2024-03-28T00:34:55.509 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-28T00:34:55.509 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625243 --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} --owner mchangir --name mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi -- /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625243/orig.config.yaml 2024-03-28T00:34:55.567 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T00:34:55.657 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625243 branch: wip-mchangir-testing-main-20240318.032620 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} email: mchangir@redhat.com first_in_suite: false job_id: '7625243' kernel: client: sha1: distro flavor: default kdb: true sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-testing-main-20240318.032620 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - slow metadata IO - SLOW_OPS - slow request sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt dry_run: true install: ceph: flavor: default sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-testing-main-20240318.032620 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 owner: mchangir priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3645 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 sleep_before_teardown: 0 subset: 132/256 suite: fs suite_branch: wip-mchangir-testing-main-20240318.032620 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi169.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDvqBRAdkinW6eathrE4TDlDVrjB27UcWoRVSmCzPzKFCw0tUqXvPEzss8vK65QBSeXuzWno0S74/iYffI/OgUM= smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE6h7J0lVtBzjACbHrnBa3xMal6Zg049NPtROAFInqMLyH5aSYl2cAGBPCnh7kVO8vNtaKX4kn4SDH7uhGbCog0= smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAK9MjTg33ou9JzFIbnx85ybNfxorqE+LxEwM4+HOpi46iUBIbOrRDZmLvuz2SuqIBSPQbDQhdc056rFTG9dNfQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/iogen.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-27_03:45:23 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-28T00:34:55.658 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa; will attempt to use it 2024-03-28T00:34:55.659 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks 2024-03-28T00:34:55.659 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-28T00:34:55.660 INFO:teuthology.task.internal:Checking packages... 2024-03-28T00:34:55.685 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3be9a5c9bc793e11e2800a8c0c696e8b46742033' 2024-03-28T00:34:55.685 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-28T00:34:55.685 INFO:teuthology.packaging:ref: None 2024-03-28T00:34:55.685 INFO:teuthology.packaging:tag: None 2024-03-28T00:34:55.685 INFO:teuthology.packaging:branch: wip-mchangir-testing-main-20240318.032620 2024-03-28T00:34:55.685 INFO:teuthology.packaging:sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T00:34:55.685 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-testing-main-20240318.032620 2024-03-28T00:34:55.833 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2145.g3be9a5c9 2024-03-28T00:34:55.835 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-28T00:34:55.841 INFO:teuthology.task.internal:no buildpackages task found 2024-03-28T00:34:55.841 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-28T00:34:55.846 INFO:teuthology.task.internal:Saving configuration 2024-03-28T00:34:55.865 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-28T00:34:55.870 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T00:34:55.896 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625243', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 00:29:37.922069', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDvqBRAdkinW6eathrE4TDlDVrjB27UcWoRVSmCzPzKFCw0tUqXvPEzss8vK65QBSeXuzWno0S74/iYffI/OgUM='} 2024-03-28T00:34:55.920 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625243', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 00:29:37.923462', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE6h7J0lVtBzjACbHrnBa3xMal6Zg049NPtROAFInqMLyH5aSYl2cAGBPCnh7kVO8vNtaKX4kn4SDH7uhGbCog0='} 2024-03-28T00:34:55.943 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625243', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 00:29:37.924661', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAK9MjTg33ou9JzFIbnx85ybNfxorqE+LxEwM4+HOpi46iUBIbOrRDZmLvuz2SuqIBSPQbDQhdc056rFTG9dNfQ='} 2024-03-28T00:34:55.944 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-28T00:34:55.949 INFO:teuthology.task.internal:roles: ubuntu@smithi169.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-28T00:34:55.949 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-28T00:34:55.949 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-28T00:34:55.949 INFO:teuthology.run_tasks:Running task console_log... 2024-03-28T00:34:56.054 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9b86263f70>, signals=[15]) 2024-03-28T00:34:56.055 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-28T00:34:56.060 INFO:teuthology.task.internal:Opening connections... 2024-03-28T00:34:56.060 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi169.front.sepia.ceph.com 2024-03-28T00:34:56.062 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T00:34:56.151 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi174.front.sepia.ceph.com 2024-03-28T00:34:56.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T00:34:56.240 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2024-03-28T00:34:56.241 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T00:34:56.330 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-28T00:34:56.336 DEBUG:teuthology.orchestra.run.smithi169:> uname -m 2024-03-28T00:34:56.353 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64 2024-03-28T00:34:56.354 DEBUG:teuthology.orchestra.run.smithi169:> cat /etc/os-release 2024-03-28T00:34:56.411 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="CentOS Stream" 2024-03-28T00:34:56.411 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="9" 2024-03-28T00:34:56.411 INFO:teuthology.orchestra.run.smithi169.stdout:ID="centos" 2024-03-28T00:34:56.412 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE="rhel fedora" 2024-03-28T00:34:56.412 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="9" 2024-03-28T00:34:56.412 INFO:teuthology.orchestra.run.smithi169.stdout:PLATFORM_ID="platform:el9" 2024-03-28T00:34:56.412 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T00:34:56.412 INFO:teuthology.orchestra.run.smithi169.stdout:ANSI_COLOR="0;31" 2024-03-28T00:34:56.412 INFO:teuthology.orchestra.run.smithi169.stdout:LOGO="fedora-logo-icon" 2024-03-28T00:34:56.412 INFO:teuthology.orchestra.run.smithi169.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T00:34:56.412 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://centos.org/" 2024-03-28T00:34:56.412 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T00:34:56.412 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T00:34:56.412 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T00:34:56.413 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2024-03-28T00:34:56.434 DEBUG:teuthology.orchestra.run.smithi174:> uname -m 2024-03-28T00:34:56.452 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2024-03-28T00:34:56.452 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release 2024-03-28T00:34:56.509 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="CentOS Stream" 2024-03-28T00:34:56.509 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="9" 2024-03-28T00:34:56.509 INFO:teuthology.orchestra.run.smithi174.stdout:ID="centos" 2024-03-28T00:34:56.509 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE="rhel fedora" 2024-03-28T00:34:56.509 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="9" 2024-03-28T00:34:56.509 INFO:teuthology.orchestra.run.smithi174.stdout:PLATFORM_ID="platform:el9" 2024-03-28T00:34:56.509 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T00:34:56.509 INFO:teuthology.orchestra.run.smithi174.stdout:ANSI_COLOR="0;31" 2024-03-28T00:34:56.509 INFO:teuthology.orchestra.run.smithi174.stdout:LOGO="fedora-logo-icon" 2024-03-28T00:34:56.509 INFO:teuthology.orchestra.run.smithi174.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T00:34:56.509 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://centos.org/" 2024-03-28T00:34:56.509 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T00:34:56.510 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T00:34:56.510 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T00:34:56.510 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2024-03-28T00:34:56.534 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2024-03-28T00:34:56.553 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2024-03-28T00:34:56.553 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-03-28T00:34:56.610 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream" 2024-03-28T00:34:56.610 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="9" 2024-03-28T00:34:56.610 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos" 2024-03-28T00:34:56.611 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora" 2024-03-28T00:34:56.611 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="9" 2024-03-28T00:34:56.611 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el9" 2024-03-28T00:34:56.611 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T00:34:56.611 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2024-03-28T00:34:56.611 INFO:teuthology.orchestra.run.smithi177.stdout:LOGO="fedora-logo-icon" 2024-03-28T00:34:56.611 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T00:34:56.611 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/" 2024-03-28T00:34:56.611 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T00:34:56.611 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T00:34:56.611 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T00:34:56.612 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-03-28T00:34:56.634 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-28T00:34:56.644 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-28T00:34:56.680 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-28T00:34:56.681 DEBUG:teuthology.orchestra.run.smithi169:> test '!' -e /home/ubuntu/cephtest 2024-03-28T00:34:56.685 DEBUG:teuthology.orchestra.run.smithi174:> test '!' -e /home/ubuntu/cephtest 2024-03-28T00:34:56.688 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2024-03-28T00:34:56.704 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-28T00:34:56.710 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-28T00:34:56.710 DEBUG:teuthology.orchestra.run.smithi169:> test -z $(ls -A /var/lib/ceph) 2024-03-28T00:34:56.739 DEBUG:teuthology.orchestra.run.smithi174:> test -z $(ls -A /var/lib/ceph) 2024-03-28T00:34:56.745 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2024-03-28T00:34:56.844 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-28T00:34:56.925 INFO:teuthology.run_tasks:Running task kernel... 2024-03-28T00:34:56.939 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'} 2024-03-28T00:34:56.940 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-28T00:34:56.940 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-28T00:34:56.940 DEBUG:teuthology.orchestra.run.smithi169:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T00:34:56.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T00:34:56.957 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-03-28T00:34:57.014 INFO:teuthology.orchestra.run.smithi169.stdout:5.14.0-361.el9.x86_64 2024-03-28T00:34:57.015 INFO:teuthology.task.kernel:Running kernel on smithi169: 5.14.0-361.el9.x86_64 2024-03-28T00:34:57.015 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum install -y kernel 2024-03-28T00:34:59.224 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-28T00:35:00.641 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-28T00:35:02.390 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 8.0 MB 00:01 2024-03-28T00:35:05.011 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-28T00:35:08.883 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.1 MB 00:00 2024-03-28T00:35:10.492 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-28T00:35:12.336 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-28T00:35:17.327 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 8.3 kB/s | 1.7 kB 00:00 2024-03-28T00:35:19.475 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-28T00:35:19.475 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-28T00:35:19.586 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-28T00:35:19.597 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-28T00:35:19.597 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-03-28T00:35:19.597 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-28T00:35:19.597 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-28T00:35:19.597 INFO:teuthology.orchestra.run.smithi169.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-28T00:35:19.597 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-28T00:35:19.597 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-28T00:35:19.597 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-28T00:35:19.597 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-28T00:35:19.597 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:35:19.598 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-28T00:35:19.598 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-28T00:35:19.598 INFO:teuthology.orchestra.run.smithi169.stdout:Install 4 Packages 2024-03-28T00:35:19.598 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:35:19.598 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 76 M 2024-03-28T00:35:19.598 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 123 M 2024-03-28T00:35:19.598 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-28T00:35:20.095 INFO:teuthology.orchestra.run.smithi169.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 172 kB/s | 66 kB 00:00 2024-03-28T00:35:34.635 INFO:teuthology.orchestra.run.smithi169.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 1.0 MB/s | 15 MB 00:14 2024-03-28T00:35:40.576 INFO:teuthology.orchestra.run.smithi169.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 1.6 MB/s | 33 MB 00:20 2024-03-28T00:35:48.975 INFO:teuthology.orchestra.run.smithi169.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 975 kB/s | 27 MB 00:28 2024-03-28T00:35:48.975 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-28T00:35:48.975 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.6 MB/s | 76 MB 00:29 2024-03-28T00:35:49.643 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-28T00:35:50.002 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-28T00:35:50.002 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-28T00:35:51.040 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-28T00:35:51.041 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-28T00:35:53.509 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-28T00:35:54.116 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-28T00:35:54.209 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T00:35:56.923 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T00:35:57.063 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T00:36:01.749 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T00:36:01.854 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-28T00:36:06.581 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T00:36:35.750 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T00:36:35.751 INFO:teuthology.orchestra.run.smithi169.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T00:36:35.751 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:36:35.792 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-28T00:36:36.167 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-28T00:36:36.167 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-28T00:36:36.167 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T00:36:36.167 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T00:36:36.564 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T00:36:36.564 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:36:36.564 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-28T00:36:36.564 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-28T00:36:36.564 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-28T00:36:36.564 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-28T00:36:36.564 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-28T00:36:36.564 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:36:36.564 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-28T00:36:36.903 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T00:36:37.006 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-28T00:36:37.006 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-28T00:36:37.006 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-28T00:36:37.006 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-28T00:36:37.006 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-28T00:36:37.007 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-28T00:36:37.007 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi169.front.sepia.ceph.com, path=None, version=distro) 2024-03-28T00:36:37.007 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T00:36:37.087 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-28T00:36:37.087 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-28T00:36:37.087 DEBUG:teuthology.orchestra.run.smithi169:> sudo rpm -qi grub2-tools 2024-03-28T00:36:37.121 INFO:teuthology.orchestra.run.smithi169.stdout:Name : grub2-tools 2024-03-28T00:36:37.121 INFO:teuthology.orchestra.run.smithi169.stdout:Epoch : 1 2024-03-28T00:36:37.122 INFO:teuthology.orchestra.run.smithi169.stdout:Version : 2.06 2024-03-28T00:36:37.122 INFO:teuthology.orchestra.run.smithi169.stdout:Release : 68.el9 2024-03-28T00:36:37.122 INFO:teuthology.orchestra.run.smithi169.stdout:Architecture: x86_64 2024-03-28T00:36:37.122 INFO:teuthology.orchestra.run.smithi169.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T00:36:37.122 INFO:teuthology.orchestra.run.smithi169.stdout:Group : Unspecified 2024-03-28T00:36:37.122 INFO:teuthology.orchestra.run.smithi169.stdout:Size : 8264254 2024-03-28T00:36:37.122 INFO:teuthology.orchestra.run.smithi169.stdout:License : GPLv3+ 2024-03-28T00:36:37.122 INFO:teuthology.orchestra.run.smithi169.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T00:36:37.122 INFO:teuthology.orchestra.run.smithi169.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T00:36:37.122 INFO:teuthology.orchestra.run.smithi169.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T00:36:37.122 INFO:teuthology.orchestra.run.smithi169.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T00:36:37.122 INFO:teuthology.orchestra.run.smithi169.stdout:Packager : builder@centos.org 2024-03-28T00:36:37.122 INFO:teuthology.orchestra.run.smithi169.stdout:Vendor : CentOS 2024-03-28T00:36:37.122 INFO:teuthology.orchestra.run.smithi169.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T00:36:37.123 INFO:teuthology.orchestra.run.smithi169.stdout:Summary : Support tools for GRUB. 2024-03-28T00:36:37.123 INFO:teuthology.orchestra.run.smithi169.stdout:Description : 2024-03-28T00:36:37.123 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:36:37.123 INFO:teuthology.orchestra.run.smithi169.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T00:36:37.123 INFO:teuthology.orchestra.run.smithi169.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T00:36:37.123 INFO:teuthology.orchestra.run.smithi169.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T00:36:37.123 INFO:teuthology.orchestra.run.smithi169.stdout:hardware devices. 2024-03-28T00:36:37.123 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:36:37.123 INFO:teuthology.orchestra.run.smithi169.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T00:36:37.124 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T00:36:37.124 INFO:teuthology.task.kernel:Updating grub on smithi169 to boot 5.14.0-432.el9.x86_64 2024-03-28T00:36:37.124 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T00:36:37.757 INFO:teuthology.orchestra.run.smithi169.stderr:Generating grub configuration file ... 2024-03-28T00:36:39.182 INFO:teuthology.orchestra.run.smithi169.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T00:36:39.215 INFO:teuthology.orchestra.run.smithi169.stderr:done 2024-03-28T00:36:39.218 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2024-03-28T00:36:39.234 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.idAMtzamFm 2024-03-28T00:36:39.235 DEBUG:teuthology.orchestra.run.smithi169:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.idAMtzamFm 2024-03-28T00:36:39.303 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.idAMtzamFm 2024-03-28T00:36:39.441 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.idAMtzamFm is 6KB 2024-03-28T00:36:39.490 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.idAMtzamFm 2024-03-28T00:36:39.504 DEBUG:teuthology.orchestra.run.smithi169:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T00:36:39.576 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T00:36:39.576 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T00:36:39.576 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T00:36:39.576 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-28T00:36:39.577 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-28T00:36:39.736 DEBUG:teuthology.orchestra.run.smithi169:> sudo shutdown -r now 2024-03-28T00:37:09.738 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-28T00:37:09.739 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-28T00:37:09.740 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T00:37:28.222 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-03-28T00:37:37.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-28T00:37:37.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T00:37:40.286 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-03-28T00:37:52.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-28T00:37:52.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T00:38:10.718 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-03-28T00:38:25.720 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-28T00:38:25.721 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T00:38:26.076 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-03-28T00:38:26.540 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-28T00:38:26.540 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-432.el9.x86_64"... 2024-03-28T00:38:26.540 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-03-28T00:38:26.555 INFO:teuthology.orchestra.run.smithi169.stdout:5.14.0-432.el9.x86_64 2024-03-28T00:38:26.555 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-28T00:38:26.555 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T00:38:26.555 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T00:38:27.556 DEBUG:teuthology.parallel:result is None 2024-03-28T00:38:27.556 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-28T00:38:27.566 INFO:teuthology.task.internal:Creating test directory... 2024-03-28T00:38:27.566 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T00:38:27.569 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T00:38:27.572 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T00:38:27.590 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-28T00:38:27.596 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-28T00:38:27.604 INFO:teuthology.task.internal:Creating archive directory... 2024-03-28T00:38:27.604 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T00:38:27.624 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T00:38:27.630 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T00:38:27.725 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-28T00:38:27.732 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-28T00:38:27.732 DEBUG:teuthology.orchestra.run.smithi174:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T00:38:27.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T00:38:27.749 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T00:38:27.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T00:38:27.765 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T00:38:27.768 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T00:38:27.792 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T00:38:27.826 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T00:38:27.843 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T00:38:27.845 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T00:38:27.863 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T00:38:27.868 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T00:38:27.958 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T00:38:27.959 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-28T00:38:27.967 INFO:teuthology.task.internal:Configuring sudo... 2024-03-28T00:38:27.967 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T00:38:27.969 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T00:38:27.971 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T00:38:28.001 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-28T00:38:28.011 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-28T00:38:28.011 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T00:38:28.037 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T00:38:28.042 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T00:38:28.057 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T00:38:28.132 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T00:38:28.180 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T00:38:28.259 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T00:38:28.299 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-28T00:38:28.300 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T00:38:28.363 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T00:38:28.406 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T00:38:28.462 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T00:38:28.545 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T00:38:28.590 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-28T00:38:28.590 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T00:38:28.658 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T00:38:28.703 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T00:38:28.757 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T00:38:28.842 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T00:38:28.885 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-28T00:38:28.885 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T00:38:28.954 DEBUG:teuthology.orchestra.run.smithi169:> sudo service rsyslog restart 2024-03-28T00:38:28.956 DEBUG:teuthology.orchestra.run.smithi174:> sudo service rsyslog restart 2024-03-28T00:38:28.958 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2024-03-28T00:38:29.000 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T00:38:29.009 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T00:38:29.034 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T00:38:29.473 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-28T00:38:29.482 INFO:teuthology.task.internal:Starting timer... 2024-03-28T00:38:29.482 INFO:teuthology.run_tasks:Running task pcp... 2024-03-28T00:38:29.492 INFO:teuthology.run_tasks:Running task selinux... 2024-03-28T00:38:29.501 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-28T00:38:29.502 DEBUG:teuthology.orchestra.run.smithi169:> sudo service auditd rotate 2024-03-28T00:38:29.570 INFO:teuthology.orchestra.run.smithi169.stdout:Rotating logs: 2024-03-28T00:38:29.572 DEBUG:teuthology.orchestra.run.smithi174:> sudo service auditd rotate 2024-03-28T00:38:29.634 INFO:teuthology.orchestra.run.smithi174.stdout:Rotating logs: 2024-03-28T00:38:29.636 DEBUG:teuthology.orchestra.run.smithi177:> sudo service auditd rotate 2024-03-28T00:38:29.690 INFO:teuthology.orchestra.run.smithi177.stdout:Rotating logs: 2024-03-28T00:38:29.692 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-28T00:38:29.692 DEBUG:teuthology.orchestra.run.smithi169:> /usr/sbin/getenforce 2024-03-28T00:38:29.709 INFO:teuthology.orchestra.run.smithi169.stdout:Permissive 2024-03-28T00:38:29.709 DEBUG:teuthology.orchestra.run.smithi174:> /usr/sbin/getenforce 2024-03-28T00:38:29.728 INFO:teuthology.orchestra.run.smithi174.stdout:Permissive 2024-03-28T00:38:29.728 DEBUG:teuthology.orchestra.run.smithi177:> /usr/sbin/getenforce 2024-03-28T00:38:29.752 INFO:teuthology.orchestra.run.smithi177.stdout:Permissive 2024-03-28T00:38:29.752 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi169.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi174.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi177.front.sepia.ceph.com': 'permissive'} 2024-03-28T00:38:29.753 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T00:38:29.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T00:38:29.780 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T00:38:29.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T00:38:29.809 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T00:38:29.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T00:38:29.837 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-28T00:38:29.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo /usr/sbin/setenforce permissive 2024-03-28T00:38:29.878 DEBUG:teuthology.orchestra.run.smithi174:> sudo /usr/sbin/setenforce permissive 2024-03-28T00:38:29.914 DEBUG:teuthology.orchestra.run.smithi177:> sudo /usr/sbin/setenforce permissive 2024-03-28T00:38:29.946 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-28T00:38:29.956 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-28T00:38:29.973 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-28T00:38:29.974 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi169.front.sepia.ceph.com,smithi174.front.sepia.ceph.com,smithi177.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-28T00:43:35.442 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi169.front.sepia.ceph.com'), Remote(name='ubuntu@smithi174.front.sepia.ceph.com'), Remote(name='ubuntu@smithi177.front.sepia.ceph.com')] 2024-03-28T00:43:35.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-28T00:43:35.445 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T00:43:35.535 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-03-28T00:43:35.617 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-28T00:43:35.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-28T00:43:35.618 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T00:43:35.706 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-03-28T00:43:35.773 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-28T00:43:35.773 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-28T00:43:35.774 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T00:43:35.861 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-03-28T00:43:35.932 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-28T00:43:35.932 INFO:teuthology.run_tasks:Running task clock... 2024-03-28T00:43:35.945 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-28T00:43:35.945 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T00:43:35.946 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T00:43:35.948 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T00:43:35.948 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T00:43:35.950 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T00:43:35.951 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T00:43:35.978 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T00:43:35.980 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T00:43:35.994 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T00:43:35.996 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T00:43:36.002 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T00:43:36.019 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T00:43:36.024 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: ntpd: command not found 2024-03-28T00:43:36.026 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: ntpd: command not found 2024-03-28T00:43:36.037 INFO:teuthology.orchestra.run.smithi169.stdout:506 Cannot talk to daemon 2024-03-28T00:43:36.040 INFO:teuthology.orchestra.run.smithi174.stdout:506 Cannot talk to daemon 2024-03-28T00:43:36.048 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: ntpd: command not found 2024-03-28T00:43:36.053 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T00:43:36.057 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T00:43:36.061 INFO:teuthology.orchestra.run.smithi177.stdout:506 Cannot talk to daemon 2024-03-28T00:43:36.070 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T00:43:36.073 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T00:43:36.078 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T00:43:36.094 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T00:43:36.117 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-03-28T00:43:36.120 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T00:43:36.120 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-03-28T00:43:36.120 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T00:43:36.121 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T00:43:36.121 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T00:43:36.121 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T00:43:36.123 INFO:teuthology.orchestra.run.smithi169.stderr:bash: line 1: ntpq: command not found 2024-03-28T00:43:36.126 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T00:43:36.126 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-03-28T00:43:36.126 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T00:43:36.127 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T00:43:36.127 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T00:43:36.127 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T00:43:36.137 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-03-28T00:43:36.140 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T00:43:36.141 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-03-28T00:43:36.141 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T00:43:36.141 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T00:43:36.141 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T00:43:36.141 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T00:43:36.142 INFO:teuthology.run_tasks:Running task install... 2024-03-28T00:43:36.151 DEBUG:teuthology.task.install:project ceph 2024-03-28T00:43:36.151 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}} 2024-03-28T00:43:36.151 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'} 2024-03-28T00:43:36.152 INFO:teuthology.task.install:Using flavor: default 2024-03-28T00:43:36.160 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-28T00:43:36.160 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-28T00:43:36.160 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T00:43:36.161 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T00:43:36.163 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T00:43:36.163 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T00:43:36.164 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T00:43:36.164 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T00:43:36.310 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T00:43:36.310 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T00:43:36.315 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T00:43:36.315 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T00:43:36.380 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T00:43:36.380 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T00:43:36.482 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T00:43:36.482 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-28T00:43:36.482 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T00:43:36.487 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T00:43:36.488 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-28T00:43:36.488 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T00:43:36.493 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T00:43:36.493 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-28T00:43:36.493 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T00:43:36.510 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T00:43:36.510 DEBUG:teuthology.orchestra.run.smithi169:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T00:43:36.516 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T00:43:36.516 DEBUG:teuthology.orchestra.run.smithi174:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T00:43:36.520 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T00:43:36.521 DEBUG:teuthology.orchestra.run.smithi177:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T00:43:36.578 DEBUG:teuthology.orchestra.run.smithi169:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T00:43:36.583 DEBUG:teuthology.orchestra.run.smithi174:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T00:43:36.588 DEBUG:teuthology.orchestra.run.smithi177:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T00:43:36.658 DEBUG:teuthology.orchestra.run.smithi169:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T00:43:36.663 DEBUG:teuthology.orchestra.run.smithi174:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T00:43:36.667 DEBUG:teuthology.orchestra.run.smithi177:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T00:43:36.724 INFO:teuthology.orchestra.run.smithi169.stdout:check_obsoletes = 1 2024-03-28T00:43:36.726 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum clean all 2024-03-28T00:43:36.729 INFO:teuthology.orchestra.run.smithi174.stdout:check_obsoletes = 1 2024-03-28T00:43:36.731 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum clean all 2024-03-28T00:43:36.733 INFO:teuthology.orchestra.run.smithi177.stdout:check_obsoletes = 1 2024-03-28T00:43:36.735 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum clean all 2024-03-28T00:43:36.964 INFO:teuthology.orchestra.run.smithi174.stdout:74 files removed 2024-03-28T00:43:36.968 INFO:teuthology.orchestra.run.smithi177.stdout:74 files removed 2024-03-28T00:43:36.969 INFO:teuthology.orchestra.run.smithi169.stdout:74 files removed 2024-03-28T00:43:36.991 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T00:43:36.994 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T00:43:36.995 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T00:43:37.701 INFO:teuthology.orchestra.run.smithi169.stdout:ceph packages for x86_64 205 kB/s | 83 kB 00:00 2024-03-28T00:43:37.763 INFO:teuthology.orchestra.run.smithi174.stdout:ceph packages for x86_64 172 kB/s | 83 kB 00:00 2024-03-28T00:43:37.936 INFO:teuthology.orchestra.run.smithi177.stdout:ceph packages for x86_64 127 kB/s | 83 kB 00:00 2024-03-28T00:43:38.068 INFO:teuthology.orchestra.run.smithi169.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-28T00:43:38.164 INFO:teuthology.orchestra.run.smithi174.stdout:ceph noarch packages 28 kB/s | 10 kB 00:00 2024-03-28T00:43:38.377 INFO:teuthology.orchestra.run.smithi169.stdout:ceph source packages 7.0 kB/s | 1.9 kB 00:00 2024-03-28T00:43:38.453 INFO:teuthology.orchestra.run.smithi177.stdout:ceph noarch packages 21 kB/s | 10 kB 00:00 2024-03-28T00:43:38.473 INFO:teuthology.orchestra.run.smithi174.stdout:ceph source packages 6.8 kB/s | 1.9 kB 00:00 2024-03-28T00:43:38.854 INFO:teuthology.orchestra.run.smithi177.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-03-28T00:43:39.213 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - AppStream 8.4 MB/s | 6.7 MB 00:00 2024-03-28T00:43:39.650 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS-9 - AppStream 5.8 MB/s | 6.7 MB 00:01 2024-03-28T00:43:40.049 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - AppStream 5.7 MB/s | 6.7 MB 00:01 2024-03-28T00:43:41.681 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - BaseOS 1.5 MB/s | 2.0 MB 00:01 2024-03-28T00:43:41.910 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS-9 - BaseOS 1.7 MB/s | 2.0 MB 00:01 2024-03-28T00:43:42.207 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - BaseOS 1.9 MB/s | 2.0 MB 00:01 2024-03-28T00:43:42.438 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-28T00:43:43.774 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.0 MB 00:01 2024-03-28T00:43:45.097 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-28T00:43:45.409 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - AppStream 32 MB/s | 19 MB 00:00 2024-03-28T00:43:49.750 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - CRB 5.5 MB/s | 6.1 MB 00:01 2024-03-28T00:43:51.107 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 16 kB 00:00 2024-03-28T00:43:51.680 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 901 kB/s | 8.0 MB 00:09 2024-03-28T00:43:52.167 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-28T00:43:56.734 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - CRB 800 kB/s | 6.1 MB 00:07 2024-03-28T00:43:57.342 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 5.9 kB/s | 1.7 kB 00:00 2024-03-28T00:43:58.176 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 16 kB 00:00 2024-03-28T00:43:58.871 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 3.1 MB/s | 19 MB 00:06 2024-03-28T00:43:59.761 INFO:teuthology.orchestra.run.smithi169.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T00:43:59.762 INFO:teuthology.orchestra.run.smithi169.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T00:43:59.774 INFO:teuthology.orchestra.run.smithi169.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T00:43:59.775 INFO:teuthology.orchestra.run.smithi169.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T00:43:59.776 INFO:teuthology.orchestra.run.smithi169.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T00:43:59.777 INFO:teuthology.orchestra.run.smithi169.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T00:43:59.779 INFO:teuthology.orchestra.run.smithi169.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T00:43:59.781 INFO:teuthology.orchestra.run.smithi169.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T00:43:59.782 INFO:teuthology.orchestra.run.smithi169.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T00:43:59.865 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-28T00:43:59.889 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-03-28T00:43:59.889 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-28T00:43:59.889 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-03-28T00:43:59.889 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-28T00:43:59.889 INFO:teuthology.orchestra.run.smithi169.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T00:43:59.889 INFO:teuthology.orchestra.run.smithi169.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T00:43:59.889 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T00:43:59.889 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T00:43:59.889 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T00:43:59.889 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T00:43:59.890 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T00:43:59.890 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T00:43:59.890 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T00:43:59.890 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T00:43:59.890 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T00:43:59.890 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T00:43:59.890 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T00:43:59.890 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T00:43:59.890 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T00:43:59.890 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T00:43:59.890 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T00:43:59.891 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T00:43:59.891 INFO:teuthology.orchestra.run.smithi169.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T00:43:59.891 INFO:teuthology.orchestra.run.smithi169.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T00:43:59.891 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T00:43:59.891 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T00:43:59.891 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T00:43:59.891 INFO:teuthology.orchestra.run.smithi169.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T00:43:59.891 INFO:teuthology.orchestra.run.smithi169.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T00:43:59.891 INFO:teuthology.orchestra.run.smithi169.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T00:43:59.891 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T00:43:59.891 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T00:43:59.892 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T00:43:59.892 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T00:43:59.892 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T00:43:59.892 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T00:43:59.892 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T00:43:59.892 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T00:43:59.892 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T00:43:59.892 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T00:43:59.892 INFO:teuthology.orchestra.run.smithi169.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T00:43:59.892 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrading: 2024-03-28T00:43:59.892 INFO:teuthology.orchestra.run.smithi169.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T00:43:59.892 INFO:teuthology.orchestra.run.smithi169.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T00:43:59.893 INFO:teuthology.orchestra.run.smithi169.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T00:43:59.893 INFO:teuthology.orchestra.run.smithi169.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T00:43:59.893 INFO:teuthology.orchestra.run.smithi169.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T00:43:59.893 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T00:43:59.893 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T00:43:59.893 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T00:43:59.893 INFO:teuthology.orchestra.run.smithi169.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T00:43:59.893 INFO:teuthology.orchestra.run.smithi169.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T00:43:59.893 INFO:teuthology.orchestra.run.smithi169.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T00:43:59.893 INFO:teuthology.orchestra.run.smithi169.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T00:43:59.893 INFO:teuthology.orchestra.run.smithi169.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T00:43:59.893 INFO:teuthology.orchestra.run.smithi169.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T00:43:59.893 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T00:43:59.893 INFO:teuthology.orchestra.run.smithi169.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T00:43:59.894 INFO:teuthology.orchestra.run.smithi169.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T00:43:59.894 INFO:teuthology.orchestra.run.smithi169.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T00:43:59.894 INFO:teuthology.orchestra.run.smithi169.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T00:43:59.894 INFO:teuthology.orchestra.run.smithi169.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T00:43:59.894 INFO:teuthology.orchestra.run.smithi169.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T00:43:59.894 INFO:teuthology.orchestra.run.smithi169.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T00:43:59.894 INFO:teuthology.orchestra.run.smithi169.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T00:43:59.894 INFO:teuthology.orchestra.run.smithi169.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T00:43:59.894 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-28T00:43:59.894 INFO:teuthology.orchestra.run.smithi169.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T00:43:59.894 INFO:teuthology.orchestra.run.smithi169.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T00:43:59.895 INFO:teuthology.orchestra.run.smithi169.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T00:43:59.895 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T00:43:59.895 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T00:43:59.895 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T00:43:59.895 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T00:43:59.895 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T00:43:59.895 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T00:43:59.895 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T00:43:59.895 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T00:43:59.895 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T00:43:59.895 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T00:43:59.895 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T00:43:59.895 INFO:teuthology.orchestra.run.smithi169.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T00:43:59.896 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T00:43:59.896 INFO:teuthology.orchestra.run.smithi169.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T00:43:59.896 INFO:teuthology.orchestra.run.smithi169.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T00:43:59.896 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T00:43:59.896 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T00:43:59.896 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T00:43:59.896 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T00:43:59.896 INFO:teuthology.orchestra.run.smithi169.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T00:43:59.896 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T00:43:59.896 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T00:43:59.896 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T00:43:59.897 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T00:43:59.897 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T00:43:59.897 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T00:43:59.897 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T00:43:59.897 INFO:teuthology.orchestra.run.smithi169.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T00:43:59.897 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T00:43:59.897 INFO:teuthology.orchestra.run.smithi169.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T00:43:59.897 INFO:teuthology.orchestra.run.smithi169.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T00:43:59.897 INFO:teuthology.orchestra.run.smithi169.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T00:43:59.897 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T00:43:59.897 INFO:teuthology.orchestra.run.smithi169.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T00:43:59.897 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T00:43:59.897 INFO:teuthology.orchestra.run.smithi169.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T00:43:59.898 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T00:43:59.898 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T00:43:59.898 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T00:43:59.898 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T00:43:59.898 INFO:teuthology.orchestra.run.smithi169.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T00:43:59.898 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T00:43:59.898 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T00:43:59.898 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T00:43:59.898 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T00:43:59.898 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T00:43:59.898 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T00:43:59.898 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T00:43:59.898 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T00:43:59.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T00:43:59.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T00:43:59.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T00:43:59.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T00:43:59.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T00:43:59.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T00:43:59.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T00:43:59.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T00:43:59.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T00:43:59.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T00:43:59.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T00:43:59.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T00:43:59.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T00:43:59.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T00:43:59.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T00:43:59.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T00:43:59.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T00:43:59.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T00:43:59.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T00:43:59.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T00:43:59.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T00:43:59.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T00:43:59.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T00:43:59.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T00:43:59.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T00:43:59.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T00:43:59.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T00:43:59.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T00:43:59.901 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T00:43:59.901 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T00:43:59.901 INFO:teuthology.orchestra.run.smithi169.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T00:43:59.901 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T00:43:59.901 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T00:43:59.901 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T00:43:59.901 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T00:43:59.901 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T00:43:59.901 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T00:43:59.901 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T00:43:59.901 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T00:43:59.901 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T00:43:59.901 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T00:43:59.901 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T00:43:59.901 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T00:43:59.902 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T00:43:59.902 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T00:43:59.902 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T00:43:59.902 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T00:43:59.902 INFO:teuthology.orchestra.run.smithi169.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T00:43:59.902 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T00:43:59.902 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T00:43:59.902 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T00:43:59.902 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T00:43:59.902 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T00:43:59.902 INFO:teuthology.orchestra.run.smithi169.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T00:43:59.902 INFO:teuthology.orchestra.run.smithi169.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T00:43:59.902 INFO:teuthology.orchestra.run.smithi169.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T00:43:59.903 INFO:teuthology.orchestra.run.smithi169.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T00:43:59.903 INFO:teuthology.orchestra.run.smithi169.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T00:43:59.903 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T00:43:59.903 INFO:teuthology.orchestra.run.smithi169.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T00:43:59.903 INFO:teuthology.orchestra.run.smithi169.stdout:Installing weak dependencies: 2024-03-28T00:43:59.903 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T00:43:59.903 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:43:59.903 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-28T00:43:59.903 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-03-28T00:43:59.903 INFO:teuthology.orchestra.run.smithi169.stdout:Install 150 Packages 2024-03-28T00:43:59.903 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrade 24 Packages 2024-03-28T00:43:59.903 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:43:59.904 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 225 M 2024-03-28T00:43:59.904 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-28T00:44:01.060 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 7.5 MB/s | 21 MB 00:02 2024-03-28T00:44:01.276 INFO:teuthology.orchestra.run.smithi169.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 13 kB/s | 6.6 kB 00:00 2024-03-28T00:44:01.643 INFO:teuthology.orchestra.run.smithi169.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 2.4 MB/s | 890 kB 00:00 2024-03-28T00:44:01.752 INFO:teuthology.orchestra.run.smithi169.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 1.3 MB/s | 145 kB 00:00 2024-03-28T00:44:02.194 INFO:teuthology.orchestra.run.smithi169.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 5.2 MB/s | 2.3 MB 00:00 2024-03-28T00:44:02.502 INFO:teuthology.orchestra.run.smithi169.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 4.9 MB/s | 1.5 MB 00:00 2024-03-28T00:44:02.769 INFO:teuthology.orchestra.run.smithi169.stdout:(6/174): ceph-common-19.0.0-2145.g3be9a5c9.el9. 10 MB/s | 20 MB 00:02 2024-03-28T00:44:02.853 INFO:teuthology.orchestra.run.smithi169.stdout:(7/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 2.6 MB/s | 5.4 MB 00:02 2024-03-28T00:44:03.245 INFO:teuthology.orchestra.run.smithi169.stdout:(8/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 6.3 MB/s | 4.7 MB 00:00 2024-03-28T00:44:03.328 INFO:teuthology.orchestra.run.smithi169.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 302 kB/s | 25 kB 00:00 2024-03-28T00:44:03.398 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 5.7 MB/s | 6.1 MB 00:01 2024-03-28T00:44:03.878 INFO:teuthology.orchestra.run.smithi169.stdout:(10/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 15 MB/s | 17 MB 00:01 2024-03-28T00:44:03.962 INFO:teuthology.orchestra.run.smithi169.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 2.6 MB/s | 220 kB 00:00 2024-03-28T00:44:04.062 INFO:teuthology.orchestra.run.smithi169.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 270 kB/s | 27 kB 00:00 2024-03-28T00:44:04.154 INFO:teuthology.orchestra.run.smithi169.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 341 kB/s | 31 kB 00:00 2024-03-28T00:44:04.263 INFO:teuthology.orchestra.run.smithi169.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 6.7 MB/s | 746 kB 00:00 2024-03-28T00:44:04.355 INFO:teuthology.orchestra.run.smithi169.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.7 MB/s | 163 kB 00:00 2024-03-28T00:44:04.438 INFO:teuthology.orchestra.run.smithi169.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.5 MB/s | 127 kB 00:00 2024-03-28T00:44:04.539 INFO:teuthology.orchestra.run.smithi169.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 4.9 MB/s | 501 kB 00:00 2024-03-28T00:44:04.989 INFO:teuthology.orchestra.run.smithi169.stdout:(18/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 11 MB/s | 5.1 MB 00:00 2024-03-28T00:44:04.997 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-28T00:44:05.064 INFO:teuthology.orchestra.run.smithi169.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 602 kB/s | 45 kB 00:00 2024-03-28T00:44:05.165 INFO:teuthology.orchestra.run.smithi169.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.3 MB/s | 131 kB 00:00 2024-03-28T00:44:05.248 INFO:teuthology.orchestra.run.smithi169.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 1.9 MB/s | 162 kB 00:00 2024-03-28T00:44:05.398 INFO:teuthology.orchestra.run.smithi169.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 2.1 MB/s | 321 kB 00:00 2024-03-28T00:44:05.507 INFO:teuthology.orchestra.run.smithi169.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 2.7 MB/s | 298 kB 00:00 2024-03-28T00:44:05.607 INFO:teuthology.orchestra.run.smithi169.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 995 kB/s | 99 kB 00:00 2024-03-28T00:44:05.708 INFO:teuthology.orchestra.run.smithi169.stdout:(25/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 853 kB/s | 85 kB 00:00 2024-03-28T00:44:05.985 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 10 kB/s | 1.7 kB 00:00 2024-03-28T00:44:06.016 INFO:teuthology.orchestra.run.smithi169.stdout:(26/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 10 MB/s | 3.1 MB 00:00 2024-03-28T00:44:06.133 INFO:teuthology.orchestra.run.smithi169.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 1.4 MB/s | 170 kB 00:00 2024-03-28T00:44:06.209 INFO:teuthology.orchestra.run.smithi169.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 452 kB/s | 34 kB 00:00 2024-03-28T00:44:06.318 INFO:teuthology.orchestra.run.smithi169.stdout:(29/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 2.8 MB/s | 9.9 MB 00:03 2024-03-28T00:44:06.343 INFO:teuthology.orchestra.run.smithi169.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 1.0 MB/s | 139 kB 00:00 2024-03-28T00:44:06.918 INFO:teuthology.orchestra.run.smithi169.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 13 MB/s | 7.4 MB 00:00 2024-03-28T00:44:06.977 INFO:teuthology.orchestra.run.smithi169.stdout:(32/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 2.7 MB/s | 1.8 MB 00:00 2024-03-28T00:44:07.002 INFO:teuthology.orchestra.run.smithi169.stdout:(33/174): ceph-mgr-modules-core-19.0.0-2145.g3b 2.9 MB/s | 248 kB 00:00 2024-03-28T00:44:07.060 INFO:teuthology.orchestra.run.smithi169.stdout:(34/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 590 kB/s | 49 kB 00:00 2024-03-28T00:44:07.086 INFO:teuthology.orchestra.run.smithi169.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2145.g3 196 kB/s | 16 kB 00:00 2024-03-28T00:44:07.194 INFO:teuthology.orchestra.run.smithi169.stdout:(36/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 6.8 MB/s | 748 kB 00:00 2024-03-28T00:44:07.207 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 21 MB 00:02 2024-03-28T00:44:07.220 INFO:teuthology.orchestra.run.smithi169.stdout:(37/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 1.6 MB/s | 264 kB 00:00 2024-03-28T00:44:07.278 INFO:teuthology.orchestra.run.smithi169.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 218 kB/s | 18 kB 00:00 2024-03-28T00:44:07.445 INFO:teuthology.orchestra.run.smithi169.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.9 MB/s | 662 kB 00:00 2024-03-28T00:44:07.496 INFO:teuthology.orchestra.run.smithi169.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 106 kB 00:00 2024-03-28T00:44:07.554 INFO:teuthology.orchestra.run.smithi169.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-03-28T00:44:07.596 INFO:teuthology.orchestra.run.smithi169.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 779 kB/s | 32 kB 00:00 2024-03-28T00:44:08.013 INFO:teuthology.orchestra.run.smithi169.stdout:(43/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 10 MB/s | 48 MB 00:04 2024-03-28T00:44:08.080 INFO:teuthology.orchestra.run.smithi169.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.2 MB/s | 3.0 MB 00:00 2024-03-28T00:44:08.097 INFO:teuthology.orchestra.run.smithi169.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 214 kB/s | 17 kB 00:00 2024-03-28T00:44:08.130 INFO:teuthology.orchestra.run.smithi169.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-03-28T00:44:08.156 INFO:teuthology.orchestra.run.smithi169.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-28T00:44:08.173 INFO:teuthology.orchestra.run.smithi169.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-28T00:44:08.232 INFO:teuthology.orchestra.run.smithi169.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-28T00:44:08.257 INFO:teuthology.orchestra.run.smithi169.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 6.7 MB/s | 578 kB 00:00 2024-03-28T00:44:08.274 INFO:teuthology.orchestra.run.smithi169.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-28T00:44:08.316 INFO:teuthology.orchestra.run.smithi169.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-28T00:44:08.341 INFO:teuthology.orchestra.run.smithi169.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-03-28T00:44:08.354 INFO:teuthology.orchestra.run.smithi174.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T00:44:08.355 INFO:teuthology.orchestra.run.smithi174.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T00:44:08.367 INFO:teuthology.orchestra.run.smithi174.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T00:44:08.367 INFO:teuthology.orchestra.run.smithi174.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T00:44:08.368 INFO:teuthology.orchestra.run.smithi174.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T00:44:08.369 INFO:teuthology.orchestra.run.smithi174.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T00:44:08.373 INFO:teuthology.orchestra.run.smithi174.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T00:44:08.374 INFO:teuthology.orchestra.run.smithi174.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T00:44:08.374 INFO:teuthology.orchestra.run.smithi174.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T00:44:08.375 INFO:teuthology.orchestra.run.smithi169.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-03-28T00:44:08.400 INFO:teuthology.orchestra.run.smithi169.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 583 kB/s | 34 kB 00:00 2024-03-28T00:44:08.453 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-28T00:44:08.468 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================================ 2024-03-28T00:44:08.469 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-03-28T00:44:08.469 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================================ 2024-03-28T00:44:08.469 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-28T00:44:08.469 INFO:teuthology.orchestra.run.smithi174.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T00:44:08.469 INFO:teuthology.orchestra.run.smithi174.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T00:44:08.469 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T00:44:08.469 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T00:44:08.469 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T00:44:08.469 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T00:44:08.469 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T00:44:08.469 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T00:44:08.469 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T00:44:08.469 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T00:44:08.469 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T00:44:08.470 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T00:44:08.470 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T00:44:08.470 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T00:44:08.470 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T00:44:08.470 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T00:44:08.470 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T00:44:08.470 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T00:44:08.470 INFO:teuthology.orchestra.run.smithi174.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T00:44:08.470 INFO:teuthology.orchestra.run.smithi174.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T00:44:08.470 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T00:44:08.470 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T00:44:08.470 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T00:44:08.471 INFO:teuthology.orchestra.run.smithi174.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T00:44:08.471 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T00:44:08.471 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T00:44:08.471 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T00:44:08.471 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T00:44:08.471 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T00:44:08.471 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T00:44:08.471 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T00:44:08.471 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T00:44:08.471 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T00:44:08.471 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T00:44:08.471 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T00:44:08.471 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T00:44:08.471 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T00:44:08.472 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrading: 2024-03-28T00:44:08.472 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T00:44:08.472 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T00:44:08.472 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T00:44:08.472 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T00:44:08.472 INFO:teuthology.orchestra.run.smithi174.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T00:44:08.472 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T00:44:08.472 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T00:44:08.472 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T00:44:08.472 INFO:teuthology.orchestra.run.smithi174.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T00:44:08.472 INFO:teuthology.orchestra.run.smithi174.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T00:44:08.472 INFO:teuthology.orchestra.run.smithi174.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T00:44:08.472 INFO:teuthology.orchestra.run.smithi174.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T00:44:08.472 INFO:teuthology.orchestra.run.smithi174.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T00:44:08.473 INFO:teuthology.orchestra.run.smithi174.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T00:44:08.473 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T00:44:08.473 INFO:teuthology.orchestra.run.smithi174.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T00:44:08.473 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T00:44:08.473 INFO:teuthology.orchestra.run.smithi174.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T00:44:08.473 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T00:44:08.473 INFO:teuthology.orchestra.run.smithi174.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T00:44:08.473 INFO:teuthology.orchestra.run.smithi174.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T00:44:08.473 INFO:teuthology.orchestra.run.smithi174.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T00:44:08.473 INFO:teuthology.orchestra.run.smithi174.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T00:44:08.473 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T00:44:08.473 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-03-28T00:44:08.473 INFO:teuthology.orchestra.run.smithi174.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T00:44:08.473 INFO:teuthology.orchestra.run.smithi174.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T00:44:08.474 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T00:44:08.474 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T00:44:08.474 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T00:44:08.474 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T00:44:08.474 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T00:44:08.474 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T00:44:08.474 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T00:44:08.474 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T00:44:08.474 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T00:44:08.474 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T00:44:08.474 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T00:44:08.474 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T00:44:08.474 INFO:teuthology.orchestra.run.smithi174.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T00:44:08.474 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T00:44:08.475 INFO:teuthology.orchestra.run.smithi174.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T00:44:08.475 INFO:teuthology.orchestra.run.smithi174.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T00:44:08.475 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T00:44:08.475 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T00:44:08.475 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T00:44:08.475 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T00:44:08.475 INFO:teuthology.orchestra.run.smithi174.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T00:44:08.475 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T00:44:08.475 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T00:44:08.475 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T00:44:08.475 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T00:44:08.475 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T00:44:08.476 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T00:44:08.476 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T00:44:08.476 INFO:teuthology.orchestra.run.smithi174.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T00:44:08.476 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T00:44:08.476 INFO:teuthology.orchestra.run.smithi174.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T00:44:08.476 INFO:teuthology.orchestra.run.smithi174.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T00:44:08.476 INFO:teuthology.orchestra.run.smithi174.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T00:44:08.476 INFO:teuthology.orchestra.run.smithi174.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T00:44:08.476 INFO:teuthology.orchestra.run.smithi174.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T00:44:08.476 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T00:44:08.476 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T00:44:08.476 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T00:44:08.476 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T00:44:08.476 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T00:44:08.477 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T00:44:08.477 INFO:teuthology.orchestra.run.smithi174.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T00:44:08.477 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T00:44:08.477 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T00:44:08.477 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T00:44:08.477 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T00:44:08.477 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T00:44:08.477 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T00:44:08.477 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T00:44:08.477 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T00:44:08.477 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T00:44:08.477 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T00:44:08.477 INFO:teuthology.orchestra.run.smithi174.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T00:44:08.477 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T00:44:08.478 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T00:44:08.478 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T00:44:08.478 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T00:44:08.478 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T00:44:08.478 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T00:44:08.478 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T00:44:08.478 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T00:44:08.478 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T00:44:08.478 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T00:44:08.478 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T00:44:08.478 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T00:44:08.478 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T00:44:08.478 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T00:44:08.478 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T00:44:08.479 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T00:44:08.479 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T00:44:08.479 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T00:44:08.479 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T00:44:08.479 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T00:44:08.479 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T00:44:08.479 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T00:44:08.479 INFO:teuthology.orchestra.run.smithi174.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T00:44:08.479 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T00:44:08.479 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T00:44:08.479 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T00:44:08.479 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T00:44:08.479 INFO:teuthology.orchestra.run.smithi174.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T00:44:08.479 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T00:44:08.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T00:44:08.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T00:44:08.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T00:44:08.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T00:44:08.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T00:44:08.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T00:44:08.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T00:44:08.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T00:44:08.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T00:44:08.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T00:44:08.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T00:44:08.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T00:44:08.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T00:44:08.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T00:44:08.481 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T00:44:08.481 INFO:teuthology.orchestra.run.smithi174.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T00:44:08.481 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T00:44:08.481 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T00:44:08.481 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T00:44:08.481 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T00:44:08.481 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T00:44:08.481 INFO:teuthology.orchestra.run.smithi174.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T00:44:08.481 INFO:teuthology.orchestra.run.smithi174.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T00:44:08.481 INFO:teuthology.orchestra.run.smithi174.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T00:44:08.481 INFO:teuthology.orchestra.run.smithi174.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T00:44:08.481 INFO:teuthology.orchestra.run.smithi174.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T00:44:08.481 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T00:44:08.481 INFO:teuthology.orchestra.run.smithi174.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T00:44:08.481 INFO:teuthology.orchestra.run.smithi174.stdout:Installing weak dependencies: 2024-03-28T00:44:08.482 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T00:44:08.482 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-28T00:44:08.482 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-28T00:44:08.482 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================================ 2024-03-28T00:44:08.482 INFO:teuthology.orchestra.run.smithi174.stdout:Install 150 Packages 2024-03-28T00:44:08.482 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrade 24 Packages 2024-03-28T00:44:08.482 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-28T00:44:08.482 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 225 M 2024-03-28T00:44:08.483 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-28T00:44:08.575 INFO:teuthology.orchestra.run.smithi169.stdout:(56/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-03-28T00:44:08.642 INFO:teuthology.orchestra.run.smithi169.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 18 MB/s | 4.7 MB 00:00 2024-03-28T00:44:08.667 INFO:teuthology.orchestra.run.smithi169.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 850 kB/s | 78 kB 00:00 2024-03-28T00:44:08.843 INFO:teuthology.orchestra.run.smithi169.stdout:(59/174): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 5.8 MB 00:00 2024-03-28T00:44:08.868 INFO:teuthology.orchestra.run.smithi169.stdout:(60/174): python3-jsonpatch-1.21-16.el9.noarch. 122 kB/s | 24 kB 00:00 2024-03-28T00:44:08.893 INFO:teuthology.orchestra.run.smithi169.stdout:(61/174): python3-jsonpointer-2.0-4.el9.noarch. 362 kB/s | 18 kB 00:00 2024-03-28T00:44:08.919 INFO:teuthology.orchestra.run.smithi169.stdout:(62/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-28T00:44:08.944 INFO:teuthology.orchestra.run.smithi169.stdout:(63/174): python3-markupsafe-1.1.1-12.el9.x86_6 626 kB/s | 32 kB 00:00 2024-03-28T00:44:09.053 INFO:teuthology.orchestra.run.smithi169.stdout:(64/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 217 kB 00:00 2024-03-28T00:44:09.120 INFO:teuthology.orchestra.run.smithi169.stdout:(65/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 4.8 MB 00:00 2024-03-28T00:44:09.145 INFO:teuthology.orchestra.run.smithi169.stdout:(66/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.9 MB/s | 174 kB 00:00 2024-03-28T00:44:09.170 INFO:teuthology.orchestra.run.smithi169.stdout:(67/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-28T00:44:09.195 INFO:teuthology.orchestra.run.smithi169.stdout:(68/174): python3-prettytable-0.7.2-27.el9.noar 826 kB/s | 41 kB 00:00 2024-03-28T00:44:09.221 INFO:teuthology.orchestra.run.smithi169.stdout:(69/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-28T00:44:09.246 INFO:teuthology.orchestra.run.smithi169.stdout:(70/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-28T00:44:09.253 INFO:teuthology.orchestra.run.smithi174.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 42 kB/s | 6.6 kB 00:00 2024-03-28T00:44:09.271 INFO:teuthology.orchestra.run.smithi169.stdout:(71/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-28T00:44:09.296 INFO:teuthology.orchestra.run.smithi169.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 947 kB/s | 47 kB 00:00 2024-03-28T00:44:09.322 INFO:teuthology.orchestra.run.smithi169.stdout:(73/174): python3-requests-oauthlib-1.3.0-12.el 855 kB/s | 43 kB 00:00 2024-03-28T00:44:09.439 INFO:teuthology.orchestra.run.smithi169.stdout:(74/174): python3-toml-0.10.2-6.el9.noarch.rpm 333 kB/s | 39 kB 00:00 2024-03-28T00:44:09.444 INFO:teuthology.orchestra.run.smithi174.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 4.6 MB/s | 890 kB 00:00 2024-03-28T00:44:09.495 INFO:teuthology.orchestra.run.smithi174.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 2.8 MB/s | 145 kB 00:00 2024-03-28T00:44:09.636 INFO:teuthology.orchestra.run.smithi174.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 16 MB/s | 2.3 MB 00:00 2024-03-28T00:44:09.745 INFO:teuthology.orchestra.run.smithi174.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 14 MB/s | 1.5 MB 00:00 2024-03-28T00:44:09.764 INFO:teuthology.orchestra.run.smithi169.stdout:(75/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-03-28T00:44:09.789 INFO:teuthology.orchestra.run.smithi169.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 118 kB/s | 41 kB 00:00 2024-03-28T00:44:09.815 INFO:teuthology.orchestra.run.smithi169.stdout:(77/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-28T00:44:09.840 INFO:teuthology.orchestra.run.smithi169.stdout:(78/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-28T00:44:09.857 INFO:teuthology.orchestra.run.smithi169.stdout:(79/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-28T00:44:09.899 INFO:teuthology.orchestra.run.smithi169.stdout:(80/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 750 kB/s | 44 kB 00:00 2024-03-28T00:44:09.924 INFO:teuthology.orchestra.run.smithi169.stdout:(81/174): fuse-2.9.9-15.el9.x86_64.rpm 1.1 MB/s | 79 kB 00:00 2024-03-28T00:44:09.958 INFO:teuthology.orchestra.run.smithi169.stdout:(82/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-28T00:44:09.995 INFO:teuthology.orchestra.run.smithi174.stdout:(6/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 19 MB/s | 4.7 MB 00:00 2024-03-28T00:44:10.008 INFO:teuthology.orchestra.run.smithi169.stdout:(83/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-28T00:44:10.034 INFO:teuthology.orchestra.run.smithi169.stdout:(84/174): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-28T00:44:10.059 INFO:teuthology.orchestra.run.smithi169.stdout:(85/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-28T00:44:10.093 INFO:teuthology.orchestra.run.smithi169.stdout:(86/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.1 MB/s | 187 kB 00:00 2024-03-28T00:44:10.193 INFO:teuthology.orchestra.run.smithi169.stdout:(87/174): bison-3.7.4-5.el9.x86_64.rpm 316 kB/s | 920 kB 00:02 2024-03-28T00:44:10.326 INFO:teuthology.orchestra.run.smithi169.stdout:(88/174): jq-1.6-16.el9.x86_64.rpm 700 kB/s | 187 kB 00:00 2024-03-28T00:44:10.410 INFO:teuthology.orchestra.run.smithi169.stdout:(89/174): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-03-28T00:44:10.477 INFO:teuthology.orchestra.run.smithi169.stdout:(90/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.0 MB/s | 803 kB 00:00 2024-03-28T00:44:10.502 INFO:teuthology.orchestra.run.smithi169.stdout:(91/174): libquadmath-11.4.1-3.el9.x86_64.rpm 687 kB/s | 195 kB 00:00 2024-03-28T00:44:10.694 INFO:teuthology.orchestra.run.smithi169.stdout:(92/174): elfutils-libelf-devel-0.190-2.el9.x86 76 kB/s | 21 kB 00:00 2024-03-28T00:44:10.761 INFO:teuthology.orchestra.run.smithi169.stdout:(93/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 90 kB/s | 25 kB 00:00 2024-03-28T00:44:10.846 INFO:teuthology.orchestra.run.smithi174.stdout:(7/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x86 20 MB/s | 17 MB 00:00 2024-03-28T00:44:11.288 INFO:teuthology.orchestra.run.smithi174.stdout:(8/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 2.4 MB/s | 5.4 MB 00:02 2024-03-28T00:44:11.404 INFO:teuthology.orchestra.run.smithi174.stdout:(9/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 18 MB/s | 9.9 MB 00:00 2024-03-28T00:44:11.430 INFO:teuthology.orchestra.run.smithi174.stdout:(10/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el 177 kB/s | 25 kB 00:00 2024-03-28T00:44:11.437 INFO:teuthology.orchestra.run.smithi169.stdout:(94/174): postgresql-contrib-13.14-1.el9.x86_64 1.3 MB/s | 911 kB 00:00 2024-03-28T00:44:11.479 INFO:teuthology.orchestra.run.smithi169.stdout:(95/174): postgresql-13.14-1.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-03-28T00:44:11.538 INFO:teuthology.orchestra.run.smithi174.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 2.0 MB/s | 220 kB 00:00 2024-03-28T00:44:11.545 INFO:teuthology.orchestra.run.smithi169.stdout:(96/174): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-03-28T00:44:11.647 INFO:teuthology.orchestra.run.smithi174.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 250 kB/s | 27 kB 00:00 2024-03-28T00:44:11.662 INFO:teuthology.orchestra.run.smithi169.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.1 MB/s | 245 kB 00:00 2024-03-28T00:44:11.747 INFO:teuthology.orchestra.run.smithi174.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 313 kB/s | 31 kB 00:00 2024-03-28T00:44:11.771 INFO:teuthology.orchestra.run.smithi169.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-03-28T00:44:11.838 INFO:teuthology.orchestra.run.smithi169.stdout:(99/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 3.1 MB/s | 4.1 MB 00:01 2024-03-28T00:44:11.880 INFO:teuthology.orchestra.run.smithi169.stdout:(100/174): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-03-28T00:44:11.947 INFO:teuthology.orchestra.run.smithi169.stdout:(101/174): qatlib-service-24.02.0-1.el9.x86_64. 330 kB/s | 36 kB 00:00 2024-03-28T00:44:11.956 INFO:teuthology.orchestra.run.smithi174.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 3.5 MB/s | 746 kB 00:00 2024-03-28T00:44:11.981 INFO:teuthology.orchestra.run.smithi169.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 472 kB/s | 47 kB 00:00 2024-03-28T00:44:12.032 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-03-28T00:44:12.064 INFO:teuthology.orchestra.run.smithi174.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.5 MB/s | 163 kB 00:00 2024-03-28T00:44:12.064 INFO:teuthology.orchestra.run.smithi169.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 802 kB/s | 94 kB 00:00 2024-03-28T00:44:12.181 INFO:teuthology.orchestra.run.smithi169.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 111 kB/s | 22 kB 00:00 2024-03-28T00:44:12.181 INFO:teuthology.orchestra.run.smithi174.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.1 MB/s | 127 kB 00:00 2024-03-28T00:44:12.436 INFO:teuthology.orchestra.run.smithi174.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 1.9 MB/s | 501 kB 00:00 2024-03-28T00:44:12.706 INFO:teuthology.orchestra.run.smithi169.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.3 MB/s | 863 kB 00:00 2024-03-28T00:44:12.915 INFO:teuthology.orchestra.run.smithi169.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.4 MB/s | 309 kB 00:00 2024-03-28T00:44:12.974 INFO:teuthology.orchestra.run.smithi169.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 335 kB/s | 19 kB 00:00 2024-03-28T00:44:13.273 INFO:teuthology.orchestra.run.smithi174.stdout:(18/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 6.1 MB/s | 5.1 MB 00:00 2024-03-28T00:44:13.291 INFO:teuthology.orchestra.run.smithi169.stdout:(108/174): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-28T00:44:13.349 INFO:teuthology.orchestra.run.smithi169.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 447 kB/s | 26 kB 00:00 2024-03-28T00:44:13.357 INFO:teuthology.orchestra.run.smithi174.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 543 kB/s | 45 kB 00:00 2024-03-28T00:44:13.416 INFO:teuthology.orchestra.run.smithi169.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 726 kB/s | 49 kB 00:00 2024-03-28T00:44:13.475 INFO:teuthology.orchestra.run.smithi169.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-28T00:44:13.542 INFO:teuthology.orchestra.run.smithi169.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-03-28T00:44:13.642 INFO:teuthology.orchestra.run.smithi169.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2024-03-28T00:44:13.726 INFO:teuthology.orchestra.run.smithi169.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-03-28T00:44:13.832 INFO:teuthology.orchestra.run.smithi174.stdout:(20/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 20 MB/s | 48 MB 00:02 2024-03-28T00:44:13.857 INFO:teuthology.orchestra.run.smithi174.stdout:(21/174): python3-ceph-common-19.0.0-2145.g3be9 262 kB/s | 131 kB 00:00 2024-03-28T00:44:13.883 INFO:teuthology.orchestra.run.smithi174.stdout:(22/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 3.2 MB/s | 162 kB 00:00 2024-03-28T00:44:13.925 INFO:teuthology.orchestra.run.smithi174.stdout:(23/174): python3-rados-19.0.0-2145.g3be9a5c9.e 4.7 MB/s | 321 kB 00:00 2024-03-28T00:44:13.934 INFO:teuthology.orchestra.run.smithi169.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 208 kB/s | 43 kB 00:00 2024-03-28T00:44:13.950 INFO:teuthology.orchestra.run.smithi174.stdout:(24/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 4.4 MB/s | 298 kB 00:00 2024-03-28T00:44:13.983 INFO:teuthology.orchestra.run.smithi174.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.7 MB/s | 99 kB 00:00 2024-03-28T00:44:13.993 INFO:teuthology.orchestra.run.smithi169.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 552 kB/s | 32 kB 00:00 2024-03-28T00:44:14.009 INFO:teuthology.orchestra.run.smithi174.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-28T00:44:14.043 INFO:teuthology.orchestra.run.smithi169.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 301 kB/s | 15 kB 00:00 2024-03-28T00:44:14.059 INFO:teuthology.orchestra.run.smithi174.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 3.3 MB/s | 170 kB 00:00 2024-03-28T00:44:14.110 INFO:teuthology.orchestra.run.smithi174.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 682 kB/s | 34 kB 00:00 2024-03-28T00:44:14.110 INFO:teuthology.orchestra.run.smithi169.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-03-28T00:44:14.176 INFO:teuthology.orchestra.run.smithi174.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.0 MB/s | 139 kB 00:00 2024-03-28T00:44:14.194 INFO:teuthology.orchestra.run.smithi169.stdout:(119/174): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-03-28T00:44:14.269 INFO:teuthology.orchestra.run.smithi169.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-03-28T00:44:14.301 INFO:teuthology.orchestra.run.smithi174.stdout:(30/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 9.7 MB/s | 3.1 MB 00:00 2024-03-28T00:44:14.328 INFO:teuthology.orchestra.run.smithi169.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 606 kB/s | 35 kB 00:00 2024-03-28T00:44:14.343 INFO:teuthology.orchestra.run.smithi174.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 11 MB/s | 1.8 MB 00:00 2024-03-28T00:44:14.390 INFO:teuthology.orchestra.run.smithi177.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T00:44:14.391 INFO:teuthology.orchestra.run.smithi177.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T00:44:14.395 INFO:teuthology.orchestra.run.smithi169.stdout:(122/174): python3-google-auth-2.28.2-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-03-28T00:44:14.405 INFO:teuthology.orchestra.run.smithi177.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T00:44:14.406 INFO:teuthology.orchestra.run.smithi177.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T00:44:14.407 INFO:teuthology.orchestra.run.smithi177.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T00:44:14.408 INFO:teuthology.orchestra.run.smithi177.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T00:44:14.410 INFO:teuthology.orchestra.run.smithi177.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T00:44:14.410 INFO:teuthology.orchestra.run.smithi174.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 3.6 MB/s | 248 kB 00:00 2024-03-28T00:44:14.413 INFO:teuthology.orchestra.run.smithi177.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T00:44:14.414 INFO:teuthology.orchestra.run.smithi177.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T00:44:14.461 INFO:teuthology.orchestra.run.smithi174.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 991 kB/s | 49 kB 00:00 2024-03-28T00:44:14.495 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-28T00:44:14.503 INFO:teuthology.orchestra.run.smithi174.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 395 kB/s | 16 kB 00:00 2024-03-28T00:44:14.522 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================================ 2024-03-28T00:44:14.522 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-28T00:44:14.522 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================================ 2024-03-28T00:44:14.522 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-28T00:44:14.522 INFO:teuthology.orchestra.run.smithi177.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T00:44:14.522 INFO:teuthology.orchestra.run.smithi177.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T00:44:14.522 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T00:44:14.522 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T00:44:14.522 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T00:44:14.523 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T00:44:14.523 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T00:44:14.523 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T00:44:14.523 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T00:44:14.523 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T00:44:14.523 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T00:44:14.523 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T00:44:14.523 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T00:44:14.523 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T00:44:14.523 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T00:44:14.523 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T00:44:14.523 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T00:44:14.523 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T00:44:14.524 INFO:teuthology.orchestra.run.smithi177.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T00:44:14.524 INFO:teuthology.orchestra.run.smithi177.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T00:44:14.524 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T00:44:14.524 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T00:44:14.524 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T00:44:14.524 INFO:teuthology.orchestra.run.smithi177.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T00:44:14.524 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T00:44:14.524 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T00:44:14.524 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T00:44:14.524 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T00:44:14.524 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T00:44:14.524 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T00:44:14.524 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T00:44:14.524 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T00:44:14.525 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T00:44:14.525 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T00:44:14.525 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T00:44:14.525 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T00:44:14.525 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T00:44:14.525 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrading: 2024-03-28T00:44:14.525 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T00:44:14.525 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T00:44:14.525 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T00:44:14.525 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T00:44:14.525 INFO:teuthology.orchestra.run.smithi177.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T00:44:14.525 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T00:44:14.525 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T00:44:14.525 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T00:44:14.526 INFO:teuthology.orchestra.run.smithi177.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T00:44:14.526 INFO:teuthology.orchestra.run.smithi177.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T00:44:14.526 INFO:teuthology.orchestra.run.smithi177.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T00:44:14.526 INFO:teuthology.orchestra.run.smithi177.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T00:44:14.526 INFO:teuthology.orchestra.run.smithi177.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T00:44:14.526 INFO:teuthology.orchestra.run.smithi177.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T00:44:14.526 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T00:44:14.526 INFO:teuthology.orchestra.run.smithi177.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T00:44:14.526 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T00:44:14.526 INFO:teuthology.orchestra.run.smithi177.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T00:44:14.526 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T00:44:14.526 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T00:44:14.527 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T00:44:14.527 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T00:44:14.527 INFO:teuthology.orchestra.run.smithi177.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T00:44:14.527 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T00:44:14.527 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-28T00:44:14.527 INFO:teuthology.orchestra.run.smithi177.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T00:44:14.527 INFO:teuthology.orchestra.run.smithi177.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T00:44:14.527 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T00:44:14.527 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T00:44:14.527 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T00:44:14.527 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T00:44:14.527 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T00:44:14.527 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T00:44:14.528 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T00:44:14.528 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T00:44:14.528 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T00:44:14.528 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T00:44:14.528 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T00:44:14.528 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T00:44:14.528 INFO:teuthology.orchestra.run.smithi177.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T00:44:14.528 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T00:44:14.528 INFO:teuthology.orchestra.run.smithi177.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T00:44:14.528 INFO:teuthology.orchestra.run.smithi177.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T00:44:14.528 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T00:44:14.528 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T00:44:14.528 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T00:44:14.528 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T00:44:14.529 INFO:teuthology.orchestra.run.smithi177.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T00:44:14.529 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T00:44:14.529 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T00:44:14.529 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T00:44:14.529 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T00:44:14.529 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T00:44:14.529 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T00:44:14.529 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T00:44:14.529 INFO:teuthology.orchestra.run.smithi177.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T00:44:14.529 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T00:44:14.529 INFO:teuthology.orchestra.run.smithi177.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T00:44:14.529 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T00:44:14.530 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T00:44:14.530 INFO:teuthology.orchestra.run.smithi177.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T00:44:14.530 INFO:teuthology.orchestra.run.smithi177.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T00:44:14.530 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T00:44:14.530 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T00:44:14.530 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T00:44:14.530 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T00:44:14.530 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T00:44:14.530 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T00:44:14.530 INFO:teuthology.orchestra.run.smithi177.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T00:44:14.530 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T00:44:14.530 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T00:44:14.530 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T00:44:14.531 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T00:44:14.531 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T00:44:14.531 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T00:44:14.531 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T00:44:14.531 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T00:44:14.531 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T00:44:14.531 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T00:44:14.531 INFO:teuthology.orchestra.run.smithi177.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T00:44:14.531 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T00:44:14.531 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T00:44:14.531 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T00:44:14.531 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T00:44:14.531 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T00:44:14.531 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T00:44:14.532 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T00:44:14.532 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T00:44:14.532 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T00:44:14.532 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T00:44:14.532 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T00:44:14.532 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T00:44:14.532 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T00:44:14.532 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T00:44:14.532 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T00:44:14.532 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T00:44:14.532 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T00:44:14.532 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T00:44:14.532 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T00:44:14.532 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T00:44:14.533 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T00:44:14.533 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T00:44:14.533 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T00:44:14.533 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T00:44:14.533 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T00:44:14.533 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T00:44:14.533 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T00:44:14.533 INFO:teuthology.orchestra.run.smithi177.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T00:44:14.533 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T00:44:14.533 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T00:44:14.533 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T00:44:14.533 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T00:44:14.533 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T00:44:14.533 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T00:44:14.534 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T00:44:14.534 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T00:44:14.534 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T00:44:14.534 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T00:44:14.534 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T00:44:14.534 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T00:44:14.534 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T00:44:14.534 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T00:44:14.534 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T00:44:14.534 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T00:44:14.534 INFO:teuthology.orchestra.run.smithi177.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T00:44:14.534 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T00:44:14.534 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T00:44:14.535 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T00:44:14.535 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T00:44:14.535 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T00:44:14.535 INFO:teuthology.orchestra.run.smithi177.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T00:44:14.535 INFO:teuthology.orchestra.run.smithi177.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T00:44:14.535 INFO:teuthology.orchestra.run.smithi177.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T00:44:14.535 INFO:teuthology.orchestra.run.smithi177.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T00:44:14.535 INFO:teuthology.orchestra.run.smithi177.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T00:44:14.535 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T00:44:14.535 INFO:teuthology.orchestra.run.smithi177.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T00:44:14.535 INFO:teuthology.orchestra.run.smithi177.stdout:Installing weak dependencies: 2024-03-28T00:44:14.535 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T00:44:14.535 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-28T00:44:14.535 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-28T00:44:14.535 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================================ 2024-03-28T00:44:14.536 INFO:teuthology.orchestra.run.smithi177.stdout:Install 150 Packages 2024-03-28T00:44:14.536 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrade 24 Packages 2024-03-28T00:44:14.536 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-28T00:44:14.536 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 225 M 2024-03-28T00:44:14.536 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-28T00:44:14.561 INFO:teuthology.orchestra.run.smithi174.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 4.4 MB/s | 264 kB 00:00 2024-03-28T00:44:14.562 INFO:teuthology.orchestra.run.smithi169.stdout:(123/174): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-03-28T00:44:14.629 INFO:teuthology.orchestra.run.smithi169.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-03-28T00:44:14.645 INFO:teuthology.orchestra.run.smithi174.stdout:(36/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 8.8 MB/s | 748 kB 00:00 2024-03-28T00:44:14.687 INFO:teuthology.orchestra.run.smithi169.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 184 kB/s | 11 kB 00:00 2024-03-28T00:44:14.704 INFO:teuthology.orchestra.run.smithi174.stdout:(37/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 313 kB/s | 18 kB 00:00 2024-03-28T00:44:14.746 INFO:teuthology.orchestra.run.smithi169.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 303 kB/s | 18 kB 00:00 2024-03-28T00:44:14.805 INFO:teuthology.orchestra.run.smithi169.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 397 kB/s | 23 kB 00:00 2024-03-28T00:44:14.854 INFO:teuthology.orchestra.run.smithi174.stdout:(38/174): ceph-mgr-diskprediction-local-19.0.0- 13 MB/s | 7.4 MB 00:00 2024-03-28T00:44:14.863 INFO:teuthology.orchestra.run.smithi169.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 334 kB/s | 19 kB 00:00 2024-03-28T00:44:14.921 INFO:teuthology.orchestra.run.smithi169.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 337 kB/s | 20 kB 00:00 2024-03-28T00:44:15.029 INFO:teuthology.orchestra.run.smithi174.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 6.0 MB/s | 920 kB 00:00 2024-03-28T00:44:15.047 INFO:teuthology.orchestra.run.smithi169.stdout:(130/174): python3-kubernetes-26.1.0-2.el9.noar 8.2 MB/s | 1.0 MB 00:00 2024-03-28T00:44:15.122 INFO:teuthology.orchestra.run.smithi169.stdout:(131/174): postgresql-server-13.14-1.el9.x86_64 1.6 MB/s | 5.8 MB 00:03 2024-03-28T00:44:15.147 INFO:teuthology.orchestra.run.smithi169.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 461 kB/s | 46 kB 00:00 2024-03-28T00:44:15.183 INFO:teuthology.orchestra.run.smithi177.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 27 kB/s | 6.6 kB 00:00 2024-03-28T00:44:15.214 INFO:teuthology.orchestra.run.smithi169.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 869 kB/s | 58 kB 00:00 2024-03-28T00:44:15.274 INFO:teuthology.orchestra.run.smithi169.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 523 kB/s | 79 kB 00:00 2024-03-28T00:44:15.298 INFO:teuthology.orchestra.run.smithi169.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-28T00:44:15.332 INFO:teuthology.orchestra.run.smithi169.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 281 kB/s | 16 kB 00:00 2024-03-28T00:44:15.349 INFO:teuthology.orchestra.run.smithi169.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-03-28T00:44:15.391 INFO:teuthology.orchestra.run.smithi169.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 480 kB/s | 28 kB 00:00 2024-03-28T00:44:15.416 INFO:teuthology.orchestra.run.smithi169.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 460 kB/s | 31 kB 00:00 2024-03-28T00:44:15.446 INFO:teuthology.orchestra.run.smithi174.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 892 kB/s | 662 kB 00:00 2024-03-28T00:44:15.471 INFO:teuthology.orchestra.run.smithi174.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 239 kB/s | 106 kB 00:00 2024-03-28T00:44:15.483 INFO:teuthology.orchestra.run.smithi177.stdout:(2/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 9.9 MB/s | 5.4 MB 00:00 2024-03-28T00:44:15.483 INFO:teuthology.orchestra.run.smithi169.stdout:(140/174): python3-rsa-4.9-2.el9.noarch.rpm 886 kB/s | 59 kB 00:00 2024-03-28T00:44:15.509 INFO:teuthology.orchestra.run.smithi169.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-03-28T00:44:15.533 INFO:teuthology.orchestra.run.smithi177.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 2.8 MB/s | 145 kB 00:00 2024-03-28T00:44:15.542 INFO:teuthology.orchestra.run.smithi169.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 609 kB/s | 36 kB 00:00 2024-03-28T00:44:15.567 INFO:teuthology.orchestra.run.smithi169.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 873 kB/s | 51 kB 00:00 2024-03-28T00:44:15.571 INFO:teuthology.orchestra.run.smithi174.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 2.4 MB/s | 307 kB 00:00 2024-03-28T00:44:15.597 INFO:teuthology.orchestra.run.smithi174.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 253 kB/s | 32 kB 00:00 2024-03-28T00:44:15.610 INFO:teuthology.orchestra.run.smithi169.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-28T00:44:15.635 INFO:teuthology.orchestra.run.smithi169.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-03-28T00:44:15.685 INFO:teuthology.orchestra.run.smithi169.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.6 MB/s | 427 kB 00:00 2024-03-28T00:44:15.700 INFO:teuthology.orchestra.run.smithi177.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 14 MB/s | 2.3 MB 00:00 2024-03-28T00:44:15.710 INFO:teuthology.orchestra.run.smithi169.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-03-28T00:44:15.734 INFO:teuthology.orchestra.run.smithi177.stdout:(5/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.6 MB/s | 890 kB 00:00 2024-03-28T00:44:15.752 INFO:teuthology.orchestra.run.smithi169.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-28T00:44:15.805 INFO:teuthology.orchestra.run.smithi174.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 82 kB/s | 17 kB 00:00 2024-03-28T00:44:15.809 INFO:teuthology.orchestra.run.smithi177.stdout:(6/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 14 MB/s | 1.5 MB 00:00 2024-03-28T00:44:15.877 INFO:teuthology.orchestra.run.smithi169.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-03-28T00:44:15.897 INFO:teuthology.orchestra.run.smithi174.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.2 MB/s | 3.0 MB 00:00 2024-03-28T00:44:15.922 INFO:teuthology.orchestra.run.smithi174.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 87 kB/s | 10 kB 00:00 2024-03-28T00:44:15.989 INFO:teuthology.orchestra.run.smithi174.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 159 kB 00:00 2024-03-28T00:44:16.015 INFO:teuthology.orchestra.run.smithi174.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 480 kB/s | 44 kB 00:00 2024-03-28T00:44:16.106 INFO:teuthology.orchestra.run.smithi174.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 661 kB 00:00 2024-03-28T00:44:16.132 INFO:teuthology.orchestra.run.smithi174.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 4.8 MB/s | 578 kB 00:00 2024-03-28T00:44:16.207 INFO:teuthology.orchestra.run.smithi174.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 467 kB/s | 46 kB 00:00 2024-03-28T00:44:16.232 INFO:teuthology.orchestra.run.smithi174.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 282 kB 00:00 2024-03-28T00:44:16.316 INFO:teuthology.orchestra.run.smithi174.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-28T00:44:16.341 INFO:teuthology.orchestra.run.smithi174.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.0 MB/s | 217 kB 00:00 2024-03-28T00:44:16.400 INFO:teuthology.orchestra.run.smithi174.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 411 kB/s | 34 kB 00:00 2024-03-28T00:44:16.575 INFO:teuthology.orchestra.run.smithi174.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 20 MB/s | 4.7 MB 00:00 2024-03-28T00:44:16.608 INFO:teuthology.orchestra.run.smithi174.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-03-28T00:44:16.674 INFO:teuthology.orchestra.run.smithi174.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 804 kB/s | 78 kB 00:00 2024-03-28T00:44:16.869 INFO:teuthology.orchestra.run.smithi169.stdout:(150/174): librados2-19.0.0-2145.g3be9a5c9.el9. 3.0 MB/s | 3.4 MB 00:01 2024-03-28T00:44:16.874 INFO:teuthology.orchestra.run.smithi174.stdout:(59/174): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-28T00:44:16.899 INFO:teuthology.orchestra.run.smithi174.stdout:(60/174): python3-jsonpatch-1.21-16.el9.noarch. 108 kB/s | 24 kB 00:00 2024-03-28T00:44:16.942 INFO:teuthology.orchestra.run.smithi177.stdout:(7/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x86 15 MB/s | 17 MB 00:01 2024-03-28T00:44:16.983 INFO:teuthology.orchestra.run.smithi174.stdout:(61/174): python3-jsonpointer-2.0-4.el9.noarch. 166 kB/s | 18 kB 00:00 2024-03-28T00:44:17.009 INFO:teuthology.orchestra.run.smithi174.stdout:(62/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 144 kB 00:00 2024-03-28T00:44:17.092 INFO:teuthology.orchestra.run.smithi174.stdout:(63/174): python3-markupsafe-1.1.1-12.el9.x86_6 291 kB/s | 32 kB 00:00 2024-03-28T00:44:17.259 INFO:teuthology.orchestra.run.smithi174.stdout:(64/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-28T00:44:17.284 INFO:teuthology.orchestra.run.smithi174.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.1 MB/s | 217 kB 00:00 2024-03-28T00:44:17.362 INFO:teuthology.orchestra.run.smithi174.stdout:(66/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.7 MB/s | 174 kB 00:00 2024-03-28T00:44:17.393 INFO:teuthology.orchestra.run.smithi174.stdout:(67/174): python3-packaging-20.9-5.el9.noarch.r 633 kB/s | 69 kB 00:00 2024-03-28T00:44:17.469 INFO:teuthology.orchestra.run.smithi174.stdout:(68/174): python3-prettytable-0.7.2-27.el9.noar 380 kB/s | 41 kB 00:00 2024-03-28T00:44:17.554 INFO:teuthology.orchestra.run.smithi174.stdout:(69/174): python3-protobuf-3.14.0-13.el9.noarch 1.5 MB/s | 238 kB 00:00 2024-03-28T00:44:17.576 INFO:teuthology.orchestra.run.smithi177.stdout:(8/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 16 MB/s | 9.9 MB 00:00 2024-03-28T00:44:17.611 INFO:teuthology.orchestra.run.smithi174.stdout:(70/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 933 kB/s | 132 kB 00:00 2024-03-28T00:44:17.612 INFO:teuthology.orchestra.run.smithi169.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 433 kB/s | 321 kB 00:00 2024-03-28T00:44:17.618 INFO:teuthology.orchestra.run.smithi177.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 607 kB/s | 25 kB 00:00 2024-03-28T00:44:17.653 INFO:teuthology.orchestra.run.smithi174.stdout:(71/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.1 MB/s | 211 kB 00:00 2024-03-28T00:44:17.695 INFO:teuthology.orchestra.run.smithi174.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 566 kB/s | 47 kB 00:00 2024-03-28T00:44:17.745 INFO:teuthology.orchestra.run.smithi174.stdout:(73/174): python3-requests-oauthlib-1.3.0-12.el 466 kB/s | 43 kB 00:00 2024-03-28T00:44:18.312 INFO:teuthology.orchestra.run.smithi169.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 3.3 MB/s | 2.3 MB 00:00 2024-03-28T00:44:18.495 INFO:teuthology.orchestra.run.smithi174.stdout:(74/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-03-28T00:44:18.512 INFO:teuthology.orchestra.run.smithi174.stdout:(75/174): python3-toml-0.10.2-6.el9.noarch.rpm 50 kB/s | 39 kB 00:00 2024-03-28T00:44:18.570 INFO:teuthology.orchestra.run.smithi169.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 7.1 MB/s | 1.8 MB 00:00 2024-03-28T00:44:18.629 INFO:teuthology.orchestra.run.smithi174.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 307 kB/s | 41 kB 00:00 2024-03-28T00:44:18.654 INFO:teuthology.orchestra.run.smithi174.stdout:(77/174): socat-1.7.4.1-5.el9.x86_64.rpm 2.1 MB/s | 300 kB 00:00 2024-03-28T00:44:18.687 INFO:teuthology.orchestra.run.smithi169.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 361 kB/s | 42 kB 00:00 2024-03-28T00:44:18.737 INFO:teuthology.orchestra.run.smithi169.stdout:(155/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 1.1 MB/s | 3.1 MB 00:02 2024-03-28T00:44:18.771 INFO:teuthology.orchestra.run.smithi174.stdout:(78/174): uuid-1.6.2-55.el9.x86_64.rpm 399 kB/s | 56 kB 00:00 2024-03-28T00:44:18.776 INFO:teuthology.orchestra.run.smithi177.stdout:(10/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x8 1.5 MB/s | 4.7 MB 00:03 2024-03-28T00:44:18.797 INFO:teuthology.orchestra.run.smithi174.stdout:(79/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 440 kB/s | 63 kB 00:00 2024-03-28T00:44:18.829 INFO:teuthology.orchestra.run.smithi169.stdout:(156/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.7 MB/s | 248 kB 00:00 2024-03-28T00:44:18.905 INFO:teuthology.orchestra.run.smithi174.stdout:(80/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 329 kB/s | 44 kB 00:00 2024-03-28T00:44:18.929 INFO:teuthology.orchestra.run.smithi169.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 364 kB/s | 36 kB 00:00 2024-03-28T00:44:18.930 INFO:teuthology.orchestra.run.smithi174.stdout:(81/174): fuse-2.9.9-15.el9.x86_64.rpm 589 kB/s | 79 kB 00:00 2024-03-28T00:44:18.960 INFO:teuthology.orchestra.run.smithi177.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 1.2 MB/s | 220 kB 00:00 2024-03-28T00:44:18.988 INFO:teuthology.orchestra.run.smithi169.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 285 kB/s | 71 kB 00:00 2024-03-28T00:44:19.013 INFO:teuthology.orchestra.run.smithi169.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.2 MB/s | 192 kB 00:00 2024-03-28T00:44:19.014 INFO:teuthology.orchestra.run.smithi174.stdout:(82/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.2 MB/s | 241 kB 00:00 2024-03-28T00:44:19.051 INFO:teuthology.orchestra.run.smithi177.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 295 kB/s | 27 kB 00:00 2024-03-28T00:44:19.064 INFO:teuthology.orchestra.run.smithi169.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.3 MB/s | 256 kB 00:00 2024-03-28T00:44:19.064 INFO:teuthology.orchestra.run.smithi174.stdout:(83/174): python3-cryptography-36.0.1-4.el9.x86 8.6 MB/s | 1.1 MB 00:00 2024-03-28T00:44:19.089 INFO:teuthology.orchestra.run.smithi169.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 306 kB/s | 23 kB 00:00 2024-03-28T00:44:19.114 INFO:teuthology.orchestra.run.smithi174.stdout:(84/174): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2024-03-28T00:44:19.139 INFO:teuthology.orchestra.run.smithi169.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 108 kB 00:00 2024-03-28T00:44:19.143 INFO:teuthology.orchestra.run.smithi177.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 341 kB/s | 31 kB 00:00 2024-03-28T00:44:19.173 INFO:teuthology.orchestra.run.smithi174.stdout:(85/174): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 124 kB 00:00 2024-03-28T00:44:19.206 INFO:teuthology.orchestra.run.smithi169.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 155 kB 00:00 2024-03-28T00:44:19.223 INFO:teuthology.orchestra.run.smithi174.stdout:(86/174): python3-urllib3-1.26.5-5.el9.noarch.r 1.7 MB/s | 187 kB 00:00 2024-03-28T00:44:19.232 INFO:teuthology.orchestra.run.smithi169.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 136 kB 00:00 2024-03-28T00:44:19.274 INFO:teuthology.orchestra.run.smithi169.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 954 kB/s | 64 kB 00:00 2024-03-28T00:44:19.315 INFO:teuthology.orchestra.run.smithi169.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 343 kB/s | 29 kB 00:00 2024-03-28T00:44:19.369 INFO:teuthology.orchestra.run.smithi177.stdout:(14/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 4.6 MB/s | 20 MB 00:04 2024-03-28T00:44:19.461 INFO:teuthology.orchestra.run.smithi177.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.7 MB/s | 163 kB 00:00 2024-03-28T00:44:19.474 INFO:teuthology.orchestra.run.smithi169.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 6.0 MB/s | 1.2 MB 00:00 2024-03-28T00:44:19.562 INFO:teuthology.orchestra.run.smithi177.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.2 MB/s | 127 kB 00:00 2024-03-28T00:44:19.703 INFO:teuthology.orchestra.run.smithi177.stdout:(17/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 1.3 MB/s | 746 kB 00:00 2024-03-28T00:44:19.716 INFO:teuthology.orchestra.run.smithi169.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 9.5 MB/s | 2.3 MB 00:00 2024-03-28T00:44:19.745 INFO:teuthology.orchestra.run.smithi177.stdout:(18/174): libradosstriper1-19.0.0-2145.g3be9a5c 2.7 MB/s | 501 kB 00:00 2024-03-28T00:44:19.810 INFO:teuthology.orchestra.run.smithi169.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 5.0 MB/s | 465 kB 00:00 2024-03-28T00:44:19.845 INFO:teuthology.orchestra.run.smithi177.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 451 kB/s | 45 kB 00:00 2024-03-28T00:44:19.917 INFO:teuthology.orchestra.run.smithi169.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 287 kB/s | 31 kB 00:00 2024-03-28T00:44:19.945 INFO:teuthology.orchestra.run.smithi177.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.3 MB/s | 131 kB 00:00 2024-03-28T00:44:19.999 INFO:teuthology.orchestra.run.smithi174.stdout:(87/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 1.9 MB/s | 20 MB 00:10 2024-03-28T00:44:20.134 INFO:teuthology.orchestra.run.smithi169.stdout:(171/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.0 MB/s | 1.1 MB 00:00 2024-03-28T00:44:20.176 INFO:teuthology.orchestra.run.smithi169.stdout:(172/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 2.5 MB/s | 2.2 MB 00:00 2024-03-28T00:44:20.241 INFO:teuthology.orchestra.run.smithi174.stdout:(88/174): jq-1.6-16.el9.x86_64.rpm 175 kB/s | 187 kB 00:01 2024-03-28T00:44:20.413 INFO:teuthology.orchestra.run.smithi177.stdout:(21/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 17 MB/s | 48 MB 00:02 2024-03-28T00:44:20.438 INFO:teuthology.orchestra.run.smithi177.stdout:(22/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 329 kB/s | 162 kB 00:00 2024-03-28T00:44:20.480 INFO:teuthology.orchestra.run.smithi177.stdout:(23/174): python3-rados-19.0.0-2145.g3be9a5c9.e 4.7 MB/s | 321 kB 00:00 2024-03-28T00:44:20.530 INFO:teuthology.orchestra.run.smithi177.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 2.0 MB/s | 99 kB 00:00 2024-03-28T00:44:20.555 INFO:teuthology.orchestra.run.smithi177.stdout:(25/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 2.5 MB/s | 298 kB 00:00 2024-03-28T00:44:20.581 INFO:teuthology.orchestra.run.smithi177.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.7 MB/s | 85 kB 00:00 2024-03-28T00:44:20.584 INFO:teuthology.orchestra.run.smithi169.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 165 kB/s | 74 kB 00:00 2024-03-28T00:44:20.631 INFO:teuthology.orchestra.run.smithi177.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 3.3 MB/s | 170 kB 00:00 2024-03-28T00:44:20.665 INFO:teuthology.orchestra.run.smithi177.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 1.0 MB/s | 34 kB 00:00 2024-03-28T00:44:20.714 INFO:teuthology.orchestra.run.smithi177.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.7 MB/s | 139 kB 00:00 2024-03-28T00:44:20.848 INFO:teuthology.orchestra.run.smithi177.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 13 MB/s | 1.8 MB 00:00 2024-03-28T00:44:20.874 INFO:teuthology.orchestra.run.smithi174.stdout:(89/174): libquadmath-11.4.1-3.el9.x86_64.rpm 223 kB/s | 195 kB 00:00 2024-03-28T00:44:21.028 INFO:teuthology.orchestra.run.smithi174.stdout:(90/174): elfutils-libelf-devel-0.190-2.el9.x86 140 kB/s | 21 kB 00:00 2024-03-28T00:44:21.058 INFO:teuthology.orchestra.run.smithi174.stdout:(91/174): python3-requests-2.25.1-8.el9.noarch. 153 kB/s | 125 kB 00:00 2024-03-28T00:44:21.083 INFO:teuthology.orchestra.run.smithi174.stdout:(92/174): libgfortran-11.4.1-3.el9.x86_64.rpm 432 kB/s | 803 kB 00:01 2024-03-28T00:44:21.183 INFO:teuthology.orchestra.run.smithi174.stdout:(93/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 165 kB/s | 25 kB 00:00 2024-03-28T00:44:21.190 INFO:teuthology.orchestra.run.smithi177.stdout:(31/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 4.9 MB/s | 3.1 MB 00:00 2024-03-28T00:44:21.282 INFO:teuthology.orchestra.run.smithi177.stdout:(32/174): ceph-mgr-diskprediction-local-19.0.0- 17 MB/s | 7.4 MB 00:00 2024-03-28T00:44:21.307 INFO:teuthology.orchestra.run.smithi177.stdout:(33/174): ceph-mgr-modules-core-19.0.0-2145.g3b 2.1 MB/s | 248 kB 00:00 2024-03-28T00:44:21.324 INFO:teuthology.orchestra.run.smithi177.stdout:(34/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 1.2 MB/s | 49 kB 00:00 2024-03-28T00:44:21.358 INFO:teuthology.orchestra.run.smithi177.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2145.g3 328 kB/s | 16 kB 00:00 2024-03-28T00:44:21.383 INFO:teuthology.orchestra.run.smithi177.stdout:(36/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 4.4 MB/s | 264 kB 00:00 2024-03-28T00:44:21.416 INFO:teuthology.orchestra.run.smithi177.stdout:(37/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 549 kB/s | 18 kB 00:00 2024-03-28T00:44:21.500 INFO:teuthology.orchestra.run.smithi177.stdout:(38/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 5.1 MB/s | 748 kB 00:00 2024-03-28T00:44:21.667 INFO:teuthology.orchestra.run.smithi174.stdout:(94/174): postgresql-13.14-1.el9.x86_64.rpm 2.7 MB/s | 1.6 MB 00:00 2024-03-28T00:44:21.667 INFO:teuthology.orchestra.run.smithi177.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 6.4 MB/s | 920 kB 00:00 2024-03-28T00:44:21.726 INFO:teuthology.orchestra.run.smithi174.stdout:(95/174): postgresql-contrib-13.14-1.el9.x86_64 1.6 MB/s | 911 kB 00:00 2024-03-28T00:44:21.809 INFO:teuthology.orchestra.run.smithi177.stdout:(40/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 2.4 MB/s | 5.1 MB 00:02 2024-03-28T00:44:21.892 INFO:teuthology.orchestra.run.smithi174.stdout:(96/174): postgresql-private-libs-13.14-1.el9.x 592 kB/s | 133 kB 00:00 2024-03-28T00:44:22.143 INFO:teuthology.orchestra.run.smithi174.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 982 kB/s | 245 kB 00:00 2024-03-28T00:44:22.159 INFO:teuthology.orchestra.run.smithi177.stdout:(41/174): automake-1.16.2-8.el9.noarch.rpm 892 kB/s | 662 kB 00:00 2024-03-28T00:44:22.184 INFO:teuthology.orchestra.run.smithi177.stdout:(42/174): boost-program-options-1.75.0-8.el9.x8 205 kB/s | 106 kB 00:00 2024-03-28T00:44:22.218 INFO:teuthology.orchestra.run.smithi177.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 752 kB/s | 307 kB 00:00 2024-03-28T00:44:22.302 INFO:teuthology.orchestra.run.smithi177.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 223 kB/s | 32 kB 00:00 2024-03-28T00:44:22.384 INFO:teuthology.orchestra.run.smithi174.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.0 MB/s | 248 kB 00:00 2024-03-28T00:44:22.443 INFO:teuthology.orchestra.run.smithi174.stdout:(99/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 3.0 MB/s | 4.1 MB 00:01 2024-03-28T00:44:22.535 INFO:teuthology.orchestra.run.smithi177.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 8.5 MB/s | 3.0 MB 00:00 2024-03-28T00:44:22.560 INFO:teuthology.orchestra.run.smithi174.stdout:(100/174): qatlib-service-24.02.0-1.el9.x86_64. 308 kB/s | 36 kB 00:00 2024-03-28T00:44:22.560 INFO:teuthology.orchestra.run.smithi177.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 50 kB/s | 17 kB 00:00 2024-03-28T00:44:22.585 INFO:teuthology.orchestra.run.smithi177.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 36 kB/s | 10 kB 00:00 2024-03-28T00:44:22.702 INFO:teuthology.orchestra.run.smithi177.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 379 kB/s | 44 kB 00:00 2024-03-28T00:44:22.718 INFO:teuthology.orchestra.run.smithi174.stdout:(101/174): qatlib-24.02.0-1.el9.x86_64.rpm 662 kB/s | 221 kB 00:00 2024-03-28T00:44:22.728 INFO:teuthology.orchestra.run.smithi177.stdout:(49/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-03-28T00:44:22.735 INFO:teuthology.orchestra.run.smithi174.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 270 kB/s | 47 kB 00:00 2024-03-28T00:44:22.761 INFO:teuthology.orchestra.run.smithi177.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 661 kB 00:00 2024-03-28T00:44:22.819 INFO:teuthology.orchestra.run.smithi174.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-03-28T00:44:22.837 INFO:teuthology.orchestra.run.smithi177.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 4.3 MB/s | 578 kB 00:00 2024-03-28T00:44:22.879 INFO:teuthology.orchestra.run.smithi177.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 397 kB/s | 46 kB 00:00 2024-03-28T00:44:22.886 INFO:teuthology.orchestra.run.smithi174.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 561 kB/s | 94 kB 00:00 2024-03-28T00:44:22.904 INFO:teuthology.orchestra.run.smithi177.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 282 kB 00:00 2024-03-28T00:44:22.963 INFO:teuthology.orchestra.run.smithi177.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-28T00:44:22.996 INFO:teuthology.orchestra.run.smithi177.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.8 MB/s | 217 kB 00:00 2024-03-28T00:44:23.011 INFO:teuthology.orchestra.run.smithi174.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.4 MB/s | 863 kB 00:00 2024-03-28T00:44:23.030 INFO:teuthology.orchestra.run.smithi177.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 270 kB/s | 34 kB 00:00 2024-03-28T00:44:23.231 INFO:teuthology.orchestra.run.smithi177.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 18 MB/s | 4.7 MB 00:00 2024-03-28T00:44:23.256 INFO:teuthology.orchestra.run.smithi177.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 345 kB/s | 78 kB 00:00 2024-03-28T00:44:23.289 INFO:teuthology.orchestra.run.smithi177.stdout:(59/174): protobuf-3.14.0-13.el9.x86_64.rpm 3.4 MB/s | 1.0 MB 00:00 2024-03-28T00:44:23.499 INFO:teuthology.orchestra.run.smithi177.stdout:(60/174): python3-jsonpatch-1.21-16.el9.noarch. 101 kB/s | 24 kB 00:00 2024-03-28T00:44:23.582 INFO:teuthology.orchestra.run.smithi177.stdout:(61/174): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2024-03-28T00:44:23.599 INFO:teuthology.orchestra.run.smithi177.stdout:(62/174): python3-jsonpointer-2.0-4.el9.noarch. 58 kB/s | 18 kB 00:00 2024-03-28T00:44:23.624 INFO:teuthology.orchestra.run.smithi177.stdout:(63/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-03-28T00:44:23.708 INFO:teuthology.orchestra.run.smithi177.stdout:(64/174): python3-markupsafe-1.1.1-12.el9.x86_6 292 kB/s | 32 kB 00:00 2024-03-28T00:44:23.778 INFO:teuthology.orchestra.run.smithi174.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 614 kB/s | 548 kB 00:00 2024-03-28T00:44:23.887 INFO:teuthology.orchestra.run.smithi174.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 180 kB/s | 19 kB 00:00 2024-03-28T00:44:23.900 INFO:teuthology.orchestra.run.smithi177.stdout:(65/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2024-03-28T00:44:23.925 INFO:teuthology.orchestra.run.smithi177.stdout:(66/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 723 kB/s | 217 kB 00:00 2024-03-28T00:44:23.950 INFO:teuthology.orchestra.run.smithi177.stdout:(67/174): python3-oauthlib-3.1.1-5.el9.noarch.r 731 kB/s | 174 kB 00:00 2024-03-28T00:44:23.953 INFO:teuthology.orchestra.run.smithi174.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 328 kB/s | 309 kB 00:00 2024-03-28T00:44:24.009 INFO:teuthology.orchestra.run.smithi177.stdout:(68/174): python3-packaging-20.9-5.el9.noarch.r 638 kB/s | 69 kB 00:00 2024-03-28T00:44:24.068 INFO:teuthology.orchestra.run.smithi177.stdout:(69/174): python3-prettytable-0.7.2-27.el9.noar 352 kB/s | 41 kB 00:00 2024-03-28T00:44:24.093 INFO:teuthology.orchestra.run.smithi177.stdout:(70/174): python3-protobuf-3.14.0-13.el9.noarch 1.6 MB/s | 238 kB 00:00 2024-03-28T00:44:24.118 INFO:teuthology.orchestra.run.smithi177.stdout:(71/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 132 kB 00:00 2024-03-28T00:44:24.138 INFO:teuthology.orchestra.run.smithi174.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 141 kB/s | 26 kB 00:00 2024-03-28T00:44:24.185 INFO:teuthology.orchestra.run.smithi177.stdout:(72/174): python3-pyasn1-modules-0.4.8-6.el9.no 1.8 MB/s | 211 kB 00:00 2024-03-28T00:44:24.210 INFO:teuthology.orchestra.run.smithi177.stdout:(73/174): python3-pytz-2021.1-5.el9.noarch.rpm 404 kB/s | 47 kB 00:00 2024-03-28T00:44:24.235 INFO:teuthology.orchestra.run.smithi177.stdout:(74/174): python3-requests-oauthlib-1.3.0-12.el 364 kB/s | 43 kB 00:00 2024-03-28T00:44:24.305 INFO:teuthology.orchestra.run.smithi174.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 292 kB/s | 49 kB 00:00 2024-03-28T00:44:24.422 INFO:teuthology.orchestra.run.smithi174.stdout:(111/174): postgresql-server-13.14-1.el9.x86_64 2.1 MB/s | 5.8 MB 00:02 2024-03-28T00:44:24.452 INFO:teuthology.orchestra.run.smithi177.stdout:(75/174): python3-toml-0.10.2-6.el9.noarch.rpm 159 kB/s | 39 kB 00:00 2024-03-28T00:44:24.605 INFO:teuthology.orchestra.run.smithi174.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 224 kB/s | 67 kB 00:00 2024-03-28T00:44:24.628 INFO:teuthology.orchestra.run.smithi177.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 104 kB/s | 41 kB 00:00 2024-03-28T00:44:24.864 INFO:teuthology.orchestra.run.smithi174.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 342 kB/s | 151 kB 00:00 2024-03-28T00:44:24.986 INFO:teuthology.orchestra.run.smithi177.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-03-28T00:44:25.011 INFO:teuthology.orchestra.run.smithi177.stdout:(78/174): socat-1.7.4.1-5.el9.x86_64.rpm 539 kB/s | 300 kB 00:00 2024-03-28T00:44:25.036 INFO:teuthology.orchestra.run.smithi177.stdout:(79/174): uuid-1.6.2-55.el9.x86_64.rpm 139 kB/s | 56 kB 00:00 2024-03-28T00:44:25.103 INFO:teuthology.orchestra.run.smithi177.stdout:(80/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 540 kB/s | 63 kB 00:00 2024-03-28T00:44:25.145 INFO:teuthology.orchestra.run.smithi177.stdout:(81/174): fuse-2.9.9-15.el9.x86_64.rpm 726 kB/s | 79 kB 00:00 2024-03-28T00:44:25.170 INFO:teuthology.orchestra.run.smithi177.stdout:(82/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 330 kB/s | 44 kB 00:00 2024-03-28T00:44:25.229 INFO:teuthology.orchestra.run.smithi177.stdout:(83/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 241 kB 00:00 2024-03-28T00:44:25.304 INFO:teuthology.orchestra.run.smithi177.stdout:(84/174): python3-cryptography-36.0.1-4.el9.x86 7.2 MB/s | 1.1 MB 00:00 2024-03-28T00:44:25.329 INFO:teuthology.orchestra.run.smithi177.stdout:(85/174): python3-ply-3.11-14.el9.noarch.rpm 648 kB/s | 103 kB 00:00 2024-03-28T00:44:25.355 INFO:teuthology.orchestra.run.smithi177.stdout:(86/174): python3-pycparser-2.20-6.el9.noarch.r 990 kB/s | 124 kB 00:00 2024-03-28T00:44:25.414 INFO:teuthology.orchestra.run.smithi174.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 2.9 MB/s | 4.4 MB 00:01 2024-03-28T00:44:25.430 INFO:teuthology.orchestra.run.smithi177.stdout:(87/174): python3-urllib3-1.26.5-5.el9.noarch.r 1.5 MB/s | 187 kB 00:00 2024-03-28T00:44:25.497 INFO:teuthology.orchestra.run.smithi177.stdout:(88/174): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-28T00:44:25.514 INFO:teuthology.orchestra.run.smithi174.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 435 kB/s | 43 kB 00:00 2024-03-28T00:44:25.548 INFO:teuthology.orchestra.run.smithi174.stdout:(116/174): python3-asyncssh-2.13.2-5.el9.noarch 802 kB/s | 548 kB 00:00 2024-03-28T00:44:25.556 INFO:teuthology.orchestra.run.smithi177.stdout:(89/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 803 kB 00:00 2024-03-28T00:44:25.581 INFO:teuthology.orchestra.run.smithi177.stdout:(90/174): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-03-28T00:44:25.606 INFO:teuthology.orchestra.run.smithi174.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 257 kB/s | 15 kB 00:00 2024-03-28T00:44:25.606 INFO:teuthology.orchestra.run.smithi177.stdout:(91/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-28T00:44:25.631 INFO:teuthology.orchestra.run.smithi174.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 275 kB/s | 32 kB 00:00 2024-03-28T00:44:25.741 INFO:teuthology.orchestra.run.smithi177.stdout:(92/174): elfutils-libelf-devel-0.190-2.el9.x86 117 kB/s | 21 kB 00:00 2024-03-28T00:44:25.765 INFO:teuthology.orchestra.run.smithi177.stdout:(93/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 138 kB/s | 25 kB 00:00 2024-03-28T00:44:25.765 INFO:teuthology.orchestra.run.smithi174.stdout:(119/174): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-03-28T00:44:25.790 INFO:teuthology.orchestra.run.smithi174.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 935 kB/s | 172 kB 00:00 2024-03-28T00:44:25.882 INFO:teuthology.orchestra.run.smithi174.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 384 kB/s | 35 kB 00:00 2024-03-28T00:44:25.908 INFO:teuthology.orchestra.run.smithi174.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.1 MB/s | 303 kB 00:00 2024-03-28T00:44:26.032 INFO:teuthology.orchestra.run.smithi177.stdout:(94/174): postgresql-contrib-13.14-1.el9.x86_64 3.3 MB/s | 911 kB 00:00 2024-03-28T00:44:26.091 INFO:teuthology.orchestra.run.smithi177.stdout:(95/174): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-03-28T00:44:26.141 INFO:teuthology.orchestra.run.smithi174.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 784 kB/s | 203 kB 00:00 2024-03-28T00:44:26.282 INFO:teuthology.orchestra.run.smithi177.stdout:(96/174): postgresql-13.14-1.el9.x86_64.rpm 3.1 MB/s | 1.6 MB 00:00 2024-03-28T00:44:26.333 INFO:teuthology.orchestra.run.smithi174.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 4.8 MB/s | 2.0 MB 00:00 2024-03-28T00:44:26.358 INFO:teuthology.orchestra.run.smithi174.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 666 kB/s | 144 kB 00:00 2024-03-28T00:44:26.391 INFO:teuthology.orchestra.run.smithi177.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-03-28T00:44:26.417 INFO:teuthology.orchestra.run.smithi174.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 128 kB/s | 11 kB 00:00 2024-03-28T00:44:26.442 INFO:teuthology.orchestra.run.smithi174.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 211 kB/s | 18 kB 00:00 2024-03-28T00:44:26.501 INFO:teuthology.orchestra.run.smithi174.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 278 kB/s | 23 kB 00:00 2024-03-28T00:44:26.508 INFO:teuthology.orchestra.run.smithi177.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.1 MB/s | 248 kB 00:00 2024-03-28T00:44:26.526 INFO:teuthology.orchestra.run.smithi174.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 233 kB/s | 19 kB 00:00 2024-03-28T00:44:26.583 INFO:teuthology.orchestra.run.smithi177.stdout:(99/174): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-03-28T00:44:26.584 INFO:teuthology.orchestra.run.smithi174.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 235 kB/s | 20 kB 00:00 2024-03-28T00:44:26.642 INFO:teuthology.orchestra.run.smithi177.stdout:(100/174): openssl-devel-3.0.7-27.el9.x86_64.rp 4.0 MB/s | 4.1 MB 00:01 2024-03-28T00:44:26.667 INFO:teuthology.orchestra.run.smithi177.stdout:(101/174): qatlib-service-24.02.0-1.el9.x86_64. 431 kB/s | 36 kB 00:00 2024-03-28T00:44:26.677 INFO:teuthology.orchestra.run.smithi174.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 504 kB/s | 46 kB 00:00 2024-03-28T00:44:26.709 INFO:teuthology.orchestra.run.smithi177.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 711 kB/s | 47 kB 00:00 2024-03-28T00:44:26.734 INFO:teuthology.orchestra.run.smithi177.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.4 MB/s | 94 kB 00:00 2024-03-28T00:44:26.802 INFO:teuthology.orchestra.run.smithi174.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 632 kB/s | 79 kB 00:00 2024-03-28T00:44:26.901 INFO:teuthology.orchestra.run.smithi177.stdout:(104/174): postgresql-server-13.14-1.el9.x86_64 7.1 MB/s | 5.8 MB 00:00 2024-03-28T00:44:26.910 INFO:teuthology.orchestra.run.smithi174.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 532 kB/s | 58 kB 00:00 2024-03-28T00:44:27.010 INFO:teuthology.orchestra.run.smithi177.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 74 kB/s | 22 kB 00:00 2024-03-28T00:44:27.086 INFO:teuthology.orchestra.run.smithi174.stdout:(134/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-03-28T00:44:27.111 INFO:teuthology.orchestra.run.smithi174.stdout:(135/174): python3-kubernetes-26.1.0-2.el9.noar 1.7 MB/s | 1.0 MB 00:00 2024-03-28T00:44:27.160 INFO:teuthology.orchestra.run.smithi177.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 2.0 MB/s | 309 kB 00:00 2024-03-28T00:44:27.170 INFO:teuthology.orchestra.run.smithi174.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 197 kB/s | 16 kB 00:00 2024-03-28T00:44:27.195 INFO:teuthology.orchestra.run.smithi174.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-03-28T00:44:27.202 INFO:teuthology.orchestra.run.smithi177.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 471 kB/s | 19 kB 00:00 2024-03-28T00:44:27.262 INFO:teuthology.orchestra.run.smithi174.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 307 kB/s | 28 kB 00:00 2024-03-28T00:44:27.285 INFO:teuthology.orchestra.run.smithi177.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.4 MB/s | 548 kB 00:00 2024-03-28T00:44:27.287 INFO:teuthology.orchestra.run.smithi174.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 334 kB/s | 31 kB 00:00 2024-03-28T00:44:27.344 INFO:teuthology.orchestra.run.smithi177.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-03-28T00:44:27.379 INFO:teuthology.orchestra.run.smithi174.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-03-28T00:44:27.402 INFO:teuthology.orchestra.run.smithi177.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 833 kB/s | 49 kB 00:00 2024-03-28T00:44:27.404 INFO:teuthology.orchestra.run.smithi174.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 506 kB/s | 59 kB 00:00 2024-03-28T00:44:27.461 INFO:teuthology.orchestra.run.smithi177.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-28T00:44:27.505 INFO:teuthology.orchestra.run.smithi174.stdout:(142/174): python3-typing-extensions-4.4.0-2.el 509 kB/s | 51 kB 00:00 2024-03-28T00:44:27.530 INFO:teuthology.orchestra.run.smithi174.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 237 kB/s | 36 kB 00:00 2024-03-28T00:44:27.553 INFO:teuthology.orchestra.run.smithi177.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-03-28T00:44:27.613 INFO:teuthology.orchestra.run.smithi174.stdout:(144/174): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-03-28T00:44:27.639 INFO:teuthology.orchestra.run.smithi174.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-28T00:44:27.670 INFO:teuthology.orchestra.run.smithi177.stdout:(113/174): protobuf-compiler-3.14.0-13.el9.x86_ 923 kB/s | 863 kB 00:00 2024-03-28T00:44:27.672 INFO:teuthology.orchestra.run.smithi174.stdout:(146/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 273 kB/s | 838 kB 00:03 2024-03-28T00:44:27.788 INFO:teuthology.orchestra.run.smithi174.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 134 kB/s | 20 kB 00:00 2024-03-28T00:44:27.814 INFO:teuthology.orchestra.run.smithi174.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.1 MB/s | 427 kB 00:00 2024-03-28T00:44:27.828 INFO:teuthology.orchestra.run.smithi177.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.0 MB/s | 838 kB 00:00 2024-03-28T00:44:27.862 INFO:teuthology.orchestra.run.smithi177.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.3 MB/s | 43 kB 00:00 2024-03-28T00:44:27.946 INFO:teuthology.orchestra.run.smithi177.stdout:(116/174): libarrow-9.0.0-10.el9.x86_64.rpm 5.9 MB/s | 4.4 MB 00:00 2024-03-28T00:44:27.962 INFO:teuthology.orchestra.run.smithi177.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 321 kB/s | 32 kB 00:00 2024-03-28T00:44:27.988 INFO:teuthology.orchestra.run.smithi177.stdout:(118/174): python3-asyncssh-2.13.2-5.el9.noarch 1.7 MB/s | 548 kB 00:00 2024-03-28T00:44:28.013 INFO:teuthology.orchestra.run.smithi177.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-03-28T00:44:28.038 INFO:teuthology.orchestra.run.smithi177.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-28T00:44:28.080 INFO:teuthology.orchestra.run.smithi177.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 849 kB/s | 35 kB 00:00 2024-03-28T00:44:28.090 INFO:teuthology.orchestra.run.smithi174.stdout:(149/174): librados2-19.0.0-2145.g3be9a5c9.el9. 12 MB/s | 3.4 MB 00:00 2024-03-28T00:44:28.105 INFO:teuthology.orchestra.run.smithi177.stdout:(122/174): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-03-28T00:44:28.131 INFO:teuthology.orchestra.run.smithi177.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-03-28T00:44:28.156 INFO:teuthology.orchestra.run.smithi177.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-03-28T00:44:28.198 INFO:teuthology.orchestra.run.smithi177.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 256 kB/s | 11 kB 00:00 2024-03-28T00:44:28.232 INFO:teuthology.orchestra.run.smithi177.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-03-28T00:44:28.256 INFO:teuthology.orchestra.run.smithi174.stdout:(150/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 19 MB/s | 3.1 MB 00:00 2024-03-28T00:44:28.257 INFO:teuthology.orchestra.run.smithi177.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 302 kB/s | 18 kB 00:00 2024-03-28T00:44:28.290 INFO:teuthology.orchestra.run.smithi177.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 397 kB/s | 23 kB 00:00 2024-03-28T00:44:28.290 INFO:teuthology.orchestra.run.smithi174.stdout:(151/174): thrift-0.15.0-2.el9.x86_64.rpm 3.1 MB/s | 1.6 MB 00:00 2024-03-28T00:44:28.315 INFO:teuthology.orchestra.run.smithi177.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 332 kB/s | 19 kB 00:00 2024-03-28T00:44:28.349 INFO:teuthology.orchestra.run.smithi177.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 336 kB/s | 20 kB 00:00 2024-03-28T00:44:28.374 INFO:teuthology.orchestra.run.smithi174.stdout:(152/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.7 MB/s | 321 kB 00:00 2024-03-28T00:44:28.408 INFO:teuthology.orchestra.run.smithi177.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 796 kB/s | 46 kB 00:00 2024-03-28T00:44:28.458 INFO:teuthology.orchestra.run.smithi177.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-03-28T00:44:28.517 INFO:teuthology.orchestra.run.smithi177.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 990 kB/s | 58 kB 00:00 2024-03-28T00:44:28.584 INFO:teuthology.orchestra.run.smithi177.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 3.8 MB/s | 1.0 MB 00:00 2024-03-28T00:44:28.626 INFO:teuthology.orchestra.run.smithi177.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-03-28T00:44:28.659 INFO:teuthology.orchestra.run.smithi177.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-03-28T00:44:28.701 INFO:teuthology.orchestra.run.smithi177.stdout:(137/174): python3-pyperclip-1.8.2-2.el9.noarch 677 kB/s | 28 kB 00:00 2024-03-28T00:44:28.726 INFO:teuthology.orchestra.run.smithi177.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 895 kB/s | 90 kB 00:00 2024-03-28T00:44:28.752 INFO:teuthology.orchestra.run.smithi177.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 614 kB/s | 31 kB 00:00 2024-03-28T00:44:28.793 INFO:teuthology.orchestra.run.smithi177.stdout:(140/174): python3-grpcio-1.46.7-10.el9.x86_64. 3.0 MB/s | 2.0 MB 00:00 2024-03-28T00:44:28.819 INFO:teuthology.orchestra.run.smithi177.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 884 kB/s | 59 kB 00:00 2024-03-28T00:44:28.844 INFO:teuthology.orchestra.run.smithi177.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 714 kB/s | 36 kB 00:00 2024-03-28T00:44:28.869 INFO:teuthology.orchestra.run.smithi177.stdout:(143/174): python3-routes-2.5.1-5.el9.noarch.rp 1.3 MB/s | 188 kB 00:00 2024-03-28T00:44:28.895 INFO:teuthology.orchestra.run.smithi177.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2024-03-28T00:44:28.920 INFO:teuthology.orchestra.run.smithi177.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-03-28T00:44:28.945 INFO:teuthology.orchestra.run.smithi177.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-28T00:44:28.970 INFO:teuthology.orchestra.run.smithi177.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-03-28T00:44:29.004 INFO:teuthology.orchestra.run.smithi177.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-28T00:44:29.029 INFO:teuthology.orchestra.run.smithi177.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-03-28T00:44:29.149 INFO:teuthology.orchestra.run.smithi174.stdout:(153/174): re2-20211101-3.el9.x86_64.rpm 130 kB/s | 192 kB 00:01 2024-03-28T00:44:29.479 INFO:teuthology.orchestra.run.smithi177.stdout:(150/174): librados2-19.0.0-2145.g3be9a5c9.el9. 7.1 MB/s | 3.4 MB 00:00 2024-03-28T00:44:29.513 INFO:teuthology.orchestra.run.smithi177.stdout:(151/174): thrift-0.15.0-2.el9.x86_64.rpm 2.9 MB/s | 1.6 MB 00:00 2024-03-28T00:44:29.833 INFO:teuthology.orchestra.run.smithi174.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 62 kB/s | 42 kB 00:00 2024-03-28T00:44:29.880 INFO:teuthology.orchestra.run.smithi177.stdout:(152/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 3.8 MB/s | 3.1 MB 00:00 2024-03-28T00:44:29.972 INFO:teuthology.orchestra.run.smithi177.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 5.0 MB/s | 2.3 MB 00:00 2024-03-28T00:44:30.022 INFO:teuthology.orchestra.run.smithi177.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 845 kB/s | 42 kB 00:00 2024-03-28T00:44:30.081 INFO:teuthology.orchestra.run.smithi177.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 4.2 MB/s | 248 kB 00:00 2024-03-28T00:44:30.122 INFO:teuthology.orchestra.run.smithi177.stdout:(156/174): NetworkManager-libnm-1.47.2-1.el9.x8 7.6 MB/s | 1.8 MB 00:00 2024-03-28T00:44:30.165 INFO:teuthology.orchestra.run.smithi177.stdout:(157/174): xfsdump-3.1.12-4.el9.x86_64.rpm 469 kB/s | 321 kB 00:00 2024-03-28T00:44:30.181 INFO:teuthology.orchestra.run.smithi177.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 623 kB/s | 36 kB 00:00 2024-03-28T00:44:30.207 INFO:teuthology.orchestra.run.smithi177.stdout:(159/174): acl-2.3.1-4.el9.x86_64.rpm 568 kB/s | 71 kB 00:00 2024-03-28T00:44:30.232 INFO:teuthology.orchestra.run.smithi177.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 5.0 MB/s | 256 kB 00:00 2024-03-28T00:44:30.249 INFO:teuthology.orchestra.run.smithi177.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 552 kB/s | 23 kB 00:00 2024-03-28T00:44:30.282 INFO:teuthology.orchestra.run.smithi177.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-28T00:44:30.316 INFO:teuthology.orchestra.run.smithi177.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 155 kB 00:00 2024-03-28T00:44:30.341 INFO:teuthology.orchestra.run.smithi177.stdout:(164/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.1 MB/s | 192 kB 00:00 2024-03-28T00:44:30.367 INFO:teuthology.orchestra.run.smithi177.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 136 kB 00:00 2024-03-28T00:44:30.392 INFO:teuthology.orchestra.run.smithi177.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 571 kB/s | 29 kB 00:00 2024-03-28T00:44:30.417 INFO:teuthology.orchestra.run.smithi177.stdout:(167/174): libsmartcols-2.37.4-18.el9.x86_64.rp 634 kB/s | 64 kB 00:00 2024-03-28T00:44:30.459 INFO:teuthology.orchestra.run.smithi177.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 13 MB/s | 1.2 MB 00:00 2024-03-28T00:44:30.518 INFO:teuthology.orchestra.run.smithi177.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 7.8 MB/s | 465 kB 00:00 2024-03-28T00:44:30.568 INFO:teuthology.orchestra.run.smithi177.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 15 MB/s | 2.3 MB 00:00 2024-03-28T00:44:30.593 INFO:teuthology.orchestra.run.smithi177.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 414 kB/s | 31 kB 00:00 2024-03-28T00:44:30.635 INFO:teuthology.orchestra.run.smithi177.stdout:(172/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 8.9 MB/s | 2.2 MB 00:00 2024-03-28T00:44:30.686 INFO:teuthology.orchestra.run.smithi177.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.2 MB/s | 1.1 MB 00:00 2024-03-28T00:44:30.744 INFO:teuthology.orchestra.run.smithi177.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 495 kB/s | 74 kB 00:00 2024-03-28T00:44:30.748 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-28T00:44:30.748 INFO:teuthology.orchestra.run.smithi177.stdout:Total 14 MB/s | 225 MB 00:16 2024-03-28T00:44:30.766 INFO:teuthology.orchestra.run.smithi174.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 266 kB/s | 248 kB 00:00 2024-03-28T00:44:31.304 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-28T00:44:31.450 INFO:teuthology.orchestra.run.smithi174.stdout:(156/174): NetworkManager-libnm-1.47.2-1.el9.x8 610 kB/s | 1.8 MB 00:03 2024-03-28T00:44:31.475 INFO:teuthology.orchestra.run.smithi174.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 101 kB/s | 71 kB 00:00 2024-03-28T00:44:31.578 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-28T00:44:31.578 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-28T00:44:32.075 INFO:teuthology.orchestra.run.smithi174.stdout:(158/174): NetworkManager-1.47.2-1.el9.x86_64.r 624 kB/s | 2.3 MB 00:03 2024-03-28T00:44:32.100 INFO:teuthology.orchestra.run.smithi174.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 56 kB/s | 36 kB 00:00 2024-03-28T00:44:32.294 INFO:teuthology.orchestra.run.smithi174.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 235 kB/s | 192 kB 00:00 2024-03-28T00:44:32.676 INFO:teuthology.orchestra.run.smithi174.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 40 kB/s | 23 kB 00:00 2024-03-28T00:44:32.976 INFO:teuthology.orchestra.run.smithi174.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 284 kB/s | 256 kB 00:00 2024-03-28T00:44:33.018 INFO:teuthology.orchestra.run.smithi174.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 149 kB/s | 108 kB 00:00 2024-03-28T00:44:33.313 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-28T00:44:33.314 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-28T00:44:33.501 INFO:teuthology.orchestra.run.smithi174.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 188 kB/s | 155 kB 00:00 2024-03-28T00:44:33.685 INFO:teuthology.orchestra.run.smithi174.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 96 kB/s | 64 kB 00:00 2024-03-28T00:44:33.727 INFO:teuthology.orchestra.run.smithi174.stdout:(166/174): libmount-2.37.4-18.el9.x86_64.rpm 181 kB/s | 136 kB 00:00 2024-03-28T00:44:34.085 INFO:teuthology.orchestra.run.smithi174.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 49 kB/s | 29 kB 00:00 2024-03-28T00:44:35.977 INFO:teuthology.orchestra.run.smithi174.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 541 kB/s | 1.2 MB 00:02 2024-03-28T00:44:37.194 INFO:teuthology.orchestra.run.smithi174.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 382 kB/s | 465 kB 00:01 2024-03-28T00:44:37.336 INFO:teuthology.orchestra.run.smithi174.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 616 kB/s | 2.2 MB 00:03 2024-03-28T00:44:37.744 INFO:teuthology.orchestra.run.smithi174.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 639 kB/s | 2.3 MB 00:03 2024-03-28T00:44:37.786 INFO:teuthology.orchestra.run.smithi174.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 53 kB/s | 31 kB 00:00 2024-03-28T00:44:37.928 INFO:teuthology.orchestra.run.smithi174.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 406 kB/s | 74 kB 00:00 2024-03-28T00:44:39.295 INFO:teuthology.orchestra.run.smithi174.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 561 kB/s | 1.1 MB 00:01 2024-03-28T00:44:39.301 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-28T00:44:39.301 INFO:teuthology.orchestra.run.smithi174.stdout:Total 7.3 MB/s | 225 MB 00:30 2024-03-28T00:44:39.854 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-28T00:44:40.124 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-28T00:44:40.124 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-28T00:44:40.161 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-28T00:44:40.264 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T00:44:40.486 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T00:44:40.554 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T00:44:40.565 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T00:44:40.621 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T00:44:40.676 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T00:44:40.716 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T00:44:40.750 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T00:44:40.793 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T00:44:41.121 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T00:44:41.176 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T00:44:41.288 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T00:44:41.310 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T00:44:41.410 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T00:44:41.439 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T00:44:41.480 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T00:44:41.662 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T00:44:41.720 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T00:44:41.748 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T00:44:41.835 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T00:44:41.846 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-28T00:44:41.846 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-28T00:44:41.858 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T00:44:41.921 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T00:44:41.961 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T00:44:41.984 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T00:44:42.000 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T00:44:42.192 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T00:44:42.223 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T00:44:42.255 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T00:44:42.350 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T00:44:42.441 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T00:44:42.484 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T00:44:42.550 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T00:44:42.589 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T00:44:42.806 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T00:44:42.905 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T00:44:43.011 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T00:44:43.056 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T00:44:43.083 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T00:44:43.118 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T00:44:43.194 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T00:44:43.246 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T00:44:43.287 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T00:44:43.351 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T00:44:43.386 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T00:44:43.430 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T00:44:43.462 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T00:44:43.490 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T00:44:43.558 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T00:44:43.595 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T00:44:43.638 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T00:44:43.667 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T00:44:43.734 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T00:44:43.781 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T00:44:43.854 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T00:44:43.987 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T00:44:44.033 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T00:44:44.331 INFO:teuthology.orchestra.run.smithi169.stdout:(174/174): abseil-cpp-20211102.0-3.el9.x86_64.r 17 kB/s | 548 kB 00:32 2024-03-28T00:44:44.337 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-28T00:44:44.338 INFO:teuthology.orchestra.run.smithi169.stdout:Total 5.1 MB/s | 225 MB 00:44 2024-03-28T00:44:44.553 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T00:44:44.628 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T00:44:44.720 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T00:44:44.765 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T00:44:44.774 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T00:44:44.805 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T00:44:44.846 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T00:44:44.884 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T00:44:44.913 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-28T00:44:44.914 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T00:44:45.150 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T00:44:45.190 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-28T00:44:45.190 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-28T00:44:45.318 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T00:44:45.400 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T00:44:45.443 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T00:44:45.572 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T00:44:45.621 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T00:44:45.677 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T00:44:45.837 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T00:44:45.872 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T00:44:45.913 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T00:44:45.955 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T00:44:45.994 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T00:44:46.043 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T00:44:46.094 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T00:44:46.226 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T00:44:46.273 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T00:44:46.304 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T00:44:46.345 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T00:44:46.439 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T00:44:46.487 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T00:44:46.526 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T00:44:46.552 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T00:44:46.577 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T00:44:46.649 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T00:44:46.769 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T00:44:46.868 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T00:44:46.910 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T00:44:46.958 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T00:44:47.022 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T00:44:47.190 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-28T00:44:47.191 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-28T00:44:47.204 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T00:44:47.257 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T00:44:47.400 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T00:44:47.446 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T00:44:47.478 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T00:44:47.509 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T00:44:47.536 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T00:44:47.580 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T00:44:47.619 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T00:44:47.650 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T00:44:47.680 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T00:44:47.755 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T00:44:47.790 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T00:44:47.824 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T00:44:47.856 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T00:44:47.891 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T00:44:47.980 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T00:44:48.061 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T00:44:48.105 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T00:44:48.137 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T00:44:48.184 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T00:44:48.280 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T00:44:48.338 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T00:44:48.742 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-28T00:44:48.857 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T00:44:48.883 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T00:44:48.922 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T00:44:48.930 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T00:44:48.971 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T00:44:49.024 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T00:44:49.058 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T00:44:49.063 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T00:44:49.075 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T00:44:49.110 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T00:44:49.117 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T00:44:49.144 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T00:44:49.191 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T00:44:49.270 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T00:44:49.303 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T00:44:49.437 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T00:44:49.480 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T00:44:49.492 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T00:44:49.509 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T00:44:49.586 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T00:44:49.596 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T00:44:49.627 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T00:44:49.717 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T00:44:49.747 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T00:44:49.788 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T00:44:49.948 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T00:44:49.996 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T00:44:50.024 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T00:44:50.126 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T00:44:50.149 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T00:44:50.205 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T00:44:50.220 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T00:44:50.253 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T00:44:50.276 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T00:44:50.290 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T00:44:50.331 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T00:44:50.442 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T00:44:50.473 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T00:44:50.513 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T00:44:50.617 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T00:44:50.707 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T00:44:50.767 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T00:44:50.837 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T00:44:50.893 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T00:44:51.117 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T00:44:51.238 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T00:44:51.335 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T00:44:51.372 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T00:44:51.399 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T00:44:51.435 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T00:44:51.509 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T00:44:51.563 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T00:44:51.601 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T00:44:51.665 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T00:44:51.700 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T00:44:51.741 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T00:44:51.743 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T00:44:51.776 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T00:44:51.812 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T00:44:51.826 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T00:44:51.856 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T00:44:51.881 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T00:44:51.887 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T00:44:51.916 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T00:44:51.917 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T00:44:51.960 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T00:44:51.989 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T00:44:52.057 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T00:44:52.111 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T00:44:52.185 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T00:44:52.227 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T00:44:52.262 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T00:44:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T00:44:52.344 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T00:44:52.372 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T00:44:52.375 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T00:44:52.411 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T00:44:52.872 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T00:44:52.891 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T00:44:52.898 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T00:44:52.969 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T00:44:52.976 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T00:44:52.991 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T00:44:53.055 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T00:44:53.095 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T00:44:53.104 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T00:44:53.135 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T00:44:53.176 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T00:44:53.222 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T00:44:53.251 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T00:44:53.508 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T00:44:53.654 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T00:44:53.720 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T00:44:53.756 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T00:44:53.898 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T00:44:53.942 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T00:44:53.997 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T00:44:54.069 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T00:44:54.093 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-28T00:44:54.125 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T00:44:54.161 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T00:44:54.194 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T00:44:54.209 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T00:44:54.228 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T00:44:54.245 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T00:44:54.266 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T00:44:54.276 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T00:44:54.287 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T00:44:54.325 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T00:44:54.327 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T00:44:54.379 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T00:44:54.415 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T00:44:54.449 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T00:44:54.460 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T00:44:54.499 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T00:44:54.502 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T00:44:54.529 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T00:44:54.571 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T00:44:54.664 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T00:44:54.712 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T00:44:54.752 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T00:44:54.778 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T00:44:54.802 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T00:44:54.832 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T00:44:54.874 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T00:44:54.909 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T00:44:55.003 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T00:44:55.030 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T00:44:55.060 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T00:44:55.106 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T00:44:55.134 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T00:44:55.151 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T00:44:55.164 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T00:44:55.199 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T00:44:55.205 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T00:44:55.253 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T00:44:55.256 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T00:44:55.288 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T00:44:55.393 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T00:44:55.439 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T00:44:55.446 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T00:44:55.473 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T00:44:55.480 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T00:44:55.569 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T00:44:55.591 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T00:44:55.612 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T00:44:55.640 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T00:44:55.653 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T00:44:55.671 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T00:44:55.685 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T00:44:55.685 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T00:44:55.701 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T00:44:55.716 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T00:44:55.744 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T00:44:55.778 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T00:44:55.817 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T00:44:55.824 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T00:44:55.848 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T00:44:55.852 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T00:44:55.858 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T00:44:55.879 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T00:44:55.883 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T00:44:55.928 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T00:44:55.941 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T00:44:55.941 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T00:44:55.941 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T00:44:55.942 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-28T00:44:55.956 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T00:44:55.989 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T00:44:56.014 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T00:44:56.026 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T00:44:56.055 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T00:44:56.090 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T00:44:56.112 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T00:44:56.180 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T00:44:56.247 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T00:44:56.270 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T00:44:56.307 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T00:44:56.312 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T00:44:56.344 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T00:44:56.391 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T00:44:56.402 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T00:44:56.450 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T00:44:56.489 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T00:44:56.553 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T00:44:56.664 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T00:44:56.773 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T00:44:56.868 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T00:44:56.906 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T00:44:56.932 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T00:44:56.968 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T00:44:57.042 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T00:44:57.096 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T00:44:57.134 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T00:44:57.198 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T00:44:57.233 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T00:44:57.277 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T00:44:57.293 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T00:44:57.325 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T00:44:57.326 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T00:44:57.353 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T00:44:57.358 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T00:44:57.406 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T00:44:57.425 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T00:44:57.459 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T00:44:57.509 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T00:44:57.522 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T00:44:57.539 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T00:44:57.608 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T00:44:57.653 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T00:44:57.713 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T00:44:57.728 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T00:44:57.861 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T00:44:57.914 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T00:44:57.979 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T00:44:58.007 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T00:44:58.084 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T00:44:58.436 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T00:44:58.518 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T00:44:58.595 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T00:44:58.646 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T00:44:58.655 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T00:44:58.686 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T00:44:58.719 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T00:44:58.738 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T00:44:58.773 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T00:44:58.804 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T00:44:58.820 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T00:44:59.084 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T00:44:59.236 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T00:44:59.313 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T00:44:59.358 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T00:44:59.515 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T00:44:59.594 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T00:44:59.649 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T00:44:59.721 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T00:44:59.752 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T00:44:59.781 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T00:44:59.828 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T00:44:59.863 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T00:44:59.907 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T00:44:59.949 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T00:45:00.021 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T00:45:00.084 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T00:45:00.106 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T00:45:00.129 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T00:45:00.151 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T00:45:00.153 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T00:45:00.185 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T00:45:00.193 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T00:45:00.212 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T00:45:00.288 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T00:45:00.334 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T00:45:00.375 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T00:45:00.400 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T00:45:00.424 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T00:45:00.502 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T00:45:00.524 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T00:45:00.550 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T00:45:00.626 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T00:45:00.641 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T00:45:00.671 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T00:45:00.707 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T00:45:00.725 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T00:45:00.766 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T00:45:00.815 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T00:45:00.871 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T00:45:01.055 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T00:45:01.103 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T00:45:01.168 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T00:45:01.195 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T00:45:01.241 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T00:45:01.249 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T00:45:01.271 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T00:45:01.277 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T00:45:01.309 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T00:45:01.340 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T00:45:01.367 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T00:45:01.394 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T00:45:01.433 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T00:45:01.464 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T00:45:01.494 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T00:45:01.587 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T00:45:01.621 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T00:45:01.655 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T00:45:01.696 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T00:45:01.731 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T00:45:01.812 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T00:45:01.903 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T00:45:01.945 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T00:45:01.977 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T00:45:02.023 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T00:45:02.121 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T00:45:02.170 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T00:45:02.899 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T00:45:02.941 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T00:45:02.975 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T00:45:03.033 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T00:45:03.120 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T00:45:03.152 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T00:45:03.324 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T00:45:03.358 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T00:45:03.438 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T00:45:03.506 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T00:45:03.534 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T00:45:04.071 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T00:45:04.078 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T00:45:04.112 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T00:45:04.193 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T00:45:04.194 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T00:45:04.194 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T00:45:04.194 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-28T00:45:04.198 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T00:45:04.221 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T00:45:05.430 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T00:45:05.525 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T00:45:05.556 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T00:45:05.588 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T00:45:05.615 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T00:45:05.930 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T00:45:05.970 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T00:45:06.045 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T00:45:06.075 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T00:45:06.111 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T00:45:06.572 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T00:45:06.598 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T00:45:06.669 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T00:45:06.700 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T00:45:07.753 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T00:45:07.753 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /sys 2024-03-28T00:45:07.753 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /proc 2024-03-28T00:45:07.754 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /mnt 2024-03-28T00:45:07.754 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /var/tmp 2024-03-28T00:45:07.754 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /home 2024-03-28T00:45:07.754 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /root 2024-03-28T00:45:07.754 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /tmp 2024-03-28T00:45:07.754 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-28T00:45:07.913 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T00:45:07.928 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T00:45:07.928 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:07.928 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T00:45:07.928 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-28T00:45:08.972 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T00:45:09.006 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T00:45:09.424 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T00:45:09.449 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T00:45:09.449 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:09.450 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T00:45:09.450 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T00:45:09.450 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T00:45:09.450 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-28T00:45:09.553 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T00:45:09.593 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T00:45:09.667 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T00:45:09.667 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T00:45:09.667 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T00:45:09.668 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:45:09.675 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T00:45:09.697 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T00:45:09.697 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:09.697 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T00:45:09.697 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T00:45:09.698 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T00:45:09.698 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-28T00:45:09.702 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T00:45:10.133 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T00:45:10.156 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T00:45:10.156 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:10.156 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T00:45:10.156 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T00:45:10.156 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T00:45:10.157 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-28T00:45:10.244 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T00:45:10.620 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T00:45:10.651 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T00:45:10.683 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T00:45:10.742 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T00:45:10.765 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T00:45:11.001 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T00:45:11.033 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T00:45:11.997 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T00:45:12.016 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T00:45:12.115 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T00:45:12.279 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T00:45:12.310 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T00:45:12.333 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T00:45:12.333 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:12.333 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T00:45:12.333 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T00:45:12.333 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T00:45:12.333 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-28T00:45:12.378 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T00:45:12.388 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T00:45:12.433 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T00:45:13.306 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T00:45:13.338 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T00:45:13.360 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T00:45:13.360 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:13.360 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T00:45:13.360 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T00:45:13.360 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T00:45:13.360 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-28T00:45:13.405 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T00:45:13.426 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T00:45:13.426 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:13.426 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T00:45:13.427 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-28T00:45:13.472 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T00:45:13.494 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T00:45:13.494 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:13.494 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T00:45:13.495 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T00:45:13.495 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T00:45:13.495 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-28T00:45:13.798 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T00:45:13.820 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T00:45:13.820 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:13.821 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T00:45:13.821 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T00:45:13.821 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T00:45:13.821 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-28T00:45:15.836 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T00:45:15.836 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /sys 2024-03-28T00:45:15.836 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /proc 2024-03-28T00:45:15.836 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /mnt 2024-03-28T00:45:15.836 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /var/tmp 2024-03-28T00:45:15.836 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /home 2024-03-28T00:45:15.836 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /root 2024-03-28T00:45:15.836 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /tmp 2024-03-28T00:45:15.836 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-28T00:45:16.000 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T00:45:16.015 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T00:45:16.016 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:16.016 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T00:45:16.016 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-28T00:45:17.478 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T00:45:17.503 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T00:45:17.503 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:17.503 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T00:45:17.504 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T00:45:17.504 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T00:45:17.504 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-28T00:45:17.720 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T00:45:17.742 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T00:45:17.742 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:17.742 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T00:45:17.742 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T00:45:17.742 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T00:45:17.743 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-28T00:45:18.179 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T00:45:18.201 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T00:45:18.201 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:18.202 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T00:45:18.202 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T00:45:18.202 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T00:45:18.202 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-28T00:45:18.298 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T00:45:18.316 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T00:45:18.346 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T00:45:18.374 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T00:45:18.401 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T00:45:18.582 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T00:45:18.700 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T00:45:18.741 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T00:45:18.773 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T00:45:18.779 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T00:45:18.804 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T00:45:18.836 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T00:45:18.840 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T00:45:18.884 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T00:45:18.897 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T00:45:18.925 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T00:45:18.927 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T00:45:18.966 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T00:45:19.003 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T00:45:19.058 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T00:45:19.118 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T00:45:19.144 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T00:45:19.160 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T00:45:19.194 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T00:45:19.769 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T00:45:19.808 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T00:45:19.828 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T00:45:20.156 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T00:45:20.186 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T00:45:20.287 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T00:45:20.457 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T00:45:20.505 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T00:45:20.528 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T00:45:20.528 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:20.528 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T00:45:20.528 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T00:45:20.528 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T00:45:20.528 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-28T00:45:20.572 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T00:45:20.583 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T00:45:20.628 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T00:45:20.646 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T00:45:20.720 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T00:45:20.760 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T00:45:20.841 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T00:45:20.881 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T00:45:20.906 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T00:45:20.932 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T00:45:20.934 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T00:45:20.934 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /sys 2024-03-28T00:45:20.934 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /proc 2024-03-28T00:45:20.934 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /mnt 2024-03-28T00:45:20.935 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /var/tmp 2024-03-28T00:45:20.935 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /home 2024-03-28T00:45:20.935 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /root 2024-03-28T00:45:20.935 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /tmp 2024-03-28T00:45:20.935 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:45:20.958 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T00:45:20.958 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T00:45:21.046 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T00:45:21.075 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T00:45:21.076 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T00:45:21.097 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T00:45:21.112 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T00:45:21.113 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:21.113 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T00:45:21.113 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:45:21.160 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T00:45:21.186 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T00:45:21.230 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T00:45:21.255 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T00:45:21.280 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T00:45:21.313 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T00:45:21.353 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T00:45:21.393 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T00:45:21.418 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T00:45:21.444 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T00:45:21.469 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T00:45:21.494 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T00:45:21.500 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T00:45:21.500 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T00:45:21.501 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T00:45:21.532 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T00:45:21.555 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T00:45:21.555 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:21.555 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T00:45:21.555 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T00:45:21.555 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T00:45:21.555 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-28T00:45:21.600 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T00:45:21.621 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T00:45:21.622 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:21.622 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T00:45:21.622 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-28T00:45:21.667 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T00:45:21.680 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T00:45:21.680 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T00:45:21.689 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T00:45:21.689 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:21.689 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T00:45:21.689 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T00:45:21.690 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T00:45:21.690 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-28T00:45:21.764 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T00:45:21.788 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T00:45:21.820 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T00:45:21.848 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T00:45:21.873 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T00:45:21.938 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T00:45:21.993 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T00:45:22.015 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T00:45:22.015 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:22.015 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T00:45:22.015 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T00:45:22.015 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T00:45:22.015 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-28T00:45:22.608 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T00:45:22.633 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T00:45:22.634 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:22.634 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T00:45:22.634 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T00:45:22.634 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T00:45:22.634 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:45:22.876 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T00:45:22.899 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T00:45:22.899 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:22.899 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T00:45:22.899 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T00:45:22.899 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T00:45:22.899 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:45:23.344 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T00:45:23.366 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T00:45:23.366 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:23.366 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T00:45:23.367 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T00:45:23.367 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T00:45:23.367 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:45:23.471 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T00:45:23.905 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T00:45:23.935 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T00:45:23.967 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T00:45:24.029 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T00:45:24.058 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T00:45:24.292 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T00:45:24.326 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T00:45:25.289 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T00:45:25.318 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T00:45:25.419 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T00:45:25.572 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T00:45:25.604 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T00:45:25.627 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T00:45:25.627 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:25.627 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T00:45:25.627 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T00:45:25.627 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T00:45:25.627 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:45:25.671 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T00:45:25.682 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T00:45:25.732 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T00:45:26.438 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T00:45:26.479 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T00:45:26.510 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T00:45:26.537 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T00:45:26.606 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T00:45:26.664 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T00:45:26.688 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T00:45:26.688 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:26.688 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T00:45:26.688 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T00:45:26.688 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T00:45:26.688 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:45:26.732 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T00:45:26.735 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T00:45:26.754 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T00:45:26.754 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:26.754 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T00:45:26.754 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:45:26.799 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T00:45:26.823 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T00:45:26.823 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T00:45:26.823 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T00:45:26.824 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T00:45:26.824 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T00:45:26.824 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T00:45:26.824 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T00:45:26.824 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T00:45:26.825 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T00:45:26.825 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T00:45:26.825 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T00:45:26.825 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T00:45:26.825 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T00:45:26.825 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T00:45:26.825 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T00:45:26.825 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T00:45:26.825 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T00:45:26.825 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T00:45:26.825 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T00:45:26.825 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T00:45:26.826 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T00:45:26.826 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T00:45:26.826 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T00:45:26.826 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T00:45:26.826 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T00:45:26.826 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T00:45:26.826 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T00:45:26.826 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T00:45:26.826 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T00:45:26.826 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T00:45:26.827 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T00:45:26.827 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T00:45:26.827 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T00:45:26.827 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T00:45:26.827 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T00:45:26.827 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T00:45:26.827 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T00:45:26.827 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T00:45:26.827 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T00:45:26.827 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T00:45:26.827 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T00:45:26.828 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T00:45:26.828 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T00:45:26.828 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T00:45:26.828 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T00:45:26.828 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T00:45:26.828 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T00:45:26.828 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T00:45:26.828 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T00:45:26.828 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T00:45:26.828 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T00:45:26.829 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T00:45:26.829 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T00:45:26.829 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T00:45:26.829 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T00:45:26.829 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T00:45:26.829 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T00:45:26.829 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T00:45:26.829 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T00:45:26.829 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T00:45:26.829 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T00:45:26.829 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T00:45:26.830 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T00:45:26.830 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T00:45:26.830 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T00:45:26.830 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T00:45:26.830 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T00:45:26.830 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T00:45:26.830 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T00:45:26.830 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T00:45:26.830 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T00:45:26.830 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T00:45:26.830 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T00:45:26.831 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T00:45:26.831 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T00:45:26.831 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T00:45:26.831 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T00:45:26.831 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T00:45:26.831 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T00:45:26.831 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T00:45:26.831 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T00:45:26.831 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T00:45:26.831 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T00:45:26.831 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T00:45:26.832 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T00:45:26.832 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T00:45:26.832 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T00:45:26.832 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T00:45:26.832 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T00:45:26.832 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T00:45:26.832 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T00:45:26.832 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T00:45:26.832 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T00:45:26.832 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T00:45:26.832 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T00:45:26.833 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T00:45:26.833 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T00:45:26.833 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T00:45:26.833 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T00:45:26.833 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T00:45:26.833 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T00:45:26.833 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T00:45:26.833 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T00:45:26.833 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T00:45:26.833 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T00:45:26.833 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T00:45:26.834 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T00:45:26.834 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T00:45:26.834 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T00:45:26.834 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T00:45:26.834 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T00:45:26.834 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T00:45:26.834 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T00:45:26.834 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T00:45:26.834 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T00:45:26.834 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T00:45:26.834 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T00:45:26.835 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T00:45:26.835 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T00:45:26.835 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T00:45:26.835 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T00:45:26.835 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T00:45:26.835 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T00:45:26.835 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T00:45:26.835 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T00:45:26.835 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T00:45:26.835 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T00:45:26.835 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T00:45:26.835 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T00:45:26.836 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T00:45:26.836 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T00:45:26.836 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T00:45:26.836 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T00:45:26.836 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T00:45:26.836 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T00:45:26.836 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T00:45:26.836 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T00:45:26.836 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T00:45:26.836 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T00:45:26.836 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T00:45:26.837 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T00:45:26.837 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T00:45:26.837 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T00:45:26.837 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T00:45:26.837 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T00:45:26.837 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T00:45:26.837 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T00:45:26.837 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T00:45:26.837 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T00:45:26.837 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T00:45:26.837 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T00:45:26.838 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T00:45:26.838 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T00:45:26.838 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T00:45:26.838 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T00:45:26.838 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T00:45:26.838 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T00:45:26.838 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T00:45:26.838 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T00:45:26.838 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T00:45:26.838 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T00:45:26.838 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T00:45:26.838 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T00:45:26.839 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T00:45:26.839 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T00:45:26.839 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T00:45:26.839 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T00:45:26.839 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T00:45:26.839 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T00:45:26.839 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T00:45:26.839 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T00:45:26.839 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T00:45:26.839 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T00:45:26.840 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T00:45:26.840 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T00:45:26.840 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T00:45:26.840 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T00:45:26.840 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T00:45:26.840 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T00:45:26.840 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T00:45:26.840 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T00:45:26.840 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T00:45:26.840 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T00:45:26.840 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T00:45:26.840 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T00:45:26.841 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T00:45:26.841 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T00:45:26.841 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T00:45:26.841 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T00:45:26.841 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T00:45:26.841 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T00:45:26.841 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T00:45:26.841 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T00:45:26.841 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T00:45:26.841 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T00:45:26.841 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T00:45:26.842 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T00:45:26.842 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T00:45:26.842 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T00:45:26.842 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:26.842 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T00:45:26.842 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T00:45:26.843 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T00:45:26.843 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:45:26.861 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T00:45:26.901 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T00:45:26.940 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T00:45:27.001 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T00:45:27.044 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T00:45:27.077 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T00:45:27.109 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T00:45:27.147 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T00:45:27.150 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T00:45:27.173 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T00:45:27.173 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T00:45:27.173 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T00:45:27.173 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T00:45:27.173 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T00:45:27.173 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:45:27.201 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T00:45:27.253 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T00:45:27.288 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T00:45:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T00:45:27.877 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T00:45:27.896 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T00:45:28.720 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T00:45:28.804 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T00:45:28.844 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T00:45:28.925 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T00:45:28.966 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T00:45:28.991 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T00:45:29.016 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T00:45:29.043 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T00:45:29.043 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T00:45:29.131 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T00:45:29.160 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T00:45:29.160 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T00:45:29.228 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T00:45:29.254 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T00:45:29.296 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T00:45:29.322 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T00:45:29.346 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T00:45:29.379 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T00:45:29.418 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T00:45:29.451 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T00:45:29.476 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T00:45:29.501 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T00:45:29.527 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T00:45:29.552 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T00:45:29.558 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T00:45:29.558 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T00:45:29.738 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T00:45:29.738 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T00:45:29.804 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T00:45:29.829 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T00:45:29.861 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T00:45:29.877 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T00:45:29.877 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-28T00:45:29.877 INFO:teuthology.orchestra.run.smithi177.stdout:Upgraded: 2024-03-28T00:45:29.877 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T00:45:29.877 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T00:45:29.877 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T00:45:29.878 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T00:45:29.878 INFO:teuthology.orchestra.run.smithi177.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T00:45:29.878 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T00:45:29.878 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T00:45:29.878 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T00:45:29.878 INFO:teuthology.orchestra.run.smithi177.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T00:45:29.878 INFO:teuthology.orchestra.run.smithi177.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T00:45:29.879 INFO:teuthology.orchestra.run.smithi177.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T00:45:29.879 INFO:teuthology.orchestra.run.smithi177.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T00:45:29.879 INFO:teuthology.orchestra.run.smithi177.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T00:45:29.879 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.879 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.879 INFO:teuthology.orchestra.run.smithi177.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T00:45:29.879 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T00:45:29.879 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T00:45:29.879 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T00:45:29.879 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T00:45:29.879 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T00:45:29.879 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T00:45:29.879 INFO:teuthology.orchestra.run.smithi177.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T00:45:29.880 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T00:45:29.880 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-28T00:45:29.880 INFO:teuthology.orchestra.run.smithi177.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T00:45:29.880 INFO:teuthology.orchestra.run.smithi177.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T00:45:29.880 INFO:teuthology.orchestra.run.smithi177.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T00:45:29.880 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T00:45:29.880 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.880 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.880 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.880 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.880 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:29.881 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.881 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.881 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.881 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:29.881 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:29.881 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:29.881 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:29.881 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:29.881 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.881 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.881 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:29.881 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.882 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.882 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.882 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:29.882 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:29.882 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.882 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.882 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:29.882 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T00:45:29.882 INFO:teuthology.orchestra.run.smithi177.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T00:45:29.882 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T00:45:29.882 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T00:45:29.882 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T00:45:29.883 INFO:teuthology.orchestra.run.smithi177.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T00:45:29.883 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T00:45:29.883 INFO:teuthology.orchestra.run.smithi177.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T00:45:29.883 INFO:teuthology.orchestra.run.smithi177.stdout: jq-1.6-16.el9.x86_64 2024-03-28T00:45:29.883 INFO:teuthology.orchestra.run.smithi177.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T00:45:29.883 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T00:45:29.883 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T00:45:29.883 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.883 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.883 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.883 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T00:45:29.883 INFO:teuthology.orchestra.run.smithi177.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T00:45:29.883 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T00:45:29.884 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T00:45:29.884 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T00:45:29.884 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.884 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.884 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T00:45:29.884 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.884 INFO:teuthology.orchestra.run.smithi177.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T00:45:29.884 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T00:45:29.884 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T00:45:29.884 INFO:teuthology.orchestra.run.smithi177.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T00:45:29.884 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T00:45:29.884 INFO:teuthology.orchestra.run.smithi177.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T00:45:29.884 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T00:45:29.885 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T00:45:29.885 INFO:teuthology.orchestra.run.smithi177.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T00:45:29.885 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T00:45:29.885 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T00:45:29.885 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T00:45:29.885 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T00:45:29.885 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T00:45:29.885 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T00:45:29.885 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T00:45:29.885 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T00:45:29.885 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T00:45:29.885 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T00:45:29.886 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T00:45:29.886 INFO:teuthology.orchestra.run.smithi177.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T00:45:29.886 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T00:45:29.886 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T00:45:29.886 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T00:45:29.886 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.886 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.886 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.886 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T00:45:29.886 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T00:45:29.886 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T00:45:29.886 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T00:45:29.886 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T00:45:29.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T00:45:29.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T00:45:29.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T00:45:29.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T00:45:29.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T00:45:29.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T00:45:29.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T00:45:29.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T00:45:29.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T00:45:29.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T00:45:29.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T00:45:29.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T00:45:29.887 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T00:45:29.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T00:45:29.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T00:45:29.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T00:45:29.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T00:45:29.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T00:45:29.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T00:45:29.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T00:45:29.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T00:45:29.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T00:45:29.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T00:45:29.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T00:45:29.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T00:45:29.888 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T00:45:29.889 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T00:45:29.889 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T00:45:29.889 INFO:teuthology.orchestra.run.smithi177.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T00:45:29.889 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T00:45:29.889 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T00:45:29.889 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T00:45:29.889 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T00:45:29.889 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T00:45:29.889 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T00:45:29.889 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.890 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.890 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T00:45:29.890 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T00:45:29.890 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T00:45:29.890 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.890 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T00:45:29.890 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T00:45:29.890 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T00:45:29.890 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T00:45:29.890 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T00:45:29.890 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T00:45:29.890 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T00:45:29.890 INFO:teuthology.orchestra.run.smithi177.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T00:45:29.891 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T00:45:29.891 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T00:45:29.891 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T00:45:29.891 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T00:45:29.891 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T00:45:29.891 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T00:45:29.891 INFO:teuthology.orchestra.run.smithi177.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T00:45:29.891 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.891 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.891 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:29.891 INFO:teuthology.orchestra.run.smithi177.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T00:45:29.891 INFO:teuthology.orchestra.run.smithi177.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T00:45:29.891 INFO:teuthology.orchestra.run.smithi177.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T00:45:29.892 INFO:teuthology.orchestra.run.smithi177.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T00:45:29.892 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T00:45:29.892 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T00:45:29.892 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T00:45:29.892 INFO:teuthology.orchestra.run.smithi177.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T00:45:29.892 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-28T00:45:29.893 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-28T00:45:29.913 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T00:45:29.977 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T00:45:30.357 DEBUG:teuthology.parallel:result is None 2024-03-28T00:45:31.663 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T00:45:31.704 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T00:45:31.750 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T00:45:31.778 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T00:45:31.970 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T00:45:32.077 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T00:45:32.118 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T00:45:32.172 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T00:45:32.226 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T00:45:32.269 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T00:45:32.294 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T00:45:32.326 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T00:45:32.355 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T00:45:32.423 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T00:45:32.490 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T00:45:32.529 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T00:45:33.094 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T00:45:33.137 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T00:45:33.158 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T00:45:34.004 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T00:45:34.074 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T00:45:34.115 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T00:45:34.208 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T00:45:34.244 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T00:45:34.269 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T00:45:34.294 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T00:45:34.321 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T00:45:34.321 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T00:45:34.409 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T00:45:34.438 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T00:45:34.439 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T00:45:34.523 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T00:45:34.550 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T00:45:34.589 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T00:45:34.593 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T00:45:34.593 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T00:45:34.593 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T00:45:34.593 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T00:45:34.593 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T00:45:34.593 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T00:45:34.593 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T00:45:34.594 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T00:45:34.594 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T00:45:34.594 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T00:45:34.594 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T00:45:34.594 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T00:45:34.594 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T00:45:34.594 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T00:45:34.594 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T00:45:34.594 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T00:45:34.594 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T00:45:34.594 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T00:45:34.594 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T00:45:34.594 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T00:45:34.594 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T00:45:34.594 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T00:45:34.595 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T00:45:34.595 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T00:45:34.595 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T00:45:34.595 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T00:45:34.595 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T00:45:34.595 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T00:45:34.595 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T00:45:34.595 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T00:45:34.595 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T00:45:34.595 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T00:45:34.595 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T00:45:34.595 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T00:45:34.595 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T00:45:34.595 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T00:45:34.596 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T00:45:34.596 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T00:45:34.596 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T00:45:34.596 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T00:45:34.596 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T00:45:34.596 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T00:45:34.596 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T00:45:34.596 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T00:45:34.596 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T00:45:34.596 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T00:45:34.596 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T00:45:34.596 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T00:45:34.596 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T00:45:34.596 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T00:45:34.596 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T00:45:34.597 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T00:45:34.597 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T00:45:34.597 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T00:45:34.597 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T00:45:34.597 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T00:45:34.597 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T00:45:34.597 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T00:45:34.597 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T00:45:34.597 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T00:45:34.597 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T00:45:34.597 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T00:45:34.597 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T00:45:34.597 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T00:45:34.597 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T00:45:34.597 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T00:45:34.598 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T00:45:34.598 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T00:45:34.598 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T00:45:34.598 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T00:45:34.598 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T00:45:34.598 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T00:45:34.598 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T00:45:34.598 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T00:45:34.598 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T00:45:34.598 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T00:45:34.598 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T00:45:34.598 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T00:45:34.598 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T00:45:34.598 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T00:45:34.598 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T00:45:34.599 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T00:45:34.599 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T00:45:34.599 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T00:45:34.599 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T00:45:34.599 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T00:45:34.599 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T00:45:34.599 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T00:45:34.599 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T00:45:34.599 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T00:45:34.599 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T00:45:34.599 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T00:45:34.599 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T00:45:34.599 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T00:45:34.599 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T00:45:34.599 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T00:45:34.600 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T00:45:34.600 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T00:45:34.600 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T00:45:34.600 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T00:45:34.600 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T00:45:34.600 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T00:45:34.600 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T00:45:34.600 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T00:45:34.600 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T00:45:34.600 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T00:45:34.600 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T00:45:34.600 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T00:45:34.600 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T00:45:34.601 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T00:45:34.601 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T00:45:34.601 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T00:45:34.601 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T00:45:34.601 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T00:45:34.601 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T00:45:34.601 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T00:45:34.601 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T00:45:34.601 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T00:45:34.601 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T00:45:34.601 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T00:45:34.601 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T00:45:34.601 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T00:45:34.601 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T00:45:34.601 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T00:45:34.602 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T00:45:34.602 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T00:45:34.602 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T00:45:34.602 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T00:45:34.602 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T00:45:34.602 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T00:45:34.602 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T00:45:34.602 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T00:45:34.602 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T00:45:34.602 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T00:45:34.602 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T00:45:34.602 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T00:45:34.602 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T00:45:34.602 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T00:45:34.603 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T00:45:34.603 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T00:45:34.603 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T00:45:34.603 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T00:45:34.603 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T00:45:34.603 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T00:45:34.603 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T00:45:34.603 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T00:45:34.603 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T00:45:34.603 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T00:45:34.603 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T00:45:34.603 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T00:45:34.603 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T00:45:34.603 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T00:45:34.604 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T00:45:34.604 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T00:45:34.604 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T00:45:34.604 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T00:45:34.604 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T00:45:34.604 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T00:45:34.604 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T00:45:34.604 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T00:45:34.604 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T00:45:34.604 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T00:45:34.604 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T00:45:34.604 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T00:45:34.604 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T00:45:34.604 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T00:45:34.605 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T00:45:34.605 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T00:45:34.605 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T00:45:34.605 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T00:45:34.605 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T00:45:34.605 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T00:45:34.605 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T00:45:34.605 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T00:45:34.605 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T00:45:34.605 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T00:45:34.605 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T00:45:34.605 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T00:45:34.605 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T00:45:34.605 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T00:45:34.605 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T00:45:34.606 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T00:45:34.606 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T00:45:34.606 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T00:45:34.606 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T00:45:34.606 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T00:45:34.606 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T00:45:34.606 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T00:45:34.606 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T00:45:34.606 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T00:45:34.606 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T00:45:34.606 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T00:45:34.606 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T00:45:34.606 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T00:45:34.607 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T00:45:34.607 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T00:45:34.607 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T00:45:34.607 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T00:45:34.616 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T00:45:34.641 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T00:45:34.678 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T00:45:34.718 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T00:45:34.746 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T00:45:34.771 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T00:45:34.797 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T00:45:34.822 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T00:45:34.847 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T00:45:34.855 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T00:45:34.855 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T00:45:35.033 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T00:45:35.033 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T00:45:35.116 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T00:45:35.141 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T00:45:35.173 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T00:45:35.192 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T00:45:35.226 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T00:45:35.291 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T00:45:37.595 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T00:45:37.595 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-28T00:45:37.595 INFO:teuthology.orchestra.run.smithi174.stdout:Upgraded: 2024-03-28T00:45:37.595 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T00:45:37.595 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T00:45:37.595 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T00:45:37.595 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T00:45:37.595 INFO:teuthology.orchestra.run.smithi174.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T00:45:37.595 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T00:45:37.595 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T00:45:37.595 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T00:45:37.596 INFO:teuthology.orchestra.run.smithi174.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T00:45:37.596 INFO:teuthology.orchestra.run.smithi174.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T00:45:37.596 INFO:teuthology.orchestra.run.smithi174.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T00:45:37.596 INFO:teuthology.orchestra.run.smithi174.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T00:45:37.596 INFO:teuthology.orchestra.run.smithi174.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T00:45:37.596 INFO:teuthology.orchestra.run.smithi174.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.596 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.596 INFO:teuthology.orchestra.run.smithi174.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T00:45:37.596 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T00:45:37.596 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T00:45:37.596 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T00:45:37.596 INFO:teuthology.orchestra.run.smithi174.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T00:45:37.596 INFO:teuthology.orchestra.run.smithi174.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T00:45:37.597 INFO:teuthology.orchestra.run.smithi174.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T00:45:37.597 INFO:teuthology.orchestra.run.smithi174.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T00:45:37.597 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T00:45:37.597 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-28T00:45:37.597 INFO:teuthology.orchestra.run.smithi174.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T00:45:37.597 INFO:teuthology.orchestra.run.smithi174.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T00:45:37.597 INFO:teuthology.orchestra.run.smithi174.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T00:45:37.597 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T00:45:37.597 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.597 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.597 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.597 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.597 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:37.597 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.597 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.598 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.598 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:37.598 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:37.598 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:37.598 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:37.598 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:37.598 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.598 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.598 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:37.598 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.598 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.598 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.598 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:37.598 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:37.599 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.599 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.599 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:37.599 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T00:45:37.599 INFO:teuthology.orchestra.run.smithi174.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T00:45:37.599 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T00:45:37.599 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T00:45:37.599 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T00:45:37.599 INFO:teuthology.orchestra.run.smithi174.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T00:45:37.599 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T00:45:37.599 INFO:teuthology.orchestra.run.smithi174.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T00:45:37.599 INFO:teuthology.orchestra.run.smithi174.stdout: jq-1.6-16.el9.x86_64 2024-03-28T00:45:37.599 INFO:teuthology.orchestra.run.smithi174.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T00:45:37.600 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T00:45:37.600 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T00:45:37.600 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.600 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.600 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.600 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T00:45:37.600 INFO:teuthology.orchestra.run.smithi174.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T00:45:37.600 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T00:45:37.600 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T00:45:37.600 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T00:45:37.600 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.600 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.600 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T00:45:37.600 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.600 INFO:teuthology.orchestra.run.smithi174.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T00:45:37.601 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T00:45:37.601 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T00:45:37.601 INFO:teuthology.orchestra.run.smithi174.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T00:45:37.601 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T00:45:37.601 INFO:teuthology.orchestra.run.smithi174.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T00:45:37.601 INFO:teuthology.orchestra.run.smithi174.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T00:45:37.601 INFO:teuthology.orchestra.run.smithi174.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T00:45:37.601 INFO:teuthology.orchestra.run.smithi174.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T00:45:37.601 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T00:45:37.601 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T00:45:37.601 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T00:45:37.601 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T00:45:37.601 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T00:45:37.601 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T00:45:37.601 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T00:45:37.602 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T00:45:37.602 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T00:45:37.602 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T00:45:37.602 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T00:45:37.602 INFO:teuthology.orchestra.run.smithi174.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T00:45:37.602 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T00:45:37.602 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T00:45:37.602 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T00:45:37.602 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.602 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.602 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.602 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T00:45:37.602 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T00:45:37.602 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T00:45:37.603 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T00:45:37.603 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T00:45:37.603 INFO:teuthology.orchestra.run.smithi174.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T00:45:37.603 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T00:45:37.603 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T00:45:37.603 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T00:45:37.603 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T00:45:37.603 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T00:45:37.603 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T00:45:37.603 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T00:45:37.603 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T00:45:37.603 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T00:45:37.603 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T00:45:37.603 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T00:45:37.603 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T00:45:37.603 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T00:45:37.604 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T00:45:37.604 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T00:45:37.604 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T00:45:37.604 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T00:45:37.604 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T00:45:37.604 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T00:45:37.604 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T00:45:37.604 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T00:45:37.604 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T00:45:37.604 INFO:teuthology.orchestra.run.smithi174.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T00:45:37.604 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T00:45:37.604 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T00:45:37.604 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T00:45:37.604 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T00:45:37.604 INFO:teuthology.orchestra.run.smithi174.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T00:45:37.605 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T00:45:37.605 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T00:45:37.605 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T00:45:37.605 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T00:45:37.605 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T00:45:37.605 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T00:45:37.605 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.605 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.605 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T00:45:37.605 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T00:45:37.605 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T00:45:37.605 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.605 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T00:45:37.605 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T00:45:37.605 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T00:45:37.605 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T00:45:37.605 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T00:45:37.606 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T00:45:37.606 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T00:45:37.606 INFO:teuthology.orchestra.run.smithi174.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T00:45:37.606 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T00:45:37.606 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T00:45:37.606 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T00:45:37.606 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T00:45:37.606 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T00:45:37.606 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T00:45:37.606 INFO:teuthology.orchestra.run.smithi174.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T00:45:37.607 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.607 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.607 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:37.607 INFO:teuthology.orchestra.run.smithi174.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T00:45:37.607 INFO:teuthology.orchestra.run.smithi174.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T00:45:37.607 INFO:teuthology.orchestra.run.smithi174.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T00:45:37.607 INFO:teuthology.orchestra.run.smithi174.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T00:45:37.607 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T00:45:37.607 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T00:45:37.607 INFO:teuthology.orchestra.run.smithi174.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T00:45:37.608 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-28T00:45:37.608 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-28T00:45:38.092 DEBUG:teuthology.parallel:result is None 2024-03-28T00:45:40.289 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T00:45:40.289 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T00:45:40.290 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T00:45:40.290 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T00:45:40.290 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T00:45:40.290 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T00:45:40.290 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T00:45:40.290 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T00:45:40.290 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T00:45:40.290 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T00:45:40.290 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T00:45:40.290 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T00:45:40.290 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T00:45:40.290 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T00:45:40.290 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T00:45:40.291 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T00:45:40.291 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T00:45:40.291 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T00:45:40.291 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T00:45:40.291 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T00:45:40.291 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T00:45:40.291 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T00:45:40.291 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T00:45:40.291 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T00:45:40.291 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T00:45:40.291 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T00:45:40.291 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T00:45:40.291 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T00:45:40.291 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T00:45:40.292 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T00:45:40.307 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T00:45:40.307 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T00:45:40.307 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T00:45:40.308 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T00:45:40.308 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T00:45:40.308 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T00:45:40.308 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T00:45:40.308 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T00:45:40.308 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T00:45:40.308 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T00:45:40.308 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T00:45:40.308 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T00:45:40.308 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T00:45:40.308 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T00:45:40.308 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T00:45:40.308 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T00:45:40.309 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T00:45:40.309 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T00:45:40.309 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T00:45:40.309 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T00:45:40.309 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T00:45:40.309 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T00:45:40.309 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T00:45:40.309 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T00:45:40.309 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T00:45:40.309 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T00:45:40.309 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T00:45:40.309 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T00:45:40.309 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T00:45:40.309 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T00:45:40.309 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T00:45:40.310 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T00:45:40.310 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T00:45:40.310 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T00:45:40.310 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T00:45:40.310 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T00:45:40.310 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T00:45:40.310 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T00:45:40.310 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T00:45:40.310 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T00:45:40.310 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T00:45:40.310 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T00:45:40.310 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T00:45:40.310 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T00:45:40.310 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T00:45:40.311 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T00:45:40.311 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T00:45:40.311 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T00:45:40.311 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T00:45:40.311 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T00:45:40.311 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T00:45:40.311 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T00:45:40.311 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T00:45:40.311 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T00:45:40.311 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T00:45:40.311 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T00:45:40.311 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T00:45:40.311 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T00:45:40.311 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T00:45:40.311 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T00:45:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T00:45:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T00:45:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T00:45:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T00:45:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T00:45:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T00:45:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T00:45:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T00:45:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T00:45:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T00:45:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T00:45:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T00:45:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T00:45:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T00:45:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T00:45:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T00:45:40.313 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T00:45:40.313 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T00:45:40.313 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T00:45:40.313 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T00:45:40.313 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T00:45:40.313 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T00:45:40.313 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T00:45:40.313 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T00:45:40.313 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T00:45:40.313 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T00:45:40.313 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T00:45:40.313 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T00:45:40.313 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T00:45:40.313 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T00:45:40.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T00:45:40.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T00:45:40.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T00:45:40.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T00:45:40.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T00:45:40.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T00:45:40.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T00:45:40.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T00:45:40.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T00:45:40.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T00:45:40.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T00:45:40.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T00:45:40.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T00:45:40.314 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T00:45:40.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T00:45:40.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T00:45:40.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T00:45:40.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T00:45:40.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T00:45:40.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T00:45:40.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T00:45:40.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T00:45:40.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T00:45:40.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T00:45:40.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T00:45:40.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T00:45:40.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T00:45:40.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T00:45:40.315 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T00:45:40.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T00:45:40.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T00:45:40.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T00:45:40.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T00:45:40.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T00:45:40.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T00:45:40.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T00:45:40.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T00:45:40.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T00:45:40.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T00:45:40.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T00:45:40.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T00:45:40.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T00:45:40.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T00:45:40.316 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T00:45:40.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T00:45:40.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T00:45:40.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T00:45:40.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T00:45:40.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T00:45:40.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T00:45:40.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T00:45:40.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T00:45:40.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T00:45:40.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T00:45:40.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T00:45:40.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T00:45:40.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T00:45:40.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T00:45:40.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T00:45:40.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T00:45:40.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T00:45:40.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T00:45:40.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T00:45:40.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T00:45:40.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T00:45:40.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T00:45:40.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T00:45:40.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T00:45:40.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T00:45:40.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T00:45:40.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T00:45:40.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T00:45:40.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T00:45:40.319 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T00:45:40.319 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T00:45:40.319 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T00:45:40.319 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T00:45:40.319 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T00:45:43.387 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T00:45:43.387 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:45:43.388 INFO:teuthology.orchestra.run.smithi169.stdout:Upgraded: 2024-03-28T00:45:43.388 INFO:teuthology.orchestra.run.smithi169.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T00:45:43.388 INFO:teuthology.orchestra.run.smithi169.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T00:45:43.388 INFO:teuthology.orchestra.run.smithi169.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T00:45:43.388 INFO:teuthology.orchestra.run.smithi169.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T00:45:43.388 INFO:teuthology.orchestra.run.smithi169.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T00:45:43.388 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T00:45:43.388 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T00:45:43.388 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T00:45:43.388 INFO:teuthology.orchestra.run.smithi169.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T00:45:43.388 INFO:teuthology.orchestra.run.smithi169.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T00:45:43.388 INFO:teuthology.orchestra.run.smithi169.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T00:45:43.388 INFO:teuthology.orchestra.run.smithi169.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T00:45:43.388 INFO:teuthology.orchestra.run.smithi169.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T00:45:43.389 INFO:teuthology.orchestra.run.smithi169.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.389 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.389 INFO:teuthology.orchestra.run.smithi169.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T00:45:43.389 INFO:teuthology.orchestra.run.smithi169.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T00:45:43.389 INFO:teuthology.orchestra.run.smithi169.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T00:45:43.389 INFO:teuthology.orchestra.run.smithi169.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T00:45:43.389 INFO:teuthology.orchestra.run.smithi169.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T00:45:43.389 INFO:teuthology.orchestra.run.smithi169.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T00:45:43.389 INFO:teuthology.orchestra.run.smithi169.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T00:45:43.389 INFO:teuthology.orchestra.run.smithi169.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T00:45:43.389 INFO:teuthology.orchestra.run.smithi169.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T00:45:43.389 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-28T00:45:43.389 INFO:teuthology.orchestra.run.smithi169.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T00:45:43.389 INFO:teuthology.orchestra.run.smithi169.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T00:45:43.390 INFO:teuthology.orchestra.run.smithi169.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T00:45:43.390 INFO:teuthology.orchestra.run.smithi169.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T00:45:43.390 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.390 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.390 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.390 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.390 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:43.390 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.390 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.390 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.390 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:43.390 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:43.390 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:43.391 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:43.391 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:43.391 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.391 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.391 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:43.391 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.391 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.391 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.391 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:43.391 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:43.391 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.391 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.391 INFO:teuthology.orchestra.run.smithi169.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T00:45:43.391 INFO:teuthology.orchestra.run.smithi169.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T00:45:43.392 INFO:teuthology.orchestra.run.smithi169.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T00:45:43.392 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T00:45:43.392 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T00:45:43.392 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T00:45:43.392 INFO:teuthology.orchestra.run.smithi169.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T00:45:43.392 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T00:45:43.392 INFO:teuthology.orchestra.run.smithi169.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T00:45:43.392 INFO:teuthology.orchestra.run.smithi169.stdout: jq-1.6-16.el9.x86_64 2024-03-28T00:45:43.392 INFO:teuthology.orchestra.run.smithi169.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T00:45:43.392 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T00:45:43.392 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T00:45:43.392 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.392 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.392 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.392 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T00:45:43.393 INFO:teuthology.orchestra.run.smithi169.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T00:45:43.393 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T00:45:43.393 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T00:45:43.393 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T00:45:43.393 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.393 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.393 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T00:45:43.393 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.393 INFO:teuthology.orchestra.run.smithi169.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T00:45:43.393 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T00:45:43.393 INFO:teuthology.orchestra.run.smithi169.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T00:45:43.393 INFO:teuthology.orchestra.run.smithi169.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T00:45:43.393 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T00:45:43.393 INFO:teuthology.orchestra.run.smithi169.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T00:45:43.393 INFO:teuthology.orchestra.run.smithi169.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T00:45:43.394 INFO:teuthology.orchestra.run.smithi169.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T00:45:43.394 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T00:45:43.394 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T00:45:43.394 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T00:45:43.394 INFO:teuthology.orchestra.run.smithi169.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T00:45:43.394 INFO:teuthology.orchestra.run.smithi169.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T00:45:43.394 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T00:45:43.394 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T00:45:43.394 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T00:45:43.394 INFO:teuthology.orchestra.run.smithi169.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T00:45:43.394 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T00:45:43.394 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T00:45:43.394 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T00:45:43.394 INFO:teuthology.orchestra.run.smithi169.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T00:45:43.394 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T00:45:43.395 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T00:45:43.395 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T00:45:43.395 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.395 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.395 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.395 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T00:45:43.395 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T00:45:43.395 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T00:45:43.395 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T00:45:43.395 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T00:45:43.395 INFO:teuthology.orchestra.run.smithi169.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T00:45:43.395 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T00:45:43.395 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T00:45:43.396 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T00:45:43.396 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T00:45:43.396 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T00:45:43.396 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T00:45:43.396 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T00:45:43.396 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T00:45:43.396 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T00:45:43.396 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T00:45:43.396 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T00:45:43.396 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T00:45:43.396 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T00:45:43.396 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T00:45:43.396 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T00:45:43.396 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T00:45:43.396 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T00:45:43.397 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T00:45:43.397 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T00:45:43.397 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T00:45:43.397 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T00:45:43.397 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T00:45:43.397 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T00:45:43.397 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T00:45:43.397 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T00:45:43.397 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T00:45:43.397 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T00:45:43.397 INFO:teuthology.orchestra.run.smithi169.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T00:45:43.397 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T00:45:43.397 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T00:45:43.397 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T00:45:43.397 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T00:45:43.398 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T00:45:43.398 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T00:45:43.398 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.398 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.398 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T00:45:43.398 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T00:45:43.398 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T00:45:43.398 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.398 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T00:45:43.398 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T00:45:43.398 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T00:45:43.398 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T00:45:43.398 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T00:45:43.398 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T00:45:43.399 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T00:45:43.399 INFO:teuthology.orchestra.run.smithi169.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T00:45:43.399 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T00:45:43.399 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T00:45:43.399 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T00:45:43.399 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T00:45:43.399 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T00:45:43.399 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T00:45:43.399 INFO:teuthology.orchestra.run.smithi169.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T00:45:43.399 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.399 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.399 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T00:45:43.399 INFO:teuthology.orchestra.run.smithi169.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T00:45:43.399 INFO:teuthology.orchestra.run.smithi169.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T00:45:43.400 INFO:teuthology.orchestra.run.smithi169.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T00:45:43.400 INFO:teuthology.orchestra.run.smithi169.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T00:45:43.400 INFO:teuthology.orchestra.run.smithi169.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T00:45:43.400 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T00:45:43.400 INFO:teuthology.orchestra.run.smithi169.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T00:45:43.400 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:45:43.400 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-28T00:45:43.855 DEBUG:teuthology.parallel:result is None 2024-03-28T00:45:43.856 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T00:45:43.999 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T00:45:44.021 INFO:teuthology.orchestra.run.smithi169.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T00:45:44.021 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T00:45:44.021 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T00:45:44.023 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T00:45:44.248 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T00:45:44.273 INFO:teuthology.orchestra.run.smithi174.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T00:45:44.274 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T00:45:44.274 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T00:45:44.275 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T00:45:44.416 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T00:45:44.442 INFO:teuthology.orchestra.run.smithi177.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T00:45:44.442 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T00:45:44.442 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T00:45:44.444 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-28T00:45:44.445 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-28T00:45:44.445 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T00:45:44.474 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-28T00:45:44.474 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T00:45:44.509 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-28T00:45:44.509 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T00:45:44.543 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-28T00:45:44.543 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-28T00:45:44.543 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T00:45:44.571 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T00:45:44.637 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-28T00:45:44.638 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T00:45:44.670 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T00:45:44.742 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-28T00:45:44.742 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T00:45:44.774 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T00:45:44.846 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-28T00:45:44.846 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-28T00:45:44.846 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T00:45:44.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T00:45:44.940 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-28T00:45:44.940 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T00:45:44.971 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T00:45:45.042 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-28T00:45:45.042 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T00:45:45.070 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T00:45:45.137 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-28T00:45:45.137 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-28T00:45:45.137 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T00:45:45.164 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T00:45:45.230 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-28T00:45:45.230 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T00:45:45.262 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T00:45:45.331 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-28T00:45:45.332 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T00:45:45.359 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T00:45:45.427 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-28T00:45:45.529 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-28T00:45:45.529 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T00:45:45.530 INFO:tasks.cephadm:Cluster fsid is 833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T00:45:45.530 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-28T00:45:45.530 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.169', 'mon.b': '172.21.15.174', 'mon.c': '172.21.15.177'} 2024-03-28T00:45:45.530 INFO:tasks.cephadm:First mon is mon.a on smithi169 2024-03-28T00:45:45.530 INFO:tasks.cephadm:First mgr is x 2024-03-28T00:45:45.530 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-28T00:45:45.531 DEBUG:teuthology.orchestra.run.smithi169:> sudo hostname $(hostname -s) 2024-03-28T00:45:45.558 DEBUG:teuthology.orchestra.run.smithi174:> sudo hostname $(hostname -s) 2024-03-28T00:45:45.587 DEBUG:teuthology.orchestra.run.smithi177:> sudo hostname $(hostname -s) 2024-03-28T00:45:45.614 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-28T00:45:45.614 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T00:45:45.755 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77877/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2145-g3be9a5c9', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2145.g3be9a5c9'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-22 05:51:42.267384', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-mchangir-testing-main-20240318.032620', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-28T00:45:45.895 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-mchangir-testing-main-20240318.032620, sha1 3be9a5c9bc793e11e2800a8c0c696e8b46742033 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T00:45:45.897 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm 2024-03-28T00:45:45.897 DEBUG:teuthology.orchestra.run.smithi169:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T00:45:46.504 INFO:teuthology.orchestra.run.smithi169.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 00:45 /home/ubuntu/cephtest/cephadm 2024-03-28T00:45:46.504 DEBUG:teuthology.orchestra.run.smithi174:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T00:45:47.154 INFO:teuthology.orchestra.run.smithi174.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 00:45 /home/ubuntu/cephtest/cephadm 2024-03-28T00:45:47.155 DEBUG:teuthology.orchestra.run.smithi177:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T00:45:47.713 INFO:teuthology.orchestra.run.smithi177.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 00:45 /home/ubuntu/cephtest/cephadm 2024-03-28T00:45:47.714 DEBUG:teuthology.orchestra.run.smithi169:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T00:45:47.733 DEBUG:teuthology.orchestra.run.smithi174:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T00:45:47.751 DEBUG:teuthology.orchestra.run.smithi177:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T00:45:47.785 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 on all hosts... 2024-03-28T00:45:47.785 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T00:45:47.788 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T00:45:47.794 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T00:45:48.607 INFO:teuthology.orchestra.run.smithi174.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T00:45:48.654 INFO:teuthology.orchestra.run.smithi177.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T00:45:48.744 INFO:teuthology.orchestra.run.smithi169.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T00:46:15.222 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:46:15.223 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T00:46:15.223 INFO:teuthology.orchestra.run.smithi169.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T00:46:15.223 INFO:teuthology.orchestra.run.smithi169.stdout: "repo_digests": [ 2024-03-28T00:46:15.223 INFO:teuthology.orchestra.run.smithi169.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T00:46:15.223 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-28T00:46:15.223 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:46:18.252 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-28T00:46:18.252 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T00:46:18.252 INFO:teuthology.orchestra.run.smithi174.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T00:46:18.252 INFO:teuthology.orchestra.run.smithi174.stdout: "repo_digests": [ 2024-03-28T00:46:18.252 INFO:teuthology.orchestra.run.smithi174.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T00:46:18.252 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-28T00:46:18.252 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-28T00:46:22.219 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-03-28T00:46:22.219 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T00:46:22.219 INFO:teuthology.orchestra.run.smithi177.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T00:46:22.219 INFO:teuthology.orchestra.run.smithi177.stdout: "repo_digests": [ 2024-03-28T00:46:22.219 INFO:teuthology.orchestra.run.smithi177.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T00:46:22.219 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-03-28T00:46:22.219 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-03-28T00:46:22.244 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph 2024-03-28T00:46:22.280 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /etc/ceph 2024-03-28T00:46:22.315 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph 2024-03-28T00:46:22.345 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 777 /etc/ceph 2024-03-28T00:46:22.377 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 777 /etc/ceph 2024-03-28T00:46:22.407 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /etc/ceph 2024-03-28T00:46:22.433 INFO:tasks.cephadm:Writing seed config... 2024-03-28T00:46:22.435 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-28T00:46:22.435 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-28T00:46:22.435 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-28T00:46:22.435 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-28T00:46:22.435 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-28T00:46:22.435 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-28T00:46:22.435 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-28T00:46:22.435 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-28T00:46:22.435 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-28T00:46:22.435 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-28T00:46:22.435 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-28T00:46:22.435 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-28T00:46:22.435 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-28T00:46:22.435 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-28T00:46:22.436 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-28T00:46:22.436 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-28T00:46:22.436 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-28T00:46:22.436 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-28T00:46:22.436 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-28T00:46:22.436 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-28T00:46:22.436 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-28T00:46:22.436 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-28T00:46:22.436 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-28T00:46:22.436 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-28T00:46:22.436 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-28T00:46:22.436 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-28T00:46:22.436 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-28T00:46:22.436 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-28T00:46:22.436 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-28T00:46:22.436 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-28T00:46:22.437 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-28T00:46:22.437 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-28T00:46:22.437 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-28T00:46:22.437 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-28T00:46:22.437 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-28T00:46:22.437 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-28T00:46:22.437 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-28T00:46:22.437 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-28T00:46:22.437 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-28T00:46:22.437 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-28T00:46:22.437 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-28T00:46:22.437 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-28T00:46:22.437 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-28T00:46:22.437 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-28T00:46:22.437 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-28T00:46:22.438 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-28T00:46:22.438 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-28T00:46:22.456 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 833368da-ec9c-11ee-95d0-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-28T00:46:22.457 DEBUG:teuthology.orchestra.run.smithi169:mon.a> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mon.a.service 2024-03-28T00:46:22.500 DEBUG:teuthology.orchestra.run.smithi169:mgr.x> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mgr.x.service 2024-03-28T00:46:22.542 INFO:tasks.cephadm:Bootstrapping... 2024-03-28T00:46:22.543 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 -v bootstrap --fsid 833368da-ec9c-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.169 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-28T00:46:22.738 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-28T00:46:22.738 INFO:teuthology.orchestra.run.smithi169.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033', '-v', 'bootstrap', '--fsid', '833368da-ec9c-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.169', '--skip-admin-label'] 2024-03-28T00:46:22.763 INFO:teuthology.orchestra.run.smithi169.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T00:46:22.764 INFO:teuthology.orchestra.run.smithi169.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-28T00:46:22.764 INFO:teuthology.orchestra.run.smithi169.stdout:Verifying podman|docker is present... 2024-03-28T00:46:22.786 INFO:teuthology.orchestra.run.smithi169.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T00:46:22.786 INFO:teuthology.orchestra.run.smithi169.stdout:Verifying lvm2 is present... 2024-03-28T00:46:22.786 INFO:teuthology.orchestra.run.smithi169.stdout:Verifying time synchronization is in place... 2024-03-28T00:46:22.794 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T00:46:22.794 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T00:46:22.800 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T00:46:22.800 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout inactive 2024-03-28T00:46:22.806 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout enabled 2024-03-28T00:46:22.812 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout active 2024-03-28T00:46:22.812 INFO:teuthology.orchestra.run.smithi169.stdout:Unit chronyd.service is enabled and running 2024-03-28T00:46:22.812 INFO:teuthology.orchestra.run.smithi169.stdout:Repeating the final host check... 2024-03-28T00:46:22.835 INFO:teuthology.orchestra.run.smithi169.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T00:46:22.835 INFO:teuthology.orchestra.run.smithi169.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-28T00:46:22.835 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl is present 2024-03-28T00:46:22.835 INFO:teuthology.orchestra.run.smithi169.stdout:lvcreate is present 2024-03-28T00:46:22.841 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T00:46:22.842 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T00:46:22.847 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T00:46:22.848 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout inactive 2024-03-28T00:46:22.853 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout enabled 2024-03-28T00:46:22.860 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout active 2024-03-28T00:46:22.860 INFO:teuthology.orchestra.run.smithi169.stdout:Unit chronyd.service is enabled and running 2024-03-28T00:46:22.860 INFO:teuthology.orchestra.run.smithi169.stdout:Host looks OK 2024-03-28T00:46:22.860 INFO:teuthology.orchestra.run.smithi169.stdout:Cluster fsid: 833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T00:46:22.860 INFO:teuthology.orchestra.run.smithi169.stdout:Acquiring lock 140026299747776 on /run/cephadm/833368da-ec9c-11ee-95d0-87774f69a715.lock 2024-03-28T00:46:22.860 INFO:teuthology.orchestra.run.smithi169.stdout:Lock 140026299747776 acquired on /run/cephadm/833368da-ec9c-11ee-95d0-87774f69a715.lock 2024-03-28T00:46:22.860 INFO:teuthology.orchestra.run.smithi169.stdout:Verifying IP 172.21.15.169 port 3300 ... 2024-03-28T00:46:22.861 INFO:teuthology.orchestra.run.smithi169.stdout:Verifying IP 172.21.15.169 port 6789 ... 2024-03-28T00:46:22.861 INFO:teuthology.orchestra.run.smithi169.stdout:Base mon IP(s) is [172.21.15.169:3300, 172.21.15.169:6789], mon addrv is [v2:172.21.15.169:3300,v1:172.21.15.169:6789] 2024-03-28T00:46:22.864 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.169 metric 100 2024-03-28T00:46:22.864 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.169 metric 100 2024-03-28T00:46:22.866 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-28T00:46:22.866 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-28T00:46:22.866 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1782sec hoplimit 64 pref medium 2024-03-28T00:46:22.869 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-28T00:46:22.869 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-28T00:46:22.869 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T00:46:22.869 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-28T00:46:22.869 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:7f4d/64 scope link 2024-03-28T00:46:22.869 INFO:teuthology.orchestra.run.smithi169.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T00:46:22.870 INFO:teuthology.orchestra.run.smithi169.stdout:Mon IP `172.21.15.169` is in CIDR network `172.21.0.0/20` 2024-03-28T00:46:22.870 INFO:teuthology.orchestra.run.smithi169.stdout:Mon IP `172.21.15.169` is in CIDR network `172.21.0.0/20` 2024-03-28T00:46:22.870 INFO:teuthology.orchestra.run.smithi169.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-28T00:46:22.870 INFO:teuthology.orchestra.run.smithi169.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-28T00:46:22.871 INFO:teuthology.orchestra.run.smithi169.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T00:46:23.332 INFO:teuthology.orchestra.run.smithi169.stdout:/bin/podman: stdout c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-28T00:46:23.332 INFO:teuthology.orchestra.run.smithi169.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T00:46:23.332 INFO:teuthology.orchestra.run.smithi169.stdout:/bin/podman: stderr Getting image source signatures 2024-03-28T00:46:23.332 INFO:teuthology.orchestra.run.smithi169.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-28T00:46:23.332 INFO:teuthology.orchestra.run.smithi169.stdout:/bin/podman: stderr Copying blob sha256:4dd83c6ee8921fe17396d77c31e205985ae94c87d3f752465b5f04aaf9ae340a 2024-03-28T00:46:23.332 INFO:teuthology.orchestra.run.smithi169.stdout:/bin/podman: stderr Copying config sha256:c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-28T00:46:23.332 INFO:teuthology.orchestra.run.smithi169.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-28T00:46:24.262 INFO:teuthology.orchestra.run.smithi169.stdout:ceph: stdout ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T00:46:24.262 INFO:teuthology.orchestra.run.smithi169.stdout:Ceph version: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T00:46:24.262 INFO:teuthology.orchestra.run.smithi169.stdout:Extracting ceph user uid/gid from container image... 2024-03-28T00:46:25.134 INFO:teuthology.orchestra.run.smithi169.stdout:stat: stdout 167 167 2024-03-28T00:46:25.134 INFO:teuthology.orchestra.run.smithi169.stdout:Creating initial keys... 2024-03-28T00:46:31.145 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph-authtool: stdout AQDhvQRmWp6yGRAAjl0uYij2qoz9Kr2lW8Gfcg== 2024-03-28T00:46:36.528 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph-authtool: stdout AQDqvQRmN1czAxAAfMfLL2qLLcpVSZo3fzOBOQ== 2024-03-28T00:46:37.575 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph-authtool: stdout AQDsvQRmvs8wMxAAF1eGaHP1ckPwx/5K+yUvdg== 2024-03-28T00:46:37.576 INFO:teuthology.orchestra.run.smithi169.stdout:Creating initial monmap... 2024-03-28T00:46:39.075 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T00:46:39.075 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-28T00:46:39.075 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T00:46:39.075 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T00:46:39.075 INFO:teuthology.orchestra.run.smithi169.stdout:monmaptool for a [v2:172.21.15.169:3300,v1:172.21.15.169:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T00:46:39.075 INFO:teuthology.orchestra.run.smithi169.stdout:setting min_mon_release = quincy 2024-03-28T00:46:39.075 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/monmaptool: set fsid to 833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T00:46:39.075 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T00:46:39.076 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:46:39.076 INFO:teuthology.orchestra.run.smithi169.stdout:Creating mon... 2024-03-28T00:46:40.056 INFO:teuthology.orchestra.run.smithi169.stdout:create mon.a on 2024-03-28T00:46:40.288 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-28T00:46:40.428 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-28T00:46:40.620 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-833368da-ec9c-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-833368da-ec9c-11ee-95d0-87774f69a715.target. 2024-03-28T00:46:40.620 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-833368da-ec9c-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-833368da-ec9c-11ee-95d0-87774f69a715.target. 2024-03-28T00:46:40.863 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-833368da-ec9c-11ee-95d0-87774f69a715@mon.a 2024-03-28T00:46:40.863 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stderr Failed to reset failed state of unit ceph-833368da-ec9c-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-833368da-ec9c-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-28T00:46:41.029 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-833368da-ec9c-11ee-95d0-87774f69a715.target.wants/ceph-833368da-ec9c-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-833368da-ec9c-11ee-95d0-87774f69a715@.service. 2024-03-28T00:46:42.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:42 smithi169 ceph-mon[31897]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T00:46:43.283 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T00:46:43.283 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout disabled 2024-03-28T00:46:43.291 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T00:46:43.291 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout inactive 2024-03-28T00:46:43.291 INFO:teuthology.orchestra.run.smithi169.stdout:firewalld.service is not enabled 2024-03-28T00:46:43.291 INFO:teuthology.orchestra.run.smithi169.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T00:46:43.292 INFO:teuthology.orchestra.run.smithi169.stdout:Waiting for mon to start... 2024-03-28T00:46:43.292 INFO:teuthology.orchestra.run.smithi169.stdout:Waiting for mon... 2024-03-28T00:46:43.537 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:43 smithi169 bash[31878]: f92ae489f9b0dcc445b66163024f19f979a728adc25e7492f6c696dbc7bcfc93 2024-03-28T00:46:43.537 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:43 smithi169 systemd[1]: Started Ceph mon.a for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T00:46:43.538 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:43 smithi169 ceph-mon[31897]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T00:46:43.538 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:43 smithi169 ceph-mon[31897]: monmap e1: 1 mons at {a=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T00:46:43.538 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:43 smithi169 ceph-mon[31897]: fsmap 2024-03-28T00:46:43.538 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:43 smithi169 ceph-mon[31897]: osdmap e1: 0 total, 0 up, 0 in 2024-03-28T00:46:43.538 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:43 smithi169 ceph-mon[31897]: mgrmap e1: no daemons active 2024-03-28T00:46:44.483 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout cluster: 2024-03-28T00:46:44.483 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout id: 833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T00:46:44.484 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-28T00:46:44.484 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-28T00:46:44.484 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout services: 2024-03-28T00:46:44.484 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 1.39744s) 2024-03-28T00:46:44.484 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-28T00:46:44.484 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-28T00:46:44.484 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-28T00:46:44.484 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout data: 2024-03-28T00:46:44.484 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-28T00:46:44.484 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-28T00:46:44.484 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-28T00:46:44.484 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout pgs: 2024-03-28T00:46:44.485 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-28T00:46:44.485 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.750+0000 7f0526440700 1 Processor -- start 2024-03-28T00:46:44.485 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.751+0000 7f0526440700 1 -- start start 2024-03-28T00:46:44.485 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.751+0000 7f0526440700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0520108620 0x7f052010aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:46:44.485 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.751+0000 7f0526440700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052007be70 con 0x7f0520108620 2024-03-28T00:46:44.485 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.751+0000 7f051ffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0520108620 0x7f052010aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:46:44.485 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.751+0000 7f051ffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0520108620 0x7f052010aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:44118/0 (socket says 172.21.15.169:44118) 2024-03-28T00:46:44.485 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.751+0000 7f051ffff700 1 -- 172.21.15.169:0/4146843317 learned_addr learned my addr 172.21.15.169:0/4146843317 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:46:44.485 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.751+0000 7f051ffff700 1 -- 172.21.15.169:0/4146843317 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052010af50 con 0x7f0520108620 2024-03-28T00:46:44.485 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.751+0000 7f051ffff700 1 --2- 172.21.15.169:0/4146843317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0520108620 0x7f052010aa10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f05140098a0 tx=0x7f0514009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a420c69d371e75ed server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:46:44.485 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.752+0000 7f051effd700 1 -- 172.21.15.169:0/4146843317 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0514004160 con 0x7f0520108620 2024-03-28T00:46:44.485 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.752+0000 7f051effd700 1 -- 172.21.15.169:0/4146843317 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0514004300 con 0x7f0520108620 2024-03-28T00:46:44.485 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.752+0000 7f051effd700 1 -- 172.21.15.169:0/4146843317 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f05140036a0 con 0x7f0520108620 2024-03-28T00:46:44.485 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.752+0000 7f0526440700 1 -- 172.21.15.169:0/4146843317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0520108620 msgr2=0x7f052010aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:46:44.486 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.752+0000 7f0526440700 1 --2- 172.21.15.169:0/4146843317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0520108620 0x7f052010aa10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f05140098a0 tx=0x7f0514009bb0 comp rx=0 tx=0).stop 2024-03-28T00:46:44.486 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.752+0000 7f0526440700 1 -- 172.21.15.169:0/4146843317 shutdown_connections 2024-03-28T00:46:44.486 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.752+0000 7f0526440700 1 --2- 172.21.15.169:0/4146843317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0520108620 0x7f052010aa10 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:46:44.486 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.752+0000 7f0526440700 1 -- 172.21.15.169:0/4146843317 >> 172.21.15.169:0/4146843317 conn(0x7f0520102030 msgr2=0x7f0520104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:46:44.486 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.752+0000 7f0526440700 1 -- 172.21.15.169:0/4146843317 shutdown_connections 2024-03-28T00:46:44.486 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.752+0000 7f0526440700 1 -- 172.21.15.169:0/4146843317 wait complete. 2024-03-28T00:46:44.486 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.753+0000 7f0526440700 1 Processor -- start 2024-03-28T00:46:44.486 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.753+0000 7f0526440700 1 -- start start 2024-03-28T00:46:44.486 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.753+0000 7f0526440700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0520108620 0x7f052019b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:46:44.486 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.753+0000 7f0526440700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052010b3c0 con 0x7f0520108620 2024-03-28T00:46:44.486 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.753+0000 7f051ffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0520108620 0x7f052019b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:46:44.486 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.753+0000 7f051ffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0520108620 0x7f052019b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:44132/0 (socket says 172.21.15.169:44132) 2024-03-28T00:46:44.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.753+0000 7f051ffff700 1 -- 172.21.15.169:0/1258424254 learned_addr learned my addr 172.21.15.169:0/1258424254 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:46:44.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.754+0000 7f051ffff700 1 -- 172.21.15.169:0/1258424254 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052019b7f0 con 0x7f0520108620 2024-03-28T00:46:44.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.754+0000 7f051ffff700 1 --2- 172.21.15.169:0/1258424254 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0520108620 0x7f052019b2b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0514000c00 tx=0x7f0514004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:46:44.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.754+0000 7f051d7fa700 1 -- 172.21.15.169:0/1258424254 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0514003db0 con 0x7f0520108620 2024-03-28T00:46:44.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.754+0000 7f051d7fa700 1 -- 172.21.15.169:0/1258424254 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0514011630 con 0x7f0520108620 2024-03-28T00:46:44.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.754+0000 7f0526440700 1 -- 172.21.15.169:0/1258424254 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f052019ba80 con 0x7f0520108620 2024-03-28T00:46:44.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.754+0000 7f051d7fa700 1 -- 172.21.15.169:0/1258424254 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0514011950 con 0x7f0520108620 2024-03-28T00:46:44.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.754+0000 7f0526440700 1 -- 172.21.15.169:0/1258424254 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f052019e770 con 0x7f0520108620 2024-03-28T00:46:44.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.754+0000 7f051d7fa700 1 -- 172.21.15.169:0/1258424254 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f051401a460 con 0x7f0520108620 2024-03-28T00:46:44.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.754+0000 7f051d7fa700 1 -- 172.21.15.169:0/1258424254 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f051401b780 con 0x7f0520108620 2024-03-28T00:46:44.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.755+0000 7f0526440700 1 -- 172.21.15.169:0/1258424254 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05200637c0 con 0x7f0520108620 2024-03-28T00:46:44.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.757+0000 7f051d7fa700 1 -- 172.21.15.169:0/1258424254 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f051401ba20 con 0x7f0520108620 2024-03-28T00:46:44.488 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.827+0000 7f0526440700 1 -- 172.21.15.169:0/1258424254 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f052002dd30 con 0x7f0520108620 2024-03-28T00:46:44.488 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.827+0000 7f051d7fa700 1 -- 172.21.15.169:0/1258424254 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+316 (secure 0 0 0) 0x7f051401bc00 con 0x7f0520108620 2024-03-28T00:46:44.488 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.828+0000 7f0526440700 1 -- 172.21.15.169:0/1258424254 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0520108620 msgr2=0x7f052019b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:46:44.488 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.828+0000 7f0526440700 1 --2- 172.21.15.169:0/1258424254 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0520108620 0x7f052019b2b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0514000c00 tx=0x7f0514004750 comp rx=0 tx=0).stop 2024-03-28T00:46:44.488 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.828+0000 7f0526440700 1 -- 172.21.15.169:0/1258424254 shutdown_connections 2024-03-28T00:46:44.488 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.828+0000 7f0526440700 1 --2- 172.21.15.169:0/1258424254 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0520108620 0x7f052019b2b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:46:44.488 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.828+0000 7f0526440700 1 -- 172.21.15.169:0/1258424254 >> 172.21.15.169:0/1258424254 conn(0x7f0520102030 msgr2=0x7f05201042f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:46:44.488 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.828+0000 7f0526440700 1 -- 172.21.15.169:0/1258424254 shutdown_connections 2024-03-28T00:46:44.488 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:43.828+0000 7f0526440700 1 -- 172.21.15.169:0/1258424254 wait complete. 2024-03-28T00:46:44.488 INFO:teuthology.orchestra.run.smithi169.stdout:mon is available 2024-03-28T00:46:44.488 INFO:teuthology.orchestra.run.smithi169.stdout:Assimilating anything we can from ceph.conf... 2024-03-28T00:46:44.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:44 smithi169 ceph-mon[31897]: from='client.? 172.21.15.169:0/1258424254' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-28T00:46:45.523 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-28T00:46:45.523 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T00:46:45.523 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout fsid = 833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T00:46:45.523 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.169:3300,v1:172.21.15.169:6789] 2024-03-28T00:46:45.523 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T00:46:45.523 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T00:46:45.523 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T00:46:45.524 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T00:46:45.524 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-28T00:46:45.524 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T00:46:45.524 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T00:46:45.524 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-28T00:46:45.524 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T00:46:45.524 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T00:46:45.524 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T00:46:45.524 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T00:46:45.524 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T00:46:45.524 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T00:46:45.524 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.894+0000 7eff36882700 1 Processor -- start 2024-03-28T00:46:45.525 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.894+0000 7eff36882700 1 -- start start 2024-03-28T00:46:45.525 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.894+0000 7eff36882700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff30109db0 0x7eff3010a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:46:45.525 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.894+0000 7eff36882700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3010a7a0 con 0x7eff30109db0 2024-03-28T00:46:45.525 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.895+0000 7eff2ffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff30109db0 0x7eff3010a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:46:45.525 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.895+0000 7eff2ffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff30109db0 0x7eff3010a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:44134/0 (socket says 172.21.15.169:44134) 2024-03-28T00:46:45.525 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.895+0000 7eff2ffff700 1 -- 172.21.15.169:0/3766968180 learned_addr learned my addr 172.21.15.169:0/3766968180 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:46:45.525 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.895+0000 7eff2ffff700 1 -- 172.21.15.169:0/3766968180 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3010af10 con 0x7eff30109db0 2024-03-28T00:46:45.525 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.895+0000 7eff2ffff700 1 --2- 172.21.15.169:0/3766968180 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff30109db0 0x7eff3010a1d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7eff200098a0 tx=0x7eff20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=836b136547e29c8a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:46:45.525 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.895+0000 7eff2effd700 1 -- 172.21.15.169:0/3766968180 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff20004160 con 0x7eff30109db0 2024-03-28T00:46:45.525 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.895+0000 7eff2effd700 1 -- 172.21.15.169:0/3766968180 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff20004300 con 0x7eff30109db0 2024-03-28T00:46:45.525 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.895+0000 7eff2effd700 1 -- 172.21.15.169:0/3766968180 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff200036a0 con 0x7eff30109db0 2024-03-28T00:46:45.525 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.896+0000 7eff36882700 1 -- 172.21.15.169:0/3766968180 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff30109db0 msgr2=0x7eff3010a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:46:45.525 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.896+0000 7eff36882700 1 --2- 172.21.15.169:0/3766968180 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff30109db0 0x7eff3010a1d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7eff200098a0 tx=0x7eff20009bb0 comp rx=0 tx=0).stop 2024-03-28T00:46:45.526 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.896+0000 7eff36882700 1 -- 172.21.15.169:0/3766968180 shutdown_connections 2024-03-28T00:46:45.526 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.896+0000 7eff36882700 1 --2- 172.21.15.169:0/3766968180 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff30109db0 0x7eff3010a1d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:46:45.526 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.896+0000 7eff36882700 1 -- 172.21.15.169:0/3766968180 >> 172.21.15.169:0/3766968180 conn(0x7eff3007d220 msgr2=0x7eff3007d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:46:45.526 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.896+0000 7eff36882700 1 -- 172.21.15.169:0/3766968180 shutdown_connections 2024-03-28T00:46:45.526 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.896+0000 7eff36882700 1 -- 172.21.15.169:0/3766968180 wait complete. 2024-03-28T00:46:45.526 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.896+0000 7eff36882700 1 Processor -- start 2024-03-28T00:46:45.526 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.896+0000 7eff36882700 1 -- start start 2024-03-28T00:46:45.526 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.897+0000 7eff36882700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff30109db0 0x7eff3019d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:46:45.526 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.897+0000 7eff36882700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3010b440 con 0x7eff30109db0 2024-03-28T00:46:45.526 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.897+0000 7eff2ffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff30109db0 0x7eff3019d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:46:45.526 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.897+0000 7eff2ffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff30109db0 0x7eff3019d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:44142/0 (socket says 172.21.15.169:44142) 2024-03-28T00:46:45.526 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.897+0000 7eff2ffff700 1 -- 172.21.15.169:0/2175955177 learned_addr learned my addr 172.21.15.169:0/2175955177 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:46:45.527 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.897+0000 7eff2ffff700 1 -- 172.21.15.169:0/2175955177 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3019da40 con 0x7eff30109db0 2024-03-28T00:46:45.527 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.897+0000 7eff2ffff700 1 --2- 172.21.15.169:0/2175955177 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff30109db0 0x7eff3019d500 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7eff20000c00 tx=0x7eff20004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:46:45.527 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.897+0000 7eff2d7fa700 1 -- 172.21.15.169:0/2175955177 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff20003db0 con 0x7eff30109db0 2024-03-28T00:46:45.527 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.897+0000 7eff2d7fa700 1 -- 172.21.15.169:0/2175955177 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff20011630 con 0x7eff30109db0 2024-03-28T00:46:45.527 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.897+0000 7eff2d7fa700 1 -- 172.21.15.169:0/2175955177 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff20011950 con 0x7eff30109db0 2024-03-28T00:46:45.527 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.897+0000 7eff36882700 1 -- 172.21.15.169:0/2175955177 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff3019dcd0 con 0x7eff30109db0 2024-03-28T00:46:45.527 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.898+0000 7eff36882700 1 -- 172.21.15.169:0/2175955177 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff301a0970 con 0x7eff30109db0 2024-03-28T00:46:45.527 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.898+0000 7eff2d7fa700 1 -- 172.21.15.169:0/2175955177 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7eff2001a460 con 0x7eff30109db0 2024-03-28T00:46:45.527 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.898+0000 7eff2d7fa700 1 -- 172.21.15.169:0/2175955177 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7eff2001b780 con 0x7eff30109db0 2024-03-28T00:46:45.527 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.898+0000 7eff36882700 1 -- 172.21.15.169:0/2175955177 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff300637c0 con 0x7eff30109db0 2024-03-28T00:46:45.527 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.901+0000 7eff2d7fa700 1 -- 172.21.15.169:0/2175955177 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7eff20031020 con 0x7eff30109db0 2024-03-28T00:46:45.527 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.969+0000 7eff36882700 1 -- 172.21.15.169:0/2175955177 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7eff3002dd30 con 0x7eff30109db0 2024-03-28T00:46:45.527 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.972+0000 7eff2d7fa700 1 -- 172.21.15.169:0/2175955177 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7eff2001b980 con 0x7eff30109db0 2024-03-28T00:46:45.528 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.972+0000 7eff2d7fa700 1 -- 172.21.15.169:0/2175955177 <== mon.0 v2:172.21.15.169:3300/0 8 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff2001ad00 con 0x7eff30109db0 2024-03-28T00:46:45.528 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.973+0000 7eff36882700 1 -- 172.21.15.169:0/2175955177 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff30109db0 msgr2=0x7eff3019d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:46:45.528 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.973+0000 7eff36882700 1 --2- 172.21.15.169:0/2175955177 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff30109db0 0x7eff3019d500 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7eff20000c00 tx=0x7eff20004750 comp rx=0 tx=0).stop 2024-03-28T00:46:45.528 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.973+0000 7eff36882700 1 -- 172.21.15.169:0/2175955177 shutdown_connections 2024-03-28T00:46:45.528 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.973+0000 7eff36882700 1 --2- 172.21.15.169:0/2175955177 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff30109db0 0x7eff3019d500 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:46:45.528 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.973+0000 7eff36882700 1 -- 172.21.15.169:0/2175955177 >> 172.21.15.169:0/2175955177 conn(0x7eff3007d220 msgr2=0x7eff30194000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:46:45.528 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.973+0000 7eff36882700 1 -- 172.21.15.169:0/2175955177 shutdown_connections 2024-03-28T00:46:45.528 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:44.973+0000 7eff36882700 1 -- 172.21.15.169:0/2175955177 wait complete. 2024-03-28T00:46:45.528 INFO:teuthology.orchestra.run.smithi169.stdout:Generating new minimal ceph.conf... 2024-03-28T00:46:45.708 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:45 smithi169 ceph-mon[31897]: from='client.? 172.21.15.169:0/2175955177' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-28T00:46:45.708 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:45 smithi169 ceph-mon[31897]: from='client.? 172.21.15.169:0/2175955177' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-28T00:46:46.578 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.894+0000 7f9038f1d700 1 Processor -- start 2024-03-28T00:46:46.578 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.894+0000 7f9038f1d700 1 -- start start 2024-03-28T00:46:46.578 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.894+0000 7f9038f1d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9034106a40 0x7f9034106e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:46:46.578 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.894+0000 7f9038f1d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9034107430 con 0x7f9034106a40 2024-03-28T00:46:46.578 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.894+0000 7f903259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9034106a40 0x7f9034106e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:46:46.578 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.895+0000 7f903259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9034106a40 0x7f9034106e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:44146/0 (socket says 172.21.15.169:44146) 2024-03-28T00:46:46.578 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.895+0000 7f903259c700 1 -- 172.21.15.169:0/1734662699 learned_addr learned my addr 172.21.15.169:0/1734662699 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:46:46.578 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.895+0000 7f903259c700 1 -- 172.21.15.169:0/1734662699 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9034107c40 con 0x7f9034106a40 2024-03-28T00:46:46.578 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.895+0000 7f903259c700 1 --2- 172.21.15.169:0/1734662699 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9034106a40 0x7f9034106e60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f90240098a0 tx=0x7f9024009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a7ed0fe2d9b124f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:46:46.578 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.895+0000 7f903159a700 1 -- 172.21.15.169:0/1734662699 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9024004160 con 0x7f9034106a40 2024-03-28T00:46:46.578 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.895+0000 7f903159a700 1 -- 172.21.15.169:0/1734662699 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9024004300 con 0x7f9034106a40 2024-03-28T00:46:46.579 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.895+0000 7f903159a700 1 -- 172.21.15.169:0/1734662699 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9024011b50 con 0x7f9034106a40 2024-03-28T00:46:46.579 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.896+0000 7f9038f1d700 1 -- 172.21.15.169:0/1734662699 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9034106a40 msgr2=0x7f9034106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:46:46.579 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.896+0000 7f9038f1d700 1 --2- 172.21.15.169:0/1734662699 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9034106a40 0x7f9034106e60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f90240098a0 tx=0x7f9024009bb0 comp rx=0 tx=0).stop 2024-03-28T00:46:46.579 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.896+0000 7f9038f1d700 1 -- 172.21.15.169:0/1734662699 shutdown_connections 2024-03-28T00:46:46.579 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.896+0000 7f9038f1d700 1 --2- 172.21.15.169:0/1734662699 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9034106a40 0x7f9034106e60 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:46:46.579 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.896+0000 7f9038f1d700 1 -- 172.21.15.169:0/1734662699 >> 172.21.15.169:0/1734662699 conn(0x7f9034102000 msgr2=0x7f9034104420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:46:46.579 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.896+0000 7f9038f1d700 1 -- 172.21.15.169:0/1734662699 shutdown_connections 2024-03-28T00:46:46.579 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.896+0000 7f9038f1d700 1 -- 172.21.15.169:0/1734662699 wait complete. 2024-03-28T00:46:46.579 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.897+0000 7f9038f1d700 1 Processor -- start 2024-03-28T00:46:46.579 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.897+0000 7f9038f1d700 1 -- start start 2024-03-28T00:46:46.579 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.897+0000 7f9038f1d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9034106a40 0x7f9034191ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:46:46.579 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.897+0000 7f9038f1d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903410aff0 con 0x7f9034106a40 2024-03-28T00:46:46.579 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.897+0000 7f903259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9034106a40 0x7f9034191ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:46:46.579 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.897+0000 7f903259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9034106a40 0x7f9034191ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:44156/0 (socket says 172.21.15.169:44156) 2024-03-28T00:46:46.580 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.897+0000 7f903259c700 1 -- 172.21.15.169:0/2145607675 learned_addr learned my addr 172.21.15.169:0/2145607675 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:46:46.580 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.897+0000 7f903259c700 1 -- 172.21.15.169:0/2145607675 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90341920e0 con 0x7f9034106a40 2024-03-28T00:46:46.580 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.897+0000 7f903259c700 1 --2- 172.21.15.169:0/2145607675 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9034106a40 0x7f9034191ba0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9024009fb0 tx=0x7f9024011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:46:46.580 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.897+0000 7f90237fe700 1 -- 172.21.15.169:0/2145607675 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90240038b0 con 0x7f9034106a40 2024-03-28T00:46:46.580 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.897+0000 7f90237fe700 1 -- 172.21.15.169:0/2145607675 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9024011e50 con 0x7f9034106a40 2024-03-28T00:46:46.580 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.897+0000 7f90237fe700 1 -- 172.21.15.169:0/2145607675 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9024019550 con 0x7f9034106a40 2024-03-28T00:46:46.580 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.897+0000 7f9038f1d700 1 -- 172.21.15.169:0/2145607675 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9034190320 con 0x7f9034106a40 2024-03-28T00:46:46.580 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.897+0000 7f9038f1d700 1 -- 172.21.15.169:0/2145607675 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9034190800 con 0x7f9034106a40 2024-03-28T00:46:46.580 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.898+0000 7f90237fe700 1 -- 172.21.15.169:0/2145607675 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9024019b90 con 0x7f9034106a40 2024-03-28T00:46:46.580 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.898+0000 7f90237fe700 1 -- 172.21.15.169:0/2145607675 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9024024730 con 0x7f9034106a40 2024-03-28T00:46:46.580 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.898+0000 7f9038f1d700 1 -- 172.21.15.169:0/2145607675 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90340637c0 con 0x7f9034106a40 2024-03-28T00:46:46.580 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.901+0000 7f90237fe700 1 -- 172.21.15.169:0/2145607675 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9024003a50 con 0x7f9034106a40 2024-03-28T00:46:46.580 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.967+0000 7f9038f1d700 1 -- 172.21.15.169:0/2145607675 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f9034195960 con 0x7f9034106a40 2024-03-28T00:46:46.580 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.968+0000 7f90237fe700 1 -- 172.21.15.169:0/2145607675 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f90240249d0 con 0x7f9034106a40 2024-03-28T00:46:46.580 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.968+0000 7f9038f1d700 1 -- 172.21.15.169:0/2145607675 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9034106a40 msgr2=0x7f9034191ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:46:46.581 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.968+0000 7f9038f1d700 1 --2- 172.21.15.169:0/2145607675 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9034106a40 0x7f9034191ba0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9024009fb0 tx=0x7f9024011bf0 comp rx=0 tx=0).stop 2024-03-28T00:46:46.581 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.969+0000 7f9038f1d700 1 -- 172.21.15.169:0/2145607675 shutdown_connections 2024-03-28T00:46:46.581 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.969+0000 7f9038f1d700 1 --2- 172.21.15.169:0/2145607675 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9034106a40 0x7f9034191ba0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:46:46.581 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.969+0000 7f9038f1d700 1 -- 172.21.15.169:0/2145607675 >> 172.21.15.169:0/2145607675 conn(0x7f9034102000 msgr2=0x7f9034102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:46:46.581 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.969+0000 7f9038f1d700 1 -- 172.21.15.169:0/2145607675 shutdown_connections 2024-03-28T00:46:46.581 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:45.969+0000 7f9038f1d700 1 -- 172.21.15.169:0/2145607675 wait complete. 2024-03-28T00:46:46.581 INFO:teuthology.orchestra.run.smithi169.stdout:Restarting the monitor... 2024-03-28T00:46:46.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:46 smithi169 ceph-mon[31897]: from='client.? 172.21.15.169:0/2145607675' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:46:46.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:46 smithi169 systemd[1]: Stopping Ceph mon.a for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T00:46:47.207 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:46 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-a[31893]: 2024-03-28T00:46:46.828+0000 7ff01d32a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T00:46:47.208 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:46 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-a[31893]: 2024-03-28T00:46:46.828+0000 7ff01d32a700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-28T00:46:47.208 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:47 smithi169 podman[32143]: 2024-03-28 00:46:47.013254054 +0000 UTC m=+0.281223364 container died f92ae489f9b0dcc445b66163024f19f979a728adc25e7492f6c696dbc7bcfc93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T00:46:47.533 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:47 smithi169 podman[32143]: 2024-03-28 00:46:47.295815432 +0000 UTC m=+0.563784747 container cleanup f92ae489f9b0dcc445b66163024f19f979a728adc25e7492f6c696dbc7bcfc93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T00:46:47.533 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:47 smithi169 bash[32143]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-a 2024-03-28T00:46:47.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:47 smithi169 podman[32157]: 2024-03-28 00:46:47.530161679 +0000 UTC m=+0.512021145 container remove f92ae489f9b0dcc445b66163024f19f979a728adc25e7492f6c696dbc7bcfc93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T00:46:47.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:47 smithi169 systemd[1]: ceph-833368da-ec9c-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-28T00:46:47.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:47 smithi169 systemd[1]: Stopped Ceph mon.a for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T00:46:47.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:47 smithi169 systemd[1]: Starting Ceph mon.a for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T00:46:48.162 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:47 smithi169 podman[32255]: 2024-03-28 00:46:47.780948519 +0000 UTC m=+0.017838708 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T00:46:48.162 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:47 smithi169 podman[32255]: 2024-03-28 00:46:47.889523533 +0000 UTC m=+0.126413709 container create 559d191b9d44bf626ddc94c817add793be07571fbeda37425420013eca489a78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-28T00:46:48.162 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 podman[32255]: 2024-03-28 00:46:48.075174836 +0000 UTC m=+0.312065015 container init 559d191b9d44bf626ddc94c817add793be07571fbeda37425420013eca489a78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T00:46:48.162 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 podman[32255]: 2024-03-28 00:46:48.078008468 +0000 UTC m=+0.314898644 container start 559d191b9d44bf626ddc94c817add793be07571fbeda37425420013eca489a78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-a, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-28T00:46:48.162 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: set uid:gid to 167:167 (ceph:ceph) 2024-03-28T00:46:48.162 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-28T00:46:48.162 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: pidfile_write: ignore empty --pid-file 2024-03-28T00:46:48.162 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: load: jerasure load: lrc 2024-03-28T00:46:48.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: RocksDB version: 7.9.2 2024-03-28T00:46:48.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Git sha 0 2024-03-28T00:46:48.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-28T00:46:48.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: DB SUMMARY 2024-03-28T00:46:48.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: DB Session ID: EW38QTSCDR1GMRE17QG6 2024-03-28T00:46:48.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: CURRENT file: CURRENT 2024-03-28T00:46:48.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: IDENTITY file: IDENTITY 2024-03-28T00:46:48.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-28T00:46:48.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-28T00:46:48.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 101075 ; 2024-03-28T00:46:48.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.error_if_exists: 0 2024-03-28T00:46:48.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.create_if_missing: 0 2024-03-28T00:46:48.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.paranoid_checks: 1 2024-03-28T00:46:48.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-28T00:46:48.163 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-28T00:46:48.164 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-28T00:46:48.164 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.env: 0x5612c47c9b20 2024-03-28T00:46:48.164 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.fs: PosixFileSystem 2024-03-28T00:46:48.164 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.info_log: 0x5612c527f920 2024-03-28T00:46:48.164 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_file_opening_threads: 16 2024-03-28T00:46:48.164 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.statistics: (nil) 2024-03-28T00:46:48.164 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.use_fsync: 0 2024-03-28T00:46:48.164 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_log_file_size: 0 2024-03-28T00:46:48.164 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-28T00:46:48.164 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-28T00:46:48.164 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.keep_log_file_num: 1000 2024-03-28T00:46:48.164 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.recycle_log_file_num: 0 2024-03-28T00:46:48.164 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.allow_fallocate: 1 2024-03-28T00:46:48.164 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.allow_mmap_reads: 0 2024-03-28T00:46:48.164 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.allow_mmap_writes: 0 2024-03-28T00:46:48.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.use_direct_reads: 0 2024-03-28T00:46:48.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-28T00:46:48.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.create_missing_column_families: 0 2024-03-28T00:46:48.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.db_log_dir: 2024-03-28T00:46:48.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.wal_dir: 2024-03-28T00:46:48.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-28T00:46:48.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-28T00:46:48.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-28T00:46:48.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-28T00:46:48.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-28T00:46:48.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-28T00:46:48.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.advise_random_on_open: 1 2024-03-28T00:46:48.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.db_write_buffer_size: 0 2024-03-28T00:46:48.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.write_buffer_manager: 0x5612c52521e0 2024-03-28T00:46:48.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-28T00:46:48.166 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-28T00:46:48.166 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-28T00:46:48.166 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.rate_limiter: (nil) 2024-03-28T00:46:48.167 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-28T00:46:48.167 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.wal_recovery_mode: 2 2024-03-28T00:46:48.167 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.enable_thread_tracking: 0 2024-03-28T00:46:48.167 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.enable_pipelined_write: 0 2024-03-28T00:46:48.167 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.unordered_write: 0 2024-03-28T00:46:48.167 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-28T00:46:48.167 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-28T00:46:48.167 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-28T00:46:48.168 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-28T00:46:48.168 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.row_cache: None 2024-03-28T00:46:48.168 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.wal_filter: None 2024-03-28T00:46:48.168 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-28T00:46:48.168 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.allow_ingest_behind: 0 2024-03-28T00:46:48.168 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.two_write_queues: 0 2024-03-28T00:46:48.168 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.manual_wal_flush: 0 2024-03-28T00:46:48.168 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.wal_compression: 0 2024-03-28T00:46:48.168 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.atomic_flush: 0 2024-03-28T00:46:48.168 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-28T00:46:48.168 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-28T00:46:48.168 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-28T00:46:48.168 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.log_readahead_size: 0 2024-03-28T00:46:48.168 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-28T00:46:48.168 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.best_efforts_recovery: 0 2024-03-28T00:46:48.169 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-28T00:46:48.169 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-28T00:46:48.169 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.allow_data_in_errors: 0 2024-03-28T00:46:48.169 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.db_host_id: __hostname__ 2024-03-28T00:46:48.169 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-28T00:46:48.169 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_background_jobs: 2 2024-03-28T00:46:48.169 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_background_compactions: -1 2024-03-28T00:46:48.169 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_subcompactions: 1 2024-03-28T00:46:48.169 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-28T00:46:48.169 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-28T00:46:48.169 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-28T00:46:48.169 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_total_wal_size: 0 2024-03-28T00:46:48.169 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-28T00:46:48.169 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-28T00:46:48.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-28T00:46:48.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-28T00:46:48.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_open_files: -1 2024-03-28T00:46:48.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.bytes_per_sync: 0 2024-03-28T00:46:48.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-28T00:46:48.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-28T00:46:48.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compaction_readahead_size: 0 2024-03-28T00:46:48.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_background_flushes: -1 2024-03-28T00:46:48.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Compression algorithms supported: 2024-03-28T00:46:48.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-28T00:46:48.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: kZSTD supported: 0 2024-03-28T00:46:48.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: kXpressCompression supported: 0 2024-03-28T00:46:48.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: kLZ4HCCompression supported: 1 2024-03-28T00:46:48.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: kZlibCompression supported: 1 2024-03-28T00:46:48.170 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: kSnappyCompression supported: 1 2024-03-28T00:46:48.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: kLZ4Compression supported: 1 2024-03-28T00:46:48.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: kBZip2Compression supported: 0 2024-03-28T00:46:48.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-28T00:46:48.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-28T00:46:48.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-28T00:46:48.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-28T00:46:48.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-28T00:46:48.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.merge_operator: 2024-03-28T00:46:48.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compaction_filter: None 2024-03-28T00:46:48.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compaction_filter_factory: None 2024-03-28T00:46:48.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.sst_partitioner_factory: None 2024-03-28T00:46:48.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-28T00:46:48.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-28T00:46:48.171 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5612c527fa20) 2024-03-28T00:46:48.171 INFO:journalctl@ceph.mon.a.smithi169.stdout: cache_index_and_filter_blocks: 1 2024-03-28T00:46:48.172 INFO:journalctl@ceph.mon.a.smithi169.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-28T00:46:48.172 INFO:journalctl@ceph.mon.a.smithi169.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-28T00:46:48.172 INFO:journalctl@ceph.mon.a.smithi169.stdout: pin_top_level_index_and_filter: 1 2024-03-28T00:46:48.172 INFO:journalctl@ceph.mon.a.smithi169.stdout: index_type: 0 2024-03-28T00:46:48.172 INFO:journalctl@ceph.mon.a.smithi169.stdout: data_block_index_type: 0 2024-03-28T00:46:48.172 INFO:journalctl@ceph.mon.a.smithi169.stdout: index_shortening: 1 2024-03-28T00:46:48.172 INFO:journalctl@ceph.mon.a.smithi169.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-28T00:46:48.172 INFO:journalctl@ceph.mon.a.smithi169.stdout: checksum: 4 2024-03-28T00:46:48.172 INFO:journalctl@ceph.mon.a.smithi169.stdout: no_block_cache: 0 2024-03-28T00:46:48.172 INFO:journalctl@ceph.mon.a.smithi169.stdout: block_cache: 0x5612c5285090 2024-03-28T00:46:48.172 INFO:journalctl@ceph.mon.a.smithi169.stdout: block_cache_name: BinnedLRUCache 2024-03-28T00:46:48.172 INFO:journalctl@ceph.mon.a.smithi169.stdout: block_cache_options: 2024-03-28T00:46:48.172 INFO:journalctl@ceph.mon.a.smithi169.stdout: capacity : 536870912 2024-03-28T00:46:48.172 INFO:journalctl@ceph.mon.a.smithi169.stdout: num_shard_bits : 4 2024-03-28T00:46:48.172 INFO:journalctl@ceph.mon.a.smithi169.stdout: strict_capacity_limit : 0 2024-03-28T00:46:48.172 INFO:journalctl@ceph.mon.a.smithi169.stdout: high_pri_pool_ratio: 0.000 2024-03-28T00:46:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout: block_cache_compressed: (nil) 2024-03-28T00:46:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout: persistent_cache: (nil) 2024-03-28T00:46:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout: block_size: 4096 2024-03-28T00:46:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout: block_size_deviation: 10 2024-03-28T00:46:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout: block_restart_interval: 16 2024-03-28T00:46:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout: index_block_restart_interval: 1 2024-03-28T00:46:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout: metadata_block_size: 4096 2024-03-28T00:46:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout: partition_filters: 0 2024-03-28T00:46:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout: use_delta_encoding: 1 2024-03-28T00:46:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout: filter_policy: bloomfilter 2024-03-28T00:46:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout: whole_key_filtering: 1 2024-03-28T00:46:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout: verify_compression: 0 2024-03-28T00:46:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout: read_amp_bytes_per_bit: 0 2024-03-28T00:46:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout: format_version: 5 2024-03-28T00:46:48.173 INFO:journalctl@ceph.mon.a.smithi169.stdout: enable_index_compression: 1 2024-03-28T00:46:48.174 INFO:journalctl@ceph.mon.a.smithi169.stdout: block_align: 0 2024-03-28T00:46:48.174 INFO:journalctl@ceph.mon.a.smithi169.stdout: max_auto_readahead_size: 262144 2024-03-28T00:46:48.174 INFO:journalctl@ceph.mon.a.smithi169.stdout: prepopulate_block_cache: 0 2024-03-28T00:46:48.174 INFO:journalctl@ceph.mon.a.smithi169.stdout: initial_auto_readahead_size: 8192 2024-03-28T00:46:48.174 INFO:journalctl@ceph.mon.a.smithi169.stdout: num_file_reads_for_auto_readahead: 2 2024-03-28T00:46:48.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.write_buffer_size: 33554432 2024-03-28T00:46:48.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_write_buffer_number: 2 2024-03-28T00:46:48.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compression: NoCompression 2024-03-28T00:46:48.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.bottommost_compression: Disabled 2024-03-28T00:46:48.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.prefix_extractor: nullptr 2024-03-28T00:46:48.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-28T00:46:48.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.num_levels: 7 2024-03-28T00:46:48.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-28T00:46:48.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-28T00:46:48.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-28T00:46:48.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-28T00:46:48.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-28T00:46:48.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-28T00:46:48.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-28T00:46:48.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-28T00:46:48.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-28T00:46:48.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-28T00:46:48.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-28T00:46:48.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-28T00:46:48.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-28T00:46:48.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compression_opts.level: 32767 2024-03-28T00:46:48.176 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compression_opts.strategy: 0 2024-03-28T00:46:48.176 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-28T00:46:48.176 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-28T00:46:48.176 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-28T00:46:48.176 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-28T00:46:48.176 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compression_opts.enabled: false 2024-03-28T00:46:48.176 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-28T00:46:48.176 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-28T00:46:48.176 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-28T00:46:48.176 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-28T00:46:48.176 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.target_file_size_base: 67108864 2024-03-28T00:46:48.176 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-28T00:46:48.176 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-28T00:46:48.176 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-28T00:46:48.176 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-28T00:46:48.177 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-28T00:46:48.177 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-28T00:46:48.177 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-28T00:46:48.177 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-28T00:46:48.177 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-28T00:46:48.177 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-28T00:46:48.177 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-28T00:46:48.177 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-28T00:46:48.177 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-28T00:46:48.177 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-28T00:46:48.177 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.arena_block_size: 1048576 2024-03-28T00:46:48.177 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-28T00:46:48.177 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-28T00:46:48.177 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.disable_auto_compactions: 0 2024-03-28T00:46:48.177 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-28T00:46:48.178 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-28T00:46:48.178 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-28T00:46:48.178 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-28T00:46:48.178 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-28T00:46:48.178 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-28T00:46:48.178 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-28T00:46:48.178 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-28T00:46:48.178 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-28T00:46:48.178 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-28T00:46:48.178 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.table_properties_collectors: 2024-03-28T00:46:48.178 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.inplace_update_support: 0 2024-03-28T00:46:48.178 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-28T00:46:48.178 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-28T00:46:48.178 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-28T00:46:48.178 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-28T00:46:48.179 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.bloom_locality: 0 2024-03-28T00:46:48.179 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.max_successive_merges: 0 2024-03-28T00:46:48.179 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-28T00:46:48.179 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.paranoid_file_checks: 0 2024-03-28T00:46:48.179 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.force_consistency_checks: 1 2024-03-28T00:46:48.179 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.report_bg_io_stats: 0 2024-03-28T00:46:48.179 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.ttl: 2592000 2024-03-28T00:46:48.179 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-28T00:46:48.179 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-28T00:46:48.179 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-28T00:46:48.179 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.enable_blob_files: false 2024-03-28T00:46:48.180 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.min_blob_size: 0 2024-03-28T00:46:48.180 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.blob_file_size: 268435456 2024-03-28T00:46:48.180 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-28T00:46:48.180 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-28T00:46:48.180 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-28T00:46:48.180 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-28T00:46:48.180 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-28T00:46:48.180 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.blob_file_starting_level: 0 2024-03-28T00:46:48.180 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-28T00:46:48.180 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-28T00:46:48.180 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-28T00:46:48.180 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: df2d0f69-3440-428c-96f3-d383ab6392b2 2024-03-28T00:46:48.180 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711586808107142, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-28T00:46:48.180 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-28T00:46:48.180 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711586808108201, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 96046, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 94036, "index_size": 292, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13964, "raw_average_key_size": 50, "raw_value_size": 86438, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 279, "num_filter_entries": 279, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711586808, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "df2d0f69-3440-428c-96f3-d383ab6392b2", "db_session_id": "EW38QTSCDR1GMRE17QG6", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-28T00:46:48.181 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711586808108284, "job": 1, "event": "recovery_finished"} 2024-03-28T00:46:48.181 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-28T00:46:48.181 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-28T00:46:48.181 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5612c5378000 2024-03-28T00:46:48.181 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: DB pointer 0x5612c5362000 2024-03-28T00:46:48.181 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-28T00:46:48.181 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-28T00:46:48.181 INFO:journalctl@ceph.mon.a.smithi169.stdout: ** DB Stats ** 2024-03-28T00:46:48.181 INFO:journalctl@ceph.mon.a.smithi169.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T00:46:48.181 INFO:journalctl@ceph.mon.a.smithi169.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-28T00:46:48.181 INFO:journalctl@ceph.mon.a.smithi169.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T00:46:48.181 INFO:journalctl@ceph.mon.a.smithi169.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T00:46:48.181 INFO:journalctl@ceph.mon.a.smithi169.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-28T00:46:48.181 INFO:journalctl@ceph.mon.a.smithi169.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T00:46:48.181 INFO:journalctl@ceph.mon.a.smithi169.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T00:46:48.182 INFO:journalctl@ceph.mon.a.smithi169.stdout: 2024-03-28T00:46:48.182 INFO:journalctl@ceph.mon.a.smithi169.stdout: ** Compaction Stats [default] ** 2024-03-28T00:46:48.182 INFO:journalctl@ceph.mon.a.smithi169.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T00:46:48.182 INFO:journalctl@ceph.mon.a.smithi169.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-28T00:46:48.182 INFO:journalctl@ceph.mon.a.smithi169.stdout: L0 2/0 95.63 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 129.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T00:46:48.182 INFO:journalctl@ceph.mon.a.smithi169.stdout: Sum 2/0 95.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 129.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T00:46:48.182 INFO:journalctl@ceph.mon.a.smithi169.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 129.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T00:46:48.182 INFO:journalctl@ceph.mon.a.smithi169.stdout: 2024-03-28T00:46:48.182 INFO:journalctl@ceph.mon.a.smithi169.stdout: ** Compaction Stats [default] ** 2024-03-28T00:46:48.182 INFO:journalctl@ceph.mon.a.smithi169.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T00:46:48.182 INFO:journalctl@ceph.mon.a.smithi169.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-28T00:46:48.182 INFO:journalctl@ceph.mon.a.smithi169.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 129.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T00:46:48.182 INFO:journalctl@ceph.mon.a.smithi169.stdout: 2024-03-28T00:46:48.182 INFO:journalctl@ceph.mon.a.smithi169.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-28T00:46:48.182 INFO:journalctl@ceph.mon.a.smithi169.stdout: 2024-03-28T00:46:48.183 INFO:journalctl@ceph.mon.a.smithi169.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T00:46:48.183 INFO:journalctl@ceph.mon.a.smithi169.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-28T00:46:48.183 INFO:journalctl@ceph.mon.a.smithi169.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-28T00:46:48.183 INFO:journalctl@ceph.mon.a.smithi169.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-28T00:46:48.183 INFO:journalctl@ceph.mon.a.smithi169.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-28T00:46:48.183 INFO:journalctl@ceph.mon.a.smithi169.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-28T00:46:48.183 INFO:journalctl@ceph.mon.a.smithi169.stdout: Cumulative compaction: 0.00 GB write, 30.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T00:46:48.183 INFO:journalctl@ceph.mon.a.smithi169.stdout: Interval compaction: 0.00 GB write, 30.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T00:46:48.183 INFO:journalctl@ceph.mon.a.smithi169.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-28T00:46:48.183 INFO:journalctl@ceph.mon.a.smithi169.stdout: Block cache BinnedLRUCache@0x5612c5285090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-28T00:46:48.183 INFO:journalctl@ceph.mon.a.smithi169.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-28T00:46:48.183 INFO:journalctl@ceph.mon.a.smithi169.stdout: 2024-03-28T00:46:48.183 INFO:journalctl@ceph.mon.a.smithi169.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-28T00:46:48.183 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: starting mon.a rank 0 at public addrs [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] at bind addrs [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T00:46:48.183 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: mon.a@-1(???) e1 preinit fsid 833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T00:46:48.183 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: mon.a@-1(???).mds e1 new map 2024-03-28T00:46:48.184 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: mon.a@-1(???).mds e1 print_map 2024-03-28T00:46:48.184 INFO:journalctl@ceph.mon.a.smithi169.stdout: e1 2024-03-28T00:46:48.184 INFO:journalctl@ceph.mon.a.smithi169.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T00:46:48.184 INFO:journalctl@ceph.mon.a.smithi169.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T00:46:48.184 INFO:journalctl@ceph.mon.a.smithi169.stdout: legacy client fscid: -1 2024-03-28T00:46:48.184 INFO:journalctl@ceph.mon.a.smithi169.stdout: 2024-03-28T00:46:48.184 INFO:journalctl@ceph.mon.a.smithi169.stdout: No filesystems configured 2024-03-28T00:46:48.184 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T00:46:48.184 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T00:46:48.184 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T00:46:48.184 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T00:46:48.185 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-28T00:46:48.185 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 bash[32255]: 559d191b9d44bf626ddc94c817add793be07571fbeda37425420013eca489a78 2024-03-28T00:46:48.185 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 systemd[1]: Started Ceph mon.a for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T00:46:48.185 INFO:teuthology.orchestra.run.smithi169.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-28T00:46:48.432 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T00:46:48.432 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: monmap e1: 1 mons at {a=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T00:46:48.432 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: fsmap 2024-03-28T00:46:48.432 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: osdmap e1: 0 total, 0 up, 0 in 2024-03-28T00:46:48.432 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:48 smithi169 ceph-mon[32274]: mgrmap e1: no daemons active 2024-03-28T00:46:49.421 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.529+0000 7fa0c6b61700 1 Processor -- start 2024-03-28T00:46:49.426 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.529+0000 7fa0c6b61700 1 -- start start 2024-03-28T00:46:49.427 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.529+0000 7fa0c6b61700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0c0106a10 0x7fa0c0106e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:46:49.427 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.529+0000 7fa0c6b61700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c0107400 con 0x7fa0c0106a10 2024-03-28T00:46:49.427 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.530+0000 7fa0c48fd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0c0106a10 0x7fa0c0106e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:46:49.427 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.530+0000 7fa0c48fd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0c0106a10 0x7fa0c0106e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:57738/0 (socket says 172.21.15.169:57738) 2024-03-28T00:46:49.427 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.530+0000 7fa0c48fd700 1 -- 172.21.15.169:0/219288943 learned_addr learned my addr 172.21.15.169:0/219288943 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:46:49.427 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.530+0000 7fa0c48fd700 1 -- 172.21.15.169:0/219288943 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c0107c10 con 0x7fa0c0106a10 2024-03-28T00:46:49.427 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.530+0000 7fa0c48fd700 1 --2- 172.21.15.169:0/219288943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0c0106a10 0x7fa0c0106e30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009e80 tx=0x7fa0bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b9b4bf2f2c43bc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:46:49.427 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.531+0000 7fa0b77fe700 1 -- 172.21.15.169:0/219288943 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0bc017070 con 0x7fa0c0106a10 2024-03-28T00:46:49.427 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.531+0000 7fa0b77fe700 1 -- 172.21.15.169:0/219288943 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0bc00b8c0 con 0x7fa0c0106a10 2024-03-28T00:46:49.427 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.531+0000 7fa0b77fe700 1 -- 172.21.15.169:0/219288943 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0bc0045c0 con 0x7fa0c0106a10 2024-03-28T00:46:49.427 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.531+0000 7fa0c6b61700 1 -- 172.21.15.169:0/219288943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0c0106a10 msgr2=0x7fa0c0106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:46:49.428 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.531+0000 7fa0c6b61700 1 --2- 172.21.15.169:0/219288943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0c0106a10 0x7fa0c0106e30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009e80 tx=0x7fa0bc00b330 comp rx=0 tx=0).stop 2024-03-28T00:46:49.428 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.531+0000 7fa0c6b61700 1 -- 172.21.15.169:0/219288943 shutdown_connections 2024-03-28T00:46:49.428 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.531+0000 7fa0c6b61700 1 --2- 172.21.15.169:0/219288943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0c0106a10 0x7fa0c0106e30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:46:49.428 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.531+0000 7fa0c6b61700 1 -- 172.21.15.169:0/219288943 >> 172.21.15.169:0/219288943 conn(0x7fa0c0101fd0 msgr2=0x7fa0c01043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:46:49.428 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.531+0000 7fa0c6b61700 1 -- 172.21.15.169:0/219288943 shutdown_connections 2024-03-28T00:46:49.428 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.531+0000 7fa0c6b61700 1 -- 172.21.15.169:0/219288943 wait complete. 2024-03-28T00:46:49.428 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.532+0000 7fa0c6b61700 1 Processor -- start 2024-03-28T00:46:49.428 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.532+0000 7fa0c6b61700 1 -- start start 2024-03-28T00:46:49.428 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.532+0000 7fa0c6b61700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0c0106a10 0x7fa0c019bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:46:49.428 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.532+0000 7fa0c6b61700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c010afc0 con 0x7fa0c0106a10 2024-03-28T00:46:49.428 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.532+0000 7fa0c48fd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0c0106a10 0x7fa0c019bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:46:49.428 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.532+0000 7fa0c48fd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0c0106a10 0x7fa0c019bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:57742/0 (socket says 172.21.15.169:57742) 2024-03-28T00:46:49.428 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.532+0000 7fa0c48fd700 1 -- 172.21.15.169:0/4022816819 learned_addr learned my addr 172.21.15.169:0/4022816819 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:46:49.428 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.532+0000 7fa0c48fd700 1 -- 172.21.15.169:0/4022816819 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c019c070 con 0x7fa0c0106a10 2024-03-28T00:46:49.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.533+0000 7fa0c48fd700 1 --2- 172.21.15.169:0/4022816819 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0c0106a10 0x7fa0c019bb30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc0072a0 tx=0x7fa0bc00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:46:49.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.533+0000 7fa0b5ffb700 1 -- 172.21.15.169:0/4022816819 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0bc020020 con 0x7fa0c0106a10 2024-03-28T00:46:49.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.533+0000 7fa0c6b61700 1 -- 172.21.15.169:0/4022816819 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c019c300 con 0x7fa0c0106a10 2024-03-28T00:46:49.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.533+0000 7fa0b5ffb700 1 -- 172.21.15.169:0/4022816819 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0bc003b50 con 0x7fa0c0106a10 2024-03-28T00:46:49.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.533+0000 7fa0b5ffb700 1 -- 172.21.15.169:0/4022816819 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0bc017040 con 0x7fa0c0106a10 2024-03-28T00:46:49.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.533+0000 7fa0c6b61700 1 -- 172.21.15.169:0/4022816819 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c019eff0 con 0x7fa0c0106a10 2024-03-28T00:46:49.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.533+0000 7fa0b5ffb700 1 -- 172.21.15.169:0/4022816819 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fa0bc023020 con 0x7fa0c0106a10 2024-03-28T00:46:49.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.533+0000 7fa0b5ffb700 1 -- 172.21.15.169:0/4022816819 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa0bc00f050 con 0x7fa0c0106a10 2024-03-28T00:46:49.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.534+0000 7fa0c6b61700 1 -- 172.21.15.169:0/4022816819 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c00637c0 con 0x7fa0c0106a10 2024-03-28T00:46:49.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.536+0000 7fa0b5ffb700 1 -- 172.21.15.169:0/4022816819 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fa0bc02a020 con 0x7fa0c0106a10 2024-03-28T00:46:49.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.606+0000 7fa0c6b61700 1 -- 172.21.15.169:0/4022816819 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fa0c002dd10 con 0x7fa0c0106a10 2024-03-28T00:46:49.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.608+0000 7fa0b5ffb700 1 -- 172.21.15.169:0/4022816819 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fa0bc003d00 con 0x7fa0c0106a10 2024-03-28T00:46:49.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.609+0000 7fa0c6b61700 1 -- 172.21.15.169:0/4022816819 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0c0106a10 msgr2=0x7fa0c019bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:46:49.429 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.609+0000 7fa0c6b61700 1 --2- 172.21.15.169:0/4022816819 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0c0106a10 0x7fa0c019bb30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc0072a0 tx=0x7fa0bc00bd50 comp rx=0 tx=0).stop 2024-03-28T00:46:49.430 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.609+0000 7fa0c6b61700 1 -- 172.21.15.169:0/4022816819 shutdown_connections 2024-03-28T00:46:49.430 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.609+0000 7fa0c6b61700 1 --2- 172.21.15.169:0/4022816819 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0c0106a10 0x7fa0c019bb30 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:46:49.430 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.609+0000 7fa0c6b61700 1 -- 172.21.15.169:0/4022816819 >> 172.21.15.169:0/4022816819 conn(0x7fa0c0101fd0 msgr2=0x7fa0c0102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:46:49.430 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.609+0000 7fa0c6b61700 1 -- 172.21.15.169:0/4022816819 shutdown_connections 2024-03-28T00:46:49.430 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:48.609+0000 7fa0c6b61700 1 -- 172.21.15.169:0/4022816819 wait complete. 2024-03-28T00:46:49.430 INFO:teuthology.orchestra.run.smithi169.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-28T00:46:49.451 INFO:teuthology.orchestra.run.smithi169.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-28T00:46:49.451 INFO:teuthology.orchestra.run.smithi169.stdout:Creating mgr... 2024-03-28T00:46:49.452 INFO:teuthology.orchestra.run.smithi169.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-28T00:46:49.452 INFO:teuthology.orchestra.run.smithi169.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-28T00:46:49.705 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-833368da-ec9c-11ee-95d0-87774f69a715@mgr.x 2024-03-28T00:46:49.705 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stderr Failed to reset failed state of unit ceph-833368da-ec9c-11ee-95d0-87774f69a715@mgr.x.service: Unit ceph-833368da-ec9c-11ee-95d0-87774f69a715@mgr.x.service not loaded. 2024-03-28T00:46:49.711 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:49 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4022816819' entity='client.admin' 2024-03-28T00:46:49.854 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-833368da-ec9c-11ee-95d0-87774f69a715.target.wants/ceph-833368da-ec9c-11ee-95d0-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-833368da-ec9c-11ee-95d0-87774f69a715@.service. 2024-03-28T00:46:50.053 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:49 smithi169 systemd[1]: Starting Ceph mgr.x for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T00:46:50.415 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:50 smithi169 podman[32492]: 2024-03-28 00:46:49.989781409 +0000 UTC m=+0.017696191 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T00:46:50.415 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:50 smithi169 podman[32492]: 2024-03-28 00:46:50.143076482 +0000 UTC m=+0.170991246 container create baba397a44a4429c5d6604d52739ce4b6b38d09d16f4c9860afb72300488736c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-03-28T00:46:50.416 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:50 smithi169 podman[32492]: 2024-03-28 00:46:50.328705855 +0000 UTC m=+0.356620618 container init baba397a44a4429c5d6604d52739ce4b6b38d09d16f4c9860afb72300488736c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-28T00:46:50.416 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:50 smithi169 podman[32492]: 2024-03-28 00:46:50.331097163 +0000 UTC m=+0.359011930 container start baba397a44a4429c5d6604d52739ce4b6b38d09d16f4c9860afb72300488736c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T00:46:50.416 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:50 smithi169 bash[32492]: baba397a44a4429c5d6604d52739ce4b6b38d09d16f4c9860afb72300488736c 2024-03-28T00:46:50.428 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T00:46:50.428 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout disabled 2024-03-28T00:46:50.436 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T00:46:50.436 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout inactive 2024-03-28T00:46:50.436 INFO:teuthology.orchestra.run.smithi169.stdout:firewalld.service is not enabled 2024-03-28T00:46:50.436 INFO:teuthology.orchestra.run.smithi169.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T00:46:50.444 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T00:46:50.444 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout disabled 2024-03-28T00:46:50.451 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T00:46:50.451 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout inactive 2024-03-28T00:46:50.452 INFO:teuthology.orchestra.run.smithi169.stdout:firewalld.service is not enabled 2024-03-28T00:46:50.452 INFO:teuthology.orchestra.run.smithi169.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-28T00:46:50.452 INFO:teuthology.orchestra.run.smithi169.stdout:Waiting for mgr to start... 2024-03-28T00:46:50.452 INFO:teuthology.orchestra.run.smithi169.stdout:Waiting for mgr... 2024-03-28T00:46:50.665 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:50 smithi169 systemd[1]: Started Ceph mgr.x for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T00:46:50.918 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:50 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:50.889+0000 7f070adc5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T00:46:51.255 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:50 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/870339547' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T00:46:51.256 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:50 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:50.990+0000 7f070adc5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T00:46:51.486 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-28T00:46:51.486 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout { 2024-03-28T00:46:51.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "fsid": "833368da-ec9c-11ee-95d0-87774f69a715", 2024-03-28T00:46:51.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T00:46:51.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T00:46:51.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T00:46:51.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T00:46:51.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:46:51.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T00:46:51.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T00:46:51.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 0 2024-03-28T00:46:51.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ], 2024-03-28T00:46:51.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T00:46:51.487 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T00:46:51.488 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ], 2024-03-28T00:46:51.488 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-28T00:46:51.488 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T00:46:51.488 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T00:46:51.488 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T00:46:51.488 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T00:46:51.488 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:46:51.488 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T00:46:51.488 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T00:46:51.488 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T00:46:51.488 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T00:46:51.488 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T00:46:51.497 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T00:46:51.497 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T00:46:51.497 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T00:46:51.497 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:46:51.497 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T00:46:51.497 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T00:46:51.497 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T00:46:51.497 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T00:46:51.497 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T00:46:51.497 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T00:46:51.497 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T00:46:51.498 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T00:46:51.498 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T00:46:51.498 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:46:51.498 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T00:46:51.498 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T00:46:51.498 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T00:46:51.498 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T00:46:51.498 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:46:51.498 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T00:46:51.498 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T00:46:51.498 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T00:46:51.498 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T00:46:51.498 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T00:46:51.499 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T00:46:51.499 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T00:46:51.499 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ], 2024-03-28T00:46:51.499 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T00:46:51.499 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:46:51.499 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T00:46:51.499 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T00:46:51.499 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T00:46:42.432708+0000", 2024-03-28T00:46:51.499 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T00:46:51.499 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:46:51.499 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T00:46:51.499 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout } 2024-03-28T00:46:51.499 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.823+0000 7fd0c5087700 1 Processor -- start 2024-03-28T00:46:51.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.823+0000 7fd0c5087700 1 -- start start 2024-03-28T00:46:51.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.823+0000 7fd0c5087700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c007dec0 0x7fd0c007c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:46:51.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.823+0000 7fd0c5087700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c007c8f0 con 0x7fd0c007dec0 2024-03-28T00:46:51.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.823+0000 7fd0bed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c007dec0 0x7fd0c007c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:46:51.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.823+0000 7fd0bed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c007dec0 0x7fd0c007c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:57764/0 (socket says 172.21.15.169:57764) 2024-03-28T00:46:51.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.823+0000 7fd0bed9d700 1 -- 172.21.15.169:0/1201673906 learned_addr learned my addr 172.21.15.169:0/1201673906 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:46:51.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.824+0000 7fd0bed9d700 1 -- 172.21.15.169:0/1201673906 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c007ca70 con 0x7fd0c007dec0 2024-03-28T00:46:51.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.824+0000 7fd0bed9d700 1 --2- 172.21.15.169:0/1201673906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c007dec0 0x7fd0c007c320 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd0b00098a0 tx=0x7fd0b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9103d680e1e26495 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:46:51.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.824+0000 7fd0bdd9b700 1 -- 172.21.15.169:0/1201673906 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0b0004160 con 0x7fd0c007dec0 2024-03-28T00:46:51.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.824+0000 7fd0bdd9b700 1 -- 172.21.15.169:0/1201673906 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd0b0004300 con 0x7fd0c007dec0 2024-03-28T00:46:51.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.824+0000 7fd0bdd9b700 1 -- 172.21.15.169:0/1201673906 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0b0011b50 con 0x7fd0c007dec0 2024-03-28T00:46:51.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.825+0000 7fd0c5087700 1 -- 172.21.15.169:0/1201673906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c007dec0 msgr2=0x7fd0c007c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:46:51.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.825+0000 7fd0c5087700 1 --2- 172.21.15.169:0/1201673906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c007dec0 0x7fd0c007c320 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd0b00098a0 tx=0x7fd0b0009bb0 comp rx=0 tx=0).stop 2024-03-28T00:46:51.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.825+0000 7fd0c5087700 1 -- 172.21.15.169:0/1201673906 shutdown_connections 2024-03-28T00:46:51.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.825+0000 7fd0c5087700 1 --2- 172.21.15.169:0/1201673906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c007dec0 0x7fd0c007c320 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:46:51.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.825+0000 7fd0c5087700 1 -- 172.21.15.169:0/1201673906 >> 172.21.15.169:0/1201673906 conn(0x7fd0c0103150 msgr2=0x7fd0c0105570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:46:51.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.825+0000 7fd0c5087700 1 -- 172.21.15.169:0/1201673906 shutdown_connections 2024-03-28T00:46:51.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.825+0000 7fd0c5087700 1 -- 172.21.15.169:0/1201673906 wait complete. 2024-03-28T00:46:51.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.826+0000 7fd0c5087700 1 Processor -- start 2024-03-28T00:46:51.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.826+0000 7fd0c5087700 1 -- start start 2024-03-28T00:46:51.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.826+0000 7fd0c5087700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c007dec0 0x7fd0c019dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:46:51.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.826+0000 7fd0c5087700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c010e160 con 0x7fd0c007dec0 2024-03-28T00:46:51.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.826+0000 7fd0bed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c007dec0 0x7fd0c019dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:46:51.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.826+0000 7fd0bed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c007dec0 0x7fd0c019dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:57766/0 (socket says 172.21.15.169:57766) 2024-03-28T00:46:51.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.826+0000 7fd0bed9d700 1 -- 172.21.15.169:0/870339547 learned_addr learned my addr 172.21.15.169:0/870339547 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:46:51.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.826+0000 7fd0bed9d700 1 -- 172.21.15.169:0/870339547 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c019e2e0 con 0x7fd0c007dec0 2024-03-28T00:46:51.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.827+0000 7fd0bed9d700 1 --2- 172.21.15.169:0/870339547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c007dec0 0x7fd0c019dda0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0009fb0 tx=0x7fd0b0011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:46:51.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.827+0000 7fd0a7fff700 1 -- 172.21.15.169:0/870339547 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0b00038b0 con 0x7fd0c007dec0 2024-03-28T00:46:51.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.827+0000 7fd0c5087700 1 -- 172.21.15.169:0/870339547 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c019e570 con 0x7fd0c007dec0 2024-03-28T00:46:51.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.827+0000 7fd0a7fff700 1 -- 172.21.15.169:0/870339547 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd0b0011e50 con 0x7fd0c007dec0 2024-03-28T00:46:51.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.827+0000 7fd0a7fff700 1 -- 172.21.15.169:0/870339547 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0b0019550 con 0x7fd0c007dec0 2024-03-28T00:46:51.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.827+0000 7fd0c5087700 1 -- 172.21.15.169:0/870339547 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c01a1220 con 0x7fd0c007dec0 2024-03-28T00:46:51.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.827+0000 7fd0a7fff700 1 -- 172.21.15.169:0/870339547 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fd0b0019be0 con 0x7fd0c007dec0 2024-03-28T00:46:51.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.827+0000 7fd0a7fff700 1 -- 172.21.15.169:0/870339547 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd0b0024730 con 0x7fd0c007dec0 2024-03-28T00:46:51.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.828+0000 7fd0c5087700 1 -- 172.21.15.169:0/870339547 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0c00637c0 con 0x7fd0c007dec0 2024-03-28T00:46:51.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.830+0000 7fd0a7fff700 1 -- 172.21.15.169:0/870339547 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fd0b0003a50 con 0x7fd0c007dec0 2024-03-28T00:46:51.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.902+0000 7fd0c5087700 1 -- 172.21.15.169:0/870339547 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd0c01a1490 con 0x7fd0c007dec0 2024-03-28T00:46:51.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.903+0000 7fd0a7fff700 1 -- 172.21.15.169:0/870339547 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fd0b00249d0 con 0x7fd0c007dec0 2024-03-28T00:46:51.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.904+0000 7fd0c5087700 1 -- 172.21.15.169:0/870339547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c007dec0 msgr2=0x7fd0c019dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:46:51.503 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.904+0000 7fd0c5087700 1 --2- 172.21.15.169:0/870339547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c007dec0 0x7fd0c019dda0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0009fb0 tx=0x7fd0b0011bf0 comp rx=0 tx=0).stop 2024-03-28T00:46:51.503 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.904+0000 7fd0c5087700 1 -- 172.21.15.169:0/870339547 shutdown_connections 2024-03-28T00:46:51.503 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.904+0000 7fd0c5087700 1 --2- 172.21.15.169:0/870339547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0c007dec0 0x7fd0c019dda0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:46:51.503 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.904+0000 7fd0c5087700 1 -- 172.21.15.169:0/870339547 >> 172.21.15.169:0/870339547 conn(0x7fd0c0103150 msgr2=0x7fd0c0103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:46:51.503 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.904+0000 7fd0c5087700 1 -- 172.21.15.169:0/870339547 shutdown_connections 2024-03-28T00:46:51.503 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:50.904+0000 7fd0c5087700 1 -- 172.21.15.169:0/870339547 wait complete. 2024-03-28T00:46:51.503 INFO:teuthology.orchestra.run.smithi169.stdout:mgr not available, waiting (1/15)... 2024-03-28T00:46:51.976 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:51 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:51.643+0000 7f070adc5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T00:46:51.976 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:51 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:51.736+0000 7f070adc5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T00:46:51.976 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:51 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:51.877+0000 7f070adc5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T00:46:52.302 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:51 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:51.972+0000 7f070adc5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T00:46:52.303 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:52 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:52.054+0000 7f070adc5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T00:46:52.612 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:52 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:52.318+0000 7f070adc5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T00:46:52.612 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:52 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:52.428+0000 7f070adc5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T00:46:52.612 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:52 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:52.513+0000 7f070adc5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T00:46:52.612 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:52 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:52.607+0000 7f070adc5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T00:46:53.052 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:52 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:52.783+0000 7f070adc5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T00:46:53.891 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:53 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:53.766+0000 7f070adc5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T00:46:54.253 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:54 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/756287653' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T00:46:54.253 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:53 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:53.904+0000 7f070adc5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T00:46:54.253 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:54 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:54.248+0000 7f070adc5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T00:46:54.537 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:54 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:54.338+0000 7f070adc5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T00:46:54.537 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:54 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:54.533+0000 7f070adc5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T00:46:54.542 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-28T00:46:54.542 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout { 2024-03-28T00:46:54.542 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "fsid": "833368da-ec9c-11ee-95d0-87774f69a715", 2024-03-28T00:46:54.542 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T00:46:54.542 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T00:46:54.542 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T00:46:54.542 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T00:46:54.542 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:46:54.543 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T00:46:54.543 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T00:46:54.543 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 0 2024-03-28T00:46:54.543 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ], 2024-03-28T00:46:54.543 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T00:46:54.543 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T00:46:54.543 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ], 2024-03-28T00:46:54.543 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-28T00:46:54.543 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T00:46:54.543 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T00:46:54.543 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T00:46:54.543 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T00:46:54.543 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:46:54.543 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T00:46:54.543 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T00:46:54.543 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T00:46:54.544 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T00:46:54.552 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T00:46:54.552 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T00:46:54.552 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T00:46:54.552 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T00:46:54.552 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:46:54.552 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T00:46:54.552 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T00:46:54.552 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T00:46:54.553 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T00:46:54.553 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T00:46:54.553 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T00:46:54.553 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T00:46:54.553 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T00:46:54.553 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T00:46:54.553 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:46:54.553 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T00:46:54.553 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T00:46:54.553 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T00:46:54.553 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T00:46:54.553 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:46:54.553 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T00:46:54.553 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T00:46:54.553 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T00:46:54.554 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T00:46:54.554 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T00:46:54.554 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T00:46:54.554 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T00:46:54.554 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ], 2024-03-28T00:46:54.554 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T00:46:54.554 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:46:54.554 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T00:46:54.554 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T00:46:54.554 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T00:46:42.432708+0000", 2024-03-28T00:46:54.554 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T00:46:54.554 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:46:54.554 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T00:46:54.554 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout } 2024-03-28T00:46:54.555 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.887+0000 7f66f50b3700 1 Processor -- start 2024-03-28T00:46:54.555 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.887+0000 7f66f50b3700 1 -- start start 2024-03-28T00:46:54.555 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.887+0000 7f66f50b3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f0106a50 0x7f66f0106e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:46:54.555 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.887+0000 7f66f50b3700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f0107440 con 0x7f66f0106a50 2024-03-28T00:46:54.555 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.888+0000 7f66eed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f0106a50 0x7f66f0106e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:46:54.555 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.888+0000 7f66eed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f0106a50 0x7f66f0106e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:57770/0 (socket says 172.21.15.169:57770) 2024-03-28T00:46:54.555 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.888+0000 7f66eed9d700 1 -- 172.21.15.169:0/2232529383 learned_addr learned my addr 172.21.15.169:0/2232529383 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:46:54.555 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.888+0000 7f66eed9d700 1 -- 172.21.15.169:0/2232529383 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f0107c50 con 0x7f66f0106a50 2024-03-28T00:46:54.555 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.888+0000 7f66eed9d700 1 --2- 172.21.15.169:0/2232529383 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f0106a50 0x7f66f0106e70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f66e000bb40 tx=0x7f66e000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=160cd556f0e1ca2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:46:54.555 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.888+0000 7f66edd9b700 1 -- 172.21.15.169:0/2232529383 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66e000dbe0 con 0x7f66f0106a50 2024-03-28T00:46:54.555 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.888+0000 7f66edd9b700 1 -- 172.21.15.169:0/2232529383 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f66e00044f0 con 0x7f66f0106a50 2024-03-28T00:46:54.555 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.888+0000 7f66edd9b700 1 -- 172.21.15.169:0/2232529383 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66e0010470 con 0x7f66f0106a50 2024-03-28T00:46:54.555 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.889+0000 7f66f50b3700 1 -- 172.21.15.169:0/2232529383 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f0106a50 msgr2=0x7f66f0106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:46:54.555 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.889+0000 7f66f50b3700 1 --2- 172.21.15.169:0/2232529383 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f0106a50 0x7f66f0106e70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f66e000bb40 tx=0x7f66e000be50 comp rx=0 tx=0).stop 2024-03-28T00:46:54.555 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.889+0000 7f66f50b3700 1 -- 172.21.15.169:0/2232529383 shutdown_connections 2024-03-28T00:46:54.556 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.889+0000 7f66f50b3700 1 --2- 172.21.15.169:0/2232529383 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f0106a50 0x7f66f0106e70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:46:54.556 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.889+0000 7f66f50b3700 1 -- 172.21.15.169:0/2232529383 >> 172.21.15.169:0/2232529383 conn(0x7f66f0102030 msgr2=0x7f66f0104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:46:54.556 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.889+0000 7f66f50b3700 1 -- 172.21.15.169:0/2232529383 shutdown_connections 2024-03-28T00:46:54.556 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.889+0000 7f66f50b3700 1 -- 172.21.15.169:0/2232529383 wait complete. 2024-03-28T00:46:54.556 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.890+0000 7f66f50b3700 1 Processor -- start 2024-03-28T00:46:54.556 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.890+0000 7f66f50b3700 1 -- start start 2024-03-28T00:46:54.556 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.890+0000 7f66f50b3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f0106a50 0x7f66f0199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:46:54.556 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.890+0000 7f66f50b3700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f007aff0 con 0x7f66f0106a50 2024-03-28T00:46:54.556 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.891+0000 7f66eed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f0106a50 0x7f66f0199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:46:54.556 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.891+0000 7f66eed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f0106a50 0x7f66f0199980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:57780/0 (socket says 172.21.15.169:57780) 2024-03-28T00:46:54.556 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.891+0000 7f66eed9d700 1 -- 172.21.15.169:0/756287653 learned_addr learned my addr 172.21.15.169:0/756287653 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:46:54.556 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.891+0000 7f66eed9d700 1 -- 172.21.15.169:0/756287653 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f0199ec0 con 0x7f66f0106a50 2024-03-28T00:46:54.556 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.891+0000 7f66eed9d700 1 --2- 172.21.15.169:0/756287653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f0106a50 0x7f66f0199980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f66e000b340 tx=0x7f66e00040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:46:54.556 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.891+0000 7f66d7fff700 1 -- 172.21.15.169:0/756287653 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66e0027070 con 0x7f66f0106a50 2024-03-28T00:46:54.557 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.891+0000 7f66d7fff700 1 -- 172.21.15.169:0/756287653 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f66e00104e0 con 0x7f66f0106a50 2024-03-28T00:46:54.557 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.891+0000 7f66f50b3700 1 -- 172.21.15.169:0/756287653 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66f019a150 con 0x7f66f0106a50 2024-03-28T00:46:54.557 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.891+0000 7f66f50b3700 1 -- 172.21.15.169:0/756287653 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f019ce40 con 0x7f66f0106a50 2024-03-28T00:46:54.557 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.891+0000 7f66d7fff700 1 -- 172.21.15.169:0/756287653 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66e0022c00 con 0x7f66f0106a50 2024-03-28T00:46:54.557 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.892+0000 7f66d7fff700 1 -- 172.21.15.169:0/756287653 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f66e002a070 con 0x7f66f0106a50 2024-03-28T00:46:54.557 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.892+0000 7f66d7fff700 1 -- 172.21.15.169:0/756287653 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f66e001f070 con 0x7f66f0106a50 2024-03-28T00:46:54.557 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.892+0000 7f66f50b3700 1 -- 172.21.15.169:0/756287653 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66f00637c0 con 0x7f66f0106a50 2024-03-28T00:46:54.557 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.894+0000 7f66d7fff700 1 -- 172.21.15.169:0/756287653 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f66e0015070 con 0x7f66f0106a50 2024-03-28T00:46:54.557 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.967+0000 7f66f50b3700 1 -- 172.21.15.169:0/756287653 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f66f002dd90 con 0x7f66f0106a50 2024-03-28T00:46:54.557 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.968+0000 7f66d7fff700 1 -- 172.21.15.169:0/756287653 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f66e0022410 con 0x7f66f0106a50 2024-03-28T00:46:54.557 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.969+0000 7f66f50b3700 1 -- 172.21.15.169:0/756287653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f0106a50 msgr2=0x7f66f0199980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:46:54.557 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.969+0000 7f66f50b3700 1 --2- 172.21.15.169:0/756287653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f0106a50 0x7f66f0199980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f66e000b340 tx=0x7f66e00040b0 comp rx=0 tx=0).stop 2024-03-28T00:46:54.557 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.969+0000 7f66f50b3700 1 -- 172.21.15.169:0/756287653 shutdown_connections 2024-03-28T00:46:54.557 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.969+0000 7f66f50b3700 1 --2- 172.21.15.169:0/756287653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f0106a50 0x7f66f0199980 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:46:54.557 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.969+0000 7f66f50b3700 1 -- 172.21.15.169:0/756287653 >> 172.21.15.169:0/756287653 conn(0x7f66f0102030 msgr2=0x7f66f0190420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:46:54.558 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.969+0000 7f66f50b3700 1 -- 172.21.15.169:0/756287653 shutdown_connections 2024-03-28T00:46:54.558 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:53.969+0000 7f66f50b3700 1 -- 172.21.15.169:0/756287653 wait complete. 2024-03-28T00:46:54.558 INFO:teuthology.orchestra.run.smithi169.stdout:mgr not available, waiting (2/15)... 2024-03-28T00:46:54.802 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:54 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:54.718+0000 7f070adc5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T00:46:55.263 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:54 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:54.998+0000 7f070adc5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T00:46:55.263 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:55 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:55.258+0000 7f070adc5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T00:46:55.552 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:55 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:55.494+0000 7f070adc5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T00:46:56.655 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:56 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:56.301+0000 7f070adc5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T00:46:56.914 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:56 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:56.886+0000 7f070adc5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T00:46:57.218 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:57 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1279602448' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T00:46:57.219 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:56 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:56.989+0000 7f070adc5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T00:46:57.219 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:46:57 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:46:57.214+0000 7f070adc5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T00:46:57.596 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-28T00:46:57.596 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout { 2024-03-28T00:46:57.596 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "fsid": "833368da-ec9c-11ee-95d0-87774f69a715", 2024-03-28T00:46:57.596 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T00:46:57.597 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T00:46:57.597 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T00:46:57.597 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T00:46:57.597 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:46:57.597 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T00:46:57.597 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T00:46:57.597 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 0 2024-03-28T00:46:57.597 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ], 2024-03-28T00:46:57.597 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T00:46:57.597 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T00:46:57.597 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ], 2024-03-28T00:46:57.597 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-28T00:46:57.598 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T00:46:57.598 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T00:46:57.598 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T00:46:57.598 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T00:46:57.598 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:46:57.598 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T00:46:57.598 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T00:46:57.598 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T00:46:57.598 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T00:46:57.598 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T00:46:57.598 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T00:46:57.598 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T00:46:57.598 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T00:46:57.598 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:46:57.599 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T00:46:57.599 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T00:46:57.599 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T00:46:57.599 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T00:46:57.599 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T00:46:57.599 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T00:46:57.599 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T00:46:57.599 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T00:46:57.599 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T00:46:57.599 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:46:57.599 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T00:46:57.599 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T00:46:57.599 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T00:46:57.599 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T00:46:57.600 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:46:57.600 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T00:46:57.600 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T00:46:57.600 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T00:46:57.600 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T00:46:57.600 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T00:46:57.600 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T00:46:57.600 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T00:46:57.600 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ], 2024-03-28T00:46:57.600 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T00:46:57.600 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:46:57.600 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T00:46:57.601 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T00:46:57.601 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T00:46:42.432708+0000", 2024-03-28T00:46:57.601 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T00:46:57.601 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:46:57.601 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T00:46:57.601 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout } 2024-03-28T00:46:57.601 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.910+0000 7faa44c53700 1 Processor -- start 2024-03-28T00:46:57.601 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.910+0000 7faa44c53700 1 -- start start 2024-03-28T00:46:57.601 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.910+0000 7faa44c53700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa40107fb0 0x7faa401083d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:46:57.601 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.910+0000 7faa44c53700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa401089a0 con 0x7faa40107fb0 2024-03-28T00:46:57.601 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.910+0000 7faa3e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa40107fb0 0x7faa401083d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:46:57.601 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.910+0000 7faa3e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa40107fb0 0x7faa401083d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53724/0 (socket says 172.21.15.169:53724) 2024-03-28T00:46:57.601 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.910+0000 7faa3e59c700 1 -- 172.21.15.169:0/1572325888 learned_addr learned my addr 172.21.15.169:0/1572325888 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:46:57.602 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.911+0000 7faa3e59c700 1 -- 172.21.15.169:0/1572325888 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa40108b20 con 0x7faa40107fb0 2024-03-28T00:46:57.602 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.911+0000 7faa3e59c700 1 --2- 172.21.15.169:0/1572325888 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa40107fb0 0x7faa401083d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7faa300098a0 tx=0x7faa30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d45e3155167c2be5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:46:57.602 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.911+0000 7faa3d59a700 1 -- 172.21.15.169:0/1572325888 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa30004160 con 0x7faa40107fb0 2024-03-28T00:46:57.602 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.911+0000 7faa3d59a700 1 -- 172.21.15.169:0/1572325888 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa30004300 con 0x7faa40107fb0 2024-03-28T00:46:57.602 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.911+0000 7faa3d59a700 1 -- 172.21.15.169:0/1572325888 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa30011b50 con 0x7faa40107fb0 2024-03-28T00:46:57.602 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.912+0000 7faa44c53700 1 -- 172.21.15.169:0/1572325888 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa40107fb0 msgr2=0x7faa401083d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:46:57.602 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.912+0000 7faa44c53700 1 --2- 172.21.15.169:0/1572325888 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa40107fb0 0x7faa401083d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7faa300098a0 tx=0x7faa30009bb0 comp rx=0 tx=0).stop 2024-03-28T00:46:57.602 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.912+0000 7faa44c53700 1 -- 172.21.15.169:0/1572325888 shutdown_connections 2024-03-28T00:46:57.602 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.912+0000 7faa44c53700 1 --2- 172.21.15.169:0/1572325888 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa40107fb0 0x7faa401083d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:46:57.602 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.912+0000 7faa44c53700 1 -- 172.21.15.169:0/1572325888 >> 172.21.15.169:0/1572325888 conn(0x7faa40101780 msgr2=0x7faa40103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:46:57.602 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.912+0000 7faa44c53700 1 -- 172.21.15.169:0/1572325888 shutdown_connections 2024-03-28T00:46:57.602 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.912+0000 7faa44c53700 1 -- 172.21.15.169:0/1572325888 wait complete. 2024-03-28T00:46:57.602 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.913+0000 7faa44c53700 1 Processor -- start 2024-03-28T00:46:57.602 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.913+0000 7faa44c53700 1 -- start start 2024-03-28T00:46:57.603 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.913+0000 7faa44c53700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa40107fb0 0x7faa4019dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:46:57.603 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.913+0000 7faa44c53700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa401068b0 con 0x7faa40107fb0 2024-03-28T00:46:57.603 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.913+0000 7faa3e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa40107fb0 0x7faa4019dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:46:57.603 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.913+0000 7faa3e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa40107fb0 0x7faa4019dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53740/0 (socket says 172.21.15.169:53740) 2024-03-28T00:46:57.603 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.913+0000 7faa3e59c700 1 -- 172.21.15.169:0/1279602448 learned_addr learned my addr 172.21.15.169:0/1279602448 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:46:57.603 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.913+0000 7faa3e59c700 1 -- 172.21.15.169:0/1279602448 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa4019e190 con 0x7faa40107fb0 2024-03-28T00:46:57.603 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.914+0000 7faa3e59c700 1 --2- 172.21.15.169:0/1279602448 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa40107fb0 0x7faa4019dc50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7faa30011b60 tx=0x7faa30011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:46:57.603 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.914+0000 7faa2f7fe700 1 -- 172.21.15.169:0/1279602448 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa30003970 con 0x7faa40107fb0 2024-03-28T00:46:57.603 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.914+0000 7faa2f7fe700 1 -- 172.21.15.169:0/1279602448 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa30003b10 con 0x7faa40107fb0 2024-03-28T00:46:57.603 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.914+0000 7faa2f7fe700 1 -- 172.21.15.169:0/1279602448 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa30024610 con 0x7faa40107fb0 2024-03-28T00:46:57.603 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.914+0000 7faa44c53700 1 -- 172.21.15.169:0/1279602448 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa4019e480 con 0x7faa40107fb0 2024-03-28T00:46:57.603 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.914+0000 7faa2f7fe700 1 -- 172.21.15.169:0/1279602448 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7faa30024c50 con 0x7faa40107fb0 2024-03-28T00:46:57.603 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.915+0000 7faa44c53700 1 -- 172.21.15.169:0/1279602448 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa401a1110 con 0x7faa40107fb0 2024-03-28T00:46:57.604 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.915+0000 7faa2f7fe700 1 -- 172.21.15.169:0/1279602448 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7faa30011e00 con 0x7faa40107fb0 2024-03-28T00:46:57.604 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.915+0000 7faa44c53700 1 -- 172.21.15.169:0/1279602448 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa400637c0 con 0x7faa40107fb0 2024-03-28T00:46:57.604 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.917+0000 7faa2f7fe700 1 -- 172.21.15.169:0/1279602448 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7faa3001a430 con 0x7faa40107fb0 2024-03-28T00:46:57.604 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.989+0000 7faa44c53700 1 -- 172.21.15.169:0/1279602448 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7faa401a14a0 con 0x7faa40107fb0 2024-03-28T00:46:57.604 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.990+0000 7faa2f7fe700 1 -- 172.21.15.169:0/1279602448 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7faa401a14a0 con 0x7faa40107fb0 2024-03-28T00:46:57.604 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.990+0000 7faa44c53700 1 -- 172.21.15.169:0/1279602448 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa40107fb0 msgr2=0x7faa4019dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:46:57.604 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.990+0000 7faa44c53700 1 --2- 172.21.15.169:0/1279602448 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa40107fb0 0x7faa4019dc50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7faa30011b60 tx=0x7faa30011c40 comp rx=0 tx=0).stop 2024-03-28T00:46:57.604 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.991+0000 7faa44c53700 1 -- 172.21.15.169:0/1279602448 shutdown_connections 2024-03-28T00:46:57.604 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.991+0000 7faa44c53700 1 --2- 172.21.15.169:0/1279602448 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa40107fb0 0x7faa4019dc50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:46:57.604 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.991+0000 7faa44c53700 1 -- 172.21.15.169:0/1279602448 >> 172.21.15.169:0/1279602448 conn(0x7faa40101780 msgr2=0x7faa40194290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:46:57.604 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.991+0000 7faa44c53700 1 -- 172.21.15.169:0/1279602448 shutdown_connections 2024-03-28T00:46:57.604 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:46:56.991+0000 7faa44c53700 1 -- 172.21.15.169:0/1279602448 wait complete. 2024-03-28T00:46:57.604 INFO:teuthology.orchestra.run.smithi169.stdout:mgr not available, waiting (3/15)... 2024-03-28T00:46:58.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:58 smithi169 ceph-mon[32274]: Activating manager daemon x 2024-03-28T00:46:58.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:58 smithi169 ceph-mon[32274]: mgrmap e2: x(active, starting, since 0.00314245s) 2024-03-28T00:46:58.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:58 smithi169 ceph-mon[32274]: from='mgr.14100 172.21.15.169:0/2379641204' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T00:46:58.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:58 smithi169 ceph-mon[32274]: from='mgr.14100 172.21.15.169:0/2379641204' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T00:46:58.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:58 smithi169 ceph-mon[32274]: from='mgr.14100 172.21.15.169:0/2379641204' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T00:46:58.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:58 smithi169 ceph-mon[32274]: from='mgr.14100 172.21.15.169:0/2379641204' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T00:46:58.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:58 smithi169 ceph-mon[32274]: from='mgr.14100 172.21.15.169:0/2379641204' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T00:46:58.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:58 smithi169 ceph-mon[32274]: Manager daemon x is now available 2024-03-28T00:46:58.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:58 smithi169 ceph-mon[32274]: from='mgr.14100 172.21.15.169:0/2379641204' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T00:46:58.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:58 smithi169 ceph-mon[32274]: from='mgr.14100 172.21.15.169:0/2379641204' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T00:46:58.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:58 smithi169 ceph-mon[32274]: from='mgr.14100 172.21.15.169:0/2379641204' entity='mgr.x' 2024-03-28T00:46:58.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:58 smithi169 ceph-mon[32274]: from='mgr.14100 172.21.15.169:0/2379641204' entity='mgr.x' 2024-03-28T00:46:58.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:58 smithi169 ceph-mon[32274]: from='mgr.14100 172.21.15.169:0/2379641204' entity='mgr.x' 2024-03-28T00:46:59.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:46:59 smithi169 ceph-mon[32274]: mgrmap e3: x(active, since 1.00956s) 2024-03-28T00:47:00.847 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-28T00:47:00.847 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout { 2024-03-28T00:47:00.848 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "fsid": "833368da-ec9c-11ee-95d0-87774f69a715", 2024-03-28T00:47:00.848 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T00:47:00.848 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T00:47:00.848 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T00:47:00.848 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T00:47:00.848 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:47:00.848 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T00:47:00.848 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T00:47:00.848 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 0 2024-03-28T00:47:00.848 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ], 2024-03-28T00:47:00.848 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T00:47:00.848 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T00:47:00.848 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ], 2024-03-28T00:47:00.848 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-28T00:47:00.848 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T00:47:00.849 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T00:47:00.849 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T00:47:00.849 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T00:47:00.849 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:47:00.849 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T00:47:00.849 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T00:47:00.849 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T00:47:00.849 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T00:47:00.849 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T00:47:00.857 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T00:47:00.858 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T00:47:00.858 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T00:47:00.858 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:47:00.858 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T00:47:00.858 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T00:47:00.858 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T00:47:00.858 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T00:47:00.858 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T00:47:00.858 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T00:47:00.858 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T00:47:00.858 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T00:47:00.859 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T00:47:00.859 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:47:00.859 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T00:47:00.859 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T00:47:00.859 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T00:47:00.859 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T00:47:00.859 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:47:00.859 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T00:47:00.859 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T00:47:00.859 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T00:47:00.859 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T00:47:00.859 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T00:47:00.859 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T00:47:00.859 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T00:47:00.860 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ], 2024-03-28T00:47:00.860 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T00:47:00.860 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:47:00.860 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T00:47:00.860 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T00:47:00.860 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T00:46:42.432708+0000", 2024-03-28T00:47:00.860 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T00:47:00.860 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout }, 2024-03-28T00:47:00.860 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T00:47:00.860 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout } 2024-03-28T00:47:00.860 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.004+0000 7f4c2955c700 1 Processor -- start 2024-03-28T00:47:00.860 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.004+0000 7f4c2955c700 1 -- start start 2024-03-28T00:47:00.860 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.004+0000 7f4c2955c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c24107af0 0x7f4c24107f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:00.860 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.004+0000 7f4c2955c700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c241084e0 con 0x7f4c24107af0 2024-03-28T00:47:00.861 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.005+0000 7f4c22ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c24107af0 0x7f4c24107f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:00.861 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.005+0000 7f4c22ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c24107af0 0x7f4c24107f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53798/0 (socket says 172.21.15.169:53798) 2024-03-28T00:47:00.861 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.005+0000 7f4c22ffd700 1 -- 172.21.15.169:0/296052984 learned_addr learned my addr 172.21.15.169:0/296052984 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:00.861 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.005+0000 7f4c22ffd700 1 -- 172.21.15.169:0/296052984 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c24108cf0 con 0x7f4c24107af0 2024-03-28T00:47:00.861 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.005+0000 7f4c22ffd700 1 --2- 172.21.15.169:0/296052984 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c24107af0 0x7f4c24107f10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c009e80 tx=0x7f4c1c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7dc246fac4b56b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:00.861 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.005+0000 7f4c21ffb700 1 -- 172.21.15.169:0/296052984 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4c1c017070 con 0x7f4c24107af0 2024-03-28T00:47:00.861 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.005+0000 7f4c21ffb700 1 -- 172.21.15.169:0/296052984 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c1c00b8c0 con 0x7f4c24107af0 2024-03-28T00:47:00.861 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.005+0000 7f4c21ffb700 1 -- 172.21.15.169:0/296052984 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4c1c0045c0 con 0x7f4c24107af0 2024-03-28T00:47:00.861 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.006+0000 7f4c2955c700 1 -- 172.21.15.169:0/296052984 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c24107af0 msgr2=0x7f4c24107f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:00.861 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.006+0000 7f4c2955c700 1 --2- 172.21.15.169:0/296052984 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c24107af0 0x7f4c24107f10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c009e80 tx=0x7f4c1c00b330 comp rx=0 tx=0).stop 2024-03-28T00:47:00.861 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.006+0000 7f4c2955c700 1 -- 172.21.15.169:0/296052984 shutdown_connections 2024-03-28T00:47:00.861 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.006+0000 7f4c2955c700 1 --2- 172.21.15.169:0/296052984 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c24107af0 0x7f4c24107f10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:00.861 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.006+0000 7f4c2955c700 1 -- 172.21.15.169:0/296052984 >> 172.21.15.169:0/296052984 conn(0x7f4c24103070 msgr2=0x7f4c241054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:00.862 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.006+0000 7f4c2955c700 1 -- 172.21.15.169:0/296052984 shutdown_connections 2024-03-28T00:47:00.862 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.006+0000 7f4c2955c700 1 -- 172.21.15.169:0/296052984 wait complete. 2024-03-28T00:47:00.862 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.006+0000 7f4c2955c700 1 Processor -- start 2024-03-28T00:47:00.862 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.007+0000 7f4c2955c700 1 -- start start 2024-03-28T00:47:00.862 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.007+0000 7f4c2955c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c24107af0 0x7f4c2419ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:00.862 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.007+0000 7f4c2955c700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2410c0a0 con 0x7f4c24107af0 2024-03-28T00:47:00.862 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.007+0000 7f4c22ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c24107af0 0x7f4c2419ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:00.865 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.007+0000 7f4c22ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c24107af0 0x7f4c2419ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53806/0 (socket says 172.21.15.169:53806) 2024-03-28T00:47:00.865 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.007+0000 7f4c22ffd700 1 -- 172.21.15.169:0/1302482539 learned_addr learned my addr 172.21.15.169:0/1302482539 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:00.865 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.007+0000 7f4c22ffd700 1 -- 172.21.15.169:0/1302482539 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2419bfc0 con 0x7f4c24107af0 2024-03-28T00:47:00.865 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.007+0000 7f4c22ffd700 1 --2- 172.21.15.169:0/1302482539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c24107af0 0x7f4c2419ba80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c0072a0 tx=0x7f4c1c00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:00.865 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.008+0000 7f4c0bfff700 1 -- 172.21.15.169:0/1302482539 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4c1c020020 con 0x7f4c24107af0 2024-03-28T00:47:00.866 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.008+0000 7f4c0bfff700 1 -- 172.21.15.169:0/1302482539 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c1c003b50 con 0x7f4c24107af0 2024-03-28T00:47:00.866 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.008+0000 7f4c2955c700 1 -- 172.21.15.169:0/1302482539 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c2419c250 con 0x7f4c24107af0 2024-03-28T00:47:00.866 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.008+0000 7f4c0bfff700 1 -- 172.21.15.169:0/1302482539 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4c1c017040 con 0x7f4c24107af0 2024-03-28T00:47:00.866 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.008+0000 7f4c2955c700 1 -- 172.21.15.169:0/1302482539 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c2419ef40 con 0x7f4c24107af0 2024-03-28T00:47:00.866 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.008+0000 7f4c0bfff700 1 -- 172.21.15.169:0/1302482539 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 3) v1 ==== 45989+0+0 (secure 0 0 0) 0x7f4c1c023020 con 0x7f4c24107af0 2024-03-28T00:47:00.866 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.009+0000 7f4c0bfff700 1 --2- 172.21.15.169:0/1302482539 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7f4c0c038d50 0x7f4c0c03b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:00.866 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.009+0000 7f4c2955c700 1 -- 172.21.15.169:0/1302482539 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c240637c0 con 0x7f4c24107af0 2024-03-28T00:47:00.866 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.009+0000 7f4c0bfff700 1 -- 172.21.15.169:0/1302482539 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4c1c00f050 con 0x7f4c24107af0 2024-03-28T00:47:00.866 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.009+0000 7f4c227fc700 1 --2- 172.21.15.169:0/1302482539 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7f4c0c038d50 0x7f4c0c03b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:00.866 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.009+0000 7f4c227fc700 1 --2- 172.21.15.169:0/1302482539 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7f4c0c038d50 0x7f4c0c03b210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4c18006fd0 tx=0x7f4c18006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:00.866 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.014+0000 7f4c0bfff700 1 -- 172.21.15.169:0/1302482539 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4c1c02a020 con 0x7f4c24107af0 2024-03-28T00:47:00.866 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.236+0000 7f4c0bfff700 1 -- 172.21.15.169:0/1302482539 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f4c1c025400 con 0x7f4c24107af0 2024-03-28T00:47:00.866 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.287+0000 7f4c2955c700 1 -- 172.21.15.169:0/1302482539 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4c2419f1b0 con 0x7f4c24107af0 2024-03-28T00:47:00.867 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.288+0000 7f4c0bfff700 1 -- 172.21.15.169:0/1302482539 <== mon.0 v2:172.21.15.169:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f4c1c01b370 con 0x7f4c24107af0 2024-03-28T00:47:00.867 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.290+0000 7f4c2955c700 1 -- 172.21.15.169:0/1302482539 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7f4c0c038d50 msgr2=0x7f4c0c03b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:00.867 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.290+0000 7f4c2955c700 1 --2- 172.21.15.169:0/1302482539 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7f4c0c038d50 0x7f4c0c03b210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4c18006fd0 tx=0x7f4c18006e40 comp rx=0 tx=0).stop 2024-03-28T00:47:00.867 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.290+0000 7f4c2955c700 1 -- 172.21.15.169:0/1302482539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c24107af0 msgr2=0x7f4c2419ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:00.867 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.290+0000 7f4c2955c700 1 --2- 172.21.15.169:0/1302482539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c24107af0 0x7f4c2419ba80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c0072a0 tx=0x7f4c1c00bd50 comp rx=0 tx=0).stop 2024-03-28T00:47:00.867 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.290+0000 7f4c2955c700 1 -- 172.21.15.169:0/1302482539 shutdown_connections 2024-03-28T00:47:00.867 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.290+0000 7f4c2955c700 1 --2- 172.21.15.169:0/1302482539 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7f4c0c038d50 0x7f4c0c03b210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:00.867 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.290+0000 7f4c2955c700 1 --2- 172.21.15.169:0/1302482539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c24107af0 0x7f4c2419ba80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:00.867 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.290+0000 7f4c2955c700 1 -- 172.21.15.169:0/1302482539 >> 172.21.15.169:0/1302482539 conn(0x7f4c24103070 msgr2=0x7f4c24103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:00.867 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.290+0000 7f4c2955c700 1 -- 172.21.15.169:0/1302482539 shutdown_connections 2024-03-28T00:47:00.867 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:00.290+0000 7f4c2955c700 1 -- 172.21.15.169:0/1302482539 wait complete. 2024-03-28T00:47:00.867 INFO:teuthology.orchestra.run.smithi169.stdout:mgr is available 2024-03-28T00:47:01.496 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:01 smithi169 ceph-mon[32274]: mgrmap e4: x(active, since 3s) 2024-03-28T00:47:01.496 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:01 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1302482539' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T00:47:02.001 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-28T00:47:02.001 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T00:47:02.001 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout fsid = 833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T00:47:02.001 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T00:47:02.001 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T00:47:02.001 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T00:47:02.001 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T00:47:02.001 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-28T00:47:02.001 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T00:47:02.001 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T00:47:02.001 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 2024-03-28T00:47:02.002 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T00:47:02.002 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T00:47:02.002 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T00:47:02.002 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T00:47:02.002 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T00:47:02.002 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T00:47:02.002 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.274+0000 7f94e7a56700 1 Processor -- start 2024-03-28T00:47:02.002 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.275+0000 7f94e7a56700 1 -- start start 2024-03-28T00:47:02.002 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.275+0000 7f94e7a56700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94e0105e30 0x7f94e0106250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:02.002 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.275+0000 7f94e7a56700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e0106820 con 0x7f94e0105e30 2024-03-28T00:47:02.002 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.275+0000 7f94e57f2700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94e0105e30 0x7f94e0106250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:02.002 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.275+0000 7f94e57f2700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94e0105e30 0x7f94e0106250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53818/0 (socket says 172.21.15.169:53818) 2024-03-28T00:47:02.002 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.275+0000 7f94e57f2700 1 -- 172.21.15.169:0/974086004 learned_addr learned my addr 172.21.15.169:0/974086004 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:02.002 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.275+0000 7f94e57f2700 1 -- 172.21.15.169:0/974086004 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e0108dc0 con 0x7f94e0105e30 2024-03-28T00:47:02.003 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.275+0000 7f94e57f2700 1 --2- 172.21.15.169:0/974086004 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94e0105e30 0x7f94e0106250 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f94d00098a0 tx=0x7f94d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf905672dbf93bb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:02.003 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.276+0000 7f94d7fff700 1 -- 172.21.15.169:0/974086004 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f94d0004160 con 0x7f94e0105e30 2024-03-28T00:47:02.003 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.276+0000 7f94d7fff700 1 -- 172.21.15.169:0/974086004 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f94d0004300 con 0x7f94e0105e30 2024-03-28T00:47:02.003 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.276+0000 7f94d7fff700 1 -- 172.21.15.169:0/974086004 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f94d0011b50 con 0x7f94e0105e30 2024-03-28T00:47:02.003 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.276+0000 7f94e7a56700 1 -- 172.21.15.169:0/974086004 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94e0105e30 msgr2=0x7f94e0106250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:02.003 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.276+0000 7f94e7a56700 1 --2- 172.21.15.169:0/974086004 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94e0105e30 0x7f94e0106250 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f94d00098a0 tx=0x7f94d0009bb0 comp rx=0 tx=0).stop 2024-03-28T00:47:02.003 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.276+0000 7f94e7a56700 1 -- 172.21.15.169:0/974086004 shutdown_connections 2024-03-28T00:47:02.003 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.276+0000 7f94e7a56700 1 --2- 172.21.15.169:0/974086004 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94e0105e30 0x7f94e0106250 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:02.003 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.276+0000 7f94e7a56700 1 -- 172.21.15.169:0/974086004 >> 172.21.15.169:0/974086004 conn(0x7f94e01017a0 msgr2=0x7f94e0103bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:02.003 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.276+0000 7f94e7a56700 1 -- 172.21.15.169:0/974086004 shutdown_connections 2024-03-28T00:47:02.003 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.276+0000 7f94e7a56700 1 -- 172.21.15.169:0/974086004 wait complete. 2024-03-28T00:47:02.003 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.277+0000 7f94e7a56700 1 Processor -- start 2024-03-28T00:47:02.003 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.277+0000 7f94e7a56700 1 -- start start 2024-03-28T00:47:02.004 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.277+0000 7f94e7a56700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94e0105e30 0x7f94e019bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:02.004 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.277+0000 7f94e7a56700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e010c170 con 0x7f94e0105e30 2024-03-28T00:47:02.004 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.277+0000 7f94e57f2700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94e0105e30 0x7f94e019bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:02.004 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.277+0000 7f94e57f2700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94e0105e30 0x7f94e019bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53824/0 (socket says 172.21.15.169:53824) 2024-03-28T00:47:02.004 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.277+0000 7f94e57f2700 1 -- 172.21.15.169:0/2483212215 learned_addr learned my addr 172.21.15.169:0/2483212215 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:02.004 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.277+0000 7f94e57f2700 1 -- 172.21.15.169:0/2483212215 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e019c0b0 con 0x7f94e0105e30 2024-03-28T00:47:02.004 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.278+0000 7f94e57f2700 1 --2- 172.21.15.169:0/2483212215 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94e0105e30 0x7f94e019bb70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f94d0009fb0 tx=0x7f94d0011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:02.004 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.278+0000 7f94d67fc700 1 -- 172.21.15.169:0/2483212215 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f94d00038b0 con 0x7f94e0105e30 2024-03-28T00:47:02.004 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.278+0000 7f94d67fc700 1 -- 172.21.15.169:0/2483212215 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f94d0011e50 con 0x7f94e0105e30 2024-03-28T00:47:02.004 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.278+0000 7f94e7a56700 1 -- 172.21.15.169:0/2483212215 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e019c340 con 0x7f94e0105e30 2024-03-28T00:47:02.004 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.278+0000 7f94d67fc700 1 -- 172.21.15.169:0/2483212215 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f94d0019550 con 0x7f94e0105e30 2024-03-28T00:47:02.004 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.278+0000 7f94e7a56700 1 -- 172.21.15.169:0/2483212215 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e019f030 con 0x7f94e0105e30 2024-03-28T00:47:02.004 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.279+0000 7f94d67fc700 1 -- 172.21.15.169:0/2483212215 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f94d0019800 con 0x7f94e0105e30 2024-03-28T00:47:02.004 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.279+0000 7f94e7a56700 1 -- 172.21.15.169:0/2483212215 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e00637c0 con 0x7f94e0105e30 2024-03-28T00:47:02.005 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.279+0000 7f94d67fc700 1 --2- 172.21.15.169:0/2483212215 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7f94cc038ed0 0x7f94cc03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:02.005 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.279+0000 7f94d67fc700 1 -- 172.21.15.169:0/2483212215 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f94d004d0c0 con 0x7f94e0105e30 2024-03-28T00:47:02.005 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.279+0000 7f94e4ff1700 1 --2- 172.21.15.169:0/2483212215 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7f94cc038ed0 0x7f94cc03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:02.005 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.280+0000 7f94e4ff1700 1 --2- 172.21.15.169:0/2483212215 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7f94cc038ed0 0x7f94cc03b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f94dc006fd0 tx=0x7f94dc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:02.005 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.284+0000 7f94d67fc700 1 -- 172.21.15.169:0/2483212215 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f94d000bc40 con 0x7f94e0105e30 2024-03-28T00:47:02.005 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.478+0000 7f94e7a56700 1 -- 172.21.15.169:0/2483212215 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f94e019f200 con 0x7f94e0105e30 2024-03-28T00:47:02.005 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.479+0000 7f94d67fc700 1 -- 172.21.15.169:0/2483212215 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f94e019f200 con 0x7f94e0105e30 2024-03-28T00:47:02.005 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.481+0000 7f94e7a56700 1 -- 172.21.15.169:0/2483212215 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7f94cc038ed0 msgr2=0x7f94cc03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:02.005 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.481+0000 7f94e7a56700 1 --2- 172.21.15.169:0/2483212215 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7f94cc038ed0 0x7f94cc03b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f94dc006fd0 tx=0x7f94dc006e40 comp rx=0 tx=0).stop 2024-03-28T00:47:02.005 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.481+0000 7f94e7a56700 1 -- 172.21.15.169:0/2483212215 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94e0105e30 msgr2=0x7f94e019bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:02.005 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.481+0000 7f94e7a56700 1 --2- 172.21.15.169:0/2483212215 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94e0105e30 0x7f94e019bb70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f94d0009fb0 tx=0x7f94d0011bf0 comp rx=0 tx=0).stop 2024-03-28T00:47:02.005 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.482+0000 7f94e7a56700 1 -- 172.21.15.169:0/2483212215 shutdown_connections 2024-03-28T00:47:02.005 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.482+0000 7f94e7a56700 1 --2- 172.21.15.169:0/2483212215 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7f94cc038ed0 0x7f94cc03b390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:02.006 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.482+0000 7f94e7a56700 1 --2- 172.21.15.169:0/2483212215 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94e0105e30 0x7f94e019bb70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:02.006 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.482+0000 7f94e7a56700 1 -- 172.21.15.169:0/2483212215 >> 172.21.15.169:0/2483212215 conn(0x7f94e01017a0 msgr2=0x7f94e0102300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:02.006 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.482+0000 7f94e7a56700 1 -- 172.21.15.169:0/2483212215 shutdown_connections 2024-03-28T00:47:02.006 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:01.482+0000 7f94e7a56700 1 -- 172.21.15.169:0/2483212215 wait complete. 2024-03-28T00:47:02.006 INFO:teuthology.orchestra.run.smithi169.stdout:Enabling cephadm module... 2024-03-28T00:47:02.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:02 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2483212215' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-28T00:47:03.539 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:03 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2744836259' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-28T00:47:03.539 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:03 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: ignoring --setuser ceph since I am not root 2024-03-28T00:47:03.539 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:03 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: ignoring --setgroup ceph since I am not root 2024-03-28T00:47:03.773 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.375+0000 7f96650b1700 1 Processor -- start 2024-03-28T00:47:03.773 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.375+0000 7f96650b1700 1 -- start start 2024-03-28T00:47:03.773 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.375+0000 7f96650b1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9660108020 0x7f9660108440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:03.773 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.375+0000 7f96650b1700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9660108a10 con 0x7f9660108020 2024-03-28T00:47:03.773 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.375+0000 7f965ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9660108020 0x7f9660108440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:03.773 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.375+0000 7f965ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9660108020 0x7f9660108440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53832/0 (socket says 172.21.15.169:53832) 2024-03-28T00:47:03.773 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.375+0000 7f965ed9d700 1 -- 172.21.15.169:0/2188206060 learned_addr learned my addr 172.21.15.169:0/2188206060 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:03.773 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.375+0000 7f965ed9d700 1 -- 172.21.15.169:0/2188206060 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9660106c80 con 0x7f9660108020 2024-03-28T00:47:03.773 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.375+0000 7f965ed9d700 1 --2- 172.21.15.169:0/2188206060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9660108020 0x7f9660108440 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9658009e80 tx=0x7f965800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=340ed300a242f857 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:03.773 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.375+0000 7f965dd9b700 1 -- 172.21.15.169:0/2188206060 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9658017070 con 0x7f9660108020 2024-03-28T00:47:03.773 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.376+0000 7f965dd9b700 1 -- 172.21.15.169:0/2188206060 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f965800b8c0 con 0x7f9660108020 2024-03-28T00:47:03.774 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.376+0000 7f965dd9b700 1 -- 172.21.15.169:0/2188206060 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f96580045c0 con 0x7f9660108020 2024-03-28T00:47:03.774 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.376+0000 7f96650b1700 1 -- 172.21.15.169:0/2188206060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9660108020 msgr2=0x7f9660108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:03.774 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.376+0000 7f96650b1700 1 --2- 172.21.15.169:0/2188206060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9660108020 0x7f9660108440 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9658009e80 tx=0x7f965800b330 comp rx=0 tx=0).stop 2024-03-28T00:47:03.774 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.376+0000 7f96650b1700 1 -- 172.21.15.169:0/2188206060 shutdown_connections 2024-03-28T00:47:03.774 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.376+0000 7f96650b1700 1 --2- 172.21.15.169:0/2188206060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9660108020 0x7f9660108440 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:03.774 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.376+0000 7f96650b1700 1 -- 172.21.15.169:0/2188206060 >> 172.21.15.169:0/2188206060 conn(0x7f96601017d0 msgr2=0x7f9660103bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:03.774 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.376+0000 7f96650b1700 1 -- 172.21.15.169:0/2188206060 shutdown_connections 2024-03-28T00:47:03.774 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.376+0000 7f96650b1700 1 -- 172.21.15.169:0/2188206060 wait complete. 2024-03-28T00:47:03.774 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.377+0000 7f96650b1700 1 Processor -- start 2024-03-28T00:47:03.774 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.377+0000 7f96650b1700 1 -- start start 2024-03-28T00:47:03.774 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.377+0000 7f96650b1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9660108020 0x7f96601065f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:03.774 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.377+0000 7f96650b1700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966010b040 con 0x7f9660108020 2024-03-28T00:47:03.774 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.377+0000 7f965ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9660108020 0x7f96601065f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:03.774 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.377+0000 7f965ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9660108020 0x7f96601065f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53846/0 (socket says 172.21.15.169:53846) 2024-03-28T00:47:03.774 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.377+0000 7f965ed9d700 1 -- 172.21.15.169:0/2744836259 learned_addr learned my addr 172.21.15.169:0/2744836259 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:03.775 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.378+0000 7f965ed9d700 1 -- 172.21.15.169:0/2744836259 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966019e8a0 con 0x7f9660108020 2024-03-28T00:47:03.775 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.378+0000 7f965ed9d700 1 --2- 172.21.15.169:0/2744836259 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9660108020 0x7f96601065f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9658009e50 tx=0x7f965800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:03.775 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.378+0000 7f9657fff700 1 -- 172.21.15.169:0/2744836259 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9658020020 con 0x7f9660108020 2024-03-28T00:47:03.775 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.378+0000 7f9657fff700 1 -- 172.21.15.169:0/2744836259 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9658003b50 con 0x7f9660108020 2024-03-28T00:47:03.775 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.378+0000 7f96650b1700 1 -- 172.21.15.169:0/2744836259 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f966019ea70 con 0x7f9660108020 2024-03-28T00:47:03.775 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.378+0000 7f9657fff700 1 -- 172.21.15.169:0/2744836259 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9658017040 con 0x7f9660108020 2024-03-28T00:47:03.775 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.378+0000 7f96650b1700 1 -- 172.21.15.169:0/2744836259 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f966019ee90 con 0x7f9660108020 2024-03-28T00:47:03.775 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.379+0000 7f9657fff700 1 -- 172.21.15.169:0/2744836259 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f9658023020 con 0x7f9660108020 2024-03-28T00:47:03.775 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.379+0000 7f9657fff700 1 --2- 172.21.15.169:0/2744836259 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7f964c038e80 0x7f964c03b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:03.775 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.379+0000 7f9657fff700 1 -- 172.21.15.169:0/2744836259 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f965800f050 con 0x7f9660108020 2024-03-28T00:47:03.775 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.379+0000 7f965e59c700 1 --2- 172.21.15.169:0/2744836259 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7f964c038e80 0x7f964c03b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:03.775 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.379+0000 7f96650b1700 1 -- 172.21.15.169:0/2744836259 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96600637c0 con 0x7f9660108020 2024-03-28T00:47:03.775 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.379+0000 7f965e59c700 1 --2- 172.21.15.169:0/2744836259 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7f964c038e80 0x7f964c03b340 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9648006fd0 tx=0x7f9648006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:03.776 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.384+0000 7f9657fff700 1 -- 172.21.15.169:0/2744836259 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f965802a020 con 0x7f9660108020 2024-03-28T00:47:03.776 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:02.634+0000 7f96650b1700 1 -- 172.21.15.169:0/2744836259 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f966019f530 con 0x7f9660108020 2024-03-28T00:47:03.776 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:03.239+0000 7f9657fff700 1 -- 172.21.15.169:0/2744836259 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f965804dc70 con 0x7f9660108020 2024-03-28T00:47:03.776 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:03.241+0000 7f9657fff700 1 -- 172.21.15.169:0/2744836259 <== mon.0 v2:172.21.15.169:3300/0 8 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f965801bcb0 con 0x7f9660108020 2024-03-28T00:47:03.776 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:03.243+0000 7f96650b1700 1 -- 172.21.15.169:0/2744836259 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7f964c038e80 msgr2=0x7f964c03b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:03.776 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:03.243+0000 7f96650b1700 1 --2- 172.21.15.169:0/2744836259 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7f964c038e80 0x7f964c03b340 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9648006fd0 tx=0x7f9648006e40 comp rx=0 tx=0).stop 2024-03-28T00:47:03.776 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:03.243+0000 7f96650b1700 1 -- 172.21.15.169:0/2744836259 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9660108020 msgr2=0x7f96601065f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:03.776 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:03.243+0000 7f96650b1700 1 --2- 172.21.15.169:0/2744836259 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9660108020 0x7f96601065f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9658009e50 tx=0x7f965800bd30 comp rx=0 tx=0).stop 2024-03-28T00:47:03.776 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:03.243+0000 7f96650b1700 1 -- 172.21.15.169:0/2744836259 shutdown_connections 2024-03-28T00:47:03.776 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:03.243+0000 7f96650b1700 1 --2- 172.21.15.169:0/2744836259 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7f964c038e80 0x7f964c03b340 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:03.776 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:03.243+0000 7f96650b1700 1 --2- 172.21.15.169:0/2744836259 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9660108020 0x7f96601065f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:03.776 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:03.243+0000 7f96650b1700 1 -- 172.21.15.169:0/2744836259 >> 172.21.15.169:0/2744836259 conn(0x7f96601017d0 msgr2=0x7f9660102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:03.776 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:03.243+0000 7f96650b1700 1 -- 172.21.15.169:0/2744836259 shutdown_connections 2024-03-28T00:47:03.776 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:03.244+0000 7f96650b1700 1 -- 172.21.15.169:0/2744836259 wait complete. 2024-03-28T00:47:04.092 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:03 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:03.804+0000 7f2c92c09200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T00:47:04.093 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:03 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:03.899+0000 7f2c92c09200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T00:47:04.397 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:04 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2744836259' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-28T00:47:04.397 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:04 smithi169 ceph-mon[32274]: mgrmap e5: x(active, since 6s) 2024-03-28T00:47:04.655 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:04 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:04.558+0000 7f2c92c09200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T00:47:04.655 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:04 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:04.651+0000 7f2c92c09200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T00:47:04.921 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:04 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:04.793+0000 7f2c92c09200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T00:47:04.921 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:04 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:04.890+0000 7f2c92c09200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T00:47:05.165 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout { 2024-03-28T00:47:05.165 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-28T00:47:05.165 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T00:47:05.165 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T00:47:05.165 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T00:47:05.165 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout } 2024-03-28T00:47:05.165 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.188+0000 7ff2eab98700 1 Processor -- start 2024-03-28T00:47:05.165 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.188+0000 7ff2eab98700 1 -- start start 2024-03-28T00:47:05.165 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.188+0000 7ff2eab98700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2e4107b50 0x7ff2e4107f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:05.166 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.188+0000 7ff2eab98700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e4108540 con 0x7ff2e4107b50 2024-03-28T00:47:05.166 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.188+0000 7ff2e8934700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2e4107b50 0x7ff2e4107f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:05.166 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.188+0000 7ff2e8934700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2e4107b50 0x7ff2e4107f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53870/0 (socket says 172.21.15.169:53870) 2024-03-28T00:47:05.166 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.188+0000 7ff2e8934700 1 -- 172.21.15.169:0/874993339 learned_addr learned my addr 172.21.15.169:0/874993339 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:05.166 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.188+0000 7ff2e8934700 1 -- 172.21.15.169:0/874993339 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e4108d50 con 0x7ff2e4107b50 2024-03-28T00:47:05.166 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.189+0000 7ff2e8934700 1 --2- 172.21.15.169:0/874993339 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2e4107b50 0x7ff2e4107f70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff2d80098a0 tx=0x7ff2d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=111b646df64fa4ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:05.166 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.189+0000 7ff2e37fe700 1 -- 172.21.15.169:0/874993339 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff2d8004160 con 0x7ff2e4107b50 2024-03-28T00:47:05.166 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.189+0000 7ff2e37fe700 1 -- 172.21.15.169:0/874993339 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2d8004300 con 0x7ff2e4107b50 2024-03-28T00:47:05.166 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.189+0000 7ff2e37fe700 1 -- 172.21.15.169:0/874993339 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff2d8011b50 con 0x7ff2e4107b50 2024-03-28T00:47:05.166 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.189+0000 7ff2eab98700 1 -- 172.21.15.169:0/874993339 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2e4107b50 msgr2=0x7ff2e4107f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:05.166 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.189+0000 7ff2eab98700 1 --2- 172.21.15.169:0/874993339 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2e4107b50 0x7ff2e4107f70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff2d80098a0 tx=0x7ff2d8009bb0 comp rx=0 tx=0).stop 2024-03-28T00:47:05.166 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.189+0000 7ff2eab98700 1 -- 172.21.15.169:0/874993339 shutdown_connections 2024-03-28T00:47:05.167 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.189+0000 7ff2eab98700 1 --2- 172.21.15.169:0/874993339 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2e4107b50 0x7ff2e4107f70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:05.167 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.189+0000 7ff2eab98700 1 -- 172.21.15.169:0/874993339 >> 172.21.15.169:0/874993339 conn(0x7ff2e4103150 msgr2=0x7ff2e4105570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:05.167 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.189+0000 7ff2eab98700 1 -- 172.21.15.169:0/874993339 shutdown_connections 2024-03-28T00:47:05.167 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.189+0000 7ff2eab98700 1 -- 172.21.15.169:0/874993339 wait complete. 2024-03-28T00:47:05.167 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.190+0000 7ff2eab98700 1 Processor -- start 2024-03-28T00:47:05.167 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.190+0000 7ff2eab98700 1 -- start start 2024-03-28T00:47:05.167 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.190+0000 7ff2eab98700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2e4107b50 0x7ff2e419bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:05.167 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.190+0000 7ff2eab98700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e410c100 con 0x7ff2e4107b50 2024-03-28T00:47:05.167 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.190+0000 7ff2e8934700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2e4107b50 0x7ff2e419bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:05.167 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.190+0000 7ff2e8934700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2e4107b50 0x7ff2e419bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53872/0 (socket says 172.21.15.169:53872) 2024-03-28T00:47:05.167 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.190+0000 7ff2e8934700 1 -- 172.21.15.169:0/1104533184 learned_addr learned my addr 172.21.15.169:0/1104533184 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:05.167 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.191+0000 7ff2e8934700 1 -- 172.21.15.169:0/1104533184 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e419c0b0 con 0x7ff2e4107b50 2024-03-28T00:47:05.168 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.191+0000 7ff2e8934700 1 --2- 172.21.15.169:0/1104533184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2e4107b50 0x7ff2e419bb70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8009fb0 tx=0x7ff2d8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:05.168 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.191+0000 7ff2e1ffb700 1 -- 172.21.15.169:0/1104533184 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff2d80038b0 con 0x7ff2e4107b50 2024-03-28T00:47:05.168 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.191+0000 7ff2e1ffb700 1 -- 172.21.15.169:0/1104533184 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2d8011e50 con 0x7ff2e4107b50 2024-03-28T00:47:05.168 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.191+0000 7ff2eab98700 1 -- 172.21.15.169:0/1104533184 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e419c340 con 0x7ff2e4107b50 2024-03-28T00:47:05.168 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.191+0000 7ff2e1ffb700 1 -- 172.21.15.169:0/1104533184 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff2d8019550 con 0x7ff2e4107b50 2024-03-28T00:47:05.168 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.191+0000 7ff2eab98700 1 -- 172.21.15.169:0/1104533184 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e419f030 con 0x7ff2e4107b50 2024-03-28T00:47:05.168 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.192+0000 7ff2e1ffb700 1 -- 172.21.15.169:0/1104533184 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7ff2d8019850 con 0x7ff2e4107b50 2024-03-28T00:47:05.168 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.192+0000 7ff2eab98700 1 -- 172.21.15.169:0/1104533184 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e40637c0 con 0x7ff2e4107b50 2024-03-28T00:47:05.168 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.192+0000 7ff2e1ffb700 1 --2- 172.21.15.169:0/1104533184 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7ff2d4038ed0 0x7ff2d403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:05.168 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.192+0000 7ff2e1ffb700 1 -- 172.21.15.169:0/1104533184 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff2d804d000 con 0x7ff2e4107b50 2024-03-28T00:47:05.168 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.192+0000 7ff2e3fff700 1 -- 172.21.15.169:0/1104533184 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7ff2d4038ed0 msgr2=0x7ff2d403b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/311965133 2024-03-28T00:47:05.168 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.192+0000 7ff2e3fff700 1 --2- 172.21.15.169:0/1104533184 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7ff2d4038ed0 0x7ff2d403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T00:47:05.168 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.197+0000 7ff2e1ffb700 1 -- 172.21.15.169:0/1104533184 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff2d8023c90 con 0x7ff2e4107b50 2024-03-28T00:47:05.169 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.393+0000 7ff2e3fff700 1 -- 172.21.15.169:0/1104533184 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7ff2d4038ed0 msgr2=0x7ff2d403b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/311965133 2024-03-28T00:47:05.169 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.393+0000 7ff2e3fff700 1 --2- 172.21.15.169:0/1104533184 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7ff2d4038ed0 0x7ff2d403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T00:47:05.169 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.451+0000 7ff2eab98700 1 -- 172.21.15.169:0/1104533184 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7ff2e419f200 con 0x7ff2e4107b50 2024-03-28T00:47:05.169 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.451+0000 7ff2e1ffb700 1 -- 172.21.15.169:0/1104533184 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7ff2e419f200 con 0x7ff2e4107b50 2024-03-28T00:47:05.169 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.453+0000 7ff2eab98700 1 -- 172.21.15.169:0/1104533184 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7ff2d4038ed0 msgr2=0x7ff2d403b390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T00:47:05.169 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.453+0000 7ff2eab98700 1 --2- 172.21.15.169:0/1104533184 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7ff2d4038ed0 0x7ff2d403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:05.169 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.453+0000 7ff2eab98700 1 -- 172.21.15.169:0/1104533184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2e4107b50 msgr2=0x7ff2e419bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:05.169 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.453+0000 7ff2eab98700 1 --2- 172.21.15.169:0/1104533184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2e4107b50 0x7ff2e419bb70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8009fb0 tx=0x7ff2d8011bf0 comp rx=0 tx=0).stop 2024-03-28T00:47:05.169 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.453+0000 7ff2eab98700 1 -- 172.21.15.169:0/1104533184 shutdown_connections 2024-03-28T00:47:05.169 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.453+0000 7ff2eab98700 1 --2- 172.21.15.169:0/1104533184 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7ff2d4038ed0 0x7ff2d403b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:05.169 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.453+0000 7ff2eab98700 1 --2- 172.21.15.169:0/1104533184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2e4107b50 0x7ff2e419bb70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:05.169 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.453+0000 7ff2eab98700 1 -- 172.21.15.169:0/1104533184 >> 172.21.15.169:0/1104533184 conn(0x7ff2e4103150 msgr2=0x7ff2e4103ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:05.169 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.454+0000 7ff2eab98700 1 -- 172.21.15.169:0/1104533184 shutdown_connections 2024-03-28T00:47:05.169 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:04.454+0000 7ff2eab98700 1 -- 172.21.15.169:0/1104533184 wait complete. 2024-03-28T00:47:05.170 INFO:teuthology.orchestra.run.smithi169.stdout:Waiting for the mgr to restart... 2024-03-28T00:47:05.170 INFO:teuthology.orchestra.run.smithi169.stdout:Waiting for mgr epoch 5... 2024-03-28T00:47:05.244 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:04 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:04.974+0000 7f2c92c09200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T00:47:05.550 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:05 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1104533184' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T00:47:05.550 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:05 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:05.247+0000 7f2c92c09200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T00:47:05.550 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:05 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:05.361+0000 7f2c92c09200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T00:47:05.550 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:05 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:05.448+0000 7f2c92c09200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T00:47:05.802 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:05 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:05.547+0000 7f2c92c09200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T00:47:05.802 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:05 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:05.724+0000 7f2c92c09200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T00:47:06.983 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:06 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:06.680+0000 7f2c92c09200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T00:47:06.983 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:06 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:06.811+0000 7f2c92c09200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T00:47:07.248 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:07 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:07.154+0000 7f2c92c09200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T00:47:07.248 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:07 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:07.244+0000 7f2c92c09200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T00:47:07.552 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:07 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:07.442+0000 7f2c92c09200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T00:47:07.920 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:07 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:07.628+0000 7f2c92c09200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T00:47:08.180 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:07 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:07.915+0000 7f2c92c09200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T00:47:08.180 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:08 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:08.175+0000 7f2c92c09200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T00:47:08.552 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:08 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:08.413+0000 7f2c92c09200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T00:47:09.552 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:09 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:09.211+0000 7f2c92c09200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T00:47:10.133 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:09 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:09.812+0000 7f2c92c09200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T00:47:10.133 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:09 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:09.910+0000 7f2c92c09200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T00:47:10.133 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:10 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:10.127+0000 7f2c92c09200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T00:47:10.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:10 smithi169 ceph-mon[32274]: Active manager daemon x restarted 2024-03-28T00:47:10.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:10 smithi169 ceph-mon[32274]: Activating manager daemon x 2024-03-28T00:47:10.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:10 smithi169 ceph-mon[32274]: osdmap e2: 0 total, 0 up, 0 in 2024-03-28T00:47:10.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:10 smithi169 ceph-mon[32274]: mgrmap e6: x(active, starting, since 0.00324081s) 2024-03-28T00:47:10.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:10 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T00:47:10.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:10 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T00:47:10.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:10 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T00:47:10.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:10 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T00:47:10.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:10 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T00:47:10.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:10 smithi169 ceph-mon[32274]: Manager daemon x is now available 2024-03-28T00:47:10.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:10 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' 2024-03-28T00:47:10.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:10 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' 2024-03-28T00:47:10.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:10 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:47:10.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:10 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:47:11.461 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:11 smithi169 ceph-mon[32274]: Found migration_current of "None". Setting to last migration. 2024-03-28T00:47:11.461 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:11 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:47:11.461 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:11 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T00:47:11.461 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:11 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T00:47:11.461 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:11 smithi169 ceph-mon[32274]: mgrmap e7: x(active, since 1.00559s) 2024-03-28T00:47:11.747 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout { 2024-03-28T00:47:11.747 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-28T00:47:11.747 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T00:47:11.747 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout } 2024-03-28T00:47:11.747 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.571+0000 7fb25b662700 1 Processor -- start 2024-03-28T00:47:11.747 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.571+0000 7fb25b662700 1 -- start start 2024-03-28T00:47:11.747 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.572+0000 7fb25b662700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2541085b0 0x7fb25410a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:11.747 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.572+0000 7fb25b662700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25407be90 con 0x7fb2541085b0 2024-03-28T00:47:11.748 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.572+0000 7fb2593fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2541085b0 0x7fb25410a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:11.748 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.572+0000 7fb2593fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2541085b0 0x7fb25410a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53880/0 (socket says 172.21.15.169:53880) 2024-03-28T00:47:11.748 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.572+0000 7fb2593fe700 1 -- 172.21.15.169:0/273861642 learned_addr learned my addr 172.21.15.169:0/273861642 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:11.748 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.572+0000 7fb2593fe700 1 -- 172.21.15.169:0/273861642 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25410aee0 con 0x7fb2541085b0 2024-03-28T00:47:11.748 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.572+0000 7fb2593fe700 1 --2- 172.21.15.169:0/273861642 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2541085b0 0x7fb25410a9a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb2440098a0 tx=0x7fb244009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff5d4e60903edc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:11.748 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.572+0000 7fb24bfff700 1 -- 172.21.15.169:0/273861642 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb244004160 con 0x7fb2541085b0 2024-03-28T00:47:11.748 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.572+0000 7fb24bfff700 1 -- 172.21.15.169:0/273861642 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb244004300 con 0x7fb2541085b0 2024-03-28T00:47:11.748 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.572+0000 7fb24bfff700 1 -- 172.21.15.169:0/273861642 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb244011b50 con 0x7fb2541085b0 2024-03-28T00:47:11.748 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.573+0000 7fb25b662700 1 -- 172.21.15.169:0/273861642 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2541085b0 msgr2=0x7fb25410a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:11.748 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.573+0000 7fb25b662700 1 --2- 172.21.15.169:0/273861642 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2541085b0 0x7fb25410a9a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb2440098a0 tx=0x7fb244009bb0 comp rx=0 tx=0).stop 2024-03-28T00:47:11.748 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.573+0000 7fb25b662700 1 -- 172.21.15.169:0/273861642 shutdown_connections 2024-03-28T00:47:11.748 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.573+0000 7fb25b662700 1 --2- 172.21.15.169:0/273861642 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2541085b0 0x7fb25410a9a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:11.748 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.573+0000 7fb25b662700 1 -- 172.21.15.169:0/273861642 >> 172.21.15.169:0/273861642 conn(0x7fb254101f80 msgr2=0x7fb2541043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:11.748 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.573+0000 7fb25b662700 1 -- 172.21.15.169:0/273861642 shutdown_connections 2024-03-28T00:47:11.749 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.573+0000 7fb25b662700 1 -- 172.21.15.169:0/273861642 wait complete. 2024-03-28T00:47:11.749 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.574+0000 7fb25b662700 1 Processor -- start 2024-03-28T00:47:11.749 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.574+0000 7fb25b662700 1 -- start start 2024-03-28T00:47:11.749 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.574+0000 7fb25b662700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2541085b0 0x7fb25419bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:11.749 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.574+0000 7fb25b662700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25410bf00 con 0x7fb2541085b0 2024-03-28T00:47:11.749 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.574+0000 7fb2593fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2541085b0 0x7fb25419bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:11.749 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.574+0000 7fb2593fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2541085b0 0x7fb25419bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53888/0 (socket says 172.21.15.169:53888) 2024-03-28T00:47:11.749 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.574+0000 7fb2593fe700 1 -- 172.21.15.169:0/1040811296 learned_addr learned my addr 172.21.15.169:0/1040811296 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:11.749 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.575+0000 7fb2593fe700 1 -- 172.21.15.169:0/1040811296 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25419bff0 con 0x7fb2541085b0 2024-03-28T00:47:11.749 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.575+0000 7fb2593fe700 1 --2- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2541085b0 0x7fb25419bab0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb244011670 tx=0x7fb244011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:11.749 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.575+0000 7fb24a7fc700 1 -- 172.21.15.169:0/1040811296 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2440039c0 con 0x7fb2541085b0 2024-03-28T00:47:11.749 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.575+0000 7fb25b662700 1 -- 172.21.15.169:0/1040811296 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb25419c280 con 0x7fb2541085b0 2024-03-28T00:47:11.749 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.575+0000 7fb24a7fc700 1 -- 172.21.15.169:0/1040811296 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb244011e50 con 0x7fb2541085b0 2024-03-28T00:47:11.749 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.575+0000 7fb24a7fc700 1 -- 172.21.15.169:0/1040811296 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb24401a5c0 con 0x7fb2541085b0 2024-03-28T00:47:11.749 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.575+0000 7fb25b662700 1 -- 172.21.15.169:0/1040811296 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25419ef70 con 0x7fb2541085b0 2024-03-28T00:47:11.750 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.576+0000 7fb24a7fc700 1 -- 172.21.15.169:0/1040811296 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fb24401a8c0 con 0x7fb2541085b0 2024-03-28T00:47:11.750 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.576+0000 7fb24a7fc700 1 --2- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7fb240038ed0 0x7fb24003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:11.750 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.576+0000 7fb24a7fc700 1 -- 172.21.15.169:0/1040811296 --> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb24003baa0 con 0x7fb240038ed0 2024-03-28T00:47:11.750 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.576+0000 7fb24a7fc700 1 -- 172.21.15.169:0/1040811296 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb24404d1e0 con 0x7fb2541085b0 2024-03-28T00:47:11.750 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.576+0000 7fb258bfd700 1 -- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7fb240038ed0 msgr2=0x7fb24003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/311965133 2024-03-28T00:47:11.750 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.576+0000 7fb258bfd700 1 --2- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7fb240038ed0 0x7fb24003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T00:47:11.750 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.777+0000 7fb258bfd700 1 -- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7fb240038ed0 msgr2=0x7fb24003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/311965133 2024-03-28T00:47:11.750 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:05.777+0000 7fb258bfd700 1 --2- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7fb240038ed0 0x7fb24003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T00:47:11.750 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:06.178+0000 7fb258bfd700 1 -- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7fb240038ed0 msgr2=0x7fb24003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/311965133 2024-03-28T00:47:11.750 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:06.178+0000 7fb258bfd700 1 --2- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7fb240038ed0 0x7fb24003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T00:47:11.750 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:06.979+0000 7fb258bfd700 1 -- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7fb240038ed0 msgr2=0x7fb24003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/311965133 2024-03-28T00:47:11.750 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:06.979+0000 7fb258bfd700 1 --2- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7fb240038ed0 0x7fb24003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T00:47:11.750 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:08.579+0000 7fb258bfd700 1 -- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7fb240038ed0 msgr2=0x7fb24003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/311965133 2024-03-28T00:47:11.751 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:08.579+0000 7fb258bfd700 1 --2- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7fb240038ed0 0x7fb24003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T00:47:11.751 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:10.143+0000 7fb24a7fc700 1 -- 172.21.15.169:0/1040811296 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mgrmap(e 6) v1 ==== 45873+0+0 (secure 0 0 0) 0x7fb24401ac00 con 0x7fb2541085b0 2024-03-28T00:47:11.751 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:10.143+0000 7fb24a7fc700 1 -- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7fb240038ed0 msgr2=0x7fb24003b390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T00:47:11.751 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:10.143+0000 7fb24a7fc700 1 --2- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7fb240038ed0 0x7fb24003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:11.751 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:11.145+0000 7fb24a7fc700 1 -- 172.21.15.169:0/1040811296 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7fb24404efd0 con 0x7fb2541085b0 2024-03-28T00:47:11.751 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:11.145+0000 7fb24a7fc700 1 --2- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7fb24003c7c0 0x7fb24003ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:11.751 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:11.145+0000 7fb24a7fc700 1 -- 172.21.15.169:0/1040811296 --> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb24401fec0 con 0x7fb24003c7c0 2024-03-28T00:47:11.751 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:11.146+0000 7fb258bfd700 1 --2- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7fb24003c7c0 0x7fb24003ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:11.751 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:11.147+0000 7fb258bfd700 1 --2- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7fb24003c7c0 0x7fb24003ebb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb250003a10 tx=0x7fb250009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:11.751 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:11.147+0000 7fb24a7fc700 1 -- 172.21.15.169:0/1040811296 <== mgr.14120 v2:172.21.15.169:6800/3995234407 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7fb24401fec0 con 0x7fb24003c7c0 2024-03-28T00:47:11.751 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:11.154+0000 7fb25b662700 1 -- 172.21.15.169:0/1040811296 --> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fb2540637c0 con 0x7fb24003c7c0 2024-03-28T00:47:11.751 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:11.154+0000 7fb24a7fc700 1 -- 172.21.15.169:0/1040811296 <== mgr.14120 v2:172.21.15.169:6800/3995234407 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fb2540637c0 con 0x7fb24003c7c0 2024-03-28T00:47:11.751 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:11.154+0000 7fb25b662700 1 -- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7fb24003c7c0 msgr2=0x7fb24003ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:11.751 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:11.154+0000 7fb25b662700 1 --2- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7fb24003c7c0 0x7fb24003ebb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb250003a10 tx=0x7fb250009290 comp rx=0 tx=0).stop 2024-03-28T00:47:11.752 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:11.154+0000 7fb25b662700 1 -- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2541085b0 msgr2=0x7fb25419bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:11.752 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:11.154+0000 7fb25b662700 1 --2- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2541085b0 0x7fb25419bab0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb244011670 tx=0x7fb244011b60 comp rx=0 tx=0).stop 2024-03-28T00:47:11.752 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:11.154+0000 7fb25b662700 1 -- 172.21.15.169:0/1040811296 shutdown_connections 2024-03-28T00:47:11.752 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:11.154+0000 7fb25b662700 1 --2- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7fb24003c7c0 0x7fb24003ebb0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:11.752 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:11.154+0000 7fb25b662700 1 --2- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2541085b0 0x7fb25419bab0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:11.752 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:11.154+0000 7fb25b662700 1 --2- 172.21.15.169:0/1040811296 >> [v2:172.21.15.169:6800/311965133,v1:172.21.15.169:6801/311965133] conn(0x7fb240038ed0 0x7fb24003b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:11.752 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:11.154+0000 7fb25b662700 1 -- 172.21.15.169:0/1040811296 >> 172.21.15.169:0/1040811296 conn(0x7fb254101f80 msgr2=0x7fb254102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:11.752 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:11.154+0000 7fb25b662700 1 -- 172.21.15.169:0/1040811296 shutdown_connections 2024-03-28T00:47:11.752 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:11.154+0000 7fb25b662700 1 -- 172.21.15.169:0/1040811296 wait complete. 2024-03-28T00:47:11.752 INFO:teuthology.orchestra.run.smithi169.stdout:mgr epoch 5 is available 2024-03-28T00:47:11.752 INFO:teuthology.orchestra.run.smithi169.stdout:Setting orchestrator backend to cephadm... 2024-03-28T00:47:12.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:12 smithi169 ceph-mon[32274]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-28T00:47:12.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:12 smithi169 ceph-mon[32274]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-28T00:47:12.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:12 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' 2024-03-28T00:47:12.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:12 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' 2024-03-28T00:47:12.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:12 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:47:12.920 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.119+0000 7f19413fd700 1 Processor -- start 2024-03-28T00:47:12.920 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.119+0000 7f19413fd700 1 -- start start 2024-03-28T00:47:12.920 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.119+0000 7f19413fd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f193c106a40 0x7f193c106e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:12.920 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.119+0000 7f19413fd700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c107430 con 0x7f193c106a40 2024-03-28T00:47:12.920 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.120+0000 7f193affd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f193c106a40 0x7f193c106e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:12.920 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.120+0000 7f193affd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f193c106a40 0x7f193c106e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:52418/0 (socket says 172.21.15.169:52418) 2024-03-28T00:47:12.920 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.120+0000 7f193affd700 1 -- 172.21.15.169:0/3522276380 learned_addr learned my addr 172.21.15.169:0/3522276380 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:12.920 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.120+0000 7f193affd700 1 -- 172.21.15.169:0/3522276380 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193c107c40 con 0x7f193c106a40 2024-03-28T00:47:12.920 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.120+0000 7f193affd700 1 --2- 172.21.15.169:0/3522276380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f193c106a40 0x7f193c106e60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f19340098a0 tx=0x7f1934009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17cbe09e34e06361 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:12.921 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.120+0000 7f1939ffb700 1 -- 172.21.15.169:0/3522276380 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1934004160 con 0x7f193c106a40 2024-03-28T00:47:12.921 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.120+0000 7f1939ffb700 1 -- 172.21.15.169:0/3522276380 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1934004300 con 0x7f193c106a40 2024-03-28T00:47:12.921 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.120+0000 7f1939ffb700 1 -- 172.21.15.169:0/3522276380 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1934011b50 con 0x7f193c106a40 2024-03-28T00:47:12.921 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.121+0000 7f19413fd700 1 -- 172.21.15.169:0/3522276380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f193c106a40 msgr2=0x7f193c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:12.921 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.121+0000 7f19413fd700 1 --2- 172.21.15.169:0/3522276380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f193c106a40 0x7f193c106e60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f19340098a0 tx=0x7f1934009bb0 comp rx=0 tx=0).stop 2024-03-28T00:47:12.921 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.121+0000 7f19413fd700 1 -- 172.21.15.169:0/3522276380 shutdown_connections 2024-03-28T00:47:12.921 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.121+0000 7f19413fd700 1 --2- 172.21.15.169:0/3522276380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f193c106a40 0x7f193c106e60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:12.921 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.121+0000 7f19413fd700 1 -- 172.21.15.169:0/3522276380 >> 172.21.15.169:0/3522276380 conn(0x7f193c102000 msgr2=0x7f193c104420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:12.921 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.121+0000 7f19413fd700 1 -- 172.21.15.169:0/3522276380 shutdown_connections 2024-03-28T00:47:12.921 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.121+0000 7f19413fd700 1 -- 172.21.15.169:0/3522276380 wait complete. 2024-03-28T00:47:12.921 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.121+0000 7f19413fd700 1 Processor -- start 2024-03-28T00:47:12.921 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.122+0000 7f19413fd700 1 -- start start 2024-03-28T00:47:12.921 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.122+0000 7f19413fd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f193c106a40 0x7f193c07ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:12.921 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.122+0000 7f19413fd700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c10aff0 con 0x7f193c106a40 2024-03-28T00:47:12.921 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.122+0000 7f193affd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f193c106a40 0x7f193c07ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:12.922 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.122+0000 7f193affd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f193c106a40 0x7f193c07ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:52424/0 (socket says 172.21.15.169:52424) 2024-03-28T00:47:12.922 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.122+0000 7f193affd700 1 -- 172.21.15.169:0/3657249103 learned_addr learned my addr 172.21.15.169:0/3657249103 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:12.922 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.122+0000 7f193affd700 1 -- 172.21.15.169:0/3657249103 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193c19e8d0 con 0x7f193c106a40 2024-03-28T00:47:12.922 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.122+0000 7f193affd700 1 --2- 172.21.15.169:0/3657249103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f193c106a40 0x7f193c07ba70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1934011850 tx=0x7f1934011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:12.922 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.122+0000 7f1923fff700 1 -- 172.21.15.169:0/3657249103 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1934003af0 con 0x7f193c106a40 2024-03-28T00:47:12.922 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.122+0000 7f1923fff700 1 -- 172.21.15.169:0/3657249103 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1934020070 con 0x7f193c106a40 2024-03-28T00:47:12.922 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.122+0000 7f19413fd700 1 -- 172.21.15.169:0/3657249103 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f193c19eaa0 con 0x7f193c106a40 2024-03-28T00:47:12.922 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.122+0000 7f1923fff700 1 -- 172.21.15.169:0/3657249103 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19340195a0 con 0x7f193c106a40 2024-03-28T00:47:12.922 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.122+0000 7f19413fd700 1 -- 172.21.15.169:0/3657249103 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193c19ee40 con 0x7f193c106a40 2024-03-28T00:47:12.922 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.123+0000 7f1923fff700 1 -- 172.21.15.169:0/3657249103 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7f19340198a0 con 0x7f193c106a40 2024-03-28T00:47:12.922 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.123+0000 7f1923fff700 1 --2- 172.21.15.169:0/3657249103 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f1924038e00 0x7f192403b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:12.922 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.123+0000 7f1923fff700 1 -- 172.21.15.169:0/3657249103 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1934011e00 con 0x7f193c106a40 2024-03-28T00:47:12.922 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.123+0000 7f193a7fc700 1 --2- 172.21.15.169:0/3657249103 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f1924038e00 0x7f192403b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:12.922 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.124+0000 7f19413fd700 1 -- 172.21.15.169:0/3657249103 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f193c0637c0 con 0x7f193c106a40 2024-03-28T00:47:12.923 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.124+0000 7f193a7fc700 1 --2- 172.21.15.169:0/3657249103 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f1924038e00 0x7f192403b2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f193000ad30 tx=0x7f19300093f0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:12.923 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.129+0000 7f1923fff700 1 -- 172.21.15.169:0/3657249103 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1934003c90 con 0x7f193c106a40 2024-03-28T00:47:12.923 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.289+0000 7f1923fff700 1 -- 172.21.15.169:0/3657249103 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f1934019be0 con 0x7f193c106a40 2024-03-28T00:47:12.923 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.341+0000 7f19413fd700 1 -- 172.21.15.169:0/3657249103 --> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f193c103310 con 0x7f1924038e00 2024-03-28T00:47:12.923 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.349+0000 7f1923fff700 1 -- 172.21.15.169:0/3657249103 <== mgr.14120 v2:172.21.15.169:6800/3995234407 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f193c103310 con 0x7f1924038e00 2024-03-28T00:47:12.923 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.351+0000 7f19413fd700 1 -- 172.21.15.169:0/3657249103 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f1924038e00 msgr2=0x7f192403b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:12.923 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.351+0000 7f19413fd700 1 --2- 172.21.15.169:0/3657249103 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f1924038e00 0x7f192403b2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f193000ad30 tx=0x7f19300093f0 comp rx=0 tx=0).stop 2024-03-28T00:47:12.923 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.351+0000 7f19413fd700 1 -- 172.21.15.169:0/3657249103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f193c106a40 msgr2=0x7f193c07ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:12.923 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.351+0000 7f19413fd700 1 --2- 172.21.15.169:0/3657249103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f193c106a40 0x7f193c07ba70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1934011850 tx=0x7f1934011c10 comp rx=0 tx=0).stop 2024-03-28T00:47:12.923 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.351+0000 7f19413fd700 1 -- 172.21.15.169:0/3657249103 shutdown_connections 2024-03-28T00:47:12.923 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.351+0000 7f19413fd700 1 --2- 172.21.15.169:0/3657249103 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f1924038e00 0x7f192403b2c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:12.923 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.351+0000 7f19413fd700 1 --2- 172.21.15.169:0/3657249103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f193c106a40 0x7f193c07ba70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:12.923 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.351+0000 7f19413fd700 1 -- 172.21.15.169:0/3657249103 >> 172.21.15.169:0/3657249103 conn(0x7f193c102000 msgr2=0x7f193c102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:12.923 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.352+0000 7f19413fd700 1 -- 172.21.15.169:0/3657249103 shutdown_connections 2024-03-28T00:47:12.923 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:12.352+0000 7f19413fd700 1 -- 172.21.15.169:0/3657249103 wait complete. 2024-03-28T00:47:13.523 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:13 smithi169 ceph-mon[32274]: [28/Mar/2024:00:47:11] ENGINE Bus STARTING 2024-03-28T00:47:13.523 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:13 smithi169 ceph-mon[32274]: [28/Mar/2024:00:47:11] ENGINE Serving on https://172.21.15.169:7150 2024-03-28T00:47:13.523 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:13 smithi169 ceph-mon[32274]: [28/Mar/2024:00:47:11] ENGINE Serving on http://172.21.15.169:8765 2024-03-28T00:47:13.523 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:13 smithi169 ceph-mon[32274]: [28/Mar/2024:00:47:11] ENGINE Bus STARTED 2024-03-28T00:47:13.523 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:13 smithi169 ceph-mon[32274]: mgrmap e8: x(active, since 2s) 2024-03-28T00:47:13.523 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:13 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' 2024-03-28T00:47:13.523 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:13 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:47:14.067 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-28T00:47:14.067 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.313+0000 7f91ed4f7700 1 Processor -- start 2024-03-28T00:47:14.067 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.313+0000 7f91ed4f7700 1 -- start start 2024-03-28T00:47:14.067 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.313+0000 7f91ed4f7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91e8108600 0x7f91e810a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:14.067 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.314+0000 7f91ed4f7700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e8075860 con 0x7f91e8108600 2024-03-28T00:47:14.067 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.314+0000 7f91e6ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91e8108600 0x7f91e810a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:14.068 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.314+0000 7f91e6ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91e8108600 0x7f91e810a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:52440/0 (socket says 172.21.15.169:52440) 2024-03-28T00:47:14.068 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.314+0000 7f91e6ffd700 1 -- 172.21.15.169:0/1354694988 learned_addr learned my addr 172.21.15.169:0/1354694988 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:14.068 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.314+0000 7f91e6ffd700 1 -- 172.21.15.169:0/1354694988 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e810af30 con 0x7f91e8108600 2024-03-28T00:47:14.068 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.314+0000 7f91e6ffd700 1 --2- 172.21.15.169:0/1354694988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91e8108600 0x7f91e810a9f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f91e0009e80 tx=0x7f91e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=424acb326f9a9d0c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:14.068 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.314+0000 7f91e5ffb700 1 -- 172.21.15.169:0/1354694988 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91e0017070 con 0x7f91e8108600 2024-03-28T00:47:14.068 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.314+0000 7f91e5ffb700 1 -- 172.21.15.169:0/1354694988 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f91e000b8c0 con 0x7f91e8108600 2024-03-28T00:47:14.068 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.314+0000 7f91e5ffb700 1 -- 172.21.15.169:0/1354694988 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91e00045c0 con 0x7f91e8108600 2024-03-28T00:47:14.068 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.315+0000 7f91ed4f7700 1 -- 172.21.15.169:0/1354694988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91e8108600 msgr2=0x7f91e810a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:14.068 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.315+0000 7f91ed4f7700 1 --2- 172.21.15.169:0/1354694988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91e8108600 0x7f91e810a9f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f91e0009e80 tx=0x7f91e000b330 comp rx=0 tx=0).stop 2024-03-28T00:47:14.068 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.315+0000 7f91ed4f7700 1 -- 172.21.15.169:0/1354694988 shutdown_connections 2024-03-28T00:47:14.068 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.315+0000 7f91ed4f7700 1 --2- 172.21.15.169:0/1354694988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91e8108600 0x7f91e810a9f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:14.069 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.315+0000 7f91ed4f7700 1 -- 172.21.15.169:0/1354694988 >> 172.21.15.169:0/1354694988 conn(0x7f91e8101fb0 msgr2=0x7f91e81043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:14.069 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.315+0000 7f91ed4f7700 1 -- 172.21.15.169:0/1354694988 shutdown_connections 2024-03-28T00:47:14.069 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.315+0000 7f91ed4f7700 1 -- 172.21.15.169:0/1354694988 wait complete. 2024-03-28T00:47:14.069 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.316+0000 7f91ed4f7700 1 Processor -- start 2024-03-28T00:47:14.069 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.316+0000 7f91ed4f7700 1 -- start start 2024-03-28T00:47:14.069 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.316+0000 7f91ed4f7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91e8108600 0x7f91e819ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:14.069 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.316+0000 7f91ed4f7700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e810bef0 con 0x7f91e8108600 2024-03-28T00:47:14.069 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.316+0000 7f91e6ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91e8108600 0x7f91e819ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:14.069 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.316+0000 7f91e6ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91e8108600 0x7f91e819ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:52444/0 (socket says 172.21.15.169:52444) 2024-03-28T00:47:14.069 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.316+0000 7f91e6ffd700 1 -- 172.21.15.169:0/3424161432 learned_addr learned my addr 172.21.15.169:0/3424161432 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:14.069 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.316+0000 7f91e6ffd700 1 -- 172.21.15.169:0/3424161432 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e819bfc0 con 0x7f91e8108600 2024-03-28T00:47:14.070 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.316+0000 7f91e6ffd700 1 --2- 172.21.15.169:0/3424161432 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91e8108600 0x7f91e819ba80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f91e0009530 tx=0x7f91e000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:14.070 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.317+0000 7f91dffff700 1 -- 172.21.15.169:0/3424161432 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91e0020020 con 0x7f91e8108600 2024-03-28T00:47:14.070 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.317+0000 7f91dffff700 1 -- 172.21.15.169:0/3424161432 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f91e0003b50 con 0x7f91e8108600 2024-03-28T00:47:14.070 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.317+0000 7f91ed4f7700 1 -- 172.21.15.169:0/3424161432 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e819c250 con 0x7f91e8108600 2024-03-28T00:47:14.070 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.317+0000 7f91dffff700 1 -- 172.21.15.169:0/3424161432 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91e0017040 con 0x7f91e8108600 2024-03-28T00:47:14.070 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.317+0000 7f91ed4f7700 1 -- 172.21.15.169:0/3424161432 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e819ef40 con 0x7f91e8108600 2024-03-28T00:47:14.070 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.317+0000 7f91dffff700 1 -- 172.21.15.169:0/3424161432 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f91e0023020 con 0x7f91e8108600 2024-03-28T00:47:14.070 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.318+0000 7f91dffff700 1 --2- 172.21.15.169:0/3424161432 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f91d4038f20 0x7f91d403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:14.070 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.318+0000 7f91dffff700 1 -- 172.21.15.169:0/3424161432 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f91e000f050 con 0x7f91e8108600 2024-03-28T00:47:14.070 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.318+0000 7f91e67fc700 1 --2- 172.21.15.169:0/3424161432 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f91d4038f20 0x7f91d403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:14.070 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.318+0000 7f91ed4f7700 1 -- 172.21.15.169:0/3424161432 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e80637c0 con 0x7f91e8108600 2024-03-28T00:47:14.070 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.318+0000 7f91e67fc700 1 --2- 172.21.15.169:0/3424161432 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f91d4038f20 0x7f91d403b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f91d0006fd0 tx=0x7f91d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:14.071 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.323+0000 7f91dffff700 1 -- 172.21.15.169:0/3424161432 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f91e007b740 con 0x7f91e8108600 2024-03-28T00:47:14.071 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.519+0000 7f91ed4f7700 1 -- 172.21.15.169:0/3424161432 --> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f91e81031b0 con 0x7f91d4038f20 2024-03-28T00:47:14.071 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.519+0000 7f91dffff700 1 -- 172.21.15.169:0/3424161432 <== mgr.14120 v2:172.21.15.169:6800/3995234407 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f91e81031b0 con 0x7f91d4038f20 2024-03-28T00:47:14.071 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.521+0000 7f91ed4f7700 1 -- 172.21.15.169:0/3424161432 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f91d4038f20 msgr2=0x7f91d403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:14.071 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.521+0000 7f91ed4f7700 1 --2- 172.21.15.169:0/3424161432 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f91d4038f20 0x7f91d403b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f91d0006fd0 tx=0x7f91d0006e40 comp rx=0 tx=0).stop 2024-03-28T00:47:14.071 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.521+0000 7f91ed4f7700 1 -- 172.21.15.169:0/3424161432 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91e8108600 msgr2=0x7f91e819ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:14.071 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.521+0000 7f91ed4f7700 1 --2- 172.21.15.169:0/3424161432 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91e8108600 0x7f91e819ba80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f91e0009530 tx=0x7f91e000bd50 comp rx=0 tx=0).stop 2024-03-28T00:47:14.071 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.521+0000 7f91ed4f7700 1 -- 172.21.15.169:0/3424161432 shutdown_connections 2024-03-28T00:47:14.071 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.521+0000 7f91ed4f7700 1 --2- 172.21.15.169:0/3424161432 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f91d4038f20 0x7f91d403b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:14.071 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.521+0000 7f91ed4f7700 1 --2- 172.21.15.169:0/3424161432 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91e8108600 0x7f91e819ba80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:14.072 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.521+0000 7f91ed4f7700 1 -- 172.21.15.169:0/3424161432 >> 172.21.15.169:0/3424161432 conn(0x7f91e8101fb0 msgr2=0x7f91e8102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:14.072 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.522+0000 7f91ed4f7700 1 -- 172.21.15.169:0/3424161432 shutdown_connections 2024-03-28T00:47:14.072 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:13.522+0000 7f91ed4f7700 1 -- 172.21.15.169:0/3424161432 wait complete. 2024-03-28T00:47:14.072 INFO:teuthology.orchestra.run.smithi169.stdout:Generating ssh key... 2024-03-28T00:47:14.454 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:14 smithi169 ceph-mon[32274]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:47:15.242 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:14 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: Generating public/private rsa key pair. 2024-03-28T00:47:15.242 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:14 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: Your identification has been saved in /tmp/tmpv8sprepb/key. 2024-03-28T00:47:15.242 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:14 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: Your public key has been saved in /tmp/tmpv8sprepb/key.pub. 2024-03-28T00:47:15.242 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:14 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: The key fingerprint is: 2024-03-28T00:47:15.242 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:14 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: SHA256:ACLGCFOSfkGv8rxWkGo1qXPzU9Dj0FnXlWsXwEYhi20 ceph-833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T00:47:15.242 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:14 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: The key's randomart image is: 2024-03-28T00:47:15.242 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:14 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: +---[RSA 3072]----+ 2024-03-28T00:47:15.242 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:14 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: |O++.. .+++..| 2024-03-28T00:47:15.242 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:14 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: |+= o.. .o.o+ o | 2024-03-28T00:47:15.242 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:14 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: |. +.+ o..E. o| 2024-03-28T00:47:15.243 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:14 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: | . B.o * . o.| 2024-03-28T00:47:15.243 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:14 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: | .=.o + S . .| 2024-03-28T00:47:15.243 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:14 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: | =+o . o | 2024-03-28T00:47:15.243 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:14 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: |. oo+ . | 2024-03-28T00:47:15.243 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:14 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: | ..o | 2024-03-28T00:47:15.243 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:14 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: | .. . | 2024-03-28T00:47:15.243 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:14 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: +----[SHA256]-----+ 2024-03-28T00:47:15.492 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:15 smithi169 ceph-mon[32274]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:47:15.492 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:15 smithi169 ceph-mon[32274]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:47:15.492 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:15 smithi169 ceph-mon[32274]: Generating ssh key... 2024-03-28T00:47:15.492 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:15 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' 2024-03-28T00:47:15.492 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:15 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' 2024-03-28T00:47:15.499 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.474+0000 7ff35e41b700 1 Processor -- start 2024-03-28T00:47:15.499 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.474+0000 7ff35e41b700 1 -- start start 2024-03-28T00:47:15.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.474+0000 7ff35e41b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff358107b10 0x7ff358107f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:15.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.474+0000 7ff35e41b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff358108500 con 0x7ff358107b10 2024-03-28T00:47:15.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.475+0000 7ff357fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff358107b10 0x7ff358107f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:15.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.475+0000 7ff357fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff358107b10 0x7ff358107f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:52454/0 (socket says 172.21.15.169:52454) 2024-03-28T00:47:15.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.475+0000 7ff357fff700 1 -- 172.21.15.169:0/624533953 learned_addr learned my addr 172.21.15.169:0/624533953 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:15.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.475+0000 7ff357fff700 1 -- 172.21.15.169:0/624533953 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff358108d10 con 0x7ff358107b10 2024-03-28T00:47:15.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.475+0000 7ff357fff700 1 --2- 172.21.15.169:0/624533953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff358107b10 0x7ff358107f30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff3480098a0 tx=0x7ff348009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d2cc6b698a1d7f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:15.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.475+0000 7ff356ffd700 1 -- 172.21.15.169:0/624533953 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff348004160 con 0x7ff358107b10 2024-03-28T00:47:15.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.475+0000 7ff356ffd700 1 -- 172.21.15.169:0/624533953 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff348004300 con 0x7ff358107b10 2024-03-28T00:47:15.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.475+0000 7ff356ffd700 1 -- 172.21.15.169:0/624533953 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff348011b50 con 0x7ff358107b10 2024-03-28T00:47:15.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.476+0000 7ff35e41b700 1 -- 172.21.15.169:0/624533953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff358107b10 msgr2=0x7ff358107f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:15.500 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.476+0000 7ff35e41b700 1 --2- 172.21.15.169:0/624533953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff358107b10 0x7ff358107f30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff3480098a0 tx=0x7ff348009bb0 comp rx=0 tx=0).stop 2024-03-28T00:47:15.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.476+0000 7ff35e41b700 1 -- 172.21.15.169:0/624533953 shutdown_connections 2024-03-28T00:47:15.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.476+0000 7ff35e41b700 1 --2- 172.21.15.169:0/624533953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff358107b10 0x7ff358107f30 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:15.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.476+0000 7ff35e41b700 1 -- 172.21.15.169:0/624533953 >> 172.21.15.169:0/624533953 conn(0x7ff358103070 msgr2=0x7ff3581054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:15.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.476+0000 7ff35e41b700 1 -- 172.21.15.169:0/624533953 shutdown_connections 2024-03-28T00:47:15.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.476+0000 7ff35e41b700 1 -- 172.21.15.169:0/624533953 wait complete. 2024-03-28T00:47:15.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.476+0000 7ff35e41b700 1 Processor -- start 2024-03-28T00:47:15.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.476+0000 7ff35e41b700 1 -- start start 2024-03-28T00:47:15.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.477+0000 7ff35e41b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff358107b10 0x7ff35819bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:15.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.477+0000 7ff35e41b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35810c0c0 con 0x7ff358107b10 2024-03-28T00:47:15.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.477+0000 7ff357fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff358107b10 0x7ff35819bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:15.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.477+0000 7ff357fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff358107b10 0x7ff35819bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:52456/0 (socket says 172.21.15.169:52456) 2024-03-28T00:47:15.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.477+0000 7ff357fff700 1 -- 172.21.15.169:0/573575914 learned_addr learned my addr 172.21.15.169:0/573575914 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:15.501 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.477+0000 7ff357fff700 1 -- 172.21.15.169:0/573575914 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35819c0a0 con 0x7ff358107b10 2024-03-28T00:47:15.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.477+0000 7ff357fff700 1 --2- 172.21.15.169:0/573575914 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff358107b10 0x7ff35819bb60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff348011850 tx=0x7ff348011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:15.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.477+0000 7ff3557fa700 1 -- 172.21.15.169:0/573575914 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff348003af0 con 0x7ff358107b10 2024-03-28T00:47:15.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.477+0000 7ff3557fa700 1 -- 172.21.15.169:0/573575914 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff348020070 con 0x7ff358107b10 2024-03-28T00:47:15.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.477+0000 7ff35e41b700 1 -- 172.21.15.169:0/573575914 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff35819c330 con 0x7ff358107b10 2024-03-28T00:47:15.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.477+0000 7ff3557fa700 1 -- 172.21.15.169:0/573575914 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff3480195a0 con 0x7ff358107b10 2024-03-28T00:47:15.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.477+0000 7ff35e41b700 1 -- 172.21.15.169:0/573575914 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff35819f020 con 0x7ff358107b10 2024-03-28T00:47:15.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.478+0000 7ff3557fa700 1 -- 172.21.15.169:0/573575914 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7ff3480198a0 con 0x7ff358107b10 2024-03-28T00:47:15.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.478+0000 7ff3557fa700 1 --2- 172.21.15.169:0/573575914 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7ff340038f20 0x7ff34003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:15.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.478+0000 7ff35e41b700 1 -- 172.21.15.169:0/573575914 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3580637c0 con 0x7ff358107b10 2024-03-28T00:47:15.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.478+0000 7ff3557fa700 1 -- 172.21.15.169:0/573575914 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff348011e00 con 0x7ff358107b10 2024-03-28T00:47:15.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.478+0000 7ff3577fe700 1 --2- 172.21.15.169:0/573575914 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7ff340038f20 0x7ff34003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:15.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.479+0000 7ff3577fe700 1 --2- 172.21.15.169:0/573575914 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7ff340038f20 0x7ff34003b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff34c006fd0 tx=0x7ff34c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:15.502 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.483+0000 7ff3557fa700 1 -- 172.21.15.169:0/573575914 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff348023530 con 0x7ff358107b10 2024-03-28T00:47:15.503 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.677+0000 7ff35e41b700 1 -- 172.21.15.169:0/573575914 --> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3581043c0 con 0x7ff340038f20 2024-03-28T00:47:15.503 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.922+0000 7ff3557fa700 1 -- 172.21.15.169:0/573575914 <== mgr.14120 v2:172.21.15.169:6800/3995234407 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff3581043c0 con 0x7ff340038f20 2024-03-28T00:47:15.503 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.924+0000 7ff35e41b700 1 -- 172.21.15.169:0/573575914 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7ff340038f20 msgr2=0x7ff34003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:15.503 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.924+0000 7ff35e41b700 1 --2- 172.21.15.169:0/573575914 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7ff340038f20 0x7ff34003b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff34c006fd0 tx=0x7ff34c006e40 comp rx=0 tx=0).stop 2024-03-28T00:47:15.503 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.924+0000 7ff35e41b700 1 -- 172.21.15.169:0/573575914 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff358107b10 msgr2=0x7ff35819bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:15.503 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.924+0000 7ff35e41b700 1 --2- 172.21.15.169:0/573575914 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff358107b10 0x7ff35819bb60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff348011850 tx=0x7ff348011c10 comp rx=0 tx=0).stop 2024-03-28T00:47:15.503 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.924+0000 7ff35e41b700 1 -- 172.21.15.169:0/573575914 shutdown_connections 2024-03-28T00:47:15.503 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.924+0000 7ff35e41b700 1 --2- 172.21.15.169:0/573575914 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7ff340038f20 0x7ff34003b3e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:15.503 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.924+0000 7ff35e41b700 1 --2- 172.21.15.169:0/573575914 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff358107b10 0x7ff35819bb60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:15.503 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.924+0000 7ff35e41b700 1 -- 172.21.15.169:0/573575914 >> 172.21.15.169:0/573575914 conn(0x7ff358103070 msgr2=0x7ff358103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:15.503 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.924+0000 7ff35e41b700 1 -- 172.21.15.169:0/573575914 shutdown_connections 2024-03-28T00:47:15.503 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:14.924+0000 7ff35e41b700 1 -- 172.21.15.169:0/573575914 wait complete. 2024-03-28T00:47:16.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:16 smithi169 ceph-mon[32274]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:47:16.691 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4X76kTFMVUKlH9cG7YjCSOAg3eID81cphRlIaiCNMlzrvjwLhCke9F8Ysl6mPYqropWQ3KW88DpdrtITRL3kx+kbFiV7vwRis8tehhOm9YM7FZp6dco4UFahT2EnkJ4mz1ixNSRyelmKYeRdnLJH5vuETtWXj8plzmflYNvznW/O8qEMURxJvU/X2T0nU+MJ5ZYlOcrIBHM3N9xN/umq4zm0h8zoJYkKqi07EREOHtGG4T2MT7hgjfwdGY4D2+oC+k+7zIc40/p5ZH5/5k4kDJqPn8eRiZwHHHQ0DHTeEaQQsL4tKEMYsafrK3J9pQ7X6lbYhUVyCHl6VJPIGbel2zy6jowrUzGzT+WQvxgZWcZgkCPlB8ARa3gQKH8voghMfMjiOAjbq+VQi9E7U/Il9C+dTYOlqZqLZtXwBANMYgSaEJTHqYlAiGoL5DtA/aauTC/sDzmRIIBvSN3KSEf2ub6nXM/RU4b+Y1LiLaGiSFS9CQvMZ/WIUbh0buyB8qjs= ceph-833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T00:47:16.691 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.916+0000 7f2873990700 1 Processor -- start 2024-03-28T00:47:16.691 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.916+0000 7f2873990700 1 -- start start 2024-03-28T00:47:16.691 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.916+0000 7f2873990700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f286c105dd0 0x7f286c1061f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:16.691 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.916+0000 7f2873990700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c1067c0 con 0x7f286c105dd0 2024-03-28T00:47:16.691 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.917+0000 7f287172c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f286c105dd0 0x7f286c1061f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:16.691 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.917+0000 7f287172c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f286c105dd0 0x7f286c1061f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:52472/0 (socket says 172.21.15.169:52472) 2024-03-28T00:47:16.691 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.917+0000 7f287172c700 1 -- 172.21.15.169:0/1662522429 learned_addr learned my addr 172.21.15.169:0/1662522429 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:16.691 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.917+0000 7f287172c700 1 -- 172.21.15.169:0/1662522429 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286c108d60 con 0x7f286c105dd0 2024-03-28T00:47:16.691 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.917+0000 7f287172c700 1 --2- 172.21.15.169:0/1662522429 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f286c105dd0 0x7f286c1061f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f285c0098a0 tx=0x7f285c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6331515e68e64d29 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:16.692 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.917+0000 7f2863fff700 1 -- 172.21.15.169:0/1662522429 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f285c004160 con 0x7f286c105dd0 2024-03-28T00:47:16.692 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.917+0000 7f2863fff700 1 -- 172.21.15.169:0/1662522429 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f285c004300 con 0x7f286c105dd0 2024-03-28T00:47:16.692 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.917+0000 7f2863fff700 1 -- 172.21.15.169:0/1662522429 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f285c011b50 con 0x7f286c105dd0 2024-03-28T00:47:16.692 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.918+0000 7f2873990700 1 -- 172.21.15.169:0/1662522429 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f286c105dd0 msgr2=0x7f286c1061f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:16.692 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.918+0000 7f2873990700 1 --2- 172.21.15.169:0/1662522429 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f286c105dd0 0x7f286c1061f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f285c0098a0 tx=0x7f285c009bb0 comp rx=0 tx=0).stop 2024-03-28T00:47:16.692 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.918+0000 7f2873990700 1 -- 172.21.15.169:0/1662522429 shutdown_connections 2024-03-28T00:47:16.692 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.918+0000 7f2873990700 1 --2- 172.21.15.169:0/1662522429 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f286c105dd0 0x7f286c1061f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:16.692 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.918+0000 7f2873990700 1 -- 172.21.15.169:0/1662522429 >> 172.21.15.169:0/1662522429 conn(0x7f286c101780 msgr2=0x7f286c103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:16.692 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.918+0000 7f2873990700 1 -- 172.21.15.169:0/1662522429 shutdown_connections 2024-03-28T00:47:16.692 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.918+0000 7f2873990700 1 -- 172.21.15.169:0/1662522429 wait complete. 2024-03-28T00:47:16.692 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.919+0000 7f2873990700 1 Processor -- start 2024-03-28T00:47:16.692 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.919+0000 7f2873990700 1 -- start start 2024-03-28T00:47:16.692 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.919+0000 7f2873990700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f286c105dd0 0x7f286c19bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:16.692 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.919+0000 7f2873990700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c10c110 con 0x7f286c105dd0 2024-03-28T00:47:16.692 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.919+0000 7f287172c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f286c105dd0 0x7f286c19bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:16.693 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.919+0000 7f287172c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f286c105dd0 0x7f286c19bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:52482/0 (socket says 172.21.15.169:52482) 2024-03-28T00:47:16.693 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.919+0000 7f287172c700 1 -- 172.21.15.169:0/139502752 learned_addr learned my addr 172.21.15.169:0/139502752 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:16.693 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.920+0000 7f287172c700 1 -- 172.21.15.169:0/139502752 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286c19c080 con 0x7f286c105dd0 2024-03-28T00:47:16.693 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.920+0000 7f287172c700 1 --2- 172.21.15.169:0/139502752 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f286c105dd0 0x7f286c19bb40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f285c009fb0 tx=0x7f285c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:16.693 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.920+0000 7f28627fc700 1 -- 172.21.15.169:0/139502752 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f285c0038b0 con 0x7f286c105dd0 2024-03-28T00:47:16.693 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.920+0000 7f28627fc700 1 -- 172.21.15.169:0/139502752 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f285c011e50 con 0x7f286c105dd0 2024-03-28T00:47:16.693 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.920+0000 7f28627fc700 1 -- 172.21.15.169:0/139502752 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f285c019550 con 0x7f286c105dd0 2024-03-28T00:47:16.693 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.920+0000 7f2873990700 1 -- 172.21.15.169:0/139502752 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f286c19c310 con 0x7f286c105dd0 2024-03-28T00:47:16.693 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.920+0000 7f2873990700 1 -- 172.21.15.169:0/139502752 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f286c19f000 con 0x7f286c105dd0 2024-03-28T00:47:16.693 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.921+0000 7f28627fc700 1 -- 172.21.15.169:0/139502752 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f285c003a50 con 0x7f286c105dd0 2024-03-28T00:47:16.693 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.921+0000 7f2873990700 1 -- 172.21.15.169:0/139502752 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286c0637c0 con 0x7f286c105dd0 2024-03-28T00:47:16.693 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.921+0000 7f28627fc700 1 --2- 172.21.15.169:0/139502752 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f2858038f20 0x7f285803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:16.693 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.921+0000 7f28627fc700 1 -- 172.21.15.169:0/139502752 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f285c04d160 con 0x7f286c105dd0 2024-03-28T00:47:16.693 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.921+0000 7f2870f2b700 1 --2- 172.21.15.169:0/139502752 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f2858038f20 0x7f285803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:16.693 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.922+0000 7f2870f2b700 1 --2- 172.21.15.169:0/139502752 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f2858038f20 0x7f285803b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2868006fd0 tx=0x7f2868006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:16.693 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:15.926+0000 7f28627fc700 1 -- 172.21.15.169:0/139502752 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f285c04cab0 con 0x7f286c105dd0 2024-03-28T00:47:16.693 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:16.132+0000 7f2873990700 1 -- 172.21.15.169:0/139502752 --> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f286c1029f0 con 0x7f2858038f20 2024-03-28T00:47:16.694 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:16.133+0000 7f28627fc700 1 -- 172.21.15.169:0/139502752 <== mgr.14120 v2:172.21.15.169:6800/3995234407 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f286c1029f0 con 0x7f2858038f20 2024-03-28T00:47:16.694 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:16.135+0000 7f2873990700 1 -- 172.21.15.169:0/139502752 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f2858038f20 msgr2=0x7f285803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:16.694 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:16.135+0000 7f2873990700 1 --2- 172.21.15.169:0/139502752 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f2858038f20 0x7f285803b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2868006fd0 tx=0x7f2868006e40 comp rx=0 tx=0).stop 2024-03-28T00:47:16.694 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:16.135+0000 7f2873990700 1 -- 172.21.15.169:0/139502752 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f286c105dd0 msgr2=0x7f286c19bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:16.694 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:16.135+0000 7f2873990700 1 --2- 172.21.15.169:0/139502752 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f286c105dd0 0x7f286c19bb40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f285c009fb0 tx=0x7f285c011bf0 comp rx=0 tx=0).stop 2024-03-28T00:47:16.694 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:16.135+0000 7f2873990700 1 -- 172.21.15.169:0/139502752 shutdown_connections 2024-03-28T00:47:16.694 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:16.135+0000 7f2873990700 1 --2- 172.21.15.169:0/139502752 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f2858038f20 0x7f285803b3e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:16.694 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:16.135+0000 7f2873990700 1 --2- 172.21.15.169:0/139502752 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f286c105dd0 0x7f286c19bb40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:16.694 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:16.135+0000 7f2873990700 1 -- 172.21.15.169:0/139502752 >> 172.21.15.169:0/139502752 conn(0x7f286c101780 msgr2=0x7f286c1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:16.694 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:16.135+0000 7f2873990700 1 -- 172.21.15.169:0/139502752 shutdown_connections 2024-03-28T00:47:16.694 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:16.135+0000 7f2873990700 1 -- 172.21.15.169:0/139502752 wait complete. 2024-03-28T00:47:16.694 INFO:teuthology.orchestra.run.smithi169.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-28T00:47:16.694 INFO:teuthology.orchestra.run.smithi169.stdout:Adding key to root@localhost authorized_keys... 2024-03-28T00:47:16.694 INFO:teuthology.orchestra.run.smithi169.stdout:Adding host smithi169... 2024-03-28T00:47:18.540 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:18 smithi169 ceph-mon[32274]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi169", "addr": "172.21.15.169", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:47:18.540 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:18 smithi169 ceph-mon[32274]: Deploying cephadm binary to smithi169 2024-03-28T00:47:22.248 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout Added host 'smithi169' with addr '172.21.15.169' 2024-03-28T00:47:22.248 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.126+0000 7f58c3ada700 1 Processor -- start 2024-03-28T00:47:22.248 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.127+0000 7f58c3ada700 1 -- start start 2024-03-28T00:47:22.248 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.127+0000 7f58c3ada700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58bc107b50 0x7f58bc107f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:22.249 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.127+0000 7f58c3ada700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc108540 con 0x7f58bc107b50 2024-03-28T00:47:22.249 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.127+0000 7f58c1876700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58bc107b50 0x7f58bc107f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:22.249 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.127+0000 7f58c1876700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58bc107b50 0x7f58bc107f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:37956/0 (socket says 172.21.15.169:37956) 2024-03-28T00:47:22.249 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.127+0000 7f58c1876700 1 -- 172.21.15.169:0/2824895150 learned_addr learned my addr 172.21.15.169:0/2824895150 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:22.249 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.127+0000 7f58c1876700 1 -- 172.21.15.169:0/2824895150 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc108d50 con 0x7f58bc107b50 2024-03-28T00:47:22.249 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.128+0000 7f58c1876700 1 --2- 172.21.15.169:0/2824895150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58bc107b50 0x7f58bc107f70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0098a0 tx=0x7f58ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=861f3547bd4b5249 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:22.249 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.128+0000 7f58c0874700 1 -- 172.21.15.169:0/2824895150 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58ac004160 con 0x7f58bc107b50 2024-03-28T00:47:22.249 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.128+0000 7f58c0874700 1 -- 172.21.15.169:0/2824895150 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58ac004300 con 0x7f58bc107b50 2024-03-28T00:47:22.249 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.128+0000 7f58c0874700 1 -- 172.21.15.169:0/2824895150 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58ac011b50 con 0x7f58bc107b50 2024-03-28T00:47:22.249 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.128+0000 7f58c3ada700 1 -- 172.21.15.169:0/2824895150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58bc107b50 msgr2=0x7f58bc107f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:22.249 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.128+0000 7f58c3ada700 1 --2- 172.21.15.169:0/2824895150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58bc107b50 0x7f58bc107f70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0098a0 tx=0x7f58ac009bb0 comp rx=0 tx=0).stop 2024-03-28T00:47:22.249 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.128+0000 7f58c3ada700 1 -- 172.21.15.169:0/2824895150 shutdown_connections 2024-03-28T00:47:22.249 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.128+0000 7f58c3ada700 1 --2- 172.21.15.169:0/2824895150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58bc107b50 0x7f58bc107f70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:22.250 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.128+0000 7f58c3ada700 1 -- 172.21.15.169:0/2824895150 >> 172.21.15.169:0/2824895150 conn(0x7f58bc103150 msgr2=0x7f58bc105570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:22.250 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.128+0000 7f58c3ada700 1 -- 172.21.15.169:0/2824895150 shutdown_connections 2024-03-28T00:47:22.250 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.128+0000 7f58c3ada700 1 -- 172.21.15.169:0/2824895150 wait complete. 2024-03-28T00:47:22.250 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.129+0000 7f58c3ada700 1 Processor -- start 2024-03-28T00:47:22.250 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.129+0000 7f58c3ada700 1 -- start start 2024-03-28T00:47:22.250 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.129+0000 7f58c3ada700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58bc107b50 0x7f58bc19bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:22.250 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.129+0000 7f58c3ada700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc10c100 con 0x7f58bc107b50 2024-03-28T00:47:22.250 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.129+0000 7f58c1876700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58bc107b50 0x7f58bc19bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:22.250 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.129+0000 7f58c1876700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58bc107b50 0x7f58bc19bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:37970/0 (socket says 172.21.15.169:37970) 2024-03-28T00:47:22.250 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.129+0000 7f58c1876700 1 -- 172.21.15.169:0/2824294745 learned_addr learned my addr 172.21.15.169:0/2824294745 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:22.250 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.130+0000 7f58c1876700 1 -- 172.21.15.169:0/2824294745 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc19c0b0 con 0x7f58bc107b50 2024-03-28T00:47:22.250 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.130+0000 7f58c1876700 1 --2- 172.21.15.169:0/2824294745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58bc107b50 0x7f58bc19bb70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009fb0 tx=0x7f58ac011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:22.250 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.130+0000 7f58b2ffd700 1 -- 172.21.15.169:0/2824294745 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58ac0038b0 con 0x7f58bc107b50 2024-03-28T00:47:22.251 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.130+0000 7f58b2ffd700 1 -- 172.21.15.169:0/2824294745 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58ac011e50 con 0x7f58bc107b50 2024-03-28T00:47:22.251 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.130+0000 7f58c3ada700 1 -- 172.21.15.169:0/2824294745 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58bc19c340 con 0x7f58bc107b50 2024-03-28T00:47:22.251 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.130+0000 7f58b2ffd700 1 -- 172.21.15.169:0/2824294745 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58ac019550 con 0x7f58bc107b50 2024-03-28T00:47:22.251 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.130+0000 7f58c3ada700 1 -- 172.21.15.169:0/2824294745 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58bc19f030 con 0x7f58bc107b50 2024-03-28T00:47:22.251 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.131+0000 7f58b2ffd700 1 -- 172.21.15.169:0/2824294745 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f58ac019800 con 0x7f58bc107b50 2024-03-28T00:47:22.251 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.131+0000 7f58c3ada700 1 -- 172.21.15.169:0/2824294745 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58bc0637c0 con 0x7f58bc107b50 2024-03-28T00:47:22.251 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.131+0000 7f58b2ffd700 1 --2- 172.21.15.169:0/2824294745 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f58a8038f20 0x7f58a803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:22.251 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.131+0000 7f58b2ffd700 1 -- 172.21.15.169:0/2824294745 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f58ac04d200 con 0x7f58bc107b50 2024-03-28T00:47:22.251 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.131+0000 7f58c1075700 1 --2- 172.21.15.169:0/2824294745 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f58a8038f20 0x7f58a803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:22.251 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.131+0000 7f58c1075700 1 --2- 172.21.15.169:0/2824294745 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f58a8038f20 0x7f58a803b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f58b8006fd0 tx=0x7f58b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:22.251 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.136+0000 7f58b2ffd700 1 -- 172.21.15.169:0/2824294745 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f58ac02d960 con 0x7f58bc107b50 2024-03-28T00:47:22.251 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:17.332+0000 7f58c3ada700 1 -- 172.21.15.169:0/2824294745 --> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi169", "addr": "172.21.15.169", "target": ["mon-mgr", ""]}) v1 -- 0x7f58bc1043c0 con 0x7f58a8038f20 2024-03-28T00:47:22.251 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:21.618+0000 7f58b2ffd700 1 -- 172.21.15.169:0/2824294745 <== mgr.14120 v2:172.21.15.169:6800/3995234407 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f58bc1043c0 con 0x7f58a8038f20 2024-03-28T00:47:22.251 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:21.621+0000 7f58c3ada700 1 -- 172.21.15.169:0/2824294745 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f58a8038f20 msgr2=0x7f58a803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:22.252 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:21.621+0000 7f58c3ada700 1 --2- 172.21.15.169:0/2824294745 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f58a8038f20 0x7f58a803b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f58b8006fd0 tx=0x7f58b8006e40 comp rx=0 tx=0).stop 2024-03-28T00:47:22.252 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:21.621+0000 7f58c3ada700 1 -- 172.21.15.169:0/2824294745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58bc107b50 msgr2=0x7f58bc19bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:22.252 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:21.621+0000 7f58c3ada700 1 --2- 172.21.15.169:0/2824294745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58bc107b50 0x7f58bc19bb70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009fb0 tx=0x7f58ac011bf0 comp rx=0 tx=0).stop 2024-03-28T00:47:22.252 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:21.621+0000 7f58c3ada700 1 -- 172.21.15.169:0/2824294745 shutdown_connections 2024-03-28T00:47:22.252 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:21.621+0000 7f58c3ada700 1 --2- 172.21.15.169:0/2824294745 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f58a8038f20 0x7f58a803b3e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:22.252 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:21.621+0000 7f58c3ada700 1 --2- 172.21.15.169:0/2824294745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58bc107b50 0x7f58bc19bb70 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:22.252 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:21.621+0000 7f58c3ada700 1 -- 172.21.15.169:0/2824294745 >> 172.21.15.169:0/2824294745 conn(0x7f58bc103150 msgr2=0x7f58bc103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:22.252 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:21.621+0000 7f58c3ada700 1 -- 172.21.15.169:0/2824294745 shutdown_connections 2024-03-28T00:47:22.252 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:21.621+0000 7f58c3ada700 1 -- 172.21.15.169:0/2824294745 wait complete. 2024-03-28T00:47:22.252 INFO:teuthology.orchestra.run.smithi169.stdout:Deploying unmanaged mon service... 2024-03-28T00:47:22.819 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:22 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' 2024-03-28T00:47:22.819 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:22 smithi169 ceph-mon[32274]: Added host smithi169 2024-03-28T00:47:22.819 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:22 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:47:23.627 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-28T00:47:23.627 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.815+0000 7fec12f86700 1 Processor -- start 2024-03-28T00:47:23.628 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.815+0000 7fec12f86700 1 -- start start 2024-03-28T00:47:23.628 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.815+0000 7fec12f86700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec0c106430 0x7fec0c106850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:23.628 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.815+0000 7fec12f86700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c07be50 con 0x7fec0c106430 2024-03-28T00:47:23.628 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.815+0000 7fec10d22700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec0c106430 0x7fec0c106850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:23.628 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.815+0000 7fec10d22700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec0c106430 0x7fec0c106850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:37974/0 (socket says 172.21.15.169:37974) 2024-03-28T00:47:23.628 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.815+0000 7fec10d22700 1 -- 172.21.15.169:0/142965959 learned_addr learned my addr 172.21.15.169:0/142965959 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:23.628 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.816+0000 7fec10d22700 1 -- 172.21.15.169:0/142965959 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0c106d90 con 0x7fec0c106430 2024-03-28T00:47:23.628 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.816+0000 7fec10d22700 1 --2- 172.21.15.169:0/142965959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec0c106430 0x7fec0c106850 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fec000098a0 tx=0x7fec00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f26b5827012f2e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:23.628 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.816+0000 7fec0b7fe700 1 -- 172.21.15.169:0/142965959 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec00004160 con 0x7fec0c106430 2024-03-28T00:47:23.628 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.816+0000 7fec0b7fe700 1 -- 172.21.15.169:0/142965959 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec00004300 con 0x7fec0c106430 2024-03-28T00:47:23.628 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.816+0000 7fec0b7fe700 1 -- 172.21.15.169:0/142965959 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec00011b50 con 0x7fec0c106430 2024-03-28T00:47:23.628 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.816+0000 7fec12f86700 1 -- 172.21.15.169:0/142965959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec0c106430 msgr2=0x7fec0c106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:23.629 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.816+0000 7fec12f86700 1 --2- 172.21.15.169:0/142965959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec0c106430 0x7fec0c106850 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fec000098a0 tx=0x7fec00009bb0 comp rx=0 tx=0).stop 2024-03-28T00:47:23.629 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.817+0000 7fec12f86700 1 -- 172.21.15.169:0/142965959 shutdown_connections 2024-03-28T00:47:23.629 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.817+0000 7fec12f86700 1 --2- 172.21.15.169:0/142965959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec0c106430 0x7fec0c106850 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:23.629 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.817+0000 7fec12f86700 1 -- 172.21.15.169:0/142965959 >> 172.21.15.169:0/142965959 conn(0x7fec0c102030 msgr2=0x7fec0c104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:23.629 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.817+0000 7fec12f86700 1 -- 172.21.15.169:0/142965959 shutdown_connections 2024-03-28T00:47:23.629 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.817+0000 7fec12f86700 1 -- 172.21.15.169:0/142965959 wait complete. 2024-03-28T00:47:23.629 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.817+0000 7fec12f86700 1 Processor -- start 2024-03-28T00:47:23.629 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.817+0000 7fec12f86700 1 -- start start 2024-03-28T00:47:23.630 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.817+0000 7fec12f86700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec0c106430 0x7fec0c19dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:23.630 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.817+0000 7fec12f86700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c107e20 con 0x7fec0c106430 2024-03-28T00:47:23.630 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.817+0000 7fec10d22700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec0c106430 0x7fec0c19dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:23.630 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.817+0000 7fec10d22700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec0c106430 0x7fec0c19dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:37990/0 (socket says 172.21.15.169:37990) 2024-03-28T00:47:23.630 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.817+0000 7fec10d22700 1 -- 172.21.15.169:0/3677928697 learned_addr learned my addr 172.21.15.169:0/3677928697 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:23.630 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.818+0000 7fec10d22700 1 -- 172.21.15.169:0/3677928697 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0c19e270 con 0x7fec0c106430 2024-03-28T00:47:23.630 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.818+0000 7fec10d22700 1 --2- 172.21.15.169:0/3677928697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec0c106430 0x7fec0c19dd30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fec00009fb0 tx=0x7fec00011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:23.630 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.818+0000 7fec09ffb700 1 -- 172.21.15.169:0/3677928697 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec000038b0 con 0x7fec0c106430 2024-03-28T00:47:23.630 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.818+0000 7fec09ffb700 1 -- 172.21.15.169:0/3677928697 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec00011e50 con 0x7fec0c106430 2024-03-28T00:47:23.630 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.818+0000 7fec12f86700 1 -- 172.21.15.169:0/3677928697 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec0c19e500 con 0x7fec0c106430 2024-03-28T00:47:23.630 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.818+0000 7fec09ffb700 1 -- 172.21.15.169:0/3677928697 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec00019420 con 0x7fec0c106430 2024-03-28T00:47:23.631 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.818+0000 7fec12f86700 1 -- 172.21.15.169:0/3677928697 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec0c1a11f0 con 0x7fec0c106430 2024-03-28T00:47:23.631 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.819+0000 7fec09ffb700 1 -- 172.21.15.169:0/3677928697 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fec00019720 con 0x7fec0c106430 2024-03-28T00:47:23.631 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.819+0000 7fec12f86700 1 -- 172.21.15.169:0/3677928697 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec0c0637c0 con 0x7fec0c106430 2024-03-28T00:47:23.631 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.819+0000 7fec09ffb700 1 --2- 172.21.15.169:0/3677928697 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7febfc038ed0 0x7febfc03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:23.631 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.819+0000 7fec09ffb700 1 -- 172.21.15.169:0/3677928697 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fec0004d210 con 0x7fec0c106430 2024-03-28T00:47:23.631 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.819+0000 7fec0bfff700 1 --2- 172.21.15.169:0/3677928697 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7febfc038ed0 0x7febfc03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:23.631 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.820+0000 7fec0bfff700 1 --2- 172.21.15.169:0/3677928697 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7febfc038ed0 0x7febfc03b390 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7febf8006fd0 tx=0x7febf8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:23.631 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:22.824+0000 7fec09ffb700 1 -- 172.21.15.169:0/3677928697 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fec0004cb60 con 0x7fec0c106430 2024-03-28T00:47:23.631 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:23.048+0000 7fec12f86700 1 -- 172.21.15.169:0/3677928697 --> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fec0c0008d0 con 0x7febfc038ed0 2024-03-28T00:47:23.632 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:23.055+0000 7fec09ffb700 1 -- 172.21.15.169:0/3677928697 <== mgr.14120 v2:172.21.15.169:6800/3995234407 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fec0c0008d0 con 0x7febfc038ed0 2024-03-28T00:47:23.632 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:23.057+0000 7fec12f86700 1 -- 172.21.15.169:0/3677928697 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7febfc038ed0 msgr2=0x7febfc03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:23.632 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:23.057+0000 7fec12f86700 1 --2- 172.21.15.169:0/3677928697 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7febfc038ed0 0x7febfc03b390 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7febf8006fd0 tx=0x7febf8006e40 comp rx=0 tx=0).stop 2024-03-28T00:47:23.632 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:23.057+0000 7fec12f86700 1 -- 172.21.15.169:0/3677928697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec0c106430 msgr2=0x7fec0c19dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:23.632 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:23.057+0000 7fec12f86700 1 --2- 172.21.15.169:0/3677928697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec0c106430 0x7fec0c19dd30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fec00009fb0 tx=0x7fec00011bf0 comp rx=0 tx=0).stop 2024-03-28T00:47:23.632 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:23.057+0000 7fec12f86700 1 -- 172.21.15.169:0/3677928697 shutdown_connections 2024-03-28T00:47:23.632 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:23.057+0000 7fec12f86700 1 --2- 172.21.15.169:0/3677928697 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7febfc038ed0 0x7febfc03b390 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:23.632 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:23.057+0000 7fec12f86700 1 --2- 172.21.15.169:0/3677928697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec0c106430 0x7fec0c19dd30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:23.632 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:23.057+0000 7fec12f86700 1 -- 172.21.15.169:0/3677928697 >> 172.21.15.169:0/3677928697 conn(0x7fec0c102030 msgr2=0x7fec0c1948e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:23.632 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:23.058+0000 7fec12f86700 1 -- 172.21.15.169:0/3677928697 shutdown_connections 2024-03-28T00:47:23.632 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:23.058+0000 7fec12f86700 1 -- 172.21.15.169:0/3677928697 wait complete. 2024-03-28T00:47:23.633 INFO:teuthology.orchestra.run.smithi169.stdout:Deploying unmanaged mgr service... 2024-03-28T00:47:24.241 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:24 smithi169 ceph-mon[32274]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:47:24.241 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:24 smithi169 ceph-mon[32274]: Saving service mon spec with placement count:5 2024-03-28T00:47:24.241 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:24 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' 2024-03-28T00:47:25.357 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-28T00:47:25.357 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.015+0000 7f326bdc6700 1 Processor -- start 2024-03-28T00:47:25.357 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.015+0000 7f326bdc6700 1 -- start start 2024-03-28T00:47:25.357 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.015+0000 7f326bdc6700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f326407cd90 0x7f326407b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:25.357 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.015+0000 7f326bdc6700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326407b7c0 con 0x7f326407cd90 2024-03-28T00:47:25.357 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.015+0000 7f3269b62700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f326407cd90 0x7f326407b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:25.357 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.015+0000 7f3269b62700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f326407cd90 0x7f326407b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38004/0 (socket says 172.21.15.169:38004) 2024-03-28T00:47:25.357 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.015+0000 7f3269b62700 1 -- 172.21.15.169:0/3136460589 learned_addr learned my addr 172.21.15.169:0/3136460589 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:25.357 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.016+0000 7f3269b62700 1 -- 172.21.15.169:0/3136460589 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326407b940 con 0x7f326407cd90 2024-03-28T00:47:25.357 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.016+0000 7f3269b62700 1 --2- 172.21.15.169:0/3136460589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f326407cd90 0x7f326407b1f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f32600098a0 tx=0x7f3260009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=247390d5ac85c6e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:25.358 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.016+0000 7f3268b60700 1 -- 172.21.15.169:0/3136460589 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3260004160 con 0x7f326407cd90 2024-03-28T00:47:25.358 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.016+0000 7f3268b60700 1 -- 172.21.15.169:0/3136460589 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3260004300 con 0x7f326407cd90 2024-03-28T00:47:25.358 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.016+0000 7f3268b60700 1 -- 172.21.15.169:0/3136460589 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3260011b50 con 0x7f326407cd90 2024-03-28T00:47:25.358 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.016+0000 7f326bdc6700 1 -- 172.21.15.169:0/3136460589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f326407cd90 msgr2=0x7f326407b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:25.358 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.016+0000 7f326bdc6700 1 --2- 172.21.15.169:0/3136460589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f326407cd90 0x7f326407b1f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f32600098a0 tx=0x7f3260009bb0 comp rx=0 tx=0).stop 2024-03-28T00:47:25.358 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.017+0000 7f326bdc6700 1 -- 172.21.15.169:0/3136460589 shutdown_connections 2024-03-28T00:47:25.358 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.017+0000 7f326bdc6700 1 --2- 172.21.15.169:0/3136460589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f326407cd90 0x7f326407b1f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:25.358 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.017+0000 7f326bdc6700 1 -- 172.21.15.169:0/3136460589 >> 172.21.15.169:0/3136460589 conn(0x7f3264103080 msgr2=0x7f32641054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:25.358 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.017+0000 7f326bdc6700 1 -- 172.21.15.169:0/3136460589 shutdown_connections 2024-03-28T00:47:25.358 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.017+0000 7f326bdc6700 1 -- 172.21.15.169:0/3136460589 wait complete. 2024-03-28T00:47:25.358 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.017+0000 7f326bdc6700 1 Processor -- start 2024-03-28T00:47:25.358 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.017+0000 7f326bdc6700 1 -- start start 2024-03-28T00:47:25.358 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.017+0000 7f326bdc6700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f326407cd90 0x7f326419dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:25.359 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.017+0000 7f326bdc6700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326410e110 con 0x7f326407cd90 2024-03-28T00:47:25.359 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.017+0000 7f3269b62700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f326407cd90 0x7f326419dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:25.359 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.017+0000 7f3269b62700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f326407cd90 0x7f326419dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38006/0 (socket says 172.21.15.169:38006) 2024-03-28T00:47:25.359 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.017+0000 7f3269b62700 1 -- 172.21.15.169:0/2845188584 learned_addr learned my addr 172.21.15.169:0/2845188584 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:25.359 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.018+0000 7f3269b62700 1 -- 172.21.15.169:0/2845188584 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326419e1c0 con 0x7f326407cd90 2024-03-28T00:47:25.359 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.018+0000 7f3269b62700 1 --2- 172.21.15.169:0/2845188584 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f326407cd90 0x7f326419dc80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3260011670 tx=0x7f3260011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:25.359 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.018+0000 7f325affd700 1 -- 172.21.15.169:0/2845188584 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32600039c0 con 0x7f326407cd90 2024-03-28T00:47:25.359 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.018+0000 7f325affd700 1 -- 172.21.15.169:0/2845188584 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3260011e50 con 0x7f326407cd90 2024-03-28T00:47:25.359 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.018+0000 7f326bdc6700 1 -- 172.21.15.169:0/2845188584 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f326419e450 con 0x7f326407cd90 2024-03-28T00:47:25.359 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.018+0000 7f325affd700 1 -- 172.21.15.169:0/2845188584 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f326001a5c0 con 0x7f326407cd90 2024-03-28T00:47:25.359 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.018+0000 7f326bdc6700 1 -- 172.21.15.169:0/2845188584 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32641a1140 con 0x7f326407cd90 2024-03-28T00:47:25.359 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.019+0000 7f325affd700 1 -- 172.21.15.169:0/2845188584 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f3260024430 con 0x7f326407cd90 2024-03-28T00:47:25.359 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.019+0000 7f326bdc6700 1 -- 172.21.15.169:0/2845188584 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32640637c0 con 0x7f326407cd90 2024-03-28T00:47:25.359 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.019+0000 7f325affd700 1 --2- 172.21.15.169:0/2845188584 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f3250038f20 0x7f325003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:25.359 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.019+0000 7f325affd700 1 -- 172.21.15.169:0/2845188584 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f326004d260 con 0x7f326407cd90 2024-03-28T00:47:25.359 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.019+0000 7f3269361700 1 --2- 172.21.15.169:0/2845188584 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f3250038f20 0x7f325003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:25.360 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.020+0000 7f3269361700 1 --2- 172.21.15.169:0/2845188584 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f3250038f20 0x7f325003b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3254006fd0 tx=0x7f3254006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:25.360 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.024+0000 7f325affd700 1 -- 172.21.15.169:0/2845188584 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f326002d910 con 0x7f326407cd90 2024-03-28T00:47:25.360 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.244+0000 7f326bdc6700 1 -- 172.21.15.169:0/2845188584 --> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f32641042d0 con 0x7f3250038f20 2024-03-28T00:47:25.360 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.252+0000 7f325affd700 1 -- 172.21.15.169:0/2845188584 <== mgr.14120 v2:172.21.15.169:6800/3995234407 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f32641042d0 con 0x7f3250038f20 2024-03-28T00:47:25.360 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.254+0000 7f326bdc6700 1 -- 172.21.15.169:0/2845188584 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f3250038f20 msgr2=0x7f325003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:25.360 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.254+0000 7f326bdc6700 1 --2- 172.21.15.169:0/2845188584 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f3250038f20 0x7f325003b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3254006fd0 tx=0x7f3254006e40 comp rx=0 tx=0).stop 2024-03-28T00:47:25.360 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.254+0000 7f326bdc6700 1 -- 172.21.15.169:0/2845188584 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f326407cd90 msgr2=0x7f326419dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:25.360 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.254+0000 7f326bdc6700 1 --2- 172.21.15.169:0/2845188584 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f326407cd90 0x7f326419dc80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3260011670 tx=0x7f3260011b60 comp rx=0 tx=0).stop 2024-03-28T00:47:25.360 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.254+0000 7f326bdc6700 1 -- 172.21.15.169:0/2845188584 shutdown_connections 2024-03-28T00:47:25.360 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.254+0000 7f326bdc6700 1 --2- 172.21.15.169:0/2845188584 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f3250038f20 0x7f325003b3e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:25.360 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.254+0000 7f326bdc6700 1 --2- 172.21.15.169:0/2845188584 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f326407cd90 0x7f326419dc80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:25.360 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.255+0000 7f326bdc6700 1 -- 172.21.15.169:0/2845188584 >> 172.21.15.169:0/2845188584 conn(0x7f3264103080 msgr2=0x7f3264103bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:25.360 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.255+0000 7f326bdc6700 1 -- 172.21.15.169:0/2845188584 shutdown_connections 2024-03-28T00:47:25.360 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:24.255+0000 7f326bdc6700 1 -- 172.21.15.169:0/2845188584 wait complete. 2024-03-28T00:47:25.457 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:25 smithi169 ceph-mon[32274]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:47:25.458 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:25 smithi169 ceph-mon[32274]: Saving service mgr spec with placement count:2 2024-03-28T00:47:25.458 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:25 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' 2024-03-28T00:47:25.458 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:25 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' 2024-03-28T00:47:26.538 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.804+0000 7fa904e60700 1 Processor -- start 2024-03-28T00:47:26.538 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.804+0000 7fa904e60700 1 -- start start 2024-03-28T00:47:26.539 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.804+0000 7fa904e60700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9001085b0 0x7fa90010a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:26.539 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.804+0000 7fa904e60700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90007be70 con 0x7fa9001085b0 2024-03-28T00:47:26.539 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.804+0000 7fa8fe59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9001085b0 0x7fa90010a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:26.539 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.804+0000 7fa8fe59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9001085b0 0x7fa90010a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38014/0 (socket says 172.21.15.169:38014) 2024-03-28T00:47:26.539 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.804+0000 7fa8fe59c700 1 -- 172.21.15.169:0/2000394717 learned_addr learned my addr 172.21.15.169:0/2000394717 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:26.539 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.805+0000 7fa8fe59c700 1 -- 172.21.15.169:0/2000394717 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90010aee0 con 0x7fa9001085b0 2024-03-28T00:47:26.539 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.805+0000 7fa8fe59c700 1 --2- 172.21.15.169:0/2000394717 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9001085b0 0x7fa90010a9a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa8f00098a0 tx=0x7fa8f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=315ee296d3617a97 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:26.539 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.805+0000 7fa8fd59a700 1 -- 172.21.15.169:0/2000394717 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8f0004160 con 0x7fa9001085b0 2024-03-28T00:47:26.539 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.805+0000 7fa8fd59a700 1 -- 172.21.15.169:0/2000394717 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8f0004300 con 0x7fa9001085b0 2024-03-28T00:47:26.539 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.805+0000 7fa8fd59a700 1 -- 172.21.15.169:0/2000394717 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8f0011b20 con 0x7fa9001085b0 2024-03-28T00:47:26.539 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.806+0000 7fa904e60700 1 -- 172.21.15.169:0/2000394717 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9001085b0 msgr2=0x7fa90010a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:26.539 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.806+0000 7fa904e60700 1 --2- 172.21.15.169:0/2000394717 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9001085b0 0x7fa90010a9a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa8f00098a0 tx=0x7fa8f0009bb0 comp rx=0 tx=0).stop 2024-03-28T00:47:26.539 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.806+0000 7fa904e60700 1 -- 172.21.15.169:0/2000394717 shutdown_connections 2024-03-28T00:47:26.540 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.806+0000 7fa904e60700 1 --2- 172.21.15.169:0/2000394717 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9001085b0 0x7fa90010a9a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:26.540 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.806+0000 7fa904e60700 1 -- 172.21.15.169:0/2000394717 >> 172.21.15.169:0/2000394717 conn(0x7fa900101f80 msgr2=0x7fa9001043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:26.540 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.806+0000 7fa904e60700 1 -- 172.21.15.169:0/2000394717 shutdown_connections 2024-03-28T00:47:26.540 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.806+0000 7fa904e60700 1 -- 172.21.15.169:0/2000394717 wait complete. 2024-03-28T00:47:26.540 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.807+0000 7fa904e60700 1 Processor -- start 2024-03-28T00:47:26.540 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.807+0000 7fa904e60700 1 -- start start 2024-03-28T00:47:26.540 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.807+0000 7fa904e60700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9001085b0 0x7fa90019b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:26.540 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.807+0000 7fa904e60700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90010bef0 con 0x7fa9001085b0 2024-03-28T00:47:26.540 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.807+0000 7fa8fe59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9001085b0 0x7fa90019b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:26.540 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.807+0000 7fa8fe59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9001085b0 0x7fa90019b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38024/0 (socket says 172.21.15.169:38024) 2024-03-28T00:47:26.540 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.807+0000 7fa8fe59c700 1 -- 172.21.15.169:0/310991438 learned_addr learned my addr 172.21.15.169:0/310991438 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:26.540 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.807+0000 7fa8fe59c700 1 -- 172.21.15.169:0/310991438 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90019bf30 con 0x7fa9001085b0 2024-03-28T00:47:26.540 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.807+0000 7fa8fe59c700 1 --2- 172.21.15.169:0/310991438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9001085b0 0x7fa90019b9f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa8f00117e0 tx=0x7fa8f0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:26.540 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.807+0000 7fa8ef7fe700 1 -- 172.21.15.169:0/310991438 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8f0003bf0 con 0x7fa9001085b0 2024-03-28T00:47:26.540 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.808+0000 7fa8ef7fe700 1 -- 172.21.15.169:0/310991438 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8f001a910 con 0x7fa9001085b0 2024-03-28T00:47:26.541 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.808+0000 7fa904e60700 1 -- 172.21.15.169:0/310991438 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa90019c1c0 con 0x7fa9001085b0 2024-03-28T00:47:26.541 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.808+0000 7fa8ef7fe700 1 -- 172.21.15.169:0/310991438 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8f0024970 con 0x7fa9001085b0 2024-03-28T00:47:26.541 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.808+0000 7fa904e60700 1 -- 172.21.15.169:0/310991438 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa90019c620 con 0x7fa9001085b0 2024-03-28T00:47:26.541 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.808+0000 7fa8ef7fe700 1 -- 172.21.15.169:0/310991438 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fa8f0022020 con 0x7fa9001085b0 2024-03-28T00:47:26.541 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.809+0000 7fa8ef7fe700 1 --2- 172.21.15.169:0/310991438 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7fa8e8038ed0 0x7fa8e803b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:26.541 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.809+0000 7fa904e60700 1 -- 172.21.15.169:0/310991438 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9000637c0 con 0x7fa9001085b0 2024-03-28T00:47:26.541 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.809+0000 7fa8ef7fe700 1 -- 172.21.15.169:0/310991438 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa8f004e1c0 con 0x7fa9001085b0 2024-03-28T00:47:26.541 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.809+0000 7fa8fdd9b700 1 --2- 172.21.15.169:0/310991438 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7fa8e8038ed0 0x7fa8e803b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:26.541 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.809+0000 7fa8fdd9b700 1 --2- 172.21.15.169:0/310991438 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7fa8e8038ed0 0x7fa8e803b390 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4006fd0 tx=0x7fa8f4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:26.541 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:25.814+0000 7fa8ef7fe700 1 -- 172.21.15.169:0/310991438 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa8f0011e20 con 0x7fa9001085b0 2024-03-28T00:47:26.541 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.019+0000 7fa904e60700 1 -- 172.21.15.169:0/310991438 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fa90019ef70 con 0x7fa9001085b0 2024-03-28T00:47:26.541 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.024+0000 7fa8ef7fe700 1 -- 172.21.15.169:0/310991438 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fa8f0019c70 con 0x7fa9001085b0 2024-03-28T00:47:26.541 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.026+0000 7fa904e60700 1 -- 172.21.15.169:0/310991438 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7fa8e8038ed0 msgr2=0x7fa8e803b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:26.541 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.026+0000 7fa904e60700 1 --2- 172.21.15.169:0/310991438 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7fa8e8038ed0 0x7fa8e803b390 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4006fd0 tx=0x7fa8f4006e40 comp rx=0 tx=0).stop 2024-03-28T00:47:26.542 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.026+0000 7fa904e60700 1 -- 172.21.15.169:0/310991438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9001085b0 msgr2=0x7fa90019b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:26.542 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.026+0000 7fa904e60700 1 --2- 172.21.15.169:0/310991438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9001085b0 0x7fa90019b9f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa8f00117e0 tx=0x7fa8f0011c10 comp rx=0 tx=0).stop 2024-03-28T00:47:26.542 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.027+0000 7fa904e60700 1 -- 172.21.15.169:0/310991438 shutdown_connections 2024-03-28T00:47:26.542 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.027+0000 7fa904e60700 1 --2- 172.21.15.169:0/310991438 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7fa8e8038ed0 0x7fa8e803b390 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:26.542 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.027+0000 7fa904e60700 1 --2- 172.21.15.169:0/310991438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9001085b0 0x7fa90019b9f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:26.542 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.027+0000 7fa904e60700 1 -- 172.21.15.169:0/310991438 >> 172.21.15.169:0/310991438 conn(0x7fa900101f80 msgr2=0x7fa9001029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:26.542 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.027+0000 7fa904e60700 1 -- 172.21.15.169:0/310991438 shutdown_connections 2024-03-28T00:47:26.542 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.027+0000 7fa904e60700 1 -- 172.21.15.169:0/310991438 wait complete. 2024-03-28T00:47:26.875 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:26 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' 2024-03-28T00:47:26.875 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:26 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/310991438' entity='client.admin' 2024-03-28T00:47:27.830 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.890+0000 7f62c4963700 1 Processor -- start 2024-03-28T00:47:27.830 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.890+0000 7f62c4963700 1 -- start start 2024-03-28T00:47:27.831 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.891+0000 7f62c4963700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c01063d0 0x7f62c01067f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:27.831 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.891+0000 7f62c4963700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c0075880 con 0x7f62c01063d0 2024-03-28T00:47:27.831 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.891+0000 7f62be59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c01063d0 0x7f62c01067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:27.831 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.891+0000 7f62be59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c01063d0 0x7f62c01067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36302/0 (socket says 172.21.15.169:36302) 2024-03-28T00:47:27.831 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.891+0000 7f62be59c700 1 -- 172.21.15.169:0/4064663619 learned_addr learned my addr 172.21.15.169:0/4064663619 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:27.831 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.891+0000 7f62be59c700 1 -- 172.21.15.169:0/4064663619 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c0106d30 con 0x7f62c01063d0 2024-03-28T00:47:27.831 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.891+0000 7f62be59c700 1 --2- 172.21.15.169:0/4064663619 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c01063d0 0x7f62c01067f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f62b00098a0 tx=0x7f62b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b20574ea41dfae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:27.831 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.891+0000 7f62bd59a700 1 -- 172.21.15.169:0/4064663619 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f62b0004160 con 0x7f62c01063d0 2024-03-28T00:47:27.831 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.891+0000 7f62bd59a700 1 -- 172.21.15.169:0/4064663619 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62b0004300 con 0x7f62c01063d0 2024-03-28T00:47:27.831 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.891+0000 7f62bd59a700 1 -- 172.21.15.169:0/4064663619 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f62b0011b20 con 0x7f62c01063d0 2024-03-28T00:47:27.831 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.892+0000 7f62c4963700 1 -- 172.21.15.169:0/4064663619 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c01063d0 msgr2=0x7f62c01067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:27.831 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.892+0000 7f62c4963700 1 --2- 172.21.15.169:0/4064663619 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c01063d0 0x7f62c01067f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f62b00098a0 tx=0x7f62b0009bb0 comp rx=0 tx=0).stop 2024-03-28T00:47:27.831 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.892+0000 7f62c4963700 1 -- 172.21.15.169:0/4064663619 shutdown_connections 2024-03-28T00:47:27.831 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.892+0000 7f62c4963700 1 --2- 172.21.15.169:0/4064663619 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c01063d0 0x7f62c01067f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:27.832 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.892+0000 7f62c4963700 1 -- 172.21.15.169:0/4064663619 >> 172.21.15.169:0/4064663619 conn(0x7f62c0101fb0 msgr2=0x7f62c01043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:27.832 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.892+0000 7f62c4963700 1 -- 172.21.15.169:0/4064663619 shutdown_connections 2024-03-28T00:47:27.832 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.892+0000 7f62c4963700 1 -- 172.21.15.169:0/4064663619 wait complete. 2024-03-28T00:47:27.832 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.893+0000 7f62c4963700 1 Processor -- start 2024-03-28T00:47:27.832 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.893+0000 7f62c4963700 1 -- start start 2024-03-28T00:47:27.832 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.893+0000 7f62c4963700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c01063d0 0x7f62c019dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:27.832 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.893+0000 7f62c4963700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c0107e00 con 0x7f62c01063d0 2024-03-28T00:47:27.832 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.893+0000 7f62be59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c01063d0 0x7f62c019dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:27.832 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.893+0000 7f62be59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c01063d0 0x7f62c019dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36318/0 (socket says 172.21.15.169:36318) 2024-03-28T00:47:27.832 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.893+0000 7f62be59c700 1 -- 172.21.15.169:0/2442260660 learned_addr learned my addr 172.21.15.169:0/2442260660 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:27.832 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.893+0000 7f62be59c700 1 -- 172.21.15.169:0/2442260660 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c019e1b0 con 0x7f62c01063d0 2024-03-28T00:47:27.832 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.893+0000 7f62be59c700 1 --2- 172.21.15.169:0/2442260660 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c01063d0 0x7f62c019dc70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f62b0011b60 tx=0x7f62b0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:27.832 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.894+0000 7f62af7fe700 1 -- 172.21.15.169:0/2442260660 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f62b0003b60 con 0x7f62c01063d0 2024-03-28T00:47:27.833 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.894+0000 7f62af7fe700 1 -- 172.21.15.169:0/2442260660 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62b001a910 con 0x7f62c01063d0 2024-03-28T00:47:27.833 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.894+0000 7f62c4963700 1 -- 172.21.15.169:0/2442260660 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c019e440 con 0x7f62c01063d0 2024-03-28T00:47:27.833 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.894+0000 7f62af7fe700 1 -- 172.21.15.169:0/2442260660 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f62b0024930 con 0x7f62c01063d0 2024-03-28T00:47:27.833 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.894+0000 7f62c4963700 1 -- 172.21.15.169:0/2442260660 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c019e8a0 con 0x7f62c01063d0 2024-03-28T00:47:27.833 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.894+0000 7f62af7fe700 1 -- 172.21.15.169:0/2442260660 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f62b0022020 con 0x7f62c01063d0 2024-03-28T00:47:27.833 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.895+0000 7f62c4963700 1 -- 172.21.15.169:0/2442260660 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c00637c0 con 0x7f62c01063d0 2024-03-28T00:47:27.833 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.895+0000 7f62af7fe700 1 --2- 172.21.15.169:0/2442260660 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f62a8038f70 0x7f62a803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:27.833 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.895+0000 7f62af7fe700 1 -- 172.21.15.169:0/2442260660 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f62b004d050 con 0x7f62c01063d0 2024-03-28T00:47:27.833 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.895+0000 7f62bdd9b700 1 --2- 172.21.15.169:0/2442260660 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f62a8038f70 0x7f62a803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:27.833 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.895+0000 7f62bdd9b700 1 --2- 172.21.15.169:0/2442260660 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f62a8038f70 0x7f62a803b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f62b4006fd0 tx=0x7f62b4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:27.833 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:26.900+0000 7f62af7fe700 1 -- 172.21.15.169:0/2442260660 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f62b001abe0 con 0x7f62c01063d0 2024-03-28T00:47:27.833 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:27.123+0000 7f62c4963700 1 -- 172.21.15.169:0/2442260660 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f62c0062da0 con 0x7f62c01063d0 2024-03-28T00:47:27.833 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:27.126+0000 7f62af7fe700 1 -- 172.21.15.169:0/2442260660 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f62b002d7d0 con 0x7f62c01063d0 2024-03-28T00:47:27.833 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:27.128+0000 7f62c4963700 1 -- 172.21.15.169:0/2442260660 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f62a8038f70 msgr2=0x7f62a803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:27.834 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:27.128+0000 7f62c4963700 1 --2- 172.21.15.169:0/2442260660 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f62a8038f70 0x7f62a803b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f62b4006fd0 tx=0x7f62b4006e40 comp rx=0 tx=0).stop 2024-03-28T00:47:27.834 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:27.128+0000 7f62c4963700 1 -- 172.21.15.169:0/2442260660 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c01063d0 msgr2=0x7f62c019dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:27.834 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:27.128+0000 7f62c4963700 1 --2- 172.21.15.169:0/2442260660 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c01063d0 0x7f62c019dc70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f62b0011b60 tx=0x7f62b0011c40 comp rx=0 tx=0).stop 2024-03-28T00:47:27.834 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:27.128+0000 7f62c4963700 1 -- 172.21.15.169:0/2442260660 shutdown_connections 2024-03-28T00:47:27.834 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:27.128+0000 7f62c4963700 1 --2- 172.21.15.169:0/2442260660 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f62a8038f70 0x7f62a803b430 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:27.834 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:27.128+0000 7f62c4963700 1 --2- 172.21.15.169:0/2442260660 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c01063d0 0x7f62c019dc70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:27.834 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:27.128+0000 7f62c4963700 1 -- 172.21.15.169:0/2442260660 >> 172.21.15.169:0/2442260660 conn(0x7f62c0101fb0 msgr2=0x7f62c01948e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:27.834 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:27.129+0000 7f62c4963700 1 -- 172.21.15.169:0/2442260660 shutdown_connections 2024-03-28T00:47:27.834 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:27.129+0000 7f62c4963700 1 -- 172.21.15.169:0/2442260660 wait complete. 2024-03-28T00:47:27.834 INFO:teuthology.orchestra.run.smithi169.stdout:Enabling the dashboard module... 2024-03-28T00:47:28.200 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:28 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2442260660' entity='client.admin' 2024-03-28T00:47:29.449 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:29 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2102858704' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-28T00:47:29.449 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:29 smithi169 ceph-mon[32274]: from='mgr.14120 172.21.15.169:0/2805663798' entity='mgr.x' 2024-03-28T00:47:29.449 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:29 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: ignoring --setuser ceph since I am not root 2024-03-28T00:47:29.449 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:29 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: ignoring --setgroup ceph since I am not root 2024-03-28T00:47:29.741 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.214+0000 7f800a1c6700 1 Processor -- start 2024-03-28T00:47:29.741 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.214+0000 7f800a1c6700 1 -- start start 2024-03-28T00:47:29.741 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.214+0000 7f800a1c6700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8004108620 0x7f800410aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:29.741 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.214+0000 7f800a1c6700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800407be70 con 0x7f8004108620 2024-03-28T00:47:29.742 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.214+0000 7f80037fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8004108620 0x7f800410aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:29.742 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.214+0000 7f80037fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8004108620 0x7f800410aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36330/0 (socket says 172.21.15.169:36330) 2024-03-28T00:47:29.742 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.214+0000 7f80037fe700 1 -- 172.21.15.169:0/3169925862 learned_addr learned my addr 172.21.15.169:0/3169925862 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:29.742 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.215+0000 7f80037fe700 1 -- 172.21.15.169:0/3169925862 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800410af50 con 0x7f8004108620 2024-03-28T00:47:29.742 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.215+0000 7f80037fe700 1 --2- 172.21.15.169:0/3169925862 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8004108620 0x7f800410aa10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7ff40098a0 tx=0x7f7ff4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=535077c4c28dca01 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:29.742 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.215+0000 7f80027fc700 1 -- 172.21.15.169:0/3169925862 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ff4004160 con 0x7f8004108620 2024-03-28T00:47:29.742 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.215+0000 7f80027fc700 1 -- 172.21.15.169:0/3169925862 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ff4004300 con 0x7f8004108620 2024-03-28T00:47:29.742 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.215+0000 7f80027fc700 1 -- 172.21.15.169:0/3169925862 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ff4011b20 con 0x7f8004108620 2024-03-28T00:47:29.742 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.215+0000 7f800a1c6700 1 -- 172.21.15.169:0/3169925862 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8004108620 msgr2=0x7f800410aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:29.742 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.215+0000 7f800a1c6700 1 --2- 172.21.15.169:0/3169925862 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8004108620 0x7f800410aa10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7ff40098a0 tx=0x7f7ff4009bb0 comp rx=0 tx=0).stop 2024-03-28T00:47:29.742 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.216+0000 7f800a1c6700 1 -- 172.21.15.169:0/3169925862 shutdown_connections 2024-03-28T00:47:29.742 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.216+0000 7f800a1c6700 1 --2- 172.21.15.169:0/3169925862 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8004108620 0x7f800410aa10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:29.742 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.216+0000 7f800a1c6700 1 -- 172.21.15.169:0/3169925862 >> 172.21.15.169:0/3169925862 conn(0x7f8004102030 msgr2=0x7f8004104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:29.743 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.216+0000 7f800a1c6700 1 -- 172.21.15.169:0/3169925862 shutdown_connections 2024-03-28T00:47:29.743 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.216+0000 7f800a1c6700 1 -- 172.21.15.169:0/3169925862 wait complete. 2024-03-28T00:47:29.743 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.216+0000 7f800a1c6700 1 Processor -- start 2024-03-28T00:47:29.743 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.216+0000 7f800a1c6700 1 -- start start 2024-03-28T00:47:29.743 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.216+0000 7f800a1c6700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8004108620 0x7f8004107ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:29.743 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.216+0000 7f800a1c6700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800410bf60 con 0x7f8004108620 2024-03-28T00:47:29.743 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.217+0000 7f80037fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8004108620 0x7f8004107ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:29.743 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.217+0000 7f80037fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8004108620 0x7f8004107ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36336/0 (socket says 172.21.15.169:36336) 2024-03-28T00:47:29.743 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.217+0000 7f80037fe700 1 -- 172.21.15.169:0/2102858704 learned_addr learned my addr 172.21.15.169:0/2102858704 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:29.743 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.217+0000 7f80037fe700 1 -- 172.21.15.169:0/2102858704 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80041083e0 con 0x7f8004108620 2024-03-28T00:47:29.743 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.217+0000 7f80037fe700 1 --2- 172.21.15.169:0/2102858704 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8004108620 0x7f8004107ea0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7ff40116a0 tx=0x7f7ff4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:29.743 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.217+0000 7f8000ff9700 1 -- 172.21.15.169:0/2102858704 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ff4003a40 con 0x7f8004108620 2024-03-28T00:47:29.743 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.217+0000 7f8000ff9700 1 -- 172.21.15.169:0/2102858704 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ff4003be0 con 0x7f8004108620 2024-03-28T00:47:29.744 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.217+0000 7f800a1c6700 1 -- 172.21.15.169:0/2102858704 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80041065d0 con 0x7f8004108620 2024-03-28T00:47:29.744 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.217+0000 7f8000ff9700 1 -- 172.21.15.169:0/2102858704 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ff401a750 con 0x7f8004108620 2024-03-28T00:47:29.744 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.217+0000 7f800a1c6700 1 -- 172.21.15.169:0/2102858704 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8004106a90 con 0x7f8004108620 2024-03-28T00:47:29.744 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.218+0000 7f8000ff9700 1 -- 172.21.15.169:0/2102858704 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f7ff401aa80 con 0x7f8004108620 2024-03-28T00:47:29.744 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.218+0000 7f800a1c6700 1 -- 172.21.15.169:0/2102858704 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80040637c0 con 0x7f8004108620 2024-03-28T00:47:29.744 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.218+0000 7f8000ff9700 1 --2- 172.21.15.169:0/2102858704 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f7fec038f20 0x7f7fec03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:29.744 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.218+0000 7f8000ff9700 1 -- 172.21.15.169:0/2102858704 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7ff404d5e0 con 0x7f8004108620 2024-03-28T00:47:29.744 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.218+0000 7f8002ffd700 1 --2- 172.21.15.169:0/2102858704 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f7fec038f20 0x7f7fec03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:29.744 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.219+0000 7f8002ffd700 1 --2- 172.21.15.169:0/2102858704 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f7fec038f20 0x7f7fec03b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8006fd0 tx=0x7f7ff8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:29.744 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.223+0000 7f8000ff9700 1 -- 172.21.15.169:0/2102858704 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7ff4011e20 con 0x7f8004108620 2024-03-28T00:47:29.744 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:28.479+0000 7f800a1c6700 1 -- 172.21.15.169:0/2102858704 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f8004106d10 con 0x7f8004108620 2024-03-28T00:47:29.744 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:29.129+0000 7f8000ff9700 1 -- 172.21.15.169:0/2102858704 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f7ff401ae20 con 0x7f8004108620 2024-03-28T00:47:29.744 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:29.131+0000 7f8000ff9700 1 -- 172.21.15.169:0/2102858704 <== mon.0 v2:172.21.15.169:3300/0 8 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f7ff402d910 con 0x7f8004108620 2024-03-28T00:47:29.744 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:29.133+0000 7f800a1c6700 1 -- 172.21.15.169:0/2102858704 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f7fec038f20 msgr2=0x7f7fec03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:29.745 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:29.133+0000 7f800a1c6700 1 --2- 172.21.15.169:0/2102858704 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f7fec038f20 0x7f7fec03b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8006fd0 tx=0x7f7ff8006e40 comp rx=0 tx=0).stop 2024-03-28T00:47:29.745 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:29.133+0000 7f800a1c6700 1 -- 172.21.15.169:0/2102858704 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8004108620 msgr2=0x7f8004107ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:29.745 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:29.133+0000 7f800a1c6700 1 --2- 172.21.15.169:0/2102858704 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8004108620 0x7f8004107ea0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7ff40116a0 tx=0x7f7ff4011ba0 comp rx=0 tx=0).stop 2024-03-28T00:47:29.745 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:29.133+0000 7f800a1c6700 1 -- 172.21.15.169:0/2102858704 shutdown_connections 2024-03-28T00:47:29.745 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:29.133+0000 7f800a1c6700 1 --2- 172.21.15.169:0/2102858704 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f7fec038f20 0x7f7fec03b3e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:29.745 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:29.133+0000 7f800a1c6700 1 --2- 172.21.15.169:0/2102858704 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8004108620 0x7f8004107ea0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:29.745 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:29.133+0000 7f800a1c6700 1 -- 172.21.15.169:0/2102858704 >> 172.21.15.169:0/2102858704 conn(0x7f8004102030 msgr2=0x7f8004102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:29.745 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:29.133+0000 7f800a1c6700 1 -- 172.21.15.169:0/2102858704 shutdown_connections 2024-03-28T00:47:29.745 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:29.133+0000 7f800a1c6700 1 -- 172.21.15.169:0/2102858704 wait complete. 2024-03-28T00:47:29.960 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:29 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:29.705+0000 7f52d4686200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T00:47:29.960 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:29 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:29.804+0000 7f52d4686200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T00:47:30.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:30 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2102858704' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-28T00:47:30.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:30 smithi169 ceph-mon[32274]: mgrmap e9: x(active, since 18s) 2024-03-28T00:47:30.706 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:30 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:30.465+0000 7f52d4686200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T00:47:30.706 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:30 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:30.558+0000 7f52d4686200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T00:47:30.998 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:30 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:30.703+0000 7f52d4686200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T00:47:30.998 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:30 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:30.798+0000 7f52d4686200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T00:47:30.998 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:30 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:30.881+0000 7f52d4686200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T00:47:31.001 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout { 2024-03-28T00:47:31.011 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-28T00:47:31.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T00:47:31.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T00:47:31.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T00:47:31.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout } 2024-03-28T00:47:31.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.139+0000 7fb59c5be700 1 Processor -- start 2024-03-28T00:47:31.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.139+0000 7fb59c5be700 1 -- start start 2024-03-28T00:47:31.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.140+0000 7fb59c5be700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb594108620 0x7fb59410aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:31.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.140+0000 7fb59c5be700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59407be70 con 0x7fb594108620 2024-03-28T00:47:31.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.140+0000 7fb59a35a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb594108620 0x7fb59410aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:31.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.140+0000 7fb59a35a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb594108620 0x7fb59410aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36364/0 (socket says 172.21.15.169:36364) 2024-03-28T00:47:31.012 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.140+0000 7fb59a35a700 1 -- 172.21.15.169:0/1317293868 learned_addr learned my addr 172.21.15.169:0/1317293868 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:31.013 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.140+0000 7fb59a35a700 1 -- 172.21.15.169:0/1317293868 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59410af50 con 0x7fb594108620 2024-03-28T00:47:31.013 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.140+0000 7fb59a35a700 1 --2- 172.21.15.169:0/1317293868 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb594108620 0x7fb59410aa10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb5900098a0 tx=0x7fb590009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d58c10ce672db1b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:31.013 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.140+0000 7fb599358700 1 -- 172.21.15.169:0/1317293868 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb590004160 con 0x7fb594108620 2024-03-28T00:47:31.013 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.140+0000 7fb599358700 1 -- 172.21.15.169:0/1317293868 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb590004300 con 0x7fb594108620 2024-03-28T00:47:31.013 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.140+0000 7fb599358700 1 -- 172.21.15.169:0/1317293868 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb590011b20 con 0x7fb594108620 2024-03-28T00:47:31.013 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.141+0000 7fb59c5be700 1 -- 172.21.15.169:0/1317293868 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb594108620 msgr2=0x7fb59410aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:31.013 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.141+0000 7fb59c5be700 1 --2- 172.21.15.169:0/1317293868 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb594108620 0x7fb59410aa10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb5900098a0 tx=0x7fb590009bb0 comp rx=0 tx=0).stop 2024-03-28T00:47:31.013 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.141+0000 7fb59c5be700 1 -- 172.21.15.169:0/1317293868 shutdown_connections 2024-03-28T00:47:31.013 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.141+0000 7fb59c5be700 1 --2- 172.21.15.169:0/1317293868 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb594108620 0x7fb59410aa10 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:31.013 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.141+0000 7fb59c5be700 1 -- 172.21.15.169:0/1317293868 >> 172.21.15.169:0/1317293868 conn(0x7fb594102030 msgr2=0x7fb594104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:31.014 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.141+0000 7fb59c5be700 1 -- 172.21.15.169:0/1317293868 shutdown_connections 2024-03-28T00:47:31.014 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.141+0000 7fb59c5be700 1 -- 172.21.15.169:0/1317293868 wait complete. 2024-03-28T00:47:31.014 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.142+0000 7fb59c5be700 1 Processor -- start 2024-03-28T00:47:31.014 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.142+0000 7fb59c5be700 1 -- start start 2024-03-28T00:47:31.014 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.142+0000 7fb59c5be700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb594108620 0x7fb594107ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:31.014 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.142+0000 7fb59c5be700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59410bf60 con 0x7fb594108620 2024-03-28T00:47:31.014 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.142+0000 7fb59a35a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb594108620 0x7fb594107ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:31.014 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.142+0000 7fb59a35a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb594108620 0x7fb594107ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36366/0 (socket says 172.21.15.169:36366) 2024-03-28T00:47:31.014 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.142+0000 7fb59a35a700 1 -- 172.21.15.169:0/2194834892 learned_addr learned my addr 172.21.15.169:0/2194834892 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:31.014 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.142+0000 7fb59a35a700 1 -- 172.21.15.169:0/2194834892 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5941083e0 con 0x7fb594108620 2024-03-28T00:47:31.014 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.143+0000 7fb59a35a700 1 --2- 172.21.15.169:0/2194834892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb594108620 0x7fb594107ea0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb5900116a0 tx=0x7fb590011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:31.015 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.143+0000 7fb58b7fe700 1 -- 172.21.15.169:0/2194834892 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb590003a40 con 0x7fb594108620 2024-03-28T00:47:31.015 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.143+0000 7fb59c5be700 1 -- 172.21.15.169:0/2194834892 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5941065d0 con 0x7fb594108620 2024-03-28T00:47:31.015 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.143+0000 7fb58b7fe700 1 -- 172.21.15.169:0/2194834892 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb590003be0 con 0x7fb594108620 2024-03-28T00:47:31.015 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.143+0000 7fb58b7fe700 1 -- 172.21.15.169:0/2194834892 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb59001a750 con 0x7fb594108620 2024-03-28T00:47:31.015 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.143+0000 7fb59c5be700 1 -- 172.21.15.169:0/2194834892 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb594106a90 con 0x7fb594108620 2024-03-28T00:47:31.015 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.144+0000 7fb58b7fe700 1 -- 172.21.15.169:0/2194834892 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fb59001aa50 con 0x7fb594108620 2024-03-28T00:47:31.015 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.144+0000 7fb58b7fe700 1 --2- 172.21.15.169:0/2194834892 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7fb580038f20 0x7fb58003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:31.015 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.144+0000 7fb58b7fe700 1 -- 172.21.15.169:0/2194834892 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb59004d5a0 con 0x7fb594108620 2024-03-28T00:47:31.016 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.144+0000 7fb599b59700 1 -- 172.21.15.169:0/2194834892 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7fb580038f20 msgr2=0x7fb58003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/3995234407 2024-03-28T00:47:31.016 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.144+0000 7fb599b59700 1 --2- 172.21.15.169:0/2194834892 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7fb580038f20 0x7fb58003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T00:47:31.016 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.144+0000 7fb59c5be700 1 -- 172.21.15.169:0/2194834892 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5940637c0 con 0x7fb594108620 2024-03-28T00:47:31.016 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.149+0000 7fb58b7fe700 1 -- 172.21.15.169:0/2194834892 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb590011e20 con 0x7fb594108620 2024-03-28T00:47:31.016 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.345+0000 7fb599b59700 1 -- 172.21.15.169:0/2194834892 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7fb580038f20 msgr2=0x7fb58003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/3995234407 2024-03-28T00:47:31.016 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.345+0000 7fb599b59700 1 --2- 172.21.15.169:0/2194834892 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7fb580038f20 0x7fb58003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T00:47:31.016 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.399+0000 7fb59c5be700 1 -- 172.21.15.169:0/2194834892 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fb594107110 con 0x7fb594108620 2024-03-28T00:47:31.016 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.400+0000 7fb58b7fe700 1 -- 172.21.15.169:0/2194834892 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fb59000f2e0 con 0x7fb594108620 2024-03-28T00:47:31.016 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.403+0000 7fb59c5be700 1 -- 172.21.15.169:0/2194834892 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7fb580038f20 msgr2=0x7fb58003b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T00:47:31.016 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.403+0000 7fb59c5be700 1 --2- 172.21.15.169:0/2194834892 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7fb580038f20 0x7fb58003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:31.016 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.403+0000 7fb59c5be700 1 -- 172.21.15.169:0/2194834892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb594108620 msgr2=0x7fb594107ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:31.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.403+0000 7fb59c5be700 1 --2- 172.21.15.169:0/2194834892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb594108620 0x7fb594107ea0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb5900116a0 tx=0x7fb590011ba0 comp rx=0 tx=0).stop 2024-03-28T00:47:31.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.403+0000 7fb59c5be700 1 -- 172.21.15.169:0/2194834892 shutdown_connections 2024-03-28T00:47:31.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.403+0000 7fb59c5be700 1 --2- 172.21.15.169:0/2194834892 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7fb580038f20 0x7fb58003b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:31.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.403+0000 7fb59c5be700 1 --2- 172.21.15.169:0/2194834892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb594108620 0x7fb594107ea0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:31.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.403+0000 7fb59c5be700 1 -- 172.21.15.169:0/2194834892 >> 172.21.15.169:0/2194834892 conn(0x7fb594102030 msgr2=0x7fb594102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:31.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.403+0000 7fb59c5be700 1 -- 172.21.15.169:0/2194834892 shutdown_connections 2024-03-28T00:47:31.017 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:30.403+0000 7fb59c5be700 1 -- 172.21.15.169:0/2194834892 wait complete. 2024-03-28T00:47:31.017 INFO:teuthology.orchestra.run.smithi169.stdout:Waiting for the mgr to restart... 2024-03-28T00:47:31.017 INFO:teuthology.orchestra.run.smithi169.stdout:Waiting for mgr epoch 9... 2024-03-28T00:47:31.250 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:31 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2194834892' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T00:47:31.251 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:31 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:31.154+0000 7f52d4686200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T00:47:31.552 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:31 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:31.268+0000 7f52d4686200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T00:47:31.552 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:31 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:31.354+0000 7f52d4686200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T00:47:31.552 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:31 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:31.455+0000 7f52d4686200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T00:47:32.052 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:31 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:31.632+0000 7f52d4686200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T00:47:32.868 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:32 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:32.579+0000 7f52d4686200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T00:47:32.868 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:32 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:32.710+0000 7f52d4686200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T00:47:33.158 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:33 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:33.065+0000 7f52d4686200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T00:47:33.158 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:33 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:33.154+0000 7f52d4686200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T00:47:33.541 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:33 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:33.352+0000 7f52d4686200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T00:47:33.541 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:33 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:33.537+0000 7f52d4686200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T00:47:34.089 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:33 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:33.828+0000 7f52d4686200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T00:47:34.089 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:34 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:34.085+0000 7f52d4686200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T00:47:34.469 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:34 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:34.318+0000 7f52d4686200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T00:47:35.552 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:35 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:35.108+0000 7f52d4686200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T00:47:35.988 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:35 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:35.671+0000 7f52d4686200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T00:47:35.988 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:35 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:35.767+0000 7f52d4686200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T00:47:35.988 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:47:35 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:47:35.984+0000 7f52d4686200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T00:47:36.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:36 smithi169 ceph-mon[32274]: Active manager daemon x restarted 2024-03-28T00:47:36.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:36 smithi169 ceph-mon[32274]: Activating manager daemon x 2024-03-28T00:47:36.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:36 smithi169 ceph-mon[32274]: osdmap e3: 0 total, 0 up, 0 in 2024-03-28T00:47:36.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:36 smithi169 ceph-mon[32274]: mgrmap e10: x(active, starting, since 0.0034254s) 2024-03-28T00:47:36.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T00:47:36.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T00:47:36.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T00:47:36.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T00:47:36.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T00:47:36.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:36 smithi169 ceph-mon[32274]: Manager daemon x is now available 2024-03-28T00:47:36.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:47:36.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:47:37.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T00:47:37.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T00:47:37.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:37 smithi169 ceph-mon[32274]: mgrmap e11: x(active, since 1.00583s) 2024-03-28T00:47:37.835 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout { 2024-03-28T00:47:37.836 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-28T00:47:37.836 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T00:47:37.836 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout } 2024-03-28T00:47:37.836 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.456+0000 7f8171710700 1 Processor -- start 2024-03-28T00:47:37.836 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.457+0000 7f8171710700 1 -- start start 2024-03-28T00:47:37.836 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.457+0000 7f8171710700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f816c108030 0x7f816c108450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:37.836 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.457+0000 7f8171710700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816c108a20 con 0x7f816c108030 2024-03-28T00:47:37.836 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.457+0000 7f816affd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f816c108030 0x7f816c108450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:37.836 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.457+0000 7f816affd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f816c108030 0x7f816c108450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36372/0 (socket says 172.21.15.169:36372) 2024-03-28T00:47:37.836 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.457+0000 7f816affd700 1 -- 172.21.15.169:0/163903523 learned_addr learned my addr 172.21.15.169:0/163903523 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:37.837 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.457+0000 7f816affd700 1 -- 172.21.15.169:0/163903523 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816c106c40 con 0x7f816c108030 2024-03-28T00:47:37.837 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.457+0000 7f816affd700 1 --2- 172.21.15.169:0/163903523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f816c108030 0x7f816c108450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f815c00bb40 tx=0x7f815c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e2d59a10e1e26b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:37.837 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.457+0000 7f8169ffb700 1 -- 172.21.15.169:0/163903523 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f815c00dbe0 con 0x7f816c108030 2024-03-28T00:47:37.837 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.458+0000 7f8169ffb700 1 -- 172.21.15.169:0/163903523 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f815c004510 con 0x7f816c108030 2024-03-28T00:47:37.837 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.458+0000 7f8169ffb700 1 -- 172.21.15.169:0/163903523 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f815c010470 con 0x7f816c108030 2024-03-28T00:47:37.837 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.458+0000 7f8171710700 1 -- 172.21.15.169:0/163903523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f816c108030 msgr2=0x7f816c108450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:37.837 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.458+0000 7f8171710700 1 --2- 172.21.15.169:0/163903523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f816c108030 0x7f816c108450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f815c00bb40 tx=0x7f815c00be50 comp rx=0 tx=0).stop 2024-03-28T00:47:37.837 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.458+0000 7f8171710700 1 -- 172.21.15.169:0/163903523 shutdown_connections 2024-03-28T00:47:37.837 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.458+0000 7f8171710700 1 --2- 172.21.15.169:0/163903523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f816c108030 0x7f816c108450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:37.837 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.458+0000 7f8171710700 1 -- 172.21.15.169:0/163903523 >> 172.21.15.169:0/163903523 conn(0x7f816c1017d0 msgr2=0x7f816c103bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:37.837 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.459+0000 7f8171710700 1 -- 172.21.15.169:0/163903523 shutdown_connections 2024-03-28T00:47:37.837 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.459+0000 7f8171710700 1 -- 172.21.15.169:0/163903523 wait complete. 2024-03-28T00:47:37.837 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.459+0000 7f8171710700 1 Processor -- start 2024-03-28T00:47:37.838 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.459+0000 7f8171710700 1 -- start start 2024-03-28T00:47:37.838 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.459+0000 7f8171710700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f816c108030 0x7f816c1999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:37.838 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.459+0000 7f8171710700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816c105b30 con 0x7f816c108030 2024-03-28T00:47:37.838 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.460+0000 7f816affd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f816c108030 0x7f816c1999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:37.838 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.460+0000 7f816affd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f816c108030 0x7f816c1999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36388/0 (socket says 172.21.15.169:36388) 2024-03-28T00:47:37.838 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.460+0000 7f816affd700 1 -- 172.21.15.169:0/3327978433 learned_addr learned my addr 172.21.15.169:0/3327978433 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:37.838 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.460+0000 7f816affd700 1 -- 172.21.15.169:0/3327978433 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816c199f10 con 0x7f816c108030 2024-03-28T00:47:37.838 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.460+0000 7f816affd700 1 --2- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f816c108030 0x7f816c1999d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f815c003bb0 tx=0x7f815c0040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:37.838 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.460+0000 7f8153fff700 1 -- 172.21.15.169:0/3327978433 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f815c027070 con 0x7f816c108030 2024-03-28T00:47:37.838 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.460+0000 7f8171710700 1 -- 172.21.15.169:0/3327978433 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f816c19a1a0 con 0x7f816c108030 2024-03-28T00:47:37.838 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.460+0000 7f8153fff700 1 -- 172.21.15.169:0/3327978433 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f815c00de20 con 0x7f816c108030 2024-03-28T00:47:37.838 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.460+0000 7f8153fff700 1 -- 172.21.15.169:0/3327978433 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f815c022ba0 con 0x7f816c108030 2024-03-28T00:47:37.838 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.460+0000 7f8171710700 1 -- 172.21.15.169:0/3327978433 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f816c19ce90 con 0x7f816c108030 2024-03-28T00:47:37.838 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.461+0000 7f8153fff700 1 -- 172.21.15.169:0/3327978433 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f815c02a070 con 0x7f816c108030 2024-03-28T00:47:37.839 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.461+0000 7f8153fff700 1 --2- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f8154038f20 0x7f815403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:37.839 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.461+0000 7f8153fff700 1 -- 172.21.15.169:0/3327978433 --> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f815403baf0 con 0x7f8154038f20 2024-03-28T00:47:37.839 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.461+0000 7f8153fff700 1 -- 172.21.15.169:0/3327978433 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f815c01f070 con 0x7f816c108030 2024-03-28T00:47:37.839 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.461+0000 7f816a7fc700 1 -- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f8154038f20 msgr2=0x7f815403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/3995234407 2024-03-28T00:47:37.839 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.461+0000 7f816a7fc700 1 --2- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f8154038f20 0x7f815403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T00:47:37.839 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.662+0000 7f816a7fc700 1 -- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f8154038f20 msgr2=0x7f815403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/3995234407 2024-03-28T00:47:37.839 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:31.662+0000 7f816a7fc700 1 --2- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f8154038f20 0x7f815403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T00:47:37.839 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:32.063+0000 7f816a7fc700 1 -- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f8154038f20 msgr2=0x7f815403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/3995234407 2024-03-28T00:47:37.839 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:32.063+0000 7f816a7fc700 1 --2- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f8154038f20 0x7f815403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T00:47:37.839 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:32.864+0000 7f816a7fc700 1 -- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f8154038f20 msgr2=0x7f815403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/3995234407 2024-03-28T00:47:37.839 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:32.864+0000 7f816a7fc700 1 --2- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f8154038f20 0x7f815403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T00:47:37.839 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:34.465+0000 7f816a7fc700 1 -- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f8154038f20 msgr2=0x7f815403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/3995234407 2024-03-28T00:47:37.839 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:34.465+0000 7f816a7fc700 1 --2- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f8154038f20 0x7f815403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T00:47:37.840 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:35.999+0000 7f8153fff700 1 -- 172.21.15.169:0/3327978433 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mgrmap(e 10) v1 ==== 45886+0+0 (secure 0 0 0) 0x7f815c022410 con 0x7f816c108030 2024-03-28T00:47:37.840 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:35.999+0000 7f8153fff700 1 -- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f8154038f20 msgr2=0x7f815403b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T00:47:37.840 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:35.999+0000 7f8153fff700 1 --2- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f8154038f20 0x7f815403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:37.840 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:37.000+0000 7f8153fff700 1 -- 172.21.15.169:0/3327978433 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7f815c02cce0 con 0x7f816c108030 2024-03-28T00:47:37.840 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:37.000+0000 7f8153fff700 1 --2- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f815403c9a0 0x7f815403ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:37.840 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:37.000+0000 7f8153fff700 1 -- 172.21.15.169:0/3327978433 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f815403baf0 con 0x7f815403c9a0 2024-03-28T00:47:37.840 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:37.001+0000 7f816a7fc700 1 --2- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f815403c9a0 0x7f815403ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:37.840 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:37.001+0000 7f816a7fc700 1 --2- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f815403c9a0 0x7f815403ed90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8160003a10 tx=0x7f81600092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:37.840 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:37.002+0000 7f8153fff700 1 -- 172.21.15.169:0/3327978433 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f815403baf0 con 0x7f815403c9a0 2024-03-28T00:47:37.840 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:37.009+0000 7f8171710700 1 -- 172.21.15.169:0/3327978433 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f816c0637c0 con 0x7f815403c9a0 2024-03-28T00:47:37.840 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:37.009+0000 7f8153fff700 1 -- 172.21.15.169:0/3327978433 <== mgr.14152 v2:172.21.15.169:6800/1069722251 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f816c0637c0 con 0x7f815403c9a0 2024-03-28T00:47:37.840 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:37.010+0000 7f8171710700 1 -- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f815403c9a0 msgr2=0x7f815403ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:37.840 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:37.010+0000 7f8171710700 1 --2- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f815403c9a0 0x7f815403ed90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8160003a10 tx=0x7f81600092b0 comp rx=0 tx=0).stop 2024-03-28T00:47:37.840 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:37.010+0000 7f8171710700 1 -- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f816c108030 msgr2=0x7f816c1999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:37.841 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:37.010+0000 7f8171710700 1 --2- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f816c108030 0x7f816c1999d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f815c003bb0 tx=0x7f815c0040b0 comp rx=0 tx=0).stop 2024-03-28T00:47:37.841 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:37.010+0000 7f8171710700 1 -- 172.21.15.169:0/3327978433 shutdown_connections 2024-03-28T00:47:37.841 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:37.010+0000 7f8171710700 1 --2- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f815403c9a0 0x7f815403ed90 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:37.841 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:37.010+0000 7f8171710700 1 --2- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f816c108030 0x7f816c1999d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:37.841 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:37.010+0000 7f8171710700 1 --2- 172.21.15.169:0/3327978433 >> [v2:172.21.15.169:6800/3995234407,v1:172.21.15.169:6801/3995234407] conn(0x7f8154038f20 0x7f815403b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:37.841 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:37.010+0000 7f8171710700 1 -- 172.21.15.169:0/3327978433 >> 172.21.15.169:0/3327978433 conn(0x7f816c1017d0 msgr2=0x7f816c190420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:37.841 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:37.010+0000 7f8171710700 1 -- 172.21.15.169:0/3327978433 shutdown_connections 2024-03-28T00:47:37.841 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:37.010+0000 7f8171710700 1 -- 172.21.15.169:0/3327978433 wait complete. 2024-03-28T00:47:37.841 INFO:teuthology.orchestra.run.smithi169.stdout:mgr epoch 9 is available 2024-03-28T00:47:37.841 INFO:teuthology.orchestra.run.smithi169.stdout:Generating a dashboard self-signed certificate... 2024-03-28T00:47:38.156 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:38 smithi169 ceph-mon[32274]: [28/Mar/2024:00:47:36] ENGINE Bus STARTING 2024-03-28T00:47:38.156 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:38 smithi169 ceph-mon[32274]: [28/Mar/2024:00:47:36] ENGINE Serving on http://172.21.15.169:8765 2024-03-28T00:47:38.156 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:38 smithi169 ceph-mon[32274]: [28/Mar/2024:00:47:36] ENGINE Serving on https://172.21.15.169:7150 2024-03-28T00:47:38.156 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:38 smithi169 ceph-mon[32274]: [28/Mar/2024:00:47:36] ENGINE Bus STARTED 2024-03-28T00:47:38.156 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:38 smithi169 ceph-mon[32274]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-28T00:47:38.156 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:38 smithi169 ceph-mon[32274]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-28T00:47:38.156 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:38 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:47:39.342 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-28T00:47:39.342 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.253+0000 7fcc78d91700 1 Processor -- start 2024-03-28T00:47:39.342 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.253+0000 7fcc78d91700 1 -- start start 2024-03-28T00:47:39.342 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.253+0000 7fcc78d91700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc74106a40 0x7fcc74106e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:39.342 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.253+0000 7fcc78d91700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc74107430 con 0x7fcc74106a40 2024-03-28T00:47:39.342 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.254+0000 7fcc7259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc74106a40 0x7fcc74106e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:39.342 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.254+0000 7fcc7259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc74106a40 0x7fcc74106e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40074/0 (socket says 172.21.15.169:40074) 2024-03-28T00:47:39.342 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.254+0000 7fcc7259c700 1 -- 172.21.15.169:0/3705914191 learned_addr learned my addr 172.21.15.169:0/3705914191 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:39.343 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.254+0000 7fcc7259c700 1 -- 172.21.15.169:0/3705914191 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc74107c40 con 0x7fcc74106a40 2024-03-28T00:47:39.343 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.254+0000 7fcc7259c700 1 --2- 172.21.15.169:0/3705914191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc74106a40 0x7fcc74106e60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcc640098a0 tx=0x7fcc64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4e925fe1a1375b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:39.343 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.254+0000 7fcc7159a700 1 -- 172.21.15.169:0/3705914191 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcc64004160 con 0x7fcc74106a40 2024-03-28T00:47:39.343 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.254+0000 7fcc7159a700 1 -- 172.21.15.169:0/3705914191 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc64004300 con 0x7fcc74106a40 2024-03-28T00:47:39.343 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.254+0000 7fcc7159a700 1 -- 172.21.15.169:0/3705914191 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcc64011b20 con 0x7fcc74106a40 2024-03-28T00:47:39.343 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.255+0000 7fcc78d91700 1 -- 172.21.15.169:0/3705914191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc74106a40 msgr2=0x7fcc74106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:39.343 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.255+0000 7fcc78d91700 1 --2- 172.21.15.169:0/3705914191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc74106a40 0x7fcc74106e60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcc640098a0 tx=0x7fcc64009bb0 comp rx=0 tx=0).stop 2024-03-28T00:47:39.343 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.255+0000 7fcc78d91700 1 -- 172.21.15.169:0/3705914191 shutdown_connections 2024-03-28T00:47:39.343 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.255+0000 7fcc78d91700 1 --2- 172.21.15.169:0/3705914191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc74106a40 0x7fcc74106e60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:39.343 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.255+0000 7fcc78d91700 1 -- 172.21.15.169:0/3705914191 >> 172.21.15.169:0/3705914191 conn(0x7fcc74102000 msgr2=0x7fcc74104420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:39.343 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.255+0000 7fcc78d91700 1 -- 172.21.15.169:0/3705914191 shutdown_connections 2024-03-28T00:47:39.343 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.255+0000 7fcc78d91700 1 -- 172.21.15.169:0/3705914191 wait complete. 2024-03-28T00:47:39.344 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.256+0000 7fcc78d91700 1 Processor -- start 2024-03-28T00:47:39.344 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.256+0000 7fcc78d91700 1 -- start start 2024-03-28T00:47:39.344 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.256+0000 7fcc78d91700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc74106a40 0x7fcc7407ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:39.344 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.256+0000 7fcc78d91700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7410aff0 con 0x7fcc74106a40 2024-03-28T00:47:39.344 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.256+0000 7fcc7259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc74106a40 0x7fcc7407ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:39.344 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.256+0000 7fcc7259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc74106a40 0x7fcc7407ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40084/0 (socket says 172.21.15.169:40084) 2024-03-28T00:47:39.344 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.256+0000 7fcc7259c700 1 -- 172.21.15.169:0/2637143758 learned_addr learned my addr 172.21.15.169:0/2637143758 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:39.344 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.257+0000 7fcc7259c700 1 -- 172.21.15.169:0/2637143758 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7419e880 con 0x7fcc74106a40 2024-03-28T00:47:39.344 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.257+0000 7fcc7259c700 1 --2- 172.21.15.169:0/2637143758 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc74106a40 0x7fcc7407ba70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcc64000c00 tx=0x7fcc64011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:39.344 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.257+0000 7fcc637fe700 1 -- 172.21.15.169:0/2637143758 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcc64003b60 con 0x7fcc74106a40 2024-03-28T00:47:39.344 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.257+0000 7fcc637fe700 1 -- 172.21.15.169:0/2637143758 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc6401a910 con 0x7fcc74106a40 2024-03-28T00:47:39.344 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.257+0000 7fcc637fe700 1 -- 172.21.15.169:0/2637143758 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcc64024930 con 0x7fcc74106a40 2024-03-28T00:47:39.344 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.257+0000 7fcc78d91700 1 -- 172.21.15.169:0/2637143758 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7419ea50 con 0x7fcc74106a40 2024-03-28T00:47:39.344 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.257+0000 7fcc78d91700 1 -- 172.21.15.169:0/2637143758 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7419edf0 con 0x7fcc74106a40 2024-03-28T00:47:39.345 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.258+0000 7fcc637fe700 1 -- 172.21.15.169:0/2637143758 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7fcc64022020 con 0x7fcc74106a40 2024-03-28T00:47:39.345 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.258+0000 7fcc637fe700 1 --2- 172.21.15.169:0/2637143758 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcc5c038e50 0x7fcc5c03b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:39.345 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.258+0000 7fcc637fe700 1 -- 172.21.15.169:0/2637143758 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcc6404cff0 con 0x7fcc74106a40 2024-03-28T00:47:39.345 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.258+0000 7fcc71d9b700 1 --2- 172.21.15.169:0/2637143758 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcc5c038e50 0x7fcc5c03b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:39.345 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.258+0000 7fcc78d91700 1 -- 172.21.15.169:0/2637143758 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc740637c0 con 0x7fcc74106a40 2024-03-28T00:47:39.345 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.259+0000 7fcc71d9b700 1 --2- 172.21.15.169:0/2637143758 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcc5c038e50 0x7fcc5c03b310 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcc68006fd0 tx=0x7fcc68006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:39.345 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.264+0000 7fcc637fe700 1 -- 172.21.15.169:0/2637143758 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcc640172b0 con 0x7fcc74106a40 2024-03-28T00:47:39.345 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.368+0000 7fcc637fe700 1 -- 172.21.15.169:0/2637143758 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fcc6402dc90 con 0x7fcc74106a40 2024-03-28T00:47:39.345 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.474+0000 7fcc78d91700 1 -- 172.21.15.169:0/2637143758 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc741032c0 con 0x7fcc5c038e50 2024-03-28T00:47:39.345 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.636+0000 7fcc637fe700 1 -- 172.21.15.169:0/2637143758 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fcc741032c0 con 0x7fcc5c038e50 2024-03-28T00:47:39.345 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.638+0000 7fcc78d91700 1 -- 172.21.15.169:0/2637143758 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcc5c038e50 msgr2=0x7fcc5c03b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:39.345 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.638+0000 7fcc78d91700 1 --2- 172.21.15.169:0/2637143758 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcc5c038e50 0x7fcc5c03b310 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcc68006fd0 tx=0x7fcc68006e40 comp rx=0 tx=0).stop 2024-03-28T00:47:39.345 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.638+0000 7fcc78d91700 1 -- 172.21.15.169:0/2637143758 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc74106a40 msgr2=0x7fcc7407ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:39.346 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.638+0000 7fcc78d91700 1 --2- 172.21.15.169:0/2637143758 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc74106a40 0x7fcc7407ba70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcc64000c00 tx=0x7fcc64011c10 comp rx=0 tx=0).stop 2024-03-28T00:47:39.346 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.638+0000 7fcc78d91700 1 -- 172.21.15.169:0/2637143758 shutdown_connections 2024-03-28T00:47:39.346 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.638+0000 7fcc78d91700 1 --2- 172.21.15.169:0/2637143758 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcc5c038e50 0x7fcc5c03b310 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:39.346 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.638+0000 7fcc78d91700 1 --2- 172.21.15.169:0/2637143758 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc74106a40 0x7fcc7407ba70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:39.346 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.638+0000 7fcc78d91700 1 -- 172.21.15.169:0/2637143758 >> 172.21.15.169:0/2637143758 conn(0x7fcc74102000 msgr2=0x7fcc74102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:39.346 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.638+0000 7fcc78d91700 1 -- 172.21.15.169:0/2637143758 shutdown_connections 2024-03-28T00:47:39.346 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:38.638+0000 7fcc78d91700 1 -- 172.21.15.169:0/2637143758 wait complete. 2024-03-28T00:47:39.346 INFO:teuthology.orchestra.run.smithi169.stdout:Creating initial admin user... 2024-03-28T00:47:39.645 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:39 smithi169 ceph-mon[32274]: mgrmap e12: x(active, since 2s) 2024-03-28T00:47:39.645 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:39 smithi169 ceph-mon[32274]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:47:39.645 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:39 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:47:39.645 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:39 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:47:39.646 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:39 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:47:40.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:40 smithi169 ceph-mon[32274]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:47:40.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:40 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:47:40.777 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$VzrMOAZESQkmJLiy2RO5gurnKfPdrlgbjB0QPeqJfJN9OwIo5ER7u", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711586860, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-28T00:47:40.777 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.744+0000 7f58d9cb4700 1 Processor -- start 2024-03-28T00:47:40.777 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.744+0000 7f58d9cb4700 1 -- start start 2024-03-28T00:47:40.777 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.745+0000 7f58d9cb4700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58d4107fe0 0x7f58d4108400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:40.777 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.745+0000 7f58d9cb4700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d41089d0 con 0x7f58d4107fe0 2024-03-28T00:47:40.777 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.745+0000 7f58d37fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58d4107fe0 0x7f58d4108400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:40.777 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.745+0000 7f58d37fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58d4107fe0 0x7f58d4108400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40092/0 (socket says 172.21.15.169:40092) 2024-03-28T00:47:40.777 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.745+0000 7f58d37fe700 1 -- 172.21.15.169:0/940600925 learned_addr learned my addr 172.21.15.169:0/940600925 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:40.777 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.745+0000 7f58d37fe700 1 -- 172.21.15.169:0/940600925 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d4106c90 con 0x7f58d4107fe0 2024-03-28T00:47:40.777 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.745+0000 7f58d37fe700 1 --2- 172.21.15.169:0/940600925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58d4107fe0 0x7f58d4108400 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f58c4009e80 tx=0x7f58c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=951758150ad4998b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:40.777 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.745+0000 7f58d27fc700 1 -- 172.21.15.169:0/940600925 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58c4017070 con 0x7f58d4107fe0 2024-03-28T00:47:40.777 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.745+0000 7f58d27fc700 1 -- 172.21.15.169:0/940600925 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58c400b8c0 con 0x7f58d4107fe0 2024-03-28T00:47:40.778 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.745+0000 7f58d27fc700 1 -- 172.21.15.169:0/940600925 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58c40045c0 con 0x7f58d4107fe0 2024-03-28T00:47:40.778 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.746+0000 7f58d9cb4700 1 -- 172.21.15.169:0/940600925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58d4107fe0 msgr2=0x7f58d4108400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:40.778 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.746+0000 7f58d9cb4700 1 --2- 172.21.15.169:0/940600925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58d4107fe0 0x7f58d4108400 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f58c4009e80 tx=0x7f58c400b330 comp rx=0 tx=0).stop 2024-03-28T00:47:40.778 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.746+0000 7f58d9cb4700 1 -- 172.21.15.169:0/940600925 shutdown_connections 2024-03-28T00:47:40.778 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.746+0000 7f58d9cb4700 1 --2- 172.21.15.169:0/940600925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58d4107fe0 0x7f58d4108400 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:40.778 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.746+0000 7f58d9cb4700 1 -- 172.21.15.169:0/940600925 >> 172.21.15.169:0/940600925 conn(0x7f58d4101780 msgr2=0x7f58d4103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:40.778 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.746+0000 7f58d9cb4700 1 -- 172.21.15.169:0/940600925 shutdown_connections 2024-03-28T00:47:40.778 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.746+0000 7f58d9cb4700 1 -- 172.21.15.169:0/940600925 wait complete. 2024-03-28T00:47:40.778 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.747+0000 7f58d9cb4700 1 Processor -- start 2024-03-28T00:47:40.778 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.747+0000 7f58d9cb4700 1 -- start start 2024-03-28T00:47:40.778 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.747+0000 7f58d9cb4700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58d4107fe0 0x7f58d4106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:40.778 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.747+0000 7f58d9cb4700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d410b050 con 0x7f58d4107fe0 2024-03-28T00:47:40.778 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.747+0000 7f58d37fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58d4107fe0 0x7f58d4106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:40.779 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.747+0000 7f58d37fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58d4107fe0 0x7f58d4106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40108/0 (socket says 172.21.15.169:40108) 2024-03-28T00:47:40.779 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.747+0000 7f58d37fe700 1 -- 172.21.15.169:0/3082751577 learned_addr learned my addr 172.21.15.169:0/3082751577 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:40.779 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.747+0000 7f58d37fe700 1 -- 172.21.15.169:0/3082751577 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d419e820 con 0x7f58d4107fe0 2024-03-28T00:47:40.779 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.747+0000 7f58d37fe700 1 --2- 172.21.15.169:0/3082751577 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58d4107fe0 0x7f58d4106620 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f58c4003fa0 tx=0x7f58c4004080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:40.779 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.748+0000 7f58d0ff9700 1 -- 172.21.15.169:0/3082751577 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58c4020020 con 0x7f58d4107fe0 2024-03-28T00:47:40.779 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.748+0000 7f58d0ff9700 1 -- 172.21.15.169:0/3082751577 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58c400bba0 con 0x7f58d4107fe0 2024-03-28T00:47:40.779 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.748+0000 7f58d9cb4700 1 -- 172.21.15.169:0/3082751577 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d419e9f0 con 0x7f58d4107fe0 2024-03-28T00:47:40.779 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.748+0000 7f58d0ff9700 1 -- 172.21.15.169:0/3082751577 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58c4017040 con 0x7f58d4107fe0 2024-03-28T00:47:40.779 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.748+0000 7f58d9cb4700 1 -- 172.21.15.169:0/3082751577 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d419ee10 con 0x7f58d4107fe0 2024-03-28T00:47:40.779 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.748+0000 7f58d0ff9700 1 -- 172.21.15.169:0/3082751577 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f58c4023020 con 0x7f58d4107fe0 2024-03-28T00:47:40.779 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.749+0000 7f58d0ff9700 1 --2- 172.21.15.169:0/3082751577 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f58c0038f80 0x7f58c003b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:40.779 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.749+0000 7f58d0ff9700 1 -- 172.21.15.169:0/3082751577 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f58c400f050 con 0x7f58d4107fe0 2024-03-28T00:47:40.779 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.749+0000 7f58d2ffd700 1 --2- 172.21.15.169:0/3082751577 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f58c0038f80 0x7f58c003b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:40.780 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.749+0000 7f58d9cb4700 1 -- 172.21.15.169:0/3082751577 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d40637c0 con 0x7f58d4107fe0 2024-03-28T00:47:40.780 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.749+0000 7f58d2ffd700 1 --2- 172.21.15.169:0/3082751577 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f58c0038f80 0x7f58c003b440 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f58bc006fd0 tx=0x7f58bc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:40.780 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.754+0000 7f58d0ff9700 1 -- 172.21.15.169:0/3082751577 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f58c402a020 con 0x7f58d4107fe0 2024-03-28T00:47:40.780 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:39.977+0000 7f58d9cb4700 1 -- 172.21.15.169:0/3082751577 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f58d41029e0 con 0x7f58c0038f80 2024-03-28T00:47:40.780 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:40.201+0000 7f58d0ff9700 1 -- 172.21.15.169:0/3082751577 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f58d41029e0 con 0x7f58c0038f80 2024-03-28T00:47:40.780 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:40.203+0000 7f58d9cb4700 1 -- 172.21.15.169:0/3082751577 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f58c0038f80 msgr2=0x7f58c003b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:40.780 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:40.203+0000 7f58d9cb4700 1 --2- 172.21.15.169:0/3082751577 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f58c0038f80 0x7f58c003b440 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f58bc006fd0 tx=0x7f58bc006e40 comp rx=0 tx=0).stop 2024-03-28T00:47:40.780 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:40.203+0000 7f58d9cb4700 1 -- 172.21.15.169:0/3082751577 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58d4107fe0 msgr2=0x7f58d4106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:40.780 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:40.203+0000 7f58d9cb4700 1 --2- 172.21.15.169:0/3082751577 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58d4107fe0 0x7f58d4106620 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f58c4003fa0 tx=0x7f58c4004080 comp rx=0 tx=0).stop 2024-03-28T00:47:40.780 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:40.203+0000 7f58d9cb4700 1 -- 172.21.15.169:0/3082751577 shutdown_connections 2024-03-28T00:47:40.780 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:40.203+0000 7f58d9cb4700 1 --2- 172.21.15.169:0/3082751577 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f58c0038f80 0x7f58c003b440 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:40.780 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:40.203+0000 7f58d9cb4700 1 --2- 172.21.15.169:0/3082751577 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58d4107fe0 0x7f58d4106620 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:40.780 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:40.203+0000 7f58d9cb4700 1 -- 172.21.15.169:0/3082751577 >> 172.21.15.169:0/3082751577 conn(0x7f58d4101780 msgr2=0x7f58d41022d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:40.780 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:40.204+0000 7f58d9cb4700 1 -- 172.21.15.169:0/3082751577 shutdown_connections 2024-03-28T00:47:40.780 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:40.204+0000 7f58d9cb4700 1 -- 172.21.15.169:0/3082751577 wait complete. 2024-03-28T00:47:40.781 INFO:teuthology.orchestra.run.smithi169.stdout:Fetching dashboard port number... 2024-03-28T00:47:42.016 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:41 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2646505564' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-28T00:47:42.221 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stdout 8443 2024-03-28T00:47:42.221 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.489+0000 7f01aca73700 1 Processor -- start 2024-03-28T00:47:42.222 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.489+0000 7f01aca73700 1 -- start start 2024-03-28T00:47:42.222 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.489+0000 7f01aca73700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01a8108600 0x7f01a810a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:42.222 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.489+0000 7f01aca73700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a807be90 con 0x7f01a8108600 2024-03-28T00:47:42.222 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.489+0000 7f01a659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01a8108600 0x7f01a810a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:42.222 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.489+0000 7f01a659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01a8108600 0x7f01a810a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40112/0 (socket says 172.21.15.169:40112) 2024-03-28T00:47:42.222 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.489+0000 7f01a659c700 1 -- 172.21.15.169:0/3849675017 learned_addr learned my addr 172.21.15.169:0/3849675017 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:42.222 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.489+0000 7f01a659c700 1 -- 172.21.15.169:0/3849675017 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a810af30 con 0x7f01a8108600 2024-03-28T00:47:42.222 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.490+0000 7f01a659c700 1 --2- 172.21.15.169:0/3849675017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01a8108600 0x7f01a810a9f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f019c0098a0 tx=0x7f019c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ef87060957d122 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:42.222 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.490+0000 7f01a559a700 1 -- 172.21.15.169:0/3849675017 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f019c004160 con 0x7f01a8108600 2024-03-28T00:47:42.222 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.490+0000 7f01a559a700 1 -- 172.21.15.169:0/3849675017 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f019c004300 con 0x7f01a8108600 2024-03-28T00:47:42.222 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.490+0000 7f01a559a700 1 -- 172.21.15.169:0/3849675017 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f019c011b20 con 0x7f01a8108600 2024-03-28T00:47:42.222 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.490+0000 7f01aca73700 1 -- 172.21.15.169:0/3849675017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01a8108600 msgr2=0x7f01a810a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:42.223 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.490+0000 7f01aca73700 1 --2- 172.21.15.169:0/3849675017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01a8108600 0x7f01a810a9f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f019c0098a0 tx=0x7f019c009bb0 comp rx=0 tx=0).stop 2024-03-28T00:47:42.223 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.490+0000 7f01aca73700 1 -- 172.21.15.169:0/3849675017 shutdown_connections 2024-03-28T00:47:42.223 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.490+0000 7f01aca73700 1 --2- 172.21.15.169:0/3849675017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01a8108600 0x7f01a810a9f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:42.223 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.490+0000 7f01aca73700 1 -- 172.21.15.169:0/3849675017 >> 172.21.15.169:0/3849675017 conn(0x7f01a8102030 msgr2=0x7f01a8104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:42.223 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.490+0000 7f01aca73700 1 -- 172.21.15.169:0/3849675017 shutdown_connections 2024-03-28T00:47:42.223 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.491+0000 7f01aca73700 1 -- 172.21.15.169:0/3849675017 wait complete. 2024-03-28T00:47:42.223 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.491+0000 7f01aca73700 1 Processor -- start 2024-03-28T00:47:42.223 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.491+0000 7f01aca73700 1 -- start start 2024-03-28T00:47:42.223 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.491+0000 7f01aca73700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01a8108600 0x7f01a819bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:42.223 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.491+0000 7f01aca73700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a810bf90 con 0x7f01a8108600 2024-03-28T00:47:42.223 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.491+0000 7f01a659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01a8108600 0x7f01a819bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:42.223 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.491+0000 7f01a659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01a8108600 0x7f01a819bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40122/0 (socket says 172.21.15.169:40122) 2024-03-28T00:47:42.223 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.492+0000 7f01a659c700 1 -- 172.21.15.169:0/2646505564 learned_addr learned my addr 172.21.15.169:0/2646505564 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:42.223 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.492+0000 7f01a659c700 1 -- 172.21.15.169:0/2646505564 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a819c000 con 0x7f01a8108600 2024-03-28T00:47:42.224 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.492+0000 7f01a659c700 1 --2- 172.21.15.169:0/2646505564 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01a8108600 0x7f01a819bac0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f019c0116a0 tx=0x7f019c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:42.224 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.492+0000 7f01977fe700 1 -- 172.21.15.169:0/2646505564 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f019c003a40 con 0x7f01a8108600 2024-03-28T00:47:42.224 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.492+0000 7f01aca73700 1 -- 172.21.15.169:0/2646505564 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a819c290 con 0x7f01a8108600 2024-03-28T00:47:42.224 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.492+0000 7f01977fe700 1 -- 172.21.15.169:0/2646505564 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f019c003be0 con 0x7f01a8108600 2024-03-28T00:47:42.224 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.492+0000 7f01977fe700 1 -- 172.21.15.169:0/2646505564 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f019c01a750 con 0x7f01a8108600 2024-03-28T00:47:42.224 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.492+0000 7f01aca73700 1 -- 172.21.15.169:0/2646505564 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a819c6f0 con 0x7f01a8108600 2024-03-28T00:47:42.224 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.493+0000 7f01977fe700 1 -- 172.21.15.169:0/2646505564 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f019c01aa50 con 0x7f01a8108600 2024-03-28T00:47:42.224 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.493+0000 7f01977fe700 1 --2- 172.21.15.169:0/2646505564 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0190038f20 0x7f019003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:42.224 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.493+0000 7f01977fe700 1 -- 172.21.15.169:0/2646505564 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f019c024aa0 con 0x7f01a8108600 2024-03-28T00:47:42.224 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.493+0000 7f01a5d9b700 1 --2- 172.21.15.169:0/2646505564 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0190038f20 0x7f019003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:42.224 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.493+0000 7f01aca73700 1 -- 172.21.15.169:0/2646505564 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a80637c0 con 0x7f01a8108600 2024-03-28T00:47:42.224 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.494+0000 7f01a5d9b700 1 --2- 172.21.15.169:0/2646505564 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0190038f20 0x7f019003b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0198006fd0 tx=0x7f0198006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:42.225 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.498+0000 7f01977fe700 1 -- 172.21.15.169:0/2646505564 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f019c011e20 con 0x7f01a8108600 2024-03-28T00:47:42.225 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.692+0000 7f01aca73700 1 -- 172.21.15.169:0/2646505564 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f01a819f0a0 con 0x7f01a8108600 2024-03-28T00:47:42.225 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.692+0000 7f01977fe700 1 -- 172.21.15.169:0/2646505564 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f019c00ba30 con 0x7f01a8108600 2024-03-28T00:47:42.225 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.695+0000 7f01aca73700 1 -- 172.21.15.169:0/2646505564 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0190038f20 msgr2=0x7f019003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:42.225 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.695+0000 7f01aca73700 1 --2- 172.21.15.169:0/2646505564 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0190038f20 0x7f019003b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0198006fd0 tx=0x7f0198006e40 comp rx=0 tx=0).stop 2024-03-28T00:47:42.225 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.695+0000 7f01aca73700 1 -- 172.21.15.169:0/2646505564 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01a8108600 msgr2=0x7f01a819bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:42.225 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.695+0000 7f01aca73700 1 --2- 172.21.15.169:0/2646505564 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01a8108600 0x7f01a819bac0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f019c0116a0 tx=0x7f019c011ba0 comp rx=0 tx=0).stop 2024-03-28T00:47:42.225 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.695+0000 7f01aca73700 1 -- 172.21.15.169:0/2646505564 shutdown_connections 2024-03-28T00:47:42.225 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.695+0000 7f01aca73700 1 --2- 172.21.15.169:0/2646505564 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0190038f20 0x7f019003b3e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:42.225 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.695+0000 7f01aca73700 1 --2- 172.21.15.169:0/2646505564 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01a8108600 0x7f01a819bac0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:42.225 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.695+0000 7f01aca73700 1 -- 172.21.15.169:0/2646505564 >> 172.21.15.169:0/2646505564 conn(0x7f01a8102030 msgr2=0x7f01a8102af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:42.225 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.695+0000 7f01aca73700 1 -- 172.21.15.169:0/2646505564 shutdown_connections 2024-03-28T00:47:42.225 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:41.695+0000 7f01aca73700 1 -- 172.21.15.169:0/2646505564 wait complete. 2024-03-28T00:47:42.227 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T00:47:42.227 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout disabled 2024-03-28T00:47:42.234 INFO:teuthology.orchestra.run.smithi169.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T00:47:42.234 INFO:teuthology.orchestra.run.smithi169.stdout:systemctl: stdout inactive 2024-03-28T00:47:42.234 INFO:teuthology.orchestra.run.smithi169.stdout:firewalld.service is not enabled 2024-03-28T00:47:42.234 INFO:teuthology.orchestra.run.smithi169.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-28T00:47:42.236 INFO:teuthology.orchestra.run.smithi169.stdout:Ceph Dashboard is now available at: 2024-03-28T00:47:42.236 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:47:42.236 INFO:teuthology.orchestra.run.smithi169.stdout: URL: https://smithi169.front.sepia.ceph.com:8443/ 2024-03-28T00:47:42.236 INFO:teuthology.orchestra.run.smithi169.stdout: User: admin 2024-03-28T00:47:42.236 INFO:teuthology.orchestra.run.smithi169.stdout: Password: pto2de5kco 2024-03-28T00:47:42.236 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:47:42.237 INFO:teuthology.orchestra.run.smithi169.stdout:Saving cluster configuration to /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config directory 2024-03-28T00:47:42.237 INFO:teuthology.orchestra.run.smithi169.stdout:Enabling autotune for osd_memory_target 2024-03-28T00:47:43.413 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.620+0000 7f6b3afee700 1 Processor -- start 2024-03-28T00:47:43.414 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.620+0000 7f6b3afee700 1 -- start start 2024-03-28T00:47:43.414 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.621+0000 7f6b3afee700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b34108010 0x7f6b34108430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:43.414 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.621+0000 7f6b3afee700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b34108a00 con 0x7f6b34108010 2024-03-28T00:47:43.414 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.621+0000 7f6b38d8a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b34108010 0x7f6b34108430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:43.414 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.621+0000 7f6b38d8a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b34108010 0x7f6b34108430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40138/0 (socket says 172.21.15.169:40138) 2024-03-28T00:47:43.414 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.621+0000 7f6b38d8a700 1 -- 172.21.15.169:0/1115950802 learned_addr learned my addr 172.21.15.169:0/1115950802 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:43.414 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.621+0000 7f6b38d8a700 1 -- 172.21.15.169:0/1115950802 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b34108b80 con 0x7f6b34108010 2024-03-28T00:47:43.414 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.621+0000 7f6b38d8a700 1 --2- 172.21.15.169:0/1115950802 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b34108010 0x7f6b34108430 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6b240098a0 tx=0x7f6b24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc706e7281075314 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:43.414 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.621+0000 7f6b337fe700 1 -- 172.21.15.169:0/1115950802 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b24004160 con 0x7f6b34108010 2024-03-28T00:47:43.414 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.621+0000 7f6b337fe700 1 -- 172.21.15.169:0/1115950802 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b24004300 con 0x7f6b34108010 2024-03-28T00:47:43.415 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.622+0000 7f6b337fe700 1 -- 172.21.15.169:0/1115950802 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b24011b20 con 0x7f6b34108010 2024-03-28T00:47:43.415 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.622+0000 7f6b3afee700 1 -- 172.21.15.169:0/1115950802 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b34108010 msgr2=0x7f6b34108430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:43.415 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.622+0000 7f6b3afee700 1 --2- 172.21.15.169:0/1115950802 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b34108010 0x7f6b34108430 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6b240098a0 tx=0x7f6b24009bb0 comp rx=0 tx=0).stop 2024-03-28T00:47:43.415 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.622+0000 7f6b3afee700 1 -- 172.21.15.169:0/1115950802 shutdown_connections 2024-03-28T00:47:43.415 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.622+0000 7f6b3afee700 1 --2- 172.21.15.169:0/1115950802 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b34108010 0x7f6b34108430 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:43.415 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.622+0000 7f6b3afee700 1 -- 172.21.15.169:0/1115950802 >> 172.21.15.169:0/1115950802 conn(0x7f6b341017a0 msgr2=0x7f6b34103bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:43.415 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.622+0000 7f6b3afee700 1 -- 172.21.15.169:0/1115950802 shutdown_connections 2024-03-28T00:47:43.415 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.622+0000 7f6b3afee700 1 -- 172.21.15.169:0/1115950802 wait complete. 2024-03-28T00:47:43.415 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.623+0000 7f6b3afee700 1 Processor -- start 2024-03-28T00:47:43.415 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.623+0000 7f6b3afee700 1 -- start start 2024-03-28T00:47:43.415 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.623+0000 7f6b3afee700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b34108010 0x7f6b3419dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:43.416 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.623+0000 7f6b3afee700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b34106a30 con 0x7f6b34108010 2024-03-28T00:47:43.416 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.623+0000 7f6b38d8a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b34108010 0x7f6b3419dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:43.416 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.623+0000 7f6b38d8a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b34108010 0x7f6b3419dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40154/0 (socket says 172.21.15.169:40154) 2024-03-28T00:47:43.416 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.623+0000 7f6b38d8a700 1 -- 172.21.15.169:0/1022516915 learned_addr learned my addr 172.21.15.169:0/1022516915 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:43.416 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.623+0000 7f6b38d8a700 1 -- 172.21.15.169:0/1022516915 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3419e180 con 0x7f6b34108010 2024-03-28T00:47:43.416 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.623+0000 7f6b38d8a700 1 --2- 172.21.15.169:0/1022516915 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b34108010 0x7f6b3419dc40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6b240116a0 tx=0x7f6b24011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:43.416 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.623+0000 7f6b31ffb700 1 -- 172.21.15.169:0/1022516915 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b24003a40 con 0x7f6b34108010 2024-03-28T00:47:43.416 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.623+0000 7f6b31ffb700 1 -- 172.21.15.169:0/1022516915 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b24003be0 con 0x7f6b34108010 2024-03-28T00:47:43.416 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.624+0000 7f6b3afee700 1 -- 172.21.15.169:0/1022516915 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3419e470 con 0x7f6b34108010 2024-03-28T00:47:43.416 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.624+0000 7f6b31ffb700 1 -- 172.21.15.169:0/1022516915 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b2401a750 con 0x7f6b34108010 2024-03-28T00:47:43.416 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.624+0000 7f6b3afee700 1 -- 172.21.15.169:0/1022516915 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b341a1100 con 0x7f6b34108010 2024-03-28T00:47:43.417 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.624+0000 7f6b31ffb700 1 -- 172.21.15.169:0/1022516915 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f6b2401aa00 con 0x7f6b34108010 2024-03-28T00:47:43.417 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.625+0000 7f6b3afee700 1 -- 172.21.15.169:0/1022516915 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b340637c0 con 0x7f6b34108010 2024-03-28T00:47:43.417 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.625+0000 7f6b31ffb700 1 --2- 172.21.15.169:0/1022516915 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6b1c038f80 0x7f6b1c03b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:43.417 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.625+0000 7f6b31ffb700 1 -- 172.21.15.169:0/1022516915 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6b2404d670 con 0x7f6b34108010 2024-03-28T00:47:43.417 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.625+0000 7f6b33fff700 1 --2- 172.21.15.169:0/1022516915 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6b1c038f80 0x7f6b1c03b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:43.417 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.625+0000 7f6b33fff700 1 --2- 172.21.15.169:0/1022516915 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6b1c038f80 0x7f6b1c03b440 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6b28006fd0 tx=0x7f6b28006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:43.417 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.630+0000 7f6b31ffb700 1 -- 172.21.15.169:0/1022516915 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6b24011e20 con 0x7f6b34108010 2024-03-28T00:47:43.417 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.745+0000 7f6b31ffb700 1 -- 172.21.15.169:0/1022516915 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f6b2402d910 con 0x7f6b34108010 2024-03-28T00:47:43.418 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.823+0000 7f6b3afee700 1 -- 172.21.15.169:0/1022516915 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f6b3402dde0 con 0x7f6b34108010 2024-03-28T00:47:43.418 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.823+0000 7f6b31ffb700 1 -- 172.21.15.169:0/1022516915 <== mon.0 v2:172.21.15.169:3300/0 8 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f6b2400b3e0 con 0x7f6b34108010 2024-03-28T00:47:43.418 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.825+0000 7f6b3afee700 1 -- 172.21.15.169:0/1022516915 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6b1c038f80 msgr2=0x7f6b1c03b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:43.418 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.825+0000 7f6b3afee700 1 --2- 172.21.15.169:0/1022516915 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6b1c038f80 0x7f6b1c03b440 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6b28006fd0 tx=0x7f6b28006e40 comp rx=0 tx=0).stop 2024-03-28T00:47:43.418 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.825+0000 7f6b3afee700 1 -- 172.21.15.169:0/1022516915 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b34108010 msgr2=0x7f6b3419dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:43.418 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.825+0000 7f6b3afee700 1 --2- 172.21.15.169:0/1022516915 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b34108010 0x7f6b3419dc40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6b240116a0 tx=0x7f6b24011ba0 comp rx=0 tx=0).stop 2024-03-28T00:47:43.418 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.825+0000 7f6b3afee700 1 -- 172.21.15.169:0/1022516915 shutdown_connections 2024-03-28T00:47:43.418 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.825+0000 7f6b3afee700 1 --2- 172.21.15.169:0/1022516915 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6b1c038f80 0x7f6b1c03b440 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:43.418 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.825+0000 7f6b3afee700 1 --2- 172.21.15.169:0/1022516915 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b34108010 0x7f6b3419dc40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:43.418 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.825+0000 7f6b3afee700 1 -- 172.21.15.169:0/1022516915 >> 172.21.15.169:0/1022516915 conn(0x7f6b341017a0 msgr2=0x7f6b34194840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:43.418 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.825+0000 7f6b3afee700 1 -- 172.21.15.169:0/1022516915 shutdown_connections 2024-03-28T00:47:43.418 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:42.825+0000 7f6b3afee700 1 -- 172.21.15.169:0/1022516915 wait complete. 2024-03-28T00:47:43.780 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:43 smithi169 ceph-mon[32274]: mgrmap e13: x(active, since 6s) 2024-03-28T00:47:44.652 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.793+0000 7fb3152ce700 1 Processor -- start 2024-03-28T00:47:44.652 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.793+0000 7fb3152ce700 1 -- start start 2024-03-28T00:47:44.652 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.793+0000 7fb3152ce700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3101069f0 0x7fb310106e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:44.652 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.793+0000 7fb3152ce700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3101073e0 con 0x7fb3101069f0 2024-03-28T00:47:44.652 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.793+0000 7fb30effd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3101069f0 0x7fb310106e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:44.652 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.793+0000 7fb30effd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3101069f0 0x7fb310106e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40158/0 (socket says 172.21.15.169:40158) 2024-03-28T00:47:44.652 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.793+0000 7fb30effd700 1 -- 172.21.15.169:0/921487984 learned_addr learned my addr 172.21.15.169:0/921487984 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:44.653 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.793+0000 7fb30effd700 1 -- 172.21.15.169:0/921487984 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb310107bf0 con 0x7fb3101069f0 2024-03-28T00:47:44.653 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.794+0000 7fb30effd700 1 --2- 172.21.15.169:0/921487984 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3101069f0 0x7fb310106e10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb3000098a0 tx=0x7fb300009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1785973e292fad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:44.653 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.794+0000 7fb30dffb700 1 -- 172.21.15.169:0/921487984 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb300004160 con 0x7fb3101069f0 2024-03-28T00:47:44.653 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.794+0000 7fb30dffb700 1 -- 172.21.15.169:0/921487984 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb300004300 con 0x7fb3101069f0 2024-03-28T00:47:44.653 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.794+0000 7fb30dffb700 1 -- 172.21.15.169:0/921487984 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb300011b20 con 0x7fb3101069f0 2024-03-28T00:47:44.653 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.794+0000 7fb3152ce700 1 -- 172.21.15.169:0/921487984 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3101069f0 msgr2=0x7fb310106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:44.653 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.794+0000 7fb3152ce700 1 --2- 172.21.15.169:0/921487984 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3101069f0 0x7fb310106e10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb3000098a0 tx=0x7fb300009bb0 comp rx=0 tx=0).stop 2024-03-28T00:47:44.653 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.794+0000 7fb3152ce700 1 -- 172.21.15.169:0/921487984 shutdown_connections 2024-03-28T00:47:44.653 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.794+0000 7fb3152ce700 1 --2- 172.21.15.169:0/921487984 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3101069f0 0x7fb310106e10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:44.653 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.794+0000 7fb3152ce700 1 -- 172.21.15.169:0/921487984 >> 172.21.15.169:0/921487984 conn(0x7fb310101fb0 msgr2=0x7fb3101043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:44.653 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.794+0000 7fb3152ce700 1 -- 172.21.15.169:0/921487984 shutdown_connections 2024-03-28T00:47:44.653 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.794+0000 7fb3152ce700 1 -- 172.21.15.169:0/921487984 wait complete. 2024-03-28T00:47:44.653 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.795+0000 7fb3152ce700 1 Processor -- start 2024-03-28T00:47:44.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.795+0000 7fb3152ce700 1 -- start start 2024-03-28T00:47:44.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.795+0000 7fb3152ce700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3101069f0 0x7fb310191ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:44.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.795+0000 7fb3152ce700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31010afa0 con 0x7fb3101069f0 2024-03-28T00:47:44.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.795+0000 7fb30effd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3101069f0 0x7fb310191ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:44.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.795+0000 7fb30effd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3101069f0 0x7fb310191ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40160/0 (socket says 172.21.15.169:40160) 2024-03-28T00:47:44.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.795+0000 7fb30effd700 1 -- 172.21.15.169:0/485980780 learned_addr learned my addr 172.21.15.169:0/485980780 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:44.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.796+0000 7fb30effd700 1 -- 172.21.15.169:0/485980780 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3101920e0 con 0x7fb3101069f0 2024-03-28T00:47:44.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.796+0000 7fb30effd700 1 --2- 172.21.15.169:0/485980780 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3101069f0 0x7fb310191ba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb3000116a0 tx=0x7fb300011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:44.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.796+0000 7fb2f7fff700 1 -- 172.21.15.169:0/485980780 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb300003a40 con 0x7fb3101069f0 2024-03-28T00:47:44.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.796+0000 7fb2f7fff700 1 -- 172.21.15.169:0/485980780 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb300003be0 con 0x7fb3101069f0 2024-03-28T00:47:44.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.796+0000 7fb3152ce700 1 -- 172.21.15.169:0/485980780 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3101902d0 con 0x7fb3101069f0 2024-03-28T00:47:44.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.796+0000 7fb2f7fff700 1 -- 172.21.15.169:0/485980780 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb30001a720 con 0x7fb3101069f0 2024-03-28T00:47:44.654 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.796+0000 7fb3152ce700 1 -- 172.21.15.169:0/485980780 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb310190790 con 0x7fb3101069f0 2024-03-28T00:47:44.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.797+0000 7fb2f7fff700 1 -- 172.21.15.169:0/485980780 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fb30001aa20 con 0x7fb3101069f0 2024-03-28T00:47:44.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.797+0000 7fb3152ce700 1 -- 172.21.15.169:0/485980780 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3100637c0 con 0x7fb3101069f0 2024-03-28T00:47:44.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.797+0000 7fb2f7fff700 1 --2- 172.21.15.169:0/485980780 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb2f8038fc0 0x7fb2f803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:44.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.797+0000 7fb2f7fff700 1 -- 172.21.15.169:0/485980780 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb30004d7f0 con 0x7fb3101069f0 2024-03-28T00:47:44.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.797+0000 7fb30e7fc700 1 --2- 172.21.15.169:0/485980780 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb2f8038fc0 0x7fb2f803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:44.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.798+0000 7fb30e7fc700 1 --2- 172.21.15.169:0/485980780 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb2f8038fc0 0x7fb2f803b480 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb304006fd0 tx=0x7fb304006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:44.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:43.802+0000 7fb2f7fff700 1 -- 172.21.15.169:0/485980780 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb300011e20 con 0x7fb3101069f0 2024-03-28T00:47:44.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:44.108+0000 7fb3152ce700 1 -- 172.21.15.169:0/485980780 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fb3101958f0 con 0x7fb3101069f0 2024-03-28T00:47:44.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:44.111+0000 7fb2f7fff700 1 -- 172.21.15.169:0/485980780 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) v1 ==== 153+0+0 (secure 0 0 0) 0x7fb30000bc40 con 0x7fb3101069f0 2024-03-28T00:47:44.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:44.114+0000 7fb3152ce700 1 -- 172.21.15.169:0/485980780 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb2f8038fc0 msgr2=0x7fb2f803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:44.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:44.114+0000 7fb3152ce700 1 --2- 172.21.15.169:0/485980780 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb2f8038fc0 0x7fb2f803b480 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb304006fd0 tx=0x7fb304006e40 comp rx=0 tx=0).stop 2024-03-28T00:47:44.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:44.114+0000 7fb3152ce700 1 -- 172.21.15.169:0/485980780 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3101069f0 msgr2=0x7fb310191ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:44.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:44.114+0000 7fb3152ce700 1 --2- 172.21.15.169:0/485980780 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3101069f0 0x7fb310191ba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb3000116a0 tx=0x7fb300011ba0 comp rx=0 tx=0).stop 2024-03-28T00:47:44.655 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:44.114+0000 7fb3152ce700 1 -- 172.21.15.169:0/485980780 shutdown_connections 2024-03-28T00:47:44.656 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:44.114+0000 7fb3152ce700 1 --2- 172.21.15.169:0/485980780 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb2f8038fc0 0x7fb2f803b480 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:44.656 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:44.114+0000 7fb3152ce700 1 --2- 172.21.15.169:0/485980780 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3101069f0 0x7fb310191ba0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:44.656 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:44.114+0000 7fb3152ce700 1 -- 172.21.15.169:0/485980780 >> 172.21.15.169:0/485980780 conn(0x7fb310101fb0 msgr2=0x7fb310102e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:44.656 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:44.114+0000 7fb3152ce700 1 -- 172.21.15.169:0/485980780 shutdown_connections 2024-03-28T00:47:44.656 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr 2024-03-28T00:47:44.114+0000 7fb3152ce700 1 -- 172.21.15.169:0/485980780 wait complete. 2024-03-28T00:47:44.656 INFO:teuthology.orchestra.run.smithi169.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-28T00:47:44.656 INFO:teuthology.orchestra.run.smithi169.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-28T00:47:44.656 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:47:44.656 INFO:teuthology.orchestra.run.smithi169.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-28T00:47:44.656 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:47:44.656 INFO:teuthology.orchestra.run.smithi169.stdout:Or, if you are only running a single cluster on this host: 2024-03-28T00:47:44.656 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:47:44.656 INFO:teuthology.orchestra.run.smithi169.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-28T00:47:44.656 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:47:44.656 INFO:teuthology.orchestra.run.smithi169.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-28T00:47:44.657 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:47:44.657 INFO:teuthology.orchestra.run.smithi169.stdout: ceph telemetry on 2024-03-28T00:47:44.657 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:47:44.657 INFO:teuthology.orchestra.run.smithi169.stdout:For more information see: 2024-03-28T00:47:44.657 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:47:44.657 INFO:teuthology.orchestra.run.smithi169.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-28T00:47:44.657 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:47:44.657 INFO:teuthology.orchestra.run.smithi169.stdout:Bootstrap complete. 2024-03-28T00:47:44.699 INFO:tasks.cephadm:Fetching config... 2024-03-28T00:47:44.699 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-28T00:47:44.700 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-28T00:47:44.715 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-28T00:47:44.715 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-28T00:47:44.715 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-28T00:47:44.771 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-28T00:47:44.771 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-28T00:47:44.771 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-28T00:47:44.839 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-28T00:47:44.839 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-28T00:47:44.839 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-28T00:47:44.895 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-28T00:47:44.895 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4X76kTFMVUKlH9cG7YjCSOAg3eID81cphRlIaiCNMlzrvjwLhCke9F8Ysl6mPYqropWQ3KW88DpdrtITRL3kx+kbFiV7vwRis8tehhOm9YM7FZp6dco4UFahT2EnkJ4mz1ixNSRyelmKYeRdnLJH5vuETtWXj8plzmflYNvznW/O8qEMURxJvU/X2T0nU+MJ5ZYlOcrIBHM3N9xN/umq4zm0h8zoJYkKqi07EREOHtGG4T2MT7hgjfwdGY4D2+oC+k+7zIc40/p5ZH5/5k4kDJqPn8eRiZwHHHQ0DHTeEaQQsL4tKEMYsafrK3J9pQ7X6lbYhUVyCHl6VJPIGbel2zy6jowrUzGzT+WQvxgZWcZgkCPlB8ARa3gQKH8voghMfMjiOAjbq+VQi9E7U/Il9C+dTYOlqZqLZtXwBANMYgSaEJTHqYlAiGoL5DtA/aauTC/sDzmRIIBvSN3KSEf2ub6nXM/RU4b+Y1LiLaGiSFS9CQvMZ/WIUbh0buyB8qjs= ceph-833368da-ec9c-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T00:47:44.978 INFO:teuthology.orchestra.run.smithi169.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4X76kTFMVUKlH9cG7YjCSOAg3eID81cphRlIaiCNMlzrvjwLhCke9F8Ysl6mPYqropWQ3KW88DpdrtITRL3kx+kbFiV7vwRis8tehhOm9YM7FZp6dco4UFahT2EnkJ4mz1ixNSRyelmKYeRdnLJH5vuETtWXj8plzmflYNvznW/O8qEMURxJvU/X2T0nU+MJ5ZYlOcrIBHM3N9xN/umq4zm0h8zoJYkKqi07EREOHtGG4T2MT7hgjfwdGY4D2+oC+k+7zIc40/p5ZH5/5k4kDJqPn8eRiZwHHHQ0DHTeEaQQsL4tKEMYsafrK3J9pQ7X6lbYhUVyCHl6VJPIGbel2zy6jowrUzGzT+WQvxgZWcZgkCPlB8ARa3gQKH8voghMfMjiOAjbq+VQi9E7U/Il9C+dTYOlqZqLZtXwBANMYgSaEJTHqYlAiGoL5DtA/aauTC/sDzmRIIBvSN3KSEf2ub6nXM/RU4b+Y1LiLaGiSFS9CQvMZ/WIUbh0buyB8qjs= ceph-833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T00:47:44.992 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4X76kTFMVUKlH9cG7YjCSOAg3eID81cphRlIaiCNMlzrvjwLhCke9F8Ysl6mPYqropWQ3KW88DpdrtITRL3kx+kbFiV7vwRis8tehhOm9YM7FZp6dco4UFahT2EnkJ4mz1ixNSRyelmKYeRdnLJH5vuETtWXj8plzmflYNvznW/O8qEMURxJvU/X2T0nU+MJ5ZYlOcrIBHM3N9xN/umq4zm0h8zoJYkKqi07EREOHtGG4T2MT7hgjfwdGY4D2+oC+k+7zIc40/p5ZH5/5k4kDJqPn8eRiZwHHHQ0DHTeEaQQsL4tKEMYsafrK3J9pQ7X6lbYhUVyCHl6VJPIGbel2zy6jowrUzGzT+WQvxgZWcZgkCPlB8ARa3gQKH8voghMfMjiOAjbq+VQi9E7U/Il9C+dTYOlqZqLZtXwBANMYgSaEJTHqYlAiGoL5DtA/aauTC/sDzmRIIBvSN3KSEf2ub6nXM/RU4b+Y1LiLaGiSFS9CQvMZ/WIUbh0buyB8qjs= ceph-833368da-ec9c-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T00:47:45.042 INFO:teuthology.orchestra.run.smithi174.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4X76kTFMVUKlH9cG7YjCSOAg3eID81cphRlIaiCNMlzrvjwLhCke9F8Ysl6mPYqropWQ3KW88DpdrtITRL3kx+kbFiV7vwRis8tehhOm9YM7FZp6dco4UFahT2EnkJ4mz1ixNSRyelmKYeRdnLJH5vuETtWXj8plzmflYNvznW/O8qEMURxJvU/X2T0nU+MJ5ZYlOcrIBHM3N9xN/umq4zm0h8zoJYkKqi07EREOHtGG4T2MT7hgjfwdGY4D2+oC+k+7zIc40/p5ZH5/5k4kDJqPn8eRiZwHHHQ0DHTeEaQQsL4tKEMYsafrK3J9pQ7X6lbYhUVyCHl6VJPIGbel2zy6jowrUzGzT+WQvxgZWcZgkCPlB8ARa3gQKH8voghMfMjiOAjbq+VQi9E7U/Il9C+dTYOlqZqLZtXwBANMYgSaEJTHqYlAiGoL5DtA/aauTC/sDzmRIIBvSN3KSEf2ub6nXM/RU4b+Y1LiLaGiSFS9CQvMZ/WIUbh0buyB8qjs= ceph-833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T00:47:45.053 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4X76kTFMVUKlH9cG7YjCSOAg3eID81cphRlIaiCNMlzrvjwLhCke9F8Ysl6mPYqropWQ3KW88DpdrtITRL3kx+kbFiV7vwRis8tehhOm9YM7FZp6dco4UFahT2EnkJ4mz1ixNSRyelmKYeRdnLJH5vuETtWXj8plzmflYNvznW/O8qEMURxJvU/X2T0nU+MJ5ZYlOcrIBHM3N9xN/umq4zm0h8zoJYkKqi07EREOHtGG4T2MT7hgjfwdGY4D2+oC+k+7zIc40/p5ZH5/5k4kDJqPn8eRiZwHHHQ0DHTeEaQQsL4tKEMYsafrK3J9pQ7X6lbYhUVyCHl6VJPIGbel2zy6jowrUzGzT+WQvxgZWcZgkCPlB8ARa3gQKH8voghMfMjiOAjbq+VQi9E7U/Il9C+dTYOlqZqLZtXwBANMYgSaEJTHqYlAiGoL5DtA/aauTC/sDzmRIIBvSN3KSEf2ub6nXM/RU4b+Y1LiLaGiSFS9CQvMZ/WIUbh0buyB8qjs= ceph-833368da-ec9c-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T00:47:45.099 INFO:teuthology.orchestra.run.smithi177.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4X76kTFMVUKlH9cG7YjCSOAg3eID81cphRlIaiCNMlzrvjwLhCke9F8Ysl6mPYqropWQ3KW88DpdrtITRL3kx+kbFiV7vwRis8tehhOm9YM7FZp6dco4UFahT2EnkJ4mz1ixNSRyelmKYeRdnLJH5vuETtWXj8plzmflYNvznW/O8qEMURxJvU/X2T0nU+MJ5ZYlOcrIBHM3N9xN/umq4zm0h8zoJYkKqi07EREOHtGG4T2MT7hgjfwdGY4D2+oC+k+7zIc40/p5ZH5/5k4kDJqPn8eRiZwHHHQ0DHTeEaQQsL4tKEMYsafrK3J9pQ7X6lbYhUVyCHl6VJPIGbel2zy6jowrUzGzT+WQvxgZWcZgkCPlB8ARa3gQKH8voghMfMjiOAjbq+VQi9E7U/Il9C+dTYOlqZqLZtXwBANMYgSaEJTHqYlAiGoL5DtA/aauTC/sDzmRIIBvSN3KSEf2ub6nXM/RU4b+Y1LiLaGiSFS9CQvMZ/WIUbh0buyB8qjs= ceph-833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T00:47:45.114 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-28T00:47:45.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:45 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/485980780' entity='client.admin' 2024-03-28T00:47:45.392 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:47:47.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.428+0000 7fe9c95e1700 1 -- 172.21.15.169:0/982892447 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe9c4100ac0 msgr2=0x7fe9c4100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:47.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.428+0000 7fe9c95e1700 1 --2- 172.21.15.169:0/982892447 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe9c4100ac0 0x7fe9c4100ee0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4009910 tx=0x7fe9b4009c20 comp rx=0 tx=0).stop 2024-03-28T00:47:47.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.428+0000 7fe9c95e1700 1 -- 172.21.15.169:0/982892447 shutdown_connections 2024-03-28T00:47:47.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.428+0000 7fe9c95e1700 1 --2- 172.21.15.169:0/982892447 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe9c4100ac0 0x7fe9c4100ee0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:47.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.428+0000 7fe9c95e1700 1 -- 172.21.15.169:0/982892447 >> 172.21.15.169:0/982892447 conn(0x7fe9c40fc080 msgr2=0x7fe9c40fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:47.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.428+0000 7fe9c95e1700 1 -- 172.21.15.169:0/982892447 shutdown_connections 2024-03-28T00:47:47.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.428+0000 7fe9c95e1700 1 -- 172.21.15.169:0/982892447 wait complete. 2024-03-28T00:47:47.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.428+0000 7fe9c95e1700 1 Processor -- start 2024-03-28T00:47:47.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.429+0000 7fe9c95e1700 1 -- start start 2024-03-28T00:47:47.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.429+0000 7fe9c95e1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe9c4100ac0 0x7fe9c41972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:47.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.429+0000 7fe9c95e1700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c4106b50 con 0x7fe9c4100ac0 2024-03-28T00:47:47.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.429+0000 7fe9c2ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe9c4100ac0 0x7fe9c41972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:47.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.429+0000 7fe9c2ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe9c4100ac0 0x7fe9c41972f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46684/0 (socket says 172.21.15.169:46684) 2024-03-28T00:47:47.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.429+0000 7fe9c2ffd700 1 -- 172.21.15.169:0/3933510703 learned_addr learned my addr 172.21.15.169:0/3933510703 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:47.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.429+0000 7fe9c2ffd700 1 -- 172.21.15.169:0/3933510703 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c4197830 con 0x7fe9c4100ac0 2024-03-28T00:47:47.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.429+0000 7fe9c2ffd700 1 --2- 172.21.15.169:0/3933510703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe9c4100ac0 0x7fe9c41972f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4006010 tx=0x7fe9b4005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:47.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.429+0000 7fe9c0ff9700 1 -- 172.21.15.169:0/3933510703 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9b4027070 con 0x7fe9c4100ac0 2024-03-28T00:47:47.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.430+0000 7fe9c95e1700 1 -- 172.21.15.169:0/3933510703 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c4197ac0 con 0x7fe9c4100ac0 2024-03-28T00:47:47.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.430+0000 7fe9c0ff9700 1 -- 172.21.15.169:0/3933510703 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe9b400be40 con 0x7fe9c4100ac0 2024-03-28T00:47:47.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.430+0000 7fe9c0ff9700 1 -- 172.21.15.169:0/3933510703 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9b400f460 con 0x7fe9c4100ac0 2024-03-28T00:47:47.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.430+0000 7fe9c95e1700 1 -- 172.21.15.169:0/3933510703 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c419a7b0 con 0x7fe9c4100ac0 2024-03-28T00:47:47.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.430+0000 7fe9c0ff9700 1 -- 172.21.15.169:0/3933510703 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fe9b402a070 con 0x7fe9c4100ac0 2024-03-28T00:47:47.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.430+0000 7fe9c95e1700 1 -- 172.21.15.169:0/3933510703 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c41072d0 con 0x7fe9c4100ac0 2024-03-28T00:47:47.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.431+0000 7fe9c0ff9700 1 --2- 172.21.15.169:0/3933510703 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe9b0038fc0 0x7fe9b003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:47.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.431+0000 7fe9c0ff9700 1 -- 172.21.15.169:0/3933510703 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe9b401e030 con 0x7fe9c4100ac0 2024-03-28T00:47:47.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.431+0000 7fe9ba7fc700 1 --2- 172.21.15.169:0/3933510703 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe9b0038fc0 0x7fe9b003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:47.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.431+0000 7fe9ba7fc700 1 --2- 172.21.15.169:0/3933510703 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe9b0038fc0 0x7fe9b003b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac006fd0 tx=0x7fe9ac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:47.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.435+0000 7fe9c0ff9700 1 -- 172.21.15.169:0/3933510703 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe9b4013070 con 0x7fe9c4100ac0 2024-03-28T00:47:47.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.631+0000 7fe9c95e1700 1 -- 172.21.15.169:0/3933510703 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fe9c419abb0 con 0x7fe9c4100ac0 2024-03-28T00:47:47.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.633+0000 7fe9c0ff9700 1 -- 172.21.15.169:0/3933510703 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fe9b402a360 con 0x7fe9c4100ac0 2024-03-28T00:47:47.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.635+0000 7fe9c95e1700 1 -- 172.21.15.169:0/3933510703 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe9b0038fc0 msgr2=0x7fe9b003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:47.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.635+0000 7fe9c95e1700 1 --2- 172.21.15.169:0/3933510703 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe9b0038fc0 0x7fe9b003b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac006fd0 tx=0x7fe9ac006e40 comp rx=0 tx=0).stop 2024-03-28T00:47:47.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.635+0000 7fe9c95e1700 1 -- 172.21.15.169:0/3933510703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe9c4100ac0 msgr2=0x7fe9c41972f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:47.640 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.635+0000 7fe9c95e1700 1 --2- 172.21.15.169:0/3933510703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe9c4100ac0 0x7fe9c41972f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4006010 tx=0x7fe9b4005050 comp rx=0 tx=0).stop 2024-03-28T00:47:47.640 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.635+0000 7fe9c95e1700 1 -- 172.21.15.169:0/3933510703 shutdown_connections 2024-03-28T00:47:47.640 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.635+0000 7fe9c95e1700 1 --2- 172.21.15.169:0/3933510703 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe9b0038fc0 0x7fe9b003b480 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:47.640 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.635+0000 7fe9c95e1700 1 --2- 172.21.15.169:0/3933510703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe9c4100ac0 0x7fe9c41972f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:47.640 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.635+0000 7fe9c95e1700 1 -- 172.21.15.169:0/3933510703 >> 172.21.15.169:0/3933510703 conn(0x7fe9c40fc080 msgr2=0x7fe9c4069f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:47.640 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.636+0000 7fe9c95e1700 1 -- 172.21.15.169:0/3933510703 shutdown_connections 2024-03-28T00:47:47.640 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:47.636+0000 7fe9c95e1700 1 -- 172.21.15.169:0/3933510703 wait complete. 2024-03-28T00:47:47.938 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:47 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:47:47.938 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:47 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:47:47.938 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:47 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi169", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:47:47.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:47 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:47:47.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:47 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:47:47.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:47 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:47:47.939 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:47 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3933510703' entity='client.admin' 2024-03-28T00:47:48.264 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-28T00:47:48.265 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-28T00:47:48.499 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:47:48.945 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:48 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:47:48.945 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:48 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:47:50.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.023+0000 7fbae514c700 1 -- 172.21.15.169:0/4206048417 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbae0100ac0 msgr2=0x7fbae0100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:50.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.023+0000 7fbae514c700 1 --2- 172.21.15.169:0/4206048417 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbae0100ac0 0x7fbae0100ee0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009910 tx=0x7fbad0009c20 comp rx=0 tx=0).stop 2024-03-28T00:47:50.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.024+0000 7fbae514c700 1 -- 172.21.15.169:0/4206048417 shutdown_connections 2024-03-28T00:47:50.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.024+0000 7fbae514c700 1 --2- 172.21.15.169:0/4206048417 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbae0100ac0 0x7fbae0100ee0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:50.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.024+0000 7fbae514c700 1 -- 172.21.15.169:0/4206048417 >> 172.21.15.169:0/4206048417 conn(0x7fbae00fc080 msgr2=0x7fbae00fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:50.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.024+0000 7fbae514c700 1 -- 172.21.15.169:0/4206048417 shutdown_connections 2024-03-28T00:47:50.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.024+0000 7fbae514c700 1 -- 172.21.15.169:0/4206048417 wait complete. 2024-03-28T00:47:50.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.024+0000 7fbae514c700 1 Processor -- start 2024-03-28T00:47:50.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.024+0000 7fbae514c700 1 -- start start 2024-03-28T00:47:50.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.025+0000 7fbae514c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbae0100ac0 0x7fbae006aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:50.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.025+0000 7fbae514c700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae0106b50 con 0x7fbae0100ac0 2024-03-28T00:47:50.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.025+0000 7fbaded9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbae0100ac0 0x7fbae006aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:50.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.025+0000 7fbaded9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbae0100ac0 0x7fbae006aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46706/0 (socket says 172.21.15.169:46706) 2024-03-28T00:47:50.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.025+0000 7fbaded9d700 1 -- 172.21.15.169:0/611559603 learned_addr learned my addr 172.21.15.169:0/611559603 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:50.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.025+0000 7fbaded9d700 1 -- 172.21.15.169:0/611559603 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae019a030 con 0x7fbae0100ac0 2024-03-28T00:47:50.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.025+0000 7fbaded9d700 1 --2- 172.21.15.169:0/611559603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbae0100ac0 0x7fbae006aa20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbad0005850 tx=0x7fbad0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:50.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.025+0000 7fbac7fff700 1 -- 172.21.15.169:0/611559603 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbad0026070 con 0x7fbae0100ac0 2024-03-28T00:47:50.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.025+0000 7fbac7fff700 1 -- 172.21.15.169:0/611559603 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbad00044a0 con 0x7fbae0100ac0 2024-03-28T00:47:50.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.025+0000 7fbae514c700 1 -- 172.21.15.169:0/611559603 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae019a200 con 0x7fbae0100ac0 2024-03-28T00:47:50.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.025+0000 7fbac7fff700 1 -- 172.21.15.169:0/611559603 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbad0003ab0 con 0x7fbae0100ac0 2024-03-28T00:47:50.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.025+0000 7fbae514c700 1 -- 172.21.15.169:0/611559603 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae019a620 con 0x7fbae0100ac0 2024-03-28T00:47:50.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.026+0000 7fbac7fff700 1 -- 172.21.15.169:0/611559603 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fbad0029070 con 0x7fbae0100ac0 2024-03-28T00:47:50.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.026+0000 7fbae514c700 1 -- 172.21.15.169:0/611559603 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae01072d0 con 0x7fbae0100ac0 2024-03-28T00:47:50.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.026+0000 7fbac7fff700 1 --2- 172.21.15.169:0/611559603 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbac8038fc0 0x7fbac803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:50.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.026+0000 7fbac7fff700 1 -- 172.21.15.169:0/611559603 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbad001e070 con 0x7fbae0100ac0 2024-03-28T00:47:50.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.027+0000 7fbade59c700 1 --2- 172.21.15.169:0/611559603 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbac8038fc0 0x7fbac803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:50.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.027+0000 7fbade59c700 1 --2- 172.21.15.169:0/611559603 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbac8038fc0 0x7fbac803b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbad4006fd0 tx=0x7fbad4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:50.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.032+0000 7fbac7fff700 1 -- 172.21.15.169:0/611559603 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbad0013070 con 0x7fbae0100ac0 2024-03-28T00:47:50.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.235+0000 7fbae514c700 1 -- 172.21.15.169:0/611559603 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fbae002dcf0 con 0x7fbac8038fc0 2024-03-28T00:47:50.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.239+0000 7fbac7fff700 1 -- 172.21.15.169:0/611559603 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbae002dcf0 con 0x7fbac8038fc0 2024-03-28T00:47:50.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.241+0000 7fbae514c700 1 -- 172.21.15.169:0/611559603 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbac8038fc0 msgr2=0x7fbac803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:50.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.241+0000 7fbae514c700 1 --2- 172.21.15.169:0/611559603 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbac8038fc0 0x7fbac803b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbad4006fd0 tx=0x7fbad4006e40 comp rx=0 tx=0).stop 2024-03-28T00:47:50.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.241+0000 7fbae514c700 1 -- 172.21.15.169:0/611559603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbae0100ac0 msgr2=0x7fbae006aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:50.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.241+0000 7fbae514c700 1 --2- 172.21.15.169:0/611559603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbae0100ac0 0x7fbae006aa20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbad0005850 tx=0x7fbad0005030 comp rx=0 tx=0).stop 2024-03-28T00:47:50.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.241+0000 7fbae514c700 1 -- 172.21.15.169:0/611559603 shutdown_connections 2024-03-28T00:47:50.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.241+0000 7fbae514c700 1 --2- 172.21.15.169:0/611559603 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbac8038fc0 0x7fbac803b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:50.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.241+0000 7fbae514c700 1 --2- 172.21.15.169:0/611559603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbae0100ac0 0x7fbae006aa20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:50.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.241+0000 7fbae514c700 1 -- 172.21.15.169:0/611559603 >> 172.21.15.169:0/611559603 conn(0x7fbae00fc080 msgr2=0x7fbae00fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:50.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.241+0000 7fbae514c700 1 -- 172.21.15.169:0/611559603 shutdown_connections 2024-03-28T00:47:50.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:50.241+0000 7fbae514c700 1 -- 172.21.15.169:0/611559603 wait complete. 2024-03-28T00:47:50.984 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi174 2024-03-28T00:47:50.985 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-28T00:47:50.985 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/etc/ceph/ceph.conf 2024-03-28T00:47:51.004 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-28T00:47:51.004 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T00:47:51.061 INFO:tasks.cephadm:Adding host smithi174 to orchestrator... 2024-03-28T00:47:51.061 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph orch host add smithi174 2024-03-28T00:47:51.304 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:47:51.520 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:51 smithi169 ceph-mon[32274]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:47:51.520 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:51 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:47:51.520 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:51 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:47:51.520 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:51 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:47:51.520 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:51 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:47:51.520 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:51 smithi169 ceph-mon[32274]: Updating smithi169:/etc/ceph/ceph.conf 2024-03-28T00:47:51.520 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:51 smithi169 ceph-mon[32274]: Updating smithi169:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T00:47:51.520 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:51 smithi169 ceph-mon[32274]: Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2024-03-28T00:47:51.520 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:51 smithi169 ceph-mon[32274]: Updating smithi169:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T00:47:51.520 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:51 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:47:51.521 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:51 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:47:51.521 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:51 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:47:52.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.847+0000 7fa0a3cf8700 1 -- 172.21.15.169:0/85929307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa09c106a50 msgr2=0x7fa09c106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:52.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.847+0000 7fa0a3cf8700 1 --2- 172.21.15.169:0/85929307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa09c106a50 0x7fa09c106e30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa08c009910 tx=0x7fa08c009c20 comp rx=0 tx=0).stop 2024-03-28T00:47:52.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.847+0000 7fa0a3cf8700 1 -- 172.21.15.169:0/85929307 shutdown_connections 2024-03-28T00:47:52.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.847+0000 7fa0a3cf8700 1 --2- 172.21.15.169:0/85929307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa09c106a50 0x7fa09c106e30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:52.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.847+0000 7fa0a3cf8700 1 -- 172.21.15.169:0/85929307 >> 172.21.15.169:0/85929307 conn(0x7fa09c0fc580 msgr2=0x7fa09c0fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:52.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.847+0000 7fa0a3cf8700 1 -- 172.21.15.169:0/85929307 shutdown_connections 2024-03-28T00:47:52.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.847+0000 7fa0a3cf8700 1 -- 172.21.15.169:0/85929307 wait complete. 2024-03-28T00:47:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.848+0000 7fa0a3cf8700 1 Processor -- start 2024-03-28T00:47:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.848+0000 7fa0a3cf8700 1 -- start start 2024-03-28T00:47:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.848+0000 7fa0a3cf8700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa09c106a50 0x7fa09c197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.848+0000 7fa0a3cf8700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c10b090 con 0x7fa09c106a50 2024-03-28T00:47:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.848+0000 7fa0a1a94700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa09c106a50 0x7fa09c197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.848+0000 7fa0a1a94700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa09c106a50 0x7fa09c197700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46718/0 (socket says 172.21.15.169:46718) 2024-03-28T00:47:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.848+0000 7fa0a1a94700 1 -- 172.21.15.169:0/409139198 learned_addr learned my addr 172.21.15.169:0/409139198 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.848+0000 7fa0a1a94700 1 -- 172.21.15.169:0/409139198 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09c197c40 con 0x7fa09c106a50 2024-03-28T00:47:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.848+0000 7fa0a1a94700 1 --2- 172.21.15.169:0/409139198 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa09c106a50 0x7fa09c197700 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa08c005850 tx=0x7fa08c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.848+0000 7fa092ffd700 1 -- 172.21.15.169:0/409139198 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa08c026070 con 0x7fa09c106a50 2024-03-28T00:47:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.848+0000 7fa092ffd700 1 -- 172.21.15.169:0/409139198 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa08c00bbd0 con 0x7fa09c106a50 2024-03-28T00:47:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.848+0000 7fa0a3cf8700 1 -- 172.21.15.169:0/409139198 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa09c197f30 con 0x7fa09c106a50 2024-03-28T00:47:52.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.849+0000 7fa092ffd700 1 -- 172.21.15.169:0/409139198 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa08c003970 con 0x7fa09c106a50 2024-03-28T00:47:52.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.849+0000 7fa0a3cf8700 1 -- 172.21.15.169:0/409139198 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa09c19bfd0 con 0x7fa09c106a50 2024-03-28T00:47:52.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.849+0000 7fa092ffd700 1 -- 172.21.15.169:0/409139198 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fa08c029070 con 0x7fa09c106a50 2024-03-28T00:47:52.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.849+0000 7fa0a3cf8700 1 -- 172.21.15.169:0/409139198 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa09c102190 con 0x7fa09c106a50 2024-03-28T00:47:52.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.850+0000 7fa092ffd700 1 --2- 172.21.15.169:0/409139198 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa088038fc0 0x7fa08803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:52.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.850+0000 7fa092ffd700 1 -- 172.21.15.169:0/409139198 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa08c01e070 con 0x7fa09c106a50 2024-03-28T00:47:52.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.850+0000 7fa0a1293700 1 --2- 172.21.15.169:0/409139198 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa088038fc0 0x7fa08803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:52.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.850+0000 7fa0a1293700 1 --2- 172.21.15.169:0/409139198 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa088038fc0 0x7fa08803b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa098006fd0 tx=0x7fa098006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:52.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:52.854+0000 7fa092ffd700 1 -- 172.21.15.169:0/409139198 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa08c050030 con 0x7fa09c106a50 2024-03-28T00:47:53.057 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:53.053+0000 7fa0a3cf8700 1 -- 172.21.15.169:0/409139198 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi174", "target": ["mon-mgr", ""]}) v1 -- 0x7fa09c198a60 con 0x7fa088038fc0 2024-03-28T00:47:53.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:53 smithi169 ceph-mon[32274]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi174", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:47:54.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:54 smithi169 ceph-mon[32274]: Deploying cephadm binary to smithi174 2024-03-28T00:47:57.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:57.127+0000 7fa092ffd700 1 -- 172.21.15.169:0/409139198 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fa09c198a60 con 0x7fa088038fc0 2024-03-28T00:47:57.131 INFO:teuthology.orchestra.run.smithi169.stdout:Added host 'smithi174' with addr '172.21.15.174' 2024-03-28T00:47:57.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:57.129+0000 7fa0a3cf8700 1 -- 172.21.15.169:0/409139198 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa088038fc0 msgr2=0x7fa08803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:57.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:57.129+0000 7fa0a3cf8700 1 --2- 172.21.15.169:0/409139198 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa088038fc0 0x7fa08803b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa098006fd0 tx=0x7fa098006e40 comp rx=0 tx=0).stop 2024-03-28T00:47:57.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:57.129+0000 7fa0a3cf8700 1 -- 172.21.15.169:0/409139198 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa09c106a50 msgr2=0x7fa09c197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:57.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:57.129+0000 7fa0a3cf8700 1 --2- 172.21.15.169:0/409139198 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa09c106a50 0x7fa09c197700 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa08c005850 tx=0x7fa08c005030 comp rx=0 tx=0).stop 2024-03-28T00:47:57.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:57.129+0000 7fa0a3cf8700 1 -- 172.21.15.169:0/409139198 shutdown_connections 2024-03-28T00:47:57.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:57.129+0000 7fa0a3cf8700 1 --2- 172.21.15.169:0/409139198 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa088038fc0 0x7fa08803b480 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:57.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:57.129+0000 7fa0a3cf8700 1 --2- 172.21.15.169:0/409139198 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa09c106a50 0x7fa09c197700 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:57.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:57.129+0000 7fa0a3cf8700 1 -- 172.21.15.169:0/409139198 >> 172.21.15.169:0/409139198 conn(0x7fa09c0fc580 msgr2=0x7fa09c190520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:57.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:57.129+0000 7fa0a3cf8700 1 -- 172.21.15.169:0/409139198 shutdown_connections 2024-03-28T00:47:57.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:57.129+0000 7fa0a3cf8700 1 -- 172.21.15.169:0/409139198 wait complete. 2024-03-28T00:47:57.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:57 smithi169 ceph-mon[32274]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:47:57.695 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T00:47:57.928 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:47:58.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:58 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:47:58.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:58 smithi169 ceph-mon[32274]: Added host smithi174 2024-03-28T00:47:58.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:58 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:47:58.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:58 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:47:59.294 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:59 smithi169 ceph-mon[32274]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:47:59.294 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:47:59 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:47:59.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.393+0000 7fd827f4e700 1 -- 172.21.15.169:0/2449225111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd820106a70 msgr2=0x7fd820106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:59.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.393+0000 7fd827f4e700 1 --2- 172.21.15.169:0/2449225111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd820106a70 0x7fd820106e50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd81800b0a0 tx=0x7fd81800b3b0 comp rx=0 tx=0).stop 2024-03-28T00:47:59.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.393+0000 7fd827f4e700 1 -- 172.21.15.169:0/2449225111 shutdown_connections 2024-03-28T00:47:59.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.393+0000 7fd827f4e700 1 --2- 172.21.15.169:0/2449225111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd820106a70 0x7fd820106e50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:59.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.393+0000 7fd827f4e700 1 -- 172.21.15.169:0/2449225111 >> 172.21.15.169:0/2449225111 conn(0x7fd8200fc560 msgr2=0x7fd8200fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:59.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.393+0000 7fd827f4e700 1 -- 172.21.15.169:0/2449225111 shutdown_connections 2024-03-28T00:47:59.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.393+0000 7fd827f4e700 1 -- 172.21.15.169:0/2449225111 wait complete. 2024-03-28T00:47:59.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.394+0000 7fd827f4e700 1 Processor -- start 2024-03-28T00:47:59.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.394+0000 7fd827f4e700 1 -- start start 2024-03-28T00:47:59.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.394+0000 7fd827f4e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd820106a70 0x7fd820197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:59.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.394+0000 7fd827f4e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82010b0b0 con 0x7fd820106a70 2024-03-28T00:47:59.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.394+0000 7fd825cea700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd820106a70 0x7fd820197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:59.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.394+0000 7fd825cea700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd820106a70 0x7fd820197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:42522/0 (socket says 172.21.15.169:42522) 2024-03-28T00:47:59.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.394+0000 7fd825cea700 1 -- 172.21.15.169:0/1174343946 learned_addr learned my addr 172.21.15.169:0/1174343946 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:47:59.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.395+0000 7fd825cea700 1 -- 172.21.15.169:0/1174343946 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd820197b50 con 0x7fd820106a70 2024-03-28T00:47:59.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.395+0000 7fd825cea700 1 --2- 172.21.15.169:0/1174343946 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd820106a70 0x7fd820197610 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd818015040 tx=0x7fd818003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:59.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.395+0000 7fd816ffd700 1 -- 172.21.15.169:0/1174343946 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd818017040 con 0x7fd820106a70 2024-03-28T00:47:59.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.395+0000 7fd816ffd700 1 -- 172.21.15.169:0/1174343946 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd818022070 con 0x7fd820106a70 2024-03-28T00:47:59.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.395+0000 7fd827f4e700 1 -- 172.21.15.169:0/1174343946 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd820197de0 con 0x7fd820106a70 2024-03-28T00:47:59.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.395+0000 7fd816ffd700 1 -- 172.21.15.169:0/1174343946 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd81801d5d0 con 0x7fd820106a70 2024-03-28T00:47:59.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.395+0000 7fd827f4e700 1 -- 172.21.15.169:0/1174343946 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd82019bee0 con 0x7fd820106a70 2024-03-28T00:47:59.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.396+0000 7fd816ffd700 1 -- 172.21.15.169:0/1174343946 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fd81801d8d0 con 0x7fd820106a70 2024-03-28T00:47:59.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.396+0000 7fd827f4e700 1 -- 172.21.15.169:0/1174343946 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8201021b0 con 0x7fd820106a70 2024-03-28T00:47:59.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.396+0000 7fd816ffd700 1 --2- 172.21.15.169:0/1174343946 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd80c038fc0 0x7fd80c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:47:59.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.396+0000 7fd816ffd700 1 -- 172.21.15.169:0/1174343946 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd81800f050 con 0x7fd820106a70 2024-03-28T00:47:59.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.396+0000 7fd8254e9700 1 --2- 172.21.15.169:0/1174343946 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd80c038fc0 0x7fd80c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:47:59.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.397+0000 7fd8254e9700 1 --2- 172.21.15.169:0/1174343946 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd80c038fc0 0x7fd80c03b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd810006fd0 tx=0x7fd810006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:47:59.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.401+0000 7fd816ffd700 1 -- 172.21.15.169:0/1174343946 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd818007e20 con 0x7fd820106a70 2024-03-28T00:47:59.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.599+0000 7fd827f4e700 1 -- 172.21.15.169:0/1174343946 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd820198970 con 0x7fd80c038fc0 2024-03-28T00:47:59.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.600+0000 7fd816ffd700 1 -- 172.21.15.169:0/1174343946 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7fd820198970 con 0x7fd80c038fc0 2024-03-28T00:47:59.604 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:47:59.604 INFO:teuthology.orchestra.run.smithi169.stdout:[{"addr": "172.21.15.169", "hostname": "smithi169", "labels": [], "status": ""}, {"addr": "172.21.15.174", "hostname": "smithi174", "labels": [], "status": ""}] 2024-03-28T00:47:59.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.602+0000 7fd827f4e700 1 -- 172.21.15.169:0/1174343946 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd80c038fc0 msgr2=0x7fd80c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:59.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.602+0000 7fd827f4e700 1 --2- 172.21.15.169:0/1174343946 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd80c038fc0 0x7fd80c03b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd810006fd0 tx=0x7fd810006e40 comp rx=0 tx=0).stop 2024-03-28T00:47:59.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.602+0000 7fd827f4e700 1 -- 172.21.15.169:0/1174343946 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd820106a70 msgr2=0x7fd820197610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:47:59.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.602+0000 7fd827f4e700 1 --2- 172.21.15.169:0/1174343946 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd820106a70 0x7fd820197610 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd818015040 tx=0x7fd818003d50 comp rx=0 tx=0).stop 2024-03-28T00:47:59.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.602+0000 7fd827f4e700 1 -- 172.21.15.169:0/1174343946 shutdown_connections 2024-03-28T00:47:59.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.602+0000 7fd827f4e700 1 --2- 172.21.15.169:0/1174343946 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd80c038fc0 0x7fd80c03b480 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:59.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.602+0000 7fd827f4e700 1 --2- 172.21.15.169:0/1174343946 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd820106a70 0x7fd820197610 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:47:59.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.602+0000 7fd827f4e700 1 -- 172.21.15.169:0/1174343946 >> 172.21.15.169:0/1174343946 conn(0x7fd8200fc560 msgr2=0x7fd820069fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:47:59.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.602+0000 7fd827f4e700 1 -- 172.21.15.169:0/1174343946 shutdown_connections 2024-03-28T00:47:59.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:47:59.602+0000 7fd827f4e700 1 -- 172.21.15.169:0/1174343946 wait complete. 2024-03-28T00:48:00.211 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi177 2024-03-28T00:48:00.211 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-28T00:48:00.211 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/etc/ceph/ceph.conf 2024-03-28T00:48:00.229 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-28T00:48:00.229 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T00:48:00.286 INFO:tasks.cephadm:Adding host smithi177 to orchestrator... 2024-03-28T00:48:00.286 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph orch host add smithi177 2024-03-28T00:48:00.523 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:48:00.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:00 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:00.585 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:00 smithi169 ceph-mon[32274]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:48:00.585 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:00 smithi169 ceph-mon[32274]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:01.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.964+0000 7f749d3c0700 1 -- 172.21.15.169:0/2415122540 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7498106a70 msgr2=0x7f7498106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:01.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.964+0000 7f749d3c0700 1 --2- 172.21.15.169:0/2415122540 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7498106a70 0x7f7498106e50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7490009910 tx=0x7f7490009c20 comp rx=0 tx=0).stop 2024-03-28T00:48:01.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.964+0000 7f749d3c0700 1 -- 172.21.15.169:0/2415122540 shutdown_connections 2024-03-28T00:48:01.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.964+0000 7f749d3c0700 1 --2- 172.21.15.169:0/2415122540 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7498106a70 0x7f7498106e50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:01.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.964+0000 7f749d3c0700 1 -- 172.21.15.169:0/2415122540 >> 172.21.15.169:0/2415122540 conn(0x7f74980fc5a0 msgr2=0x7f74980fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:48:01.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.964+0000 7f749d3c0700 1 -- 172.21.15.169:0/2415122540 shutdown_connections 2024-03-28T00:48:01.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.964+0000 7f749d3c0700 1 -- 172.21.15.169:0/2415122540 wait complete. 2024-03-28T00:48:01.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.964+0000 7f749d3c0700 1 Processor -- start 2024-03-28T00:48:01.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.965+0000 7f749d3c0700 1 -- start start 2024-03-28T00:48:01.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.965+0000 7f749d3c0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7498106a70 0x7f7498196590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:01.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.965+0000 7f749d3c0700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749810b0b0 con 0x7f7498106a70 2024-03-28T00:48:01.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.965+0000 7f7496ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7498106a70 0x7f7498196590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:01.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.965+0000 7f7496ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7498106a70 0x7f7498196590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:42540/0 (socket says 172.21.15.169:42540) 2024-03-28T00:48:01.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.965+0000 7f7496ffd700 1 -- 172.21.15.169:0/904377954 learned_addr learned my addr 172.21.15.169:0/904377954 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:48:01.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.965+0000 7f7496ffd700 1 -- 172.21.15.169:0/904377954 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7498196ad0 con 0x7f7498106a70 2024-03-28T00:48:01.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.965+0000 7f7496ffd700 1 --2- 172.21.15.169:0/904377954 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7498106a70 0x7f7498196590 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7490005850 tx=0x7f7490005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:48:01.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.965+0000 7f747ffff700 1 -- 172.21.15.169:0/904377954 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7490026070 con 0x7f7498106a70 2024-03-28T00:48:01.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.965+0000 7f747ffff700 1 -- 172.21.15.169:0/904377954 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f749000bbd0 con 0x7f7498106a70 2024-03-28T00:48:01.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.965+0000 7f749d3c0700 1 -- 172.21.15.169:0/904377954 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7498196dc0 con 0x7f7498106a70 2024-03-28T00:48:01.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.965+0000 7f747ffff700 1 -- 172.21.15.169:0/904377954 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7490003970 con 0x7f7498106a70 2024-03-28T00:48:01.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.965+0000 7f749d3c0700 1 -- 172.21.15.169:0/904377954 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f749819ae60 con 0x7f7498106a70 2024-03-28T00:48:01.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.966+0000 7f747ffff700 1 -- 172.21.15.169:0/904377954 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f7490029070 con 0x7f7498106a70 2024-03-28T00:48:01.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.966+0000 7f747ffff700 1 --2- 172.21.15.169:0/904377954 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f748003b1d0 0x7f748003d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:01.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.966+0000 7f749d3c0700 1 -- 172.21.15.169:0/904377954 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74981021b0 con 0x7f7498106a70 2024-03-28T00:48:01.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.966+0000 7f747ffff700 1 -- 172.21.15.169:0/904377954 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f749001e070 con 0x7f7498106a70 2024-03-28T00:48:01.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.967+0000 7f74967fc700 1 --2- 172.21.15.169:0/904377954 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f748003b1d0 0x7f748003d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:01.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.967+0000 7f74967fc700 1 --2- 172.21.15.169:0/904377954 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f748003b1d0 0x7f748003d690 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f748c006fd0 tx=0x7f748c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:48:01.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:01.971+0000 7f747ffff700 1 -- 172.21.15.169:0/904377954 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7490013070 con 0x7f7498106a70 2024-03-28T00:48:02.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:02.172+0000 7f749d3c0700 1 -- 172.21.15.169:0/904377954 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}) v1 -- 0x7f74981978f0 con 0x7f748003b1d0 2024-03-28T00:48:03.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:03 smithi169 ceph-mon[32274]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:03.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:03 smithi169 ceph-mon[32274]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:48:03.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:03 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:03.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:03 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:03.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:03 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:03.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:03 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:03.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:03 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:48:03.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:03 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:03.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:03 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:48:04.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:04 smithi169 ceph-mon[32274]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-28T00:48:04.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:04 smithi169 ceph-mon[32274]: Updating smithi174:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T00:48:04.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:04 smithi169 ceph-mon[32274]: Deploying cephadm binary to smithi177 2024-03-28T00:48:04.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:04 smithi169 ceph-mon[32274]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-03-28T00:48:04.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:04 smithi169 ceph-mon[32274]: Updating smithi174:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T00:48:04.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:04 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:04.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:04 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:04.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:04 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:05.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:05 smithi169 ceph-mon[32274]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:06.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:06.389+0000 7f747ffff700 1 -- 172.21.15.169:0/904377954 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f74981978f0 con 0x7f748003b1d0 2024-03-28T00:48:06.393 INFO:teuthology.orchestra.run.smithi169.stdout:Added host 'smithi177' with addr '172.21.15.177' 2024-03-28T00:48:06.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:06.392+0000 7f749d3c0700 1 -- 172.21.15.169:0/904377954 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f748003b1d0 msgr2=0x7f748003d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:06.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:06.392+0000 7f749d3c0700 1 --2- 172.21.15.169:0/904377954 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f748003b1d0 0x7f748003d690 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f748c006fd0 tx=0x7f748c006e40 comp rx=0 tx=0).stop 2024-03-28T00:48:06.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:06.392+0000 7f749d3c0700 1 -- 172.21.15.169:0/904377954 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7498106a70 msgr2=0x7f7498196590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:06.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:06.392+0000 7f749d3c0700 1 --2- 172.21.15.169:0/904377954 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7498106a70 0x7f7498196590 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7490005850 tx=0x7f7490005030 comp rx=0 tx=0).stop 2024-03-28T00:48:06.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:06.392+0000 7f749d3c0700 1 -- 172.21.15.169:0/904377954 shutdown_connections 2024-03-28T00:48:06.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:06.392+0000 7f749d3c0700 1 --2- 172.21.15.169:0/904377954 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f748003b1d0 0x7f748003d690 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:06.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:06.392+0000 7f749d3c0700 1 --2- 172.21.15.169:0/904377954 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7498106a70 0x7f7498196590 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:06.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:06.392+0000 7f749d3c0700 1 -- 172.21.15.169:0/904377954 >> 172.21.15.169:0/904377954 conn(0x7f74980fc5a0 msgr2=0x7f74980fd2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:48:06.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:06.392+0000 7f749d3c0700 1 -- 172.21.15.169:0/904377954 shutdown_connections 2024-03-28T00:48:06.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:06.392+0000 7f749d3c0700 1 -- 172.21.15.169:0/904377954 wait complete. 2024-03-28T00:48:06.989 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T00:48:07.225 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:48:07.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:07 smithi169 ceph-mon[32274]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:07.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:07 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:07.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:07 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:48:07.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:07 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:08.166 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:08 smithi169 ceph-mon[32274]: Added host smithi177 2024-03-28T00:48:08.166 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:08 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:08.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.677+0000 7f1457b71700 1 -- 172.21.15.169:0/2498271490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1450106a70 msgr2=0x7f1450106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:08.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.677+0000 7f1457b71700 1 --2- 172.21.15.169:0/2498271490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1450106a70 0x7f1450106e50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f144c009910 tx=0x7f144c009c20 comp rx=0 tx=0).stop 2024-03-28T00:48:08.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.677+0000 7f1457b71700 1 -- 172.21.15.169:0/2498271490 shutdown_connections 2024-03-28T00:48:08.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.677+0000 7f1457b71700 1 --2- 172.21.15.169:0/2498271490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1450106a70 0x7f1450106e50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:08.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.677+0000 7f1457b71700 1 -- 172.21.15.169:0/2498271490 >> 172.21.15.169:0/2498271490 conn(0x7f14500fc5a0 msgr2=0x7f14500fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:48:08.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.677+0000 7f1457b71700 1 -- 172.21.15.169:0/2498271490 shutdown_connections 2024-03-28T00:48:08.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.677+0000 7f1457b71700 1 -- 172.21.15.169:0/2498271490 wait complete. 2024-03-28T00:48:08.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.678+0000 7f1457b71700 1 Processor -- start 2024-03-28T00:48:08.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.678+0000 7f1457b71700 1 -- start start 2024-03-28T00:48:08.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.678+0000 7f1457b71700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1450106a70 0x7f14500fca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:08.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.678+0000 7f1457b71700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145010b0b0 con 0x7f1450106a70 2024-03-28T00:48:08.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.678+0000 7f145590d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1450106a70 0x7f14500fca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:08.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.678+0000 7f145590d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1450106a70 0x7f14500fca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34718/0 (socket says 172.21.15.169:34718) 2024-03-28T00:48:08.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.678+0000 7f145590d700 1 -- 172.21.15.169:0/1858217105 learned_addr learned my addr 172.21.15.169:0/1858217105 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:48:08.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.679+0000 7f145590d700 1 -- 172.21.15.169:0/1858217105 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14500fe940 con 0x7f1450106a70 2024-03-28T00:48:08.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.679+0000 7f145590d700 1 --2- 172.21.15.169:0/1858217105 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1450106a70 0x7f14500fca50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f144c006010 tx=0x7f144c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:48:08.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.679+0000 7f1446ffd700 1 -- 172.21.15.169:0/1858217105 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f144c026070 con 0x7f1450106a70 2024-03-28T00:48:08.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.679+0000 7f1457b71700 1 -- 172.21.15.169:0/1858217105 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14500fcff0 con 0x7f1450106a70 2024-03-28T00:48:08.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.679+0000 7f1446ffd700 1 -- 172.21.15.169:0/1858217105 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f144c00bc30 con 0x7f1450106a70 2024-03-28T00:48:08.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.679+0000 7f1446ffd700 1 -- 172.21.15.169:0/1858217105 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f144c0039d0 con 0x7f1450106a70 2024-03-28T00:48:08.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.679+0000 7f1457b71700 1 -- 172.21.15.169:0/1858217105 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14500fd4b0 con 0x7f1450106a70 2024-03-28T00:48:08.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.680+0000 7f1446ffd700 1 -- 172.21.15.169:0/1858217105 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f144c029070 con 0x7f1450106a70 2024-03-28T00:48:08.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.680+0000 7f1457b71700 1 -- 172.21.15.169:0/1858217105 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14501021b0 con 0x7f1450106a70 2024-03-28T00:48:08.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.680+0000 7f1446ffd700 1 --2- 172.21.15.169:0/1858217105 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f143c038fc0 0x7f143c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:08.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.680+0000 7f1446ffd700 1 -- 172.21.15.169:0/1858217105 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f144c01e070 con 0x7f1450106a70 2024-03-28T00:48:08.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.680+0000 7f145510c700 1 --2- 172.21.15.169:0/1858217105 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f143c038fc0 0x7f143c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:08.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.680+0000 7f145510c700 1 --2- 172.21.15.169:0/1858217105 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f143c038fc0 0x7f143c03b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1440006fd0 tx=0x7f1440006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:48:08.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.685+0000 7f1446ffd700 1 -- 172.21.15.169:0/1858217105 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f144c013070 con 0x7f1450106a70 2024-03-28T00:48:08.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.887+0000 7f1457b71700 1 -- 172.21.15.169:0/1858217105 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f14500fdf80 con 0x7f143c038fc0 2024-03-28T00:48:08.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.888+0000 7f1446ffd700 1 -- 172.21.15.169:0/1858217105 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+241 (secure 0 0 0) 0x7f14500fdf80 con 0x7f143c038fc0 2024-03-28T00:48:08.891 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:48:08.892 INFO:teuthology.orchestra.run.smithi169.stdout:[{"addr": "172.21.15.169", "hostname": "smithi169", "labels": [], "status": ""}, {"addr": "172.21.15.174", "hostname": "smithi174", "labels": [], "status": ""}, {"addr": "172.21.15.177", "hostname": "smithi177", "labels": [], "status": ""}] 2024-03-28T00:48:08.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.890+0000 7f1457b71700 1 -- 172.21.15.169:0/1858217105 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f143c038fc0 msgr2=0x7f143c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:08.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.890+0000 7f1457b71700 1 --2- 172.21.15.169:0/1858217105 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f143c038fc0 0x7f143c03b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1440006fd0 tx=0x7f1440006e40 comp rx=0 tx=0).stop 2024-03-28T00:48:08.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.890+0000 7f1457b71700 1 -- 172.21.15.169:0/1858217105 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1450106a70 msgr2=0x7f14500fca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:08.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.890+0000 7f1457b71700 1 --2- 172.21.15.169:0/1858217105 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1450106a70 0x7f14500fca50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f144c006010 tx=0x7f144c005030 comp rx=0 tx=0).stop 2024-03-28T00:48:08.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.890+0000 7f1457b71700 1 -- 172.21.15.169:0/1858217105 shutdown_connections 2024-03-28T00:48:08.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.890+0000 7f1457b71700 1 --2- 172.21.15.169:0/1858217105 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f143c038fc0 0x7f143c03b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:08.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.890+0000 7f1457b71700 1 --2- 172.21.15.169:0/1858217105 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1450106a70 0x7f14500fca50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:08.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.890+0000 7f1457b71700 1 -- 172.21.15.169:0/1858217105 >> 172.21.15.169:0/1858217105 conn(0x7f14500fc5a0 msgr2=0x7f1450069fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:48:08.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.890+0000 7f1457b71700 1 -- 172.21.15.169:0/1858217105 shutdown_connections 2024-03-28T00:48:08.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:08.890+0000 7f1457b71700 1 -- 172.21.15.169:0/1858217105 wait complete. 2024-03-28T00:48:09.440 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:09 smithi169 ceph-mon[32274]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:09.440 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:09 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:09.467 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-28T00:48:09.467 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-28T00:48:09.699 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:48:10.471 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:10 smithi169 ceph-mon[32274]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:48:11.091 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:11 smithi169 ceph-mon[32274]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:11.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.116+0000 7f648d0cd700 1 -- 172.21.15.169:0/2977448901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6488101db0 msgr2=0x7f6488102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:11.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.116+0000 7f648d0cd700 1 --2- 172.21.15.169:0/2977448901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6488101db0 0x7f6488102190 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f647c009910 tx=0x7f647c009c20 comp rx=0 tx=0).stop 2024-03-28T00:48:11.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.116+0000 7f648d0cd700 1 -- 172.21.15.169:0/2977448901 shutdown_connections 2024-03-28T00:48:11.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.116+0000 7f648d0cd700 1 --2- 172.21.15.169:0/2977448901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6488101db0 0x7f6488102190 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:11.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.116+0000 7f648d0cd700 1 -- 172.21.15.169:0/2977448901 >> 172.21.15.169:0/2977448901 conn(0x7f64880fd660 msgr2=0x7f64880ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:48:11.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.116+0000 7f648d0cd700 1 -- 172.21.15.169:0/2977448901 shutdown_connections 2024-03-28T00:48:11.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.116+0000 7f648d0cd700 1 -- 172.21.15.169:0/2977448901 wait complete. 2024-03-28T00:48:11.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.117+0000 7f648d0cd700 1 Processor -- start 2024-03-28T00:48:11.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.117+0000 7f648d0cd700 1 -- start start 2024-03-28T00:48:11.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.117+0000 7f648d0cd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6488101db0 0x7f6488197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:11.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.117+0000 7f648d0cd700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648810c180 con 0x7f6488101db0 2024-03-28T00:48:11.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.117+0000 7f6486d9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6488101db0 0x7f6488197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:11.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.117+0000 7f6486d9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6488101db0 0x7f6488197720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34740/0 (socket says 172.21.15.169:34740) 2024-03-28T00:48:11.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.117+0000 7f6486d9d700 1 -- 172.21.15.169:0/3967892764 learned_addr learned my addr 172.21.15.169:0/3967892764 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:48:11.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.118+0000 7f6486d9d700 1 -- 172.21.15.169:0/3967892764 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6488197c60 con 0x7f6488101db0 2024-03-28T00:48:11.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.118+0000 7f6486d9d700 1 --2- 172.21.15.169:0/3967892764 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6488101db0 0x7f6488197720 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f647c005850 tx=0x7f647c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:48:11.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.118+0000 7f646ffff700 1 -- 172.21.15.169:0/3967892764 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f647c026070 con 0x7f6488101db0 2024-03-28T00:48:11.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.118+0000 7f646ffff700 1 -- 172.21.15.169:0/3967892764 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f647c00bbd0 con 0x7f6488101db0 2024-03-28T00:48:11.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.118+0000 7f648d0cd700 1 -- 172.21.15.169:0/3967892764 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6488197f50 con 0x7f6488101db0 2024-03-28T00:48:11.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.118+0000 7f646ffff700 1 -- 172.21.15.169:0/3967892764 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f647c003970 con 0x7f6488101db0 2024-03-28T00:48:11.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.118+0000 7f648d0cd700 1 -- 172.21.15.169:0/3967892764 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f648819bff0 con 0x7f6488101db0 2024-03-28T00:48:11.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.119+0000 7f646ffff700 1 -- 172.21.15.169:0/3967892764 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f647c029070 con 0x7f6488101db0 2024-03-28T00:48:11.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.119+0000 7f648d0cd700 1 -- 172.21.15.169:0/3967892764 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6488103290 con 0x7f6488101db0 2024-03-28T00:48:11.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.119+0000 7f646ffff700 1 --2- 172.21.15.169:0/3967892764 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6470038fc0 0x7f647003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:11.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.119+0000 7f646ffff700 1 -- 172.21.15.169:0/3967892764 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f647c01e070 con 0x7f6488101db0 2024-03-28T00:48:11.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.119+0000 7f648659c700 1 --2- 172.21.15.169:0/3967892764 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6470038fc0 0x7f647003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:11.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.119+0000 7f648659c700 1 --2- 172.21.15.169:0/3967892764 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6470038fc0 0x7f647003b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6478006fd0 tx=0x7f6478006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:48:11.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.124+0000 7f646ffff700 1 -- 172.21.15.169:0/3967892764 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f647c052020 con 0x7f6488101db0 2024-03-28T00:48:11.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:11.323+0000 7f648d0cd700 1 -- 172.21.15.169:0/3967892764 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f6488198a80 con 0x7f6488101db0 2024-03-28T00:48:12.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:12.076+0000 7f646ffff700 1 -- 172.21.15.169:0/3967892764 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f647c030020 con 0x7f6488101db0 2024-03-28T00:48:12.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:12.079+0000 7f648d0cd700 1 -- 172.21.15.169:0/3967892764 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6470038fc0 msgr2=0x7f647003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:12.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:12.079+0000 7f648d0cd700 1 --2- 172.21.15.169:0/3967892764 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6470038fc0 0x7f647003b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6478006fd0 tx=0x7f6478006e40 comp rx=0 tx=0).stop 2024-03-28T00:48:12.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:12.079+0000 7f648d0cd700 1 -- 172.21.15.169:0/3967892764 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6488101db0 msgr2=0x7f6488197720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:12.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:12.079+0000 7f648d0cd700 1 --2- 172.21.15.169:0/3967892764 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6488101db0 0x7f6488197720 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f647c005850 tx=0x7f647c005030 comp rx=0 tx=0).stop 2024-03-28T00:48:12.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:12.079+0000 7f648d0cd700 1 -- 172.21.15.169:0/3967892764 shutdown_connections 2024-03-28T00:48:12.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:12.079+0000 7f648d0cd700 1 --2- 172.21.15.169:0/3967892764 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6470038fc0 0x7f647003b480 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:12.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:12.079+0000 7f648d0cd700 1 --2- 172.21.15.169:0/3967892764 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6488101db0 0x7f6488197720 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:12.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:12.079+0000 7f648d0cd700 1 -- 172.21.15.169:0/3967892764 >> 172.21.15.169:0/3967892764 conn(0x7f64880fd660 msgr2=0x7f6488104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:48:12.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:12.080+0000 7f648d0cd700 1 -- 172.21.15.169:0/3967892764 shutdown_connections 2024-03-28T00:48:12.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:12.080+0000 7f648d0cd700 1 -- 172.21.15.169:0/3967892764 wait complete. 2024-03-28T00:48:12.084 INFO:teuthology.orchestra.run.smithi169.stderr:adjusted tunables profile to default 2024-03-28T00:48:12.461 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:12 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3967892764' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T00:48:12.461 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:12 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:12.461 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:12 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:12.461 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:12 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:12.461 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:12 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:12.461 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:12 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:48:12.461 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:12 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:12.461 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:12 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:48:12.764 INFO:tasks.cephadm:Adding mon.a on smithi169 2024-03-28T00:48:12.764 INFO:tasks.cephadm:Adding mon.b on smithi174 2024-03-28T00:48:12.764 INFO:tasks.cephadm:Adding mon.c on smithi177 2024-03-28T00:48:12.764 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi169:172.21.15.169=a;smithi174:172.21.15.174=b;smithi177:172.21.15.177=c' 2024-03-28T00:48:12.997 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T00:48:13.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:13 smithi169 ceph-mon[32274]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:13.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:13 smithi169 ceph-mon[32274]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-28T00:48:13.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:13 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3967892764' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T00:48:13.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:13 smithi169 ceph-mon[32274]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T00:48:13.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:13 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:13.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:13 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:13.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:13 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:14.286 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.280+0000 7fc846b60700 1 -- 172.21.15.177:0/897919401 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc840101dd0 msgr2=0x7fc8401021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:14.286 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.280+0000 7fc846b60700 1 --2- 172.21.15.177:0/897919401 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc840101dd0 0x7fc8401021b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc83c00b0a0 tx=0x7fc83c00b3b0 comp rx=0 tx=0).stop 2024-03-28T00:48:14.286 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.281+0000 7fc846b60700 1 -- 172.21.15.177:0/897919401 shutdown_connections 2024-03-28T00:48:14.286 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.281+0000 7fc846b60700 1 --2- 172.21.15.177:0/897919401 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc840101dd0 0x7fc8401021b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:14.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.281+0000 7fc846b60700 1 -- 172.21.15.177:0/897919401 >> 172.21.15.177:0/897919401 conn(0x7fc8400fd660 msgr2=0x7fc8400ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:48:14.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.281+0000 7fc846b60700 1 -- 172.21.15.177:0/897919401 shutdown_connections 2024-03-28T00:48:14.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.281+0000 7fc846b60700 1 -- 172.21.15.177:0/897919401 wait complete. 2024-03-28T00:48:14.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.281+0000 7fc846b60700 1 Processor -- start 2024-03-28T00:48:14.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.281+0000 7fc846b60700 1 -- start start 2024-03-28T00:48:14.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.281+0000 7fc846b60700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc840101dd0 0x7fc840197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:14.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.281+0000 7fc846b60700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84010c1a0 con 0x7fc840101dd0 2024-03-28T00:48:14.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.282+0000 7fc8448fc700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc840101dd0 0x7fc840197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:14.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.282+0000 7fc8448fc700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc840101dd0 0x7fc840197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.177:44164/0 (socket says 172.21.15.177:44164) 2024-03-28T00:48:14.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.282+0000 7fc8448fc700 1 -- 172.21.15.177:0/3487567782 learned_addr learned my addr 172.21.15.177:0/3487567782 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-28T00:48:14.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.282+0000 7fc8448fc700 1 -- 172.21.15.177:0/3487567782 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc840197b50 con 0x7fc840101dd0 2024-03-28T00:48:14.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.282+0000 7fc8448fc700 1 --2- 172.21.15.177:0/3487567782 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc840101dd0 0x7fc840197610 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc83c015040 tx=0x7fc83c003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:48:14.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.282+0000 7fc835ffb700 1 -- 172.21.15.177:0/3487567782 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc83c017040 con 0x7fc840101dd0 2024-03-28T00:48:14.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.282+0000 7fc835ffb700 1 -- 172.21.15.177:0/3487567782 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc83c022070 con 0x7fc840101dd0 2024-03-28T00:48:14.288 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.282+0000 7fc846b60700 1 -- 172.21.15.177:0/3487567782 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc840197de0 con 0x7fc840101dd0 2024-03-28T00:48:14.288 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.282+0000 7fc835ffb700 1 -- 172.21.15.177:0/3487567782 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc83c01d5d0 con 0x7fc840101dd0 2024-03-28T00:48:14.288 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.282+0000 7fc846b60700 1 -- 172.21.15.177:0/3487567782 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc84019bee0 con 0x7fc840101dd0 2024-03-28T00:48:14.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.283+0000 7fc846b60700 1 -- 172.21.15.177:0/3487567782 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8401032b0 con 0x7fc840101dd0 2024-03-28T00:48:14.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.283+0000 7fc835ffb700 1 -- 172.21.15.177:0/3487567782 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fc83c004000 con 0x7fc840101dd0 2024-03-28T00:48:14.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.283+0000 7fc835ffb700 1 --2- 172.21.15.177:0/3487567782 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc828039020 0x7fc82803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:14.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.283+0000 7fc835ffb700 1 -- 172.21.15.177:0/3487567782 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc83c00f050 con 0x7fc840101dd0 2024-03-28T00:48:14.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.284+0000 7fc837fff700 1 --2- 172.21.15.177:0/3487567782 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc828039020 0x7fc82803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:14.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.284+0000 7fc837fff700 1 --2- 172.21.15.177:0/3487567782 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc828039020 0x7fc82803b4e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc830006fd0 tx=0x7fc830006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:48:14.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.289+0000 7fc835ffb700 1 -- 172.21.15.177:0/3487567782 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc83c007e20 con 0x7fc840101dd0 2024-03-28T00:48:14.454 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:14 smithi169 ceph-mon[32274]: Updating smithi177:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T00:48:14.454 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:14 smithi169 ceph-mon[32274]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-28T00:48:14.454 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:14 smithi169 ceph-mon[32274]: Updating smithi177:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T00:48:14.499 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.495+0000 7fc846b60700 1 -- 172.21.15.177:0/3487567782 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi169:172.21.15.169=a;smithi174:172.21.15.174=b;smithi177:172.21.15.177=c", "target": ["mon-mgr", ""]}) v1 -- 0x7fc840198970 con 0x7fc828039020 2024-03-28T00:48:14.506 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.503+0000 7fc835ffb700 1 -- 172.21.15.177:0/3487567782 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc840198970 con 0x7fc828039020 2024-03-28T00:48:14.506 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled mon update... 2024-03-28T00:48:14.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.504+0000 7fc846b60700 1 -- 172.21.15.177:0/3487567782 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc828039020 msgr2=0x7fc82803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:14.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.504+0000 7fc846b60700 1 --2- 172.21.15.177:0/3487567782 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc828039020 0x7fc82803b4e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc830006fd0 tx=0x7fc830006e40 comp rx=0 tx=0).stop 2024-03-28T00:48:14.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.504+0000 7fc846b60700 1 -- 172.21.15.177:0/3487567782 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc840101dd0 msgr2=0x7fc840197610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:14.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.504+0000 7fc846b60700 1 --2- 172.21.15.177:0/3487567782 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc840101dd0 0x7fc840197610 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc83c015040 tx=0x7fc83c003d50 comp rx=0 tx=0).stop 2024-03-28T00:48:14.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.504+0000 7fc846b60700 1 -- 172.21.15.177:0/3487567782 shutdown_connections 2024-03-28T00:48:14.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.504+0000 7fc846b60700 1 --2- 172.21.15.177:0/3487567782 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc828039020 0x7fc82803b4e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:14.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.504+0000 7fc846b60700 1 --2- 172.21.15.177:0/3487567782 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc840101dd0 0x7fc840197610 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:14.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.504+0000 7fc846b60700 1 -- 172.21.15.177:0/3487567782 >> 172.21.15.177:0/3487567782 conn(0x7fc8400fd660 msgr2=0x7fc8401037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:48:14.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.504+0000 7fc846b60700 1 -- 172.21.15.177:0/3487567782 shutdown_connections 2024-03-28T00:48:14.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:14.504+0000 7fc846b60700 1 -- 172.21.15.177:0/3487567782 wait complete. 2024-03-28T00:48:15.005 DEBUG:teuthology.orchestra.run.smithi174:mon.b> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mon.b.service 2024-03-28T00:48:15.008 DEBUG:teuthology.orchestra.run.smithi177:mon.c> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mon.c.service 2024-03-28T00:48:15.011 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T00:48:15.011 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T00:48:15.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:15 smithi169 ceph-mon[32274]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:15.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:15 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:15.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:15 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:48:15.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:15 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:15.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:15 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:48:15.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:15 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:15.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:15 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T00:48:15.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:15 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:15.832 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T00:48:16.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:16 smithi169 ceph-mon[32274]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi169:172.21.15.169=a;smithi174:172.21.15.174=b;smithi177:172.21.15.177=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:48:16.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:16 smithi169 ceph-mon[32274]: Saving service mon spec with placement smithi169:172.21.15.169=a;smithi174:172.21.15.174=b;smithi177:172.21.15.177=c;count:3 2024-03-28T00:48:16.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:16 smithi169 ceph-mon[32274]: Deploying daemon mon.c on smithi177 2024-03-28T00:48:17.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:17 smithi169 ceph-mon[32274]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:18.703 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.697+0000 7fa42e83b700 1 -- 172.21.15.177:0/3228612434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa428101dd0 msgr2=0x7fa4281021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:18.703 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.697+0000 7fa42e83b700 1 --2- 172.21.15.177:0/3228612434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa428101dd0 0x7fa4281021b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa418009910 tx=0x7fa418009c20 comp rx=0 tx=0).stop 2024-03-28T00:48:18.703 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.697+0000 7fa42e83b700 1 -- 172.21.15.177:0/3228612434 shutdown_connections 2024-03-28T00:48:18.703 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.697+0000 7fa42e83b700 1 --2- 172.21.15.177:0/3228612434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa428101dd0 0x7fa4281021b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:18.703 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.697+0000 7fa42e83b700 1 -- 172.21.15.177:0/3228612434 >> 172.21.15.177:0/3228612434 conn(0x7fa4280fd660 msgr2=0x7fa4280ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:48:18.704 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.697+0000 7fa42e83b700 1 -- 172.21.15.177:0/3228612434 shutdown_connections 2024-03-28T00:48:18.704 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.697+0000 7fa42e83b700 1 -- 172.21.15.177:0/3228612434 wait complete. 2024-03-28T00:48:18.704 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.698+0000 7fa42e83b700 1 Processor -- start 2024-03-28T00:48:18.704 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.698+0000 7fa42e83b700 1 -- start start 2024-03-28T00:48:18.704 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.698+0000 7fa42e83b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa428101dd0 0x7fa428074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:18.704 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.698+0000 7fa42e83b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42810c1a0 con 0x7fa428101dd0 2024-03-28T00:48:18.704 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.699+0000 7fa427fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa428101dd0 0x7fa428074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:18.704 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.699+0000 7fa427fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa428101dd0 0x7fa428074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.177:44174/0 (socket says 172.21.15.177:44174) 2024-03-28T00:48:18.704 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.699+0000 7fa427fff700 1 -- 172.21.15.177:0/874860942 learned_addr learned my addr 172.21.15.177:0/874860942 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-28T00:48:18.704 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.699+0000 7fa427fff700 1 -- 172.21.15.177:0/874860942 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa428076380 con 0x7fa428101dd0 2024-03-28T00:48:18.704 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.699+0000 7fa427fff700 1 --2- 172.21.15.177:0/874860942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa428101dd0 0x7fa428074490 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa418005850 tx=0x7fa418005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:48:18.704 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.699+0000 7fa4257fa700 1 -- 172.21.15.177:0/874860942 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa418026070 con 0x7fa428101dd0 2024-03-28T00:48:18.704 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.699+0000 7fa4257fa700 1 -- 172.21.15.177:0/874860942 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa41800bc30 con 0x7fa428101dd0 2024-03-28T00:48:18.704 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.699+0000 7fa42e83b700 1 -- 172.21.15.177:0/874860942 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa428074a30 con 0x7fa428101dd0 2024-03-28T00:48:18.706 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.699+0000 7fa4257fa700 1 -- 172.21.15.177:0/874860942 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa418003a20 con 0x7fa428101dd0 2024-03-28T00:48:18.706 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.699+0000 7fa42e83b700 1 -- 172.21.15.177:0/874860942 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa428074ef0 con 0x7fa428101dd0 2024-03-28T00:48:18.706 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.700+0000 7fa4257fa700 1 -- 172.21.15.177:0/874860942 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fa418029070 con 0x7fa428101dd0 2024-03-28T00:48:18.706 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.700+0000 7fa42e83b700 1 -- 172.21.15.177:0/874860942 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4281032b0 con 0x7fa428101dd0 2024-03-28T00:48:18.706 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.700+0000 7fa4257fa700 1 --2- 172.21.15.177:0/874860942 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa410038fc0 0x7fa41003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:18.707 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.701+0000 7fa4257fa700 1 -- 172.21.15.177:0/874860942 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa41801e070 con 0x7fa428101dd0 2024-03-28T00:48:18.707 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.701+0000 7fa4277fe700 1 --2- 172.21.15.177:0/874860942 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa410038fc0 0x7fa41003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:18.707 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.701+0000 7fa4277fe700 1 --2- 172.21.15.177:0/874860942 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa410038fc0 0x7fa41003b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa41c006fd0 tx=0x7fa41c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:48:18.710 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:18.706+0000 7fa4257fa700 1 -- 172.21.15.177:0/874860942 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa418013070 con 0x7fa428101dd0 2024-03-28T00:48:18.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:18 smithi177 systemd[1]: Starting Ceph mon.c for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T00:48:19.005 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:19.001+0000 7fa42e83b700 1 -- 172.21.15.177:0/874860942 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4280759c0 con 0x7fa428101dd0 2024-03-28T00:48:19.005 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:19.002+0000 7fa4257fa700 1 -- 172.21.15.177:0/874860942 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+749 (secure 0 0 0) 0x7fa418030020 con 0x7fa428101dd0 2024-03-28T00:48:19.006 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-28T00:48:19.006 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","modified":"2024-03-28T00:46:37.887122Z","created":"2024-03-28T00:46:37.887122Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-28T00:48:19.008 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:19.004+0000 7fa42e83b700 1 -- 172.21.15.177:0/874860942 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa410038fc0 msgr2=0x7fa41003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:19.008 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:19.004+0000 7fa42e83b700 1 --2- 172.21.15.177:0/874860942 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa410038fc0 0x7fa41003b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa41c006fd0 tx=0x7fa41c006e40 comp rx=0 tx=0).stop 2024-03-28T00:48:19.008 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:19.004+0000 7fa42e83b700 1 -- 172.21.15.177:0/874860942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa428101dd0 msgr2=0x7fa428074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:19.008 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:19.004+0000 7fa42e83b700 1 --2- 172.21.15.177:0/874860942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa428101dd0 0x7fa428074490 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa418005850 tx=0x7fa418005030 comp rx=0 tx=0).stop 2024-03-28T00:48:19.008 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:19.004+0000 7fa42e83b700 1 -- 172.21.15.177:0/874860942 shutdown_connections 2024-03-28T00:48:19.008 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:19.004+0000 7fa42e83b700 1 --2- 172.21.15.177:0/874860942 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa410038fc0 0x7fa41003b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:19.008 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:19.004+0000 7fa42e83b700 1 --2- 172.21.15.177:0/874860942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa428101dd0 0x7fa428074490 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:19.008 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:19.004+0000 7fa42e83b700 1 -- 172.21.15.177:0/874860942 >> 172.21.15.177:0/874860942 conn(0x7fa4280fd660 msgr2=0x7fa4281037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:48:19.008 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:19.004+0000 7fa42e83b700 1 -- 172.21.15.177:0/874860942 shutdown_connections 2024-03-28T00:48:19.008 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:19.004+0000 7fa42e83b700 1 -- 172.21.15.177:0/874860942 wait complete. 2024-03-28T00:48:19.009 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-03-28T00:48:19.288 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:18 smithi177 podman[36171]: 2024-03-28 00:48:18.931880752 +0000 UTC m=+0.101028066 container create e37383dc5f0d4f108e093043ceb3f246ae85b2c7d9d8498a955bc2a4912e0b7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-28T00:48:19.289 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:18 smithi177 podman[36171]: 2024-03-28 00:48:18.849313135 +0000 UTC m=+0.018460455 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T00:48:19.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:19 smithi169 ceph-mon[32274]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:19.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:19 smithi169 ceph-mon[32274]: from='client.? 172.21.15.177:0/874860942' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T00:48:19.918 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 podman[36171]: 2024-03-28 00:48:19.66951411 +0000 UTC m=+0.838661433 container init e37383dc5f0d4f108e093043ceb3f246ae85b2c7d9d8498a955bc2a4912e0b7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-28T00:48:19.918 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 podman[36171]: 2024-03-28 00:48:19.672647818 +0000 UTC m=+0.841795124 container start e37383dc5f0d4f108e093043ceb3f246ae85b2c7d9d8498a955bc2a4912e0b7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-28T00:48:19.918 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: set uid:gid to 167:167 (ceph:ceph) 2024-03-28T00:48:19.919 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-28T00:48:19.919 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: pidfile_write: ignore empty --pid-file 2024-03-28T00:48:19.919 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: load: jerasure load: lrc 2024-03-28T00:48:19.919 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: RocksDB version: 7.9.2 2024-03-28T00:48:19.919 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Git sha 0 2024-03-28T00:48:19.919 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-28T00:48:19.919 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: DB SUMMARY 2024-03-28T00:48:19.919 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: DB Session ID: UIW4UBTEMM4QW9IJIWI5 2024-03-28T00:48:19.919 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: CURRENT file: CURRENT 2024-03-28T00:48:19.920 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: IDENTITY file: IDENTITY 2024-03-28T00:48:19.920 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-28T00:48:19.920 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-28T00:48:19.920 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-28T00:48:19.920 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.error_if_exists: 0 2024-03-28T00:48:19.920 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.create_if_missing: 0 2024-03-28T00:48:19.920 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.paranoid_checks: 1 2024-03-28T00:48:19.920 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-28T00:48:19.920 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-28T00:48:19.920 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-28T00:48:19.920 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.env: 0x56257fdcab20 2024-03-28T00:48:19.920 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.fs: PosixFileSystem 2024-03-28T00:48:19.921 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.info_log: 0x5625812b3920 2024-03-28T00:48:19.921 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_file_opening_threads: 16 2024-03-28T00:48:19.922 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.statistics: (nil) 2024-03-28T00:48:19.923 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.use_fsync: 0 2024-03-28T00:48:19.923 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_log_file_size: 0 2024-03-28T00:48:19.923 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-28T00:48:19.923 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-28T00:48:19.923 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.keep_log_file_num: 1000 2024-03-28T00:48:19.923 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.recycle_log_file_num: 0 2024-03-28T00:48:19.923 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.allow_fallocate: 1 2024-03-28T00:48:19.923 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.allow_mmap_reads: 0 2024-03-28T00:48:19.923 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.allow_mmap_writes: 0 2024-03-28T00:48:19.923 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.use_direct_reads: 0 2024-03-28T00:48:19.923 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-28T00:48:19.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.create_missing_column_families: 0 2024-03-28T00:48:19.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.db_log_dir: 2024-03-28T00:48:19.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.wal_dir: 2024-03-28T00:48:19.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-28T00:48:19.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-28T00:48:19.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-28T00:48:19.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-28T00:48:19.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-28T00:48:19.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-28T00:48:19.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.advise_random_on_open: 1 2024-03-28T00:48:19.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.db_write_buffer_size: 0 2024-03-28T00:48:19.924 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.write_buffer_manager: 0x5625812861e0 2024-03-28T00:48:19.925 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-28T00:48:19.925 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-28T00:48:19.925 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-28T00:48:19.925 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.rate_limiter: (nil) 2024-03-28T00:48:19.925 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-28T00:48:19.925 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.wal_recovery_mode: 2 2024-03-28T00:48:19.925 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.enable_thread_tracking: 0 2024-03-28T00:48:19.925 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.enable_pipelined_write: 0 2024-03-28T00:48:19.925 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.unordered_write: 0 2024-03-28T00:48:19.925 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-28T00:48:19.925 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-28T00:48:19.926 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-28T00:48:19.926 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-28T00:48:19.926 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.row_cache: None 2024-03-28T00:48:19.926 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.wal_filter: None 2024-03-28T00:48:19.926 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-28T00:48:19.926 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.allow_ingest_behind: 0 2024-03-28T00:48:19.926 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.two_write_queues: 0 2024-03-28T00:48:19.926 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.manual_wal_flush: 0 2024-03-28T00:48:19.926 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.wal_compression: 0 2024-03-28T00:48:19.926 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.atomic_flush: 0 2024-03-28T00:48:19.926 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-28T00:48:19.927 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-28T00:48:19.927 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-28T00:48:19.927 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.log_readahead_size: 0 2024-03-28T00:48:19.927 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-28T00:48:19.927 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.best_efforts_recovery: 0 2024-03-28T00:48:19.927 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-28T00:48:19.927 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-28T00:48:19.927 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.allow_data_in_errors: 0 2024-03-28T00:48:19.927 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.db_host_id: __hostname__ 2024-03-28T00:48:19.927 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-28T00:48:19.927 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_background_jobs: 2 2024-03-28T00:48:19.927 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_background_compactions: -1 2024-03-28T00:48:19.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_subcompactions: 1 2024-03-28T00:48:19.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-28T00:48:19.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-28T00:48:19.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-28T00:48:19.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_total_wal_size: 0 2024-03-28T00:48:19.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-28T00:48:19.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-28T00:48:19.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-28T00:48:19.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-28T00:48:19.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_open_files: -1 2024-03-28T00:48:19.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.bytes_per_sync: 0 2024-03-28T00:48:19.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-28T00:48:19.928 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-28T00:48:19.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compaction_readahead_size: 0 2024-03-28T00:48:19.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_background_flushes: -1 2024-03-28T00:48:19.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Compression algorithms supported: 2024-03-28T00:48:19.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-28T00:48:19.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: kZSTD supported: 0 2024-03-28T00:48:19.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: kXpressCompression supported: 0 2024-03-28T00:48:19.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: kLZ4HCCompression supported: 1 2024-03-28T00:48:19.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: kZlibCompression supported: 1 2024-03-28T00:48:19.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: kSnappyCompression supported: 1 2024-03-28T00:48:19.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: kLZ4Compression supported: 1 2024-03-28T00:48:19.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: kBZip2Compression supported: 0 2024-03-28T00:48:19.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-28T00:48:19.929 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-28T00:48:19.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-28T00:48:19.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-28T00:48:19.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-28T00:48:19.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.merge_operator: 2024-03-28T00:48:19.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compaction_filter: None 2024-03-28T00:48:19.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compaction_filter_factory: None 2024-03-28T00:48:19.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.sst_partitioner_factory: None 2024-03-28T00:48:19.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-28T00:48:19.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-28T00:48:19.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5625812b3a20) 2024-03-28T00:48:19.930 INFO:journalctl@ceph.mon.c.smithi177.stdout: cache_index_and_filter_blocks: 1 2024-03-28T00:48:19.931 INFO:journalctl@ceph.mon.c.smithi177.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-28T00:48:19.931 INFO:journalctl@ceph.mon.c.smithi177.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-28T00:48:19.931 INFO:journalctl@ceph.mon.c.smithi177.stdout: pin_top_level_index_and_filter: 1 2024-03-28T00:48:19.931 INFO:journalctl@ceph.mon.c.smithi177.stdout: index_type: 0 2024-03-28T00:48:19.931 INFO:journalctl@ceph.mon.c.smithi177.stdout: data_block_index_type: 0 2024-03-28T00:48:19.931 INFO:journalctl@ceph.mon.c.smithi177.stdout: index_shortening: 1 2024-03-28T00:48:19.931 INFO:journalctl@ceph.mon.c.smithi177.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-28T00:48:19.931 INFO:journalctl@ceph.mon.c.smithi177.stdout: checksum: 4 2024-03-28T00:48:19.931 INFO:journalctl@ceph.mon.c.smithi177.stdout: no_block_cache: 0 2024-03-28T00:48:19.931 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_cache: 0x5625812b9090 2024-03-28T00:48:19.931 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_cache_name: BinnedLRUCache 2024-03-28T00:48:19.931 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_cache_options: 2024-03-28T00:48:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout: capacity : 536870912 2024-03-28T00:48:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout: num_shard_bits : 4 2024-03-28T00:48:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout: strict_capacity_limit : 0 2024-03-28T00:48:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout: high_pri_pool_ratio: 0.000 2024-03-28T00:48:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_cache_compressed: (nil) 2024-03-28T00:48:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout: persistent_cache: (nil) 2024-03-28T00:48:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_size: 4096 2024-03-28T00:48:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_size_deviation: 10 2024-03-28T00:48:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_restart_interval: 16 2024-03-28T00:48:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout: index_block_restart_interval: 1 2024-03-28T00:48:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout: metadata_block_size: 4096 2024-03-28T00:48:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout: partition_filters: 0 2024-03-28T00:48:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout: use_delta_encoding: 1 2024-03-28T00:48:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout: filter_policy: bloomfilter 2024-03-28T00:48:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout: whole_key_filtering: 1 2024-03-28T00:48:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout: verify_compression: 0 2024-03-28T00:48:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout: read_amp_bytes_per_bit: 0 2024-03-28T00:48:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout: format_version: 5 2024-03-28T00:48:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout: enable_index_compression: 1 2024-03-28T00:48:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_align: 0 2024-03-28T00:48:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout: max_auto_readahead_size: 262144 2024-03-28T00:48:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout: prepopulate_block_cache: 0 2024-03-28T00:48:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout: initial_auto_readahead_size: 8192 2024-03-28T00:48:19.934 INFO:journalctl@ceph.mon.c.smithi177.stdout: num_file_reads_for_auto_readahead: 2 2024-03-28T00:48:19.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.write_buffer_size: 33554432 2024-03-28T00:48:19.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_write_buffer_number: 2 2024-03-28T00:48:19.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compression: NoCompression 2024-03-28T00:48:19.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.bottommost_compression: Disabled 2024-03-28T00:48:19.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.prefix_extractor: nullptr 2024-03-28T00:48:19.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-28T00:48:19.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.num_levels: 7 2024-03-28T00:48:19.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-28T00:48:19.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-28T00:48:19.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-28T00:48:19.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-28T00:48:19.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-28T00:48:19.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-28T00:48:19.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-28T00:48:19.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-28T00:48:19.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-28T00:48:19.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-28T00:48:19.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-28T00:48:19.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-28T00:48:19.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-28T00:48:19.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compression_opts.level: 32767 2024-03-28T00:48:19.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compression_opts.strategy: 0 2024-03-28T00:48:19.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-28T00:48:19.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-28T00:48:19.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-28T00:48:19.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-28T00:48:19.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compression_opts.enabled: false 2024-03-28T00:48:19.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-28T00:48:19.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-28T00:48:19.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-28T00:48:19.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-28T00:48:19.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.target_file_size_base: 67108864 2024-03-28T00:48:19.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-28T00:48:19.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-28T00:48:19.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-28T00:48:19.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-28T00:48:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-28T00:48:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-28T00:48:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-28T00:48:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-28T00:48:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-28T00:48:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-28T00:48:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-28T00:48:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-28T00:48:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-28T00:48:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-28T00:48:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.arena_block_size: 1048576 2024-03-28T00:48:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-28T00:48:19.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-28T00:48:19.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.disable_auto_compactions: 0 2024-03-28T00:48:19.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-28T00:48:19.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-28T00:48:19.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-28T00:48:19.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-28T00:48:19.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-28T00:48:19.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-28T00:48:19.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-28T00:48:19.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-28T00:48:19.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-28T00:48:19.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-28T00:48:19.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.table_properties_collectors: 2024-03-28T00:48:19.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.inplace_update_support: 0 2024-03-28T00:48:19.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-28T00:48:19.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-28T00:48:19.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-28T00:48:19.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-28T00:48:19.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.bloom_locality: 0 2024-03-28T00:48:19.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.max_successive_merges: 0 2024-03-28T00:48:19.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-28T00:48:19.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.paranoid_file_checks: 0 2024-03-28T00:48:19.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.force_consistency_checks: 1 2024-03-28T00:48:19.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.report_bg_io_stats: 0 2024-03-28T00:48:19.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.ttl: 2592000 2024-03-28T00:48:19.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-28T00:48:19.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-28T00:48:19.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-28T00:48:19.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.enable_blob_files: false 2024-03-28T00:48:19.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.min_blob_size: 0 2024-03-28T00:48:19.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.blob_file_size: 268435456 2024-03-28T00:48:19.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-28T00:48:19.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-28T00:48:19.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-28T00:48:19.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-28T00:48:19.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-28T00:48:19.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.blob_file_starting_level: 0 2024-03-28T00:48:19.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-28T00:48:19.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-28T00:48:19.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-28T00:48:19.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: df9b68ae-b4d3-44ba-b2b1-045ef787a8f0 2024-03-28T00:48:19.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711586899702283, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-28T00:48:19.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-28T00:48:19.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711586899702725, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711586899, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "df9b68ae-b4d3-44ba-b2b1-045ef787a8f0", "db_session_id": "UIW4UBTEMM4QW9IJIWI5", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-28T00:48:19.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711586899702804, "job": 1, "event": "recovery_finished"} 2024-03-28T00:48:19.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-28T00:48:19.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-28T00:48:19.941 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5625813ac000 2024-03-28T00:48:19.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: DB pointer 0x562581396000 2024-03-28T00:48:19.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-28T00:48:19.942 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-28T00:48:19.942 INFO:journalctl@ceph.mon.c.smithi177.stdout: ** DB Stats ** 2024-03-28T00:48:19.942 INFO:journalctl@ceph.mon.c.smithi177.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T00:48:19.942 INFO:journalctl@ceph.mon.c.smithi177.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-28T00:48:19.942 INFO:journalctl@ceph.mon.c.smithi177.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T00:48:19.942 INFO:journalctl@ceph.mon.c.smithi177.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T00:48:19.942 INFO:journalctl@ceph.mon.c.smithi177.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-28T00:48:19.942 INFO:journalctl@ceph.mon.c.smithi177.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T00:48:19.942 INFO:journalctl@ceph.mon.c.smithi177.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T00:48:19.943 INFO:journalctl@ceph.mon.c.smithi177.stdout: 2024-03-28T00:48:19.943 INFO:journalctl@ceph.mon.c.smithi177.stdout: ** Compaction Stats [default] ** 2024-03-28T00:48:19.943 INFO:journalctl@ceph.mon.c.smithi177.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T00:48:19.943 INFO:journalctl@ceph.mon.c.smithi177.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-28T00:48:19.943 INFO:journalctl@ceph.mon.c.smithi177.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T00:48:19.943 INFO:journalctl@ceph.mon.c.smithi177.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T00:48:19.943 INFO:journalctl@ceph.mon.c.smithi177.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T00:48:19.943 INFO:journalctl@ceph.mon.c.smithi177.stdout: 2024-03-28T00:48:19.943 INFO:journalctl@ceph.mon.c.smithi177.stdout: ** Compaction Stats [default] ** 2024-03-28T00:48:19.943 INFO:journalctl@ceph.mon.c.smithi177.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T00:48:19.943 INFO:journalctl@ceph.mon.c.smithi177.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-28T00:48:19.943 INFO:journalctl@ceph.mon.c.smithi177.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T00:48:19.944 INFO:journalctl@ceph.mon.c.smithi177.stdout: 2024-03-28T00:48:19.944 INFO:journalctl@ceph.mon.c.smithi177.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-28T00:48:19.944 INFO:journalctl@ceph.mon.c.smithi177.stdout: 2024-03-28T00:48:19.944 INFO:journalctl@ceph.mon.c.smithi177.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T00:48:19.944 INFO:journalctl@ceph.mon.c.smithi177.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-28T00:48:19.944 INFO:journalctl@ceph.mon.c.smithi177.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-28T00:48:19.944 INFO:journalctl@ceph.mon.c.smithi177.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-28T00:48:19.944 INFO:journalctl@ceph.mon.c.smithi177.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-28T00:48:19.944 INFO:journalctl@ceph.mon.c.smithi177.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-28T00:48:19.944 INFO:journalctl@ceph.mon.c.smithi177.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T00:48:19.944 INFO:journalctl@ceph.mon.c.smithi177.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T00:48:19.944 INFO:journalctl@ceph.mon.c.smithi177.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-28T00:48:19.945 INFO:journalctl@ceph.mon.c.smithi177.stdout: Block cache BinnedLRUCache@0x5625812b9090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-28T00:48:19.945 INFO:journalctl@ceph.mon.c.smithi177.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-28T00:48:19.945 INFO:journalctl@ceph.mon.c.smithi177.stdout: 2024-03-28T00:48:19.945 INFO:journalctl@ceph.mon.c.smithi177.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-28T00:48:19.945 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-28T00:48:19.945 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: using public_addr v2:172.21.15.177:0/0 -> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-03-28T00:48:19.945 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: starting mon.c rank -1 at public addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] at bind addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T00:48:19.945 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: mon.c@-1(???) e0 preinit fsid 833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T00:48:19.945 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: mon.c@-1(synchronizing).mds e1 new map 2024-03-28T00:48:19.945 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-28T00:48:19.945 INFO:journalctl@ceph.mon.c.smithi177.stdout: e1 2024-03-28T00:48:19.945 INFO:journalctl@ceph.mon.c.smithi177.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T00:48:19.945 INFO:journalctl@ceph.mon.c.smithi177.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T00:48:19.946 INFO:journalctl@ceph.mon.c.smithi177.stdout: legacy client fscid: -1 2024-03-28T00:48:19.946 INFO:journalctl@ceph.mon.c.smithi177.stdout: 2024-03-28T00:48:19.946 INFO:journalctl@ceph.mon.c.smithi177.stdout: No filesystems configured 2024-03-28T00:48:19.946 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-28T00:48:19.946 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-28T00:48:19.946 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-28T00:48:19.946 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-28T00:48:19.946 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-28T00:48:19.946 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-28T00:48:19.946 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T00:48:19.946 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T00:48:19.946 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T00:48:19.947 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T00:48:19.947 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:19.947 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:48:19.947 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:19.947 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:19.947 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:19.947 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:19.947 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:48:19.947 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:19.947 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:48:19.947 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-28T00:48:19.947 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: Updating smithi174:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T00:48:19.948 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: Deploying cephadm binary to smithi177 2024-03-28T00:48:19.948 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-03-28T00:48:19.948 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: Updating smithi174:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T00:48:19.948 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:19.948 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:19.948 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:19.948 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:19.948 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:19.948 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:19.948 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:48:19.948 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:19.948 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: Added host smithi177 2024-03-28T00:48:19.949 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:19.949 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:19.949 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:19.949 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:48:19.949 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:19.949 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3967892764' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T00:48:19.949 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:19.949 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:19.949 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:19.949 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:19.949 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:48:19.949 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:19.949 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:48:19.950 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:19.950 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-28T00:48:19.950 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3967892764' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T00:48:19.950 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T00:48:19.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:19.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:19.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:19.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: Updating smithi177:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T00:48:19.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-28T00:48:19.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: Updating smithi177:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T00:48:19.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:19.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:19.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:48:19.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:19.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:48:19.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:19.952 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T00:48:19.952 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:19.952 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi169:172.21.15.169=a;smithi174:172.21.15.174=b;smithi177:172.21.15.177=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:48:19.952 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: Saving service mon spec with placement smithi169:172.21.15.169=a;smithi174:172.21.15.174=b;smithi177:172.21.15.177=c;count:3 2024-03-28T00:48:19.952 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: Deploying daemon mon.c on smithi177 2024-03-28T00:48:19.952 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:19.952 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:19.952 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: from='client.? 172.21.15.177:0/874860942' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T00:48:19.952 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 ceph-mon[36206]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-28T00:48:19.952 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 bash[36171]: e37383dc5f0d4f108e093043ceb3f246ae85b2c7d9d8498a955bc2a4912e0b7a 2024-03-28T00:48:19.952 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:19 smithi177 systemd[1]: Started Ceph mon.c for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T00:48:20.564 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T00:48:20.564 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T00:48:20.799 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T00:48:25.065 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.061+0000 7fdf53d27700 1 -- 172.21.15.177:0/2948481901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf380035c0 msgr2=0x7fdf38005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:25.065 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.061+0000 7fdf53d27700 1 --2- 172.21.15.177:0/2948481901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf380035c0 0x7fdf38005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c100a90 tx=0x7fdf4800feb0 comp rx=0 tx=0).stop 2024-03-28T00:48:25.065 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.061+0000 7fdf53d27700 1 -- 172.21.15.177:0/2948481901 shutdown_connections 2024-03-28T00:48:25.066 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.061+0000 7fdf53d27700 1 --2- 172.21.15.177:0/2948481901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf380035c0 0x7fdf38005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:25.066 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.061+0000 7fdf53d27700 1 -- 172.21.15.177:0/2948481901 >> 172.21.15.177:0/2948481901 conn(0x7fdf4c0fc630 msgr2=0x7fdf4c0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:48:25.066 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.061+0000 7fdf53d27700 1 -- 172.21.15.177:0/2948481901 shutdown_connections 2024-03-28T00:48:25.066 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.061+0000 7fdf53d27700 1 -- 172.21.15.177:0/2948481901 wait complete. 2024-03-28T00:48:25.068 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.062+0000 7fdf53d27700 1 Processor -- start 2024-03-28T00:48:25.068 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.062+0000 7fdf53d27700 1 -- start start 2024-03-28T00:48:25.068 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.062+0000 7fdf53d27700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf380035c0 0x7fdf4c19d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:25.068 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.062+0000 7fdf53d27700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf4c19dad0 0x7fdf4c197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:25.069 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.062+0000 7fdf53d27700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c108bf0 con 0x7fdf380035c0 2024-03-28T00:48:25.069 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.062+0000 7fdf53d27700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c108a70 con 0x7fdf4c19dad0 2024-03-28T00:48:25.069 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.062+0000 7fdf512c2700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf4c19dad0 0x7fdf4c197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:25.069 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.063+0000 7fdf512c2700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf4c19dad0 0x7fdf4c197660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:36858/0 (socket says 172.21.15.177:36858) 2024-03-28T00:48:25.069 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.063+0000 7fdf512c2700 1 -- 172.21.15.177:0/3225717599 learned_addr learned my addr 172.21.15.177:0/3225717599 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-28T00:48:25.069 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.063+0000 7fdf51ac3700 1 --2- 172.21.15.177:0/3225717599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf380035c0 0x7fdf4c19d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:25.069 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.063+0000 7fdf512c2700 1 -- 172.21.15.177:0/3225717599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf4c19dad0 msgr2=0x7fdf4c197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-28T00:48:25.070 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.063+0000 7fdf512c2700 1 -- 172.21.15.177:0/3225717599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf4c19dad0 msgr2=0x7fdf4c197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T00:48:25.070 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.063+0000 7fdf512c2700 1 --2- 172.21.15.177:0/3225717599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf4c19dad0 0x7fdf4c197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T00:48:25.070 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.063+0000 7fdf512c2700 1 --2- 172.21.15.177:0/3225717599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf4c19dad0 0x7fdf4c197660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T00:48:25.070 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.063+0000 7fdf51ac3700 1 -- 172.21.15.177:0/3225717599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf4c19dad0 msgr2=0x7fdf4c197660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T00:48:25.070 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.063+0000 7fdf51ac3700 1 --2- 172.21.15.177:0/3225717599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf4c19dad0 0x7fdf4c197660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:25.070 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.063+0000 7fdf51ac3700 1 -- 172.21.15.177:0/3225717599 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c19e050 con 0x7fdf380035c0 2024-03-28T00:48:25.070 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.063+0000 7fdf51ac3700 1 --2- 172.21.15.177:0/3225717599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf380035c0 0x7fdf4c19d590 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdf480059d0 tx=0x7fdf480047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:48:25.070 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.063+0000 7fdf42ffd700 1 -- 172.21.15.177:0/3225717599 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fdf48013b00 con 0x7fdf380035c0 2024-03-28T00:48:25.070 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.064+0000 7fdf42ffd700 1 -- 172.21.15.177:0/3225717599 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdf48004500 con 0x7fdf380035c0 2024-03-28T00:48:25.070 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.064+0000 7fdf53d27700 1 -- 172.21.15.177:0/3225717599 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf4c197c00 con 0x7fdf380035c0 2024-03-28T00:48:25.070 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.064+0000 7fdf42ffd700 1 -- 172.21.15.177:0/3225717599 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fdf48011470 con 0x7fdf380035c0 2024-03-28T00:48:25.070 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.064+0000 7fdf53d27700 1 -- 172.21.15.177:0/3225717599 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf4c198190 con 0x7fdf380035c0 2024-03-28T00:48:25.070 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.065+0000 7fdf42ffd700 1 -- 172.21.15.177:0/3225717599 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fdf48023070 con 0x7fdf380035c0 2024-03-28T00:48:25.071 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.065+0000 7fdf42ffd700 1 --2- 172.21.15.177:0/3225717599 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdf30038d60 0x7fdf3003b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:25.071 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.065+0000 7fdf42ffd700 1 -- 172.21.15.177:0/3225717599 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdf48020050 con 0x7fdf380035c0 2024-03-28T00:48:25.071 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.065+0000 7fdf53d27700 1 -- 172.21.15.177:0/3225717599 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf4c1022a0 con 0x7fdf380035c0 2024-03-28T00:48:25.072 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.065+0000 7fdf512c2700 1 --2- 172.21.15.177:0/3225717599 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdf30038d60 0x7fdf3003b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:25.072 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.066+0000 7fdf512c2700 1 --2- 172.21.15.177:0/3225717599 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdf30038d60 0x7fdf3003b220 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c009180 tx=0x7fdf3c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:48:25.074 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.070+0000 7fdf42ffd700 1 -- 172.21.15.177:0/3225717599 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdf48031020 con 0x7fdf380035c0 2024-03-28T00:48:25.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T00:48:25.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: mon.a calling monitor election 2024-03-28T00:48:25.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:48:25.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:25.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:48:25.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:48:25.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: mon.c calling monitor election 2024-03-28T00:48:25.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:25.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:48:25.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:48:25.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:25.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:48:25.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T00:48:25.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: monmap e2: 2 mons at {a=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],c=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T00:48:25.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: fsmap 2024-03-28T00:48:25.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T00:48:25.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: mgrmap e13: x(active, since 48s) 2024-03-28T00:48:25.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: overall HEALTH_OK 2024-03-28T00:48:25.169 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:25.169 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:25.169 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:25.169 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T00:48:25.169 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:24 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T00:48:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: mon.a calling monitor election 2024-03-28T00:48:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:48:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:48:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:48:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: mon.c calling monitor election 2024-03-28T00:48:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:48:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:48:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:25.174 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:48:25.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T00:48:25.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: monmap e2: 2 mons at {a=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],c=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T00:48:25.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: fsmap 2024-03-28T00:48:25.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T00:48:25.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: mgrmap e13: x(active, since 48s) 2024-03-28T00:48:25.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: overall HEALTH_OK 2024-03-28T00:48:25.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:25.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:25.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:25.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T00:48:25.175 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:24 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:25.354 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.350+0000 7fdf53d27700 1 -- 172.21.15.177:0/3225717599 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf4c198b90 con 0x7fdf380035c0 2024-03-28T00:48:25.355 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.351+0000 7fdf42ffd700 1 -- 172.21.15.177:0/3225717599 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1019 (secure 0 0 0) 0x7fdf48028030 con 0x7fdf380035c0 2024-03-28T00:48:25.355 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-28T00:48:25.355 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":2,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","modified":"2024-03-28T00:48:19.725538Z","created":"2024-03-28T00:46:37.887122Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-28T00:48:25.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.353+0000 7fdf53d27700 1 -- 172.21.15.177:0/3225717599 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdf30038d60 msgr2=0x7fdf3003b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:25.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.353+0000 7fdf53d27700 1 --2- 172.21.15.177:0/3225717599 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdf30038d60 0x7fdf3003b220 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c009180 tx=0x7fdf3c008040 comp rx=0 tx=0).stop 2024-03-28T00:48:25.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.353+0000 7fdf53d27700 1 -- 172.21.15.177:0/3225717599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf380035c0 msgr2=0x7fdf4c19d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:25.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.353+0000 7fdf53d27700 1 --2- 172.21.15.177:0/3225717599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf380035c0 0x7fdf4c19d590 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdf480059d0 tx=0x7fdf480047a0 comp rx=0 tx=0).stop 2024-03-28T00:48:25.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.353+0000 7fdf53d27700 1 -- 172.21.15.177:0/3225717599 shutdown_connections 2024-03-28T00:48:25.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.353+0000 7fdf53d27700 1 --2- 172.21.15.177:0/3225717599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf380035c0 0x7fdf4c19d590 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:25.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.353+0000 7fdf53d27700 1 --2- 172.21.15.177:0/3225717599 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdf30038d60 0x7fdf3003b220 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:25.357 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.353+0000 7fdf53d27700 1 --2- 172.21.15.177:0/3225717599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf4c19dad0 0x7fdf4c197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:25.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.353+0000 7fdf53d27700 1 -- 172.21.15.177:0/3225717599 >> 172.21.15.177:0/3225717599 conn(0x7fdf4c0fc630 msgr2=0x7fdf4c0ffb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:48:25.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.353+0000 7fdf53d27700 1 -- 172.21.15.177:0/3225717599 shutdown_connections 2024-03-28T00:48:25.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:25.353+0000 7fdf53d27700 1 -- 172.21.15.177:0/3225717599 wait complete. 2024-03-28T00:48:25.358 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 2 2024-03-28T00:48:25.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:25 smithi177 ceph-mon[36206]: Deploying daemon mon.b on smithi174 2024-03-28T00:48:25.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:25 smithi177 ceph-mon[36206]: from='client.? 172.21.15.177:0/3225717599' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T00:48:25.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:25 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:48:26.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:25 smithi169 ceph-mon[32274]: Deploying daemon mon.b on smithi174 2024-03-28T00:48:26.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:25 smithi169 ceph-mon[32274]: from='client.? 172.21.15.177:0/3225717599' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T00:48:26.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:25 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:48:26.961 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T00:48:26.961 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T00:48:27.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:26 smithi177 ceph-mon[36206]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:27.194 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T00:48:27.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:26 smithi169 ceph-mon[32274]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:28.660 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.655+0000 7f26c098b700 1 -- 172.21.15.177:0/1250773532 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26bc101ed0 msgr2=0x7f26bc1022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:28.660 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.655+0000 7f26c098b700 1 --2- 172.21.15.177:0/1250773532 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26bc101ed0 0x7f26bc1022b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f26ac009910 tx=0x7f26ac009c20 comp rx=0 tx=0).stop 2024-03-28T00:48:28.660 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.655+0000 7f26c098b700 1 -- 172.21.15.177:0/1250773532 shutdown_connections 2024-03-28T00:48:28.660 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.655+0000 7f26c098b700 1 --2- 172.21.15.177:0/1250773532 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26bc101ed0 0x7f26bc1022b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:28.660 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.655+0000 7f26c098b700 1 -- 172.21.15.177:0/1250773532 >> 172.21.15.177:0/1250773532 conn(0x7f26bc0fd760 msgr2=0x7f26bc0ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:48:28.660 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.655+0000 7f26c098b700 1 -- 172.21.15.177:0/1250773532 shutdown_connections 2024-03-28T00:48:28.660 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.655+0000 7f26c098b700 1 -- 172.21.15.177:0/1250773532 wait complete. 2024-03-28T00:48:28.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.656+0000 7f26c098b700 1 Processor -- start 2024-03-28T00:48:28.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.656+0000 7f26c098b700 1 -- start start 2024-03-28T00:48:28.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.656+0000 7f26c098b700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26bc101ed0 0x7f26bc19d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:28.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.656+0000 7f26c098b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26bc19db60 0x7f26bc197790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:28.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.656+0000 7f26c098b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26bc10c2a0 con 0x7f26bc19db60 2024-03-28T00:48:28.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.656+0000 7f26c098b700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26bc10c120 con 0x7f26bc101ed0 2024-03-28T00:48:28.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.656+0000 7f26ba59c700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26bc101ed0 0x7f26bc19d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:28.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.656+0000 7f26ba59c700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26bc101ed0 0x7f26bc19d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:36872/0 (socket says 172.21.15.177:36872) 2024-03-28T00:48:28.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.656+0000 7f26ba59c700 1 -- 172.21.15.177:0/3950149728 learned_addr learned my addr 172.21.15.177:0/3950149728 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-28T00:48:28.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.657+0000 7f26ba59c700 1 -- 172.21.15.177:0/3950149728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26bc101ed0 msgr2=0x7f26bc19d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-28T00:48:28.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.657+0000 7f26ba59c700 1 -- 172.21.15.177:0/3950149728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26bc101ed0 msgr2=0x7f26bc19d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T00:48:28.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.657+0000 7f26ba59c700 1 --2- 172.21.15.177:0/3950149728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26bc101ed0 0x7f26bc19d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T00:48:28.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.657+0000 7f26ba59c700 1 --2- 172.21.15.177:0/3950149728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26bc101ed0 0x7f26bc19d620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T00:48:28.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.657+0000 7f26b1d9b700 1 --2- 172.21.15.177:0/3950149728 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26bc19db60 0x7f26bc197790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:28.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.657+0000 7f26b1d9b700 1 -- 172.21.15.177:0/3950149728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26bc101ed0 msgr2=0x7f26bc19d620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T00:48:28.662 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.657+0000 7f26b1d9b700 1 --2- 172.21.15.177:0/3950149728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26bc101ed0 0x7f26bc19d620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:28.662 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.657+0000 7f26b1d9b700 1 -- 172.21.15.177:0/3950149728 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26bc19e0e0 con 0x7f26bc19db60 2024-03-28T00:48:28.662 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.657+0000 7f26b1d9b700 1 --2- 172.21.15.177:0/3950149728 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26bc19db60 0x7f26bc197790 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f26a400b4d0 tx=0x7f26a400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:48:28.662 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.657+0000 7f26b3fff700 1 -- 172.21.15.177:0/3950149728 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f26a4011630 con 0x7f26bc19db60 2024-03-28T00:48:28.662 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.657+0000 7f26b3fff700 1 -- 172.21.15.177:0/3950149728 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26a4011cb0 con 0x7f26bc19db60 2024-03-28T00:48:28.662 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.657+0000 7f26c098b700 1 -- 172.21.15.177:0/3950149728 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26bc197d30 con 0x7f26bc19db60 2024-03-28T00:48:28.662 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.657+0000 7f26b3fff700 1 -- 172.21.15.177:0/3950149728 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f26a4010e60 con 0x7f26bc19db60 2024-03-28T00:48:28.662 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.657+0000 7f26c098b700 1 -- 172.21.15.177:0/3950149728 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26bc1982c0 con 0x7f26bc19db60 2024-03-28T00:48:28.663 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.658+0000 7f26b3fff700 1 -- 172.21.15.177:0/3950149728 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f26a4018020 con 0x7f26bc19db60 2024-03-28T00:48:28.663 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.658+0000 7f26c098b700 1 -- 172.21.15.177:0/3950149728 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26bc1033b0 con 0x7f26bc19db60 2024-03-28T00:48:28.663 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.658+0000 7f26b3fff700 1 --2- 172.21.15.177:0/3950149728 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f26a8038d10 0x7f26a803b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:28.663 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.658+0000 7f26b3fff700 1 -- 172.21.15.177:0/3950149728 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f26a404bc50 con 0x7f26bc19db60 2024-03-28T00:48:28.663 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.659+0000 7f26ba59c700 1 --2- 172.21.15.177:0/3950149728 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f26a8038d10 0x7f26a803b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:28.663 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.659+0000 7f26ba59c700 1 --2- 172.21.15.177:0/3950149728 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f26a8038d10 0x7f26a803b1d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f26ac000c00 tx=0x7f26ac00b580 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:48:28.666 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.664+0000 7f26b3fff700 1 -- 172.21.15.177:0/3950149728 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f26a401f020 con 0x7f26bc19db60 2024-03-28T00:48:28.770 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.767+0000 7f26b3fff700 1 -- 172.21.15.177:0/3950149728 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a40178b0 con 0x7f26bc19db60 2024-03-28T00:48:28.788 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:28 smithi174 ceph-mon[36060]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-28T00:48:28.788 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:28 smithi174 bash[36041]: 19bd7a2dd6fec9218147a3188bb19d3962f792cde05ec15c5571e92af254d1e0 2024-03-28T00:48:28.944 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:28.941+0000 7f26c098b700 1 -- 172.21.15.177:0/3950149728 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26bc198630 con 0x7f26bc19db60 2024-03-28T00:48:29.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:28 smithi174 systemd[1]: Started Ceph mon.b for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T00:48:33.831 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:33.827+0000 7f26b3fff700 1 -- 172.21.15.177:0/3950149728 <== mon.0 v2:172.21.15.169:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1289 (secure 0 0 0) 0x7f26a403e520 con 0x7f26bc19db60 2024-03-28T00:48:33.831 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-28T00:48:33.831 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":3,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","modified":"2024-03-28T00:48:28.766239Z","created":"2024-03-28T00:46:37.887122Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-28T00:48:33.834 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:33.830+0000 7f26c098b700 1 -- 172.21.15.177:0/3950149728 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f26a8038d10 msgr2=0x7f26a803b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:33.834 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:33.830+0000 7f26c098b700 1 --2- 172.21.15.177:0/3950149728 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f26a8038d10 0x7f26a803b1d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f26ac000c00 tx=0x7f26ac00b580 comp rx=0 tx=0).stop 2024-03-28T00:48:33.834 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:33.830+0000 7f26c098b700 1 -- 172.21.15.177:0/3950149728 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26bc19db60 msgr2=0x7f26bc197790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:33.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:33.830+0000 7f26c098b700 1 --2- 172.21.15.177:0/3950149728 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26bc19db60 0x7f26bc197790 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f26a400b4d0 tx=0x7f26a400b890 comp rx=0 tx=0).stop 2024-03-28T00:48:33.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:33.830+0000 7f26c098b700 1 -- 172.21.15.177:0/3950149728 shutdown_connections 2024-03-28T00:48:33.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:33.830+0000 7f26c098b700 1 --2- 172.21.15.177:0/3950149728 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f26a8038d10 0x7f26a803b1d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:33.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:33.830+0000 7f26c098b700 1 --2- 172.21.15.177:0/3950149728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26bc101ed0 0x7f26bc19d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:33.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:33.830+0000 7f26c098b700 1 --2- 172.21.15.177:0/3950149728 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26bc19db60 0x7f26bc197790 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:33.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:33.830+0000 7f26c098b700 1 -- 172.21.15.177:0/3950149728 >> 172.21.15.177:0/3950149728 conn(0x7f26bc0fd760 msgr2=0x7f26bc1038a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:48:33.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:33.830+0000 7f26c098b700 1 -- 172.21.15.177:0/3950149728 shutdown_connections 2024-03-28T00:48:33.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:33.830+0000 7f26c098b700 1 -- 172.21.15.177:0/3950149728 wait complete. 2024-03-28T00:48:33.836 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 3 2024-03-28T00:48:34.085 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T00:48:34.085 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: mon.a calling monitor election 2024-03-28T00:48:34.085 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: mon.c calling monitor election 2024-03-28T00:48:34.086 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:48:34.086 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:48:34.086 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: from='client.? 172.21.15.177:0/3950149728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T00:48:34.086 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:48:34.086 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:34.086 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:48:34.086 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: mon.b calling monitor election 2024-03-28T00:48:34.086 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:48:34.086 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:34.086 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:48:34.086 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:48:34.086 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T00:48:34.086 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: monmap e3: 3 mons at {a=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],b=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],c=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T00:48:34.086 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: fsmap 2024-03-28T00:48:34.087 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T00:48:34.087 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: mgrmap e13: x(active, since 57s) 2024-03-28T00:48:34.087 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: overall HEALTH_OK 2024-03-28T00:48:34.087 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:34.087 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:34.087 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:34.087 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:34.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T00:48:34.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: mon.a calling monitor election 2024-03-28T00:48:34.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: mon.c calling monitor election 2024-03-28T00:48:34.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:48:34.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:48:34.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: from='client.? 172.21.15.177:0/3950149728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T00:48:34.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:48:34.304 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:34.304 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:48:34.304 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: mon.b calling monitor election 2024-03-28T00:48:34.304 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:48:34.304 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:34.304 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:48:34.304 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:48:34.304 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T00:48:34.304 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: monmap e3: 3 mons at {a=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],b=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],c=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T00:48:34.305 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: fsmap 2024-03-28T00:48:34.305 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T00:48:34.305 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: mgrmap e13: x(active, since 57s) 2024-03-28T00:48:34.305 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: overall HEALTH_OK 2024-03-28T00:48:34.305 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:34.305 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:34.305 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:34.306 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:34.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T00:48:34.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: mon.a calling monitor election 2024-03-28T00:48:34.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: mon.c calling monitor election 2024-03-28T00:48:34.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:48:34.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:48:34.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: from='client.? 172.21.15.177:0/3950149728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T00:48:34.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:48:34.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:34.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:48:34.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: mon.b calling monitor election 2024-03-28T00:48:34.360 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:48:34.360 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:34.360 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:48:34.360 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:48:34.360 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T00:48:34.360 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: monmap e3: 3 mons at {a=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],b=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],c=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T00:48:34.360 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: fsmap 2024-03-28T00:48:34.360 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T00:48:34.360 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: mgrmap e13: x(active, since 57s) 2024-03-28T00:48:34.360 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: overall HEALTH_OK 2024-03-28T00:48:34.360 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:34.360 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:34.360 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:34.360 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:34.396 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-28T00:48:34.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-28T00:48:34.644 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:48:35.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:34 smithi169 ceph-mon[32274]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:35.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:34 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:48:35.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:34 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:48:35.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:34 smithi174 ceph-mon[36060]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:35.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:34 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:48:35.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:34 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:48:35.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:34 smithi177 ceph-mon[36206]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:35.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:34 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:48:35.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:34 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:48:36.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.265+0000 7f79dec6e700 1 -- 172.21.15.169:0/2900020060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79d8106a50 msgr2=0x7f79d8106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:36.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.265+0000 7f79dec6e700 1 --2- 172.21.15.169:0/2900020060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79d8106a50 0x7f79d8106e30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f79cc009910 tx=0x7f79cc009c20 comp rx=0 tx=0).stop 2024-03-28T00:48:36.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.265+0000 7f79dec6e700 1 -- 172.21.15.169:0/2900020060 shutdown_connections 2024-03-28T00:48:36.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.265+0000 7f79dec6e700 1 --2- 172.21.15.169:0/2900020060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79d8106a50 0x7f79d8106e30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:36.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.265+0000 7f79dec6e700 1 -- 172.21.15.169:0/2900020060 >> 172.21.15.169:0/2900020060 conn(0x7f79d80fc580 msgr2=0x7f79d80fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:48:36.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.265+0000 7f79dec6e700 1 -- 172.21.15.169:0/2900020060 shutdown_connections 2024-03-28T00:48:36.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.265+0000 7f79dec6e700 1 -- 172.21.15.169:0/2900020060 wait complete. 2024-03-28T00:48:36.270 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.270 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.270 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.270 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.270 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:36.270 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:48:36.270 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.270 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.270 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.270 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.271 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.271 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.271 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.271 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.271 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.271 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.271 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.271 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T00:48:36.271 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T00:48:36.271 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:36.271 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.266+0000 7f79dec6e700 1 Processor -- start 2024-03-28T00:48:36.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.266+0000 7f79dec6e700 1 -- start start 2024-03-28T00:48:36.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.266+0000 7f79dec6e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79d8106a50 0x7f79d8198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:36.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.266+0000 7f79dec6e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79d8198c90 0x7f79d8199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:36.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.266+0000 7f79dec6e700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79d81a17c0 0x7f79d81a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:36.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.266+0000 7f79dec6e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d810b090 con 0x7f79d8106a50 2024-03-28T00:48:36.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.266+0000 7f79dec6e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d810af10 con 0x7f79d8198c90 2024-03-28T00:48:36.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.266+0000 7f79dec6e700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d810b210 con 0x7f79d81a17c0 2024-03-28T00:48:36.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.266+0000 7f79dca0a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79d8106a50 0x7f79d8198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:36.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.266+0000 7f79dca0a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79d8106a50 0x7f79d8198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48196/0 (socket says 172.21.15.169:48196) 2024-03-28T00:48:36.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.266+0000 7f79dca0a700 1 -- 172.21.15.169:0/3262290385 learned_addr learned my addr 172.21.15.169:0/3262290385 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:48:36.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.266+0000 7f79d7fff700 1 --2- 172.21.15.169:0/3262290385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79d8198c90 0x7f79d8199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:36.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.266+0000 7f79dd20b700 1 --2- 172.21.15.169:0/3262290385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79d81a17c0 0x7f79d81a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:36.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.267+0000 7f79dca0a700 1 -- 172.21.15.169:0/3262290385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79d81a17c0 msgr2=0x7f79d81a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:36.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.267+0000 7f79dca0a700 1 --2- 172.21.15.169:0/3262290385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79d81a17c0 0x7f79d81a3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:36.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.267+0000 7f79dca0a700 1 -- 172.21.15.169:0/3262290385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79d8198c90 msgr2=0x7f79d8199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:36.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.267+0000 7f79dca0a700 1 --2- 172.21.15.169:0/3262290385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79d8198c90 0x7f79d8199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:36.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.267+0000 7f79dca0a700 1 -- 172.21.15.169:0/3262290385 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d81a43c0 con 0x7f79d8106a50 2024-03-28T00:48:36.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.267+0000 7f79dca0a700 1 --2- 172.21.15.169:0/3262290385 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79d8106a50 0x7f79d8198750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f79cc005850 tx=0x7f79cc004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:48:36.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.267+0000 7f79d5ffb700 1 -- 172.21.15.169:0/3262290385 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc026070 con 0x7f79d8106a50 2024-03-28T00:48:36.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.267+0000 7f79dec6e700 1 -- 172.21.15.169:0/3262290385 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d81965a0 con 0x7f79d8106a50 2024-03-28T00:48:36.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.267+0000 7f79d5ffb700 1 -- 172.21.15.169:0/3262290385 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79cc00b6f0 con 0x7f79d8106a50 2024-03-28T00:48:36.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.267+0000 7f79d5ffb700 1 -- 172.21.15.169:0/3262290385 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc0038e0 con 0x7f79d8106a50 2024-03-28T00:48:36.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.267+0000 7f79dec6e700 1 -- 172.21.15.169:0/3262290385 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d8196b30 con 0x7f79d8106a50 2024-03-28T00:48:36.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.268+0000 7f79d5ffb700 1 -- 172.21.15.169:0/3262290385 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f79cc029070 con 0x7f79d8106a50 2024-03-28T00:48:36.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.268+0000 7f79d5ffb700 1 --2- 172.21.15.169:0/3262290385 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f79c8039450 0x7f79c803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:36.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.268+0000 7f79d5ffb700 1 -- 172.21.15.169:0/3262290385 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f79cc04dc80 con 0x7f79d8106a50 2024-03-28T00:48:36.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.269+0000 7f79d7fff700 1 --2- 172.21.15.169:0/3262290385 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f79c8039450 0x7f79c803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:36.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.269+0000 7f79dec6e700 1 -- 172.21.15.169:0/3262290385 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d80637c0 con 0x7f79d8106a50 2024-03-28T00:48:36.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.269+0000 7f79d7fff700 1 --2- 172.21.15.169:0/3262290385 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f79c8039450 0x7f79c803b910 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f79c4009ad0 tx=0x7f79c4009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:48:36.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.274+0000 7f79d5ffb700 1 -- 172.21.15.169:0/3262290385 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f79cc030070 con 0x7f79d8106a50 2024-03-28T00:48:36.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.496+0000 7f79dec6e700 1 -- 172.21.15.169:0/3262290385 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f79d8196d90 con 0x7f79d8106a50 2024-03-28T00:48:36.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.496+0000 7f79d5ffb700 1 -- 172.21.15.169:0/3262290385 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+277 (secure 0 0 0) 0x7f79cc013070 con 0x7f79d8106a50 2024-03-28T00:48:36.500 INFO:teuthology.orchestra.run.smithi169.stdout:# minimal ceph.conf for 833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T00:48:36.500 INFO:teuthology.orchestra.run.smithi169.stdout:[global] 2024-03-28T00:48:36.500 INFO:teuthology.orchestra.run.smithi169.stdout: fsid = 833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T00:48:36.500 INFO:teuthology.orchestra.run.smithi169.stdout: mon_host = [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-03-28T00:48:36.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.499+0000 7f79dec6e700 1 -- 172.21.15.169:0/3262290385 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f79c8039450 msgr2=0x7f79c803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:36.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.499+0000 7f79dec6e700 1 --2- 172.21.15.169:0/3262290385 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f79c8039450 0x7f79c803b910 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f79c4009ad0 tx=0x7f79c4009210 comp rx=0 tx=0).stop 2024-03-28T00:48:36.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.499+0000 7f79dec6e700 1 -- 172.21.15.169:0/3262290385 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79d8106a50 msgr2=0x7f79d8198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:36.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.499+0000 7f79dec6e700 1 --2- 172.21.15.169:0/3262290385 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79d8106a50 0x7f79d8198750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f79cc005850 tx=0x7f79cc004f80 comp rx=0 tx=0).stop 2024-03-28T00:48:36.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.499+0000 7f79dec6e700 1 -- 172.21.15.169:0/3262290385 shutdown_connections 2024-03-28T00:48:36.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.499+0000 7f79dec6e700 1 --2- 172.21.15.169:0/3262290385 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f79c8039450 0x7f79c803b910 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:36.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.499+0000 7f79dec6e700 1 --2- 172.21.15.169:0/3262290385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79d81a17c0 0x7f79d81a3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:36.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.499+0000 7f79dec6e700 1 --2- 172.21.15.169:0/3262290385 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79d8106a50 0x7f79d8198750 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:36.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.499+0000 7f79dec6e700 1 --2- 172.21.15.169:0/3262290385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79d8198c90 0x7f79d8199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:36.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.499+0000 7f79dec6e700 1 -- 172.21.15.169:0/3262290385 >> 172.21.15.169:0/3262290385 conn(0x7f79d80fc580 msgr2=0x7f79d8102550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:48:36.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.499+0000 7f79dec6e700 1 -- 172.21.15.169:0/3262290385 shutdown_connections 2024-03-28T00:48:36.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:36.499+0000 7f79dec6e700 1 -- 172.21.15.169:0/3262290385 wait complete. 2024-03-28T00:48:36.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:36.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:48:36.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.610 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.610 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.610 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.610 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.610 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T00:48:36.610 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T00:48:36.610 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:36.610 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:36.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:48:36.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:36.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T00:48:36.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T00:48:36.669 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:36.669 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:37.479 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:37 smithi169 ceph-mon[32274]: Updating smithi169:/etc/ceph/ceph.conf 2024-03-28T00:48:37.480 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:37 smithi169 ceph-mon[32274]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-28T00:48:37.480 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:37 smithi169 ceph-mon[32274]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-28T00:48:37.480 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:37 smithi169 ceph-mon[32274]: Updating smithi177:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T00:48:37.480 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:37 smithi169 ceph-mon[32274]: Updating smithi174:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T00:48:37.480 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:37 smithi169 ceph-mon[32274]: Updating smithi169:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T00:48:37.480 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:37 smithi169 ceph-mon[32274]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T00:48:37.480 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:37 smithi169 ceph-mon[32274]: Reconfiguring daemon mon.a on smithi169 2024-03-28T00:48:37.480 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:37 smithi169 ceph-mon[32274]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:37.480 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:37 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3262290385' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:37.507 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-28T00:48:37.507 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-28T00:48:37.507 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T00:48:37.537 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-28T00:48:37.537 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T00:48:37.605 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-28T00:48:37.605 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T00:48:37.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:37 smithi174 ceph-mon[36060]: Updating smithi169:/etc/ceph/ceph.conf 2024-03-28T00:48:37.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:37 smithi174 ceph-mon[36060]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-28T00:48:37.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:37 smithi174 ceph-mon[36060]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-28T00:48:37.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:37 smithi174 ceph-mon[36060]: Updating smithi177:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T00:48:37.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:37 smithi174 ceph-mon[36060]: Updating smithi174:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T00:48:37.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:37 smithi174 ceph-mon[36060]: Updating smithi169:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T00:48:37.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:37 smithi174 ceph-mon[36060]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T00:48:37.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:37 smithi174 ceph-mon[36060]: Reconfiguring daemon mon.a on smithi169 2024-03-28T00:48:37.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:37 smithi174 ceph-mon[36060]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:37.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:37 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3262290385' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:37.640 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-28T00:48:37.640 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T00:48:37.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:37 smithi177 ceph-mon[36206]: Updating smithi169:/etc/ceph/ceph.conf 2024-03-28T00:48:37.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:37 smithi177 ceph-mon[36206]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-28T00:48:37.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:37 smithi177 ceph-mon[36206]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-28T00:48:37.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:37 smithi177 ceph-mon[36206]: Updating smithi177:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T00:48:37.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:37 smithi177 ceph-mon[36206]: Updating smithi174:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T00:48:37.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:37 smithi177 ceph-mon[36206]: Updating smithi169:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T00:48:37.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:37 smithi177 ceph-mon[36206]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T00:48:37.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:37 smithi177 ceph-mon[36206]: Reconfiguring daemon mon.a on smithi169 2024-03-28T00:48:37.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:37 smithi177 ceph-mon[36206]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:37.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:37 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3262290385' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:37.712 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-28T00:48:37.712 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T00:48:37.746 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-28T00:48:37.746 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T00:48:37.817 INFO:tasks.cephadm:Adding mgr.x on smithi169 2024-03-28T00:48:37.818 INFO:tasks.cephadm:Adding mgr.y on smithi174 2024-03-28T00:48:37.818 INFO:tasks.cephadm:Adding mgr.z on smithi177 2024-03-28T00:48:37.818 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph orch apply mgr '3;smithi169=x;smithi174=y;smithi177=z' 2024-03-28T00:48:38.095 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T00:48:39.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:39 smithi169 ceph-mon[32274]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:39.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:39 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:39.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:39 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:39.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:39 smithi169 ceph-mon[32274]: Reconfiguring mon.b (monmap changed)... 2024-03-28T00:48:39.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:39 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T00:48:39.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:39 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T00:48:39.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:39 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:39.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:39 smithi169 ceph-mon[32274]: Reconfiguring daemon mon.b on smithi174 2024-03-28T00:48:39.355 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:39 smithi177 ceph-mon[36206]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:39.355 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:39 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:39.355 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:39 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:39.355 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:39 smithi177 ceph-mon[36206]: Reconfiguring mon.b (monmap changed)... 2024-03-28T00:48:39.355 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:39 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T00:48:39.355 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:39 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T00:48:39.355 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:39 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:39.355 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:39 smithi177 ceph-mon[36206]: Reconfiguring daemon mon.b on smithi174 2024-03-28T00:48:39.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:39 smithi174 ceph-mon[36060]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:39.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:39 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:39.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:39 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:39.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:39 smithi174 ceph-mon[36060]: Reconfiguring mon.b (monmap changed)... 2024-03-28T00:48:39.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:39 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T00:48:39.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:39 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T00:48:39.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:39 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:39.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:39 smithi174 ceph-mon[36060]: Reconfiguring daemon mon.b on smithi174 2024-03-28T00:48:39.544 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.538+0000 7f7170c15700 1 -- 172.21.15.177:0/548986689 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f716c100bf0 msgr2=0x7f716c101010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:39.544 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.538+0000 7f7170c15700 1 --2- 172.21.15.177:0/548986689 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f716c100bf0 0x7f716c101010 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7160009910 tx=0x7f7160009c20 comp rx=0 tx=0).stop 2024-03-28T00:48:39.544 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.538+0000 7f7170c15700 1 -- 172.21.15.177:0/548986689 shutdown_connections 2024-03-28T00:48:39.544 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.538+0000 7f7170c15700 1 --2- 172.21.15.177:0/548986689 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f716c100bf0 0x7f716c101010 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:39.544 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.538+0000 7f7170c15700 1 -- 172.21.15.177:0/548986689 >> 172.21.15.177:0/548986689 conn(0x7f716c0fc190 msgr2=0x7f716c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:48:39.544 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.538+0000 7f7170c15700 1 -- 172.21.15.177:0/548986689 shutdown_connections 2024-03-28T00:48:39.544 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.538+0000 7f7170c15700 1 -- 172.21.15.177:0/548986689 wait complete. 2024-03-28T00:48:39.544 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.539+0000 7f7170c15700 1 Processor -- start 2024-03-28T00:48:39.544 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.539+0000 7f7170c15700 1 -- start start 2024-03-28T00:48:39.544 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.539+0000 7f7170c15700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f716c100bf0 0x7f716c06aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:39.545 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.539+0000 7f7170c15700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f716c19a260 0x7f716c19c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:39.545 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.539+0000 7f7170c15700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f716c19cbe0 0x7f716c19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:39.545 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.539+0000 7f7170c15700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716c106c30 con 0x7f716c100bf0 2024-03-28T00:48:39.545 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.539+0000 7f7170c15700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716c106ab0 con 0x7f716c19cbe0 2024-03-28T00:48:39.545 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.539+0000 7f7170c15700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716c106db0 con 0x7f716c19a260 2024-03-28T00:48:39.545 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.539+0000 7f7169d9b700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f716c19a260 0x7f716c19c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:39.545 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.539+0000 7f7169d9b700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f716c19a260 0x7f716c19c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:51062/0 (socket says 172.21.15.177:51062) 2024-03-28T00:48:39.545 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.539+0000 7f7169d9b700 1 -- 172.21.15.177:0/3101299233 learned_addr learned my addr 172.21.15.177:0/3101299233 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-28T00:48:39.545 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.539+0000 7f716a59c700 1 --2- 172.21.15.177:0/3101299233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f716c100bf0 0x7f716c06aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:39.545 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.539+0000 7f716ad9d700 1 --2- 172.21.15.177:0/3101299233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f716c19cbe0 0x7f716c19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:39.545 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.540+0000 7f716a59c700 1 -- 172.21.15.177:0/3101299233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f716c19a260 msgr2=0x7f716c19c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:39.545 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.540+0000 7f716a59c700 1 --2- 172.21.15.177:0/3101299233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f716c19a260 0x7f716c19c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:39.545 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.540+0000 7f716a59c700 1 -- 172.21.15.177:0/3101299233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f716c19cbe0 msgr2=0x7f716c19f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:39.545 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.540+0000 7f716a59c700 1 --2- 172.21.15.177:0/3101299233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f716c19cbe0 0x7f716c19f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:39.546 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.540+0000 7f716a59c700 1 -- 172.21.15.177:0/3101299233 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716c19f940 con 0x7f716c100bf0 2024-03-28T00:48:39.546 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.540+0000 7f716ad9d700 1 --2- 172.21.15.177:0/3101299233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f716c19cbe0 0x7f716c19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:48:39.546 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.540+0000 7f716a59c700 1 --2- 172.21.15.177:0/3101299233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f716c100bf0 0x7f716c06aed0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f716000b5c0 tx=0x7f7160004f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:48:39.546 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.540+0000 7f715b7fe700 1 -- 172.21.15.177:0/3101299233 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7160027070 con 0x7f716c100bf0 2024-03-28T00:48:39.546 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.540+0000 7f715b7fe700 1 -- 172.21.15.177:0/3101299233 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7160005da0 con 0x7f716c100bf0 2024-03-28T00:48:39.546 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.540+0000 7f715b7fe700 1 -- 172.21.15.177:0/3101299233 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7160022e10 con 0x7f716c100bf0 2024-03-28T00:48:39.546 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.540+0000 7f7170c15700 1 -- 172.21.15.177:0/3101299233 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f716c19fbd0 con 0x7f716c100bf0 2024-03-28T00:48:39.546 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.540+0000 7f7170c15700 1 -- 172.21.15.177:0/3101299233 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f716c1a0060 con 0x7f716c100bf0 2024-03-28T00:48:39.547 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.541+0000 7f715b7fe700 1 -- 172.21.15.177:0/3101299233 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f716002a070 con 0x7f716c100bf0 2024-03-28T00:48:39.547 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.541+0000 7f715b7fe700 1 --2- 172.21.15.177:0/3101299233 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7154039450 0x7f715403b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:39.547 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.541+0000 7f715b7fe700 1 -- 172.21.15.177:0/3101299233 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f716004cfe0 con 0x7f716c100bf0 2024-03-28T00:48:39.547 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.541+0000 7f7169d9b700 1 --2- 172.21.15.177:0/3101299233 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7154039450 0x7f715403b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:39.547 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.541+0000 7f7170c15700 1 -- 172.21.15.177:0/3101299233 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f716c02da30 con 0x7f716c100bf0 2024-03-28T00:48:39.547 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.542+0000 7f7169d9b700 1 --2- 172.21.15.177:0/3101299233 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7154039450 0x7f715403b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f716c06b450 tx=0x7f715c0092c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:48:39.550 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.547+0000 7f715b7fe700 1 -- 172.21.15.177:0/3101299233 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7160013070 con 0x7f716c100bf0 2024-03-28T00:48:39.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.772+0000 7f7170c15700 1 -- 172.21.15.177:0/3101299233 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi169=x;smithi174=y;smithi177=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f716c0625d0 con 0x7f7154039450 2024-03-28T00:48:39.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.779+0000 7f715b7fe700 1 -- 172.21.15.177:0/3101299233 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f716c0625d0 con 0x7f7154039450 2024-03-28T00:48:39.783 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled mgr update... 2024-03-28T00:48:39.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.781+0000 7f7170c15700 1 -- 172.21.15.177:0/3101299233 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7154039450 msgr2=0x7f715403b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:39.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.781+0000 7f7170c15700 1 --2- 172.21.15.177:0/3101299233 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7154039450 0x7f715403b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f716c06b450 tx=0x7f715c0092c0 comp rx=0 tx=0).stop 2024-03-28T00:48:39.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.781+0000 7f7170c15700 1 -- 172.21.15.177:0/3101299233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f716c100bf0 msgr2=0x7f716c06aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:39.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.781+0000 7f7170c15700 1 --2- 172.21.15.177:0/3101299233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f716c100bf0 0x7f716c06aed0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f716000b5c0 tx=0x7f7160004f60 comp rx=0 tx=0).stop 2024-03-28T00:48:39.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.781+0000 7f7170c15700 1 -- 172.21.15.177:0/3101299233 shutdown_connections 2024-03-28T00:48:39.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.781+0000 7f7170c15700 1 --2- 172.21.15.177:0/3101299233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f716c19cbe0 0x7f716c19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:39.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.781+0000 7f7170c15700 1 --2- 172.21.15.177:0/3101299233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f716c100bf0 0x7f716c06aed0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:39.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.781+0000 7f7170c15700 1 --2- 172.21.15.177:0/3101299233 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7154039450 0x7f715403b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:39.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.781+0000 7f7170c15700 1 --2- 172.21.15.177:0/3101299233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f716c19a260 0x7f716c19c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:39.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.781+0000 7f7170c15700 1 -- 172.21.15.177:0/3101299233 >> 172.21.15.177:0/3101299233 conn(0x7f716c0fc190 msgr2=0x7f716c0fceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:48:39.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.781+0000 7f7170c15700 1 -- 172.21.15.177:0/3101299233 shutdown_connections 2024-03-28T00:48:39.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:48:39.782+0000 7f7170c15700 1 -- 172.21.15.177:0/3101299233 wait complete. 2024-03-28T00:48:40.391 DEBUG:teuthology.orchestra.run.smithi174:mgr.y> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mgr.y.service 2024-03-28T00:48:40.394 DEBUG:teuthology.orchestra.run.smithi177:mgr.z> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mgr.z.service 2024-03-28T00:48:40.396 INFO:tasks.cephadm:Deploying OSDs... 2024-03-28T00:48:40.396 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-28T00:48:40.397 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T00:48:40.417 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T00:48:40.417 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_1 2024-03-28T00:48:40.475 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T00:48:40.475 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T00:48:40.475 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-28T00:48:40.476 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T00:48:40.476 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T00:48:40.476 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-28 00:47:45.697168726 +0000 2024-03-28T00:48:40.476 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-28 00:42:29.933287141 +0000 2024-03-28T00:48:40.476 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-28 00:42:29.933287141 +0000 2024-03-28T00:48:40.476 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-28 00:42:29.933287141 +0000 2024-03-28T00:48:40.476 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T00:48:40.542 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:40 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:40.542 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:40 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:40.542 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:40 smithi177 ceph-mon[36206]: Reconfiguring mon.c (monmap changed)... 2024-03-28T00:48:40.542 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:40 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T00:48:40.542 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:40 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T00:48:40.542 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:40 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:40.542 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:40 smithi177 ceph-mon[36206]: Reconfiguring daemon mon.c on smithi177 2024-03-28T00:48:40.542 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:40 smithi177 ceph-mon[36206]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi169=x;smithi174=y;smithi177=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:48:40.542 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:40 smithi177 ceph-mon[36206]: Saving service mgr spec with placement smithi169=x;smithi174=y;smithi177=z;count:3 2024-03-28T00:48:40.542 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:40 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:40.542 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:40 smithi177 ceph-mon[36206]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:40.547 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-28T00:48:40.547 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-28T00:48:40.547 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000244704 s, 2.1 MB/s 2024-03-28T00:48:40.548 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T00:48:40.606 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_2 2024-03-28T00:48:40.663 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T00:48:40.663 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T00:48:40.664 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-28T00:48:40.664 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T00:48:40.664 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T00:48:40.664 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-28 00:47:45.726168092 +0000 2024-03-28T00:48:40.664 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-28 00:42:30.284279473 +0000 2024-03-28T00:48:40.664 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-28 00:42:30.284279473 +0000 2024-03-28T00:48:40.664 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-28 00:42:30.284279473 +0000 2024-03-28T00:48:40.664 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T00:48:40.729 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:40 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:40.729 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:40 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:40.729 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:40 smithi169 ceph-mon[32274]: Reconfiguring mon.c (monmap changed)... 2024-03-28T00:48:40.729 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:40 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T00:48:40.729 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:40 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T00:48:40.729 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:40 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:40.729 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:40 smithi169 ceph-mon[32274]: Reconfiguring daemon mon.c on smithi177 2024-03-28T00:48:40.729 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:40 smithi169 ceph-mon[32274]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi169=x;smithi174=y;smithi177=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:48:40.730 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:40 smithi169 ceph-mon[32274]: Saving service mgr spec with placement smithi169=x;smithi174=y;smithi177=z;count:3 2024-03-28T00:48:40.730 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:40 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:40.730 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:40 smithi169 ceph-mon[32274]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:40.731 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-28T00:48:40.732 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-28T00:48:40.732 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000244271 s, 2.1 MB/s 2024-03-28T00:48:40.733 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T00:48:40.790 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_3 2024-03-28T00:48:40.846 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T00:48:40.846 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T00:48:40.846 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-28T00:48:40.846 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T00:48:40.847 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T00:48:40.847 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-28 00:47:45.753167501 +0000 2024-03-28T00:48:40.847 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-28 00:42:30.641271673 +0000 2024-03-28T00:48:40.847 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-28 00:42:30.641271673 +0000 2024-03-28T00:48:40.847 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-28 00:42:30.641271673 +0000 2024-03-28T00:48:40.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T00:48:40.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:40 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:40.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:40 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:40.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:40 smithi174 ceph-mon[36060]: Reconfiguring mon.c (monmap changed)... 2024-03-28T00:48:40.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:40 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T00:48:40.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:40 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T00:48:40.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:40 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:40.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:40 smithi174 ceph-mon[36060]: Reconfiguring daemon mon.c on smithi177 2024-03-28T00:48:40.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:40 smithi174 ceph-mon[36060]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi169=x;smithi174=y;smithi177=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:48:40.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:40 smithi174 ceph-mon[36060]: Saving service mgr spec with placement smithi169=x;smithi174=y;smithi177=z;count:3 2024-03-28T00:48:40.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:40 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:40.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:40 smithi174 ceph-mon[36060]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:40.913 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-28T00:48:40.913 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-28T00:48:40.913 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000221885 s, 2.3 MB/s 2024-03-28T00:48:40.915 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T00:48:40.971 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_4 2024-03-28T00:48:41.026 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T00:48:41.026 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T00:48:41.026 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-28T00:48:41.027 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T00:48:41.027 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T00:48:41.027 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-28 00:47:45.781166888 +0000 2024-03-28T00:48:41.027 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-28 00:42:30.980264267 +0000 2024-03-28T00:48:41.027 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-28 00:42:30.980264267 +0000 2024-03-28T00:48:41.027 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-28 00:42:30.980264267 +0000 2024-03-28T00:48:41.027 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T00:48:41.092 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-28T00:48:41.092 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-28T00:48:41.092 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000202814 s, 2.5 MB/s 2024-03-28T00:48:41.094 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T00:48:41.150 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-28T00:48:41.150 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T00:48:41.169 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T00:48:41.169 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_1 2024-03-28T00:48:41.227 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T00:48:41.227 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T00:48:41.227 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-28T00:48:41.227 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T00:48:41.227 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T00:48:41.227 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-28 00:48:28.220189861 +0000 2024-03-28T00:48:41.227 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-28 00:42:35.842660286 +0000 2024-03-28T00:48:41.227 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-28 00:42:35.842660286 +0000 2024-03-28T00:48:41.227 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-28 00:42:35.842660286 +0000 2024-03-28T00:48:41.228 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T00:48:41.296 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-28T00:48:41.296 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-28T00:48:41.296 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000311703 s, 1.6 MB/s 2024-03-28T00:48:41.298 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T00:48:41.356 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_2 2024-03-28T00:48:41.413 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T00:48:41.413 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T00:48:41.413 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-28T00:48:41.413 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T00:48:41.413 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T00:48:41.413 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-28 00:48:28.220189861 +0000 2024-03-28T00:48:41.414 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-28 00:42:36.194655426 +0000 2024-03-28T00:48:41.414 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-28 00:42:36.194655426 +0000 2024-03-28T00:48:41.414 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-28 00:42:36.194655426 +0000 2024-03-28T00:48:41.414 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T00:48:41.481 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-28T00:48:41.482 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-28T00:48:41.482 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.00022093 s, 2.3 MB/s 2024-03-28T00:48:41.483 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T00:48:41.544 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_3 2024-03-28T00:48:41.602 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T00:48:41.602 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T00:48:41.602 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-28T00:48:41.602 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T00:48:41.603 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T00:48:41.603 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-28 00:48:28.220189861 +0000 2024-03-28T00:48:41.603 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-28 00:42:36.534650732 +0000 2024-03-28T00:48:41.603 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-28 00:42:36.534650732 +0000 2024-03-28T00:48:41.603 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-28 00:42:36.534650732 +0000 2024-03-28T00:48:41.603 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T00:48:41.672 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-28T00:48:41.672 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-28T00:48:41.672 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000210606 s, 2.4 MB/s 2024-03-28T00:48:41.674 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T00:48:41.731 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_4 2024-03-28T00:48:41.787 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T00:48:41.787 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T00:48:41.787 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-03-28T00:48:41.787 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T00:48:41.787 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T00:48:41.787 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-28 00:48:28.221189848 +0000 2024-03-28T00:48:41.787 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-28 00:42:36.874646038 +0000 2024-03-28T00:48:41.787 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-28 00:42:36.874646038 +0000 2024-03-28T00:48:41.787 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-28 00:42:36.874646038 +0000 2024-03-28T00:48:41.788 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T00:48:41.855 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-28T00:48:41.855 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-28T00:48:41.855 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000278539 s, 1.8 MB/s 2024-03-28T00:48:41.856 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T00:48:41.912 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-28T00:48:41.912 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T00:48:41.931 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T00:48:41.931 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2024-03-28T00:48:41.989 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T00:48:41.989 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T00:48:41.989 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-03-28T00:48:41.989 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T00:48:41.989 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T00:48:41.989 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-28 00:48:18.641330691 +0000 2024-03-28T00:48:41.989 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-28 00:42:52.569500237 +0000 2024-03-28T00:48:41.989 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-28 00:42:52.569500237 +0000 2024-03-28T00:48:41.989 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-28 00:42:52.569500237 +0000 2024-03-28T00:48:41.990 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T00:48:42.059 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-28T00:48:42.059 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-28T00:48:42.059 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000303852 s, 1.7 MB/s 2024-03-28T00:48:42.060 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T00:48:42.116 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2024-03-28T00:48:42.171 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T00:48:42.171 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T00:48:42.171 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-03-28T00:48:42.171 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T00:48:42.171 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T00:48:42.171 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-28 00:48:18.641330691 +0000 2024-03-28T00:48:42.171 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-28 00:42:52.917490122 +0000 2024-03-28T00:48:42.171 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-28 00:42:52.917490122 +0000 2024-03-28T00:48:42.171 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-28 00:42:52.917490122 +0000 2024-03-28T00:48:42.172 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T00:48:42.237 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-28T00:48:42.237 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-28T00:48:42.237 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000178903 s, 2.9 MB/s 2024-03-28T00:48:42.239 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T00:48:42.294 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2024-03-28T00:48:42.349 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T00:48:42.349 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T00:48:42.349 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 700 Links: 1 2024-03-28T00:48:42.349 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T00:48:42.349 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T00:48:42.349 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-28 00:48:18.642330662 +0000 2024-03-28T00:48:42.349 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-28 00:42:53.276479701 +0000 2024-03-28T00:48:42.350 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-28 00:42:53.276479701 +0000 2024-03-28T00:48:42.350 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-28 00:42:53.276479701 +0000 2024-03-28T00:48:42.350 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T00:48:42.415 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-28T00:48:42.415 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-28T00:48:42.415 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000170047 s, 3.0 MB/s 2024-03-28T00:48:42.417 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T00:48:42.472 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2024-03-28T00:48:42.527 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T00:48:42.527 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T00:48:42.527 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-03-28T00:48:42.528 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T00:48:42.528 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T00:48:42.528 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-28 00:48:18.642330662 +0000 2024-03-28T00:48:42.528 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-28 00:42:53.621469704 +0000 2024-03-28T00:48:42.528 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-28 00:42:53.621469704 +0000 2024-03-28T00:48:42.528 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-28 00:42:53.621469704 +0000 2024-03-28T00:48:42.528 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T00:48:42.595 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-28T00:48:42.595 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-28T00:48:42.595 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000197483 s, 2.6 MB/s 2024-03-28T00:48:42.596 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T00:48:42.652 INFO:tasks.cephadm:Deploying osd.0 on smithi169 with /dev/vg_nvme/lv_4... 2024-03-28T00:48:42.652 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T00:48:42.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:42 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:42.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:42 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:42.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:42 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:48:42.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:42 smithi177 ceph-mon[36206]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:42.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:42 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:42.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:42 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:48:42.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:42 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:42.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:42 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T00:48:42.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:42 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T00:48:42.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:42 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T00:48:42.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:42 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:42.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:42 smithi177 ceph-mon[36206]: Deploying daemon mgr.z on smithi177 2024-03-28T00:48:42.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:42 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:42.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:42 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:42.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:42 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:48:42.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:42 smithi169 ceph-mon[32274]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:42.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:42 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:42.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:42 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:48:42.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:42 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:42.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:42 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T00:48:42.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:42 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T00:48:42.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:42 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T00:48:42.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:42 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:42.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:42 smithi169 ceph-mon[32274]: Deploying daemon mgr.z on smithi177 2024-03-28T00:48:42.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:42 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:42.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:42 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:42.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:42 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:48:42.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:42 smithi174 ceph-mon[36060]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:42.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:42 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:42.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:42 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:48:42.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:42 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:42.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:42 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T00:48:42.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:42 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T00:48:42.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:42 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T00:48:42.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:42 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:42.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:42 smithi174 ceph-mon[36060]: Deploying daemon mgr.z on smithi177 2024-03-28T00:48:42.888 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:48:44.313 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:44 smithi177 systemd[1]: Starting Ceph mgr.z for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T00:48:44.668 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:44 smithi177 podman[38045]: 2024-03-28 00:48:44.310084854 +0000 UTC m=+0.101464681 container create a1b0b6a946b255625c6f21de2e7ecd4d87085aa8f6cdcbecb018d29bc99bc3f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-28T00:48:44.668 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:44 smithi177 podman[38045]: 2024-03-28 00:48:44.22653731 +0000 UTC m=+0.017917143 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T00:48:44.668 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:44 smithi177 podman[38045]: 2024-03-28 00:48:44.478945352 +0000 UTC m=+0.270325184 container init a1b0b6a946b255625c6f21de2e7ecd4d87085aa8f6cdcbecb018d29bc99bc3f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True) 2024-03-28T00:48:44.668 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:44 smithi177 podman[38045]: 2024-03-28 00:48:44.481555423 +0000 UTC m=+0.272935250 container start a1b0b6a946b255625c6f21de2e7ecd4d87085aa8f6cdcbecb018d29bc99bc3f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-28T00:48:44.668 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:44 smithi177 ceph-mgr[38064]: -- 172.21.15.177:0/2029538577 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563ffed2e5a0 con 0x563ffed02800 2024-03-28T00:48:44.668 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:44 smithi177 bash[38045]: a1b0b6a946b255625c6f21de2e7ecd4d87085aa8f6cdcbecb018d29bc99bc3f7 2024-03-28T00:48:44.668 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:44 smithi177 systemd[1]: Started Ceph mgr.z for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T00:48:44.765 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T00:48:44.765 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T00:48:44.765 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2024-03-28T00:48:44.765 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2024-03-28T00:48:44.765 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113762 s, 922 MB/s 2024-03-28T00:48:44.765 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2024-03-28T00:48:45.328 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:45 smithi169 ceph-mon[32274]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:45.329 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:45 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:45.329 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:45 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:45.329 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:45 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:45.329 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:45 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T00:48:45.329 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:45 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T00:48:45.329 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:45 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T00:48:45.329 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:45 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:45.355 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_4 2024-03-28T00:48:45.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:45 smithi174 ceph-mon[36060]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:45.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:45 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:45.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:45 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:45 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:45 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T00:48:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:45 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T00:48:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:45 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T00:48:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:45 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:45.417 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:45 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:45.025+0000 7fa5e60bd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T00:48:45.417 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:45 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:45.119+0000 7fa5e60bd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T00:48:45.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:45 smithi177 ceph-mon[36206]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:45.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:45 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:45.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:45 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:45.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:45 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:45.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:45 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T00:48:45.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:45 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T00:48:45.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:45 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T00:48:45.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:45 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:45.590 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:48:46.072 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:45 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:45.752+0000 7fa5e60bd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T00:48:46.072 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:45 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:45.844+0000 7fa5e60bd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T00:48:46.072 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:45 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:45.984+0000 7fa5e60bd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T00:48:46.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:46 smithi169 ceph-mon[32274]: Deploying daemon mgr.y on smithi174 2024-03-28T00:48:46.354 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:46 smithi174 ceph-mon[36060]: Deploying daemon mgr.y on smithi174 2024-03-28T00:48:46.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:46 smithi177 ceph-mon[36206]: Deploying daemon mgr.y on smithi174 2024-03-28T00:48:46.417 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:46 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:46.079+0000 7fa5e60bd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T00:48:46.417 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:46 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:46.161+0000 7fa5e60bd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T00:48:46.609 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:46 smithi174 systemd[1]: Starting Ceph mgr.y for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T00:48:46.684 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:46 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:46.422+0000 7fa5e60bd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T00:48:46.684 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:46 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:46.529+0000 7fa5e60bd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T00:48:46.684 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:46 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:46.615+0000 7fa5e60bd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T00:48:46.884 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:46 smithi174 podman[37433]: 2024-03-28 00:48:46.661166045 +0000 UTC m=+0.017916851 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T00:48:46.885 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:46 smithi174 podman[37433]: 2024-03-28 00:48:46.758519068 +0000 UTC m=+0.115269864 container create fbd1ef7452fc0364dda36b7ed13789ff03f3c3a6d573ba088c749e48285dd336 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T00:48:46.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.941+0000 7fc753ca5700 1 -- 172.21.15.169:0/456691557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc74c1021c0 msgr2=0x7fc74c1025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:46.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.941+0000 7fc753ca5700 1 --2- 172.21.15.169:0/456691557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc74c1021c0 0x7fc74c1025e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc7480099c0 tx=0x7fc748009cd0 comp rx=0 tx=0).stop 2024-03-28T00:48:46.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.941+0000 7fc753ca5700 1 -- 172.21.15.169:0/456691557 shutdown_connections 2024-03-28T00:48:46.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.941+0000 7fc753ca5700 1 --2- 172.21.15.169:0/456691557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc74c103d60 0x7fc74c10a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:46.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.941+0000 7fc753ca5700 1 --2- 172.21.15.169:0/456691557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc74c1021c0 0x7fc74c1025e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:46.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.941+0000 7fc753ca5700 1 --2- 172.21.15.169:0/456691557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc74c1033c0 0x7fc74c103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:46.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.941+0000 7fc753ca5700 1 -- 172.21.15.169:0/456691557 >> 172.21.15.169:0/456691557 conn(0x7fc74c0fd7a0 msgr2=0x7fc74c0ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:48:46.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.941+0000 7fc753ca5700 1 -- 172.21.15.169:0/456691557 shutdown_connections 2024-03-28T00:48:46.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.941+0000 7fc753ca5700 1 -- 172.21.15.169:0/456691557 wait complete. 2024-03-28T00:48:46.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.941+0000 7fc753ca5700 1 Processor -- start 2024-03-28T00:48:46.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.941+0000 7fc753ca5700 1 -- start start 2024-03-28T00:48:46.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.941+0000 7fc753ca5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc74c1021c0 0x7fc74c075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:46.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.941+0000 7fc753ca5700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc74c1033c0 0x7fc74c074580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:46.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.941+0000 7fc753ca5700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc74c103d60 0x7fc74c074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:46.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.941+0000 7fc753ca5700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c10ddf0 con 0x7fc74c1021c0 2024-03-28T00:48:46.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.941+0000 7fc753ca5700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c10dc70 con 0x7fc74c1033c0 2024-03-28T00:48:46.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.941+0000 7fc753ca5700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c10df70 con 0x7fc74c103d60 2024-03-28T00:48:46.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.942+0000 7fc751a41700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc74c1021c0 0x7fc74c075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:46.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.942+0000 7fc751240700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc74c1033c0 0x7fc74c074580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:46.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.942+0000 7fc751a41700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc74c1021c0 0x7fc74c075f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48946/0 (socket says 172.21.15.169:48946) 2024-03-28T00:48:46.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.942+0000 7fc751a41700 1 -- 172.21.15.169:0/3300906674 learned_addr learned my addr 172.21.15.169:0/3300906674 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:48:46.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.942+0000 7fc752242700 1 --2- 172.21.15.169:0/3300906674 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc74c103d60 0x7fc74c074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:46.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.942+0000 7fc751a41700 1 -- 172.21.15.169:0/3300906674 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc74c103d60 msgr2=0x7fc74c074ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:46.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.942+0000 7fc751a41700 1 --2- 172.21.15.169:0/3300906674 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc74c103d60 0x7fc74c074ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:46.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.942+0000 7fc751a41700 1 -- 172.21.15.169:0/3300906674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc74c1033c0 msgr2=0x7fc74c074580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:48:46.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.942+0000 7fc751a41700 1 --2- 172.21.15.169:0/3300906674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc74c1033c0 0x7fc74c074580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:48:46.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.942+0000 7fc751a41700 1 -- 172.21.15.169:0/3300906674 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74c075370 con 0x7fc74c1021c0 2024-03-28T00:48:46.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.942+0000 7fc751240700 1 --2- 172.21.15.169:0/3300906674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc74c1033c0 0x7fc74c074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:48:46.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.942+0000 7fc751a41700 1 --2- 172.21.15.169:0/3300906674 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc74c1021c0 0x7fc74c075f30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc74800b580 tx=0x7fc74800bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:48:46.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.942+0000 7fc742ffd700 1 -- 172.21.15.169:0/3300906674 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc748009e60 con 0x7fc74c1021c0 2024-03-28T00:48:46.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.942+0000 7fc742ffd700 1 -- 172.21.15.169:0/3300906674 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc74800f460 con 0x7fc74c1021c0 2024-03-28T00:48:46.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.942+0000 7fc753ca5700 1 -- 172.21.15.169:0/3300906674 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc74c1a2590 con 0x7fc74c1021c0 2024-03-28T00:48:46.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.942+0000 7fc742ffd700 1 -- 172.21.15.169:0/3300906674 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc748017650 con 0x7fc74c1021c0 2024-03-28T00:48:46.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.942+0000 7fc753ca5700 1 -- 172.21.15.169:0/3300906674 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc74c1a29e0 con 0x7fc74c1021c0 2024-03-28T00:48:46.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.943+0000 7fc742ffd700 1 -- 172.21.15.169:0/3300906674 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fc7480177f0 con 0x7fc74c1021c0 2024-03-28T00:48:46.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.943+0000 7fc753ca5700 1 -- 172.21.15.169:0/3300906674 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc74c068e50 con 0x7fc74c1021c0 2024-03-28T00:48:46.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.943+0000 7fc742ffd700 1 --2- 172.21.15.169:0/3300906674 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc738039200 0x7fc73803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:48:46.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.943+0000 7fc742ffd700 1 -- 172.21.15.169:0/3300906674 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc74804d0a0 con 0x7fc74c1021c0 2024-03-28T00:48:46.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.943+0000 7fc751240700 1 --2- 172.21.15.169:0/3300906674 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc738039200 0x7fc73803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:48:46.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.944+0000 7fc751240700 1 --2- 172.21.15.169:0/3300906674 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc738039200 0x7fc73803b6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc73c010420 tx=0x7fc73c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:48:46.951 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:46 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:46.710+0000 7fa5e60bd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T00:48:46.951 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:46 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:46.893+0000 7fa5e60bd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T00:48:46.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:46.948+0000 7fc742ffd700 1 -- 172.21.15.169:0/3300906674 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc748013070 con 0x7fc74c1021c0 2024-03-28T00:48:47.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:47.161+0000 7fc753ca5700 1 -- 172.21.15.169:0/3300906674 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fc74c0625d0 con 0x7fc738039200 2024-03-28T00:48:47.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:47 smithi174 ceph-mon[36060]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:47.359 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:46 smithi174 podman[37433]: 2024-03-28 00:48:46.969307062 +0000 UTC m=+0.326057864 container init fbd1ef7452fc0364dda36b7ed13789ff03f3c3a6d573ba088c749e48285dd336 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T00:48:47.359 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:46 smithi174 podman[37433]: 2024-03-28 00:48:46.97207627 +0000 UTC m=+0.328827062 container start fbd1ef7452fc0364dda36b7ed13789ff03f3c3a6d573ba088c749e48285dd336 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-28T00:48:47.359 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:46 smithi174 ceph-mgr[37452]: -- 172.21.15.174:0/1975040624 <== mon.2 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560ea2ad85a0 con 0x560ea2aacc00 2024-03-28T00:48:47.359 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:47 smithi174 bash[37433]: fbd1ef7452fc0364dda36b7ed13789ff03f3c3a6d573ba088c749e48285dd336 2024-03-28T00:48:47.359 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:47 smithi174 systemd[1]: Started Ceph mgr.y for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T00:48:47.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:47 smithi177 ceph-mon[36206]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:47.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:47 smithi169 ceph-mon[32274]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:47.858 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:47 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:47.524+0000 7f6c6774e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T00:48:47.858 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:47 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:47.621+0000 7f6c6774e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T00:48:48.268 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:47 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:47.961+0000 7fa5e60bd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T00:48:48.268 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:48 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:48.099+0000 7fa5e60bd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T00:48:48.268 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:48 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:48.268 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:48 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:48.268 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:48 smithi177 ceph-mon[36206]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:48:48.268 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:48 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:48:48.269 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:48 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:48:48.269 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:48 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:48.269 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:48 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:48.269 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:48 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:48.269 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:48 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:48:48.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:48 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:48.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:48 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:48.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:48 smithi169 ceph-mon[32274]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:48:48.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:48 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:48:48.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:48 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:48:48.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:48 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:48.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:48 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:48.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:48 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:48.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:48 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:48:48.400 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:48 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:48.303+0000 7f6c6774e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T00:48:48.400 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:48 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:48.396+0000 7f6c6774e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T00:48:48.401 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:48 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:48.401 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:48 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:48.401 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:48 smithi174 ceph-mon[36060]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:48:48.401 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:48 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:48:48.401 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:48 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:48:48.401 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:48 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:48.401 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:48 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:48.401 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:48 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:48.401 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:48 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:48:48.541 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:48 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:48.445+0000 7fa5e60bd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T00:48:48.541 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:48 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:48.537+0000 7fa5e60bd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T00:48:48.650 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:48 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:48.552+0000 7f6c6774e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T00:48:48.651 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:48 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:48.646+0000 7f6c6774e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T00:48:48.917 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:48 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:48.757+0000 7fa5e60bd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T00:48:48.999 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:48 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:48.729+0000 7f6c6774e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T00:48:48.999 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:48 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:48.994+0000 7f6c6774e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T00:48:49.201 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:48 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:48.943+0000 7fa5e60bd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T00:48:49.285 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:49 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:49.103+0000 7f6c6774e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T00:48:49.285 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:49 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:49.187+0000 7f6c6774e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T00:48:49.285 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:49 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:49.280+0000 7f6c6774e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T00:48:49.285 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:49 smithi174 ceph-mon[36060]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:49.285 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:49 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:49.285 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:49 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:49.285 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:49 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:49.285 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:49 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:49.286 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:49 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:49.286 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:49 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:48:49.286 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:49 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:49.286 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:49 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T00:48:49.286 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:49 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T00:48:49.286 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:49 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:49.493 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:49 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:49.228+0000 7fa5e60bd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T00:48:49.493 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:49 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:49.489+0000 7fa5e60bd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T00:48:49.493 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:49 smithi177 ceph-mon[36206]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:49.493 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:49 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:49.493 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:49 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:49.493 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:49 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:49.493 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:49 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:49.493 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:49 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:49.493 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:49 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:48:49.493 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:49 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:49.494 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:49 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T00:48:49.494 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:49 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T00:48:49.494 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:49 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:49.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:49 smithi169 ceph-mon[32274]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:49.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:49 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:49.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:49 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:49.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:49 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:49.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:49 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:49.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:49 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:49.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:49 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:48:49.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:49 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:49.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:49 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T00:48:49.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:49 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T00:48:49.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:49 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:49.608 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:49 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:49.457+0000 7f6c6774e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T00:48:49.917 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:49 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:49.724+0000 7fa5e60bd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T00:48:50.531 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:50 smithi177 ceph-mon[36206]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T00:48:50.531 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:50 smithi177 ceph-mon[36206]: Reconfiguring daemon mgr.x on smithi169 2024-03-28T00:48:50.531 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:50 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3317743438' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37a188e7-9de4-4f22-a2d2-f524c64e1923"}]: dispatch 2024-03-28T00:48:50.531 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:50 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3317743438' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37a188e7-9de4-4f22-a2d2-f524c64e1923"}]': finished 2024-03-28T00:48:50.531 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:50 smithi177 ceph-mon[36206]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T00:48:50.531 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:50 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T00:48:50.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:50 smithi169 ceph-mon[32274]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T00:48:50.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:50 smithi169 ceph-mon[32274]: Reconfiguring daemon mgr.x on smithi169 2024-03-28T00:48:50.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:50 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3317743438' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37a188e7-9de4-4f22-a2d2-f524c64e1923"}]: dispatch 2024-03-28T00:48:50.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:50 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3317743438' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37a188e7-9de4-4f22-a2d2-f524c64e1923"}]': finished 2024-03-28T00:48:50.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:50 smithi169 ceph-mon[32274]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T00:48:50.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:50 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T00:48:50.553 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:50 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:50.419+0000 7f6c6774e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T00:48:50.553 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:50 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:50.548+0000 7f6c6774e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T00:48:50.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:50 smithi174 ceph-mon[36060]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T00:48:50.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:50 smithi174 ceph-mon[36060]: Reconfiguring daemon mgr.x on smithi169 2024-03-28T00:48:50.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:50 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3317743438' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37a188e7-9de4-4f22-a2d2-f524c64e1923"}]: dispatch 2024-03-28T00:48:50.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:50 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3317743438' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37a188e7-9de4-4f22-a2d2-f524c64e1923"}]': finished 2024-03-28T00:48:50.554 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:50 smithi174 ceph-mon[36060]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T00:48:50.554 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:50 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T00:48:50.917 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:50 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:50.527+0000 7fa5e60bd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T00:48:51.174 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:50 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:50.890+0000 7f6c6774e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T00:48:51.174 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:50 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:50.978+0000 7f6c6774e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T00:48:51.174 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:51 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:51.169+0000 7f6c6774e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T00:48:51.415 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:51 smithi177 ceph-mon[36206]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:51.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:51 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/495503663' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:48:51.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:51 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:51.416 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:51 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:51.097+0000 7fa5e60bd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T00:48:51.416 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:51 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:51.194+0000 7fa5e60bd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T00:48:51.416 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 00:48:51 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z[38060]: 2024-03-28T00:48:51.411+0000 7fa5e60bd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T00:48:51.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:51 smithi169 ceph-mon[32274]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:51.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:51 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/495503663' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:48:51.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:51 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:51.608 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:51 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:51.353+0000 7f6c6774e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T00:48:51.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:51 smithi174 ceph-mon[36060]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:51.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:51 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/495503663' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:48:51.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:51 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:51.895 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:51 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:51.630+0000 7f6c6774e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T00:48:51.895 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:51 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:51.891+0000 7f6c6774e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T00:48:52.206 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:52 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:52.125+0000 7f6c6774e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T00:48:52.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:52.212+0000 7fc742ffd700 1 -- 172.21.15.169:0/3300906674 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mgrmap(e 14) v1 ==== 91699+0+0 (secure 0 0 0) 0x7fc748021cc0 con 0x7fc74c1021c0 2024-03-28T00:48:52.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:52 smithi169 ceph-mon[32274]: Standby manager daemon z started 2024-03-28T00:48:52.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:52 smithi169 ceph-mon[32274]: from='mgr.? 172.21.15.177:0/149627457' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T00:48:52.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:52 smithi169 ceph-mon[32274]: from='mgr.? 172.21.15.177:0/149627457' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T00:48:52.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:52 smithi169 ceph-mon[32274]: from='mgr.? 172.21.15.177:0/149627457' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T00:48:52.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:52 smithi169 ceph-mon[32274]: from='mgr.? 172.21.15.177:0/149627457' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T00:48:52.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:52 smithi169 ceph-mon[32274]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:52.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:52 smithi174 ceph-mon[36060]: Standby manager daemon z started 2024-03-28T00:48:52.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:52 smithi174 ceph-mon[36060]: from='mgr.? 172.21.15.177:0/149627457' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T00:48:52.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:52 smithi174 ceph-mon[36060]: from='mgr.? 172.21.15.177:0/149627457' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T00:48:52.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:52 smithi174 ceph-mon[36060]: from='mgr.? 172.21.15.177:0/149627457' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T00:48:52.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:52 smithi174 ceph-mon[36060]: from='mgr.? 172.21.15.177:0/149627457' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T00:48:52.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:52 smithi174 ceph-mon[36060]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:52.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:52 smithi177 ceph-mon[36206]: Standby manager daemon z started 2024-03-28T00:48:52.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:52 smithi177 ceph-mon[36206]: from='mgr.? 172.21.15.177:0/149627457' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T00:48:52.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:52 smithi177 ceph-mon[36206]: from='mgr.? 172.21.15.177:0/149627457' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T00:48:52.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:52 smithi177 ceph-mon[36206]: from='mgr.? 172.21.15.177:0/149627457' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T00:48:52.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:52 smithi177 ceph-mon[36206]: from='mgr.? 172.21.15.177:0/149627457' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T00:48:52.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:52 smithi177 ceph-mon[36206]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:53.214 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:52 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:52.916+0000 7f6c6774e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T00:48:53.486 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:53 smithi174 ceph-mon[36060]: mgrmap e14: x(active, since 76s), standbys: z 2024-03-28T00:48:53.486 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:53 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T00:48:53.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:53 smithi169 ceph-mon[32274]: mgrmap e14: x(active, since 76s), standbys: z 2024-03-28T00:48:53.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:53 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T00:48:53.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:53 smithi177 ceph-mon[36206]: mgrmap e14: x(active, since 76s), standbys: z 2024-03-28T00:48:53.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:53 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T00:48:53.793 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:53 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:53.482+0000 7f6c6774e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T00:48:53.794 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:53 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:53.577+0000 7f6c6774e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T00:48:53.794 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 00:48:53 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y[37448]: 2024-03-28T00:48:53.789+0000 7f6c6774e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T00:48:54.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:48:54.223+0000 7fc742ffd700 1 -- 172.21.15.169:0/3300906674 <== mon.0 v2:172.21.15.169:3300/0 8 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7fc7480163a0 con 0x7fc74c1021c0 2024-03-28T00:48:54.370 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:54 smithi169 ceph-mon[32274]: Standby manager daemon y started 2024-03-28T00:48:54.370 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:54 smithi169 ceph-mon[32274]: from='mgr.? 172.21.15.174:0/3675889950' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T00:48:54.370 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:54 smithi169 ceph-mon[32274]: from='mgr.? 172.21.15.174:0/3675889950' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T00:48:54.370 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:54 smithi169 ceph-mon[32274]: from='mgr.? 172.21.15.174:0/3675889950' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T00:48:54.370 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:54 smithi169 ceph-mon[32274]: from='mgr.? 172.21.15.174:0/3675889950' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T00:48:54.370 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:54 smithi169 ceph-mon[32274]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:54.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:54 smithi174 ceph-mon[36060]: Standby manager daemon y started 2024-03-28T00:48:54.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:54 smithi174 ceph-mon[36060]: from='mgr.? 172.21.15.174:0/3675889950' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T00:48:54.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:54 smithi174 ceph-mon[36060]: from='mgr.? 172.21.15.174:0/3675889950' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T00:48:54.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:54 smithi174 ceph-mon[36060]: from='mgr.? 172.21.15.174:0/3675889950' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T00:48:54.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:54 smithi174 ceph-mon[36060]: from='mgr.? 172.21.15.174:0/3675889950' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T00:48:54.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:54 smithi174 ceph-mon[36060]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:54.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:54 smithi177 ceph-mon[36206]: Standby manager daemon y started 2024-03-28T00:48:54.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:54 smithi177 ceph-mon[36206]: from='mgr.? 172.21.15.174:0/3675889950' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T00:48:54.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:54 smithi177 ceph-mon[36206]: from='mgr.? 172.21.15.174:0/3675889950' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T00:48:54.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:54 smithi177 ceph-mon[36206]: from='mgr.? 172.21.15.174:0/3675889950' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T00:48:54.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:54 smithi177 ceph-mon[36206]: from='mgr.? 172.21.15.174:0/3675889950' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T00:48:54.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:54 smithi177 ceph-mon[36206]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:55.487 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:55 smithi169 ceph-mon[32274]: mgrmap e15: x(active, since 78s), standbys: z, y 2024-03-28T00:48:55.488 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:55 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T00:48:55.488 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:55 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:55.488 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:55 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:55.488 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:55 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:48:55.488 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:55 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:55.488 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:55 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:48:55.488 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:55 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:55.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:55 smithi174 ceph-mon[36060]: mgrmap e15: x(active, since 78s), standbys: z, y 2024-03-28T00:48:55.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:55 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T00:48:55.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:55 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:55.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:55 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:55.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:55 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:48:55.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:55 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:55.610 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:55 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:48:55.610 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:55 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:55.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:55 smithi177 ceph-mon[36206]: mgrmap e15: x(active, since 78s), standbys: z, y 2024-03-28T00:48:55.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:55 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T00:48:55.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:55 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:55.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:55 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:55.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:55 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:48:55.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:55 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:55.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:55 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:48:55.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:55 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:48:56.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:56 smithi174 ceph-mon[36060]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:56.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:56 smithi169 ceph-mon[32274]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:56.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:56 smithi177 ceph-mon[36206]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:57.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:57 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T00:48:57.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:57 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:57.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:57 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T00:48:57.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:57 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:57.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:57 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T00:48:57.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:57 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:48:58.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:58 smithi169 ceph-mon[32274]: Deploying daemon osd.0 on smithi169 2024-03-28T00:48:58.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:48:58 smithi169 ceph-mon[32274]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:58.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:58 smithi177 ceph-mon[36206]: Deploying daemon osd.0 on smithi169 2024-03-28T00:48:58.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:48:58 smithi177 ceph-mon[36206]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:48:58.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:58 smithi174 ceph-mon[36060]: Deploying daemon osd.0 on smithi169 2024-03-28T00:48:58.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:48:58 smithi174 ceph-mon[36060]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:49:01.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:01 smithi169 ceph-mon[32274]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:49:01.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:01 smithi174 ceph-mon[36060]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:49:01.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:01 smithi177 ceph-mon[36206]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:49:03.249 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:03 smithi169 ceph-mon[32274]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:49:03.250 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:03 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:03.250 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:03 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:03.250 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:03 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:49:03.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:03 smithi174 ceph-mon[36060]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:49:03.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:03 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:03.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:03 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:03.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:03 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:49:03.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:03 smithi177 ceph-mon[36206]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:49:03.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:03 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:03.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:03 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:03.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:03 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:49:05.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:05 smithi169 ceph-mon[32274]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:49:05.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:05 smithi174 ceph-mon[36060]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:49:05.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:05 smithi177 ceph-mon[36206]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:49:05.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:05.924+0000 7fc742ffd700 1 -- 172.21.15.169:0/3300906674 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc74c0625d0 con 0x7fc738039200 2024-03-28T00:49:05.929 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 0 on host 'smithi169' 2024-03-28T00:49:05.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:05.927+0000 7fc753ca5700 1 -- 172.21.15.169:0/3300906674 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc738039200 msgr2=0x7fc73803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:49:05.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:05.927+0000 7fc753ca5700 1 --2- 172.21.15.169:0/3300906674 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc738039200 0x7fc73803b6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc73c010420 tx=0x7fc73c0073d0 comp rx=0 tx=0).stop 2024-03-28T00:49:05.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:05.927+0000 7fc753ca5700 1 -- 172.21.15.169:0/3300906674 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc74c1021c0 msgr2=0x7fc74c075f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:49:05.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:05.927+0000 7fc753ca5700 1 --2- 172.21.15.169:0/3300906674 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc74c1021c0 0x7fc74c075f30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc74800b580 tx=0x7fc74800bee0 comp rx=0 tx=0).stop 2024-03-28T00:49:05.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:05.927+0000 7fc753ca5700 1 -- 172.21.15.169:0/3300906674 shutdown_connections 2024-03-28T00:49:05.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:05.927+0000 7fc753ca5700 1 --2- 172.21.15.169:0/3300906674 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc738039200 0x7fc73803b6c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:05.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:05.927+0000 7fc753ca5700 1 --2- 172.21.15.169:0/3300906674 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc74c103d60 0x7fc74c074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:05.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:05.927+0000 7fc753ca5700 1 --2- 172.21.15.169:0/3300906674 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc74c1021c0 0x7fc74c075f30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:05.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:05.927+0000 7fc753ca5700 1 --2- 172.21.15.169:0/3300906674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc74c1033c0 0x7fc74c074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:05.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:05.927+0000 7fc753ca5700 1 -- 172.21.15.169:0/3300906674 >> 172.21.15.169:0/3300906674 conn(0x7fc74c0fd7a0 msgr2=0x7fc74c1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:49:05.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:05.927+0000 7fc753ca5700 1 -- 172.21.15.169:0/3300906674 shutdown_connections 2024-03-28T00:49:05.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:05.927+0000 7fc753ca5700 1 -- 172.21.15.169:0/3300906674 wait complete. 2024-03-28T00:49:06.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:06 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:06.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:06 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:06.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:06 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:06.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:06 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:49:06.278 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:06 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:06.278 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:06 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:06.278 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:06 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:06.278 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:06 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:49:06.594 DEBUG:teuthology.orchestra.run.smithi169:osd.0> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.0.service 2024-03-28T00:49:06.596 INFO:tasks.cephadm:Deploying osd.1 on smithi174 with /dev/vg_nvme/lv_4... 2024-03-28T00:49:06.596 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T00:49:06.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:06 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:06.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:06 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:06.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:06 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:06.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:06 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:49:06.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:06 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:06.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:06 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:06.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:06 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:06.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:06 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:49:06.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:06 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:06.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:06 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:06.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:06 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:06.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:06 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:49:06.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:06 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:06.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:06 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:06.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:06 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:06.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:06 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:49:06.838 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T00:49:07.251 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:07 smithi174 ceph-mon[36060]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:49:07.251 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:07 smithi174 ceph-mon[36060]: from='osd.0 [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T00:49:07.476 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:07 smithi169 ceph-mon[32274]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:49:07.476 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:07 smithi169 ceph-mon[32274]: from='osd.0 [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T00:49:07.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:07 smithi177 ceph-mon[36206]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:49:07.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:07 smithi177 ceph-mon[36206]: from='osd.0 [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T00:49:08.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:08 smithi174 ceph-mon[36060]: from='osd.0 [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T00:49:08.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:08 smithi174 ceph-mon[36060]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T00:49:08.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:08 smithi174 ceph-mon[36060]: from='osd.0 [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-28T00:49:08.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:08 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T00:49:08.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:08 smithi174 ceph-mon[36060]: from='osd.0 [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-28T00:49:08.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:08 smithi174 ceph-mon[36060]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T00:49:08.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:08 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T00:49:08.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:08 smithi169 ceph-mon[32274]: from='osd.0 [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T00:49:08.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:08 smithi169 ceph-mon[32274]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T00:49:08.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:08 smithi169 ceph-mon[32274]: from='osd.0 [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-28T00:49:08.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:08 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T00:49:08.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:08 smithi169 ceph-mon[32274]: from='osd.0 [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-28T00:49:08.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:08 smithi169 ceph-mon[32274]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T00:49:08.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:08 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T00:49:08.655 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T00:49:08.655 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T00:49:08.656 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-03-28T00:49:08.656 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-03-28T00:49:08.656 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113306 s, 925 MB/s 2024-03-28T00:49:08.656 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-03-28T00:49:08.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:08 smithi177 ceph-mon[36206]: from='osd.0 [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T00:49:08.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:08 smithi177 ceph-mon[36206]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T00:49:08.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:08 smithi177 ceph-mon[36206]: from='osd.0 [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-28T00:49:08.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:08 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T00:49:08.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:08 smithi177 ceph-mon[36206]: from='osd.0 [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-28T00:49:08.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:08 smithi177 ceph-mon[36206]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T00:49:08.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:08 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T00:49:08.943 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 00:49:08 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-0[44478]: 2024-03-28T00:49:08.613+0000 7f7e2cc2e700 -1 osd.0 0 waiting for initial osdmap 2024-03-28T00:49:08.944 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 00:49:08 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-0[44478]: 2024-03-28T00:49:08.617+0000 7f7e24a19700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T00:49:09.226 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:09 smithi174 ceph-mon[36060]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:49:09.226 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:09 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T00:49:09.226 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:09 smithi174 ceph-mon[36060]: from='osd.0 [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737]' entity='osd.0' 2024-03-28T00:49:09.226 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:09 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T00:49:09.233 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:09 smithi169 ceph-mon[32274]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:49:09.233 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:09 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T00:49:09.234 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:09 smithi169 ceph-mon[32274]: from='osd.0 [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737]' entity='osd.0' 2024-03-28T00:49:09.234 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:09 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T00:49:09.253 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi174:vg_nvme/lv_4 2024-03-28T00:49:09.541 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T00:49:09.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:09 smithi177 ceph-mon[36206]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:49:09.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:09 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T00:49:09.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:09 smithi177 ceph-mon[36206]: from='osd.0 [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737]' entity='osd.0' 2024-03-28T00:49:09.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:09 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T00:49:10.478 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:10 smithi174 ceph-mon[36060]: purged_snaps scrub starts 2024-03-28T00:49:10.478 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:10 smithi174 ceph-mon[36060]: purged_snaps scrub ok 2024-03-28T00:49:10.478 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:10.478 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:10.479 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:49:10.479 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:10.479 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:10.479 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:49:10.479 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:10.479 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:10 smithi174 ceph-mon[36060]: osd.0 [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737] boot 2024-03-28T00:49:10.479 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:10 smithi174 ceph-mon[36060]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T00:49:10.479 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T00:49:10.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:10 smithi169 ceph-mon[32274]: purged_snaps scrub starts 2024-03-28T00:49:10.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:10 smithi169 ceph-mon[32274]: purged_snaps scrub ok 2024-03-28T00:49:10.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:10.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:10.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:49:10.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:10.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:10.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:49:10.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:10.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:10 smithi169 ceph-mon[32274]: osd.0 [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737] boot 2024-03-28T00:49:10.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:10 smithi169 ceph-mon[32274]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T00:49:10.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T00:49:10.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:10 smithi177 ceph-mon[36206]: purged_snaps scrub starts 2024-03-28T00:49:10.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:10 smithi177 ceph-mon[36206]: purged_snaps scrub ok 2024-03-28T00:49:10.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:10.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:10.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:49:10.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:10.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:10.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:49:10.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:10.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:10 smithi177 ceph-mon[36206]: osd.0 [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737] boot 2024-03-28T00:49:10.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:10 smithi177 ceph-mon[36206]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T00:49:10.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T00:49:11.106 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.100+0000 7fdb0314d700 1 -- 172.21.15.174:0/1247332816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdafc1021b0 msgr2=0x7fdafc1025d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:49:11.106 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.100+0000 7fdb0314d700 1 --2- 172.21.15.174:0/1247332816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdafc1021b0 0x7fdafc1025d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdaf00099c0 tx=0x7fdaf0009cd0 comp rx=0 tx=0).stop 2024-03-28T00:49:11.106 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.100+0000 7fdb0314d700 1 -- 172.21.15.174:0/1247332816 shutdown_connections 2024-03-28T00:49:11.106 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.100+0000 7fdb0314d700 1 --2- 172.21.15.174:0/1247332816 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdafc103d50 0x7fdafc10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:11.106 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.100+0000 7fdb0314d700 1 --2- 172.21.15.174:0/1247332816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdafc1021b0 0x7fdafc1025d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:11.106 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.100+0000 7fdb0314d700 1 --2- 172.21.15.174:0/1247332816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdafc1033b0 0x7fdafc103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:11.106 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.100+0000 7fdb0314d700 1 -- 172.21.15.174:0/1247332816 >> 172.21.15.174:0/1247332816 conn(0x7fdafc0fd770 msgr2=0x7fdafc0ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:49:11.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.100+0000 7fdb0314d700 1 -- 172.21.15.174:0/1247332816 shutdown_connections 2024-03-28T00:49:11.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.100+0000 7fdb0314d700 1 -- 172.21.15.174:0/1247332816 wait complete. 2024-03-28T00:49:11.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.100+0000 7fdb0314d700 1 Processor -- start 2024-03-28T00:49:11.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.100+0000 7fdb0314d700 1 -- start start 2024-03-28T00:49:11.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.101+0000 7fdb0314d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdafc1021b0 0x7fdafc1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:49:11.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.101+0000 7fdb0314d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdafc1033b0 0x7fdafc1989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:49:11.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.101+0000 7fdb0314d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdafc103d50 0x7fdafc19fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:49:11.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.101+0000 7fdb0314d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc10de00 con 0x7fdafc1033b0 2024-03-28T00:49:11.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.101+0000 7fdb0314d700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc10dc80 con 0x7fdafc103d50 2024-03-28T00:49:11.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.101+0000 7fdb0314d700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc10df80 con 0x7fdafc1021b0 2024-03-28T00:49:11.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.101+0000 7fdb016ea700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdafc103d50 0x7fdafc19fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:49:11.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.101+0000 7fdb016ea700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdafc103d50 0x7fdafc19fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:47598/0 (socket says 172.21.15.174:47598) 2024-03-28T00:49:11.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.101+0000 7fdb016ea700 1 -- 172.21.15.174:0/1636521858 learned_addr learned my addr 172.21.15.174:0/1636521858 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-28T00:49:11.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.101+0000 7fdb00ee9700 1 --2- 172.21.15.174:0/1636521858 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdafc1021b0 0x7fdafc1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:49:11.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.101+0000 7fdafbfff700 1 --2- 172.21.15.174:0/1636521858 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdafc1033b0 0x7fdafc1989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:49:11.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.101+0000 7fdb016ea700 1 -- 172.21.15.174:0/1636521858 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdafc1021b0 msgr2=0x7fdafc1984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:49:11.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.101+0000 7fdb016ea700 1 --2- 172.21.15.174:0/1636521858 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdafc1021b0 0x7fdafc1984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:11.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.101+0000 7fdb016ea700 1 -- 172.21.15.174:0/1636521858 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdafc1033b0 msgr2=0x7fdafc1989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:49:11.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.101+0000 7fdb016ea700 1 --2- 172.21.15.174:0/1636521858 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdafc1033b0 0x7fdafc1989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:11.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.101+0000 7fdb016ea700 1 -- 172.21.15.174:0/1636521858 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdafc1a0100 con 0x7fdafc103d50 2024-03-28T00:49:11.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.101+0000 7fdb016ea700 1 --2- 172.21.15.174:0/1636521858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdafc103d50 0x7fdafc19fa60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdaf400ada0 tx=0x7fdaf400c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:49:11.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.101+0000 7fdaf9ffb700 1 -- 172.21.15.174:0/1636521858 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf4017020 con 0x7fdafc103d50 2024-03-28T00:49:11.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.101+0000 7fdaf9ffb700 1 -- 172.21.15.174:0/1636521858 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdaf400a0d0 con 0x7fdafc103d50 2024-03-28T00:49:11.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.101+0000 7fdb0314d700 1 -- 172.21.15.174:0/1636521858 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdafc1a03f0 con 0x7fdafc103d50 2024-03-28T00:49:11.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.101+0000 7fdaf9ffb700 1 -- 172.21.15.174:0/1636521858 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf4005480 con 0x7fdafc103d50 2024-03-28T00:49:11.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.102+0000 7fdb0314d700 1 -- 172.21.15.174:0/1636521858 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdafc1a08e0 con 0x7fdafc103d50 2024-03-28T00:49:11.109 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.102+0000 7fdb0314d700 1 -- 172.21.15.174:0/1636521858 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdafc068e50 con 0x7fdafc103d50 2024-03-28T00:49:11.109 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.103+0000 7fdaf9ffb700 1 -- 172.21.15.174:0/1636521858 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7fdaf4005620 con 0x7fdafc103d50 2024-03-28T00:49:11.109 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.104+0000 7fdaf9ffb700 1 --2- 172.21.15.174:0/1636521858 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdaec0a2800 0x7fdaec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:49:11.110 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.104+0000 7fdaf9ffb700 1 -- 172.21.15.174:0/1636521858 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1767+0+0 (secure 0 0 0) 0x7fdaf40cc750 con 0x7fdafc103d50 2024-03-28T00:49:11.110 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.105+0000 7fdb00ee9700 1 --2- 172.21.15.174:0/1636521858 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdaec0a2800 0x7fdaec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:49:11.110 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.105+0000 7fdb00ee9700 1 --2- 172.21.15.174:0/1636521858 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdaec0a2800 0x7fdaec0a4cc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdaf000b580 tx=0x7fdaf0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:49:11.113 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.109+0000 7fdaf9ffb700 1 -- 172.21.15.174:0/1636521858 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdaf4021020 con 0x7fdafc103d50 2024-03-28T00:49:11.317 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:11.312+0000 7fdb0314d700 1 -- 172.21.15.174:0/1636521858 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fdafc0625d0 con 0x7fdaec0a2800 2024-03-28T00:49:11.317 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:11 smithi174 ceph-mon[36060]: Detected new or changed devices on smithi169 2024-03-28T00:49:11.317 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:11 smithi174 ceph-mon[36060]: Adjusting osd_memory_target on smithi169 to 17088M 2024-03-28T00:49:11.317 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:11 smithi174 ceph-mon[36060]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:49:11.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:11 smithi169 ceph-mon[32274]: Detected new or changed devices on smithi169 2024-03-28T00:49:11.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:11 smithi169 ceph-mon[32274]: Adjusting osd_memory_target on smithi169 to 17088M 2024-03-28T00:49:11.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:11 smithi169 ceph-mon[32274]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:49:11.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:11 smithi177 ceph-mon[36206]: Detected new or changed devices on smithi169 2024-03-28T00:49:11.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:11 smithi177 ceph-mon[36206]: Adjusting osd_memory_target on smithi169 to 17088M 2024-03-28T00:49:11.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:11 smithi177 ceph-mon[36206]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T00:49:12.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:12 smithi169 ceph-mon[32274]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T00:49:12.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:12 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:49:12.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:12 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:49:12.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:12 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:12.556 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:12 smithi174 ceph-mon[36060]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T00:49:12.556 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:12 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:49:12.556 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:12 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:49:12.556 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:12 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:12.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:12 smithi177 ceph-mon[36206]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T00:49:12.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:12 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:49:12.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:12 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:49:12.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:12 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:13.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:13 smithi174 ceph-mon[36060]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:49:13.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:13 smithi174 ceph-mon[36060]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:13.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:13 smithi169 ceph-mon[32274]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:49:13.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:13 smithi169 ceph-mon[32274]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:13.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:13 smithi177 ceph-mon[36206]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:49:13.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:13 smithi177 ceph-mon[36206]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:14.463 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:14 smithi169 ceph-mon[32274]: from='client.? 172.21.15.174:0/1156775927' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "269d9b9a-b3cd-4dda-bc9e-7d980cad5d12"}]: dispatch 2024-03-28T00:49:14.463 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:14 smithi169 ceph-mon[32274]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "269d9b9a-b3cd-4dda-bc9e-7d980cad5d12"}]: dispatch 2024-03-28T00:49:14.463 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:14 smithi169 ceph-mon[32274]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "269d9b9a-b3cd-4dda-bc9e-7d980cad5d12"}]': finished 2024-03-28T00:49:14.463 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:14 smithi169 ceph-mon[32274]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T00:49:14.463 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:14 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T00:49:14.463 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:14 smithi169 ceph-mon[32274]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:14.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:14 smithi174 ceph-mon[36060]: from='client.? 172.21.15.174:0/1156775927' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "269d9b9a-b3cd-4dda-bc9e-7d980cad5d12"}]: dispatch 2024-03-28T00:49:14.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:14 smithi174 ceph-mon[36060]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "269d9b9a-b3cd-4dda-bc9e-7d980cad5d12"}]: dispatch 2024-03-28T00:49:14.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:14 smithi174 ceph-mon[36060]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "269d9b9a-b3cd-4dda-bc9e-7d980cad5d12"}]': finished 2024-03-28T00:49:14.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:14 smithi174 ceph-mon[36060]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T00:49:14.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:14 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T00:49:14.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:14 smithi174 ceph-mon[36060]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:14.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:14 smithi177 ceph-mon[36206]: from='client.? 172.21.15.174:0/1156775927' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "269d9b9a-b3cd-4dda-bc9e-7d980cad5d12"}]: dispatch 2024-03-28T00:49:14.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:14 smithi177 ceph-mon[36206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "269d9b9a-b3cd-4dda-bc9e-7d980cad5d12"}]: dispatch 2024-03-28T00:49:14.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:14 smithi177 ceph-mon[36206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "269d9b9a-b3cd-4dda-bc9e-7d980cad5d12"}]': finished 2024-03-28T00:49:14.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:14 smithi177 ceph-mon[36206]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T00:49:14.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:14 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T00:49:14.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:14 smithi177 ceph-mon[36206]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:15.440 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:15 smithi177 ceph-mon[36206]: from='client.? 172.21.15.174:0/731867590' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:49:15.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:15 smithi174 ceph-mon[36060]: from='client.? 172.21.15.174:0/731867590' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:49:15.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:15 smithi169 ceph-mon[32274]: from='client.? 172.21.15.174:0/731867590' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:49:16.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:16 smithi169 ceph-mon[32274]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:16.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:16 smithi174 ceph-mon[36060]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:16.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:16 smithi177 ceph-mon[36206]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:19.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:19 smithi174 ceph-mon[36060]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:19.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:19 smithi177 ceph-mon[36206]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:19.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:19 smithi169 ceph-mon[32274]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:21.334 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:21 smithi174 ceph-mon[36060]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:21.335 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:21 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T00:49:21.335 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:21 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:21.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:21 smithi177 ceph-mon[36206]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:21.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:21 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T00:49:21.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:21 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:21.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:21 smithi169 ceph-mon[32274]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:21.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:21 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T00:49:21.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:21 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:22.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:22 smithi174 ceph-mon[36060]: Deploying daemon osd.1 on smithi174 2024-03-28T00:49:22.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:22 smithi177 ceph-mon[36206]: Deploying daemon osd.1 on smithi174 2024-03-28T00:49:22.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:22 smithi169 ceph-mon[32274]: Deploying daemon osd.1 on smithi174 2024-03-28T00:49:23.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:23 smithi174 ceph-mon[36060]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:23.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:23 smithi177 ceph-mon[36206]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:23.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:23 smithi169 ceph-mon[32274]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:25.335 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:25 smithi174 ceph-mon[36060]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:25.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:25 smithi177 ceph-mon[36206]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:25.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:25 smithi169 ceph-mon[32274]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:26.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:26 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:49:26.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:26 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:26.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:26 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:26.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:26 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:49:26.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:26 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:26.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:26 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:26.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:26 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:49:26.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:26 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:26.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:26 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:27.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:27 smithi174 ceph-mon[36060]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:27.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:27 smithi177 ceph-mon[36206]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:27.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:27 smithi169 ceph-mon[32274]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:29.128 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:29.125+0000 7fdaf9ffb700 1 -- 172.21.15.174:0/1636521858 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fdafc0625d0 con 0x7fdaec0a2800 2024-03-28T00:49:29.128 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 1 on host 'smithi174' 2024-03-28T00:49:29.130 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:29.127+0000 7fdb0314d700 1 -- 172.21.15.174:0/1636521858 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdaec0a2800 msgr2=0x7fdaec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:49:29.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:29.127+0000 7fdb0314d700 1 --2- 172.21.15.174:0/1636521858 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdaec0a2800 0x7fdaec0a4cc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdaf000b580 tx=0x7fdaf0005fb0 comp rx=0 tx=0).stop 2024-03-28T00:49:29.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:29.127+0000 7fdb0314d700 1 -- 172.21.15.174:0/1636521858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdafc103d50 msgr2=0x7fdafc19fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:49:29.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:29.127+0000 7fdb0314d700 1 --2- 172.21.15.174:0/1636521858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdafc103d50 0x7fdafc19fa60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdaf400ada0 tx=0x7fdaf400c330 comp rx=0 tx=0).stop 2024-03-28T00:49:29.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:29.127+0000 7fdb0314d700 1 -- 172.21.15.174:0/1636521858 shutdown_connections 2024-03-28T00:49:29.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:29.127+0000 7fdb0314d700 1 --2- 172.21.15.174:0/1636521858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdafc103d50 0x7fdafc19fa60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:29.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:29.127+0000 7fdb0314d700 1 --2- 172.21.15.174:0/1636521858 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdaec0a2800 0x7fdaec0a4cc0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:29.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:29.127+0000 7fdb0314d700 1 --2- 172.21.15.174:0/1636521858 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdafc1021b0 0x7fdafc1984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:29.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:29.127+0000 7fdb0314d700 1 --2- 172.21.15.174:0/1636521858 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdafc1033b0 0x7fdafc1989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:29.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:29.127+0000 7fdb0314d700 1 -- 172.21.15.174:0/1636521858 >> 172.21.15.174:0/1636521858 conn(0x7fdafc0fd770 msgr2=0x7fdafc106700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:49:29.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:29.128+0000 7fdb0314d700 1 -- 172.21.15.174:0/1636521858 shutdown_connections 2024-03-28T00:49:29.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:49:29.128+0000 7fdb0314d700 1 -- 172.21.15.174:0/1636521858 wait complete. 2024-03-28T00:49:29.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:29 smithi174 ceph-mon[36060]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:29.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:29 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:29.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:29 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:29.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:29 smithi177 ceph-mon[36206]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:29.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:29 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:29.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:29 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:29.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:29 smithi169 ceph-mon[32274]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:29.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:29 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:29.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:29 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:29.825 DEBUG:teuthology.orchestra.run.smithi174:osd.1> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.1.service 2024-03-28T00:49:29.827 INFO:tasks.cephadm:Deploying osd.2 on smithi177 with /dev/vg_nvme/lv_4... 2024-03-28T00:49:29.828 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T00:49:30.071 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T00:49:30.778 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:30 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:30.778 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:30 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:30.778 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:30 smithi174 ceph-mon[36060]: from='osd.1 [v2:172.21.15.174:6800/3333696361,v1:172.21.15.174:6801/3333696361]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T00:49:30.778 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:30 smithi174 ceph-mon[36060]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T00:49:30.779 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:30 smithi174 ceph-mon[36060]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:30.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:30 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:30.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:30 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:30.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:30 smithi169 ceph-mon[32274]: from='osd.1 [v2:172.21.15.174:6800/3333696361,v1:172.21.15.174:6801/3333696361]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T00:49:30.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:30 smithi169 ceph-mon[32274]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T00:49:30.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:30 smithi169 ceph-mon[32274]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:30.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:30 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:30.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:30 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:30.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:30 smithi177 ceph-mon[36206]: from='osd.1 [v2:172.21.15.174:6800/3333696361,v1:172.21.15.174:6801/3333696361]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T00:49:30.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:30 smithi177 ceph-mon[36206]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T00:49:30.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:30 smithi177 ceph-mon[36206]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:31.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:31 smithi177 ceph-mon[36206]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T00:49:31.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:31 smithi177 ceph-mon[36206]: from='osd.1 [v2:172.21.15.174:6800/3333696361,v1:172.21.15.174:6801/3333696361]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:49:31.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:31 smithi177 ceph-mon[36206]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T00:49:31.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:31 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T00:49:31.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:31 smithi177 ceph-mon[36206]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:49:31.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:31 smithi169 ceph-mon[32274]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T00:49:31.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:31 smithi169 ceph-mon[32274]: from='osd.1 [v2:172.21.15.174:6800/3333696361,v1:172.21.15.174:6801/3333696361]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:49:31.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:31 smithi169 ceph-mon[32274]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T00:49:31.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:31 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T00:49:31.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:31 smithi169 ceph-mon[32274]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:49:31.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:31 smithi174 ceph-mon[36060]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T00:49:31.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:31 smithi174 ceph-mon[36060]: from='osd.1 [v2:172.21.15.174:6800/3333696361,v1:172.21.15.174:6801/3333696361]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:49:31.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:31 smithi174 ceph-mon[36060]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T00:49:31.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:31 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T00:49:31.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:31 smithi174 ceph-mon[36060]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:49:31.859 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 00:49:31 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-1[40666]: 2024-03-28T00:49:31.488+0000 7fbd73f17700 -1 osd.1 0 waiting for initial osdmap 2024-03-28T00:49:31.859 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 00:49:31 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-1[40666]: 2024-03-28T00:49:31.491+0000 7fbd6fd0a700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T00:49:31.942 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T00:49:31.942 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T00:49:31.942 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-03-28T00:49:31.942 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-03-28T00:49:31.942 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0153762 s, 682 MB/s 2024-03-28T00:49:31.942 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-03-28T00:49:32.560 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi177:vg_nvme/lv_4 2024-03-28T00:49:32.778 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:32 smithi174 ceph-mon[36060]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-28T00:49:32.778 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:32 smithi174 ceph-mon[36060]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T00:49:32.778 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:32 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T00:49:32.778 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:32 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T00:49:32.778 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:32 smithi174 ceph-mon[36060]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:32.778 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:32 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T00:49:32.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:32 smithi169 ceph-mon[32274]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-28T00:49:32.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:32 smithi169 ceph-mon[32274]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T00:49:32.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:32 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T00:49:32.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:32 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T00:49:32.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:32 smithi169 ceph-mon[32274]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:32.804 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:32 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T00:49:32.805 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T00:49:32.833 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:32 smithi177 ceph-mon[36206]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-28T00:49:32.833 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:32 smithi177 ceph-mon[36206]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T00:49:32.833 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:32 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T00:49:32.833 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:32 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T00:49:32.833 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:32 smithi177 ceph-mon[36206]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T00:49:32.833 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:32 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T00:49:33.774 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:33 smithi177 ceph-mon[36206]: purged_snaps scrub starts 2024-03-28T00:49:33.774 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:33 smithi177 ceph-mon[36206]: purged_snaps scrub ok 2024-03-28T00:49:33.774 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:33 smithi177 ceph-mon[36206]: osd.1 [v2:172.21.15.174:6800/3333696361,v1:172.21.15.174:6801/3333696361] boot 2024-03-28T00:49:33.774 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:33 smithi177 ceph-mon[36206]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T00:49:33.774 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T00:49:33.774 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:33 smithi177 ceph-mon[36206]: Detected new or changed devices on smithi174 2024-03-28T00:49:33.774 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:33.774 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:33.775 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:49:33.775 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:33 smithi177 ceph-mon[36206]: Adjusting osd_memory_target on smithi174 to 17080M 2024-03-28T00:49:33.775 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:33.775 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:33.775 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:49:33.775 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:33.775 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:49:33.775 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:33.775 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:49:33.775 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:33.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:33 smithi169 ceph-mon[32274]: purged_snaps scrub starts 2024-03-28T00:49:33.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:33 smithi169 ceph-mon[32274]: purged_snaps scrub ok 2024-03-28T00:49:33.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:33 smithi169 ceph-mon[32274]: osd.1 [v2:172.21.15.174:6800/3333696361,v1:172.21.15.174:6801/3333696361] boot 2024-03-28T00:49:33.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:33 smithi169 ceph-mon[32274]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T00:49:33.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T00:49:33.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:33 smithi169 ceph-mon[32274]: Detected new or changed devices on smithi174 2024-03-28T00:49:33.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:33.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:33.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:49:33.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:33 smithi169 ceph-mon[32274]: Adjusting osd_memory_target on smithi174 to 17080M 2024-03-28T00:49:33.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:33.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:33.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:49:33.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:33.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:49:33.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:33.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:49:33.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:33.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:33 smithi174 ceph-mon[36060]: purged_snaps scrub starts 2024-03-28T00:49:33.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:33 smithi174 ceph-mon[36060]: purged_snaps scrub ok 2024-03-28T00:49:33.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:33 smithi174 ceph-mon[36060]: osd.1 [v2:172.21.15.174:6800/3333696361,v1:172.21.15.174:6801/3333696361] boot 2024-03-28T00:49:33.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:33 smithi174 ceph-mon[36060]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T00:49:33.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T00:49:33.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:33 smithi174 ceph-mon[36060]: Detected new or changed devices on smithi174 2024-03-28T00:49:33.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:33.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:33.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:49:33.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:33 smithi174 ceph-mon[36060]: Adjusting osd_memory_target on smithi174 to 17080M 2024-03-28T00:49:33.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:33.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:33.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:49:33.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:33.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:49:33.860 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:33.860 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:49:33.860 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:34.245 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.239+0000 7f9e2f24b700 1 -- 172.21.15.177:0/1721325086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e281021f0 msgr2=0x7f9e28102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:49:34.245 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.239+0000 7f9e2f24b700 1 --2- 172.21.15.177:0/1721325086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e281021f0 0x7f9e28102610 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c009990 tx=0x7f9e1c009ca0 comp rx=0 tx=0).stop 2024-03-28T00:49:34.245 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.239+0000 7f9e2f24b700 1 -- 172.21.15.177:0/1721325086 shutdown_connections 2024-03-28T00:49:34.245 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.239+0000 7f9e2f24b700 1 --2- 172.21.15.177:0/1721325086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e28103d90 0x7f9e2810a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:34.245 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.239+0000 7f9e2f24b700 1 --2- 172.21.15.177:0/1721325086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e281021f0 0x7f9e28102610 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:34.245 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.239+0000 7f9e2f24b700 1 --2- 172.21.15.177:0/1721325086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e281033f0 0x7f9e28103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:34.246 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.239+0000 7f9e2f24b700 1 -- 172.21.15.177:0/1721325086 >> 172.21.15.177:0/1721325086 conn(0x7f9e280fd790 msgr2=0x7f9e280ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:49:34.246 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.239+0000 7f9e2f24b700 1 -- 172.21.15.177:0/1721325086 shutdown_connections 2024-03-28T00:49:34.246 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.239+0000 7f9e2f24b700 1 -- 172.21.15.177:0/1721325086 wait complete. 2024-03-28T00:49:34.246 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.240+0000 7f9e2f24b700 1 Processor -- start 2024-03-28T00:49:34.246 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.240+0000 7f9e2f24b700 1 -- start start 2024-03-28T00:49:34.246 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.240+0000 7f9e2f24b700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e281021f0 0x7f9e28198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:49:34.246 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.240+0000 7f9e2f24b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e281033f0 0x7f9e28198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:49:34.246 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.240+0000 7f9e2f24b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e28103d90 0x7f9e2819fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:49:34.246 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.240+0000 7f9e2f24b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2810deb0 con 0x7f9e28103d90 2024-03-28T00:49:34.246 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.240+0000 7f9e2f24b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2810dd30 con 0x7f9e281033f0 2024-03-28T00:49:34.246 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.240+0000 7f9e2f24b700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2810e030 con 0x7f9e281021f0 2024-03-28T00:49:34.246 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.240+0000 7f9e2cfe7700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e281021f0 0x7f9e28198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:49:34.247 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.240+0000 7f9e2cfe7700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e281021f0 0x7f9e28198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:36280/0 (socket says 172.21.15.177:36280) 2024-03-28T00:49:34.247 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.240+0000 7f9e2cfe7700 1 -- 172.21.15.177:0/2024233798 learned_addr learned my addr 172.21.15.177:0/2024233798 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-28T00:49:34.247 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.240+0000 7f9e27fff700 1 --2- 172.21.15.177:0/2024233798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e281033f0 0x7f9e28198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:49:34.247 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.240+0000 7f9e2d7e8700 1 --2- 172.21.15.177:0/2024233798 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e28103d90 0x7f9e2819fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:49:34.247 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.240+0000 7f9e2cfe7700 1 -- 172.21.15.177:0/2024233798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e281033f0 msgr2=0x7f9e28198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:49:34.247 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.240+0000 7f9e2cfe7700 1 --2- 172.21.15.177:0/2024233798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e281033f0 0x7f9e28198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:34.247 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.240+0000 7f9e2cfe7700 1 -- 172.21.15.177:0/2024233798 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e28103d90 msgr2=0x7f9e2819fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:49:34.247 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.240+0000 7f9e2cfe7700 1 --2- 172.21.15.177:0/2024233798 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e28103d90 0x7f9e2819fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:34.247 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.240+0000 7f9e2cfe7700 1 -- 172.21.15.177:0/2024233798 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e281a01b0 con 0x7f9e281021f0 2024-03-28T00:49:34.247 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.240+0000 7f9e2d7e8700 1 --2- 172.21.15.177:0/2024233798 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e28103d90 0x7f9e2819fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:49:34.247 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.241+0000 7f9e2cfe7700 1 --2- 172.21.15.177:0/2024233798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e281021f0 0x7f9e28198550 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0047b0 tx=0x7f9e1c009ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:49:34.247 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.241+0000 7f9e25ffb700 1 -- 172.21.15.177:0/2024233798 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c026070 con 0x7f9e281021f0 2024-03-28T00:49:34.247 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.241+0000 7f9e25ffb700 1 -- 172.21.15.177:0/2024233798 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e1c004210 con 0x7f9e281021f0 2024-03-28T00:49:34.248 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.241+0000 7f9e25ffb700 1 -- 172.21.15.177:0/2024233798 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c0053b0 con 0x7f9e281021f0 2024-03-28T00:49:34.248 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.241+0000 7f9e2f24b700 1 -- 172.21.15.177:0/2024233798 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e281a0440 con 0x7f9e281021f0 2024-03-28T00:49:34.248 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.241+0000 7f9e2f24b700 1 -- 172.21.15.177:0/2024233798 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e281a0970 con 0x7f9e281021f0 2024-03-28T00:49:34.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.242+0000 7f9e2f24b700 1 -- 172.21.15.177:0/2024233798 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e28068e50 con 0x7f9e281021f0 2024-03-28T00:49:34.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.244+0000 7f9e25ffb700 1 -- 172.21.15.177:0/2024233798 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7f9e1c029070 con 0x7f9e281021f0 2024-03-28T00:49:34.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.244+0000 7f9e25ffb700 1 --2- 172.21.15.177:0/2024233798 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9e180a2880 0x7f9e180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:49:34.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.244+0000 7f9e25ffb700 1 -- 172.21.15.177:0/2024233798 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f9e1c0ce280 con 0x7f9e281021f0 2024-03-28T00:49:34.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.245+0000 7f9e27fff700 1 --2- 172.21.15.177:0/2024233798 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9e180a2880 0x7f9e180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:49:34.249 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.245+0000 7f9e27fff700 1 --2- 172.21.15.177:0/2024233798 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9e180a2880 0x7f9e180a4d40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9e14009a10 tx=0x7f9e14009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:49:34.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.248+0000 7f9e25ffb700 1 -- 172.21.15.177:0/2024233798 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9e1c030020 con 0x7f9e281021f0 2024-03-28T00:49:34.456 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:34.452+0000 7f9e2f24b700 1 -- 172.21.15.177:0/2024233798 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e280625d0 con 0x7f9e180a2880 2024-03-28T00:49:34.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:34 smithi169 ceph-mon[32274]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T00:49:34.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:34 smithi169 ceph-mon[32274]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:34.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:34 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:49:34.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:34 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:49:34.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:34 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:34.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:34 smithi174 ceph-mon[36060]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T00:49:34.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:34 smithi174 ceph-mon[36060]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:34.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:34 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:49:34.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:34 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:49:34.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:34 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:34.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:34 smithi177 ceph-mon[36206]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T00:49:34.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:34 smithi177 ceph-mon[36206]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:34.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:34 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:49:34.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:34 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:49:34.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:34 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:35.604 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:35 smithi177 ceph-mon[36206]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:49:35.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:35 smithi169 ceph-mon[32274]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:49:35.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:35 smithi174 ceph-mon[36060]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:49:36.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:36 smithi169 ceph-mon[32274]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:36.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:36 smithi174 ceph-mon[36060]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:36.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:36 smithi177 ceph-mon[36206]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:37.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:37 smithi177 ceph-mon[36206]: from='client.? 172.21.15.177:0/4233240422' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a6108e3-e6eb-489b-8f90-1caba1b3da34"}]: dispatch 2024-03-28T00:49:37.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:37 smithi177 ceph-mon[36206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a6108e3-e6eb-489b-8f90-1caba1b3da34"}]: dispatch 2024-03-28T00:49:37.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:37 smithi177 ceph-mon[36206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a6108e3-e6eb-489b-8f90-1caba1b3da34"}]': finished 2024-03-28T00:49:37.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:37 smithi177 ceph-mon[36206]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T00:49:37.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:37 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T00:49:37.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:37 smithi169 ceph-mon[32274]: from='client.? 172.21.15.177:0/4233240422' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a6108e3-e6eb-489b-8f90-1caba1b3da34"}]: dispatch 2024-03-28T00:49:37.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:37 smithi169 ceph-mon[32274]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a6108e3-e6eb-489b-8f90-1caba1b3da34"}]: dispatch 2024-03-28T00:49:37.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:37 smithi169 ceph-mon[32274]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a6108e3-e6eb-489b-8f90-1caba1b3da34"}]': finished 2024-03-28T00:49:37.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:37 smithi169 ceph-mon[32274]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T00:49:37.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T00:49:37.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:37 smithi174 ceph-mon[36060]: from='client.? 172.21.15.177:0/4233240422' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a6108e3-e6eb-489b-8f90-1caba1b3da34"}]: dispatch 2024-03-28T00:49:37.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:37 smithi174 ceph-mon[36060]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a6108e3-e6eb-489b-8f90-1caba1b3da34"}]: dispatch 2024-03-28T00:49:37.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:37 smithi174 ceph-mon[36060]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a6108e3-e6eb-489b-8f90-1caba1b3da34"}]': finished 2024-03-28T00:49:37.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:37 smithi174 ceph-mon[36060]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T00:49:37.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:37 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T00:49:38.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:38 smithi169 ceph-mon[32274]: from='client.? 172.21.15.177:0/1676080023' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:49:38.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:38 smithi169 ceph-mon[32274]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:38.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:38 smithi174 ceph-mon[36060]: from='client.? 172.21.15.177:0/1676080023' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:49:38.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:38 smithi174 ceph-mon[36060]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:38.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:38 smithi177 ceph-mon[36206]: from='client.? 172.21.15.177:0/1676080023' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:49:38.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:38 smithi177 ceph-mon[36206]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:41.229 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:41 smithi177 ceph-mon[36206]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:41.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:41 smithi174 ceph-mon[36060]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:41.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:41 smithi169 ceph-mon[32274]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:43.266 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:43 smithi177 ceph-mon[36206]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:43.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:43 smithi174 ceph-mon[36060]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:43.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:43 smithi169 ceph-mon[32274]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:44.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:44 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T00:49:44.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:44 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:44.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:44 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T00:49:44.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:44 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:44.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:44 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T00:49:44.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:44 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:45.320 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:45 smithi177 ceph-mon[36206]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:45.321 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:45 smithi177 ceph-mon[36206]: Deploying daemon osd.2 on smithi177 2024-03-28T00:49:45.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:45 smithi174 ceph-mon[36060]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:45.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:45 smithi174 ceph-mon[36060]: Deploying daemon osd.2 on smithi177 2024-03-28T00:49:45.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:45 smithi169 ceph-mon[32274]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:45.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:45 smithi169 ceph-mon[32274]: Deploying daemon osd.2 on smithi177 2024-03-28T00:49:47.352 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:47 smithi177 ceph-mon[36206]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:47.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:47 smithi174 ceph-mon[36060]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:47.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:47 smithi169 ceph-mon[32274]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:49.210 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:49 smithi177 ceph-mon[36206]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:49.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:49 smithi174 ceph-mon[36060]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:49.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:49 smithi169 ceph-mon[32274]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:50.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:50 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:50.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:50 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:50.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:50 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:49:50.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:50 smithi174 ceph-mon[36060]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:50.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:50 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:50.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:50 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:50.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:50 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:49:50.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:50 smithi177 ceph-mon[36206]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:50.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:50 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:50.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:50 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:50.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:50 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:49:50.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:50 smithi169 ceph-mon[32274]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:52.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:52.668+0000 7f9e25ffb700 1 -- 172.21.15.177:0/2024233798 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9e280625d0 con 0x7f9e180a2880 2024-03-28T00:49:52.672 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 2 on host 'smithi177' 2024-03-28T00:49:52.674 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:52.670+0000 7f9e2f24b700 1 -- 172.21.15.177:0/2024233798 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9e180a2880 msgr2=0x7f9e180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:49:52.674 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:52.670+0000 7f9e2f24b700 1 --2- 172.21.15.177:0/2024233798 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9e180a2880 0x7f9e180a4d40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9e14009a10 tx=0x7f9e14009190 comp rx=0 tx=0).stop 2024-03-28T00:49:52.674 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:52.670+0000 7f9e2f24b700 1 -- 172.21.15.177:0/2024233798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e281021f0 msgr2=0x7f9e28198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:49:52.675 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:52.670+0000 7f9e2f24b700 1 --2- 172.21.15.177:0/2024233798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e281021f0 0x7f9e28198550 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0047b0 tx=0x7f9e1c009ee0 comp rx=0 tx=0).stop 2024-03-28T00:49:52.675 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:52.671+0000 7f9e2f24b700 1 -- 172.21.15.177:0/2024233798 shutdown_connections 2024-03-28T00:49:52.675 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:52.671+0000 7f9e2f24b700 1 --2- 172.21.15.177:0/2024233798 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e28103d90 0x7f9e2819fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:52.675 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:52.671+0000 7f9e2f24b700 1 --2- 172.21.15.177:0/2024233798 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9e180a2880 0x7f9e180a4d40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:52.675 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:52.671+0000 7f9e2f24b700 1 --2- 172.21.15.177:0/2024233798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e281021f0 0x7f9e28198550 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:52.675 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:52.671+0000 7f9e2f24b700 1 --2- 172.21.15.177:0/2024233798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e281033f0 0x7f9e28198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:52.675 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:52.671+0000 7f9e2f24b700 1 -- 172.21.15.177:0/2024233798 >> 172.21.15.177:0/2024233798 conn(0x7f9e280fd790 msgr2=0x7f9e28106770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:49:52.675 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:52.671+0000 7f9e2f24b700 1 -- 172.21.15.177:0/2024233798 shutdown_connections 2024-03-28T00:49:52.675 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:49:52.671+0000 7f9e2f24b700 1 -- 172.21.15.177:0/2024233798 wait complete. 2024-03-28T00:49:52.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:52 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:52.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:52 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:52.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:52 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:52.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:52 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:49:52.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:52 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:52.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:52 smithi177 ceph-mon[36206]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:52.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:52 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:52.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:52 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:52.918 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:52 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:49:53.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:52 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:53.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:52 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:53.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:52 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:53.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:52 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:49:53.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:52 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:53.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:52 smithi169 ceph-mon[32274]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:53.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:52 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:53.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:52 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:53.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:52 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:49:53.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:52 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:53.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:52 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:53.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:52 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:53.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:52 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:49:53.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:52 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:53.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:52 smithi174 ceph-mon[36060]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:53.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:52 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:53.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:52 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:53.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:52 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:49:53.313 DEBUG:teuthology.orchestra.run.smithi177:osd.2> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.2.service 2024-03-28T00:49:53.315 INFO:tasks.cephadm:Deploying osd.3 on smithi169 with /dev/vg_nvme/lv_3... 2024-03-28T00:49:53.315 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T00:49:53.565 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:49:53.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:53 smithi177 ceph-mon[36206]: from='osd.2 [v2:172.21.15.177:6800/1495865602,v1:172.21.15.177:6801/1495865602]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T00:49:53.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:53 smithi177 ceph-mon[36206]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T00:49:53.958 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:53 smithi169 ceph-mon[32274]: from='osd.2 [v2:172.21.15.177:6800/1495865602,v1:172.21.15.177:6801/1495865602]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T00:49:53.958 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:53 smithi169 ceph-mon[32274]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T00:49:54.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:53 smithi174 ceph-mon[36060]: from='osd.2 [v2:172.21.15.177:6800/1495865602,v1:172.21.15.177:6801/1495865602]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T00:49:54.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:53 smithi174 ceph-mon[36060]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T00:49:55.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:54 smithi169 ceph-mon[32274]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T00:49:55.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:54 smithi169 ceph-mon[32274]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T00:49:55.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:54 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T00:49:55.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:54 smithi169 ceph-mon[32274]: from='osd.2 [v2:172.21.15.177:6800/1495865602,v1:172.21.15.177:6801/1495865602]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:49:55.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:54 smithi169 ceph-mon[32274]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:49:55.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:54 smithi169 ceph-mon[32274]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:55.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:54 smithi174 ceph-mon[36060]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T00:49:55.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:54 smithi174 ceph-mon[36060]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T00:49:55.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:54 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T00:49:55.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:54 smithi174 ceph-mon[36060]: from='osd.2 [v2:172.21.15.177:6800/1495865602,v1:172.21.15.177:6801/1495865602]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:49:55.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:54 smithi174 ceph-mon[36060]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:49:55.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:54 smithi174 ceph-mon[36060]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:55.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:54 smithi177 ceph-mon[36206]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T00:49:55.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:54 smithi177 ceph-mon[36206]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T00:49:55.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:54 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T00:49:55.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:54 smithi177 ceph-mon[36206]: from='osd.2 [v2:172.21.15.177:6800/1495865602,v1:172.21.15.177:6801/1495865602]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:49:55.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:54 smithi177 ceph-mon[36206]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:49:55.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:54 smithi177 ceph-mon[36206]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T00:49:55.168 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 00:49:54 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-2[41334]: 2024-03-28T00:49:54.815+0000 7ff88bb35700 -1 osd.2 0 waiting for initial osdmap 2024-03-28T00:49:55.168 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 00:49:54 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-2[41334]: 2024-03-28T00:49:54.821+0000 7ff887928700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T00:49:55.641 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T00:49:55.641 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T00:49:55.641 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2024-03-28T00:49:55.641 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2024-03-28T00:49:55.641 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0133163 s, 787 MB/s 2024-03-28T00:49:55.641 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2024-03-28T00:49:55.983 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:55 smithi177 ceph-mon[36206]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-28T00:49:55.983 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:55 smithi177 ceph-mon[36206]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T00:49:55.983 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:55 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T00:49:55.983 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:55 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T00:49:56.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:55 smithi174 ceph-mon[36060]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-28T00:49:56.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:55 smithi174 ceph-mon[36060]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T00:49:56.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:55 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T00:49:56.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:55 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T00:49:56.115 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:55 smithi169 ceph-mon[32274]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-28T00:49:56.115 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:55 smithi169 ceph-mon[32274]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T00:49:56.115 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:55 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T00:49:56.115 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:55 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T00:49:56.254 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_3 2024-03-28T00:49:56.495 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:49:56.898 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:56 smithi169 ceph-mon[32274]: purged_snaps scrub starts 2024-03-28T00:49:56.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:56 smithi169 ceph-mon[32274]: purged_snaps scrub ok 2024-03-28T00:49:56.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:56 smithi169 ceph-mon[32274]: osd.2 [v2:172.21.15.177:6800/1495865602,v1:172.21.15.177:6801/1495865602] boot 2024-03-28T00:49:56.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:56 smithi169 ceph-mon[32274]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T00:49:56.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T00:49:56.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:56 smithi169 ceph-mon[32274]: pgmap v78: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T00:49:56.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:56 smithi169 ceph-mon[32274]: Detected new or changed devices on smithi177 2024-03-28T00:49:56.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:56.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:56.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:49:56.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:56 smithi169 ceph-mon[32274]: Adjusting osd_memory_target on smithi177 to 17080M 2024-03-28T00:49:56.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:56.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:56.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:49:56.899 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:56.900 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T00:49:57.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:56 smithi174 ceph-mon[36060]: purged_snaps scrub starts 2024-03-28T00:49:57.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:56 smithi174 ceph-mon[36060]: purged_snaps scrub ok 2024-03-28T00:49:57.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:56 smithi174 ceph-mon[36060]: osd.2 [v2:172.21.15.177:6800/1495865602,v1:172.21.15.177:6801/1495865602] boot 2024-03-28T00:49:57.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:56 smithi174 ceph-mon[36060]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T00:49:57.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T00:49:57.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:56 smithi174 ceph-mon[36060]: pgmap v78: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T00:49:57.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:56 smithi174 ceph-mon[36060]: Detected new or changed devices on smithi177 2024-03-28T00:49:57.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:57.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:57.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:49:57.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:56 smithi174 ceph-mon[36060]: Adjusting osd_memory_target on smithi177 to 17080M 2024-03-28T00:49:57.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:57.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:57.110 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:49:57.110 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:57.110 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T00:49:57.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:56 smithi177 ceph-mon[36206]: purged_snaps scrub starts 2024-03-28T00:49:57.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:56 smithi177 ceph-mon[36206]: purged_snaps scrub ok 2024-03-28T00:49:57.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:56 smithi177 ceph-mon[36206]: osd.2 [v2:172.21.15.177:6800/1495865602,v1:172.21.15.177:6801/1495865602] boot 2024-03-28T00:49:57.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:56 smithi177 ceph-mon[36206]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T00:49:57.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T00:49:57.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:56 smithi177 ceph-mon[36206]: pgmap v78: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T00:49:57.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:56 smithi177 ceph-mon[36206]: Detected new or changed devices on smithi177 2024-03-28T00:49:57.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:57.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:57.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:49:57.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:56 smithi177 ceph-mon[36206]: Adjusting osd_memory_target on smithi177 to 17080M 2024-03-28T00:49:57.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:57.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:57.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:49:57.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:57.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T00:49:58.053 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.048+0000 7f34a2e7d700 1 -- 172.21.15.169:0/459106364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f349c1021c0 msgr2=0x7f349c1025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:49:58.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.048+0000 7f34a2e7d700 1 --2- 172.21.15.169:0/459106364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f349c1021c0 0x7f349c1025e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f34900099c0 tx=0x7f3490009cd0 comp rx=0 tx=0).stop 2024-03-28T00:49:58.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.048+0000 7f34a2e7d700 1 -- 172.21.15.169:0/459106364 shutdown_connections 2024-03-28T00:49:58.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.048+0000 7f34a2e7d700 1 --2- 172.21.15.169:0/459106364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f349c103d60 0x7f349c10a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:58.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.048+0000 7f34a2e7d700 1 --2- 172.21.15.169:0/459106364 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f349c1021c0 0x7f349c1025e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:58.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.048+0000 7f34a2e7d700 1 --2- 172.21.15.169:0/459106364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f349c1033c0 0x7f349c103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:58.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.048+0000 7f34a2e7d700 1 -- 172.21.15.169:0/459106364 >> 172.21.15.169:0/459106364 conn(0x7f349c0fd7a0 msgr2=0x7f349c0ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:49:58.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.048+0000 7f34a2e7d700 1 -- 172.21.15.169:0/459106364 shutdown_connections 2024-03-28T00:49:58.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.048+0000 7f34a2e7d700 1 -- 172.21.15.169:0/459106364 wait complete. 2024-03-28T00:49:58.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.049+0000 7f34a2e7d700 1 Processor -- start 2024-03-28T00:49:58.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.049+0000 7f34a2e7d700 1 -- start start 2024-03-28T00:49:58.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.049+0000 7f34a2e7d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f349c1021c0 0x7f349c198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:49:58.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.049+0000 7f34a2e7d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f349c1033c0 0x7f349c198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:49:58.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.049+0000 7f34a2e7d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f349c103d60 0x7f349c19f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:49:58.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.049+0000 7f34a2e7d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c10dda0 con 0x7f349c1021c0 2024-03-28T00:49:58.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.049+0000 7f34a2e7d700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c10dc20 con 0x7f349c1033c0 2024-03-28T00:49:58.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.049+0000 7f34a2e7d700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c10df20 con 0x7f349c103d60 2024-03-28T00:49:58.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.049+0000 7f34a0c19700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f349c1021c0 0x7f349c198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:49:58.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.049+0000 7f34a0c19700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f349c1021c0 0x7f349c198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46970/0 (socket says 172.21.15.169:46970) 2024-03-28T00:49:58.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.049+0000 7f34a0c19700 1 -- 172.21.15.169:0/3394741209 learned_addr learned my addr 172.21.15.169:0/3394741209 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:49:58.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.049+0000 7f34a141a700 1 --2- 172.21.15.169:0/3394741209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f349c103d60 0x7f349c19f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:49:58.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.049+0000 7f34a0c19700 1 -- 172.21.15.169:0/3394741209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f349c103d60 msgr2=0x7f349c19f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:49:58.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.049+0000 7f34a0c19700 1 --2- 172.21.15.169:0/3394741209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f349c103d60 0x7f349c19f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:58.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.049+0000 7f34a0c19700 1 -- 172.21.15.169:0/3394741209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f349c1033c0 msgr2=0x7f349c198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:49:58.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.049+0000 7f349bfff700 1 --2- 172.21.15.169:0/3394741209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f349c1033c0 0x7f349c198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:49:58.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.049+0000 7f34a0c19700 1 --2- 172.21.15.169:0/3394741209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f349c1033c0 0x7f349c198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:49:58.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.049+0000 7f34a0c19700 1 -- 172.21.15.169:0/3394741209 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c1a0080 con 0x7f349c1021c0 2024-03-28T00:49:58.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.050+0000 7f349bfff700 1 --2- 172.21.15.169:0/3394741209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f349c1033c0 0x7f349c198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:49:58.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.050+0000 7f34a0c19700 1 --2- 172.21.15.169:0/3394741209 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f349c1021c0 0x7f349c198420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3490009510 tx=0x7f3490009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:49:58.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.050+0000 7f3499ffb700 1 -- 172.21.15.169:0/3394741209 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3490027070 con 0x7f349c1021c0 2024-03-28T00:49:58.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.050+0000 7f3499ffb700 1 -- 172.21.15.169:0/3394741209 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3490004150 con 0x7f349c1021c0 2024-03-28T00:49:58.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.050+0000 7f34a2e7d700 1 -- 172.21.15.169:0/3394741209 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f349c1a0310 con 0x7f349c1021c0 2024-03-28T00:49:58.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.050+0000 7f3499ffb700 1 -- 172.21.15.169:0/3394741209 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349000f460 con 0x7f349c1021c0 2024-03-28T00:49:58.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.050+0000 7f34a2e7d700 1 -- 172.21.15.169:0/3394741209 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f349c1a0840 con 0x7f349c1021c0 2024-03-28T00:49:58.057 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.051+0000 7f34a2e7d700 1 -- 172.21.15.169:0/3394741209 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f349c068e50 con 0x7f349c1021c0 2024-03-28T00:49:58.057 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.052+0000 7f3499ffb700 1 -- 172.21.15.169:0/3394741209 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7f349002a070 con 0x7f349c1021c0 2024-03-28T00:49:58.057 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.053+0000 7f3499ffb700 1 --2- 172.21.15.169:0/3394741209 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f348c0a2880 0x7f348c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:49:58.057 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.053+0000 7f349bfff700 1 --2- 172.21.15.169:0/3394741209 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f348c0a2880 0x7f348c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:49:58.057 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.053+0000 7f3499ffb700 1 -- 172.21.15.169:0/3394741209 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3226+0+0 (secure 0 0 0) 0x7f34900ce500 con 0x7f349c1021c0 2024-03-28T00:49:58.057 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.053+0000 7f349bfff700 1 --2- 172.21.15.169:0/3394741209 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f348c0a2880 0x7f348c0a4d40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3488005fd0 tx=0x7f3488005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:49:58.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.057+0000 7f3499ffb700 1 -- 172.21.15.169:0/3394741209 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f349005e740 con 0x7f349c1021c0 2024-03-28T00:49:58.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:58.279+0000 7f34a2e7d700 1 -- 172.21.15.169:0/3394741209 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f349c0625d0 con 0x7f348c0a2880 2024-03-28T00:49:58.283 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:58 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T00:49:58.283 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:58 smithi169 ceph-mon[32274]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T00:49:58.283 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:58 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T00:49:58.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:58 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T00:49:58.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:58 smithi174 ceph-mon[36060]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T00:49:58.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:58 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T00:49:58.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:58 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T00:49:58.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:58 smithi177 ceph-mon[36206]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T00:49:58.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:58 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T00:49:58.552 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 00:49:58 smithi169 sudo[49209]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T00:49:58.552 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 00:49:58 smithi169 sudo[49209]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T00:49:58.552 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 00:49:58 smithi169 sudo[49209]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T00:49:58.552 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 00:49:58 smithi169 sudo[49209]: pam_unix(sudo:session): session closed for user root 2024-03-28T00:49:59.029 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 00:49:58 smithi169 sudo[49212]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T00:49:59.029 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 00:49:58 smithi169 sudo[49212]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T00:49:59.029 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 00:49:58 smithi169 sudo[49212]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T00:49:59.030 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 00:49:58 smithi169 sudo[49212]: pam_unix(sudo:session): session closed for user root 2024-03-28T00:49:59.215 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 00:49:58 smithi174 sudo[44372]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T00:49:59.215 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 00:49:58 smithi174 sudo[44372]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T00:49:59.215 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 00:49:58 smithi174 sudo[44372]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T00:49:59.215 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 00:49:58 smithi174 sudo[44372]: pam_unix(sudo:session): session closed for user root 2024-03-28T00:49:59.215 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:59 smithi174 ceph-mon[36060]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T00:49:59.215 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:59 smithi174 ceph-mon[36060]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T00:49:59.216 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:59 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T00:49:59.216 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:59 smithi174 ceph-mon[36060]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T00:49:59.216 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:59 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:49:59.216 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:59 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:49:59.216 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:59 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:59.216 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:49:59 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:59.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:59 smithi169 ceph-mon[32274]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T00:49:59.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:59 smithi169 ceph-mon[32274]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T00:49:59.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:59 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T00:49:59.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:59 smithi169 ceph-mon[32274]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T00:49:59.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:59 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:49:59.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:59 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:49:59.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:59 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:59.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:49:59 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:59.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:59 smithi177 ceph-mon[36206]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T00:49:59.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:59 smithi177 ceph-mon[36206]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T00:49:59.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:59 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T00:49:59.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:59 smithi177 ceph-mon[36206]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T00:49:59.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:59 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:49:59.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:59 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:49:59.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:59 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:49:59.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:49:59 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:49:59.608 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 00:49:59 smithi174 sudo[44375]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T00:49:59.608 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 00:49:59 smithi174 sudo[44375]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T00:49:59.608 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 00:49:59 smithi174 sudo[44375]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T00:49:59.608 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 00:49:59 smithi174 sudo[44375]: pam_unix(sudo:session): session closed for user root 2024-03-28T00:49:59.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:49:59.721+0000 7f3499ffb700 1 -- 172.21.15.169:0/3394741209 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3490098a60 con 0x7f349c1021c0 2024-03-28T00:49:59.733 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 00:49:59 smithi177 sudo[45108]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T00:49:59.733 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 00:49:59 smithi177 sudo[45108]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T00:49:59.733 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 00:49:59 smithi177 sudo[45108]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T00:49:59.733 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 00:49:59 smithi177 sudo[45108]: pam_unix(sudo:session): session closed for user root 2024-03-28T00:50:00.155 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:00 smithi169 sudo[49311]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T00:50:00.155 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:00 smithi169 sudo[49311]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T00:50:00.155 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:00 smithi169 sudo[49311]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T00:50:00.155 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:00 smithi169 sudo[49311]: pam_unix(sudo:session): session closed for user root 2024-03-28T00:50:00.167 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 00:49:59 smithi177 sudo[45111]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T00:50:00.167 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 00:49:59 smithi177 sudo[45111]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T00:50:00.167 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 00:49:59 smithi177 sudo[45111]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T00:50:00.167 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 00:49:59 smithi177 sudo[45111]: pam_unix(sudo:session): session closed for user root 2024-03-28T00:50:00.416 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:00 smithi169 sudo[49320]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T00:50:00.417 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:00 smithi169 sudo[49320]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T00:50:00.417 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:00 smithi169 sudo[49320]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T00:50:00.417 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:00 smithi169 sudo[49320]: pam_unix(sudo:session): session closed for user root 2024-03-28T00:50:00.417 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:00 smithi169 ceph-mon[32274]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:50:00.417 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:00 smithi169 ceph-mon[32274]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T00:50:00.417 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:00 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:00.417 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:00 smithi169 ceph-mon[32274]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T00:50:00.417 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:00 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:00.417 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:00 smithi169 ceph-mon[32274]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T00:50:00.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:00 smithi177 ceph-mon[36206]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:50:00.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:00 smithi177 ceph-mon[36206]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T00:50:00.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:00 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:00.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:00 smithi177 ceph-mon[36206]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T00:50:00.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:00 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:00.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:00 smithi177 ceph-mon[36206]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T00:50:00.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:00 smithi177 sudo[45114]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T00:50:00.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:00 smithi177 sudo[45114]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T00:50:00.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:00 smithi177 sudo[45114]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T00:50:00.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:00 smithi177 sudo[45114]: pam_unix(sudo:session): session closed for user root 2024-03-28T00:50:00.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:00 smithi174 ceph-mon[36060]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:50:00.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:00 smithi174 ceph-mon[36060]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T00:50:00.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:00 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:00.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:00 smithi174 ceph-mon[36060]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T00:50:00.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:00 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:00.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:00 smithi174 ceph-mon[36060]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T00:50:00.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:00 smithi177 sudo[45117]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T00:50:00.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:00 smithi177 sudo[45117]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T00:50:00.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:00 smithi177 sudo[45117]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T00:50:00.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:00 smithi177 sudo[45117]: pam_unix(sudo:session): session closed for user root 2024-03-28T00:50:01.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 sudo[44378]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T00:50:01.256 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 sudo[44378]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T00:50:01.257 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 sudo[44378]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T00:50:01.257 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 sudo[44378]: pam_unix(sudo:session): session closed for user root 2024-03-28T00:50:01.257 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 sudo[44381]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T00:50:01.257 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 sudo[44381]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T00:50:01.257 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 sudo[44381]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T00:50:01.257 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 sudo[44381]: pam_unix(sudo:session): session closed for user root 2024-03-28T00:50:01.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T00:50:01.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T00:50:01.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-03-28T00:50:01.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-03-28T00:50:01.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: application not enabled on pool '.mgr' 2024-03-28T00:50:01.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-28T00:50:01.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T00:50:01.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:50:01.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:50:01.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T00:50:01.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T00:50:01.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:50:01.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:50:01.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T00:50:01.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T00:50:01.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T00:50:01.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:50:01.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:50:01.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1658933282' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7eae99d-476b-43c0-968f-8b5bf6d07222"}]: dispatch 2024-03-28T00:50:01.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T00:50:01.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1658933282' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7eae99d-476b-43c0-968f-8b5bf6d07222"}]': finished 2024-03-28T00:50:01.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: osdmap e22: 4 total, 3 up, 4 in 2024-03-28T00:50:01.554 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:01 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T00:50:01.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T00:50:01.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T00:50:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-03-28T00:50:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-03-28T00:50:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: application not enabled on pool '.mgr' 2024-03-28T00:50:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-28T00:50:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T00:50:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:50:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:50:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T00:50:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T00:50:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:50:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:50:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T00:50:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T00:50:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T00:50:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:50:01.610 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:50:01.610 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1658933282' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7eae99d-476b-43c0-968f-8b5bf6d07222"}]: dispatch 2024-03-28T00:50:01.610 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T00:50:01.610 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1658933282' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7eae99d-476b-43c0-968f-8b5bf6d07222"}]': finished 2024-03-28T00:50:01.610 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: osdmap e22: 4 total, 3 up, 4 in 2024-03-28T00:50:01.610 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:01 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T00:50:01.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T00:50:01.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T00:50:01.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-03-28T00:50:01.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-03-28T00:50:01.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: application not enabled on pool '.mgr' 2024-03-28T00:50:01.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-28T00:50:01.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T00:50:01.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:50:01.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:50:01.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T00:50:01.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T00:50:01.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:50:01.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:50:01.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T00:50:01.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T00:50:01.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T00:50:01.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T00:50:01.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T00:50:01.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1658933282' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7eae99d-476b-43c0-968f-8b5bf6d07222"}]: dispatch 2024-03-28T00:50:01.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T00:50:01.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1658933282' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7eae99d-476b-43c0-968f-8b5bf6d07222"}]': finished 2024-03-28T00:50:01.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: osdmap e22: 4 total, 3 up, 4 in 2024-03-28T00:50:01.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:01 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T00:50:02.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:02 smithi169 ceph-mon[32274]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T00:50:02.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:02 smithi169 ceph-mon[32274]: Cluster is now healthy 2024-03-28T00:50:02.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:02 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1470110121' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:50:02.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:02 smithi169 ceph-mon[32274]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T00:50:02.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:02 smithi174 ceph-mon[36060]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T00:50:02.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:02 smithi174 ceph-mon[36060]: Cluster is now healthy 2024-03-28T00:50:02.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:02 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1470110121' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:50:02.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:02 smithi174 ceph-mon[36060]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T00:50:02.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:02 smithi177 ceph-mon[36206]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T00:50:02.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:02 smithi177 ceph-mon[36206]: Cluster is now healthy 2024-03-28T00:50:02.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:02 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1470110121' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:50:02.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:02 smithi177 ceph-mon[36206]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T00:50:05.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:05 smithi169 ceph-mon[32274]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 7.3 KiB/s rd, 370 KiB/s wr, 24 op/s 2024-03-28T00:50:05.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:05 smithi174 ceph-mon[36060]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 7.3 KiB/s rd, 370 KiB/s wr, 24 op/s 2024-03-28T00:50:05.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:05 smithi177 ceph-mon[36206]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 7.3 KiB/s rd, 370 KiB/s wr, 24 op/s 2024-03-28T00:50:07.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:07 smithi177 ceph-mon[36206]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 5.2 KiB/s rd, 263 KiB/s wr, 17 op/s 2024-03-28T00:50:07.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:07 smithi174 ceph-mon[36060]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 5.2 KiB/s rd, 263 KiB/s wr, 17 op/s 2024-03-28T00:50:07.405 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:07 smithi169 ceph-mon[32274]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 5.2 KiB/s rd, 263 KiB/s wr, 17 op/s 2024-03-28T00:50:09.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:09 smithi174 ceph-mon[36060]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 204 KiB/s wr, 13 op/s 2024-03-28T00:50:09.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:09 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T00:50:09.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:09 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:09.380 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:09 smithi169 ceph-mon[32274]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 204 KiB/s wr, 13 op/s 2024-03-28T00:50:09.380 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:09 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T00:50:09.380 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:09 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:09.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:09 smithi177 ceph-mon[36206]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 204 KiB/s wr, 13 op/s 2024-03-28T00:50:09.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:09 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T00:50:09.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:09 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:10.197 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:10 smithi169 ceph-mon[32274]: Deploying daemon osd.3 on smithi169 2024-03-28T00:50:10.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:10 smithi174 ceph-mon[36060]: Deploying daemon osd.3 on smithi169 2024-03-28T00:50:10.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:10 smithi177 ceph-mon[36206]: Deploying daemon osd.3 on smithi169 2024-03-28T00:50:11.183 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:11 smithi169 ceph-mon[32274]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-28T00:50:11.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:11 smithi174 ceph-mon[36060]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-28T00:50:11.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:11 smithi177 ceph-mon[36206]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-28T00:50:13.243 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:13 smithi169 ceph-mon[32274]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 170 KiB/s wr, 11 op/s 2024-03-28T00:50:13.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:13 smithi177 ceph-mon[36206]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 170 KiB/s wr, 11 op/s 2024-03-28T00:50:13.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:13 smithi174 ceph-mon[36060]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 170 KiB/s wr, 11 op/s 2024-03-28T00:50:15.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:14 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:15.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:14 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:15.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:14 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:50:15.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:14 smithi169 ceph-mon[32274]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-28T00:50:15.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:14 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:15.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:14 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:15.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:14 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:50:15.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:14 smithi174 ceph-mon[36060]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-28T00:50:15.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:14 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:15.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:14 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:15.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:14 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:50:15.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:14 smithi177 ceph-mon[36206]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-28T00:50:17.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:17 smithi169 ceph-mon[32274]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T00:50:17.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:17 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:17.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:17 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:17.297 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:17 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:17.297 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:17 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:50:17.297 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:17 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:17.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:17 smithi174 ceph-mon[36060]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T00:50:17.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:17 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:17.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:17 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:17.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:17 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:17.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:17 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:50:17.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:17 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:17.366 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:17 smithi177 ceph-mon[36206]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T00:50:17.366 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:17 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:17.366 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:17 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:17.366 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:17 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:17.366 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:17 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:50:17.366 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:17 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:17.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:50:17.659+0000 7f3499ffb700 1 -- 172.21.15.169:0/3394741209 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f349c0625d0 con 0x7f348c0a2880 2024-03-28T00:50:17.663 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 3 on host 'smithi169' 2024-03-28T00:50:17.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:50:17.662+0000 7f34a2e7d700 1 -- 172.21.15.169:0/3394741209 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f348c0a2880 msgr2=0x7f348c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:50:17.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:50:17.662+0000 7f34a2e7d700 1 --2- 172.21.15.169:0/3394741209 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f348c0a2880 0x7f348c0a4d40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3488005fd0 tx=0x7f3488005ee0 comp rx=0 tx=0).stop 2024-03-28T00:50:17.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:50:17.662+0000 7f34a2e7d700 1 -- 172.21.15.169:0/3394741209 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f349c1021c0 msgr2=0x7f349c198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:50:17.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:50:17.662+0000 7f34a2e7d700 1 --2- 172.21.15.169:0/3394741209 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f349c1021c0 0x7f349c198420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3490009510 tx=0x7f3490009f10 comp rx=0 tx=0).stop 2024-03-28T00:50:17.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:50:17.662+0000 7f34a2e7d700 1 -- 172.21.15.169:0/3394741209 shutdown_connections 2024-03-28T00:50:17.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:50:17.662+0000 7f34a2e7d700 1 --2- 172.21.15.169:0/3394741209 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f348c0a2880 0x7f348c0a4d40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:50:17.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:50:17.662+0000 7f34a2e7d700 1 --2- 172.21.15.169:0/3394741209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f349c103d60 0x7f349c19f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:50:17.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:50:17.662+0000 7f34a2e7d700 1 --2- 172.21.15.169:0/3394741209 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f349c1021c0 0x7f349c198420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:50:17.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:50:17.662+0000 7f34a2e7d700 1 --2- 172.21.15.169:0/3394741209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f349c1033c0 0x7f349c198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:50:17.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:50:17.662+0000 7f34a2e7d700 1 -- 172.21.15.169:0/3394741209 >> 172.21.15.169:0/3394741209 conn(0x7f349c0fd7a0 msgr2=0x7f349c106740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:50:17.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:50:17.662+0000 7f34a2e7d700 1 -- 172.21.15.169:0/3394741209 shutdown_connections 2024-03-28T00:50:17.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:50:17.662+0000 7f34a2e7d700 1 -- 172.21.15.169:0/3394741209 wait complete. 2024-03-28T00:50:18.276 DEBUG:teuthology.orchestra.run.smithi169:osd.3> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.3.service 2024-03-28T00:50:18.278 INFO:tasks.cephadm:Deploying osd.4 on smithi174 with /dev/vg_nvme/lv_3... 2024-03-28T00:50:18.279 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T00:50:18.536 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T00:50:18.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:18 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:18.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:18 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:18.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:18 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:50:18.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:18 smithi169 ceph-mon[32274]: from='osd.3 [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T00:50:18.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:18 smithi169 ceph-mon[32274]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T00:50:18.915 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:18 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:18.915 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:18 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:18.915 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:18 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:50:18.915 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:18 smithi174 ceph-mon[36060]: from='osd.3 [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T00:50:18.915 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:18 smithi174 ceph-mon[36060]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T00:50:18.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:18 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:18.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:18 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:18.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:18 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:50:18.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:18 smithi177 ceph-mon[36206]: from='osd.3 [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T00:50:18.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:18 smithi177 ceph-mon[36206]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T00:50:19.895 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:19 smithi169 ceph-mon[32274]: from='osd.3 [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T00:50:19.896 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:19 smithi169 ceph-mon[32274]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T00:50:19.896 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:19 smithi169 ceph-mon[32274]: from='osd.3 [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-28T00:50:19.896 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:19 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T00:50:19.896 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 28 00:50:19 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-3[51405]: 2024-03-28T00:50:19.646+0000 7fb55b9bb700 -1 osd.3 0 waiting for initial osdmap 2024-03-28T00:50:19.896 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 28 00:50:19 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-3[51405]: 2024-03-28T00:50:19.651+0000 7fb5577ae700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T00:50:19.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:19 smithi177 ceph-mon[36206]: from='osd.3 [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T00:50:19.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:19 smithi177 ceph-mon[36206]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T00:50:19.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:19 smithi177 ceph-mon[36206]: from='osd.3 [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-28T00:50:19.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T00:50:20.001 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:19 smithi174 ceph-mon[36060]: from='osd.3 [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T00:50:20.001 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:19 smithi174 ceph-mon[36060]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T00:50:20.001 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:19 smithi174 ceph-mon[36060]: from='osd.3 [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-28T00:50:20.001 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:19 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T00:50:20.557 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T00:50:20.557 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T00:50:20.557 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-03-28T00:50:20.557 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-03-28T00:50:20.557 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011287 s, 929 MB/s 2024-03-28T00:50:20.557 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-03-28T00:50:20.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:20 smithi169 ceph-mon[32274]: from='osd.3 [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-28T00:50:20.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:20 smithi169 ceph-mon[32274]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T00:50:20.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:20 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T00:50:20.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:20 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T00:50:20.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:20 smithi169 ceph-mon[32274]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T00:50:20.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:20 smithi177 ceph-mon[36206]: from='osd.3 [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-28T00:50:20.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:20 smithi177 ceph-mon[36206]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T00:50:20.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:20 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T00:50:20.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:20 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T00:50:20.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:20 smithi177 ceph-mon[36206]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T00:50:21.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:20 smithi174 ceph-mon[36060]: from='osd.3 [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-28T00:50:21.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:20 smithi174 ceph-mon[36060]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T00:50:21.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:20 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T00:50:21.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:20 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T00:50:21.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:20 smithi174 ceph-mon[36060]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T00:50:21.196 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi174:vg_nvme/lv_3 2024-03-28T00:50:21.440 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T00:50:21.797 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:21 smithi174 ceph-mon[36060]: purged_snaps scrub starts 2024-03-28T00:50:21.798 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:21 smithi174 ceph-mon[36060]: purged_snaps scrub ok 2024-03-28T00:50:21.798 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:21 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T00:50:21.798 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:21 smithi174 ceph-mon[36060]: osd.3 [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866] boot 2024-03-28T00:50:21.798 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:21 smithi174 ceph-mon[36060]: osdmap e25: 4 total, 4 up, 4 in 2024-03-28T00:50:21.798 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:21 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T00:50:21.798 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:21 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:21.798 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:21 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:21.798 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:21 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:50:21.798 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:21 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:50:21.798 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:21 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:21.798 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:21 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:21.798 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:21 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:50:21.799 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:21 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:21.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:21 smithi177 ceph-mon[36206]: purged_snaps scrub starts 2024-03-28T00:50:21.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:21 smithi177 ceph-mon[36206]: purged_snaps scrub ok 2024-03-28T00:50:21.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:21 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T00:50:21.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:21 smithi177 ceph-mon[36206]: osd.3 [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866] boot 2024-03-28T00:50:21.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:21 smithi177 ceph-mon[36206]: osdmap e25: 4 total, 4 up, 4 in 2024-03-28T00:50:21.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:21 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T00:50:21.918 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:21 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:21.918 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:21 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:21.918 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:21 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:50:21.918 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:21 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:50:21.918 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:21 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:21.918 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:21 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:21.918 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:21 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:50:21.918 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:21 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:22.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:21 smithi169 ceph-mon[32274]: purged_snaps scrub starts 2024-03-28T00:50:22.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:21 smithi169 ceph-mon[32274]: purged_snaps scrub ok 2024-03-28T00:50:22.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:21 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T00:50:22.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:21 smithi169 ceph-mon[32274]: osd.3 [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866] boot 2024-03-28T00:50:22.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:21 smithi169 ceph-mon[32274]: osdmap e25: 4 total, 4 up, 4 in 2024-03-28T00:50:22.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:21 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T00:50:22.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:21 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:22.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:21 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:22.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:21 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:50:22.053 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:21 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:50:22.053 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:21 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:22.053 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:21 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:22.053 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:21 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:50:22.053 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:21 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:22.830 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:22 smithi174 ceph-mon[36060]: Detected new or changed devices on smithi169 2024-03-28T00:50:22.830 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:22 smithi174 ceph-mon[36060]: Adjusting osd_memory_target on smithi169 to 8544M 2024-03-28T00:50:22.830 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:22 smithi174 ceph-mon[36060]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T00:50:22.830 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:22 smithi174 ceph-mon[36060]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T00:50:22.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:22 smithi177 ceph-mon[36206]: Detected new or changed devices on smithi169 2024-03-28T00:50:22.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:22 smithi177 ceph-mon[36206]: Adjusting osd_memory_target on smithi169 to 8544M 2024-03-28T00:50:22.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:22 smithi177 ceph-mon[36206]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T00:50:22.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:22 smithi177 ceph-mon[36206]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T00:50:23.015 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.011+0000 7f1d57ba7700 1 -- 172.21.15.174:0/3250866378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d50107eb0 msgr2=0x7f1d5010a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:50:23.015 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.011+0000 7f1d57ba7700 1 --2- 172.21.15.174:0/3250866378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d50107eb0 0x7f1d5010a320 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1d48009990 tx=0x7f1d48009ca0 comp rx=0 tx=0).stop 2024-03-28T00:50:23.015 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.011+0000 7f1d57ba7700 1 -- 172.21.15.174:0/3250866378 shutdown_connections 2024-03-28T00:50:23.015 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.011+0000 7f1d57ba7700 1 --2- 172.21.15.174:0/3250866378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d50107eb0 0x7f1d5010a320 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:50:23.015 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.011+0000 7f1d57ba7700 1 --2- 172.21.15.174:0/3250866378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d50102c50 0x7f1d50105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:50:23.015 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.011+0000 7f1d57ba7700 1 --2- 172.21.15.174:0/3250866378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d50105580 0x7f1d50107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:50:23.015 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.011+0000 7f1d57ba7700 1 -- 172.21.15.174:0/3250866378 >> 172.21.15.174:0/3250866378 conn(0x7f1d500fc660 msgr2=0x7f1d500feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:50:23.015 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.011+0000 7f1d57ba7700 1 -- 172.21.15.174:0/3250866378 shutdown_connections 2024-03-28T00:50:23.015 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.011+0000 7f1d57ba7700 1 -- 172.21.15.174:0/3250866378 wait complete. 2024-03-28T00:50:23.017 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.012+0000 7f1d57ba7700 1 Processor -- start 2024-03-28T00:50:23.017 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.012+0000 7f1d57ba7700 1 -- start start 2024-03-28T00:50:23.017 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.012+0000 7f1d57ba7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d50102c50 0x7f1d501025b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:50:23.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.012+0000 7f1d57ba7700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d50105580 0x7f1d50100c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:50:23.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.012+0000 7f1d57ba7700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d50107eb0 0x7f1d50101140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:50:23.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.012+0000 7f1d57ba7700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5010dda0 con 0x7f1d50102c50 2024-03-28T00:50:23.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.012+0000 7f1d57ba7700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5010dc20 con 0x7f1d50105580 2024-03-28T00:50:23.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.012+0000 7f1d57ba7700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5010df20 con 0x7f1d50107eb0 2024-03-28T00:50:23.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.012+0000 7f1d55142700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d50105580 0x7f1d50100c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:50:23.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.012+0000 7f1d55142700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d50105580 0x7f1d50100c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:47224/0 (socket says 172.21.15.174:47224) 2024-03-28T00:50:23.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.012+0000 7f1d55142700 1 -- 172.21.15.174:0/4164549309 learned_addr learned my addr 172.21.15.174:0/4164549309 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-28T00:50:23.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.012+0000 7f1d55943700 1 --2- 172.21.15.174:0/4164549309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d50102c50 0x7f1d501025b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:50:23.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.012+0000 7f1d56144700 1 --2- 172.21.15.174:0/4164549309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d50107eb0 0x7f1d50101140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:50:23.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.012+0000 7f1d55142700 1 -- 172.21.15.174:0/4164549309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d50107eb0 msgr2=0x7f1d50101140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:50:23.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.012+0000 7f1d55142700 1 --2- 172.21.15.174:0/4164549309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d50107eb0 0x7f1d50101140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:50:23.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.012+0000 7f1d55142700 1 -- 172.21.15.174:0/4164549309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d50102c50 msgr2=0x7f1d501025b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:50:23.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.012+0000 7f1d55142700 1 --2- 172.21.15.174:0/4164549309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d50102c50 0x7f1d501025b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:50:23.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.012+0000 7f1d55142700 1 -- 172.21.15.174:0/4164549309 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d501019f0 con 0x7f1d50105580 2024-03-28T00:50:23.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.012+0000 7f1d55142700 1 --2- 172.21.15.174:0/4164549309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d50105580 0x7f1d50100c00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1d4000c970 tx=0x7f1d4000cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:50:23.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.013+0000 7f1d46ffd700 1 -- 172.21.15.174:0/4164549309 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d40007aa0 con 0x7f1d50105580 2024-03-28T00:50:23.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.013+0000 7f1d46ffd700 1 -- 172.21.15.174:0/4164549309 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d40004d10 con 0x7f1d50105580 2024-03-28T00:50:23.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.013+0000 7f1d57ba7700 1 -- 172.21.15.174:0/4164549309 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d501a2580 con 0x7f1d50105580 2024-03-28T00:50:23.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.013+0000 7f1d46ffd700 1 -- 172.21.15.174:0/4164549309 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4000f450 con 0x7f1d50105580 2024-03-28T00:50:23.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.013+0000 7f1d57ba7700 1 -- 172.21.15.174:0/4164549309 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d501a2a90 con 0x7f1d50105580 2024-03-28T00:50:23.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.014+0000 7f1d57ba7700 1 -- 172.21.15.174:0/4164549309 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d50068e50 con 0x7f1d50105580 2024-03-28T00:50:23.020 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.015+0000 7f1d46ffd700 1 -- 172.21.15.174:0/4164549309 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1d40004750 con 0x7f1d50105580 2024-03-28T00:50:23.020 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.015+0000 7f1d46ffd700 1 --2- 172.21.15.174:0/4164549309 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1d3c0a2910 0x7f1d3c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:50:23.020 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.015+0000 7f1d46ffd700 1 -- 172.21.15.174:0/4164549309 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 3698+0+0 (secure 0 0 0) 0x7f1d400cd5b0 con 0x7f1d50105580 2024-03-28T00:50:23.020 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.015+0000 7f1d55943700 1 --2- 172.21.15.174:0/4164549309 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1d3c0a2910 0x7f1d3c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:50:23.020 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.016+0000 7f1d55943700 1 --2- 172.21.15.174:0/4164549309 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1d3c0a2910 0x7f1d3c0a4dd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c0060b0 tx=0x7f1d4c00c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:50:23.023 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.020+0000 7f1d46ffd700 1 -- 172.21.15.174:0/4164549309 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1d4009b8e0 con 0x7f1d50105580 2024-03-28T00:50:23.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:22 smithi169 ceph-mon[32274]: Detected new or changed devices on smithi169 2024-03-28T00:50:23.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:22 smithi169 ceph-mon[32274]: Adjusting osd_memory_target on smithi169 to 8544M 2024-03-28T00:50:23.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:22 smithi169 ceph-mon[32274]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T00:50:23.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:22 smithi169 ceph-mon[32274]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T00:50:23.227 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:23.223+0000 7f1d57ba7700 1 -- 172.21.15.174:0/4164549309 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d500625d0 con 0x7f1d3c0a2910 2024-03-28T00:50:23.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:23 smithi177 ceph-mon[36206]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T00:50:23.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:23 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:50:23.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:23 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:50:23.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:23 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:23.940 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:23 smithi174 ceph-mon[36060]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T00:50:23.940 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:23 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:50:23.940 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:23 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:50:23.940 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:23 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:24.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:23 smithi169 ceph-mon[32274]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T00:50:24.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:23 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:50:24.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:23 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:50:24.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:23 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:24.953 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:24 smithi174 ceph-mon[36060]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:50:24.953 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:24 smithi174 ceph-mon[36060]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T00:50:24.953 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:24 smithi174 ceph-mon[36060]: pgmap v102: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T00:50:25.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:24 smithi169 ceph-mon[32274]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:50:25.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:24 smithi169 ceph-mon[32274]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T00:50:25.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:24 smithi169 ceph-mon[32274]: pgmap v102: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T00:50:25.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:24 smithi177 ceph-mon[36206]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:50:25.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:24 smithi177 ceph-mon[36206]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T00:50:25.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:24 smithi177 ceph-mon[36206]: pgmap v102: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T00:50:26.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:25 smithi169 ceph-mon[32274]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T00:50:26.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:25 smithi169 ceph-mon[32274]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T00:50:26.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:25 smithi174 ceph-mon[36060]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T00:50:26.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:25 smithi174 ceph-mon[36060]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T00:50:26.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:25 smithi177 ceph-mon[36206]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T00:50:26.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:25 smithi177 ceph-mon[36206]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T00:50:27.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:26 smithi169 ceph-mon[32274]: from='client.? 172.21.15.174:0/34039166' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73da9d2f-f108-40a2-a13d-639b240706f2"}]: dispatch 2024-03-28T00:50:27.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:26 smithi169 ceph-mon[32274]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73da9d2f-f108-40a2-a13d-639b240706f2"}]: dispatch 2024-03-28T00:50:27.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:26 smithi169 ceph-mon[32274]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "73da9d2f-f108-40a2-a13d-639b240706f2"}]': finished 2024-03-28T00:50:27.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:26 smithi169 ceph-mon[32274]: osdmap e30: 5 total, 4 up, 5 in 2024-03-28T00:50:27.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:26 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T00:50:27.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:26 smithi169 ceph-mon[32274]: pgmap v105: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T00:50:27.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:26 smithi169 ceph-mon[32274]: from='client.? 172.21.15.174:0/1832423204' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:50:27.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:26 smithi174 ceph-mon[36060]: from='client.? 172.21.15.174:0/34039166' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73da9d2f-f108-40a2-a13d-639b240706f2"}]: dispatch 2024-03-28T00:50:27.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:26 smithi174 ceph-mon[36060]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73da9d2f-f108-40a2-a13d-639b240706f2"}]: dispatch 2024-03-28T00:50:27.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:26 smithi174 ceph-mon[36060]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "73da9d2f-f108-40a2-a13d-639b240706f2"}]': finished 2024-03-28T00:50:27.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:26 smithi174 ceph-mon[36060]: osdmap e30: 5 total, 4 up, 5 in 2024-03-28T00:50:27.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:26 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T00:50:27.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:26 smithi174 ceph-mon[36060]: pgmap v105: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T00:50:27.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:26 smithi174 ceph-mon[36060]: from='client.? 172.21.15.174:0/1832423204' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:50:27.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:26 smithi177 ceph-mon[36206]: from='client.? 172.21.15.174:0/34039166' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73da9d2f-f108-40a2-a13d-639b240706f2"}]: dispatch 2024-03-28T00:50:27.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:26 smithi177 ceph-mon[36206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73da9d2f-f108-40a2-a13d-639b240706f2"}]: dispatch 2024-03-28T00:50:27.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:26 smithi177 ceph-mon[36206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "73da9d2f-f108-40a2-a13d-639b240706f2"}]': finished 2024-03-28T00:50:27.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:26 smithi177 ceph-mon[36206]: osdmap e30: 5 total, 4 up, 5 in 2024-03-28T00:50:27.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:26 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T00:50:27.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:26 smithi177 ceph-mon[36206]: pgmap v105: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T00:50:27.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:26 smithi177 ceph-mon[36206]: from='client.? 172.21.15.174:0/1832423204' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:50:29.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:29 smithi174 ceph-mon[36060]: pgmap v106: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T00:50:29.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:29 smithi177 ceph-mon[36206]: pgmap v106: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T00:50:29.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:29 smithi169 ceph-mon[32274]: pgmap v106: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T00:50:30.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:30 smithi177 ceph-mon[36206]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-28T00:50:30.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:30 smithi177 ceph-mon[36206]: Cluster is now healthy 2024-03-28T00:50:30.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:30 smithi169 ceph-mon[32274]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-28T00:50:30.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:30 smithi169 ceph-mon[32274]: Cluster is now healthy 2024-03-28T00:50:30.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:30 smithi174 ceph-mon[36060]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-28T00:50:30.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:30 smithi174 ceph-mon[36060]: Cluster is now healthy 2024-03-28T00:50:31.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:31 smithi177 ceph-mon[36206]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-28T00:50:31.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:31 smithi169 ceph-mon[32274]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-28T00:50:31.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:31 smithi174 ceph-mon[36060]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-28T00:50:33.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:33 smithi177 ceph-mon[36206]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T00:50:33.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:33 smithi169 ceph-mon[32274]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T00:50:33.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:33 smithi174 ceph-mon[36060]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T00:50:34.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:34 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T00:50:34.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:34 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:34.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:34 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T00:50:34.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:34 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:34.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:34 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T00:50:34.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:34 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:35.232 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:35 smithi174 ceph-mon[36060]: Deploying daemon osd.4 on smithi174 2024-03-28T00:50:35.232 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:35 smithi174 ceph-mon[36060]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-28T00:50:35.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:35 smithi177 ceph-mon[36206]: Deploying daemon osd.4 on smithi174 2024-03-28T00:50:35.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:35 smithi177 ceph-mon[36206]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-28T00:50:35.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:35 smithi169 ceph-mon[32274]: Deploying daemon osd.4 on smithi174 2024-03-28T00:50:35.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:35 smithi169 ceph-mon[32274]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-28T00:50:37.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:37 smithi174 ceph-mon[36060]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-28T00:50:37.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:37 smithi177 ceph-mon[36206]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-28T00:50:37.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:37 smithi169 ceph-mon[32274]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-28T00:50:39.320 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:39 smithi174 ceph-mon[36060]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-28T00:50:39.320 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:39 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:39.320 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:39 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:39.320 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:39 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:50:39.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:39 smithi177 ceph-mon[36206]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-28T00:50:39.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:39 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:39.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:39 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:39.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:39 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:50:39.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:39 smithi169 ceph-mon[32274]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-28T00:50:39.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:39 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:39.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:39 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:39.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:39 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:50:41.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:41 smithi174 ceph-mon[36060]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-28T00:50:41.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:41 smithi177 ceph-mon[36206]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-28T00:50:41.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:41 smithi169 ceph-mon[32274]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-28T00:50:41.945 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 4 on host 'smithi174' 2024-03-28T00:50:41.946 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:41.939+0000 7f1d46ffd700 1 -- 172.21.15.174:0/4164549309 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1d500625d0 con 0x7f1d3c0a2910 2024-03-28T00:50:41.946 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:41.941+0000 7f1d57ba7700 1 -- 172.21.15.174:0/4164549309 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1d3c0a2910 msgr2=0x7f1d3c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:50:41.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:41.941+0000 7f1d57ba7700 1 --2- 172.21.15.174:0/4164549309 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1d3c0a2910 0x7f1d3c0a4dd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c0060b0 tx=0x7f1d4c00c040 comp rx=0 tx=0).stop 2024-03-28T00:50:41.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:41.941+0000 7f1d57ba7700 1 -- 172.21.15.174:0/4164549309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d50105580 msgr2=0x7f1d50100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:50:41.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:41.941+0000 7f1d57ba7700 1 --2- 172.21.15.174:0/4164549309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d50105580 0x7f1d50100c00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1d4000c970 tx=0x7f1d4000cc80 comp rx=0 tx=0).stop 2024-03-28T00:50:41.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:41.941+0000 7f1d57ba7700 1 -- 172.21.15.174:0/4164549309 shutdown_connections 2024-03-28T00:50:41.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:41.941+0000 7f1d57ba7700 1 --2- 172.21.15.174:0/4164549309 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1d3c0a2910 0x7f1d3c0a4dd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:50:41.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:41.941+0000 7f1d57ba7700 1 --2- 172.21.15.174:0/4164549309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d50107eb0 0x7f1d50101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:50:41.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:41.941+0000 7f1d57ba7700 1 --2- 172.21.15.174:0/4164549309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d50102c50 0x7f1d501025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:50:41.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:41.941+0000 7f1d57ba7700 1 --2- 172.21.15.174:0/4164549309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d50105580 0x7f1d50100c00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:50:41.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:41.941+0000 7f1d57ba7700 1 -- 172.21.15.174:0/4164549309 >> 172.21.15.174:0/4164549309 conn(0x7f1d500fc660 msgr2=0x7f1d50103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:50:41.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:41.941+0000 7f1d57ba7700 1 -- 172.21.15.174:0/4164549309 shutdown_connections 2024-03-28T00:50:41.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:50:41.941+0000 7f1d57ba7700 1 -- 172.21.15.174:0/4164549309 wait complete. 2024-03-28T00:50:42.750 DEBUG:teuthology.orchestra.run.smithi174:osd.4> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.4.service 2024-03-28T00:50:42.753 INFO:tasks.cephadm:Deploying osd.5 on smithi177 with /dev/vg_nvme/lv_3... 2024-03-28T00:50:42.753 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T00:50:43.000 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T00:50:43.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:42 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:43.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:42 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:43.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:42 smithi174 ceph-mon[36060]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T00:50:43.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:42 smithi174 ceph-mon[36060]: from='osd.4 [v2:172.21.15.174:6808/701854773,v1:172.21.15.174:6809/701854773]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T00:50:43.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:42 smithi174 ceph-mon[36060]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T00:50:43.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:42 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:43.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:42 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:43.121 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:42 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:43.121 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:42 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:43.121 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:42 smithi177 ceph-mon[36206]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T00:50:43.121 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:42 smithi177 ceph-mon[36206]: from='osd.4 [v2:172.21.15.174:6808/701854773,v1:172.21.15.174:6809/701854773]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T00:50:43.121 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:42 smithi177 ceph-mon[36206]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T00:50:43.121 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:42 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:43.121 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:42 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:43.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:42 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:43.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:42 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:43.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:42 smithi169 ceph-mon[32274]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T00:50:43.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:42 smithi169 ceph-mon[32274]: from='osd.4 [v2:172.21.15.174:6808/701854773,v1:172.21.15.174:6809/701854773]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T00:50:43.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:42 smithi169 ceph-mon[32274]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T00:50:43.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:42 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:43.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:42 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:44.051 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:43 smithi177 ceph-mon[36206]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T00:50:44.051 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:43 smithi177 ceph-mon[36206]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T00:50:44.051 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:43 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T00:50:44.051 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:43 smithi177 ceph-mon[36206]: from='osd.4 [v2:172.21.15.174:6808/701854773,v1:172.21.15.174:6809/701854773]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:50:44.051 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:43 smithi177 ceph-mon[36206]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:50:44.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:43 smithi174 ceph-mon[36060]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T00:50:44.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:43 smithi174 ceph-mon[36060]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T00:50:44.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:43 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T00:50:44.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:43 smithi174 ceph-mon[36060]: from='osd.4 [v2:172.21.15.174:6808/701854773,v1:172.21.15.174:6809/701854773]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:50:44.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:43 smithi174 ceph-mon[36060]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:50:44.203 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:43 smithi169 ceph-mon[32274]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T00:50:44.203 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:43 smithi169 ceph-mon[32274]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T00:50:44.203 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:43 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T00:50:44.203 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:43 smithi169 ceph-mon[32274]: from='osd.4 [v2:172.21.15.174:6808/701854773,v1:172.21.15.174:6809/701854773]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:50:44.204 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:43 smithi169 ceph-mon[32274]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:50:44.597 INFO:journalctl@ceph.osd.4.smithi174.stdout:Mar 28 00:50:44 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-4[47179]: 2024-03-28T00:50:44.317+0000 7f12aa108700 -1 osd.4 0 waiting for initial osdmap 2024-03-28T00:50:44.597 INFO:journalctl@ceph.osd.4.smithi174.stdout:Mar 28 00:50:44 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-4[47179]: 2024-03-28T00:50:44.324+0000 7f12a1ef3700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T00:50:44.901 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T00:50:44.901 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T00:50:44.901 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-03-28T00:50:44.901 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-03-28T00:50:44.901 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0148747 s, 705 MB/s 2024-03-28T00:50:44.901 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-03-28T00:50:44.926 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:44 smithi177 ceph-mon[36206]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-28T00:50:44.926 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:44 smithi177 ceph-mon[36206]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T00:50:44.926 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:44 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T00:50:44.926 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:44 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T00:50:44.926 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:44 smithi177 ceph-mon[36206]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T00:50:44.926 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:44 smithi177 ceph-mon[36206]: from='osd.4 ' entity='osd.4' 2024-03-28T00:50:45.215 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:44 smithi169 ceph-mon[32274]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-28T00:50:45.215 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:44 smithi169 ceph-mon[32274]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T00:50:45.216 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:44 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T00:50:45.216 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:44 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T00:50:45.216 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:44 smithi169 ceph-mon[32274]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T00:50:45.216 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:44 smithi169 ceph-mon[32274]: from='osd.4 ' entity='osd.4' 2024-03-28T00:50:45.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:44 smithi174 ceph-mon[36060]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-28T00:50:45.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:44 smithi174 ceph-mon[36060]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T00:50:45.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:44 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T00:50:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:44 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T00:50:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:44 smithi174 ceph-mon[36060]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-28T00:50:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:44 smithi174 ceph-mon[36060]: from='osd.4 ' entity='osd.4' 2024-03-28T00:50:45.493 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi177:vg_nvme/lv_3 2024-03-28T00:50:45.731 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T00:50:46.094 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:45 smithi177 ceph-mon[36206]: purged_snaps scrub starts 2024-03-28T00:50:46.094 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:45 smithi177 ceph-mon[36206]: purged_snaps scrub ok 2024-03-28T00:50:46.094 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:45 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T00:50:46.094 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:45 smithi177 ceph-mon[36206]: osd.4 [v2:172.21.15.174:6808/701854773,v1:172.21.15.174:6809/701854773] boot 2024-03-28T00:50:46.094 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:45 smithi177 ceph-mon[36206]: osdmap e33: 5 total, 5 up, 5 in 2024-03-28T00:50:46.094 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:45 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T00:50:46.197 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:45 smithi174 ceph-mon[36060]: purged_snaps scrub starts 2024-03-28T00:50:46.197 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:45 smithi174 ceph-mon[36060]: purged_snaps scrub ok 2024-03-28T00:50:46.198 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:45 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T00:50:46.198 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:45 smithi174 ceph-mon[36060]: osd.4 [v2:172.21.15.174:6808/701854773,v1:172.21.15.174:6809/701854773] boot 2024-03-28T00:50:46.198 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:45 smithi174 ceph-mon[36060]: osdmap e33: 5 total, 5 up, 5 in 2024-03-28T00:50:46.198 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:45 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T00:50:46.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:45 smithi169 ceph-mon[32274]: purged_snaps scrub starts 2024-03-28T00:50:46.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:45 smithi169 ceph-mon[32274]: purged_snaps scrub ok 2024-03-28T00:50:46.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:45 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T00:50:46.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:45 smithi169 ceph-mon[32274]: osd.4 [v2:172.21.15.174:6808/701854773,v1:172.21.15.174:6809/701854773] boot 2024-03-28T00:50:46.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:45 smithi169 ceph-mon[32274]: osdmap e33: 5 total, 5 up, 5 in 2024-03-28T00:50:46.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:45 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T00:50:46.994 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:46 smithi177 ceph-mon[36206]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:50:46.994 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:46 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:46.994 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:46 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:46.994 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:46 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:50:46.994 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:46 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:50:46.995 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:46 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:46.995 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:46 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:46.995 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:46 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:50:46.995 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:46 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:46.995 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:46 smithi177 ceph-mon[36206]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T00:50:46.995 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:46 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:50:46.995 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:46 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:46.995 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:46 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:50:46.995 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:46 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:47.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:46 smithi169 ceph-mon[32274]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:50:47.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:46 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:47.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:46 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:47.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:46 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:50:47.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:46 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:50:47.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:46 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:47.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:46 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:47.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:46 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:50:47.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:46 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:47.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:46 smithi169 ceph-mon[32274]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T00:50:47.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:46 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:50:47.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:46 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:47.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:46 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:50:47.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:46 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:47.321 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.317+0000 7f0b51d21700 1 -- 172.21.15.177:0/635470046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b4c105580 msgr2=0x7f0b4c107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:50:47.321 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.317+0000 7f0b51d21700 1 --2- 172.21.15.177:0/635470046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b4c105580 0x7f0b4c107970 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0b40009990 tx=0x7f0b40009ca0 comp rx=0 tx=0).stop 2024-03-28T00:50:47.321 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.317+0000 7f0b51d21700 1 -- 172.21.15.177:0/635470046 shutdown_connections 2024-03-28T00:50:47.321 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.317+0000 7f0b51d21700 1 --2- 172.21.15.177:0/635470046 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b4c107eb0 0x7f0b4c10a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:50:47.321 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.317+0000 7f0b51d21700 1 --2- 172.21.15.177:0/635470046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b4c102c50 0x7f0b4c105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:50:47.321 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.317+0000 7f0b51d21700 1 --2- 172.21.15.177:0/635470046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b4c105580 0x7f0b4c107970 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:50:47.321 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.317+0000 7f0b51d21700 1 -- 172.21.15.177:0/635470046 >> 172.21.15.177:0/635470046 conn(0x7f0b4c0fc680 msgr2=0x7f0b4c0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:50:47.321 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.317+0000 7f0b51d21700 1 -- 172.21.15.177:0/635470046 shutdown_connections 2024-03-28T00:50:47.321 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.317+0000 7f0b51d21700 1 -- 172.21.15.177:0/635470046 wait complete. 2024-03-28T00:50:47.321 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.317+0000 7f0b51d21700 1 Processor -- start 2024-03-28T00:50:47.324 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.318+0000 7f0b51d21700 1 -- start start 2024-03-28T00:50:47.324 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.318+0000 7f0b51d21700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b4c102c50 0x7f0b4c198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:50:47.324 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.318+0000 7f0b51d21700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b4c105580 0x7f0b4c198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:50:47.324 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.318+0000 7f0b51d21700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b4c107eb0 0x7f0b4c19f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:50:47.324 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.318+0000 7f0b51d21700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c10ddd0 con 0x7f0b4c107eb0 2024-03-28T00:50:47.324 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.318+0000 7f0b51d21700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c10dc50 con 0x7f0b4c105580 2024-03-28T00:50:47.324 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.318+0000 7f0b51d21700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c10df50 con 0x7f0b4c102c50 2024-03-28T00:50:47.324 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.318+0000 7f0b4b7fe700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b4c102c50 0x7f0b4c198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:50:47.324 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.318+0000 7f0b4b7fe700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b4c102c50 0x7f0b4c198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:43140/0 (socket says 172.21.15.177:43140) 2024-03-28T00:50:47.324 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.318+0000 7f0b4b7fe700 1 -- 172.21.15.177:0/113118033 learned_addr learned my addr 172.21.15.177:0/113118033 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-28T00:50:47.324 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.318+0000 7f0b4affd700 1 --2- 172.21.15.177:0/113118033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b4c105580 0x7f0b4c198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:50:47.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.318+0000 7f0b4bfff700 1 --2- 172.21.15.177:0/113118033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b4c107eb0 0x7f0b4c19f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:50:47.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.318+0000 7f0b4b7fe700 1 -- 172.21.15.177:0/113118033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b4c105580 msgr2=0x7f0b4c198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:50:47.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.318+0000 7f0b4b7fe700 1 --2- 172.21.15.177:0/113118033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b4c105580 0x7f0b4c198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:50:47.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.318+0000 7f0b4b7fe700 1 -- 172.21.15.177:0/113118033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b4c107eb0 msgr2=0x7f0b4c19f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:50:47.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.318+0000 7f0b4b7fe700 1 --2- 172.21.15.177:0/113118033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b4c107eb0 0x7f0b4c19f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:50:47.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.318+0000 7f0b4b7fe700 1 -- 172.21.15.177:0/113118033 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b4c1a0060 con 0x7f0b4c102c50 2024-03-28T00:50:47.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.318+0000 7f0b4bfff700 1 --2- 172.21.15.177:0/113118033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b4c107eb0 0x7f0b4c19f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:50:47.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.318+0000 7f0b4b7fe700 1 --2- 172.21.15.177:0/113118033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b4c102c50 0x7f0b4c198400 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c00c9a0 tx=0x7f0b3c00ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:50:47.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.319+0000 7f0b48ff9700 1 -- 172.21.15.177:0/113118033 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c007a60 con 0x7f0b4c102c50 2024-03-28T00:50:47.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.319+0000 7f0b51d21700 1 -- 172.21.15.177:0/113118033 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b4c1a0350 con 0x7f0b4c102c50 2024-03-28T00:50:47.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.319+0000 7f0b48ff9700 1 -- 172.21.15.177:0/113118033 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b3c004d10 con 0x7f0b4c102c50 2024-03-28T00:50:47.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.319+0000 7f0b48ff9700 1 -- 172.21.15.177:0/113118033 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c005690 con 0x7f0b4c102c50 2024-03-28T00:50:47.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.319+0000 7f0b51d21700 1 -- 172.21.15.177:0/113118033 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b4c1a0840 con 0x7f0b4c102c50 2024-03-28T00:50:47.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.320+0000 7f0b51d21700 1 -- 172.21.15.177:0/113118033 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b4c068e50 con 0x7f0b4c102c50 2024-03-28T00:50:47.326 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.320+0000 7f0b48ff9700 1 -- 172.21.15.177:0/113118033 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0b3c007c00 con 0x7f0b4c102c50 2024-03-28T00:50:47.326 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.321+0000 7f0b48ff9700 1 --2- 172.21.15.177:0/113118033 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0b340a2890 0x7f0b340a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:50:47.326 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.321+0000 7f0b48ff9700 1 -- 172.21.15.177:0/113118033 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f0b3c0cd7e0 con 0x7f0b4c102c50 2024-03-28T00:50:47.326 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.321+0000 7f0b4affd700 1 --2- 172.21.15.177:0/113118033 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0b340a2890 0x7f0b340a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:50:47.327 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.322+0000 7f0b4affd700 1 --2- 172.21.15.177:0/113118033 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0b340a2890 0x7f0b340a4d50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0b4000b5c0 tx=0x7f0b40005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:50:47.329 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.326+0000 7f0b48ff9700 1 -- 172.21.15.177:0/113118033 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0b3c09b980 con 0x7f0b4c102c50 2024-03-28T00:50:47.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:46 smithi174 ceph-mon[36060]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:50:47.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:46 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:47.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:46 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:47.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:46 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:50:47.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:46 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:50:47.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:46 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:47.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:46 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:47.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:46 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:50:47.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:46 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:47.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:46 smithi174 ceph-mon[36060]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T00:50:47.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:46 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:50:47.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:46 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:47.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:46 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:50:47.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:46 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:50:47.530 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:50:47.527+0000 7f0b51d21700 1 -- 172.21.15.177:0/113118033 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b4c02de90 con 0x7f0b340a2890 2024-03-28T00:50:48.249 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:47 smithi177 ceph-mon[36206]: Detected new or changed devices on smithi174 2024-03-28T00:50:48.249 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:47 smithi177 ceph-mon[36206]: Adjusting osd_memory_target on smithi174 to 8540M 2024-03-28T00:50:48.249 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:47 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:50:48.249 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:47 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:50:48.249 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:47 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:48.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:47 smithi169 ceph-mon[32274]: Detected new or changed devices on smithi174 2024-03-28T00:50:48.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:47 smithi169 ceph-mon[32274]: Adjusting osd_memory_target on smithi174 to 8540M 2024-03-28T00:50:48.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:47 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:50:48.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:47 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:50:48.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:47 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:48.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:47 smithi174 ceph-mon[36060]: Detected new or changed devices on smithi174 2024-03-28T00:50:48.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:47 smithi174 ceph-mon[36060]: Adjusting osd_memory_target on smithi174 to 8540M 2024-03-28T00:50:48.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:47 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:50:48.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:47 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:50:48.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:47 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:49.261 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:48 smithi177 ceph-mon[36206]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:50:49.262 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:48 smithi177 ceph-mon[36206]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:50:49.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:48 smithi169 ceph-mon[32274]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:50:49.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:48 smithi169 ceph-mon[32274]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:50:49.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:48 smithi174 ceph-mon[36060]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:50:49.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:48 smithi174 ceph-mon[36060]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:50:51.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:51 smithi174 ceph-mon[36060]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:50:51.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:51 smithi174 ceph-mon[36060]: from='client.? 172.21.15.177:0/302109281' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd80a3c2-8ec1-495c-a8cb-b662242df637"}]: dispatch 2024-03-28T00:50:51.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:51 smithi174 ceph-mon[36060]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd80a3c2-8ec1-495c-a8cb-b662242df637"}]: dispatch 2024-03-28T00:50:51.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:51 smithi174 ceph-mon[36060]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd80a3c2-8ec1-495c-a8cb-b662242df637"}]': finished 2024-03-28T00:50:51.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:51 smithi174 ceph-mon[36060]: osdmap e35: 6 total, 5 up, 6 in 2024-03-28T00:50:51.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:51 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T00:50:51.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:51 smithi174 ceph-mon[36060]: from='client.? 172.21.15.177:0/16417438' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:50:51.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:51 smithi177 ceph-mon[36206]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:50:51.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:51 smithi177 ceph-mon[36206]: from='client.? 172.21.15.177:0/302109281' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd80a3c2-8ec1-495c-a8cb-b662242df637"}]: dispatch 2024-03-28T00:50:51.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:51 smithi177 ceph-mon[36206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd80a3c2-8ec1-495c-a8cb-b662242df637"}]: dispatch 2024-03-28T00:50:51.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:51 smithi177 ceph-mon[36206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd80a3c2-8ec1-495c-a8cb-b662242df637"}]': finished 2024-03-28T00:50:51.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:51 smithi177 ceph-mon[36206]: osdmap e35: 6 total, 5 up, 6 in 2024-03-28T00:50:51.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:51 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T00:50:51.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:51 smithi177 ceph-mon[36206]: from='client.? 172.21.15.177:0/16417438' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:50:51.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:51 smithi169 ceph-mon[32274]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:50:51.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:51 smithi169 ceph-mon[32274]: from='client.? 172.21.15.177:0/302109281' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd80a3c2-8ec1-495c-a8cb-b662242df637"}]: dispatch 2024-03-28T00:50:51.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:51 smithi169 ceph-mon[32274]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd80a3c2-8ec1-495c-a8cb-b662242df637"}]: dispatch 2024-03-28T00:50:51.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:51 smithi169 ceph-mon[32274]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd80a3c2-8ec1-495c-a8cb-b662242df637"}]': finished 2024-03-28T00:50:51.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:51 smithi169 ceph-mon[32274]: osdmap e35: 6 total, 5 up, 6 in 2024-03-28T00:50:51.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:51 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T00:50:51.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:51 smithi169 ceph-mon[32274]: from='client.? 172.21.15.177:0/16417438' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:50:53.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:53 smithi174 ceph-mon[36060]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:50:53.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:53 smithi177 ceph-mon[36206]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:50:53.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:53 smithi169 ceph-mon[32274]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:50:55.349 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:55 smithi177 ceph-mon[36206]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:50:55.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:55 smithi174 ceph-mon[36060]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:50:55.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:55 smithi169 ceph-mon[32274]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:50:57.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:57 smithi174 ceph-mon[36060]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:50:57.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:57 smithi177 ceph-mon[36206]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:50:57.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:57 smithi169 ceph-mon[32274]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:50:58.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:58 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T00:50:58.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:58 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:58.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:58 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T00:50:58.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:58 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:58.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:58 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T00:50:58.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:58 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:50:59.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:59 smithi174 ceph-mon[36060]: Deploying daemon osd.5 on smithi177 2024-03-28T00:50:59.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:50:59 smithi174 ceph-mon[36060]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:50:59.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:59 smithi177 ceph-mon[36206]: Deploying daemon osd.5 on smithi177 2024-03-28T00:50:59.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:50:59 smithi177 ceph-mon[36206]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:50:59.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:59 smithi169 ceph-mon[32274]: Deploying daemon osd.5 on smithi177 2024-03-28T00:50:59.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:50:59 smithi169 ceph-mon[32274]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:51:01.375 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:01 smithi177 ceph-mon[36206]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:51:01.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:01 smithi169 ceph-mon[32274]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:51:01.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:01 smithi174 ceph-mon[36060]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:51:03.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:03 smithi177 ceph-mon[36206]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:51:03.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:03 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:03.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:03 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:03.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:03 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:51:03.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:03 smithi169 ceph-mon[32274]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:51:03.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:03 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:03.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:03 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:03.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:03 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:51:03.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:03 smithi174 ceph-mon[36060]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:51:03.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:03 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:03.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:03 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:03.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:03 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:51:05.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:05 smithi177 ceph-mon[36206]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:51:05.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:05 smithi169 ceph-mon[32274]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:51:05.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:05 smithi174 ceph-mon[36060]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:51:06.601 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 5 on host 'smithi177' 2024-03-28T00:51:06.602 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:51:06.595+0000 7f0b48ff9700 1 -- 172.21.15.177:0/113118033 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0b4000b7d0 con 0x7f0b340a2890 2024-03-28T00:51:06.602 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:51:06.598+0000 7f0b51d21700 1 -- 172.21.15.177:0/113118033 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0b340a2890 msgr2=0x7f0b340a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:51:06.602 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:51:06.598+0000 7f0b51d21700 1 --2- 172.21.15.177:0/113118033 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0b340a2890 0x7f0b340a4d50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0b4000b5c0 tx=0x7f0b40005fd0 comp rx=0 tx=0).stop 2024-03-28T00:51:06.602 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:51:06.598+0000 7f0b51d21700 1 -- 172.21.15.177:0/113118033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b4c102c50 msgr2=0x7f0b4c198400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:51:06.603 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:51:06.598+0000 7f0b51d21700 1 --2- 172.21.15.177:0/113118033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b4c102c50 0x7f0b4c198400 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c00c9a0 tx=0x7f0b3c00ccb0 comp rx=0 tx=0).stop 2024-03-28T00:51:06.603 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:51:06.598+0000 7f0b51d21700 1 -- 172.21.15.177:0/113118033 shutdown_connections 2024-03-28T00:51:06.603 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:51:06.598+0000 7f0b51d21700 1 --2- 172.21.15.177:0/113118033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b4c107eb0 0x7f0b4c19f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:06.603 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:51:06.598+0000 7f0b51d21700 1 --2- 172.21.15.177:0/113118033 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0b340a2890 0x7f0b340a4d50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:06.603 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:51:06.598+0000 7f0b51d21700 1 --2- 172.21.15.177:0/113118033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b4c102c50 0x7f0b4c198400 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:06.603 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:51:06.598+0000 7f0b51d21700 1 --2- 172.21.15.177:0/113118033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b4c105580 0x7f0b4c198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:06.603 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:51:06.598+0000 7f0b51d21700 1 -- 172.21.15.177:0/113118033 >> 172.21.15.177:0/113118033 conn(0x7f0b4c0fc680 msgr2=0x7f0b4c103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:51:06.603 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:51:06.598+0000 7f0b51d21700 1 -- 172.21.15.177:0/113118033 shutdown_connections 2024-03-28T00:51:06.603 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:51:06.598+0000 7f0b51d21700 1 -- 172.21.15.177:0/113118033 wait complete. 2024-03-28T00:51:07.222 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:07 smithi177 ceph-mon[36206]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:51:07.222 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:07 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:07.223 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:07 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:07.223 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:07 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:07.223 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:07 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:07.223 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:07 smithi177 ceph-mon[36206]: from='osd.5 [v2:172.21.15.177:6808/3583242703,v1:172.21.15.177:6809/3583242703]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T00:51:07.223 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:07 smithi177 ceph-mon[36206]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T00:51:07.255 DEBUG:teuthology.orchestra.run.smithi177:osd.5> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.5.service 2024-03-28T00:51:07.257 INFO:tasks.cephadm:Deploying osd.6 on smithi169 with /dev/vg_nvme/lv_2... 2024-03-28T00:51:07.257 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T00:51:07.506 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:51:07.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:07 smithi169 ceph-mon[32274]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:51:07.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:07 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:07.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:07 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:07.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:07 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:07.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:07 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:07.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:07 smithi169 ceph-mon[32274]: from='osd.5 [v2:172.21.15.177:6808/3583242703,v1:172.21.15.177:6809/3583242703]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T00:51:07.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:07 smithi169 ceph-mon[32274]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T00:51:07.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:07 smithi174 ceph-mon[36060]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:51:07.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:07 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:07.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:07 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:07.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:07 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:07.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:07 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:07.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:07 smithi174 ceph-mon[36060]: from='osd.5 [v2:172.21.15.177:6808/3583242703,v1:172.21.15.177:6809/3583242703]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T00:51:07.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:07 smithi174 ceph-mon[36060]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T00:51:08.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:08 smithi169 ceph-mon[32274]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T00:51:08.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:08 smithi169 ceph-mon[32274]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T00:51:08.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:08 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T00:51:08.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:08 smithi169 ceph-mon[32274]: from='osd.5 [v2:172.21.15.177:6808/3583242703,v1:172.21.15.177:6809/3583242703]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:51:08.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:08 smithi169 ceph-mon[32274]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:51:08.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:08 smithi169 ceph-mon[32274]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:51:08.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:08 smithi177 ceph-mon[36206]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T00:51:08.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:08 smithi177 ceph-mon[36206]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T00:51:08.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:08 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T00:51:08.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:08 smithi177 ceph-mon[36206]: from='osd.5 [v2:172.21.15.177:6808/3583242703,v1:172.21.15.177:6809/3583242703]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:51:08.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:08 smithi177 ceph-mon[36206]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:51:08.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:08 smithi177 ceph-mon[36206]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:51:08.917 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 00:51:08 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T00:51:08.671+0000 7faf8787c700 -1 osd.5 0 waiting for initial osdmap 2024-03-28T00:51:08.918 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 00:51:08 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T00:51:08.679+0000 7faf8366f700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T00:51:09.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:08 smithi174 ceph-mon[36060]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T00:51:09.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:08 smithi174 ceph-mon[36060]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T00:51:09.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:08 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T00:51:09.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:08 smithi174 ceph-mon[36060]: from='osd.5 [v2:172.21.15.177:6808/3583242703,v1:172.21.15.177:6809/3583242703]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:51:09.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:08 smithi174 ceph-mon[36060]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:51:09.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:08 smithi174 ceph-mon[36060]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:51:09.370 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T00:51:09.370 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T00:51:09.370 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2024-03-28T00:51:09.370 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2024-03-28T00:51:09.370 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115449 s, 908 MB/s 2024-03-28T00:51:09.370 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2024-03-28T00:51:09.820 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:09 smithi177 ceph-mon[36206]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-28T00:51:09.821 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:09 smithi177 ceph-mon[36206]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T00:51:09.821 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:09 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T00:51:09.821 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:09 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T00:51:10.037 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:09 smithi169 ceph-mon[32274]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-28T00:51:10.037 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:09 smithi169 ceph-mon[32274]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T00:51:10.038 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:09 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T00:51:10.038 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:09 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T00:51:10.064 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_2 2024-03-28T00:51:10.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:09 smithi174 ceph-mon[36060]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-28T00:51:10.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:09 smithi174 ceph-mon[36060]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T00:51:10.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:09 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T00:51:10.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:09 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T00:51:10.313 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:51:10.715 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:10 smithi169 ceph-mon[32274]: purged_snaps scrub starts 2024-03-28T00:51:10.716 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:10 smithi169 ceph-mon[32274]: purged_snaps scrub ok 2024-03-28T00:51:10.716 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T00:51:10.716 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:10 smithi169 ceph-mon[32274]: osd.5 [v2:172.21.15.177:6808/3583242703,v1:172.21.15.177:6809/3583242703] boot 2024-03-28T00:51:10.716 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:10 smithi169 ceph-mon[32274]: osdmap e38: 6 total, 6 up, 6 in 2024-03-28T00:51:10.716 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T00:51:10.716 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:10 smithi169 ceph-mon[32274]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:51:10.716 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:10 smithi169 ceph-mon[32274]: Detected new or changed devices on smithi177 2024-03-28T00:51:10.716 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:10.716 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:10.716 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:51:10.716 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:51:10.716 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:10 smithi169 ceph-mon[32274]: Adjusting osd_memory_target on smithi177 to 8540M 2024-03-28T00:51:10.716 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:10.716 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:10.717 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:51:10.717 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:10.717 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:51:10.717 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:10.717 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:51:10.717 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:11.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:10 smithi174 ceph-mon[36060]: purged_snaps scrub starts 2024-03-28T00:51:11.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:10 smithi174 ceph-mon[36060]: purged_snaps scrub ok 2024-03-28T00:51:11.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T00:51:11.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:10 smithi174 ceph-mon[36060]: osd.5 [v2:172.21.15.177:6808/3583242703,v1:172.21.15.177:6809/3583242703] boot 2024-03-28T00:51:11.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:10 smithi174 ceph-mon[36060]: osdmap e38: 6 total, 6 up, 6 in 2024-03-28T00:51:11.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T00:51:11.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:10 smithi174 ceph-mon[36060]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:51:11.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:10 smithi174 ceph-mon[36060]: Detected new or changed devices on smithi177 2024-03-28T00:51:11.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:11.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:11.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:51:11.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:51:11.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:10 smithi174 ceph-mon[36060]: Adjusting osd_memory_target on smithi177 to 8540M 2024-03-28T00:51:11.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:11.110 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:11.110 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:51:11.110 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:11.110 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:51:11.110 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:11.110 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:51:11.110 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:11.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:10 smithi177 ceph-mon[36206]: purged_snaps scrub starts 2024-03-28T00:51:11.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:10 smithi177 ceph-mon[36206]: purged_snaps scrub ok 2024-03-28T00:51:11.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T00:51:11.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:10 smithi177 ceph-mon[36206]: osd.5 [v2:172.21.15.177:6808/3583242703,v1:172.21.15.177:6809/3583242703] boot 2024-03-28T00:51:11.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:10 smithi177 ceph-mon[36206]: osdmap e38: 6 total, 6 up, 6 in 2024-03-28T00:51:11.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T00:51:11.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:10 smithi177 ceph-mon[36206]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T00:51:11.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:10 smithi177 ceph-mon[36206]: Detected new or changed devices on smithi177 2024-03-28T00:51:11.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:11.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:11.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:51:11.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:51:11.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:10 smithi177 ceph-mon[36206]: Adjusting osd_memory_target on smithi177 to 8540M 2024-03-28T00:51:11.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:11.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:11.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:51:11.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:11.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:51:11.169 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:11.169 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:51:11.169 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:11.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.807+0000 7fb16f105700 1 -- 172.21.15.169:0/4178555555 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1681022c0 msgr2=0x7fb168102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:51:11.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.807+0000 7fb16f105700 1 --2- 172.21.15.169:0/4178555555 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1681022c0 0x7fb168102720 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb1540099c0 tx=0x7fb154009cd0 comp rx=0 tx=0).stop 2024-03-28T00:51:11.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.807+0000 7fb16f105700 1 -- 172.21.15.169:0/4178555555 shutdown_connections 2024-03-28T00:51:11.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.807+0000 7fb16f105700 1 --2- 172.21.15.169:0/4178555555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb168102c60 0x7fb168109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:11.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.807+0000 7fb16f105700 1 --2- 172.21.15.169:0/4178555555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1681010c0 0x7fb1681014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:11.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.807+0000 7fb16f105700 1 --2- 172.21.15.169:0/4178555555 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1681022c0 0x7fb168102720 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:11.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.807+0000 7fb16f105700 1 -- 172.21.15.169:0/4178555555 >> 172.21.15.169:0/4178555555 conn(0x7fb1680fc680 msgr2=0x7fb1680feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:51:11.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.807+0000 7fb16f105700 1 -- 172.21.15.169:0/4178555555 shutdown_connections 2024-03-28T00:51:11.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.807+0000 7fb16f105700 1 -- 172.21.15.169:0/4178555555 wait complete. 2024-03-28T00:51:11.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.808+0000 7fb16f105700 1 Processor -- start 2024-03-28T00:51:11.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.808+0000 7fb16f105700 1 -- start start 2024-03-28T00:51:11.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.808+0000 7fb16f105700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1681010c0 0x7fb1681983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:51:11.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.808+0000 7fb16f105700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1681022c0 0x7fb168198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:51:11.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.808+0000 7fb16f105700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb168102c60 0x7fb16819f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:51:11.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.808+0000 7fb16f105700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16810ccd0 con 0x7fb168102c60 2024-03-28T00:51:11.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.808+0000 7fb16f105700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16810cb50 con 0x7fb1681010c0 2024-03-28T00:51:11.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.808+0000 7fb16f105700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16810ce50 con 0x7fb1681022c0 2024-03-28T00:51:11.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.808+0000 7fb16d6a2700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb168102c60 0x7fb16819f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:51:11.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.808+0000 7fb16cea1700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1681010c0 0x7fb1681983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:51:11.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.808+0000 7fb16d6a2700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb168102c60 0x7fb16819f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:45068/0 (socket says 172.21.15.169:45068) 2024-03-28T00:51:11.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.808+0000 7fb16d6a2700 1 -- 172.21.15.169:0/2483314949 learned_addr learned my addr 172.21.15.169:0/2483314949 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:51:11.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.808+0000 7fb167fff700 1 --2- 172.21.15.169:0/2483314949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1681022c0 0x7fb168198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:51:11.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.808+0000 7fb16d6a2700 1 -- 172.21.15.169:0/2483314949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1681022c0 msgr2=0x7fb168198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:51:11.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.808+0000 7fb16d6a2700 1 --2- 172.21.15.169:0/2483314949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1681022c0 0x7fb168198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:11.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.808+0000 7fb16d6a2700 1 -- 172.21.15.169:0/2483314949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1681010c0 msgr2=0x7fb1681983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:51:11.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.808+0000 7fb16d6a2700 1 --2- 172.21.15.169:0/2483314949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1681010c0 0x7fb1681983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:11.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.808+0000 7fb16d6a2700 1 -- 172.21.15.169:0/2483314949 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1681a0030 con 0x7fb168102c60 2024-03-28T00:51:11.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.808+0000 7fb16d6a2700 1 --2- 172.21.15.169:0/2483314949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb168102c60 0x7fb16819f990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb16000add0 tx=0x7fb16000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:51:11.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.809+0000 7fb165ffb700 1 -- 172.21.15.169:0/2483314949 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb160017020 con 0x7fb168102c60 2024-03-28T00:51:11.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.809+0000 7fb165ffb700 1 -- 172.21.15.169:0/2483314949 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb16000ae00 con 0x7fb168102c60 2024-03-28T00:51:11.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.809+0000 7fb16f105700 1 -- 172.21.15.169:0/2483314949 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1681a0320 con 0x7fb168102c60 2024-03-28T00:51:11.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.809+0000 7fb165ffb700 1 -- 172.21.15.169:0/2483314949 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1600053e0 con 0x7fb168102c60 2024-03-28T00:51:11.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.809+0000 7fb16f105700 1 -- 172.21.15.169:0/2483314949 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1681a0860 con 0x7fb168102c60 2024-03-28T00:51:11.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.810+0000 7fb16f105700 1 -- 172.21.15.169:0/2483314949 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb168068e50 con 0x7fb168102c60 2024-03-28T00:51:11.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.811+0000 7fb165ffb700 1 -- 172.21.15.169:0/2483314949 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb16000c890 con 0x7fb168102c60 2024-03-28T00:51:11.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.811+0000 7fb165ffb700 1 --2- 172.21.15.169:0/2483314949 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1580a2910 0x7fb1580a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:51:11.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.812+0000 7fb16cea1700 1 --2- 172.21.15.169:0/2483314949 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1580a2910 0x7fb1580a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:51:11.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.812+0000 7fb165ffb700 1 -- 172.21.15.169:0/2483314949 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4562+0+0 (secure 0 0 0) 0x7fb1600cd2a0 con 0x7fb168102c60 2024-03-28T00:51:11.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.812+0000 7fb16cea1700 1 --2- 172.21.15.169:0/2483314949 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1580a2910 0x7fb1580a4dd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb168102120 tx=0x7fb15c00a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:51:11.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:11.816+0000 7fb165ffb700 1 -- 172.21.15.169:0/2483314949 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb160021020 con 0x7fb168102c60 2024-03-28T00:51:12.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:12.023+0000 7fb16f105700 1 -- 172.21.15.169:0/2483314949 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1680fe7d0 con 0x7fb1580a2910 2024-03-28T00:51:12.026 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:11 smithi169 ceph-mon[32274]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T00:51:12.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:11 smithi174 ceph-mon[36060]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T00:51:12.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:11 smithi177 ceph-mon[36206]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T00:51:12.885 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:12 smithi169 ceph-mon[32274]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T00:51:12.885 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:12 smithi169 ceph-mon[32274]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:51:12.885 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:12 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:51:12.885 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:12 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:51:12.885 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:12 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:12.885 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:12 smithi169 ceph-mon[32274]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:13.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:12 smithi174 ceph-mon[36060]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T00:51:13.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:12 smithi174 ceph-mon[36060]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:51:13.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:12 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:51:13.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:12 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:51:13.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:12 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:13.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:12 smithi174 ceph-mon[36060]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:13.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:12 smithi177 ceph-mon[36206]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T00:51:13.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:12 smithi177 ceph-mon[36206]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:51:13.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:12 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:51:13.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:12 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:51:13.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:12 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:13.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:12 smithi177 ceph-mon[36206]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:13.958 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:13 smithi169 ceph-mon[32274]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T00:51:14.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:13 smithi174 ceph-mon[36060]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T00:51:14.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:13 smithi177 ceph-mon[36206]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T00:51:15.009 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:14 smithi174 ceph-mon[36060]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T00:51:15.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:14 smithi174 ceph-mon[36060]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T00:51:15.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:14 smithi169 ceph-mon[32274]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T00:51:15.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:14 smithi169 ceph-mon[32274]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T00:51:15.069 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:14 smithi177 ceph-mon[36206]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T00:51:15.069 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:14 smithi177 ceph-mon[36206]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T00:51:16.023 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:15 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2531973729' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4"}]: dispatch 2024-03-28T00:51:16.023 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:15 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2531973729' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4"}]': finished 2024-03-28T00:51:16.024 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:15 smithi174 ceph-mon[36060]: osdmap e43: 7 total, 6 up, 7 in 2024-03-28T00:51:16.024 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:15 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T00:51:16.024 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:15 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/4278655384' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:51:16.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:15 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2531973729' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4"}]: dispatch 2024-03-28T00:51:16.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:15 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2531973729' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4"}]': finished 2024-03-28T00:51:16.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:15 smithi169 ceph-mon[32274]: osdmap e43: 7 total, 6 up, 7 in 2024-03-28T00:51:16.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:15 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T00:51:16.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:15 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4278655384' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:51:16.083 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:15 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2531973729' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4"}]: dispatch 2024-03-28T00:51:16.083 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:15 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2531973729' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4"}]': finished 2024-03-28T00:51:16.083 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:15 smithi177 ceph-mon[36206]: osdmap e43: 7 total, 6 up, 7 in 2024-03-28T00:51:16.083 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:15 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T00:51:16.083 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:15 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/4278655384' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:51:17.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:16 smithi169 ceph-mon[32274]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-28T00:51:17.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:16 smithi174 ceph-mon[36060]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-28T00:51:17.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:16 smithi177 ceph-mon[36206]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-28T00:51:19.232 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:19 smithi169 ceph-mon[32274]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T00:51:19.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:19 smithi174 ceph-mon[36060]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T00:51:19.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:19 smithi177 ceph-mon[36206]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T00:51:21.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:21 smithi169 ceph-mon[32274]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-28T00:51:21.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:21 smithi174 ceph-mon[36060]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-28T00:51:21.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:21 smithi177 ceph-mon[36206]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-28T00:51:22.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:22 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T00:51:22.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:22 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:22.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:22 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T00:51:22.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:22 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:22.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:22 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T00:51:22.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:22 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:23.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:23 smithi169 ceph-mon[32274]: Deploying daemon osd.6 on smithi169 2024-03-28T00:51:23.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:23 smithi169 ceph-mon[32274]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:23.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:23 smithi174 ceph-mon[36060]: Deploying daemon osd.6 on smithi169 2024-03-28T00:51:23.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:23 smithi174 ceph-mon[36060]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:23.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:23 smithi177 ceph-mon[36206]: Deploying daemon osd.6 on smithi169 2024-03-28T00:51:23.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:23 smithi177 ceph-mon[36206]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:25.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:25 smithi169 ceph-mon[32274]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:25.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:25 smithi174 ceph-mon[36060]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:25.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:25 smithi177 ceph-mon[36206]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:27.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:27 smithi169 ceph-mon[32274]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:27.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:27 smithi177 ceph-mon[36206]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:27.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:27 smithi174 ceph-mon[36060]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:28.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:28 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:28.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:28 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:51:28.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:28 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:28.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:28 smithi169 ceph-mon[32274]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:28.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:28 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:28.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:28 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:51:28.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:28 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:28.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:28 smithi174 ceph-mon[36060]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:28.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:28 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:28.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:28 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:51:28.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:28 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:28.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:28 smithi177 ceph-mon[36206]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:31.257 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:31 smithi169 ceph-mon[32274]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:31.257 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:31 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:31.257 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:31 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:31.258 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:31 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:31.258 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:31 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:51:31.258 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:31 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:31.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:31.335+0000 7fb165ffb700 1 -- 172.21.15.169:0/2483314949 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb1680fe7d0 con 0x7fb1580a2910 2024-03-28T00:51:31.339 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 6 on host 'smithi169' 2024-03-28T00:51:31.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:31.337+0000 7fb16f105700 1 -- 172.21.15.169:0/2483314949 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1580a2910 msgr2=0x7fb1580a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:51:31.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:31.337+0000 7fb16f105700 1 --2- 172.21.15.169:0/2483314949 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1580a2910 0x7fb1580a4dd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb168102120 tx=0x7fb15c00a3b0 comp rx=0 tx=0).stop 2024-03-28T00:51:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:31.337+0000 7fb16f105700 1 -- 172.21.15.169:0/2483314949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb168102c60 msgr2=0x7fb16819f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:51:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:31.337+0000 7fb16f105700 1 --2- 172.21.15.169:0/2483314949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb168102c60 0x7fb16819f990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb16000add0 tx=0x7fb16000c330 comp rx=0 tx=0).stop 2024-03-28T00:51:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:31.337+0000 7fb16f105700 1 -- 172.21.15.169:0/2483314949 shutdown_connections 2024-03-28T00:51:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:31.337+0000 7fb16f105700 1 --2- 172.21.15.169:0/2483314949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb168102c60 0x7fb16819f990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:31.337+0000 7fb16f105700 1 --2- 172.21.15.169:0/2483314949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1681010c0 0x7fb1681983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:31.337+0000 7fb16f105700 1 --2- 172.21.15.169:0/2483314949 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1580a2910 0x7fb1580a4dd0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:31.337+0000 7fb16f105700 1 --2- 172.21.15.169:0/2483314949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1681022c0 0x7fb168198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:31.337+0000 7fb16f105700 1 -- 172.21.15.169:0/2483314949 >> 172.21.15.169:0/2483314949 conn(0x7fb1680fc680 msgr2=0x7fb1680fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:51:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:31.337+0000 7fb16f105700 1 -- 172.21.15.169:0/2483314949 shutdown_connections 2024-03-28T00:51:31.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:51:31.337+0000 7fb16f105700 1 -- 172.21.15.169:0/2483314949 wait complete. 2024-03-28T00:51:31.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:31 smithi174 ceph-mon[36060]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:31.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:31 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:31.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:31 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:31.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:31 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:31.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:31 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:51:31.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:31 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:31.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:31 smithi177 ceph-mon[36206]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:31.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:31 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:31.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:31 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:31.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:31 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:31.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:31 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:51:31.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:31 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:31.938 DEBUG:teuthology.orchestra.run.smithi169:osd.6> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.6.service 2024-03-28T00:51:31.941 INFO:tasks.cephadm:Deploying osd.7 on smithi174 with /dev/vg_nvme/lv_2... 2024-03-28T00:51:31.941 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T00:51:32.186 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T00:51:32.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:32 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:32.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:32 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:32.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:32 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:51:32.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:32 smithi169 ceph-mon[32274]: from='osd.6 [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T00:51:32.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:32 smithi169 ceph-mon[32274]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:32.597 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:32 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:32.597 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:32 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:32.597 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:32 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:51:32.597 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:32 smithi174 ceph-mon[36060]: from='osd.6 [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T00:51:32.597 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:32 smithi174 ceph-mon[36060]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:32.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:32 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:32.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:32 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:32.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:32 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:51:32.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:32 smithi177 ceph-mon[36206]: from='osd.6 [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T00:51:32.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:32 smithi177 ceph-mon[36206]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:33.422 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:33 smithi169 ceph-mon[32274]: from='osd.6 [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T00:51:33.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:33 smithi169 ceph-mon[32274]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T00:51:33.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:33 smithi169 ceph-mon[32274]: from='osd.6 [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-28T00:51:33.423 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T00:51:33.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:33 smithi174 ceph-mon[36060]: from='osd.6 [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T00:51:33.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:33 smithi174 ceph-mon[36060]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T00:51:33.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:33 smithi174 ceph-mon[36060]: from='osd.6 [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-28T00:51:33.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T00:51:33.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:33 smithi177 ceph-mon[36206]: from='osd.6 [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T00:51:33.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:33 smithi177 ceph-mon[36206]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T00:51:33.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:33 smithi177 ceph-mon[36206]: from='osd.6 [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-28T00:51:33.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T00:51:34.052 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 00:51:33 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T00:51:33.765+0000 7fa2f4640700 -1 osd.6 0 waiting for initial osdmap 2024-03-28T00:51:34.052 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 00:51:33 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T00:51:33.774+0000 7fa2ec42b700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T00:51:34.360 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T00:51:34.360 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T00:51:34.360 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-03-28T00:51:34.360 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-03-28T00:51:34.360 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0133903 s, 783 MB/s 2024-03-28T00:51:34.361 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-03-28T00:51:34.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:34 smithi169 ceph-mon[32274]: from='osd.6 [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-28T00:51:34.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:34 smithi169 ceph-mon[32274]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T00:51:34.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:34 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T00:51:34.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:34 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T00:51:34.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:34 smithi169 ceph-mon[32274]: from='osd.6 [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252]' entity='osd.6' 2024-03-28T00:51:34.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:34 smithi169 ceph-mon[32274]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:34.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:34 smithi174 ceph-mon[36060]: from='osd.6 [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-28T00:51:34.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:34 smithi174 ceph-mon[36060]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T00:51:34.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:34 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T00:51:34.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:34 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T00:51:34.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:34 smithi174 ceph-mon[36060]: from='osd.6 [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252]' entity='osd.6' 2024-03-28T00:51:34.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:34 smithi174 ceph-mon[36060]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:34.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:34 smithi177 ceph-mon[36206]: from='osd.6 [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-28T00:51:34.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:34 smithi177 ceph-mon[36206]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T00:51:34.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:34 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T00:51:34.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:34 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T00:51:34.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:34 smithi177 ceph-mon[36206]: from='osd.6 [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252]' entity='osd.6' 2024-03-28T00:51:34.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:34 smithi177 ceph-mon[36206]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-28T00:51:35.002 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi174:vg_nvme/lv_2 2024-03-28T00:51:35.258 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T00:51:35.590 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:35 smithi174 ceph-mon[36060]: purged_snaps scrub starts 2024-03-28T00:51:35.591 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:35 smithi174 ceph-mon[36060]: purged_snaps scrub ok 2024-03-28T00:51:35.591 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:35 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T00:51:35.591 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:35 smithi174 ceph-mon[36060]: osd.6 [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252] boot 2024-03-28T00:51:35.591 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:35 smithi174 ceph-mon[36060]: osdmap e46: 7 total, 7 up, 7 in 2024-03-28T00:51:35.591 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:35 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T00:51:35.591 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:35 smithi174 ceph-mon[36060]: Detected new or changed devices on smithi169 2024-03-28T00:51:35.591 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:35 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:35.591 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:35 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:35.591 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:35 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:51:35.591 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:35 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:51:35.592 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:35 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:51:35.592 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:35 smithi174 ceph-mon[36060]: Adjusting osd_memory_target on smithi169 to 5696M 2024-03-28T00:51:35.592 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:35 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:35.592 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:35 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:35.592 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:35 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:51:35.592 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:35 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:35.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:35 smithi177 ceph-mon[36206]: purged_snaps scrub starts 2024-03-28T00:51:35.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:35 smithi177 ceph-mon[36206]: purged_snaps scrub ok 2024-03-28T00:51:35.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:35 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T00:51:35.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:35 smithi177 ceph-mon[36206]: osd.6 [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252] boot 2024-03-28T00:51:35.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:35 smithi177 ceph-mon[36206]: osdmap e46: 7 total, 7 up, 7 in 2024-03-28T00:51:35.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:35 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T00:51:35.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:35 smithi177 ceph-mon[36206]: Detected new or changed devices on smithi169 2024-03-28T00:51:35.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:35 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:35.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:35 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:35.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:35 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:51:35.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:35 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:51:35.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:35 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:51:35.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:35 smithi177 ceph-mon[36206]: Adjusting osd_memory_target on smithi169 to 5696M 2024-03-28T00:51:35.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:35 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:35.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:35 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:35.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:35 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:51:35.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:35 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:35.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:35 smithi169 ceph-mon[32274]: purged_snaps scrub starts 2024-03-28T00:51:35.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:35 smithi169 ceph-mon[32274]: purged_snaps scrub ok 2024-03-28T00:51:35.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:35 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T00:51:35.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:35 smithi169 ceph-mon[32274]: osd.6 [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252] boot 2024-03-28T00:51:35.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:35 smithi169 ceph-mon[32274]: osdmap e46: 7 total, 7 up, 7 in 2024-03-28T00:51:35.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:35 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T00:51:35.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:35 smithi169 ceph-mon[32274]: Detected new or changed devices on smithi169 2024-03-28T00:51:35.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:35 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:35.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:35 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:35.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:35 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:51:35.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:35 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:51:35.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:35 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:51:35.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:35 smithi169 ceph-mon[32274]: Adjusting osd_memory_target on smithi169 to 5696M 2024-03-28T00:51:35.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:35 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:35.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:35 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:35.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:35 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:51:35.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:35 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:36.574 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:36 smithi174 ceph-mon[36060]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:36.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:36 smithi177 ceph-mon[36206]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:36.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:36 smithi169 ceph-mon[32274]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:36.916 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.912+0000 7f2b262af700 1 -- 172.21.15.174:0/2345746552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b201022f0 msgr2=0x7f2b20102750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:51:36.917 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.912+0000 7f2b262af700 1 --2- 172.21.15.174:0/2345746552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b201022f0 0x7f2b20102750 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2b100099c0 tx=0x7f2b10009cd0 comp rx=0 tx=0).stop 2024-03-28T00:51:36.917 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.912+0000 7f2b262af700 1 -- 172.21.15.174:0/2345746552 shutdown_connections 2024-03-28T00:51:36.917 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.912+0000 7f2b262af700 1 --2- 172.21.15.174:0/2345746552 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b20102c90 0x7f2b20109540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:36.917 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.912+0000 7f2b262af700 1 --2- 172.21.15.174:0/2345746552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b201010f0 0x7f2b20101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:36.917 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.912+0000 7f2b262af700 1 --2- 172.21.15.174:0/2345746552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b201022f0 0x7f2b20102750 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:36.917 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.912+0000 7f2b262af700 1 -- 172.21.15.174:0/2345746552 >> 172.21.15.174:0/2345746552 conn(0x7f2b200fc6b0 msgr2=0x7f2b200fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:51:36.917 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.912+0000 7f2b262af700 1 -- 172.21.15.174:0/2345746552 shutdown_connections 2024-03-28T00:51:36.917 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.912+0000 7f2b262af700 1 -- 172.21.15.174:0/2345746552 wait complete. 2024-03-28T00:51:36.917 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.913+0000 7f2b262af700 1 Processor -- start 2024-03-28T00:51:36.917 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.913+0000 7f2b262af700 1 -- start start 2024-03-28T00:51:36.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.913+0000 7f2b262af700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b201010f0 0x7f2b201983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:51:36.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.913+0000 7f2b262af700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b201022f0 0x7f2b20198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:51:36.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.913+0000 7f2b262af700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b20102c90 0x7f2b2019f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:51:36.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.913+0000 7f2b262af700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2010ccb0 con 0x7f2b201022f0 2024-03-28T00:51:36.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.913+0000 7f2b262af700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2010cb30 con 0x7f2b20102c90 2024-03-28T00:51:36.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.913+0000 7f2b262af700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2010ce30 con 0x7f2b201010f0 2024-03-28T00:51:36.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.914+0000 7f2b2484c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b20102c90 0x7f2b2019f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:51:36.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.914+0000 7f2b2484c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b20102c90 0x7f2b2019f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:54750/0 (socket says 172.21.15.174:54750) 2024-03-28T00:51:36.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.914+0000 7f2b2484c700 1 -- 172.21.15.174:0/3507643435 learned_addr learned my addr 172.21.15.174:0/3507643435 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-28T00:51:36.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.914+0000 7f2b1f7fe700 1 --2- 172.21.15.174:0/3507643435 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b201022f0 0x7f2b20198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:51:36.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.914+0000 7f2b1ffff700 1 --2- 172.21.15.174:0/3507643435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b201010f0 0x7f2b201983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:51:36.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.914+0000 7f2b2484c700 1 -- 172.21.15.174:0/3507643435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b201010f0 msgr2=0x7f2b201983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:51:36.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.914+0000 7f2b2484c700 1 --2- 172.21.15.174:0/3507643435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b201010f0 0x7f2b201983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:36.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.914+0000 7f2b2484c700 1 -- 172.21.15.174:0/3507643435 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b201022f0 msgr2=0x7f2b20198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:51:36.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.914+0000 7f2b2484c700 1 --2- 172.21.15.174:0/3507643435 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b201022f0 0x7f2b20198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:36.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.914+0000 7f2b2484c700 1 -- 172.21.15.174:0/3507643435 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b201a0030 con 0x7f2b20102c90 2024-03-28T00:51:36.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.914+0000 7f2b2484c700 1 --2- 172.21.15.174:0/3507643435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b20102c90 0x7f2b2019f990 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2b18007ee0 tx=0x7f2b1800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:51:36.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.914+0000 7f2b1d7fa700 1 -- 172.21.15.174:0/3507643435 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1800dbd0 con 0x7f2b20102c90 2024-03-28T00:51:36.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.914+0000 7f2b262af700 1 -- 172.21.15.174:0/3507643435 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b201a0320 con 0x7f2b20102c90 2024-03-28T00:51:36.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.914+0000 7f2b1d7fa700 1 -- 172.21.15.174:0/3507643435 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2b1800dd70 con 0x7f2b20102c90 2024-03-28T00:51:36.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.914+0000 7f2b1d7fa700 1 -- 172.21.15.174:0/3507643435 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b18005a40 con 0x7f2b20102c90 2024-03-28T00:51:36.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.914+0000 7f2b262af700 1 -- 172.21.15.174:0/3507643435 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b201a0860 con 0x7f2b20102c90 2024-03-28T00:51:36.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.915+0000 7f2b262af700 1 -- 172.21.15.174:0/3507643435 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b20068e50 con 0x7f2b20102c90 2024-03-28T00:51:36.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.916+0000 7f2b1d7fa700 1 -- 172.21.15.174:0/3507643435 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2b1801a020 con 0x7f2b20102c90 2024-03-28T00:51:36.922 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.917+0000 7f2b1d7fa700 1 --2- 172.21.15.174:0/3507643435 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2b0c0a2890 0x7f2b0c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:51:36.922 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.917+0000 7f2b1d7fa700 1 -- 172.21.15.174:0/3507643435 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f2b180cdad0 con 0x7f2b20102c90 2024-03-28T00:51:36.922 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.917+0000 7f2b1ffff700 1 --2- 172.21.15.174:0/3507643435 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2b0c0a2890 0x7f2b0c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:51:36.922 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.918+0000 7f2b1ffff700 1 --2- 172.21.15.174:0/3507643435 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2b0c0a2890 0x7f2b0c0a4d50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2b08005950 tx=0x7f2b080058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:51:36.924 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:36.921+0000 7f2b1d7fa700 1 -- 172.21.15.174:0/3507643435 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2b18010070 con 0x7f2b20102c90 2024-03-28T00:51:37.128 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:37.124+0000 7f2b262af700 1 -- 172.21.15.174:0/3507643435 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b201a0b80 con 0x7f2b0c0a2890 2024-03-28T00:51:37.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:37 smithi174 ceph-mon[36060]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T00:51:37.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:37 smithi174 ceph-mon[36060]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:51:37.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:37 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:51:37.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:37 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:51:37.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:37 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:37.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:37 smithi177 ceph-mon[36206]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T00:51:37.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:37 smithi177 ceph-mon[36206]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:51:37.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:37 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:51:37.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:37 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:51:37.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:37 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:37.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:37 smithi169 ceph-mon[32274]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T00:51:37.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:37 smithi169 ceph-mon[32274]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:51:37.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:51:37.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:51:37.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:38.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:38 smithi174 ceph-mon[36060]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:38.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:38 smithi177 ceph-mon[36206]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:38.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:38 smithi169 ceph-mon[32274]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:40.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:39 smithi177 ceph-mon[36206]: from='client.? 172.21.15.174:0/2701253073' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5378575b-122b-4369-9fc8-1230aee2dbd1"}]: dispatch 2024-03-28T00:51:40.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:39 smithi177 ceph-mon[36206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5378575b-122b-4369-9fc8-1230aee2dbd1"}]: dispatch 2024-03-28T00:51:40.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:39 smithi177 ceph-mon[36206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5378575b-122b-4369-9fc8-1230aee2dbd1"}]': finished 2024-03-28T00:51:40.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:39 smithi177 ceph-mon[36206]: osdmap e48: 8 total, 7 up, 8 in 2024-03-28T00:51:40.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:39 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T00:51:40.200 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:39 smithi174 ceph-mon[36060]: from='client.? 172.21.15.174:0/2701253073' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5378575b-122b-4369-9fc8-1230aee2dbd1"}]: dispatch 2024-03-28T00:51:40.201 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:39 smithi174 ceph-mon[36060]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5378575b-122b-4369-9fc8-1230aee2dbd1"}]: dispatch 2024-03-28T00:51:40.201 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:39 smithi174 ceph-mon[36060]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5378575b-122b-4369-9fc8-1230aee2dbd1"}]': finished 2024-03-28T00:51:40.201 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:39 smithi174 ceph-mon[36060]: osdmap e48: 8 total, 7 up, 8 in 2024-03-28T00:51:40.201 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:39 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T00:51:40.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:39 smithi169 ceph-mon[32274]: from='client.? 172.21.15.174:0/2701253073' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5378575b-122b-4369-9fc8-1230aee2dbd1"}]: dispatch 2024-03-28T00:51:40.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:39 smithi169 ceph-mon[32274]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5378575b-122b-4369-9fc8-1230aee2dbd1"}]: dispatch 2024-03-28T00:51:40.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:39 smithi169 ceph-mon[32274]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5378575b-122b-4369-9fc8-1230aee2dbd1"}]': finished 2024-03-28T00:51:40.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:39 smithi169 ceph-mon[32274]: osdmap e48: 8 total, 7 up, 8 in 2024-03-28T00:51:40.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:39 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T00:51:41.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:40 smithi177 ceph-mon[36206]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:41.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:40 smithi177 ceph-mon[36206]: from='client.? 172.21.15.174:0/5816153' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:51:41.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:40 smithi169 ceph-mon[32274]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:41.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:40 smithi169 ceph-mon[32274]: from='client.? 172.21.15.174:0/5816153' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:51:41.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:40 smithi174 ceph-mon[36060]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:41.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:40 smithi174 ceph-mon[36060]: from='client.? 172.21.15.174:0/5816153' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:51:43.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:43 smithi174 ceph-mon[36060]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:43.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:43 smithi177 ceph-mon[36206]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:43.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:43 smithi169 ceph-mon[32274]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:45.354 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:45 smithi169 ceph-mon[32274]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:45.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:45 smithi174 ceph-mon[36060]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:45.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:45 smithi177 ceph-mon[36206]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:47.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:47 smithi174 ceph-mon[36060]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:47.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:47 smithi177 ceph-mon[36206]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:47.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:47 smithi169 ceph-mon[32274]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:48.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:48 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T00:51:48.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:48 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:48.419 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:48 smithi177 ceph-mon[36206]: Deploying daemon osd.7 on smithi174 2024-03-28T00:51:48.486 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:48 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T00:51:48.487 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:48 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:48.487 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:48 smithi174 ceph-mon[36060]: Deploying daemon osd.7 on smithi174 2024-03-28T00:51:48.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:48 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T00:51:48.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:48 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:48.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:48 smithi169 ceph-mon[32274]: Deploying daemon osd.7 on smithi174 2024-03-28T00:51:49.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:49 smithi177 ceph-mon[36206]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:49.515 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:49 smithi174 ceph-mon[36060]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:49.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:49 smithi169 ceph-mon[32274]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:51.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:51 smithi177 ceph-mon[36206]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:51.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:51 smithi169 ceph-mon[32274]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:51.572 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:51 smithi174 ceph-mon[36060]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:53.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:53 smithi177 ceph-mon[36206]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:53.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:53 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:53.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:53 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:53.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:53 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:51:53.473 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:53 smithi174 ceph-mon[36060]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:53.473 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:53 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:53.473 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:53 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:53.473 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:53 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:51:53.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:53 smithi169 ceph-mon[32274]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:53.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:53 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:53.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:53 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:53.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:53 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:51:55.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:55 smithi177 ceph-mon[36206]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:55.480 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:55 smithi174 ceph-mon[36060]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:55.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:55 smithi169 ceph-mon[32274]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:56.796 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 7 on host 'smithi174' 2024-03-28T00:51:56.797 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:56.793+0000 7f2b1d7fa700 1 -- 172.21.15.174:0/3507643435 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2b201a0b80 con 0x7f2b0c0a2890 2024-03-28T00:51:56.798 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:56.795+0000 7f2b262af700 1 -- 172.21.15.174:0/3507643435 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2b0c0a2890 msgr2=0x7f2b0c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:51:56.798 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:56.795+0000 7f2b262af700 1 --2- 172.21.15.174:0/3507643435 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2b0c0a2890 0x7f2b0c0a4d50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2b08005950 tx=0x7f2b080058e0 comp rx=0 tx=0).stop 2024-03-28T00:51:56.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:56.795+0000 7f2b262af700 1 -- 172.21.15.174:0/3507643435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b20102c90 msgr2=0x7f2b2019f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:51:56.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:56.795+0000 7f2b262af700 1 --2- 172.21.15.174:0/3507643435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b20102c90 0x7f2b2019f990 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2b18007ee0 tx=0x7f2b1800d3b0 comp rx=0 tx=0).stop 2024-03-28T00:51:56.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:56.795+0000 7f2b262af700 1 -- 172.21.15.174:0/3507643435 shutdown_connections 2024-03-28T00:51:56.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:56.795+0000 7f2b262af700 1 --2- 172.21.15.174:0/3507643435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b20102c90 0x7f2b2019f990 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:56.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:56.795+0000 7f2b262af700 1 --2- 172.21.15.174:0/3507643435 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2b0c0a2890 0x7f2b0c0a4d50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:56.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:56.795+0000 7f2b262af700 1 --2- 172.21.15.174:0/3507643435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b201010f0 0x7f2b201983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:56.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:56.795+0000 7f2b262af700 1 --2- 172.21.15.174:0/3507643435 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2b201022f0 0x7f2b20198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:51:56.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:56.795+0000 7f2b262af700 1 -- 172.21.15.174:0/3507643435 >> 172.21.15.174:0/3507643435 conn(0x7f2b200fc6b0 msgr2=0x7f2b200fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:51:56.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:56.795+0000 7f2b262af700 1 -- 172.21.15.174:0/3507643435 shutdown_connections 2024-03-28T00:51:56.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:51:56.795+0000 7f2b262af700 1 -- 172.21.15.174:0/3507643435 wait complete. 2024-03-28T00:51:57.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:57.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:57.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:57.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:51:57.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:57.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:56 smithi169 ceph-mon[32274]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:57.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:56 smithi169 ceph-mon[32274]: from='osd.7 [v2:172.21.15.174:6816/2274626586,v1:172.21.15.174:6817/2274626586]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T00:51:57.053 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:56 smithi169 ceph-mon[32274]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T00:51:57.053 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:57.053 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:57.053 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:51:57.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:57.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:57.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:57.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:51:57.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:57.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:56 smithi174 ceph-mon[36060]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:57.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:56 smithi174 ceph-mon[36060]: from='osd.7 [v2:172.21.15.174:6816/2274626586,v1:172.21.15.174:6817/2274626586]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T00:51:57.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:56 smithi174 ceph-mon[36060]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T00:51:57.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:57.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:57.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:51:57.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:57.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:57.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:51:57.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:51:57.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:57.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:56 smithi177 ceph-mon[36206]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:57.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:56 smithi177 ceph-mon[36206]: from='osd.7 [v2:172.21.15.174:6816/2274626586,v1:172.21.15.174:6817/2274626586]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T00:51:57.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:56 smithi177 ceph-mon[36206]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T00:51:57.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:57.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:51:57.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:51:57.444 DEBUG:teuthology.orchestra.run.smithi174:osd.7> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.7.service 2024-03-28T00:51:57.446 INFO:tasks.cephadm:Deploying osd.8 on smithi177 with /dev/vg_nvme/lv_2... 2024-03-28T00:51:57.447 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T00:51:57.691 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T00:51:58.077 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:57 smithi177 ceph-mon[36206]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T00:51:58.077 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:57 smithi177 ceph-mon[36206]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T00:51:58.077 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:57 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T00:51:58.077 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:57 smithi177 ceph-mon[36206]: from='osd.7 [v2:172.21.15.174:6816/2274626586,v1:172.21.15.174:6817/2274626586]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:51:58.077 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:57 smithi177 ceph-mon[36206]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:51:58.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:57 smithi174 ceph-mon[36060]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T00:51:58.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:57 smithi174 ceph-mon[36060]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T00:51:58.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:57 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T00:51:58.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:57 smithi174 ceph-mon[36060]: from='osd.7 [v2:172.21.15.174:6816/2274626586,v1:172.21.15.174:6817/2274626586]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:51:58.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:57 smithi174 ceph-mon[36060]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:51:58.109 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 00:51:57 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T00:51:57.841+0000 7fbaeb83b700 -1 osd.7 0 waiting for initial osdmap 2024-03-28T00:51:58.109 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 00:51:57 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T00:51:57.850+0000 7fbae762e700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T00:51:58.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:57 smithi169 ceph-mon[32274]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T00:51:58.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:57 smithi169 ceph-mon[32274]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T00:51:58.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:57 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T00:51:58.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:57 smithi169 ceph-mon[32274]: from='osd.7 [v2:172.21.15.174:6816/2274626586,v1:172.21.15.174:6817/2274626586]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:51:58.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:57 smithi169 ceph-mon[32274]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:51:58.975 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:58 smithi174 ceph-mon[36060]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-28T00:51:58.976 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:58 smithi174 ceph-mon[36060]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T00:51:58.976 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:58 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T00:51:58.976 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:58 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T00:51:58.976 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:58 smithi174 ceph-mon[36060]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:58.976 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:58 smithi174 ceph-mon[36060]: osd.7 [v2:172.21.15.174:6816/2274626586,v1:172.21.15.174:6817/2274626586] boot 2024-03-28T00:51:58.976 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:58 smithi174 ceph-mon[36060]: osdmap e51: 8 total, 8 up, 8 in 2024-03-28T00:51:58.976 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:51:58 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T00:51:59.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:58 smithi177 ceph-mon[36206]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-28T00:51:59.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:58 smithi177 ceph-mon[36206]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T00:51:59.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:58 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T00:51:59.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:58 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T00:51:59.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:58 smithi177 ceph-mon[36206]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:59.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:58 smithi177 ceph-mon[36206]: osd.7 [v2:172.21.15.174:6816/2274626586,v1:172.21.15.174:6817/2274626586] boot 2024-03-28T00:51:59.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:58 smithi177 ceph-mon[36206]: osdmap e51: 8 total, 8 up, 8 in 2024-03-28T00:51:59.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:51:58 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T00:51:59.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:58 smithi169 ceph-mon[32274]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-28T00:51:59.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:58 smithi169 ceph-mon[32274]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T00:51:59.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:58 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T00:51:59.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:58 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T00:51:59.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:58 smithi169 ceph-mon[32274]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-28T00:51:59.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:58 smithi169 ceph-mon[32274]: osd.7 [v2:172.21.15.174:6816/2274626586,v1:172.21.15.174:6817/2274626586] boot 2024-03-28T00:51:59.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:58 smithi169 ceph-mon[32274]: osdmap e51: 8 total, 8 up, 8 in 2024-03-28T00:51:59.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:51:58 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T00:51:59.777 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T00:51:59.777 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T00:51:59.777 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-03-28T00:51:59.777 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-03-28T00:51:59.778 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112057 s, 936 MB/s 2024-03-28T00:51:59.778 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-03-28T00:52:00.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:00 smithi177 ceph-mon[36206]: purged_snaps scrub starts 2024-03-28T00:52:00.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:00 smithi177 ceph-mon[36206]: purged_snaps scrub ok 2024-03-28T00:52:00.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:00 smithi177 ceph-mon[36206]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T00:52:00.449 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi177:vg_nvme/lv_2 2024-03-28T00:52:00.517 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:00 smithi174 ceph-mon[36060]: purged_snaps scrub starts 2024-03-28T00:52:00.517 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:00 smithi174 ceph-mon[36060]: purged_snaps scrub ok 2024-03-28T00:52:00.517 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:00 smithi174 ceph-mon[36060]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T00:52:00.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:00 smithi169 ceph-mon[32274]: purged_snaps scrub starts 2024-03-28T00:52:00.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:00 smithi169 ceph-mon[32274]: purged_snaps scrub ok 2024-03-28T00:52:00.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:00 smithi169 ceph-mon[32274]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T00:52:00.700 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T00:52:01.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:01 smithi177 ceph-mon[36206]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:01.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:01 smithi177 ceph-mon[36206]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T00:52:01.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:01 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:01.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:01 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:01.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:01 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:01.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:01 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:01.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:01 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:01.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:01 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:01.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:01 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:01.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:01 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:52:01.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:01 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:01.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:01 smithi169 ceph-mon[32274]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:01.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:01 smithi169 ceph-mon[32274]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T00:52:01.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:01 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:01.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:01 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:01.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:01 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:01.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:01 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:01.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:01 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:01.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:01 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:01.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:01 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:01.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:01 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:52:01.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:01 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:01.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:01 smithi174 ceph-mon[36060]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:01.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:01 smithi174 ceph-mon[36060]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T00:52:01.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:01 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:01.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:01 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:01 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:01 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:01 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:01 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:01 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:01 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:52:01.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:01 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:02.300 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:02 smithi177 ceph-mon[36206]: Detected new or changed devices on smithi174 2024-03-28T00:52:02.300 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:02 smithi177 ceph-mon[36206]: Adjusting osd_memory_target on smithi174 to 5693M 2024-03-28T00:52:02.300 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:02 smithi177 ceph-mon[36206]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T00:52:02.300 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:02 smithi177 ceph-mon[36206]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T00:52:02.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.508+0000 7f34ae400700 1 -- 172.21.15.177:0/1778083749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a8101160 msgr2=0x7f34a8101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:52:02.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.508+0000 7f34ae400700 1 --2- 172.21.15.177:0/1778083749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a8101160 0x7f34a8101580 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3498009990 tx=0x7f3498009ca0 comp rx=0 tx=0).stop 2024-03-28T00:52:02.512 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.508+0000 7f34ae400700 1 -- 172.21.15.177:0/1778083749 shutdown_connections 2024-03-28T00:52:02.512 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.508+0000 7f34ae400700 1 --2- 172.21.15.177:0/1778083749 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34a8102d00 0x7f34a81095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:02.512 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.508+0000 7f34ae400700 1 --2- 172.21.15.177:0/1778083749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a8101160 0x7f34a8101580 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:02.512 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.508+0000 7f34ae400700 1 --2- 172.21.15.177:0/1778083749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34a8102360 0x7f34a81027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:02.512 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.508+0000 7f34ae400700 1 -- 172.21.15.177:0/1778083749 >> 172.21.15.177:0/1778083749 conn(0x7f34a80fc720 msgr2=0x7f34a80feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:52:02.512 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.508+0000 7f34ae400700 1 -- 172.21.15.177:0/1778083749 shutdown_connections 2024-03-28T00:52:02.512 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.508+0000 7f34ae400700 1 -- 172.21.15.177:0/1778083749 wait complete. 2024-03-28T00:52:02.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.508+0000 7f34ae400700 1 Processor -- start 2024-03-28T00:52:02.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.509+0000 7f34ae400700 1 -- start start 2024-03-28T00:52:02.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.509+0000 7f34ae400700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34a8101160 0x7f34a806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:52:02.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.509+0000 7f34ae400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a8102360 0x7f34a819d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:52:02.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.509+0000 7f34ae400700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34a8102d00 0x7f34a819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:52:02.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.509+0000 7f34ae400700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a810cce0 con 0x7f34a8102d00 2024-03-28T00:52:02.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.509+0000 7f34ae400700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a810cb60 con 0x7f34a8101160 2024-03-28T00:52:02.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.509+0000 7f34ae400700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a810ce60 con 0x7f34a8102360 2024-03-28T00:52:02.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.509+0000 7f34a77fe700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a8102360 0x7f34a819d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:52:02.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.509+0000 7f34a77fe700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a8102360 0x7f34a819d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:47708/0 (socket says 172.21.15.177:47708) 2024-03-28T00:52:02.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.509+0000 7f34a77fe700 1 -- 172.21.15.177:0/1543649381 learned_addr learned my addr 172.21.15.177:0/1543649381 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-28T00:52:02.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.509+0000 7f34a7fff700 1 --2- 172.21.15.177:0/1543649381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34a8101160 0x7f34a806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:52:02.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.509+0000 7f34a77fe700 1 -- 172.21.15.177:0/1543649381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34a8101160 msgr2=0x7f34a806b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:52:02.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.509+0000 7f34a77fe700 1 --2- 172.21.15.177:0/1543649381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34a8101160 0x7f34a806b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:02.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.509+0000 7f34a77fe700 1 -- 172.21.15.177:0/1543649381 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34a8102d00 msgr2=0x7f34a819f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:52:02.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.509+0000 7f34ac99d700 1 --2- 172.21.15.177:0/1543649381 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34a8102d00 0x7f34a819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:52:02.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.509+0000 7f34a77fe700 1 --2- 172.21.15.177:0/1543649381 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34a8102d00 0x7f34a819f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:02.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.509+0000 7f34a77fe700 1 -- 172.21.15.177:0/1543649381 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a81a0080 con 0x7f34a8102360 2024-03-28T00:52:02.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.509+0000 7f34a77fe700 1 --2- 172.21.15.177:0/1543649381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a8102360 0x7f34a819d160 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f349c00d980 tx=0x7f349c00dd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:52:02.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.510+0000 7f34a57fa700 1 -- 172.21.15.177:0/1543649381 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c017070 con 0x7f34a8102360 2024-03-28T00:52:02.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.510+0000 7f34ae400700 1 -- 172.21.15.177:0/1543649381 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a81a0370 con 0x7f34a8102360 2024-03-28T00:52:02.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.510+0000 7f34a57fa700 1 -- 172.21.15.177:0/1543649381 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f349c004d10 con 0x7f34a8102360 2024-03-28T00:52:02.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.510+0000 7f34a57fa700 1 -- 172.21.15.177:0/1543649381 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c0055c0 con 0x7f34a8102360 2024-03-28T00:52:02.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.510+0000 7f34ae400700 1 -- 172.21.15.177:0/1543649381 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a81a0860 con 0x7f34a8102360 2024-03-28T00:52:02.521 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.511+0000 7f34ae400700 1 -- 172.21.15.177:0/1543649381 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a8068e50 con 0x7f34a8102360 2024-03-28T00:52:02.522 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.512+0000 7f34a57fa700 1 -- 172.21.15.177:0/1543649381 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f349c004750 con 0x7f34a8102360 2024-03-28T00:52:02.522 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.512+0000 7f34a57fa700 1 --2- 172.21.15.177:0/1543649381 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f34900a2890 0x7f34900a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:52:02.522 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.512+0000 7f34a57fa700 1 -- 172.21.15.177:0/1543649381 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5397+0+0 (secure 0 0 0) 0x7f349c0ce270 con 0x7f34a8102360 2024-03-28T00:52:02.522 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.513+0000 7f34a7fff700 1 --2- 172.21.15.177:0/1543649381 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f34900a2890 0x7f34900a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:52:02.522 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.513+0000 7f34a7fff700 1 --2- 172.21.15.177:0/1543649381 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f34900a2890 0x7f34900a4d50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f34980039c0 tx=0x7f3498005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:52:02.522 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.516+0000 7f34a57fa700 1 -- 172.21.15.177:0/1543649381 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f349c020020 con 0x7f34a8102360 2024-03-28T00:52:02.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:02 smithi169 ceph-mon[32274]: Detected new or changed devices on smithi174 2024-03-28T00:52:02.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:02 smithi169 ceph-mon[32274]: Adjusting osd_memory_target on smithi174 to 5693M 2024-03-28T00:52:02.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:02 smithi169 ceph-mon[32274]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T00:52:02.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:02 smithi169 ceph-mon[32274]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T00:52:02.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:02 smithi174 ceph-mon[36060]: Detected new or changed devices on smithi174 2024-03-28T00:52:02.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:02 smithi174 ceph-mon[36060]: Adjusting osd_memory_target on smithi174 to 5693M 2024-03-28T00:52:02.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:02 smithi174 ceph-mon[36060]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T00:52:02.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:02 smithi174 ceph-mon[36060]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T00:52:02.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:02.722+0000 7f34ae400700 1 -- 172.21.15.177:0/1543649381 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f34a80625d0 con 0x7f34900a2890 2024-03-28T00:52:03.527 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:03 smithi177 ceph-mon[36206]: pgmap v177: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:03.527 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:03 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:52:03.527 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:03 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:52:03.527 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:03 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:03.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:03 smithi169 ceph-mon[32274]: pgmap v177: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:03.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:03 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:52:03.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:03 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:52:03.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:03 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:03.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:03 smithi174 ceph-mon[36060]: pgmap v177: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:03.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:03 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:52:03.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:03 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:52:03.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:03 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:04.461 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:04 smithi177 ceph-mon[36206]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:52:04.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:04 smithi169 ceph-mon[32274]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:52:04.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:04 smithi174 ceph-mon[36060]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:52:05.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:05 smithi169 ceph-mon[32274]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:05.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:05 smithi174 ceph-mon[36060]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:05.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:05 smithi177 ceph-mon[36206]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:06.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:06 smithi169 ceph-mon[32274]: from='client.? 172.21.15.177:0/4270513592' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "874c0869-f1fe-4927-9549-61d6e46c498a"}]: dispatch 2024-03-28T00:52:06.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:06 smithi169 ceph-mon[32274]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "874c0869-f1fe-4927-9549-61d6e46c498a"}]: dispatch 2024-03-28T00:52:06.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:06 smithi169 ceph-mon[32274]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "874c0869-f1fe-4927-9549-61d6e46c498a"}]': finished 2024-03-28T00:52:06.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:06 smithi169 ceph-mon[32274]: osdmap e56: 9 total, 8 up, 9 in 2024-03-28T00:52:06.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:06 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T00:52:06.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:06 smithi169 ceph-mon[32274]: from='client.? 172.21.15.177:0/3623289990' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:52:06.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:06 smithi174 ceph-mon[36060]: from='client.? 172.21.15.177:0/4270513592' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "874c0869-f1fe-4927-9549-61d6e46c498a"}]: dispatch 2024-03-28T00:52:06.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:06 smithi174 ceph-mon[36060]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "874c0869-f1fe-4927-9549-61d6e46c498a"}]: dispatch 2024-03-28T00:52:06.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:06 smithi174 ceph-mon[36060]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "874c0869-f1fe-4927-9549-61d6e46c498a"}]': finished 2024-03-28T00:52:06.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:06 smithi174 ceph-mon[36060]: osdmap e56: 9 total, 8 up, 9 in 2024-03-28T00:52:06.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:06 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T00:52:06.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:06 smithi174 ceph-mon[36060]: from='client.? 172.21.15.177:0/3623289990' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:52:06.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:06 smithi177 ceph-mon[36206]: from='client.? 172.21.15.177:0/4270513592' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "874c0869-f1fe-4927-9549-61d6e46c498a"}]: dispatch 2024-03-28T00:52:06.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:06 smithi177 ceph-mon[36206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "874c0869-f1fe-4927-9549-61d6e46c498a"}]: dispatch 2024-03-28T00:52:06.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:06 smithi177 ceph-mon[36206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "874c0869-f1fe-4927-9549-61d6e46c498a"}]': finished 2024-03-28T00:52:06.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:06 smithi177 ceph-mon[36206]: osdmap e56: 9 total, 8 up, 9 in 2024-03-28T00:52:06.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:06 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T00:52:06.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:06 smithi177 ceph-mon[36206]: from='client.? 172.21.15.177:0/3623289990' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:52:07.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:07 smithi169 ceph-mon[32274]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:07.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:07 smithi174 ceph-mon[36060]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:07.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:07 smithi177 ceph-mon[36206]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:09.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:09 smithi177 ceph-mon[36206]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:09.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:09 smithi169 ceph-mon[32274]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:09.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:09 smithi174 ceph-mon[36060]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:11.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:11 smithi177 ceph-mon[36206]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:11.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:11 smithi169 ceph-mon[32274]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:11.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:11 smithi174 ceph-mon[36060]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:13.487 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:13 smithi177 ceph-mon[36206]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:13.487 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:13 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T00:52:13.487 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:13 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:13.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:13 smithi169 ceph-mon[32274]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:13.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:13 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T00:52:13.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:13 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:13.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:13 smithi174 ceph-mon[36060]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:13.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:13 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T00:52:13.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:13 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:14.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:14 smithi177 ceph-mon[36206]: Deploying daemon osd.8 on smithi177 2024-03-28T00:52:14.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:14 smithi169 ceph-mon[32274]: Deploying daemon osd.8 on smithi177 2024-03-28T00:52:14.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:14 smithi174 ceph-mon[36060]: Deploying daemon osd.8 on smithi177 2024-03-28T00:52:15.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:15 smithi177 ceph-mon[36206]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:15.509 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:15 smithi174 ceph-mon[36060]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:15.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:15 smithi169 ceph-mon[32274]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:17.465 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:17 smithi177 ceph-mon[36206]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:17.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:17 smithi169 ceph-mon[32274]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:17.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:17 smithi174 ceph-mon[36060]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:18.355 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:18 smithi177 ceph-mon[36206]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:18.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:18 smithi169 ceph-mon[32274]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:18.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:18 smithi174 ceph-mon[36060]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:19.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:19 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:19.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:19 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:52:19.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:19 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:19.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:19 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:19.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:19 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:52:19.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:19 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:19.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:19.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:52:19.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:19 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:20.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:20 smithi177 ceph-mon[36206]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:20.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:20 smithi169 ceph-mon[32274]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:20.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:20 smithi174 ceph-mon[36060]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:22.046 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 8 on host 'smithi177' 2024-03-28T00:52:22.046 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:22.042+0000 7f34a57fa700 1 -- 172.21.15.177:0/1543649381 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f34a80625d0 con 0x7f34900a2890 2024-03-28T00:52:22.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:22.044+0000 7f34ae400700 1 -- 172.21.15.177:0/1543649381 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f34900a2890 msgr2=0x7f34900a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:52:22.048 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:22.044+0000 7f34ae400700 1 --2- 172.21.15.177:0/1543649381 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f34900a2890 0x7f34900a4d50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f34980039c0 tx=0x7f3498005fb0 comp rx=0 tx=0).stop 2024-03-28T00:52:22.048 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:22.044+0000 7f34ae400700 1 -- 172.21.15.177:0/1543649381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a8102360 msgr2=0x7f34a819d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:52:22.048 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:22.044+0000 7f34ae400700 1 --2- 172.21.15.177:0/1543649381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a8102360 0x7f34a819d160 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f349c00d980 tx=0x7f349c00dd40 comp rx=0 tx=0).stop 2024-03-28T00:52:22.048 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:22.044+0000 7f34ae400700 1 -- 172.21.15.177:0/1543649381 shutdown_connections 2024-03-28T00:52:22.048 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:22.044+0000 7f34ae400700 1 --2- 172.21.15.177:0/1543649381 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34a8102d00 0x7f34a819f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:22.048 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:22.044+0000 7f34ae400700 1 --2- 172.21.15.177:0/1543649381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34a8101160 0x7f34a806b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:22.048 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:22.044+0000 7f34ae400700 1 --2- 172.21.15.177:0/1543649381 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f34900a2890 0x7f34900a4d50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:22.048 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:22.044+0000 7f34ae400700 1 --2- 172.21.15.177:0/1543649381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a8102360 0x7f34a819d160 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:22.048 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:22.044+0000 7f34ae400700 1 -- 172.21.15.177:0/1543649381 >> 172.21.15.177:0/1543649381 conn(0x7f34a80fc720 msgr2=0x7f34a81056b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:52:22.048 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:22.044+0000 7f34ae400700 1 -- 172.21.15.177:0/1543649381 shutdown_connections 2024-03-28T00:52:22.048 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:52:22.044+0000 7f34ae400700 1 -- 172.21.15.177:0/1543649381 wait complete. 2024-03-28T00:52:22.730 DEBUG:teuthology.orchestra.run.smithi177:osd.8> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.8.service 2024-03-28T00:52:22.732 INFO:tasks.cephadm:Deploying osd.9 on smithi169 with /dev/vg_nvme/lv_1... 2024-03-28T00:52:22.733 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T00:52:22.982 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:52:23.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:23 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:23.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:23 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:23.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:23 smithi169 ceph-mon[32274]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:23.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:23 smithi169 ceph-mon[32274]: from='osd.8 [v2:172.21.15.177:6816/4108616824,v1:172.21.15.177:6817/4108616824]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T00:52:23.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:23 smithi169 ceph-mon[32274]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T00:52:23.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:23 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:23.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:23 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:23.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:23 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:23.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:23 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:23.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:23 smithi174 ceph-mon[36060]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:23.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:23 smithi174 ceph-mon[36060]: from='osd.8 [v2:172.21.15.177:6816/4108616824,v1:172.21.15.177:6817/4108616824]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T00:52:23.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:23 smithi174 ceph-mon[36060]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T00:52:23.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:23 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:23.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:23 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:23.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:23 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:23.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:23 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:23.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:23 smithi177 ceph-mon[36206]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:23.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:23 smithi177 ceph-mon[36206]: from='osd.8 [v2:172.21.15.177:6816/4108616824,v1:172.21.15.177:6817/4108616824]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T00:52:23.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:23 smithi177 ceph-mon[36206]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T00:52:23.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:23 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:23.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:23 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:24.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:24 smithi177 ceph-mon[36206]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T00:52:24.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:24 smithi177 ceph-mon[36206]: from='osd.8 [v2:172.21.15.177:6816/4108616824,v1:172.21.15.177:6817/4108616824]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:52:24.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:24 smithi177 ceph-mon[36206]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T00:52:24.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:24 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T00:52:24.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:24 smithi177 ceph-mon[36206]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:52:24.168 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 00:52:24 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T00:52:24.046+0000 7fdc3fb97700 -1 osd.8 0 waiting for initial osdmap 2024-03-28T00:52:24.168 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 00:52:24 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T00:52:24.058+0000 7fdc3b98a700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T00:52:24.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:24 smithi169 ceph-mon[32274]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T00:52:24.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:24 smithi169 ceph-mon[32274]: from='osd.8 [v2:172.21.15.177:6816/4108616824,v1:172.21.15.177:6817/4108616824]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:52:24.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:24 smithi169 ceph-mon[32274]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T00:52:24.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:24 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T00:52:24.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:24 smithi169 ceph-mon[32274]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:52:24.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:24 smithi174 ceph-mon[36060]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T00:52:24.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:24 smithi174 ceph-mon[36060]: from='osd.8 [v2:172.21.15.177:6816/4108616824,v1:172.21.15.177:6817/4108616824]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:52:24.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:24 smithi174 ceph-mon[36060]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T00:52:24.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:24 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T00:52:24.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:24 smithi174 ceph-mon[36060]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:52:24.967 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T00:52:24.967 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T00:52:24.967 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2024-03-28T00:52:24.967 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2024-03-28T00:52:24.967 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0139405 s, 752 MB/s 2024-03-28T00:52:24.967 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2024-03-28T00:52:25.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:25 smithi174 ceph-mon[36060]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-28T00:52:25.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:25 smithi174 ceph-mon[36060]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T00:52:25.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:25 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T00:52:25.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:25 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T00:52:25.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:25 smithi174 ceph-mon[36060]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:25.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:25 smithi177 ceph-mon[36206]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-28T00:52:25.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:25 smithi177 ceph-mon[36206]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T00:52:25.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:25 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T00:52:25.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:25 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T00:52:25.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:25 smithi177 ceph-mon[36206]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:25.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:25 smithi169 ceph-mon[32274]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-28T00:52:25.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:25 smithi169 ceph-mon[32274]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T00:52:25.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:25 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T00:52:25.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:25 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T00:52:25.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:25 smithi169 ceph-mon[32274]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-28T00:52:25.732 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_1 2024-03-28T00:52:25.970 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:52:26.313 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:26 smithi177 ceph-mon[36206]: purged_snaps scrub starts 2024-03-28T00:52:26.313 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:26 smithi177 ceph-mon[36206]: purged_snaps scrub ok 2024-03-28T00:52:26.313 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:26 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T00:52:26.313 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:26 smithi177 ceph-mon[36206]: osd.8 [v2:172.21.15.177:6816/4108616824,v1:172.21.15.177:6817/4108616824] boot 2024-03-28T00:52:26.313 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:26 smithi177 ceph-mon[36206]: osdmap e59: 9 total, 9 up, 9 in 2024-03-28T00:52:26.313 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:26 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T00:52:26.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:26 smithi174 ceph-mon[36060]: purged_snaps scrub starts 2024-03-28T00:52:26.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:26 smithi174 ceph-mon[36060]: purged_snaps scrub ok 2024-03-28T00:52:26.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:26 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T00:52:26.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:26 smithi174 ceph-mon[36060]: osd.8 [v2:172.21.15.177:6816/4108616824,v1:172.21.15.177:6817/4108616824] boot 2024-03-28T00:52:26.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:26 smithi174 ceph-mon[36060]: osdmap e59: 9 total, 9 up, 9 in 2024-03-28T00:52:26.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:26 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T00:52:26.375 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:26 smithi169 ceph-mon[32274]: purged_snaps scrub starts 2024-03-28T00:52:26.375 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:26 smithi169 ceph-mon[32274]: purged_snaps scrub ok 2024-03-28T00:52:26.375 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:26 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T00:52:26.375 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:26 smithi169 ceph-mon[32274]: osd.8 [v2:172.21.15.177:6816/4108616824,v1:172.21.15.177:6817/4108616824] boot 2024-03-28T00:52:26.375 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:26 smithi169 ceph-mon[32274]: osdmap e59: 9 total, 9 up, 9 in 2024-03-28T00:52:26.375 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:26 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T00:52:27.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:27 smithi174 ceph-mon[36060]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:27.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:27 smithi174 ceph-mon[36060]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T00:52:27.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:27 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:27.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:27 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:27.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:27 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:27.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:27 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:27.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:27 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:27.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:27 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:27.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:27 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:27.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:27 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:52:27.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:27 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:27.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:27 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:52:27.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:27 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:27.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:27 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:52:27.360 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:27 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:27.375 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:27 smithi169 ceph-mon[32274]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:27.375 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:27 smithi169 ceph-mon[32274]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T00:52:27.375 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:27 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:27.375 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:27 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:27.375 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:27 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:27.375 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:27 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:27.376 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:27 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:27.376 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:27 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:27.376 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:27 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:27.376 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:27 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:52:27.376 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:27 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:27.376 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:27 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:52:27.376 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:27 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:27.376 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:27 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:52:27.376 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:27 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:27.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:27 smithi177 ceph-mon[36206]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:27.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:27 smithi177 ceph-mon[36206]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T00:52:27.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:27 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:27.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:27 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:27.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:27 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:27.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:27 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:27.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:27 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:27.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:27 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:27.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:27 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:27.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:27 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:52:27.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:27 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:27.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:27 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:52:27.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:27 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:27.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:27 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:52:27.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:27 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:27.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.637+0000 7feb16e20700 1 -- 172.21.15.169:0/1077248071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb10107f20 msgr2=0x7feb1010a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:52:27.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.637+0000 7feb16e20700 1 --2- 172.21.15.169:0/1077248071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb10107f20 0x7feb1010a390 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7feb0800b0a0 tx=0x7feb0800b3b0 comp rx=0 tx=0).stop 2024-03-28T00:52:27.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.637+0000 7feb16e20700 1 -- 172.21.15.169:0/1077248071 shutdown_connections 2024-03-28T00:52:27.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.637+0000 7feb16e20700 1 --2- 172.21.15.169:0/1077248071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb10107f20 0x7feb1010a390 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:27.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.637+0000 7feb16e20700 1 --2- 172.21.15.169:0/1077248071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb10102cc0 0x7feb101050b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:27.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.637+0000 7feb16e20700 1 --2- 172.21.15.169:0/1077248071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb101055f0 0x7feb101079e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:27.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.637+0000 7feb16e20700 1 -- 172.21.15.169:0/1077248071 >> 172.21.15.169:0/1077248071 conn(0x7feb100fc6f0 msgr2=0x7feb100feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:52:27.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.637+0000 7feb16e20700 1 -- 172.21.15.169:0/1077248071 shutdown_connections 2024-03-28T00:52:27.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.637+0000 7feb16e20700 1 -- 172.21.15.169:0/1077248071 wait complete. 2024-03-28T00:52:27.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.637+0000 7feb16e20700 1 Processor -- start 2024-03-28T00:52:27.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.637+0000 7feb16e20700 1 -- start start 2024-03-28T00:52:27.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.638+0000 7feb16e20700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb10102cc0 0x7feb10198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:52:27.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.638+0000 7feb16e20700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb101055f0 0x7feb10198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:52:27.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.638+0000 7feb16e20700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb10107f20 0x7feb1019f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:52:27.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.638+0000 7feb16e20700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb1010ddd0 con 0x7feb10107f20 2024-03-28T00:52:27.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.638+0000 7feb16e20700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb1010dc50 con 0x7feb10102cc0 2024-03-28T00:52:27.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.638+0000 7feb16e20700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb1010df50 con 0x7feb101055f0 2024-03-28T00:52:27.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.638+0000 7feb153bd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb10107f20 0x7feb1019f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:52:27.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.638+0000 7feb153bd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb10107f20 0x7feb1019f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60522/0 (socket says 172.21.15.169:60522) 2024-03-28T00:52:27.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.638+0000 7feb153bd700 1 -- 172.21.15.169:0/35063487 learned_addr learned my addr 172.21.15.169:0/35063487 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:52:27.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.638+0000 7feb07fff700 1 --2- 172.21.15.169:0/35063487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb101055f0 0x7feb10198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:52:27.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.638+0000 7feb14bbc700 1 --2- 172.21.15.169:0/35063487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb10102cc0 0x7feb10198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:52:27.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.638+0000 7feb153bd700 1 -- 172.21.15.169:0/35063487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb101055f0 msgr2=0x7feb10198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:52:27.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.638+0000 7feb153bd700 1 --2- 172.21.15.169:0/35063487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb101055f0 0x7feb10198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:27.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.638+0000 7feb153bd700 1 -- 172.21.15.169:0/35063487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb10102cc0 msgr2=0x7feb10198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:52:27.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.638+0000 7feb153bd700 1 --2- 172.21.15.169:0/35063487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb10102cc0 0x7feb10198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:27.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.638+0000 7feb153bd700 1 -- 172.21.15.169:0/35063487 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb1019ffb0 con 0x7feb10107f20 2024-03-28T00:52:27.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.638+0000 7feb07fff700 1 --2- 172.21.15.169:0/35063487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb101055f0 0x7feb10198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:52:27.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.638+0000 7feb153bd700 1 --2- 172.21.15.169:0/35063487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb10107f20 0x7feb1019f910 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7feb0800d010 tx=0x7feb08004ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:52:27.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.638+0000 7feb05ffb700 1 -- 172.21.15.169:0/35063487 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb08021070 con 0x7feb10107f20 2024-03-28T00:52:27.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.639+0000 7feb16e20700 1 -- 172.21.15.169:0/35063487 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb101a0240 con 0x7feb10107f20 2024-03-28T00:52:27.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.639+0000 7feb05ffb700 1 -- 172.21.15.169:0/35063487 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feb08007ca0 con 0x7feb10107f20 2024-03-28T00:52:27.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.639+0000 7feb05ffb700 1 -- 172.21.15.169:0/35063487 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb080055d0 con 0x7feb10107f20 2024-03-28T00:52:27.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.639+0000 7feb16e20700 1 -- 172.21.15.169:0/35063487 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb101a0740 con 0x7feb10107f20 2024-03-28T00:52:27.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.640+0000 7feb16e20700 1 -- 172.21.15.169:0/35063487 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb10068e50 con 0x7feb10107f20 2024-03-28T00:52:27.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.641+0000 7feb05ffb700 1 -- 172.21.15.169:0/35063487 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7feb08024070 con 0x7feb10107f20 2024-03-28T00:52:27.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.641+0000 7feb05ffb700 1 --2- 172.21.15.169:0/35063487 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7feaf80a2890 0x7feaf80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:52:27.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.641+0000 7feb14bbc700 1 --2- 172.21.15.169:0/35063487 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7feaf80a2890 0x7feaf80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:52:27.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.641+0000 7feb05ffb700 1 -- 172.21.15.169:0/35063487 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5858+0+0 (secure 0 0 0) 0x7feb0800f020 con 0x7feb10107f20 2024-03-28T00:52:27.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.642+0000 7feb14bbc700 1 --2- 172.21.15.169:0/35063487 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7feaf80a2890 0x7feaf80a4d50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7feb0c005950 tx=0x7feb0c005eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:52:27.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.646+0000 7feb05ffb700 1 -- 172.21.15.169:0/35063487 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7feb0802b020 con 0x7feb10107f20 2024-03-28T00:52:27.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:27.850+0000 7feb16e20700 1 -- 172.21.15.169:0/35063487 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7feb100625d0 con 0x7feaf80a2890 2024-03-28T00:52:28.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:28 smithi177 ceph-mon[36206]: Detected new or changed devices on smithi177 2024-03-28T00:52:28.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:28 smithi177 ceph-mon[36206]: Adjusting osd_memory_target on smithi177 to 5693M 2024-03-28T00:52:28.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:28 smithi177 ceph-mon[36206]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T00:52:28.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:28 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:52:28.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:28 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:52:28.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:28 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:28.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:28 smithi174 ceph-mon[36060]: Detected new or changed devices on smithi177 2024-03-28T00:52:28.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:28 smithi174 ceph-mon[36060]: Adjusting osd_memory_target on smithi177 to 5693M 2024-03-28T00:52:28.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:28 smithi174 ceph-mon[36060]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T00:52:28.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:28 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:52:28.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:28 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:52:28.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:28 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:28.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:28 smithi169 ceph-mon[32274]: Detected new or changed devices on smithi177 2024-03-28T00:52:28.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:28 smithi169 ceph-mon[32274]: Adjusting osd_memory_target on smithi177 to 5693M 2024-03-28T00:52:28.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:28 smithi169 ceph-mon[32274]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T00:52:28.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:28 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:52:28.453 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:28 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:52:28.453 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:28 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:29.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:29 smithi174 ceph-mon[36060]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:52:29.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:29 smithi174 ceph-mon[36060]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:29.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:29 smithi174 ceph-mon[36060]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T00:52:29.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:29 smithi177 ceph-mon[36206]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:52:29.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:29 smithi177 ceph-mon[36206]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:29.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:29 smithi177 ceph-mon[36206]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T00:52:29.455 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:29 smithi169 ceph-mon[32274]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:52:29.455 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:29 smithi169 ceph-mon[32274]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:29.455 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:29 smithi169 ceph-mon[32274]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T00:52:30.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:30 smithi169 ceph-mon[32274]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T00:52:30.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:30 smithi174 ceph-mon[36060]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T00:52:30.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:30 smithi177 ceph-mon[36206]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T00:52:31.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:31 smithi169 ceph-mon[32274]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T00:52:31.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:31 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1024878243' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6a429af-798a-48e3-8368-ad9338daa9b8"}]: dispatch 2024-03-28T00:52:31.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:31 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1024878243' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6a429af-798a-48e3-8368-ad9338daa9b8"}]': finished 2024-03-28T00:52:31.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:31 smithi169 ceph-mon[32274]: osdmap e64: 10 total, 9 up, 10 in 2024-03-28T00:52:31.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:31 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T00:52:31.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:31 smithi174 ceph-mon[36060]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T00:52:31.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:31 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1024878243' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6a429af-798a-48e3-8368-ad9338daa9b8"}]: dispatch 2024-03-28T00:52:31.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:31 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1024878243' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6a429af-798a-48e3-8368-ad9338daa9b8"}]': finished 2024-03-28T00:52:31.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:31 smithi174 ceph-mon[36060]: osdmap e64: 10 total, 9 up, 10 in 2024-03-28T00:52:31.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:31 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T00:52:31.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:31 smithi177 ceph-mon[36206]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T00:52:31.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:31 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1024878243' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6a429af-798a-48e3-8368-ad9338daa9b8"}]: dispatch 2024-03-28T00:52:31.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:31 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1024878243' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6a429af-798a-48e3-8368-ad9338daa9b8"}]': finished 2024-03-28T00:52:31.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:31 smithi177 ceph-mon[36206]: osdmap e64: 10 total, 9 up, 10 in 2024-03-28T00:52:31.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:31 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T00:52:32.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:32 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2205988553' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:52:32.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:32 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2205988553' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:52:32.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:32 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2205988553' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:52:33.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:33 smithi174 ceph-mon[36060]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-28T00:52:33.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:33 smithi177 ceph-mon[36206]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-28T00:52:33.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:33 smithi169 ceph-mon[32274]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-28T00:52:35.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:35 smithi174 ceph-mon[36060]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T00:52:35.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:35 smithi177 ceph-mon[36206]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T00:52:35.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:35 smithi169 ceph-mon[32274]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T00:52:37.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:37 smithi174 ceph-mon[36060]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T00:52:37.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:37 smithi177 ceph-mon[36206]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T00:52:37.503 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:37 smithi169 ceph-mon[32274]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T00:52:38.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:38 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T00:52:38.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:38 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:38.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:38 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T00:52:38.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:38 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:38.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:38 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T00:52:38.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:38 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:39.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:39 smithi169 ceph-mon[32274]: Deploying daemon osd.9 on smithi169 2024-03-28T00:52:39.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:39 smithi169 ceph-mon[32274]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:39.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:39 smithi177 ceph-mon[36206]: Deploying daemon osd.9 on smithi169 2024-03-28T00:52:39.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:39 smithi177 ceph-mon[36206]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:39.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:39 smithi174 ceph-mon[36060]: Deploying daemon osd.9 on smithi169 2024-03-28T00:52:39.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:39 smithi174 ceph-mon[36060]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:41.413 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:41 smithi169 ceph-mon[32274]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:41.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:41 smithi177 ceph-mon[36206]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:41.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:41 smithi174 ceph-mon[36060]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:43.226 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:43 smithi169 ceph-mon[32274]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:43.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:43 smithi177 ceph-mon[36206]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:43.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:43 smithi174 ceph-mon[36060]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:45.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:44 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:45.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:44 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:52:45.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:44 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:45.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:44 smithi169 ceph-mon[32274]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:45.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:44 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:45.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:44 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:52:45.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:44 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:45.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:44 smithi174 ceph-mon[36060]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:45.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:44 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:45.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:44 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:52:45.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:44 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:45.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:44 smithi177 ceph-mon[36206]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:47.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:47 smithi177 ceph-mon[36206]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:47.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:47 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:47.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:47 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:47.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:47 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:47.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:47 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:52:47.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:47 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:47.480 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:47 smithi169 ceph-mon[32274]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:47.480 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:47 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:47.481 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:47 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:47.481 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:47 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:47.481 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:47 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:52:47.481 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:47 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:47.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:47 smithi174 ceph-mon[36060]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:47.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:47 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:47.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:47 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:47.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:47 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:47.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:47 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:52:47.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:47 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:47.873 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 9 on host 'smithi169' 2024-03-28T00:52:47.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:47.869+0000 7feb05ffb700 1 -- 172.21.15.169:0/35063487 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7feb100625d0 con 0x7feaf80a2890 2024-03-28T00:52:47.876 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:47.872+0000 7feb16e20700 1 -- 172.21.15.169:0/35063487 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7feaf80a2890 msgr2=0x7feaf80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:52:47.876 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:47.872+0000 7feb16e20700 1 --2- 172.21.15.169:0/35063487 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7feaf80a2890 0x7feaf80a4d50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7feb0c005950 tx=0x7feb0c005eb0 comp rx=0 tx=0).stop 2024-03-28T00:52:47.876 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:47.872+0000 7feb16e20700 1 -- 172.21.15.169:0/35063487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb10107f20 msgr2=0x7feb1019f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:52:47.876 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:47.872+0000 7feb16e20700 1 --2- 172.21.15.169:0/35063487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb10107f20 0x7feb1019f910 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7feb0800d010 tx=0x7feb08004ef0 comp rx=0 tx=0).stop 2024-03-28T00:52:47.876 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:47.873+0000 7feb16e20700 1 -- 172.21.15.169:0/35063487 shutdown_connections 2024-03-28T00:52:47.876 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:47.873+0000 7feb16e20700 1 --2- 172.21.15.169:0/35063487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb10107f20 0x7feb1019f910 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:47.876 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:47.873+0000 7feb16e20700 1 --2- 172.21.15.169:0/35063487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb10102cc0 0x7feb10198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:47.876 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:47.873+0000 7feb16e20700 1 --2- 172.21.15.169:0/35063487 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7feaf80a2890 0x7feaf80a4d50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:47.876 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:47.873+0000 7feb16e20700 1 --2- 172.21.15.169:0/35063487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb101055f0 0x7feb10198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:47.876 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:47.873+0000 7feb16e20700 1 -- 172.21.15.169:0/35063487 >> 172.21.15.169:0/35063487 conn(0x7feb100fc6f0 msgr2=0x7feb10103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:52:47.877 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:47.873+0000 7feb16e20700 1 -- 172.21.15.169:0/35063487 shutdown_connections 2024-03-28T00:52:47.877 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:52:47.873+0000 7feb16e20700 1 -- 172.21.15.169:0/35063487 wait complete. 2024-03-28T00:52:48.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:48 smithi177 ceph-mon[36206]: from='osd.9 [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T00:52:48.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:48 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:48.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:48 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:48.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:48 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:52:48.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:48 smithi169 ceph-mon[32274]: from='osd.9 [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T00:52:48.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:48 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:48.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:48 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:48.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:48 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:52:48.608 DEBUG:teuthology.orchestra.run.smithi169:osd.9> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.9.service 2024-03-28T00:52:48.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:48 smithi174 ceph-mon[36060]: from='osd.9 [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T00:52:48.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:48 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:48.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:48 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:48.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:48 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:52:48.611 INFO:tasks.cephadm:Deploying osd.10 on smithi174 with /dev/vg_nvme/lv_1... 2024-03-28T00:52:48.611 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T00:52:48.860 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T00:52:49.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:49 smithi169 ceph-mon[32274]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:49.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:49 smithi169 ceph-mon[32274]: from='osd.9 [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T00:52:49.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:49 smithi169 ceph-mon[32274]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T00:52:49.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:49 smithi169 ceph-mon[32274]: from='osd.9 [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-28T00:52:49.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:49 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T00:52:49.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 00:52:49 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T00:52:49.123+0000 7f5d2f6d2700 -1 osd.9 0 waiting for initial osdmap 2024-03-28T00:52:49.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 00:52:49 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T00:52:49.136+0000 7f5d2b4c5700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T00:52:49.309 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:49 smithi174 ceph-mon[36060]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:49.309 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:49 smithi174 ceph-mon[36060]: from='osd.9 [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T00:52:49.309 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:49 smithi174 ceph-mon[36060]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T00:52:49.309 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:49 smithi174 ceph-mon[36060]: from='osd.9 [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-28T00:52:49.309 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:49 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T00:52:49.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:49 smithi177 ceph-mon[36206]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:49.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:49 smithi177 ceph-mon[36206]: from='osd.9 [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T00:52:49.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:49 smithi177 ceph-mon[36206]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T00:52:49.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:49 smithi177 ceph-mon[36206]: from='osd.9 [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-03-28T00:52:49.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:49 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T00:52:50.251 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:50 smithi174 ceph-mon[36060]: from='osd.9 [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-28T00:52:50.251 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:50 smithi174 ceph-mon[36060]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T00:52:50.251 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:50 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T00:52:50.251 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:50 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T00:52:50.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:50 smithi177 ceph-mon[36206]: from='osd.9 [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-28T00:52:50.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:50 smithi177 ceph-mon[36206]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T00:52:50.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:50 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T00:52:50.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:50 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T00:52:50.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:50 smithi169 ceph-mon[32274]: from='osd.9 [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-03-28T00:52:50.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:50 smithi169 ceph-mon[32274]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T00:52:50.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:50 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T00:52:50.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:50 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T00:52:50.821 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T00:52:50.821 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T00:52:50.821 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-03-28T00:52:50.821 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-03-28T00:52:50.821 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113821 s, 921 MB/s 2024-03-28T00:52:50.821 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-03-28T00:52:51.267 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:51 smithi169 ceph-mon[32274]: purged_snaps scrub starts 2024-03-28T00:52:51.267 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:51 smithi169 ceph-mon[32274]: purged_snaps scrub ok 2024-03-28T00:52:51.267 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:51 smithi169 ceph-mon[32274]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:51.267 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:51 smithi169 ceph-mon[32274]: osd.9 [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290] boot 2024-03-28T00:52:51.267 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:51 smithi169 ceph-mon[32274]: osdmap e67: 10 total, 10 up, 10 in 2024-03-28T00:52:51.267 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:51 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T00:52:51.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:51 smithi177 ceph-mon[36206]: purged_snaps scrub starts 2024-03-28T00:52:51.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:51 smithi177 ceph-mon[36206]: purged_snaps scrub ok 2024-03-28T00:52:51.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:51 smithi177 ceph-mon[36206]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:51.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:51 smithi177 ceph-mon[36206]: osd.9 [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290] boot 2024-03-28T00:52:51.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:51 smithi177 ceph-mon[36206]: osdmap e67: 10 total, 10 up, 10 in 2024-03-28T00:52:51.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:51 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T00:52:51.495 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:51 smithi174 ceph-mon[36060]: purged_snaps scrub starts 2024-03-28T00:52:51.495 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:51 smithi174 ceph-mon[36060]: purged_snaps scrub ok 2024-03-28T00:52:51.495 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:51 smithi174 ceph-mon[36060]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-28T00:52:51.495 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:51 smithi174 ceph-mon[36060]: osd.9 [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290] boot 2024-03-28T00:52:51.495 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:51 smithi174 ceph-mon[36060]: osdmap e67: 10 total, 10 up, 10 in 2024-03-28T00:52:51.495 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:51 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T00:52:51.522 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi174:vg_nvme/lv_1 2024-03-28T00:52:51.758 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T00:52:52.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:52 smithi177 ceph-mon[36206]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T00:52:52.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:52 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:52.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:52 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:52.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:52 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:52.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:52 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:52.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:52 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:52.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:52 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:52.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:52 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:52.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:52 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:52.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:52 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:52:52.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:52 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:52.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:52 smithi169 ceph-mon[32274]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T00:52:52.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:52 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:52.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:52 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:52.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:52 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:52.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:52 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:52.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:52 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:52.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:52 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:52.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:52 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:52.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:52 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:52.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:52 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:52:52.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:52 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:52.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:52 smithi174 ceph-mon[36060]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T00:52:52.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:52 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:52.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:52 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:52.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:52 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:52.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:52 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:52.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:52 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:52.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:52 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:52:52.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:52 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:52.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:52 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:52.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:52 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:52:52.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:52 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:52:53.414 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:53 smithi174 ceph-mon[36060]: Detected new or changed devices on smithi169 2024-03-28T00:52:53.414 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:53 smithi174 ceph-mon[36060]: Adjusting osd_memory_target on smithi169 to 4272M 2024-03-28T00:52:53.414 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:53 smithi174 ceph-mon[36060]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:52:53.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:53 smithi177 ceph-mon[36206]: Detected new or changed devices on smithi169 2024-03-28T00:52:53.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:53 smithi177 ceph-mon[36206]: Adjusting osd_memory_target on smithi169 to 4272M 2024-03-28T00:52:53.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:53 smithi177 ceph-mon[36206]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:52:53.427 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.421+0000 7f3e47649700 1 -- 172.21.15.174:0/2902940259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e40103480 msgr2=0x7f3e401038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:52:53.427 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.421+0000 7f3e47649700 1 --2- 172.21.15.174:0/2902940259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e40103480 0x7f3e401038e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3e30009990 tx=0x7f3e30009ca0 comp rx=0 tx=0).stop 2024-03-28T00:52:53.427 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.421+0000 7f3e47649700 1 -- 172.21.15.174:0/2902940259 shutdown_connections 2024-03-28T00:52:53.427 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.421+0000 7f3e47649700 1 --2- 172.21.15.174:0/2902940259 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e40103e20 0x7f3e4010a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:53.427 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.421+0000 7f3e47649700 1 --2- 172.21.15.174:0/2902940259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e401004a0 0x7f3e401008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:53.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.421+0000 7f3e47649700 1 --2- 172.21.15.174:0/2902940259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e40103480 0x7f3e401038e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:53.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.421+0000 7f3e47649700 1 -- 172.21.15.174:0/2902940259 >> 172.21.15.174:0/2902940259 conn(0x7f3e400fbde0 msgr2=0x7f3e400fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:52:53.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.421+0000 7f3e47649700 1 -- 172.21.15.174:0/2902940259 shutdown_connections 2024-03-28T00:52:53.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.421+0000 7f3e47649700 1 -- 172.21.15.174:0/2902940259 wait complete. 2024-03-28T00:52:53.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.422+0000 7f3e47649700 1 Processor -- start 2024-03-28T00:52:53.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.422+0000 7f3e47649700 1 -- start start 2024-03-28T00:52:53.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.422+0000 7f3e47649700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e401004a0 0x7f3e4019a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:52:53.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.422+0000 7f3e47649700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e40103480 0x7f3e4019aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:52:53.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.422+0000 7f3e47649700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e40103e20 0x7f3e401a1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:52:53.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.422+0000 7f3e47649700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4010ffd0 con 0x7f3e40103e20 2024-03-28T00:52:53.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.422+0000 7f3e47649700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4010fe50 con 0x7f3e40103480 2024-03-28T00:52:53.428 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.422+0000 7f3e47649700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e40110150 con 0x7f3e401004a0 2024-03-28T00:52:53.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.422+0000 7f3e44be4700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e40103480 0x7f3e4019aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:52:53.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.422+0000 7f3e453e5700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e401004a0 0x7f3e4019a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:52:53.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.422+0000 7f3e44be4700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e40103480 0x7f3e4019aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:40176/0 (socket says 172.21.15.174:40176) 2024-03-28T00:52:53.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.422+0000 7f3e44be4700 1 -- 172.21.15.174:0/817221417 learned_addr learned my addr 172.21.15.174:0/817221417 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-03-28T00:52:53.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.422+0000 7f3e45be6700 1 --2- 172.21.15.174:0/817221417 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e40103e20 0x7f3e401a1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:52:53.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.422+0000 7f3e44be4700 1 -- 172.21.15.174:0/817221417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e401004a0 msgr2=0x7f3e4019a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:52:53.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.422+0000 7f3e44be4700 1 --2- 172.21.15.174:0/817221417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e401004a0 0x7f3e4019a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:53.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.422+0000 7f3e44be4700 1 -- 172.21.15.174:0/817221417 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e40103e20 msgr2=0x7f3e401a1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:52:53.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.422+0000 7f3e44be4700 1 --2- 172.21.15.174:0/817221417 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e40103e20 0x7f3e401a1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:52:53.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.423+0000 7f3e44be4700 1 -- 172.21.15.174:0/817221417 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e401a21a0 con 0x7f3e40103480 2024-03-28T00:52:53.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.423+0000 7f3e44be4700 1 --2- 172.21.15.174:0/817221417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e40103480 0x7f3e4019aa80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3e300098e0 tx=0x7f3e300057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:52:53.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.423+0000 7f3e367fc700 1 -- 172.21.15.174:0/817221417 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e30027070 con 0x7f3e40103480 2024-03-28T00:52:53.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.423+0000 7f3e367fc700 1 -- 172.21.15.174:0/817221417 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e30004150 con 0x7f3e40103480 2024-03-28T00:52:53.429 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.423+0000 7f3e47649700 1 -- 172.21.15.174:0/817221417 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e401a2430 con 0x7f3e40103480 2024-03-28T00:52:53.430 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.423+0000 7f3e367fc700 1 -- 172.21.15.174:0/817221417 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3000f460 con 0x7f3e40103480 2024-03-28T00:52:53.430 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.423+0000 7f3e47649700 1 -- 172.21.15.174:0/817221417 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e401a2880 con 0x7f3e40103480 2024-03-28T00:52:53.430 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.424+0000 7f3e47649700 1 -- 172.21.15.174:0/817221417 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e40068e50 con 0x7f3e40103480 2024-03-28T00:52:53.430 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.425+0000 7f3e367fc700 1 -- 172.21.15.174:0/817221417 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3e3002a070 con 0x7f3e40103480 2024-03-28T00:52:53.431 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.425+0000 7f3e367fc700 1 --2- 172.21.15.174:0/817221417 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3e2c0a2890 0x7f3e2c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:52:53.431 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.425+0000 7f3e367fc700 1 -- 172.21.15.174:0/817221417 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f3e300cf860 con 0x7f3e40103480 2024-03-28T00:52:53.431 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.426+0000 7f3e453e5700 1 --2- 172.21.15.174:0/817221417 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3e2c0a2890 0x7f3e2c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:52:53.431 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.426+0000 7f3e453e5700 1 --2- 172.21.15.174:0/817221417 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3e2c0a2890 0x7f3e2c0a4d50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3e4006a710 tx=0x7f3e3c007a30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:52:53.432 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.429+0000 7f3e367fc700 1 -- 172.21.15.174:0/817221417 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3e30031020 con 0x7f3e40103480 2024-03-28T00:52:53.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:53 smithi169 ceph-mon[32274]: Detected new or changed devices on smithi169 2024-03-28T00:52:53.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:53 smithi169 ceph-mon[32274]: Adjusting osd_memory_target on smithi169 to 4272M 2024-03-28T00:52:53.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:53 smithi169 ceph-mon[32274]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:52:53.637 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:52:53.633+0000 7f3e47649700 1 -- 172.21.15.174:0/817221417 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e400625d0 con 0x7f3e2c0a2890 2024-03-28T00:52:54.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:54 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:52:54.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:54 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:52:54.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:54 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:54.466 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:54 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:52:54.466 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:54 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:52:54.467 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:54 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:54.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:54 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:52:54.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:54 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:52:54.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:54 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:52:55.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:55 smithi177 ceph-mon[36206]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:52:55.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:55 smithi177 ceph-mon[36206]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:52:55.471 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:55 smithi174 ceph-mon[36060]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:52:55.471 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:55 smithi174 ceph-mon[36060]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:52:55.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:55 smithi169 ceph-mon[32274]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:52:55.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:55 smithi169 ceph-mon[32274]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:52:57.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:57 smithi177 ceph-mon[36206]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:52:57.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:57 smithi177 ceph-mon[36206]: from='client.? 172.21.15.174:0/947607117' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba098e93-e8fd-4dca-b2ff-1722425e2524"}]: dispatch 2024-03-28T00:52:57.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:57 smithi177 ceph-mon[36206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba098e93-e8fd-4dca-b2ff-1722425e2524"}]: dispatch 2024-03-28T00:52:57.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:57 smithi177 ceph-mon[36206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba098e93-e8fd-4dca-b2ff-1722425e2524"}]': finished 2024-03-28T00:52:57.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:57 smithi177 ceph-mon[36206]: osdmap e69: 11 total, 10 up, 11 in 2024-03-28T00:52:57.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:57 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T00:52:57.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:57 smithi177 ceph-mon[36206]: from='client.? 172.21.15.174:0/214743175' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:52:57.418 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:57 smithi169 ceph-mon[32274]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:52:57.418 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:57 smithi169 ceph-mon[32274]: from='client.? 172.21.15.174:0/947607117' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba098e93-e8fd-4dca-b2ff-1722425e2524"}]: dispatch 2024-03-28T00:52:57.418 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:57 smithi169 ceph-mon[32274]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba098e93-e8fd-4dca-b2ff-1722425e2524"}]: dispatch 2024-03-28T00:52:57.418 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:57 smithi169 ceph-mon[32274]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba098e93-e8fd-4dca-b2ff-1722425e2524"}]': finished 2024-03-28T00:52:57.418 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:57 smithi169 ceph-mon[32274]: osdmap e69: 11 total, 10 up, 11 in 2024-03-28T00:52:57.418 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:57 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T00:52:57.418 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:57 smithi169 ceph-mon[32274]: from='client.? 172.21.15.174:0/214743175' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:52:57.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:57 smithi174 ceph-mon[36060]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:52:57.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:57 smithi174 ceph-mon[36060]: from='client.? 172.21.15.174:0/947607117' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba098e93-e8fd-4dca-b2ff-1722425e2524"}]: dispatch 2024-03-28T00:52:57.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:57 smithi174 ceph-mon[36060]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba098e93-e8fd-4dca-b2ff-1722425e2524"}]: dispatch 2024-03-28T00:52:57.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:57 smithi174 ceph-mon[36060]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba098e93-e8fd-4dca-b2ff-1722425e2524"}]': finished 2024-03-28T00:52:57.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:57 smithi174 ceph-mon[36060]: osdmap e69: 11 total, 10 up, 11 in 2024-03-28T00:52:57.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:57 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T00:52:57.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:57 smithi174 ceph-mon[36060]: from='client.? 172.21.15.174:0/214743175' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:52:59.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:52:59 smithi177 ceph-mon[36206]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:52:59.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:52:59 smithi169 ceph-mon[32274]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:52:59.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:52:59 smithi174 ceph-mon[36060]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:01.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:01 smithi177 ceph-mon[36206]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:01.492 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:01 smithi174 ceph-mon[36060]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:01.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:01 smithi169 ceph-mon[32274]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:03.172 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:03 smithi174 ceph-mon[36060]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:03.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:03 smithi177 ceph-mon[36206]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:03.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:03 smithi169 ceph-mon[32274]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:04.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:04 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T00:53:04.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:04 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:04.438 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:04 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T00:53:04.438 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:04 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:04.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:04 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T00:53:04.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:04 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:05.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:05 smithi177 ceph-mon[36206]: Deploying daemon osd.10 on smithi174 2024-03-28T00:53:05.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:05 smithi177 ceph-mon[36206]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:05.441 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:05 smithi174 ceph-mon[36060]: Deploying daemon osd.10 on smithi174 2024-03-28T00:53:05.441 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:05 smithi174 ceph-mon[36060]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:05.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:05 smithi169 ceph-mon[32274]: Deploying daemon osd.10 on smithi174 2024-03-28T00:53:05.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:05 smithi169 ceph-mon[32274]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:07.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:07 smithi174 ceph-mon[36060]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:07.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:07 smithi169 ceph-mon[32274]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:07.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:07 smithi177 ceph-mon[36206]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:09.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:09 smithi169 ceph-mon[32274]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:09.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:09 smithi174 ceph-mon[36060]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:09.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:09 smithi177 ceph-mon[36206]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:10.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:10.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:53:10.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:10.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:10 smithi174 ceph-mon[36060]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:10.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:10.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:53:10.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:10.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:10 smithi169 ceph-mon[32274]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:10.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:10.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:53:10.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:10.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:10 smithi177 ceph-mon[36206]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:13.266 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:13 smithi174 ceph-mon[36060]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:13.266 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:13 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:13.266 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:13 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:13.267 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:13 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:13.267 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:13 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:53:13.267 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:13 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:13.309 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 10 on host 'smithi174' 2024-03-28T00:53:13.309 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:53:13.306+0000 7f3e367fc700 1 -- 172.21.15.174:0/817221417 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f3e400625d0 con 0x7f3e2c0a2890 2024-03-28T00:53:13.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:53:13.308+0000 7f3e47649700 1 -- 172.21.15.174:0/817221417 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3e2c0a2890 msgr2=0x7f3e2c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:13.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:53:13.308+0000 7f3e47649700 1 --2- 172.21.15.174:0/817221417 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3e2c0a2890 0x7f3e2c0a4d50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3e4006a710 tx=0x7f3e3c007a30 comp rx=0 tx=0).stop 2024-03-28T00:53:13.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:53:13.308+0000 7f3e47649700 1 -- 172.21.15.174:0/817221417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e40103480 msgr2=0x7f3e4019aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:13.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:53:13.308+0000 7f3e47649700 1 --2- 172.21.15.174:0/817221417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e40103480 0x7f3e4019aa80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3e300098e0 tx=0x7f3e300057c0 comp rx=0 tx=0).stop 2024-03-28T00:53:13.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:53:13.309+0000 7f3e47649700 1 -- 172.21.15.174:0/817221417 shutdown_connections 2024-03-28T00:53:13.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:53:13.309+0000 7f3e47649700 1 --2- 172.21.15.174:0/817221417 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e40103e20 0x7f3e401a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:13.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:53:13.309+0000 7f3e47649700 1 --2- 172.21.15.174:0/817221417 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3e2c0a2890 0x7f3e2c0a4d50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:13.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:53:13.309+0000 7f3e47649700 1 --2- 172.21.15.174:0/817221417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e401004a0 0x7f3e4019a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:13.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:53:13.309+0000 7f3e47649700 1 --2- 172.21.15.174:0/817221417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e40103480 0x7f3e4019aa80 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:13.313 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:53:13.309+0000 7f3e47649700 1 -- 172.21.15.174:0/817221417 >> 172.21.15.174:0/817221417 conn(0x7f3e400fbde0 msgr2=0x7f3e40106780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:53:13.313 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:53:13.309+0000 7f3e47649700 1 -- 172.21.15.174:0/817221417 shutdown_connections 2024-03-28T00:53:13.313 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-28T00:53:13.309+0000 7f3e47649700 1 -- 172.21.15.174:0/817221417 wait complete. 2024-03-28T00:53:13.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:13 smithi177 ceph-mon[36206]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:13.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:13 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:13.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:13 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:13.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:13 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:13.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:13 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:53:13.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:13 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:13.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:13 smithi169 ceph-mon[32274]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:13.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:13 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:13.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:13 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:13.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:13 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:13.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:13 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:53:13.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:13 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:13.919 DEBUG:teuthology.orchestra.run.smithi174:osd.10> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.10.service 2024-03-28T00:53:13.921 INFO:tasks.cephadm:Deploying osd.11 on smithi177 with /dev/vg_nvme/lv_1... 2024-03-28T00:53:13.921 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T00:53:14.168 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T00:53:14.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:14 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:14.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:14 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:14.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:14 smithi169 ceph-mon[32274]: from='osd.10 [v2:172.21.15.174:6824/3371468956,v1:172.21.15.174:6825/3371468956]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T00:53:14.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:14 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:53:14.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:14 smithi169 ceph-mon[32274]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T00:53:14.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:14 smithi169 ceph-mon[32274]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:14.553 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:14 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:14.553 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:14 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:14.553 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:14 smithi177 ceph-mon[36206]: from='osd.10 [v2:172.21.15.174:6824/3371468956,v1:172.21.15.174:6825/3371468956]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T00:53:14.553 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:14 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:53:14.553 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:14 smithi177 ceph-mon[36206]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T00:53:14.554 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:14 smithi177 ceph-mon[36206]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:14.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:14 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:14.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:14 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:14.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:14 smithi174 ceph-mon[36060]: from='osd.10 [v2:172.21.15.174:6824/3371468956,v1:172.21.15.174:6825/3371468956]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T00:53:14.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:14 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:53:14.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:14 smithi174 ceph-mon[36060]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T00:53:14.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:14 smithi174 ceph-mon[36060]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:15.582 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:15 smithi174 ceph-mon[36060]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T00:53:15.582 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:15 smithi174 ceph-mon[36060]: from='osd.10 [v2:172.21.15.174:6824/3371468956,v1:172.21.15.174:6825/3371468956]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:53:15.582 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:15 smithi174 ceph-mon[36060]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T00:53:15.582 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:15 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T00:53:15.582 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:15 smithi174 ceph-mon[36060]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:53:15.606 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:15 smithi177 ceph-mon[36206]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T00:53:15.607 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:15 smithi177 ceph-mon[36206]: from='osd.10 [v2:172.21.15.174:6824/3371468956,v1:172.21.15.174:6825/3371468956]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:53:15.607 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:15 smithi177 ceph-mon[36206]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T00:53:15.607 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:15 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T00:53:15.607 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:15 smithi177 ceph-mon[36206]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:53:15.722 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:15 smithi169 ceph-mon[32274]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T00:53:15.722 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:15 smithi169 ceph-mon[32274]: from='osd.10 [v2:172.21.15.174:6824/3371468956,v1:172.21.15.174:6825/3371468956]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:53:15.722 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:15 smithi169 ceph-mon[32274]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T00:53:15.722 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:15 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T00:53:15.722 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:15 smithi169 ceph-mon[32274]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-28T00:53:15.858 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 00:53:15 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T00:53:15.755+0000 7f1284918700 -1 osd.10 0 waiting for initial osdmap 2024-03-28T00:53:15.858 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 00:53:15 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T00:53:15.769+0000 7f127c703700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T00:53:16.342 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T00:53:16.342 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T00:53:16.342 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-03-28T00:53:16.342 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-03-28T00:53:16.343 INFO:teuthology.orchestra.run.smithi177.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0116687 s, 899 MB/s 2024-03-28T00:53:16.343 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-03-28T00:53:16.552 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:16 smithi174 ceph-mon[36060]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-28T00:53:16.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:16 smithi174 ceph-mon[36060]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T00:53:16.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:16 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T00:53:16.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:16 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T00:53:16.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:16 smithi174 ceph-mon[36060]: from='osd.10 ' entity='osd.10' 2024-03-28T00:53:16.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:16 smithi174 ceph-mon[36060]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:16.640 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:16 smithi177 ceph-mon[36206]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-28T00:53:16.640 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:16 smithi177 ceph-mon[36206]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T00:53:16.640 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:16 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T00:53:16.640 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:16 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T00:53:16.640 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:16 smithi177 ceph-mon[36206]: from='osd.10 ' entity='osd.10' 2024-03-28T00:53:16.640 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:16 smithi177 ceph-mon[36206]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:16.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:16 smithi169 ceph-mon[32274]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-28T00:53:16.673 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:16 smithi169 ceph-mon[32274]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T00:53:16.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:16 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T00:53:16.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:16 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T00:53:16.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:16 smithi169 ceph-mon[32274]: from='osd.10 ' entity='osd.10' 2024-03-28T00:53:16.674 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:16 smithi169 ceph-mon[32274]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-03-28T00:53:17.186 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi177:vg_nvme/lv_1 2024-03-28T00:53:17.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:17 smithi177 ceph-mon[36206]: purged_snaps scrub starts 2024-03-28T00:53:17.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:17 smithi177 ceph-mon[36206]: purged_snaps scrub ok 2024-03-28T00:53:17.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:17 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T00:53:17.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:17 smithi177 ceph-mon[36206]: osd.10 [v2:172.21.15.174:6824/3371468956,v1:172.21.15.174:6825/3371468956] boot 2024-03-28T00:53:17.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:17 smithi177 ceph-mon[36206]: osdmap e72: 11 total, 11 up, 11 in 2024-03-28T00:53:17.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:17 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T00:53:17.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:17 smithi177 ceph-mon[36206]: Detected new or changed devices on smithi174 2024-03-28T00:53:17.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:17 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:17.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:17 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:17.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:17 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:17.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:17 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:17.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:17 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:17.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:17 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:17.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:17 smithi177 ceph-mon[36206]: Adjusting osd_memory_target on smithi174 to 4270M 2024-03-28T00:53:17.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:17 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:17.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:17 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:17.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:17 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:53:17.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:17 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:17.437 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T00:53:17.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:17 smithi174 ceph-mon[36060]: purged_snaps scrub starts 2024-03-28T00:53:17.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:17 smithi174 ceph-mon[36060]: purged_snaps scrub ok 2024-03-28T00:53:17.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:17 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T00:53:17.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:17 smithi174 ceph-mon[36060]: osd.10 [v2:172.21.15.174:6824/3371468956,v1:172.21.15.174:6825/3371468956] boot 2024-03-28T00:53:17.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:17 smithi174 ceph-mon[36060]: osdmap e72: 11 total, 11 up, 11 in 2024-03-28T00:53:17.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:17 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T00:53:17.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:17 smithi174 ceph-mon[36060]: Detected new or changed devices on smithi174 2024-03-28T00:53:17.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:17 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:17.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:17 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:17.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:17 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:17.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:17 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:17.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:17 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:17.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:17 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:17.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:17 smithi174 ceph-mon[36060]: Adjusting osd_memory_target on smithi174 to 4270M 2024-03-28T00:53:17.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:17 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:17.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:17 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:17.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:17 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:53:17.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:17 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:17.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:17 smithi169 ceph-mon[32274]: purged_snaps scrub starts 2024-03-28T00:53:17.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:17 smithi169 ceph-mon[32274]: purged_snaps scrub ok 2024-03-28T00:53:17.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:17 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T00:53:17.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:17 smithi169 ceph-mon[32274]: osd.10 [v2:172.21.15.174:6824/3371468956,v1:172.21.15.174:6825/3371468956] boot 2024-03-28T00:53:17.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:17 smithi169 ceph-mon[32274]: osdmap e72: 11 total, 11 up, 11 in 2024-03-28T00:53:17.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:17 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T00:53:17.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:17 smithi169 ceph-mon[32274]: Detected new or changed devices on smithi174 2024-03-28T00:53:17.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:17 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:17.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:17 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:17.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:17 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:17.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:17 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:17.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:17 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:17.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:17 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:17.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:17 smithi169 ceph-mon[32274]: Adjusting osd_memory_target on smithi174 to 4270M 2024-03-28T00:53:17.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:17 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:17.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:17 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:17.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:17 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:53:17.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:17 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:18.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:18 smithi174 ceph-mon[36060]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T00:53:18.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:18 smithi174 ceph-mon[36060]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:18.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:18 smithi177 ceph-mon[36206]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T00:53:18.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:18 smithi177 ceph-mon[36206]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:18.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:18 smithi169 ceph-mon[32274]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T00:53:18.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:18 smithi169 ceph-mon[32274]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:19.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.220+0000 7fa44501d700 1 -- 172.21.15.177:0/538121296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4401022c0 msgr2=0x7fa440102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:19.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.220+0000 7fa44501d700 1 --2- 172.21.15.177:0/538121296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4401022c0 0x7fa440102720 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa434009990 tx=0x7fa434009ca0 comp rx=0 tx=0).stop 2024-03-28T00:53:19.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.220+0000 7fa44501d700 1 -- 172.21.15.177:0/538121296 shutdown_connections 2024-03-28T00:53:19.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.220+0000 7fa44501d700 1 --2- 172.21.15.177:0/538121296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa440102c60 0x7fa440109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:19.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.220+0000 7fa44501d700 1 --2- 172.21.15.177:0/538121296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4401010c0 0x7fa4401014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:19.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.220+0000 7fa44501d700 1 --2- 172.21.15.177:0/538121296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4401022c0 0x7fa440102720 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:19.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.220+0000 7fa44501d700 1 -- 172.21.15.177:0/538121296 >> 172.21.15.177:0/538121296 conn(0x7fa4400fc660 msgr2=0x7fa4400feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:53:19.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.220+0000 7fa44501d700 1 -- 172.21.15.177:0/538121296 shutdown_connections 2024-03-28T00:53:19.227 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.220+0000 7fa44501d700 1 -- 172.21.15.177:0/538121296 wait complete. 2024-03-28T00:53:19.227 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.221+0000 7fa44501d700 1 Processor -- start 2024-03-28T00:53:19.227 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.221+0000 7fa44501d700 1 -- start start 2024-03-28T00:53:19.227 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.221+0000 7fa44501d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4401010c0 0x7fa440198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:19.227 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.221+0000 7fa44501d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4401022c0 0x7fa440198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:19.227 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.221+0000 7fa44501d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa440102c60 0x7fa44019fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:19.227 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.221+0000 7fa44501d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44010cd20 con 0x7fa4401022c0 2024-03-28T00:53:19.227 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.221+0000 7fa44501d700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44010cba0 con 0x7fa440102c60 2024-03-28T00:53:19.227 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.221+0000 7fa44501d700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44010cea0 con 0x7fa4401010c0 2024-03-28T00:53:19.227 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.221+0000 7fa43ed9d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4401010c0 0x7fa440198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:19.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.222+0000 7fa43ed9d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4401010c0 0x7fa440198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:44692/0 (socket says 172.21.15.177:44692) 2024-03-28T00:53:19.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.222+0000 7fa43ed9d700 1 -- 172.21.15.177:0/3148344676 learned_addr learned my addr 172.21.15.177:0/3148344676 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-28T00:53:19.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.222+0000 7fa43e59c700 1 --2- 172.21.15.177:0/3148344676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4401022c0 0x7fa440198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:19.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.222+0000 7fa43f59e700 1 --2- 172.21.15.177:0/3148344676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa440102c60 0x7fa44019fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:19.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.222+0000 7fa43ed9d700 1 -- 172.21.15.177:0/3148344676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa440102c60 msgr2=0x7fa44019fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:19.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.222+0000 7fa43ed9d700 1 --2- 172.21.15.177:0/3148344676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa440102c60 0x7fa44019fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:19.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.222+0000 7fa43ed9d700 1 -- 172.21.15.177:0/3148344676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4401022c0 msgr2=0x7fa440198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:19.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.222+0000 7fa43ed9d700 1 --2- 172.21.15.177:0/3148344676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4401022c0 0x7fa440198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:19.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.222+0000 7fa43ed9d700 1 -- 172.21.15.177:0/3148344676 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4401a0110 con 0x7fa4401010c0 2024-03-28T00:53:19.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.222+0000 7fa43f59e700 1 --2- 172.21.15.177:0/3148344676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa440102c60 0x7fa44019fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:53:19.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.222+0000 7fa43ed9d700 1 --2- 172.21.15.177:0/3148344676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4401010c0 0x7fa440198500 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa43000b530 tx=0x7fa43000b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:53:19.229 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.222+0000 7fa427fff700 1 -- 172.21.15.177:0/3148344676 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa430015020 con 0x7fa4401010c0 2024-03-28T00:53:19.229 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.222+0000 7fa44501d700 1 -- 172.21.15.177:0/3148344676 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa440103540 con 0x7fa4401010c0 2024-03-28T00:53:19.229 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.222+0000 7fa427fff700 1 -- 172.21.15.177:0/3148344676 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa430011d10 con 0x7fa4401010c0 2024-03-28T00:53:19.229 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.222+0000 7fa427fff700 1 -- 172.21.15.177:0/3148344676 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43000f2e0 con 0x7fa4401010c0 2024-03-28T00:53:19.229 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.222+0000 7fa44501d700 1 -- 172.21.15.177:0/3148344676 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa440103b50 con 0x7fa4401010c0 2024-03-28T00:53:19.230 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.223+0000 7fa44501d700 1 -- 172.21.15.177:0/3148344676 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa440068e50 con 0x7fa4401010c0 2024-03-28T00:53:19.230 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.225+0000 7fa427fff700 1 -- 172.21.15.177:0/3148344676 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa430018020 con 0x7fa4401010c0 2024-03-28T00:53:19.230 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.226+0000 7fa427fff700 1 --2- 172.21.15.177:0/3148344676 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa4280a2890 0x7fa4280a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:19.230 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.226+0000 7fa427fff700 1 -- 172.21.15.177:0/3148344676 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6723+0+0 (secure 0 0 0) 0x7fa4300ce000 con 0x7fa4401010c0 2024-03-28T00:53:19.230 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.226+0000 7fa43e59c700 1 --2- 172.21.15.177:0/3148344676 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa4280a2890 0x7fa4280a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:19.230 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.226+0000 7fa43e59c700 1 --2- 172.21.15.177:0/3148344676 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa4280a2890 0x7fa4280a4d50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa43400b5c0 tx=0x7fa434003960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:53:19.232 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.229+0000 7fa427fff700 1 -- 172.21.15.177:0/3148344676 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa43001f020 con 0x7fa4401010c0 2024-03-28T00:53:19.445 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:19.442+0000 7fa44501d700 1 -- 172.21.15.177:0/3148344676 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4400fe870 con 0x7fa4280a2890 2024-03-28T00:53:19.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:19 smithi174 ceph-mon[36060]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T00:53:19.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:19 smithi177 ceph-mon[36206]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T00:53:19.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:19 smithi169 ceph-mon[32274]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T00:53:20.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:20 smithi174 ceph-mon[36060]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T00:53:20.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:20 smithi174 ceph-mon[36060]: from='client.24352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:53:20.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:20 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:53:20.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:20 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:53:20.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:20 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:20.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:20 smithi174 ceph-mon[36060]: pgmap v237: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T00:53:20.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:20 smithi177 ceph-mon[36206]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T00:53:20.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:20 smithi177 ceph-mon[36206]: from='client.24352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:53:20.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:20 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:53:20.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:20 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:53:20.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:20 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:20.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:20 smithi177 ceph-mon[36206]: pgmap v237: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T00:53:20.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:20 smithi169 ceph-mon[32274]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T00:53:20.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:20 smithi169 ceph-mon[32274]: from='client.24352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:53:20.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:20 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T00:53:20.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:20 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T00:53:20.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:20 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:20.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:20 smithi169 ceph-mon[32274]: pgmap v237: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T00:53:21.410 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:21 smithi177 ceph-mon[36206]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T00:53:21.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:21 smithi174 ceph-mon[36060]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T00:53:21.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:21 smithi169 ceph-mon[32274]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T00:53:22.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:22 smithi177 ceph-mon[36206]: pgmap v239: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-28T00:53:22.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:22 smithi177 ceph-mon[36206]: from='client.? 172.21.15.177:0/1678511419' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1e65122-4e70-4022-933a-c9bf2f0f2e41"}]: dispatch 2024-03-28T00:53:22.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:22 smithi177 ceph-mon[36206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1e65122-4e70-4022-933a-c9bf2f0f2e41"}]: dispatch 2024-03-28T00:53:22.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:22 smithi177 ceph-mon[36206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b1e65122-4e70-4022-933a-c9bf2f0f2e41"}]': finished 2024-03-28T00:53:22.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:22 smithi177 ceph-mon[36206]: osdmap e77: 12 total, 11 up, 12 in 2024-03-28T00:53:22.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:22 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T00:53:22.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:22 smithi169 ceph-mon[32274]: pgmap v239: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-28T00:53:22.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:22 smithi169 ceph-mon[32274]: from='client.? 172.21.15.177:0/1678511419' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1e65122-4e70-4022-933a-c9bf2f0f2e41"}]: dispatch 2024-03-28T00:53:22.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:22 smithi169 ceph-mon[32274]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1e65122-4e70-4022-933a-c9bf2f0f2e41"}]: dispatch 2024-03-28T00:53:22.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:22 smithi169 ceph-mon[32274]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b1e65122-4e70-4022-933a-c9bf2f0f2e41"}]': finished 2024-03-28T00:53:22.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:22 smithi169 ceph-mon[32274]: osdmap e77: 12 total, 11 up, 12 in 2024-03-28T00:53:22.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:22 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T00:53:22.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:22 smithi174 ceph-mon[36060]: pgmap v239: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-28T00:53:22.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:22 smithi174 ceph-mon[36060]: from='client.? 172.21.15.177:0/1678511419' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1e65122-4e70-4022-933a-c9bf2f0f2e41"}]: dispatch 2024-03-28T00:53:22.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:22 smithi174 ceph-mon[36060]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1e65122-4e70-4022-933a-c9bf2f0f2e41"}]: dispatch 2024-03-28T00:53:22.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:22 smithi174 ceph-mon[36060]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b1e65122-4e70-4022-933a-c9bf2f0f2e41"}]': finished 2024-03-28T00:53:22.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:22 smithi174 ceph-mon[36060]: osdmap e77: 12 total, 11 up, 12 in 2024-03-28T00:53:22.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:22 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T00:53:23.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:23 smithi177 ceph-mon[36206]: from='client.? 172.21.15.177:0/2720726468' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:53:23.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:23 smithi169 ceph-mon[32274]: from='client.? 172.21.15.177:0/2720726468' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:53:23.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:23 smithi174 ceph-mon[36060]: from='client.? 172.21.15.177:0/2720726468' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T00:53:24.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:24 smithi177 ceph-mon[36206]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-28T00:53:24.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:24 smithi169 ceph-mon[32274]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-28T00:53:24.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:24 smithi174 ceph-mon[36060]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-28T00:53:27.358 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:27 smithi177 ceph-mon[36206]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:27.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:27 smithi174 ceph-mon[36060]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:27.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:27 smithi169 ceph-mon[32274]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:29.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:29 smithi177 ceph-mon[36206]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:29.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:29 smithi169 ceph-mon[32274]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:29.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:29 smithi174 ceph-mon[36060]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:30.382 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:30 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T00:53:30.382 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:30 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:30.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:30 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T00:53:30.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:30 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:30.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:30 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T00:53:30.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:30 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:31.409 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:31 smithi177 ceph-mon[36206]: Deploying daemon osd.11 on smithi177 2024-03-28T00:53:31.409 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:31 smithi177 ceph-mon[36206]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:31.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:31 smithi169 ceph-mon[32274]: Deploying daemon osd.11 on smithi177 2024-03-28T00:53:31.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:31 smithi169 ceph-mon[32274]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:31.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:31 smithi174 ceph-mon[36060]: Deploying daemon osd.11 on smithi177 2024-03-28T00:53:31.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:31 smithi174 ceph-mon[36060]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:33.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:33 smithi177 ceph-mon[36206]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:33.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:33 smithi169 ceph-mon[32274]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:33.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:33 smithi174 ceph-mon[36060]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:35.194 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:35 smithi177 ceph-mon[36206]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:35.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:35 smithi169 ceph-mon[32274]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:35.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:35 smithi174 ceph-mon[36060]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:36.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:36.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:36.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:53:36.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:36 smithi177 ceph-mon[36206]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:37.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:37.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:37.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:53:37.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:36 smithi169 ceph-mon[32274]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:37.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:37.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:37.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:53:37.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:36 smithi174 ceph-mon[36060]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:39.211 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 11 on host 'smithi177' 2024-03-28T00:53:39.211 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:39.207+0000 7fa427fff700 1 -- 172.21.15.177:0/3148344676 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fa4400fe870 con 0x7fa4280a2890 2024-03-28T00:53:39.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:39.209+0000 7fa44501d700 1 -- 172.21.15.177:0/3148344676 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa4280a2890 msgr2=0x7fa4280a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:39.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:39.209+0000 7fa44501d700 1 --2- 172.21.15.177:0/3148344676 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa4280a2890 0x7fa4280a4d50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa43400b5c0 tx=0x7fa434003960 comp rx=0 tx=0).stop 2024-03-28T00:53:39.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:39.209+0000 7fa44501d700 1 -- 172.21.15.177:0/3148344676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4401010c0 msgr2=0x7fa440198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:39.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:39.209+0000 7fa44501d700 1 --2- 172.21.15.177:0/3148344676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4401010c0 0x7fa440198500 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa43000b530 tx=0x7fa43000b840 comp rx=0 tx=0).stop 2024-03-28T00:53:39.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:39.210+0000 7fa44501d700 1 -- 172.21.15.177:0/3148344676 shutdown_connections 2024-03-28T00:53:39.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:39.210+0000 7fa44501d700 1 --2- 172.21.15.177:0/3148344676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa440102c60 0x7fa44019fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:39.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:39.210+0000 7fa44501d700 1 --2- 172.21.15.177:0/3148344676 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa4280a2890 0x7fa4280a4d50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:39.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:39.210+0000 7fa44501d700 1 --2- 172.21.15.177:0/3148344676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4401010c0 0x7fa440198500 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:39.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:39.210+0000 7fa44501d700 1 --2- 172.21.15.177:0/3148344676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4401022c0 0x7fa440198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:39.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:39.210+0000 7fa44501d700 1 -- 172.21.15.177:0/3148344676 >> 172.21.15.177:0/3148344676 conn(0x7fa4400fc660 msgr2=0x7fa4400fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:53:39.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:39.210+0000 7fa44501d700 1 -- 172.21.15.177:0/3148344676 shutdown_connections 2024-03-28T00:53:39.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:39.210+0000 7fa44501d700 1 -- 172.21.15.177:0/3148344676 wait complete. 2024-03-28T00:53:39.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:39 smithi177 ceph-mon[36206]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:39.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:39 smithi169 ceph-mon[32274]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:39.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:39 smithi174 ceph-mon[36060]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:39.948 DEBUG:teuthology.orchestra.run.smithi177:osd.11> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.11.service 2024-03-28T00:53:39.951 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-28T00:53:39.951 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-28T00:53:40.209 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:53:40.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:40 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:40.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:40 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:40.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:40 smithi169 ceph-mon[32274]: from='osd.11 [v2:172.21.15.177:6824/2109004868,v1:172.21.15.177:6825/2109004868]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T00:53:40.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:40 smithi169 ceph-mon[32274]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T00:53:40.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:40 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:40.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:40 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:40.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:40 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:40.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:40 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:40.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:40 smithi174 ceph-mon[36060]: from='osd.11 [v2:172.21.15.177:6824/2109004868,v1:172.21.15.177:6825/2109004868]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T00:53:40.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:40 smithi174 ceph-mon[36060]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T00:53:40.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:40 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:40.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:40 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:40.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:40 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:40.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:40 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:40.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:40 smithi177 ceph-mon[36206]: from='osd.11 [v2:172.21.15.177:6824/2109004868,v1:172.21.15.177:6825/2109004868]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T00:53:40.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:40 smithi177 ceph-mon[36206]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T00:53:40.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:40 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:40.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:40 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:41.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:41 smithi177 ceph-mon[36206]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:41.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:41 smithi177 ceph-mon[36206]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T00:53:41.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:41 smithi177 ceph-mon[36206]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T00:53:41.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:41 smithi177 ceph-mon[36206]: from='osd.11 [v2:172.21.15.177:6824/2109004868,v1:172.21.15.177:6825/2109004868]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:53:41.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:41 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T00:53:41.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:41 smithi177 ceph-mon[36206]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:53:41.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:41 smithi169 ceph-mon[32274]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:41.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:41 smithi169 ceph-mon[32274]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T00:53:41.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:41 smithi169 ceph-mon[32274]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T00:53:41.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:41 smithi169 ceph-mon[32274]: from='osd.11 [v2:172.21.15.177:6824/2109004868,v1:172.21.15.177:6825/2109004868]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:53:41.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:41 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T00:53:41.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:41 smithi169 ceph-mon[32274]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:53:41.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:41 smithi174 ceph-mon[36060]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:41.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:41 smithi174 ceph-mon[36060]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T00:53:41.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:41 smithi174 ceph-mon[36060]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T00:53:41.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:41 smithi174 ceph-mon[36060]: from='osd.11 [v2:172.21.15.177:6824/2109004868,v1:172.21.15.177:6825/2109004868]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:53:41.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:41 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T00:53:41.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:41 smithi174 ceph-mon[36060]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-28T00:53:41.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.937+0000 7f2f77a8a700 1 -- 172.21.15.169:0/533970869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f70102c60 msgr2=0x7f2f70109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:41.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.937+0000 7f2f77a8a700 1 --2- 172.21.15.169:0/533970869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f70102c60 0x7f2f70109510 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800b0a0 tx=0x7f2f6800b3b0 comp rx=0 tx=0).stop 2024-03-28T00:53:41.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.937+0000 7f2f77a8a700 1 -- 172.21.15.169:0/533970869 shutdown_connections 2024-03-28T00:53:41.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.937+0000 7f2f77a8a700 1 --2- 172.21.15.169:0/533970869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f70102c60 0x7f2f70109510 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:41.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.937+0000 7f2f77a8a700 1 --2- 172.21.15.169:0/533970869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f701010c0 0x7f2f701014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:41.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.937+0000 7f2f77a8a700 1 --2- 172.21.15.169:0/533970869 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f701022c0 0x7f2f70102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:41.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.937+0000 7f2f77a8a700 1 -- 172.21.15.169:0/533970869 >> 172.21.15.169:0/533970869 conn(0x7f2f700fc680 msgr2=0x7f2f700feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:53:41.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.937+0000 7f2f77a8a700 1 -- 172.21.15.169:0/533970869 shutdown_connections 2024-03-28T00:53:41.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.937+0000 7f2f77a8a700 1 -- 172.21.15.169:0/533970869 wait complete. 2024-03-28T00:53:41.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.938+0000 7f2f77a8a700 1 Processor -- start 2024-03-28T00:53:41.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.938+0000 7f2f77a8a700 1 -- start start 2024-03-28T00:53:41.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.938+0000 7f2f77a8a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f701010c0 0x7f2f70105050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:41.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.938+0000 7f2f77a8a700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f701022c0 0x7f2f701036a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:41.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.938+0000 7f2f77a8a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f70102c60 0x7f2f70103be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:41.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.938+0000 7f2f77a8a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7010ccb0 con 0x7f2f70102c60 2024-03-28T00:53:41.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.938+0000 7f2f77a8a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7010cb30 con 0x7f2f701010c0 2024-03-28T00:53:41.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.938+0000 7f2f77a8a700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7010ce30 con 0x7f2f701022c0 2024-03-28T00:53:41.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.938+0000 7f2f76027700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f70102c60 0x7f2f70103be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:41.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.938+0000 7f2f76027700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f70102c60 0x7f2f70103be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39110/0 (socket says 172.21.15.169:39110) 2024-03-28T00:53:41.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.938+0000 7f2f75025700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f701022c0 0x7f2f701036a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:41.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.938+0000 7f2f76027700 1 -- 172.21.15.169:0/4931141 learned_addr learned my addr 172.21.15.169:0/4931141 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:53:41.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.938+0000 7f2f75826700 1 --2- 172.21.15.169:0/4931141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f701010c0 0x7f2f70105050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:41.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.938+0000 7f2f76027700 1 -- 172.21.15.169:0/4931141 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f701022c0 msgr2=0x7f2f701036a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:41.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.938+0000 7f2f76027700 1 --2- 172.21.15.169:0/4931141 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f701022c0 0x7f2f701036a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:41.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.938+0000 7f2f76027700 1 -- 172.21.15.169:0/4931141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f701010c0 msgr2=0x7f2f70105050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:41.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.938+0000 7f2f76027700 1 --2- 172.21.15.169:0/4931141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f701010c0 0x7f2f70105050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:41.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.938+0000 7f2f76027700 1 -- 172.21.15.169:0/4931141 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f70104490 con 0x7f2f70102c60 2024-03-28T00:53:41.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.938+0000 7f2f75025700 1 --2- 172.21.15.169:0/4931141 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f701022c0 0x7f2f701036a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:53:41.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.938+0000 7f2f76027700 1 --2- 172.21.15.169:0/4931141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f70102c60 0x7f2f70103be0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800b780 tx=0x7f2f680094f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:53:41.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.939+0000 7f2f66ffd700 1 -- 172.21.15.169:0/4931141 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68021070 con 0x7f2f70102c60 2024-03-28T00:53:41.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.939+0000 7f2f77a8a700 1 -- 172.21.15.169:0/4931141 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f701a2490 con 0x7f2f70102c60 2024-03-28T00:53:41.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.939+0000 7f2f66ffd700 1 -- 172.21.15.169:0/4931141 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f68007c40 con 0x7f2f70102c60 2024-03-28T00:53:41.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.939+0000 7f2f66ffd700 1 -- 172.21.15.169:0/4931141 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68005260 con 0x7f2f70102c60 2024-03-28T00:53:41.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.939+0000 7f2f77a8a700 1 -- 172.21.15.169:0/4931141 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f701a28f0 con 0x7f2f70102c60 2024-03-28T00:53:41.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.940+0000 7f2f77a8a700 1 -- 172.21.15.169:0/4931141 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f70068e50 con 0x7f2f70102c60 2024-03-28T00:53:41.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.941+0000 7f2f66ffd700 1 -- 172.21.15.169:0/4931141 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2f68024070 con 0x7f2f70102c60 2024-03-28T00:53:41.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.941+0000 7f2f66ffd700 1 --2- 172.21.15.169:0/4931141 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2f5c0a2890 0x7f2f5c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:41.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.941+0000 7f2f75826700 1 --2- 172.21.15.169:0/4931141 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2f5c0a2890 0x7f2f5c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:41.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.942+0000 7f2f66ffd700 1 -- 172.21.15.169:0/4931141 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6847+0+0 (secure 0 0 0) 0x7f2f680cf610 con 0x7f2f70102c60 2024-03-28T00:53:41.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.942+0000 7f2f75826700 1 --2- 172.21.15.169:0/4931141 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2f5c0a2890 0x7f2f5c0a4d50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009680 tx=0x7f2f6c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:53:41.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:41.945+0000 7f2f66ffd700 1 -- 172.21.15.169:0/4931141 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2f6809cd00 con 0x7f2f70102c60 2024-03-28T00:53:42.098 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 00:53:41 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T00:53:41.686+0000 7feb9c79a700 -1 osd.11 0 waiting for initial osdmap 2024-03-28T00:53:42.098 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 00:53:41 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T00:53:41.701+0000 7feb94585700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T00:53:42.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:42.154+0000 7f2f77a8a700 1 -- 172.21.15.169:0/4931141 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f2f701a2ce0 con 0x7f2f70102c60 2024-03-28T00:53:42.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:42.154+0000 7f2f66ffd700 1 -- 172.21.15.169:0/4931141 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) v1 ==== 74+0+133 (secure 0 0 0) 0x7f2f6809c600 con 0x7f2f70102c60 2024-03-28T00:53:42.158 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:53:42.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:42.157+0000 7f2f77a8a700 1 -- 172.21.15.169:0/4931141 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2f5c0a2890 msgr2=0x7f2f5c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:42.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:42.157+0000 7f2f77a8a700 1 --2- 172.21.15.169:0/4931141 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2f5c0a2890 0x7f2f5c0a4d50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009680 tx=0x7f2f6c009260 comp rx=0 tx=0).stop 2024-03-28T00:53:42.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:42.157+0000 7f2f77a8a700 1 -- 172.21.15.169:0/4931141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f70102c60 msgr2=0x7f2f70103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:42.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:42.157+0000 7f2f77a8a700 1 --2- 172.21.15.169:0/4931141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f70102c60 0x7f2f70103be0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800b780 tx=0x7f2f680094f0 comp rx=0 tx=0).stop 2024-03-28T00:53:42.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:42.157+0000 7f2f77a8a700 1 -- 172.21.15.169:0/4931141 shutdown_connections 2024-03-28T00:53:42.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:42.157+0000 7f2f77a8a700 1 --2- 172.21.15.169:0/4931141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f70102c60 0x7f2f70103be0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:42.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:42.157+0000 7f2f77a8a700 1 --2- 172.21.15.169:0/4931141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f701010c0 0x7f2f70105050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:42.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:42.157+0000 7f2f77a8a700 1 --2- 172.21.15.169:0/4931141 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2f5c0a2890 0x7f2f5c0a4d50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:42.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:42.157+0000 7f2f77a8a700 1 --2- 172.21.15.169:0/4931141 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f701022c0 0x7f2f701036a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:42.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:42.157+0000 7f2f77a8a700 1 -- 172.21.15.169:0/4931141 >> 172.21.15.169:0/4931141 conn(0x7f2f700fc680 msgr2=0x7f2f700fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:53:42.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:42.157+0000 7f2f77a8a700 1 -- 172.21.15.169:0/4931141 shutdown_connections 2024-03-28T00:53:42.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:42.157+0000 7f2f77a8a700 1 -- 172.21.15.169:0/4931141 wait complete. 2024-03-28T00:53:42.384 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:42 smithi177 ceph-mon[36206]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-28T00:53:42.384 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:42 smithi177 ceph-mon[36206]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T00:53:42.385 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:42 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T00:53:42.385 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:42 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T00:53:42.385 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:42 smithi177 ceph-mon[36206]: from='osd.11 ' entity='osd.11' 2024-03-28T00:53:42.385 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:42 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/4931141' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T00:53:42.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:42 smithi169 ceph-mon[32274]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-28T00:53:42.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:42 smithi169 ceph-mon[32274]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T00:53:42.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:42 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T00:53:42.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:42 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T00:53:42.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:42 smithi169 ceph-mon[32274]: from='osd.11 ' entity='osd.11' 2024-03-28T00:53:42.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:42 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4931141' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T00:53:42.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:42 smithi174 ceph-mon[36060]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-28T00:53:42.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:42 smithi174 ceph-mon[36060]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T00:53:42.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:42 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T00:53:42.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:42 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T00:53:42.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:42 smithi174 ceph-mon[36060]: from='osd.11 ' entity='osd.11' 2024-03-28T00:53:42.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:42 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/4931141' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T00:53:42.780 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1711587196,"num_in_osds":12,"osd_in_since":1711587202,"num_remapped_pgs":0} 2024-03-28T00:53:43.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:43 smithi177 ceph-mon[36206]: purged_snaps scrub starts 2024-03-28T00:53:43.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:43 smithi177 ceph-mon[36206]: purged_snaps scrub ok 2024-03-28T00:53:43.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:43 smithi177 ceph-mon[36206]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:43.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:43 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T00:53:43.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:43 smithi177 ceph-mon[36206]: osd.11 [v2:172.21.15.177:6824/2109004868,v1:172.21.15.177:6825/2109004868] boot 2024-03-28T00:53:43.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:43 smithi177 ceph-mon[36206]: osdmap e80: 12 total, 12 up, 12 in 2024-03-28T00:53:43.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:43 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T00:53:43.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:43 smithi169 ceph-mon[32274]: purged_snaps scrub starts 2024-03-28T00:53:43.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:43 smithi169 ceph-mon[32274]: purged_snaps scrub ok 2024-03-28T00:53:43.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:43 smithi169 ceph-mon[32274]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:43.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:43 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T00:53:43.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:43 smithi169 ceph-mon[32274]: osd.11 [v2:172.21.15.177:6824/2109004868,v1:172.21.15.177:6825/2109004868] boot 2024-03-28T00:53:43.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:43 smithi169 ceph-mon[32274]: osdmap e80: 12 total, 12 up, 12 in 2024-03-28T00:53:43.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:43 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T00:53:43.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:43 smithi174 ceph-mon[36060]: purged_snaps scrub starts 2024-03-28T00:53:43.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:43 smithi174 ceph-mon[36060]: purged_snaps scrub ok 2024-03-28T00:53:43.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:43 smithi174 ceph-mon[36060]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-28T00:53:43.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:43 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T00:53:43.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:43 smithi174 ceph-mon[36060]: osd.11 [v2:172.21.15.177:6824/2109004868,v1:172.21.15.177:6825/2109004868] boot 2024-03-28T00:53:43.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:43 smithi174 ceph-mon[36060]: osdmap e80: 12 total, 12 up, 12 in 2024-03-28T00:53:43.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:43 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T00:53:43.782 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-28T00:53:44.033 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:53:45.199 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:44 smithi169 ceph-mon[32274]: Detected new or changed devices on smithi177 2024-03-28T00:53:45.199 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:44 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:45.199 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:44 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:45.199 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:44 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:45.199 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:44 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:45.199 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:44 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:45.200 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:44 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:45.200 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:44 smithi169 ceph-mon[32274]: Adjusting osd_memory_target on smithi177 to 4270M 2024-03-28T00:53:45.200 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:44 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:45.200 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:44 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:45.200 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:44 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:53:45.200 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:44 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:45.200 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:44 smithi169 ceph-mon[32274]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:45.200 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:44 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:53:45.200 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:44 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:45.200 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:44 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:53:45.200 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:44 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:45.200 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:44 smithi169 ceph-mon[32274]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T00:53:45.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:44 smithi174 ceph-mon[36060]: Detected new or changed devices on smithi177 2024-03-28T00:53:45.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:44 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:45.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:44 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:44 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:44 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:44 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:44 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:44 smithi174 ceph-mon[36060]: Adjusting osd_memory_target on smithi177 to 4270M 2024-03-28T00:53:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:44 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:44 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:44 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:53:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:44 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:44 smithi174 ceph-mon[36060]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:44 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:53:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:44 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:44 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:53:45.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:44 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:45.360 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:44 smithi174 ceph-mon[36060]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T00:53:45.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:44 smithi177 ceph-mon[36206]: Detected new or changed devices on smithi177 2024-03-28T00:53:45.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:44 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:45.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:44 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:45.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:44 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:45.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:44 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:45.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:44 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:45.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:44 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T00:53:45.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:44 smithi177 ceph-mon[36206]: Adjusting osd_memory_target on smithi177 to 4270M 2024-03-28T00:53:45.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:44 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:45.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:44 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:45.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:44 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:53:45.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:44 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:45.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:44 smithi177 ceph-mon[36206]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:45.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:44 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:53:45.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:44 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:45.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:44 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:53:45.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:44 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:45.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:44 smithi177 ceph-mon[36206]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T00:53:45.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.799+0000 7f2a679ff700 1 -- 172.21.15.169:0/2552629565 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a60101d50 msgr2=0x7f2a60109610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:45.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.799+0000 7f2a679ff700 1 --2- 172.21.15.169:0/2552629565 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a60101d50 0x7f2a60109610 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2a5800b0d0 tx=0x7f2a5800b3e0 comp rx=0 tx=0).stop 2024-03-28T00:53:45.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.799+0000 7f2a679ff700 1 -- 172.21.15.169:0/2552629565 shutdown_connections 2024-03-28T00:53:45.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.799+0000 7f2a679ff700 1 --2- 172.21.15.169:0/2552629565 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a60101d50 0x7f2a60109610 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:45.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.799+0000 7f2a679ff700 1 --2- 172.21.15.169:0/2552629565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a60102670 0x7f2a60102a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:45.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.799+0000 7f2a679ff700 1 --2- 172.21.15.169:0/2552629565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a601013b0 0x7f2a60101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:45.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.799+0000 7f2a679ff700 1 -- 172.21.15.169:0/2552629565 >> 172.21.15.169:0/2552629565 conn(0x7f2a600fbde0 msgr2=0x7f2a600fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:53:45.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.800+0000 7f2a679ff700 1 -- 172.21.15.169:0/2552629565 shutdown_connections 2024-03-28T00:53:45.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.800+0000 7f2a679ff700 1 -- 172.21.15.169:0/2552629565 wait complete. 2024-03-28T00:53:45.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.800+0000 7f2a679ff700 1 Processor -- start 2024-03-28T00:53:45.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.800+0000 7f2a679ff700 1 -- start start 2024-03-28T00:53:45.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.800+0000 7f2a679ff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a601013b0 0x7f2a60105150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:45.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.800+0000 7f2a679ff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a60101d50 0x7f2a601037a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:45.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.800+0000 7f2a679ff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a60102670 0x7f2a60103ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:45.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.800+0000 7f2a679ff700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6010ccd0 con 0x7f2a60101d50 2024-03-28T00:53:45.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.800+0000 7f2a679ff700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6010cb50 con 0x7f2a601013b0 2024-03-28T00:53:45.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.800+0000 7f2a679ff700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6010ce50 con 0x7f2a60102670 2024-03-28T00:53:45.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.800+0000 7f2a64f9a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a60101d50 0x7f2a601037a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:45.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.800+0000 7f2a64f9a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a60101d50 0x7f2a601037a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39126/0 (socket says 172.21.15.169:39126) 2024-03-28T00:53:45.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.800+0000 7f2a64f9a700 1 -- 172.21.15.169:0/3376515776 learned_addr learned my addr 172.21.15.169:0/3376515776 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:53:45.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.800+0000 7f2a6579b700 1 --2- 172.21.15.169:0/3376515776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a601013b0 0x7f2a60105150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:45.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.801+0000 7f2a65f9c700 1 --2- 172.21.15.169:0/3376515776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a60102670 0x7f2a60103ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:45.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.801+0000 7f2a64f9a700 1 -- 172.21.15.169:0/3376515776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a60102670 msgr2=0x7f2a60103ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:45.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.801+0000 7f2a64f9a700 1 --2- 172.21.15.169:0/3376515776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a60102670 0x7f2a60103ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:45.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.801+0000 7f2a64f9a700 1 -- 172.21.15.169:0/3376515776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a601013b0 msgr2=0x7f2a60105150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:45.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.801+0000 7f2a64f9a700 1 --2- 172.21.15.169:0/3376515776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a601013b0 0x7f2a60105150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:45.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.801+0000 7f2a64f9a700 1 -- 172.21.15.169:0/3376515776 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a60104590 con 0x7f2a60101d50 2024-03-28T00:53:45.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.801+0000 7f2a64f9a700 1 --2- 172.21.15.169:0/3376515776 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a60101d50 0x7f2a601037a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000d950 tx=0x7f2a5000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:53:45.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.801+0000 7f2a567fc700 1 -- 172.21.15.169:0/3376515776 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a50016070 con 0x7f2a60101d50 2024-03-28T00:53:45.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.801+0000 7f2a567fc700 1 -- 172.21.15.169:0/3376515776 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a50009e20 con 0x7f2a60101d50 2024-03-28T00:53:45.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.801+0000 7f2a567fc700 1 -- 172.21.15.169:0/3376515776 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5000f620 con 0x7f2a60101d50 2024-03-28T00:53:45.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.801+0000 7f2a679ff700 1 -- 172.21.15.169:0/3376515776 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a601a2570 con 0x7f2a60101d50 2024-03-28T00:53:45.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.801+0000 7f2a679ff700 1 -- 172.21.15.169:0/3376515776 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a601a2ab0 con 0x7f2a60101d50 2024-03-28T00:53:45.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.802+0000 7f2a679ff700 1 -- 172.21.15.169:0/3376515776 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a60068e50 con 0x7f2a60101d50 2024-03-28T00:53:45.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.803+0000 7f2a567fc700 1 -- 172.21.15.169:0/3376515776 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2a50019070 con 0x7f2a60101d50 2024-03-28T00:53:45.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.804+0000 7f2a567fc700 1 --2- 172.21.15.169:0/3376515776 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2a4c0a2910 0x7f2a4c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:45.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.804+0000 7f2a6579b700 1 --2- 172.21.15.169:0/3376515776 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2a4c0a2910 0x7f2a4c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:45.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.804+0000 7f2a567fc700 1 -- 172.21.15.169:0/3376515776 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2a500ce100 con 0x7f2a60101d50 2024-03-28T00:53:45.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.804+0000 7f2a6579b700 1 --2- 172.21.15.169:0/3376515776 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2a4c0a2910 0x7f2a4c0a4dd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c005950 tx=0x7f2a5c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:53:45.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:45.808+0000 7f2a567fc700 1 -- 172.21.15.169:0/3376515776 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2a5009b640 con 0x7f2a60101d50 2024-03-28T00:53:46.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:46.002+0000 7f2a679ff700 1 -- 172.21.15.169:0/3376515776 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f2a601a2ed0 con 0x7f2a60101d50 2024-03-28T00:53:46.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:46.002+0000 7f2a567fc700 1 -- 172.21.15.169:0/3376515776 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) v1 ==== 74+0+133 (secure 0 0 0) 0x7f2a5009af40 con 0x7f2a60101d50 2024-03-28T00:53:46.006 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:53:46.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:46.005+0000 7f2a679ff700 1 -- 172.21.15.169:0/3376515776 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2a4c0a2910 msgr2=0x7f2a4c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:46.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:46.005+0000 7f2a679ff700 1 --2- 172.21.15.169:0/3376515776 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2a4c0a2910 0x7f2a4c0a4dd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c005950 tx=0x7f2a5c00a300 comp rx=0 tx=0).stop 2024-03-28T00:53:46.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:46.005+0000 7f2a679ff700 1 -- 172.21.15.169:0/3376515776 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a60101d50 msgr2=0x7f2a601037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:46.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:46.005+0000 7f2a679ff700 1 --2- 172.21.15.169:0/3376515776 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a60101d50 0x7f2a601037a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000d950 tx=0x7f2a5000dc60 comp rx=0 tx=0).stop 2024-03-28T00:53:46.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:46.005+0000 7f2a679ff700 1 -- 172.21.15.169:0/3376515776 shutdown_connections 2024-03-28T00:53:46.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:46.005+0000 7f2a679ff700 1 --2- 172.21.15.169:0/3376515776 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2a4c0a2910 0x7f2a4c0a4dd0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:46.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:46.005+0000 7f2a679ff700 1 --2- 172.21.15.169:0/3376515776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a60102670 0x7f2a60103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:46.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:46.005+0000 7f2a679ff700 1 --2- 172.21.15.169:0/3376515776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a601013b0 0x7f2a60105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:46.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:46.005+0000 7f2a679ff700 1 --2- 172.21.15.169:0/3376515776 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a60101d50 0x7f2a601037a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:46.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:46.005+0000 7f2a679ff700 1 -- 172.21.15.169:0/3376515776 >> 172.21.15.169:0/3376515776 conn(0x7f2a600fbde0 msgr2=0x7f2a600fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:53:46.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:46.005+0000 7f2a679ff700 1 -- 172.21.15.169:0/3376515776 shutdown_connections 2024-03-28T00:53:46.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:46.005+0000 7f2a679ff700 1 -- 172.21.15.169:0/3376515776 wait complete. 2024-03-28T00:53:46.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:46 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3376515776' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T00:53:46.340 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:46 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3376515776' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T00:53:46.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:46 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3376515776' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T00:53:46.760 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1711587222,"num_in_osds":12,"osd_in_since":1711587202,"num_remapped_pgs":0} 2024-03-28T00:53:46.761 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T00:53:46.995 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:53:47.335 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:47 smithi169 ceph-mon[32274]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:47.357 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:47 smithi174 ceph-mon[36060]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:47.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:47 smithi177 ceph-mon[36206]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.610+0000 7fad57798700 1 -- 172.21.15.169:0/1838586418 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad501070d0 msgr2=0x7fad501074b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:48.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.610+0000 7fad57798700 1 --2- 172.21.15.169:0/1838586418 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad501070d0 0x7fad501074b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fad48009990 tx=0x7fad48009ca0 comp rx=0 tx=0).stop 2024-03-28T00:53:48.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.610+0000 7fad57798700 1 -- 172.21.15.169:0/1838586418 shutdown_connections 2024-03-28T00:53:48.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.610+0000 7fad57798700 1 --2- 172.21.15.169:0/1838586418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad50101a30 0x7fad5010df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:48.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.610+0000 7fad57798700 1 --2- 172.21.15.169:0/1838586418 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad501070d0 0x7fad501074b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:48.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.610+0000 7fad57798700 1 --2- 172.21.15.169:0/1838586418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad50101090 0x7fad501014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:48.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.610+0000 7fad57798700 1 -- 172.21.15.169:0/1838586418 >> 172.21.15.169:0/1838586418 conn(0x7fad50077fc0 msgr2=0x7fad500783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:53:48.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.610+0000 7fad57798700 1 -- 172.21.15.169:0/1838586418 shutdown_connections 2024-03-28T00:53:48.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.610+0000 7fad57798700 1 -- 172.21.15.169:0/1838586418 wait complete. 2024-03-28T00:53:48.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.611+0000 7fad57798700 1 Processor -- start 2024-03-28T00:53:48.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.611+0000 7fad57798700 1 -- start start 2024-03-28T00:53:48.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.611+0000 7fad57798700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad50101090 0x7fad501a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:48.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.611+0000 7fad57798700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad50101a30 0x7fad501a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:48.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.611+0000 7fad57798700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad501070d0 0x7fad5019a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.611+0000 7fad57798700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad50111310 con 0x7fad50101090 2024-03-28T00:53:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.611+0000 7fad57798700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad50111190 con 0x7fad501070d0 2024-03-28T00:53:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.611+0000 7fad57798700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad50111490 con 0x7fad50101a30 2024-03-28T00:53:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.611+0000 7fad55534700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad50101090 0x7fad501a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.611+0000 7fad55534700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad50101090 0x7fad501a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38672/0 (socket says 172.21.15.169:38672) 2024-03-28T00:53:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.611+0000 7fad55534700 1 -- 172.21.15.169:0/2673813419 learned_addr learned my addr 172.21.15.169:0/2673813419 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:53:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.611+0000 7fad55d35700 1 --2- 172.21.15.169:0/2673813419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad501070d0 0x7fad5019a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.611+0000 7fad54d33700 1 --2- 172.21.15.169:0/2673813419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad50101a30 0x7fad501a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.612+0000 7fad55534700 1 -- 172.21.15.169:0/2673813419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad50101a30 msgr2=0x7fad501a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.612+0000 7fad55534700 1 --2- 172.21.15.169:0/2673813419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad50101a30 0x7fad501a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.612+0000 7fad55534700 1 -- 172.21.15.169:0/2673813419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad501070d0 msgr2=0x7fad5019a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.612+0000 7fad55534700 1 --2- 172.21.15.169:0/2673813419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad501070d0 0x7fad5019a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.612+0000 7fad55534700 1 -- 172.21.15.169:0/2673813419 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5019afe0 con 0x7fad50101090 2024-03-28T00:53:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.612+0000 7fad54d33700 1 --2- 172.21.15.169:0/2673813419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad50101a30 0x7fad501a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:53:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.612+0000 7fad55534700 1 --2- 172.21.15.169:0/2673813419 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad50101090 0x7fad501a06c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fad480038b0 tx=0x7fad48009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:53:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.612+0000 7fad4e7fc700 1 -- 172.21.15.169:0/2673813419 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad48027070 con 0x7fad50101090 2024-03-28T00:53:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.612+0000 7fad57798700 1 -- 172.21.15.169:0/2673813419 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad5019b270 con 0x7fad50101090 2024-03-28T00:53:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.612+0000 7fad4e7fc700 1 -- 172.21.15.169:0/2673813419 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad48004050 con 0x7fad50101090 2024-03-28T00:53:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.612+0000 7fad4e7fc700 1 -- 172.21.15.169:0/2673813419 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad48022cd0 con 0x7fad50101090 2024-03-28T00:53:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.612+0000 7fad57798700 1 -- 172.21.15.169:0/2673813419 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad501a7590 con 0x7fad50101090 2024-03-28T00:53:48.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.613+0000 7fad57798700 1 -- 172.21.15.169:0/2673813419 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad50068e50 con 0x7fad50101090 2024-03-28T00:53:48.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.614+0000 7fad4e7fc700 1 -- 172.21.15.169:0/2673813419 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fad4802a070 con 0x7fad50101090 2024-03-28T00:53:48.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.615+0000 7fad4e7fc700 1 --2- 172.21.15.169:0/2673813419 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fad380a2890 0x7fad380a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:48.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.615+0000 7fad54d33700 1 --2- 172.21.15.169:0/2673813419 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fad380a2890 0x7fad380a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:48.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.615+0000 7fad4e7fc700 1 -- 172.21.15.169:0/2673813419 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fad480cfc10 con 0x7fad50101090 2024-03-28T00:53:48.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.615+0000 7fad54d33700 1 --2- 172.21.15.169:0/2673813419 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fad380a2890 0x7fad380a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fad5006ae90 tx=0x7fad44009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:53:48.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.618+0000 7fad4e7fc700 1 -- 172.21.15.169:0/2673813419 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fad48031020 con 0x7fad50101090 2024-03-28T00:53:48.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.814+0000 7fad57798700 1 -- 172.21.15.169:0/2673813419 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad5019bdb0 con 0x7fad50101090 2024-03-28T00:53:48.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.815+0000 7fad4e7fc700 1 -- 172.21.15.169:0/2673813419 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18514 (secure 0 0 0) 0x7fad48013070 con 0x7fad50101090 2024-03-28T00:53:48.818 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:53:48.818 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":81,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:53:44.212143+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:53:48.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.817+0000 7fad57798700 1 -- 172.21.15.169:0/2673813419 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fad380a2890 msgr2=0x7fad380a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:48.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.817+0000 7fad57798700 1 --2- 172.21.15.169:0/2673813419 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fad380a2890 0x7fad380a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fad5006ae90 tx=0x7fad44009040 comp rx=0 tx=0).stop 2024-03-28T00:53:48.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.817+0000 7fad57798700 1 -- 172.21.15.169:0/2673813419 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad50101090 msgr2=0x7fad501a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:48.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.817+0000 7fad57798700 1 --2- 172.21.15.169:0/2673813419 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad50101090 0x7fad501a06c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fad480038b0 tx=0x7fad48009f50 comp rx=0 tx=0).stop 2024-03-28T00:53:48.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.817+0000 7fad57798700 1 -- 172.21.15.169:0/2673813419 shutdown_connections 2024-03-28T00:53:48.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.817+0000 7fad57798700 1 --2- 172.21.15.169:0/2673813419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad501070d0 0x7fad5019a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:48.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.817+0000 7fad57798700 1 --2- 172.21.15.169:0/2673813419 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad50101090 0x7fad501a06c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:48.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.817+0000 7fad57798700 1 --2- 172.21.15.169:0/2673813419 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fad380a2890 0x7fad380a4d50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:48.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.817+0000 7fad57798700 1 --2- 172.21.15.169:0/2673813419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad50101a30 0x7fad501a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:48.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.817+0000 7fad57798700 1 -- 172.21.15.169:0/2673813419 >> 172.21.15.169:0/2673813419 conn(0x7fad50077fc0 msgr2=0x7fad50069e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:53:48.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.817+0000 7fad57798700 1 -- 172.21.15.169:0/2673813419 shutdown_connections 2024-03-28T00:53:48.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:48.817+0000 7fad57798700 1 -- 172.21.15.169:0/2673813419 wait complete. 2024-03-28T00:53:49.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:49 smithi177 ceph-mon[36206]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:49.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:49 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2673813419' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:53:49.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:49 smithi169 ceph-mon[32274]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:49.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:49 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2673813419' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:53:49.582 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T00:49:56.339525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T00:53:49.582 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-28T00:53:49.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:49 smithi174 ceph-mon[36060]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:49.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:49 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2673813419' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:53:49.823 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:53:51.269 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:51 smithi169 ceph-mon[32274]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:51.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:51 smithi177 ceph-mon[36206]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:51.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.517+0000 7f38b1690700 1 -- 172.21.15.169:0/1611572663 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38ac102ba0 msgr2=0x7f38ac10f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:51.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.517+0000 7f38b1690700 1 --2- 172.21.15.169:0/1611572663 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38ac102ba0 0x7f38ac10f040 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f38a400b0a0 tx=0x7f38a400b3b0 comp rx=0 tx=0).stop 2024-03-28T00:53:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.517+0000 7f38b1690700 1 -- 172.21.15.169:0/1611572663 shutdown_connections 2024-03-28T00:53:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.517+0000 7f38b1690700 1 --2- 172.21.15.169:0/1611572663 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38ac102ba0 0x7f38ac10f040 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.517+0000 7f38b1690700 1 --2- 172.21.15.169:0/1611572663 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38ac108290 0x7f38ac108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.517+0000 7f38b1690700 1 --2- 172.21.15.169:0/1611572663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38ac1021e0 0x7f38ac102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.517+0000 7f38b1690700 1 -- 172.21.15.169:0/1611572663 >> 172.21.15.169:0/1611572663 conn(0x7f38ac0792c0 msgr2=0x7f38ac0796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:53:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.517+0000 7f38b1690700 1 -- 172.21.15.169:0/1611572663 shutdown_connections 2024-03-28T00:53:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.517+0000 7f38b1690700 1 -- 172.21.15.169:0/1611572663 wait complete. 2024-03-28T00:53:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.518+0000 7f38b1690700 1 Processor -- start 2024-03-28T00:53:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.518+0000 7f38b1690700 1 -- start start 2024-03-28T00:53:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.518+0000 7f38b1690700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38ac1021e0 0x7f38ac19e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.518+0000 7f38b1690700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38ac102ba0 0x7f38ac19e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.518+0000 7f38b1690700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38ac108290 0x7f38ac198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.518+0000 7f38b1690700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac1124a0 con 0x7f38ac102ba0 2024-03-28T00:53:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.518+0000 7f38b1690700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac112320 con 0x7f38ac1021e0 2024-03-28T00:53:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.518+0000 7f38b1690700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac112620 con 0x7f38ac108290 2024-03-28T00:53:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.518+0000 7f38aa7fc700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38ac102ba0 0x7f38ac19e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.518+0000 7f38aa7fc700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38ac102ba0 0x7f38ac19e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38678/0 (socket says 172.21.15.169:38678) 2024-03-28T00:53:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.518+0000 7f38aa7fc700 1 -- 172.21.15.169:0/1622664549 learned_addr learned my addr 172.21.15.169:0/1622664549 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:53:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.518+0000 7f38aaffd700 1 --2- 172.21.15.169:0/1622664549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38ac1021e0 0x7f38ac19e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.519+0000 7f38ab7fe700 1 --2- 172.21.15.169:0/1622664549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38ac108290 0x7f38ac198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.519+0000 7f38aa7fc700 1 -- 172.21.15.169:0/1622664549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38ac108290 msgr2=0x7f38ac198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.519+0000 7f38aa7fc700 1 --2- 172.21.15.169:0/1622664549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38ac108290 0x7f38ac198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.519+0000 7f38aa7fc700 1 -- 172.21.15.169:0/1622664549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38ac1021e0 msgr2=0x7f38ac19e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.519+0000 7f38aa7fc700 1 --2- 172.21.15.169:0/1622664549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38ac1021e0 0x7f38ac19e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.519+0000 7f38aa7fc700 1 -- 172.21.15.169:0/1622664549 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ac198d80 con 0x7f38ac102ba0 2024-03-28T00:53:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.519+0000 7f38aa7fc700 1 --2- 172.21.15.169:0/1622664549 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38ac102ba0 0x7f38ac19e9a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f38a000c9b0 tx=0x7f38a000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:53:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.519+0000 7f3893fff700 1 -- 172.21.15.169:0/1622664549 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a0007b00 con 0x7f38ac102ba0 2024-03-28T00:53:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.519+0000 7f3893fff700 1 -- 172.21.15.169:0/1622664549 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38a0007ca0 con 0x7f38ac102ba0 2024-03-28T00:53:51.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.519+0000 7f3893fff700 1 -- 172.21.15.169:0/1622664549 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a0018730 con 0x7f38ac102ba0 2024-03-28T00:53:51.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.519+0000 7f38b1690700 1 -- 172.21.15.169:0/1622664549 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38ac199070 con 0x7f38ac102ba0 2024-03-28T00:53:51.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.519+0000 7f38b1690700 1 -- 172.21.15.169:0/1622664549 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38ac0743c0 con 0x7f38ac102ba0 2024-03-28T00:53:51.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.520+0000 7f38b1690700 1 -- 172.21.15.169:0/1622664549 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38ac068e50 con 0x7f38ac102ba0 2024-03-28T00:53:51.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.522+0000 7f3893fff700 1 -- 172.21.15.169:0/1622664549 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f38a00188d0 con 0x7f38ac102ba0 2024-03-28T00:53:51.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.522+0000 7f3893fff700 1 --2- 172.21.15.169:0/1622664549 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f38940a2910 0x7f38940a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:51.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.522+0000 7f38aaffd700 1 --2- 172.21.15.169:0/1622664549 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f38940a2910 0x7f38940a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:51.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.522+0000 7f3893fff700 1 -- 172.21.15.169:0/1622664549 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f38a00ce7f0 con 0x7f38ac102ba0 2024-03-28T00:53:51.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.523+0000 7f38aaffd700 1 --2- 172.21.15.169:0/1622664549 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f38940a2910 0x7f38940a4dd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f389c005950 tx=0x7f389c010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:53:51.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.526+0000 7f3893fff700 1 -- 172.21.15.169:0/1622664549 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f38a0014030 con 0x7f38ac102ba0 2024-03-28T00:53:51.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:51 smithi174 ceph-mon[36060]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:51.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.722+0000 7f38b1690700 1 -- 172.21.15.169:0/1622664549 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f38ac199b50 con 0x7f38ac102ba0 2024-03-28T00:53:51.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.722+0000 7f3893fff700 1 -- 172.21.15.169:0/1622664549 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f38a009be40 con 0x7f38ac102ba0 2024-03-28T00:53:51.725 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 1 2024-03-28T00:53:51.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.724+0000 7f38b1690700 1 -- 172.21.15.169:0/1622664549 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f38940a2910 msgr2=0x7f38940a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:51.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.724+0000 7f38b1690700 1 --2- 172.21.15.169:0/1622664549 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f38940a2910 0x7f38940a4dd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f389c005950 tx=0x7f389c010040 comp rx=0 tx=0).stop 2024-03-28T00:53:51.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.724+0000 7f38b1690700 1 -- 172.21.15.169:0/1622664549 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38ac102ba0 msgr2=0x7f38ac19e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:51.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.724+0000 7f38b1690700 1 --2- 172.21.15.169:0/1622664549 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38ac102ba0 0x7f38ac19e9a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f38a000c9b0 tx=0x7f38a000ccc0 comp rx=0 tx=0).stop 2024-03-28T00:53:51.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.724+0000 7f38b1690700 1 -- 172.21.15.169:0/1622664549 shutdown_connections 2024-03-28T00:53:51.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.724+0000 7f38b1690700 1 --2- 172.21.15.169:0/1622664549 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f38940a2910 0x7f38940a4dd0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:51.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.724+0000 7f38b1690700 1 --2- 172.21.15.169:0/1622664549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38ac108290 0x7f38ac198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:51.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.724+0000 7f38b1690700 1 --2- 172.21.15.169:0/1622664549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38ac1021e0 0x7f38ac19e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:51.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.724+0000 7f38b1690700 1 --2- 172.21.15.169:0/1622664549 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38ac102ba0 0x7f38ac19e9a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:51.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.724+0000 7f38b1690700 1 -- 172.21.15.169:0/1622664549 >> 172.21.15.169:0/1622664549 conn(0x7f38ac0792c0 msgr2=0x7f38ac100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:53:51.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.724+0000 7f38b1690700 1 -- 172.21.15.169:0/1622664549 shutdown_connections 2024-03-28T00:53:51.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:51.724+0000 7f38b1690700 1 -- 172.21.15.169:0/1622664549 wait complete. 2024-03-28T00:53:52.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:52 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1622664549' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:53:52.379 INFO:tasks.cephadm:Adding mds.a on smithi169 2024-03-28T00:53:52.379 INFO:tasks.cephadm:Adding mds.d on smithi169 2024-03-28T00:53:52.379 INFO:tasks.cephadm:Adding mds.g on smithi169 2024-03-28T00:53:52.379 INFO:tasks.cephadm:Adding mds.j on smithi169 2024-03-28T00:53:52.379 INFO:tasks.cephadm:Adding mds.b on smithi174 2024-03-28T00:53:52.379 INFO:tasks.cephadm:Adding mds.e on smithi174 2024-03-28T00:53:52.380 INFO:tasks.cephadm:Adding mds.h on smithi174 2024-03-28T00:53:52.380 INFO:tasks.cephadm:Adding mds.k on smithi174 2024-03-28T00:53:52.380 INFO:tasks.cephadm:Adding mds.c on smithi177 2024-03-28T00:53:52.380 INFO:tasks.cephadm:Adding mds.f on smithi177 2024-03-28T00:53:52.380 INFO:tasks.cephadm:Adding mds.i on smithi177 2024-03-28T00:53:52.380 INFO:tasks.cephadm:Adding mds.l on smithi177 2024-03-28T00:53:52.380 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph orch apply mds all '12;smithi169=a;smithi169=d;smithi169=g;smithi169=j;smithi174=b;smithi174=e;smithi174=h;smithi174=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l' 2024-03-28T00:53:52.411 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:52 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1622664549' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:53:52.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:52 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1622664549' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:53:52.630 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T00:53:53.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:53 smithi177 ceph-mon[36206]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:53.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:53 smithi169 ceph-mon[32274]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:53.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:53 smithi174 ceph-mon[36060]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:54.340 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.335+0000 7ff53743a700 1 -- 172.21.15.177:0/3909606820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5301010c0 msgr2=0x7ff5301014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:54.340 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.335+0000 7ff53743a700 1 --2- 172.21.15.177:0/3909606820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5301010c0 0x7ff5301014e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff520009990 tx=0x7ff520009ca0 comp rx=0 tx=0).stop 2024-03-28T00:53:54.340 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.335+0000 7ff53743a700 1 -- 172.21.15.177:0/3909606820 shutdown_connections 2024-03-28T00:53:54.340 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.335+0000 7ff53743a700 1 --2- 172.21.15.177:0/3909606820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff530102c60 0x7ff530109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:54.340 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.335+0000 7ff53743a700 1 --2- 172.21.15.177:0/3909606820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5301010c0 0x7ff5301014e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:54.340 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.335+0000 7ff53743a700 1 --2- 172.21.15.177:0/3909606820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5301022c0 0x7ff530102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:54.340 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.335+0000 7ff53743a700 1 -- 172.21.15.177:0/3909606820 >> 172.21.15.177:0/3909606820 conn(0x7ff5300fc660 msgr2=0x7ff5300feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:53:54.340 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.335+0000 7ff53743a700 1 -- 172.21.15.177:0/3909606820 shutdown_connections 2024-03-28T00:53:54.340 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.335+0000 7ff53743a700 1 -- 172.21.15.177:0/3909606820 wait complete. 2024-03-28T00:53:54.341 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.336+0000 7ff53743a700 1 Processor -- start 2024-03-28T00:53:54.341 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.336+0000 7ff53743a700 1 -- start start 2024-03-28T00:53:54.341 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.336+0000 7ff53743a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5301010c0 0x7ff5301984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:54.341 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.336+0000 7ff53743a700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5301022c0 0x7ff530198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:54.341 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.336+0000 7ff53743a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff530102c60 0x7ff53019fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:54.341 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.336+0000 7ff53743a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53010cd10 con 0x7ff5301010c0 2024-03-28T00:53:54.341 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.336+0000 7ff53743a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53010cb90 con 0x7ff530102c60 2024-03-28T00:53:54.341 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.336+0000 7ff53743a700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53010ce90 con 0x7ff5301022c0 2024-03-28T00:53:54.341 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.336+0000 7ff5349d5700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5301022c0 0x7ff530198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:54.341 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.336+0000 7ff5349d5700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5301022c0 0x7ff530198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:34998/0 (socket says 172.21.15.177:34998) 2024-03-28T00:53:54.341 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.336+0000 7ff5349d5700 1 -- 172.21.15.177:0/3324131693 learned_addr learned my addr 172.21.15.177:0/3324131693 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-28T00:53:54.341 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.336+0000 7ff5359d7700 1 --2- 172.21.15.177:0/3324131693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff530102c60 0x7ff53019fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:54.341 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.336+0000 7ff5351d6700 1 --2- 172.21.15.177:0/3324131693 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5301010c0 0x7ff5301984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:54.341 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.336+0000 7ff5349d5700 1 -- 172.21.15.177:0/3324131693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff530102c60 msgr2=0x7ff53019fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:54.342 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.336+0000 7ff5349d5700 1 --2- 172.21.15.177:0/3324131693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff530102c60 0x7ff53019fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:54.342 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.336+0000 7ff5349d5700 1 -- 172.21.15.177:0/3324131693 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5301010c0 msgr2=0x7ff5301984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:54.342 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.336+0000 7ff5349d5700 1 --2- 172.21.15.177:0/3324131693 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5301010c0 0x7ff5301984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:54.342 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.336+0000 7ff5349d5700 1 -- 172.21.15.177:0/3324131693 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5301a0130 con 0x7ff5301022c0 2024-03-28T00:53:54.342 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.336+0000 7ff5349d5700 1 --2- 172.21.15.177:0/3324131693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5301022c0 0x7ff530198a10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff52c00d980 tx=0x7ff52c00dd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:53:54.342 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.336+0000 7ff5267fc700 1 -- 172.21.15.177:0/3324131693 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c016070 con 0x7ff5301022c0 2024-03-28T00:53:54.342 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.337+0000 7ff5267fc700 1 -- 172.21.15.177:0/3324131693 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff52c009b40 con 0x7ff5301022c0 2024-03-28T00:53:54.342 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.337+0000 7ff53743a700 1 -- 172.21.15.177:0/3324131693 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5301a0420 con 0x7ff5301022c0 2024-03-28T00:53:54.342 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.337+0000 7ff5267fc700 1 -- 172.21.15.177:0/3324131693 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c00f620 con 0x7ff5301022c0 2024-03-28T00:53:54.342 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.337+0000 7ff53743a700 1 -- 172.21.15.177:0/3324131693 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5301a09e0 con 0x7ff5301022c0 2024-03-28T00:53:54.343 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.337+0000 7ff53743a700 1 -- 172.21.15.177:0/3324131693 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff530068e50 con 0x7ff5301022c0 2024-03-28T00:53:54.343 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.339+0000 7ff5267fc700 1 -- 172.21.15.177:0/3324131693 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff52c019070 con 0x7ff5301022c0 2024-03-28T00:53:54.343 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.339+0000 7ff5267fc700 1 --2- 172.21.15.177:0/3324131693 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff51c0a2890 0x7ff51c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:54.343 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.339+0000 7ff5267fc700 1 -- 172.21.15.177:0/3324131693 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff52c0ce1b0 con 0x7ff5301022c0 2024-03-28T00:53:54.343 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.340+0000 7ff5351d6700 1 --2- 172.21.15.177:0/3324131693 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff51c0a2890 0x7ff51c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:54.343 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.340+0000 7ff5351d6700 1 --2- 172.21.15.177:0/3324131693 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff51c0a2890 0x7ff51c0a4d50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff52000b5c0 tx=0x7ff520005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:53:54.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.343+0000 7ff5267fc700 1 -- 172.21.15.177:0/3324131693 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff52c09b6f0 con 0x7ff5301022c0 2024-03-28T00:53:54.548 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.545+0000 7ff53743a700 1 -- 172.21.15.177:0/3324131693 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi169=a;smithi169=d;smithi169=g;smithi169=j;smithi174=b;smithi174=e;smithi174=h;smithi174=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l", "target": ["mon-mgr", ""]}) v1 -- 0x7ff53006a550 con 0x7ff51c0a2890 2024-03-28T00:53:54.556 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.553+0000 7ff5267fc700 1 -- 172.21.15.177:0/3324131693 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7ff53006a550 con 0x7ff51c0a2890 2024-03-28T00:53:54.556 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled mds.all update... 2024-03-28T00:53:54.558 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.555+0000 7ff53743a700 1 -- 172.21.15.177:0/3324131693 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff51c0a2890 msgr2=0x7ff51c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:54.558 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.555+0000 7ff53743a700 1 --2- 172.21.15.177:0/3324131693 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff51c0a2890 0x7ff51c0a4d50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff52000b5c0 tx=0x7ff520005fb0 comp rx=0 tx=0).stop 2024-03-28T00:53:54.559 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.555+0000 7ff53743a700 1 -- 172.21.15.177:0/3324131693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5301022c0 msgr2=0x7ff530198a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:54.559 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.555+0000 7ff53743a700 1 --2- 172.21.15.177:0/3324131693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5301022c0 0x7ff530198a10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff52c00d980 tx=0x7ff52c00dd40 comp rx=0 tx=0).stop 2024-03-28T00:53:54.559 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.555+0000 7ff53743a700 1 -- 172.21.15.177:0/3324131693 shutdown_connections 2024-03-28T00:53:54.559 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.555+0000 7ff53743a700 1 --2- 172.21.15.177:0/3324131693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff530102c60 0x7ff53019fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:54.559 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.555+0000 7ff53743a700 1 --2- 172.21.15.177:0/3324131693 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff5301010c0 0x7ff5301984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:54.559 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.555+0000 7ff53743a700 1 --2- 172.21.15.177:0/3324131693 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff51c0a2890 0x7ff51c0a4d50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:54.559 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.555+0000 7ff53743a700 1 --2- 172.21.15.177:0/3324131693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5301022c0 0x7ff530198a10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:54.559 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.555+0000 7ff53743a700 1 -- 172.21.15.177:0/3324131693 >> 172.21.15.177:0/3324131693 conn(0x7ff5300fc660 msgr2=0x7ff530069e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:53:54.559 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.555+0000 7ff53743a700 1 -- 172.21.15.177:0/3324131693 shutdown_connections 2024-03-28T00:53:54.559 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-28T00:53:54.555+0000 7ff53743a700 1 -- 172.21.15.177:0/3324131693 wait complete. 2024-03-28T00:53:55.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:55 smithi177 ceph-mon[36206]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:55.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:55 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:55.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:55 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:53:55.242 DEBUG:teuthology.orchestra.run.smithi169:mds.a> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.a.service 2024-03-28T00:53:55.245 DEBUG:teuthology.orchestra.run.smithi169:mds.d> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.d.service 2024-03-28T00:53:55.287 DEBUG:teuthology.orchestra.run.smithi169:mds.g> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.g.service 2024-03-28T00:53:55.330 DEBUG:teuthology.orchestra.run.smithi169:mds.j> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.j.service 2024-03-28T00:53:55.373 DEBUG:teuthology.orchestra.run.smithi174:mds.b> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.b.service 2024-03-28T00:53:55.376 DEBUG:teuthology.orchestra.run.smithi174:mds.e> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.e.service 2024-03-28T00:53:55.394 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:55 smithi169 ceph-mon[32274]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:55.395 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:55 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:55.395 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:55 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:53:55.399 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:55 smithi174 ceph-mon[36060]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:55.399 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:55 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:55.399 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:55 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:53:55.400 DEBUG:teuthology.orchestra.run.smithi174:mds.h> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.h.service 2024-03-28T00:53:55.443 DEBUG:teuthology.orchestra.run.smithi174:mds.k> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.k.service 2024-03-28T00:53:55.486 DEBUG:teuthology.orchestra.run.smithi177:mds.c> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.c.service 2024-03-28T00:53:55.488 DEBUG:teuthology.orchestra.run.smithi177:mds.f> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.f.service 2024-03-28T00:53:55.530 DEBUG:teuthology.orchestra.run.smithi177:mds.i> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.i.service 2024-03-28T00:53:55.574 DEBUG:teuthology.orchestra.run.smithi177:mds.l> sudo journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.l.service 2024-03-28T00:53:55.617 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-28T00:53:55.617 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T00:53:55.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.725+0000 7f12eaa19640 1 -- 172.21.15.169:0/1003198975 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e4102170 msgr2=0x7f12e4108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:55.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.725+0000 7f12eaa19640 1 --2- 172.21.15.169:0/1003198975 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e4102170 0x7f12e4108a00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f12dc009ca0 tx=0x7f12dc02c0f0 comp rx=0 tx=0).stop 2024-03-28T00:53:55.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.725+0000 7f12eaa19640 1 -- 172.21.15.169:0/1003198975 shutdown_connections 2024-03-28T00:53:55.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.725+0000 7f12eaa19640 1 --2- 172.21.15.169:0/1003198975 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e4102170 0x7f12e4108a00 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:55.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.725+0000 7f12eaa19640 1 --2- 172.21.15.169:0/1003198975 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e41017b0 0x7f12e4101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:55.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.725+0000 7f12eaa19640 1 --2- 172.21.15.169:0/1003198975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12e41005c0 0x7f12e41009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:55.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.725+0000 7f12eaa19640 1 -- 172.21.15.169:0/1003198975 >> 172.21.15.169:0/1003198975 conn(0x7f12e40fbb30 msgr2=0x7f12e40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:53:55.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.725+0000 7f12eaa19640 1 -- 172.21.15.169:0/1003198975 shutdown_connections 2024-03-28T00:53:55.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.725+0000 7f12eaa19640 1 -- 172.21.15.169:0/1003198975 wait complete. 2024-03-28T00:53:55.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.725+0000 7f12eaa19640 1 Processor -- start 2024-03-28T00:53:55.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.725+0000 7f12eaa19640 1 -- start start 2024-03-28T00:53:55.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.726+0000 7f12eaa19640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e41005c0 0x7f12e4196840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:55.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.726+0000 7f12eaa19640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e41017b0 0x7f12e4196d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:55.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.726+0000 7f12eaa19640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12e4102170 0x7f12e419ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:55.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.726+0000 7f12eaa19640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e410c220 con 0x7f12e41005c0 2024-03-28T00:53:55.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.726+0000 7f12eaa19640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e410c0a0 con 0x7f12e4102170 2024-03-28T00:53:55.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.726+0000 7f12eaa19640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e410c3a0 con 0x7f12e41017b0 2024-03-28T00:53:55.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.726+0000 7f12e3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e41005c0 0x7f12e4196840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:55.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.726+0000 7f12e3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e41005c0 0x7f12e4196840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38706/0 (socket says 172.21.15.169:38706) 2024-03-28T00:53:55.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.726+0000 7f12e3fff640 1 -- 172.21.15.169:0/2474175128 learned_addr learned my addr 172.21.15.169:0/2474175128 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:53:55.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.726+0000 7f12e37fe640 1 --2- 172.21.15.169:0/2474175128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e41017b0 0x7f12e4196d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:55.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.726+0000 7f12e3fff640 1 -- 172.21.15.169:0/2474175128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e41017b0 msgr2=0x7f12e4196d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:55.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.726+0000 7f12e3fff640 1 --2- 172.21.15.169:0/2474175128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e41017b0 0x7f12e4196d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:55.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.726+0000 7f12e3fff640 1 -- 172.21.15.169:0/2474175128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12e4102170 msgr2=0x7f12e419ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:55.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.726+0000 7f12e3fff640 1 --2- 172.21.15.169:0/2474175128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12e4102170 0x7f12e419ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:55.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.726+0000 7f12e3fff640 1 -- 172.21.15.169:0/2474175128 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e419e460 con 0x7f12e41005c0 2024-03-28T00:53:55.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.726+0000 7f12e37fe640 1 --2- 172.21.15.169:0/2474175128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e41017b0 0x7f12e4196d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:53:55.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.726+0000 7f12e3fff640 1 --2- 172.21.15.169:0/2474175128 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e41005c0 0x7f12e4196840 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f12d800e9c0 tx=0x7f12d800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:53:55.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.726+0000 7f12e17fa640 1 -- 172.21.15.169:0/2474175128 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d800cd80 con 0x7f12e41005c0 2024-03-28T00:53:55.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.727+0000 7f12e17fa640 1 -- 172.21.15.169:0/2474175128 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12d80035f0 con 0x7f12e41005c0 2024-03-28T00:53:55.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.727+0000 7f12eaa19640 1 -- 172.21.15.169:0/2474175128 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e419e750 con 0x7f12e41005c0 2024-03-28T00:53:55.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.727+0000 7f12e17fa640 1 -- 172.21.15.169:0/2474175128 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d8010690 con 0x7f12e41005c0 2024-03-28T00:53:55.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.727+0000 7f12eaa19640 1 -- 172.21.15.169:0/2474175128 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e419eca0 con 0x7f12e41005c0 2024-03-28T00:53:55.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.727+0000 7f12eaa19640 1 -- 172.21.15.169:0/2474175128 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e4107880 con 0x7f12e41005c0 2024-03-28T00:53:55.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.729+0000 7f12e17fa640 1 -- 172.21.15.169:0/2474175128 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f12d80037a0 con 0x7f12e41005c0 2024-03-28T00:53:55.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.729+0000 7f12e17fa640 1 --2- 172.21.15.169:0/2474175128 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f12cc0a2800 0x7f12cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:55.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.729+0000 7f12e37fe640 1 --2- 172.21.15.169:0/2474175128 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f12cc0a2800 0x7f12cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:55.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.729+0000 7f12e17fa640 1 -- 172.21.15.169:0/2474175128 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f12d80ceae0 con 0x7f12e41005c0 2024-03-28T00:53:55.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.730+0000 7f12e37fe640 1 --2- 172.21.15.169:0/2474175128 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f12cc0a2800 0x7f12cc0a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f12e4197d50 tx=0x7f12d400b5f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:53:55.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.734+0000 7f12e17fa640 1 -- 172.21.15.169:0/2474175128 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f12d809c020 con 0x7f12e41005c0 2024-03-28T00:53:55.914 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.911+0000 7f12eaa19640 1 -- 172.21.15.169:0/2474175128 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12e402d290 con 0x7f12e41005c0 2024-03-28T00:53:55.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.912+0000 7f12e17fa640 1 -- 172.21.15.169:0/2474175128 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18514 (secure 0 0 0) 0x7f12d809b920 con 0x7f12e41005c0 2024-03-28T00:53:55.915 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:53:55.915 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":81,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:53:44.212143+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:53:55.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.914+0000 7f12eaa19640 1 -- 172.21.15.169:0/2474175128 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f12cc0a2800 msgr2=0x7f12cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:55.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.914+0000 7f12eaa19640 1 --2- 172.21.15.169:0/2474175128 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f12cc0a2800 0x7f12cc0a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f12e4197d50 tx=0x7f12d400b5f0 comp rx=0 tx=0).stop 2024-03-28T00:53:55.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.914+0000 7f12eaa19640 1 -- 172.21.15.169:0/2474175128 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e41005c0 msgr2=0x7f12e4196840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:55.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.914+0000 7f12eaa19640 1 --2- 172.21.15.169:0/2474175128 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e41005c0 0x7f12e4196840 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f12d800e9c0 tx=0x7f12d800ee80 comp rx=0 tx=0).stop 2024-03-28T00:53:55.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.914+0000 7f12eaa19640 1 -- 172.21.15.169:0/2474175128 shutdown_connections 2024-03-28T00:53:55.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.914+0000 7f12eaa19640 1 --2- 172.21.15.169:0/2474175128 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f12cc0a2800 0x7f12cc0a4cc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:55.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.914+0000 7f12eaa19640 1 --2- 172.21.15.169:0/2474175128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12e4102170 0x7f12e419ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:55.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.914+0000 7f12eaa19640 1 --2- 172.21.15.169:0/2474175128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e41017b0 0x7f12e4196d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:55.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.914+0000 7f12eaa19640 1 --2- 172.21.15.169:0/2474175128 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e41005c0 0x7f12e4196840 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:55.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.914+0000 7f12eaa19640 1 -- 172.21.15.169:0/2474175128 >> 172.21.15.169:0/2474175128 conn(0x7f12e40fbb30 msgr2=0x7f12e4104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:53:55.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.915+0000 7f12eaa19640 1 -- 172.21.15.169:0/2474175128 shutdown_connections 2024-03-28T00:53:55.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:55.915+0000 7f12eaa19640 1 -- 172.21.15.169:0/2474175128 wait complete. 2024-03-28T00:53:55.929 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T00:49:56.339525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T00:53:55.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T00:53:56.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.032+0000 7f220eaea640 1 -- 172.21.15.169:0/2693328974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2208101830 msgr2=0x7f2208101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:56.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.032+0000 7f220eaea640 1 --2- 172.21.15.169:0/2693328974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2208101830 0x7f2208101cb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f21f4009900 tx=0x7f21f402bc30 comp rx=0 tx=0).stop 2024-03-28T00:53:56.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.033+0000 7f220eaea640 1 -- 172.21.15.169:0/2693328974 shutdown_connections 2024-03-28T00:53:56.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.033+0000 7f220eaea640 1 --2- 172.21.15.169:0/2693328974 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22081021f0 0x7f2208108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:56.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.033+0000 7f220eaea640 1 --2- 172.21.15.169:0/2693328974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2208101830 0x7f2208101cb0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:56.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.033+0000 7f220eaea640 1 --2- 172.21.15.169:0/2693328974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2208100640 0x7f2208100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:56.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.033+0000 7f220eaea640 1 -- 172.21.15.169:0/2693328974 >> 172.21.15.169:0/2693328974 conn(0x7f22080fbbb0 msgr2=0x7f22080fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:53:56.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.033+0000 7f220eaea640 1 -- 172.21.15.169:0/2693328974 shutdown_connections 2024-03-28T00:53:56.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.033+0000 7f220eaea640 1 -- 172.21.15.169:0/2693328974 wait complete. 2024-03-28T00:53:56.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.033+0000 7f220eaea640 1 Processor -- start 2024-03-28T00:53:56.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.033+0000 7f220eaea640 1 -- start start 2024-03-28T00:53:56.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.033+0000 7f220eaea640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2208100640 0x7f2208197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:56.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.033+0000 7f220eaea640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2208101830 0x7f2208198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:56.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.033+0000 7f220eaea640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22081021f0 0x7f220819f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:56.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.033+0000 7f220eaea640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220810d440 con 0x7f22081021f0 2024-03-28T00:53:56.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.033+0000 7f220eaea640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220810d2c0 con 0x7f2208100640 2024-03-28T00:53:56.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.033+0000 7f220eaea640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220810d5c0 con 0x7f2208101830 2024-03-28T00:53:56.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.033+0000 7f220d087640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22081021f0 0x7f220819f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:56.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.033+0000 7f220d087640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22081021f0 0x7f220819f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38734/0 (socket says 172.21.15.169:38734) 2024-03-28T00:53:56.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.033+0000 7f220d087640 1 -- 172.21.15.169:0/4052840699 learned_addr learned my addr 172.21.15.169:0/4052840699 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:53:56.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.033+0000 7f220c886640 1 --2- 172.21.15.169:0/4052840699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2208100640 0x7f2208197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:56.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.034+0000 7f2207fff640 1 --2- 172.21.15.169:0/4052840699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2208101830 0x7f2208198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:56.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.034+0000 7f220d087640 1 -- 172.21.15.169:0/4052840699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2208101830 msgr2=0x7f2208198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:56.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.034+0000 7f220d087640 1 --2- 172.21.15.169:0/4052840699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2208101830 0x7f2208198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:56.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.034+0000 7f220d087640 1 -- 172.21.15.169:0/4052840699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2208100640 msgr2=0x7f2208197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:56.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.034+0000 7f220d087640 1 --2- 172.21.15.169:0/4052840699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2208100640 0x7f2208197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:56.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.034+0000 7f220d087640 1 -- 172.21.15.169:0/4052840699 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220819f750 con 0x7f22081021f0 2024-03-28T00:53:56.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.034+0000 7f220d087640 1 --2- 172.21.15.169:0/4052840699 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22081021f0 0x7f220819f0e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f220000eb10 tx=0x7f220000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:53:56.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.034+0000 7f2205ffb640 1 -- 172.21.15.169:0/4052840699 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220000ce20 con 0x7f22081021f0 2024-03-28T00:53:56.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.034+0000 7f2205ffb640 1 -- 172.21.15.169:0/4052840699 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2200005020 con 0x7f22081021f0 2024-03-28T00:53:56.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.034+0000 7f220eaea640 1 -- 172.21.15.169:0/4052840699 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f220819fa40 con 0x7f22081021f0 2024-03-28T00:53:56.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.034+0000 7f2205ffb640 1 -- 172.21.15.169:0/4052840699 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2200010690 con 0x7f22081021f0 2024-03-28T00:53:56.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.034+0000 7f220eaea640 1 -- 172.21.15.169:0/4052840699 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f220819ff10 con 0x7f22081021f0 2024-03-28T00:53:56.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.035+0000 7f220eaea640 1 -- 172.21.15.169:0/4052840699 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2208107960 con 0x7f22081021f0 2024-03-28T00:53:56.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.036+0000 7f2205ffb640 1 -- 172.21.15.169:0/4052840699 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2200005540 con 0x7f22081021f0 2024-03-28T00:53:56.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.037+0000 7f2205ffb640 1 --2- 172.21.15.169:0/4052840699 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f21f80a2800 0x7f21f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:56.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.037+0000 7f2205ffb640 1 -- 172.21.15.169:0/4052840699 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f22000ceb50 con 0x7f22081021f0 2024-03-28T00:53:56.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.037+0000 7f220c886640 1 --2- 172.21.15.169:0/4052840699 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f21f80a2800 0x7f21f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:56.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.037+0000 7f220c886640 1 --2- 172.21.15.169:0/4052840699 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f21f80a2800 0x7f21f80a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2208101690 tx=0x7f21fc009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:53:56.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.042+0000 7f2205ffb640 1 -- 172.21.15.169:0/4052840699 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f220009c090 con 0x7f22081021f0 2024-03-28T00:53:56.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.247+0000 7f220eaea640 1 -- 172.21.15.169:0/4052840699 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f220802d290 con 0x7f22081021f0 2024-03-28T00:53:56.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.247+0000 7f2205ffb640 1 -- 172.21.15.169:0/4052840699 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f220009b990 con 0x7f22081021f0 2024-03-28T00:53:56.250 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 1 2024-03-28T00:53:56.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.249+0000 7f220eaea640 1 -- 172.21.15.169:0/4052840699 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f21f80a2800 msgr2=0x7f21f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:56.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.249+0000 7f220eaea640 1 --2- 172.21.15.169:0/4052840699 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f21f80a2800 0x7f21f80a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2208101690 tx=0x7f21fc009110 comp rx=0 tx=0).stop 2024-03-28T00:53:56.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.249+0000 7f220eaea640 1 -- 172.21.15.169:0/4052840699 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22081021f0 msgr2=0x7f220819f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:56.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.249+0000 7f220eaea640 1 --2- 172.21.15.169:0/4052840699 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22081021f0 0x7f220819f0e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f220000eb10 tx=0x7f220000ef10 comp rx=0 tx=0).stop 2024-03-28T00:53:56.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.249+0000 7f220eaea640 1 -- 172.21.15.169:0/4052840699 shutdown_connections 2024-03-28T00:53:56.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.249+0000 7f220eaea640 1 --2- 172.21.15.169:0/4052840699 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f21f80a2800 0x7f21f80a4cc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:56.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.249+0000 7f220eaea640 1 --2- 172.21.15.169:0/4052840699 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22081021f0 0x7f220819f0e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:56.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.249+0000 7f220eaea640 1 --2- 172.21.15.169:0/4052840699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2208101830 0x7f2208198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:56.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.249+0000 7f220eaea640 1 --2- 172.21.15.169:0/4052840699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2208100640 0x7f2208197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:56.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.249+0000 7f220eaea640 1 -- 172.21.15.169:0/4052840699 >> 172.21.15.169:0/4052840699 conn(0x7f22080fbbb0 msgr2=0x7f2208104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:53:56.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.249+0000 7f220eaea640 1 -- 172.21.15.169:0/4052840699 shutdown_connections 2024-03-28T00:53:56.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.249+0000 7f220eaea640 1 -- 172.21.15.169:0/4052840699 wait complete. 2024-03-28T00:53:56.262 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:56 smithi169 ceph-mon[32274]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi169=a;smithi169=d;smithi169=g;smithi169=j;smithi174=b;smithi174=e;smithi174=h;smithi174=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:53:56.262 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:56 smithi169 ceph-mon[32274]: Saving service mds.all spec with placement smithi169=a;smithi169=d;smithi169=g;smithi169=j;smithi174=b;smithi174=e;smithi174=h;smithi174=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l;count:12 2024-03-28T00:53:56.262 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:56.262 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:53:56.263 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:56.263 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:53:56.263 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:53:56.263 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:56 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:56.263 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:56 smithi169 ceph-mon[32274]: Deploying daemon mds.l on smithi177 2024-03-28T00:53:56.263 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:56 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2474175128' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:53:56.264 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:53:56.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.363+0000 7fde4300b640 1 -- 172.21.15.169:0/4012165484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde3c1073a0 msgr2=0x7fde3c1097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:56.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.363+0000 7fde4300b640 1 --2- 172.21.15.169:0/4012165484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde3c1073a0 0x7fde3c1097f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009900 tx=0x7fde2c02bc30 comp rx=0 tx=0).stop 2024-03-28T00:53:56.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.363+0000 7fde4300b640 1 -- 172.21.15.169:0/4012165484 shutdown_connections 2024-03-28T00:53:56.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.363+0000 7fde4300b640 1 --2- 172.21.15.169:0/4012165484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde3c1073a0 0x7fde3c1097f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:56.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.363+0000 7fde4300b640 1 --2- 172.21.15.169:0/4012165484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde3c104a70 0x7fde3c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:56.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.363+0000 7fde4300b640 1 --2- 172.21.15.169:0/4012165484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde3c102140 0x7fde3c104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:56.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.363+0000 7fde4300b640 1 -- 172.21.15.169:0/4012165484 >> 172.21.15.169:0/4012165484 conn(0x7fde3c0fbb30 msgr2=0x7fde3c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:53:56.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.363+0000 7fde4300b640 1 -- 172.21.15.169:0/4012165484 shutdown_connections 2024-03-28T00:53:56.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.363+0000 7fde4300b640 1 -- 172.21.15.169:0/4012165484 wait complete. 2024-03-28T00:53:56.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.364+0000 7fde4300b640 1 Processor -- start 2024-03-28T00:53:56.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.364+0000 7fde4300b640 1 -- start start 2024-03-28T00:53:56.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.364+0000 7fde4300b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde3c102140 0x7fde3c197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:56.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.364+0000 7fde4300b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde3c104a70 0x7fde3c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:56.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.364+0000 7fde4300b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde3c1073a0 0x7fde3c19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:56.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.364+0000 7fde4300b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3c10d3b0 con 0x7fde3c104a70 2024-03-28T00:53:56.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.364+0000 7fde4300b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3c10d230 con 0x7fde3c102140 2024-03-28T00:53:56.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.364+0000 7fde4300b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3c10d530 con 0x7fde3c1073a0 2024-03-28T00:53:56.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.364+0000 7fde3bfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde3c104a70 0x7fde3c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:56.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.364+0000 7fde3bfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde3c104a70 0x7fde3c198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38738/0 (socket says 172.21.15.169:38738) 2024-03-28T00:53:56.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.364+0000 7fde3bfff640 1 -- 172.21.15.169:0/317669271 learned_addr learned my addr 172.21.15.169:0/317669271 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:53:56.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.364+0000 7fde415a8640 1 --2- 172.21.15.169:0/317669271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde3c1073a0 0x7fde3c19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:56.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.364+0000 7fde40da7640 1 --2- 172.21.15.169:0/317669271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde3c102140 0x7fde3c197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:56.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.364+0000 7fde3bfff640 1 -- 172.21.15.169:0/317669271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde3c1073a0 msgr2=0x7fde3c19ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:56.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.364+0000 7fde3bfff640 1 --2- 172.21.15.169:0/317669271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde3c1073a0 0x7fde3c19ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:56.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.364+0000 7fde3bfff640 1 -- 172.21.15.169:0/317669271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde3c102140 msgr2=0x7fde3c197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:56.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.364+0000 7fde3bfff640 1 --2- 172.21.15.169:0/317669271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde3c102140 0x7fde3c197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:56.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.364+0000 7fde3bfff640 1 -- 172.21.15.169:0/317669271 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde3c19f5d0 con 0x7fde3c104a70 2024-03-28T00:53:56.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.365+0000 7fde40da7640 1 --2- 172.21.15.169:0/317669271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde3c102140 0x7fde3c197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:53:56.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.365+0000 7fde3bfff640 1 --2- 172.21.15.169:0/317669271 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde3c104a70 0x7fde3c198000 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fde3000d770 tx=0x7fde3000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:53:56.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.365+0000 7fde39ffb640 1 -- 172.21.15.169:0/317669271 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde30017070 con 0x7fde3c104a70 2024-03-28T00:53:56.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.365+0000 7fde39ffb640 1 -- 172.21.15.169:0/317669271 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde30009c30 con 0x7fde3c104a70 2024-03-28T00:53:56.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.365+0000 7fde4300b640 1 -- 172.21.15.169:0/317669271 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde3c19f8c0 con 0x7fde3c104a70 2024-03-28T00:53:56.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.365+0000 7fde39ffb640 1 -- 172.21.15.169:0/317669271 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3000f620 con 0x7fde3c104a70 2024-03-28T00:53:56.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.365+0000 7fde4300b640 1 -- 172.21.15.169:0/317669271 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde3c19fe60 con 0x7fde3c104a70 2024-03-28T00:53:56.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.366+0000 7fde4300b640 1 -- 172.21.15.169:0/317669271 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde3c0fd620 con 0x7fde3c104a70 2024-03-28T00:53:56.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.367+0000 7fde39ffb640 1 -- 172.21.15.169:0/317669271 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fde3001a070 con 0x7fde3c104a70 2024-03-28T00:53:56.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.368+0000 7fde39ffb640 1 --2- 172.21.15.169:0/317669271 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fde240a2880 0x7fde240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:56.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.368+0000 7fde40da7640 1 --2- 172.21.15.169:0/317669271 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fde240a2880 0x7fde240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:56.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.368+0000 7fde39ffb640 1 -- 172.21.15.169:0/317669271 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fde300ce010 con 0x7fde3c104a70 2024-03-28T00:53:56.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.368+0000 7fde40da7640 1 --2- 172.21.15.169:0/317669271 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fde240a2880 0x7fde240a4d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fde34005fd0 tx=0x7fde34009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:53:56.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.373+0000 7fde39ffb640 1 -- 172.21.15.169:0/317669271 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fde30009de0 con 0x7fde3c104a70 2024-03-28T00:53:56.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:56 smithi177 ceph-mon[36206]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi169=a;smithi169=d;smithi169=g;smithi169=j;smithi174=b;smithi174=e;smithi174=h;smithi174=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:53:56.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:56 smithi177 ceph-mon[36206]: Saving service mds.all spec with placement smithi169=a;smithi169=d;smithi169=g;smithi169=j;smithi174=b;smithi174=e;smithi174=h;smithi174=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l;count:12 2024-03-28T00:53:56.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:56.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:53:56.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:56.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:53:56.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:53:56.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:56 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:56.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:56 smithi177 ceph-mon[36206]: Deploying daemon mds.l on smithi177 2024-03-28T00:53:56.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:56 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2474175128' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:53:56.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.584+0000 7fde4300b640 1 -- 172.21.15.169:0/317669271 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde3c02d250 con 0x7fde3c104a70 2024-03-28T00:53:56.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.585+0000 7fde39ffb640 1 -- 172.21.15.169:0/317669271 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7fde3009b550 con 0x7fde3c104a70 2024-03-28T00:53:56.588 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:53:56.588 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-28T00:53:56.588 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 1 2024-03-28T00:53:56.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.586+0000 7fde4300b640 1 -- 172.21.15.169:0/317669271 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fde240a2880 msgr2=0x7fde240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:56.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.587+0000 7fde4300b640 1 --2- 172.21.15.169:0/317669271 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fde240a2880 0x7fde240a4d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fde34005fd0 tx=0x7fde34009500 comp rx=0 tx=0).stop 2024-03-28T00:53:56.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.587+0000 7fde4300b640 1 -- 172.21.15.169:0/317669271 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde3c104a70 msgr2=0x7fde3c198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:56.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.587+0000 7fde4300b640 1 --2- 172.21.15.169:0/317669271 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde3c104a70 0x7fde3c198000 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fde3000d770 tx=0x7fde3000dc30 comp rx=0 tx=0).stop 2024-03-28T00:53:56.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.587+0000 7fde4300b640 1 -- 172.21.15.169:0/317669271 shutdown_connections 2024-03-28T00:53:56.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.587+0000 7fde4300b640 1 --2- 172.21.15.169:0/317669271 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fde240a2880 0x7fde240a4d40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:56.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.587+0000 7fde4300b640 1 --2- 172.21.15.169:0/317669271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde3c1073a0 0x7fde3c19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:56.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.587+0000 7fde4300b640 1 --2- 172.21.15.169:0/317669271 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde3c104a70 0x7fde3c198000 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:56.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.587+0000 7fde4300b640 1 --2- 172.21.15.169:0/317669271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde3c102140 0x7fde3c197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:56.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.587+0000 7fde4300b640 1 -- 172.21.15.169:0/317669271 >> 172.21.15.169:0/317669271 conn(0x7fde3c0fbb30 msgr2=0x7fde3c102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:53:56.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.587+0000 7fde4300b640 1 -- 172.21.15.169:0/317669271 shutdown_connections 2024-03-28T00:53:56.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:56.587+0000 7fde4300b640 1 -- 172.21.15.169:0/317669271 wait complete. 2024-03-28T00:53:56.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:56 smithi174 ceph-mon[36060]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi169=a;smithi169=d;smithi169=g;smithi169=j;smithi174=b;smithi174=e;smithi174=h;smithi174=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:53:56.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:56 smithi174 ceph-mon[36060]: Saving service mds.all spec with placement smithi169=a;smithi169=d;smithi169=g;smithi169=j;smithi174=b;smithi174=e;smithi174=h;smithi174=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l;count:12 2024-03-28T00:53:56.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:56.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:53:56.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:56.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:53:56.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:53:56.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:56 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:56.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:56 smithi174 ceph-mon[36060]: Deploying daemon mds.l on smithi177 2024-03-28T00:53:56.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:56 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2474175128' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:53:57.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:57 smithi174 ceph-mon[36060]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:57.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:57 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/4052840699' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:53:57.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:57 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/317669271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:53:57.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:57 smithi177 ceph-mon[36206]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:57.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:57 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/4052840699' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:53:57.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:57 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/317669271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:53:57.417 INFO:journalctl@ceph.mds.l.smithi177.stdout:Mar 28 00:53:57 smithi177 systemd[1]: Starting Ceph mds.l for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T00:53:57.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:57 smithi169 ceph-mon[32274]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:57.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:57 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4052840699' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:53:57.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:57 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/317669271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:53:57.687 INFO:journalctl@ceph.mds.l.smithi177.stdout:Mar 28 00:53:57 smithi177 podman[66718]: 2024-03-28 00:53:57.336507542 +0000 UTC m=+0.018039798 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T00:53:57.687 INFO:journalctl@ceph.mds.l.smithi177.stdout:Mar 28 00:53:57 smithi177 podman[66718]: 2024-03-28 00:53:57.465325828 +0000 UTC m=+0.146858111 container create eda74141a6aaefbf1db0184b68eaffc66f71f05660683d9241d950aaa61b8de3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-l, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T00:53:57.687 INFO:journalctl@ceph.mds.l.smithi177.stdout:Mar 28 00:53:57 smithi177 podman[66718]: 2024-03-28 00:53:57.68525928 +0000 UTC m=+0.366791513 container init eda74141a6aaefbf1db0184b68eaffc66f71f05660683d9241d950aaa61b8de3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-l, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T00:53:58.167 INFO:journalctl@ceph.mds.l.smithi177.stdout:Mar 28 00:53:57 smithi177 podman[66718]: 2024-03-28 00:53:57.687891862 +0000 UTC m=+0.369424104 container start eda74141a6aaefbf1db0184b68eaffc66f71f05660683d9241d950aaa61b8de3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-l, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-03-28T00:53:58.167 INFO:journalctl@ceph.mds.l.smithi177.stdout:Mar 28 00:53:57 smithi177 ceph-mds[66736]: -- 172.21.15.177:0/3597165465 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5579b4501a40 con 0x5579b44d5000 2024-03-28T00:53:58.167 INFO:journalctl@ceph.mds.l.smithi177.stdout:Mar 28 00:53:57 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-l[66732]: starting mds.l at 2024-03-28T00:53:58.167 INFO:journalctl@ceph.mds.l.smithi177.stdout:Mar 28 00:53:57 smithi177 bash[66718]: eda74141a6aaefbf1db0184b68eaffc66f71f05660683d9241d950aaa61b8de3 2024-03-28T00:53:58.167 INFO:journalctl@ceph.mds.l.smithi177.stdout:Mar 28 00:53:57 smithi177 systemd[1]: Started Ceph mds.l for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T00:53:58.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:53:58.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.712+0000 7feb67812640 1 -- 172.21.15.169:0/1086042689 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb600ff8f0 msgr2=0x7feb600ffd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:58.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.712+0000 7feb67812640 1 --2- 172.21.15.169:0/1086042689 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb600ff8f0 0x7feb600ffd10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7feb5c00b3f0 tx=0x7feb5c02c4d0 comp rx=0 tx=0).stop 2024-03-28T00:53:58.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.712+0000 7feb67812640 1 -- 172.21.15.169:0/1086042689 shutdown_connections 2024-03-28T00:53:58.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.712+0000 7feb67812640 1 --2- 172.21.15.169:0/1086042689 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb601014a0 0x7feb60107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:58.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.712+0000 7feb67812640 1 --2- 172.21.15.169:0/1086042689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb60100ae0 0x7feb60100f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:58.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.712+0000 7feb67812640 1 --2- 172.21.15.169:0/1086042689 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb600ff8f0 0x7feb600ffd10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:58.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.712+0000 7feb67812640 1 -- 172.21.15.169:0/1086042689 >> 172.21.15.169:0/1086042689 conn(0x7feb600faea0 msgr2=0x7feb600fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:53:58.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.713+0000 7feb67812640 1 -- 172.21.15.169:0/1086042689 shutdown_connections 2024-03-28T00:53:58.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.713+0000 7feb67812640 1 -- 172.21.15.169:0/1086042689 wait complete. 2024-03-28T00:53:58.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.713+0000 7feb67812640 1 Processor -- start 2024-03-28T00:53:58.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.713+0000 7feb67812640 1 -- start start 2024-03-28T00:53:58.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.713+0000 7feb67812640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb600ff8f0 0x7feb600fcfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:58.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.713+0000 7feb67812640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb60100ae0 0x7feb6019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:58.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.713+0000 7feb67812640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb601014a0 0x7feb6019ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:58.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.713+0000 7feb67812640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6010b140 con 0x7feb60100ae0 2024-03-28T00:53:58.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.713+0000 7feb67812640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6010afc0 con 0x7feb601014a0 2024-03-28T00:53:58.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.713+0000 7feb67812640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6010b2c0 con 0x7feb600ff8f0 2024-03-28T00:53:58.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.713+0000 7feb64dad640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb60100ae0 0x7feb6019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:58.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.713+0000 7feb64dad640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb60100ae0 0x7feb6019c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40514/0 (socket says 172.21.15.169:40514) 2024-03-28T00:53:58.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.713+0000 7feb64dad640 1 -- 172.21.15.169:0/3067024602 learned_addr learned my addr 172.21.15.169:0/3067024602 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:53:58.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.714+0000 7feb65daf640 1 --2- 172.21.15.169:0/3067024602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb601014a0 0x7feb6019ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:58.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.714+0000 7feb64dad640 1 -- 172.21.15.169:0/3067024602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb600ff8f0 msgr2=0x7feb600fcfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:58.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.714+0000 7feb64dad640 1 --2- 172.21.15.169:0/3067024602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb600ff8f0 0x7feb600fcfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:58.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.714+0000 7feb64dad640 1 -- 172.21.15.169:0/3067024602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb601014a0 msgr2=0x7feb6019ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:58.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.714+0000 7feb64dad640 1 --2- 172.21.15.169:0/3067024602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb601014a0 0x7feb6019ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:58.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.714+0000 7feb64dad640 1 -- 172.21.15.169:0/3067024602 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6019f620 con 0x7feb60100ae0 2024-03-28T00:53:58.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.714+0000 7feb65daf640 1 --2- 172.21.15.169:0/3067024602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb601014a0 0x7feb6019ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:53:58.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.714+0000 7feb64dad640 1 --2- 172.21.15.169:0/3067024602 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb60100ae0 0x7feb6019c6f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7feb5000c920 tx=0x7feb5000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:53:58.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.714+0000 7feb567fc640 1 -- 172.21.15.169:0/3067024602 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb50007c70 con 0x7feb60100ae0 2024-03-28T00:53:58.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.714+0000 7feb67812640 1 -- 172.21.15.169:0/3067024602 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb6019f8b0 con 0x7feb60100ae0 2024-03-28T00:53:58.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.714+0000 7feb567fc640 1 -- 172.21.15.169:0/3067024602 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feb50007e10 con 0x7feb60100ae0 2024-03-28T00:53:58.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.714+0000 7feb567fc640 1 -- 172.21.15.169:0/3067024602 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb50018890 con 0x7feb60100ae0 2024-03-28T00:53:58.720 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.714+0000 7feb67812640 1 -- 172.21.15.169:0/3067024602 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb6019fe00 con 0x7feb60100ae0 2024-03-28T00:53:58.720 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.715+0000 7feb67812640 1 -- 172.21.15.169:0/3067024602 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb60107c40 con 0x7feb60100ae0 2024-03-28T00:53:58.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.716+0000 7feb567fc640 1 -- 172.21.15.169:0/3067024602 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7feb50018a70 con 0x7feb60100ae0 2024-03-28T00:53:58.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.717+0000 7feb567fc640 1 --2- 172.21.15.169:0/3067024602 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7feb4c0a2800 0x7feb4c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:53:58.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.717+0000 7feb655ae640 1 --2- 172.21.15.169:0/3067024602 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7feb4c0a2800 0x7feb4c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:53:58.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.717+0000 7feb567fc640 1 -- 172.21.15.169:0/3067024602 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7feb500ce7d0 con 0x7feb60100ae0 2024-03-28T00:53:58.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.717+0000 7feb655ae640 1 --2- 172.21.15.169:0/3067024602 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7feb4c0a2800 0x7feb4c0a4cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7feb5c005c20 tx=0x7feb5c033400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:53:58.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.721+0000 7feb567fc640 1 -- 172.21.15.169:0/3067024602 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7feb50014030 con 0x7feb60100ae0 2024-03-28T00:53:58.752 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:58 smithi174 ceph-mon[36060]: mds.? [v2:172.21.15.177:6832/3466292652,v1:172.21.15.177:6833/3466292652] up:boot 2024-03-28T00:53:58.752 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:58 smithi174 ceph-mon[36060]: fsmap 1 up:standby 2024-03-28T00:53:58.752 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:58 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T00:53:58.752 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:58 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:58.752 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:58 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:58.752 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:58 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:58.752 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:58 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:53:58.752 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:58 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:53:58.752 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:58 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:58.752 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:58 smithi174 ceph-mon[36060]: Deploying daemon mds.k on smithi174 2024-03-28T00:53:58.752 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:58 smithi174 ceph-mon[36060]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:58.938 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.935+0000 7feb67812640 1 -- 172.21.15.169:0/3067024602 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb6002d070 con 0x7feb60100ae0 2024-03-28T00:53:58.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.936+0000 7feb567fc640 1 -- 172.21.15.169:0/3067024602 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7feb5000de20 con 0x7feb60100ae0 2024-03-28T00:53:58.939 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:53:58.939 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T00:53:58.939 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 2 2024-03-28T00:53:58.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.938+0000 7feb67812640 1 -- 172.21.15.169:0/3067024602 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7feb4c0a2800 msgr2=0x7feb4c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:58.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.938+0000 7feb67812640 1 --2- 172.21.15.169:0/3067024602 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7feb4c0a2800 0x7feb4c0a4cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7feb5c005c20 tx=0x7feb5c033400 comp rx=0 tx=0).stop 2024-03-28T00:53:58.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.938+0000 7feb67812640 1 -- 172.21.15.169:0/3067024602 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb60100ae0 msgr2=0x7feb6019c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:53:58.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.938+0000 7feb67812640 1 --2- 172.21.15.169:0/3067024602 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb60100ae0 0x7feb6019c6f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7feb5000c920 tx=0x7feb5000cde0 comp rx=0 tx=0).stop 2024-03-28T00:53:58.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.938+0000 7feb67812640 1 -- 172.21.15.169:0/3067024602 shutdown_connections 2024-03-28T00:53:58.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.938+0000 7feb67812640 1 --2- 172.21.15.169:0/3067024602 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7feb4c0a2800 0x7feb4c0a4cc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:58.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.938+0000 7feb67812640 1 --2- 172.21.15.169:0/3067024602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb601014a0 0x7feb6019ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:58.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.938+0000 7feb67812640 1 --2- 172.21.15.169:0/3067024602 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7feb60100ae0 0x7feb6019c6f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:58.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.938+0000 7feb67812640 1 --2- 172.21.15.169:0/3067024602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb600ff8f0 0x7feb600fcfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:53:58.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.938+0000 7feb67812640 1 -- 172.21.15.169:0/3067024602 >> 172.21.15.169:0/3067024602 conn(0x7feb600faea0 msgr2=0x7feb60104e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:53:58.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.938+0000 7feb67812640 1 -- 172.21.15.169:0/3067024602 shutdown_connections 2024-03-28T00:53:58.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:53:58.938+0000 7feb67812640 1 -- 172.21.15.169:0/3067024602 wait complete. 2024-03-28T00:53:58.951 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:58 smithi169 ceph-mon[32274]: mds.? [v2:172.21.15.177:6832/3466292652,v1:172.21.15.177:6833/3466292652] up:boot 2024-03-28T00:53:58.951 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:58 smithi169 ceph-mon[32274]: fsmap 1 up:standby 2024-03-28T00:53:58.951 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:58 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T00:53:58.951 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:58 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:58.951 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:58 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:58.951 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:58 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:58.951 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:58 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:53:58.951 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:58 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:53:58.952 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:58 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:58.952 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:58 smithi169 ceph-mon[32274]: Deploying daemon mds.k on smithi174 2024-03-28T00:53:58.952 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:58 smithi169 ceph-mon[32274]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:53:59.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:58 smithi177 ceph-mon[36206]: mds.? [v2:172.21.15.177:6832/3466292652,v1:172.21.15.177:6833/3466292652] up:boot 2024-03-28T00:53:59.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:58 smithi177 ceph-mon[36206]: fsmap 1 up:standby 2024-03-28T00:53:59.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:58 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T00:53:59.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:58 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:59.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:58 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:59.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:58 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:53:59.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:58 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:53:59.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:58 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:53:59.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:58 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:53:59.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:58 smithi177 ceph-mon[36206]: Deploying daemon mds.k on smithi174 2024-03-28T00:53:59.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:58 smithi177 ceph-mon[36206]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:00.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:53:59 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3067024602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:00.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:53:59 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3067024602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:00.318 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:53:59 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3067024602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:00.318 INFO:journalctl@ceph.mds.k.smithi174.stdout:Mar 28 00:54:00 smithi174 systemd[1]: Starting Ceph mds.k for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T00:54:00.597 INFO:journalctl@ceph.mds.k.smithi174.stdout:Mar 28 00:54:00 smithi174 podman[65143]: 2024-03-28 00:54:00.218105567 +0000 UTC m=+0.017389757 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T00:54:00.597 INFO:journalctl@ceph.mds.k.smithi174.stdout:Mar 28 00:54:00 smithi174 podman[65143]: 2024-03-28 00:54:00.333760824 +0000 UTC m=+0.133044996 container create d59d450f31ef1f29786e7a8df3e3c4805e7de6fa75124dc0c43b42be64a08ac3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-k, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T00:54:00.858 INFO:journalctl@ceph.mds.k.smithi174.stdout:Mar 28 00:54:00 smithi174 podman[65143]: 2024-03-28 00:54:00.594796582 +0000 UTC m=+0.394080746 container init d59d450f31ef1f29786e7a8df3e3c4805e7de6fa75124dc0c43b42be64a08ac3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-k, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-28T00:54:00.858 INFO:journalctl@ceph.mds.k.smithi174.stdout:Mar 28 00:54:00 smithi174 podman[65143]: 2024-03-28 00:54:00.597553451 +0000 UTC m=+0.396837625 container start d59d450f31ef1f29786e7a8df3e3c4805e7de6fa75124dc0c43b42be64a08ac3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-k, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-28T00:54:00.858 INFO:journalctl@ceph.mds.k.smithi174.stdout:Mar 28 00:54:00 smithi174 ceph-mds[65162]: -- 172.21.15.174:0/2107478519 <== mon.2 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ef9f62da40 con 0x55ef9f601000 2024-03-28T00:54:00.858 INFO:journalctl@ceph.mds.k.smithi174.stdout:Mar 28 00:54:00 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-k[65158]: starting mds.k at 2024-03-28T00:54:00.858 INFO:journalctl@ceph.mds.k.smithi174.stdout:Mar 28 00:54:00 smithi174 bash[65143]: d59d450f31ef1f29786e7a8df3e3c4805e7de6fa75124dc0c43b42be64a08ac3 2024-03-28T00:54:00.859 INFO:journalctl@ceph.mds.k.smithi174.stdout:Mar 28 00:54:00 smithi174 systemd[1]: Started Ceph mds.k for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T00:54:00.953 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:54:01.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.054+0000 7f313ca85640 1 -- 172.21.15.169:0/2535403955 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31381017b0 msgr2=0x7f3138101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:01.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.054+0000 7f313ca85640 1 --2- 172.21.15.169:0/2535403955 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31381017b0 0x7f3138101c30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3128009900 tx=0x7f312802bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:01.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.055+0000 7f313ca85640 1 -- 172.21.15.169:0/2535403955 shutdown_connections 2024-03-28T00:54:01.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.055+0000 7f313ca85640 1 --2- 172.21.15.169:0/2535403955 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3138102170 0x7f3138108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:01.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.055+0000 7f313ca85640 1 --2- 172.21.15.169:0/2535403955 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31381017b0 0x7f3138101c30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:01.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.055+0000 7f313ca85640 1 --2- 172.21.15.169:0/2535403955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31381005c0 0x7f31381009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:01.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.055+0000 7f313ca85640 1 -- 172.21.15.169:0/2535403955 >> 172.21.15.169:0/2535403955 conn(0x7f31380fbb30 msgr2=0x7f31380fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:01.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.055+0000 7f313ca85640 1 -- 172.21.15.169:0/2535403955 shutdown_connections 2024-03-28T00:54:01.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.055+0000 7f313ca85640 1 -- 172.21.15.169:0/2535403955 wait complete. 2024-03-28T00:54:01.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.055+0000 7f313ca85640 1 Processor -- start 2024-03-28T00:54:01.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.055+0000 7f313ca85640 1 -- start start 2024-03-28T00:54:01.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.055+0000 7f313ca85640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31381005c0 0x7f3138197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:01.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.055+0000 7f313ca85640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31381017b0 0x7f3138197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:01.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.055+0000 7f313ca85640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3138102170 0x7f313819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:01.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.056+0000 7f313ca85640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313810c220 con 0x7f31381005c0 2024-03-28T00:54:01.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.056+0000 7f313ca85640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313810c0a0 con 0x7f31381017b0 2024-03-28T00:54:01.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.056+0000 7f313ca85640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313810c3a0 con 0x7f3138102170 2024-03-28T00:54:01.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.056+0000 7f313659c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31381005c0 0x7f3138197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:01.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.056+0000 7f313659c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31381005c0 0x7f3138197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40530/0 (socket says 172.21.15.169:40530) 2024-03-28T00:54:01.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.056+0000 7f313659c640 1 -- 172.21.15.169:0/1720404743 learned_addr learned my addr 172.21.15.169:0/1720404743 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:01.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.056+0000 7f3136d9d640 1 --2- 172.21.15.169:0/1720404743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3138102170 0x7f313819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:01.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.056+0000 7f3135d9b640 1 --2- 172.21.15.169:0/1720404743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31381017b0 0x7f3138197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:01.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.056+0000 7f313659c640 1 -- 172.21.15.169:0/1720404743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3138102170 msgr2=0x7f313819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:01.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.056+0000 7f313659c640 1 --2- 172.21.15.169:0/1720404743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3138102170 0x7f313819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:01.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.056+0000 7f313659c640 1 -- 172.21.15.169:0/1720404743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31381017b0 msgr2=0x7f3138197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:01.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.056+0000 7f313659c640 1 --2- 172.21.15.169:0/1720404743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31381017b0 0x7f3138197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:01.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.056+0000 7f313659c640 1 -- 172.21.15.169:0/1720404743 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313819f660 con 0x7f31381005c0 2024-03-28T00:54:01.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.056+0000 7f313659c640 1 --2- 172.21.15.169:0/1720404743 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31381005c0 0x7f3138197a40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f312c00e9c0 tx=0x7f312c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:01.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.056+0000 7f31277fe640 1 -- 172.21.15.169:0/1720404743 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c00cd80 con 0x7f31381005c0 2024-03-28T00:54:01.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.056+0000 7f31277fe640 1 -- 172.21.15.169:0/1720404743 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f312c0035f0 con 0x7f31381005c0 2024-03-28T00:54:01.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.056+0000 7f313ca85640 1 -- 172.21.15.169:0/1720404743 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f313819f950 con 0x7f31381005c0 2024-03-28T00:54:01.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.056+0000 7f31277fe640 1 -- 172.21.15.169:0/1720404743 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c010690 con 0x7f31381005c0 2024-03-28T00:54:01.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.056+0000 7f313ca85640 1 -- 172.21.15.169:0/1720404743 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313819fe70 con 0x7f31381005c0 2024-03-28T00:54:01.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.057+0000 7f313ca85640 1 -- 172.21.15.169:0/1720404743 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31381a0100 con 0x7f31381005c0 2024-03-28T00:54:01.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.059+0000 7f31277fe640 1 -- 172.21.15.169:0/1720404743 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f312c0037a0 con 0x7f31381005c0 2024-03-28T00:54:01.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.059+0000 7f31277fe640 1 --2- 172.21.15.169:0/1720404743 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f31200a2800 0x7f31200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:01.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.059+0000 7f3135d9b640 1 --2- 172.21.15.169:0/1720404743 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f31200a2800 0x7f31200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:01.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.060+0000 7f31277fe640 1 -- 172.21.15.169:0/1720404743 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f312c0cea20 con 0x7f31381005c0 2024-03-28T00:54:01.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.060+0000 7f3135d9b640 1 --2- 172.21.15.169:0/1720404743 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f31200a2800 0x7f31200a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3128006010 tx=0x7f3128000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:01.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.064+0000 7f31277fe640 1 -- 172.21.15.169:0/1720404743 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f312c09bf60 con 0x7f31381005c0 2024-03-28T00:54:01.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:00 smithi177 ceph-mon[36206]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:01.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:00 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:01.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:00 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:01.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:00 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:01.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:00 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:01.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:00 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:01.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:00 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:01.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.289+0000 7f313ca85640 1 -- 172.21.15.169:0/1720404743 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31381a0830 con 0x7f31381005c0 2024-03-28T00:54:01.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.290+0000 7f31277fe640 1 -- 172.21.15.169:0/1720404743 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2122 (secure 0 0 0) 0x7f312c09b860 con 0x7f31381005c0 2024-03-28T00:54:01.294 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:01.295 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T00:54:01.295 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 3 2024-03-28T00:54:01.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.292+0000 7f313ca85640 1 -- 172.21.15.169:0/1720404743 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f31200a2800 msgr2=0x7f31200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:01.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.292+0000 7f313ca85640 1 --2- 172.21.15.169:0/1720404743 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f31200a2800 0x7f31200a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3128006010 tx=0x7f3128000bc0 comp rx=0 tx=0).stop 2024-03-28T00:54:01.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.292+0000 7f313ca85640 1 -- 172.21.15.169:0/1720404743 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31381005c0 msgr2=0x7f3138197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:01.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.292+0000 7f313ca85640 1 --2- 172.21.15.169:0/1720404743 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31381005c0 0x7f3138197a40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f312c00e9c0 tx=0x7f312c00ee80 comp rx=0 tx=0).stop 2024-03-28T00:54:01.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.292+0000 7f313ca85640 1 -- 172.21.15.169:0/1720404743 shutdown_connections 2024-03-28T00:54:01.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.292+0000 7f313ca85640 1 --2- 172.21.15.169:0/1720404743 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f31200a2800 0x7f31200a4cc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:01.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.292+0000 7f313ca85640 1 --2- 172.21.15.169:0/1720404743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3138102170 0x7f313819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:01.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.292+0000 7f313ca85640 1 --2- 172.21.15.169:0/1720404743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31381017b0 0x7f3138197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:01.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.292+0000 7f313ca85640 1 --2- 172.21.15.169:0/1720404743 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31381005c0 0x7f3138197a40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:01.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.292+0000 7f313ca85640 1 -- 172.21.15.169:0/1720404743 >> 172.21.15.169:0/1720404743 conn(0x7f31380fbb30 msgr2=0x7f31380fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:01.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.292+0000 7f313ca85640 1 -- 172.21.15.169:0/1720404743 shutdown_connections 2024-03-28T00:54:01.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:01.292+0000 7f313ca85640 1 -- 172.21.15.169:0/1720404743 wait complete. 2024-03-28T00:54:01.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:00 smithi169 ceph-mon[32274]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:01.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:00 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:01.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:00 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:01.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:00 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:01.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:00 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:01.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:00 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:01.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:00 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:01.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:00 smithi174 ceph-mon[36060]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:01.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:00 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:01.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:00 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:01.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:00 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:01.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:00 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:01.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:00 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:01.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:00 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:02.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:01 smithi177 ceph-mon[36206]: Deploying daemon mds.j on smithi169 2024-03-28T00:54:02.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:01 smithi177 ceph-mon[36206]: mds.? [v2:172.21.15.174:6832/1840169743,v1:172.21.15.174:6833/1840169743] up:boot 2024-03-28T00:54:02.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:01 smithi177 ceph-mon[36206]: fsmap 2 up:standby 2024-03-28T00:54:02.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:01 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T00:54:02.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:01 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1720404743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:02.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:01 smithi169 ceph-mon[32274]: Deploying daemon mds.j on smithi169 2024-03-28T00:54:02.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:01 smithi169 ceph-mon[32274]: mds.? [v2:172.21.15.174:6832/1840169743,v1:172.21.15.174:6833/1840169743] up:boot 2024-03-28T00:54:02.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:01 smithi169 ceph-mon[32274]: fsmap 2 up:standby 2024-03-28T00:54:02.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:01 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T00:54:02.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:01 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1720404743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:02.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:01 smithi174 ceph-mon[36060]: Deploying daemon mds.j on smithi169 2024-03-28T00:54:02.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:01 smithi174 ceph-mon[36060]: mds.? [v2:172.21.15.174:6832/1840169743,v1:172.21.15.174:6833/1840169743] up:boot 2024-03-28T00:54:02.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:01 smithi174 ceph-mon[36060]: fsmap 2 up:standby 2024-03-28T00:54:02.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:01 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T00:54:02.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:01 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1720404743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:03.024 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:02 smithi174 ceph-mon[36060]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:03.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:02 smithi169 ceph-mon[32274]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:03.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:02 smithi177 ceph-mon[36206]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:03.310 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:54:03.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.415+0000 7f9bb3805640 1 -- 172.21.15.169:0/667792569 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bac1005c0 msgr2=0x7f9bac1009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:03.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.415+0000 7f9bb3805640 1 --2- 172.21.15.169:0/667792569 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bac1005c0 0x7f9bac1009e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8009900 tx=0x7f9ba802bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:03.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.415+0000 7f9bb3805640 1 -- 172.21.15.169:0/667792569 shutdown_connections 2024-03-28T00:54:03.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.415+0000 7f9bb3805640 1 --2- 172.21.15.169:0/667792569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bac102170 0x7f9bac108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:03.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.415+0000 7f9bb3805640 1 --2- 172.21.15.169:0/667792569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bac1017b0 0x7f9bac101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:03.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.415+0000 7f9bb3805640 1 --2- 172.21.15.169:0/667792569 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bac1005c0 0x7f9bac1009e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:03.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.415+0000 7f9bb3805640 1 -- 172.21.15.169:0/667792569 >> 172.21.15.169:0/667792569 conn(0x7f9bac0fbb30 msgr2=0x7f9bac0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:03.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.415+0000 7f9bb3805640 1 -- 172.21.15.169:0/667792569 shutdown_connections 2024-03-28T00:54:03.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.416+0000 7f9bb3805640 1 -- 172.21.15.169:0/667792569 wait complete. 2024-03-28T00:54:03.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.416+0000 7f9bb3805640 1 Processor -- start 2024-03-28T00:54:03.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.416+0000 7f9bb3805640 1 -- start start 2024-03-28T00:54:03.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.416+0000 7f9bb3805640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bac1005c0 0x7f9bac1968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:03.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.416+0000 7f9bb3805640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bac1017b0 0x7f9bac196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:03.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.416+0000 7f9bb3805640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bac102170 0x7f9bac19de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:03.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.416+0000 7f9bb3805640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac10c290 con 0x7f9bac102170 2024-03-28T00:54:03.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.416+0000 7f9bb3805640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac10c110 con 0x7f9bac1005c0 2024-03-28T00:54:03.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.416+0000 7f9bb3805640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac10c410 con 0x7f9bac1017b0 2024-03-28T00:54:03.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.416+0000 7f9bb15a1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bac1005c0 0x7f9bac1968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:03.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.416+0000 7f9bb1da2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bac102170 0x7f9bac19de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:03.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.416+0000 7f9bb15a1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bac1005c0 0x7f9bac1968e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:41470/0 (socket says 172.21.15.169:41470) 2024-03-28T00:54:03.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.416+0000 7f9bb1da2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bac102170 0x7f9bac19de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40540/0 (socket says 172.21.15.169:40540) 2024-03-28T00:54:03.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.416+0000 7f9bb15a1640 1 -- 172.21.15.169:0/1557058844 learned_addr learned my addr 172.21.15.169:0/1557058844 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:03.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.416+0000 7f9bb0da0640 1 --2- 172.21.15.169:0/1557058844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bac1017b0 0x7f9bac196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:03.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.417+0000 7f9bb15a1640 1 -- 172.21.15.169:0/1557058844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bac1017b0 msgr2=0x7f9bac196e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:03.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.417+0000 7f9bb15a1640 1 --2- 172.21.15.169:0/1557058844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bac1017b0 0x7f9bac196e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:03.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.417+0000 7f9bb15a1640 1 -- 172.21.15.169:0/1557058844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bac102170 msgr2=0x7f9bac19de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:03.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.417+0000 7f9bb15a1640 1 --2- 172.21.15.169:0/1557058844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bac102170 0x7f9bac19de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:03.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.417+0000 7f9bb15a1640 1 -- 172.21.15.169:0/1557058844 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bac19e500 con 0x7f9bac1005c0 2024-03-28T00:54:03.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.417+0000 7f9bb1da2640 1 --2- 172.21.15.169:0/1557058844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bac102170 0x7f9bac19de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:03.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.417+0000 7f9bb15a1640 1 --2- 172.21.15.169:0/1557058844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bac1005c0 0x7f9bac1968e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8000c00 tx=0x7f9ba802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:03.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.417+0000 7f9ba27fc640 1 -- 172.21.15.169:0/1557058844 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8043020 con 0x7f9bac1005c0 2024-03-28T00:54:03.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.417+0000 7f9bb3805640 1 -- 172.21.15.169:0/1557058844 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bac19e790 con 0x7f9bac1005c0 2024-03-28T00:54:03.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.417+0000 7f9ba27fc640 1 -- 172.21.15.169:0/1557058844 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ba80054e0 con 0x7f9bac1005c0 2024-03-28T00:54:03.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.417+0000 7f9ba27fc640 1 -- 172.21.15.169:0/1557058844 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8003c40 con 0x7f9bac1005c0 2024-03-28T00:54:03.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.417+0000 7f9bb3805640 1 -- 172.21.15.169:0/1557058844 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bac19ece0 con 0x7f9bac1005c0 2024-03-28T00:54:03.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.418+0000 7f9bb3805640 1 -- 172.21.15.169:0/1557058844 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bac1078e0 con 0x7f9bac1005c0 2024-03-28T00:54:03.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.419+0000 7f9ba27fc640 1 -- 172.21.15.169:0/1557058844 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9ba8046020 con 0x7f9bac1005c0 2024-03-28T00:54:03.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.420+0000 7f9ba27fc640 1 --2- 172.21.15.169:0/1557058844 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9b980a2800 0x7f9b980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:03.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.420+0000 7f9bb0da0640 1 --2- 172.21.15.169:0/1557058844 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9b980a2800 0x7f9b980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:03.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.420+0000 7f9ba27fc640 1 -- 172.21.15.169:0/1557058844 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9ba80f0580 con 0x7f9bac1005c0 2024-03-28T00:54:03.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.420+0000 7f9bb0da0640 1 --2- 172.21.15.169:0/1557058844 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9b980a2800 0x7f9b980a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9bac197df0 tx=0x7f9b9c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:03.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.425+0000 7f9ba27fc640 1 -- 172.21.15.169:0/1557058844 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9ba804d020 con 0x7f9bac1005c0 2024-03-28T00:54:03.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.651+0000 7f9bb3805640 1 -- 172.21.15.169:0/1557058844 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bac02d290 con 0x7f9bac1005c0 2024-03-28T00:54:03.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.652+0000 7f9ba27fc640 1 -- 172.21.15.169:0/1557058844 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2122 (secure 0 0 0) 0x7f9ba80bdac0 con 0x7f9bac1005c0 2024-03-28T00:54:03.655 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:03.655 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T00:54:03.655 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 3 2024-03-28T00:54:03.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.654+0000 7f9bb3805640 1 -- 172.21.15.169:0/1557058844 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9b980a2800 msgr2=0x7f9b980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:03.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.654+0000 7f9bb3805640 1 --2- 172.21.15.169:0/1557058844 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9b980a2800 0x7f9b980a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9bac197df0 tx=0x7f9b9c005f90 comp rx=0 tx=0).stop 2024-03-28T00:54:03.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.654+0000 7f9bb3805640 1 -- 172.21.15.169:0/1557058844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bac1005c0 msgr2=0x7f9bac1968e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:03.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.654+0000 7f9bb3805640 1 --2- 172.21.15.169:0/1557058844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bac1005c0 0x7f9bac1968e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8000c00 tx=0x7f9ba802c650 comp rx=0 tx=0).stop 2024-03-28T00:54:03.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.654+0000 7f9bb3805640 1 -- 172.21.15.169:0/1557058844 shutdown_connections 2024-03-28T00:54:03.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.654+0000 7f9bb3805640 1 --2- 172.21.15.169:0/1557058844 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9b980a2800 0x7f9b980a4cc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:03.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.654+0000 7f9bb3805640 1 --2- 172.21.15.169:0/1557058844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bac102170 0x7f9bac19de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:03.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.654+0000 7f9bb3805640 1 --2- 172.21.15.169:0/1557058844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bac1017b0 0x7f9bac196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:03.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.654+0000 7f9bb3805640 1 --2- 172.21.15.169:0/1557058844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bac1005c0 0x7f9bac1968e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:03.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.654+0000 7f9bb3805640 1 -- 172.21.15.169:0/1557058844 >> 172.21.15.169:0/1557058844 conn(0x7f9bac0fbb30 msgr2=0x7f9bac104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:03.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.654+0000 7f9bb3805640 1 -- 172.21.15.169:0/1557058844 shutdown_connections 2024-03-28T00:54:03.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:03.654+0000 7f9bb3805640 1 -- 172.21.15.169:0/1557058844 wait complete. 2024-03-28T00:54:03.667 INFO:journalctl@ceph.mds.j.smithi169.stdout:Mar 28 00:54:03 smithi169 systemd[1]: Started Ceph mds.j for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T00:54:04.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:04 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:04.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:04 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:04.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:04 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:04.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:04 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:04.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:04 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:04.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:04 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:04.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:04 smithi169 ceph-mon[32274]: Deploying daemon mds.d on smithi169 2024-03-28T00:54:04.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:04 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1557058844' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:04.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:04 smithi169 ceph-mon[32274]: mds.? [v2:172.21.15.169:6834/3964885254,v1:172.21.15.169:6835/3964885254] up:boot 2024-03-28T00:54:04.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:04 smithi169 ceph-mon[32274]: fsmap 3 up:standby 2024-03-28T00:54:04.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:04 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T00:54:04.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:04 smithi169 ceph-mon[32274]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:04.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:04 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:04.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:04 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:04.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:04 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:04.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:04 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:04.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:04 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:04.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:04 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:04.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:04 smithi174 ceph-mon[36060]: Deploying daemon mds.d on smithi169 2024-03-28T00:54:04.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:04 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1557058844' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:04.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:04 smithi174 ceph-mon[36060]: mds.? [v2:172.21.15.169:6834/3964885254,v1:172.21.15.169:6835/3964885254] up:boot 2024-03-28T00:54:04.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:04 smithi174 ceph-mon[36060]: fsmap 3 up:standby 2024-03-28T00:54:04.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:04 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T00:54:04.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:04 smithi174 ceph-mon[36060]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:04.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:04 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:04.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:04 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:04.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:04 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:04.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:04 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:04.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:04 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:04.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:04 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:04.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:04 smithi177 ceph-mon[36206]: Deploying daemon mds.d on smithi169 2024-03-28T00:54:04.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:04 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1557058844' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:04.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:04 smithi177 ceph-mon[36206]: mds.? [v2:172.21.15.169:6834/3964885254,v1:172.21.15.169:6835/3964885254] up:boot 2024-03-28T00:54:04.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:04 smithi177 ceph-mon[36206]: fsmap 3 up:standby 2024-03-28T00:54:04.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:04 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T00:54:04.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:04 smithi177 ceph-mon[36206]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:05.669 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:54:05.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.859+0000 7fe907d16640 1 -- 172.21.15.169:0/974661460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe900102160 msgr2=0x7fe900104550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:05.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.859+0000 7fe907d16640 1 --2- 172.21.15.169:0/974661460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe900102160 0x7fe900104550 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009900 tx=0x7fe8fc02bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:05.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.859+0000 7fe907d16640 1 -- 172.21.15.169:0/974661460 shutdown_connections 2024-03-28T00:54:05.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.859+0000 7fe907d16640 1 --2- 172.21.15.169:0/974661460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9001073c0 0x7fe900109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:05.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.859+0000 7fe907d16640 1 --2- 172.21.15.169:0/974661460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe900104a90 0x7fe900106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:05.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.859+0000 7fe907d16640 1 --2- 172.21.15.169:0/974661460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe900102160 0x7fe900104550 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:05.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.860+0000 7fe907d16640 1 -- 172.21.15.169:0/974661460 >> 172.21.15.169:0/974661460 conn(0x7fe9000fbb50 msgr2=0x7fe9000fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:05.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.860+0000 7fe907d16640 1 -- 172.21.15.169:0/974661460 shutdown_connections 2024-03-28T00:54:05.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.860+0000 7fe907d16640 1 -- 172.21.15.169:0/974661460 wait complete. 2024-03-28T00:54:05.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.860+0000 7fe907d16640 1 Processor -- start 2024-03-28T00:54:05.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.860+0000 7fe907d16640 1 -- start start 2024-03-28T00:54:05.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe907d16640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe900102160 0x7fe900197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:05.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe907d16640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe900104a90 0x7fe900198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:05.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe907d16640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9001073c0 0x7fe90019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:05.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe907d16640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90010d4a0 con 0x7fe900102160 2024-03-28T00:54:05.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe907d16640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90010d320 con 0x7fe9001073c0 2024-03-28T00:54:05.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe907d16640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90010d620 con 0x7fe900104a90 2024-03-28T00:54:05.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe905ab2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe900102160 0x7fe900197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:05.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe905ab2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe900102160 0x7fe900197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40590/0 (socket says 172.21.15.169:40590) 2024-03-28T00:54:05.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe905ab2640 1 -- 172.21.15.169:0/2582827873 learned_addr learned my addr 172.21.15.169:0/2582827873 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:05.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe9052b1640 1 --2- 172.21.15.169:0/2582827873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe900104a90 0x7fe900198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:05.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe9062b3640 1 --2- 172.21.15.169:0/2582827873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9001073c0 0x7fe90019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:05.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe905ab2640 1 -- 172.21.15.169:0/2582827873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe900104a90 msgr2=0x7fe900198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:05.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe905ab2640 1 --2- 172.21.15.169:0/2582827873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe900104a90 0x7fe900198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:05.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe905ab2640 1 -- 172.21.15.169:0/2582827873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9001073c0 msgr2=0x7fe90019f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:05.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe905ab2640 1 --2- 172.21.15.169:0/2582827873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9001073c0 0x7fe90019f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:05.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe905ab2640 1 -- 172.21.15.169:0/2582827873 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90019f730 con 0x7fe900102160 2024-03-28T00:54:05.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe9052b1640 1 --2- 172.21.15.169:0/2582827873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe900104a90 0x7fe900198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:54:05.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe9062b3640 1 --2- 172.21.15.169:0/2582827873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9001073c0 0x7fe90019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:54:05.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe905ab2640 1 --2- 172.21.15.169:0/2582827873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe900102160 0x7fe900197b10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc000c00 tx=0x7fe8fc02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:05.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe8f6ffd640 1 -- 172.21.15.169:0/2582827873 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc043020 con 0x7fe900102160 2024-03-28T00:54:05.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe907d16640 1 -- 172.21.15.169:0/2582827873 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe90019f9c0 con 0x7fe900102160 2024-03-28T00:54:05.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe8f6ffd640 1 -- 172.21.15.169:0/2582827873 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe8fc0054e0 con 0x7fe900102160 2024-03-28T00:54:05.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe8f6ffd640 1 -- 172.21.15.169:0/2582827873 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc003e50 con 0x7fe900102160 2024-03-28T00:54:05.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.861+0000 7fe907d16640 1 -- 172.21.15.169:0/2582827873 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90019ff10 con 0x7fe900102160 2024-03-28T00:54:05.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.862+0000 7fe907d16640 1 -- 172.21.15.169:0/2582827873 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9000fd640 con 0x7fe900102160 2024-03-28T00:54:05.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.864+0000 7fe8f6ffd640 1 -- 172.21.15.169:0/2582827873 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe8fc046020 con 0x7fe900102160 2024-03-28T00:54:05.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.865+0000 7fe8f6ffd640 1 --2- 172.21.15.169:0/2582827873 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe8ec0a29a0 0x7fe8ec0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:05.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.865+0000 7fe9052b1640 1 --2- 172.21.15.169:0/2582827873 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe8ec0a29a0 0x7fe8ec0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:05.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.865+0000 7fe8f6ffd640 1 -- 172.21.15.169:0/2582827873 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe8fc0f0eb0 con 0x7fe900102160 2024-03-28T00:54:05.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.865+0000 7fe9052b1640 1 --2- 172.21.15.169:0/2582827873 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe8ec0a29a0 0x7fe8ec0a4e60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe900199020 tx=0x7fe8f0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:05.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:05.868+0000 7fe8f6ffd640 1 -- 172.21.15.169:0/2582827873 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe8fc04d020 con 0x7fe900102160 2024-03-28T00:54:06.015 INFO:journalctl@ceph.mds.d.smithi169.stdout:Mar 28 00:54:05 smithi169 systemd[1]: Starting Ceph mds.d for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T00:54:06.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:06.090+0000 7fe907d16640 1 -- 172.21.15.169:0/2582827873 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe90002d190 con 0x7fe900102160 2024-03-28T00:54:06.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:06.090+0000 7fe8f6ffd640 1 -- 172.21.15.169:0/2582827873 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2916 (secure 0 0 0) 0x7fe8fc0be3d0 con 0x7fe900102160 2024-03-28T00:54:06.094 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:06.094 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T00:54:06.094 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 4 2024-03-28T00:54:06.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:06.092+0000 7fe907d16640 1 -- 172.21.15.169:0/2582827873 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe8ec0a29a0 msgr2=0x7fe8ec0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:06.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:06.092+0000 7fe907d16640 1 --2- 172.21.15.169:0/2582827873 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe8ec0a29a0 0x7fe8ec0a4e60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe900199020 tx=0x7fe8f0008040 comp rx=0 tx=0).stop 2024-03-28T00:54:06.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:06.092+0000 7fe907d16640 1 -- 172.21.15.169:0/2582827873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe900102160 msgr2=0x7fe900197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:06.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:06.092+0000 7fe907d16640 1 --2- 172.21.15.169:0/2582827873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe900102160 0x7fe900197b10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc000c00 tx=0x7fe8fc02c6c0 comp rx=0 tx=0).stop 2024-03-28T00:54:06.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:06.092+0000 7fe907d16640 1 -- 172.21.15.169:0/2582827873 shutdown_connections 2024-03-28T00:54:06.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:06.092+0000 7fe907d16640 1 --2- 172.21.15.169:0/2582827873 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe8ec0a29a0 0x7fe8ec0a4e60 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:06.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:06.092+0000 7fe907d16640 1 --2- 172.21.15.169:0/2582827873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9001073c0 0x7fe90019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:06.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:06.092+0000 7fe907d16640 1 --2- 172.21.15.169:0/2582827873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe900104a90 0x7fe900198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:06.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:06.092+0000 7fe907d16640 1 --2- 172.21.15.169:0/2582827873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe900102160 0x7fe900197b10 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:06.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:06.092+0000 7fe907d16640 1 -- 172.21.15.169:0/2582827873 >> 172.21.15.169:0/2582827873 conn(0x7fe9000fbb50 msgr2=0x7fe900102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:06.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:06.092+0000 7fe907d16640 1 -- 172.21.15.169:0/2582827873 shutdown_connections 2024-03-28T00:54:06.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:06.092+0000 7fe907d16640 1 -- 172.21.15.169:0/2582827873 wait complete. 2024-03-28T00:54:06.275 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:06 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2582827873' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:06.276 INFO:journalctl@ceph.mds.d.smithi169.stdout:Mar 28 00:54:06 smithi169 podman[71260]: 2024-03-28 00:54:06.011888752 +0000 UTC m=+0.113278256 container create 95e1f3625dc388656717ebbadde95089f39f2bd12699dbe880b6788812247355 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-28T00:54:06.276 INFO:journalctl@ceph.mds.d.smithi169.stdout:Mar 28 00:54:06 smithi169 podman[71260]: 2024-03-28 00:54:05.91658696 +0000 UTC m=+0.017976480 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T00:54:06.276 INFO:journalctl@ceph.mds.d.smithi169.stdout:Mar 28 00:54:06 smithi169 podman[71260]: 2024-03-28 00:54:06.248174497 +0000 UTC m=+0.349564000 container init 95e1f3625dc388656717ebbadde95089f39f2bd12699dbe880b6788812247355 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-d, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T00:54:06.276 INFO:journalctl@ceph.mds.d.smithi169.stdout:Mar 28 00:54:06 smithi169 podman[71260]: 2024-03-28 00:54:06.250953671 +0000 UTC m=+0.352343181 container start 95e1f3625dc388656717ebbadde95089f39f2bd12699dbe880b6788812247355 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-d, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-28T00:54:06.276 INFO:journalctl@ceph.mds.d.smithi169.stdout:Mar 28 00:54:06 smithi169 ceph-mds[71281]: -- 172.21.15.169:0/3837932296 <== mon.0 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562608399a40 con 0x56260836d000 2024-03-28T00:54:06.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:06 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2582827873' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:06.551 INFO:journalctl@ceph.mds.d.smithi169.stdout:Mar 28 00:54:06 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-d[71277]: starting mds.d at 2024-03-28T00:54:06.551 INFO:journalctl@ceph.mds.d.smithi169.stdout:Mar 28 00:54:06 smithi169 bash[71260]: 95e1f3625dc388656717ebbadde95089f39f2bd12699dbe880b6788812247355 2024-03-28T00:54:06.552 INFO:journalctl@ceph.mds.d.smithi169.stdout:Mar 28 00:54:06 smithi169 systemd[1]: Started Ceph mds.d for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T00:54:06.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:06 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2582827873' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:07.317 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:07 smithi174 ceph-mon[36060]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:07.317 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:07 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:07.317 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:07 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:07.317 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:07 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:07.317 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:07 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:07.317 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:07 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:07.317 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:07 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:07.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:07 smithi177 ceph-mon[36206]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:07.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:07 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:07.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:07 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:07.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:07 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:07.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:07 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:07.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:07 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:07.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:07 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:07.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:07 smithi169 ceph-mon[32274]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:07.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:07 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:07.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:07 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:07.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:07 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:07.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:07 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:07.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:07 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:07.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:07 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:08.108 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:54:08.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.222+0000 7fd67a66e640 1 -- 172.21.15.169:0/2705838996 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd674103310 msgr2=0x7fd674109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:08.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.222+0000 7fd67a66e640 1 --2- 172.21.15.169:0/2705838996 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd674103310 0x7fd674109ba0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd66c009cd0 tx=0x7fd66c02c0f0 comp rx=0 tx=0).stop 2024-03-28T00:54:08.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.222+0000 7fd67a66e640 1 -- 172.21.15.169:0/2705838996 shutdown_connections 2024-03-28T00:54:08.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.222+0000 7fd67a66e640 1 --2- 172.21.15.169:0/2705838996 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd674103310 0x7fd674109ba0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:08.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.222+0000 7fd67a66e640 1 --2- 172.21.15.169:0/2705838996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd674102950 0x7fd674102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:08.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.222+0000 7fd67a66e640 1 --2- 172.21.15.169:0/2705838996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6740ff980 0x7fd6740ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:08.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.222+0000 7fd67a66e640 1 -- 172.21.15.169:0/2705838996 >> 172.21.15.169:0/2705838996 conn(0x7fd6740fb2d0 msgr2=0x7fd6740fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:08.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.222+0000 7fd67a66e640 1 -- 172.21.15.169:0/2705838996 shutdown_connections 2024-03-28T00:54:08.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.222+0000 7fd67a66e640 1 -- 172.21.15.169:0/2705838996 wait complete. 2024-03-28T00:54:08.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.222+0000 7fd67a66e640 1 Processor -- start 2024-03-28T00:54:08.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.222+0000 7fd67a66e640 1 -- start start 2024-03-28T00:54:08.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.223+0000 7fd67a66e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6740ff980 0x7fd6741022c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:08.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.223+0000 7fd67a66e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd674102950 0x7fd674100900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:08.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.223+0000 7fd67a66e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd674103310 0x7fd674100e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:08.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.223+0000 7fd67a66e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67410d450 con 0x7fd6740ff980 2024-03-28T00:54:08.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.223+0000 7fd67a66e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67410d2d0 con 0x7fd674102950 2024-03-28T00:54:08.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.223+0000 7fd67a66e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67410d5d0 con 0x7fd674103310 2024-03-28T00:54:08.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.223+0000 7fd673fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6740ff980 0x7fd6741022c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:08.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.223+0000 7fd673fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6740ff980 0x7fd6741022c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:55512/0 (socket says 172.21.15.169:55512) 2024-03-28T00:54:08.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.223+0000 7fd673fff640 1 -- 172.21.15.169:0/4014193649 learned_addr learned my addr 172.21.15.169:0/4014193649 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:08.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.223+0000 7fd678c0b640 1 --2- 172.21.15.169:0/4014193649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd674103310 0x7fd674100e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:08.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.223+0000 7fd6737fe640 1 --2- 172.21.15.169:0/4014193649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd674102950 0x7fd674100900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:08.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.223+0000 7fd673fff640 1 -- 172.21.15.169:0/4014193649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd674103310 msgr2=0x7fd674100e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:08.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.223+0000 7fd673fff640 1 --2- 172.21.15.169:0/4014193649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd674103310 0x7fd674100e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:08.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.223+0000 7fd673fff640 1 -- 172.21.15.169:0/4014193649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd674102950 msgr2=0x7fd674100900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:08.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.223+0000 7fd673fff640 1 --2- 172.21.15.169:0/4014193649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd674102950 0x7fd674100900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:08.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.223+0000 7fd673fff640 1 -- 172.21.15.169:0/4014193649 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6741016d0 con 0x7fd6740ff980 2024-03-28T00:54:08.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.223+0000 7fd678c0b640 1 --2- 172.21.15.169:0/4014193649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd674103310 0x7fd674100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:54:08.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.223+0000 7fd673fff640 1 --2- 172.21.15.169:0/4014193649 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6740ff980 0x7fd6741022c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd66400b450 tx=0x7fd66400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:08.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.223+0000 7fd6717fa640 1 -- 172.21.15.169:0/4014193649 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664015020 con 0x7fd6740ff980 2024-03-28T00:54:08.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.224+0000 7fd67a66e640 1 -- 172.21.15.169:0/4014193649 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6741a1bb0 con 0x7fd6740ff980 2024-03-28T00:54:08.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.224+0000 7fd6717fa640 1 -- 172.21.15.169:0/4014193649 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd664011d70 con 0x7fd6740ff980 2024-03-28T00:54:08.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.224+0000 7fd6717fa640 1 -- 172.21.15.169:0/4014193649 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66400f340 con 0x7fd6740ff980 2024-03-28T00:54:08.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.224+0000 7fd67a66e640 1 -- 172.21.15.169:0/4014193649 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6741a20b0 con 0x7fd6740ff980 2024-03-28T00:54:08.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.225+0000 7fd67a66e640 1 -- 172.21.15.169:0/4014193649 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd674108b00 con 0x7fd6740ff980 2024-03-28T00:54:08.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.226+0000 7fd6717fa640 1 -- 172.21.15.169:0/4014193649 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd664018020 con 0x7fd6740ff980 2024-03-28T00:54:08.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.226+0000 7fd6717fa640 1 --2- 172.21.15.169:0/4014193649 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd6600a2800 0x7fd6600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:08.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.226+0000 7fd6737fe640 1 --2- 172.21.15.169:0/4014193649 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd6600a2800 0x7fd6600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:08.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.227+0000 7fd6717fa640 1 -- 172.21.15.169:0/4014193649 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd6640ce1d0 con 0x7fd6740ff980 2024-03-28T00:54:08.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.227+0000 7fd6737fe640 1 --2- 172.21.15.169:0/4014193649 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd6600a2800 0x7fd6600a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd674102030 tx=0x7fd65c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:08.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.231+0000 7fd6717fa640 1 -- 172.21.15.169:0/4014193649 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd66401f020 con 0x7fd6740ff980 2024-03-28T00:54:08.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:08 smithi174 ceph-mon[36060]: Deploying daemon mds.e on smithi174 2024-03-28T00:54:08.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:08 smithi174 ceph-mon[36060]: mds.? [v2:172.21.15.169:6836/2130558218,v1:172.21.15.169:6837/2130558218] up:boot 2024-03-28T00:54:08.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:08 smithi174 ceph-mon[36060]: fsmap 4 up:standby 2024-03-28T00:54:08.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:08 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T00:54:08.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:08 smithi177 ceph-mon[36206]: Deploying daemon mds.e on smithi174 2024-03-28T00:54:08.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:08 smithi177 ceph-mon[36206]: mds.? [v2:172.21.15.169:6836/2130558218,v1:172.21.15.169:6837/2130558218] up:boot 2024-03-28T00:54:08.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:08 smithi177 ceph-mon[36206]: fsmap 4 up:standby 2024-03-28T00:54:08.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:08 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T00:54:08.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.444+0000 7fd67a66e640 1 -- 172.21.15.169:0/4014193649 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd67402d070 con 0x7fd6740ff980 2024-03-28T00:54:08.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.445+0000 7fd6717fa640 1 -- 172.21.15.169:0/4014193649 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3710 (secure 0 0 0) 0x7fd66400d040 con 0x7fd6740ff980 2024-03-28T00:54:08.448 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:08.448 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T00:54:08.448 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 5 2024-03-28T00:54:08.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.447+0000 7fd67a66e640 1 -- 172.21.15.169:0/4014193649 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd6600a2800 msgr2=0x7fd6600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:08.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.447+0000 7fd67a66e640 1 --2- 172.21.15.169:0/4014193649 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd6600a2800 0x7fd6600a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd674102030 tx=0x7fd65c008040 comp rx=0 tx=0).stop 2024-03-28T00:54:08.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.447+0000 7fd67a66e640 1 -- 172.21.15.169:0/4014193649 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6740ff980 msgr2=0x7fd6741022c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:08.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.447+0000 7fd67a66e640 1 --2- 172.21.15.169:0/4014193649 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6740ff980 0x7fd6741022c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd66400b450 tx=0x7fd66400b910 comp rx=0 tx=0).stop 2024-03-28T00:54:08.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.447+0000 7fd67a66e640 1 -- 172.21.15.169:0/4014193649 shutdown_connections 2024-03-28T00:54:08.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.447+0000 7fd67a66e640 1 --2- 172.21.15.169:0/4014193649 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd6600a2800 0x7fd6600a4cc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:08.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.447+0000 7fd67a66e640 1 --2- 172.21.15.169:0/4014193649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd674103310 0x7fd674100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:08.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.447+0000 7fd67a66e640 1 --2- 172.21.15.169:0/4014193649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd674102950 0x7fd674100900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:08.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.447+0000 7fd67a66e640 1 --2- 172.21.15.169:0/4014193649 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6740ff980 0x7fd6741022c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:08.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.447+0000 7fd67a66e640 1 -- 172.21.15.169:0/4014193649 >> 172.21.15.169:0/4014193649 conn(0x7fd6740fb2d0 msgr2=0x7fd674105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:08.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.447+0000 7fd67a66e640 1 -- 172.21.15.169:0/4014193649 shutdown_connections 2024-03-28T00:54:08.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:08.447+0000 7fd67a66e640 1 -- 172.21.15.169:0/4014193649 wait complete. 2024-03-28T00:54:08.460 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:08 smithi169 ceph-mon[32274]: Deploying daemon mds.e on smithi174 2024-03-28T00:54:08.460 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:08 smithi169 ceph-mon[32274]: mds.? [v2:172.21.15.169:6836/2130558218,v1:172.21.15.169:6837/2130558218] up:boot 2024-03-28T00:54:08.460 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:08 smithi169 ceph-mon[32274]: fsmap 4 up:standby 2024-03-28T00:54:08.460 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:08 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T00:54:08.850 INFO:journalctl@ceph.mds.e.smithi174.stdout:Mar 28 00:54:08 smithi174 systemd[1]: Starting Ceph mds.e for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T00:54:09.108 INFO:journalctl@ceph.mds.e.smithi174.stdout:Mar 28 00:54:08 smithi174 podman[65464]: 2024-03-28 00:54:08.750758673 +0000 UTC m=+0.017822685 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T00:54:09.108 INFO:journalctl@ceph.mds.e.smithi174.stdout:Mar 28 00:54:08 smithi174 podman[65464]: 2024-03-28 00:54:08.871487694 +0000 UTC m=+0.138551692 container create 66562be0736e3bc3339f230efa51f144bb48a4a6b47c37a9626f12b1370c8306 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-28T00:54:09.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:09 smithi177 ceph-mon[36206]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:09.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:09 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/4014193649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:09.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:09 smithi169 ceph-mon[32274]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:09.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:09 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4014193649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:09.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:09 smithi174 ceph-mon[36060]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:09.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:09 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/4014193649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:09.608 INFO:journalctl@ceph.mds.e.smithi174.stdout:Mar 28 00:54:09 smithi174 podman[65464]: 2024-03-28 00:54:09.107258831 +0000 UTC m=+0.374322828 container init 66562be0736e3bc3339f230efa51f144bb48a4a6b47c37a9626f12b1370c8306 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-e, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-03-28T00:54:09.608 INFO:journalctl@ceph.mds.e.smithi174.stdout:Mar 28 00:54:09 smithi174 podman[65464]: 2024-03-28 00:54:09.109885644 +0000 UTC m=+0.376949645 container start 66562be0736e3bc3339f230efa51f144bb48a4a6b47c37a9626f12b1370c8306 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-e, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-28T00:54:09.608 INFO:journalctl@ceph.mds.e.smithi174.stdout:Mar 28 00:54:09 smithi174 ceph-mds[65483]: -- 172.21.15.174:0/766231878 <== mon.2 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56026c2f5a40 con 0x56026c2c8c00 2024-03-28T00:54:09.609 INFO:journalctl@ceph.mds.e.smithi174.stdout:Mar 28 00:54:09 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-e[65479]: starting mds.e at 2024-03-28T00:54:09.609 INFO:journalctl@ceph.mds.e.smithi174.stdout:Mar 28 00:54:09 smithi174 bash[65464]: 66562be0736e3bc3339f230efa51f144bb48a4a6b47c37a9626f12b1370c8306 2024-03-28T00:54:09.609 INFO:journalctl@ceph.mds.e.smithi174.stdout:Mar 28 00:54:09 smithi174 systemd[1]: Started Ceph mds.e for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T00:54:10.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:10 smithi177 ceph-mon[36206]: mds.? [v2:172.21.15.174:6834/3213155599,v1:172.21.15.174:6835/3213155599] up:boot 2024-03-28T00:54:10.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:10 smithi177 ceph-mon[36206]: fsmap 5 up:standby 2024-03-28T00:54:10.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T00:54:10.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:10.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:10.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:10.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:10.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:10.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:10 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:10.462 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:54:10.493 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:10 smithi169 ceph-mon[32274]: mds.? [v2:172.21.15.174:6834/3213155599,v1:172.21.15.174:6835/3213155599] up:boot 2024-03-28T00:54:10.494 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:10 smithi169 ceph-mon[32274]: fsmap 5 up:standby 2024-03-28T00:54:10.494 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T00:54:10.494 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:10.494 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:10.494 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:10.494 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:10.494 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:10.494 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:10 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:10.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.573+0000 7ffa30cdc640 1 -- 172.21.15.169:0/2542866115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa2c1021a0 msgr2=0x7ffa2c104590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:10.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.573+0000 7ffa30cdc640 1 --2- 172.21.15.169:0/2542866115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa2c1021a0 0x7ffa2c104590 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ffa20009900 tx=0x7ffa2002bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:10.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.573+0000 7ffa30cdc640 1 -- 172.21.15.169:0/2542866115 shutdown_connections 2024-03-28T00:54:10.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.573+0000 7ffa30cdc640 1 --2- 172.21.15.169:0/2542866115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa2c107400 0x7ffa2c1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:10.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.573+0000 7ffa30cdc640 1 --2- 172.21.15.169:0/2542866115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa2c104ad0 0x7ffa2c106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:10.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.573+0000 7ffa30cdc640 1 --2- 172.21.15.169:0/2542866115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa2c1021a0 0x7ffa2c104590 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:10.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.573+0000 7ffa30cdc640 1 -- 172.21.15.169:0/2542866115 >> 172.21.15.169:0/2542866115 conn(0x7ffa2c0fbb30 msgr2=0x7ffa2c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:10.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.573+0000 7ffa30cdc640 1 -- 172.21.15.169:0/2542866115 shutdown_connections 2024-03-28T00:54:10.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.573+0000 7ffa30cdc640 1 -- 172.21.15.169:0/2542866115 wait complete. 2024-03-28T00:54:10.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.574+0000 7ffa30cdc640 1 Processor -- start 2024-03-28T00:54:10.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.574+0000 7ffa30cdc640 1 -- start start 2024-03-28T00:54:10.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.574+0000 7ffa30cdc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa2c1021a0 0x7ffa2c1979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:10.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.574+0000 7ffa30cdc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa2c104ad0 0x7ffa2c197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:10.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.574+0000 7ffa30cdc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa2c107400 0x7ffa2c19ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:10.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.574+0000 7ffa30cdc640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c10d340 con 0x7ffa2c1021a0 2024-03-28T00:54:10.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.574+0000 7ffa30cdc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c10d1c0 con 0x7ffa2c107400 2024-03-28T00:54:10.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.574+0000 7ffa30cdc640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c10d4c0 con 0x7ffa2c104ad0 2024-03-28T00:54:10.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.574+0000 7ffa2a59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa2c1021a0 0x7ffa2c1979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:10.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.574+0000 7ffa2a59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa2c1021a0 0x7ffa2c1979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:55518/0 (socket says 172.21.15.169:55518) 2024-03-28T00:54:10.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.574+0000 7ffa2a59c640 1 -- 172.21.15.169:0/1695068079 learned_addr learned my addr 172.21.15.169:0/1695068079 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:10.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.574+0000 7ffa2ad9d640 1 --2- 172.21.15.169:0/1695068079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa2c107400 0x7ffa2c19ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:10.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.575+0000 7ffa29d9b640 1 --2- 172.21.15.169:0/1695068079 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa2c104ad0 0x7ffa2c197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:10.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.575+0000 7ffa2a59c640 1 -- 172.21.15.169:0/1695068079 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa2c104ad0 msgr2=0x7ffa2c197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:10.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.575+0000 7ffa2a59c640 1 --2- 172.21.15.169:0/1695068079 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa2c104ad0 0x7ffa2c197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:10.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.575+0000 7ffa2a59c640 1 -- 172.21.15.169:0/1695068079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa2c107400 msgr2=0x7ffa2c19ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:10.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.575+0000 7ffa2a59c640 1 --2- 172.21.15.169:0/1695068079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa2c107400 0x7ffa2c19ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:10.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.575+0000 7ffa2a59c640 1 -- 172.21.15.169:0/1695068079 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c19f600 con 0x7ffa2c1021a0 2024-03-28T00:54:10.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.575+0000 7ffa2a59c640 1 --2- 172.21.15.169:0/1695068079 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa2c1021a0 0x7ffa2c1979e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffa20009230 tx=0x7ffa20005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:10.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.575+0000 7ffa1b7fe640 1 -- 172.21.15.169:0/1695068079 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa20041070 con 0x7ffa2c1021a0 2024-03-28T00:54:10.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.575+0000 7ffa30cdc640 1 -- 172.21.15.169:0/1695068079 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2c19f890 con 0x7ffa2c1021a0 2024-03-28T00:54:10.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.575+0000 7ffa30cdc640 1 -- 172.21.15.169:0/1695068079 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2c19fdd0 con 0x7ffa2c1021a0 2024-03-28T00:54:10.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.575+0000 7ffa1b7fe640 1 -- 172.21.15.169:0/1695068079 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa200054e0 con 0x7ffa2c1021a0 2024-03-28T00:54:10.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.575+0000 7ffa1b7fe640 1 -- 172.21.15.169:0/1695068079 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2004b060 con 0x7ffa2c1021a0 2024-03-28T00:54:10.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.576+0000 7ffa30cdc640 1 -- 172.21.15.169:0/1695068079 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa0c0052c0 con 0x7ffa2c1021a0 2024-03-28T00:54:10.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.577+0000 7ffa1b7fe640 1 -- 172.21.15.169:0/1695068079 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ffa20044070 con 0x7ffa2c1021a0 2024-03-28T00:54:10.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.578+0000 7ffa1b7fe640 1 --2- 172.21.15.169:0/1695068079 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffa140a2800 0x7ffa140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:10.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.578+0000 7ffa29d9b640 1 --2- 172.21.15.169:0/1695068079 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffa140a2800 0x7ffa140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:10.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.578+0000 7ffa1b7fe640 1 -- 172.21.15.169:0/1695068079 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ffa200f01a0 con 0x7ffa2c1021a0 2024-03-28T00:54:10.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.578+0000 7ffa29d9b640 1 --2- 172.21.15.169:0/1695068079 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffa140a2800 0x7ffa140a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c005fd0 tx=0x7ffa1c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:10.584 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.581+0000 7ffa1b7fe640 1 -- 172.21.15.169:0/1695068079 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffa200bd6e0 con 0x7ffa2c1021a0 2024-03-28T00:54:10.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:10 smithi174 ceph-mon[36060]: mds.? [v2:172.21.15.174:6834/3213155599,v1:172.21.15.174:6835/3213155599] up:boot 2024-03-28T00:54:10.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:10 smithi174 ceph-mon[36060]: fsmap 5 up:standby 2024-03-28T00:54:10.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T00:54:10.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:10.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:10.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:10.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:10.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:10.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:10 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:10.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.792+0000 7ffa30cdc640 1 -- 172.21.15.169:0/1695068079 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa0c0055b0 con 0x7ffa2c1021a0 2024-03-28T00:54:10.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.793+0000 7ffa1b7fe640 1 -- 172.21.15.169:0/1695068079 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4504 (secure 0 0 0) 0x7ffa200bcfe0 con 0x7ffa2c1021a0 2024-03-28T00:54:10.796 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:10.796 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-28T00:54:10.796 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 6 2024-03-28T00:54:10.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.795+0000 7ffa30cdc640 1 -- 172.21.15.169:0/1695068079 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffa140a2800 msgr2=0x7ffa140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:10.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.795+0000 7ffa30cdc640 1 --2- 172.21.15.169:0/1695068079 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffa140a2800 0x7ffa140a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c005fd0 tx=0x7ffa1c005850 comp rx=0 tx=0).stop 2024-03-28T00:54:10.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.795+0000 7ffa30cdc640 1 -- 172.21.15.169:0/1695068079 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa2c1021a0 msgr2=0x7ffa2c1979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:10.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.795+0000 7ffa30cdc640 1 --2- 172.21.15.169:0/1695068079 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa2c1021a0 0x7ffa2c1979e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffa20009230 tx=0x7ffa20005dd0 comp rx=0 tx=0).stop 2024-03-28T00:54:10.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.795+0000 7ffa30cdc640 1 -- 172.21.15.169:0/1695068079 shutdown_connections 2024-03-28T00:54:10.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.795+0000 7ffa30cdc640 1 --2- 172.21.15.169:0/1695068079 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffa140a2800 0x7ffa140a4cc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:10.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.795+0000 7ffa30cdc640 1 --2- 172.21.15.169:0/1695068079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa2c107400 0x7ffa2c19ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:10.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.795+0000 7ffa30cdc640 1 --2- 172.21.15.169:0/1695068079 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa2c104ad0 0x7ffa2c197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:10.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.795+0000 7ffa30cdc640 1 --2- 172.21.15.169:0/1695068079 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa2c1021a0 0x7ffa2c1979e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:10.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.795+0000 7ffa30cdc640 1 -- 172.21.15.169:0/1695068079 >> 172.21.15.169:0/1695068079 conn(0x7ffa2c0fbb30 msgr2=0x7ffa2c102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:10.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.795+0000 7ffa30cdc640 1 -- 172.21.15.169:0/1695068079 shutdown_connections 2024-03-28T00:54:10.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:10.795+0000 7ffa30cdc640 1 -- 172.21.15.169:0/1695068079 wait complete. 2024-03-28T00:54:11.247 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:11 smithi177 ceph-mon[36206]: Deploying daemon mds.f on smithi177 2024-03-28T00:54:11.247 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:11 smithi177 ceph-mon[36206]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:11.247 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:11 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1695068079' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:11.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:11 smithi169 ceph-mon[32274]: Deploying daemon mds.f on smithi177 2024-03-28T00:54:11.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:11 smithi169 ceph-mon[32274]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:11.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:11 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1695068079' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:11.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:11 smithi174 ceph-mon[36060]: Deploying daemon mds.f on smithi177 2024-03-28T00:54:11.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:11 smithi174 ceph-mon[36060]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:11.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:11 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1695068079' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:11.630 INFO:journalctl@ceph.mds.f.smithi177.stdout:Mar 28 00:54:11 smithi177 systemd[1]: Starting Ceph mds.f for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T00:54:11.883 INFO:journalctl@ceph.mds.f.smithi177.stdout:Mar 28 00:54:11 smithi177 podman[67025]: 2024-03-28 00:54:11.530004875 +0000 UTC m=+0.017762684 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T00:54:11.883 INFO:journalctl@ceph.mds.f.smithi177.stdout:Mar 28 00:54:11 smithi177 podman[67025]: 2024-03-28 00:54:11.645034226 +0000 UTC m=+0.132792015 container create 8f7ef40b1835aeed41908232748f893a9e40bbeeb2fbc400ab19c1a4104a4fbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-f, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-28T00:54:11.884 INFO:journalctl@ceph.mds.f.smithi177.stdout:Mar 28 00:54:11 smithi177 podman[67025]: 2024-03-28 00:54:11.881385905 +0000 UTC m=+0.369143704 container init 8f7ef40b1835aeed41908232748f893a9e40bbeeb2fbc400ab19c1a4104a4fbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-f, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-28T00:54:12.166 INFO:journalctl@ceph.mds.f.smithi177.stdout:Mar 28 00:54:11 smithi177 podman[67025]: 2024-03-28 00:54:11.88456312 +0000 UTC m=+0.372320907 container start 8f7ef40b1835aeed41908232748f893a9e40bbeeb2fbc400ab19c1a4104a4fbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-f, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-28T00:54:12.167 INFO:journalctl@ceph.mds.f.smithi177.stdout:Mar 28 00:54:11 smithi177 ceph-mds[67044]: -- 172.21.15.177:0/245043241 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56490d6cfa40 con 0x56490d6a2800 2024-03-28T00:54:12.167 INFO:journalctl@ceph.mds.f.smithi177.stdout:Mar 28 00:54:11 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-f[67040]: starting mds.f at 2024-03-28T00:54:12.167 INFO:journalctl@ceph.mds.f.smithi177.stdout:Mar 28 00:54:11 smithi177 bash[67025]: 8f7ef40b1835aeed41908232748f893a9e40bbeeb2fbc400ab19c1a4104a4fbe 2024-03-28T00:54:12.167 INFO:journalctl@ceph.mds.f.smithi177.stdout:Mar 28 00:54:11 smithi177 systemd[1]: Started Ceph mds.f for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T00:54:12.810 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:54:12.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.922+0000 7f4cde177640 1 -- 172.21.15.169:0/1628706455 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4cd81073a0 msgr2=0x7f4cd81097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:12.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.922+0000 7f4cde177640 1 --2- 172.21.15.169:0/1628706455 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4cd81073a0 0x7f4cd81097f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009ca0 tx=0x7f4cd002c0f0 comp rx=0 tx=0).stop 2024-03-28T00:54:12.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.922+0000 7f4cde177640 1 -- 172.21.15.169:0/1628706455 shutdown_connections 2024-03-28T00:54:12.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.922+0000 7f4cde177640 1 --2- 172.21.15.169:0/1628706455 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4cd81073a0 0x7f4cd81097f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:12.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.922+0000 7f4cde177640 1 --2- 172.21.15.169:0/1628706455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cd8104a70 0x7f4cd8106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:12.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.922+0000 7f4cde177640 1 --2- 172.21.15.169:0/1628706455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4cd8102140 0x7f4cd8104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:12.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.922+0000 7f4cde177640 1 -- 172.21.15.169:0/1628706455 >> 172.21.15.169:0/1628706455 conn(0x7f4cd80fbb30 msgr2=0x7f4cd80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:12.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.922+0000 7f4cde177640 1 -- 172.21.15.169:0/1628706455 shutdown_connections 2024-03-28T00:54:12.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.922+0000 7f4cde177640 1 -- 172.21.15.169:0/1628706455 wait complete. 2024-03-28T00:54:12.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.923+0000 7f4cde177640 1 Processor -- start 2024-03-28T00:54:12.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.923+0000 7f4cde177640 1 -- start start 2024-03-28T00:54:12.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.923+0000 7f4cde177640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cd8102140 0x7f4cd8197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:12.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.923+0000 7f4cde177640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4cd8104a70 0x7f4cd8197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:12.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.923+0000 7f4cde177640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4cd81073a0 0x7f4cd819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:12.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.923+0000 7f4cde177640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd810d360 con 0x7f4cd8104a70 2024-03-28T00:54:12.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.923+0000 7f4cde177640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd810d1e0 con 0x7f4cd8102140 2024-03-28T00:54:12.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.923+0000 7f4cde177640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd810d4e0 con 0x7f4cd81073a0 2024-03-28T00:54:12.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.923+0000 7f4cd6ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4cd8104a70 0x7f4cd8197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:12.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.923+0000 7f4cd6ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4cd8104a70 0x7f4cd8197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:55532/0 (socket says 172.21.15.169:55532) 2024-03-28T00:54:12.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.923+0000 7f4cd6ffd640 1 -- 172.21.15.169:0/2486299568 learned_addr learned my addr 172.21.15.169:0/2486299568 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:12.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.923+0000 7f4cd7fff640 1 --2- 172.21.15.169:0/2486299568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4cd81073a0 0x7f4cd819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:12.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.923+0000 7f4cd77fe640 1 --2- 172.21.15.169:0/2486299568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cd8102140 0x7f4cd8197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:12.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.924+0000 7f4cd6ffd640 1 -- 172.21.15.169:0/2486299568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4cd81073a0 msgr2=0x7f4cd819ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:12.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.924+0000 7f4cd6ffd640 1 --2- 172.21.15.169:0/2486299568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4cd81073a0 0x7f4cd819ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:12.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.924+0000 7f4cd6ffd640 1 -- 172.21.15.169:0/2486299568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cd8102140 msgr2=0x7f4cd8197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:12.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.924+0000 7f4cd6ffd640 1 --2- 172.21.15.169:0/2486299568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cd8102140 0x7f4cd8197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:12.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.924+0000 7f4cd6ffd640 1 -- 172.21.15.169:0/2486299568 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd819f590 con 0x7f4cd8104a70 2024-03-28T00:54:12.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.924+0000 7f4cd7fff640 1 --2- 172.21.15.169:0/2486299568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4cd81073a0 0x7f4cd819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:12.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.924+0000 7f4cd77fe640 1 --2- 172.21.15.169:0/2486299568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cd8102140 0x7f4cd8197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:54:12.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.924+0000 7f4cd6ffd640 1 --2- 172.21.15.169:0/2486299568 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4cd8104a70 0x7f4cd8197eb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc00e9c0 tx=0x7f4ccc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:12.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.924+0000 7f4cd4ff9640 1 -- 172.21.15.169:0/2486299568 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc00cd80 con 0x7f4cd8104a70 2024-03-28T00:54:12.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.924+0000 7f4cde177640 1 -- 172.21.15.169:0/2486299568 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd819f880 con 0x7f4cd8104a70 2024-03-28T00:54:12.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.924+0000 7f4cd4ff9640 1 -- 172.21.15.169:0/2486299568 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ccc005500 con 0x7f4cd8104a70 2024-03-28T00:54:12.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.924+0000 7f4cd4ff9640 1 -- 172.21.15.169:0/2486299568 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc010690 con 0x7f4cd8104a70 2024-03-28T00:54:12.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.924+0000 7f4cde177640 1 -- 172.21.15.169:0/2486299568 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd819fe20 con 0x7f4cd8104a70 2024-03-28T00:54:12.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.925+0000 7f4cde177640 1 -- 172.21.15.169:0/2486299568 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd80fd620 con 0x7f4cd8104a70 2024-03-28T00:54:12.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.926+0000 7f4cd4ff9640 1 -- 172.21.15.169:0/2486299568 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4ccc0046c0 con 0x7f4cd8104a70 2024-03-28T00:54:12.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.927+0000 7f4cd4ff9640 1 --2- 172.21.15.169:0/2486299568 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4cc00a2800 0x7f4cc00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:12.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.927+0000 7f4cd4ff9640 1 -- 172.21.15.169:0/2486299568 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4ccc0cebf0 con 0x7f4cd8104a70 2024-03-28T00:54:12.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.927+0000 7f4cd77fe640 1 --2- 172.21.15.169:0/2486299568 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4cc00a2800 0x7f4cc00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:12.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.927+0000 7f4cd77fe640 1 --2- 172.21.15.169:0/2486299568 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4cc00a2800 0x7f4cc00a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8009c20 tx=0x7f4cc8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:12.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:12.930+0000 7f4cd4ff9640 1 -- 172.21.15.169:0/2486299568 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4ccc053740 con 0x7f4cd8104a70 2024-03-28T00:54:13.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:13.144+0000 7f4cde177640 1 -- 172.21.15.169:0/2486299568 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cd802d470 con 0x7f4cd8104a70 2024-03-28T00:54:13.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:13.145+0000 7f4cd4ff9640 1 -- 172.21.15.169:0/2486299568 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5298 (secure 0 0 0) 0x7f4ccc09c130 con 0x7f4cd8104a70 2024-03-28T00:54:13.148 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:13.148 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-28T00:54:13.148 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 7 2024-03-28T00:54:13.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:13.147+0000 7f4cde177640 1 -- 172.21.15.169:0/2486299568 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4cc00a2800 msgr2=0x7f4cc00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:13.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:13.147+0000 7f4cde177640 1 --2- 172.21.15.169:0/2486299568 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4cc00a2800 0x7f4cc00a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8009c20 tx=0x7f4cc8005f90 comp rx=0 tx=0).stop 2024-03-28T00:54:13.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:13.147+0000 7f4cde177640 1 -- 172.21.15.169:0/2486299568 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4cd8104a70 msgr2=0x7f4cd8197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:13.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:13.147+0000 7f4cde177640 1 --2- 172.21.15.169:0/2486299568 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4cd8104a70 0x7f4cd8197eb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc00e9c0 tx=0x7f4ccc00ee80 comp rx=0 tx=0).stop 2024-03-28T00:54:13.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:13.147+0000 7f4cde177640 1 -- 172.21.15.169:0/2486299568 shutdown_connections 2024-03-28T00:54:13.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:13.147+0000 7f4cde177640 1 --2- 172.21.15.169:0/2486299568 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4cc00a2800 0x7f4cc00a4cc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:13.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:13.147+0000 7f4cde177640 1 --2- 172.21.15.169:0/2486299568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4cd81073a0 0x7f4cd819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:13.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:13.147+0000 7f4cde177640 1 --2- 172.21.15.169:0/2486299568 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4cd8104a70 0x7f4cd8197eb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:13.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:13.147+0000 7f4cde177640 1 --2- 172.21.15.169:0/2486299568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cd8102140 0x7f4cd8197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:13.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:13.147+0000 7f4cde177640 1 -- 172.21.15.169:0/2486299568 >> 172.21.15.169:0/2486299568 conn(0x7f4cd80fbb30 msgr2=0x7f4cd8102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:13.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:13.147+0000 7f4cde177640 1 -- 172.21.15.169:0/2486299568 shutdown_connections 2024-03-28T00:54:13.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:13.147+0000 7f4cde177640 1 -- 172.21.15.169:0/2486299568 wait complete. 2024-03-28T00:54:13.160 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:13 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:13.160 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:13 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:13.160 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:13 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:13.161 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:13 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:13.161 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:13 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:13.161 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:13 smithi169 ceph-mon[32274]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:13.161 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:13 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:13.161 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:13 smithi169 ceph-mon[32274]: Deploying daemon mds.c on smithi177 2024-03-28T00:54:13.161 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:13 smithi169 ceph-mon[32274]: mds.? [v2:172.21.15.177:6834/1050217570,v1:172.21.15.177:6835/1050217570] up:boot 2024-03-28T00:54:13.161 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:13 smithi169 ceph-mon[32274]: fsmap 6 up:standby 2024-03-28T00:54:13.161 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:13 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T00:54:13.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:13 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:13.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:13 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:13.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:13 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:13.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:13 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:13.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:13 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:13.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:13 smithi174 ceph-mon[36060]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:13.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:13 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:13.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:13 smithi174 ceph-mon[36060]: Deploying daemon mds.c on smithi177 2024-03-28T00:54:13.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:13 smithi174 ceph-mon[36060]: mds.? [v2:172.21.15.177:6834/1050217570,v1:172.21.15.177:6835/1050217570] up:boot 2024-03-28T00:54:13.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:13 smithi174 ceph-mon[36060]: fsmap 6 up:standby 2024-03-28T00:54:13.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:13 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T00:54:13.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:13 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:13.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:13 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:13.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:13 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:13.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:13 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:13.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:13 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:13.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:13 smithi177 ceph-mon[36206]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:13.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:13 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:13.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:13 smithi177 ceph-mon[36206]: Deploying daemon mds.c on smithi177 2024-03-28T00:54:13.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:13 smithi177 ceph-mon[36206]: mds.? [v2:172.21.15.177:6834/1050217570,v1:172.21.15.177:6835/1050217570] up:boot 2024-03-28T00:54:13.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:13 smithi177 ceph-mon[36206]: fsmap 6 up:standby 2024-03-28T00:54:13.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:13 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T00:54:14.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:14 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2486299568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:14.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:14 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2486299568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:14.417 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 00:54:14 smithi177 systemd[1]: Starting Ceph mds.c for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T00:54:14.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:14 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2486299568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:14.748 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 00:54:14 smithi177 podman[67342]: 2024-03-28 00:54:14.38455955 +0000 UTC m=+0.017496863 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T00:54:14.748 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 00:54:14 smithi177 podman[67342]: 2024-03-28 00:54:14.509330323 +0000 UTC m=+0.142267679 container create 729e7063b6ebb665fda672bf336029c586c6da36c5d5dc10aa6d72214811c61c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T00:54:14.748 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 00:54:14 smithi177 podman[67342]: 2024-03-28 00:54:14.720196544 +0000 UTC m=+0.353133862 container init 729e7063b6ebb665fda672bf336029c586c6da36c5d5dc10aa6d72214811c61c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-c, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-28T00:54:14.748 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 00:54:14 smithi177 podman[67342]: 2024-03-28 00:54:14.722658096 +0000 UTC m=+0.355595411 container start 729e7063b6ebb665fda672bf336029c586c6da36c5d5dc10aa6d72214811c61c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-28T00:54:14.748 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 00:54:14 smithi177 ceph-mds[67361]: -- 172.21.15.177:0/4153510380 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55eeb9cd5a40 con 0x55eeb9ca8800 2024-03-28T00:54:15.139 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 00:54:14 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-c[67357]: starting mds.c at 2024-03-28T00:54:15.139 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 00:54:14 smithi177 bash[67342]: 729e7063b6ebb665fda672bf336029c586c6da36c5d5dc10aa6d72214811c61c 2024-03-28T00:54:15.139 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 00:54:14 smithi177 systemd[1]: Started Ceph mds.c for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T00:54:15.163 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:54:15.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.274+0000 7f542c71d640 1 -- 172.21.15.169:0/2369860916 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54241073a0 msgr2=0x7f54241097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:15.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.274+0000 7f542c71d640 1 --2- 172.21.15.169:0/2369860916 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54241073a0 0x7f54241097f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f541c009900 tx=0x7f541c02bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:15.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.275+0000 7f542c71d640 1 -- 172.21.15.169:0/2369860916 shutdown_connections 2024-03-28T00:54:15.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.275+0000 7f542c71d640 1 --2- 172.21.15.169:0/2369860916 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54241073a0 0x7f54241097f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:15.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.275+0000 7f542c71d640 1 --2- 172.21.15.169:0/2369860916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5424104a70 0x7f5424106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:15.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.275+0000 7f542c71d640 1 --2- 172.21.15.169:0/2369860916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5424102140 0x7f5424104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:15.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.275+0000 7f542c71d640 1 -- 172.21.15.169:0/2369860916 >> 172.21.15.169:0/2369860916 conn(0x7f54240fbb30 msgr2=0x7f54240fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:15.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.275+0000 7f542c71d640 1 -- 172.21.15.169:0/2369860916 shutdown_connections 2024-03-28T00:54:15.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.275+0000 7f542c71d640 1 -- 172.21.15.169:0/2369860916 wait complete. 2024-03-28T00:54:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.275+0000 7f542c71d640 1 Processor -- start 2024-03-28T00:54:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.275+0000 7f542c71d640 1 -- start start 2024-03-28T00:54:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.275+0000 7f542c71d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5424102140 0x7f54241979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.275+0000 7f542c71d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5424104a70 0x7f5424197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.275+0000 7f542c71d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54241073a0 0x7f542419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.275+0000 7f542c71d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542410d360 con 0x7f5424104a70 2024-03-28T00:54:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.275+0000 7f542c71d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542410d1e0 con 0x7f54241073a0 2024-03-28T00:54:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.275+0000 7f542c71d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542410d4e0 con 0x7f5424102140 2024-03-28T00:54:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.276+0000 7f5429cb8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5424104a70 0x7f5424197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.276+0000 7f5429cb8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5424104a70 0x7f5424197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:55548/0 (socket says 172.21.15.169:55548) 2024-03-28T00:54:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.276+0000 7f5429cb8640 1 -- 172.21.15.169:0/1412006456 learned_addr learned my addr 172.21.15.169:0/1412006456 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.276+0000 7f542acba640 1 --2- 172.21.15.169:0/1412006456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54241073a0 0x7f542419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.276+0000 7f542a4b9640 1 --2- 172.21.15.169:0/1412006456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5424102140 0x7f54241979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.276+0000 7f5429cb8640 1 -- 172.21.15.169:0/1412006456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5424102140 msgr2=0x7f54241979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:15.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.276+0000 7f5429cb8640 1 --2- 172.21.15.169:0/1412006456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5424102140 0x7f54241979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:15.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.276+0000 7f5429cb8640 1 -- 172.21.15.169:0/1412006456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54241073a0 msgr2=0x7f542419ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:15.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.276+0000 7f5429cb8640 1 --2- 172.21.15.169:0/1412006456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54241073a0 0x7f542419ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:15.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.276+0000 7f5429cb8640 1 -- 172.21.15.169:0/1412006456 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f542419f5d0 con 0x7f5424104a70 2024-03-28T00:54:15.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.276+0000 7f542a4b9640 1 --2- 172.21.15.169:0/1412006456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5424102140 0x7f54241979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:54:15.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.276+0000 7f5429cb8640 1 --2- 172.21.15.169:0/1412006456 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5424104a70 0x7f5424197ef0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f541400e9c0 tx=0x7f541400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:15.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.276+0000 7f541b7fe640 1 -- 172.21.15.169:0/1412006456 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541400cd80 con 0x7f5424104a70 2024-03-28T00:54:15.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.276+0000 7f542c71d640 1 -- 172.21.15.169:0/1412006456 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f542419f8c0 con 0x7f5424104a70 2024-03-28T00:54:15.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.276+0000 7f541b7fe640 1 -- 172.21.15.169:0/1412006456 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5414005500 con 0x7f5424104a70 2024-03-28T00:54:15.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.276+0000 7f541b7fe640 1 -- 172.21.15.169:0/1412006456 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5414010690 con 0x7f5424104a70 2024-03-28T00:54:15.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.276+0000 7f542c71d640 1 -- 172.21.15.169:0/1412006456 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f542419fe10 con 0x7f5424104a70 2024-03-28T00:54:15.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.277+0000 7f542c71d640 1 -- 172.21.15.169:0/1412006456 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54240fd620 con 0x7f5424104a70 2024-03-28T00:54:15.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.278+0000 7f541b7fe640 1 -- 172.21.15.169:0/1412006456 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f54140046c0 con 0x7f5424104a70 2024-03-28T00:54:15.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.279+0000 7f541b7fe640 1 --2- 172.21.15.169:0/1412006456 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f54100a2800 0x7f54100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:15.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.279+0000 7f542a4b9640 1 --2- 172.21.15.169:0/1412006456 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f54100a2800 0x7f54100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:15.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.279+0000 7f541b7fe640 1 -- 172.21.15.169:0/1412006456 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f54140ce9d0 con 0x7f5424104a70 2024-03-28T00:54:15.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.279+0000 7f542a4b9640 1 --2- 172.21.15.169:0/1412006456 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f54100a2800 0x7f54100a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5420005fd0 tx=0x7f5420009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:15.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.284+0000 7f541b7fe640 1 -- 172.21.15.169:0/1412006456 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f541409bf10 con 0x7f5424104a70 2024-03-28T00:54:15.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:15 smithi177 ceph-mon[36206]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:15.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:15 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:15.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:15 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:15.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:15 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:15.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:15 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:15.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:15 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:15.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:15 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:15.453 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:15 smithi169 ceph-mon[32274]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:15.453 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:15 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:15.453 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:15 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:15.453 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:15 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:15.453 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:15 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:15.453 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:15 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:15.453 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:15 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:15.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.500+0000 7f542c71d640 1 -- 172.21.15.169:0/1412006456 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f542402d250 con 0x7f5424104a70 2024-03-28T00:54:15.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.501+0000 7f541b7fe640 1 -- 172.21.15.169:0/1412006456 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6092 (secure 0 0 0) 0x7f541409b810 con 0x7f5424104a70 2024-03-28T00:54:15.504 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:15.504 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-28T00:54:15.504 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 8 2024-03-28T00:54:15.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.503+0000 7f542c71d640 1 -- 172.21.15.169:0/1412006456 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f54100a2800 msgr2=0x7f54100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:15.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.503+0000 7f542c71d640 1 --2- 172.21.15.169:0/1412006456 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f54100a2800 0x7f54100a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5420005fd0 tx=0x7f5420009500 comp rx=0 tx=0).stop 2024-03-28T00:54:15.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.503+0000 7f542c71d640 1 -- 172.21.15.169:0/1412006456 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5424104a70 msgr2=0x7f5424197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:15.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.503+0000 7f542c71d640 1 --2- 172.21.15.169:0/1412006456 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5424104a70 0x7f5424197ef0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f541400e9c0 tx=0x7f541400ee80 comp rx=0 tx=0).stop 2024-03-28T00:54:15.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.503+0000 7f542c71d640 1 -- 172.21.15.169:0/1412006456 shutdown_connections 2024-03-28T00:54:15.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.503+0000 7f542c71d640 1 --2- 172.21.15.169:0/1412006456 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f54100a2800 0x7f54100a4cc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:15.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.503+0000 7f542c71d640 1 --2- 172.21.15.169:0/1412006456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54241073a0 0x7f542419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:15.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.503+0000 7f542c71d640 1 --2- 172.21.15.169:0/1412006456 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5424104a70 0x7f5424197ef0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:15.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.503+0000 7f542c71d640 1 --2- 172.21.15.169:0/1412006456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5424102140 0x7f54241979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:15.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.503+0000 7f542c71d640 1 -- 172.21.15.169:0/1412006456 >> 172.21.15.169:0/1412006456 conn(0x7f54240fbb30 msgr2=0x7f5424102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:15.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.503+0000 7f542c71d640 1 -- 172.21.15.169:0/1412006456 shutdown_connections 2024-03-28T00:54:15.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:15.503+0000 7f542c71d640 1 -- 172.21.15.169:0/1412006456 wait complete. 2024-03-28T00:54:15.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:15 smithi174 ceph-mon[36060]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:15.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:15 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:15.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:15 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:15.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:15 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:15.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:15 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:15.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:15 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:15.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:15 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:16.412 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:16 smithi177 ceph-mon[36206]: Deploying daemon mds.h on smithi174 2024-03-28T00:54:16.412 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:16 smithi177 ceph-mon[36206]: mds.? [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] up:boot 2024-03-28T00:54:16.412 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:16 smithi177 ceph-mon[36206]: fsmap 7 up:standby 2024-03-28T00:54:16.412 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:16 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T00:54:16.413 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:16 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1412006456' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:16.470 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:16 smithi174 ceph-mon[36060]: Deploying daemon mds.h on smithi174 2024-03-28T00:54:16.470 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:16 smithi174 ceph-mon[36060]: mds.? [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] up:boot 2024-03-28T00:54:16.470 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:16 smithi174 ceph-mon[36060]: fsmap 7 up:standby 2024-03-28T00:54:16.470 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:16 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T00:54:16.470 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:16 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1412006456' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:16.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:16 smithi169 ceph-mon[32274]: Deploying daemon mds.h on smithi174 2024-03-28T00:54:16.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:16 smithi169 ceph-mon[32274]: mds.? [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] up:boot 2024-03-28T00:54:16.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:16 smithi169 ceph-mon[32274]: fsmap 7 up:standby 2024-03-28T00:54:16.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:16 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T00:54:16.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:16 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1412006456' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:17.398 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:17 smithi174 ceph-mon[36060]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:17.398 INFO:journalctl@ceph.mds.h.smithi174.stdout:Mar 28 00:54:17 smithi174 systemd[1]: Starting Ceph mds.h for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T00:54:17.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:17 smithi177 ceph-mon[36206]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:17.518 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:54:17.546 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:17 smithi169 ceph-mon[32274]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:17.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.627+0000 7fda551e1640 1 -- 172.21.15.169:0/3634426048 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda50102160 msgr2=0x7fda50104550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:17.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.627+0000 7fda551e1640 1 --2- 172.21.15.169:0/3634426048 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda50102160 0x7fda50104550 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fda40009900 tx=0x7fda4002bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:17.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.627+0000 7fda551e1640 1 -- 172.21.15.169:0/3634426048 shutdown_connections 2024-03-28T00:54:17.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.627+0000 7fda551e1640 1 --2- 172.21.15.169:0/3634426048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda501073c0 0x7fda50109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:17.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.627+0000 7fda551e1640 1 --2- 172.21.15.169:0/3634426048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda50104a90 0x7fda50106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:17.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.627+0000 7fda551e1640 1 --2- 172.21.15.169:0/3634426048 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda50102160 0x7fda50104550 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:17.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.627+0000 7fda551e1640 1 -- 172.21.15.169:0/3634426048 >> 172.21.15.169:0/3634426048 conn(0x7fda500fbb50 msgr2=0x7fda500fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:17.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.627+0000 7fda551e1640 1 -- 172.21.15.169:0/3634426048 shutdown_connections 2024-03-28T00:54:17.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.627+0000 7fda551e1640 1 -- 172.21.15.169:0/3634426048 wait complete. 2024-03-28T00:54:17.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.628+0000 7fda551e1640 1 Processor -- start 2024-03-28T00:54:17.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.628+0000 7fda551e1640 1 -- start start 2024-03-28T00:54:17.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.628+0000 7fda551e1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda50102160 0x7fda50197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:17.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.628+0000 7fda551e1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda50104a90 0x7fda50197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:17.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.628+0000 7fda551e1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda501073c0 0x7fda5019f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:17.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.628+0000 7fda551e1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5010d3e0 con 0x7fda50104a90 2024-03-28T00:54:17.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.628+0000 7fda551e1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5010d260 con 0x7fda501073c0 2024-03-28T00:54:17.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.628+0000 7fda551e1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5010d560 con 0x7fda50102160 2024-03-28T00:54:17.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.628+0000 7fda4e59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda50104a90 0x7fda50197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:17.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.628+0000 7fda4e59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda50104a90 0x7fda50197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51326/0 (socket says 172.21.15.169:51326) 2024-03-28T00:54:17.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.628+0000 7fda4e59c640 1 -- 172.21.15.169:0/1034766404 learned_addr learned my addr 172.21.15.169:0/1034766404 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:17.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.629+0000 7fda4f59e640 1 --2- 172.21.15.169:0/1034766404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda501073c0 0x7fda5019f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:17.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.629+0000 7fda4ed9d640 1 --2- 172.21.15.169:0/1034766404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda50102160 0x7fda50197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:17.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.629+0000 7fda4e59c640 1 -- 172.21.15.169:0/1034766404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda50102160 msgr2=0x7fda50197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:17.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.629+0000 7fda4e59c640 1 --2- 172.21.15.169:0/1034766404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda50102160 0x7fda50197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:17.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.629+0000 7fda4e59c640 1 -- 172.21.15.169:0/1034766404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda501073c0 msgr2=0x7fda5019f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:17.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.629+0000 7fda4e59c640 1 --2- 172.21.15.169:0/1034766404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda501073c0 0x7fda5019f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:17.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.629+0000 7fda4e59c640 1 -- 172.21.15.169:0/1034766404 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5019f670 con 0x7fda50104a90 2024-03-28T00:54:17.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.629+0000 7fda4f59e640 1 --2- 172.21.15.169:0/1034766404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda501073c0 0x7fda5019f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:54:17.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.629+0000 7fda4e59c640 1 --2- 172.21.15.169:0/1034766404 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda50104a90 0x7fda50197f90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fda4400d740 tx=0x7fda4400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:17.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.629+0000 7fda37fff640 1 -- 172.21.15.169:0/1034766404 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda44017070 con 0x7fda50104a90 2024-03-28T00:54:17.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.629+0000 7fda551e1640 1 -- 172.21.15.169:0/1034766404 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda5019f960 con 0x7fda50104a90 2024-03-28T00:54:17.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.629+0000 7fda37fff640 1 -- 172.21.15.169:0/1034766404 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda4400de50 con 0x7fda50104a90 2024-03-28T00:54:17.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.629+0000 7fda37fff640 1 -- 172.21.15.169:0/1034766404 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda44004e00 con 0x7fda50104a90 2024-03-28T00:54:17.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.629+0000 7fda551e1640 1 -- 172.21.15.169:0/1034766404 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda5019fe60 con 0x7fda50104a90 2024-03-28T00:54:17.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.630+0000 7fda551e1640 1 -- 172.21.15.169:0/1034766404 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda500fd640 con 0x7fda50104a90 2024-03-28T00:54:17.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.631+0000 7fda37fff640 1 -- 172.21.15.169:0/1034766404 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fda44004fa0 con 0x7fda50104a90 2024-03-28T00:54:17.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.632+0000 7fda37fff640 1 --2- 172.21.15.169:0/1034766404 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fda380a2800 0x7fda380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:17.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.632+0000 7fda4ed9d640 1 --2- 172.21.15.169:0/1034766404 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fda380a2800 0x7fda380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:17.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.632+0000 7fda37fff640 1 -- 172.21.15.169:0/1034766404 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fda440cdff0 con 0x7fda50104a90 2024-03-28T00:54:17.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.632+0000 7fda4ed9d640 1 --2- 172.21.15.169:0/1034766404 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fda380a2800 0x7fda380a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fda400096c0 tx=0x7fda4002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:17.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.636+0000 7fda37fff640 1 -- 172.21.15.169:0/1034766404 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fda4409b530 con 0x7fda50104a90 2024-03-28T00:54:17.690 INFO:journalctl@ceph.mds.h.smithi174.stdout:Mar 28 00:54:17 smithi174 podman[65786]: 2024-03-28 00:54:17.299230517 +0000 UTC m=+0.018292569 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T00:54:17.690 INFO:journalctl@ceph.mds.h.smithi174.stdout:Mar 28 00:54:17 smithi174 podman[65786]: 2024-03-28 00:54:17.426024084 +0000 UTC m=+0.145086121 container create 7c47ea8c36a858aaaf82bcd5671484eed5a96cf29bafa8c1d53a0b830a36416c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-h, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-28T00:54:17.691 INFO:journalctl@ceph.mds.h.smithi174.stdout:Mar 28 00:54:17 smithi174 podman[65786]: 2024-03-28 00:54:17.687690592 +0000 UTC m=+0.406752649 container init 7c47ea8c36a858aaaf82bcd5671484eed5a96cf29bafa8c1d53a0b830a36416c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-h, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T00:54:17.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.849+0000 7fda551e1640 1 -- 172.21.15.169:0/1034766404 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda5002d070 con 0x7fda50104a90 2024-03-28T00:54:17.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.850+0000 7fda37fff640 1 -- 172.21.15.169:0/1034766404 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6092 (secure 0 0 0) 0x7fda4409ae30 con 0x7fda50104a90 2024-03-28T00:54:17.853 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:17.853 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-28T00:54:17.853 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 8 2024-03-28T00:54:17.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.852+0000 7fda551e1640 1 -- 172.21.15.169:0/1034766404 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fda380a2800 msgr2=0x7fda380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:17.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.852+0000 7fda551e1640 1 --2- 172.21.15.169:0/1034766404 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fda380a2800 0x7fda380a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fda400096c0 tx=0x7fda4002df90 comp rx=0 tx=0).stop 2024-03-28T00:54:17.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.852+0000 7fda551e1640 1 -- 172.21.15.169:0/1034766404 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda50104a90 msgr2=0x7fda50197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:17.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.852+0000 7fda551e1640 1 --2- 172.21.15.169:0/1034766404 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda50104a90 0x7fda50197f90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fda4400d740 tx=0x7fda4400dc00 comp rx=0 tx=0).stop 2024-03-28T00:54:17.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.852+0000 7fda551e1640 1 -- 172.21.15.169:0/1034766404 shutdown_connections 2024-03-28T00:54:17.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.852+0000 7fda551e1640 1 --2- 172.21.15.169:0/1034766404 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fda380a2800 0x7fda380a4cc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:17.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.852+0000 7fda551e1640 1 --2- 172.21.15.169:0/1034766404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda501073c0 0x7fda5019f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:17.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.852+0000 7fda551e1640 1 --2- 172.21.15.169:0/1034766404 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda50104a90 0x7fda50197f90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:17.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.852+0000 7fda551e1640 1 --2- 172.21.15.169:0/1034766404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda50102160 0x7fda50197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:17.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.852+0000 7fda551e1640 1 -- 172.21.15.169:0/1034766404 >> 172.21.15.169:0/1034766404 conn(0x7fda500fbb50 msgr2=0x7fda50102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:17.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.852+0000 7fda551e1640 1 -- 172.21.15.169:0/1034766404 shutdown_connections 2024-03-28T00:54:17.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:17.852+0000 7fda551e1640 1 -- 172.21.15.169:0/1034766404 wait complete. 2024-03-28T00:54:18.108 INFO:journalctl@ceph.mds.h.smithi174.stdout:Mar 28 00:54:17 smithi174 podman[65786]: 2024-03-28 00:54:17.690366152 +0000 UTC m=+0.409428204 container start 7c47ea8c36a858aaaf82bcd5671484eed5a96cf29bafa8c1d53a0b830a36416c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-28T00:54:18.108 INFO:journalctl@ceph.mds.h.smithi174.stdout:Mar 28 00:54:17 smithi174 ceph-mds[65804]: -- 172.21.15.174:0/4294102849 <== mon.2 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563aba08da40 con 0x563aba061000 2024-03-28T00:54:18.108 INFO:journalctl@ceph.mds.h.smithi174.stdout:Mar 28 00:54:17 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-h[65800]: starting mds.h at 2024-03-28T00:54:18.108 INFO:journalctl@ceph.mds.h.smithi174.stdout:Mar 28 00:54:17 smithi174 bash[65786]: 7c47ea8c36a858aaaf82bcd5671484eed5a96cf29bafa8c1d53a0b830a36416c 2024-03-28T00:54:18.109 INFO:journalctl@ceph.mds.h.smithi174.stdout:Mar 28 00:54:17 smithi174 systemd[1]: Started Ceph mds.h for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T00:54:19.104 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:18 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:19.104 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:18 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:19.105 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:18 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1034766404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:19.105 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:18 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:19.105 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:18 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:19.105 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:18 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:19.105 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:18 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:19.105 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:18 smithi177 ceph-mon[36206]: Deploying daemon mds.i on smithi177 2024-03-28T00:54:19.105 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:18 smithi177 ceph-mon[36206]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:19.105 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:18 smithi177 ceph-mon[36206]: mds.? [v2:172.21.15.174:6836/3629228577,v1:172.21.15.174:6837/3629228577] up:boot 2024-03-28T00:54:19.105 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:18 smithi177 ceph-mon[36206]: fsmap 8 up:standby 2024-03-28T00:54:19.105 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:18 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T00:54:19.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:18 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:19.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:18 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:19.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:18 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1034766404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:19.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:18 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:19.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:18 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:19.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:18 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:19.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:18 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:19.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:18 smithi174 ceph-mon[36060]: Deploying daemon mds.i on smithi177 2024-03-28T00:54:19.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:18 smithi174 ceph-mon[36060]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:19.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:18 smithi174 ceph-mon[36060]: mds.? [v2:172.21.15.174:6836/3629228577,v1:172.21.15.174:6837/3629228577] up:boot 2024-03-28T00:54:19.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:18 smithi174 ceph-mon[36060]: fsmap 8 up:standby 2024-03-28T00:54:19.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:18 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T00:54:19.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:18 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:19.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:18 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:19.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:18 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1034766404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:19.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:18 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:19.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:18 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:19.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:18 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:19.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:18 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:19.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:18 smithi169 ceph-mon[32274]: Deploying daemon mds.i on smithi177 2024-03-28T00:54:19.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:18 smithi169 ceph-mon[32274]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:19.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:18 smithi169 ceph-mon[32274]: mds.? [v2:172.21.15.174:6836/3629228577,v1:172.21.15.174:6837/3629228577] up:boot 2024-03-28T00:54:19.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:18 smithi169 ceph-mon[32274]: fsmap 8 up:standby 2024-03-28T00:54:19.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:18 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T00:54:19.867 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:54:19.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.978+0000 7f18d09c6640 1 -- 172.21.15.169:0/3244084122 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18cc1011f0 msgr2=0x7f18cc108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:19.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.978+0000 7f18d09c6640 1 --2- 172.21.15.169:0/3244084122 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18cc1011f0 0x7f18cc108a90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f18c4009ca0 tx=0x7f18c402c0f0 comp rx=0 tx=0).stop 2024-03-28T00:54:19.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.978+0000 7f18d09c6640 1 -- 172.21.15.169:0/3244084122 shutdown_connections 2024-03-28T00:54:19.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.978+0000 7f18d09c6640 1 --2- 172.21.15.169:0/3244084122 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18cc1011f0 0x7f18cc108a90 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:19.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.978+0000 7f18d09c6640 1 --2- 172.21.15.169:0/3244084122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18cc100830 0x7f18cc100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:19.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.978+0000 7f18d09c6640 1 --2- 172.21.15.169:0/3244084122 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18cc101be0 0x7f18cc102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:19.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.978+0000 7f18d09c6640 1 -- 172.21.15.169:0/3244084122 >> 172.21.15.169:0/3244084122 conn(0x7f18cc0fb2c0 msgr2=0x7f18cc0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:19.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.978+0000 7f18d09c6640 1 -- 172.21.15.169:0/3244084122 shutdown_connections 2024-03-28T00:54:19.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.978+0000 7f18d09c6640 1 -- 172.21.15.169:0/3244084122 wait complete. 2024-03-28T00:54:19.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.979+0000 7f18d09c6640 1 Processor -- start 2024-03-28T00:54:19.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.979+0000 7f18d09c6640 1 -- start start 2024-03-28T00:54:19.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.979+0000 7f18d09c6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18cc100830 0x7f18cc1967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:19.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.979+0000 7f18d09c6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18cc1011f0 0x7f18cc196d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:19.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.979+0000 7f18d09c6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18cc101be0 0x7f18cc19dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:19.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.979+0000 7f18d09c6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18cc10c270 con 0x7f18cc1011f0 2024-03-28T00:54:19.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.979+0000 7f18d09c6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18cc10c0f0 con 0x7f18cc101be0 2024-03-28T00:54:19.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.979+0000 7f18d09c6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18cc10c3f0 con 0x7f18cc100830 2024-03-28T00:54:19.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.979+0000 7f18c9d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18cc1011f0 0x7f18cc196d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:19.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.979+0000 7f18c9d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18cc1011f0 0x7f18cc196d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51342/0 (socket says 172.21.15.169:51342) 2024-03-28T00:54:19.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.979+0000 7f18c9d9b640 1 -- 172.21.15.169:0/3758489230 learned_addr learned my addr 172.21.15.169:0/3758489230 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:19.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.979+0000 7f18ca59c640 1 --2- 172.21.15.169:0/3758489230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18cc100830 0x7f18cc1967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:19.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.980+0000 7f18cad9d640 1 --2- 172.21.15.169:0/3758489230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18cc101be0 0x7f18cc19dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:19.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.980+0000 7f18c9d9b640 1 -- 172.21.15.169:0/3758489230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18cc100830 msgr2=0x7f18cc1967f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:19.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.980+0000 7f18c9d9b640 1 --2- 172.21.15.169:0/3758489230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18cc100830 0x7f18cc1967f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:19.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.980+0000 7f18c9d9b640 1 -- 172.21.15.169:0/3758489230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18cc101be0 msgr2=0x7f18cc19dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:19.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.980+0000 7f18c9d9b640 1 --2- 172.21.15.169:0/3758489230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18cc101be0 0x7f18cc19dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:19.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.980+0000 7f18c9d9b640 1 -- 172.21.15.169:0/3758489230 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18cc19e410 con 0x7f18cc1011f0 2024-03-28T00:54:19.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.980+0000 7f18ca59c640 1 --2- 172.21.15.169:0/3758489230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18cc100830 0x7f18cc1967f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:19.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.980+0000 7f18c9d9b640 1 --2- 172.21.15.169:0/3758489230 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18cc1011f0 0x7f18cc196d30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f18c000c970 tx=0x7f18c000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:19.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.980+0000 7f18bb7fe640 1 -- 172.21.15.169:0/3758489230 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c0007c10 con 0x7f18cc1011f0 2024-03-28T00:54:19.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.980+0000 7f18bb7fe640 1 -- 172.21.15.169:0/3758489230 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f18c0007db0 con 0x7f18cc1011f0 2024-03-28T00:54:19.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.980+0000 7f18d09c6640 1 -- 172.21.15.169:0/3758489230 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18cc19e700 con 0x7f18cc1011f0 2024-03-28T00:54:19.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.980+0000 7f18bb7fe640 1 -- 172.21.15.169:0/3758489230 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c000f940 con 0x7f18cc1011f0 2024-03-28T00:54:19.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.980+0000 7f18d09c6640 1 -- 172.21.15.169:0/3758489230 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18cc19ec50 con 0x7f18cc1011f0 2024-03-28T00:54:19.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.981+0000 7f18d09c6640 1 -- 172.21.15.169:0/3758489230 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18cc1078a0 con 0x7f18cc1011f0 2024-03-28T00:54:19.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.982+0000 7f18bb7fe640 1 -- 172.21.15.169:0/3758489230 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f18c000fae0 con 0x7f18cc1011f0 2024-03-28T00:54:19.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.983+0000 7f18bb7fe640 1 --2- 172.21.15.169:0/3758489230 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f18b40a2800 0x7f18b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:19.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.983+0000 7f18ca59c640 1 --2- 172.21.15.169:0/3758489230 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f18b40a2800 0x7f18b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:19.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.983+0000 7f18bb7fe640 1 -- 172.21.15.169:0/3758489230 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f18c00ce8b0 con 0x7f18cc1011f0 2024-03-28T00:54:19.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.983+0000 7f18ca59c640 1 --2- 172.21.15.169:0/3758489230 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f18b40a2800 0x7f18b40a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f18cc069b30 tx=0x7f18bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:19.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:19.987+0000 7f18bb7fe640 1 -- 172.21.15.169:0/3758489230 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f18c009be80 con 0x7f18cc1011f0 2024-03-28T00:54:20.120 INFO:journalctl@ceph.mds.i.smithi177.stdout:Mar 28 00:54:19 smithi177 systemd[1]: Starting Ceph mds.i for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T00:54:20.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:20.205+0000 7f18d09c6640 1 -- 172.21.15.169:0/3758489230 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18cc02d070 con 0x7f18cc1011f0 2024-03-28T00:54:20.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:20.205+0000 7f18bb7fe640 1 -- 172.21.15.169:0/3758489230 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6886 (secure 0 0 0) 0x7f18c009b780 con 0x7f18cc1011f0 2024-03-28T00:54:20.208 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:20.209 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24452,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-28T00:54:20.209 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 9 2024-03-28T00:54:20.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:20.207+0000 7f18d09c6640 1 -- 172.21.15.169:0/3758489230 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f18b40a2800 msgr2=0x7f18b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:20.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:20.207+0000 7f18d09c6640 1 --2- 172.21.15.169:0/3758489230 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f18b40a2800 0x7f18b40a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f18cc069b30 tx=0x7f18bc005f90 comp rx=0 tx=0).stop 2024-03-28T00:54:20.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:20.207+0000 7f18d09c6640 1 -- 172.21.15.169:0/3758489230 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18cc1011f0 msgr2=0x7f18cc196d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:20.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:20.207+0000 7f18d09c6640 1 --2- 172.21.15.169:0/3758489230 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18cc1011f0 0x7f18cc196d30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f18c000c970 tx=0x7f18c000ce30 comp rx=0 tx=0).stop 2024-03-28T00:54:20.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:20.207+0000 7f18d09c6640 1 -- 172.21.15.169:0/3758489230 shutdown_connections 2024-03-28T00:54:20.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:20.207+0000 7f18d09c6640 1 --2- 172.21.15.169:0/3758489230 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f18b40a2800 0x7f18b40a4cc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:20.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:20.207+0000 7f18d09c6640 1 --2- 172.21.15.169:0/3758489230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18cc101be0 0x7f18cc19dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:20.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:20.207+0000 7f18d09c6640 1 --2- 172.21.15.169:0/3758489230 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18cc1011f0 0x7f18cc196d30 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:20.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:20.207+0000 7f18d09c6640 1 --2- 172.21.15.169:0/3758489230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18cc100830 0x7f18cc1967f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:20.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:20.207+0000 7f18d09c6640 1 -- 172.21.15.169:0/3758489230 >> 172.21.15.169:0/3758489230 conn(0x7f18cc0fb2c0 msgr2=0x7f18cc104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:20.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:20.207+0000 7f18d09c6640 1 -- 172.21.15.169:0/3758489230 shutdown_connections 2024-03-28T00:54:20.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:20.207+0000 7f18d09c6640 1 -- 172.21.15.169:0/3758489230 wait complete. 2024-03-28T00:54:20.386 INFO:journalctl@ceph.mds.i.smithi177.stdout:Mar 28 00:54:20 smithi177 podman[67661]: 2024-03-28 00:54:20.019913254 +0000 UTC m=+0.017582730 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T00:54:20.386 INFO:journalctl@ceph.mds.i.smithi177.stdout:Mar 28 00:54:20 smithi177 podman[67661]: 2024-03-28 00:54:20.131114295 +0000 UTC m=+0.128783771 container create 6863e73af0f029d5fb6b74fe8f227e93374d09a5a8cc8428558426824e5d4f49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True) 2024-03-28T00:54:20.386 INFO:journalctl@ceph.mds.i.smithi177.stdout:Mar 28 00:54:20 smithi177 podman[67661]: 2024-03-28 00:54:20.359025596 +0000 UTC m=+0.356695064 container init 6863e73af0f029d5fb6b74fe8f227e93374d09a5a8cc8428558426824e5d4f49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-i, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-28T00:54:20.386 INFO:journalctl@ceph.mds.i.smithi177.stdout:Mar 28 00:54:20 smithi177 podman[67661]: 2024-03-28 00:54:20.361666136 +0000 UTC m=+0.359335604 container start 6863e73af0f029d5fb6b74fe8f227e93374d09a5a8cc8428558426824e5d4f49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-i, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-28T00:54:20.666 INFO:journalctl@ceph.mds.i.smithi177.stdout:Mar 28 00:54:20 smithi177 ceph-mds[67680]: -- 172.21.15.177:0/230078808 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5624abd5fa40 con 0x5624abd32800 2024-03-28T00:54:20.667 INFO:journalctl@ceph.mds.i.smithi177.stdout:Mar 28 00:54:20 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-i[67676]: starting mds.i at 2024-03-28T00:54:20.667 INFO:journalctl@ceph.mds.i.smithi177.stdout:Mar 28 00:54:20 smithi177 bash[67661]: 6863e73af0f029d5fb6b74fe8f227e93374d09a5a8cc8428558426824e5d4f49 2024-03-28T00:54:20.667 INFO:journalctl@ceph.mds.i.smithi177.stdout:Mar 28 00:54:20 smithi177 systemd[1]: Started Ceph mds.i for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T00:54:21.413 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:21 smithi169 ceph-mon[32274]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:21.473 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:21 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3758489230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:21.473 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:21 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:21.473 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:21 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:21.474 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:21 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:21.474 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:21 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:21.474 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:21 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:21.474 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:21 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:21.474 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:21 smithi177 ceph-mon[36206]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:21.474 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:21 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3758489230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:21.475 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:21 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:21.475 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:21 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:21.475 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:21 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:21.475 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:21 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:21.475 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:21 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:21.475 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:21 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:21.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:21 smithi174 ceph-mon[36060]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:21.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:21 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3758489230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:21.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:21 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:21.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:21 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:21.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:21 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:21.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:21 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:21.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:21 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:21.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:21 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:22.222 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:54:22.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.329+0000 7fe58b297640 1 -- 172.21.15.169:0/1326250582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe584103270 msgr2=0x7fe584109b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:22.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.329+0000 7fe58b297640 1 --2- 172.21.15.169:0/1326250582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe584103270 0x7fe584109b00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe57c009ca0 tx=0x7fe57c02c0f0 comp rx=0 tx=0).stop 2024-03-28T00:54:22.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.329+0000 7fe58b297640 1 -- 172.21.15.169:0/1326250582 shutdown_connections 2024-03-28T00:54:22.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.329+0000 7fe58b297640 1 --2- 172.21.15.169:0/1326250582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe584103270 0x7fe584109b00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:22.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.329+0000 7fe58b297640 1 --2- 172.21.15.169:0/1326250582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5841028b0 0x7fe584102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:22.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.329+0000 7fe58b297640 1 --2- 172.21.15.169:0/1326250582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5841016c0 0x7fe584101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:22.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.329+0000 7fe58b297640 1 -- 172.21.15.169:0/1326250582 >> 172.21.15.169:0/1326250582 conn(0x7fe5840fcc50 msgr2=0x7fe5840ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:22.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.329+0000 7fe58b297640 1 -- 172.21.15.169:0/1326250582 shutdown_connections 2024-03-28T00:54:22.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.329+0000 7fe58b297640 1 -- 172.21.15.169:0/1326250582 wait complete. 2024-03-28T00:54:22.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.330+0000 7fe58b297640 1 Processor -- start 2024-03-28T00:54:22.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.330+0000 7fe58b297640 1 -- start start 2024-03-28T00:54:22.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.330+0000 7fe58b297640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5841016c0 0x7fe5841979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:22.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.330+0000 7fe58b297640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5841028b0 0x7fe584197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:22.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.330+0000 7fe58b297640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe584103270 0x7fe58419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:22.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.330+0000 7fe58b297640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58410d3e0 con 0x7fe5841028b0 2024-03-28T00:54:22.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.330+0000 7fe58b297640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58410d260 con 0x7fe5841016c0 2024-03-28T00:54:22.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.330+0000 7fe58b297640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58410d560 con 0x7fe584103270 2024-03-28T00:54:22.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.330+0000 7fe588832640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5841028b0 0x7fe584197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:22.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.330+0000 7fe588832640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5841028b0 0x7fe584197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51348/0 (socket says 172.21.15.169:51348) 2024-03-28T00:54:22.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.330+0000 7fe588832640 1 -- 172.21.15.169:0/3843627227 learned_addr learned my addr 172.21.15.169:0/3843627227 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:22.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.330+0000 7fe589033640 1 --2- 172.21.15.169:0/3843627227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5841016c0 0x7fe5841979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:22.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.330+0000 7fe588832640 1 -- 172.21.15.169:0/3843627227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe584103270 msgr2=0x7fe58419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:22.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.330+0000 7fe588832640 1 --2- 172.21.15.169:0/3843627227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe584103270 0x7fe58419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:22.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.330+0000 7fe588832640 1 -- 172.21.15.169:0/3843627227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5841016c0 msgr2=0x7fe5841979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:22.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.330+0000 7fe588832640 1 --2- 172.21.15.169:0/3843627227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5841016c0 0x7fe5841979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:22.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.330+0000 7fe588832640 1 -- 172.21.15.169:0/3843627227 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58419f610 con 0x7fe5841028b0 2024-03-28T00:54:22.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.331+0000 7fe588832640 1 --2- 172.21.15.169:0/3843627227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5841028b0 0x7fe584197f30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe57400c970 tx=0x7fe57400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:22.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.331+0000 7fe57a7fc640 1 -- 172.21.15.169:0/3843627227 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe574007bb0 con 0x7fe5841028b0 2024-03-28T00:54:22.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.331+0000 7fe57a7fc640 1 -- 172.21.15.169:0/3843627227 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe574007d50 con 0x7fe5841028b0 2024-03-28T00:54:22.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.331+0000 7fe58b297640 1 -- 172.21.15.169:0/3843627227 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe58419f900 con 0x7fe5841028b0 2024-03-28T00:54:22.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.331+0000 7fe57a7fc640 1 -- 172.21.15.169:0/3843627227 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57400f450 con 0x7fe5841028b0 2024-03-28T00:54:22.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.331+0000 7fe58b297640 1 -- 172.21.15.169:0/3843627227 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe58419fe00 con 0x7fe5841028b0 2024-03-28T00:54:22.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.332+0000 7fe58b297640 1 -- 172.21.15.169:0/3843627227 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe584108980 con 0x7fe5841028b0 2024-03-28T00:54:22.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.333+0000 7fe57a7fc640 1 -- 172.21.15.169:0/3843627227 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe574005020 con 0x7fe5841028b0 2024-03-28T00:54:22.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.333+0000 7fe57a7fc640 1 --2- 172.21.15.169:0/3843627227 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe5700a2800 0x7fe5700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:22.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.333+0000 7fe589033640 1 --2- 172.21.15.169:0/3843627227 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe5700a2800 0x7fe5700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:22.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.333+0000 7fe57a7fc640 1 -- 172.21.15.169:0/3843627227 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe5740ce440 con 0x7fe5841028b0 2024-03-28T00:54:22.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.334+0000 7fe589033640 1 --2- 172.21.15.169:0/3843627227 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe5700a2800 0x7fe5700a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe584102710 tx=0x7fe580008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:22.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.338+0000 7fe57a7fc640 1 -- 172.21.15.169:0/3843627227 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe574014030 con 0x7fe5841028b0 2024-03-28T00:54:22.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:22 smithi177 ceph-mon[36206]: Deploying daemon mds.g on smithi169 2024-03-28T00:54:22.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:22 smithi177 ceph-mon[36206]: mds.? [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] up:boot 2024-03-28T00:54:22.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:22 smithi177 ceph-mon[36206]: fsmap 9 up:standby 2024-03-28T00:54:22.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:22 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T00:54:22.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:22 smithi169 ceph-mon[32274]: Deploying daemon mds.g on smithi169 2024-03-28T00:54:22.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:22 smithi169 ceph-mon[32274]: mds.? [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] up:boot 2024-03-28T00:54:22.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:22 smithi169 ceph-mon[32274]: fsmap 9 up:standby 2024-03-28T00:54:22.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:22 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T00:54:22.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.575+0000 7fe58b297640 1 -- 172.21.15.169:0/3843627227 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe58402d220 con 0x7fe5841028b0 2024-03-28T00:54:22.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.575+0000 7fe57a7fc640 1 -- 172.21.15.169:0/3843627227 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7679 (secure 0 0 0) 0x7fe57409b980 con 0x7fe5841028b0 2024-03-28T00:54:22.579 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:22.579 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24452,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T00:54:22.579 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 10 2024-03-28T00:54:22.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.577+0000 7fe58b297640 1 -- 172.21.15.169:0/3843627227 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe5700a2800 msgr2=0x7fe5700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:22.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.577+0000 7fe58b297640 1 --2- 172.21.15.169:0/3843627227 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe5700a2800 0x7fe5700a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe584102710 tx=0x7fe580008040 comp rx=0 tx=0).stop 2024-03-28T00:54:22.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.577+0000 7fe58b297640 1 -- 172.21.15.169:0/3843627227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5841028b0 msgr2=0x7fe584197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:22.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.577+0000 7fe58b297640 1 --2- 172.21.15.169:0/3843627227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5841028b0 0x7fe584197f30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe57400c970 tx=0x7fe57400ce30 comp rx=0 tx=0).stop 2024-03-28T00:54:22.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.578+0000 7fe58b297640 1 -- 172.21.15.169:0/3843627227 shutdown_connections 2024-03-28T00:54:22.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.578+0000 7fe58b297640 1 --2- 172.21.15.169:0/3843627227 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe5700a2800 0x7fe5700a4cc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:22.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.578+0000 7fe58b297640 1 --2- 172.21.15.169:0/3843627227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe584103270 0x7fe58419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:22.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.578+0000 7fe58b297640 1 --2- 172.21.15.169:0/3843627227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5841028b0 0x7fe584197f30 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:22.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.578+0000 7fe58b297640 1 --2- 172.21.15.169:0/3843627227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5841016c0 0x7fe5841979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:22.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.578+0000 7fe58b297640 1 -- 172.21.15.169:0/3843627227 >> 172.21.15.169:0/3843627227 conn(0x7fe5840fcc50 msgr2=0x7fe584105b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:22.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.578+0000 7fe58b297640 1 -- 172.21.15.169:0/3843627227 shutdown_connections 2024-03-28T00:54:22.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:22.578+0000 7fe58b297640 1 -- 172.21.15.169:0/3843627227 wait complete. 2024-03-28T00:54:22.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:22 smithi174 ceph-mon[36060]: Deploying daemon mds.g on smithi169 2024-03-28T00:54:22.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:22 smithi174 ceph-mon[36060]: mds.? [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] up:boot 2024-03-28T00:54:22.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:22 smithi174 ceph-mon[36060]: fsmap 9 up:standby 2024-03-28T00:54:22.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:22 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T00:54:22.961 INFO:journalctl@ceph.mds.g.smithi169.stdout:Mar 28 00:54:22 smithi169 systemd[1]: Starting Ceph mds.g for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T00:54:23.241 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:23 smithi169 ceph-mon[32274]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:23.241 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:23 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3843627227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:23.242 INFO:journalctl@ceph.mds.g.smithi169.stdout:Mar 28 00:54:22 smithi169 podman[71866]: 2024-03-28 00:54:22.861627268 +0000 UTC m=+0.018130129 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T00:54:23.242 INFO:journalctl@ceph.mds.g.smithi169.stdout:Mar 28 00:54:22 smithi169 podman[71866]: 2024-03-28 00:54:22.978101718 +0000 UTC m=+0.134604621 container create 73be7983f9dae61f75fbb29a7c5b8ac728b372731674f786a4e7618d202dbe60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-g, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0) 2024-03-28T00:54:23.242 INFO:journalctl@ceph.mds.g.smithi169.stdout:Mar 28 00:54:23 smithi169 podman[71866]: 2024-03-28 00:54:23.239492423 +0000 UTC m=+0.395995284 container init 73be7983f9dae61f75fbb29a7c5b8ac728b372731674f786a4e7618d202dbe60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-g, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-28T00:54:23.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:23 smithi177 ceph-mon[36206]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:23.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:23 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3843627227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:23.551 INFO:journalctl@ceph.mds.g.smithi169.stdout:Mar 28 00:54:23 smithi169 podman[71866]: 2024-03-28 00:54:23.242652776 +0000 UTC m=+0.399155637 container start 73be7983f9dae61f75fbb29a7c5b8ac728b372731674f786a4e7618d202dbe60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-g, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-28T00:54:23.551 INFO:journalctl@ceph.mds.g.smithi169.stdout:Mar 28 00:54:23 smithi169 ceph-mds[71886]: -- 172.21.15.169:0/1690928081 <== mon.0 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e683b29a40 con 0x55e683afc800 2024-03-28T00:54:23.551 INFO:journalctl@ceph.mds.g.smithi169.stdout:Mar 28 00:54:23 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-g[71882]: starting mds.g at 2024-03-28T00:54:23.552 INFO:journalctl@ceph.mds.g.smithi169.stdout:Mar 28 00:54:23 smithi169 bash[71866]: 73be7983f9dae61f75fbb29a7c5b8ac728b372731674f786a4e7618d202dbe60 2024-03-28T00:54:23.552 INFO:journalctl@ceph.mds.g.smithi169.stdout:Mar 28 00:54:23 smithi169 systemd[1]: Started Ceph mds.g for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T00:54:23.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:23 smithi174 ceph-mon[36060]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:23.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:23 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3843627227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:24.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:24 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:24.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:24 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:24.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:24 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:24.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:24 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:24.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:24 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:24.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:24 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:24.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:24 smithi169 ceph-mon[32274]: Deploying daemon mds.a on smithi169 2024-03-28T00:54:24.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:24 smithi169 ceph-mon[32274]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:24.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:24 smithi169 ceph-mon[32274]: mds.? [v2:172.21.15.169:6838/4072876730,v1:172.21.15.169:6839/4072876730] up:boot 2024-03-28T00:54:24.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:24 smithi169 ceph-mon[32274]: fsmap 10 up:standby 2024-03-28T00:54:24.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:24 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T00:54:24.596 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:54:24.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:24 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:24.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:24 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:24.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:24 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:24.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:24 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:24.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:24 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:24.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:24 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:24.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:24 smithi177 ceph-mon[36206]: Deploying daemon mds.a on smithi169 2024-03-28T00:54:24.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:24 smithi177 ceph-mon[36206]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:24.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:24 smithi177 ceph-mon[36206]: mds.? [v2:172.21.15.169:6838/4072876730,v1:172.21.15.169:6839/4072876730] up:boot 2024-03-28T00:54:24.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:24 smithi177 ceph-mon[36206]: fsmap 10 up:standby 2024-03-28T00:54:24.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:24 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T00:54:24.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.700+0000 7f7bba145640 1 -- 172.21.15.169:0/2027790982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bb4103270 msgr2=0x7f7bb4109b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:24.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.700+0000 7f7bba145640 1 --2- 172.21.15.169:0/2027790982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bb4103270 0x7f7bb4109b00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7bac009ca0 tx=0x7f7bac02c0f0 comp rx=0 tx=0).stop 2024-03-28T00:54:24.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.700+0000 7f7bba145640 1 -- 172.21.15.169:0/2027790982 shutdown_connections 2024-03-28T00:54:24.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.700+0000 7f7bba145640 1 --2- 172.21.15.169:0/2027790982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bb4103270 0x7f7bb4109b00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:24.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.700+0000 7f7bba145640 1 --2- 172.21.15.169:0/2027790982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bb41028b0 0x7f7bb4102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:24.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.700+0000 7f7bba145640 1 --2- 172.21.15.169:0/2027790982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bb41016c0 0x7f7bb4101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:24.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.700+0000 7f7bba145640 1 -- 172.21.15.169:0/2027790982 >> 172.21.15.169:0/2027790982 conn(0x7f7bb40fcc50 msgr2=0x7f7bb40ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:24.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.700+0000 7f7bba145640 1 -- 172.21.15.169:0/2027790982 shutdown_connections 2024-03-28T00:54:24.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.700+0000 7f7bba145640 1 -- 172.21.15.169:0/2027790982 wait complete. 2024-03-28T00:54:24.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.700+0000 7f7bba145640 1 Processor -- start 2024-03-28T00:54:24.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.700+0000 7f7bba145640 1 -- start start 2024-03-28T00:54:24.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.701+0000 7f7bba145640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bb41016c0 0x7f7bb41979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:24.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.701+0000 7f7bba145640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bb41028b0 0x7f7bb4197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:24.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.701+0000 7f7bba145640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bb4103270 0x7f7bb419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:24.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.701+0000 7f7bba145640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb410d3a0 con 0x7f7bb41028b0 2024-03-28T00:54:24.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.701+0000 7f7bba145640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb410d220 con 0x7f7bb4103270 2024-03-28T00:54:24.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.701+0000 7f7bba145640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb410d520 con 0x7f7bb41016c0 2024-03-28T00:54:24.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.701+0000 7f7baaffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bb41028b0 0x7f7bb4197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:24.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.701+0000 7f7baaffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bb41028b0 0x7f7bb4197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51376/0 (socket says 172.21.15.169:51376) 2024-03-28T00:54:24.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.701+0000 7f7baaffd640 1 -- 172.21.15.169:0/4266264638 learned_addr learned my addr 172.21.15.169:0/4266264638 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:24.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.701+0000 7f7bb3fff640 1 --2- 172.21.15.169:0/4266264638 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bb4103270 0x7f7bb419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:24.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.701+0000 7f7baaffd640 1 -- 172.21.15.169:0/4266264638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bb41016c0 msgr2=0x7f7bb41979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:24.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.701+0000 7f7bb37fe640 1 --2- 172.21.15.169:0/4266264638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bb41016c0 0x7f7bb41979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:24.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.701+0000 7f7baaffd640 1 --2- 172.21.15.169:0/4266264638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bb41016c0 0x7f7bb41979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:24.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.701+0000 7f7baaffd640 1 -- 172.21.15.169:0/4266264638 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bb4103270 msgr2=0x7f7bb419ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:24.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.701+0000 7f7baaffd640 1 --2- 172.21.15.169:0/4266264638 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bb4103270 0x7f7bb419ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:24.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.701+0000 7f7baaffd640 1 -- 172.21.15.169:0/4266264638 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb419f5d0 con 0x7f7bb41028b0 2024-03-28T00:54:24.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.701+0000 7f7baaffd640 1 --2- 172.21.15.169:0/4266264638 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bb41028b0 0x7f7bb4197ef0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c00b4a0 tx=0x7f7b9c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:24.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.701+0000 7f7bb17fa640 1 -- 172.21.15.169:0/4266264638 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c014070 con 0x7f7bb41028b0 2024-03-28T00:54:24.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.701+0000 7f7bb17fa640 1 -- 172.21.15.169:0/4266264638 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b9c010af0 con 0x7f7bb41028b0 2024-03-28T00:54:24.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.701+0000 7f7bba145640 1 -- 172.21.15.169:0/4266264638 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb419f8c0 con 0x7f7bb41028b0 2024-03-28T00:54:24.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.702+0000 7f7bb17fa640 1 -- 172.21.15.169:0/4266264638 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c017d10 con 0x7f7bb41028b0 2024-03-28T00:54:24.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.702+0000 7f7bba145640 1 -- 172.21.15.169:0/4266264638 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb419fdc0 con 0x7f7bb41028b0 2024-03-28T00:54:24.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.702+0000 7f7bba145640 1 -- 172.21.15.169:0/4266264638 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb41089e0 con 0x7f7bb41028b0 2024-03-28T00:54:24.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.704+0000 7f7bb17fa640 1 -- 172.21.15.169:0/4266264638 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7b9c017070 con 0x7f7bb41028b0 2024-03-28T00:54:24.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.705+0000 7f7bb17fa640 1 --2- 172.21.15.169:0/4266264638 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7ba00a2800 0x7f7ba00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:24.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.705+0000 7f7bb37fe640 1 --2- 172.21.15.169:0/4266264638 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7ba00a2800 0x7f7ba00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:24.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.705+0000 7f7bb17fa640 1 -- 172.21.15.169:0/4266264638 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7b9c0cde70 con 0x7f7bb41028b0 2024-03-28T00:54:24.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.705+0000 7f7bb37fe640 1 --2- 172.21.15.169:0/4266264638 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7ba00a2800 0x7f7ba00a4cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4005fd0 tx=0x7f7ba4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:24.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.708+0000 7f7bb17fa640 1 -- 172.21.15.169:0/4266264638 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7b9c09b310 con 0x7f7bb41028b0 2024-03-28T00:54:24.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:24 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:24.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:24 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:24.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:24 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:24.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:24 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:24.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:24 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:24.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:24 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:24.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:24 smithi174 ceph-mon[36060]: Deploying daemon mds.a on smithi169 2024-03-28T00:54:24.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:24 smithi174 ceph-mon[36060]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:24.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:24 smithi174 ceph-mon[36060]: mds.? [v2:172.21.15.169:6838/4072876730,v1:172.21.15.169:6839/4072876730] up:boot 2024-03-28T00:54:24.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:24 smithi174 ceph-mon[36060]: fsmap 10 up:standby 2024-03-28T00:54:24.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:24 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T00:54:24.926 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.924+0000 7f7bba145640 1 -- 172.21.15.169:0/4266264638 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bb402d070 con 0x7f7bb41028b0 2024-03-28T00:54:24.927 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.924+0000 7f7bb17fa640 1 -- 172.21.15.169:0/4266264638 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8474 (secure 0 0 0) 0x7f7b9c09ac10 con 0x7f7bb41028b0 2024-03-28T00:54:24.927 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:24.927 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24452,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T00:54:24.928 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 11 2024-03-28T00:54:24.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.926+0000 7f7bba145640 1 -- 172.21.15.169:0/4266264638 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7ba00a2800 msgr2=0x7f7ba00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:24.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.926+0000 7f7bba145640 1 --2- 172.21.15.169:0/4266264638 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7ba00a2800 0x7f7ba00a4cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4005fd0 tx=0x7f7ba4005850 comp rx=0 tx=0).stop 2024-03-28T00:54:24.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.926+0000 7f7bba145640 1 -- 172.21.15.169:0/4266264638 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bb41028b0 msgr2=0x7f7bb4197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:24.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.926+0000 7f7bba145640 1 --2- 172.21.15.169:0/4266264638 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bb41028b0 0x7f7bb4197ef0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c00b4a0 tx=0x7f7b9c00b960 comp rx=0 tx=0).stop 2024-03-28T00:54:24.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.926+0000 7f7bba145640 1 -- 172.21.15.169:0/4266264638 shutdown_connections 2024-03-28T00:54:24.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.926+0000 7f7bba145640 1 --2- 172.21.15.169:0/4266264638 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7ba00a2800 0x7f7ba00a4cc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:24.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.926+0000 7f7bba145640 1 --2- 172.21.15.169:0/4266264638 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bb4103270 0x7f7bb419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:24.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.926+0000 7f7bba145640 1 --2- 172.21.15.169:0/4266264638 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7bb41028b0 0x7f7bb4197ef0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:24.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.926+0000 7f7bba145640 1 --2- 172.21.15.169:0/4266264638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bb41016c0 0x7f7bb41979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:24.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.926+0000 7f7bba145640 1 -- 172.21.15.169:0/4266264638 >> 172.21.15.169:0/4266264638 conn(0x7f7bb40fcc50 msgr2=0x7f7bb4105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:24.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.927+0000 7f7bba145640 1 -- 172.21.15.169:0/4266264638 shutdown_connections 2024-03-28T00:54:24.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:24.927+0000 7f7bba145640 1 -- 172.21.15.169:0/4266264638 wait complete. 2024-03-28T00:54:25.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:25 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4266264638' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:25.552 INFO:journalctl@ceph.mds.a.smithi169.stdout:Mar 28 00:54:25 smithi169 systemd[1]: Starting Ceph mds.a for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T00:54:25.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:25 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/4266264638' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:25.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:25 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/4266264638' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:25.949 INFO:journalctl@ceph.mds.a.smithi169.stdout:Mar 28 00:54:25 smithi169 podman[72212]: 2024-03-28 00:54:25.594385071 +0000 UTC m=+0.017648426 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T00:54:25.949 INFO:journalctl@ceph.mds.a.smithi169.stdout:Mar 28 00:54:25 smithi169 podman[72212]: 2024-03-28 00:54:25.700867186 +0000 UTC m=+0.124130525 container create d4b5685df0c66548a8fa3e1a36ddd7c036aabb86f460efb60bac56d364db0bf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-a, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-28T00:54:25.949 INFO:journalctl@ceph.mds.a.smithi169.stdout:Mar 28 00:54:25 smithi169 podman[72212]: 2024-03-28 00:54:25.920627374 +0000 UTC m=+0.343890711 container init d4b5685df0c66548a8fa3e1a36ddd7c036aabb86f460efb60bac56d364db0bf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-28T00:54:25.949 INFO:journalctl@ceph.mds.a.smithi169.stdout:Mar 28 00:54:25 smithi169 podman[72212]: 2024-03-28 00:54:25.923487027 +0000 UTC m=+0.346750367 container start d4b5685df0c66548a8fa3e1a36ddd7c036aabb86f460efb60bac56d364db0bf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-a, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-28T00:54:25.949 INFO:journalctl@ceph.mds.a.smithi169.stdout:Mar 28 00:54:25 smithi169 ceph-mds[72231]: -- 172.21.15.169:0/1055272434 <== mon.0 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a8cb967a40 con 0x55a8cb93b000 2024-03-28T00:54:26.301 INFO:journalctl@ceph.mds.a.smithi169.stdout:Mar 28 00:54:25 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-a[72227]: starting mds.a at 2024-03-28T00:54:26.301 INFO:journalctl@ceph.mds.a.smithi169.stdout:Mar 28 00:54:26 smithi169 bash[72212]: d4b5685df0c66548a8fa3e1a36ddd7c036aabb86f460efb60bac56d364db0bf2 2024-03-28T00:54:26.301 INFO:journalctl@ceph.mds.a.smithi169.stdout:Mar 28 00:54:26 smithi169 systemd[1]: Started Ceph mds.a for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T00:54:26.941 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:54:27.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.057+0000 7fbd0ee0b640 1 -- 172.21.15.169:0/548535498 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd081017b0 msgr2=0x7fbd08101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:27.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.057+0000 7fbd0ee0b640 1 --2- 172.21.15.169:0/548535498 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd081017b0 0x7fbd08101c30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc009900 tx=0x7fbcfc02bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:27.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.057+0000 7fbd0ee0b640 1 -- 172.21.15.169:0/548535498 shutdown_connections 2024-03-28T00:54:27.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.057+0000 7fbd0ee0b640 1 --2- 172.21.15.169:0/548535498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd08102170 0x7fbd08108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:27.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.057+0000 7fbd0ee0b640 1 --2- 172.21.15.169:0/548535498 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd081017b0 0x7fbd08101c30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:27.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.057+0000 7fbd0ee0b640 1 --2- 172.21.15.169:0/548535498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd081005c0 0x7fbd081009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:27.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.057+0000 7fbd0ee0b640 1 -- 172.21.15.169:0/548535498 >> 172.21.15.169:0/548535498 conn(0x7fbd080fbb30 msgr2=0x7fbd080fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:27.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.057+0000 7fbd0ee0b640 1 -- 172.21.15.169:0/548535498 shutdown_connections 2024-03-28T00:54:27.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.057+0000 7fbd0ee0b640 1 -- 172.21.15.169:0/548535498 wait complete. 2024-03-28T00:54:27.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.057+0000 7fbd0ee0b640 1 Processor -- start 2024-03-28T00:54:27.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.057+0000 7fbd0ee0b640 1 -- start start 2024-03-28T00:54:27.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd0ee0b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd081005c0 0x7fbd080693d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:27.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd0ee0b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd081017b0 0x7fbd0819c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:27.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd0ee0b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd08102170 0x7fbd0819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:27.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd0ee0b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0810c2d0 con 0x7fbd08102170 2024-03-28T00:54:27.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd0ee0b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0810c150 con 0x7fbd081005c0 2024-03-28T00:54:27.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd0ee0b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0810c450 con 0x7fbd081017b0 2024-03-28T00:54:27.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd0d3a8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd08102170 0x7fbd0819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:27.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd0d3a8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd08102170 0x7fbd0819edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33822/0 (socket says 172.21.15.169:33822) 2024-03-28T00:54:27.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd0d3a8640 1 -- 172.21.15.169:0/3077831877 learned_addr learned my addr 172.21.15.169:0/3077831877 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:27.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd0cba7640 1 --2- 172.21.15.169:0/3077831877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd081005c0 0x7fbd080693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:27.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd0d3a8640 1 -- 172.21.15.169:0/3077831877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd081017b0 msgr2=0x7fbd0819c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:27.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd07fff640 1 --2- 172.21.15.169:0/3077831877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd081017b0 0x7fbd0819c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:27.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd0d3a8640 1 --2- 172.21.15.169:0/3077831877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd081017b0 0x7fbd0819c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:27.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd0d3a8640 1 -- 172.21.15.169:0/3077831877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd081005c0 msgr2=0x7fbd080693d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:27.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd0d3a8640 1 --2- 172.21.15.169:0/3077831877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd081005c0 0x7fbd080693d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:27.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd0d3a8640 1 -- 172.21.15.169:0/3077831877 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0819f690 con 0x7fbd08102170 2024-03-28T00:54:27.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd0cba7640 1 --2- 172.21.15.169:0/3077831877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd081005c0 0x7fbd080693d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:54:27.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd0d3a8640 1 --2- 172.21.15.169:0/3077831877 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd08102170 0x7fbd0819edd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbd0000eaa0 tx=0x7fbd0000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:27.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd05ffb640 1 -- 172.21.15.169:0/3077831877 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0000cdb0 con 0x7fbd08102170 2024-03-28T00:54:27.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd05ffb640 1 -- 172.21.15.169:0/3077831877 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd00005020 con 0x7fbd08102170 2024-03-28T00:54:27.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd05ffb640 1 -- 172.21.15.169:0/3077831877 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd000116b0 con 0x7fbd08102170 2024-03-28T00:54:27.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.058+0000 7fbd0ee0b640 1 -- 172.21.15.169:0/3077831877 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd0819f980 con 0x7fbd08102170 2024-03-28T00:54:27.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.059+0000 7fbd0ee0b640 1 -- 172.21.15.169:0/3077831877 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd0819fe80 con 0x7fbd08102170 2024-03-28T00:54:27.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.060+0000 7fbd0ee0b640 1 -- 172.21.15.169:0/3077831877 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd081078e0 con 0x7fbd08102170 2024-03-28T00:54:27.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.061+0000 7fbd05ffb640 1 -- 172.21.15.169:0/3077831877 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbd00005560 con 0x7fbd08102170 2024-03-28T00:54:27.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.061+0000 7fbd05ffb640 1 --2- 172.21.15.169:0/3077831877 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbcf00a2880 0x7fbcf00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:27.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.061+0000 7fbd0cba7640 1 --2- 172.21.15.169:0/3077831877 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbcf00a2880 0x7fbcf00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:27.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.061+0000 7fbd05ffb640 1 -- 172.21.15.169:0/3077831877 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbd000ce810 con 0x7fbd08102170 2024-03-28T00:54:27.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.062+0000 7fbd0cba7640 1 --2- 172.21.15.169:0/3077831877 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbcf00a2880 0x7fbcf00a4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8006fd0 tx=0x7fbcf8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:27.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.066+0000 7fbd05ffb640 1 -- 172.21.15.169:0/3077831877 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbd00016030 con 0x7fbd08102170 2024-03-28T00:54:27.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:27 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:27.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:27 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:27.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:27 smithi174 ceph-mon[36060]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:27.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:27 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:27.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:27 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:27.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:27 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:27.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:27 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:27.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:27 smithi174 ceph-mon[36060]: Deploying daemon mds.b on smithi174 2024-03-28T00:54:27.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:27 smithi174 ceph-mon[36060]: mds.? [v2:172.21.15.169:6840/2980464895,v1:172.21.15.169:6841/2980464895] up:boot 2024-03-28T00:54:27.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:27 smithi174 ceph-mon[36060]: fsmap 11 up:standby 2024-03-28T00:54:27.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:27 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T00:54:27.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:27 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:27.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:27 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:27.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:27 smithi177 ceph-mon[36206]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:27.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:27 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:27.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:27 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:27.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:27 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:27.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:27 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:27.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:27 smithi177 ceph-mon[36206]: Deploying daemon mds.b on smithi174 2024-03-28T00:54:27.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:27 smithi177 ceph-mon[36206]: mds.? [v2:172.21.15.169:6840/2980464895,v1:172.21.15.169:6841/2980464895] up:boot 2024-03-28T00:54:27.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:27 smithi177 ceph-mon[36206]: fsmap 11 up:standby 2024-03-28T00:54:27.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:27 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T00:54:27.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.280+0000 7fbd0ee0b640 1 -- 172.21.15.169:0/3077831877 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd0802d070 con 0x7fbd08102170 2024-03-28T00:54:27.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.280+0000 7fbd05ffb640 1 -- 172.21.15.169:0/3077831877 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9269 (secure 0 0 0) 0x7fbd0009bd50 con 0x7fbd08102170 2024-03-28T00:54:27.284 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:27.284 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24452,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T00:54:27.284 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 12 2024-03-28T00:54:27.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.282+0000 7fbd0ee0b640 1 -- 172.21.15.169:0/3077831877 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbcf00a2880 msgr2=0x7fbcf00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:27.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.282+0000 7fbd0ee0b640 1 --2- 172.21.15.169:0/3077831877 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbcf00a2880 0x7fbcf00a4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8006fd0 tx=0x7fbcf8008040 comp rx=0 tx=0).stop 2024-03-28T00:54:27.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.282+0000 7fbd0ee0b640 1 -- 172.21.15.169:0/3077831877 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd08102170 msgr2=0x7fbd0819edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:27.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.282+0000 7fbd0ee0b640 1 --2- 172.21.15.169:0/3077831877 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd08102170 0x7fbd0819edd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbd0000eaa0 tx=0x7fbd0000ef60 comp rx=0 tx=0).stop 2024-03-28T00:54:27.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.282+0000 7fbd0ee0b640 1 -- 172.21.15.169:0/3077831877 shutdown_connections 2024-03-28T00:54:27.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.282+0000 7fbd0ee0b640 1 --2- 172.21.15.169:0/3077831877 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbcf00a2880 0x7fbcf00a4d40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:27.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.282+0000 7fbd0ee0b640 1 --2- 172.21.15.169:0/3077831877 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd08102170 0x7fbd0819edd0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:27.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.282+0000 7fbd0ee0b640 1 --2- 172.21.15.169:0/3077831877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd081017b0 0x7fbd0819c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:27.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.282+0000 7fbd0ee0b640 1 --2- 172.21.15.169:0/3077831877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd081005c0 0x7fbd080693d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:27.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.282+0000 7fbd0ee0b640 1 -- 172.21.15.169:0/3077831877 >> 172.21.15.169:0/3077831877 conn(0x7fbd080fbb30 msgr2=0x7fbd08104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:27.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.283+0000 7fbd0ee0b640 1 -- 172.21.15.169:0/3077831877 shutdown_connections 2024-03-28T00:54:27.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:27.283+0000 7fbd0ee0b640 1 -- 172.21.15.169:0/3077831877 wait complete. 2024-03-28T00:54:27.295 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:27 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:27.295 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:27 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:27.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:27 smithi169 ceph-mon[32274]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:27.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:27 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:27.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:27 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T00:54:27.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:27 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T00:54:27.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:27 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:27.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:27 smithi169 ceph-mon[32274]: Deploying daemon mds.b on smithi174 2024-03-28T00:54:27.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:27 smithi169 ceph-mon[32274]: mds.? [v2:172.21.15.169:6840/2980464895,v1:172.21.15.169:6841/2980464895] up:boot 2024-03-28T00:54:27.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:27 smithi169 ceph-mon[32274]: fsmap 11 up:standby 2024-03-28T00:54:27.296 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:27 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T00:54:28.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:28 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3077831877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:28.358 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 00:54:28 smithi174 systemd[1]: Starting Ceph mds.b for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T00:54:28.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:28 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3077831877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:28.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:28 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3077831877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:28.752 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 00:54:28 smithi174 podman[66088]: 2024-03-28 00:54:28.392890503 +0000 UTC m=+0.018631325 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T00:54:28.753 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 00:54:28 smithi174 podman[66088]: 2024-03-28 00:54:28.522718779 +0000 UTC m=+0.148459586 container create 08f12830df86263c166a955fd1af29076d22527d64c31b952e2569d916a500f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-28T00:54:28.753 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 00:54:28 smithi174 podman[66088]: 2024-03-28 00:54:28.750246567 +0000 UTC m=+0.375987388 container init 08f12830df86263c166a955fd1af29076d22527d64c31b952e2569d916a500f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-03-28T00:54:29.011 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 00:54:28 smithi174 podman[66088]: 2024-03-28 00:54:28.752822232 +0000 UTC m=+0.378563040 container start 08f12830df86263c166a955fd1af29076d22527d64c31b952e2569d916a500f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T00:54:29.011 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 00:54:28 smithi174 ceph-mds[66107]: -- 172.21.15.174:0/853414242 <== mon.2 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560674993a40 con 0x560674966c00 2024-03-28T00:54:29.012 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 00:54:28 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-b[66103]: starting mds.b at 2024-03-28T00:54:29.298 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:54:29.358 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 00:54:29 smithi174 bash[66088]: 08f12830df86263c166a955fd1af29076d22527d64c31b952e2569d916a500f5 2024-03-28T00:54:29.358 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 00:54:29 smithi174 systemd[1]: Started Ceph mds.b for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T00:54:29.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:29 smithi174 ceph-mon[36060]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:29.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:29 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:29.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:29 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:29.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:29 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:29.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:29 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:29.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:29 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:29.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.407+0000 7f0979a6e640 1 -- 172.21.15.169:0/3709210056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0974102170 msgr2=0x7f0974108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:29.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.407+0000 7f0979a6e640 1 --2- 172.21.15.169:0/3709210056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0974102170 0x7f0974108a00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f096c009ca0 tx=0x7f096c02c0f0 comp rx=0 tx=0).stop 2024-03-28T00:54:29.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.407+0000 7f0979a6e640 1 -- 172.21.15.169:0/3709210056 shutdown_connections 2024-03-28T00:54:29.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.407+0000 7f0979a6e640 1 --2- 172.21.15.169:0/3709210056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0974102170 0x7f0974108a00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:29.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.407+0000 7f0979a6e640 1 --2- 172.21.15.169:0/3709210056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09741017b0 0x7f0974101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:29.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.407+0000 7f0979a6e640 1 --2- 172.21.15.169:0/3709210056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09741005c0 0x7f09741009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:29.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.407+0000 7f0979a6e640 1 -- 172.21.15.169:0/3709210056 >> 172.21.15.169:0/3709210056 conn(0x7f09740fbb30 msgr2=0x7f09740fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:29.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.407+0000 7f0979a6e640 1 -- 172.21.15.169:0/3709210056 shutdown_connections 2024-03-28T00:54:29.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.407+0000 7f0979a6e640 1 -- 172.21.15.169:0/3709210056 wait complete. 2024-03-28T00:54:29.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.407+0000 7f0979a6e640 1 Processor -- start 2024-03-28T00:54:29.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.408+0000 7f0979a6e640 1 -- start start 2024-03-28T00:54:29.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.408+0000 7f0979a6e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09741005c0 0x7f0974104550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:29.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.408+0000 7f0979a6e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09741017b0 0x7f0974102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:29.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.408+0000 7f0979a6e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0974102170 0x7f09741030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:29.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.408+0000 7f0979a6e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097410c240 con 0x7f0974102170 2024-03-28T00:54:29.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.408+0000 7f0979a6e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097410c0c0 con 0x7f09741005c0 2024-03-28T00:54:29.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.408+0000 7f0979a6e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097410c3c0 con 0x7f09741017b0 2024-03-28T00:54:29.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.408+0000 7f0973fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0974102170 0x7f09741030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:29.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.408+0000 7f0972ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09741017b0 0x7f0974102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:29.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.408+0000 7f0973fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0974102170 0x7f09741030d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33836/0 (socket says 172.21.15.169:33836) 2024-03-28T00:54:29.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.408+0000 7f0973fff640 1 -- 172.21.15.169:0/351543041 learned_addr learned my addr 172.21.15.169:0/351543041 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:29.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.408+0000 7f09737fe640 1 --2- 172.21.15.169:0/351543041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09741005c0 0x7f0974104550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:29.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.408+0000 7f0972ffd640 1 -- 172.21.15.169:0/351543041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09741005c0 msgr2=0x7f0974104550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:29.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.408+0000 7f0972ffd640 1 --2- 172.21.15.169:0/351543041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09741005c0 0x7f0974104550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:29.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.408+0000 7f0972ffd640 1 -- 172.21.15.169:0/351543041 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0974102170 msgr2=0x7f09741030d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:29.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.408+0000 7f0972ffd640 1 --2- 172.21.15.169:0/351543041 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0974102170 0x7f09741030d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:29.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.408+0000 7f0972ffd640 1 -- 172.21.15.169:0/351543041 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0974103960 con 0x7f09741017b0 2024-03-28T00:54:29.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.408+0000 7f0973fff640 1 --2- 172.21.15.169:0/351543041 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0974102170 0x7f09741030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:29.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.408+0000 7f09737fe640 1 --2- 172.21.15.169:0/351543041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09741005c0 0x7f0974104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:29.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.408+0000 7f0972ffd640 1 --2- 172.21.15.169:0/351543041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09741017b0 0x7f0974102b90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f096400e9f0 tx=0x7f096400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:29.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.409+0000 7f0970ff9640 1 -- 172.21.15.169:0/351543041 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096400ccc0 con 0x7f09741017b0 2024-03-28T00:54:29.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.409+0000 7f0979a6e640 1 -- 172.21.15.169:0/351543041 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09741a1ac0 con 0x7f09741017b0 2024-03-28T00:54:29.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.409+0000 7f0970ff9640 1 -- 172.21.15.169:0/351543041 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0964005020 con 0x7f09741017b0 2024-03-28T00:54:29.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.409+0000 7f0970ff9640 1 -- 172.21.15.169:0/351543041 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09640049e0 con 0x7f09741017b0 2024-03-28T00:54:29.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.409+0000 7f0979a6e640 1 -- 172.21.15.169:0/351543041 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09741a2010 con 0x7f09741017b0 2024-03-28T00:54:29.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.410+0000 7f0979a6e640 1 -- 172.21.15.169:0/351543041 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09741a2240 con 0x7f09741017b0 2024-03-28T00:54:29.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:29 smithi177 ceph-mon[36206]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:29.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:29 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:29.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:29 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:29.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:29 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:29.419 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:29 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:29.419 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:29 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:29.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.411+0000 7f0970ff9640 1 -- 172.21.15.169:0/351543041 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0964005560 con 0x7f09741017b0 2024-03-28T00:54:29.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.411+0000 7f0970ff9640 1 --2- 172.21.15.169:0/351543041 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f095c0a2800 0x7f095c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:29.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.411+0000 7f0970ff9640 1 -- 172.21.15.169:0/351543041 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f09640ce5e0 con 0x7f09741017b0 2024-03-28T00:54:29.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.411+0000 7f09737fe640 1 --2- 172.21.15.169:0/351543041 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f095c0a2800 0x7f095c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:29.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.412+0000 7f09737fe640 1 --2- 172.21.15.169:0/351543041 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f095c0a2800 0x7f095c0a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0974101610 tx=0x7f0968005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:29.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.416+0000 7f0970ff9640 1 -- 172.21.15.169:0/351543041 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f09640051d0 con 0x7f09741017b0 2024-03-28T00:54:29.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:29 smithi169 ceph-mon[32274]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:29.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:29 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:29.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:29 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:29.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:29 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:29.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:29 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:29.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:29 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:29.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.639+0000 7f0979a6e640 1 -- 172.21.15.169:0/351543041 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09741a24b0 con 0x7f09741017b0 2024-03-28T00:54:29.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.640+0000 7f0970ff9640 1 -- 172.21.15.169:0/351543041 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10064 (secure 0 0 0) 0x7f096409bb20 con 0x7f09741017b0 2024-03-28T00:54:29.643 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:29.643 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24452,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24479,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-28T00:54:29.644 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 13 2024-03-28T00:54:29.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.642+0000 7f0979a6e640 1 -- 172.21.15.169:0/351543041 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f095c0a2800 msgr2=0x7f095c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:29.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.642+0000 7f0979a6e640 1 --2- 172.21.15.169:0/351543041 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f095c0a2800 0x7f095c0a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0974101610 tx=0x7f0968005f90 comp rx=0 tx=0).stop 2024-03-28T00:54:29.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.642+0000 7f0979a6e640 1 -- 172.21.15.169:0/351543041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09741017b0 msgr2=0x7f0974102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:29.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.643+0000 7f0979a6e640 1 --2- 172.21.15.169:0/351543041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09741017b0 0x7f0974102b90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f096400e9f0 tx=0x7f096400eeb0 comp rx=0 tx=0).stop 2024-03-28T00:54:29.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.643+0000 7f0979a6e640 1 -- 172.21.15.169:0/351543041 shutdown_connections 2024-03-28T00:54:29.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.643+0000 7f0979a6e640 1 --2- 172.21.15.169:0/351543041 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f095c0a2800 0x7f095c0a4cc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:29.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.643+0000 7f0979a6e640 1 --2- 172.21.15.169:0/351543041 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0974102170 0x7f09741030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:29.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.643+0000 7f0979a6e640 1 --2- 172.21.15.169:0/351543041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09741017b0 0x7f0974102b90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:29.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.643+0000 7f0979a6e640 1 --2- 172.21.15.169:0/351543041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09741005c0 0x7f0974104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:29.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.643+0000 7f0979a6e640 1 -- 172.21.15.169:0/351543041 >> 172.21.15.169:0/351543041 conn(0x7f09740fbb30 msgr2=0x7f09740fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:29.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.643+0000 7f0979a6e640 1 -- 172.21.15.169:0/351543041 shutdown_connections 2024-03-28T00:54:29.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.643+0000 7f0979a6e640 1 -- 172.21.15.169:0/351543041 wait complete. 2024-03-28T00:54:29.657 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T00:54:29.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.760+0000 7ffbe257e640 1 -- 172.21.15.169:0/4253968648 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffbdc101c00 msgr2=0x7ffbdc102020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:29.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.760+0000 7ffbe257e640 1 --2- 172.21.15.169:0/4253968648 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffbdc101c00 0x7ffbdc102020 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ffbd0009900 tx=0x7ffbd002bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:29.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.760+0000 7ffbe257e640 1 -- 172.21.15.169:0/4253968648 shutdown_connections 2024-03-28T00:54:29.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.760+0000 7ffbe257e640 1 --2- 172.21.15.169:0/4253968648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbdc101210 0x7ffbdc108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:29.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.760+0000 7ffbe257e640 1 --2- 172.21.15.169:0/4253968648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbdc100850 0x7ffbdc100cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:29.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.760+0000 7ffbe257e640 1 --2- 172.21.15.169:0/4253968648 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffbdc101c00 0x7ffbdc102020 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:29.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.761+0000 7ffbe257e640 1 -- 172.21.15.169:0/4253968648 >> 172.21.15.169:0/4253968648 conn(0x7ffbdc0fb2e0 msgr2=0x7ffbdc0fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:29.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.761+0000 7ffbe257e640 1 -- 172.21.15.169:0/4253968648 shutdown_connections 2024-03-28T00:54:29.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.761+0000 7ffbe257e640 1 -- 172.21.15.169:0/4253968648 wait complete. 2024-03-28T00:54:29.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.761+0000 7ffbe257e640 1 Processor -- start 2024-03-28T00:54:29.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.761+0000 7ffbe257e640 1 -- start start 2024-03-28T00:54:29.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.761+0000 7ffbe257e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbdc100850 0x7ffbdc1002b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:29.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbe257e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbdc101210 0x7ffbdc19c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:29.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbe257e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffbdc101c00 0x7ffbdc19edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:29.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbe257e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbdc10c270 con 0x7ffbdc101c00 2024-03-28T00:54:29.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbe257e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbdc10c0f0 con 0x7ffbdc100850 2024-03-28T00:54:29.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbe257e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbdc10c3f0 con 0x7ffbdc101210 2024-03-28T00:54:29.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbe0b1b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffbdc101c00 0x7ffbdc19edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:29.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbe0b1b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffbdc101c00 0x7ffbdc19edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33850/0 (socket says 172.21.15.169:33850) 2024-03-28T00:54:29.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbdbfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbdc100850 0x7ffbdc1002b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:29.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbe0b1b640 1 -- 172.21.15.169:0/3524948296 learned_addr learned my addr 172.21.15.169:0/3524948296 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:29.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbdb7fe640 1 --2- 172.21.15.169:0/3524948296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbdc101210 0x7ffbdc19c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:29.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbdbfff640 1 -- 172.21.15.169:0/3524948296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbdc101210 msgr2=0x7ffbdc19c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:29.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbdbfff640 1 --2- 172.21.15.169:0/3524948296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbdc101210 0x7ffbdc19c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:29.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbdbfff640 1 -- 172.21.15.169:0/3524948296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffbdc101c00 msgr2=0x7ffbdc19edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:29.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbdbfff640 1 --2- 172.21.15.169:0/3524948296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffbdc101c00 0x7ffbdc19edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:29.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbdbfff640 1 -- 172.21.15.169:0/3524948296 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbdc19f6c0 con 0x7ffbdc100850 2024-03-28T00:54:29.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbe0b1b640 1 --2- 172.21.15.169:0/3524948296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffbdc101c00 0x7ffbdc19edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:29.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbdbfff640 1 --2- 172.21.15.169:0/3524948296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbdc100850 0x7ffbdc1002b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ffbd002c170 tx=0x7ffbd002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:29.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbd97fa640 1 -- 172.21.15.169:0/3524948296 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd0041070 con 0x7ffbdc100850 2024-03-28T00:54:29.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbd97fa640 1 -- 172.21.15.169:0/3524948296 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffbd00054e0 con 0x7ffbdc100850 2024-03-28T00:54:29.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbe257e640 1 -- 172.21.15.169:0/3524948296 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbdc19f8f0 con 0x7ffbdc100850 2024-03-28T00:54:29.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbd97fa640 1 -- 172.21.15.169:0/3524948296 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd0034440 con 0x7ffbdc100850 2024-03-28T00:54:29.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.762+0000 7ffbe257e640 1 -- 172.21.15.169:0/3524948296 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbdc19fdb0 con 0x7ffbdc100850 2024-03-28T00:54:29.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.763+0000 7ffbe257e640 1 -- 172.21.15.169:0/3524948296 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbdc107a60 con 0x7ffbdc100850 2024-03-28T00:54:29.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.764+0000 7ffbd97fa640 1 -- 172.21.15.169:0/3524948296 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ffbd0044070 con 0x7ffbdc100850 2024-03-28T00:54:29.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.765+0000 7ffbd97fa640 1 --2- 172.21.15.169:0/3524948296 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffbc40a2800 0x7ffbc40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:29.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.765+0000 7ffbdb7fe640 1 --2- 172.21.15.169:0/3524948296 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffbc40a2800 0x7ffbc40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:29.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.765+0000 7ffbd97fa640 1 -- 172.21.15.169:0/3524948296 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ffbd00f0250 con 0x7ffbdc100850 2024-03-28T00:54:29.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.766+0000 7ffbdb7fe640 1 --2- 172.21.15.169:0/3524948296 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffbc40a2800 0x7ffbc40a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc100020 tx=0x7ffbcc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:29.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.769+0000 7ffbd97fa640 1 -- 172.21.15.169:0/3524948296 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffbd00bd0a0 con 0x7ffbdc100850 2024-03-28T00:54:29.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.960+0000 7ffbe257e640 1 -- 172.21.15.169:0/3524948296 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffbdc02d240 con 0x7ffbdc100850 2024-03-28T00:54:29.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.961+0000 7ffbd97fa640 1 -- 172.21.15.169:0/3524948296 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18514 (secure 0 0 0) 0x7ffbd00c1140 con 0x7ffbdc100850 2024-03-28T00:54:29.965 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:29.965 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":81,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:53:44.212143+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:54:29.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.963+0000 7ffbe257e640 1 -- 172.21.15.169:0/3524948296 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffbc40a2800 msgr2=0x7ffbc40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:29.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.963+0000 7ffbe257e640 1 --2- 172.21.15.169:0/3524948296 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffbc40a2800 0x7ffbc40a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc100020 tx=0x7ffbcc005f90 comp rx=0 tx=0).stop 2024-03-28T00:54:29.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.964+0000 7ffbe257e640 1 -- 172.21.15.169:0/3524948296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbdc100850 msgr2=0x7ffbdc1002b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:29.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.964+0000 7ffbe257e640 1 --2- 172.21.15.169:0/3524948296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbdc100850 0x7ffbdc1002b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ffbd002c170 tx=0x7ffbd002c650 comp rx=0 tx=0).stop 2024-03-28T00:54:29.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.964+0000 7ffbe257e640 1 -- 172.21.15.169:0/3524948296 shutdown_connections 2024-03-28T00:54:29.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.964+0000 7ffbe257e640 1 --2- 172.21.15.169:0/3524948296 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffbc40a2800 0x7ffbc40a4cc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:29.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.964+0000 7ffbe257e640 1 --2- 172.21.15.169:0/3524948296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffbdc101c00 0x7ffbdc19edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:29.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.964+0000 7ffbe257e640 1 --2- 172.21.15.169:0/3524948296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbdc101210 0x7ffbdc19c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:29.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.964+0000 7ffbe257e640 1 --2- 172.21.15.169:0/3524948296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbdc100850 0x7ffbdc1002b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:29.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.964+0000 7ffbe257e640 1 -- 172.21.15.169:0/3524948296 >> 172.21.15.169:0/3524948296 conn(0x7ffbdc0fb2e0 msgr2=0x7ffbdc104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:29.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.964+0000 7ffbe257e640 1 -- 172.21.15.169:0/3524948296 shutdown_connections 2024-03-28T00:54:29.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:29.964+0000 7ffbe257e640 1 -- 172.21.15.169:0/3524948296 wait complete. 2024-03-28T00:54:29.978 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T00:49:56.339525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T00:54:29.978 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T00:54:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.081+0000 7fcb5dc9a640 1 -- 172.21.15.169:0/1025800150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb58104a50 msgr2=0x7fcb58106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:30.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.081+0000 7fcb5dc9a640 1 --2- 172.21.15.169:0/1025800150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb58104a50 0x7fcb58106e40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c009900 tx=0x7fcb4c02bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:30.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.081+0000 7fcb5dc9a640 1 -- 172.21.15.169:0/1025800150 shutdown_connections 2024-03-28T00:54:30.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.081+0000 7fcb5dc9a640 1 --2- 172.21.15.169:0/1025800150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb58107380 0x7fcb581097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.081+0000 7fcb5dc9a640 1 --2- 172.21.15.169:0/1025800150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb58104a50 0x7fcb58106e40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.081+0000 7fcb5dc9a640 1 --2- 172.21.15.169:0/1025800150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb58102120 0x7fcb58104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.081+0000 7fcb5dc9a640 1 -- 172.21.15.169:0/1025800150 >> 172.21.15.169:0/1025800150 conn(0x7fcb580fbb30 msgr2=0x7fcb580fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:30.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.081+0000 7fcb5dc9a640 1 -- 172.21.15.169:0/1025800150 shutdown_connections 2024-03-28T00:54:30.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.081+0000 7fcb5dc9a640 1 -- 172.21.15.169:0/1025800150 wait complete. 2024-03-28T00:54:30.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.082+0000 7fcb5dc9a640 1 Processor -- start 2024-03-28T00:54:30.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.082+0000 7fcb5dc9a640 1 -- start start 2024-03-28T00:54:30.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.082+0000 7fcb5dc9a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb58102120 0x7fcb581979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:30.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.082+0000 7fcb5dc9a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb58104a50 0x7fcb58197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:30.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.082+0000 7fcb5dc9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb58107380 0x7fcb5819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:30.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.082+0000 7fcb5dc9a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5810d320 con 0x7fcb58107380 2024-03-28T00:54:30.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.082+0000 7fcb5dc9a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5810d1a0 con 0x7fcb58104a50 2024-03-28T00:54:30.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.082+0000 7fcb5dc9a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5810d4a0 con 0x7fcb58102120 2024-03-28T00:54:30.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.082+0000 7fcb57fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb58107380 0x7fcb5819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:30.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.082+0000 7fcb57fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb58107380 0x7fcb5819efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33860/0 (socket says 172.21.15.169:33860) 2024-03-28T00:54:30.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.082+0000 7fcb57fff640 1 -- 172.21.15.169:0/1152779681 learned_addr learned my addr 172.21.15.169:0/1152779681 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.082+0000 7fcb56ffd640 1 --2- 172.21.15.169:0/1152779681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb58104a50 0x7fcb58197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.082+0000 7fcb577fe640 1 --2- 172.21.15.169:0/1152779681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb58102120 0x7fcb581979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.082+0000 7fcb57fff640 1 -- 172.21.15.169:0/1152779681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb58102120 msgr2=0x7fcb581979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.082+0000 7fcb57fff640 1 --2- 172.21.15.169:0/1152779681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb58102120 0x7fcb581979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.082+0000 7fcb57fff640 1 -- 172.21.15.169:0/1152779681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb58104a50 msgr2=0x7fcb58197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.082+0000 7fcb57fff640 1 --2- 172.21.15.169:0/1152779681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb58104a50 0x7fcb58197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.083+0000 7fcb57fff640 1 -- 172.21.15.169:0/1152779681 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5819f610 con 0x7fcb58107380 2024-03-28T00:54:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.083+0000 7fcb577fe640 1 --2- 172.21.15.169:0/1152779681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb58102120 0x7fcb581979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:54:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.083+0000 7fcb56ffd640 1 --2- 172.21.15.169:0/1152779681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb58104a50 0x7fcb58197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.083+0000 7fcb57fff640 1 --2- 172.21.15.169:0/1152779681 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb58107380 0x7fcb5819efa0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcb5000eb10 tx=0x7fcb5000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.083+0000 7fcb54ff9640 1 -- 172.21.15.169:0/1152779681 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb50004b20 con 0x7fcb58107380 2024-03-28T00:54:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.083+0000 7fcb54ff9640 1 -- 172.21.15.169:0/1152779681 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcb50004cc0 con 0x7fcb58107380 2024-03-28T00:54:30.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.083+0000 7fcb5dc9a640 1 -- 172.21.15.169:0/1152779681 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5819f900 con 0x7fcb58107380 2024-03-28T00:54:30.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.083+0000 7fcb54ff9640 1 -- 172.21.15.169:0/1152779681 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb50010760 con 0x7fcb58107380 2024-03-28T00:54:30.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.083+0000 7fcb5dc9a640 1 -- 172.21.15.169:0/1152779681 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb5819fdd0 con 0x7fcb58107380 2024-03-28T00:54:30.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.084+0000 7fcb5dc9a640 1 -- 172.21.15.169:0/1152779681 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb580fd620 con 0x7fcb58107380 2024-03-28T00:54:30.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.085+0000 7fcb54ff9640 1 -- 172.21.15.169:0/1152779681 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fcb50005020 con 0x7fcb58107380 2024-03-28T00:54:30.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.085+0000 7fcb54ff9640 1 --2- 172.21.15.169:0/1152779681 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcb400a2800 0x7fcb400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:30.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.086+0000 7fcb577fe640 1 --2- 172.21.15.169:0/1152779681 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcb400a2800 0x7fcb400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:30.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.086+0000 7fcb54ff9640 1 -- 172.21.15.169:0/1152779681 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcb500cebe0 con 0x7fcb58107380 2024-03-28T00:54:30.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.086+0000 7fcb577fe640 1 --2- 172.21.15.169:0/1152779681 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcb400a2800 0x7fcb400a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcb480058c0 tx=0x7fcb48005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:30.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.090+0000 7fcb54ff9640 1 -- 172.21.15.169:0/1152779681 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcb5009c120 con 0x7fcb58107380 2024-03-28T00:54:30.260 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:30 smithi169 ceph-mon[32274]: mds.? [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] up:boot 2024-03-28T00:54:30.261 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:30 smithi169 ceph-mon[32274]: fsmap 12 up:standby 2024-03-28T00:54:30.261 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:30 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T00:54:30.261 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:30 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:54:30.261 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:30 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/351543041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:30.261 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:30 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3524948296' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:30.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.290+0000 7fcb5dc9a640 1 -- 172.21.15.169:0/1152779681 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcb580622a0 con 0x7fcb58107380 2024-03-28T00:54:30.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.290+0000 7fcb54ff9640 1 -- 172.21.15.169:0/1152779681 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcb5009ba20 con 0x7fcb58107380 2024-03-28T00:54:30.293 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 1 2024-03-28T00:54:30.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.292+0000 7fcb5dc9a640 1 -- 172.21.15.169:0/1152779681 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcb400a2800 msgr2=0x7fcb400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:30.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.292+0000 7fcb5dc9a640 1 --2- 172.21.15.169:0/1152779681 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcb400a2800 0x7fcb400a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcb480058c0 tx=0x7fcb48005850 comp rx=0 tx=0).stop 2024-03-28T00:54:30.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.292+0000 7fcb5dc9a640 1 -- 172.21.15.169:0/1152779681 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb58107380 msgr2=0x7fcb5819efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:30.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.292+0000 7fcb5dc9a640 1 --2- 172.21.15.169:0/1152779681 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb58107380 0x7fcb5819efa0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcb5000eb10 tx=0x7fcb5000c2d0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.292+0000 7fcb5dc9a640 1 -- 172.21.15.169:0/1152779681 shutdown_connections 2024-03-28T00:54:30.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.292+0000 7fcb5dc9a640 1 --2- 172.21.15.169:0/1152779681 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcb400a2800 0x7fcb400a4cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.292+0000 7fcb5dc9a640 1 --2- 172.21.15.169:0/1152779681 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb58107380 0x7fcb5819efa0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.292+0000 7fcb5dc9a640 1 --2- 172.21.15.169:0/1152779681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb58104a50 0x7fcb58197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.292+0000 7fcb5dc9a640 1 --2- 172.21.15.169:0/1152779681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb58102120 0x7fcb581979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.292+0000 7fcb5dc9a640 1 -- 172.21.15.169:0/1152779681 >> 172.21.15.169:0/1152779681 conn(0x7fcb580fbb30 msgr2=0x7fcb58102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:30.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.293+0000 7fcb5dc9a640 1 -- 172.21.15.169:0/1152779681 shutdown_connections 2024-03-28T00:54:30.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.293+0000 7fcb5dc9a640 1 -- 172.21.15.169:0/1152779681 wait complete. 2024-03-28T00:54:30.307 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T00:54:30.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.407+0000 7f7975377640 1 -- 172.21.15.169:0/1626378598 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79701017b0 msgr2=0x7f7970101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:30.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.407+0000 7f7975377640 1 --2- 172.21.15.169:0/1626378598 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79701017b0 0x7f7970101c30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7958009900 tx=0x7f795802bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:30.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.407+0000 7f7975377640 1 -- 172.21.15.169:0/1626378598 shutdown_connections 2024-03-28T00:54:30.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.407+0000 7f7975377640 1 --2- 172.21.15.169:0/1626378598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7970102170 0x7f7970108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.407+0000 7f7975377640 1 --2- 172.21.15.169:0/1626378598 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79701017b0 0x7f7970101c30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.407+0000 7f7975377640 1 --2- 172.21.15.169:0/1626378598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79701005c0 0x7f79701009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.407+0000 7f7975377640 1 -- 172.21.15.169:0/1626378598 >> 172.21.15.169:0/1626378598 conn(0x7f79700fbb30 msgr2=0x7f79700fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:30.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.407+0000 7f7975377640 1 -- 172.21.15.169:0/1626378598 shutdown_connections 2024-03-28T00:54:30.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.407+0000 7f7975377640 1 -- 172.21.15.169:0/1626378598 wait complete. 2024-03-28T00:54:30.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.408+0000 7f7975377640 1 Processor -- start 2024-03-28T00:54:30.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.408+0000 7f7975377640 1 -- start start 2024-03-28T00:54:30.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.408+0000 7f7975377640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79701005c0 0x7f7970104620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:30.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.408+0000 7f7975377640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79701017b0 0x7f7970102c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:30.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.408+0000 7f7975377640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7970102170 0x7f79701031a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:30.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.408+0000 7f7975377640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797010c240 con 0x7f7970102170 2024-03-28T00:54:30.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.408+0000 7f7975377640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797010c0c0 con 0x7f79701005c0 2024-03-28T00:54:30.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.408+0000 7f7975377640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797010c3c0 con 0x7f79701017b0 2024-03-28T00:54:30.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.408+0000 7f796f7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7970102170 0x7f79701031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:30.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.408+0000 7f796f7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7970102170 0x7f79701031a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33878/0 (socket says 172.21.15.169:33878) 2024-03-28T00:54:30.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.408+0000 7f796f7fe640 1 -- 172.21.15.169:0/41370181 learned_addr learned my addr 172.21.15.169:0/41370181 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:30.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.408+0000 7f796effd640 1 --2- 172.21.15.169:0/41370181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79701005c0 0x7f7970104620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:30.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.408+0000 7f7967fff640 1 --2- 172.21.15.169:0/41370181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79701017b0 0x7f7970102c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:30.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.408+0000 7f796f7fe640 1 -- 172.21.15.169:0/41370181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79701017b0 msgr2=0x7f7970102c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:30.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.408+0000 7f796f7fe640 1 --2- 172.21.15.169:0/41370181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79701017b0 0x7f7970102c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.408+0000 7f796f7fe640 1 -- 172.21.15.169:0/41370181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79701005c0 msgr2=0x7f7970104620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:30.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.408+0000 7f796f7fe640 1 --2- 172.21.15.169:0/41370181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79701005c0 0x7f7970104620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.408+0000 7f796f7fe640 1 -- 172.21.15.169:0/41370181 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7970103a30 con 0x7f7970102170 2024-03-28T00:54:30.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.408+0000 7f7967fff640 1 --2- 172.21.15.169:0/41370181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79701017b0 0x7f7970102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:54:30.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.408+0000 7f796effd640 1 --2- 172.21.15.169:0/41370181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79701005c0 0x7f7970104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:54:30.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.408+0000 7f796f7fe640 1 --2- 172.21.15.169:0/41370181 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7970102170 0x7f79701031a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f796800cad0 tx=0x7f796800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:30.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.409+0000 7f796cff9640 1 -- 172.21.15.169:0/41370181 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796800ae20 con 0x7f7970102170 2024-03-28T00:54:30.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.409+0000 7f796cff9640 1 -- 172.21.15.169:0/41370181 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7968005020 con 0x7f7970102170 2024-03-28T00:54:30.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.409+0000 7f7975377640 1 -- 172.21.15.169:0/41370181 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79701a1b20 con 0x7f7970102170 2024-03-28T00:54:30.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.409+0000 7f796cff9640 1 -- 172.21.15.169:0/41370181 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796800f6f0 con 0x7f7970102170 2024-03-28T00:54:30.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.409+0000 7f7975377640 1 -- 172.21.15.169:0/41370181 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79701a2070 con 0x7f7970102170 2024-03-28T00:54:30.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.410+0000 7f7975377640 1 -- 172.21.15.169:0/41370181 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79701a22a0 con 0x7f7970102170 2024-03-28T00:54:30.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.411+0000 7f796cff9640 1 -- 172.21.15.169:0/41370181 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f79680051d0 con 0x7f7970102170 2024-03-28T00:54:30.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:30 smithi177 ceph-mon[36206]: mds.? [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] up:boot 2024-03-28T00:54:30.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:30 smithi177 ceph-mon[36206]: fsmap 12 up:standby 2024-03-28T00:54:30.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:30 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T00:54:30.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:30 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:54:30.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:30 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/351543041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:30.418 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:30 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3524948296' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:30.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.411+0000 7f796cff9640 1 --2- 172.21.15.169:0/41370181 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f795c0a2800 0x7f795c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:30.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.411+0000 7f796effd640 1 --2- 172.21.15.169:0/41370181 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f795c0a2800 0x7f795c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:30.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.411+0000 7f796cff9640 1 -- 172.21.15.169:0/41370181 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f79680cea30 con 0x7f7970102170 2024-03-28T00:54:30.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.412+0000 7f796effd640 1 --2- 172.21.15.169:0/41370181 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f795c0a2800 0x7f795c0a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7970101610 tx=0x7f7960005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:30.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.415+0000 7f796cff9640 1 -- 172.21.15.169:0/41370181 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7968014030 con 0x7f7970102170 2024-03-28T00:54:30.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.593+0000 7f7975377640 1 -- 172.21.15.169:0/41370181 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f79701a2590 con 0x7f7970102170 2024-03-28T00:54:30.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.594+0000 7f796cff9640 1 -- 172.21.15.169:0/41370181 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) v1 ==== 84+0+65 (secure 0 0 0) 0x7f796809bf70 con 0x7f7970102170 2024-03-28T00:54:30.597 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:30.597 INFO:teuthology.orchestra.run.smithi169.stdout:[ 2024-03-28T00:54:30.597 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T00:54:30.597 INFO:teuthology.orchestra.run.smithi169.stdout: "poolnum": 1, 2024-03-28T00:54:30.597 INFO:teuthology.orchestra.run.smithi169.stdout: "poolname": ".mgr" 2024-03-28T00:54:30.597 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T00:54:30.598 INFO:teuthology.orchestra.run.smithi169.stdout:] 2024-03-28T00:54:30.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.596+0000 7f7975377640 1 -- 172.21.15.169:0/41370181 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f795c0a2800 msgr2=0x7f795c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:30.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.596+0000 7f7975377640 1 --2- 172.21.15.169:0/41370181 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f795c0a2800 0x7f795c0a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7970101610 tx=0x7f7960005f90 comp rx=0 tx=0).stop 2024-03-28T00:54:30.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.596+0000 7f7975377640 1 -- 172.21.15.169:0/41370181 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7970102170 msgr2=0x7f79701031a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:30.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.596+0000 7f7975377640 1 --2- 172.21.15.169:0/41370181 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7970102170 0x7f79701031a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f796800cad0 tx=0x7f796800cf90 comp rx=0 tx=0).stop 2024-03-28T00:54:30.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.596+0000 7f7975377640 1 -- 172.21.15.169:0/41370181 shutdown_connections 2024-03-28T00:54:30.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.596+0000 7f7975377640 1 --2- 172.21.15.169:0/41370181 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f795c0a2800 0x7f795c0a4cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.596+0000 7f7975377640 1 --2- 172.21.15.169:0/41370181 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7970102170 0x7f79701031a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.596+0000 7f7975377640 1 --2- 172.21.15.169:0/41370181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79701017b0 0x7f7970102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.596+0000 7f7975377640 1 --2- 172.21.15.169:0/41370181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79701005c0 0x7f7970104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.596+0000 7f7975377640 1 -- 172.21.15.169:0/41370181 >> 172.21.15.169:0/41370181 conn(0x7f79700fbb30 msgr2=0x7f7970068800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:30.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.596+0000 7f7975377640 1 -- 172.21.15.169:0/41370181 shutdown_connections 2024-03-28T00:54:30.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.596+0000 7f7975377640 1 -- 172.21.15.169:0/41370181 wait complete. 2024-03-28T00:54:30.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:30 smithi174 ceph-mon[36060]: mds.? [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] up:boot 2024-03-28T00:54:30.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:30 smithi174 ceph-mon[36060]: fsmap 12 up:standby 2024-03-28T00:54:30.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:30 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T00:54:30.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:30 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:54:30.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:30 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/351543041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:30.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:30 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3524948296' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:30.610 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T00:54:30.610 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T00:54:30.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.717+0000 7fed57a4b640 1 -- 172.21.15.169:0/2395342793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed501005a0 msgr2=0x7fed501009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:30.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.717+0000 7fed57a4b640 1 --2- 172.21.15.169:0/2395342793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed501005a0 0x7fed501009c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fed4c009900 tx=0x7fed4c02bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:30.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.717+0000 7fed57a4b640 1 -- 172.21.15.169:0/2395342793 shutdown_connections 2024-03-28T00:54:30.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.717+0000 7fed57a4b640 1 --2- 172.21.15.169:0/2395342793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed50102150 0x7fed501089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.717+0000 7fed57a4b640 1 --2- 172.21.15.169:0/2395342793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed50101790 0x7fed50101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.717+0000 7fed57a4b640 1 --2- 172.21.15.169:0/2395342793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed501005a0 0x7fed501009c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.717+0000 7fed57a4b640 1 -- 172.21.15.169:0/2395342793 >> 172.21.15.169:0/2395342793 conn(0x7fed500fbb30 msgr2=0x7fed500fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:30.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.717+0000 7fed57a4b640 1 -- 172.21.15.169:0/2395342793 shutdown_connections 2024-03-28T00:54:30.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.717+0000 7fed57a4b640 1 -- 172.21.15.169:0/2395342793 wait complete. 2024-03-28T00:54:30.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.718+0000 7fed57a4b640 1 Processor -- start 2024-03-28T00:54:30.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.718+0000 7fed57a4b640 1 -- start start 2024-03-28T00:54:30.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.718+0000 7fed57a4b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed501005a0 0x7fed500693d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:30.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.718+0000 7fed57a4b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed50101790 0x7fed5019c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:30.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.718+0000 7fed57a4b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed50102150 0x7fed5019ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:30.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.718+0000 7fed57a4b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5010c230 con 0x7fed50102150 2024-03-28T00:54:30.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.718+0000 7fed57a4b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5010c0b0 con 0x7fed501005a0 2024-03-28T00:54:30.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.718+0000 7fed57a4b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5010c3b0 con 0x7fed50101790 2024-03-28T00:54:30.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.718+0000 7fed55fe8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed50102150 0x7fed5019ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:30.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.718+0000 7fed55fe8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed50102150 0x7fed5019ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33896/0 (socket says 172.21.15.169:33896) 2024-03-28T00:54:30.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.718+0000 7fed55fe8640 1 -- 172.21.15.169:0/3399635248 learned_addr learned my addr 172.21.15.169:0/3399635248 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:30.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.718+0000 7fed557e7640 1 --2- 172.21.15.169:0/3399635248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed501005a0 0x7fed500693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:30.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.718+0000 7fed54fe6640 1 --2- 172.21.15.169:0/3399635248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed50101790 0x7fed5019c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:30.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.719+0000 7fed55fe8640 1 -- 172.21.15.169:0/3399635248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed50101790 msgr2=0x7fed5019c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:30.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.719+0000 7fed55fe8640 1 --2- 172.21.15.169:0/3399635248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed50101790 0x7fed5019c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.719+0000 7fed55fe8640 1 -- 172.21.15.169:0/3399635248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed501005a0 msgr2=0x7fed500693d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:30.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.719+0000 7fed55fe8640 1 --2- 172.21.15.169:0/3399635248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed501005a0 0x7fed500693d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:30.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.719+0000 7fed55fe8640 1 -- 172.21.15.169:0/3399635248 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5019f610 con 0x7fed50102150 2024-03-28T00:54:30.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.719+0000 7fed55fe8640 1 --2- 172.21.15.169:0/3399635248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed50102150 0x7fed5019ed50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fed4800eb10 tx=0x7fed4800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:30.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.720+0000 7fed467fc640 1 -- 172.21.15.169:0/3399635248 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed48004b20 con 0x7fed50102150 2024-03-28T00:54:30.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.720+0000 7fed467fc640 1 -- 172.21.15.169:0/3399635248 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed48004cc0 con 0x7fed50102150 2024-03-28T00:54:30.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.720+0000 7fed57a4b640 1 -- 172.21.15.169:0/3399635248 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed5019f900 con 0x7fed50102150 2024-03-28T00:54:30.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.720+0000 7fed467fc640 1 -- 172.21.15.169:0/3399635248 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed48010760 con 0x7fed50102150 2024-03-28T00:54:30.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.720+0000 7fed57a4b640 1 -- 172.21.15.169:0/3399635248 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed5019fe50 con 0x7fed50102150 2024-03-28T00:54:30.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.721+0000 7fed57a4b640 1 -- 172.21.15.169:0/3399635248 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed50107960 con 0x7fed50102150 2024-03-28T00:54:30.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.722+0000 7fed467fc640 1 -- 172.21.15.169:0/3399635248 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fed48005000 con 0x7fed50102150 2024-03-28T00:54:30.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.723+0000 7fed467fc640 1 --2- 172.21.15.169:0/3399635248 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed3c0a2880 0x7fed3c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:30.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.723+0000 7fed557e7640 1 --2- 172.21.15.169:0/3399635248 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed3c0a2880 0x7fed3c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:30.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.723+0000 7fed467fc640 1 -- 172.21.15.169:0/3399635248 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fed480cece0 con 0x7fed50102150 2024-03-28T00:54:30.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.723+0000 7fed557e7640 1 --2- 172.21.15.169:0/3399635248 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed3c0a2880 0x7fed3c0a4d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fed4c000c00 tx=0x7fed4c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:30.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.726+0000 7fed467fc640 1 -- 172.21.15.169:0/3399635248 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fed48053740 con 0x7fed50102150 2024-03-28T00:54:30.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:30.916+0000 7fed57a4b640 1 -- 172.21.15.169:0/3399635248 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fed5002d2c0 con 0x7fed50102150 2024-03-28T00:54:31.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.233+0000 7fed467fc640 1 -- 172.21.15.169:0/3399635248 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) v1 ==== 138+0+0 (secure 0 0 0) 0x7fed4809c220 con 0x7fed50102150 2024-03-28T00:54:31.236 INFO:teuthology.orchestra.run.smithi169.stderr:pool 'cephfs_metadata' created 2024-03-28T00:54:31.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.235+0000 7fed57a4b640 1 -- 172.21.15.169:0/3399635248 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed3c0a2880 msgr2=0x7fed3c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:31.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.235+0000 7fed57a4b640 1 --2- 172.21.15.169:0/3399635248 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed3c0a2880 0x7fed3c0a4d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fed4c000c00 tx=0x7fed4c005fd0 comp rx=0 tx=0).stop 2024-03-28T00:54:31.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.235+0000 7fed57a4b640 1 -- 172.21.15.169:0/3399635248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed50102150 msgr2=0x7fed5019ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:31.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.235+0000 7fed57a4b640 1 --2- 172.21.15.169:0/3399635248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed50102150 0x7fed5019ed50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fed4800eb10 tx=0x7fed4800c2d0 comp rx=0 tx=0).stop 2024-03-28T00:54:31.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.235+0000 7fed57a4b640 1 -- 172.21.15.169:0/3399635248 shutdown_connections 2024-03-28T00:54:31.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.235+0000 7fed57a4b640 1 --2- 172.21.15.169:0/3399635248 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed3c0a2880 0x7fed3c0a4d40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:31.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.235+0000 7fed57a4b640 1 --2- 172.21.15.169:0/3399635248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed50102150 0x7fed5019ed50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:31.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.235+0000 7fed57a4b640 1 --2- 172.21.15.169:0/3399635248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed50101790 0x7fed5019c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:31.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.235+0000 7fed57a4b640 1 --2- 172.21.15.169:0/3399635248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed501005a0 0x7fed500693d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:31.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.235+0000 7fed57a4b640 1 -- 172.21.15.169:0/3399635248 >> 172.21.15.169:0/3399635248 conn(0x7fed500fbb30 msgr2=0x7fed50104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:31.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.236+0000 7fed57a4b640 1 -- 172.21.15.169:0/3399635248 shutdown_connections 2024-03-28T00:54:31.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.236+0000 7fed57a4b640 1 -- 172.21.15.169:0/3399635248 wait complete. 2024-03-28T00:54:31.252 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-28T00:54:31.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.359+0000 7f609a663640 1 -- 172.21.15.169:0/1388108598 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60941005a0 msgr2=0x7f60941009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:31.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.359+0000 7f609a663640 1 --2- 172.21.15.169:0/1388108598 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60941005a0 0x7f60941009c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6084009900 tx=0x7f608402bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:31.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.359+0000 7f609a663640 1 -- 172.21.15.169:0/1388108598 shutdown_connections 2024-03-28T00:54:31.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.359+0000 7f609a663640 1 --2- 172.21.15.169:0/1388108598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6094102150 0x7f60941089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:31.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.359+0000 7f609a663640 1 --2- 172.21.15.169:0/1388108598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6094101790 0x7f6094101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:31.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.359+0000 7f609a663640 1 --2- 172.21.15.169:0/1388108598 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60941005a0 0x7f60941009c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:31.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.359+0000 7f609a663640 1 -- 172.21.15.169:0/1388108598 >> 172.21.15.169:0/1388108598 conn(0x7f60940fbb30 msgr2=0x7f60940fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:31.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.359+0000 7f609a663640 1 -- 172.21.15.169:0/1388108598 shutdown_connections 2024-03-28T00:54:31.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.359+0000 7f609a663640 1 -- 172.21.15.169:0/1388108598 wait complete. 2024-03-28T00:54:31.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.359+0000 7f609a663640 1 Processor -- start 2024-03-28T00:54:31.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.360+0000 7f609a663640 1 -- start start 2024-03-28T00:54:31.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.360+0000 7f609a663640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60941005a0 0x7f6094196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:31.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.360+0000 7f609a663640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6094101790 0x7f6094196e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:31.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.360+0000 7f609a663640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6094102150 0x7f609419def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:31.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.360+0000 7f609a663640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609410c200 con 0x7f6094101790 2024-03-28T00:54:31.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.360+0000 7f609a663640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609410c080 con 0x7f60941005a0 2024-03-28T00:54:31.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.360+0000 7f609a663640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609410c380 con 0x7f6094102150 2024-03-28T00:54:31.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.360+0000 7f60937fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6094101790 0x7f6094196e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:31.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.360+0000 7f60937fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6094101790 0x7f6094196e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33910/0 (socket says 172.21.15.169:33910) 2024-03-28T00:54:31.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.360+0000 7f60937fe640 1 -- 172.21.15.169:0/4079905238 learned_addr learned my addr 172.21.15.169:0/4079905238 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:31.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.360+0000 7f6093fff640 1 --2- 172.21.15.169:0/4079905238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60941005a0 0x7f6094196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:31.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.360+0000 7f6098c00640 1 --2- 172.21.15.169:0/4079905238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6094102150 0x7f609419def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:31.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.360+0000 7f60937fe640 1 -- 172.21.15.169:0/4079905238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6094102150 msgr2=0x7f609419def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:31.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.360+0000 7f60937fe640 1 --2- 172.21.15.169:0/4079905238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6094102150 0x7f609419def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:31.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.360+0000 7f60937fe640 1 -- 172.21.15.169:0/4079905238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60941005a0 msgr2=0x7f6094196940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:31.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.360+0000 7f60937fe640 1 --2- 172.21.15.169:0/4079905238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60941005a0 0x7f6094196940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:31.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.360+0000 7f60937fe640 1 -- 172.21.15.169:0/4079905238 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609419e560 con 0x7f6094101790 2024-03-28T00:54:31.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.361+0000 7f60937fe640 1 --2- 172.21.15.169:0/4079905238 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6094101790 0x7f6094196e80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f608800d950 tx=0x7f608800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:31.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.361+0000 7f60917fa640 1 -- 172.21.15.169:0/4079905238 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6088017070 con 0x7f6094101790 2024-03-28T00:54:31.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.361+0000 7f609a663640 1 -- 172.21.15.169:0/4079905238 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f609419e850 con 0x7f6094101790 2024-03-28T00:54:31.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.361+0000 7f60917fa640 1 -- 172.21.15.169:0/4079905238 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6088009c30 con 0x7f6094101790 2024-03-28T00:54:31.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.361+0000 7f60917fa640 1 -- 172.21.15.169:0/4079905238 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608800f6b0 con 0x7f6094101790 2024-03-28T00:54:31.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.361+0000 7f609a663640 1 -- 172.21.15.169:0/4079905238 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f609419edf0 con 0x7f6094101790 2024-03-28T00:54:31.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.362+0000 7f609a663640 1 -- 172.21.15.169:0/4079905238 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60941079e0 con 0x7f6094101790 2024-03-28T00:54:31.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.363+0000 7f60917fa640 1 -- 172.21.15.169:0/4079905238 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f60880109d0 con 0x7f6094101790 2024-03-28T00:54:31.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.363+0000 7f60917fa640 1 --2- 172.21.15.169:0/4079905238 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f607c0a2800 0x7f607c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:31.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.364+0000 7f6093fff640 1 --2- 172.21.15.169:0/4079905238 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f607c0a2800 0x7f607c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:31.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.364+0000 7f60917fa640 1 -- 172.21.15.169:0/4079905238 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f60880ce2f0 con 0x7f6094101790 2024-03-28T00:54:31.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.364+0000 7f6093fff640 1 --2- 172.21.15.169:0/4079905238 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f607c0a2800 0x7f607c0a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6084000c00 tx=0x7f6084005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:31.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.368+0000 7f60917fa640 1 -- 172.21.15.169:0/4079905238 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6088009de0 con 0x7f6094101790 2024-03-28T00:54:31.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:31.544+0000 7f609a663640 1 -- 172.21.15.169:0/4079905238 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f60940622a0 con 0x7f6094101790 2024-03-28T00:54:31.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:31 smithi169 ceph-mon[32274]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:31.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:31 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1152779681' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:54:31.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:31 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/41370181' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T00:54:31.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:31 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3399635248' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T00:54:31.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:31 smithi174 ceph-mon[36060]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:31.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:31 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1152779681' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:54:31.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:31 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/41370181' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T00:54:31.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:31 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3399635248' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T00:54:31.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:31 smithi177 ceph-mon[36206]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:31.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:31 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1152779681' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:54:31.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:31 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/41370181' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T00:54:31.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:31 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3399635248' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T00:54:32.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.235+0000 7f60917fa640 1 -- 172.21.15.169:0/4079905238 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) v1 ==== 171+0+0 (secure 0 0 0) 0x7f608809b070 con 0x7f6094101790 2024-03-28T00:54:32.238 INFO:teuthology.orchestra.run.smithi169.stderr:pool 'cephfs_data' created 2024-03-28T00:54:32.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.238+0000 7f609a663640 1 -- 172.21.15.169:0/4079905238 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f607c0a2800 msgr2=0x7f607c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:32.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.238+0000 7f609a663640 1 --2- 172.21.15.169:0/4079905238 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f607c0a2800 0x7f607c0a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6084000c00 tx=0x7f6084005fd0 comp rx=0 tx=0).stop 2024-03-28T00:54:32.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.238+0000 7f609a663640 1 -- 172.21.15.169:0/4079905238 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6094101790 msgr2=0x7f6094196e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:32.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.238+0000 7f609a663640 1 --2- 172.21.15.169:0/4079905238 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6094101790 0x7f6094196e80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f608800d950 tx=0x7f608800de10 comp rx=0 tx=0).stop 2024-03-28T00:54:32.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.239+0000 7f609a663640 1 -- 172.21.15.169:0/4079905238 shutdown_connections 2024-03-28T00:54:32.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.239+0000 7f609a663640 1 --2- 172.21.15.169:0/4079905238 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f607c0a2800 0x7f607c0a4cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:32.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.239+0000 7f609a663640 1 --2- 172.21.15.169:0/4079905238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6094102150 0x7f609419def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:32.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.239+0000 7f609a663640 1 --2- 172.21.15.169:0/4079905238 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6094101790 0x7f6094196e80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:32.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.239+0000 7f609a663640 1 --2- 172.21.15.169:0/4079905238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60941005a0 0x7f6094196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:32.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.239+0000 7f609a663640 1 -- 172.21.15.169:0/4079905238 >> 172.21.15.169:0/4079905238 conn(0x7f60940fbb30 msgr2=0x7f6094104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:32.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.241+0000 7f609a663640 1 -- 172.21.15.169:0/4079905238 shutdown_connections 2024-03-28T00:54:32.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.241+0000 7f609a663640 1 -- 172.21.15.169:0/4079905238 wait complete. 2024-03-28T00:54:32.268 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T00:54:32.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.374+0000 7f67df836640 1 -- 172.21.15.169:0/3307366603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67d81005a0 msgr2=0x7f67d81009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:32.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.374+0000 7f67df836640 1 --2- 172.21.15.169:0/3307366603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67d81005a0 0x7f67d81009c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f67d4009900 tx=0x7f67d402bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:32.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.374+0000 7f67df836640 1 -- 172.21.15.169:0/3307366603 shutdown_connections 2024-03-28T00:54:32.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.374+0000 7f67df836640 1 --2- 172.21.15.169:0/3307366603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67d8102150 0x7f67d81089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:32.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.374+0000 7f67df836640 1 --2- 172.21.15.169:0/3307366603 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d8101790 0x7f67d8101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:32.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.374+0000 7f67df836640 1 --2- 172.21.15.169:0/3307366603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67d81005a0 0x7f67d81009c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:32.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.374+0000 7f67df836640 1 -- 172.21.15.169:0/3307366603 >> 172.21.15.169:0/3307366603 conn(0x7f67d80fbb30 msgr2=0x7f67d80fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:32.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.374+0000 7f67df836640 1 -- 172.21.15.169:0/3307366603 shutdown_connections 2024-03-28T00:54:32.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.374+0000 7f67df836640 1 -- 172.21.15.169:0/3307366603 wait complete. 2024-03-28T00:54:32.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.374+0000 7f67df836640 1 Processor -- start 2024-03-28T00:54:32.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.374+0000 7f67df836640 1 -- start start 2024-03-28T00:54:32.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67df836640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67d81005a0 0x7f67d8196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:32.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67df836640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67d8101790 0x7f67d8196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:32.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67df836640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d8102150 0x7f67d819dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:32.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67df836640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d810c240 con 0x7f67d81005a0 2024-03-28T00:54:32.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67df836640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d810c0c0 con 0x7f67d8101790 2024-03-28T00:54:32.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67df836640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d810c3c0 con 0x7f67d8102150 2024-03-28T00:54:32.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67dd5d2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67d81005a0 0x7f67d8196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:32.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67dd5d2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67d81005a0 0x7f67d8196930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33936/0 (socket says 172.21.15.169:33936) 2024-03-28T00:54:32.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67dd5d2640 1 -- 172.21.15.169:0/3552982194 learned_addr learned my addr 172.21.15.169:0/3552982194 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:32.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67dcdd1640 1 --2- 172.21.15.169:0/3552982194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67d8101790 0x7f67d8196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:32.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67dddd3640 1 --2- 172.21.15.169:0/3552982194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d8102150 0x7f67d819dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:32.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67dddd3640 1 -- 172.21.15.169:0/3552982194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67d8101790 msgr2=0x7f67d8196e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:32.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67dddd3640 1 --2- 172.21.15.169:0/3552982194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67d8101790 0x7f67d8196e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:32.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67dddd3640 1 -- 172.21.15.169:0/3552982194 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67d81005a0 msgr2=0x7f67d8196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:32.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67dddd3640 1 --2- 172.21.15.169:0/3552982194 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67d81005a0 0x7f67d8196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:32.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67dddd3640 1 -- 172.21.15.169:0/3552982194 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d819e550 con 0x7f67d8102150 2024-03-28T00:54:32.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67dd5d2640 1 --2- 172.21.15.169:0/3552982194 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67d81005a0 0x7f67d8196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:32.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67dddd3640 1 --2- 172.21.15.169:0/3552982194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d8102150 0x7f67d819dee0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f67d0009fd0 tx=0x7f67d000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:32.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67ce7fc640 1 -- 172.21.15.169:0/3552982194 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d000ce20 con 0x7f67d8102150 2024-03-28T00:54:32.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67ce7fc640 1 -- 172.21.15.169:0/3552982194 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67d0005020 con 0x7f67d8102150 2024-03-28T00:54:32.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67df836640 1 -- 172.21.15.169:0/3552982194 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d819e840 con 0x7f67d8102150 2024-03-28T00:54:32.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.375+0000 7f67ce7fc640 1 -- 172.21.15.169:0/3552982194 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d0010720 con 0x7f67d8102150 2024-03-28T00:54:32.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.376+0000 7f67df836640 1 -- 172.21.15.169:0/3552982194 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d819ed40 con 0x7f67d8102150 2024-03-28T00:54:32.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.376+0000 7f67df836640 1 -- 172.21.15.169:0/3552982194 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d8107990 con 0x7f67d8102150 2024-03-28T00:54:32.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.378+0000 7f67ce7fc640 1 -- 172.21.15.169:0/3552982194 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f67d0005540 con 0x7f67d8102150 2024-03-28T00:54:32.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.378+0000 7f67ce7fc640 1 --2- 172.21.15.169:0/3552982194 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f67c40a2800 0x7f67c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:32.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.378+0000 7f67ce7fc640 1 -- 172.21.15.169:0/3552982194 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f67d00cefc0 con 0x7f67d8102150 2024-03-28T00:54:32.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.379+0000 7f67dd5d2640 1 --2- 172.21.15.169:0/3552982194 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f67c40a2800 0x7f67c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:32.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.379+0000 7f67dd5d2640 1 --2- 172.21.15.169:0/3552982194 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f67c40a2800 0x7f67c40a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f67d4006010 tx=0x7f67d402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:32.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.382+0000 7f67ce7fc640 1 -- 172.21.15.169:0/3552982194 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f67d009c170 con 0x7f67d8102150 2024-03-28T00:54:32.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:32 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:32 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:32 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:32 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:32 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:32 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:32 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:32 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3399635248' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T00:54:32.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:32 smithi169 ceph-mon[32274]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T00:54:32.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:32 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4079905238' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-28T00:54:32.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:32 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:32.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:32 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:54:32.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:32 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:32 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:54:32.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:32 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:32 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:32 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:32 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:32 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:32 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:32 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:32 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3399635248' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T00:54:32.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:32 smithi174 ceph-mon[36060]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T00:54:32.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:32 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/4079905238' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-28T00:54:32.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:32 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:32.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:32 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:54:32.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:32 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:32 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:54:32.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:32.611+0000 7f67df836640 1 -- 172.21.15.169:0/3552982194 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f67d802d070 con 0x7f67d8102150 2024-03-28T00:54:32.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:32 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:32 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:32 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:32 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:32 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:32 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:32 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:32 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3399635248' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T00:54:32.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:32 smithi177 ceph-mon[36206]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T00:54:32.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:32 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/4079905238' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-28T00:54:32.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:32 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:32.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:32 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:54:32.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:32 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:32.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:32 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:54:33.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.239+0000 7f67ce7fc640 1 -- 172.21.15.169:0/3552982194 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f67d009ba70 con 0x7f67d8102150 2024-03-28T00:54:33.242 INFO:teuthology.orchestra.run.smithi169.stderr:new fs with metadata pool 2 and data pool 3 2024-03-28T00:54:33.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.241+0000 7f67df836640 1 -- 172.21.15.169:0/3552982194 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f67c40a2800 msgr2=0x7f67c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:33.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.241+0000 7f67df836640 1 --2- 172.21.15.169:0/3552982194 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f67c40a2800 0x7f67c40a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f67d4006010 tx=0x7f67d402df90 comp rx=0 tx=0).stop 2024-03-28T00:54:33.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.241+0000 7f67df836640 1 -- 172.21.15.169:0/3552982194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d8102150 msgr2=0x7f67d819dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:33.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.241+0000 7f67df836640 1 --2- 172.21.15.169:0/3552982194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d8102150 0x7f67d819dee0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f67d0009fd0 tx=0x7f67d000ef10 comp rx=0 tx=0).stop 2024-03-28T00:54:33.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.241+0000 7f67df836640 1 -- 172.21.15.169:0/3552982194 shutdown_connections 2024-03-28T00:54:33.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.241+0000 7f67df836640 1 --2- 172.21.15.169:0/3552982194 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f67c40a2800 0x7f67c40a4cc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:33.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.241+0000 7f67df836640 1 --2- 172.21.15.169:0/3552982194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d8102150 0x7f67d819dee0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:33.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.241+0000 7f67df836640 1 --2- 172.21.15.169:0/3552982194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67d8101790 0x7f67d8196e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:33.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.241+0000 7f67df836640 1 --2- 172.21.15.169:0/3552982194 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67d81005a0 0x7f67d8196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:33.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.241+0000 7f67df836640 1 -- 172.21.15.169:0/3552982194 >> 172.21.15.169:0/3552982194 conn(0x7f67d80fbb30 msgr2=0x7f67d8104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:33.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.242+0000 7f67df836640 1 -- 172.21.15.169:0/3552982194 shutdown_connections 2024-03-28T00:54:33.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.242+0000 7f67df836640 1 -- 172.21.15.169:0/3552982194 wait complete. 2024-03-28T00:54:33.247 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:33 smithi174 ceph-mon[36060]: pgmap v280: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:33.247 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:33 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/4079905238' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-28T00:54:33.247 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:33 smithi174 ceph-mon[36060]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T00:54:33.247 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:33 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3552982194' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T00:54:33.247 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:33 smithi174 ceph-mon[36060]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T00:54:33.247 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:33 smithi174 ceph-mon[36060]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T00:54:33.261 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-28T00:54:33.262 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-28T00:54:33.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.375+0000 7fe2ca2dc640 1 -- 172.21.15.169:0/160196306 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2c4107420 msgr2=0x7fe2c4109810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:33.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.375+0000 7fe2ca2dc640 1 --2- 172.21.15.169:0/160196306 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2c4107420 0x7fe2c4109810 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc009ca0 tx=0x7fe2bc02c0f0 comp rx=0 tx=0).stop 2024-03-28T00:54:33.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.375+0000 7fe2ca2dc640 1 -- 172.21.15.169:0/160196306 shutdown_connections 2024-03-28T00:54:33.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.375+0000 7fe2ca2dc640 1 --2- 172.21.15.169:0/160196306 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2c4107420 0x7fe2c4109810 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:33.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.375+0000 7fe2ca2dc640 1 --2- 172.21.15.169:0/160196306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2c4104af0 0x7fe2c4106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:33.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.375+0000 7fe2ca2dc640 1 --2- 172.21.15.169:0/160196306 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe2c41021c0 0x7fe2c41045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:33.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.375+0000 7fe2ca2dc640 1 -- 172.21.15.169:0/160196306 >> 172.21.15.169:0/160196306 conn(0x7fe2c40fbb50 msgr2=0x7fe2c40fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:33.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.375+0000 7fe2ca2dc640 1 -- 172.21.15.169:0/160196306 shutdown_connections 2024-03-28T00:54:33.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.375+0000 7fe2ca2dc640 1 -- 172.21.15.169:0/160196306 wait complete. 2024-03-28T00:54:33.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.376+0000 7fe2ca2dc640 1 Processor -- start 2024-03-28T00:54:33.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.376+0000 7fe2ca2dc640 1 -- start start 2024-03-28T00:54:33.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.376+0000 7fe2ca2dc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2c41021c0 0x7fe2c4197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:33.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.376+0000 7fe2ca2dc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe2c4104af0 0x7fe2c4197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:33.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.376+0000 7fe2ca2dc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2c4107420 0x7fe2c419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:33.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.376+0000 7fe2ca2dc640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c410d380 con 0x7fe2c4107420 2024-03-28T00:54:33.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.376+0000 7fe2ca2dc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c410d200 con 0x7fe2c4104af0 2024-03-28T00:54:33.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.376+0000 7fe2ca2dc640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c410d500 con 0x7fe2c41021c0 2024-03-28T00:54:33.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.376+0000 7fe2c8879640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2c4107420 0x7fe2c419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:33.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.376+0000 7fe2c8879640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2c4107420 0x7fe2c419eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33954/0 (socket says 172.21.15.169:33954) 2024-03-28T00:54:33.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.376+0000 7fe2c8879640 1 -- 172.21.15.169:0/3980427328 learned_addr learned my addr 172.21.15.169:0/3980427328 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:33.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.376+0000 7fe2c3fff640 1 --2- 172.21.15.169:0/3980427328 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2c41021c0 0x7fe2c4197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:33.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.376+0000 7fe2c37fe640 1 --2- 172.21.15.169:0/3980427328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe2c4104af0 0x7fe2c4197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:33.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.376+0000 7fe2c8879640 1 -- 172.21.15.169:0/3980427328 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2c41021c0 msgr2=0x7fe2c4197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:33.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.376+0000 7fe2c8879640 1 --2- 172.21.15.169:0/3980427328 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2c41021c0 0x7fe2c4197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:33.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.376+0000 7fe2c8879640 1 -- 172.21.15.169:0/3980427328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe2c4104af0 msgr2=0x7fe2c4197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:33.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.376+0000 7fe2c8879640 1 --2- 172.21.15.169:0/3980427328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe2c4104af0 0x7fe2c4197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:33.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.376+0000 7fe2c8879640 1 -- 172.21.15.169:0/3980427328 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c419f660 con 0x7fe2c4107420 2024-03-28T00:54:33.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.377+0000 7fe2c8879640 1 --2- 172.21.15.169:0/3980427328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2c4107420 0x7fe2c419eff0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc005cc0 tx=0x7fe2bc032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:33.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.377+0000 7fe2c17fa640 1 -- 172.21.15.169:0/3980427328 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc041020 con 0x7fe2c4107420 2024-03-28T00:54:33.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.377+0000 7fe2ca2dc640 1 -- 172.21.15.169:0/3980427328 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c419f8f0 con 0x7fe2c4107420 2024-03-28T00:54:33.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.377+0000 7fe2c17fa640 1 -- 172.21.15.169:0/3980427328 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2bc034cd0 con 0x7fe2c4107420 2024-03-28T00:54:33.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.377+0000 7fe2c17fa640 1 -- 172.21.15.169:0/3980427328 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc039040 con 0x7fe2c4107420 2024-03-28T00:54:33.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.377+0000 7fe2ca2dc640 1 -- 172.21.15.169:0/3980427328 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c419fe00 con 0x7fe2c4107420 2024-03-28T00:54:33.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.378+0000 7fe2ca2dc640 1 -- 172.21.15.169:0/3980427328 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c40fd640 con 0x7fe2c4107420 2024-03-28T00:54:33.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.379+0000 7fe2c17fa640 1 -- 172.21.15.169:0/3980427328 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe2bc044020 con 0x7fe2c4107420 2024-03-28T00:54:33.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.379+0000 7fe2c17fa640 1 --2- 172.21.15.169:0/3980427328 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe2ac0a2800 0x7fe2ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:33.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.380+0000 7fe2c3fff640 1 --2- 172.21.15.169:0/3980427328 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe2ac0a2800 0x7fe2ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:33.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.380+0000 7fe2c17fa640 1 -- 172.21.15.169:0/3980427328 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe2bc0f0a90 con 0x7fe2c4107420 2024-03-28T00:54:33.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.380+0000 7fe2c3fff640 1 --2- 172.21.15.169:0/3980427328 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe2ac0a2800 0x7fe2ac0a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4009bd0 tx=0x7fe2b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:33.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.383+0000 7fe2c17fa640 1 -- 172.21.15.169:0/3980427328 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe2bc04b020 con 0x7fe2c4107420 2024-03-28T00:54:33.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:33 smithi169 ceph-mon[32274]: pgmap v280: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:33.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:33 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4079905238' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-28T00:54:33.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:33 smithi169 ceph-mon[32274]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T00:54:33.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:33 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3552982194' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T00:54:33.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:33 smithi169 ceph-mon[32274]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T00:54:33.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:33 smithi169 ceph-mon[32274]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T00:54:33.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:33 smithi169 ceph-mon[32274]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T00:54:33.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:33 smithi169 ceph-mon[32274]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T00:54:33.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:33 smithi174 ceph-mon[36060]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T00:54:33.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:33 smithi174 ceph-mon[36060]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T00:54:33.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.617+0000 7fe2ca2dc640 1 -- 172.21.15.169:0/3980427328 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7fe2c40622a0 con 0x7fe2c4107420 2024-03-28T00:54:33.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:33 smithi177 ceph-mon[36206]: pgmap v280: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:33.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:33 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/4079905238' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-28T00:54:33.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:33 smithi177 ceph-mon[36206]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T00:54:33.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:33 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3552982194' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T00:54:33.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:33 smithi177 ceph-mon[36206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T00:54:33.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:33 smithi177 ceph-mon[36206]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T00:54:33.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:33 smithi177 ceph-mon[36206]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T00:54:33.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:33 smithi177 ceph-mon[36206]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T00:54:33.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.788+0000 7fe2c17fa640 1 -- 172.21.15.169:0/3980427328 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) v1 ==== 157+0+0 (secure 0 0 0) 0x7fe2bc0bdd10 con 0x7fe2c4107420 2024-03-28T00:54:33.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.790+0000 7fe2ca2dc640 1 -- 172.21.15.169:0/3980427328 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe2ac0a2800 msgr2=0x7fe2ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:33.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.790+0000 7fe2ca2dc640 1 --2- 172.21.15.169:0/3980427328 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe2ac0a2800 0x7fe2ac0a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4009bd0 tx=0x7fe2b4005f90 comp rx=0 tx=0).stop 2024-03-28T00:54:33.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.790+0000 7fe2ca2dc640 1 -- 172.21.15.169:0/3980427328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2c4107420 msgr2=0x7fe2c419eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:33.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.790+0000 7fe2ca2dc640 1 --2- 172.21.15.169:0/3980427328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2c4107420 0x7fe2c419eff0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc005cc0 tx=0x7fe2bc032f40 comp rx=0 tx=0).stop 2024-03-28T00:54:33.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.790+0000 7fe2ca2dc640 1 -- 172.21.15.169:0/3980427328 shutdown_connections 2024-03-28T00:54:33.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.790+0000 7fe2ca2dc640 1 --2- 172.21.15.169:0/3980427328 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe2ac0a2800 0x7fe2ac0a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:33.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.790+0000 7fe2ca2dc640 1 --2- 172.21.15.169:0/3980427328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe2c4107420 0x7fe2c419eff0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:33.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.790+0000 7fe2ca2dc640 1 --2- 172.21.15.169:0/3980427328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe2c4104af0 0x7fe2c4197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:33.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.790+0000 7fe2ca2dc640 1 --2- 172.21.15.169:0/3980427328 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2c41021c0 0x7fe2c4197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:33.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.791+0000 7fe2ca2dc640 1 -- 172.21.15.169:0/3980427328 >> 172.21.15.169:0/3980427328 conn(0x7fe2c40fbb50 msgr2=0x7fe2c4102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:33.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.791+0000 7fe2ca2dc640 1 -- 172.21.15.169:0/3980427328 shutdown_connections 2024-03-28T00:54:33.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.791+0000 7fe2ca2dc640 1 -- 172.21.15.169:0/3980427328 wait complete. 2024-03-28T00:54:33.805 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T00:54:33.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.913+0000 7f63ff075640 1 -- 172.21.15.169:0/3907528853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63f81028b0 msgr2=0x7f63f8102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:33.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.913+0000 7f63ff075640 1 --2- 172.21.15.169:0/3907528853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63f81028b0 0x7f63f8102d30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f63ec009900 tx=0x7f63ec02bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:33.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.913+0000 7f63ff075640 1 -- 172.21.15.169:0/3907528853 shutdown_connections 2024-03-28T00:54:33.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.913+0000 7f63ff075640 1 --2- 172.21.15.169:0/3907528853 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63f8103270 0x7f63f8109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:33.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.913+0000 7f63ff075640 1 --2- 172.21.15.169:0/3907528853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63f81028b0 0x7f63f8102d30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:33.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.913+0000 7f63ff075640 1 --2- 172.21.15.169:0/3907528853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63f81016c0 0x7f63f8101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:33.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.913+0000 7f63ff075640 1 -- 172.21.15.169:0/3907528853 >> 172.21.15.169:0/3907528853 conn(0x7f63f80fcc50 msgr2=0x7f63f80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:33.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.913+0000 7f63ff075640 1 -- 172.21.15.169:0/3907528853 shutdown_connections 2024-03-28T00:54:33.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.913+0000 7f63ff075640 1 -- 172.21.15.169:0/3907528853 wait complete. 2024-03-28T00:54:33.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.913+0000 7f63ff075640 1 Processor -- start 2024-03-28T00:54:33.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.913+0000 7f63ff075640 1 -- start start 2024-03-28T00:54:33.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.913+0000 7f63ff075640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63f81016c0 0x7f63f8197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:33.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.913+0000 7f63ff075640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63f81028b0 0x7f63f8197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:33.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.913+0000 7f63ff075640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63f8103270 0x7f63f819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:33.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.914+0000 7f63ff075640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f810d3f0 con 0x7f63f81028b0 2024-03-28T00:54:33.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.914+0000 7f63ff075640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f810d270 con 0x7f63f8103270 2024-03-28T00:54:33.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.914+0000 7f63ff075640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f810d570 con 0x7f63f81016c0 2024-03-28T00:54:33.920 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.914+0000 7f63f7fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63f81028b0 0x7f63f8197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:33.920 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.914+0000 7f63f7fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63f81028b0 0x7f63f8197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33984/0 (socket says 172.21.15.169:33984) 2024-03-28T00:54:33.920 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.914+0000 7f63f7fff640 1 -- 172.21.15.169:0/1779349246 learned_addr learned my addr 172.21.15.169:0/1779349246 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:33.920 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.914+0000 7f63fce11640 1 --2- 172.21.15.169:0/1779349246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63f81016c0 0x7f63f8197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:33.920 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.914+0000 7f63fd612640 1 --2- 172.21.15.169:0/1779349246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63f8103270 0x7f63f819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:33.920 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.914+0000 7f63f7fff640 1 -- 172.21.15.169:0/1779349246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63f81016c0 msgr2=0x7f63f8197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:33.920 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.914+0000 7f63f7fff640 1 --2- 172.21.15.169:0/1779349246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63f81016c0 0x7f63f8197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:33.920 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.914+0000 7f63f7fff640 1 -- 172.21.15.169:0/1779349246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63f8103270 msgr2=0x7f63f819f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:33.920 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.914+0000 7f63f7fff640 1 --2- 172.21.15.169:0/1779349246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63f8103270 0x7f63f819f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:33.920 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.914+0000 7f63f7fff640 1 -- 172.21.15.169:0/1779349246 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f819f680 con 0x7f63f81028b0 2024-03-28T00:54:33.920 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.914+0000 7f63fce11640 1 --2- 172.21.15.169:0/1779349246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63f81016c0 0x7f63f8197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:54:33.920 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.914+0000 7f63fd612640 1 --2- 172.21.15.169:0/1779349246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63f8103270 0x7f63f819f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:54:33.920 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.914+0000 7f63f7fff640 1 --2- 172.21.15.169:0/1779349246 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63f81028b0 0x7f63f8197fa0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f63ec000c00 tx=0x7f63ec005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:33.920 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.914+0000 7f63f5ffb640 1 -- 172.21.15.169:0/1779349246 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec043020 con 0x7f63f81028b0 2024-03-28T00:54:33.921 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.914+0000 7f63ff075640 1 -- 172.21.15.169:0/1779349246 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f819f910 con 0x7f63f81028b0 2024-03-28T00:54:33.921 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.914+0000 7f63f5ffb640 1 -- 172.21.15.169:0/1779349246 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f63ec0054e0 con 0x7f63f81028b0 2024-03-28T00:54:33.921 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.914+0000 7f63f5ffb640 1 -- 172.21.15.169:0/1779349246 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec003e60 con 0x7f63f81028b0 2024-03-28T00:54:33.921 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.915+0000 7f63ff075640 1 -- 172.21.15.169:0/1779349246 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f819fe60 con 0x7f63f81028b0 2024-03-28T00:54:33.921 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.915+0000 7f63ff075640 1 -- 172.21.15.169:0/1779349246 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f81089e0 con 0x7f63f81028b0 2024-03-28T00:54:33.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.916+0000 7f63f5ffb640 1 -- 172.21.15.169:0/1779349246 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f63ec046020 con 0x7f63f81028b0 2024-03-28T00:54:33.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.917+0000 7f63f5ffb640 1 --2- 172.21.15.169:0/1779349246 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f63e00a2800 0x7f63e00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:33.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.917+0000 7f63fce11640 1 --2- 172.21.15.169:0/1779349246 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f63e00a2800 0x7f63e00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:33.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.917+0000 7f63f5ffb640 1 -- 172.21.15.169:0/1779349246 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f63ec0f0df0 con 0x7f63f81028b0 2024-03-28T00:54:33.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.917+0000 7f63fce11640 1 --2- 172.21.15.169:0/1779349246 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f63e00a2800 0x7f63e00a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f63e8005d00 tx=0x7f63e8005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:33.924 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:33.921+0000 7f63f5ffb640 1 -- 172.21.15.169:0/1779349246 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f63ec04d020 con 0x7f63f81028b0 2024-03-28T00:54:34.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:34.098+0000 7f63ff075640 1 -- 172.21.15.169:0/1779349246 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f63f802d070 con 0x7f63f81028b0 2024-03-28T00:54:34.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:34 smithi169 ceph-mon[32274]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T00:54:34.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:34 smithi169 ceph-mon[32274]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T00:54:34.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:34 smithi169 ceph-mon[32274]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T00:54:34.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:34 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:34.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:34 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:34.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:34 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3980427328' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T00:54:34.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:34 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:34.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:34 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:34.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:34 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:34.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:34 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:34.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:34 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:34.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:34 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:54:34.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:34 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:34.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:34 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3980427328' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-28T00:54:34.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:34 smithi169 ceph-mon[32274]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T00:54:34.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:34 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1779349246' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T00:54:34.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:34 smithi174 ceph-mon[36060]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T00:54:34.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:34 smithi174 ceph-mon[36060]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T00:54:34.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:34 smithi174 ceph-mon[36060]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T00:54:34.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:34 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:34.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:34 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:34.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:34 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3980427328' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T00:54:34.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:34 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:34.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:34 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:34.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:34 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:34.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:34 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:34.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:34 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:34.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:34 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:54:34.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:34 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:34.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:34 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3980427328' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-28T00:54:34.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:34 smithi174 ceph-mon[36060]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T00:54:34.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:34 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1779349246' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T00:54:34.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:34 smithi177 ceph-mon[36206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T00:54:34.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:34 smithi177 ceph-mon[36206]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T00:54:34.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:34 smithi177 ceph-mon[36206]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T00:54:34.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:34 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:34.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:34 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:34.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:34 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3980427328' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T00:54:34.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:34 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:34.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:34 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:34.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:34 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:34.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:34 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:34.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:34 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:54:34.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:34 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:54:34.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:34 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:54:34.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:34 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3980427328' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-28T00:54:34.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:34 smithi177 ceph-mon[36206]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T00:54:34.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:34 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1779349246' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T00:54:35.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:35 smithi169 ceph-mon[32274]: pgmap v284: 97 pgs: 6 creating+peering, 29 active+clean, 62 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:35.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:35 smithi169 ceph-mon[32274]: mds.? [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] up:active 2024-03-28T00:54:35.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:35 smithi169 ceph-mon[32274]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T00:54:35.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:35 smithi169 ceph-mon[32274]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T00:54:35.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:35 smithi169 ceph-mon[32274]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T00:54:35.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:35 smithi169 ceph-mon[32274]: Cluster is now healthy 2024-03-28T00:54:35.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:35 smithi169 ceph-mon[32274]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T00:54:35.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:35 smithi174 ceph-mon[36060]: pgmap v284: 97 pgs: 6 creating+peering, 29 active+clean, 62 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:35.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:35 smithi174 ceph-mon[36060]: mds.? [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] up:active 2024-03-28T00:54:35.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:35 smithi174 ceph-mon[36060]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T00:54:35.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:35 smithi174 ceph-mon[36060]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T00:54:35.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:35 smithi174 ceph-mon[36060]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T00:54:35.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:35 smithi174 ceph-mon[36060]: Cluster is now healthy 2024-03-28T00:54:35.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:35 smithi174 ceph-mon[36060]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T00:54:35.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:35 smithi177 ceph-mon[36206]: pgmap v284: 97 pgs: 6 creating+peering, 29 active+clean, 62 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:35.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:35 smithi177 ceph-mon[36206]: mds.? [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] up:active 2024-03-28T00:54:35.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:35 smithi177 ceph-mon[36206]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T00:54:35.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:35 smithi177 ceph-mon[36206]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T00:54:35.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:35 smithi177 ceph-mon[36206]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T00:54:35.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:35 smithi177 ceph-mon[36206]: Cluster is now healthy 2024-03-28T00:54:35.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:35 smithi177 ceph-mon[36206]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T00:54:35.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.794+0000 7f63f5ffb640 1 -- 172.21.15.169:0/1779349246 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) v1 ==== 228+0+0 (secure 0 0 0) 0x7f63ec03c350 con 0x7f63f81028b0 2024-03-28T00:54:35.797 INFO:teuthology.orchestra.run.smithi169.stderr:pool 'cephfs_data_ec' created 2024-03-28T00:54:35.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.797+0000 7f63ff075640 1 -- 172.21.15.169:0/1779349246 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f63e00a2800 msgr2=0x7f63e00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:35.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.797+0000 7f63ff075640 1 --2- 172.21.15.169:0/1779349246 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f63e00a2800 0x7f63e00a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f63e8005d00 tx=0x7f63e8005c90 comp rx=0 tx=0).stop 2024-03-28T00:54:35.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.797+0000 7f63ff075640 1 -- 172.21.15.169:0/1779349246 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63f81028b0 msgr2=0x7f63f8197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:35.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.797+0000 7f63ff075640 1 --2- 172.21.15.169:0/1779349246 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63f81028b0 0x7f63f8197fa0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f63ec000c00 tx=0x7f63ec005dd0 comp rx=0 tx=0).stop 2024-03-28T00:54:35.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.797+0000 7f63ff075640 1 -- 172.21.15.169:0/1779349246 shutdown_connections 2024-03-28T00:54:35.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.797+0000 7f63ff075640 1 --2- 172.21.15.169:0/1779349246 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f63e00a2800 0x7f63e00a4cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:35.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.797+0000 7f63ff075640 1 --2- 172.21.15.169:0/1779349246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63f8103270 0x7f63f819f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:35.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.797+0000 7f63ff075640 1 --2- 172.21.15.169:0/1779349246 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63f81028b0 0x7f63f8197fa0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:35.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.797+0000 7f63ff075640 1 --2- 172.21.15.169:0/1779349246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63f81016c0 0x7f63f8197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:35.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.797+0000 7f63ff075640 1 -- 172.21.15.169:0/1779349246 >> 172.21.15.169:0/1779349246 conn(0x7f63f80fcc50 msgr2=0x7f63f8105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:35.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.797+0000 7f63ff075640 1 -- 172.21.15.169:0/1779349246 shutdown_connections 2024-03-28T00:54:35.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.797+0000 7f63ff075640 1 -- 172.21.15.169:0/1779349246 wait complete. 2024-03-28T00:54:35.819 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-28T00:54:35.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.928+0000 7fcf9ad77640 1 -- 172.21.15.169:0/3231667098 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf94101790 msgr2=0x7fcf94101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:35.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.928+0000 7fcf9ad77640 1 --2- 172.21.15.169:0/3231667098 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf94101790 0x7fcf94101c10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcf80009900 tx=0x7fcf8002bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:35.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.928+0000 7fcf9ad77640 1 -- 172.21.15.169:0/3231667098 shutdown_connections 2024-03-28T00:54:35.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.928+0000 7fcf9ad77640 1 --2- 172.21.15.169:0/3231667098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf94102150 0x7fcf941089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.928+0000 7fcf9ad77640 1 --2- 172.21.15.169:0/3231667098 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf94101790 0x7fcf94101c10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.928+0000 7fcf9ad77640 1 --2- 172.21.15.169:0/3231667098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf941005a0 0x7fcf941009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.928+0000 7fcf9ad77640 1 -- 172.21.15.169:0/3231667098 >> 172.21.15.169:0/3231667098 conn(0x7fcf940fbb30 msgr2=0x7fcf940fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.928+0000 7fcf9ad77640 1 -- 172.21.15.169:0/3231667098 shutdown_connections 2024-03-28T00:54:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.928+0000 7fcf9ad77640 1 -- 172.21.15.169:0/3231667098 wait complete. 2024-03-28T00:54:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.928+0000 7fcf9ad77640 1 Processor -- start 2024-03-28T00:54:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.928+0000 7fcf9ad77640 1 -- start start 2024-03-28T00:54:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.929+0000 7fcf9ad77640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf941005a0 0x7fcf941968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.929+0000 7fcf9ad77640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf94101790 0x7fcf94196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.929+0000 7fcf9ad77640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf94102150 0x7fcf9419de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.929+0000 7fcf9ad77640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9410c230 con 0x7fcf94102150 2024-03-28T00:54:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.929+0000 7fcf9ad77640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9410c0b0 con 0x7fcf94101790 2024-03-28T00:54:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.929+0000 7fcf9ad77640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9410c3b0 con 0x7fcf941005a0 2024-03-28T00:54:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.929+0000 7fcf99314640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf94102150 0x7fcf9419de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.929+0000 7fcf99314640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf94102150 0x7fcf9419de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34000/0 (socket says 172.21.15.169:34000) 2024-03-28T00:54:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.929+0000 7fcf99314640 1 -- 172.21.15.169:0/1024990051 learned_addr learned my addr 172.21.15.169:0/1024990051 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.929+0000 7fcf93fff640 1 --2- 172.21.15.169:0/1024990051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf94101790 0x7fcf94196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.929+0000 7fcf98b13640 1 --2- 172.21.15.169:0/1024990051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf941005a0 0x7fcf941968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.929+0000 7fcf99314640 1 -- 172.21.15.169:0/1024990051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf941005a0 msgr2=0x7fcf941968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.929+0000 7fcf99314640 1 --2- 172.21.15.169:0/1024990051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf941005a0 0x7fcf941968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.929+0000 7fcf99314640 1 -- 172.21.15.169:0/1024990051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf94101790 msgr2=0x7fcf94196e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.929+0000 7fcf99314640 1 --2- 172.21.15.169:0/1024990051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf94101790 0x7fcf94196e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.929+0000 7fcf99314640 1 -- 172.21.15.169:0/1024990051 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9419e500 con 0x7fcf94102150 2024-03-28T00:54:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.929+0000 7fcf99314640 1 --2- 172.21.15.169:0/1024990051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf94102150 0x7fcf9419de90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00eb10 tx=0x7fcf8c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.929+0000 7fcf91ffb640 1 -- 172.21.15.169:0/1024990051 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c004b20 con 0x7fcf94102150 2024-03-28T00:54:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.929+0000 7fcf91ffb640 1 -- 172.21.15.169:0/1024990051 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf8c004cc0 con 0x7fcf94102150 2024-03-28T00:54:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.929+0000 7fcf9ad77640 1 -- 172.21.15.169:0/1024990051 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf9419e7f0 con 0x7fcf94102150 2024-03-28T00:54:35.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.929+0000 7fcf91ffb640 1 -- 172.21.15.169:0/1024990051 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c010760 con 0x7fcf94102150 2024-03-28T00:54:35.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.930+0000 7fcf9ad77640 1 -- 172.21.15.169:0/1024990051 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9419ecc0 con 0x7fcf94102150 2024-03-28T00:54:35.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.930+0000 7fcf9ad77640 1 -- 172.21.15.169:0/1024990051 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf94107910 con 0x7fcf94102150 2024-03-28T00:54:35.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.931+0000 7fcf91ffb640 1 -- 172.21.15.169:0/1024990051 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fcf8c005020 con 0x7fcf94102150 2024-03-28T00:54:35.937 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.932+0000 7fcf91ffb640 1 --2- 172.21.15.169:0/1024990051 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcf840a2800 0x7fcf840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:35.937 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.932+0000 7fcf98b13640 1 --2- 172.21.15.169:0/1024990051 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcf840a2800 0x7fcf840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:35.937 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.932+0000 7fcf91ffb640 1 -- 172.21.15.169:0/1024990051 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8770+0+0 (secure 0 0 0) 0x7fcf8c0cf1f0 con 0x7fcf94102150 2024-03-28T00:54:35.937 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.932+0000 7fcf98b13640 1 --2- 172.21.15.169:0/1024990051 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcf840a2800 0x7fcf840a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcf880058c0 tx=0x7fcf8800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:35.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:35.937+0000 7fcf91ffb640 1 -- 172.21.15.169:0/1024990051 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcf8c09c070 con 0x7fcf94102150 2024-03-28T00:54:36.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.116+0000 7fcf9ad77640 1 -- 172.21.15.169:0/1024990051 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7fcf9402d070 con 0x7fcf94102150 2024-03-28T00:54:36.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.802+0000 7fcf91ffb640 1 -- 172.21.15.169:0/1024990051 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) v1 ==== 169+0+0 (secure 0 0 0) 0x7fcf8c09b970 con 0x7fcf94102150 2024-03-28T00:54:36.805 INFO:teuthology.orchestra.run.smithi169.stderr:set pool 4 allow_ec_overwrites to true 2024-03-28T00:54:36.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.805+0000 7fcf9ad77640 1 -- 172.21.15.169:0/1024990051 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcf840a2800 msgr2=0x7fcf840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:36.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.805+0000 7fcf9ad77640 1 --2- 172.21.15.169:0/1024990051 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcf840a2800 0x7fcf840a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcf880058c0 tx=0x7fcf8800a2d0 comp rx=0 tx=0).stop 2024-03-28T00:54:36.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.805+0000 7fcf9ad77640 1 -- 172.21.15.169:0/1024990051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf94102150 msgr2=0x7fcf9419de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:36.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.805+0000 7fcf9ad77640 1 --2- 172.21.15.169:0/1024990051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf94102150 0x7fcf9419de90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00eb10 tx=0x7fcf8c00c2d0 comp rx=0 tx=0).stop 2024-03-28T00:54:36.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.805+0000 7fcf9ad77640 1 -- 172.21.15.169:0/1024990051 shutdown_connections 2024-03-28T00:54:36.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.805+0000 7fcf9ad77640 1 --2- 172.21.15.169:0/1024990051 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcf840a2800 0x7fcf840a4cc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:36.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.805+0000 7fcf9ad77640 1 --2- 172.21.15.169:0/1024990051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf94102150 0x7fcf9419de90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:36.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.805+0000 7fcf9ad77640 1 --2- 172.21.15.169:0/1024990051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf94101790 0x7fcf94196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:36.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.805+0000 7fcf9ad77640 1 --2- 172.21.15.169:0/1024990051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf941005a0 0x7fcf941968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:36.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.805+0000 7fcf9ad77640 1 -- 172.21.15.169:0/1024990051 >> 172.21.15.169:0/1024990051 conn(0x7fcf940fbb30 msgr2=0x7fcf94104ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:36.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.805+0000 7fcf9ad77640 1 -- 172.21.15.169:0/1024990051 shutdown_connections 2024-03-28T00:54:36.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.805+0000 7fcf9ad77640 1 -- 172.21.15.169:0/1024990051 wait complete. 2024-03-28T00:54:36.830 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-28T00:54:36.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.937+0000 7f6fc6bc4640 1 -- 172.21.15.169:0/2257269725 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6fc0075660 msgr2=0x7f6fc0073ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:36.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.937+0000 7f6fc6bc4640 1 --2- 172.21.15.169:0/2257269725 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6fc0075660 0x7f6fc0073ab0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8009ca0 tx=0x7f6fb802c180 comp rx=0 tx=0).stop 2024-03-28T00:54:36.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.937+0000 7f6fc6bc4640 1 -- 172.21.15.169:0/2257269725 shutdown_connections 2024-03-28T00:54:36.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.937+0000 7f6fc6bc4640 1 --2- 172.21.15.169:0/2257269725 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fc0109560 0x7f6fc010b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:36.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.937+0000 7f6fc6bc4640 1 --2- 172.21.15.169:0/2257269725 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fc0074080 0x7f6fc00744e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:36.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.937+0000 7f6fc6bc4640 1 --2- 172.21.15.169:0/2257269725 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6fc0075660 0x7f6fc0073ab0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:36.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.937+0000 7f6fc6bc4640 1 -- 172.21.15.169:0/2257269725 >> 172.21.15.169:0/2257269725 conn(0x7f6fc00fcc40 msgr2=0x7f6fc00ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:36.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.937+0000 7f6fc6bc4640 1 -- 172.21.15.169:0/2257269725 shutdown_connections 2024-03-28T00:54:36.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.937+0000 7f6fc6bc4640 1 -- 172.21.15.169:0/2257269725 wait complete. 2024-03-28T00:54:36.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.937+0000 7f6fc6bc4640 1 Processor -- start 2024-03-28T00:54:36.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.937+0000 7f6fc6bc4640 1 -- start start 2024-03-28T00:54:36.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.937+0000 7f6fc6bc4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fc0074080 0x7f6fc0199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:36.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.937+0000 7f6fc6bc4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6fc0075660 0x7f6fc019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:36.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.937+0000 7f6fc6bc4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fc0109560 0x7f6fc01a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:36.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.937+0000 7f6fc6bc4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc010f590 con 0x7f6fc0075660 2024-03-28T00:54:36.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.937+0000 7f6fc6bc4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc010f410 con 0x7f6fc0109560 2024-03-28T00:54:36.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.937+0000 7f6fc6bc4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc010f710 con 0x7f6fc0074080 2024-03-28T00:54:36.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.938+0000 7f6fbffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6fc0075660 0x7f6fc019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:36.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.938+0000 7f6fbffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6fc0075660 0x7f6fc019a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48760/0 (socket says 172.21.15.169:48760) 2024-03-28T00:54:36.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.938+0000 7f6fbffff640 1 -- 172.21.15.169:0/705466430 learned_addr learned my addr 172.21.15.169:0/705466430 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:36.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.938+0000 7f6fc5161640 1 --2- 172.21.15.169:0/705466430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fc0109560 0x7f6fc01a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:36.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.938+0000 7f6fc4960640 1 --2- 172.21.15.169:0/705466430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fc0074080 0x7f6fc0199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:36.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.938+0000 7f6fbffff640 1 -- 172.21.15.169:0/705466430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fc0074080 msgr2=0x7f6fc0199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:36.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.938+0000 7f6fbffff640 1 --2- 172.21.15.169:0/705466430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fc0074080 0x7f6fc0199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:36.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.938+0000 7f6fbffff640 1 -- 172.21.15.169:0/705466430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fc0109560 msgr2=0x7f6fc01a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:36.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.938+0000 7f6fbffff640 1 --2- 172.21.15.169:0/705466430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fc0109560 0x7f6fc01a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:36.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.938+0000 7f6fbffff640 1 -- 172.21.15.169:0/705466430 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc01a1880 con 0x7f6fc0075660 2024-03-28T00:54:36.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.938+0000 7f6fbffff640 1 --2- 172.21.15.169:0/705466430 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6fc0075660 0x7f6fc019a1b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6fb400e9f0 tx=0x7f6fb400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:36.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.938+0000 7f6fbdffb640 1 -- 172.21.15.169:0/705466430 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb400cd80 con 0x7f6fc0075660 2024-03-28T00:54:36.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.938+0000 7f6fc6bc4640 1 -- 172.21.15.169:0/705466430 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc01a1b70 con 0x7f6fc0075660 2024-03-28T00:54:36.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.938+0000 7f6fbdffb640 1 -- 172.21.15.169:0/705466430 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6fb4010910 con 0x7f6fc0075660 2024-03-28T00:54:36.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.938+0000 7f6fbdffb640 1 -- 172.21.15.169:0/705466430 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb40188f0 con 0x7f6fc0075660 2024-03-28T00:54:36.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.938+0000 7f6fc6bc4640 1 -- 172.21.15.169:0/705466430 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc01a2110 con 0x7f6fc0075660 2024-03-28T00:54:36.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.939+0000 7f6fc6bc4640 1 -- 172.21.15.169:0/705466430 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc0108320 con 0x7f6fc0075660 2024-03-28T00:54:36.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.940+0000 7f6fbdffb640 1 -- 172.21.15.169:0/705466430 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6fb4010ac0 con 0x7f6fc0075660 2024-03-28T00:54:36.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.941+0000 7f6fbdffb640 1 --2- 172.21.15.169:0/705466430 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6fa80a2800 0x7f6fa80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:36.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.941+0000 7f6fc4960640 1 --2- 172.21.15.169:0/705466430 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6fa80a2800 0x7f6fa80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:36.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.941+0000 7f6fbdffb640 1 -- 172.21.15.169:0/705466430 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8770+0+0 (secure 0 0 0) 0x7f6fb40cf0c0 con 0x7f6fc0075660 2024-03-28T00:54:36.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.941+0000 7f6fc4960640 1 --2- 172.21.15.169:0/705466430 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6fa80a2800 0x7f6fa80a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6fb802c790 tx=0x7f6fb8032670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:36.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:36.945+0000 7f6fbdffb640 1 -- 172.21.15.169:0/705466430 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6fb409b8f0 con 0x7f6fc0075660 2024-03-28T00:54:37.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:36 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1779349246' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T00:54:37.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:36 smithi174 ceph-mon[36060]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T00:54:37.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:36 smithi174 ceph-mon[36060]: pgmap v287: 129 pgs: 18 creating+peering, 78 active+clean, 33 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 8 op/s 2024-03-28T00:54:37.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:36 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1024990051' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-28T00:54:37.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.158+0000 7f6fc6bc4640 1 -- 172.21.15.169:0/705466430 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f6fc00622a0 con 0x7f6fc0075660 2024-03-28T00:54:37.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:36 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1779349246' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T00:54:37.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:36 smithi177 ceph-mon[36206]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T00:54:37.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:36 smithi177 ceph-mon[36206]: pgmap v287: 129 pgs: 18 creating+peering, 78 active+clean, 33 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 8 op/s 2024-03-28T00:54:37.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:36 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1024990051' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-28T00:54:37.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:36 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1779349246' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T00:54:37.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:36 smithi169 ceph-mon[32274]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T00:54:37.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:36 smithi169 ceph-mon[32274]: pgmap v287: 129 pgs: 18 creating+peering, 78 active+clean, 33 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 8 op/s 2024-03-28T00:54:37.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:36 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1024990051' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-28T00:54:37.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.803+0000 7f6fbdffb640 1 -- 172.21.15.169:0/705466430 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7f6fb409f990 con 0x7f6fc0075660 2024-03-28T00:54:37.806 INFO:teuthology.orchestra.run.smithi169.stderr:added data pool 4 to fsmap 2024-03-28T00:54:37.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.806+0000 7f6fc6bc4640 1 -- 172.21.15.169:0/705466430 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6fa80a2800 msgr2=0x7f6fa80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:37.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.806+0000 7f6fc6bc4640 1 --2- 172.21.15.169:0/705466430 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6fa80a2800 0x7f6fa80a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6fb802c790 tx=0x7f6fb8032670 comp rx=0 tx=0).stop 2024-03-28T00:54:37.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.806+0000 7f6fc6bc4640 1 -- 172.21.15.169:0/705466430 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6fc0075660 msgr2=0x7f6fc019a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:37.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.806+0000 7f6fc6bc4640 1 --2- 172.21.15.169:0/705466430 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6fc0075660 0x7f6fc019a1b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6fb400e9f0 tx=0x7f6fb400eeb0 comp rx=0 tx=0).stop 2024-03-28T00:54:37.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.806+0000 7f6fc6bc4640 1 -- 172.21.15.169:0/705466430 shutdown_connections 2024-03-28T00:54:37.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.806+0000 7f6fc6bc4640 1 --2- 172.21.15.169:0/705466430 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6fa80a2800 0x7f6fa80a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:37.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.806+0000 7f6fc6bc4640 1 --2- 172.21.15.169:0/705466430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fc0109560 0x7f6fc01a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:37.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.806+0000 7f6fc6bc4640 1 --2- 172.21.15.169:0/705466430 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6fc0075660 0x7f6fc019a1b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:37.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.806+0000 7f6fc6bc4640 1 --2- 172.21.15.169:0/705466430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fc0074080 0x7f6fc0199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:37.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.806+0000 7f6fc6bc4640 1 -- 172.21.15.169:0/705466430 >> 172.21.15.169:0/705466430 conn(0x7f6fc00fcc40 msgr2=0x7f6fc01054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:37.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.806+0000 7f6fc6bc4640 1 -- 172.21.15.169:0/705466430 shutdown_connections 2024-03-28T00:54:37.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.806+0000 7f6fc6bc4640 1 -- 172.21.15.169:0/705466430 wait complete. 2024-03-28T00:54:37.822 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:54:37.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.923+0000 7f97df5ab640 1 -- 172.21.15.169:0/1275553226 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97d8102d10 msgr2=0x7f97d8103130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:37.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.923+0000 7f97df5ab640 1 --2- 172.21.15.169:0/1275553226 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97d8102d10 0x7f97d8103130 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f97c8009900 tx=0x7f97c802bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:37.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.923+0000 7f97df5ab640 1 -- 172.21.15.169:0/1275553226 shutdown_connections 2024-03-28T00:54:37.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.923+0000 7f97df5ab640 1 --2- 172.21.15.169:0/1275553226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97d8100140 0x7f97d8109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:37.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.923+0000 7f97df5ab640 1 --2- 172.21.15.169:0/1275553226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97d80ff780 0x7f97d80ffc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:37.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.923+0000 7f97df5ab640 1 --2- 172.21.15.169:0/1275553226 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97d8102d10 0x7f97d8103130 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:37.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.923+0000 7f97df5ab640 1 -- 172.21.15.169:0/1275553226 >> 172.21.15.169:0/1275553226 conn(0x7f97d80fb2e0 msgr2=0x7f97d80fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:37.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.923+0000 7f97df5ab640 1 -- 172.21.15.169:0/1275553226 shutdown_connections 2024-03-28T00:54:37.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.923+0000 7f97df5ab640 1 -- 172.21.15.169:0/1275553226 wait complete. 2024-03-28T00:54:37.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.923+0000 7f97df5ab640 1 Processor -- start 2024-03-28T00:54:37.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.923+0000 7f97df5ab640 1 -- start start 2024-03-28T00:54:37.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.923+0000 7f97df5ab640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97d80ff780 0x7f97d8199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:37.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97df5ab640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97d8100140 0x7f97d819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:37.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97df5ab640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97d8102d10 0x7f97d81a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:37.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97df5ab640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d810d340 con 0x7f97d8102d10 2024-03-28T00:54:37.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97df5ab640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d810d1c0 con 0x7f97d80ff780 2024-03-28T00:54:37.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97df5ab640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d810d4c0 con 0x7f97d8100140 2024-03-28T00:54:37.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97ddb48640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97d8102d10 0x7f97d81a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:37.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97ddb48640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97d8102d10 0x7f97d81a11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48780/0 (socket says 172.21.15.169:48780) 2024-03-28T00:54:37.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97ddb48640 1 -- 172.21.15.169:0/2999372768 learned_addr learned my addr 172.21.15.169:0/2999372768 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:37.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97dd347640 1 --2- 172.21.15.169:0/2999372768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97d80ff780 0x7f97d8199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:37.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97dcb46640 1 --2- 172.21.15.169:0/2999372768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97d8100140 0x7f97d819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:37.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97ddb48640 1 -- 172.21.15.169:0/2999372768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97d8100140 msgr2=0x7f97d819a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:37.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97ddb48640 1 --2- 172.21.15.169:0/2999372768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97d8100140 0x7f97d819a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:37.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97ddb48640 1 -- 172.21.15.169:0/2999372768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97d80ff780 msgr2=0x7f97d8199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:37.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97ddb48640 1 --2- 172.21.15.169:0/2999372768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97d80ff780 0x7f97d8199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:37.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97ddb48640 1 -- 172.21.15.169:0/2999372768 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d81a1860 con 0x7f97d8102d10 2024-03-28T00:54:37.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97dd347640 1 --2- 172.21.15.169:0/2999372768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97d80ff780 0x7f97d8199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:37.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97ddb48640 1 --2- 172.21.15.169:0/2999372768 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97d8102d10 0x7f97d81a11c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f97d0009fd0 tx=0x7f97d000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:37.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97ce7fc640 1 -- 172.21.15.169:0/2999372768 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0017070 con 0x7f97d8102d10 2024-03-28T00:54:37.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97ce7fc640 1 -- 172.21.15.169:0/2999372768 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97d0003e50 con 0x7f97d8102d10 2024-03-28T00:54:37.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97df5ab640 1 -- 172.21.15.169:0/2999372768 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d81a1b50 con 0x7f97d8102d10 2024-03-28T00:54:37.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97ce7fc640 1 -- 172.21.15.169:0/2999372768 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d00106c0 con 0x7f97d8102d10 2024-03-28T00:54:37.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.924+0000 7f97df5ab640 1 -- 172.21.15.169:0/2999372768 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d81a20c0 con 0x7f97d8102d10 2024-03-28T00:54:37.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.925+0000 7f97df5ab640 1 -- 172.21.15.169:0/2999372768 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d81a2290 con 0x7f97d8102d10 2024-03-28T00:54:37.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.926+0000 7f97ce7fc640 1 -- 172.21.15.169:0/2999372768 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f97d0003970 con 0x7f97d8102d10 2024-03-28T00:54:37.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.927+0000 7f97ce7fc640 1 --2- 172.21.15.169:0/2999372768 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f97c40a2880 0x7f97c40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:37.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.927+0000 7f97dd347640 1 --2- 172.21.15.169:0/2999372768 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f97c40a2880 0x7f97c40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:37.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.927+0000 7f97ce7fc640 1 -- 172.21.15.169:0/2999372768 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f97d00ced10 con 0x7f97d8102d10 2024-03-28T00:54:37.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.928+0000 7f97dd347640 1 --2- 172.21.15.169:0/2999372768 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f97c40a2880 0x7f97c40a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f97c8000c00 tx=0x7f97c8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:37.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:37.931+0000 7f97ce7fc640 1 -- 172.21.15.169:0/2999372768 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f97d009b5a0 con 0x7f97d8102d10 2024-03-28T00:54:38.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:37 smithi174 ceph-mon[36060]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T00:54:38.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:37 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1024990051' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-28T00:54:38.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:37 smithi174 ceph-mon[36060]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T00:54:38.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:37 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/705466430' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-28T00:54:38.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.145+0000 7f97df5ab640 1 -- 172.21.15.169:0/2999372768 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97d81a2620 con 0x7f97d8102d10 2024-03-28T00:54:38.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.146+0000 7f97ce7fc640 1 -- 172.21.15.169:0/2999372768 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11348 (secure 0 0 0) 0x7f97d009f640 con 0x7f97d8102d10 2024-03-28T00:54:38.149 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:38.149 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24452,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:54:37.160291+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24479,"qdb_cluster":[24479]},"id":1}]} 2024-03-28T00:54:38.149 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 17 2024-03-28T00:54:38.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.148+0000 7f97df5ab640 1 -- 172.21.15.169:0/2999372768 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f97c40a2880 msgr2=0x7f97c40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:38.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.148+0000 7f97df5ab640 1 --2- 172.21.15.169:0/2999372768 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f97c40a2880 0x7f97c40a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f97c8000c00 tx=0x7f97c8005fd0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.148+0000 7f97df5ab640 1 -- 172.21.15.169:0/2999372768 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97d8102d10 msgr2=0x7f97d81a11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:38.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.148+0000 7f97df5ab640 1 --2- 172.21.15.169:0/2999372768 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97d8102d10 0x7f97d81a11c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f97d0009fd0 tx=0x7f97d000b260 comp rx=0 tx=0).stop 2024-03-28T00:54:38.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.148+0000 7f97df5ab640 1 -- 172.21.15.169:0/2999372768 shutdown_connections 2024-03-28T00:54:38.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.148+0000 7f97df5ab640 1 --2- 172.21.15.169:0/2999372768 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f97c40a2880 0x7f97c40a4d40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.148+0000 7f97df5ab640 1 --2- 172.21.15.169:0/2999372768 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97d8102d10 0x7f97d81a11c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.148+0000 7f97df5ab640 1 --2- 172.21.15.169:0/2999372768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97d8100140 0x7f97d819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.148+0000 7f97df5ab640 1 --2- 172.21.15.169:0/2999372768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97d80ff780 0x7f97d8199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.148+0000 7f97df5ab640 1 -- 172.21.15.169:0/2999372768 >> 172.21.15.169:0/2999372768 conn(0x7f97d80fb2e0 msgr2=0x7f97d8100890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:38.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.148+0000 7f97df5ab640 1 -- 172.21.15.169:0/2999372768 shutdown_connections 2024-03-28T00:54:38.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.148+0000 7f97df5ab640 1 -- 172.21.15.169:0/2999372768 wait complete. 2024-03-28T00:54:38.161 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:37 smithi169 ceph-mon[32274]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T00:54:38.161 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:37 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1024990051' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-28T00:54:38.161 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:37 smithi169 ceph-mon[32274]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T00:54:38.161 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:37 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/705466430' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-28T00:54:38.162 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T00:54:38.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:37 smithi177 ceph-mon[36206]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T00:54:38.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:37 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1024990051' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-28T00:54:38.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:37 smithi177 ceph-mon[36206]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T00:54:38.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:37 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/705466430' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-28T00:54:38.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.264+0000 7fa49f95b640 1 -- 172.21.15.169:0/22942460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4981028e0 msgr2=0x7fa498102d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:38.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.264+0000 7fa49f95b640 1 --2- 172.21.15.169:0/22942460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4981028e0 0x7fa498102d60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa48c009ca0 tx=0x7fa48c02c0f0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.265+0000 7fa49f95b640 1 -- 172.21.15.169:0/22942460 shutdown_connections 2024-03-28T00:54:38.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.265+0000 7fa49f95b640 1 --2- 172.21.15.169:0/22942460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4981032a0 0x7fa498109b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.265+0000 7fa49f95b640 1 --2- 172.21.15.169:0/22942460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4981028e0 0x7fa498102d60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.265+0000 7fa49f95b640 1 --2- 172.21.15.169:0/22942460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4981016f0 0x7fa498101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.265+0000 7fa49f95b640 1 -- 172.21.15.169:0/22942460 >> 172.21.15.169:0/22942460 conn(0x7fa4980fcc40 msgr2=0x7fa4980ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:38.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.265+0000 7fa49f95b640 1 -- 172.21.15.169:0/22942460 shutdown_connections 2024-03-28T00:54:38.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.265+0000 7fa49f95b640 1 -- 172.21.15.169:0/22942460 wait complete. 2024-03-28T00:54:38.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.265+0000 7fa49f95b640 1 Processor -- start 2024-03-28T00:54:38.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.265+0000 7fa49f95b640 1 -- start start 2024-03-28T00:54:38.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.265+0000 7fa49f95b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4981016f0 0x7fa498197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:38.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.265+0000 7fa49f95b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4981028e0 0x7fa498197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:38.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.265+0000 7fa49f95b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4981032a0 0x7fa49819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:38.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.265+0000 7fa49f95b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49810d370 con 0x7fa4981032a0 2024-03-28T00:54:38.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.265+0000 7fa49f95b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49810d1f0 con 0x7fa4981016f0 2024-03-28T00:54:38.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.265+0000 7fa49f95b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49810d4f0 con 0x7fa4981028e0 2024-03-28T00:54:38.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.265+0000 7fa49def8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4981032a0 0x7fa49819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:38.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.265+0000 7fa49def8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4981032a0 0x7fa49819efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48796/0 (socket says 172.21.15.169:48796) 2024-03-28T00:54:38.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.265+0000 7fa49def8640 1 -- 172.21.15.169:0/744349472 learned_addr learned my addr 172.21.15.169:0/744349472 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:38.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.265+0000 7fa49d6f7640 1 --2- 172.21.15.169:0/744349472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4981016f0 0x7fa498197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:38.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.266+0000 7fa49cef6640 1 --2- 172.21.15.169:0/744349472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4981028e0 0x7fa498197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:38.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.266+0000 7fa49def8640 1 -- 172.21.15.169:0/744349472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4981028e0 msgr2=0x7fa498197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:38.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.266+0000 7fa49def8640 1 --2- 172.21.15.169:0/744349472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4981028e0 0x7fa498197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.266+0000 7fa49def8640 1 -- 172.21.15.169:0/744349472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4981016f0 msgr2=0x7fa498197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:38.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.266+0000 7fa49def8640 1 --2- 172.21.15.169:0/744349472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4981016f0 0x7fa498197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.266+0000 7fa49def8640 1 -- 172.21.15.169:0/744349472 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49819f630 con 0x7fa4981032a0 2024-03-28T00:54:38.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.266+0000 7fa49d6f7640 1 --2- 172.21.15.169:0/744349472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4981016f0 0x7fa498197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:38.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.266+0000 7fa49cef6640 1 --2- 172.21.15.169:0/744349472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4981028e0 0x7fa498197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:54:38.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.266+0000 7fa49def8640 1 --2- 172.21.15.169:0/744349472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4981032a0 0x7fa49819efc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa48800e970 tx=0x7fa48800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:38.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.266+0000 7fa4967fc640 1 -- 172.21.15.169:0/744349472 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48800ccc0 con 0x7fa4981032a0 2024-03-28T00:54:38.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.266+0000 7fa49f95b640 1 -- 172.21.15.169:0/744349472 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa49819f920 con 0x7fa4981032a0 2024-03-28T00:54:38.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.266+0000 7fa4967fc640 1 -- 172.21.15.169:0/744349472 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa48800ce60 con 0x7fa4981032a0 2024-03-28T00:54:38.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.266+0000 7fa4967fc640 1 -- 172.21.15.169:0/744349472 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa488010490 con 0x7fa4981032a0 2024-03-28T00:54:38.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.266+0000 7fa49f95b640 1 -- 172.21.15.169:0/744349472 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa49819fe20 con 0x7fa4981032a0 2024-03-28T00:54:38.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.267+0000 7fa49f95b640 1 -- 172.21.15.169:0/744349472 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa498108a10 con 0x7fa4981032a0 2024-03-28T00:54:38.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.268+0000 7fa4967fc640 1 -- 172.21.15.169:0/744349472 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa488005020 con 0x7fa4981032a0 2024-03-28T00:54:38.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.269+0000 7fa4967fc640 1 --2- 172.21.15.169:0/744349472 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa4800a2800 0x7fa4800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:38.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.269+0000 7fa49d6f7640 1 --2- 172.21.15.169:0/744349472 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa4800a2800 0x7fa4800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:38.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.269+0000 7fa4967fc640 1 -- 172.21.15.169:0/744349472 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa4880cf1a0 con 0x7fa4981032a0 2024-03-28T00:54:38.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.269+0000 7fa49d6f7640 1 --2- 172.21.15.169:0/744349472 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa4800a2800 0x7fa4800a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa498102740 tx=0x7fa490005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:38.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.273+0000 7fa4967fc640 1 -- 172.21.15.169:0/744349472 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa48809c000 con 0x7fa4981032a0 2024-03-28T00:54:38.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.449+0000 7fa49f95b640 1 -- 172.21.15.169:0/744349472 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4980622a0 con 0x7fa4981032a0 2024-03-28T00:54:38.453 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.450+0000 7fa4967fc640 1 -- 172.21.15.169:0/744349472 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24199 (secure 0 0 0) 0x7fa48809b900 con 0x7fa4981032a0 2024-03-28T00:54:38.453 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:38.454 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":89,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:54:37.160100+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:54:38.455 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.452+0000 7fa49f95b640 1 -- 172.21.15.169:0/744349472 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa4800a2800 msgr2=0x7fa4800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:38.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.452+0000 7fa49f95b640 1 --2- 172.21.15.169:0/744349472 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa4800a2800 0x7fa4800a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa498102740 tx=0x7fa490005f90 comp rx=0 tx=0).stop 2024-03-28T00:54:38.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.452+0000 7fa49f95b640 1 -- 172.21.15.169:0/744349472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4981032a0 msgr2=0x7fa49819efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:38.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.452+0000 7fa49f95b640 1 --2- 172.21.15.169:0/744349472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4981032a0 0x7fa49819efc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa48800e970 tx=0x7fa48800ed70 comp rx=0 tx=0).stop 2024-03-28T00:54:38.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.452+0000 7fa49f95b640 1 -- 172.21.15.169:0/744349472 shutdown_connections 2024-03-28T00:54:38.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.452+0000 7fa49f95b640 1 --2- 172.21.15.169:0/744349472 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa4800a2800 0x7fa4800a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.452+0000 7fa49f95b640 1 --2- 172.21.15.169:0/744349472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4981032a0 0x7fa49819efc0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.452+0000 7fa49f95b640 1 --2- 172.21.15.169:0/744349472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4981028e0 0x7fa498197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.452+0000 7fa49f95b640 1 --2- 172.21.15.169:0/744349472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4981016f0 0x7fa498197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.452+0000 7fa49f95b640 1 -- 172.21.15.169:0/744349472 >> 172.21.15.169:0/744349472 conn(0x7fa4980fcc40 msgr2=0x7fa498105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:38.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.452+0000 7fa49f95b640 1 -- 172.21.15.169:0/744349472 shutdown_connections 2024-03-28T00:54:38.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.453+0000 7fa49f95b640 1 -- 172.21.15.169:0/744349472 wait complete. 2024-03-28T00:54:38.467 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T00:54:38.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.566+0000 7f2d7b59e640 1 -- 172.21.15.169:0/1255490665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d7c101f70 msgr2=0x7f2d7c104360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:38.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.566+0000 7f2d7b59e640 1 --2- 172.21.15.169:0/1255490665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d7c101f70 0x7f2d7c104360 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c009ca0 tx=0x7f2d6c02c0f0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.567+0000 7f2d7b59e640 1 -- 172.21.15.169:0/1255490665 shutdown_connections 2024-03-28T00:54:38.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.567+0000 7f2d7b59e640 1 --2- 172.21.15.169:0/1255490665 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d7c1071d0 0x7f2d7c109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.567+0000 7f2d7b59e640 1 --2- 172.21.15.169:0/1255490665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d7c1048a0 0x7f2d7c106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.567+0000 7f2d7b59e640 1 --2- 172.21.15.169:0/1255490665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d7c101f70 0x7f2d7c104360 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.567+0000 7f2d7b59e640 1 -- 172.21.15.169:0/1255490665 >> 172.21.15.169:0/1255490665 conn(0x7f2d7c0fb980 msgr2=0x7f2d7c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:38.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.567+0000 7f2d7b59e640 1 -- 172.21.15.169:0/1255490665 shutdown_connections 2024-03-28T00:54:38.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.567+0000 7f2d7b59e640 1 -- 172.21.15.169:0/1255490665 wait complete. 2024-03-28T00:54:38.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.567+0000 7f2d7b59e640 1 Processor -- start 2024-03-28T00:54:38.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.567+0000 7f2d7b59e640 1 -- start start 2024-03-28T00:54:38.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.567+0000 7f2d7b59e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d7c101f70 0x7f2d7c197850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:38.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.567+0000 7f2d7b59e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d7c1048a0 0x7f2d7c197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:38.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.567+0000 7f2d7b59e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d7c1071d0 0x7f2d7c19ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:38.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.568+0000 7f2d7b59e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c10d180 con 0x7f2d7c101f70 2024-03-28T00:54:38.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.568+0000 7f2d7b59e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c10d000 con 0x7f2d7c1048a0 2024-03-28T00:54:38.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.568+0000 7f2d7b59e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c10d300 con 0x7f2d7c1071d0 2024-03-28T00:54:38.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.568+0000 7f2d7a59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d7c101f70 0x7f2d7c197850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:38.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.568+0000 7f2d7a59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d7c101f70 0x7f2d7c197850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48808/0 (socket says 172.21.15.169:48808) 2024-03-28T00:54:38.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.568+0000 7f2d7a59c640 1 -- 172.21.15.169:0/2423484652 learned_addr learned my addr 172.21.15.169:0/2423484652 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:38.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.568+0000 7f2d79d9b640 1 --2- 172.21.15.169:0/2423484652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d7c1048a0 0x7f2d7c197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:38.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.568+0000 7f2d7ad9d640 1 --2- 172.21.15.169:0/2423484652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d7c1071d0 0x7f2d7c19ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:38.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.568+0000 7f2d7a59c640 1 -- 172.21.15.169:0/2423484652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d7c1071d0 msgr2=0x7f2d7c19ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:38.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.568+0000 7f2d7a59c640 1 --2- 172.21.15.169:0/2423484652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d7c1071d0 0x7f2d7c19ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.568+0000 7f2d7a59c640 1 -- 172.21.15.169:0/2423484652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d7c1048a0 msgr2=0x7f2d7c197d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:38.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.568+0000 7f2d7a59c640 1 --2- 172.21.15.169:0/2423484652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d7c1048a0 0x7f2d7c197d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.568+0000 7f2d7a59c640 1 -- 172.21.15.169:0/2423484652 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c19f470 con 0x7f2d7c101f70 2024-03-28T00:54:38.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.568+0000 7f2d7a59c640 1 --2- 172.21.15.169:0/2423484652 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d7c101f70 0x7f2d7c197850 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c009dc0 tx=0x7f2d6c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:38.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.568+0000 7f2d6b7fe640 1 -- 172.21.15.169:0/2423484652 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c041020 con 0x7f2d7c101f70 2024-03-28T00:54:38.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.568+0000 7f2d7b59e640 1 -- 172.21.15.169:0/2423484652 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7c19f700 con 0x7f2d7c101f70 2024-03-28T00:54:38.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.568+0000 7f2d6b7fe640 1 -- 172.21.15.169:0/2423484652 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d6c034cd0 con 0x7f2d7c101f70 2024-03-28T00:54:38.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.568+0000 7f2d6b7fe640 1 -- 172.21.15.169:0/2423484652 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c039040 con 0x7f2d7c101f70 2024-03-28T00:54:38.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.568+0000 7f2d7b59e640 1 -- 172.21.15.169:0/2423484652 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7c19fc40 con 0x7f2d7c101f70 2024-03-28T00:54:38.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.569+0000 7f2d7b59e640 1 -- 172.21.15.169:0/2423484652 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7c0fd470 con 0x7f2d7c101f70 2024-03-28T00:54:38.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.570+0000 7f2d6b7fe640 1 -- 172.21.15.169:0/2423484652 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2d6c044020 con 0x7f2d7c101f70 2024-03-28T00:54:38.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.571+0000 7f2d6b7fe640 1 --2- 172.21.15.169:0/2423484652 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2d640a2800 0x7f2d640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:38.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.571+0000 7f2d79d9b640 1 --2- 172.21.15.169:0/2423484652 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2d640a2800 0x7f2d640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:38.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.571+0000 7f2d6b7fe640 1 -- 172.21.15.169:0/2423484652 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f2d6c0f0f00 con 0x7f2d7c101f70 2024-03-28T00:54:38.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.571+0000 7f2d79d9b640 1 --2- 172.21.15.169:0/2423484652 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2d640a2800 0x7f2d640a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c198d60 tx=0x7f2d70005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:38.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.574+0000 7f2d6b7fe640 1 -- 172.21.15.169:0/2423484652 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2d6c04b020 con 0x7f2d7c101f70 2024-03-28T00:54:38.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.751+0000 7f2d7b59e640 1 -- 172.21.15.169:0/2423484652 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d7c0622a0 con 0x7f2d7c101f70 2024-03-28T00:54:38.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.753+0000 7f2d6b7fe640 1 -- 172.21.15.169:0/2423484652 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24199 (secure 0 0 0) 0x7f2d6c0bdd60 con 0x7f2d7c101f70 2024-03-28T00:54:38.756 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:38.756 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":89,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:54:37.160100+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:54:38.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.755+0000 7f2d7b59e640 1 -- 172.21.15.169:0/2423484652 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2d640a2800 msgr2=0x7f2d640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:38.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.755+0000 7f2d7b59e640 1 --2- 172.21.15.169:0/2423484652 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2d640a2800 0x7f2d640a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c198d60 tx=0x7f2d70005f90 comp rx=0 tx=0).stop 2024-03-28T00:54:38.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.755+0000 7f2d7b59e640 1 -- 172.21.15.169:0/2423484652 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d7c101f70 msgr2=0x7f2d7c197850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:38.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.755+0000 7f2d7b59e640 1 --2- 172.21.15.169:0/2423484652 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d7c101f70 0x7f2d7c197850 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c009dc0 tx=0x7f2d6c032f20 comp rx=0 tx=0).stop 2024-03-28T00:54:38.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.755+0000 7f2d7b59e640 1 -- 172.21.15.169:0/2423484652 shutdown_connections 2024-03-28T00:54:38.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.755+0000 7f2d7b59e640 1 --2- 172.21.15.169:0/2423484652 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2d640a2800 0x7f2d640a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.755+0000 7f2d7b59e640 1 --2- 172.21.15.169:0/2423484652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d7c1071d0 0x7f2d7c19ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.755+0000 7f2d7b59e640 1 --2- 172.21.15.169:0/2423484652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d7c1048a0 0x7f2d7c197d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.755+0000 7f2d7b59e640 1 --2- 172.21.15.169:0/2423484652 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d7c101f70 0x7f2d7c197850 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.755+0000 7f2d7b59e640 1 -- 172.21.15.169:0/2423484652 >> 172.21.15.169:0/2423484652 conn(0x7f2d7c0fb980 msgr2=0x7f2d7c1028d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:38.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.755+0000 7f2d7b59e640 1 -- 172.21.15.169:0/2423484652 shutdown_connections 2024-03-28T00:54:38.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.755+0000 7f2d7b59e640 1 -- 172.21.15.169:0/2423484652 wait complete. 2024-03-28T00:54:38.788 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:54:38.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.889+0000 7f7fb8797640 1 -- 172.21.15.169:0/71106892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fb01005c0 msgr2=0x7f7fb01009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:38.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.889+0000 7f7fb8797640 1 --2- 172.21.15.169:0/71106892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fb01005c0 0x7f7fb01009e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009ca0 tx=0x7f7fa802c0f0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.890+0000 7f7fb8797640 1 -- 172.21.15.169:0/71106892 shutdown_connections 2024-03-28T00:54:38.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.890+0000 7f7fb8797640 1 --2- 172.21.15.169:0/71106892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7fb0102170 0x7f7fb0108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.890+0000 7f7fb8797640 1 --2- 172.21.15.169:0/71106892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fb01017b0 0x7f7fb0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.890+0000 7f7fb8797640 1 --2- 172.21.15.169:0/71106892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fb01005c0 0x7f7fb01009e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.890+0000 7f7fb8797640 1 -- 172.21.15.169:0/71106892 >> 172.21.15.169:0/71106892 conn(0x7f7fb00fbb30 msgr2=0x7f7fb00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:38.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.890+0000 7f7fb8797640 1 -- 172.21.15.169:0/71106892 shutdown_connections 2024-03-28T00:54:38.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.890+0000 7f7fb8797640 1 -- 172.21.15.169:0/71106892 wait complete. 2024-03-28T00:54:38.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.890+0000 7f7fb8797640 1 Processor -- start 2024-03-28T00:54:38.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.890+0000 7f7fb8797640 1 -- start start 2024-03-28T00:54:38.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.890+0000 7f7fb8797640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7fb01005c0 0x7f7fb0197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:38.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.890+0000 7f7fb8797640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fb01017b0 0x7f7fb0197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:38.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.890+0000 7f7fb8797640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fb0102170 0x7f7fb019ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:38.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.890+0000 7f7fb8797640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb010c210 con 0x7f7fb01017b0 2024-03-28T00:54:38.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.890+0000 7f7fb8797640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb010c090 con 0x7f7fb01005c0 2024-03-28T00:54:38.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.890+0000 7f7fb8797640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb010c390 con 0x7f7fb0102170 2024-03-28T00:54:38.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.890+0000 7f7fb5d32640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fb01017b0 0x7f7fb0197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:38.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.890+0000 7f7fb5d32640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fb01017b0 0x7f7fb0197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48822/0 (socket says 172.21.15.169:48822) 2024-03-28T00:54:38.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.890+0000 7f7fb5d32640 1 -- 172.21.15.169:0/2661804980 learned_addr learned my addr 172.21.15.169:0/2661804980 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:38.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.890+0000 7f7fb6533640 1 --2- 172.21.15.169:0/2661804980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7fb01005c0 0x7f7fb0197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:38.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.891+0000 7f7fb6d34640 1 --2- 172.21.15.169:0/2661804980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fb0102170 0x7f7fb019ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:38.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.891+0000 7f7fb6533640 1 -- 172.21.15.169:0/2661804980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fb0102170 msgr2=0x7f7fb019ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:38.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.891+0000 7f7fb6533640 1 --2- 172.21.15.169:0/2661804980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fb0102170 0x7f7fb019ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.891+0000 7f7fb6533640 1 -- 172.21.15.169:0/2661804980 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fb01017b0 msgr2=0x7f7fb0197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:38.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.891+0000 7f7fb6533640 1 --2- 172.21.15.169:0/2661804980 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fb01017b0 0x7f7fb0197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:38.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.891+0000 7f7fb6533640 1 -- 172.21.15.169:0/2661804980 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb019f5b0 con 0x7f7fb01005c0 2024-03-28T00:54:38.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.891+0000 7f7fb5d32640 1 --2- 172.21.15.169:0/2661804980 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fb01017b0 0x7f7fb0197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:38.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.891+0000 7f7fb6d34640 1 --2- 172.21.15.169:0/2661804980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fb0102170 0x7f7fb019ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:54:38.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.891+0000 7f7fb6533640 1 --2- 172.21.15.169:0/2661804980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7fb01005c0 0x7f7fb0197990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8037040 tx=0x7f7fa8032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:38.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.891+0000 7f7fa77fe640 1 -- 172.21.15.169:0/2661804980 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8041020 con 0x7f7fb01005c0 2024-03-28T00:54:38.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.891+0000 7f7fa77fe640 1 -- 172.21.15.169:0/2661804980 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7fa8034d90 con 0x7f7fb01005c0 2024-03-28T00:54:38.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.891+0000 7f7fb8797640 1 -- 172.21.15.169:0/2661804980 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb019f840 con 0x7f7fb01005c0 2024-03-28T00:54:38.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.891+0000 7f7fa77fe640 1 -- 172.21.15.169:0/2661804980 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8039040 con 0x7f7fb01005c0 2024-03-28T00:54:38.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.891+0000 7f7fb8797640 1 -- 172.21.15.169:0/2661804980 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb019fd00 con 0x7f7fb01005c0 2024-03-28T00:54:38.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.892+0000 7f7fb8797640 1 -- 172.21.15.169:0/2661804980 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb019ff60 con 0x7f7fb01005c0 2024-03-28T00:54:38.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.893+0000 7f7fa77fe640 1 -- 172.21.15.169:0/2661804980 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7fa8044020 con 0x7f7fb01005c0 2024-03-28T00:54:38.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.893+0000 7f7fa77fe640 1 --2- 172.21.15.169:0/2661804980 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7f9c0a2800 0x7f7f9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:38.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.893+0000 7f7fb5d32640 1 --2- 172.21.15.169:0/2661804980 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7f9c0a2800 0x7f7f9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:38.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.893+0000 7f7fa77fe640 1 -- 172.21.15.169:0/2661804980 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f7fa80f0b80 con 0x7f7fb01005c0 2024-03-28T00:54:38.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.894+0000 7f7fb5d32640 1 --2- 172.21.15.169:0/2661804980 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7f9c0a2800 0x7f7f9c0a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0198ea0 tx=0x7f7fa000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:38.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:38.898+0000 7f7fa77fe640 1 -- 172.21.15.169:0/2661804980 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7fa804b020 con 0x7f7fb01005c0 2024-03-28T00:54:39.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:38 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/705466430' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-28T00:54:39.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:38 smithi169 ceph-mon[32274]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T00:54:39.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:38 smithi169 ceph-mon[32274]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T00:54:39.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:38 smithi169 ceph-mon[32274]: pgmap v290: 129 pgs: 18 creating+peering, 97 active+clean, 14 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2024-03-28T00:54:39.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:38 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2999372768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:39.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:38 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/744349472' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:39.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:38 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2423484652' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:39.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:38 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/705466430' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-28T00:54:39.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:38 smithi174 ceph-mon[36060]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T00:54:39.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:38 smithi174 ceph-mon[36060]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T00:54:39.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:38 smithi174 ceph-mon[36060]: pgmap v290: 129 pgs: 18 creating+peering, 97 active+clean, 14 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2024-03-28T00:54:39.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:38 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2999372768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:39.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:38 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/744349472' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:39.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:38 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2423484652' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:39.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.111+0000 7f7fb8797640 1 -- 172.21.15.169:0/2661804980 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fb01a0130 con 0x7f7fb01005c0 2024-03-28T00:54:39.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.112+0000 7f7fa77fe640 1 -- 172.21.15.169:0/2661804980 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11348 (secure 0 0 0) 0x7f7fa80bd9e0 con 0x7f7fb01005c0 2024-03-28T00:54:39.115 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:39.115 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24452,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:54:37.160291+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24479,"qdb_cluster":[24479]},"id":1}]} 2024-03-28T00:54:39.115 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 17 2024-03-28T00:54:39.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.114+0000 7f7fb8797640 1 -- 172.21.15.169:0/2661804980 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7f9c0a2800 msgr2=0x7f7f9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:39.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.114+0000 7f7fb8797640 1 --2- 172.21.15.169:0/2661804980 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7f9c0a2800 0x7f7f9c0a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0198ea0 tx=0x7f7fa000a380 comp rx=0 tx=0).stop 2024-03-28T00:54:39.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.114+0000 7f7fb8797640 1 -- 172.21.15.169:0/2661804980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7fb01005c0 msgr2=0x7f7fb0197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:39.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.114+0000 7f7fb8797640 1 --2- 172.21.15.169:0/2661804980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7fb01005c0 0x7f7fb0197990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8037040 tx=0x7f7fa8032ea0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.114+0000 7f7fb8797640 1 -- 172.21.15.169:0/2661804980 shutdown_connections 2024-03-28T00:54:39.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.114+0000 7f7fb8797640 1 --2- 172.21.15.169:0/2661804980 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7f9c0a2800 0x7f7f9c0a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.114+0000 7f7fb8797640 1 --2- 172.21.15.169:0/2661804980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fb0102170 0x7f7fb019ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.114+0000 7f7fb8797640 1 --2- 172.21.15.169:0/2661804980 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7fb01017b0 0x7f7fb0197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.114+0000 7f7fb8797640 1 --2- 172.21.15.169:0/2661804980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7fb01005c0 0x7f7fb0197990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.114+0000 7f7fb8797640 1 -- 172.21.15.169:0/2661804980 >> 172.21.15.169:0/2661804980 conn(0x7f7fb00fbb30 msgr2=0x7f7fb0068800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:39.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.114+0000 7f7fb8797640 1 -- 172.21.15.169:0/2661804980 shutdown_connections 2024-03-28T00:54:39.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.114+0000 7f7fb8797640 1 -- 172.21.15.169:0/2661804980 wait complete. 2024-03-28T00:54:39.128 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T00:54:39.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:38 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/705466430' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-28T00:54:39.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:38 smithi177 ceph-mon[36206]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T00:54:39.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:38 smithi177 ceph-mon[36206]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T00:54:39.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:38 smithi177 ceph-mon[36206]: pgmap v290: 129 pgs: 18 creating+peering, 97 active+clean, 14 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2024-03-28T00:54:39.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:38 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2999372768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:39.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:38 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/744349472' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:39.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:38 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2423484652' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:39.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.227+0000 7f5e2b3f3640 1 -- 172.21.15.169:0/1635272395 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e241073c0 msgr2=0x7f5e24109810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:39.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.227+0000 7f5e2b3f3640 1 --2- 172.21.15.169:0/1635272395 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e241073c0 0x7f5e24109810 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c009ca0 tx=0x7f5e1c02c0f0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.227+0000 7f5e2b3f3640 1 -- 172.21.15.169:0/1635272395 shutdown_connections 2024-03-28T00:54:39.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.227+0000 7f5e2b3f3640 1 --2- 172.21.15.169:0/1635272395 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e241073c0 0x7f5e24109810 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.227+0000 7f5e2b3f3640 1 --2- 172.21.15.169:0/1635272395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e24104a90 0x7f5e24106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.227+0000 7f5e2b3f3640 1 --2- 172.21.15.169:0/1635272395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e24102160 0x7f5e24104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.227+0000 7f5e2b3f3640 1 -- 172.21.15.169:0/1635272395 >> 172.21.15.169:0/1635272395 conn(0x7f5e240fbb50 msgr2=0x7f5e240fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:39.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.227+0000 7f5e2b3f3640 1 -- 172.21.15.169:0/1635272395 shutdown_connections 2024-03-28T00:54:39.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.227+0000 7f5e2b3f3640 1 -- 172.21.15.169:0/1635272395 wait complete. 2024-03-28T00:54:39.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.228+0000 7f5e2b3f3640 1 Processor -- start 2024-03-28T00:54:39.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.228+0000 7f5e2b3f3640 1 -- start start 2024-03-28T00:54:39.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.228+0000 7f5e2b3f3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e24102160 0x7f5e241979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.228+0000 7f5e2b3f3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e24104a90 0x7f5e24197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.228+0000 7f5e2b3f3640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e241073c0 0x7f5e2419ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.228+0000 7f5e2b3f3640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2410d380 con 0x7f5e241073c0 2024-03-28T00:54:39.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.228+0000 7f5e2b3f3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2410d200 con 0x7f5e24104a90 2024-03-28T00:54:39.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.228+0000 7f5e2b3f3640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2410d500 con 0x7f5e24102160 2024-03-28T00:54:39.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.228+0000 7f5e29990640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e241073c0 0x7f5e2419ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.228+0000 7f5e29990640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e241073c0 0x7f5e2419ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48830/0 (socket says 172.21.15.169:48830) 2024-03-28T00:54:39.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.228+0000 7f5e29990640 1 -- 172.21.15.169:0/1312931542 learned_addr learned my addr 172.21.15.169:0/1312931542 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:39.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.228+0000 7f5e2898e640 1 --2- 172.21.15.169:0/1312931542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e24104a90 0x7f5e24197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.228+0000 7f5e2918f640 1 --2- 172.21.15.169:0/1312931542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e24102160 0x7f5e241979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.228+0000 7f5e29990640 1 -- 172.21.15.169:0/1312931542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e24102160 msgr2=0x7f5e241979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:39.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.228+0000 7f5e29990640 1 --2- 172.21.15.169:0/1312931542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e24102160 0x7f5e241979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.228+0000 7f5e29990640 1 -- 172.21.15.169:0/1312931542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e24104a90 msgr2=0x7f5e24197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:39.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.228+0000 7f5e29990640 1 --2- 172.21.15.169:0/1312931542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e24104a90 0x7f5e24197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.228+0000 7f5e29990640 1 -- 172.21.15.169:0/1312931542 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2419f5c0 con 0x7f5e241073c0 2024-03-28T00:54:39.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.229+0000 7f5e29990640 1 --2- 172.21.15.169:0/1312931542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e241073c0 0x7f5e2419ef50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c037040 tx=0x7f5e1c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:39.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.229+0000 7f5e1a7fc640 1 -- 172.21.15.169:0/1312931542 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c041020 con 0x7f5e241073c0 2024-03-28T00:54:39.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.229+0000 7f5e2b3f3640 1 -- 172.21.15.169:0/1312931542 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e2419f850 con 0x7f5e241073c0 2024-03-28T00:54:39.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.229+0000 7f5e1a7fc640 1 -- 172.21.15.169:0/1312931542 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e1c034d90 con 0x7f5e241073c0 2024-03-28T00:54:39.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.229+0000 7f5e1a7fc640 1 -- 172.21.15.169:0/1312931542 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c039040 con 0x7f5e241073c0 2024-03-28T00:54:39.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.229+0000 7f5e2b3f3640 1 -- 172.21.15.169:0/1312931542 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e2419fd40 con 0x7f5e241073c0 2024-03-28T00:54:39.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.230+0000 7f5e2b3f3640 1 -- 172.21.15.169:0/1312931542 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e240fd640 con 0x7f5e241073c0 2024-03-28T00:54:39.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.231+0000 7f5e1a7fc640 1 -- 172.21.15.169:0/1312931542 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5e1c044020 con 0x7f5e241073c0 2024-03-28T00:54:39.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.231+0000 7f5e1a7fc640 1 --2- 172.21.15.169:0/1312931542 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5e100a2800 0x7f5e100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.231+0000 7f5e2918f640 1 --2- 172.21.15.169:0/1312931542 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5e100a2800 0x7f5e100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.231+0000 7f5e1a7fc640 1 -- 172.21.15.169:0/1312931542 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f5e1c0f0fd0 con 0x7f5e241073c0 2024-03-28T00:54:39.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.232+0000 7f5e2918f640 1 --2- 172.21.15.169:0/1312931542 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5e100a2800 0x7f5e100a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5e20009bd0 tx=0x7f5e20005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:39.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.235+0000 7f5e1a7fc640 1 -- 172.21.15.169:0/1312931542 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5e1c04b020 con 0x7f5e241073c0 2024-03-28T00:54:39.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.412+0000 7f5e2b3f3640 1 -- 172.21.15.169:0/1312931542 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e240622a0 con 0x7f5e241073c0 2024-03-28T00:54:39.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.414+0000 7f5e1a7fc640 1 -- 172.21.15.169:0/1312931542 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24186 (secure 0 0 0) 0x7f5e1c0bde30 con 0x7f5e241073c0 2024-03-28T00:54:39.417 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:39.418 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":90,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:54:38.804421+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:54:39.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.416+0000 7f5e2b3f3640 1 -- 172.21.15.169:0/1312931542 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5e100a2800 msgr2=0x7f5e100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:39.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.416+0000 7f5e2b3f3640 1 --2- 172.21.15.169:0/1312931542 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5e100a2800 0x7f5e100a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5e20009bd0 tx=0x7f5e20005f90 comp rx=0 tx=0).stop 2024-03-28T00:54:39.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.416+0000 7f5e2b3f3640 1 -- 172.21.15.169:0/1312931542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e241073c0 msgr2=0x7f5e2419ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:39.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.416+0000 7f5e2b3f3640 1 --2- 172.21.15.169:0/1312931542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e241073c0 0x7f5e2419ef50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c037040 tx=0x7f5e1c032f20 comp rx=0 tx=0).stop 2024-03-28T00:54:39.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.416+0000 7f5e2b3f3640 1 -- 172.21.15.169:0/1312931542 shutdown_connections 2024-03-28T00:54:39.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.416+0000 7f5e2b3f3640 1 --2- 172.21.15.169:0/1312931542 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5e100a2800 0x7f5e100a4cc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.416+0000 7f5e2b3f3640 1 --2- 172.21.15.169:0/1312931542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e241073c0 0x7f5e2419ef50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.416+0000 7f5e2b3f3640 1 --2- 172.21.15.169:0/1312931542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e24104a90 0x7f5e24197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.416+0000 7f5e2b3f3640 1 --2- 172.21.15.169:0/1312931542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e24102160 0x7f5e241979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.416+0000 7f5e2b3f3640 1 -- 172.21.15.169:0/1312931542 >> 172.21.15.169:0/1312931542 conn(0x7f5e240fbb50 msgr2=0x7f5e24102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:39.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.416+0000 7f5e2b3f3640 1 -- 172.21.15.169:0/1312931542 shutdown_connections 2024-03-28T00:54:39.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.416+0000 7f5e2b3f3640 1 -- 172.21.15.169:0/1312931542 wait complete. 2024-03-28T00:54:39.431 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T00:54:32.613335+0000', 'modified': '2024-03-28T00:54:37.160291+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24479}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24479': {'gid': 24479, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.174:6839/1152414321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6838', 'nonce': 1152414321}, {'type': 'v1', 'addr': '172.21.15.174:6839', 'nonce': 1152414321}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24479, 'qdb_cluster': [24479]} 2024-03-28T00:54:39.431 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T00:54:39.431 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T00:54:39.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.533+0000 7fa277c7e640 1 -- 172.21.15.169:0/4033853995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2701005b0 msgr2=0x7fa2701009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:39.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.533+0000 7fa277c7e640 1 --2- 172.21.15.169:0/4033853995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2701005b0 0x7fa2701009d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa26c009900 tx=0x7fa26c02bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:39.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.533+0000 7fa277c7e640 1 -- 172.21.15.169:0/4033853995 shutdown_connections 2024-03-28T00:54:39.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.533+0000 7fa277c7e640 1 --2- 172.21.15.169:0/4033853995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa270102160 0x7fa2701089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.533+0000 7fa277c7e640 1 --2- 172.21.15.169:0/4033853995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2701017a0 0x7fa270101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.533+0000 7fa277c7e640 1 --2- 172.21.15.169:0/4033853995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2701005b0 0x7fa2701009d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.533+0000 7fa277c7e640 1 -- 172.21.15.169:0/4033853995 >> 172.21.15.169:0/4033853995 conn(0x7fa2700fbb40 msgr2=0x7fa2700fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:39.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.533+0000 7fa277c7e640 1 -- 172.21.15.169:0/4033853995 shutdown_connections 2024-03-28T00:54:39.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.533+0000 7fa277c7e640 1 -- 172.21.15.169:0/4033853995 wait complete. 2024-03-28T00:54:39.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.534+0000 7fa277c7e640 1 Processor -- start 2024-03-28T00:54:39.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.534+0000 7fa277c7e640 1 -- start start 2024-03-28T00:54:39.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.534+0000 7fa277c7e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2701005b0 0x7fa27006a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.534+0000 7fa277c7e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2701017a0 0x7fa27019c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.534+0000 7fa277c7e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa270102160 0x7fa27019ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.534+0000 7fa277c7e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27010c200 con 0x7fa2701005b0 2024-03-28T00:54:39.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.534+0000 7fa277c7e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27010c080 con 0x7fa2701017a0 2024-03-28T00:54:39.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.534+0000 7fa277c7e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27010c380 con 0x7fa270102160 2024-03-28T00:54:39.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.534+0000 7fa275a1a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2701005b0 0x7fa27006a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.534+0000 7fa275a1a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2701005b0 0x7fa27006a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48856/0 (socket says 172.21.15.169:48856) 2024-03-28T00:54:39.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.534+0000 7fa275a1a640 1 -- 172.21.15.169:0/2723091750 learned_addr learned my addr 172.21.15.169:0/2723091750 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:39.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.534+0000 7fa275219640 1 --2- 172.21.15.169:0/2723091750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2701017a0 0x7fa27019c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.534+0000 7fa27621b640 1 --2- 172.21.15.169:0/2723091750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa270102160 0x7fa27019ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.534+0000 7fa275a1a640 1 -- 172.21.15.169:0/2723091750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa270102160 msgr2=0x7fa27019ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:39.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.534+0000 7fa275a1a640 1 --2- 172.21.15.169:0/2723091750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa270102160 0x7fa27019ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.534+0000 7fa275a1a640 1 -- 172.21.15.169:0/2723091750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2701017a0 msgr2=0x7fa27019c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:39.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.534+0000 7fa275a1a640 1 --2- 172.21.15.169:0/2723091750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2701017a0 0x7fa27019c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.534+0000 7fa275a1a640 1 -- 172.21.15.169:0/2723091750 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27019f6a0 con 0x7fa2701005b0 2024-03-28T00:54:39.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.534+0000 7fa275219640 1 --2- 172.21.15.169:0/2723091750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2701017a0 0x7fa27019c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:39.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.534+0000 7fa27621b640 1 --2- 172.21.15.169:0/2723091750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa270102160 0x7fa27019ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:54:39.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.535+0000 7fa275a1a640 1 --2- 172.21.15.169:0/2723091750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2701005b0 0x7fa27006a7e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa26c000c00 tx=0x7fa26c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:39.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.535+0000 7fa266ffd640 1 -- 172.21.15.169:0/2723091750 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c041070 con 0x7fa2701005b0 2024-03-28T00:54:39.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.535+0000 7fa277c7e640 1 -- 172.21.15.169:0/2723091750 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa27019f8d0 con 0x7fa2701005b0 2024-03-28T00:54:39.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.535+0000 7fa266ffd640 1 -- 172.21.15.169:0/2723091750 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa26c0044d0 con 0x7fa2701005b0 2024-03-28T00:54:39.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.535+0000 7fa266ffd640 1 -- 172.21.15.169:0/2723091750 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c034440 con 0x7fa2701005b0 2024-03-28T00:54:39.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.535+0000 7fa277c7e640 1 -- 172.21.15.169:0/2723091750 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa27019fe10 con 0x7fa2701005b0 2024-03-28T00:54:39.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.537+0000 7fa266ffd640 1 -- 172.21.15.169:0/2723091750 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa26c044070 con 0x7fa2701005b0 2024-03-28T00:54:39.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.537+0000 7fa27a863740 1 Processor -- start 2024-03-28T00:54:39.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.537+0000 7fa27a863740 1 -- start start 2024-03-28T00:54:39.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa27a863740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f609740 0x55f33f609b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa27a863740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f60ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa27a863740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f33f60f2a0 0x55f33f611730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa27a863740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33f56ec20 con 0x55f33f60a130 2024-03-28T00:54:39.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa27a863740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33f517a40 con 0x55f33f609740 2024-03-28T00:54:39.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa27a863740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33f56acc0 con 0x55f33f60f2a0 2024-03-28T00:54:39.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa275a1a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f60ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa27621b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f609740 0x55f33f609b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa27621b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f609740 0x55f33f609b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:37736/0 (socket says 172.21.15.169:37736) 2024-03-28T00:54:39.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa275a1a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f60ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48868/0 (socket says 172.21.15.169:48868) 2024-03-28T00:54:39.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa27621b640 1 -- 172.21.15.169:0/568494546 learned_addr learned my addr 172.21.15.169:0/568494546 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:39.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa275219640 1 --2- 172.21.15.169:0/568494546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f33f60f2a0 0x55f33f611730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa266ffd640 1 --2- 172.21.15.169:0/2723091750 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa25c0a2800 0x7fa25c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa27621b640 1 -- 172.21.15.169:0/568494546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f33f60f2a0 msgr2=0x55f33f611730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:54:39.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa27621b640 1 --2- 172.21.15.169:0/568494546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f33f60f2a0 0x55f33f611730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa27621b640 1 -- 172.21.15.169:0/568494546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 msgr2=0x55f33f60ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:54:39.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa27621b640 1 --2- 172.21.15.169:0/568494546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f60ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa27621b640 1 -- 172.21.15.169:0/568494546 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f33f539f90 con 0x55f33f609740 2024-03-28T00:54:39.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa275a1a640 1 --2- 172.21.15.169:0/568494546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f60ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:39.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa266ffd640 1 -- 172.21.15.169:0/2723091750 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa26c0f0840 con 0x7fa2701005b0 2024-03-28T00:54:39.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa27621b640 1 --2- 172.21.15.169:0/2723091750 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa25c0a2800 0x7fa25c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.538+0000 7fa27621b640 1 --2- 172.21.15.169:0/568494546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f609740 0x55f33f609b60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa26800e450 tx=0x7fa26800e910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff5604f3d36a2294 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:39.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.539+0000 7fa25bfff640 1 -- 172.21.15.169:0/568494546 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268012e60 con 0x55f33f609740 2024-03-28T00:54:39.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.539+0000 7fa25bfff640 1 -- 172.21.15.169:0/568494546 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa26801ce30 con 0x55f33f609740 2024-03-28T00:54:39.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.539+0000 7fa27621b640 1 --2- 172.21.15.169:0/2723091750 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa25c0a2800 0x7fa25c0a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa26801f000 tx=0x7fa268020000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:39.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.539+0000 7fa25bfff640 1 -- 172.21.15.169:0/568494546 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268024620 con 0x55f33f609740 2024-03-28T00:54:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.539+0000 7fa27a863740 1 -- 172.21.15.169:0/568494546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f609740 msgr2=0x55f33f609b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.539+0000 7fa27a863740 1 --2- 172.21.15.169:0/568494546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f609740 0x55f33f609b60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa26800e450 tx=0x7fa26800e910 comp rx=0 tx=0).stop 2024-03-28T00:54:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.539+0000 7fa27a863740 1 -- 172.21.15.169:0/568494546 shutdown_connections 2024-03-28T00:54:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.539+0000 7fa27a863740 1 --2- 172.21.15.169:0/568494546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f33f60f2a0 0x55f33f611730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.539+0000 7fa27a863740 1 --2- 172.21.15.169:0/568494546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f60ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.539+0000 7fa27a863740 1 --2- 172.21.15.169:0/568494546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f609740 0x55f33f609b60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.539+0000 7fa27a863740 1 -- 172.21.15.169:0/568494546 >> 172.21.15.169:0/568494546 conn(0x55f33f5d1a60 msgr2=0x55f33f6050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.539+0000 7fa27a863740 1 -- 172.21.15.169:0/568494546 shutdown_connections 2024-03-28T00:54:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.539+0000 7fa27a863740 1 -- 172.21.15.169:0/568494546 wait complete. 2024-03-28T00:54:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.540+0000 7fa27a863740 1 -- start start 2024-03-28T00:54:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.540+0000 7fa27a863740 1 Processor -- start 2024-03-28T00:54:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.540+0000 7fa27a863740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f609740 0x55f33f613c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.540+0000 7fa27a863740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f614190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.540+0000 7fa27a863740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f33f60f2a0 0x55f33f61b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.540+0000 7fa27a863740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33f612250 con 0x55f33f60a130 2024-03-28T00:54:39.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.540+0000 7fa27a863740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33f52ae90 con 0x55f33f609740 2024-03-28T00:54:39.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.540+0000 7fa27a863740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33f6123d0 con 0x55f33f60f2a0 2024-03-28T00:54:39.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.540+0000 7fa27a863740 10 client.0 _ll_register_callbacks cb 0x55f33f282ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:54:39.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.540+0000 7fa275219640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f614190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.540+0000 7fa275a1a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f609740 0x55f33f613c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.540+0000 7fa275219640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f614190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48870/0 (socket says 172.21.15.169:48870) 2024-03-28T00:54:39.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.540+0000 7fa275219640 1 -- 172.21.15.169:0/198576034 learned_addr learned my addr 172.21.15.169:0/198576034 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:39.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.540+0000 7fa27621b640 1 --2- 172.21.15.169:0/198576034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f33f60f2a0 0x55f33f61b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa275a1a640 1 -- 172.21.15.169:0/198576034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f33f60f2a0 msgr2=0x55f33f61b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:54:39.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa275a1a640 1 --2- 172.21.15.169:0/198576034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f33f60f2a0 0x55f33f61b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa275a1a640 1 -- 172.21.15.169:0/198576034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 msgr2=0x55f33f614190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:54:39.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa275a1a640 1 --2- 172.21.15.169:0/198576034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f614190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa275a1a640 1 -- 172.21.15.169:0/198576034 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f33f5379d0 con 0x55f33f609740 2024-03-28T00:54:39.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa275219640 1 --2- 172.21.15.169:0/198576034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f614190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:39.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa275a1a640 1 --2- 172.21.15.169:0/198576034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f609740 0x55f33f613c50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0ee150 tx=0x7fa26c02ce50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c651f64744ce5ec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:39.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa2597fa640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:54:39.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa2597fa640 1 -- 172.21.15.169:0/198576034 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c04b070 con 0x55f33f609740 2024-03-28T00:54:39.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa27a863740 1 -- 172.21.15.169:0/198576034 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa26801ce30 con 0x55f33f609740 2024-03-28T00:54:39.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa2597fa640 1 -- 172.21.15.169:0/198576034 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa26c0c52b0 con 0x55f33f609740 2024-03-28T00:54:39.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa2597fa640 1 -- 172.21.15.169:0/198576034 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa26c0bcae0 con 0x55f33f609740 2024-03-28T00:54:39.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa27a863740 10 client.24578 fetch_fsmap learned FSMap version 17 2024-03-28T00:54:39.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa27a863740 1 -- 172.21.15.169:0/198576034 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x55f33f2832d0 con 0x55f33f609740 2024-03-28T00:54:39.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa2597fa640 1 -- 172.21.15.169:0/198576034 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa26801ce30 con 0x55f33f609740 2024-03-28T00:54:39.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa2597fa640 1 -- 172.21.15.169:0/198576034 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 17) v1 ==== 7922+0+0 (secure 0 0 0) 0x7fa26c0bc3f0 con 0x55f33f609740 2024-03-28T00:54:39.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa27a863740 10 client.24578 fetch_fsmap finished waiting for FSMap version 17 2024-03-28T00:54:39.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa27a863740 10 client.24578 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T00:54:39.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa27a863740 20 client.24578 populate_metadata read hostname 'smithi169' 2024-03-28T00:54:39.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa27a863740 1 --2- 172.21.15.169:0/198576034 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f33f620390 0x55f33f61f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa27a863740 4 client.24578 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:54:39.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.541+0000 7fa27a863740 1 -- 172.21.15.169:0/198576034 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f33f57c160 con 0x55f33f620390 2024-03-28T00:54:39.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.542+0000 7fa275219640 1 --2- 172.21.15.169:0/198576034 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f33f620390 0x55f33f61f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.542+0000 7fa2597fa640 10 client.24578 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:54:39.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.542+0000 7fa275219640 1 --2- 172.21.15.169:0/198576034 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f33f620390 0x55f33f61f330 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6ff961aafb5044d server_cookie=1ec80be8c3ec0d94 in_seq=0 out_seq=0 2024-03-28T00:54:39.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.542+0000 7fa2597fa640 1 -- 172.21.15.169:0/198576034 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f33f57c160 con 0x55f33f620390 2024-03-28T00:54:39.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.542+0000 7fa2597fa640 10 client.24578 handle_command_reply: tid=0 2024-03-28T00:54:39.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.543+0000 7fa27a863740 1 client.24578 shutdown 2024-03-28T00:54:39.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.543+0000 7fa27a863740 1 -- 172.21.15.169:0/198576034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f609740 msgr2=0x55f33f613c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:39.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.543+0000 7fa27a863740 1 --2- 172.21.15.169:0/198576034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f609740 0x55f33f613c50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0ee150 tx=0x7fa26c02ce50 comp rx=0 tx=0).stop 2024-03-28T00:54:39.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.543+0000 7fa27a863740 1 -- 172.21.15.169:0/198576034 shutdown_connections 2024-03-28T00:54:39.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.543+0000 7fa27a863740 1 --2- 172.21.15.169:0/198576034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f33f60f2a0 0x55f33f61b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.543+0000 7fa27a863740 1 --2- 172.21.15.169:0/198576034 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f33f620390 0x55f33f61f330 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.543+0000 7fa27a863740 1 --2- 172.21.15.169:0/198576034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f614190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.543+0000 7fa2597fa640 0 client.24578 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:54:39.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.543+0000 7fa27a863740 1 --2- 172.21.15.169:0/198576034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f609740 0x55f33f613c50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.543+0000 7fa27a863740 1 -- 172.21.15.169:0/198576034 >> 172.21.15.169:0/198576034 conn(0x55f33f5d1a60 msgr2=0x55f33f6042f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:39.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.543+0000 7fa27a863740 1 -- 172.21.15.169:0/198576034 shutdown_connections 2024-03-28T00:54:39.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.543+0000 7fa27a863740 1 -- 172.21.15.169:0/198576034 wait complete. 2024-03-28T00:54:39.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.543+0000 7fa27a863740 20 client.24578 trim_cache size 0 max 16384 2024-03-28T00:54:39.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.559+0000 7fa27a863740 1 Processor -- start 2024-03-28T00:54:39.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.559+0000 7fa27a863740 1 -- start start 2024-03-28T00:54:39.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.559+0000 7fa27a863740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f33f609740 0x55f33f604bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.559+0000 7fa27a863740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f61b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.559+0000 7fa27a863740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f60f2a0 0x55f33f605130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.559+0000 7fa27a863740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33f61fbb0 con 0x55f33f60a130 2024-03-28T00:54:39.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.559+0000 7fa27a863740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33f620210 con 0x55f33f60f2a0 2024-03-28T00:54:39.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.559+0000 7fa27a863740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33f620870 con 0x55f33f609740 2024-03-28T00:54:39.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.560+0000 7fa275219640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f61b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.560+0000 7fa275219640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f61b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48878/0 (socket says 172.21.15.169:48878) 2024-03-28T00:54:39.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.560+0000 7fa275219640 1 -- 172.21.15.169:0/3472803056 learned_addr learned my addr 172.21.15.169:0/3472803056 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:39.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.560+0000 7fa27621b640 1 --2- 172.21.15.169:0/3472803056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f60f2a0 0x55f33f605130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.560+0000 7fa275a1a640 1 --2- 172.21.15.169:0/3472803056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f33f609740 0x55f33f604bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.560+0000 7fa275219640 1 -- 172.21.15.169:0/3472803056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f33f609740 msgr2=0x55f33f604bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:54:39.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.560+0000 7fa275219640 1 --2- 172.21.15.169:0/3472803056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f33f609740 0x55f33f604bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.560+0000 7fa275219640 1 -- 172.21.15.169:0/3472803056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f60f2a0 msgr2=0x55f33f605130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:54:39.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.560+0000 7fa275219640 1 --2- 172.21.15.169:0/3472803056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f60f2a0 0x55f33f605130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.560+0000 7fa275219640 1 -- 172.21.15.169:0/3472803056 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f33f34d6b0 con 0x55f33f60a130 2024-03-28T00:54:39.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.560+0000 7fa275219640 1 --2- 172.21.15.169:0/3472803056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f61b190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa260000c00 tx=0x7fa260005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265d1962bae7689f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:39.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.560+0000 7fa25b7fe640 1 -- 172.21.15.169:0/3472803056 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2600120d0 con 0x55f33f60a130 2024-03-28T00:54:39.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.560+0000 7fa25b7fe640 1 -- 172.21.15.169:0/3472803056 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa260012270 con 0x55f33f60a130 2024-03-28T00:54:39.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.560+0000 7fa25b7fe640 1 -- 172.21.15.169:0/3472803056 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa260010690 con 0x55f33f60a130 2024-03-28T00:54:39.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.561+0000 7fa27a863740 1 -- 172.21.15.169:0/3472803056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 msgr2=0x55f33f61b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:39.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.561+0000 7fa27a863740 1 --2- 172.21.15.169:0/3472803056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f61b190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa260000c00 tx=0x7fa260005f90 comp rx=0 tx=0).stop 2024-03-28T00:54:39.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.561+0000 7fa27a863740 1 -- 172.21.15.169:0/3472803056 shutdown_connections 2024-03-28T00:54:39.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.561+0000 7fa27a863740 1 --2- 172.21.15.169:0/3472803056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f60f2a0 0x55f33f605130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.561+0000 7fa27a863740 1 --2- 172.21.15.169:0/3472803056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f61b190 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.561+0000 7fa27a863740 1 --2- 172.21.15.169:0/3472803056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f33f609740 0x55f33f604bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.561+0000 7fa27a863740 1 -- 172.21.15.169:0/3472803056 >> 172.21.15.169:0/3472803056 conn(0x55f33f5d1a60 msgr2=0x55f33f613c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:39.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.561+0000 7fa27a863740 1 -- 172.21.15.169:0/3472803056 shutdown_connections 2024-03-28T00:54:39.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.561+0000 7fa27a863740 1 -- 172.21.15.169:0/3472803056 wait complete. 2024-03-28T00:54:39.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.561+0000 7fa27a863740 1 -- start start 2024-03-28T00:54:39.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.561+0000 7fa27a863740 1 Processor -- start 2024-03-28T00:54:39.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.561+0000 7fa27a863740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f609740 0x55f33f61afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.561+0000 7fa27a863740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f617ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.561+0000 7fa27a863740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f33f60f2a0 0x55f33f618010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.561+0000 7fa27a863740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33f621530 con 0x55f33f60a130 2024-03-28T00:54:39.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.561+0000 7fa27a863740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33f620ed0 con 0x55f33f609740 2024-03-28T00:54:39.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.561+0000 7fa27a863740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33f621b90 con 0x55f33f60f2a0 2024-03-28T00:54:39.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.561+0000 7fa27a863740 10 client.0 _ll_register_callbacks cb 0x7fa26c0f6240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:54:39.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa275a1a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f609740 0x55f33f61afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa275219640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f617ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa275219640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f617ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48894/0 (socket says 172.21.15.169:48894) 2024-03-28T00:54:39.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa275a1a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f609740 0x55f33f61afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:37770/0 (socket says 172.21.15.169:37770) 2024-03-28T00:54:39.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa275219640 1 -- 172.21.15.169:0/3253747163 learned_addr learned my addr 172.21.15.169:0/3253747163 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:39.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa27621b640 1 --2- 172.21.15.169:0/3253747163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f33f60f2a0 0x55f33f618010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa275a1a640 1 -- 172.21.15.169:0/3253747163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f33f60f2a0 msgr2=0x55f33f618010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:54:39.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa275a1a640 1 --2- 172.21.15.169:0/3253747163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f33f60f2a0 0x55f33f618010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa275a1a640 1 -- 172.21.15.169:0/3253747163 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 msgr2=0x55f33f617ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:54:39.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa275a1a640 1 --2- 172.21.15.169:0/3253747163 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f617ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa275a1a640 1 -- 172.21.15.169:0/3253747163 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f33f67b3f0 con 0x55f33f609740 2024-03-28T00:54:39.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa27621b640 1 --2- 172.21.15.169:0/3253747163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f33f60f2a0 0x55f33f618010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:54:39.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa275219640 1 --2- 172.21.15.169:0/3253747163 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f617ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:39.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa275a1a640 1 --2- 172.21.15.169:0/3253747163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f609740 0x55f33f61afc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0b9970 tx=0x7fa26c0bc620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=925fe297e0ada9a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:39.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa2597fa640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:54:39.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa2597fa640 1 -- 172.21.15.169:0/3253747163 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c04b070 con 0x55f33f609740 2024-03-28T00:54:39.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa2597fa640 1 -- 172.21.15.169:0/3253747163 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa26c0b9010 con 0x55f33f609740 2024-03-28T00:54:39.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa27a863740 1 -- 172.21.15.169:0/3253747163 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa260012270 con 0x55f33f609740 2024-03-28T00:54:39.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa2597fa640 1 -- 172.21.15.169:0/3253747163 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa26c0bc900 con 0x55f33f609740 2024-03-28T00:54:39.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa27a863740 10 client.24584 fetch_fsmap learned FSMap version 17 2024-03-28T00:54:39.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa27a863740 1 -- 172.21.15.169:0/3253747163 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x55f33f67b3f0 con 0x55f33f609740 2024-03-28T00:54:39.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.562+0000 7fa2597fa640 1 -- 172.21.15.169:0/3253747163 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa260012270 con 0x55f33f609740 2024-03-28T00:54:39.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa2597fa640 1 -- 172.21.15.169:0/3253747163 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 17) v1 ==== 7922+0+0 (secure 0 0 0) 0x7fa26c0c52b0 con 0x55f33f609740 2024-03-28T00:54:39.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa27a863740 10 client.24584 fetch_fsmap finished waiting for FSMap version 17 2024-03-28T00:54:39.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa27a863740 10 client.24584 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T00:54:39.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa27a863740 20 client.24584 populate_metadata read hostname 'smithi169' 2024-03-28T00:54:39.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa27a863740 1 --2- 172.21.15.169:0/3253747163 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f33f680940 0x55f33f67c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:39.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa27a863740 4 client.24584 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T00:54:39.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa27a863740 1 -- 172.21.15.169:0/3253747163 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f33f5764b0 con 0x55f33f680940 2024-03-28T00:54:39.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa275219640 1 --2- 172.21.15.169:0/3253747163 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f33f680940 0x55f33f67c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:39.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa2597fa640 10 client.24584 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:54:39.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa275219640 1 --2- 172.21.15.169:0/3253747163 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f33f680940 0x55f33f67c6f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c315fc33246c5857 server_cookie=aabfaf69722e6d55 in_seq=0 out_seq=0 2024-03-28T00:54:39.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa2597fa640 1 -- 172.21.15.169:0/3253747163 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55f33f5764b0 con 0x55f33f680940 2024-03-28T00:54:39.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa2597fa640 10 client.24584 handle_command_reply: tid=0 2024-03-28T00:54:39.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa27a863740 1 client.24584 shutdown 2024-03-28T00:54:39.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa27a863740 1 -- 172.21.15.169:0/3253747163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f609740 msgr2=0x55f33f61afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:39.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa27a863740 1 --2- 172.21.15.169:0/3253747163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f609740 0x55f33f61afc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0b9970 tx=0x7fa26c0bc620 comp rx=0 tx=0).stop 2024-03-28T00:54:39.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa27a863740 1 -- 172.21.15.169:0/3253747163 shutdown_connections 2024-03-28T00:54:39.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa27a863740 1 --2- 172.21.15.169:0/3253747163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f33f60f2a0 0x55f33f618010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa27a863740 1 --2- 172.21.15.169:0/3253747163 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f33f680940 0x55f33f67c6f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa27a863740 1 --2- 172.21.15.169:0/3253747163 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f33f60a130 0x55f33f617ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa27a863740 1 --2- 172.21.15.169:0/3253747163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f33f609740 0x55f33f61afc0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa2597fa640 0 client.24584 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:54:39.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.563+0000 7fa27a863740 1 -- 172.21.15.169:0/3253747163 >> 172.21.15.169:0/3253747163 conn(0x55f33f5d1a60 msgr2=0x55f33f6082f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:39.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.564+0000 7fa27a863740 1 -- 172.21.15.169:0/3253747163 shutdown_connections 2024-03-28T00:54:39.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.564+0000 7fa27a863740 1 -- 172.21.15.169:0/3253747163 wait complete. 2024-03-28T00:54:39.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.564+0000 7fa27a863740 20 client.24584 trim_cache size 0 max 16384 2024-03-28T00:54:39.575 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:54:39.575 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_fsid": "833368da-ec9c-11ee-95d0-87774f69a715", 2024-03-28T00:54:39.575 INFO:teuthology.orchestra.run.smithi169.stdout: "whoami": 0, 2024-03-28T00:54:39.575 INFO:teuthology.orchestra.run.smithi169.stdout: "id": 24479, 2024-03-28T00:54:39.575 INFO:teuthology.orchestra.run.smithi169.stdout: "want_state": "up:active", 2024-03-28T00:54:39.575 INFO:teuthology.orchestra.run.smithi169.stdout: "state": "up:active", 2024-03-28T00:54:39.575 INFO:teuthology.orchestra.run.smithi169.stdout: "fs_name": "cephfs", 2024-03-28T00:54:39.575 INFO:teuthology.orchestra.run.smithi169.stdout: "rank_uptime": 6.3259668380000003, 2024-03-28T00:54:39.575 INFO:teuthology.orchestra.run.smithi169.stdout: "mdsmap_epoch": 17, 2024-03-28T00:54:39.575 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_epoch": 90, 2024-03-28T00:54:39.575 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_epoch_barrier": 88, 2024-03-28T00:54:39.576 INFO:teuthology.orchestra.run.smithi169.stdout: "uptime": 10.785887194000001 2024-03-28T00:54:39.576 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:54:39.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.564+0000 7fa277c7e640 1 -- 172.21.15.169:0/2723091750 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa25c0a2800 msgr2=0x7fa25c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:39.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.564+0000 7fa277c7e640 1 --2- 172.21.15.169:0/2723091750 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa25c0a2800 0x7fa25c0a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa26801f000 tx=0x7fa268020000 comp rx=0 tx=0).stop 2024-03-28T00:54:39.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.564+0000 7fa277c7e640 1 -- 172.21.15.169:0/2723091750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2701005b0 msgr2=0x7fa27006a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:39.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.564+0000 7fa277c7e640 1 --2- 172.21.15.169:0/2723091750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2701005b0 0x7fa27006a7e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa26c000c00 tx=0x7fa26c005d50 comp rx=0 tx=0).stop 2024-03-28T00:54:39.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.564+0000 7fa277c7e640 1 -- 172.21.15.169:0/2723091750 shutdown_connections 2024-03-28T00:54:39.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.564+0000 7fa277c7e640 1 --2- 172.21.15.169:0/2723091750 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa25c0a2800 0x7fa25c0a4cc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.564+0000 7fa277c7e640 1 --2- 172.21.15.169:0/2723091750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa270102160 0x7fa27019ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.564+0000 7fa277c7e640 1 --2- 172.21.15.169:0/2723091750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2701017a0 0x7fa27019c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.564+0000 7fa277c7e640 1 --2- 172.21.15.169:0/2723091750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2701005b0 0x7fa27006a7e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:39.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.564+0000 7fa277c7e640 1 -- 172.21.15.169:0/2723091750 >> 172.21.15.169:0/2723091750 conn(0x7fa2700fbb40 msgr2=0x7fa270104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:39.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.564+0000 7fa277c7e640 1 -- 172.21.15.169:0/2723091750 shutdown_connections 2024-03-28T00:54:39.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:39.564+0000 7fa277c7e640 1 -- 172.21.15.169:0/2723091750 wait complete. 2024-03-28T00:54:39.578 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T00:54:39.578 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-28T00:54:39.579 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph fs ls 2024-03-28T00:54:39.608 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 00:54:39 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-b[66103]: 2024-03-28T00:54:39.138+0000 7f2508032700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T00:54:39.816 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:54:40.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:39 smithi174 ceph-mon[36060]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T00:54:40.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:39 smithi174 ceph-mon[36060]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T00:54:40.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:39 smithi174 ceph-mon[36060]: Cluster is now healthy 2024-03-28T00:54:40.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:39 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2661804980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:40.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:39 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1312931542' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:40.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:39 smithi177 ceph-mon[36206]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T00:54:40.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:39 smithi177 ceph-mon[36206]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T00:54:40.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:39 smithi177 ceph-mon[36206]: Cluster is now healthy 2024-03-28T00:54:40.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:39 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2661804980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:40.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:39 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1312931542' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:40.195 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:39 smithi169 ceph-mon[32274]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T00:54:40.195 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:39 smithi169 ceph-mon[32274]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T00:54:40.195 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:39 smithi169 ceph-mon[32274]: Cluster is now healthy 2024-03-28T00:54:40.195 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:39 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2661804980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:40.195 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:39 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1312931542' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:41.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:40 smithi174 ceph-mon[36060]: pgmap v292: 129 pgs: 18 creating+peering, 102 active+clean, 9 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s wr, 8 op/s 2024-03-28T00:54:41.165 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:40 smithi169 ceph-mon[32274]: pgmap v292: 129 pgs: 18 creating+peering, 102 active+clean, 9 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s wr, 8 op/s 2024-03-28T00:54:41.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:40 smithi177 ceph-mon[36206]: pgmap v292: 129 pgs: 18 creating+peering, 102 active+clean, 9 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s wr, 8 op/s 2024-03-28T00:54:41.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.535+0000 7f52b8fbf700 1 -- 172.21.15.169:0/2578315404 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52b4100d30 msgr2=0x7f52b4101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:41.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.535+0000 7f52b8fbf700 1 --2- 172.21.15.169:0/2578315404 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52b4100d30 0x7f52b4101110 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f52a40099c0 tx=0x7f52a4009cd0 comp rx=0 tx=0).stop 2024-03-28T00:54:41.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.535+0000 7f52b8fbf700 1 -- 172.21.15.169:0/2578315404 shutdown_connections 2024-03-28T00:54:41.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.535+0000 7f52b8fbf700 1 --2- 172.21.15.169:0/2578315404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52b4107550 0x7f52b410f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:41.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.535+0000 7f52b8fbf700 1 --2- 172.21.15.169:0/2578315404 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52b4100d30 0x7f52b4101110 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:41.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.535+0000 7f52b8fbf700 1 --2- 172.21.15.169:0/2578315404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52b41016e0 0x7f52b4106e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:41.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.535+0000 7f52b8fbf700 1 -- 172.21.15.169:0/2578315404 >> 172.21.15.169:0/2578315404 conn(0x7f52b40763d0 msgr2=0x7f52b40767e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:41.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.535+0000 7f52b8fbf700 1 -- 172.21.15.169:0/2578315404 shutdown_connections 2024-03-28T00:54:41.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.535+0000 7f52b8fbf700 1 -- 172.21.15.169:0/2578315404 wait complete. 2024-03-28T00:54:41.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.536+0000 7f52b8fbf700 1 Processor -- start 2024-03-28T00:54:41.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.536+0000 7f52b8fbf700 1 -- start start 2024-03-28T00:54:41.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.536+0000 7f52b8fbf700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52b4100d30 0x7f52b419e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:41.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.536+0000 7f52b8fbf700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52b41016e0 0x7f52b419eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:41.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.536+0000 7f52b8fbf700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52b4107550 0x7f52b41986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:41.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.536+0000 7f52b8fbf700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b41124e0 con 0x7f52b4107550 2024-03-28T00:54:41.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.536+0000 7f52b8fbf700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b4112360 con 0x7f52b4100d30 2024-03-28T00:54:41.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.536+0000 7f52b8fbf700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b4112660 con 0x7f52b41016e0 2024-03-28T00:54:41.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.537+0000 7f52b2d9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52b4107550 0x7f52b41986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:41.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.537+0000 7f52b259c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52b4100d30 0x7f52b419e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:41.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.537+0000 7f52b2d9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52b4107550 0x7f52b41986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48912/0 (socket says 172.21.15.169:48912) 2024-03-28T00:54:41.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.537+0000 7f52b259c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52b4100d30 0x7f52b419e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:37782/0 (socket says 172.21.15.169:37782) 2024-03-28T00:54:41.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.537+0000 7f52b2d9d700 1 -- 172.21.15.169:0/1154092015 learned_addr learned my addr 172.21.15.169:0/1154092015 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:41.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.537+0000 7f52b1d9b700 1 --2- 172.21.15.169:0/1154092015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52b41016e0 0x7f52b419eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:41.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.537+0000 7f52b259c700 1 -- 172.21.15.169:0/1154092015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52b41016e0 msgr2=0x7f52b419eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:41.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.537+0000 7f52b259c700 1 --2- 172.21.15.169:0/1154092015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52b41016e0 0x7f52b419eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:41.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.537+0000 7f52b259c700 1 -- 172.21.15.169:0/1154092015 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52b4107550 msgr2=0x7f52b41986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:41.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.537+0000 7f52b259c700 1 --2- 172.21.15.169:0/1154092015 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52b4107550 0x7f52b41986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:41.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.537+0000 7f52b259c700 1 -- 172.21.15.169:0/1154092015 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b4198fc0 con 0x7f52b4100d30 2024-03-28T00:54:41.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.537+0000 7f52b259c700 1 --2- 172.21.15.169:0/1154092015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52b4100d30 0x7f52b419e5c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f52a400b580 tx=0x7f52a4003e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:41.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.537+0000 7f52a37fe700 1 -- 172.21.15.169:0/1154092015 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a4009e60 con 0x7f52b4100d30 2024-03-28T00:54:41.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.537+0000 7f52b8fbf700 1 -- 172.21.15.169:0/1154092015 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b4199250 con 0x7f52b4100d30 2024-03-28T00:54:41.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.537+0000 7f52a37fe700 1 -- 172.21.15.169:0/1154092015 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52a4003680 con 0x7f52b4100d30 2024-03-28T00:54:41.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.537+0000 7f52a37fe700 1 -- 172.21.15.169:0/1154092015 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a400fda0 con 0x7f52b4100d30 2024-03-28T00:54:41.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.537+0000 7f52b8fbf700 1 -- 172.21.15.169:0/1154092015 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b41a5370 con 0x7f52b4100d30 2024-03-28T00:54:41.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.538+0000 7f52b8fbf700 1 -- 172.21.15.169:0/1154092015 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b406a3a0 con 0x7f52b4100d30 2024-03-28T00:54:41.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.539+0000 7f52a37fe700 1 -- 172.21.15.169:0/1154092015 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f52a400f460 con 0x7f52b4100d30 2024-03-28T00:54:41.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.540+0000 7f52a37fe700 1 --2- 172.21.15.169:0/1154092015 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f529c0a2890 0x7f529c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:41.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.540+0000 7f52b1d9b700 1 --2- 172.21.15.169:0/1154092015 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f529c0a2890 0x7f529c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:41.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.540+0000 7f52a37fe700 1 -- 172.21.15.169:0/1154092015 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f52a40cf9c0 con 0x7f52b4100d30 2024-03-28T00:54:41.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.540+0000 7f52b1d9b700 1 --2- 172.21.15.169:0/1154092015 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f529c0a2890 0x7f529c0a4d50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f52b4199e50 tx=0x7f52a8005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:41.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.544+0000 7f52a37fe700 1 -- 172.21.15.169:0/1154092015 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f52a40d1e20 con 0x7f52b4100d30 2024-03-28T00:54:41.788 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.785+0000 7f52b8fbf700 1 -- 172.21.15.169:0/1154092015 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f52b410a740 con 0x7f52b4100d30 2024-03-28T00:54:41.788 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.785+0000 7f52a37fe700 1 -- 172.21.15.169:0/1154092015 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7f52a4013070 con 0x7f52b4100d30 2024-03-28T00:54:41.788 INFO:teuthology.orchestra.run.smithi169.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-28T00:54:41.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.787+0000 7f52b8fbf700 1 -- 172.21.15.169:0/1154092015 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f529c0a2890 msgr2=0x7f529c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:41.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.787+0000 7f52b8fbf700 1 --2- 172.21.15.169:0/1154092015 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f529c0a2890 0x7f529c0a4d50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f52b4199e50 tx=0x7f52a8005d50 comp rx=0 tx=0).stop 2024-03-28T00:54:41.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.787+0000 7f52b8fbf700 1 -- 172.21.15.169:0/1154092015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52b4100d30 msgr2=0x7f52b419e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:41.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.787+0000 7f52b8fbf700 1 --2- 172.21.15.169:0/1154092015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52b4100d30 0x7f52b419e5c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f52a400b580 tx=0x7f52a4003e00 comp rx=0 tx=0).stop 2024-03-28T00:54:41.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.787+0000 7f52b8fbf700 1 -- 172.21.15.169:0/1154092015 shutdown_connections 2024-03-28T00:54:41.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.787+0000 7f52b8fbf700 1 --2- 172.21.15.169:0/1154092015 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f52b4107550 0x7f52b41986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:41.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.787+0000 7f52b8fbf700 1 --2- 172.21.15.169:0/1154092015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52b4100d30 0x7f52b419e5c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:41.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.787+0000 7f52b8fbf700 1 --2- 172.21.15.169:0/1154092015 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f529c0a2890 0x7f529c0a4d50 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:41.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.787+0000 7f52b8fbf700 1 --2- 172.21.15.169:0/1154092015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52b41016e0 0x7f52b419eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:41.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.787+0000 7f52b8fbf700 1 -- 172.21.15.169:0/1154092015 >> 172.21.15.169:0/1154092015 conn(0x7f52b40763d0 msgr2=0x7f52b40fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:41.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.787+0000 7f52b8fbf700 1 -- 172.21.15.169:0/1154092015 shutdown_connections 2024-03-28T00:54:41.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:41.787+0000 7f52b8fbf700 1 -- 172.21.15.169:0/1154092015 wait complete. 2024-03-28T00:54:42.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:41 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1154092015' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T00:54:42.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:41 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1154092015' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T00:54:42.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:41 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1154092015' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T00:54:42.488 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-28T00:54:42.488 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T00:54:42.488 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi169.front.sepia.ceph.com 2024-03-28T00:54:42.488 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-28T00:54:42.488 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-28T00:54:42.488 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-28T00:54:42.488 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T00:54:42.488 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T00:54:42.488 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-28T00:54:42.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:54:42.489 DEBUG:teuthology.orchestra.run.smithi169:> ip addr 2024-03-28T00:54:42.506 INFO:teuthology.orchestra.run.smithi169.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T00:54:42.506 INFO:teuthology.orchestra.run.smithi169.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T00:54:42.506 INFO:teuthology.orchestra.run.smithi169.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T00:54:42.506 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-28T00:54:42.506 INFO:teuthology.orchestra.run.smithi169.stdout: inet6 ::1/128 scope host 2024-03-28T00:54:42.506 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-28T00:54:42.506 INFO:teuthology.orchestra.run.smithi169.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T00:54:42.506 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:d9:3b:b4 brd ff:ff:ff:ff:ff:ff 2024-03-28T00:54:42.506 INFO:teuthology.orchestra.run.smithi169.stdout: altname enp7s0f0 2024-03-28T00:54:42.506 INFO:teuthology.orchestra.run.smithi169.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T00:54:42.506 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:d9:3b:b5 brd ff:ff:ff:ff:ff:ff 2024-03-28T00:54:42.507 INFO:teuthology.orchestra.run.smithi169.stdout: altname enp7s0f1 2024-03-28T00:54:42.507 INFO:teuthology.orchestra.run.smithi169.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T00:54:42.507 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:88:7f:4c brd ff:ff:ff:ff:ff:ff 2024-03-28T00:54:42.507 INFO:teuthology.orchestra.run.smithi169.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T00:54:42.507 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:88:7f:4d brd ff:ff:ff:ff:ff:ff 2024-03-28T00:54:42.507 INFO:teuthology.orchestra.run.smithi169.stdout: inet 172.21.15.169/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-28T00:54:42.507 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft 42221sec preferred_lft 42221sec 2024-03-28T00:54:42.507 INFO:teuthology.orchestra.run.smithi169.stdout: inet6 fe80::ec4:7aff:fe88:7f4d/64 scope link 2024-03-28T00:54:42.507 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-28T00:54:42.508 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T00:54:42.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:54:42.508 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2024-03-28T00:54:42.508 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-03-28T00:54:42.508 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link add name ceph-brx type bridge 2024-03-28T00:54:42.508 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip addr flush dev ceph-brx 2024-03-28T00:54:42.508 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link set ceph-brx up 2024-03-28T00:54:42.508 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T00:54:42.508 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-03-28T00:54:42.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:54:42.683 DEBUG:teuthology.orchestra.run.smithi169:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T00:54:42.710 INFO:teuthology.orchestra.run.smithi169.stdout:1 2024-03-28T00:54:42.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:54:42.712 DEBUG:teuthology.orchestra.run.smithi169:> route 2024-03-28T00:54:42.778 INFO:teuthology.orchestra.run.smithi169.stdout:Kernel IP routing table 2024-03-28T00:54:42.778 INFO:teuthology.orchestra.run.smithi169.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T00:54:42.778 INFO:teuthology.orchestra.run.smithi169.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-28T00:54:42.779 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-28T00:54:42.779 INFO:teuthology.orchestra.run.smithi169.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T00:54:42.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:54:42.779 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2024-03-28T00:54:42.779 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-03-28T00:54:42.779 DEBUG:teuthology.orchestra.run.smithi169:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-28T00:54:42.779 DEBUG:teuthology.orchestra.run.smithi169:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-28T00:54:42.779 DEBUG:teuthology.orchestra.run.smithi169:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-28T00:54:42.780 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-03-28T00:54:42.967 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:42 smithi169 ceph-mon[32274]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 1 op/s 2024-03-28T00:54:43.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:54:43.026 DEBUG:teuthology.orchestra.run.smithi169:> ip netns list 2024-03-28T00:54:43.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:54:43.042 DEBUG:teuthology.orchestra.run.smithi169:> ip netns list-id 2024-03-28T00:54:43.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:54:43.097 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2024-03-28T00:54:43.097 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-03-28T00:54:43.097 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T00:54:43.097 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-28T00:54:43.097 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-03-28T00:54:43.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:42 smithi174 ceph-mon[36060]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 1 op/s 2024-03-28T00:54:43.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:42 smithi177 ceph-mon[36206]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 1 op/s 2024-03-28T00:54:43.180 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-28T00:54:43.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:54:43.181 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2024-03-28T00:54:43.181 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-03-28T00:54:43.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-28T00:54:43.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T00:54:43.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-28T00:54:43.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-28T00:54:43.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-28T00:54:43.182 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-03-28T00:54:43.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:54:43.320 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2024-03-28T00:54:43.320 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-03-28T00:54:43.320 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link set brx.0 up 2024-03-28T00:54:43.320 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T00:54:43.320 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-03-28T00:54:43.362 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-28T00:54:43.362 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T00:54:43.362 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-28T00:54:43.418 INFO:teuthology.orchestra.run.smithi169.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-28T00:54:43.418 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T00:54:43.418 DEBUG:teuthology.orchestra.run.smithi169:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-28T00:54:43.472 DEBUG:teuthology.orchestra.run.smithi169:> sudo modprobe fuse 2024-03-28T00:54:43.540 DEBUG:teuthology.orchestra.run.smithi169:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T00:54:43.596 INFO:teuthology.orchestra.run.smithi169.stdout:/proc 2024-03-28T00:54:43.597 INFO:teuthology.orchestra.run.smithi169.stdout:/sys 2024-03-28T00:54:43.597 INFO:teuthology.orchestra.run.smithi169.stdout:/dev 2024-03-28T00:54:43.597 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/security 2024-03-28T00:54:43.597 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/shm 2024-03-28T00:54:43.597 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/pts 2024-03-28T00:54:43.597 INFO:teuthology.orchestra.run.smithi169.stdout:/run 2024-03-28T00:54:43.597 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/cgroup 2024-03-28T00:54:43.597 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/pstore 2024-03-28T00:54:43.597 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/bpf 2024-03-28T00:54:43.597 INFO:teuthology.orchestra.run.smithi169.stdout:/ 2024-03-28T00:54:43.597 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/selinux 2024-03-28T00:54:43.597 INFO:teuthology.orchestra.run.smithi169.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T00:54:43.597 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/hugepages 2024-03-28T00:54:43.598 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/mqueue 2024-03-28T00:54:43.598 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/debug 2024-03-28T00:54:43.598 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/tracing 2024-03-28T00:54:43.598 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/fuse/connections 2024-03-28T00:54:43.598 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/config 2024-03-28T00:54:43.598 INFO:teuthology.orchestra.run.smithi169.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T00:54:43.598 INFO:teuthology.orchestra.run.smithi169.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T00:54:43.598 INFO:teuthology.orchestra.run.smithi169.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T00:54:43.598 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T00:54:43.598 INFO:teuthology.orchestra.run.smithi169.stdout:/run/user/1000 2024-03-28T00:54:43.598 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph 2024-03-28T00:54:43.598 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay 2024-03-28T00:54:43.598 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay/454a34bbe5c281390dba8ef0c2af31da204a3c3f2a829d7ce03aa6bd046e74fa/merged 2024-03-28T00:54:43.599 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay/e606b5d878fcd1ad1fa1f83ba983f3b87e61e9e888ad6fa6223dc95bf845d1c2/merged 2024-03-28T00:54:43.599 INFO:teuthology.orchestra.run.smithi169.stdout:/run/user/0 2024-03-28T00:54:43.599 INFO:teuthology.orchestra.run.smithi169.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T00:54:43.599 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay/b0d116cc70bfe47cf356626f11285000bab25dde4f663eaad8d5cd4d826ddbaa/merged 2024-03-28T00:54:43.599 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay/75bd5bf7e92f9367bb6055deb25a52352826696a0d7ee2afe72d18fdae67ca45/merged 2024-03-28T00:54:43.599 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay/fbb967b832edbd118ef50818fcbe9788df2caaa129091a07093f3ab352facdc4/merged 2024-03-28T00:54:43.599 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay/679e79184319284edc180ed0ca34a1e28b024cfdfc358fbd82c8ee9e0ac5231d/merged 2024-03-28T00:54:43.599 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay/94a6f9c8f51b92d5819ea4f2551be1839c5b7c7b59e6145a8fad0d86ea721a04/merged 2024-03-28T00:54:43.599 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay/0f3bd14a5c2879df9518a6b491501c44a2ce521df8ce4664682bef6a90b90835/merged 2024-03-28T00:54:43.599 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay/d1bf068c4e82148a283b36602e06ba015d129b1186315c8f7231beb8a4703ec4/merged 2024-03-28T00:54:43.599 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay/2f938f19770bc16f74836b1acf5d8c8c1ad7df0cde769aa9b9f34f3ba6b250a5/merged 2024-03-28T00:54:43.599 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns 2024-03-28T00:54:43.599 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T00:54:43.599 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T00:54:43.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:54:43.600 DEBUG:teuthology.orchestra.run.smithi169:> ls /sys/fs/fuse/connections 2024-03-28T00:54:43.651 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-28T00:54:43.651 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-28T00:54:43.694 DEBUG:teuthology.orchestra.run.smithi169:> sudo modprobe fuse 2024-03-28T00:54:43.721 DEBUG:teuthology.orchestra.run.smithi169:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T00:54:43.777 INFO:teuthology.orchestra.run.smithi169.stdout:/proc 2024-03-28T00:54:43.777 INFO:teuthology.orchestra.run.smithi169.stdout:/sys 2024-03-28T00:54:43.777 INFO:teuthology.orchestra.run.smithi169.stdout:/dev 2024-03-28T00:54:43.777 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/security 2024-03-28T00:54:43.777 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/shm 2024-03-28T00:54:43.777 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/pts 2024-03-28T00:54:43.777 INFO:teuthology.orchestra.run.smithi169.stdout:/run 2024-03-28T00:54:43.777 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/cgroup 2024-03-28T00:54:43.777 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/pstore 2024-03-28T00:54:43.778 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/bpf 2024-03-28T00:54:43.778 INFO:teuthology.orchestra.run.smithi169.stdout:/ 2024-03-28T00:54:43.778 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/selinux 2024-03-28T00:54:43.778 INFO:teuthology.orchestra.run.smithi169.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T00:54:43.778 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/hugepages 2024-03-28T00:54:43.778 INFO:teuthology.orchestra.run.smithi169.stdout:/dev/mqueue 2024-03-28T00:54:43.778 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/debug 2024-03-28T00:54:43.778 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/tracing 2024-03-28T00:54:43.778 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/fs/fuse/connections 2024-03-28T00:54:43.779 INFO:teuthology.orchestra.run.smithi169.stdout:/sys/kernel/config 2024-03-28T00:54:43.779 INFO:teuthology.orchestra.run.smithi169.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T00:54:43.779 INFO:teuthology.orchestra.run.smithi169.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T00:54:43.779 INFO:teuthology.orchestra.run.smithi169.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T00:54:43.779 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T00:54:43.779 INFO:teuthology.orchestra.run.smithi169.stdout:/run/user/1000 2024-03-28T00:54:43.779 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/ceph 2024-03-28T00:54:43.779 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay 2024-03-28T00:54:43.779 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay/454a34bbe5c281390dba8ef0c2af31da204a3c3f2a829d7ce03aa6bd046e74fa/merged 2024-03-28T00:54:43.779 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay/e606b5d878fcd1ad1fa1f83ba983f3b87e61e9e888ad6fa6223dc95bf845d1c2/merged 2024-03-28T00:54:43.779 INFO:teuthology.orchestra.run.smithi169.stdout:/run/user/0 2024-03-28T00:54:43.779 INFO:teuthology.orchestra.run.smithi169.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T00:54:43.779 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay/b0d116cc70bfe47cf356626f11285000bab25dde4f663eaad8d5cd4d826ddbaa/merged 2024-03-28T00:54:43.780 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay/75bd5bf7e92f9367bb6055deb25a52352826696a0d7ee2afe72d18fdae67ca45/merged 2024-03-28T00:54:43.780 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay/fbb967b832edbd118ef50818fcbe9788df2caaa129091a07093f3ab352facdc4/merged 2024-03-28T00:54:43.780 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay/679e79184319284edc180ed0ca34a1e28b024cfdfc358fbd82c8ee9e0ac5231d/merged 2024-03-28T00:54:43.780 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay/94a6f9c8f51b92d5819ea4f2551be1839c5b7c7b59e6145a8fad0d86ea721a04/merged 2024-03-28T00:54:43.780 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay/0f3bd14a5c2879df9518a6b491501c44a2ce521df8ce4664682bef6a90b90835/merged 2024-03-28T00:54:43.780 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay/d1bf068c4e82148a283b36602e06ba015d129b1186315c8f7231beb8a4703ec4/merged 2024-03-28T00:54:43.780 INFO:teuthology.orchestra.run.smithi169.stdout:/var/lib/containers/storage/overlay/2f938f19770bc16f74836b1acf5d8c8c1ad7df0cde769aa9b9f34f3ba6b250a5/merged 2024-03-28T00:54:43.780 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns 2024-03-28T00:54:43.780 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T00:54:43.780 INFO:teuthology.orchestra.run.smithi169.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T00:54:43.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:54:43.780 DEBUG:teuthology.orchestra.run.smithi169:> ls /sys/fs/fuse/connections 2024-03-28T00:54:43.786 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi169.stderr:2024-03-28T00:54:43.783+0000 7f2dcd3fb480 -1 init, newargv = 0x556685f66500 newargc=15 2024-03-28T00:54:43.786 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi169.stderr:ceph-fuse[74331]: starting ceph client 2024-03-28T00:54:43.791 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi169.stderr:ceph-fuse[74331]: starting fuse 2024-03-28T00:54:43.801 INFO:teuthology.orchestra.run.smithi169.stdout:119 2024-03-28T00:54:43.801 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-28T00:54:43.801 DEBUG:teuthology.orchestra.run.smithi169:> sudo stdin-killer -- python3 -c ' 2024-03-28T00:54:43.801 DEBUG:teuthology.orchestra.run.smithi169:> import glob 2024-03-28T00:54:43.801 DEBUG:teuthology.orchestra.run.smithi169:> import re 2024-03-28T00:54:43.801 DEBUG:teuthology.orchestra.run.smithi169:> import os 2024-03-28T00:54:43.801 DEBUG:teuthology.orchestra.run.smithi169:> import subprocess 2024-03-28T00:54:43.801 DEBUG:teuthology.orchestra.run.smithi169:> 2024-03-28T00:54:43.801 DEBUG:teuthology.orchestra.run.smithi169:> def _find_admin_socket(client_name): 2024-03-28T00:54:43.801 DEBUG:teuthology.orchestra.run.smithi169:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-28T00:54:43.802 DEBUG:teuthology.orchestra.run.smithi169:> files = glob.glob(asok_path) 2024-03-28T00:54:43.802 DEBUG:teuthology.orchestra.run.smithi169:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-28T00:54:43.802 DEBUG:teuthology.orchestra.run.smithi169:> 2024-03-28T00:54:43.802 DEBUG:teuthology.orchestra.run.smithi169:> # Given a non-glob path, it better be there 2024-03-28T00:54:43.802 DEBUG:teuthology.orchestra.run.smithi169:> if "*" not in asok_path: 2024-03-28T00:54:43.802 DEBUG:teuthology.orchestra.run.smithi169:> assert(len(files) == 1) 2024-03-28T00:54:43.802 DEBUG:teuthology.orchestra.run.smithi169:> return files[0] 2024-03-28T00:54:43.802 DEBUG:teuthology.orchestra.run.smithi169:> 2024-03-28T00:54:43.802 DEBUG:teuthology.orchestra.run.smithi169:> for f in files: 2024-03-28T00:54:43.802 DEBUG:teuthology.orchestra.run.smithi169:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-28T00:54:43.802 DEBUG:teuthology.orchestra.run.smithi169:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-28T00:54:43.802 DEBUG:teuthology.orchestra.run.smithi169:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-28T00:54:43.802 DEBUG:teuthology.orchestra.run.smithi169:> contents = proc_f.read() 2024-03-28T00:54:43.802 DEBUG:teuthology.orchestra.run.smithi169:> if mountpoint in contents: 2024-03-28T00:54:43.803 DEBUG:teuthology.orchestra.run.smithi169:> return f 2024-03-28T00:54:43.803 DEBUG:teuthology.orchestra.run.smithi169:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-28T00:54:43.803 DEBUG:teuthology.orchestra.run.smithi169:> 2024-03-28T00:54:43.803 DEBUG:teuthology.orchestra.run.smithi169:> print(_find_admin_socket("client.admin")) 2024-03-28T00:54:43.803 DEBUG:teuthology.orchestra.run.smithi169:> ' 2024-03-28T00:54:43.922 INFO:teuthology.orchestra.run.smithi169.stdout:/var/run/ceph/ceph-client.admin.74331.asok 2024-03-28T00:54:43.926 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T00:54:43.933 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.74331.asok 2024-03-28T00:54:43.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:54:43.933 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.74331.asok status 2024-03-28T00:54:44.069 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:54:44.069 INFO:teuthology.orchestra.run.smithi169.stdout: "metadata": { 2024-03-28T00:54:44.069 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-28T00:54:44.069 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T00:54:44.069 INFO:teuthology.orchestra.run.smithi169.stdout: "entity_id": "admin", 2024-03-28T00:54:44.069 INFO:teuthology.orchestra.run.smithi169.stdout: "hostname": "smithi169", 2024-03-28T00:54:44.069 INFO:teuthology.orchestra.run.smithi169.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-28T00:54:44.069 INFO:teuthology.orchestra.run.smithi169.stdout: "pid": "74331", 2024-03-28T00:54:44.070 INFO:teuthology.orchestra.run.smithi169.stdout: "root": "/" 2024-03-28T00:54:44.070 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T00:54:44.070 INFO:teuthology.orchestra.run.smithi169.stdout: "dentry_count": 0, 2024-03-28T00:54:44.070 INFO:teuthology.orchestra.run.smithi169.stdout: "dentry_pinned_count": 0, 2024-03-28T00:54:44.070 INFO:teuthology.orchestra.run.smithi169.stdout: "id": 14769, 2024-03-28T00:54:44.070 INFO:teuthology.orchestra.run.smithi169.stdout: "inst": { 2024-03-28T00:54:44.070 INFO:teuthology.orchestra.run.smithi169.stdout: "name": { 2024-03-28T00:54:44.070 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "client", 2024-03-28T00:54:44.070 INFO:teuthology.orchestra.run.smithi169.stdout: "num": 14769 2024-03-28T00:54:44.070 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T00:54:44.070 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": { 2024-03-28T00:54:44.071 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v1", 2024-03-28T00:54:44.071 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "192.168.0.1:0", 2024-03-28T00:54:44.071 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3617942795 2024-03-28T00:54:44.071 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T00:54:44.071 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T00:54:44.071 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": { 2024-03-28T00:54:44.071 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v1", 2024-03-28T00:54:44.071 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "192.168.0.1:0", 2024-03-28T00:54:44.071 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 3617942795 2024-03-28T00:54:44.071 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T00:54:44.071 INFO:teuthology.orchestra.run.smithi169.stdout: "inst_str": "client.14769 192.168.0.1:0/3617942795", 2024-03-28T00:54:44.071 INFO:teuthology.orchestra.run.smithi169.stdout: "addr_str": "192.168.0.1:0/3617942795", 2024-03-28T00:54:44.071 INFO:teuthology.orchestra.run.smithi169.stdout: "inode_count": 1, 2024-03-28T00:54:44.071 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_epoch": 17, 2024-03-28T00:54:44.071 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_epoch": 90, 2024-03-28T00:54:44.072 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_epoch_barrier": 0, 2024-03-28T00:54:44.072 INFO:teuthology.orchestra.run.smithi169.stdout: "blocklisted": false, 2024-03-28T00:54:44.072 INFO:teuthology.orchestra.run.smithi169.stdout: "fs_name": "cephfs" 2024-03-28T00:54:44.072 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:54:44.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:54:44.078 DEBUG:teuthology.orchestra.run.smithi169:> stat --file-system '--printf=%T 2024-03-28T00:54:44.078 DEBUG:teuthology.orchestra.run.smithi169:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-28T00:54:44.093 INFO:teuthology.orchestra.run.smithi169.stdout:fuseblk 2024-03-28T00:54:44.093 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-28T00:54:44.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:54:44.094 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-28T00:54:44.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:54:44.162 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-28T00:54:44.252 INFO:teuthology.orchestra.run.smithi169.stdout:# file: . 2024-03-28T00:54:44.252 INFO:teuthology.orchestra.run.smithi169.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-28T00:54:44.252 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:44.252 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-28T00:54:44.252 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T00:54:44.269 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T00:54:44.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.408+0000 7f246bf97640 1 -- 172.21.15.169:0/1823798447 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24641028b0 msgr2=0x7f2464102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:44.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.408+0000 7f246bf97640 1 --2- 172.21.15.169:0/1823798447 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24641028b0 0x7f2464102d30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2460009860 tx=0x7f246002bb70 comp rx=0 tx=0).stop 2024-03-28T00:54:44.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.408+0000 7f246bf97640 1 -- 172.21.15.169:0/1823798447 shutdown_connections 2024-03-28T00:54:44.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.408+0000 7f246bf97640 1 --2- 172.21.15.169:0/1823798447 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2464103270 0x7f2464109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:44.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.408+0000 7f246bf97640 1 --2- 172.21.15.169:0/1823798447 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24641028b0 0x7f2464102d30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:44.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.408+0000 7f246bf97640 1 --2- 172.21.15.169:0/1823798447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24641016c0 0x7f2464101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:44.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.408+0000 7f246bf97640 1 -- 172.21.15.169:0/1823798447 >> 172.21.15.169:0/1823798447 conn(0x7f24640fcc50 msgr2=0x7f24640ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:44.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.408+0000 7f246bf97640 1 -- 172.21.15.169:0/1823798447 shutdown_connections 2024-03-28T00:54:44.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.408+0000 7f246bf97640 1 -- 172.21.15.169:0/1823798447 wait complete. 2024-03-28T00:54:44.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f246bf97640 1 Processor -- start 2024-03-28T00:54:44.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f246bf97640 1 -- start start 2024-03-28T00:54:44.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f246bf97640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24641016c0 0x7f2464197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:44.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f246bf97640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24641028b0 0x7f2464197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:44.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f246bf97640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2464103270 0x7f246419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:44.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f246bf97640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246410d340 con 0x7f2464103270 2024-03-28T00:54:44.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f246bf97640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246410d1c0 con 0x7f24641028b0 2024-03-28T00:54:44.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f246bf97640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246410d4c0 con 0x7f24641016c0 2024-03-28T00:54:44.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f246a534640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2464103270 0x7f246419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:44.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f246a534640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2464103270 0x7f246419efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48930/0 (socket says 172.21.15.169:48930) 2024-03-28T00:54:44.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f246a534640 1 -- 172.21.15.169:0/1306063072 learned_addr learned my addr 172.21.15.169:0/1306063072 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:44.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f2469532640 1 --2- 172.21.15.169:0/1306063072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24641028b0 0x7f2464197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:44.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f2469d33640 1 --2- 172.21.15.169:0/1306063072 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24641016c0 0x7f2464197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:44.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f246a534640 1 -- 172.21.15.169:0/1306063072 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24641016c0 msgr2=0x7f2464197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:44.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f246a534640 1 --2- 172.21.15.169:0/1306063072 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24641016c0 0x7f2464197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:44.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f246a534640 1 -- 172.21.15.169:0/1306063072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24641028b0 msgr2=0x7f2464197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:44.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f246a534640 1 --2- 172.21.15.169:0/1306063072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24641028b0 0x7f2464197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:44.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f246a534640 1 -- 172.21.15.169:0/1306063072 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246419f620 con 0x7f2464103270 2024-03-28T00:54:44.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f2469532640 1 --2- 172.21.15.169:0/1306063072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24641028b0 0x7f2464197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:44.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f246a534640 1 --2- 172.21.15.169:0/1306063072 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2464103270 0x7f246419efb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f245c00efc0 tx=0x7f245c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:44.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f2456ffd640 1 -- 172.21.15.169:0/1306063072 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c003970 con 0x7f2464103270 2024-03-28T00:54:44.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f246bf97640 1 -- 172.21.15.169:0/1306063072 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f246419f910 con 0x7f2464103270 2024-03-28T00:54:44.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f2456ffd640 1 -- 172.21.15.169:0/1306063072 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f245c004020 con 0x7f2464103270 2024-03-28T00:54:44.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.409+0000 7f2456ffd640 1 -- 172.21.15.169:0/1306063072 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c00b9f0 con 0x7f2464103270 2024-03-28T00:54:44.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.410+0000 7f246bf97640 1 -- 172.21.15.169:0/1306063072 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f246419fde0 con 0x7f2464103270 2024-03-28T00:54:44.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.411+0000 7f246bf97640 1 -- 172.21.15.169:0/1306063072 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2464108ab0 con 0x7f2464103270 2024-03-28T00:54:44.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.412+0000 7f2456ffd640 1 -- 172.21.15.169:0/1306063072 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f245c01d020 con 0x7f2464103270 2024-03-28T00:54:44.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.412+0000 7f2456ffd640 1 --2- 172.21.15.169:0/1306063072 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f24500a2800 0x7f24500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:44.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.412+0000 7f2456ffd640 1 -- 172.21.15.169:0/1306063072 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f245c0cf170 con 0x7f2464103270 2024-03-28T00:54:44.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.412+0000 7f2469d33640 1 --2- 172.21.15.169:0/1306063072 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f24500a2800 0x7f24500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:44.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.413+0000 7f2469d33640 1 --2- 172.21.15.169:0/1306063072 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f24500a2800 0x7f24500a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2458005fd0 tx=0x7f245800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:44.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.416+0000 7f2456ffd640 1 -- 172.21.15.169:0/1306063072 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f245c09bfd0 con 0x7f2464103270 2024-03-28T00:54:44.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.594+0000 7f246bf97640 1 -- 172.21.15.169:0/1306063072 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24640622a0 con 0x7f2464103270 2024-03-28T00:54:44.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.596+0000 7f2456ffd640 1 -- 172.21.15.169:0/1306063072 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24186 (secure 0 0 0) 0x7f245c09b8d0 con 0x7f2464103270 2024-03-28T00:54:44.599 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:44.599 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":90,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:54:38.804421+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:54:44.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.598+0000 7f246bf97640 1 -- 172.21.15.169:0/1306063072 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f24500a2800 msgr2=0x7f24500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:44.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.598+0000 7f246bf97640 1 --2- 172.21.15.169:0/1306063072 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f24500a2800 0x7f24500a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2458005fd0 tx=0x7f245800b390 comp rx=0 tx=0).stop 2024-03-28T00:54:44.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.598+0000 7f246bf97640 1 -- 172.21.15.169:0/1306063072 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2464103270 msgr2=0x7f246419efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:44.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.598+0000 7f246bf97640 1 --2- 172.21.15.169:0/1306063072 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2464103270 0x7f246419efb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f245c00efc0 tx=0x7f245c00c800 comp rx=0 tx=0).stop 2024-03-28T00:54:44.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.598+0000 7f246bf97640 1 -- 172.21.15.169:0/1306063072 shutdown_connections 2024-03-28T00:54:44.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.598+0000 7f246bf97640 1 --2- 172.21.15.169:0/1306063072 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f24500a2800 0x7f24500a4cc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:44.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.598+0000 7f246bf97640 1 --2- 172.21.15.169:0/1306063072 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2464103270 0x7f246419efb0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:44.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.598+0000 7f246bf97640 1 --2- 172.21.15.169:0/1306063072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24641028b0 0x7f2464197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:44.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.598+0000 7f246bf97640 1 --2- 172.21.15.169:0/1306063072 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24641016c0 0x7f2464197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:44.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.598+0000 7f246bf97640 1 -- 172.21.15.169:0/1306063072 >> 172.21.15.169:0/1306063072 conn(0x7f24640fcc50 msgr2=0x7f2464105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:44.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.598+0000 7f246bf97640 1 -- 172.21.15.169:0/1306063072 shutdown_connections 2024-03-28T00:54:44.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.598+0000 7f246bf97640 1 -- 172.21.15.169:0/1306063072 wait complete. 2024-03-28T00:54:44.614 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T00:49:56.339525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T00:54:31.182562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T00:54:31.780429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T00:54:35.035724+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T00:54:44.614 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T00:54:44.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.718+0000 7f19b0019640 1 -- 172.21.15.169:0/2220305282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19a81028a0 msgr2=0x7f19a8102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:44.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.718+0000 7f19b0019640 1 --2- 172.21.15.169:0/2220305282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19a81028a0 0x7f19a8102d20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f19a4009900 tx=0x7f19a402bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:44.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.718+0000 7f19b0019640 1 -- 172.21.15.169:0/2220305282 shutdown_connections 2024-03-28T00:54:44.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.718+0000 7f19b0019640 1 --2- 172.21.15.169:0/2220305282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19a8103260 0x7f19a8109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:44.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.718+0000 7f19b0019640 1 --2- 172.21.15.169:0/2220305282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19a81028a0 0x7f19a8102d20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:44.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.718+0000 7f19b0019640 1 --2- 172.21.15.169:0/2220305282 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19a81016b0 0x7f19a8101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:44.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.718+0000 7f19b0019640 1 -- 172.21.15.169:0/2220305282 >> 172.21.15.169:0/2220305282 conn(0x7f19a80fcc40 msgr2=0x7f19a80ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:44.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.718+0000 7f19b0019640 1 -- 172.21.15.169:0/2220305282 shutdown_connections 2024-03-28T00:54:44.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.718+0000 7f19b0019640 1 -- 172.21.15.169:0/2220305282 wait complete. 2024-03-28T00:54:44.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f19b0019640 1 Processor -- start 2024-03-28T00:54:44.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f19b0019640 1 -- start start 2024-03-28T00:54:44.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f19b0019640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19a81016b0 0x7f19a81979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:44.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f19b0019640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19a81028a0 0x7f19a8197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:44.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f19b0019640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19a8103260 0x7f19a819ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:44.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f19b0019640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a810d330 con 0x7f19a81016b0 2024-03-28T00:54:44.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f19b0019640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a810d1b0 con 0x7f19a81028a0 2024-03-28T00:54:44.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f19b0019640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a810d4b0 con 0x7f19a8103260 2024-03-28T00:54:44.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f19addb5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19a81016b0 0x7f19a81979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:44.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f19addb5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19a81016b0 0x7f19a81979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48954/0 (socket says 172.21.15.169:48954) 2024-03-28T00:54:44.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f19addb5640 1 -- 172.21.15.169:0/1000155011 learned_addr learned my addr 172.21.15.169:0/1000155011 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:44.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f19ad5b4640 1 --2- 172.21.15.169:0/1000155011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19a81028a0 0x7f19a8197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:44.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f19ae5b6640 1 --2- 172.21.15.169:0/1000155011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19a8103260 0x7f19a819ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:44.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f19addb5640 1 -- 172.21.15.169:0/1000155011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19a8103260 msgr2=0x7f19a819ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:44.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f19addb5640 1 --2- 172.21.15.169:0/1000155011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19a8103260 0x7f19a819ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:44.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f19addb5640 1 -- 172.21.15.169:0/1000155011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19a81028a0 msgr2=0x7f19a8197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:44.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f19addb5640 1 --2- 172.21.15.169:0/1000155011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19a81028a0 0x7f19a8197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:44.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f19addb5640 1 -- 172.21.15.169:0/1000155011 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a819f5f0 con 0x7f19a81016b0 2024-03-28T00:54:44.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f19addb5640 1 --2- 172.21.15.169:0/1000155011 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19a81016b0 0x7f19a81979d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f199800e9c0 tx=0x7f199800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:44.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f199effd640 1 -- 172.21.15.169:0/1000155011 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199800cd80 con 0x7f19a81016b0 2024-03-28T00:54:44.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f199effd640 1 -- 172.21.15.169:0/1000155011 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19980035f0 con 0x7f19a81016b0 2024-03-28T00:54:44.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.719+0000 7f19b0019640 1 -- 172.21.15.169:0/1000155011 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a819f8e0 con 0x7f19a81016b0 2024-03-28T00:54:44.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.720+0000 7f199effd640 1 -- 172.21.15.169:0/1000155011 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1998010690 con 0x7f19a81016b0 2024-03-28T00:54:44.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.720+0000 7f19b0019640 1 -- 172.21.15.169:0/1000155011 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a819fe00 con 0x7f19a81016b0 2024-03-28T00:54:44.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.720+0000 7f19b0019640 1 -- 172.21.15.169:0/1000155011 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a8108a50 con 0x7f19a81016b0 2024-03-28T00:54:44.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.722+0000 7f199effd640 1 -- 172.21.15.169:0/1000155011 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f19980037a0 con 0x7f19a81016b0 2024-03-28T00:54:44.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.722+0000 7f199effd640 1 --2- 172.21.15.169:0/1000155011 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f19940a2800 0x7f19940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:44.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.722+0000 7f19ad5b4640 1 --2- 172.21.15.169:0/1000155011 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f19940a2800 0x7f19940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:44.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.722+0000 7f199effd640 1 -- 172.21.15.169:0/1000155011 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f19980cf050 con 0x7f19a81016b0 2024-03-28T00:54:44.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.723+0000 7f19ad5b4640 1 --2- 172.21.15.169:0/1000155011 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f19940a2800 0x7f19940a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f19a402c7b0 tx=0x7f19a4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:44.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.727+0000 7f199effd640 1 -- 172.21.15.169:0/1000155011 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f199809beb0 con 0x7f19a81016b0 2024-03-28T00:54:44.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.903+0000 7f19b0019640 1 -- 172.21.15.169:0/1000155011 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f19a80622a0 con 0x7f19a81016b0 2024-03-28T00:54:44.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.903+0000 7f199effd640 1 -- 172.21.15.169:0/1000155011 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f199809b7b0 con 0x7f19a81016b0 2024-03-28T00:54:44.907 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 1 2024-03-28T00:54:44.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.905+0000 7f19b0019640 1 -- 172.21.15.169:0/1000155011 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f19940a2800 msgr2=0x7f19940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:44.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.905+0000 7f19b0019640 1 --2- 172.21.15.169:0/1000155011 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f19940a2800 0x7f19940a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f19a402c7b0 tx=0x7f19a4005fd0 comp rx=0 tx=0).stop 2024-03-28T00:54:44.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.905+0000 7f19b0019640 1 -- 172.21.15.169:0/1000155011 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19a81016b0 msgr2=0x7f19a81979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:44.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.905+0000 7f19b0019640 1 --2- 172.21.15.169:0/1000155011 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19a81016b0 0x7f19a81979d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f199800e9c0 tx=0x7f199800ee80 comp rx=0 tx=0).stop 2024-03-28T00:54:44.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.905+0000 7f19b0019640 1 -- 172.21.15.169:0/1000155011 shutdown_connections 2024-03-28T00:54:44.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.905+0000 7f19b0019640 1 --2- 172.21.15.169:0/1000155011 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f19940a2800 0x7f19940a4cc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:44.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.905+0000 7f19b0019640 1 --2- 172.21.15.169:0/1000155011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19a8103260 0x7f19a819ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:44.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.905+0000 7f19b0019640 1 --2- 172.21.15.169:0/1000155011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19a81028a0 0x7f19a8197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:44.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.905+0000 7f19b0019640 1 --2- 172.21.15.169:0/1000155011 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19a81016b0 0x7f19a81979d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:44.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.905+0000 7f19b0019640 1 -- 172.21.15.169:0/1000155011 >> 172.21.15.169:0/1000155011 conn(0x7f19a80fcc40 msgr2=0x7f19a8105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:44.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.905+0000 7f19b0019640 1 -- 172.21.15.169:0/1000155011 shutdown_connections 2024-03-28T00:54:44.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:44.905+0000 7f19b0019640 1 -- 172.21.15.169:0/1000155011 wait complete. 2024-03-28T00:54:44.920 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T00:54:45.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.021+0000 7f5750377640 1 -- 172.21.15.169:0/82787099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57481021a0 msgr2=0x7f5748104590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:45.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.021+0000 7f5750377640 1 --2- 172.21.15.169:0/82787099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57481021a0 0x7f5748104590 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5744009860 tx=0x7f574402bb70 comp rx=0 tx=0).stop 2024-03-28T00:54:45.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.021+0000 7f5750377640 1 -- 172.21.15.169:0/82787099 shutdown_connections 2024-03-28T00:54:45.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.021+0000 7f5750377640 1 --2- 172.21.15.169:0/82787099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5748107400 0x7f57481097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.021+0000 7f5750377640 1 --2- 172.21.15.169:0/82787099 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5748104ad0 0x7f5748106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.021+0000 7f5750377640 1 --2- 172.21.15.169:0/82787099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57481021a0 0x7f5748104590 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.021+0000 7f5750377640 1 -- 172.21.15.169:0/82787099 >> 172.21.15.169:0/82787099 conn(0x7f57480fbb50 msgr2=0x7f57480fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:45.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.021+0000 7f5750377640 1 -- 172.21.15.169:0/82787099 shutdown_connections 2024-03-28T00:54:45.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.021+0000 7f5750377640 1 -- 172.21.15.169:0/82787099 wait complete. 2024-03-28T00:54:45.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.022+0000 7f5750377640 1 Processor -- start 2024-03-28T00:54:45.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.022+0000 7f5750377640 1 -- start start 2024-03-28T00:54:45.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.022+0000 7f5750377640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57481021a0 0x7f5748197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:45.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.022+0000 7f5750377640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5748104ad0 0x7f5748197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:45.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.022+0000 7f5750377640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5748107400 0x7f574819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:45.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.022+0000 7f5750377640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574810d340 con 0x7f5748104ad0 2024-03-28T00:54:45.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.022+0000 7f5750377640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574810d1c0 con 0x7f57481021a0 2024-03-28T00:54:45.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.022+0000 7f5750377640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574810d4c0 con 0x7f5748107400 2024-03-28T00:54:45.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.022+0000 7f574d912640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5748104ad0 0x7f5748197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:45.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.022+0000 7f574d912640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5748104ad0 0x7f5748197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48976/0 (socket says 172.21.15.169:48976) 2024-03-28T00:54:45.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.022+0000 7f574e113640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57481021a0 0x7f5748197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:45.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.022+0000 7f574d912640 1 -- 172.21.15.169:0/3650561964 learned_addr learned my addr 172.21.15.169:0/3650561964 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:45.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.022+0000 7f574e113640 1 -- 172.21.15.169:0/3650561964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5748107400 msgr2=0x7f574819ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:45.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.022+0000 7f574e113640 1 --2- 172.21.15.169:0/3650561964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5748107400 0x7f574819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.022+0000 7f574e113640 1 -- 172.21.15.169:0/3650561964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5748104ad0 msgr2=0x7f5748197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:45.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.022+0000 7f574e113640 1 --2- 172.21.15.169:0/3650561964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5748104ad0 0x7f5748197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.022+0000 7f574e113640 1 -- 172.21.15.169:0/3650561964 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574819f5b0 con 0x7f57481021a0 2024-03-28T00:54:45.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.022+0000 7f574d912640 1 --2- 172.21.15.169:0/3650561964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5748104ad0 0x7f5748197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:45.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.023+0000 7f574e113640 1 --2- 172.21.15.169:0/3650561964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57481021a0 0x7f5748197990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f574402c0b0 tx=0x7f57440042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:45.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.023+0000 7f573f7fe640 1 -- 172.21.15.169:0/3650561964 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5744041020 con 0x7f57481021a0 2024-03-28T00:54:45.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.023+0000 7f573f7fe640 1 -- 172.21.15.169:0/3650561964 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57440347b0 con 0x7f57481021a0 2024-03-28T00:54:45.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.023+0000 7f573f7fe640 1 -- 172.21.15.169:0/3650561964 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5744039040 con 0x7f57481021a0 2024-03-28T00:54:45.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.023+0000 7f5750377640 1 -- 172.21.15.169:0/3650561964 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f574819f840 con 0x7f57481021a0 2024-03-28T00:54:45.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.023+0000 7f5750377640 1 -- 172.21.15.169:0/3650561964 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574819fd30 con 0x7f57481021a0 2024-03-28T00:54:45.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.024+0000 7f5750377640 1 -- 172.21.15.169:0/3650561964 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57480fd640 con 0x7f57481021a0 2024-03-28T00:54:45.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.025+0000 7f573f7fe640 1 -- 172.21.15.169:0/3650561964 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5744044020 con 0x7f57481021a0 2024-03-28T00:54:45.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.025+0000 7f573f7fe640 1 --2- 172.21.15.169:0/3650561964 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f57340a2800 0x7f57340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:45.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.026+0000 7f574d912640 1 --2- 172.21.15.169:0/3650561964 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f57340a2800 0x7f57340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:45.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.026+0000 7f573f7fe640 1 -- 172.21.15.169:0/3650561964 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f57440f09a0 con 0x7f57481021a0 2024-03-28T00:54:45.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.026+0000 7f574d912640 1 --2- 172.21.15.169:0/3650561964 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f57340a2800 0x7f57340a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5748198ea0 tx=0x7f5738009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:45.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.029+0000 7f573f7fe640 1 -- 172.21.15.169:0/3650561964 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f574404b020 con 0x7f57481021a0 2024-03-28T00:54:45.207 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.203+0000 7f5750377640 1 -- 172.21.15.169:0/3650561964 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5748062340 con 0x7f57481021a0 2024-03-28T00:54:45.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.205+0000 7f573f7fe640 1 -- 172.21.15.169:0/3650561964 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f57440bd800 con 0x7f57481021a0 2024-03-28T00:54:45.208 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T00:54:45.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.205+0000 7f5750377640 1 -- 172.21.15.169:0/3650561964 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f57340a2800 msgr2=0x7f57340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:45.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.205+0000 7f5750377640 1 --2- 172.21.15.169:0/3650561964 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f57340a2800 0x7f57340a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5748198ea0 tx=0x7f5738009260 comp rx=0 tx=0).stop 2024-03-28T00:54:45.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.206+0000 7f5750377640 1 -- 172.21.15.169:0/3650561964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57481021a0 msgr2=0x7f5748197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:45.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.206+0000 7f5750377640 1 --2- 172.21.15.169:0/3650561964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57481021a0 0x7f5748197990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f574402c0b0 tx=0x7f57440042f0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.206+0000 7f5750377640 1 -- 172.21.15.169:0/3650561964 shutdown_connections 2024-03-28T00:54:45.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.206+0000 7f5750377640 1 --2- 172.21.15.169:0/3650561964 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f57340a2800 0x7f57340a4cc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.206+0000 7f5750377640 1 --2- 172.21.15.169:0/3650561964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5748107400 0x7f574819ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.206+0000 7f5750377640 1 --2- 172.21.15.169:0/3650561964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5748104ad0 0x7f5748197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.206+0000 7f5750377640 1 --2- 172.21.15.169:0/3650561964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57481021a0 0x7f5748197990 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.206+0000 7f5750377640 1 -- 172.21.15.169:0/3650561964 >> 172.21.15.169:0/3650561964 conn(0x7f57480fbb50 msgr2=0x7f5748102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:45.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.206+0000 7f5750377640 1 -- 172.21.15.169:0/3650561964 shutdown_connections 2024-03-28T00:54:45.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.206+0000 7f5750377640 1 -- 172.21.15.169:0/3650561964 wait complete. 2024-03-28T00:54:45.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T00:54:45.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.325+0000 7f197b153640 1 -- 172.21.15.169:0/2525740405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1974101790 msgr2=0x7f1974101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:45.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.325+0000 7f197b153640 1 --2- 172.21.15.169:0/2525740405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1974101790 0x7f1974101c10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1960009900 tx=0x7f196002bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:45.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.325+0000 7f197b153640 1 -- 172.21.15.169:0/2525740405 shutdown_connections 2024-03-28T00:54:45.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.325+0000 7f197b153640 1 --2- 172.21.15.169:0/2525740405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1974102150 0x7f19741089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.325+0000 7f197b153640 1 --2- 172.21.15.169:0/2525740405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1974101790 0x7f1974101c10 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.325+0000 7f197b153640 1 --2- 172.21.15.169:0/2525740405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19741005a0 0x7f19741009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.325+0000 7f197b153640 1 -- 172.21.15.169:0/2525740405 >> 172.21.15.169:0/2525740405 conn(0x7f19740fbb30 msgr2=0x7f19740fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:45.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.325+0000 7f197b153640 1 -- 172.21.15.169:0/2525740405 shutdown_connections 2024-03-28T00:54:45.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.325+0000 7f197b153640 1 -- 172.21.15.169:0/2525740405 wait complete. 2024-03-28T00:54:45.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.326+0000 7f197b153640 1 Processor -- start 2024-03-28T00:54:45.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.326+0000 7f197b153640 1 -- start start 2024-03-28T00:54:45.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.326+0000 7f197b153640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19741005a0 0x7f1974069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:45.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.326+0000 7f197b153640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1974101790 0x7f197419c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:45.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.326+0000 7f197b153640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1974102150 0x7f197419ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:45.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.326+0000 7f197b153640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197410c230 con 0x7f1974102150 2024-03-28T00:54:45.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.326+0000 7f197b153640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197410c0b0 con 0x7f1974101790 2024-03-28T00:54:45.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.326+0000 7f197b153640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197410c3b0 con 0x7f19741005a0 2024-03-28T00:54:45.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.326+0000 7f19796f0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1974102150 0x7f197419ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:45.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.326+0000 7f1973fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1974101790 0x7f197419c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:45.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.326+0000 7f19796f0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1974102150 0x7f197419ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48996/0 (socket says 172.21.15.169:48996) 2024-03-28T00:54:45.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.326+0000 7f19796f0640 1 -- 172.21.15.169:0/4224973896 learned_addr learned my addr 172.21.15.169:0/4224973896 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:45.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.326+0000 7f1978eef640 1 --2- 172.21.15.169:0/4224973896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19741005a0 0x7f1974069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:45.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.327+0000 7f1973fff640 1 -- 172.21.15.169:0/4224973896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19741005a0 msgr2=0x7f1974069390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:45.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.327+0000 7f1973fff640 1 --2- 172.21.15.169:0/4224973896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19741005a0 0x7f1974069390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.327+0000 7f1973fff640 1 -- 172.21.15.169:0/4224973896 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1974102150 msgr2=0x7f197419ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:45.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.327+0000 7f1973fff640 1 --2- 172.21.15.169:0/4224973896 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1974102150 0x7f197419ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.327+0000 7f1973fff640 1 -- 172.21.15.169:0/4224973896 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197419f600 con 0x7f1974101790 2024-03-28T00:54:45.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.327+0000 7f19796f0640 1 --2- 172.21.15.169:0/4224973896 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1974102150 0x7f197419ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:45.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.327+0000 7f1973fff640 1 --2- 172.21.15.169:0/4224973896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1974101790 0x7f197419c700 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1960009230 tx=0x7f196002c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:45.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.327+0000 7f1971ffb640 1 -- 172.21.15.169:0/4224973896 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960041070 con 0x7f1974101790 2024-03-28T00:54:45.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.327+0000 7f1971ffb640 1 -- 172.21.15.169:0/4224973896 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1960004450 con 0x7f1974101790 2024-03-28T00:54:45.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.327+0000 7f1971ffb640 1 -- 172.21.15.169:0/4224973896 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960034440 con 0x7f1974101790 2024-03-28T00:54:45.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.327+0000 7f197b153640 1 -- 172.21.15.169:0/4224973896 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f197419f890 con 0x7f1974101790 2024-03-28T00:54:45.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.327+0000 7f197b153640 1 -- 172.21.15.169:0/4224973896 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f197419fdd0 con 0x7f1974101790 2024-03-28T00:54:45.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.328+0000 7f197b153640 1 -- 172.21.15.169:0/4224973896 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19741078e0 con 0x7f1974101790 2024-03-28T00:54:45.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.329+0000 7f1971ffb640 1 -- 172.21.15.169:0/4224973896 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1960044070 con 0x7f1974101790 2024-03-28T00:54:45.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.329+0000 7f1971ffb640 1 --2- 172.21.15.169:0/4224973896 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f19640a2880 0x7f19640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:45.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.330+0000 7f1978eef640 1 --2- 172.21.15.169:0/4224973896 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f19640a2880 0x7f19640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:45.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.330+0000 7f1971ffb640 1 -- 172.21.15.169:0/4224973896 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f19600f0800 con 0x7f1974101790 2024-03-28T00:54:45.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.330+0000 7f1978eef640 1 --2- 172.21.15.169:0/4224973896 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f19640a2880 0x7f19640a4d40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1968005d00 tx=0x7f1968005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:45.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.334+0000 7f1971ffb640 1 -- 172.21.15.169:0/4224973896 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f19600bd660 con 0x7f1974101790 2024-03-28T00:54:45.493 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:45 smithi169 ceph-mon[32274]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 422 B/s wr, 0 op/s 2024-03-28T00:54:45.493 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:45 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1306063072' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:45.493 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:45 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1000155011' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:54:45.493 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:45 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3650561964' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:54:45.520 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.516+0000 7f197b153640 1 -- 172.21.15.169:0/4224973896 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f19740622a0 con 0x7f1974101790 2024-03-28T00:54:45.520 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.516+0000 7f1971ffb640 1 -- 172.21.15.169:0/4224973896 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7f19600bcf60 con 0x7f1974101790 2024-03-28T00:54:45.520 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 64 2024-03-28T00:54:45.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.518+0000 7f197b153640 1 -- 172.21.15.169:0/4224973896 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f19640a2880 msgr2=0x7f19640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:45.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.518+0000 7f197b153640 1 --2- 172.21.15.169:0/4224973896 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f19640a2880 0x7f19640a4d40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1968005d00 tx=0x7f1968005c50 comp rx=0 tx=0).stop 2024-03-28T00:54:45.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.518+0000 7f197b153640 1 -- 172.21.15.169:0/4224973896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1974101790 msgr2=0x7f197419c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:45.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.518+0000 7f197b153640 1 --2- 172.21.15.169:0/4224973896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1974101790 0x7f197419c700 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1960009230 tx=0x7f196002c6c0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.518+0000 7f197b153640 1 -- 172.21.15.169:0/4224973896 shutdown_connections 2024-03-28T00:54:45.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.518+0000 7f197b153640 1 --2- 172.21.15.169:0/4224973896 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f19640a2880 0x7f19640a4d40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.518+0000 7f197b153640 1 --2- 172.21.15.169:0/4224973896 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1974102150 0x7f197419ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.518+0000 7f197b153640 1 --2- 172.21.15.169:0/4224973896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1974101790 0x7f197419c700 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.518+0000 7f197b153640 1 --2- 172.21.15.169:0/4224973896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19741005a0 0x7f1974069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.518+0000 7f197b153640 1 -- 172.21.15.169:0/4224973896 >> 172.21.15.169:0/4224973896 conn(0x7f19740fbb30 msgr2=0x7f1974104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:45.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.519+0000 7f197b153640 1 -- 172.21.15.169:0/4224973896 shutdown_connections 2024-03-28T00:54:45.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.519+0000 7f197b153640 1 -- 172.21.15.169:0/4224973896 wait complete. 2024-03-28T00:54:45.533 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T00:54:45.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:45 smithi174 ceph-mon[36060]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 422 B/s wr, 0 op/s 2024-03-28T00:54:45.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:45 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1306063072' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:45.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:45 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1000155011' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:54:45.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:45 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3650561964' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:54:45.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.640+0000 7fd0963a9640 1 -- 172.21.15.169:0/283354316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0901017a0 msgr2=0x7fd090101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:45.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.640+0000 7fd0963a9640 1 --2- 172.21.15.169:0/283354316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0901017a0 0x7fd090101c20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd084009860 tx=0x7fd08402bb70 comp rx=0 tx=0).stop 2024-03-28T00:54:45.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.640+0000 7fd0963a9640 1 -- 172.21.15.169:0/283354316 shutdown_connections 2024-03-28T00:54:45.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.640+0000 7fd0963a9640 1 --2- 172.21.15.169:0/283354316 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd090102160 0x7fd0901089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.640+0000 7fd0963a9640 1 --2- 172.21.15.169:0/283354316 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0901017a0 0x7fd090101c20 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.640+0000 7fd0963a9640 1 --2- 172.21.15.169:0/283354316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0901005b0 0x7fd0901009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.640+0000 7fd0963a9640 1 -- 172.21.15.169:0/283354316 >> 172.21.15.169:0/283354316 conn(0x7fd0900fbb40 msgr2=0x7fd0900fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:45.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.640+0000 7fd0963a9640 1 -- 172.21.15.169:0/283354316 shutdown_connections 2024-03-28T00:54:45.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.640+0000 7fd0963a9640 1 -- 172.21.15.169:0/283354316 wait complete. 2024-03-28T00:54:45.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.641+0000 7fd0963a9640 1 Processor -- start 2024-03-28T00:54:45.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.641+0000 7fd0963a9640 1 -- start start 2024-03-28T00:54:45.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.641+0000 7fd0963a9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0901005b0 0x7fd09006a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:45.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.641+0000 7fd0963a9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0901017a0 0x7fd09019c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:45.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.641+0000 7fd0963a9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd090102160 0x7fd09019eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:45.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.641+0000 7fd0963a9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09010c230 con 0x7fd090102160 2024-03-28T00:54:45.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.641+0000 7fd0963a9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09010c0b0 con 0x7fd0901017a0 2024-03-28T00:54:45.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.641+0000 7fd0963a9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09010c3b0 con 0x7fd0901005b0 2024-03-28T00:54:45.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.641+0000 7fd094946640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd090102160 0x7fd09019eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:45.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.641+0000 7fd094946640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd090102160 0x7fd09019eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49018/0 (socket says 172.21.15.169:49018) 2024-03-28T00:54:45.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.641+0000 7fd094946640 1 -- 172.21.15.169:0/1461319752 learned_addr learned my addr 172.21.15.169:0/1461319752 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:45.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.641+0000 7fd08f7fe640 1 --2- 172.21.15.169:0/1461319752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0901017a0 0x7fd09019c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:45.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.641+0000 7fd08ffff640 1 --2- 172.21.15.169:0/1461319752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0901005b0 0x7fd09006a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:45.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.641+0000 7fd08f7fe640 1 -- 172.21.15.169:0/1461319752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0901005b0 msgr2=0x7fd09006a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:45.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.641+0000 7fd08f7fe640 1 --2- 172.21.15.169:0/1461319752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0901005b0 0x7fd09006a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.641+0000 7fd08f7fe640 1 -- 172.21.15.169:0/1461319752 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd090102160 msgr2=0x7fd09019eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:45.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.641+0000 7fd08f7fe640 1 --2- 172.21.15.169:0/1461319752 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd090102160 0x7fd09019eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.641+0000 7fd08f7fe640 1 -- 172.21.15.169:0/1461319752 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09019f660 con 0x7fd0901017a0 2024-03-28T00:54:45.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.641+0000 7fd094946640 1 --2- 172.21.15.169:0/1461319752 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd090102160 0x7fd09019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:45.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.642+0000 7fd08f7fe640 1 --2- 172.21.15.169:0/1461319752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0901017a0 0x7fd09019c760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd08402c0b0 tx=0x7fd084032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:45.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.642+0000 7fd08d7fa640 1 -- 172.21.15.169:0/1461319752 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd084041020 con 0x7fd0901017a0 2024-03-28T00:54:45.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.642+0000 7fd0963a9640 1 -- 172.21.15.169:0/1461319752 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd09019f8f0 con 0x7fd0901017a0 2024-03-28T00:54:45.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.642+0000 7fd08d7fa640 1 -- 172.21.15.169:0/1461319752 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd08402c9f0 con 0x7fd0901017a0 2024-03-28T00:54:45.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.642+0000 7fd08d7fa640 1 -- 172.21.15.169:0/1461319752 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd084039040 con 0x7fd0901017a0 2024-03-28T00:54:45.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.642+0000 7fd0963a9640 1 -- 172.21.15.169:0/1461319752 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd09019fe00 con 0x7fd0901017a0 2024-03-28T00:54:45.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.643+0000 7fd0963a9640 1 -- 172.21.15.169:0/1461319752 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd090107940 con 0x7fd0901017a0 2024-03-28T00:54:45.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.644+0000 7fd08d7fa640 1 -- 172.21.15.169:0/1461319752 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd084044020 con 0x7fd0901017a0 2024-03-28T00:54:45.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.644+0000 7fd08d7fa640 1 --2- 172.21.15.169:0/1461319752 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd0780a2800 0x7fd0780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:45.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.645+0000 7fd08ffff640 1 --2- 172.21.15.169:0/1461319752 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd0780a2800 0x7fd0780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:45.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.645+0000 7fd08d7fa640 1 -- 172.21.15.169:0/1461319752 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd0840f0910 con 0x7fd0901017a0 2024-03-28T00:54:45.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.645+0000 7fd08ffff640 1 --2- 172.21.15.169:0/1461319752 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd0780a2800 0x7fd0780a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd090101600 tx=0x7fd080009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:45.656 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.652+0000 7fd08d7fa640 1 -- 172.21.15.169:0/1461319752 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd08404b020 con 0x7fd0901017a0 2024-03-28T00:54:45.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:45 smithi177 ceph-mon[36206]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 422 B/s wr, 0 op/s 2024-03-28T00:54:45.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:45 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1306063072' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:45.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:45 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1000155011' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:54:45.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:45 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3650561964' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:54:45.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.836+0000 7fd0963a9640 1 -- 172.21.15.169:0/1461319752 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd09002d270 con 0x7fd0901017a0 2024-03-28T00:54:45.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.837+0000 7fd08d7fa640 1 -- 172.21.15.169:0/1461319752 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd0840bd770 con 0x7fd0901017a0 2024-03-28T00:54:45.841 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T00:54:45.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.839+0000 7fd0963a9640 1 -- 172.21.15.169:0/1461319752 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd0780a2800 msgr2=0x7fd0780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:45.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.839+0000 7fd0963a9640 1 --2- 172.21.15.169:0/1461319752 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd0780a2800 0x7fd0780a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd090101600 tx=0x7fd080009260 comp rx=0 tx=0).stop 2024-03-28T00:54:45.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.839+0000 7fd0963a9640 1 -- 172.21.15.169:0/1461319752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0901017a0 msgr2=0x7fd09019c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:45.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.839+0000 7fd0963a9640 1 --2- 172.21.15.169:0/1461319752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0901017a0 0x7fd09019c760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd08402c0b0 tx=0x7fd084032430 comp rx=0 tx=0).stop 2024-03-28T00:54:45.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.839+0000 7fd0963a9640 1 -- 172.21.15.169:0/1461319752 shutdown_connections 2024-03-28T00:54:45.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.839+0000 7fd0963a9640 1 --2- 172.21.15.169:0/1461319752 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd0780a2800 0x7fd0780a4cc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.839+0000 7fd0963a9640 1 --2- 172.21.15.169:0/1461319752 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd090102160 0x7fd09019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.839+0000 7fd0963a9640 1 --2- 172.21.15.169:0/1461319752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0901017a0 0x7fd09019c760 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.839+0000 7fd0963a9640 1 --2- 172.21.15.169:0/1461319752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0901005b0 0x7fd09006a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.839+0000 7fd0963a9640 1 -- 172.21.15.169:0/1461319752 >> 172.21.15.169:0/1461319752 conn(0x7fd0900fbb40 msgr2=0x7fd090104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:45.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.839+0000 7fd0963a9640 1 -- 172.21.15.169:0/1461319752 shutdown_connections 2024-03-28T00:54:45.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.839+0000 7fd0963a9640 1 -- 172.21.15.169:0/1461319752 wait complete. 2024-03-28T00:54:45.855 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T00:54:45.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.958+0000 7f0a1b53b640 1 -- 172.21.15.169:0/2749089975 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a141073e0 msgr2=0x7f0a141097d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:45.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.958+0000 7f0a1b53b640 1 --2- 172.21.15.169:0/2749089975 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a141073e0 0x7f0a141097d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009900 tx=0x7f0a1002bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:45.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.958+0000 7f0a1b53b640 1 -- 172.21.15.169:0/2749089975 shutdown_connections 2024-03-28T00:54:45.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.958+0000 7f0a1b53b640 1 --2- 172.21.15.169:0/2749089975 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a141073e0 0x7f0a141097d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.958+0000 7f0a1b53b640 1 --2- 172.21.15.169:0/2749089975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a14104ab0 0x7f0a14106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.958+0000 7f0a1b53b640 1 --2- 172.21.15.169:0/2749089975 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a14102180 0x7f0a14104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.958+0000 7f0a1b53b640 1 -- 172.21.15.169:0/2749089975 >> 172.21.15.169:0/2749089975 conn(0x7f0a140fbb30 msgr2=0x7f0a140fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:45.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.959+0000 7f0a1b53b640 1 -- 172.21.15.169:0/2749089975 shutdown_connections 2024-03-28T00:54:45.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.959+0000 7f0a1b53b640 1 -- 172.21.15.169:0/2749089975 wait complete. 2024-03-28T00:54:45.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.959+0000 7f0a1b53b640 1 Processor -- start 2024-03-28T00:54:45.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.959+0000 7f0a1b53b640 1 -- start start 2024-03-28T00:54:45.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.959+0000 7f0a1b53b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a14102180 0x7f0a14199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:45.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.959+0000 7f0a1b53b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a14104ab0 0x7f0a1419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:45.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.959+0000 7f0a1b53b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a141073e0 0x7f0a141a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:45.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.959+0000 7f0a1b53b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1410d340 con 0x7f0a14104ab0 2024-03-28T00:54:45.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.959+0000 7f0a1b53b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1410d1c0 con 0x7f0a141073e0 2024-03-28T00:54:45.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.959+0000 7f0a1b53b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1410d4c0 con 0x7f0a14102180 2024-03-28T00:54:45.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.959+0000 7f0a18ad6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a14104ab0 0x7f0a1419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:45.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.959+0000 7f0a18ad6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a14104ab0 0x7f0a1419a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49038/0 (socket says 172.21.15.169:49038) 2024-03-28T00:54:45.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.959+0000 7f0a18ad6640 1 -- 172.21.15.169:0/2177062196 learned_addr learned my addr 172.21.15.169:0/2177062196 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:45.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.959+0000 7f0a192d7640 1 --2- 172.21.15.169:0/2177062196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a14102180 0x7f0a14199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:45.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.959+0000 7f0a19ad8640 1 --2- 172.21.15.169:0/2177062196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a141073e0 0x7f0a141a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:45.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.960+0000 7f0a192d7640 1 -- 172.21.15.169:0/2177062196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a141073e0 msgr2=0x7f0a141a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:45.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.960+0000 7f0a192d7640 1 --2- 172.21.15.169:0/2177062196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a141073e0 0x7f0a141a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.960+0000 7f0a192d7640 1 -- 172.21.15.169:0/2177062196 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a14104ab0 msgr2=0x7f0a1419a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:45.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.960+0000 7f0a192d7640 1 --2- 172.21.15.169:0/2177062196 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a14104ab0 0x7f0a1419a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:45.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.960+0000 7f0a192d7640 1 -- 172.21.15.169:0/2177062196 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a141a1850 con 0x7f0a14102180 2024-03-28T00:54:45.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.960+0000 7f0a19ad8640 1 --2- 172.21.15.169:0/2177062196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a141073e0 0x7f0a141a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:45.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.960+0000 7f0a18ad6640 1 --2- 172.21.15.169:0/2177062196 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a14104ab0 0x7f0a1419a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:45.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.960+0000 7f0a192d7640 1 --2- 172.21.15.169:0/2177062196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a14102180 0x7f0a14199cb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c009fd0 tx=0x7f0a0c00ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:45.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.960+0000 7f0a0a7fc640 1 -- 172.21.15.169:0/2177062196 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c004b20 con 0x7f0a14102180 2024-03-28T00:54:45.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.960+0000 7f0a0a7fc640 1 -- 172.21.15.169:0/2177062196 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a0c004cc0 con 0x7f0a14102180 2024-03-28T00:54:45.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.960+0000 7f0a1b53b640 1 -- 172.21.15.169:0/2177062196 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a141a1b40 con 0x7f0a14102180 2024-03-28T00:54:45.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.960+0000 7f0a0a7fc640 1 -- 172.21.15.169:0/2177062196 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c0107b0 con 0x7f0a14102180 2024-03-28T00:54:45.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.960+0000 7f0a1b53b640 1 -- 172.21.15.169:0/2177062196 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a141a2040 con 0x7f0a14102180 2024-03-28T00:54:45.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.961+0000 7f0a1b53b640 1 -- 172.21.15.169:0/2177062196 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a14108ed0 con 0x7f0a14102180 2024-03-28T00:54:45.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.962+0000 7f0a0a7fc640 1 -- 172.21.15.169:0/2177062196 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0a0c005020 con 0x7f0a14102180 2024-03-28T00:54:45.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.963+0000 7f0a0a7fc640 1 --2- 172.21.15.169:0/2177062196 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0a000a2800 0x7f0a000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:45.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.963+0000 7f0a18ad6640 1 --2- 172.21.15.169:0/2177062196 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0a000a2800 0x7f0a000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:45.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.963+0000 7f0a0a7fc640 1 -- 172.21.15.169:0/2177062196 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f0a0c0cf800 con 0x7f0a14102180 2024-03-28T00:54:45.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.963+0000 7f0a18ad6640 1 --2- 172.21.15.169:0/2177062196 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0a000a2800 0x7f0a000a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0a040058c0 tx=0x7f0a04005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:45.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:45.966+0000 7f0a0a7fc640 1 -- 172.21.15.169:0/2177062196 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0a0c09c660 con 0x7f0a14102180 2024-03-28T00:54:46.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.158+0000 7f0a1b53b640 1 -- 172.21.15.169:0/2177062196 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0a1402d070 con 0x7f0a14102180 2024-03-28T00:54:46.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.159+0000 7f0a0a7fc640 1 -- 172.21.15.169:0/2177062196 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) v1 ==== 81+0+600 (secure 0 0 0) 0x7f0a0c09bf60 con 0x7f0a14102180 2024-03-28T00:54:46.163 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:0/2299683991 2024-03-29T00:47:35.997770+0000 2024-03-28T00:54:46.163 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:6801/3995234407 2024-03-29T00:47:35.997770+0000 2024-03-28T00:54:46.163 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:0/3955974323 2024-03-29T00:47:10.143139+0000 2024-03-28T00:54:46.163 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:6800/3995234407 2024-03-29T00:47:35.997770+0000 2024-03-28T00:54:46.163 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:0/1567973250 2024-03-29T00:47:10.143139+0000 2024-03-28T00:54:46.163 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:0/1799473331 2024-03-29T00:47:10.143139+0000 2024-03-28T00:54:46.163 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:0/1949169969 2024-03-29T00:47:35.997770+0000 2024-03-28T00:54:46.164 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:6800/311965133 2024-03-29T00:47:10.143139+0000 2024-03-28T00:54:46.164 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:6801/311965133 2024-03-29T00:47:10.143139+0000 2024-03-28T00:54:46.164 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:0/2928378386 2024-03-29T00:47:35.997770+0000 2024-03-28T00:54:46.164 INFO:teuthology.orchestra.run.smithi169.stderr:listed 10 entries 2024-03-28T00:54:46.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.161+0000 7f0a1b53b640 1 -- 172.21.15.169:0/2177062196 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0a000a2800 msgr2=0x7f0a000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:46.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.161+0000 7f0a1b53b640 1 --2- 172.21.15.169:0/2177062196 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0a000a2800 0x7f0a000a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0a040058c0 tx=0x7f0a04005f90 comp rx=0 tx=0).stop 2024-03-28T00:54:46.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.161+0000 7f0a1b53b640 1 -- 172.21.15.169:0/2177062196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a14102180 msgr2=0x7f0a14199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:46.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.161+0000 7f0a1b53b640 1 --2- 172.21.15.169:0/2177062196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a14102180 0x7f0a14199cb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c009fd0 tx=0x7f0a0c00ef90 comp rx=0 tx=0).stop 2024-03-28T00:54:46.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.161+0000 7f0a1b53b640 1 -- 172.21.15.169:0/2177062196 shutdown_connections 2024-03-28T00:54:46.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.161+0000 7f0a1b53b640 1 --2- 172.21.15.169:0/2177062196 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0a000a2800 0x7f0a000a4cc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.161+0000 7f0a1b53b640 1 --2- 172.21.15.169:0/2177062196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a141073e0 0x7f0a141a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.161+0000 7f0a1b53b640 1 --2- 172.21.15.169:0/2177062196 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a14104ab0 0x7f0a1419a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.161+0000 7f0a1b53b640 1 --2- 172.21.15.169:0/2177062196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a14102180 0x7f0a14199cb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.161+0000 7f0a1b53b640 1 -- 172.21.15.169:0/2177062196 >> 172.21.15.169:0/2177062196 conn(0x7f0a140fbb30 msgr2=0x7f0a14101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:46.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.161+0000 7f0a1b53b640 1 -- 172.21.15.169:0/2177062196 shutdown_connections 2024-03-28T00:54:46.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.161+0000 7f0a1b53b640 1 -- 172.21.15.169:0/2177062196 wait complete. 2024-03-28T00:54:46.176 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-28T00:54:46.176 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T00:54:46.194 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T00:54:46.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.334+0000 7f1a42a85640 1 -- 172.21.15.169:0/3905530964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a3c1017b0 msgr2=0x7f1a3c101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:46.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.334+0000 7f1a42a85640 1 --2- 172.21.15.169:0/3905530964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a3c1017b0 0x7f1a3c101c30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1a28009900 tx=0x7f1a2802bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:46.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.334+0000 7f1a42a85640 1 -- 172.21.15.169:0/3905530964 shutdown_connections 2024-03-28T00:54:46.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.334+0000 7f1a42a85640 1 --2- 172.21.15.169:0/3905530964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a3c102170 0x7f1a3c108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.334+0000 7f1a42a85640 1 --2- 172.21.15.169:0/3905530964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a3c1017b0 0x7f1a3c101c30 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.334+0000 7f1a42a85640 1 --2- 172.21.15.169:0/3905530964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a3c1005c0 0x7f1a3c1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.334+0000 7f1a42a85640 1 -- 172.21.15.169:0/3905530964 >> 172.21.15.169:0/3905530964 conn(0x7f1a3c0fbb30 msgr2=0x7f1a3c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:46.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.334+0000 7f1a42a85640 1 -- 172.21.15.169:0/3905530964 shutdown_connections 2024-03-28T00:54:46.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.334+0000 7f1a42a85640 1 -- 172.21.15.169:0/3905530964 wait complete. 2024-03-28T00:54:46.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.334+0000 7f1a42a85640 1 Processor -- start 2024-03-28T00:54:46.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.334+0000 7f1a42a85640 1 -- start start 2024-03-28T00:54:46.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.334+0000 7f1a42a85640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a3c1005c0 0x7f1a3c069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:46.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.334+0000 7f1a42a85640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a3c1017b0 0x7f1a3c19eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:46.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.334+0000 7f1a42a85640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a3c102170 0x7f1a3c1a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:46.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.334+0000 7f1a42a85640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c10c240 con 0x7f1a3c102170 2024-03-28T00:54:46.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.334+0000 7f1a42a85640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c10c0c0 con 0x7f1a3c1017b0 2024-03-28T00:54:46.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.334+0000 7f1a42a85640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c10c3c0 con 0x7f1a3c1005c0 2024-03-28T00:54:46.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.334+0000 7f1a41022640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a3c102170 0x7f1a3c1a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:46.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.335+0000 7f1a3bfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a3c1017b0 0x7f1a3c19eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:46.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.335+0000 7f1a41022640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a3c102170 0x7f1a3c1a1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49064/0 (socket says 172.21.15.169:49064) 2024-03-28T00:54:46.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.335+0000 7f1a41022640 1 -- 172.21.15.169:0/401358622 learned_addr learned my addr 172.21.15.169:0/401358622 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:46.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.335+0000 7f1a40821640 1 --2- 172.21.15.169:0/401358622 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a3c1005c0 0x7f1a3c069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:46.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.335+0000 7f1a3bfff640 1 -- 172.21.15.169:0/401358622 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a3c1005c0 msgr2=0x7f1a3c069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:46.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.335+0000 7f1a3bfff640 1 --2- 172.21.15.169:0/401358622 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a3c1005c0 0x7f1a3c069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.335+0000 7f1a3bfff640 1 -- 172.21.15.169:0/401358622 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a3c102170 msgr2=0x7f1a3c1a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:46.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.335+0000 7f1a3bfff640 1 --2- 172.21.15.169:0/401358622 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a3c102170 0x7f1a3c1a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.335+0000 7f1a3bfff640 1 -- 172.21.15.169:0/401358622 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3c1a1870 con 0x7f1a3c1017b0 2024-03-28T00:54:46.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.335+0000 7f1a40821640 1 --2- 172.21.15.169:0/401358622 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a3c1005c0 0x7f1a3c069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:54:46.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.335+0000 7f1a41022640 1 --2- 172.21.15.169:0/401358622 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a3c102170 0x7f1a3c1a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:46.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.335+0000 7f1a3bfff640 1 --2- 172.21.15.169:0/401358622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a3c1017b0 0x7f1a3c19eb30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1a28009230 tx=0x7f1a2802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:46.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.335+0000 7f1a39ffb640 1 -- 172.21.15.169:0/401358622 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a28041070 con 0x7f1a3c1017b0 2024-03-28T00:54:46.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.335+0000 7f1a42a85640 1 -- 172.21.15.169:0/401358622 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a3c1a1aa0 con 0x7f1a3c1017b0 2024-03-28T00:54:46.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.335+0000 7f1a39ffb640 1 -- 172.21.15.169:0/401358622 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a28004450 con 0x7f1a3c1017b0 2024-03-28T00:54:46.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.335+0000 7f1a39ffb640 1 -- 172.21.15.169:0/401358622 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a28003df0 con 0x7f1a3c1017b0 2024-03-28T00:54:46.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.335+0000 7f1a42a85640 1 -- 172.21.15.169:0/401358622 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a3c1a1f90 con 0x7f1a3c1017b0 2024-03-28T00:54:46.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.336+0000 7f1a42a85640 1 -- 172.21.15.169:0/401358622 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a3c1078e0 con 0x7f1a3c1017b0 2024-03-28T00:54:46.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.337+0000 7f1a39ffb640 1 -- 172.21.15.169:0/401358622 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1a28044070 con 0x7f1a3c1017b0 2024-03-28T00:54:46.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.338+0000 7f1a39ffb640 1 --2- 172.21.15.169:0/401358622 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1a2c0a2830 0x7f1a2c0a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:46.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.338+0000 7f1a39ffb640 1 -- 172.21.15.169:0/401358622 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f1a280f0750 con 0x7f1a3c1017b0 2024-03-28T00:54:46.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.338+0000 7f1a40821640 1 --2- 172.21.15.169:0/401358622 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1a2c0a2830 0x7f1a2c0a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:46.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.338+0000 7f1a40821640 1 --2- 172.21.15.169:0/401358622 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1a2c0a2830 0x7f1a2c0a4cf0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c101610 tx=0x7f1a30005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:46.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.342+0000 7f1a39ffb640 1 -- 172.21.15.169:0/401358622 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1a280bd5b0 con 0x7f1a3c1017b0 2024-03-28T00:54:46.466 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:46 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4224973896' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:54:46.466 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:46 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1461319752' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:54:46.466 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:46 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2177062196' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T00:54:46.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.524+0000 7f1a42a85640 1 -- 172.21.15.169:0/401358622 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a3c0622a0 con 0x7f1a3c1017b0 2024-03-28T00:54:46.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.526+0000 7f1a39ffb640 1 -- 172.21.15.169:0/401358622 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24186 (secure 0 0 0) 0x7f1a280bceb0 con 0x7f1a3c1017b0 2024-03-28T00:54:46.530 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:46.530 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":90,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:54:38.804421+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:54:46.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.528+0000 7f1a42a85640 1 -- 172.21.15.169:0/401358622 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1a2c0a2830 msgr2=0x7f1a2c0a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:46.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.528+0000 7f1a42a85640 1 --2- 172.21.15.169:0/401358622 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1a2c0a2830 0x7f1a2c0a4cf0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c101610 tx=0x7f1a30005f90 comp rx=0 tx=0).stop 2024-03-28T00:54:46.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.528+0000 7f1a42a85640 1 -- 172.21.15.169:0/401358622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a3c1017b0 msgr2=0x7f1a3c19eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:46.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.528+0000 7f1a42a85640 1 --2- 172.21.15.169:0/401358622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a3c1017b0 0x7f1a3c19eb30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1a28009230 tx=0x7f1a2802c650 comp rx=0 tx=0).stop 2024-03-28T00:54:46.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.528+0000 7f1a42a85640 1 -- 172.21.15.169:0/401358622 shutdown_connections 2024-03-28T00:54:46.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.528+0000 7f1a42a85640 1 --2- 172.21.15.169:0/401358622 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1a2c0a2830 0x7f1a2c0a4cf0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.528+0000 7f1a42a85640 1 --2- 172.21.15.169:0/401358622 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a3c102170 0x7f1a3c1a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.528+0000 7f1a42a85640 1 --2- 172.21.15.169:0/401358622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a3c1017b0 0x7f1a3c19eb30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.528+0000 7f1a42a85640 1 --2- 172.21.15.169:0/401358622 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a3c1005c0 0x7f1a3c069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.528+0000 7f1a42a85640 1 -- 172.21.15.169:0/401358622 >> 172.21.15.169:0/401358622 conn(0x7f1a3c0fbb30 msgr2=0x7f1a3c104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:46.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.528+0000 7f1a42a85640 1 -- 172.21.15.169:0/401358622 shutdown_connections 2024-03-28T00:54:46.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.528+0000 7f1a42a85640 1 -- 172.21.15.169:0/401358622 wait complete. 2024-03-28T00:54:46.543 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T00:49:56.339525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T00:54:31.182562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T00:54:31.780429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T00:54:35.035724+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T00:54:46.544 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T00:54:46.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:46 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/4224973896' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:54:46.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:46 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1461319752' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:54:46.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:46 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2177062196' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T00:54:46.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.645+0000 7f85cd5c6640 1 -- 172.21.15.169:0/3988084989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85c8102160 msgr2=0x7f85c81089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:46.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.645+0000 7f85cd5c6640 1 --2- 172.21.15.169:0/3988084989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85c8102160 0x7f85c81089f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f85c0009ca0 tx=0x7f85c002c0f0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.645+0000 7f85cd5c6640 1 -- 172.21.15.169:0/3988084989 shutdown_connections 2024-03-28T00:54:46.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.645+0000 7f85cd5c6640 1 --2- 172.21.15.169:0/3988084989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85c8102160 0x7f85c81089f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.645+0000 7f85cd5c6640 1 --2- 172.21.15.169:0/3988084989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85c81017a0 0x7f85c8101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.645+0000 7f85cd5c6640 1 --2- 172.21.15.169:0/3988084989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85c81005b0 0x7f85c81009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.645+0000 7f85cd5c6640 1 -- 172.21.15.169:0/3988084989 >> 172.21.15.169:0/3988084989 conn(0x7f85c80fbb40 msgr2=0x7f85c80fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:46.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.645+0000 7f85cd5c6640 1 -- 172.21.15.169:0/3988084989 shutdown_connections 2024-03-28T00:54:46.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.645+0000 7f85cd5c6640 1 -- 172.21.15.169:0/3988084989 wait complete. 2024-03-28T00:54:46.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.645+0000 7f85cd5c6640 1 Processor -- start 2024-03-28T00:54:46.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.645+0000 7f85cd5c6640 1 -- start start 2024-03-28T00:54:46.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85cd5c6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85c81005b0 0x7f85c810af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85cd5c6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85c81017a0 0x7f85c8109570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85cd5c6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85c8102160 0x7f85c8109ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85cd5c6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c810d350 con 0x7f85c81017a0 2024-03-28T00:54:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85cd5c6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c810d1d0 con 0x7f85c8102160 2024-03-28T00:54:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85cd5c6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c810d4d0 con 0x7f85c81005b0 2024-03-28T00:54:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85c67fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85c81017a0 0x7f85c8109570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85c67fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85c81017a0 0x7f85c8109570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38590/0 (socket says 172.21.15.169:38590) 2024-03-28T00:54:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85c67fc640 1 -- 172.21.15.169:0/1691582704 learned_addr learned my addr 172.21.15.169:0/1691582704 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85c6ffd640 1 --2- 172.21.15.169:0/1691582704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85c81005b0 0x7f85c810af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85c77fe640 1 --2- 172.21.15.169:0/1691582704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85c8102160 0x7f85c8109ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85c67fc640 1 -- 172.21.15.169:0/1691582704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85c81005b0 msgr2=0x7f85c810af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85c67fc640 1 --2- 172.21.15.169:0/1691582704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85c81005b0 0x7f85c810af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85c67fc640 1 -- 172.21.15.169:0/1691582704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85c8102160 msgr2=0x7f85c8109ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85c67fc640 1 --2- 172.21.15.169:0/1691582704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85c8102160 0x7f85c8109ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85c67fc640 1 -- 172.21.15.169:0/1691582704 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c810a340 con 0x7f85c81017a0 2024-03-28T00:54:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85c6ffd640 1 --2- 172.21.15.169:0/1691582704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85c81005b0 0x7f85c810af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85c67fc640 1 --2- 172.21.15.169:0/1691582704 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85c81017a0 0x7f85c8109570 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f85bc00e9f0 tx=0x7f85bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85affff640 1 -- 172.21.15.169:0/1691582704 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85bc00cd80 con 0x7f85c81017a0 2024-03-28T00:54:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85cd5c6640 1 -- 172.21.15.169:0/1691582704 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c81a1ac0 con 0x7f85c81017a0 2024-03-28T00:54:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85affff640 1 -- 172.21.15.169:0/1691582704 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f85bc010910 con 0x7f85c81017a0 2024-03-28T00:54:46.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85affff640 1 -- 172.21.15.169:0/1691582704 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85bc018950 con 0x7f85c81017a0 2024-03-28T00:54:46.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.646+0000 7f85cd5c6640 1 -- 172.21.15.169:0/1691582704 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c81a2010 con 0x7f85c81017a0 2024-03-28T00:54:46.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.647+0000 7f85cd5c6640 1 -- 172.21.15.169:0/1691582704 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85c81078d0 con 0x7f85c81017a0 2024-03-28T00:54:46.656 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.649+0000 7f85affff640 1 -- 172.21.15.169:0/1691582704 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f85bc010ac0 con 0x7f85c81017a0 2024-03-28T00:54:46.656 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.649+0000 7f85affff640 1 --2- 172.21.15.169:0/1691582704 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f85b00a2800 0x7f85b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:46.656 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.649+0000 7f85c6ffd640 1 --2- 172.21.15.169:0/1691582704 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f85b00a2800 0x7f85b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:46.656 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.649+0000 7f85affff640 1 -- 172.21.15.169:0/1691582704 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f85bc0cf400 con 0x7f85c81017a0 2024-03-28T00:54:46.656 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.650+0000 7f85c6ffd640 1 --2- 172.21.15.169:0/1691582704 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f85b00a2800 0x7f85b00a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f85c8101600 tx=0x7f85b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:46.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.653+0000 7f85affff640 1 -- 172.21.15.169:0/1691582704 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f85bc09bc10 con 0x7f85c81017a0 2024-03-28T00:54:46.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:46 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/4224973896' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:54:46.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:46 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1461319752' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:54:46.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:46 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2177062196' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T00:54:46.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.829+0000 7f85cd5c6640 1 -- 172.21.15.169:0/1691582704 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f85c802d200 con 0x7f85c81017a0 2024-03-28T00:54:46.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.829+0000 7f85affff640 1 -- 172.21.15.169:0/1691582704 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f85bc09fcb0 con 0x7f85c81017a0 2024-03-28T00:54:46.834 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 1 2024-03-28T00:54:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.831+0000 7f85cd5c6640 1 -- 172.21.15.169:0/1691582704 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f85b00a2800 msgr2=0x7f85b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.831+0000 7f85cd5c6640 1 --2- 172.21.15.169:0/1691582704 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f85b00a2800 0x7f85b00a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f85c8101600 tx=0x7f85b8005f90 comp rx=0 tx=0).stop 2024-03-28T00:54:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.831+0000 7f85cd5c6640 1 -- 172.21.15.169:0/1691582704 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85c81017a0 msgr2=0x7f85c8109570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.831+0000 7f85cd5c6640 1 --2- 172.21.15.169:0/1691582704 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85c81017a0 0x7f85c8109570 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f85bc00e9f0 tx=0x7f85bc00eeb0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.831+0000 7f85cd5c6640 1 -- 172.21.15.169:0/1691582704 shutdown_connections 2024-03-28T00:54:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.831+0000 7f85cd5c6640 1 --2- 172.21.15.169:0/1691582704 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f85b00a2800 0x7f85b00a4cc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.831+0000 7f85cd5c6640 1 --2- 172.21.15.169:0/1691582704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85c8102160 0x7f85c8109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.831+0000 7f85cd5c6640 1 --2- 172.21.15.169:0/1691582704 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85c81017a0 0x7f85c8109570 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.831+0000 7f85cd5c6640 1 --2- 172.21.15.169:0/1691582704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85c81005b0 0x7f85c810af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.831+0000 7f85cd5c6640 1 -- 172.21.15.169:0/1691582704 >> 172.21.15.169:0/1691582704 conn(0x7f85c80fbb40 msgr2=0x7f85c8104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.832+0000 7f85cd5c6640 1 -- 172.21.15.169:0/1691582704 shutdown_connections 2024-03-28T00:54:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.832+0000 7f85cd5c6640 1 -- 172.21.15.169:0/1691582704 wait complete. 2024-03-28T00:54:46.846 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T00:54:46.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.948+0000 7f644bcac640 1 -- 172.21.15.169:0/1978861511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6444100f40 msgr2=0x7f64441013c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:46.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.948+0000 7f644bcac640 1 --2- 172.21.15.169:0/1978861511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6444100f40 0x7f64441013c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6440009900 tx=0x7f644002bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:46.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.948+0000 7f644bcac640 1 -- 172.21.15.169:0/1978861511 shutdown_connections 2024-03-28T00:54:46.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.948+0000 7f644bcac640 1 --2- 172.21.15.169:0/1978861511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6444101900 0x7f6444108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.948+0000 7f644bcac640 1 --2- 172.21.15.169:0/1978861511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6444100f40 0x7f64441013c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.948+0000 7f644bcac640 1 --2- 172.21.15.169:0/1978861511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64440ffd50 0x7f6444100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.948+0000 7f644bcac640 1 -- 172.21.15.169:0/1978861511 >> 172.21.15.169:0/1978861511 conn(0x7f64440fb2a0 msgr2=0x7f64440fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:46.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.948+0000 7f644bcac640 1 -- 172.21.15.169:0/1978861511 shutdown_connections 2024-03-28T00:54:46.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.948+0000 7f644bcac640 1 -- 172.21.15.169:0/1978861511 wait complete. 2024-03-28T00:54:46.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.948+0000 7f644bcac640 1 Processor -- start 2024-03-28T00:54:46.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.948+0000 7f644bcac640 1 -- start start 2024-03-28T00:54:46.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f644bcac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64440ffd50 0x7f6444197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:46.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f644bcac640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6444100f40 0x7f6444197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:46.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f644bcac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6444101900 0x7f644419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f644bcac640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644410c250 con 0x7f6444101900 2024-03-28T00:54:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f644bcac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644410c0d0 con 0x7f64440ffd50 2024-03-28T00:54:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f644bcac640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644410c3d0 con 0x7f6444100f40 2024-03-28T00:54:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f644a249640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6444101900 0x7f644419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f6449a48640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64440ffd50 0x7f6444197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f644a249640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6444101900 0x7f644419efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38606/0 (socket says 172.21.15.169:38606) 2024-03-28T00:54:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f644a249640 1 -- 172.21.15.169:0/2715333004 learned_addr learned my addr 172.21.15.169:0/2715333004 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f6449247640 1 --2- 172.21.15.169:0/2715333004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6444100f40 0x7f6444197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f644a249640 1 -- 172.21.15.169:0/2715333004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6444100f40 msgr2=0x7f6444197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f644a249640 1 --2- 172.21.15.169:0/2715333004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6444100f40 0x7f6444197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f644a249640 1 -- 172.21.15.169:0/2715333004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64440ffd50 msgr2=0x7f6444197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:46.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f644a249640 1 --2- 172.21.15.169:0/2715333004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64440ffd50 0x7f6444197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:46.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f644a249640 1 -- 172.21.15.169:0/2715333004 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644419f630 con 0x7f6444101900 2024-03-28T00:54:46.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f644a249640 1 --2- 172.21.15.169:0/2715333004 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6444101900 0x7f644419efc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f643c00cad0 tx=0x7f643c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:46.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f643affd640 1 -- 172.21.15.169:0/2715333004 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c00ae20 con 0x7f6444101900 2024-03-28T00:54:46.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f644bcac640 1 -- 172.21.15.169:0/2715333004 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f644419f920 con 0x7f6444101900 2024-03-28T00:54:46.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f643affd640 1 -- 172.21.15.169:0/2715333004 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f643c005020 con 0x7f6444101900 2024-03-28T00:54:46.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f643affd640 1 -- 172.21.15.169:0/2715333004 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c00f6b0 con 0x7f6444101900 2024-03-28T00:54:46.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.949+0000 7f644bcac640 1 -- 172.21.15.169:0/2715333004 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f644419fe90 con 0x7f6444101900 2024-03-28T00:54:46.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.950+0000 7f644bcac640 1 -- 172.21.15.169:0/2715333004 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64441a0060 con 0x7f6444101900 2024-03-28T00:54:46.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.951+0000 7f643affd640 1 -- 172.21.15.169:0/2715333004 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f643c0051d0 con 0x7f6444101900 2024-03-28T00:54:46.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.952+0000 7f643affd640 1 --2- 172.21.15.169:0/2715333004 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f64300a2800 0x7f64300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:46.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.952+0000 7f6449a48640 1 --2- 172.21.15.169:0/2715333004 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f64300a2800 0x7f64300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:46.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.952+0000 7f643affd640 1 -- 172.21.15.169:0/2715333004 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f643c0cef30 con 0x7f6444101900 2024-03-28T00:54:46.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.952+0000 7f6449a48640 1 --2- 172.21.15.169:0/2715333004 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f64300a2800 0x7f64300a4cc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6444100da0 tx=0x7f6434005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:46.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:46.956+0000 7f643affd640 1 -- 172.21.15.169:0/2715333004 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f643c014030 con 0x7f6444101900 2024-03-28T00:54:47.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.128+0000 7f644bcac640 1 -- 172.21.15.169:0/2715333004 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f64441a0230 con 0x7f6444101900 2024-03-28T00:54:47.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.129+0000 7f643affd640 1 -- 172.21.15.169:0/2715333004 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f643c09b740 con 0x7f6444101900 2024-03-28T00:54:47.133 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T00:54:47.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.132+0000 7f644bcac640 1 -- 172.21.15.169:0/2715333004 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f64300a2800 msgr2=0x7f64300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:47.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.132+0000 7f644bcac640 1 --2- 172.21.15.169:0/2715333004 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f64300a2800 0x7f64300a4cc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6444100da0 tx=0x7f6434005f90 comp rx=0 tx=0).stop 2024-03-28T00:54:47.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.132+0000 7f644bcac640 1 -- 172.21.15.169:0/2715333004 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6444101900 msgr2=0x7f644419efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:47.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.132+0000 7f644bcac640 1 --2- 172.21.15.169:0/2715333004 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6444101900 0x7f644419efc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f643c00cad0 tx=0x7f643c00cf90 comp rx=0 tx=0).stop 2024-03-28T00:54:47.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.132+0000 7f644bcac640 1 -- 172.21.15.169:0/2715333004 shutdown_connections 2024-03-28T00:54:47.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.132+0000 7f644bcac640 1 --2- 172.21.15.169:0/2715333004 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f64300a2800 0x7f64300a4cc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.132+0000 7f644bcac640 1 --2- 172.21.15.169:0/2715333004 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6444101900 0x7f644419efc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.132+0000 7f644bcac640 1 --2- 172.21.15.169:0/2715333004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6444100f40 0x7f6444197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.132+0000 7f644bcac640 1 --2- 172.21.15.169:0/2715333004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64440ffd50 0x7f6444197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.132+0000 7f644bcac640 1 -- 172.21.15.169:0/2715333004 >> 172.21.15.169:0/2715333004 conn(0x7f64440fb2a0 msgr2=0x7f6444108b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:47.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.132+0000 7f644bcac640 1 -- 172.21.15.169:0/2715333004 shutdown_connections 2024-03-28T00:54:47.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.132+0000 7f644bcac640 1 -- 172.21.15.169:0/2715333004 wait complete. 2024-03-28T00:54:47.146 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T00:54:47.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.248+0000 7fcb3700b640 1 -- 172.21.15.169:0/1335756415 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb301005a0 msgr2=0x7fcb301009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:47.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.248+0000 7fcb3700b640 1 --2- 172.21.15.169:0/1335756415 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb301005a0 0x7fcb301009c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcb28009900 tx=0x7fcb2802bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:47.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.248+0000 7fcb3700b640 1 -- 172.21.15.169:0/1335756415 shutdown_connections 2024-03-28T00:54:47.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.248+0000 7fcb3700b640 1 --2- 172.21.15.169:0/1335756415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb30102150 0x7fcb301089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.248+0000 7fcb3700b640 1 --2- 172.21.15.169:0/1335756415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb30101790 0x7fcb30101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.248+0000 7fcb3700b640 1 --2- 172.21.15.169:0/1335756415 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb301005a0 0x7fcb301009c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.248+0000 7fcb3700b640 1 -- 172.21.15.169:0/1335756415 >> 172.21.15.169:0/1335756415 conn(0x7fcb300fbb30 msgr2=0x7fcb300fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.249+0000 7fcb3700b640 1 -- 172.21.15.169:0/1335756415 shutdown_connections 2024-03-28T00:54:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.249+0000 7fcb3700b640 1 -- 172.21.15.169:0/1335756415 wait complete. 2024-03-28T00:54:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.249+0000 7fcb3700b640 1 Processor -- start 2024-03-28T00:54:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.249+0000 7fcb3700b640 1 -- start start 2024-03-28T00:54:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.249+0000 7fcb3700b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb301005a0 0x7fcb30069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.249+0000 7fcb3700b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb30101790 0x7fcb3019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.249+0000 7fcb3700b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb30102150 0x7fcb3019eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.249+0000 7fcb3700b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3010c230 con 0x7fcb30101790 2024-03-28T00:54:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.249+0000 7fcb3700b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3010c0b0 con 0x7fcb301005a0 2024-03-28T00:54:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.249+0000 7fcb3700b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3010c3b0 con 0x7fcb30102150 2024-03-28T00:54:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.249+0000 7fcb34da7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb301005a0 0x7fcb30069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.249+0000 7fcb2ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb30101790 0x7fcb3019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.249+0000 7fcb34da7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb301005a0 0x7fcb30069370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:34906/0 (socket says 172.21.15.169:34906) 2024-03-28T00:54:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.249+0000 7fcb34da7640 1 -- 172.21.15.169:0/1605159557 learned_addr learned my addr 172.21.15.169:0/1605159557 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.250+0000 7fcb355a8640 1 --2- 172.21.15.169:0/1605159557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb30102150 0x7fcb3019eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.250+0000 7fcb2ffff640 1 -- 172.21.15.169:0/1605159557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb30102150 msgr2=0x7fcb3019eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.250+0000 7fcb2ffff640 1 --2- 172.21.15.169:0/1605159557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb30102150 0x7fcb3019eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.250+0000 7fcb2ffff640 1 -- 172.21.15.169:0/1605159557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb301005a0 msgr2=0x7fcb30069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.250+0000 7fcb2ffff640 1 --2- 172.21.15.169:0/1605159557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb301005a0 0x7fcb30069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.250+0000 7fcb2ffff640 1 -- 172.21.15.169:0/1605159557 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3019f6b0 con 0x7fcb30101790 2024-03-28T00:54:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.250+0000 7fcb355a8640 1 --2- 172.21.15.169:0/1605159557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb30102150 0x7fcb3019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:54:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.250+0000 7fcb34da7640 1 --2- 172.21.15.169:0/1605159557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb301005a0 0x7fcb30069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.250+0000 7fcb2ffff640 1 --2- 172.21.15.169:0/1605159557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb30101790 0x7fcb3019c780 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcb20009fd0 tx=0x7fcb2000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.250+0000 7fcb2dffb640 1 -- 172.21.15.169:0/1605159557 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb20004b20 con 0x7fcb30101790 2024-03-28T00:54:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.250+0000 7fcb2dffb640 1 -- 172.21.15.169:0/1605159557 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcb20004cc0 con 0x7fcb30101790 2024-03-28T00:54:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.250+0000 7fcb3700b640 1 -- 172.21.15.169:0/1605159557 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3019f940 con 0x7fcb30101790 2024-03-28T00:54:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.250+0000 7fcb2dffb640 1 -- 172.21.15.169:0/1605159557 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb200107b0 con 0x7fcb30101790 2024-03-28T00:54:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.250+0000 7fcb3700b640 1 -- 172.21.15.169:0/1605159557 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb3019fe40 con 0x7fcb30101790 2024-03-28T00:54:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.251+0000 7fcb3700b640 1 -- 172.21.15.169:0/1605159557 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb301079e0 con 0x7fcb30101790 2024-03-28T00:54:47.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.252+0000 7fcb2dffb640 1 -- 172.21.15.169:0/1605159557 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fcb20005020 con 0x7fcb30101790 2024-03-28T00:54:47.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.253+0000 7fcb2dffb640 1 --2- 172.21.15.169:0/1605159557 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcb1c0a2800 0x7fcb1c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:47.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.253+0000 7fcb34da7640 1 --2- 172.21.15.169:0/1605159557 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcb1c0a2800 0x7fcb1c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:47.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.253+0000 7fcb2dffb640 1 -- 172.21.15.169:0/1605159557 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fcb200cf4a0 con 0x7fcb30101790 2024-03-28T00:54:47.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.253+0000 7fcb34da7640 1 --2- 172.21.15.169:0/1605159557 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcb1c0a2800 0x7fcb1c0a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcb2802c8e0 tx=0x7fcb28005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:47.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.258+0000 7fcb2dffb640 1 -- 172.21.15.169:0/1605159557 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcb2009c300 con 0x7fcb30101790 2024-03-28T00:54:47.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.435+0000 7fcb3700b640 1 -- 172.21.15.169:0/1605159557 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcb300622a0 con 0x7fcb30101790 2024-03-28T00:54:47.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.435+0000 7fcb2dffb640 1 -- 172.21.15.169:0/1605159557 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcb2009bc00 con 0x7fcb30101790 2024-03-28T00:54:47.439 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 64 2024-03-28T00:54:47.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.437+0000 7fcb3700b640 1 -- 172.21.15.169:0/1605159557 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcb1c0a2800 msgr2=0x7fcb1c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:47.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.437+0000 7fcb3700b640 1 --2- 172.21.15.169:0/1605159557 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcb1c0a2800 0x7fcb1c0a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcb2802c8e0 tx=0x7fcb28005b50 comp rx=0 tx=0).stop 2024-03-28T00:54:47.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.437+0000 7fcb3700b640 1 -- 172.21.15.169:0/1605159557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb30101790 msgr2=0x7fcb3019c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:47.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.437+0000 7fcb3700b640 1 --2- 172.21.15.169:0/1605159557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb30101790 0x7fcb3019c780 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcb20009fd0 tx=0x7fcb2000ef90 comp rx=0 tx=0).stop 2024-03-28T00:54:47.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.437+0000 7fcb3700b640 1 -- 172.21.15.169:0/1605159557 shutdown_connections 2024-03-28T00:54:47.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.437+0000 7fcb3700b640 1 --2- 172.21.15.169:0/1605159557 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcb1c0a2800 0x7fcb1c0a4cc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.437+0000 7fcb3700b640 1 --2- 172.21.15.169:0/1605159557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb30102150 0x7fcb3019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.437+0000 7fcb3700b640 1 --2- 172.21.15.169:0/1605159557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb30101790 0x7fcb3019c780 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.437+0000 7fcb3700b640 1 --2- 172.21.15.169:0/1605159557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb301005a0 0x7fcb30069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.438+0000 7fcb3700b640 1 -- 172.21.15.169:0/1605159557 >> 172.21.15.169:0/1605159557 conn(0x7fcb300fbb30 msgr2=0x7fcb30104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:47.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.438+0000 7fcb3700b640 1 -- 172.21.15.169:0/1605159557 shutdown_connections 2024-03-28T00:54:47.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.438+0000 7fcb3700b640 1 -- 172.21.15.169:0/1605159557 wait complete. 2024-03-28T00:54:47.451 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:47 smithi169 ceph-mon[32274]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:47.451 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:47 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/401358622' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:47.451 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:47 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1691582704' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:54:47.451 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:47 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2715333004' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:54:47.452 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T00:54:47.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.554+0000 7f18d2d13640 1 -- 172.21.15.169:0/1842937048 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18cc101790 msgr2=0x7f18cc101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:47.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.554+0000 7f18d2d13640 1 --2- 172.21.15.169:0/1842937048 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18cc101790 0x7f18cc101c10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f18b8009900 tx=0x7f18b802bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:47.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.554+0000 7f18d2d13640 1 -- 172.21.15.169:0/1842937048 shutdown_connections 2024-03-28T00:54:47.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.554+0000 7f18d2d13640 1 --2- 172.21.15.169:0/1842937048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18cc102150 0x7f18cc1089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.554+0000 7f18d2d13640 1 --2- 172.21.15.169:0/1842937048 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18cc101790 0x7f18cc101c10 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.554+0000 7f18d2d13640 1 --2- 172.21.15.169:0/1842937048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18cc1005a0 0x7f18cc1009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.554+0000 7f18d2d13640 1 -- 172.21.15.169:0/1842937048 >> 172.21.15.169:0/1842937048 conn(0x7f18cc0fbb30 msgr2=0x7f18cc0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:47.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.554+0000 7f18d2d13640 1 -- 172.21.15.169:0/1842937048 shutdown_connections 2024-03-28T00:54:47.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.554+0000 7f18d2d13640 1 -- 172.21.15.169:0/1842937048 wait complete. 2024-03-28T00:54:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.554+0000 7f18d2d13640 1 Processor -- start 2024-03-28T00:54:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.554+0000 7f18d2d13640 1 -- start start 2024-03-28T00:54:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18d2d13640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18cc1005a0 0x7f18cc06a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18d2d13640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18cc101790 0x7f18cc19c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18d2d13640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18cc102150 0x7f18cc19ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18d2d13640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18cc10c230 con 0x7f18cc101790 2024-03-28T00:54:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18d2d13640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18cc10c0b0 con 0x7f18cc1005a0 2024-03-28T00:54:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18d2d13640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18cc10c3b0 con 0x7f18cc102150 2024-03-28T00:54:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18cbfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18cc101790 0x7f18cc19c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18cbfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18cc101790 0x7f18cc19c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38652/0 (socket says 172.21.15.169:38652) 2024-03-28T00:54:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18d0aaf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18cc1005a0 0x7f18cc06a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18cbfff640 1 -- 172.21.15.169:0/1299493008 learned_addr learned my addr 172.21.15.169:0/1299493008 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18d12b0640 1 --2- 172.21.15.169:0/1299493008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18cc102150 0x7f18cc19ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18cbfff640 1 -- 172.21.15.169:0/1299493008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18cc102150 msgr2=0x7f18cc19ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18cbfff640 1 --2- 172.21.15.169:0/1299493008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18cc102150 0x7f18cc19ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18cbfff640 1 -- 172.21.15.169:0/1299493008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18cc1005a0 msgr2=0x7f18cc06a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18cbfff640 1 --2- 172.21.15.169:0/1299493008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18cc1005a0 0x7f18cc06a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18cbfff640 1 -- 172.21.15.169:0/1299493008 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18cc19f610 con 0x7f18cc101790 2024-03-28T00:54:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18d0aaf640 1 --2- 172.21.15.169:0/1299493008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18cc1005a0 0x7f18cc06a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18cbfff640 1 --2- 172.21.15.169:0/1299493008 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18cc101790 0x7f18cc19c6e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f18b8009230 tx=0x7f18b8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18c9ffb640 1 -- 172.21.15.169:0/1299493008 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b8041070 con 0x7f18cc101790 2024-03-28T00:54:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18c9ffb640 1 -- 172.21.15.169:0/1299493008 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f18b80044d0 con 0x7f18cc101790 2024-03-28T00:54:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18d2d13640 1 -- 172.21.15.169:0/1299493008 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18cc19f840 con 0x7f18cc101790 2024-03-28T00:54:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.555+0000 7f18c9ffb640 1 -- 172.21.15.169:0/1299493008 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b8034440 con 0x7f18cc101790 2024-03-28T00:54:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.556+0000 7f18d2d13640 1 -- 172.21.15.169:0/1299493008 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18cc19fd30 con 0x7f18cc101790 2024-03-28T00:54:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.557+0000 7f18d2d13640 1 -- 172.21.15.169:0/1299493008 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18cc107940 con 0x7f18cc101790 2024-03-28T00:54:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.558+0000 7f18c9ffb640 1 -- 172.21.15.169:0/1299493008 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f18b8044070 con 0x7f18cc101790 2024-03-28T00:54:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.558+0000 7f18c9ffb640 1 --2- 172.21.15.169:0/1299493008 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f18bc0a2800 0x7f18bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.558+0000 7f18d0aaf640 1 --2- 172.21.15.169:0/1299493008 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f18bc0a2800 0x7f18bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.558+0000 7f18c9ffb640 1 -- 172.21.15.169:0/1299493008 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f18b80f0c00 con 0x7f18cc101790 2024-03-28T00:54:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.559+0000 7f18d0aaf640 1 --2- 172.21.15.169:0/1299493008 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f18bc0a2800 0x7f18bc0a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f18cc1015f0 tx=0x7f18c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:47.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.562+0000 7f18c9ffb640 1 -- 172.21.15.169:0/1299493008 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f18b80bd410 con 0x7f18cc101790 2024-03-28T00:54:47.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:47 smithi174 ceph-mon[36060]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:47.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:47 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/401358622' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:47.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:47 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1691582704' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:54:47.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:47 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2715333004' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:54:47.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:47 smithi177 ceph-mon[36206]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T00:54:47.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:47 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/401358622' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:47.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:47 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1691582704' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:54:47.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:47 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2715333004' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:54:47.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.737+0000 7f18d2d13640 1 -- 172.21.15.169:0/1299493008 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f18cc02d200 con 0x7f18cc101790 2024-03-28T00:54:47.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.737+0000 7f18c9ffb640 1 -- 172.21.15.169:0/1299493008 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) v1 ==== 103+0+11 (secure 0 0 0) 0x7f18b80c14b0 con 0x7f18cc101790 2024-03-28T00:54:47.742 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T00:54:47.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.740+0000 7f18d2d13640 1 -- 172.21.15.169:0/1299493008 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f18bc0a2800 msgr2=0x7f18bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:47.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.740+0000 7f18d2d13640 1 --2- 172.21.15.169:0/1299493008 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f18bc0a2800 0x7f18bc0a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f18cc1015f0 tx=0x7f18c0005f90 comp rx=0 tx=0).stop 2024-03-28T00:54:47.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.740+0000 7f18d2d13640 1 -- 172.21.15.169:0/1299493008 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18cc101790 msgr2=0x7f18cc19c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:47.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.740+0000 7f18d2d13640 1 --2- 172.21.15.169:0/1299493008 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18cc101790 0x7f18cc19c6e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f18b8009230 tx=0x7f18b8005d50 comp rx=0 tx=0).stop 2024-03-28T00:54:47.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.740+0000 7f18d2d13640 1 -- 172.21.15.169:0/1299493008 shutdown_connections 2024-03-28T00:54:47.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.740+0000 7f18d2d13640 1 --2- 172.21.15.169:0/1299493008 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f18bc0a2800 0x7f18bc0a4cc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.740+0000 7f18d2d13640 1 --2- 172.21.15.169:0/1299493008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18cc102150 0x7f18cc19ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.740+0000 7f18d2d13640 1 --2- 172.21.15.169:0/1299493008 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18cc101790 0x7f18cc19c6e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.740+0000 7f18d2d13640 1 --2- 172.21.15.169:0/1299493008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18cc1005a0 0x7f18cc06a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.740+0000 7f18d2d13640 1 -- 172.21.15.169:0/1299493008 >> 172.21.15.169:0/1299493008 conn(0x7f18cc0fbb30 msgr2=0x7f18cc104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:47.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.740+0000 7f18d2d13640 1 -- 172.21.15.169:0/1299493008 shutdown_connections 2024-03-28T00:54:47.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.740+0000 7f18d2d13640 1 -- 172.21.15.169:0/1299493008 wait complete. 2024-03-28T00:54:47.755 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T00:54:47.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.854+0000 7f99a4737640 1 -- 172.21.15.169:0/341507211 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f999c1016c0 msgr2=0x7f999c101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:47.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.854+0000 7f99a4737640 1 --2- 172.21.15.169:0/341507211 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f999c1016c0 0x7f999c101ae0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f998c009900 tx=0x7f998c02bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:47.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.854+0000 7f99a4737640 1 -- 172.21.15.169:0/341507211 shutdown_connections 2024-03-28T00:54:47.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.854+0000 7f99a4737640 1 --2- 172.21.15.169:0/341507211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f999c103270 0x7f999c109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.854+0000 7f99a4737640 1 --2- 172.21.15.169:0/341507211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f999c1028b0 0x7f999c102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.854+0000 7f99a4737640 1 --2- 172.21.15.169:0/341507211 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f999c1016c0 0x7f999c101ae0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.854+0000 7f99a4737640 1 -- 172.21.15.169:0/341507211 >> 172.21.15.169:0/341507211 conn(0x7f999c0fcc50 msgr2=0x7f999c0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:47.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.854+0000 7f99a4737640 1 -- 172.21.15.169:0/341507211 shutdown_connections 2024-03-28T00:54:47.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.854+0000 7f99a4737640 1 -- 172.21.15.169:0/341507211 wait complete. 2024-03-28T00:54:47.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.854+0000 7f99a4737640 1 Processor -- start 2024-03-28T00:54:47.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.854+0000 7f99a4737640 1 -- start start 2024-03-28T00:54:47.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99a4737640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f999c1016c0 0x7f999c075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:47.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99a4737640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f999c1028b0 0x7f999c073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:47.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99a4737640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f999c103270 0x7f999c073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:47.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99a4737640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999c10d310 con 0x7f999c1028b0 2024-03-28T00:54:47.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99a4737640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999c10d190 con 0x7f999c1016c0 2024-03-28T00:54:47.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99a4737640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999c10d490 con 0x7f999c103270 2024-03-28T00:54:47.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99a1cd2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f999c1028b0 0x7f999c073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:47.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99a1cd2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f999c1028b0 0x7f999c073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38680/0 (socket says 172.21.15.169:38680) 2024-03-28T00:54:47.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99a1cd2640 1 -- 172.21.15.169:0/2956468685 learned_addr learned my addr 172.21.15.169:0/2956468685 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:47.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99a24d3640 1 --2- 172.21.15.169:0/2956468685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f999c1016c0 0x7f999c075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:47.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99a2cd4640 1 --2- 172.21.15.169:0/2956468685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f999c103270 0x7f999c073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:47.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99a1cd2640 1 -- 172.21.15.169:0/2956468685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f999c103270 msgr2=0x7f999c073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:47.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99a1cd2640 1 --2- 172.21.15.169:0/2956468685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f999c103270 0x7f999c073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99a1cd2640 1 -- 172.21.15.169:0/2956468685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f999c1016c0 msgr2=0x7f999c075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:47.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99a1cd2640 1 --2- 172.21.15.169:0/2956468685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f999c1016c0 0x7f999c075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:47.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99a1cd2640 1 -- 172.21.15.169:0/2956468685 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999c074860 con 0x7f999c1028b0 2024-03-28T00:54:47.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99a24d3640 1 --2- 172.21.15.169:0/2956468685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f999c1016c0 0x7f999c075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:47.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99a1cd2640 1 --2- 172.21.15.169:0/2956468685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f999c1028b0 0x7f999c073a90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f999800d950 tx=0x7f999800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:47.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99937fe640 1 -- 172.21.15.169:0/2956468685 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9998017030 con 0x7f999c1028b0 2024-03-28T00:54:47.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99937fe640 1 -- 172.21.15.169:0/2956468685 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9998004020 con 0x7f999c1028b0 2024-03-28T00:54:47.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99a4737640 1 -- 172.21.15.169:0/2956468685 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f999c1a1b00 con 0x7f999c1028b0 2024-03-28T00:54:47.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.855+0000 7f99937fe640 1 -- 172.21.15.169:0/2956468685 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99980047a0 con 0x7f999c1028b0 2024-03-28T00:54:47.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.856+0000 7f99a4737640 1 -- 172.21.15.169:0/2956468685 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f999c1a2050 con 0x7f999c1028b0 2024-03-28T00:54:47.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.857+0000 7f99a4737640 1 -- 172.21.15.169:0/2956468685 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f999c108ab0 con 0x7f999c1028b0 2024-03-28T00:54:47.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.858+0000 7f99937fe640 1 -- 172.21.15.169:0/2956468685 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f99980041d0 con 0x7f999c1028b0 2024-03-28T00:54:47.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.858+0000 7f99937fe640 1 --2- 172.21.15.169:0/2956468685 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f99880a2800 0x7f99880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:47.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.858+0000 7f99a24d3640 1 --2- 172.21.15.169:0/2956468685 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f99880a2800 0x7f99880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:47.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.858+0000 7f99937fe640 1 -- 172.21.15.169:0/2956468685 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f99980ce750 con 0x7f999c1028b0 2024-03-28T00:54:47.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.859+0000 7f99a24d3640 1 --2- 172.21.15.169:0/2956468685 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f99880a2800 0x7f99880a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f998c000c00 tx=0x7f998c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:47.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:47.863+0000 7f99937fe640 1 -- 172.21.15.169:0/2956468685 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f999809b5b0 con 0x7f999c1028b0 2024-03-28T00:54:48.053 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.049+0000 7f99a4737640 1 -- 172.21.15.169:0/2956468685 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f999c0622d0 con 0x7f999c1028b0 2024-03-28T00:54:48.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.051+0000 7f99937fe640 1 -- 172.21.15.169:0/2956468685 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) v1 ==== 81+0+600 (secure 0 0 0) 0x7f999809aeb0 con 0x7f999c1028b0 2024-03-28T00:54:48.054 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:0/2299683991 2024-03-29T00:47:35.997770+0000 2024-03-28T00:54:48.054 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:6801/3995234407 2024-03-29T00:47:35.997770+0000 2024-03-28T00:54:48.054 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:0/3955974323 2024-03-29T00:47:10.143139+0000 2024-03-28T00:54:48.054 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:6800/3995234407 2024-03-29T00:47:35.997770+0000 2024-03-28T00:54:48.054 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:0/1567973250 2024-03-29T00:47:10.143139+0000 2024-03-28T00:54:48.054 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:0/1799473331 2024-03-29T00:47:10.143139+0000 2024-03-28T00:54:48.054 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:0/1949169969 2024-03-29T00:47:35.997770+0000 2024-03-28T00:54:48.055 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:6800/311965133 2024-03-29T00:47:10.143139+0000 2024-03-28T00:54:48.055 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:6801/311965133 2024-03-29T00:47:10.143139+0000 2024-03-28T00:54:48.055 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:0/2928378386 2024-03-29T00:47:35.997770+0000 2024-03-28T00:54:48.055 INFO:teuthology.orchestra.run.smithi169.stderr:listed 10 entries 2024-03-28T00:54:48.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.053+0000 7f99a4737640 1 -- 172.21.15.169:0/2956468685 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f99880a2800 msgr2=0x7f99880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:48.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.053+0000 7f99a4737640 1 --2- 172.21.15.169:0/2956468685 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f99880a2800 0x7f99880a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f998c000c00 tx=0x7f998c005fd0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.053+0000 7f99a4737640 1 -- 172.21.15.169:0/2956468685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f999c1028b0 msgr2=0x7f999c073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:48.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.053+0000 7f99a4737640 1 --2- 172.21.15.169:0/2956468685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f999c1028b0 0x7f999c073a90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f999800d950 tx=0x7f999800de10 comp rx=0 tx=0).stop 2024-03-28T00:54:48.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.053+0000 7f99a4737640 1 -- 172.21.15.169:0/2956468685 shutdown_connections 2024-03-28T00:54:48.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.053+0000 7f99a4737640 1 --2- 172.21.15.169:0/2956468685 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f99880a2800 0x7f99880a4cc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.053+0000 7f99a4737640 1 --2- 172.21.15.169:0/2956468685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f999c103270 0x7f999c073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.053+0000 7f99a4737640 1 --2- 172.21.15.169:0/2956468685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f999c1028b0 0x7f999c073a90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.053+0000 7f99a4737640 1 --2- 172.21.15.169:0/2956468685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f999c1016c0 0x7f999c075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.053+0000 7f99a4737640 1 -- 172.21.15.169:0/2956468685 >> 172.21.15.169:0/2956468685 conn(0x7f999c0fcc50 msgr2=0x7f999c105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:48.057 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.053+0000 7f99a4737640 1 -- 172.21.15.169:0/2956468685 shutdown_connections 2024-03-28T00:54:48.057 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.053+0000 7f99a4737640 1 -- 172.21.15.169:0/2956468685 wait complete. 2024-03-28T00:54:48.067 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi169.front.sepia.ceph.com... 2024-03-28T00:54:48.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:54:48.067 DEBUG:teuthology.orchestra.run.smithi169:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-28T00:54:48.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi169.stderr:ceph-fuse[74331]: fuse finished with error 0 and tester_r 0 2024-03-28T00:54:48.095 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T00:54:48.173 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-28T00:54:48.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:54:48.174 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-28T00:54:48.193 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T00:54:48.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.334+0000 7f6371390640 1 -- 172.21.15.169:0/998464685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f636c1016c0 msgr2=0x7f636c101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:48.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.334+0000 7f6371390640 1 --2- 172.21.15.169:0/998464685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f636c1016c0 0x7f636c101ae0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6364009ca0 tx=0x7f636402c0f0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.335+0000 7f6371390640 1 -- 172.21.15.169:0/998464685 shutdown_connections 2024-03-28T00:54:48.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.335+0000 7f6371390640 1 --2- 172.21.15.169:0/998464685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f636c103270 0x7f636c109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.335+0000 7f6371390640 1 --2- 172.21.15.169:0/998464685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f636c1028b0 0x7f636c102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.335+0000 7f6371390640 1 --2- 172.21.15.169:0/998464685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f636c1016c0 0x7f636c101ae0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.335+0000 7f6371390640 1 -- 172.21.15.169:0/998464685 >> 172.21.15.169:0/998464685 conn(0x7f636c0fcc50 msgr2=0x7f636c0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:48.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.335+0000 7f6371390640 1 -- 172.21.15.169:0/998464685 shutdown_connections 2024-03-28T00:54:48.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.335+0000 7f6371390640 1 -- 172.21.15.169:0/998464685 wait complete. 2024-03-28T00:54:48.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.335+0000 7f6371390640 1 Processor -- start 2024-03-28T00:54:48.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.335+0000 7f6371390640 1 -- start start 2024-03-28T00:54:48.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.335+0000 7f6371390640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f636c1016c0 0x7f636c1979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:48.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.335+0000 7f6371390640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f636c1028b0 0x7f636c197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:48.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.335+0000 7f6371390640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f636c103270 0x7f636c19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:48.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.335+0000 7f6371390640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636c10d340 con 0x7f636c103270 2024-03-28T00:54:48.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.335+0000 7f6371390640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636c10d1c0 con 0x7f636c1016c0 2024-03-28T00:54:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.335+0000 7f6371390640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636c10d4c0 con 0x7f636c1028b0 2024-03-28T00:54:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.336+0000 7f636b7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f636c103270 0x7f636c19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.336+0000 7f636b7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f636c103270 0x7f636c19ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38704/0 (socket says 172.21.15.169:38704) 2024-03-28T00:54:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.336+0000 7f636b7fe640 1 -- 172.21.15.169:0/3073720943 learned_addr learned my addr 172.21.15.169:0/3073720943 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.336+0000 7f636affd640 1 --2- 172.21.15.169:0/3073720943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f636c1016c0 0x7f636c1979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.336+0000 7f636a7fc640 1 --2- 172.21.15.169:0/3073720943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f636c1028b0 0x7f636c197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.336+0000 7f636b7fe640 1 -- 172.21.15.169:0/3073720943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f636c1028b0 msgr2=0x7f636c197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.336+0000 7f636b7fe640 1 --2- 172.21.15.169:0/3073720943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f636c1028b0 0x7f636c197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.336+0000 7f636b7fe640 1 -- 172.21.15.169:0/3073720943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f636c1016c0 msgr2=0x7f636c1979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.336+0000 7f636b7fe640 1 --2- 172.21.15.169:0/3073720943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f636c1016c0 0x7f636c1979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.336+0000 7f636b7fe640 1 -- 172.21.15.169:0/3073720943 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636c19f5d0 con 0x7f636c103270 2024-03-28T00:54:48.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.336+0000 7f636affd640 1 --2- 172.21.15.169:0/3073720943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f636c1016c0 0x7f636c1979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:48.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.336+0000 7f636b7fe640 1 --2- 172.21.15.169:0/3073720943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f636c103270 0x7f636c19ef60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f635c00d750 tx=0x7f635c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:48.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.336+0000 7f6363fff640 1 -- 172.21.15.169:0/3073720943 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c017070 con 0x7f636c103270 2024-03-28T00:54:48.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.336+0000 7f6371390640 1 -- 172.21.15.169:0/3073720943 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f636c19f8c0 con 0x7f636c103270 2024-03-28T00:54:48.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.336+0000 7f6363fff640 1 -- 172.21.15.169:0/3073720943 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f635c005500 con 0x7f636c103270 2024-03-28T00:54:48.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.336+0000 7f6363fff640 1 -- 172.21.15.169:0/3073720943 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c004d90 con 0x7f636c103270 2024-03-28T00:54:48.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.336+0000 7f6371390640 1 -- 172.21.15.169:0/3073720943 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f636c19fe10 con 0x7f636c103270 2024-03-28T00:54:48.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.337+0000 7f6371390640 1 -- 172.21.15.169:0/3073720943 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f636c1089e0 con 0x7f636c103270 2024-03-28T00:54:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.338+0000 7f6363fff640 1 -- 172.21.15.169:0/3073720943 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f635c004f30 con 0x7f636c103270 2024-03-28T00:54:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.339+0000 7f6363fff640 1 --2- 172.21.15.169:0/3073720943 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f63580a2800 0x7f63580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:48.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.339+0000 7f636affd640 1 --2- 172.21.15.169:0/3073720943 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f63580a2800 0x7f63580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:48.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.339+0000 7f6363fff640 1 -- 172.21.15.169:0/3073720943 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f635c0ce640 con 0x7f636c103270 2024-03-28T00:54:48.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.339+0000 7f636affd640 1 --2- 172.21.15.169:0/3073720943 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f63580a2800 0x7f63580a4cc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f636402c630 tx=0x7f6364009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:48.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.343+0000 7f6363fff640 1 -- 172.21.15.169:0/3073720943 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f635c09b4a0 con 0x7f636c103270 2024-03-28T00:54:48.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.521+0000 7f6371390640 1 -- 172.21.15.169:0/3073720943 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f636c0622a0 con 0x7f636c103270 2024-03-28T00:54:48.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.523+0000 7f6363fff640 1 -- 172.21.15.169:0/3073720943 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24186 (secure 0 0 0) 0x7f635c09ada0 con 0x7f636c103270 2024-03-28T00:54:48.526 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:48.526 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":90,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:54:38.804421+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:54:48.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.525+0000 7f6371390640 1 -- 172.21.15.169:0/3073720943 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f63580a2800 msgr2=0x7f63580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:48.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.525+0000 7f6371390640 1 --2- 172.21.15.169:0/3073720943 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f63580a2800 0x7f63580a4cc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f636402c630 tx=0x7f6364009360 comp rx=0 tx=0).stop 2024-03-28T00:54:48.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.525+0000 7f6371390640 1 -- 172.21.15.169:0/3073720943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f636c103270 msgr2=0x7f636c19ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:48.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.525+0000 7f6371390640 1 --2- 172.21.15.169:0/3073720943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f636c103270 0x7f636c19ef60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f635c00d750 tx=0x7f635c00dc10 comp rx=0 tx=0).stop 2024-03-28T00:54:48.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.525+0000 7f6371390640 1 -- 172.21.15.169:0/3073720943 shutdown_connections 2024-03-28T00:54:48.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.525+0000 7f6371390640 1 --2- 172.21.15.169:0/3073720943 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f63580a2800 0x7f63580a4cc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.525+0000 7f6371390640 1 --2- 172.21.15.169:0/3073720943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f636c103270 0x7f636c19ef60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.525+0000 7f6371390640 1 --2- 172.21.15.169:0/3073720943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f636c1028b0 0x7f636c197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.525+0000 7f6371390640 1 --2- 172.21.15.169:0/3073720943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f636c1016c0 0x7f636c1979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.525+0000 7f6371390640 1 -- 172.21.15.169:0/3073720943 >> 172.21.15.169:0/3073720943 conn(0x7f636c0fcc50 msgr2=0x7f636c105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:48.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.525+0000 7f6371390640 1 -- 172.21.15.169:0/3073720943 shutdown_connections 2024-03-28T00:54:48.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.525+0000 7f6371390640 1 -- 172.21.15.169:0/3073720943 wait complete. 2024-03-28T00:54:48.538 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:48 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1605159557' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:54:48.539 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:48 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1299493008' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:54:48.539 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:48 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2956468685' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T00:54:48.540 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T00:54:48.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:48 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1605159557' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:54:48.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:48 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1299493008' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:54:48.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:48 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2956468685' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T00:54:48.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.639+0000 7fc60c776640 1 -- 172.21.15.169:0/249402904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc604100120 msgr2=0x7fc604109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:48.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.639+0000 7fc60c776640 1 --2- 172.21.15.169:0/249402904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc604100120 0x7fc604109cd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc009ca0 tx=0x7fc5fc02c0f0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.639+0000 7fc60c776640 1 -- 172.21.15.169:0/249402904 shutdown_connections 2024-03-28T00:54:48.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.639+0000 7fc60c776640 1 --2- 172.21.15.169:0/249402904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc604100120 0x7fc604109cd0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.639+0000 7fc60c776640 1 --2- 172.21.15.169:0/249402904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6040ff760 0x7fc6040ffbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.639+0000 7fc60c776640 1 --2- 172.21.15.169:0/249402904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc604102cf0 0x7fc604103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.639+0000 7fc60c776640 1 -- 172.21.15.169:0/249402904 >> 172.21.15.169:0/249402904 conn(0x7fc6040fb2c0 msgr2=0x7fc6040fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:48.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.639+0000 7fc60c776640 1 -- 172.21.15.169:0/249402904 shutdown_connections 2024-03-28T00:54:48.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.639+0000 7fc60c776640 1 -- 172.21.15.169:0/249402904 wait complete. 2024-03-28T00:54:48.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc60c776640 1 Processor -- start 2024-03-28T00:54:48.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc60c776640 1 -- start start 2024-03-28T00:54:48.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc60c776640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6040ff760 0x7fc6041978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:48.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc60c776640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc604100120 0x7fc604197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:48.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc60c776640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc604102cf0 0x7fc60419ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:48.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc60c776640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60410d320 con 0x7fc6040ff760 2024-03-28T00:54:48.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc60c776640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60410d1a0 con 0x7fc604102cf0 2024-03-28T00:54:48.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc60c776640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60410d4a0 con 0x7fc604100120 2024-03-28T00:54:48.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc60a512640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6040ff760 0x7fc6041978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:48.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc60a512640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6040ff760 0x7fc6041978a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38718/0 (socket says 172.21.15.169:38718) 2024-03-28T00:54:48.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc60a512640 1 -- 172.21.15.169:0/3936013352 learned_addr learned my addr 172.21.15.169:0/3936013352 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:48.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc60ad13640 1 --2- 172.21.15.169:0/3936013352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc604102cf0 0x7fc60419ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:48.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc609d11640 1 --2- 172.21.15.169:0/3936013352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc604100120 0x7fc604197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:48.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc60a512640 1 -- 172.21.15.169:0/3936013352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc604100120 msgr2=0x7fc604197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:48.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc60a512640 1 --2- 172.21.15.169:0/3936013352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc604100120 0x7fc604197de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc60a512640 1 -- 172.21.15.169:0/3936013352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc604102cf0 msgr2=0x7fc60419ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:48.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc60a512640 1 --2- 172.21.15.169:0/3936013352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc604102cf0 0x7fc60419ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc60a512640 1 -- 172.21.15.169:0/3936013352 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc60419f4c0 con 0x7fc6040ff760 2024-03-28T00:54:48.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc609d11640 1 --2- 172.21.15.169:0/3936013352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc604100120 0x7fc604197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:54:48.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc60ad13640 1 --2- 172.21.15.169:0/3936013352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc604102cf0 0x7fc60419ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:54:48.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc60a512640 1 --2- 172.21.15.169:0/3936013352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6040ff760 0x7fc6041978a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc60000e9c0 tx=0x7fc60000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:48.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc5fb7fe640 1 -- 172.21.15.169:0/3936013352 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60000cde0 con 0x7fc6040ff760 2024-03-28T00:54:48.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc60c776640 1 -- 172.21.15.169:0/3936013352 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc60419f7b0 con 0x7fc6040ff760 2024-03-28T00:54:48.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc5fb7fe640 1 -- 172.21.15.169:0/3936013352 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc600010910 con 0x7fc6040ff760 2024-03-28T00:54:48.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc5fb7fe640 1 -- 172.21.15.169:0/3936013352 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc600018a10 con 0x7fc6040ff760 2024-03-28T00:54:48.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.640+0000 7fc60c776640 1 -- 172.21.15.169:0/3936013352 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc60419fd50 con 0x7fc6040ff760 2024-03-28T00:54:48.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.641+0000 7fc60c776640 1 -- 172.21.15.169:0/3936013352 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc604108ae0 con 0x7fc6040ff760 2024-03-28T00:54:48.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.642+0000 7fc5fb7fe640 1 -- 172.21.15.169:0/3936013352 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc60001f020 con 0x7fc6040ff760 2024-03-28T00:54:48.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.643+0000 7fc5fb7fe640 1 --2- 172.21.15.169:0/3936013352 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc5f00a2800 0x7fc5f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:48.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.643+0000 7fc609d11640 1 --2- 172.21.15.169:0/3936013352 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc5f00a2800 0x7fc5f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:48.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.643+0000 7fc5fb7fe640 1 -- 172.21.15.169:0/3936013352 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc6000cf5d0 con 0x7fc6040ff760 2024-03-28T00:54:48.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.644+0000 7fc609d11640 1 --2- 172.21.15.169:0/3936013352 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc5f00a2800 0x7fc5f00a4cc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc604198db0 tx=0x7fc5f400a5c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:48.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.647+0000 7fc5fb7fe640 1 -- 172.21.15.169:0/3936013352 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc60009c430 con 0x7fc6040ff760 2024-03-28T00:54:48.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:48 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1605159557' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:54:48.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:48 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1299493008' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:54:48.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:48 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2956468685' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T00:54:48.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.826+0000 7fc60c776640 1 -- 172.21.15.169:0/3936013352 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6040622a0 con 0x7fc6040ff760 2024-03-28T00:54:48.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.828+0000 7fc5fb7fe640 1 -- 172.21.15.169:0/3936013352 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24186 (secure 0 0 0) 0x7fc600020e40 con 0x7fc6040ff760 2024-03-28T00:54:48.831 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:48.831 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":90,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:54:38.804421+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:54:48.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.830+0000 7fc60c776640 1 -- 172.21.15.169:0/3936013352 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc5f00a2800 msgr2=0x7fc5f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:48.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.830+0000 7fc60c776640 1 --2- 172.21.15.169:0/3936013352 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc5f00a2800 0x7fc5f00a4cc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc604198db0 tx=0x7fc5f400a5c0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.830+0000 7fc60c776640 1 -- 172.21.15.169:0/3936013352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6040ff760 msgr2=0x7fc6041978a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:48.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.830+0000 7fc60c776640 1 --2- 172.21.15.169:0/3936013352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6040ff760 0x7fc6041978a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc60000e9c0 tx=0x7fc60000ee80 comp rx=0 tx=0).stop 2024-03-28T00:54:48.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.830+0000 7fc60c776640 1 -- 172.21.15.169:0/3936013352 shutdown_connections 2024-03-28T00:54:48.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.830+0000 7fc60c776640 1 --2- 172.21.15.169:0/3936013352 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc5f00a2800 0x7fc5f00a4cc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.830+0000 7fc60c776640 1 --2- 172.21.15.169:0/3936013352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc604102cf0 0x7fc60419ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.830+0000 7fc60c776640 1 --2- 172.21.15.169:0/3936013352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc604100120 0x7fc604197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.830+0000 7fc60c776640 1 --2- 172.21.15.169:0/3936013352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6040ff760 0x7fc6041978a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.830+0000 7fc60c776640 1 -- 172.21.15.169:0/3936013352 >> 172.21.15.169:0/3936013352 conn(0x7fc6040fb2c0 msgr2=0x7fc604105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:48.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.830+0000 7fc60c776640 1 -- 172.21.15.169:0/3936013352 shutdown_connections 2024-03-28T00:54:48.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.830+0000 7fc60c776640 1 -- 172.21.15.169:0/3936013352 wait complete. 2024-03-28T00:54:48.844 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T00:54:48.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.946+0000 7fed9aa27640 1 -- 172.21.15.169:0/1282681038 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed94102150 msgr2=0x7fed941089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:48.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.946+0000 7fed9aa27640 1 --2- 172.21.15.169:0/1282681038 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed94102150 0x7fed941089e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fed8c009ca0 tx=0x7fed8c02c0f0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.946+0000 7fed9aa27640 1 -- 172.21.15.169:0/1282681038 shutdown_connections 2024-03-28T00:54:48.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.946+0000 7fed9aa27640 1 --2- 172.21.15.169:0/1282681038 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed94102150 0x7fed941089e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.946+0000 7fed9aa27640 1 --2- 172.21.15.169:0/1282681038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed94101790 0x7fed94101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.946+0000 7fed9aa27640 1 --2- 172.21.15.169:0/1282681038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed941005a0 0x7fed941009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.946+0000 7fed9aa27640 1 -- 172.21.15.169:0/1282681038 >> 172.21.15.169:0/1282681038 conn(0x7fed940fbb30 msgr2=0x7fed940fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:48.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.946+0000 7fed9aa27640 1 -- 172.21.15.169:0/1282681038 shutdown_connections 2024-03-28T00:54:48.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.946+0000 7fed9aa27640 1 -- 172.21.15.169:0/1282681038 wait complete. 2024-03-28T00:54:48.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.947+0000 7fed9aa27640 1 Processor -- start 2024-03-28T00:54:48.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.947+0000 7fed9aa27640 1 -- start start 2024-03-28T00:54:48.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.947+0000 7fed9aa27640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed941005a0 0x7fed941967d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:48.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.947+0000 7fed9aa27640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed94101790 0x7fed94196d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:48.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.947+0000 7fed9aa27640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed94102150 0x7fed9419dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:48.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.947+0000 7fed9aa27640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9410c230 con 0x7fed94102150 2024-03-28T00:54:48.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.947+0000 7fed9aa27640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9410c0b0 con 0x7fed941005a0 2024-03-28T00:54:48.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.947+0000 7fed9aa27640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9410c3b0 con 0x7fed94101790 2024-03-28T00:54:48.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.947+0000 7fed98fc4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed94102150 0x7fed9419dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:48.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.947+0000 7fed98fc4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed94102150 0x7fed9419dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38742/0 (socket says 172.21.15.169:38742) 2024-03-28T00:54:48.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.947+0000 7fed98fc4640 1 -- 172.21.15.169:0/3034341037 learned_addr learned my addr 172.21.15.169:0/3034341037 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:48.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.947+0000 7fed93fff640 1 --2- 172.21.15.169:0/3034341037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed941005a0 0x7fed941967d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:48.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.948+0000 7fed937fe640 1 --2- 172.21.15.169:0/3034341037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed94101790 0x7fed94196d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:48.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.948+0000 7fed98fc4640 1 -- 172.21.15.169:0/3034341037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed94101790 msgr2=0x7fed94196d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:48.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.948+0000 7fed98fc4640 1 --2- 172.21.15.169:0/3034341037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed94101790 0x7fed94196d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.948+0000 7fed98fc4640 1 -- 172.21.15.169:0/3034341037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed941005a0 msgr2=0x7fed941967d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:48.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.948+0000 7fed98fc4640 1 --2- 172.21.15.169:0/3034341037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed941005a0 0x7fed941967d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:48.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.948+0000 7fed98fc4640 1 -- 172.21.15.169:0/3034341037 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9419e3f0 con 0x7fed94102150 2024-03-28T00:54:48.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.948+0000 7fed98fc4640 1 --2- 172.21.15.169:0/3034341037 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed94102150 0x7fed9419dd80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fed8c037040 tx=0x7fed8c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:48.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.948+0000 7fed917fa640 1 -- 172.21.15.169:0/3034341037 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8c041020 con 0x7fed94102150 2024-03-28T00:54:48.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.948+0000 7fed917fa640 1 -- 172.21.15.169:0/3034341037 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed8c034d90 con 0x7fed94102150 2024-03-28T00:54:48.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.948+0000 7fed9aa27640 1 -- 172.21.15.169:0/3034341037 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed9419e680 con 0x7fed94102150 2024-03-28T00:54:48.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.948+0000 7fed917fa640 1 -- 172.21.15.169:0/3034341037 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8c039040 con 0x7fed94102150 2024-03-28T00:54:48.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.948+0000 7fed9aa27640 1 -- 172.21.15.169:0/3034341037 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed9419eb70 con 0x7fed94102150 2024-03-28T00:54:48.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.949+0000 7fed9aa27640 1 -- 172.21.15.169:0/3034341037 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed941078c0 con 0x7fed94102150 2024-03-28T00:54:48.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.950+0000 7fed917fa640 1 -- 172.21.15.169:0/3034341037 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fed8c044020 con 0x7fed94102150 2024-03-28T00:54:48.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.951+0000 7fed917fa640 1 --2- 172.21.15.169:0/3034341037 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed7c0a2800 0x7fed7c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:48.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.951+0000 7fed93fff640 1 --2- 172.21.15.169:0/3034341037 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed7c0a2800 0x7fed7c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:48.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.951+0000 7fed917fa640 1 -- 172.21.15.169:0/3034341037 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fed8c0f0c20 con 0x7fed94102150 2024-03-28T00:54:48.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.951+0000 7fed93fff640 1 --2- 172.21.15.169:0/3034341037 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed7c0a2800 0x7fed7c0a4cc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fed84005fd0 tx=0x7fed8400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:48.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:48.955+0000 7fed917fa640 1 -- 172.21.15.169:0/3034341037 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fed8c04b020 con 0x7fed94102150 2024-03-28T00:54:49.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.168+0000 7fed9aa27640 1 -- 172.21.15.169:0/3034341037 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fed940622a0 con 0x7fed94102150 2024-03-28T00:54:49.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.241+0000 7fed917fa640 1 -- 172.21.15.169:0/3034341037 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7fed8c0bda80 con 0x7fed94102150 2024-03-28T00:54:49.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.243+0000 7fed9aa27640 1 -- 172.21.15.169:0/3034341037 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed7c0a2800 msgr2=0x7fed7c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:49.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.243+0000 7fed9aa27640 1 --2- 172.21.15.169:0/3034341037 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed7c0a2800 0x7fed7c0a4cc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fed84005fd0 tx=0x7fed8400b410 comp rx=0 tx=0).stop 2024-03-28T00:54:49.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.243+0000 7fed9aa27640 1 -- 172.21.15.169:0/3034341037 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed94102150 msgr2=0x7fed9419dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:49.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.243+0000 7fed9aa27640 1 --2- 172.21.15.169:0/3034341037 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed94102150 0x7fed9419dd80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fed8c037040 tx=0x7fed8c032f20 comp rx=0 tx=0).stop 2024-03-28T00:54:49.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.243+0000 7fed9aa27640 1 -- 172.21.15.169:0/3034341037 shutdown_connections 2024-03-28T00:54:49.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.243+0000 7fed9aa27640 1 --2- 172.21.15.169:0/3034341037 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed7c0a2800 0x7fed7c0a4cc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:49.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.243+0000 7fed9aa27640 1 --2- 172.21.15.169:0/3034341037 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed94102150 0x7fed9419dd80 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:49.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.243+0000 7fed9aa27640 1 --2- 172.21.15.169:0/3034341037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed94101790 0x7fed94196d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:49.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.243+0000 7fed9aa27640 1 --2- 172.21.15.169:0/3034341037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed941005a0 0x7fed941967d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:49.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.243+0000 7fed9aa27640 1 -- 172.21.15.169:0/3034341037 >> 172.21.15.169:0/3034341037 conn(0x7fed940fbb30 msgr2=0x7fed94104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:49.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.244+0000 7fed9aa27640 1 -- 172.21.15.169:0/3034341037 shutdown_connections 2024-03-28T00:54:49.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.244+0000 7fed9aa27640 1 -- 172.21.15.169:0/3034341037 wait complete. 2024-03-28T00:54:49.258 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-28T00:54:49.259 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-28T00:54:49.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.361+0000 7fb19880b640 1 -- 172.21.15.169:0/1449253224 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1941016c0 msgr2=0x7fb194101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:49.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.361+0000 7fb19880b640 1 --2- 172.21.15.169:0/1449253224 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1941016c0 0x7fb194101ae0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb184009900 tx=0x7fb18402bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.361+0000 7fb19880b640 1 -- 172.21.15.169:0/1449253224 shutdown_connections 2024-03-28T00:54:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.361+0000 7fb19880b640 1 --2- 172.21.15.169:0/1449253224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb194103270 0x7fb194109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.361+0000 7fb19880b640 1 --2- 172.21.15.169:0/1449253224 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1941028b0 0x7fb194102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.361+0000 7fb19880b640 1 --2- 172.21.15.169:0/1449253224 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1941016c0 0x7fb194101ae0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.361+0000 7fb19880b640 1 -- 172.21.15.169:0/1449253224 >> 172.21.15.169:0/1449253224 conn(0x7fb1940fcc50 msgr2=0x7fb1940ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.361+0000 7fb19880b640 1 -- 172.21.15.169:0/1449253224 shutdown_connections 2024-03-28T00:54:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.361+0000 7fb19880b640 1 -- 172.21.15.169:0/1449253224 wait complete. 2024-03-28T00:54:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.362+0000 7fb19880b640 1 Processor -- start 2024-03-28T00:54:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.362+0000 7fb19880b640 1 -- start start 2024-03-28T00:54:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.362+0000 7fb19880b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1941016c0 0x7fb1941979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.362+0000 7fb19880b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1941028b0 0x7fb194197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.362+0000 7fb19880b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb194103270 0x7fb19419ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.362+0000 7fb19880b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19410d390 con 0x7fb194103270 2024-03-28T00:54:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.362+0000 7fb19880b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19410d210 con 0x7fb1941028b0 2024-03-28T00:54:49.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.362+0000 7fb19880b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19410d510 con 0x7fb1941016c0 2024-03-28T00:54:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.362+0000 7fb192d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb194103270 0x7fb19419ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.362+0000 7fb192d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb194103270 0x7fb19419ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38768/0 (socket says 172.21.15.169:38768) 2024-03-28T00:54:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.362+0000 7fb192d9d640 1 -- 172.21.15.169:0/2090518226 learned_addr learned my addr 172.21.15.169:0/2090518226 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.362+0000 7fb191d9b640 1 --2- 172.21.15.169:0/2090518226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1941028b0 0x7fb194197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.362+0000 7fb19259c640 1 --2- 172.21.15.169:0/2090518226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1941016c0 0x7fb1941979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.362+0000 7fb192d9d640 1 -- 172.21.15.169:0/2090518226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1941016c0 msgr2=0x7fb1941979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.362+0000 7fb192d9d640 1 --2- 172.21.15.169:0/2090518226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1941016c0 0x7fb1941979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.362+0000 7fb192d9d640 1 -- 172.21.15.169:0/2090518226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1941028b0 msgr2=0x7fb194197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.362+0000 7fb192d9d640 1 --2- 172.21.15.169:0/2090518226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1941028b0 0x7fb194197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:49.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.362+0000 7fb192d9d640 1 -- 172.21.15.169:0/2090518226 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19419f5c0 con 0x7fb194103270 2024-03-28T00:54:49.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.362+0000 7fb191d9b640 1 --2- 172.21.15.169:0/2090518226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1941028b0 0x7fb194197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:49.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.363+0000 7fb192d9d640 1 --2- 172.21.15.169:0/2090518226 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb194103270 0x7fb19419ef50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb18c00eaa0 tx=0x7fb18c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:49.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.363+0000 7fb1837fe640 1 -- 172.21.15.169:0/2090518226 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18c00cdb0 con 0x7fb194103270 2024-03-28T00:54:49.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.363+0000 7fb1837fe640 1 -- 172.21.15.169:0/2090518226 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb18c005020 con 0x7fb194103270 2024-03-28T00:54:49.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.363+0000 7fb19880b640 1 -- 172.21.15.169:0/2090518226 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb19419f8b0 con 0x7fb194103270 2024-03-28T00:54:49.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.363+0000 7fb1837fe640 1 -- 172.21.15.169:0/2090518226 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18c0116b0 con 0x7fb194103270 2024-03-28T00:54:49.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.363+0000 7fb19880b640 1 -- 172.21.15.169:0/2090518226 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb19419fe00 con 0x7fb194103270 2024-03-28T00:54:49.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.364+0000 7fb19880b640 1 -- 172.21.15.169:0/2090518226 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1941089e0 con 0x7fb194103270 2024-03-28T00:54:49.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.365+0000 7fb1837fe640 1 -- 172.21.15.169:0/2090518226 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb18c005560 con 0x7fb194103270 2024-03-28T00:54:49.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.365+0000 7fb1837fe640 1 --2- 172.21.15.169:0/2090518226 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb17c0a2800 0x7fb17c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:49.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.365+0000 7fb19259c640 1 --2- 172.21.15.169:0/2090518226 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb17c0a2800 0x7fb17c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:49.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.365+0000 7fb1837fe640 1 -- 172.21.15.169:0/2090518226 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb18c0cee40 con 0x7fb194103270 2024-03-28T00:54:49.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.366+0000 7fb19259c640 1 --2- 172.21.15.169:0/2090518226 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb17c0a2800 0x7fb17c0a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb18402c170 tx=0x7fb18402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:49.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.370+0000 7fb1837fe640 1 -- 172.21.15.169:0/2090518226 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb18c016030 con 0x7fb194103270 2024-03-28T00:54:49.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:49 smithi169 ceph-mon[32274]: pgmap v296: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-03-28T00:54:49.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:49 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3073720943' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:49.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:49 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3936013352' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:49.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:49 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3034341037' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T00:54:49.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:49.587+0000 7fb19880b640 1 -- 172.21.15.169:0/2090518226 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7fb1940622a0 con 0x7fb194103270 2024-03-28T00:54:49.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:49 smithi174 ceph-mon[36060]: pgmap v296: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-03-28T00:54:49.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:49 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3073720943' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:49.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:49 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3936013352' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:49.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:49 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3034341037' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T00:54:49.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:49 smithi177 ceph-mon[36206]: pgmap v296: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-03-28T00:54:49.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:49 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3073720943' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:49.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:49 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3936013352' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:49.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:49 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3034341037' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T00:54:50.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.255+0000 7fb1837fe640 1 -- 172.21.15.169:0/2090518226 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7fb18c09b650 con 0x7fb194103270 2024-03-28T00:54:50.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.258+0000 7fb19880b640 1 -- 172.21.15.169:0/2090518226 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb17c0a2800 msgr2=0x7fb17c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:50.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.258+0000 7fb19880b640 1 --2- 172.21.15.169:0/2090518226 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb17c0a2800 0x7fb17c0a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb18402c170 tx=0x7fb18402df90 comp rx=0 tx=0).stop 2024-03-28T00:54:50.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.258+0000 7fb19880b640 1 -- 172.21.15.169:0/2090518226 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb194103270 msgr2=0x7fb19419ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:50.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.258+0000 7fb19880b640 1 --2- 172.21.15.169:0/2090518226 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb194103270 0x7fb19419ef50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb18c00eaa0 tx=0x7fb18c00ef60 comp rx=0 tx=0).stop 2024-03-28T00:54:50.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.258+0000 7fb19880b640 1 -- 172.21.15.169:0/2090518226 shutdown_connections 2024-03-28T00:54:50.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.258+0000 7fb19880b640 1 --2- 172.21.15.169:0/2090518226 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb17c0a2800 0x7fb17c0a4cc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:50.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.258+0000 7fb19880b640 1 --2- 172.21.15.169:0/2090518226 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb194103270 0x7fb19419ef50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:50.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.258+0000 7fb19880b640 1 --2- 172.21.15.169:0/2090518226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1941028b0 0x7fb194197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:50.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.258+0000 7fb19880b640 1 --2- 172.21.15.169:0/2090518226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1941016c0 0x7fb1941979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:50.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.258+0000 7fb19880b640 1 -- 172.21.15.169:0/2090518226 >> 172.21.15.169:0/2090518226 conn(0x7fb1940fcc50 msgr2=0x7fb194105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:50.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.259+0000 7fb19880b640 1 -- 172.21.15.169:0/2090518226 shutdown_connections 2024-03-28T00:54:50.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.259+0000 7fb19880b640 1 -- 172.21.15.169:0/2090518226 wait complete. 2024-03-28T00:54:50.278 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-28T00:54:50.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.385+0000 7f9db5a7e640 1 -- 172.21.15.169:0/2710564044 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9db01016a0 msgr2=0x7f9db0101ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:50.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.385+0000 7f9db5a7e640 1 --2- 172.21.15.169:0/2710564044 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9db01016a0 0x7f9db0101ac0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9da4009900 tx=0x7f9da402bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:50.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.385+0000 7f9db5a7e640 1 -- 172.21.15.169:0/2710564044 shutdown_connections 2024-03-28T00:54:50.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.385+0000 7f9db5a7e640 1 --2- 172.21.15.169:0/2710564044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9db0103250 0x7f9db0109ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:50.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.385+0000 7f9db5a7e640 1 --2- 172.21.15.169:0/2710564044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db0102890 0x7f9db0102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:50.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.385+0000 7f9db5a7e640 1 --2- 172.21.15.169:0/2710564044 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9db01016a0 0x7f9db0101ac0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:50.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.385+0000 7f9db5a7e640 1 -- 172.21.15.169:0/2710564044 >> 172.21.15.169:0/2710564044 conn(0x7f9db00fcc10 msgr2=0x7f9db00ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:50.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.385+0000 7f9db5a7e640 1 -- 172.21.15.169:0/2710564044 shutdown_connections 2024-03-28T00:54:50.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.385+0000 7f9db5a7e640 1 -- 172.21.15.169:0/2710564044 wait complete. 2024-03-28T00:54:50.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.385+0000 7f9db5a7e640 1 Processor -- start 2024-03-28T00:54:50.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.386+0000 7f9db5a7e640 1 -- start start 2024-03-28T00:54:50.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.386+0000 7f9db5a7e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9db01016a0 0x7f9db01979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:50.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.386+0000 7f9db5a7e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db0102890 0x7f9db0197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:50.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.386+0000 7f9db5a7e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9db0103250 0x7f9db019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:50.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.386+0000 7f9db5a7e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db010d370 con 0x7f9db0103250 2024-03-28T00:54:50.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.386+0000 7f9db5a7e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db010d1f0 con 0x7f9db0102890 2024-03-28T00:54:50.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.386+0000 7f9db5a7e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db010d4f0 con 0x7f9db01016a0 2024-03-28T00:54:50.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.386+0000 7f9daffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9db0103250 0x7f9db019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:50.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.386+0000 7f9daffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9db0103250 0x7f9db019ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38780/0 (socket says 172.21.15.169:38780) 2024-03-28T00:54:50.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.386+0000 7f9daffff640 1 -- 172.21.15.169:0/1663163664 learned_addr learned my addr 172.21.15.169:0/1663163664 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:50.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.386+0000 7f9daeffd640 1 --2- 172.21.15.169:0/1663163664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db0102890 0x7f9db0197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:50.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.386+0000 7f9daf7fe640 1 --2- 172.21.15.169:0/1663163664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9db01016a0 0x7f9db01979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:50.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.386+0000 7f9daffff640 1 -- 172.21.15.169:0/1663163664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9db01016a0 msgr2=0x7f9db01979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:50.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.386+0000 7f9daffff640 1 --2- 172.21.15.169:0/1663163664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9db01016a0 0x7f9db01979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:50.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.386+0000 7f9daffff640 1 -- 172.21.15.169:0/1663163664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db0102890 msgr2=0x7f9db0197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:50.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.386+0000 7f9daffff640 1 --2- 172.21.15.169:0/1663163664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db0102890 0x7f9db0197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:50.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.386+0000 7f9daffff640 1 -- 172.21.15.169:0/1663163664 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db019f5d0 con 0x7f9db0103250 2024-03-28T00:54:50.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.386+0000 7f9daf7fe640 1 --2- 172.21.15.169:0/1663163664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9db01016a0 0x7f9db01979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:54:50.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.386+0000 7f9daffff640 1 --2- 172.21.15.169:0/1663163664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9db0103250 0x7f9db019ef60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9da800cad0 tx=0x7f9da800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:50.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.387+0000 7f9dacff9640 1 -- 172.21.15.169:0/1663163664 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da800ae20 con 0x7f9db0103250 2024-03-28T00:54:50.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.387+0000 7f9db5a7e640 1 -- 172.21.15.169:0/1663163664 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db019f8c0 con 0x7f9db0103250 2024-03-28T00:54:50.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.387+0000 7f9dacff9640 1 -- 172.21.15.169:0/1663163664 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9da8005020 con 0x7f9db0103250 2024-03-28T00:54:50.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.387+0000 7f9dacff9640 1 -- 172.21.15.169:0/1663163664 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da800f6b0 con 0x7f9db0103250 2024-03-28T00:54:50.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.387+0000 7f9db5a7e640 1 -- 172.21.15.169:0/1663163664 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db019fe10 con 0x7f9db0103250 2024-03-28T00:54:50.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.388+0000 7f9db5a7e640 1 -- 172.21.15.169:0/1663163664 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db01089c0 con 0x7f9db0103250 2024-03-28T00:54:50.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.389+0000 7f9dacff9640 1 -- 172.21.15.169:0/1663163664 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9da80051d0 con 0x7f9db0103250 2024-03-28T00:54:50.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.389+0000 7f9dacff9640 1 --2- 172.21.15.169:0/1663163664 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9d980a2800 0x7f9d980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:50.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.389+0000 7f9daf7fe640 1 --2- 172.21.15.169:0/1663163664 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9d980a2800 0x7f9d980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:50.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.389+0000 7f9dacff9640 1 -- 172.21.15.169:0/1663163664 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f9da80cf020 con 0x7f9db0103250 2024-03-28T00:54:50.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.390+0000 7f9daf7fe640 1 --2- 172.21.15.169:0/1663163664 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9d980a2800 0x7f9d980a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9da40096c0 tx=0x7f9da402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:50.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.393+0000 7f9dacff9640 1 -- 172.21.15.169:0/1663163664 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9da8014030 con 0x7f9db0103250 2024-03-28T00:54:50.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:50 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3034341037' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T00:54:50.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:50 smithi169 ceph-mon[32274]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T00:54:50.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:50 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2090518226' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-28T00:54:50.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:50 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3034341037' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T00:54:50.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:50 smithi174 ceph-mon[36060]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T00:54:50.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:50 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2090518226' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-28T00:54:50.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:50.608+0000 7f9db5a7e640 1 -- 172.21.15.169:0/1663163664 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f9db00622a0 con 0x7f9db0103250 2024-03-28T00:54:50.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:50 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3034341037' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T00:54:50.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:50 smithi177 ceph-mon[36206]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T00:54:50.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:50 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2090518226' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-28T00:54:51.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.262+0000 7f9dacff9640 1 -- 172.21.15.169:0/1663163664 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7f9da809be80 con 0x7f9db0103250 2024-03-28T00:54:51.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.265+0000 7f9db5a7e640 1 -- 172.21.15.169:0/1663163664 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9d980a2800 msgr2=0x7f9d980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:51.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.265+0000 7f9db5a7e640 1 --2- 172.21.15.169:0/1663163664 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9d980a2800 0x7f9d980a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9da40096c0 tx=0x7f9da402df90 comp rx=0 tx=0).stop 2024-03-28T00:54:51.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.265+0000 7f9db5a7e640 1 -- 172.21.15.169:0/1663163664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9db0103250 msgr2=0x7f9db019ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:51.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.265+0000 7f9db5a7e640 1 --2- 172.21.15.169:0/1663163664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9db0103250 0x7f9db019ef60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9da800cad0 tx=0x7f9da800cf90 comp rx=0 tx=0).stop 2024-03-28T00:54:51.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.265+0000 7f9db5a7e640 1 -- 172.21.15.169:0/1663163664 shutdown_connections 2024-03-28T00:54:51.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.265+0000 7f9db5a7e640 1 --2- 172.21.15.169:0/1663163664 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9d980a2800 0x7f9d980a4cc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:51.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.265+0000 7f9db5a7e640 1 --2- 172.21.15.169:0/1663163664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9db0103250 0x7f9db019ef60 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:51.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.265+0000 7f9db5a7e640 1 --2- 172.21.15.169:0/1663163664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db0102890 0x7f9db0197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:51.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.265+0000 7f9db5a7e640 1 --2- 172.21.15.169:0/1663163664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9db01016a0 0x7f9db01979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:51.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.265+0000 7f9db5a7e640 1 -- 172.21.15.169:0/1663163664 >> 172.21.15.169:0/1663163664 conn(0x7f9db00fcc10 msgr2=0x7f9db0105b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:51.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.265+0000 7f9db5a7e640 1 -- 172.21.15.169:0/1663163664 shutdown_connections 2024-03-28T00:54:51.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.265+0000 7f9db5a7e640 1 -- 172.21.15.169:0/1663163664 wait complete. 2024-03-28T00:54:51.284 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-28T00:54:51.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.388+0000 7fbf742a1640 1 -- 172.21.15.169:0/91690626 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbf6c1016b0 msgr2=0x7fbf6c101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:51.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.388+0000 7fbf742a1640 1 --2- 172.21.15.169:0/91690626 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbf6c1016b0 0x7fbf6c101ad0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fbf68009900 tx=0x7fbf6802bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:51.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.388+0000 7fbf742a1640 1 -- 172.21.15.169:0/91690626 shutdown_connections 2024-03-28T00:54:51.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.388+0000 7fbf742a1640 1 --2- 172.21.15.169:0/91690626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf6c103260 0x7fbf6c109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:51.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.388+0000 7fbf742a1640 1 --2- 172.21.15.169:0/91690626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf6c1028a0 0x7fbf6c102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:51.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.388+0000 7fbf742a1640 1 --2- 172.21.15.169:0/91690626 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbf6c1016b0 0x7fbf6c101ad0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:51.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.388+0000 7fbf742a1640 1 -- 172.21.15.169:0/91690626 >> 172.21.15.169:0/91690626 conn(0x7fbf6c0fcc40 msgr2=0x7fbf6c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:51.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.388+0000 7fbf742a1640 1 -- 172.21.15.169:0/91690626 shutdown_connections 2024-03-28T00:54:51.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.388+0000 7fbf742a1640 1 -- 172.21.15.169:0/91690626 wait complete. 2024-03-28T00:54:51.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.389+0000 7fbf742a1640 1 Processor -- start 2024-03-28T00:54:51.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.389+0000 7fbf742a1640 1 -- start start 2024-03-28T00:54:51.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.389+0000 7fbf742a1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbf6c1016b0 0x7fbf6c197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:51.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.389+0000 7fbf742a1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf6c1028a0 0x7fbf6c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:51.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.389+0000 7fbf742a1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf6c103260 0x7fbf6c19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:51.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.389+0000 7fbf742a1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c10d330 con 0x7fbf6c1016b0 2024-03-28T00:54:51.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.389+0000 7fbf742a1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c10d1b0 con 0x7fbf6c103260 2024-03-28T00:54:51.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.389+0000 7fbf742a1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c10d4b0 con 0x7fbf6c1028a0 2024-03-28T00:54:51.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.389+0000 7fbf7203d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbf6c1016b0 0x7fbf6c197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:51.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.389+0000 7fbf7203d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbf6c1016b0 0x7fbf6c197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38796/0 (socket says 172.21.15.169:38796) 2024-03-28T00:54:51.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.389+0000 7fbf7203d640 1 -- 172.21.15.169:0/478819451 learned_addr learned my addr 172.21.15.169:0/478819451 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:51.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.389+0000 7fbf7183c640 1 --2- 172.21.15.169:0/478819451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf6c1028a0 0x7fbf6c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:51.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.389+0000 7fbf7283e640 1 --2- 172.21.15.169:0/478819451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf6c103260 0x7fbf6c19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:51.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.389+0000 7fbf7203d640 1 -- 172.21.15.169:0/478819451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf6c1028a0 msgr2=0x7fbf6c197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:51.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.389+0000 7fbf7203d640 1 --2- 172.21.15.169:0/478819451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf6c1028a0 0x7fbf6c197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:51.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.389+0000 7fbf7203d640 1 -- 172.21.15.169:0/478819451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf6c103260 msgr2=0x7fbf6c19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:51.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.389+0000 7fbf7203d640 1 --2- 172.21.15.169:0/478819451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf6c103260 0x7fbf6c19f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:51.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.389+0000 7fbf7203d640 1 -- 172.21.15.169:0/478819451 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6c19f670 con 0x7fbf6c1016b0 2024-03-28T00:54:51.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.389+0000 7fbf7183c640 1 --2- 172.21.15.169:0/478819451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf6c1028a0 0x7fbf6c197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:51.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.390+0000 7fbf7203d640 1 --2- 172.21.15.169:0/478819451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbf6c1016b0 0x7fbf6c197a50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbf68000c00 tx=0x7fbf6802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:51.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.390+0000 7fbf637fe640 1 -- 172.21.15.169:0/478819451 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf68041070 con 0x7fbf6c1016b0 2024-03-28T00:54:51.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.390+0000 7fbf742a1640 1 -- 172.21.15.169:0/478819451 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6c19f900 con 0x7fbf6c1016b0 2024-03-28T00:54:51.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.390+0000 7fbf637fe640 1 -- 172.21.15.169:0/478819451 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf68004450 con 0x7fbf6c1016b0 2024-03-28T00:54:51.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.390+0000 7fbf637fe640 1 -- 172.21.15.169:0/478819451 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf68034440 con 0x7fbf6c1016b0 2024-03-28T00:54:51.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.390+0000 7fbf742a1640 1 -- 172.21.15.169:0/478819451 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf6c19fdf0 con 0x7fbf6c1016b0 2024-03-28T00:54:51.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.391+0000 7fbf742a1640 1 -- 172.21.15.169:0/478819451 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf6c108af0 con 0x7fbf6c1016b0 2024-03-28T00:54:51.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.392+0000 7fbf637fe640 1 -- 172.21.15.169:0/478819451 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbf68044070 con 0x7fbf6c1016b0 2024-03-28T00:54:51.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.393+0000 7fbf637fe640 1 --2- 172.21.15.169:0/478819451 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbf580a2800 0x7fbf580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:51.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.393+0000 7fbf7183c640 1 --2- 172.21.15.169:0/478819451 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbf580a2800 0x7fbf580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:51.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.393+0000 7fbf637fe640 1 -- 172.21.15.169:0/478819451 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fbf680f0760 con 0x7fbf6c1016b0 2024-03-28T00:54:51.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.393+0000 7fbf7183c640 1 --2- 172.21.15.169:0/478819451 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbf580a2800 0x7fbf580a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c198f60 tx=0x7fbf5c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:51.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.397+0000 7fbf637fe640 1 -- 172.21.15.169:0/478819451 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbf680bd000 con 0x7fbf6c1016b0 2024-03-28T00:54:51.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:51 smithi169 ceph-mon[32274]: pgmap v297: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 272 B/s wr, 0 op/s 2024-03-28T00:54:51.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:51 smithi169 ceph-mon[32274]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T00:54:51.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:51 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2090518226' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-28T00:54:51.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:51 smithi169 ceph-mon[32274]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T00:54:51.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:51 smithi169 ceph-mon[32274]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T00:54:51.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:51 smithi169 ceph-mon[32274]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-28T00:54:51.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:51 smithi169 ceph-mon[32274]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-28T00:54:51.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:51 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1663163664' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T00:54:51.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:51 smithi174 ceph-mon[36060]: pgmap v297: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 272 B/s wr, 0 op/s 2024-03-28T00:54:51.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:51 smithi174 ceph-mon[36060]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T00:54:51.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:51 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2090518226' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-28T00:54:51.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:51 smithi174 ceph-mon[36060]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T00:54:51.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:51 smithi174 ceph-mon[36060]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T00:54:51.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:51 smithi174 ceph-mon[36060]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-28T00:54:51.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:51 smithi174 ceph-mon[36060]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-28T00:54:51.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:51 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1663163664' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T00:54:51.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:51.615+0000 7fbf742a1640 1 -- 172.21.15.169:0/478819451 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fbf6c0622a0 con 0x7fbf6c1016b0 2024-03-28T00:54:51.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:51 smithi177 ceph-mon[36206]: pgmap v297: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 272 B/s wr, 0 op/s 2024-03-28T00:54:51.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:51 smithi177 ceph-mon[36206]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T00:54:51.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:51 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2090518226' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-28T00:54:51.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:51 smithi177 ceph-mon[36206]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T00:54:51.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:51 smithi177 ceph-mon[36206]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T00:54:51.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:51 smithi177 ceph-mon[36206]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-28T00:54:51.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:51 smithi177 ceph-mon[36206]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-28T00:54:51.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:51 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1663163664' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T00:54:52.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.270+0000 7fbf637fe640 1 -- 172.21.15.169:0/478819451 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7fbf680c10a0 con 0x7fbf6c1016b0 2024-03-28T00:54:52.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.273+0000 7fbf742a1640 1 -- 172.21.15.169:0/478819451 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbf580a2800 msgr2=0x7fbf580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:52.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.273+0000 7fbf742a1640 1 --2- 172.21.15.169:0/478819451 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbf580a2800 0x7fbf580a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c198f60 tx=0x7fbf5c005f90 comp rx=0 tx=0).stop 2024-03-28T00:54:52.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.273+0000 7fbf742a1640 1 -- 172.21.15.169:0/478819451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbf6c1016b0 msgr2=0x7fbf6c197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:52.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.273+0000 7fbf742a1640 1 --2- 172.21.15.169:0/478819451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbf6c1016b0 0x7fbf6c197a50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbf68000c00 tx=0x7fbf6802c640 comp rx=0 tx=0).stop 2024-03-28T00:54:52.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.273+0000 7fbf742a1640 1 -- 172.21.15.169:0/478819451 shutdown_connections 2024-03-28T00:54:52.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.273+0000 7fbf742a1640 1 --2- 172.21.15.169:0/478819451 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbf580a2800 0x7fbf580a4cc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.273+0000 7fbf742a1640 1 --2- 172.21.15.169:0/478819451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf6c103260 0x7fbf6c19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.273+0000 7fbf742a1640 1 --2- 172.21.15.169:0/478819451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf6c1028a0 0x7fbf6c197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.273+0000 7fbf742a1640 1 --2- 172.21.15.169:0/478819451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbf6c1016b0 0x7fbf6c197a50 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.273+0000 7fbf742a1640 1 -- 172.21.15.169:0/478819451 >> 172.21.15.169:0/478819451 conn(0x7fbf6c0fcc40 msgr2=0x7fbf6c105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:52.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.273+0000 7fbf742a1640 1 -- 172.21.15.169:0/478819451 shutdown_connections 2024-03-28T00:54:52.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.273+0000 7fbf742a1640 1 -- 172.21.15.169:0/478819451 wait complete. 2024-03-28T00:54:52.289 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-03-28T00:54:52.289 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-28T00:54:52.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.392+0000 7f421cc36640 1 -- 172.21.15.169:0/1627618014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4218102160 msgr2=0x7f42181089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:52.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.392+0000 7f421cc36640 1 --2- 172.21.15.169:0/1627618014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4218102160 0x7f42181089f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4210009ca0 tx=0x7f421002c0f0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.392+0000 7f421cc36640 1 -- 172.21.15.169:0/1627618014 shutdown_connections 2024-03-28T00:54:52.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.392+0000 7f421cc36640 1 --2- 172.21.15.169:0/1627618014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4218102160 0x7f42181089f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.393+0000 7f421cc36640 1 --2- 172.21.15.169:0/1627618014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42181017a0 0x7f4218101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.393+0000 7f421cc36640 1 --2- 172.21.15.169:0/1627618014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42181005b0 0x7f42181009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.393+0000 7f421cc36640 1 -- 172.21.15.169:0/1627618014 >> 172.21.15.169:0/1627618014 conn(0x7f42180fbb40 msgr2=0x7f42180fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:52.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.393+0000 7f421cc36640 1 -- 172.21.15.169:0/1627618014 shutdown_connections 2024-03-28T00:54:52.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.393+0000 7f421cc36640 1 -- 172.21.15.169:0/1627618014 wait complete. 2024-03-28T00:54:52.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.393+0000 7f421cc36640 1 Processor -- start 2024-03-28T00:54:52.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.393+0000 7f421cc36640 1 -- start start 2024-03-28T00:54:52.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.393+0000 7f421cc36640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42181005b0 0x7f4218197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:52.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.393+0000 7f421cc36640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42181017a0 0x7f4218197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:52.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.393+0000 7f421cc36640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4218102160 0x7f421819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:52.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.393+0000 7f421cc36640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421810c230 con 0x7f4218102160 2024-03-28T00:54:52.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.393+0000 7f421cc36640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421810c0b0 con 0x7f42181005b0 2024-03-28T00:54:52.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.393+0000 7f421cc36640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421810c3b0 con 0x7f42181017a0 2024-03-28T00:54:52.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.394+0000 7f4216d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4218102160 0x7f421819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:52.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.394+0000 7f4216d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4218102160 0x7f421819ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38812/0 (socket says 172.21.15.169:38812) 2024-03-28T00:54:52.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.394+0000 7f4216d9d640 1 -- 172.21.15.169:0/2473236957 learned_addr learned my addr 172.21.15.169:0/2473236957 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:52.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.394+0000 7f420ffff640 1 --2- 172.21.15.169:0/2473236957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42181017a0 0x7f4218197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:52.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.394+0000 7f421659c640 1 --2- 172.21.15.169:0/2473236957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42181005b0 0x7f4218197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:52.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.394+0000 7f4216d9d640 1 -- 172.21.15.169:0/2473236957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42181017a0 msgr2=0x7f4218197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:52.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.394+0000 7f4216d9d640 1 --2- 172.21.15.169:0/2473236957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42181017a0 0x7f4218197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.394+0000 7f4216d9d640 1 -- 172.21.15.169:0/2473236957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42181005b0 msgr2=0x7f4218197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:52.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.394+0000 7f4216d9d640 1 --2- 172.21.15.169:0/2473236957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42181005b0 0x7f4218197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.394+0000 7f4216d9d640 1 -- 172.21.15.169:0/2473236957 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421819f590 con 0x7f4218102160 2024-03-28T00:54:52.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.394+0000 7f4216d9d640 1 --2- 172.21.15.169:0/2473236957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4218102160 0x7f421819ef20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4210037040 tx=0x7f4210032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:52.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.394+0000 7f420f7fe640 1 -- 172.21.15.169:0/2473236957 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4210041020 con 0x7f4218102160 2024-03-28T00:54:52.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.394+0000 7f420f7fe640 1 -- 172.21.15.169:0/2473236957 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4210034cd0 con 0x7f4218102160 2024-03-28T00:54:52.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.394+0000 7f421cc36640 1 -- 172.21.15.169:0/2473236957 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f421819f820 con 0x7f4218102160 2024-03-28T00:54:52.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.394+0000 7f420f7fe640 1 -- 172.21.15.169:0/2473236957 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4210039040 con 0x7f4218102160 2024-03-28T00:54:52.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.394+0000 7f421cc36640 1 -- 172.21.15.169:0/2473236957 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f421819fd10 con 0x7f4218102160 2024-03-28T00:54:52.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.395+0000 7f421cc36640 1 -- 172.21.15.169:0/2473236957 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f421819ff70 con 0x7f4218102160 2024-03-28T00:54:52.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.396+0000 7f420f7fe640 1 -- 172.21.15.169:0/2473236957 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4210044020 con 0x7f4218102160 2024-03-28T00:54:52.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.397+0000 7f420f7fe640 1 --2- 172.21.15.169:0/2473236957 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f42040a2800 0x7f42040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:52.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.397+0000 7f421659c640 1 --2- 172.21.15.169:0/2473236957 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f42040a2800 0x7f42040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:52.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.397+0000 7f420f7fe640 1 -- 172.21.15.169:0/2473236957 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f42100f0b40 con 0x7f4218102160 2024-03-28T00:54:52.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.397+0000 7f421659c640 1 --2- 172.21.15.169:0/2473236957 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f42040a2800 0x7f42040a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4218101600 tx=0x7f4208005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:52.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.401+0000 7f420f7fe640 1 -- 172.21.15.169:0/2473236957 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f421004b020 con 0x7f4218102160 2024-03-28T00:54:52.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:52 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1663163664' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T00:54:52.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:52 smithi169 ceph-mon[32274]: mds.? [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] up:active 2024-03-28T00:54:52.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:52 smithi169 ceph-mon[32274]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T00:54:52.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:52 smithi169 ceph-mon[32274]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T00:54:52.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:52 smithi169 ceph-mon[32274]: Cluster is now healthy 2024-03-28T00:54:52.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:52 smithi169 ceph-mon[32274]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-28T00:54:52.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:52 smithi169 ceph-mon[32274]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-28T00:54:52.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:52 smithi169 ceph-mon[32274]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-28T00:54:52.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:52 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/478819451' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T00:54:52.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:52 smithi169 ceph-mon[32274]: pgmap v298: 129 pgs: 129 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-03-28T00:54:52.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.602+0000 7f421cc36640 1 -- 172.21.15.169:0/2473236957 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f42180fdcc0 con 0x7f42040a2800 2024-03-28T00:54:52.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:52 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1663163664' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T00:54:52.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:52 smithi174 ceph-mon[36060]: mds.? [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] up:active 2024-03-28T00:54:52.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:52 smithi174 ceph-mon[36060]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T00:54:52.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:52 smithi174 ceph-mon[36060]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T00:54:52.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:52 smithi174 ceph-mon[36060]: Cluster is now healthy 2024-03-28T00:54:52.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:52 smithi174 ceph-mon[36060]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-28T00:54:52.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:52 smithi174 ceph-mon[36060]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-28T00:54:52.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:52 smithi174 ceph-mon[36060]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-28T00:54:52.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:52 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/478819451' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T00:54:52.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:52 smithi174 ceph-mon[36060]: pgmap v298: 129 pgs: 129 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-03-28T00:54:52.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.624+0000 7f420f7fe640 1 -- 172.21.15.169:0/2473236957 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f42180fdcc0 con 0x7f42040a2800 2024-03-28T00:54:52.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.627+0000 7f421cc36640 1 -- 172.21.15.169:0/2473236957 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f42040a2800 msgr2=0x7f42040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:52.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.627+0000 7f421cc36640 1 --2- 172.21.15.169:0/2473236957 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f42040a2800 0x7f42040a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4218101600 tx=0x7f4208005f90 comp rx=0 tx=0).stop 2024-03-28T00:54:52.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.627+0000 7f421cc36640 1 -- 172.21.15.169:0/2473236957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4218102160 msgr2=0x7f421819ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:52.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.627+0000 7f421cc36640 1 --2- 172.21.15.169:0/2473236957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4218102160 0x7f421819ef20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4210037040 tx=0x7f4210032f40 comp rx=0 tx=0).stop 2024-03-28T00:54:52.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.627+0000 7f421cc36640 1 -- 172.21.15.169:0/2473236957 shutdown_connections 2024-03-28T00:54:52.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.627+0000 7f421cc36640 1 --2- 172.21.15.169:0/2473236957 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f42040a2800 0x7f42040a4cc0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.627+0000 7f421cc36640 1 --2- 172.21.15.169:0/2473236957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4218102160 0x7f421819ef20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.627+0000 7f421cc36640 1 --2- 172.21.15.169:0/2473236957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42181017a0 0x7f4218197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.627+0000 7f421cc36640 1 --2- 172.21.15.169:0/2473236957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42181005b0 0x7f4218197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.627+0000 7f421cc36640 1 -- 172.21.15.169:0/2473236957 >> 172.21.15.169:0/2473236957 conn(0x7f42180fbb40 msgr2=0x7f42180fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:52.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.627+0000 7f421cc36640 1 -- 172.21.15.169:0/2473236957 shutdown_connections 2024-03-28T00:54:52.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.627+0000 7f421cc36640 1 -- 172.21.15.169:0/2473236957 wait complete. 2024-03-28T00:54:52.641 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-28T00:54:52.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:52 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1663163664' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T00:54:52.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:52 smithi177 ceph-mon[36206]: mds.? [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] up:active 2024-03-28T00:54:52.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:52 smithi177 ceph-mon[36206]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T00:54:52.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:52 smithi177 ceph-mon[36206]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T00:54:52.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:52 smithi177 ceph-mon[36206]: Cluster is now healthy 2024-03-28T00:54:52.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:52 smithi177 ceph-mon[36206]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-28T00:54:52.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:52 smithi177 ceph-mon[36206]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-28T00:54:52.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:52 smithi177 ceph-mon[36206]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-28T00:54:52.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:52 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/478819451' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T00:54:52.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:52 smithi177 ceph-mon[36206]: pgmap v298: 129 pgs: 129 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-03-28T00:54:52.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.746+0000 7f6787dda640 1 -- 172.21.15.169:0/4262254342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67801017a0 msgr2=0x7f6780101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:52.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.746+0000 7f6787dda640 1 --2- 172.21.15.169:0/4262254342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67801017a0 0x7f6780101c20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6770009900 tx=0x7f677002bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:52.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.746+0000 7f6787dda640 1 -- 172.21.15.169:0/4262254342 shutdown_connections 2024-03-28T00:54:52.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.746+0000 7f6787dda640 1 --2- 172.21.15.169:0/4262254342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6780102160 0x7f67801089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.746+0000 7f6787dda640 1 --2- 172.21.15.169:0/4262254342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67801017a0 0x7f6780101c20 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.746+0000 7f6787dda640 1 --2- 172.21.15.169:0/4262254342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67801005b0 0x7f67801009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.746+0000 7f6787dda640 1 -- 172.21.15.169:0/4262254342 >> 172.21.15.169:0/4262254342 conn(0x7f67800fbb40 msgr2=0x7f67800fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:52.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.746+0000 7f6787dda640 1 -- 172.21.15.169:0/4262254342 shutdown_connections 2024-03-28T00:54:52.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.746+0000 7f6787dda640 1 -- 172.21.15.169:0/4262254342 wait complete. 2024-03-28T00:54:52.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.747+0000 7f6787dda640 1 Processor -- start 2024-03-28T00:54:52.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.747+0000 7f6787dda640 1 -- start start 2024-03-28T00:54:52.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.747+0000 7f6787dda640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67801005b0 0x7f67801968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:52.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.747+0000 7f6787dda640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67801017a0 0x7f6780196df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:52.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.747+0000 7f6787dda640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6780102160 0x7f678019de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:52.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.747+0000 7f6787dda640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678010c230 con 0x7f67801017a0 2024-03-28T00:54:52.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.747+0000 7f6787dda640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678010c0b0 con 0x7f6780102160 2024-03-28T00:54:52.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.747+0000 7f6787dda640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678010c3b0 con 0x7f67801005b0 2024-03-28T00:54:52.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.747+0000 7f6785375640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67801017a0 0x7f6780196df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:52.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.747+0000 7f6785b76640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67801005b0 0x7f67801968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:52.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.747+0000 7f6785b76640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67801005b0 0x7f67801968b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:42072/0 (socket says 172.21.15.169:42072) 2024-03-28T00:54:52.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.747+0000 7f6785b76640 1 -- 172.21.15.169:0/4174395859 learned_addr learned my addr 172.21.15.169:0/4174395859 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:52.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.747+0000 7f6785375640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67801017a0 0x7f6780196df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38852/0 (socket says 172.21.15.169:38852) 2024-03-28T00:54:52.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.747+0000 7f6786377640 1 --2- 172.21.15.169:0/4174395859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6780102160 0x7f678019de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:52.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.747+0000 7f6785b76640 1 -- 172.21.15.169:0/4174395859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6780102160 msgr2=0x7f678019de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:52.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.747+0000 7f6785b76640 1 --2- 172.21.15.169:0/4174395859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6780102160 0x7f678019de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.747+0000 7f6785b76640 1 -- 172.21.15.169:0/4174395859 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67801017a0 msgr2=0x7f6780196df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:52.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.747+0000 7f6785b76640 1 --2- 172.21.15.169:0/4174395859 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67801017a0 0x7f6780196df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.747+0000 7f6785b76640 1 -- 172.21.15.169:0/4174395859 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678019e4d0 con 0x7f67801005b0 2024-03-28T00:54:52.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.747+0000 7f6785375640 1 --2- 172.21.15.169:0/4174395859 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67801017a0 0x7f6780196df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:52.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.748+0000 7f6785b76640 1 --2- 172.21.15.169:0/4174395859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67801005b0 0x7f67801968b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f677c00ead0 tx=0x7f677c00ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:52.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.748+0000 7f6776ffd640 1 -- 172.21.15.169:0/4174395859 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677c01c070 con 0x7f67801005b0 2024-03-28T00:54:52.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.748+0000 7f6787dda640 1 -- 172.21.15.169:0/4174395859 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6780069cf0 con 0x7f67801005b0 2024-03-28T00:54:52.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.748+0000 7f6776ffd640 1 -- 172.21.15.169:0/4174395859 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f677c00bbd0 con 0x7f67801005b0 2024-03-28T00:54:52.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.748+0000 7f6787dda640 1 -- 172.21.15.169:0/4174395859 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f678006a1c0 con 0x7f67801005b0 2024-03-28T00:54:52.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.748+0000 7f6776ffd640 1 -- 172.21.15.169:0/4174395859 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677c026060 con 0x7f67801005b0 2024-03-28T00:54:52.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.749+0000 7f6787dda640 1 -- 172.21.15.169:0/4174395859 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67800688e0 con 0x7f67801005b0 2024-03-28T00:54:52.755 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.750+0000 7f6776ffd640 1 -- 172.21.15.169:0/4174395859 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f677c01f070 con 0x7f67801005b0 2024-03-28T00:54:52.755 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.751+0000 7f6776ffd640 1 --2- 172.21.15.169:0/4174395859 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f676c0a2800 0x7f676c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:52.755 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.751+0000 7f6785375640 1 --2- 172.21.15.169:0/4174395859 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f676c0a2800 0x7f676c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:52.755 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.751+0000 7f6776ffd640 1 -- 172.21.15.169:0/4174395859 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f677c0d0bb0 con 0x7f67801005b0 2024-03-28T00:54:52.755 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.751+0000 7f6785375640 1 --2- 172.21.15.169:0/4174395859 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f676c0a2800 0x7f676c0a4cc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f677002c7b0 tx=0x7f6770005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:52.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.756+0000 7f6776ffd640 1 -- 172.21.15.169:0/4174395859 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f677c09d360 con 0x7f67801005b0 2024-03-28T00:54:52.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.962+0000 7f6787dda640 1 -- 172.21.15.169:0/4174395859 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f678006a530 con 0x7f676c0a2800 2024-03-28T00:54:52.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.985+0000 7f6776ffd640 1 -- 172.21.15.169:0/4174395859 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f678006a530 con 0x7f676c0a2800 2024-03-28T00:54:52.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.987+0000 7f6787dda640 1 -- 172.21.15.169:0/4174395859 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f676c0a2800 msgr2=0x7f676c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:52.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.987+0000 7f6787dda640 1 --2- 172.21.15.169:0/4174395859 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f676c0a2800 0x7f676c0a4cc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f677002c7b0 tx=0x7f6770005fd0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.987+0000 7f6787dda640 1 -- 172.21.15.169:0/4174395859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67801005b0 msgr2=0x7f67801968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:52.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.987+0000 7f6787dda640 1 --2- 172.21.15.169:0/4174395859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67801005b0 0x7f67801968b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f677c00ead0 tx=0x7f677c00ef90 comp rx=0 tx=0).stop 2024-03-28T00:54:52.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.987+0000 7f6787dda640 1 -- 172.21.15.169:0/4174395859 shutdown_connections 2024-03-28T00:54:52.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.987+0000 7f6787dda640 1 --2- 172.21.15.169:0/4174395859 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f676c0a2800 0x7f676c0a4cc0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.987+0000 7f6787dda640 1 --2- 172.21.15.169:0/4174395859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6780102160 0x7f678019de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.987+0000 7f6787dda640 1 --2- 172.21.15.169:0/4174395859 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67801017a0 0x7f6780196df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.987+0000 7f6787dda640 1 --2- 172.21.15.169:0/4174395859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67801005b0 0x7f67801968b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:52.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.987+0000 7f6787dda640 1 -- 172.21.15.169:0/4174395859 >> 172.21.15.169:0/4174395859 conn(0x7f67800fbb40 msgr2=0x7f6780104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:52.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.988+0000 7f6787dda640 1 -- 172.21.15.169:0/4174395859 shutdown_connections 2024-03-28T00:54:52.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:52.988+0000 7f6787dda640 1 -- 172.21.15.169:0/4174395859 wait complete. 2024-03-28T00:54:53.000 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:54:52 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:54:52.607+0000 7f5236b5a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T00:54:53.000 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:54:52 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:54:52.607+0000 7f5236b5a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T00:54:53.001 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:54:52 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:54:52.607+0000 7f5236b5a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T00:54:53.001 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:54:52 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:54:52.607+0000 7f5236b5a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T00:54:53.001 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:54:52 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:54:52.607+0000 7f5236b5a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T00:54:53.002 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-28T00:54:53.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.106+0000 7f854de75640 1 -- 172.21.15.169:0/1676536694 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8548101790 msgr2=0x7f8548101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:53.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.106+0000 7f854de75640 1 --2- 172.21.15.169:0/1676536694 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8548101790 0x7f8548101c10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f853c009900 tx=0x7f853c02bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:53.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.106+0000 7f854de75640 1 -- 172.21.15.169:0/1676536694 shutdown_connections 2024-03-28T00:54:53.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.106+0000 7f854de75640 1 --2- 172.21.15.169:0/1676536694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8548102150 0x7f85481089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.106+0000 7f854de75640 1 --2- 172.21.15.169:0/1676536694 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8548101790 0x7f8548101c10 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.106+0000 7f854de75640 1 --2- 172.21.15.169:0/1676536694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85481005a0 0x7f85481009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.106+0000 7f854de75640 1 -- 172.21.15.169:0/1676536694 >> 172.21.15.169:0/1676536694 conn(0x7f85480fbb30 msgr2=0x7f85480fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:53.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.106+0000 7f854de75640 1 -- 172.21.15.169:0/1676536694 shutdown_connections 2024-03-28T00:54:53.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.107+0000 7f854de75640 1 -- 172.21.15.169:0/1676536694 wait complete. 2024-03-28T00:54:53.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.107+0000 7f854de75640 1 Processor -- start 2024-03-28T00:54:53.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.107+0000 7f854de75640 1 -- start start 2024-03-28T00:54:53.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.107+0000 7f854de75640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85481005a0 0x7f85481979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:53.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.107+0000 7f854de75640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8548101790 0x7f8548197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:53.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.107+0000 7f854de75640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8548102150 0x7f854819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:53.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.107+0000 7f854de75640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854810c230 con 0x7f85481005a0 2024-03-28T00:54:53.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.107+0000 7f854de75640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854810c0b0 con 0x7f8548101790 2024-03-28T00:54:53.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.107+0000 7f854de75640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854810c3b0 con 0x7f8548102150 2024-03-28T00:54:53.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.107+0000 7f85477fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85481005a0 0x7f85481979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:53.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.107+0000 7f85477fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85481005a0 0x7f85481979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38868/0 (socket says 172.21.15.169:38868) 2024-03-28T00:54:53.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.107+0000 7f85477fe640 1 -- 172.21.15.169:0/613348412 learned_addr learned my addr 172.21.15.169:0/613348412 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:53.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.107+0000 7f8547fff640 1 --2- 172.21.15.169:0/613348412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8548102150 0x7f854819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:53.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.108+0000 7f8546ffd640 1 --2- 172.21.15.169:0/613348412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8548101790 0x7f8548197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:53.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.108+0000 7f85477fe640 1 -- 172.21.15.169:0/613348412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8548102150 msgr2=0x7f854819ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:53.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.108+0000 7f85477fe640 1 --2- 172.21.15.169:0/613348412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8548102150 0x7f854819ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.108+0000 7f85477fe640 1 -- 172.21.15.169:0/613348412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8548101790 msgr2=0x7f8548197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:53.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.108+0000 7f85477fe640 1 --2- 172.21.15.169:0/613348412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8548101790 0x7f8548197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.108+0000 7f85477fe640 1 -- 172.21.15.169:0/613348412 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854819f5d0 con 0x7f85481005a0 2024-03-28T00:54:53.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.108+0000 7f85477fe640 1 --2- 172.21.15.169:0/613348412 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85481005a0 0x7f85481979b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f853800e9c0 tx=0x7f853800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:53.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.108+0000 7f8544ff9640 1 -- 172.21.15.169:0/613348412 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853800cde0 con 0x7f85481005a0 2024-03-28T00:54:53.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.108+0000 7f8544ff9640 1 -- 172.21.15.169:0/613348412 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8538010910 con 0x7f85481005a0 2024-03-28T00:54:53.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.108+0000 7f854de75640 1 -- 172.21.15.169:0/613348412 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f854819f8c0 con 0x7f85481005a0 2024-03-28T00:54:53.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.108+0000 7f8544ff9640 1 -- 172.21.15.169:0/613348412 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8538018a00 con 0x7f85481005a0 2024-03-28T00:54:53.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.108+0000 7f854de75640 1 -- 172.21.15.169:0/613348412 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f854819fde0 con 0x7f85481005a0 2024-03-28T00:54:53.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.109+0000 7f854de75640 1 -- 172.21.15.169:0/613348412 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f854819ffe0 con 0x7f85481005a0 2024-03-28T00:54:53.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.110+0000 7f8544ff9640 1 -- 172.21.15.169:0/613348412 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f853801f020 con 0x7f85481005a0 2024-03-28T00:54:53.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.111+0000 7f8544ff9640 1 --2- 172.21.15.169:0/613348412 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f85300a2880 0x7f85300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:53.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.111+0000 7f8546ffd640 1 --2- 172.21.15.169:0/613348412 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f85300a2880 0x7f85300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:53.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.111+0000 7f8544ff9640 1 -- 172.21.15.169:0/613348412 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f85380ceee0 con 0x7f85481005a0 2024-03-28T00:54:53.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.111+0000 7f8546ffd640 1 --2- 172.21.15.169:0/613348412 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f85300a2880 0x7f85300a4d40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f853c02c7b0 tx=0x7f853c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:53.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.116+0000 7f8544ff9640 1 -- 172.21.15.169:0/613348412 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f853809bce0 con 0x7f85481005a0 2024-03-28T00:54:53.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.321+0000 7f854de75640 1 -- 172.21.15.169:0/613348412 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f85480fdd20 con 0x7f85300a2880 2024-03-28T00:54:53.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.329+0000 7f8544ff9640 1 -- 172.21.15.169:0/613348412 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f85480fdd20 con 0x7f85300a2880 2024-03-28T00:54:53.332 INFO:teuthology.orchestra.run.smithi169.stdout:/volumes/qa/sv_0/d69e00e6-09a9-4c58-9e86-0a77c2669635 2024-03-28T00:54:53.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.331+0000 7f854de75640 1 -- 172.21.15.169:0/613348412 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f85300a2880 msgr2=0x7f85300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:53.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.331+0000 7f854de75640 1 --2- 172.21.15.169:0/613348412 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f85300a2880 0x7f85300a4d40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f853c02c7b0 tx=0x7f853c005fd0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.331+0000 7f854de75640 1 -- 172.21.15.169:0/613348412 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85481005a0 msgr2=0x7f85481979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:53.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.331+0000 7f854de75640 1 --2- 172.21.15.169:0/613348412 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85481005a0 0x7f85481979b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f853800e9c0 tx=0x7f853800ee80 comp rx=0 tx=0).stop 2024-03-28T00:54:53.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.331+0000 7f854de75640 1 -- 172.21.15.169:0/613348412 shutdown_connections 2024-03-28T00:54:53.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.331+0000 7f854de75640 1 --2- 172.21.15.169:0/613348412 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f85300a2880 0x7f85300a4d40 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.331+0000 7f854de75640 1 --2- 172.21.15.169:0/613348412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8548102150 0x7f854819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.331+0000 7f854de75640 1 --2- 172.21.15.169:0/613348412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8548101790 0x7f8548197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.331+0000 7f854de75640 1 --2- 172.21.15.169:0/613348412 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85481005a0 0x7f85481979b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.331+0000 7f854de75640 1 -- 172.21.15.169:0/613348412 >> 172.21.15.169:0/613348412 conn(0x7f85480fbb30 msgr2=0x7f85480fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:53.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.331+0000 7f854de75640 1 -- 172.21.15.169:0/613348412 shutdown_connections 2024-03-28T00:54:53.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.331+0000 7f854de75640 1 -- 172.21.15.169:0/613348412 wait complete. 2024-03-28T00:54:53.345 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-28T00:54:53.455 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.449+0000 7f3900e34640 1 -- 172.21.15.169:0/1753913152 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38fc101800 msgr2=0x7f38fc101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:53.455 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.449+0000 7f3900e34640 1 --2- 172.21.15.169:0/1753913152 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38fc101800 0x7f38fc101c80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f38f0009900 tx=0x7f38f002bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:53.455 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.450+0000 7f3900e34640 1 -- 172.21.15.169:0/1753913152 shutdown_connections 2024-03-28T00:54:53.455 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.450+0000 7f3900e34640 1 --2- 172.21.15.169:0/1753913152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38fc1021c0 0x7f38fc108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.455 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.450+0000 7f3900e34640 1 --2- 172.21.15.169:0/1753913152 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38fc101800 0x7f38fc101c80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.455 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.450+0000 7f3900e34640 1 --2- 172.21.15.169:0/1753913152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38fc100af0 0x7f38fc100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.455 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.450+0000 7f3900e34640 1 -- 172.21.15.169:0/1753913152 >> 172.21.15.169:0/1753913152 conn(0x7f38fc0fb2a0 msgr2=0x7f38fc0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:53.455 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.450+0000 7f3900e34640 1 -- 172.21.15.169:0/1753913152 shutdown_connections 2024-03-28T00:54:53.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.450+0000 7f3900e34640 1 -- 172.21.15.169:0/1753913152 wait complete. 2024-03-28T00:54:53.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.450+0000 7f3900e34640 1 Processor -- start 2024-03-28T00:54:53.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.450+0000 7f3900e34640 1 -- start start 2024-03-28T00:54:53.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.450+0000 7f3900e34640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38fc100af0 0x7f38fc197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:53.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.450+0000 7f3900e34640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38fc101800 0x7f38fc198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:53.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.450+0000 7f3900e34640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38fc1021c0 0x7f38fc19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:53.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.450+0000 7f3900e34640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc10c240 con 0x7f38fc101800 2024-03-28T00:54:53.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.450+0000 7f3900e34640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc10c0c0 con 0x7f38fc1021c0 2024-03-28T00:54:53.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.450+0000 7f3900e34640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc10c3c0 con 0x7f38fc100af0 2024-03-28T00:54:53.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.451+0000 7f38f9d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38fc101800 0x7f38fc198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:53.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.451+0000 7f38f9d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38fc101800 0x7f38fc198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38884/0 (socket says 172.21.15.169:38884) 2024-03-28T00:54:53.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.451+0000 7f38f9d9b640 1 -- 172.21.15.169:0/2262606659 learned_addr learned my addr 172.21.15.169:0/2262606659 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:53.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.451+0000 7f38fa59c640 1 --2- 172.21.15.169:0/2262606659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38fc100af0 0x7f38fc197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:53.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.451+0000 7f38fad9d640 1 --2- 172.21.15.169:0/2262606659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38fc1021c0 0x7f38fc19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:53.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.451+0000 7f38f9d9b640 1 -- 172.21.15.169:0/2262606659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38fc100af0 msgr2=0x7f38fc197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:53.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.451+0000 7f38f9d9b640 1 --2- 172.21.15.169:0/2262606659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38fc100af0 0x7f38fc197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.451+0000 7f38f9d9b640 1 -- 172.21.15.169:0/2262606659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38fc1021c0 msgr2=0x7f38fc19f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:53.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.451+0000 7f38f9d9b640 1 --2- 172.21.15.169:0/2262606659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38fc1021c0 0x7f38fc19f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.451+0000 7f38f9d9b640 1 -- 172.21.15.169:0/2262606659 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38fc19f6a0 con 0x7f38fc101800 2024-03-28T00:54:53.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.451+0000 7f38f9d9b640 1 --2- 172.21.15.169:0/2262606659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38fc101800 0x7f38fc198030 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f38f0005ab0 tx=0x7f38f002c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:53.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.451+0000 7f38eb7fe640 1 -- 172.21.15.169:0/2262606659 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f0041070 con 0x7f38fc101800 2024-03-28T00:54:53.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.451+0000 7f38eb7fe640 1 -- 172.21.15.169:0/2262606659 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38f0004450 con 0x7f38fc101800 2024-03-28T00:54:53.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.451+0000 7f38eb7fe640 1 -- 172.21.15.169:0/2262606659 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f0003e50 con 0x7f38fc101800 2024-03-28T00:54:53.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.451+0000 7f3900e34640 1 -- 172.21.15.169:0/2262606659 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38fc19f930 con 0x7f38fc101800 2024-03-28T00:54:53.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.451+0000 7f3900e34640 1 -- 172.21.15.169:0/2262606659 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38fc19fe20 con 0x7f38fc101800 2024-03-28T00:54:53.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.452+0000 7f3900e34640 1 -- 172.21.15.169:0/2262606659 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38fc19fff0 con 0x7f38fc101800 2024-03-28T00:54:53.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.453+0000 7f38eb7fe640 1 -- 172.21.15.169:0/2262606659 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f38f0044070 con 0x7f38fc101800 2024-03-28T00:54:53.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.454+0000 7f38eb7fe640 1 --2- 172.21.15.169:0/2262606659 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f38e40a29a0 0x7f38e40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:53.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.454+0000 7f38fa59c640 1 --2- 172.21.15.169:0/2262606659 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f38e40a29a0 0x7f38e40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:53.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.454+0000 7f38eb7fe640 1 -- 172.21.15.169:0/2262606659 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f38f00f0840 con 0x7f38fc101800 2024-03-28T00:54:53.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.454+0000 7f38fa59c640 1 --2- 172.21.15.169:0/2262606659 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f38e40a29a0 0x7f38e40a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f38ec0095f0 tx=0x7f38ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:53.461 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.459+0000 7f38eb7fe640 1 -- 172.21.15.169:0/2262606659 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f38f00bd050 con 0x7f38fc101800 2024-03-28T00:54:53.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:53 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/478819451' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T00:54:53.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:53 smithi169 ceph-mon[32274]: mds.? [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] up:active 2024-03-28T00:54:53.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:53 smithi169 ceph-mon[32274]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-28T00:54:53.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:53 smithi169 ceph-mon[32274]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T00:54:53.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:53 smithi169 ceph-mon[32274]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:54:53.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:53 smithi169 ceph-mon[32274]: from='client.24688 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:54:53.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:53 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/478819451' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T00:54:53.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:53 smithi174 ceph-mon[36060]: mds.? [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] up:active 2024-03-28T00:54:53.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:53 smithi174 ceph-mon[36060]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-28T00:54:53.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:53 smithi174 ceph-mon[36060]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T00:54:53.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:53 smithi174 ceph-mon[36060]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:54:53.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:53 smithi174 ceph-mon[36060]: from='client.24688 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:54:53.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.661+0000 7f3900e34640 1 -- 172.21.15.169:0/2262606659 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f38fc0fd4a0 con 0x7f38e40a29a0 2024-03-28T00:54:53.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:53 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/478819451' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T00:54:53.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:53 smithi177 ceph-mon[36206]: mds.? [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] up:active 2024-03-28T00:54:53.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:53 smithi177 ceph-mon[36206]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-28T00:54:53.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:53 smithi177 ceph-mon[36206]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T00:54:53.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:53 smithi177 ceph-mon[36206]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:54:53.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:53 smithi177 ceph-mon[36206]: from='client.24688 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:54:53.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.686+0000 7f38eb7fe640 1 -- 172.21.15.169:0/2262606659 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f38fc0fd4a0 con 0x7f38e40a29a0 2024-03-28T00:54:53.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.688+0000 7f3900e34640 1 -- 172.21.15.169:0/2262606659 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f38e40a29a0 msgr2=0x7f38e40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.688+0000 7f3900e34640 1 --2- 172.21.15.169:0/2262606659 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f38e40a29a0 0x7f38e40a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f38ec0095f0 tx=0x7f38ec005f90 comp rx=0 tx=0).stop 2024-03-28T00:54:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.688+0000 7f3900e34640 1 -- 172.21.15.169:0/2262606659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38fc101800 msgr2=0x7f38fc198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.688+0000 7f3900e34640 1 --2- 172.21.15.169:0/2262606659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38fc101800 0x7f38fc198030 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f38f0005ab0 tx=0x7f38f002c6c0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.688+0000 7f3900e34640 1 -- 172.21.15.169:0/2262606659 shutdown_connections 2024-03-28T00:54:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.688+0000 7f3900e34640 1 --2- 172.21.15.169:0/2262606659 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f38e40a29a0 0x7f38e40a4e60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.688+0000 7f3900e34640 1 --2- 172.21.15.169:0/2262606659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38fc1021c0 0x7f38fc19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.688+0000 7f3900e34640 1 --2- 172.21.15.169:0/2262606659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38fc101800 0x7f38fc198030 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.688+0000 7f3900e34640 1 --2- 172.21.15.169:0/2262606659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38fc100af0 0x7f38fc197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.688+0000 7f3900e34640 1 -- 172.21.15.169:0/2262606659 >> 172.21.15.169:0/2262606659 conn(0x7f38fc0fb2a0 msgr2=0x7f38fc0fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.688+0000 7f3900e34640 1 -- 172.21.15.169:0/2262606659 shutdown_connections 2024-03-28T00:54:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.688+0000 7f3900e34640 1 -- 172.21.15.169:0/2262606659 wait complete. 2024-03-28T00:54:53.702 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-28T00:54:53.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.807+0000 7f4990a7e640 1 -- 172.21.15.169:0/2068574618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f498c1005a0 msgr2=0x7f498c1009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:53.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.807+0000 7f4990a7e640 1 --2- 172.21.15.169:0/2068574618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f498c1005a0 0x7f498c1009c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f497c009860 tx=0x7f497c02bb70 comp rx=0 tx=0).stop 2024-03-28T00:54:53.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.807+0000 7f4990a7e640 1 -- 172.21.15.169:0/2068574618 shutdown_connections 2024-03-28T00:54:53.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.807+0000 7f4990a7e640 1 --2- 172.21.15.169:0/2068574618 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f498c102150 0x7f498c1089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.807+0000 7f4990a7e640 1 --2- 172.21.15.169:0/2068574618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f498c101790 0x7f498c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.807+0000 7f4990a7e640 1 --2- 172.21.15.169:0/2068574618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f498c1005a0 0x7f498c1009c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.807+0000 7f4990a7e640 1 -- 172.21.15.169:0/2068574618 >> 172.21.15.169:0/2068574618 conn(0x7f498c0fbb30 msgr2=0x7f498c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:53.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.807+0000 7f4990a7e640 1 -- 172.21.15.169:0/2068574618 shutdown_connections 2024-03-28T00:54:53.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.807+0000 7f4990a7e640 1 -- 172.21.15.169:0/2068574618 wait complete. 2024-03-28T00:54:53.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.807+0000 7f4990a7e640 1 Processor -- start 2024-03-28T00:54:53.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.807+0000 7f4990a7e640 1 -- start start 2024-03-28T00:54:53.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f4990a7e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f498c1005a0 0x7f498c069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:53.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f4990a7e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f498c101790 0x7f498c19c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:53.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f4990a7e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f498c102150 0x7f498c19edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:53.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f4990a7e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498c10c230 con 0x7f498c101790 2024-03-28T00:54:53.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f4990a7e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498c10c0b0 con 0x7f498c1005a0 2024-03-28T00:54:53.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f4990a7e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498c10c3b0 con 0x7f498c102150 2024-03-28T00:54:53.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f4989d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f498c101790 0x7f498c19c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:53.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f4989d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f498c101790 0x7f498c19c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38900/0 (socket says 172.21.15.169:38900) 2024-03-28T00:54:53.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f4989d9b640 1 -- 172.21.15.169:0/2592328735 learned_addr learned my addr 172.21.15.169:0/2592328735 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:53.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f498ad9d640 1 --2- 172.21.15.169:0/2592328735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f498c102150 0x7f498c19edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:53.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f498a59c640 1 --2- 172.21.15.169:0/2592328735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f498c1005a0 0x7f498c069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:53.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f4989d9b640 1 -- 172.21.15.169:0/2592328735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f498c102150 msgr2=0x7f498c19edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:53.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f4989d9b640 1 --2- 172.21.15.169:0/2592328735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f498c102150 0x7f498c19edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f4989d9b640 1 -- 172.21.15.169:0/2592328735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f498c1005a0 msgr2=0x7f498c069390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:53.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f4989d9b640 1 --2- 172.21.15.169:0/2592328735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f498c1005a0 0x7f498c069390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:53.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f4989d9b640 1 -- 172.21.15.169:0/2592328735 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498c19f690 con 0x7f498c101790 2024-03-28T00:54:53.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f498a59c640 1 --2- 172.21.15.169:0/2592328735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f498c1005a0 0x7f498c069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:53.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f4989d9b640 1 --2- 172.21.15.169:0/2592328735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f498c101790 0x7f498c19c790 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f498000e9f0 tx=0x7f498000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:53.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f497b7fe640 1 -- 172.21.15.169:0/2592328735 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498000cd80 con 0x7f498c101790 2024-03-28T00:54:53.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f497b7fe640 1 -- 172.21.15.169:0/2592328735 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4980005500 con 0x7f498c101790 2024-03-28T00:54:53.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f4990a7e640 1 -- 172.21.15.169:0/2592328735 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f498c19f980 con 0x7f498c101790 2024-03-28T00:54:53.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f497b7fe640 1 -- 172.21.15.169:0/2592328735 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4980010690 con 0x7f498c101790 2024-03-28T00:54:53.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.808+0000 7f4990a7e640 1 -- 172.21.15.169:0/2592328735 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f498c19fe80 con 0x7f498c101790 2024-03-28T00:54:53.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.809+0000 7f4990a7e640 1 -- 172.21.15.169:0/2592328735 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f498c107980 con 0x7f498c101790 2024-03-28T00:54:53.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.810+0000 7f497b7fe640 1 -- 172.21.15.169:0/2592328735 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f49800046c0 con 0x7f498c101790 2024-03-28T00:54:53.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.811+0000 7f497b7fe640 1 --2- 172.21.15.169:0/2592328735 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f49740a2800 0x7f49740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:53.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.811+0000 7f498a59c640 1 --2- 172.21.15.169:0/2592328735 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f49740a2800 0x7f49740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:53.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.811+0000 7f497b7fe640 1 -- 172.21.15.169:0/2592328735 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f49800cf270 con 0x7f498c101790 2024-03-28T00:54:53.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.812+0000 7f498a59c640 1 --2- 172.21.15.169:0/2592328735 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f49740a2800 0x7f49740a4cc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f497c009830 tx=0x7f497c02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:53.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:53.814+0000 7f497b7fe640 1 -- 172.21.15.169:0/2592328735 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4980053740 con 0x7f498c101790 2024-03-28T00:54:54.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.018+0000 7f4990a7e640 1 -- 172.21.15.169:0/2592328735 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f498c061ae0 con 0x7f49740a2800 2024-03-28T00:54:54.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.025+0000 7f497b7fe640 1 -- 172.21.15.169:0/2592328735 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f498c061ae0 con 0x7f49740a2800 2024-03-28T00:54:54.029 INFO:teuthology.orchestra.run.smithi169.stdout:/volumes/qa/sv_1/42e8c166-c61f-4c6b-9595-a337acbab519 2024-03-28T00:54:54.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.028+0000 7f4990a7e640 1 -- 172.21.15.169:0/2592328735 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f49740a2800 msgr2=0x7f49740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:54.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.028+0000 7f4990a7e640 1 --2- 172.21.15.169:0/2592328735 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f49740a2800 0x7f49740a4cc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f497c009830 tx=0x7f497c02e040 comp rx=0 tx=0).stop 2024-03-28T00:54:54.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.028+0000 7f4990a7e640 1 -- 172.21.15.169:0/2592328735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f498c101790 msgr2=0x7f498c19c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:54.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.028+0000 7f4990a7e640 1 --2- 172.21.15.169:0/2592328735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f498c101790 0x7f498c19c790 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f498000e9f0 tx=0x7f498000edf0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.028+0000 7f4990a7e640 1 -- 172.21.15.169:0/2592328735 shutdown_connections 2024-03-28T00:54:54.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.028+0000 7f4990a7e640 1 --2- 172.21.15.169:0/2592328735 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f49740a2800 0x7f49740a4cc0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.028+0000 7f4990a7e640 1 --2- 172.21.15.169:0/2592328735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f498c102150 0x7f498c19edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.028+0000 7f4990a7e640 1 --2- 172.21.15.169:0/2592328735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f498c101790 0x7f498c19c790 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.028+0000 7f4990a7e640 1 --2- 172.21.15.169:0/2592328735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f498c1005a0 0x7f498c069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.028+0000 7f4990a7e640 1 -- 172.21.15.169:0/2592328735 >> 172.21.15.169:0/2592328735 conn(0x7f498c0fbb30 msgr2=0x7f498c104b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:54.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.028+0000 7f4990a7e640 1 -- 172.21.15.169:0/2592328735 shutdown_connections 2024-03-28T00:54:54.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.028+0000 7f4990a7e640 1 -- 172.21.15.169:0/2592328735 wait complete. 2024-03-28T00:54:54.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:54:54.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.141+0000 7f2603bd4640 1 -- 172.21.15.169:0/586630085 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25fc107420 msgr2=0x7f25fc109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:54.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.141+0000 7f2603bd4640 1 --2- 172.21.15.169:0/586630085 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25fc107420 0x7f25fc109870 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f25f4009ca0 tx=0x7f25f402c0f0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.141+0000 7f2603bd4640 1 -- 172.21.15.169:0/586630085 shutdown_connections 2024-03-28T00:54:54.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.141+0000 7f2603bd4640 1 --2- 172.21.15.169:0/586630085 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25fc107420 0x7f25fc109870 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.141+0000 7f2603bd4640 1 --2- 172.21.15.169:0/586630085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25fc104af0 0x7f25fc106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.141+0000 7f2603bd4640 1 --2- 172.21.15.169:0/586630085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25fc1021c0 0x7f25fc1045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.141+0000 7f2603bd4640 1 -- 172.21.15.169:0/586630085 >> 172.21.15.169:0/586630085 conn(0x7f25fc0fbbd0 msgr2=0x7f25fc0fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:54.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.141+0000 7f2603bd4640 1 -- 172.21.15.169:0/586630085 shutdown_connections 2024-03-28T00:54:54.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.141+0000 7f2603bd4640 1 -- 172.21.15.169:0/586630085 wait complete. 2024-03-28T00:54:54.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.142+0000 7f2603bd4640 1 Processor -- start 2024-03-28T00:54:54.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.142+0000 7f2603bd4640 1 -- start start 2024-03-28T00:54:54.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.142+0000 7f2603bd4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25fc1021c0 0x7f25fc197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:54.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.142+0000 7f2603bd4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25fc104af0 0x7f25fc197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:54.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.142+0000 7f2603bd4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25fc107420 0x7f25fc19ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:54.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.142+0000 7f2603bd4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc10d350 con 0x7f25fc107420 2024-03-28T00:54:54.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.142+0000 7f2603bd4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc10d1d0 con 0x7f25fc104af0 2024-03-28T00:54:54.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.142+0000 7f2603bd4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc10d4d0 con 0x7f25fc1021c0 2024-03-28T00:54:54.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.142+0000 7f2602171640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25fc107420 0x7f25fc19ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:54.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.142+0000 7f2602171640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25fc107420 0x7f25fc19ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38916/0 (socket says 172.21.15.169:38916) 2024-03-28T00:54:54.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.142+0000 7f2601970640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25fc1021c0 0x7f25fc197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:54.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.142+0000 7f2602171640 1 -- 172.21.15.169:0/3356206730 learned_addr learned my addr 172.21.15.169:0/3356206730 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:54.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.142+0000 7f260116f640 1 --2- 172.21.15.169:0/3356206730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25fc104af0 0x7f25fc197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:54.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.142+0000 7f2602171640 1 -- 172.21.15.169:0/3356206730 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25fc1021c0 msgr2=0x7f25fc197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:54.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.142+0000 7f2602171640 1 --2- 172.21.15.169:0/3356206730 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25fc1021c0 0x7f25fc197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.142+0000 7f2602171640 1 -- 172.21.15.169:0/3356206730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25fc104af0 msgr2=0x7f25fc197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:54.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.142+0000 7f2602171640 1 --2- 172.21.15.169:0/3356206730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25fc104af0 0x7f25fc197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.142+0000 7f2602171640 1 -- 172.21.15.169:0/3356206730 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25fc19f570 con 0x7f25fc107420 2024-03-28T00:54:54.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.143+0000 7f2602171640 1 --2- 172.21.15.169:0/3356206730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25fc107420 0x7f25fc19ef00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f25f4037040 tx=0x7f25f4032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:54.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.143+0000 7f25f2ffd640 1 -- 172.21.15.169:0/3356206730 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f4041020 con 0x7f25fc107420 2024-03-28T00:54:54.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.143+0000 7f25f2ffd640 1 -- 172.21.15.169:0/3356206730 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f25f4034cd0 con 0x7f25fc107420 2024-03-28T00:54:54.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.143+0000 7f2603bd4640 1 -- 172.21.15.169:0/3356206730 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25fc0fffb0 con 0x7f25fc107420 2024-03-28T00:54:54.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.143+0000 7f25f2ffd640 1 -- 172.21.15.169:0/3356206730 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f4039040 con 0x7f25fc107420 2024-03-28T00:54:54.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.143+0000 7f2603bd4640 1 -- 172.21.15.169:0/3356206730 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25fc1004a0 con 0x7f25fc107420 2024-03-28T00:54:54.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.144+0000 7f2603bd4640 1 -- 172.21.15.169:0/3356206730 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25fc0fd6c0 con 0x7f25fc107420 2024-03-28T00:54:54.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.145+0000 7f25f2ffd640 1 -- 172.21.15.169:0/3356206730 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f25f4044020 con 0x7f25fc107420 2024-03-28T00:54:54.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.145+0000 7f25f2ffd640 1 --2- 172.21.15.169:0/3356206730 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f25e80a2800 0x7f25e80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:54.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.146+0000 7f25f2ffd640 1 -- 172.21.15.169:0/3356206730 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f25f40f0f00 con 0x7f25fc107420 2024-03-28T00:54:54.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.146+0000 7f2601970640 1 --2- 172.21.15.169:0/3356206730 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f25e80a2800 0x7f25e80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:54.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.146+0000 7f2601970640 1 --2- 172.21.15.169:0/3356206730 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f25e80a2800 0x7f25e80a4cc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f25f80095f0 tx=0x7f25f8012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:54.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.149+0000 7f25f2ffd640 1 -- 172.21.15.169:0/3356206730 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f25f404b020 con 0x7f25fc107420 2024-03-28T00:54:54.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:54 smithi169 ceph-mon[32274]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:54:54.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:54 smithi169 ceph-mon[32274]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:54:54.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:54 smithi169 ceph-mon[32274]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:54:54.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:54 smithi169 ceph-mon[32274]: pgmap v299: 129 pgs: 129 active+clean; 583 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-28T00:54:54.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.367+0000 7f2603bd4640 1 -- 172.21.15.169:0/3356206730 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25fc02d230 con 0x7f25fc107420 2024-03-28T00:54:54.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.368+0000 7f25f2ffd640 1 -- 172.21.15.169:0/3356206730 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11420 (secure 0 0 0) 0x7f25fc02d230 con 0x7f25fc107420 2024-03-28T00:54:54.371 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:54.371 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:54:52.274335+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24457,24478]},"id":1}]} 2024-03-28T00:54:54.371 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 24 2024-03-28T00:54:54.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.370+0000 7f2603bd4640 1 -- 172.21.15.169:0/3356206730 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f25e80a2800 msgr2=0x7f25e80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:54.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.370+0000 7f2603bd4640 1 --2- 172.21.15.169:0/3356206730 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f25e80a2800 0x7f25e80a4cc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f25f80095f0 tx=0x7f25f8012040 comp rx=0 tx=0).stop 2024-03-28T00:54:54.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.370+0000 7f2603bd4640 1 -- 172.21.15.169:0/3356206730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25fc107420 msgr2=0x7f25fc19ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:54.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.370+0000 7f2603bd4640 1 --2- 172.21.15.169:0/3356206730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25fc107420 0x7f25fc19ef00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f25f4037040 tx=0x7f25f4032f40 comp rx=0 tx=0).stop 2024-03-28T00:54:54.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.370+0000 7f2603bd4640 1 -- 172.21.15.169:0/3356206730 shutdown_connections 2024-03-28T00:54:54.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.370+0000 7f2603bd4640 1 --2- 172.21.15.169:0/3356206730 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f25e80a2800 0x7f25e80a4cc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.370+0000 7f2603bd4640 1 --2- 172.21.15.169:0/3356206730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25fc107420 0x7f25fc19ef00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.370+0000 7f2603bd4640 1 --2- 172.21.15.169:0/3356206730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25fc104af0 0x7f25fc197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.370+0000 7f2603bd4640 1 --2- 172.21.15.169:0/3356206730 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25fc1021c0 0x7f25fc197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.370+0000 7f2603bd4640 1 -- 172.21.15.169:0/3356206730 >> 172.21.15.169:0/3356206730 conn(0x7f25fc0fbbd0 msgr2=0x7f25fc102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:54.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.370+0000 7f2603bd4640 1 -- 172.21.15.169:0/3356206730 shutdown_connections 2024-03-28T00:54:54.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.370+0000 7f2603bd4640 1 -- 172.21.15.169:0/3356206730 wait complete. 2024-03-28T00:54:54.384 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T00:54:54.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.483+0000 7f8e459b9640 1 -- 172.21.15.169:0/3015763170 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e40102a20 msgr2=0x7f8e40102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:54.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.483+0000 7f8e459b9640 1 --2- 172.21.15.169:0/3015763170 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e40102a20 0x7f8e40102ea0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8e28009930 tx=0x7f8e2802bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:54.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.483+0000 7f8e459b9640 1 -- 172.21.15.169:0/3015763170 shutdown_connections 2024-03-28T00:54:54.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.483+0000 7f8e459b9640 1 --2- 172.21.15.169:0/3015763170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e401033e0 0x7f8e40109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.483+0000 7f8e459b9640 1 --2- 172.21.15.169:0/3015763170 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e40102a20 0x7f8e40102ea0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.483+0000 7f8e459b9640 1 --2- 172.21.15.169:0/3015763170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e400ff9b0 0x7f8e400ffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.483+0000 7f8e459b9640 1 -- 172.21.15.169:0/3015763170 >> 172.21.15.169:0/3015763170 conn(0x7f8e400fb2c0 msgr2=0x7f8e400fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:54.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.483+0000 7f8e459b9640 1 -- 172.21.15.169:0/3015763170 shutdown_connections 2024-03-28T00:54:54.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.483+0000 7f8e459b9640 1 -- 172.21.15.169:0/3015763170 wait complete. 2024-03-28T00:54:54.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.484+0000 7f8e459b9640 1 Processor -- start 2024-03-28T00:54:54.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.484+0000 7f8e459b9640 1 -- start start 2024-03-28T00:54:54.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.484+0000 7f8e459b9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e400ff9b0 0x7f8e40197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:54.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.484+0000 7f8e459b9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e40102a20 0x7f8e40197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:54.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.484+0000 7f8e459b9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e401033e0 0x7f8e4019efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:54.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.484+0000 7f8e459b9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4010d360 con 0x7f8e401033e0 2024-03-28T00:54:54.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.484+0000 7f8e459b9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4010d1e0 con 0x7f8e400ff9b0 2024-03-28T00:54:54.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.484+0000 7f8e459b9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4010d4e0 con 0x7f8e40102a20 2024-03-28T00:54:54.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.484+0000 7f8e3f7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e401033e0 0x7f8e4019efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:54.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.484+0000 7f8e3f7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e401033e0 0x7f8e4019efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38936/0 (socket says 172.21.15.169:38936) 2024-03-28T00:54:54.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.484+0000 7f8e3effd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e400ff9b0 0x7f8e40197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:54.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.484+0000 7f8e3f7fe640 1 -- 172.21.15.169:0/473636366 learned_addr learned my addr 172.21.15.169:0/473636366 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:54.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.484+0000 7f8e367fc640 1 --2- 172.21.15.169:0/473636366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e40102a20 0x7f8e40197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:54.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.485+0000 7f8e3f7fe640 1 -- 172.21.15.169:0/473636366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e40102a20 msgr2=0x7f8e40197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:54.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.485+0000 7f8e3f7fe640 1 --2- 172.21.15.169:0/473636366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e40102a20 0x7f8e40197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.485+0000 7f8e3f7fe640 1 -- 172.21.15.169:0/473636366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e400ff9b0 msgr2=0x7f8e40197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:54.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.485+0000 7f8e3f7fe640 1 --2- 172.21.15.169:0/473636366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e400ff9b0 0x7f8e40197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.485+0000 7f8e3f7fe640 1 -- 172.21.15.169:0/473636366 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4019f640 con 0x7f8e401033e0 2024-03-28T00:54:54.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.485+0000 7f8e3effd640 1 --2- 172.21.15.169:0/473636366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e400ff9b0 0x7f8e40197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:54.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.485+0000 7f8e3f7fe640 1 --2- 172.21.15.169:0/473636366 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e401033e0 0x7f8e4019efd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8e3800eb10 tx=0x7f8e3800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:54.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.485+0000 7f8e449b7640 1 -- 172.21.15.169:0/473636366 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e38004b20 con 0x7f8e401033e0 2024-03-28T00:54:54.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.485+0000 7f8e449b7640 1 -- 172.21.15.169:0/473636366 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e38004cc0 con 0x7f8e401033e0 2024-03-28T00:54:54.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.485+0000 7f8e459b9640 1 -- 172.21.15.169:0/473636366 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e4019f930 con 0x7f8e401033e0 2024-03-28T00:54:54.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.485+0000 7f8e449b7640 1 -- 172.21.15.169:0/473636366 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e38010760 con 0x7f8e401033e0 2024-03-28T00:54:54.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.485+0000 7f8e459b9640 1 -- 172.21.15.169:0/473636366 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e4019fe00 con 0x7f8e401033e0 2024-03-28T00:54:54.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.486+0000 7f8e459b9640 1 -- 172.21.15.169:0/473636366 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e40108c20 con 0x7f8e401033e0 2024-03-28T00:54:54.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.487+0000 7f8e449b7640 1 -- 172.21.15.169:0/473636366 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8e38010900 con 0x7f8e401033e0 2024-03-28T00:54:54.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.487+0000 7f8e449b7640 1 --2- 172.21.15.169:0/473636366 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8e2c0a2800 0x7f8e2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:54.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.488+0000 7f8e3effd640 1 --2- 172.21.15.169:0/473636366 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8e2c0a2800 0x7f8e2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:54.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.488+0000 7f8e449b7640 1 -- 172.21.15.169:0/473636366 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f8e380cf440 con 0x7f8e401033e0 2024-03-28T00:54:54.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.488+0000 7f8e3effd640 1 --2- 172.21.15.169:0/473636366 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8e2c0a2800 0x7f8e2c0a4cc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8e40069a10 tx=0x7f8e30005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:54.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.491+0000 7f8e449b7640 1 -- 172.21.15.169:0/473636366 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8e38053740 con 0x7f8e401033e0 2024-03-28T00:54:54.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:54 smithi174 ceph-mon[36060]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:54:54.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:54 smithi174 ceph-mon[36060]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:54:54.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:54 smithi174 ceph-mon[36060]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:54:54.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:54 smithi174 ceph-mon[36060]: pgmap v299: 129 pgs: 129 active+clean; 583 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-28T00:54:54.608 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 00:54:54 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-b[66103]: 2024-03-28T00:54:54.139+0000 7f2508032700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T00:54:54.608 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 00:54:54 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-b[66103]: 2024-03-28T00:54:54.139+0000 7f2508032700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-28T00:54:54.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:54 smithi177 ceph-mon[36206]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:54:54.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:54 smithi177 ceph-mon[36206]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:54:54.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:54 smithi177 ceph-mon[36206]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:54:54.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:54 smithi177 ceph-mon[36206]: pgmap v299: 129 pgs: 129 active+clean; 583 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-28T00:54:54.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.669+0000 7f8e459b9640 1 -- 172.21.15.169:0/473636366 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e4002d270 con 0x7f8e401033e0 2024-03-28T00:54:54.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.670+0000 7f8e449b7640 1 -- 172.21.15.169:0/473636366 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24186 (secure 0 0 0) 0x7f8e3809c2a0 con 0x7f8e401033e0 2024-03-28T00:54:54.673 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:54.674 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":90,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:54:38.804421+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:54:54.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.672+0000 7f8e459b9640 1 -- 172.21.15.169:0/473636366 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8e2c0a2800 msgr2=0x7f8e2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:54.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.672+0000 7f8e459b9640 1 --2- 172.21.15.169:0/473636366 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8e2c0a2800 0x7f8e2c0a4cc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8e40069a10 tx=0x7f8e30005f90 comp rx=0 tx=0).stop 2024-03-28T00:54:54.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.672+0000 7f8e459b9640 1 -- 172.21.15.169:0/473636366 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e401033e0 msgr2=0x7f8e4019efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:54.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.672+0000 7f8e459b9640 1 --2- 172.21.15.169:0/473636366 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e401033e0 0x7f8e4019efd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8e3800eb10 tx=0x7f8e3800c2d0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.672+0000 7f8e459b9640 1 -- 172.21.15.169:0/473636366 shutdown_connections 2024-03-28T00:54:54.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.672+0000 7f8e459b9640 1 --2- 172.21.15.169:0/473636366 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8e2c0a2800 0x7f8e2c0a4cc0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.672+0000 7f8e459b9640 1 --2- 172.21.15.169:0/473636366 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8e401033e0 0x7f8e4019efd0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.672+0000 7f8e459b9640 1 --2- 172.21.15.169:0/473636366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e40102a20 0x7f8e40197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.672+0000 7f8e459b9640 1 --2- 172.21.15.169:0/473636366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e400ff9b0 0x7f8e40197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.672+0000 7f8e459b9640 1 -- 172.21.15.169:0/473636366 >> 172.21.15.169:0/473636366 conn(0x7f8e400fb2c0 msgr2=0x7f8e40105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:54.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.672+0000 7f8e459b9640 1 -- 172.21.15.169:0/473636366 shutdown_connections 2024-03-28T00:54:54.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.673+0000 7f8e459b9640 1 -- 172.21.15.169:0/473636366 wait complete. 2024-03-28T00:54:54.687 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T00:54:54.687 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T00:54:54.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.788+0000 7ff4b37b3640 1 -- 172.21.15.169:0/3065532971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4ac1005c0 msgr2=0x7ff4ac1009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:54.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.788+0000 7ff4b37b3640 1 --2- 172.21.15.169:0/3065532971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4ac1005c0 0x7ff4ac1009e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8009900 tx=0x7ff4a802bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:54.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.788+0000 7ff4b37b3640 1 -- 172.21.15.169:0/3065532971 shutdown_connections 2024-03-28T00:54:54.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.788+0000 7ff4b37b3640 1 --2- 172.21.15.169:0/3065532971 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4ac102170 0x7ff4ac108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.788+0000 7ff4b37b3640 1 --2- 172.21.15.169:0/3065532971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4ac1017b0 0x7ff4ac101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.788+0000 7ff4b37b3640 1 --2- 172.21.15.169:0/3065532971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4ac1005c0 0x7ff4ac1009e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.788+0000 7ff4b37b3640 1 -- 172.21.15.169:0/3065532971 >> 172.21.15.169:0/3065532971 conn(0x7ff4ac0fbb30 msgr2=0x7ff4ac0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:54.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.788+0000 7ff4b37b3640 1 -- 172.21.15.169:0/3065532971 shutdown_connections 2024-03-28T00:54:54.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.788+0000 7ff4b37b3640 1 -- 172.21.15.169:0/3065532971 wait complete. 2024-03-28T00:54:54.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.789+0000 7ff4b37b3640 1 Processor -- start 2024-03-28T00:54:54.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.789+0000 7ff4b37b3640 1 -- start start 2024-03-28T00:54:54.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.789+0000 7ff4b37b3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4ac1005c0 0x7ff4ac197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:54.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.789+0000 7ff4b37b3640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4ac1017b0 0x7ff4ac197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:54.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.789+0000 7ff4b37b3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4ac102170 0x7ff4ac19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:54.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.789+0000 7ff4b37b3640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac10c230 con 0x7ff4ac1017b0 2024-03-28T00:54:54.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.789+0000 7ff4b37b3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac10c0b0 con 0x7ff4ac102170 2024-03-28T00:54:54.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.789+0000 7ff4b37b3640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac10c3b0 con 0x7ff4ac1005c0 2024-03-28T00:54:54.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.789+0000 7ff4b0d4e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4ac1017b0 0x7ff4ac197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:54.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.789+0000 7ff4b154f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4ac1005c0 0x7ff4ac197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:54.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.789+0000 7ff4b1d50640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4ac102170 0x7ff4ac19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:54.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.789+0000 7ff4b0d4e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4ac1017b0 0x7ff4ac197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38964/0 (socket says 172.21.15.169:38964) 2024-03-28T00:54:54.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.789+0000 7ff4b0d4e640 1 -- 172.21.15.169:0/4166749450 learned_addr learned my addr 172.21.15.169:0/4166749450 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:54.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.789+0000 7ff4b154f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4ac1005c0 0x7ff4ac197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:42174/0 (socket says 172.21.15.169:42174) 2024-03-28T00:54:54.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.789+0000 7ff4b0d4e640 1 -- 172.21.15.169:0/4166749450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4ac1005c0 msgr2=0x7ff4ac197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:54.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.789+0000 7ff4b0d4e640 1 --2- 172.21.15.169:0/4166749450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4ac1005c0 0x7ff4ac197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.789+0000 7ff4b0d4e640 1 -- 172.21.15.169:0/4166749450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4ac102170 msgr2=0x7ff4ac19efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:54.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.789+0000 7ff4b0d4e640 1 --2- 172.21.15.169:0/4166749450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4ac102170 0x7ff4ac19efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.789+0000 7ff4b0d4e640 1 -- 172.21.15.169:0/4166749450 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ac19f630 con 0x7ff4ac1017b0 2024-03-28T00:54:54.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.789+0000 7ff4b154f640 1 --2- 172.21.15.169:0/4166749450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4ac1005c0 0x7ff4ac197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:54.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.789+0000 7ff4b1d50640 1 --2- 172.21.15.169:0/4166749450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4ac102170 0x7ff4ac19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:54:54.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.790+0000 7ff4b0d4e640 1 --2- 172.21.15.169:0/4166749450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4ac1017b0 0x7ff4ac197f50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff49c00e9f0 tx=0x7ff49c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:54.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.790+0000 7ff4a27fc640 1 -- 172.21.15.169:0/4166749450 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c00cd80 con 0x7ff4ac1017b0 2024-03-28T00:54:54.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.790+0000 7ff4b37b3640 1 -- 172.21.15.169:0/4166749450 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4ac19f920 con 0x7ff4ac1017b0 2024-03-28T00:54:54.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.790+0000 7ff4a27fc640 1 -- 172.21.15.169:0/4166749450 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff49c005500 con 0x7ff4ac1017b0 2024-03-28T00:54:54.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.790+0000 7ff4a27fc640 1 -- 172.21.15.169:0/4166749450 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c010690 con 0x7ff4ac1017b0 2024-03-28T00:54:54.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.790+0000 7ff4b37b3640 1 -- 172.21.15.169:0/4166749450 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ac19fe20 con 0x7ff4ac1017b0 2024-03-28T00:54:54.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.791+0000 7ff4b37b3640 1 -- 172.21.15.169:0/4166749450 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4ac1a00b0 con 0x7ff4ac1017b0 2024-03-28T00:54:54.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.792+0000 7ff4a27fc640 1 -- 172.21.15.169:0/4166749450 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff49c0046c0 con 0x7ff4ac1017b0 2024-03-28T00:54:54.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.792+0000 7ff4a27fc640 1 --2- 172.21.15.169:0/4166749450 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff4980a2800 0x7ff4980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:54.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.792+0000 7ff4b154f640 1 --2- 172.21.15.169:0/4166749450 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff4980a2800 0x7ff4980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:54.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.792+0000 7ff4a27fc640 1 -- 172.21.15.169:0/4166749450 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7ff49c0cf080 con 0x7ff4ac1017b0 2024-03-28T00:54:54.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.793+0000 7ff4b154f640 1 --2- 172.21.15.169:0/4166749450 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff4980a2800 0x7ff4980a4cc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8000c00 tx=0x7ff4a8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:54.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.797+0000 7ff4a27fc640 1 -- 172.21.15.169:0/4166749450 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff49c09bee0 con 0x7ff4ac1017b0 2024-03-28T00:54:54.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.969+0000 7ff4b37b3640 1 -- 172.21.15.169:0/4166749450 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4ac1a0280 con 0x7ff4980a2800 2024-03-28T00:54:54.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.978+0000 7ff4a27fc640 1 -- 172.21.15.169:0/4166749450 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340604 (secure 0 0 0) 0x7ff4ac1a0280 con 0x7ff4980a2800 2024-03-28T00:54:54.986 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:54.988 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-28T00:54:54.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.982+0000 7ff4b37b3640 1 -- 172.21.15.169:0/4166749450 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff4980a2800 msgr2=0x7ff4980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:54.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.982+0000 7ff4b37b3640 1 --2- 172.21.15.169:0/4166749450 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff4980a2800 0x7ff4980a4cc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8000c00 tx=0x7ff4a8005fd0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.982+0000 7ff4b37b3640 1 -- 172.21.15.169:0/4166749450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4ac1017b0 msgr2=0x7ff4ac197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:54.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.982+0000 7ff4b37b3640 1 --2- 172.21.15.169:0/4166749450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4ac1017b0 0x7ff4ac197f50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff49c00e9f0 tx=0x7ff49c00edf0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.982+0000 7ff4b37b3640 1 -- 172.21.15.169:0/4166749450 shutdown_connections 2024-03-28T00:54:54.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.982+0000 7ff4b37b3640 1 --2- 172.21.15.169:0/4166749450 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff4980a2800 0x7ff4980a4cc0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.982+0000 7ff4b37b3640 1 --2- 172.21.15.169:0/4166749450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4ac102170 0x7ff4ac19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.982+0000 7ff4b37b3640 1 --2- 172.21.15.169:0/4166749450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4ac1017b0 0x7ff4ac197f50 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.982+0000 7ff4b37b3640 1 --2- 172.21.15.169:0/4166749450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4ac1005c0 0x7ff4ac197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:54.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.982+0000 7ff4b37b3640 1 -- 172.21.15.169:0/4166749450 >> 172.21.15.169:0/4166749450 conn(0x7ff4ac0fbb30 msgr2=0x7ff4ac0fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:54.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.982+0000 7ff4b37b3640 1 -- 172.21.15.169:0/4166749450 shutdown_connections 2024-03-28T00:54:54.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:54.982+0000 7ff4b37b3640 1 -- 172.21.15.169:0/4166749450 wait complete. 2024-03-28T00:54:54.996 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-03-28T00:54:54.084888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596531,"num_objects":45,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":131,"num_read_kb":116,"num_write":275,"num_write_kb":2969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":220,"ondisk_log_size":220,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356752,"kb_used_data":11736,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585072,"statfs":{"total":1151940427776,"available":1151575113728,"internally_reserved":0,"allocated":12017664,"data_stored":7637183,"data_compressed":319182,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3753,"num_objects":21,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":49,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003184"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.675896+0000","last_change":"2024-03-28T00:54:32.809951+0000","last_active":"2024-03-28T00:54:39.675896+0000","last_peered":"2024-03-28T00:54:39.675896+0000","last_clean":"2024-03-28T00:54:39.675896+0000","last_became_active":"2024-03-28T00:54:32.808871+0000","last_became_peered":"2024-03-28T00:54:32.808871+0000","last_unstale":"2024-03-28T00:54:39.675896+0000","last_undegraded":"2024-03-28T00:54:39.675896+0000","last_fullsized":"2024-03-28T00:54:39.675896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:15:17.152487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511120+0000","last_change":"2024-03-28T00:54:32.816707+0000","last_active":"2024-03-28T00:54:39.511120+0000","last_peered":"2024-03-28T00:54:39.511120+0000","last_clean":"2024-03-28T00:54:39.511120+0000","last_became_active":"2024-03-28T00:54:32.816000+0000","last_became_peered":"2024-03-28T00:54:32.816000+0000","last_unstale":"2024-03-28T00:54:39.511120+0000","last_undegraded":"2024-03-28T00:54:39.511120+0000","last_fullsized":"2024-03-28T00:54:39.511120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:56:19.532808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510653+0000","last_change":"2024-03-28T00:54:32.817023+0000","last_active":"2024-03-28T00:54:39.510653+0000","last_peered":"2024-03-28T00:54:39.510653+0000","last_clean":"2024-03-28T00:54:39.510653+0000","last_became_active":"2024-03-28T00:54:32.816712+0000","last_became_peered":"2024-03-28T00:54:32.816712+0000","last_unstale":"2024-03-28T00:54:39.510653+0000","last_undegraded":"2024-03-28T00:54:39.510653+0000","last_fullsized":"2024-03-28T00:54:39.510653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:13:25.334294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511180+0000","last_change":"2024-03-28T00:54:32.807259+0000","last_active":"2024-03-28T00:54:39.511180+0000","last_peered":"2024-03-28T00:54:39.511180+0000","last_clean":"2024-03-28T00:54:39.511180+0000","last_became_active":"2024-03-28T00:54:32.806944+0000","last_became_peered":"2024-03-28T00:54:32.806944+0000","last_unstale":"2024-03-28T00:54:39.511180+0000","last_undegraded":"2024-03-28T00:54:39.511180+0000","last_fullsized":"2024-03-28T00:54:39.511180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:46:04.852351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510926+0000","last_change":"2024-03-28T00:54:32.807613+0000","last_active":"2024-03-28T00:54:39.510926+0000","last_peered":"2024-03-28T00:54:39.510926+0000","last_clean":"2024-03-28T00:54:39.510926+0000","last_became_active":"2024-03-28T00:54:32.807280+0000","last_became_peered":"2024-03-28T00:54:32.807280+0000","last_unstale":"2024-03-28T00:54:39.510926+0000","last_undegraded":"2024-03-28T00:54:39.510926+0000","last_fullsized":"2024-03-28T00:54:39.510926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:56:37.133675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.148314+0000","last_change":"2024-03-28T00:54:32.806804+0000","last_active":"2024-03-28T00:54:39.148314+0000","last_peered":"2024-03-28T00:54:39.148314+0000","last_clean":"2024-03-28T00:54:39.148314+0000","last_became_active":"2024-03-28T00:54:32.805343+0000","last_became_peered":"2024-03-28T00:54:32.805343+0000","last_unstale":"2024-03-28T00:54:39.148314+0000","last_undegraded":"2024-03-28T00:54:39.148314+0000","last_fullsized":"2024-03-28T00:54:39.148314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:31:44.849926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808333+0000","last_change":"2024-03-28T00:54:32.814513+0000","last_active":"2024-03-28T00:54:38.808333+0000","last_peered":"2024-03-28T00:54:38.808333+0000","last_clean":"2024-03-28T00:54:38.808333+0000","last_became_active":"2024-03-28T00:54:32.814165+0000","last_became_peered":"2024-03-28T00:54:32.814165+0000","last_unstale":"2024-03-28T00:54:38.808333+0000","last_undegraded":"2024-03-28T00:54:38.808333+0000","last_fullsized":"2024-03-28T00:54:38.808333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:36:36.160751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510529+0000","last_change":"2024-03-28T00:54:32.808058+0000","last_active":"2024-03-28T00:54:39.510529+0000","last_peered":"2024-03-28T00:54:39.510529+0000","last_clean":"2024-03-28T00:54:39.510529+0000","last_became_active":"2024-03-28T00:54:32.807755+0000","last_became_peered":"2024-03-28T00:54:32.807755+0000","last_unstale":"2024-03-28T00:54:39.510529+0000","last_undegraded":"2024-03-28T00:54:39.510529+0000","last_fullsized":"2024-03-28T00:54:39.510529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:08:42.707015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510705+0000","last_change":"2024-03-28T00:54:32.800667+0000","last_active":"2024-03-28T00:54:39.510705+0000","last_peered":"2024-03-28T00:54:39.510705+0000","last_clean":"2024-03-28T00:54:39.510705+0000","last_became_active":"2024-03-28T00:54:32.800332+0000","last_became_peered":"2024-03-28T00:54:32.800332+0000","last_unstale":"2024-03-28T00:54:39.510705+0000","last_undegraded":"2024-03-28T00:54:39.510705+0000","last_fullsized":"2024-03-28T00:54:39.510705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:41:27.577871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.148665+0000","last_change":"2024-03-28T00:54:32.807958+0000","last_active":"2024-03-28T00:54:39.148665+0000","last_peered":"2024-03-28T00:54:39.148665+0000","last_clean":"2024-03-28T00:54:39.148665+0000","last_became_active":"2024-03-28T00:54:32.807657+0000","last_became_peered":"2024-03-28T00:54:32.807657+0000","last_unstale":"2024-03-28T00:54:39.148665+0000","last_undegraded":"2024-03-28T00:54:39.148665+0000","last_fullsized":"2024-03-28T00:54:39.148665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:30:28.000690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808607+0000","last_change":"2024-03-28T00:54:32.811543+0000","last_active":"2024-03-28T00:54:38.808607+0000","last_peered":"2024-03-28T00:54:38.808607+0000","last_clean":"2024-03-28T00:54:38.808607+0000","last_became_active":"2024-03-28T00:54:32.811210+0000","last_became_peered":"2024-03-28T00:54:32.811210+0000","last_unstale":"2024-03-28T00:54:38.808607+0000","last_undegraded":"2024-03-28T00:54:38.808607+0000","last_fullsized":"2024-03-28T00:54:38.808607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:47:19.539107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511181+0000","last_change":"2024-03-28T00:54:32.812648+0000","last_active":"2024-03-28T00:54:39.511181+0000","last_peered":"2024-03-28T00:54:39.511181+0000","last_clean":"2024-03-28T00:54:39.511181+0000","last_became_active":"2024-03-28T00:54:32.812340+0000","last_became_peered":"2024-03-28T00:54:32.812340+0000","last_unstale":"2024-03-28T00:54:39.511181+0000","last_undegraded":"2024-03-28T00:54:39.511181+0000","last_fullsized":"2024-03-28T00:54:39.511181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:20:45.449953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.810970+0000","last_change":"2024-03-28T00:54:32.801807+0000","last_active":"2024-03-28T00:54:38.810970+0000","last_peered":"2024-03-28T00:54:38.810970+0000","last_clean":"2024-03-28T00:54:38.810970+0000","last_became_active":"2024-03-28T00:54:32.801498+0000","last_became_peered":"2024-03-28T00:54:32.801498+0000","last_unstale":"2024-03-28T00:54:38.810970+0000","last_undegraded":"2024-03-28T00:54:38.810970+0000","last_fullsized":"2024-03-28T00:54:38.810970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:35:35.827856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510434+0000","last_change":"2024-03-28T00:54:32.818033+0000","last_active":"2024-03-28T00:54:39.510434+0000","last_peered":"2024-03-28T00:54:39.510434+0000","last_clean":"2024-03-28T00:54:39.510434+0000","last_became_active":"2024-03-28T00:54:32.817309+0000","last_became_peered":"2024-03-28T00:54:32.817309+0000","last_unstale":"2024-03-28T00:54:39.510434+0000","last_undegraded":"2024-03-28T00:54:39.510434+0000","last_fullsized":"2024-03-28T00:54:39.510434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:39:42.682780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510512+0000","last_change":"2024-03-28T00:54:32.815224+0000","last_active":"2024-03-28T00:54:39.510512+0000","last_peered":"2024-03-28T00:54:39.510512+0000","last_clean":"2024-03-28T00:54:39.510512+0000","last_became_active":"2024-03-28T00:54:32.814936+0000","last_became_peered":"2024-03-28T00:54:32.814936+0000","last_unstale":"2024-03-28T00:54:39.510512+0000","last_undegraded":"2024-03-28T00:54:39.510512+0000","last_fullsized":"2024-03-28T00:54:39.510512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:08:40.200393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510035+0000","last_change":"2024-03-28T00:54:32.807342+0000","last_active":"2024-03-28T00:54:39.510035+0000","last_peered":"2024-03-28T00:54:39.510035+0000","last_clean":"2024-03-28T00:54:39.510035+0000","last_became_active":"2024-03-28T00:54:32.806978+0000","last_became_peered":"2024-03-28T00:54:32.806978+0000","last_unstale":"2024-03-28T00:54:39.510035+0000","last_undegraded":"2024-03-28T00:54:39.510035+0000","last_fullsized":"2024-03-28T00:54:39.510035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:17:58.599693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.675702+0000","last_change":"2024-03-28T00:54:32.810036+0000","last_active":"2024-03-28T00:54:39.675702+0000","last_peered":"2024-03-28T00:54:39.675702+0000","last_clean":"2024-03-28T00:54:39.675702+0000","last_became_active":"2024-03-28T00:54:32.809438+0000","last_became_peered":"2024-03-28T00:54:32.809438+0000","last_unstale":"2024-03-28T00:54:39.675702+0000","last_undegraded":"2024-03-28T00:54:39.675702+0000","last_fullsized":"2024-03-28T00:54:39.675702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:18:50.447474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.290406+0000","last_change":"2024-03-28T00:54:32.815398+0000","last_active":"2024-03-28T00:54:39.290406+0000","last_peered":"2024-03-28T00:54:39.290406+0000","last_clean":"2024-03-28T00:54:39.290406+0000","last_became_active":"2024-03-28T00:54:32.815045+0000","last_became_peered":"2024-03-28T00:54:32.815045+0000","last_unstale":"2024-03-28T00:54:39.290406+0000","last_undegraded":"2024-03-28T00:54:39.290406+0000","last_fullsized":"2024-03-28T00:54:39.290406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:40:29.060217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511901+0000","last_change":"2024-03-28T00:54:32.805736+0000","last_active":"2024-03-28T00:54:39.511901+0000","last_peered":"2024-03-28T00:54:39.511901+0000","last_clean":"2024-03-28T00:54:39.511901+0000","last_became_active":"2024-03-28T00:54:32.805435+0000","last_became_peered":"2024-03-28T00:54:32.805435+0000","last_unstale":"2024-03-28T00:54:39.511901+0000","last_undegraded":"2024-03-28T00:54:39.511901+0000","last_fullsized":"2024-03-28T00:54:39.511901+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:24:30.893583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511476+0000","last_change":"2024-03-28T00:54:36.824531+0000","last_active":"2024-03-28T00:54:39.511476+0000","last_peered":"2024-03-28T00:54:39.511476+0000","last_clean":"2024-03-28T00:54:39.511476+0000","last_became_active":"2024-03-28T00:54:36.824123+0000","last_became_peered":"2024-03-28T00:54:36.824123+0000","last_unstale":"2024-03-28T00:54:39.511476+0000","last_undegraded":"2024-03-28T00:54:39.511476+0000","last_fullsized":"2024-03-28T00:54:39.511476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:10:47.503171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.290598+0000","last_change":"2024-03-28T00:54:32.256226+0000","last_active":"2024-03-28T00:54:39.290598+0000","last_peered":"2024-03-28T00:54:39.290598+0000","last_clean":"2024-03-28T00:54:39.290598+0000","last_became_active":"2024-03-28T00:54:32.255921+0000","last_became_peered":"2024-03-28T00:54:32.255921+0000","last_unstale":"2024-03-28T00:54:39.290598+0000","last_undegraded":"2024-03-28T00:54:39.290598+0000","last_fullsized":"2024-03-28T00:54:39.290598+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:56:04.196311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511112+0000","last_change":"2024-03-28T00:54:32.808248+0000","last_active":"2024-03-28T00:54:39.511112+0000","last_peered":"2024-03-28T00:54:39.511112+0000","last_clean":"2024-03-28T00:54:39.511112+0000","last_became_active":"2024-03-28T00:54:32.807923+0000","last_became_peered":"2024-03-28T00:54:32.807923+0000","last_unstale":"2024-03-28T00:54:39.511112+0000","last_undegraded":"2024-03-28T00:54:39.511112+0000","last_fullsized":"2024-03-28T00:54:39.511112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:59:56.002564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510503+0000","last_change":"2024-03-28T00:54:36.830556+0000","last_active":"2024-03-28T00:54:39.510503+0000","last_peered":"2024-03-28T00:54:39.510503+0000","last_clean":"2024-03-28T00:54:39.510503+0000","last_became_active":"2024-03-28T00:54:36.830151+0000","last_became_peered":"2024-03-28T00:54:36.830151+0000","last_unstale":"2024-03-28T00:54:39.510503+0000","last_undegraded":"2024-03-28T00:54:39.510503+0000","last_fullsized":"2024-03-28T00:54:39.510503+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:19:23.073410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511484+0000","last_change":"2024-03-28T00:54:32.260635+0000","last_active":"2024-03-28T00:54:39.511484+0000","last_peered":"2024-03-28T00:54:39.511484+0000","last_clean":"2024-03-28T00:54:39.511484+0000","last_became_active":"2024-03-28T00:54:32.260084+0000","last_became_peered":"2024-03-28T00:54:32.260084+0000","last_unstale":"2024-03-28T00:54:39.511484+0000","last_undegraded":"2024-03-28T00:54:39.511484+0000","last_fullsized":"2024-03-28T00:54:39.511484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:06.858519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511232+0000","last_change":"2024-03-28T00:54:32.809901+0000","last_active":"2024-03-28T00:54:39.511232+0000","last_peered":"2024-03-28T00:54:39.511232+0000","last_clean":"2024-03-28T00:54:39.511232+0000","last_became_active":"2024-03-28T00:54:32.809557+0000","last_became_peered":"2024-03-28T00:54:32.809557+0000","last_unstale":"2024-03-28T00:54:39.511232+0000","last_undegraded":"2024-03-28T00:54:39.511232+0000","last_fullsized":"2024-03-28T00:54:39.511232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:05:31.363487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510608+0000","last_change":"2024-03-28T00:54:36.824661+0000","last_active":"2024-03-28T00:54:39.510608+0000","last_peered":"2024-03-28T00:54:39.510608+0000","last_clean":"2024-03-28T00:54:39.510608+0000","last_became_active":"2024-03-28T00:54:36.824223+0000","last_became_peered":"2024-03-28T00:54:36.824223+0000","last_unstale":"2024-03-28T00:54:39.510608+0000","last_undegraded":"2024-03-28T00:54:39.510608+0000","last_fullsized":"2024-03-28T00:54:39.510608+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:02:25.509248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511745+0000","last_change":"2024-03-28T00:54:32.256436+0000","last_active":"2024-03-28T00:54:39.511745+0000","last_peered":"2024-03-28T00:54:39.511745+0000","last_clean":"2024-03-28T00:54:39.511745+0000","last_became_active":"2024-03-28T00:54:32.256069+0000","last_became_peered":"2024-03-28T00:54:32.256069+0000","last_unstale":"2024-03-28T00:54:39.511745+0000","last_undegraded":"2024-03-28T00:54:39.511745+0000","last_fullsized":"2024-03-28T00:54:39.511745+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:43.688192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510343+0000","last_change":"2024-03-28T00:54:32.799269+0000","last_active":"2024-03-28T00:54:39.510343+0000","last_peered":"2024-03-28T00:54:39.510343+0000","last_clean":"2024-03-28T00:54:39.510343+0000","last_became_active":"2024-03-28T00:54:32.798848+0000","last_became_peered":"2024-03-28T00:54:32.798848+0000","last_unstale":"2024-03-28T00:54:39.510343+0000","last_undegraded":"2024-03-28T00:54:39.510343+0000","last_fullsized":"2024-03-28T00:54:39.510343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:56:20.318466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510479+0000","last_change":"2024-03-28T00:54:36.823153+0000","last_active":"2024-03-28T00:54:39.510479+0000","last_peered":"2024-03-28T00:54:39.510479+0000","last_clean":"2024-03-28T00:54:39.510479+0000","last_became_active":"2024-03-28T00:54:36.822817+0000","last_became_peered":"2024-03-28T00:54:36.822817+0000","last_unstale":"2024-03-28T00:54:39.510479+0000","last_undegraded":"2024-03-28T00:54:39.510479+0000","last_fullsized":"2024-03-28T00:54:39.510479+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:20:23.001599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511059+0000","last_change":"2024-03-28T00:54:32.250439+0000","last_active":"2024-03-28T00:54:39.511059+0000","last_peered":"2024-03-28T00:54:39.511059+0000","last_clean":"2024-03-28T00:54:39.511059+0000","last_became_active":"2024-03-28T00:54:32.250006+0000","last_became_peered":"2024-03-28T00:54:32.250006+0000","last_unstale":"2024-03-28T00:54:39.511059+0000","last_undegraded":"2024-03-28T00:54:39.511059+0000","last_fullsized":"2024-03-28T00:54:39.511059+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:31:38.957429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.809145+0000","last_change":"2024-03-28T00:54:32.806888+0000","last_active":"2024-03-28T00:54:38.809145+0000","last_peered":"2024-03-28T00:54:38.809145+0000","last_clean":"2024-03-28T00:54:38.809145+0000","last_became_active":"2024-03-28T00:54:32.806581+0000","last_became_peered":"2024-03-28T00:54:32.806581+0000","last_unstale":"2024-03-28T00:54:38.809145+0000","last_undegraded":"2024-03-28T00:54:38.809145+0000","last_fullsized":"2024-03-28T00:54:38.809145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:25:52.331819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511783+0000","last_change":"2024-03-28T00:54:36.824041+0000","last_active":"2024-03-28T00:54:39.511783+0000","last_peered":"2024-03-28T00:54:39.511783+0000","last_clean":"2024-03-28T00:54:39.511783+0000","last_became_active":"2024-03-28T00:54:36.823658+0000","last_became_peered":"2024-03-28T00:54:36.823658+0000","last_unstale":"2024-03-28T00:54:39.511783+0000","last_undegraded":"2024-03-28T00:54:39.511783+0000","last_fullsized":"2024-03-28T00:54:39.511783+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:11:03.236009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.273501+0000","last_change":"2024-03-28T00:54:32.257534+0000","last_active":"2024-03-28T00:54:50.273501+0000","last_peered":"2024-03-28T00:54:50.273501+0000","last_clean":"2024-03-28T00:54:50.273501+0000","last_became_active":"2024-03-28T00:54:32.256404+0000","last_became_peered":"2024-03-28T00:54:32.256404+0000","last_unstale":"2024-03-28T00:54:50.273501+0000","last_undegraded":"2024-03-28T00:54:50.273501+0000","last_fullsized":"2024-03-28T00:54:50.273501+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:39:38.329799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510814+0000","last_change":"2024-03-28T00:54:32.806099+0000","last_active":"2024-03-28T00:54:39.510814+0000","last_peered":"2024-03-28T00:54:39.510814+0000","last_clean":"2024-03-28T00:54:39.510814+0000","last_became_active":"2024-03-28T00:54:32.805811+0000","last_became_peered":"2024-03-28T00:54:32.805811+0000","last_unstale":"2024-03-28T00:54:39.510814+0000","last_undegraded":"2024-03-28T00:54:39.510814+0000","last_fullsized":"2024-03-28T00:54:39.510814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:42:32.820245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511596+0000","last_change":"2024-03-28T00:54:36.827198+0000","last_active":"2024-03-28T00:54:39.511596+0000","last_peered":"2024-03-28T00:54:39.511596+0000","last_clean":"2024-03-28T00:54:39.511596+0000","last_became_active":"2024-03-28T00:54:36.826804+0000","last_became_peered":"2024-03-28T00:54:36.826804+0000","last_unstale":"2024-03-28T00:54:39.511596+0000","last_undegraded":"2024-03-28T00:54:39.511596+0000","last_fullsized":"2024-03-28T00:54:39.511596+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:38:11.030404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:51.278214+0000","last_change":"2024-03-28T00:54:32.249394+0000","last_active":"2024-03-28T00:54:51.278214+0000","last_peered":"2024-03-28T00:54:51.278214+0000","last_clean":"2024-03-28T00:54:51.278214+0000","last_became_active":"2024-03-28T00:54:32.249089+0000","last_became_peered":"2024-03-28T00:54:32.249089+0000","last_unstale":"2024-03-28T00:54:51.278214+0000","last_undegraded":"2024-03-28T00:54:51.278214+0000","last_fullsized":"2024-03-28T00:54:51.278214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:05:01.720053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808932+0000","last_change":"2024-03-28T00:54:32.809133+0000","last_active":"2024-03-28T00:54:38.808932+0000","last_peered":"2024-03-28T00:54:38.808932+0000","last_clean":"2024-03-28T00:54:38.808932+0000","last_became_active":"2024-03-28T00:54:32.808841+0000","last_became_peered":"2024-03-28T00:54:32.808841+0000","last_unstale":"2024-03-28T00:54:38.808932+0000","last_undegraded":"2024-03-28T00:54:38.808932+0000","last_fullsized":"2024-03-28T00:54:38.808932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:19:37.859542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.675758+0000","last_change":"2024-03-28T00:54:36.824483+0000","last_active":"2024-03-28T00:54:39.675758+0000","last_peered":"2024-03-28T00:54:39.675758+0000","last_clean":"2024-03-28T00:54:39.675758+0000","last_became_active":"2024-03-28T00:54:36.823973+0000","last_became_peered":"2024-03-28T00:54:36.823973+0000","last_unstale":"2024-03-28T00:54:39.675758+0000","last_undegraded":"2024-03-28T00:54:39.675758+0000","last_fullsized":"2024-03-28T00:54:39.675758+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:01:06.251119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.810690+0000","last_change":"2024-03-28T00:54:32.251287+0000","last_active":"2024-03-28T00:54:38.810690+0000","last_peered":"2024-03-28T00:54:38.810690+0000","last_clean":"2024-03-28T00:54:38.810690+0000","last_became_active":"2024-03-28T00:54:32.250904+0000","last_became_peered":"2024-03-28T00:54:32.250904+0000","last_unstale":"2024-03-28T00:54:38.810690+0000","last_undegraded":"2024-03-28T00:54:38.810690+0000","last_fullsized":"2024-03-28T00:54:38.810690+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:18:09.051056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510973+0000","last_change":"2024-03-28T00:54:32.803124+0000","last_active":"2024-03-28T00:54:39.510973+0000","last_peered":"2024-03-28T00:54:39.510973+0000","last_clean":"2024-03-28T00:54:39.510973+0000","last_became_active":"2024-03-28T00:54:32.802825+0000","last_became_peered":"2024-03-28T00:54:32.802825+0000","last_unstale":"2024-03-28T00:54:39.510973+0000","last_undegraded":"2024-03-28T00:54:39.510973+0000","last_fullsized":"2024-03-28T00:54:39.510973+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:34:21.825314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511236+0000","last_change":"2024-03-28T00:54:36.832151+0000","last_active":"2024-03-28T00:54:39.511236+0000","last_peered":"2024-03-28T00:54:39.511236+0000","last_clean":"2024-03-28T00:54:39.511236+0000","last_became_active":"2024-03-28T00:54:36.831748+0000","last_became_peered":"2024-03-28T00:54:36.831748+0000","last_unstale":"2024-03-28T00:54:39.511236+0000","last_undegraded":"2024-03-28T00:54:39.511236+0000","last_fullsized":"2024-03-28T00:54:39.511236+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:45:32.284386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808616+0000","last_change":"2024-03-28T00:54:32.259048+0000","last_active":"2024-03-28T00:54:38.808616+0000","last_peered":"2024-03-28T00:54:38.808616+0000","last_clean":"2024-03-28T00:54:38.808616+0000","last_became_active":"2024-03-28T00:54:32.258038+0000","last_became_peered":"2024-03-28T00:54:32.258038+0000","last_unstale":"2024-03-28T00:54:38.808616+0000","last_undegraded":"2024-03-28T00:54:38.808616+0000","last_fullsized":"2024-03-28T00:54:38.808616+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:45:23.525516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.290042+0000","last_change":"2024-03-28T00:54:32.809771+0000","last_active":"2024-03-28T00:54:39.290042+0000","last_peered":"2024-03-28T00:54:39.290042+0000","last_clean":"2024-03-28T00:54:39.290042+0000","last_became_active":"2024-03-28T00:54:32.809467+0000","last_became_peered":"2024-03-28T00:54:32.809467+0000","last_unstale":"2024-03-28T00:54:39.290042+0000","last_undegraded":"2024-03-28T00:54:39.290042+0000","last_fullsized":"2024-03-28T00:54:39.290042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:30:15.440202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511045+0000","last_change":"2024-03-28T00:54:36.823313+0000","last_active":"2024-03-28T00:54:39.511045+0000","last_peered":"2024-03-28T00:54:39.511045+0000","last_clean":"2024-03-28T00:54:39.511045+0000","last_became_active":"2024-03-28T00:54:36.822923+0000","last_became_peered":"2024-03-28T00:54:36.822923+0000","last_unstale":"2024-03-28T00:54:39.511045+0000","last_undegraded":"2024-03-28T00:54:39.511045+0000","last_fullsized":"2024-03-28T00:54:39.511045+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:22:21.125028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.272884+0000","last_change":"2024-03-28T00:54:32.255084+0000","last_active":"2024-03-28T00:54:50.272884+0000","last_peered":"2024-03-28T00:54:50.272884+0000","last_clean":"2024-03-28T00:54:50.272884+0000","last_became_active":"2024-03-28T00:54:32.254744+0000","last_became_peered":"2024-03-28T00:54:32.254744+0000","last_unstale":"2024-03-28T00:54:50.272884+0000","last_undegraded":"2024-03-28T00:54:50.272884+0000","last_fullsized":"2024-03-28T00:54:50.272884+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:32:03.906826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510832+0000","last_change":"2024-03-28T00:54:32.810050+0000","last_active":"2024-03-28T00:54:39.510832+0000","last_peered":"2024-03-28T00:54:39.510832+0000","last_clean":"2024-03-28T00:54:39.510832+0000","last_became_active":"2024-03-28T00:54:32.809731+0000","last_became_peered":"2024-03-28T00:54:32.809731+0000","last_unstale":"2024-03-28T00:54:39.510832+0000","last_undegraded":"2024-03-28T00:54:39.510832+0000","last_fullsized":"2024-03-28T00:54:39.510832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:20:11.876731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.289968+0000","last_change":"2024-03-28T00:54:36.826224+0000","last_active":"2024-03-28T00:54:39.289968+0000","last_peered":"2024-03-28T00:54:39.289968+0000","last_clean":"2024-03-28T00:54:39.289968+0000","last_became_active":"2024-03-28T00:54:36.825853+0000","last_became_peered":"2024-03-28T00:54:36.825853+0000","last_unstale":"2024-03-28T00:54:39.289968+0000","last_undegraded":"2024-03-28T00:54:39.289968+0000","last_fullsized":"2024-03-28T00:54:39.289968+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:32:09.131482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.271148+0000","last_change":"2024-03-28T00:54:32.259159+0000","last_active":"2024-03-28T00:54:50.271148+0000","last_peered":"2024-03-28T00:54:50.271148+0000","last_clean":"2024-03-28T00:54:50.271148+0000","last_became_active":"2024-03-28T00:54:32.258708+0000","last_became_peered":"2024-03-28T00:54:32.258708+0000","last_unstale":"2024-03-28T00:54:50.271148+0000","last_undegraded":"2024-03-28T00:54:50.271148+0000","last_fullsized":"2024-03-28T00:54:50.271148+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:37:01.524878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511134+0000","last_change":"2024-03-28T00:54:32.814358+0000","last_active":"2024-03-28T00:54:39.511134+0000","last_peered":"2024-03-28T00:54:39.511134+0000","last_clean":"2024-03-28T00:54:39.511134+0000","last_became_active":"2024-03-28T00:54:32.814018+0000","last_became_peered":"2024-03-28T00:54:32.814018+0000","last_unstale":"2024-03-28T00:54:39.511134+0000","last_undegraded":"2024-03-28T00:54:39.511134+0000","last_fullsized":"2024-03-28T00:54:39.511134+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:24:38.287853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510938+0000","last_change":"2024-03-28T00:54:36.822161+0000","last_active":"2024-03-28T00:54:39.510938+0000","last_peered":"2024-03-28T00:54:39.510938+0000","last_clean":"2024-03-28T00:54:39.510938+0000","last_became_active":"2024-03-28T00:54:36.821750+0000","last_became_peered":"2024-03-28T00:54:36.821750+0000","last_unstale":"2024-03-28T00:54:39.510938+0000","last_undegraded":"2024-03-28T00:54:39.510938+0000","last_fullsized":"2024-03-28T00:54:39.510938+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:13:36.892453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.809075+0000","last_change":"2024-03-28T00:54:32.257038+0000","last_active":"2024-03-28T00:54:38.809075+0000","last_peered":"2024-03-28T00:54:38.809075+0000","last_clean":"2024-03-28T00:54:38.809075+0000","last_became_active":"2024-03-28T00:54:32.256702+0000","last_became_peered":"2024-03-28T00:54:32.256702+0000","last_unstale":"2024-03-28T00:54:38.809075+0000","last_undegraded":"2024-03-28T00:54:38.809075+0000","last_fullsized":"2024-03-28T00:54:38.809075+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:35:17.294116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510425+0000","last_change":"2024-03-28T00:54:32.813514+0000","last_active":"2024-03-28T00:54:39.510425+0000","last_peered":"2024-03-28T00:54:39.510425+0000","last_clean":"2024-03-28T00:54:39.510425+0000","last_became_active":"2024-03-28T00:54:32.813192+0000","last_became_peered":"2024-03-28T00:54:32.813192+0000","last_unstale":"2024-03-28T00:54:39.510425+0000","last_undegraded":"2024-03-28T00:54:39.510425+0000","last_fullsized":"2024-03-28T00:54:39.510425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:55:03.410818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.290178+0000","last_change":"2024-03-28T00:54:36.820995+0000","last_active":"2024-03-28T00:54:39.290178+0000","last_peered":"2024-03-28T00:54:39.290178+0000","last_clean":"2024-03-28T00:54:39.290178+0000","last_became_active":"2024-03-28T00:54:36.820628+0000","last_became_peered":"2024-03-28T00:54:36.820628+0000","last_unstale":"2024-03-28T00:54:39.290178+0000","last_undegraded":"2024-03-28T00:54:39.290178+0000","last_fullsized":"2024-03-28T00:54:39.290178+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:18:58.771527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'11","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:52.281220+0000","last_change":"2024-03-28T00:54:32.251092+0000","last_active":"2024-03-28T00:54:52.281220+0000","last_peered":"2024-03-28T00:54:52.281220+0000","last_clean":"2024-03-28T00:54:52.281220+0000","last_became_active":"2024-03-28T00:54:32.250759+0000","last_became_peered":"2024-03-28T00:54:32.250759+0000","last_unstale":"2024-03-28T00:54:52.281220+0000","last_undegraded":"2024-03-28T00:54:52.281220+0000","last_fullsized":"2024-03-28T00:54:52.281220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:57:30.283365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":12,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511534+0000","last_change":"2024-03-28T00:54:32.809580+0000","last_active":"2024-03-28T00:54:39.511534+0000","last_peered":"2024-03-28T00:54:39.511534+0000","last_clean":"2024-03-28T00:54:39.511534+0000","last_became_active":"2024-03-28T00:54:32.809238+0000","last_became_peered":"2024-03-28T00:54:32.809238+0000","last_unstale":"2024-03-28T00:54:39.511534+0000","last_undegraded":"2024-03-28T00:54:39.511534+0000","last_fullsized":"2024-03-28T00:54:39.511534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:20:17.397594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.809524+0000","last_change":"2024-03-28T00:54:36.826851+0000","last_active":"2024-03-28T00:54:38.809524+0000","last_peered":"2024-03-28T00:54:38.809524+0000","last_clean":"2024-03-28T00:54:38.809524+0000","last_became_active":"2024-03-28T00:54:36.826359+0000","last_became_peered":"2024-03-28T00:54:36.826359+0000","last_unstale":"2024-03-28T00:54:38.809524+0000","last_undegraded":"2024-03-28T00:54:38.809524+0000","last_fullsized":"2024-03-28T00:54:38.809524+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:27:48.857368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.271112+0000","last_change":"2024-03-28T00:54:32.251701+0000","last_active":"2024-03-28T00:54:50.271112+0000","last_peered":"2024-03-28T00:54:50.271112+0000","last_clean":"2024-03-28T00:54:50.271112+0000","last_became_active":"2024-03-28T00:54:32.251403+0000","last_became_peered":"2024-03-28T00:54:32.251403+0000","last_unstale":"2024-03-28T00:54:50.271112+0000","last_undegraded":"2024-03-28T00:54:50.271112+0000","last_fullsized":"2024-03-28T00:54:50.271112+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:28:16.127643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.810795+0000","last_change":"2024-03-28T00:54:32.806810+0000","last_active":"2024-03-28T00:54:38.810795+0000","last_peered":"2024-03-28T00:54:38.810795+0000","last_clean":"2024-03-28T00:54:38.810795+0000","last_became_active":"2024-03-28T00:54:32.806468+0000","last_became_peered":"2024-03-28T00:54:32.806468+0000","last_unstale":"2024-03-28T00:54:38.810795+0000","last_undegraded":"2024-03-28T00:54:38.810795+0000","last_fullsized":"2024-03-28T00:54:38.810795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:54:16.196032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.675486+0000","last_change":"2024-03-28T00:54:36.827450+0000","last_active":"2024-03-28T00:54:39.675486+0000","last_peered":"2024-03-28T00:54:39.675486+0000","last_clean":"2024-03-28T00:54:39.675486+0000","last_became_active":"2024-03-28T00:54:36.826837+0000","last_became_peered":"2024-03-28T00:54:36.826837+0000","last_unstale":"2024-03-28T00:54:39.675486+0000","last_undegraded":"2024-03-28T00:54:39.675486+0000","last_fullsized":"2024-03-28T00:54:39.675486+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:00:57.813913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.271705+0000","last_change":"2024-03-28T00:54:32.259440+0000","last_active":"2024-03-28T00:54:50.271705+0000","last_peered":"2024-03-28T00:54:50.271705+0000","last_clean":"2024-03-28T00:54:50.271705+0000","last_became_active":"2024-03-28T00:54:32.258225+0000","last_became_peered":"2024-03-28T00:54:32.258225+0000","last_unstale":"2024-03-28T00:54:50.271705+0000","last_undegraded":"2024-03-28T00:54:50.271705+0000","last_fullsized":"2024-03-28T00:54:50.271705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:07:18.398302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808358+0000","last_change":"2024-03-28T00:54:32.806853+0000","last_active":"2024-03-28T00:54:38.808358+0000","last_peered":"2024-03-28T00:54:38.808358+0000","last_clean":"2024-03-28T00:54:38.808358+0000","last_became_active":"2024-03-28T00:54:32.806551+0000","last_became_peered":"2024-03-28T00:54:32.806551+0000","last_unstale":"2024-03-28T00:54:38.808358+0000","last_undegraded":"2024-03-28T00:54:38.808358+0000","last_fullsized":"2024-03-28T00:54:38.808358+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:49:45.774252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.290516+0000","last_change":"2024-03-28T00:54:36.827542+0000","last_active":"2024-03-28T00:54:39.290516+0000","last_peered":"2024-03-28T00:54:39.290516+0000","last_clean":"2024-03-28T00:54:39.290516+0000","last_became_active":"2024-03-28T00:54:36.827163+0000","last_became_peered":"2024-03-28T00:54:36.827163+0000","last_unstale":"2024-03-28T00:54:39.290516+0000","last_undegraded":"2024-03-28T00:54:39.290516+0000","last_fullsized":"2024-03-28T00:54:39.290516+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:36:15.278111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511011+0000","last_change":"2024-03-28T00:54:32.253231+0000","last_active":"2024-03-28T00:54:39.511011+0000","last_peered":"2024-03-28T00:54:39.511011+0000","last_clean":"2024-03-28T00:54:39.511011+0000","last_became_active":"2024-03-28T00:54:32.252731+0000","last_became_peered":"2024-03-28T00:54:32.252731+0000","last_unstale":"2024-03-28T00:54:39.511011+0000","last_undegraded":"2024-03-28T00:54:39.511011+0000","last_fullsized":"2024-03-28T00:54:39.511011+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:05:01.716654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808635+0000","last_change":"2024-03-28T00:54:32.805073+0000","last_active":"2024-03-28T00:54:38.808635+0000","last_peered":"2024-03-28T00:54:38.808635+0000","last_clean":"2024-03-28T00:54:38.808635+0000","last_became_active":"2024-03-28T00:54:32.803690+0000","last_became_peered":"2024-03-28T00:54:32.803690+0000","last_unstale":"2024-03-28T00:54:38.808635+0000","last_undegraded":"2024-03-28T00:54:38.808635+0000","last_fullsized":"2024-03-28T00:54:38.808635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:41:16.586168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.811287+0000","last_change":"2024-03-28T00:54:36.835463+0000","last_active":"2024-03-28T00:54:38.811287+0000","last_peered":"2024-03-28T00:54:38.811287+0000","last_clean":"2024-03-28T00:54:38.811287+0000","last_became_active":"2024-03-28T00:54:36.834982+0000","last_became_peered":"2024-03-28T00:54:36.834982+0000","last_unstale":"2024-03-28T00:54:38.811287+0000","last_undegraded":"2024-03-28T00:54:38.811287+0000","last_fullsized":"2024-03-28T00:54:38.811287+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:27:32.259586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'10","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.271666+0000","last_change":"2024-03-28T00:54:32.248461+0000","last_active":"2024-03-28T00:54:50.271666+0000","last_peered":"2024-03-28T00:54:50.271666+0000","last_clean":"2024-03-28T00:54:50.271666+0000","last_became_active":"2024-03-28T00:54:32.248140+0000","last_became_peered":"2024-03-28T00:54:32.248140+0000","last_unstale":"2024-03-28T00:54:50.271666+0000","last_undegraded":"2024-03-28T00:54:50.271666+0000","last_fullsized":"2024-03-28T00:54:50.271666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:13:38.784288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808767+0000","last_change":"2024-03-28T00:54:32.806651+0000","last_active":"2024-03-28T00:54:38.808767+0000","last_peered":"2024-03-28T00:54:38.808767+0000","last_clean":"2024-03-28T00:54:38.808767+0000","last_became_active":"2024-03-28T00:54:32.806329+0000","last_became_peered":"2024-03-28T00:54:32.806329+0000","last_unstale":"2024-03-28T00:54:38.808767+0000","last_undegraded":"2024-03-28T00:54:38.808767+0000","last_fullsized":"2024-03-28T00:54:38.808767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:28:32.815396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808685+0000","last_change":"2024-03-28T00:54:36.827016+0000","last_active":"2024-03-28T00:54:38.808685+0000","last_peered":"2024-03-28T00:54:38.808685+0000","last_clean":"2024-03-28T00:54:38.808685+0000","last_became_active":"2024-03-28T00:54:36.826428+0000","last_became_peered":"2024-03-28T00:54:36.826428+0000","last_unstale":"2024-03-28T00:54:38.808685+0000","last_undegraded":"2024-03-28T00:54:38.808685+0000","last_fullsized":"2024-03-28T00:54:38.808685+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:33:01.593990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511799+0000","last_change":"2024-03-28T00:54:32.252532+0000","last_active":"2024-03-28T00:54:39.511799+0000","last_peered":"2024-03-28T00:54:39.511799+0000","last_clean":"2024-03-28T00:54:39.511799+0000","last_became_active":"2024-03-28T00:54:32.252204+0000","last_became_peered":"2024-03-28T00:54:32.252204+0000","last_unstale":"2024-03-28T00:54:39.511799+0000","last_undegraded":"2024-03-28T00:54:39.511799+0000","last_fullsized":"2024-03-28T00:54:39.511799+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:03:59.157278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.675715+0000","last_change":"2024-03-28T00:54:32.808064+0000","last_active":"2024-03-28T00:54:39.675715+0000","last_peered":"2024-03-28T00:54:39.675715+0000","last_clean":"2024-03-28T00:54:39.675715+0000","last_became_active":"2024-03-28T00:54:32.807181+0000","last_became_peered":"2024-03-28T00:54:32.807181+0000","last_unstale":"2024-03-28T00:54:39.675715+0000","last_undegraded":"2024-03-28T00:54:39.675715+0000","last_fullsized":"2024-03-28T00:54:39.675715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:26:01.917500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510838+0000","last_change":"2024-03-28T00:54:37.638175+0000","last_active":"2024-03-28T00:54:39.510838+0000","last_peered":"2024-03-28T00:54:39.510838+0000","last_clean":"2024-03-28T00:54:39.510838+0000","last_became_active":"2024-03-28T00:54:37.637662+0000","last_became_peered":"2024-03-28T00:54:37.637662+0000","last_unstale":"2024-03-28T00:54:39.510838+0000","last_undegraded":"2024-03-28T00:54:39.510838+0000","last_fullsized":"2024-03-28T00:54:39.510838+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:38:21.353519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.290306+0000","last_change":"2024-03-28T00:54:32.264038+0000","last_active":"2024-03-28T00:54:39.290306+0000","last_peered":"2024-03-28T00:54:39.290306+0000","last_clean":"2024-03-28T00:54:39.290306+0000","last_became_active":"2024-03-28T00:54:32.263706+0000","last_became_peered":"2024-03-28T00:54:32.263706+0000","last_unstale":"2024-03-28T00:54:39.290306+0000","last_undegraded":"2024-03-28T00:54:39.290306+0000","last_fullsized":"2024-03-28T00:54:39.290306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:03.244556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.148552+0000","last_change":"2024-03-28T00:54:32.804727+0000","last_active":"2024-03-28T00:54:39.148552+0000","last_peered":"2024-03-28T00:54:39.148552+0000","last_clean":"2024-03-28T00:54:39.148552+0000","last_became_active":"2024-03-28T00:54:32.804422+0000","last_became_peered":"2024-03-28T00:54:32.804422+0000","last_unstale":"2024-03-28T00:54:39.148552+0000","last_undegraded":"2024-03-28T00:54:39.148552+0000","last_fullsized":"2024-03-28T00:54:39.148552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:29:57.983722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.811349+0000","last_change":"2024-03-28T00:54:36.833416+0000","last_active":"2024-03-28T00:54:38.811349+0000","last_peered":"2024-03-28T00:54:38.811349+0000","last_clean":"2024-03-28T00:54:38.811349+0000","last_became_active":"2024-03-28T00:54:36.832985+0000","last_became_peered":"2024-03-28T00:54:36.832985+0000","last_unstale":"2024-03-28T00:54:38.811349+0000","last_undegraded":"2024-03-28T00:54:38.811349+0000","last_fullsized":"2024-03-28T00:54:38.811349+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:47:30.376032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808276+0000","last_change":"2024-03-28T00:54:32.256417+0000","last_active":"2024-03-28T00:54:38.808276+0000","last_peered":"2024-03-28T00:54:38.808276+0000","last_clean":"2024-03-28T00:54:38.808276+0000","last_became_active":"2024-03-28T00:54:32.256088+0000","last_became_peered":"2024-03-28T00:54:32.256088+0000","last_unstale":"2024-03-28T00:54:38.808276+0000","last_undegraded":"2024-03-28T00:54:38.808276+0000","last_fullsized":"2024-03-28T00:54:38.808276+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:58.661597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.148398+0000","last_change":"2024-03-28T00:54:32.817604+0000","last_active":"2024-03-28T00:54:39.148398+0000","last_peered":"2024-03-28T00:54:39.148398+0000","last_clean":"2024-03-28T00:54:39.148398+0000","last_became_active":"2024-03-28T00:54:32.817211+0000","last_became_peered":"2024-03-28T00:54:32.817211+0000","last_unstale":"2024-03-28T00:54:39.148398+0000","last_undegraded":"2024-03-28T00:54:39.148398+0000","last_fullsized":"2024-03-28T00:54:39.148398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:02:28.335075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.811147+0000","last_change":"2024-03-28T00:54:37.638968+0000","last_active":"2024-03-28T00:54:38.811147+0000","last_peered":"2024-03-28T00:54:38.811147+0000","last_clean":"2024-03-28T00:54:38.811147+0000","last_became_active":"2024-03-28T00:54:37.638446+0000","last_became_peered":"2024-03-28T00:54:37.638446+0000","last_unstale":"2024-03-28T00:54:38.811147+0000","last_undegraded":"2024-03-28T00:54:38.811147+0000","last_fullsized":"2024-03-28T00:54:38.811147+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:54:56.348760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808684+0000","last_change":"2024-03-28T00:54:32.809822+0000","last_active":"2024-03-28T00:54:38.808684+0000","last_peered":"2024-03-28T00:54:38.808684+0000","last_clean":"2024-03-28T00:54:38.808684+0000","last_became_active":"2024-03-28T00:54:32.808284+0000","last_became_peered":"2024-03-28T00:54:32.808284+0000","last_unstale":"2024-03-28T00:54:38.808684+0000","last_undegraded":"2024-03-28T00:54:38.808684+0000","last_fullsized":"2024-03-28T00:54:38.808684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:49:11.651960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.810716+0000","last_change":"2024-03-28T00:54:32.261806+0000","last_active":"2024-03-28T00:54:38.810716+0000","last_peered":"2024-03-28T00:54:38.810716+0000","last_clean":"2024-03-28T00:54:38.810716+0000","last_became_active":"2024-03-28T00:54:32.261493+0000","last_became_peered":"2024-03-28T00:54:32.261493+0000","last_unstale":"2024-03-28T00:54:38.810716+0000","last_undegraded":"2024-03-28T00:54:38.810716+0000","last_fullsized":"2024-03-28T00:54:38.810716+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:36:04.283408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.856729+0000","last_change":"2024-03-28T00:54:36.819332+0000","last_active":"2024-03-28T00:54:38.856729+0000","last_peered":"2024-03-28T00:54:38.856729+0000","last_clean":"2024-03-28T00:54:38.856729+0000","last_became_active":"2024-03-28T00:54:36.818932+0000","last_became_peered":"2024-03-28T00:54:36.818932+0000","last_unstale":"2024-03-28T00:54:38.856729+0000","last_undegraded":"2024-03-28T00:54:38.856729+0000","last_fullsized":"2024-03-28T00:54:38.856729+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:00:39.446738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511417+0000","last_change":"2024-03-28T00:54:32.806093+0000","last_active":"2024-03-28T00:54:39.511417+0000","last_peered":"2024-03-28T00:54:39.511417+0000","last_clean":"2024-03-28T00:54:39.511417+0000","last_became_active":"2024-03-28T00:54:32.805734+0000","last_became_peered":"2024-03-28T00:54:32.805734+0000","last_unstale":"2024-03-28T00:54:39.511417+0000","last_undegraded":"2024-03-28T00:54:39.511417+0000","last_fullsized":"2024-03-28T00:54:39.511417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:51:41.066225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.270104+0000","last_change":"2024-03-28T00:54:32.245182+0000","last_active":"2024-03-28T00:54:50.270104+0000","last_peered":"2024-03-28T00:54:50.270104+0000","last_clean":"2024-03-28T00:54:50.270104+0000","last_became_active":"2024-03-28T00:54:32.244855+0000","last_became_peered":"2024-03-28T00:54:32.244855+0000","last_unstale":"2024-03-28T00:54:50.270104+0000","last_undegraded":"2024-03-28T00:54:50.270104+0000","last_fullsized":"2024-03-28T00:54:50.270104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:39:09.997199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":347,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808208+0000","last_change":"2024-03-28T00:53:20.355628+0000","last_active":"2024-03-28T00:54:38.808208+0000","last_peered":"2024-03-28T00:54:38.808208+0000","last_clean":"2024-03-28T00:54:38.808208+0000","last_became_active":"2024-03-28T00:53:20.355175+0000","last_became_peered":"2024-03-28T00:53:20.355175+0000","last_unstale":"2024-03-28T00:54:38.808208+0000","last_undegraded":"2024-03-28T00:54:38.808208+0000","last_fullsized":"2024-03-28T00:54:38.808208+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:49:57.054435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:49:57.054435+0000","last_clean_scrub_stamp":"2024-03-28T00:49:57.054435+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:55:46.753799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.809409+0000","last_change":"2024-03-28T00:54:37.639760+0000","last_active":"2024-03-28T00:54:38.809409+0000","last_peered":"2024-03-28T00:54:38.809409+0000","last_clean":"2024-03-28T00:54:38.809409+0000","last_became_active":"2024-03-28T00:54:37.639287+0000","last_became_peered":"2024-03-28T00:54:37.639287+0000","last_unstale":"2024-03-28T00:54:38.809409+0000","last_undegraded":"2024-03-28T00:54:38.809409+0000","last_fullsized":"2024-03-28T00:54:38.809409+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:50:02.840312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.810531+0000","last_change":"2024-03-28T00:54:32.807764+0000","last_active":"2024-03-28T00:54:38.810531+0000","last_peered":"2024-03-28T00:54:38.810531+0000","last_clean":"2024-03-28T00:54:38.810531+0000","last_became_active":"2024-03-28T00:54:32.807441+0000","last_became_peered":"2024-03-28T00:54:32.807441+0000","last_unstale":"2024-03-28T00:54:38.810531+0000","last_undegraded":"2024-03-28T00:54:38.810531+0000","last_fullsized":"2024-03-28T00:54:38.810531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:37:40.540514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:51.277141+0000","last_change":"2024-03-28T00:54:32.259192+0000","last_active":"2024-03-28T00:54:51.277141+0000","last_peered":"2024-03-28T00:54:51.277141+0000","last_clean":"2024-03-28T00:54:51.277141+0000","last_became_active":"2024-03-28T00:54:32.258880+0000","last_became_peered":"2024-03-28T00:54:32.258880+0000","last_unstale":"2024-03-28T00:54:51.277141+0000","last_undegraded":"2024-03-28T00:54:51.277141+0000","last_fullsized":"2024-03-28T00:54:51.277141+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:20:11.384633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.675482+0000","last_change":"2024-03-28T00:54:36.822268+0000","last_active":"2024-03-28T00:54:39.675482+0000","last_peered":"2024-03-28T00:54:39.675482+0000","last_clean":"2024-03-28T00:54:39.675482+0000","last_became_active":"2024-03-28T00:54:36.821773+0000","last_became_peered":"2024-03-28T00:54:36.821773+0000","last_unstale":"2024-03-28T00:54:39.675482+0000","last_undegraded":"2024-03-28T00:54:39.675482+0000","last_fullsized":"2024-03-28T00:54:39.675482+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:56:11.206975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511679+0000","last_change":"2024-03-28T00:54:32.256133+0000","last_active":"2024-03-28T00:54:39.511679+0000","last_peered":"2024-03-28T00:54:39.511679+0000","last_clean":"2024-03-28T00:54:39.511679+0000","last_became_active":"2024-03-28T00:54:32.250535+0000","last_became_peered":"2024-03-28T00:54:32.250535+0000","last_unstale":"2024-03-28T00:54:39.511679+0000","last_undegraded":"2024-03-28T00:54:39.511679+0000","last_fullsized":"2024-03-28T00:54:39.511679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:18:50.874181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.811341+0000","last_change":"2024-03-28T00:54:32.808883+0000","last_active":"2024-03-28T00:54:38.811341+0000","last_peered":"2024-03-28T00:54:38.811341+0000","last_clean":"2024-03-28T00:54:38.811341+0000","last_became_active":"2024-03-28T00:54:32.808587+0000","last_became_peered":"2024-03-28T00:54:32.808587+0000","last_unstale":"2024-03-28T00:54:38.811341+0000","last_undegraded":"2024-03-28T00:54:38.811341+0000","last_fullsized":"2024-03-28T00:54:38.811341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:09:55.164876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510865+0000","last_change":"2024-03-28T00:54:37.637743+0000","last_active":"2024-03-28T00:54:39.510865+0000","last_peered":"2024-03-28T00:54:39.510865+0000","last_clean":"2024-03-28T00:54:39.510865+0000","last_became_active":"2024-03-28T00:54:37.637280+0000","last_became_peered":"2024-03-28T00:54:37.637280+0000","last_unstale":"2024-03-28T00:54:39.510865+0000","last_undegraded":"2024-03-28T00:54:39.510865+0000","last_fullsized":"2024-03-28T00:54:39.510865+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:40:51.431757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.290423+0000","last_change":"2024-03-28T00:54:32.246444+0000","last_active":"2024-03-28T00:54:39.290423+0000","last_peered":"2024-03-28T00:54:39.290423+0000","last_clean":"2024-03-28T00:54:39.290423+0000","last_became_active":"2024-03-28T00:54:32.245106+0000","last_became_peered":"2024-03-28T00:54:32.245106+0000","last_unstale":"2024-03-28T00:54:39.290423+0000","last_undegraded":"2024-03-28T00:54:39.290423+0000","last_fullsized":"2024-03-28T00:54:39.290423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:13:45.123625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511204+0000","last_change":"2024-03-28T00:54:32.812851+0000","last_active":"2024-03-28T00:54:39.511204+0000","last_peered":"2024-03-28T00:54:39.511204+0000","last_clean":"2024-03-28T00:54:39.511204+0000","last_became_active":"2024-03-28T00:54:32.812518+0000","last_became_peered":"2024-03-28T00:54:32.812518+0000","last_unstale":"2024-03-28T00:54:39.511204+0000","last_undegraded":"2024-03-28T00:54:39.511204+0000","last_fullsized":"2024-03-28T00:54:39.511204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:08:41.518888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511256+0000","last_change":"2024-03-28T00:54:36.818709+0000","last_active":"2024-03-28T00:54:39.511256+0000","last_peered":"2024-03-28T00:54:39.511256+0000","last_clean":"2024-03-28T00:54:39.511256+0000","last_became_active":"2024-03-28T00:54:36.818304+0000","last_became_peered":"2024-03-28T00:54:36.818304+0000","last_unstale":"2024-03-28T00:54:39.511256+0000","last_undegraded":"2024-03-28T00:54:39.511256+0000","last_fullsized":"2024-03-28T00:54:39.511256+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:03:07.642685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.269754+0000","last_change":"2024-03-28T00:54:32.256031+0000","last_active":"2024-03-28T00:54:50.269754+0000","last_peered":"2024-03-28T00:54:50.269754+0000","last_clean":"2024-03-28T00:54:50.269754+0000","last_became_active":"2024-03-28T00:54:32.255597+0000","last_became_peered":"2024-03-28T00:54:32.255597+0000","last_unstale":"2024-03-28T00:54:50.269754+0000","last_undegraded":"2024-03-28T00:54:50.269754+0000","last_fullsized":"2024-03-28T00:54:50.269754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:31:10.837212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511212+0000","last_change":"2024-03-28T00:54:32.816961+0000","last_active":"2024-03-28T00:54:39.511212+0000","last_peered":"2024-03-28T00:54:39.511212+0000","last_clean":"2024-03-28T00:54:39.511212+0000","last_became_active":"2024-03-28T00:54:32.816408+0000","last_became_peered":"2024-03-28T00:54:32.816408+0000","last_unstale":"2024-03-28T00:54:39.511212+0000","last_undegraded":"2024-03-28T00:54:39.511212+0000","last_fullsized":"2024-03-28T00:54:39.511212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:08:57.441492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510781+0000","last_change":"2024-03-28T00:54:37.637811+0000","last_active":"2024-03-28T00:54:39.510781+0000","last_peered":"2024-03-28T00:54:39.510781+0000","last_clean":"2024-03-28T00:54:39.510781+0000","last_became_active":"2024-03-28T00:54:37.637347+0000","last_became_peered":"2024-03-28T00:54:37.637347+0000","last_unstale":"2024-03-28T00:54:39.510781+0000","last_undegraded":"2024-03-28T00:54:39.510781+0000","last_fullsized":"2024-03-28T00:54:39.510781+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:17:52.687297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.272465+0000","last_change":"2024-03-28T00:54:32.250301+0000","last_active":"2024-03-28T00:54:50.272465+0000","last_peered":"2024-03-28T00:54:50.272465+0000","last_clean":"2024-03-28T00:54:50.272465+0000","last_became_active":"2024-03-28T00:54:32.249993+0000","last_became_peered":"2024-03-28T00:54:32.249993+0000","last_unstale":"2024-03-28T00:54:50.272465+0000","last_undegraded":"2024-03-28T00:54:50.272465+0000","last_fullsized":"2024-03-28T00:54:50.272465+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:53:34.826687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510650+0000","last_change":"2024-03-28T00:54:32.811592+0000","last_active":"2024-03-28T00:54:39.510650+0000","last_peered":"2024-03-28T00:54:39.510650+0000","last_clean":"2024-03-28T00:54:39.510650+0000","last_became_active":"2024-03-28T00:54:32.811228+0000","last_became_peered":"2024-03-28T00:54:32.811228+0000","last_unstale":"2024-03-28T00:54:39.510650+0000","last_undegraded":"2024-03-28T00:54:39.510650+0000","last_fullsized":"2024-03-28T00:54:39.510650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:14:19.585237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.290628+0000","last_change":"2024-03-28T00:54:36.822501+0000","last_active":"2024-03-28T00:54:39.290628+0000","last_peered":"2024-03-28T00:54:39.290628+0000","last_clean":"2024-03-28T00:54:39.290628+0000","last_became_active":"2024-03-28T00:54:36.821630+0000","last_became_peered":"2024-03-28T00:54:36.821630+0000","last_unstale":"2024-03-28T00:54:39.290628+0000","last_undegraded":"2024-03-28T00:54:39.290628+0000","last_fullsized":"2024-03-28T00:54:39.290628+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:20:52.247055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.271642+0000","last_change":"2024-03-28T00:54:32.247328+0000","last_active":"2024-03-28T00:54:50.271642+0000","last_peered":"2024-03-28T00:54:50.271642+0000","last_clean":"2024-03-28T00:54:50.271642+0000","last_became_active":"2024-03-28T00:54:32.247003+0000","last_became_peered":"2024-03-28T00:54:32.247003+0000","last_unstale":"2024-03-28T00:54:50.271642+0000","last_undegraded":"2024-03-28T00:54:50.271642+0000","last_fullsized":"2024-03-28T00:54:50.271642+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:33:09.764946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.675291+0000","last_change":"2024-03-28T00:54:32.805977+0000","last_active":"2024-03-28T00:54:39.675291+0000","last_peered":"2024-03-28T00:54:39.675291+0000","last_clean":"2024-03-28T00:54:39.675291+0000","last_became_active":"2024-03-28T00:54:32.805636+0000","last_became_peered":"2024-03-28T00:54:32.805636+0000","last_unstale":"2024-03-28T00:54:39.675291+0000","last_undegraded":"2024-03-28T00:54:39.675291+0000","last_fullsized":"2024-03-28T00:54:39.675291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:04:42.901732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510860+0000","last_change":"2024-03-28T00:54:36.822103+0000","last_active":"2024-03-28T00:54:39.510860+0000","last_peered":"2024-03-28T00:54:39.510860+0000","last_clean":"2024-03-28T00:54:39.510860+0000","last_became_active":"2024-03-28T00:54:36.821695+0000","last_became_peered":"2024-03-28T00:54:36.821695+0000","last_unstale":"2024-03-28T00:54:39.510860+0000","last_undegraded":"2024-03-28T00:54:39.510860+0000","last_fullsized":"2024-03-28T00:54:39.510860+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:45:00.613955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511084+0000","last_change":"2024-03-28T00:54:32.251523+0000","last_active":"2024-03-28T00:54:39.511084+0000","last_peered":"2024-03-28T00:54:39.511084+0000","last_clean":"2024-03-28T00:54:39.511084+0000","last_became_active":"2024-03-28T00:54:32.251223+0000","last_became_peered":"2024-03-28T00:54:32.251223+0000","last_unstale":"2024-03-28T00:54:39.511084+0000","last_undegraded":"2024-03-28T00:54:39.511084+0000","last_fullsized":"2024-03-28T00:54:39.511084+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:08:20.489848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511358+0000","last_change":"2024-03-28T00:54:32.817641+0000","last_active":"2024-03-28T00:54:39.511358+0000","last_peered":"2024-03-28T00:54:39.511358+0000","last_clean":"2024-03-28T00:54:39.511358+0000","last_became_active":"2024-03-28T00:54:32.817252+0000","last_became_peered":"2024-03-28T00:54:32.817252+0000","last_unstale":"2024-03-28T00:54:39.511358+0000","last_undegraded":"2024-03-28T00:54:39.511358+0000","last_fullsized":"2024-03-28T00:54:39.511358+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:54:48.275005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510849+0000","last_change":"2024-03-28T00:54:37.638990+0000","last_active":"2024-03-28T00:54:39.510849+0000","last_peered":"2024-03-28T00:54:39.510849+0000","last_clean":"2024-03-28T00:54:39.510849+0000","last_became_active":"2024-03-28T00:54:37.638538+0000","last_became_peered":"2024-03-28T00:54:37.638538+0000","last_unstale":"2024-03-28T00:54:39.510849+0000","last_undegraded":"2024-03-28T00:54:39.510849+0000","last_fullsized":"2024-03-28T00:54:39.510849+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:28:10.791805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.271077+0000","last_change":"2024-03-28T00:54:32.253254+0000","last_active":"2024-03-28T00:54:50.271077+0000","last_peered":"2024-03-28T00:54:50.271077+0000","last_clean":"2024-03-28T00:54:50.271077+0000","last_became_active":"2024-03-28T00:54:32.252879+0000","last_became_peered":"2024-03-28T00:54:32.252879+0000","last_unstale":"2024-03-28T00:54:50.271077+0000","last_undegraded":"2024-03-28T00:54:50.271077+0000","last_fullsized":"2024-03-28T00:54:50.271077+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:59:51.039215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808305+0000","last_change":"2024-03-28T00:54:32.811288+0000","last_active":"2024-03-28T00:54:38.808305+0000","last_peered":"2024-03-28T00:54:38.808305+0000","last_clean":"2024-03-28T00:54:38.808305+0000","last_became_active":"2024-03-28T00:54:32.810968+0000","last_became_peered":"2024-03-28T00:54:32.810968+0000","last_unstale":"2024-03-28T00:54:38.808305+0000","last_undegraded":"2024-03-28T00:54:38.808305+0000","last_fullsized":"2024-03-28T00:54:38.808305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:16:56.113046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510257+0000","last_change":"2024-03-28T00:54:36.830714+0000","last_active":"2024-03-28T00:54:39.510257+0000","last_peered":"2024-03-28T00:54:39.510257+0000","last_clean":"2024-03-28T00:54:39.510257+0000","last_became_active":"2024-03-28T00:54:36.830253+0000","last_became_peered":"2024-03-28T00:54:36.830253+0000","last_unstale":"2024-03-28T00:54:39.510257+0000","last_undegraded":"2024-03-28T00:54:39.510257+0000","last_fullsized":"2024-03-28T00:54:39.510257+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:24:43.696150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511755+0000","last_change":"2024-03-28T00:54:32.254009+0000","last_active":"2024-03-28T00:54:39.511755+0000","last_peered":"2024-03-28T00:54:39.511755+0000","last_clean":"2024-03-28T00:54:39.511755+0000","last_became_active":"2024-03-28T00:54:32.253697+0000","last_became_peered":"2024-03-28T00:54:32.253697+0000","last_unstale":"2024-03-28T00:54:39.511755+0000","last_undegraded":"2024-03-28T00:54:39.511755+0000","last_fullsized":"2024-03-28T00:54:39.511755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:17:44.629367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808729+0000","last_change":"2024-03-28T00:54:32.812828+0000","last_active":"2024-03-28T00:54:38.808729+0000","last_peered":"2024-03-28T00:54:38.808729+0000","last_clean":"2024-03-28T00:54:38.808729+0000","last_became_active":"2024-03-28T00:54:32.811591+0000","last_became_peered":"2024-03-28T00:54:32.811591+0000","last_unstale":"2024-03-28T00:54:38.808729+0000","last_undegraded":"2024-03-28T00:54:38.808729+0000","last_fullsized":"2024-03-28T00:54:38.808729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:04.613623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510810+0000","last_change":"2024-03-28T00:54:36.823284+0000","last_active":"2024-03-28T00:54:39.510810+0000","last_peered":"2024-03-28T00:54:39.510810+0000","last_clean":"2024-03-28T00:54:39.510810+0000","last_became_active":"2024-03-28T00:54:36.822883+0000","last_became_peered":"2024-03-28T00:54:36.822883+0000","last_unstale":"2024-03-28T00:54:39.510810+0000","last_undegraded":"2024-03-28T00:54:39.510810+0000","last_fullsized":"2024-03-28T00:54:39.510810+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:37:10.266063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.809170+0000","last_change":"2024-03-28T00:54:32.253946+0000","last_active":"2024-03-28T00:54:38.809170+0000","last_peered":"2024-03-28T00:54:38.809170+0000","last_clean":"2024-03-28T00:54:38.809170+0000","last_became_active":"2024-03-28T00:54:32.253650+0000","last_became_peered":"2024-03-28T00:54:32.253650+0000","last_unstale":"2024-03-28T00:54:38.809170+0000","last_undegraded":"2024-03-28T00:54:38.809170+0000","last_fullsized":"2024-03-28T00:54:38.809170+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:11:44.930074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808579+0000","last_change":"2024-03-28T00:54:32.815410+0000","last_active":"2024-03-28T00:54:38.808579+0000","last_peered":"2024-03-28T00:54:38.808579+0000","last_clean":"2024-03-28T00:54:38.808579+0000","last_became_active":"2024-03-28T00:54:32.815088+0000","last_became_peered":"2024-03-28T00:54:32.815088+0000","last_unstale":"2024-03-28T00:54:38.808579+0000","last_undegraded":"2024-03-28T00:54:38.808579+0000","last_fullsized":"2024-03-28T00:54:38.808579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:16:24.862822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808474+0000","last_change":"2024-03-28T00:54:36.823182+0000","last_active":"2024-03-28T00:54:38.808474+0000","last_peered":"2024-03-28T00:54:38.808474+0000","last_clean":"2024-03-28T00:54:38.808474+0000","last_became_active":"2024-03-28T00:54:36.821446+0000","last_became_peered":"2024-03-28T00:54:36.821446+0000","last_unstale":"2024-03-28T00:54:38.808474+0000","last_undegraded":"2024-03-28T00:54:38.808474+0000","last_fullsized":"2024-03-28T00:54:38.808474+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:45:20.030213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.811026+0000","last_change":"2024-03-28T00:54:32.257412+0000","last_active":"2024-03-28T00:54:38.811026+0000","last_peered":"2024-03-28T00:54:38.811026+0000","last_clean":"2024-03-28T00:54:38.811026+0000","last_became_active":"2024-03-28T00:54:32.257115+0000","last_became_peered":"2024-03-28T00:54:32.257115+0000","last_unstale":"2024-03-28T00:54:38.811026+0000","last_undegraded":"2024-03-28T00:54:38.811026+0000","last_fullsized":"2024-03-28T00:54:38.811026+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:04:29.850867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808830+0000","last_change":"2024-03-28T00:54:32.805879+0000","last_active":"2024-03-28T00:54:38.808830+0000","last_peered":"2024-03-28T00:54:38.808830+0000","last_clean":"2024-03-28T00:54:38.808830+0000","last_became_active":"2024-03-28T00:54:32.805507+0000","last_became_peered":"2024-03-28T00:54:32.805507+0000","last_unstale":"2024-03-28T00:54:38.808830+0000","last_undegraded":"2024-03-28T00:54:38.808830+0000","last_fullsized":"2024-03-28T00:54:38.808830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:51:51.229085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.675457+0000","last_change":"2024-03-28T00:54:32.807219+0000","last_active":"2024-03-28T00:54:39.675457+0000","last_peered":"2024-03-28T00:54:39.675457+0000","last_clean":"2024-03-28T00:54:39.675457+0000","last_became_active":"2024-03-28T00:54:32.806856+0000","last_became_peered":"2024-03-28T00:54:32.806856+0000","last_unstale":"2024-03-28T00:54:39.675457+0000","last_undegraded":"2024-03-28T00:54:39.675457+0000","last_fullsized":"2024-03-28T00:54:39.675457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:05:49.914636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511501+0000","last_change":"2024-03-28T00:54:32.806509+0000","last_active":"2024-03-28T00:54:39.511501+0000","last_peered":"2024-03-28T00:54:39.511501+0000","last_clean":"2024-03-28T00:54:39.511501+0000","last_became_active":"2024-03-28T00:54:32.806211+0000","last_became_peered":"2024-03-28T00:54:32.806211+0000","last_unstale":"2024-03-28T00:54:39.511501+0000","last_undegraded":"2024-03-28T00:54:39.511501+0000","last_fullsized":"2024-03-28T00:54:39.511501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:30:19.711477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510959+0000","last_change":"2024-03-28T00:54:32.813921+0000","last_active":"2024-03-28T00:54:39.510959+0000","last_peered":"2024-03-28T00:54:39.510959+0000","last_clean":"2024-03-28T00:54:39.510959+0000","last_became_active":"2024-03-28T00:54:32.812877+0000","last_became_peered":"2024-03-28T00:54:32.812877+0000","last_unstale":"2024-03-28T00:54:39.510959+0000","last_undegraded":"2024-03-28T00:54:39.510959+0000","last_fullsized":"2024-03-28T00:54:39.510959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:26:54.305074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510924+0000","last_change":"2024-03-28T00:54:32.815725+0000","last_active":"2024-03-28T00:54:39.510924+0000","last_peered":"2024-03-28T00:54:39.510924+0000","last_clean":"2024-03-28T00:54:39.510924+0000","last_became_active":"2024-03-28T00:54:32.815444+0000","last_became_peered":"2024-03-28T00:54:32.815444+0000","last_unstale":"2024-03-28T00:54:39.510924+0000","last_undegraded":"2024-03-28T00:54:39.510924+0000","last_fullsized":"2024-03-28T00:54:39.510924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:23:17.326589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.811200+0000","last_change":"2024-03-28T00:54:32.806595+0000","last_active":"2024-03-28T00:54:38.811200+0000","last_peered":"2024-03-28T00:54:38.811200+0000","last_clean":"2024-03-28T00:54:38.811200+0000","last_became_active":"2024-03-28T00:54:32.806256+0000","last_became_peered":"2024-03-28T00:54:32.806256+0000","last_unstale":"2024-03-28T00:54:38.811200+0000","last_undegraded":"2024-03-28T00:54:38.811200+0000","last_fullsized":"2024-03-28T00:54:38.811200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:04:22.551511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.675726+0000","last_change":"2024-03-28T00:54:32.805000+0000","last_active":"2024-03-28T00:54:39.675726+0000","last_peered":"2024-03-28T00:54:39.675726+0000","last_clean":"2024-03-28T00:54:39.675726+0000","last_became_active":"2024-03-28T00:54:32.804637+0000","last_became_peered":"2024-03-28T00:54:32.804637+0000","last_unstale":"2024-03-28T00:54:39.675726+0000","last_undegraded":"2024-03-28T00:54:39.675726+0000","last_fullsized":"2024-03-28T00:54:39.675726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:59:46.888014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511538+0000","last_change":"2024-03-28T00:54:32.802783+0000","last_active":"2024-03-28T00:54:39.511538+0000","last_peered":"2024-03-28T00:54:39.511538+0000","last_clean":"2024-03-28T00:54:39.511538+0000","last_became_active":"2024-03-28T00:54:32.802457+0000","last_became_peered":"2024-03-28T00:54:32.802457+0000","last_unstale":"2024-03-28T00:54:39.511538+0000","last_undegraded":"2024-03-28T00:54:39.511538+0000","last_fullsized":"2024-03-28T00:54:39.511538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:28:24.146047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511618+0000","last_change":"2024-03-28T00:54:32.810179+0000","last_active":"2024-03-28T00:54:39.511618+0000","last_peered":"2024-03-28T00:54:39.511618+0000","last_clean":"2024-03-28T00:54:39.511618+0000","last_became_active":"2024-03-28T00:54:32.809009+0000","last_became_peered":"2024-03-28T00:54:32.809009+0000","last_unstale":"2024-03-28T00:54:39.511618+0000","last_undegraded":"2024-03-28T00:54:39.511618+0000","last_fullsized":"2024-03-28T00:54:39.511618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:08:44.184910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808642+0000","last_change":"2024-03-28T00:54:32.810084+0000","last_active":"2024-03-28T00:54:38.808642+0000","last_peered":"2024-03-28T00:54:38.808642+0000","last_clean":"2024-03-28T00:54:38.808642+0000","last_became_active":"2024-03-28T00:54:32.809386+0000","last_became_peered":"2024-03-28T00:54:32.809386+0000","last_unstale":"2024-03-28T00:54:38.808642+0000","last_undegraded":"2024-03-28T00:54:38.808642+0000","last_fullsized":"2024-03-28T00:54:38.808642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:49:13.754797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511237+0000","last_change":"2024-03-28T00:54:32.809933+0000","last_active":"2024-03-28T00:54:39.511237+0000","last_peered":"2024-03-28T00:54:39.511237+0000","last_clean":"2024-03-28T00:54:39.511237+0000","last_became_active":"2024-03-28T00:54:32.808974+0000","last_became_peered":"2024-03-28T00:54:32.808974+0000","last_unstale":"2024-03-28T00:54:39.511237+0000","last_undegraded":"2024-03-28T00:54:39.511237+0000","last_fullsized":"2024-03-28T00:54:39.511237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:23:59.317887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510532+0000","last_change":"2024-03-28T00:54:32.808764+0000","last_active":"2024-03-28T00:54:39.510532+0000","last_peered":"2024-03-28T00:54:39.510532+0000","last_clean":"2024-03-28T00:54:39.510532+0000","last_became_active":"2024-03-28T00:54:32.808034+0000","last_became_peered":"2024-03-28T00:54:32.808034+0000","last_unstale":"2024-03-28T00:54:39.510532+0000","last_undegraded":"2024-03-28T00:54:39.510532+0000","last_fullsized":"2024-03-28T00:54:39.510532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:27:12.762520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.811066+0000","last_change":"2024-03-28T00:54:32.805470+0000","last_active":"2024-03-28T00:54:38.811066+0000","last_peered":"2024-03-28T00:54:38.811066+0000","last_clean":"2024-03-28T00:54:38.811066+0000","last_became_active":"2024-03-28T00:54:32.805146+0000","last_became_peered":"2024-03-28T00:54:32.805146+0000","last_unstale":"2024-03-28T00:54:38.811066+0000","last_undegraded":"2024-03-28T00:54:38.811066+0000","last_fullsized":"2024-03-28T00:54:38.811066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:24:14.297966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.675580+0000","last_change":"2024-03-28T00:54:32.808202+0000","last_active":"2024-03-28T00:54:39.675580+0000","last_peered":"2024-03-28T00:54:39.675580+0000","last_clean":"2024-03-28T00:54:39.675580+0000","last_became_active":"2024-03-28T00:54:32.807856+0000","last_became_peered":"2024-03-28T00:54:32.807856+0000","last_unstale":"2024-03-28T00:54:39.675580+0000","last_undegraded":"2024-03-28T00:54:39.675580+0000","last_fullsized":"2024-03-28T00:54:39.675580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:52:11.514904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282084,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6046,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":93,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":47711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383697,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":487204,"data_compressed":17627,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645334,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":1082410,"data_compressed":53513,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808858,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":487406,"data_compressed":17627,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070496,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":487248,"data_compressed":17627,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332133,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":1087365,"data_compressed":53513,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495657,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":974848,"data_stored":486622,"data_compressed":17627,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":490464,"data_compressed":17627,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":486712,"data_compressed":17627,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":491120,"data_compressed":17627,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182456,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":486816,"data_compressed":17627,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920820,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":486644,"data_compressed":17627,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":1077172,"data_compressed":53513,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34218,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34218,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34218,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":31746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T00:54:55.002 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T00:54:55.142 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.136+0000 7f98a9f9d640 1 -- 172.21.15.169:0/3039184763 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98a4102980 msgr2=0x7f98a4102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:55.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.136+0000 7f98a9f9d640 1 --2- 172.21.15.169:0/3039184763 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98a4102980 0x7f98a4102e00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f988c009930 tx=0x7f988c02bc30 comp rx=0 tx=0).stop 2024-03-28T00:54:55.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.136+0000 7f98a9f9d640 1 -- 172.21.15.169:0/3039184763 shutdown_connections 2024-03-28T00:54:55.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.136+0000 7f98a9f9d640 1 --2- 172.21.15.169:0/3039184763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98a4103340 0x7f98a4109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:55.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.136+0000 7f98a9f9d640 1 --2- 172.21.15.169:0/3039184763 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98a4102980 0x7f98a4102e00 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:55.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.136+0000 7f98a9f9d640 1 --2- 172.21.15.169:0/3039184763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98a40ff9b0 0x7f98a40ffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:55.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.136+0000 7f98a9f9d640 1 -- 172.21.15.169:0/3039184763 >> 172.21.15.169:0/3039184763 conn(0x7f98a40fb2e0 msgr2=0x7f98a40fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:55.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.136+0000 7f98a9f9d640 1 -- 172.21.15.169:0/3039184763 shutdown_connections 2024-03-28T00:54:55.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.136+0000 7f98a9f9d640 1 -- 172.21.15.169:0/3039184763 wait complete. 2024-03-28T00:54:55.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.137+0000 7f98a9f9d640 1 Processor -- start 2024-03-28T00:54:55.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.137+0000 7f98a9f9d640 1 -- start start 2024-03-28T00:54:55.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.137+0000 7f98a9f9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98a40ff9b0 0x7f98a4197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:55.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.137+0000 7f98a9f9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98a4102980 0x7f98a4197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:55.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.137+0000 7f98a9f9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98a4103340 0x7f98a419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:55.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.137+0000 7f98a9f9d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a410d370 con 0x7f98a4103340 2024-03-28T00:54:55.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.137+0000 7f98a9f9d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a410d1f0 con 0x7f98a40ff9b0 2024-03-28T00:54:55.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.137+0000 7f98a9f9d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a410d4f0 con 0x7f98a4102980 2024-03-28T00:54:55.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.137+0000 7f98a3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98a4103340 0x7f98a419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:55.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.137+0000 7f98a3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98a4103340 0x7f98a419efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38994/0 (socket says 172.21.15.169:38994) 2024-03-28T00:54:55.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.137+0000 7f98a3fff640 1 -- 172.21.15.169:0/3699622264 learned_addr learned my addr 172.21.15.169:0/3699622264 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:55.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.137+0000 7f98a37fe640 1 --2- 172.21.15.169:0/3699622264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98a40ff9b0 0x7f98a4197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:55.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.137+0000 7f989bfff640 1 --2- 172.21.15.169:0/3699622264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98a4102980 0x7f98a4197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:55.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.137+0000 7f98a3fff640 1 -- 172.21.15.169:0/3699622264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98a4102980 msgr2=0x7f98a4197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:55.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.137+0000 7f98a3fff640 1 --2- 172.21.15.169:0/3699622264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98a4102980 0x7f98a4197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:55.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.137+0000 7f98a3fff640 1 -- 172.21.15.169:0/3699622264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98a40ff9b0 msgr2=0x7f98a4197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:55.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.137+0000 7f98a3fff640 1 --2- 172.21.15.169:0/3699622264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98a40ff9b0 0x7f98a4197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:55.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.137+0000 7f98a3fff640 1 -- 172.21.15.169:0/3699622264 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a419f650 con 0x7f98a4103340 2024-03-28T00:54:55.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.138+0000 7f98a3fff640 1 --2- 172.21.15.169:0/3699622264 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98a4103340 0x7f98a419efe0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f989c00eb10 tx=0x7f989c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:55.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.138+0000 7f98a17fa640 1 -- 172.21.15.169:0/3699622264 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c004b20 con 0x7f98a4103340 2024-03-28T00:54:55.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.138+0000 7f98a17fa640 1 -- 172.21.15.169:0/3699622264 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f989c004cc0 con 0x7f98a4103340 2024-03-28T00:54:55.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.138+0000 7f98a9f9d640 1 -- 172.21.15.169:0/3699622264 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a419f940 con 0x7f98a4103340 2024-03-28T00:54:55.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.138+0000 7f98a17fa640 1 -- 172.21.15.169:0/3699622264 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c010760 con 0x7f98a4103340 2024-03-28T00:54:55.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.138+0000 7f98a9f9d640 1 -- 172.21.15.169:0/3699622264 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a419fe10 con 0x7f98a4103340 2024-03-28T00:54:55.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.139+0000 7f98a9f9d640 1 -- 172.21.15.169:0/3699622264 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a4108b80 con 0x7f98a4103340 2024-03-28T00:54:55.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.140+0000 7f98a17fa640 1 -- 172.21.15.169:0/3699622264 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f989c005020 con 0x7f98a4103340 2024-03-28T00:54:55.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.141+0000 7f98a17fa640 1 --2- 172.21.15.169:0/3699622264 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f98900a2800 0x7f98900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:55.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.141+0000 7f98a37fe640 1 --2- 172.21.15.169:0/3699622264 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f98900a2800 0x7f98900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:55.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.141+0000 7f98a17fa640 1 -- 172.21.15.169:0/3699622264 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f989c0cf210 con 0x7f98a4103340 2024-03-28T00:54:55.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.141+0000 7f98a37fe640 1 --2- 172.21.15.169:0/3699622264 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f98900a2800 0x7f98900a4cc0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f98a4069af0 tx=0x7f9894005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:55.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.145+0000 7f98a17fa640 1 -- 172.21.15.169:0/3699622264 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f989c09c070 con 0x7f98a4103340 2024-03-28T00:54:55.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.291+0000 7f98a17fa640 1 -- 172.21.15.169:0/3699622264 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f989c097fe0 con 0x7f98a4103340 2024-03-28T00:54:55.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.323+0000 7f98a9f9d640 1 -- 172.21.15.169:0/3699622264 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f98a4061ae0 con 0x7f98900a2800 2024-03-28T00:54:55.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.332+0000 7f98a17fa640 1 -- 172.21.15.169:0/3699622264 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340604 (secure 0 0 0) 0x7f98a4061ae0 con 0x7f98900a2800 2024-03-28T00:54:55.340 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:54:55.342 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-28T00:54:55.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.336+0000 7f98a9f9d640 1 -- 172.21.15.169:0/3699622264 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f98900a2800 msgr2=0x7f98900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:55.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.336+0000 7f98a9f9d640 1 --2- 172.21.15.169:0/3699622264 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f98900a2800 0x7f98900a4cc0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f98a4069af0 tx=0x7f9894005f90 comp rx=0 tx=0).stop 2024-03-28T00:54:55.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.336+0000 7f98a9f9d640 1 -- 172.21.15.169:0/3699622264 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98a4103340 msgr2=0x7f98a419efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:55.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.336+0000 7f98a9f9d640 1 --2- 172.21.15.169:0/3699622264 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98a4103340 0x7f98a419efe0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f989c00eb10 tx=0x7f989c00c2d0 comp rx=0 tx=0).stop 2024-03-28T00:54:55.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.336+0000 7f98a9f9d640 1 -- 172.21.15.169:0/3699622264 shutdown_connections 2024-03-28T00:54:55.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.336+0000 7f98a9f9d640 1 --2- 172.21.15.169:0/3699622264 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f98900a2800 0x7f98900a4cc0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:55.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.336+0000 7f98a9f9d640 1 --2- 172.21.15.169:0/3699622264 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98a4103340 0x7f98a419efe0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:55.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.336+0000 7f98a9f9d640 1 --2- 172.21.15.169:0/3699622264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98a4102980 0x7f98a4197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:55.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.336+0000 7f98a9f9d640 1 --2- 172.21.15.169:0/3699622264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98a40ff9b0 0x7f98a4197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:55.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.336+0000 7f98a9f9d640 1 -- 172.21.15.169:0/3699622264 >> 172.21.15.169:0/3699622264 conn(0x7f98a40fb2e0 msgr2=0x7f98a4105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:55.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.336+0000 7f98a9f9d640 1 -- 172.21.15.169:0/3699622264 shutdown_connections 2024-03-28T00:54:55.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:55.336+0000 7f98a9f9d640 1 -- 172.21.15.169:0/3699622264 wait complete. 2024-03-28T00:54:55.350 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-03-28T00:54:54.084888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596531,"num_objects":45,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":131,"num_read_kb":116,"num_write":275,"num_write_kb":2969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":220,"ondisk_log_size":220,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356752,"kb_used_data":11736,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585072,"statfs":{"total":1151940427776,"available":1151575113728,"internally_reserved":0,"allocated":12017664,"data_stored":7637183,"data_compressed":319182,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3753,"num_objects":21,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":49,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003184"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.675896+0000","last_change":"2024-03-28T00:54:32.809951+0000","last_active":"2024-03-28T00:54:39.675896+0000","last_peered":"2024-03-28T00:54:39.675896+0000","last_clean":"2024-03-28T00:54:39.675896+0000","last_became_active":"2024-03-28T00:54:32.808871+0000","last_became_peered":"2024-03-28T00:54:32.808871+0000","last_unstale":"2024-03-28T00:54:39.675896+0000","last_undegraded":"2024-03-28T00:54:39.675896+0000","last_fullsized":"2024-03-28T00:54:39.675896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:15:17.152487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511120+0000","last_change":"2024-03-28T00:54:32.816707+0000","last_active":"2024-03-28T00:54:39.511120+0000","last_peered":"2024-03-28T00:54:39.511120+0000","last_clean":"2024-03-28T00:54:39.511120+0000","last_became_active":"2024-03-28T00:54:32.816000+0000","last_became_peered":"2024-03-28T00:54:32.816000+0000","last_unstale":"2024-03-28T00:54:39.511120+0000","last_undegraded":"2024-03-28T00:54:39.511120+0000","last_fullsized":"2024-03-28T00:54:39.511120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:56:19.532808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510653+0000","last_change":"2024-03-28T00:54:32.817023+0000","last_active":"2024-03-28T00:54:39.510653+0000","last_peered":"2024-03-28T00:54:39.510653+0000","last_clean":"2024-03-28T00:54:39.510653+0000","last_became_active":"2024-03-28T00:54:32.816712+0000","last_became_peered":"2024-03-28T00:54:32.816712+0000","last_unstale":"2024-03-28T00:54:39.510653+0000","last_undegraded":"2024-03-28T00:54:39.510653+0000","last_fullsized":"2024-03-28T00:54:39.510653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:13:25.334294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511180+0000","last_change":"2024-03-28T00:54:32.807259+0000","last_active":"2024-03-28T00:54:39.511180+0000","last_peered":"2024-03-28T00:54:39.511180+0000","last_clean":"2024-03-28T00:54:39.511180+0000","last_became_active":"2024-03-28T00:54:32.806944+0000","last_became_peered":"2024-03-28T00:54:32.806944+0000","last_unstale":"2024-03-28T00:54:39.511180+0000","last_undegraded":"2024-03-28T00:54:39.511180+0000","last_fullsized":"2024-03-28T00:54:39.511180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:46:04.852351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510926+0000","last_change":"2024-03-28T00:54:32.807613+0000","last_active":"2024-03-28T00:54:39.510926+0000","last_peered":"2024-03-28T00:54:39.510926+0000","last_clean":"2024-03-28T00:54:39.510926+0000","last_became_active":"2024-03-28T00:54:32.807280+0000","last_became_peered":"2024-03-28T00:54:32.807280+0000","last_unstale":"2024-03-28T00:54:39.510926+0000","last_undegraded":"2024-03-28T00:54:39.510926+0000","last_fullsized":"2024-03-28T00:54:39.510926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:56:37.133675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.148314+0000","last_change":"2024-03-28T00:54:32.806804+0000","last_active":"2024-03-28T00:54:39.148314+0000","last_peered":"2024-03-28T00:54:39.148314+0000","last_clean":"2024-03-28T00:54:39.148314+0000","last_became_active":"2024-03-28T00:54:32.805343+0000","last_became_peered":"2024-03-28T00:54:32.805343+0000","last_unstale":"2024-03-28T00:54:39.148314+0000","last_undegraded":"2024-03-28T00:54:39.148314+0000","last_fullsized":"2024-03-28T00:54:39.148314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:31:44.849926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808333+0000","last_change":"2024-03-28T00:54:32.814513+0000","last_active":"2024-03-28T00:54:38.808333+0000","last_peered":"2024-03-28T00:54:38.808333+0000","last_clean":"2024-03-28T00:54:38.808333+0000","last_became_active":"2024-03-28T00:54:32.814165+0000","last_became_peered":"2024-03-28T00:54:32.814165+0000","last_unstale":"2024-03-28T00:54:38.808333+0000","last_undegraded":"2024-03-28T00:54:38.808333+0000","last_fullsized":"2024-03-28T00:54:38.808333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:36:36.160751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510529+0000","last_change":"2024-03-28T00:54:32.808058+0000","last_active":"2024-03-28T00:54:39.510529+0000","last_peered":"2024-03-28T00:54:39.510529+0000","last_clean":"2024-03-28T00:54:39.510529+0000","last_became_active":"2024-03-28T00:54:32.807755+0000","last_became_peered":"2024-03-28T00:54:32.807755+0000","last_unstale":"2024-03-28T00:54:39.510529+0000","last_undegraded":"2024-03-28T00:54:39.510529+0000","last_fullsized":"2024-03-28T00:54:39.510529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:08:42.707015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510705+0000","last_change":"2024-03-28T00:54:32.800667+0000","last_active":"2024-03-28T00:54:39.510705+0000","last_peered":"2024-03-28T00:54:39.510705+0000","last_clean":"2024-03-28T00:54:39.510705+0000","last_became_active":"2024-03-28T00:54:32.800332+0000","last_became_peered":"2024-03-28T00:54:32.800332+0000","last_unstale":"2024-03-28T00:54:39.510705+0000","last_undegraded":"2024-03-28T00:54:39.510705+0000","last_fullsized":"2024-03-28T00:54:39.510705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:41:27.577871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.148665+0000","last_change":"2024-03-28T00:54:32.807958+0000","last_active":"2024-03-28T00:54:39.148665+0000","last_peered":"2024-03-28T00:54:39.148665+0000","last_clean":"2024-03-28T00:54:39.148665+0000","last_became_active":"2024-03-28T00:54:32.807657+0000","last_became_peered":"2024-03-28T00:54:32.807657+0000","last_unstale":"2024-03-28T00:54:39.148665+0000","last_undegraded":"2024-03-28T00:54:39.148665+0000","last_fullsized":"2024-03-28T00:54:39.148665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:30:28.000690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808607+0000","last_change":"2024-03-28T00:54:32.811543+0000","last_active":"2024-03-28T00:54:38.808607+0000","last_peered":"2024-03-28T00:54:38.808607+0000","last_clean":"2024-03-28T00:54:38.808607+0000","last_became_active":"2024-03-28T00:54:32.811210+0000","last_became_peered":"2024-03-28T00:54:32.811210+0000","last_unstale":"2024-03-28T00:54:38.808607+0000","last_undegraded":"2024-03-28T00:54:38.808607+0000","last_fullsized":"2024-03-28T00:54:38.808607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:47:19.539107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511181+0000","last_change":"2024-03-28T00:54:32.812648+0000","last_active":"2024-03-28T00:54:39.511181+0000","last_peered":"2024-03-28T00:54:39.511181+0000","last_clean":"2024-03-28T00:54:39.511181+0000","last_became_active":"2024-03-28T00:54:32.812340+0000","last_became_peered":"2024-03-28T00:54:32.812340+0000","last_unstale":"2024-03-28T00:54:39.511181+0000","last_undegraded":"2024-03-28T00:54:39.511181+0000","last_fullsized":"2024-03-28T00:54:39.511181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:20:45.449953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.810970+0000","last_change":"2024-03-28T00:54:32.801807+0000","last_active":"2024-03-28T00:54:38.810970+0000","last_peered":"2024-03-28T00:54:38.810970+0000","last_clean":"2024-03-28T00:54:38.810970+0000","last_became_active":"2024-03-28T00:54:32.801498+0000","last_became_peered":"2024-03-28T00:54:32.801498+0000","last_unstale":"2024-03-28T00:54:38.810970+0000","last_undegraded":"2024-03-28T00:54:38.810970+0000","last_fullsized":"2024-03-28T00:54:38.810970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:35:35.827856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510434+0000","last_change":"2024-03-28T00:54:32.818033+0000","last_active":"2024-03-28T00:54:39.510434+0000","last_peered":"2024-03-28T00:54:39.510434+0000","last_clean":"2024-03-28T00:54:39.510434+0000","last_became_active":"2024-03-28T00:54:32.817309+0000","last_became_peered":"2024-03-28T00:54:32.817309+0000","last_unstale":"2024-03-28T00:54:39.510434+0000","last_undegraded":"2024-03-28T00:54:39.510434+0000","last_fullsized":"2024-03-28T00:54:39.510434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:39:42.682780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510512+0000","last_change":"2024-03-28T00:54:32.815224+0000","last_active":"2024-03-28T00:54:39.510512+0000","last_peered":"2024-03-28T00:54:39.510512+0000","last_clean":"2024-03-28T00:54:39.510512+0000","last_became_active":"2024-03-28T00:54:32.814936+0000","last_became_peered":"2024-03-28T00:54:32.814936+0000","last_unstale":"2024-03-28T00:54:39.510512+0000","last_undegraded":"2024-03-28T00:54:39.510512+0000","last_fullsized":"2024-03-28T00:54:39.510512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:08:40.200393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510035+0000","last_change":"2024-03-28T00:54:32.807342+0000","last_active":"2024-03-28T00:54:39.510035+0000","last_peered":"2024-03-28T00:54:39.510035+0000","last_clean":"2024-03-28T00:54:39.510035+0000","last_became_active":"2024-03-28T00:54:32.806978+0000","last_became_peered":"2024-03-28T00:54:32.806978+0000","last_unstale":"2024-03-28T00:54:39.510035+0000","last_undegraded":"2024-03-28T00:54:39.510035+0000","last_fullsized":"2024-03-28T00:54:39.510035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:17:58.599693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.675702+0000","last_change":"2024-03-28T00:54:32.810036+0000","last_active":"2024-03-28T00:54:39.675702+0000","last_peered":"2024-03-28T00:54:39.675702+0000","last_clean":"2024-03-28T00:54:39.675702+0000","last_became_active":"2024-03-28T00:54:32.809438+0000","last_became_peered":"2024-03-28T00:54:32.809438+0000","last_unstale":"2024-03-28T00:54:39.675702+0000","last_undegraded":"2024-03-28T00:54:39.675702+0000","last_fullsized":"2024-03-28T00:54:39.675702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:18:50.447474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.290406+0000","last_change":"2024-03-28T00:54:32.815398+0000","last_active":"2024-03-28T00:54:39.290406+0000","last_peered":"2024-03-28T00:54:39.290406+0000","last_clean":"2024-03-28T00:54:39.290406+0000","last_became_active":"2024-03-28T00:54:32.815045+0000","last_became_peered":"2024-03-28T00:54:32.815045+0000","last_unstale":"2024-03-28T00:54:39.290406+0000","last_undegraded":"2024-03-28T00:54:39.290406+0000","last_fullsized":"2024-03-28T00:54:39.290406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:40:29.060217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511901+0000","last_change":"2024-03-28T00:54:32.805736+0000","last_active":"2024-03-28T00:54:39.511901+0000","last_peered":"2024-03-28T00:54:39.511901+0000","last_clean":"2024-03-28T00:54:39.511901+0000","last_became_active":"2024-03-28T00:54:32.805435+0000","last_became_peered":"2024-03-28T00:54:32.805435+0000","last_unstale":"2024-03-28T00:54:39.511901+0000","last_undegraded":"2024-03-28T00:54:39.511901+0000","last_fullsized":"2024-03-28T00:54:39.511901+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:24:30.893583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511476+0000","last_change":"2024-03-28T00:54:36.824531+0000","last_active":"2024-03-28T00:54:39.511476+0000","last_peered":"2024-03-28T00:54:39.511476+0000","last_clean":"2024-03-28T00:54:39.511476+0000","last_became_active":"2024-03-28T00:54:36.824123+0000","last_became_peered":"2024-03-28T00:54:36.824123+0000","last_unstale":"2024-03-28T00:54:39.511476+0000","last_undegraded":"2024-03-28T00:54:39.511476+0000","last_fullsized":"2024-03-28T00:54:39.511476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:10:47.503171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.290598+0000","last_change":"2024-03-28T00:54:32.256226+0000","last_active":"2024-03-28T00:54:39.290598+0000","last_peered":"2024-03-28T00:54:39.290598+0000","last_clean":"2024-03-28T00:54:39.290598+0000","last_became_active":"2024-03-28T00:54:32.255921+0000","last_became_peered":"2024-03-28T00:54:32.255921+0000","last_unstale":"2024-03-28T00:54:39.290598+0000","last_undegraded":"2024-03-28T00:54:39.290598+0000","last_fullsized":"2024-03-28T00:54:39.290598+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:56:04.196311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511112+0000","last_change":"2024-03-28T00:54:32.808248+0000","last_active":"2024-03-28T00:54:39.511112+0000","last_peered":"2024-03-28T00:54:39.511112+0000","last_clean":"2024-03-28T00:54:39.511112+0000","last_became_active":"2024-03-28T00:54:32.807923+0000","last_became_peered":"2024-03-28T00:54:32.807923+0000","last_unstale":"2024-03-28T00:54:39.511112+0000","last_undegraded":"2024-03-28T00:54:39.511112+0000","last_fullsized":"2024-03-28T00:54:39.511112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:59:56.002564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510503+0000","last_change":"2024-03-28T00:54:36.830556+0000","last_active":"2024-03-28T00:54:39.510503+0000","last_peered":"2024-03-28T00:54:39.510503+0000","last_clean":"2024-03-28T00:54:39.510503+0000","last_became_active":"2024-03-28T00:54:36.830151+0000","last_became_peered":"2024-03-28T00:54:36.830151+0000","last_unstale":"2024-03-28T00:54:39.510503+0000","last_undegraded":"2024-03-28T00:54:39.510503+0000","last_fullsized":"2024-03-28T00:54:39.510503+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:19:23.073410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511484+0000","last_change":"2024-03-28T00:54:32.260635+0000","last_active":"2024-03-28T00:54:39.511484+0000","last_peered":"2024-03-28T00:54:39.511484+0000","last_clean":"2024-03-28T00:54:39.511484+0000","last_became_active":"2024-03-28T00:54:32.260084+0000","last_became_peered":"2024-03-28T00:54:32.260084+0000","last_unstale":"2024-03-28T00:54:39.511484+0000","last_undegraded":"2024-03-28T00:54:39.511484+0000","last_fullsized":"2024-03-28T00:54:39.511484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:06.858519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511232+0000","last_change":"2024-03-28T00:54:32.809901+0000","last_active":"2024-03-28T00:54:39.511232+0000","last_peered":"2024-03-28T00:54:39.511232+0000","last_clean":"2024-03-28T00:54:39.511232+0000","last_became_active":"2024-03-28T00:54:32.809557+0000","last_became_peered":"2024-03-28T00:54:32.809557+0000","last_unstale":"2024-03-28T00:54:39.511232+0000","last_undegraded":"2024-03-28T00:54:39.511232+0000","last_fullsized":"2024-03-28T00:54:39.511232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:05:31.363487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510608+0000","last_change":"2024-03-28T00:54:36.824661+0000","last_active":"2024-03-28T00:54:39.510608+0000","last_peered":"2024-03-28T00:54:39.510608+0000","last_clean":"2024-03-28T00:54:39.510608+0000","last_became_active":"2024-03-28T00:54:36.824223+0000","last_became_peered":"2024-03-28T00:54:36.824223+0000","last_unstale":"2024-03-28T00:54:39.510608+0000","last_undegraded":"2024-03-28T00:54:39.510608+0000","last_fullsized":"2024-03-28T00:54:39.510608+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:02:25.509248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511745+0000","last_change":"2024-03-28T00:54:32.256436+0000","last_active":"2024-03-28T00:54:39.511745+0000","last_peered":"2024-03-28T00:54:39.511745+0000","last_clean":"2024-03-28T00:54:39.511745+0000","last_became_active":"2024-03-28T00:54:32.256069+0000","last_became_peered":"2024-03-28T00:54:32.256069+0000","last_unstale":"2024-03-28T00:54:39.511745+0000","last_undegraded":"2024-03-28T00:54:39.511745+0000","last_fullsized":"2024-03-28T00:54:39.511745+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:43.688192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510343+0000","last_change":"2024-03-28T00:54:32.799269+0000","last_active":"2024-03-28T00:54:39.510343+0000","last_peered":"2024-03-28T00:54:39.510343+0000","last_clean":"2024-03-28T00:54:39.510343+0000","last_became_active":"2024-03-28T00:54:32.798848+0000","last_became_peered":"2024-03-28T00:54:32.798848+0000","last_unstale":"2024-03-28T00:54:39.510343+0000","last_undegraded":"2024-03-28T00:54:39.510343+0000","last_fullsized":"2024-03-28T00:54:39.510343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:56:20.318466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510479+0000","last_change":"2024-03-28T00:54:36.823153+0000","last_active":"2024-03-28T00:54:39.510479+0000","last_peered":"2024-03-28T00:54:39.510479+0000","last_clean":"2024-03-28T00:54:39.510479+0000","last_became_active":"2024-03-28T00:54:36.822817+0000","last_became_peered":"2024-03-28T00:54:36.822817+0000","last_unstale":"2024-03-28T00:54:39.510479+0000","last_undegraded":"2024-03-28T00:54:39.510479+0000","last_fullsized":"2024-03-28T00:54:39.510479+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:20:23.001599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511059+0000","last_change":"2024-03-28T00:54:32.250439+0000","last_active":"2024-03-28T00:54:39.511059+0000","last_peered":"2024-03-28T00:54:39.511059+0000","last_clean":"2024-03-28T00:54:39.511059+0000","last_became_active":"2024-03-28T00:54:32.250006+0000","last_became_peered":"2024-03-28T00:54:32.250006+0000","last_unstale":"2024-03-28T00:54:39.511059+0000","last_undegraded":"2024-03-28T00:54:39.511059+0000","last_fullsized":"2024-03-28T00:54:39.511059+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:31:38.957429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.809145+0000","last_change":"2024-03-28T00:54:32.806888+0000","last_active":"2024-03-28T00:54:38.809145+0000","last_peered":"2024-03-28T00:54:38.809145+0000","last_clean":"2024-03-28T00:54:38.809145+0000","last_became_active":"2024-03-28T00:54:32.806581+0000","last_became_peered":"2024-03-28T00:54:32.806581+0000","last_unstale":"2024-03-28T00:54:38.809145+0000","last_undegraded":"2024-03-28T00:54:38.809145+0000","last_fullsized":"2024-03-28T00:54:38.809145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:25:52.331819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511783+0000","last_change":"2024-03-28T00:54:36.824041+0000","last_active":"2024-03-28T00:54:39.511783+0000","last_peered":"2024-03-28T00:54:39.511783+0000","last_clean":"2024-03-28T00:54:39.511783+0000","last_became_active":"2024-03-28T00:54:36.823658+0000","last_became_peered":"2024-03-28T00:54:36.823658+0000","last_unstale":"2024-03-28T00:54:39.511783+0000","last_undegraded":"2024-03-28T00:54:39.511783+0000","last_fullsized":"2024-03-28T00:54:39.511783+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:11:03.236009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.273501+0000","last_change":"2024-03-28T00:54:32.257534+0000","last_active":"2024-03-28T00:54:50.273501+0000","last_peered":"2024-03-28T00:54:50.273501+0000","last_clean":"2024-03-28T00:54:50.273501+0000","last_became_active":"2024-03-28T00:54:32.256404+0000","last_became_peered":"2024-03-28T00:54:32.256404+0000","last_unstale":"2024-03-28T00:54:50.273501+0000","last_undegraded":"2024-03-28T00:54:50.273501+0000","last_fullsized":"2024-03-28T00:54:50.273501+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:39:38.329799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510814+0000","last_change":"2024-03-28T00:54:32.806099+0000","last_active":"2024-03-28T00:54:39.510814+0000","last_peered":"2024-03-28T00:54:39.510814+0000","last_clean":"2024-03-28T00:54:39.510814+0000","last_became_active":"2024-03-28T00:54:32.805811+0000","last_became_peered":"2024-03-28T00:54:32.805811+0000","last_unstale":"2024-03-28T00:54:39.510814+0000","last_undegraded":"2024-03-28T00:54:39.510814+0000","last_fullsized":"2024-03-28T00:54:39.510814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:42:32.820245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511596+0000","last_change":"2024-03-28T00:54:36.827198+0000","last_active":"2024-03-28T00:54:39.511596+0000","last_peered":"2024-03-28T00:54:39.511596+0000","last_clean":"2024-03-28T00:54:39.511596+0000","last_became_active":"2024-03-28T00:54:36.826804+0000","last_became_peered":"2024-03-28T00:54:36.826804+0000","last_unstale":"2024-03-28T00:54:39.511596+0000","last_undegraded":"2024-03-28T00:54:39.511596+0000","last_fullsized":"2024-03-28T00:54:39.511596+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:38:11.030404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:51.278214+0000","last_change":"2024-03-28T00:54:32.249394+0000","last_active":"2024-03-28T00:54:51.278214+0000","last_peered":"2024-03-28T00:54:51.278214+0000","last_clean":"2024-03-28T00:54:51.278214+0000","last_became_active":"2024-03-28T00:54:32.249089+0000","last_became_peered":"2024-03-28T00:54:32.249089+0000","last_unstale":"2024-03-28T00:54:51.278214+0000","last_undegraded":"2024-03-28T00:54:51.278214+0000","last_fullsized":"2024-03-28T00:54:51.278214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:05:01.720053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808932+0000","last_change":"2024-03-28T00:54:32.809133+0000","last_active":"2024-03-28T00:54:38.808932+0000","last_peered":"2024-03-28T00:54:38.808932+0000","last_clean":"2024-03-28T00:54:38.808932+0000","last_became_active":"2024-03-28T00:54:32.808841+0000","last_became_peered":"2024-03-28T00:54:32.808841+0000","last_unstale":"2024-03-28T00:54:38.808932+0000","last_undegraded":"2024-03-28T00:54:38.808932+0000","last_fullsized":"2024-03-28T00:54:38.808932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:19:37.859542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.675758+0000","last_change":"2024-03-28T00:54:36.824483+0000","last_active":"2024-03-28T00:54:39.675758+0000","last_peered":"2024-03-28T00:54:39.675758+0000","last_clean":"2024-03-28T00:54:39.675758+0000","last_became_active":"2024-03-28T00:54:36.823973+0000","last_became_peered":"2024-03-28T00:54:36.823973+0000","last_unstale":"2024-03-28T00:54:39.675758+0000","last_undegraded":"2024-03-28T00:54:39.675758+0000","last_fullsized":"2024-03-28T00:54:39.675758+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:01:06.251119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.810690+0000","last_change":"2024-03-28T00:54:32.251287+0000","last_active":"2024-03-28T00:54:38.810690+0000","last_peered":"2024-03-28T00:54:38.810690+0000","last_clean":"2024-03-28T00:54:38.810690+0000","last_became_active":"2024-03-28T00:54:32.250904+0000","last_became_peered":"2024-03-28T00:54:32.250904+0000","last_unstale":"2024-03-28T00:54:38.810690+0000","last_undegraded":"2024-03-28T00:54:38.810690+0000","last_fullsized":"2024-03-28T00:54:38.810690+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:18:09.051056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510973+0000","last_change":"2024-03-28T00:54:32.803124+0000","last_active":"2024-03-28T00:54:39.510973+0000","last_peered":"2024-03-28T00:54:39.510973+0000","last_clean":"2024-03-28T00:54:39.510973+0000","last_became_active":"2024-03-28T00:54:32.802825+0000","last_became_peered":"2024-03-28T00:54:32.802825+0000","last_unstale":"2024-03-28T00:54:39.510973+0000","last_undegraded":"2024-03-28T00:54:39.510973+0000","last_fullsized":"2024-03-28T00:54:39.510973+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:34:21.825314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511236+0000","last_change":"2024-03-28T00:54:36.832151+0000","last_active":"2024-03-28T00:54:39.511236+0000","last_peered":"2024-03-28T00:54:39.511236+0000","last_clean":"2024-03-28T00:54:39.511236+0000","last_became_active":"2024-03-28T00:54:36.831748+0000","last_became_peered":"2024-03-28T00:54:36.831748+0000","last_unstale":"2024-03-28T00:54:39.511236+0000","last_undegraded":"2024-03-28T00:54:39.511236+0000","last_fullsized":"2024-03-28T00:54:39.511236+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:45:32.284386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808616+0000","last_change":"2024-03-28T00:54:32.259048+0000","last_active":"2024-03-28T00:54:38.808616+0000","last_peered":"2024-03-28T00:54:38.808616+0000","last_clean":"2024-03-28T00:54:38.808616+0000","last_became_active":"2024-03-28T00:54:32.258038+0000","last_became_peered":"2024-03-28T00:54:32.258038+0000","last_unstale":"2024-03-28T00:54:38.808616+0000","last_undegraded":"2024-03-28T00:54:38.808616+0000","last_fullsized":"2024-03-28T00:54:38.808616+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:45:23.525516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.290042+0000","last_change":"2024-03-28T00:54:32.809771+0000","last_active":"2024-03-28T00:54:39.290042+0000","last_peered":"2024-03-28T00:54:39.290042+0000","last_clean":"2024-03-28T00:54:39.290042+0000","last_became_active":"2024-03-28T00:54:32.809467+0000","last_became_peered":"2024-03-28T00:54:32.809467+0000","last_unstale":"2024-03-28T00:54:39.290042+0000","last_undegraded":"2024-03-28T00:54:39.290042+0000","last_fullsized":"2024-03-28T00:54:39.290042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:30:15.440202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511045+0000","last_change":"2024-03-28T00:54:36.823313+0000","last_active":"2024-03-28T00:54:39.511045+0000","last_peered":"2024-03-28T00:54:39.511045+0000","last_clean":"2024-03-28T00:54:39.511045+0000","last_became_active":"2024-03-28T00:54:36.822923+0000","last_became_peered":"2024-03-28T00:54:36.822923+0000","last_unstale":"2024-03-28T00:54:39.511045+0000","last_undegraded":"2024-03-28T00:54:39.511045+0000","last_fullsized":"2024-03-28T00:54:39.511045+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:22:21.125028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.272884+0000","last_change":"2024-03-28T00:54:32.255084+0000","last_active":"2024-03-28T00:54:50.272884+0000","last_peered":"2024-03-28T00:54:50.272884+0000","last_clean":"2024-03-28T00:54:50.272884+0000","last_became_active":"2024-03-28T00:54:32.254744+0000","last_became_peered":"2024-03-28T00:54:32.254744+0000","last_unstale":"2024-03-28T00:54:50.272884+0000","last_undegraded":"2024-03-28T00:54:50.272884+0000","last_fullsized":"2024-03-28T00:54:50.272884+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:32:03.906826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510832+0000","last_change":"2024-03-28T00:54:32.810050+0000","last_active":"2024-03-28T00:54:39.510832+0000","last_peered":"2024-03-28T00:54:39.510832+0000","last_clean":"2024-03-28T00:54:39.510832+0000","last_became_active":"2024-03-28T00:54:32.809731+0000","last_became_peered":"2024-03-28T00:54:32.809731+0000","last_unstale":"2024-03-28T00:54:39.510832+0000","last_undegraded":"2024-03-28T00:54:39.510832+0000","last_fullsized":"2024-03-28T00:54:39.510832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:20:11.876731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.289968+0000","last_change":"2024-03-28T00:54:36.826224+0000","last_active":"2024-03-28T00:54:39.289968+0000","last_peered":"2024-03-28T00:54:39.289968+0000","last_clean":"2024-03-28T00:54:39.289968+0000","last_became_active":"2024-03-28T00:54:36.825853+0000","last_became_peered":"2024-03-28T00:54:36.825853+0000","last_unstale":"2024-03-28T00:54:39.289968+0000","last_undegraded":"2024-03-28T00:54:39.289968+0000","last_fullsized":"2024-03-28T00:54:39.289968+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:32:09.131482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.271148+0000","last_change":"2024-03-28T00:54:32.259159+0000","last_active":"2024-03-28T00:54:50.271148+0000","last_peered":"2024-03-28T00:54:50.271148+0000","last_clean":"2024-03-28T00:54:50.271148+0000","last_became_active":"2024-03-28T00:54:32.258708+0000","last_became_peered":"2024-03-28T00:54:32.258708+0000","last_unstale":"2024-03-28T00:54:50.271148+0000","last_undegraded":"2024-03-28T00:54:50.271148+0000","last_fullsized":"2024-03-28T00:54:50.271148+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:37:01.524878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511134+0000","last_change":"2024-03-28T00:54:32.814358+0000","last_active":"2024-03-28T00:54:39.511134+0000","last_peered":"2024-03-28T00:54:39.511134+0000","last_clean":"2024-03-28T00:54:39.511134+0000","last_became_active":"2024-03-28T00:54:32.814018+0000","last_became_peered":"2024-03-28T00:54:32.814018+0000","last_unstale":"2024-03-28T00:54:39.511134+0000","last_undegraded":"2024-03-28T00:54:39.511134+0000","last_fullsized":"2024-03-28T00:54:39.511134+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:24:38.287853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510938+0000","last_change":"2024-03-28T00:54:36.822161+0000","last_active":"2024-03-28T00:54:39.510938+0000","last_peered":"2024-03-28T00:54:39.510938+0000","last_clean":"2024-03-28T00:54:39.510938+0000","last_became_active":"2024-03-28T00:54:36.821750+0000","last_became_peered":"2024-03-28T00:54:36.821750+0000","last_unstale":"2024-03-28T00:54:39.510938+0000","last_undegraded":"2024-03-28T00:54:39.510938+0000","last_fullsized":"2024-03-28T00:54:39.510938+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:13:36.892453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.809075+0000","last_change":"2024-03-28T00:54:32.257038+0000","last_active":"2024-03-28T00:54:38.809075+0000","last_peered":"2024-03-28T00:54:38.809075+0000","last_clean":"2024-03-28T00:54:38.809075+0000","last_became_active":"2024-03-28T00:54:32.256702+0000","last_became_peered":"2024-03-28T00:54:32.256702+0000","last_unstale":"2024-03-28T00:54:38.809075+0000","last_undegraded":"2024-03-28T00:54:38.809075+0000","last_fullsized":"2024-03-28T00:54:38.809075+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:35:17.294116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510425+0000","last_change":"2024-03-28T00:54:32.813514+0000","last_active":"2024-03-28T00:54:39.510425+0000","last_peered":"2024-03-28T00:54:39.510425+0000","last_clean":"2024-03-28T00:54:39.510425+0000","last_became_active":"2024-03-28T00:54:32.813192+0000","last_became_peered":"2024-03-28T00:54:32.813192+0000","last_unstale":"2024-03-28T00:54:39.510425+0000","last_undegraded":"2024-03-28T00:54:39.510425+0000","last_fullsized":"2024-03-28T00:54:39.510425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:55:03.410818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.290178+0000","last_change":"2024-03-28T00:54:36.820995+0000","last_active":"2024-03-28T00:54:39.290178+0000","last_peered":"2024-03-28T00:54:39.290178+0000","last_clean":"2024-03-28T00:54:39.290178+0000","last_became_active":"2024-03-28T00:54:36.820628+0000","last_became_peered":"2024-03-28T00:54:36.820628+0000","last_unstale":"2024-03-28T00:54:39.290178+0000","last_undegraded":"2024-03-28T00:54:39.290178+0000","last_fullsized":"2024-03-28T00:54:39.290178+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:18:58.771527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'11","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:52.281220+0000","last_change":"2024-03-28T00:54:32.251092+0000","last_active":"2024-03-28T00:54:52.281220+0000","last_peered":"2024-03-28T00:54:52.281220+0000","last_clean":"2024-03-28T00:54:52.281220+0000","last_became_active":"2024-03-28T00:54:32.250759+0000","last_became_peered":"2024-03-28T00:54:32.250759+0000","last_unstale":"2024-03-28T00:54:52.281220+0000","last_undegraded":"2024-03-28T00:54:52.281220+0000","last_fullsized":"2024-03-28T00:54:52.281220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:57:30.283365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":12,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511534+0000","last_change":"2024-03-28T00:54:32.809580+0000","last_active":"2024-03-28T00:54:39.511534+0000","last_peered":"2024-03-28T00:54:39.511534+0000","last_clean":"2024-03-28T00:54:39.511534+0000","last_became_active":"2024-03-28T00:54:32.809238+0000","last_became_peered":"2024-03-28T00:54:32.809238+0000","last_unstale":"2024-03-28T00:54:39.511534+0000","last_undegraded":"2024-03-28T00:54:39.511534+0000","last_fullsized":"2024-03-28T00:54:39.511534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:20:17.397594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.809524+0000","last_change":"2024-03-28T00:54:36.826851+0000","last_active":"2024-03-28T00:54:38.809524+0000","last_peered":"2024-03-28T00:54:38.809524+0000","last_clean":"2024-03-28T00:54:38.809524+0000","last_became_active":"2024-03-28T00:54:36.826359+0000","last_became_peered":"2024-03-28T00:54:36.826359+0000","last_unstale":"2024-03-28T00:54:38.809524+0000","last_undegraded":"2024-03-28T00:54:38.809524+0000","last_fullsized":"2024-03-28T00:54:38.809524+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:27:48.857368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.271112+0000","last_change":"2024-03-28T00:54:32.251701+0000","last_active":"2024-03-28T00:54:50.271112+0000","last_peered":"2024-03-28T00:54:50.271112+0000","last_clean":"2024-03-28T00:54:50.271112+0000","last_became_active":"2024-03-28T00:54:32.251403+0000","last_became_peered":"2024-03-28T00:54:32.251403+0000","last_unstale":"2024-03-28T00:54:50.271112+0000","last_undegraded":"2024-03-28T00:54:50.271112+0000","last_fullsized":"2024-03-28T00:54:50.271112+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:28:16.127643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.810795+0000","last_change":"2024-03-28T00:54:32.806810+0000","last_active":"2024-03-28T00:54:38.810795+0000","last_peered":"2024-03-28T00:54:38.810795+0000","last_clean":"2024-03-28T00:54:38.810795+0000","last_became_active":"2024-03-28T00:54:32.806468+0000","last_became_peered":"2024-03-28T00:54:32.806468+0000","last_unstale":"2024-03-28T00:54:38.810795+0000","last_undegraded":"2024-03-28T00:54:38.810795+0000","last_fullsized":"2024-03-28T00:54:38.810795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:54:16.196032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.675486+0000","last_change":"2024-03-28T00:54:36.827450+0000","last_active":"2024-03-28T00:54:39.675486+0000","last_peered":"2024-03-28T00:54:39.675486+0000","last_clean":"2024-03-28T00:54:39.675486+0000","last_became_active":"2024-03-28T00:54:36.826837+0000","last_became_peered":"2024-03-28T00:54:36.826837+0000","last_unstale":"2024-03-28T00:54:39.675486+0000","last_undegraded":"2024-03-28T00:54:39.675486+0000","last_fullsized":"2024-03-28T00:54:39.675486+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:00:57.813913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.271705+0000","last_change":"2024-03-28T00:54:32.259440+0000","last_active":"2024-03-28T00:54:50.271705+0000","last_peered":"2024-03-28T00:54:50.271705+0000","last_clean":"2024-03-28T00:54:50.271705+0000","last_became_active":"2024-03-28T00:54:32.258225+0000","last_became_peered":"2024-03-28T00:54:32.258225+0000","last_unstale":"2024-03-28T00:54:50.271705+0000","last_undegraded":"2024-03-28T00:54:50.271705+0000","last_fullsized":"2024-03-28T00:54:50.271705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:07:18.398302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808358+0000","last_change":"2024-03-28T00:54:32.806853+0000","last_active":"2024-03-28T00:54:38.808358+0000","last_peered":"2024-03-28T00:54:38.808358+0000","last_clean":"2024-03-28T00:54:38.808358+0000","last_became_active":"2024-03-28T00:54:32.806551+0000","last_became_peered":"2024-03-28T00:54:32.806551+0000","last_unstale":"2024-03-28T00:54:38.808358+0000","last_undegraded":"2024-03-28T00:54:38.808358+0000","last_fullsized":"2024-03-28T00:54:38.808358+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:49:45.774252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.290516+0000","last_change":"2024-03-28T00:54:36.827542+0000","last_active":"2024-03-28T00:54:39.290516+0000","last_peered":"2024-03-28T00:54:39.290516+0000","last_clean":"2024-03-28T00:54:39.290516+0000","last_became_active":"2024-03-28T00:54:36.827163+0000","last_became_peered":"2024-03-28T00:54:36.827163+0000","last_unstale":"2024-03-28T00:54:39.290516+0000","last_undegraded":"2024-03-28T00:54:39.290516+0000","last_fullsized":"2024-03-28T00:54:39.290516+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:36:15.278111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511011+0000","last_change":"2024-03-28T00:54:32.253231+0000","last_active":"2024-03-28T00:54:39.511011+0000","last_peered":"2024-03-28T00:54:39.511011+0000","last_clean":"2024-03-28T00:54:39.511011+0000","last_became_active":"2024-03-28T00:54:32.252731+0000","last_became_peered":"2024-03-28T00:54:32.252731+0000","last_unstale":"2024-03-28T00:54:39.511011+0000","last_undegraded":"2024-03-28T00:54:39.511011+0000","last_fullsized":"2024-03-28T00:54:39.511011+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:05:01.716654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808635+0000","last_change":"2024-03-28T00:54:32.805073+0000","last_active":"2024-03-28T00:54:38.808635+0000","last_peered":"2024-03-28T00:54:38.808635+0000","last_clean":"2024-03-28T00:54:38.808635+0000","last_became_active":"2024-03-28T00:54:32.803690+0000","last_became_peered":"2024-03-28T00:54:32.803690+0000","last_unstale":"2024-03-28T00:54:38.808635+0000","last_undegraded":"2024-03-28T00:54:38.808635+0000","last_fullsized":"2024-03-28T00:54:38.808635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:41:16.586168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.811287+0000","last_change":"2024-03-28T00:54:36.835463+0000","last_active":"2024-03-28T00:54:38.811287+0000","last_peered":"2024-03-28T00:54:38.811287+0000","last_clean":"2024-03-28T00:54:38.811287+0000","last_became_active":"2024-03-28T00:54:36.834982+0000","last_became_peered":"2024-03-28T00:54:36.834982+0000","last_unstale":"2024-03-28T00:54:38.811287+0000","last_undegraded":"2024-03-28T00:54:38.811287+0000","last_fullsized":"2024-03-28T00:54:38.811287+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:27:32.259586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'10","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.271666+0000","last_change":"2024-03-28T00:54:32.248461+0000","last_active":"2024-03-28T00:54:50.271666+0000","last_peered":"2024-03-28T00:54:50.271666+0000","last_clean":"2024-03-28T00:54:50.271666+0000","last_became_active":"2024-03-28T00:54:32.248140+0000","last_became_peered":"2024-03-28T00:54:32.248140+0000","last_unstale":"2024-03-28T00:54:50.271666+0000","last_undegraded":"2024-03-28T00:54:50.271666+0000","last_fullsized":"2024-03-28T00:54:50.271666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:13:38.784288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808767+0000","last_change":"2024-03-28T00:54:32.806651+0000","last_active":"2024-03-28T00:54:38.808767+0000","last_peered":"2024-03-28T00:54:38.808767+0000","last_clean":"2024-03-28T00:54:38.808767+0000","last_became_active":"2024-03-28T00:54:32.806329+0000","last_became_peered":"2024-03-28T00:54:32.806329+0000","last_unstale":"2024-03-28T00:54:38.808767+0000","last_undegraded":"2024-03-28T00:54:38.808767+0000","last_fullsized":"2024-03-28T00:54:38.808767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:28:32.815396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808685+0000","last_change":"2024-03-28T00:54:36.827016+0000","last_active":"2024-03-28T00:54:38.808685+0000","last_peered":"2024-03-28T00:54:38.808685+0000","last_clean":"2024-03-28T00:54:38.808685+0000","last_became_active":"2024-03-28T00:54:36.826428+0000","last_became_peered":"2024-03-28T00:54:36.826428+0000","last_unstale":"2024-03-28T00:54:38.808685+0000","last_undegraded":"2024-03-28T00:54:38.808685+0000","last_fullsized":"2024-03-28T00:54:38.808685+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:33:01.593990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511799+0000","last_change":"2024-03-28T00:54:32.252532+0000","last_active":"2024-03-28T00:54:39.511799+0000","last_peered":"2024-03-28T00:54:39.511799+0000","last_clean":"2024-03-28T00:54:39.511799+0000","last_became_active":"2024-03-28T00:54:32.252204+0000","last_became_peered":"2024-03-28T00:54:32.252204+0000","last_unstale":"2024-03-28T00:54:39.511799+0000","last_undegraded":"2024-03-28T00:54:39.511799+0000","last_fullsized":"2024-03-28T00:54:39.511799+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:03:59.157278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.675715+0000","last_change":"2024-03-28T00:54:32.808064+0000","last_active":"2024-03-28T00:54:39.675715+0000","last_peered":"2024-03-28T00:54:39.675715+0000","last_clean":"2024-03-28T00:54:39.675715+0000","last_became_active":"2024-03-28T00:54:32.807181+0000","last_became_peered":"2024-03-28T00:54:32.807181+0000","last_unstale":"2024-03-28T00:54:39.675715+0000","last_undegraded":"2024-03-28T00:54:39.675715+0000","last_fullsized":"2024-03-28T00:54:39.675715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:26:01.917500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510838+0000","last_change":"2024-03-28T00:54:37.638175+0000","last_active":"2024-03-28T00:54:39.510838+0000","last_peered":"2024-03-28T00:54:39.510838+0000","last_clean":"2024-03-28T00:54:39.510838+0000","last_became_active":"2024-03-28T00:54:37.637662+0000","last_became_peered":"2024-03-28T00:54:37.637662+0000","last_unstale":"2024-03-28T00:54:39.510838+0000","last_undegraded":"2024-03-28T00:54:39.510838+0000","last_fullsized":"2024-03-28T00:54:39.510838+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:38:21.353519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.290306+0000","last_change":"2024-03-28T00:54:32.264038+0000","last_active":"2024-03-28T00:54:39.290306+0000","last_peered":"2024-03-28T00:54:39.290306+0000","last_clean":"2024-03-28T00:54:39.290306+0000","last_became_active":"2024-03-28T00:54:32.263706+0000","last_became_peered":"2024-03-28T00:54:32.263706+0000","last_unstale":"2024-03-28T00:54:39.290306+0000","last_undegraded":"2024-03-28T00:54:39.290306+0000","last_fullsized":"2024-03-28T00:54:39.290306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:03.244556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.148552+0000","last_change":"2024-03-28T00:54:32.804727+0000","last_active":"2024-03-28T00:54:39.148552+0000","last_peered":"2024-03-28T00:54:39.148552+0000","last_clean":"2024-03-28T00:54:39.148552+0000","last_became_active":"2024-03-28T00:54:32.804422+0000","last_became_peered":"2024-03-28T00:54:32.804422+0000","last_unstale":"2024-03-28T00:54:39.148552+0000","last_undegraded":"2024-03-28T00:54:39.148552+0000","last_fullsized":"2024-03-28T00:54:39.148552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:29:57.983722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.811349+0000","last_change":"2024-03-28T00:54:36.833416+0000","last_active":"2024-03-28T00:54:38.811349+0000","last_peered":"2024-03-28T00:54:38.811349+0000","last_clean":"2024-03-28T00:54:38.811349+0000","last_became_active":"2024-03-28T00:54:36.832985+0000","last_became_peered":"2024-03-28T00:54:36.832985+0000","last_unstale":"2024-03-28T00:54:38.811349+0000","last_undegraded":"2024-03-28T00:54:38.811349+0000","last_fullsized":"2024-03-28T00:54:38.811349+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:47:30.376032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808276+0000","last_change":"2024-03-28T00:54:32.256417+0000","last_active":"2024-03-28T00:54:38.808276+0000","last_peered":"2024-03-28T00:54:38.808276+0000","last_clean":"2024-03-28T00:54:38.808276+0000","last_became_active":"2024-03-28T00:54:32.256088+0000","last_became_peered":"2024-03-28T00:54:32.256088+0000","last_unstale":"2024-03-28T00:54:38.808276+0000","last_undegraded":"2024-03-28T00:54:38.808276+0000","last_fullsized":"2024-03-28T00:54:38.808276+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:58.661597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.148398+0000","last_change":"2024-03-28T00:54:32.817604+0000","last_active":"2024-03-28T00:54:39.148398+0000","last_peered":"2024-03-28T00:54:39.148398+0000","last_clean":"2024-03-28T00:54:39.148398+0000","last_became_active":"2024-03-28T00:54:32.817211+0000","last_became_peered":"2024-03-28T00:54:32.817211+0000","last_unstale":"2024-03-28T00:54:39.148398+0000","last_undegraded":"2024-03-28T00:54:39.148398+0000","last_fullsized":"2024-03-28T00:54:39.148398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:02:28.335075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.811147+0000","last_change":"2024-03-28T00:54:37.638968+0000","last_active":"2024-03-28T00:54:38.811147+0000","last_peered":"2024-03-28T00:54:38.811147+0000","last_clean":"2024-03-28T00:54:38.811147+0000","last_became_active":"2024-03-28T00:54:37.638446+0000","last_became_peered":"2024-03-28T00:54:37.638446+0000","last_unstale":"2024-03-28T00:54:38.811147+0000","last_undegraded":"2024-03-28T00:54:38.811147+0000","last_fullsized":"2024-03-28T00:54:38.811147+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:54:56.348760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808684+0000","last_change":"2024-03-28T00:54:32.809822+0000","last_active":"2024-03-28T00:54:38.808684+0000","last_peered":"2024-03-28T00:54:38.808684+0000","last_clean":"2024-03-28T00:54:38.808684+0000","last_became_active":"2024-03-28T00:54:32.808284+0000","last_became_peered":"2024-03-28T00:54:32.808284+0000","last_unstale":"2024-03-28T00:54:38.808684+0000","last_undegraded":"2024-03-28T00:54:38.808684+0000","last_fullsized":"2024-03-28T00:54:38.808684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:49:11.651960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.810716+0000","last_change":"2024-03-28T00:54:32.261806+0000","last_active":"2024-03-28T00:54:38.810716+0000","last_peered":"2024-03-28T00:54:38.810716+0000","last_clean":"2024-03-28T00:54:38.810716+0000","last_became_active":"2024-03-28T00:54:32.261493+0000","last_became_peered":"2024-03-28T00:54:32.261493+0000","last_unstale":"2024-03-28T00:54:38.810716+0000","last_undegraded":"2024-03-28T00:54:38.810716+0000","last_fullsized":"2024-03-28T00:54:38.810716+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:36:04.283408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.856729+0000","last_change":"2024-03-28T00:54:36.819332+0000","last_active":"2024-03-28T00:54:38.856729+0000","last_peered":"2024-03-28T00:54:38.856729+0000","last_clean":"2024-03-28T00:54:38.856729+0000","last_became_active":"2024-03-28T00:54:36.818932+0000","last_became_peered":"2024-03-28T00:54:36.818932+0000","last_unstale":"2024-03-28T00:54:38.856729+0000","last_undegraded":"2024-03-28T00:54:38.856729+0000","last_fullsized":"2024-03-28T00:54:38.856729+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:00:39.446738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511417+0000","last_change":"2024-03-28T00:54:32.806093+0000","last_active":"2024-03-28T00:54:39.511417+0000","last_peered":"2024-03-28T00:54:39.511417+0000","last_clean":"2024-03-28T00:54:39.511417+0000","last_became_active":"2024-03-28T00:54:32.805734+0000","last_became_peered":"2024-03-28T00:54:32.805734+0000","last_unstale":"2024-03-28T00:54:39.511417+0000","last_undegraded":"2024-03-28T00:54:39.511417+0000","last_fullsized":"2024-03-28T00:54:39.511417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:51:41.066225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.270104+0000","last_change":"2024-03-28T00:54:32.245182+0000","last_active":"2024-03-28T00:54:50.270104+0000","last_peered":"2024-03-28T00:54:50.270104+0000","last_clean":"2024-03-28T00:54:50.270104+0000","last_became_active":"2024-03-28T00:54:32.244855+0000","last_became_peered":"2024-03-28T00:54:32.244855+0000","last_unstale":"2024-03-28T00:54:50.270104+0000","last_undegraded":"2024-03-28T00:54:50.270104+0000","last_fullsized":"2024-03-28T00:54:50.270104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:39:09.997199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":347,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808208+0000","last_change":"2024-03-28T00:53:20.355628+0000","last_active":"2024-03-28T00:54:38.808208+0000","last_peered":"2024-03-28T00:54:38.808208+0000","last_clean":"2024-03-28T00:54:38.808208+0000","last_became_active":"2024-03-28T00:53:20.355175+0000","last_became_peered":"2024-03-28T00:53:20.355175+0000","last_unstale":"2024-03-28T00:54:38.808208+0000","last_undegraded":"2024-03-28T00:54:38.808208+0000","last_fullsized":"2024-03-28T00:54:38.808208+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:49:57.054435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:49:57.054435+0000","last_clean_scrub_stamp":"2024-03-28T00:49:57.054435+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:55:46.753799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.809409+0000","last_change":"2024-03-28T00:54:37.639760+0000","last_active":"2024-03-28T00:54:38.809409+0000","last_peered":"2024-03-28T00:54:38.809409+0000","last_clean":"2024-03-28T00:54:38.809409+0000","last_became_active":"2024-03-28T00:54:37.639287+0000","last_became_peered":"2024-03-28T00:54:37.639287+0000","last_unstale":"2024-03-28T00:54:38.809409+0000","last_undegraded":"2024-03-28T00:54:38.809409+0000","last_fullsized":"2024-03-28T00:54:38.809409+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:50:02.840312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.810531+0000","last_change":"2024-03-28T00:54:32.807764+0000","last_active":"2024-03-28T00:54:38.810531+0000","last_peered":"2024-03-28T00:54:38.810531+0000","last_clean":"2024-03-28T00:54:38.810531+0000","last_became_active":"2024-03-28T00:54:32.807441+0000","last_became_peered":"2024-03-28T00:54:32.807441+0000","last_unstale":"2024-03-28T00:54:38.810531+0000","last_undegraded":"2024-03-28T00:54:38.810531+0000","last_fullsized":"2024-03-28T00:54:38.810531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:37:40.540514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:51.277141+0000","last_change":"2024-03-28T00:54:32.259192+0000","last_active":"2024-03-28T00:54:51.277141+0000","last_peered":"2024-03-28T00:54:51.277141+0000","last_clean":"2024-03-28T00:54:51.277141+0000","last_became_active":"2024-03-28T00:54:32.258880+0000","last_became_peered":"2024-03-28T00:54:32.258880+0000","last_unstale":"2024-03-28T00:54:51.277141+0000","last_undegraded":"2024-03-28T00:54:51.277141+0000","last_fullsized":"2024-03-28T00:54:51.277141+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:20:11.384633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.675482+0000","last_change":"2024-03-28T00:54:36.822268+0000","last_active":"2024-03-28T00:54:39.675482+0000","last_peered":"2024-03-28T00:54:39.675482+0000","last_clean":"2024-03-28T00:54:39.675482+0000","last_became_active":"2024-03-28T00:54:36.821773+0000","last_became_peered":"2024-03-28T00:54:36.821773+0000","last_unstale":"2024-03-28T00:54:39.675482+0000","last_undegraded":"2024-03-28T00:54:39.675482+0000","last_fullsized":"2024-03-28T00:54:39.675482+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:56:11.206975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511679+0000","last_change":"2024-03-28T00:54:32.256133+0000","last_active":"2024-03-28T00:54:39.511679+0000","last_peered":"2024-03-28T00:54:39.511679+0000","last_clean":"2024-03-28T00:54:39.511679+0000","last_became_active":"2024-03-28T00:54:32.250535+0000","last_became_peered":"2024-03-28T00:54:32.250535+0000","last_unstale":"2024-03-28T00:54:39.511679+0000","last_undegraded":"2024-03-28T00:54:39.511679+0000","last_fullsized":"2024-03-28T00:54:39.511679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:18:50.874181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.811341+0000","last_change":"2024-03-28T00:54:32.808883+0000","last_active":"2024-03-28T00:54:38.811341+0000","last_peered":"2024-03-28T00:54:38.811341+0000","last_clean":"2024-03-28T00:54:38.811341+0000","last_became_active":"2024-03-28T00:54:32.808587+0000","last_became_peered":"2024-03-28T00:54:32.808587+0000","last_unstale":"2024-03-28T00:54:38.811341+0000","last_undegraded":"2024-03-28T00:54:38.811341+0000","last_fullsized":"2024-03-28T00:54:38.811341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:09:55.164876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510865+0000","last_change":"2024-03-28T00:54:37.637743+0000","last_active":"2024-03-28T00:54:39.510865+0000","last_peered":"2024-03-28T00:54:39.510865+0000","last_clean":"2024-03-28T00:54:39.510865+0000","last_became_active":"2024-03-28T00:54:37.637280+0000","last_became_peered":"2024-03-28T00:54:37.637280+0000","last_unstale":"2024-03-28T00:54:39.510865+0000","last_undegraded":"2024-03-28T00:54:39.510865+0000","last_fullsized":"2024-03-28T00:54:39.510865+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:40:51.431757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.290423+0000","last_change":"2024-03-28T00:54:32.246444+0000","last_active":"2024-03-28T00:54:39.290423+0000","last_peered":"2024-03-28T00:54:39.290423+0000","last_clean":"2024-03-28T00:54:39.290423+0000","last_became_active":"2024-03-28T00:54:32.245106+0000","last_became_peered":"2024-03-28T00:54:32.245106+0000","last_unstale":"2024-03-28T00:54:39.290423+0000","last_undegraded":"2024-03-28T00:54:39.290423+0000","last_fullsized":"2024-03-28T00:54:39.290423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:13:45.123625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511204+0000","last_change":"2024-03-28T00:54:32.812851+0000","last_active":"2024-03-28T00:54:39.511204+0000","last_peered":"2024-03-28T00:54:39.511204+0000","last_clean":"2024-03-28T00:54:39.511204+0000","last_became_active":"2024-03-28T00:54:32.812518+0000","last_became_peered":"2024-03-28T00:54:32.812518+0000","last_unstale":"2024-03-28T00:54:39.511204+0000","last_undegraded":"2024-03-28T00:54:39.511204+0000","last_fullsized":"2024-03-28T00:54:39.511204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:08:41.518888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511256+0000","last_change":"2024-03-28T00:54:36.818709+0000","last_active":"2024-03-28T00:54:39.511256+0000","last_peered":"2024-03-28T00:54:39.511256+0000","last_clean":"2024-03-28T00:54:39.511256+0000","last_became_active":"2024-03-28T00:54:36.818304+0000","last_became_peered":"2024-03-28T00:54:36.818304+0000","last_unstale":"2024-03-28T00:54:39.511256+0000","last_undegraded":"2024-03-28T00:54:39.511256+0000","last_fullsized":"2024-03-28T00:54:39.511256+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:03:07.642685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.269754+0000","last_change":"2024-03-28T00:54:32.256031+0000","last_active":"2024-03-28T00:54:50.269754+0000","last_peered":"2024-03-28T00:54:50.269754+0000","last_clean":"2024-03-28T00:54:50.269754+0000","last_became_active":"2024-03-28T00:54:32.255597+0000","last_became_peered":"2024-03-28T00:54:32.255597+0000","last_unstale":"2024-03-28T00:54:50.269754+0000","last_undegraded":"2024-03-28T00:54:50.269754+0000","last_fullsized":"2024-03-28T00:54:50.269754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:31:10.837212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511212+0000","last_change":"2024-03-28T00:54:32.816961+0000","last_active":"2024-03-28T00:54:39.511212+0000","last_peered":"2024-03-28T00:54:39.511212+0000","last_clean":"2024-03-28T00:54:39.511212+0000","last_became_active":"2024-03-28T00:54:32.816408+0000","last_became_peered":"2024-03-28T00:54:32.816408+0000","last_unstale":"2024-03-28T00:54:39.511212+0000","last_undegraded":"2024-03-28T00:54:39.511212+0000","last_fullsized":"2024-03-28T00:54:39.511212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:08:57.441492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510781+0000","last_change":"2024-03-28T00:54:37.637811+0000","last_active":"2024-03-28T00:54:39.510781+0000","last_peered":"2024-03-28T00:54:39.510781+0000","last_clean":"2024-03-28T00:54:39.510781+0000","last_became_active":"2024-03-28T00:54:37.637347+0000","last_became_peered":"2024-03-28T00:54:37.637347+0000","last_unstale":"2024-03-28T00:54:39.510781+0000","last_undegraded":"2024-03-28T00:54:39.510781+0000","last_fullsized":"2024-03-28T00:54:39.510781+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:17:52.687297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.272465+0000","last_change":"2024-03-28T00:54:32.250301+0000","last_active":"2024-03-28T00:54:50.272465+0000","last_peered":"2024-03-28T00:54:50.272465+0000","last_clean":"2024-03-28T00:54:50.272465+0000","last_became_active":"2024-03-28T00:54:32.249993+0000","last_became_peered":"2024-03-28T00:54:32.249993+0000","last_unstale":"2024-03-28T00:54:50.272465+0000","last_undegraded":"2024-03-28T00:54:50.272465+0000","last_fullsized":"2024-03-28T00:54:50.272465+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:53:34.826687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510650+0000","last_change":"2024-03-28T00:54:32.811592+0000","last_active":"2024-03-28T00:54:39.510650+0000","last_peered":"2024-03-28T00:54:39.510650+0000","last_clean":"2024-03-28T00:54:39.510650+0000","last_became_active":"2024-03-28T00:54:32.811228+0000","last_became_peered":"2024-03-28T00:54:32.811228+0000","last_unstale":"2024-03-28T00:54:39.510650+0000","last_undegraded":"2024-03-28T00:54:39.510650+0000","last_fullsized":"2024-03-28T00:54:39.510650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:14:19.585237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.290628+0000","last_change":"2024-03-28T00:54:36.822501+0000","last_active":"2024-03-28T00:54:39.290628+0000","last_peered":"2024-03-28T00:54:39.290628+0000","last_clean":"2024-03-28T00:54:39.290628+0000","last_became_active":"2024-03-28T00:54:36.821630+0000","last_became_peered":"2024-03-28T00:54:36.821630+0000","last_unstale":"2024-03-28T00:54:39.290628+0000","last_undegraded":"2024-03-28T00:54:39.290628+0000","last_fullsized":"2024-03-28T00:54:39.290628+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:20:52.247055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.271642+0000","last_change":"2024-03-28T00:54:32.247328+0000","last_active":"2024-03-28T00:54:50.271642+0000","last_peered":"2024-03-28T00:54:50.271642+0000","last_clean":"2024-03-28T00:54:50.271642+0000","last_became_active":"2024-03-28T00:54:32.247003+0000","last_became_peered":"2024-03-28T00:54:32.247003+0000","last_unstale":"2024-03-28T00:54:50.271642+0000","last_undegraded":"2024-03-28T00:54:50.271642+0000","last_fullsized":"2024-03-28T00:54:50.271642+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:33:09.764946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.675291+0000","last_change":"2024-03-28T00:54:32.805977+0000","last_active":"2024-03-28T00:54:39.675291+0000","last_peered":"2024-03-28T00:54:39.675291+0000","last_clean":"2024-03-28T00:54:39.675291+0000","last_became_active":"2024-03-28T00:54:32.805636+0000","last_became_peered":"2024-03-28T00:54:32.805636+0000","last_unstale":"2024-03-28T00:54:39.675291+0000","last_undegraded":"2024-03-28T00:54:39.675291+0000","last_fullsized":"2024-03-28T00:54:39.675291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:04:42.901732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510860+0000","last_change":"2024-03-28T00:54:36.822103+0000","last_active":"2024-03-28T00:54:39.510860+0000","last_peered":"2024-03-28T00:54:39.510860+0000","last_clean":"2024-03-28T00:54:39.510860+0000","last_became_active":"2024-03-28T00:54:36.821695+0000","last_became_peered":"2024-03-28T00:54:36.821695+0000","last_unstale":"2024-03-28T00:54:39.510860+0000","last_undegraded":"2024-03-28T00:54:39.510860+0000","last_fullsized":"2024-03-28T00:54:39.510860+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:45:00.613955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511084+0000","last_change":"2024-03-28T00:54:32.251523+0000","last_active":"2024-03-28T00:54:39.511084+0000","last_peered":"2024-03-28T00:54:39.511084+0000","last_clean":"2024-03-28T00:54:39.511084+0000","last_became_active":"2024-03-28T00:54:32.251223+0000","last_became_peered":"2024-03-28T00:54:32.251223+0000","last_unstale":"2024-03-28T00:54:39.511084+0000","last_undegraded":"2024-03-28T00:54:39.511084+0000","last_fullsized":"2024-03-28T00:54:39.511084+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:08:20.489848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511358+0000","last_change":"2024-03-28T00:54:32.817641+0000","last_active":"2024-03-28T00:54:39.511358+0000","last_peered":"2024-03-28T00:54:39.511358+0000","last_clean":"2024-03-28T00:54:39.511358+0000","last_became_active":"2024-03-28T00:54:32.817252+0000","last_became_peered":"2024-03-28T00:54:32.817252+0000","last_unstale":"2024-03-28T00:54:39.511358+0000","last_undegraded":"2024-03-28T00:54:39.511358+0000","last_fullsized":"2024-03-28T00:54:39.511358+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:54:48.275005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510849+0000","last_change":"2024-03-28T00:54:37.638990+0000","last_active":"2024-03-28T00:54:39.510849+0000","last_peered":"2024-03-28T00:54:39.510849+0000","last_clean":"2024-03-28T00:54:39.510849+0000","last_became_active":"2024-03-28T00:54:37.638538+0000","last_became_peered":"2024-03-28T00:54:37.638538+0000","last_unstale":"2024-03-28T00:54:39.510849+0000","last_undegraded":"2024-03-28T00:54:39.510849+0000","last_fullsized":"2024-03-28T00:54:39.510849+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:28:10.791805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:50.271077+0000","last_change":"2024-03-28T00:54:32.253254+0000","last_active":"2024-03-28T00:54:50.271077+0000","last_peered":"2024-03-28T00:54:50.271077+0000","last_clean":"2024-03-28T00:54:50.271077+0000","last_became_active":"2024-03-28T00:54:32.252879+0000","last_became_peered":"2024-03-28T00:54:32.252879+0000","last_unstale":"2024-03-28T00:54:50.271077+0000","last_undegraded":"2024-03-28T00:54:50.271077+0000","last_fullsized":"2024-03-28T00:54:50.271077+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:59:51.039215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808305+0000","last_change":"2024-03-28T00:54:32.811288+0000","last_active":"2024-03-28T00:54:38.808305+0000","last_peered":"2024-03-28T00:54:38.808305+0000","last_clean":"2024-03-28T00:54:38.808305+0000","last_became_active":"2024-03-28T00:54:32.810968+0000","last_became_peered":"2024-03-28T00:54:32.810968+0000","last_unstale":"2024-03-28T00:54:38.808305+0000","last_undegraded":"2024-03-28T00:54:38.808305+0000","last_fullsized":"2024-03-28T00:54:38.808305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:16:56.113046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510257+0000","last_change":"2024-03-28T00:54:36.830714+0000","last_active":"2024-03-28T00:54:39.510257+0000","last_peered":"2024-03-28T00:54:39.510257+0000","last_clean":"2024-03-28T00:54:39.510257+0000","last_became_active":"2024-03-28T00:54:36.830253+0000","last_became_peered":"2024-03-28T00:54:36.830253+0000","last_unstale":"2024-03-28T00:54:39.510257+0000","last_undegraded":"2024-03-28T00:54:39.510257+0000","last_fullsized":"2024-03-28T00:54:39.510257+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:24:43.696150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511755+0000","last_change":"2024-03-28T00:54:32.254009+0000","last_active":"2024-03-28T00:54:39.511755+0000","last_peered":"2024-03-28T00:54:39.511755+0000","last_clean":"2024-03-28T00:54:39.511755+0000","last_became_active":"2024-03-28T00:54:32.253697+0000","last_became_peered":"2024-03-28T00:54:32.253697+0000","last_unstale":"2024-03-28T00:54:39.511755+0000","last_undegraded":"2024-03-28T00:54:39.511755+0000","last_fullsized":"2024-03-28T00:54:39.511755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:17:44.629367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808729+0000","last_change":"2024-03-28T00:54:32.812828+0000","last_active":"2024-03-28T00:54:38.808729+0000","last_peered":"2024-03-28T00:54:38.808729+0000","last_clean":"2024-03-28T00:54:38.808729+0000","last_became_active":"2024-03-28T00:54:32.811591+0000","last_became_peered":"2024-03-28T00:54:32.811591+0000","last_unstale":"2024-03-28T00:54:38.808729+0000","last_undegraded":"2024-03-28T00:54:38.808729+0000","last_fullsized":"2024-03-28T00:54:38.808729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:04.613623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510810+0000","last_change":"2024-03-28T00:54:36.823284+0000","last_active":"2024-03-28T00:54:39.510810+0000","last_peered":"2024-03-28T00:54:39.510810+0000","last_clean":"2024-03-28T00:54:39.510810+0000","last_became_active":"2024-03-28T00:54:36.822883+0000","last_became_peered":"2024-03-28T00:54:36.822883+0000","last_unstale":"2024-03-28T00:54:39.510810+0000","last_undegraded":"2024-03-28T00:54:39.510810+0000","last_fullsized":"2024-03-28T00:54:39.510810+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:37:10.266063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.809170+0000","last_change":"2024-03-28T00:54:32.253946+0000","last_active":"2024-03-28T00:54:38.809170+0000","last_peered":"2024-03-28T00:54:38.809170+0000","last_clean":"2024-03-28T00:54:38.809170+0000","last_became_active":"2024-03-28T00:54:32.253650+0000","last_became_peered":"2024-03-28T00:54:32.253650+0000","last_unstale":"2024-03-28T00:54:38.809170+0000","last_undegraded":"2024-03-28T00:54:38.809170+0000","last_fullsized":"2024-03-28T00:54:38.809170+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:11:44.930074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808579+0000","last_change":"2024-03-28T00:54:32.815410+0000","last_active":"2024-03-28T00:54:38.808579+0000","last_peered":"2024-03-28T00:54:38.808579+0000","last_clean":"2024-03-28T00:54:38.808579+0000","last_became_active":"2024-03-28T00:54:32.815088+0000","last_became_peered":"2024-03-28T00:54:32.815088+0000","last_unstale":"2024-03-28T00:54:38.808579+0000","last_undegraded":"2024-03-28T00:54:38.808579+0000","last_fullsized":"2024-03-28T00:54:38.808579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:16:24.862822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808474+0000","last_change":"2024-03-28T00:54:36.823182+0000","last_active":"2024-03-28T00:54:38.808474+0000","last_peered":"2024-03-28T00:54:38.808474+0000","last_clean":"2024-03-28T00:54:38.808474+0000","last_became_active":"2024-03-28T00:54:36.821446+0000","last_became_peered":"2024-03-28T00:54:36.821446+0000","last_unstale":"2024-03-28T00:54:38.808474+0000","last_undegraded":"2024-03-28T00:54:38.808474+0000","last_fullsized":"2024-03-28T00:54:38.808474+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:45:20.030213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.811026+0000","last_change":"2024-03-28T00:54:32.257412+0000","last_active":"2024-03-28T00:54:38.811026+0000","last_peered":"2024-03-28T00:54:38.811026+0000","last_clean":"2024-03-28T00:54:38.811026+0000","last_became_active":"2024-03-28T00:54:32.257115+0000","last_became_peered":"2024-03-28T00:54:32.257115+0000","last_unstale":"2024-03-28T00:54:38.811026+0000","last_undegraded":"2024-03-28T00:54:38.811026+0000","last_fullsized":"2024-03-28T00:54:38.811026+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:04:29.850867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808830+0000","last_change":"2024-03-28T00:54:32.805879+0000","last_active":"2024-03-28T00:54:38.808830+0000","last_peered":"2024-03-28T00:54:38.808830+0000","last_clean":"2024-03-28T00:54:38.808830+0000","last_became_active":"2024-03-28T00:54:32.805507+0000","last_became_peered":"2024-03-28T00:54:32.805507+0000","last_unstale":"2024-03-28T00:54:38.808830+0000","last_undegraded":"2024-03-28T00:54:38.808830+0000","last_fullsized":"2024-03-28T00:54:38.808830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:51:51.229085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.675457+0000","last_change":"2024-03-28T00:54:32.807219+0000","last_active":"2024-03-28T00:54:39.675457+0000","last_peered":"2024-03-28T00:54:39.675457+0000","last_clean":"2024-03-28T00:54:39.675457+0000","last_became_active":"2024-03-28T00:54:32.806856+0000","last_became_peered":"2024-03-28T00:54:32.806856+0000","last_unstale":"2024-03-28T00:54:39.675457+0000","last_undegraded":"2024-03-28T00:54:39.675457+0000","last_fullsized":"2024-03-28T00:54:39.675457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:05:49.914636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511501+0000","last_change":"2024-03-28T00:54:32.806509+0000","last_active":"2024-03-28T00:54:39.511501+0000","last_peered":"2024-03-28T00:54:39.511501+0000","last_clean":"2024-03-28T00:54:39.511501+0000","last_became_active":"2024-03-28T00:54:32.806211+0000","last_became_peered":"2024-03-28T00:54:32.806211+0000","last_unstale":"2024-03-28T00:54:39.511501+0000","last_undegraded":"2024-03-28T00:54:39.511501+0000","last_fullsized":"2024-03-28T00:54:39.511501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:30:19.711477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510959+0000","last_change":"2024-03-28T00:54:32.813921+0000","last_active":"2024-03-28T00:54:39.510959+0000","last_peered":"2024-03-28T00:54:39.510959+0000","last_clean":"2024-03-28T00:54:39.510959+0000","last_became_active":"2024-03-28T00:54:32.812877+0000","last_became_peered":"2024-03-28T00:54:32.812877+0000","last_unstale":"2024-03-28T00:54:39.510959+0000","last_undegraded":"2024-03-28T00:54:39.510959+0000","last_fullsized":"2024-03-28T00:54:39.510959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:26:54.305074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510924+0000","last_change":"2024-03-28T00:54:32.815725+0000","last_active":"2024-03-28T00:54:39.510924+0000","last_peered":"2024-03-28T00:54:39.510924+0000","last_clean":"2024-03-28T00:54:39.510924+0000","last_became_active":"2024-03-28T00:54:32.815444+0000","last_became_peered":"2024-03-28T00:54:32.815444+0000","last_unstale":"2024-03-28T00:54:39.510924+0000","last_undegraded":"2024-03-28T00:54:39.510924+0000","last_fullsized":"2024-03-28T00:54:39.510924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:23:17.326589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.811200+0000","last_change":"2024-03-28T00:54:32.806595+0000","last_active":"2024-03-28T00:54:38.811200+0000","last_peered":"2024-03-28T00:54:38.811200+0000","last_clean":"2024-03-28T00:54:38.811200+0000","last_became_active":"2024-03-28T00:54:32.806256+0000","last_became_peered":"2024-03-28T00:54:32.806256+0000","last_unstale":"2024-03-28T00:54:38.811200+0000","last_undegraded":"2024-03-28T00:54:38.811200+0000","last_fullsized":"2024-03-28T00:54:38.811200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:04:22.551511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.675726+0000","last_change":"2024-03-28T00:54:32.805000+0000","last_active":"2024-03-28T00:54:39.675726+0000","last_peered":"2024-03-28T00:54:39.675726+0000","last_clean":"2024-03-28T00:54:39.675726+0000","last_became_active":"2024-03-28T00:54:32.804637+0000","last_became_peered":"2024-03-28T00:54:32.804637+0000","last_unstale":"2024-03-28T00:54:39.675726+0000","last_undegraded":"2024-03-28T00:54:39.675726+0000","last_fullsized":"2024-03-28T00:54:39.675726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:59:46.888014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511538+0000","last_change":"2024-03-28T00:54:32.802783+0000","last_active":"2024-03-28T00:54:39.511538+0000","last_peered":"2024-03-28T00:54:39.511538+0000","last_clean":"2024-03-28T00:54:39.511538+0000","last_became_active":"2024-03-28T00:54:32.802457+0000","last_became_peered":"2024-03-28T00:54:32.802457+0000","last_unstale":"2024-03-28T00:54:39.511538+0000","last_undegraded":"2024-03-28T00:54:39.511538+0000","last_fullsized":"2024-03-28T00:54:39.511538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:28:24.146047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511618+0000","last_change":"2024-03-28T00:54:32.810179+0000","last_active":"2024-03-28T00:54:39.511618+0000","last_peered":"2024-03-28T00:54:39.511618+0000","last_clean":"2024-03-28T00:54:39.511618+0000","last_became_active":"2024-03-28T00:54:32.809009+0000","last_became_peered":"2024-03-28T00:54:32.809009+0000","last_unstale":"2024-03-28T00:54:39.511618+0000","last_undegraded":"2024-03-28T00:54:39.511618+0000","last_fullsized":"2024-03-28T00:54:39.511618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:08:44.184910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.808642+0000","last_change":"2024-03-28T00:54:32.810084+0000","last_active":"2024-03-28T00:54:38.808642+0000","last_peered":"2024-03-28T00:54:38.808642+0000","last_clean":"2024-03-28T00:54:38.808642+0000","last_became_active":"2024-03-28T00:54:32.809386+0000","last_became_peered":"2024-03-28T00:54:32.809386+0000","last_unstale":"2024-03-28T00:54:38.808642+0000","last_undegraded":"2024-03-28T00:54:38.808642+0000","last_fullsized":"2024-03-28T00:54:38.808642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:49:13.754797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.511237+0000","last_change":"2024-03-28T00:54:32.809933+0000","last_active":"2024-03-28T00:54:39.511237+0000","last_peered":"2024-03-28T00:54:39.511237+0000","last_clean":"2024-03-28T00:54:39.511237+0000","last_became_active":"2024-03-28T00:54:32.808974+0000","last_became_peered":"2024-03-28T00:54:32.808974+0000","last_unstale":"2024-03-28T00:54:39.511237+0000","last_undegraded":"2024-03-28T00:54:39.511237+0000","last_fullsized":"2024-03-28T00:54:39.511237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:23:59.317887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.510532+0000","last_change":"2024-03-28T00:54:32.808764+0000","last_active":"2024-03-28T00:54:39.510532+0000","last_peered":"2024-03-28T00:54:39.510532+0000","last_clean":"2024-03-28T00:54:39.510532+0000","last_became_active":"2024-03-28T00:54:32.808034+0000","last_became_peered":"2024-03-28T00:54:32.808034+0000","last_unstale":"2024-03-28T00:54:39.510532+0000","last_undegraded":"2024-03-28T00:54:39.510532+0000","last_fullsized":"2024-03-28T00:54:39.510532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:27:12.762520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:38.811066+0000","last_change":"2024-03-28T00:54:32.805470+0000","last_active":"2024-03-28T00:54:38.811066+0000","last_peered":"2024-03-28T00:54:38.811066+0000","last_clean":"2024-03-28T00:54:38.811066+0000","last_became_active":"2024-03-28T00:54:32.805146+0000","last_became_peered":"2024-03-28T00:54:32.805146+0000","last_unstale":"2024-03-28T00:54:38.811066+0000","last_undegraded":"2024-03-28T00:54:38.811066+0000","last_fullsized":"2024-03-28T00:54:38.811066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:24:14.297966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T00:54:39.675580+0000","last_change":"2024-03-28T00:54:32.808202+0000","last_active":"2024-03-28T00:54:39.675580+0000","last_peered":"2024-03-28T00:54:39.675580+0000","last_clean":"2024-03-28T00:54:39.675580+0000","last_became_active":"2024-03-28T00:54:32.807856+0000","last_became_peered":"2024-03-28T00:54:32.807856+0000","last_unstale":"2024-03-28T00:54:39.675580+0000","last_undegraded":"2024-03-28T00:54:39.675580+0000","last_fullsized":"2024-03-28T00:54:39.675580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:52:11.514904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282084,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6046,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":93,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":47711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383697,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":487204,"data_compressed":17627,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645334,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":1082410,"data_compressed":53513,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808858,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":487406,"data_compressed":17627,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070496,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":487248,"data_compressed":17627,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332133,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":1087365,"data_compressed":53513,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495657,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":974848,"data_stored":486622,"data_compressed":17627,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":490464,"data_compressed":17627,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":486712,"data_compressed":17627,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":491120,"data_compressed":17627,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182456,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":486816,"data_compressed":17627,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920820,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":486644,"data_compressed":17627,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":1077172,"data_compressed":53513,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34218,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34218,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34218,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":31746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T00:54:55.359 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T00:54:55.360 INFO:tasks.cephadm:Setting up client nodes... 2024-03-28T00:54:55.361 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-28T00:54:55.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:55 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3356206730' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:55.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:55 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/473636366' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:55.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:55 smithi169 ceph-mon[32274]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:54:55.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:55 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3356206730' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:55.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:55 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/473636366' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:55.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:55 smithi174 ceph-mon[36060]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:54:55.634 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:54:55.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:55 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3356206730' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:54:55.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:55 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/473636366' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:54:55.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:55 smithi177 ceph-mon[36206]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:54:56.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:56 smithi169 ceph-mon[32274]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-28T00:54:56.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:56 smithi169 ceph-mon[32274]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:54:56.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:56 smithi169 ceph-mon[32274]: pgmap v300: 129 pgs: 129 active+clean; 584 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 2.6 KiB/s wr, 9 op/s 2024-03-28T00:54:56.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:56 smithi174 ceph-mon[36060]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-28T00:54:56.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:56 smithi174 ceph-mon[36060]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:54:56.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:56 smithi174 ceph-mon[36060]: pgmap v300: 129 pgs: 129 active+clean; 584 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 2.6 KiB/s wr, 9 op/s 2024-03-28T00:54:56.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:56 smithi177 ceph-mon[36206]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-28T00:54:56.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:56 smithi177 ceph-mon[36206]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:54:56.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:56 smithi177 ceph-mon[36206]: pgmap v300: 129 pgs: 129 active+clean; 584 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 2.6 KiB/s wr, 9 op/s 2024-03-28T00:54:57.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.377+0000 7fb124bc7700 1 -- 172.21.15.169:0/2305435198 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1201010e0 msgr2=0x7fb120101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:57.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.377+0000 7fb124bc7700 1 --2- 172.21.15.169:0/2305435198 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1201010e0 0x7fb120101500 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb1140099c0 tx=0x7fb114009cd0 comp rx=0 tx=0).stop 2024-03-28T00:54:57.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.377+0000 7fb124bc7700 1 -- 172.21.15.169:0/2305435198 shutdown_connections 2024-03-28T00:54:57.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.377+0000 7fb124bc7700 1 --2- 172.21.15.169:0/2305435198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb120102c80 0x7fb120109530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:57.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.377+0000 7fb124bc7700 1 --2- 172.21.15.169:0/2305435198 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1201010e0 0x7fb120101500 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:57.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.377+0000 7fb124bc7700 1 --2- 172.21.15.169:0/2305435198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1201022e0 0x7fb120102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:57.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.377+0000 7fb124bc7700 1 -- 172.21.15.169:0/2305435198 >> 172.21.15.169:0/2305435198 conn(0x7fb1200fc680 msgr2=0x7fb1200feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:57.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.377+0000 7fb124bc7700 1 -- 172.21.15.169:0/2305435198 shutdown_connections 2024-03-28T00:54:57.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.377+0000 7fb124bc7700 1 -- 172.21.15.169:0/2305435198 wait complete. 2024-03-28T00:54:57.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.377+0000 7fb124bc7700 1 Processor -- start 2024-03-28T00:54:57.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.377+0000 7fb124bc7700 1 -- start start 2024-03-28T00:54:57.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.378+0000 7fb124bc7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1201010e0 0x7fb12006b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:57.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.378+0000 7fb124bc7700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1201022e0 0x7fb12019d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:57.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.378+0000 7fb124bc7700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb120102c80 0x7fb12019f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:57.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.378+0000 7fb124bc7700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12010ccb0 con 0x7fb1201010e0 2024-03-28T00:54:57.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.378+0000 7fb124bc7700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12010cb30 con 0x7fb120102c80 2024-03-28T00:54:57.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.378+0000 7fb124bc7700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12010ce30 con 0x7fb1201022e0 2024-03-28T00:54:57.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.378+0000 7fb11e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1201010e0 0x7fb12006b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:57.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.378+0000 7fb11e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1201010e0 0x7fb12006b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:50536/0 (socket says 172.21.15.169:50536) 2024-03-28T00:54:57.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.378+0000 7fb11e59c700 1 -- 172.21.15.169:0/634201248 learned_addr learned my addr 172.21.15.169:0/634201248 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:54:57.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.378+0000 7fb11dd9b700 1 --2- 172.21.15.169:0/634201248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1201022e0 0x7fb12019d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:57.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.378+0000 7fb11ed9d700 1 --2- 172.21.15.169:0/634201248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb120102c80 0x7fb12019f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:57.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.378+0000 7fb11e59c700 1 -- 172.21.15.169:0/634201248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1201022e0 msgr2=0x7fb12019d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:57.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.378+0000 7fb11e59c700 1 --2- 172.21.15.169:0/634201248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1201022e0 0x7fb12019d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:57.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.378+0000 7fb11e59c700 1 -- 172.21.15.169:0/634201248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb120102c80 msgr2=0x7fb12019f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:57.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.378+0000 7fb11e59c700 1 --2- 172.21.15.169:0/634201248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb120102c80 0x7fb12019f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:57.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.378+0000 7fb11e59c700 1 -- 172.21.15.169:0/634201248 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12019ffc0 con 0x7fb1201010e0 2024-03-28T00:54:57.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.378+0000 7fb11e59c700 1 --2- 172.21.15.169:0/634201248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1201010e0 0x7fb12006b470 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb11400bf60 tx=0x7fb11400bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:57.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.378+0000 7fb10f7fe700 1 -- 172.21.15.169:0/634201248 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb114009e60 con 0x7fb1201010e0 2024-03-28T00:54:57.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.378+0000 7fb10f7fe700 1 -- 172.21.15.169:0/634201248 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb114003e30 con 0x7fb1201010e0 2024-03-28T00:54:57.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.378+0000 7fb124bc7700 1 -- 172.21.15.169:0/634201248 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb120196e30 con 0x7fb1201010e0 2024-03-28T00:54:57.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.378+0000 7fb10f7fe700 1 -- 172.21.15.169:0/634201248 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11400fd80 con 0x7fb1201010e0 2024-03-28T00:54:57.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.379+0000 7fb124bc7700 1 -- 172.21.15.169:0/634201248 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb120197310 con 0x7fb1201010e0 2024-03-28T00:54:57.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.380+0000 7fb124bc7700 1 -- 172.21.15.169:0/634201248 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1200506a0 con 0x7fb1201010e0 2024-03-28T00:54:57.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.380+0000 7fb10f7fe700 1 -- 172.21.15.169:0/634201248 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fb11400f460 con 0x7fb1201010e0 2024-03-28T00:54:57.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.381+0000 7fb10f7fe700 1 --2- 172.21.15.169:0/634201248 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1080a29d0 0x7fb1080a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:54:57.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.381+0000 7fb11dd9b700 1 --2- 172.21.15.169:0/634201248 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1080a29d0 0x7fb1080a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:54:57.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.381+0000 7fb10f7fe700 1 -- 172.21.15.169:0/634201248 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb1140cfb10 con 0x7fb1201010e0 2024-03-28T00:54:57.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.381+0000 7fb11dd9b700 1 --2- 172.21.15.169:0/634201248 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1080a29d0 0x7fb1080a4e90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb110009680 tx=0x7fb110009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:54:57.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.385+0000 7fb10f7fe700 1 -- 172.21.15.169:0/634201248 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb114013070 con 0x7fb1201010e0 2024-03-28T00:54:57.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.658+0000 7fb124bc7700 1 -- 172.21.15.169:0/634201248 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fb12002da30 con 0x7fb1201010e0 2024-03-28T00:54:57.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.661+0000 7fb10f7fe700 1 -- 172.21.15.169:0/634201248 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7fb11409c8a0 con 0x7fb1201010e0 2024-03-28T00:54:57.664 INFO:teuthology.orchestra.run.smithi169.stdout:[client.0] 2024-03-28T00:54:57.664 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQDhvwRm33lZJxAAJ5geNVRN5P/LAwdInZOLlw== 2024-03-28T00:54:57.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.664+0000 7fb124bc7700 1 -- 172.21.15.169:0/634201248 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1080a29d0 msgr2=0x7fb1080a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:57.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.664+0000 7fb124bc7700 1 --2- 172.21.15.169:0/634201248 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1080a29d0 0x7fb1080a4e90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb110009680 tx=0x7fb110009260 comp rx=0 tx=0).stop 2024-03-28T00:54:57.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.664+0000 7fb124bc7700 1 -- 172.21.15.169:0/634201248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1201010e0 msgr2=0x7fb12006b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:54:57.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.664+0000 7fb124bc7700 1 --2- 172.21.15.169:0/634201248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1201010e0 0x7fb12006b470 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb11400bf60 tx=0x7fb11400bf90 comp rx=0 tx=0).stop 2024-03-28T00:54:57.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.664+0000 7fb124bc7700 1 -- 172.21.15.169:0/634201248 shutdown_connections 2024-03-28T00:54:57.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.664+0000 7fb124bc7700 1 --2- 172.21.15.169:0/634201248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb120102c80 0x7fb12019f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:57.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.664+0000 7fb124bc7700 1 --2- 172.21.15.169:0/634201248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1201010e0 0x7fb12006b470 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:57.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.664+0000 7fb124bc7700 1 --2- 172.21.15.169:0/634201248 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1080a29d0 0x7fb1080a4e90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:57.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.664+0000 7fb124bc7700 1 --2- 172.21.15.169:0/634201248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1201022e0 0x7fb12019d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:54:57.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.664+0000 7fb124bc7700 1 -- 172.21.15.169:0/634201248 >> 172.21.15.169:0/634201248 conn(0x7fb1200fc680 msgr2=0x7fb120105580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:54:57.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.664+0000 7fb124bc7700 1 -- 172.21.15.169:0/634201248 shutdown_connections 2024-03-28T00:54:57.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:54:57.664+0000 7fb124bc7700 1 -- 172.21.15.169:0/634201248 wait complete. 2024-03-28T00:54:57.998 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:57 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/634201248' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T00:54:57.998 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:57 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/634201248' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T00:54:58.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:57 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/634201248' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T00:54:58.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:57 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/634201248' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T00:54:58.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:57 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/634201248' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T00:54:58.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:57 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/634201248' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T00:54:58.291 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-28T00:54:58.291 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-28T00:54:58.291 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-28T00:54:58.335 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-28T00:54:58.335 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-28T00:54:58.335 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-28T00:54:58.620 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:54:58.894 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:54:58 smithi169 ceph-mon[32274]: pgmap v301: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 7.5 KiB/s wr, 13 op/s 2024-03-28T00:54:59.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:54:58 smithi174 ceph-mon[36060]: pgmap v301: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 7.5 KiB/s wr, 13 op/s 2024-03-28T00:54:59.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:54:58 smithi177 ceph-mon[36206]: pgmap v301: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 7.5 KiB/s wr, 13 op/s 2024-03-28T00:55:00.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.369+0000 7f6b51fe7700 1 -- 172.21.15.169:0/2816729703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b4c101320 msgr2=0x7f6b4c101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:00.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.369+0000 7f6b51fe7700 1 --2- 172.21.15.169:0/2816729703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b4c101320 0x7f6b4c101700 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6b400099c0 tx=0x7f6b40009cd0 comp rx=0 tx=0).stop 2024-03-28T00:55:00.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.369+0000 7f6b51fe7700 1 -- 172.21.15.169:0/2816729703 shutdown_connections 2024-03-28T00:55:00.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.369+0000 7f6b51fe7700 1 --2- 172.21.15.169:0/2816729703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b4c106370 0x7f6b4c10df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:00.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.369+0000 7f6b51fe7700 1 --2- 172.21.15.169:0/2816729703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b4c101320 0x7f6b4c101700 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:00.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.369+0000 7f6b51fe7700 1 --2- 172.21.15.169:0/2816729703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b4c101cd0 0x7f6b4c105c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:00.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.369+0000 7f6b51fe7700 1 -- 172.21.15.169:0/2816729703 >> 172.21.15.169:0/2816729703 conn(0x7f6b4c0770e0 msgr2=0x7f6b4c0774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:00.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.369+0000 7f6b51fe7700 1 -- 172.21.15.169:0/2816729703 shutdown_connections 2024-03-28T00:55:00.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.369+0000 7f6b51fe7700 1 -- 172.21.15.169:0/2816729703 wait complete. 2024-03-28T00:55:00.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.370+0000 7f6b51fe7700 1 Processor -- start 2024-03-28T00:55:00.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.370+0000 7f6b51fe7700 1 -- start start 2024-03-28T00:55:00.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.370+0000 7f6b51fe7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b4c101320 0x7f6b4c19e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:00.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.371+0000 7f6b51fe7700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b4c101cd0 0x7f6b4c19ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:00.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.371+0000 7f6b51fe7700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b4c106370 0x7f6b4c1988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:00.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.371+0000 7f6b51fe7700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c1113d0 con 0x7f6b4c101320 2024-03-28T00:55:00.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.371+0000 7f6b51fe7700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c111250 con 0x7f6b4c101cd0 2024-03-28T00:55:00.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.371+0000 7f6b51fe7700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c111550 con 0x7f6b4c106370 2024-03-28T00:55:00.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.371+0000 7f6b4b7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b4c101320 0x7f6b4c19e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:00.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.371+0000 7f6b4b7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b4c101320 0x7f6b4c19e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:50546/0 (socket says 172.21.15.169:50546) 2024-03-28T00:55:00.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.371+0000 7f6b4b7fe700 1 -- 172.21.15.169:0/939643117 learned_addr learned my addr 172.21.15.169:0/939643117 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:00.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.371+0000 7f6b4bfff700 1 --2- 172.21.15.169:0/939643117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b4c106370 0x7f6b4c1988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:00.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.371+0000 7f6b4affd700 1 --2- 172.21.15.169:0/939643117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b4c101cd0 0x7f6b4c19ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:00.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.371+0000 7f6b4b7fe700 1 -- 172.21.15.169:0/939643117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b4c106370 msgr2=0x7f6b4c1988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:00.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.371+0000 7f6b4b7fe700 1 --2- 172.21.15.169:0/939643117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b4c106370 0x7f6b4c1988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:00.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.371+0000 7f6b4b7fe700 1 -- 172.21.15.169:0/939643117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b4c101cd0 msgr2=0x7f6b4c19ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:00.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.371+0000 7f6b4b7fe700 1 --2- 172.21.15.169:0/939643117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b4c101cd0 0x7f6b4c19ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:00.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.371+0000 7f6b4b7fe700 1 -- 172.21.15.169:0/939643117 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b4c199020 con 0x7f6b4c101320 2024-03-28T00:55:00.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.371+0000 7f6b4b7fe700 1 --2- 172.21.15.169:0/939643117 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b4c101320 0x7f6b4c19e6b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6b40006010 tx=0x7f6b40003a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:00.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.371+0000 7f6b48ff9700 1 -- 172.21.15.169:0/939643117 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b40003e50 con 0x7f6b4c101320 2024-03-28T00:55:00.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.371+0000 7f6b48ff9700 1 -- 172.21.15.169:0/939643117 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b400055a0 con 0x7f6b4c101320 2024-03-28T00:55:00.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.371+0000 7f6b51fe7700 1 -- 172.21.15.169:0/939643117 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b4c1992b0 con 0x7f6b4c101320 2024-03-28T00:55:00.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.371+0000 7f6b48ff9700 1 -- 172.21.15.169:0/939643117 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4000fe30 con 0x7f6b4c101320 2024-03-28T00:55:00.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.371+0000 7f6b51fe7700 1 -- 172.21.15.169:0/939643117 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b4c1a5580 con 0x7f6b4c101320 2024-03-28T00:55:00.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.373+0000 7f6b51fe7700 1 -- 172.21.15.169:0/939643117 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b4c102770 con 0x7f6b4c101320 2024-03-28T00:55:00.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.373+0000 7f6b48ff9700 1 -- 172.21.15.169:0/939643117 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f6b400050c0 con 0x7f6b4c101320 2024-03-28T00:55:00.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.374+0000 7f6b48ff9700 1 --2- 172.21.15.169:0/939643117 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6b340a2ac0 0x7f6b340a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:00.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.374+0000 7f6b4affd700 1 --2- 172.21.15.169:0/939643117 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6b340a2ac0 0x7f6b340a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:00.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.374+0000 7f6b48ff9700 1 -- 172.21.15.169:0/939643117 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f6b400cfba0 con 0x7f6b4c101320 2024-03-28T00:55:00.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.375+0000 7f6b4affd700 1 --2- 172.21.15.169:0/939643117 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6b340a2ac0 0x7f6b340a4f80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c100f90 tx=0x7f6b3c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:00.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.378+0000 7f6b48ff9700 1 -- 172.21.15.169:0/939643117 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6b40013070 con 0x7f6b4c101320 2024-03-28T00:55:00.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.632+0000 7f6b51fe7700 1 -- 172.21.15.169:0/939643117 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f6b4c068e50 con 0x7f6b4c101320 2024-03-28T00:55:00.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.637+0000 7f6b48ff9700 1 -- 172.21.15.169:0/939643117 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+262669 (secure 0 0 0) 0x7f6b4009c930 con 0x7f6b4c101320 2024-03-28T00:55:00.643 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:55:00.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.640+0000 7f6b51fe7700 1 -- 172.21.15.169:0/939643117 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6b340a2ac0 msgr2=0x7f6b340a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:00.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.640+0000 7f6b51fe7700 1 --2- 172.21.15.169:0/939643117 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6b340a2ac0 0x7f6b340a4f80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c100f90 tx=0x7f6b3c009260 comp rx=0 tx=0).stop 2024-03-28T00:55:00.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.640+0000 7f6b51fe7700 1 -- 172.21.15.169:0/939643117 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b4c101320 msgr2=0x7f6b4c19e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:00.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.640+0000 7f6b51fe7700 1 --2- 172.21.15.169:0/939643117 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b4c101320 0x7f6b4c19e6b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6b40006010 tx=0x7f6b40003a90 comp rx=0 tx=0).stop 2024-03-28T00:55:00.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.640+0000 7f6b51fe7700 1 -- 172.21.15.169:0/939643117 shutdown_connections 2024-03-28T00:55:00.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.640+0000 7f6b51fe7700 1 --2- 172.21.15.169:0/939643117 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6b340a2ac0 0x7f6b340a4f80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:00.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.640+0000 7f6b51fe7700 1 --2- 172.21.15.169:0/939643117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b4c106370 0x7f6b4c1988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:00.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.640+0000 7f6b51fe7700 1 --2- 172.21.15.169:0/939643117 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b4c101320 0x7f6b4c19e6b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:00.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.640+0000 7f6b51fe7700 1 --2- 172.21.15.169:0/939643117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b4c101cd0 0x7f6b4c19ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:00.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.640+0000 7f6b51fe7700 1 -- 172.21.15.169:0/939643117 >> 172.21.15.169:0/939643117 conn(0x7f6b4c0770e0 msgr2=0x7f6b4c06aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:00.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.640+0000 7f6b51fe7700 1 -- 172.21.15.169:0/939643117 shutdown_connections 2024-03-28T00:55:00.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:00.640+0000 7f6b51fe7700 1 -- 172.21.15.169:0/939643117 wait complete. 2024-03-28T00:55:01.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:01 smithi169 ceph-mon[32274]: pgmap v302: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 7.2 KiB/s wr, 12 op/s 2024-03-28T00:55:01.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:01 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/939643117' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T00:55:01.365 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":1069722251},{"type":"v1","addr":"172.21.15.169:6801","nonce":1069722251}]},"active_addr":"172.21.15.169:6801/1069722251","active_change":"2024-03-28T00:47:35.997884+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.169:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.169:0","nonce":1250999478}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.169:0","nonce":511304831}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.169:0","nonce":1494547996}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.169:0","nonce":1388450074}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.169:0","nonce":1887947213}]}]} 2024-03-28T00:55:01.370 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-28T00:55:01.370 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-28T00:55:01.370 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T00:55:01.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:01 smithi177 ceph-mon[36206]: pgmap v302: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 7.2 KiB/s wr, 12 op/s 2024-03-28T00:55:01.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:01 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/939643117' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T00:55:01.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:01 smithi174 ceph-mon[36060]: pgmap v302: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 7.2 KiB/s wr, 12 op/s 2024-03-28T00:55:01.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:01 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/939643117' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T00:55:01.612 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:03.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.264+0000 7fdd3bfed700 1 -- 172.21.15.169:0/2163858986 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd341021e0 msgr2=0x7fdd34102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:03.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.264+0000 7fdd3bfed700 1 --2- 172.21.15.169:0/2163858986 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd341021e0 0x7fdd34102660 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fdd300099c0 tx=0x7fdd30009cd0 comp rx=0 tx=0).stop 2024-03-28T00:55:03.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.264+0000 7fdd3bfed700 1 -- 172.21.15.169:0/2163858986 shutdown_connections 2024-03-28T00:55:03.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.264+0000 7fdd3bfed700 1 --2- 172.21.15.169:0/2163858986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd34102ba0 0x7fdd3410efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:03.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.264+0000 7fdd3bfed700 1 --2- 172.21.15.169:0/2163858986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd341081c0 0x7fdd341085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:03.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.264+0000 7fdd3bfed700 1 --2- 172.21.15.169:0/2163858986 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd341021e0 0x7fdd34102660 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:03.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.264+0000 7fdd3bfed700 1 -- 172.21.15.169:0/2163858986 >> 172.21.15.169:0/2163858986 conn(0x7fdd340792c0 msgr2=0x7fdd340796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:03.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.264+0000 7fdd3bfed700 1 -- 172.21.15.169:0/2163858986 shutdown_connections 2024-03-28T00:55:03.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.264+0000 7fdd3bfed700 1 -- 172.21.15.169:0/2163858986 wait complete. 2024-03-28T00:55:03.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.264+0000 7fdd3bfed700 1 Processor -- start 2024-03-28T00:55:03.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.264+0000 7fdd3bfed700 1 -- start start 2024-03-28T00:55:03.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.265+0000 7fdd3bfed700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd341021e0 0x7fdd3419e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:03.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.265+0000 7fdd3bfed700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd34102ba0 0x7fdd3419ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:03.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.265+0000 7fdd3bfed700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd341081c0 0x7fdd341987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:03.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.265+0000 7fdd3bfed700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd34112470 con 0x7fdd341081c0 2024-03-28T00:55:03.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.265+0000 7fdd3bfed700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd341122f0 con 0x7fdd341021e0 2024-03-28T00:55:03.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.265+0000 7fdd3bfed700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd341125f0 con 0x7fdd34102ba0 2024-03-28T00:55:03.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.265+0000 7fdd3a58a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd341081c0 0x7fdd341987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:03.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.265+0000 7fdd3a58a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd341081c0 0x7fdd341987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:50556/0 (socket says 172.21.15.169:50556) 2024-03-28T00:55:03.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.265+0000 7fdd3a58a700 1 -- 172.21.15.169:0/2221210901 learned_addr learned my addr 172.21.15.169:0/2221210901 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:03.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.265+0000 7fdd39588700 1 --2- 172.21.15.169:0/2221210901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd34102ba0 0x7fdd3419ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:03.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.265+0000 7fdd39d89700 1 --2- 172.21.15.169:0/2221210901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd341021e0 0x7fdd3419e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:03.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.265+0000 7fdd3a58a700 1 -- 172.21.15.169:0/2221210901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd34102ba0 msgr2=0x7fdd3419ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:03.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.265+0000 7fdd3a58a700 1 --2- 172.21.15.169:0/2221210901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd34102ba0 0x7fdd3419ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:03.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.265+0000 7fdd3a58a700 1 -- 172.21.15.169:0/2221210901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd341021e0 msgr2=0x7fdd3419e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:03.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.265+0000 7fdd3a58a700 1 --2- 172.21.15.169:0/2221210901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd341021e0 0x7fdd3419e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:03.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.265+0000 7fdd3a58a700 1 -- 172.21.15.169:0/2221210901 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd34199090 con 0x7fdd341081c0 2024-03-28T00:55:03.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.265+0000 7fdd3a58a700 1 --2- 172.21.15.169:0/2221210901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd341081c0 0x7fdd341987b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c00ef30 tx=0x7fdd2c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:03.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.265+0000 7fdd2affd700 1 -- 172.21.15.169:0/2221210901 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c0092e0 con 0x7fdd341081c0 2024-03-28T00:55:03.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.265+0000 7fdd2affd700 1 -- 172.21.15.169:0/2221210901 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd2c004bb0 con 0x7fdd341081c0 2024-03-28T00:55:03.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.265+0000 7fdd3bfed700 1 -- 172.21.15.169:0/2221210901 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd34199320 con 0x7fdd341081c0 2024-03-28T00:55:03.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.265+0000 7fdd2affd700 1 -- 172.21.15.169:0/2221210901 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c004280 con 0x7fdd341081c0 2024-03-28T00:55:03.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.266+0000 7fdd3bfed700 1 -- 172.21.15.169:0/2221210901 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd341a5610 con 0x7fdd341081c0 2024-03-28T00:55:03.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.267+0000 7fdd3bfed700 1 -- 172.21.15.169:0/2221210901 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd34103870 con 0x7fdd341081c0 2024-03-28T00:55:03.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.268+0000 7fdd2affd700 1 -- 172.21.15.169:0/2221210901 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fdd2c01b030 con 0x7fdd341081c0 2024-03-28T00:55:03.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.268+0000 7fdd2affd700 1 --2- 172.21.15.169:0/2221210901 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdd200a2920 0x7fdd200a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:03.272 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:03 smithi169 ceph-mon[32274]: pgmap v303: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.2 KiB/s wr, 14 op/s 2024-03-28T00:55:03.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.268+0000 7fdd2affd700 1 -- 172.21.15.169:0/2221210901 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fdd2c0ce8b0 con 0x7fdd341081c0 2024-03-28T00:55:03.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.268+0000 7fdd39d89700 1 --2- 172.21.15.169:0/2221210901 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdd200a2920 0x7fdd200a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:03.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.269+0000 7fdd39d89700 1 --2- 172.21.15.169:0/2221210901 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdd200a2920 0x7fdd200a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdd24009be0 tx=0x7fdd24009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:03.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.273+0000 7fdd2affd700 1 -- 172.21.15.169:0/2221210901 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdd2c022020 con 0x7fdd341081c0 2024-03-28T00:55:03.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:03 smithi177 ceph-mon[36206]: pgmap v303: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.2 KiB/s wr, 14 op/s 2024-03-28T00:55:03.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.471+0000 7fdd3bfed700 1 -- 172.21.15.169:0/2221210901 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd3402da30 con 0x7fdd341081c0 2024-03-28T00:55:03.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.473+0000 7fdd2affd700 1 -- 172.21.15.169:0/2221210901 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24186 (secure 0 0 0) 0x7fdd2c010040 con 0x7fdd341081c0 2024-03-28T00:55:03.476 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:55:03.476 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":90,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:54:38.804421+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:55:03.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.475+0000 7fdd3bfed700 1 -- 172.21.15.169:0/2221210901 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdd200a2920 msgr2=0x7fdd200a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:03.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.475+0000 7fdd3bfed700 1 --2- 172.21.15.169:0/2221210901 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdd200a2920 0x7fdd200a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdd24009be0 tx=0x7fdd24009260 comp rx=0 tx=0).stop 2024-03-28T00:55:03.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.475+0000 7fdd3bfed700 1 -- 172.21.15.169:0/2221210901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd341081c0 msgr2=0x7fdd341987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:03.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.475+0000 7fdd3bfed700 1 --2- 172.21.15.169:0/2221210901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd341081c0 0x7fdd341987b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c00ef30 tx=0x7fdd2c00c3a0 comp rx=0 tx=0).stop 2024-03-28T00:55:03.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.475+0000 7fdd3bfed700 1 -- 172.21.15.169:0/2221210901 shutdown_connections 2024-03-28T00:55:03.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.475+0000 7fdd3bfed700 1 --2- 172.21.15.169:0/2221210901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdd341081c0 0x7fdd341987b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:03.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.475+0000 7fdd3bfed700 1 --2- 172.21.15.169:0/2221210901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd341021e0 0x7fdd3419e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:03.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.475+0000 7fdd3bfed700 1 --2- 172.21.15.169:0/2221210901 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdd200a2920 0x7fdd200a4de0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:03.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.475+0000 7fdd3bfed700 1 --2- 172.21.15.169:0/2221210901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd34102ba0 0x7fdd3419ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:03.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.475+0000 7fdd3bfed700 1 -- 172.21.15.169:0/2221210901 >> 172.21.15.169:0/2221210901 conn(0x7fdd340792c0 msgr2=0x7fdd34100eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:03.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.475+0000 7fdd3bfed700 1 -- 172.21.15.169:0/2221210901 shutdown_connections 2024-03-28T00:55:03.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:03.475+0000 7fdd3bfed700 1 -- 172.21.15.169:0/2221210901 wait complete. 2024-03-28T00:55:03.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:03 smithi174 ceph-mon[36060]: pgmap v303: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.2 KiB/s wr, 14 op/s 2024-03-28T00:55:04.177 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-28T00:55:04.177 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T00:55:04.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:04 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2221210901' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:55:04.420 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:04.540 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:04 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2221210901' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:55:04.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:04 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2221210901' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:55:05.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:05 smithi177 ceph-mon[36206]: pgmap v304: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.0 KiB/s wr, 14 op/s 2024-03-28T00:55:05.464 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:05 smithi169 ceph-mon[32274]: pgmap v304: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.0 KiB/s wr, 14 op/s 2024-03-28T00:55:05.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:05 smithi174 ceph-mon[36060]: pgmap v304: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.0 KiB/s wr, 14 op/s 2024-03-28T00:55:05.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.991+0000 7fe11c447700 1 -- 172.21.15.169:0/2382269750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe114101cd0 msgr2=0x7fe114105cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:05.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.991+0000 7fe11c447700 1 --2- 172.21.15.169:0/2382269750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe114101cd0 0x7fe114105cd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe110009990 tx=0x7fe110009ca0 comp rx=0 tx=0).stop 2024-03-28T00:55:05.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.991+0000 7fe11c447700 1 -- 172.21.15.169:0/2382269750 shutdown_connections 2024-03-28T00:55:05.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.991+0000 7fe11c447700 1 --2- 172.21.15.169:0/2382269750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1141063b0 0x7fe11410df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:05.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.991+0000 7fe11c447700 1 --2- 172.21.15.169:0/2382269750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe114101320 0x7fe114101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:05.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.991+0000 7fe11c447700 1 --2- 172.21.15.169:0/2382269750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe114101cd0 0x7fe114105cd0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:05.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.991+0000 7fe11c447700 1 -- 172.21.15.169:0/2382269750 >> 172.21.15.169:0/2382269750 conn(0x7fe1140770b0 msgr2=0x7fe1140774c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:05.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.991+0000 7fe11c447700 1 -- 172.21.15.169:0/2382269750 shutdown_connections 2024-03-28T00:55:05.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.991+0000 7fe11c447700 1 -- 172.21.15.169:0/2382269750 wait complete. 2024-03-28T00:55:05.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.992+0000 7fe11c447700 1 Processor -- start 2024-03-28T00:55:05.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.992+0000 7fe11c447700 1 -- start start 2024-03-28T00:55:05.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.992+0000 7fe11c447700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe114101320 0x7fe11419e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:05.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.992+0000 7fe11c447700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe114101cd0 0x7fe11419eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:05.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.992+0000 7fe11c447700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1141063b0 0x7fe114198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:05.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.992+0000 7fe11c447700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe114111390 con 0x7fe1141063b0 2024-03-28T00:55:05.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.992+0000 7fe11c447700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe114111210 con 0x7fe114101cd0 2024-03-28T00:55:05.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.992+0000 7fe11c447700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe114111510 con 0x7fe114101320 2024-03-28T00:55:05.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.992+0000 7fe11a9e4700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1141063b0 0x7fe114198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:05.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.992+0000 7fe1199e2700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe114101cd0 0x7fe11419eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:05.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.992+0000 7fe11a9e4700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1141063b0 0x7fe114198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:50580/0 (socket says 172.21.15.169:50580) 2024-03-28T00:55:05.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.992+0000 7fe11a9e4700 1 -- 172.21.15.169:0/4036791036 learned_addr learned my addr 172.21.15.169:0/4036791036 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:05.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.992+0000 7fe11a1e3700 1 --2- 172.21.15.169:0/4036791036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe114101320 0x7fe11419e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:05.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.993+0000 7fe1199e2700 1 -- 172.21.15.169:0/4036791036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe114101320 msgr2=0x7fe11419e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:05.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.993+0000 7fe1199e2700 1 --2- 172.21.15.169:0/4036791036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe114101320 0x7fe11419e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:05.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.993+0000 7fe1199e2700 1 -- 172.21.15.169:0/4036791036 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1141063b0 msgr2=0x7fe114198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:05.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.993+0000 7fe1199e2700 1 --2- 172.21.15.169:0/4036791036 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1141063b0 0x7fe114198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:05.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.993+0000 7fe1199e2700 1 -- 172.21.15.169:0/4036791036 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe114198f60 con 0x7fe114101cd0 2024-03-28T00:55:05.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.993+0000 7fe1199e2700 1 --2- 172.21.15.169:0/4036791036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe114101cd0 0x7fe11419eb40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe1100098e0 tx=0x7fe11000bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:05.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.993+0000 7fe10b7fe700 1 -- 172.21.15.169:0/4036791036 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe110027070 con 0x7fe114101cd0 2024-03-28T00:55:05.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.993+0000 7fe11c447700 1 -- 172.21.15.169:0/4036791036 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1141991f0 con 0x7fe114101cd0 2024-03-28T00:55:05.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.993+0000 7fe10b7fe700 1 -- 172.21.15.169:0/4036791036 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe110003d70 con 0x7fe114101cd0 2024-03-28T00:55:05.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.993+0000 7fe10b7fe700 1 -- 172.21.15.169:0/4036791036 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11000f460 con 0x7fe114101cd0 2024-03-28T00:55:05.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.993+0000 7fe11c447700 1 -- 172.21.15.169:0/4036791036 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1141a5460 con 0x7fe114101cd0 2024-03-28T00:55:06.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.994+0000 7fe11c447700 1 -- 172.21.15.169:0/4036791036 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1141027b0 con 0x7fe114101cd0 2024-03-28T00:55:06.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.995+0000 7fe10b7fe700 1 -- 172.21.15.169:0/4036791036 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fe11002a070 con 0x7fe114101cd0 2024-03-28T00:55:06.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.996+0000 7fe10b7fe700 1 --2- 172.21.15.169:0/4036791036 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe1000a2920 0x7fe1000a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:06.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.996+0000 7fe11a1e3700 1 --2- 172.21.15.169:0/4036791036 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe1000a2920 0x7fe1000a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:06.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.996+0000 7fe10b7fe700 1 -- 172.21.15.169:0/4036791036 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fe1100cff70 con 0x7fe114101cd0 2024-03-28T00:55:06.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.996+0000 7fe11a1e3700 1 --2- 172.21.15.169:0/4036791036 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe1000a2920 0x7fe1000a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe104009c60 tx=0x7fe104009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:06.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:05.999+0000 7fe10b7fe700 1 -- 172.21.15.169:0/4036791036 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe110013070 con 0x7fe114101cd0 2024-03-28T00:55:06.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:06.199+0000 7fe11c447700 1 -- 172.21.15.169:0/4036791036 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe114199cb0 con 0x7fe114101cd0 2024-03-28T00:55:06.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:06.200+0000 7fe10b7fe700 1 -- 172.21.15.169:0/4036791036 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24186 (secure 0 0 0) 0x7fe11009cca0 con 0x7fe114101cd0 2024-03-28T00:55:06.204 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:55:06.204 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":90,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:54:38.804421+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:55:06.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:06.202+0000 7fe11c447700 1 -- 172.21.15.169:0/4036791036 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe1000a2920 msgr2=0x7fe1000a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:06.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:06.202+0000 7fe11c447700 1 --2- 172.21.15.169:0/4036791036 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe1000a2920 0x7fe1000a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe104009c60 tx=0x7fe104009260 comp rx=0 tx=0).stop 2024-03-28T00:55:06.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:06.202+0000 7fe11c447700 1 -- 172.21.15.169:0/4036791036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe114101cd0 msgr2=0x7fe11419eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:06.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:06.202+0000 7fe11c447700 1 --2- 172.21.15.169:0/4036791036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe114101cd0 0x7fe11419eb40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe1100098e0 tx=0x7fe11000bab0 comp rx=0 tx=0).stop 2024-03-28T00:55:06.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:06.202+0000 7fe11c447700 1 -- 172.21.15.169:0/4036791036 shutdown_connections 2024-03-28T00:55:06.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:06.202+0000 7fe11c447700 1 --2- 172.21.15.169:0/4036791036 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1141063b0 0x7fe114198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:06.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:06.202+0000 7fe11c447700 1 --2- 172.21.15.169:0/4036791036 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe1000a2920 0x7fe1000a4de0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:06.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:06.202+0000 7fe11c447700 1 --2- 172.21.15.169:0/4036791036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe114101320 0x7fe11419e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:06.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:06.202+0000 7fe11c447700 1 --2- 172.21.15.169:0/4036791036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe114101cd0 0x7fe11419eb40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:06.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:06.202+0000 7fe11c447700 1 -- 172.21.15.169:0/4036791036 >> 172.21.15.169:0/4036791036 conn(0x7fe1140770b0 msgr2=0x7fe114100860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:06.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:06.203+0000 7fe11c447700 1 -- 172.21.15.169:0/4036791036 shutdown_connections 2024-03-28T00:55:06.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:06.203+0000 7fe11c447700 1 -- 172.21.15.169:0/4036791036 wait complete. 2024-03-28T00:55:07.016 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-28T00:55:07.016 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-28T00:55:07.017 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-28T00:55:07.018 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-28T00:55:07.018 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-28T00:55:07.019 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-28T00:55:07.020 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-28T00:55:07.020 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-28T00:55:07.021 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-28T00:55:07.022 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-28T00:55:07.022 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-28T00:55:07.023 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-28T00:55:07.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:07 smithi169 ceph-mon[32274]: pgmap v305: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.7 KiB/s wr, 13 op/s 2024-03-28T00:55:07.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:07 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4036791036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:55:07.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:07 smithi177 ceph-mon[36206]: pgmap v305: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.7 KiB/s wr, 13 op/s 2024-03-28T00:55:07.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:07 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/4036791036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:55:07.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:07 smithi174 ceph-mon[36060]: pgmap v305: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.7 KiB/s wr, 13 op/s 2024-03-28T00:55:07.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:07 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/4036791036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:55:07.793 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:07.793 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:07.797 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:07.800 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:07.800 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:07.804 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:07.804 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:07.805 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:07.814 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:07.815 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:07.818 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:07.820 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:09.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:09 smithi177 ceph-mon[36206]: pgmap v306: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.6 KiB/s wr, 9 op/s 2024-03-28T00:55:09.510 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:09 smithi169 ceph-mon[32274]: pgmap v306: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.6 KiB/s wr, 9 op/s 2024-03-28T00:55:09.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:09 smithi174 ceph-mon[36060]: pgmap v306: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.6 KiB/s wr, 9 op/s 2024-03-28T00:55:11.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:11 smithi177 ceph-mon[36206]: pgmap v307: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-28T00:55:11.521 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:11 smithi169 ceph-mon[32274]: pgmap v307: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-28T00:55:11.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:11 smithi174 ceph-mon[36060]: pgmap v307: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-28T00:55:13.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:13 smithi169 ceph-mon[32274]: pgmap v308: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-28T00:55:13.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:13 smithi177 ceph-mon[36206]: pgmap v308: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-28T00:55:13.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:13 smithi174 ceph-mon[36060]: pgmap v308: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-28T00:55:15.293 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:15 smithi169 ceph-mon[32274]: pgmap v309: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-28T00:55:15.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:15 smithi177 ceph-mon[36206]: pgmap v309: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-28T00:55:15.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:15 smithi174 ceph-mon[36060]: pgmap v309: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-28T00:55:17.470 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:17 smithi169 ceph-mon[32274]: pgmap v310: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:55:17.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:17 smithi174 ceph-mon[36060]: pgmap v310: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:55:17.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:17 smithi177 ceph-mon[36206]: pgmap v310: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:55:19.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:19 smithi169 ceph-mon[32274]: pgmap v311: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:55:19.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:19 smithi174 ceph-mon[36060]: pgmap v311: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:55:19.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:19 smithi177 ceph-mon[36206]: pgmap v311: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:55:20.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.929+0000 7f72345cf700 1 -- 172.21.15.169:0/3804154299 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f722c06ab30 msgr2=0x7f722c06af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:20.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.929+0000 7f72345cf700 1 --2- 172.21.15.169:0/3804154299 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f722c06ab30 0x7f722c06af10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f721c009990 tx=0x7f721c009ca0 comp rx=0 tx=0).stop 2024-03-28T00:55:20.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.929+0000 7f72345cf700 1 -- 172.21.15.169:0/3804154299 shutdown_connections 2024-03-28T00:55:20.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.929+0000 7f72345cf700 1 --2- 172.21.15.169:0/3804154299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f722c1075d0 0x7f722c10f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:20.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.929+0000 7f72345cf700 1 --2- 172.21.15.169:0/3804154299 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f722c06ab30 0x7f722c06af10 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:20.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.929+0000 7f72345cf700 1 --2- 172.21.15.169:0/3804154299 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f722c06b4e0 0x7f722c106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:20.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.929+0000 7f72345cf700 1 -- 172.21.15.169:0/3804154299 >> 172.21.15.169:0/3804154299 conn(0x7f722c077ef0 msgr2=0x7f722c078300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:20.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.929+0000 7f72345cf700 1 -- 172.21.15.169:0/3804154299 shutdown_connections 2024-03-28T00:55:20.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.929+0000 7f72345cf700 1 -- 172.21.15.169:0/3804154299 wait complete. 2024-03-28T00:55:20.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.930+0000 7f72345cf700 1 Processor -- start 2024-03-28T00:55:20.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.930+0000 7f72345cf700 1 -- start start 2024-03-28T00:55:20.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.930+0000 7f72345cf700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f722c06ab30 0x7f722c19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:20.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.930+0000 7f72345cf700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f722c06b4e0 0x7f722c19eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:20.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.930+0000 7f72345cf700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f722c1075d0 0x7f722c1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:20.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.930+0000 7f72345cf700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c112530 con 0x7f722c1075d0 2024-03-28T00:55:20.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.930+0000 7f72345cf700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c1123b0 con 0x7f722c06ab30 2024-03-28T00:55:20.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.930+0000 7f72345cf700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c1126b0 con 0x7f722c06b4e0 2024-03-28T00:55:20.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.930+0000 7f723236b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f722c06ab30 0x7f722c19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:20.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.930+0000 7f7232b6c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f722c1075d0 0x7f722c1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:20.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.930+0000 7f723236b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f722c06ab30 0x7f722c19e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:52108/0 (socket says 172.21.15.169:52108) 2024-03-28T00:55:20.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.930+0000 7f7232b6c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f722c1075d0 0x7f722c1987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40602/0 (socket says 172.21.15.169:40602) 2024-03-28T00:55:20.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.931+0000 7f723236b700 1 -- 172.21.15.169:0/874111817 learned_addr learned my addr 172.21.15.169:0/874111817 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:20.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.931+0000 7f7231b6a700 1 --2- 172.21.15.169:0/874111817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f722c06b4e0 0x7f722c19eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:20.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.931+0000 7f723236b700 1 -- 172.21.15.169:0/874111817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f722c06b4e0 msgr2=0x7f722c19eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:20.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.931+0000 7f723236b700 1 --2- 172.21.15.169:0/874111817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f722c06b4e0 0x7f722c19eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:20.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.931+0000 7f723236b700 1 -- 172.21.15.169:0/874111817 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f722c1075d0 msgr2=0x7f722c1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:20.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.931+0000 7f723236b700 1 --2- 172.21.15.169:0/874111817 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f722c1075d0 0x7f722c1987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:20.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.931+0000 7f723236b700 1 -- 172.21.15.169:0/874111817 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722c198f90 con 0x7f722c06ab30 2024-03-28T00:55:20.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.931+0000 7f7232b6c700 1 --2- 172.21.15.169:0/874111817 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f722c1075d0 0x7f722c1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:55:20.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.931+0000 7f723236b700 1 --2- 172.21.15.169:0/874111817 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f722c06ab30 0x7f722c19e630 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f721c000c00 tx=0x7f721c00bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:20.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.931+0000 7f72237fe700 1 -- 172.21.15.169:0/874111817 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721c027070 con 0x7f722c06ab30 2024-03-28T00:55:20.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.931+0000 7f72345cf700 1 -- 172.21.15.169:0/874111817 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f722c199220 con 0x7f722c06ab30 2024-03-28T00:55:20.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.931+0000 7f72237fe700 1 -- 172.21.15.169:0/874111817 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f721c003d70 con 0x7f722c06ab30 2024-03-28T00:55:20.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.931+0000 7f72237fe700 1 -- 172.21.15.169:0/874111817 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721c00f460 con 0x7f722c06ab30 2024-03-28T00:55:20.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.931+0000 7f72345cf700 1 -- 172.21.15.169:0/874111817 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f722c1a5490 con 0x7f722c06ab30 2024-03-28T00:55:20.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.932+0000 7f72345cf700 1 -- 172.21.15.169:0/874111817 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f722c062860 con 0x7f722c06ab30 2024-03-28T00:55:20.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.933+0000 7f72237fe700 1 -- 172.21.15.169:0/874111817 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f721c02a070 con 0x7f722c06ab30 2024-03-28T00:55:20.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.934+0000 7f72237fe700 1 --2- 172.21.15.169:0/874111817 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f72180a2920 0x7f72180a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:20.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.934+0000 7f7231b6a700 1 --2- 172.21.15.169:0/874111817 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f72180a2920 0x7f72180a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:20.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.934+0000 7f72237fe700 1 -- 172.21.15.169:0/874111817 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f721c0d0210 con 0x7f722c06ab30 2024-03-28T00:55:20.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.934+0000 7f72237fe700 1 --2- 172.21.15.169:0/874111817 >> [v2:172.21.15.177:6824/2109004868,v1:172.21.15.177:6825/2109004868] conn(0x7f72180ad560 0x7f72180af9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:20.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.934+0000 7f72237fe700 1 -- 172.21.15.169:0/874111817 --> [v2:172.21.15.177:6824/2109004868,v1:172.21.15.177:6825/2109004868] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72180b0070 con 0x7f72180ad560 2024-03-28T00:55:20.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.934+0000 7f72237fe700 1 -- 172.21.15.169:0/874111817 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f721c0d0650 con 0x7f722c06ab30 2024-03-28T00:55:20.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.934+0000 7f7231b6a700 1 --2- 172.21.15.169:0/874111817 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f72180a2920 0x7f72180a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f722c1089b0 tx=0x7f7228009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:20.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.934+0000 7f7232b6c700 1 --2- 172.21.15.169:0/874111817 >> [v2:172.21.15.177:6824/2109004868,v1:172.21.15.177:6825/2109004868] conn(0x7f72180ad560 0x7f72180af9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:20.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.934+0000 7f7232b6c700 1 --2- 172.21.15.169:0/874111817 >> [v2:172.21.15.177:6824/2109004868,v1:172.21.15.177:6825/2109004868] conn(0x7f72180ad560 0x7f72180af9a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:20.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.936+0000 7f72237fe700 1 -- 172.21.15.169:0/874111817 <== osd.11 v2:172.21.15.177:6824/2109004868 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f72180b0070 con 0x7f72180ad560 2024-03-28T00:55:20.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.952+0000 7f72345cf700 1 -- 172.21.15.169:0/874111817 --> [v2:172.21.15.177:6824/2109004868,v1:172.21.15.177:6825/2109004868] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f722c062c90 con 0x7f72180ad560 2024-03-28T00:55:20.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.953+0000 7f72237fe700 1 -- 172.21.15.169:0/874111817 <== osd.11 v2:172.21.15.177:6824/2109004868 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f722c062c90 con 0x7f72180ad560 2024-03-28T00:55:20.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.953+0000 7f72345cf700 1 -- 172.21.15.169:0/874111817 >> [v2:172.21.15.177:6824/2109004868,v1:172.21.15.177:6825/2109004868] conn(0x7f72180ad560 msgr2=0x7f72180af9a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:20.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.953+0000 7f72345cf700 1 --2- 172.21.15.169:0/874111817 >> [v2:172.21.15.177:6824/2109004868,v1:172.21.15.177:6825/2109004868] conn(0x7f72180ad560 0x7f72180af9a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:20.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.953+0000 7f72345cf700 1 -- 172.21.15.169:0/874111817 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f72180a2920 msgr2=0x7f72180a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:20.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.953+0000 7f72345cf700 1 --2- 172.21.15.169:0/874111817 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f72180a2920 0x7f72180a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f722c1089b0 tx=0x7f7228009260 comp rx=0 tx=0).stop 2024-03-28T00:55:20.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.953+0000 7f72345cf700 1 -- 172.21.15.169:0/874111817 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f722c06ab30 msgr2=0x7f722c19e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:20.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.953+0000 7f72345cf700 1 --2- 172.21.15.169:0/874111817 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f722c06ab30 0x7f722c19e630 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f721c000c00 tx=0x7f721c00bab0 comp rx=0 tx=0).stop 2024-03-28T00:55:20.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.953+0000 7f7232b6c700 1 -- 172.21.15.169:0/874111817 reap_dead start 2024-03-28T00:55:20.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.953+0000 7f72345cf700 1 -- 172.21.15.169:0/874111817 shutdown_connections 2024-03-28T00:55:20.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.953+0000 7f72345cf700 1 -- 172.21.15.169:0/874111817 >> 172.21.15.169:0/874111817 conn(0x7f722c077ef0 msgr2=0x7f722c0ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:20.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.953+0000 7f72345cf700 1 -- 172.21.15.169:0/874111817 shutdown_connections 2024-03-28T00:55:20.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:17.953+0000 7f72345cf700 1 -- 172.21.15.169:0/874111817 wait complete. 2024-03-28T00:55:20.889 INFO:teuthology.orchestra.run.smithi169.stdout:343597383702 2024-03-28T00:55:20.890 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-28T00:55:21.512 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:21 smithi169 ceph-mon[32274]: pgmap v312: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-28T00:55:21.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:21 smithi174 ceph-mon[36060]: pgmap v312: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-28T00:55:21.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:21 smithi177 ceph-mon[36206]: pgmap v312: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-28T00:55:22.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.441+0000 7f5062794700 1 -- 172.21.15.169:0/2629530609 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f505c101cd0 msgr2=0x7f505c105cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:22.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.441+0000 7f5062794700 1 --2- 172.21.15.169:0/2629530609 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f505c101cd0 0x7f505c105cd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5050009990 tx=0x7f5050009ca0 comp rx=0 tx=0).stop 2024-03-28T00:55:22.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.441+0000 7f5062794700 1 -- 172.21.15.169:0/2629530609 shutdown_connections 2024-03-28T00:55:22.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.441+0000 7f5062794700 1 --2- 172.21.15.169:0/2629530609 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f505c1063b0 0x7f505c10df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:22.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.441+0000 7f5062794700 1 --2- 172.21.15.169:0/2629530609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f505c101320 0x7f505c101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:22.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.441+0000 7f5062794700 1 --2- 172.21.15.169:0/2629530609 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f505c101cd0 0x7f505c105cd0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:22.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.441+0000 7f5062794700 1 -- 172.21.15.169:0/2629530609 >> 172.21.15.169:0/2629530609 conn(0x7f505c0770b0 msgr2=0x7f505c0774c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:22.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.441+0000 7f5062794700 1 -- 172.21.15.169:0/2629530609 shutdown_connections 2024-03-28T00:55:22.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.442+0000 7f5062794700 1 -- 172.21.15.169:0/2629530609 wait complete. 2024-03-28T00:55:22.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.442+0000 7f5062794700 1 Processor -- start 2024-03-28T00:55:22.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.442+0000 7f5062794700 1 -- start start 2024-03-28T00:55:22.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.443+0000 7f5062794700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f505c101320 0x7f505c1997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:22.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.443+0000 7f5062794700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f505c101cd0 0x7f505c199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:22.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.443+0000 7f5062794700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f505c1063b0 0x7f505c1a4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:22.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.443+0000 7f5062794700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c111370 con 0x7f505c1063b0 2024-03-28T00:55:22.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.443+0000 7f5062794700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c1111f0 con 0x7f505c101320 2024-03-28T00:55:22.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.443+0000 7f5062794700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c1114f0 con 0x7f505c101cd0 2024-03-28T00:55:22.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.443+0000 7f5060d31700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f505c1063b0 0x7f505c1a4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:22.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.443+0000 7f5060d31700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f505c1063b0 0x7f505c1a4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40620/0 (socket says 172.21.15.169:40620) 2024-03-28T00:55:22.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.443+0000 7f5060d31700 1 -- 172.21.15.169:0/1649978201 learned_addr learned my addr 172.21.15.169:0/1649978201 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:22.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.443+0000 7f505bfff700 1 --2- 172.21.15.169:0/1649978201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f505c101320 0x7f505c1997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:22.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.443+0000 7f505b7fe700 1 --2- 172.21.15.169:0/1649978201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f505c101cd0 0x7f505c199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:22.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.443+0000 7f5060d31700 1 -- 172.21.15.169:0/1649978201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f505c101cd0 msgr2=0x7f505c199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:22.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.443+0000 7f5060d31700 1 --2- 172.21.15.169:0/1649978201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f505c101cd0 0x7f505c199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:22.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.443+0000 7f5060d31700 1 -- 172.21.15.169:0/1649978201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f505c101320 msgr2=0x7f505c1997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:22.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.443+0000 7f5060d31700 1 --2- 172.21.15.169:0/1649978201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f505c101320 0x7f505c1997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:22.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.443+0000 7f5060d31700 1 -- 172.21.15.169:0/1649978201 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505c1a4d70 con 0x7f505c1063b0 2024-03-28T00:55:22.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.443+0000 7f505bfff700 1 --2- 172.21.15.169:0/1649978201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f505c101320 0x7f505c1997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:55:22.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.443+0000 7f5060d31700 1 --2- 172.21.15.169:0/1649978201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f505c1063b0 0x7f505c1a4830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f505400ef30 tx=0x7f505400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:22.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.443+0000 7f50597fa700 1 -- 172.21.15.169:0/1649978201 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50540092e0 con 0x7f505c1063b0 2024-03-28T00:55:22.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.444+0000 7f50597fa700 1 -- 172.21.15.169:0/1649978201 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5054004bb0 con 0x7f505c1063b0 2024-03-28T00:55:22.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.444+0000 7f5062794700 1 -- 172.21.15.169:0/1649978201 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f505c1a4f40 con 0x7f505c1063b0 2024-03-28T00:55:22.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.444+0000 7f50597fa700 1 -- 172.21.15.169:0/1649978201 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5054004280 con 0x7f505c1063b0 2024-03-28T00:55:22.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.444+0000 7f5062794700 1 -- 172.21.15.169:0/1649978201 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f505c1a5340 con 0x7f505c1063b0 2024-03-28T00:55:22.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.444+0000 7f5062794700 1 -- 172.21.15.169:0/1649978201 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f505c0628a0 con 0x7f505c1063b0 2024-03-28T00:55:22.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.446+0000 7f50597fa700 1 -- 172.21.15.169:0/1649978201 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f505401b030 con 0x7f505c1063b0 2024-03-28T00:55:22.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.446+0000 7f50597fa700 1 --2- 172.21.15.169:0/1649978201 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f50440a2920 0x7f50440a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:22.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.446+0000 7f505bfff700 1 --2- 172.21.15.169:0/1649978201 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f50440a2920 0x7f50440a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:22.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.446+0000 7f50597fa700 1 -- 172.21.15.169:0/1649978201 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f50540cf180 con 0x7f505c1063b0 2024-03-28T00:55:22.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.446+0000 7f50597fa700 1 --2- 172.21.15.169:0/1649978201 >> [v2:172.21.15.174:6808/701854773,v1:172.21.15.174:6809/701854773] conn(0x7f50440ad5e0 0x7f50440afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:22.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.446+0000 7f50597fa700 1 -- 172.21.15.169:0/1649978201 --> [v2:172.21.15.174:6808/701854773,v1:172.21.15.174:6809/701854773] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f505401fe50 con 0x7f50440ad5e0 2024-03-28T00:55:22.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.446+0000 7f50597fa700 1 -- 172.21.15.169:0/1649978201 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50540cf5d0 con 0x7f505c1063b0 2024-03-28T00:55:22.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.447+0000 7f505bfff700 1 --2- 172.21.15.169:0/1649978201 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f50440a2920 0x7f50440a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f504c011490 tx=0x7f504c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:22.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.447+0000 7f505b7fe700 1 --2- 172.21.15.169:0/1649978201 >> [v2:172.21.15.174:6808/701854773,v1:172.21.15.174:6809/701854773] conn(0x7f50440ad5e0 0x7f50440afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:22.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.447+0000 7f505b7fe700 1 --2- 172.21.15.169:0/1649978201 >> [v2:172.21.15.174:6808/701854773,v1:172.21.15.174:6809/701854773] conn(0x7f50440ad5e0 0x7f50440afa20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:22.453 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.448+0000 7f50597fa700 1 -- 172.21.15.169:0/1649978201 <== osd.4 v2:172.21.15.174:6808/701854773 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f505401fe50 con 0x7f50440ad5e0 2024-03-28T00:55:22.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.464+0000 7f5062794700 1 -- 172.21.15.169:0/1649978201 --> [v2:172.21.15.174:6808/701854773,v1:172.21.15.174:6809/701854773] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f505c062d20 con 0x7f50440ad5e0 2024-03-28T00:55:22.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.464+0000 7f50597fa700 1 -- 172.21.15.169:0/1649978201 <== osd.4 v2:172.21.15.174:6808/701854773 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f505c062d20 con 0x7f50440ad5e0 2024-03-28T00:55:22.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.464+0000 7f5062794700 1 -- 172.21.15.169:0/1649978201 >> [v2:172.21.15.174:6808/701854773,v1:172.21.15.174:6809/701854773] conn(0x7f50440ad5e0 msgr2=0x7f50440afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:22.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.464+0000 7f5062794700 1 --2- 172.21.15.169:0/1649978201 >> [v2:172.21.15.174:6808/701854773,v1:172.21.15.174:6809/701854773] conn(0x7f50440ad5e0 0x7f50440afa20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:22.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.465+0000 7f5062794700 1 -- 172.21.15.169:0/1649978201 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f50440a2920 msgr2=0x7f50440a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:22.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.465+0000 7f5062794700 1 --2- 172.21.15.169:0/1649978201 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f50440a2920 0x7f50440a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f504c011490 tx=0x7f504c009260 comp rx=0 tx=0).stop 2024-03-28T00:55:22.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.465+0000 7f5062794700 1 -- 172.21.15.169:0/1649978201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f505c1063b0 msgr2=0x7f505c1a4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:22.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.465+0000 7f5062794700 1 --2- 172.21.15.169:0/1649978201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f505c1063b0 0x7f505c1a4830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f505400ef30 tx=0x7f505400c3a0 comp rx=0 tx=0).stop 2024-03-28T00:55:22.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.465+0000 7f5060d31700 1 -- 172.21.15.169:0/1649978201 reap_dead start 2024-03-28T00:55:22.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.465+0000 7f5062794700 1 -- 172.21.15.169:0/1649978201 shutdown_connections 2024-03-28T00:55:22.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.465+0000 7f5062794700 1 -- 172.21.15.169:0/1649978201 >> 172.21.15.169:0/1649978201 conn(0x7f505c0770b0 msgr2=0x7f505c100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:22.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.465+0000 7f5062794700 1 -- 172.21.15.169:0/1649978201 shutdown_connections 2024-03-28T00:55:22.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:22.465+0000 7f5062794700 1 -- 172.21.15.169:0/1649978201 wait complete. 2024-03-28T00:55:23.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.122+0000 7fe76b4b0700 1 -- 172.21.15.169:0/4088870773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe76406b440 msgr2=0x7fe764106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:23.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.122+0000 7fe76b4b0700 1 --2- 172.21.15.169:0/4088870773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe76406b440 0x7fe764106dd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe7540098f0 tx=0x7fe754009c00 comp rx=0 tx=0).stop 2024-03-28T00:55:23.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.122+0000 7fe76b4b0700 1 -- 172.21.15.169:0/4088870773 shutdown_connections 2024-03-28T00:55:23.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.122+0000 7fe76b4b0700 1 --2- 172.21.15.169:0/4088870773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe764107500 0x7fe76410f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:23.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.122+0000 7fe76b4b0700 1 --2- 172.21.15.169:0/4088870773 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe76406aa90 0x7fe76406ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:23.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.122+0000 7fe76b4b0700 1 --2- 172.21.15.169:0/4088870773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe76406b440 0x7fe764106dd0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:23.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.122+0000 7fe76b4b0700 1 -- 172.21.15.169:0/4088870773 >> 172.21.15.169:0/4088870773 conn(0x7fe764078000 msgr2=0x7fe764078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:23.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.122+0000 7fe76b4b0700 1 -- 172.21.15.169:0/4088870773 shutdown_connections 2024-03-28T00:55:23.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.122+0000 7fe76b4b0700 1 -- 172.21.15.169:0/4088870773 wait complete. 2024-03-28T00:55:23.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.122+0000 7fe76b4b0700 1 Processor -- start 2024-03-28T00:55:23.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.122+0000 7fe76b4b0700 1 -- start start 2024-03-28T00:55:23.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.123+0000 7fe76b4b0700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe76406aa90 0x7fe76419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:23.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.123+0000 7fe76b4b0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe76406b440 0x7fe76419eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:23.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.123+0000 7fe76b4b0700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe764107500 0x7fe7641987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:23.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.123+0000 7fe76b4b0700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe764112450 con 0x7fe76406b440 2024-03-28T00:55:23.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.123+0000 7fe76b4b0700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7641122d0 con 0x7fe764107500 2024-03-28T00:55:23.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.123+0000 7fe76b4b0700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7641125d0 con 0x7fe76406aa90 2024-03-28T00:55:23.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.123+0000 7fe768a4b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe76406b440 0x7fe76419eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:23.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.123+0000 7fe768a4b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe76406b440 0x7fe76419eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40628/0 (socket says 172.21.15.169:40628) 2024-03-28T00:55:23.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.123+0000 7fe768a4b700 1 -- 172.21.15.169:0/294635228 learned_addr learned my addr 172.21.15.169:0/294635228 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:23.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.123+0000 7fe76924c700 1 --2- 172.21.15.169:0/294635228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe76406aa90 0x7fe76419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:23.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.123+0000 7fe769a4d700 1 --2- 172.21.15.169:0/294635228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe764107500 0x7fe7641987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:23.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.123+0000 7fe768a4b700 1 -- 172.21.15.169:0/294635228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe76406aa90 msgr2=0x7fe76419e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:23.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.123+0000 7fe768a4b700 1 --2- 172.21.15.169:0/294635228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe76406aa90 0x7fe76419e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:23.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.123+0000 7fe768a4b700 1 -- 172.21.15.169:0/294635228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe764107500 msgr2=0x7fe7641987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:23.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.123+0000 7fe768a4b700 1 --2- 172.21.15.169:0/294635228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe764107500 0x7fe7641987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:23.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.123+0000 7fe768a4b700 1 -- 172.21.15.169:0/294635228 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe764198fb0 con 0x7fe76406b440 2024-03-28T00:55:23.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.123+0000 7fe76924c700 1 --2- 172.21.15.169:0/294635228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe76406aa90 0x7fe76419e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:55:23.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.123+0000 7fe768a4b700 1 --2- 172.21.15.169:0/294635228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe76406b440 0x7fe76419eb90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe754000c00 tx=0x7fe75400ff20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:23.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.123+0000 7fe75a7fc700 1 -- 172.21.15.169:0/294635228 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe754005490 con 0x7fe76406b440 2024-03-28T00:55:23.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.124+0000 7fe76b4b0700 1 -- 172.21.15.169:0/294635228 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe764199240 con 0x7fe76406b440 2024-03-28T00:55:23.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.124+0000 7fe75a7fc700 1 -- 172.21.15.169:0/294635228 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe754005630 con 0x7fe76406b440 2024-03-28T00:55:23.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.124+0000 7fe75a7fc700 1 -- 172.21.15.169:0/294635228 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe754003dd0 con 0x7fe76406b440 2024-03-28T00:55:23.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.124+0000 7fe76b4b0700 1 -- 172.21.15.169:0/294635228 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7641a54b0 con 0x7fe76406b440 2024-03-28T00:55:23.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.124+0000 7fe76b4b0700 1 -- 172.21.15.169:0/294635228 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe764062920 con 0x7fe76406b440 2024-03-28T00:55:23.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.126+0000 7fe75a7fc700 1 -- 172.21.15.169:0/294635228 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fe75400b560 con 0x7fe76406b440 2024-03-28T00:55:23.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.126+0000 7fe75a7fc700 1 --2- 172.21.15.169:0/294635228 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe7500a2920 0x7fe7500a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:23.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.126+0000 7fe76924c700 1 --2- 172.21.15.169:0/294635228 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe7500a2920 0x7fe7500a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:23.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.126+0000 7fe75a7fc700 1 -- 172.21.15.169:0/294635228 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fe7540cfe60 con 0x7fe76406b440 2024-03-28T00:55:23.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.126+0000 7fe75a7fc700 1 --2- 172.21.15.169:0/294635228 >> [v2:172.21.15.174:6816/2274626586,v1:172.21.15.174:6817/2274626586] conn(0x7fe7500ad5e0 0x7fe7500afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:23.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.126+0000 7fe75a7fc700 1 -- 172.21.15.169:0/294635228 --> [v2:172.21.15.174:6816/2274626586,v1:172.21.15.174:6817/2274626586] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe754010ea0 con 0x7fe7500ad5e0 2024-03-28T00:55:23.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.126+0000 7fe75a7fc700 1 -- 172.21.15.169:0/294635228 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7540d02b0 con 0x7fe76406b440 2024-03-28T00:55:23.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.127+0000 7fe76924c700 1 --2- 172.21.15.169:0/294635228 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe7500a2920 0x7fe7500a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe760009750 tx=0x7fe760009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:23.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.127+0000 7fe769a4d700 1 --2- 172.21.15.169:0/294635228 >> [v2:172.21.15.174:6816/2274626586,v1:172.21.15.174:6817/2274626586] conn(0x7fe7500ad5e0 0x7fe7500afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:23.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.127+0000 7fe769a4d700 1 --2- 172.21.15.169:0/294635228 >> [v2:172.21.15.174:6816/2274626586,v1:172.21.15.174:6817/2274626586] conn(0x7fe7500ad5e0 0x7fe7500afa20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:23.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.128+0000 7fe75a7fc700 1 -- 172.21.15.169:0/294635228 <== osd.7 v2:172.21.15.174:6816/2274626586 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe754010ea0 con 0x7fe7500ad5e0 2024-03-28T00:55:23.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.144+0000 7fe76b4b0700 1 -- 172.21.15.169:0/294635228 --> [v2:172.21.15.174:6816/2274626586,v1:172.21.15.174:6817/2274626586] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe764062d20 con 0x7fe7500ad5e0 2024-03-28T00:55:23.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.144+0000 7fe75a7fc700 1 -- 172.21.15.169:0/294635228 <== osd.7 v2:172.21.15.174:6816/2274626586 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe764062d20 con 0x7fe7500ad5e0 2024-03-28T00:55:23.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.144+0000 7fe76b4b0700 1 -- 172.21.15.169:0/294635228 >> [v2:172.21.15.174:6816/2274626586,v1:172.21.15.174:6817/2274626586] conn(0x7fe7500ad5e0 msgr2=0x7fe7500afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:23.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.144+0000 7fe76b4b0700 1 --2- 172.21.15.169:0/294635228 >> [v2:172.21.15.174:6816/2274626586,v1:172.21.15.174:6817/2274626586] conn(0x7fe7500ad5e0 0x7fe7500afa20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:23.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.144+0000 7fe76b4b0700 1 -- 172.21.15.169:0/294635228 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe7500a2920 msgr2=0x7fe7500a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:23.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.144+0000 7fe76b4b0700 1 --2- 172.21.15.169:0/294635228 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe7500a2920 0x7fe7500a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe760009750 tx=0x7fe760009310 comp rx=0 tx=0).stop 2024-03-28T00:55:23.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.144+0000 7fe76b4b0700 1 -- 172.21.15.169:0/294635228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe76406b440 msgr2=0x7fe76419eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:23.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.144+0000 7fe76b4b0700 1 --2- 172.21.15.169:0/294635228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe76406b440 0x7fe76419eb90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe754000c00 tx=0x7fe75400ff20 comp rx=0 tx=0).stop 2024-03-28T00:55:23.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.144+0000 7fe769a4d700 1 -- 172.21.15.169:0/294635228 reap_dead start 2024-03-28T00:55:23.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.145+0000 7fe76b4b0700 1 -- 172.21.15.169:0/294635228 shutdown_connections 2024-03-28T00:55:23.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.145+0000 7fe76b4b0700 1 -- 172.21.15.169:0/294635228 >> 172.21.15.169:0/294635228 conn(0x7fe764078000 msgr2=0x7fe7640ff5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:23.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.145+0000 7fe76b4b0700 1 -- 172.21.15.169:0/294635228 shutdown_connections 2024-03-28T00:55:23.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.145+0000 7fe76b4b0700 1 -- 172.21.15.169:0/294635228 wait complete. 2024-03-28T00:55:23.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.324+0000 7fb13e83e700 1 -- 172.21.15.169:0/60091107 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb138102240 msgr2=0x7fb1381026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:23.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.324+0000 7fb13e83e700 1 --2- 172.21.15.169:0/60091107 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb138102240 0x7fb1381026c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb12c009990 tx=0x7fb12c009ca0 comp rx=0 tx=0).stop 2024-03-28T00:55:23.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.324+0000 7fb13e83e700 1 -- 172.21.15.169:0/60091107 shutdown_connections 2024-03-28T00:55:23.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.324+0000 7fb13e83e700 1 --2- 172.21.15.169:0/60091107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb138102c00 0x7fb13810eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:23.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.324+0000 7fb13e83e700 1 --2- 172.21.15.169:0/60091107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1381082b0 0x7fb138108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:23.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.324+0000 7fb13e83e700 1 --2- 172.21.15.169:0/60091107 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb138102240 0x7fb1381026c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:23.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.324+0000 7fb13e83e700 1 -- 172.21.15.169:0/60091107 >> 172.21.15.169:0/60091107 conn(0x7fb138079260 msgr2=0x7fb138079670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:23.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.324+0000 7fb13e83e700 1 -- 172.21.15.169:0/60091107 shutdown_connections 2024-03-28T00:55:23.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.324+0000 7fb13e83e700 1 -- 172.21.15.169:0/60091107 wait complete. 2024-03-28T00:55:23.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.325+0000 7fb13e83e700 1 Processor -- start 2024-03-28T00:55:23.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.325+0000 7fb13e83e700 1 -- start start 2024-03-28T00:55:23.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.325+0000 7fb13e83e700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb138102240 0x7fb138074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:23.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.325+0000 7fb13e83e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb138102c00 0x7fb1380749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:23.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.325+0000 7fb13e83e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1381082b0 0x7fb138074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:23.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.325+0000 7fb13e83e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1381124d0 con 0x7fb1381082b0 2024-03-28T00:55:23.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.325+0000 7fb13e83e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb138112350 con 0x7fb138102c00 2024-03-28T00:55:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.325+0000 7fb13e83e700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb138112650 con 0x7fb138102240 2024-03-28T00:55:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.325+0000 7fb13cddb700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1381082b0 0x7fb138074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.325+0000 7fb1377fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb138102c00 0x7fb1380749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.325+0000 7fb137fff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb138102240 0x7fb138074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.325+0000 7fb137fff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb138102240 0x7fb138074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:53046/0 (socket says 172.21.15.169:53046) 2024-03-28T00:55:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.325+0000 7fb1377fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb138102c00 0x7fb1380749d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:52152/0 (socket says 172.21.15.169:52152) 2024-03-28T00:55:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.325+0000 7fb137fff700 1 -- 172.21.15.169:0/3767933097 learned_addr learned my addr 172.21.15.169:0/3767933097 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.326+0000 7fb137fff700 1 -- 172.21.15.169:0/3767933097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb138102c00 msgr2=0x7fb1380749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.326+0000 7fb137fff700 1 --2- 172.21.15.169:0/3767933097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb138102c00 0x7fb1380749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.326+0000 7fb137fff700 1 -- 172.21.15.169:0/3767933097 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1381082b0 msgr2=0x7fb138074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.326+0000 7fb137fff700 1 --2- 172.21.15.169:0/3767933097 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1381082b0 0x7fb138074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.326+0000 7fb137fff700 1 -- 172.21.15.169:0/3767933097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1381a6ec0 con 0x7fb138102240 2024-03-28T00:55:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.326+0000 7fb1377fe700 1 --2- 172.21.15.169:0/3767933097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb138102c00 0x7fb1380749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:55:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.326+0000 7fb137fff700 1 --2- 172.21.15.169:0/3767933097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb138102240 0x7fb138074490 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb12800e9a0 tx=0x7fb12800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.326+0000 7fb1357fa700 1 -- 172.21.15.169:0/3767933097 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12800ca10 con 0x7fb138102240 2024-03-28T00:55:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.326+0000 7fb1357fa700 1 -- 172.21.15.169:0/3767933097 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb12800cbb0 con 0x7fb138102240 2024-03-28T00:55:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.326+0000 7fb13e83e700 1 -- 172.21.15.169:0/3767933097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1381a71b0 con 0x7fb138102240 2024-03-28T00:55:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.326+0000 7fb1357fa700 1 -- 172.21.15.169:0/3767933097 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb128018680 con 0x7fb138102240 2024-03-28T00:55:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.326+0000 7fb13e83e700 1 -- 172.21.15.169:0/3767933097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1381a76c0 con 0x7fb138102240 2024-03-28T00:55:23.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.327+0000 7fb13e83e700 1 -- 172.21.15.169:0/3767933097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb1380628a0 con 0x7fb138102240 2024-03-28T00:55:23.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.328+0000 7fb1357fa700 1 -- 172.21.15.169:0/3767933097 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fb128010430 con 0x7fb138102240 2024-03-28T00:55:23.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.328+0000 7fb1357fa700 1 --2- 172.21.15.169:0/3767933097 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1200a2920 0x7fb1200a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:23.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.328+0000 7fb1377fe700 1 --2- 172.21.15.169:0/3767933097 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1200a2920 0x7fb1200a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:23.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.328+0000 7fb1357fa700 1 -- 172.21.15.169:0/3767933097 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb1280cefe0 con 0x7fb138102240 2024-03-28T00:55:23.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.329+0000 7fb1357fa700 1 --2- 172.21.15.169:0/3767933097 >> [v2:172.21.15.177:6816/4108616824,v1:172.21.15.177:6817/4108616824] conn(0x7fb1200ad5e0 0x7fb1200afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:23.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.329+0000 7fb1357fa700 1 -- 172.21.15.169:0/3767933097 --> [v2:172.21.15.177:6816/4108616824,v1:172.21.15.177:6817/4108616824] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb128012ec0 con 0x7fb1200ad5e0 2024-03-28T00:55:23.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.329+0000 7fb1357fa700 1 -- 172.21.15.169:0/3767933097 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb1280d6050 con 0x7fb138102240 2024-03-28T00:55:23.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.329+0000 7fb13cddb700 1 --2- 172.21.15.169:0/3767933097 >> [v2:172.21.15.177:6816/4108616824,v1:172.21.15.177:6817/4108616824] conn(0x7fb1200ad5e0 0x7fb1200afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:23.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.329+0000 7fb1377fe700 1 --2- 172.21.15.169:0/3767933097 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1200a2920 0x7fb1200a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb12c00b2e0 tx=0x7fb12c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:23.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.329+0000 7fb13cddb700 1 --2- 172.21.15.169:0/3767933097 >> [v2:172.21.15.177:6816/4108616824,v1:172.21.15.177:6817/4108616824] conn(0x7fb1200ad5e0 0x7fb1200afa20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:23.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.330+0000 7fb1357fa700 1 -- 172.21.15.169:0/3767933097 <== osd.8 v2:172.21.15.177:6816/4108616824 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb128012ec0 con 0x7fb1200ad5e0 2024-03-28T00:55:23.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.347+0000 7fb13e83e700 1 -- 172.21.15.169:0/3767933097 --> [v2:172.21.15.177:6816/4108616824,v1:172.21.15.177:6817/4108616824] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb138062ca0 con 0x7fb1200ad5e0 2024-03-28T00:55:23.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.348+0000 7fb1357fa700 1 -- 172.21.15.169:0/3767933097 <== osd.8 v2:172.21.15.177:6816/4108616824 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb138062ca0 con 0x7fb1200ad5e0 2024-03-28T00:55:23.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.348+0000 7fb13e83e700 1 -- 172.21.15.169:0/3767933097 >> [v2:172.21.15.177:6816/4108616824,v1:172.21.15.177:6817/4108616824] conn(0x7fb1200ad5e0 msgr2=0x7fb1200afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:23.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.348+0000 7fb13e83e700 1 --2- 172.21.15.169:0/3767933097 >> [v2:172.21.15.177:6816/4108616824,v1:172.21.15.177:6817/4108616824] conn(0x7fb1200ad5e0 0x7fb1200afa20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:23.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.348+0000 7fb13e83e700 1 -- 172.21.15.169:0/3767933097 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1200a2920 msgr2=0x7fb1200a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:23.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.348+0000 7fb13e83e700 1 --2- 172.21.15.169:0/3767933097 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1200a2920 0x7fb1200a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb12c00b2e0 tx=0x7fb12c005960 comp rx=0 tx=0).stop 2024-03-28T00:55:23.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.348+0000 7fb13e83e700 1 -- 172.21.15.169:0/3767933097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb138102240 msgr2=0x7fb138074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:23.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.348+0000 7fb13e83e700 1 --2- 172.21.15.169:0/3767933097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb138102240 0x7fb138074490 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb12800e9a0 tx=0x7fb12800ecb0 comp rx=0 tx=0).stop 2024-03-28T00:55:23.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.348+0000 7fb13cddb700 1 -- 172.21.15.169:0/3767933097 reap_dead start 2024-03-28T00:55:23.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.348+0000 7fb13e83e700 1 -- 172.21.15.169:0/3767933097 shutdown_connections 2024-03-28T00:55:23.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.348+0000 7fb13e83e700 1 -- 172.21.15.169:0/3767933097 >> 172.21.15.169:0/3767933097 conn(0x7fb138079260 msgr2=0x7fb138100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:23.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.348+0000 7fb13e83e700 1 -- 172.21.15.169:0/3767933097 shutdown_connections 2024-03-28T00:55:23.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:23.348+0000 7fb13e83e700 1 -- 172.21.15.169:0/3767933097 wait complete. 2024-03-28T00:55:23.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:23 smithi169 ceph-mon[32274]: pgmap v313: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:55:23.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:23 smithi174 ceph-mon[36060]: pgmap v313: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:55:23.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:23 smithi177 ceph-mon[36206]: pgmap v313: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:55:24.732 INFO:teuthology.orchestra.run.smithi169.stdout:141733920827 2024-03-28T00:55:24.732 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-28T00:55:25.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.019+0000 7fa1c9aa3700 1 -- 172.21.15.169:0/539947128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1c4069e40 msgr2=0x7fa1c406a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:25.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.019+0000 7fa1c9aa3700 1 --2- 172.21.15.169:0/539947128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1c4069e40 0x7fa1c406a2a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8009920 tx=0x7fa1b8009c30 comp rx=0 tx=0).stop 2024-03-28T00:55:25.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.019+0000 7fa1c9aa3700 1 -- 172.21.15.169:0/539947128 shutdown_connections 2024-03-28T00:55:25.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.019+0000 7fa1c9aa3700 1 --2- 172.21.15.169:0/539947128 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1c406a7e0 0x7fa1c410f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:25.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.019+0000 7fa1c9aa3700 1 --2- 172.21.15.169:0/539947128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1c41081e0 0x7fa1c41085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:25.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.019+0000 7fa1c9aa3700 1 --2- 172.21.15.169:0/539947128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1c4069e40 0x7fa1c406a2a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:25.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.019+0000 7fa1c9aa3700 1 -- 172.21.15.169:0/539947128 >> 172.21.15.169:0/539947128 conn(0x7fa1c4077fc0 msgr2=0x7fa1c40783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:25.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.019+0000 7fa1c9aa3700 1 -- 172.21.15.169:0/539947128 shutdown_connections 2024-03-28T00:55:25.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.019+0000 7fa1c9aa3700 1 -- 172.21.15.169:0/539947128 wait complete. 2024-03-28T00:55:25.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.020+0000 7fa1c9aa3700 1 Processor -- start 2024-03-28T00:55:25.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.020+0000 7fa1c9aa3700 1 -- start start 2024-03-28T00:55:25.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.020+0000 7fa1c9aa3700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1c4069e40 0x7fa1c419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:25.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.020+0000 7fa1c9aa3700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1c406a7e0 0x7fa1c419eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:25.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.020+0000 7fa1c9aa3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1c41081e0 0x7fa1c41987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:25.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.020+0000 7fa1c9aa3700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c4112440 con 0x7fa1c41081e0 2024-03-28T00:55:25.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.020+0000 7fa1c9aa3700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c41122c0 con 0x7fa1c4069e40 2024-03-28T00:55:25.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.020+0000 7fa1c9aa3700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c41125c0 con 0x7fa1c406a7e0 2024-03-28T00:55:25.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.020+0000 7fa1c37fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1c4069e40 0x7fa1c419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:25.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.020+0000 7fa1c37fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1c4069e40 0x7fa1c419e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:52184/0 (socket says 172.21.15.169:52184) 2024-03-28T00:55:25.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.020+0000 7fa1c37fe700 1 -- 172.21.15.169:0/2286360612 learned_addr learned my addr 172.21.15.169:0/2286360612 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:25.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.020+0000 7fa1c3fff700 1 --2- 172.21.15.169:0/2286360612 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1c41081e0 0x7fa1c41987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:25.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.021+0000 7fa1c2ffd700 1 --2- 172.21.15.169:0/2286360612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1c406a7e0 0x7fa1c419eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:25.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.021+0000 7fa1c37fe700 1 -- 172.21.15.169:0/2286360612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1c406a7e0 msgr2=0x7fa1c419eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:25.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.021+0000 7fa1c37fe700 1 --2- 172.21.15.169:0/2286360612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1c406a7e0 0x7fa1c419eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:25.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.021+0000 7fa1c37fe700 1 -- 172.21.15.169:0/2286360612 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1c41081e0 msgr2=0x7fa1c41987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:25.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.021+0000 7fa1c37fe700 1 --2- 172.21.15.169:0/2286360612 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa1c41081e0 0x7fa1c41987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:25.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.021+0000 7fa1c37fe700 1 -- 172.21.15.169:0/2286360612 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c4198fa0 con 0x7fa1c4069e40 2024-03-28T00:55:25.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.021+0000 7fa1c37fe700 1 --2- 172.21.15.169:0/2286360612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1c4069e40 0x7fa1c419e640 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa1b400e9a0 tx=0x7fa1b400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:25.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.021+0000 7fa1c0ff9700 1 -- 172.21.15.169:0/2286360612 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b400ca10 con 0x7fa1c4069e40 2024-03-28T00:55:25.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.021+0000 7fa1c9aa3700 1 -- 172.21.15.169:0/2286360612 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c4199290 con 0x7fa1c4069e40 2024-03-28T00:55:25.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.021+0000 7fa1c0ff9700 1 -- 172.21.15.169:0/2286360612 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1b400cbb0 con 0x7fa1c4069e40 2024-03-28T00:55:25.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.021+0000 7fa1c0ff9700 1 -- 172.21.15.169:0/2286360612 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b4018680 con 0x7fa1c4069e40 2024-03-28T00:55:25.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.021+0000 7fa1c9aa3700 1 -- 172.21.15.169:0/2286360612 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c41a54a0 con 0x7fa1c4069e40 2024-03-28T00:55:25.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.022+0000 7fa1c9aa3700 1 -- 172.21.15.169:0/2286360612 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa1c4062920 con 0x7fa1c4069e40 2024-03-28T00:55:25.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.023+0000 7fa1c0ff9700 1 -- 172.21.15.169:0/2286360612 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fa1b4010430 con 0x7fa1c4069e40 2024-03-28T00:55:25.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.024+0000 7fa1c0ff9700 1 --2- 172.21.15.169:0/2286360612 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa1ac0a2920 0x7fa1ac0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:25.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.024+0000 7fa1c2ffd700 1 --2- 172.21.15.169:0/2286360612 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa1ac0a2920 0x7fa1ac0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:25.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.024+0000 7fa1c0ff9700 1 -- 172.21.15.169:0/2286360612 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa1b40cfb70 con 0x7fa1c4069e40 2024-03-28T00:55:25.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.024+0000 7fa1c0ff9700 1 --2- 172.21.15.169:0/2286360612 >> [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737] conn(0x7fa1ac0ad5e0 0x7fa1ac0afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:25.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.024+0000 7fa1c0ff9700 1 -- 172.21.15.169:0/2286360612 --> [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa1b401ae50 con 0x7fa1ac0ad5e0 2024-03-28T00:55:25.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.024+0000 7fa1c0ff9700 1 -- 172.21.15.169:0/2286360612 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1b40cffc0 con 0x7fa1c4069e40 2024-03-28T00:55:25.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.024+0000 7fa1c2ffd700 1 --2- 172.21.15.169:0/2286360612 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa1ac0a2920 0x7fa1ac0a4de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa1b800c010 tx=0x7fa1b8019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:25.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.024+0000 7fa1c3fff700 1 --2- 172.21.15.169:0/2286360612 >> [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737] conn(0x7fa1ac0ad5e0 0x7fa1ac0afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:25.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.024+0000 7fa1c3fff700 1 --2- 172.21.15.169:0/2286360612 >> [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737] conn(0x7fa1ac0ad5e0 0x7fa1ac0afa20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:25.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.025+0000 7fa1c0ff9700 1 -- 172.21.15.169:0/2286360612 <== osd.0 v2:172.21.15.169:6802/1362598737 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa1b401ae50 con 0x7fa1ac0ad5e0 2024-03-28T00:55:25.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.042+0000 7fa1c9aa3700 1 -- 172.21.15.169:0/2286360612 --> [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa1c4062da0 con 0x7fa1ac0ad5e0 2024-03-28T00:55:25.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.043+0000 7fa1c0ff9700 1 -- 172.21.15.169:0/2286360612 <== osd.0 v2:172.21.15.169:6802/1362598737 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa1c4062da0 con 0x7fa1ac0ad5e0 2024-03-28T00:55:25.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.043+0000 7fa1c9aa3700 1 -- 172.21.15.169:0/2286360612 >> [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737] conn(0x7fa1ac0ad5e0 msgr2=0x7fa1ac0afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:25.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.043+0000 7fa1c9aa3700 1 --2- 172.21.15.169:0/2286360612 >> [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737] conn(0x7fa1ac0ad5e0 0x7fa1ac0afa20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:25.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.043+0000 7fa1c9aa3700 1 -- 172.21.15.169:0/2286360612 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa1ac0a2920 msgr2=0x7fa1ac0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:25.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.043+0000 7fa1c9aa3700 1 --2- 172.21.15.169:0/2286360612 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa1ac0a2920 0x7fa1ac0a4de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa1b800c010 tx=0x7fa1b8019040 comp rx=0 tx=0).stop 2024-03-28T00:55:25.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.043+0000 7fa1c9aa3700 1 -- 172.21.15.169:0/2286360612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1c4069e40 msgr2=0x7fa1c419e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:25.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.043+0000 7fa1c9aa3700 1 --2- 172.21.15.169:0/2286360612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1c4069e40 0x7fa1c419e640 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa1b400e9a0 tx=0x7fa1b400ecb0 comp rx=0 tx=0).stop 2024-03-28T00:55:25.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.043+0000 7fa1c3fff700 1 -- 172.21.15.169:0/2286360612 reap_dead start 2024-03-28T00:55:25.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.043+0000 7fa1c9aa3700 1 -- 172.21.15.169:0/2286360612 shutdown_connections 2024-03-28T00:55:25.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.043+0000 7fa1c9aa3700 1 -- 172.21.15.169:0/2286360612 >> 172.21.15.169:0/2286360612 conn(0x7fa1c4077fc0 msgr2=0x7fa1c40fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:25.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.043+0000 7fa1c9aa3700 1 -- 172.21.15.169:0/2286360612 shutdown_connections 2024-03-28T00:55:25.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:25.043+0000 7fa1c9aa3700 1 -- 172.21.15.169:0/2286360612 wait complete. 2024-03-28T00:55:25.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:25 smithi169 ceph-mon[32274]: pgmap v314: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-28T00:55:25.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:25 smithi174 ceph-mon[36060]: pgmap v314: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-28T00:55:25.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:25 smithi177 ceph-mon[36206]: pgmap v314: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-28T00:55:26.044 INFO:teuthology.orchestra.run.smithi169.stdout:219043332139 2024-03-28T00:55:26.045 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-28T00:55:26.520 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.929+0000 7f88c8405700 1 -- 172.21.15.169:0/3214282866 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88c0107510 msgr2=0x7f88c010f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:26.520 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.929+0000 7f88c8405700 1 --2- 172.21.15.169:0/3214282866 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88c0107510 0x7f88c010f090 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f88b800b0d0 tx=0x7f88b800b3e0 comp rx=0 tx=0).stop 2024-03-28T00:55:26.520 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.929+0000 7f88c8405700 1 -- 172.21.15.169:0/3214282866 shutdown_connections 2024-03-28T00:55:26.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.929+0000 7f88c8405700 1 --2- 172.21.15.169:0/3214282866 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88c0107510 0x7f88c010f090 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:26.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.929+0000 7f88c8405700 1 --2- 172.21.15.169:0/3214282866 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88c006a930 0x7f88c006ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:26.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.929+0000 7f88c8405700 1 --2- 172.21.15.169:0/3214282866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88c006b2e0 0x7f88c0106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:26.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.929+0000 7f88c8405700 1 -- 172.21.15.169:0/3214282866 >> 172.21.15.169:0/3214282866 conn(0x7f88c00770e0 msgr2=0x7f88c00774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:26.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.929+0000 7f88c8405700 1 -- 172.21.15.169:0/3214282866 shutdown_connections 2024-03-28T00:55:26.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.929+0000 7f88c8405700 1 -- 172.21.15.169:0/3214282866 wait complete. 2024-03-28T00:55:26.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.930+0000 7f88c8405700 1 Processor -- start 2024-03-28T00:55:26.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.930+0000 7f88c8405700 1 -- start start 2024-03-28T00:55:26.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.930+0000 7f88c8405700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88c006a930 0x7f88c019e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:26.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.930+0000 7f88c8405700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88c006b2e0 0x7f88c019eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:26.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.930+0000 7f88c8405700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88c0107510 0x7f88c0198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:26.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.930+0000 7f88c8405700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c01124d0 con 0x7f88c006a930 2024-03-28T00:55:26.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.930+0000 7f88c8405700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c0112350 con 0x7f88c0107510 2024-03-28T00:55:26.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.930+0000 7f88c8405700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c0112650 con 0x7f88c006b2e0 2024-03-28T00:55:26.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.930+0000 7f88c61a1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88c006a930 0x7f88c019e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:26.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.930+0000 7f88c61a1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88c006a930 0x7f88c019e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40658/0 (socket says 172.21.15.169:40658) 2024-03-28T00:55:26.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.930+0000 7f88c61a1700 1 -- 172.21.15.169:0/549379973 learned_addr learned my addr 172.21.15.169:0/549379973 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:26.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.930+0000 7f88c69a2700 1 --2- 172.21.15.169:0/549379973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88c0107510 0x7f88c0198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:26.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.930+0000 7f88c59a0700 1 --2- 172.21.15.169:0/549379973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88c006b2e0 0x7f88c019eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:26.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.931+0000 7f88c69a2700 1 -- 172.21.15.169:0/549379973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88c006b2e0 msgr2=0x7f88c019eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:26.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.931+0000 7f88c69a2700 1 --2- 172.21.15.169:0/549379973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88c006b2e0 0x7f88c019eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:26.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.931+0000 7f88c69a2700 1 -- 172.21.15.169:0/549379973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88c006a930 msgr2=0x7f88c019e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:26.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.931+0000 7f88c69a2700 1 --2- 172.21.15.169:0/549379973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88c006a930 0x7f88c019e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:26.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.931+0000 7f88c69a2700 1 -- 172.21.15.169:0/549379973 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c0199010 con 0x7f88c0107510 2024-03-28T00:55:26.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.931+0000 7f88c61a1700 1 --2- 172.21.15.169:0/549379973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88c006a930 0x7f88c019e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:55:26.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.931+0000 7f88c69a2700 1 --2- 172.21.15.169:0/549379973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88c0107510 0x7f88c0198730 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f88b800b770 tx=0x7f88b8007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:26.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.931+0000 7f88b77fe700 1 -- 172.21.15.169:0/549379973 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8021020 con 0x7f88c0107510 2024-03-28T00:55:26.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.931+0000 7f88c8405700 1 -- 172.21.15.169:0/549379973 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c01992a0 con 0x7f88c0107510 2024-03-28T00:55:26.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.931+0000 7f88b77fe700 1 -- 172.21.15.169:0/549379973 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88b8004c50 con 0x7f88c0107510 2024-03-28T00:55:26.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.931+0000 7f88b77fe700 1 -- 172.21.15.169:0/549379973 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b80052b0 con 0x7f88c0107510 2024-03-28T00:55:26.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.931+0000 7f88c8405700 1 -- 172.21.15.169:0/549379973 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c01a55e0 con 0x7f88c0107510 2024-03-28T00:55:26.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.932+0000 7f88c8405700 1 -- 172.21.15.169:0/549379973 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f88c00628e0 con 0x7f88c0107510 2024-03-28T00:55:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.933+0000 7f88b77fe700 1 -- 172.21.15.169:0/549379973 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f88b8024020 con 0x7f88c0107510 2024-03-28T00:55:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.933+0000 7f88b77fe700 1 --2- 172.21.15.169:0/549379973 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f88ac0a2920 0x7f88ac0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.934+0000 7f88c61a1700 1 --2- 172.21.15.169:0/549379973 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f88ac0a2920 0x7f88ac0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.934+0000 7f88b77fe700 1 -- 172.21.15.169:0/549379973 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f88b800f020 con 0x7f88c0107510 2024-03-28T00:55:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.934+0000 7f88b77fe700 1 --2- 172.21.15.169:0/549379973 >> [v2:172.21.15.177:6800/1495865602,v1:172.21.15.177:6801/1495865602] conn(0x7f88ac0ad5e0 0x7f88ac0afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.934+0000 7f88b77fe700 1 -- 172.21.15.169:0/549379973 --> [v2:172.21.15.177:6800/1495865602,v1:172.21.15.177:6801/1495865602] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88ac0b00f0 con 0x7f88ac0ad5e0 2024-03-28T00:55:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.934+0000 7f88b77fe700 1 -- 172.21.15.169:0/549379973 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88b80d0830 con 0x7f88c0107510 2024-03-28T00:55:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.934+0000 7f88c59a0700 1 --2- 172.21.15.169:0/549379973 >> [v2:172.21.15.177:6800/1495865602,v1:172.21.15.177:6801/1495865602] conn(0x7f88ac0ad5e0 0x7f88ac0afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.934+0000 7f88c61a1700 1 --2- 172.21.15.169:0/549379973 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f88ac0a2920 0x7f88ac0a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f88bc011590 tx=0x7f88bc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.934+0000 7f88c59a0700 1 --2- 172.21.15.169:0/549379973 >> [v2:172.21.15.177:6800/1495865602,v1:172.21.15.177:6801/1495865602] conn(0x7f88ac0ad5e0 0x7f88ac0afa20 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.935+0000 7f88b77fe700 1 -- 172.21.15.169:0/549379973 <== osd.2 v2:172.21.15.177:6800/1495865602 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f88ac0b00f0 con 0x7f88ac0ad5e0 2024-03-28T00:55:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.951+0000 7f88c8405700 1 -- 172.21.15.169:0/549379973 --> [v2:172.21.15.177:6800/1495865602,v1:172.21.15.177:6801/1495865602] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f88c0062ce0 con 0x7f88ac0ad5e0 2024-03-28T00:55:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.952+0000 7f88b77fe700 1 -- 172.21.15.169:0/549379973 <== osd.2 v2:172.21.15.177:6800/1495865602 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f88c0062ce0 con 0x7f88ac0ad5e0 2024-03-28T00:55:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.952+0000 7f88c8405700 1 -- 172.21.15.169:0/549379973 >> [v2:172.21.15.177:6800/1495865602,v1:172.21.15.177:6801/1495865602] conn(0x7f88ac0ad5e0 msgr2=0x7f88ac0afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:26.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.952+0000 7f88c8405700 1 --2- 172.21.15.169:0/549379973 >> [v2:172.21.15.177:6800/1495865602,v1:172.21.15.177:6801/1495865602] conn(0x7f88ac0ad5e0 0x7f88ac0afa20 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:26.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.952+0000 7f88c8405700 1 -- 172.21.15.169:0/549379973 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f88ac0a2920 msgr2=0x7f88ac0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:26.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.952+0000 7f88c8405700 1 --2- 172.21.15.169:0/549379973 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f88ac0a2920 0x7f88ac0a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f88bc011590 tx=0x7f88bc009260 comp rx=0 tx=0).stop 2024-03-28T00:55:26.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.952+0000 7f88c8405700 1 -- 172.21.15.169:0/549379973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88c0107510 msgr2=0x7f88c0198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:26.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.952+0000 7f88c8405700 1 --2- 172.21.15.169:0/549379973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88c0107510 0x7f88c0198730 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f88b800b770 tx=0x7f88b8007a20 comp rx=0 tx=0).stop 2024-03-28T00:55:26.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.952+0000 7f88c69a2700 1 -- 172.21.15.169:0/549379973 reap_dead start 2024-03-28T00:55:26.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.952+0000 7f88c8405700 1 -- 172.21.15.169:0/549379973 shutdown_connections 2024-03-28T00:55:26.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.952+0000 7f88c8405700 1 -- 172.21.15.169:0/549379973 >> 172.21.15.169:0/549379973 conn(0x7f88c00770e0 msgr2=0x7f88c00ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:26.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.953+0000 7f88c8405700 1 -- 172.21.15.169:0/549379973 shutdown_connections 2024-03-28T00:55:26.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:24.953+0000 7f88c8405700 1 -- 172.21.15.169:0/549379973 wait complete. 2024-03-28T00:55:27.161 INFO:teuthology.orchestra.run.smithi169.stdout:77309411396 2024-03-28T00:55:27.162 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-28T00:55:27.395 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:27 smithi169 ceph-mon[32274]: pgmap v315: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T00:55:27.610 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:27 smithi174 ceph-mon[36060]: pgmap v315: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T00:55:27.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:27 smithi177 ceph-mon[36206]: pgmap v315: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T00:55:27.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.832+0000 7fb2205b6700 1 -- 172.21.15.169:0/2823139811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2181021e0 msgr2=0x7fb218102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:27.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.832+0000 7fb2205b6700 1 --2- 172.21.15.169:0/2823139811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2181021e0 0x7fb218102660 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb214009990 tx=0x7fb214009ca0 comp rx=0 tx=0).stop 2024-03-28T00:55:27.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.832+0000 7fb2205b6700 1 -- 172.21.15.169:0/2823139811 shutdown_connections 2024-03-28T00:55:27.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.832+0000 7fb2205b6700 1 --2- 172.21.15.169:0/2823139811 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb218102ba0 0x7fb21810f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.832+0000 7fb2205b6700 1 --2- 172.21.15.169:0/2823139811 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb218108290 0x7fb218108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.832+0000 7fb2205b6700 1 --2- 172.21.15.169:0/2823139811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2181021e0 0x7fb218102660 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.832+0000 7fb2205b6700 1 -- 172.21.15.169:0/2823139811 >> 172.21.15.169:0/2823139811 conn(0x7fb2180792c0 msgr2=0x7fb2180796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.832+0000 7fb2205b6700 1 -- 172.21.15.169:0/2823139811 shutdown_connections 2024-03-28T00:55:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.832+0000 7fb2205b6700 1 -- 172.21.15.169:0/2823139811 wait complete. 2024-03-28T00:55:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.833+0000 7fb2205b6700 1 Processor -- start 2024-03-28T00:55:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.833+0000 7fb2205b6700 1 -- start start 2024-03-28T00:55:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.833+0000 7fb2205b6700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2181021e0 0x7fb21819e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.833+0000 7fb2205b6700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb218102ba0 0x7fb21819eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.833+0000 7fb2205b6700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb218108290 0x7fb218198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.833+0000 7fb2205b6700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2181124a0 con 0x7fb218108290 2024-03-28T00:55:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.833+0000 7fb2205b6700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb218112320 con 0x7fb2181021e0 2024-03-28T00:55:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.833+0000 7fb2205b6700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb218112620 con 0x7fb218102ba0 2024-03-28T00:55:27.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.834+0000 7fb21e352700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2181021e0 0x7fb21819e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:27.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.834+0000 7fb21eb53700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb218108290 0x7fb218198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:27.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.834+0000 7fb21db51700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb218102ba0 0x7fb21819eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:27.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.834+0000 7fb21e352700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2181021e0 0x7fb21819e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:43372/0 (socket says 172.21.15.169:43372) 2024-03-28T00:55:27.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.834+0000 7fb21e352700 1 -- 172.21.15.169:0/2583515075 learned_addr learned my addr 172.21.15.169:0/2583515075 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:27.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.834+0000 7fb21db51700 1 -- 172.21.15.169:0/2583515075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2181021e0 msgr2=0x7fb21819e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:27.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.834+0000 7fb21db51700 1 --2- 172.21.15.169:0/2583515075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2181021e0 0x7fb21819e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:27.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.834+0000 7fb21db51700 1 -- 172.21.15.169:0/2583515075 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb218108290 msgr2=0x7fb218198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:27.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.834+0000 7fb21db51700 1 --2- 172.21.15.169:0/2583515075 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb218108290 0x7fb218198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:27.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.834+0000 7fb21db51700 1 -- 172.21.15.169:0/2583515075 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb218198f60 con 0x7fb218102ba0 2024-03-28T00:55:27.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.834+0000 7fb21eb53700 1 --2- 172.21.15.169:0/2583515075 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb218108290 0x7fb218198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:55:27.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.834+0000 7fb21db51700 1 --2- 172.21.15.169:0/2583515075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb218102ba0 0x7fb21819eb70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb2140059d0 tx=0x7fb21400bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:27.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.834+0000 7fb20f7fe700 1 -- 172.21.15.169:0/2583515075 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb214027070 con 0x7fb218102ba0 2024-03-28T00:55:27.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.834+0000 7fb20f7fe700 1 -- 172.21.15.169:0/2583515075 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb214003d70 con 0x7fb218102ba0 2024-03-28T00:55:27.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.834+0000 7fb2205b6700 1 -- 172.21.15.169:0/2583515075 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2181991f0 con 0x7fb218102ba0 2024-03-28T00:55:27.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.834+0000 7fb20f7fe700 1 -- 172.21.15.169:0/2583515075 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21400f460 con 0x7fb218102ba0 2024-03-28T00:55:27.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.834+0000 7fb2205b6700 1 -- 172.21.15.169:0/2583515075 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2180743c0 con 0x7fb218102ba0 2024-03-28T00:55:27.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.835+0000 7fb2205b6700 1 -- 172.21.15.169:0/2583515075 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb218062860 con 0x7fb218102ba0 2024-03-28T00:55:27.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.836+0000 7fb20f7fe700 1 -- 172.21.15.169:0/2583515075 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fb21402a070 con 0x7fb218102ba0 2024-03-28T00:55:27.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.837+0000 7fb20f7fe700 1 --2- 172.21.15.169:0/2583515075 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb2040a2920 0x7fb2040a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:27.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.837+0000 7fb21e352700 1 --2- 172.21.15.169:0/2583515075 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb2040a2920 0x7fb2040a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:27.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.837+0000 7fb20f7fe700 1 -- 172.21.15.169:0/2583515075 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb2140d0820 con 0x7fb218102ba0 2024-03-28T00:55:27.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.837+0000 7fb20f7fe700 1 --2- 172.21.15.169:0/2583515075 >> [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866] conn(0x7fb2040ad5e0 0x7fb2040afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:27.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.837+0000 7fb20f7fe700 1 -- 172.21.15.169:0/2583515075 --> [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb214032e50 con 0x7fb2040ad5e0 2024-03-28T00:55:27.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.837+0000 7fb20f7fe700 1 -- 172.21.15.169:0/2583515075 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2140d0c70 con 0x7fb218102ba0 2024-03-28T00:55:27.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.838+0000 7fb21e352700 1 --2- 172.21.15.169:0/2583515075 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb2040a2920 0x7fb2040a4de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb208009680 tx=0x7fb208009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:27.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.838+0000 7fb21eb53700 1 --2- 172.21.15.169:0/2583515075 >> [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866] conn(0x7fb2040ad5e0 0x7fb2040afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:27.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.838+0000 7fb21eb53700 1 --2- 172.21.15.169:0/2583515075 >> [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866] conn(0x7fb2040ad5e0 0x7fb2040afa20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:27.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.839+0000 7fb20f7fe700 1 -- 172.21.15.169:0/2583515075 <== osd.3 v2:172.21.15.169:6810/149376866 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb214032e50 con 0x7fb2040ad5e0 2024-03-28T00:55:27.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.855+0000 7fb2205b6700 1 -- 172.21.15.169:0/2583515075 --> [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb218062c60 con 0x7fb2040ad5e0 2024-03-28T00:55:27.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.855+0000 7fb20f7fe700 1 -- 172.21.15.169:0/2583515075 <== osd.3 v2:172.21.15.169:6810/149376866 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb218062c60 con 0x7fb2040ad5e0 2024-03-28T00:55:27.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.855+0000 7fb2205b6700 1 -- 172.21.15.169:0/2583515075 >> [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866] conn(0x7fb2040ad5e0 msgr2=0x7fb2040afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:27.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.855+0000 7fb2205b6700 1 --2- 172.21.15.169:0/2583515075 >> [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866] conn(0x7fb2040ad5e0 0x7fb2040afa20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:27.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.855+0000 7fb2205b6700 1 -- 172.21.15.169:0/2583515075 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb2040a2920 msgr2=0x7fb2040a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:27.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.855+0000 7fb2205b6700 1 --2- 172.21.15.169:0/2583515075 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb2040a2920 0x7fb2040a4de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb208009680 tx=0x7fb208009260 comp rx=0 tx=0).stop 2024-03-28T00:55:27.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.855+0000 7fb2205b6700 1 -- 172.21.15.169:0/2583515075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb218102ba0 msgr2=0x7fb21819eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:27.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.855+0000 7fb2205b6700 1 --2- 172.21.15.169:0/2583515075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb218102ba0 0x7fb21819eb70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb2140059d0 tx=0x7fb21400bab0 comp rx=0 tx=0).stop 2024-03-28T00:55:27.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.855+0000 7fb21eb53700 1 -- 172.21.15.169:0/2583515075 reap_dead start 2024-03-28T00:55:27.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.855+0000 7fb2205b6700 1 -- 172.21.15.169:0/2583515075 shutdown_connections 2024-03-28T00:55:27.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.855+0000 7fb2205b6700 1 -- 172.21.15.169:0/2583515075 >> 172.21.15.169:0/2583515075 conn(0x7fb2180792c0 msgr2=0x7fb2181010e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:27.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.856+0000 7fb2205b6700 1 -- 172.21.15.169:0/2583515075 shutdown_connections 2024-03-28T00:55:27.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:27.856+0000 7fb2205b6700 1 -- 172.21.15.169:0/2583515075 wait complete. 2024-03-28T00:55:28.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.044+0000 7fe43d8c4700 1 -- 172.21.15.169:0/2094203949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4381030d0 msgr2=0x7fe4381034b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:28.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.044+0000 7fe43d8c4700 1 --2- 172.21.15.169:0/2094203949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4381030d0 0x7fe4381034b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009990 tx=0x7fe42c009ca0 comp rx=0 tx=0).stop 2024-03-28T00:55:28.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.044+0000 7fe43d8c4700 1 -- 172.21.15.169:0/2094203949 shutdown_connections 2024-03-28T00:55:28.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.044+0000 7fe43d8c4700 1 --2- 172.21.15.169:0/2094203949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe438101310 0x7fe438110f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:28.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.044+0000 7fe43d8c4700 1 --2- 172.21.15.169:0/2094203949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4381030d0 0x7fe4381034b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:28.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.044+0000 7fe43d8c4700 1 --2- 172.21.15.169:0/2094203949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe438100780 0x7fe438100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:28.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.044+0000 7fe43d8c4700 1 -- 172.21.15.169:0/2094203949 >> 172.21.15.169:0/2094203949 conn(0x7fe438077690 msgr2=0x7fe4380feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:28.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.044+0000 7fe43d8c4700 1 -- 172.21.15.169:0/2094203949 shutdown_connections 2024-03-28T00:55:28.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.044+0000 7fe43d8c4700 1 -- 172.21.15.169:0/2094203949 wait complete. 2024-03-28T00:55:28.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.044+0000 7fe43d8c4700 1 Processor -- start 2024-03-28T00:55:28.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.045+0000 7fe43d8c4700 1 -- start start 2024-03-28T00:55:28.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.045+0000 7fe43d8c4700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe438100780 0x7fe4381a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:28.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.045+0000 7fe43d8c4700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe438101310 0x7fe4381a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:28.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.045+0000 7fe43d8c4700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4381030d0 0x7fe43819a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:28.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.045+0000 7fe43d8c4700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe438114710 con 0x7fe4381030d0 2024-03-28T00:55:28.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.045+0000 7fe43d8c4700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe438114590 con 0x7fe438101310 2024-03-28T00:55:28.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.045+0000 7fe43d8c4700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe438114890 con 0x7fe438100780 2024-03-28T00:55:28.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.045+0000 7fe436ffd700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe438100780 0x7fe4381a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:28.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.045+0000 7fe4367fc700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe438101310 0x7fe4381a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:28.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.045+0000 7fe436ffd700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe438100780 0x7fe4381a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:38760/0 (socket says 172.21.15.169:38760) 2024-03-28T00:55:28.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.045+0000 7fe4377fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4381030d0 0x7fe43819a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:28.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.045+0000 7fe436ffd700 1 -- 172.21.15.169:0/771891460 learned_addr learned my addr 172.21.15.169:0/771891460 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:28.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.045+0000 7fe436ffd700 1 -- 172.21.15.169:0/771891460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe438101310 msgr2=0x7fe4381a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:28.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.045+0000 7fe436ffd700 1 --2- 172.21.15.169:0/771891460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe438101310 0x7fe4381a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:28.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.045+0000 7fe436ffd700 1 -- 172.21.15.169:0/771891460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4381030d0 msgr2=0x7fe43819a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:28.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.045+0000 7fe436ffd700 1 --2- 172.21.15.169:0/771891460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4381030d0 0x7fe43819a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:28.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.045+0000 7fe436ffd700 1 -- 172.21.15.169:0/771891460 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe43819b130 con 0x7fe438100780 2024-03-28T00:55:28.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.045+0000 7fe4367fc700 1 --2- 172.21.15.169:0/771891460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe438101310 0x7fe4381a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:55:28.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.045+0000 7fe4377fe700 1 --2- 172.21.15.169:0/771891460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe4381030d0 0x7fe43819a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:55:28.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.046+0000 7fe436ffd700 1 --2- 172.21.15.169:0/771891460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe438100780 0x7fe4381a0810 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe42c00bae0 tx=0x7fe42c00bb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:28.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.046+0000 7fe43c8c2700 1 -- 172.21.15.169:0/771891460 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42c026070 con 0x7fe438100780 2024-03-28T00:55:28.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.046+0000 7fe43c8c2700 1 -- 172.21.15.169:0/771891460 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe42c003d70 con 0x7fe438100780 2024-03-28T00:55:28.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.046+0000 7fe43c8c2700 1 -- 172.21.15.169:0/771891460 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42c021410 con 0x7fe438100780 2024-03-28T00:55:28.053 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.046+0000 7fe43d8c4700 1 -- 172.21.15.169:0/771891460 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe43819b3c0 con 0x7fe438100780 2024-03-28T00:55:28.053 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.046+0000 7fe43d8c4700 1 -- 172.21.15.169:0/771891460 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4381a76e0 con 0x7fe438100780 2024-03-28T00:55:28.053 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.047+0000 7fe43d8c4700 1 -- 172.21.15.169:0/771891460 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe438062860 con 0x7fe438100780 2024-03-28T00:55:28.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.048+0000 7fe43c8c2700 1 -- 172.21.15.169:0/771891460 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fe42c029070 con 0x7fe438100780 2024-03-28T00:55:28.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.049+0000 7fe43c8c2700 1 --2- 172.21.15.169:0/771891460 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe4200a29a0 0x7fe4200a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:28.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.049+0000 7fe4367fc700 1 --2- 172.21.15.169:0/771891460 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe4200a29a0 0x7fe4200a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:28.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.049+0000 7fe43c8c2700 1 -- 172.21.15.169:0/771891460 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fe42c0cfe80 con 0x7fe438100780 2024-03-28T00:55:28.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.049+0000 7fe43c8c2700 1 --2- 172.21.15.169:0/771891460 >> [v2:172.21.15.177:6808/3583242703,v1:172.21.15.177:6809/3583242703] conn(0x7fe4200ad530 0x7fe4200af970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:28.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.049+0000 7fe43c8c2700 1 -- 172.21.15.169:0/771891460 --> [v2:172.21.15.177:6808/3583242703,v1:172.21.15.177:6809/3583242703] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe42c015ec0 con 0x7fe4200ad530 2024-03-28T00:55:28.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.049+0000 7fe43c8c2700 1 -- 172.21.15.169:0/771891460 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe42c0d02d0 con 0x7fe438100780 2024-03-28T00:55:28.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.050+0000 7fe4377fe700 1 --2- 172.21.15.169:0/771891460 >> [v2:172.21.15.177:6808/3583242703,v1:172.21.15.177:6809/3583242703] conn(0x7fe4200ad530 0x7fe4200af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:28.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.050+0000 7fe4367fc700 1 --2- 172.21.15.169:0/771891460 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe4200a29a0 0x7fe4200a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe4380ffd20 tx=0x7fe428009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:28.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.050+0000 7fe4377fe700 1 --2- 172.21.15.169:0/771891460 >> [v2:172.21.15.177:6808/3583242703,v1:172.21.15.177:6809/3583242703] conn(0x7fe4200ad530 0x7fe4200af970 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:28.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.051+0000 7fe43c8c2700 1 -- 172.21.15.169:0/771891460 <== osd.5 v2:172.21.15.177:6808/3583242703 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe42c015ec0 con 0x7fe4200ad530 2024-03-28T00:55:28.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.067+0000 7fe43d8c4700 1 -- 172.21.15.169:0/771891460 --> [v2:172.21.15.177:6808/3583242703,v1:172.21.15.177:6809/3583242703] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe438062c90 con 0x7fe4200ad530 2024-03-28T00:55:28.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.067+0000 7fe43c8c2700 1 -- 172.21.15.169:0/771891460 <== osd.5 v2:172.21.15.177:6808/3583242703 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe438062c90 con 0x7fe4200ad530 2024-03-28T00:55:28.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.067+0000 7fe43d8c4700 1 -- 172.21.15.169:0/771891460 >> [v2:172.21.15.177:6808/3583242703,v1:172.21.15.177:6809/3583242703] conn(0x7fe4200ad530 msgr2=0x7fe4200af970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:28.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.067+0000 7fe43d8c4700 1 --2- 172.21.15.169:0/771891460 >> [v2:172.21.15.177:6808/3583242703,v1:172.21.15.177:6809/3583242703] conn(0x7fe4200ad530 0x7fe4200af970 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:28.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.067+0000 7fe43d8c4700 1 -- 172.21.15.169:0/771891460 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe4200a29a0 msgr2=0x7fe4200a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:28.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.067+0000 7fe43d8c4700 1 --2- 172.21.15.169:0/771891460 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe4200a29a0 0x7fe4200a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe4380ffd20 tx=0x7fe428009260 comp rx=0 tx=0).stop 2024-03-28T00:55:28.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.067+0000 7fe43d8c4700 1 -- 172.21.15.169:0/771891460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe438100780 msgr2=0x7fe4381a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:28.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.067+0000 7fe43d8c4700 1 --2- 172.21.15.169:0/771891460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe438100780 0x7fe4381a0810 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe42c00bae0 tx=0x7fe42c00bb10 comp rx=0 tx=0).stop 2024-03-28T00:55:28.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.067+0000 7fe4377fe700 1 -- 172.21.15.169:0/771891460 reap_dead start 2024-03-28T00:55:28.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.068+0000 7fe43d8c4700 1 -- 172.21.15.169:0/771891460 shutdown_connections 2024-03-28T00:55:28.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.068+0000 7fe43d8c4700 1 -- 172.21.15.169:0/771891460 >> 172.21.15.169:0/771891460 conn(0x7fe438077690 msgr2=0x7fe4380fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:28.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.068+0000 7fe43d8c4700 1 -- 172.21.15.169:0/771891460 shutdown_connections 2024-03-28T00:55:28.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.068+0000 7fe43d8c4700 1 -- 172.21.15.169:0/771891460 wait complete. 2024-03-28T00:55:28.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.259+0000 7fed4e988700 1 -- 172.21.15.169:0/2123480392 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed48102dc0 msgr2=0x7fed48106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:28.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.259+0000 7fed4e988700 1 --2- 172.21.15.169:0/2123480392 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed48102dc0 0x7fed48106e10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fed3c0099c0 tx=0x7fed3c009cd0 comp rx=0 tx=0).stop 2024-03-28T00:55:28.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.259+0000 7fed4e988700 1 -- 172.21.15.169:0/2123480392 shutdown_connections 2024-03-28T00:55:28.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.259+0000 7fed4e988700 1 --2- 172.21.15.169:0/2123480392 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed481074f0 0x7fed4810f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:28.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.259+0000 7fed4e988700 1 --2- 172.21.15.169:0/2123480392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed48102410 0x7fed481027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:28.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.259+0000 7fed4e988700 1 --2- 172.21.15.169:0/2123480392 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed48102dc0 0x7fed48106e10 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:28.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.259+0000 7fed4e988700 1 -- 172.21.15.169:0/2123480392 >> 172.21.15.169:0/2123480392 conn(0x7fed480fdc60 msgr2=0x7fed48100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:28.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.259+0000 7fed4e988700 1 -- 172.21.15.169:0/2123480392 shutdown_connections 2024-03-28T00:55:28.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.259+0000 7fed4e988700 1 -- 172.21.15.169:0/2123480392 wait complete. 2024-03-28T00:55:28.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.260+0000 7fed4e988700 1 Processor -- start 2024-03-28T00:55:28.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.260+0000 7fed4e988700 1 -- start start 2024-03-28T00:55:28.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.260+0000 7fed4e988700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed48102410 0x7fed48074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:28.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.260+0000 7fed4e988700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed48102dc0 0x7fed480749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:28.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.260+0000 7fed4e988700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed481074f0 0x7fed48074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:28.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.260+0000 7fed4e988700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed481124b0 con 0x7fed48102dc0 2024-03-28T00:55:28.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.260+0000 7fed4e988700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed48112330 con 0x7fed481074f0 2024-03-28T00:55:28.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.260+0000 7fed4e988700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed48112630 con 0x7fed48102410 2024-03-28T00:55:28.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.260+0000 7fed4cf25700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed481074f0 0x7fed48074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:28.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.260+0000 7fed47fff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed48102410 0x7fed48074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:28.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.260+0000 7fed4cf25700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed481074f0 0x7fed48074f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:43400/0 (socket says 172.21.15.169:43400) 2024-03-28T00:55:28.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.260+0000 7fed4cf25700 1 -- 172.21.15.169:0/463732422 learned_addr learned my addr 172.21.15.169:0/463732422 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:28.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.260+0000 7fed47fff700 1 --2- 172.21.15.169:0/463732422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed48102410 0x7fed48074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:38780/0 (socket says 172.21.15.169:38780) 2024-03-28T00:55:28.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.260+0000 7fed477fe700 1 --2- 172.21.15.169:0/463732422 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed48102dc0 0x7fed480749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:28.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.261+0000 7fed4cf25700 1 -- 172.21.15.169:0/463732422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed48102410 msgr2=0x7fed48074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:28.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.261+0000 7fed4cf25700 1 --2- 172.21.15.169:0/463732422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed48102410 0x7fed48074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:28.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.261+0000 7fed4cf25700 1 -- 172.21.15.169:0/463732422 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed48102dc0 msgr2=0x7fed480749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:28.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.261+0000 7fed4cf25700 1 --2- 172.21.15.169:0/463732422 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed48102dc0 0x7fed480749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:28.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.261+0000 7fed4cf25700 1 -- 172.21.15.169:0/463732422 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed481a6e80 con 0x7fed481074f0 2024-03-28T00:55:28.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.261+0000 7fed4cf25700 1 --2- 172.21.15.169:0/463732422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed481074f0 0x7fed48074f10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fed4000ef30 tx=0x7fed4000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:28.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.261+0000 7fed457fa700 1 -- 172.21.15.169:0/463732422 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed400092e0 con 0x7fed481074f0 2024-03-28T00:55:28.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.261+0000 7fed457fa700 1 -- 172.21.15.169:0/463732422 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed40004bb0 con 0x7fed481074f0 2024-03-28T00:55:28.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.261+0000 7fed457fa700 1 -- 172.21.15.169:0/463732422 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40004280 con 0x7fed481074f0 2024-03-28T00:55:28.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.262+0000 7fed4e988700 1 -- 172.21.15.169:0/463732422 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed481a7110 con 0x7fed481074f0 2024-03-28T00:55:28.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.262+0000 7fed4e988700 1 -- 172.21.15.169:0/463732422 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed481a75d0 con 0x7fed481074f0 2024-03-28T00:55:28.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.262+0000 7fed4e988700 1 -- 172.21.15.169:0/463732422 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fed480628a0 con 0x7fed481074f0 2024-03-28T00:55:28.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.264+0000 7fed457fa700 1 -- 172.21.15.169:0/463732422 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fed4001b030 con 0x7fed481074f0 2024-03-28T00:55:28.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.264+0000 7fed457fa700 1 --2- 172.21.15.169:0/463732422 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed300a2920 0x7fed300a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:28.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.265+0000 7fed47fff700 1 --2- 172.21.15.169:0/463732422 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed300a2920 0x7fed300a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:28.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.265+0000 7fed457fa700 1 -- 172.21.15.169:0/463732422 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fed400cf1e0 con 0x7fed481074f0 2024-03-28T00:55:28.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.265+0000 7fed457fa700 1 --2- 172.21.15.169:0/463732422 >> [v2:172.21.15.174:6824/3371468956,v1:172.21.15.174:6825/3371468956] conn(0x7fed300ad5e0 0x7fed300afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:28.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.265+0000 7fed457fa700 1 -- 172.21.15.169:0/463732422 --> [v2:172.21.15.174:6824/3371468956,v1:172.21.15.174:6825/3371468956] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fed4001fe50 con 0x7fed300ad5e0 2024-03-28T00:55:28.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.265+0000 7fed457fa700 1 -- 172.21.15.169:0/463732422 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed400cf620 con 0x7fed481074f0 2024-03-28T00:55:28.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.265+0000 7fed477fe700 1 --2- 172.21.15.169:0/463732422 >> [v2:172.21.15.174:6824/3371468956,v1:172.21.15.174:6825/3371468956] conn(0x7fed300ad5e0 0x7fed300afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:28.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.265+0000 7fed47fff700 1 --2- 172.21.15.169:0/463732422 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed300a2920 0x7fed300a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fed38009c60 tx=0x7fed38009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:28.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.265+0000 7fed477fe700 1 --2- 172.21.15.169:0/463732422 >> [v2:172.21.15.174:6824/3371468956,v1:172.21.15.174:6825/3371468956] conn(0x7fed300ad5e0 0x7fed300afa20 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:28.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.266+0000 7fed457fa700 1 -- 172.21.15.169:0/463732422 <== osd.10 v2:172.21.15.174:6824/3371468956 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fed4001fe50 con 0x7fed300ad5e0 2024-03-28T00:55:28.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.283+0000 7fed4e988700 1 -- 172.21.15.169:0/463732422 --> [v2:172.21.15.174:6824/3371468956,v1:172.21.15.174:6825/3371468956] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fed48062d20 con 0x7fed300ad5e0 2024-03-28T00:55:28.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.283+0000 7fed457fa700 1 -- 172.21.15.169:0/463732422 <== osd.10 v2:172.21.15.174:6824/3371468956 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fed48062d20 con 0x7fed300ad5e0 2024-03-28T00:55:28.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.283+0000 7fed4e988700 1 -- 172.21.15.169:0/463732422 >> [v2:172.21.15.174:6824/3371468956,v1:172.21.15.174:6825/3371468956] conn(0x7fed300ad5e0 msgr2=0x7fed300afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:28.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.283+0000 7fed4e988700 1 --2- 172.21.15.169:0/463732422 >> [v2:172.21.15.174:6824/3371468956,v1:172.21.15.174:6825/3371468956] conn(0x7fed300ad5e0 0x7fed300afa20 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:28.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.283+0000 7fed4e988700 1 -- 172.21.15.169:0/463732422 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed300a2920 msgr2=0x7fed300a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:28.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.283+0000 7fed4e988700 1 --2- 172.21.15.169:0/463732422 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed300a2920 0x7fed300a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fed38009c60 tx=0x7fed38009260 comp rx=0 tx=0).stop 2024-03-28T00:55:28.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.283+0000 7fed4e988700 1 -- 172.21.15.169:0/463732422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed481074f0 msgr2=0x7fed48074f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:28.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.283+0000 7fed4e988700 1 --2- 172.21.15.169:0/463732422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed481074f0 0x7fed48074f10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fed4000ef30 tx=0x7fed4000c3a0 comp rx=0 tx=0).stop 2024-03-28T00:55:28.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.284+0000 7fed4cf25700 1 -- 172.21.15.169:0/463732422 reap_dead start 2024-03-28T00:55:28.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.284+0000 7fed4e988700 1 -- 172.21.15.169:0/463732422 shutdown_connections 2024-03-28T00:55:28.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.284+0000 7fed4e988700 1 -- 172.21.15.169:0/463732422 >> 172.21.15.169:0/463732422 conn(0x7fed480fdc60 msgr2=0x7fed480fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:28.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.284+0000 7fed4e988700 1 -- 172.21.15.169:0/463732422 shutdown_connections 2024-03-28T00:55:28.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:28.284+0000 7fed4e988700 1 -- 172.21.15.169:0/463732422 wait complete. 2024-03-28T00:55:29.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.053+0000 7fa9ae7fe700 1 -- 172.21.15.169:0/2753953589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9a81070d0 msgr2=0x7fa9a81074b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:29.059 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.053+0000 7fa9ae7fe700 1 --2- 172.21.15.169:0/2753953589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9a81070d0 0x7fa9a81074b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa998009990 tx=0x7fa998009ca0 comp rx=0 tx=0).stop 2024-03-28T00:55:29.059 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.053+0000 7fa9ae7fe700 1 -- 172.21.15.169:0/2753953589 shutdown_connections 2024-03-28T00:55:29.059 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.053+0000 7fa9ae7fe700 1 --2- 172.21.15.169:0/2753953589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9a8101a30 0x7fa9a810df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:29.059 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.053+0000 7fa9ae7fe700 1 --2- 172.21.15.169:0/2753953589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9a81070d0 0x7fa9a81074b0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:29.059 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.053+0000 7fa9ae7fe700 1 --2- 172.21.15.169:0/2753953589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9a8101090 0x7fa9a81014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:29.059 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.053+0000 7fa9ae7fe700 1 -- 172.21.15.169:0/2753953589 >> 172.21.15.169:0/2753953589 conn(0x7fa9a8077fc0 msgr2=0x7fa9a80783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:29.059 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.053+0000 7fa9ae7fe700 1 -- 172.21.15.169:0/2753953589 shutdown_connections 2024-03-28T00:55:29.059 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.054+0000 7fa9ae7fe700 1 -- 172.21.15.169:0/2753953589 wait complete. 2024-03-28T00:55:29.059 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.054+0000 7fa9ae7fe700 1 Processor -- start 2024-03-28T00:55:29.059 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.054+0000 7fa9ae7fe700 1 -- start start 2024-03-28T00:55:29.059 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.054+0000 7fa9ae7fe700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9a8101090 0x7fa9a819d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:29.059 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.054+0000 7fa9ae7fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9a8101a30 0x7fa9a819d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:29.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.055+0000 7fa9ae7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9a81070d0 0x7fa9a8197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:29.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.055+0000 7fa9ae7fe700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a8111360 con 0x7fa9a81070d0 2024-03-28T00:55:29.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.055+0000 7fa9ae7fe700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a81111e0 con 0x7fa9a8101a30 2024-03-28T00:55:29.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.055+0000 7fa9ae7fe700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a81114e0 con 0x7fa9a8101090 2024-03-28T00:55:29.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.055+0000 7fa9a7fff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9a8101090 0x7fa9a819d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:29.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.055+0000 7fa9a7fff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9a8101090 0x7fa9a819d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:38788/0 (socket says 172.21.15.169:38788) 2024-03-28T00:55:29.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.055+0000 7fa9a7fff700 1 -- 172.21.15.169:0/3467873796 learned_addr learned my addr 172.21.15.169:0/3467873796 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:29.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.055+0000 7fa9acd9b700 1 --2- 172.21.15.169:0/3467873796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9a81070d0 0x7fa9a8197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:29.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.055+0000 7fa9a77fe700 1 --2- 172.21.15.169:0/3467873796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9a8101a30 0x7fa9a819d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:29.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.055+0000 7fa9a7fff700 1 -- 172.21.15.169:0/3467873796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9a8101a30 msgr2=0x7fa9a819d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:29.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.055+0000 7fa9a7fff700 1 --2- 172.21.15.169:0/3467873796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9a8101a30 0x7fa9a819d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:29.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.055+0000 7fa9a7fff700 1 -- 172.21.15.169:0/3467873796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9a81070d0 msgr2=0x7fa9a8197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:29.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.055+0000 7fa9a7fff700 1 --2- 172.21.15.169:0/3467873796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9a81070d0 0x7fa9a8197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:29.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.055+0000 7fa9a7fff700 1 -- 172.21.15.169:0/3467873796 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a8197d80 con 0x7fa9a8101090 2024-03-28T00:55:29.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.055+0000 7fa9a7fff700 1 --2- 172.21.15.169:0/3467873796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9a8101090 0x7fa9a819d460 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa998005910 tx=0x7fa99800bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:29.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.055+0000 7fa9a57fa700 1 -- 172.21.15.169:0/3467873796 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998027070 con 0x7fa9a8101090 2024-03-28T00:55:29.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.055+0000 7fa9a57fa700 1 -- 172.21.15.169:0/3467873796 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa998004650 con 0x7fa9a8101090 2024-03-28T00:55:29.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.055+0000 7fa9a57fa700 1 -- 172.21.15.169:0/3467873796 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99800f460 con 0x7fa9a8101090 2024-03-28T00:55:29.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.055+0000 7fa9ae7fe700 1 -- 172.21.15.169:0/3467873796 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a8198010 con 0x7fa9a8101090 2024-03-28T00:55:29.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.056+0000 7fa9ae7fe700 1 -- 172.21.15.169:0/3467873796 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a81a4330 con 0x7fa9a8101090 2024-03-28T00:55:29.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.056+0000 7fa9ae7fe700 1 -- 172.21.15.169:0/3467873796 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa9a8062920 con 0x7fa9a8101090 2024-03-28T00:55:29.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.057+0000 7fa9a57fa700 1 -- 172.21.15.169:0/3467873796 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fa99802a070 con 0x7fa9a8101090 2024-03-28T00:55:29.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.058+0000 7fa9a57fa700 1 --2- 172.21.15.169:0/3467873796 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa9900a2920 0x7fa9900a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:29.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.058+0000 7fa9a77fe700 1 --2- 172.21.15.169:0/3467873796 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa9900a2920 0x7fa9900a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:29.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.058+0000 7fa9a57fa700 1 -- 172.21.15.169:0/3467873796 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa9980d0410 con 0x7fa9a8101090 2024-03-28T00:55:29.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.058+0000 7fa9a57fa700 1 --2- 172.21.15.169:0/3467873796 >> [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252] conn(0x7fa9900ad5e0 0x7fa9900afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:29.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.058+0000 7fa9a57fa700 1 -- 172.21.15.169:0/3467873796 --> [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa998010ec0 con 0x7fa9900ad5e0 2024-03-28T00:55:29.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.058+0000 7fa9a57fa700 1 -- 172.21.15.169:0/3467873796 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa9980d07c0 con 0x7fa9a8101090 2024-03-28T00:55:29.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.058+0000 7fa9acd9b700 1 --2- 172.21.15.169:0/3467873796 >> [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252] conn(0x7fa9900ad5e0 0x7fa9900afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:29.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.059+0000 7fa9a77fe700 1 --2- 172.21.15.169:0/3467873796 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa9900a2920 0x7fa9900a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80785c0 tx=0x7fa99c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:29.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.059+0000 7fa9acd9b700 1 --2- 172.21.15.169:0/3467873796 >> [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252] conn(0x7fa9900ad5e0 0x7fa9900afa20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:29.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.059+0000 7fa9a57fa700 1 -- 172.21.15.169:0/3467873796 <== osd.6 v2:172.21.15.169:6818/783670252 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa998010ec0 con 0x7fa9900ad5e0 2024-03-28T00:55:29.078 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.076+0000 7fa9ae7fe700 1 -- 172.21.15.169:0/3467873796 --> [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa9a8062da0 con 0x7fa9900ad5e0 2024-03-28T00:55:29.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.076+0000 7fa9a57fa700 1 -- 172.21.15.169:0/3467873796 <== osd.6 v2:172.21.15.169:6818/783670252 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa9a8062da0 con 0x7fa9900ad5e0 2024-03-28T00:55:29.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.076+0000 7fa9ae7fe700 1 -- 172.21.15.169:0/3467873796 >> [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252] conn(0x7fa9900ad5e0 msgr2=0x7fa9900afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:29.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.076+0000 7fa9ae7fe700 1 --2- 172.21.15.169:0/3467873796 >> [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252] conn(0x7fa9900ad5e0 0x7fa9900afa20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:29.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.076+0000 7fa9ae7fe700 1 -- 172.21.15.169:0/3467873796 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa9900a2920 msgr2=0x7fa9900a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:29.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.076+0000 7fa9ae7fe700 1 --2- 172.21.15.169:0/3467873796 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa9900a2920 0x7fa9900a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80785c0 tx=0x7fa99c009260 comp rx=0 tx=0).stop 2024-03-28T00:55:29.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.076+0000 7fa9ae7fe700 1 -- 172.21.15.169:0/3467873796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9a8101090 msgr2=0x7fa9a819d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:29.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.076+0000 7fa9ae7fe700 1 --2- 172.21.15.169:0/3467873796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9a8101090 0x7fa9a819d460 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa998005910 tx=0x7fa99800bab0 comp rx=0 tx=0).stop 2024-03-28T00:55:29.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.076+0000 7fa9acd9b700 1 -- 172.21.15.169:0/3467873796 reap_dead start 2024-03-28T00:55:29.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.076+0000 7fa9ae7fe700 1 -- 172.21.15.169:0/3467873796 shutdown_connections 2024-03-28T00:55:29.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.076+0000 7fa9ae7fe700 1 -- 172.21.15.169:0/3467873796 >> 172.21.15.169:0/3467873796 conn(0x7fa9a8077fc0 msgr2=0x7fa9a80ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:29.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.076+0000 7fa9ae7fe700 1 -- 172.21.15.169:0/3467873796 shutdown_connections 2024-03-28T00:55:29.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.077+0000 7fa9ae7fe700 1 -- 172.21.15.169:0/3467873796 wait complete. 2024-03-28T00:55:29.248 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:29 smithi169 ceph-mon[32274]: pgmap v316: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T00:55:29.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.245+0000 7f5835f30700 1 -- 172.21.15.169:0/2222640420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5830101490 msgr2=0x7f583010d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:29.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.245+0000 7f5835f30700 1 --2- 172.21.15.169:0/2222640420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5830101490 0x7f583010d6d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5828009f90 tx=0x7f582800b330 comp rx=0 tx=0).stop 2024-03-28T00:55:29.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.245+0000 7f5835f30700 1 -- 172.21.15.169:0/2222640420 shutdown_connections 2024-03-28T00:55:29.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.245+0000 7f5835f30700 1 --2- 172.21.15.169:0/2222640420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5830101490 0x7f583010d6d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:29.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.245+0000 7f5835f30700 1 --2- 172.21.15.169:0/2222640420 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58301068b0 0x7f5830106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:29.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.245+0000 7f5835f30700 1 --2- 172.21.15.169:0/2222640420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5830100ad0 0x7f5830100f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:29.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.245+0000 7f5835f30700 1 -- 172.21.15.169:0/2222640420 >> 172.21.15.169:0/2222640420 conn(0x7f58300763f0 msgr2=0x7f5830076800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:29.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.245+0000 7f5835f30700 1 -- 172.21.15.169:0/2222640420 shutdown_connections 2024-03-28T00:55:29.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.245+0000 7f5835f30700 1 -- 172.21.15.169:0/2222640420 wait complete. 2024-03-28T00:55:29.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.245+0000 7f5835f30700 1 Processor -- start 2024-03-28T00:55:29.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.246+0000 7f5835f30700 1 -- start start 2024-03-28T00:55:29.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.246+0000 7f5835f30700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5830100ad0 0x7f583019e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:29.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.246+0000 7f5835f30700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5830101490 0x7f583019eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:29.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.246+0000 7f5835f30700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58301068b0 0x7f58301986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:29.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.246+0000 7f5835f30700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58301113a0 con 0x7f5830100ad0 2024-03-28T00:55:29.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.246+0000 7f5835f30700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5830111220 con 0x7f5830101490 2024-03-28T00:55:29.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.246+0000 7f5835f30700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5830111520 con 0x7f58301068b0 2024-03-28T00:55:29.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.246+0000 7f582f7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5830100ad0 0x7f583019e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:29.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.246+0000 7f582ffff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58301068b0 0x7f58301986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:29.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.246+0000 7f582f7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5830100ad0 0x7f583019e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:58654/0 (socket says 172.21.15.169:58654) 2024-03-28T00:55:29.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.246+0000 7f582f7fe700 1 -- 172.21.15.169:0/2885625625 learned_addr learned my addr 172.21.15.169:0/2885625625 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:29.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.246+0000 7f582effd700 1 --2- 172.21.15.169:0/2885625625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5830101490 0x7f583019eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:29.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.246+0000 7f582ffff700 1 -- 172.21.15.169:0/2885625625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5830101490 msgr2=0x7f583019eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:29.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.246+0000 7f582ffff700 1 --2- 172.21.15.169:0/2885625625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5830101490 0x7f583019eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:29.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.246+0000 7f582ffff700 1 -- 172.21.15.169:0/2885625625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5830100ad0 msgr2=0x7f583019e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:29.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.246+0000 7f582ffff700 1 --2- 172.21.15.169:0/2885625625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5830100ad0 0x7f583019e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:29.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.246+0000 7f582ffff700 1 -- 172.21.15.169:0/2885625625 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5830198f90 con 0x7f58301068b0 2024-03-28T00:55:29.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.246+0000 7f582ffff700 1 --2- 172.21.15.169:0/2885625625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58301068b0 0x7f58301986b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5828015000 tx=0x7f5828004f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:29.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.247+0000 7f582cff9700 1 -- 172.21.15.169:0/2885625625 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5828021070 con 0x7f58301068b0 2024-03-28T00:55:29.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.247+0000 7f582cff9700 1 -- 172.21.15.169:0/2885625625 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5828007ac0 con 0x7f58301068b0 2024-03-28T00:55:29.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.247+0000 7f5835f30700 1 -- 172.21.15.169:0/2885625625 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5830199220 con 0x7f58301068b0 2024-03-28T00:55:29.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.247+0000 7f582cff9700 1 -- 172.21.15.169:0/2885625625 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58280052a0 con 0x7f58301068b0 2024-03-28T00:55:29.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.247+0000 7f5835f30700 1 -- 172.21.15.169:0/2885625625 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58301a5560 con 0x7f58301068b0 2024-03-28T00:55:29.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.247+0000 7f5835f30700 1 -- 172.21.15.169:0/2885625625 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f58300628e0 con 0x7f58301068b0 2024-03-28T00:55:29.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.248+0000 7f582cff9700 1 -- 172.21.15.169:0/2885625625 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f5828024070 con 0x7f58301068b0 2024-03-28T00:55:29.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.249+0000 7f582cff9700 1 --2- 172.21.15.169:0/2885625625 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f581c0a2920 0x7f581c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:29.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.249+0000 7f582f7fe700 1 --2- 172.21.15.169:0/2885625625 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f581c0a2920 0x7f581c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:29.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.249+0000 7f582cff9700 1 -- 172.21.15.169:0/2885625625 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f582800f020 con 0x7f58301068b0 2024-03-28T00:55:29.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.249+0000 7f582cff9700 1 --2- 172.21.15.169:0/2885625625 >> [v2:172.21.15.174:6800/3333696361,v1:172.21.15.174:6801/3333696361] conn(0x7f581c0ad5e0 0x7f581c0afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:29.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.249+0000 7f582cff9700 1 -- 172.21.15.169:0/2885625625 --> [v2:172.21.15.174:6800/3333696361,v1:172.21.15.174:6801/3333696361] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5828013e70 con 0x7f581c0ad5e0 2024-03-28T00:55:29.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.249+0000 7f582cff9700 1 -- 172.21.15.169:0/2885625625 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58280d08c0 con 0x7f58301068b0 2024-03-28T00:55:29.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.249+0000 7f582effd700 1 --2- 172.21.15.169:0/2885625625 >> [v2:172.21.15.174:6800/3333696361,v1:172.21.15.174:6801/3333696361] conn(0x7f581c0ad5e0 0x7f581c0afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:29.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.249+0000 7f582f7fe700 1 --2- 172.21.15.169:0/2885625625 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f581c0a2920 0x7f581c0a4de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5820005d10 tx=0x7f5820009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:29.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.250+0000 7f582effd700 1 --2- 172.21.15.169:0/2885625625 >> [v2:172.21.15.174:6800/3333696361,v1:172.21.15.174:6801/3333696361] conn(0x7f581c0ad5e0 0x7f581c0afa20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:29.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.250+0000 7f582cff9700 1 -- 172.21.15.169:0/2885625625 <== osd.1 v2:172.21.15.174:6800/3333696361 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5828013e70 con 0x7f581c0ad5e0 2024-03-28T00:55:29.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.267+0000 7f5835f30700 1 -- 172.21.15.169:0/2885625625 --> [v2:172.21.15.174:6800/3333696361,v1:172.21.15.174:6801/3333696361] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5830062d60 con 0x7f581c0ad5e0 2024-03-28T00:55:29.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.267+0000 7f582cff9700 1 -- 172.21.15.169:0/2885625625 <== osd.1 v2:172.21.15.174:6800/3333696361 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5830062d60 con 0x7f581c0ad5e0 2024-03-28T00:55:29.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.267+0000 7f5835f30700 1 -- 172.21.15.169:0/2885625625 >> [v2:172.21.15.174:6800/3333696361,v1:172.21.15.174:6801/3333696361] conn(0x7f581c0ad5e0 msgr2=0x7f581c0afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:29.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.267+0000 7f5835f30700 1 --2- 172.21.15.169:0/2885625625 >> [v2:172.21.15.174:6800/3333696361,v1:172.21.15.174:6801/3333696361] conn(0x7f581c0ad5e0 0x7f581c0afa20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:29.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.267+0000 7f5835f30700 1 -- 172.21.15.169:0/2885625625 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f581c0a2920 msgr2=0x7f581c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:29.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.267+0000 7f5835f30700 1 --2- 172.21.15.169:0/2885625625 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f581c0a2920 0x7f581c0a4de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5820005d10 tx=0x7f5820009310 comp rx=0 tx=0).stop 2024-03-28T00:55:29.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.267+0000 7f5835f30700 1 -- 172.21.15.169:0/2885625625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58301068b0 msgr2=0x7f58301986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:29.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.267+0000 7f5835f30700 1 --2- 172.21.15.169:0/2885625625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58301068b0 0x7f58301986b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5828015000 tx=0x7f5828004f00 comp rx=0 tx=0).stop 2024-03-28T00:55:29.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.268+0000 7f582ffff700 1 -- 172.21.15.169:0/2885625625 reap_dead start 2024-03-28T00:55:29.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.268+0000 7f5835f30700 1 -- 172.21.15.169:0/2885625625 shutdown_connections 2024-03-28T00:55:29.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.268+0000 7f5835f30700 1 -- 172.21.15.169:0/2885625625 >> 172.21.15.169:0/2885625625 conn(0x7f58300763f0 msgr2=0x7f583010f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:29.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.268+0000 7f5835f30700 1 -- 172.21.15.169:0/2885625625 shutdown_connections 2024-03-28T00:55:29.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:29.268+0000 7f5835f30700 1 -- 172.21.15.169:0/2885625625 wait complete. 2024-03-28T00:55:29.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:29 smithi174 ceph-mon[36060]: pgmap v316: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T00:55:29.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:29 smithi177 ceph-mon[36206]: pgmap v316: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T00:55:30.076 INFO:teuthology.orchestra.run.smithi169.stdout:55834574923 2024-03-28T00:55:30.076 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-28T00:55:30.506 INFO:teuthology.orchestra.run.smithi169.stdout:163208757303 2024-03-28T00:55:30.507 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-28T00:55:31.183 INFO:teuthology.orchestra.run.smithi169.stdout:253403070503 2024-03-28T00:55:31.184 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-28T00:55:31.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:31 smithi169 ceph-mon[32274]: pgmap v317: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T00:55:31.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:31 smithi174 ceph-mon[36060]: pgmap v317: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T00:55:31.627 INFO:teuthology.orchestra.run.smithi169.stdout:197568495665 2024-03-28T00:55:31.628 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-28T00:55:31.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:31 smithi177 ceph-mon[36206]: pgmap v317: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T00:55:32.443 INFO:teuthology.orchestra.run.smithi169.stdout:107374182464 2024-03-28T00:55:32.443 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-28T00:55:32.795 INFO:teuthology.orchestra.run.smithi169.stdout:309237645342 2024-03-28T00:55:32.796 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-28T00:55:33.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.247+0000 7f20485ee700 1 -- 172.21.15.169:0/4185274853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2040101350 msgr2=0x7f2040101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:33.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.247+0000 7f20485ee700 1 --2- 172.21.15.169:0/4185274853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2040101350 0x7f2040101730 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f203c0099c0 tx=0x7f203c009cd0 comp rx=0 tx=0).stop 2024-03-28T00:55:33.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.247+0000 7f20485ee700 1 -- 172.21.15.169:0/4185274853 shutdown_connections 2024-03-28T00:55:33.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.247+0000 7f20485ee700 1 --2- 172.21.15.169:0/4185274853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20401063a0 0x7f204010df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:33.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.247+0000 7f20485ee700 1 --2- 172.21.15.169:0/4185274853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2040101350 0x7f2040101730 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:33.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.247+0000 7f20485ee700 1 --2- 172.21.15.169:0/4185274853 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2040101d00 0x7f2040105cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:33.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.247+0000 7f20485ee700 1 -- 172.21.15.169:0/4185274853 >> 172.21.15.169:0/4185274853 conn(0x7f20400770e0 msgr2=0x7f20400774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:33.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.247+0000 7f20485ee700 1 -- 172.21.15.169:0/4185274853 shutdown_connections 2024-03-28T00:55:33.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.247+0000 7f20485ee700 1 -- 172.21.15.169:0/4185274853 wait complete. 2024-03-28T00:55:33.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.247+0000 7f20485ee700 1 Processor -- start 2024-03-28T00:55:33.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f20485ee700 1 -- start start 2024-03-28T00:55:33.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f20485ee700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2040101350 0x7f204019d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:33.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f20485ee700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2040101d00 0x7f204019dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:33.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f20485ee700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20401063a0 0x7f2040197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:33.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f20485ee700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20401113b0 con 0x7f20401063a0 2024-03-28T00:55:33.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f20485ee700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2040111230 con 0x7f2040101d00 2024-03-28T00:55:33.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f20485ee700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2040111530 con 0x7f2040101350 2024-03-28T00:55:33.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f2046b8b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20401063a0 0x7f2040197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:33.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f2046b8b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20401063a0 0x7f2040197670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:58678/0 (socket says 172.21.15.169:58678) 2024-03-28T00:55:33.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f2046b8b700 1 -- 172.21.15.169:0/3144295704 learned_addr learned my addr 172.21.15.169:0/3144295704 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:33.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f204638a700 1 --2- 172.21.15.169:0/3144295704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2040101350 0x7f204019d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:33.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f2045b89700 1 --2- 172.21.15.169:0/3144295704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2040101d00 0x7f204019dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:33.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f2046b8b700 1 -- 172.21.15.169:0/3144295704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2040101350 msgr2=0x7f204019d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:33.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f2046b8b700 1 --2- 172.21.15.169:0/3144295704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2040101350 0x7f204019d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:33.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f2046b8b700 1 -- 172.21.15.169:0/3144295704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2040101d00 msgr2=0x7f204019dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:33.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f2046b8b700 1 --2- 172.21.15.169:0/3144295704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2040101d00 0x7f204019dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:33.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f2046b8b700 1 -- 172.21.15.169:0/3144295704 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2040197f50 con 0x7f20401063a0 2024-03-28T00:55:33.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f204638a700 1 --2- 172.21.15.169:0/3144295704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2040101350 0x7f204019d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:55:33.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f2046b8b700 1 --2- 172.21.15.169:0/3144295704 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20401063a0 0x7f2040197670 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f203800ef30 tx=0x7f203800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:33.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f20377fe700 1 -- 172.21.15.169:0/3144295704 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20380092e0 con 0x7f20401063a0 2024-03-28T00:55:33.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f20485ee700 1 -- 172.21.15.169:0/3144295704 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20401981e0 con 0x7f20401063a0 2024-03-28T00:55:33.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f20377fe700 1 -- 172.21.15.169:0/3144295704 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2038004bb0 con 0x7f20401063a0 2024-03-28T00:55:33.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.248+0000 7f20377fe700 1 -- 172.21.15.169:0/3144295704 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2038004280 con 0x7f20401063a0 2024-03-28T00:55:33.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.249+0000 7f20485ee700 1 -- 172.21.15.169:0/3144295704 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20401a4520 con 0x7f20401063a0 2024-03-28T00:55:33.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.249+0000 7f20485ee700 1 -- 172.21.15.169:0/3144295704 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f20400628a0 con 0x7f20401063a0 2024-03-28T00:55:33.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.251+0000 7f20377fe700 1 -- 172.21.15.169:0/3144295704 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f203801b030 con 0x7f20401063a0 2024-03-28T00:55:33.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.251+0000 7f20377fe700 1 --2- 172.21.15.169:0/3144295704 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f202c0a2920 0x7f202c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:33.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.251+0000 7f204638a700 1 --2- 172.21.15.169:0/3144295704 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f202c0a2920 0x7f202c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:33.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.251+0000 7f20377fe700 1 -- 172.21.15.169:0/3144295704 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f20380cf1e0 con 0x7f20401063a0 2024-03-28T00:55:33.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.251+0000 7f20377fe700 1 --2- 172.21.15.169:0/3144295704 >> [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290] conn(0x7f202c0ad5e0 0x7f202c0afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:33.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.251+0000 7f20377fe700 1 -- 172.21.15.169:0/3144295704 --> [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f203801fe50 con 0x7f202c0ad5e0 2024-03-28T00:55:33.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.251+0000 7f20377fe700 1 -- 172.21.15.169:0/3144295704 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20380cf630 con 0x7f20401063a0 2024-03-28T00:55:33.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.252+0000 7f2045b89700 1 --2- 172.21.15.169:0/3144295704 >> [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290] conn(0x7f202c0ad5e0 0x7f202c0afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:33.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.252+0000 7f204638a700 1 --2- 172.21.15.169:0/3144295704 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f202c0a2920 0x7f202c0a4de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f203c000c00 tx=0x7f203c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:33.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.252+0000 7f2045b89700 1 --2- 172.21.15.169:0/3144295704 >> [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290] conn(0x7f202c0ad5e0 0x7f202c0afa20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:33.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.253+0000 7f20377fe700 1 -- 172.21.15.169:0/3144295704 <== osd.9 v2:172.21.15.169:6826/3660250290 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f203801fe50 con 0x7f202c0ad5e0 2024-03-28T00:55:33.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.269+0000 7f20485ee700 1 -- 172.21.15.169:0/3144295704 --> [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2040062cd0 con 0x7f202c0ad5e0 2024-03-28T00:55:33.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.270+0000 7f20377fe700 1 -- 172.21.15.169:0/3144295704 <== osd.9 v2:172.21.15.169:6826/3660250290 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2040062cd0 con 0x7f202c0ad5e0 2024-03-28T00:55:33.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.270+0000 7f20485ee700 1 -- 172.21.15.169:0/3144295704 >> [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290] conn(0x7f202c0ad5e0 msgr2=0x7f202c0afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:33.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.270+0000 7f20485ee700 1 --2- 172.21.15.169:0/3144295704 >> [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290] conn(0x7f202c0ad5e0 0x7f202c0afa20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:33.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.270+0000 7f20485ee700 1 -- 172.21.15.169:0/3144295704 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f202c0a2920 msgr2=0x7f202c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:33.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.270+0000 7f20485ee700 1 --2- 172.21.15.169:0/3144295704 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f202c0a2920 0x7f202c0a4de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f203c000c00 tx=0x7f203c005fb0 comp rx=0 tx=0).stop 2024-03-28T00:55:33.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.270+0000 7f20485ee700 1 -- 172.21.15.169:0/3144295704 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20401063a0 msgr2=0x7f2040197670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:33.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.270+0000 7f20485ee700 1 --2- 172.21.15.169:0/3144295704 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20401063a0 0x7f2040197670 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f203800ef30 tx=0x7f203800c3a0 comp rx=0 tx=0).stop 2024-03-28T00:55:33.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.270+0000 7f2046b8b700 1 -- 172.21.15.169:0/3144295704 reap_dead start 2024-03-28T00:55:33.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.270+0000 7f20485ee700 1 -- 172.21.15.169:0/3144295704 shutdown_connections 2024-03-28T00:55:33.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.270+0000 7f20485ee700 1 -- 172.21.15.169:0/3144295704 >> 172.21.15.169:0/3144295704 conn(0x7f20400770e0 msgr2=0x7f20400ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:33.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.270+0000 7f20485ee700 1 -- 172.21.15.169:0/3144295704 shutdown_connections 2024-03-28T00:55:33.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:33.270+0000 7f20485ee700 1 -- 172.21.15.169:0/3144295704 wait complete. 2024-03-28T00:55:33.459 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:33 smithi169 ceph-mon[32274]: pgmap v318: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T00:55:33.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:33 smithi174 ceph-mon[36060]: pgmap v318: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T00:55:33.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:33 smithi177 ceph-mon[36206]: pgmap v318: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T00:55:33.822 INFO:teuthology.orchestra.run.smithi169.stdout:34359738446 2024-03-28T00:55:33.823 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-28T00:55:34.257 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:34.261 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:34.276 INFO:teuthology.orchestra.run.smithi169.stdout:287762808867 2024-03-28T00:55:34.277 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-28T00:55:34.283 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:34.295 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:34.314 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:34.517 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:34 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:55:34.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:34 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:55:34.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:34 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:55:35.439 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:35 smithi169 ceph-mon[32274]: pgmap v319: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T00:55:35.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:35 smithi174 ceph-mon[36060]: pgmap v319: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T00:55:35.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:35 smithi177 ceph-mon[36206]: pgmap v319: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T00:55:35.859 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:36.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:55:36.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:55:36.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:55:36.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:55:36.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-28T00:55:36.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-03-28T00:55:36.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2024-03-28T00:55:36.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-03-28T00:55:36.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-28T00:55:36.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-03-28T00:55:36.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-03-28T00:55:36.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2024-03-28T00:55:36.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-03-28T00:55:36.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:36 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-03-28T00:55:36.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:36 smithi169 ceph-mon[32274]: pgmap v320: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T00:55:36.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:55:36.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:55:36.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:55:36.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:55:36.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-28T00:55:36.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-03-28T00:55:36.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2024-03-28T00:55:36.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-03-28T00:55:36.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-28T00:55:36.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-03-28T00:55:36.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-03-28T00:55:36.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2024-03-28T00:55:36.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-03-28T00:55:36.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:36 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-03-28T00:55:36.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:36 smithi174 ceph-mon[36060]: pgmap v320: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T00:55:36.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:55:36.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:55:36.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:55:36.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:55:36.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-28T00:55:36.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-03-28T00:55:36.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2024-03-28T00:55:36.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-03-28T00:55:36.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-28T00:55:36.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-03-28T00:55:36.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-03-28T00:55:36.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2024-03-28T00:55:36.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-03-28T00:55:36.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:36 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-03-28T00:55:36.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:36 smithi177 ceph-mon[36206]: pgmap v320: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T00:55:37.853 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-28T00:55:37.853 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-03-28T00:55:37.853 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2024-03-28T00:55:37.854 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-03-28T00:55:37.854 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-28T00:55:37.854 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-03-28T00:55:37.854 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-03-28T00:55:37.854 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2024-03-28T00:55:37.854 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-03-28T00:55:37.854 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-03-28T00:55:37.854 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:37 smithi169 ceph-mon[32274]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T00:55:37.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:37 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-28T00:55:37.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:37 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-03-28T00:55:37.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:37 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2024-03-28T00:55:37.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:37 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-03-28T00:55:37.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:37 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-28T00:55:37.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:37 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-03-28T00:55:37.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:37 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-03-28T00:55:37.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:37 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2024-03-28T00:55:37.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:37 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-03-28T00:55:37.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:37 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-03-28T00:55:37.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:37 smithi174 ceph-mon[36060]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T00:55:37.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:37 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-28T00:55:37.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:37 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-03-28T00:55:37.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:37 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2024-03-28T00:55:37.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:37 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-03-28T00:55:37.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:37 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-28T00:55:37.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:37 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-03-28T00:55:37.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:37 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-03-28T00:55:37.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:37 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2024-03-28T00:55:37.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:37 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-03-28T00:55:37.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:37 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-03-28T00:55:37.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:37 smithi177 ceph-mon[36206]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T00:55:38.420 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:38.430 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:38.430 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:38.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:38 smithi169 ceph-mon[32274]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T00:55:38.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:38 smithi169 ceph-mon[32274]: pgmap v323: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-28T00:55:38.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:38 smithi174 ceph-mon[36060]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T00:55:38.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:38 smithi174 ceph-mon[36060]: pgmap v323: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-28T00:55:38.877 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:38.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:38 smithi177 ceph-mon[36206]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T00:55:38.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:38 smithi177 ceph-mon[36206]: pgmap v323: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-28T00:55:40.608 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:41.234 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:41 smithi169 ceph-mon[32274]: pgmap v324: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T00:55:41.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:41 smithi177 ceph-mon[36206]: pgmap v324: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T00:55:41.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:41 smithi174 ceph-mon[36060]: pgmap v324: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T00:55:42.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:42 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1038860474' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T00:55:42.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:42 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1038860474' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T00:55:42.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.235+0000 7fd4d27d5700 1 -- 172.21.15.169:0/3662377439 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4cc106370 msgr2=0x7fd4cc10df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:42.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.235+0000 7fd4d27d5700 1 --2- 172.21.15.169:0/3662377439 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4cc106370 0x7fd4cc10df40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd4c400b0d0 tx=0x7fd4c400b3e0 comp rx=0 tx=0).stop 2024-03-28T00:55:42.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.235+0000 7fd4d27d5700 1 -- 172.21.15.169:0/3662377439 shutdown_connections 2024-03-28T00:55:42.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.235+0000 7fd4d27d5700 1 --2- 172.21.15.169:0/3662377439 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4cc106370 0x7fd4cc10df40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:42.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.235+0000 7fd4d27d5700 1 --2- 172.21.15.169:0/3662377439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4cc101320 0x7fd4cc101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:42.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.235+0000 7fd4d27d5700 1 --2- 172.21.15.169:0/3662377439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4cc101cd0 0x7fd4cc105c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:42.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.235+0000 7fd4d27d5700 1 -- 172.21.15.169:0/3662377439 >> 172.21.15.169:0/3662377439 conn(0x7fd4cc0770e0 msgr2=0x7fd4cc0774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:42.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.235+0000 7fd4d27d5700 1 -- 172.21.15.169:0/3662377439 shutdown_connections 2024-03-28T00:55:42.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.235+0000 7fd4d27d5700 1 -- 172.21.15.169:0/3662377439 wait complete. 2024-03-28T00:55:42.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.236+0000 7fd4d27d5700 1 Processor -- start 2024-03-28T00:55:42.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.236+0000 7fd4d27d5700 1 -- start start 2024-03-28T00:55:42.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.236+0000 7fd4d27d5700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4cc101320 0x7fd4cc19e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:42.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.236+0000 7fd4d27d5700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4cc101cd0 0x7fd4cc19eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:42.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.236+0000 7fd4d27d5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4cc106370 0x7fd4cc198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:42.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.236+0000 7fd4d27d5700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc111360 con 0x7fd4cc106370 2024-03-28T00:55:42.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.236+0000 7fd4d27d5700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc1111e0 con 0x7fd4cc101cd0 2024-03-28T00:55:42.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.236+0000 7fd4d27d5700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc1114e0 con 0x7fd4cc101320 2024-03-28T00:55:42.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.236+0000 7fd4d0d72700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4cc106370 0x7fd4cc198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:42.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.236+0000 7fd4d0d72700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4cc106370 0x7fd4cc198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:59686/0 (socket says 172.21.15.169:59686) 2024-03-28T00:55:42.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.236+0000 7fd4d0d72700 1 -- 172.21.15.169:0/1038860474 learned_addr learned my addr 172.21.15.169:0/1038860474 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:42.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.236+0000 7fd4cb7fe700 1 --2- 172.21.15.169:0/1038860474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4cc101cd0 0x7fd4cc19eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:42.464 INFO:teuthology.orchestra.run.smithi169.stdout:219043332143 2024-03-28T00:55:42.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.236+0000 7fd4cbfff700 1 --2- 172.21.15.169:0/1038860474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4cc101320 0x7fd4cc19e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:42.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.236+0000 7fd4d0d72700 1 -- 172.21.15.169:0/1038860474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4cc101320 msgr2=0x7fd4cc19e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:42.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.236+0000 7fd4d0d72700 1 --2- 172.21.15.169:0/1038860474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4cc101320 0x7fd4cc19e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:42.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.236+0000 7fd4d0d72700 1 -- 172.21.15.169:0/1038860474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4cc101cd0 msgr2=0x7fd4cc19eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:42.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.237+0000 7fd4d0d72700 1 --2- 172.21.15.169:0/1038860474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4cc101cd0 0x7fd4cc19eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:42.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.237+0000 7fd4d0d72700 1 -- 172.21.15.169:0/1038860474 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4cc199020 con 0x7fd4cc106370 2024-03-28T00:55:42.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.237+0000 7fd4cb7fe700 1 --2- 172.21.15.169:0/1038860474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4cc101cd0 0x7fd4cc19eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:55:42.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.237+0000 7fd4d0d72700 1 --2- 172.21.15.169:0/1038860474 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4cc106370 0x7fd4cc198740 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4006070 tx=0x7fd4c4007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:42.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.237+0000 7fd4c97fa700 1 -- 172.21.15.169:0/1038860474 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c4003c00 con 0x7fd4cc106370 2024-03-28T00:55:42.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.237+0000 7fd4d27d5700 1 -- 172.21.15.169:0/1038860474 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4cc1992b0 con 0x7fd4cc106370 2024-03-28T00:55:42.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.237+0000 7fd4c97fa700 1 -- 172.21.15.169:0/1038860474 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd4c4022070 con 0x7fd4cc106370 2024-03-28T00:55:42.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.237+0000 7fd4c97fa700 1 -- 172.21.15.169:0/1038860474 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c401b6c0 con 0x7fd4cc106370 2024-03-28T00:55:42.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.237+0000 7fd4d27d5700 1 -- 172.21.15.169:0/1038860474 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4cc1a55a0 con 0x7fd4cc106370 2024-03-28T00:55:42.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.238+0000 7fd4d27d5700 1 -- 172.21.15.169:0/1038860474 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4cc199610 con 0x7fd4cc106370 2024-03-28T00:55:42.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.239+0000 7fd4c97fa700 1 -- 172.21.15.169:0/1038860474 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fd4c401d470 con 0x7fd4cc106370 2024-03-28T00:55:42.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.240+0000 7fd4c97fa700 1 --2- 172.21.15.169:0/1038860474 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd4b40a2920 0x7fd4b40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:42.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.240+0000 7fd4cbfff700 1 --2- 172.21.15.169:0/1038860474 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd4b40a2920 0x7fd4b40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:42.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.240+0000 7fd4c97fa700 1 -- 172.21.15.169:0/1038860474 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fd4c400f020 con 0x7fd4cc106370 2024-03-28T00:55:42.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.240+0000 7fd4cbfff700 1 --2- 172.21.15.169:0/1038860474 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd4b40a2920 0x7fd4b40a4de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc009680 tx=0x7fd4bc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:42.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.244+0000 7fd4c97fa700 1 -- 172.21.15.169:0/1038860474 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd4c409c8c0 con 0x7fd4cc106370 2024-03-28T00:55:42.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.447+0000 7fd4d27d5700 1 -- 172.21.15.169:0/1038860474 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fd4cc199c20 con 0x7fd4cc106370 2024-03-28T00:55:42.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.447+0000 7fd4c97fa700 1 -- 172.21.15.169:0/1038860474 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd4c409c1c0 con 0x7fd4cc106370 2024-03-28T00:55:42.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.449+0000 7fd4d27d5700 1 -- 172.21.15.169:0/1038860474 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd4b40a2920 msgr2=0x7fd4b40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:42.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.449+0000 7fd4d27d5700 1 --2- 172.21.15.169:0/1038860474 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd4b40a2920 0x7fd4b40a4de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc009680 tx=0x7fd4bc009260 comp rx=0 tx=0).stop 2024-03-28T00:55:42.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.449+0000 7fd4d27d5700 1 -- 172.21.15.169:0/1038860474 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4cc106370 msgr2=0x7fd4cc198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:42.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.449+0000 7fd4d27d5700 1 --2- 172.21.15.169:0/1038860474 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4cc106370 0x7fd4cc198740 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4006070 tx=0x7fd4c4007a20 comp rx=0 tx=0).stop 2024-03-28T00:55:42.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.450+0000 7fd4d27d5700 1 -- 172.21.15.169:0/1038860474 shutdown_connections 2024-03-28T00:55:42.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.450+0000 7fd4d27d5700 1 --2- 172.21.15.169:0/1038860474 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4cc106370 0x7fd4cc198740 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:42.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.450+0000 7fd4d27d5700 1 --2- 172.21.15.169:0/1038860474 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd4b40a2920 0x7fd4b40a4de0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:42.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.450+0000 7fd4d27d5700 1 --2- 172.21.15.169:0/1038860474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4cc101320 0x7fd4cc19e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:42.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.450+0000 7fd4d27d5700 1 --2- 172.21.15.169:0/1038860474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4cc101cd0 0x7fd4cc19eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:42.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.450+0000 7fd4d27d5700 1 -- 172.21.15.169:0/1038860474 >> 172.21.15.169:0/1038860474 conn(0x7fd4cc0770e0 msgr2=0x7fd4cc06a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:42.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.450+0000 7fd4d27d5700 1 -- 172.21.15.169:0/1038860474 shutdown_connections 2024-03-28T00:55:42.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:41.450+0000 7fd4d27d5700 1 -- 172.21.15.169:0/1038860474 wait complete. 2024-03-28T00:55:42.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:42 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1038860474' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T00:55:43.019 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332139 got 219043332143 for osd.7 2024-03-28T00:55:43.020 DEBUG:teuthology.parallel:result is None 2024-03-28T00:55:43.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.177+0000 7fed257e5700 1 -- 172.21.15.169:0/3856096103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed201074b0 msgr2=0x7fed2010f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:43.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.177+0000 7fed257e5700 1 --2- 172.21.15.169:0/3856096103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed201074b0 0x7fed2010f080 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fed1800b0d0 tx=0x7fed1800b3e0 comp rx=0 tx=0).stop 2024-03-28T00:55:43.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.177+0000 7fed257e5700 1 -- 172.21.15.169:0/3856096103 shutdown_connections 2024-03-28T00:55:43.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.177+0000 7fed257e5700 1 --2- 172.21.15.169:0/3856096103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed201074b0 0x7fed2010f080 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:43.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.177+0000 7fed257e5700 1 --2- 172.21.15.169:0/3856096103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed201023d0 0x7fed201027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:43.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.177+0000 7fed257e5700 1 --2- 172.21.15.169:0/3856096103 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed20102d80 0x7fed20106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:43.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.177+0000 7fed257e5700 1 -- 172.21.15.169:0/3856096103 >> 172.21.15.169:0/3856096103 conn(0x7fed200fdc40 msgr2=0x7fed20100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:43.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.177+0000 7fed257e5700 1 -- 172.21.15.169:0/3856096103 shutdown_connections 2024-03-28T00:55:43.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.177+0000 7fed257e5700 1 -- 172.21.15.169:0/3856096103 wait complete. 2024-03-28T00:55:43.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.177+0000 7fed257e5700 1 Processor -- start 2024-03-28T00:55:43.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.178+0000 7fed257e5700 1 -- start start 2024-03-28T00:55:43.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.178+0000 7fed257e5700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed201023d0 0x7fed2019e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:43.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.178+0000 7fed257e5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed20102d80 0x7fed2019eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:43.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.178+0000 7fed257e5700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed201074b0 0x7fed20198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:43.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.178+0000 7fed257e5700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed201124c0 con 0x7fed20102d80 2024-03-28T00:55:43.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.178+0000 7fed257e5700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed20112340 con 0x7fed201074b0 2024-03-28T00:55:43.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.178+0000 7fed257e5700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed20112640 con 0x7fed201023d0 2024-03-28T00:55:43.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.178+0000 7fed167fc700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed20102d80 0x7fed2019eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:43.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.178+0000 7fed1effd700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed201023d0 0x7fed2019e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:43.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.178+0000 7fed167fc700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed20102d80 0x7fed2019eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:59702/0 (socket says 172.21.15.169:59702) 2024-03-28T00:55:43.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.178+0000 7fed167fc700 1 -- 172.21.15.169:0/1071802014 learned_addr learned my addr 172.21.15.169:0/1071802014 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:43.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.178+0000 7fed1f7fe700 1 --2- 172.21.15.169:0/1071802014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed201074b0 0x7fed20198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:43.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.178+0000 7fed1f7fe700 1 -- 172.21.15.169:0/1071802014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed201023d0 msgr2=0x7fed2019e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:43.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.178+0000 7fed1f7fe700 1 --2- 172.21.15.169:0/1071802014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed201023d0 0x7fed2019e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:43.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.178+0000 7fed1f7fe700 1 -- 172.21.15.169:0/1071802014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed20102d80 msgr2=0x7fed2019eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:43.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.178+0000 7fed1f7fe700 1 --2- 172.21.15.169:0/1071802014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed20102d80 0x7fed2019eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:43.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.178+0000 7fed1f7fe700 1 -- 172.21.15.169:0/1071802014 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed20199000 con 0x7fed201074b0 2024-03-28T00:55:43.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.178+0000 7fed167fc700 1 --2- 172.21.15.169:0/1071802014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed20102d80 0x7fed2019eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:55:43.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.179+0000 7fed1f7fe700 1 --2- 172.21.15.169:0/1071802014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed201074b0 0x7fed20198720 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fed1800b770 tx=0x7fed18007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:43.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.179+0000 7fed1cff9700 1 -- 172.21.15.169:0/1071802014 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed18021020 con 0x7fed201074b0 2024-03-28T00:55:43.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.179+0000 7fed1cff9700 1 -- 172.21.15.169:0/1071802014 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed18004c50 con 0x7fed201074b0 2024-03-28T00:55:43.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.179+0000 7fed257e5700 1 -- 172.21.15.169:0/1071802014 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed20199290 con 0x7fed201074b0 2024-03-28T00:55:43.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.179+0000 7fed1cff9700 1 -- 172.21.15.169:0/1071802014 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed180052b0 con 0x7fed201074b0 2024-03-28T00:55:43.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.179+0000 7fed257e5700 1 -- 172.21.15.169:0/1071802014 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed201a55d0 con 0x7fed201074b0 2024-03-28T00:55:43.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.180+0000 7fed257e5700 1 -- 172.21.15.169:0/1071802014 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed201038b0 con 0x7fed201074b0 2024-03-28T00:55:43.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.181+0000 7fed1cff9700 1 -- 172.21.15.169:0/1071802014 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fed18024020 con 0x7fed201074b0 2024-03-28T00:55:43.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.181+0000 7fed1cff9700 1 --2- 172.21.15.169:0/1071802014 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed0c0a29a0 0x7fed0c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:43.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.181+0000 7fed1effd700 1 --2- 172.21.15.169:0/1071802014 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed0c0a29a0 0x7fed0c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:43.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.181+0000 7fed1cff9700 1 -- 172.21.15.169:0/1071802014 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fed1800f020 con 0x7fed201074b0 2024-03-28T00:55:43.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.182+0000 7fed1effd700 1 --2- 172.21.15.169:0/1071802014 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed0c0a29a0 0x7fed0c0a4e60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fed10009c60 tx=0x7fed10009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:43.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.186+0000 7fed1cff9700 1 -- 172.21.15.169:0/1071802014 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fed1809ca10 con 0x7fed201074b0 2024-03-28T00:55:43.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.386+0000 7fed257e5700 1 -- 172.21.15.169:0/1071802014 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fed20199c50 con 0x7fed201074b0 2024-03-28T00:55:43.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.386+0000 7fed1cff9700 1 -- 172.21.15.169:0/1071802014 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fed1801de20 con 0x7fed201074b0 2024-03-28T00:55:43.390 INFO:teuthology.orchestra.run.smithi169.stdout:197568495668 2024-03-28T00:55:43.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.388+0000 7fed257e5700 1 -- 172.21.15.169:0/1071802014 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed0c0a29a0 msgr2=0x7fed0c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:43.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.388+0000 7fed257e5700 1 --2- 172.21.15.169:0/1071802014 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed0c0a29a0 0x7fed0c0a4e60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fed10009c60 tx=0x7fed10009260 comp rx=0 tx=0).stop 2024-03-28T00:55:43.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.388+0000 7fed257e5700 1 -- 172.21.15.169:0/1071802014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed201074b0 msgr2=0x7fed20198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:43.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.388+0000 7fed257e5700 1 --2- 172.21.15.169:0/1071802014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed201074b0 0x7fed20198720 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fed1800b770 tx=0x7fed18007a20 comp rx=0 tx=0).stop 2024-03-28T00:55:43.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.389+0000 7fed257e5700 1 -- 172.21.15.169:0/1071802014 shutdown_connections 2024-03-28T00:55:43.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.389+0000 7fed257e5700 1 --2- 172.21.15.169:0/1071802014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed201074b0 0x7fed20198720 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:43.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.389+0000 7fed257e5700 1 --2- 172.21.15.169:0/1071802014 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fed0c0a29a0 0x7fed0c0a4e60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:43.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.389+0000 7fed257e5700 1 --2- 172.21.15.169:0/1071802014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed201023d0 0x7fed2019e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:43.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.389+0000 7fed257e5700 1 --2- 172.21.15.169:0/1071802014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed20102d80 0x7fed2019eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:43.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.389+0000 7fed257e5700 1 -- 172.21.15.169:0/1071802014 >> 172.21.15.169:0/1071802014 conn(0x7fed200fdc40 msgr2=0x7fed200fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:43.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.389+0000 7fed257e5700 1 -- 172.21.15.169:0/1071802014 shutdown_connections 2024-03-28T00:55:43.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.389+0000 7fed257e5700 1 -- 172.21.15.169:0/1071802014 wait complete. 2024-03-28T00:55:43.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:43 smithi177 ceph-mon[36206]: pgmap v325: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T00:55:43.496 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:43 smithi169 ceph-mon[32274]: pgmap v325: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T00:55:43.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.501+0000 7f96dff86700 1 -- 172.21.15.169:0/3800015462 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96d8106370 msgr2=0x7f96d810df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:43.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.501+0000 7f96dff86700 1 --2- 172.21.15.169:0/3800015462 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96d8106370 0x7f96d810df40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f96d40099c0 tx=0x7f96d4009cd0 comp rx=0 tx=0).stop 2024-03-28T00:55:43.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.501+0000 7f96dff86700 1 -- 172.21.15.169:0/3800015462 shutdown_connections 2024-03-28T00:55:43.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.501+0000 7f96dff86700 1 --2- 172.21.15.169:0/3800015462 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96d8106370 0x7f96d810df40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:43.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.501+0000 7f96dff86700 1 --2- 172.21.15.169:0/3800015462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96d8101320 0x7f96d8101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:43.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.501+0000 7f96dff86700 1 --2- 172.21.15.169:0/3800015462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96d8101cd0 0x7f96d8105c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:43.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.501+0000 7f96dff86700 1 -- 172.21.15.169:0/3800015462 >> 172.21.15.169:0/3800015462 conn(0x7f96d80770e0 msgr2=0x7f96d80774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:43.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.501+0000 7f96dff86700 1 -- 172.21.15.169:0/3800015462 shutdown_connections 2024-03-28T00:55:43.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.501+0000 7f96dff86700 1 -- 172.21.15.169:0/3800015462 wait complete. 2024-03-28T00:55:43.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.502+0000 7f96dff86700 1 Processor -- start 2024-03-28T00:55:43.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.502+0000 7f96dff86700 1 -- start start 2024-03-28T00:55:43.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.502+0000 7f96dff86700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96d8101320 0x7f96d8199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:43.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.502+0000 7f96dff86700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96d8101cd0 0x7f96d8199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:43.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.502+0000 7f96dff86700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96d8106370 0x7f96d81a47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:43.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.502+0000 7f96dff86700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d8111380 con 0x7f96d8106370 2024-03-28T00:55:43.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.502+0000 7f96dff86700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d8111200 con 0x7f96d8101cd0 2024-03-28T00:55:43.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.502+0000 7f96dff86700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d8111500 con 0x7f96d8101320 2024-03-28T00:55:43.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.502+0000 7f96dd521700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96d8101cd0 0x7f96d8199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:43.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.502+0000 7f96ddd22700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96d8101320 0x7f96d8199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:43.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.502+0000 7f96dd521700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96d8101cd0 0x7f96d8199ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49516/0 (socket says 172.21.15.169:49516) 2024-03-28T00:55:43.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.502+0000 7f96dd521700 1 -- 172.21.15.169:0/1000245201 learned_addr learned my addr 172.21.15.169:0/1000245201 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:43.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.502+0000 7f96ddd22700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96d8101320 0x7f96d8199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:33016/0 (socket says 172.21.15.169:33016) 2024-03-28T00:55:43.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.502+0000 7f96de523700 1 --2- 172.21.15.169:0/1000245201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96d8106370 0x7f96d81a47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:43.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.502+0000 7f96dd521700 1 -- 172.21.15.169:0/1000245201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96d8101320 msgr2=0x7f96d8199760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:43.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.502+0000 7f96dd521700 1 --2- 172.21.15.169:0/1000245201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96d8101320 0x7f96d8199760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:43.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.502+0000 7f96dd521700 1 -- 172.21.15.169:0/1000245201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96d8106370 msgr2=0x7f96d81a47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:43.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.502+0000 7f96dd521700 1 --2- 172.21.15.169:0/1000245201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96d8106370 0x7f96d81a47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:43.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.502+0000 7f96dd521700 1 -- 172.21.15.169:0/1000245201 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d81a4d20 con 0x7f96d8101cd0 2024-03-28T00:55:43.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.502+0000 7f96ddd22700 1 --2- 172.21.15.169:0/1000245201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96d8101320 0x7f96d8199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:55:43.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.503+0000 7f96dd521700 1 --2- 172.21.15.169:0/1000245201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96d8101cd0 0x7f96d8199ca0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f96c800e9f0 tx=0x7f96c800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:43.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.503+0000 7f96ceffd700 1 -- 172.21.15.169:0/1000245201 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c8004a30 con 0x7f96d8101cd0 2024-03-28T00:55:43.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.503+0000 7f96ceffd700 1 -- 172.21.15.169:0/1000245201 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96c800cca0 con 0x7f96d8101cd0 2024-03-28T00:55:43.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.503+0000 7f96dff86700 1 -- 172.21.15.169:0/1000245201 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96d81a4ef0 con 0x7f96d8101cd0 2024-03-28T00:55:43.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.503+0000 7f96ceffd700 1 -- 172.21.15.169:0/1000245201 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c80056c0 con 0x7f96d8101cd0 2024-03-28T00:55:43.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.503+0000 7f96dff86700 1 -- 172.21.15.169:0/1000245201 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96d81a53a0 con 0x7f96d8101cd0 2024-03-28T00:55:43.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.504+0000 7f96dff86700 1 -- 172.21.15.169:0/1000245201 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96d8068e50 con 0x7f96d8101cd0 2024-03-28T00:55:43.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.505+0000 7f96ceffd700 1 -- 172.21.15.169:0/1000245201 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f96c800c7c0 con 0x7f96d8101cd0 2024-03-28T00:55:43.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.506+0000 7f96ceffd700 1 --2- 172.21.15.169:0/1000245201 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f96c40a29a0 0x7f96c40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:43.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.506+0000 7f96ceffd700 1 -- 172.21.15.169:0/1000245201 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f96c80cf040 con 0x7f96d8101cd0 2024-03-28T00:55:43.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.506+0000 7f96ddd22700 1 --2- 172.21.15.169:0/1000245201 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f96c40a29a0 0x7f96c40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:43.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.507+0000 7f96ddd22700 1 --2- 172.21.15.169:0/1000245201 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f96c40a29a0 0x7f96c40a4e60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f96d0009480 tx=0x7f96d000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:43.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.510+0000 7f96ceffd700 1 -- 172.21.15.169:0/1000245201 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f96c809bc20 con 0x7f96d8101cd0 2024-03-28T00:55:43.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:43 smithi174 ceph-mon[36060]: pgmap v325: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T00:55:43.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.712+0000 7f96dff86700 1 -- 172.21.15.169:0/1000245201 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f96d802da30 con 0x7f96d8101cd0 2024-03-28T00:55:43.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.713+0000 7f96ceffd700 1 -- 172.21.15.169:0/1000245201 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f96c809b520 con 0x7f96d8101cd0 2024-03-28T00:55:43.716 INFO:teuthology.orchestra.run.smithi169.stdout:55834574925 2024-03-28T00:55:43.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.715+0000 7f96dff86700 1 -- 172.21.15.169:0/1000245201 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f96c40a29a0 msgr2=0x7f96c40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:43.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.715+0000 7f96dff86700 1 --2- 172.21.15.169:0/1000245201 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f96c40a29a0 0x7f96c40a4e60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f96d0009480 tx=0x7f96d000b040 comp rx=0 tx=0).stop 2024-03-28T00:55:43.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.715+0000 7f96dff86700 1 -- 172.21.15.169:0/1000245201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96d8101cd0 msgr2=0x7f96d8199ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:43.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.715+0000 7f96dff86700 1 --2- 172.21.15.169:0/1000245201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96d8101cd0 0x7f96d8199ca0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f96c800e9f0 tx=0x7f96c800ed00 comp rx=0 tx=0).stop 2024-03-28T00:55:43.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.715+0000 7f96dff86700 1 -- 172.21.15.169:0/1000245201 shutdown_connections 2024-03-28T00:55:43.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.715+0000 7f96dff86700 1 --2- 172.21.15.169:0/1000245201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96d8106370 0x7f96d81a47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:43.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.715+0000 7f96dff86700 1 --2- 172.21.15.169:0/1000245201 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f96c40a29a0 0x7f96c40a4e60 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:43.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.715+0000 7f96dff86700 1 --2- 172.21.15.169:0/1000245201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96d8101320 0x7f96d8199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:43.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.715+0000 7f96dff86700 1 --2- 172.21.15.169:0/1000245201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96d8101cd0 0x7f96d8199ca0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:43.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.715+0000 7f96dff86700 1 -- 172.21.15.169:0/1000245201 >> 172.21.15.169:0/1000245201 conn(0x7f96d80770e0 msgr2=0x7f96d80ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:43.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.715+0000 7f96dff86700 1 -- 172.21.15.169:0/1000245201 shutdown_connections 2024-03-28T00:55:43.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:43.715+0000 7f96dff86700 1 -- 172.21.15.169:0/1000245201 wait complete. 2024-03-28T00:55:44.121 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495665 got 197568495668 for osd.6 2024-03-28T00:55:44.122 DEBUG:teuthology.parallel:result is None 2024-03-28T00:55:44.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:44 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1071802014' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T00:55:44.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:44 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1000245201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T00:55:44.529 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:44 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1071802014' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T00:55:44.529 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:44 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1000245201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T00:55:44.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:44 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1071802014' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T00:55:44.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:44 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1000245201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T00:55:45.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:45 smithi177 ceph-mon[36206]: pgmap v326: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T00:55:45.434 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:45 smithi169 ceph-mon[32274]: pgmap v326: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T00:55:45.509 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:45 smithi174 ceph-mon[36060]: pgmap v326: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T00:55:45.869 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574923 got 55834574925 for osd.1 2024-03-28T00:55:45.870 DEBUG:teuthology.parallel:result is None 2024-03-28T00:55:46.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.903+0000 7f95fbe54700 1 -- 172.21.15.169:0/4000645403 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95f41021e0 msgr2=0x7f95f4102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:46.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.903+0000 7f95fbe54700 1 --2- 172.21.15.169:0/4000645403 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95f41021e0 0x7f95f4102660 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f95e4009990 tx=0x7f95e4009ca0 comp rx=0 tx=0).stop 2024-03-28T00:55:46.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.903+0000 7f95fbe54700 1 -- 172.21.15.169:0/4000645403 shutdown_connections 2024-03-28T00:55:46.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.903+0000 7f95fbe54700 1 --2- 172.21.15.169:0/4000645403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95f4102ba0 0x7f95f410f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:46.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.903+0000 7f95fbe54700 1 --2- 172.21.15.169:0/4000645403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95f4108290 0x7f95f4108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:46.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.903+0000 7f95fbe54700 1 --2- 172.21.15.169:0/4000645403 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95f41021e0 0x7f95f4102660 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:46.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.903+0000 7f95fbe54700 1 -- 172.21.15.169:0/4000645403 >> 172.21.15.169:0/4000645403 conn(0x7f95f40792c0 msgr2=0x7f95f40796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:46.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.903+0000 7f95fbe54700 1 -- 172.21.15.169:0/4000645403 shutdown_connections 2024-03-28T00:55:46.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.903+0000 7f95fbe54700 1 -- 172.21.15.169:0/4000645403 wait complete. 2024-03-28T00:55:46.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.903+0000 7f95fbe54700 1 Processor -- start 2024-03-28T00:55:46.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.903+0000 7f95fbe54700 1 -- start start 2024-03-28T00:55:46.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.904+0000 7f95fbe54700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95f41021e0 0x7f95f419e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:46.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.904+0000 7f95fbe54700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95f4102ba0 0x7f95f419ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:46.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.904+0000 7f95fbe54700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95f4108290 0x7f95f4198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:46.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.904+0000 7f95fbe54700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f41124c0 con 0x7f95f41021e0 2024-03-28T00:55:46.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.904+0000 7f95fbe54700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f4112340 con 0x7f95f4108290 2024-03-28T00:55:46.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.904+0000 7f95fbe54700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f4112640 con 0x7f95f4102ba0 2024-03-28T00:55:46.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.904+0000 7f95f9bf0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95f41021e0 0x7f95f419e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:46.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.904+0000 7f95f93ef700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95f4102ba0 0x7f95f419ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:46.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.904+0000 7f95f93ef700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95f4102ba0 0x7f95f419ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:57376/0 (socket says 172.21.15.169:57376) 2024-03-28T00:55:46.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.904+0000 7f95f93ef700 1 -- 172.21.15.169:0/3601439554 learned_addr learned my addr 172.21.15.169:0/3601439554 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:46.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.904+0000 7f95fa3f1700 1 --2- 172.21.15.169:0/3601439554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95f4108290 0x7f95f4198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:46.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.904+0000 7f95f93ef700 1 -- 172.21.15.169:0/3601439554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95f4108290 msgr2=0x7f95f4198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:46.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.904+0000 7f95f93ef700 1 --2- 172.21.15.169:0/3601439554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95f4108290 0x7f95f4198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:46.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.904+0000 7f95f93ef700 1 -- 172.21.15.169:0/3601439554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95f41021e0 msgr2=0x7f95f419e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:46.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.904+0000 7f95f93ef700 1 --2- 172.21.15.169:0/3601439554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95f41021e0 0x7f95f419e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:46.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.904+0000 7f95f93ef700 1 -- 172.21.15.169:0/3601439554 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f4199020 con 0x7f95f4102ba0 2024-03-28T00:55:46.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.904+0000 7f95f93ef700 1 --2- 172.21.15.169:0/3601439554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95f4102ba0 0x7f95f419ebb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f95e4000c00 tx=0x7f95e400bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:46.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.904+0000 7f95eaffd700 1 -- 172.21.15.169:0/3601439554 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e4027070 con 0x7f95f4102ba0 2024-03-28T00:55:46.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.905+0000 7f95fbe54700 1 -- 172.21.15.169:0/3601439554 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f41992b0 con 0x7f95f4102ba0 2024-03-28T00:55:46.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.905+0000 7f95eaffd700 1 -- 172.21.15.169:0/3601439554 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95e4003d70 con 0x7f95f4102ba0 2024-03-28T00:55:46.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.905+0000 7f95fbe54700 1 -- 172.21.15.169:0/3601439554 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f41a5420 con 0x7f95f4102ba0 2024-03-28T00:55:46.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.905+0000 7f95eaffd700 1 -- 172.21.15.169:0/3601439554 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e400f460 con 0x7f95f4102ba0 2024-03-28T00:55:46.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.905+0000 7f95fbe54700 1 -- 172.21.15.169:0/3601439554 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f41038d0 con 0x7f95f4102ba0 2024-03-28T00:55:46.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.907+0000 7f95eaffd700 1 -- 172.21.15.169:0/3601439554 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f95e402a070 con 0x7f95f4102ba0 2024-03-28T00:55:46.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.907+0000 7f95eaffd700 1 --2- 172.21.15.169:0/3601439554 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f95e00a2920 0x7f95e00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:46.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.907+0000 7f95f9bf0700 1 --2- 172.21.15.169:0/3601439554 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f95e00a2920 0x7f95e00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:46.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.907+0000 7f95eaffd700 1 -- 172.21.15.169:0/3601439554 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f95e40cfd20 con 0x7f95f4102ba0 2024-03-28T00:55:46.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.908+0000 7f95f9bf0700 1 --2- 172.21.15.169:0/3601439554 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f95e00a2920 0x7f95e00a4de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f95f0009c60 tx=0x7f95f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:46.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:46.915+0000 7f95eaffd700 1 -- 172.21.15.169:0/3601439554 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f95e4013070 con 0x7f95f4102ba0 2024-03-28T00:55:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.087+0000 7fe57b09a700 1 -- 172.21.15.169:0/4023897943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe574101090 msgr2=0x7fe5741014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.087+0000 7fe57b09a700 1 --2- 172.21.15.169:0/4023897943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe574101090 0x7fe5741014f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe5600098f0 tx=0x7fe560009c00 comp rx=0 tx=0).stop 2024-03-28T00:55:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.087+0000 7fe57b09a700 1 -- 172.21.15.169:0/4023897943 shutdown_connections 2024-03-28T00:55:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.087+0000 7fe57b09a700 1 --2- 172.21.15.169:0/4023897943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe574101a30 0x7fe57410df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.087+0000 7fe57b09a700 1 --2- 172.21.15.169:0/4023897943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5741070d0 0x7fe5741074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.087+0000 7fe57b09a700 1 --2- 172.21.15.169:0/4023897943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe574101090 0x7fe5741014f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.087+0000 7fe57b09a700 1 -- 172.21.15.169:0/4023897943 >> 172.21.15.169:0/4023897943 conn(0x7fe574077fc0 msgr2=0x7fe5740783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.087+0000 7fe57b09a700 1 -- 172.21.15.169:0/4023897943 shutdown_connections 2024-03-28T00:55:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.087+0000 7fe57b09a700 1 -- 172.21.15.169:0/4023897943 wait complete. 2024-03-28T00:55:47.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.087+0000 7fe57b09a700 1 Processor -- start 2024-03-28T00:55:47.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.088+0000 7fe57b09a700 1 -- start start 2024-03-28T00:55:47.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.088+0000 7fe57b09a700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe574101090 0x7fe574199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:47.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.088+0000 7fe57b09a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe574101a30 0x7fe574199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:47.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.088+0000 7fe57b09a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5741070d0 0x7fe5741a4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:47.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.088+0000 7fe57b09a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe574111380 con 0x7fe5741070d0 2024-03-28T00:55:47.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.088+0000 7fe57b09a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe574111200 con 0x7fe574101a30 2024-03-28T00:55:47.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.088+0000 7fe57b09a700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe574111500 con 0x7fe574101090 2024-03-28T00:55:47.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.088+0000 7fe579637700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5741070d0 0x7fe5741a4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:47.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.088+0000 7fe578e36700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe574101090 0x7fe574199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:47.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.088+0000 7fe579637700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5741070d0 0x7fe5741a4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:32968/0 (socket says 172.21.15.169:32968) 2024-03-28T00:55:47.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.088+0000 7fe579637700 1 -- 172.21.15.169:0/3413535692 learned_addr learned my addr 172.21.15.169:0/3413535692 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:47.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.088+0000 7fe573fff700 1 --2- 172.21.15.169:0/3413535692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe574101a30 0x7fe574199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:47.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.088+0000 7fe578e36700 1 -- 172.21.15.169:0/3413535692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe574101a30 msgr2=0x7fe574199ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:47.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.088+0000 7fe578e36700 1 --2- 172.21.15.169:0/3413535692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe574101a30 0x7fe574199ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:47.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.088+0000 7fe578e36700 1 -- 172.21.15.169:0/3413535692 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5741070d0 msgr2=0x7fe5741a4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:47.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.088+0000 7fe578e36700 1 --2- 172.21.15.169:0/3413535692 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5741070d0 0x7fe5741a4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:47.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.088+0000 7fe578e36700 1 -- 172.21.15.169:0/3413535692 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5741a4f50 con 0x7fe574101090 2024-03-28T00:55:47.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.088+0000 7fe573fff700 1 --2- 172.21.15.169:0/3413535692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe574101a30 0x7fe574199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:55:47.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.088+0000 7fe579637700 1 --2- 172.21.15.169:0/3413535692 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5741070d0 0x7fe5741a4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:55:47.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.088+0000 7fe578e36700 1 --2- 172.21.15.169:0/3413535692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe574101090 0x7fe574199990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe56800eb90 tx=0x7fe56800eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:47.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.089+0000 7fe571ffb700 1 -- 172.21.15.169:0/3413535692 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56800ccf0 con 0x7fe574101090 2024-03-28T00:55:47.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.089+0000 7fe571ffb700 1 -- 172.21.15.169:0/3413535692 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe568004500 con 0x7fe574101090 2024-03-28T00:55:47.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.089+0000 7fe57b09a700 1 -- 172.21.15.169:0/3413535692 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5741a5180 con 0x7fe574101090 2024-03-28T00:55:47.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.089+0000 7fe571ffb700 1 -- 172.21.15.169:0/3413535692 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe568010490 con 0x7fe574101090 2024-03-28T00:55:47.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.089+0000 7fe57b09a700 1 -- 172.21.15.169:0/3413535692 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5741a5690 con 0x7fe574101090 2024-03-28T00:55:47.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.090+0000 7fe57b09a700 1 -- 172.21.15.169:0/3413535692 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe574068e50 con 0x7fe574101090 2024-03-28T00:55:47.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.091+0000 7fe571ffb700 1 -- 172.21.15.169:0/3413535692 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fe568004000 con 0x7fe574101090 2024-03-28T00:55:47.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.091+0000 7fe571ffb700 1 --2- 172.21.15.169:0/3413535692 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe5640a2920 0x7fe5640a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:47.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.091+0000 7fe573fff700 1 --2- 172.21.15.169:0/3413535692 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe5640a2920 0x7fe5640a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:47.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.091+0000 7fe571ffb700 1 -- 172.21.15.169:0/3413535692 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fe5680cf1e0 con 0x7fe574101090 2024-03-28T00:55:47.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.092+0000 7fe573fff700 1 --2- 172.21.15.169:0/3413535692 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe5640a2920 0x7fe5640a4de0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe560000c00 tx=0x7fe560005b40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:47.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.100+0000 7fe571ffb700 1 -- 172.21.15.169:0/3413535692 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe56809bdc0 con 0x7fe574101090 2024-03-28T00:55:47.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.119+0000 7f95fbe54700 1 -- 172.21.15.169:0/3601439554 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f95f402da30 con 0x7f95f4102ba0 2024-03-28T00:55:47.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.120+0000 7f95eaffd700 1 -- 172.21.15.169:0/3601439554 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f95e409c990 con 0x7f95f4102ba0 2024-03-28T00:55:47.123 INFO:teuthology.orchestra.run.smithi169.stdout:77309411401 2024-03-28T00:55:47.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.122+0000 7f95fbe54700 1 -- 172.21.15.169:0/3601439554 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f95e00a2920 msgr2=0x7f95e00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:47.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.122+0000 7f95fbe54700 1 --2- 172.21.15.169:0/3601439554 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f95e00a2920 0x7f95e00a4de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f95f0009c60 tx=0x7f95f0009260 comp rx=0 tx=0).stop 2024-03-28T00:55:47.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.122+0000 7f95fbe54700 1 -- 172.21.15.169:0/3601439554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95f4102ba0 msgr2=0x7f95f419ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:47.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.122+0000 7f95fbe54700 1 --2- 172.21.15.169:0/3601439554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95f4102ba0 0x7f95f419ebb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f95e4000c00 tx=0x7f95e400bab0 comp rx=0 tx=0).stop 2024-03-28T00:55:47.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.122+0000 7f95fbe54700 1 -- 172.21.15.169:0/3601439554 shutdown_connections 2024-03-28T00:55:47.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.122+0000 7f95fbe54700 1 --2- 172.21.15.169:0/3601439554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95f4108290 0x7f95f4198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:47.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.122+0000 7f95fbe54700 1 --2- 172.21.15.169:0/3601439554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f95f41021e0 0x7f95f419e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:47.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.122+0000 7f95fbe54700 1 --2- 172.21.15.169:0/3601439554 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f95e00a2920 0x7f95e00a4de0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:47.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.122+0000 7f95fbe54700 1 --2- 172.21.15.169:0/3601439554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95f4102ba0 0x7f95f419ebb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:47.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.122+0000 7f95fbe54700 1 -- 172.21.15.169:0/3601439554 >> 172.21.15.169:0/3601439554 conn(0x7f95f40792c0 msgr2=0x7f95f4100eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:47.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.122+0000 7f95fbe54700 1 -- 172.21.15.169:0/3601439554 shutdown_connections 2024-03-28T00:55:47.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.122+0000 7f95fbe54700 1 -- 172.21.15.169:0/3601439554 wait complete. 2024-03-28T00:55:47.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:47 smithi169 ceph-mon[32274]: pgmap v327: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T00:55:47.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:47 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3601439554' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T00:55:47.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.314+0000 7fe57b09a700 1 -- 172.21.15.169:0/3413535692 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7fe57402da30 con 0x7fe574101090 2024-03-28T00:55:47.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.314+0000 7fe571ffb700 1 -- 172.21.15.169:0/3413535692 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fe56809b6c0 con 0x7fe574101090 2024-03-28T00:55:47.317 INFO:teuthology.orchestra.run.smithi169.stdout:343597383708 2024-03-28T00:55:47.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.316+0000 7fe57b09a700 1 -- 172.21.15.169:0/3413535692 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe5640a2920 msgr2=0x7fe5640a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:47.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.316+0000 7fe57b09a700 1 --2- 172.21.15.169:0/3413535692 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe5640a2920 0x7fe5640a4de0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe560000c00 tx=0x7fe560005b40 comp rx=0 tx=0).stop 2024-03-28T00:55:47.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.316+0000 7fe57b09a700 1 -- 172.21.15.169:0/3413535692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe574101090 msgr2=0x7fe574199990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:47.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.316+0000 7fe57b09a700 1 --2- 172.21.15.169:0/3413535692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe574101090 0x7fe574199990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe56800eb90 tx=0x7fe56800eea0 comp rx=0 tx=0).stop 2024-03-28T00:55:47.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.316+0000 7fe57b09a700 1 -- 172.21.15.169:0/3413535692 shutdown_connections 2024-03-28T00:55:47.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.316+0000 7fe57b09a700 1 --2- 172.21.15.169:0/3413535692 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5741070d0 0x7fe5741a4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:47.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.316+0000 7fe57b09a700 1 --2- 172.21.15.169:0/3413535692 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe5640a2920 0x7fe5640a4de0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:47.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.316+0000 7fe57b09a700 1 --2- 172.21.15.169:0/3413535692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe574101090 0x7fe574199990 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:47.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.316+0000 7fe57b09a700 1 --2- 172.21.15.169:0/3413535692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe574101a30 0x7fe574199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:47.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.316+0000 7fe57b09a700 1 -- 172.21.15.169:0/3413535692 >> 172.21.15.169:0/3413535692 conn(0x7fe574077fc0 msgr2=0x7fe5740ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:47.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.317+0000 7fe57b09a700 1 -- 172.21.15.169:0/3413535692 shutdown_connections 2024-03-28T00:55:47.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:47.317+0000 7fe57b09a700 1 -- 172.21.15.169:0/3413535692 wait complete. 2024-03-28T00:55:47.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:47 smithi174 ceph-mon[36060]: pgmap v327: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T00:55:47.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:47 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3601439554' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T00:55:47.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:47 smithi177 ceph-mon[36206]: pgmap v327: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T00:55:47.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:47 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3601439554' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T00:55:47.991 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383702 got 343597383708 for osd.11 2024-03-28T00:55:47.991 DEBUG:teuthology.parallel:result is None 2024-03-28T00:55:48.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:48 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3413535692' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T00:55:48.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:48 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3413535692' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T00:55:48.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:48 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3413535692' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T00:55:49.271 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:49 smithi169 ceph-mon[32274]: pgmap v328: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T00:55:49.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:49 smithi174 ceph-mon[36060]: pgmap v328: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T00:55:49.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:49 smithi177 ceph-mon[36206]: pgmap v328: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T00:55:49.682 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411396 got 77309411401 for osd.2 2024-03-28T00:55:49.682 DEBUG:teuthology.parallel:result is None 2024-03-28T00:55:50.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.341+0000 7fe1824c9700 1 -- 172.21.15.169:0/3676281398 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe17c107200 msgr2=0x7fe17c1075e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:50.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.341+0000 7fe1824c9700 1 --2- 172.21.15.169:0/3676281398 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe17c107200 0x7fe17c1075e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009990 tx=0x7fe16c009ca0 comp rx=0 tx=0).stop 2024-03-28T00:55:50.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.341+0000 7fe1824c9700 1 -- 172.21.15.169:0/3676281398 shutdown_connections 2024-03-28T00:55:50.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.341+0000 7fe1824c9700 1 --2- 172.21.15.169:0/3676281398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe17c101140 0x7fe17c10f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:50.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.341+0000 7fe1824c9700 1 --2- 172.21.15.169:0/3676281398 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe17c107200 0x7fe17c1075e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:50.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.341+0000 7fe1824c9700 1 --2- 172.21.15.169:0/3676281398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe17c100780 0x7fe17c100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:50.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.341+0000 7fe1824c9700 1 -- 172.21.15.169:0/3676281398 >> 172.21.15.169:0/3676281398 conn(0x7fe17c077690 msgr2=0x7fe17c0feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:50.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.341+0000 7fe1824c9700 1 -- 172.21.15.169:0/3676281398 shutdown_connections 2024-03-28T00:55:50.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.341+0000 7fe1824c9700 1 -- 172.21.15.169:0/3676281398 wait complete. 2024-03-28T00:55:50.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.341+0000 7fe1824c9700 1 Processor -- start 2024-03-28T00:55:50.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.341+0000 7fe1824c9700 1 -- start start 2024-03-28T00:55:50.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.342+0000 7fe1824c9700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe17c100780 0x7fe17c101960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:50.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.342+0000 7fe1824c9700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe17c101140 0x7fe17c101ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:50.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.342+0000 7fe1824c9700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe17c107200 0x7fe17c1023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:50.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.342+0000 7fe1824c9700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c1125c0 con 0x7fe17c107200 2024-03-28T00:55:50.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.342+0000 7fe1824c9700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c112440 con 0x7fe17c100780 2024-03-28T00:55:50.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.342+0000 7fe1824c9700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c112740 con 0x7fe17c101140 2024-03-28T00:55:50.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.342+0000 7fe180a66700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe17c107200 0x7fe17c1023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:50.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.342+0000 7fe17bfff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe17c100780 0x7fe17c101960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:50.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.342+0000 7fe180a66700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe17c107200 0x7fe17c1023e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:32990/0 (socket says 172.21.15.169:32990) 2024-03-28T00:55:50.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.342+0000 7fe180a66700 1 -- 172.21.15.169:0/269964530 learned_addr learned my addr 172.21.15.169:0/269964530 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:50.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.342+0000 7fe17b7fe700 1 --2- 172.21.15.169:0/269964530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe17c101140 0x7fe17c101ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:50.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.342+0000 7fe17b7fe700 1 -- 172.21.15.169:0/269964530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe17c100780 msgr2=0x7fe17c101960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:50.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.342+0000 7fe17b7fe700 1 --2- 172.21.15.169:0/269964530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe17c100780 0x7fe17c101960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:50.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.342+0000 7fe17b7fe700 1 -- 172.21.15.169:0/269964530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe17c107200 msgr2=0x7fe17c1023e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:50.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.342+0000 7fe17b7fe700 1 --2- 172.21.15.169:0/269964530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe17c107200 0x7fe17c1023e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:50.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.342+0000 7fe17b7fe700 1 -- 172.21.15.169:0/269964530 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17c1a6f50 con 0x7fe17c101140 2024-03-28T00:55:50.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.342+0000 7fe180a66700 1 --2- 172.21.15.169:0/269964530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe17c107200 0x7fe17c1023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:55:50.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.343+0000 7fe17b7fe700 1 --2- 172.21.15.169:0/269964530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe17c101140 0x7fe17c101ea0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe17000e9f0 tx=0x7fe17000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:50.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.343+0000 7fe1797fa700 1 -- 172.21.15.169:0/269964530 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17000c9c0 con 0x7fe17c101140 2024-03-28T00:55:50.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.343+0000 7fe1797fa700 1 -- 172.21.15.169:0/269964530 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe17000cb60 con 0x7fe17c101140 2024-03-28T00:55:50.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.343+0000 7fe1824c9700 1 -- 172.21.15.169:0/269964530 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe17c1a7240 con 0x7fe17c101140 2024-03-28T00:55:50.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.343+0000 7fe1797fa700 1 -- 172.21.15.169:0/269964530 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe170018680 con 0x7fe17c101140 2024-03-28T00:55:50.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.343+0000 7fe1824c9700 1 -- 172.21.15.169:0/269964530 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe17c1a7780 con 0x7fe17c101140 2024-03-28T00:55:50.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.344+0000 7fe1824c9700 1 -- 172.21.15.169:0/269964530 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe17c10a7d0 con 0x7fe17c101140 2024-03-28T00:55:50.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.345+0000 7fe1797fa700 1 -- 172.21.15.169:0/269964530 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fe170010430 con 0x7fe17c101140 2024-03-28T00:55:50.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.345+0000 7fe1797fa700 1 --2- 172.21.15.169:0/269964530 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe1640a2920 0x7fe1640a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:50.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.345+0000 7fe17bfff700 1 --2- 172.21.15.169:0/269964530 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe1640a2920 0x7fe1640a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:50.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.345+0000 7fe1797fa700 1 -- 172.21.15.169:0/269964530 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fe1700cf0e0 con 0x7fe17c101140 2024-03-28T00:55:50.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.346+0000 7fe17bfff700 1 --2- 172.21.15.169:0/269964530 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe1640a2920 0x7fe1640a4de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe16c00b2e0 tx=0x7fe16c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:50.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.350+0000 7fe1797fa700 1 -- 172.21.15.169:0/269964530 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe17009bc60 con 0x7fe17c101140 2024-03-28T00:55:50.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.549+0000 7fe1824c9700 1 -- 172.21.15.169:0/269964530 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe17c103a80 con 0x7fe17c101140 2024-03-28T00:55:50.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.549+0000 7fe1797fa700 1 -- 172.21.15.169:0/269964530 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe17009b560 con 0x7fe17c101140 2024-03-28T00:55:50.552 INFO:teuthology.orchestra.run.smithi169.stdout:107374182468 2024-03-28T00:55:50.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.551+0000 7fe1824c9700 1 -- 172.21.15.169:0/269964530 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe1640a2920 msgr2=0x7fe1640a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:50.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.551+0000 7fe1824c9700 1 --2- 172.21.15.169:0/269964530 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe1640a2920 0x7fe1640a4de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe16c00b2e0 tx=0x7fe16c005960 comp rx=0 tx=0).stop 2024-03-28T00:55:50.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.551+0000 7fe1824c9700 1 -- 172.21.15.169:0/269964530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe17c101140 msgr2=0x7fe17c101ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:50.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.551+0000 7fe1824c9700 1 --2- 172.21.15.169:0/269964530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe17c101140 0x7fe17c101ea0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe17000e9f0 tx=0x7fe17000ed00 comp rx=0 tx=0).stop 2024-03-28T00:55:50.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.552+0000 7fe1824c9700 1 -- 172.21.15.169:0/269964530 shutdown_connections 2024-03-28T00:55:50.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.552+0000 7fe1824c9700 1 --2- 172.21.15.169:0/269964530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe17c107200 0x7fe17c1023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:50.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.552+0000 7fe1824c9700 1 --2- 172.21.15.169:0/269964530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe17c100780 0x7fe17c101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:50.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.552+0000 7fe1824c9700 1 --2- 172.21.15.169:0/269964530 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe1640a2920 0x7fe1640a4de0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:50.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.552+0000 7fe1824c9700 1 --2- 172.21.15.169:0/269964530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe17c101140 0x7fe17c101ea0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:50.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.552+0000 7fe1824c9700 1 -- 172.21.15.169:0/269964530 >> 172.21.15.169:0/269964530 conn(0x7fe17c077690 msgr2=0x7fe17c1052a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:50.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.552+0000 7fe1824c9700 1 -- 172.21.15.169:0/269964530 shutdown_connections 2024-03-28T00:55:50.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:50.552+0000 7fe1824c9700 1 -- 172.21.15.169:0/269964530 wait complete. 2024-03-28T00:55:51.290 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:51 smithi169 ceph-mon[32274]: pgmap v329: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s; 1 B/s, 0 objects/s recovering 2024-03-28T00:55:51.291 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:51 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/269964530' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T00:55:51.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:51 smithi174 ceph-mon[36060]: pgmap v329: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s; 1 B/s, 0 objects/s recovering 2024-03-28T00:55:51.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:51 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/269964530' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T00:55:51.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:51 smithi177 ceph-mon[36206]: pgmap v329: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s; 1 B/s, 0 objects/s recovering 2024-03-28T00:55:51.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:51 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/269964530' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T00:55:51.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.812+0000 7f352f112700 1 -- 172.21.15.169:0/4037847997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3528100ad0 msgr2=0x7f3528100f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:51.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.812+0000 7f352f112700 1 --2- 172.21.15.169:0/4037847997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3528100ad0 0x7f3528100f50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f351c0098f0 tx=0x7f351c009c00 comp rx=0 tx=0).stop 2024-03-28T00:55:51.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.812+0000 7f352f112700 1 -- 172.21.15.169:0/4037847997 shutdown_connections 2024-03-28T00:55:51.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.812+0000 7f352f112700 1 --2- 172.21.15.169:0/4037847997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3528101490 0x7f352810d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:51.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.812+0000 7f352f112700 1 --2- 172.21.15.169:0/4037847997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35281068b0 0x7f3528106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:51.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.812+0000 7f352f112700 1 --2- 172.21.15.169:0/4037847997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3528100ad0 0x7f3528100f50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:51.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.812+0000 7f352f112700 1 -- 172.21.15.169:0/4037847997 >> 172.21.15.169:0/4037847997 conn(0x7f35280763d0 msgr2=0x7f35280767e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:51.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.812+0000 7f352f112700 1 -- 172.21.15.169:0/4037847997 shutdown_connections 2024-03-28T00:55:51.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.812+0000 7f352f112700 1 -- 172.21.15.169:0/4037847997 wait complete. 2024-03-28T00:55:51.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.813+0000 7f352f112700 1 Processor -- start 2024-03-28T00:55:51.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.813+0000 7f352f112700 1 -- start start 2024-03-28T00:55:51.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.813+0000 7f352f112700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3528100ad0 0x7f352819e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:51.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.813+0000 7f352f112700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3528101490 0x7f352819eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:51.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.813+0000 7f352f112700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35281068b0 0x7f35281986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:51.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.813+0000 7f352f112700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3528110b80 con 0x7f35281068b0 2024-03-28T00:55:51.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.813+0000 7f352f112700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3528110a00 con 0x7f3528101490 2024-03-28T00:55:51.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.813+0000 7f352f112700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3528110d00 con 0x7f3528100ad0 2024-03-28T00:55:51.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.813+0000 7f352d6af700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35281068b0 0x7f35281986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:51.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.813+0000 7f352ceae700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3528100ad0 0x7f352819e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:51.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.813+0000 7f352d6af700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35281068b0 0x7f35281986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33008/0 (socket says 172.21.15.169:33008) 2024-03-28T00:55:51.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.813+0000 7f352d6af700 1 -- 172.21.15.169:0/3180291426 learned_addr learned my addr 172.21.15.169:0/3180291426 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:51.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.813+0000 7f352ceae700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3528100ad0 0x7f352819e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:57450/0 (socket says 172.21.15.169:57450) 2024-03-28T00:55:51.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.813+0000 7f3527fff700 1 --2- 172.21.15.169:0/3180291426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3528101490 0x7f352819eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:51.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.813+0000 7f352d6af700 1 -- 172.21.15.169:0/3180291426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3528100ad0 msgr2=0x7f352819e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:51.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.813+0000 7f352d6af700 1 --2- 172.21.15.169:0/3180291426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3528100ad0 0x7f352819e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:51.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.813+0000 7f352d6af700 1 -- 172.21.15.169:0/3180291426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3528101490 msgr2=0x7f352819eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:51.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.813+0000 7f352d6af700 1 --2- 172.21.15.169:0/3180291426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3528101490 0x7f352819eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:51.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.813+0000 7f352d6af700 1 -- 172.21.15.169:0/3180291426 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3528198fb0 con 0x7f35281068b0 2024-03-28T00:55:51.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.813+0000 7f352ceae700 1 --2- 172.21.15.169:0/3180291426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3528100ad0 0x7f352819e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:55:51.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.813+0000 7f352d6af700 1 --2- 172.21.15.169:0/3180291426 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35281068b0 0x7f35281986d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f352000c110 tx=0x7f352000c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:51.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.814+0000 7f3525ffb700 1 -- 172.21.15.169:0/3180291426 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35200092e0 con 0x7f35281068b0 2024-03-28T00:55:51.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.814+0000 7f352f112700 1 -- 172.21.15.169:0/3180291426 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3528199240 con 0x7f35281068b0 2024-03-28T00:55:51.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.814+0000 7f3525ffb700 1 -- 172.21.15.169:0/3180291426 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3520004bb0 con 0x7f35281068b0 2024-03-28T00:55:51.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.814+0000 7f3525ffb700 1 -- 172.21.15.169:0/3180291426 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3520007d10 con 0x7f35281068b0 2024-03-28T00:55:51.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.814+0000 7f352f112700 1 -- 172.21.15.169:0/3180291426 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35281a5580 con 0x7f35281068b0 2024-03-28T00:55:51.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.815+0000 7f352f112700 1 -- 172.21.15.169:0/3180291426 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3528101f40 con 0x7f35281068b0 2024-03-28T00:55:51.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.816+0000 7f3525ffb700 1 -- 172.21.15.169:0/3180291426 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f352001a020 con 0x7f35281068b0 2024-03-28T00:55:51.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.817+0000 7f3525ffb700 1 --2- 172.21.15.169:0/3180291426 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f35100a2920 0x7f35100a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:51.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.817+0000 7f352ceae700 1 --2- 172.21.15.169:0/3180291426 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f35100a2920 0x7f35100a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:51.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.817+0000 7f3525ffb700 1 -- 172.21.15.169:0/3180291426 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f35200cf4a0 con 0x7f35281068b0 2024-03-28T00:55:51.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.817+0000 7f352ceae700 1 --2- 172.21.15.169:0/3180291426 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f35100a2920 0x7f35100a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3518011490 tx=0x7f3518009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:51.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:51.820+0000 7f3525ffb700 1 -- 172.21.15.169:0/3180291426 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3520010040 con 0x7f35281068b0 2024-03-28T00:55:52.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:52.018+0000 7f352f112700 1 -- 172.21.15.169:0/3180291426 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f3528199580 con 0x7f35281068b0 2024-03-28T00:55:52.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:52.019+0000 7f3525ffb700 1 -- 172.21.15.169:0/3180291426 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f352009c080 con 0x7f35281068b0 2024-03-28T00:55:52.022 INFO:teuthology.orchestra.run.smithi169.stdout:287762808870 2024-03-28T00:55:52.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:52.022+0000 7f352f112700 1 -- 172.21.15.169:0/3180291426 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f35100a2920 msgr2=0x7f35100a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:52.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:52.022+0000 7f352f112700 1 --2- 172.21.15.169:0/3180291426 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f35100a2920 0x7f35100a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3518011490 tx=0x7f3518009260 comp rx=0 tx=0).stop 2024-03-28T00:55:52.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:52.022+0000 7f352f112700 1 -- 172.21.15.169:0/3180291426 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35281068b0 msgr2=0x7f35281986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:52.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:52.022+0000 7f352f112700 1 --2- 172.21.15.169:0/3180291426 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35281068b0 0x7f35281986d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f352000c110 tx=0x7f352000c420 comp rx=0 tx=0).stop 2024-03-28T00:55:52.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:52.022+0000 7f352f112700 1 -- 172.21.15.169:0/3180291426 shutdown_connections 2024-03-28T00:55:52.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:52.022+0000 7f352f112700 1 --2- 172.21.15.169:0/3180291426 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f35281068b0 0x7f35281986d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:52.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:52.022+0000 7f352f112700 1 --2- 172.21.15.169:0/3180291426 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f35100a2920 0x7f35100a4de0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:52.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:52.022+0000 7f352f112700 1 --2- 172.21.15.169:0/3180291426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3528100ad0 0x7f352819e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:52.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:52.022+0000 7f352f112700 1 --2- 172.21.15.169:0/3180291426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3528101490 0x7f352819eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:52.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:52.022+0000 7f352f112700 1 -- 172.21.15.169:0/3180291426 >> 172.21.15.169:0/3180291426 conn(0x7f35280763d0 msgr2=0x7f3528107660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:52.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:52.022+0000 7f352f112700 1 -- 172.21.15.169:0/3180291426 shutdown_connections 2024-03-28T00:55:52.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:52.022+0000 7f352f112700 1 -- 172.21.15.169:0/3180291426 wait complete. 2024-03-28T00:55:52.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:52 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3180291426' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T00:55:52.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:52 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3180291426' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T00:55:52.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:52 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3180291426' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T00:55:53.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.055+0000 7ff247757700 1 -- 172.21.15.169:0/1128715518 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2401074d0 msgr2=0x7ff24010f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:53.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.055+0000 7ff247757700 1 --2- 172.21.15.169:0/1128715518 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2401074d0 0x7ff24010f0a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff23800b0d0 tx=0x7ff23800b3e0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.055+0000 7ff247757700 1 -- 172.21.15.169:0/1128715518 shutdown_connections 2024-03-28T00:55:53.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.055+0000 7ff247757700 1 --2- 172.21.15.169:0/1128715518 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2401074d0 0x7ff24010f0a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.055+0000 7ff247757700 1 --2- 172.21.15.169:0/1128715518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2401023f0 0x7ff2401027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.055+0000 7ff247757700 1 --2- 172.21.15.169:0/1128715518 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff240102da0 0x7ff240106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.055+0000 7ff247757700 1 -- 172.21.15.169:0/1128715518 >> 172.21.15.169:0/1128715518 conn(0x7ff2400fdc60 msgr2=0x7ff240100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:53.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.055+0000 7ff247757700 1 -- 172.21.15.169:0/1128715518 shutdown_connections 2024-03-28T00:55:53.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.055+0000 7ff247757700 1 -- 172.21.15.169:0/1128715518 wait complete. 2024-03-28T00:55:53.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.055+0000 7ff247757700 1 Processor -- start 2024-03-28T00:55:53.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.055+0000 7ff247757700 1 -- start start 2024-03-28T00:55:53.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.055+0000 7ff247757700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2401023f0 0x7ff24019e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:53.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.055+0000 7ff247757700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff240102da0 0x7ff24019eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:53.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff247757700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2401074d0 0x7ff2401987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:53.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff247757700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2401124e0 con 0x7ff2401023f0 2024-03-28T00:55:53.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff247757700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff240112360 con 0x7ff240102da0 2024-03-28T00:55:53.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff247757700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff240112660 con 0x7ff2401074d0 2024-03-28T00:55:53.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff2454f3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2401023f0 0x7ff24019e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:53.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff2454f3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2401023f0 0x7ff24019e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33036/0 (socket says 172.21.15.169:33036) 2024-03-28T00:55:53.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff2454f3700 1 -- 172.21.15.169:0/2290182472 learned_addr learned my addr 172.21.15.169:0/2290182472 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:53.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff245cf4700 1 --2- 172.21.15.169:0/2290182472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2401074d0 0x7ff2401987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:53.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff244cf2700 1 --2- 172.21.15.169:0/2290182472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff240102da0 0x7ff24019eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:53.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff2454f3700 1 -- 172.21.15.169:0/2290182472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2401074d0 msgr2=0x7ff2401987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:53.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff2454f3700 1 --2- 172.21.15.169:0/2290182472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2401074d0 0x7ff2401987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff2454f3700 1 -- 172.21.15.169:0/2290182472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff240102da0 msgr2=0x7ff24019eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:53.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff2454f3700 1 --2- 172.21.15.169:0/2290182472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff240102da0 0x7ff24019eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff2454f3700 1 -- 172.21.15.169:0/2290182472 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff240198fc0 con 0x7ff2401023f0 2024-03-28T00:55:53.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff245cf4700 1 --2- 172.21.15.169:0/2290182472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2401074d0 0x7ff2401987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:55:53.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff244cf2700 1 --2- 172.21.15.169:0/2290182472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff240102da0 0x7ff24019eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:55:53.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff2454f3700 1 --2- 172.21.15.169:0/2290182472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2401023f0 0x7ff24019e660 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00e9a0 tx=0x7ff23c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:53.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff2367fc700 1 -- 172.21.15.169:0/2290182472 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23c00ca10 con 0x7ff2401023f0 2024-03-28T00:55:53.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff247757700 1 -- 172.21.15.169:0/2290182472 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2401992b0 con 0x7ff2401023f0 2024-03-28T00:55:53.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff2367fc700 1 -- 172.21.15.169:0/2290182472 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff23c00cbb0 con 0x7ff2401023f0 2024-03-28T00:55:53.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff2367fc700 1 -- 172.21.15.169:0/2290182472 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23c018680 con 0x7ff2401023f0 2024-03-28T00:55:53.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.056+0000 7ff247757700 1 -- 172.21.15.169:0/2290182472 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2401a54c0 con 0x7ff2401023f0 2024-03-28T00:55:53.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.057+0000 7ff247757700 1 -- 172.21.15.169:0/2290182472 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2401038d0 con 0x7ff2401023f0 2024-03-28T00:55:53.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.058+0000 7ff2367fc700 1 -- 172.21.15.169:0/2290182472 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7ff23c010430 con 0x7ff2401023f0 2024-03-28T00:55:53.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.059+0000 7ff2367fc700 1 --2- 172.21.15.169:0/2290182472 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff22c0a29a0 0x7ff22c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:53.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.059+0000 7ff244cf2700 1 --2- 172.21.15.169:0/2290182472 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff22c0a29a0 0x7ff22c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:53.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.059+0000 7ff2367fc700 1 -- 172.21.15.169:0/2290182472 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7ff23c0cf5a0 con 0x7ff2401023f0 2024-03-28T00:55:53.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.059+0000 7ff244cf2700 1 --2- 172.21.15.169:0/2290182472 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff22c0a29a0 0x7ff22c0a4e60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff2401013d0 tx=0x7ff230009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:53.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.063+0000 7ff2367fc700 1 -- 172.21.15.169:0/2290182472 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff23c09c180 con 0x7ff2401023f0 2024-03-28T00:55:53.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.250+0000 7fc886c41700 1 -- 172.21.15.169:0/4079723044 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc880101cd0 msgr2=0x7fc880105c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:53.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.250+0000 7fc886c41700 1 --2- 172.21.15.169:0/4079723044 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc880101cd0 0x7fc880105c90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc86c0099c0 tx=0x7fc86c009cd0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.250+0000 7fc886c41700 1 -- 172.21.15.169:0/4079723044 shutdown_connections 2024-03-28T00:55:53.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.250+0000 7fc886c41700 1 --2- 172.21.15.169:0/4079723044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc880106370 0x7fc88010df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.250+0000 7fc886c41700 1 --2- 172.21.15.169:0/4079723044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc880101320 0x7fc880101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.250+0000 7fc886c41700 1 --2- 172.21.15.169:0/4079723044 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc880101cd0 0x7fc880105c90 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.250+0000 7fc886c41700 1 -- 172.21.15.169:0/4079723044 >> 172.21.15.169:0/4079723044 conn(0x7fc8800770e0 msgr2=0x7fc8800774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:53.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.250+0000 7fc886c41700 1 -- 172.21.15.169:0/4079723044 shutdown_connections 2024-03-28T00:55:53.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.250+0000 7fc886c41700 1 -- 172.21.15.169:0/4079723044 wait complete. 2024-03-28T00:55:53.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.251+0000 7fc886c41700 1 Processor -- start 2024-03-28T00:55:53.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.251+0000 7fc886c41700 1 -- start start 2024-03-28T00:55:53.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.251+0000 7fc886c41700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc880101320 0x7fc88019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:53.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.251+0000 7fc886c41700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc880101cd0 0x7fc88019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:53.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.251+0000 7fc886c41700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc880106370 0x7fc880198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:53.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.251+0000 7fc886c41700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc880111330 con 0x7fc880101cd0 2024-03-28T00:55:53.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.251+0000 7fc886c41700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8801111b0 con 0x7fc880106370 2024-03-28T00:55:53.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.251+0000 7fc886c41700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8801114b0 con 0x7fc880101320 2024-03-28T00:55:53.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.251+0000 7fc8849dd700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc880101320 0x7fc88019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:53.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.251+0000 7fc8849dd700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc880101320 0x7fc88019e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:57494/0 (socket says 172.21.15.169:57494) 2024-03-28T00:55:53.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.251+0000 7fc87ffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc880101cd0 0x7fc88019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:53.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.251+0000 7fc8849dd700 1 -- 172.21.15.169:0/3997123681 learned_addr learned my addr 172.21.15.169:0/3997123681 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:53.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.251+0000 7fc8851de700 1 --2- 172.21.15.169:0/3997123681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc880106370 0x7fc880198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:53.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.251+0000 7fc8849dd700 1 -- 172.21.15.169:0/3997123681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc880106370 msgr2=0x7fc880198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:53.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.251+0000 7fc8849dd700 1 --2- 172.21.15.169:0/3997123681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc880106370 0x7fc880198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.251+0000 7fc8849dd700 1 -- 172.21.15.169:0/3997123681 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc880101cd0 msgr2=0x7fc88019ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:53.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.251+0000 7fc8849dd700 1 --2- 172.21.15.169:0/3997123681 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc880101cd0 0x7fc88019ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.251+0000 7fc8849dd700 1 -- 172.21.15.169:0/3997123681 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc880199020 con 0x7fc880101320 2024-03-28T00:55:53.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.252+0000 7fc8849dd700 1 --2- 172.21.15.169:0/3997123681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc880101320 0x7fc88019e670 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc87400e9a0 tx=0x7fc87400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:53.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.252+0000 7fc87dffb700 1 -- 172.21.15.169:0/3997123681 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87400ca10 con 0x7fc880101320 2024-03-28T00:55:53.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.252+0000 7fc87dffb700 1 -- 172.21.15.169:0/3997123681 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc87400cbb0 con 0x7fc880101320 2024-03-28T00:55:53.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.252+0000 7fc87dffb700 1 -- 172.21.15.169:0/3997123681 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc874018680 con 0x7fc880101320 2024-03-28T00:55:53.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.252+0000 7fc886c41700 1 -- 172.21.15.169:0/3997123681 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc880199310 con 0x7fc880101320 2024-03-28T00:55:53.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.252+0000 7fc886c41700 1 -- 172.21.15.169:0/3997123681 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8801a55f0 con 0x7fc880101320 2024-03-28T00:55:53.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.253+0000 7fc886c41700 1 -- 172.21.15.169:0/3997123681 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc880102770 con 0x7fc880101320 2024-03-28T00:55:53.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.254+0000 7fc87dffb700 1 -- 172.21.15.169:0/3997123681 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fc874047a90 con 0x7fc880101320 2024-03-28T00:55:53.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.255+0000 7fc87dffb700 1 --2- 172.21.15.169:0/3997123681 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc8700a2920 0x7fc8700a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:53.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.255+0000 7fc87ffff700 1 --2- 172.21.15.169:0/3997123681 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc8700a2920 0x7fc8700a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:53.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.255+0000 7fc87dffb700 1 -- 172.21.15.169:0/3997123681 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fc8740cf1e0 con 0x7fc880101320 2024-03-28T00:55:53.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.255+0000 7fc87ffff700 1 --2- 172.21.15.169:0/3997123681 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc8700a2920 0x7fc8700a4de0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc86c000c00 tx=0x7fc86c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:53.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.263+0000 7fc87dffb700 1 -- 172.21.15.169:0/3997123681 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc87409bd60 con 0x7fc880101320 2024-03-28T00:55:53.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.269+0000 7ff247757700 1 -- 172.21.15.169:0/2290182472 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff24002da30 con 0x7ff2401023f0 2024-03-28T00:55:53.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.269+0000 7ff2367fc700 1 -- 172.21.15.169:0/2290182472 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff23c09ba80 con 0x7ff2401023f0 2024-03-28T00:55:53.272 INFO:teuthology.orchestra.run.smithi169.stdout:141733920833 2024-03-28T00:55:53.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.271+0000 7ff247757700 1 -- 172.21.15.169:0/2290182472 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff22c0a29a0 msgr2=0x7ff22c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:53.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.271+0000 7ff247757700 1 --2- 172.21.15.169:0/2290182472 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff22c0a29a0 0x7ff22c0a4e60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff2401013d0 tx=0x7ff230009260 comp rx=0 tx=0).stop 2024-03-28T00:55:53.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.271+0000 7ff247757700 1 -- 172.21.15.169:0/2290182472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2401023f0 msgr2=0x7ff24019e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:53.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.271+0000 7ff247757700 1 --2- 172.21.15.169:0/2290182472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2401023f0 0x7ff24019e660 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00e9a0 tx=0x7ff23c00ecb0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.271+0000 7ff247757700 1 -- 172.21.15.169:0/2290182472 shutdown_connections 2024-03-28T00:55:53.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.271+0000 7ff247757700 1 --2- 172.21.15.169:0/2290182472 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff22c0a29a0 0x7ff22c0a4e60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.271+0000 7ff247757700 1 --2- 172.21.15.169:0/2290182472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2401074d0 0x7ff2401987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.271+0000 7ff247757700 1 --2- 172.21.15.169:0/2290182472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2401023f0 0x7ff24019e660 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.271+0000 7ff247757700 1 --2- 172.21.15.169:0/2290182472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff240102da0 0x7ff24019eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.271+0000 7ff247757700 1 -- 172.21.15.169:0/2290182472 >> 172.21.15.169:0/2290182472 conn(0x7ff2400fdc60 msgr2=0x7ff2400fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:53.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.271+0000 7ff247757700 1 -- 172.21.15.169:0/2290182472 shutdown_connections 2024-03-28T00:55:53.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.271+0000 7ff247757700 1 -- 172.21.15.169:0/2290182472 wait complete. 2024-03-28T00:55:53.285 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:53 smithi169 ceph-mon[32274]: pgmap v330: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-28T00:55:53.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.478+0000 7fc886c41700 1 -- 172.21.15.169:0/3997123681 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7fc880199540 con 0x7fc880101320 2024-03-28T00:55:53.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.478+0000 7fc87dffb700 1 -- 172.21.15.169:0/3997123681 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fc87409b660 con 0x7fc880101320 2024-03-28T00:55:53.481 INFO:teuthology.orchestra.run.smithi169.stdout:309237645346 2024-03-28T00:55:53.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.480+0000 7fc886c41700 1 -- 172.21.15.169:0/3997123681 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc8700a2920 msgr2=0x7fc8700a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:53.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.480+0000 7fc886c41700 1 --2- 172.21.15.169:0/3997123681 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc8700a2920 0x7fc8700a4de0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc86c000c00 tx=0x7fc86c005fb0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.480+0000 7fc886c41700 1 -- 172.21.15.169:0/3997123681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc880101320 msgr2=0x7fc88019e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:53.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.480+0000 7fc886c41700 1 --2- 172.21.15.169:0/3997123681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc880101320 0x7fc88019e670 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc87400e9a0 tx=0x7fc87400ecb0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.481+0000 7fc886c41700 1 -- 172.21.15.169:0/3997123681 shutdown_connections 2024-03-28T00:55:53.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.481+0000 7fc886c41700 1 --2- 172.21.15.169:0/3997123681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc880106370 0x7fc880198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.481+0000 7fc886c41700 1 --2- 172.21.15.169:0/3997123681 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc8700a2920 0x7fc8700a4de0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.481+0000 7fc886c41700 1 --2- 172.21.15.169:0/3997123681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc880101320 0x7fc88019e670 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.481+0000 7fc886c41700 1 --2- 172.21.15.169:0/3997123681 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc880101cd0 0x7fc88019ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.481+0000 7fc886c41700 1 -- 172.21.15.169:0/3997123681 >> 172.21.15.169:0/3997123681 conn(0x7fc8800770e0 msgr2=0x7fc88006a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:53.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.481+0000 7fc886c41700 1 -- 172.21.15.169:0/3997123681 shutdown_connections 2024-03-28T00:55:53.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.481+0000 7fc886c41700 1 -- 172.21.15.169:0/3997123681 wait complete. 2024-03-28T00:55:53.534 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.529+0000 7f05075b5700 1 -- 172.21.15.169:0/842233400 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0500107200 msgr2=0x7f05001075e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:53.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.529+0000 7f05075b5700 1 --2- 172.21.15.169:0/842233400 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0500107200 0x7f05001075e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f04fc009990 tx=0x7f04fc009ca0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.529+0000 7f05075b5700 1 -- 172.21.15.169:0/842233400 shutdown_connections 2024-03-28T00:55:53.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.529+0000 7f05075b5700 1 --2- 172.21.15.169:0/842233400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0500101140 0x7f050010f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.529+0000 7f05075b5700 1 --2- 172.21.15.169:0/842233400 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0500107200 0x7f05001075e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.529+0000 7f05075b5700 1 --2- 172.21.15.169:0/842233400 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0500100780 0x7f0500100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.529+0000 7f05075b5700 1 -- 172.21.15.169:0/842233400 >> 172.21.15.169:0/842233400 conn(0x7f0500077690 msgr2=0x7f05000feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:53.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.529+0000 7f05075b5700 1 -- 172.21.15.169:0/842233400 shutdown_connections 2024-03-28T00:55:53.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.529+0000 7f05075b5700 1 -- 172.21.15.169:0/842233400 wait complete. 2024-03-28T00:55:53.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.529+0000 7f05075b5700 1 Processor -- start 2024-03-28T00:55:53.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.529+0000 7f05075b5700 1 -- start start 2024-03-28T00:55:53.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f05075b5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0500100780 0x7f050019e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:53.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f05075b5700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0500101140 0x7f050019ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:53.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f05075b5700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0500107200 0x7f0500198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:53.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f05075b5700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05001125c0 con 0x7f0500100780 2024-03-28T00:55:53.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f05075b5700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0500112440 con 0x7f0500101140 2024-03-28T00:55:53.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f05075b5700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0500112740 con 0x7f0500107200 2024-03-28T00:55:53.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f0505351700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0500100780 0x7f050019e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:53.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f0505351700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0500100780 0x7f050019e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33072/0 (socket says 172.21.15.169:33072) 2024-03-28T00:55:53.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f0505351700 1 -- 172.21.15.169:0/1240504801 learned_addr learned my addr 172.21.15.169:0/1240504801 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:53.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f0505b52700 1 --2- 172.21.15.169:0/1240504801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0500107200 0x7f0500198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:53.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f0504b50700 1 --2- 172.21.15.169:0/1240504801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0500101140 0x7f050019ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:53.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f0505351700 1 -- 172.21.15.169:0/1240504801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0500107200 msgr2=0x7f0500198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:53.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f0505351700 1 --2- 172.21.15.169:0/1240504801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0500107200 0x7f0500198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f0505351700 1 -- 172.21.15.169:0/1240504801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0500101140 msgr2=0x7f050019ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:53.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f0505351700 1 --2- 172.21.15.169:0/1240504801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0500101140 0x7f050019ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f0505351700 1 -- 172.21.15.169:0/1240504801 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0500199060 con 0x7f0500100780 2024-03-28T00:55:53.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f0505351700 1 --2- 172.21.15.169:0/1240504801 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0500100780 0x7f050019e6b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00bae0 tx=0x7f04fc00bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:53.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f04f67fc700 1 -- 172.21.15.169:0/1240504801 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc026070 con 0x7f0500100780 2024-03-28T00:55:53.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f05075b5700 1 -- 172.21.15.169:0/1240504801 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05001992f0 con 0x7f0500100780 2024-03-28T00:55:53.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f04f67fc700 1 -- 172.21.15.169:0/1240504801 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04fc003d70 con 0x7f0500100780 2024-03-28T00:55:53.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f04f67fc700 1 -- 172.21.15.169:0/1240504801 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc021410 con 0x7f0500100780 2024-03-28T00:55:53.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.530+0000 7f05075b5700 1 -- 172.21.15.169:0/1240504801 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05001a5510 con 0x7f0500100780 2024-03-28T00:55:53.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.532+0000 7f05075b5700 1 -- 172.21.15.169:0/1240504801 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0500103aa0 con 0x7f0500100780 2024-03-28T00:55:53.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.533+0000 7f04f67fc700 1 -- 172.21.15.169:0/1240504801 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f04fc029070 con 0x7f0500100780 2024-03-28T00:55:53.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.533+0000 7f04f67fc700 1 --2- 172.21.15.169:0/1240504801 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04ec0a2920 0x7f04ec0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:53.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.533+0000 7f04f67fc700 1 -- 172.21.15.169:0/1240504801 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f04fc0d01d0 con 0x7f0500100780 2024-03-28T00:55:53.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.533+0000 7f0504b50700 1 --2- 172.21.15.169:0/1240504801 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04ec0a2920 0x7f04ec0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:53.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.534+0000 7f0504b50700 1 --2- 172.21.15.169:0/1240504801 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04ec0a2920 0x7f04ec0a4de0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f04f0009680 tx=0x7f04f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:53.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.537+0000 7f04f67fc700 1 -- 172.21.15.169:0/1240504801 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f04fc030080 con 0x7f0500100780 2024-03-28T00:55:53.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:53 smithi174 ceph-mon[36060]: pgmap v330: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-28T00:55:53.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:53 smithi177 ceph-mon[36206]: pgmap v330: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-28T00:55:53.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.738+0000 7f05075b5700 1 -- 172.21.15.169:0/1240504801 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f0500199d60 con 0x7f0500100780 2024-03-28T00:55:53.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.738+0000 7f04f67fc700 1 -- 172.21.15.169:0/1240504801 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f04fc0bc070 con 0x7f0500100780 2024-03-28T00:55:53.741 INFO:teuthology.orchestra.run.smithi169.stdout:163208757308 2024-03-28T00:55:53.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.740+0000 7f05075b5700 1 -- 172.21.15.169:0/1240504801 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04ec0a2920 msgr2=0x7f04ec0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:53.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.740+0000 7f05075b5700 1 --2- 172.21.15.169:0/1240504801 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04ec0a2920 0x7f04ec0a4de0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f04f0009680 tx=0x7f04f0009260 comp rx=0 tx=0).stop 2024-03-28T00:55:53.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.740+0000 7f05075b5700 1 -- 172.21.15.169:0/1240504801 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0500100780 msgr2=0x7f050019e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:53.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.740+0000 7f05075b5700 1 --2- 172.21.15.169:0/1240504801 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0500100780 0x7f050019e6b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00bae0 tx=0x7f04fc00bb10 comp rx=0 tx=0).stop 2024-03-28T00:55:53.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.740+0000 7f05075b5700 1 -- 172.21.15.169:0/1240504801 shutdown_connections 2024-03-28T00:55:53.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.740+0000 7f05075b5700 1 --2- 172.21.15.169:0/1240504801 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04ec0a2920 0x7f04ec0a4de0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.740+0000 7f05075b5700 1 --2- 172.21.15.169:0/1240504801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0500107200 0x7f0500198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.740+0000 7f05075b5700 1 --2- 172.21.15.169:0/1240504801 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0500100780 0x7f050019e6b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.740+0000 7f05075b5700 1 --2- 172.21.15.169:0/1240504801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0500101140 0x7f050019ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:53.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.740+0000 7f05075b5700 1 -- 172.21.15.169:0/1240504801 >> 172.21.15.169:0/1240504801 conn(0x7f0500077690 msgr2=0x7f05001052a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:53.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.740+0000 7f05075b5700 1 -- 172.21.15.169:0/1240504801 shutdown_connections 2024-03-28T00:55:53.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:53.740+0000 7f05075b5700 1 -- 172.21.15.169:0/1240504801 wait complete. 2024-03-28T00:55:54.149 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645342 got 309237645346 for osd.10 2024-03-28T00:55:54.149 DEBUG:teuthology.parallel:result is None 2024-03-28T00:55:54.503 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:54 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2290182472' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T00:55:54.504 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:54 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3997123681' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T00:55:54.504 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:54 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1240504801' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T00:55:54.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:54 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2290182472' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T00:55:54.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:54 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3997123681' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T00:55:54.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:54 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1240504801' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T00:55:54.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:54 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2290182472' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T00:55:54.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:54 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3997123681' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T00:55:54.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:54 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1240504801' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T00:55:54.841 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182464 got 107374182468 for osd.3 2024-03-28T00:55:54.841 DEBUG:teuthology.parallel:result is None 2024-03-28T00:55:55.377 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:55:55.395 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920827 got 141733920833 for osd.4 2024-03-28T00:55:55.395 DEBUG:teuthology.parallel:result is None 2024-03-28T00:55:55.503 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:55 smithi169 ceph-mon[32274]: pgmap v331: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T00:55:55.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:55 smithi174 ceph-mon[36060]: pgmap v331: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T00:55:55.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:55 smithi177 ceph-mon[36206]: pgmap v331: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T00:55:55.825 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808867 got 287762808870 for osd.9 2024-03-28T00:55:55.825 DEBUG:teuthology.parallel:result is None 2024-03-28T00:55:56.237 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757303 got 163208757308 for osd.5 2024-03-28T00:55:56.237 DEBUG:teuthology.parallel:result is None 2024-03-28T00:55:57.224 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:57 smithi169 ceph-mon[32274]: pgmap v332: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T00:55:57.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:57 smithi174 ceph-mon[36060]: pgmap v332: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T00:55:57.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:57 smithi177 ceph-mon[36206]: pgmap v332: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T00:55:58.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.368+0000 7f9a9fc51700 1 -- 172.21.15.169:0/1683044071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a98106850 msgr2=0x7f9a98106c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:58.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.368+0000 7f9a9fc51700 1 --2- 172.21.15.169:0/1683044071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a98106850 0x7f9a98106c30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9a90009990 tx=0x7f9a90009ca0 comp rx=0 tx=0).stop 2024-03-28T00:55:58.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.368+0000 7f9a9fc51700 1 -- 172.21.15.169:0/1683044071 shutdown_connections 2024-03-28T00:55:58.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.368+0000 7f9a9fc51700 1 --2- 172.21.15.169:0/1683044071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a98101210 0x7f9a9810d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:58.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.368+0000 7f9a9fc51700 1 --2- 172.21.15.169:0/1683044071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a98106850 0x7f9a98106c30 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:58.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.368+0000 7f9a9fc51700 1 --2- 172.21.15.169:0/1683044071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a98100850 0x7f9a98100cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:58.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.368+0000 7f9a9fc51700 1 -- 172.21.15.169:0/1683044071 >> 172.21.15.169:0/1683044071 conn(0x7f9a98077690 msgr2=0x7f9a980feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:58.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.368+0000 7f9a9fc51700 1 -- 172.21.15.169:0/1683044071 shutdown_connections 2024-03-28T00:55:58.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.368+0000 7f9a9fc51700 1 -- 172.21.15.169:0/1683044071 wait complete. 2024-03-28T00:55:58.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.368+0000 7f9a9fc51700 1 Processor -- start 2024-03-28T00:55:58.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.368+0000 7f9a9fc51700 1 -- start start 2024-03-28T00:55:58.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a9fc51700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a98100850 0x7f9a9819c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:58.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a9fc51700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a98101210 0x7f9a9819c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:58.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a9fc51700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a98106850 0x7f9a981965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:58.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a9fc51700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a981102f0 con 0x7f9a98100850 2024-03-28T00:55:58.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a9fc51700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a98110170 con 0x7f9a98106850 2024-03-28T00:55:58.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a9fc51700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a98110470 con 0x7f9a98101210 2024-03-28T00:55:58.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a9d9ed700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a98100850 0x7f9a9819c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:58.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a9d9ed700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a98100850 0x7f9a9819c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48170/0 (socket says 172.21.15.169:48170) 2024-03-28T00:55:58.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a9d9ed700 1 -- 172.21.15.169:0/2057119405 learned_addr learned my addr 172.21.15.169:0/2057119405 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:58.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a9d1ec700 1 --2- 172.21.15.169:0/2057119405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a98101210 0x7f9a9819c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:58.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a9e1ee700 1 --2- 172.21.15.169:0/2057119405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a98106850 0x7f9a981965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:58.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a9d9ed700 1 -- 172.21.15.169:0/2057119405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a98101210 msgr2=0x7f9a9819c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:58.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a9d9ed700 1 --2- 172.21.15.169:0/2057119405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a98101210 0x7f9a9819c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:58.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a9d9ed700 1 -- 172.21.15.169:0/2057119405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a98106850 msgr2=0x7f9a981965b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:58.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a9d9ed700 1 --2- 172.21.15.169:0/2057119405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a98106850 0x7f9a981965b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:58.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a9d9ed700 1 -- 172.21.15.169:0/2057119405 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a98196d10 con 0x7f9a98100850 2024-03-28T00:55:58.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a9e1ee700 1 --2- 172.21.15.169:0/2057119405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a98106850 0x7f9a981965b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:55:58.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a9d9ed700 1 --2- 172.21.15.169:0/2057119405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a98100850 0x7f9a9819c3f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9a90000c00 tx=0x7f9a9000bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:58.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a8effd700 1 -- 172.21.15.169:0/2057119405 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a90027070 con 0x7f9a98100850 2024-03-28T00:55:58.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a9fc51700 1 -- 172.21.15.169:0/2057119405 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a98196fa0 con 0x7f9a98100850 2024-03-28T00:55:58.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a8effd700 1 -- 172.21.15.169:0/2057119405 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a90003d70 con 0x7f9a98100850 2024-03-28T00:55:58.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a8effd700 1 -- 172.21.15.169:0/2057119405 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9000f460 con 0x7f9a98100850 2024-03-28T00:55:58.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.369+0000 7f9a9fc51700 1 -- 172.21.15.169:0/2057119405 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a981a32c0 con 0x7f9a98100850 2024-03-28T00:55:58.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.370+0000 7f9a9fc51700 1 -- 172.21.15.169:0/2057119405 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a98108de0 con 0x7f9a98100850 2024-03-28T00:55:58.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.371+0000 7f9a8effd700 1 -- 172.21.15.169:0/2057119405 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f9a9002a070 con 0x7f9a98100850 2024-03-28T00:55:58.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.372+0000 7f9a8effd700 1 --2- 172.21.15.169:0/2057119405 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9a840a2920 0x7f9a840a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:58.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.372+0000 7f9a9d1ec700 1 --2- 172.21.15.169:0/2057119405 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9a840a2920 0x7f9a840a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:58.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.372+0000 7f9a8effd700 1 -- 172.21.15.169:0/2057119405 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f9a900cfca0 con 0x7f9a98100850 2024-03-28T00:55:58.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.372+0000 7f9a9d1ec700 1 --2- 172.21.15.169:0/2057119405 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9a840a2920 0x7f9a840a4de0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9a98103fe0 tx=0x7f9a88009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:58.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.377+0000 7f9a8effd700 1 -- 172.21.15.169:0/2057119405 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9a90013070 con 0x7f9a98100850 2024-03-28T00:55:58.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.577+0000 7f9a9fc51700 1 -- 172.21.15.169:0/2057119405 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f9a98068e50 con 0x7f9a98100850 2024-03-28T00:55:58.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.578+0000 7f9a8effd700 1 -- 172.21.15.169:0/2057119405 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9a9009c910 con 0x7f9a98100850 2024-03-28T00:55:58.581 INFO:teuthology.orchestra.run.smithi169.stdout:253403070510 2024-03-28T00:55:58.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.580+0000 7f9a9fc51700 1 -- 172.21.15.169:0/2057119405 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9a840a2920 msgr2=0x7f9a840a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:58.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.580+0000 7f9a9fc51700 1 --2- 172.21.15.169:0/2057119405 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9a840a2920 0x7f9a840a4de0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9a98103fe0 tx=0x7f9a88009260 comp rx=0 tx=0).stop 2024-03-28T00:55:58.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.580+0000 7f9a9fc51700 1 -- 172.21.15.169:0/2057119405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a98100850 msgr2=0x7f9a9819c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:58.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.580+0000 7f9a9fc51700 1 --2- 172.21.15.169:0/2057119405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a98100850 0x7f9a9819c3f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9a90000c00 tx=0x7f9a9000bab0 comp rx=0 tx=0).stop 2024-03-28T00:55:58.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.580+0000 7f9a9fc51700 1 -- 172.21.15.169:0/2057119405 shutdown_connections 2024-03-28T00:55:58.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.580+0000 7f9a9fc51700 1 --2- 172.21.15.169:0/2057119405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a98106850 0x7f9a981965b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:58.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.580+0000 7f9a9fc51700 1 --2- 172.21.15.169:0/2057119405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a98100850 0x7f9a9819c3f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:58.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.580+0000 7f9a9fc51700 1 --2- 172.21.15.169:0/2057119405 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9a840a2920 0x7f9a840a4de0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:58.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.580+0000 7f9a9fc51700 1 --2- 172.21.15.169:0/2057119405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a98101210 0x7f9a9819c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:58.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.580+0000 7f9a9fc51700 1 -- 172.21.15.169:0/2057119405 >> 172.21.15.169:0/2057119405 conn(0x7f9a98077690 msgr2=0x7f9a980feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:58.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.580+0000 7f9a9fc51700 1 -- 172.21.15.169:0/2057119405 shutdown_connections 2024-03-28T00:55:58.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:58.580+0000 7f9a9fc51700 1 -- 172.21.15.169:0/2057119405 wait complete. 2024-03-28T00:55:59.250 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:59 smithi169 ceph-mon[32274]: pgmap v333: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T00:55:59.250 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:55:59 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2057119405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T00:55:59.280 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070503 got 253403070510 for osd.8 2024-03-28T00:55:59.281 DEBUG:teuthology.parallel:result is None 2024-03-28T00:55:59.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.477+0000 7f66ff0ab700 1 -- 172.21.15.169:0/3105745782 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f8101320 msgr2=0x7f66f8101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:59.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.477+0000 7f66ff0ab700 1 --2- 172.21.15.169:0/3105745782 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f8101320 0x7f66f8101700 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f66ec0099c0 tx=0x7f66ec009cd0 comp rx=0 tx=0).stop 2024-03-28T00:55:59.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.477+0000 7f66ff0ab700 1 -- 172.21.15.169:0/3105745782 shutdown_connections 2024-03-28T00:55:59.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.477+0000 7f66ff0ab700 1 --2- 172.21.15.169:0/3105745782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66f8106370 0x7f66f810df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:59.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.477+0000 7f66ff0ab700 1 --2- 172.21.15.169:0/3105745782 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f8101320 0x7f66f8101700 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:59.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.477+0000 7f66ff0ab700 1 --2- 172.21.15.169:0/3105745782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66f8101cd0 0x7f66f8105c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:59.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.478+0000 7f66ff0ab700 1 -- 172.21.15.169:0/3105745782 >> 172.21.15.169:0/3105745782 conn(0x7f66f80770e0 msgr2=0x7f66f80774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:59.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.478+0000 7f66ff0ab700 1 -- 172.21.15.169:0/3105745782 shutdown_connections 2024-03-28T00:55:59.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.478+0000 7f66ff0ab700 1 -- 172.21.15.169:0/3105745782 wait complete. 2024-03-28T00:55:59.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.478+0000 7f66ff0ab700 1 Processor -- start 2024-03-28T00:55:59.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.478+0000 7f66ff0ab700 1 -- start start 2024-03-28T00:55:59.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66ff0ab700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f8101320 0x7f66f81997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:59.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66ff0ab700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66f8101cd0 0x7f66f8199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:59.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66ff0ab700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66f8106370 0x7f66f81a4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:59.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66ff0ab700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f8111370 con 0x7f66f8101320 2024-03-28T00:55:59.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66ff0ab700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f81111f0 con 0x7f66f8101cd0 2024-03-28T00:55:59.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66ff0ab700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f81114f0 con 0x7f66f8106370 2024-03-28T00:55:59.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66fce47700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f8101320 0x7f66f81997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:59.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66f7fff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66f8101cd0 0x7f66f8199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:59.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66f7fff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66f8101cd0 0x7f66f8199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:39780/0 (socket says 172.21.15.169:39780) 2024-03-28T00:55:59.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66fce47700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f8101320 0x7f66f81997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48188/0 (socket says 172.21.15.169:48188) 2024-03-28T00:55:59.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66f7fff700 1 -- 172.21.15.169:0/2141383644 learned_addr learned my addr 172.21.15.169:0/2141383644 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:55:59.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66fd648700 1 --2- 172.21.15.169:0/2141383644 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66f8106370 0x7f66f81a4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:59.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66f7fff700 1 -- 172.21.15.169:0/2141383644 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66f8106370 msgr2=0x7f66f81a4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:59.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66f7fff700 1 --2- 172.21.15.169:0/2141383644 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66f8106370 0x7f66f81a4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:59.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66f7fff700 1 -- 172.21.15.169:0/2141383644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f8101320 msgr2=0x7f66f81997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:59.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66f7fff700 1 --2- 172.21.15.169:0/2141383644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f8101320 0x7f66f81997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:59.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66f7fff700 1 -- 172.21.15.169:0/2141383644 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f81a4d70 con 0x7f66f8101cd0 2024-03-28T00:55:59.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66fce47700 1 --2- 172.21.15.169:0/2141383644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f8101320 0x7f66f81997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:55:59.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66f7fff700 1 --2- 172.21.15.169:0/2141383644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66f8101cd0 0x7f66f8199cf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f66e400e9f0 tx=0x7f66e400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:59.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66f5ffb700 1 -- 172.21.15.169:0/2141383644 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e4004a30 con 0x7f66f8101cd0 2024-03-28T00:55:59.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66f5ffb700 1 -- 172.21.15.169:0/2141383644 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66e400cca0 con 0x7f66f8101cd0 2024-03-28T00:55:59.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66ff0ab700 1 -- 172.21.15.169:0/2141383644 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66f81a4fa0 con 0x7f66f8101cd0 2024-03-28T00:55:59.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.479+0000 7f66f5ffb700 1 -- 172.21.15.169:0/2141383644 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e40056c0 con 0x7f66f8101cd0 2024-03-28T00:55:59.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.480+0000 7f66ff0ab700 1 -- 172.21.15.169:0/2141383644 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f81a53f0 con 0x7f66f8101cd0 2024-03-28T00:55:59.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.480+0000 7f66ff0ab700 1 -- 172.21.15.169:0/2141383644 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66f8068e50 con 0x7f66f8101cd0 2024-03-28T00:55:59.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.481+0000 7f66f5ffb700 1 -- 172.21.15.169:0/2141383644 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f66e400c7c0 con 0x7f66f8101cd0 2024-03-28T00:55:59.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.482+0000 7f66f5ffb700 1 --2- 172.21.15.169:0/2141383644 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f66e80a2920 0x7f66e80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:55:59.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.482+0000 7f66fce47700 1 --2- 172.21.15.169:0/2141383644 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f66e80a2920 0x7f66e80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:55:59.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.482+0000 7f66f5ffb700 1 -- 172.21.15.169:0/2141383644 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f66e40cf310 con 0x7f66f8101cd0 2024-03-28T00:55:59.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.482+0000 7f66fce47700 1 --2- 172.21.15.169:0/2141383644 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f66e80a2920 0x7f66e80a4de0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f66ec000c00 tx=0x7f66ec005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:55:59.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.486+0000 7f66f5ffb700 1 -- 172.21.15.169:0/2141383644 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f66e409be90 con 0x7f66f8101cd0 2024-03-28T00:55:59.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:59 smithi174 ceph-mon[36060]: pgmap v333: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T00:55:59.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:55:59 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2057119405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T00:55:59.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:59 smithi177 ceph-mon[36206]: pgmap v333: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T00:55:59.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:55:59 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2057119405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T00:55:59.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.694+0000 7f66ff0ab700 1 -- 172.21.15.169:0/2141383644 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f66f802dbf0 con 0x7f66f8101cd0 2024-03-28T00:55:59.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.694+0000 7f66f5ffb700 1 -- 172.21.15.169:0/2141383644 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f66e409b790 con 0x7f66f8101cd0 2024-03-28T00:55:59.697 INFO:teuthology.orchestra.run.smithi169.stdout:34359738453 2024-03-28T00:55:59.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.697+0000 7f66ff0ab700 1 -- 172.21.15.169:0/2141383644 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f66e80a2920 msgr2=0x7f66e80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:59.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.697+0000 7f66ff0ab700 1 --2- 172.21.15.169:0/2141383644 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f66e80a2920 0x7f66e80a4de0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f66ec000c00 tx=0x7f66ec005fb0 comp rx=0 tx=0).stop 2024-03-28T00:55:59.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.697+0000 7f66ff0ab700 1 -- 172.21.15.169:0/2141383644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66f8101cd0 msgr2=0x7f66f8199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:55:59.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.697+0000 7f66ff0ab700 1 --2- 172.21.15.169:0/2141383644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66f8101cd0 0x7f66f8199cf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f66e400e9f0 tx=0x7f66e400ed00 comp rx=0 tx=0).stop 2024-03-28T00:55:59.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.697+0000 7f66ff0ab700 1 -- 172.21.15.169:0/2141383644 shutdown_connections 2024-03-28T00:55:59.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.697+0000 7f66ff0ab700 1 --2- 172.21.15.169:0/2141383644 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f66e80a2920 0x7f66e80a4de0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:59.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.697+0000 7f66ff0ab700 1 --2- 172.21.15.169:0/2141383644 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66f8106370 0x7f66f81a4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:59.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.697+0000 7f66ff0ab700 1 --2- 172.21.15.169:0/2141383644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66f8101320 0x7f66f81997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:59.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.697+0000 7f66ff0ab700 1 --2- 172.21.15.169:0/2141383644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66f8101cd0 0x7f66f8199cf0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:55:59.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.697+0000 7f66ff0ab700 1 -- 172.21.15.169:0/2141383644 >> 172.21.15.169:0/2141383644 conn(0x7f66f80770e0 msgr2=0x7f66f8105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:55:59.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.697+0000 7f66ff0ab700 1 -- 172.21.15.169:0/2141383644 shutdown_connections 2024-03-28T00:55:59.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:55:59.698+0000 7f66ff0ab700 1 -- 172.21.15.169:0/2141383644 wait complete. 2024-03-28T00:56:00.494 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:00 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2141383644' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T00:56:00.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:00 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2141383644' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T00:56:00.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:00 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2141383644' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T00:56:00.838 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738446 got 34359738453 for osd.0 2024-03-28T00:56:00.839 DEBUG:teuthology.parallel:result is None 2024-03-28T00:56:00.839 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-28T00:56:00.839 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T00:56:01.113 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:56:01.507 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:01 smithi169 ceph-mon[32274]: pgmap v334: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T00:56:01.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:01 smithi174 ceph-mon[36060]: pgmap v334: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T00:56:01.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:01 smithi177 ceph-mon[36206]: pgmap v334: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T00:56:02.246 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:02 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:56:02.246 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:02 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:56:02.246 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:02 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:56:02.246 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:02 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:56:02.246 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:02 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:56:02.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:02 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:56:02.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:02 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:56:02.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:02 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:56:02.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:02 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:56:02.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:02 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:56:02.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:02 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:56:02.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:02 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:56:02.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:02 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:56:02.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:02 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:56:02.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:02 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:56:02.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.764+0000 7fc3d8654700 1 -- 172.21.15.169:0/3195664697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc3d0101a30 msgr2=0x7fc3d010df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:02.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.764+0000 7fc3d8654700 1 --2- 172.21.15.169:0/3195664697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc3d0101a30 0x7fc3d010df10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc3c800b0a0 tx=0x7fc3c800b3b0 comp rx=0 tx=0).stop 2024-03-28T00:56:02.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.764+0000 7fc3d8654700 1 -- 172.21.15.169:0/3195664697 shutdown_connections 2024-03-28T00:56:02.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.764+0000 7fc3d8654700 1 --2- 172.21.15.169:0/3195664697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc3d0101a30 0x7fc3d010df10 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:02.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.764+0000 7fc3d8654700 1 --2- 172.21.15.169:0/3195664697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3d01070d0 0x7fc3d01074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:02.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.764+0000 7fc3d8654700 1 --2- 172.21.15.169:0/3195664697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3d0101090 0x7fc3d01014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:02.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.764+0000 7fc3d8654700 1 -- 172.21.15.169:0/3195664697 >> 172.21.15.169:0/3195664697 conn(0x7fc3d0077fc0 msgr2=0x7fc3d00783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:02.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.764+0000 7fc3d8654700 1 -- 172.21.15.169:0/3195664697 shutdown_connections 2024-03-28T00:56:02.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.764+0000 7fc3d8654700 1 -- 172.21.15.169:0/3195664697 wait complete. 2024-03-28T00:56:02.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.765+0000 7fc3d8654700 1 Processor -- start 2024-03-28T00:56:02.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.765+0000 7fc3d8654700 1 -- start start 2024-03-28T00:56:02.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.765+0000 7fc3d8654700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc3d0101090 0x7fc3d019e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:02.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.765+0000 7fc3d8654700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3d0101a30 0x7fc3d019ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:02.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.765+0000 7fc3d8654700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3d01070d0 0x7fc3d0198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:02.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.765+0000 7fc3d8654700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d0111350 con 0x7fc3d0101090 2024-03-28T00:56:02.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.765+0000 7fc3d8654700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d01111d0 con 0x7fc3d01070d0 2024-03-28T00:56:02.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.765+0000 7fc3d8654700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d01114d0 con 0x7fc3d0101a30 2024-03-28T00:56:02.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.765+0000 7fc3d63f0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc3d0101090 0x7fc3d019e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:02.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.765+0000 7fc3d63f0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc3d0101090 0x7fc3d019e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48198/0 (socket says 172.21.15.169:48198) 2024-03-28T00:56:02.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.765+0000 7fc3d63f0700 1 -- 172.21.15.169:0/4207211797 learned_addr learned my addr 172.21.15.169:0/4207211797 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:02.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.765+0000 7fc3d6bf1700 1 --2- 172.21.15.169:0/4207211797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3d01070d0 0x7fc3d0198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:02.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.765+0000 7fc3d5bef700 1 --2- 172.21.15.169:0/4207211797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3d0101a30 0x7fc3d019ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:02.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.765+0000 7fc3d63f0700 1 -- 172.21.15.169:0/4207211797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3d0101a30 msgr2=0x7fc3d019ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:02.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.765+0000 7fc3d63f0700 1 --2- 172.21.15.169:0/4207211797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3d0101a30 0x7fc3d019ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:02.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.765+0000 7fc3d63f0700 1 -- 172.21.15.169:0/4207211797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3d01070d0 msgr2=0x7fc3d0198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:02.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.765+0000 7fc3d63f0700 1 --2- 172.21.15.169:0/4207211797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3d01070d0 0x7fc3d0198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:02.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.765+0000 7fc3d63f0700 1 -- 172.21.15.169:0/4207211797 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d0199040 con 0x7fc3d0101090 2024-03-28T00:56:02.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.765+0000 7fc3d63f0700 1 --2- 172.21.15.169:0/4207211797 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc3d0101090 0x7fc3d019e690 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc00e9a0 tx=0x7fc3cc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:02.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.766+0000 7fc3c77fe700 1 -- 172.21.15.169:0/4207211797 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc00ca10 con 0x7fc3d0101090 2024-03-28T00:56:02.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.766+0000 7fc3c77fe700 1 -- 172.21.15.169:0/4207211797 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3cc00cbb0 con 0x7fc3d0101090 2024-03-28T00:56:02.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.766+0000 7fc3c77fe700 1 -- 172.21.15.169:0/4207211797 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc018680 con 0x7fc3d0101090 2024-03-28T00:56:02.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.766+0000 7fc3d8654700 1 -- 172.21.15.169:0/4207211797 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d0199330 con 0x7fc3d0101090 2024-03-28T00:56:02.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.766+0000 7fc3d8654700 1 -- 172.21.15.169:0/4207211797 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d01a5610 con 0x7fc3d0101090 2024-03-28T00:56:02.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.767+0000 7fc3d8654700 1 -- 172.21.15.169:0/4207211797 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d0102780 con 0x7fc3d0101090 2024-03-28T00:56:02.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.768+0000 7fc3c77fe700 1 -- 172.21.15.169:0/4207211797 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fc3cc010430 con 0x7fc3d0101090 2024-03-28T00:56:02.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.768+0000 7fc3c77fe700 1 --2- 172.21.15.169:0/4207211797 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc3bc0a29a0 0x7fc3bc0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:02.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.768+0000 7fc3d5bef700 1 --2- 172.21.15.169:0/4207211797 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc3bc0a29a0 0x7fc3bc0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:02.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.768+0000 7fc3c77fe700 1 -- 172.21.15.169:0/4207211797 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fc3cc0cf510 con 0x7fc3d0101090 2024-03-28T00:56:02.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.769+0000 7fc3d5bef700 1 --2- 172.21.15.169:0/4207211797 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc3bc0a29a0 0x7fc3bc0a4e60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0199f20 tx=0x7fc3c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:02.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.772+0000 7fc3c77fe700 1 -- 172.21.15.169:0/4207211797 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc3cc09c0f0 con 0x7fc3d0101090 2024-03-28T00:56:02.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.964+0000 7fc3d8654700 1 -- 172.21.15.169:0/4207211797 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc3d0104e80 con 0x7fc3bc0a29a0 2024-03-28T00:56:02.976 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.974+0000 7fc3c77fe700 1 -- 172.21.15.169:0/4207211797 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340629 (secure 0 0 0) 0x7fc3d0104e80 con 0x7fc3bc0a29a0 2024-03-28T00:56:02.982 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:02.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.977+0000 7fc3d8654700 1 -- 172.21.15.169:0/4207211797 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc3bc0a29a0 msgr2=0x7fc3bc0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:02.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.977+0000 7fc3d8654700 1 --2- 172.21.15.169:0/4207211797 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc3bc0a29a0 0x7fc3bc0a4e60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0199f20 tx=0x7fc3c0009260 comp rx=0 tx=0).stop 2024-03-28T00:56:02.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.977+0000 7fc3d8654700 1 -- 172.21.15.169:0/4207211797 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc3d0101090 msgr2=0x7fc3d019e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:02.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.977+0000 7fc3d8654700 1 --2- 172.21.15.169:0/4207211797 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc3d0101090 0x7fc3d019e690 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc00e9a0 tx=0x7fc3cc00ecb0 comp rx=0 tx=0).stop 2024-03-28T00:56:02.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.977+0000 7fc3d8654700 1 -- 172.21.15.169:0/4207211797 shutdown_connections 2024-03-28T00:56:02.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.977+0000 7fc3d8654700 1 --2- 172.21.15.169:0/4207211797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3d01070d0 0x7fc3d0198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:02.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.977+0000 7fc3d8654700 1 --2- 172.21.15.169:0/4207211797 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc3d0101090 0x7fc3d019e690 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:02.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.977+0000 7fc3d8654700 1 --2- 172.21.15.169:0/4207211797 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc3bc0a29a0 0x7fc3bc0a4e60 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:02.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.977+0000 7fc3d8654700 1 --2- 172.21.15.169:0/4207211797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3d0101a30 0x7fc3d019ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:02.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.977+0000 7fc3d8654700 1 -- 172.21.15.169:0/4207211797 >> 172.21.15.169:0/4207211797 conn(0x7fc3d0077fc0 msgr2=0x7fc3d0100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:02.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.977+0000 7fc3d8654700 1 -- 172.21.15.169:0/4207211797 shutdown_connections 2024-03-28T00:56:02.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:02.977+0000 7fc3d8654700 1 -- 172.21.15.169:0/4207211797 wait complete. 2024-03-28T00:56:02.985 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-28T00:56:03.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:03 smithi169 ceph-mon[32274]: pgmap v335: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T00:56:03.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:03 smithi174 ceph-mon[36060]: pgmap v335: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T00:56:03.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:03 smithi177 ceph-mon[36206]: pgmap v335: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T00:56:03.675 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-03-28T00:56:02.104194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661347,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":552,"num_read_kb":393,"num_write":343,"num_write_kb":3057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279,"ondisk_log_size":279,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357384,"kb_used_data":12320,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584440,"statfs":{"total":1151940427776,"available":1151574466560,"internally_reserved":0,"allocated":12615680,"data_stored":8082635,"data_compressed":390594,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003603"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.561655+0000","last_change":"2024-03-28T00:54:32.809951+0000","last_active":"2024-03-28T00:55:37.561655+0000","last_peered":"2024-03-28T00:55:37.561655+0000","last_clean":"2024-03-28T00:55:37.561655+0000","last_became_active":"2024-03-28T00:54:32.808871+0000","last_became_peered":"2024-03-28T00:54:32.808871+0000","last_unstale":"2024-03-28T00:55:37.561655+0000","last_undegraded":"2024-03-28T00:55:37.561655+0000","last_fullsized":"2024-03-28T00:55:37.561655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:15:17.152487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.565720+0000","last_change":"2024-03-28T00:54:32.816707+0000","last_active":"2024-03-28T00:55:37.565720+0000","last_peered":"2024-03-28T00:55:37.565720+0000","last_clean":"2024-03-28T00:55:37.565720+0000","last_became_active":"2024-03-28T00:54:32.816000+0000","last_became_peered":"2024-03-28T00:54:32.816000+0000","last_unstale":"2024-03-28T00:55:37.565720+0000","last_undegraded":"2024-03-28T00:55:37.565720+0000","last_fullsized":"2024-03-28T00:55:37.565720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:56:19.532808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.565389+0000","last_change":"2024-03-28T00:54:32.817023+0000","last_active":"2024-03-28T00:55:37.565389+0000","last_peered":"2024-03-28T00:55:37.565389+0000","last_clean":"2024-03-28T00:55:37.565389+0000","last_became_active":"2024-03-28T00:54:32.816712+0000","last_became_peered":"2024-03-28T00:54:32.816712+0000","last_unstale":"2024-03-28T00:55:37.565389+0000","last_undegraded":"2024-03-28T00:55:37.565389+0000","last_fullsized":"2024-03-28T00:55:37.565389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:13:25.334294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559693+0000","last_change":"2024-03-28T00:54:32.807259+0000","last_active":"2024-03-28T00:55:37.559693+0000","last_peered":"2024-03-28T00:55:37.559693+0000","last_clean":"2024-03-28T00:55:37.559693+0000","last_became_active":"2024-03-28T00:54:32.806944+0000","last_became_peered":"2024-03-28T00:54:32.806944+0000","last_unstale":"2024-03-28T00:55:37.559693+0000","last_undegraded":"2024-03-28T00:55:37.559693+0000","last_fullsized":"2024-03-28T00:55:37.559693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:46:04.852351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559260+0000","last_change":"2024-03-28T00:54:32.807613+0000","last_active":"2024-03-28T00:55:37.559260+0000","last_peered":"2024-03-28T00:55:37.559260+0000","last_clean":"2024-03-28T00:55:37.559260+0000","last_became_active":"2024-03-28T00:54:32.807280+0000","last_became_peered":"2024-03-28T00:54:32.807280+0000","last_unstale":"2024-03-28T00:55:37.559260+0000","last_undegraded":"2024-03-28T00:55:37.559260+0000","last_fullsized":"2024-03-28T00:55:37.559260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:56:37.133675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917502+0000","last_change":"2024-03-28T00:54:32.806804+0000","last_active":"2024-03-28T00:55:37.917502+0000","last_peered":"2024-03-28T00:55:37.917502+0000","last_clean":"2024-03-28T00:55:37.917502+0000","last_became_active":"2024-03-28T00:54:32.805343+0000","last_became_peered":"2024-03-28T00:54:32.805343+0000","last_unstale":"2024-03-28T00:55:37.917502+0000","last_undegraded":"2024-03-28T00:55:37.917502+0000","last_fullsized":"2024-03-28T00:55:37.917502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:31:44.849926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563371+0000","last_change":"2024-03-28T00:55:37.563371+0000","last_active":"2024-03-28T00:55:37.563371+0000","last_peered":"2024-03-28T00:55:37.563371+0000","last_clean":"2024-03-28T00:55:37.563371+0000","last_became_active":"2024-03-28T00:55:37.563000+0000","last_became_peered":"2024-03-28T00:55:37.563000+0000","last_unstale":"2024-03-28T00:55:37.563371+0000","last_undegraded":"2024-03-28T00:55:37.563371+0000","last_fullsized":"2024-03-28T00:55:37.563371+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:56:58.782502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917264+0000","last_change":"2024-03-28T00:54:32.808058+0000","last_active":"2024-03-28T00:55:37.917264+0000","last_peered":"2024-03-28T00:55:37.917264+0000","last_clean":"2024-03-28T00:55:37.917264+0000","last_became_active":"2024-03-28T00:54:32.807755+0000","last_became_peered":"2024-03-28T00:54:32.807755+0000","last_unstale":"2024-03-28T00:55:37.917264+0000","last_undegraded":"2024-03-28T00:55:37.917264+0000","last_fullsized":"2024-03-28T00:55:37.917264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:08:42.707015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.564001+0000","last_change":"2024-03-28T00:54:32.800667+0000","last_active":"2024-03-28T00:55:37.564001+0000","last_peered":"2024-03-28T00:55:37.564001+0000","last_clean":"2024-03-28T00:55:37.564001+0000","last_became_active":"2024-03-28T00:54:32.800332+0000","last_became_peered":"2024-03-28T00:54:32.800332+0000","last_unstale":"2024-03-28T00:55:37.564001+0000","last_undegraded":"2024-03-28T00:55:37.564001+0000","last_fullsized":"2024-03-28T00:55:37.564001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:41:27.577871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917751+0000","last_change":"2024-03-28T00:54:32.807958+0000","last_active":"2024-03-28T00:55:37.917751+0000","last_peered":"2024-03-28T00:55:37.917751+0000","last_clean":"2024-03-28T00:55:37.917751+0000","last_became_active":"2024-03-28T00:54:32.807657+0000","last_became_peered":"2024-03-28T00:54:32.807657+0000","last_unstale":"2024-03-28T00:55:37.917751+0000","last_undegraded":"2024-03-28T00:55:37.917751+0000","last_fullsized":"2024-03-28T00:55:37.917751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:30:28.000690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560698+0000","last_change":"2024-03-28T00:54:32.811543+0000","last_active":"2024-03-28T00:55:37.560698+0000","last_peered":"2024-03-28T00:55:37.560698+0000","last_clean":"2024-03-28T00:55:37.560698+0000","last_became_active":"2024-03-28T00:54:32.811210+0000","last_became_peered":"2024-03-28T00:54:32.811210+0000","last_unstale":"2024-03-28T00:55:37.560698+0000","last_undegraded":"2024-03-28T00:55:37.560698+0000","last_fullsized":"2024-03-28T00:55:37.560698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:47:19.539107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559659+0000","last_change":"2024-03-28T00:54:32.812648+0000","last_active":"2024-03-28T00:55:37.559659+0000","last_peered":"2024-03-28T00:55:37.559659+0000","last_clean":"2024-03-28T00:55:37.559659+0000","last_became_active":"2024-03-28T00:54:32.812340+0000","last_became_peered":"2024-03-28T00:54:32.812340+0000","last_unstale":"2024-03-28T00:55:37.559659+0000","last_undegraded":"2024-03-28T00:55:37.559659+0000","last_fullsized":"2024-03-28T00:55:37.559659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:20:45.449953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918534+0000","last_change":"2024-03-28T00:54:32.801807+0000","last_active":"2024-03-28T00:55:37.918534+0000","last_peered":"2024-03-28T00:55:37.918534+0000","last_clean":"2024-03-28T00:55:37.918534+0000","last_became_active":"2024-03-28T00:54:32.801498+0000","last_became_peered":"2024-03-28T00:54:32.801498+0000","last_unstale":"2024-03-28T00:55:37.918534+0000","last_undegraded":"2024-03-28T00:55:37.918534+0000","last_fullsized":"2024-03-28T00:55:37.918534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:35:35.827856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.565160+0000","last_change":"2024-03-28T00:54:32.818033+0000","last_active":"2024-03-28T00:55:37.565160+0000","last_peered":"2024-03-28T00:55:37.565160+0000","last_clean":"2024-03-28T00:55:37.565160+0000","last_became_active":"2024-03-28T00:54:32.817309+0000","last_became_peered":"2024-03-28T00:54:32.817309+0000","last_unstale":"2024-03-28T00:55:37.565160+0000","last_undegraded":"2024-03-28T00:55:37.565160+0000","last_fullsized":"2024-03-28T00:55:37.565160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:39:42.682780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.565411+0000","last_change":"2024-03-28T00:54:32.815224+0000","last_active":"2024-03-28T00:55:37.565411+0000","last_peered":"2024-03-28T00:55:37.565411+0000","last_clean":"2024-03-28T00:55:37.565411+0000","last_became_active":"2024-03-28T00:54:32.814936+0000","last_became_peered":"2024-03-28T00:54:32.814936+0000","last_unstale":"2024-03-28T00:55:37.565411+0000","last_undegraded":"2024-03-28T00:55:37.565411+0000","last_fullsized":"2024-03-28T00:55:37.565411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:08:40.200393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563117+0000","last_change":"2024-03-28T00:54:32.807342+0000","last_active":"2024-03-28T00:55:37.563117+0000","last_peered":"2024-03-28T00:55:37.563117+0000","last_clean":"2024-03-28T00:55:37.563117+0000","last_became_active":"2024-03-28T00:54:32.806978+0000","last_became_peered":"2024-03-28T00:54:32.806978+0000","last_unstale":"2024-03-28T00:55:37.563117+0000","last_undegraded":"2024-03-28T00:55:37.563117+0000","last_fullsized":"2024-03-28T00:55:37.563117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:17:58.599693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.561239+0000","last_change":"2024-03-28T00:54:32.810036+0000","last_active":"2024-03-28T00:55:37.561239+0000","last_peered":"2024-03-28T00:55:37.561239+0000","last_clean":"2024-03-28T00:55:37.561239+0000","last_became_active":"2024-03-28T00:54:32.809438+0000","last_became_peered":"2024-03-28T00:54:32.809438+0000","last_unstale":"2024-03-28T00:55:37.561239+0000","last_undegraded":"2024-03-28T00:55:37.561239+0000","last_fullsized":"2024-03-28T00:55:37.561239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:18:50.447474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.558744+0000","last_change":"2024-03-28T00:54:32.815398+0000","last_active":"2024-03-28T00:55:37.558744+0000","last_peered":"2024-03-28T00:55:37.558744+0000","last_clean":"2024-03-28T00:55:37.558744+0000","last_became_active":"2024-03-28T00:54:32.815045+0000","last_became_peered":"2024-03-28T00:54:32.815045+0000","last_unstale":"2024-03-28T00:55:37.558744+0000","last_undegraded":"2024-03-28T00:55:37.558744+0000","last_fullsized":"2024-03-28T00:55:37.558744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:40:29.060217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563241+0000","last_change":"2024-03-28T00:54:32.805736+0000","last_active":"2024-03-28T00:55:37.563241+0000","last_peered":"2024-03-28T00:55:37.563241+0000","last_clean":"2024-03-28T00:55:37.563241+0000","last_became_active":"2024-03-28T00:54:32.805435+0000","last_became_peered":"2024-03-28T00:54:32.805435+0000","last_unstale":"2024-03-28T00:55:37.563241+0000","last_undegraded":"2024-03-28T00:55:37.563241+0000","last_fullsized":"2024-03-28T00:55:37.563241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:24:30.893583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563132+0000","last_change":"2024-03-28T00:54:36.824531+0000","last_active":"2024-03-28T00:55:37.563132+0000","last_peered":"2024-03-28T00:55:37.563132+0000","last_clean":"2024-03-28T00:55:37.563132+0000","last_became_active":"2024-03-28T00:54:36.824123+0000","last_became_peered":"2024-03-28T00:54:36.824123+0000","last_unstale":"2024-03-28T00:55:37.563132+0000","last_undegraded":"2024-03-28T00:55:37.563132+0000","last_fullsized":"2024-03-28T00:55:37.563132+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:10:47.503171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.558883+0000","last_change":"2024-03-28T00:54:32.256226+0000","last_active":"2024-03-28T00:55:37.558883+0000","last_peered":"2024-03-28T00:55:37.558883+0000","last_clean":"2024-03-28T00:55:37.558883+0000","last_became_active":"2024-03-28T00:54:32.255921+0000","last_became_peered":"2024-03-28T00:54:32.255921+0000","last_unstale":"2024-03-28T00:55:37.558883+0000","last_undegraded":"2024-03-28T00:55:37.558883+0000","last_fullsized":"2024-03-28T00:55:37.558883+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:56:04.196311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918062+0000","last_change":"2024-03-28T00:54:32.808248+0000","last_active":"2024-03-28T00:55:37.918062+0000","last_peered":"2024-03-28T00:55:37.918062+0000","last_clean":"2024-03-28T00:55:37.918062+0000","last_became_active":"2024-03-28T00:54:32.807923+0000","last_became_peered":"2024-03-28T00:54:32.807923+0000","last_unstale":"2024-03-28T00:55:37.918062+0000","last_undegraded":"2024-03-28T00:55:37.918062+0000","last_fullsized":"2024-03-28T00:55:37.918062+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:59:56.002564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559108+0000","last_change":"2024-03-28T00:54:36.830556+0000","last_active":"2024-03-28T00:55:37.559108+0000","last_peered":"2024-03-28T00:55:37.559108+0000","last_clean":"2024-03-28T00:55:37.559108+0000","last_became_active":"2024-03-28T00:54:36.830151+0000","last_became_peered":"2024-03-28T00:54:36.830151+0000","last_unstale":"2024-03-28T00:55:37.559108+0000","last_undegraded":"2024-03-28T00:55:37.559108+0000","last_fullsized":"2024-03-28T00:55:37.559108+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:19:23.073410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.919021+0000","last_change":"2024-03-28T00:54:32.260635+0000","last_active":"2024-03-28T00:55:37.919021+0000","last_peered":"2024-03-28T00:55:37.919021+0000","last_clean":"2024-03-28T00:55:37.919021+0000","last_became_active":"2024-03-28T00:54:32.260084+0000","last_became_peered":"2024-03-28T00:54:32.260084+0000","last_unstale":"2024-03-28T00:55:37.919021+0000","last_undegraded":"2024-03-28T00:55:37.919021+0000","last_fullsized":"2024-03-28T00:55:37.919021+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:06.858519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.565721+0000","last_change":"2024-03-28T00:54:32.809901+0000","last_active":"2024-03-28T00:55:37.565721+0000","last_peered":"2024-03-28T00:55:37.565721+0000","last_clean":"2024-03-28T00:55:37.565721+0000","last_became_active":"2024-03-28T00:54:32.809557+0000","last_became_peered":"2024-03-28T00:54:32.809557+0000","last_unstale":"2024-03-28T00:55:37.565721+0000","last_undegraded":"2024-03-28T00:55:37.565721+0000","last_fullsized":"2024-03-28T00:55:37.565721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:05:31.363487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559387+0000","last_change":"2024-03-28T00:54:36.824661+0000","last_active":"2024-03-28T00:55:37.559387+0000","last_peered":"2024-03-28T00:55:37.559387+0000","last_clean":"2024-03-28T00:55:37.559387+0000","last_became_active":"2024-03-28T00:54:36.824223+0000","last_became_peered":"2024-03-28T00:54:36.824223+0000","last_unstale":"2024-03-28T00:55:37.559387+0000","last_undegraded":"2024-03-28T00:55:37.559387+0000","last_fullsized":"2024-03-28T00:55:37.559387+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:02:25.509248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559890+0000","last_change":"2024-03-28T00:54:32.256436+0000","last_active":"2024-03-28T00:55:37.559890+0000","last_peered":"2024-03-28T00:55:37.559890+0000","last_clean":"2024-03-28T00:55:37.559890+0000","last_became_active":"2024-03-28T00:54:32.256069+0000","last_became_peered":"2024-03-28T00:54:32.256069+0000","last_unstale":"2024-03-28T00:55:37.559890+0000","last_undegraded":"2024-03-28T00:55:37.559890+0000","last_fullsized":"2024-03-28T00:55:37.559890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:43.688192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563466+0000","last_change":"2024-03-28T00:54:32.799269+0000","last_active":"2024-03-28T00:55:37.563466+0000","last_peered":"2024-03-28T00:55:37.563466+0000","last_clean":"2024-03-28T00:55:37.563466+0000","last_became_active":"2024-03-28T00:54:32.798848+0000","last_became_peered":"2024-03-28T00:54:32.798848+0000","last_unstale":"2024-03-28T00:55:37.563466+0000","last_undegraded":"2024-03-28T00:55:37.563466+0000","last_fullsized":"2024-03-28T00:55:37.563466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:56:20.318466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559246+0000","last_change":"2024-03-28T00:54:36.823153+0000","last_active":"2024-03-28T00:55:37.559246+0000","last_peered":"2024-03-28T00:55:37.559246+0000","last_clean":"2024-03-28T00:55:37.559246+0000","last_became_active":"2024-03-28T00:54:36.822817+0000","last_became_peered":"2024-03-28T00:54:36.822817+0000","last_unstale":"2024-03-28T00:55:37.559246+0000","last_undegraded":"2024-03-28T00:55:37.559246+0000","last_fullsized":"2024-03-28T00:55:37.559246+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:20:23.001599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559731+0000","last_change":"2024-03-28T00:54:32.250439+0000","last_active":"2024-03-28T00:55:37.559731+0000","last_peered":"2024-03-28T00:55:37.559731+0000","last_clean":"2024-03-28T00:55:37.559731+0000","last_became_active":"2024-03-28T00:54:32.250006+0000","last_became_peered":"2024-03-28T00:54:32.250006+0000","last_unstale":"2024-03-28T00:55:37.559731+0000","last_undegraded":"2024-03-28T00:55:37.559731+0000","last_fullsized":"2024-03-28T00:55:37.559731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:31:38.957429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.567916+0000","last_change":"2024-03-28T00:54:32.806888+0000","last_active":"2024-03-28T00:55:37.567916+0000","last_peered":"2024-03-28T00:55:37.567916+0000","last_clean":"2024-03-28T00:55:37.567916+0000","last_became_active":"2024-03-28T00:54:32.806581+0000","last_became_peered":"2024-03-28T00:54:32.806581+0000","last_unstale":"2024-03-28T00:55:37.567916+0000","last_undegraded":"2024-03-28T00:55:37.567916+0000","last_fullsized":"2024-03-28T00:55:37.567916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:25:52.331819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563729+0000","last_change":"2024-03-28T00:54:36.824041+0000","last_active":"2024-03-28T00:55:37.563729+0000","last_peered":"2024-03-28T00:55:37.563729+0000","last_clean":"2024-03-28T00:55:37.563729+0000","last_became_active":"2024-03-28T00:54:36.823658+0000","last_became_peered":"2024-03-28T00:54:36.823658+0000","last_unstale":"2024-03-28T00:55:37.563729+0000","last_undegraded":"2024-03-28T00:55:37.563729+0000","last_fullsized":"2024-03-28T00:55:37.563729+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:11:03.236009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":107,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:59.350361+0000","last_change":"2024-03-28T00:54:32.257534+0000","last_active":"2024-03-28T00:55:59.350361+0000","last_peered":"2024-03-28T00:55:59.350361+0000","last_clean":"2024-03-28T00:55:59.350361+0000","last_became_active":"2024-03-28T00:54:32.256404+0000","last_became_peered":"2024-03-28T00:54:32.256404+0000","last_unstale":"2024-03-28T00:55:59.350361+0000","last_undegraded":"2024-03-28T00:55:59.350361+0000","last_fullsized":"2024-03-28T00:55:59.350361+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:39:38.329799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559601+0000","last_change":"2024-03-28T00:54:32.806099+0000","last_active":"2024-03-28T00:55:37.559601+0000","last_peered":"2024-03-28T00:55:37.559601+0000","last_clean":"2024-03-28T00:55:37.559601+0000","last_became_active":"2024-03-28T00:54:32.805811+0000","last_became_peered":"2024-03-28T00:54:32.805811+0000","last_unstale":"2024-03-28T00:55:37.559601+0000","last_undegraded":"2024-03-28T00:55:37.559601+0000","last_fullsized":"2024-03-28T00:55:37.559601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:42:32.820245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.562926+0000","last_change":"2024-03-28T00:54:36.827198+0000","last_active":"2024-03-28T00:55:37.562926+0000","last_peered":"2024-03-28T00:55:37.562926+0000","last_clean":"2024-03-28T00:55:37.562926+0000","last_became_active":"2024-03-28T00:54:36.826804+0000","last_became_peered":"2024-03-28T00:54:36.826804+0000","last_unstale":"2024-03-28T00:55:37.562926+0000","last_undegraded":"2024-03-28T00:55:37.562926+0000","last_fullsized":"2024-03-28T00:55:37.562926+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:38:11.030404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917395+0000","last_change":"2024-03-28T00:54:32.249394+0000","last_active":"2024-03-28T00:55:37.917395+0000","last_peered":"2024-03-28T00:55:37.917395+0000","last_clean":"2024-03-28T00:55:37.917395+0000","last_became_active":"2024-03-28T00:54:32.249089+0000","last_became_peered":"2024-03-28T00:54:32.249089+0000","last_unstale":"2024-03-28T00:55:37.917395+0000","last_undegraded":"2024-03-28T00:55:37.917395+0000","last_fullsized":"2024-03-28T00:55:37.917395+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:05:01.720053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.568653+0000","last_change":"2024-03-28T00:54:32.809133+0000","last_active":"2024-03-28T00:55:37.568653+0000","last_peered":"2024-03-28T00:55:37.568653+0000","last_clean":"2024-03-28T00:55:37.568653+0000","last_became_active":"2024-03-28T00:54:32.808841+0000","last_became_peered":"2024-03-28T00:54:32.808841+0000","last_unstale":"2024-03-28T00:55:37.568653+0000","last_undegraded":"2024-03-28T00:55:37.568653+0000","last_fullsized":"2024-03-28T00:55:37.568653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:19:37.859542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.561131+0000","last_change":"2024-03-28T00:54:36.824483+0000","last_active":"2024-03-28T00:55:37.561131+0000","last_peered":"2024-03-28T00:55:37.561131+0000","last_clean":"2024-03-28T00:55:37.561131+0000","last_became_active":"2024-03-28T00:54:36.823973+0000","last_became_peered":"2024-03-28T00:54:36.823973+0000","last_unstale":"2024-03-28T00:55:37.561131+0000","last_undegraded":"2024-03-28T00:55:37.561131+0000","last_fullsized":"2024-03-28T00:55:37.561131+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:01:06.251119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918137+0000","last_change":"2024-03-28T00:54:32.251287+0000","last_active":"2024-03-28T00:55:37.918137+0000","last_peered":"2024-03-28T00:55:37.918137+0000","last_clean":"2024-03-28T00:55:37.918137+0000","last_became_active":"2024-03-28T00:54:32.250904+0000","last_became_peered":"2024-03-28T00:54:32.250904+0000","last_unstale":"2024-03-28T00:55:37.918137+0000","last_undegraded":"2024-03-28T00:55:37.918137+0000","last_fullsized":"2024-03-28T00:55:37.918137+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:18:09.051056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917473+0000","last_change":"2024-03-28T00:54:32.803124+0000","last_active":"2024-03-28T00:55:37.917473+0000","last_peered":"2024-03-28T00:55:37.917473+0000","last_clean":"2024-03-28T00:55:37.917473+0000","last_became_active":"2024-03-28T00:54:32.802825+0000","last_became_peered":"2024-03-28T00:54:32.802825+0000","last_unstale":"2024-03-28T00:55:37.917473+0000","last_undegraded":"2024-03-28T00:55:37.917473+0000","last_fullsized":"2024-03-28T00:55:37.917473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:34:21.825314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.565481+0000","last_change":"2024-03-28T00:54:36.832151+0000","last_active":"2024-03-28T00:55:37.565481+0000","last_peered":"2024-03-28T00:55:37.565481+0000","last_clean":"2024-03-28T00:55:37.565481+0000","last_became_active":"2024-03-28T00:54:36.831748+0000","last_became_peered":"2024-03-28T00:54:36.831748+0000","last_unstale":"2024-03-28T00:55:37.565481+0000","last_undegraded":"2024-03-28T00:55:37.565481+0000","last_fullsized":"2024-03-28T00:55:37.565481+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:45:32.284386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.566817+0000","last_change":"2024-03-28T00:54:32.259048+0000","last_active":"2024-03-28T00:55:37.566817+0000","last_peered":"2024-03-28T00:55:37.566817+0000","last_clean":"2024-03-28T00:55:37.566817+0000","last_became_active":"2024-03-28T00:54:32.258038+0000","last_became_peered":"2024-03-28T00:54:32.258038+0000","last_unstale":"2024-03-28T00:55:37.566817+0000","last_undegraded":"2024-03-28T00:55:37.566817+0000","last_fullsized":"2024-03-28T00:55:37.566817+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:45:23.525516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.558384+0000","last_change":"2024-03-28T00:54:32.809771+0000","last_active":"2024-03-28T00:55:37.558384+0000","last_peered":"2024-03-28T00:55:37.558384+0000","last_clean":"2024-03-28T00:55:37.558384+0000","last_became_active":"2024-03-28T00:54:32.809467+0000","last_became_peered":"2024-03-28T00:54:32.809467+0000","last_unstale":"2024-03-28T00:55:37.558384+0000","last_undegraded":"2024-03-28T00:55:37.558384+0000","last_fullsized":"2024-03-28T00:55:37.558384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:30:15.440202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917482+0000","last_change":"2024-03-28T00:54:36.823313+0000","last_active":"2024-03-28T00:55:37.917482+0000","last_peered":"2024-03-28T00:55:37.917482+0000","last_clean":"2024-03-28T00:55:37.917482+0000","last_became_active":"2024-03-28T00:54:36.822923+0000","last_became_peered":"2024-03-28T00:54:36.822923+0000","last_unstale":"2024-03-28T00:55:37.917482+0000","last_undegraded":"2024-03-28T00:55:37.917482+0000","last_fullsized":"2024-03-28T00:55:37.917482+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:22:21.125028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":97,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:55.346085+0000","last_change":"2024-03-28T00:54:32.255084+0000","last_active":"2024-03-28T00:55:55.346085+0000","last_peered":"2024-03-28T00:55:55.346085+0000","last_clean":"2024-03-28T00:55:55.346085+0000","last_became_active":"2024-03-28T00:54:32.254744+0000","last_became_peered":"2024-03-28T00:54:32.254744+0000","last_unstale":"2024-03-28T00:55:55.346085+0000","last_undegraded":"2024-03-28T00:55:55.346085+0000","last_fullsized":"2024-03-28T00:55:55.346085+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:32:03.906826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":65,"num_read_kb":65,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563748+0000","last_change":"2024-03-28T00:54:32.810050+0000","last_active":"2024-03-28T00:55:37.563748+0000","last_peered":"2024-03-28T00:55:37.563748+0000","last_clean":"2024-03-28T00:55:37.563748+0000","last_became_active":"2024-03-28T00:54:32.809731+0000","last_became_peered":"2024-03-28T00:54:32.809731+0000","last_unstale":"2024-03-28T00:55:37.563748+0000","last_undegraded":"2024-03-28T00:55:37.563748+0000","last_fullsized":"2024-03-28T00:55:37.563748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:20:11.876731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.567437+0000","last_change":"2024-03-28T00:55:37.566900+0000","last_active":"2024-03-28T00:55:37.567437+0000","last_peered":"2024-03-28T00:55:37.567437+0000","last_clean":"2024-03-28T00:55:37.567437+0000","last_became_active":"2024-03-28T00:55:37.566491+0000","last_became_peered":"2024-03-28T00:55:37.566491+0000","last_unstale":"2024-03-28T00:55:37.567437+0000","last_undegraded":"2024-03-28T00:55:37.567437+0000","last_fullsized":"2024-03-28T00:55:37.567437+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:31:11.489908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560419+0000","last_change":"2024-03-28T00:54:32.259159+0000","last_active":"2024-03-28T00:55:37.560419+0000","last_peered":"2024-03-28T00:55:37.560419+0000","last_clean":"2024-03-28T00:55:37.560419+0000","last_became_active":"2024-03-28T00:54:32.258708+0000","last_became_peered":"2024-03-28T00:54:32.258708+0000","last_unstale":"2024-03-28T00:55:37.560419+0000","last_undegraded":"2024-03-28T00:55:37.560419+0000","last_fullsized":"2024-03-28T00:55:37.560419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:37:01.524878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.565610+0000","last_change":"2024-03-28T00:54:32.814358+0000","last_active":"2024-03-28T00:55:37.565610+0000","last_peered":"2024-03-28T00:55:37.565610+0000","last_clean":"2024-03-28T00:55:37.565610+0000","last_became_active":"2024-03-28T00:54:32.814018+0000","last_became_peered":"2024-03-28T00:54:32.814018+0000","last_unstale":"2024-03-28T00:55:37.565610+0000","last_undegraded":"2024-03-28T00:55:37.565610+0000","last_fullsized":"2024-03-28T00:55:37.565610+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:24:38.287853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559495+0000","last_change":"2024-03-28T00:54:36.822161+0000","last_active":"2024-03-28T00:55:37.559495+0000","last_peered":"2024-03-28T00:55:37.559495+0000","last_clean":"2024-03-28T00:55:37.559495+0000","last_became_active":"2024-03-28T00:54:36.821750+0000","last_became_peered":"2024-03-28T00:54:36.821750+0000","last_unstale":"2024-03-28T00:55:37.559495+0000","last_undegraded":"2024-03-28T00:55:37.559495+0000","last_fullsized":"2024-03-28T00:55:37.559495+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:13:36.892453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.567834+0000","last_change":"2024-03-28T00:54:32.257038+0000","last_active":"2024-03-28T00:55:37.567834+0000","last_peered":"2024-03-28T00:55:37.567834+0000","last_clean":"2024-03-28T00:55:37.567834+0000","last_became_active":"2024-03-28T00:54:32.256702+0000","last_became_peered":"2024-03-28T00:54:32.256702+0000","last_unstale":"2024-03-28T00:55:37.567834+0000","last_undegraded":"2024-03-28T00:55:37.567834+0000","last_fullsized":"2024-03-28T00:55:37.567834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:35:17.294116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563608+0000","last_change":"2024-03-28T00:54:32.813514+0000","last_active":"2024-03-28T00:55:37.563608+0000","last_peered":"2024-03-28T00:55:37.563608+0000","last_clean":"2024-03-28T00:55:37.563608+0000","last_became_active":"2024-03-28T00:54:32.813192+0000","last_became_peered":"2024-03-28T00:54:32.813192+0000","last_unstale":"2024-03-28T00:55:37.563608+0000","last_undegraded":"2024-03-28T00:55:37.563608+0000","last_fullsized":"2024-03-28T00:55:37.563608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:55:03.410818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.558388+0000","last_change":"2024-03-28T00:54:36.820995+0000","last_active":"2024-03-28T00:55:37.558388+0000","last_peered":"2024-03-28T00:55:37.558388+0000","last_clean":"2024-03-28T00:55:37.558388+0000","last_became_active":"2024-03-28T00:54:36.820628+0000","last_became_peered":"2024-03-28T00:54:36.820628+0000","last_unstale":"2024-03-28T00:55:37.558388+0000","last_undegraded":"2024-03-28T00:55:37.558388+0000","last_fullsized":"2024-03-28T00:55:37.558388+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:18:58.771527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'34","reported_seq":200,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:56.346944+0000","last_change":"2024-03-28T00:54:32.251092+0000","last_active":"2024-03-28T00:55:56.346944+0000","last_peered":"2024-03-28T00:55:56.346944+0000","last_clean":"2024-03-28T00:55:56.346944+0000","last_became_active":"2024-03-28T00:54:32.250759+0000","last_became_peered":"2024-03-28T00:54:32.250759+0000","last_unstale":"2024-03-28T00:55:56.346944+0000","last_undegraded":"2024-03-28T00:55:56.346944+0000","last_fullsized":"2024-03-28T00:55:56.346944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:57:30.283365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66744,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":137,"num_read_kb":133,"num_write":35,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559613+0000","last_change":"2024-03-28T00:54:32.809580+0000","last_active":"2024-03-28T00:55:37.559613+0000","last_peered":"2024-03-28T00:55:37.559613+0000","last_clean":"2024-03-28T00:55:37.559613+0000","last_became_active":"2024-03-28T00:54:32.809238+0000","last_became_peered":"2024-03-28T00:54:32.809238+0000","last_unstale":"2024-03-28T00:55:37.559613+0000","last_undegraded":"2024-03-28T00:55:37.559613+0000","last_fullsized":"2024-03-28T00:55:37.559613+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:20:17.397594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.568088+0000","last_change":"2024-03-28T00:54:36.826851+0000","last_active":"2024-03-28T00:55:37.568088+0000","last_peered":"2024-03-28T00:55:37.568088+0000","last_clean":"2024-03-28T00:55:37.568088+0000","last_became_active":"2024-03-28T00:54:36.826359+0000","last_became_peered":"2024-03-28T00:54:36.826359+0000","last_unstale":"2024-03-28T00:55:37.568088+0000","last_undegraded":"2024-03-28T00:55:37.568088+0000","last_fullsized":"2024-03-28T00:55:37.568088+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:27:48.857368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.562536+0000","last_change":"2024-03-28T00:54:32.251701+0000","last_active":"2024-03-28T00:55:37.562536+0000","last_peered":"2024-03-28T00:55:37.562536+0000","last_clean":"2024-03-28T00:55:37.562536+0000","last_became_active":"2024-03-28T00:54:32.251403+0000","last_became_peered":"2024-03-28T00:54:32.251403+0000","last_unstale":"2024-03-28T00:55:37.562536+0000","last_undegraded":"2024-03-28T00:55:37.562536+0000","last_fullsized":"2024-03-28T00:55:37.562536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:28:16.127643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918391+0000","last_change":"2024-03-28T00:54:32.806810+0000","last_active":"2024-03-28T00:55:37.918391+0000","last_peered":"2024-03-28T00:55:37.918391+0000","last_clean":"2024-03-28T00:55:37.918391+0000","last_became_active":"2024-03-28T00:54:32.806468+0000","last_became_peered":"2024-03-28T00:54:32.806468+0000","last_unstale":"2024-03-28T00:55:37.918391+0000","last_undegraded":"2024-03-28T00:55:37.918391+0000","last_fullsized":"2024-03-28T00:55:37.918391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:54:16.196032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.919873+0000","last_change":"2024-03-28T00:55:37.919873+0000","last_active":"2024-03-28T00:55:37.919873+0000","last_peered":"2024-03-28T00:55:37.919873+0000","last_clean":"2024-03-28T00:55:37.919873+0000","last_became_active":"2024-03-28T00:55:37.919396+0000","last_became_peered":"2024-03-28T00:55:37.919396+0000","last_unstale":"2024-03-28T00:55:37.919873+0000","last_undegraded":"2024-03-28T00:55:37.919873+0000","last_fullsized":"2024-03-28T00:55:37.919873+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:05:59.524285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.568254+0000","last_change":"2024-03-28T00:54:32.259440+0000","last_active":"2024-03-28T00:55:37.568254+0000","last_peered":"2024-03-28T00:55:37.568254+0000","last_clean":"2024-03-28T00:55:37.568254+0000","last_became_active":"2024-03-28T00:54:32.258225+0000","last_became_peered":"2024-03-28T00:54:32.258225+0000","last_unstale":"2024-03-28T00:55:37.568254+0000","last_undegraded":"2024-03-28T00:55:37.568254+0000","last_fullsized":"2024-03-28T00:55:37.568254+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:07:18.398302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559996+0000","last_change":"2024-03-28T00:54:32.806853+0000","last_active":"2024-03-28T00:55:37.559996+0000","last_peered":"2024-03-28T00:55:37.559996+0000","last_clean":"2024-03-28T00:55:37.559996+0000","last_became_active":"2024-03-28T00:54:32.806551+0000","last_became_peered":"2024-03-28T00:54:32.806551+0000","last_unstale":"2024-03-28T00:55:37.559996+0000","last_undegraded":"2024-03-28T00:55:37.559996+0000","last_fullsized":"2024-03-28T00:55:37.559996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:49:45.774252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.558659+0000","last_change":"2024-03-28T00:54:36.827542+0000","last_active":"2024-03-28T00:55:37.558659+0000","last_peered":"2024-03-28T00:55:37.558659+0000","last_clean":"2024-03-28T00:55:37.558659+0000","last_became_active":"2024-03-28T00:54:36.827163+0000","last_became_peered":"2024-03-28T00:54:36.827163+0000","last_unstale":"2024-03-28T00:55:37.558659+0000","last_undegraded":"2024-03-28T00:55:37.558659+0000","last_fullsized":"2024-03-28T00:55:37.558659+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:36:15.278111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'3","reported_seq":103,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:59.350625+0000","last_change":"2024-03-28T00:54:32.253231+0000","last_active":"2024-03-28T00:55:59.350625+0000","last_peered":"2024-03-28T00:55:59.350625+0000","last_clean":"2024-03-28T00:55:59.350625+0000","last_became_active":"2024-03-28T00:54:32.252731+0000","last_became_peered":"2024-03-28T00:54:32.252731+0000","last_unstale":"2024-03-28T00:55:59.350625+0000","last_undegraded":"2024-03-28T00:55:59.350625+0000","last_fullsized":"2024-03-28T00:55:59.350625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:05:01.716654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560503+0000","last_change":"2024-03-28T00:54:32.805073+0000","last_active":"2024-03-28T00:55:37.560503+0000","last_peered":"2024-03-28T00:55:37.560503+0000","last_clean":"2024-03-28T00:55:37.560503+0000","last_became_active":"2024-03-28T00:54:32.803690+0000","last_became_peered":"2024-03-28T00:54:32.803690+0000","last_unstale":"2024-03-28T00:55:37.560503+0000","last_undegraded":"2024-03-28T00:55:37.560503+0000","last_fullsized":"2024-03-28T00:55:37.560503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:41:16.586168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918498+0000","last_change":"2024-03-28T00:54:36.835463+0000","last_active":"2024-03-28T00:55:37.918498+0000","last_peered":"2024-03-28T00:55:37.918498+0000","last_clean":"2024-03-28T00:55:37.918498+0000","last_became_active":"2024-03-28T00:54:36.834982+0000","last_became_peered":"2024-03-28T00:54:36.834982+0000","last_unstale":"2024-03-28T00:55:37.918498+0000","last_undegraded":"2024-03-28T00:55:37.918498+0000","last_fullsized":"2024-03-28T00:55:37.918498+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:27:32.259586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559446+0000","last_change":"2024-03-28T00:54:32.248461+0000","last_active":"2024-03-28T00:55:37.559446+0000","last_peered":"2024-03-28T00:55:37.559446+0000","last_clean":"2024-03-28T00:55:37.559446+0000","last_became_active":"2024-03-28T00:54:32.248140+0000","last_became_peered":"2024-03-28T00:54:32.248140+0000","last_unstale":"2024-03-28T00:55:37.559446+0000","last_undegraded":"2024-03-28T00:55:37.559446+0000","last_fullsized":"2024-03-28T00:55:37.559446+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:13:38.784288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.567018+0000","last_change":"2024-03-28T00:54:32.806651+0000","last_active":"2024-03-28T00:55:37.567018+0000","last_peered":"2024-03-28T00:55:37.567018+0000","last_clean":"2024-03-28T00:55:37.567018+0000","last_became_active":"2024-03-28T00:54:32.806329+0000","last_became_peered":"2024-03-28T00:54:32.806329+0000","last_unstale":"2024-03-28T00:55:37.567018+0000","last_undegraded":"2024-03-28T00:55:37.567018+0000","last_fullsized":"2024-03-28T00:55:37.567018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:28:32.815396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.566934+0000","last_change":"2024-03-28T00:54:36.827016+0000","last_active":"2024-03-28T00:55:37.566934+0000","last_peered":"2024-03-28T00:55:37.566934+0000","last_clean":"2024-03-28T00:55:37.566934+0000","last_became_active":"2024-03-28T00:54:36.826428+0000","last_became_peered":"2024-03-28T00:54:36.826428+0000","last_unstale":"2024-03-28T00:55:37.566934+0000","last_undegraded":"2024-03-28T00:55:37.566934+0000","last_fullsized":"2024-03-28T00:55:37.566934+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:33:01.593990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.573875+0000","last_change":"2024-03-28T00:55:37.573420+0000","last_active":"2024-03-28T00:55:37.573875+0000","last_peered":"2024-03-28T00:55:37.573875+0000","last_clean":"2024-03-28T00:55:37.573875+0000","last_became_active":"2024-03-28T00:55:37.568657+0000","last_became_peered":"2024-03-28T00:55:37.568657+0000","last_unstale":"2024-03-28T00:55:37.573875+0000","last_undegraded":"2024-03-28T00:55:37.573875+0000","last_fullsized":"2024-03-28T00:55:37.573875+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:03:21.483054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.919698+0000","last_change":"2024-03-28T00:55:37.919698+0000","last_active":"2024-03-28T00:55:37.919698+0000","last_peered":"2024-03-28T00:55:37.919698+0000","last_clean":"2024-03-28T00:55:37.919698+0000","last_became_active":"2024-03-28T00:55:37.919276+0000","last_became_peered":"2024-03-28T00:55:37.919276+0000","last_unstale":"2024-03-28T00:55:37.919698+0000","last_undegraded":"2024-03-28T00:55:37.919698+0000","last_fullsized":"2024-03-28T00:55:37.919698+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:12:39.042250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918600+0000","last_change":"2024-03-28T00:55:37.918600+0000","last_active":"2024-03-28T00:55:37.918600+0000","last_peered":"2024-03-28T00:55:37.918600+0000","last_clean":"2024-03-28T00:55:37.918600+0000","last_became_active":"2024-03-28T00:55:37.918088+0000","last_became_peered":"2024-03-28T00:55:37.918088+0000","last_unstale":"2024-03-28T00:55:37.918600+0000","last_undegraded":"2024-03-28T00:55:37.918600+0000","last_fullsized":"2024-03-28T00:55:37.918600+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:29:23.492261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.558694+0000","last_change":"2024-03-28T00:54:32.264038+0000","last_active":"2024-03-28T00:55:37.558694+0000","last_peered":"2024-03-28T00:55:37.558694+0000","last_clean":"2024-03-28T00:55:37.558694+0000","last_became_active":"2024-03-28T00:54:32.263706+0000","last_became_peered":"2024-03-28T00:54:32.263706+0000","last_unstale":"2024-03-28T00:55:37.558694+0000","last_undegraded":"2024-03-28T00:55:37.558694+0000","last_fullsized":"2024-03-28T00:55:37.558694+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:03.244556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917653+0000","last_change":"2024-03-28T00:54:32.804727+0000","last_active":"2024-03-28T00:55:37.917653+0000","last_peered":"2024-03-28T00:55:37.917653+0000","last_clean":"2024-03-28T00:55:37.917653+0000","last_became_active":"2024-03-28T00:54:32.804422+0000","last_became_peered":"2024-03-28T00:54:32.804422+0000","last_unstale":"2024-03-28T00:55:37.917653+0000","last_undegraded":"2024-03-28T00:55:37.917653+0000","last_fullsized":"2024-03-28T00:55:37.917653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:29:57.983722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918350+0000","last_change":"2024-03-28T00:54:36.833416+0000","last_active":"2024-03-28T00:55:37.918350+0000","last_peered":"2024-03-28T00:55:37.918350+0000","last_clean":"2024-03-28T00:55:37.918350+0000","last_became_active":"2024-03-28T00:54:36.832985+0000","last_became_peered":"2024-03-28T00:54:36.832985+0000","last_unstale":"2024-03-28T00:55:37.918350+0000","last_undegraded":"2024-03-28T00:55:37.918350+0000","last_fullsized":"2024-03-28T00:55:37.918350+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:47:30.376032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560050+0000","last_change":"2024-03-28T00:54:32.256417+0000","last_active":"2024-03-28T00:55:37.560050+0000","last_peered":"2024-03-28T00:55:37.560050+0000","last_clean":"2024-03-28T00:55:37.560050+0000","last_became_active":"2024-03-28T00:54:32.256088+0000","last_became_peered":"2024-03-28T00:54:32.256088+0000","last_unstale":"2024-03-28T00:55:37.560050+0000","last_undegraded":"2024-03-28T00:55:37.560050+0000","last_fullsized":"2024-03-28T00:55:37.560050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:58.661597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917228+0000","last_change":"2024-03-28T00:54:32.817604+0000","last_active":"2024-03-28T00:55:37.917228+0000","last_peered":"2024-03-28T00:55:37.917228+0000","last_clean":"2024-03-28T00:55:37.917228+0000","last_became_active":"2024-03-28T00:54:32.817211+0000","last_became_peered":"2024-03-28T00:54:32.817211+0000","last_unstale":"2024-03-28T00:55:37.917228+0000","last_undegraded":"2024-03-28T00:55:37.917228+0000","last_fullsized":"2024-03-28T00:55:37.917228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:02:28.335075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918570+0000","last_change":"2024-03-28T00:54:37.638968+0000","last_active":"2024-03-28T00:55:37.918570+0000","last_peered":"2024-03-28T00:55:37.918570+0000","last_clean":"2024-03-28T00:55:37.918570+0000","last_became_active":"2024-03-28T00:54:37.638446+0000","last_became_peered":"2024-03-28T00:54:37.638446+0000","last_unstale":"2024-03-28T00:55:37.918570+0000","last_undegraded":"2024-03-28T00:55:37.918570+0000","last_fullsized":"2024-03-28T00:55:37.918570+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:54:56.348760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.567023+0000","last_change":"2024-03-28T00:54:32.809822+0000","last_active":"2024-03-28T00:55:37.567023+0000","last_peered":"2024-03-28T00:55:37.567023+0000","last_clean":"2024-03-28T00:55:37.567023+0000","last_became_active":"2024-03-28T00:54:32.808284+0000","last_became_peered":"2024-03-28T00:54:32.808284+0000","last_unstale":"2024-03-28T00:55:37.567023+0000","last_undegraded":"2024-03-28T00:55:37.567023+0000","last_fullsized":"2024-03-28T00:55:37.567023+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:49:11.651960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":101,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:58.349353+0000","last_change":"2024-03-28T00:54:32.261806+0000","last_active":"2024-03-28T00:55:58.349353+0000","last_peered":"2024-03-28T00:55:58.349353+0000","last_clean":"2024-03-28T00:55:58.349353+0000","last_became_active":"2024-03-28T00:54:32.261493+0000","last_became_peered":"2024-03-28T00:54:32.261493+0000","last_unstale":"2024-03-28T00:55:58.349353+0000","last_undegraded":"2024-03-28T00:55:58.349353+0000","last_fullsized":"2024-03-28T00:55:58.349353+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:36:04.283408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":68,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560561+0000","last_change":"2024-03-28T00:54:36.819332+0000","last_active":"2024-03-28T00:55:37.560561+0000","last_peered":"2024-03-28T00:55:37.560561+0000","last_clean":"2024-03-28T00:55:37.560561+0000","last_became_active":"2024-03-28T00:54:36.818932+0000","last_became_peered":"2024-03-28T00:54:36.818932+0000","last_unstale":"2024-03-28T00:55:37.560561+0000","last_undegraded":"2024-03-28T00:55:37.560561+0000","last_fullsized":"2024-03-28T00:55:37.560561+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:00:39.446738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917573+0000","last_change":"2024-03-28T00:54:32.806093+0000","last_active":"2024-03-28T00:55:37.917573+0000","last_peered":"2024-03-28T00:55:37.917573+0000","last_clean":"2024-03-28T00:55:37.917573+0000","last_became_active":"2024-03-28T00:54:32.805734+0000","last_became_peered":"2024-03-28T00:54:32.805734+0000","last_unstale":"2024-03-28T00:55:37.917573+0000","last_undegraded":"2024-03-28T00:55:37.917573+0000","last_fullsized":"2024-03-28T00:55:37.917573+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:51:41.066225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559416+0000","last_change":"2024-03-28T00:54:32.245182+0000","last_active":"2024-03-28T00:55:37.559416+0000","last_peered":"2024-03-28T00:55:37.559416+0000","last_clean":"2024-03-28T00:55:37.559416+0000","last_became_active":"2024-03-28T00:54:32.244855+0000","last_became_peered":"2024-03-28T00:54:32.244855+0000","last_unstale":"2024-03-28T00:55:37.559416+0000","last_undegraded":"2024-03-28T00:55:37.559416+0000","last_fullsized":"2024-03-28T00:55:37.559416+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:39:09.997199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":351,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560022+0000","last_change":"2024-03-28T00:53:20.355628+0000","last_active":"2024-03-28T00:55:37.560022+0000","last_peered":"2024-03-28T00:55:37.560022+0000","last_clean":"2024-03-28T00:55:37.560022+0000","last_became_active":"2024-03-28T00:53:20.355175+0000","last_became_peered":"2024-03-28T00:53:20.355175+0000","last_unstale":"2024-03-28T00:55:37.560022+0000","last_undegraded":"2024-03-28T00:55:37.560022+0000","last_fullsized":"2024-03-28T00:55:37.560022+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:49:57.054435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:49:57.054435+0000","last_clean_scrub_stamp":"2024-03-28T00:49:57.054435+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:55:46.753799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.569168+0000","last_change":"2024-03-28T00:54:37.639760+0000","last_active":"2024-03-28T00:55:37.569168+0000","last_peered":"2024-03-28T00:55:37.569168+0000","last_clean":"2024-03-28T00:55:37.569168+0000","last_became_active":"2024-03-28T00:54:37.639287+0000","last_became_peered":"2024-03-28T00:54:37.639287+0000","last_unstale":"2024-03-28T00:55:37.569168+0000","last_undegraded":"2024-03-28T00:55:37.569168+0000","last_fullsized":"2024-03-28T00:55:37.569168+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:50:02.840312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917791+0000","last_change":"2024-03-28T00:54:32.807764+0000","last_active":"2024-03-28T00:55:37.917791+0000","last_peered":"2024-03-28T00:55:37.917791+0000","last_clean":"2024-03-28T00:55:37.917791+0000","last_became_active":"2024-03-28T00:54:32.807441+0000","last_became_peered":"2024-03-28T00:54:32.807441+0000","last_unstale":"2024-03-28T00:55:37.917791+0000","last_undegraded":"2024-03-28T00:55:37.917791+0000","last_fullsized":"2024-03-28T00:55:37.917791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:37:40.540514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560200+0000","last_change":"2024-03-28T00:54:32.259192+0000","last_active":"2024-03-28T00:55:37.560200+0000","last_peered":"2024-03-28T00:55:37.560200+0000","last_clean":"2024-03-28T00:55:37.560200+0000","last_became_active":"2024-03-28T00:54:32.258880+0000","last_became_peered":"2024-03-28T00:54:32.258880+0000","last_unstale":"2024-03-28T00:55:37.560200+0000","last_undegraded":"2024-03-28T00:55:37.560200+0000","last_fullsized":"2024-03-28T00:55:37.560200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:20:11.384633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560440+0000","last_change":"2024-03-28T00:54:36.822268+0000","last_active":"2024-03-28T00:55:37.560440+0000","last_peered":"2024-03-28T00:55:37.560440+0000","last_clean":"2024-03-28T00:55:37.560440+0000","last_became_active":"2024-03-28T00:54:36.821773+0000","last_became_peered":"2024-03-28T00:54:36.821773+0000","last_unstale":"2024-03-28T00:55:37.560440+0000","last_undegraded":"2024-03-28T00:55:37.560440+0000","last_fullsized":"2024-03-28T00:55:37.560440+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:56:11.206975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559722+0000","last_change":"2024-03-28T00:54:32.256133+0000","last_active":"2024-03-28T00:55:37.559722+0000","last_peered":"2024-03-28T00:55:37.559722+0000","last_clean":"2024-03-28T00:55:37.559722+0000","last_became_active":"2024-03-28T00:54:32.250535+0000","last_became_peered":"2024-03-28T00:54:32.250535+0000","last_unstale":"2024-03-28T00:55:37.559722+0000","last_undegraded":"2024-03-28T00:55:37.559722+0000","last_fullsized":"2024-03-28T00:55:37.559722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:18:50.874181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918606+0000","last_change":"2024-03-28T00:54:32.808883+0000","last_active":"2024-03-28T00:55:37.918606+0000","last_peered":"2024-03-28T00:55:37.918606+0000","last_clean":"2024-03-28T00:55:37.918606+0000","last_became_active":"2024-03-28T00:54:32.808587+0000","last_became_peered":"2024-03-28T00:54:32.808587+0000","last_unstale":"2024-03-28T00:55:37.918606+0000","last_undegraded":"2024-03-28T00:55:37.918606+0000","last_fullsized":"2024-03-28T00:55:37.918606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:09:55.164876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917403+0000","last_change":"2024-03-28T00:54:37.637743+0000","last_active":"2024-03-28T00:55:37.917403+0000","last_peered":"2024-03-28T00:55:37.917403+0000","last_clean":"2024-03-28T00:55:37.917403+0000","last_became_active":"2024-03-28T00:54:37.637280+0000","last_became_peered":"2024-03-28T00:54:37.637280+0000","last_unstale":"2024-03-28T00:55:37.917403+0000","last_undegraded":"2024-03-28T00:55:37.917403+0000","last_fullsized":"2024-03-28T00:55:37.917403+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:40:51.431757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'7","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.558852+0000","last_change":"2024-03-28T00:54:32.246444+0000","last_active":"2024-03-28T00:55:37.558852+0000","last_peered":"2024-03-28T00:55:37.558852+0000","last_clean":"2024-03-28T00:55:37.558852+0000","last_became_active":"2024-03-28T00:54:32.245106+0000","last_became_peered":"2024-03-28T00:54:32.245106+0000","last_unstale":"2024-03-28T00:55:37.558852+0000","last_undegraded":"2024-03-28T00:55:37.558852+0000","last_fullsized":"2024-03-28T00:55:37.558852+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:13:45.123625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.919052+0000","last_change":"2024-03-28T00:54:32.812851+0000","last_active":"2024-03-28T00:55:37.919052+0000","last_peered":"2024-03-28T00:55:37.919052+0000","last_clean":"2024-03-28T00:55:37.919052+0000","last_became_active":"2024-03-28T00:54:32.812518+0000","last_became_peered":"2024-03-28T00:54:32.812518+0000","last_unstale":"2024-03-28T00:55:37.919052+0000","last_undegraded":"2024-03-28T00:55:37.919052+0000","last_fullsized":"2024-03-28T00:55:37.919052+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:08:41.518888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.562736+0000","last_change":"2024-03-28T00:54:36.818709+0000","last_active":"2024-03-28T00:55:37.562736+0000","last_peered":"2024-03-28T00:55:37.562736+0000","last_clean":"2024-03-28T00:55:37.562736+0000","last_became_active":"2024-03-28T00:54:36.818304+0000","last_became_peered":"2024-03-28T00:54:36.818304+0000","last_unstale":"2024-03-28T00:55:37.562736+0000","last_undegraded":"2024-03-28T00:55:37.562736+0000","last_fullsized":"2024-03-28T00:55:37.562736+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:03:07.642685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559719+0000","last_change":"2024-03-28T00:54:32.256031+0000","last_active":"2024-03-28T00:55:37.559719+0000","last_peered":"2024-03-28T00:55:37.559719+0000","last_clean":"2024-03-28T00:55:37.559719+0000","last_became_active":"2024-03-28T00:54:32.255597+0000","last_became_peered":"2024-03-28T00:54:32.255597+0000","last_unstale":"2024-03-28T00:55:37.559719+0000","last_undegraded":"2024-03-28T00:55:37.559719+0000","last_fullsized":"2024-03-28T00:55:37.559719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:31:10.837212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.565632+0000","last_change":"2024-03-28T00:54:32.816961+0000","last_active":"2024-03-28T00:55:37.565632+0000","last_peered":"2024-03-28T00:55:37.565632+0000","last_clean":"2024-03-28T00:55:37.565632+0000","last_became_active":"2024-03-28T00:54:32.816408+0000","last_became_peered":"2024-03-28T00:54:32.816408+0000","last_unstale":"2024-03-28T00:55:37.565632+0000","last_undegraded":"2024-03-28T00:55:37.565632+0000","last_fullsized":"2024-03-28T00:55:37.565632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:08:57.441492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918692+0000","last_change":"2024-03-28T00:55:37.918692+0000","last_active":"2024-03-28T00:55:37.918692+0000","last_peered":"2024-03-28T00:55:37.918692+0000","last_clean":"2024-03-28T00:55:37.918692+0000","last_became_active":"2024-03-28T00:55:37.918195+0000","last_became_peered":"2024-03-28T00:55:37.918195+0000","last_unstale":"2024-03-28T00:55:37.918692+0000","last_undegraded":"2024-03-28T00:55:37.918692+0000","last_fullsized":"2024-03-28T00:55:37.918692+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:01:06.012325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.570700+0000","last_change":"2024-03-28T00:55:37.570700+0000","last_active":"2024-03-28T00:55:37.570700+0000","last_peered":"2024-03-28T00:55:37.570700+0000","last_clean":"2024-03-28T00:55:37.570700+0000","last_became_active":"2024-03-28T00:55:37.566298+0000","last_became_peered":"2024-03-28T00:55:37.566298+0000","last_unstale":"2024-03-28T00:55:37.570700+0000","last_undegraded":"2024-03-28T00:55:37.570700+0000","last_fullsized":"2024-03-28T00:55:37.570700+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:46:48.352817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.558967+0000","last_change":"2024-03-28T00:54:32.811592+0000","last_active":"2024-03-28T00:55:37.558967+0000","last_peered":"2024-03-28T00:55:37.558967+0000","last_clean":"2024-03-28T00:55:37.558967+0000","last_became_active":"2024-03-28T00:54:32.811228+0000","last_became_peered":"2024-03-28T00:54:32.811228+0000","last_unstale":"2024-03-28T00:55:37.558967+0000","last_undegraded":"2024-03-28T00:55:37.558967+0000","last_fullsized":"2024-03-28T00:55:37.558967+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:14:19.585237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.558764+0000","last_change":"2024-03-28T00:54:36.822501+0000","last_active":"2024-03-28T00:55:37.558764+0000","last_peered":"2024-03-28T00:55:37.558764+0000","last_clean":"2024-03-28T00:55:37.558764+0000","last_became_active":"2024-03-28T00:54:36.821630+0000","last_became_peered":"2024-03-28T00:54:36.821630+0000","last_unstale":"2024-03-28T00:55:37.558764+0000","last_undegraded":"2024-03-28T00:55:37.558764+0000","last_fullsized":"2024-03-28T00:55:37.558764+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:20:52.247055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917914+0000","last_change":"2024-03-28T00:54:32.247328+0000","last_active":"2024-03-28T00:55:37.917914+0000","last_peered":"2024-03-28T00:55:37.917914+0000","last_clean":"2024-03-28T00:55:37.917914+0000","last_became_active":"2024-03-28T00:54:32.247003+0000","last_became_peered":"2024-03-28T00:54:32.247003+0000","last_unstale":"2024-03-28T00:55:37.917914+0000","last_undegraded":"2024-03-28T00:55:37.917914+0000","last_fullsized":"2024-03-28T00:55:37.917914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:33:09.764946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560789+0000","last_change":"2024-03-28T00:54:32.805977+0000","last_active":"2024-03-28T00:55:37.560789+0000","last_peered":"2024-03-28T00:55:37.560789+0000","last_clean":"2024-03-28T00:55:37.560789+0000","last_became_active":"2024-03-28T00:54:32.805636+0000","last_became_peered":"2024-03-28T00:54:32.805636+0000","last_unstale":"2024-03-28T00:55:37.560789+0000","last_undegraded":"2024-03-28T00:55:37.560789+0000","last_fullsized":"2024-03-28T00:55:37.560789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:04:42.901732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559561+0000","last_change":"2024-03-28T00:54:36.822103+0000","last_active":"2024-03-28T00:55:37.559561+0000","last_peered":"2024-03-28T00:55:37.559561+0000","last_clean":"2024-03-28T00:55:37.559561+0000","last_became_active":"2024-03-28T00:54:36.821695+0000","last_became_peered":"2024-03-28T00:54:36.821695+0000","last_unstale":"2024-03-28T00:55:37.559561+0000","last_undegraded":"2024-03-28T00:55:37.559561+0000","last_fullsized":"2024-03-28T00:55:37.559561+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:45:00.613955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.562670+0000","last_change":"2024-03-28T00:54:32.251523+0000","last_active":"2024-03-28T00:55:37.562670+0000","last_peered":"2024-03-28T00:55:37.562670+0000","last_clean":"2024-03-28T00:55:37.562670+0000","last_became_active":"2024-03-28T00:54:32.251223+0000","last_became_peered":"2024-03-28T00:54:32.251223+0000","last_unstale":"2024-03-28T00:55:37.562670+0000","last_undegraded":"2024-03-28T00:55:37.562670+0000","last_fullsized":"2024-03-28T00:55:37.562670+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:08:20.489848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917787+0000","last_change":"2024-03-28T00:54:32.817641+0000","last_active":"2024-03-28T00:55:37.917787+0000","last_peered":"2024-03-28T00:55:37.917787+0000","last_clean":"2024-03-28T00:55:37.917787+0000","last_became_active":"2024-03-28T00:54:32.817252+0000","last_became_peered":"2024-03-28T00:54:32.817252+0000","last_unstale":"2024-03-28T00:55:37.917787+0000","last_undegraded":"2024-03-28T00:55:37.917787+0000","last_fullsized":"2024-03-28T00:55:37.917787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:54:48.275005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559331+0000","last_change":"2024-03-28T00:54:37.638990+0000","last_active":"2024-03-28T00:55:37.559331+0000","last_peered":"2024-03-28T00:55:37.559331+0000","last_clean":"2024-03-28T00:55:37.559331+0000","last_became_active":"2024-03-28T00:54:37.638538+0000","last_became_peered":"2024-03-28T00:54:37.638538+0000","last_unstale":"2024-03-28T00:55:37.559331+0000","last_undegraded":"2024-03-28T00:55:37.559331+0000","last_fullsized":"2024-03-28T00:55:37.559331+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:28:10.791805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559360+0000","last_change":"2024-03-28T00:54:32.253254+0000","last_active":"2024-03-28T00:55:37.559360+0000","last_peered":"2024-03-28T00:55:37.559360+0000","last_clean":"2024-03-28T00:55:37.559360+0000","last_became_active":"2024-03-28T00:54:32.252879+0000","last_became_peered":"2024-03-28T00:54:32.252879+0000","last_unstale":"2024-03-28T00:55:37.559360+0000","last_undegraded":"2024-03-28T00:55:37.559360+0000","last_fullsized":"2024-03-28T00:55:37.559360+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:59:51.039215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560179+0000","last_change":"2024-03-28T00:54:32.811288+0000","last_active":"2024-03-28T00:55:37.560179+0000","last_peered":"2024-03-28T00:55:37.560179+0000","last_clean":"2024-03-28T00:55:37.560179+0000","last_became_active":"2024-03-28T00:54:32.810968+0000","last_became_peered":"2024-03-28T00:54:32.810968+0000","last_unstale":"2024-03-28T00:55:37.560179+0000","last_undegraded":"2024-03-28T00:55:37.560179+0000","last_fullsized":"2024-03-28T00:55:37.560179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:16:56.113046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563140+0000","last_change":"2024-03-28T00:54:36.830714+0000","last_active":"2024-03-28T00:55:37.563140+0000","last_peered":"2024-03-28T00:55:37.563140+0000","last_clean":"2024-03-28T00:55:37.563140+0000","last_became_active":"2024-03-28T00:54:36.830253+0000","last_became_peered":"2024-03-28T00:54:36.830253+0000","last_unstale":"2024-03-28T00:55:37.563140+0000","last_undegraded":"2024-03-28T00:55:37.563140+0000","last_fullsized":"2024-03-28T00:55:37.563140+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:24:43.696150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563132+0000","last_change":"2024-03-28T00:54:32.254009+0000","last_active":"2024-03-28T00:55:37.563132+0000","last_peered":"2024-03-28T00:55:37.563132+0000","last_clean":"2024-03-28T00:55:37.563132+0000","last_became_active":"2024-03-28T00:54:32.253697+0000","last_became_peered":"2024-03-28T00:54:32.253697+0000","last_unstale":"2024-03-28T00:55:37.563132+0000","last_undegraded":"2024-03-28T00:55:37.563132+0000","last_fullsized":"2024-03-28T00:55:37.563132+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:17:44.629367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560857+0000","last_change":"2024-03-28T00:54:32.812828+0000","last_active":"2024-03-28T00:55:37.560857+0000","last_peered":"2024-03-28T00:55:37.560857+0000","last_clean":"2024-03-28T00:55:37.560857+0000","last_became_active":"2024-03-28T00:54:32.811591+0000","last_became_peered":"2024-03-28T00:54:32.811591+0000","last_unstale":"2024-03-28T00:55:37.560857+0000","last_undegraded":"2024-03-28T00:55:37.560857+0000","last_fullsized":"2024-03-28T00:55:37.560857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:04.613623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.565201+0000","last_change":"2024-03-28T00:54:36.823284+0000","last_active":"2024-03-28T00:55:37.565201+0000","last_peered":"2024-03-28T00:55:37.565201+0000","last_clean":"2024-03-28T00:55:37.565201+0000","last_became_active":"2024-03-28T00:54:36.822883+0000","last_became_peered":"2024-03-28T00:54:36.822883+0000","last_unstale":"2024-03-28T00:55:37.565201+0000","last_undegraded":"2024-03-28T00:55:37.565201+0000","last_fullsized":"2024-03-28T00:55:37.565201+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:37:10.266063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.567943+0000","last_change":"2024-03-28T00:54:32.253946+0000","last_active":"2024-03-28T00:55:37.567943+0000","last_peered":"2024-03-28T00:55:37.567943+0000","last_clean":"2024-03-28T00:55:37.567943+0000","last_became_active":"2024-03-28T00:54:32.253650+0000","last_became_peered":"2024-03-28T00:54:32.253650+0000","last_unstale":"2024-03-28T00:55:37.567943+0000","last_undegraded":"2024-03-28T00:55:37.567943+0000","last_fullsized":"2024-03-28T00:55:37.567943+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:11:44.930074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560346+0000","last_change":"2024-03-28T00:54:32.815410+0000","last_active":"2024-03-28T00:55:37.560346+0000","last_peered":"2024-03-28T00:55:37.560346+0000","last_clean":"2024-03-28T00:55:37.560346+0000","last_became_active":"2024-03-28T00:54:32.815088+0000","last_became_peered":"2024-03-28T00:54:32.815088+0000","last_unstale":"2024-03-28T00:55:37.560346+0000","last_undegraded":"2024-03-28T00:55:37.560346+0000","last_fullsized":"2024-03-28T00:55:37.560346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:16:24.862822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.566363+0000","last_change":"2024-03-28T00:54:36.823182+0000","last_active":"2024-03-28T00:55:37.566363+0000","last_peered":"2024-03-28T00:55:37.566363+0000","last_clean":"2024-03-28T00:55:37.566363+0000","last_became_active":"2024-03-28T00:54:36.821446+0000","last_became_peered":"2024-03-28T00:54:36.821446+0000","last_unstale":"2024-03-28T00:55:37.566363+0000","last_undegraded":"2024-03-28T00:55:37.566363+0000","last_fullsized":"2024-03-28T00:55:37.566363+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:45:20.030213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918503+0000","last_change":"2024-03-28T00:54:32.257412+0000","last_active":"2024-03-28T00:55:37.918503+0000","last_peered":"2024-03-28T00:55:37.918503+0000","last_clean":"2024-03-28T00:55:37.918503+0000","last_became_active":"2024-03-28T00:54:32.257115+0000","last_became_peered":"2024-03-28T00:54:32.257115+0000","last_unstale":"2024-03-28T00:55:37.918503+0000","last_undegraded":"2024-03-28T00:55:37.918503+0000","last_fullsized":"2024-03-28T00:55:37.918503+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:04:29.850867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560733+0000","last_change":"2024-03-28T00:54:32.805879+0000","last_active":"2024-03-28T00:55:37.560733+0000","last_peered":"2024-03-28T00:55:37.560733+0000","last_clean":"2024-03-28T00:55:37.560733+0000","last_became_active":"2024-03-28T00:54:32.805507+0000","last_became_peered":"2024-03-28T00:54:32.805507+0000","last_unstale":"2024-03-28T00:55:37.560733+0000","last_undegraded":"2024-03-28T00:55:37.560733+0000","last_fullsized":"2024-03-28T00:55:37.560733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:51:51.229085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560889+0000","last_change":"2024-03-28T00:54:32.807219+0000","last_active":"2024-03-28T00:55:37.560889+0000","last_peered":"2024-03-28T00:55:37.560889+0000","last_clean":"2024-03-28T00:55:37.560889+0000","last_became_active":"2024-03-28T00:54:32.806856+0000","last_became_peered":"2024-03-28T00:54:32.806856+0000","last_unstale":"2024-03-28T00:55:37.560889+0000","last_undegraded":"2024-03-28T00:55:37.560889+0000","last_fullsized":"2024-03-28T00:55:37.560889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:05:49.914636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.569156+0000","last_change":"2024-03-28T00:55:37.569156+0000","last_active":"2024-03-28T00:55:37.569156+0000","last_peered":"2024-03-28T00:55:37.569156+0000","last_clean":"2024-03-28T00:55:37.569156+0000","last_became_active":"2024-03-28T00:55:37.568775+0000","last_became_peered":"2024-03-28T00:55:37.568775+0000","last_unstale":"2024-03-28T00:55:37.569156+0000","last_undegraded":"2024-03-28T00:55:37.569156+0000","last_fullsized":"2024-03-28T00:55:37.569156+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:00:21.458550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.562565+0000","last_change":"2024-03-28T00:54:32.813921+0000","last_active":"2024-03-28T00:55:37.562565+0000","last_peered":"2024-03-28T00:55:37.562565+0000","last_clean":"2024-03-28T00:55:37.562565+0000","last_became_active":"2024-03-28T00:54:32.812877+0000","last_became_peered":"2024-03-28T00:54:32.812877+0000","last_unstale":"2024-03-28T00:55:37.562565+0000","last_undegraded":"2024-03-28T00:55:37.562565+0000","last_fullsized":"2024-03-28T00:55:37.562565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:26:54.305074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.565481+0000","last_change":"2024-03-28T00:54:32.815725+0000","last_active":"2024-03-28T00:55:37.565481+0000","last_peered":"2024-03-28T00:55:37.565481+0000","last_clean":"2024-03-28T00:55:37.565481+0000","last_became_active":"2024-03-28T00:54:32.815444+0000","last_became_peered":"2024-03-28T00:54:32.815444+0000","last_unstale":"2024-03-28T00:55:37.565481+0000","last_undegraded":"2024-03-28T00:55:37.565481+0000","last_fullsized":"2024-03-28T00:55:37.565481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:23:17.326589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918293+0000","last_change":"2024-03-28T00:54:32.806595+0000","last_active":"2024-03-28T00:55:37.918293+0000","last_peered":"2024-03-28T00:55:37.918293+0000","last_clean":"2024-03-28T00:55:37.918293+0000","last_became_active":"2024-03-28T00:54:32.806256+0000","last_became_peered":"2024-03-28T00:54:32.806256+0000","last_unstale":"2024-03-28T00:55:37.918293+0000","last_undegraded":"2024-03-28T00:55:37.918293+0000","last_fullsized":"2024-03-28T00:55:37.918293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:04:22.551511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560849+0000","last_change":"2024-03-28T00:54:32.805000+0000","last_active":"2024-03-28T00:55:37.560849+0000","last_peered":"2024-03-28T00:55:37.560849+0000","last_clean":"2024-03-28T00:55:37.560849+0000","last_became_active":"2024-03-28T00:54:32.804637+0000","last_became_peered":"2024-03-28T00:54:32.804637+0000","last_unstale":"2024-03-28T00:55:37.560849+0000","last_undegraded":"2024-03-28T00:55:37.560849+0000","last_fullsized":"2024-03-28T00:55:37.560849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:59:46.888014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563618+0000","last_change":"2024-03-28T00:54:32.802783+0000","last_active":"2024-03-28T00:55:37.563618+0000","last_peered":"2024-03-28T00:55:37.563618+0000","last_clean":"2024-03-28T00:55:37.563618+0000","last_became_active":"2024-03-28T00:54:32.802457+0000","last_became_peered":"2024-03-28T00:54:32.802457+0000","last_unstale":"2024-03-28T00:55:37.563618+0000","last_undegraded":"2024-03-28T00:55:37.563618+0000","last_fullsized":"2024-03-28T00:55:37.563618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:28:24.146047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.919114+0000","last_change":"2024-03-28T00:54:32.810179+0000","last_active":"2024-03-28T00:55:37.919114+0000","last_peered":"2024-03-28T00:55:37.919114+0000","last_clean":"2024-03-28T00:55:37.919114+0000","last_became_active":"2024-03-28T00:54:32.809009+0000","last_became_peered":"2024-03-28T00:54:32.809009+0000","last_unstale":"2024-03-28T00:55:37.919114+0000","last_undegraded":"2024-03-28T00:55:37.919114+0000","last_fullsized":"2024-03-28T00:55:37.919114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:08:44.184910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.566932+0000","last_change":"2024-03-28T00:54:32.810084+0000","last_active":"2024-03-28T00:55:37.566932+0000","last_peered":"2024-03-28T00:55:37.566932+0000","last_clean":"2024-03-28T00:55:37.566932+0000","last_became_active":"2024-03-28T00:54:32.809386+0000","last_became_peered":"2024-03-28T00:54:32.809386+0000","last_unstale":"2024-03-28T00:55:37.566932+0000","last_undegraded":"2024-03-28T00:55:37.566932+0000","last_fullsized":"2024-03-28T00:55:37.566932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:49:13.754797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917587+0000","last_change":"2024-03-28T00:54:32.809933+0000","last_active":"2024-03-28T00:55:37.917587+0000","last_peered":"2024-03-28T00:55:37.917587+0000","last_clean":"2024-03-28T00:55:37.917587+0000","last_became_active":"2024-03-28T00:54:32.808974+0000","last_became_peered":"2024-03-28T00:54:32.808974+0000","last_unstale":"2024-03-28T00:55:37.917587+0000","last_undegraded":"2024-03-28T00:55:37.917587+0000","last_fullsized":"2024-03-28T00:55:37.917587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:23:59.317887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563436+0000","last_change":"2024-03-28T00:54:32.808764+0000","last_active":"2024-03-28T00:55:37.563436+0000","last_peered":"2024-03-28T00:55:37.563436+0000","last_clean":"2024-03-28T00:55:37.563436+0000","last_became_active":"2024-03-28T00:54:32.808034+0000","last_became_peered":"2024-03-28T00:54:32.808034+0000","last_unstale":"2024-03-28T00:55:37.563436+0000","last_undegraded":"2024-03-28T00:55:37.563436+0000","last_fullsized":"2024-03-28T00:55:37.563436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:27:12.762520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918624+0000","last_change":"2024-03-28T00:54:32.805470+0000","last_active":"2024-03-28T00:55:37.918624+0000","last_peered":"2024-03-28T00:55:37.918624+0000","last_clean":"2024-03-28T00:55:37.918624+0000","last_became_active":"2024-03-28T00:54:32.805146+0000","last_became_peered":"2024-03-28T00:54:32.805146+0000","last_unstale":"2024-03-28T00:55:37.918624+0000","last_undegraded":"2024-03-28T00:55:37.918624+0000","last_fullsized":"2024-03-28T00:55:37.918624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:24:14.297966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.566696+0000","last_change":"2024-03-28T00:55:37.566696+0000","last_active":"2024-03-28T00:55:37.566696+0000","last_peered":"2024-03-28T00:55:37.566696+0000","last_clean":"2024-03-28T00:55:37.566696+0000","last_became_active":"2024-03-28T00:55:37.566334+0000","last_became_peered":"2024-03-28T00:55:37.566334+0000","last_unstale":"2024-03-28T00:55:37.566696+0000","last_undegraded":"2024-03-28T00:55:37.566696+0000","last_fullsized":"2024-03-28T00:55:37.566696+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:41:37.752201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282084,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70745,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":426,"num_read_kb":284,"num_write":160,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":212235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383711,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1011712,"data_stored":510244,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645348,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32312,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712840,"statfs":{"total":95995035648,"available":95961948160,"internally_reserved":0,"allocated":1159168,"data_stored":1164234,"data_compressed":59464,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808873,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":514554,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070511,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":510356,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332147,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1073152,"data_stored":1106287,"data_compressed":59464,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495672,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1003520,"data_stored":505634,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1077248,"data_stored":572288,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738454,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":507022,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":574029,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182471,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":505828,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920835,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":510284,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757310,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":1101875,"data_compressed":59464,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34218,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34218,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34218,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":66744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":68485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":68322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T00:56:03.681 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T00:56:03.925 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:56:04.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:04 smithi169 ceph-mon[32274]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:56:04.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:04 smithi174 ceph-mon[36060]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:56:04.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:04 smithi177 ceph-mon[36206]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:56:05.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:05 smithi169 ceph-mon[32274]: pgmap v336: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T00:56:05.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:05 smithi174 ceph-mon[36060]: pgmap v336: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T00:56:05.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.617+0000 7f49da174700 1 -- 172.21.15.169:0/1211061522 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49d41023d0 msgr2=0x7f49d41027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:05.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.617+0000 7f49da174700 1 --2- 172.21.15.169:0/1211061522 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49d41023d0 0x7f49d41027b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f49c40099c0 tx=0x7f49c4009cd0 comp rx=0 tx=0).stop 2024-03-28T00:56:05.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.617+0000 7f49da174700 1 -- 172.21.15.169:0/1211061522 shutdown_connections 2024-03-28T00:56:05.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.617+0000 7f49da174700 1 --2- 172.21.15.169:0/1211061522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49d41074b0 0x7f49d410f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:05.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.617+0000 7f49da174700 1 --2- 172.21.15.169:0/1211061522 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49d41023d0 0x7f49d41027b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:05.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.617+0000 7f49da174700 1 --2- 172.21.15.169:0/1211061522 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49d4102d80 0x7f49d4106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:05.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.617+0000 7f49da174700 1 -- 172.21.15.169:0/1211061522 >> 172.21.15.169:0/1211061522 conn(0x7f49d40fdc40 msgr2=0x7f49d4100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:05.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.617+0000 7f49da174700 1 -- 172.21.15.169:0/1211061522 shutdown_connections 2024-03-28T00:56:05.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.617+0000 7f49da174700 1 -- 172.21.15.169:0/1211061522 wait complete. 2024-03-28T00:56:05.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.617+0000 7f49da174700 1 Processor -- start 2024-03-28T00:56:05.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.618+0000 7f49da174700 1 -- start start 2024-03-28T00:56:05.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.618+0000 7f49da174700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49d41023d0 0x7f49d419e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:05.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.618+0000 7f49da174700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49d4102d80 0x7f49d419ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:05.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.618+0000 7f49da174700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49d41074b0 0x7f49d4198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:05.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.618+0000 7f49da174700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d41124c0 con 0x7f49d41023d0 2024-03-28T00:56:05.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.618+0000 7f49da174700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d4112340 con 0x7f49d41074b0 2024-03-28T00:56:05.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.618+0000 7f49da174700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d4112640 con 0x7f49d4102d80 2024-03-28T00:56:05.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.618+0000 7f49d37fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49d41023d0 0x7f49d419e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:05.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.618+0000 7f49d37fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49d41023d0 0x7f49d419e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48220/0 (socket says 172.21.15.169:48220) 2024-03-28T00:56:05.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.618+0000 7f49d37fe700 1 -- 172.21.15.169:0/2565679229 learned_addr learned my addr 172.21.15.169:0/2565679229 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:05.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.618+0000 7f49d2ffd700 1 --2- 172.21.15.169:0/2565679229 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49d4102d80 0x7f49d419ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:05.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.618+0000 7f49d3fff700 1 --2- 172.21.15.169:0/2565679229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49d41074b0 0x7f49d4198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:05.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.618+0000 7f49d37fe700 1 -- 172.21.15.169:0/2565679229 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49d4102d80 msgr2=0x7f49d419ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:05.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.618+0000 7f49d37fe700 1 --2- 172.21.15.169:0/2565679229 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49d4102d80 0x7f49d419ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:05.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.618+0000 7f49d37fe700 1 -- 172.21.15.169:0/2565679229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49d41074b0 msgr2=0x7f49d4198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:05.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.618+0000 7f49d37fe700 1 --2- 172.21.15.169:0/2565679229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49d41074b0 0x7f49d4198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:05.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.618+0000 7f49d37fe700 1 -- 172.21.15.169:0/2565679229 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d4199040 con 0x7f49d41023d0 2024-03-28T00:56:05.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.619+0000 7f49d37fe700 1 --2- 172.21.15.169:0/2565679229 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49d41023d0 0x7f49d419e690 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f49c40098e0 tx=0x7f49c400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:05.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.619+0000 7f49d0ff9700 1 -- 172.21.15.169:0/2565679229 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c4009e60 con 0x7f49d41023d0 2024-03-28T00:56:05.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.619+0000 7f49d0ff9700 1 -- 172.21.15.169:0/2565679229 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49c40050d0 con 0x7f49d41023d0 2024-03-28T00:56:05.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.619+0000 7f49da174700 1 -- 172.21.15.169:0/2565679229 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d41992d0 con 0x7f49d41023d0 2024-03-28T00:56:05.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.619+0000 7f49d0ff9700 1 -- 172.21.15.169:0/2565679229 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c4003df0 con 0x7f49d41023d0 2024-03-28T00:56:05.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.619+0000 7f49da174700 1 -- 172.21.15.169:0/2565679229 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d41a5440 con 0x7f49d41023d0 2024-03-28T00:56:05.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.620+0000 7f49da174700 1 -- 172.21.15.169:0/2565679229 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d41038b0 con 0x7f49d41023d0 2024-03-28T00:56:05.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.621+0000 7f49d0ff9700 1 -- 172.21.15.169:0/2565679229 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f49c4005280 con 0x7f49d41023d0 2024-03-28T00:56:05.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.621+0000 7f49d0ff9700 1 --2- 172.21.15.169:0/2565679229 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f49bc0a2920 0x7f49bc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:05.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.621+0000 7f49d2ffd700 1 --2- 172.21.15.169:0/2565679229 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f49bc0a2920 0x7f49bc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:05.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.621+0000 7f49d0ff9700 1 -- 172.21.15.169:0/2565679229 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f49c40cfc40 con 0x7f49d41023d0 2024-03-28T00:56:05.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.622+0000 7f49d2ffd700 1 --2- 172.21.15.169:0/2565679229 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f49bc0a2920 0x7f49bc0a4de0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f49d4199f20 tx=0x7f49c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:05.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.625+0000 7f49d0ff9700 1 -- 172.21.15.169:0/2565679229 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f49c4013070 con 0x7f49d41023d0 2024-03-28T00:56:05.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:05 smithi177 ceph-mon[36206]: pgmap v336: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T00:56:05.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.819+0000 7f49da174700 1 -- 172.21.15.169:0/2565679229 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f49d40625d0 con 0x7f49bc0a2920 2024-03-28T00:56:05.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.828+0000 7f49d0ff9700 1 -- 172.21.15.169:0/2565679229 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340630 (secure 0 0 0) 0x7f49d40625d0 con 0x7f49bc0a2920 2024-03-28T00:56:05.836 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:05.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.832+0000 7f49da174700 1 -- 172.21.15.169:0/2565679229 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f49bc0a2920 msgr2=0x7f49bc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:05.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.832+0000 7f49da174700 1 --2- 172.21.15.169:0/2565679229 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f49bc0a2920 0x7f49bc0a4de0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f49d4199f20 tx=0x7f49c8009260 comp rx=0 tx=0).stop 2024-03-28T00:56:05.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.832+0000 7f49da174700 1 -- 172.21.15.169:0/2565679229 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49d41023d0 msgr2=0x7f49d419e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:05.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.832+0000 7f49da174700 1 --2- 172.21.15.169:0/2565679229 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49d41023d0 0x7f49d419e690 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f49c40098e0 tx=0x7f49c400f690 comp rx=0 tx=0).stop 2024-03-28T00:56:05.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.832+0000 7f49da174700 1 -- 172.21.15.169:0/2565679229 shutdown_connections 2024-03-28T00:56:05.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.832+0000 7f49da174700 1 --2- 172.21.15.169:0/2565679229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49d41074b0 0x7f49d4198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:05.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.832+0000 7f49da174700 1 --2- 172.21.15.169:0/2565679229 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49d41023d0 0x7f49d419e690 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:05.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.832+0000 7f49da174700 1 --2- 172.21.15.169:0/2565679229 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f49bc0a2920 0x7f49bc0a4de0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:05.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.832+0000 7f49da174700 1 --2- 172.21.15.169:0/2565679229 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49d4102d80 0x7f49d419ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:05.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.832+0000 7f49da174700 1 -- 172.21.15.169:0/2565679229 >> 172.21.15.169:0/2565679229 conn(0x7f49d40fdc40 msgr2=0x7f49d40fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:05.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.832+0000 7f49da174700 1 -- 172.21.15.169:0/2565679229 shutdown_connections 2024-03-28T00:56:05.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:05.832+0000 7f49da174700 1 -- 172.21.15.169:0/2565679229 wait complete. 2024-03-28T00:56:05.839 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-28T00:56:06.492 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-03-28T00:56:04.104811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661347,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":567,"num_read_kb":403,"num_write":343,"num_write_kb":3057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279,"ondisk_log_size":279,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357384,"kb_used_data":12320,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584440,"statfs":{"total":1151940427776,"available":1151574466560,"internally_reserved":0,"allocated":12615680,"data_stored":8082635,"data_compressed":390594,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003690"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.561655+0000","last_change":"2024-03-28T00:54:32.809951+0000","last_active":"2024-03-28T00:55:37.561655+0000","last_peered":"2024-03-28T00:55:37.561655+0000","last_clean":"2024-03-28T00:55:37.561655+0000","last_became_active":"2024-03-28T00:54:32.808871+0000","last_became_peered":"2024-03-28T00:54:32.808871+0000","last_unstale":"2024-03-28T00:55:37.561655+0000","last_undegraded":"2024-03-28T00:55:37.561655+0000","last_fullsized":"2024-03-28T00:55:37.561655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:15:17.152487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.565720+0000","last_change":"2024-03-28T00:54:32.816707+0000","last_active":"2024-03-28T00:55:37.565720+0000","last_peered":"2024-03-28T00:55:37.565720+0000","last_clean":"2024-03-28T00:55:37.565720+0000","last_became_active":"2024-03-28T00:54:32.816000+0000","last_became_peered":"2024-03-28T00:54:32.816000+0000","last_unstale":"2024-03-28T00:55:37.565720+0000","last_undegraded":"2024-03-28T00:55:37.565720+0000","last_fullsized":"2024-03-28T00:55:37.565720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:56:19.532808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.565389+0000","last_change":"2024-03-28T00:54:32.817023+0000","last_active":"2024-03-28T00:55:37.565389+0000","last_peered":"2024-03-28T00:55:37.565389+0000","last_clean":"2024-03-28T00:55:37.565389+0000","last_became_active":"2024-03-28T00:54:32.816712+0000","last_became_peered":"2024-03-28T00:54:32.816712+0000","last_unstale":"2024-03-28T00:55:37.565389+0000","last_undegraded":"2024-03-28T00:55:37.565389+0000","last_fullsized":"2024-03-28T00:55:37.565389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:13:25.334294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559693+0000","last_change":"2024-03-28T00:54:32.807259+0000","last_active":"2024-03-28T00:55:37.559693+0000","last_peered":"2024-03-28T00:55:37.559693+0000","last_clean":"2024-03-28T00:55:37.559693+0000","last_became_active":"2024-03-28T00:54:32.806944+0000","last_became_peered":"2024-03-28T00:54:32.806944+0000","last_unstale":"2024-03-28T00:55:37.559693+0000","last_undegraded":"2024-03-28T00:55:37.559693+0000","last_fullsized":"2024-03-28T00:55:37.559693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:46:04.852351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559260+0000","last_change":"2024-03-28T00:54:32.807613+0000","last_active":"2024-03-28T00:55:37.559260+0000","last_peered":"2024-03-28T00:55:37.559260+0000","last_clean":"2024-03-28T00:55:37.559260+0000","last_became_active":"2024-03-28T00:54:32.807280+0000","last_became_peered":"2024-03-28T00:54:32.807280+0000","last_unstale":"2024-03-28T00:55:37.559260+0000","last_undegraded":"2024-03-28T00:55:37.559260+0000","last_fullsized":"2024-03-28T00:55:37.559260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:56:37.133675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917502+0000","last_change":"2024-03-28T00:54:32.806804+0000","last_active":"2024-03-28T00:55:37.917502+0000","last_peered":"2024-03-28T00:55:37.917502+0000","last_clean":"2024-03-28T00:55:37.917502+0000","last_became_active":"2024-03-28T00:54:32.805343+0000","last_became_peered":"2024-03-28T00:54:32.805343+0000","last_unstale":"2024-03-28T00:55:37.917502+0000","last_undegraded":"2024-03-28T00:55:37.917502+0000","last_fullsized":"2024-03-28T00:55:37.917502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:31:44.849926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563371+0000","last_change":"2024-03-28T00:55:37.563371+0000","last_active":"2024-03-28T00:55:37.563371+0000","last_peered":"2024-03-28T00:55:37.563371+0000","last_clean":"2024-03-28T00:55:37.563371+0000","last_became_active":"2024-03-28T00:55:37.563000+0000","last_became_peered":"2024-03-28T00:55:37.563000+0000","last_unstale":"2024-03-28T00:55:37.563371+0000","last_undegraded":"2024-03-28T00:55:37.563371+0000","last_fullsized":"2024-03-28T00:55:37.563371+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:56:58.782502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917264+0000","last_change":"2024-03-28T00:54:32.808058+0000","last_active":"2024-03-28T00:55:37.917264+0000","last_peered":"2024-03-28T00:55:37.917264+0000","last_clean":"2024-03-28T00:55:37.917264+0000","last_became_active":"2024-03-28T00:54:32.807755+0000","last_became_peered":"2024-03-28T00:54:32.807755+0000","last_unstale":"2024-03-28T00:55:37.917264+0000","last_undegraded":"2024-03-28T00:55:37.917264+0000","last_fullsized":"2024-03-28T00:55:37.917264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:08:42.707015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.564001+0000","last_change":"2024-03-28T00:54:32.800667+0000","last_active":"2024-03-28T00:55:37.564001+0000","last_peered":"2024-03-28T00:55:37.564001+0000","last_clean":"2024-03-28T00:55:37.564001+0000","last_became_active":"2024-03-28T00:54:32.800332+0000","last_became_peered":"2024-03-28T00:54:32.800332+0000","last_unstale":"2024-03-28T00:55:37.564001+0000","last_undegraded":"2024-03-28T00:55:37.564001+0000","last_fullsized":"2024-03-28T00:55:37.564001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:41:27.577871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917751+0000","last_change":"2024-03-28T00:54:32.807958+0000","last_active":"2024-03-28T00:55:37.917751+0000","last_peered":"2024-03-28T00:55:37.917751+0000","last_clean":"2024-03-28T00:55:37.917751+0000","last_became_active":"2024-03-28T00:54:32.807657+0000","last_became_peered":"2024-03-28T00:54:32.807657+0000","last_unstale":"2024-03-28T00:55:37.917751+0000","last_undegraded":"2024-03-28T00:55:37.917751+0000","last_fullsized":"2024-03-28T00:55:37.917751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:30:28.000690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560698+0000","last_change":"2024-03-28T00:54:32.811543+0000","last_active":"2024-03-28T00:55:37.560698+0000","last_peered":"2024-03-28T00:55:37.560698+0000","last_clean":"2024-03-28T00:55:37.560698+0000","last_became_active":"2024-03-28T00:54:32.811210+0000","last_became_peered":"2024-03-28T00:54:32.811210+0000","last_unstale":"2024-03-28T00:55:37.560698+0000","last_undegraded":"2024-03-28T00:55:37.560698+0000","last_fullsized":"2024-03-28T00:55:37.560698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:47:19.539107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559659+0000","last_change":"2024-03-28T00:54:32.812648+0000","last_active":"2024-03-28T00:55:37.559659+0000","last_peered":"2024-03-28T00:55:37.559659+0000","last_clean":"2024-03-28T00:55:37.559659+0000","last_became_active":"2024-03-28T00:54:32.812340+0000","last_became_peered":"2024-03-28T00:54:32.812340+0000","last_unstale":"2024-03-28T00:55:37.559659+0000","last_undegraded":"2024-03-28T00:55:37.559659+0000","last_fullsized":"2024-03-28T00:55:37.559659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:20:45.449953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918534+0000","last_change":"2024-03-28T00:54:32.801807+0000","last_active":"2024-03-28T00:55:37.918534+0000","last_peered":"2024-03-28T00:55:37.918534+0000","last_clean":"2024-03-28T00:55:37.918534+0000","last_became_active":"2024-03-28T00:54:32.801498+0000","last_became_peered":"2024-03-28T00:54:32.801498+0000","last_unstale":"2024-03-28T00:55:37.918534+0000","last_undegraded":"2024-03-28T00:55:37.918534+0000","last_fullsized":"2024-03-28T00:55:37.918534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:35:35.827856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.565160+0000","last_change":"2024-03-28T00:54:32.818033+0000","last_active":"2024-03-28T00:55:37.565160+0000","last_peered":"2024-03-28T00:55:37.565160+0000","last_clean":"2024-03-28T00:55:37.565160+0000","last_became_active":"2024-03-28T00:54:32.817309+0000","last_became_peered":"2024-03-28T00:54:32.817309+0000","last_unstale":"2024-03-28T00:55:37.565160+0000","last_undegraded":"2024-03-28T00:55:37.565160+0000","last_fullsized":"2024-03-28T00:55:37.565160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:39:42.682780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.565411+0000","last_change":"2024-03-28T00:54:32.815224+0000","last_active":"2024-03-28T00:55:37.565411+0000","last_peered":"2024-03-28T00:55:37.565411+0000","last_clean":"2024-03-28T00:55:37.565411+0000","last_became_active":"2024-03-28T00:54:32.814936+0000","last_became_peered":"2024-03-28T00:54:32.814936+0000","last_unstale":"2024-03-28T00:55:37.565411+0000","last_undegraded":"2024-03-28T00:55:37.565411+0000","last_fullsized":"2024-03-28T00:55:37.565411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:08:40.200393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563117+0000","last_change":"2024-03-28T00:54:32.807342+0000","last_active":"2024-03-28T00:55:37.563117+0000","last_peered":"2024-03-28T00:55:37.563117+0000","last_clean":"2024-03-28T00:55:37.563117+0000","last_became_active":"2024-03-28T00:54:32.806978+0000","last_became_peered":"2024-03-28T00:54:32.806978+0000","last_unstale":"2024-03-28T00:55:37.563117+0000","last_undegraded":"2024-03-28T00:55:37.563117+0000","last_fullsized":"2024-03-28T00:55:37.563117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:17:58.599693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.561239+0000","last_change":"2024-03-28T00:54:32.810036+0000","last_active":"2024-03-28T00:55:37.561239+0000","last_peered":"2024-03-28T00:55:37.561239+0000","last_clean":"2024-03-28T00:55:37.561239+0000","last_became_active":"2024-03-28T00:54:32.809438+0000","last_became_peered":"2024-03-28T00:54:32.809438+0000","last_unstale":"2024-03-28T00:55:37.561239+0000","last_undegraded":"2024-03-28T00:55:37.561239+0000","last_fullsized":"2024-03-28T00:55:37.561239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:18:50.447474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.558744+0000","last_change":"2024-03-28T00:54:32.815398+0000","last_active":"2024-03-28T00:55:37.558744+0000","last_peered":"2024-03-28T00:55:37.558744+0000","last_clean":"2024-03-28T00:55:37.558744+0000","last_became_active":"2024-03-28T00:54:32.815045+0000","last_became_peered":"2024-03-28T00:54:32.815045+0000","last_unstale":"2024-03-28T00:55:37.558744+0000","last_undegraded":"2024-03-28T00:55:37.558744+0000","last_fullsized":"2024-03-28T00:55:37.558744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:40:29.060217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563241+0000","last_change":"2024-03-28T00:54:32.805736+0000","last_active":"2024-03-28T00:55:37.563241+0000","last_peered":"2024-03-28T00:55:37.563241+0000","last_clean":"2024-03-28T00:55:37.563241+0000","last_became_active":"2024-03-28T00:54:32.805435+0000","last_became_peered":"2024-03-28T00:54:32.805435+0000","last_unstale":"2024-03-28T00:55:37.563241+0000","last_undegraded":"2024-03-28T00:55:37.563241+0000","last_fullsized":"2024-03-28T00:55:37.563241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:24:30.893583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563132+0000","last_change":"2024-03-28T00:54:36.824531+0000","last_active":"2024-03-28T00:55:37.563132+0000","last_peered":"2024-03-28T00:55:37.563132+0000","last_clean":"2024-03-28T00:55:37.563132+0000","last_became_active":"2024-03-28T00:54:36.824123+0000","last_became_peered":"2024-03-28T00:54:36.824123+0000","last_unstale":"2024-03-28T00:55:37.563132+0000","last_undegraded":"2024-03-28T00:55:37.563132+0000","last_fullsized":"2024-03-28T00:55:37.563132+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:10:47.503171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.558883+0000","last_change":"2024-03-28T00:54:32.256226+0000","last_active":"2024-03-28T00:55:37.558883+0000","last_peered":"2024-03-28T00:55:37.558883+0000","last_clean":"2024-03-28T00:55:37.558883+0000","last_became_active":"2024-03-28T00:54:32.255921+0000","last_became_peered":"2024-03-28T00:54:32.255921+0000","last_unstale":"2024-03-28T00:55:37.558883+0000","last_undegraded":"2024-03-28T00:55:37.558883+0000","last_fullsized":"2024-03-28T00:55:37.558883+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:56:04.196311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918062+0000","last_change":"2024-03-28T00:54:32.808248+0000","last_active":"2024-03-28T00:55:37.918062+0000","last_peered":"2024-03-28T00:55:37.918062+0000","last_clean":"2024-03-28T00:55:37.918062+0000","last_became_active":"2024-03-28T00:54:32.807923+0000","last_became_peered":"2024-03-28T00:54:32.807923+0000","last_unstale":"2024-03-28T00:55:37.918062+0000","last_undegraded":"2024-03-28T00:55:37.918062+0000","last_fullsized":"2024-03-28T00:55:37.918062+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:59:56.002564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559108+0000","last_change":"2024-03-28T00:54:36.830556+0000","last_active":"2024-03-28T00:55:37.559108+0000","last_peered":"2024-03-28T00:55:37.559108+0000","last_clean":"2024-03-28T00:55:37.559108+0000","last_became_active":"2024-03-28T00:54:36.830151+0000","last_became_peered":"2024-03-28T00:54:36.830151+0000","last_unstale":"2024-03-28T00:55:37.559108+0000","last_undegraded":"2024-03-28T00:55:37.559108+0000","last_fullsized":"2024-03-28T00:55:37.559108+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:19:23.073410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.919021+0000","last_change":"2024-03-28T00:54:32.260635+0000","last_active":"2024-03-28T00:55:37.919021+0000","last_peered":"2024-03-28T00:55:37.919021+0000","last_clean":"2024-03-28T00:55:37.919021+0000","last_became_active":"2024-03-28T00:54:32.260084+0000","last_became_peered":"2024-03-28T00:54:32.260084+0000","last_unstale":"2024-03-28T00:55:37.919021+0000","last_undegraded":"2024-03-28T00:55:37.919021+0000","last_fullsized":"2024-03-28T00:55:37.919021+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:06.858519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.565721+0000","last_change":"2024-03-28T00:54:32.809901+0000","last_active":"2024-03-28T00:55:37.565721+0000","last_peered":"2024-03-28T00:55:37.565721+0000","last_clean":"2024-03-28T00:55:37.565721+0000","last_became_active":"2024-03-28T00:54:32.809557+0000","last_became_peered":"2024-03-28T00:54:32.809557+0000","last_unstale":"2024-03-28T00:55:37.565721+0000","last_undegraded":"2024-03-28T00:55:37.565721+0000","last_fullsized":"2024-03-28T00:55:37.565721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:05:31.363487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559387+0000","last_change":"2024-03-28T00:54:36.824661+0000","last_active":"2024-03-28T00:55:37.559387+0000","last_peered":"2024-03-28T00:55:37.559387+0000","last_clean":"2024-03-28T00:55:37.559387+0000","last_became_active":"2024-03-28T00:54:36.824223+0000","last_became_peered":"2024-03-28T00:54:36.824223+0000","last_unstale":"2024-03-28T00:55:37.559387+0000","last_undegraded":"2024-03-28T00:55:37.559387+0000","last_fullsized":"2024-03-28T00:55:37.559387+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:02:25.509248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559890+0000","last_change":"2024-03-28T00:54:32.256436+0000","last_active":"2024-03-28T00:55:37.559890+0000","last_peered":"2024-03-28T00:55:37.559890+0000","last_clean":"2024-03-28T00:55:37.559890+0000","last_became_active":"2024-03-28T00:54:32.256069+0000","last_became_peered":"2024-03-28T00:54:32.256069+0000","last_unstale":"2024-03-28T00:55:37.559890+0000","last_undegraded":"2024-03-28T00:55:37.559890+0000","last_fullsized":"2024-03-28T00:55:37.559890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:43.688192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563466+0000","last_change":"2024-03-28T00:54:32.799269+0000","last_active":"2024-03-28T00:55:37.563466+0000","last_peered":"2024-03-28T00:55:37.563466+0000","last_clean":"2024-03-28T00:55:37.563466+0000","last_became_active":"2024-03-28T00:54:32.798848+0000","last_became_peered":"2024-03-28T00:54:32.798848+0000","last_unstale":"2024-03-28T00:55:37.563466+0000","last_undegraded":"2024-03-28T00:55:37.563466+0000","last_fullsized":"2024-03-28T00:55:37.563466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:56:20.318466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559246+0000","last_change":"2024-03-28T00:54:36.823153+0000","last_active":"2024-03-28T00:55:37.559246+0000","last_peered":"2024-03-28T00:55:37.559246+0000","last_clean":"2024-03-28T00:55:37.559246+0000","last_became_active":"2024-03-28T00:54:36.822817+0000","last_became_peered":"2024-03-28T00:54:36.822817+0000","last_unstale":"2024-03-28T00:55:37.559246+0000","last_undegraded":"2024-03-28T00:55:37.559246+0000","last_fullsized":"2024-03-28T00:55:37.559246+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:20:23.001599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559731+0000","last_change":"2024-03-28T00:54:32.250439+0000","last_active":"2024-03-28T00:55:37.559731+0000","last_peered":"2024-03-28T00:55:37.559731+0000","last_clean":"2024-03-28T00:55:37.559731+0000","last_became_active":"2024-03-28T00:54:32.250006+0000","last_became_peered":"2024-03-28T00:54:32.250006+0000","last_unstale":"2024-03-28T00:55:37.559731+0000","last_undegraded":"2024-03-28T00:55:37.559731+0000","last_fullsized":"2024-03-28T00:55:37.559731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:31:38.957429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.567916+0000","last_change":"2024-03-28T00:54:32.806888+0000","last_active":"2024-03-28T00:55:37.567916+0000","last_peered":"2024-03-28T00:55:37.567916+0000","last_clean":"2024-03-28T00:55:37.567916+0000","last_became_active":"2024-03-28T00:54:32.806581+0000","last_became_peered":"2024-03-28T00:54:32.806581+0000","last_unstale":"2024-03-28T00:55:37.567916+0000","last_undegraded":"2024-03-28T00:55:37.567916+0000","last_fullsized":"2024-03-28T00:55:37.567916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:25:52.331819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563729+0000","last_change":"2024-03-28T00:54:36.824041+0000","last_active":"2024-03-28T00:55:37.563729+0000","last_peered":"2024-03-28T00:55:37.563729+0000","last_clean":"2024-03-28T00:55:37.563729+0000","last_became_active":"2024-03-28T00:54:36.823658+0000","last_became_peered":"2024-03-28T00:54:36.823658+0000","last_unstale":"2024-03-28T00:55:37.563729+0000","last_undegraded":"2024-03-28T00:55:37.563729+0000","last_fullsized":"2024-03-28T00:55:37.563729+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:11:03.236009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":107,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:59.350361+0000","last_change":"2024-03-28T00:54:32.257534+0000","last_active":"2024-03-28T00:55:59.350361+0000","last_peered":"2024-03-28T00:55:59.350361+0000","last_clean":"2024-03-28T00:55:59.350361+0000","last_became_active":"2024-03-28T00:54:32.256404+0000","last_became_peered":"2024-03-28T00:54:32.256404+0000","last_unstale":"2024-03-28T00:55:59.350361+0000","last_undegraded":"2024-03-28T00:55:59.350361+0000","last_fullsized":"2024-03-28T00:55:59.350361+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:39:38.329799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559601+0000","last_change":"2024-03-28T00:54:32.806099+0000","last_active":"2024-03-28T00:55:37.559601+0000","last_peered":"2024-03-28T00:55:37.559601+0000","last_clean":"2024-03-28T00:55:37.559601+0000","last_became_active":"2024-03-28T00:54:32.805811+0000","last_became_peered":"2024-03-28T00:54:32.805811+0000","last_unstale":"2024-03-28T00:55:37.559601+0000","last_undegraded":"2024-03-28T00:55:37.559601+0000","last_fullsized":"2024-03-28T00:55:37.559601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:42:32.820245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.562926+0000","last_change":"2024-03-28T00:54:36.827198+0000","last_active":"2024-03-28T00:55:37.562926+0000","last_peered":"2024-03-28T00:55:37.562926+0000","last_clean":"2024-03-28T00:55:37.562926+0000","last_became_active":"2024-03-28T00:54:36.826804+0000","last_became_peered":"2024-03-28T00:54:36.826804+0000","last_unstale":"2024-03-28T00:55:37.562926+0000","last_undegraded":"2024-03-28T00:55:37.562926+0000","last_fullsized":"2024-03-28T00:55:37.562926+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:38:11.030404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917395+0000","last_change":"2024-03-28T00:54:32.249394+0000","last_active":"2024-03-28T00:55:37.917395+0000","last_peered":"2024-03-28T00:55:37.917395+0000","last_clean":"2024-03-28T00:55:37.917395+0000","last_became_active":"2024-03-28T00:54:32.249089+0000","last_became_peered":"2024-03-28T00:54:32.249089+0000","last_unstale":"2024-03-28T00:55:37.917395+0000","last_undegraded":"2024-03-28T00:55:37.917395+0000","last_fullsized":"2024-03-28T00:55:37.917395+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:05:01.720053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.568653+0000","last_change":"2024-03-28T00:54:32.809133+0000","last_active":"2024-03-28T00:55:37.568653+0000","last_peered":"2024-03-28T00:55:37.568653+0000","last_clean":"2024-03-28T00:55:37.568653+0000","last_became_active":"2024-03-28T00:54:32.808841+0000","last_became_peered":"2024-03-28T00:54:32.808841+0000","last_unstale":"2024-03-28T00:55:37.568653+0000","last_undegraded":"2024-03-28T00:55:37.568653+0000","last_fullsized":"2024-03-28T00:55:37.568653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:19:37.859542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.561131+0000","last_change":"2024-03-28T00:54:36.824483+0000","last_active":"2024-03-28T00:55:37.561131+0000","last_peered":"2024-03-28T00:55:37.561131+0000","last_clean":"2024-03-28T00:55:37.561131+0000","last_became_active":"2024-03-28T00:54:36.823973+0000","last_became_peered":"2024-03-28T00:54:36.823973+0000","last_unstale":"2024-03-28T00:55:37.561131+0000","last_undegraded":"2024-03-28T00:55:37.561131+0000","last_fullsized":"2024-03-28T00:55:37.561131+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:01:06.251119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918137+0000","last_change":"2024-03-28T00:54:32.251287+0000","last_active":"2024-03-28T00:55:37.918137+0000","last_peered":"2024-03-28T00:55:37.918137+0000","last_clean":"2024-03-28T00:55:37.918137+0000","last_became_active":"2024-03-28T00:54:32.250904+0000","last_became_peered":"2024-03-28T00:54:32.250904+0000","last_unstale":"2024-03-28T00:55:37.918137+0000","last_undegraded":"2024-03-28T00:55:37.918137+0000","last_fullsized":"2024-03-28T00:55:37.918137+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:18:09.051056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917473+0000","last_change":"2024-03-28T00:54:32.803124+0000","last_active":"2024-03-28T00:55:37.917473+0000","last_peered":"2024-03-28T00:55:37.917473+0000","last_clean":"2024-03-28T00:55:37.917473+0000","last_became_active":"2024-03-28T00:54:32.802825+0000","last_became_peered":"2024-03-28T00:54:32.802825+0000","last_unstale":"2024-03-28T00:55:37.917473+0000","last_undegraded":"2024-03-28T00:55:37.917473+0000","last_fullsized":"2024-03-28T00:55:37.917473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:34:21.825314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.565481+0000","last_change":"2024-03-28T00:54:36.832151+0000","last_active":"2024-03-28T00:55:37.565481+0000","last_peered":"2024-03-28T00:55:37.565481+0000","last_clean":"2024-03-28T00:55:37.565481+0000","last_became_active":"2024-03-28T00:54:36.831748+0000","last_became_peered":"2024-03-28T00:54:36.831748+0000","last_unstale":"2024-03-28T00:55:37.565481+0000","last_undegraded":"2024-03-28T00:55:37.565481+0000","last_fullsized":"2024-03-28T00:55:37.565481+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:45:32.284386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.566817+0000","last_change":"2024-03-28T00:54:32.259048+0000","last_active":"2024-03-28T00:55:37.566817+0000","last_peered":"2024-03-28T00:55:37.566817+0000","last_clean":"2024-03-28T00:55:37.566817+0000","last_became_active":"2024-03-28T00:54:32.258038+0000","last_became_peered":"2024-03-28T00:54:32.258038+0000","last_unstale":"2024-03-28T00:55:37.566817+0000","last_undegraded":"2024-03-28T00:55:37.566817+0000","last_fullsized":"2024-03-28T00:55:37.566817+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:45:23.525516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.558384+0000","last_change":"2024-03-28T00:54:32.809771+0000","last_active":"2024-03-28T00:55:37.558384+0000","last_peered":"2024-03-28T00:55:37.558384+0000","last_clean":"2024-03-28T00:55:37.558384+0000","last_became_active":"2024-03-28T00:54:32.809467+0000","last_became_peered":"2024-03-28T00:54:32.809467+0000","last_unstale":"2024-03-28T00:55:37.558384+0000","last_undegraded":"2024-03-28T00:55:37.558384+0000","last_fullsized":"2024-03-28T00:55:37.558384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:30:15.440202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917482+0000","last_change":"2024-03-28T00:54:36.823313+0000","last_active":"2024-03-28T00:55:37.917482+0000","last_peered":"2024-03-28T00:55:37.917482+0000","last_clean":"2024-03-28T00:55:37.917482+0000","last_became_active":"2024-03-28T00:54:36.822923+0000","last_became_peered":"2024-03-28T00:54:36.822923+0000","last_unstale":"2024-03-28T00:55:37.917482+0000","last_undegraded":"2024-03-28T00:55:37.917482+0000","last_fullsized":"2024-03-28T00:55:37.917482+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:22:21.125028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":102,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:56:00.350876+0000","last_change":"2024-03-28T00:54:32.255084+0000","last_active":"2024-03-28T00:56:00.350876+0000","last_peered":"2024-03-28T00:56:00.350876+0000","last_clean":"2024-03-28T00:56:00.350876+0000","last_became_active":"2024-03-28T00:54:32.254744+0000","last_became_peered":"2024-03-28T00:54:32.254744+0000","last_unstale":"2024-03-28T00:56:00.350876+0000","last_undegraded":"2024-03-28T00:56:00.350876+0000","last_fullsized":"2024-03-28T00:56:00.350876+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:32:03.906826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563748+0000","last_change":"2024-03-28T00:54:32.810050+0000","last_active":"2024-03-28T00:55:37.563748+0000","last_peered":"2024-03-28T00:55:37.563748+0000","last_clean":"2024-03-28T00:55:37.563748+0000","last_became_active":"2024-03-28T00:54:32.809731+0000","last_became_peered":"2024-03-28T00:54:32.809731+0000","last_unstale":"2024-03-28T00:55:37.563748+0000","last_undegraded":"2024-03-28T00:55:37.563748+0000","last_fullsized":"2024-03-28T00:55:37.563748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:20:11.876731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.567437+0000","last_change":"2024-03-28T00:55:37.566900+0000","last_active":"2024-03-28T00:55:37.567437+0000","last_peered":"2024-03-28T00:55:37.567437+0000","last_clean":"2024-03-28T00:55:37.567437+0000","last_became_active":"2024-03-28T00:55:37.566491+0000","last_became_peered":"2024-03-28T00:55:37.566491+0000","last_unstale":"2024-03-28T00:55:37.567437+0000","last_undegraded":"2024-03-28T00:55:37.567437+0000","last_fullsized":"2024-03-28T00:55:37.567437+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:31:11.489908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560419+0000","last_change":"2024-03-28T00:54:32.259159+0000","last_active":"2024-03-28T00:55:37.560419+0000","last_peered":"2024-03-28T00:55:37.560419+0000","last_clean":"2024-03-28T00:55:37.560419+0000","last_became_active":"2024-03-28T00:54:32.258708+0000","last_became_peered":"2024-03-28T00:54:32.258708+0000","last_unstale":"2024-03-28T00:55:37.560419+0000","last_undegraded":"2024-03-28T00:55:37.560419+0000","last_fullsized":"2024-03-28T00:55:37.560419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:37:01.524878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.565610+0000","last_change":"2024-03-28T00:54:32.814358+0000","last_active":"2024-03-28T00:55:37.565610+0000","last_peered":"2024-03-28T00:55:37.565610+0000","last_clean":"2024-03-28T00:55:37.565610+0000","last_became_active":"2024-03-28T00:54:32.814018+0000","last_became_peered":"2024-03-28T00:54:32.814018+0000","last_unstale":"2024-03-28T00:55:37.565610+0000","last_undegraded":"2024-03-28T00:55:37.565610+0000","last_fullsized":"2024-03-28T00:55:37.565610+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:24:38.287853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559495+0000","last_change":"2024-03-28T00:54:36.822161+0000","last_active":"2024-03-28T00:55:37.559495+0000","last_peered":"2024-03-28T00:55:37.559495+0000","last_clean":"2024-03-28T00:55:37.559495+0000","last_became_active":"2024-03-28T00:54:36.821750+0000","last_became_peered":"2024-03-28T00:54:36.821750+0000","last_unstale":"2024-03-28T00:55:37.559495+0000","last_undegraded":"2024-03-28T00:55:37.559495+0000","last_fullsized":"2024-03-28T00:55:37.559495+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:13:36.892453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.567834+0000","last_change":"2024-03-28T00:54:32.257038+0000","last_active":"2024-03-28T00:55:37.567834+0000","last_peered":"2024-03-28T00:55:37.567834+0000","last_clean":"2024-03-28T00:55:37.567834+0000","last_became_active":"2024-03-28T00:54:32.256702+0000","last_became_peered":"2024-03-28T00:54:32.256702+0000","last_unstale":"2024-03-28T00:55:37.567834+0000","last_undegraded":"2024-03-28T00:55:37.567834+0000","last_fullsized":"2024-03-28T00:55:37.567834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:35:17.294116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563608+0000","last_change":"2024-03-28T00:54:32.813514+0000","last_active":"2024-03-28T00:55:37.563608+0000","last_peered":"2024-03-28T00:55:37.563608+0000","last_clean":"2024-03-28T00:55:37.563608+0000","last_became_active":"2024-03-28T00:54:32.813192+0000","last_became_peered":"2024-03-28T00:54:32.813192+0000","last_unstale":"2024-03-28T00:55:37.563608+0000","last_undegraded":"2024-03-28T00:55:37.563608+0000","last_fullsized":"2024-03-28T00:55:37.563608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:55:03.410818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.558388+0000","last_change":"2024-03-28T00:54:36.820995+0000","last_active":"2024-03-28T00:55:37.558388+0000","last_peered":"2024-03-28T00:55:37.558388+0000","last_clean":"2024-03-28T00:55:37.558388+0000","last_became_active":"2024-03-28T00:54:36.820628+0000","last_became_peered":"2024-03-28T00:54:36.820628+0000","last_unstale":"2024-03-28T00:55:37.558388+0000","last_undegraded":"2024-03-28T00:55:37.558388+0000","last_fullsized":"2024-03-28T00:55:37.558388+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:18:58.771527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'34","reported_seq":210,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:56:01.351650+0000","last_change":"2024-03-28T00:54:32.251092+0000","last_active":"2024-03-28T00:56:01.351650+0000","last_peered":"2024-03-28T00:56:01.351650+0000","last_clean":"2024-03-28T00:56:01.351650+0000","last_became_active":"2024-03-28T00:54:32.250759+0000","last_became_peered":"2024-03-28T00:54:32.250759+0000","last_unstale":"2024-03-28T00:56:01.351650+0000","last_undegraded":"2024-03-28T00:56:01.351650+0000","last_fullsized":"2024-03-28T00:56:01.351650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:57:30.283365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66744,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":147,"num_read_kb":138,"num_write":35,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559613+0000","last_change":"2024-03-28T00:54:32.809580+0000","last_active":"2024-03-28T00:55:37.559613+0000","last_peered":"2024-03-28T00:55:37.559613+0000","last_clean":"2024-03-28T00:55:37.559613+0000","last_became_active":"2024-03-28T00:54:32.809238+0000","last_became_peered":"2024-03-28T00:54:32.809238+0000","last_unstale":"2024-03-28T00:55:37.559613+0000","last_undegraded":"2024-03-28T00:55:37.559613+0000","last_fullsized":"2024-03-28T00:55:37.559613+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:20:17.397594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.568088+0000","last_change":"2024-03-28T00:54:36.826851+0000","last_active":"2024-03-28T00:55:37.568088+0000","last_peered":"2024-03-28T00:55:37.568088+0000","last_clean":"2024-03-28T00:55:37.568088+0000","last_became_active":"2024-03-28T00:54:36.826359+0000","last_became_peered":"2024-03-28T00:54:36.826359+0000","last_unstale":"2024-03-28T00:55:37.568088+0000","last_undegraded":"2024-03-28T00:55:37.568088+0000","last_fullsized":"2024-03-28T00:55:37.568088+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:27:48.857368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.562536+0000","last_change":"2024-03-28T00:54:32.251701+0000","last_active":"2024-03-28T00:55:37.562536+0000","last_peered":"2024-03-28T00:55:37.562536+0000","last_clean":"2024-03-28T00:55:37.562536+0000","last_became_active":"2024-03-28T00:54:32.251403+0000","last_became_peered":"2024-03-28T00:54:32.251403+0000","last_unstale":"2024-03-28T00:55:37.562536+0000","last_undegraded":"2024-03-28T00:55:37.562536+0000","last_fullsized":"2024-03-28T00:55:37.562536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:28:16.127643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918391+0000","last_change":"2024-03-28T00:54:32.806810+0000","last_active":"2024-03-28T00:55:37.918391+0000","last_peered":"2024-03-28T00:55:37.918391+0000","last_clean":"2024-03-28T00:55:37.918391+0000","last_became_active":"2024-03-28T00:54:32.806468+0000","last_became_peered":"2024-03-28T00:54:32.806468+0000","last_unstale":"2024-03-28T00:55:37.918391+0000","last_undegraded":"2024-03-28T00:55:37.918391+0000","last_fullsized":"2024-03-28T00:55:37.918391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:54:16.196032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.919873+0000","last_change":"2024-03-28T00:55:37.919873+0000","last_active":"2024-03-28T00:55:37.919873+0000","last_peered":"2024-03-28T00:55:37.919873+0000","last_clean":"2024-03-28T00:55:37.919873+0000","last_became_active":"2024-03-28T00:55:37.919396+0000","last_became_peered":"2024-03-28T00:55:37.919396+0000","last_unstale":"2024-03-28T00:55:37.919873+0000","last_undegraded":"2024-03-28T00:55:37.919873+0000","last_fullsized":"2024-03-28T00:55:37.919873+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:05:59.524285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.568254+0000","last_change":"2024-03-28T00:54:32.259440+0000","last_active":"2024-03-28T00:55:37.568254+0000","last_peered":"2024-03-28T00:55:37.568254+0000","last_clean":"2024-03-28T00:55:37.568254+0000","last_became_active":"2024-03-28T00:54:32.258225+0000","last_became_peered":"2024-03-28T00:54:32.258225+0000","last_unstale":"2024-03-28T00:55:37.568254+0000","last_undegraded":"2024-03-28T00:55:37.568254+0000","last_fullsized":"2024-03-28T00:55:37.568254+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:07:18.398302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559996+0000","last_change":"2024-03-28T00:54:32.806853+0000","last_active":"2024-03-28T00:55:37.559996+0000","last_peered":"2024-03-28T00:55:37.559996+0000","last_clean":"2024-03-28T00:55:37.559996+0000","last_became_active":"2024-03-28T00:54:32.806551+0000","last_became_peered":"2024-03-28T00:54:32.806551+0000","last_unstale":"2024-03-28T00:55:37.559996+0000","last_undegraded":"2024-03-28T00:55:37.559996+0000","last_fullsized":"2024-03-28T00:55:37.559996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:49:45.774252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.558659+0000","last_change":"2024-03-28T00:54:36.827542+0000","last_active":"2024-03-28T00:55:37.558659+0000","last_peered":"2024-03-28T00:55:37.558659+0000","last_clean":"2024-03-28T00:55:37.558659+0000","last_became_active":"2024-03-28T00:54:36.827163+0000","last_became_peered":"2024-03-28T00:54:36.827163+0000","last_unstale":"2024-03-28T00:55:37.558659+0000","last_undegraded":"2024-03-28T00:55:37.558659+0000","last_fullsized":"2024-03-28T00:55:37.558659+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:36:15.278111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'3","reported_seq":103,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:59.350625+0000","last_change":"2024-03-28T00:54:32.253231+0000","last_active":"2024-03-28T00:55:59.350625+0000","last_peered":"2024-03-28T00:55:59.350625+0000","last_clean":"2024-03-28T00:55:59.350625+0000","last_became_active":"2024-03-28T00:54:32.252731+0000","last_became_peered":"2024-03-28T00:54:32.252731+0000","last_unstale":"2024-03-28T00:55:59.350625+0000","last_undegraded":"2024-03-28T00:55:59.350625+0000","last_fullsized":"2024-03-28T00:55:59.350625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:05:01.716654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560503+0000","last_change":"2024-03-28T00:54:32.805073+0000","last_active":"2024-03-28T00:55:37.560503+0000","last_peered":"2024-03-28T00:55:37.560503+0000","last_clean":"2024-03-28T00:55:37.560503+0000","last_became_active":"2024-03-28T00:54:32.803690+0000","last_became_peered":"2024-03-28T00:54:32.803690+0000","last_unstale":"2024-03-28T00:55:37.560503+0000","last_undegraded":"2024-03-28T00:55:37.560503+0000","last_fullsized":"2024-03-28T00:55:37.560503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:41:16.586168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918498+0000","last_change":"2024-03-28T00:54:36.835463+0000","last_active":"2024-03-28T00:55:37.918498+0000","last_peered":"2024-03-28T00:55:37.918498+0000","last_clean":"2024-03-28T00:55:37.918498+0000","last_became_active":"2024-03-28T00:54:36.834982+0000","last_became_peered":"2024-03-28T00:54:36.834982+0000","last_unstale":"2024-03-28T00:55:37.918498+0000","last_undegraded":"2024-03-28T00:55:37.918498+0000","last_fullsized":"2024-03-28T00:55:37.918498+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:27:32.259586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559446+0000","last_change":"2024-03-28T00:54:32.248461+0000","last_active":"2024-03-28T00:55:37.559446+0000","last_peered":"2024-03-28T00:55:37.559446+0000","last_clean":"2024-03-28T00:55:37.559446+0000","last_became_active":"2024-03-28T00:54:32.248140+0000","last_became_peered":"2024-03-28T00:54:32.248140+0000","last_unstale":"2024-03-28T00:55:37.559446+0000","last_undegraded":"2024-03-28T00:55:37.559446+0000","last_fullsized":"2024-03-28T00:55:37.559446+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:13:38.784288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.567018+0000","last_change":"2024-03-28T00:54:32.806651+0000","last_active":"2024-03-28T00:55:37.567018+0000","last_peered":"2024-03-28T00:55:37.567018+0000","last_clean":"2024-03-28T00:55:37.567018+0000","last_became_active":"2024-03-28T00:54:32.806329+0000","last_became_peered":"2024-03-28T00:54:32.806329+0000","last_unstale":"2024-03-28T00:55:37.567018+0000","last_undegraded":"2024-03-28T00:55:37.567018+0000","last_fullsized":"2024-03-28T00:55:37.567018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:28:32.815396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.566934+0000","last_change":"2024-03-28T00:54:36.827016+0000","last_active":"2024-03-28T00:55:37.566934+0000","last_peered":"2024-03-28T00:55:37.566934+0000","last_clean":"2024-03-28T00:55:37.566934+0000","last_became_active":"2024-03-28T00:54:36.826428+0000","last_became_peered":"2024-03-28T00:54:36.826428+0000","last_unstale":"2024-03-28T00:55:37.566934+0000","last_undegraded":"2024-03-28T00:55:37.566934+0000","last_fullsized":"2024-03-28T00:55:37.566934+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:33:01.593990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.573875+0000","last_change":"2024-03-28T00:55:37.573420+0000","last_active":"2024-03-28T00:55:37.573875+0000","last_peered":"2024-03-28T00:55:37.573875+0000","last_clean":"2024-03-28T00:55:37.573875+0000","last_became_active":"2024-03-28T00:55:37.568657+0000","last_became_peered":"2024-03-28T00:55:37.568657+0000","last_unstale":"2024-03-28T00:55:37.573875+0000","last_undegraded":"2024-03-28T00:55:37.573875+0000","last_fullsized":"2024-03-28T00:55:37.573875+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:03:21.483054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.919698+0000","last_change":"2024-03-28T00:55:37.919698+0000","last_active":"2024-03-28T00:55:37.919698+0000","last_peered":"2024-03-28T00:55:37.919698+0000","last_clean":"2024-03-28T00:55:37.919698+0000","last_became_active":"2024-03-28T00:55:37.919276+0000","last_became_peered":"2024-03-28T00:55:37.919276+0000","last_unstale":"2024-03-28T00:55:37.919698+0000","last_undegraded":"2024-03-28T00:55:37.919698+0000","last_fullsized":"2024-03-28T00:55:37.919698+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:12:39.042250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918600+0000","last_change":"2024-03-28T00:55:37.918600+0000","last_active":"2024-03-28T00:55:37.918600+0000","last_peered":"2024-03-28T00:55:37.918600+0000","last_clean":"2024-03-28T00:55:37.918600+0000","last_became_active":"2024-03-28T00:55:37.918088+0000","last_became_peered":"2024-03-28T00:55:37.918088+0000","last_unstale":"2024-03-28T00:55:37.918600+0000","last_undegraded":"2024-03-28T00:55:37.918600+0000","last_fullsized":"2024-03-28T00:55:37.918600+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:29:23.492261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.558694+0000","last_change":"2024-03-28T00:54:32.264038+0000","last_active":"2024-03-28T00:55:37.558694+0000","last_peered":"2024-03-28T00:55:37.558694+0000","last_clean":"2024-03-28T00:55:37.558694+0000","last_became_active":"2024-03-28T00:54:32.263706+0000","last_became_peered":"2024-03-28T00:54:32.263706+0000","last_unstale":"2024-03-28T00:55:37.558694+0000","last_undegraded":"2024-03-28T00:55:37.558694+0000","last_fullsized":"2024-03-28T00:55:37.558694+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:03.244556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917653+0000","last_change":"2024-03-28T00:54:32.804727+0000","last_active":"2024-03-28T00:55:37.917653+0000","last_peered":"2024-03-28T00:55:37.917653+0000","last_clean":"2024-03-28T00:55:37.917653+0000","last_became_active":"2024-03-28T00:54:32.804422+0000","last_became_peered":"2024-03-28T00:54:32.804422+0000","last_unstale":"2024-03-28T00:55:37.917653+0000","last_undegraded":"2024-03-28T00:55:37.917653+0000","last_fullsized":"2024-03-28T00:55:37.917653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:29:57.983722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918350+0000","last_change":"2024-03-28T00:54:36.833416+0000","last_active":"2024-03-28T00:55:37.918350+0000","last_peered":"2024-03-28T00:55:37.918350+0000","last_clean":"2024-03-28T00:55:37.918350+0000","last_became_active":"2024-03-28T00:54:36.832985+0000","last_became_peered":"2024-03-28T00:54:36.832985+0000","last_unstale":"2024-03-28T00:55:37.918350+0000","last_undegraded":"2024-03-28T00:55:37.918350+0000","last_fullsized":"2024-03-28T00:55:37.918350+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:47:30.376032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560050+0000","last_change":"2024-03-28T00:54:32.256417+0000","last_active":"2024-03-28T00:55:37.560050+0000","last_peered":"2024-03-28T00:55:37.560050+0000","last_clean":"2024-03-28T00:55:37.560050+0000","last_became_active":"2024-03-28T00:54:32.256088+0000","last_became_peered":"2024-03-28T00:54:32.256088+0000","last_unstale":"2024-03-28T00:55:37.560050+0000","last_undegraded":"2024-03-28T00:55:37.560050+0000","last_fullsized":"2024-03-28T00:55:37.560050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:58.661597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917228+0000","last_change":"2024-03-28T00:54:32.817604+0000","last_active":"2024-03-28T00:55:37.917228+0000","last_peered":"2024-03-28T00:55:37.917228+0000","last_clean":"2024-03-28T00:55:37.917228+0000","last_became_active":"2024-03-28T00:54:32.817211+0000","last_became_peered":"2024-03-28T00:54:32.817211+0000","last_unstale":"2024-03-28T00:55:37.917228+0000","last_undegraded":"2024-03-28T00:55:37.917228+0000","last_fullsized":"2024-03-28T00:55:37.917228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:02:28.335075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918570+0000","last_change":"2024-03-28T00:54:37.638968+0000","last_active":"2024-03-28T00:55:37.918570+0000","last_peered":"2024-03-28T00:55:37.918570+0000","last_clean":"2024-03-28T00:55:37.918570+0000","last_became_active":"2024-03-28T00:54:37.638446+0000","last_became_peered":"2024-03-28T00:54:37.638446+0000","last_unstale":"2024-03-28T00:55:37.918570+0000","last_undegraded":"2024-03-28T00:55:37.918570+0000","last_fullsized":"2024-03-28T00:55:37.918570+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:54:56.348760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.567023+0000","last_change":"2024-03-28T00:54:32.809822+0000","last_active":"2024-03-28T00:55:37.567023+0000","last_peered":"2024-03-28T00:55:37.567023+0000","last_clean":"2024-03-28T00:55:37.567023+0000","last_became_active":"2024-03-28T00:54:32.808284+0000","last_became_peered":"2024-03-28T00:54:32.808284+0000","last_unstale":"2024-03-28T00:55:37.567023+0000","last_undegraded":"2024-03-28T00:55:37.567023+0000","last_fullsized":"2024-03-28T00:55:37.567023+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:49:11.651960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":101,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:58.349353+0000","last_change":"2024-03-28T00:54:32.261806+0000","last_active":"2024-03-28T00:55:58.349353+0000","last_peered":"2024-03-28T00:55:58.349353+0000","last_clean":"2024-03-28T00:55:58.349353+0000","last_became_active":"2024-03-28T00:54:32.261493+0000","last_became_peered":"2024-03-28T00:54:32.261493+0000","last_unstale":"2024-03-28T00:55:58.349353+0000","last_undegraded":"2024-03-28T00:55:58.349353+0000","last_fullsized":"2024-03-28T00:55:58.349353+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:36:04.283408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":68,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560561+0000","last_change":"2024-03-28T00:54:36.819332+0000","last_active":"2024-03-28T00:55:37.560561+0000","last_peered":"2024-03-28T00:55:37.560561+0000","last_clean":"2024-03-28T00:55:37.560561+0000","last_became_active":"2024-03-28T00:54:36.818932+0000","last_became_peered":"2024-03-28T00:54:36.818932+0000","last_unstale":"2024-03-28T00:55:37.560561+0000","last_undegraded":"2024-03-28T00:55:37.560561+0000","last_fullsized":"2024-03-28T00:55:37.560561+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:00:39.446738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917573+0000","last_change":"2024-03-28T00:54:32.806093+0000","last_active":"2024-03-28T00:55:37.917573+0000","last_peered":"2024-03-28T00:55:37.917573+0000","last_clean":"2024-03-28T00:55:37.917573+0000","last_became_active":"2024-03-28T00:54:32.805734+0000","last_became_peered":"2024-03-28T00:54:32.805734+0000","last_unstale":"2024-03-28T00:55:37.917573+0000","last_undegraded":"2024-03-28T00:55:37.917573+0000","last_fullsized":"2024-03-28T00:55:37.917573+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:51:41.066225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559416+0000","last_change":"2024-03-28T00:54:32.245182+0000","last_active":"2024-03-28T00:55:37.559416+0000","last_peered":"2024-03-28T00:55:37.559416+0000","last_clean":"2024-03-28T00:55:37.559416+0000","last_became_active":"2024-03-28T00:54:32.244855+0000","last_became_peered":"2024-03-28T00:54:32.244855+0000","last_unstale":"2024-03-28T00:55:37.559416+0000","last_undegraded":"2024-03-28T00:55:37.559416+0000","last_fullsized":"2024-03-28T00:55:37.559416+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:39:09.997199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":352,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560071+0000","last_change":"2024-03-28T00:53:20.355628+0000","last_active":"2024-03-28T00:55:37.560071+0000","last_peered":"2024-03-28T00:55:37.560071+0000","last_clean":"2024-03-28T00:55:37.560071+0000","last_became_active":"2024-03-28T00:53:20.355175+0000","last_became_peered":"2024-03-28T00:53:20.355175+0000","last_unstale":"2024-03-28T00:55:37.560071+0000","last_undegraded":"2024-03-28T00:55:37.560071+0000","last_fullsized":"2024-03-28T00:55:37.560071+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:49:57.054435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:49:57.054435+0000","last_clean_scrub_stamp":"2024-03-28T00:49:57.054435+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:55:46.753799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.569168+0000","last_change":"2024-03-28T00:54:37.639760+0000","last_active":"2024-03-28T00:55:37.569168+0000","last_peered":"2024-03-28T00:55:37.569168+0000","last_clean":"2024-03-28T00:55:37.569168+0000","last_became_active":"2024-03-28T00:54:37.639287+0000","last_became_peered":"2024-03-28T00:54:37.639287+0000","last_unstale":"2024-03-28T00:55:37.569168+0000","last_undegraded":"2024-03-28T00:55:37.569168+0000","last_fullsized":"2024-03-28T00:55:37.569168+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:50:02.840312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917791+0000","last_change":"2024-03-28T00:54:32.807764+0000","last_active":"2024-03-28T00:55:37.917791+0000","last_peered":"2024-03-28T00:55:37.917791+0000","last_clean":"2024-03-28T00:55:37.917791+0000","last_became_active":"2024-03-28T00:54:32.807441+0000","last_became_peered":"2024-03-28T00:54:32.807441+0000","last_unstale":"2024-03-28T00:55:37.917791+0000","last_undegraded":"2024-03-28T00:55:37.917791+0000","last_fullsized":"2024-03-28T00:55:37.917791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:37:40.540514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560200+0000","last_change":"2024-03-28T00:54:32.259192+0000","last_active":"2024-03-28T00:55:37.560200+0000","last_peered":"2024-03-28T00:55:37.560200+0000","last_clean":"2024-03-28T00:55:37.560200+0000","last_became_active":"2024-03-28T00:54:32.258880+0000","last_became_peered":"2024-03-28T00:54:32.258880+0000","last_unstale":"2024-03-28T00:55:37.560200+0000","last_undegraded":"2024-03-28T00:55:37.560200+0000","last_fullsized":"2024-03-28T00:55:37.560200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:20:11.384633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560440+0000","last_change":"2024-03-28T00:54:36.822268+0000","last_active":"2024-03-28T00:55:37.560440+0000","last_peered":"2024-03-28T00:55:37.560440+0000","last_clean":"2024-03-28T00:55:37.560440+0000","last_became_active":"2024-03-28T00:54:36.821773+0000","last_became_peered":"2024-03-28T00:54:36.821773+0000","last_unstale":"2024-03-28T00:55:37.560440+0000","last_undegraded":"2024-03-28T00:55:37.560440+0000","last_fullsized":"2024-03-28T00:55:37.560440+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:56:11.206975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559722+0000","last_change":"2024-03-28T00:54:32.256133+0000","last_active":"2024-03-28T00:55:37.559722+0000","last_peered":"2024-03-28T00:55:37.559722+0000","last_clean":"2024-03-28T00:55:37.559722+0000","last_became_active":"2024-03-28T00:54:32.250535+0000","last_became_peered":"2024-03-28T00:54:32.250535+0000","last_unstale":"2024-03-28T00:55:37.559722+0000","last_undegraded":"2024-03-28T00:55:37.559722+0000","last_fullsized":"2024-03-28T00:55:37.559722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:18:50.874181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918606+0000","last_change":"2024-03-28T00:54:32.808883+0000","last_active":"2024-03-28T00:55:37.918606+0000","last_peered":"2024-03-28T00:55:37.918606+0000","last_clean":"2024-03-28T00:55:37.918606+0000","last_became_active":"2024-03-28T00:54:32.808587+0000","last_became_peered":"2024-03-28T00:54:32.808587+0000","last_unstale":"2024-03-28T00:55:37.918606+0000","last_undegraded":"2024-03-28T00:55:37.918606+0000","last_fullsized":"2024-03-28T00:55:37.918606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:09:55.164876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917403+0000","last_change":"2024-03-28T00:54:37.637743+0000","last_active":"2024-03-28T00:55:37.917403+0000","last_peered":"2024-03-28T00:55:37.917403+0000","last_clean":"2024-03-28T00:55:37.917403+0000","last_became_active":"2024-03-28T00:54:37.637280+0000","last_became_peered":"2024-03-28T00:54:37.637280+0000","last_unstale":"2024-03-28T00:55:37.917403+0000","last_undegraded":"2024-03-28T00:55:37.917403+0000","last_fullsized":"2024-03-28T00:55:37.917403+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:40:51.431757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'7","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.558852+0000","last_change":"2024-03-28T00:54:32.246444+0000","last_active":"2024-03-28T00:55:37.558852+0000","last_peered":"2024-03-28T00:55:37.558852+0000","last_clean":"2024-03-28T00:55:37.558852+0000","last_became_active":"2024-03-28T00:54:32.245106+0000","last_became_peered":"2024-03-28T00:54:32.245106+0000","last_unstale":"2024-03-28T00:55:37.558852+0000","last_undegraded":"2024-03-28T00:55:37.558852+0000","last_fullsized":"2024-03-28T00:55:37.558852+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:13:45.123625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.919052+0000","last_change":"2024-03-28T00:54:32.812851+0000","last_active":"2024-03-28T00:55:37.919052+0000","last_peered":"2024-03-28T00:55:37.919052+0000","last_clean":"2024-03-28T00:55:37.919052+0000","last_became_active":"2024-03-28T00:54:32.812518+0000","last_became_peered":"2024-03-28T00:54:32.812518+0000","last_unstale":"2024-03-28T00:55:37.919052+0000","last_undegraded":"2024-03-28T00:55:37.919052+0000","last_fullsized":"2024-03-28T00:55:37.919052+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:08:41.518888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.562736+0000","last_change":"2024-03-28T00:54:36.818709+0000","last_active":"2024-03-28T00:55:37.562736+0000","last_peered":"2024-03-28T00:55:37.562736+0000","last_clean":"2024-03-28T00:55:37.562736+0000","last_became_active":"2024-03-28T00:54:36.818304+0000","last_became_peered":"2024-03-28T00:54:36.818304+0000","last_unstale":"2024-03-28T00:55:37.562736+0000","last_undegraded":"2024-03-28T00:55:37.562736+0000","last_fullsized":"2024-03-28T00:55:37.562736+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:03:07.642685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559719+0000","last_change":"2024-03-28T00:54:32.256031+0000","last_active":"2024-03-28T00:55:37.559719+0000","last_peered":"2024-03-28T00:55:37.559719+0000","last_clean":"2024-03-28T00:55:37.559719+0000","last_became_active":"2024-03-28T00:54:32.255597+0000","last_became_peered":"2024-03-28T00:54:32.255597+0000","last_unstale":"2024-03-28T00:55:37.559719+0000","last_undegraded":"2024-03-28T00:55:37.559719+0000","last_fullsized":"2024-03-28T00:55:37.559719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:31:10.837212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.565632+0000","last_change":"2024-03-28T00:54:32.816961+0000","last_active":"2024-03-28T00:55:37.565632+0000","last_peered":"2024-03-28T00:55:37.565632+0000","last_clean":"2024-03-28T00:55:37.565632+0000","last_became_active":"2024-03-28T00:54:32.816408+0000","last_became_peered":"2024-03-28T00:54:32.816408+0000","last_unstale":"2024-03-28T00:55:37.565632+0000","last_undegraded":"2024-03-28T00:55:37.565632+0000","last_fullsized":"2024-03-28T00:55:37.565632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:08:57.441492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918692+0000","last_change":"2024-03-28T00:55:37.918692+0000","last_active":"2024-03-28T00:55:37.918692+0000","last_peered":"2024-03-28T00:55:37.918692+0000","last_clean":"2024-03-28T00:55:37.918692+0000","last_became_active":"2024-03-28T00:55:37.918195+0000","last_became_peered":"2024-03-28T00:55:37.918195+0000","last_unstale":"2024-03-28T00:55:37.918692+0000","last_undegraded":"2024-03-28T00:55:37.918692+0000","last_fullsized":"2024-03-28T00:55:37.918692+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:01:06.012325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.570700+0000","last_change":"2024-03-28T00:55:37.570700+0000","last_active":"2024-03-28T00:55:37.570700+0000","last_peered":"2024-03-28T00:55:37.570700+0000","last_clean":"2024-03-28T00:55:37.570700+0000","last_became_active":"2024-03-28T00:55:37.566298+0000","last_became_peered":"2024-03-28T00:55:37.566298+0000","last_unstale":"2024-03-28T00:55:37.570700+0000","last_undegraded":"2024-03-28T00:55:37.570700+0000","last_fullsized":"2024-03-28T00:55:37.570700+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:46:48.352817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.558967+0000","last_change":"2024-03-28T00:54:32.811592+0000","last_active":"2024-03-28T00:55:37.558967+0000","last_peered":"2024-03-28T00:55:37.558967+0000","last_clean":"2024-03-28T00:55:37.558967+0000","last_became_active":"2024-03-28T00:54:32.811228+0000","last_became_peered":"2024-03-28T00:54:32.811228+0000","last_unstale":"2024-03-28T00:55:37.558967+0000","last_undegraded":"2024-03-28T00:55:37.558967+0000","last_fullsized":"2024-03-28T00:55:37.558967+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:14:19.585237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.558764+0000","last_change":"2024-03-28T00:54:36.822501+0000","last_active":"2024-03-28T00:55:37.558764+0000","last_peered":"2024-03-28T00:55:37.558764+0000","last_clean":"2024-03-28T00:55:37.558764+0000","last_became_active":"2024-03-28T00:54:36.821630+0000","last_became_peered":"2024-03-28T00:54:36.821630+0000","last_unstale":"2024-03-28T00:55:37.558764+0000","last_undegraded":"2024-03-28T00:55:37.558764+0000","last_fullsized":"2024-03-28T00:55:37.558764+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:20:52.247055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917914+0000","last_change":"2024-03-28T00:54:32.247328+0000","last_active":"2024-03-28T00:55:37.917914+0000","last_peered":"2024-03-28T00:55:37.917914+0000","last_clean":"2024-03-28T00:55:37.917914+0000","last_became_active":"2024-03-28T00:54:32.247003+0000","last_became_peered":"2024-03-28T00:54:32.247003+0000","last_unstale":"2024-03-28T00:55:37.917914+0000","last_undegraded":"2024-03-28T00:55:37.917914+0000","last_fullsized":"2024-03-28T00:55:37.917914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:33:09.764946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560789+0000","last_change":"2024-03-28T00:54:32.805977+0000","last_active":"2024-03-28T00:55:37.560789+0000","last_peered":"2024-03-28T00:55:37.560789+0000","last_clean":"2024-03-28T00:55:37.560789+0000","last_became_active":"2024-03-28T00:54:32.805636+0000","last_became_peered":"2024-03-28T00:54:32.805636+0000","last_unstale":"2024-03-28T00:55:37.560789+0000","last_undegraded":"2024-03-28T00:55:37.560789+0000","last_fullsized":"2024-03-28T00:55:37.560789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:04:42.901732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559561+0000","last_change":"2024-03-28T00:54:36.822103+0000","last_active":"2024-03-28T00:55:37.559561+0000","last_peered":"2024-03-28T00:55:37.559561+0000","last_clean":"2024-03-28T00:55:37.559561+0000","last_became_active":"2024-03-28T00:54:36.821695+0000","last_became_peered":"2024-03-28T00:54:36.821695+0000","last_unstale":"2024-03-28T00:55:37.559561+0000","last_undegraded":"2024-03-28T00:55:37.559561+0000","last_fullsized":"2024-03-28T00:55:37.559561+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:45:00.613955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.562670+0000","last_change":"2024-03-28T00:54:32.251523+0000","last_active":"2024-03-28T00:55:37.562670+0000","last_peered":"2024-03-28T00:55:37.562670+0000","last_clean":"2024-03-28T00:55:37.562670+0000","last_became_active":"2024-03-28T00:54:32.251223+0000","last_became_peered":"2024-03-28T00:54:32.251223+0000","last_unstale":"2024-03-28T00:55:37.562670+0000","last_undegraded":"2024-03-28T00:55:37.562670+0000","last_fullsized":"2024-03-28T00:55:37.562670+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:08:20.489848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917787+0000","last_change":"2024-03-28T00:54:32.817641+0000","last_active":"2024-03-28T00:55:37.917787+0000","last_peered":"2024-03-28T00:55:37.917787+0000","last_clean":"2024-03-28T00:55:37.917787+0000","last_became_active":"2024-03-28T00:54:32.817252+0000","last_became_peered":"2024-03-28T00:54:32.817252+0000","last_unstale":"2024-03-28T00:55:37.917787+0000","last_undegraded":"2024-03-28T00:55:37.917787+0000","last_fullsized":"2024-03-28T00:55:37.917787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:54:48.275005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559331+0000","last_change":"2024-03-28T00:54:37.638990+0000","last_active":"2024-03-28T00:55:37.559331+0000","last_peered":"2024-03-28T00:55:37.559331+0000","last_clean":"2024-03-28T00:55:37.559331+0000","last_became_active":"2024-03-28T00:54:37.638538+0000","last_became_peered":"2024-03-28T00:54:37.638538+0000","last_unstale":"2024-03-28T00:55:37.559331+0000","last_undegraded":"2024-03-28T00:55:37.559331+0000","last_fullsized":"2024-03-28T00:55:37.559331+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:28:10.791805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.559360+0000","last_change":"2024-03-28T00:54:32.253254+0000","last_active":"2024-03-28T00:55:37.559360+0000","last_peered":"2024-03-28T00:55:37.559360+0000","last_clean":"2024-03-28T00:55:37.559360+0000","last_became_active":"2024-03-28T00:54:32.252879+0000","last_became_peered":"2024-03-28T00:54:32.252879+0000","last_unstale":"2024-03-28T00:55:37.559360+0000","last_undegraded":"2024-03-28T00:55:37.559360+0000","last_fullsized":"2024-03-28T00:55:37.559360+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:59:51.039215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560179+0000","last_change":"2024-03-28T00:54:32.811288+0000","last_active":"2024-03-28T00:55:37.560179+0000","last_peered":"2024-03-28T00:55:37.560179+0000","last_clean":"2024-03-28T00:55:37.560179+0000","last_became_active":"2024-03-28T00:54:32.810968+0000","last_became_peered":"2024-03-28T00:54:32.810968+0000","last_unstale":"2024-03-28T00:55:37.560179+0000","last_undegraded":"2024-03-28T00:55:37.560179+0000","last_fullsized":"2024-03-28T00:55:37.560179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:16:56.113046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563140+0000","last_change":"2024-03-28T00:54:36.830714+0000","last_active":"2024-03-28T00:55:37.563140+0000","last_peered":"2024-03-28T00:55:37.563140+0000","last_clean":"2024-03-28T00:55:37.563140+0000","last_became_active":"2024-03-28T00:54:36.830253+0000","last_became_peered":"2024-03-28T00:54:36.830253+0000","last_unstale":"2024-03-28T00:55:37.563140+0000","last_undegraded":"2024-03-28T00:55:37.563140+0000","last_fullsized":"2024-03-28T00:55:37.563140+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:24:43.696150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563132+0000","last_change":"2024-03-28T00:54:32.254009+0000","last_active":"2024-03-28T00:55:37.563132+0000","last_peered":"2024-03-28T00:55:37.563132+0000","last_clean":"2024-03-28T00:55:37.563132+0000","last_became_active":"2024-03-28T00:54:32.253697+0000","last_became_peered":"2024-03-28T00:54:32.253697+0000","last_unstale":"2024-03-28T00:55:37.563132+0000","last_undegraded":"2024-03-28T00:55:37.563132+0000","last_fullsized":"2024-03-28T00:55:37.563132+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:17:44.629367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560857+0000","last_change":"2024-03-28T00:54:32.812828+0000","last_active":"2024-03-28T00:55:37.560857+0000","last_peered":"2024-03-28T00:55:37.560857+0000","last_clean":"2024-03-28T00:55:37.560857+0000","last_became_active":"2024-03-28T00:54:32.811591+0000","last_became_peered":"2024-03-28T00:54:32.811591+0000","last_unstale":"2024-03-28T00:55:37.560857+0000","last_undegraded":"2024-03-28T00:55:37.560857+0000","last_fullsized":"2024-03-28T00:55:37.560857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:04.613623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.565201+0000","last_change":"2024-03-28T00:54:36.823284+0000","last_active":"2024-03-28T00:55:37.565201+0000","last_peered":"2024-03-28T00:55:37.565201+0000","last_clean":"2024-03-28T00:55:37.565201+0000","last_became_active":"2024-03-28T00:54:36.822883+0000","last_became_peered":"2024-03-28T00:54:36.822883+0000","last_unstale":"2024-03-28T00:55:37.565201+0000","last_undegraded":"2024-03-28T00:55:37.565201+0000","last_fullsized":"2024-03-28T00:55:37.565201+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:37:10.266063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.567943+0000","last_change":"2024-03-28T00:54:32.253946+0000","last_active":"2024-03-28T00:55:37.567943+0000","last_peered":"2024-03-28T00:55:37.567943+0000","last_clean":"2024-03-28T00:55:37.567943+0000","last_became_active":"2024-03-28T00:54:32.253650+0000","last_became_peered":"2024-03-28T00:54:32.253650+0000","last_unstale":"2024-03-28T00:55:37.567943+0000","last_undegraded":"2024-03-28T00:55:37.567943+0000","last_fullsized":"2024-03-28T00:55:37.567943+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:11:44.930074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560346+0000","last_change":"2024-03-28T00:54:32.815410+0000","last_active":"2024-03-28T00:55:37.560346+0000","last_peered":"2024-03-28T00:55:37.560346+0000","last_clean":"2024-03-28T00:55:37.560346+0000","last_became_active":"2024-03-28T00:54:32.815088+0000","last_became_peered":"2024-03-28T00:54:32.815088+0000","last_unstale":"2024-03-28T00:55:37.560346+0000","last_undegraded":"2024-03-28T00:55:37.560346+0000","last_fullsized":"2024-03-28T00:55:37.560346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:16:24.862822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.566363+0000","last_change":"2024-03-28T00:54:36.823182+0000","last_active":"2024-03-28T00:55:37.566363+0000","last_peered":"2024-03-28T00:55:37.566363+0000","last_clean":"2024-03-28T00:55:37.566363+0000","last_became_active":"2024-03-28T00:54:36.821446+0000","last_became_peered":"2024-03-28T00:54:36.821446+0000","last_unstale":"2024-03-28T00:55:37.566363+0000","last_undegraded":"2024-03-28T00:55:37.566363+0000","last_fullsized":"2024-03-28T00:55:37.566363+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:45:20.030213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918503+0000","last_change":"2024-03-28T00:54:32.257412+0000","last_active":"2024-03-28T00:55:37.918503+0000","last_peered":"2024-03-28T00:55:37.918503+0000","last_clean":"2024-03-28T00:55:37.918503+0000","last_became_active":"2024-03-28T00:54:32.257115+0000","last_became_peered":"2024-03-28T00:54:32.257115+0000","last_unstale":"2024-03-28T00:55:37.918503+0000","last_undegraded":"2024-03-28T00:55:37.918503+0000","last_fullsized":"2024-03-28T00:55:37.918503+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:04:29.850867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560733+0000","last_change":"2024-03-28T00:54:32.805879+0000","last_active":"2024-03-28T00:55:37.560733+0000","last_peered":"2024-03-28T00:55:37.560733+0000","last_clean":"2024-03-28T00:55:37.560733+0000","last_became_active":"2024-03-28T00:54:32.805507+0000","last_became_peered":"2024-03-28T00:54:32.805507+0000","last_unstale":"2024-03-28T00:55:37.560733+0000","last_undegraded":"2024-03-28T00:55:37.560733+0000","last_fullsized":"2024-03-28T00:55:37.560733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:51:51.229085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560889+0000","last_change":"2024-03-28T00:54:32.807219+0000","last_active":"2024-03-28T00:55:37.560889+0000","last_peered":"2024-03-28T00:55:37.560889+0000","last_clean":"2024-03-28T00:55:37.560889+0000","last_became_active":"2024-03-28T00:54:32.806856+0000","last_became_peered":"2024-03-28T00:54:32.806856+0000","last_unstale":"2024-03-28T00:55:37.560889+0000","last_undegraded":"2024-03-28T00:55:37.560889+0000","last_fullsized":"2024-03-28T00:55:37.560889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:05:49.914636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.569156+0000","last_change":"2024-03-28T00:55:37.569156+0000","last_active":"2024-03-28T00:55:37.569156+0000","last_peered":"2024-03-28T00:55:37.569156+0000","last_clean":"2024-03-28T00:55:37.569156+0000","last_became_active":"2024-03-28T00:55:37.568775+0000","last_became_peered":"2024-03-28T00:55:37.568775+0000","last_unstale":"2024-03-28T00:55:37.569156+0000","last_undegraded":"2024-03-28T00:55:37.569156+0000","last_fullsized":"2024-03-28T00:55:37.569156+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:00:21.458550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.562565+0000","last_change":"2024-03-28T00:54:32.813921+0000","last_active":"2024-03-28T00:55:37.562565+0000","last_peered":"2024-03-28T00:55:37.562565+0000","last_clean":"2024-03-28T00:55:37.562565+0000","last_became_active":"2024-03-28T00:54:32.812877+0000","last_became_peered":"2024-03-28T00:54:32.812877+0000","last_unstale":"2024-03-28T00:55:37.562565+0000","last_undegraded":"2024-03-28T00:55:37.562565+0000","last_fullsized":"2024-03-28T00:55:37.562565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:26:54.305074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.565481+0000","last_change":"2024-03-28T00:54:32.815725+0000","last_active":"2024-03-28T00:55:37.565481+0000","last_peered":"2024-03-28T00:55:37.565481+0000","last_clean":"2024-03-28T00:55:37.565481+0000","last_became_active":"2024-03-28T00:54:32.815444+0000","last_became_peered":"2024-03-28T00:54:32.815444+0000","last_unstale":"2024-03-28T00:55:37.565481+0000","last_undegraded":"2024-03-28T00:55:37.565481+0000","last_fullsized":"2024-03-28T00:55:37.565481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:23:17.326589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918293+0000","last_change":"2024-03-28T00:54:32.806595+0000","last_active":"2024-03-28T00:55:37.918293+0000","last_peered":"2024-03-28T00:55:37.918293+0000","last_clean":"2024-03-28T00:55:37.918293+0000","last_became_active":"2024-03-28T00:54:32.806256+0000","last_became_peered":"2024-03-28T00:54:32.806256+0000","last_unstale":"2024-03-28T00:55:37.918293+0000","last_undegraded":"2024-03-28T00:55:37.918293+0000","last_fullsized":"2024-03-28T00:55:37.918293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:04:22.551511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.560849+0000","last_change":"2024-03-28T00:54:32.805000+0000","last_active":"2024-03-28T00:55:37.560849+0000","last_peered":"2024-03-28T00:55:37.560849+0000","last_clean":"2024-03-28T00:55:37.560849+0000","last_became_active":"2024-03-28T00:54:32.804637+0000","last_became_peered":"2024-03-28T00:54:32.804637+0000","last_unstale":"2024-03-28T00:55:37.560849+0000","last_undegraded":"2024-03-28T00:55:37.560849+0000","last_fullsized":"2024-03-28T00:55:37.560849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:59:46.888014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563618+0000","last_change":"2024-03-28T00:54:32.802783+0000","last_active":"2024-03-28T00:55:37.563618+0000","last_peered":"2024-03-28T00:55:37.563618+0000","last_clean":"2024-03-28T00:55:37.563618+0000","last_became_active":"2024-03-28T00:54:32.802457+0000","last_became_peered":"2024-03-28T00:54:32.802457+0000","last_unstale":"2024-03-28T00:55:37.563618+0000","last_undegraded":"2024-03-28T00:55:37.563618+0000","last_fullsized":"2024-03-28T00:55:37.563618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:28:24.146047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.919114+0000","last_change":"2024-03-28T00:54:32.810179+0000","last_active":"2024-03-28T00:55:37.919114+0000","last_peered":"2024-03-28T00:55:37.919114+0000","last_clean":"2024-03-28T00:55:37.919114+0000","last_became_active":"2024-03-28T00:54:32.809009+0000","last_became_peered":"2024-03-28T00:54:32.809009+0000","last_unstale":"2024-03-28T00:55:37.919114+0000","last_undegraded":"2024-03-28T00:55:37.919114+0000","last_fullsized":"2024-03-28T00:55:37.919114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:08:44.184910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.566932+0000","last_change":"2024-03-28T00:54:32.810084+0000","last_active":"2024-03-28T00:55:37.566932+0000","last_peered":"2024-03-28T00:55:37.566932+0000","last_clean":"2024-03-28T00:55:37.566932+0000","last_became_active":"2024-03-28T00:54:32.809386+0000","last_became_peered":"2024-03-28T00:54:32.809386+0000","last_unstale":"2024-03-28T00:55:37.566932+0000","last_undegraded":"2024-03-28T00:55:37.566932+0000","last_fullsized":"2024-03-28T00:55:37.566932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:49:13.754797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.917587+0000","last_change":"2024-03-28T00:54:32.809933+0000","last_active":"2024-03-28T00:55:37.917587+0000","last_peered":"2024-03-28T00:55:37.917587+0000","last_clean":"2024-03-28T00:55:37.917587+0000","last_became_active":"2024-03-28T00:54:32.808974+0000","last_became_peered":"2024-03-28T00:54:32.808974+0000","last_unstale":"2024-03-28T00:55:37.917587+0000","last_undegraded":"2024-03-28T00:55:37.917587+0000","last_fullsized":"2024-03-28T00:55:37.917587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:23:59.317887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.563436+0000","last_change":"2024-03-28T00:54:32.808764+0000","last_active":"2024-03-28T00:55:37.563436+0000","last_peered":"2024-03-28T00:55:37.563436+0000","last_clean":"2024-03-28T00:55:37.563436+0000","last_became_active":"2024-03-28T00:54:32.808034+0000","last_became_peered":"2024-03-28T00:54:32.808034+0000","last_unstale":"2024-03-28T00:55:37.563436+0000","last_undegraded":"2024-03-28T00:55:37.563436+0000","last_fullsized":"2024-03-28T00:55:37.563436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:27:12.762520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.918624+0000","last_change":"2024-03-28T00:54:32.805470+0000","last_active":"2024-03-28T00:55:37.918624+0000","last_peered":"2024-03-28T00:55:37.918624+0000","last_clean":"2024-03-28T00:55:37.918624+0000","last_became_active":"2024-03-28T00:54:32.805146+0000","last_became_peered":"2024-03-28T00:54:32.805146+0000","last_unstale":"2024-03-28T00:55:37.918624+0000","last_undegraded":"2024-03-28T00:55:37.918624+0000","last_fullsized":"2024-03-28T00:55:37.918624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:24:14.297966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T00:55:37.566696+0000","last_change":"2024-03-28T00:55:37.566696+0000","last_active":"2024-03-28T00:55:37.566696+0000","last_peered":"2024-03-28T00:55:37.566696+0000","last_clean":"2024-03-28T00:55:37.566696+0000","last_became_active":"2024-03-28T00:55:37.566334+0000","last_became_peered":"2024-03-28T00:55:37.566334+0000","last_unstale":"2024-03-28T00:55:37.566696+0000","last_undegraded":"2024-03-28T00:55:37.566696+0000","last_fullsized":"2024-03-28T00:55:37.566696+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:41:37.752201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282084,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70745,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":441,"num_read_kb":294,"num_write":160,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":212235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383712,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1011712,"data_stored":510244,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645349,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32312,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712840,"statfs":{"total":95995035648,"available":95961948160,"internally_reserved":0,"allocated":1159168,"data_stored":1164234,"data_compressed":59464,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808873,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":514554,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070511,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":510356,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332148,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1073152,"data_stored":1106287,"data_compressed":59464,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495672,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1003520,"data_stored":505634,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1077248,"data_stored":572288,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738454,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":507022,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":574029,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182471,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":505828,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920835,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":510284,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757310,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":1101875,"data_compressed":59464,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34218,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34218,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34218,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":66744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":68485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":68322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T00:56:06.498 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-28T00:56:06.498 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-28T00:56:06.498 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-28T00:56:06.498 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-28T00:56:06.740 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:56:07.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:07 smithi169 ceph-mon[32274]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:56:07.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:07 smithi169 ceph-mon[32274]: pgmap v337: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T00:56:07.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:07 smithi174 ceph-mon[36060]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:56:07.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:07 smithi174 ceph-mon[36060]: pgmap v337: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T00:56:07.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:07 smithi177 ceph-mon[36206]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:56:07.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:07 smithi177 ceph-mon[36206]: pgmap v337: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T00:56:08.431 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:08 smithi169 ceph-mon[32274]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-28T00:56:08.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.526+0000 7fc7dc0a8700 1 -- 172.21.15.169:0/3281308358 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7d4100ac0 msgr2=0x7fc7d4100f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:08.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.526+0000 7fc7dc0a8700 1 --2- 172.21.15.169:0/3281308358 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7d4100ac0 0x7fc7d4100f40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8009990 tx=0x7fc7c8009ca0 comp rx=0 tx=0).stop 2024-03-28T00:56:08.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.527+0000 7fc7dc0a8700 1 -- 172.21.15.169:0/3281308358 shutdown_connections 2024-03-28T00:56:08.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.527+0000 7fc7dc0a8700 1 --2- 172.21.15.169:0/3281308358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7d4101480 0x7fc7d410d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:08.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.527+0000 7fc7dc0a8700 1 --2- 172.21.15.169:0/3281308358 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d41068a0 0x7fc7d4106c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:08.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.527+0000 7fc7dc0a8700 1 --2- 172.21.15.169:0/3281308358 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7d4100ac0 0x7fc7d4100f40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:08.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.527+0000 7fc7dc0a8700 1 -- 172.21.15.169:0/3281308358 >> 172.21.15.169:0/3281308358 conn(0x7fc7d40763c0 msgr2=0x7fc7d40767d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:08.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.527+0000 7fc7dc0a8700 1 -- 172.21.15.169:0/3281308358 shutdown_connections 2024-03-28T00:56:08.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.527+0000 7fc7dc0a8700 1 -- 172.21.15.169:0/3281308358 wait complete. 2024-03-28T00:56:08.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.527+0000 7fc7dc0a8700 1 Processor -- start 2024-03-28T00:56:08.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.527+0000 7fc7dc0a8700 1 -- start start 2024-03-28T00:56:08.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7dc0a8700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7d4100ac0 0x7fc7d419e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:08.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7dc0a8700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d4101480 0x7fc7d419eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:08.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7dc0a8700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7d41068a0 0x7fc7d4198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:08.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7dc0a8700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d41102b0 con 0x7fc7d41068a0 2024-03-28T00:56:08.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7dc0a8700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d4110130 con 0x7fc7d4101480 2024-03-28T00:56:08.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7dc0a8700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d4110430 con 0x7fc7d4100ac0 2024-03-28T00:56:08.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7da645700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7d41068a0 0x7fc7d4198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:08.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7d9e44700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7d4100ac0 0x7fc7d419e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:08.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7d9e44700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7d4100ac0 0x7fc7d419e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:44186/0 (socket says 172.21.15.169:44186) 2024-03-28T00:56:08.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7d9e44700 1 -- 172.21.15.169:0/3723233288 learned_addr learned my addr 172.21.15.169:0/3723233288 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:08.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7d9643700 1 --2- 172.21.15.169:0/3723233288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d4101480 0x7fc7d419eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:08.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7da645700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7d41068a0 0x7fc7d4198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38066/0 (socket says 172.21.15.169:38066) 2024-03-28T00:56:08.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7d9e44700 1 -- 172.21.15.169:0/3723233288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d4101480 msgr2=0x7fc7d419eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:08.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7d9e44700 1 --2- 172.21.15.169:0/3723233288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d4101480 0x7fc7d419eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:08.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7d9e44700 1 -- 172.21.15.169:0/3723233288 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7d41068a0 msgr2=0x7fc7d4198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:08.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7d9e44700 1 --2- 172.21.15.169:0/3723233288 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7d41068a0 0x7fc7d4198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:08.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7d9e44700 1 -- 172.21.15.169:0/3723233288 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d4198fe0 con 0x7fc7d4100ac0 2024-03-28T00:56:08.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7d9e44700 1 --2- 172.21.15.169:0/3723233288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7d4100ac0 0x7fc7d419e630 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc009fb0 tx=0x7fc7cc0076c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:08.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7d2ffd700 1 -- 172.21.15.169:0/3723233288 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc004bb0 con 0x7fc7d4100ac0 2024-03-28T00:56:08.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7d2ffd700 1 -- 172.21.15.169:0/3723233288 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7cc004d50 con 0x7fc7d4100ac0 2024-03-28T00:56:08.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7dc0a8700 1 -- 172.21.15.169:0/3723233288 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d41992d0 con 0x7fc7d4100ac0 2024-03-28T00:56:08.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7d2ffd700 1 -- 172.21.15.169:0/3723233288 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc005350 con 0x7fc7d4100ac0 2024-03-28T00:56:08.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.528+0000 7fc7dc0a8700 1 -- 172.21.15.169:0/3723233288 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d41a55b0 con 0x7fc7d4100ac0 2024-03-28T00:56:08.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.529+0000 7fc7dc0a8700 1 -- 172.21.15.169:0/3723233288 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d4101f30 con 0x7fc7d4100ac0 2024-03-28T00:56:08.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.530+0000 7fc7d2ffd700 1 -- 172.21.15.169:0/3723233288 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc7cc01b020 con 0x7fc7d4100ac0 2024-03-28T00:56:08.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.531+0000 7fc7d2ffd700 1 --2- 172.21.15.169:0/3723233288 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc7bc0a2890 0x7fc7bc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:08.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.531+0000 7fc7d9643700 1 --2- 172.21.15.169:0/3723233288 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc7bc0a2890 0x7fc7bc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:08.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.531+0000 7fc7d2ffd700 1 -- 172.21.15.169:0/3723233288 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fc7cc0ceb40 con 0x7fc7d4100ac0 2024-03-28T00:56:08.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.531+0000 7fc7d9643700 1 --2- 172.21.15.169:0/3723233288 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc7bc0a2890 0x7fc7bc0a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4199ec0 tx=0x7fc7c800b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:08.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.534+0000 7fc7d2ffd700 1 -- 172.21.15.169:0/3723233288 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc7cc011070 con 0x7fc7d4100ac0 2024-03-28T00:56:08.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:08 smithi174 ceph-mon[36060]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-28T00:56:08.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:08 smithi177 ceph-mon[36206]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-28T00:56:08.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.775+0000 7fc7dc0a8700 1 -- 172.21.15.169:0/3723233288 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc7d4199c30 con 0x7fc7d4100ac0 2024-03-28T00:56:08.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.776+0000 7fc7d2ffd700 1 -- 172.21.15.169:0/3723233288 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fc7cc09b880 con 0x7fc7d4100ac0 2024-03-28T00:56:08.779 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:08.779 INFO:teuthology.orchestra.run.smithi169.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-28T00:56:08.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.778+0000 7fc7dc0a8700 1 -- 172.21.15.169:0/3723233288 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc7bc0a2890 msgr2=0x7fc7bc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:08.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.778+0000 7fc7dc0a8700 1 --2- 172.21.15.169:0/3723233288 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc7bc0a2890 0x7fc7bc0a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4199ec0 tx=0x7fc7c800b280 comp rx=0 tx=0).stop 2024-03-28T00:56:08.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.778+0000 7fc7dc0a8700 1 -- 172.21.15.169:0/3723233288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7d4100ac0 msgr2=0x7fc7d419e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:08.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.778+0000 7fc7dc0a8700 1 --2- 172.21.15.169:0/3723233288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7d4100ac0 0x7fc7d419e630 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc009fb0 tx=0x7fc7cc0076c0 comp rx=0 tx=0).stop 2024-03-28T00:56:08.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.778+0000 7fc7dc0a8700 1 -- 172.21.15.169:0/3723233288 shutdown_connections 2024-03-28T00:56:08.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.778+0000 7fc7dc0a8700 1 --2- 172.21.15.169:0/3723233288 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc7d41068a0 0x7fc7d4198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:08.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.778+0000 7fc7dc0a8700 1 --2- 172.21.15.169:0/3723233288 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc7bc0a2890 0x7fc7bc0a4d50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:08.782 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.778+0000 7fc7dc0a8700 1 --2- 172.21.15.169:0/3723233288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7d4100ac0 0x7fc7d419e630 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:08.782 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.778+0000 7fc7dc0a8700 1 --2- 172.21.15.169:0/3723233288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d4101480 0x7fc7d419eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:08.782 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.778+0000 7fc7dc0a8700 1 -- 172.21.15.169:0/3723233288 >> 172.21.15.169:0/3723233288 conn(0x7fc7d40763c0 msgr2=0x7fc7d40fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:08.782 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.778+0000 7fc7dc0a8700 1 -- 172.21.15.169:0/3723233288 shutdown_connections 2024-03-28T00:56:08.782 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:08.778+0000 7fc7dc0a8700 1 -- 172.21.15.169:0/3723233288 wait complete. 2024-03-28T00:56:09.495 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:09 smithi169 ceph-mon[32274]: pgmap v338: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T00:56:09.495 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:09 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3723233288' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T00:56:09.523 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-28T00:56:09.524 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T00:56:09.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:09 smithi174 ceph-mon[36060]: pgmap v338: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T00:56:09.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:09 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3723233288' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T00:56:09.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.628+0000 7fc0080b2640 1 -- 172.21.15.169:0/333039048 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0000756e0 msgr2=0x7fc000073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:09.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.628+0000 7fc0080b2640 1 --2- 172.21.15.169:0/333039048 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0000756e0 0x7fc000073b30 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fbffc009930 tx=0x7fbffc02bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:09.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.628+0000 7fc0080b2640 1 -- 172.21.15.169:0/333039048 shutdown_connections 2024-03-28T00:56:09.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.628+0000 7fc0080b2640 1 --2- 172.21.15.169:0/333039048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc00010dc80 0x7fc000110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:09.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.628+0000 7fc0080b2640 1 --2- 172.21.15.169:0/333039048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc000074100 0x7fc000074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:09.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.628+0000 7fc0080b2640 1 --2- 172.21.15.169:0/333039048 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0000756e0 0x7fc000073b30 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:09.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.628+0000 7fc0080b2640 1 -- 172.21.15.169:0/333039048 >> 172.21.15.169:0/333039048 conn(0x7fc0000fcc80 msgr2=0x7fc0000ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:09.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.628+0000 7fc0080b2640 1 -- 172.21.15.169:0/333039048 shutdown_connections 2024-03-28T00:56:09.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.628+0000 7fc0080b2640 1 -- 172.21.15.169:0/333039048 wait complete. 2024-03-28T00:56:09.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.628+0000 7fc0080b2640 1 Processor -- start 2024-03-28T00:56:09.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.628+0000 7fc0080b2640 1 -- start start 2024-03-28T00:56:09.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fc0080b2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc000074100 0x7fc00019dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:09.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fc0080b2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0000756e0 0x7fc00019e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:09.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fc0080b2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc00010dc80 0x7fc0001a55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:09.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fc0080b2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc000113970 con 0x7fc0000756e0 2024-03-28T00:56:09.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fc0080b2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0001137f0 con 0x7fc000074100 2024-03-28T00:56:09.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fc0080b2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc000113af0 con 0x7fc00010dc80 2024-03-28T00:56:09.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fc00564d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0000756e0 0x7fc00019e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:09.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fc00564d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0000756e0 0x7fc00019e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38084/0 (socket says 172.21.15.169:38084) 2024-03-28T00:56:09.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fc00564d640 1 -- 172.21.15.169:0/2984015557 learned_addr learned my addr 172.21.15.169:0/2984015557 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:09.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fc005e4e640 1 --2- 172.21.15.169:0/2984015557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc000074100 0x7fc00019dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:09.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fc00664f640 1 --2- 172.21.15.169:0/2984015557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc00010dc80 0x7fc0001a55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:09.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fc00564d640 1 -- 172.21.15.169:0/2984015557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc00010dc80 msgr2=0x7fc0001a55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:09.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fc00564d640 1 --2- 172.21.15.169:0/2984015557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc00010dc80 0x7fc0001a55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:09.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fc00564d640 1 -- 172.21.15.169:0/2984015557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc000074100 msgr2=0x7fc00019dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:09.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fc00564d640 1 --2- 172.21.15.169:0/2984015557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc000074100 0x7fc00019dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:09.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fc00564d640 1 -- 172.21.15.169:0/2984015557 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0001a5c10 con 0x7fc0000756e0 2024-03-28T00:56:09.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fc00564d640 1 --2- 172.21.15.169:0/2984015557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0000756e0 0x7fc00019e530 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fbff000e9f0 tx=0x7fbff000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:09.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fbff6ffd640 1 -- 172.21.15.169:0/2984015557 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff000cd80 con 0x7fc0000756e0 2024-03-28T00:56:09.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fbff6ffd640 1 -- 172.21.15.169:0/2984015557 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbff0005500 con 0x7fc0000756e0 2024-03-28T00:56:09.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fc0080b2640 1 -- 172.21.15.169:0/2984015557 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0001a5f00 con 0x7fc0000756e0 2024-03-28T00:56:09.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fbff6ffd640 1 -- 172.21.15.169:0/2984015557 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff0010690 con 0x7fc0000756e0 2024-03-28T00:56:09.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.629+0000 7fc0080b2640 1 -- 172.21.15.169:0/2984015557 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0001a6370 con 0x7fc0000756e0 2024-03-28T00:56:09.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.630+0000 7fc0080b2640 1 -- 172.21.15.169:0/2984015557 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc000103e50 con 0x7fc0000756e0 2024-03-28T00:56:09.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.631+0000 7fbff6ffd640 1 -- 172.21.15.169:0/2984015557 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbff00046c0 con 0x7fc0000756e0 2024-03-28T00:56:09.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.632+0000 7fbff6ffd640 1 --2- 172.21.15.169:0/2984015557 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbfec0a2800 0x7fbfec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:09.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.632+0000 7fc005e4e640 1 --2- 172.21.15.169:0/2984015557 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbfec0a2800 0x7fbfec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:09.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.632+0000 7fbff6ffd640 1 -- 172.21.15.169:0/2984015557 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fbff00cf190 con 0x7fc0000756e0 2024-03-28T00:56:09.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.633+0000 7fc005e4e640 1 --2- 172.21.15.169:0/2984015557 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbfec0a2800 0x7fbfec0a4cc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbffc036040 tx=0x7fbffc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:09.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.636+0000 7fbff6ffd640 1 -- 172.21.15.169:0/2984015557 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbff009bed0 con 0x7fc0000756e0 2024-03-28T00:56:09.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:09 smithi177 ceph-mon[36206]: pgmap v338: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T00:56:09.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:09 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3723233288' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T00:56:09.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.813+0000 7fc0080b2640 1 -- 172.21.15.169:0/2984015557 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc000062340 con 0x7fc0000756e0 2024-03-28T00:56:09.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.815+0000 7fbff6ffd640 1 -- 172.21.15.169:0/2984015557 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+24711 (secure 0 0 0) 0x7fbff009b7d0 con 0x7fc0000756e0 2024-03-28T00:56:09.818 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:09.818 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":92,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:55:37.555451+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:56:09.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.817+0000 7fc0080b2640 1 -- 172.21.15.169:0/2984015557 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbfec0a2800 msgr2=0x7fbfec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:09.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.817+0000 7fc0080b2640 1 --2- 172.21.15.169:0/2984015557 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbfec0a2800 0x7fbfec0a4cc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbffc036040 tx=0x7fbffc02df90 comp rx=0 tx=0).stop 2024-03-28T00:56:09.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.817+0000 7fc0080b2640 1 -- 172.21.15.169:0/2984015557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0000756e0 msgr2=0x7fc00019e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:09.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.817+0000 7fc0080b2640 1 --2- 172.21.15.169:0/2984015557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0000756e0 0x7fc00019e530 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fbff000e9f0 tx=0x7fbff000edf0 comp rx=0 tx=0).stop 2024-03-28T00:56:09.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.817+0000 7fc0080b2640 1 -- 172.21.15.169:0/2984015557 shutdown_connections 2024-03-28T00:56:09.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.817+0000 7fc0080b2640 1 --2- 172.21.15.169:0/2984015557 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbfec0a2800 0x7fbfec0a4cc0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:09.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.817+0000 7fc0080b2640 1 --2- 172.21.15.169:0/2984015557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc00010dc80 0x7fc0001a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:09.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.817+0000 7fc0080b2640 1 --2- 172.21.15.169:0/2984015557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0000756e0 0x7fc00019e530 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:09.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.817+0000 7fc0080b2640 1 --2- 172.21.15.169:0/2984015557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc000074100 0x7fc00019dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:09.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.817+0000 7fc0080b2640 1 -- 172.21.15.169:0/2984015557 >> 172.21.15.169:0/2984015557 conn(0x7fc0000fcc80 msgr2=0x7fc000101020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:09.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.817+0000 7fc0080b2640 1 -- 172.21.15.169:0/2984015557 shutdown_connections 2024-03-28T00:56:09.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.817+0000 7fc0080b2640 1 -- 172.21.15.169:0/2984015557 wait complete. 2024-03-28T00:56:09.831 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T00:49:56.339525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T00:54:31.182562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T00:54:31.780429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T00:54:35.035724+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T00:56:09.832 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T00:56:09.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.934+0000 7fa2c1dcd640 1 -- 172.21.15.169:0/1672302526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2bc1073a0 msgr2=0x7fa2bc1097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:09.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.934+0000 7fa2c1dcd640 1 --2- 172.21.15.169:0/1672302526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2bc1073a0 0x7fa2bc1097f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4009ca0 tx=0x7fa2b402c0f0 comp rx=0 tx=0).stop 2024-03-28T00:56:09.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.934+0000 7fa2c1dcd640 1 -- 172.21.15.169:0/1672302526 shutdown_connections 2024-03-28T00:56:09.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.934+0000 7fa2c1dcd640 1 --2- 172.21.15.169:0/1672302526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2bc1073a0 0x7fa2bc1097f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:09.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.934+0000 7fa2c1dcd640 1 --2- 172.21.15.169:0/1672302526 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2bc104a70 0x7fa2bc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:09.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.934+0000 7fa2c1dcd640 1 --2- 172.21.15.169:0/1672302526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2bc102140 0x7fa2bc104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:09.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.934+0000 7fa2c1dcd640 1 -- 172.21.15.169:0/1672302526 >> 172.21.15.169:0/1672302526 conn(0x7fa2bc0fbb50 msgr2=0x7fa2bc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:09.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.934+0000 7fa2c1dcd640 1 -- 172.21.15.169:0/1672302526 shutdown_connections 2024-03-28T00:56:09.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.934+0000 7fa2c1dcd640 1 -- 172.21.15.169:0/1672302526 wait complete. 2024-03-28T00:56:09.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.934+0000 7fa2c1dcd640 1 Processor -- start 2024-03-28T00:56:09.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.934+0000 7fa2c1dcd640 1 -- start start 2024-03-28T00:56:09.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.935+0000 7fa2c1dcd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2bc102140 0x7fa2bc197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:09.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.935+0000 7fa2c1dcd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2bc104a70 0x7fa2bc197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:09.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.935+0000 7fa2c1dcd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2bc1073a0 0x7fa2bc19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:09.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.935+0000 7fa2c1dcd640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc10d360 con 0x7fa2bc104a70 2024-03-28T00:56:09.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.935+0000 7fa2c1dcd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc10d1e0 con 0x7fa2bc102140 2024-03-28T00:56:09.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.935+0000 7fa2c1dcd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc10d4e0 con 0x7fa2bc1073a0 2024-03-28T00:56:09.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.935+0000 7fa2b2ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2bc104a70 0x7fa2bc197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:09.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.935+0000 7fa2b2ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2bc104a70 0x7fa2bc197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38098/0 (socket says 172.21.15.169:38098) 2024-03-28T00:56:09.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.935+0000 7fa2b2ffd640 1 -- 172.21.15.169:0/1217064713 learned_addr learned my addr 172.21.15.169:0/1217064713 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:09.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.935+0000 7fa2bbfff640 1 --2- 172.21.15.169:0/1217064713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2bc1073a0 0x7fa2bc19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:09.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.935+0000 7fa2bb7fe640 1 --2- 172.21.15.169:0/1217064713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2bc102140 0x7fa2bc197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:09.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.935+0000 7fa2b2ffd640 1 -- 172.21.15.169:0/1217064713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2bc1073a0 msgr2=0x7fa2bc19ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:09.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.935+0000 7fa2b2ffd640 1 --2- 172.21.15.169:0/1217064713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2bc1073a0 0x7fa2bc19ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:09.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.935+0000 7fa2b2ffd640 1 -- 172.21.15.169:0/1217064713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2bc102140 msgr2=0x7fa2bc197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:09.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.935+0000 7fa2b2ffd640 1 --2- 172.21.15.169:0/1217064713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2bc102140 0x7fa2bc197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:09.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.935+0000 7fa2b2ffd640 1 -- 172.21.15.169:0/1217064713 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2bc19f5d0 con 0x7fa2bc104a70 2024-03-28T00:56:09.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.935+0000 7fa2b2ffd640 1 --2- 172.21.15.169:0/1217064713 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2bc104a70 0x7fa2bc197f40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa2a400e9f0 tx=0x7fa2a400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:09.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.935+0000 7fa2b97fa640 1 -- 172.21.15.169:0/1217064713 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a400ccc0 con 0x7fa2bc104a70 2024-03-28T00:56:09.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.936+0000 7fa2b97fa640 1 -- 172.21.15.169:0/1217064713 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2a4005500 con 0x7fa2bc104a70 2024-03-28T00:56:09.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.935+0000 7fa2c1dcd640 1 -- 172.21.15.169:0/1217064713 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2bc19f8c0 con 0x7fa2bc104a70 2024-03-28T00:56:09.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.936+0000 7fa2b97fa640 1 -- 172.21.15.169:0/1217064713 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4010690 con 0x7fa2bc104a70 2024-03-28T00:56:09.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.936+0000 7fa2c1dcd640 1 -- 172.21.15.169:0/1217064713 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2bc19fdc0 con 0x7fa2bc104a70 2024-03-28T00:56:09.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.937+0000 7fa2c1dcd640 1 -- 172.21.15.169:0/1217064713 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2bc0fd640 con 0x7fa2bc104a70 2024-03-28T00:56:09.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.938+0000 7fa2b97fa640 1 -- 172.21.15.169:0/1217064713 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa2a40046c0 con 0x7fa2bc104a70 2024-03-28T00:56:09.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.938+0000 7fa2b97fa640 1 --2- 172.21.15.169:0/1217064713 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa2a80a2800 0x7fa2a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:09.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.938+0000 7fa2bb7fe640 1 --2- 172.21.15.169:0/1217064713 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa2a80a2800 0x7fa2a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:09.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.939+0000 7fa2b97fa640 1 -- 172.21.15.169:0/1217064713 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fa2a40cf120 con 0x7fa2bc104a70 2024-03-28T00:56:09.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.939+0000 7fa2bb7fe640 1 --2- 172.21.15.169:0/1217064713 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa2a80a2800 0x7fa2a80a4cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac009bd0 tx=0x7fa2ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:09.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:09.943+0000 7fa2b97fa640 1 -- 172.21.15.169:0/1217064713 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa2a409be60 con 0x7fa2bc104a70 2024-03-28T00:56:10.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.121+0000 7fa2c1dcd640 1 -- 172.21.15.169:0/1217064713 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa2bc0622a0 con 0x7fa2bc104a70 2024-03-28T00:56:10.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.121+0000 7fa2b97fa640 1 -- 172.21.15.169:0/1217064713 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa2a409b760 con 0x7fa2bc104a70 2024-03-28T00:56:10.124 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 1 2024-03-28T00:56:10.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.123+0000 7fa2c1dcd640 1 -- 172.21.15.169:0/1217064713 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa2a80a2800 msgr2=0x7fa2a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:10.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.123+0000 7fa2c1dcd640 1 --2- 172.21.15.169:0/1217064713 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa2a80a2800 0x7fa2a80a4cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac009bd0 tx=0x7fa2ac005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:10.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.123+0000 7fa2c1dcd640 1 -- 172.21.15.169:0/1217064713 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2bc104a70 msgr2=0x7fa2bc197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:10.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.123+0000 7fa2c1dcd640 1 --2- 172.21.15.169:0/1217064713 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2bc104a70 0x7fa2bc197f40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa2a400e9f0 tx=0x7fa2a400eeb0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.123+0000 7fa2c1dcd640 1 -- 172.21.15.169:0/1217064713 shutdown_connections 2024-03-28T00:56:10.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.123+0000 7fa2c1dcd640 1 --2- 172.21.15.169:0/1217064713 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa2a80a2800 0x7fa2a80a4cc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.123+0000 7fa2c1dcd640 1 --2- 172.21.15.169:0/1217064713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2bc1073a0 0x7fa2bc19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.123+0000 7fa2c1dcd640 1 --2- 172.21.15.169:0/1217064713 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2bc104a70 0x7fa2bc197f40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.123+0000 7fa2c1dcd640 1 --2- 172.21.15.169:0/1217064713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2bc102140 0x7fa2bc197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.123+0000 7fa2c1dcd640 1 -- 172.21.15.169:0/1217064713 >> 172.21.15.169:0/1217064713 conn(0x7fa2bc0fbb50 msgr2=0x7fa2bc102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:10.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.123+0000 7fa2c1dcd640 1 -- 172.21.15.169:0/1217064713 shutdown_connections 2024-03-28T00:56:10.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.123+0000 7fa2c1dcd640 1 -- 172.21.15.169:0/1217064713 wait complete. 2024-03-28T00:56:10.137 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T00:56:10.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.240+0000 7fe0228b5640 1 -- 172.21.15.169:0/489097220 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe01c1017a0 msgr2=0x7fe01c101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:10.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.240+0000 7fe0228b5640 1 --2- 172.21.15.169:0/489097220 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe01c1017a0 0x7fe01c101c20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe010009900 tx=0x7fe01002bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:10.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.241+0000 7fe0228b5640 1 -- 172.21.15.169:0/489097220 shutdown_connections 2024-03-28T00:56:10.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.241+0000 7fe0228b5640 1 --2- 172.21.15.169:0/489097220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe01c102160 0x7fe01c1089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.241+0000 7fe0228b5640 1 --2- 172.21.15.169:0/489097220 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe01c1017a0 0x7fe01c101c20 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.241+0000 7fe0228b5640 1 --2- 172.21.15.169:0/489097220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe01c1005b0 0x7fe01c1009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.241+0000 7fe0228b5640 1 -- 172.21.15.169:0/489097220 >> 172.21.15.169:0/489097220 conn(0x7fe01c0fbb40 msgr2=0x7fe01c0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:10.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.241+0000 7fe0228b5640 1 -- 172.21.15.169:0/489097220 shutdown_connections 2024-03-28T00:56:10.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.241+0000 7fe0228b5640 1 -- 172.21.15.169:0/489097220 wait complete. 2024-03-28T00:56:10.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.241+0000 7fe0228b5640 1 Processor -- start 2024-03-28T00:56:10.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.241+0000 7fe0228b5640 1 -- start start 2024-03-28T00:56:10.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.241+0000 7fe0228b5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe01c1005b0 0x7fe01c1979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:10.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.241+0000 7fe0228b5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe01c1017a0 0x7fe01c197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:10.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.241+0000 7fe0228b5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe01c102160 0x7fe01c19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:10.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.241+0000 7fe0228b5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c10c210 con 0x7fe01c1017a0 2024-03-28T00:56:10.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.241+0000 7fe0228b5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c10c090 con 0x7fe01c1005b0 2024-03-28T00:56:10.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.241+0000 7fe0228b5640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c10c390 con 0x7fe01c102160 2024-03-28T00:56:10.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.242+0000 7fe01b7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe01c1017a0 0x7fe01c197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:10.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.242+0000 7fe01bfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe01c1005b0 0x7fe01c1979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:10.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.242+0000 7fe01b7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe01c1017a0 0x7fe01c197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38116/0 (socket says 172.21.15.169:38116) 2024-03-28T00:56:10.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.242+0000 7fe01b7fe640 1 -- 172.21.15.169:0/3655762676 learned_addr learned my addr 172.21.15.169:0/3655762676 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:10.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.242+0000 7fe020e52640 1 --2- 172.21.15.169:0/3655762676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe01c102160 0x7fe01c19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:10.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.242+0000 7fe01b7fe640 1 -- 172.21.15.169:0/3655762676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe01c102160 msgr2=0x7fe01c19ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:10.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.242+0000 7fe01b7fe640 1 --2- 172.21.15.169:0/3655762676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe01c102160 0x7fe01c19ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.242+0000 7fe01b7fe640 1 -- 172.21.15.169:0/3655762676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe01c1005b0 msgr2=0x7fe01c1979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:10.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.242+0000 7fe01b7fe640 1 --2- 172.21.15.169:0/3655762676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe01c1005b0 0x7fe01c1979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.242+0000 7fe01b7fe640 1 -- 172.21.15.169:0/3655762676 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01c19f5f0 con 0x7fe01c1017a0 2024-03-28T00:56:10.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.242+0000 7fe01bfff640 1 --2- 172.21.15.169:0/3655762676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe01c1005b0 0x7fe01c1979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:10.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.242+0000 7fe01b7fe640 1 --2- 172.21.15.169:0/3655762676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe01c1017a0 0x7fe01c197f10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe010009230 tx=0x7fe01002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:10.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.242+0000 7fe0197fa640 1 -- 172.21.15.169:0/3655762676 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe010041070 con 0x7fe01c1017a0 2024-03-28T00:56:10.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.242+0000 7fe0228b5640 1 -- 172.21.15.169:0/3655762676 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe01c19f880 con 0x7fe01c1017a0 2024-03-28T00:56:10.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.242+0000 7fe0197fa640 1 -- 172.21.15.169:0/3655762676 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe010004450 con 0x7fe01c1017a0 2024-03-28T00:56:10.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.242+0000 7fe0228b5640 1 -- 172.21.15.169:0/3655762676 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe01c19fd70 con 0x7fe01c1017a0 2024-03-28T00:56:10.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.242+0000 7fe0197fa640 1 -- 172.21.15.169:0/3655762676 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01004b060 con 0x7fe01c1017a0 2024-03-28T00:56:10.249 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.243+0000 7fe0228b5640 1 -- 172.21.15.169:0/3655762676 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe01c19ffc0 con 0x7fe01c1017a0 2024-03-28T00:56:10.249 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.244+0000 7fe0197fa640 1 -- 172.21.15.169:0/3655762676 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe010044070 con 0x7fe01c1017a0 2024-03-28T00:56:10.249 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.245+0000 7fe0197fa640 1 --2- 172.21.15.169:0/3655762676 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe0040a2800 0x7fe0040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:10.249 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.245+0000 7fe01bfff640 1 --2- 172.21.15.169:0/3655762676 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe0040a2800 0x7fe0040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:10.249 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.245+0000 7fe0197fa640 1 -- 172.21.15.169:0/3655762676 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fe0100f0c80 con 0x7fe01c1017a0 2024-03-28T00:56:10.249 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.245+0000 7fe01bfff640 1 --2- 172.21.15.169:0/3655762676 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe0040a2800 0x7fe0040a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe01c101600 tx=0x7fe00c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:10.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.249+0000 7fe0197fa640 1 -- 172.21.15.169:0/3655762676 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe010003e20 con 0x7fe01c1017a0 2024-03-28T00:56:10.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.426+0000 7fe0228b5640 1 -- 172.21.15.169:0/3655762676 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe01c1a0230 con 0x7fe01c1017a0 2024-03-28T00:56:10.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.426+0000 7fe0197fa640 1 -- 172.21.15.169:0/3655762676 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe0100bd370 con 0x7fe01c1017a0 2024-03-28T00:56:10.429 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T00:56:10.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.429+0000 7fe0228b5640 1 -- 172.21.15.169:0/3655762676 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe0040a2800 msgr2=0x7fe0040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:10.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.429+0000 7fe0228b5640 1 --2- 172.21.15.169:0/3655762676 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe0040a2800 0x7fe0040a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe01c101600 tx=0x7fe00c005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:10.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.429+0000 7fe0228b5640 1 -- 172.21.15.169:0/3655762676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe01c1017a0 msgr2=0x7fe01c197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:10.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.429+0000 7fe0228b5640 1 --2- 172.21.15.169:0/3655762676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe01c1017a0 0x7fe01c197f10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe010009230 tx=0x7fe01002c650 comp rx=0 tx=0).stop 2024-03-28T00:56:10.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.429+0000 7fe0228b5640 1 -- 172.21.15.169:0/3655762676 shutdown_connections 2024-03-28T00:56:10.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.429+0000 7fe0228b5640 1 --2- 172.21.15.169:0/3655762676 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe0040a2800 0x7fe0040a4cc0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.429+0000 7fe0228b5640 1 --2- 172.21.15.169:0/3655762676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe01c102160 0x7fe01c19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.429+0000 7fe0228b5640 1 --2- 172.21.15.169:0/3655762676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe01c1017a0 0x7fe01c197f10 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.429+0000 7fe0228b5640 1 --2- 172.21.15.169:0/3655762676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe01c1005b0 0x7fe01c1979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.429+0000 7fe0228b5640 1 -- 172.21.15.169:0/3655762676 >> 172.21.15.169:0/3655762676 conn(0x7fe01c0fbb40 msgr2=0x7fe01c069c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:10.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.429+0000 7fe0228b5640 1 -- 172.21.15.169:0/3655762676 shutdown_connections 2024-03-28T00:56:10.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.429+0000 7fe0228b5640 1 -- 172.21.15.169:0/3655762676 wait complete. 2024-03-28T00:56:10.442 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:10 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2984015557' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:10.442 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:10 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1217064713' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:56:10.443 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T00:56:10.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.545+0000 7f6402044640 1 -- 172.21.15.169:0/801636545 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63fc1028a0 msgr2=0x7f63fc102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:10.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.545+0000 7f6402044640 1 --2- 172.21.15.169:0/801636545 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63fc1028a0 0x7f63fc102d20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f63e4009900 tx=0x7f63e402bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:10.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.545+0000 7f6402044640 1 -- 172.21.15.169:0/801636545 shutdown_connections 2024-03-28T00:56:10.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.545+0000 7f6402044640 1 --2- 172.21.15.169:0/801636545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63fc103260 0x7f63fc109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.545+0000 7f6402044640 1 --2- 172.21.15.169:0/801636545 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63fc1028a0 0x7f63fc102d20 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.545+0000 7f6402044640 1 --2- 172.21.15.169:0/801636545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63fc1016b0 0x7f63fc101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.545+0000 7f6402044640 1 -- 172.21.15.169:0/801636545 >> 172.21.15.169:0/801636545 conn(0x7f63fc0fcc40 msgr2=0x7f63fc0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:10.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.545+0000 7f6402044640 1 -- 172.21.15.169:0/801636545 shutdown_connections 2024-03-28T00:56:10.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.545+0000 7f6402044640 1 -- 172.21.15.169:0/801636545 wait complete. 2024-03-28T00:56:10.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.546+0000 7f6402044640 1 Processor -- start 2024-03-28T00:56:10.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.546+0000 7f6402044640 1 -- start start 2024-03-28T00:56:10.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.546+0000 7f6402044640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63fc1016b0 0x7f63fc1979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:10.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.546+0000 7f6402044640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63fc1028a0 0x7f63fc197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:10.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.546+0000 7f6402044640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63fc103260 0x7f63fc19ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:10.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.546+0000 7f6402044640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc10d310 con 0x7f63fc103260 2024-03-28T00:56:10.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.546+0000 7f6402044640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc10d190 con 0x7f63fc1028a0 2024-03-28T00:56:10.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.546+0000 7f6402044640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc10d490 con 0x7f63fc1016b0 2024-03-28T00:56:10.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.546+0000 7f63fbfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63fc103260 0x7f63fc19ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:10.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.546+0000 7f63f2ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63fc1028a0 0x7f63fc197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:10.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.546+0000 7f63fbfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63fc103260 0x7f63fc19ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38134/0 (socket says 172.21.15.169:38134) 2024-03-28T00:56:10.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.546+0000 7f63fbfff640 1 -- 172.21.15.169:0/2913765114 learned_addr learned my addr 172.21.15.169:0/2913765114 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:10.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.546+0000 7f63fb7fe640 1 --2- 172.21.15.169:0/2913765114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63fc1016b0 0x7f63fc1979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:10.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.546+0000 7f63fbfff640 1 -- 172.21.15.169:0/2913765114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63fc1016b0 msgr2=0x7f63fc1979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:10.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.546+0000 7f63fbfff640 1 --2- 172.21.15.169:0/2913765114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63fc1016b0 0x7f63fc1979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.546+0000 7f63fbfff640 1 -- 172.21.15.169:0/2913765114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63fc1028a0 msgr2=0x7f63fc197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:10.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.546+0000 7f63fbfff640 1 --2- 172.21.15.169:0/2913765114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63fc1028a0 0x7f63fc197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.546+0000 7f63fbfff640 1 -- 172.21.15.169:0/2913765114 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63fc19f5c0 con 0x7f63fc103260 2024-03-28T00:56:10.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.546+0000 7f63f2ffd640 1 --2- 172.21.15.169:0/2913765114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63fc1028a0 0x7f63fc197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:10.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.547+0000 7f63fbfff640 1 --2- 172.21.15.169:0/2913765114 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63fc103260 0x7f63fc19ef50 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f63f400eb10 tx=0x7f63f400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:10.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.547+0000 7f63f97fa640 1 -- 172.21.15.169:0/2913765114 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4004b20 con 0x7f63fc103260 2024-03-28T00:56:10.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.547+0000 7f63f97fa640 1 -- 172.21.15.169:0/2913765114 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f63f4004cc0 con 0x7f63fc103260 2024-03-28T00:56:10.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.547+0000 7f6402044640 1 -- 172.21.15.169:0/2913765114 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63fc19f8b0 con 0x7f63fc103260 2024-03-28T00:56:10.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.547+0000 7f63f97fa640 1 -- 172.21.15.169:0/2913765114 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4010760 con 0x7f63fc103260 2024-03-28T00:56:10.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.547+0000 7f6402044640 1 -- 172.21.15.169:0/2913765114 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63fc19fd80 con 0x7f63fc103260 2024-03-28T00:56:10.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.548+0000 7f6402044640 1 -- 172.21.15.169:0/2913765114 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63fc108aa0 con 0x7f63fc103260 2024-03-28T00:56:10.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.549+0000 7f63f97fa640 1 -- 172.21.15.169:0/2913765114 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f63f4005000 con 0x7f63fc103260 2024-03-28T00:56:10.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.549+0000 7f63f97fa640 1 --2- 172.21.15.169:0/2913765114 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f63e80a2800 0x7f63e80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:10.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.549+0000 7f63fb7fe640 1 --2- 172.21.15.169:0/2913765114 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f63e80a2800 0x7f63e80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:10.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.549+0000 7f63f97fa640 1 -- 172.21.15.169:0/2913765114 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f63f40cf560 con 0x7f63fc103260 2024-03-28T00:56:10.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.550+0000 7f63fb7fe640 1 --2- 172.21.15.169:0/2913765114 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f63e80a2800 0x7f63e80a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f63ec0095f0 tx=0x7f63ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:10.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.553+0000 7f63f97fa640 1 -- 172.21.15.169:0/2913765114 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f63f4053740 con 0x7f63fc103260 2024-03-28T00:56:10.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:10 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2984015557' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:10.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:10 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1217064713' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:56:10.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:10 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2984015557' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:10.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:10 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1217064713' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:56:10.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.733+0000 7f6402044640 1 -- 172.21.15.169:0/2913765114 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f63fc02d200 con 0x7f63fc103260 2024-03-28T00:56:10.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.733+0000 7f63f97fa640 1 -- 172.21.15.169:0/2913765114 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7f63f409c2a0 con 0x7f63fc103260 2024-03-28T00:56:10.736 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 64 2024-03-28T00:56:10.738 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.735+0000 7f6402044640 1 -- 172.21.15.169:0/2913765114 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f63e80a2800 msgr2=0x7f63e80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:10.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.735+0000 7f6402044640 1 --2- 172.21.15.169:0/2913765114 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f63e80a2800 0x7f63e80a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f63ec0095f0 tx=0x7f63ec005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:10.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.735+0000 7f6402044640 1 -- 172.21.15.169:0/2913765114 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63fc103260 msgr2=0x7f63fc19ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:10.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.735+0000 7f6402044640 1 --2- 172.21.15.169:0/2913765114 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63fc103260 0x7f63fc19ef50 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f63f400eb10 tx=0x7f63f400c2d0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.736+0000 7f6402044640 1 -- 172.21.15.169:0/2913765114 shutdown_connections 2024-03-28T00:56:10.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.736+0000 7f6402044640 1 --2- 172.21.15.169:0/2913765114 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f63e80a2800 0x7f63e80a4cc0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.736+0000 7f6402044640 1 --2- 172.21.15.169:0/2913765114 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f63fc103260 0x7f63fc19ef50 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.736+0000 7f6402044640 1 --2- 172.21.15.169:0/2913765114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63fc1028a0 0x7f63fc197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.736+0000 7f6402044640 1 --2- 172.21.15.169:0/2913765114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63fc1016b0 0x7f63fc1979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.736+0000 7f6402044640 1 -- 172.21.15.169:0/2913765114 >> 172.21.15.169:0/2913765114 conn(0x7f63fc0fcc40 msgr2=0x7f63fc105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:10.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.736+0000 7f6402044640 1 -- 172.21.15.169:0/2913765114 shutdown_connections 2024-03-28T00:56:10.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.736+0000 7f6402044640 1 -- 172.21.15.169:0/2913765114 wait complete. 2024-03-28T00:56:10.750 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T00:56:10.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.852+0000 7f9b346ea640 1 -- 172.21.15.169:0/2613833213 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b2c100640 msgr2=0x7f9b2c100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:10.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.852+0000 7f9b346ea640 1 --2- 172.21.15.169:0/2613833213 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b2c100640 0x7f9b2c100a60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9b28009860 tx=0x7f9b2802bb70 comp rx=0 tx=0).stop 2024-03-28T00:56:10.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.852+0000 7f9b346ea640 1 -- 172.21.15.169:0/2613833213 shutdown_connections 2024-03-28T00:56:10.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.852+0000 7f9b346ea640 1 --2- 172.21.15.169:0/2613833213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b2c1021f0 0x7f9b2c108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.852+0000 7f9b346ea640 1 --2- 172.21.15.169:0/2613833213 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b2c101830 0x7f9b2c101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.852+0000 7f9b346ea640 1 --2- 172.21.15.169:0/2613833213 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b2c100640 0x7f9b2c100a60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.852+0000 7f9b346ea640 1 -- 172.21.15.169:0/2613833213 >> 172.21.15.169:0/2613833213 conn(0x7f9b2c0fbbb0 msgr2=0x7f9b2c0fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:10.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.852+0000 7f9b346ea640 1 -- 172.21.15.169:0/2613833213 shutdown_connections 2024-03-28T00:56:10.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.852+0000 7f9b346ea640 1 -- 172.21.15.169:0/2613833213 wait complete. 2024-03-28T00:56:10.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.853+0000 7f9b346ea640 1 Processor -- start 2024-03-28T00:56:10.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.853+0000 7f9b346ea640 1 -- start start 2024-03-28T00:56:10.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.853+0000 7f9b346ea640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b2c100640 0x7f9b2c196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:10.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.853+0000 7f9b346ea640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b2c101830 0x7f9b2c196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:10.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.853+0000 7f9b346ea640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b2c1021f0 0x7f9b2c19ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:10.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.853+0000 7f9b346ea640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c10c220 con 0x7f9b2c100640 2024-03-28T00:56:10.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.853+0000 7f9b346ea640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c10c0a0 con 0x7f9b2c101830 2024-03-28T00:56:10.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.853+0000 7f9b346ea640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c10c3a0 con 0x7f9b2c1021f0 2024-03-28T00:56:10.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.853+0000 7f9b32486640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b2c100640 0x7f9b2c196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:10.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.853+0000 7f9b32486640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b2c100640 0x7f9b2c196920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38150/0 (socket says 172.21.15.169:38150) 2024-03-28T00:56:10.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.853+0000 7f9b32486640 1 -- 172.21.15.169:0/3830480882 learned_addr learned my addr 172.21.15.169:0/3830480882 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:10.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.853+0000 7f9b32c87640 1 --2- 172.21.15.169:0/3830480882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b2c1021f0 0x7f9b2c19ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:10.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.853+0000 7f9b31c85640 1 --2- 172.21.15.169:0/3830480882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b2c101830 0x7f9b2c196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:10.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.853+0000 7f9b32486640 1 -- 172.21.15.169:0/3830480882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b2c1021f0 msgr2=0x7f9b2c19ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:10.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.853+0000 7f9b32486640 1 --2- 172.21.15.169:0/3830480882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b2c1021f0 0x7f9b2c19ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.853+0000 7f9b32486640 1 -- 172.21.15.169:0/3830480882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b2c101830 msgr2=0x7f9b2c196e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:10.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.853+0000 7f9b32486640 1 --2- 172.21.15.169:0/3830480882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b2c101830 0x7f9b2c196e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:10.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.853+0000 7f9b32486640 1 -- 172.21.15.169:0/3830480882 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2c069cd0 con 0x7f9b2c100640 2024-03-28T00:56:10.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.853+0000 7f9b32486640 1 --2- 172.21.15.169:0/3830480882 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b2c100640 0x7f9b2c196920 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9b2802c0b0 tx=0x7f9b28032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:10.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.853+0000 7f9b237fe640 1 -- 172.21.15.169:0/3830480882 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b28041020 con 0x7f9b2c100640 2024-03-28T00:56:10.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.853+0000 7f9b346ea640 1 -- 172.21.15.169:0/3830480882 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b2c069f60 con 0x7f9b2c100640 2024-03-28T00:56:10.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.854+0000 7f9b237fe640 1 -- 172.21.15.169:0/3830480882 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b2802c9f0 con 0x7f9b2c100640 2024-03-28T00:56:10.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.854+0000 7f9b237fe640 1 -- 172.21.15.169:0/3830480882 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b28039040 con 0x7f9b2c100640 2024-03-28T00:56:10.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.854+0000 7f9b346ea640 1 -- 172.21.15.169:0/3830480882 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b2c06a450 con 0x7f9b2c100640 2024-03-28T00:56:10.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.855+0000 7f9b346ea640 1 -- 172.21.15.169:0/3830480882 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b2c107a00 con 0x7f9b2c100640 2024-03-28T00:56:10.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.856+0000 7f9b237fe640 1 -- 172.21.15.169:0/3830480882 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9b28044020 con 0x7f9b2c100640 2024-03-28T00:56:10.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.856+0000 7f9b237fe640 1 --2- 172.21.15.169:0/3830480882 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9b180a2800 0x7f9b180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:10.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.856+0000 7f9b31c85640 1 --2- 172.21.15.169:0/3830480882 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9b180a2800 0x7f9b180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:10.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.856+0000 7f9b237fe640 1 -- 172.21.15.169:0/3830480882 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f9b280f06a0 con 0x7f9b2c100640 2024-03-28T00:56:10.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.857+0000 7f9b31c85640 1 --2- 172.21.15.169:0/3830480882 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9b180a2800 0x7f9b180a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c197e30 tx=0x7f9b1c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:10.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:10.861+0000 7f9b237fe640 1 -- 172.21.15.169:0/3830480882 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9b2804b020 con 0x7f9b2c100640 2024-03-28T00:56:11.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.036+0000 7f9b346ea640 1 -- 172.21.15.169:0/3830480882 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9b2c06a6b0 con 0x7f9b2c100640 2024-03-28T00:56:11.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.036+0000 7f9b237fe640 1 -- 172.21.15.169:0/3830480882 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9b280bd3e0 con 0x7f9b2c100640 2024-03-28T00:56:11.039 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T00:56:11.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.038+0000 7f9b346ea640 1 -- 172.21.15.169:0/3830480882 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9b180a2800 msgr2=0x7f9b180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:11.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.038+0000 7f9b346ea640 1 --2- 172.21.15.169:0/3830480882 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9b180a2800 0x7f9b180a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c197e30 tx=0x7f9b1c009260 comp rx=0 tx=0).stop 2024-03-28T00:56:11.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.038+0000 7f9b346ea640 1 -- 172.21.15.169:0/3830480882 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b2c100640 msgr2=0x7f9b2c196920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:11.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.038+0000 7f9b346ea640 1 --2- 172.21.15.169:0/3830480882 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b2c100640 0x7f9b2c196920 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9b2802c0b0 tx=0x7f9b28032430 comp rx=0 tx=0).stop 2024-03-28T00:56:11.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.038+0000 7f9b346ea640 1 -- 172.21.15.169:0/3830480882 shutdown_connections 2024-03-28T00:56:11.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.038+0000 7f9b346ea640 1 --2- 172.21.15.169:0/3830480882 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9b180a2800 0x7f9b180a4cc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.038+0000 7f9b346ea640 1 --2- 172.21.15.169:0/3830480882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b2c1021f0 0x7f9b2c19ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.038+0000 7f9b346ea640 1 --2- 172.21.15.169:0/3830480882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b2c101830 0x7f9b2c196e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.038+0000 7f9b346ea640 1 --2- 172.21.15.169:0/3830480882 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b2c100640 0x7f9b2c196920 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.038+0000 7f9b346ea640 1 -- 172.21.15.169:0/3830480882 >> 172.21.15.169:0/3830480882 conn(0x7f9b2c0fbbb0 msgr2=0x7f9b2c104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:11.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.038+0000 7f9b346ea640 1 -- 172.21.15.169:0/3830480882 shutdown_connections 2024-03-28T00:56:11.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.038+0000 7f9b346ea640 1 -- 172.21.15.169:0/3830480882 wait complete. 2024-03-28T00:56:11.053 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:56:11.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.152+0000 7f3f233cd640 1 -- 172.21.15.169:0/1664090010 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f1c1017d0 msgr2=0x7f3f1c101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:11.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.152+0000 7f3f233cd640 1 --2- 172.21.15.169:0/1664090010 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f1c1017d0 0x7f3f1c101c50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3f10009900 tx=0x7f3f1002bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:11.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.152+0000 7f3f233cd640 1 -- 172.21.15.169:0/1664090010 shutdown_connections 2024-03-28T00:56:11.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.152+0000 7f3f233cd640 1 --2- 172.21.15.169:0/1664090010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f1c102190 0x7f3f1c108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.152+0000 7f3f233cd640 1 --2- 172.21.15.169:0/1664090010 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f1c1017d0 0x7f3f1c101c50 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.152+0000 7f3f233cd640 1 --2- 172.21.15.169:0/1664090010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f1c1005e0 0x7f3f1c100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.152+0000 7f3f233cd640 1 -- 172.21.15.169:0/1664090010 >> 172.21.15.169:0/1664090010 conn(0x7f3f1c0fbb50 msgr2=0x7f3f1c0fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:11.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.153+0000 7f3f233cd640 1 -- 172.21.15.169:0/1664090010 shutdown_connections 2024-03-28T00:56:11.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.153+0000 7f3f233cd640 1 -- 172.21.15.169:0/1664090010 wait complete. 2024-03-28T00:56:11.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.153+0000 7f3f233cd640 1 Processor -- start 2024-03-28T00:56:11.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.153+0000 7f3f233cd640 1 -- start start 2024-03-28T00:56:11.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.153+0000 7f3f233cd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f1c1005e0 0x7f3f1c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:11.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.153+0000 7f3f233cd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f1c1017d0 0x7f3f1c19c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:11.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.153+0000 7f3f233cd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f1c102190 0x7f3f1c19edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:11.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.153+0000 7f3f233cd640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c10c240 con 0x7f3f1c102190 2024-03-28T00:56:11.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.153+0000 7f3f233cd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c10c0c0 con 0x7f3f1c1017d0 2024-03-28T00:56:11.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.153+0000 7f3f233cd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c10c3c0 con 0x7f3f1c1005e0 2024-03-28T00:56:11.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.153+0000 7f3f21169640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f1c1005e0 0x7f3f1c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:11.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.153+0000 7f3f2196a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f1c102190 0x7f3f1c19edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:11.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.153+0000 7f3f21169640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f1c1005e0 0x7f3f1c06a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:44286/0 (socket says 172.21.15.169:44286) 2024-03-28T00:56:11.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.153+0000 7f3f21169640 1 -- 172.21.15.169:0/740273870 learned_addr learned my addr 172.21.15.169:0/740273870 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:11.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.153+0000 7f3f20968640 1 --2- 172.21.15.169:0/740273870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f1c1017d0 0x7f3f1c19c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:11.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.154+0000 7f3f21169640 1 -- 172.21.15.169:0/740273870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f1c1017d0 msgr2=0x7f3f1c19c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:11.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.154+0000 7f3f21169640 1 --2- 172.21.15.169:0/740273870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f1c1017d0 0x7f3f1c19c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.154+0000 7f3f21169640 1 -- 172.21.15.169:0/740273870 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f1c102190 msgr2=0x7f3f1c19edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:11.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.154+0000 7f3f21169640 1 --2- 172.21.15.169:0/740273870 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f1c102190 0x7f3f1c19edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.154+0000 7f3f21169640 1 -- 172.21.15.169:0/740273870 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1c19f6d0 con 0x7f3f1c1005e0 2024-03-28T00:56:11.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.154+0000 7f3f20968640 1 --2- 172.21.15.169:0/740273870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f1c1017d0 0x7f3f1c19c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:11.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.154+0000 7f3f21169640 1 --2- 172.21.15.169:0/740273870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f1c1005e0 0x7f3f1c06a840 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3f1400ebc0 tx=0x7f3f1400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:11.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.154+0000 7f3f1a7fc640 1 -- 172.21.15.169:0/740273870 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f14003b70 con 0x7f3f1c1005e0 2024-03-28T00:56:11.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.154+0000 7f3f233cd640 1 -- 172.21.15.169:0/740273870 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f1c19f960 con 0x7f3f1c1005e0 2024-03-28T00:56:11.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.154+0000 7f3f1a7fc640 1 -- 172.21.15.169:0/740273870 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f14003d10 con 0x7f3f1c1005e0 2024-03-28T00:56:11.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.154+0000 7f3f1a7fc640 1 -- 172.21.15.169:0/740273870 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f14019980 con 0x7f3f1c1005e0 2024-03-28T00:56:11.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.154+0000 7f3f233cd640 1 -- 172.21.15.169:0/740273870 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f1c19fed0 con 0x7f3f1c1005e0 2024-03-28T00:56:11.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.155+0000 7f3f233cd640 1 -- 172.21.15.169:0/740273870 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f1c0688e0 con 0x7f3f1c1005e0 2024-03-28T00:56:11.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.156+0000 7f3f1a7fc640 1 -- 172.21.15.169:0/740273870 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3f14019b20 con 0x7f3f1c1005e0 2024-03-28T00:56:11.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.157+0000 7f3f1a7fc640 1 --2- 172.21.15.169:0/740273870 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3f040a2800 0x7f3f040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:11.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.157+0000 7f3f20968640 1 --2- 172.21.15.169:0/740273870 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3f040a2800 0x7f3f040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:11.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.157+0000 7f3f1a7fc640 1 -- 172.21.15.169:0/740273870 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f3f140cf6e0 con 0x7f3f1c1005e0 2024-03-28T00:56:11.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.157+0000 7f3f20968640 1 --2- 172.21.15.169:0/740273870 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3f040a2800 0x7f3f040a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3f10009230 tx=0x7f3f10005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:11.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.161+0000 7f3f1a7fc640 1 -- 172.21.15.169:0/740273870 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3f1409c4b0 con 0x7f3f1c1005e0 2024-03-28T00:56:11.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.375+0000 7f3f233cd640 1 -- 172.21.15.169:0/740273870 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f1c02d110 con 0x7f3f1c1005e0 2024-03-28T00:56:11.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.376+0000 7f3f1a7fc640 1 -- 172.21.15.169:0/740273870 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11420 (secure 0 0 0) 0x7f3f140a0e20 con 0x7f3f1c1005e0 2024-03-28T00:56:11.379 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:11.379 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:54:52.274335+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24457,24478]},"id":1}]} 2024-03-28T00:56:11.379 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 24 2024-03-28T00:56:11.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.378+0000 7f3f233cd640 1 -- 172.21.15.169:0/740273870 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3f040a2800 msgr2=0x7f3f040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:11.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.378+0000 7f3f233cd640 1 --2- 172.21.15.169:0/740273870 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3f040a2800 0x7f3f040a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3f10009230 tx=0x7f3f10005fd0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.378+0000 7f3f233cd640 1 -- 172.21.15.169:0/740273870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f1c1005e0 msgr2=0x7f3f1c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:11.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.378+0000 7f3f233cd640 1 --2- 172.21.15.169:0/740273870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f1c1005e0 0x7f3f1c06a840 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3f1400ebc0 tx=0x7f3f1400c6a0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.378+0000 7f3f233cd640 1 -- 172.21.15.169:0/740273870 shutdown_connections 2024-03-28T00:56:11.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.378+0000 7f3f233cd640 1 --2- 172.21.15.169:0/740273870 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3f040a2800 0x7f3f040a4cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.378+0000 7f3f233cd640 1 --2- 172.21.15.169:0/740273870 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f1c102190 0x7f3f1c19edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.378+0000 7f3f233cd640 1 --2- 172.21.15.169:0/740273870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f1c1017d0 0x7f3f1c19c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.378+0000 7f3f233cd640 1 --2- 172.21.15.169:0/740273870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f1c1005e0 0x7f3f1c06a840 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.378+0000 7f3f233cd640 1 -- 172.21.15.169:0/740273870 >> 172.21.15.169:0/740273870 conn(0x7f3f1c0fbb50 msgr2=0x7f3f1c104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:11.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.378+0000 7f3f233cd640 1 -- 172.21.15.169:0/740273870 shutdown_connections 2024-03-28T00:56:11.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.378+0000 7f3f233cd640 1 -- 172.21.15.169:0/740273870 wait complete. 2024-03-28T00:56:11.391 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:11 smithi169 ceph-mon[32274]: pgmap v339: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T00:56:11.391 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:11 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3655762676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:56:11.391 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:11 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2913765114' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:56:11.391 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:11 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3830480882' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:56:11.393 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T00:56:11.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.492+0000 7f3803c36640 1 -- 172.21.15.169:0/3256972554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37fc1005c0 msgr2=0x7f37fc1009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:11.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.492+0000 7f3803c36640 1 --2- 172.21.15.169:0/3256972554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37fc1005c0 0x7f37fc1009e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f37ec009900 tx=0x7f37ec02bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:11.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.492+0000 7f3803c36640 1 -- 172.21.15.169:0/3256972554 shutdown_connections 2024-03-28T00:56:11.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.492+0000 7f3803c36640 1 --2- 172.21.15.169:0/3256972554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37fc102170 0x7f37fc108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.492+0000 7f3803c36640 1 --2- 172.21.15.169:0/3256972554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37fc1017b0 0x7f37fc101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.492+0000 7f3803c36640 1 --2- 172.21.15.169:0/3256972554 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37fc1005c0 0x7f37fc1009e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.492+0000 7f3803c36640 1 -- 172.21.15.169:0/3256972554 >> 172.21.15.169:0/3256972554 conn(0x7f37fc0fbb30 msgr2=0x7f37fc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:11.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.493+0000 7f3803c36640 1 -- 172.21.15.169:0/3256972554 shutdown_connections 2024-03-28T00:56:11.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.493+0000 7f3803c36640 1 -- 172.21.15.169:0/3256972554 wait complete. 2024-03-28T00:56:11.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.493+0000 7f3803c36640 1 Processor -- start 2024-03-28T00:56:11.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.493+0000 7f3803c36640 1 -- start start 2024-03-28T00:56:11.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.493+0000 7f3803c36640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37fc1005c0 0x7f37fc197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:11.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.493+0000 7f3803c36640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37fc1017b0 0x7f37fc197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:11.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.493+0000 7f3803c36640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37fc102170 0x7f37fc19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:11.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.493+0000 7f3803c36640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc10c210 con 0x7f37fc1017b0 2024-03-28T00:56:11.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.493+0000 7f3803c36640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc10c090 con 0x7f37fc1005c0 2024-03-28T00:56:11.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.493+0000 7f3803c36640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc10c390 con 0x7f37fc102170 2024-03-28T00:56:11.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.493+0000 7f38011d1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37fc1017b0 0x7f37fc197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:11.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.493+0000 7f38011d1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37fc1017b0 0x7f37fc197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38194/0 (socket says 172.21.15.169:38194) 2024-03-28T00:56:11.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.493+0000 7f38011d1640 1 -- 172.21.15.169:0/3150521184 learned_addr learned my addr 172.21.15.169:0/3150521184 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:11.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.493+0000 7f38019d2640 1 --2- 172.21.15.169:0/3150521184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37fc1005c0 0x7f37fc197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:11.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.494+0000 7f38021d3640 1 --2- 172.21.15.169:0/3150521184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37fc102170 0x7f37fc19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:11.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.494+0000 7f38011d1640 1 -- 172.21.15.169:0/3150521184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37fc102170 msgr2=0x7f37fc19efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:11.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.494+0000 7f38011d1640 1 --2- 172.21.15.169:0/3150521184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37fc102170 0x7f37fc19efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.494+0000 7f38011d1640 1 -- 172.21.15.169:0/3150521184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37fc1005c0 msgr2=0x7f37fc197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:11.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.494+0000 7f38011d1640 1 --2- 172.21.15.169:0/3150521184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37fc1005c0 0x7f37fc197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.494+0000 7f38011d1640 1 -- 172.21.15.169:0/3150521184 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37fc19f630 con 0x7f37fc1017b0 2024-03-28T00:56:11.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.494+0000 7f38019d2640 1 --2- 172.21.15.169:0/3150521184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37fc1005c0 0x7f37fc197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:11.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.494+0000 7f38011d1640 1 --2- 172.21.15.169:0/3150521184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37fc1017b0 0x7f37fc197f50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f37f800ea60 tx=0x7f37f800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:11.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.494+0000 7f37f2ffd640 1 -- 172.21.15.169:0/3150521184 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f800cde0 con 0x7f37fc1017b0 2024-03-28T00:56:11.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.494+0000 7f37f2ffd640 1 -- 172.21.15.169:0/3150521184 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f37f80035f0 con 0x7f37fc1017b0 2024-03-28T00:56:11.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.494+0000 7f3803c36640 1 -- 172.21.15.169:0/3150521184 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37fc19f920 con 0x7f37fc1017b0 2024-03-28T00:56:11.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.494+0000 7f37f2ffd640 1 -- 172.21.15.169:0/3150521184 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f8010670 con 0x7f37fc1017b0 2024-03-28T00:56:11.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.494+0000 7f3803c36640 1 -- 172.21.15.169:0/3150521184 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37fc19fe70 con 0x7f37fc1017b0 2024-03-28T00:56:11.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.495+0000 7f3803c36640 1 -- 172.21.15.169:0/3150521184 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37fc1a0100 con 0x7f37fc1017b0 2024-03-28T00:56:11.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.496+0000 7f37f2ffd640 1 -- 172.21.15.169:0/3150521184 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f37f80037a0 con 0x7f37fc1017b0 2024-03-28T00:56:11.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.497+0000 7f37f2ffd640 1 --2- 172.21.15.169:0/3150521184 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f37e80a2800 0x7f37e80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:11.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.497+0000 7f38019d2640 1 --2- 172.21.15.169:0/3150521184 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f37e80a2800 0x7f37e80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:11.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.497+0000 7f37f2ffd640 1 -- 172.21.15.169:0/3150521184 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f37f80cf0b0 con 0x7f37fc1017b0 2024-03-28T00:56:11.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.497+0000 7f38019d2640 1 --2- 172.21.15.169:0/3150521184 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f37e80a2800 0x7f37e80a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f37ec000c00 tx=0x7f37ec005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:11.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.501+0000 7f37f2ffd640 1 -- 172.21.15.169:0/3150521184 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f37f809bdf0 con 0x7f37fc1017b0 2024-03-28T00:56:11.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:11 smithi174 ceph-mon[36060]: pgmap v339: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T00:56:11.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:11 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3655762676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:56:11.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:11 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2913765114' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:56:11.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:11 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3830480882' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:56:11.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:11 smithi177 ceph-mon[36206]: pgmap v339: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-28T00:56:11.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:11 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3655762676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:56:11.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:11 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2913765114' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:56:11.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:11 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3830480882' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:56:11.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.677+0000 7f3803c36640 1 -- 172.21.15.169:0/3150521184 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37fc1a02d0 con 0x7f37fc1017b0 2024-03-28T00:56:11.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.678+0000 7f37f2ffd640 1 -- 172.21.15.169:0/3150521184 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+24711 (secure 0 0 0) 0x7f37f809b6f0 con 0x7f37fc1017b0 2024-03-28T00:56:11.682 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:11.683 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":92,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:55:37.555451+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:56:11.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.681+0000 7f3803c36640 1 -- 172.21.15.169:0/3150521184 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f37e80a2800 msgr2=0x7f37e80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:11.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.681+0000 7f3803c36640 1 --2- 172.21.15.169:0/3150521184 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f37e80a2800 0x7f37e80a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f37ec000c00 tx=0x7f37ec005fd0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.681+0000 7f3803c36640 1 -- 172.21.15.169:0/3150521184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37fc1017b0 msgr2=0x7f37fc197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:11.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.681+0000 7f3803c36640 1 --2- 172.21.15.169:0/3150521184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37fc1017b0 0x7f37fc197f50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f37f800ea60 tx=0x7f37f800ef20 comp rx=0 tx=0).stop 2024-03-28T00:56:11.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.681+0000 7f3803c36640 1 -- 172.21.15.169:0/3150521184 shutdown_connections 2024-03-28T00:56:11.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.681+0000 7f3803c36640 1 --2- 172.21.15.169:0/3150521184 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f37e80a2800 0x7f37e80a4cc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.681+0000 7f3803c36640 1 --2- 172.21.15.169:0/3150521184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37fc102170 0x7f37fc19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.681+0000 7f3803c36640 1 --2- 172.21.15.169:0/3150521184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37fc1017b0 0x7f37fc197f50 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.681+0000 7f3803c36640 1 --2- 172.21.15.169:0/3150521184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37fc1005c0 0x7f37fc197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.681+0000 7f3803c36640 1 -- 172.21.15.169:0/3150521184 >> 172.21.15.169:0/3150521184 conn(0x7f37fc0fbb30 msgr2=0x7f37fc0fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:11.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.681+0000 7f3803c36640 1 -- 172.21.15.169:0/3150521184 shutdown_connections 2024-03-28T00:56:11.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.681+0000 7f3803c36640 1 -- 172.21.15.169:0/3150521184 wait complete. 2024-03-28T00:56:11.697 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T00:49:56.339525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T00:54:31.182562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T00:54:31.780429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T00:54:35.035724+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T00:56:11.698 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T00:56:11.811 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.800+0000 7f9e5a459640 1 -- 172.21.15.169:0/3989905723 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e541028a0 msgr2=0x7f9e54102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:11.811 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.800+0000 7f9e5a459640 1 --2- 172.21.15.169:0/3989905723 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e541028a0 0x7f9e54102d20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009900 tx=0x7f9e4802bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:11.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.800+0000 7f9e5a459640 1 -- 172.21.15.169:0/3989905723 shutdown_connections 2024-03-28T00:56:11.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.800+0000 7f9e5a459640 1 --2- 172.21.15.169:0/3989905723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e54103260 0x7f9e54109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.800+0000 7f9e5a459640 1 --2- 172.21.15.169:0/3989905723 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e541028a0 0x7f9e54102d20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.800+0000 7f9e5a459640 1 --2- 172.21.15.169:0/3989905723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e541016b0 0x7f9e54101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.800+0000 7f9e5a459640 1 -- 172.21.15.169:0/3989905723 >> 172.21.15.169:0/3989905723 conn(0x7f9e540fcc40 msgr2=0x7f9e540ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:11.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.800+0000 7f9e5a459640 1 -- 172.21.15.169:0/3989905723 shutdown_connections 2024-03-28T00:56:11.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.800+0000 7f9e5a459640 1 -- 172.21.15.169:0/3989905723 wait complete. 2024-03-28T00:56:11.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.801+0000 7f9e5a459640 1 Processor -- start 2024-03-28T00:56:11.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.801+0000 7f9e5a459640 1 -- start start 2024-03-28T00:56:11.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.801+0000 7f9e5a459640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e541016b0 0x7f9e54075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:11.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.801+0000 7f9e5a459640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e541028a0 0x7f9e54073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:11.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.801+0000 7f9e5a459640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e54103260 0x7f9e54073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:11.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.801+0000 7f9e5a459640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5410d310 con 0x7f9e541016b0 2024-03-28T00:56:11.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.801+0000 7f9e5a459640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5410d190 con 0x7f9e54103260 2024-03-28T00:56:11.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.801+0000 7f9e5a459640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5410d490 con 0x7f9e541028a0 2024-03-28T00:56:11.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.801+0000 7f9e53fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e541016b0 0x7f9e54075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:11.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.801+0000 7f9e53fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e541016b0 0x7f9e54075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38208/0 (socket says 172.21.15.169:38208) 2024-03-28T00:56:11.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.801+0000 7f9e53fff640 1 -- 172.21.15.169:0/93551808 learned_addr learned my addr 172.21.15.169:0/93551808 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:11.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.801+0000 7f9e537fe640 1 --2- 172.21.15.169:0/93551808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e541028a0 0x7f9e54073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:11.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.801+0000 7f9e589f6640 1 --2- 172.21.15.169:0/93551808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e54103260 0x7f9e54073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:11.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.802+0000 7f9e53fff640 1 -- 172.21.15.169:0/93551808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e541028a0 msgr2=0x7f9e54073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:11.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.802+0000 7f9e53fff640 1 --2- 172.21.15.169:0/93551808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e541028a0 0x7f9e54073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.802+0000 7f9e53fff640 1 -- 172.21.15.169:0/93551808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e54103260 msgr2=0x7f9e54073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:11.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.802+0000 7f9e53fff640 1 --2- 172.21.15.169:0/93551808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e54103260 0x7f9e54073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.802+0000 7f9e53fff640 1 -- 172.21.15.169:0/93551808 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e54074860 con 0x7f9e541016b0 2024-03-28T00:56:11.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.802+0000 7f9e53fff640 1 --2- 172.21.15.169:0/93551808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e541016b0 0x7f9e54075450 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9e4400e9c0 tx=0x7f9e4400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:11.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.802+0000 7f9e517fa640 1 -- 172.21.15.169:0/93551808 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4400cd80 con 0x7f9e541016b0 2024-03-28T00:56:11.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.802+0000 7f9e517fa640 1 -- 172.21.15.169:0/93551808 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e44010910 con 0x7f9e541016b0 2024-03-28T00:56:11.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.802+0000 7f9e5a459640 1 -- 172.21.15.169:0/93551808 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e541a1b30 con 0x7f9e541016b0 2024-03-28T00:56:11.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.802+0000 7f9e517fa640 1 -- 172.21.15.169:0/93551808 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e44018950 con 0x7f9e541016b0 2024-03-28T00:56:11.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.802+0000 7f9e5a459640 1 -- 172.21.15.169:0/93551808 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e541a2050 con 0x7f9e541016b0 2024-03-28T00:56:11.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.803+0000 7f9e5a459640 1 -- 172.21.15.169:0/93551808 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e54108a50 con 0x7f9e541016b0 2024-03-28T00:56:11.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.804+0000 7f9e517fa640 1 -- 172.21.15.169:0/93551808 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9e44010ac0 con 0x7f9e541016b0 2024-03-28T00:56:11.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.805+0000 7f9e517fa640 1 --2- 172.21.15.169:0/93551808 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9e3c0a2800 0x7f9e3c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:11.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.805+0000 7f9e537fe640 1 --2- 172.21.15.169:0/93551808 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9e3c0a2800 0x7f9e3c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:11.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.805+0000 7f9e517fa640 1 -- 172.21.15.169:0/93551808 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f9e440cf1e0 con 0x7f9e541016b0 2024-03-28T00:56:11.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.805+0000 7f9e537fe640 1 --2- 172.21.15.169:0/93551808 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9e3c0a2800 0x7f9e3c0a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9e4802c7b0 tx=0x7f9e48005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:11.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.809+0000 7f9e517fa640 1 -- 172.21.15.169:0/93551808 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9e44053740 con 0x7f9e541016b0 2024-03-28T00:56:11.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.983+0000 7f9e5a459640 1 -- 172.21.15.169:0/93551808 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9e5402d070 con 0x7f9e541016b0 2024-03-28T00:56:11.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.983+0000 7f9e517fa640 1 -- 172.21.15.169:0/93551808 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9e4409bf20 con 0x7f9e541016b0 2024-03-28T00:56:11.987 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 1 2024-03-28T00:56:11.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.985+0000 7f9e5a459640 1 -- 172.21.15.169:0/93551808 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9e3c0a2800 msgr2=0x7f9e3c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:11.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.985+0000 7f9e5a459640 1 --2- 172.21.15.169:0/93551808 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9e3c0a2800 0x7f9e3c0a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9e4802c7b0 tx=0x7f9e48005fd0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.985+0000 7f9e5a459640 1 -- 172.21.15.169:0/93551808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e541016b0 msgr2=0x7f9e54075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:11.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.985+0000 7f9e5a459640 1 --2- 172.21.15.169:0/93551808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e541016b0 0x7f9e54075450 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9e4400e9c0 tx=0x7f9e4400edc0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.985+0000 7f9e5a459640 1 -- 172.21.15.169:0/93551808 shutdown_connections 2024-03-28T00:56:11.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.985+0000 7f9e5a459640 1 --2- 172.21.15.169:0/93551808 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9e3c0a2800 0x7f9e3c0a4cc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.985+0000 7f9e5a459640 1 --2- 172.21.15.169:0/93551808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e54103260 0x7f9e54073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.985+0000 7f9e5a459640 1 --2- 172.21.15.169:0/93551808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e541028a0 0x7f9e54073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.985+0000 7f9e5a459640 1 --2- 172.21.15.169:0/93551808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e541016b0 0x7f9e54075450 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:11.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.985+0000 7f9e5a459640 1 -- 172.21.15.169:0/93551808 >> 172.21.15.169:0/93551808 conn(0x7f9e540fcc40 msgr2=0x7f9e54105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:11.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.985+0000 7f9e5a459640 1 -- 172.21.15.169:0/93551808 shutdown_connections 2024-03-28T00:56:11.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:11.985+0000 7f9e5a459640 1 -- 172.21.15.169:0/93551808 wait complete. 2024-03-28T00:56:12.000 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T00:56:12.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.102+0000 7fb9b9219640 1 -- 172.21.15.169:0/861282176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9b41073a0 msgr2=0x7fb9b41097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:12.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.102+0000 7fb9b9219640 1 --2- 172.21.15.169:0/861282176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9b41073a0 0x7fb9b41097f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac009ca0 tx=0x7fb9ac02c0f0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.102+0000 7fb9b9219640 1 -- 172.21.15.169:0/861282176 shutdown_connections 2024-03-28T00:56:12.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.102+0000 7fb9b9219640 1 --2- 172.21.15.169:0/861282176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9b41073a0 0x7fb9b41097f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.102+0000 7fb9b9219640 1 --2- 172.21.15.169:0/861282176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9b4104a70 0x7fb9b4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.102+0000 7fb9b9219640 1 --2- 172.21.15.169:0/861282176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9b4102140 0x7fb9b4104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.102+0000 7fb9b9219640 1 -- 172.21.15.169:0/861282176 >> 172.21.15.169:0/861282176 conn(0x7fb9b40fbb50 msgr2=0x7fb9b40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:12.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.102+0000 7fb9b9219640 1 -- 172.21.15.169:0/861282176 shutdown_connections 2024-03-28T00:56:12.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.102+0000 7fb9b9219640 1 -- 172.21.15.169:0/861282176 wait complete. 2024-03-28T00:56:12.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.103+0000 7fb9b9219640 1 Processor -- start 2024-03-28T00:56:12.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.103+0000 7fb9b9219640 1 -- start start 2024-03-28T00:56:12.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.103+0000 7fb9b9219640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9b4102140 0x7fb9b4197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:12.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.103+0000 7fb9b9219640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9b4104a70 0x7fb9b4197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:12.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.103+0000 7fb9b9219640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9b41073a0 0x7fb9b419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:12.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.103+0000 7fb9b9219640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b410d360 con 0x7fb9b4102140 2024-03-28T00:56:12.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.103+0000 7fb9b9219640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b410d1e0 con 0x7fb9b4104a70 2024-03-28T00:56:12.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.103+0000 7fb9b9219640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b410d4e0 con 0x7fb9b41073a0 2024-03-28T00:56:12.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.103+0000 7fb9b2d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9b4102140 0x7fb9b4197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:12.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.103+0000 7fb9abfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9b4104a70 0x7fb9b4197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:12.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.103+0000 7fb9b2d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9b4102140 0x7fb9b4197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38222/0 (socket says 172.21.15.169:38222) 2024-03-28T00:56:12.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.103+0000 7fb9b2d9d640 1 -- 172.21.15.169:0/1258658790 learned_addr learned my addr 172.21.15.169:0/1258658790 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:12.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.103+0000 7fb9b359e640 1 --2- 172.21.15.169:0/1258658790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9b41073a0 0x7fb9b419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:12.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.103+0000 7fb9abfff640 1 -- 172.21.15.169:0/1258658790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9b41073a0 msgr2=0x7fb9b419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:12.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.103+0000 7fb9abfff640 1 --2- 172.21.15.169:0/1258658790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9b41073a0 0x7fb9b419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.103+0000 7fb9abfff640 1 -- 172.21.15.169:0/1258658790 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9b4102140 msgr2=0x7fb9b4197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:12.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.103+0000 7fb9abfff640 1 --2- 172.21.15.169:0/1258658790 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9b4102140 0x7fb9b4197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.103+0000 7fb9abfff640 1 -- 172.21.15.169:0/1258658790 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b419f620 con 0x7fb9b4104a70 2024-03-28T00:56:12.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.104+0000 7fb9abfff640 1 --2- 172.21.15.169:0/1258658790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9b4104a70 0x7fb9b4197f40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb99c00e9c0 tx=0x7fb99c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:12.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.104+0000 7fb9b0d99640 1 -- 172.21.15.169:0/1258658790 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb99c00cd80 con 0x7fb9b4104a70 2024-03-28T00:56:12.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.104+0000 7fb9b9219640 1 -- 172.21.15.169:0/1258658790 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9b419f910 con 0x7fb9b4104a70 2024-03-28T00:56:12.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.104+0000 7fb9b0d99640 1 -- 172.21.15.169:0/1258658790 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb99c0035f0 con 0x7fb9b4104a70 2024-03-28T00:56:12.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.104+0000 7fb9b0d99640 1 -- 172.21.15.169:0/1258658790 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb99c010670 con 0x7fb9b4104a70 2024-03-28T00:56:12.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.104+0000 7fb9b9219640 1 -- 172.21.15.169:0/1258658790 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9b419feb0 con 0x7fb9b4104a70 2024-03-28T00:56:12.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.105+0000 7fb9b9219640 1 -- 172.21.15.169:0/1258658790 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9b40fd640 con 0x7fb9b4104a70 2024-03-28T00:56:12.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.106+0000 7fb9b0d99640 1 -- 172.21.15.169:0/1258658790 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb99c0037a0 con 0x7fb9b4104a70 2024-03-28T00:56:12.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.106+0000 7fb9b0d99640 1 --2- 172.21.15.169:0/1258658790 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb9a00a2800 0x7fb9a00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:12.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.107+0000 7fb9b2d9d640 1 --2- 172.21.15.169:0/1258658790 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb9a00a2800 0x7fb9a00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:12.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.107+0000 7fb9b0d99640 1 -- 172.21.15.169:0/1258658790 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fb99c0cf0c0 con 0x7fb9b4104a70 2024-03-28T00:56:12.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.107+0000 7fb9b2d9d640 1 --2- 172.21.15.169:0/1258658790 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb9a00a2800 0x7fb9a00a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb9a4009cd0 tx=0x7fb9a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:12.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.110+0000 7fb9b0d99640 1 -- 172.21.15.169:0/1258658790 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb99c09bda0 con 0x7fb9b4104a70 2024-03-28T00:56:12.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.285+0000 7fb9b9219640 1 -- 172.21.15.169:0/1258658790 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb9b402d230 con 0x7fb9b4104a70 2024-03-28T00:56:12.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.286+0000 7fb9b0d99640 1 -- 172.21.15.169:0/1258658790 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb99c09b6a0 con 0x7fb9b4104a70 2024-03-28T00:56:12.289 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T00:56:12.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.288+0000 7fb9b9219640 1 -- 172.21.15.169:0/1258658790 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb9a00a2800 msgr2=0x7fb9a00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:12.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.288+0000 7fb9b9219640 1 --2- 172.21.15.169:0/1258658790 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb9a00a2800 0x7fb9a00a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb9a4009cd0 tx=0x7fb9a4005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:12.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.288+0000 7fb9b9219640 1 -- 172.21.15.169:0/1258658790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9b4104a70 msgr2=0x7fb9b4197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:12.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.288+0000 7fb9b9219640 1 --2- 172.21.15.169:0/1258658790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9b4104a70 0x7fb9b4197f40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb99c00e9c0 tx=0x7fb99c00ee80 comp rx=0 tx=0).stop 2024-03-28T00:56:12.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.288+0000 7fb9b9219640 1 -- 172.21.15.169:0/1258658790 shutdown_connections 2024-03-28T00:56:12.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.288+0000 7fb9b9219640 1 --2- 172.21.15.169:0/1258658790 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb9a00a2800 0x7fb9a00a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.288+0000 7fb9b9219640 1 --2- 172.21.15.169:0/1258658790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9b41073a0 0x7fb9b419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.288+0000 7fb9b9219640 1 --2- 172.21.15.169:0/1258658790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9b4104a70 0x7fb9b4197f40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.288+0000 7fb9b9219640 1 --2- 172.21.15.169:0/1258658790 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9b4102140 0x7fb9b4197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.288+0000 7fb9b9219640 1 -- 172.21.15.169:0/1258658790 >> 172.21.15.169:0/1258658790 conn(0x7fb9b40fbb50 msgr2=0x7fb9b4102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:12.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.288+0000 7fb9b9219640 1 -- 172.21.15.169:0/1258658790 shutdown_connections 2024-03-28T00:56:12.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.288+0000 7fb9b9219640 1 -- 172.21.15.169:0/1258658790 wait complete. 2024-03-28T00:56:12.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:12 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/740273870' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:56:12.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:12 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3150521184' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:12.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:12 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/93551808' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:56:12.302 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T00:56:12.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.404+0000 7f7c49ac6640 1 -- 172.21.15.169:0/2889082189 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c441005c0 msgr2=0x7f7c441009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:12.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.404+0000 7f7c49ac6640 1 --2- 172.21.15.169:0/2889082189 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c441005c0 0x7f7c441009e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7c34009900 tx=0x7f7c3402bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:12.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.404+0000 7f7c49ac6640 1 -- 172.21.15.169:0/2889082189 shutdown_connections 2024-03-28T00:56:12.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.404+0000 7f7c49ac6640 1 --2- 172.21.15.169:0/2889082189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c44102170 0x7f7c44108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.404+0000 7f7c49ac6640 1 --2- 172.21.15.169:0/2889082189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c441017b0 0x7f7c44101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.404+0000 7f7c49ac6640 1 --2- 172.21.15.169:0/2889082189 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c441005c0 0x7f7c441009e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.404+0000 7f7c49ac6640 1 -- 172.21.15.169:0/2889082189 >> 172.21.15.169:0/2889082189 conn(0x7f7c440fbb30 msgr2=0x7f7c440fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:12.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.405+0000 7f7c49ac6640 1 -- 172.21.15.169:0/2889082189 shutdown_connections 2024-03-28T00:56:12.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.405+0000 7f7c49ac6640 1 -- 172.21.15.169:0/2889082189 wait complete. 2024-03-28T00:56:12.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.405+0000 7f7c49ac6640 1 Processor -- start 2024-03-28T00:56:12.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.405+0000 7f7c49ac6640 1 -- start start 2024-03-28T00:56:12.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.405+0000 7f7c49ac6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c441005c0 0x7f7c44197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:12.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.405+0000 7f7c49ac6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c441017b0 0x7f7c44197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:12.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.405+0000 7f7c49ac6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c44102170 0x7f7c4419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:12.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.405+0000 7f7c49ac6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4410d360 con 0x7f7c441005c0 2024-03-28T00:56:12.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.405+0000 7f7c49ac6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4410d1e0 con 0x7f7c44102170 2024-03-28T00:56:12.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.405+0000 7f7c49ac6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4410d4e0 con 0x7f7c441017b0 2024-03-28T00:56:12.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.406+0000 7f7c437fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c441005c0 0x7f7c44197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:12.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.406+0000 7f7c437fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c441005c0 0x7f7c44197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38238/0 (socket says 172.21.15.169:38238) 2024-03-28T00:56:12.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.406+0000 7f7c437fe640 1 -- 172.21.15.169:0/4199658410 learned_addr learned my addr 172.21.15.169:0/4199658410 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:12.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.406+0000 7f7c42ffd640 1 --2- 172.21.15.169:0/4199658410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c441017b0 0x7f7c44197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:12.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.406+0000 7f7c43fff640 1 --2- 172.21.15.169:0/4199658410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c44102170 0x7f7c4419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:12.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.406+0000 7f7c437fe640 1 -- 172.21.15.169:0/4199658410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c441017b0 msgr2=0x7f7c44197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:12.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.406+0000 7f7c437fe640 1 --2- 172.21.15.169:0/4199658410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c441017b0 0x7f7c44197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.406+0000 7f7c437fe640 1 -- 172.21.15.169:0/4199658410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c44102170 msgr2=0x7f7c4419f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:12.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.406+0000 7f7c437fe640 1 --2- 172.21.15.169:0/4199658410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c44102170 0x7f7c4419f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.406+0000 7f7c437fe640 1 -- 172.21.15.169:0/4199658410 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c4419f690 con 0x7f7c441005c0 2024-03-28T00:56:12.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.406+0000 7f7c437fe640 1 --2- 172.21.15.169:0/4199658410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c441005c0 0x7f7c44197a70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7c34000c00 tx=0x7f7c34004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:12.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.406+0000 7f7c40ff9640 1 -- 172.21.15.169:0/4199658410 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c34041070 con 0x7f7c441005c0 2024-03-28T00:56:12.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.406+0000 7f7c40ff9640 1 -- 172.21.15.169:0/4199658410 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c34004450 con 0x7f7c441005c0 2024-03-28T00:56:12.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.406+0000 7f7c49ac6640 1 -- 172.21.15.169:0/4199658410 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c4419f920 con 0x7f7c441005c0 2024-03-28T00:56:12.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.406+0000 7f7c40ff9640 1 -- 172.21.15.169:0/4199658410 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c34003df0 con 0x7f7c441005c0 2024-03-28T00:56:12.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.406+0000 7f7c49ac6640 1 -- 172.21.15.169:0/4199658410 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c4419fe10 con 0x7f7c441005c0 2024-03-28T00:56:12.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.407+0000 7f7c49ac6640 1 -- 172.21.15.169:0/4199658410 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c44107a00 con 0x7f7c441005c0 2024-03-28T00:56:12.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.408+0000 7f7c40ff9640 1 -- 172.21.15.169:0/4199658410 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7c34044070 con 0x7f7c441005c0 2024-03-28T00:56:12.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.409+0000 7f7c40ff9640 1 --2- 172.21.15.169:0/4199658410 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7c2c0a2800 0x7f7c2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:12.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.409+0000 7f7c42ffd640 1 --2- 172.21.15.169:0/4199658410 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7c2c0a2800 0x7f7c2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:12.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.409+0000 7f7c40ff9640 1 -- 172.21.15.169:0/4199658410 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f7c340f0860 con 0x7f7c441005c0 2024-03-28T00:56:12.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.409+0000 7f7c42ffd640 1 --2- 172.21.15.169:0/4199658410 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7c2c0a2800 0x7f7c2c0a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7c44198f80 tx=0x7f7c38005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:12.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.413+0000 7f7c40ff9640 1 -- 172.21.15.169:0/4199658410 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7c340bcf50 con 0x7f7c441005c0 2024-03-28T00:56:12.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.587+0000 7f7c49ac6640 1 -- 172.21.15.169:0/4199658410 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7c4402d070 con 0x7f7c441005c0 2024-03-28T00:56:12.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.588+0000 7f7c40ff9640 1 -- 172.21.15.169:0/4199658410 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7c340c0ff0 con 0x7f7c441005c0 2024-03-28T00:56:12.591 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 64 2024-03-28T00:56:12.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.590+0000 7f7c49ac6640 1 -- 172.21.15.169:0/4199658410 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7c2c0a2800 msgr2=0x7f7c2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:12.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.590+0000 7f7c49ac6640 1 --2- 172.21.15.169:0/4199658410 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7c2c0a2800 0x7f7c2c0a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7c44198f80 tx=0x7f7c38005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:12.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.590+0000 7f7c49ac6640 1 -- 172.21.15.169:0/4199658410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c441005c0 msgr2=0x7f7c44197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:12.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.590+0000 7f7c49ac6640 1 --2- 172.21.15.169:0/4199658410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c441005c0 0x7f7c44197a70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7c34000c00 tx=0x7f7c34004960 comp rx=0 tx=0).stop 2024-03-28T00:56:12.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.590+0000 7f7c49ac6640 1 -- 172.21.15.169:0/4199658410 shutdown_connections 2024-03-28T00:56:12.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.590+0000 7f7c49ac6640 1 --2- 172.21.15.169:0/4199658410 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7c2c0a2800 0x7f7c2c0a4cc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.590+0000 7f7c49ac6640 1 --2- 172.21.15.169:0/4199658410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c44102170 0x7f7c4419f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.590+0000 7f7c49ac6640 1 --2- 172.21.15.169:0/4199658410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c441017b0 0x7f7c44197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.590+0000 7f7c49ac6640 1 --2- 172.21.15.169:0/4199658410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c441005c0 0x7f7c44197a70 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.590+0000 7f7c49ac6640 1 -- 172.21.15.169:0/4199658410 >> 172.21.15.169:0/4199658410 conn(0x7f7c440fbb30 msgr2=0x7f7c44104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:12.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.590+0000 7f7c49ac6640 1 -- 172.21.15.169:0/4199658410 shutdown_connections 2024-03-28T00:56:12.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.590+0000 7f7c49ac6640 1 -- 172.21.15.169:0/4199658410 wait complete. 2024-03-28T00:56:12.607 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T00:56:12.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:12 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/740273870' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:56:12.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:12 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3150521184' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:12.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:12 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/93551808' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:56:12.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:12 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/740273870' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:56:12.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:12 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3150521184' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:12.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:12 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/93551808' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:56:12.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.710+0000 7f2ebb0ae640 1 -- 172.21.15.169:0/3341966460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2eb41005a0 msgr2=0x7f2eb41009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:12.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.710+0000 7f2ebb0ae640 1 --2- 172.21.15.169:0/3341966460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2eb41005a0 0x7f2eb41009c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8009900 tx=0x7f2ea802bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:12.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.710+0000 7f2ebb0ae640 1 -- 172.21.15.169:0/3341966460 shutdown_connections 2024-03-28T00:56:12.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.710+0000 7f2ebb0ae640 1 --2- 172.21.15.169:0/3341966460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2eb4102150 0x7f2eb41089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.710+0000 7f2ebb0ae640 1 --2- 172.21.15.169:0/3341966460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2eb4101790 0x7f2eb4101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.710+0000 7f2ebb0ae640 1 --2- 172.21.15.169:0/3341966460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2eb41005a0 0x7f2eb41009c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.710+0000 7f2ebb0ae640 1 -- 172.21.15.169:0/3341966460 >> 172.21.15.169:0/3341966460 conn(0x7f2eb40fbb30 msgr2=0x7f2eb40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:12.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.710+0000 7f2ebb0ae640 1 -- 172.21.15.169:0/3341966460 shutdown_connections 2024-03-28T00:56:12.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.710+0000 7f2ebb0ae640 1 -- 172.21.15.169:0/3341966460 wait complete. 2024-03-28T00:56:12.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.710+0000 7f2ebb0ae640 1 Processor -- start 2024-03-28T00:56:12.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.710+0000 7f2ebb0ae640 1 -- start start 2024-03-28T00:56:12.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.710+0000 7f2ebb0ae640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2eb41005a0 0x7f2eb4069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:12.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.710+0000 7f2ebb0ae640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2eb4101790 0x7f2eb419c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:12.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.710+0000 7f2ebb0ae640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2eb4102150 0x7f2eb419edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:12.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.710+0000 7f2ebb0ae640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb410c200 con 0x7f2eb4101790 2024-03-28T00:56:12.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.710+0000 7f2ebb0ae640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb410c080 con 0x7f2eb4102150 2024-03-28T00:56:12.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.710+0000 7f2ebb0ae640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb410c380 con 0x7f2eb41005a0 2024-03-28T00:56:12.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.711+0000 7f2eb3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2eb4101790 0x7f2eb419c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:12.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.711+0000 7f2eb3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2eb4101790 0x7f2eb419c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38246/0 (socket says 172.21.15.169:38246) 2024-03-28T00:56:12.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.711+0000 7f2eb3fff640 1 -- 172.21.15.169:0/3235138285 learned_addr learned my addr 172.21.15.169:0/3235138285 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:12.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.711+0000 7f2eb8e4a640 1 --2- 172.21.15.169:0/3235138285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2eb41005a0 0x7f2eb4069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:12.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.711+0000 7f2eb964b640 1 --2- 172.21.15.169:0/3235138285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2eb4102150 0x7f2eb419edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:12.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.711+0000 7f2eb3fff640 1 -- 172.21.15.169:0/3235138285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2eb41005a0 msgr2=0x7f2eb4069390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:12.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.711+0000 7f2eb3fff640 1 --2- 172.21.15.169:0/3235138285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2eb41005a0 0x7f2eb4069390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.711+0000 7f2eb3fff640 1 -- 172.21.15.169:0/3235138285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2eb4102150 msgr2=0x7f2eb419edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:12.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.711+0000 7f2eb3fff640 1 --2- 172.21.15.169:0/3235138285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2eb4102150 0x7f2eb419edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.711+0000 7f2eb3fff640 1 -- 172.21.15.169:0/3235138285 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb419f670 con 0x7f2eb4101790 2024-03-28T00:56:12.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.711+0000 7f2eb8e4a640 1 --2- 172.21.15.169:0/3235138285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2eb41005a0 0x7f2eb4069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:12.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.711+0000 7f2eb964b640 1 --2- 172.21.15.169:0/3235138285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2eb4102150 0x7f2eb419edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:12.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.711+0000 7f2eb3fff640 1 --2- 172.21.15.169:0/3235138285 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2eb4101790 0x7f2eb419c770 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000ea60 tx=0x7f2ea000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:12.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.711+0000 7f2eb1ffb640 1 -- 172.21.15.169:0/3235138285 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea000cd20 con 0x7f2eb4101790 2024-03-28T00:56:12.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.711+0000 7f2eb1ffb640 1 -- 172.21.15.169:0/3235138285 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ea00035f0 con 0x7f2eb4101790 2024-03-28T00:56:12.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.711+0000 7f2ebb0ae640 1 -- 172.21.15.169:0/3235138285 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb419f960 con 0x7f2eb4101790 2024-03-28T00:56:12.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.711+0000 7f2eb1ffb640 1 -- 172.21.15.169:0/3235138285 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0010690 con 0x7f2eb4101790 2024-03-28T00:56:12.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.711+0000 7f2ebb0ae640 1 -- 172.21.15.169:0/3235138285 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb419feb0 con 0x7f2eb4101790 2024-03-28T00:56:12.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.712+0000 7f2ebb0ae640 1 -- 172.21.15.169:0/3235138285 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb41079e0 con 0x7f2eb4101790 2024-03-28T00:56:12.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.713+0000 7f2eb1ffb640 1 -- 172.21.15.169:0/3235138285 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2ea00037a0 con 0x7f2eb4101790 2024-03-28T00:56:12.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.714+0000 7f2eb1ffb640 1 --2- 172.21.15.169:0/3235138285 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2ea40a2800 0x7f2ea40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:12.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.714+0000 7f2eb8e4a640 1 --2- 172.21.15.169:0/3235138285 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2ea40a2800 0x7f2ea40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:12.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.714+0000 7f2eb1ffb640 1 -- 172.21.15.169:0/3235138285 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f2ea00cf0d0 con 0x7f2eb4101790 2024-03-28T00:56:12.720 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.714+0000 7f2eb8e4a640 1 --2- 172.21.15.169:0/3235138285 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2ea40a2800 0x7f2ea40a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8000c00 tx=0x7f2ea8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:12.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.718+0000 7f2eb1ffb640 1 -- 172.21.15.169:0/3235138285 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2ea009be10 con 0x7f2eb4101790 2024-03-28T00:56:12.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.895+0000 7f2ebb0ae640 1 -- 172.21.15.169:0/3235138285 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f2eb402d070 con 0x7f2eb4101790 2024-03-28T00:56:12.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.896+0000 7f2eb1ffb640 1 -- 172.21.15.169:0/3235138285 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) v1 ==== 103+0+11 (secure 0 0 0) 0x7f2ea009b710 con 0x7f2eb4101790 2024-03-28T00:56:12.900 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T00:56:12.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.898+0000 7f2ebb0ae640 1 -- 172.21.15.169:0/3235138285 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2ea40a2800 msgr2=0x7f2ea40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:12.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.898+0000 7f2ebb0ae640 1 --2- 172.21.15.169:0/3235138285 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2ea40a2800 0x7f2ea40a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8000c00 tx=0x7f2ea8005fd0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.898+0000 7f2ebb0ae640 1 -- 172.21.15.169:0/3235138285 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2eb4101790 msgr2=0x7f2eb419c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:12.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.898+0000 7f2ebb0ae640 1 --2- 172.21.15.169:0/3235138285 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2eb4101790 0x7f2eb419c770 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000ea60 tx=0x7f2ea000ee60 comp rx=0 tx=0).stop 2024-03-28T00:56:12.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.898+0000 7f2ebb0ae640 1 -- 172.21.15.169:0/3235138285 shutdown_connections 2024-03-28T00:56:12.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.898+0000 7f2ebb0ae640 1 --2- 172.21.15.169:0/3235138285 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2ea40a2800 0x7f2ea40a4cc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.898+0000 7f2ebb0ae640 1 --2- 172.21.15.169:0/3235138285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2eb4102150 0x7f2eb419edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.898+0000 7f2ebb0ae640 1 --2- 172.21.15.169:0/3235138285 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2eb4101790 0x7f2eb419c770 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.898+0000 7f2ebb0ae640 1 --2- 172.21.15.169:0/3235138285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2eb41005a0 0x7f2eb4069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:12.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.898+0000 7f2ebb0ae640 1 -- 172.21.15.169:0/3235138285 >> 172.21.15.169:0/3235138285 conn(0x7f2eb40fbb30 msgr2=0x7f2eb4104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:12.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.898+0000 7f2ebb0ae640 1 -- 172.21.15.169:0/3235138285 shutdown_connections 2024-03-28T00:56:12.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:12.898+0000 7f2ebb0ae640 1 -- 172.21.15.169:0/3235138285 wait complete. 2024-03-28T00:56:12.913 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:56:13.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.013+0000 7f158f481640 1 -- 172.21.15.169:0/1425929325 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15880ff990 msgr2=0x7f15880ffdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:13.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.013+0000 7f158f481640 1 --2- 172.21.15.169:0/1425929325 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15880ff990 0x7f15880ffdb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1584009930 tx=0x7f158402bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:13.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.013+0000 7f158f481640 1 -- 172.21.15.169:0/1425929325 shutdown_connections 2024-03-28T00:56:13.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.013+0000 7f158f481640 1 --2- 172.21.15.169:0/1425929325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1588103320 0x7f1588109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.013+0000 7f158f481640 1 --2- 172.21.15.169:0/1425929325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1588102960 0x7f1588102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.013+0000 7f158f481640 1 --2- 172.21.15.169:0/1425929325 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15880ff990 0x7f15880ffdb0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.013+0000 7f158f481640 1 -- 172.21.15.169:0/1425929325 >> 172.21.15.169:0/1425929325 conn(0x7f15880fb2c0 msgr2=0x7f15880fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:13.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.013+0000 7f158f481640 1 -- 172.21.15.169:0/1425929325 shutdown_connections 2024-03-28T00:56:13.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.013+0000 7f158f481640 1 -- 172.21.15.169:0/1425929325 wait complete. 2024-03-28T00:56:13.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.013+0000 7f158f481640 1 Processor -- start 2024-03-28T00:56:13.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.013+0000 7f158f481640 1 -- start start 2024-03-28T00:56:13.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f158f481640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15880ff990 0x7f1588197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:13.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f158f481640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1588102960 0x7f1588197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:13.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f158f481640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1588103320 0x7f158819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:13.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f158f481640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158810d340 con 0x7f1588102960 2024-03-28T00:56:13.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f158f481640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158810d1c0 con 0x7f1588103320 2024-03-28T00:56:13.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f158f481640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158810d4c0 con 0x7f15880ff990 2024-03-28T00:56:13.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f158ca1c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1588102960 0x7f1588197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:13.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f158d21d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15880ff990 0x7f1588197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:13.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f158ca1c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1588102960 0x7f1588197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38270/0 (socket says 172.21.15.169:38270) 2024-03-28T00:56:13.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f158ca1c640 1 -- 172.21.15.169:0/1524480212 learned_addr learned my addr 172.21.15.169:0/1524480212 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:13.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f158da1e640 1 --2- 172.21.15.169:0/1524480212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1588103320 0x7f158819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:13.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f158d21d640 1 -- 172.21.15.169:0/1524480212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1588103320 msgr2=0x7f158819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:13.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f158d21d640 1 --2- 172.21.15.169:0/1524480212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1588103320 0x7f158819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f158d21d640 1 -- 172.21.15.169:0/1524480212 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1588102960 msgr2=0x7f1588197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:13.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f158d21d640 1 --2- 172.21.15.169:0/1524480212 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1588102960 0x7f1588197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f158d21d640 1 -- 172.21.15.169:0/1524480212 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158819f660 con 0x7f15880ff990 2024-03-28T00:56:13.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f158ca1c640 1 --2- 172.21.15.169:0/1524480212 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1588102960 0x7f1588197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:13.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f158d21d640 1 --2- 172.21.15.169:0/1524480212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15880ff990 0x7f1588197a40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1584036040 tx=0x7f158402c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:13.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f157e7fc640 1 -- 172.21.15.169:0/1524480212 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1584041070 con 0x7f15880ff990 2024-03-28T00:56:13.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f158f481640 1 -- 172.21.15.169:0/1524480212 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f158819f8f0 con 0x7f15880ff990 2024-03-28T00:56:13.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f157e7fc640 1 -- 172.21.15.169:0/1524480212 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1584004450 con 0x7f15880ff990 2024-03-28T00:56:13.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f157e7fc640 1 -- 172.21.15.169:0/1524480212 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1584003e20 con 0x7f15880ff990 2024-03-28T00:56:13.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.014+0000 7f158f481640 1 -- 172.21.15.169:0/1524480212 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f158819fe00 con 0x7f15880ff990 2024-03-28T00:56:13.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.015+0000 7f158f481640 1 -- 172.21.15.169:0/1524480212 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1588108ae0 con 0x7f15880ff990 2024-03-28T00:56:13.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.016+0000 7f157e7fc640 1 -- 172.21.15.169:0/1524480212 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1584044070 con 0x7f15880ff990 2024-03-28T00:56:13.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.017+0000 7f157e7fc640 1 --2- 172.21.15.169:0/1524480212 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f15740a2800 0x7f15740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:13.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.017+0000 7f158ca1c640 1 --2- 172.21.15.169:0/1524480212 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f15740a2800 0x7f15740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:13.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.017+0000 7f157e7fc640 1 -- 172.21.15.169:0/1524480212 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f15840f0a00 con 0x7f15880ff990 2024-03-28T00:56:13.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.017+0000 7f158ca1c640 1 --2- 172.21.15.169:0/1524480212 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f15740a2800 0x7f15740a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f15780095f0 tx=0x7f1578005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:13.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.020+0000 7f157e7fc640 1 -- 172.21.15.169:0/1524480212 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f15840bd740 con 0x7f15880ff990 2024-03-28T00:56:13.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.235+0000 7f158f481640 1 -- 172.21.15.169:0/1524480212 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f158802d070 con 0x7f15880ff990 2024-03-28T00:56:13.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.236+0000 7f157e7fc640 1 -- 172.21.15.169:0/1524480212 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11420 (secure 0 0 0) 0x7f15840bd040 con 0x7f15880ff990 2024-03-28T00:56:13.239 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:13.240 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:54:52.274335+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24457,24478]},"id":1}]} 2024-03-28T00:56:13.241 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 24 2024-03-28T00:56:13.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.238+0000 7f158f481640 1 -- 172.21.15.169:0/1524480212 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f15740a2800 msgr2=0x7f15740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:13.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.238+0000 7f158f481640 1 --2- 172.21.15.169:0/1524480212 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f15740a2800 0x7f15740a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f15780095f0 tx=0x7f1578005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:13.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.238+0000 7f158f481640 1 -- 172.21.15.169:0/1524480212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15880ff990 msgr2=0x7f1588197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:13.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.238+0000 7f158f481640 1 --2- 172.21.15.169:0/1524480212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15880ff990 0x7f1588197a40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1584036040 tx=0x7f158402c650 comp rx=0 tx=0).stop 2024-03-28T00:56:13.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.238+0000 7f158f481640 1 -- 172.21.15.169:0/1524480212 shutdown_connections 2024-03-28T00:56:13.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.238+0000 7f158f481640 1 --2- 172.21.15.169:0/1524480212 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f15740a2800 0x7f15740a4cc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.238+0000 7f158f481640 1 --2- 172.21.15.169:0/1524480212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1588103320 0x7f158819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.238+0000 7f158f481640 1 --2- 172.21.15.169:0/1524480212 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1588102960 0x7f1588197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.238+0000 7f158f481640 1 --2- 172.21.15.169:0/1524480212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15880ff990 0x7f1588197a40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.238+0000 7f158f481640 1 -- 172.21.15.169:0/1524480212 >> 172.21.15.169:0/1524480212 conn(0x7f15880fb2c0 msgr2=0x7f1588105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:13.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.238+0000 7f158f481640 1 -- 172.21.15.169:0/1524480212 shutdown_connections 2024-03-28T00:56:13.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.238+0000 7f158f481640 1 -- 172.21.15.169:0/1524480212 wait complete. 2024-03-28T00:56:13.252 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T00:56:13.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.352+0000 7eff0eef7640 1 -- 172.21.15.169:0/833956549 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff081005c0 msgr2=0x7eff081009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:13.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.352+0000 7eff0eef7640 1 --2- 172.21.15.169:0/833956549 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff081005c0 0x7eff081009e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7efefc009900 tx=0x7efefc02bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:13.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.352+0000 7eff0eef7640 1 -- 172.21.15.169:0/833956549 shutdown_connections 2024-03-28T00:56:13.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.352+0000 7eff0eef7640 1 --2- 172.21.15.169:0/833956549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff08102170 0x7eff08108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.352+0000 7eff0eef7640 1 --2- 172.21.15.169:0/833956549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff081017b0 0x7eff08101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.352+0000 7eff0eef7640 1 --2- 172.21.15.169:0/833956549 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff081005c0 0x7eff081009e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.352+0000 7eff0eef7640 1 -- 172.21.15.169:0/833956549 >> 172.21.15.169:0/833956549 conn(0x7eff080fbb30 msgr2=0x7eff080fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:13.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.352+0000 7eff0eef7640 1 -- 172.21.15.169:0/833956549 shutdown_connections 2024-03-28T00:56:13.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.353+0000 7eff0eef7640 1 -- 172.21.15.169:0/833956549 wait complete. 2024-03-28T00:56:13.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.353+0000 7eff0eef7640 1 Processor -- start 2024-03-28T00:56:13.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.353+0000 7eff0eef7640 1 -- start start 2024-03-28T00:56:13.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.353+0000 7eff0eef7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff081005c0 0x7eff08197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:13.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.353+0000 7eff0eef7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff081017b0 0x7eff08197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:13.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.353+0000 7eff0eef7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff08102170 0x7eff0819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:13.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.353+0000 7eff0eef7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0810c230 con 0x7eff081005c0 2024-03-28T00:56:13.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.353+0000 7eff0eef7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0810c0b0 con 0x7eff081017b0 2024-03-28T00:56:13.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.353+0000 7eff0eef7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0810c3b0 con 0x7eff08102170 2024-03-28T00:56:13.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.353+0000 7eff0cc93640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff081005c0 0x7eff08197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:13.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.353+0000 7eff0cc93640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff081005c0 0x7eff08197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38284/0 (socket says 172.21.15.169:38284) 2024-03-28T00:56:13.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.353+0000 7eff0cc93640 1 -- 172.21.15.169:0/2321287731 learned_addr learned my addr 172.21.15.169:0/2321287731 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:13.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.353+0000 7eff07fff640 1 --2- 172.21.15.169:0/2321287731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff081017b0 0x7eff08197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:13.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.353+0000 7eff0d494640 1 --2- 172.21.15.169:0/2321287731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff08102170 0x7eff0819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:13.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.354+0000 7eff0cc93640 1 -- 172.21.15.169:0/2321287731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff08102170 msgr2=0x7eff0819efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:13.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.354+0000 7eff0cc93640 1 --2- 172.21.15.169:0/2321287731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff08102170 0x7eff0819efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.354+0000 7eff0cc93640 1 -- 172.21.15.169:0/2321287731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff081017b0 msgr2=0x7eff08197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:13.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.354+0000 7eff0cc93640 1 --2- 172.21.15.169:0/2321287731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff081017b0 0x7eff08197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.354+0000 7eff0cc93640 1 -- 172.21.15.169:0/2321287731 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0819f650 con 0x7eff081005c0 2024-03-28T00:56:13.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.354+0000 7eff0cc93640 1 --2- 172.21.15.169:0/2321287731 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff081005c0 0x7eff08197a30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7efefc000c00 tx=0x7efefc02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:13.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.354+0000 7eff05ffb640 1 -- 172.21.15.169:0/2321287731 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc041070 con 0x7eff081005c0 2024-03-28T00:56:13.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.354+0000 7eff05ffb640 1 -- 172.21.15.169:0/2321287731 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efefc004450 con 0x7eff081005c0 2024-03-28T00:56:13.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.354+0000 7eff0eef7640 1 -- 172.21.15.169:0/2321287731 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0819f8e0 con 0x7eff081005c0 2024-03-28T00:56:13.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.354+0000 7eff05ffb640 1 -- 172.21.15.169:0/2321287731 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc003df0 con 0x7eff081005c0 2024-03-28T00:56:13.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.354+0000 7eff0eef7640 1 -- 172.21.15.169:0/2321287731 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff0819fdd0 con 0x7eff081005c0 2024-03-28T00:56:13.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.355+0000 7eff0eef7640 1 -- 172.21.15.169:0/2321287731 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff0819ffd0 con 0x7eff081005c0 2024-03-28T00:56:13.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.356+0000 7eff05ffb640 1 -- 172.21.15.169:0/2321287731 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7efefc044070 con 0x7eff081005c0 2024-03-28T00:56:13.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.357+0000 7eff05ffb640 1 --2- 172.21.15.169:0/2321287731 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efef80a2800 0x7efef80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:13.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.357+0000 7eff07fff640 1 --2- 172.21.15.169:0/2321287731 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efef80a2800 0x7efef80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:13.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.357+0000 7eff05ffb640 1 -- 172.21.15.169:0/2321287731 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7efefc0f08b0 con 0x7eff081005c0 2024-03-28T00:56:13.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.357+0000 7eff07fff640 1 --2- 172.21.15.169:0/2321287731 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efef80a2800 0x7efef80a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7eff08198f40 tx=0x7efef4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:13.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.361+0000 7eff05ffb640 1 -- 172.21.15.169:0/2321287731 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efefc0bd680 con 0x7eff081005c0 2024-03-28T00:56:13.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.537+0000 7eff0eef7640 1 -- 172.21.15.169:0/2321287731 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff081a06c0 con 0x7eff081005c0 2024-03-28T00:56:13.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.538+0000 7eff05ffb640 1 -- 172.21.15.169:0/2321287731 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+24711 (secure 0 0 0) 0x7efefc0bcf80 con 0x7eff081005c0 2024-03-28T00:56:13.541 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:13.542 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":92,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:55:37.555451+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:56:13.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.540+0000 7eff0eef7640 1 -- 172.21.15.169:0/2321287731 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efef80a2800 msgr2=0x7efef80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:13.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.540+0000 7eff0eef7640 1 --2- 172.21.15.169:0/2321287731 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efef80a2800 0x7efef80a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7eff08198f40 tx=0x7efef4005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:13.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.540+0000 7eff0eef7640 1 -- 172.21.15.169:0/2321287731 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff081005c0 msgr2=0x7eff08197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:13.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.540+0000 7eff0eef7640 1 --2- 172.21.15.169:0/2321287731 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff081005c0 0x7eff08197a30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7efefc000c00 tx=0x7efefc02c640 comp rx=0 tx=0).stop 2024-03-28T00:56:13.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.540+0000 7eff0eef7640 1 -- 172.21.15.169:0/2321287731 shutdown_connections 2024-03-28T00:56:13.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.540+0000 7eff0eef7640 1 --2- 172.21.15.169:0/2321287731 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efef80a2800 0x7efef80a4cc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.540+0000 7eff0eef7640 1 --2- 172.21.15.169:0/2321287731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff08102170 0x7eff0819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.540+0000 7eff0eef7640 1 --2- 172.21.15.169:0/2321287731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff081017b0 0x7eff08197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.540+0000 7eff0eef7640 1 --2- 172.21.15.169:0/2321287731 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff081005c0 0x7eff08197a30 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.541+0000 7eff0eef7640 1 -- 172.21.15.169:0/2321287731 >> 172.21.15.169:0/2321287731 conn(0x7eff080fbb30 msgr2=0x7eff08068800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:13.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.541+0000 7eff0eef7640 1 -- 172.21.15.169:0/2321287731 shutdown_connections 2024-03-28T00:56:13.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.541+0000 7eff0eef7640 1 -- 172.21.15.169:0/2321287731 wait complete. 2024-03-28T00:56:13.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:13 smithi169 ceph-mon[32274]: pgmap v340: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:56:13.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:13 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1258658790' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:56:13.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:13 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4199658410' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:56:13.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:13 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3235138285' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:56:13.555 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:56:13.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:13 smithi174 ceph-mon[36060]: pgmap v340: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:56:13.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:13 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1258658790' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:56:13.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:13 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/4199658410' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:56:13.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:13 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3235138285' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:56:13.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.656+0000 7f7996553640 1 -- 172.21.15.169:0/1319518846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79900ff980 msgr2=0x7f79900ffda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:13.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.656+0000 7f7996553640 1 --2- 172.21.15.169:0/1319518846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79900ff980 0x7f79900ffda0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7980009930 tx=0x7f798002bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:13.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.656+0000 7f7996553640 1 -- 172.21.15.169:0/1319518846 shutdown_connections 2024-03-28T00:56:13.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.656+0000 7f7996553640 1 --2- 172.21.15.169:0/1319518846 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7990103310 0x7f7990109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.656+0000 7f7996553640 1 --2- 172.21.15.169:0/1319518846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7990102950 0x7f7990102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.656+0000 7f7996553640 1 --2- 172.21.15.169:0/1319518846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79900ff980 0x7f79900ffda0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.656+0000 7f7996553640 1 -- 172.21.15.169:0/1319518846 >> 172.21.15.169:0/1319518846 conn(0x7f79900fb2d0 msgr2=0x7f79900fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:13.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.656+0000 7f7996553640 1 -- 172.21.15.169:0/1319518846 shutdown_connections 2024-03-28T00:56:13.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.656+0000 7f7996553640 1 -- 172.21.15.169:0/1319518846 wait complete. 2024-03-28T00:56:13.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.656+0000 7f7996553640 1 Processor -- start 2024-03-28T00:56:13.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.656+0000 7f7996553640 1 -- start start 2024-03-28T00:56:13.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f7996553640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79900ff980 0x7f7990197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:13.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f7996553640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7990102950 0x7f7990197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:13.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f7996553640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7990103310 0x7f799019efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:13.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f7996553640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799010d380 con 0x7f7990103310 2024-03-28T00:56:13.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f7996553640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799010d200 con 0x7f79900ff980 2024-03-28T00:56:13.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f7996553640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799010d500 con 0x7f7990102950 2024-03-28T00:56:13.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f7994af0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7990103310 0x7f799019efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:13.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f798ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79900ff980 0x7f7990197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:13.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f7994af0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7990103310 0x7f799019efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38300/0 (socket says 172.21.15.169:38300) 2024-03-28T00:56:13.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f7994af0640 1 -- 172.21.15.169:0/420421297 learned_addr learned my addr 172.21.15.169:0/420421297 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:13.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f798f7fe640 1 --2- 172.21.15.169:0/420421297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7990102950 0x7f7990197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:13.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f7994af0640 1 -- 172.21.15.169:0/420421297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7990102950 msgr2=0x7f7990197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:13.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f7994af0640 1 --2- 172.21.15.169:0/420421297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7990102950 0x7f7990197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f7994af0640 1 -- 172.21.15.169:0/420421297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79900ff980 msgr2=0x7f7990197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:13.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f7994af0640 1 --2- 172.21.15.169:0/420421297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79900ff980 0x7f7990197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f7994af0640 1 -- 172.21.15.169:0/420421297 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799019f640 con 0x7f7990103310 2024-03-28T00:56:13.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f798ffff640 1 --2- 172.21.15.169:0/420421297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79900ff980 0x7f7990197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:13.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f7994af0640 1 --2- 172.21.15.169:0/420421297 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7990103310 0x7f799019efd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f798800eb10 tx=0x7f798800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:13.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f798d7fa640 1 -- 172.21.15.169:0/420421297 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7988004b20 con 0x7f7990103310 2024-03-28T00:56:13.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f7996553640 1 -- 172.21.15.169:0/420421297 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799019f930 con 0x7f7990103310 2024-03-28T00:56:13.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f798d7fa640 1 -- 172.21.15.169:0/420421297 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7988004cc0 con 0x7f7990103310 2024-03-28T00:56:13.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f798d7fa640 1 -- 172.21.15.169:0/420421297 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7988010760 con 0x7f7990103310 2024-03-28T00:56:13.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.657+0000 7f7996553640 1 -- 172.21.15.169:0/420421297 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f799019fe80 con 0x7f7990103310 2024-03-28T00:56:13.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.658+0000 7f7996553640 1 -- 172.21.15.169:0/420421297 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7990108b00 con 0x7f7990103310 2024-03-28T00:56:13.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.659+0000 7f798d7fa640 1 -- 172.21.15.169:0/420421297 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7988005020 con 0x7f7990103310 2024-03-28T00:56:13.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.660+0000 7f798d7fa640 1 --2- 172.21.15.169:0/420421297 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f79780a2800 0x7f79780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:13.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.660+0000 7f798ffff640 1 --2- 172.21.15.169:0/420421297 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f79780a2800 0x7f79780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:13.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.660+0000 7f798d7fa640 1 -- 172.21.15.169:0/420421297 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f79880cf330 con 0x7f7990103310 2024-03-28T00:56:13.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.660+0000 7f798ffff640 1 --2- 172.21.15.169:0/420421297 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f79780a2800 0x7f79780a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7980009230 tx=0x7f798002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:13.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:13 smithi177 ceph-mon[36206]: pgmap v340: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:56:13.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:13 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1258658790' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:56:13.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:13 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/4199658410' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:56:13.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:13 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3235138285' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:56:13.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.664+0000 7f798d7fa640 1 -- 172.21.15.169:0/420421297 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f798809c070 con 0x7f7990103310 2024-03-28T00:56:13.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.878+0000 7f7996553640 1 -- 172.21.15.169:0/420421297 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f799002d070 con 0x7f7990103310 2024-03-28T00:56:13.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.879+0000 7f798d7fa640 1 -- 172.21.15.169:0/420421297 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11420 (secure 0 0 0) 0x7f798809b970 con 0x7f7990103310 2024-03-28T00:56:13.882 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:13.882 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:54:52.274335+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24457,24478]},"id":1}]} 2024-03-28T00:56:13.883 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 24 2024-03-28T00:56:13.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.881+0000 7f7996553640 1 -- 172.21.15.169:0/420421297 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f79780a2800 msgr2=0x7f79780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:13.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.881+0000 7f7996553640 1 --2- 172.21.15.169:0/420421297 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f79780a2800 0x7f79780a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7980009230 tx=0x7f798002df90 comp rx=0 tx=0).stop 2024-03-28T00:56:13.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.881+0000 7f7996553640 1 -- 172.21.15.169:0/420421297 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7990103310 msgr2=0x7f799019efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:13.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.881+0000 7f7996553640 1 --2- 172.21.15.169:0/420421297 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7990103310 0x7f799019efd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f798800eb10 tx=0x7f798800c2d0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.882+0000 7f7996553640 1 -- 172.21.15.169:0/420421297 shutdown_connections 2024-03-28T00:56:13.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.882+0000 7f7996553640 1 --2- 172.21.15.169:0/420421297 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f79780a2800 0x7f79780a4cc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.882+0000 7f7996553640 1 --2- 172.21.15.169:0/420421297 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7990103310 0x7f799019efd0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.882+0000 7f7996553640 1 --2- 172.21.15.169:0/420421297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7990102950 0x7f7990197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.882+0000 7f7996553640 1 --2- 172.21.15.169:0/420421297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79900ff980 0x7f7990197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:13.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.882+0000 7f7996553640 1 -- 172.21.15.169:0/420421297 >> 172.21.15.169:0/420421297 conn(0x7f79900fb2d0 msgr2=0x7f7990105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:13.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.882+0000 7f7996553640 1 -- 172.21.15.169:0/420421297 shutdown_connections 2024-03-28T00:56:13.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.882+0000 7f7996553640 1 -- 172.21.15.169:0/420421297 wait complete. 2024-03-28T00:56:13.896 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T00:54:32.613335+0000', 'modified': '2024-03-28T00:54:52.274335+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24479, 'mds_1': 24478, 'mds_2': 24457}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24419': {'gid': 24419, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6835/3213155599', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6834', 'nonce': 3213155599}, {'type': 'v1', 'addr': '172.21.15.174:6835', 'nonce': 3213155599}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24445': {'gid': 24445, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6835/1050217570', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6834', 'nonce': 1050217570}, {'type': 'v1', 'addr': '172.21.15.177:6835', 'nonce': 1050217570}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24452': {'gid': 24452, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6837/3629228577', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6836', 'nonce': 3629228577}, {'type': 'v1', 'addr': '172.21.15.174:6837', 'nonce': 3629228577}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24457': {'gid': 24457, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.177:6837/3931247435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6836', 'nonce': 3931247435}, {'type': 'v1', 'addr': '172.21.15.177:6837', 'nonce': 3931247435}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24478': {'gid': 24478, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.177:6839/152605729', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6838', 'nonce': 152605729}, {'type': 'v1', 'addr': '172.21.15.177:6839', 'nonce': 152605729}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24479': {'gid': 24479, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.174:6839/1152414321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6838', 'nonce': 1152414321}, {'type': 'v1', 'addr': '172.21.15.174:6839', 'nonce': 1152414321}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24479, 'qdb_cluster': [24479, 24457, 24478]} 2024-03-28T00:56:13.896 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-28T00:56:13.897 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-28T00:56:14.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.997+0000 7f2eaa1df640 1 -- 172.21.15.169:0/1138032875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ea4103260 msgr2=0x7f2ea4109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.997+0000 7f2eaa1df640 1 --2- 172.21.15.169:0/1138032875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ea4103260 0x7f2ea4109af0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c009ca0 tx=0x7f2e9c02c0f0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.997+0000 7f2eaa1df640 1 -- 172.21.15.169:0/1138032875 shutdown_connections 2024-03-28T00:56:14.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.997+0000 7f2eaa1df640 1 --2- 172.21.15.169:0/1138032875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ea4103260 0x7f2ea4109af0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.997+0000 7f2eaa1df640 1 --2- 172.21.15.169:0/1138032875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ea41028a0 0x7f2ea4102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.997+0000 7f2eaa1df640 1 --2- 172.21.15.169:0/1138032875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ea41016b0 0x7f2ea4101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.997+0000 7f2eaa1df640 1 -- 172.21.15.169:0/1138032875 >> 172.21.15.169:0/1138032875 conn(0x7f2ea40fcc40 msgr2=0x7f2ea40ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:14.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.997+0000 7f2eaa1df640 1 -- 172.21.15.169:0/1138032875 shutdown_connections 2024-03-28T00:56:14.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.997+0000 7f2eaa1df640 1 -- 172.21.15.169:0/1138032875 wait complete. 2024-03-28T00:56:14.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.998+0000 7f2eaa1df640 1 Processor -- start 2024-03-28T00:56:14.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.998+0000 7f2eaa1df640 1 -- start start 2024-03-28T00:56:14.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.998+0000 7f2eaa1df640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ea41016b0 0x7f2ea4197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.998+0000 7f2eaa1df640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ea41028a0 0x7f2ea4197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.998+0000 7f2eaa1df640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ea4103260 0x7f2ea419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.998+0000 7f2eaa1df640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea410d310 con 0x7f2ea4103260 2024-03-28T00:56:14.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.998+0000 7f2eaa1df640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea410d190 con 0x7f2ea41016b0 2024-03-28T00:56:14.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.998+0000 7f2eaa1df640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea410d490 con 0x7f2ea41028a0 2024-03-28T00:56:14.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.998+0000 7f2ea3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ea4103260 0x7f2ea419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.998+0000 7f2ea37fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ea41016b0 0x7f2ea4197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.998+0000 7f2ea3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ea4103260 0x7f2ea419ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38314/0 (socket says 172.21.15.169:38314) 2024-03-28T00:56:14.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.998+0000 7f2ea3fff640 1 -- 172.21.15.169:0/3445509673 learned_addr learned my addr 172.21.15.169:0/3445509673 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:14.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.998+0000 7f2ea2ffd640 1 --2- 172.21.15.169:0/3445509673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ea41028a0 0x7f2ea4197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.999+0000 7f2ea37fe640 1 -- 172.21.15.169:0/3445509673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ea41028a0 msgr2=0x7f2ea4197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.999+0000 7f2ea37fe640 1 --2- 172.21.15.169:0/3445509673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ea41028a0 0x7f2ea4197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.999+0000 7f2ea37fe640 1 -- 172.21.15.169:0/3445509673 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ea4103260 msgr2=0x7f2ea419ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.999+0000 7f2ea37fe640 1 --2- 172.21.15.169:0/3445509673 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ea4103260 0x7f2ea419ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.999+0000 7f2ea37fe640 1 -- 172.21.15.169:0/3445509673 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea419f570 con 0x7f2ea41016b0 2024-03-28T00:56:14.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.999+0000 7f2ea3fff640 1 --2- 172.21.15.169:0/3445509673 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ea4103260 0x7f2ea419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:14.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.999+0000 7f2ea37fe640 1 --2- 172.21.15.169:0/3445509673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ea41016b0 0x7f2ea4197950 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400e9c0 tx=0x7f2e9400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:14.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.999+0000 7f2ea0ff9640 1 -- 172.21.15.169:0/3445509673 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9400cd80 con 0x7f2ea41016b0 2024-03-28T00:56:14.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.999+0000 7f2eaa1df640 1 -- 172.21.15.169:0/3445509673 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea419f860 con 0x7f2ea41016b0 2024-03-28T00:56:14.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.999+0000 7f2ea0ff9640 1 -- 172.21.15.169:0/3445509673 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e94010910 con 0x7f2ea41016b0 2024-03-28T00:56:14.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.999+0000 7f2eaa1df640 1 -- 172.21.15.169:0/3445509673 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea419fe00 con 0x7f2ea41016b0 2024-03-28T00:56:14.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:13.999+0000 7f2ea0ff9640 1 -- 172.21.15.169:0/3445509673 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e94018980 con 0x7f2ea41016b0 2024-03-28T00:56:14.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.002+0000 7f2ea0ff9640 1 -- 172.21.15.169:0/3445509673 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2e9401f020 con 0x7f2ea41016b0 2024-03-28T00:56:14.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.002+0000 7f2eacdc7740 1 Processor -- start 2024-03-28T00:56:14.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.002+0000 7f2eacdc7740 1 -- start start 2024-03-28T00:56:14.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.002+0000 7f2eacdc7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56178872ab50 0x56178872af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.002+0000 7f2eacdc7740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56178872b540 0x561788730040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.002+0000 7f2eacdc7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617887306b0 0x561788732b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.002+0000 7f2eacdc7740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56178868bc20 con 0x56178872b540 2024-03-28T00:56:14.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.002+0000 7f2eacdc7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x561788634a40 con 0x56178872ab50 2024-03-28T00:56:14.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.002+0000 7f2eacdc7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561788687cc0 con 0x5617887306b0 2024-03-28T00:56:14.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.002+0000 7f2ea37fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56178872b540 0x561788730040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.002+0000 7f2ea3fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56178872ab50 0x56178872af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.002+0000 7f2ea37fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56178872b540 0x561788730040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38324/0 (socket says 172.21.15.169:38324) 2024-03-28T00:56:14.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.002+0000 7f2ea3fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56178872ab50 0x56178872af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:54696/0 (socket says 172.21.15.169:54696) 2024-03-28T00:56:14.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.002+0000 7f2ea37fe640 1 -- 172.21.15.169:0/3855613158 learned_addr learned my addr 172.21.15.169:0/3855613158 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:14.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.002+0000 7f2ea2ffd640 1 --2- 172.21.15.169:0/3855613158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617887306b0 0x561788732b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.002+0000 7f2ea0ff9640 1 --2- 172.21.15.169:0/3445509673 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2e8c0a2800 0x7f2e8c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.003+0000 7f2ea37fe640 1 -- 172.21.15.169:0/3855613158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617887306b0 msgr2=0x561788732b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.003+0000 7f2ea37fe640 1 --2- 172.21.15.169:0/3855613158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617887306b0 0x561788732b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.003+0000 7f2ea37fe640 1 -- 172.21.15.169:0/3855613158 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56178872ab50 msgr2=0x56178872af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.003+0000 7f2ea37fe640 1 --2- 172.21.15.169:0/3855613158 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56178872ab50 0x56178872af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.003+0000 7f2ea37fe640 1 -- 172.21.15.169:0/3855613158 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561788656f90 con 0x56178872b540 2024-03-28T00:56:14.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.003+0000 7f2ea2ffd640 1 --2- 172.21.15.169:0/3855613158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617887306b0 0x561788732b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:14.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.003+0000 7f2ea3fff640 1 --2- 172.21.15.169:0/3855613158 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56178872ab50 0x56178872af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:14.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.003+0000 7f2ea3fff640 1 --2- 172.21.15.169:0/3445509673 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2e8c0a2800 0x7f2e8c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.003+0000 7f2ea0ff9640 1 -- 172.21.15.169:0/3445509673 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f2e940cedb0 con 0x7f2ea41016b0 2024-03-28T00:56:14.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.003+0000 7f2ea37fe640 1 --2- 172.21.15.169:0/3855613158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56178872b540 0x561788730040 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2e94097dc0 tx=0x7f2e9400efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32650251c16358b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:14.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.003+0000 7f2e89ffb640 1 -- 172.21.15.169:0/3855613158 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9409b1c0 con 0x56178872b540 2024-03-28T00:56:14.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.003+0000 7f2ea3fff640 1 --2- 172.21.15.169:0/3445509673 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2e8c0a2800 0x7f2e8c0a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c032420 tx=0x7f2e9c02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:14.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.003+0000 7f2e89ffb640 1 -- 172.21.15.169:0/3855613158 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e9409baf0 con 0x56178872b540 2024-03-28T00:56:14.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.003+0000 7f2e89ffb640 1 -- 172.21.15.169:0/3855613158 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e940071c0 con 0x56178872b540 2024-03-28T00:56:14.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.004+0000 7f2eacdc7740 1 -- 172.21.15.169:0/3855613158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56178872b540 msgr2=0x561788730040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.004+0000 7f2eacdc7740 1 --2- 172.21.15.169:0/3855613158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56178872b540 0x561788730040 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2e94097dc0 tx=0x7f2e9400efb0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.004+0000 7f2eacdc7740 1 -- 172.21.15.169:0/3855613158 shutdown_connections 2024-03-28T00:56:14.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.004+0000 7f2eacdc7740 1 --2- 172.21.15.169:0/3855613158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617887306b0 0x561788732b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.004+0000 7f2eacdc7740 1 --2- 172.21.15.169:0/3855613158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56178872b540 0x561788730040 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.004+0000 7f2eacdc7740 1 --2- 172.21.15.169:0/3855613158 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56178872ab50 0x56178872af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.004+0000 7f2eacdc7740 1 -- 172.21.15.169:0/3855613158 >> 172.21.15.169:0/3855613158 conn(0x5617886eea60 msgr2=0x5617887220b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:14.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.004+0000 7f2eacdc7740 1 -- 172.21.15.169:0/3855613158 shutdown_connections 2024-03-28T00:56:14.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.004+0000 7f2eacdc7740 1 -- 172.21.15.169:0/3855613158 wait complete. 2024-03-28T00:56:14.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.004+0000 7f2eacdc7740 1 -- start start 2024-03-28T00:56:14.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.004+0000 7f2eacdc7740 1 Processor -- start 2024-03-28T00:56:14.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.004+0000 7f2eacdc7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56178872ab50 0x561788735060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.004+0000 7f2eacdc7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56178872b540 0x5617887355a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.004+0000 7f2eacdc7740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5617887306b0 0x56178873c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.004+0000 7f2eacdc7740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x561788733600 con 0x5617887306b0 2024-03-28T00:56:14.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.004+0000 7f2eacdc7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x561788647e90 con 0x56178872ab50 2024-03-28T00:56:14.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.004+0000 7f2eacdc7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561788733780 con 0x56178872b540 2024-03-28T00:56:14.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.005+0000 7f2eacdc7740 10 client.0 _ll_register_callbacks cb 0x56178839fac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:56:14.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.005+0000 7f2ea3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5617887306b0 0x56178873c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.005+0000 7f2ea37fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56178872ab50 0x561788735060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.005+0000 7f2ea3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5617887306b0 0x56178873c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38334/0 (socket says 172.21.15.169:38334) 2024-03-28T00:56:14.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.005+0000 7f2ea3fff640 1 -- 172.21.15.169:0/2669716429 learned_addr learned my addr 172.21.15.169:0/2669716429 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:14.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.005+0000 7f2ea2ffd640 1 --2- 172.21.15.169:0/2669716429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56178872b540 0x5617887355a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.005+0000 7f2ea3fff640 1 -- 172.21.15.169:0/2669716429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56178872b540 msgr2=0x5617887355a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.005+0000 7f2ea3fff640 1 --2- 172.21.15.169:0/2669716429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56178872b540 0x5617887355a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.005+0000 7f2ea3fff640 1 -- 172.21.15.169:0/2669716429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56178872ab50 msgr2=0x561788735060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.005+0000 7f2ea3fff640 1 --2- 172.21.15.169:0/2669716429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56178872ab50 0x561788735060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.005+0000 7f2ea3fff640 1 -- 172.21.15.169:0/2669716429 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5617886549d0 con 0x5617887306b0 2024-03-28T00:56:14.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.005+0000 7f2ea3fff640 1 --2- 172.21.15.169:0/2669716429 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5617887306b0 0x56178873c5e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c037040 tx=0x7f2e9c0325b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=532141cd8c4a89c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:14.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.005+0000 7f2e82ffd640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:56:14.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.005+0000 7f2e82ffd640 1 -- 172.21.15.169:0/2669716429 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c046070 con 0x5617887306b0 2024-03-28T00:56:14.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.005+0000 7f2e82ffd640 1 -- 172.21.15.169:0/2669716429 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f2e9c034d80 con 0x5617887306b0 2024-03-28T00:56:14.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.005+0000 7f2eacdc7740 1 -- 172.21.15.169:0/2669716429 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2e9409baf0 con 0x5617887306b0 2024-03-28T00:56:14.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.005+0000 7f2e82ffd640 1 -- 172.21.15.169:0/2669716429 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e9c039040 con 0x5617887306b0 2024-03-28T00:56:14.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.005+0000 7f2eacdc7740 10 client.15219 fetch_fsmap learned FSMap version 24 2024-03-28T00:56:14.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.005+0000 7f2eacdc7740 1 -- 172.21.15.169:0/2669716429 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5617883a02d0 con 0x5617887306b0 2024-03-28T00:56:14.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.006+0000 7f2e82ffd640 1 -- 172.21.15.169:0/2669716429 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e9409baf0 con 0x5617887306b0 2024-03-28T00:56:14.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.006+0000 7f2e82ffd640 1 -- 172.21.15.169:0/2669716429 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2e9c03f040 con 0x5617887306b0 2024-03-28T00:56:14.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.006+0000 7f2eacdc7740 10 client.15219 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T00:56:14.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.006+0000 7f2eacdc7740 10 client.15219 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T00:56:14.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.006+0000 7f2eacdc7740 20 client.15219 populate_metadata read hostname 'smithi169' 2024-03-28T00:56:14.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.006+0000 7f2eacdc7740 1 --2- 172.21.15.169:0/2669716429 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x5617887418d0 0x5617887408c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.006+0000 7f2eacdc7740 4 client.15219 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:56:14.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.006+0000 7f2eacdc7740 1 -- 172.21.15.169:0/2669716429 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561788699160 con 0x5617887418d0 2024-03-28T00:56:14.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.006+0000 7f2ea37fe640 1 --2- 172.21.15.169:0/2669716429 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x5617887418d0 0x5617887408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.006+0000 7f2e82ffd640 10 client.15219 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T00:56:14.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.006+0000 7f2ea37fe640 1 --2- 172.21.15.169:0/2669716429 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x5617887418d0 0x5617887408c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=25c2438e52b9db13 server_cookie=f51bbacc0a9e3086 in_seq=0 out_seq=0 2024-03-28T00:56:14.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.007+0000 7f2e82ffd640 1 -- 172.21.15.169:0/2669716429 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561788699160 con 0x5617887418d0 2024-03-28T00:56:14.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.007+0000 7f2e82ffd640 10 client.15219 handle_command_reply: tid=0 2024-03-28T00:56:14.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.007+0000 7f2eacdc7740 1 client.15219 shutdown 2024-03-28T00:56:14.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.007+0000 7f2eacdc7740 1 -- 172.21.15.169:0/2669716429 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5617887306b0 msgr2=0x56178873c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.007+0000 7f2eacdc7740 1 --2- 172.21.15.169:0/2669716429 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5617887306b0 0x56178873c5e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c037040 tx=0x7f2e9c0325b0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.007+0000 7f2eacdc7740 1 -- 172.21.15.169:0/2669716429 shutdown_connections 2024-03-28T00:56:14.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.007+0000 7f2eacdc7740 1 --2- 172.21.15.169:0/2669716429 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x5617887418d0 0x5617887408c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.007+0000 7f2eacdc7740 1 --2- 172.21.15.169:0/2669716429 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5617887306b0 0x56178873c5e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.007+0000 7f2eacdc7740 1 --2- 172.21.15.169:0/2669716429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56178872b540 0x5617887355a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.007+0000 7f2eacdc7740 1 --2- 172.21.15.169:0/2669716429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56178872ab50 0x561788735060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.007+0000 7f2eacdc7740 1 -- 172.21.15.169:0/2669716429 >> 172.21.15.169:0/2669716429 conn(0x5617886eea60 msgr2=0x5617887212f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:14.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.007+0000 7f2e82ffd640 0 client.15219 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T00:56:14.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.007+0000 7f2eacdc7740 1 -- 172.21.15.169:0/2669716429 shutdown_connections 2024-03-28T00:56:14.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.007+0000 7f2eacdc7740 1 -- 172.21.15.169:0/2669716429 wait complete. 2024-03-28T00:56:14.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.007+0000 7f2eacdc7740 20 client.15219 trim_cache size 0 max 16384 2024-03-28T00:56:14.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.024+0000 7f2eacdc7740 1 Processor -- start 2024-03-28T00:56:14.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.024+0000 7f2eacdc7740 1 -- start start 2024-03-28T00:56:14.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.024+0000 7f2eacdc7740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56178872ab50 0x56178872ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.024+0000 7f2eacdc7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56178872b540 0x56178872cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.024+0000 7f2eacdc7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617887306b0 0x56178873c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.024+0000 7f2eacdc7740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617887410f0 con 0x56178872ab50 2024-03-28T00:56:14.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.024+0000 7f2eacdc7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x561788741750 con 0x56178872b540 2024-03-28T00:56:14.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.024+0000 7f2eacdc7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561788741db0 con 0x5617887306b0 2024-03-28T00:56:14.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.024+0000 7f2ea37fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56178872ab50 0x56178872ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.024+0000 7f2ea37fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56178872ab50 0x56178872ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38342/0 (socket says 172.21.15.169:38342) 2024-03-28T00:56:14.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.024+0000 7f2ea37fe640 1 -- 172.21.15.169:0/461024581 learned_addr learned my addr 172.21.15.169:0/461024581 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:14.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.024+0000 7f2ea2ffd640 1 --2- 172.21.15.169:0/461024581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56178872b540 0x56178872cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.024+0000 7f2ea3fff640 1 --2- 172.21.15.169:0/461024581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617887306b0 0x56178873c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.024+0000 7f2ea37fe640 1 -- 172.21.15.169:0/461024581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617887306b0 msgr2=0x56178873c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.024+0000 7f2ea37fe640 1 --2- 172.21.15.169:0/461024581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617887306b0 0x56178873c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.024+0000 7f2ea37fe640 1 -- 172.21.15.169:0/461024581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56178872b540 msgr2=0x56178872cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.024+0000 7f2ea37fe640 1 --2- 172.21.15.169:0/461024581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56178872b540 0x56178872cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.024+0000 7f2ea37fe640 1 -- 172.21.15.169:0/461024581 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56178846a6b0 con 0x56178872ab50 2024-03-28T00:56:14.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.024+0000 7f2ea2ffd640 1 --2- 172.21.15.169:0/461024581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56178872b540 0x56178872cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:56:14.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.024+0000 7f2ea37fe640 1 --2- 172.21.15.169:0/461024581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56178872ab50 0x56178872ca40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2e94000c00 tx=0x7f2e940cc4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dac24c4aa93d009 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:14.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.025+0000 7f2e897fa640 1 -- 172.21.15.169:0/461024581 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9409ad50 con 0x56178872ab50 2024-03-28T00:56:14.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.025+0000 7f2e897fa640 1 -- 172.21.15.169:0/461024581 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e940a3b90 con 0x56178872ab50 2024-03-28T00:56:14.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.025+0000 7f2e897fa640 1 -- 172.21.15.169:0/461024581 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9409af30 con 0x56178872ab50 2024-03-28T00:56:14.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.025+0000 7f2eacdc7740 1 -- 172.21.15.169:0/461024581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56178872ab50 msgr2=0x56178872ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.025+0000 7f2eacdc7740 1 --2- 172.21.15.169:0/461024581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56178872ab50 0x56178872ca40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2e94000c00 tx=0x7f2e940cc4c0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.025+0000 7f2eacdc7740 1 -- 172.21.15.169:0/461024581 shutdown_connections 2024-03-28T00:56:14.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.025+0000 7f2eacdc7740 1 --2- 172.21.15.169:0/461024581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617887306b0 0x56178873c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.025+0000 7f2eacdc7740 1 --2- 172.21.15.169:0/461024581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56178872b540 0x56178872cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.025+0000 7f2eacdc7740 1 --2- 172.21.15.169:0/461024581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56178872ab50 0x56178872ca40 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.025+0000 7f2eacdc7740 1 -- 172.21.15.169:0/461024581 >> 172.21.15.169:0/461024581 conn(0x5617886eea60 msgr2=0x5617887408c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:14.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.025+0000 7f2eacdc7740 1 -- 172.21.15.169:0/461024581 shutdown_connections 2024-03-28T00:56:14.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.025+0000 7f2eacdc7740 1 -- 172.21.15.169:0/461024581 wait complete. 2024-03-28T00:56:14.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.025+0000 7f2eacdc7740 1 -- start start 2024-03-28T00:56:14.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2eacdc7740 1 Processor -- start 2024-03-28T00:56:14.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2eacdc7740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56178872ab50 0x5617887385b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2eacdc7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56178872b540 0x561788738af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2eacdc7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5617887306b0 0x56178873bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2eacdc7740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x561788742a70 con 0x56178872ab50 2024-03-28T00:56:14.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2eacdc7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x561788742410 con 0x5617887306b0 2024-03-28T00:56:14.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2eacdc7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617887430d0 con 0x56178872b540 2024-03-28T00:56:14.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2eacdc7740 10 client.0 _ll_register_callbacks cb 0x7f2e98005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:56:14.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2ea37fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56178872ab50 0x5617887385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2ea37fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56178872ab50 0x5617887385b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38348/0 (socket says 172.21.15.169:38348) 2024-03-28T00:56:14.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2ea37fe640 1 -- 172.21.15.169:0/1047672840 learned_addr learned my addr 172.21.15.169:0/1047672840 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:14.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2ea2ffd640 1 --2- 172.21.15.169:0/1047672840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56178872b540 0x561788738af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2ea3fff640 1 --2- 172.21.15.169:0/1047672840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5617887306b0 0x56178873bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2ea37fe640 1 -- 172.21.15.169:0/1047672840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56178872b540 msgr2=0x561788738af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2ea37fe640 1 --2- 172.21.15.169:0/1047672840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56178872b540 0x561788738af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2ea37fe640 1 -- 172.21.15.169:0/1047672840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5617887306b0 msgr2=0x56178873bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2ea37fe640 1 --2- 172.21.15.169:0/1047672840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5617887306b0 0x56178873bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2ea37fe640 1 -- 172.21.15.169:0/1047672840 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56178873c0b0 con 0x56178872ab50 2024-03-28T00:56:14.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2ea37fe640 1 --2- 172.21.15.169:0/1047672840 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56178872ab50 0x5617887385b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400cf60 tx=0x7f2e94009490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2ed281a34163df3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:14.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2e82ffd640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:56:14.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2e82ffd640 1 -- 172.21.15.169:0/1047672840 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e940b4060 con 0x56178872ab50 2024-03-28T00:56:14.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2e82ffd640 1 -- 172.21.15.169:0/1047672840 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f2e94099490 con 0x56178872ab50 2024-03-28T00:56:14.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.026+0000 7f2eacdc7740 1 -- 172.21.15.169:0/1047672840 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2e940a3b90 con 0x56178872ab50 2024-03-28T00:56:14.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.027+0000 7f2e82ffd640 1 -- 172.21.15.169:0/1047672840 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e9409ad50 con 0x56178872ab50 2024-03-28T00:56:14.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.027+0000 7f2eacdc7740 10 client.15225 fetch_fsmap learned FSMap version 24 2024-03-28T00:56:14.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.027+0000 7f2eacdc7740 1 -- 172.21.15.169:0/1047672840 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56178879d110 con 0x56178872ab50 2024-03-28T00:56:14.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.027+0000 7f2e82ffd640 1 -- 172.21.15.169:0/1047672840 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e940a3b90 con 0x56178872ab50 2024-03-28T00:56:14.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.027+0000 7f2e82ffd640 1 -- 172.21.15.169:0/1047672840 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2e94097a60 con 0x56178872ab50 2024-03-28T00:56:14.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.027+0000 7f2eacdc7740 10 client.15225 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T00:56:14.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.027+0000 7f2eacdc7740 10 client.15225 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T00:56:14.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.027+0000 7f2eacdc7740 20 client.15225 populate_metadata read hostname 'smithi169' 2024-03-28T00:56:14.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.027+0000 7f2eacdc7740 1 --2- 172.21.15.169:0/1047672840 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x5617887a10f0 0x56178879d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.027+0000 7f2eacdc7740 4 client.15225 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T00:56:14.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.027+0000 7f2eacdc7740 1 -- 172.21.15.169:0/1047672840 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5617886934b0 con 0x5617887a10f0 2024-03-28T00:56:14.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.027+0000 7f2ea2ffd640 1 --2- 172.21.15.169:0/1047672840 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x5617887a10f0 0x56178879d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.028+0000 7f2e82ffd640 10 client.15225 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T00:56:14.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.028+0000 7f2ea2ffd640 1 --2- 172.21.15.169:0/1047672840 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x5617887a10f0 0x56178879d2e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d9a562b00b52069d server_cookie=fafd224e15bde9c8 in_seq=0 out_seq=0 2024-03-28T00:56:14.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.028+0000 7f2e82ffd640 1 -- 172.21.15.169:0/1047672840 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5617886934b0 con 0x5617887a10f0 2024-03-28T00:56:14.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.028+0000 7f2e82ffd640 10 client.15225 handle_command_reply: tid=0 2024-03-28T00:56:14.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.028+0000 7f2eacdc7740 1 client.15225 shutdown 2024-03-28T00:56:14.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.028+0000 7f2eacdc7740 1 -- 172.21.15.169:0/1047672840 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56178872ab50 msgr2=0x5617887385b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.028+0000 7f2eacdc7740 1 --2- 172.21.15.169:0/1047672840 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56178872ab50 0x5617887385b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400cf60 tx=0x7f2e94009490 comp rx=0 tx=0).stop 2024-03-28T00:56:14.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.028+0000 7f2eacdc7740 1 -- 172.21.15.169:0/1047672840 shutdown_connections 2024-03-28T00:56:14.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.028+0000 7f2eacdc7740 1 --2- 172.21.15.169:0/1047672840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5617887306b0 0x56178873bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.028+0000 7f2eacdc7740 1 --2- 172.21.15.169:0/1047672840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56178872b540 0x561788738af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.028+0000 7f2eacdc7740 1 --2- 172.21.15.169:0/1047672840 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x5617887a10f0 0x56178879d2e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.028+0000 7f2eacdc7740 1 --2- 172.21.15.169:0/1047672840 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56178872ab50 0x5617887385b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.028+0000 7f2eacdc7740 1 -- 172.21.15.169:0/1047672840 >> 172.21.15.169:0/1047672840 conn(0x5617886eea60 msgr2=0x561788721bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:14.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.028+0000 7f2e82ffd640 0 client.15225 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T00:56:14.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.028+0000 7f2eacdc7740 1 -- 172.21.15.169:0/1047672840 shutdown_connections 2024-03-28T00:56:14.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.028+0000 7f2eacdc7740 1 -- 172.21.15.169:0/1047672840 wait complete. 2024-03-28T00:56:14.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.028+0000 7f2eacdc7740 20 client.15225 trim_cache size 0 max 16384 2024-03-28T00:56:14.039 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:56:14.039 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_fsid": "833368da-ec9c-11ee-95d0-87774f69a715", 2024-03-28T00:56:14.039 INFO:teuthology.orchestra.run.smithi169.stdout: "whoami": 2, 2024-03-28T00:56:14.039 INFO:teuthology.orchestra.run.smithi169.stdout: "id": 24457, 2024-03-28T00:56:14.039 INFO:teuthology.orchestra.run.smithi169.stdout: "want_state": "up:active", 2024-03-28T00:56:14.039 INFO:teuthology.orchestra.run.smithi169.stdout: "state": "up:active", 2024-03-28T00:56:14.039 INFO:teuthology.orchestra.run.smithi169.stdout: "fs_name": "cephfs", 2024-03-28T00:56:14.039 INFO:teuthology.orchestra.run.smithi169.stdout: "rank_uptime": 82.760187939999994, 2024-03-28T00:56:14.039 INFO:teuthology.orchestra.run.smithi169.stdout: "mdsmap_epoch": 24, 2024-03-28T00:56:14.039 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_epoch": 92, 2024-03-28T00:56:14.039 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_epoch_barrier": 90, 2024-03-28T00:56:14.040 INFO:teuthology.orchestra.run.smithi169.stdout: "uptime": 119.280805359 2024-03-28T00:56:14.040 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:56:14.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.029+0000 7f2eaa1df640 1 -- 172.21.15.169:0/3445509673 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2e8c0a2800 msgr2=0x7f2e8c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.029+0000 7f2eaa1df640 1 --2- 172.21.15.169:0/3445509673 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2e8c0a2800 0x7f2e8c0a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c032420 tx=0x7f2e9c02e040 comp rx=0 tx=0).stop 2024-03-28T00:56:14.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.029+0000 7f2eaa1df640 1 -- 172.21.15.169:0/3445509673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ea41016b0 msgr2=0x7f2ea4197950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.029+0000 7f2eaa1df640 1 --2- 172.21.15.169:0/3445509673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ea41016b0 0x7f2ea4197950 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400e9c0 tx=0x7f2e9400ee80 comp rx=0 tx=0).stop 2024-03-28T00:56:14.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.029+0000 7f2eaa1df640 1 -- 172.21.15.169:0/3445509673 shutdown_connections 2024-03-28T00:56:14.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.029+0000 7f2eaa1df640 1 --2- 172.21.15.169:0/3445509673 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2e8c0a2800 0x7f2e8c0a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.029+0000 7f2eaa1df640 1 --2- 172.21.15.169:0/3445509673 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ea4103260 0x7f2ea419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.029+0000 7f2eaa1df640 1 --2- 172.21.15.169:0/3445509673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ea41028a0 0x7f2ea4197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.029+0000 7f2eaa1df640 1 --2- 172.21.15.169:0/3445509673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ea41016b0 0x7f2ea4197950 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.029+0000 7f2eaa1df640 1 -- 172.21.15.169:0/3445509673 >> 172.21.15.169:0/3445509673 conn(0x7f2ea40fcc40 msgr2=0x7f2ea4105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:14.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.029+0000 7f2eaa1df640 1 -- 172.21.15.169:0/3445509673 shutdown_connections 2024-03-28T00:56:14.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.029+0000 7f2eaa1df640 1 -- 172.21.15.169:0/3445509673 wait complete. 2024-03-28T00:56:14.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-28T00:56:14.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.143+0000 7fd6f6075640 1 -- 172.21.15.169:0/1877979995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6f0102970 msgr2=0x7fd6f0102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.143+0000 7fd6f6075640 1 --2- 172.21.15.169:0/1877979995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6f0102970 0x7fd6f0102df0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4009890 tx=0x7fd6e402bb70 comp rx=0 tx=0).stop 2024-03-28T00:56:14.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.143+0000 7fd6f6075640 1 -- 172.21.15.169:0/1877979995 shutdown_connections 2024-03-28T00:56:14.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.143+0000 7fd6f6075640 1 --2- 172.21.15.169:0/1877979995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6f0103330 0x7fd6f0109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.143+0000 7fd6f6075640 1 --2- 172.21.15.169:0/1877979995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6f0102970 0x7fd6f0102df0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.143+0000 7fd6f6075640 1 --2- 172.21.15.169:0/1877979995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6f00ff9a0 0x7fd6f00ffdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.143+0000 7fd6f6075640 1 -- 172.21.15.169:0/1877979995 >> 172.21.15.169:0/1877979995 conn(0x7fd6f00fb2d0 msgr2=0x7fd6f00fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:14.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.143+0000 7fd6f6075640 1 -- 172.21.15.169:0/1877979995 shutdown_connections 2024-03-28T00:56:14.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.143+0000 7fd6f6075640 1 -- 172.21.15.169:0/1877979995 wait complete. 2024-03-28T00:56:14.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.144+0000 7fd6f6075640 1 Processor -- start 2024-03-28T00:56:14.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.144+0000 7fd6f6075640 1 -- start start 2024-03-28T00:56:14.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.144+0000 7fd6f6075640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6f00ff9a0 0x7fd6f0197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.144+0000 7fd6f6075640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6f0102970 0x7fd6f0197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.144+0000 7fd6f6075640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6f0103330 0x7fd6f019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.144+0000 7fd6f6075640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f010d360 con 0x7fd6f00ff9a0 2024-03-28T00:56:14.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.144+0000 7fd6f6075640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f010d1e0 con 0x7fd6f0103330 2024-03-28T00:56:14.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.144+0000 7fd6f6075640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f010d4e0 con 0x7fd6f0102970 2024-03-28T00:56:14.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.144+0000 7fd6ef7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6f00ff9a0 0x7fd6f0197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.144+0000 7fd6ef7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6f00ff9a0 0x7fd6f0197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38352/0 (socket says 172.21.15.169:38352) 2024-03-28T00:56:14.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.144+0000 7fd6eeffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6f0102970 0x7fd6f0197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.144+0000 7fd6ef7fe640 1 -- 172.21.15.169:0/4098611352 learned_addr learned my addr 172.21.15.169:0/4098611352 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:14.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.144+0000 7fd6effff640 1 --2- 172.21.15.169:0/4098611352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6f0103330 0x7fd6f019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.144+0000 7fd6ef7fe640 1 -- 172.21.15.169:0/4098611352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6f0102970 msgr2=0x7fd6f0197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.144+0000 7fd6ef7fe640 1 --2- 172.21.15.169:0/4098611352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6f0102970 0x7fd6f0197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.144+0000 7fd6ef7fe640 1 -- 172.21.15.169:0/4098611352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6f0103330 msgr2=0x7fd6f019efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.144+0000 7fd6ef7fe640 1 --2- 172.21.15.169:0/4098611352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6f0103330 0x7fd6f019efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.144+0000 7fd6ef7fe640 1 -- 172.21.15.169:0/4098611352 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f019f630 con 0x7fd6f00ff9a0 2024-03-28T00:56:14.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.144+0000 7fd6eeffd640 1 --2- 172.21.15.169:0/4098611352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6f0102970 0x7fd6f0197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:14.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.144+0000 7fd6effff640 1 --2- 172.21.15.169:0/4098611352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6f0103330 0x7fd6f019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:14.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.145+0000 7fd6ef7fe640 1 --2- 172.21.15.169:0/4098611352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6f00ff9a0 0x7fd6f0197a10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd6e000e9a0 tx=0x7fd6e000eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:14.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.145+0000 7fd6ecff9640 1 -- 172.21.15.169:0/4098611352 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e000ccc0 con 0x7fd6f00ff9a0 2024-03-28T00:56:14.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.145+0000 7fd6f6075640 1 -- 172.21.15.169:0/4098611352 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f019f920 con 0x7fd6f00ff9a0 2024-03-28T00:56:14.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.145+0000 7fd6ecff9640 1 -- 172.21.15.169:0/4098611352 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6e0010910 con 0x7fd6f00ff9a0 2024-03-28T00:56:14.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.145+0000 7fd6ecff9640 1 -- 172.21.15.169:0/4098611352 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0018a30 con 0x7fd6f00ff9a0 2024-03-28T00:56:14.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.145+0000 7fd6f6075640 1 -- 172.21.15.169:0/4098611352 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f019fe90 con 0x7fd6f00ff9a0 2024-03-28T00:56:14.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.147+0000 7fd6ecff9640 1 -- 172.21.15.169:0/4098611352 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd6e001f020 con 0x7fd6f00ff9a0 2024-03-28T00:56:14.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.147+0000 7fd6f8c5b740 1 Processor -- start 2024-03-28T00:56:14.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.147+0000 7fd6f8c5b740 1 -- start start 2024-03-28T00:56:14.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.148+0000 7fd6f8c5b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556a2d5f740 0x5556a2d5fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.148+0000 7fd6f8c5b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d60130 0x5556a2d64c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.148+0000 7fd6f8c5b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d652a0 0x5556a2d67730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.148+0000 7fd6f8c5b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a2cc4c20 con 0x5556a2d652a0 2024-03-28T00:56:14.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.148+0000 7fd6f8c5b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a2c6da40 con 0x5556a2d60130 2024-03-28T00:56:14.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.148+0000 7fd6f8c5b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a2cc0cc0 con 0x5556a2d5f740 2024-03-28T00:56:14.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.148+0000 7fd6ecff9640 1 --2- 172.21.15.169:0/4098611352 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd6d80a2800 0x7fd6d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.148+0000 7fd6eeffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d652a0 0x5556a2d67730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.148+0000 7fd6effff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556a2d5f740 0x5556a2d5fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.148+0000 7fd6eeffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d652a0 0x5556a2d67730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38358/0 (socket says 172.21.15.169:38358) 2024-03-28T00:56:14.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.148+0000 7fd6eeffd640 1 -- 172.21.15.169:0/4105915735 learned_addr learned my addr 172.21.15.169:0/4105915735 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:14.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.148+0000 7fd6ef7fe640 1 --2- 172.21.15.169:0/4105915735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d60130 0x5556a2d64c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.148+0000 7fd6effff640 1 --2- 172.21.15.169:0/4098611352 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd6d80a2800 0x7fd6d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.148+0000 7fd6ecff9640 1 -- 172.21.15.169:0/4098611352 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fd6e00cefa0 con 0x7fd6f00ff9a0 2024-03-28T00:56:14.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.148+0000 7fd6eeffd640 1 -- 172.21.15.169:0/4105915735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556a2d5f740 msgr2=0x5556a2d5fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.148+0000 7fd6eeffd640 1 --2- 172.21.15.169:0/4105915735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556a2d5f740 0x5556a2d5fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.148+0000 7fd6eeffd640 1 -- 172.21.15.169:0/4105915735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d60130 msgr2=0x5556a2d64c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.148+0000 7fd6eeffd640 1 --2- 172.21.15.169:0/4105915735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d60130 0x5556a2d64c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.148+0000 7fd6eeffd640 1 -- 172.21.15.169:0/4105915735 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556a2c8ff90 con 0x5556a2d652a0 2024-03-28T00:56:14.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.148+0000 7fd6eeffd640 1 --2- 172.21.15.169:0/4105915735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d652a0 0x5556a2d67730 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd6e40349e0 tx=0x7fd6e4034ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92945966c8a3eeac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:14.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.148+0000 7fd6effff640 1 --2- 172.21.15.169:0/4098611352 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd6d80a2800 0x7fd6d80a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x5556a2d60990 tx=0x7fd6e8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:14.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.149+0000 7fd6d5ffb640 1 -- 172.21.15.169:0/4105915735 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4039040 con 0x5556a2d652a0 2024-03-28T00:56:14.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.149+0000 7fd6d5ffb640 1 -- 172.21.15.169:0/4105915735 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6e4034640 con 0x5556a2d652a0 2024-03-28T00:56:14.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.149+0000 7fd6d5ffb640 1 -- 172.21.15.169:0/4105915735 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e402cad0 con 0x5556a2d652a0 2024-03-28T00:56:14.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.149+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/4105915735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d652a0 msgr2=0x5556a2d67730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.149+0000 7fd6f8c5b740 1 --2- 172.21.15.169:0/4105915735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d652a0 0x5556a2d67730 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd6e40349e0 tx=0x7fd6e4034ad0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.149+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/4105915735 shutdown_connections 2024-03-28T00:56:14.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.149+0000 7fd6f8c5b740 1 --2- 172.21.15.169:0/4105915735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d652a0 0x5556a2d67730 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.149+0000 7fd6f8c5b740 1 --2- 172.21.15.169:0/4105915735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d60130 0x5556a2d64c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.149+0000 7fd6f8c5b740 1 --2- 172.21.15.169:0/4105915735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556a2d5f740 0x5556a2d5fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.149+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/4105915735 >> 172.21.15.169:0/4105915735 conn(0x5556a2d27a60 msgr2=0x5556a2d5b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:14.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.149+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/4105915735 shutdown_connections 2024-03-28T00:56:14.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.149+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/4105915735 wait complete. 2024-03-28T00:56:14.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.150+0000 7fd6f8c5b740 1 -- start start 2024-03-28T00:56:14.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.150+0000 7fd6f8c5b740 1 Processor -- start 2024-03-28T00:56:14.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.150+0000 7fd6f8c5b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d5f740 0x5556a2d69b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.150+0000 7fd6f8c5b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d60130 0x5556a2d6a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.150+0000 7fd6f8c5b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556a2d652a0 0x5556a2d710d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.150+0000 7fd6f8c5b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a2d681f0 con 0x5556a2d60130 2024-03-28T00:56:14.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.150+0000 7fd6f8c5b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a2c80e90 con 0x5556a2d5f740 2024-03-28T00:56:14.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.150+0000 7fd6f8c5b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a2d68370 con 0x5556a2d652a0 2024-03-28T00:56:14.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.150+0000 7fd6f8c5b740 10 client.0 _ll_register_callbacks cb 0x5556a29d8ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:56:14.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.150+0000 7fd6ef7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d5f740 0x5556a2d69b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.150+0000 7fd6eeffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d60130 0x5556a2d6a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.150+0000 7fd6ef7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d5f740 0x5556a2d69b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:54770/0 (socket says 172.21.15.169:54770) 2024-03-28T00:56:14.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.150+0000 7fd6ef7fe640 1 -- 172.21.15.169:0/1729774652 learned_addr learned my addr 172.21.15.169:0/1729774652 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:14.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.150+0000 7fd6effff640 1 --2- 172.21.15.169:0/1729774652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556a2d652a0 0x5556a2d710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.150+0000 7fd6ef7fe640 1 -- 172.21.15.169:0/1729774652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556a2d652a0 msgr2=0x5556a2d710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.150+0000 7fd6ef7fe640 1 --2- 172.21.15.169:0/1729774652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556a2d652a0 0x5556a2d710d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.150+0000 7fd6ef7fe640 1 -- 172.21.15.169:0/1729774652 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d60130 msgr2=0x5556a2d6a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.151+0000 7fd6ef7fe640 1 --2- 172.21.15.169:0/1729774652 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d60130 0x5556a2d6a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.151+0000 7fd6ef7fe640 1 -- 172.21.15.169:0/1729774652 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5556a2c8d9d0 con 0x5556a2d5f740 2024-03-28T00:56:14.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.151+0000 7fd6eeffd640 1 --2- 172.21.15.169:0/1729774652 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d60130 0x5556a2d6a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:14.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.151+0000 7fd6ef7fe640 1 --2- 172.21.15.169:0/1729774652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d5f740 0x5556a2d69b50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd6e000c6a0 tx=0x7fd6e006b100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad4a2ed20d69bda5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:14.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.151+0000 7fd6ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:56:14.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.151+0000 7fd6ceffd640 1 -- 172.21.15.169:0/1729774652 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e006c070 con 0x5556a2d5f740 2024-03-28T00:56:14.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.151+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/1729774652 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6e4034640 con 0x5556a2d5f740 2024-03-28T00:56:14.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.151+0000 7fd6ceffd640 1 -- 172.21.15.169:0/1729774652 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fd6e00d10b0 con 0x5556a2d5f740 2024-03-28T00:56:14.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.151+0000 7fd6ceffd640 1 -- 172.21.15.169:0/1729774652 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6e009b120 con 0x5556a2d5f740 2024-03-28T00:56:14.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.151+0000 7fd6f8c5b740 10 client.25061 fetch_fsmap learned FSMap version 24 2024-03-28T00:56:14.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.151+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/1729774652 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5556a29d92d0 con 0x5556a2d5f740 2024-03-28T00:56:14.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.151+0000 7fd6ceffd640 1 -- 172.21.15.169:0/1729774652 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6e4034640 con 0x5556a2d5f740 2024-03-28T00:56:14.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.151+0000 7fd6ceffd640 1 -- 172.21.15.169:0/1729774652 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd6e00a38f0 con 0x5556a2d5f740 2024-03-28T00:56:14.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.151+0000 7fd6f8c5b740 10 client.25061 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T00:56:14.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.151+0000 7fd6f8c5b740 10 client.25061 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T00:56:14.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.151+0000 7fd6f8c5b740 20 client.25061 populate_metadata read hostname 'smithi169' 2024-03-28T00:56:14.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.151+0000 7fd6f8c5b740 1 --2- 172.21.15.169:0/1729774652 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5556a2d76390 0x5556a2d753b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.151+0000 7fd6f8c5b740 4 client.25061 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:56:14.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.151+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/1729774652 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5556a2cd2160 con 0x5556a2d76390 2024-03-28T00:56:14.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.152+0000 7fd6eeffd640 1 --2- 172.21.15.169:0/1729774652 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5556a2d76390 0x5556a2d753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.152+0000 7fd6eeffd640 1 --2- 172.21.15.169:0/1729774652 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5556a2d76390 0x5556a2d753b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f6a1d93a1cf547e2 server_cookie=15db89e7a907ed23 in_seq=0 out_seq=0 2024-03-28T00:56:14.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.152+0000 7fd6ceffd640 10 client.25061 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T00:56:14.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.153+0000 7fd6ceffd640 1 -- 172.21.15.169:0/1729774652 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5556a2cd2160 con 0x5556a2d76390 2024-03-28T00:56:14.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.153+0000 7fd6ceffd640 10 client.25061 handle_command_reply: tid=0 2024-03-28T00:56:14.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.153+0000 7fd6f8c5b740 1 client.25061 shutdown 2024-03-28T00:56:14.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.153+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/1729774652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d5f740 msgr2=0x5556a2d69b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.153+0000 7fd6f8c5b740 1 --2- 172.21.15.169:0/1729774652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d5f740 0x5556a2d69b50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd6e000c6a0 tx=0x7fd6e006b100 comp rx=0 tx=0).stop 2024-03-28T00:56:14.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.153+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/1729774652 shutdown_connections 2024-03-28T00:56:14.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.153+0000 7fd6f8c5b740 1 --2- 172.21.15.169:0/1729774652 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5556a2d76390 0x5556a2d753b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.153+0000 7fd6f8c5b740 1 --2- 172.21.15.169:0/1729774652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556a2d652a0 0x5556a2d710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.153+0000 7fd6ceffd640 0 client.25061 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T00:56:14.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.153+0000 7fd6f8c5b740 1 --2- 172.21.15.169:0/1729774652 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d60130 0x5556a2d6a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.153+0000 7fd6f8c5b740 1 --2- 172.21.15.169:0/1729774652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d5f740 0x5556a2d69b50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.153+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/1729774652 >> 172.21.15.169:0/1729774652 conn(0x5556a2d27a60 msgr2=0x5556a2d5a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:14.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.153+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/1729774652 shutdown_connections 2024-03-28T00:56:14.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.153+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/1729774652 wait complete. 2024-03-28T00:56:14.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.153+0000 7fd6f8c5b740 20 client.25061 trim_cache size 0 max 16384 2024-03-28T00:56:14.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.170+0000 7fd6f8c5b740 1 Processor -- start 2024-03-28T00:56:14.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.170+0000 7fd6f8c5b740 1 -- start start 2024-03-28T00:56:14.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.170+0000 7fd6f8c5b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d5f740 0x5556a2d5ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.170+0000 7fd6f8c5b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d60130 0x5556a2d68770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.170+0000 7fd6f8c5b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556a2d652a0 0x5556a2d68cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.170+0000 7fd6f8c5b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a2d75bb0 con 0x5556a2d60130 2024-03-28T00:56:14.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.170+0000 7fd6f8c5b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a2d76210 con 0x5556a2d5f740 2024-03-28T00:56:14.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.170+0000 7fd6f8c5b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a2d76870 con 0x5556a2d652a0 2024-03-28T00:56:14.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.170+0000 7fd6eeffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d60130 0x5556a2d68770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.170+0000 7fd6ef7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d5f740 0x5556a2d5ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.170+0000 7fd6eeffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d60130 0x5556a2d68770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38368/0 (socket says 172.21.15.169:38368) 2024-03-28T00:56:14.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.170+0000 7fd6eeffd640 1 -- 172.21.15.169:0/3684286396 learned_addr learned my addr 172.21.15.169:0/3684286396 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:14.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.170+0000 7fd6effff640 1 --2- 172.21.15.169:0/3684286396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556a2d652a0 0x5556a2d68cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.170+0000 7fd6eeffd640 1 -- 172.21.15.169:0/3684286396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556a2d652a0 msgr2=0x5556a2d68cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.170+0000 7fd6eeffd640 1 --2- 172.21.15.169:0/3684286396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556a2d652a0 0x5556a2d68cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.170+0000 7fd6eeffd640 1 -- 172.21.15.169:0/3684286396 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d5f740 msgr2=0x5556a2d5ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.170+0000 7fd6eeffd640 1 --2- 172.21.15.169:0/3684286396 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d5f740 0x5556a2d5ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.170+0000 7fd6eeffd640 1 -- 172.21.15.169:0/3684286396 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556a2aa36b0 con 0x5556a2d60130 2024-03-28T00:56:14.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.170+0000 7fd6ef7fe640 1 --2- 172.21.15.169:0/3684286396 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d5f740 0x5556a2d5ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:14.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.170+0000 7fd6eeffd640 1 --2- 172.21.15.169:0/3684286396 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d60130 0x5556a2d68770 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd6e40057c0 tx=0x7fd6e402bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ef693f4812ff9a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:14.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.171+0000 7fd6d57fa640 1 -- 172.21.15.169:0/3684286396 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4039040 con 0x5556a2d60130 2024-03-28T00:56:14.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.171+0000 7fd6d57fa640 1 -- 172.21.15.169:0/3684286396 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6e402c510 con 0x5556a2d60130 2024-03-28T00:56:14.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.171+0000 7fd6d57fa640 1 -- 172.21.15.169:0/3684286396 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e40035f0 con 0x5556a2d60130 2024-03-28T00:56:14.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.171+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/3684286396 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d60130 msgr2=0x5556a2d68770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.171+0000 7fd6f8c5b740 1 --2- 172.21.15.169:0/3684286396 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d60130 0x5556a2d68770 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd6e40057c0 tx=0x7fd6e402bb70 comp rx=0 tx=0).stop 2024-03-28T00:56:14.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.171+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/3684286396 shutdown_connections 2024-03-28T00:56:14.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.171+0000 7fd6f8c5b740 1 --2- 172.21.15.169:0/3684286396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556a2d652a0 0x5556a2d68cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.171+0000 7fd6f8c5b740 1 --2- 172.21.15.169:0/3684286396 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d60130 0x5556a2d68770 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.171+0000 7fd6f8c5b740 1 --2- 172.21.15.169:0/3684286396 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d5f740 0x5556a2d5ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.171+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/3684286396 >> 172.21.15.169:0/3684286396 conn(0x5556a2d27a60 msgr2=0x5556a2d5e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:14.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.171+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/3684286396 shutdown_connections 2024-03-28T00:56:14.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.171+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/3684286396 wait complete. 2024-03-28T00:56:14.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.171+0000 7fd6f8c5b740 1 -- start start 2024-03-28T00:56:14.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.172+0000 7fd6f8c5b740 1 Processor -- start 2024-03-28T00:56:14.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.172+0000 7fd6f8c5b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556a2d5f740 0x5556a2dcd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.172+0000 7fd6f8c5b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d60130 0x5556a2dcdb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.172+0000 7fd6f8c5b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d652a0 0x5556a2dd49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.172+0000 7fd6f8c5b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a2d77530 con 0x5556a2d652a0 2024-03-28T00:56:14.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.172+0000 7fd6f8c5b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a2d76ed0 con 0x5556a2d60130 2024-03-28T00:56:14.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.172+0000 7fd6f8c5b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a2d77b90 con 0x5556a2d5f740 2024-03-28T00:56:14.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.172+0000 7fd6f8c5b740 10 client.0 _ll_register_callbacks cb 0x7fd6e00046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:56:14.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.172+0000 7fd6effff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d652a0 0x5556a2dd49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.172+0000 7fd6effff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d652a0 0x5556a2dd49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38384/0 (socket says 172.21.15.169:38384) 2024-03-28T00:56:14.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.172+0000 7fd6effff640 1 -- 172.21.15.169:0/2188544659 learned_addr learned my addr 172.21.15.169:0/2188544659 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:14.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.172+0000 7fd6eeffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d60130 0x5556a2dcdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.172+0000 7fd6ef7fe640 1 --2- 172.21.15.169:0/2188544659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556a2d5f740 0x5556a2dcd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.172+0000 7fd6effff640 1 -- 172.21.15.169:0/2188544659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556a2d5f740 msgr2=0x5556a2dcd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.172+0000 7fd6effff640 1 --2- 172.21.15.169:0/2188544659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556a2d5f740 0x5556a2dcd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.172+0000 7fd6effff640 1 -- 172.21.15.169:0/2188544659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d60130 msgr2=0x5556a2dcdb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.172+0000 7fd6effff640 1 --2- 172.21.15.169:0/2188544659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d60130 0x5556a2dcdb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.172+0000 7fd6effff640 1 -- 172.21.15.169:0/2188544659 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5556a2dd4fb0 con 0x5556a2d652a0 2024-03-28T00:56:14.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.172+0000 7fd6eeffd640 1 --2- 172.21.15.169:0/2188544659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d60130 0x5556a2dcdb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:14.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.172+0000 7fd6effff640 1 --2- 172.21.15.169:0/2188544659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d652a0 0x5556a2dd49a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8003ec0 tx=0x7fd6e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c9d4406df46f6ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:14.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.172+0000 7fd6ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:56:14.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.173+0000 7fd6ceffd640 1 -- 172.21.15.169:0/2188544659 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8004d20 con 0x5556a2d652a0 2024-03-28T00:56:14.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.173+0000 7fd6ceffd640 1 -- 172.21.15.169:0/2188544659 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fd6e8020720 con 0x5556a2d652a0 2024-03-28T00:56:14.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.173+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/2188544659 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6e402c510 con 0x5556a2d652a0 2024-03-28T00:56:14.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.173+0000 7fd6ceffd640 1 -- 172.21.15.169:0/2188544659 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6e8035050 con 0x5556a2d652a0 2024-03-28T00:56:14.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.173+0000 7fd6f8c5b740 10 client.15243 fetch_fsmap learned FSMap version 24 2024-03-28T00:56:14.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.173+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/2188544659 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5556a2dd6350 con 0x5556a2d652a0 2024-03-28T00:56:14.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.173+0000 7fd6ceffd640 1 -- 172.21.15.169:0/2188544659 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6e402c510 con 0x5556a2d652a0 2024-03-28T00:56:14.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.173+0000 7fd6ceffd640 1 -- 172.21.15.169:0/2188544659 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd6e801d040 con 0x5556a2d652a0 2024-03-28T00:56:14.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.173+0000 7fd6f8c5b740 10 client.15243 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T00:56:14.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.173+0000 7fd6f8c5b740 10 client.15243 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T00:56:14.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.173+0000 7fd6f8c5b740 20 client.15243 populate_metadata read hostname 'smithi169' 2024-03-28T00:56:14.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.173+0000 7fd6f8c5b740 1 --2- 172.21.15.169:0/2188544659 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5556a2ddaae0 0x5556a2dd6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.173+0000 7fd6f8c5b740 4 client.15243 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T00:56:14.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.173+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/2188544659 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5556a2d610a0 con 0x5556a2ddaae0 2024-03-28T00:56:14.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.173+0000 7fd6ef7fe640 1 --2- 172.21.15.169:0/2188544659 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5556a2ddaae0 0x5556a2dd6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.174+0000 7fd6ceffd640 10 client.15243 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T00:56:14.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.174+0000 7fd6ef7fe640 1 --2- 172.21.15.169:0/2188544659 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5556a2ddaae0 0x5556a2dd6690 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7dc9e4b2ac66c65 server_cookie=9d77a0e13bea52c0 in_seq=0 out_seq=0 2024-03-28T00:56:14.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.174+0000 7fd6ceffd640 1 -- 172.21.15.169:0/2188544659 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5556a2d610a0 con 0x5556a2ddaae0 2024-03-28T00:56:14.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.174+0000 7fd6ceffd640 10 client.15243 handle_command_reply: tid=0 2024-03-28T00:56:14.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.174+0000 7fd6f8c5b740 1 client.15243 shutdown 2024-03-28T00:56:14.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.174+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/2188544659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d652a0 msgr2=0x5556a2dd49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.174+0000 7fd6f8c5b740 1 --2- 172.21.15.169:0/2188544659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d652a0 0x5556a2dd49a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8003ec0 tx=0x7fd6e8004a40 comp rx=0 tx=0).stop 2024-03-28T00:56:14.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.174+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/2188544659 shutdown_connections 2024-03-28T00:56:14.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.174+0000 7fd6f8c5b740 1 --2- 172.21.15.169:0/2188544659 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5556a2ddaae0 0x5556a2dd6690 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.174+0000 7fd6f8c5b740 1 --2- 172.21.15.169:0/2188544659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556a2d652a0 0x5556a2dd49a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.174+0000 7fd6f8c5b740 1 --2- 172.21.15.169:0/2188544659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556a2d60130 0x5556a2dcdb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.174+0000 7fd6ceffd640 0 client.15243 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T00:56:14.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.174+0000 7fd6f8c5b740 1 --2- 172.21.15.169:0/2188544659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556a2d5f740 0x5556a2dcd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.174+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/2188544659 >> 172.21.15.169:0/2188544659 conn(0x5556a2d27a60 msgr2=0x5556a2d5eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:14.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.174+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/2188544659 shutdown_connections 2024-03-28T00:56:14.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.174+0000 7fd6f8c5b740 1 -- 172.21.15.169:0/2188544659 wait complete. 2024-03-28T00:56:14.196 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.174+0000 7fd6f8c5b740 20 client.15243 trim_cache size 0 max 16384 2024-03-28T00:56:14.196 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:56:14.196 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_fsid": "833368da-ec9c-11ee-95d0-87774f69a715", 2024-03-28T00:56:14.196 INFO:teuthology.orchestra.run.smithi169.stdout: "whoami": 1, 2024-03-28T00:56:14.197 INFO:teuthology.orchestra.run.smithi169.stdout: "id": 24478, 2024-03-28T00:56:14.197 INFO:teuthology.orchestra.run.smithi169.stdout: "want_state": "up:active", 2024-03-28T00:56:14.197 INFO:teuthology.orchestra.run.smithi169.stdout: "state": "up:active", 2024-03-28T00:56:14.197 INFO:teuthology.orchestra.run.smithi169.stdout: "fs_name": "cephfs", 2024-03-28T00:56:14.197 INFO:teuthology.orchestra.run.smithi169.stdout: "rank_uptime": 83.911973360000005, 2024-03-28T00:56:14.197 INFO:teuthology.orchestra.run.smithi169.stdout: "mdsmap_epoch": 24, 2024-03-28T00:56:14.197 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_epoch": 92, 2024-03-28T00:56:14.198 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_epoch_barrier": 90, 2024-03-28T00:56:14.198 INFO:teuthology.orchestra.run.smithi169.stdout: "uptime": 113.788701101 2024-03-28T00:56:14.198 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:56:14.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.175+0000 7fd6f6075640 1 -- 172.21.15.169:0/4098611352 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd6d80a2800 msgr2=0x7fd6d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.175+0000 7fd6f6075640 1 --2- 172.21.15.169:0/4098611352 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd6d80a2800 0x7fd6d80a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x5556a2d60990 tx=0x7fd6e8009500 comp rx=0 tx=0).stop 2024-03-28T00:56:14.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.175+0000 7fd6f6075640 1 -- 172.21.15.169:0/4098611352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6f00ff9a0 msgr2=0x7fd6f0197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.175+0000 7fd6f6075640 1 --2- 172.21.15.169:0/4098611352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6f00ff9a0 0x7fd6f0197a10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd6e000e9a0 tx=0x7fd6e000eda0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.175+0000 7fd6f6075640 1 -- 172.21.15.169:0/4098611352 shutdown_connections 2024-03-28T00:56:14.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.175+0000 7fd6f6075640 1 --2- 172.21.15.169:0/4098611352 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd6d80a2800 0x7fd6d80a4cc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.175+0000 7fd6f6075640 1 --2- 172.21.15.169:0/4098611352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6f0103330 0x7fd6f019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.175+0000 7fd6f6075640 1 --2- 172.21.15.169:0/4098611352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6f0102970 0x7fd6f0197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.175+0000 7fd6f6075640 1 --2- 172.21.15.169:0/4098611352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd6f00ff9a0 0x7fd6f0197a10 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.175+0000 7fd6f6075640 1 -- 172.21.15.169:0/4098611352 >> 172.21.15.169:0/4098611352 conn(0x7fd6f00fb2d0 msgr2=0x7fd6f0105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:14.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.175+0000 7fd6f6075640 1 -- 172.21.15.169:0/4098611352 shutdown_connections 2024-03-28T00:56:14.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.175+0000 7fd6f6075640 1 -- 172.21.15.169:0/4098611352 wait complete. 2024-03-28T00:56:14.201 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T00:56:14.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.303+0000 7f7ee13d4640 1 -- 172.21.15.169:0/3374923805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc101ba0 msgr2=0x7f7edc101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.303+0000 7f7ee13d4640 1 --2- 172.21.15.169:0/3374923805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc101ba0 0x7f7edc101fc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc009900 tx=0x7f7ecc02bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:14.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.303+0000 7f7ee13d4640 1 -- 172.21.15.169:0/3374923805 shutdown_connections 2024-03-28T00:56:14.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.303+0000 7f7ee13d4640 1 --2- 172.21.15.169:0/3374923805 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7edc101200 0x7f7edc108aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.303+0000 7f7ee13d4640 1 --2- 172.21.15.169:0/3374923805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7edc100840 0x7f7edc100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.303+0000 7f7ee13d4640 1 --2- 172.21.15.169:0/3374923805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc101ba0 0x7f7edc101fc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.303+0000 7f7ee13d4640 1 -- 172.21.15.169:0/3374923805 >> 172.21.15.169:0/3374923805 conn(0x7f7edc0fb2b0 msgr2=0x7f7edc0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:14.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.303+0000 7f7ee13d4640 1 -- 172.21.15.169:0/3374923805 shutdown_connections 2024-03-28T00:56:14.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.303+0000 7f7ee13d4640 1 -- 172.21.15.169:0/3374923805 wait complete. 2024-03-28T00:56:14.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.303+0000 7f7ee13d4640 1 Processor -- start 2024-03-28T00:56:14.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.304+0000 7f7ee13d4640 1 -- start start 2024-03-28T00:56:14.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.304+0000 7f7ee13d4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7edc100840 0x7f7edc1968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.304+0000 7f7ee13d4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7edc101200 0x7f7edc196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.304+0000 7f7ee13d4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc101ba0 0x7f7edc19de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.304+0000 7f7ee13d4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc10c250 con 0x7f7edc101200 2024-03-28T00:56:14.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.304+0000 7f7ee13d4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc10c0d0 con 0x7f7edc101ba0 2024-03-28T00:56:14.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.304+0000 7f7ee13d4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc10c3d0 con 0x7f7edc100840 2024-03-28T00:56:14.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.304+0000 7f7eda7fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7edc101200 0x7f7edc196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.304+0000 7f7edaffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7edc100840 0x7f7edc1968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.304+0000 7f7eda7fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7edc101200 0x7f7edc196e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38404/0 (socket says 172.21.15.169:38404) 2024-03-28T00:56:14.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.304+0000 7f7edb7fe640 1 --2- 172.21.15.169:0/1704879806 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc101ba0 0x7f7edc19de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.304+0000 7f7eda7fc640 1 -- 172.21.15.169:0/1704879806 learned_addr learned my addr 172.21.15.169:0/1704879806 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:14.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.304+0000 7f7edaffd640 1 -- 172.21.15.169:0/1704879806 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc101ba0 msgr2=0x7f7edc19de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.304+0000 7f7edaffd640 1 --2- 172.21.15.169:0/1704879806 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc101ba0 0x7f7edc19de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.304+0000 7f7edaffd640 1 -- 172.21.15.169:0/1704879806 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7edc101200 msgr2=0x7f7edc196e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.304+0000 7f7edaffd640 1 --2- 172.21.15.169:0/1704879806 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7edc101200 0x7f7edc196e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.304+0000 7f7edaffd640 1 -- 172.21.15.169:0/1704879806 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7edc19e4e0 con 0x7f7edc100840 2024-03-28T00:56:14.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.304+0000 7f7eda7fc640 1 --2- 172.21.15.169:0/1704879806 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7edc101200 0x7f7edc196e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:14.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.304+0000 7f7edb7fe640 1 --2- 172.21.15.169:0/1704879806 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc101ba0 0x7f7edc19de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:14.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.305+0000 7f7edaffd640 1 --2- 172.21.15.169:0/1704879806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7edc100840 0x7f7edc1968c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc02c170 tx=0x7f7ecc02c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:14.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.305+0000 7f7ec3fff640 1 -- 172.21.15.169:0/1704879806 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc043020 con 0x7f7edc100840 2024-03-28T00:56:14.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.305+0000 7f7ee13d4640 1 -- 172.21.15.169:0/1704879806 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7edc19e770 con 0x7f7edc100840 2024-03-28T00:56:14.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.305+0000 7f7ec3fff640 1 -- 172.21.15.169:0/1704879806 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ecc0054e0 con 0x7f7edc100840 2024-03-28T00:56:14.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.305+0000 7f7ee13d4640 1 -- 172.21.15.169:0/1704879806 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7edc19ec90 con 0x7f7edc100840 2024-03-28T00:56:14.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.305+0000 7f7ec3fff640 1 -- 172.21.15.169:0/1704879806 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc003c40 con 0x7f7edc100840 2024-03-28T00:56:14.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.307+0000 7f7ec3fff640 1 -- 172.21.15.169:0/1704879806 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7ecc046020 con 0x7f7edc100840 2024-03-28T00:56:14.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.307+0000 7f7ee3fc4740 1 Processor -- start 2024-03-28T00:56:14.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.307+0000 7f7ee3fc4740 1 -- start start 2024-03-28T00:56:14.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.307+0000 7f7ee3fc4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf19d740 0x5607cf19db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.307+0000 7f7ee3fc4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19e130 0x5607cf1a2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.307+0000 7f7ee3fc4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf1a32a0 0x5607cf1a5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.307+0000 7f7ee3fc4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607cf102c20 con 0x5607cf19d740 2024-03-28T00:56:14.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.307+0000 7f7ee3fc4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607cf0aba40 con 0x5607cf19e130 2024-03-28T00:56:14.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.307+0000 7f7ee3fc4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607cf0fecc0 con 0x5607cf1a32a0 2024-03-28T00:56:14.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.307+0000 7f7ec3fff640 1 --2- 172.21.15.169:0/1704879806 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7ec40a2800 0x7f7ec40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.307+0000 7f7edb7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf19d740 0x5607cf19db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.307+0000 7f7eda7fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf1a32a0 0x5607cf1a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.307+0000 7f7edb7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf19d740 0x5607cf19db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38420/0 (socket says 172.21.15.169:38420) 2024-03-28T00:56:14.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.307+0000 7f7eda7fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf1a32a0 0x5607cf1a5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:44546/0 (socket says 172.21.15.169:44546) 2024-03-28T00:56:14.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.307+0000 7f7edb7fe640 1 -- 172.21.15.169:0/1240342068 learned_addr learned my addr 172.21.15.169:0/1240342068 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:14.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.307+0000 7f7edaffd640 1 --2- 172.21.15.169:0/1240342068 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19e130 0x5607cf1a2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.308+0000 7f7edb7fe640 1 --2- 172.21.15.169:0/1704879806 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7ec40a2800 0x7f7ec40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.308+0000 7f7ec3fff640 1 -- 172.21.15.169:0/1704879806 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f7ecc0f0dc0 con 0x7f7edc100840 2024-03-28T00:56:14.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.308+0000 7f7edb7fe640 1 -- 172.21.15.169:0/1240342068 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf1a32a0 msgr2=0x5607cf1a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.308+0000 7f7edb7fe640 1 --2- 172.21.15.169:0/1240342068 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf1a32a0 0x5607cf1a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.308+0000 7f7edb7fe640 1 -- 172.21.15.169:0/1240342068 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19e130 msgr2=0x5607cf1a2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.308+0000 7f7edb7fe640 1 --2- 172.21.15.169:0/1240342068 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19e130 0x5607cf1a2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.308+0000 7f7edb7fe640 1 -- 172.21.15.169:0/1240342068 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607cf0cdf90 con 0x5607cf19d740 2024-03-28T00:56:14.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.308+0000 7f7edb7fe640 1 --2- 172.21.15.169:0/1240342068 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf19d740 0x5607cf19db60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7ed400f7e0 tx=0x7f7ed400fca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5440d289067a539f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:14.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.308+0000 7f7edb7fe640 1 --2- 172.21.15.169:0/1704879806 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7ec40a2800 0x7f7ec40a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7ed40099e0 tx=0x7f7ed4020000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:14.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.308+0000 7f7ec17fa640 1 -- 172.21.15.169:0/1240342068 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed402d070 con 0x5607cf19d740 2024-03-28T00:56:14.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.308+0000 7f7ec17fa640 1 -- 172.21.15.169:0/1240342068 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ed4003e50 con 0x5607cf19d740 2024-03-28T00:56:14.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.308+0000 7f7ec17fa640 1 -- 172.21.15.169:0/1240342068 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed4024070 con 0x5607cf19d740 2024-03-28T00:56:14.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.309+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/1240342068 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf19d740 msgr2=0x5607cf19db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.309+0000 7f7ee3fc4740 1 --2- 172.21.15.169:0/1240342068 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf19d740 0x5607cf19db60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7ed400f7e0 tx=0x7f7ed400fca0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.309+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/1240342068 shutdown_connections 2024-03-28T00:56:14.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.309+0000 7f7ee3fc4740 1 --2- 172.21.15.169:0/1240342068 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf1a32a0 0x5607cf1a5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.309+0000 7f7ee3fc4740 1 --2- 172.21.15.169:0/1240342068 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19e130 0x5607cf1a2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.309+0000 7f7ee3fc4740 1 --2- 172.21.15.169:0/1240342068 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf19d740 0x5607cf19db60 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.309+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/1240342068 >> 172.21.15.169:0/1240342068 conn(0x5607cf165a60 msgr2=0x5607cf1990b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:14.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.309+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/1240342068 shutdown_connections 2024-03-28T00:56:14.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.309+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/1240342068 wait complete. 2024-03-28T00:56:14.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.309+0000 7f7ee3fc4740 1 -- start start 2024-03-28T00:56:14.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.309+0000 7f7ee3fc4740 1 Processor -- start 2024-03-28T00:56:14.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.309+0000 7f7ee3fc4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19d740 0x5607cf1a7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.309+0000 7f7ee3fc4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf19e130 0x5607cf1a8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.309+0000 7f7ee3fc4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf1a32a0 0x5607cf1af150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.309+0000 7f7ee3fc4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607cf1a61f0 con 0x5607cf1a32a0 2024-03-28T00:56:14.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.309+0000 7f7ee3fc4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607cf0bee90 con 0x5607cf19d740 2024-03-28T00:56:14.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.309+0000 7f7ee3fc4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607cf1a6370 con 0x5607cf19e130 2024-03-28T00:56:14.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.309+0000 7f7ee3fc4740 10 client.0 _ll_register_callbacks cb 0x5607cee16ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:56:14.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7edaffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19d740 0x5607cf1a7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7edb7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf1a32a0 0x5607cf1af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7eda7fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf19e130 0x5607cf1a8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7edb7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf1a32a0 0x5607cf1af150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38422/0 (socket says 172.21.15.169:38422) 2024-03-28T00:56:14.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7edaffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19d740 0x5607cf1a7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:54834/0 (socket says 172.21.15.169:54834) 2024-03-28T00:56:14.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7edb7fe640 1 -- 172.21.15.169:0/2887762721 learned_addr learned my addr 172.21.15.169:0/2887762721 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:14.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7eda7fc640 1 -- 172.21.15.169:0/2887762721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19d740 msgr2=0x5607cf1a7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7eda7fc640 1 --2- 172.21.15.169:0/2887762721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19d740 0x5607cf1a7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7eda7fc640 1 -- 172.21.15.169:0/2887762721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf1a32a0 msgr2=0x5607cf1af150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7eda7fc640 1 --2- 172.21.15.169:0/2887762721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf1a32a0 0x5607cf1af150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7eda7fc640 1 -- 172.21.15.169:0/2887762721 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5607cf0cb9d0 con 0x5607cf19e130 2024-03-28T00:56:14.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7edaffd640 1 --2- 172.21.15.169:0/2887762721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19d740 0x5607cf1a7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:14.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7edb7fe640 1 --2- 172.21.15.169:0/2887762721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf1a32a0 0x5607cf1af150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:14.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7eda7fc640 1 --2- 172.21.15.169:0/2887762721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf19e130 0x5607cf1a8110 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0009640 tx=0x7f7ed000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c34bc5f83a5ad77 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:14.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7ebeff5640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:56:14.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7ebeff5640 1 -- 172.21.15.169:0/2887762721 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed00120d0 con 0x5607cf19e130 2024-03-28T00:56:14.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/2887762721 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ed4003e50 con 0x5607cf19e130 2024-03-28T00:56:14.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7ebeff5640 1 -- 172.21.15.169:0/2887762721 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f7ed0005020 con 0x5607cf19e130 2024-03-28T00:56:14.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7ebeff5640 1 -- 172.21.15.169:0/2887762721 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ed0025050 con 0x5607cf19e130 2024-03-28T00:56:14.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7ee3fc4740 10 client.25048 fetch_fsmap learned FSMap version 24 2024-03-28T00:56:14.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/2887762721 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5607cee172d0 con 0x5607cf19e130 2024-03-28T00:56:14.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.310+0000 7f7ebeff5640 1 -- 172.21.15.169:0/2887762721 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ed4003e50 con 0x5607cf19e130 2024-03-28T00:56:14.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.311+0000 7f7ebeff5640 1 -- 172.21.15.169:0/2887762721 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f7ed001c470 con 0x5607cf19e130 2024-03-28T00:56:14.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.311+0000 7f7ee3fc4740 10 client.25048 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T00:56:14.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.311+0000 7f7ee3fc4740 10 client.25048 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T00:56:14.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.311+0000 7f7ee3fc4740 20 client.25048 populate_metadata read hostname 'smithi169' 2024-03-28T00:56:14.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.311+0000 7f7ee3fc4740 1 --2- 172.21.15.169:0/2887762721 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5607cf1b44b0 0x5607cf1b3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.311+0000 7f7ee3fc4740 4 client.25048 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:56:14.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.311+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/2887762721 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5607cf110160 con 0x5607cf1b44b0 2024-03-28T00:56:14.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.311+0000 7f7edaffd640 1 --2- 172.21.15.169:0/2887762721 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5607cf1b44b0 0x5607cf1b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.311+0000 7f7edaffd640 1 --2- 172.21.15.169:0/2887762721 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5607cf1b44b0 0x5607cf1b3540 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e9a1f7af6a9ce11 server_cookie=e4e5a899afad05cc in_seq=0 out_seq=0 2024-03-28T00:56:14.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.311+0000 7f7ebeff5640 10 client.25048 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:56:14.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.312+0000 7f7ebeff5640 1 -- 172.21.15.169:0/2887762721 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5607cf110160 con 0x5607cf1b44b0 2024-03-28T00:56:14.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.312+0000 7f7ebeff5640 10 client.25048 handle_command_reply: tid=0 2024-03-28T00:56:14.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.312+0000 7f7ee3fc4740 1 client.25048 shutdown 2024-03-28T00:56:14.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.312+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/2887762721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf19e130 msgr2=0x5607cf1a8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.312+0000 7f7ee3fc4740 1 --2- 172.21.15.169:0/2887762721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf19e130 0x5607cf1a8110 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0009640 tx=0x7f7ed000f790 comp rx=0 tx=0).stop 2024-03-28T00:56:14.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.312+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/2887762721 shutdown_connections 2024-03-28T00:56:14.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.312+0000 7f7ee3fc4740 1 --2- 172.21.15.169:0/2887762721 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5607cf1b44b0 0x5607cf1b3540 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.312+0000 7f7ee3fc4740 1 --2- 172.21.15.169:0/2887762721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf1a32a0 0x5607cf1af150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.312+0000 7f7ee3fc4740 1 --2- 172.21.15.169:0/2887762721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf19e130 0x5607cf1a8110 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.312+0000 7f7ebeff5640 0 client.25048 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:56:14.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.312+0000 7f7ee3fc4740 1 --2- 172.21.15.169:0/2887762721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19d740 0x5607cf1a7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.312+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/2887762721 >> 172.21.15.169:0/2887762721 conn(0x5607cf165a60 msgr2=0x5607cf1982f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:14.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.312+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/2887762721 shutdown_connections 2024-03-28T00:56:14.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.312+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/2887762721 wait complete. 2024-03-28T00:56:14.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.312+0000 7f7ee3fc4740 20 client.25048 trim_cache size 0 max 16384 2024-03-28T00:56:14.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.329+0000 7f7ee3fc4740 1 Processor -- start 2024-03-28T00:56:14.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.329+0000 7f7ee3fc4740 1 -- start start 2024-03-28T00:56:14.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.329+0000 7f7ee3fc4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf19d740 0x5607cf19c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.329+0000 7f7ee3fc4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19e130 0x5607cf19c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.329+0000 7f7ee3fc4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf1a32a0 0x5607cf19cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.329+0000 7f7ee3fc4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607cf1b3cd0 con 0x5607cf19d740 2024-03-28T00:56:14.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.329+0000 7f7ee3fc4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607cf1b4330 con 0x5607cf19e130 2024-03-28T00:56:14.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.329+0000 7f7ee3fc4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607cf1b4990 con 0x5607cf1a32a0 2024-03-28T00:56:14.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.329+0000 7f7edaffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf19d740 0x5607cf19c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.329+0000 7f7edaffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf19d740 0x5607cf19c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38424/0 (socket says 172.21.15.169:38424) 2024-03-28T00:56:14.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.329+0000 7f7edaffd640 1 -- 172.21.15.169:0/1959261649 learned_addr learned my addr 172.21.15.169:0/1959261649 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:14.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.329+0000 7f7edb7fe640 1 --2- 172.21.15.169:0/1959261649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf1a32a0 0x5607cf19cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.329+0000 7f7eda7fc640 1 --2- 172.21.15.169:0/1959261649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19e130 0x5607cf19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.330+0000 7f7edaffd640 1 -- 172.21.15.169:0/1959261649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf1a32a0 msgr2=0x5607cf19cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.330+0000 7f7edaffd640 1 --2- 172.21.15.169:0/1959261649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf1a32a0 0x5607cf19cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.330+0000 7f7edaffd640 1 -- 172.21.15.169:0/1959261649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19e130 msgr2=0x5607cf19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.330+0000 7f7edaffd640 1 --2- 172.21.15.169:0/1959261649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19e130 0x5607cf19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.330+0000 7f7edaffd640 1 -- 172.21.15.169:0/1959261649 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607ceee16b0 con 0x5607cf19d740 2024-03-28T00:56:14.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.330+0000 7f7edaffd640 1 --2- 172.21.15.169:0/1959261649 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf19d740 0x5607cf19c2f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc0bd390 tx=0x7f7ecc005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86cce92052891910 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:14.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.330+0000 7f7ec0ff9640 1 -- 172.21.15.169:0/1959261649 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc0bdb00 con 0x5607cf19d740 2024-03-28T00:56:14.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.330+0000 7f7ec0ff9640 1 -- 172.21.15.169:0/1959261649 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ecc0bd400 con 0x5607cf19d740 2024-03-28T00:56:14.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.330+0000 7f7ec0ff9640 1 -- 172.21.15.169:0/1959261649 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc0bb4a0 con 0x5607cf19d740 2024-03-28T00:56:14.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.330+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/1959261649 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf19d740 msgr2=0x5607cf19c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.330+0000 7f7ee3fc4740 1 --2- 172.21.15.169:0/1959261649 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf19d740 0x5607cf19c2f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc0bd390 tx=0x7f7ecc005e50 comp rx=0 tx=0).stop 2024-03-28T00:56:14.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.330+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/1959261649 shutdown_connections 2024-03-28T00:56:14.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.330+0000 7f7ee3fc4740 1 --2- 172.21.15.169:0/1959261649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf1a32a0 0x5607cf19cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.330+0000 7f7ee3fc4740 1 --2- 172.21.15.169:0/1959261649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19e130 0x5607cf19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.330+0000 7f7ee3fc4740 1 --2- 172.21.15.169:0/1959261649 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf19d740 0x5607cf19c2f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.330+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/1959261649 >> 172.21.15.169:0/1959261649 conn(0x5607cf165a60 msgr2=0x5607cf19f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:14.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.330+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/1959261649 shutdown_connections 2024-03-28T00:56:14.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.330+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/1959261649 wait complete. 2024-03-28T00:56:14.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.331+0000 7f7ee3fc4740 1 -- start start 2024-03-28T00:56:14.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.331+0000 7f7ee3fc4740 1 Processor -- start 2024-03-28T00:56:14.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.331+0000 7f7ee3fc4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf19d740 0x5607cf20b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.331+0000 7f7ee3fc4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19e130 0x5607cf20bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.331+0000 7f7ee3fc4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf1a32a0 0x5607cf212b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.331+0000 7f7ee3fc4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607cf1b5650 con 0x5607cf1a32a0 2024-03-28T00:56:14.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.331+0000 7f7ee3fc4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607cf1b4ff0 con 0x5607cf19e130 2024-03-28T00:56:14.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.331+0000 7f7ee3fc4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607cf1b5cb0 con 0x5607cf19d740 2024-03-28T00:56:14.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.331+0000 7f7ee3fc4740 10 client.0 _ll_register_callbacks cb 0x7f7ed0012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:56:14.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.331+0000 7f7edb7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf1a32a0 0x5607cf212b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.331+0000 7f7edb7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf1a32a0 0x5607cf212b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38440/0 (socket says 172.21.15.169:38440) 2024-03-28T00:56:14.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.331+0000 7f7edb7fe640 1 -- 172.21.15.169:0/2770685842 learned_addr learned my addr 172.21.15.169:0/2770685842 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:14.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.331+0000 7f7eda7fc640 1 --2- 172.21.15.169:0/2770685842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19e130 0x5607cf20bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.331+0000 7f7edaffd640 1 --2- 172.21.15.169:0/2770685842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf19d740 0x5607cf20b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.331+0000 7f7edb7fe640 1 -- 172.21.15.169:0/2770685842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf19d740 msgr2=0x5607cf20b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.331+0000 7f7edb7fe640 1 --2- 172.21.15.169:0/2770685842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf19d740 0x5607cf20b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.331+0000 7f7edb7fe640 1 -- 172.21.15.169:0/2770685842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19e130 msgr2=0x5607cf20bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:14.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.331+0000 7f7edb7fe640 1 --2- 172.21.15.169:0/2770685842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19e130 0x5607cf20bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.331+0000 7f7edb7fe640 1 -- 172.21.15.169:0/2770685842 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5607cf213120 con 0x5607cf1a32a0 2024-03-28T00:56:14.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.331+0000 7f7edb7fe640 1 --2- 172.21.15.169:0/2770685842 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf1a32a0 0x5607cf212b30 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4009540 tx=0x7f7ed40134e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6e475b106470514 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:14.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.331+0000 7f7ebeff5640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:56:14.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.332+0000 7f7ebeff5640 1 -- 172.21.15.169:0/2770685842 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed4036060 con 0x5607cf1a32a0 2024-03-28T00:56:14.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.332+0000 7f7ebeff5640 1 -- 172.21.15.169:0/2770685842 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f7ed4013840 con 0x5607cf1a32a0 2024-03-28T00:56:14.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.332+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/2770685842 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ecc0bd400 con 0x5607cf1a32a0 2024-03-28T00:56:14.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.332+0000 7f7ebeff5640 1 -- 172.21.15.169:0/2770685842 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ed403d050 con 0x5607cf1a32a0 2024-03-28T00:56:14.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.332+0000 7f7ee3fc4740 10 client.15261 fetch_fsmap learned FSMap version 24 2024-03-28T00:56:14.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.332+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/2770685842 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5607cf214490 con 0x5607cf1a32a0 2024-03-28T00:56:14.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.332+0000 7f7ebeff5640 1 -- 172.21.15.169:0/2770685842 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ecc0bd400 con 0x5607cf1a32a0 2024-03-28T00:56:14.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.332+0000 7f7ebeff5640 1 -- 172.21.15.169:0/2770685842 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f7ed40051d0 con 0x5607cf1a32a0 2024-03-28T00:56:14.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.332+0000 7f7ee3fc4740 10 client.15261 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T00:56:14.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.332+0000 7f7ee3fc4740 10 client.15261 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T00:56:14.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.332+0000 7f7ee3fc4740 20 client.15261 populate_metadata read hostname 'smithi169' 2024-03-28T00:56:14.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.332+0000 7f7ee3fc4740 1 --2- 172.21.15.169:0/2770685842 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5607cf218a20 0x5607cf214770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:14.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.332+0000 7f7ee3fc4740 4 client.15261 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T00:56:14.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.332+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/2770685842 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5607cf10a4b0 con 0x5607cf218a20 2024-03-28T00:56:14.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.332+0000 7f7edaffd640 1 --2- 172.21.15.169:0/2770685842 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5607cf218a20 0x5607cf214770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:14.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.333+0000 7f7edaffd640 1 --2- 172.21.15.169:0/2770685842 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5607cf218a20 0x5607cf214770 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f769be0aa111ab server_cookie=2a9476ae47a60745 in_seq=0 out_seq=0 2024-03-28T00:56:14.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.333+0000 7f7ebeff5640 10 client.15261 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:56:14.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.333+0000 7f7ebeff5640 1 -- 172.21.15.169:0/2770685842 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5607cf10a4b0 con 0x5607cf218a20 2024-03-28T00:56:14.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.333+0000 7f7ebeff5640 10 client.15261 handle_command_reply: tid=0 2024-03-28T00:56:14.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.333+0000 7f7ee3fc4740 1 client.15261 shutdown 2024-03-28T00:56:14.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.333+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/2770685842 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf1a32a0 msgr2=0x5607cf212b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.333+0000 7f7ee3fc4740 1 --2- 172.21.15.169:0/2770685842 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf1a32a0 0x5607cf212b30 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4009540 tx=0x7f7ed40134e0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.333+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/2770685842 shutdown_connections 2024-03-28T00:56:14.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.333+0000 7f7ee3fc4740 1 --2- 172.21.15.169:0/2770685842 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5607cf218a20 0x5607cf214770 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.333+0000 7f7ee3fc4740 1 --2- 172.21.15.169:0/2770685842 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5607cf1a32a0 0x5607cf212b30 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.333+0000 7f7ee3fc4740 1 --2- 172.21.15.169:0/2770685842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607cf19e130 0x5607cf20bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.333+0000 7f7ebeff5640 0 client.15261 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:56:14.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.333+0000 7f7ee3fc4740 1 --2- 172.21.15.169:0/2770685842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5607cf19d740 0x5607cf20b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.333+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/2770685842 >> 172.21.15.169:0/2770685842 conn(0x5607cf165a60 msgr2=0x5607cf197a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:14.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.333+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/2770685842 shutdown_connections 2024-03-28T00:56:14.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.333+0000 7f7ee3fc4740 1 -- 172.21.15.169:0/2770685842 wait complete. 2024-03-28T00:56:14.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.333+0000 7f7ee3fc4740 20 client.15261 trim_cache size 0 max 16384 2024-03-28T00:56:14.344 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:56:14.344 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_fsid": "833368da-ec9c-11ee-95d0-87774f69a715", 2024-03-28T00:56:14.344 INFO:teuthology.orchestra.run.smithi169.stdout: "whoami": 0, 2024-03-28T00:56:14.344 INFO:teuthology.orchestra.run.smithi169.stdout: "id": 24479, 2024-03-28T00:56:14.344 INFO:teuthology.orchestra.run.smithi169.stdout: "want_state": "up:active", 2024-03-28T00:56:14.344 INFO:teuthology.orchestra.run.smithi169.stdout: "state": "up:active", 2024-03-28T00:56:14.345 INFO:teuthology.orchestra.run.smithi169.stdout: "fs_name": "cephfs", 2024-03-28T00:56:14.345 INFO:teuthology.orchestra.run.smithi169.stdout: "rank_uptime": 101.095613403, 2024-03-28T00:56:14.345 INFO:teuthology.orchestra.run.smithi169.stdout: "mdsmap_epoch": 24, 2024-03-28T00:56:14.345 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_epoch": 92, 2024-03-28T00:56:14.345 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_epoch_barrier": 90, 2024-03-28T00:56:14.345 INFO:teuthology.orchestra.run.smithi169.stdout: "uptime": 105.55553378099999 2024-03-28T00:56:14.345 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:56:14.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.334+0000 7f7ee13d4640 1 -- 172.21.15.169:0/1704879806 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7ec40a2800 msgr2=0x7f7ec40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.334+0000 7f7ee13d4640 1 --2- 172.21.15.169:0/1704879806 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7ec40a2800 0x7f7ec40a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7ed40099e0 tx=0x7f7ed4020000 comp rx=0 tx=0).stop 2024-03-28T00:56:14.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.334+0000 7f7ee13d4640 1 -- 172.21.15.169:0/1704879806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7edc100840 msgr2=0x7f7edc1968c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:14.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.334+0000 7f7ee13d4640 1 --2- 172.21.15.169:0/1704879806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7edc100840 0x7f7edc1968c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc02c170 tx=0x7f7ecc02c640 comp rx=0 tx=0).stop 2024-03-28T00:56:14.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.334+0000 7f7ee13d4640 1 -- 172.21.15.169:0/1704879806 shutdown_connections 2024-03-28T00:56:14.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.334+0000 7f7ee13d4640 1 --2- 172.21.15.169:0/1704879806 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7ec40a2800 0x7f7ec40a4cc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.334+0000 7f7ee13d4640 1 --2- 172.21.15.169:0/1704879806 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc101ba0 0x7f7edc19de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.334+0000 7f7ee13d4640 1 --2- 172.21.15.169:0/1704879806 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7edc101200 0x7f7edc196e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.334+0000 7f7ee13d4640 1 --2- 172.21.15.169:0/1704879806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7edc100840 0x7f7edc1968c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:14.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.334+0000 7f7ee13d4640 1 -- 172.21.15.169:0/1704879806 >> 172.21.15.169:0/1704879806 conn(0x7f7edc0fb2b0 msgr2=0x7f7edc104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:14.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.334+0000 7f7ee13d4640 1 -- 172.21.15.169:0/1704879806 shutdown_connections 2024-03-28T00:56:14.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:14.334+0000 7f7ee13d4640 1 -- 172.21.15.169:0/1704879806 wait complete. 2024-03-28T00:56:14.348 INFO:tasks.cephadm:Setup complete, yielding 2024-03-28T00:56:14.348 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T00:56:14.530 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi169.front.sepia.ceph.com 2024-03-28T00:56:14.530 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- bash -c 'ceph orch status' 2024-03-28T00:56:14.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:14 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1524480212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:56:14.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:14 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2321287731' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:14.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:14 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/420421297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:56:14.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:14 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1524480212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:56:14.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:14 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2321287731' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:14.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:14 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/420421297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:56:14.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:14 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1524480212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:56:14.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:14 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2321287731' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:14.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:14 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/420421297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:56:14.772 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:56:15.509 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:15 smithi174 ceph-mon[36060]: pgmap v341: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:56:15.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:15 smithi169 ceph-mon[32274]: pgmap v341: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:56:15.567 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:15 smithi177 ceph-mon[36206]: pgmap v341: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:56:16.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.616+0000 7f37dad26700 1 -- 172.21.15.169:0/2819182909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37d4101d20 msgr2=0x7f37d41095c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:16.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.616+0000 7f37dad26700 1 --2- 172.21.15.169:0/2819182909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37d4101d20 0x7f37d41095c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f37cc009f90 tx=0x7f37cc00b330 comp rx=0 tx=0).stop 2024-03-28T00:56:16.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.616+0000 7f37dad26700 1 -- 172.21.15.169:0/2819182909 shutdown_connections 2024-03-28T00:56:16.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.616+0000 7f37dad26700 1 --2- 172.21.15.169:0/2819182909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37d4101d20 0x7f37d41095c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:16.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.616+0000 7f37dad26700 1 --2- 172.21.15.169:0/2819182909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37d4102790 0x7f37d4102bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:16.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.616+0000 7f37dad26700 1 --2- 172.21.15.169:0/2819182909 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37d4101360 0x7f37d41017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:16.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.616+0000 7f37dad26700 1 -- 172.21.15.169:0/2819182909 >> 172.21.15.169:0/2819182909 conn(0x7f37d40fbe30 msgr2=0x7f37d40fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:16.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.616+0000 7f37dad26700 1 -- 172.21.15.169:0/2819182909 shutdown_connections 2024-03-28T00:56:16.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.616+0000 7f37dad26700 1 -- 172.21.15.169:0/2819182909 wait complete. 2024-03-28T00:56:16.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.616+0000 7f37dad26700 1 Processor -- start 2024-03-28T00:56:16.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.616+0000 7f37dad26700 1 -- start start 2024-03-28T00:56:16.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.617+0000 7f37dad26700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37d4101360 0x7f37d4197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:16.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.617+0000 7f37dad26700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37d4101d20 0x7f37d4197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:16.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.617+0000 7f37dad26700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37d4102790 0x7f37d419e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:16.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.617+0000 7f37dad26700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d410cd80 con 0x7f37d4101360 2024-03-28T00:56:16.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.617+0000 7f37dad26700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d410cc00 con 0x7f37d4102790 2024-03-28T00:56:16.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.617+0000 7f37dad26700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d410cf00 con 0x7f37d4101d20 2024-03-28T00:56:16.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.617+0000 7f37d3fff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37d4101d20 0x7f37d4197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:16.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.617+0000 7f37d3fff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37d4101d20 0x7f37d4197780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:37038/0 (socket says 172.21.15.169:37038) 2024-03-28T00:56:16.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.617+0000 7f37d3fff700 1 -- 172.21.15.169:0/2568442621 learned_addr learned my addr 172.21.15.169:0/2568442621 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:16.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.617+0000 7f37d92c3700 1 --2- 172.21.15.169:0/2568442621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37d4102790 0x7f37d419e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:16.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.617+0000 7f37d8ac2700 1 --2- 172.21.15.169:0/2568442621 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37d4101360 0x7f37d4197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:16.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.617+0000 7f37d3fff700 1 -- 172.21.15.169:0/2568442621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37d4102790 msgr2=0x7f37d419e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:16.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.617+0000 7f37d3fff700 1 --2- 172.21.15.169:0/2568442621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37d4102790 0x7f37d419e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:16.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.617+0000 7f37d3fff700 1 -- 172.21.15.169:0/2568442621 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37d4101360 msgr2=0x7f37d4197240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:16.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.617+0000 7f37d3fff700 1 --2- 172.21.15.169:0/2568442621 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37d4101360 0x7f37d4197240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:16.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.617+0000 7f37d3fff700 1 -- 172.21.15.169:0/2568442621 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d419eea0 con 0x7f37d4101d20 2024-03-28T00:56:16.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.617+0000 7f37d92c3700 1 --2- 172.21.15.169:0/2568442621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37d4102790 0x7f37d419e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:16.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.618+0000 7f37d3fff700 1 --2- 172.21.15.169:0/2568442621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37d4101d20 0x7f37d4197780 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f37c000ec40 tx=0x7f37c000ef50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:16.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.618+0000 7f37d1ffb700 1 -- 172.21.15.169:0/2568442621 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c000ccf0 con 0x7f37d4101d20 2024-03-28T00:56:16.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.618+0000 7f37d1ffb700 1 -- 172.21.15.169:0/2568442621 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f37c000ce90 con 0x7f37d4101d20 2024-03-28T00:56:16.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.618+0000 7f37dad26700 1 -- 172.21.15.169:0/2568442621 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37d419f190 con 0x7f37d4101d20 2024-03-28T00:56:16.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.618+0000 7f37d1ffb700 1 -- 172.21.15.169:0/2568442621 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c0010690 con 0x7f37d4101d20 2024-03-28T00:56:16.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.618+0000 7f37dad26700 1 -- 172.21.15.169:0/2568442621 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37d419f6d0 con 0x7f37d4101d20 2024-03-28T00:56:16.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.619+0000 7f37dad26700 1 -- 172.21.15.169:0/2568442621 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37d4068e50 con 0x7f37d4101d20 2024-03-28T00:56:16.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.620+0000 7f37d1ffb700 1 -- 172.21.15.169:0/2568442621 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f37c0003e10 con 0x7f37d4101d20 2024-03-28T00:56:16.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.620+0000 7f37d1ffb700 1 --2- 172.21.15.169:0/2568442621 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f37c40a2890 0x7f37c40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:16.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.620+0000 7f37d8ac2700 1 --2- 172.21.15.169:0/2568442621 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f37c40a2890 0x7f37c40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:16.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.620+0000 7f37d1ffb700 1 -- 172.21.15.169:0/2568442621 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f37c00cf130 con 0x7f37d4101d20 2024-03-28T00:56:16.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.621+0000 7f37d8ac2700 1 --2- 172.21.15.169:0/2568442621 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f37c40a2890 0x7f37c40a4d50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f37c8009c60 tx=0x7f37c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:16.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.625+0000 7f37d1ffb700 1 -- 172.21.15.169:0/2568442621 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f37c009be70 con 0x7f37d4101d20 2024-03-28T00:56:16.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.840+0000 7f37dad26700 1 -- 172.21.15.169:0/2568442621 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f37d40625d0 con 0x7f37c40a2890 2024-03-28T00:56:16.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.841+0000 7f37d1ffb700 1 -- 172.21.15.169:0/2568442621 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f37d40625d0 con 0x7f37c40a2890 2024-03-28T00:56:16.845 INFO:teuthology.orchestra.run.smithi169.stdout:Backend: cephadm 2024-03-28T00:56:16.845 INFO:teuthology.orchestra.run.smithi169.stdout:Available: Yes 2024-03-28T00:56:16.845 INFO:teuthology.orchestra.run.smithi169.stdout:Paused: No 2024-03-28T00:56:16.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.843+0000 7f37dad26700 1 -- 172.21.15.169:0/2568442621 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f37c40a2890 msgr2=0x7f37c40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:16.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.843+0000 7f37dad26700 1 --2- 172.21.15.169:0/2568442621 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f37c40a2890 0x7f37c40a4d50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f37c8009c60 tx=0x7f37c8009260 comp rx=0 tx=0).stop 2024-03-28T00:56:16.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.843+0000 7f37dad26700 1 -- 172.21.15.169:0/2568442621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37d4101d20 msgr2=0x7f37d4197780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:16.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.843+0000 7f37dad26700 1 --2- 172.21.15.169:0/2568442621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37d4101d20 0x7f37d4197780 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f37c000ec40 tx=0x7f37c000ef50 comp rx=0 tx=0).stop 2024-03-28T00:56:16.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.843+0000 7f37dad26700 1 -- 172.21.15.169:0/2568442621 shutdown_connections 2024-03-28T00:56:16.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.843+0000 7f37dad26700 1 --2- 172.21.15.169:0/2568442621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37d4102790 0x7f37d419e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:16.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.843+0000 7f37dad26700 1 --2- 172.21.15.169:0/2568442621 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37d4101360 0x7f37d4197240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:16.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.843+0000 7f37dad26700 1 --2- 172.21.15.169:0/2568442621 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f37c40a2890 0x7f37c40a4d50 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:16.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.843+0000 7f37dad26700 1 --2- 172.21.15.169:0/2568442621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37d4101d20 0x7f37d4197780 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:16.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.844+0000 7f37dad26700 1 -- 172.21.15.169:0/2568442621 >> 172.21.15.169:0/2568442621 conn(0x7f37d40fbe30 msgr2=0x7f37d41055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:16.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.844+0000 7f37dad26700 1 -- 172.21.15.169:0/2568442621 shutdown_connections 2024-03-28T00:56:16.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:16.844+0000 7f37dad26700 1 -- 172.21.15.169:0/2568442621 wait complete. 2024-03-28T00:56:17.450 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:17 smithi169 ceph-mon[32274]: pgmap v342: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:56:17.478 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-28T00:56:17.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:17 smithi174 ceph-mon[36060]: pgmap v342: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:56:17.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:17 smithi177 ceph-mon[36206]: pgmap v342: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:56:17.768 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:56:18.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:18 smithi169 ceph-mon[32274]: from='client.25060 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:56:18.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:18 smithi174 ceph-mon[36060]: from='client.25060 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:56:18.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:18 smithi177 ceph-mon[36206]: from='client.25060 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:56:19.505 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:19 smithi169 ceph-mon[32274]: pgmap v343: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:56:19.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.505+0000 7fa03ae99700 1 -- 172.21.15.169:0/1900287502 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0341021c0 msgr2=0x7fa0341025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:19.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.505+0000 7fa03ae99700 1 --2- 172.21.15.169:0/1900287502 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0341021c0 0x7fa0341025e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa0280099c0 tx=0x7fa028009cd0 comp rx=0 tx=0).stop 2024-03-28T00:56:19.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.505+0000 7fa03ae99700 1 -- 172.21.15.169:0/1900287502 shutdown_connections 2024-03-28T00:56:19.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.505+0000 7fa03ae99700 1 --2- 172.21.15.169:0/1900287502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa034103d60 0x7fa03410a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:19.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.505+0000 7fa03ae99700 1 --2- 172.21.15.169:0/1900287502 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0341021c0 0x7fa0341025e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:19.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.505+0000 7fa03ae99700 1 --2- 172.21.15.169:0/1900287502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0341033c0 0x7fa034103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:19.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.505+0000 7fa03ae99700 1 -- 172.21.15.169:0/1900287502 >> 172.21.15.169:0/1900287502 conn(0x7fa0340fd7a0 msgr2=0x7fa0340ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:19.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.505+0000 7fa03ae99700 1 -- 172.21.15.169:0/1900287502 shutdown_connections 2024-03-28T00:56:19.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.505+0000 7fa03ae99700 1 -- 172.21.15.169:0/1900287502 wait complete. 2024-03-28T00:56:19.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.506+0000 7fa03ae99700 1 Processor -- start 2024-03-28T00:56:19.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.506+0000 7fa03ae99700 1 -- start start 2024-03-28T00:56:19.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.506+0000 7fa03ae99700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0341021c0 0x7fa034198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:19.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.506+0000 7fa03ae99700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0341033c0 0x7fa034198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:19.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.506+0000 7fa03ae99700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa034103d60 0x7fa03419f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:19.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.506+0000 7fa03ae99700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03410ddb0 con 0x7fa034103d60 2024-03-28T00:56:19.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.506+0000 7fa03ae99700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03410dc30 con 0x7fa0341033c0 2024-03-28T00:56:19.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.506+0000 7fa03ae99700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03410df30 con 0x7fa0341021c0 2024-03-28T00:56:19.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.506+0000 7fa039436700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa034103d60 0x7fa03419f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:19.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.506+0000 7fa039436700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa034103d60 0x7fa03419f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48446/0 (socket says 172.21.15.169:48446) 2024-03-28T00:56:19.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.506+0000 7fa039436700 1 -- 172.21.15.169:0/3555914305 learned_addr learned my addr 172.21.15.169:0/3555914305 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:19.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.506+0000 7fa033fff700 1 --2- 172.21.15.169:0/3555914305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0341033c0 0x7fa034198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:19.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.506+0000 7fa038c35700 1 --2- 172.21.15.169:0/3555914305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0341021c0 0x7fa034198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:19.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.507+0000 7fa039436700 1 -- 172.21.15.169:0/3555914305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0341021c0 msgr2=0x7fa034198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:19.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.507+0000 7fa039436700 1 --2- 172.21.15.169:0/3555914305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0341021c0 0x7fa034198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:19.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.507+0000 7fa039436700 1 -- 172.21.15.169:0/3555914305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0341033c0 msgr2=0x7fa034198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:19.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.507+0000 7fa039436700 1 --2- 172.21.15.169:0/3555914305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0341033c0 0x7fa034198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:19.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.507+0000 7fa039436700 1 -- 172.21.15.169:0/3555914305 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0341a0070 con 0x7fa034103d60 2024-03-28T00:56:19.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.507+0000 7fa038c35700 1 --2- 172.21.15.169:0/3555914305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0341021c0 0x7fa034198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:19.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.507+0000 7fa039436700 1 --2- 172.21.15.169:0/3555914305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa034103d60 0x7fa03419f9d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa02c00ef60 tx=0x7fa02c00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:19.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.507+0000 7fa031ffb700 1 -- 172.21.15.169:0/3555914305 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c0092e0 con 0x7fa034103d60 2024-03-28T00:56:19.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.507+0000 7fa03ae99700 1 -- 172.21.15.169:0/3555914305 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0341a0300 con 0x7fa034103d60 2024-03-28T00:56:19.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.507+0000 7fa031ffb700 1 -- 172.21.15.169:0/3555914305 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa02c004bb0 con 0x7fa034103d60 2024-03-28T00:56:19.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.507+0000 7fa031ffb700 1 -- 172.21.15.169:0/3555914305 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c004280 con 0x7fa034103d60 2024-03-28T00:56:19.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.507+0000 7fa03ae99700 1 -- 172.21.15.169:0/3555914305 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0341a0840 con 0x7fa034103d60 2024-03-28T00:56:19.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.508+0000 7fa03ae99700 1 -- 172.21.15.169:0/3555914305 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa034068e50 con 0x7fa034103d60 2024-03-28T00:56:19.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.509+0000 7fa031ffb700 1 -- 172.21.15.169:0/3555914305 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa02c01b030 con 0x7fa034103d60 2024-03-28T00:56:19.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.510+0000 7fa031ffb700 1 --2- 172.21.15.169:0/3555914305 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa0240a2890 0x7fa0240a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:19.514 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.510+0000 7fa038c35700 1 --2- 172.21.15.169:0/3555914305 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa0240a2890 0x7fa0240a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:19.514 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.510+0000 7fa031ffb700 1 -- 172.21.15.169:0/3555914305 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fa02c0ce830 con 0x7fa034103d60 2024-03-28T00:56:19.514 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.510+0000 7fa038c35700 1 --2- 172.21.15.169:0/3555914305 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa0240a2890 0x7fa0240a4d50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa02800b580 tx=0x7fa028005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:19.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.514+0000 7fa031ffb700 1 -- 172.21.15.169:0/3555914305 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa02c022020 con 0x7fa034103d60 2024-03-28T00:56:19.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:19 smithi174 ceph-mon[36060]: pgmap v343: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:56:19.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:19 smithi177 ceph-mon[36206]: pgmap v343: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:56:19.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.720+0000 7fa03ae99700 1 -- 172.21.15.169:0/3555914305 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0340625d0 con 0x7fa0240a2890 2024-03-28T00:56:19.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.734+0000 7fa031ffb700 1 -- 172.21.15.169:0/3555914305 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7fa0340625d0 con 0x7fa0240a2890 2024-03-28T00:56:19.737 INFO:teuthology.orchestra.run.smithi169.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-28T00:56:19.737 INFO:teuthology.orchestra.run.smithi169.stdout:mds.a smithi169 running (114s) 18s ago 113s 13.8M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 d4b5685df0c6 2024-03-28T00:56:19.737 INFO:teuthology.orchestra.run.smithi169.stdout:mds.b smithi174 running (111s) 44s ago 110s 26.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 08f12830df86 2024-03-28T00:56:19.737 INFO:teuthology.orchestra.run.smithi169.stdout:mds.c smithi177 running (2m) 44s ago 2m 18.9M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 729e7063b6eb 2024-03-28T00:56:19.737 INFO:teuthology.orchestra.run.smithi169.stdout:mds.d smithi169 running (2m) 18s ago 2m 13.9M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 95e1f3625dc3 2024-03-28T00:56:19.738 INFO:teuthology.orchestra.run.smithi169.stdout:mds.e smithi174 running (2m) 44s ago 2m 15.5M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 66562be0736e 2024-03-28T00:56:19.738 INFO:teuthology.orchestra.run.smithi169.stdout:mds.f smithi177 running (2m) 44s ago 2m 17.7M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 8f7ef40b1835 2024-03-28T00:56:19.738 INFO:teuthology.orchestra.run.smithi169.stdout:mds.g smithi169 running (116s) 18s ago 116s 12.7M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 73be7983f9da 2024-03-28T00:56:19.738 INFO:teuthology.orchestra.run.smithi169.stdout:mds.h smithi174 running (2m) 44s ago 2m 18.7M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 7c47ea8c36a8 2024-03-28T00:56:19.738 INFO:teuthology.orchestra.run.smithi169.stdout:mds.i smithi177 running (119s) 44s ago 119s 18.0M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 6863e73af0f0 2024-03-28T00:56:19.738 INFO:teuthology.orchestra.run.smithi169.stdout:mds.j smithi169 running (2m) 18s ago 2m 15.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 0433ace5a545 2024-03-28T00:56:19.739 INFO:teuthology.orchestra.run.smithi169.stdout:mds.k smithi174 running (2m) 44s ago 2m 16.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 d59d450f31ef 2024-03-28T00:56:19.739 INFO:teuthology.orchestra.run.smithi169.stdout:mds.l smithi177 running (2m) 44s ago 2m 13.1M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 eda74141a6aa 2024-03-28T00:56:19.739 INFO:teuthology.orchestra.run.smithi169.stdout:mgr.x smithi169 *:9283,8765 running (9m) 18s ago 9m 517M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 baba397a44a4 2024-03-28T00:56:19.739 INFO:teuthology.orchestra.run.smithi169.stdout:mgr.y smithi174 *:8443,8765 running (7m) 44s ago 7m 453M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 fbd1ef7452fc 2024-03-28T00:56:19.739 INFO:teuthology.orchestra.run.smithi169.stdout:mgr.z smithi177 *:8443,8765 running (7m) 44s ago 7m 453M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 a1b0b6a946b2 2024-03-28T00:56:19.739 INFO:teuthology.orchestra.run.smithi169.stdout:mon.a smithi169 running (9m) 18s ago 9m 49.5M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 559d191b9d44 2024-03-28T00:56:19.739 INFO:teuthology.orchestra.run.smithi169.stdout:mon.b smithi174 running (7m) 44s ago 7m 39.0M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 19bd7a2dd6fe 2024-03-28T00:56:19.739 INFO:teuthology.orchestra.run.smithi169.stdout:mon.c smithi177 running (8m) 44s ago 7m 39.3M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 e37383dc5f0d 2024-03-28T00:56:19.739 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0 smithi169 running (7m) 18s ago 7m 56.1M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 d9b13d3fea96 2024-03-28T00:56:19.739 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1 smithi174 running (6m) 44s ago 6m 46.8M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 01586815ac4a 2024-03-28T00:56:19.739 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2 smithi177 running (6m) 44s ago 6m 47.3M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 cc239ec94f15 2024-03-28T00:56:19.739 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3 smithi169 running (6m) 18s ago 6m 44.5M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 14077a9a999a 2024-03-28T00:56:19.740 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4 smithi174 running (5m) 44s ago 5m 53.3M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 592754e094e7 2024-03-28T00:56:19.740 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5 smithi177 running (5m) 44s ago 5m 47.3M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 2a21aa6e83e3 2024-03-28T00:56:19.740 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6 smithi169 running (4m) 18s ago 4m 54.2M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 8df1010710b3 2024-03-28T00:56:19.740 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7 smithi174 running (4m) 44s ago 4m 44.2M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 6df5fc0be6c8 2024-03-28T00:56:19.740 INFO:teuthology.orchestra.run.smithi169.stdout:osd.8 smithi177 running (4m) 44s ago 4m 42.3M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 b1ce4b588bac 2024-03-28T00:56:19.740 INFO:teuthology.orchestra.run.smithi169.stdout:osd.9 smithi169 running (3m) 18s ago 3m 43.1M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 d68890c7d7f3 2024-03-28T00:56:19.740 INFO:teuthology.orchestra.run.smithi169.stdout:osd.10 smithi174 running (3m) 44s ago 3m 55.4M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 84a25479b76f 2024-03-28T00:56:19.740 INFO:teuthology.orchestra.run.smithi169.stdout:osd.11 smithi177 running (2m) 44s ago 2m 54.0M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 03e05207a750 2024-03-28T00:56:19.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.736+0000 7fa03ae99700 1 -- 172.21.15.169:0/3555914305 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa0240a2890 msgr2=0x7fa0240a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:19.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.736+0000 7fa03ae99700 1 --2- 172.21.15.169:0/3555914305 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa0240a2890 0x7fa0240a4d50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa02800b580 tx=0x7fa028005fb0 comp rx=0 tx=0).stop 2024-03-28T00:56:19.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.736+0000 7fa03ae99700 1 -- 172.21.15.169:0/3555914305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa034103d60 msgr2=0x7fa03419f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:19.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.736+0000 7fa03ae99700 1 --2- 172.21.15.169:0/3555914305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa034103d60 0x7fa03419f9d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa02c00ef60 tx=0x7fa02c00ef90 comp rx=0 tx=0).stop 2024-03-28T00:56:19.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.736+0000 7fa03ae99700 1 -- 172.21.15.169:0/3555914305 shutdown_connections 2024-03-28T00:56:19.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.736+0000 7fa03ae99700 1 --2- 172.21.15.169:0/3555914305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa034103d60 0x7fa03419f9d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:19.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.736+0000 7fa03ae99700 1 --2- 172.21.15.169:0/3555914305 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa0240a2890 0x7fa0240a4d50 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:19.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.736+0000 7fa03ae99700 1 --2- 172.21.15.169:0/3555914305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0341021c0 0x7fa034198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:19.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.736+0000 7fa03ae99700 1 --2- 172.21.15.169:0/3555914305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0341033c0 0x7fa034198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:19.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.736+0000 7fa03ae99700 1 -- 172.21.15.169:0/3555914305 >> 172.21.15.169:0/3555914305 conn(0x7fa0340fd7a0 msgr2=0x7fa034106760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:19.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.736+0000 7fa03ae99700 1 -- 172.21.15.169:0/3555914305 shutdown_connections 2024-03-28T00:56:19.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:19.736+0000 7fa03ae99700 1 -- 172.21.15.169:0/3555914305 wait complete. 2024-03-28T00:56:20.521 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-28T00:56:20.766 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:56:21.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:21 smithi169 ceph-mon[32274]: from='client.15273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:56:21.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:21 smithi169 ceph-mon[32274]: pgmap v344: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T00:56:21.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:21 smithi174 ceph-mon[36060]: from='client.15273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:56:21.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:21 smithi174 ceph-mon[36060]: pgmap v344: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T00:56:21.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:21 smithi177 ceph-mon[36206]: from='client.15273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:56:21.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:21 smithi177 ceph-mon[36206]: pgmap v344: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T00:56:22.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.572+0000 7f0c006c5700 1 -- 172.21.15.169:0/4014682136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0bf81021c0 msgr2=0x7f0bf81025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:22.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.572+0000 7f0c006c5700 1 --2- 172.21.15.169:0/4014682136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0bf81021c0 0x7f0bf81025e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40099c0 tx=0x7f0bf4009cd0 comp rx=0 tx=0).stop 2024-03-28T00:56:22.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.572+0000 7f0c006c5700 1 -- 172.21.15.169:0/4014682136 shutdown_connections 2024-03-28T00:56:22.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.572+0000 7f0c006c5700 1 --2- 172.21.15.169:0/4014682136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bf8103d60 0x7f0bf810a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:22.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.572+0000 7f0c006c5700 1 --2- 172.21.15.169:0/4014682136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0bf81021c0 0x7f0bf81025e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:22.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.572+0000 7f0c006c5700 1 --2- 172.21.15.169:0/4014682136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0bf81033c0 0x7f0bf8103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:22.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.572+0000 7f0c006c5700 1 -- 172.21.15.169:0/4014682136 >> 172.21.15.169:0/4014682136 conn(0x7f0bf80fd7a0 msgr2=0x7f0bf80ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:22.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.572+0000 7f0c006c5700 1 -- 172.21.15.169:0/4014682136 shutdown_connections 2024-03-28T00:56:22.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.573+0000 7f0c006c5700 1 -- 172.21.15.169:0/4014682136 wait complete. 2024-03-28T00:56:22.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.573+0000 7f0c006c5700 1 Processor -- start 2024-03-28T00:56:22.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.573+0000 7f0c006c5700 1 -- start start 2024-03-28T00:56:22.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.573+0000 7f0c006c5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0bf81021c0 0x7f0bf8198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:22.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.573+0000 7f0c006c5700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bf81033c0 0x7f0bf8198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:22.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.573+0000 7f0c006c5700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0bf8103d60 0x7f0bf819f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:22.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.573+0000 7f0c006c5700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf810ddb0 con 0x7f0bf81021c0 2024-03-28T00:56:22.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.573+0000 7f0c006c5700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf810dc30 con 0x7f0bf8103d60 2024-03-28T00:56:22.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.573+0000 7f0c006c5700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf810df30 con 0x7f0bf81033c0 2024-03-28T00:56:22.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.573+0000 7f0bfe461700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0bf81021c0 0x7f0bf8198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:22.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.573+0000 7f0bfe461700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0bf81021c0 0x7f0bf8198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48478/0 (socket says 172.21.15.169:48478) 2024-03-28T00:56:22.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.573+0000 7f0bfe461700 1 -- 172.21.15.169:0/3336411537 learned_addr learned my addr 172.21.15.169:0/3336411537 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:22.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.574+0000 7f0bfdc60700 1 --2- 172.21.15.169:0/3336411537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bf81033c0 0x7f0bf8198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:22.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.574+0000 7f0bfec62700 1 --2- 172.21.15.169:0/3336411537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0bf8103d60 0x7f0bf819f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:22.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.574+0000 7f0bfdc60700 1 -- 172.21.15.169:0/3336411537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0bf8103d60 msgr2=0x7f0bf819f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:22.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.574+0000 7f0bfdc60700 1 --2- 172.21.15.169:0/3336411537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0bf8103d60 0x7f0bf819f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:22.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.574+0000 7f0bfdc60700 1 -- 172.21.15.169:0/3336411537 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0bf81021c0 msgr2=0x7f0bf8198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:22.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.574+0000 7f0bfdc60700 1 --2- 172.21.15.169:0/3336411537 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0bf81021c0 0x7f0bf8198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:22.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.574+0000 7f0bfdc60700 1 -- 172.21.15.169:0/3336411537 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf81a0070 con 0x7f0bf81033c0 2024-03-28T00:56:22.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.574+0000 7f0bfe461700 1 --2- 172.21.15.169:0/3336411537 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0bf81021c0 0x7f0bf8198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:22.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.574+0000 7f0bfdc60700 1 --2- 172.21.15.169:0/3336411537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bf81033c0 0x7f0bf8198950 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0be800ea20 tx=0x7f0be800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:22.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.574+0000 7f0bef7fe700 1 -- 172.21.15.169:0/3336411537 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be800cbd0 con 0x7f0bf81033c0 2024-03-28T00:56:22.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.574+0000 7f0bef7fe700 1 -- 172.21.15.169:0/3336411537 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0be800ee60 con 0x7f0bf81033c0 2024-03-28T00:56:22.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.574+0000 7f0c006c5700 1 -- 172.21.15.169:0/3336411537 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf81a0360 con 0x7f0bf81033c0 2024-03-28T00:56:22.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.574+0000 7f0bef7fe700 1 -- 172.21.15.169:0/3336411537 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be8018680 con 0x7f0bf81033c0 2024-03-28T00:56:22.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.574+0000 7f0c006c5700 1 -- 172.21.15.169:0/3336411537 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf81a08a0 con 0x7f0bf81033c0 2024-03-28T00:56:22.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.575+0000 7f0c006c5700 1 -- 172.21.15.169:0/3336411537 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf8068e50 con 0x7f0bf81033c0 2024-03-28T00:56:22.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.576+0000 7f0bef7fe700 1 -- 172.21.15.169:0/3336411537 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0be8047a60 con 0x7f0bf81033c0 2024-03-28T00:56:22.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.577+0000 7f0bef7fe700 1 --2- 172.21.15.169:0/3336411537 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0be40a2890 0x7f0be40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:22.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.577+0000 7f0bfe461700 1 --2- 172.21.15.169:0/3336411537 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0be40a2890 0x7f0be40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:22.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.577+0000 7f0bef7fe700 1 -- 172.21.15.169:0/3336411537 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f0be80cee70 con 0x7f0bf81033c0 2024-03-28T00:56:22.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.577+0000 7f0bfe461700 1 --2- 172.21.15.169:0/3336411537 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0be40a2890 0x7f0be40a4d50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4000c00 tx=0x7f0bf4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:22.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.580+0000 7f0bef7fe700 1 -- 172.21.15.169:0/3336411537 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0be809bbb0 con 0x7f0bf81033c0 2024-03-28T00:56:22.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.784+0000 7f0c006c5700 1 -- 172.21.15.169:0/3336411537 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bf80625d0 con 0x7f0be40a2890 2024-03-28T00:56:22.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.790+0000 7f0bef7fe700 1 -- 172.21.15.169:0/3336411537 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f0bf80625d0 con 0x7f0be40a2890 2024-03-28T00:56:22.793 INFO:teuthology.orchestra.run.smithi169.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-28T00:56:22.793 INFO:teuthology.orchestra.run.smithi169.stdout:mds.all 12/12 47s ago 2m smithi169=a;smithi169=d;smithi169=g;smithi169=j;smithi174=b;smithi174=e;smithi174=h;smithi174=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l;count:12 2024-03-28T00:56:22.793 INFO:teuthology.orchestra.run.smithi169.stdout:mgr 3/3 47s ago 7m smithi169=x;smithi174=y;smithi177=z;count:3 2024-03-28T00:56:22.793 INFO:teuthology.orchestra.run.smithi169.stdout:mon 3/3 47s ago 8m smithi169:172.21.15.169=a;smithi174:172.21.15.174=b;smithi177:172.21.15.177=c;count:3 2024-03-28T00:56:22.793 INFO:teuthology.orchestra.run.smithi169.stdout:osd 12 47s ago - 2024-03-28T00:56:22.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.792+0000 7f0c006c5700 1 -- 172.21.15.169:0/3336411537 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0be40a2890 msgr2=0x7f0be40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:22.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.792+0000 7f0c006c5700 1 --2- 172.21.15.169:0/3336411537 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0be40a2890 0x7f0be40a4d50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4000c00 tx=0x7f0bf4005fb0 comp rx=0 tx=0).stop 2024-03-28T00:56:22.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.792+0000 7f0c006c5700 1 -- 172.21.15.169:0/3336411537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bf81033c0 msgr2=0x7f0bf8198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:22.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.792+0000 7f0c006c5700 1 --2- 172.21.15.169:0/3336411537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bf81033c0 0x7f0bf8198950 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0be800ea20 tx=0x7f0be800ed30 comp rx=0 tx=0).stop 2024-03-28T00:56:22.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.792+0000 7f0c006c5700 1 -- 172.21.15.169:0/3336411537 shutdown_connections 2024-03-28T00:56:22.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.792+0000 7f0c006c5700 1 --2- 172.21.15.169:0/3336411537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0bf8103d60 0x7f0bf819f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:22.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.792+0000 7f0c006c5700 1 --2- 172.21.15.169:0/3336411537 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0bf81021c0 0x7f0bf8198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:22.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.792+0000 7f0c006c5700 1 --2- 172.21.15.169:0/3336411537 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0be40a2890 0x7f0be40a4d50 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:22.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.792+0000 7f0c006c5700 1 --2- 172.21.15.169:0/3336411537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bf81033c0 0x7f0bf8198950 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:22.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.792+0000 7f0c006c5700 1 -- 172.21.15.169:0/3336411537 >> 172.21.15.169:0/3336411537 conn(0x7f0bf80fd7a0 msgr2=0x7f0bf8106760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:22.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.792+0000 7f0c006c5700 1 -- 172.21.15.169:0/3336411537 shutdown_connections 2024-03-28T00:56:22.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:22.792+0000 7f0c006c5700 1 -- 172.21.15.169:0/3336411537 wait complete. 2024-03-28T00:56:23.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:23 smithi169 ceph-mon[32274]: pgmap v345: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-28T00:56:23.604 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-28T00:56:23.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:23 smithi174 ceph-mon[36060]: pgmap v345: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-28T00:56:23.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:23 smithi177 ceph-mon[36206]: pgmap v345: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-28T00:56:23.848 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:56:24.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:24 smithi169 ceph-mon[32274]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:56:24.625 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:24 smithi174 ceph-mon[36060]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:56:24.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:24 smithi177 ceph-mon[36206]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:56:25.428 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:25 smithi169 ceph-mon[32274]: pgmap v346: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T00:56:25.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:25 smithi174 ceph-mon[36060]: pgmap v346: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T00:56:25.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.636+0000 7f4ba1a93700 1 -- 172.21.15.169:0/1054002173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b9c1004c0 msgr2=0x7f4b9c1008e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:25.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.636+0000 7f4ba1a93700 1 --2- 172.21.15.169:0/1054002173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b9c1004c0 0x7f4b9c1008e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c009990 tx=0x7f4b8c009ca0 comp rx=0 tx=0).stop 2024-03-28T00:56:25.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.636+0000 7f4ba1a93700 1 -- 172.21.15.169:0/1054002173 shutdown_connections 2024-03-28T00:56:25.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.636+0000 7f4ba1a93700 1 --2- 172.21.15.169:0/1054002173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b9c103e40 0x7f4b9c10a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:25.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.636+0000 7f4ba1a93700 1 --2- 172.21.15.169:0/1054002173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b9c1004c0 0x7f4b9c1008e0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:25.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.636+0000 7f4ba1a93700 1 --2- 172.21.15.169:0/1054002173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b9c1034a0 0x7f4b9c103900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:25.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.636+0000 7f4ba1a93700 1 -- 172.21.15.169:0/1054002173 >> 172.21.15.169:0/1054002173 conn(0x7f4b9c0fbe20 msgr2=0x7f4b9c0fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:25.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.636+0000 7f4ba1a93700 1 -- 172.21.15.169:0/1054002173 shutdown_connections 2024-03-28T00:56:25.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.636+0000 7f4ba1a93700 1 -- 172.21.15.169:0/1054002173 wait complete. 2024-03-28T00:56:25.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.636+0000 7f4ba1a93700 1 Processor -- start 2024-03-28T00:56:25.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.637+0000 7f4ba1a93700 1 -- start start 2024-03-28T00:56:25.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.637+0000 7f4ba1a93700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b9c1004c0 0x7f4b9c1983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:25.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.637+0000 7f4ba1a93700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b9c1034a0 0x7f4b9c198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:25.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.637+0000 7f4ba1a93700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b9c103e40 0x7f4b9c19f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:25.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.637+0000 7f4ba1a93700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c10ddd0 con 0x7f4b9c1004c0 2024-03-28T00:56:25.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.637+0000 7f4ba1a93700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c10dc50 con 0x7f4b9c1034a0 2024-03-28T00:56:25.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.637+0000 7f4ba1a93700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c10df50 con 0x7f4b9c103e40 2024-03-28T00:56:25.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.637+0000 7f4b9b7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b9c1004c0 0x7f4b9c1983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:25.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.637+0000 7f4b9b7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b9c1004c0 0x7f4b9c1983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48494/0 (socket says 172.21.15.169:48494) 2024-03-28T00:56:25.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.637+0000 7f4b9b7fe700 1 -- 172.21.15.169:0/2126649240 learned_addr learned my addr 172.21.15.169:0/2126649240 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:25.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.637+0000 7f4b9bfff700 1 --2- 172.21.15.169:0/2126649240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b9c103e40 0x7f4b9c19f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:25.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.637+0000 7f4b9affd700 1 --2- 172.21.15.169:0/2126649240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b9c1034a0 0x7f4b9c198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:25.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.637+0000 7f4b9b7fe700 1 -- 172.21.15.169:0/2126649240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b9c103e40 msgr2=0x7f4b9c19f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:25.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.637+0000 7f4b9b7fe700 1 --2- 172.21.15.169:0/2126649240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b9c103e40 0x7f4b9c19f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:25.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.637+0000 7f4b9b7fe700 1 -- 172.21.15.169:0/2126649240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b9c1034a0 msgr2=0x7f4b9c198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:25.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.637+0000 7f4b9b7fe700 1 --2- 172.21.15.169:0/2126649240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b9c1034a0 0x7f4b9c198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:25.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.637+0000 7f4b9b7fe700 1 -- 172.21.15.169:0/2126649240 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9c1a0040 con 0x7f4b9c1004c0 2024-03-28T00:56:25.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.637+0000 7f4b9b7fe700 1 --2- 172.21.15.169:0/2126649240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b9c1004c0 0x7f4b9c1983e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c005f50 tx=0x7f4b8c00bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:25.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.638+0000 7f4b98ff9700 1 -- 172.21.15.169:0/2126649240 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c026070 con 0x7f4b9c1004c0 2024-03-28T00:56:25.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.638+0000 7f4ba1a93700 1 -- 172.21.15.169:0/2126649240 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b9c1a02d0 con 0x7f4b9c1004c0 2024-03-28T00:56:25.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.638+0000 7f4b98ff9700 1 -- 172.21.15.169:0/2126649240 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b8c00f460 con 0x7f4b9c1004c0 2024-03-28T00:56:25.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.638+0000 7f4b98ff9700 1 -- 172.21.15.169:0/2126649240 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c021670 con 0x7f4b9c1004c0 2024-03-28T00:56:25.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.638+0000 7f4ba1a93700 1 -- 172.21.15.169:0/2126649240 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b9c1a0780 con 0x7f4b9c1004c0 2024-03-28T00:56:25.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.639+0000 7f4ba1a93700 1 -- 172.21.15.169:0/2126649240 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b9c068e50 con 0x7f4b9c1004c0 2024-03-28T00:56:25.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.640+0000 7f4b98ff9700 1 -- 172.21.15.169:0/2126649240 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4b8c029070 con 0x7f4b9c1004c0 2024-03-28T00:56:25.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.640+0000 7f4b98ff9700 1 --2- 172.21.15.169:0/2126649240 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4b840a2890 0x7f4b840a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:25.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.640+0000 7f4b9affd700 1 --2- 172.21.15.169:0/2126649240 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4b840a2890 0x7f4b840a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:25.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.640+0000 7f4b98ff9700 1 -- 172.21.15.169:0/2126649240 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f4b8c0cff10 con 0x7f4b9c1004c0 2024-03-28T00:56:25.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.641+0000 7f4b9affd700 1 --2- 172.21.15.169:0/2126649240 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4b840a2890 0x7f4b840a4d50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4b90009c60 tx=0x7f4b90009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:25.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.644+0000 7f4b98ff9700 1 -- 172.21.15.169:0/2126649240 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4b8c030020 con 0x7f4b9c1004c0 2024-03-28T00:56:25.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:25 smithi177 ceph-mon[36206]: pgmap v346: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T00:56:25.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.847+0000 7f4ba1a93700 1 -- 172.21.15.169:0/2126649240 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b9c0625d0 con 0x7f4b840a2890 2024-03-28T00:56:25.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.848+0000 7f4b98ff9700 1 -- 172.21.15.169:0/2126649240 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f4b9c0625d0 con 0x7f4b840a2890 2024-03-28T00:56:25.851 INFO:teuthology.orchestra.run.smithi169.stdout:HOST ADDR LABELS STATUS 2024-03-28T00:56:25.851 INFO:teuthology.orchestra.run.smithi169.stdout:smithi169 172.21.15.169 2024-03-28T00:56:25.851 INFO:teuthology.orchestra.run.smithi169.stdout:smithi174 172.21.15.174 2024-03-28T00:56:25.851 INFO:teuthology.orchestra.run.smithi169.stdout:smithi177 172.21.15.177 2024-03-28T00:56:25.851 INFO:teuthology.orchestra.run.smithi169.stdout:3 hosts in cluster 2024-03-28T00:56:25.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.850+0000 7f4ba1a93700 1 -- 172.21.15.169:0/2126649240 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4b840a2890 msgr2=0x7f4b840a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:25.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.850+0000 7f4ba1a93700 1 --2- 172.21.15.169:0/2126649240 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4b840a2890 0x7f4b840a4d50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4b90009c60 tx=0x7f4b90009260 comp rx=0 tx=0).stop 2024-03-28T00:56:25.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.850+0000 7f4ba1a93700 1 -- 172.21.15.169:0/2126649240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b9c1004c0 msgr2=0x7f4b9c1983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:25.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.850+0000 7f4ba1a93700 1 --2- 172.21.15.169:0/2126649240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b9c1004c0 0x7f4b9c1983e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c005f50 tx=0x7f4b8c00bbb0 comp rx=0 tx=0).stop 2024-03-28T00:56:25.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.850+0000 7f4ba1a93700 1 -- 172.21.15.169:0/2126649240 shutdown_connections 2024-03-28T00:56:25.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.850+0000 7f4ba1a93700 1 --2- 172.21.15.169:0/2126649240 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4b840a2890 0x7f4b840a4d50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:25.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.850+0000 7f4ba1a93700 1 --2- 172.21.15.169:0/2126649240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b9c103e40 0x7f4b9c19f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:25.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.850+0000 7f4ba1a93700 1 --2- 172.21.15.169:0/2126649240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4b9c1004c0 0x7f4b9c1983e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:25.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.850+0000 7f4ba1a93700 1 --2- 172.21.15.169:0/2126649240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b9c1034a0 0x7f4b9c198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:25.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.850+0000 7f4ba1a93700 1 -- 172.21.15.169:0/2126649240 >> 172.21.15.169:0/2126649240 conn(0x7f4b9c0fbe20 msgr2=0x7f4b9c106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:25.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.850+0000 7f4ba1a93700 1 -- 172.21.15.169:0/2126649240 shutdown_connections 2024-03-28T00:56:25.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:25.850+0000 7f4ba1a93700 1 -- 172.21.15.169:0/2126649240 wait complete. 2024-03-28T00:56:26.618 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-28T00:56:26.865 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:56:27.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:27 smithi169 ceph-mon[32274]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:56:27.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:27 smithi169 ceph-mon[32274]: pgmap v347: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T00:56:27.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:27 smithi174 ceph-mon[36060]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:56:27.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:27 smithi174 ceph-mon[36060]: pgmap v347: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T00:56:27.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:27 smithi177 ceph-mon[36206]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:56:27.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:27 smithi177 ceph-mon[36206]: pgmap v347: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T00:56:28.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.697+0000 7ff3cbce4700 1 -- 172.21.15.169:0/3459643570 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3c4103d60 msgr2=0x7ff3c410a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:28.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.697+0000 7ff3cbce4700 1 --2- 172.21.15.169:0/3459643570 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3c4103d60 0x7ff3c410a610 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc00b0d0 tx=0x7ff3bc00b3e0 comp rx=0 tx=0).stop 2024-03-28T00:56:28.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.697+0000 7ff3cbce4700 1 -- 172.21.15.169:0/3459643570 shutdown_connections 2024-03-28T00:56:28.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.697+0000 7ff3cbce4700 1 --2- 172.21.15.169:0/3459643570 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3c4103d60 0x7ff3c410a610 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:28.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.697+0000 7ff3cbce4700 1 --2- 172.21.15.169:0/3459643570 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3c41021c0 0x7ff3c41025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:28.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.697+0000 7ff3cbce4700 1 --2- 172.21.15.169:0/3459643570 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3c41033c0 0x7ff3c4103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:28.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.697+0000 7ff3cbce4700 1 -- 172.21.15.169:0/3459643570 >> 172.21.15.169:0/3459643570 conn(0x7ff3c40fd7a0 msgr2=0x7ff3c40ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:28.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.697+0000 7ff3cbce4700 1 -- 172.21.15.169:0/3459643570 shutdown_connections 2024-03-28T00:56:28.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.697+0000 7ff3cbce4700 1 -- 172.21.15.169:0/3459643570 wait complete. 2024-03-28T00:56:28.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.698+0000 7ff3cbce4700 1 Processor -- start 2024-03-28T00:56:28.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.698+0000 7ff3cbce4700 1 -- start start 2024-03-28T00:56:28.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.698+0000 7ff3cbce4700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3c41021c0 0x7ff3c4198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:28.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.698+0000 7ff3cbce4700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3c41033c0 0x7ff3c41988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:28.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.698+0000 7ff3cbce4700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3c4103d60 0x7ff3c419f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:28.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.698+0000 7ff3cbce4700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c410dda0 con 0x7ff3c41033c0 2024-03-28T00:56:28.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.698+0000 7ff3cbce4700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c410dc20 con 0x7ff3c41021c0 2024-03-28T00:56:28.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.698+0000 7ff3cbce4700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c410df20 con 0x7ff3c4103d60 2024-03-28T00:56:28.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.698+0000 7ff3c927f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3c41033c0 0x7ff3c41988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:28.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.698+0000 7ff3c927f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3c41033c0 0x7ff3c41988a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47406/0 (socket says 172.21.15.169:47406) 2024-03-28T00:56:28.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.698+0000 7ff3c927f700 1 -- 172.21.15.169:0/3161254606 learned_addr learned my addr 172.21.15.169:0/3161254606 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:28.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.698+0000 7ff3c9a80700 1 --2- 172.21.15.169:0/3161254606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3c41021c0 0x7ff3c4198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:28.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.699+0000 7ff3ca281700 1 --2- 172.21.15.169:0/3161254606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3c4103d60 0x7ff3c419f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:28.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.699+0000 7ff3c927f700 1 -- 172.21.15.169:0/3161254606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3c4103d60 msgr2=0x7ff3c419f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:28.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.699+0000 7ff3c927f700 1 --2- 172.21.15.169:0/3161254606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3c4103d60 0x7ff3c419f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:28.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.699+0000 7ff3c927f700 1 -- 172.21.15.169:0/3161254606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3c41021c0 msgr2=0x7ff3c4198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:28.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.699+0000 7ff3c927f700 1 --2- 172.21.15.169:0/3161254606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3c41021c0 0x7ff3c4198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:28.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.699+0000 7ff3c927f700 1 -- 172.21.15.169:0/3161254606 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c419ffc0 con 0x7ff3c41033c0 2024-03-28T00:56:28.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.699+0000 7ff3c9a80700 1 --2- 172.21.15.169:0/3161254606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3c41021c0 0x7ff3c4198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:28.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.699+0000 7ff3c927f700 1 --2- 172.21.15.169:0/3161254606 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3c41033c0 0x7ff3c41988a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000e9d0 tx=0x7ff3c000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:28.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.699+0000 7ff3baffd700 1 -- 172.21.15.169:0/3161254606 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c000cbd0 con 0x7ff3c41033c0 2024-03-28T00:56:28.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.699+0000 7ff3baffd700 1 -- 172.21.15.169:0/3161254606 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3c000ee70 con 0x7ff3c41033c0 2024-03-28T00:56:28.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.699+0000 7ff3baffd700 1 -- 172.21.15.169:0/3161254606 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0018660 con 0x7ff3c41033c0 2024-03-28T00:56:28.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.699+0000 7ff3cbce4700 1 -- 172.21.15.169:0/3161254606 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c41a02b0 con 0x7ff3c41033c0 2024-03-28T00:56:28.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.699+0000 7ff3cbce4700 1 -- 172.21.15.169:0/3161254606 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c41a07f0 con 0x7ff3c41033c0 2024-03-28T00:56:28.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.700+0000 7ff3cbce4700 1 -- 172.21.15.169:0/3161254606 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c4068e50 con 0x7ff3c41033c0 2024-03-28T00:56:28.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.701+0000 7ff3baffd700 1 -- 172.21.15.169:0/3161254606 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff3c0010430 con 0x7ff3c41033c0 2024-03-28T00:56:28.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.702+0000 7ff3baffd700 1 --2- 172.21.15.169:0/3161254606 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff3b00a2890 0x7ff3b00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:28.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.702+0000 7ff3c9a80700 1 --2- 172.21.15.169:0/3161254606 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff3b00a2890 0x7ff3b00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:28.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.702+0000 7ff3baffd700 1 -- 172.21.15.169:0/3161254606 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7ff3c00cefe0 con 0x7ff3c41033c0 2024-03-28T00:56:28.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.702+0000 7ff3c9a80700 1 --2- 172.21.15.169:0/3161254606 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff3b00a2890 0x7ff3b00a4d50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4103220 tx=0x7ff3b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:28.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.706+0000 7ff3baffd700 1 -- 172.21.15.169:0/3161254606 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff3c009bd20 con 0x7ff3c41033c0 2024-03-28T00:56:28.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.916+0000 7ff3cbce4700 1 -- 172.21.15.169:0/3161254606 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3c4062690 con 0x7ff3b00a2890 2024-03-28T00:56:28.920 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.917+0000 7ff3baffd700 1 -- 172.21.15.169:0/3161254606 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7ff3c4062690 con 0x7ff3b00a2890 2024-03-28T00:56:28.920 INFO:teuthology.orchestra.run.smithi169.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-28T00:56:28.920 INFO:teuthology.orchestra.run.smithi169.stdout:smithi169 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300LN400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-28T00:56:28.921 INFO:teuthology.orchestra.run.smithi169.stdout:smithi174 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62050074400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-28T00:56:28.921 INFO:teuthology.orchestra.run.smithi169.stdout:smithi177 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233007S400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-28T00:56:28.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.920+0000 7ff3cbce4700 1 -- 172.21.15.169:0/3161254606 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff3b00a2890 msgr2=0x7ff3b00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:28.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.920+0000 7ff3cbce4700 1 --2- 172.21.15.169:0/3161254606 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff3b00a2890 0x7ff3b00a4d50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4103220 tx=0x7ff3b4009260 comp rx=0 tx=0).stop 2024-03-28T00:56:28.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.920+0000 7ff3cbce4700 1 -- 172.21.15.169:0/3161254606 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3c41033c0 msgr2=0x7ff3c41988a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:28.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.920+0000 7ff3cbce4700 1 --2- 172.21.15.169:0/3161254606 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3c41033c0 0x7ff3c41988a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000e9d0 tx=0x7ff3c000ece0 comp rx=0 tx=0).stop 2024-03-28T00:56:28.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.920+0000 7ff3cbce4700 1 -- 172.21.15.169:0/3161254606 shutdown_connections 2024-03-28T00:56:28.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.920+0000 7ff3cbce4700 1 --2- 172.21.15.169:0/3161254606 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff3b00a2890 0x7ff3b00a4d50 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:28.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.920+0000 7ff3cbce4700 1 --2- 172.21.15.169:0/3161254606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3c4103d60 0x7ff3c419f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:28.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.920+0000 7ff3cbce4700 1 --2- 172.21.15.169:0/3161254606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3c41021c0 0x7ff3c4198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:28.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.920+0000 7ff3cbce4700 1 --2- 172.21.15.169:0/3161254606 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3c41033c0 0x7ff3c41988a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:28.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.920+0000 7ff3cbce4700 1 -- 172.21.15.169:0/3161254606 >> 172.21.15.169:0/3161254606 conn(0x7ff3c40fd7a0 msgr2=0x7ff3c41066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:28.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.920+0000 7ff3cbce4700 1 -- 172.21.15.169:0/3161254606 shutdown_connections 2024-03-28T00:56:28.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:28.920+0000 7ff3cbce4700 1 -- 172.21.15.169:0/3161254606 wait complete. 2024-03-28T00:56:29.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:29 smithi169 ceph-mon[32274]: pgmap v348: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T00:56:29.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:29 smithi174 ceph-mon[36060]: pgmap v348: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T00:56:29.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:29 smithi177 ceph-mon[36206]: pgmap v348: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T00:56:29.706 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T00:56:29.715 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi169.front.sepia.ceph.com 2024-03-28T00:56:29.715 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-28T00:56:29.967 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:56:30.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:30 smithi169 ceph-mon[32274]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:56:30.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:30 smithi174 ceph-mon[36060]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:56:30.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:30 smithi177 ceph-mon[36206]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:56:31.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:31 smithi169 ceph-mon[32274]: pgmap v349: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T00:56:31.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:31 smithi174 ceph-mon[36060]: pgmap v349: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T00:56:31.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:31 smithi177 ceph-mon[36206]: pgmap v349: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T00:56:31.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.815+0000 7f67224b2700 1 -- 172.21.15.169:0/3864823083 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f671c101130 msgr2=0x7f671c101550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:31.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.815+0000 7f67224b2700 1 --2- 172.21.15.169:0/3864823083 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f671c101130 0x7f671c101550 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f670c0099c0 tx=0x7f670c009cd0 comp rx=0 tx=0).stop 2024-03-28T00:56:31.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.815+0000 7f67224b2700 1 -- 172.21.15.169:0/3864823083 shutdown_connections 2024-03-28T00:56:31.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.815+0000 7f67224b2700 1 --2- 172.21.15.169:0/3864823083 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f671c102cd0 0x7f671c109580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:31.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.815+0000 7f67224b2700 1 --2- 172.21.15.169:0/3864823083 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f671c101130 0x7f671c101550 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:31.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.815+0000 7f67224b2700 1 --2- 172.21.15.169:0/3864823083 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f671c102330 0x7f671c102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:31.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.815+0000 7f67224b2700 1 -- 172.21.15.169:0/3864823083 >> 172.21.15.169:0/3864823083 conn(0x7f671c0fc6f0 msgr2=0x7f671c0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:31.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.815+0000 7f67224b2700 1 -- 172.21.15.169:0/3864823083 shutdown_connections 2024-03-28T00:56:31.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.815+0000 7f67224b2700 1 -- 172.21.15.169:0/3864823083 wait complete. 2024-03-28T00:56:31.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.815+0000 7f67224b2700 1 Processor -- start 2024-03-28T00:56:31.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.815+0000 7f67224b2700 1 -- start start 2024-03-28T00:56:31.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.816+0000 7f67224b2700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f671c101130 0x7f671c1a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:31.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.816+0000 7f67224b2700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f671c102330 0x7f671c1a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:31.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.816+0000 7f67224b2700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f671c102cd0 0x7f671c1a5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:31.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.816+0000 7f67224b2700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c10cc90 con 0x7f671c102cd0 2024-03-28T00:56:31.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.816+0000 7f67224b2700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c10cb10 con 0x7f671c102330 2024-03-28T00:56:31.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.816+0000 7f67224b2700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c10ce10 con 0x7f671c101130 2024-03-28T00:56:31.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.816+0000 7f6720a4f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f671c102cd0 0x7f671c1a5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:31.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.816+0000 7f671bfff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f671c101130 0x7f671c1a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:31.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.816+0000 7f671bfff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f671c101130 0x7f671c1a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56936/0 (socket says 172.21.15.169:56936) 2024-03-28T00:56:31.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.816+0000 7f6720a4f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f671c102cd0 0x7f671c1a5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47418/0 (socket says 172.21.15.169:47418) 2024-03-28T00:56:31.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.816+0000 7f671b7fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f671c102330 0x7f671c1a36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:31.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.816+0000 7f671bfff700 1 -- 172.21.15.169:0/1219296967 learned_addr learned my addr 172.21.15.169:0/1219296967 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:31.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.816+0000 7f6720a4f700 1 -- 172.21.15.169:0/1219296967 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f671c101130 msgr2=0x7f671c1a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:31.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.816+0000 7f6720a4f700 1 --2- 172.21.15.169:0/1219296967 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f671c101130 0x7f671c1a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:31.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.816+0000 7f6720a4f700 1 -- 172.21.15.169:0/1219296967 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f671c102330 msgr2=0x7f671c1a36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:31.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.816+0000 7f6720a4f700 1 --2- 172.21.15.169:0/1219296967 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f671c102330 0x7f671c1a36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:31.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.816+0000 7f6720a4f700 1 -- 172.21.15.169:0/1219296967 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671c1a6680 con 0x7f671c102cd0 2024-03-28T00:56:31.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.816+0000 7f671bfff700 1 --2- 172.21.15.169:0/1219296967 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f671c101130 0x7f671c1a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:31.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.816+0000 7f6720a4f700 1 --2- 172.21.15.169:0/1219296967 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f671c102cd0 0x7f671c1a5d80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f671400efc0 tx=0x7f671400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:31.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.817+0000 7f67197fa700 1 -- 172.21.15.169:0/1219296967 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67140092e0 con 0x7f671c102cd0 2024-03-28T00:56:31.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.817+0000 7f67197fa700 1 -- 172.21.15.169:0/1219296967 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6714004bb0 con 0x7f671c102cd0 2024-03-28T00:56:31.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.817+0000 7f67197fa700 1 -- 172.21.15.169:0/1219296967 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6714004280 con 0x7f671c102cd0 2024-03-28T00:56:31.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.817+0000 7f67224b2700 1 -- 172.21.15.169:0/1219296967 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671c1a6910 con 0x7f671c102cd0 2024-03-28T00:56:31.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.817+0000 7f67224b2700 1 -- 172.21.15.169:0/1219296967 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671c1a6e50 con 0x7f671c102cd0 2024-03-28T00:56:31.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.818+0000 7f67224b2700 1 -- 172.21.15.169:0/1219296967 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f671c068e80 con 0x7f671c102cd0 2024-03-28T00:56:31.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.819+0000 7f67197fa700 1 -- 172.21.15.169:0/1219296967 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f671401b030 con 0x7f671c102cd0 2024-03-28T00:56:31.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.819+0000 7f67197fa700 1 --2- 172.21.15.169:0/1219296967 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f67040a2910 0x7f67040a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:31.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.820+0000 7f671bfff700 1 --2- 172.21.15.169:0/1219296967 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f67040a2910 0x7f67040a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:31.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.820+0000 7f67197fa700 1 -- 172.21.15.169:0/1219296967 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f67140cf270 con 0x7f671c102cd0 2024-03-28T00:56:31.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.820+0000 7f671bfff700 1 --2- 172.21.15.169:0/1219296967 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f67040a2910 0x7f67040a4dd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f671c109a90 tx=0x7f670c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:31.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:31.823+0000 7f67197fa700 1 -- 172.21.15.169:0/1219296967 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f671c068e80 con 0x7f671c102cd0 2024-03-28T00:56:32.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:32.061+0000 7f67224b2700 1 -- 172.21.15.169:0/1219296967 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f671c1a72c0 con 0x7f671c102cd0 2024-03-28T00:56:32.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:32.061+0000 7f67197fa700 1 -- 172.21.15.169:0/1219296967 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+3011 (secure 0 0 0) 0x7f6714022020 con 0x7f671c102cd0 2024-03-28T00:56:32.064 INFO:teuthology.orchestra.run.smithi169.stdout:e24 2024-03-28T00:56:32.064 INFO:teuthology.orchestra.run.smithi169.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T00:56:32.065 INFO:teuthology.orchestra.run.smithi169.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T00:56:32.065 INFO:teuthology.orchestra.run.smithi169.stdout:legacy client fscid: 1 2024-03-28T00:56:32.065 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:32.065 INFO:teuthology.orchestra.run.smithi169.stdout:Filesystem 'cephfs' (1) 2024-03-28T00:56:32.065 INFO:teuthology.orchestra.run.smithi169.stdout:fs_name cephfs 2024-03-28T00:56:32.065 INFO:teuthology.orchestra.run.smithi169.stdout:epoch 24 2024-03-28T00:56:32.065 INFO:teuthology.orchestra.run.smithi169.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-28T00:56:32.065 INFO:teuthology.orchestra.run.smithi169.stdout:created 2024-03-28T00:54:32.613335+0000 2024-03-28T00:56:32.065 INFO:teuthology.orchestra.run.smithi169.stdout:modified 2024-03-28T00:54:52.274335+0000 2024-03-28T00:56:32.065 INFO:teuthology.orchestra.run.smithi169.stdout:tableserver 0 2024-03-28T00:56:32.065 INFO:teuthology.orchestra.run.smithi169.stdout:root 0 2024-03-28T00:56:32.065 INFO:teuthology.orchestra.run.smithi169.stdout:session_timeout 300 2024-03-28T00:56:32.065 INFO:teuthology.orchestra.run.smithi169.stdout:session_autoclose 300 2024-03-28T00:56:32.065 INFO:teuthology.orchestra.run.smithi169.stdout:max_file_size 1099511627776 2024-03-28T00:56:32.065 INFO:teuthology.orchestra.run.smithi169.stdout:max_xattr_size 65536 2024-03-28T00:56:32.066 INFO:teuthology.orchestra.run.smithi169.stdout:required_client_features {} 2024-03-28T00:56:32.066 INFO:teuthology.orchestra.run.smithi169.stdout:last_failure 0 2024-03-28T00:56:32.066 INFO:teuthology.orchestra.run.smithi169.stdout:last_failure_osd_epoch 0 2024-03-28T00:56:32.066 INFO:teuthology.orchestra.run.smithi169.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T00:56:32.066 INFO:teuthology.orchestra.run.smithi169.stdout:max_mds 3 2024-03-28T00:56:32.066 INFO:teuthology.orchestra.run.smithi169.stdout:in 0,1,2 2024-03-28T00:56:32.066 INFO:teuthology.orchestra.run.smithi169.stdout:up {0=24479,1=24478,2=24457} 2024-03-28T00:56:32.066 INFO:teuthology.orchestra.run.smithi169.stdout:failed 2024-03-28T00:56:32.066 INFO:teuthology.orchestra.run.smithi169.stdout:damaged 2024-03-28T00:56:32.066 INFO:teuthology.orchestra.run.smithi169.stdout:stopped 2024-03-28T00:56:32.066 INFO:teuthology.orchestra.run.smithi169.stdout:data_pools [3,4] 2024-03-28T00:56:32.066 INFO:teuthology.orchestra.run.smithi169.stdout:metadata_pool 2 2024-03-28T00:56:32.066 INFO:teuthology.orchestra.run.smithi169.stdout:inline_data disabled 2024-03-28T00:56:32.066 INFO:teuthology.orchestra.run.smithi169.stdout:balancer 2024-03-28T00:56:32.067 INFO:teuthology.orchestra.run.smithi169.stdout:bal_rank_mask -1 2024-03-28T00:56:32.067 INFO:teuthology.orchestra.run.smithi169.stdout:standby_count_wanted 0 2024-03-28T00:56:32.067 INFO:teuthology.orchestra.run.smithi169.stdout:qdb_cluster leader: 24479 members: 24479,24457,24478 2024-03-28T00:56:32.067 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.b{0:24479} state up:active seq 3 addr [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T00:56:32.067 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.h{0:24452} state up:standby-replay seq 1 addr [v2:172.21.15.174:6836/3629228577,v1:172.21.15.174:6837/3629228577] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T00:56:32.067 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.i{1:24478} state up:active seq 9 addr [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T00:56:32.067 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.f{1:24445} state up:standby-replay seq 1 addr [v2:172.21.15.177:6834/1050217570,v1:172.21.15.177:6835/1050217570] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T00:56:32.067 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.c{2:24457} state up:active seq 11 addr [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T00:56:32.067 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.e{2:24419} state up:standby-replay seq 1 addr [v2:172.21.15.174:6834/3213155599,v1:172.21.15.174:6835/3213155599] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T00:56:32.067 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:32.067 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:32.067 INFO:teuthology.orchestra.run.smithi169.stdout:Standby daemons: 2024-03-28T00:56:32.067 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:32.067 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.j{-1:14544} state up:standby seq 1 addr [v2:172.21.15.169:6834/3964885254,v1:172.21.15.169:6835/3964885254] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T00:56:32.067 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.d{-1:14556} state up:standby seq 1 addr [v2:172.21.15.169:6836/2130558218,v1:172.21.15.169:6837/2130558218] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T00:56:32.068 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.g{-1:14619} state up:standby seq 1 addr [v2:172.21.15.169:6838/4072876730,v1:172.21.15.169:6839/4072876730] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T00:56:32.068 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.a{-1:14631} state up:standby seq 1 addr [v2:172.21.15.169:6840/2980464895,v1:172.21.15.169:6841/2980464895] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T00:56:32.068 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.k{-1:24389} state up:standby seq 1 addr [v2:172.21.15.174:6832/1840169743,v1:172.21.15.174:6833/1840169743] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T00:56:32.068 INFO:teuthology.orchestra.run.smithi169.stdout:[mds.l{-1:24412} state up:standby seq 1 addr [v2:172.21.15.177:6832/3466292652,v1:172.21.15.177:6833/3466292652] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T00:56:32.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:32.063+0000 7f67224b2700 1 -- 172.21.15.169:0/1219296967 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f67040a2910 msgr2=0x7f67040a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:32.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:32.063+0000 7f67224b2700 1 --2- 172.21.15.169:0/1219296967 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f67040a2910 0x7f67040a4dd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f671c109a90 tx=0x7f670c005fb0 comp rx=0 tx=0).stop 2024-03-28T00:56:32.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:32.063+0000 7f67224b2700 1 -- 172.21.15.169:0/1219296967 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f671c102cd0 msgr2=0x7f671c1a5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:32.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:32.063+0000 7f67224b2700 1 --2- 172.21.15.169:0/1219296967 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f671c102cd0 0x7f671c1a5d80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f671400efc0 tx=0x7f671400c3a0 comp rx=0 tx=0).stop 2024-03-28T00:56:32.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:32.063+0000 7f67224b2700 1 -- 172.21.15.169:0/1219296967 shutdown_connections 2024-03-28T00:56:32.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:32.063+0000 7f67224b2700 1 --2- 172.21.15.169:0/1219296967 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f671c102cd0 0x7f671c1a5d80 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:32.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:32.063+0000 7f67224b2700 1 --2- 172.21.15.169:0/1219296967 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f67040a2910 0x7f67040a4dd0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:32.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:32.063+0000 7f67224b2700 1 --2- 172.21.15.169:0/1219296967 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f671c101130 0x7f671c1a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:32.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:32.063+0000 7f67224b2700 1 --2- 172.21.15.169:0/1219296967 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f671c102330 0x7f671c1a36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:32.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:32.063+0000 7f67224b2700 1 -- 172.21.15.169:0/1219296967 >> 172.21.15.169:0/1219296967 conn(0x7f671c0fc6f0 msgr2=0x7f671c06a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:32.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:32.063+0000 7f67224b2700 1 -- 172.21.15.169:0/1219296967 shutdown_connections 2024-03-28T00:56:32.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:32.063+0000 7f67224b2700 1 -- 172.21.15.169:0/1219296967 wait complete. 2024-03-28T00:56:32.069 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 24 2024-03-28T00:56:32.428 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:32 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1219296967' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T00:56:32.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:32 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1219296967' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T00:56:32.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:32 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1219296967' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T00:56:32.849 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-28T00:56:33.093 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:56:33.475 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:33 smithi169 ceph-mon[32274]: pgmap v350: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T00:56:33.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:33 smithi174 ceph-mon[36060]: pgmap v350: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T00:56:33.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:33 smithi177 ceph-mon[36206]: pgmap v350: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T00:56:34.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:34 smithi169 ceph-mon[32274]: pgmap v351: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T00:56:34.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:34 smithi174 ceph-mon[36060]: pgmap v351: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T00:56:34.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:34 smithi177 ceph-mon[36206]: pgmap v351: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T00:56:34.927 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.919+0000 7f65750ae700 1 -- 172.21.15.169:0/2410554959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65701010c0 msgr2=0x7f65701014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:34.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.919+0000 7f65750ae700 1 --2- 172.21.15.169:0/2410554959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65701010c0 0x7f65701014e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f656800b0d0 tx=0x7f656800b3e0 comp rx=0 tx=0).stop 2024-03-28T00:56:34.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.919+0000 7f65750ae700 1 -- 172.21.15.169:0/2410554959 shutdown_connections 2024-03-28T00:56:34.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.919+0000 7f65750ae700 1 --2- 172.21.15.169:0/2410554959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6570102c60 0x7f6570109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:34.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.919+0000 7f65750ae700 1 --2- 172.21.15.169:0/2410554959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f65701010c0 0x7f65701014e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:34.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.919+0000 7f65750ae700 1 --2- 172.21.15.169:0/2410554959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65701022c0 0x7f6570102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:34.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.919+0000 7f65750ae700 1 -- 172.21.15.169:0/2410554959 >> 172.21.15.169:0/2410554959 conn(0x7f65700fc660 msgr2=0x7f65700feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:34.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.919+0000 7f65750ae700 1 -- 172.21.15.169:0/2410554959 shutdown_connections 2024-03-28T00:56:34.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.919+0000 7f65750ae700 1 -- 172.21.15.169:0/2410554959 wait complete. 2024-03-28T00:56:34.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.920+0000 7f65750ae700 1 Processor -- start 2024-03-28T00:56:34.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.920+0000 7f65750ae700 1 -- start start 2024-03-28T00:56:34.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.920+0000 7f65750ae700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65701010c0 0x7f6570198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:34.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.920+0000 7f65750ae700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65701022c0 0x7f6570198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:34.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.920+0000 7f65750ae700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6570102c60 0x7f657019fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:34.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.920+0000 7f65750ae700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657010ccb0 con 0x7f6570102c60 2024-03-28T00:56:34.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.920+0000 7f65750ae700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657010cb30 con 0x7f65701010c0 2024-03-28T00:56:34.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.920+0000 7f65750ae700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657010ce30 con 0x7f65701022c0 2024-03-28T00:56:34.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.920+0000 7f656ed9d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65701010c0 0x7f6570198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:34.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.920+0000 7f656f59e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6570102c60 0x7f657019fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:34.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.920+0000 7f656e59c700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65701022c0 0x7f6570198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:34.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.920+0000 7f656ed9d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65701010c0 0x7f6570198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:46420/0 (socket says 172.21.15.169:46420) 2024-03-28T00:56:34.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.920+0000 7f656ed9d700 1 -- 172.21.15.169:0/4193647878 learned_addr learned my addr 172.21.15.169:0/4193647878 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:34.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.921+0000 7f656ed9d700 1 -- 172.21.15.169:0/4193647878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65701022c0 msgr2=0x7f6570198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:34.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.921+0000 7f656ed9d700 1 --2- 172.21.15.169:0/4193647878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65701022c0 0x7f6570198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:34.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.921+0000 7f656ed9d700 1 -- 172.21.15.169:0/4193647878 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6570102c60 msgr2=0x7f657019fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:34.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.921+0000 7f656ed9d700 1 --2- 172.21.15.169:0/4193647878 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6570102c60 0x7f657019fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:34.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.921+0000 7f656ed9d700 1 -- 172.21.15.169:0/4193647878 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65701a00b0 con 0x7f65701010c0 2024-03-28T00:56:34.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.921+0000 7f656e59c700 1 --2- 172.21.15.169:0/4193647878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65701022c0 0x7f6570198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:34.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.921+0000 7f656ed9d700 1 --2- 172.21.15.169:0/4193647878 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65701010c0 0x7f6570198450 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6568007a20 tx=0x7f6568007a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:34.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.921+0000 7f6567fff700 1 -- 172.21.15.169:0/4193647878 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6568021020 con 0x7f65701010c0 2024-03-28T00:56:34.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.921+0000 7f6567fff700 1 -- 172.21.15.169:0/4193647878 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6568004c50 con 0x7f65701010c0 2024-03-28T00:56:34.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.921+0000 7f65750ae700 1 -- 172.21.15.169:0/4193647878 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65701034e0 con 0x7f65701010c0 2024-03-28T00:56:34.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.921+0000 7f6567fff700 1 -- 172.21.15.169:0/4193647878 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6568005340 con 0x7f65701010c0 2024-03-28T00:56:34.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.921+0000 7f65750ae700 1 -- 172.21.15.169:0/4193647878 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6570103a20 con 0x7f65701010c0 2024-03-28T00:56:34.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.922+0000 7f65750ae700 1 -- 172.21.15.169:0/4193647878 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6570068e50 con 0x7f65701010c0 2024-03-28T00:56:34.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.923+0000 7f6567fff700 1 -- 172.21.15.169:0/4193647878 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6568024020 con 0x7f65701010c0 2024-03-28T00:56:34.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.923+0000 7f6567fff700 1 --2- 172.21.15.169:0/4193647878 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f655c0a2890 0x7f655c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:34.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.924+0000 7f656e59c700 1 --2- 172.21.15.169:0/4193647878 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f655c0a2890 0x7f655c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:34.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.924+0000 7f6567fff700 1 -- 172.21.15.169:0/4193647878 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f656800f020 con 0x7f65701010c0 2024-03-28T00:56:34.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.924+0000 7f656e59c700 1 --2- 172.21.15.169:0/4193647878 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f655c0a2890 0x7f655c0a4d50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6558011490 tx=0x7f6558009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:34.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:34.927+0000 7f6567fff700 1 -- 172.21.15.169:0/4193647878 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f656809cb70 con 0x7f65701010c0 2024-03-28T00:56:35.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.119+0000 7f65750ae700 1 -- 172.21.15.169:0/4193647878 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f6570103ed0 con 0x7f65701010c0 2024-03-28T00:56:35.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.120+0000 7f6567fff700 1 -- 172.21.15.169:0/4193647878 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v92) v1 ==== 56+0+5626 (secure 0 0 0) 0x7f656809c470 con 0x7f65701010c0 2024-03-28T00:56:35.123 INFO:teuthology.orchestra.run.smithi169.stdout:epoch 92 2024-03-28T00:56:35.123 INFO:teuthology.orchestra.run.smithi169.stdout:fsid 833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T00:56:35.124 INFO:teuthology.orchestra.run.smithi169.stdout:created 2024-03-28T00:46:42.432153+0000 2024-03-28T00:56:35.124 INFO:teuthology.orchestra.run.smithi169.stdout:modified 2024-03-28T00:55:37.555451+0000 2024-03-28T00:56:35.124 INFO:teuthology.orchestra.run.smithi169.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-28T00:56:35.124 INFO:teuthology.orchestra.run.smithi169.stdout:crush_version 27 2024-03-28T00:56:35.124 INFO:teuthology.orchestra.run.smithi169.stdout:full_ratio 0.95 2024-03-28T00:56:35.124 INFO:teuthology.orchestra.run.smithi169.stdout:backfillfull_ratio 0.9 2024-03-28T00:56:35.124 INFO:teuthology.orchestra.run.smithi169.stdout:nearfull_ratio 0.85 2024-03-28T00:56:35.124 INFO:teuthology.orchestra.run.smithi169.stdout:require_min_compat_client luminous 2024-03-28T00:56:35.124 INFO:teuthology.orchestra.run.smithi169.stdout:min_compat_client luminous 2024-03-28T00:56:35.124 INFO:teuthology.orchestra.run.smithi169.stdout:require_osd_release squid 2024-03-28T00:56:35.124 INFO:teuthology.orchestra.run.smithi169.stdout:stretch_mode_enabled false 2024-03-28T00:56:35.124 INFO:teuthology.orchestra.run.smithi169.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-28T00:56:35.124 INFO:teuthology.orchestra.run.smithi169.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-28T00:56:35.125 INFO:teuthology.orchestra.run.smithi169.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-28T00:56:35.125 INFO:teuthology.orchestra.run.smithi169.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 90 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-28T00:56:35.125 INFO:teuthology.orchestra.run.smithi169.stdout:max_osd 12 2024-03-28T00:56:35.125 INFO:teuthology.orchestra.run.smithi169.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.169:6802/1362598737,v1:172.21.15.169:6803/1362598737] [v2:172.21.15.169:6804/1362598737,v1:172.21.15.169:6805/1362598737] exists,up 37a188e7-9de4-4f22-a2d2-f524c64e1923 2024-03-28T00:56:35.125 INFO:teuthology.orchestra.run.smithi169.stdout:osd.1 up in weight 1 up_from 13 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6800/3333696361,v1:172.21.15.174:6801/3333696361] [v2:172.21.15.174:6802/3333696361,v1:172.21.15.174:6803/3333696361] exists,up 269d9b9a-b3cd-4dda-bc9e-7d980cad5d12 2024-03-28T00:56:35.125 INFO:teuthology.orchestra.run.smithi169.stdout:osd.2 up in weight 1 up_from 18 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6800/1495865602,v1:172.21.15.177:6801/1495865602] [v2:172.21.15.177:6802/1495865602,v1:172.21.15.177:6803/1495865602] exists,up 4a6108e3-e6eb-489b-8f90-1caba1b3da34 2024-03-28T00:56:35.125 INFO:teuthology.orchestra.run.smithi169.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.169:6810/149376866,v1:172.21.15.169:6811/149376866] [v2:172.21.15.169:6812/149376866,v1:172.21.15.169:6813/149376866] exists,up a7eae99d-476b-43c0-968f-8b5bf6d07222 2024-03-28T00:56:35.125 INFO:teuthology.orchestra.run.smithi169.stdout:osd.4 up in weight 1 up_from 33 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6808/701854773,v1:172.21.15.174:6809/701854773] [v2:172.21.15.174:6810/701854773,v1:172.21.15.174:6811/701854773] exists,up 73da9d2f-f108-40a2-a13d-639b240706f2 2024-03-28T00:56:35.125 INFO:teuthology.orchestra.run.smithi169.stdout:osd.5 up in weight 1 up_from 38 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6808/3583242703,v1:172.21.15.177:6809/3583242703] [v2:172.21.15.177:6810/3583242703,v1:172.21.15.177:6811/3583242703] exists,up dd80a3c2-8ec1-495c-a8cb-b662242df637 2024-03-28T00:56:35.125 INFO:teuthology.orchestra.run.smithi169.stdout:osd.6 up in weight 1 up_from 46 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.169:6818/783670252,v1:172.21.15.169:6819/783670252] [v2:172.21.15.169:6820/783670252,v1:172.21.15.169:6821/783670252] exists,up 7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4 2024-03-28T00:56:35.125 INFO:teuthology.orchestra.run.smithi169.stdout:osd.7 up in weight 1 up_from 51 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6816/2274626586,v1:172.21.15.174:6817/2274626586] [v2:172.21.15.174:6818/2274626586,v1:172.21.15.174:6819/2274626586] exists,up 5378575b-122b-4369-9fc8-1230aee2dbd1 2024-03-28T00:56:35.125 INFO:teuthology.orchestra.run.smithi169.stdout:osd.8 up in weight 1 up_from 59 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6816/4108616824,v1:172.21.15.177:6817/4108616824] [v2:172.21.15.177:6818/4108616824,v1:172.21.15.177:6819/4108616824] exists,up 874c0869-f1fe-4927-9549-61d6e46c498a 2024-03-28T00:56:35.125 INFO:teuthology.orchestra.run.smithi169.stdout:osd.9 up in weight 1 up_from 67 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.169:6826/3660250290,v1:172.21.15.169:6827/3660250290] [v2:172.21.15.169:6828/3660250290,v1:172.21.15.169:6829/3660250290] exists,up e6a429af-798a-48e3-8368-ad9338daa9b8 2024-03-28T00:56:35.126 INFO:teuthology.orchestra.run.smithi169.stdout:osd.10 up in weight 1 up_from 72 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6824/3371468956,v1:172.21.15.174:6825/3371468956] [v2:172.21.15.174:6826/3371468956,v1:172.21.15.174:6827/3371468956] exists,up ba098e93-e8fd-4dca-b2ff-1722425e2524 2024-03-28T00:56:35.126 INFO:teuthology.orchestra.run.smithi169.stdout:osd.11 up in weight 1 up_from 80 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6824/2109004868,v1:172.21.15.177:6825/2109004868] [v2:172.21.15.177:6826/2109004868,v1:172.21.15.177:6827/2109004868] exists,up b1e65122-4e70-4022-933a-c9bf2f0f2e41 2024-03-28T00:56:35.126 INFO:teuthology.orchestra.run.smithi169.stdout:pg_upmap_items 2.8 [2,11] 2024-03-28T00:56:35.126 INFO:teuthology.orchestra.run.smithi169.stdout:pg_upmap_items 2.f [8,11] 2024-03-28T00:56:35.126 INFO:teuthology.orchestra.run.smithi169.stdout:pg_upmap_items 3.e [4,6] 2024-03-28T00:56:35.126 INFO:teuthology.orchestra.run.smithi169.stdout:pg_upmap_items 3.21 [4,8] 2024-03-28T00:56:35.126 INFO:teuthology.orchestra.run.smithi169.stdout:pg_upmap_items 3.2c [4,9] 2024-03-28T00:56:35.126 INFO:teuthology.orchestra.run.smithi169.stdout:pg_upmap_items 3.39 [5,8] 2024-03-28T00:56:35.126 INFO:teuthology.orchestra.run.smithi169.stdout:pg_upmap_items 4.2 [9,1] 2024-03-28T00:56:35.126 INFO:teuthology.orchestra.run.smithi169.stdout:pg_upmap_items 4.e [5,8] 2024-03-28T00:56:35.126 INFO:teuthology.orchestra.run.smithi169.stdout:pg_upmap_items 4.10 [9,8] 2024-03-28T00:56:35.126 INFO:teuthology.orchestra.run.smithi169.stdout:pg_upmap_items 4.14 [9,8] 2024-03-28T00:56:35.126 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:0/2299683991 expires 2024-03-29T00:47:35.997770+0000 2024-03-28T00:56:35.126 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:6801/3995234407 expires 2024-03-29T00:47:35.997770+0000 2024-03-28T00:56:35.127 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:0/3955974323 expires 2024-03-29T00:47:10.143139+0000 2024-03-28T00:56:35.127 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:6800/3995234407 expires 2024-03-29T00:47:35.997770+0000 2024-03-28T00:56:35.127 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:0/1567973250 expires 2024-03-29T00:47:10.143139+0000 2024-03-28T00:56:35.127 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:0/1799473331 expires 2024-03-29T00:47:10.143139+0000 2024-03-28T00:56:35.127 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:0/1949169969 expires 2024-03-29T00:47:35.997770+0000 2024-03-28T00:56:35.127 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:6800/311965133 expires 2024-03-29T00:47:10.143139+0000 2024-03-28T00:56:35.127 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:6801/311965133 expires 2024-03-29T00:47:10.143139+0000 2024-03-28T00:56:35.127 INFO:teuthology.orchestra.run.smithi169.stdout:blocklist 172.21.15.169:0/2928378386 expires 2024-03-29T00:47:35.997770+0000 2024-03-28T00:56:35.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.123+0000 7f65750ae700 1 -- 172.21.15.169:0/4193647878 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f655c0a2890 msgr2=0x7f655c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:35.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.123+0000 7f65750ae700 1 --2- 172.21.15.169:0/4193647878 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f655c0a2890 0x7f655c0a4d50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6558011490 tx=0x7f6558009260 comp rx=0 tx=0).stop 2024-03-28T00:56:35.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.123+0000 7f65750ae700 1 -- 172.21.15.169:0/4193647878 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65701010c0 msgr2=0x7f6570198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:35.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.123+0000 7f65750ae700 1 --2- 172.21.15.169:0/4193647878 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65701010c0 0x7f6570198450 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6568007a20 tx=0x7f6568007a50 comp rx=0 tx=0).stop 2024-03-28T00:56:35.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.123+0000 7f65750ae700 1 -- 172.21.15.169:0/4193647878 shutdown_connections 2024-03-28T00:56:35.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.123+0000 7f65750ae700 1 --2- 172.21.15.169:0/4193647878 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6570102c60 0x7f657019fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:35.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.123+0000 7f65750ae700 1 --2- 172.21.15.169:0/4193647878 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65701010c0 0x7f6570198450 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:35.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.123+0000 7f65750ae700 1 --2- 172.21.15.169:0/4193647878 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f655c0a2890 0x7f655c0a4d50 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:35.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.123+0000 7f65750ae700 1 --2- 172.21.15.169:0/4193647878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65701022c0 0x7f6570198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:35.129 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.123+0000 7f65750ae700 1 -- 172.21.15.169:0/4193647878 >> 172.21.15.169:0/4193647878 conn(0x7f65700fc660 msgr2=0x7f65700fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:35.129 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.123+0000 7f65750ae700 1 -- 172.21.15.169:0/4193647878 shutdown_connections 2024-03-28T00:56:35.129 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.123+0000 7f65750ae700 1 -- 172.21.15.169:0/4193647878 wait complete. 2024-03-28T00:56:35.498 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:35 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4193647878' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T00:56:35.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:35 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/4193647878' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T00:56:35.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:35 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/4193647878' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T00:56:35.812 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-28T00:56:35.823 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T00:56:35.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.928+0000 7fcc05b1b640 1 -- 172.21.15.169:0/1422228813 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc001028d0 msgr2=0x7fcc00102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:35.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.928+0000 7fcc05b1b640 1 --2- 172.21.15.169:0/1422228813 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc001028d0 0x7fcc00102d50 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009900 tx=0x7fcbe802bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:35.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.928+0000 7fcc05b1b640 1 -- 172.21.15.169:0/1422228813 shutdown_connections 2024-03-28T00:56:35.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.928+0000 7fcc05b1b640 1 --2- 172.21.15.169:0/1422228813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc00103290 0x7fcc00109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:35.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.928+0000 7fcc05b1b640 1 --2- 172.21.15.169:0/1422228813 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc001028d0 0x7fcc00102d50 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:35.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.928+0000 7fcc05b1b640 1 --2- 172.21.15.169:0/1422228813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc001016e0 0x7fcc00101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:35.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.928+0000 7fcc05b1b640 1 -- 172.21.15.169:0/1422228813 >> 172.21.15.169:0/1422228813 conn(0x7fcc000fcc50 msgr2=0x7fcc000ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:35.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.928+0000 7fcc05b1b640 1 -- 172.21.15.169:0/1422228813 shutdown_connections 2024-03-28T00:56:35.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.928+0000 7fcc05b1b640 1 -- 172.21.15.169:0/1422228813 wait complete. 2024-03-28T00:56:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.928+0000 7fcc05b1b640 1 Processor -- start 2024-03-28T00:56:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.928+0000 7fcc05b1b640 1 -- start start 2024-03-28T00:56:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcc05b1b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc001016e0 0x7fcc001979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcc05b1b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc001028d0 0x7fcc00197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcc05b1b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc00103290 0x7fcc0019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcc05b1b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0010d360 con 0x7fcc00103290 2024-03-28T00:56:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcc05b1b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0010d1e0 con 0x7fcc001016e0 2024-03-28T00:56:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcc05b1b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0010d4e0 con 0x7fcc001028d0 2024-03-28T00:56:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcbfffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc00103290 0x7fcc0019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcbfffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc00103290 0x7fcc0019ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47470/0 (socket says 172.21.15.169:47470) 2024-03-28T00:56:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcbfffff640 1 -- 172.21.15.169:0/4135777434 learned_addr learned my addr 172.21.15.169:0/4135777434 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcbff7fe640 1 --2- 172.21.15.169:0/4135777434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc001016e0 0x7fcc001979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcbf7fff640 1 --2- 172.21.15.169:0/4135777434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc001028d0 0x7fcc00197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:35.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcbfffff640 1 -- 172.21.15.169:0/4135777434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc001028d0 msgr2=0x7fcc00197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcbfffff640 1 --2- 172.21.15.169:0/4135777434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc001028d0 0x7fcc00197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcbfffff640 1 -- 172.21.15.169:0/4135777434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc001016e0 msgr2=0x7fcc001979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcbfffff640 1 --2- 172.21.15.169:0/4135777434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc001016e0 0x7fcc001979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcbfffff640 1 -- 172.21.15.169:0/4135777434 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0019f5f0 con 0x7fcc00103290 2024-03-28T00:56:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcbfffff640 1 --2- 172.21.15.169:0/4135777434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc00103290 0x7fcc0019ef80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800eb10 tx=0x7fcbf800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcbfd7fa640 1 -- 172.21.15.169:0/4135777434 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8004b20 con 0x7fcc00103290 2024-03-28T00:56:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcbfd7fa640 1 -- 172.21.15.169:0/4135777434 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbf8004cc0 con 0x7fcc00103290 2024-03-28T00:56:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcc05b1b640 1 -- 172.21.15.169:0/4135777434 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0019f8e0 con 0x7fcc00103290 2024-03-28T00:56:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcbfd7fa640 1 -- 172.21.15.169:0/4135777434 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8010760 con 0x7fcc00103290 2024-03-28T00:56:35.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.929+0000 7fcc05b1b640 1 -- 172.21.15.169:0/4135777434 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc0019fe00 con 0x7fcc00103290 2024-03-28T00:56:35.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.930+0000 7fcc05b1b640 1 -- 172.21.15.169:0/4135777434 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc00108a00 con 0x7fcc00103290 2024-03-28T00:56:35.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.931+0000 7fcbfd7fa640 1 -- 172.21.15.169:0/4135777434 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fcbf8005020 con 0x7fcc00103290 2024-03-28T00:56:35.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.932+0000 7fcbfd7fa640 1 --2- 172.21.15.169:0/4135777434 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcbec0a2800 0x7fcbec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:35.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.932+0000 7fcbff7fe640 1 --2- 172.21.15.169:0/4135777434 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcbec0a2800 0x7fcbec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:35.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.932+0000 7fcbfd7fa640 1 -- 172.21.15.169:0/4135777434 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fcbf80cf4a0 con 0x7fcc00103290 2024-03-28T00:56:35.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.932+0000 7fcbff7fe640 1 --2- 172.21.15.169:0/4135777434 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcbec0a2800 0x7fcbec0a4cc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcc00102730 tx=0x7fcbf0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:35.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:35.937+0000 7fcbfd7fa640 1 -- 172.21.15.169:0/4135777434 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcbf809c1e0 con 0x7fcc00103290 2024-03-28T00:56:36.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.123+0000 7fcc05b1b640 1 -- 172.21.15.169:0/4135777434 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc0002d0c0 con 0x7fcc00103290 2024-03-28T00:56:36.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.125+0000 7fcbfd7fa640 1 -- 172.21.15.169:0/4135777434 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+24711 (secure 0 0 0) 0x7fcbf809bae0 con 0x7fcc00103290 2024-03-28T00:56:36.128 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:36.128 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":92,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:55:37.555451+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:56:36.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.127+0000 7fcc05b1b640 1 -- 172.21.15.169:0/4135777434 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcbec0a2800 msgr2=0x7fcbec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:36.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.127+0000 7fcc05b1b640 1 --2- 172.21.15.169:0/4135777434 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcbec0a2800 0x7fcbec0a4cc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcc00102730 tx=0x7fcbf0005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:36.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.127+0000 7fcc05b1b640 1 -- 172.21.15.169:0/4135777434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc00103290 msgr2=0x7fcc0019ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:36.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.127+0000 7fcc05b1b640 1 --2- 172.21.15.169:0/4135777434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc00103290 0x7fcc0019ef80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800eb10 tx=0x7fcbf800c2d0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.127+0000 7fcc05b1b640 1 -- 172.21.15.169:0/4135777434 shutdown_connections 2024-03-28T00:56:36.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.127+0000 7fcc05b1b640 1 --2- 172.21.15.169:0/4135777434 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcbec0a2800 0x7fcbec0a4cc0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.127+0000 7fcc05b1b640 1 --2- 172.21.15.169:0/4135777434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcc00103290 0x7fcc0019ef80 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.127+0000 7fcc05b1b640 1 --2- 172.21.15.169:0/4135777434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc001028d0 0x7fcc00197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.127+0000 7fcc05b1b640 1 --2- 172.21.15.169:0/4135777434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc001016e0 0x7fcc001979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.127+0000 7fcc05b1b640 1 -- 172.21.15.169:0/4135777434 >> 172.21.15.169:0/4135777434 conn(0x7fcc000fcc50 msgr2=0x7fcc00105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:36.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.127+0000 7fcc05b1b640 1 -- 172.21.15.169:0/4135777434 shutdown_connections 2024-03-28T00:56:36.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.127+0000 7fcc05b1b640 1 -- 172.21.15.169:0/4135777434 wait complete. 2024-03-28T00:56:36.142 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T00:49:56.339525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T00:54:31.182562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T00:54:31.780429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T00:54:35.035724+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T00:56:36.143 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T00:56:36.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.245+0000 7f3b2efbc640 1 -- 172.21.15.169:0/3149757605 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b28101790 msgr2=0x7f3b28101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:36.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.245+0000 7f3b2efbc640 1 --2- 172.21.15.169:0/3149757605 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b28101790 0x7f3b28101c10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009900 tx=0x7f3b1c02bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:36.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.245+0000 7f3b2efbc640 1 -- 172.21.15.169:0/3149757605 shutdown_connections 2024-03-28T00:56:36.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.245+0000 7f3b2efbc640 1 --2- 172.21.15.169:0/3149757605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b28102150 0x7f3b281089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.245+0000 7f3b2efbc640 1 --2- 172.21.15.169:0/3149757605 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b28101790 0x7f3b28101c10 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.245+0000 7f3b2efbc640 1 --2- 172.21.15.169:0/3149757605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b281005a0 0x7f3b281009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.245+0000 7f3b2efbc640 1 -- 172.21.15.169:0/3149757605 >> 172.21.15.169:0/3149757605 conn(0x7f3b280fbb30 msgr2=0x7f3b280fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:36.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.245+0000 7f3b2efbc640 1 -- 172.21.15.169:0/3149757605 shutdown_connections 2024-03-28T00:56:36.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.245+0000 7f3b2efbc640 1 -- 172.21.15.169:0/3149757605 wait complete. 2024-03-28T00:56:36.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.246+0000 7f3b2efbc640 1 Processor -- start 2024-03-28T00:56:36.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.246+0000 7f3b2efbc640 1 -- start start 2024-03-28T00:56:36.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.246+0000 7f3b2efbc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b281005a0 0x7f3b28197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:36.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.246+0000 7f3b2efbc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b28101790 0x7f3b28197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:36.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.246+0000 7f3b2efbc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b28102150 0x7f3b2819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:36.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.246+0000 7f3b2efbc640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2810c230 con 0x7f3b281005a0 2024-03-28T00:56:36.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.246+0000 7f3b2efbc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2810c0b0 con 0x7f3b28102150 2024-03-28T00:56:36.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.246+0000 7f3b2efbc640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2810c3b0 con 0x7f3b28101790 2024-03-28T00:56:36.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.246+0000 7f3b2cd58640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b281005a0 0x7f3b28197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:36.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.246+0000 7f3b2cd58640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b281005a0 0x7f3b28197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47492/0 (socket says 172.21.15.169:47492) 2024-03-28T00:56:36.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.246+0000 7f3b27fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b28101790 0x7f3b28197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:36.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.246+0000 7f3b2cd58640 1 -- 172.21.15.169:0/9556015 learned_addr learned my addr 172.21.15.169:0/9556015 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:36.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.246+0000 7f3b2d559640 1 --2- 172.21.15.169:0/9556015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b28102150 0x7f3b2819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:36.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.247+0000 7f3b2cd58640 1 -- 172.21.15.169:0/9556015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b28101790 msgr2=0x7f3b28197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:36.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.247+0000 7f3b2cd58640 1 --2- 172.21.15.169:0/9556015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b28101790 0x7f3b28197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.247+0000 7f3b2cd58640 1 -- 172.21.15.169:0/9556015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b28102150 msgr2=0x7f3b2819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:36.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.247+0000 7f3b2cd58640 1 --2- 172.21.15.169:0/9556015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b28102150 0x7f3b2819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.247+0000 7f3b2cd58640 1 -- 172.21.15.169:0/9556015 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2819f620 con 0x7f3b281005a0 2024-03-28T00:56:36.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.247+0000 7f3b2cd58640 1 --2- 172.21.15.169:0/9556015 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b281005a0 0x7f3b28197a00 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800e9c0 tx=0x7f3b1800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:36.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.247+0000 7f3b25ffb640 1 -- 172.21.15.169:0/9556015 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1800cd80 con 0x7f3b281005a0 2024-03-28T00:56:36.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.247+0000 7f3b25ffb640 1 -- 172.21.15.169:0/9556015 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b18010910 con 0x7f3b281005a0 2024-03-28T00:56:36.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.247+0000 7f3b2efbc640 1 -- 172.21.15.169:0/9556015 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b2819f910 con 0x7f3b281005a0 2024-03-28T00:56:36.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.247+0000 7f3b25ffb640 1 -- 172.21.15.169:0/9556015 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b18018950 con 0x7f3b281005a0 2024-03-28T00:56:36.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.247+0000 7f3b2efbc640 1 -- 172.21.15.169:0/9556015 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2819fe30 con 0x7f3b281005a0 2024-03-28T00:56:36.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.248+0000 7f3b2efbc640 1 -- 172.21.15.169:0/9556015 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b281a0000 con 0x7f3b281005a0 2024-03-28T00:56:36.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.249+0000 7f3b25ffb640 1 -- 172.21.15.169:0/9556015 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3b1801f020 con 0x7f3b281005a0 2024-03-28T00:56:36.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.250+0000 7f3b25ffb640 1 --2- 172.21.15.169:0/9556015 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3b100a2800 0x7f3b100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:36.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.250+0000 7f3b27fff640 1 --2- 172.21.15.169:0/9556015 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3b100a2800 0x7f3b100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:36.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.250+0000 7f3b25ffb640 1 -- 172.21.15.169:0/9556015 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f3b180cee80 con 0x7f3b281005a0 2024-03-28T00:56:36.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.250+0000 7f3b27fff640 1 --2- 172.21.15.169:0/9556015 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3b100a2800 0x7f3b100a4cc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c02c7b0 tx=0x7f3b1c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:36.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.254+0000 7f3b25ffb640 1 -- 172.21.15.169:0/9556015 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3b1809bbc0 con 0x7f3b281005a0 2024-03-28T00:56:36.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.431+0000 7f3b2efbc640 1 -- 172.21.15.169:0/9556015 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3b281a06f0 con 0x7f3b281005a0 2024-03-28T00:56:36.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.431+0000 7f3b25ffb640 1 -- 172.21.15.169:0/9556015 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3b1809b4c0 con 0x7f3b281005a0 2024-03-28T00:56:36.434 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 1 2024-03-28T00:56:36.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.433+0000 7f3b2efbc640 1 -- 172.21.15.169:0/9556015 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3b100a2800 msgr2=0x7f3b100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:36.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.433+0000 7f3b2efbc640 1 --2- 172.21.15.169:0/9556015 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3b100a2800 0x7f3b100a4cc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c02c7b0 tx=0x7f3b1c005fd0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.433+0000 7f3b2efbc640 1 -- 172.21.15.169:0/9556015 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b281005a0 msgr2=0x7f3b28197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:36.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.433+0000 7f3b2efbc640 1 --2- 172.21.15.169:0/9556015 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b281005a0 0x7f3b28197a00 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800e9c0 tx=0x7f3b1800edc0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.433+0000 7f3b2efbc640 1 -- 172.21.15.169:0/9556015 shutdown_connections 2024-03-28T00:56:36.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.433+0000 7f3b2efbc640 1 --2- 172.21.15.169:0/9556015 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3b100a2800 0x7f3b100a4cc0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.433+0000 7f3b2efbc640 1 --2- 172.21.15.169:0/9556015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b28102150 0x7f3b2819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.433+0000 7f3b2efbc640 1 --2- 172.21.15.169:0/9556015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b28101790 0x7f3b28197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.433+0000 7f3b2efbc640 1 --2- 172.21.15.169:0/9556015 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b281005a0 0x7f3b28197a00 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.433+0000 7f3b2efbc640 1 -- 172.21.15.169:0/9556015 >> 172.21.15.169:0/9556015 conn(0x7f3b280fbb30 msgr2=0x7f3b28068800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:36.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.433+0000 7f3b2efbc640 1 -- 172.21.15.169:0/9556015 shutdown_connections 2024-03-28T00:56:36.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.433+0000 7f3b2efbc640 1 -- 172.21.15.169:0/9556015 wait complete. 2024-03-28T00:56:36.446 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:36 smithi169 ceph-mon[32274]: pgmap v352: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T00:56:36.446 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:36 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4135777434' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:36.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T00:56:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.550+0000 7ff716c4b640 1 -- 172.21.15.169:0/2725710161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7101005c0 msgr2=0x7ff7101009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.550+0000 7ff716c4b640 1 --2- 172.21.15.169:0/2725710161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7101005c0 0x7ff7101009e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff700009900 tx=0x7ff70002bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.550+0000 7ff716c4b640 1 -- 172.21.15.169:0/2725710161 shutdown_connections 2024-03-28T00:56:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.550+0000 7ff716c4b640 1 --2- 172.21.15.169:0/2725710161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff710102170 0x7ff710108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.550+0000 7ff716c4b640 1 --2- 172.21.15.169:0/2725710161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7101017b0 0x7ff710101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.550+0000 7ff716c4b640 1 --2- 172.21.15.169:0/2725710161 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7101005c0 0x7ff7101009e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.550+0000 7ff716c4b640 1 -- 172.21.15.169:0/2725710161 >> 172.21.15.169:0/2725710161 conn(0x7ff7100fbb30 msgr2=0x7ff7100fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.550+0000 7ff716c4b640 1 -- 172.21.15.169:0/2725710161 shutdown_connections 2024-03-28T00:56:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.551+0000 7ff716c4b640 1 -- 172.21.15.169:0/2725710161 wait complete. 2024-03-28T00:56:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.551+0000 7ff716c4b640 1 Processor -- start 2024-03-28T00:56:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.551+0000 7ff716c4b640 1 -- start start 2024-03-28T00:56:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.551+0000 7ff716c4b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7101005c0 0x7ff71006a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.551+0000 7ff716c4b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7101017b0 0x7ff71019c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.551+0000 7ff716c4b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff710102170 0x7ff71019ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.551+0000 7ff716c4b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71010c280 con 0x7ff7101017b0 2024-03-28T00:56:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.551+0000 7ff716c4b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71010c100 con 0x7ff7101005c0 2024-03-28T00:56:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.551+0000 7ff716c4b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71010c400 con 0x7ff710102170 2024-03-28T00:56:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.551+0000 7ff70ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7101017b0 0x7ff71019c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.551+0000 7ff70ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7101017b0 0x7ff71019c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:50920/0 (socket says 172.21.15.169:50920) 2024-03-28T00:56:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.551+0000 7ff70ffff640 1 -- 172.21.15.169:0/265093194 learned_addr learned my addr 172.21.15.169:0/265093194 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.551+0000 7ff7149e7640 1 --2- 172.21.15.169:0/265093194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7101005c0 0x7ff71006a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.551+0000 7ff7151e8640 1 --2- 172.21.15.169:0/265093194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff710102170 0x7ff71019ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.552+0000 7ff70ffff640 1 -- 172.21.15.169:0/265093194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff710102170 msgr2=0x7ff71019ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.552+0000 7ff70ffff640 1 --2- 172.21.15.169:0/265093194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff710102170 0x7ff71019ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.552+0000 7ff70ffff640 1 -- 172.21.15.169:0/265093194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7101005c0 msgr2=0x7ff71006a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.552+0000 7ff70ffff640 1 --2- 172.21.15.169:0/265093194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7101005c0 0x7ff71006a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.552+0000 7ff70ffff640 1 -- 172.21.15.169:0/265093194 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71019f6c0 con 0x7ff7101017b0 2024-03-28T00:56:36.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.552+0000 7ff70ffff640 1 --2- 172.21.15.169:0/265093194 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7101017b0 0x7ff71019c7c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff70400e9f0 tx=0x7ff70400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:36.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.552+0000 7ff70dffb640 1 -- 172.21.15.169:0/265093194 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70400cd80 con 0x7ff7101017b0 2024-03-28T00:56:36.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.552+0000 7ff70dffb640 1 -- 172.21.15.169:0/265093194 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff704010910 con 0x7ff7101017b0 2024-03-28T00:56:36.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.552+0000 7ff716c4b640 1 -- 172.21.15.169:0/265093194 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff71019f9b0 con 0x7ff7101017b0 2024-03-28T00:56:36.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.552+0000 7ff70dffb640 1 -- 172.21.15.169:0/265093194 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff704018990 con 0x7ff7101017b0 2024-03-28T00:56:36.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.552+0000 7ff716c4b640 1 -- 172.21.15.169:0/265093194 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff71019ff00 con 0x7ff7101017b0 2024-03-28T00:56:36.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.553+0000 7ff716c4b640 1 -- 172.21.15.169:0/265093194 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff710107a00 con 0x7ff7101017b0 2024-03-28T00:56:36.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.554+0000 7ff70dffb640 1 -- 172.21.15.169:0/265093194 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff704010ac0 con 0x7ff7101017b0 2024-03-28T00:56:36.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.555+0000 7ff70dffb640 1 --2- 172.21.15.169:0/265093194 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff6f80a2800 0x7ff6f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:36.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.555+0000 7ff7149e7640 1 --2- 172.21.15.169:0/265093194 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff6f80a2800 0x7ff6f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:36.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.555+0000 7ff70dffb640 1 -- 172.21.15.169:0/265093194 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7ff7040cf4e0 con 0x7ff7101017b0 2024-03-28T00:56:36.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.555+0000 7ff7149e7640 1 --2- 172.21.15.169:0/265093194 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff6f80a2800 0x7ff6f80a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff70002c240 tx=0x7ff70002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:36.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.558+0000 7ff70dffb640 1 -- 172.21.15.169:0/265093194 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff70409bbd0 con 0x7ff7101017b0 2024-03-28T00:56:36.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:36 smithi174 ceph-mon[36060]: pgmap v352: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T00:56:36.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:36 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/4135777434' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:36.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:36 smithi177 ceph-mon[36206]: pgmap v352: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T00:56:36.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:36 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/4135777434' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:36.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.734+0000 7ff716c4b640 1 -- 172.21.15.169:0/265093194 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff71002d200 con 0x7ff7101017b0 2024-03-28T00:56:36.738 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.735+0000 7ff70dffb640 1 -- 172.21.15.169:0/265093194 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff70409fc70 con 0x7ff7101017b0 2024-03-28T00:56:36.738 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T00:56:36.740 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.737+0000 7ff716c4b640 1 -- 172.21.15.169:0/265093194 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff6f80a2800 msgr2=0x7ff6f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:36.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.737+0000 7ff716c4b640 1 --2- 172.21.15.169:0/265093194 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff6f80a2800 0x7ff6f80a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff70002c240 tx=0x7ff70002df90 comp rx=0 tx=0).stop 2024-03-28T00:56:36.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.737+0000 7ff716c4b640 1 -- 172.21.15.169:0/265093194 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7101017b0 msgr2=0x7ff71019c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:36.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.737+0000 7ff716c4b640 1 --2- 172.21.15.169:0/265093194 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7101017b0 0x7ff71019c7c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff70400e9f0 tx=0x7ff70400eeb0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.737+0000 7ff716c4b640 1 -- 172.21.15.169:0/265093194 shutdown_connections 2024-03-28T00:56:36.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.737+0000 7ff716c4b640 1 --2- 172.21.15.169:0/265093194 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff6f80a2800 0x7ff6f80a4cc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.737+0000 7ff716c4b640 1 --2- 172.21.15.169:0/265093194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff710102170 0x7ff71019ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.737+0000 7ff716c4b640 1 --2- 172.21.15.169:0/265093194 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7101017b0 0x7ff71019c7c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.737+0000 7ff716c4b640 1 --2- 172.21.15.169:0/265093194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7101005c0 0x7ff71006a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.737+0000 7ff716c4b640 1 -- 172.21.15.169:0/265093194 >> 172.21.15.169:0/265093194 conn(0x7ff7100fbb30 msgr2=0x7ff710104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:36.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.737+0000 7ff716c4b640 1 -- 172.21.15.169:0/265093194 shutdown_connections 2024-03-28T00:56:36.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.737+0000 7ff716c4b640 1 -- 172.21.15.169:0/265093194 wait complete. 2024-03-28T00:56:36.754 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T00:56:36.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.858+0000 7ff7f655b640 1 -- 172.21.15.169:0/954664124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7f0104ab0 msgr2=0x7ff7f0106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:36.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.858+0000 7ff7f655b640 1 --2- 172.21.15.169:0/954664124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7f0104ab0 0x7ff7f0106ea0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4009900 tx=0x7ff7e402bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:36.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.858+0000 7ff7f655b640 1 -- 172.21.15.169:0/954664124 shutdown_connections 2024-03-28T00:56:36.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.858+0000 7ff7f655b640 1 --2- 172.21.15.169:0/954664124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7f01073e0 0x7ff7f01097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.858+0000 7ff7f655b640 1 --2- 172.21.15.169:0/954664124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7f0104ab0 0x7ff7f0106ea0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.858+0000 7ff7f655b640 1 --2- 172.21.15.169:0/954664124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7f0102180 0x7ff7f0104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.858+0000 7ff7f655b640 1 -- 172.21.15.169:0/954664124 >> 172.21.15.169:0/954664124 conn(0x7ff7f00fbb30 msgr2=0x7ff7f00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:36.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.858+0000 7ff7f655b640 1 -- 172.21.15.169:0/954664124 shutdown_connections 2024-03-28T00:56:36.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.858+0000 7ff7f655b640 1 -- 172.21.15.169:0/954664124 wait complete. 2024-03-28T00:56:36.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.859+0000 7ff7f655b640 1 Processor -- start 2024-03-28T00:56:36.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.859+0000 7ff7f655b640 1 -- start start 2024-03-28T00:56:36.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.859+0000 7ff7f655b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7f0102180 0x7ff7f0197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:36.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.859+0000 7ff7f655b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7f0104ab0 0x7ff7f0197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:36.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.859+0000 7ff7f655b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7f01073e0 0x7ff7f019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:36.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.859+0000 7ff7f655b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f010d300 con 0x7ff7f0102180 2024-03-28T00:56:36.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.859+0000 7ff7f655b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f010d180 con 0x7ff7f01073e0 2024-03-28T00:56:36.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.859+0000 7ff7f655b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f010d480 con 0x7ff7f0104ab0 2024-03-28T00:56:36.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.859+0000 7ff7effff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7f0102180 0x7ff7f0197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:36.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.859+0000 7ff7effff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7f0102180 0x7ff7f0197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:50930/0 (socket says 172.21.15.169:50930) 2024-03-28T00:56:36.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.859+0000 7ff7effff640 1 -- 172.21.15.169:0/386955007 learned_addr learned my addr 172.21.15.169:0/386955007 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:36.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.859+0000 7ff7ef7fe640 1 --2- 172.21.15.169:0/386955007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7f0104ab0 0x7ff7f0197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:36.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.859+0000 7ff7f4af8640 1 --2- 172.21.15.169:0/386955007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7f01073e0 0x7ff7f019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:36.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.860+0000 7ff7effff640 1 -- 172.21.15.169:0/386955007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7f0104ab0 msgr2=0x7ff7f0197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:36.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.860+0000 7ff7effff640 1 --2- 172.21.15.169:0/386955007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7f0104ab0 0x7ff7f0197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.860+0000 7ff7effff640 1 -- 172.21.15.169:0/386955007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7f01073e0 msgr2=0x7ff7f019efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:36.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.860+0000 7ff7effff640 1 --2- 172.21.15.169:0/386955007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7f01073e0 0x7ff7f019efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:36.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.860+0000 7ff7effff640 1 -- 172.21.15.169:0/386955007 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f019f620 con 0x7ff7f0102180 2024-03-28T00:56:36.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.860+0000 7ff7effff640 1 --2- 172.21.15.169:0/386955007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7f0102180 0x7ff7f0197a00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000e9f0 tx=0x7ff7e000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:36.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.860+0000 7ff7ed7fa640 1 -- 172.21.15.169:0/386955007 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e000cd80 con 0x7ff7f0102180 2024-03-28T00:56:36.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.860+0000 7ff7f655b640 1 -- 172.21.15.169:0/386955007 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7f019f910 con 0x7ff7f0102180 2024-03-28T00:56:36.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.860+0000 7ff7ed7fa640 1 -- 172.21.15.169:0/386955007 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff7e0010910 con 0x7ff7f0102180 2024-03-28T00:56:36.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.860+0000 7ff7ed7fa640 1 -- 172.21.15.169:0/386955007 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e00188f0 con 0x7ff7f0102180 2024-03-28T00:56:36.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.860+0000 7ff7f655b640 1 -- 172.21.15.169:0/386955007 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f019fe80 con 0x7ff7f0102180 2024-03-28T00:56:36.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.861+0000 7ff7f655b640 1 -- 172.21.15.169:0/386955007 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7f00fd620 con 0x7ff7f0102180 2024-03-28T00:56:36.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.862+0000 7ff7ed7fa640 1 -- 172.21.15.169:0/386955007 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff7e0010ac0 con 0x7ff7f0102180 2024-03-28T00:56:36.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.863+0000 7ff7ed7fa640 1 --2- 172.21.15.169:0/386955007 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff7d80a2800 0x7ff7d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:36.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.863+0000 7ff7ef7fe640 1 --2- 172.21.15.169:0/386955007 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff7d80a2800 0x7ff7d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:36.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.863+0000 7ff7ed7fa640 1 -- 172.21.15.169:0/386955007 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7ff7e00cf540 con 0x7ff7f0102180 2024-03-28T00:56:36.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.863+0000 7ff7ef7fe640 1 --2- 172.21.15.169:0/386955007 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff7d80a2800 0x7ff7d80a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4000c00 tx=0x7ff7e4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:36.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:36.866+0000 7ff7ed7fa640 1 -- 172.21.15.169:0/386955007 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff7e009bc30 con 0x7ff7f0102180 2024-03-28T00:56:37.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.039+0000 7ff7f655b640 1 -- 172.21.15.169:0/386955007 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff7f002d320 con 0x7ff7f0102180 2024-03-28T00:56:37.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.040+0000 7ff7ed7fa640 1 -- 172.21.15.169:0/386955007 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff7e009fcd0 con 0x7ff7f0102180 2024-03-28T00:56:37.043 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 64 2024-03-28T00:56:37.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.042+0000 7ff7f655b640 1 -- 172.21.15.169:0/386955007 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff7d80a2800 msgr2=0x7ff7d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:37.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.042+0000 7ff7f655b640 1 --2- 172.21.15.169:0/386955007 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff7d80a2800 0x7ff7d80a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4000c00 tx=0x7ff7e4005fd0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.042+0000 7ff7f655b640 1 -- 172.21.15.169:0/386955007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7f0102180 msgr2=0x7ff7f0197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:37.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.042+0000 7ff7f655b640 1 --2- 172.21.15.169:0/386955007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7f0102180 0x7ff7f0197a00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000e9f0 tx=0x7ff7e000eeb0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.042+0000 7ff7f655b640 1 -- 172.21.15.169:0/386955007 shutdown_connections 2024-03-28T00:56:37.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.042+0000 7ff7f655b640 1 --2- 172.21.15.169:0/386955007 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff7d80a2800 0x7ff7d80a4cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.042+0000 7ff7f655b640 1 --2- 172.21.15.169:0/386955007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7f01073e0 0x7ff7f019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.042+0000 7ff7f655b640 1 --2- 172.21.15.169:0/386955007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7f0104ab0 0x7ff7f0197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.042+0000 7ff7f655b640 1 --2- 172.21.15.169:0/386955007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff7f0102180 0x7ff7f0197a00 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.042+0000 7ff7f655b640 1 -- 172.21.15.169:0/386955007 >> 172.21.15.169:0/386955007 conn(0x7ff7f00fbb30 msgr2=0x7ff7f0102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:37.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.042+0000 7ff7f655b640 1 -- 172.21.15.169:0/386955007 shutdown_connections 2024-03-28T00:56:37.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.042+0000 7ff7f655b640 1 -- 172.21.15.169:0/386955007 wait complete. 2024-03-28T00:56:37.056 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T00:56:37.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.158+0000 7fb0c0ad1640 1 -- 172.21.15.169:0/2171760021 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0bc101790 msgr2=0x7fb0bc101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:37.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.158+0000 7fb0c0ad1640 1 --2- 172.21.15.169:0/2171760021 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0bc101790 0x7fb0bc101c10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4009860 tx=0x7fb0a402bb70 comp rx=0 tx=0).stop 2024-03-28T00:56:37.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.159+0000 7fb0c0ad1640 1 -- 172.21.15.169:0/2171760021 shutdown_connections 2024-03-28T00:56:37.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.159+0000 7fb0c0ad1640 1 --2- 172.21.15.169:0/2171760021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0bc102150 0x7fb0bc1089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.159+0000 7fb0c0ad1640 1 --2- 172.21.15.169:0/2171760021 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0bc101790 0x7fb0bc101c10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.159+0000 7fb0c0ad1640 1 --2- 172.21.15.169:0/2171760021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0bc1005a0 0x7fb0bc1009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.159+0000 7fb0c0ad1640 1 -- 172.21.15.169:0/2171760021 >> 172.21.15.169:0/2171760021 conn(0x7fb0bc0fbb30 msgr2=0x7fb0bc0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:37.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.159+0000 7fb0c0ad1640 1 -- 172.21.15.169:0/2171760021 shutdown_connections 2024-03-28T00:56:37.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.159+0000 7fb0c0ad1640 1 -- 172.21.15.169:0/2171760021 wait complete. 2024-03-28T00:56:37.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.159+0000 7fb0c0ad1640 1 Processor -- start 2024-03-28T00:56:37.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.159+0000 7fb0c0ad1640 1 -- start start 2024-03-28T00:56:37.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.159+0000 7fb0c0ad1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0bc1005a0 0x7fb0bc1957c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:37.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.159+0000 7fb0c0ad1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0bc101790 0x7fb0bc195d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:37.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.160+0000 7fb0c0ad1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0bc102150 0x7fb0bc19cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:37.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.160+0000 7fb0c0ad1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc10b120 con 0x7fb0bc101790 2024-03-28T00:56:37.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.160+0000 7fb0c0ad1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc10afa0 con 0x7fb0bc102150 2024-03-28T00:56:37.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.160+0000 7fb0c0ad1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc10b2a0 con 0x7fb0bc1005a0 2024-03-28T00:56:37.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.160+0000 7fb0b3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0bc101790 0x7fb0bc195d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:37.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.160+0000 7fb0b3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0bc101790 0x7fb0bc195d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:50962/0 (socket says 172.21.15.169:50962) 2024-03-28T00:56:37.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.160+0000 7fb0b3fff640 1 -- 172.21.15.169:0/185410709 learned_addr learned my addr 172.21.15.169:0/185410709 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:37.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.160+0000 7fb0ba59c640 1 --2- 172.21.15.169:0/185410709 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0bc1005a0 0x7fb0bc1957c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:37.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.160+0000 7fb0bad9d640 1 --2- 172.21.15.169:0/185410709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0bc102150 0x7fb0bc19cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:37.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.160+0000 7fb0b3fff640 1 -- 172.21.15.169:0/185410709 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0bc1005a0 msgr2=0x7fb0bc1957c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:37.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.160+0000 7fb0b3fff640 1 --2- 172.21.15.169:0/185410709 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0bc1005a0 0x7fb0bc1957c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.160+0000 7fb0b3fff640 1 -- 172.21.15.169:0/185410709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0bc102150 msgr2=0x7fb0bc19cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:37.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.160+0000 7fb0b3fff640 1 --2- 172.21.15.169:0/185410709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0bc102150 0x7fb0bc19cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.160+0000 7fb0b3fff640 1 -- 172.21.15.169:0/185410709 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0bc19d3e0 con 0x7fb0bc101790 2024-03-28T00:56:37.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.160+0000 7fb0b3fff640 1 --2- 172.21.15.169:0/185410709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0bc101790 0x7fb0bc195d00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4005940 tx=0x7fb0a4032470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:37.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.160+0000 7fb0b37fe640 1 -- 172.21.15.169:0/185410709 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4041020 con 0x7fb0bc101790 2024-03-28T00:56:37.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.160+0000 7fb0c0ad1640 1 -- 172.21.15.169:0/185410709 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0bc19d670 con 0x7fb0bc101790 2024-03-28T00:56:37.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.160+0000 7fb0b37fe640 1 -- 172.21.15.169:0/185410709 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0a402c9f0 con 0x7fb0bc101790 2024-03-28T00:56:37.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.160+0000 7fb0b37fe640 1 -- 172.21.15.169:0/185410709 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4039040 con 0x7fb0bc101790 2024-03-28T00:56:37.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.160+0000 7fb0c0ad1640 1 -- 172.21.15.169:0/185410709 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0bc19db60 con 0x7fb0bc101790 2024-03-28T00:56:37.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.161+0000 7fb0c0ad1640 1 -- 172.21.15.169:0/185410709 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0bc1078e0 con 0x7fb0bc101790 2024-03-28T00:56:37.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.163+0000 7fb0b37fe640 1 -- 172.21.15.169:0/185410709 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb0a4044020 con 0x7fb0bc101790 2024-03-28T00:56:37.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.163+0000 7fb0b37fe640 1 --2- 172.21.15.169:0/185410709 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb0a80a2800 0x7fb0a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:37.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.163+0000 7fb0ba59c640 1 --2- 172.21.15.169:0/185410709 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb0a80a2800 0x7fb0a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:37.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.163+0000 7fb0b37fe640 1 -- 172.21.15.169:0/185410709 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fb0a40f0710 con 0x7fb0bc101790 2024-03-28T00:56:37.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.164+0000 7fb0ba59c640 1 --2- 172.21.15.169:0/185410709 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb0a80a2800 0x7fb0a80a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc1015f0 tx=0x7fb0ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:37.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.168+0000 7fb0b37fe640 1 -- 172.21.15.169:0/185410709 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb0a404b020 con 0x7fb0bc101790 2024-03-28T00:56:37.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.352+0000 7fb0c0ad1640 1 -- 172.21.15.169:0/185410709 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fb0bc0622a0 con 0x7fb0bc101790 2024-03-28T00:56:37.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.353+0000 7fb0b37fe640 1 -- 172.21.15.169:0/185410709 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7fb0a40bd450 con 0x7fb0bc101790 2024-03-28T00:56:37.356 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T00:56:37.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.355+0000 7fb0c0ad1640 1 -- 172.21.15.169:0/185410709 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb0a80a2800 msgr2=0x7fb0a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:37.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.355+0000 7fb0c0ad1640 1 --2- 172.21.15.169:0/185410709 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb0a80a2800 0x7fb0a80a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc1015f0 tx=0x7fb0ac005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:37.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.355+0000 7fb0c0ad1640 1 -- 172.21.15.169:0/185410709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0bc101790 msgr2=0x7fb0bc195d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:37.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.355+0000 7fb0c0ad1640 1 --2- 172.21.15.169:0/185410709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0bc101790 0x7fb0bc195d00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4005940 tx=0x7fb0a4032470 comp rx=0 tx=0).stop 2024-03-28T00:56:37.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.355+0000 7fb0c0ad1640 1 -- 172.21.15.169:0/185410709 shutdown_connections 2024-03-28T00:56:37.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.355+0000 7fb0c0ad1640 1 --2- 172.21.15.169:0/185410709 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb0a80a2800 0x7fb0a80a4cc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.355+0000 7fb0c0ad1640 1 --2- 172.21.15.169:0/185410709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0bc102150 0x7fb0bc19cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.355+0000 7fb0c0ad1640 1 --2- 172.21.15.169:0/185410709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0bc101790 0x7fb0bc195d00 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.355+0000 7fb0c0ad1640 1 --2- 172.21.15.169:0/185410709 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0bc1005a0 0x7fb0bc1957c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.355+0000 7fb0c0ad1640 1 -- 172.21.15.169:0/185410709 >> 172.21.15.169:0/185410709 conn(0x7fb0bc0fbb30 msgr2=0x7fb0bc104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:37.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.355+0000 7fb0c0ad1640 1 -- 172.21.15.169:0/185410709 shutdown_connections 2024-03-28T00:56:37.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.355+0000 7fb0c0ad1640 1 -- 172.21.15.169:0/185410709 wait complete. 2024-03-28T00:56:37.370 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:37 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/9556015' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:56:37.370 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-28T00:56:37.370 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9, 5, 7]}]: dispatch 2024-03-28T00:56:37.370 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:37 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/265093194' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:56:37.370 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:37 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/386955007' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:56:37.371 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:56:37.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.470+0000 7f57450dc640 1 -- 172.21.15.169:0/3690392577 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57400ff990 msgr2=0x7f57400ffdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:37.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.470+0000 7f57450dc640 1 --2- 172.21.15.169:0/3690392577 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57400ff990 0x7f57400ffdb0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5738009930 tx=0x7f573802bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:37.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.470+0000 7f57450dc640 1 -- 172.21.15.169:0/3690392577 shutdown_connections 2024-03-28T00:56:37.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.470+0000 7f57450dc640 1 --2- 172.21.15.169:0/3690392577 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5740103320 0x7f5740109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.470+0000 7f57450dc640 1 --2- 172.21.15.169:0/3690392577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5740102960 0x7f5740102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.470+0000 7f57450dc640 1 --2- 172.21.15.169:0/3690392577 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57400ff990 0x7f57400ffdb0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.470+0000 7f57450dc640 1 -- 172.21.15.169:0/3690392577 >> 172.21.15.169:0/3690392577 conn(0x7f57400fb2c0 msgr2=0x7f57400fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:37.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.470+0000 7f57450dc640 1 -- 172.21.15.169:0/3690392577 shutdown_connections 2024-03-28T00:56:37.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.470+0000 7f57450dc640 1 -- 172.21.15.169:0/3690392577 wait complete. 2024-03-28T00:56:37.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.471+0000 7f57450dc640 1 Processor -- start 2024-03-28T00:56:37.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.471+0000 7f57450dc640 1 -- start start 2024-03-28T00:56:37.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.471+0000 7f57450dc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f57400ff990 0x7f5740197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:37.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.471+0000 7f57450dc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5740102960 0x7f5740197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:37.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.471+0000 7f57450dc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5740103320 0x7f574019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:37.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.471+0000 7f57450dc640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574010d370 con 0x7f5740103320 2024-03-28T00:56:37.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.471+0000 7f57450dc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574010d1f0 con 0x7f5740102960 2024-03-28T00:56:37.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.471+0000 7f57450dc640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574010d4f0 con 0x7f57400ff990 2024-03-28T00:56:37.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.471+0000 7f573f59e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5740103320 0x7f574019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:37.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.471+0000 7f573f59e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5740103320 0x7f574019efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:50970/0 (socket says 172.21.15.169:50970) 2024-03-28T00:56:37.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.471+0000 7f573ed9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f57400ff990 0x7f5740197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:37.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.471+0000 7f573f59e640 1 -- 172.21.15.169:0/1891947523 learned_addr learned my addr 172.21.15.169:0/1891947523 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:37.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.471+0000 7f573e59c640 1 --2- 172.21.15.169:0/1891947523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5740102960 0x7f5740197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:37.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.472+0000 7f573ed9d640 1 -- 172.21.15.169:0/1891947523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5740102960 msgr2=0x7f5740197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:37.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.472+0000 7f573ed9d640 1 --2- 172.21.15.169:0/1891947523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5740102960 0x7f5740197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.472+0000 7f573ed9d640 1 -- 172.21.15.169:0/1891947523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5740103320 msgr2=0x7f574019efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:37.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.472+0000 7f573ed9d640 1 --2- 172.21.15.169:0/1891947523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5740103320 0x7f574019efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.472+0000 7f573ed9d640 1 -- 172.21.15.169:0/1891947523 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574019f650 con 0x7f57400ff990 2024-03-28T00:56:37.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.472+0000 7f573ed9d640 1 --2- 172.21.15.169:0/1891947523 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f57400ff990 0x7f5740197a30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5738036040 tx=0x7f573802c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:37.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.472+0000 7f5727fff640 1 -- 172.21.15.169:0/1891947523 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5738041070 con 0x7f57400ff990 2024-03-28T00:56:37.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.472+0000 7f57450dc640 1 -- 172.21.15.169:0/1891947523 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f574019f8e0 con 0x7f57400ff990 2024-03-28T00:56:37.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.472+0000 7f5727fff640 1 -- 172.21.15.169:0/1891947523 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57380054e0 con 0x7f57400ff990 2024-03-28T00:56:37.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.472+0000 7f5727fff640 1 -- 172.21.15.169:0/1891947523 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5738034440 con 0x7f57400ff990 2024-03-28T00:56:37.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.472+0000 7f57450dc640 1 -- 172.21.15.169:0/1891947523 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574019fdf0 con 0x7f57400ff990 2024-03-28T00:56:37.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.473+0000 7f57450dc640 1 -- 172.21.15.169:0/1891947523 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5740108b60 con 0x7f57400ff990 2024-03-28T00:56:37.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.474+0000 7f5727fff640 1 -- 172.21.15.169:0/1891947523 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5738044070 con 0x7f57400ff990 2024-03-28T00:56:37.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.475+0000 7f5727fff640 1 --2- 172.21.15.169:0/1891947523 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f57280a2800 0x7f57280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:37.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.475+0000 7f573e59c640 1 --2- 172.21.15.169:0/1891947523 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f57280a2800 0x7f57280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:37.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.475+0000 7f5727fff640 1 -- 172.21.15.169:0/1891947523 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f57380f0b90 con 0x7f57400ff990 2024-03-28T00:56:37.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.475+0000 7f573e59c640 1 --2- 172.21.15.169:0/1891947523 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f57280a2800 0x7f57280a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5740198f40 tx=0x7f5734005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:37.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.479+0000 7f5727fff640 1 -- 172.21.15.169:0/1891947523 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f57380bd240 con 0x7f57400ff990 2024-03-28T00:56:37.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:37 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/9556015' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:56:37.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:37 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-28T00:56:37.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:37 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9, 5, 7]}]: dispatch 2024-03-28T00:56:37.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:37 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/265093194' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:56:37.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:37 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/386955007' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:56:37.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:37 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/9556015' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:56:37.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:37 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-28T00:56:37.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:37 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9, 5, 7]}]: dispatch 2024-03-28T00:56:37.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:37 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/265093194' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:56:37.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:37 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/386955007' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:56:37.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.690+0000 7f57450dc640 1 -- 172.21.15.169:0/1891947523 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57400622a0 con 0x7f57400ff990 2024-03-28T00:56:37.694 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.691+0000 7f5727fff640 1 -- 172.21.15.169:0/1891947523 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11420 (secure 0 0 0) 0x7f57380c12e0 con 0x7f57400ff990 2024-03-28T00:56:37.694 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:37.694 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:54:52.274335+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24457,24478]},"id":1}]} 2024-03-28T00:56:37.695 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 24 2024-03-28T00:56:37.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.693+0000 7f57450dc640 1 -- 172.21.15.169:0/1891947523 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f57280a2800 msgr2=0x7f57280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:37.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.693+0000 7f57450dc640 1 --2- 172.21.15.169:0/1891947523 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f57280a2800 0x7f57280a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5740198f40 tx=0x7f5734005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:37.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.693+0000 7f57450dc640 1 -- 172.21.15.169:0/1891947523 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f57400ff990 msgr2=0x7f5740197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:37.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.693+0000 7f57450dc640 1 --2- 172.21.15.169:0/1891947523 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f57400ff990 0x7f5740197a30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5738036040 tx=0x7f573802c650 comp rx=0 tx=0).stop 2024-03-28T00:56:37.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.693+0000 7f57450dc640 1 -- 172.21.15.169:0/1891947523 shutdown_connections 2024-03-28T00:56:37.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.693+0000 7f57450dc640 1 --2- 172.21.15.169:0/1891947523 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f57280a2800 0x7f57280a4cc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.693+0000 7f57450dc640 1 --2- 172.21.15.169:0/1891947523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5740103320 0x7f574019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.693+0000 7f57450dc640 1 --2- 172.21.15.169:0/1891947523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5740102960 0x7f5740197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.693+0000 7f57450dc640 1 --2- 172.21.15.169:0/1891947523 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f57400ff990 0x7f5740197a30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.693+0000 7f57450dc640 1 -- 172.21.15.169:0/1891947523 >> 172.21.15.169:0/1891947523 conn(0x7f57400fb2c0 msgr2=0x7f5740105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:37.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.693+0000 7f57450dc640 1 -- 172.21.15.169:0/1891947523 shutdown_connections 2024-03-28T00:56:37.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.693+0000 7f57450dc640 1 -- 172.21.15.169:0/1891947523 wait complete. 2024-03-28T00:56:37.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T00:56:37.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.807+0000 7f7b430ac640 1 -- 172.21.15.169:0/1890404482 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b3c1017b0 msgr2=0x7f7b3c101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:37.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.807+0000 7f7b430ac640 1 --2- 172.21.15.169:0/1890404482 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b3c1017b0 0x7f7b3c101c30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7b28009900 tx=0x7f7b2802bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:37.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.807+0000 7f7b430ac640 1 -- 172.21.15.169:0/1890404482 shutdown_connections 2024-03-28T00:56:37.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.807+0000 7f7b430ac640 1 --2- 172.21.15.169:0/1890404482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b3c102170 0x7f7b3c108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.807+0000 7f7b430ac640 1 --2- 172.21.15.169:0/1890404482 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b3c1017b0 0x7f7b3c101c30 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.807+0000 7f7b430ac640 1 --2- 172.21.15.169:0/1890404482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b3c1005c0 0x7f7b3c1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.807+0000 7f7b430ac640 1 -- 172.21.15.169:0/1890404482 >> 172.21.15.169:0/1890404482 conn(0x7f7b3c0fbb50 msgr2=0x7f7b3c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:37.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.807+0000 7f7b430ac640 1 -- 172.21.15.169:0/1890404482 shutdown_connections 2024-03-28T00:56:37.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.807+0000 7f7b430ac640 1 -- 172.21.15.169:0/1890404482 wait complete. 2024-03-28T00:56:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.807+0000 7f7b430ac640 1 Processor -- start 2024-03-28T00:56:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.807+0000 7f7b430ac640 1 -- start start 2024-03-28T00:56:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.808+0000 7f7b430ac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b3c1005c0 0x7f7b3c199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.808+0000 7f7b430ac640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b3c1017b0 0x7f7b3c19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.808+0000 7f7b430ac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b3c102170 0x7f7b3c1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.808+0000 7f7b430ac640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c10d360 con 0x7f7b3c102170 2024-03-28T00:56:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.808+0000 7f7b430ac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c10d1e0 con 0x7f7b3c1005c0 2024-03-28T00:56:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.808+0000 7f7b430ac640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c10d4e0 con 0x7f7b3c1017b0 2024-03-28T00:56:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.808+0000 7f7b41649640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b3c102170 0x7f7b3c1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.808+0000 7f7b41649640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b3c102170 0x7f7b3c1a1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:50992/0 (socket says 172.21.15.169:50992) 2024-03-28T00:56:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.808+0000 7f7b41649640 1 -- 172.21.15.169:0/358442861 learned_addr learned my addr 172.21.15.169:0/358442861 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.808+0000 7f7b3bfff640 1 --2- 172.21.15.169:0/358442861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b3c1017b0 0x7f7b3c19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.808+0000 7f7b40e48640 1 --2- 172.21.15.169:0/358442861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b3c1005c0 0x7f7b3c199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.808+0000 7f7b41649640 1 -- 172.21.15.169:0/358442861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b3c1017b0 msgr2=0x7f7b3c19a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.808+0000 7f7b41649640 1 --2- 172.21.15.169:0/358442861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b3c1017b0 0x7f7b3c19a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.808+0000 7f7b41649640 1 -- 172.21.15.169:0/358442861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b3c1005c0 msgr2=0x7f7b3c199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.808+0000 7f7b41649640 1 --2- 172.21.15.169:0/358442861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b3c1005c0 0x7f7b3c199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.808+0000 7f7b41649640 1 -- 172.21.15.169:0/358442861 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b3c1a1800 con 0x7f7b3c102170 2024-03-28T00:56:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.808+0000 7f7b41649640 1 --2- 172.21.15.169:0/358442861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b3c102170 0x7f7b3c1a1190 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400eb10 tx=0x7f7b3400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.808+0000 7f7b39ffb640 1 -- 172.21.15.169:0/358442861 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b34004b20 con 0x7f7b3c102170 2024-03-28T00:56:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.808+0000 7f7b430ac640 1 -- 172.21.15.169:0/358442861 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b3c1a1af0 con 0x7f7b3c102170 2024-03-28T00:56:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.808+0000 7f7b39ffb640 1 -- 172.21.15.169:0/358442861 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b34004cc0 con 0x7f7b3c102170 2024-03-28T00:56:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.808+0000 7f7b39ffb640 1 -- 172.21.15.169:0/358442861 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b34010760 con 0x7f7b3c102170 2024-03-28T00:56:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.809+0000 7f7b430ac640 1 -- 172.21.15.169:0/358442861 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3c1a2010 con 0x7f7b3c102170 2024-03-28T00:56:37.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.810+0000 7f7b430ac640 1 -- 172.21.15.169:0/358442861 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b3c1a2240 con 0x7f7b3c102170 2024-03-28T00:56:37.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.810+0000 7f7b39ffb640 1 -- 172.21.15.169:0/358442861 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7b34005020 con 0x7f7b3c102170 2024-03-28T00:56:37.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.811+0000 7f7b39ffb640 1 --2- 172.21.15.169:0/358442861 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7b2c0a2800 0x7f7b2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:37.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.811+0000 7f7b40e48640 1 --2- 172.21.15.169:0/358442861 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7b2c0a2800 0x7f7b2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:37.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.811+0000 7f7b39ffb640 1 -- 172.21.15.169:0/358442861 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f7b340cf370 con 0x7f7b3c102170 2024-03-28T00:56:37.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.812+0000 7f7b40e48640 1 --2- 172.21.15.169:0/358442861 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7b2c0a2800 0x7f7b2c0a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c101610 tx=0x7f7b30005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:37.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.815+0000 7f7b39ffb640 1 -- 172.21.15.169:0/358442861 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7b3409c070 con 0x7f7b3c102170 2024-03-28T00:56:37.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.989+0000 7f7b430ac640 1 -- 172.21.15.169:0/358442861 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b3c1a24b0 con 0x7f7b3c102170 2024-03-28T00:56:37.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.991+0000 7f7b39ffb640 1 -- 172.21.15.169:0/358442861 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24823 (secure 0 0 0) 0x7f7b3409b970 con 0x7f7b3c102170 2024-03-28T00:56:37.994 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:37.994 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":93,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:56:37.280509+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9},{"from":5,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:56:37.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.993+0000 7f7b430ac640 1 -- 172.21.15.169:0/358442861 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7b2c0a2800 msgr2=0x7f7b2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:37.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.993+0000 7f7b430ac640 1 --2- 172.21.15.169:0/358442861 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7b2c0a2800 0x7f7b2c0a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c101610 tx=0x7f7b30005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:37.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.993+0000 7f7b430ac640 1 -- 172.21.15.169:0/358442861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b3c102170 msgr2=0x7f7b3c1a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:37.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.993+0000 7f7b430ac640 1 --2- 172.21.15.169:0/358442861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b3c102170 0x7f7b3c1a1190 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400eb10 tx=0x7f7b3400c2d0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.993+0000 7f7b430ac640 1 -- 172.21.15.169:0/358442861 shutdown_connections 2024-03-28T00:56:37.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.993+0000 7f7b430ac640 1 --2- 172.21.15.169:0/358442861 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7b2c0a2800 0x7f7b2c0a4cc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.993+0000 7f7b430ac640 1 --2- 172.21.15.169:0/358442861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b3c102170 0x7f7b3c1a1190 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.993+0000 7f7b430ac640 1 --2- 172.21.15.169:0/358442861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b3c1017b0 0x7f7b3c19a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.993+0000 7f7b430ac640 1 --2- 172.21.15.169:0/358442861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b3c1005c0 0x7f7b3c199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:37.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.993+0000 7f7b430ac640 1 -- 172.21.15.169:0/358442861 >> 172.21.15.169:0/358442861 conn(0x7f7b3c0fbb50 msgr2=0x7f7b3c1093c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:37.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.993+0000 7f7b430ac640 1 -- 172.21.15.169:0/358442861 shutdown_connections 2024-03-28T00:56:37.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:37.993+0000 7f7b430ac640 1 -- 172.21.15.169:0/358442861 wait complete. 2024-03-28T00:56:38.007 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T00:49:56.339525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T00:54:31.182562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T00:54:31.780429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T00:54:35.035724+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T00:56:38.008 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T00:56:38.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.110+0000 7f548a6a1640 1 -- 172.21.15.169:0/1288457864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54841005a0 msgr2=0x7f54841009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:38.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.110+0000 7f548a6a1640 1 --2- 172.21.15.169:0/1288457864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54841005a0 0x7f54841009c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f546c009860 tx=0x7f546c02bb70 comp rx=0 tx=0).stop 2024-03-28T00:56:38.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.110+0000 7f548a6a1640 1 -- 172.21.15.169:0/1288457864 shutdown_connections 2024-03-28T00:56:38.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.110+0000 7f548a6a1640 1 --2- 172.21.15.169:0/1288457864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5484102150 0x7f54841089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.110+0000 7f548a6a1640 1 --2- 172.21.15.169:0/1288457864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5484101790 0x7f5484101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.110+0000 7f548a6a1640 1 --2- 172.21.15.169:0/1288457864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54841005a0 0x7f54841009c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.110+0000 7f548a6a1640 1 -- 172.21.15.169:0/1288457864 >> 172.21.15.169:0/1288457864 conn(0x7f54840fbb30 msgr2=0x7f54840fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:38.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.110+0000 7f548a6a1640 1 -- 172.21.15.169:0/1288457864 shutdown_connections 2024-03-28T00:56:38.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.110+0000 7f548a6a1640 1 -- 172.21.15.169:0/1288457864 wait complete. 2024-03-28T00:56:38.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.110+0000 7f548a6a1640 1 Processor -- start 2024-03-28T00:56:38.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.110+0000 7f548a6a1640 1 -- start start 2024-03-28T00:56:38.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f548a6a1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54841005a0 0x7f5484196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:38.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f548a6a1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5484101790 0x7f5484196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:38.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f548a6a1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5484102150 0x7f548419dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:38.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f548a6a1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548410c200 con 0x7f54841005a0 2024-03-28T00:56:38.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f548a6a1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548410c080 con 0x7f5484101790 2024-03-28T00:56:38.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f548a6a1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548410c380 con 0x7f5484102150 2024-03-28T00:56:38.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f5483fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54841005a0 0x7f5484196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:38.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f5483fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54841005a0 0x7f5484196930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51004/0 (socket says 172.21.15.169:51004) 2024-03-28T00:56:38.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f54837fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5484101790 0x7f5484196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:38.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f5483fff640 1 -- 172.21.15.169:0/1994609387 learned_addr learned my addr 172.21.15.169:0/1994609387 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:38.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f5488c3e640 1 --2- 172.21.15.169:0/1994609387 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5484102150 0x7f548419dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:38.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f5483fff640 1 -- 172.21.15.169:0/1994609387 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5484102150 msgr2=0x7f548419dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:38.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f5483fff640 1 --2- 172.21.15.169:0/1994609387 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5484102150 0x7f548419dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f5483fff640 1 -- 172.21.15.169:0/1994609387 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5484101790 msgr2=0x7f5484196e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:38.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f5483fff640 1 --2- 172.21.15.169:0/1994609387 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5484101790 0x7f5484196e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f5483fff640 1 -- 172.21.15.169:0/1994609387 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548419e550 con 0x7f54841005a0 2024-03-28T00:56:38.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f54837fe640 1 --2- 172.21.15.169:0/1994609387 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5484101790 0x7f5484196e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:38.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f5483fff640 1 --2- 172.21.15.169:0/1994609387 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54841005a0 0x7f5484196930 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f546c02c0b0 tx=0x7f546c032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:38.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f54817fa640 1 -- 172.21.15.169:0/1994609387 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c041020 con 0x7f54841005a0 2024-03-28T00:56:38.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f548a6a1640 1 -- 172.21.15.169:0/1994609387 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548419e7e0 con 0x7f54841005a0 2024-03-28T00:56:38.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f54817fa640 1 -- 172.21.15.169:0/1994609387 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f546c02c9f0 con 0x7f54841005a0 2024-03-28T00:56:38.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f54817fa640 1 -- 172.21.15.169:0/1994609387 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c039040 con 0x7f54841005a0 2024-03-28T00:56:38.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.111+0000 7f548a6a1640 1 -- 172.21.15.169:0/1994609387 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f548419ecd0 con 0x7f54841005a0 2024-03-28T00:56:38.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.112+0000 7f548a6a1640 1 -- 172.21.15.169:0/1994609387 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54841079e0 con 0x7f54841005a0 2024-03-28T00:56:38.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.113+0000 7f54817fa640 1 -- 172.21.15.169:0/1994609387 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f546c044020 con 0x7f54841005a0 2024-03-28T00:56:38.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.114+0000 7f54817fa640 1 --2- 172.21.15.169:0/1994609387 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f54700a2800 0x7f54700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:38.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.114+0000 7f54837fe640 1 --2- 172.21.15.169:0/1994609387 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f54700a2800 0x7f54700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:38.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.114+0000 7f54817fa640 1 -- 172.21.15.169:0/1994609387 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f546c0f06f0 con 0x7f54841005a0 2024-03-28T00:56:38.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.114+0000 7f54837fe640 1 --2- 172.21.15.169:0/1994609387 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f54700a2800 0x7f54700a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5484197e40 tx=0x7f5474005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:38.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.118+0000 7f54817fa640 1 -- 172.21.15.169:0/1994609387 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f546c04b020 con 0x7f54841005a0 2024-03-28T00:56:38.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.297+0000 7f548a6a1640 1 -- 172.21.15.169:0/1994609387 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f548402d070 con 0x7f54841005a0 2024-03-28T00:56:38.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.298+0000 7f54817fa640 1 -- 172.21.15.169:0/1994609387 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7f546c0bd3f0 con 0x7f54841005a0 2024-03-28T00:56:38.301 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 1 2024-03-28T00:56:38.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.300+0000 7f548a6a1640 1 -- 172.21.15.169:0/1994609387 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f54700a2800 msgr2=0x7f54700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:38.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.300+0000 7f548a6a1640 1 --2- 172.21.15.169:0/1994609387 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f54700a2800 0x7f54700a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5484197e40 tx=0x7f5474005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:38.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.300+0000 7f548a6a1640 1 -- 172.21.15.169:0/1994609387 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54841005a0 msgr2=0x7f5484196930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:38.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.300+0000 7f548a6a1640 1 --2- 172.21.15.169:0/1994609387 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54841005a0 0x7f5484196930 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f546c02c0b0 tx=0x7f546c032430 comp rx=0 tx=0).stop 2024-03-28T00:56:38.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.300+0000 7f548a6a1640 1 -- 172.21.15.169:0/1994609387 shutdown_connections 2024-03-28T00:56:38.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.300+0000 7f548a6a1640 1 --2- 172.21.15.169:0/1994609387 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f54700a2800 0x7f54700a4cc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.300+0000 7f548a6a1640 1 --2- 172.21.15.169:0/1994609387 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5484102150 0x7f548419dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.300+0000 7f548a6a1640 1 --2- 172.21.15.169:0/1994609387 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5484101790 0x7f5484196e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.300+0000 7f548a6a1640 1 --2- 172.21.15.169:0/1994609387 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54841005a0 0x7f5484196930 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.300+0000 7f548a6a1640 1 -- 172.21.15.169:0/1994609387 >> 172.21.15.169:0/1994609387 conn(0x7f54840fbb30 msgr2=0x7f5484104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:38.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.300+0000 7f548a6a1640 1 -- 172.21.15.169:0/1994609387 shutdown_connections 2024-03-28T00:56:38.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.300+0000 7f548a6a1640 1 -- 172.21.15.169:0/1994609387 wait complete. 2024-03-28T00:56:38.315 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T00:56:38.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.418+0000 7fc759244640 1 -- 172.21.15.169:0/3883183702 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc754103260 msgr2=0x7fc754109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:38.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.418+0000 7fc759244640 1 --2- 172.21.15.169:0/3883183702 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc754103260 0x7fc754109af0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc74c009ca0 tx=0x7fc74c02c0f0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.418+0000 7fc759244640 1 -- 172.21.15.169:0/3883183702 shutdown_connections 2024-03-28T00:56:38.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.418+0000 7fc759244640 1 --2- 172.21.15.169:0/3883183702 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc754103260 0x7fc754109af0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.418+0000 7fc759244640 1 --2- 172.21.15.169:0/3883183702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7541028a0 0x7fc754102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.418+0000 7fc759244640 1 --2- 172.21.15.169:0/3883183702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7541016b0 0x7fc754101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.418+0000 7fc759244640 1 -- 172.21.15.169:0/3883183702 >> 172.21.15.169:0/3883183702 conn(0x7fc7540fcc40 msgr2=0x7fc7540ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:38.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.418+0000 7fc759244640 1 -- 172.21.15.169:0/3883183702 shutdown_connections 2024-03-28T00:56:38.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.418+0000 7fc759244640 1 -- 172.21.15.169:0/3883183702 wait complete. 2024-03-28T00:56:38.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.418+0000 7fc759244640 1 Processor -- start 2024-03-28T00:56:38.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.418+0000 7fc759244640 1 -- start start 2024-03-28T00:56:38.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc759244640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7541016b0 0x7fc754197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:38.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc759244640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7541028a0 0x7fc754197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:38.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc759244640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc754103260 0x7fc75419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:38.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc759244640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75410d330 con 0x7fc754103260 2024-03-28T00:56:38.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc759244640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75410d1b0 con 0x7fc7541016b0 2024-03-28T00:56:38.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc759244640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75410d4b0 con 0x7fc7541028a0 2024-03-28T00:56:38.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc75359e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc754103260 0x7fc75419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:38.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc74bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7541028a0 0x7fc754197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:38.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc75359e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc754103260 0x7fc75419ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51026/0 (socket says 172.21.15.169:51026) 2024-03-28T00:56:38.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc75359e640 1 -- 172.21.15.169:0/4205056589 learned_addr learned my addr 172.21.15.169:0/4205056589 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:38.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc752d9d640 1 --2- 172.21.15.169:0/4205056589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7541016b0 0x7fc754197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:38.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc75359e640 1 -- 172.21.15.169:0/4205056589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7541028a0 msgr2=0x7fc754197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:38.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc75359e640 1 --2- 172.21.15.169:0/4205056589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7541028a0 0x7fc754197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc75359e640 1 -- 172.21.15.169:0/4205056589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7541016b0 msgr2=0x7fc754197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:38.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc75359e640 1 --2- 172.21.15.169:0/4205056589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7541016b0 0x7fc754197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc75359e640 1 -- 172.21.15.169:0/4205056589 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75419f570 con 0x7fc754103260 2024-03-28T00:56:38.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc752d9d640 1 --2- 172.21.15.169:0/4205056589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7541016b0 0x7fc754197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:38.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc74bfff640 1 --2- 172.21.15.169:0/4205056589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7541028a0 0x7fc754197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:38.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc75359e640 1 --2- 172.21.15.169:0/4205056589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc754103260 0x7fc75419ef00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc74c037040 tx=0x7fc74c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:38.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc750d99640 1 -- 172.21.15.169:0/4205056589 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c041020 con 0x7fc754103260 2024-03-28T00:56:38.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc759244640 1 -- 172.21.15.169:0/4205056589 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc75419f800 con 0x7fc754103260 2024-03-28T00:56:38.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc750d99640 1 -- 172.21.15.169:0/4205056589 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc74c034cd0 con 0x7fc754103260 2024-03-28T00:56:38.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.419+0000 7fc750d99640 1 -- 172.21.15.169:0/4205056589 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c039040 con 0x7fc754103260 2024-03-28T00:56:38.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.420+0000 7fc759244640 1 -- 172.21.15.169:0/4205056589 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc75419fd40 con 0x7fc754103260 2024-03-28T00:56:38.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.421+0000 7fc759244640 1 -- 172.21.15.169:0/4205056589 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7541089d0 con 0x7fc754103260 2024-03-28T00:56:38.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.421+0000 7fc750d99640 1 -- 172.21.15.169:0/4205056589 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc74c044020 con 0x7fc754103260 2024-03-28T00:56:38.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.422+0000 7fc750d99640 1 --2- 172.21.15.169:0/4205056589 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc7400a2800 0x7fc7400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:38.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.422+0000 7fc752d9d640 1 --2- 172.21.15.169:0/4205056589 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc7400a2800 0x7fc7400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:38.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.422+0000 7fc750d99640 1 -- 172.21.15.169:0/4205056589 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc74c0f0ca0 con 0x7fc754103260 2024-03-28T00:56:38.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.422+0000 7fc752d9d640 1 --2- 172.21.15.169:0/4205056589 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc7400a2800 0x7fc7400a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc754102700 tx=0x7fc744005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:38.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.427+0000 7fc750d99640 1 -- 172.21.15.169:0/4205056589 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc74c04b020 con 0x7fc754103260 2024-03-28T00:56:38.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:38 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-28T00:56:38.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:38 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9, 5, 7]}]': finished 2024-03-28T00:56:38.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:38 smithi169 ceph-mon[32274]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T00:56:38.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:38 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/185410709' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:56:38.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:38 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1891947523' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:56:38.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:38 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/358442861' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:38.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:38 smithi169 ceph-mon[32274]: pgmap v354: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T00:56:38.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:38 smithi169 ceph-mon[32274]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T00:56:38.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.601+0000 7fc759244640 1 -- 172.21.15.169:0/4205056589 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc7540622a0 con 0x7fc754103260 2024-03-28T00:56:38.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.601+0000 7fc750d99640 1 -- 172.21.15.169:0/4205056589 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc74c0bda30 con 0x7fc754103260 2024-03-28T00:56:38.604 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T00:56:38.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.603+0000 7fc759244640 1 -- 172.21.15.169:0/4205056589 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc7400a2800 msgr2=0x7fc7400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:38.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.603+0000 7fc759244640 1 --2- 172.21.15.169:0/4205056589 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc7400a2800 0x7fc7400a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc754102700 tx=0x7fc744005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:38.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.603+0000 7fc759244640 1 -- 172.21.15.169:0/4205056589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc754103260 msgr2=0x7fc75419ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:38.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.603+0000 7fc759244640 1 --2- 172.21.15.169:0/4205056589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc754103260 0x7fc75419ef00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc74c037040 tx=0x7fc74c032f40 comp rx=0 tx=0).stop 2024-03-28T00:56:38.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.603+0000 7fc759244640 1 -- 172.21.15.169:0/4205056589 shutdown_connections 2024-03-28T00:56:38.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.603+0000 7fc759244640 1 --2- 172.21.15.169:0/4205056589 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc7400a2800 0x7fc7400a4cc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.603+0000 7fc759244640 1 --2- 172.21.15.169:0/4205056589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc754103260 0x7fc75419ef00 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.603+0000 7fc759244640 1 --2- 172.21.15.169:0/4205056589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7541028a0 0x7fc754197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.603+0000 7fc759244640 1 --2- 172.21.15.169:0/4205056589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7541016b0 0x7fc754197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.603+0000 7fc759244640 1 -- 172.21.15.169:0/4205056589 >> 172.21.15.169:0/4205056589 conn(0x7fc7540fcc40 msgr2=0x7fc754105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:38.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.603+0000 7fc759244640 1 -- 172.21.15.169:0/4205056589 shutdown_connections 2024-03-28T00:56:38.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.603+0000 7fc759244640 1 -- 172.21.15.169:0/4205056589 wait complete. 2024-03-28T00:56:38.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:38 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-28T00:56:38.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:38 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9, 5, 7]}]': finished 2024-03-28T00:56:38.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:38 smithi174 ceph-mon[36060]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T00:56:38.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:38 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/185410709' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:56:38.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:38 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1891947523' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:56:38.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:38 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/358442861' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:38.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:38 smithi174 ceph-mon[36060]: pgmap v354: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T00:56:38.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:38 smithi174 ceph-mon[36060]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T00:56:38.617 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T00:56:38.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:38 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-28T00:56:38.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:38 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9, 5, 7]}]': finished 2024-03-28T00:56:38.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:38 smithi177 ceph-mon[36206]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T00:56:38.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:38 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/185410709' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:56:38.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:38 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1891947523' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:56:38.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:38 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/358442861' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:38.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:38 smithi177 ceph-mon[36206]: pgmap v354: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T00:56:38.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:38 smithi177 ceph-mon[36206]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T00:56:38.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.720+0000 7fa163b3b640 1 -- 172.21.15.169:0/545225738 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa15c102150 msgr2=0x7fa15c1089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:38.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.720+0000 7fa163b3b640 1 --2- 172.21.15.169:0/545225738 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa15c102150 0x7fa15c1089e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fa154009ca0 tx=0x7fa15402c0f0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.720+0000 7fa163b3b640 1 -- 172.21.15.169:0/545225738 shutdown_connections 2024-03-28T00:56:38.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.720+0000 7fa163b3b640 1 --2- 172.21.15.169:0/545225738 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa15c102150 0x7fa15c1089e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.720+0000 7fa163b3b640 1 --2- 172.21.15.169:0/545225738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa15c101790 0x7fa15c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.720+0000 7fa163b3b640 1 --2- 172.21.15.169:0/545225738 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa15c1005a0 0x7fa15c1009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.720+0000 7fa163b3b640 1 -- 172.21.15.169:0/545225738 >> 172.21.15.169:0/545225738 conn(0x7fa15c0fbb30 msgr2=0x7fa15c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:38.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.720+0000 7fa163b3b640 1 -- 172.21.15.169:0/545225738 shutdown_connections 2024-03-28T00:56:38.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.720+0000 7fa163b3b640 1 -- 172.21.15.169:0/545225738 wait complete. 2024-03-28T00:56:38.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.720+0000 7fa163b3b640 1 Processor -- start 2024-03-28T00:56:38.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.720+0000 7fa163b3b640 1 -- start start 2024-03-28T00:56:38.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.720+0000 7fa163b3b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa15c1005a0 0x7fa15c1979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:38.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.720+0000 7fa163b3b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa15c101790 0x7fa15c197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:38.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.720+0000 7fa163b3b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa15c102150 0x7fa15c19ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:38.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.720+0000 7fa163b3b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c10c230 con 0x7fa15c102150 2024-03-28T00:56:38.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.720+0000 7fa163b3b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c10c0b0 con 0x7fa15c101790 2024-03-28T00:56:38.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.720+0000 7fa163b3b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c10c3b0 con 0x7fa15c1005a0 2024-03-28T00:56:38.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.721+0000 7fa1618d7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa15c1005a0 0x7fa15c1979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:38.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.721+0000 7fa1620d8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa15c102150 0x7fa15c19ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:38.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.721+0000 7fa1620d8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa15c102150 0x7fa15c19ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51040/0 (socket says 172.21.15.169:51040) 2024-03-28T00:56:38.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.721+0000 7fa1618d7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa15c1005a0 0x7fa15c1979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:55816/0 (socket says 172.21.15.169:55816) 2024-03-28T00:56:38.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.721+0000 7fa1620d8640 1 -- 172.21.15.169:0/4025375528 learned_addr learned my addr 172.21.15.169:0/4025375528 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:38.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.721+0000 7fa1610d6640 1 --2- 172.21.15.169:0/4025375528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa15c101790 0x7fa15c197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:38.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.721+0000 7fa1620d8640 1 -- 172.21.15.169:0/4025375528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa15c1005a0 msgr2=0x7fa15c1979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:38.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.721+0000 7fa1620d8640 1 --2- 172.21.15.169:0/4025375528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa15c1005a0 0x7fa15c1979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.721+0000 7fa1620d8640 1 -- 172.21.15.169:0/4025375528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa15c101790 msgr2=0x7fa15c197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:38.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.721+0000 7fa1620d8640 1 --2- 172.21.15.169:0/4025375528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa15c101790 0x7fa15c197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.721+0000 7fa1620d8640 1 -- 172.21.15.169:0/4025375528 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15c19f5c0 con 0x7fa15c102150 2024-03-28T00:56:38.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.721+0000 7fa1620d8640 1 --2- 172.21.15.169:0/4025375528 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa15c102150 0x7fa15c19ef50 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa154037040 tx=0x7fa154032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:38.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.721+0000 7fa152ffd640 1 -- 172.21.15.169:0/4025375528 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa154041020 con 0x7fa15c102150 2024-03-28T00:56:38.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.721+0000 7fa163b3b640 1 -- 172.21.15.169:0/4025375528 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa15c19f850 con 0x7fa15c102150 2024-03-28T00:56:38.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.721+0000 7fa152ffd640 1 -- 172.21.15.169:0/4025375528 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa154034cd0 con 0x7fa15c102150 2024-03-28T00:56:38.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.721+0000 7fa152ffd640 1 -- 172.21.15.169:0/4025375528 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa154039040 con 0x7fa15c102150 2024-03-28T00:56:38.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.721+0000 7fa163b3b640 1 -- 172.21.15.169:0/4025375528 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa15c19fd40 con 0x7fa15c102150 2024-03-28T00:56:38.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.722+0000 7fa163b3b640 1 -- 172.21.15.169:0/4025375528 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa15c19ffa0 con 0x7fa15c102150 2024-03-28T00:56:38.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.723+0000 7fa152ffd640 1 -- 172.21.15.169:0/4025375528 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa154044020 con 0x7fa15c102150 2024-03-28T00:56:38.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.724+0000 7fa152ffd640 1 --2- 172.21.15.169:0/4025375528 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa1480a2800 0x7fa1480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:38.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.724+0000 7fa152ffd640 1 -- 172.21.15.169:0/4025375528 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa1540f1060 con 0x7fa15c102150 2024-03-28T00:56:38.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.724+0000 7fa1618d7640 1 --2- 172.21.15.169:0/4025375528 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa1480a2800 0x7fa1480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:38.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.724+0000 7fa1618d7640 1 --2- 172.21.15.169:0/4025375528 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa1480a2800 0x7fa1480a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa15c1015f0 tx=0x7fa158005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:38.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.727+0000 7fa152ffd640 1 -- 172.21.15.169:0/4025375528 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa15404b020 con 0x7fa15c102150 2024-03-28T00:56:38.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.901+0000 7fa163b3b640 1 -- 172.21.15.169:0/4025375528 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa15c1a0290 con 0x7fa15c102150 2024-03-28T00:56:38.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.902+0000 7fa152ffd640 1 -- 172.21.15.169:0/4025375528 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa1540bddf0 con 0x7fa15c102150 2024-03-28T00:56:38.905 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 64 2024-03-28T00:56:38.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.904+0000 7fa163b3b640 1 -- 172.21.15.169:0/4025375528 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa1480a2800 msgr2=0x7fa1480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:38.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.904+0000 7fa163b3b640 1 --2- 172.21.15.169:0/4025375528 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa1480a2800 0x7fa1480a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa15c1015f0 tx=0x7fa158005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:38.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.904+0000 7fa163b3b640 1 -- 172.21.15.169:0/4025375528 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa15c102150 msgr2=0x7fa15c19ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:38.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.904+0000 7fa163b3b640 1 --2- 172.21.15.169:0/4025375528 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa15c102150 0x7fa15c19ef50 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa154037040 tx=0x7fa154032f40 comp rx=0 tx=0).stop 2024-03-28T00:56:38.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.904+0000 7fa163b3b640 1 -- 172.21.15.169:0/4025375528 shutdown_connections 2024-03-28T00:56:38.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.904+0000 7fa163b3b640 1 --2- 172.21.15.169:0/4025375528 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa1480a2800 0x7fa1480a4cc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.904+0000 7fa163b3b640 1 --2- 172.21.15.169:0/4025375528 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa15c102150 0x7fa15c19ef50 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.904+0000 7fa163b3b640 1 --2- 172.21.15.169:0/4025375528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa15c101790 0x7fa15c197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.904+0000 7fa163b3b640 1 --2- 172.21.15.169:0/4025375528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa15c1005a0 0x7fa15c1979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:38.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.904+0000 7fa163b3b640 1 -- 172.21.15.169:0/4025375528 >> 172.21.15.169:0/4025375528 conn(0x7fa15c0fbb30 msgr2=0x7fa15c0fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:38.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.904+0000 7fa163b3b640 1 -- 172.21.15.169:0/4025375528 shutdown_connections 2024-03-28T00:56:38.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:38.904+0000 7fa163b3b640 1 -- 172.21.15.169:0/4025375528 wait complete. 2024-03-28T00:56:38.918 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T00:56:39.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.020+0000 7f85b75a9640 1 -- 172.21.15.169:0/3151378776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85b01018f0 msgr2=0x7f85b0108180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:39.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.020+0000 7f85b75a9640 1 --2- 172.21.15.169:0/3151378776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85b01018f0 0x7f85b0108180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f85a8009ca0 tx=0x7f85a802c0f0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.020+0000 7f85b75a9640 1 -- 172.21.15.169:0/3151378776 shutdown_connections 2024-03-28T00:56:39.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.020+0000 7f85b75a9640 1 --2- 172.21.15.169:0/3151378776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85b01018f0 0x7f85b0108180 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.020+0000 7f85b75a9640 1 --2- 172.21.15.169:0/3151378776 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85b0100f30 0x7f85b01013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.020+0000 7f85b75a9640 1 --2- 172.21.15.169:0/3151378776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85b00ffd40 0x7f85b0100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.020+0000 7f85b75a9640 1 -- 172.21.15.169:0/3151378776 >> 172.21.15.169:0/3151378776 conn(0x7f85b00fb2d0 msgr2=0x7f85b00fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:39.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.020+0000 7f85b75a9640 1 -- 172.21.15.169:0/3151378776 shutdown_connections 2024-03-28T00:56:39.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.020+0000 7f85b75a9640 1 -- 172.21.15.169:0/3151378776 wait complete. 2024-03-28T00:56:39.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.021+0000 7f85b75a9640 1 Processor -- start 2024-03-28T00:56:39.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.021+0000 7f85b75a9640 1 -- start start 2024-03-28T00:56:39.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.021+0000 7f85b75a9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85b00ffd40 0x7f85b0199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.021+0000 7f85b75a9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85b0100f30 0x7f85b019a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.021+0000 7f85b75a9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85b01018f0 0x7f85b01a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.021+0000 7f85b75a9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b010b150 con 0x7f85b01018f0 2024-03-28T00:56:39.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.021+0000 7f85b75a9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b010afd0 con 0x7f85b00ffd40 2024-03-28T00:56:39.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.021+0000 7f85b75a9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b010b2d0 con 0x7f85b0100f30 2024-03-28T00:56:39.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.021+0000 7f85b5b46640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85b01018f0 0x7f85b01a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.021+0000 7f85b4b44640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85b0100f30 0x7f85b019a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.021+0000 7f85b5b46640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85b01018f0 0x7f85b01a1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51054/0 (socket says 172.21.15.169:51054) 2024-03-28T00:56:39.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.021+0000 7f85b5b46640 1 -- 172.21.15.169:0/1760066720 learned_addr learned my addr 172.21.15.169:0/1760066720 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:39.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.021+0000 7f85b5345640 1 --2- 172.21.15.169:0/1760066720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85b00ffd40 0x7f85b0199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.021+0000 7f85b5b46640 1 -- 172.21.15.169:0/1760066720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85b0100f30 msgr2=0x7f85b019a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:39.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.021+0000 7f85b5b46640 1 --2- 172.21.15.169:0/1760066720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85b0100f30 0x7f85b019a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.021+0000 7f85b5b46640 1 -- 172.21.15.169:0/1760066720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85b00ffd40 msgr2=0x7f85b0199b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:39.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.021+0000 7f85b5b46640 1 --2- 172.21.15.169:0/1760066720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85b00ffd40 0x7f85b0199b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.021+0000 7f85b5b46640 1 -- 172.21.15.169:0/1760066720 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b01a17b0 con 0x7f85b01018f0 2024-03-28T00:56:39.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.021+0000 7f85b5b46640 1 --2- 172.21.15.169:0/1760066720 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85b01018f0 0x7f85b01a1140 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f85a802c630 tx=0x7f85a8032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:39.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.021+0000 7f85a67fc640 1 -- 172.21.15.169:0/1760066720 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a8041020 con 0x7f85b01018f0 2024-03-28T00:56:39.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.022+0000 7f85b75a9640 1 -- 172.21.15.169:0/1760066720 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b01a1a40 con 0x7f85b01018f0 2024-03-28T00:56:39.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.022+0000 7f85a67fc640 1 -- 172.21.15.169:0/1760066720 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f85a8034cd0 con 0x7f85b01018f0 2024-03-28T00:56:39.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.022+0000 7f85a67fc640 1 -- 172.21.15.169:0/1760066720 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a8039040 con 0x7f85b01018f0 2024-03-28T00:56:39.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.022+0000 7f85b75a9640 1 -- 172.21.15.169:0/1760066720 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b01a1f30 con 0x7f85b01018f0 2024-03-28T00:56:39.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.022+0000 7f85b75a9640 1 -- 172.21.15.169:0/1760066720 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b0107000 con 0x7f85b01018f0 2024-03-28T00:56:39.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.024+0000 7f85a67fc640 1 -- 172.21.15.169:0/1760066720 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f85a8044020 con 0x7f85b01018f0 2024-03-28T00:56:39.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.024+0000 7f85a67fc640 1 --2- 172.21.15.169:0/1760066720 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f859c0a2800 0x7f859c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.024+0000 7f85b5345640 1 --2- 172.21.15.169:0/1760066720 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f859c0a2800 0x7f859c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.024+0000 7f85a67fc640 1 -- 172.21.15.169:0/1760066720 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f85a80f0c20 con 0x7f85b01018f0 2024-03-28T00:56:39.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.025+0000 7f85b5345640 1 --2- 172.21.15.169:0/1760066720 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f859c0a2800 0x7f859c0a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f85b0100d90 tx=0x7f85ac010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:39.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.029+0000 7f85a67fc640 1 -- 172.21.15.169:0/1760066720 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f85a804b020 con 0x7f85b01018f0 2024-03-28T00:56:39.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.208+0000 7f85b75a9640 1 -- 172.21.15.169:0/1760066720 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f85b00622a0 con 0x7f85b01018f0 2024-03-28T00:56:39.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.209+0000 7f85a67fc640 1 -- 172.21.15.169:0/1760066720 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) v1 ==== 103+0+11 (secure 0 0 0) 0x7f85a80b7e20 con 0x7f85b01018f0 2024-03-28T00:56:39.212 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T00:56:39.214 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.211+0000 7f85b75a9640 1 -- 172.21.15.169:0/1760066720 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f859c0a2800 msgr2=0x7f859c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:39.214 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.211+0000 7f85b75a9640 1 --2- 172.21.15.169:0/1760066720 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f859c0a2800 0x7f859c0a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f85b0100d90 tx=0x7f85ac010040 comp rx=0 tx=0).stop 2024-03-28T00:56:39.214 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.211+0000 7f85b75a9640 1 -- 172.21.15.169:0/1760066720 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85b01018f0 msgr2=0x7f85b01a1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:39.214 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.211+0000 7f85b75a9640 1 --2- 172.21.15.169:0/1760066720 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85b01018f0 0x7f85b01a1140 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f85a802c630 tx=0x7f85a8032ea0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.214 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.211+0000 7f85b75a9640 1 -- 172.21.15.169:0/1760066720 shutdown_connections 2024-03-28T00:56:39.214 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.211+0000 7f85b75a9640 1 --2- 172.21.15.169:0/1760066720 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f859c0a2800 0x7f859c0a4cc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.214 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.211+0000 7f85b75a9640 1 --2- 172.21.15.169:0/1760066720 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85b01018f0 0x7f85b01a1140 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.211+0000 7f85b75a9640 1 --2- 172.21.15.169:0/1760066720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85b0100f30 0x7f85b019a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.211+0000 7f85b75a9640 1 --2- 172.21.15.169:0/1760066720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85b00ffd40 0x7f85b0199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.211+0000 7f85b75a9640 1 -- 172.21.15.169:0/1760066720 >> 172.21.15.169:0/1760066720 conn(0x7f85b00fb2d0 msgr2=0x7f85b01041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:39.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.211+0000 7f85b75a9640 1 -- 172.21.15.169:0/1760066720 shutdown_connections 2024-03-28T00:56:39.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.211+0000 7f85b75a9640 1 -- 172.21.15.169:0/1760066720 wait complete. 2024-03-28T00:56:39.225 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:56:39.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.326+0000 7f1a78b09640 1 -- 172.21.15.169:0/105126829 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a741016c0 msgr2=0x7f1a74101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:39.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.326+0000 7f1a78b09640 1 --2- 172.21.15.169:0/105126829 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a741016c0 0x7f1a74101ae0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1a64009900 tx=0x7f1a6402bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:39.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.326+0000 7f1a78b09640 1 -- 172.21.15.169:0/105126829 shutdown_connections 2024-03-28T00:56:39.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.326+0000 7f1a78b09640 1 --2- 172.21.15.169:0/105126829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a74103270 0x7f1a74109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.326+0000 7f1a78b09640 1 --2- 172.21.15.169:0/105126829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a741028b0 0x7f1a74102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.326+0000 7f1a78b09640 1 --2- 172.21.15.169:0/105126829 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a741016c0 0x7f1a74101ae0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.326+0000 7f1a78b09640 1 -- 172.21.15.169:0/105126829 >> 172.21.15.169:0/105126829 conn(0x7f1a740fcc50 msgr2=0x7f1a740ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:39.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.326+0000 7f1a78b09640 1 -- 172.21.15.169:0/105126829 shutdown_connections 2024-03-28T00:56:39.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.326+0000 7f1a78b09640 1 -- 172.21.15.169:0/105126829 wait complete. 2024-03-28T00:56:39.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.327+0000 7f1a78b09640 1 Processor -- start 2024-03-28T00:56:39.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.327+0000 7f1a78b09640 1 -- start start 2024-03-28T00:56:39.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.327+0000 7f1a78b09640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a741016c0 0x7f1a741979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.327+0000 7f1a78b09640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a741028b0 0x7f1a74197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.327+0000 7f1a78b09640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a74103270 0x7f1a7419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.327+0000 7f1a78b09640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7410d340 con 0x7f1a74103270 2024-03-28T00:56:39.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.327+0000 7f1a78b09640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7410d1c0 con 0x7f1a741028b0 2024-03-28T00:56:39.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.327+0000 7f1a78b09640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7410d4c0 con 0x7f1a741016c0 2024-03-28T00:56:39.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.327+0000 7f1a72d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a74103270 0x7f1a7419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.327+0000 7f1a7259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a741016c0 0x7f1a741979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.327+0000 7f1a72d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a74103270 0x7f1a7419ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51062/0 (socket says 172.21.15.169:51062) 2024-03-28T00:56:39.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.327+0000 7f1a72d9d640 1 -- 172.21.15.169:0/3597073715 learned_addr learned my addr 172.21.15.169:0/3597073715 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:39.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.327+0000 7f1a6bfff640 1 --2- 172.21.15.169:0/3597073715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a741028b0 0x7f1a74197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.327+0000 7f1a72d9d640 1 -- 172.21.15.169:0/3597073715 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a741016c0 msgr2=0x7f1a741979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:39.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.327+0000 7f1a72d9d640 1 --2- 172.21.15.169:0/3597073715 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a741016c0 0x7f1a741979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.327+0000 7f1a72d9d640 1 -- 172.21.15.169:0/3597073715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a741028b0 msgr2=0x7f1a74197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:39.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.327+0000 7f1a72d9d640 1 --2- 172.21.15.169:0/3597073715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a741028b0 0x7f1a74197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.327+0000 7f1a72d9d640 1 -- 172.21.15.169:0/3597073715 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7419f5d0 con 0x7f1a74103270 2024-03-28T00:56:39.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.327+0000 7f1a6bfff640 1 --2- 172.21.15.169:0/3597073715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a741028b0 0x7f1a74197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:39.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.327+0000 7f1a72d9d640 1 --2- 172.21.15.169:0/3597073715 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a74103270 0x7f1a7419ef60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c00eb10 tx=0x7f1a6c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:39.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.328+0000 7f1a6b7fe640 1 -- 172.21.15.169:0/3597073715 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c004b20 con 0x7f1a74103270 2024-03-28T00:56:39.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.328+0000 7f1a78b09640 1 -- 172.21.15.169:0/3597073715 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a7419f8c0 con 0x7f1a74103270 2024-03-28T00:56:39.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.328+0000 7f1a6b7fe640 1 -- 172.21.15.169:0/3597073715 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a6c004cc0 con 0x7f1a74103270 2024-03-28T00:56:39.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.328+0000 7f1a6b7fe640 1 -- 172.21.15.169:0/3597073715 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c010760 con 0x7f1a74103270 2024-03-28T00:56:39.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.328+0000 7f1a78b09640 1 -- 172.21.15.169:0/3597073715 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a7419fe60 con 0x7f1a74103270 2024-03-28T00:56:39.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.329+0000 7f1a78b09640 1 -- 172.21.15.169:0/3597073715 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a741089e0 con 0x7f1a74103270 2024-03-28T00:56:39.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.330+0000 7f1a6b7fe640 1 -- 172.21.15.169:0/3597073715 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1a6c005020 con 0x7f1a74103270 2024-03-28T00:56:39.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.330+0000 7f1a6b7fe640 1 --2- 172.21.15.169:0/3597073715 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1a600a2800 0x7f1a600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.330+0000 7f1a7259c640 1 --2- 172.21.15.169:0/3597073715 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1a600a2800 0x7f1a600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.331+0000 7f1a6b7fe640 1 -- 172.21.15.169:0/3597073715 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f1a6c0cf4e0 con 0x7f1a74103270 2024-03-28T00:56:39.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.331+0000 7f1a7259c640 1 --2- 172.21.15.169:0/3597073715 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1a600a2800 0x7f1a600a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1a64000c00 tx=0x7f1a64005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:39.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.335+0000 7f1a6b7fe640 1 -- 172.21.15.169:0/3597073715 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1a6c09c1e0 con 0x7f1a74103270 2024-03-28T00:56:39.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.548+0000 7f1a78b09640 1 -- 172.21.15.169:0/3597073715 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a740622a0 con 0x7f1a74103270 2024-03-28T00:56:39.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:39 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1994609387' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:56:39.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:39 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4205056589' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:56:39.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:39 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4025375528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:56:39.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:39 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1760066720' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:56:39.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.549+0000 7f1a6b7fe640 1 -- 172.21.15.169:0/3597073715 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11420 (secure 0 0 0) 0x7f1a6c09bae0 con 0x7f1a74103270 2024-03-28T00:56:39.552 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:39.552 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:54:52.274335+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24457,24478]},"id":1}]} 2024-03-28T00:56:39.552 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 24 2024-03-28T00:56:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.551+0000 7f1a78b09640 1 -- 172.21.15.169:0/3597073715 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1a600a2800 msgr2=0x7f1a600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.551+0000 7f1a78b09640 1 --2- 172.21.15.169:0/3597073715 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1a600a2800 0x7f1a600a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1a64000c00 tx=0x7f1a64005fd0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.551+0000 7f1a78b09640 1 -- 172.21.15.169:0/3597073715 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a74103270 msgr2=0x7f1a7419ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.551+0000 7f1a78b09640 1 --2- 172.21.15.169:0/3597073715 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a74103270 0x7f1a7419ef60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c00eb10 tx=0x7f1a6c00c2d0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.551+0000 7f1a78b09640 1 -- 172.21.15.169:0/3597073715 shutdown_connections 2024-03-28T00:56:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.551+0000 7f1a78b09640 1 --2- 172.21.15.169:0/3597073715 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1a600a2800 0x7f1a600a4cc0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.551+0000 7f1a78b09640 1 --2- 172.21.15.169:0/3597073715 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1a74103270 0x7f1a7419ef60 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.551+0000 7f1a78b09640 1 --2- 172.21.15.169:0/3597073715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a741028b0 0x7f1a74197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.551+0000 7f1a78b09640 1 --2- 172.21.15.169:0/3597073715 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a741016c0 0x7f1a741979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.551+0000 7f1a78b09640 1 -- 172.21.15.169:0/3597073715 >> 172.21.15.169:0/3597073715 conn(0x7f1a740fcc50 msgr2=0x7f1a74105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:39.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.551+0000 7f1a78b09640 1 -- 172.21.15.169:0/3597073715 shutdown_connections 2024-03-28T00:56:39.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.551+0000 7f1a78b09640 1 -- 172.21.15.169:0/3597073715 wait complete. 2024-03-28T00:56:39.565 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T00:56:39.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:39 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1994609387' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:56:39.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:39 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/4205056589' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:56:39.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:39 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/4025375528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:56:39.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:39 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1760066720' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:56:39.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:39 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1994609387' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:56:39.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:39 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/4205056589' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:56:39.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:39 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/4025375528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:56:39.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:39 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1760066720' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:56:39.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.664+0000 7f758e909640 1 -- 172.21.15.169:0/3040183022 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7588102960 msgr2=0x7f7588102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:39.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.664+0000 7f758e909640 1 --2- 172.21.15.169:0/3040183022 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7588102960 0x7f7588102de0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f757c009930 tx=0x7f757c02bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:39.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.664+0000 7f758e909640 1 -- 172.21.15.169:0/3040183022 shutdown_connections 2024-03-28T00:56:39.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.664+0000 7f758e909640 1 --2- 172.21.15.169:0/3040183022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7588103320 0x7f7588109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.664+0000 7f758e909640 1 --2- 172.21.15.169:0/3040183022 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7588102960 0x7f7588102de0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.664+0000 7f758e909640 1 --2- 172.21.15.169:0/3040183022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75880ff990 0x7f75880ffdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.664+0000 7f758e909640 1 -- 172.21.15.169:0/3040183022 >> 172.21.15.169:0/3040183022 conn(0x7f75880fb2c0 msgr2=0x7f75880fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:39.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.664+0000 7f758e909640 1 -- 172.21.15.169:0/3040183022 shutdown_connections 2024-03-28T00:56:39.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.664+0000 7f758e909640 1 -- 172.21.15.169:0/3040183022 wait complete. 2024-03-28T00:56:39.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.665+0000 7f758e909640 1 Processor -- start 2024-03-28T00:56:39.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.665+0000 7f758e909640 1 -- start start 2024-03-28T00:56:39.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.665+0000 7f758e909640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75880ff990 0x7f7588197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.665+0000 7f758e909640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7588102960 0x7f7588197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.665+0000 7f758e909640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7588103320 0x7f758819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.665+0000 7f758e909640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758810d370 con 0x7f7588103320 2024-03-28T00:56:39.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.665+0000 7f758e909640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758810d1f0 con 0x7f7588102960 2024-03-28T00:56:39.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.665+0000 7f758e909640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758810d4f0 con 0x7f75880ff990 2024-03-28T00:56:39.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.665+0000 7f758cea6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7588103320 0x7f758819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.665+0000 7f758cea6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7588103320 0x7f758819efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51076/0 (socket says 172.21.15.169:51076) 2024-03-28T00:56:39.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.665+0000 7f758cea6640 1 -- 172.21.15.169:0/3443197598 learned_addr learned my addr 172.21.15.169:0/3443197598 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:39.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.665+0000 7f7587fff640 1 --2- 172.21.15.169:0/3443197598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75880ff990 0x7f7588197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.665+0000 7f75877fe640 1 --2- 172.21.15.169:0/3443197598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7588102960 0x7f7588197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.666+0000 7f758cea6640 1 -- 172.21.15.169:0/3443197598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75880ff990 msgr2=0x7f7588197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:39.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.666+0000 7f758cea6640 1 --2- 172.21.15.169:0/3443197598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75880ff990 0x7f7588197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.666+0000 7f758cea6640 1 -- 172.21.15.169:0/3443197598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7588102960 msgr2=0x7f7588197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:39.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.666+0000 7f758cea6640 1 --2- 172.21.15.169:0/3443197598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7588102960 0x7f7588197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.666+0000 7f758cea6640 1 -- 172.21.15.169:0/3443197598 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758819f630 con 0x7f7588103320 2024-03-28T00:56:39.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.666+0000 7f758cea6640 1 --2- 172.21.15.169:0/3443197598 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7588103320 0x7f758819efc0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f758000eb10 tx=0x7f758000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:39.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.666+0000 7f75857fa640 1 -- 172.21.15.169:0/3443197598 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7580004b20 con 0x7f7588103320 2024-03-28T00:56:39.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.666+0000 7f75857fa640 1 -- 172.21.15.169:0/3443197598 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7580004cc0 con 0x7f7588103320 2024-03-28T00:56:39.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.666+0000 7f758e909640 1 -- 172.21.15.169:0/3443197598 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f758819f920 con 0x7f7588103320 2024-03-28T00:56:39.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.666+0000 7f75857fa640 1 -- 172.21.15.169:0/3443197598 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7580010760 con 0x7f7588103320 2024-03-28T00:56:39.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.666+0000 7f758e909640 1 -- 172.21.15.169:0/3443197598 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f758819fe40 con 0x7f7588103320 2024-03-28T00:56:39.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.667+0000 7f758e909640 1 -- 172.21.15.169:0/3443197598 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7588108ba0 con 0x7f7588103320 2024-03-28T00:56:39.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.668+0000 7f75857fa640 1 -- 172.21.15.169:0/3443197598 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7580005020 con 0x7f7588103320 2024-03-28T00:56:39.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.668+0000 7f75857fa640 1 --2- 172.21.15.169:0/3443197598 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f75700a2800 0x7f75700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.669+0000 7f75857fa640 1 -- 172.21.15.169:0/3443197598 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f75800cf4e0 con 0x7f7588103320 2024-03-28T00:56:39.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.669+0000 7f7587fff640 1 --2- 172.21.15.169:0/3443197598 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f75700a2800 0x7f75700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.669+0000 7f7587fff640 1 --2- 172.21.15.169:0/3443197598 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f75700a2800 0x7f75700a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7588069ad0 tx=0x7f7578005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:39.675 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.673+0000 7f75857fa640 1 -- 172.21.15.169:0/3443197598 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f758009c1e0 con 0x7f7588103320 2024-03-28T00:56:39.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.848+0000 7f758e909640 1 -- 172.21.15.169:0/3443197598 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f758802d070 con 0x7f7588103320 2024-03-28T00:56:39.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.850+0000 7f75857fa640 1 -- 172.21.15.169:0/3443197598 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+24823 (secure 0 0 0) 0x7f758009bae0 con 0x7f7588103320 2024-03-28T00:56:39.853 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:39.853 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":94,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:56:38.281826+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9},{"from":5,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:56:39.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.852+0000 7f758e909640 1 -- 172.21.15.169:0/3443197598 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f75700a2800 msgr2=0x7f75700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:39.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.852+0000 7f758e909640 1 --2- 172.21.15.169:0/3443197598 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f75700a2800 0x7f75700a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7588069ad0 tx=0x7f7578005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:39.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.852+0000 7f758e909640 1 -- 172.21.15.169:0/3443197598 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7588103320 msgr2=0x7f758819efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:39.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.852+0000 7f758e909640 1 --2- 172.21.15.169:0/3443197598 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7588103320 0x7f758819efc0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f758000eb10 tx=0x7f758000c2d0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.852+0000 7f758e909640 1 -- 172.21.15.169:0/3443197598 shutdown_connections 2024-03-28T00:56:39.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.852+0000 7f758e909640 1 --2- 172.21.15.169:0/3443197598 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f75700a2800 0x7f75700a4cc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.852+0000 7f758e909640 1 --2- 172.21.15.169:0/3443197598 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7588103320 0x7f758819efc0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.852+0000 7f758e909640 1 --2- 172.21.15.169:0/3443197598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7588102960 0x7f7588197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.852+0000 7f758e909640 1 --2- 172.21.15.169:0/3443197598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75880ff990 0x7f7588197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.852+0000 7f758e909640 1 -- 172.21.15.169:0/3443197598 >> 172.21.15.169:0/3443197598 conn(0x7f75880fb2c0 msgr2=0x7f7588105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:39.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.852+0000 7f758e909640 1 -- 172.21.15.169:0/3443197598 shutdown_connections 2024-03-28T00:56:39.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.852+0000 7f758e909640 1 -- 172.21.15.169:0/3443197598 wait complete. 2024-03-28T00:56:39.867 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T00:54:32.613335+0000', 'modified': '2024-03-28T00:54:52.274335+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24479, 'mds_1': 24478, 'mds_2': 24457}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24419': {'gid': 24419, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6835/3213155599', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6834', 'nonce': 3213155599}, {'type': 'v1', 'addr': '172.21.15.174:6835', 'nonce': 3213155599}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24445': {'gid': 24445, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6835/1050217570', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6834', 'nonce': 1050217570}, {'type': 'v1', 'addr': '172.21.15.177:6835', 'nonce': 1050217570}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24452': {'gid': 24452, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6837/3629228577', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6836', 'nonce': 3629228577}, {'type': 'v1', 'addr': '172.21.15.174:6837', 'nonce': 3629228577}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24457': {'gid': 24457, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.177:6837/3931247435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6836', 'nonce': 3931247435}, {'type': 'v1', 'addr': '172.21.15.177:6837', 'nonce': 3931247435}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24478': {'gid': 24478, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.177:6839/152605729', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6838', 'nonce': 152605729}, {'type': 'v1', 'addr': '172.21.15.177:6839', 'nonce': 152605729}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24479': {'gid': 24479, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.174:6839/1152414321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6838', 'nonce': 1152414321}, {'type': 'v1', 'addr': '172.21.15.174:6839', 'nonce': 1152414321}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24479, 'qdb_cluster': [24479, 24457, 24478]} 2024-03-28T00:56:39.867 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-28T00:56:39.867 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-28T00:56:39.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.968+0000 7f8ecfba9640 1 -- 172.21.15.169:0/787021796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ec81016b0 msgr2=0x7f8ec8101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:39.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.968+0000 7f8ecfba9640 1 --2- 172.21.15.169:0/787021796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ec81016b0 0x7f8ec8101ad0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009900 tx=0x7f8ebc02bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:39.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.968+0000 7f8ecfba9640 1 -- 172.21.15.169:0/787021796 shutdown_connections 2024-03-28T00:56:39.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.968+0000 7f8ecfba9640 1 --2- 172.21.15.169:0/787021796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ec8103260 0x7f8ec8109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.968+0000 7f8ecfba9640 1 --2- 172.21.15.169:0/787021796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ec81028a0 0x7f8ec8102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.968+0000 7f8ecfba9640 1 --2- 172.21.15.169:0/787021796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ec81016b0 0x7f8ec8101ad0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.968+0000 7f8ecfba9640 1 -- 172.21.15.169:0/787021796 >> 172.21.15.169:0/787021796 conn(0x7f8ec80fcc40 msgr2=0x7f8ec80ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:39.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.968+0000 7f8ecfba9640 1 -- 172.21.15.169:0/787021796 shutdown_connections 2024-03-28T00:56:39.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.968+0000 7f8ecfba9640 1 -- 172.21.15.169:0/787021796 wait complete. 2024-03-28T00:56:39.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.968+0000 7f8ecfba9640 1 Processor -- start 2024-03-28T00:56:39.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.968+0000 7f8ecfba9640 1 -- start start 2024-03-28T00:56:39.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.968+0000 7f8ecfba9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ec81016b0 0x7f8ec8197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.968+0000 7f8ecfba9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ec81028a0 0x7f8ec8197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.969+0000 7f8ecfba9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ec8103260 0x7f8ec819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.969+0000 7f8ecfba9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec810d300 con 0x7f8ec8103260 2024-03-28T00:56:39.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.969+0000 7f8ecfba9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec810d180 con 0x7f8ec81028a0 2024-03-28T00:56:39.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.969+0000 7f8ecfba9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec810d480 con 0x7f8ec81016b0 2024-03-28T00:56:39.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.969+0000 7f8ece146640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ec8103260 0x7f8ec819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.969+0000 7f8ece146640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ec8103260 0x7f8ec819efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51090/0 (socket says 172.21.15.169:51090) 2024-03-28T00:56:39.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.969+0000 7f8ece146640 1 -- 172.21.15.169:0/2641153779 learned_addr learned my addr 172.21.15.169:0/2641153779 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:39.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.969+0000 7f8ecd945640 1 --2- 172.21.15.169:0/2641153779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ec81016b0 0x7f8ec8197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.969+0000 7f8ecd144640 1 --2- 172.21.15.169:0/2641153779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ec81028a0 0x7f8ec8197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.969+0000 7f8ece146640 1 -- 172.21.15.169:0/2641153779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ec81016b0 msgr2=0x7f8ec8197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:39.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.969+0000 7f8ece146640 1 --2- 172.21.15.169:0/2641153779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ec81016b0 0x7f8ec8197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.969+0000 7f8ece146640 1 -- 172.21.15.169:0/2641153779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ec81028a0 msgr2=0x7f8ec8197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:39.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.969+0000 7f8ece146640 1 --2- 172.21.15.169:0/2641153779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ec81028a0 0x7f8ec8197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.969+0000 7f8ece146640 1 -- 172.21.15.169:0/2641153779 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec819f650 con 0x7f8ec8103260 2024-03-28T00:56:39.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.969+0000 7f8ece146640 1 --2- 172.21.15.169:0/2641153779 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ec8103260 0x7f8ec819efe0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8ec000efd0 tx=0x7f8ec000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:39.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.969+0000 7f8ebaffd640 1 -- 172.21.15.169:0/2641153779 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0003b70 con 0x7f8ec8103260 2024-03-28T00:56:39.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.969+0000 7f8ebaffd640 1 -- 172.21.15.169:0/2641153779 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ec0003d10 con 0x7f8ec8103260 2024-03-28T00:56:39.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.969+0000 7f8ecfba9640 1 -- 172.21.15.169:0/2641153779 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec819f940 con 0x7f8ec8103260 2024-03-28T00:56:39.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.969+0000 7f8ebaffd640 1 -- 172.21.15.169:0/2641153779 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec00109f0 con 0x7f8ec8103260 2024-03-28T00:56:39.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.969+0000 7f8ecfba9640 1 -- 172.21.15.169:0/2641153779 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec819fe90 con 0x7f8ec8103260 2024-03-28T00:56:39.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.971+0000 7f8ebaffd640 1 -- 172.21.15.169:0/2641153779 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8ec0004020 con 0x7f8ec8103260 2024-03-28T00:56:39.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.972+0000 7f8ed2790740 1 Processor -- start 2024-03-28T00:56:39.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.972+0000 7f8ed2790740 1 -- start start 2024-03-28T00:56:39.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.972+0000 7f8ed2790740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbfea740 0x55bcfbfeab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.972+0000 7f8ed2790740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbfeb130 0x55bcfbfefc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.972+0000 7f8ed2790740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbff02a0 0x55bcfbff2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.972+0000 7f8ed2790740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcfbf4fc20 con 0x55bcfbff02a0 2024-03-28T00:56:39.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.972+0000 7f8ed2790740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcfbef8a40 con 0x55bcfbfeb130 2024-03-28T00:56:39.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.972+0000 7f8ed2790740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcfbf4bcc0 con 0x55bcfbfea740 2024-03-28T00:56:39.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.972+0000 7f8ebaffd640 1 --2- 172.21.15.169:0/2641153779 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8eb40a2880 0x7f8eb40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.972+0000 7f8ecd144640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbff02a0 0x55bcfbff2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.972+0000 7f8ecd945640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbfeb130 0x55bcfbfefc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.972+0000 7f8ecd945640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbfeb130 0x55bcfbfefc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57820/0 (socket says 172.21.15.169:57820) 2024-03-28T00:56:39.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.972+0000 7f8ecd144640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbfea740 0x55bcfbfeab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.972+0000 7f8ecd945640 1 -- 172.21.15.169:0/861048084 learned_addr learned my addr 172.21.15.169:0/861048084 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:39.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.973+0000 7f8ece146640 1 --2- 172.21.15.169:0/2641153779 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8eb40a2880 0x7f8eb40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.973+0000 7f8ebaffd640 1 -- 172.21.15.169:0/2641153779 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f8ec00cf1f0 con 0x7f8ec8103260 2024-03-28T00:56:39.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.973+0000 7f8ecd945640 1 -- 172.21.15.169:0/861048084 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbfea740 msgr2=0x55bcfbfeab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:39.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.973+0000 7f8ecd945640 1 --2- 172.21.15.169:0/861048084 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbfea740 0x55bcfbfeab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.973+0000 7f8ecd945640 1 -- 172.21.15.169:0/861048084 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbff02a0 msgr2=0x55bcfbff2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:39.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.973+0000 7f8ecd945640 1 --2- 172.21.15.169:0/861048084 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbff02a0 0x55bcfbff2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.973+0000 7f8ecd945640 1 -- 172.21.15.169:0/861048084 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcfbf1af90 con 0x55bcfbfeb130 2024-03-28T00:56:39.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.973+0000 7f8ecd144640 1 --2- 172.21.15.169:0/861048084 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbfea740 0x55bcfbfeab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:39.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.973+0000 7f8ece146640 1 --2- 172.21.15.169:0/2641153779 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8eb40a2880 0x7f8eb40a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8ec009b780 tx=0x7f8ec0013700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:39.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.973+0000 7f8ecd945640 1 --2- 172.21.15.169:0/861048084 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbfeb130 0x55bcfbfefc30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009900 tx=0x7f8ebc009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33272bab54ac19d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:39.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.973+0000 7f8eb3fff640 1 -- 172.21.15.169:0/861048084 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc02cb80 con 0x55bcfbfeb130 2024-03-28T00:56:39.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.973+0000 7f8eb3fff640 1 -- 172.21.15.169:0/861048084 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ebc005040 con 0x55bcfbfeb130 2024-03-28T00:56:39.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.973+0000 7f8eb3fff640 1 -- 172.21.15.169:0/861048084 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc004920 con 0x55bcfbfeb130 2024-03-28T00:56:39.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.974+0000 7f8ed2790740 1 -- 172.21.15.169:0/861048084 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbfeb130 msgr2=0x55bcfbfefc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:39.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.974+0000 7f8ed2790740 1 --2- 172.21.15.169:0/861048084 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbfeb130 0x55bcfbfefc30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009900 tx=0x7f8ebc009560 comp rx=0 tx=0).stop 2024-03-28T00:56:39.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.974+0000 7f8ed2790740 1 -- 172.21.15.169:0/861048084 shutdown_connections 2024-03-28T00:56:39.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.974+0000 7f8ed2790740 1 --2- 172.21.15.169:0/861048084 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbff02a0 0x55bcfbff2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.974+0000 7f8ed2790740 1 --2- 172.21.15.169:0/861048084 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbfeb130 0x55bcfbfefc30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.974+0000 7f8ed2790740 1 --2- 172.21.15.169:0/861048084 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbfea740 0x55bcfbfeab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.974+0000 7f8ed2790740 1 -- 172.21.15.169:0/861048084 >> 172.21.15.169:0/861048084 conn(0x55bcfbfb2a60 msgr2=0x55bcfbfe60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:39.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.974+0000 7f8ed2790740 1 -- 172.21.15.169:0/861048084 shutdown_connections 2024-03-28T00:56:39.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.974+0000 7f8ed2790740 1 -- 172.21.15.169:0/861048084 wait complete. 2024-03-28T00:56:39.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.974+0000 7f8ed2790740 1 -- start start 2024-03-28T00:56:39.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.974+0000 7f8ed2790740 1 Processor -- start 2024-03-28T00:56:39.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.975+0000 7f8ed2790740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbfea740 0x55bcfbff4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.975+0000 7f8ed2790740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbfeb130 0x55bcfbff51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.975+0000 7f8ed2790740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbff02a0 0x55bcfbffc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.975+0000 7f8ed2790740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcfbff31f0 con 0x55bcfbfea740 2024-03-28T00:56:39.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.975+0000 7f8ed2790740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcfbf0be90 con 0x55bcfbfeb130 2024-03-28T00:56:39.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.975+0000 7f8ed2790740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcfbff3370 con 0x55bcfbff02a0 2024-03-28T00:56:39.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.975+0000 7f8ed2790740 10 client.0 _ll_register_callbacks cb 0x55bcfbc63ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:56:39.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.975+0000 7f8ecd144640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbfea740 0x55bcfbff4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.975+0000 7f8ecd144640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbfea740 0x55bcfbff4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51110/0 (socket says 172.21.15.169:51110) 2024-03-28T00:56:39.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.975+0000 7f8ecd144640 1 -- 172.21.15.169:0/2915770683 learned_addr learned my addr 172.21.15.169:0/2915770683 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:39.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.975+0000 7f8ecd945640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbfeb130 0x55bcfbff51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.975+0000 7f8ecd144640 1 --2- 172.21.15.169:0/2915770683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbff02a0 0x55bcfbffc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.975+0000 7f8ecd945640 1 -- 172.21.15.169:0/2915770683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbff02a0 msgr2=0x55bcfbffc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:39.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.975+0000 7f8ecd945640 1 --2- 172.21.15.169:0/2915770683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbff02a0 0x55bcfbffc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.975+0000 7f8ecd945640 1 -- 172.21.15.169:0/2915770683 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbfea740 msgr2=0x55bcfbff4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:39.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.975+0000 7f8ecd945640 1 --2- 172.21.15.169:0/2915770683 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbfea740 0x55bcfbff4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.975+0000 7f8ecd945640 1 -- 172.21.15.169:0/2915770683 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bcfbf189d0 con 0x55bcfbfeb130 2024-03-28T00:56:39.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.975+0000 7f8ecd144640 1 --2- 172.21.15.169:0/2915770683 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbfea740 0x55bcfbff4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:39.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.975+0000 7f8ecd945640 1 --2- 172.21.15.169:0/2915770683 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbfeb130 0x55bcfbff51f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x55bcfbfe7000 tx=0x7f8ebc0054a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=109529611f4e413 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:39.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.975+0000 7f8eb17fa640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:56:39.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.976+0000 7f8eb17fa640 1 -- 172.21.15.169:0/2915770683 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc004240 con 0x55bcfbfeb130 2024-03-28T00:56:39.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.976+0000 7f8eb17fa640 1 -- 172.21.15.169:0/2915770683 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f8ebc0043e0 con 0x55bcfbfeb130 2024-03-28T00:56:39.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.976+0000 7f8ed2790740 1 -- 172.21.15.169:0/2915770683 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ebc005040 con 0x55bcfbfeb130 2024-03-28T00:56:39.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.976+0000 7f8eb17fa640 1 -- 172.21.15.169:0/2915770683 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ebc054050 con 0x55bcfbfeb130 2024-03-28T00:56:39.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.976+0000 7f8ed2790740 10 client.25217 fetch_fsmap learned FSMap version 24 2024-03-28T00:56:39.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.976+0000 7f8ed2790740 1 -- 172.21.15.169:0/2915770683 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55bcfbc642d0 con 0x55bcfbfeb130 2024-03-28T00:56:39.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.976+0000 7f8eb17fa640 1 -- 172.21.15.169:0/2915770683 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ebc005040 con 0x55bcfbfeb130 2024-03-28T00:56:39.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.976+0000 7f8eb17fa640 1 -- 172.21.15.169:0/2915770683 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f8ebc0037b0 con 0x55bcfbfeb130 2024-03-28T00:56:39.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.976+0000 7f8ed2790740 10 client.25217 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T00:56:39.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.976+0000 7f8ed2790740 10 client.25217 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T00:56:39.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.976+0000 7f8ed2790740 20 client.25217 populate_metadata read hostname 'smithi169' 2024-03-28T00:56:39.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.976+0000 7f8ed2790740 1 --2- 172.21.15.169:0/2915770683 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55bcfc000320 0x55bcfc0053f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:39.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.976+0000 7f8ed2790740 4 client.25217 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:56:39.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.976+0000 7f8ed2790740 1 -- 172.21.15.169:0/2915770683 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bcfbf5d160 con 0x55bcfc000320 2024-03-28T00:56:39.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.976+0000 7f8ecd144640 1 --2- 172.21.15.169:0/2915770683 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55bcfc000320 0x55bcfc0053f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:39.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.977+0000 7f8eb17fa640 10 client.25217 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T00:56:39.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.977+0000 7f8ecd144640 1 --2- 172.21.15.169:0/2915770683 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55bcfc000320 0x55bcfc0053f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=efce49f3617cc301 server_cookie=fe171a0f4508d7b in_seq=0 out_seq=0 2024-03-28T00:56:39.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.977+0000 7f8eb17fa640 1 -- 172.21.15.169:0/2915770683 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bcfbf5d160 con 0x55bcfc000320 2024-03-28T00:56:39.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.977+0000 7f8eb17fa640 10 client.25217 handle_command_reply: tid=0 2024-03-28T00:56:39.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.977+0000 7f8ed2790740 1 client.25217 shutdown 2024-03-28T00:56:39.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.977+0000 7f8ed2790740 1 -- 172.21.15.169:0/2915770683 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbfeb130 msgr2=0x55bcfbff51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:39.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.977+0000 7f8ed2790740 1 --2- 172.21.15.169:0/2915770683 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbfeb130 0x55bcfbff51f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x55bcfbfe7000 tx=0x7f8ebc0054a0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.978+0000 7f8ed2790740 1 -- 172.21.15.169:0/2915770683 shutdown_connections 2024-03-28T00:56:39.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.978+0000 7f8ed2790740 1 --2- 172.21.15.169:0/2915770683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbff02a0 0x55bcfbffc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.978+0000 7f8ed2790740 1 --2- 172.21.15.169:0/2915770683 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbfeb130 0x55bcfbff51f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.978+0000 7f8ed2790740 1 --2- 172.21.15.169:0/2915770683 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55bcfc000320 0x55bcfc0053f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.978+0000 7f8ed2790740 1 --2- 172.21.15.169:0/2915770683 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbfea740 0x55bcfbff4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:39.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.978+0000 7f8eb17fa640 0 client.25217 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T00:56:39.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.978+0000 7f8ed2790740 1 -- 172.21.15.169:0/2915770683 >> 172.21.15.169:0/2915770683 conn(0x55bcfbfb2a60 msgr2=0x55bcfbfe52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:39.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.978+0000 7f8ed2790740 1 -- 172.21.15.169:0/2915770683 shutdown_connections 2024-03-28T00:56:39.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.978+0000 7f8ed2790740 1 -- 172.21.15.169:0/2915770683 wait complete. 2024-03-28T00:56:39.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.978+0000 7f8ed2790740 20 client.25217 trim_cache size 0 max 16384 2024-03-28T00:56:40.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.994+0000 7f8ed2790740 1 Processor -- start 2024-03-28T00:56:40.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.994+0000 7f8ed2790740 1 -- start start 2024-03-28T00:56:40.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.994+0000 7f8ed2790740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbfea740 0x55bcfbfec670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.994+0000 7f8ed2790740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbfeb130 0x55bcfbffc180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.994+0000 7f8ed2790740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbff02a0 0x55bcfbfecbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.994+0000 7f8ed2790740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcfc000790 con 0x55bcfbfea740 2024-03-28T00:56:40.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.994+0000 7f8ed2790740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcfc000df0 con 0x55bcfbff02a0 2024-03-28T00:56:40.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.994+0000 7f8ed2790740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcfc001450 con 0x55bcfbfeb130 2024-03-28T00:56:40.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.994+0000 7f8ecd144640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbfea740 0x55bcfbfec670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.994+0000 7f8ecd144640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbfea740 0x55bcfbfec670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51114/0 (socket says 172.21.15.169:51114) 2024-03-28T00:56:40.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.994+0000 7f8ecd144640 1 -- 172.21.15.169:0/2346255729 learned_addr learned my addr 172.21.15.169:0/2346255729 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:40.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.994+0000 7f8ecd945640 1 --2- 172.21.15.169:0/2346255729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbfeb130 0x55bcfbffc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.995+0000 7f8ecd144640 1 --2- 172.21.15.169:0/2346255729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbff02a0 0x55bcfbfecbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.995+0000 7f8ecd144640 1 -- 172.21.15.169:0/2346255729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbfeb130 msgr2=0x55bcfbffc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:40.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.995+0000 7f8ecd144640 1 --2- 172.21.15.169:0/2346255729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbfeb130 0x55bcfbffc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.995+0000 7f8ecd144640 1 -- 172.21.15.169:0/2346255729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbff02a0 msgr2=0x55bcfbfecbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:40.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.995+0000 7f8ecd144640 1 --2- 172.21.15.169:0/2346255729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbff02a0 0x55bcfbfecbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.995+0000 7f8ecd144640 1 -- 172.21.15.169:0/2346255729 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcfbd2e6b0 con 0x55bcfbfea740 2024-03-28T00:56:40.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.995+0000 7f8ecd945640 1 --2- 172.21.15.169:0/2346255729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbfeb130 0x55bcfbffc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:40.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.995+0000 7f8ecd144640 1 --2- 172.21.15.169:0/2346255729 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbfea740 0x55bcfbfec670 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x55bcfc002bc0 tx=0x7f8ec4009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=307fcc42ea78096a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:40.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.995+0000 7f8eb37fe640 1 -- 172.21.15.169:0/2346255729 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4019040 con 0x55bcfbfea740 2024-03-28T00:56:40.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.995+0000 7f8eb37fe640 1 -- 172.21.15.169:0/2346255729 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ec40047a0 con 0x55bcfbfea740 2024-03-28T00:56:40.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.995+0000 7f8eb37fe640 1 -- 172.21.15.169:0/2346255729 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4022650 con 0x55bcfbfea740 2024-03-28T00:56:40.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ed2790740 1 -- 172.21.15.169:0/2346255729 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbfea740 msgr2=0x55bcfbfec670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ed2790740 1 --2- 172.21.15.169:0/2346255729 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbfea740 0x55bcfbfec670 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x55bcfc002bc0 tx=0x7f8ec4009b40 comp rx=0 tx=0).stop 2024-03-28T00:56:40.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ed2790740 1 -- 172.21.15.169:0/2346255729 shutdown_connections 2024-03-28T00:56:40.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ed2790740 1 --2- 172.21.15.169:0/2346255729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbff02a0 0x55bcfbfecbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ed2790740 1 --2- 172.21.15.169:0/2346255729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbfeb130 0x55bcfbffc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ed2790740 1 --2- 172.21.15.169:0/2346255729 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbfea740 0x55bcfbfec670 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ed2790740 1 -- 172.21.15.169:0/2346255729 >> 172.21.15.169:0/2346255729 conn(0x55bcfbfb2a60 msgr2=0x55bcfbff37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:40.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ed2790740 1 -- 172.21.15.169:0/2346255729 shutdown_connections 2024-03-28T00:56:40.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ed2790740 1 -- 172.21.15.169:0/2346255729 wait complete. 2024-03-28T00:56:40.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ed2790740 1 -- start start 2024-03-28T00:56:40.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ed2790740 1 Processor -- start 2024-03-28T00:56:40.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ed2790740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbfea740 0x55bcfbff78c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ed2790740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbfeb130 0x55bcfbff7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ed2790740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbff02a0 0x55bcfbffae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ed2790740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcfc002110 con 0x55bcfbff02a0 2024-03-28T00:56:40.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ed2790740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcfc001ab0 con 0x55bcfbfeb130 2024-03-28T00:56:40.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ed2790740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcfc002770 con 0x55bcfbfea740 2024-03-28T00:56:40.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ed2790740 10 client.0 _ll_register_callbacks cb 0x7f8ec4009dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:56:40.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ecd144640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbff02a0 0x55bcfbffae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ecd144640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbff02a0 0x55bcfbffae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51118/0 (socket says 172.21.15.169:51118) 2024-03-28T00:56:40.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ecd144640 1 -- 172.21.15.169:0/3236116865 learned_addr learned my addr 172.21.15.169:0/3236116865 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:40.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.996+0000 7f8ecd144640 1 --2- 172.21.15.169:0/3236116865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbfea740 0x55bcfbff78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.997+0000 7f8ecd945640 1 --2- 172.21.15.169:0/3236116865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbfeb130 0x55bcfbff7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.997+0000 7f8ecd945640 1 -- 172.21.15.169:0/3236116865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbfea740 msgr2=0x55bcfbff78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:40.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.997+0000 7f8ecd945640 1 --2- 172.21.15.169:0/3236116865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbfea740 0x55bcfbff78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.997+0000 7f8ecd945640 1 -- 172.21.15.169:0/3236116865 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbff02a0 msgr2=0x55bcfbffae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:40.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.997+0000 7f8ecd945640 1 --2- 172.21.15.169:0/3236116865 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbff02a0 0x55bcfbffae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.997+0000 7f8ecd945640 1 -- 172.21.15.169:0/3236116865 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bcfbffb410 con 0x55bcfbfeb130 2024-03-28T00:56:40.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.997+0000 7f8ecd144640 1 --2- 172.21.15.169:0/3236116865 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbff02a0 0x55bcfbffae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:40.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.997+0000 7f8ecd945640 1 --2- 172.21.15.169:0/3236116865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbfeb130 0x55bcfbff7e00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc02e000 tx=0x7f8ebc005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e6725cd722a6cec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:40.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.997+0000 7f8eb17fa640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:56:40.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.997+0000 7f8eb17fa640 1 -- 172.21.15.169:0/3236116865 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc05d060 con 0x55bcfbfeb130 2024-03-28T00:56:40.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.997+0000 7f8ed2790740 1 -- 172.21.15.169:0/3236116865 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ec40047a0 con 0x55bcfbfeb130 2024-03-28T00:56:40.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.997+0000 7f8eb17fa640 1 -- 172.21.15.169:0/3236116865 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f8ebc004240 con 0x55bcfbfeb130 2024-03-28T00:56:40.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.997+0000 7f8eb17fa640 1 -- 172.21.15.169:0/3236116865 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ebc064050 con 0x55bcfbfeb130 2024-03-28T00:56:40.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.997+0000 7f8ed2790740 10 client.25223 fetch_fsmap learned FSMap version 24 2024-03-28T00:56:40.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.997+0000 7f8ed2790740 1 -- 172.21.15.169:0/3236116865 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55bcfc05c570 con 0x55bcfbfeb130 2024-03-28T00:56:40.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.997+0000 7f8eb17fa640 1 -- 172.21.15.169:0/3236116865 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ec40047a0 con 0x55bcfbfeb130 2024-03-28T00:56:40.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.998+0000 7f8eb17fa640 1 -- 172.21.15.169:0/3236116865 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f8ebc034510 con 0x55bcfbfeb130 2024-03-28T00:56:40.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.998+0000 7f8ed2790740 10 client.25223 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T00:56:40.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.998+0000 7f8ed2790740 10 client.25223 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T00:56:40.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.998+0000 7f8ed2790740 20 client.25223 populate_metadata read hostname 'smithi169' 2024-03-28T00:56:40.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.998+0000 7f8ed2790740 1 --2- 172.21.15.169:0/3236116865 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55bcfc060b40 0x55bcfc05c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.998+0000 7f8ed2790740 4 client.25223 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T00:56:40.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.998+0000 7f8ed2790740 1 -- 172.21.15.169:0/3236116865 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bcfbf574b0 con 0x55bcfc060b40 2024-03-28T00:56:40.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.998+0000 7f8ecd144640 1 --2- 172.21.15.169:0/3236116865 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55bcfc060b40 0x55bcfc05c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.998+0000 7f8eb17fa640 10 client.25223 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T00:56:40.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.998+0000 7f8ecd144640 1 --2- 172.21.15.169:0/3236116865 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55bcfc060b40 0x55bcfc05c7a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=87419635a8d5223f server_cookie=84e432dcafffdd18 in_seq=0 out_seq=0 2024-03-28T00:56:40.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.998+0000 7f8eb17fa640 1 -- 172.21.15.169:0/3236116865 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55bcfbf574b0 con 0x55bcfc060b40 2024-03-28T00:56:40.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.998+0000 7f8eb17fa640 10 client.25223 handle_command_reply: tid=0 2024-03-28T00:56:40.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.998+0000 7f8ed2790740 1 client.25223 shutdown 2024-03-28T00:56:40.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.998+0000 7f8ed2790740 1 -- 172.21.15.169:0/3236116865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbfeb130 msgr2=0x55bcfbff7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.998+0000 7f8ed2790740 1 --2- 172.21.15.169:0/3236116865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbfeb130 0x55bcfbff7e00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc02e000 tx=0x7f8ebc005b50 comp rx=0 tx=0).stop 2024-03-28T00:56:40.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8ed2790740 1 -- 172.21.15.169:0/3236116865 shutdown_connections 2024-03-28T00:56:40.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8ed2790740 1 --2- 172.21.15.169:0/3236116865 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55bcfc060b40 0x55bcfc05c7a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8ed2790740 1 --2- 172.21.15.169:0/3236116865 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bcfbff02a0 0x55bcfbffae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8ed2790740 1 --2- 172.21.15.169:0/3236116865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bcfbfeb130 0x55bcfbff7e00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8ed2790740 1 --2- 172.21.15.169:0/3236116865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bcfbfea740 0x55bcfbff78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8eb17fa640 0 client.25223 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T00:56:40.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8ed2790740 1 -- 172.21.15.169:0/3236116865 >> 172.21.15.169:0/3236116865 conn(0x55bcfbfb2a60 msgr2=0x55bcfbfe5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:40.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8ed2790740 1 -- 172.21.15.169:0/3236116865 shutdown_connections 2024-03-28T00:56:40.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8ed2790740 1 -- 172.21.15.169:0/3236116865 wait complete. 2024-03-28T00:56:40.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8ed2790740 20 client.25223 trim_cache size 0 max 16384 2024-03-28T00:56:40.011 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:56:40.011 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_fsid": "833368da-ec9c-11ee-95d0-87774f69a715", 2024-03-28T00:56:40.011 INFO:teuthology.orchestra.run.smithi169.stdout: "whoami": 2, 2024-03-28T00:56:40.012 INFO:teuthology.orchestra.run.smithi169.stdout: "id": 24457, 2024-03-28T00:56:40.012 INFO:teuthology.orchestra.run.smithi169.stdout: "want_state": "up:active", 2024-03-28T00:56:40.012 INFO:teuthology.orchestra.run.smithi169.stdout: "state": "up:active", 2024-03-28T00:56:40.012 INFO:teuthology.orchestra.run.smithi169.stdout: "fs_name": "cephfs", 2024-03-28T00:56:40.012 INFO:teuthology.orchestra.run.smithi169.stdout: "rank_uptime": 108.73078374799999, 2024-03-28T00:56:40.012 INFO:teuthology.orchestra.run.smithi169.stdout: "mdsmap_epoch": 24, 2024-03-28T00:56:40.012 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_epoch": 94, 2024-03-28T00:56:40.012 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_epoch_barrier": 90, 2024-03-28T00:56:40.012 INFO:teuthology.orchestra.run.smithi169.stdout: "uptime": 145.251400682 2024-03-28T00:56:40.012 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:56:40.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8ecfba9640 1 -- 172.21.15.169:0/2641153779 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8eb40a2880 msgr2=0x7f8eb40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8ecfba9640 1 --2- 172.21.15.169:0/2641153779 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8eb40a2880 0x7f8eb40a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8ec009b780 tx=0x7f8ec0013700 comp rx=0 tx=0).stop 2024-03-28T00:56:40.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8ecfba9640 1 -- 172.21.15.169:0/2641153779 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ec8103260 msgr2=0x7f8ec819efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8ecfba9640 1 --2- 172.21.15.169:0/2641153779 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ec8103260 0x7f8ec819efe0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8ec000efd0 tx=0x7f8ec000c6a0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8ecfba9640 1 -- 172.21.15.169:0/2641153779 shutdown_connections 2024-03-28T00:56:40.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8ecfba9640 1 --2- 172.21.15.169:0/2641153779 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8eb40a2880 0x7f8eb40a4d40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8ecfba9640 1 --2- 172.21.15.169:0/2641153779 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ec8103260 0x7f8ec819efe0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8ecfba9640 1 --2- 172.21.15.169:0/2641153779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ec81028a0 0x7f8ec8197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8ecfba9640 1 --2- 172.21.15.169:0/2641153779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ec81016b0 0x7f8ec8197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8ecfba9640 1 -- 172.21.15.169:0/2641153779 >> 172.21.15.169:0/2641153779 conn(0x7f8ec80fcc40 msgr2=0x7f8ec8105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:40.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8ecfba9640 1 -- 172.21.15.169:0/2641153779 shutdown_connections 2024-03-28T00:56:40.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:39.999+0000 7f8ecfba9640 1 -- 172.21.15.169:0/2641153779 wait complete. 2024-03-28T00:56:40.015 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-28T00:56:40.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.118+0000 7f8b813f3640 1 -- 172.21.15.169:0/1842810030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b7c1017a0 msgr2=0x7f8b7c101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.118+0000 7f8b813f3640 1 --2- 172.21.15.169:0/1842810030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b7c1017a0 0x7f8b7c101c20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8b64009860 tx=0x7f8b6402bb70 comp rx=0 tx=0).stop 2024-03-28T00:56:40.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.118+0000 7f8b813f3640 1 -- 172.21.15.169:0/1842810030 shutdown_connections 2024-03-28T00:56:40.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.118+0000 7f8b813f3640 1 --2- 172.21.15.169:0/1842810030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b7c102160 0x7f8b7c1089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.118+0000 7f8b813f3640 1 --2- 172.21.15.169:0/1842810030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b7c1017a0 0x7f8b7c101c20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.118+0000 7f8b813f3640 1 --2- 172.21.15.169:0/1842810030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b7c1005b0 0x7f8b7c1009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.118+0000 7f8b813f3640 1 -- 172.21.15.169:0/1842810030 >> 172.21.15.169:0/1842810030 conn(0x7f8b7c0fbb40 msgr2=0x7f8b7c0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:40.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.118+0000 7f8b813f3640 1 -- 172.21.15.169:0/1842810030 shutdown_connections 2024-03-28T00:56:40.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.118+0000 7f8b813f3640 1 -- 172.21.15.169:0/1842810030 wait complete. 2024-03-28T00:56:40.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.119+0000 7f8b813f3640 1 Processor -- start 2024-03-28T00:56:40.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.119+0000 7f8b813f3640 1 -- start start 2024-03-28T00:56:40.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.119+0000 7f8b813f3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b7c1005b0 0x7f8b7c1968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.119+0000 7f8b813f3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b7c1017a0 0x7f8b7c196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.119+0000 7f8b813f3640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b7c102160 0x7f8b7c19de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.119+0000 7f8b813f3640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c10c210 con 0x7f8b7c102160 2024-03-28T00:56:40.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.119+0000 7f8b813f3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c10c090 con 0x7f8b7c1017a0 2024-03-28T00:56:40.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.119+0000 7f8b813f3640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c10c390 con 0x7f8b7c1005b0 2024-03-28T00:56:40.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.119+0000 7f8b7b7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b7c102160 0x7f8b7c19de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.119+0000 7f8b7b7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b7c102160 0x7f8b7c19de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51140/0 (socket says 172.21.15.169:51140) 2024-03-28T00:56:40.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.119+0000 7f8b73fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b7c1017a0 0x7f8b7c196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.119+0000 7f8b7affd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b7c1005b0 0x7f8b7c1968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.119+0000 7f8b7b7fe640 1 -- 172.21.15.169:0/2042639650 learned_addr learned my addr 172.21.15.169:0/2042639650 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:40.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.119+0000 7f8b7b7fe640 1 -- 172.21.15.169:0/2042639650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b7c1005b0 msgr2=0x7f8b7c1968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.119+0000 7f8b7b7fe640 1 --2- 172.21.15.169:0/2042639650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b7c1005b0 0x7f8b7c1968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.119+0000 7f8b7b7fe640 1 -- 172.21.15.169:0/2042639650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b7c1017a0 msgr2=0x7f8b7c196e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.119+0000 7f8b7b7fe640 1 --2- 172.21.15.169:0/2042639650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b7c1017a0 0x7f8b7c196e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.119+0000 7f8b7b7fe640 1 -- 172.21.15.169:0/2042639650 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b7c19e500 con 0x7f8b7c102160 2024-03-28T00:56:40.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.119+0000 7f8b7affd640 1 --2- 172.21.15.169:0/2042639650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b7c1005b0 0x7f8b7c1968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:40.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.119+0000 7f8b73fff640 1 --2- 172.21.15.169:0/2042639650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b7c1017a0 0x7f8b7c196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:40.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.120+0000 7f8b7b7fe640 1 --2- 172.21.15.169:0/2042639650 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b7c102160 0x7f8b7c19de90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8b7400b5e0 tx=0x7f8b7400baa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:40.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.120+0000 7f8b78ff9640 1 -- 172.21.15.169:0/2042639650 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b74003970 con 0x7f8b7c102160 2024-03-28T00:56:40.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.120+0000 7f8b813f3640 1 -- 172.21.15.169:0/2042639650 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b7c069cf0 con 0x7f8b7c102160 2024-03-28T00:56:40.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.120+0000 7f8b78ff9640 1 -- 172.21.15.169:0/2042639650 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b7400a680 con 0x7f8b7c102160 2024-03-28T00:56:40.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.120+0000 7f8b78ff9640 1 -- 172.21.15.169:0/2042639650 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b740044c0 con 0x7f8b7c102160 2024-03-28T00:56:40.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.120+0000 7f8b813f3640 1 -- 172.21.15.169:0/2042639650 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b7c06a1c0 con 0x7f8b7c102160 2024-03-28T00:56:40.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.122+0000 7f8b78ff9640 1 -- 172.21.15.169:0/2042639650 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8b74004660 con 0x7f8b7c102160 2024-03-28T00:56:40.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.122+0000 7f8b83fd6740 1 Processor -- start 2024-03-28T00:56:40.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.122+0000 7f8b83fd6740 1 -- start start 2024-03-28T00:56:40.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.122+0000 7f8b83fd6740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44602740 0x559f44602b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b83fd6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f44603130 0x559f44607c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b83fd6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f446082a0 0x559f4460a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b83fd6740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f44567c20 con 0x559f44602740 2024-03-28T00:56:40.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b83fd6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f44510a40 con 0x559f44603130 2024-03-28T00:56:40.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b83fd6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f44563cc0 con 0x559f446082a0 2024-03-28T00:56:40.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b73fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44602740 0x559f44602b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b78ff9640 1 --2- 172.21.15.169:0/2042639650 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8b680a2800 0x7f8b680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b73fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44602740 0x559f44602b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51154/0 (socket says 172.21.15.169:51154) 2024-03-28T00:56:40.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b73fff640 1 -- 172.21.15.169:0/4035300964 learned_addr learned my addr 172.21.15.169:0/4035300964 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:40.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b7affd640 1 --2- 172.21.15.169:0/4035300964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f44603130 0x559f44607c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b73fff640 1 --2- 172.21.15.169:0/4035300964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f446082a0 0x559f4460a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b7b7fe640 1 --2- 172.21.15.169:0/2042639650 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8b680a2800 0x7f8b680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b73fff640 1 -- 172.21.15.169:0/4035300964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f446082a0 msgr2=0x559f4460a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:40.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b73fff640 1 --2- 172.21.15.169:0/4035300964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f446082a0 0x559f4460a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b73fff640 1 -- 172.21.15.169:0/4035300964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f44603130 msgr2=0x559f44607c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:40.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b73fff640 1 --2- 172.21.15.169:0/4035300964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f44603130 0x559f44607c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b73fff640 1 -- 172.21.15.169:0/4035300964 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f44532f90 con 0x559f44602740 2024-03-28T00:56:40.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b78ff9640 1 -- 172.21.15.169:0/2042639650 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f8b740d0cb0 con 0x7f8b7c102160 2024-03-28T00:56:40.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b7affd640 1 --2- 172.21.15.169:0/4035300964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f44603130 0x559f44607c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:40.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b73fff640 1 --2- 172.21.15.169:0/4035300964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44602740 0x559f44602b60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f8b64037040 tx=0x7f8b64034ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a8103984e2c3f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:40.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b7b7fe640 1 --2- 172.21.15.169:0/2042639650 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8b680a2800 0x7f8b680a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8b740a5520 tx=0x7f8b74068120 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:40.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b717fa640 1 -- 172.21.15.169:0/4035300964 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b64039040 con 0x559f44602740 2024-03-28T00:56:40.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b717fa640 1 -- 172.21.15.169:0/4035300964 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b64034740 con 0x559f44602740 2024-03-28T00:56:40.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.123+0000 7f8b717fa640 1 -- 172.21.15.169:0/4035300964 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b64032640 con 0x559f44602740 2024-03-28T00:56:40.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.124+0000 7f8b83fd6740 1 -- 172.21.15.169:0/4035300964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44602740 msgr2=0x559f44602b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.124+0000 7f8b83fd6740 1 --2- 172.21.15.169:0/4035300964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44602740 0x559f44602b60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f8b64037040 tx=0x7f8b64034ad0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.124+0000 7f8b83fd6740 1 -- 172.21.15.169:0/4035300964 shutdown_connections 2024-03-28T00:56:40.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.124+0000 7f8b83fd6740 1 --2- 172.21.15.169:0/4035300964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f446082a0 0x559f4460a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.124+0000 7f8b83fd6740 1 --2- 172.21.15.169:0/4035300964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f44603130 0x559f44607c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.124+0000 7f8b83fd6740 1 --2- 172.21.15.169:0/4035300964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44602740 0x559f44602b60 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.124+0000 7f8b83fd6740 1 -- 172.21.15.169:0/4035300964 >> 172.21.15.169:0/4035300964 conn(0x559f445caa60 msgr2=0x559f445fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:40.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.124+0000 7f8b83fd6740 1 -- 172.21.15.169:0/4035300964 shutdown_connections 2024-03-28T00:56:40.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.124+0000 7f8b83fd6740 1 -- 172.21.15.169:0/4035300964 wait complete. 2024-03-28T00:56:40.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.124+0000 7f8b83fd6740 1 -- start start 2024-03-28T00:56:40.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b83fd6740 1 Processor -- start 2024-03-28T00:56:40.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b83fd6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f44602740 0x559f4460cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b83fd6740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44603130 0x559f4460d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b83fd6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f446082a0 0x559f446141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b83fd6740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f4460b1f0 con 0x559f44603130 2024-03-28T00:56:40.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b83fd6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f44523e90 con 0x559f446082a0 2024-03-28T00:56:40.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b83fd6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f4460b370 con 0x559f44602740 2024-03-28T00:56:40.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b83fd6740 10 client.0 _ll_register_callbacks cb 0x559f4427bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:56:40.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b7affd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44603130 0x559f4460d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b7affd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44603130 0x559f4460d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51158/0 (socket says 172.21.15.169:51158) 2024-03-28T00:56:40.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b7affd640 1 -- 172.21.15.169:0/1172391939 learned_addr learned my addr 172.21.15.169:0/1172391939 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:40.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b73fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f44602740 0x559f4460cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b73fff640 1 --2- 172.21.15.169:0/1172391939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f446082a0 0x559f446141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b7affd640 1 -- 172.21.15.169:0/1172391939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f44602740 msgr2=0x559f4460cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:40.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b7affd640 1 --2- 172.21.15.169:0/1172391939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f44602740 0x559f4460cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b7affd640 1 -- 172.21.15.169:0/1172391939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f446082a0 msgr2=0x559f446141d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:40.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b7affd640 1 --2- 172.21.15.169:0/1172391939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f446082a0 0x559f446141d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b7affd640 1 -- 172.21.15.169:0/1172391939 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559f445309d0 con 0x559f44603130 2024-03-28T00:56:40.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b7affd640 1 --2- 172.21.15.169:0/1172391939 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44603130 0x559f4460d190 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c012d20 tx=0x7f8b6c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54df27a3b9cbbf10 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:40.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b627fc640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:56:40.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b627fc640 1 -- 172.21.15.169:0/1172391939 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6c0120d0 con 0x559f44603130 2024-03-28T00:56:40.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.125+0000 7f8b627fc640 1 -- 172.21.15.169:0/1172391939 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f8b6c0047a0 con 0x559f44603130 2024-03-28T00:56:40.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.126+0000 7f8b83fd6740 1 -- 172.21.15.169:0/1172391939 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8b64034740 con 0x559f44603130 2024-03-28T00:56:40.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.126+0000 7f8b627fc640 1 -- 172.21.15.169:0/1172391939 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b6c025050 con 0x559f44603130 2024-03-28T00:56:40.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.126+0000 7f8b83fd6740 10 client.15411 fetch_fsmap learned FSMap version 24 2024-03-28T00:56:40.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.126+0000 7f8b83fd6740 1 -- 172.21.15.169:0/1172391939 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x559f4427c2d0 con 0x559f44603130 2024-03-28T00:56:40.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.126+0000 7f8b627fc640 1 -- 172.21.15.169:0/1172391939 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b64034740 con 0x559f44603130 2024-03-28T00:56:40.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.126+0000 7f8b627fc640 1 -- 172.21.15.169:0/1172391939 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f8b6c010430 con 0x559f44603130 2024-03-28T00:56:40.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.126+0000 7f8b83fd6740 10 client.15411 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T00:56:40.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.126+0000 7f8b83fd6740 10 client.15411 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T00:56:40.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.126+0000 7f8b83fd6740 20 client.15411 populate_metadata read hostname 'smithi169' 2024-03-28T00:56:40.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.126+0000 7f8b83fd6740 1 --2- 172.21.15.169:0/1172391939 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x559f446194c0 0x559f44618570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.126+0000 7f8b83fd6740 4 client.15411 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:56:40.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.126+0000 7f8b83fd6740 1 -- 172.21.15.169:0/1172391939 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559f44575160 con 0x559f446194c0 2024-03-28T00:56:40.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.126+0000 7f8b73fff640 1 --2- 172.21.15.169:0/1172391939 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x559f446194c0 0x559f44618570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.127+0000 7f8b73fff640 1 --2- 172.21.15.169:0/1172391939 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x559f446194c0 0x559f44618570 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=23dd325c8eb7d4a server_cookie=ae7d1e4b76ae247f in_seq=0 out_seq=0 2024-03-28T00:56:40.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.127+0000 7f8b627fc640 10 client.15411 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T00:56:40.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.127+0000 7f8b627fc640 1 -- 172.21.15.169:0/1172391939 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559f44575160 con 0x559f446194c0 2024-03-28T00:56:40.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.127+0000 7f8b627fc640 10 client.15411 handle_command_reply: tid=0 2024-03-28T00:56:40.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.128+0000 7f8b83fd6740 1 client.15411 shutdown 2024-03-28T00:56:40.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.128+0000 7f8b83fd6740 1 -- 172.21.15.169:0/1172391939 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44603130 msgr2=0x559f4460d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.128+0000 7f8b83fd6740 1 --2- 172.21.15.169:0/1172391939 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44603130 0x559f4460d190 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c012d20 tx=0x7f8b6c00f790 comp rx=0 tx=0).stop 2024-03-28T00:56:40.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.128+0000 7f8b83fd6740 1 -- 172.21.15.169:0/1172391939 shutdown_connections 2024-03-28T00:56:40.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.128+0000 7f8b83fd6740 1 --2- 172.21.15.169:0/1172391939 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x559f446194c0 0x559f44618570 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.128+0000 7f8b83fd6740 1 --2- 172.21.15.169:0/1172391939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f446082a0 0x559f446141d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.128+0000 7f8b83fd6740 1 --2- 172.21.15.169:0/1172391939 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44603130 0x559f4460d190 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.128+0000 7f8b627fc640 0 client.15411 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T00:56:40.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.128+0000 7f8b83fd6740 1 --2- 172.21.15.169:0/1172391939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f44602740 0x559f4460cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.128+0000 7f8b83fd6740 1 -- 172.21.15.169:0/1172391939 >> 172.21.15.169:0/1172391939 conn(0x559f445caa60 msgr2=0x559f445fd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:40.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.128+0000 7f8b83fd6740 1 -- 172.21.15.169:0/1172391939 shutdown_connections 2024-03-28T00:56:40.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.128+0000 7f8b83fd6740 1 -- 172.21.15.169:0/1172391939 wait complete. 2024-03-28T00:56:40.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.128+0000 7f8b83fd6740 20 client.15411 trim_cache size 0 max 16384 2024-03-28T00:56:40.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.144+0000 7f8b83fd6740 1 Processor -- start 2024-03-28T00:56:40.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.144+0000 7f8b83fd6740 1 -- start start 2024-03-28T00:56:40.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.144+0000 7f8b83fd6740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44602740 0x559f446012f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.144+0000 7f8b83fd6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f44603130 0x559f44614190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.145+0000 7f8b83fd6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f446082a0 0x559f44601830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.145+0000 7f8b83fd6740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f44618ce0 con 0x559f44602740 2024-03-28T00:56:40.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.145+0000 7f8b83fd6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f44619340 con 0x559f446082a0 2024-03-28T00:56:40.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.145+0000 7f8b83fd6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f446199a0 con 0x559f44603130 2024-03-28T00:56:40.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.145+0000 7f8b73fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44602740 0x559f446012f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.145+0000 7f8b73fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44602740 0x559f446012f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51168/0 (socket says 172.21.15.169:51168) 2024-03-28T00:56:40.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.145+0000 7f8b73fff640 1 -- 172.21.15.169:0/2638141212 learned_addr learned my addr 172.21.15.169:0/2638141212 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:40.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.145+0000 7f8b7affd640 1 --2- 172.21.15.169:0/2638141212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f44603130 0x559f44614190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.145+0000 7f8b73fff640 1 --2- 172.21.15.169:0/2638141212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f446082a0 0x559f44601830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.145+0000 7f8b73fff640 1 -- 172.21.15.169:0/2638141212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f44603130 msgr2=0x559f44614190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:40.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.145+0000 7f8b73fff640 1 --2- 172.21.15.169:0/2638141212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f44603130 0x559f44614190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.145+0000 7f8b73fff640 1 -- 172.21.15.169:0/2638141212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f446082a0 msgr2=0x559f44601830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:40.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.145+0000 7f8b73fff640 1 --2- 172.21.15.169:0/2638141212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f446082a0 0x559f44601830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.145+0000 7f8b73fff640 1 -- 172.21.15.169:0/2638141212 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f443466b0 con 0x559f44602740 2024-03-28T00:56:40.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.145+0000 7f8b7affd640 1 --2- 172.21.15.169:0/2638141212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f44603130 0x559f44614190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:40.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.145+0000 7f8b73fff640 1 --2- 172.21.15.169:0/2638141212 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44602740 0x559f446012f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8b640349e0 tx=0x7f8b64050040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=417cf698e0509b5a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:40.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.145+0000 7f8b70ff9640 1 -- 172.21.15.169:0/2638141212 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b64039040 con 0x559f44602740 2024-03-28T00:56:40.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.145+0000 7f8b70ff9640 1 -- 172.21.15.169:0/2638141212 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b640582e0 con 0x559f44602740 2024-03-28T00:56:40.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.145+0000 7f8b70ff9640 1 -- 172.21.15.169:0/2638141212 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b64032930 con 0x559f44602740 2024-03-28T00:56:40.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.146+0000 7f8b83fd6740 1 -- 172.21.15.169:0/2638141212 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44602740 msgr2=0x559f446012f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.146+0000 7f8b83fd6740 1 --2- 172.21.15.169:0/2638141212 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44602740 0x559f446012f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8b640349e0 tx=0x7f8b64050040 comp rx=0 tx=0).stop 2024-03-28T00:56:40.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.146+0000 7f8b83fd6740 1 -- 172.21.15.169:0/2638141212 shutdown_connections 2024-03-28T00:56:40.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.146+0000 7f8b83fd6740 1 --2- 172.21.15.169:0/2638141212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f446082a0 0x559f44601830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.146+0000 7f8b83fd6740 1 --2- 172.21.15.169:0/2638141212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f44603130 0x559f44614190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.146+0000 7f8b83fd6740 1 --2- 172.21.15.169:0/2638141212 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44602740 0x559f446012f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.146+0000 7f8b83fd6740 1 -- 172.21.15.169:0/2638141212 >> 172.21.15.169:0/2638141212 conn(0x559f445caa60 msgr2=0x559f4460b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:40.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.146+0000 7f8b83fd6740 1 -- 172.21.15.169:0/2638141212 shutdown_connections 2024-03-28T00:56:40.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.146+0000 7f8b83fd6740 1 -- 172.21.15.169:0/2638141212 wait complete. 2024-03-28T00:56:40.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.146+0000 7f8b83fd6740 1 -- start start 2024-03-28T00:56:40.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.146+0000 7f8b83fd6740 1 Processor -- start 2024-03-28T00:56:40.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b83fd6740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44602740 0x559f446103c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b83fd6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f44603130 0x559f44610900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b83fd6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f446082a0 0x559f44613450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b83fd6740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f4461a660 con 0x559f44602740 2024-03-28T00:56:40.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b83fd6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f4461a000 con 0x559f44603130 2024-03-28T00:56:40.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b83fd6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f4461acc0 con 0x559f446082a0 2024-03-28T00:56:40.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b83fd6740 10 client.0 _ll_register_callbacks cb 0x7f8b6c012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:56:40.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b73fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44602740 0x559f446103c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b7affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f44603130 0x559f44610900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b7affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f44603130 0x559f44610900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57924/0 (socket says 172.21.15.169:57924) 2024-03-28T00:56:40.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b7affd640 1 -- 172.21.15.169:0/2628151548 learned_addr learned my addr 172.21.15.169:0/2628151548 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:40.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b73fff640 1 --2- 172.21.15.169:0/2628151548 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f446082a0 0x559f44613450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b7affd640 1 -- 172.21.15.169:0/2628151548 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f446082a0 msgr2=0x559f44613450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:40.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b7affd640 1 --2- 172.21.15.169:0/2628151548 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f446082a0 0x559f44613450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b7affd640 1 -- 172.21.15.169:0/2628151548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44602740 msgr2=0x559f446103c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:40.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b7affd640 1 --2- 172.21.15.169:0/2628151548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44602740 0x559f446103c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b7affd640 1 -- 172.21.15.169:0/2628151548 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559f44613a40 con 0x559f44603130 2024-03-28T00:56:40.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b7affd640 1 --2- 172.21.15.169:0/2628151548 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f44603130 0x559f44610900 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c010880 tx=0x7f8b6c010c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2da83410456f870 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:40.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b627fc640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:56:40.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b627fc640 1 -- 172.21.15.169:0/2628151548 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6c00fe80 con 0x559f44603130 2024-03-28T00:56:40.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b627fc640 1 -- 172.21.15.169:0/2628151548 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f8b6c018040 con 0x559f44603130 2024-03-28T00:56:40.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.147+0000 7f8b83fd6740 1 -- 172.21.15.169:0/2628151548 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8b640582e0 con 0x559f44603130 2024-03-28T00:56:40.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.148+0000 7f8b627fc640 1 -- 172.21.15.169:0/2628151548 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b6c038050 con 0x559f44603130 2024-03-28T00:56:40.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.148+0000 7f8b83fd6740 10 client.25241 fetch_fsmap learned FSMap version 24 2024-03-28T00:56:40.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.148+0000 7f8b83fd6740 1 -- 172.21.15.169:0/2628151548 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x559f44674c40 con 0x559f44603130 2024-03-28T00:56:40.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.148+0000 7f8b627fc640 1 -- 172.21.15.169:0/2628151548 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b640582e0 con 0x559f44603130 2024-03-28T00:56:40.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.148+0000 7f8b627fc640 1 -- 172.21.15.169:0/2628151548 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f8b6c0035f0 con 0x559f44603130 2024-03-28T00:56:40.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.148+0000 7f8b83fd6740 10 client.25241 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T00:56:40.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.148+0000 7f8b83fd6740 10 client.25241 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T00:56:40.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.148+0000 7f8b83fd6740 20 client.25241 populate_metadata read hostname 'smithi169' 2024-03-28T00:56:40.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.148+0000 7f8b83fd6740 1 --2- 172.21.15.169:0/2628151548 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x559f446792a0 0x559f44674e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.148+0000 7f8b83fd6740 4 client.25241 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T00:56:40.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.148+0000 7f8b83fd6740 1 -- 172.21.15.169:0/2628151548 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559f4456f4b0 con 0x559f446792a0 2024-03-28T00:56:40.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.148+0000 7f8b73fff640 1 --2- 172.21.15.169:0/2628151548 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x559f446792a0 0x559f44674e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.148+0000 7f8b627fc640 10 client.25241 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T00:56:40.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.148+0000 7f8b73fff640 1 --2- 172.21.15.169:0/2628151548 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x559f446792a0 0x559f44674e70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f1828da89e3df612 server_cookie=3718d4a021fc1210 in_seq=0 out_seq=0 2024-03-28T00:56:40.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b627fc640 1 -- 172.21.15.169:0/2628151548 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x559f4456f4b0 con 0x559f446792a0 2024-03-28T00:56:40.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b627fc640 10 client.25241 handle_command_reply: tid=0 2024-03-28T00:56:40.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b83fd6740 1 client.25241 shutdown 2024-03-28T00:56:40.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b83fd6740 1 -- 172.21.15.169:0/2628151548 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f44603130 msgr2=0x559f44610900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b83fd6740 1 --2- 172.21.15.169:0/2628151548 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f44603130 0x559f44610900 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c010880 tx=0x7f8b6c010c20 comp rx=0 tx=0).stop 2024-03-28T00:56:40.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b83fd6740 1 -- 172.21.15.169:0/2628151548 shutdown_connections 2024-03-28T00:56:40.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b83fd6740 1 --2- 172.21.15.169:0/2628151548 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x559f446792a0 0x559f44674e70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b83fd6740 1 --2- 172.21.15.169:0/2628151548 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f446082a0 0x559f44613450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b83fd6740 1 --2- 172.21.15.169:0/2628151548 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f44603130 0x559f44610900 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b83fd6740 1 --2- 172.21.15.169:0/2628151548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559f44602740 0x559f446103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b627fc640 0 client.25241 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T00:56:40.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b83fd6740 1 -- 172.21.15.169:0/2628151548 >> 172.21.15.169:0/2628151548 conn(0x559f445caa60 msgr2=0x559f446012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:40.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b83fd6740 1 -- 172.21.15.169:0/2628151548 shutdown_connections 2024-03-28T00:56:40.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b83fd6740 1 -- 172.21.15.169:0/2628151548 wait complete. 2024-03-28T00:56:40.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b83fd6740 20 client.25241 trim_cache size 0 max 16384 2024-03-28T00:56:40.159 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:56:40.159 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_fsid": "833368da-ec9c-11ee-95d0-87774f69a715", 2024-03-28T00:56:40.159 INFO:teuthology.orchestra.run.smithi169.stdout: "whoami": 1, 2024-03-28T00:56:40.159 INFO:teuthology.orchestra.run.smithi169.stdout: "id": 24478, 2024-03-28T00:56:40.159 INFO:teuthology.orchestra.run.smithi169.stdout: "want_state": "up:active", 2024-03-28T00:56:40.159 INFO:teuthology.orchestra.run.smithi169.stdout: "state": "up:active", 2024-03-28T00:56:40.160 INFO:teuthology.orchestra.run.smithi169.stdout: "fs_name": "cephfs", 2024-03-28T00:56:40.160 INFO:teuthology.orchestra.run.smithi169.stdout: "rank_uptime": 109.886677203, 2024-03-28T00:56:40.160 INFO:teuthology.orchestra.run.smithi169.stdout: "mdsmap_epoch": 24, 2024-03-28T00:56:40.160 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_epoch": 94, 2024-03-28T00:56:40.160 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_epoch_barrier": 90, 2024-03-28T00:56:40.160 INFO:teuthology.orchestra.run.smithi169.stdout: "uptime": 139.76340746299999 2024-03-28T00:56:40.160 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:56:40.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b813f3640 1 -- 172.21.15.169:0/2042639650 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8b680a2800 msgr2=0x7f8b680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b813f3640 1 --2- 172.21.15.169:0/2042639650 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8b680a2800 0x7f8b680a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8b740a5520 tx=0x7f8b74068120 comp rx=0 tx=0).stop 2024-03-28T00:56:40.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b813f3640 1 -- 172.21.15.169:0/2042639650 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b7c102160 msgr2=0x7f8b7c19de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b813f3640 1 --2- 172.21.15.169:0/2042639650 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b7c102160 0x7f8b7c19de90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8b7400b5e0 tx=0x7f8b7400baa0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b813f3640 1 -- 172.21.15.169:0/2042639650 shutdown_connections 2024-03-28T00:56:40.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b813f3640 1 --2- 172.21.15.169:0/2042639650 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8b680a2800 0x7f8b680a4cc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b813f3640 1 --2- 172.21.15.169:0/2042639650 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b7c102160 0x7f8b7c19de90 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b813f3640 1 --2- 172.21.15.169:0/2042639650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b7c1017a0 0x7f8b7c196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b813f3640 1 --2- 172.21.15.169:0/2042639650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b7c1005b0 0x7f8b7c1968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b813f3640 1 -- 172.21.15.169:0/2042639650 >> 172.21.15.169:0/2042639650 conn(0x7f8b7c0fbb40 msgr2=0x7f8b7c104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:40.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b813f3640 1 -- 172.21.15.169:0/2042639650 shutdown_connections 2024-03-28T00:56:40.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.149+0000 7f8b813f3640 1 -- 172.21.15.169:0/2042639650 wait complete. 2024-03-28T00:56:40.163 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T00:56:40.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.264+0000 7f3a287c4640 1 -- 172.21.15.169:0/3903633392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a201016d0 msgr2=0x7f3a20101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.264+0000 7f3a287c4640 1 --2- 172.21.15.169:0/3903633392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a201016d0 0x7f3a20101af0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c009860 tx=0x7f3a1c02bb70 comp rx=0 tx=0).stop 2024-03-28T00:56:40.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.265+0000 7f3a287c4640 1 -- 172.21.15.169:0/3903633392 shutdown_connections 2024-03-28T00:56:40.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.265+0000 7f3a287c4640 1 --2- 172.21.15.169:0/3903633392 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a20103280 0x7f3a20109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.265+0000 7f3a287c4640 1 --2- 172.21.15.169:0/3903633392 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a201028c0 0x7f3a20102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.265+0000 7f3a287c4640 1 --2- 172.21.15.169:0/3903633392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a201016d0 0x7f3a20101af0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.265+0000 7f3a287c4640 1 -- 172.21.15.169:0/3903633392 >> 172.21.15.169:0/3903633392 conn(0x7f3a200fcc40 msgr2=0x7f3a200ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:40.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.265+0000 7f3a287c4640 1 -- 172.21.15.169:0/3903633392 shutdown_connections 2024-03-28T00:56:40.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.265+0000 7f3a287c4640 1 -- 172.21.15.169:0/3903633392 wait complete. 2024-03-28T00:56:40.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.265+0000 7f3a287c4640 1 Processor -- start 2024-03-28T00:56:40.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.265+0000 7f3a287c4640 1 -- start start 2024-03-28T00:56:40.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.265+0000 7f3a287c4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a201016d0 0x7f3a201979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.265+0000 7f3a287c4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a201028c0 0x7f3a20197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.265+0000 7f3a287c4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a20103280 0x7f3a2019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.265+0000 7f3a287c4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2010d320 con 0x7f3a201016d0 2024-03-28T00:56:40.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.265+0000 7f3a287c4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2010d1a0 con 0x7f3a201028c0 2024-03-28T00:56:40.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.265+0000 7f3a287c4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2010d4a0 con 0x7f3a20103280 2024-03-28T00:56:40.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.265+0000 7f3a26560640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a201016d0 0x7f3a201979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.266+0000 7f3a26560640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a201016d0 0x7f3a201979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51198/0 (socket says 172.21.15.169:51198) 2024-03-28T00:56:40.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.266+0000 7f3a25d5f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a201028c0 0x7f3a20197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.266+0000 7f3a26560640 1 -- 172.21.15.169:0/1297537940 learned_addr learned my addr 172.21.15.169:0/1297537940 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:40.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.266+0000 7f3a26d61640 1 --2- 172.21.15.169:0/1297537940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a20103280 0x7f3a2019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.266+0000 7f3a26560640 1 -- 172.21.15.169:0/1297537940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a20103280 msgr2=0x7f3a2019efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.266+0000 7f3a26560640 1 --2- 172.21.15.169:0/1297537940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a20103280 0x7f3a2019efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.266+0000 7f3a26560640 1 -- 172.21.15.169:0/1297537940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a201028c0 msgr2=0x7f3a20197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.266+0000 7f3a26560640 1 --2- 172.21.15.169:0/1297537940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a201028c0 0x7f3a20197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.266+0000 7f3a26560640 1 -- 172.21.15.169:0/1297537940 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2019f610 con 0x7f3a201016d0 2024-03-28T00:56:40.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.266+0000 7f3a26d61640 1 --2- 172.21.15.169:0/1297537940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a20103280 0x7f3a2019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:40.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.266+0000 7f3a26560640 1 --2- 172.21.15.169:0/1297537940 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a201016d0 0x7f3a201979f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c02c0b0 tx=0x7f3a1c032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:40.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.266+0000 7f3a177fe640 1 -- 172.21.15.169:0/1297537940 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c041020 con 0x7f3a201016d0 2024-03-28T00:56:40.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.266+0000 7f3a177fe640 1 -- 172.21.15.169:0/1297537940 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a1c02c9f0 con 0x7f3a201016d0 2024-03-28T00:56:40.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.266+0000 7f3a287c4640 1 -- 172.21.15.169:0/1297537940 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a2019f8a0 con 0x7f3a201016d0 2024-03-28T00:56:40.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.266+0000 7f3a177fe640 1 -- 172.21.15.169:0/1297537940 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c039040 con 0x7f3a201016d0 2024-03-28T00:56:40.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.266+0000 7f3a287c4640 1 -- 172.21.15.169:0/1297537940 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a2019fd90 con 0x7f3a201016d0 2024-03-28T00:56:40.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.268+0000 7f3a177fe640 1 -- 172.21.15.169:0/1297537940 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3a1c044020 con 0x7f3a201016d0 2024-03-28T00:56:40.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a2b3a9740 1 Processor -- start 2024-03-28T00:56:40.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a2b3a9740 1 -- start start 2024-03-28T00:56:40.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a2b3a9740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1ac740 0x55b8df1acb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a177fe640 1 --2- 172.21.15.169:0/1297537940 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3a0c0a2800 0x7f3a0c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a2b3a9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1ad130 0x55b8df1b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a2b3a9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b8df1b22a0 0x55b8df1b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a2b3a9740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8df111c20 con 0x55b8df1ac740 2024-03-28T00:56:40.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a2b3a9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8df0baa40 con 0x55b8df1ad130 2024-03-28T00:56:40.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a2b3a9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8df10dcc0 con 0x55b8df1b22a0 2024-03-28T00:56:40.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a26d61640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1ac740 0x55b8df1acb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a26d61640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1ac740 0x55b8df1acb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51200/0 (socket says 172.21.15.169:51200) 2024-03-28T00:56:40.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a26d61640 1 -- 172.21.15.169:0/2048480491 learned_addr learned my addr 172.21.15.169:0/2048480491 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:40.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a25d5f640 1 --2- 172.21.15.169:0/2048480491 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1ad130 0x55b8df1b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a26560640 1 --2- 172.21.15.169:0/1297537940 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3a0c0a2800 0x7f3a0c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a26d61640 1 --2- 172.21.15.169:0/2048480491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b8df1b22a0 0x55b8df1b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a177fe640 1 -- 172.21.15.169:0/1297537940 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f3a1c0f06e0 con 0x7f3a201016d0 2024-03-28T00:56:40.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a26d61640 1 -- 172.21.15.169:0/2048480491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b8df1b22a0 msgr2=0x55b8df1b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:40.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a26d61640 1 --2- 172.21.15.169:0/2048480491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b8df1b22a0 0x55b8df1b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a26d61640 1 -- 172.21.15.169:0/2048480491 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1ad130 msgr2=0x55b8df1b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:40.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a26d61640 1 --2- 172.21.15.169:0/2048480491 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1ad130 0x55b8df1b1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a26d61640 1 -- 172.21.15.169:0/2048480491 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b8df0dcf90 con 0x55b8df1ac740 2024-03-28T00:56:40.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.269+0000 7f3a25d5f640 1 --2- 172.21.15.169:0/2048480491 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1ad130 0x55b8df1b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:40.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.270+0000 7f3a26d61640 1 --2- 172.21.15.169:0/2048480491 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1ac740 0x55b8df1acb60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3a1800ed10 tx=0x7f3a18009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc486705f9c28e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:40.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.270+0000 7f3a26560640 1 --2- 172.21.15.169:0/1297537940 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3a0c0a2800 0x7f3a0c0a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0c4f50 tx=0x7f3a1c087aa0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:40.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.270+0000 7f3a14ff9640 1 -- 172.21.15.169:0/2048480491 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a18017040 con 0x55b8df1ac740 2024-03-28T00:56:40.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.270+0000 7f3a14ff9640 1 -- 172.21.15.169:0/2048480491 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a18003e50 con 0x55b8df1ac740 2024-03-28T00:56:40.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.270+0000 7f3a14ff9640 1 -- 172.21.15.169:0/2048480491 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a18004690 con 0x55b8df1ac740 2024-03-28T00:56:40.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.270+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/2048480491 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1ac740 msgr2=0x55b8df1acb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.270+0000 7f3a2b3a9740 1 --2- 172.21.15.169:0/2048480491 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1ac740 0x55b8df1acb60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3a1800ed10 tx=0x7f3a18009ca0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.271+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/2048480491 shutdown_connections 2024-03-28T00:56:40.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.271+0000 7f3a2b3a9740 1 --2- 172.21.15.169:0/2048480491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b8df1b22a0 0x55b8df1b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.271+0000 7f3a2b3a9740 1 --2- 172.21.15.169:0/2048480491 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1ad130 0x55b8df1b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.271+0000 7f3a2b3a9740 1 --2- 172.21.15.169:0/2048480491 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1ac740 0x55b8df1acb60 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.271+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/2048480491 >> 172.21.15.169:0/2048480491 conn(0x55b8df174a60 msgr2=0x55b8df1a80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:40.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.271+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/2048480491 shutdown_connections 2024-03-28T00:56:40.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.271+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/2048480491 wait complete. 2024-03-28T00:56:40.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.271+0000 7f3a2b3a9740 1 -- start start 2024-03-28T00:56:40.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.271+0000 7f3a2b3a9740 1 Processor -- start 2024-03-28T00:56:40.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.271+0000 7f3a2b3a9740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1ac740 0x55b8df1b6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.271+0000 7f3a2b3a9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1ad130 0x55b8df1b71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.271+0000 7f3a2b3a9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b8df1b22a0 0x55b8df1be230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.271+0000 7f3a2b3a9740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8df1b5250 con 0x55b8df1ac740 2024-03-28T00:56:40.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.271+0000 7f3a2b3a9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8df0cde90 con 0x55b8df1ad130 2024-03-28T00:56:40.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.271+0000 7f3a2b3a9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8df1b53d0 con 0x55b8df1b22a0 2024-03-28T00:56:40.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.271+0000 7f3a26d61640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1ac740 0x55b8df1b6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.271+0000 7f3a2b3a9740 10 client.0 _ll_register_callbacks cb 0x55b8dee25ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:56:40.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.271+0000 7f3a26d61640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1ac740 0x55b8df1b6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51212/0 (socket says 172.21.15.169:51212) 2024-03-28T00:56:40.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.271+0000 7f3a26d61640 1 -- 172.21.15.169:0/3986248580 learned_addr learned my addr 172.21.15.169:0/3986248580 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:40.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.271+0000 7f3a25d5f640 1 --2- 172.21.15.169:0/3986248580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1ad130 0x55b8df1b71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.272+0000 7f3a26d61640 1 --2- 172.21.15.169:0/3986248580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b8df1b22a0 0x55b8df1be230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.272+0000 7f3a26d61640 1 -- 172.21.15.169:0/3986248580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b8df1b22a0 msgr2=0x55b8df1be230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:40.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.272+0000 7f3a26d61640 1 --2- 172.21.15.169:0/3986248580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b8df1b22a0 0x55b8df1be230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.272+0000 7f3a26d61640 1 -- 172.21.15.169:0/3986248580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1ad130 msgr2=0x55b8df1b71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:40.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.272+0000 7f3a26d61640 1 --2- 172.21.15.169:0/3986248580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1ad130 0x55b8df1b71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.272+0000 7f3a26d61640 1 -- 172.21.15.169:0/3986248580 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b8df0da9d0 con 0x55b8df1ac740 2024-03-28T00:56:40.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.272+0000 7f3a26d61640 1 --2- 172.21.15.169:0/3986248580 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1ac740 0x55b8df1b6cb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3a18009620 tx=0x7f3a180097e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f31f17f5304d17a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:40.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.272+0000 7f3a09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:56:40.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.272+0000 7f3a09ffb640 1 -- 172.21.15.169:0/3986248580 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a18026040 con 0x55b8df1ac740 2024-03-28T00:56:40.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.272+0000 7f3a09ffb640 1 -- 172.21.15.169:0/3986248580 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f3a18005230 con 0x55b8df1ac740 2024-03-28T00:56:40.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.272+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/3986248580 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3a18003e50 con 0x55b8df1ac740 2024-03-28T00:56:40.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.272+0000 7f3a09ffb640 1 -- 172.21.15.169:0/3986248580 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a18033050 con 0x55b8df1ac740 2024-03-28T00:56:40.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.272+0000 7f3a2b3a9740 10 client.15429 fetch_fsmap learned FSMap version 24 2024-03-28T00:56:40.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.272+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/3986248580 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55b8dee262d0 con 0x55b8df1ac740 2024-03-28T00:56:40.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.272+0000 7f3a09ffb640 1 -- 172.21.15.169:0/3986248580 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a18003e50 con 0x55b8df1ac740 2024-03-28T00:56:40.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.272+0000 7f3a09ffb640 1 -- 172.21.15.169:0/3986248580 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f3a1800b040 con 0x55b8df1ac740 2024-03-28T00:56:40.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.272+0000 7f3a2b3a9740 10 client.15429 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T00:56:40.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.273+0000 7f3a2b3a9740 10 client.15429 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T00:56:40.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.273+0000 7f3a2b3a9740 20 client.15429 populate_metadata read hostname 'smithi169' 2024-03-28T00:56:40.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.273+0000 7f3a2b3a9740 1 --2- 172.21.15.169:0/3986248580 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b8df1c31b0 0x55b8df1c2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.273+0000 7f3a2b3a9740 4 client.15429 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:56:40.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.273+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/3986248580 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b8df11f160 con 0x55b8df1c31b0 2024-03-28T00:56:40.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.273+0000 7f3a25d5f640 1 --2- 172.21.15.169:0/3986248580 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b8df1c31b0 0x55b8df1c2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.273+0000 7f3a25d5f640 1 --2- 172.21.15.169:0/3986248580 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b8df1c31b0 0x55b8df1c2450 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1c83a4e39ba7e47 server_cookie=aedba1166eedd48e in_seq=0 out_seq=0 2024-03-28T00:56:40.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.273+0000 7f3a09ffb640 10 client.15429 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:56:40.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.274+0000 7f3a09ffb640 1 -- 172.21.15.169:0/3986248580 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b8df11f160 con 0x55b8df1c31b0 2024-03-28T00:56:40.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.274+0000 7f3a09ffb640 10 client.15429 handle_command_reply: tid=0 2024-03-28T00:56:40.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.274+0000 7f3a2b3a9740 1 client.15429 shutdown 2024-03-28T00:56:40.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.274+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/3986248580 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1ac740 msgr2=0x55b8df1b6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.274+0000 7f3a2b3a9740 1 --2- 172.21.15.169:0/3986248580 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1ac740 0x55b8df1b6cb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3a18009620 tx=0x7f3a180097e0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.274+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/3986248580 shutdown_connections 2024-03-28T00:56:40.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.274+0000 7f3a2b3a9740 1 --2- 172.21.15.169:0/3986248580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b8df1b22a0 0x55b8df1be230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.274+0000 7f3a2b3a9740 1 --2- 172.21.15.169:0/3986248580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1ad130 0x55b8df1b71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.274+0000 7f3a2b3a9740 1 --2- 172.21.15.169:0/3986248580 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b8df1c31b0 0x55b8df1c2450 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.274+0000 7f3a2b3a9740 1 --2- 172.21.15.169:0/3986248580 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1ac740 0x55b8df1b6cb0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.274+0000 7f3a09ffb640 0 client.15429 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:56:40.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.274+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/3986248580 >> 172.21.15.169:0/3986248580 conn(0x55b8df174a60 msgr2=0x55b8df1a72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:40.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.274+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/3986248580 shutdown_connections 2024-03-28T00:56:40.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.274+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/3986248580 wait complete. 2024-03-28T00:56:40.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.274+0000 7f3a2b3a9740 20 client.15429 trim_cache size 0 max 16384 2024-03-28T00:56:40.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.290+0000 7f3a2b3a9740 1 Processor -- start 2024-03-28T00:56:40.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.290+0000 7f3a2b3a9740 1 -- start start 2024-03-28T00:56:40.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.291+0000 7f3a2b3a9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1ac740 0x55b8df1ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.291+0000 7f3a2b3a9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b8df1ad130 0x55b8df1aec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.291+0000 7f3a2b3a9740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1b22a0 0x55b8df1be1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.291+0000 7f3a2b3a9740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8df1c29d0 con 0x55b8df1b22a0 2024-03-28T00:56:40.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.291+0000 7f3a2b3a9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8df1c3030 con 0x55b8df1ac740 2024-03-28T00:56:40.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.291+0000 7f3a2b3a9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8df1c3690 con 0x55b8df1ad130 2024-03-28T00:56:40.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.291+0000 7f3a26d61640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1b22a0 0x55b8df1be1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.291+0000 7f3a26d61640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1b22a0 0x55b8df1be1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51218/0 (socket says 172.21.15.169:51218) 2024-03-28T00:56:40.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.291+0000 7f3a26d61640 1 -- 172.21.15.169:0/2853437000 learned_addr learned my addr 172.21.15.169:0/2853437000 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:40.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.291+0000 7f3a26d61640 1 --2- 172.21.15.169:0/2853437000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1ac740 0x55b8df1ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.291+0000 7f3a25d5f640 1 --2- 172.21.15.169:0/2853437000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b8df1ad130 0x55b8df1aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.291+0000 7f3a26d61640 1 -- 172.21.15.169:0/2853437000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b8df1ad130 msgr2=0x55b8df1aec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:40.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.291+0000 7f3a26d61640 1 --2- 172.21.15.169:0/2853437000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b8df1ad130 0x55b8df1aec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.291+0000 7f3a26d61640 1 -- 172.21.15.169:0/2853437000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1ac740 msgr2=0x55b8df1ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:40.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.291+0000 7f3a26d61640 1 --2- 172.21.15.169:0/2853437000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1ac740 0x55b8df1ae6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.291+0000 7f3a26d61640 1 -- 172.21.15.169:0/2853437000 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b8deef06b0 con 0x55b8df1b22a0 2024-03-28T00:56:40.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.291+0000 7f3a26d61640 1 --2- 172.21.15.169:0/2853437000 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1b22a0 0x55b8df1be1c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x55b8df1b7f80 tx=0x7f3a18003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484fea87cc1e7270 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:40.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.291+0000 7f3a0bfff640 1 -- 172.21.15.169:0/2853437000 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1802f070 con 0x55b8df1b22a0 2024-03-28T00:56:40.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.291+0000 7f3a0bfff640 1 -- 172.21.15.169:0/2853437000 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a18009ca0 con 0x55b8df1b22a0 2024-03-28T00:56:40.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.291+0000 7f3a0bfff640 1 -- 172.21.15.169:0/2853437000 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a18004230 con 0x55b8df1b22a0 2024-03-28T00:56:40.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.292+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/2853437000 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1b22a0 msgr2=0x55b8df1be1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.292+0000 7f3a2b3a9740 1 --2- 172.21.15.169:0/2853437000 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1b22a0 0x55b8df1be1c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x55b8df1b7f80 tx=0x7f3a18003c80 comp rx=0 tx=0).stop 2024-03-28T00:56:40.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.292+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/2853437000 shutdown_connections 2024-03-28T00:56:40.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.292+0000 7f3a2b3a9740 1 --2- 172.21.15.169:0/2853437000 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1b22a0 0x55b8df1be1c0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.292+0000 7f3a2b3a9740 1 --2- 172.21.15.169:0/2853437000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b8df1ad130 0x55b8df1aec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.292+0000 7f3a2b3a9740 1 --2- 172.21.15.169:0/2853437000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1ac740 0x55b8df1ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.292+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/2853437000 >> 172.21.15.169:0/2853437000 conn(0x55b8df174a60 msgr2=0x55b8df1b5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:40.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.292+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/2853437000 shutdown_connections 2024-03-28T00:56:40.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.292+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/2853437000 wait complete. 2024-03-28T00:56:40.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.292+0000 7f3a2b3a9740 1 -- start start 2024-03-28T00:56:40.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.292+0000 7f3a2b3a9740 1 Processor -- start 2024-03-28T00:56:40.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.292+0000 7f3a2b3a9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b8df1ac740 0x55b8df1ba380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a2b3a9740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1ad130 0x55b8df1ba8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a2b3a9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1b22a0 0x55b8df1bd820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a2b3a9740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8df1c4350 con 0x55b8df1ad130 2024-03-28T00:56:40.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a2b3a9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8df1c3cf0 con 0x55b8df1b22a0 2024-03-28T00:56:40.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a2b3a9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8df1c49b0 con 0x55b8df1ac740 2024-03-28T00:56:40.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a2b3a9740 10 client.0 _ll_register_callbacks cb 0x7f3a10009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:56:40.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a25d5f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1ad130 0x55b8df1ba8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a25d5f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1ad130 0x55b8df1ba8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51234/0 (socket says 172.21.15.169:51234) 2024-03-28T00:56:40.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a25d5f640 1 -- 172.21.15.169:0/364545060 learned_addr learned my addr 172.21.15.169:0/364545060 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:40.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a26d61640 1 --2- 172.21.15.169:0/364545060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b8df1ac740 0x55b8df1ba380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a26d61640 1 --2- 172.21.15.169:0/364545060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1b22a0 0x55b8df1bd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a25d5f640 1 -- 172.21.15.169:0/364545060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b8df1ac740 msgr2=0x55b8df1ba380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:40.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a25d5f640 1 --2- 172.21.15.169:0/364545060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b8df1ac740 0x55b8df1ba380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a25d5f640 1 -- 172.21.15.169:0/364545060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1b22a0 msgr2=0x55b8df1bd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:40.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a25d5f640 1 --2- 172.21.15.169:0/364545060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1b22a0 0x55b8df1bd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a25d5f640 1 -- 172.21.15.169:0/364545060 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b8df1bde10 con 0x55b8df1ad130 2024-03-28T00:56:40.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a26d61640 1 --2- 172.21.15.169:0/364545060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1b22a0 0x55b8df1bd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:56:40.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a25d5f640 1 --2- 172.21.15.169:0/364545060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1ad130 0x55b8df1ba8c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3a10000c00 tx=0x7f3a10010a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dbe95031f5a4add server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:40.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:56:40.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a09ffb640 1 -- 172.21.15.169:0/364545060 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10005020 con 0x55b8df1ad130 2024-03-28T00:56:40.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a09ffb640 1 -- 172.21.15.169:0/364545060 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f3a100051c0 con 0x55b8df1ad130 2024-03-28T00:56:40.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/364545060 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3a18009ca0 con 0x55b8df1ad130 2024-03-28T00:56:40.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a09ffb640 1 -- 172.21.15.169:0/364545060 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a10030050 con 0x55b8df1ad130 2024-03-28T00:56:40.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a2b3a9740 10 client.15435 fetch_fsmap learned FSMap version 24 2024-03-28T00:56:40.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/364545060 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55b8df21ebc0 con 0x55b8df1ad130 2024-03-28T00:56:40.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.293+0000 7f3a09ffb640 1 -- 172.21.15.169:0/364545060 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a18009ca0 con 0x55b8df1ad130 2024-03-28T00:56:40.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.294+0000 7f3a09ffb640 1 -- 172.21.15.169:0/364545060 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 24) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f3a10020900 con 0x55b8df1ad130 2024-03-28T00:56:40.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.294+0000 7f3a2b3a9740 10 client.15435 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T00:56:40.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.294+0000 7f3a2b3a9740 10 client.15435 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T00:56:40.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.294+0000 7f3a2b3a9740 20 client.15435 populate_metadata read hostname 'smithi169' 2024-03-28T00:56:40.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.294+0000 7f3a2b3a9740 1 --2- 172.21.15.169:0/364545060 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b8df2232f0 0x55b8df21eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:40.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.294+0000 7f3a2b3a9740 4 client.15435 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T00:56:40.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.294+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/364545060 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b8df1194b0 con 0x55b8df2232f0 2024-03-28T00:56:40.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.294+0000 7f3a26d61640 1 --2- 172.21.15.169:0/364545060 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b8df2232f0 0x55b8df21eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:40.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.294+0000 7f3a09ffb640 10 client.15435 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:56:40.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.294+0000 7f3a26d61640 1 --2- 172.21.15.169:0/364545060 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b8df2232f0 0x55b8df21eeb0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e536ad0b3c5e636e server_cookie=267a806a4f200768 in_seq=0 out_seq=0 2024-03-28T00:56:40.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.294+0000 7f3a09ffb640 1 -- 172.21.15.169:0/364545060 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x55b8df1194b0 con 0x55b8df2232f0 2024-03-28T00:56:40.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.294+0000 7f3a09ffb640 10 client.15435 handle_command_reply: tid=0 2024-03-28T00:56:40.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.294+0000 7f3a2b3a9740 1 client.15435 shutdown 2024-03-28T00:56:40.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.294+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/364545060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1ad130 msgr2=0x55b8df1ba8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.294+0000 7f3a2b3a9740 1 --2- 172.21.15.169:0/364545060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1ad130 0x55b8df1ba8c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3a10000c00 tx=0x7f3a10010a20 comp rx=0 tx=0).stop 2024-03-28T00:56:40.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/364545060 shutdown_connections 2024-03-28T00:56:40.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a2b3a9740 1 --2- 172.21.15.169:0/364545060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8df1b22a0 0x55b8df1bd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a2b3a9740 1 --2- 172.21.15.169:0/364545060 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b8df2232f0 0x55b8df21eeb0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a2b3a9740 1 --2- 172.21.15.169:0/364545060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b8df1ad130 0x55b8df1ba8c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a2b3a9740 1 --2- 172.21.15.169:0/364545060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b8df1ac740 0x55b8df1ba380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a09ffb640 0 client.15435 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:56:40.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/364545060 >> 172.21.15.169:0/364545060 conn(0x55b8df174a60 msgr2=0x55b8df1ae6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:40.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/364545060 shutdown_connections 2024-03-28T00:56:40.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a2b3a9740 1 -- 172.21.15.169:0/364545060 wait complete. 2024-03-28T00:56:40.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a2b3a9740 20 client.15435 trim_cache size 0 max 16384 2024-03-28T00:56:40.312 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:56:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: "cluster_fsid": "833368da-ec9c-11ee-95d0-87774f69a715", 2024-03-28T00:56:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: "whoami": 0, 2024-03-28T00:56:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: "id": 24479, 2024-03-28T00:56:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: "want_state": "up:active", 2024-03-28T00:56:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: "state": "up:active", 2024-03-28T00:56:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: "fs_name": "cephfs", 2024-03-28T00:56:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: "rank_uptime": 127.057051008, 2024-03-28T00:56:40.312 INFO:teuthology.orchestra.run.smithi169.stdout: "mdsmap_epoch": 24, 2024-03-28T00:56:40.313 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_epoch": 94, 2024-03-28T00:56:40.313 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_epoch_barrier": 90, 2024-03-28T00:56:40.313 INFO:teuthology.orchestra.run.smithi169.stdout: "uptime": 131.516971547 2024-03-28T00:56:40.313 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:56:40.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a287c4640 1 -- 172.21.15.169:0/1297537940 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3a0c0a2800 msgr2=0x7f3a0c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a287c4640 1 --2- 172.21.15.169:0/1297537940 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3a0c0a2800 0x7f3a0c0a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0c4f50 tx=0x7f3a1c087aa0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a287c4640 1 -- 172.21.15.169:0/1297537940 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a201016d0 msgr2=0x7f3a201979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:40.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a287c4640 1 --2- 172.21.15.169:0/1297537940 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a201016d0 0x7f3a201979f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c02c0b0 tx=0x7f3a1c032430 comp rx=0 tx=0).stop 2024-03-28T00:56:40.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a287c4640 1 -- 172.21.15.169:0/1297537940 shutdown_connections 2024-03-28T00:56:40.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a287c4640 1 --2- 172.21.15.169:0/1297537940 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3a0c0a2800 0x7f3a0c0a4cc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a287c4640 1 --2- 172.21.15.169:0/1297537940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a20103280 0x7f3a2019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a287c4640 1 --2- 172.21.15.169:0/1297537940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a201028c0 0x7f3a20197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a287c4640 1 --2- 172.21.15.169:0/1297537940 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3a201016d0 0x7f3a201979f0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:40.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a287c4640 1 -- 172.21.15.169:0/1297537940 >> 172.21.15.169:0/1297537940 conn(0x7f3a200fcc40 msgr2=0x7f3a20105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:40.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a287c4640 1 -- 172.21.15.169:0/1297537940 shutdown_connections 2024-03-28T00:56:40.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:40.295+0000 7f3a287c4640 1 -- 172.21.15.169:0/1297537940 wait complete. 2024-03-28T00:56:40.316 INFO:teuthology.run_tasks:Running task kclient... 2024-03-28T00:56:40.344 INFO:tasks.kclient:Mounting kernel clients... 2024-03-28T00:56:40.345 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-28T00:56:40.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:56:40.345 DEBUG:teuthology.orchestra.run.smithi169:> ip netns list 2024-03-28T00:56:40.361 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-28T00:56:40.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:56:40.362 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T00:56:40.427 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:40 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3597073715' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:56:40.428 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:40 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3443197598' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:40.428 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:40 smithi169 ceph-mon[32274]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-28T00:56:40.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:56:40.432 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link delete ceph-brx 2024-03-28T00:56:40.530 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-28T00:56:40.531 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T00:56:40.531 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T00:56:40.531 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph fs ls 2024-03-28T00:56:40.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:40 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3597073715' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:56:40.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:40 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3443197598' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:40.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:40 smithi174 ceph-mon[36060]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-28T00:56:40.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:40 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3597073715' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:56:40.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:40 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3443197598' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:40.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:40 smithi177 ceph-mon[36206]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-28T00:56:40.773 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T00:56:42.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.471+0000 7f97c6729700 1 -- 172.21.15.169:0/2644239396 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97c0103080 msgr2=0x7f97c0103460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:42.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.471+0000 7f97c6729700 1 --2- 172.21.15.169:0/2644239396 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97c0103080 0x7f97c0103460 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f97b0009990 tx=0x7f97b0009ca0 comp rx=0 tx=0).stop 2024-03-28T00:56:42.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.471+0000 7f97c6729700 1 -- 172.21.15.169:0/2644239396 shutdown_connections 2024-03-28T00:56:42.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.471+0000 7f97c6729700 1 --2- 172.21.15.169:0/2644239396 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97c0101af0 0x7f97c010e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:42.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.471+0000 7f97c6729700 1 --2- 172.21.15.169:0/2644239396 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97c0103080 0x7f97c0103460 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:42.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.471+0000 7f97c6729700 1 --2- 172.21.15.169:0/2644239396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97c0101130 0x7f97c01015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:42.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.471+0000 7f97c6729700 1 -- 172.21.15.169:0/2644239396 >> 172.21.15.169:0/2644239396 conn(0x7f97c00770e0 msgr2=0x7f97c00774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:42.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.471+0000 7f97c6729700 1 -- 172.21.15.169:0/2644239396 shutdown_connections 2024-03-28T00:56:42.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.471+0000 7f97c6729700 1 -- 172.21.15.169:0/2644239396 wait complete. 2024-03-28T00:56:42.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.471+0000 7f97c6729700 1 Processor -- start 2024-03-28T00:56:42.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.471+0000 7f97c6729700 1 -- start start 2024-03-28T00:56:42.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.472+0000 7f97c6729700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97c0101130 0x7f97c019f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:42.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.472+0000 7f97c6729700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97c0101af0 0x7f97c019f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:42.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.472+0000 7f97c6729700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97c0103080 0x7f97c01a3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:42.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.472+0000 7f97c6729700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c01114c0 con 0x7f97c0101130 2024-03-28T00:56:42.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.472+0000 7f97c6729700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c0111340 con 0x7f97c0101af0 2024-03-28T00:56:42.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.472+0000 7f97c6729700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c0111640 con 0x7f97c0103080 2024-03-28T00:56:42.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.472+0000 7f97bffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97c0101130 0x7f97c019f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:42.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.472+0000 7f97bffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97c0101130 0x7f97c019f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51250/0 (socket says 172.21.15.169:51250) 2024-03-28T00:56:42.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.472+0000 7f97bffff700 1 -- 172.21.15.169:0/1092133065 learned_addr learned my addr 172.21.15.169:0/1092133065 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:42.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.472+0000 7f97bf7fe700 1 --2- 172.21.15.169:0/1092133065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97c0101af0 0x7f97c019f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:42.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.472+0000 7f97c4cc6700 1 --2- 172.21.15.169:0/1092133065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97c0103080 0x7f97c01a3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.472+0000 7f97bf7fe700 1 -- 172.21.15.169:0/1092133065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97c0103080 msgr2=0x7f97c01a3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.472+0000 7f97bf7fe700 1 --2- 172.21.15.169:0/1092133065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97c0103080 0x7f97c01a3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.472+0000 7f97bf7fe700 1 -- 172.21.15.169:0/1092133065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97c0101130 msgr2=0x7f97c019f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.472+0000 7f97bf7fe700 1 --2- 172.21.15.169:0/1092133065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97c0101130 0x7f97c019f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.472+0000 7f97bf7fe700 1 -- 172.21.15.169:0/1092133065 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c01a4060 con 0x7f97c0101af0 2024-03-28T00:56:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.472+0000 7f97c4cc6700 1 --2- 172.21.15.169:0/1092133065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97c0103080 0x7f97c01a3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.472+0000 7f97bffff700 1 --2- 172.21.15.169:0/1092133065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97c0101130 0x7f97c019f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.473+0000 7f97bf7fe700 1 --2- 172.21.15.169:0/1092133065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97c0101af0 0x7f97c019f8b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f97b400e9f0 tx=0x7f97b400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.473+0000 7f97bd7fa700 1 -- 172.21.15.169:0/1092133065 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b4004220 con 0x7f97c0101af0 2024-03-28T00:56:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.473+0000 7f97bd7fa700 1 -- 172.21.15.169:0/1092133065 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97b4004d10 con 0x7f97c0101af0 2024-03-28T00:56:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.473+0000 7f97c6729700 1 -- 172.21.15.169:0/1092133065 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c01a4350 con 0x7f97c0101af0 2024-03-28T00:56:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.473+0000 7f97bd7fa700 1 -- 172.21.15.169:0/1092133065 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b4005020 con 0x7f97c0101af0 2024-03-28T00:56:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.473+0000 7f97c6729700 1 -- 172.21.15.169:0/1092133065 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c01abbd0 con 0x7f97c0101af0 2024-03-28T00:56:42.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.474+0000 7f97c6729700 1 -- 172.21.15.169:0/1092133065 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c0068e50 con 0x7f97c0101af0 2024-03-28T00:56:42.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.475+0000 7f97bd7fa700 1 -- 172.21.15.169:0/1092133065 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f97b4004750 con 0x7f97c0101af0 2024-03-28T00:56:42.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.475+0000 7f97bd7fa700 1 --2- 172.21.15.169:0/1092133065 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f97a80a2890 0x7f97a80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:42.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.475+0000 7f97bd7fa700 1 -- 172.21.15.169:0/1092133065 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f97b40ced10 con 0x7f97c0101af0 2024-03-28T00:56:42.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.476+0000 7f97bffff700 1 --2- 172.21.15.169:0/1092133065 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f97a80a2890 0x7f97a80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:42.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.476+0000 7f97bffff700 1 --2- 172.21.15.169:0/1092133065 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f97a80a2890 0x7f97a80a4d50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f97b000b2e0 tx=0x7f97b0009590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:42.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.480+0000 7f97bd7fa700 1 -- 172.21.15.169:0/1092133065 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f97b40d4020 con 0x7f97c0101af0 2024-03-28T00:56:42.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.720+0000 7f97c6729700 1 -- 172.21.15.169:0/1092133065 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f97c01abda0 con 0x7f97c0101af0 2024-03-28T00:56:42.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.720+0000 7f97bd7fa700 1 -- 172.21.15.169:0/1092133065 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) v1 ==== 53+0+88 (secure 0 0 0) 0x7f97b4095e20 con 0x7f97c0101af0 2024-03-28T00:56:42.723 INFO:teuthology.orchestra.run.smithi169.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-28T00:56:42.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.722+0000 7f97c6729700 1 -- 172.21.15.169:0/1092133065 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f97a80a2890 msgr2=0x7f97a80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:42.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.722+0000 7f97c6729700 1 --2- 172.21.15.169:0/1092133065 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f97a80a2890 0x7f97a80a4d50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f97b000b2e0 tx=0x7f97b0009590 comp rx=0 tx=0).stop 2024-03-28T00:56:42.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.723+0000 7f97c6729700 1 -- 172.21.15.169:0/1092133065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97c0101af0 msgr2=0x7f97c019f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:42.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.723+0000 7f97c6729700 1 --2- 172.21.15.169:0/1092133065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97c0101af0 0x7f97c019f8b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f97b400e9f0 tx=0x7f97b400ed00 comp rx=0 tx=0).stop 2024-03-28T00:56:42.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.723+0000 7f97c6729700 1 -- 172.21.15.169:0/1092133065 shutdown_connections 2024-03-28T00:56:42.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.723+0000 7f97c6729700 1 --2- 172.21.15.169:0/1092133065 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f97a80a2890 0x7f97a80a4d50 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:42.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.723+0000 7f97c6729700 1 --2- 172.21.15.169:0/1092133065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97c0103080 0x7f97c01a3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:42.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.723+0000 7f97c6729700 1 --2- 172.21.15.169:0/1092133065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97c0101130 0x7f97c019f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:42.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.723+0000 7f97c6729700 1 --2- 172.21.15.169:0/1092133065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97c0101af0 0x7f97c019f8b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:42.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.723+0000 7f97c6729700 1 -- 172.21.15.169:0/1092133065 >> 172.21.15.169:0/1092133065 conn(0x7f97c00770e0 msgr2=0x7f97c006a930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:42.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.723+0000 7f97c6729700 1 -- 172.21.15.169:0/1092133065 shutdown_connections 2024-03-28T00:56:42.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:42.723+0000 7f97c6729700 1 -- 172.21.15.169:0/1092133065 wait complete. 2024-03-28T00:56:43.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:43 smithi169 ceph-mon[32274]: pgmap v357: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T00:56:43.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:43 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1092133065' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T00:56:43.388 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-28T00:56:43.388 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T00:56:43.388 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi169.front.sepia.ceph.com 2024-03-28T00:56:43.388 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-28T00:56:43.388 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T00:56:43.388 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-28T00:56:43.388 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T00:56:43.388 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T00:56:43.388 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-28T00:56:43.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:56:43.389 DEBUG:teuthology.orchestra.run.smithi169:> ip addr 2024-03-28T00:56:43.405 INFO:teuthology.orchestra.run.smithi169.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T00:56:43.405 INFO:teuthology.orchestra.run.smithi169.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T00:56:43.405 INFO:teuthology.orchestra.run.smithi169.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T00:56:43.405 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-28T00:56:43.405 INFO:teuthology.orchestra.run.smithi169.stdout: inet6 ::1/128 scope host 2024-03-28T00:56:43.405 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-28T00:56:43.405 INFO:teuthology.orchestra.run.smithi169.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T00:56:43.405 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:d9:3b:b4 brd ff:ff:ff:ff:ff:ff 2024-03-28T00:56:43.405 INFO:teuthology.orchestra.run.smithi169.stdout: altname enp7s0f0 2024-03-28T00:56:43.405 INFO:teuthology.orchestra.run.smithi169.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T00:56:43.405 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:d9:3b:b5 brd ff:ff:ff:ff:ff:ff 2024-03-28T00:56:43.405 INFO:teuthology.orchestra.run.smithi169.stdout: altname enp7s0f1 2024-03-28T00:56:43.405 INFO:teuthology.orchestra.run.smithi169.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T00:56:43.405 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:88:7f:4c brd ff:ff:ff:ff:ff:ff 2024-03-28T00:56:43.406 INFO:teuthology.orchestra.run.smithi169.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T00:56:43.406 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:88:7f:4d brd ff:ff:ff:ff:ff:ff 2024-03-28T00:56:43.406 INFO:teuthology.orchestra.run.smithi169.stdout: inet 172.21.15.169/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-28T00:56:43.406 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft 42100sec preferred_lft 42100sec 2024-03-28T00:56:43.406 INFO:teuthology.orchestra.run.smithi169.stdout: inet6 fe80::ec4:7aff:fe88:7f4d/64 scope link 2024-03-28T00:56:43.406 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-03-28T00:56:43.406 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T00:56:43.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:56:43.406 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2024-03-28T00:56:43.407 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-03-28T00:56:43.407 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link add name ceph-brx type bridge 2024-03-28T00:56:43.407 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip addr flush dev ceph-brx 2024-03-28T00:56:43.407 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link set ceph-brx up 2024-03-28T00:56:43.407 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T00:56:43.407 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-03-28T00:56:43.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:56:43.516 DEBUG:teuthology.orchestra.run.smithi169:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T00:56:43.543 INFO:teuthology.orchestra.run.smithi169.stdout:1 2024-03-28T00:56:43.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:56:43.545 DEBUG:teuthology.orchestra.run.smithi169:> route 2024-03-28T00:56:43.600 INFO:teuthology.orchestra.run.smithi169.stdout:Kernel IP routing table 2024-03-28T00:56:43.600 INFO:teuthology.orchestra.run.smithi169.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T00:56:43.600 INFO:teuthology.orchestra.run.smithi169.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-28T00:56:43.600 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-28T00:56:43.600 INFO:teuthology.orchestra.run.smithi169.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T00:56:43.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:56:43.601 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2024-03-28T00:56:43.601 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-03-28T00:56:43.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-28T00:56:43.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-28T00:56:43.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-28T00:56:43.601 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-03-28T00:56:43.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:43 smithi174 ceph-mon[36060]: pgmap v357: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T00:56:43.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:43 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1092133065' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T00:56:43.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:43 smithi177 ceph-mon[36206]: pgmap v357: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-28T00:56:43.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:43 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1092133065' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T00:56:43.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:56:43.697 DEBUG:teuthology.orchestra.run.smithi169:> ip netns list 2024-03-28T00:56:43.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:56:43.713 DEBUG:teuthology.orchestra.run.smithi169:> ip netns list-id 2024-03-28T00:56:43.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:56:43.768 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2024-03-28T00:56:43.768 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-03-28T00:56:43.768 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T00:56:43.768 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-28T00:56:43.768 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-03-28T00:56:43.850 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-28T00:56:43.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:56:43.850 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2024-03-28T00:56:43.850 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-03-28T00:56:43.850 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-28T00:56:43.850 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T00:56:43.850 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-28T00:56:43.850 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-28T00:56:43.851 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-28T00:56:43.851 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-03-28T00:56:43.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:56:43.977 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2024-03-28T00:56:43.977 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-03-28T00:56:43.977 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link set brx.0 up 2024-03-28T00:56:43.978 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T00:56:43.978 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-03-28T00:56:44.019 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T00:56:44.019 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-28T00:56:44.074 INFO:teuthology.orchestra.run.smithi169.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-28T00:56:44.075 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T00:56:44.075 DEBUG:teuthology.orchestra.run.smithi169:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-28T00:56:44.132 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-28T00:56:44.132 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-28T00:56:44.132 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/42e8c166-c61f-4c6b-9595-a337acbab519 2024-03-28T00:56:44.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:56:44.133 DEBUG:teuthology.orchestra.run.smithi169:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/42e8c166-c61f-4c6b-9595-a337acbab519 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-03-28T00:56:44.217 INFO:teuthology.orchestra.run.smithi169.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-03-28T00:56:44.217 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-03-28T00:56:44.358 INFO:teuthology.orchestra.run.smithi169.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-03-28T00:56:44.358 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-03-28T00:56:44.358 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: resolved to: "172.21.15.169:3300,172.21.15.174:3300,172.21.15.177:3300" 2024-03-28T00:56:44.358 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: trying mount with new device syntax: 0@833368da-ec9c-11ee-95d0-87774f69a715.cephfs=/volumes/qa/sv_1/42e8c166-c61f-4c6b-9595-a337acbab519 2024-03-28T00:56:44.358 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync,name=0,key=0,mon_addr=172.21.15.169:3300/172.21.15.174:3300/172.21.15.177:3300" will pass to kernel 2024-03-28T00:56:44.359 INFO:teuthology.orchestra.run.smithi169.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-28T00:56:44.359 INFO:teuthology.orchestra.run.smithi169.stdout: You just mounted a file system that supports labels which does not 2024-03-28T00:56:44.359 INFO:teuthology.orchestra.run.smithi169.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-28T00:56:44.359 INFO:teuthology.orchestra.run.smithi169.stdout: applications will generate AVC messages and not be allowed access to 2024-03-28T00:56:44.359 INFO:teuthology.orchestra.run.smithi169.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-28T00:56:44.362 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-28T00:56:44.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:56:44.362 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-28T00:56:44.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:56:44.431 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T00:56:44.489 INFO:teuthology.orchestra.run.smithi169.stdout:client25237 2024-03-28T00:56:44.489 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-28T00:56:44.489 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T00:56:44.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:56:44.546 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T00:56:44.602 INFO:teuthology.orchestra.run.smithi169.stdout:client25237 2024-03-28T00:56:44.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:56:44.602 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T00:56:44.659 INFO:teuthology.orchestra.run.smithi169.stdout:client25237 2024-03-28T00:56:44.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:56:44.659 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/833368da-ec9c-11ee-95d0-87774f69a715.client25237/status') 2024-03-28T00:56:44.727 INFO:teuthology.orchestra.run.smithi169.stdout:instance: client.25237 (3)172.21.15.169:0/423342855 2024-03-28T00:56:44.727 INFO:teuthology.orchestra.run.smithi169.stdout:blocklisted: false 2024-03-28T00:56:44.727 INFO:teuthology.orchestra.run.smithi169.stderr:0+1 records in 2024-03-28T00:56:44.728 INFO:teuthology.orchestra.run.smithi169.stderr:0+1 records out 2024-03-28T00:56:44.728 INFO:teuthology.orchestra.run.smithi169.stderr:71 bytes copied, 3.714e-05 s, 1.9 MB/s 2024-03-28T00:56:44.730 INFO:teuthology.run_tasks:Running task exec... 2024-03-28T00:56:44.741 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T00:56:44.741 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi169.front.sepia.ceph.com 2024-03-28T00:56:44.741 DEBUG:teuthology.orchestra.run.smithi169:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-28T00:56:44.877 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.872+0000 7fdaba9e1640 1 -- 172.21.15.169:0/2465697193 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdab41017c0 msgr2=0x7fdab4101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:44.877 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.872+0000 7fdaba9e1640 1 --2- 172.21.15.169:0/2465697193 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdab41017c0 0x7fdab4101c40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8009900 tx=0x7fdaa802bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:44.877 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.872+0000 7fdaba9e1640 1 -- 172.21.15.169:0/2465697193 shutdown_connections 2024-03-28T00:56:44.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.872+0000 7fdaba9e1640 1 --2- 172.21.15.169:0/2465697193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdab4102180 0x7fdab4108a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:44.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.872+0000 7fdaba9e1640 1 --2- 172.21.15.169:0/2465697193 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdab41017c0 0x7fdab4101c40 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:44.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.872+0000 7fdaba9e1640 1 --2- 172.21.15.169:0/2465697193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdab41005d0 0x7fdab41009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:44.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.872+0000 7fdaba9e1640 1 -- 172.21.15.169:0/2465697193 >> 172.21.15.169:0/2465697193 conn(0x7fdab40fbb60 msgr2=0x7fdab40fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:44.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.872+0000 7fdaba9e1640 1 -- 172.21.15.169:0/2465697193 shutdown_connections 2024-03-28T00:56:44.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.872+0000 7fdaba9e1640 1 -- 172.21.15.169:0/2465697193 wait complete. 2024-03-28T00:56:44.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.872+0000 7fdaba9e1640 1 Processor -- start 2024-03-28T00:56:44.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.872+0000 7fdaba9e1640 1 -- start start 2024-03-28T00:56:44.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.873+0000 7fdaba9e1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdab41005d0 0x7fdab4197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:44.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.873+0000 7fdaba9e1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdab41017c0 0x7fdab4197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:44.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.873+0000 7fdaba9e1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdab4102180 0x7fdab419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:44.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.873+0000 7fdaba9e1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab410c290 con 0x7fdab41005d0 2024-03-28T00:56:44.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.873+0000 7fdaba9e1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab410c110 con 0x7fdab41017c0 2024-03-28T00:56:44.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.873+0000 7fdaba9e1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab410c410 con 0x7fdab4102180 2024-03-28T00:56:44.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.873+0000 7fdab3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdab41005d0 0x7fdab4197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:44.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.873+0000 7fdab3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdab41005d0 0x7fdab4197950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51266/0 (socket says 172.21.15.169:51266) 2024-03-28T00:56:44.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.873+0000 7fdab3fff640 1 -- 172.21.15.169:0/10065003 learned_addr learned my addr 172.21.15.169:0/10065003 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:44.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.873+0000 7fdab8f7e640 1 --2- 172.21.15.169:0/10065003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdab4102180 0x7fdab419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:44.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.873+0000 7fdab37fe640 1 --2- 172.21.15.169:0/10065003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdab41017c0 0x7fdab4197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:44.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.873+0000 7fdab3fff640 1 -- 172.21.15.169:0/10065003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdab4102180 msgr2=0x7fdab419ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:44.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.873+0000 7fdab3fff640 1 --2- 172.21.15.169:0/10065003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdab4102180 0x7fdab419ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:44.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.873+0000 7fdab3fff640 1 -- 172.21.15.169:0/10065003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdab41017c0 msgr2=0x7fdab4197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:44.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.873+0000 7fdab3fff640 1 --2- 172.21.15.169:0/10065003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdab41017c0 0x7fdab4197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:44.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.873+0000 7fdab3fff640 1 -- 172.21.15.169:0/10065003 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab419f570 con 0x7fdab41005d0 2024-03-28T00:56:44.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.873+0000 7fdab8f7e640 1 --2- 172.21.15.169:0/10065003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdab4102180 0x7fdab419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:44.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.873+0000 7fdab3fff640 1 --2- 172.21.15.169:0/10065003 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdab41005d0 0x7fdab4197950 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdaa400e9c0 tx=0x7fdaa400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:44.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.873+0000 7fdab17fa640 1 -- 172.21.15.169:0/10065003 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa400ccc0 con 0x7fdab41005d0 2024-03-28T00:56:44.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.874+0000 7fdab17fa640 1 -- 172.21.15.169:0/10065003 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdaa40035f0 con 0x7fdab41005d0 2024-03-28T00:56:44.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.874+0000 7fdab17fa640 1 -- 172.21.15.169:0/10065003 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa400ce60 con 0x7fdab41005d0 2024-03-28T00:56:44.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.874+0000 7fdaba9e1640 1 -- 172.21.15.169:0/10065003 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab419f860 con 0x7fdab41005d0 2024-03-28T00:56:44.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.874+0000 7fdaba9e1640 1 -- 172.21.15.169:0/10065003 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab419fd80 con 0x7fdab41005d0 2024-03-28T00:56:44.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.874+0000 7fdaba9e1640 1 -- 172.21.15.169:0/10065003 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab419ffe0 con 0x7fdab41005d0 2024-03-28T00:56:44.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.876+0000 7fdab17fa640 1 -- 172.21.15.169:0/10065003 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fdaa40037a0 con 0x7fdab41005d0 2024-03-28T00:56:44.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.876+0000 7fdab17fa640 1 --2- 172.21.15.169:0/10065003 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fda9c0a28a0 0x7fda9c0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:44.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.876+0000 7fdab37fe640 1 --2- 172.21.15.169:0/10065003 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fda9c0a28a0 0x7fda9c0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:44.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.876+0000 7fdab17fa640 1 -- 172.21.15.169:0/10065003 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fdaa40cf160 con 0x7fdab41005d0 2024-03-28T00:56:44.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.877+0000 7fdab37fe640 1 --2- 172.21.15.169:0/10065003 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fda9c0a28a0 0x7fda9c0a4d60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fdaa802c240 tx=0x7fdaa802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:44.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:44.881+0000 7fdab17fa640 1 -- 172.21.15.169:0/10065003 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdaa409be60 con 0x7fdab41005d0 2024-03-28T00:56:45.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.095+0000 7fdaba9e1640 1 -- 172.21.15.169:0/10065003 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) v1 -- 0x7fdab41a01b0 con 0x7fdab41005d0 2024-03-28T00:56:45.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.169+0000 7fdab17fa640 1 -- 172.21.15.169:0/10065003 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v25) v1 ==== 118+0+0 (secure 0 0 0) 0x7fdaa409b760 con 0x7fdab41005d0 2024-03-28T00:56:45.174 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.171+0000 7fdaba9e1640 1 -- 172.21.15.169:0/10065003 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fda9c0a28a0 msgr2=0x7fda9c0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:45.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.171+0000 7fdaba9e1640 1 --2- 172.21.15.169:0/10065003 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fda9c0a28a0 0x7fda9c0a4d60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fdaa802c240 tx=0x7fdaa802df90 comp rx=0 tx=0).stop 2024-03-28T00:56:45.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.171+0000 7fdaba9e1640 1 -- 172.21.15.169:0/10065003 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdab41005d0 msgr2=0x7fdab4197950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:45.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.171+0000 7fdaba9e1640 1 --2- 172.21.15.169:0/10065003 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdab41005d0 0x7fdab4197950 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdaa400e9c0 tx=0x7fdaa400edc0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.172+0000 7fdaba9e1640 1 -- 172.21.15.169:0/10065003 shutdown_connections 2024-03-28T00:56:45.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.172+0000 7fdaba9e1640 1 --2- 172.21.15.169:0/10065003 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fda9c0a28a0 0x7fda9c0a4d60 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.172+0000 7fdaba9e1640 1 --2- 172.21.15.169:0/10065003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdab4102180 0x7fdab419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.172+0000 7fdaba9e1640 1 --2- 172.21.15.169:0/10065003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdab41017c0 0x7fdab4197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.172+0000 7fdaba9e1640 1 --2- 172.21.15.169:0/10065003 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdab41005d0 0x7fdab4197950 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.172+0000 7fdaba9e1640 1 -- 172.21.15.169:0/10065003 >> 172.21.15.169:0/10065003 conn(0x7fdab40fbb60 msgr2=0x7fdab4069f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:45.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.172+0000 7fdaba9e1640 1 -- 172.21.15.169:0/10065003 shutdown_connections 2024-03-28T00:56:45.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.172+0000 7fdaba9e1640 1 -- 172.21.15.169:0/10065003 wait complete. 2024-03-28T00:56:45.186 DEBUG:teuthology.orchestra.run.smithi169:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-28T00:56:45.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.292+0000 7fd9d4d3b640 1 -- 172.21.15.169:0/4206828792 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9d01005e0 msgr2=0x7fd9d0100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:45.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.292+0000 7fd9d4d3b640 1 --2- 172.21.15.169:0/4206828792 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9d01005e0 0x7fd9d0100a00 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4009900 tx=0x7fd9c402bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:45.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.292+0000 7fd9d4d3b640 1 -- 172.21.15.169:0/4206828792 shutdown_connections 2024-03-28T00:56:45.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.292+0000 7fd9d4d3b640 1 --2- 172.21.15.169:0/4206828792 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9d0102190 0x7fd9d0108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.292+0000 7fd9d4d3b640 1 --2- 172.21.15.169:0/4206828792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9d01017d0 0x7fd9d0101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.292+0000 7fd9d4d3b640 1 --2- 172.21.15.169:0/4206828792 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9d01005e0 0x7fd9d0100a00 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.292+0000 7fd9d4d3b640 1 -- 172.21.15.169:0/4206828792 >> 172.21.15.169:0/4206828792 conn(0x7fd9d00fbb70 msgr2=0x7fd9d00fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:45.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.292+0000 7fd9d4d3b640 1 -- 172.21.15.169:0/4206828792 shutdown_connections 2024-03-28T00:56:45.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.292+0000 7fd9d4d3b640 1 -- 172.21.15.169:0/4206828792 wait complete. 2024-03-28T00:56:45.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.293+0000 7fd9d4d3b640 1 Processor -- start 2024-03-28T00:56:45.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.293+0000 7fd9d4d3b640 1 -- start start 2024-03-28T00:56:45.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.293+0000 7fd9d4d3b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9d01005e0 0x7fd9d006a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:45.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.293+0000 7fd9d4d3b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9d01017d0 0x7fd9d019c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:45.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.293+0000 7fd9d4d3b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9d0102190 0x7fd9d019ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:45.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.293+0000 7fd9d4d3b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d010c230 con 0x7fd9d0102190 2024-03-28T00:56:45.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.293+0000 7fd9d4d3b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d010c0b0 con 0x7fd9d01005e0 2024-03-28T00:56:45.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.293+0000 7fd9d4d3b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d010c3b0 con 0x7fd9d01017d0 2024-03-28T00:56:45.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.293+0000 7fd9ced9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9d0102190 0x7fd9d019ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:45.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.293+0000 7fd9cdd9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9d01017d0 0x7fd9d019c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:45.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.293+0000 7fd9ced9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9d0102190 0x7fd9d019ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51280/0 (socket says 172.21.15.169:51280) 2024-03-28T00:56:45.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.293+0000 7fd9ce59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9d01005e0 0x7fd9d006a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:45.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.293+0000 7fd9ced9d640 1 -- 172.21.15.169:0/3765955836 learned_addr learned my addr 172.21.15.169:0/3765955836 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:45.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.293+0000 7fd9cdd9b640 1 -- 172.21.15.169:0/3765955836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9d01005e0 msgr2=0x7fd9d006a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:45.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.293+0000 7fd9cdd9b640 1 --2- 172.21.15.169:0/3765955836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9d01005e0 0x7fd9d006a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.293+0000 7fd9cdd9b640 1 -- 172.21.15.169:0/3765955836 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9d0102190 msgr2=0x7fd9d019ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:45.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.293+0000 7fd9cdd9b640 1 --2- 172.21.15.169:0/3765955836 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9d0102190 0x7fd9d019ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.293+0000 7fd9cdd9b640 1 -- 172.21.15.169:0/3765955836 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d019f590 con 0x7fd9d01017d0 2024-03-28T00:56:45.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.293+0000 7fd9ce59c640 1 --2- 172.21.15.169:0/3765955836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9d01005e0 0x7fd9d006a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:45.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.293+0000 7fd9ced9d640 1 --2- 172.21.15.169:0/3765955836 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9d0102190 0x7fd9d019ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:45.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.294+0000 7fd9cdd9b640 1 --2- 172.21.15.169:0/3765955836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9d01017d0 0x7fd9d019c660 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd9c000e9f0 tx=0x7fd9c000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:45.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.294+0000 7fd9bf7fe640 1 -- 172.21.15.169:0/3765955836 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c000cd80 con 0x7fd9d01017d0 2024-03-28T00:56:45.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.294+0000 7fd9bf7fe640 1 -- 172.21.15.169:0/3765955836 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9c00035f0 con 0x7fd9d01017d0 2024-03-28T00:56:45.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.294+0000 7fd9d4d3b640 1 -- 172.21.15.169:0/3765955836 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d019f880 con 0x7fd9d01017d0 2024-03-28T00:56:45.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.294+0000 7fd9bf7fe640 1 -- 172.21.15.169:0/3765955836 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c0010690 con 0x7fd9d01017d0 2024-03-28T00:56:45.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.294+0000 7fd9d4d3b640 1 -- 172.21.15.169:0/3765955836 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d019fdd0 con 0x7fd9d01017d0 2024-03-28T00:56:45.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.295+0000 7fd9d4d3b640 1 -- 172.21.15.169:0/3765955836 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d0107900 con 0x7fd9d01017d0 2024-03-28T00:56:45.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.296+0000 7fd9bf7fe640 1 -- 172.21.15.169:0/3765955836 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd9c0003b30 con 0x7fd9d01017d0 2024-03-28T00:56:45.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.296+0000 7fd9bf7fe640 1 --2- 172.21.15.169:0/3765955836 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd9b80a28a0 0x7fd9b80a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:45.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.296+0000 7fd9ce59c640 1 --2- 172.21.15.169:0/3765955836 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd9b80a28a0 0x7fd9b80a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:45.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.296+0000 7fd9bf7fe640 1 -- 172.21.15.169:0/3765955836 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fd9c00cf110 con 0x7fd9d01017d0 2024-03-28T00:56:45.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.297+0000 7fd9ce59c640 1 --2- 172.21.15.169:0/3765955836 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd9b80a28a0 0x7fd9b80a4d60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4000c00 tx=0x7fd9c4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:45.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.301+0000 7fd9bf7fe640 1 -- 172.21.15.169:0/3765955836 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd9c009bdb0 con 0x7fd9d01017d0 2024-03-28T00:56:45.327 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:45 smithi169 ceph-mon[32274]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-28T00:56:45.327 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:45 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/10065003' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-28T00:56:45.468 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:45 smithi177 ceph-mon[36206]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-28T00:56:45.468 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:45 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/10065003' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-28T00:56:45.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.513+0000 7fd9d4d3b640 1 -- 172.21.15.169:0/3765955836 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9d0061b80 con 0x7fd9b80a28a0 2024-03-28T00:56:45.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.528+0000 7fd9bf7fe640 1 -- 172.21.15.169:0/3765955836 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3 (secure 0 0 0) 0x7fd9d0061b80 con 0x7fd9b80a28a0 2024-03-28T00:56:45.530 INFO:teuthology.orchestra.run.smithi169.stdout:{} 2024-03-28T00:56:45.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.530+0000 7fd9d4d3b640 1 -- 172.21.15.169:0/3765955836 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd9b80a28a0 msgr2=0x7fd9b80a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:45.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.530+0000 7fd9d4d3b640 1 --2- 172.21.15.169:0/3765955836 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd9b80a28a0 0x7fd9b80a4d60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4000c00 tx=0x7fd9c4005fd0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.530+0000 7fd9d4d3b640 1 -- 172.21.15.169:0/3765955836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9d01017d0 msgr2=0x7fd9d019c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:45.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.530+0000 7fd9d4d3b640 1 --2- 172.21.15.169:0/3765955836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9d01017d0 0x7fd9d019c660 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd9c000e9f0 tx=0x7fd9c000eeb0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.530+0000 7fd9d4d3b640 1 -- 172.21.15.169:0/3765955836 shutdown_connections 2024-03-28T00:56:45.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.530+0000 7fd9d4d3b640 1 --2- 172.21.15.169:0/3765955836 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd9b80a28a0 0x7fd9b80a4d60 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.530+0000 7fd9d4d3b640 1 --2- 172.21.15.169:0/3765955836 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9d0102190 0x7fd9d019ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.530+0000 7fd9d4d3b640 1 --2- 172.21.15.169:0/3765955836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9d01017d0 0x7fd9d019c660 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.530+0000 7fd9d4d3b640 1 --2- 172.21.15.169:0/3765955836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9d01005e0 0x7fd9d006a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.534 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.530+0000 7fd9d4d3b640 1 -- 172.21.15.169:0/3765955836 >> 172.21.15.169:0/3765955836 conn(0x7fd9d00fbb70 msgr2=0x7fd9d0104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:45.534 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.530+0000 7fd9d4d3b640 1 -- 172.21.15.169:0/3765955836 shutdown_connections 2024-03-28T00:56:45.534 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.530+0000 7fd9d4d3b640 1 -- 172.21.15.169:0/3765955836 wait complete. 2024-03-28T00:56:45.544 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-28T00:56:45.561 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-28T00:56:45.572 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T00:56:45.596 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:56:45 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:56:45.518+0000 7f5236b5a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T00:56:45.597 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:56:45 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:56:45.518+0000 7f5236b5a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T00:56:45.597 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:56:45 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:56:45.518+0000 7f5236b5a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T00:56:45.597 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:56:45 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:56:45.518+0000 7f5236b5a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T00:56:45.597 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 00:56:45 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T00:56:45.518+0000 7f5236b5a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T00:56:45.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:45 smithi174 ceph-mon[36060]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-28T00:56:45.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:45 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/10065003' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-28T00:56:45.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.675+0000 7ff6f7d77640 1 -- 172.21.15.169:0/2852492751 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6f01005c0 msgr2=0x7ff6f01009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:45.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.675+0000 7ff6f7d77640 1 --2- 172.21.15.169:0/2852492751 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6f01005c0 0x7ff6f01009e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec009860 tx=0x7ff6ec02bb70 comp rx=0 tx=0).stop 2024-03-28T00:56:45.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.675+0000 7ff6f7d77640 1 -- 172.21.15.169:0/2852492751 shutdown_connections 2024-03-28T00:56:45.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.675+0000 7ff6f7d77640 1 --2- 172.21.15.169:0/2852492751 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6f0102170 0x7ff6f0108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.675+0000 7ff6f7d77640 1 --2- 172.21.15.169:0/2852492751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6f01017b0 0x7ff6f0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.675+0000 7ff6f7d77640 1 --2- 172.21.15.169:0/2852492751 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6f01005c0 0x7ff6f01009e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.675+0000 7ff6f7d77640 1 -- 172.21.15.169:0/2852492751 >> 172.21.15.169:0/2852492751 conn(0x7ff6f00fbb30 msgr2=0x7ff6f00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:45.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.675+0000 7ff6f7d77640 1 -- 172.21.15.169:0/2852492751 shutdown_connections 2024-03-28T00:56:45.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.675+0000 7ff6f7d77640 1 -- 172.21.15.169:0/2852492751 wait complete. 2024-03-28T00:56:45.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.676+0000 7ff6f7d77640 1 Processor -- start 2024-03-28T00:56:45.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.676+0000 7ff6f7d77640 1 -- start start 2024-03-28T00:56:45.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.676+0000 7ff6f7d77640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6f01005c0 0x7ff6f0196900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:45.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.676+0000 7ff6f7d77640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6f01017b0 0x7ff6f0196e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:45.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.676+0000 7ff6f7d77640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6f0102170 0x7ff6f019deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:45.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.676+0000 7ff6f7d77640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f010c230 con 0x7ff6f01005c0 2024-03-28T00:56:45.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.676+0000 7ff6f7d77640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f010c0b0 con 0x7ff6f0102170 2024-03-28T00:56:45.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.676+0000 7ff6f7d77640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f010c3b0 con 0x7ff6f01017b0 2024-03-28T00:56:45.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.676+0000 7ff6f5b13640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6f01005c0 0x7ff6f0196900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:45.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.676+0000 7ff6f5b13640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6f01005c0 0x7ff6f0196900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51318/0 (socket says 172.21.15.169:51318) 2024-03-28T00:56:45.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.676+0000 7ff6f5b13640 1 -- 172.21.15.169:0/1813060011 learned_addr learned my addr 172.21.15.169:0/1813060011 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:45.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.676+0000 7ff6f6314640 1 --2- 172.21.15.169:0/1813060011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6f0102170 0x7ff6f019deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:45.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.676+0000 7ff6f5312640 1 --2- 172.21.15.169:0/1813060011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6f01017b0 0x7ff6f0196e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:45.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.677+0000 7ff6f5b13640 1 -- 172.21.15.169:0/1813060011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6f01017b0 msgr2=0x7ff6f0196e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:45.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.677+0000 7ff6f5b13640 1 --2- 172.21.15.169:0/1813060011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6f01017b0 0x7ff6f0196e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.677+0000 7ff6f5b13640 1 -- 172.21.15.169:0/1813060011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6f0102170 msgr2=0x7ff6f019deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:45.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.677+0000 7ff6f5b13640 1 --2- 172.21.15.169:0/1813060011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6f0102170 0x7ff6f019deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.677+0000 7ff6f5b13640 1 -- 172.21.15.169:0/1813060011 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f019e520 con 0x7ff6f01005c0 2024-03-28T00:56:45.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.677+0000 7ff6f5b13640 1 --2- 172.21.15.169:0/1813060011 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6f01005c0 0x7ff6f0196900 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec02c0b0 tx=0x7ff6ec032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:45.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.677+0000 7ff6e6ffd640 1 -- 172.21.15.169:0/1813060011 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ec041020 con 0x7ff6f01005c0 2024-03-28T00:56:45.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.677+0000 7ff6f7d77640 1 -- 172.21.15.169:0/1813060011 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f019e7b0 con 0x7ff6f01005c0 2024-03-28T00:56:45.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.677+0000 7ff6e6ffd640 1 -- 172.21.15.169:0/1813060011 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6ec02c9f0 con 0x7ff6f01005c0 2024-03-28T00:56:45.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.677+0000 7ff6e6ffd640 1 -- 172.21.15.169:0/1813060011 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ec039040 con 0x7ff6f01005c0 2024-03-28T00:56:45.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.677+0000 7ff6f7d77640 1 -- 172.21.15.169:0/1813060011 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f019ecf0 con 0x7ff6f01005c0 2024-03-28T00:56:45.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.678+0000 7ff6f7d77640 1 -- 172.21.15.169:0/1813060011 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f0107930 con 0x7ff6f01005c0 2024-03-28T00:56:45.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.679+0000 7ff6e6ffd640 1 -- 172.21.15.169:0/1813060011 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff6ec044020 con 0x7ff6f01005c0 2024-03-28T00:56:45.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.680+0000 7ff6e6ffd640 1 --2- 172.21.15.169:0/1813060011 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff6dc0a2800 0x7ff6dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:45.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.680+0000 7ff6f5312640 1 --2- 172.21.15.169:0/1813060011 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff6dc0a2800 0x7ff6dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:45.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.680+0000 7ff6e6ffd640 1 -- 172.21.15.169:0/1813060011 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ff6ec0f0aa0 con 0x7ff6f01005c0 2024-03-28T00:56:45.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.680+0000 7ff6f5312640 1 --2- 172.21.15.169:0/1813060011 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff6dc0a2800 0x7ff6dc0a4cc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0197e10 tx=0x7ff6e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:45.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.687+0000 7ff6e6ffd640 1 -- 172.21.15.169:0/1813060011 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff6ec04b020 con 0x7ff6f01005c0 2024-03-28T00:56:45.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.872+0000 7ff6f7d77640 1 -- 172.21.15.169:0/1813060011 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6f002d1f0 con 0x7ff6f01005c0 2024-03-28T00:56:45.876 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.873+0000 7ff6e6ffd640 1 -- 172.21.15.169:0/1813060011 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+24823 (secure 0 0 0) 0x7ff6ec0bd7a0 con 0x7ff6f01005c0 2024-03-28T00:56:45.877 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:45.877 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":94,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:56:38.281826+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9},{"from":5,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:56:45.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.875+0000 7ff6f7d77640 1 -- 172.21.15.169:0/1813060011 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff6dc0a2800 msgr2=0x7ff6dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:45.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.875+0000 7ff6f7d77640 1 --2- 172.21.15.169:0/1813060011 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff6dc0a2800 0x7ff6dc0a4cc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0197e10 tx=0x7ff6e0005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:45.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.875+0000 7ff6f7d77640 1 -- 172.21.15.169:0/1813060011 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6f01005c0 msgr2=0x7ff6f0196900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:45.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.875+0000 7ff6f7d77640 1 --2- 172.21.15.169:0/1813060011 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6f01005c0 0x7ff6f0196900 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec02c0b0 tx=0x7ff6ec032430 comp rx=0 tx=0).stop 2024-03-28T00:56:45.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.876+0000 7ff6f7d77640 1 -- 172.21.15.169:0/1813060011 shutdown_connections 2024-03-28T00:56:45.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.876+0000 7ff6f7d77640 1 --2- 172.21.15.169:0/1813060011 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff6dc0a2800 0x7ff6dc0a4cc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.876+0000 7ff6f7d77640 1 --2- 172.21.15.169:0/1813060011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6f0102170 0x7ff6f019deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.876+0000 7ff6f7d77640 1 --2- 172.21.15.169:0/1813060011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6f01017b0 0x7ff6f0196e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.876+0000 7ff6f7d77640 1 --2- 172.21.15.169:0/1813060011 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff6f01005c0 0x7ff6f0196900 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:45.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.876+0000 7ff6f7d77640 1 -- 172.21.15.169:0/1813060011 >> 172.21.15.169:0/1813060011 conn(0x7ff6f00fbb30 msgr2=0x7ff6f0104b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:45.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.876+0000 7ff6f7d77640 1 -- 172.21.15.169:0/1813060011 shutdown_connections 2024-03-28T00:56:45.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:45.876+0000 7ff6f7d77640 1 -- 172.21.15.169:0/1813060011 wait complete. 2024-03-28T00:56:45.890 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T00:49:56.339525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T00:54:31.182562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T00:54:31.780429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T00:54:35.035724+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T00:56:45.890 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T00:56:46.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.032+0000 7f73a3a8c640 1 -- 172.21.15.169:0/212799338 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f739c104a50 msgr2=0x7f739c106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:46.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.032+0000 7f73a3a8c640 1 --2- 172.21.15.169:0/212799338 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f739c104a50 0x7f739c106e40 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7398009900 tx=0x7f739802bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:46.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.033+0000 7f73a3a8c640 1 -- 172.21.15.169:0/212799338 shutdown_connections 2024-03-28T00:56:46.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.033+0000 7f73a3a8c640 1 --2- 172.21.15.169:0/212799338 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f739c107380 0x7f739c1097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.033+0000 7f73a3a8c640 1 --2- 172.21.15.169:0/212799338 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f739c104a50 0x7f739c106e40 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.033+0000 7f73a3a8c640 1 --2- 172.21.15.169:0/212799338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f739c102120 0x7f739c104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.033+0000 7f73a3a8c640 1 -- 172.21.15.169:0/212799338 >> 172.21.15.169:0/212799338 conn(0x7f739c0fbb30 msgr2=0x7f739c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:46.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.033+0000 7f73a3a8c640 1 -- 172.21.15.169:0/212799338 shutdown_connections 2024-03-28T00:56:46.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.033+0000 7f73a3a8c640 1 -- 172.21.15.169:0/212799338 wait complete. 2024-03-28T00:56:46.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.033+0000 7f73a3a8c640 1 Processor -- start 2024-03-28T00:56:46.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.033+0000 7f73a3a8c640 1 -- start start 2024-03-28T00:56:46.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.033+0000 7f73a3a8c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f739c102120 0x7f739c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:46.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.033+0000 7f73a3a8c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f739c104a50 0x7f739c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:46.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.033+0000 7f73a3a8c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f739c107380 0x7f739c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:46.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.033+0000 7f73a3a8c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c10d300 con 0x7f739c104a50 2024-03-28T00:56:46.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.033+0000 7f73a3a8c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c10d180 con 0x7f739c102120 2024-03-28T00:56:46.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.033+0000 7f73a3a8c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c10d480 con 0x7f739c107380 2024-03-28T00:56:46.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.033+0000 7f73a1027640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f739c104a50 0x7f739c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:46.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.034+0000 7f73a1027640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f739c104a50 0x7f739c197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51344/0 (socket says 172.21.15.169:51344) 2024-03-28T00:56:46.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.034+0000 7f73a1027640 1 -- 172.21.15.169:0/3953880617 learned_addr learned my addr 172.21.15.169:0/3953880617 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:46.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.034+0000 7f73a1828640 1 --2- 172.21.15.169:0/3953880617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f739c102120 0x7f739c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:46.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.034+0000 7f73a2029640 1 --2- 172.21.15.169:0/3953880617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f739c107380 0x7f739c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:46.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.034+0000 7f73a1027640 1 -- 172.21.15.169:0/3953880617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f739c107380 msgr2=0x7f739c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:46.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.034+0000 7f73a1027640 1 --2- 172.21.15.169:0/3953880617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f739c107380 0x7f739c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.034+0000 7f73a1027640 1 -- 172.21.15.169:0/3953880617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f739c102120 msgr2=0x7f739c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:46.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.034+0000 7f73a1027640 1 --2- 172.21.15.169:0/3953880617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f739c102120 0x7f739c197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.034+0000 7f73a1027640 1 -- 172.21.15.169:0/3953880617 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739c19f650 con 0x7f739c104a50 2024-03-28T00:56:46.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.034+0000 7f73a1828640 1 --2- 172.21.15.169:0/3953880617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f739c102120 0x7f739c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:46.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.034+0000 7f73a1027640 1 --2- 172.21.15.169:0/3953880617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f739c104a50 0x7f739c197f70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7398000c00 tx=0x7f739802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:46.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.034+0000 7f7392ffd640 1 -- 172.21.15.169:0/3953880617 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7398041070 con 0x7f739c104a50 2024-03-28T00:56:46.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.034+0000 7f73a3a8c640 1 -- 172.21.15.169:0/3953880617 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739c19f8e0 con 0x7f739c104a50 2024-03-28T00:56:46.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.034+0000 7f7392ffd640 1 -- 172.21.15.169:0/3953880617 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7398004450 con 0x7f739c104a50 2024-03-28T00:56:46.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.034+0000 7f7392ffd640 1 -- 172.21.15.169:0/3953880617 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7398003df0 con 0x7f739c104a50 2024-03-28T00:56:46.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.034+0000 7f73a3a8c640 1 -- 172.21.15.169:0/3953880617 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739c19fe20 con 0x7f739c104a50 2024-03-28T00:56:46.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.035+0000 7f73a3a8c640 1 -- 172.21.15.169:0/3953880617 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f739c0fd620 con 0x7f739c104a50 2024-03-28T00:56:46.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.036+0000 7f7392ffd640 1 -- 172.21.15.169:0/3953880617 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7398044070 con 0x7f739c104a50 2024-03-28T00:56:46.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.037+0000 7f7392ffd640 1 --2- 172.21.15.169:0/3953880617 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f73880a2800 0x7f73880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:46.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.037+0000 7f7392ffd640 1 -- 172.21.15.169:0/3953880617 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f73980f0ac0 con 0x7f739c104a50 2024-03-28T00:56:46.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.037+0000 7f73a1828640 1 --2- 172.21.15.169:0/3953880617 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f73880a2800 0x7f73880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:46.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.037+0000 7f73a1828640 1 --2- 172.21.15.169:0/3953880617 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f73880a2800 0x7f73880a4cc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f738c0095f0 tx=0x7f738c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:46.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.045+0000 7f7392ffd640 1 -- 172.21.15.169:0/3953880617 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f73980bd850 con 0x7f739c104a50 2024-03-28T00:56:46.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.225+0000 7f73a3a8c640 1 -- 172.21.15.169:0/3953880617 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f739c02d070 con 0x7f739c104a50 2024-03-28T00:56:46.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.225+0000 7f7392ffd640 1 -- 172.21.15.169:0/3953880617 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7f73980bd150 con 0x7f739c104a50 2024-03-28T00:56:46.228 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 1 2024-03-28T00:56:46.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.227+0000 7f73a3a8c640 1 -- 172.21.15.169:0/3953880617 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f73880a2800 msgr2=0x7f73880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:46.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.227+0000 7f73a3a8c640 1 --2- 172.21.15.169:0/3953880617 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f73880a2800 0x7f73880a4cc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f738c0095f0 tx=0x7f738c005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:46.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.227+0000 7f73a3a8c640 1 -- 172.21.15.169:0/3953880617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f739c104a50 msgr2=0x7f739c197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:46.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.227+0000 7f73a3a8c640 1 --2- 172.21.15.169:0/3953880617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f739c104a50 0x7f739c197f70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7398000c00 tx=0x7f739802c640 comp rx=0 tx=0).stop 2024-03-28T00:56:46.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.228+0000 7f73a3a8c640 1 -- 172.21.15.169:0/3953880617 shutdown_connections 2024-03-28T00:56:46.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.228+0000 7f73a3a8c640 1 --2- 172.21.15.169:0/3953880617 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f73880a2800 0x7f73880a4cc0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.228+0000 7f73a3a8c640 1 --2- 172.21.15.169:0/3953880617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f739c107380 0x7f739c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.228+0000 7f73a3a8c640 1 --2- 172.21.15.169:0/3953880617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f739c104a50 0x7f739c197f70 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.228+0000 7f73a3a8c640 1 --2- 172.21.15.169:0/3953880617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f739c102120 0x7f739c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.228+0000 7f73a3a8c640 1 -- 172.21.15.169:0/3953880617 >> 172.21.15.169:0/3953880617 conn(0x7f739c0fbb30 msgr2=0x7f739c102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:46.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.228+0000 7f73a3a8c640 1 -- 172.21.15.169:0/3953880617 shutdown_connections 2024-03-28T00:56:46.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.228+0000 7f73a3a8c640 1 -- 172.21.15.169:0/3953880617 wait complete. 2024-03-28T00:56:46.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T00:56:46.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.346+0000 7fcfa466c640 1 -- 172.21.15.169:0/342140088 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf9c103260 msgr2=0x7fcf9c109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:46.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.346+0000 7fcfa466c640 1 --2- 172.21.15.169:0/342140088 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf9c103260 0x7fcf9c109af0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fcf94009ca0 tx=0x7fcf9402c0f0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.346+0000 7fcfa466c640 1 -- 172.21.15.169:0/342140088 shutdown_connections 2024-03-28T00:56:46.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.346+0000 7fcfa466c640 1 --2- 172.21.15.169:0/342140088 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf9c103260 0x7fcf9c109af0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.346+0000 7fcfa466c640 1 --2- 172.21.15.169:0/342140088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf9c1028a0 0x7fcf9c102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.346+0000 7fcfa466c640 1 --2- 172.21.15.169:0/342140088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf9c1016b0 0x7fcf9c101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.346+0000 7fcfa466c640 1 -- 172.21.15.169:0/342140088 >> 172.21.15.169:0/342140088 conn(0x7fcf9c0fcc20 msgr2=0x7fcf9c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:46.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.346+0000 7fcfa466c640 1 -- 172.21.15.169:0/342140088 shutdown_connections 2024-03-28T00:56:46.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.346+0000 7fcfa466c640 1 -- 172.21.15.169:0/342140088 wait complete. 2024-03-28T00:56:46.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.347+0000 7fcfa466c640 1 Processor -- start 2024-03-28T00:56:46.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.347+0000 7fcfa466c640 1 -- start start 2024-03-28T00:56:46.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.347+0000 7fcfa466c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf9c1016b0 0x7fcf9c1979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:46.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.347+0000 7fcfa466c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf9c1028a0 0x7fcf9c197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:46.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.347+0000 7fcfa466c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf9c103260 0x7fcf9c19ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:46.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.347+0000 7fcfa466c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c10d330 con 0x7fcf9c1028a0 2024-03-28T00:56:46.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.347+0000 7fcfa466c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c10d1b0 con 0x7fcf9c103260 2024-03-28T00:56:46.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.347+0000 7fcfa466c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c10d4b0 con 0x7fcf9c1016b0 2024-03-28T00:56:46.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.347+0000 7fcfa1c07640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf9c1028a0 0x7fcf9c197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:46.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.347+0000 7fcfa1c07640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf9c1028a0 0x7fcf9c197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51364/0 (socket says 172.21.15.169:51364) 2024-03-28T00:56:46.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.347+0000 7fcfa1c07640 1 -- 172.21.15.169:0/4118586690 learned_addr learned my addr 172.21.15.169:0/4118586690 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:46.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.347+0000 7fcfa2408640 1 --2- 172.21.15.169:0/4118586690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf9c1016b0 0x7fcf9c1979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:46.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.347+0000 7fcfa2c09640 1 --2- 172.21.15.169:0/4118586690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf9c103260 0x7fcf9c19ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:46.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.348+0000 7fcfa1c07640 1 -- 172.21.15.169:0/4118586690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf9c1016b0 msgr2=0x7fcf9c1979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:46.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.348+0000 7fcfa1c07640 1 --2- 172.21.15.169:0/4118586690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf9c1016b0 0x7fcf9c1979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.348+0000 7fcfa1c07640 1 -- 172.21.15.169:0/4118586690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf9c103260 msgr2=0x7fcf9c19ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:46.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.348+0000 7fcfa1c07640 1 --2- 172.21.15.169:0/4118586690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf9c103260 0x7fcf9c19ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.348+0000 7fcfa1c07640 1 -- 172.21.15.169:0/4118586690 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c19f5c0 con 0x7fcf9c1028a0 2024-03-28T00:56:46.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.348+0000 7fcfa2408640 1 --2- 172.21.15.169:0/4118586690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf9c1016b0 0x7fcf9c1979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:46.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.348+0000 7fcfa2c09640 1 --2- 172.21.15.169:0/4118586690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf9c103260 0x7fcf9c19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:46.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.348+0000 7fcfa1c07640 1 --2- 172.21.15.169:0/4118586690 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf9c1028a0 0x7fcf9c197ee0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00e9f0 tx=0x7fcf8c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:46.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.348+0000 7fcf937fe640 1 -- 172.21.15.169:0/4118586690 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c00cd80 con 0x7fcf9c1028a0 2024-03-28T00:56:46.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.348+0000 7fcf937fe640 1 -- 172.21.15.169:0/4118586690 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf8c010910 con 0x7fcf9c1028a0 2024-03-28T00:56:46.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.348+0000 7fcfa466c640 1 -- 172.21.15.169:0/4118586690 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf9c19f8b0 con 0x7fcf9c1028a0 2024-03-28T00:56:46.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.348+0000 7fcf937fe640 1 -- 172.21.15.169:0/4118586690 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c018990 con 0x7fcf9c1028a0 2024-03-28T00:56:46.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.348+0000 7fcfa466c640 1 -- 172.21.15.169:0/4118586690 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9c19fe00 con 0x7fcf9c1028a0 2024-03-28T00:56:46.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.349+0000 7fcfa466c640 1 -- 172.21.15.169:0/4118586690 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf9c108a20 con 0x7fcf9c1028a0 2024-03-28T00:56:46.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.350+0000 7fcf937fe640 1 -- 172.21.15.169:0/4118586690 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fcf8c010ac0 con 0x7fcf9c1028a0 2024-03-28T00:56:46.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.351+0000 7fcf937fe640 1 --2- 172.21.15.169:0/4118586690 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcf880a2800 0x7fcf880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:46.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.351+0000 7fcf937fe640 1 -- 172.21.15.169:0/4118586690 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fcf8c0cf510 con 0x7fcf9c1028a0 2024-03-28T00:56:46.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.351+0000 7fcfa2408640 1 --2- 172.21.15.169:0/4118586690 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcf880a2800 0x7fcf880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:46.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.351+0000 7fcfa2408640 1 --2- 172.21.15.169:0/4118586690 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcf880a2800 0x7fcf880a4cc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c102700 tx=0x7fcf98005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:46.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.354+0000 7fcf937fe640 1 -- 172.21.15.169:0/4118586690 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcf8c09bbc0 con 0x7fcf9c1028a0 2024-03-28T00:56:46.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.529+0000 7fcfa466c640 1 -- 172.21.15.169:0/4118586690 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcf9c02d200 con 0x7fcf9c1028a0 2024-03-28T00:56:46.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.529+0000 7fcf937fe640 1 -- 172.21.15.169:0/4118586690 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcf8c09fc60 con 0x7fcf9c1028a0 2024-03-28T00:56:46.532 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T00:56:46.534 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.531+0000 7fcfa466c640 1 -- 172.21.15.169:0/4118586690 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcf880a2800 msgr2=0x7fcf880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:46.534 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.531+0000 7fcfa466c640 1 --2- 172.21.15.169:0/4118586690 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcf880a2800 0x7fcf880a4cc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c102700 tx=0x7fcf98005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:46.534 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.531+0000 7fcfa466c640 1 -- 172.21.15.169:0/4118586690 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf9c1028a0 msgr2=0x7fcf9c197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:46.534 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.531+0000 7fcfa466c640 1 --2- 172.21.15.169:0/4118586690 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf9c1028a0 0x7fcf9c197ee0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00e9f0 tx=0x7fcf8c00eeb0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.531+0000 7fcfa466c640 1 -- 172.21.15.169:0/4118586690 shutdown_connections 2024-03-28T00:56:46.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.531+0000 7fcfa466c640 1 --2- 172.21.15.169:0/4118586690 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcf880a2800 0x7fcf880a4cc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.531+0000 7fcfa466c640 1 --2- 172.21.15.169:0/4118586690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf9c103260 0x7fcf9c19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.531+0000 7fcfa466c640 1 --2- 172.21.15.169:0/4118586690 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf9c1028a0 0x7fcf9c197ee0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.531+0000 7fcfa466c640 1 --2- 172.21.15.169:0/4118586690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf9c1016b0 0x7fcf9c1979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.531+0000 7fcfa466c640 1 -- 172.21.15.169:0/4118586690 >> 172.21.15.169:0/4118586690 conn(0x7fcf9c0fcc20 msgr2=0x7fcf9c105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:46.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.531+0000 7fcfa466c640 1 -- 172.21.15.169:0/4118586690 shutdown_connections 2024-03-28T00:56:46.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.531+0000 7fcfa466c640 1 -- 172.21.15.169:0/4118586690 wait complete. 2024-03-28T00:56:46.544 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:46 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/10065003' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-28T00:56:46.544 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:46 smithi169 ceph-mon[32274]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T00:56:46.544 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:46 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1813060011' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:46.545 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T00:56:46.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:46 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/10065003' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-28T00:56:46.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:46 smithi174 ceph-mon[36060]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T00:56:46.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:46 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1813060011' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:46.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.648+0000 7f32d239d640 1 -- 172.21.15.169:0/2097460443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32cc101580 msgr2=0x7f32cc1019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.648+0000 7f32d239d640 1 --2- 172.21.15.169:0/2097460443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32cc101580 0x7f32cc1019a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f32c8009860 tx=0x7f32c802bb70 comp rx=0 tx=0).stop 2024-03-28T00:56:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.648+0000 7f32d239d640 1 -- 172.21.15.169:0/2097460443 shutdown_connections 2024-03-28T00:56:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.648+0000 7f32d239d640 1 --2- 172.21.15.169:0/2097460443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32cc103130 0x7f32cc1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.648+0000 7f32d239d640 1 --2- 172.21.15.169:0/2097460443 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32cc102770 0x7f32cc102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.648+0000 7f32d239d640 1 --2- 172.21.15.169:0/2097460443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32cc101580 0x7f32cc1019a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.648+0000 7f32d239d640 1 -- 172.21.15.169:0/2097460443 >> 172.21.15.169:0/2097460443 conn(0x7f32cc0fcb10 msgr2=0x7f32cc0fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.648+0000 7f32d239d640 1 -- 172.21.15.169:0/2097460443 shutdown_connections 2024-03-28T00:56:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.648+0000 7f32d239d640 1 -- 172.21.15.169:0/2097460443 wait complete. 2024-03-28T00:56:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.648+0000 7f32d239d640 1 Processor -- start 2024-03-28T00:56:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.648+0000 7f32d239d640 1 -- start start 2024-03-28T00:56:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.648+0000 7f32d239d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32cc101580 0x7f32cc197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.648+0000 7f32d239d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32cc102770 0x7f32cc197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.648+0000 7f32d239d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32cc103130 0x7f32cc19ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.648+0000 7f32d239d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc10d210 con 0x7f32cc103130 2024-03-28T00:56:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.648+0000 7f32d239d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc10d090 con 0x7f32cc101580 2024-03-28T00:56:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.648+0000 7f32d239d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc10d390 con 0x7f32cc102770 2024-03-28T00:56:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.649+0000 7f32d1b9c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32cc103130 0x7f32cc19ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.649+0000 7f32d139b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32cc101580 0x7f32cc197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.649+0000 7f32d0b9a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32cc102770 0x7f32cc197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.649+0000 7f32d1b9c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32cc103130 0x7f32cc19ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34626/0 (socket says 172.21.15.169:34626) 2024-03-28T00:56:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.649+0000 7f32d1b9c640 1 -- 172.21.15.169:0/3344783440 learned_addr learned my addr 172.21.15.169:0/3344783440 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.649+0000 7f32d1b9c640 1 -- 172.21.15.169:0/3344783440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32cc102770 msgr2=0x7f32cc197da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.649+0000 7f32d1b9c640 1 --2- 172.21.15.169:0/3344783440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32cc102770 0x7f32cc197da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.649+0000 7f32d1b9c640 1 -- 172.21.15.169:0/3344783440 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32cc101580 msgr2=0x7f32cc197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:46.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.649+0000 7f32d1b9c640 1 --2- 172.21.15.169:0/3344783440 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32cc101580 0x7f32cc197860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.649+0000 7f32d1b9c640 1 -- 172.21.15.169:0/3344783440 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32cc19f480 con 0x7f32cc103130 2024-03-28T00:56:46.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.649+0000 7f32d139b640 1 --2- 172.21.15.169:0/3344783440 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32cc101580 0x7f32cc197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:46.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.649+0000 7f32d1b9c640 1 --2- 172.21.15.169:0/3344783440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32cc103130 0x7f32cc19ee10 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f32c400eb10 tx=0x7f32c400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:46.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.649+0000 7f32c27fc640 1 -- 172.21.15.169:0/3344783440 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c4004b20 con 0x7f32cc103130 2024-03-28T00:56:46.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.649+0000 7f32d239d640 1 -- 172.21.15.169:0/3344783440 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32cc19f770 con 0x7f32cc103130 2024-03-28T00:56:46.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.649+0000 7f32c27fc640 1 -- 172.21.15.169:0/3344783440 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32c4004cc0 con 0x7f32cc103130 2024-03-28T00:56:46.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.649+0000 7f32c27fc640 1 -- 172.21.15.169:0/3344783440 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c4010760 con 0x7f32cc103130 2024-03-28T00:56:46.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.649+0000 7f32d239d640 1 -- 172.21.15.169:0/3344783440 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32cc19fcc0 con 0x7f32cc103130 2024-03-28T00:56:46.656 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.650+0000 7f32d239d640 1 -- 172.21.15.169:0/3344783440 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32cc1088a0 con 0x7f32cc103130 2024-03-28T00:56:46.656 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.651+0000 7f32c27fc640 1 -- 172.21.15.169:0/3344783440 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f32c4005000 con 0x7f32cc103130 2024-03-28T00:56:46.656 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.652+0000 7f32c27fc640 1 --2- 172.21.15.169:0/3344783440 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f32b80a2800 0x7f32b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:46.656 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.652+0000 7f32d139b640 1 --2- 172.21.15.169:0/3344783440 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f32b80a2800 0x7f32b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:46.656 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.652+0000 7f32c27fc640 1 -- 172.21.15.169:0/3344783440 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f32c40cf850 con 0x7f32cc103130 2024-03-28T00:56:46.656 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.652+0000 7f32d139b640 1 --2- 172.21.15.169:0/3344783440 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f32b80a2800 0x7f32b80a4cc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f32c8009830 tx=0x7f32c802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:46.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.655+0000 7f32c27fc640 1 -- 172.21.15.169:0/3344783440 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f32c409c550 con 0x7f32cc103130 2024-03-28T00:56:46.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:46 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/10065003' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-28T00:56:46.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:46 smithi177 ceph-mon[36206]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T00:56:46.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:46 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1813060011' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:46.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.831+0000 7f32d239d640 1 -- 172.21.15.169:0/3344783440 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f32cc02d070 con 0x7f32cc103130 2024-03-28T00:56:46.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.831+0000 7f32c27fc640 1 -- 172.21.15.169:0/3344783440 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) v1 ==== 100+0+11 (secure 0 0 0) 0x7f32c409be50 con 0x7f32cc103130 2024-03-28T00:56:46.834 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 64 2024-03-28T00:56:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.833+0000 7f32d239d640 1 -- 172.21.15.169:0/3344783440 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f32b80a2800 msgr2=0x7f32b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.833+0000 7f32d239d640 1 --2- 172.21.15.169:0/3344783440 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f32b80a2800 0x7f32b80a4cc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f32c8009830 tx=0x7f32c802e040 comp rx=0 tx=0).stop 2024-03-28T00:56:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.833+0000 7f32d239d640 1 -- 172.21.15.169:0/3344783440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32cc103130 msgr2=0x7f32cc19ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.833+0000 7f32d239d640 1 --2- 172.21.15.169:0/3344783440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32cc103130 0x7f32cc19ee10 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f32c400eb10 tx=0x7f32c400c2d0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.833+0000 7f32d239d640 1 -- 172.21.15.169:0/3344783440 shutdown_connections 2024-03-28T00:56:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.833+0000 7f32d239d640 1 --2- 172.21.15.169:0/3344783440 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f32b80a2800 0x7f32b80a4cc0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.833+0000 7f32d239d640 1 --2- 172.21.15.169:0/3344783440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32cc103130 0x7f32cc19ee10 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.833+0000 7f32d239d640 1 --2- 172.21.15.169:0/3344783440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32cc102770 0x7f32cc197da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.833+0000 7f32d239d640 1 --2- 172.21.15.169:0/3344783440 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32cc101580 0x7f32cc197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.833+0000 7f32d239d640 1 -- 172.21.15.169:0/3344783440 >> 172.21.15.169:0/3344783440 conn(0x7f32cc0fcb10 msgr2=0x7f32cc105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:46.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.833+0000 7f32d239d640 1 -- 172.21.15.169:0/3344783440 shutdown_connections 2024-03-28T00:56:46.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.833+0000 7f32d239d640 1 -- 172.21.15.169:0/3344783440 wait complete. 2024-03-28T00:56:46.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T00:56:46.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.949+0000 7faed273b640 1 -- 172.21.15.169:0/2654252420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faecc1028a0 msgr2=0x7faecc102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:46.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.949+0000 7faed273b640 1 --2- 172.21.15.169:0/2654252420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faecc1028a0 0x7faecc102d20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7faec0009900 tx=0x7faec002bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:46.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.949+0000 7faed273b640 1 -- 172.21.15.169:0/2654252420 shutdown_connections 2024-03-28T00:56:46.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.949+0000 7faed273b640 1 --2- 172.21.15.169:0/2654252420 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faecc103260 0x7faecc109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.949+0000 7faed273b640 1 --2- 172.21.15.169:0/2654252420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faecc1028a0 0x7faecc102d20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.949+0000 7faed273b640 1 --2- 172.21.15.169:0/2654252420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faecc1016b0 0x7faecc101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.949+0000 7faed273b640 1 -- 172.21.15.169:0/2654252420 >> 172.21.15.169:0/2654252420 conn(0x7faecc0fcc40 msgr2=0x7faecc0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:46.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.949+0000 7faed273b640 1 -- 172.21.15.169:0/2654252420 shutdown_connections 2024-03-28T00:56:46.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.949+0000 7faed273b640 1 -- 172.21.15.169:0/2654252420 wait complete. 2024-03-28T00:56:46.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.950+0000 7faed273b640 1 Processor -- start 2024-03-28T00:56:46.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.950+0000 7faed273b640 1 -- start start 2024-03-28T00:56:46.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.950+0000 7faed273b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faecc1016b0 0x7faecc1979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:46.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.950+0000 7faed273b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faecc1028a0 0x7faecc197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:46.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.950+0000 7faed273b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faecc103260 0x7faecc19ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:46.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.950+0000 7faed273b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc10d330 con 0x7faecc103260 2024-03-28T00:56:46.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.950+0000 7faed273b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc10d1b0 con 0x7faecc1028a0 2024-03-28T00:56:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.950+0000 7faed273b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc10d4b0 con 0x7faecc1016b0 2024-03-28T00:56:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.950+0000 7faed0cd8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faecc103260 0x7faecc19ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.950+0000 7faed0cd8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faecc103260 0x7faecc19ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34650/0 (socket says 172.21.15.169:34650) 2024-03-28T00:56:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.950+0000 7faed0cd8640 1 -- 172.21.15.169:0/1003765485 learned_addr learned my addr 172.21.15.169:0/1003765485 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.950+0000 7faecb7fe640 1 --2- 172.21.15.169:0/1003765485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faecc1028a0 0x7faecc197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.950+0000 7faecbfff640 1 --2- 172.21.15.169:0/1003765485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faecc1016b0 0x7faecc1979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.950+0000 7faed0cd8640 1 -- 172.21.15.169:0/1003765485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faecc1016b0 msgr2=0x7faecc1979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.950+0000 7faed0cd8640 1 --2- 172.21.15.169:0/1003765485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faecc1016b0 0x7faecc1979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.950+0000 7faed0cd8640 1 -- 172.21.15.169:0/1003765485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faecc1028a0 msgr2=0x7faecc197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.950+0000 7faed0cd8640 1 --2- 172.21.15.169:0/1003765485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faecc1028a0 0x7faecc197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.950+0000 7faed0cd8640 1 -- 172.21.15.169:0/1003765485 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc19f600 con 0x7faecc103260 2024-03-28T00:56:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.951+0000 7faed0cd8640 1 --2- 172.21.15.169:0/1003765485 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faecc103260 0x7faecc19ef90 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7faec400efd0 tx=0x7faec400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.951+0000 7faec97fa640 1 -- 172.21.15.169:0/1003765485 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec4003b70 con 0x7faecc103260 2024-03-28T00:56:46.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.951+0000 7faec97fa640 1 -- 172.21.15.169:0/1003765485 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faec4003d10 con 0x7faecc103260 2024-03-28T00:56:46.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.951+0000 7faed273b640 1 -- 172.21.15.169:0/1003765485 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faecc19f8f0 con 0x7faecc103260 2024-03-28T00:56:46.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.951+0000 7faec97fa640 1 -- 172.21.15.169:0/1003765485 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec40109f0 con 0x7faecc103260 2024-03-28T00:56:46.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.951+0000 7faed273b640 1 -- 172.21.15.169:0/1003765485 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faecc19fdc0 con 0x7faecc103260 2024-03-28T00:56:46.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.952+0000 7faed273b640 1 -- 172.21.15.169:0/1003765485 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faecc108a90 con 0x7faecc103260 2024-03-28T00:56:46.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.953+0000 7faec97fa640 1 -- 172.21.15.169:0/1003765485 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7faec4004020 con 0x7faecc103260 2024-03-28T00:56:46.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.954+0000 7faec97fa640 1 --2- 172.21.15.169:0/1003765485 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faeb40a2800 0x7faeb40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:46.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.954+0000 7faecbfff640 1 --2- 172.21.15.169:0/1003765485 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faeb40a2800 0x7faeb40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:46.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.954+0000 7faec97fa640 1 -- 172.21.15.169:0/1003765485 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7faec40cf270 con 0x7faecc103260 2024-03-28T00:56:46.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.954+0000 7faecbfff640 1 --2- 172.21.15.169:0/1003765485 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faeb40a2800 0x7faeb40a4cc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7faecc102700 tx=0x7faebc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:46.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:46.958+0000 7faec97fa640 1 -- 172.21.15.169:0/1003765485 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faec409bf70 con 0x7faecc103260 2024-03-28T00:56:47.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.133+0000 7faed273b640 1 -- 172.21.15.169:0/1003765485 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7faecc0622a0 con 0x7faecc103260 2024-03-28T00:56:47.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.134+0000 7faec97fa640 1 -- 172.21.15.169:0/1003765485 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) v1 ==== 103+0+11 (secure 0 0 0) 0x7faec409b870 con 0x7faecc103260 2024-03-28T00:56:47.137 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T00:56:47.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.136+0000 7faed273b640 1 -- 172.21.15.169:0/1003765485 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faeb40a2800 msgr2=0x7faeb40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:47.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.136+0000 7faed273b640 1 --2- 172.21.15.169:0/1003765485 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faeb40a2800 0x7faeb40a4cc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7faecc102700 tx=0x7faebc005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:47.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.136+0000 7faed273b640 1 -- 172.21.15.169:0/1003765485 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faecc103260 msgr2=0x7faecc19ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:47.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.136+0000 7faed273b640 1 --2- 172.21.15.169:0/1003765485 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faecc103260 0x7faecc19ef90 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7faec400efd0 tx=0x7faec400c6a0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.136+0000 7faed273b640 1 -- 172.21.15.169:0/1003765485 shutdown_connections 2024-03-28T00:56:47.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.136+0000 7faed273b640 1 --2- 172.21.15.169:0/1003765485 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faeb40a2800 0x7faeb40a4cc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.136+0000 7faed273b640 1 --2- 172.21.15.169:0/1003765485 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faecc103260 0x7faecc19ef90 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.136+0000 7faed273b640 1 --2- 172.21.15.169:0/1003765485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faecc1028a0 0x7faecc197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.136+0000 7faed273b640 1 --2- 172.21.15.169:0/1003765485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faecc1016b0 0x7faecc1979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.136+0000 7faed273b640 1 -- 172.21.15.169:0/1003765485 >> 172.21.15.169:0/1003765485 conn(0x7faecc0fcc40 msgr2=0x7faecc105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:47.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.136+0000 7faed273b640 1 -- 172.21.15.169:0/1003765485 shutdown_connections 2024-03-28T00:56:47.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.136+0000 7faed273b640 1 -- 172.21.15.169:0/1003765485 wait complete. 2024-03-28T00:56:47.150 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T00:56:47.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.249+0000 7f82a53e6640 1 -- 172.21.15.169:0/3259529109 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82a01017b0 msgr2=0x7f82a0101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:47.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.249+0000 7f82a53e6640 1 --2- 172.21.15.169:0/3259529109 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82a01017b0 0x7f82a0101c30 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8294009900 tx=0x7f829402bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.249+0000 7f82a53e6640 1 -- 172.21.15.169:0/3259529109 shutdown_connections 2024-03-28T00:56:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.249+0000 7f82a53e6640 1 --2- 172.21.15.169:0/3259529109 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82a0102170 0x7f82a0108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.249+0000 7f82a53e6640 1 --2- 172.21.15.169:0/3259529109 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82a01017b0 0x7f82a0101c30 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.249+0000 7f82a53e6640 1 --2- 172.21.15.169:0/3259529109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82a01005c0 0x7f82a01009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.249+0000 7f82a53e6640 1 -- 172.21.15.169:0/3259529109 >> 172.21.15.169:0/3259529109 conn(0x7f82a00fbb30 msgr2=0x7f82a00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.249+0000 7f82a53e6640 1 -- 172.21.15.169:0/3259529109 shutdown_connections 2024-03-28T00:56:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.250+0000 7f82a53e6640 1 -- 172.21.15.169:0/3259529109 wait complete. 2024-03-28T00:56:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.250+0000 7f82a53e6640 1 Processor -- start 2024-03-28T00:56:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.250+0000 7f82a53e6640 1 -- start start 2024-03-28T00:56:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.250+0000 7f82a53e6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82a01005c0 0x7f82a0104620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.250+0000 7f82a53e6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82a01017b0 0x7f82a0102c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:47.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.250+0000 7f82a53e6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82a0102170 0x7f82a01031a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.250+0000 7f82a53e6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a010c220 con 0x7f82a01005c0 2024-03-28T00:56:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.250+0000 7f82a53e6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a010c0a0 con 0x7f82a01017b0 2024-03-28T00:56:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.250+0000 7f82a53e6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a010c3a0 con 0x7f82a0102170 2024-03-28T00:56:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.251+0000 7f829effd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82a01005c0 0x7f82a0104620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.251+0000 7f829effd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82a01005c0 0x7f82a0104620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34678/0 (socket says 172.21.15.169:34678) 2024-03-28T00:56:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.251+0000 7f829effd640 1 -- 172.21.15.169:0/1261889938 learned_addr learned my addr 172.21.15.169:0/1261889938 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.251+0000 7f829e7fc640 1 --2- 172.21.15.169:0/1261889938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82a01017b0 0x7f82a0102c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.251+0000 7f829f7fe640 1 --2- 172.21.15.169:0/1261889938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82a0102170 0x7f82a01031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.251+0000 7f829effd640 1 -- 172.21.15.169:0/1261889938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82a0102170 msgr2=0x7f82a01031a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.251+0000 7f829effd640 1 --2- 172.21.15.169:0/1261889938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82a0102170 0x7f82a01031a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.251+0000 7f829effd640 1 -- 172.21.15.169:0/1261889938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82a01017b0 msgr2=0x7f82a0102c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.251+0000 7f829effd640 1 --2- 172.21.15.169:0/1261889938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82a01017b0 0x7f82a0102c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.251+0000 7f829effd640 1 -- 172.21.15.169:0/1261889938 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a0103a30 con 0x7f82a01005c0 2024-03-28T00:56:47.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.251+0000 7f829effd640 1 --2- 172.21.15.169:0/1261889938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82a01005c0 0x7f82a0104620 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f829000e9c0 tx=0x7f829000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:47.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.251+0000 7f8287fff640 1 -- 172.21.15.169:0/1261889938 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829000cd80 con 0x7f82a01005c0 2024-03-28T00:56:47.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.251+0000 7f8287fff640 1 -- 172.21.15.169:0/1261889938 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82900035f0 con 0x7f82a01005c0 2024-03-28T00:56:47.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.251+0000 7f82a53e6640 1 -- 172.21.15.169:0/1261889938 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82a01a1b00 con 0x7f82a01005c0 2024-03-28T00:56:47.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.251+0000 7f8287fff640 1 -- 172.21.15.169:0/1261889938 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8290010690 con 0x7f82a01005c0 2024-03-28T00:56:47.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.251+0000 7f82a53e6640 1 -- 172.21.15.169:0/1261889938 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a01a2020 con 0x7f82a01005c0 2024-03-28T00:56:47.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.252+0000 7f82a53e6640 1 -- 172.21.15.169:0/1261889938 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82a0068490 con 0x7f82a01005c0 2024-03-28T00:56:47.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.253+0000 7f8287fff640 1 -- 172.21.15.169:0/1261889938 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f82900037a0 con 0x7f82a01005c0 2024-03-28T00:56:47.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.254+0000 7f8287fff640 1 --2- 172.21.15.169:0/1261889938 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f82880a2800 0x7f82880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:47.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.254+0000 7f829e7fc640 1 --2- 172.21.15.169:0/1261889938 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f82880a2800 0x7f82880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:47.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.254+0000 7f8287fff640 1 -- 172.21.15.169:0/1261889938 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f82900cf1f0 con 0x7f82a01005c0 2024-03-28T00:56:47.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.254+0000 7f829e7fc640 1 --2- 172.21.15.169:0/1261889938 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f82880a2800 0x7f82880a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f829402c7b0 tx=0x7f8294005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:47.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.258+0000 7f8287fff640 1 -- 172.21.15.169:0/1261889938 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f829009bef0 con 0x7f82a01005c0 2024-03-28T00:56:47.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.435+0000 7f82a53e6640 1 -- 172.21.15.169:0/1261889938 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82a01a29e0 con 0x7f82a01005c0 2024-03-28T00:56:47.440 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.437+0000 7f8287fff640 1 -- 172.21.15.169:0/1261889938 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+24823 (secure 0 0 0) 0x7f829009b7f0 con 0x7f82a01005c0 2024-03-28T00:56:47.440 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:47.441 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":94,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:56:38.281826+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9},{"from":5,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:56:47.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.439+0000 7f82a53e6640 1 -- 172.21.15.169:0/1261889938 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f82880a2800 msgr2=0x7f82880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:47.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.439+0000 7f82a53e6640 1 --2- 172.21.15.169:0/1261889938 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f82880a2800 0x7f82880a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f829402c7b0 tx=0x7f8294005fd0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.439+0000 7f82a53e6640 1 -- 172.21.15.169:0/1261889938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82a01005c0 msgr2=0x7f82a0104620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:47.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.439+0000 7f82a53e6640 1 --2- 172.21.15.169:0/1261889938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82a01005c0 0x7f82a0104620 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f829000e9c0 tx=0x7f829000ee80 comp rx=0 tx=0).stop 2024-03-28T00:56:47.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.439+0000 7f82a53e6640 1 -- 172.21.15.169:0/1261889938 shutdown_connections 2024-03-28T00:56:47.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.439+0000 7f82a53e6640 1 --2- 172.21.15.169:0/1261889938 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f82880a2800 0x7f82880a4cc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.439+0000 7f82a53e6640 1 --2- 172.21.15.169:0/1261889938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82a0102170 0x7f82a01031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.439+0000 7f82a53e6640 1 --2- 172.21.15.169:0/1261889938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82a01017b0 0x7f82a0102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.439+0000 7f82a53e6640 1 --2- 172.21.15.169:0/1261889938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82a01005c0 0x7f82a0104620 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.439+0000 7f82a53e6640 1 -- 172.21.15.169:0/1261889938 >> 172.21.15.169:0/1261889938 conn(0x7f82a00fbb30 msgr2=0x7f82a0068800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:47.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.439+0000 7f82a53e6640 1 -- 172.21.15.169:0/1261889938 shutdown_connections 2024-03-28T00:56:47.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.439+0000 7f82a53e6640 1 -- 172.21.15.169:0/1261889938 wait complete. 2024-03-28T00:56:47.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:47 smithi169 ceph-mon[32274]: from='client.25249 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:56:47.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:47 smithi169 ceph-mon[32274]: pgmap v359: 129 pgs: 129 active+clean; 647 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 231 B/s wr, 7 op/s 2024-03-28T00:56:47.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:47 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3953880617' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:56:47.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:47 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4118586690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:56:47.452 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:47 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3344783440' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:56:47.453 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:47 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1003765485' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:56:47.454 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T00:49:56.339525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T00:54:31.182562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T00:54:31.780429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T00:54:35.035724+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T00:56:47.455 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T00:56:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.558+0000 7f4d88297640 1 -- 172.21.15.169:0/3080135850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4d801005a0 msgr2=0x7f4d801009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.558+0000 7f4d88297640 1 --2- 172.21.15.169:0/3080135850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4d801005a0 0x7f4d801009c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009860 tx=0x7f4d7c02bb70 comp rx=0 tx=0).stop 2024-03-28T00:56:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.558+0000 7f4d88297640 1 -- 172.21.15.169:0/3080135850 shutdown_connections 2024-03-28T00:56:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.558+0000 7f4d88297640 1 --2- 172.21.15.169:0/3080135850 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d80102150 0x7f4d801089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.558+0000 7f4d88297640 1 --2- 172.21.15.169:0/3080135850 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d80101790 0x7f4d80101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.558+0000 7f4d88297640 1 --2- 172.21.15.169:0/3080135850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4d801005a0 0x7f4d801009c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.558+0000 7f4d88297640 1 -- 172.21.15.169:0/3080135850 >> 172.21.15.169:0/3080135850 conn(0x7f4d800fbb30 msgr2=0x7f4d800fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.558+0000 7f4d88297640 1 -- 172.21.15.169:0/3080135850 shutdown_connections 2024-03-28T00:56:47.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.558+0000 7f4d88297640 1 -- 172.21.15.169:0/3080135850 wait complete. 2024-03-28T00:56:47.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.558+0000 7f4d88297640 1 Processor -- start 2024-03-28T00:56:47.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.558+0000 7f4d88297640 1 -- start start 2024-03-28T00:56:47.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.558+0000 7f4d88297640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d801005a0 0x7f4d801979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:47.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.558+0000 7f4d88297640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d80101790 0x7f4d80197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:47.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.558+0000 7f4d88297640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4d80102150 0x7f4d8019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:47.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.558+0000 7f4d88297640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8010c230 con 0x7f4d801005a0 2024-03-28T00:56:47.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.558+0000 7f4d88297640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8010c0b0 con 0x7f4d80102150 2024-03-28T00:56:47.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.558+0000 7f4d88297640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8010c3b0 con 0x7f4d80101790 2024-03-28T00:56:47.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.559+0000 7f4d86033640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d801005a0 0x7f4d801979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:47.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.559+0000 7f4d86033640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d801005a0 0x7f4d801979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34698/0 (socket says 172.21.15.169:34698) 2024-03-28T00:56:47.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.559+0000 7f4d86033640 1 -- 172.21.15.169:0/2264489908 learned_addr learned my addr 172.21.15.169:0/2264489908 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:47.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.559+0000 7f4d85832640 1 --2- 172.21.15.169:0/2264489908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d80101790 0x7f4d80197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:47.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.559+0000 7f4d86834640 1 --2- 172.21.15.169:0/2264489908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4d80102150 0x7f4d8019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:47.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.559+0000 7f4d86033640 1 -- 172.21.15.169:0/2264489908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d80101790 msgr2=0x7f4d80197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:47.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.559+0000 7f4d86033640 1 --2- 172.21.15.169:0/2264489908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d80101790 0x7f4d80197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.559+0000 7f4d86033640 1 -- 172.21.15.169:0/2264489908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4d80102150 msgr2=0x7f4d8019efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:47.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.559+0000 7f4d86033640 1 --2- 172.21.15.169:0/2264489908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4d80102150 0x7f4d8019efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.559+0000 7f4d86033640 1 -- 172.21.15.169:0/2264489908 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8019f610 con 0x7f4d801005a0 2024-03-28T00:56:47.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.559+0000 7f4d86033640 1 --2- 172.21.15.169:0/2264489908 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d801005a0 0x7f4d801979f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c02c0b0 tx=0x7f4d7c032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:47.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.559+0000 7f4d777fe640 1 -- 172.21.15.169:0/2264489908 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c041020 con 0x7f4d801005a0 2024-03-28T00:56:47.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.559+0000 7f4d88297640 1 -- 172.21.15.169:0/2264489908 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d8019f8a0 con 0x7f4d801005a0 2024-03-28T00:56:47.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.559+0000 7f4d777fe640 1 -- 172.21.15.169:0/2264489908 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d7c02c9f0 con 0x7f4d801005a0 2024-03-28T00:56:47.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.559+0000 7f4d777fe640 1 -- 172.21.15.169:0/2264489908 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c039040 con 0x7f4d801005a0 2024-03-28T00:56:47.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.559+0000 7f4d88297640 1 -- 172.21.15.169:0/2264489908 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8019fde0 con 0x7f4d801005a0 2024-03-28T00:56:47.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.560+0000 7f4d88297640 1 -- 172.21.15.169:0/2264489908 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d801a00a0 con 0x7f4d801005a0 2024-03-28T00:56:47.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.561+0000 7f4d777fe640 1 -- 172.21.15.169:0/2264489908 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4d7c044020 con 0x7f4d801005a0 2024-03-28T00:56:47.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.562+0000 7f4d777fe640 1 --2- 172.21.15.169:0/2264489908 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4d6c0a2800 0x7f4d6c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:47.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.562+0000 7f4d85832640 1 --2- 172.21.15.169:0/2264489908 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4d6c0a2800 0x7f4d6c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:47.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.562+0000 7f4d777fe640 1 -- 172.21.15.169:0/2264489908 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f4d7c0f06d0 con 0x7f4d801005a0 2024-03-28T00:56:47.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.562+0000 7f4d85832640 1 --2- 172.21.15.169:0/2264489908 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4d6c0a2800 0x7f4d6c0a4cc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4d700095f0 tx=0x7f4d70005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:47.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.566+0000 7f4d777fe640 1 -- 172.21.15.169:0/2264489908 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4d7c04b020 con 0x7f4d801005a0 2024-03-28T00:56:47.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:47 smithi174 ceph-mon[36060]: from='client.25249 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:56:47.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:47 smithi174 ceph-mon[36060]: pgmap v359: 129 pgs: 129 active+clean; 647 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 231 B/s wr, 7 op/s 2024-03-28T00:56:47.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:47 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3953880617' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:56:47.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:47 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/4118586690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:56:47.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:47 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3344783440' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:56:47.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:47 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1003765485' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:56:47.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:47 smithi177 ceph-mon[36206]: from='client.25249 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T00:56:47.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:47 smithi177 ceph-mon[36206]: pgmap v359: 129 pgs: 129 active+clean; 647 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 231 B/s wr, 7 op/s 2024-03-28T00:56:47.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:47 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3953880617' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:56:47.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:47 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/4118586690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:56:47.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:47 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3344783440' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:56:47.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:47 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1003765485' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:56:47.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.741+0000 7f4d88297640 1 -- 172.21.15.169:0/2264489908 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4d801a0270 con 0x7f4d801005a0 2024-03-28T00:56:47.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.742+0000 7f4d777fe640 1 -- 172.21.15.169:0/2264489908 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4d7c0bd3d0 con 0x7f4d801005a0 2024-03-28T00:56:47.745 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 1 2024-03-28T00:56:47.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.744+0000 7f4d88297640 1 -- 172.21.15.169:0/2264489908 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4d6c0a2800 msgr2=0x7f4d6c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:47.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.744+0000 7f4d88297640 1 --2- 172.21.15.169:0/2264489908 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4d6c0a2800 0x7f4d6c0a4cc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4d700095f0 tx=0x7f4d70005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:47.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.744+0000 7f4d88297640 1 -- 172.21.15.169:0/2264489908 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d801005a0 msgr2=0x7f4d801979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:47.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.744+0000 7f4d88297640 1 --2- 172.21.15.169:0/2264489908 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d801005a0 0x7f4d801979f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c02c0b0 tx=0x7f4d7c032430 comp rx=0 tx=0).stop 2024-03-28T00:56:47.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.744+0000 7f4d88297640 1 -- 172.21.15.169:0/2264489908 shutdown_connections 2024-03-28T00:56:47.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.744+0000 7f4d88297640 1 --2- 172.21.15.169:0/2264489908 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4d6c0a2800 0x7f4d6c0a4cc0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.744+0000 7f4d88297640 1 --2- 172.21.15.169:0/2264489908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4d80102150 0x7f4d8019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.744+0000 7f4d88297640 1 --2- 172.21.15.169:0/2264489908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d80101790 0x7f4d80197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.744+0000 7f4d88297640 1 --2- 172.21.15.169:0/2264489908 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d801005a0 0x7f4d801979f0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.744+0000 7f4d88297640 1 -- 172.21.15.169:0/2264489908 >> 172.21.15.169:0/2264489908 conn(0x7f4d800fbb30 msgr2=0x7f4d80068800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:47.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.744+0000 7f4d88297640 1 -- 172.21.15.169:0/2264489908 shutdown_connections 2024-03-28T00:56:47.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.744+0000 7f4d88297640 1 -- 172.21.15.169:0/2264489908 wait complete. 2024-03-28T00:56:47.758 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T00:56:47.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.860+0000 7f5114e9a640 1 -- 172.21.15.169:0/1128009380 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51101000d0 msgr2=0x7f5110109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:47.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.860+0000 7f5114e9a640 1 --2- 172.21.15.169:0/1128009380 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51101000d0 0x7f5110109c80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5108009be0 tx=0x7f510802bef0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.860+0000 7f5114e9a640 1 -- 172.21.15.169:0/1128009380 shutdown_connections 2024-03-28T00:56:47.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.860+0000 7f5114e9a640 1 --2- 172.21.15.169:0/1128009380 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51101000d0 0x7f5110109c80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.860+0000 7f5114e9a640 1 --2- 172.21.15.169:0/1128009380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51100ff710 0x7f51100ffb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.860+0000 7f5114e9a640 1 --2- 172.21.15.169:0/1128009380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5110102ca0 0x7f51101030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.860+0000 7f5114e9a640 1 -- 172.21.15.169:0/1128009380 >> 172.21.15.169:0/1128009380 conn(0x7f51100fb2d0 msgr2=0x7f51100fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:47.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.860+0000 7f5114e9a640 1 -- 172.21.15.169:0/1128009380 shutdown_connections 2024-03-28T00:56:47.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.860+0000 7f5114e9a640 1 -- 172.21.15.169:0/1128009380 wait complete. 2024-03-28T00:56:47.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.860+0000 7f5114e9a640 1 Processor -- start 2024-03-28T00:56:47.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.860+0000 7f5114e9a640 1 -- start start 2024-03-28T00:56:47.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f5114e9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51100ff710 0x7f5110197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:47.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f5114e9a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51101000d0 0x7f5110197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:47.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f5114e9a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5110102ca0 0x7f511019eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:47.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f5114e9a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511010d340 con 0x7f51100ff710 2024-03-28T00:56:47.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f5114e9a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511010d1c0 con 0x7f5110102ca0 2024-03-28T00:56:47.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f5114e9a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511010d4c0 con 0x7f51101000d0 2024-03-28T00:56:47.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f510e59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51100ff710 0x7f5110197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:47.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f510e59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51100ff710 0x7f5110197920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34720/0 (socket says 172.21.15.169:34720) 2024-03-28T00:56:47.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f510e59c640 1 -- 172.21.15.169:0/3671801509 learned_addr learned my addr 172.21.15.169:0/3671801509 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:47.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f510dd9b640 1 --2- 172.21.15.169:0/3671801509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51101000d0 0x7f5110197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:47.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f510ed9d640 1 --2- 172.21.15.169:0/3671801509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5110102ca0 0x7f511019eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:47.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f510e59c640 1 -- 172.21.15.169:0/3671801509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51101000d0 msgr2=0x7f5110197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:47.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f510e59c640 1 --2- 172.21.15.169:0/3671801509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51101000d0 0x7f5110197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f510e59c640 1 -- 172.21.15.169:0/3671801509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5110102ca0 msgr2=0x7f511019eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:47.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f510e59c640 1 --2- 172.21.15.169:0/3671801509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5110102ca0 0x7f511019eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:47.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f510e59c640 1 -- 172.21.15.169:0/3671801509 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511019f540 con 0x7f51100ff710 2024-03-28T00:56:47.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f510ed9d640 1 --2- 172.21.15.169:0/3671801509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5110102ca0 0x7f511019eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:47.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f510e59c640 1 --2- 172.21.15.169:0/3671801509 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51100ff710 0x7f5110197920 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f510000e9c0 tx=0x7f510000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:47.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f50ff7fe640 1 -- 172.21.15.169:0/3671801509 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510000cd80 con 0x7f51100ff710 2024-03-28T00:56:47.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f50ff7fe640 1 -- 172.21.15.169:0/3671801509 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51000035f0 con 0x7f51100ff710 2024-03-28T00:56:47.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f5114e9a640 1 -- 172.21.15.169:0/3671801509 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f511019f830 con 0x7f51100ff710 2024-03-28T00:56:47.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.861+0000 7f50ff7fe640 1 -- 172.21.15.169:0/3671801509 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5100010690 con 0x7f51100ff710 2024-03-28T00:56:47.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.862+0000 7f5114e9a640 1 -- 172.21.15.169:0/3671801509 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f511019fd30 con 0x7f51100ff710 2024-03-28T00:56:47.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.863+0000 7f5114e9a640 1 -- 172.21.15.169:0/3671801509 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5110108b60 con 0x7f51100ff710 2024-03-28T00:56:47.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.863+0000 7f50ff7fe640 1 -- 172.21.15.169:0/3671801509 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f51000037a0 con 0x7f51100ff710 2024-03-28T00:56:47.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.864+0000 7f50ff7fe640 1 --2- 172.21.15.169:0/3671801509 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f50f80a2800 0x7f50f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:47.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.864+0000 7f510dd9b640 1 --2- 172.21.15.169:0/3671801509 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f50f80a2800 0x7f50f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:47.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.864+0000 7f50ff7fe640 1 -- 172.21.15.169:0/3671801509 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f51000cf160 con 0x7f51100ff710 2024-03-28T00:56:47.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.864+0000 7f510dd9b640 1 --2- 172.21.15.169:0/3671801509 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f50f80a2800 0x7f50f80a4cc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5110198e30 tx=0x7f5104009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:47.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:47.868+0000 7f50ff7fe640 1 -- 172.21.15.169:0/3671801509 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f510009be60 con 0x7f51100ff710 2024-03-28T00:56:48.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.042+0000 7f5114e9a640 1 -- 172.21.15.169:0/3671801509 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f51100622a0 con 0x7f51100ff710 2024-03-28T00:56:48.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.042+0000 7f50ff7fe640 1 -- 172.21.15.169:0/3671801509 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) v1 ==== 104+0+11 (secure 0 0 0) 0x7f510009b760 con 0x7f51100ff710 2024-03-28T00:56:48.045 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T00:56:48.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.044+0000 7f5114e9a640 1 -- 172.21.15.169:0/3671801509 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f50f80a2800 msgr2=0x7f50f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:48.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.044+0000 7f5114e9a640 1 --2- 172.21.15.169:0/3671801509 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f50f80a2800 0x7f50f80a4cc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5110198e30 tx=0x7f5104009260 comp rx=0 tx=0).stop 2024-03-28T00:56:48.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.044+0000 7f5114e9a640 1 -- 172.21.15.169:0/3671801509 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51100ff710 msgr2=0x7f5110197920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:48.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.044+0000 7f5114e9a640 1 --2- 172.21.15.169:0/3671801509 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51100ff710 0x7f5110197920 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f510000e9c0 tx=0x7f510000ee80 comp rx=0 tx=0).stop 2024-03-28T00:56:48.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.044+0000 7f5114e9a640 1 -- 172.21.15.169:0/3671801509 shutdown_connections 2024-03-28T00:56:48.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.044+0000 7f5114e9a640 1 --2- 172.21.15.169:0/3671801509 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f50f80a2800 0x7f50f80a4cc0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.044+0000 7f5114e9a640 1 --2- 172.21.15.169:0/3671801509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5110102ca0 0x7f511019eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.044+0000 7f5114e9a640 1 --2- 172.21.15.169:0/3671801509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51101000d0 0x7f5110197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.044+0000 7f5114e9a640 1 --2- 172.21.15.169:0/3671801509 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51100ff710 0x7f5110197920 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.044+0000 7f5114e9a640 1 -- 172.21.15.169:0/3671801509 >> 172.21.15.169:0/3671801509 conn(0x7f51100fb2d0 msgr2=0x7f5110105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:48.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.044+0000 7f5114e9a640 1 -- 172.21.15.169:0/3671801509 shutdown_connections 2024-03-28T00:56:48.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.044+0000 7f5114e9a640 1 -- 172.21.15.169:0/3671801509 wait complete. 2024-03-28T00:56:48.058 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T00:56:48.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.161+0000 7f02ff9a9640 1 -- 172.21.15.169:0/2946131661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f02f8102160 msgr2=0x7f02f81089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:48.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.161+0000 7f02ff9a9640 1 --2- 172.21.15.169:0/2946131661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f02f8102160 0x7f02f81089f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009ca0 tx=0x7f02f402c0f0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.161+0000 7f02ff9a9640 1 -- 172.21.15.169:0/2946131661 shutdown_connections 2024-03-28T00:56:48.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.161+0000 7f02ff9a9640 1 --2- 172.21.15.169:0/2946131661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f02f8102160 0x7f02f81089f0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.161+0000 7f02ff9a9640 1 --2- 172.21.15.169:0/2946131661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f81017a0 0x7f02f8101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.161+0000 7f02ff9a9640 1 --2- 172.21.15.169:0/2946131661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02f81005b0 0x7f02f81009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.161+0000 7f02ff9a9640 1 -- 172.21.15.169:0/2946131661 >> 172.21.15.169:0/2946131661 conn(0x7f02f80fbb40 msgr2=0x7f02f80fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:48.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.161+0000 7f02ff9a9640 1 -- 172.21.15.169:0/2946131661 shutdown_connections 2024-03-28T00:56:48.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.161+0000 7f02ff9a9640 1 -- 172.21.15.169:0/2946131661 wait complete. 2024-03-28T00:56:48.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.162+0000 7f02ff9a9640 1 Processor -- start 2024-03-28T00:56:48.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.162+0000 7f02ff9a9640 1 -- start start 2024-03-28T00:56:48.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.162+0000 7f02ff9a9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02f81005b0 0x7f02f81968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:48.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.162+0000 7f02ff9a9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f81017a0 0x7f02f8196de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:48.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.162+0000 7f02ff9a9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f02f8102160 0x7f02f819de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:48.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.162+0000 7f02ff9a9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f810c230 con 0x7f02f8102160 2024-03-28T00:56:48.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.162+0000 7f02ff9a9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f810c0b0 con 0x7f02f81005b0 2024-03-28T00:56:48.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.162+0000 7f02ff9a9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f810c3b0 con 0x7f02f81017a0 2024-03-28T00:56:48.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.162+0000 7f02fdf46640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f02f8102160 0x7f02f819de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:48.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.162+0000 7f02fdf46640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f02f8102160 0x7f02f819de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34728/0 (socket says 172.21.15.169:34728) 2024-03-28T00:56:48.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.162+0000 7f02fdf46640 1 -- 172.21.15.169:0/2626492422 learned_addr learned my addr 172.21.15.169:0/2626492422 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:48.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.162+0000 7f02fd745640 1 --2- 172.21.15.169:0/2626492422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02f81005b0 0x7f02f81968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:48.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.162+0000 7f02fcf44640 1 --2- 172.21.15.169:0/2626492422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f81017a0 0x7f02f8196de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:48.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.162+0000 7f02fdf46640 1 -- 172.21.15.169:0/2626492422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f81017a0 msgr2=0x7f02f8196de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:48.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.162+0000 7f02fdf46640 1 --2- 172.21.15.169:0/2626492422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f81017a0 0x7f02f8196de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.162+0000 7f02fdf46640 1 -- 172.21.15.169:0/2626492422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02f81005b0 msgr2=0x7f02f81968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:48.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.162+0000 7f02fdf46640 1 --2- 172.21.15.169:0/2626492422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02f81005b0 0x7f02f81968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.162+0000 7f02fdf46640 1 -- 172.21.15.169:0/2626492422 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f819e4c0 con 0x7f02f8102160 2024-03-28T00:56:48.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.162+0000 7f02fd745640 1 --2- 172.21.15.169:0/2626492422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02f81005b0 0x7f02f81968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:48.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.163+0000 7f02fdf46640 1 --2- 172.21.15.169:0/2626492422 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f02f8102160 0x7f02f819de50 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f02f4037040 tx=0x7f02f4032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:48.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.163+0000 7f02ee7fc640 1 -- 172.21.15.169:0/2626492422 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f4041020 con 0x7f02f8102160 2024-03-28T00:56:48.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.163+0000 7f02ff9a9640 1 -- 172.21.15.169:0/2626492422 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f819e750 con 0x7f02f8102160 2024-03-28T00:56:48.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.163+0000 7f02ee7fc640 1 -- 172.21.15.169:0/2626492422 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02f4034cd0 con 0x7f02f8102160 2024-03-28T00:56:48.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.163+0000 7f02ee7fc640 1 -- 172.21.15.169:0/2626492422 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f4039040 con 0x7f02f8102160 2024-03-28T00:56:48.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.163+0000 7f02ff9a9640 1 -- 172.21.15.169:0/2626492422 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f819ec90 con 0x7f02f8102160 2024-03-28T00:56:48.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.164+0000 7f02ff9a9640 1 -- 172.21.15.169:0/2626492422 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f81078d0 con 0x7f02f8102160 2024-03-28T00:56:48.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.165+0000 7f02ee7fc640 1 -- 172.21.15.169:0/2626492422 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f02f4044020 con 0x7f02f8102160 2024-03-28T00:56:48.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.165+0000 7f02ee7fc640 1 --2- 172.21.15.169:0/2626492422 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f02e40a2800 0x7f02e40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:48.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.165+0000 7f02fd745640 1 --2- 172.21.15.169:0/2626492422 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f02e40a2800 0x7f02e40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:48.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.165+0000 7f02ee7fc640 1 -- 172.21.15.169:0/2626492422 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f02f40f1100 con 0x7f02f8102160 2024-03-28T00:56:48.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.166+0000 7f02fd745640 1 --2- 172.21.15.169:0/2626492422 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f02e40a2800 0x7f02e40a4cc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f02f8101600 tx=0x7f02f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:48.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.169+0000 7f02ee7fc640 1 -- 172.21.15.169:0/2626492422 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f02f404b020 con 0x7f02f8102160 2024-03-28T00:56:48.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.196+0000 7f02ee7fc640 1 -- 172.21.15.169:0/2626492422 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f02f40b9e00 con 0x7f02f8102160 2024-03-28T00:56:48.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.354+0000 7f02ff9a9640 1 -- 172.21.15.169:0/2626492422 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f02f802d070 con 0x7f02f8102160 2024-03-28T00:56:48.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.354+0000 7f02ee7fc640 1 -- 172.21.15.169:0/2626492422 <== mon.0 v2:172.21.15.169:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) v1 ==== 100+0+11 (secure 0 0 0) 0x7f02f40bde90 con 0x7f02f8102160 2024-03-28T00:56:48.357 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 64 2024-03-28T00:56:48.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.356+0000 7f02ff9a9640 1 -- 172.21.15.169:0/2626492422 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f02e40a2800 msgr2=0x7f02e40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:48.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.356+0000 7f02ff9a9640 1 --2- 172.21.15.169:0/2626492422 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f02e40a2800 0x7f02e40a4cc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f02f8101600 tx=0x7f02f0005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:48.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.356+0000 7f02ff9a9640 1 -- 172.21.15.169:0/2626492422 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f02f8102160 msgr2=0x7f02f819de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:48.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.356+0000 7f02ff9a9640 1 --2- 172.21.15.169:0/2626492422 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f02f8102160 0x7f02f819de50 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f02f4037040 tx=0x7f02f4032f40 comp rx=0 tx=0).stop 2024-03-28T00:56:48.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.356+0000 7f02ff9a9640 1 -- 172.21.15.169:0/2626492422 shutdown_connections 2024-03-28T00:56:48.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.356+0000 7f02ff9a9640 1 --2- 172.21.15.169:0/2626492422 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f02e40a2800 0x7f02e40a4cc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.356+0000 7f02ff9a9640 1 --2- 172.21.15.169:0/2626492422 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f02f8102160 0x7f02f819de50 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.356+0000 7f02ff9a9640 1 --2- 172.21.15.169:0/2626492422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f81017a0 0x7f02f8196de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.356+0000 7f02ff9a9640 1 --2- 172.21.15.169:0/2626492422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02f81005b0 0x7f02f81968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.356+0000 7f02ff9a9640 1 -- 172.21.15.169:0/2626492422 >> 172.21.15.169:0/2626492422 conn(0x7f02f80fbb40 msgr2=0x7f02f8104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:48.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.356+0000 7f02ff9a9640 1 -- 172.21.15.169:0/2626492422 shutdown_connections 2024-03-28T00:56:48.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.356+0000 7f02ff9a9640 1 -- 172.21.15.169:0/2626492422 wait complete. 2024-03-28T00:56:48.369 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:48 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1261889938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:48.369 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:48 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2264489908' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:56:48.369 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:48 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3671801509' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:56:48.371 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T00:56:48.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.473+0000 7fc0bc88c640 1 -- 172.21.15.169:0/4186413203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0b8102120 msgr2=0x7fc0b8104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:48.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.473+0000 7fc0bc88c640 1 --2- 172.21.15.169:0/4186413203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0b8102120 0x7fc0b8104510 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8009860 tx=0x7fc0a802bb70 comp rx=0 tx=0).stop 2024-03-28T00:56:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.474+0000 7fc0bc88c640 1 -- 172.21.15.169:0/4186413203 shutdown_connections 2024-03-28T00:56:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.474+0000 7fc0bc88c640 1 --2- 172.21.15.169:0/4186413203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0b8107380 0x7fc0b81097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.474+0000 7fc0bc88c640 1 --2- 172.21.15.169:0/4186413203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0b8104a50 0x7fc0b8106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.474+0000 7fc0bc88c640 1 --2- 172.21.15.169:0/4186413203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0b8102120 0x7fc0b8104510 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.474+0000 7fc0bc88c640 1 -- 172.21.15.169:0/4186413203 >> 172.21.15.169:0/4186413203 conn(0x7fc0b80fbb30 msgr2=0x7fc0b80fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.474+0000 7fc0bc88c640 1 -- 172.21.15.169:0/4186413203 shutdown_connections 2024-03-28T00:56:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.474+0000 7fc0bc88c640 1 -- 172.21.15.169:0/4186413203 wait complete. 2024-03-28T00:56:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.474+0000 7fc0bc88c640 1 Processor -- start 2024-03-28T00:56:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.474+0000 7fc0bc88c640 1 -- start start 2024-03-28T00:56:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.474+0000 7fc0bc88c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0b8102120 0x7fc0b8197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.474+0000 7fc0bc88c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0b8104a50 0x7fc0b8197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.474+0000 7fc0bc88c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0b8107380 0x7fc0b819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.474+0000 7fc0bc88c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b810d330 con 0x7fc0b8104a50 2024-03-28T00:56:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.474+0000 7fc0bc88c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b810d1b0 con 0x7fc0b8107380 2024-03-28T00:56:48.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.474+0000 7fc0bc88c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b810d4b0 con 0x7fc0b8102120 2024-03-28T00:56:48.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.475+0000 7fc0b5d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0b8104a50 0x7fc0b8197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:48.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.475+0000 7fc0b659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0b8102120 0x7fc0b8197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:48.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.475+0000 7fc0b5d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0b8104a50 0x7fc0b8197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34742/0 (socket says 172.21.15.169:34742) 2024-03-28T00:56:48.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.475+0000 7fc0b5d9b640 1 -- 172.21.15.169:0/1777128797 learned_addr learned my addr 172.21.15.169:0/1777128797 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:48.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.475+0000 7fc0b6d9d640 1 --2- 172.21.15.169:0/1777128797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0b8107380 0x7fc0b819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:48.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.475+0000 7fc0b5d9b640 1 -- 172.21.15.169:0/1777128797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0b8102120 msgr2=0x7fc0b8197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:48.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.475+0000 7fc0b5d9b640 1 --2- 172.21.15.169:0/1777128797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0b8102120 0x7fc0b8197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.475+0000 7fc0b5d9b640 1 -- 172.21.15.169:0/1777128797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0b8107380 msgr2=0x7fc0b819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:48.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.475+0000 7fc0b5d9b640 1 --2- 172.21.15.169:0/1777128797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0b8107380 0x7fc0b819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.475+0000 7fc0b5d9b640 1 -- 172.21.15.169:0/1777128797 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b819f620 con 0x7fc0b8104a50 2024-03-28T00:56:48.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.475+0000 7fc0b659c640 1 --2- 172.21.15.169:0/1777128797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0b8102120 0x7fc0b8197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:48.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.475+0000 7fc0b5d9b640 1 --2- 172.21.15.169:0/1777128797 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0b8104a50 0x7fc0b8197f40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac00e9f0 tx=0x7fc0ac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:48.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.475+0000 7fc0a77fe640 1 -- 172.21.15.169:0/1777128797 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac00ccc0 con 0x7fc0b8104a50 2024-03-28T00:56:48.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.475+0000 7fc0a77fe640 1 -- 172.21.15.169:0/1777128797 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0ac005500 con 0x7fc0b8104a50 2024-03-28T00:56:48.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.475+0000 7fc0bc88c640 1 -- 172.21.15.169:0/1777128797 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b819f910 con 0x7fc0b8104a50 2024-03-28T00:56:48.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.475+0000 7fc0a77fe640 1 -- 172.21.15.169:0/1777128797 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac010690 con 0x7fc0b8104a50 2024-03-28T00:56:48.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.475+0000 7fc0bc88c640 1 -- 172.21.15.169:0/1777128797 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b8100130 con 0x7fc0b8104a50 2024-03-28T00:56:48.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.476+0000 7fc0bc88c640 1 -- 172.21.15.169:0/1777128797 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b80fd620 con 0x7fc0b8104a50 2024-03-28T00:56:48.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.477+0000 7fc0a77fe640 1 -- 172.21.15.169:0/1777128797 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fc0ac0046c0 con 0x7fc0b8104a50 2024-03-28T00:56:48.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.478+0000 7fc0a77fe640 1 --2- 172.21.15.169:0/1777128797 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc0a00a2890 0x7fc0a00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:48.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.478+0000 7fc0b659c640 1 --2- 172.21.15.169:0/1777128797 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc0a00a2890 0x7fc0a00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:48.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.478+0000 7fc0a77fe640 1 -- 172.21.15.169:0/1777128797 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc0ac0cf2d0 con 0x7fc0b8104a50 2024-03-28T00:56:48.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.478+0000 7fc0b659c640 1 --2- 172.21.15.169:0/1777128797 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc0a00a2890 0x7fc0a00a4d50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc0a802c180 tx=0x7fc0a802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:48.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.482+0000 7fc0a77fe640 1 -- 172.21.15.169:0/1777128797 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc0ac09be70 con 0x7fc0b8104a50 2024-03-28T00:56:48.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:48 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1261889938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:48.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:48 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2264489908' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:56:48.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:48 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3671801509' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:56:48.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.656+0000 7fc0bc88c640 1 -- 172.21.15.169:0/1777128797 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc0b802d270 con 0x7fc0b8104a50 2024-03-28T00:56:48.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.657+0000 7fc0a77fe640 1 -- 172.21.15.169:0/1777128797 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc0ac09b770 con 0x7fc0b8104a50 2024-03-28T00:56:48.660 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T00:56:48.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.659+0000 7fc0bc88c640 1 -- 172.21.15.169:0/1777128797 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc0a00a2890 msgr2=0x7fc0a00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:48.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.659+0000 7fc0bc88c640 1 --2- 172.21.15.169:0/1777128797 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc0a00a2890 0x7fc0a00a4d50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc0a802c180 tx=0x7fc0a802e040 comp rx=0 tx=0).stop 2024-03-28T00:56:48.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.659+0000 7fc0bc88c640 1 -- 172.21.15.169:0/1777128797 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0b8104a50 msgr2=0x7fc0b8197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:48.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.659+0000 7fc0bc88c640 1 --2- 172.21.15.169:0/1777128797 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0b8104a50 0x7fc0b8197f40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac00e9f0 tx=0x7fc0ac00eeb0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.659+0000 7fc0bc88c640 1 -- 172.21.15.169:0/1777128797 shutdown_connections 2024-03-28T00:56:48.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.659+0000 7fc0bc88c640 1 --2- 172.21.15.169:0/1777128797 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc0a00a2890 0x7fc0a00a4d50 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.659+0000 7fc0bc88c640 1 --2- 172.21.15.169:0/1777128797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0b8107380 0x7fc0b819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.659+0000 7fc0bc88c640 1 --2- 172.21.15.169:0/1777128797 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0b8104a50 0x7fc0b8197f40 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.659+0000 7fc0bc88c640 1 --2- 172.21.15.169:0/1777128797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0b8102120 0x7fc0b8197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.659+0000 7fc0bc88c640 1 -- 172.21.15.169:0/1777128797 >> 172.21.15.169:0/1777128797 conn(0x7fc0b80fbb30 msgr2=0x7fc0b8102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:48.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.659+0000 7fc0bc88c640 1 -- 172.21.15.169:0/1777128797 shutdown_connections 2024-03-28T00:56:48.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.659+0000 7fc0bc88c640 1 -- 172.21.15.169:0/1777128797 wait complete. 2024-03-28T00:56:48.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:48 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1261889938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:48.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:48 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2264489908' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:56:48.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:48 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3671801509' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:56:48.673 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-28T00:56:48.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:56:48.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.772+0000 7f6180977640 1 -- 172.21.15.169:0/1461862679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f617c102170 msgr2=0x7f617c108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:48.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.772+0000 7f6180977640 1 --2- 172.21.15.169:0/1461862679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f617c102170 0x7f617c108a00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6174009ca0 tx=0x7f617402c0f0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.773+0000 7f6180977640 1 -- 172.21.15.169:0/1461862679 shutdown_connections 2024-03-28T00:56:48.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.773+0000 7f6180977640 1 --2- 172.21.15.169:0/1461862679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f617c102170 0x7f617c108a00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.773+0000 7f6180977640 1 --2- 172.21.15.169:0/1461862679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f617c1017b0 0x7f617c101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.773+0000 7f6180977640 1 --2- 172.21.15.169:0/1461862679 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f617c1005c0 0x7f617c1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.773+0000 7f6180977640 1 -- 172.21.15.169:0/1461862679 >> 172.21.15.169:0/1461862679 conn(0x7f617c0fbb30 msgr2=0x7f617c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:48.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.773+0000 7f6180977640 1 -- 172.21.15.169:0/1461862679 shutdown_connections 2024-03-28T00:56:48.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.773+0000 7f6180977640 1 -- 172.21.15.169:0/1461862679 wait complete. 2024-03-28T00:56:48.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.773+0000 7f6180977640 1 Processor -- start 2024-03-28T00:56:48.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.773+0000 7f6180977640 1 -- start start 2024-03-28T00:56:48.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.773+0000 7f6180977640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f617c1005c0 0x7f617c069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:48.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.773+0000 7f6180977640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f617c1017b0 0x7f617c19c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:48.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.773+0000 7f6180977640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f617c102170 0x7f617c19ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:48.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.773+0000 7f6180977640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c10c220 con 0x7f617c1005c0 2024-03-28T00:56:48.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.773+0000 7f6180977640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c10c0a0 con 0x7f617c102170 2024-03-28T00:56:48.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.773+0000 7f6180977640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c10c3a0 con 0x7f617c1017b0 2024-03-28T00:56:48.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.774+0000 7f617a59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f617c1005c0 0x7f617c069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:48.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.774+0000 7f617a59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f617c1005c0 0x7f617c069370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34766/0 (socket says 172.21.15.169:34766) 2024-03-28T00:56:48.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.774+0000 7f617a59c640 1 -- 172.21.15.169:0/556264606 learned_addr learned my addr 172.21.15.169:0/556264606 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:48.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.774+0000 7f6173fff640 1 --2- 172.21.15.169:0/556264606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f617c1017b0 0x7f617c19c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:48.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.774+0000 7f617ad9d640 1 --2- 172.21.15.169:0/556264606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f617c102170 0x7f617c19ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:48.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.774+0000 7f617a59c640 1 -- 172.21.15.169:0/556264606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f617c1017b0 msgr2=0x7f617c19c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:48.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.774+0000 7f617a59c640 1 --2- 172.21.15.169:0/556264606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f617c1017b0 0x7f617c19c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.774+0000 7f617a59c640 1 -- 172.21.15.169:0/556264606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f617c102170 msgr2=0x7f617c19ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:48.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.774+0000 7f617a59c640 1 --2- 172.21.15.169:0/556264606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f617c102170 0x7f617c19ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:48.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.774+0000 7f617a59c640 1 -- 172.21.15.169:0/556264606 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617c19f5c0 con 0x7f617c1005c0 2024-03-28T00:56:48.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.774+0000 7f617a59c640 1 --2- 172.21.15.169:0/556264606 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f617c1005c0 0x7f617c069370 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f616c00e9c0 tx=0x7f616c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:48.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.774+0000 7f61737fe640 1 -- 172.21.15.169:0/556264606 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c00cd80 con 0x7f617c1005c0 2024-03-28T00:56:48.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.774+0000 7f6180977640 1 -- 172.21.15.169:0/556264606 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f617c19f850 con 0x7f617c1005c0 2024-03-28T00:56:48.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.774+0000 7f61737fe640 1 -- 172.21.15.169:0/556264606 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f616c0035f0 con 0x7f617c1005c0 2024-03-28T00:56:48.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.774+0000 7f61737fe640 1 -- 172.21.15.169:0/556264606 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c010690 con 0x7f617c1005c0 2024-03-28T00:56:48.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.774+0000 7f6180977640 1 -- 172.21.15.169:0/556264606 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f617c19fdf0 con 0x7f617c1005c0 2024-03-28T00:56:48.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.775+0000 7f6180977640 1 -- 172.21.15.169:0/556264606 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f617c068490 con 0x7f617c1005c0 2024-03-28T00:56:48.782 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.776+0000 7f61737fe640 1 -- 172.21.15.169:0/556264606 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f616c0037a0 con 0x7f617c1005c0 2024-03-28T00:56:48.782 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.777+0000 7f61737fe640 1 --2- 172.21.15.169:0/556264606 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f61680a2890 0x7f61680a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:48.782 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.777+0000 7f6173fff640 1 --2- 172.21.15.169:0/556264606 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f61680a2890 0x7f61680a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:48.782 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.777+0000 7f61737fe640 1 -- 172.21.15.169:0/556264606 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f616c0cf320 con 0x7f617c1005c0 2024-03-28T00:56:48.782 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.777+0000 7f6173fff640 1 --2- 172.21.15.169:0/556264606 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f61680a2890 0x7f61680a4d50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f61640095f0 tx=0x7f6164005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:48.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.782+0000 7f61737fe640 1 -- 172.21.15.169:0/556264606 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f616c09bec0 con 0x7f617c1005c0 2024-03-28T00:56:48.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.997+0000 7f6180977640 1 -- 172.21.15.169:0/556264606 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f617c1078e0 con 0x7f617c1005c0 2024-03-28T00:56:49.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.997+0000 7f61737fe640 1 -- 172.21.15.169:0/556264606 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7f617c1078e0 con 0x7f617c1005c0 2024-03-28T00:56:49.000 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:49.000 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T00:56:49.001 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T00:56:49.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.999+0000 7f6180977640 1 -- 172.21.15.169:0/556264606 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f61680a2890 msgr2=0x7f61680a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:49.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.999+0000 7f6180977640 1 --2- 172.21.15.169:0/556264606 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f61680a2890 0x7f61680a4d50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f61640095f0 tx=0x7f6164005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:49.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.999+0000 7f6180977640 1 -- 172.21.15.169:0/556264606 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f617c1005c0 msgr2=0x7f617c069370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:49.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.999+0000 7f6180977640 1 --2- 172.21.15.169:0/556264606 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f617c1005c0 0x7f617c069370 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f616c00e9c0 tx=0x7f616c00ee80 comp rx=0 tx=0).stop 2024-03-28T00:56:49.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.999+0000 7f6180977640 1 -- 172.21.15.169:0/556264606 shutdown_connections 2024-03-28T00:56:49.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.999+0000 7f6180977640 1 --2- 172.21.15.169:0/556264606 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f61680a2890 0x7f61680a4d50 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.999+0000 7f6180977640 1 --2- 172.21.15.169:0/556264606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f617c102170 0x7f617c19ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.999+0000 7f6180977640 1 --2- 172.21.15.169:0/556264606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f617c1017b0 0x7f617c19c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.999+0000 7f6180977640 1 --2- 172.21.15.169:0/556264606 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f617c1005c0 0x7f617c069370 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:48.999+0000 7f6180977640 1 -- 172.21.15.169:0/556264606 >> 172.21.15.169:0/556264606 conn(0x7f617c0fbb30 msgr2=0x7f617c104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:49.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.000+0000 7f6180977640 1 -- 172.21.15.169:0/556264606 shutdown_connections 2024-03-28T00:56:49.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.000+0000 7f6180977640 1 -- 172.21.15.169:0/556264606 wait complete. 2024-03-28T00:56:49.014 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-28T00:56:49.014 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-28T00:56:49.014 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T00:56:49.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.115+0000 7f0529634640 1 -- 172.21.15.169:0/2626636812 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05241028b0 msgr2=0x7f0524102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:49.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.115+0000 7f0529634640 1 --2- 172.21.15.169:0/2626636812 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05241028b0 0x7f0524102d30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0518009ca0 tx=0x7f051802c0f0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.115+0000 7f0529634640 1 -- 172.21.15.169:0/2626636812 shutdown_connections 2024-03-28T00:56:49.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.115+0000 7f0529634640 1 --2- 172.21.15.169:0/2626636812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0524103270 0x7f0524109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.115+0000 7f0529634640 1 --2- 172.21.15.169:0/2626636812 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05241028b0 0x7f0524102d30 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.115+0000 7f0529634640 1 --2- 172.21.15.169:0/2626636812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05241016c0 0x7f0524101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.115+0000 7f0529634640 1 -- 172.21.15.169:0/2626636812 >> 172.21.15.169:0/2626636812 conn(0x7f05240fcc50 msgr2=0x7f05240ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:49.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.115+0000 7f0529634640 1 -- 172.21.15.169:0/2626636812 shutdown_connections 2024-03-28T00:56:49.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.115+0000 7f0529634640 1 -- 172.21.15.169:0/2626636812 wait complete. 2024-03-28T00:56:49.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.115+0000 7f0529634640 1 Processor -- start 2024-03-28T00:56:49.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.115+0000 7f0529634640 1 -- start start 2024-03-28T00:56:49.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f0529634640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05241016c0 0x7f0524075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:49.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f0529634640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05241028b0 0x7f0524073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:49.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f0529634640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0524103270 0x7f0524073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:49.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f0529634640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052410d320 con 0x7f05241028b0 2024-03-28T00:56:49.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f0529634640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052410d1a0 con 0x7f05241016c0 2024-03-28T00:56:49.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f0529634640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052410d4a0 con 0x7f0524103270 2024-03-28T00:56:49.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f05227fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05241028b0 0x7f0524073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:49.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f05227fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05241028b0 0x7f0524073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34794/0 (socket says 172.21.15.169:34794) 2024-03-28T00:56:49.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f05227fc640 1 -- 172.21.15.169:0/711585177 learned_addr learned my addr 172.21.15.169:0/711585177 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:49.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f0522ffd640 1 --2- 172.21.15.169:0/711585177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05241016c0 0x7f0524075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:49.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f05237fe640 1 --2- 172.21.15.169:0/711585177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0524103270 0x7f0524073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:49.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f05227fc640 1 -- 172.21.15.169:0/711585177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0524103270 msgr2=0x7f0524073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:49.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f05227fc640 1 --2- 172.21.15.169:0/711585177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0524103270 0x7f0524073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f05227fc640 1 -- 172.21.15.169:0/711585177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05241016c0 msgr2=0x7f0524075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:49.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f05227fc640 1 --2- 172.21.15.169:0/711585177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05241016c0 0x7f0524075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f05227fc640 1 -- 172.21.15.169:0/711585177 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0524074860 con 0x7f05241028b0 2024-03-28T00:56:49.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f05227fc640 1 --2- 172.21.15.169:0/711585177 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05241028b0 0x7f0524073a90 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0518009dc0 tx=0x7f0518032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:49.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f050bfff640 1 -- 172.21.15.169:0/711585177 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0518041020 con 0x7f05241028b0 2024-03-28T00:56:49.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f0529634640 1 -- 172.21.15.169:0/711585177 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05241a1a80 con 0x7f05241028b0 2024-03-28T00:56:49.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f050bfff640 1 -- 172.21.15.169:0/711585177 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0518034cd0 con 0x7f05241028b0 2024-03-28T00:56:49.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f050bfff640 1 -- 172.21.15.169:0/711585177 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0518039040 con 0x7f05241028b0 2024-03-28T00:56:49.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.116+0000 7f0529634640 1 -- 172.21.15.169:0/711585177 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05241a2020 con 0x7f05241028b0 2024-03-28T00:56:49.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.118+0000 7f0529634640 1 -- 172.21.15.169:0/711585177 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05241089e0 con 0x7f05241028b0 2024-03-28T00:56:49.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.119+0000 7f050bfff640 1 -- 172.21.15.169:0/711585177 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f0518044020 con 0x7f05241028b0 2024-03-28T00:56:49.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.119+0000 7f050bfff640 1 --2- 172.21.15.169:0/711585177 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f050c0a2890 0x7f050c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:49.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.119+0000 7f0522ffd640 1 --2- 172.21.15.169:0/711585177 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f050c0a2890 0x7f050c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:49.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.119+0000 7f050bfff640 1 -- 172.21.15.169:0/711585177 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f05180f0c90 con 0x7f05241028b0 2024-03-28T00:56:49.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.120+0000 7f0522ffd640 1 --2- 172.21.15.169:0/711585177 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f050c0a2890 0x7f050c0a4d50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0524102710 tx=0x7f051c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:49.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.123+0000 7f050bfff640 1 -- 172.21.15.169:0/711585177 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f05180f6020 con 0x7f05241028b0 2024-03-28T00:56:49.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:49 smithi169 ceph-mon[32274]: pgmap v360: 129 pgs: 129 active+clean; 653 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-28T00:56:49.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:49 smithi169 ceph-mon[32274]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-28T00:56:49.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:49 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2626492422' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:56:49.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:49 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1777128797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:56:49.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:49 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/556264606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:56:49.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.299+0000 7f0529634640 1 -- 172.21.15.169:0/711585177 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0524061ae0 con 0x7f050c0a2890 2024-03-28T00:56:49.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.309+0000 7f050bfff640 1 -- 172.21.15.169:0/711585177 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340752 (secure 0 0 0) 0x7f0524061ae0 con 0x7f050c0a2890 2024-03-28T00:56:49.316 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:49.318 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-28T00:56:49.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.313+0000 7f0529634640 1 -- 172.21.15.169:0/711585177 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f050c0a2890 msgr2=0x7f050c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:49.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.313+0000 7f0529634640 1 --2- 172.21.15.169:0/711585177 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f050c0a2890 0x7f050c0a4d50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0524102710 tx=0x7f051c005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:49.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.313+0000 7f0529634640 1 -- 172.21.15.169:0/711585177 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05241028b0 msgr2=0x7f0524073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:49.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.313+0000 7f0529634640 1 --2- 172.21.15.169:0/711585177 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05241028b0 0x7f0524073a90 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0518009dc0 tx=0x7f0518032f20 comp rx=0 tx=0).stop 2024-03-28T00:56:49.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.313+0000 7f0529634640 1 -- 172.21.15.169:0/711585177 shutdown_connections 2024-03-28T00:56:49.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.313+0000 7f0529634640 1 --2- 172.21.15.169:0/711585177 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f050c0a2890 0x7f050c0a4d50 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.313+0000 7f0529634640 1 --2- 172.21.15.169:0/711585177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0524103270 0x7f0524073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.313+0000 7f0529634640 1 --2- 172.21.15.169:0/711585177 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05241028b0 0x7f0524073a90 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.313+0000 7f0529634640 1 --2- 172.21.15.169:0/711585177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05241016c0 0x7f0524075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.313+0000 7f0529634640 1 -- 172.21.15.169:0/711585177 >> 172.21.15.169:0/711585177 conn(0x7f05240fcc50 msgr2=0x7f0524105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:49.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.313+0000 7f0529634640 1 -- 172.21.15.169:0/711585177 shutdown_connections 2024-03-28T00:56:49.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.313+0000 7f0529634640 1 -- 172.21.15.169:0/711585177 wait complete. 2024-03-28T00:56:49.327 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-03-28T00:56:48.116387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":668508,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":842,"num_read_kb":548,"num_write":358,"num_write_kb":3074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":792,"ondisk_log_size":792,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357676,"kb_used_data":12616,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584148,"statfs":{"total":1151940427776,"available":1151574167552,"internally_reserved":0,"allocated":12918784,"data_stored":8322532,"data_compressed":462210,"data_compressed_allocated":626688,"data_compressed_original":2752512,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6771,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":63,"num_read_kb":38,"num_write":8,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002486"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308755+0000","last_change":"2024-03-28T00:54:32.809951+0000","last_active":"2024-03-28T00:56:38.308755+0000","last_peered":"2024-03-28T00:56:38.308755+0000","last_clean":"2024-03-28T00:56:38.308755+0000","last_became_active":"2024-03-28T00:54:32.808871+0000","last_became_peered":"2024-03-28T00:54:32.808871+0000","last_unstale":"2024-03-28T00:56:38.308755+0000","last_undegraded":"2024-03-28T00:56:38.308755+0000","last_fullsized":"2024-03-28T00:56:38.308755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:15:17.152487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303536+0000","last_change":"2024-03-28T00:54:32.816707+0000","last_active":"2024-03-28T00:56:38.303536+0000","last_peered":"2024-03-28T00:56:38.303536+0000","last_clean":"2024-03-28T00:56:38.303536+0000","last_became_active":"2024-03-28T00:54:32.816000+0000","last_became_peered":"2024-03-28T00:54:32.816000+0000","last_unstale":"2024-03-28T00:56:38.303536+0000","last_undegraded":"2024-03-28T00:56:38.303536+0000","last_fullsized":"2024-03-28T00:56:38.303536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:56:19.532808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303346+0000","last_change":"2024-03-28T00:54:32.817023+0000","last_active":"2024-03-28T00:56:38.303346+0000","last_peered":"2024-03-28T00:56:38.303346+0000","last_clean":"2024-03-28T00:56:38.303346+0000","last_became_active":"2024-03-28T00:54:32.816712+0000","last_became_peered":"2024-03-28T00:54:32.816712+0000","last_unstale":"2024-03-28T00:56:38.303346+0000","last_undegraded":"2024-03-28T00:56:38.303346+0000","last_fullsized":"2024-03-28T00:56:38.303346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:13:25.334294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285135+0000","last_change":"2024-03-28T00:54:32.807259+0000","last_active":"2024-03-28T00:56:38.285135+0000","last_peered":"2024-03-28T00:56:38.285135+0000","last_clean":"2024-03-28T00:56:38.285135+0000","last_became_active":"2024-03-28T00:54:32.806944+0000","last_became_peered":"2024-03-28T00:54:32.806944+0000","last_unstale":"2024-03-28T00:56:38.285135+0000","last_undegraded":"2024-03-28T00:56:38.285135+0000","last_fullsized":"2024-03-28T00:56:38.285135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:46:04.852351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308386+0000","last_change":"2024-03-28T00:54:32.807613+0000","last_active":"2024-03-28T00:56:38.308386+0000","last_peered":"2024-03-28T00:56:38.308386+0000","last_clean":"2024-03-28T00:56:38.308386+0000","last_became_active":"2024-03-28T00:54:32.807280+0000","last_became_peered":"2024-03-28T00:54:32.807280+0000","last_unstale":"2024-03-28T00:56:38.308386+0000","last_undegraded":"2024-03-28T00:56:38.308386+0000","last_fullsized":"2024-03-28T00:56:38.308386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:56:37.133675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286708+0000","last_change":"2024-03-28T00:54:32.806804+0000","last_active":"2024-03-28T00:56:38.286708+0000","last_peered":"2024-03-28T00:56:38.286708+0000","last_clean":"2024-03-28T00:56:38.286708+0000","last_became_active":"2024-03-28T00:54:32.805343+0000","last_became_peered":"2024-03-28T00:54:32.805343+0000","last_unstale":"2024-03-28T00:56:38.286708+0000","last_undegraded":"2024-03-28T00:56:38.286708+0000","last_fullsized":"2024-03-28T00:56:38.286708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:31:44.849926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307907+0000","last_change":"2024-03-28T00:55:37.563371+0000","last_active":"2024-03-28T00:56:38.307907+0000","last_peered":"2024-03-28T00:56:38.307907+0000","last_clean":"2024-03-28T00:56:38.307907+0000","last_became_active":"2024-03-28T00:55:37.563000+0000","last_became_peered":"2024-03-28T00:55:37.563000+0000","last_unstale":"2024-03-28T00:56:38.307907+0000","last_undegraded":"2024-03-28T00:56:38.307907+0000","last_fullsized":"2024-03-28T00:56:38.307907+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:56:58.782502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.309667+0000","last_change":"2024-03-28T00:56:38.309328+0000","last_active":"2024-03-28T00:56:38.309667+0000","last_peered":"2024-03-28T00:56:38.309667+0000","last_clean":"2024-03-28T00:56:38.309667+0000","last_became_active":"2024-03-28T00:56:38.308978+0000","last_became_peered":"2024-03-28T00:56:38.308978+0000","last_unstale":"2024-03-28T00:56:38.309667+0000","last_undegraded":"2024-03-28T00:56:38.309667+0000","last_fullsized":"2024-03-28T00:56:38.309667+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:45:52.643890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307485+0000","last_change":"2024-03-28T00:54:32.800667+0000","last_active":"2024-03-28T00:56:38.307485+0000","last_peered":"2024-03-28T00:56:38.307485+0000","last_clean":"2024-03-28T00:56:38.307485+0000","last_became_active":"2024-03-28T00:54:32.800332+0000","last_became_peered":"2024-03-28T00:54:32.800332+0000","last_unstale":"2024-03-28T00:56:38.307485+0000","last_undegraded":"2024-03-28T00:56:38.307485+0000","last_fullsized":"2024-03-28T00:56:38.307485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:41:27.577871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.288488+0000","last_change":"2024-03-28T00:56:38.288488+0000","last_active":"2024-03-28T00:56:38.288488+0000","last_peered":"2024-03-28T00:56:38.288488+0000","last_clean":"2024-03-28T00:56:38.288488+0000","last_became_active":"2024-03-28T00:56:38.288164+0000","last_became_peered":"2024-03-28T00:56:38.288164+0000","last_unstale":"2024-03-28T00:56:38.288488+0000","last_undegraded":"2024-03-28T00:56:38.288488+0000","last_fullsized":"2024-03-28T00:56:38.288488+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:50:33.312420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308456+0000","last_change":"2024-03-28T00:54:32.811543+0000","last_active":"2024-03-28T00:56:38.308456+0000","last_peered":"2024-03-28T00:56:38.308456+0000","last_clean":"2024-03-28T00:56:38.308456+0000","last_became_active":"2024-03-28T00:54:32.811210+0000","last_became_peered":"2024-03-28T00:54:32.811210+0000","last_unstale":"2024-03-28T00:56:38.308456+0000","last_undegraded":"2024-03-28T00:56:38.308456+0000","last_fullsized":"2024-03-28T00:56:38.308456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:47:19.539107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285098+0000","last_change":"2024-03-28T00:54:32.812648+0000","last_active":"2024-03-28T00:56:38.285098+0000","last_peered":"2024-03-28T00:56:38.285098+0000","last_clean":"2024-03-28T00:56:38.285098+0000","last_became_active":"2024-03-28T00:54:32.812340+0000","last_became_peered":"2024-03-28T00:54:32.812340+0000","last_unstale":"2024-03-28T00:56:38.285098+0000","last_undegraded":"2024-03-28T00:56:38.285098+0000","last_fullsized":"2024-03-28T00:56:38.285098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:20:45.449953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350843+0000","last_change":"2024-03-28T00:54:32.801807+0000","last_active":"2024-03-28T00:56:38.350843+0000","last_peered":"2024-03-28T00:56:38.350843+0000","last_clean":"2024-03-28T00:56:38.350843+0000","last_became_active":"2024-03-28T00:54:32.801498+0000","last_became_peered":"2024-03-28T00:54:32.801498+0000","last_unstale":"2024-03-28T00:56:38.350843+0000","last_undegraded":"2024-03-28T00:56:38.350843+0000","last_fullsized":"2024-03-28T00:56:38.350843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:35:35.827856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303105+0000","last_change":"2024-03-28T00:54:32.818033+0000","last_active":"2024-03-28T00:56:38.303105+0000","last_peered":"2024-03-28T00:56:38.303105+0000","last_clean":"2024-03-28T00:56:38.303105+0000","last_became_active":"2024-03-28T00:54:32.817309+0000","last_became_peered":"2024-03-28T00:54:32.817309+0000","last_unstale":"2024-03-28T00:56:38.303105+0000","last_undegraded":"2024-03-28T00:56:38.303105+0000","last_fullsized":"2024-03-28T00:56:38.303105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:39:42.682780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303139+0000","last_change":"2024-03-28T00:54:32.815224+0000","last_active":"2024-03-28T00:56:38.303139+0000","last_peered":"2024-03-28T00:56:38.303139+0000","last_clean":"2024-03-28T00:56:38.303139+0000","last_became_active":"2024-03-28T00:54:32.814936+0000","last_became_peered":"2024-03-28T00:54:32.814936+0000","last_unstale":"2024-03-28T00:56:38.303139+0000","last_undegraded":"2024-03-28T00:56:38.303139+0000","last_fullsized":"2024-03-28T00:56:38.303139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:08:40.200393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.306895+0000","last_change":"2024-03-28T00:54:32.807342+0000","last_active":"2024-03-28T00:56:38.306895+0000","last_peered":"2024-03-28T00:56:38.306895+0000","last_clean":"2024-03-28T00:56:38.306895+0000","last_became_active":"2024-03-28T00:54:32.806978+0000","last_became_peered":"2024-03-28T00:54:32.806978+0000","last_unstale":"2024-03-28T00:56:38.306895+0000","last_undegraded":"2024-03-28T00:56:38.306895+0000","last_fullsized":"2024-03-28T00:56:38.306895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:17:58.599693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308639+0000","last_change":"2024-03-28T00:54:32.810036+0000","last_active":"2024-03-28T00:56:38.308639+0000","last_peered":"2024-03-28T00:56:38.308639+0000","last_clean":"2024-03-28T00:56:38.308639+0000","last_became_active":"2024-03-28T00:54:32.809438+0000","last_became_peered":"2024-03-28T00:54:32.809438+0000","last_unstale":"2024-03-28T00:56:38.308639+0000","last_undegraded":"2024-03-28T00:56:38.308639+0000","last_fullsized":"2024-03-28T00:56:38.308639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:18:50.447474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286485+0000","last_change":"2024-03-28T00:54:32.815398+0000","last_active":"2024-03-28T00:56:38.286485+0000","last_peered":"2024-03-28T00:56:38.286485+0000","last_clean":"2024-03-28T00:56:38.286485+0000","last_became_active":"2024-03-28T00:54:32.815045+0000","last_became_peered":"2024-03-28T00:54:32.815045+0000","last_unstale":"2024-03-28T00:56:38.286485+0000","last_undegraded":"2024-03-28T00:56:38.286485+0000","last_fullsized":"2024-03-28T00:56:38.286485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:40:29.060217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285333+0000","last_change":"2024-03-28T00:54:32.805736+0000","last_active":"2024-03-28T00:56:38.285333+0000","last_peered":"2024-03-28T00:56:38.285333+0000","last_clean":"2024-03-28T00:56:38.285333+0000","last_became_active":"2024-03-28T00:54:32.805435+0000","last_became_peered":"2024-03-28T00:54:32.805435+0000","last_unstale":"2024-03-28T00:56:38.285333+0000","last_undegraded":"2024-03-28T00:56:38.285333+0000","last_fullsized":"2024-03-28T00:56:38.285333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:24:30.893583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285426+0000","last_change":"2024-03-28T00:54:36.824531+0000","last_active":"2024-03-28T00:56:38.285426+0000","last_peered":"2024-03-28T00:56:38.285426+0000","last_clean":"2024-03-28T00:56:38.285426+0000","last_became_active":"2024-03-28T00:54:36.824123+0000","last_became_peered":"2024-03-28T00:54:36.824123+0000","last_unstale":"2024-03-28T00:56:38.285426+0000","last_undegraded":"2024-03-28T00:56:38.285426+0000","last_fullsized":"2024-03-28T00:56:38.285426+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:10:47.503171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286759+0000","last_change":"2024-03-28T00:54:32.256226+0000","last_active":"2024-03-28T00:56:38.286759+0000","last_peered":"2024-03-28T00:56:38.286759+0000","last_clean":"2024-03-28T00:56:38.286759+0000","last_became_active":"2024-03-28T00:54:32.255921+0000","last_became_peered":"2024-03-28T00:54:32.255921+0000","last_unstale":"2024-03-28T00:56:38.286759+0000","last_undegraded":"2024-03-28T00:56:38.286759+0000","last_fullsized":"2024-03-28T00:56:38.286759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:56:04.196311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284800+0000","last_change":"2024-03-28T00:54:32.808248+0000","last_active":"2024-03-28T00:56:38.284800+0000","last_peered":"2024-03-28T00:56:38.284800+0000","last_clean":"2024-03-28T00:56:38.284800+0000","last_became_active":"2024-03-28T00:54:32.807923+0000","last_became_peered":"2024-03-28T00:54:32.807923+0000","last_unstale":"2024-03-28T00:56:38.284800+0000","last_undegraded":"2024-03-28T00:56:38.284800+0000","last_fullsized":"2024-03-28T00:56:38.284800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:59:56.002564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284703+0000","last_change":"2024-03-28T00:54:36.830556+0000","last_active":"2024-03-28T00:56:38.284703+0000","last_peered":"2024-03-28T00:56:38.284703+0000","last_clean":"2024-03-28T00:56:38.284703+0000","last_became_active":"2024-03-28T00:54:36.830151+0000","last_became_peered":"2024-03-28T00:54:36.830151+0000","last_unstale":"2024-03-28T00:56:38.284703+0000","last_undegraded":"2024-03-28T00:56:38.284703+0000","last_fullsized":"2024-03-28T00:56:38.284703+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:19:23.073410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285141+0000","last_change":"2024-03-28T00:54:32.260635+0000","last_active":"2024-03-28T00:56:38.285141+0000","last_peered":"2024-03-28T00:56:38.285141+0000","last_clean":"2024-03-28T00:56:38.285141+0000","last_became_active":"2024-03-28T00:54:32.260084+0000","last_became_peered":"2024-03-28T00:54:32.260084+0000","last_unstale":"2024-03-28T00:56:38.285141+0000","last_undegraded":"2024-03-28T00:56:38.285141+0000","last_fullsized":"2024-03-28T00:56:38.285141+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:06.858519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303455+0000","last_change":"2024-03-28T00:54:32.809901+0000","last_active":"2024-03-28T00:56:38.303455+0000","last_peered":"2024-03-28T00:56:38.303455+0000","last_clean":"2024-03-28T00:56:38.303455+0000","last_became_active":"2024-03-28T00:54:32.809557+0000","last_became_peered":"2024-03-28T00:54:32.809557+0000","last_unstale":"2024-03-28T00:56:38.303455+0000","last_undegraded":"2024-03-28T00:56:38.303455+0000","last_fullsized":"2024-03-28T00:56:38.303455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:05:31.363487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284824+0000","last_change":"2024-03-28T00:54:36.824661+0000","last_active":"2024-03-28T00:56:38.284824+0000","last_peered":"2024-03-28T00:56:38.284824+0000","last_clean":"2024-03-28T00:56:38.284824+0000","last_became_active":"2024-03-28T00:54:36.824223+0000","last_became_peered":"2024-03-28T00:54:36.824223+0000","last_unstale":"2024-03-28T00:56:38.284824+0000","last_undegraded":"2024-03-28T00:56:38.284824+0000","last_fullsized":"2024-03-28T00:56:38.284824+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:02:25.509248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308563+0000","last_change":"2024-03-28T00:54:32.256436+0000","last_active":"2024-03-28T00:56:38.308563+0000","last_peered":"2024-03-28T00:56:38.308563+0000","last_clean":"2024-03-28T00:56:38.308563+0000","last_became_active":"2024-03-28T00:54:32.256069+0000","last_became_peered":"2024-03-28T00:54:32.256069+0000","last_unstale":"2024-03-28T00:56:38.308563+0000","last_undegraded":"2024-03-28T00:56:38.308563+0000","last_fullsized":"2024-03-28T00:56:38.308563+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:43.688192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307212+0000","last_change":"2024-03-28T00:54:32.799269+0000","last_active":"2024-03-28T00:56:38.307212+0000","last_peered":"2024-03-28T00:56:38.307212+0000","last_clean":"2024-03-28T00:56:38.307212+0000","last_became_active":"2024-03-28T00:54:32.798848+0000","last_became_peered":"2024-03-28T00:54:32.798848+0000","last_unstale":"2024-03-28T00:56:38.307212+0000","last_undegraded":"2024-03-28T00:56:38.307212+0000","last_fullsized":"2024-03-28T00:56:38.307212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:56:20.318466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284703+0000","last_change":"2024-03-28T00:54:36.823153+0000","last_active":"2024-03-28T00:56:38.284703+0000","last_peered":"2024-03-28T00:56:38.284703+0000","last_clean":"2024-03-28T00:56:38.284703+0000","last_became_active":"2024-03-28T00:54:36.822817+0000","last_became_peered":"2024-03-28T00:54:36.822817+0000","last_unstale":"2024-03-28T00:56:38.284703+0000","last_undegraded":"2024-03-28T00:56:38.284703+0000","last_fullsized":"2024-03-28T00:56:38.284703+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:20:23.001599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308631+0000","last_change":"2024-03-28T00:54:32.250439+0000","last_active":"2024-03-28T00:56:38.308631+0000","last_peered":"2024-03-28T00:56:38.308631+0000","last_clean":"2024-03-28T00:56:38.308631+0000","last_became_active":"2024-03-28T00:54:32.250006+0000","last_became_peered":"2024-03-28T00:54:32.250006+0000","last_unstale":"2024-03-28T00:56:38.308631+0000","last_undegraded":"2024-03-28T00:56:38.308631+0000","last_fullsized":"2024-03-28T00:56:38.308631+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:31:38.957429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308661+0000","last_change":"2024-03-28T00:54:32.806888+0000","last_active":"2024-03-28T00:56:38.308661+0000","last_peered":"2024-03-28T00:56:38.308661+0000","last_clean":"2024-03-28T00:56:38.308661+0000","last_became_active":"2024-03-28T00:54:32.806581+0000","last_became_peered":"2024-03-28T00:54:32.806581+0000","last_unstale":"2024-03-28T00:56:38.308661+0000","last_undegraded":"2024-03-28T00:56:38.308661+0000","last_fullsized":"2024-03-28T00:56:38.308661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:25:52.331819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285199+0000","last_change":"2024-03-28T00:54:36.824041+0000","last_active":"2024-03-28T00:56:38.285199+0000","last_peered":"2024-03-28T00:56:38.285199+0000","last_clean":"2024-03-28T00:56:38.285199+0000","last_became_active":"2024-03-28T00:54:36.823658+0000","last_became_peered":"2024-03-28T00:54:36.823658+0000","last_unstale":"2024-03-28T00:56:38.285199+0000","last_undegraded":"2024-03-28T00:56:38.285199+0000","last_fullsized":"2024-03-28T00:56:38.285199+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:11:03.236009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'9","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:45.395923+0000","last_change":"2024-03-28T00:54:32.257534+0000","last_active":"2024-03-28T00:56:45.395923+0000","last_peered":"2024-03-28T00:56:45.395923+0000","last_clean":"2024-03-28T00:56:45.395923+0000","last_became_active":"2024-03-28T00:54:32.256404+0000","last_became_peered":"2024-03-28T00:54:32.256404+0000","last_unstale":"2024-03-28T00:56:45.395923+0000","last_undegraded":"2024-03-28T00:56:45.395923+0000","last_fullsized":"2024-03-28T00:56:45.395923+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:39:38.329799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1853,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308137+0000","last_change":"2024-03-28T00:54:32.806099+0000","last_active":"2024-03-28T00:56:38.308137+0000","last_peered":"2024-03-28T00:56:38.308137+0000","last_clean":"2024-03-28T00:56:38.308137+0000","last_became_active":"2024-03-28T00:54:32.805811+0000","last_became_peered":"2024-03-28T00:54:32.805811+0000","last_unstale":"2024-03-28T00:56:38.308137+0000","last_undegraded":"2024-03-28T00:56:38.308137+0000","last_fullsized":"2024-03-28T00:56:38.308137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:42:32.820245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285325+0000","last_change":"2024-03-28T00:54:36.827198+0000","last_active":"2024-03-28T00:56:38.285325+0000","last_peered":"2024-03-28T00:56:38.285325+0000","last_clean":"2024-03-28T00:56:38.285325+0000","last_became_active":"2024-03-28T00:54:36.826804+0000","last_became_peered":"2024-03-28T00:54:36.826804+0000","last_unstale":"2024-03-28T00:56:38.285325+0000","last_undegraded":"2024-03-28T00:56:38.285325+0000","last_fullsized":"2024-03-28T00:56:38.285325+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:38:11.030404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286690+0000","last_change":"2024-03-28T00:54:32.249394+0000","last_active":"2024-03-28T00:56:38.286690+0000","last_peered":"2024-03-28T00:56:38.286690+0000","last_clean":"2024-03-28T00:56:38.286690+0000","last_became_active":"2024-03-28T00:54:32.249089+0000","last_became_peered":"2024-03-28T00:54:32.249089+0000","last_unstale":"2024-03-28T00:56:38.286690+0000","last_undegraded":"2024-03-28T00:56:38.286690+0000","last_fullsized":"2024-03-28T00:56:38.286690+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:05:01.720053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307883+0000","last_change":"2024-03-28T00:54:32.809133+0000","last_active":"2024-03-28T00:56:38.307883+0000","last_peered":"2024-03-28T00:56:38.307883+0000","last_clean":"2024-03-28T00:56:38.307883+0000","last_became_active":"2024-03-28T00:54:32.808841+0000","last_became_peered":"2024-03-28T00:54:32.808841+0000","last_unstale":"2024-03-28T00:56:38.307883+0000","last_undegraded":"2024-03-28T00:56:38.307883+0000","last_fullsized":"2024-03-28T00:56:38.307883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:19:37.859542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308573+0000","last_change":"2024-03-28T00:54:36.824483+0000","last_active":"2024-03-28T00:56:38.308573+0000","last_peered":"2024-03-28T00:56:38.308573+0000","last_clean":"2024-03-28T00:56:38.308573+0000","last_became_active":"2024-03-28T00:54:36.823973+0000","last_became_peered":"2024-03-28T00:54:36.823973+0000","last_unstale":"2024-03-28T00:56:38.308573+0000","last_undegraded":"2024-03-28T00:56:38.308573+0000","last_fullsized":"2024-03-28T00:56:38.308573+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:01:06.251119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350674+0000","last_change":"2024-03-28T00:54:32.251287+0000","last_active":"2024-03-28T00:56:38.350674+0000","last_peered":"2024-03-28T00:56:38.350674+0000","last_clean":"2024-03-28T00:56:38.350674+0000","last_became_active":"2024-03-28T00:54:32.250904+0000","last_became_peered":"2024-03-28T00:54:32.250904+0000","last_unstale":"2024-03-28T00:56:38.350674+0000","last_undegraded":"2024-03-28T00:56:38.350674+0000","last_fullsized":"2024-03-28T00:56:38.350674+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:18:09.051056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284458+0000","last_change":"2024-03-28T00:54:32.803124+0000","last_active":"2024-03-28T00:56:38.284458+0000","last_peered":"2024-03-28T00:56:38.284458+0000","last_clean":"2024-03-28T00:56:38.284458+0000","last_became_active":"2024-03-28T00:54:32.802825+0000","last_became_peered":"2024-03-28T00:54:32.802825+0000","last_unstale":"2024-03-28T00:56:38.284458+0000","last_undegraded":"2024-03-28T00:56:38.284458+0000","last_fullsized":"2024-03-28T00:56:38.284458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:34:21.825314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303447+0000","last_change":"2024-03-28T00:54:36.832151+0000","last_active":"2024-03-28T00:56:38.303447+0000","last_peered":"2024-03-28T00:56:38.303447+0000","last_clean":"2024-03-28T00:56:38.303447+0000","last_became_active":"2024-03-28T00:54:36.831748+0000","last_became_peered":"2024-03-28T00:54:36.831748+0000","last_unstale":"2024-03-28T00:56:38.303447+0000","last_undegraded":"2024-03-28T00:56:38.303447+0000","last_fullsized":"2024-03-28T00:56:38.303447+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:45:32.284386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307890+0000","last_change":"2024-03-28T00:54:32.259048+0000","last_active":"2024-03-28T00:56:38.307890+0000","last_peered":"2024-03-28T00:56:38.307890+0000","last_clean":"2024-03-28T00:56:38.307890+0000","last_became_active":"2024-03-28T00:54:32.258038+0000","last_became_peered":"2024-03-28T00:54:32.258038+0000","last_unstale":"2024-03-28T00:56:38.307890+0000","last_undegraded":"2024-03-28T00:56:38.307890+0000","last_fullsized":"2024-03-28T00:56:38.307890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:45:23.525516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286245+0000","last_change":"2024-03-28T00:54:32.809771+0000","last_active":"2024-03-28T00:56:38.286245+0000","last_peered":"2024-03-28T00:56:38.286245+0000","last_clean":"2024-03-28T00:56:38.286245+0000","last_became_active":"2024-03-28T00:54:32.809467+0000","last_became_peered":"2024-03-28T00:54:32.809467+0000","last_unstale":"2024-03-28T00:56:38.286245+0000","last_undegraded":"2024-03-28T00:56:38.286245+0000","last_fullsized":"2024-03-28T00:56:38.286245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:30:15.440202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284418+0000","last_change":"2024-03-28T00:54:36.823313+0000","last_active":"2024-03-28T00:56:38.284418+0000","last_peered":"2024-03-28T00:56:38.284418+0000","last_clean":"2024-03-28T00:56:38.284418+0000","last_became_active":"2024-03-28T00:54:36.822923+0000","last_became_peered":"2024-03-28T00:54:36.822923+0000","last_unstale":"2024-03-28T00:56:38.284418+0000","last_undegraded":"2024-03-28T00:56:38.284418+0000","last_fullsized":"2024-03-28T00:56:38.284418+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:22:21.125028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'5","reported_seq":153,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:45.395588+0000","last_change":"2024-03-28T00:54:32.255084+0000","last_active":"2024-03-28T00:56:45.395588+0000","last_peered":"2024-03-28T00:56:45.395588+0000","last_clean":"2024-03-28T00:56:45.395588+0000","last_became_active":"2024-03-28T00:54:32.254744+0000","last_became_peered":"2024-03-28T00:54:32.254744+0000","last_unstale":"2024-03-28T00:56:45.395588+0000","last_undegraded":"2024-03-28T00:56:45.395588+0000","last_fullsized":"2024-03-28T00:56:45.395588+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:32:03.906826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":115,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307394+0000","last_change":"2024-03-28T00:54:32.810050+0000","last_active":"2024-03-28T00:56:38.307394+0000","last_peered":"2024-03-28T00:56:38.307394+0000","last_clean":"2024-03-28T00:56:38.307394+0000","last_became_active":"2024-03-28T00:54:32.809731+0000","last_became_peered":"2024-03-28T00:54:32.809731+0000","last_unstale":"2024-03-28T00:56:38.307394+0000","last_undegraded":"2024-03-28T00:56:38.307394+0000","last_fullsized":"2024-03-28T00:56:38.307394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:20:11.876731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307671+0000","last_change":"2024-03-28T00:55:37.566900+0000","last_active":"2024-03-28T00:56:38.307671+0000","last_peered":"2024-03-28T00:56:38.307671+0000","last_clean":"2024-03-28T00:56:38.307671+0000","last_became_active":"2024-03-28T00:55:37.566491+0000","last_became_peered":"2024-03-28T00:55:37.566491+0000","last_unstale":"2024-03-28T00:56:38.307671+0000","last_undegraded":"2024-03-28T00:56:38.307671+0000","last_fullsized":"2024-03-28T00:56:38.307671+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:31:11.489908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308398+0000","last_change":"2024-03-28T00:54:32.259159+0000","last_active":"2024-03-28T00:56:38.308398+0000","last_peered":"2024-03-28T00:56:38.308398+0000","last_clean":"2024-03-28T00:56:38.308398+0000","last_became_active":"2024-03-28T00:54:32.258708+0000","last_became_peered":"2024-03-28T00:54:32.258708+0000","last_unstale":"2024-03-28T00:56:38.308398+0000","last_undegraded":"2024-03-28T00:56:38.308398+0000","last_fullsized":"2024-03-28T00:56:38.308398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:37:01.524878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303253+0000","last_change":"2024-03-28T00:54:32.814358+0000","last_active":"2024-03-28T00:56:38.303253+0000","last_peered":"2024-03-28T00:56:38.303253+0000","last_clean":"2024-03-28T00:56:38.303253+0000","last_became_active":"2024-03-28T00:54:32.814018+0000","last_became_peered":"2024-03-28T00:54:32.814018+0000","last_unstale":"2024-03-28T00:56:38.303253+0000","last_undegraded":"2024-03-28T00:56:38.303253+0000","last_fullsized":"2024-03-28T00:56:38.303253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:24:38.287853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284939+0000","last_change":"2024-03-28T00:54:36.822161+0000","last_active":"2024-03-28T00:56:38.284939+0000","last_peered":"2024-03-28T00:56:38.284939+0000","last_clean":"2024-03-28T00:56:38.284939+0000","last_became_active":"2024-03-28T00:54:36.821750+0000","last_became_peered":"2024-03-28T00:54:36.821750+0000","last_unstale":"2024-03-28T00:56:38.284939+0000","last_undegraded":"2024-03-28T00:56:38.284939+0000","last_fullsized":"2024-03-28T00:56:38.284939+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:13:36.892453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308562+0000","last_change":"2024-03-28T00:54:32.257038+0000","last_active":"2024-03-28T00:56:38.308562+0000","last_peered":"2024-03-28T00:56:38.308562+0000","last_clean":"2024-03-28T00:56:38.308562+0000","last_became_active":"2024-03-28T00:54:32.256702+0000","last_became_peered":"2024-03-28T00:54:32.256702+0000","last_unstale":"2024-03-28T00:56:38.308562+0000","last_undegraded":"2024-03-28T00:56:38.308562+0000","last_fullsized":"2024-03-28T00:56:38.308562+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:35:17.294116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307329+0000","last_change":"2024-03-28T00:54:32.813514+0000","last_active":"2024-03-28T00:56:38.307329+0000","last_peered":"2024-03-28T00:56:38.307329+0000","last_clean":"2024-03-28T00:56:38.307329+0000","last_became_active":"2024-03-28T00:54:32.813192+0000","last_became_peered":"2024-03-28T00:54:32.813192+0000","last_unstale":"2024-03-28T00:56:38.307329+0000","last_undegraded":"2024-03-28T00:56:38.307329+0000","last_fullsized":"2024-03-28T00:56:38.307329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:55:03.410818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286147+0000","last_change":"2024-03-28T00:54:36.820995+0000","last_active":"2024-03-28T00:56:38.286147+0000","last_peered":"2024-03-28T00:56:38.286147+0000","last_clean":"2024-03-28T00:56:38.286147+0000","last_became_active":"2024-03-28T00:54:36.820628+0000","last_became_peered":"2024-03-28T00:54:36.820628+0000","last_unstale":"2024-03-28T00:56:38.286147+0000","last_undegraded":"2024-03-28T00:56:38.286147+0000","last_fullsized":"2024-03-28T00:56:38.286147+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:18:58.771527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'40","reported_seq":313,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:46.396459+0000","last_change":"2024-03-28T00:54:32.251092+0000","last_active":"2024-03-28T00:56:46.396459+0000","last_peered":"2024-03-28T00:56:46.396459+0000","last_clean":"2024-03-28T00:56:46.396459+0000","last_became_active":"2024-03-28T00:54:32.250759+0000","last_became_peered":"2024-03-28T00:54:32.250759+0000","last_unstale":"2024-03-28T00:56:46.396459+0000","last_undegraded":"2024-03-28T00:56:46.396459+0000","last_fullsized":"2024-03-28T00:56:46.396459+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:57:30.283365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73065,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":240,"num_read_kb":191,"num_write":41,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308272+0000","last_change":"2024-03-28T00:54:32.809580+0000","last_active":"2024-03-28T00:56:38.308272+0000","last_peered":"2024-03-28T00:56:38.308272+0000","last_clean":"2024-03-28T00:56:38.308272+0000","last_became_active":"2024-03-28T00:54:32.809238+0000","last_became_peered":"2024-03-28T00:54:32.809238+0000","last_unstale":"2024-03-28T00:56:38.308272+0000","last_undegraded":"2024-03-28T00:56:38.308272+0000","last_fullsized":"2024-03-28T00:56:38.308272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:20:17.397594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308935+0000","last_change":"2024-03-28T00:54:36.826851+0000","last_active":"2024-03-28T00:56:38.308935+0000","last_peered":"2024-03-28T00:56:38.308935+0000","last_clean":"2024-03-28T00:56:38.308935+0000","last_became_active":"2024-03-28T00:54:36.826359+0000","last_became_peered":"2024-03-28T00:54:36.826359+0000","last_unstale":"2024-03-28T00:56:38.308935+0000","last_undegraded":"2024-03-28T00:56:38.308935+0000","last_fullsized":"2024-03-28T00:56:38.308935+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:27:48.857368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285191+0000","last_change":"2024-03-28T00:54:32.251701+0000","last_active":"2024-03-28T00:56:38.285191+0000","last_peered":"2024-03-28T00:56:38.285191+0000","last_clean":"2024-03-28T00:56:38.285191+0000","last_became_active":"2024-03-28T00:54:32.251403+0000","last_became_peered":"2024-03-28T00:54:32.251403+0000","last_unstale":"2024-03-28T00:56:38.285191+0000","last_undegraded":"2024-03-28T00:56:38.285191+0000","last_fullsized":"2024-03-28T00:56:38.285191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:28:16.127643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350770+0000","last_change":"2024-03-28T00:54:32.806810+0000","last_active":"2024-03-28T00:56:38.350770+0000","last_peered":"2024-03-28T00:56:38.350770+0000","last_clean":"2024-03-28T00:56:38.350770+0000","last_became_active":"2024-03-28T00:54:32.806468+0000","last_became_peered":"2024-03-28T00:54:32.806468+0000","last_unstale":"2024-03-28T00:56:38.350770+0000","last_undegraded":"2024-03-28T00:56:38.350770+0000","last_fullsized":"2024-03-28T00:56:38.350770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:54:16.196032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307882+0000","last_change":"2024-03-28T00:55:37.919873+0000","last_active":"2024-03-28T00:56:38.307882+0000","last_peered":"2024-03-28T00:56:38.307882+0000","last_clean":"2024-03-28T00:56:38.307882+0000","last_became_active":"2024-03-28T00:55:37.919396+0000","last_became_peered":"2024-03-28T00:55:37.919396+0000","last_unstale":"2024-03-28T00:56:38.307882+0000","last_undegraded":"2024-03-28T00:56:38.307882+0000","last_fullsized":"2024-03-28T00:56:38.307882+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:05:59.524285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307902+0000","last_change":"2024-03-28T00:54:32.259440+0000","last_active":"2024-03-28T00:56:38.307902+0000","last_peered":"2024-03-28T00:56:38.307902+0000","last_clean":"2024-03-28T00:56:38.307902+0000","last_became_active":"2024-03-28T00:54:32.258225+0000","last_became_peered":"2024-03-28T00:54:32.258225+0000","last_unstale":"2024-03-28T00:56:38.307902+0000","last_undegraded":"2024-03-28T00:56:38.307902+0000","last_fullsized":"2024-03-28T00:56:38.307902+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:07:18.398302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308098+0000","last_change":"2024-03-28T00:54:32.806853+0000","last_active":"2024-03-28T00:56:38.308098+0000","last_peered":"2024-03-28T00:56:38.308098+0000","last_clean":"2024-03-28T00:56:38.308098+0000","last_became_active":"2024-03-28T00:54:32.806551+0000","last_became_peered":"2024-03-28T00:54:32.806551+0000","last_unstale":"2024-03-28T00:56:38.308098+0000","last_undegraded":"2024-03-28T00:56:38.308098+0000","last_fullsized":"2024-03-28T00:56:38.308098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:49:45.774252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286620+0000","last_change":"2024-03-28T00:54:36.827542+0000","last_active":"2024-03-28T00:56:38.286620+0000","last_peered":"2024-03-28T00:56:38.286620+0000","last_clean":"2024-03-28T00:56:38.286620+0000","last_became_active":"2024-03-28T00:54:36.827163+0000","last_became_peered":"2024-03-28T00:54:36.827163+0000","last_unstale":"2024-03-28T00:56:38.286620+0000","last_undegraded":"2024-03-28T00:56:38.286620+0000","last_fullsized":"2024-03-28T00:56:38.286620+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:36:15.278111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'3","reported_seq":152,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:44.395209+0000","last_change":"2024-03-28T00:54:32.253231+0000","last_active":"2024-03-28T00:56:44.395209+0000","last_peered":"2024-03-28T00:56:44.395209+0000","last_clean":"2024-03-28T00:56:44.395209+0000","last_became_active":"2024-03-28T00:54:32.252731+0000","last_became_peered":"2024-03-28T00:54:32.252731+0000","last_unstale":"2024-03-28T00:56:44.395209+0000","last_undegraded":"2024-03-28T00:56:44.395209+0000","last_fullsized":"2024-03-28T00:56:44.395209+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:05:01.716654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308421+0000","last_change":"2024-03-28T00:54:32.805073+0000","last_active":"2024-03-28T00:56:38.308421+0000","last_peered":"2024-03-28T00:56:38.308421+0000","last_clean":"2024-03-28T00:56:38.308421+0000","last_became_active":"2024-03-28T00:54:32.803690+0000","last_became_peered":"2024-03-28T00:54:32.803690+0000","last_unstale":"2024-03-28T00:56:38.308421+0000","last_undegraded":"2024-03-28T00:56:38.308421+0000","last_fullsized":"2024-03-28T00:56:38.308421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:41:16.586168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350844+0000","last_change":"2024-03-28T00:54:36.835463+0000","last_active":"2024-03-28T00:56:38.350844+0000","last_peered":"2024-03-28T00:56:38.350844+0000","last_clean":"2024-03-28T00:56:38.350844+0000","last_became_active":"2024-03-28T00:54:36.834982+0000","last_became_peered":"2024-03-28T00:54:36.834982+0000","last_unstale":"2024-03-28T00:56:38.350844+0000","last_undegraded":"2024-03-28T00:56:38.350844+0000","last_fullsized":"2024-03-28T00:56:38.350844+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:27:32.259586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307377+0000","last_change":"2024-03-28T00:54:32.248461+0000","last_active":"2024-03-28T00:56:38.307377+0000","last_peered":"2024-03-28T00:56:38.307377+0000","last_clean":"2024-03-28T00:56:38.307377+0000","last_became_active":"2024-03-28T00:54:32.248140+0000","last_became_peered":"2024-03-28T00:54:32.248140+0000","last_unstale":"2024-03-28T00:56:38.307377+0000","last_undegraded":"2024-03-28T00:56:38.307377+0000","last_fullsized":"2024-03-28T00:56:38.307377+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:13:38.784288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307746+0000","last_change":"2024-03-28T00:54:32.806651+0000","last_active":"2024-03-28T00:56:38.307746+0000","last_peered":"2024-03-28T00:56:38.307746+0000","last_clean":"2024-03-28T00:56:38.307746+0000","last_became_active":"2024-03-28T00:54:32.806329+0000","last_became_peered":"2024-03-28T00:54:32.806329+0000","last_unstale":"2024-03-28T00:56:38.307746+0000","last_undegraded":"2024-03-28T00:56:38.307746+0000","last_fullsized":"2024-03-28T00:56:38.307746+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:28:32.815396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307991+0000","last_change":"2024-03-28T00:54:36.827016+0000","last_active":"2024-03-28T00:56:38.307991+0000","last_peered":"2024-03-28T00:56:38.307991+0000","last_clean":"2024-03-28T00:56:38.307991+0000","last_became_active":"2024-03-28T00:54:36.826428+0000","last_became_peered":"2024-03-28T00:54:36.826428+0000","last_unstale":"2024-03-28T00:56:38.307991+0000","last_undegraded":"2024-03-28T00:56:38.307991+0000","last_fullsized":"2024-03-28T00:56:38.307991+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:33:01.593990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308142+0000","last_change":"2024-03-28T00:55:37.573420+0000","last_active":"2024-03-28T00:56:38.308142+0000","last_peered":"2024-03-28T00:56:38.308142+0000","last_clean":"2024-03-28T00:56:38.308142+0000","last_became_active":"2024-03-28T00:55:37.568657+0000","last_became_peered":"2024-03-28T00:55:37.568657+0000","last_unstale":"2024-03-28T00:56:38.308142+0000","last_undegraded":"2024-03-28T00:56:38.308142+0000","last_fullsized":"2024-03-28T00:56:38.308142+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:03:21.483054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308301+0000","last_change":"2024-03-28T00:55:37.919698+0000","last_active":"2024-03-28T00:56:38.308301+0000","last_peered":"2024-03-28T00:56:38.308301+0000","last_clean":"2024-03-28T00:56:38.308301+0000","last_became_active":"2024-03-28T00:55:37.919276+0000","last_became_peered":"2024-03-28T00:55:37.919276+0000","last_unstale":"2024-03-28T00:56:38.308301+0000","last_undegraded":"2024-03-28T00:56:38.308301+0000","last_fullsized":"2024-03-28T00:56:38.308301+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:12:39.042250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308070+0000","last_change":"2024-03-28T00:55:37.918600+0000","last_active":"2024-03-28T00:56:38.308070+0000","last_peered":"2024-03-28T00:56:38.308070+0000","last_clean":"2024-03-28T00:56:38.308070+0000","last_became_active":"2024-03-28T00:55:37.918088+0000","last_became_peered":"2024-03-28T00:55:37.918088+0000","last_unstale":"2024-03-28T00:56:38.308070+0000","last_undegraded":"2024-03-28T00:56:38.308070+0000","last_fullsized":"2024-03-28T00:56:38.308070+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:29:23.492261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286573+0000","last_change":"2024-03-28T00:54:32.264038+0000","last_active":"2024-03-28T00:56:38.286573+0000","last_peered":"2024-03-28T00:56:38.286573+0000","last_clean":"2024-03-28T00:56:38.286573+0000","last_became_active":"2024-03-28T00:54:32.263706+0000","last_became_peered":"2024-03-28T00:54:32.263706+0000","last_unstale":"2024-03-28T00:56:38.286573+0000","last_undegraded":"2024-03-28T00:56:38.286573+0000","last_fullsized":"2024-03-28T00:56:38.286573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:03.244556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286955+0000","last_change":"2024-03-28T00:54:32.804727+0000","last_active":"2024-03-28T00:56:38.286955+0000","last_peered":"2024-03-28T00:56:38.286955+0000","last_clean":"2024-03-28T00:56:38.286955+0000","last_became_active":"2024-03-28T00:54:32.804422+0000","last_became_peered":"2024-03-28T00:54:32.804422+0000","last_unstale":"2024-03-28T00:56:38.286955+0000","last_undegraded":"2024-03-28T00:56:38.286955+0000","last_fullsized":"2024-03-28T00:56:38.286955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:29:57.983722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.351075+0000","last_change":"2024-03-28T00:54:36.833416+0000","last_active":"2024-03-28T00:56:38.351075+0000","last_peered":"2024-03-28T00:56:38.351075+0000","last_clean":"2024-03-28T00:56:38.351075+0000","last_became_active":"2024-03-28T00:54:36.832985+0000","last_became_peered":"2024-03-28T00:54:36.832985+0000","last_unstale":"2024-03-28T00:56:38.351075+0000","last_undegraded":"2024-03-28T00:56:38.351075+0000","last_fullsized":"2024-03-28T00:56:38.351075+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:47:30.376032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308027+0000","last_change":"2024-03-28T00:54:32.256417+0000","last_active":"2024-03-28T00:56:38.308027+0000","last_peered":"2024-03-28T00:56:38.308027+0000","last_clean":"2024-03-28T00:56:38.308027+0000","last_became_active":"2024-03-28T00:54:32.256088+0000","last_became_peered":"2024-03-28T00:54:32.256088+0000","last_unstale":"2024-03-28T00:56:38.308027+0000","last_undegraded":"2024-03-28T00:56:38.308027+0000","last_fullsized":"2024-03-28T00:56:38.308027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:58.661597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286728+0000","last_change":"2024-03-28T00:54:32.817604+0000","last_active":"2024-03-28T00:56:38.286728+0000","last_peered":"2024-03-28T00:56:38.286728+0000","last_clean":"2024-03-28T00:56:38.286728+0000","last_became_active":"2024-03-28T00:54:32.817211+0000","last_became_peered":"2024-03-28T00:54:32.817211+0000","last_unstale":"2024-03-28T00:56:38.286728+0000","last_undegraded":"2024-03-28T00:56:38.286728+0000","last_fullsized":"2024-03-28T00:56:38.286728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:02:28.335075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350947+0000","last_change":"2024-03-28T00:54:37.638968+0000","last_active":"2024-03-28T00:56:38.350947+0000","last_peered":"2024-03-28T00:56:38.350947+0000","last_clean":"2024-03-28T00:56:38.350947+0000","last_became_active":"2024-03-28T00:54:37.638446+0000","last_became_peered":"2024-03-28T00:54:37.638446+0000","last_unstale":"2024-03-28T00:56:38.350947+0000","last_undegraded":"2024-03-28T00:56:38.350947+0000","last_fullsized":"2024-03-28T00:56:38.350947+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:54:56.348760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308145+0000","last_change":"2024-03-28T00:54:32.809822+0000","last_active":"2024-03-28T00:56:38.308145+0000","last_peered":"2024-03-28T00:56:38.308145+0000","last_clean":"2024-03-28T00:56:38.308145+0000","last_became_active":"2024-03-28T00:54:32.808284+0000","last_became_peered":"2024-03-28T00:54:32.808284+0000","last_unstale":"2024-03-28T00:56:38.308145+0000","last_undegraded":"2024-03-28T00:56:38.308145+0000","last_fullsized":"2024-03-28T00:56:38.308145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:49:11.651960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":151,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:43.393776+0000","last_change":"2024-03-28T00:54:32.261806+0000","last_active":"2024-03-28T00:56:43.393776+0000","last_peered":"2024-03-28T00:56:43.393776+0000","last_clean":"2024-03-28T00:56:43.393776+0000","last_became_active":"2024-03-28T00:54:32.261493+0000","last_became_peered":"2024-03-28T00:54:32.261493+0000","last_unstale":"2024-03-28T00:56:43.393776+0000","last_undegraded":"2024-03-28T00:56:43.393776+0000","last_fullsized":"2024-03-28T00:56:43.393776+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:36:04.283408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308358+0000","last_change":"2024-03-28T00:54:36.819332+0000","last_active":"2024-03-28T00:56:38.308358+0000","last_peered":"2024-03-28T00:56:38.308358+0000","last_clean":"2024-03-28T00:56:38.308358+0000","last_became_active":"2024-03-28T00:54:36.818932+0000","last_became_peered":"2024-03-28T00:54:36.818932+0000","last_unstale":"2024-03-28T00:56:38.308358+0000","last_undegraded":"2024-03-28T00:56:38.308358+0000","last_fullsized":"2024-03-28T00:56:38.308358+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:00:39.446738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284570+0000","last_change":"2024-03-28T00:54:32.806093+0000","last_active":"2024-03-28T00:56:38.284570+0000","last_peered":"2024-03-28T00:56:38.284570+0000","last_clean":"2024-03-28T00:56:38.284570+0000","last_became_active":"2024-03-28T00:54:32.805734+0000","last_became_peered":"2024-03-28T00:54:32.805734+0000","last_unstale":"2024-03-28T00:56:38.284570+0000","last_undegraded":"2024-03-28T00:56:38.284570+0000","last_fullsized":"2024-03-28T00:56:38.284570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:51:41.066225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284968+0000","last_change":"2024-03-28T00:54:32.245182+0000","last_active":"2024-03-28T00:56:38.284968+0000","last_peered":"2024-03-28T00:56:38.284968+0000","last_clean":"2024-03-28T00:56:38.284968+0000","last_became_active":"2024-03-28T00:54:32.244855+0000","last_became_peered":"2024-03-28T00:54:32.244855+0000","last_unstale":"2024-03-28T00:56:38.284968+0000","last_undegraded":"2024-03-28T00:56:38.284968+0000","last_fullsized":"2024-03-28T00:56:38.284968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:39:09.997199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308243+0000","last_change":"2024-03-28T00:53:20.355628+0000","last_active":"2024-03-28T00:56:38.308243+0000","last_peered":"2024-03-28T00:56:38.308243+0000","last_clean":"2024-03-28T00:56:38.308243+0000","last_became_active":"2024-03-28T00:53:20.355175+0000","last_became_peered":"2024-03-28T00:53:20.355175+0000","last_unstale":"2024-03-28T00:56:38.308243+0000","last_undegraded":"2024-03-28T00:56:38.308243+0000","last_fullsized":"2024-03-28T00:56:38.308243+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:49:57.054435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:49:57.054435+0000","last_clean_scrub_stamp":"2024-03-28T00:49:57.054435+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:55:46.753799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308032+0000","last_change":"2024-03-28T00:54:37.639760+0000","last_active":"2024-03-28T00:56:38.308032+0000","last_peered":"2024-03-28T00:56:38.308032+0000","last_clean":"2024-03-28T00:56:38.308032+0000","last_became_active":"2024-03-28T00:54:37.639287+0000","last_became_peered":"2024-03-28T00:54:37.639287+0000","last_unstale":"2024-03-28T00:56:38.308032+0000","last_undegraded":"2024-03-28T00:56:38.308032+0000","last_fullsized":"2024-03-28T00:56:38.308032+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:50:02.840312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350597+0000","last_change":"2024-03-28T00:54:32.807764+0000","last_active":"2024-03-28T00:56:38.350597+0000","last_peered":"2024-03-28T00:56:38.350597+0000","last_clean":"2024-03-28T00:56:38.350597+0000","last_became_active":"2024-03-28T00:54:32.807441+0000","last_became_peered":"2024-03-28T00:54:32.807441+0000","last_unstale":"2024-03-28T00:56:38.350597+0000","last_undegraded":"2024-03-28T00:56:38.350597+0000","last_fullsized":"2024-03-28T00:56:38.350597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:37:40.540514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308361+0000","last_change":"2024-03-28T00:54:32.259192+0000","last_active":"2024-03-28T00:56:38.308361+0000","last_peered":"2024-03-28T00:56:38.308361+0000","last_clean":"2024-03-28T00:56:38.308361+0000","last_became_active":"2024-03-28T00:54:32.258880+0000","last_became_peered":"2024-03-28T00:54:32.258880+0000","last_unstale":"2024-03-28T00:56:38.308361+0000","last_undegraded":"2024-03-28T00:56:38.308361+0000","last_fullsized":"2024-03-28T00:56:38.308361+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:20:11.384633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307815+0000","last_change":"2024-03-28T00:54:36.822268+0000","last_active":"2024-03-28T00:56:38.307815+0000","last_peered":"2024-03-28T00:56:38.307815+0000","last_clean":"2024-03-28T00:56:38.307815+0000","last_became_active":"2024-03-28T00:54:36.821773+0000","last_became_peered":"2024-03-28T00:54:36.821773+0000","last_unstale":"2024-03-28T00:56:38.307815+0000","last_undegraded":"2024-03-28T00:56:38.307815+0000","last_fullsized":"2024-03-28T00:56:38.307815+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:56:11.206975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308433+0000","last_change":"2024-03-28T00:54:32.256133+0000","last_active":"2024-03-28T00:56:38.308433+0000","last_peered":"2024-03-28T00:56:38.308433+0000","last_clean":"2024-03-28T00:56:38.308433+0000","last_became_active":"2024-03-28T00:54:32.250535+0000","last_became_peered":"2024-03-28T00:54:32.250535+0000","last_unstale":"2024-03-28T00:56:38.308433+0000","last_undegraded":"2024-03-28T00:56:38.308433+0000","last_fullsized":"2024-03-28T00:56:38.308433+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:18:50.874181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.351152+0000","last_change":"2024-03-28T00:54:32.808883+0000","last_active":"2024-03-28T00:56:38.351152+0000","last_peered":"2024-03-28T00:56:38.351152+0000","last_clean":"2024-03-28T00:56:38.351152+0000","last_became_active":"2024-03-28T00:54:32.808587+0000","last_became_peered":"2024-03-28T00:54:32.808587+0000","last_unstale":"2024-03-28T00:56:38.351152+0000","last_undegraded":"2024-03-28T00:56:38.351152+0000","last_fullsized":"2024-03-28T00:56:38.351152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:09:55.164876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284641+0000","last_change":"2024-03-28T00:54:37.637743+0000","last_active":"2024-03-28T00:56:38.284641+0000","last_peered":"2024-03-28T00:56:38.284641+0000","last_clean":"2024-03-28T00:56:38.284641+0000","last_became_active":"2024-03-28T00:54:37.637280+0000","last_became_peered":"2024-03-28T00:54:37.637280+0000","last_unstale":"2024-03-28T00:56:38.284641+0000","last_undegraded":"2024-03-28T00:56:38.284641+0000","last_fullsized":"2024-03-28T00:56:38.284641+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:40:51.431757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'9","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:45.255094+0000","last_change":"2024-03-28T00:54:32.246444+0000","last_active":"2024-03-28T00:56:45.255094+0000","last_peered":"2024-03-28T00:56:45.255094+0000","last_clean":"2024-03-28T00:56:45.255094+0000","last_became_active":"2024-03-28T00:54:32.245106+0000","last_became_peered":"2024-03-28T00:54:32.245106+0000","last_unstale":"2024-03-28T00:56:45.255094+0000","last_undegraded":"2024-03-28T00:56:45.255094+0000","last_fullsized":"2024-03-28T00:56:45.255094+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:13:45.123625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285364+0000","last_change":"2024-03-28T00:54:32.812851+0000","last_active":"2024-03-28T00:56:38.285364+0000","last_peered":"2024-03-28T00:56:38.285364+0000","last_clean":"2024-03-28T00:56:38.285364+0000","last_became_active":"2024-03-28T00:54:32.812518+0000","last_became_peered":"2024-03-28T00:54:32.812518+0000","last_unstale":"2024-03-28T00:56:38.285364+0000","last_undegraded":"2024-03-28T00:56:38.285364+0000","last_fullsized":"2024-03-28T00:56:38.285364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:08:41.518888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284865+0000","last_change":"2024-03-28T00:54:36.818709+0000","last_active":"2024-03-28T00:56:38.284865+0000","last_peered":"2024-03-28T00:56:38.284865+0000","last_clean":"2024-03-28T00:56:38.284865+0000","last_became_active":"2024-03-28T00:54:36.818304+0000","last_became_peered":"2024-03-28T00:54:36.818304+0000","last_unstale":"2024-03-28T00:56:38.284865+0000","last_undegraded":"2024-03-28T00:56:38.284865+0000","last_fullsized":"2024-03-28T00:56:38.284865+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:03:07.642685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285251+0000","last_change":"2024-03-28T00:54:32.256031+0000","last_active":"2024-03-28T00:56:38.285251+0000","last_peered":"2024-03-28T00:56:38.285251+0000","last_clean":"2024-03-28T00:56:38.285251+0000","last_became_active":"2024-03-28T00:54:32.255597+0000","last_became_peered":"2024-03-28T00:54:32.255597+0000","last_unstale":"2024-03-28T00:56:38.285251+0000","last_undegraded":"2024-03-28T00:56:38.285251+0000","last_fullsized":"2024-03-28T00:56:38.285251+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:31:10.837212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303584+0000","last_change":"2024-03-28T00:54:32.816961+0000","last_active":"2024-03-28T00:56:38.303584+0000","last_peered":"2024-03-28T00:56:38.303584+0000","last_clean":"2024-03-28T00:56:38.303584+0000","last_became_active":"2024-03-28T00:54:32.816408+0000","last_became_peered":"2024-03-28T00:54:32.816408+0000","last_unstale":"2024-03-28T00:56:38.303584+0000","last_undegraded":"2024-03-28T00:56:38.303584+0000","last_fullsized":"2024-03-28T00:56:38.303584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:08:57.441492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285039+0000","last_change":"2024-03-28T00:55:37.918692+0000","last_active":"2024-03-28T00:56:38.285039+0000","last_peered":"2024-03-28T00:56:38.285039+0000","last_clean":"2024-03-28T00:56:38.285039+0000","last_became_active":"2024-03-28T00:55:37.918195+0000","last_became_peered":"2024-03-28T00:55:37.918195+0000","last_unstale":"2024-03-28T00:56:38.285039+0000","last_undegraded":"2024-03-28T00:56:38.285039+0000","last_fullsized":"2024-03-28T00:56:38.285039+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:01:06.012325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.306972+0000","last_change":"2024-03-28T00:55:37.570700+0000","last_active":"2024-03-28T00:56:38.306972+0000","last_peered":"2024-03-28T00:56:38.306972+0000","last_clean":"2024-03-28T00:56:38.306972+0000","last_became_active":"2024-03-28T00:55:37.566298+0000","last_became_peered":"2024-03-28T00:55:37.566298+0000","last_unstale":"2024-03-28T00:56:38.306972+0000","last_undegraded":"2024-03-28T00:56:38.306972+0000","last_fullsized":"2024-03-28T00:56:38.306972+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:46:48.352817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307628+0000","last_change":"2024-03-28T00:54:32.811592+0000","last_active":"2024-03-28T00:56:38.307628+0000","last_peered":"2024-03-28T00:56:38.307628+0000","last_clean":"2024-03-28T00:56:38.307628+0000","last_became_active":"2024-03-28T00:54:32.811228+0000","last_became_peered":"2024-03-28T00:54:32.811228+0000","last_unstale":"2024-03-28T00:56:38.307628+0000","last_undegraded":"2024-03-28T00:56:38.307628+0000","last_fullsized":"2024-03-28T00:56:38.307628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:14:19.585237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286681+0000","last_change":"2024-03-28T00:54:36.822501+0000","last_active":"2024-03-28T00:56:38.286681+0000","last_peered":"2024-03-28T00:56:38.286681+0000","last_clean":"2024-03-28T00:56:38.286681+0000","last_became_active":"2024-03-28T00:54:36.821630+0000","last_became_peered":"2024-03-28T00:54:36.821630+0000","last_unstale":"2024-03-28T00:56:38.286681+0000","last_undegraded":"2024-03-28T00:56:38.286681+0000","last_fullsized":"2024-03-28T00:56:38.286681+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:20:52.247055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284556+0000","last_change":"2024-03-28T00:54:32.247328+0000","last_active":"2024-03-28T00:56:38.284556+0000","last_peered":"2024-03-28T00:56:38.284556+0000","last_clean":"2024-03-28T00:56:38.284556+0000","last_became_active":"2024-03-28T00:54:32.247003+0000","last_became_peered":"2024-03-28T00:54:32.247003+0000","last_unstale":"2024-03-28T00:56:38.284556+0000","last_undegraded":"2024-03-28T00:56:38.284556+0000","last_fullsized":"2024-03-28T00:56:38.284556+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:33:09.764946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308401+0000","last_change":"2024-03-28T00:54:32.805977+0000","last_active":"2024-03-28T00:56:38.308401+0000","last_peered":"2024-03-28T00:56:38.308401+0000","last_clean":"2024-03-28T00:56:38.308401+0000","last_became_active":"2024-03-28T00:54:32.805636+0000","last_became_peered":"2024-03-28T00:54:32.805636+0000","last_unstale":"2024-03-28T00:56:38.308401+0000","last_undegraded":"2024-03-28T00:56:38.308401+0000","last_fullsized":"2024-03-28T00:56:38.308401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:04:42.901732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284987+0000","last_change":"2024-03-28T00:54:36.822103+0000","last_active":"2024-03-28T00:56:38.284987+0000","last_peered":"2024-03-28T00:56:38.284987+0000","last_clean":"2024-03-28T00:56:38.284987+0000","last_became_active":"2024-03-28T00:54:36.821695+0000","last_became_peered":"2024-03-28T00:54:36.821695+0000","last_unstale":"2024-03-28T00:56:38.284987+0000","last_undegraded":"2024-03-28T00:56:38.284987+0000","last_fullsized":"2024-03-28T00:56:38.284987+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:45:00.613955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285112+0000","last_change":"2024-03-28T00:54:32.251523+0000","last_active":"2024-03-28T00:56:38.285112+0000","last_peered":"2024-03-28T00:56:38.285112+0000","last_clean":"2024-03-28T00:56:38.285112+0000","last_became_active":"2024-03-28T00:54:32.251223+0000","last_became_peered":"2024-03-28T00:54:32.251223+0000","last_unstale":"2024-03-28T00:56:38.285112+0000","last_undegraded":"2024-03-28T00:56:38.285112+0000","last_fullsized":"2024-03-28T00:56:38.285112+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:08:20.489848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284729+0000","last_change":"2024-03-28T00:54:32.817641+0000","last_active":"2024-03-28T00:56:38.284729+0000","last_peered":"2024-03-28T00:56:38.284729+0000","last_clean":"2024-03-28T00:56:38.284729+0000","last_became_active":"2024-03-28T00:54:32.817252+0000","last_became_peered":"2024-03-28T00:54:32.817252+0000","last_unstale":"2024-03-28T00:56:38.284729+0000","last_undegraded":"2024-03-28T00:56:38.284729+0000","last_fullsized":"2024-03-28T00:56:38.284729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:54:48.275005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285020+0000","last_change":"2024-03-28T00:54:37.638990+0000","last_active":"2024-03-28T00:56:38.285020+0000","last_peered":"2024-03-28T00:56:38.285020+0000","last_clean":"2024-03-28T00:56:38.285020+0000","last_became_active":"2024-03-28T00:54:37.638538+0000","last_became_peered":"2024-03-28T00:54:37.638538+0000","last_unstale":"2024-03-28T00:56:38.285020+0000","last_undegraded":"2024-03-28T00:56:38.285020+0000","last_fullsized":"2024-03-28T00:56:38.285020+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:28:10.791805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308482+0000","last_change":"2024-03-28T00:54:32.253254+0000","last_active":"2024-03-28T00:56:38.308482+0000","last_peered":"2024-03-28T00:56:38.308482+0000","last_clean":"2024-03-28T00:56:38.308482+0000","last_became_active":"2024-03-28T00:54:32.252879+0000","last_became_peered":"2024-03-28T00:54:32.252879+0000","last_unstale":"2024-03-28T00:56:38.308482+0000","last_undegraded":"2024-03-28T00:56:38.308482+0000","last_fullsized":"2024-03-28T00:56:38.308482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:59:51.039215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308081+0000","last_change":"2024-03-28T00:54:32.811288+0000","last_active":"2024-03-28T00:56:38.308081+0000","last_peered":"2024-03-28T00:56:38.308081+0000","last_clean":"2024-03-28T00:56:38.308081+0000","last_became_active":"2024-03-28T00:54:32.810968+0000","last_became_peered":"2024-03-28T00:54:32.810968+0000","last_unstale":"2024-03-28T00:56:38.308081+0000","last_undegraded":"2024-03-28T00:56:38.308081+0000","last_fullsized":"2024-03-28T00:56:38.308081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:16:56.113046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307049+0000","last_change":"2024-03-28T00:54:36.830714+0000","last_active":"2024-03-28T00:56:38.307049+0000","last_peered":"2024-03-28T00:56:38.307049+0000","last_clean":"2024-03-28T00:56:38.307049+0000","last_became_active":"2024-03-28T00:54:36.830253+0000","last_became_peered":"2024-03-28T00:54:36.830253+0000","last_unstale":"2024-03-28T00:56:38.307049+0000","last_undegraded":"2024-03-28T00:56:38.307049+0000","last_fullsized":"2024-03-28T00:56:38.307049+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:24:43.696150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285475+0000","last_change":"2024-03-28T00:54:32.254009+0000","last_active":"2024-03-28T00:56:38.285475+0000","last_peered":"2024-03-28T00:56:38.285475+0000","last_clean":"2024-03-28T00:56:38.285475+0000","last_became_active":"2024-03-28T00:54:32.253697+0000","last_became_peered":"2024-03-28T00:54:32.253697+0000","last_unstale":"2024-03-28T00:56:38.285475+0000","last_undegraded":"2024-03-28T00:56:38.285475+0000","last_fullsized":"2024-03-28T00:56:38.285475+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:17:44.629367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308505+0000","last_change":"2024-03-28T00:54:32.812828+0000","last_active":"2024-03-28T00:56:38.308505+0000","last_peered":"2024-03-28T00:56:38.308505+0000","last_clean":"2024-03-28T00:56:38.308505+0000","last_became_active":"2024-03-28T00:54:32.811591+0000","last_became_peered":"2024-03-28T00:54:32.811591+0000","last_unstale":"2024-03-28T00:56:38.308505+0000","last_undegraded":"2024-03-28T00:56:38.308505+0000","last_fullsized":"2024-03-28T00:56:38.308505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:04.613623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303210+0000","last_change":"2024-03-28T00:54:36.823284+0000","last_active":"2024-03-28T00:56:38.303210+0000","last_peered":"2024-03-28T00:56:38.303210+0000","last_clean":"2024-03-28T00:56:38.303210+0000","last_became_active":"2024-03-28T00:54:36.822883+0000","last_became_peered":"2024-03-28T00:54:36.822883+0000","last_unstale":"2024-03-28T00:56:38.303210+0000","last_undegraded":"2024-03-28T00:56:38.303210+0000","last_fullsized":"2024-03-28T00:56:38.303210+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:37:10.266063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308755+0000","last_change":"2024-03-28T00:54:32.253946+0000","last_active":"2024-03-28T00:56:38.308755+0000","last_peered":"2024-03-28T00:56:38.308755+0000","last_clean":"2024-03-28T00:56:38.308755+0000","last_became_active":"2024-03-28T00:54:32.253650+0000","last_became_peered":"2024-03-28T00:54:32.253650+0000","last_unstale":"2024-03-28T00:56:38.308755+0000","last_undegraded":"2024-03-28T00:56:38.308755+0000","last_fullsized":"2024-03-28T00:56:38.308755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:11:44.930074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308286+0000","last_change":"2024-03-28T00:54:32.815410+0000","last_active":"2024-03-28T00:56:38.308286+0000","last_peered":"2024-03-28T00:56:38.308286+0000","last_clean":"2024-03-28T00:56:38.308286+0000","last_became_active":"2024-03-28T00:54:32.815088+0000","last_became_peered":"2024-03-28T00:54:32.815088+0000","last_unstale":"2024-03-28T00:56:38.308286+0000","last_undegraded":"2024-03-28T00:56:38.308286+0000","last_fullsized":"2024-03-28T00:56:38.308286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:16:24.862822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307742+0000","last_change":"2024-03-28T00:54:36.823182+0000","last_active":"2024-03-28T00:56:38.307742+0000","last_peered":"2024-03-28T00:56:38.307742+0000","last_clean":"2024-03-28T00:56:38.307742+0000","last_became_active":"2024-03-28T00:54:36.821446+0000","last_became_peered":"2024-03-28T00:54:36.821446+0000","last_unstale":"2024-03-28T00:56:38.307742+0000","last_undegraded":"2024-03-28T00:56:38.307742+0000","last_fullsized":"2024-03-28T00:56:38.307742+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:45:20.030213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350942+0000","last_change":"2024-03-28T00:54:32.257412+0000","last_active":"2024-03-28T00:56:38.350942+0000","last_peered":"2024-03-28T00:56:38.350942+0000","last_clean":"2024-03-28T00:56:38.350942+0000","last_became_active":"2024-03-28T00:54:32.257115+0000","last_became_peered":"2024-03-28T00:54:32.257115+0000","last_unstale":"2024-03-28T00:56:38.350942+0000","last_undegraded":"2024-03-28T00:56:38.350942+0000","last_fullsized":"2024-03-28T00:56:38.350942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:04:29.850867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308522+0000","last_change":"2024-03-28T00:54:32.805879+0000","last_active":"2024-03-28T00:56:38.308522+0000","last_peered":"2024-03-28T00:56:38.308522+0000","last_clean":"2024-03-28T00:56:38.308522+0000","last_became_active":"2024-03-28T00:54:32.805507+0000","last_became_peered":"2024-03-28T00:54:32.805507+0000","last_unstale":"2024-03-28T00:56:38.308522+0000","last_undegraded":"2024-03-28T00:56:38.308522+0000","last_fullsized":"2024-03-28T00:56:38.308522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:51:51.229085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308515+0000","last_change":"2024-03-28T00:54:32.807219+0000","last_active":"2024-03-28T00:56:38.308515+0000","last_peered":"2024-03-28T00:56:38.308515+0000","last_clean":"2024-03-28T00:56:38.308515+0000","last_became_active":"2024-03-28T00:54:32.806856+0000","last_became_peered":"2024-03-28T00:54:32.806856+0000","last_unstale":"2024-03-28T00:56:38.308515+0000","last_undegraded":"2024-03-28T00:56:38.308515+0000","last_fullsized":"2024-03-28T00:56:38.308515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:05:49.914636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285057+0000","last_change":"2024-03-28T00:55:37.569156+0000","last_active":"2024-03-28T00:56:38.285057+0000","last_peered":"2024-03-28T00:56:38.285057+0000","last_clean":"2024-03-28T00:56:38.285057+0000","last_became_active":"2024-03-28T00:55:37.568775+0000","last_became_peered":"2024-03-28T00:55:37.568775+0000","last_unstale":"2024-03-28T00:56:38.285057+0000","last_undegraded":"2024-03-28T00:56:38.285057+0000","last_fullsized":"2024-03-28T00:56:38.285057+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:00:21.458550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285006+0000","last_change":"2024-03-28T00:54:32.813921+0000","last_active":"2024-03-28T00:56:38.285006+0000","last_peered":"2024-03-28T00:56:38.285006+0000","last_clean":"2024-03-28T00:56:38.285006+0000","last_became_active":"2024-03-28T00:54:32.812877+0000","last_became_peered":"2024-03-28T00:54:32.812877+0000","last_unstale":"2024-03-28T00:56:38.285006+0000","last_undegraded":"2024-03-28T00:56:38.285006+0000","last_fullsized":"2024-03-28T00:56:38.285006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:26:54.305074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303553+0000","last_change":"2024-03-28T00:54:32.815725+0000","last_active":"2024-03-28T00:56:38.303553+0000","last_peered":"2024-03-28T00:56:38.303553+0000","last_clean":"2024-03-28T00:56:38.303553+0000","last_became_active":"2024-03-28T00:54:32.815444+0000","last_became_peered":"2024-03-28T00:54:32.815444+0000","last_unstale":"2024-03-28T00:56:38.303553+0000","last_undegraded":"2024-03-28T00:56:38.303553+0000","last_fullsized":"2024-03-28T00:56:38.303553+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:23:17.326589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.351026+0000","last_change":"2024-03-28T00:54:32.806595+0000","last_active":"2024-03-28T00:56:38.351026+0000","last_peered":"2024-03-28T00:56:38.351026+0000","last_clean":"2024-03-28T00:56:38.351026+0000","last_became_active":"2024-03-28T00:54:32.806256+0000","last_became_peered":"2024-03-28T00:54:32.806256+0000","last_unstale":"2024-03-28T00:56:38.351026+0000","last_undegraded":"2024-03-28T00:56:38.351026+0000","last_fullsized":"2024-03-28T00:56:38.351026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:04:22.551511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308462+0000","last_change":"2024-03-28T00:54:32.805000+0000","last_active":"2024-03-28T00:56:38.308462+0000","last_peered":"2024-03-28T00:56:38.308462+0000","last_clean":"2024-03-28T00:56:38.308462+0000","last_became_active":"2024-03-28T00:54:32.804637+0000","last_became_peered":"2024-03-28T00:54:32.804637+0000","last_unstale":"2024-03-28T00:56:38.308462+0000","last_undegraded":"2024-03-28T00:56:38.308462+0000","last_fullsized":"2024-03-28T00:56:38.308462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:59:46.888014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285395+0000","last_change":"2024-03-28T00:54:32.802783+0000","last_active":"2024-03-28T00:56:38.285395+0000","last_peered":"2024-03-28T00:56:38.285395+0000","last_clean":"2024-03-28T00:56:38.285395+0000","last_became_active":"2024-03-28T00:54:32.802457+0000","last_became_peered":"2024-03-28T00:54:32.802457+0000","last_unstale":"2024-03-28T00:56:38.285395+0000","last_undegraded":"2024-03-28T00:56:38.285395+0000","last_fullsized":"2024-03-28T00:56:38.285395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:28:24.146047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285531+0000","last_change":"2024-03-28T00:54:32.810179+0000","last_active":"2024-03-28T00:56:38.285531+0000","last_peered":"2024-03-28T00:56:38.285531+0000","last_clean":"2024-03-28T00:56:38.285531+0000","last_became_active":"2024-03-28T00:54:32.809009+0000","last_became_peered":"2024-03-28T00:54:32.809009+0000","last_unstale":"2024-03-28T00:56:38.285531+0000","last_undegraded":"2024-03-28T00:56:38.285531+0000","last_fullsized":"2024-03-28T00:56:38.285531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:08:44.184910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308081+0000","last_change":"2024-03-28T00:54:32.810084+0000","last_active":"2024-03-28T00:56:38.308081+0000","last_peered":"2024-03-28T00:56:38.308081+0000","last_clean":"2024-03-28T00:56:38.308081+0000","last_became_active":"2024-03-28T00:54:32.809386+0000","last_became_peered":"2024-03-28T00:54:32.809386+0000","last_unstale":"2024-03-28T00:56:38.308081+0000","last_undegraded":"2024-03-28T00:56:38.308081+0000","last_fullsized":"2024-03-28T00:56:38.308081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:49:13.754797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284532+0000","last_change":"2024-03-28T00:54:32.809933+0000","last_active":"2024-03-28T00:56:38.284532+0000","last_peered":"2024-03-28T00:56:38.284532+0000","last_clean":"2024-03-28T00:56:38.284532+0000","last_became_active":"2024-03-28T00:54:32.808974+0000","last_became_peered":"2024-03-28T00:54:32.808974+0000","last_unstale":"2024-03-28T00:56:38.284532+0000","last_undegraded":"2024-03-28T00:56:38.284532+0000","last_fullsized":"2024-03-28T00:56:38.284532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:23:59.317887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307220+0000","last_change":"2024-03-28T00:54:32.808764+0000","last_active":"2024-03-28T00:56:38.307220+0000","last_peered":"2024-03-28T00:56:38.307220+0000","last_clean":"2024-03-28T00:56:38.307220+0000","last_became_active":"2024-03-28T00:54:32.808034+0000","last_became_peered":"2024-03-28T00:54:32.808034+0000","last_unstale":"2024-03-28T00:56:38.307220+0000","last_undegraded":"2024-03-28T00:56:38.307220+0000","last_fullsized":"2024-03-28T00:56:38.307220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:27:12.762520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350903+0000","last_change":"2024-03-28T00:54:32.805470+0000","last_active":"2024-03-28T00:56:38.350903+0000","last_peered":"2024-03-28T00:56:38.350903+0000","last_clean":"2024-03-28T00:56:38.350903+0000","last_became_active":"2024-03-28T00:54:32.805146+0000","last_became_peered":"2024-03-28T00:54:32.805146+0000","last_unstale":"2024-03-28T00:56:38.350903+0000","last_undegraded":"2024-03-28T00:56:38.350903+0000","last_fullsized":"2024-03-28T00:56:38.350903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:24:14.297966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308078+0000","last_change":"2024-03-28T00:55:37.566696+0000","last_active":"2024-03-28T00:56:38.308078+0000","last_peered":"2024-03-28T00:56:38.308078+0000","last_clean":"2024-03-28T00:56:38.308078+0000","last_became_active":"2024-03-28T00:55:37.566334+0000","last_became_peered":"2024-03-28T00:55:37.566334+0000","last_unstale":"2024-03-28T00:56:38.308078+0000","last_undegraded":"2024-03-28T00:56:38.308078+0000","last_fullsized":"2024-03-28T00:56:38.308078+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:41:37.752201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282084,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":77906,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":716,"num_read_kb":439,"num_write":175,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450560,"data_stored":226947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153,"ondisk_log_size":153,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383720,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":529042,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645358,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32336,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712816,"statfs":{"total":95995035648,"available":95961923584,"internally_reserved":0,"allocated":1187840,"data_stored":1189353,"data_compressed":65432,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808882,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":533352,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070520,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":529154,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332156,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1097728,"data_stored":1125226,"data_compressed":65432,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495681,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":524432,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":591335,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1052672,"data_stored":525820,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":599988,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182480,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1040384,"data_stored":524626,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920844,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":529082,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":1121122,"data_compressed":65432,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34218,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34218,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34218,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":66993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":75646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":74643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T00:56:49.333 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T00:56:49.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.468+0000 7f7269bb2640 1 -- 172.21.15.169:0/887606369 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7264103340 msgr2=0x7f7264109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:49.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.468+0000 7f7269bb2640 1 --2- 172.21.15.169:0/887606369 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7264103340 0x7f7264109bd0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f725c009cd0 tx=0x7f725c02c0f0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.468+0000 7f7269bb2640 1 -- 172.21.15.169:0/887606369 shutdown_connections 2024-03-28T00:56:49.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.468+0000 7f7269bb2640 1 --2- 172.21.15.169:0/887606369 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7264103340 0x7f7264109bd0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.468+0000 7f7269bb2640 1 --2- 172.21.15.169:0/887606369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7264102980 0x7f7264102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.468+0000 7f7269bb2640 1 --2- 172.21.15.169:0/887606369 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72640ff9b0 0x7f72640ffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.468+0000 7f7269bb2640 1 -- 172.21.15.169:0/887606369 >> 172.21.15.169:0/887606369 conn(0x7f72640fb2e0 msgr2=0x7f72640fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:49.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.468+0000 7f7269bb2640 1 -- 172.21.15.169:0/887606369 shutdown_connections 2024-03-28T00:56:49.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.468+0000 7f7269bb2640 1 -- 172.21.15.169:0/887606369 wait complete. 2024-03-28T00:56:49.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.469+0000 7f7269bb2640 1 Processor -- start 2024-03-28T00:56:49.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.469+0000 7f7269bb2640 1 -- start start 2024-03-28T00:56:49.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.469+0000 7f7269bb2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72640ff9b0 0x7f72641979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:49.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.469+0000 7f7269bb2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7264102980 0x7f7264197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:49.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.469+0000 7f7269bb2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7264103340 0x7f726419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:49.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.469+0000 7f7269bb2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726410d390 con 0x7f7264102980 2024-03-28T00:56:49.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.469+0000 7f7269bb2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726410d210 con 0x7f7264103340 2024-03-28T00:56:49.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.469+0000 7f7269bb2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726410d510 con 0x7f72640ff9b0 2024-03-28T00:56:49.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.469+0000 7f725affd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7264102980 0x7f7264197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:49.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.469+0000 7f725affd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7264102980 0x7f7264197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34824/0 (socket says 172.21.15.169:34824) 2024-03-28T00:56:49.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.469+0000 7f725affd640 1 -- 172.21.15.169:0/1926242633 learned_addr learned my addr 172.21.15.169:0/1926242633 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:49.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.469+0000 7f7263fff640 1 --2- 172.21.15.169:0/1926242633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7264103340 0x7f726419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:49.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.469+0000 7f72637fe640 1 --2- 172.21.15.169:0/1926242633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72640ff9b0 0x7f72641979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:49.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.470+0000 7f725affd640 1 -- 172.21.15.169:0/1926242633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72640ff9b0 msgr2=0x7f72641979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:49.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.470+0000 7f725affd640 1 --2- 172.21.15.169:0/1926242633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72640ff9b0 0x7f72641979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.470+0000 7f725affd640 1 -- 172.21.15.169:0/1926242633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7264103340 msgr2=0x7f726419ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:49.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.470+0000 7f725affd640 1 --2- 172.21.15.169:0/1926242633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7264103340 0x7f726419ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.470+0000 7f725affd640 1 -- 172.21.15.169:0/1926242633 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726419f5f0 con 0x7f7264102980 2024-03-28T00:56:49.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.470+0000 7f7263fff640 1 --2- 172.21.15.169:0/1926242633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7264103340 0x7f726419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:49.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.470+0000 7f725affd640 1 --2- 172.21.15.169:0/1926242633 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7264102980 0x7f7264197f10 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f724c00e9f0 tx=0x7f724c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:49.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.470+0000 7f72617fa640 1 -- 172.21.15.169:0/1926242633 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c00cd80 con 0x7f7264102980 2024-03-28T00:56:49.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.470+0000 7f7269bb2640 1 -- 172.21.15.169:0/1926242633 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f726419f8e0 con 0x7f7264102980 2024-03-28T00:56:49.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.470+0000 7f72617fa640 1 -- 172.21.15.169:0/1926242633 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f724c005500 con 0x7f7264102980 2024-03-28T00:56:49.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.470+0000 7f72617fa640 1 -- 172.21.15.169:0/1926242633 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c010690 con 0x7f7264102980 2024-03-28T00:56:49.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.470+0000 7f7269bb2640 1 -- 172.21.15.169:0/1926242633 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f726419fde0 con 0x7f7264102980 2024-03-28T00:56:49.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.471+0000 7f7269bb2640 1 -- 172.21.15.169:0/1926242633 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7264108ab0 con 0x7f7264102980 2024-03-28T00:56:49.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.472+0000 7f72617fa640 1 -- 172.21.15.169:0/1926242633 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f724c0046c0 con 0x7f7264102980 2024-03-28T00:56:49.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.473+0000 7f72617fa640 1 --2- 172.21.15.169:0/1926242633 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f72500a2890 0x7f72500a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:49.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.473+0000 7f72637fe640 1 --2- 172.21.15.169:0/1926242633 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f72500a2890 0x7f72500a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:49.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.473+0000 7f72617fa640 1 -- 172.21.15.169:0/1926242633 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f724c0cf5b0 con 0x7f7264102980 2024-03-28T00:56:49.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.473+0000 7f72637fe640 1 --2- 172.21.15.169:0/1926242633 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f72500a2890 0x7f72500a4d50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7264069af0 tx=0x7f7254005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:49.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.476+0000 7f72617fa640 1 -- 172.21.15.169:0/1926242633 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f724c053740 con 0x7f7264102980 2024-03-28T00:56:49.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:49 smithi174 ceph-mon[36060]: pgmap v360: 129 pgs: 129 active+clean; 653 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-28T00:56:49.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:49 smithi174 ceph-mon[36060]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-28T00:56:49.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:49 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2626492422' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:56:49.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:49 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1777128797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:56:49.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:49 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/556264606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:56:49.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.649+0000 7f7269bb2640 1 -- 172.21.15.169:0/1926242633 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7264061ae0 con 0x7f72500a2890 2024-03-28T00:56:49.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.658+0000 7f72617fa640 1 -- 172.21.15.169:0/1926242633 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340752 (secure 0 0 0) 0x7f7264061ae0 con 0x7f72500a2890 2024-03-28T00:56:49.665 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:49.667 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-28T00:56:49.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:49 smithi177 ceph-mon[36206]: pgmap v360: 129 pgs: 129 active+clean; 653 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-28T00:56:49.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:49 smithi177 ceph-mon[36206]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-28T00:56:49.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:49 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2626492422' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:56:49.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:49 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1777128797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:56:49.668 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:49 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/556264606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:56:49.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.662+0000 7f7269bb2640 1 -- 172.21.15.169:0/1926242633 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f72500a2890 msgr2=0x7f72500a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:49.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.662+0000 7f7269bb2640 1 --2- 172.21.15.169:0/1926242633 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f72500a2890 0x7f72500a4d50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7264069af0 tx=0x7f7254005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:49.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.662+0000 7f7269bb2640 1 -- 172.21.15.169:0/1926242633 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7264102980 msgr2=0x7f7264197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:49.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.662+0000 7f7269bb2640 1 --2- 172.21.15.169:0/1926242633 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7264102980 0x7f7264197f10 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f724c00e9f0 tx=0x7f724c00edf0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.662+0000 7f7269bb2640 1 -- 172.21.15.169:0/1926242633 shutdown_connections 2024-03-28T00:56:49.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.662+0000 7f7269bb2640 1 --2- 172.21.15.169:0/1926242633 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f72500a2890 0x7f72500a4d50 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.662+0000 7f7269bb2640 1 --2- 172.21.15.169:0/1926242633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7264103340 0x7f726419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.662+0000 7f7269bb2640 1 --2- 172.21.15.169:0/1926242633 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7264102980 0x7f7264197f10 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.662+0000 7f7269bb2640 1 --2- 172.21.15.169:0/1926242633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72640ff9b0 0x7f72641979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.662+0000 7f7269bb2640 1 -- 172.21.15.169:0/1926242633 >> 172.21.15.169:0/1926242633 conn(0x7f72640fb2e0 msgr2=0x7f7264105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:49.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.662+0000 7f7269bb2640 1 -- 172.21.15.169:0/1926242633 shutdown_connections 2024-03-28T00:56:49.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.662+0000 7f7269bb2640 1 -- 172.21.15.169:0/1926242633 wait complete. 2024-03-28T00:56:49.676 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-03-28T00:56:48.116387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":668508,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":842,"num_read_kb":548,"num_write":358,"num_write_kb":3074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":792,"ondisk_log_size":792,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357676,"kb_used_data":12616,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584148,"statfs":{"total":1151940427776,"available":1151574167552,"internally_reserved":0,"allocated":12918784,"data_stored":8322532,"data_compressed":462210,"data_compressed_allocated":626688,"data_compressed_original":2752512,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6771,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":63,"num_read_kb":38,"num_write":8,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002486"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308755+0000","last_change":"2024-03-28T00:54:32.809951+0000","last_active":"2024-03-28T00:56:38.308755+0000","last_peered":"2024-03-28T00:56:38.308755+0000","last_clean":"2024-03-28T00:56:38.308755+0000","last_became_active":"2024-03-28T00:54:32.808871+0000","last_became_peered":"2024-03-28T00:54:32.808871+0000","last_unstale":"2024-03-28T00:56:38.308755+0000","last_undegraded":"2024-03-28T00:56:38.308755+0000","last_fullsized":"2024-03-28T00:56:38.308755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:15:17.152487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303536+0000","last_change":"2024-03-28T00:54:32.816707+0000","last_active":"2024-03-28T00:56:38.303536+0000","last_peered":"2024-03-28T00:56:38.303536+0000","last_clean":"2024-03-28T00:56:38.303536+0000","last_became_active":"2024-03-28T00:54:32.816000+0000","last_became_peered":"2024-03-28T00:54:32.816000+0000","last_unstale":"2024-03-28T00:56:38.303536+0000","last_undegraded":"2024-03-28T00:56:38.303536+0000","last_fullsized":"2024-03-28T00:56:38.303536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:56:19.532808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303346+0000","last_change":"2024-03-28T00:54:32.817023+0000","last_active":"2024-03-28T00:56:38.303346+0000","last_peered":"2024-03-28T00:56:38.303346+0000","last_clean":"2024-03-28T00:56:38.303346+0000","last_became_active":"2024-03-28T00:54:32.816712+0000","last_became_peered":"2024-03-28T00:54:32.816712+0000","last_unstale":"2024-03-28T00:56:38.303346+0000","last_undegraded":"2024-03-28T00:56:38.303346+0000","last_fullsized":"2024-03-28T00:56:38.303346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:13:25.334294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285135+0000","last_change":"2024-03-28T00:54:32.807259+0000","last_active":"2024-03-28T00:56:38.285135+0000","last_peered":"2024-03-28T00:56:38.285135+0000","last_clean":"2024-03-28T00:56:38.285135+0000","last_became_active":"2024-03-28T00:54:32.806944+0000","last_became_peered":"2024-03-28T00:54:32.806944+0000","last_unstale":"2024-03-28T00:56:38.285135+0000","last_undegraded":"2024-03-28T00:56:38.285135+0000","last_fullsized":"2024-03-28T00:56:38.285135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:46:04.852351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308386+0000","last_change":"2024-03-28T00:54:32.807613+0000","last_active":"2024-03-28T00:56:38.308386+0000","last_peered":"2024-03-28T00:56:38.308386+0000","last_clean":"2024-03-28T00:56:38.308386+0000","last_became_active":"2024-03-28T00:54:32.807280+0000","last_became_peered":"2024-03-28T00:54:32.807280+0000","last_unstale":"2024-03-28T00:56:38.308386+0000","last_undegraded":"2024-03-28T00:56:38.308386+0000","last_fullsized":"2024-03-28T00:56:38.308386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:56:37.133675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286708+0000","last_change":"2024-03-28T00:54:32.806804+0000","last_active":"2024-03-28T00:56:38.286708+0000","last_peered":"2024-03-28T00:56:38.286708+0000","last_clean":"2024-03-28T00:56:38.286708+0000","last_became_active":"2024-03-28T00:54:32.805343+0000","last_became_peered":"2024-03-28T00:54:32.805343+0000","last_unstale":"2024-03-28T00:56:38.286708+0000","last_undegraded":"2024-03-28T00:56:38.286708+0000","last_fullsized":"2024-03-28T00:56:38.286708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:31:44.849926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307907+0000","last_change":"2024-03-28T00:55:37.563371+0000","last_active":"2024-03-28T00:56:38.307907+0000","last_peered":"2024-03-28T00:56:38.307907+0000","last_clean":"2024-03-28T00:56:38.307907+0000","last_became_active":"2024-03-28T00:55:37.563000+0000","last_became_peered":"2024-03-28T00:55:37.563000+0000","last_unstale":"2024-03-28T00:56:38.307907+0000","last_undegraded":"2024-03-28T00:56:38.307907+0000","last_fullsized":"2024-03-28T00:56:38.307907+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:56:58.782502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.309667+0000","last_change":"2024-03-28T00:56:38.309328+0000","last_active":"2024-03-28T00:56:38.309667+0000","last_peered":"2024-03-28T00:56:38.309667+0000","last_clean":"2024-03-28T00:56:38.309667+0000","last_became_active":"2024-03-28T00:56:38.308978+0000","last_became_peered":"2024-03-28T00:56:38.308978+0000","last_unstale":"2024-03-28T00:56:38.309667+0000","last_undegraded":"2024-03-28T00:56:38.309667+0000","last_fullsized":"2024-03-28T00:56:38.309667+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:45:52.643890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307485+0000","last_change":"2024-03-28T00:54:32.800667+0000","last_active":"2024-03-28T00:56:38.307485+0000","last_peered":"2024-03-28T00:56:38.307485+0000","last_clean":"2024-03-28T00:56:38.307485+0000","last_became_active":"2024-03-28T00:54:32.800332+0000","last_became_peered":"2024-03-28T00:54:32.800332+0000","last_unstale":"2024-03-28T00:56:38.307485+0000","last_undegraded":"2024-03-28T00:56:38.307485+0000","last_fullsized":"2024-03-28T00:56:38.307485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:41:27.577871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.288488+0000","last_change":"2024-03-28T00:56:38.288488+0000","last_active":"2024-03-28T00:56:38.288488+0000","last_peered":"2024-03-28T00:56:38.288488+0000","last_clean":"2024-03-28T00:56:38.288488+0000","last_became_active":"2024-03-28T00:56:38.288164+0000","last_became_peered":"2024-03-28T00:56:38.288164+0000","last_unstale":"2024-03-28T00:56:38.288488+0000","last_undegraded":"2024-03-28T00:56:38.288488+0000","last_fullsized":"2024-03-28T00:56:38.288488+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:50:33.312420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308456+0000","last_change":"2024-03-28T00:54:32.811543+0000","last_active":"2024-03-28T00:56:38.308456+0000","last_peered":"2024-03-28T00:56:38.308456+0000","last_clean":"2024-03-28T00:56:38.308456+0000","last_became_active":"2024-03-28T00:54:32.811210+0000","last_became_peered":"2024-03-28T00:54:32.811210+0000","last_unstale":"2024-03-28T00:56:38.308456+0000","last_undegraded":"2024-03-28T00:56:38.308456+0000","last_fullsized":"2024-03-28T00:56:38.308456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:47:19.539107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285098+0000","last_change":"2024-03-28T00:54:32.812648+0000","last_active":"2024-03-28T00:56:38.285098+0000","last_peered":"2024-03-28T00:56:38.285098+0000","last_clean":"2024-03-28T00:56:38.285098+0000","last_became_active":"2024-03-28T00:54:32.812340+0000","last_became_peered":"2024-03-28T00:54:32.812340+0000","last_unstale":"2024-03-28T00:56:38.285098+0000","last_undegraded":"2024-03-28T00:56:38.285098+0000","last_fullsized":"2024-03-28T00:56:38.285098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:20:45.449953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350843+0000","last_change":"2024-03-28T00:54:32.801807+0000","last_active":"2024-03-28T00:56:38.350843+0000","last_peered":"2024-03-28T00:56:38.350843+0000","last_clean":"2024-03-28T00:56:38.350843+0000","last_became_active":"2024-03-28T00:54:32.801498+0000","last_became_peered":"2024-03-28T00:54:32.801498+0000","last_unstale":"2024-03-28T00:56:38.350843+0000","last_undegraded":"2024-03-28T00:56:38.350843+0000","last_fullsized":"2024-03-28T00:56:38.350843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:35:35.827856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303105+0000","last_change":"2024-03-28T00:54:32.818033+0000","last_active":"2024-03-28T00:56:38.303105+0000","last_peered":"2024-03-28T00:56:38.303105+0000","last_clean":"2024-03-28T00:56:38.303105+0000","last_became_active":"2024-03-28T00:54:32.817309+0000","last_became_peered":"2024-03-28T00:54:32.817309+0000","last_unstale":"2024-03-28T00:56:38.303105+0000","last_undegraded":"2024-03-28T00:56:38.303105+0000","last_fullsized":"2024-03-28T00:56:38.303105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:39:42.682780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303139+0000","last_change":"2024-03-28T00:54:32.815224+0000","last_active":"2024-03-28T00:56:38.303139+0000","last_peered":"2024-03-28T00:56:38.303139+0000","last_clean":"2024-03-28T00:56:38.303139+0000","last_became_active":"2024-03-28T00:54:32.814936+0000","last_became_peered":"2024-03-28T00:54:32.814936+0000","last_unstale":"2024-03-28T00:56:38.303139+0000","last_undegraded":"2024-03-28T00:56:38.303139+0000","last_fullsized":"2024-03-28T00:56:38.303139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:08:40.200393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.306895+0000","last_change":"2024-03-28T00:54:32.807342+0000","last_active":"2024-03-28T00:56:38.306895+0000","last_peered":"2024-03-28T00:56:38.306895+0000","last_clean":"2024-03-28T00:56:38.306895+0000","last_became_active":"2024-03-28T00:54:32.806978+0000","last_became_peered":"2024-03-28T00:54:32.806978+0000","last_unstale":"2024-03-28T00:56:38.306895+0000","last_undegraded":"2024-03-28T00:56:38.306895+0000","last_fullsized":"2024-03-28T00:56:38.306895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:17:58.599693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308639+0000","last_change":"2024-03-28T00:54:32.810036+0000","last_active":"2024-03-28T00:56:38.308639+0000","last_peered":"2024-03-28T00:56:38.308639+0000","last_clean":"2024-03-28T00:56:38.308639+0000","last_became_active":"2024-03-28T00:54:32.809438+0000","last_became_peered":"2024-03-28T00:54:32.809438+0000","last_unstale":"2024-03-28T00:56:38.308639+0000","last_undegraded":"2024-03-28T00:56:38.308639+0000","last_fullsized":"2024-03-28T00:56:38.308639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:18:50.447474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286485+0000","last_change":"2024-03-28T00:54:32.815398+0000","last_active":"2024-03-28T00:56:38.286485+0000","last_peered":"2024-03-28T00:56:38.286485+0000","last_clean":"2024-03-28T00:56:38.286485+0000","last_became_active":"2024-03-28T00:54:32.815045+0000","last_became_peered":"2024-03-28T00:54:32.815045+0000","last_unstale":"2024-03-28T00:56:38.286485+0000","last_undegraded":"2024-03-28T00:56:38.286485+0000","last_fullsized":"2024-03-28T00:56:38.286485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:40:29.060217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285333+0000","last_change":"2024-03-28T00:54:32.805736+0000","last_active":"2024-03-28T00:56:38.285333+0000","last_peered":"2024-03-28T00:56:38.285333+0000","last_clean":"2024-03-28T00:56:38.285333+0000","last_became_active":"2024-03-28T00:54:32.805435+0000","last_became_peered":"2024-03-28T00:54:32.805435+0000","last_unstale":"2024-03-28T00:56:38.285333+0000","last_undegraded":"2024-03-28T00:56:38.285333+0000","last_fullsized":"2024-03-28T00:56:38.285333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:24:30.893583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285426+0000","last_change":"2024-03-28T00:54:36.824531+0000","last_active":"2024-03-28T00:56:38.285426+0000","last_peered":"2024-03-28T00:56:38.285426+0000","last_clean":"2024-03-28T00:56:38.285426+0000","last_became_active":"2024-03-28T00:54:36.824123+0000","last_became_peered":"2024-03-28T00:54:36.824123+0000","last_unstale":"2024-03-28T00:56:38.285426+0000","last_undegraded":"2024-03-28T00:56:38.285426+0000","last_fullsized":"2024-03-28T00:56:38.285426+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:10:47.503171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286759+0000","last_change":"2024-03-28T00:54:32.256226+0000","last_active":"2024-03-28T00:56:38.286759+0000","last_peered":"2024-03-28T00:56:38.286759+0000","last_clean":"2024-03-28T00:56:38.286759+0000","last_became_active":"2024-03-28T00:54:32.255921+0000","last_became_peered":"2024-03-28T00:54:32.255921+0000","last_unstale":"2024-03-28T00:56:38.286759+0000","last_undegraded":"2024-03-28T00:56:38.286759+0000","last_fullsized":"2024-03-28T00:56:38.286759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:56:04.196311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284800+0000","last_change":"2024-03-28T00:54:32.808248+0000","last_active":"2024-03-28T00:56:38.284800+0000","last_peered":"2024-03-28T00:56:38.284800+0000","last_clean":"2024-03-28T00:56:38.284800+0000","last_became_active":"2024-03-28T00:54:32.807923+0000","last_became_peered":"2024-03-28T00:54:32.807923+0000","last_unstale":"2024-03-28T00:56:38.284800+0000","last_undegraded":"2024-03-28T00:56:38.284800+0000","last_fullsized":"2024-03-28T00:56:38.284800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:59:56.002564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284703+0000","last_change":"2024-03-28T00:54:36.830556+0000","last_active":"2024-03-28T00:56:38.284703+0000","last_peered":"2024-03-28T00:56:38.284703+0000","last_clean":"2024-03-28T00:56:38.284703+0000","last_became_active":"2024-03-28T00:54:36.830151+0000","last_became_peered":"2024-03-28T00:54:36.830151+0000","last_unstale":"2024-03-28T00:56:38.284703+0000","last_undegraded":"2024-03-28T00:56:38.284703+0000","last_fullsized":"2024-03-28T00:56:38.284703+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:19:23.073410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285141+0000","last_change":"2024-03-28T00:54:32.260635+0000","last_active":"2024-03-28T00:56:38.285141+0000","last_peered":"2024-03-28T00:56:38.285141+0000","last_clean":"2024-03-28T00:56:38.285141+0000","last_became_active":"2024-03-28T00:54:32.260084+0000","last_became_peered":"2024-03-28T00:54:32.260084+0000","last_unstale":"2024-03-28T00:56:38.285141+0000","last_undegraded":"2024-03-28T00:56:38.285141+0000","last_fullsized":"2024-03-28T00:56:38.285141+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:06.858519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303455+0000","last_change":"2024-03-28T00:54:32.809901+0000","last_active":"2024-03-28T00:56:38.303455+0000","last_peered":"2024-03-28T00:56:38.303455+0000","last_clean":"2024-03-28T00:56:38.303455+0000","last_became_active":"2024-03-28T00:54:32.809557+0000","last_became_peered":"2024-03-28T00:54:32.809557+0000","last_unstale":"2024-03-28T00:56:38.303455+0000","last_undegraded":"2024-03-28T00:56:38.303455+0000","last_fullsized":"2024-03-28T00:56:38.303455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:05:31.363487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284824+0000","last_change":"2024-03-28T00:54:36.824661+0000","last_active":"2024-03-28T00:56:38.284824+0000","last_peered":"2024-03-28T00:56:38.284824+0000","last_clean":"2024-03-28T00:56:38.284824+0000","last_became_active":"2024-03-28T00:54:36.824223+0000","last_became_peered":"2024-03-28T00:54:36.824223+0000","last_unstale":"2024-03-28T00:56:38.284824+0000","last_undegraded":"2024-03-28T00:56:38.284824+0000","last_fullsized":"2024-03-28T00:56:38.284824+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:02:25.509248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308563+0000","last_change":"2024-03-28T00:54:32.256436+0000","last_active":"2024-03-28T00:56:38.308563+0000","last_peered":"2024-03-28T00:56:38.308563+0000","last_clean":"2024-03-28T00:56:38.308563+0000","last_became_active":"2024-03-28T00:54:32.256069+0000","last_became_peered":"2024-03-28T00:54:32.256069+0000","last_unstale":"2024-03-28T00:56:38.308563+0000","last_undegraded":"2024-03-28T00:56:38.308563+0000","last_fullsized":"2024-03-28T00:56:38.308563+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:43.688192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307212+0000","last_change":"2024-03-28T00:54:32.799269+0000","last_active":"2024-03-28T00:56:38.307212+0000","last_peered":"2024-03-28T00:56:38.307212+0000","last_clean":"2024-03-28T00:56:38.307212+0000","last_became_active":"2024-03-28T00:54:32.798848+0000","last_became_peered":"2024-03-28T00:54:32.798848+0000","last_unstale":"2024-03-28T00:56:38.307212+0000","last_undegraded":"2024-03-28T00:56:38.307212+0000","last_fullsized":"2024-03-28T00:56:38.307212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:56:20.318466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284703+0000","last_change":"2024-03-28T00:54:36.823153+0000","last_active":"2024-03-28T00:56:38.284703+0000","last_peered":"2024-03-28T00:56:38.284703+0000","last_clean":"2024-03-28T00:56:38.284703+0000","last_became_active":"2024-03-28T00:54:36.822817+0000","last_became_peered":"2024-03-28T00:54:36.822817+0000","last_unstale":"2024-03-28T00:56:38.284703+0000","last_undegraded":"2024-03-28T00:56:38.284703+0000","last_fullsized":"2024-03-28T00:56:38.284703+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:20:23.001599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308631+0000","last_change":"2024-03-28T00:54:32.250439+0000","last_active":"2024-03-28T00:56:38.308631+0000","last_peered":"2024-03-28T00:56:38.308631+0000","last_clean":"2024-03-28T00:56:38.308631+0000","last_became_active":"2024-03-28T00:54:32.250006+0000","last_became_peered":"2024-03-28T00:54:32.250006+0000","last_unstale":"2024-03-28T00:56:38.308631+0000","last_undegraded":"2024-03-28T00:56:38.308631+0000","last_fullsized":"2024-03-28T00:56:38.308631+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:31:38.957429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308661+0000","last_change":"2024-03-28T00:54:32.806888+0000","last_active":"2024-03-28T00:56:38.308661+0000","last_peered":"2024-03-28T00:56:38.308661+0000","last_clean":"2024-03-28T00:56:38.308661+0000","last_became_active":"2024-03-28T00:54:32.806581+0000","last_became_peered":"2024-03-28T00:54:32.806581+0000","last_unstale":"2024-03-28T00:56:38.308661+0000","last_undegraded":"2024-03-28T00:56:38.308661+0000","last_fullsized":"2024-03-28T00:56:38.308661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:25:52.331819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285199+0000","last_change":"2024-03-28T00:54:36.824041+0000","last_active":"2024-03-28T00:56:38.285199+0000","last_peered":"2024-03-28T00:56:38.285199+0000","last_clean":"2024-03-28T00:56:38.285199+0000","last_became_active":"2024-03-28T00:54:36.823658+0000","last_became_peered":"2024-03-28T00:54:36.823658+0000","last_unstale":"2024-03-28T00:56:38.285199+0000","last_undegraded":"2024-03-28T00:56:38.285199+0000","last_fullsized":"2024-03-28T00:56:38.285199+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:11:03.236009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'9","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:45.395923+0000","last_change":"2024-03-28T00:54:32.257534+0000","last_active":"2024-03-28T00:56:45.395923+0000","last_peered":"2024-03-28T00:56:45.395923+0000","last_clean":"2024-03-28T00:56:45.395923+0000","last_became_active":"2024-03-28T00:54:32.256404+0000","last_became_peered":"2024-03-28T00:54:32.256404+0000","last_unstale":"2024-03-28T00:56:45.395923+0000","last_undegraded":"2024-03-28T00:56:45.395923+0000","last_fullsized":"2024-03-28T00:56:45.395923+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:39:38.329799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1853,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308137+0000","last_change":"2024-03-28T00:54:32.806099+0000","last_active":"2024-03-28T00:56:38.308137+0000","last_peered":"2024-03-28T00:56:38.308137+0000","last_clean":"2024-03-28T00:56:38.308137+0000","last_became_active":"2024-03-28T00:54:32.805811+0000","last_became_peered":"2024-03-28T00:54:32.805811+0000","last_unstale":"2024-03-28T00:56:38.308137+0000","last_undegraded":"2024-03-28T00:56:38.308137+0000","last_fullsized":"2024-03-28T00:56:38.308137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:42:32.820245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285325+0000","last_change":"2024-03-28T00:54:36.827198+0000","last_active":"2024-03-28T00:56:38.285325+0000","last_peered":"2024-03-28T00:56:38.285325+0000","last_clean":"2024-03-28T00:56:38.285325+0000","last_became_active":"2024-03-28T00:54:36.826804+0000","last_became_peered":"2024-03-28T00:54:36.826804+0000","last_unstale":"2024-03-28T00:56:38.285325+0000","last_undegraded":"2024-03-28T00:56:38.285325+0000","last_fullsized":"2024-03-28T00:56:38.285325+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:38:11.030404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286690+0000","last_change":"2024-03-28T00:54:32.249394+0000","last_active":"2024-03-28T00:56:38.286690+0000","last_peered":"2024-03-28T00:56:38.286690+0000","last_clean":"2024-03-28T00:56:38.286690+0000","last_became_active":"2024-03-28T00:54:32.249089+0000","last_became_peered":"2024-03-28T00:54:32.249089+0000","last_unstale":"2024-03-28T00:56:38.286690+0000","last_undegraded":"2024-03-28T00:56:38.286690+0000","last_fullsized":"2024-03-28T00:56:38.286690+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:05:01.720053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307883+0000","last_change":"2024-03-28T00:54:32.809133+0000","last_active":"2024-03-28T00:56:38.307883+0000","last_peered":"2024-03-28T00:56:38.307883+0000","last_clean":"2024-03-28T00:56:38.307883+0000","last_became_active":"2024-03-28T00:54:32.808841+0000","last_became_peered":"2024-03-28T00:54:32.808841+0000","last_unstale":"2024-03-28T00:56:38.307883+0000","last_undegraded":"2024-03-28T00:56:38.307883+0000","last_fullsized":"2024-03-28T00:56:38.307883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:19:37.859542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308573+0000","last_change":"2024-03-28T00:54:36.824483+0000","last_active":"2024-03-28T00:56:38.308573+0000","last_peered":"2024-03-28T00:56:38.308573+0000","last_clean":"2024-03-28T00:56:38.308573+0000","last_became_active":"2024-03-28T00:54:36.823973+0000","last_became_peered":"2024-03-28T00:54:36.823973+0000","last_unstale":"2024-03-28T00:56:38.308573+0000","last_undegraded":"2024-03-28T00:56:38.308573+0000","last_fullsized":"2024-03-28T00:56:38.308573+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:01:06.251119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350674+0000","last_change":"2024-03-28T00:54:32.251287+0000","last_active":"2024-03-28T00:56:38.350674+0000","last_peered":"2024-03-28T00:56:38.350674+0000","last_clean":"2024-03-28T00:56:38.350674+0000","last_became_active":"2024-03-28T00:54:32.250904+0000","last_became_peered":"2024-03-28T00:54:32.250904+0000","last_unstale":"2024-03-28T00:56:38.350674+0000","last_undegraded":"2024-03-28T00:56:38.350674+0000","last_fullsized":"2024-03-28T00:56:38.350674+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:18:09.051056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284458+0000","last_change":"2024-03-28T00:54:32.803124+0000","last_active":"2024-03-28T00:56:38.284458+0000","last_peered":"2024-03-28T00:56:38.284458+0000","last_clean":"2024-03-28T00:56:38.284458+0000","last_became_active":"2024-03-28T00:54:32.802825+0000","last_became_peered":"2024-03-28T00:54:32.802825+0000","last_unstale":"2024-03-28T00:56:38.284458+0000","last_undegraded":"2024-03-28T00:56:38.284458+0000","last_fullsized":"2024-03-28T00:56:38.284458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:34:21.825314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303447+0000","last_change":"2024-03-28T00:54:36.832151+0000","last_active":"2024-03-28T00:56:38.303447+0000","last_peered":"2024-03-28T00:56:38.303447+0000","last_clean":"2024-03-28T00:56:38.303447+0000","last_became_active":"2024-03-28T00:54:36.831748+0000","last_became_peered":"2024-03-28T00:54:36.831748+0000","last_unstale":"2024-03-28T00:56:38.303447+0000","last_undegraded":"2024-03-28T00:56:38.303447+0000","last_fullsized":"2024-03-28T00:56:38.303447+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:45:32.284386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307890+0000","last_change":"2024-03-28T00:54:32.259048+0000","last_active":"2024-03-28T00:56:38.307890+0000","last_peered":"2024-03-28T00:56:38.307890+0000","last_clean":"2024-03-28T00:56:38.307890+0000","last_became_active":"2024-03-28T00:54:32.258038+0000","last_became_peered":"2024-03-28T00:54:32.258038+0000","last_unstale":"2024-03-28T00:56:38.307890+0000","last_undegraded":"2024-03-28T00:56:38.307890+0000","last_fullsized":"2024-03-28T00:56:38.307890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:45:23.525516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286245+0000","last_change":"2024-03-28T00:54:32.809771+0000","last_active":"2024-03-28T00:56:38.286245+0000","last_peered":"2024-03-28T00:56:38.286245+0000","last_clean":"2024-03-28T00:56:38.286245+0000","last_became_active":"2024-03-28T00:54:32.809467+0000","last_became_peered":"2024-03-28T00:54:32.809467+0000","last_unstale":"2024-03-28T00:56:38.286245+0000","last_undegraded":"2024-03-28T00:56:38.286245+0000","last_fullsized":"2024-03-28T00:56:38.286245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:30:15.440202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284418+0000","last_change":"2024-03-28T00:54:36.823313+0000","last_active":"2024-03-28T00:56:38.284418+0000","last_peered":"2024-03-28T00:56:38.284418+0000","last_clean":"2024-03-28T00:56:38.284418+0000","last_became_active":"2024-03-28T00:54:36.822923+0000","last_became_peered":"2024-03-28T00:54:36.822923+0000","last_unstale":"2024-03-28T00:56:38.284418+0000","last_undegraded":"2024-03-28T00:56:38.284418+0000","last_fullsized":"2024-03-28T00:56:38.284418+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:22:21.125028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'5","reported_seq":153,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:45.395588+0000","last_change":"2024-03-28T00:54:32.255084+0000","last_active":"2024-03-28T00:56:45.395588+0000","last_peered":"2024-03-28T00:56:45.395588+0000","last_clean":"2024-03-28T00:56:45.395588+0000","last_became_active":"2024-03-28T00:54:32.254744+0000","last_became_peered":"2024-03-28T00:54:32.254744+0000","last_unstale":"2024-03-28T00:56:45.395588+0000","last_undegraded":"2024-03-28T00:56:45.395588+0000","last_fullsized":"2024-03-28T00:56:45.395588+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:32:03.906826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":115,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307394+0000","last_change":"2024-03-28T00:54:32.810050+0000","last_active":"2024-03-28T00:56:38.307394+0000","last_peered":"2024-03-28T00:56:38.307394+0000","last_clean":"2024-03-28T00:56:38.307394+0000","last_became_active":"2024-03-28T00:54:32.809731+0000","last_became_peered":"2024-03-28T00:54:32.809731+0000","last_unstale":"2024-03-28T00:56:38.307394+0000","last_undegraded":"2024-03-28T00:56:38.307394+0000","last_fullsized":"2024-03-28T00:56:38.307394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:20:11.876731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307671+0000","last_change":"2024-03-28T00:55:37.566900+0000","last_active":"2024-03-28T00:56:38.307671+0000","last_peered":"2024-03-28T00:56:38.307671+0000","last_clean":"2024-03-28T00:56:38.307671+0000","last_became_active":"2024-03-28T00:55:37.566491+0000","last_became_peered":"2024-03-28T00:55:37.566491+0000","last_unstale":"2024-03-28T00:56:38.307671+0000","last_undegraded":"2024-03-28T00:56:38.307671+0000","last_fullsized":"2024-03-28T00:56:38.307671+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:31:11.489908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308398+0000","last_change":"2024-03-28T00:54:32.259159+0000","last_active":"2024-03-28T00:56:38.308398+0000","last_peered":"2024-03-28T00:56:38.308398+0000","last_clean":"2024-03-28T00:56:38.308398+0000","last_became_active":"2024-03-28T00:54:32.258708+0000","last_became_peered":"2024-03-28T00:54:32.258708+0000","last_unstale":"2024-03-28T00:56:38.308398+0000","last_undegraded":"2024-03-28T00:56:38.308398+0000","last_fullsized":"2024-03-28T00:56:38.308398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:37:01.524878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303253+0000","last_change":"2024-03-28T00:54:32.814358+0000","last_active":"2024-03-28T00:56:38.303253+0000","last_peered":"2024-03-28T00:56:38.303253+0000","last_clean":"2024-03-28T00:56:38.303253+0000","last_became_active":"2024-03-28T00:54:32.814018+0000","last_became_peered":"2024-03-28T00:54:32.814018+0000","last_unstale":"2024-03-28T00:56:38.303253+0000","last_undegraded":"2024-03-28T00:56:38.303253+0000","last_fullsized":"2024-03-28T00:56:38.303253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:24:38.287853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284939+0000","last_change":"2024-03-28T00:54:36.822161+0000","last_active":"2024-03-28T00:56:38.284939+0000","last_peered":"2024-03-28T00:56:38.284939+0000","last_clean":"2024-03-28T00:56:38.284939+0000","last_became_active":"2024-03-28T00:54:36.821750+0000","last_became_peered":"2024-03-28T00:54:36.821750+0000","last_unstale":"2024-03-28T00:56:38.284939+0000","last_undegraded":"2024-03-28T00:56:38.284939+0000","last_fullsized":"2024-03-28T00:56:38.284939+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:13:36.892453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308562+0000","last_change":"2024-03-28T00:54:32.257038+0000","last_active":"2024-03-28T00:56:38.308562+0000","last_peered":"2024-03-28T00:56:38.308562+0000","last_clean":"2024-03-28T00:56:38.308562+0000","last_became_active":"2024-03-28T00:54:32.256702+0000","last_became_peered":"2024-03-28T00:54:32.256702+0000","last_unstale":"2024-03-28T00:56:38.308562+0000","last_undegraded":"2024-03-28T00:56:38.308562+0000","last_fullsized":"2024-03-28T00:56:38.308562+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:35:17.294116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307329+0000","last_change":"2024-03-28T00:54:32.813514+0000","last_active":"2024-03-28T00:56:38.307329+0000","last_peered":"2024-03-28T00:56:38.307329+0000","last_clean":"2024-03-28T00:56:38.307329+0000","last_became_active":"2024-03-28T00:54:32.813192+0000","last_became_peered":"2024-03-28T00:54:32.813192+0000","last_unstale":"2024-03-28T00:56:38.307329+0000","last_undegraded":"2024-03-28T00:56:38.307329+0000","last_fullsized":"2024-03-28T00:56:38.307329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:55:03.410818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286147+0000","last_change":"2024-03-28T00:54:36.820995+0000","last_active":"2024-03-28T00:56:38.286147+0000","last_peered":"2024-03-28T00:56:38.286147+0000","last_clean":"2024-03-28T00:56:38.286147+0000","last_became_active":"2024-03-28T00:54:36.820628+0000","last_became_peered":"2024-03-28T00:54:36.820628+0000","last_unstale":"2024-03-28T00:56:38.286147+0000","last_undegraded":"2024-03-28T00:56:38.286147+0000","last_fullsized":"2024-03-28T00:56:38.286147+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:18:58.771527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'40","reported_seq":313,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:46.396459+0000","last_change":"2024-03-28T00:54:32.251092+0000","last_active":"2024-03-28T00:56:46.396459+0000","last_peered":"2024-03-28T00:56:46.396459+0000","last_clean":"2024-03-28T00:56:46.396459+0000","last_became_active":"2024-03-28T00:54:32.250759+0000","last_became_peered":"2024-03-28T00:54:32.250759+0000","last_unstale":"2024-03-28T00:56:46.396459+0000","last_undegraded":"2024-03-28T00:56:46.396459+0000","last_fullsized":"2024-03-28T00:56:46.396459+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:57:30.283365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73065,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":240,"num_read_kb":191,"num_write":41,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308272+0000","last_change":"2024-03-28T00:54:32.809580+0000","last_active":"2024-03-28T00:56:38.308272+0000","last_peered":"2024-03-28T00:56:38.308272+0000","last_clean":"2024-03-28T00:56:38.308272+0000","last_became_active":"2024-03-28T00:54:32.809238+0000","last_became_peered":"2024-03-28T00:54:32.809238+0000","last_unstale":"2024-03-28T00:56:38.308272+0000","last_undegraded":"2024-03-28T00:56:38.308272+0000","last_fullsized":"2024-03-28T00:56:38.308272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:20:17.397594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308935+0000","last_change":"2024-03-28T00:54:36.826851+0000","last_active":"2024-03-28T00:56:38.308935+0000","last_peered":"2024-03-28T00:56:38.308935+0000","last_clean":"2024-03-28T00:56:38.308935+0000","last_became_active":"2024-03-28T00:54:36.826359+0000","last_became_peered":"2024-03-28T00:54:36.826359+0000","last_unstale":"2024-03-28T00:56:38.308935+0000","last_undegraded":"2024-03-28T00:56:38.308935+0000","last_fullsized":"2024-03-28T00:56:38.308935+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:27:48.857368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285191+0000","last_change":"2024-03-28T00:54:32.251701+0000","last_active":"2024-03-28T00:56:38.285191+0000","last_peered":"2024-03-28T00:56:38.285191+0000","last_clean":"2024-03-28T00:56:38.285191+0000","last_became_active":"2024-03-28T00:54:32.251403+0000","last_became_peered":"2024-03-28T00:54:32.251403+0000","last_unstale":"2024-03-28T00:56:38.285191+0000","last_undegraded":"2024-03-28T00:56:38.285191+0000","last_fullsized":"2024-03-28T00:56:38.285191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:28:16.127643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350770+0000","last_change":"2024-03-28T00:54:32.806810+0000","last_active":"2024-03-28T00:56:38.350770+0000","last_peered":"2024-03-28T00:56:38.350770+0000","last_clean":"2024-03-28T00:56:38.350770+0000","last_became_active":"2024-03-28T00:54:32.806468+0000","last_became_peered":"2024-03-28T00:54:32.806468+0000","last_unstale":"2024-03-28T00:56:38.350770+0000","last_undegraded":"2024-03-28T00:56:38.350770+0000","last_fullsized":"2024-03-28T00:56:38.350770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:54:16.196032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307882+0000","last_change":"2024-03-28T00:55:37.919873+0000","last_active":"2024-03-28T00:56:38.307882+0000","last_peered":"2024-03-28T00:56:38.307882+0000","last_clean":"2024-03-28T00:56:38.307882+0000","last_became_active":"2024-03-28T00:55:37.919396+0000","last_became_peered":"2024-03-28T00:55:37.919396+0000","last_unstale":"2024-03-28T00:56:38.307882+0000","last_undegraded":"2024-03-28T00:56:38.307882+0000","last_fullsized":"2024-03-28T00:56:38.307882+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:05:59.524285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307902+0000","last_change":"2024-03-28T00:54:32.259440+0000","last_active":"2024-03-28T00:56:38.307902+0000","last_peered":"2024-03-28T00:56:38.307902+0000","last_clean":"2024-03-28T00:56:38.307902+0000","last_became_active":"2024-03-28T00:54:32.258225+0000","last_became_peered":"2024-03-28T00:54:32.258225+0000","last_unstale":"2024-03-28T00:56:38.307902+0000","last_undegraded":"2024-03-28T00:56:38.307902+0000","last_fullsized":"2024-03-28T00:56:38.307902+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:07:18.398302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308098+0000","last_change":"2024-03-28T00:54:32.806853+0000","last_active":"2024-03-28T00:56:38.308098+0000","last_peered":"2024-03-28T00:56:38.308098+0000","last_clean":"2024-03-28T00:56:38.308098+0000","last_became_active":"2024-03-28T00:54:32.806551+0000","last_became_peered":"2024-03-28T00:54:32.806551+0000","last_unstale":"2024-03-28T00:56:38.308098+0000","last_undegraded":"2024-03-28T00:56:38.308098+0000","last_fullsized":"2024-03-28T00:56:38.308098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:49:45.774252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286620+0000","last_change":"2024-03-28T00:54:36.827542+0000","last_active":"2024-03-28T00:56:38.286620+0000","last_peered":"2024-03-28T00:56:38.286620+0000","last_clean":"2024-03-28T00:56:38.286620+0000","last_became_active":"2024-03-28T00:54:36.827163+0000","last_became_peered":"2024-03-28T00:54:36.827163+0000","last_unstale":"2024-03-28T00:56:38.286620+0000","last_undegraded":"2024-03-28T00:56:38.286620+0000","last_fullsized":"2024-03-28T00:56:38.286620+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:36:15.278111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'3","reported_seq":152,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:44.395209+0000","last_change":"2024-03-28T00:54:32.253231+0000","last_active":"2024-03-28T00:56:44.395209+0000","last_peered":"2024-03-28T00:56:44.395209+0000","last_clean":"2024-03-28T00:56:44.395209+0000","last_became_active":"2024-03-28T00:54:32.252731+0000","last_became_peered":"2024-03-28T00:54:32.252731+0000","last_unstale":"2024-03-28T00:56:44.395209+0000","last_undegraded":"2024-03-28T00:56:44.395209+0000","last_fullsized":"2024-03-28T00:56:44.395209+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:05:01.716654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308421+0000","last_change":"2024-03-28T00:54:32.805073+0000","last_active":"2024-03-28T00:56:38.308421+0000","last_peered":"2024-03-28T00:56:38.308421+0000","last_clean":"2024-03-28T00:56:38.308421+0000","last_became_active":"2024-03-28T00:54:32.803690+0000","last_became_peered":"2024-03-28T00:54:32.803690+0000","last_unstale":"2024-03-28T00:56:38.308421+0000","last_undegraded":"2024-03-28T00:56:38.308421+0000","last_fullsized":"2024-03-28T00:56:38.308421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:41:16.586168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350844+0000","last_change":"2024-03-28T00:54:36.835463+0000","last_active":"2024-03-28T00:56:38.350844+0000","last_peered":"2024-03-28T00:56:38.350844+0000","last_clean":"2024-03-28T00:56:38.350844+0000","last_became_active":"2024-03-28T00:54:36.834982+0000","last_became_peered":"2024-03-28T00:54:36.834982+0000","last_unstale":"2024-03-28T00:56:38.350844+0000","last_undegraded":"2024-03-28T00:56:38.350844+0000","last_fullsized":"2024-03-28T00:56:38.350844+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:27:32.259586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307377+0000","last_change":"2024-03-28T00:54:32.248461+0000","last_active":"2024-03-28T00:56:38.307377+0000","last_peered":"2024-03-28T00:56:38.307377+0000","last_clean":"2024-03-28T00:56:38.307377+0000","last_became_active":"2024-03-28T00:54:32.248140+0000","last_became_peered":"2024-03-28T00:54:32.248140+0000","last_unstale":"2024-03-28T00:56:38.307377+0000","last_undegraded":"2024-03-28T00:56:38.307377+0000","last_fullsized":"2024-03-28T00:56:38.307377+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:13:38.784288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307746+0000","last_change":"2024-03-28T00:54:32.806651+0000","last_active":"2024-03-28T00:56:38.307746+0000","last_peered":"2024-03-28T00:56:38.307746+0000","last_clean":"2024-03-28T00:56:38.307746+0000","last_became_active":"2024-03-28T00:54:32.806329+0000","last_became_peered":"2024-03-28T00:54:32.806329+0000","last_unstale":"2024-03-28T00:56:38.307746+0000","last_undegraded":"2024-03-28T00:56:38.307746+0000","last_fullsized":"2024-03-28T00:56:38.307746+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:28:32.815396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307991+0000","last_change":"2024-03-28T00:54:36.827016+0000","last_active":"2024-03-28T00:56:38.307991+0000","last_peered":"2024-03-28T00:56:38.307991+0000","last_clean":"2024-03-28T00:56:38.307991+0000","last_became_active":"2024-03-28T00:54:36.826428+0000","last_became_peered":"2024-03-28T00:54:36.826428+0000","last_unstale":"2024-03-28T00:56:38.307991+0000","last_undegraded":"2024-03-28T00:56:38.307991+0000","last_fullsized":"2024-03-28T00:56:38.307991+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:33:01.593990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308142+0000","last_change":"2024-03-28T00:55:37.573420+0000","last_active":"2024-03-28T00:56:38.308142+0000","last_peered":"2024-03-28T00:56:38.308142+0000","last_clean":"2024-03-28T00:56:38.308142+0000","last_became_active":"2024-03-28T00:55:37.568657+0000","last_became_peered":"2024-03-28T00:55:37.568657+0000","last_unstale":"2024-03-28T00:56:38.308142+0000","last_undegraded":"2024-03-28T00:56:38.308142+0000","last_fullsized":"2024-03-28T00:56:38.308142+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:03:21.483054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308301+0000","last_change":"2024-03-28T00:55:37.919698+0000","last_active":"2024-03-28T00:56:38.308301+0000","last_peered":"2024-03-28T00:56:38.308301+0000","last_clean":"2024-03-28T00:56:38.308301+0000","last_became_active":"2024-03-28T00:55:37.919276+0000","last_became_peered":"2024-03-28T00:55:37.919276+0000","last_unstale":"2024-03-28T00:56:38.308301+0000","last_undegraded":"2024-03-28T00:56:38.308301+0000","last_fullsized":"2024-03-28T00:56:38.308301+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:12:39.042250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308070+0000","last_change":"2024-03-28T00:55:37.918600+0000","last_active":"2024-03-28T00:56:38.308070+0000","last_peered":"2024-03-28T00:56:38.308070+0000","last_clean":"2024-03-28T00:56:38.308070+0000","last_became_active":"2024-03-28T00:55:37.918088+0000","last_became_peered":"2024-03-28T00:55:37.918088+0000","last_unstale":"2024-03-28T00:56:38.308070+0000","last_undegraded":"2024-03-28T00:56:38.308070+0000","last_fullsized":"2024-03-28T00:56:38.308070+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:29:23.492261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286573+0000","last_change":"2024-03-28T00:54:32.264038+0000","last_active":"2024-03-28T00:56:38.286573+0000","last_peered":"2024-03-28T00:56:38.286573+0000","last_clean":"2024-03-28T00:56:38.286573+0000","last_became_active":"2024-03-28T00:54:32.263706+0000","last_became_peered":"2024-03-28T00:54:32.263706+0000","last_unstale":"2024-03-28T00:56:38.286573+0000","last_undegraded":"2024-03-28T00:56:38.286573+0000","last_fullsized":"2024-03-28T00:56:38.286573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:03.244556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286955+0000","last_change":"2024-03-28T00:54:32.804727+0000","last_active":"2024-03-28T00:56:38.286955+0000","last_peered":"2024-03-28T00:56:38.286955+0000","last_clean":"2024-03-28T00:56:38.286955+0000","last_became_active":"2024-03-28T00:54:32.804422+0000","last_became_peered":"2024-03-28T00:54:32.804422+0000","last_unstale":"2024-03-28T00:56:38.286955+0000","last_undegraded":"2024-03-28T00:56:38.286955+0000","last_fullsized":"2024-03-28T00:56:38.286955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:29:57.983722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.351075+0000","last_change":"2024-03-28T00:54:36.833416+0000","last_active":"2024-03-28T00:56:38.351075+0000","last_peered":"2024-03-28T00:56:38.351075+0000","last_clean":"2024-03-28T00:56:38.351075+0000","last_became_active":"2024-03-28T00:54:36.832985+0000","last_became_peered":"2024-03-28T00:54:36.832985+0000","last_unstale":"2024-03-28T00:56:38.351075+0000","last_undegraded":"2024-03-28T00:56:38.351075+0000","last_fullsized":"2024-03-28T00:56:38.351075+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:47:30.376032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308027+0000","last_change":"2024-03-28T00:54:32.256417+0000","last_active":"2024-03-28T00:56:38.308027+0000","last_peered":"2024-03-28T00:56:38.308027+0000","last_clean":"2024-03-28T00:56:38.308027+0000","last_became_active":"2024-03-28T00:54:32.256088+0000","last_became_peered":"2024-03-28T00:54:32.256088+0000","last_unstale":"2024-03-28T00:56:38.308027+0000","last_undegraded":"2024-03-28T00:56:38.308027+0000","last_fullsized":"2024-03-28T00:56:38.308027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:58.661597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286728+0000","last_change":"2024-03-28T00:54:32.817604+0000","last_active":"2024-03-28T00:56:38.286728+0000","last_peered":"2024-03-28T00:56:38.286728+0000","last_clean":"2024-03-28T00:56:38.286728+0000","last_became_active":"2024-03-28T00:54:32.817211+0000","last_became_peered":"2024-03-28T00:54:32.817211+0000","last_unstale":"2024-03-28T00:56:38.286728+0000","last_undegraded":"2024-03-28T00:56:38.286728+0000","last_fullsized":"2024-03-28T00:56:38.286728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:02:28.335075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350947+0000","last_change":"2024-03-28T00:54:37.638968+0000","last_active":"2024-03-28T00:56:38.350947+0000","last_peered":"2024-03-28T00:56:38.350947+0000","last_clean":"2024-03-28T00:56:38.350947+0000","last_became_active":"2024-03-28T00:54:37.638446+0000","last_became_peered":"2024-03-28T00:54:37.638446+0000","last_unstale":"2024-03-28T00:56:38.350947+0000","last_undegraded":"2024-03-28T00:56:38.350947+0000","last_fullsized":"2024-03-28T00:56:38.350947+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:54:56.348760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308145+0000","last_change":"2024-03-28T00:54:32.809822+0000","last_active":"2024-03-28T00:56:38.308145+0000","last_peered":"2024-03-28T00:56:38.308145+0000","last_clean":"2024-03-28T00:56:38.308145+0000","last_became_active":"2024-03-28T00:54:32.808284+0000","last_became_peered":"2024-03-28T00:54:32.808284+0000","last_unstale":"2024-03-28T00:56:38.308145+0000","last_undegraded":"2024-03-28T00:56:38.308145+0000","last_fullsized":"2024-03-28T00:56:38.308145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:49:11.651960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":151,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:43.393776+0000","last_change":"2024-03-28T00:54:32.261806+0000","last_active":"2024-03-28T00:56:43.393776+0000","last_peered":"2024-03-28T00:56:43.393776+0000","last_clean":"2024-03-28T00:56:43.393776+0000","last_became_active":"2024-03-28T00:54:32.261493+0000","last_became_peered":"2024-03-28T00:54:32.261493+0000","last_unstale":"2024-03-28T00:56:43.393776+0000","last_undegraded":"2024-03-28T00:56:43.393776+0000","last_fullsized":"2024-03-28T00:56:43.393776+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:36:04.283408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308358+0000","last_change":"2024-03-28T00:54:36.819332+0000","last_active":"2024-03-28T00:56:38.308358+0000","last_peered":"2024-03-28T00:56:38.308358+0000","last_clean":"2024-03-28T00:56:38.308358+0000","last_became_active":"2024-03-28T00:54:36.818932+0000","last_became_peered":"2024-03-28T00:54:36.818932+0000","last_unstale":"2024-03-28T00:56:38.308358+0000","last_undegraded":"2024-03-28T00:56:38.308358+0000","last_fullsized":"2024-03-28T00:56:38.308358+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:00:39.446738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284570+0000","last_change":"2024-03-28T00:54:32.806093+0000","last_active":"2024-03-28T00:56:38.284570+0000","last_peered":"2024-03-28T00:56:38.284570+0000","last_clean":"2024-03-28T00:56:38.284570+0000","last_became_active":"2024-03-28T00:54:32.805734+0000","last_became_peered":"2024-03-28T00:54:32.805734+0000","last_unstale":"2024-03-28T00:56:38.284570+0000","last_undegraded":"2024-03-28T00:56:38.284570+0000","last_fullsized":"2024-03-28T00:56:38.284570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:51:41.066225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284968+0000","last_change":"2024-03-28T00:54:32.245182+0000","last_active":"2024-03-28T00:56:38.284968+0000","last_peered":"2024-03-28T00:56:38.284968+0000","last_clean":"2024-03-28T00:56:38.284968+0000","last_became_active":"2024-03-28T00:54:32.244855+0000","last_became_peered":"2024-03-28T00:54:32.244855+0000","last_unstale":"2024-03-28T00:56:38.284968+0000","last_undegraded":"2024-03-28T00:56:38.284968+0000","last_fullsized":"2024-03-28T00:56:38.284968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:39:09.997199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308243+0000","last_change":"2024-03-28T00:53:20.355628+0000","last_active":"2024-03-28T00:56:38.308243+0000","last_peered":"2024-03-28T00:56:38.308243+0000","last_clean":"2024-03-28T00:56:38.308243+0000","last_became_active":"2024-03-28T00:53:20.355175+0000","last_became_peered":"2024-03-28T00:53:20.355175+0000","last_unstale":"2024-03-28T00:56:38.308243+0000","last_undegraded":"2024-03-28T00:56:38.308243+0000","last_fullsized":"2024-03-28T00:56:38.308243+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:49:57.054435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:49:57.054435+0000","last_clean_scrub_stamp":"2024-03-28T00:49:57.054435+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:55:46.753799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308032+0000","last_change":"2024-03-28T00:54:37.639760+0000","last_active":"2024-03-28T00:56:38.308032+0000","last_peered":"2024-03-28T00:56:38.308032+0000","last_clean":"2024-03-28T00:56:38.308032+0000","last_became_active":"2024-03-28T00:54:37.639287+0000","last_became_peered":"2024-03-28T00:54:37.639287+0000","last_unstale":"2024-03-28T00:56:38.308032+0000","last_undegraded":"2024-03-28T00:56:38.308032+0000","last_fullsized":"2024-03-28T00:56:38.308032+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:50:02.840312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350597+0000","last_change":"2024-03-28T00:54:32.807764+0000","last_active":"2024-03-28T00:56:38.350597+0000","last_peered":"2024-03-28T00:56:38.350597+0000","last_clean":"2024-03-28T00:56:38.350597+0000","last_became_active":"2024-03-28T00:54:32.807441+0000","last_became_peered":"2024-03-28T00:54:32.807441+0000","last_unstale":"2024-03-28T00:56:38.350597+0000","last_undegraded":"2024-03-28T00:56:38.350597+0000","last_fullsized":"2024-03-28T00:56:38.350597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:37:40.540514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308361+0000","last_change":"2024-03-28T00:54:32.259192+0000","last_active":"2024-03-28T00:56:38.308361+0000","last_peered":"2024-03-28T00:56:38.308361+0000","last_clean":"2024-03-28T00:56:38.308361+0000","last_became_active":"2024-03-28T00:54:32.258880+0000","last_became_peered":"2024-03-28T00:54:32.258880+0000","last_unstale":"2024-03-28T00:56:38.308361+0000","last_undegraded":"2024-03-28T00:56:38.308361+0000","last_fullsized":"2024-03-28T00:56:38.308361+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:20:11.384633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307815+0000","last_change":"2024-03-28T00:54:36.822268+0000","last_active":"2024-03-28T00:56:38.307815+0000","last_peered":"2024-03-28T00:56:38.307815+0000","last_clean":"2024-03-28T00:56:38.307815+0000","last_became_active":"2024-03-28T00:54:36.821773+0000","last_became_peered":"2024-03-28T00:54:36.821773+0000","last_unstale":"2024-03-28T00:56:38.307815+0000","last_undegraded":"2024-03-28T00:56:38.307815+0000","last_fullsized":"2024-03-28T00:56:38.307815+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:56:11.206975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308433+0000","last_change":"2024-03-28T00:54:32.256133+0000","last_active":"2024-03-28T00:56:38.308433+0000","last_peered":"2024-03-28T00:56:38.308433+0000","last_clean":"2024-03-28T00:56:38.308433+0000","last_became_active":"2024-03-28T00:54:32.250535+0000","last_became_peered":"2024-03-28T00:54:32.250535+0000","last_unstale":"2024-03-28T00:56:38.308433+0000","last_undegraded":"2024-03-28T00:56:38.308433+0000","last_fullsized":"2024-03-28T00:56:38.308433+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:18:50.874181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.351152+0000","last_change":"2024-03-28T00:54:32.808883+0000","last_active":"2024-03-28T00:56:38.351152+0000","last_peered":"2024-03-28T00:56:38.351152+0000","last_clean":"2024-03-28T00:56:38.351152+0000","last_became_active":"2024-03-28T00:54:32.808587+0000","last_became_peered":"2024-03-28T00:54:32.808587+0000","last_unstale":"2024-03-28T00:56:38.351152+0000","last_undegraded":"2024-03-28T00:56:38.351152+0000","last_fullsized":"2024-03-28T00:56:38.351152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:09:55.164876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284641+0000","last_change":"2024-03-28T00:54:37.637743+0000","last_active":"2024-03-28T00:56:38.284641+0000","last_peered":"2024-03-28T00:56:38.284641+0000","last_clean":"2024-03-28T00:56:38.284641+0000","last_became_active":"2024-03-28T00:54:37.637280+0000","last_became_peered":"2024-03-28T00:54:37.637280+0000","last_unstale":"2024-03-28T00:56:38.284641+0000","last_undegraded":"2024-03-28T00:56:38.284641+0000","last_fullsized":"2024-03-28T00:56:38.284641+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:40:51.431757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'9","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:45.255094+0000","last_change":"2024-03-28T00:54:32.246444+0000","last_active":"2024-03-28T00:56:45.255094+0000","last_peered":"2024-03-28T00:56:45.255094+0000","last_clean":"2024-03-28T00:56:45.255094+0000","last_became_active":"2024-03-28T00:54:32.245106+0000","last_became_peered":"2024-03-28T00:54:32.245106+0000","last_unstale":"2024-03-28T00:56:45.255094+0000","last_undegraded":"2024-03-28T00:56:45.255094+0000","last_fullsized":"2024-03-28T00:56:45.255094+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:13:45.123625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285364+0000","last_change":"2024-03-28T00:54:32.812851+0000","last_active":"2024-03-28T00:56:38.285364+0000","last_peered":"2024-03-28T00:56:38.285364+0000","last_clean":"2024-03-28T00:56:38.285364+0000","last_became_active":"2024-03-28T00:54:32.812518+0000","last_became_peered":"2024-03-28T00:54:32.812518+0000","last_unstale":"2024-03-28T00:56:38.285364+0000","last_undegraded":"2024-03-28T00:56:38.285364+0000","last_fullsized":"2024-03-28T00:56:38.285364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:08:41.518888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284865+0000","last_change":"2024-03-28T00:54:36.818709+0000","last_active":"2024-03-28T00:56:38.284865+0000","last_peered":"2024-03-28T00:56:38.284865+0000","last_clean":"2024-03-28T00:56:38.284865+0000","last_became_active":"2024-03-28T00:54:36.818304+0000","last_became_peered":"2024-03-28T00:54:36.818304+0000","last_unstale":"2024-03-28T00:56:38.284865+0000","last_undegraded":"2024-03-28T00:56:38.284865+0000","last_fullsized":"2024-03-28T00:56:38.284865+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:03:07.642685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285251+0000","last_change":"2024-03-28T00:54:32.256031+0000","last_active":"2024-03-28T00:56:38.285251+0000","last_peered":"2024-03-28T00:56:38.285251+0000","last_clean":"2024-03-28T00:56:38.285251+0000","last_became_active":"2024-03-28T00:54:32.255597+0000","last_became_peered":"2024-03-28T00:54:32.255597+0000","last_unstale":"2024-03-28T00:56:38.285251+0000","last_undegraded":"2024-03-28T00:56:38.285251+0000","last_fullsized":"2024-03-28T00:56:38.285251+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:31:10.837212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303584+0000","last_change":"2024-03-28T00:54:32.816961+0000","last_active":"2024-03-28T00:56:38.303584+0000","last_peered":"2024-03-28T00:56:38.303584+0000","last_clean":"2024-03-28T00:56:38.303584+0000","last_became_active":"2024-03-28T00:54:32.816408+0000","last_became_peered":"2024-03-28T00:54:32.816408+0000","last_unstale":"2024-03-28T00:56:38.303584+0000","last_undegraded":"2024-03-28T00:56:38.303584+0000","last_fullsized":"2024-03-28T00:56:38.303584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:08:57.441492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285039+0000","last_change":"2024-03-28T00:55:37.918692+0000","last_active":"2024-03-28T00:56:38.285039+0000","last_peered":"2024-03-28T00:56:38.285039+0000","last_clean":"2024-03-28T00:56:38.285039+0000","last_became_active":"2024-03-28T00:55:37.918195+0000","last_became_peered":"2024-03-28T00:55:37.918195+0000","last_unstale":"2024-03-28T00:56:38.285039+0000","last_undegraded":"2024-03-28T00:56:38.285039+0000","last_fullsized":"2024-03-28T00:56:38.285039+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:01:06.012325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.306972+0000","last_change":"2024-03-28T00:55:37.570700+0000","last_active":"2024-03-28T00:56:38.306972+0000","last_peered":"2024-03-28T00:56:38.306972+0000","last_clean":"2024-03-28T00:56:38.306972+0000","last_became_active":"2024-03-28T00:55:37.566298+0000","last_became_peered":"2024-03-28T00:55:37.566298+0000","last_unstale":"2024-03-28T00:56:38.306972+0000","last_undegraded":"2024-03-28T00:56:38.306972+0000","last_fullsized":"2024-03-28T00:56:38.306972+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:46:48.352817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307628+0000","last_change":"2024-03-28T00:54:32.811592+0000","last_active":"2024-03-28T00:56:38.307628+0000","last_peered":"2024-03-28T00:56:38.307628+0000","last_clean":"2024-03-28T00:56:38.307628+0000","last_became_active":"2024-03-28T00:54:32.811228+0000","last_became_peered":"2024-03-28T00:54:32.811228+0000","last_unstale":"2024-03-28T00:56:38.307628+0000","last_undegraded":"2024-03-28T00:56:38.307628+0000","last_fullsized":"2024-03-28T00:56:38.307628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:14:19.585237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286681+0000","last_change":"2024-03-28T00:54:36.822501+0000","last_active":"2024-03-28T00:56:38.286681+0000","last_peered":"2024-03-28T00:56:38.286681+0000","last_clean":"2024-03-28T00:56:38.286681+0000","last_became_active":"2024-03-28T00:54:36.821630+0000","last_became_peered":"2024-03-28T00:54:36.821630+0000","last_unstale":"2024-03-28T00:56:38.286681+0000","last_undegraded":"2024-03-28T00:56:38.286681+0000","last_fullsized":"2024-03-28T00:56:38.286681+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:20:52.247055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284556+0000","last_change":"2024-03-28T00:54:32.247328+0000","last_active":"2024-03-28T00:56:38.284556+0000","last_peered":"2024-03-28T00:56:38.284556+0000","last_clean":"2024-03-28T00:56:38.284556+0000","last_became_active":"2024-03-28T00:54:32.247003+0000","last_became_peered":"2024-03-28T00:54:32.247003+0000","last_unstale":"2024-03-28T00:56:38.284556+0000","last_undegraded":"2024-03-28T00:56:38.284556+0000","last_fullsized":"2024-03-28T00:56:38.284556+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:33:09.764946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308401+0000","last_change":"2024-03-28T00:54:32.805977+0000","last_active":"2024-03-28T00:56:38.308401+0000","last_peered":"2024-03-28T00:56:38.308401+0000","last_clean":"2024-03-28T00:56:38.308401+0000","last_became_active":"2024-03-28T00:54:32.805636+0000","last_became_peered":"2024-03-28T00:54:32.805636+0000","last_unstale":"2024-03-28T00:56:38.308401+0000","last_undegraded":"2024-03-28T00:56:38.308401+0000","last_fullsized":"2024-03-28T00:56:38.308401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:04:42.901732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284987+0000","last_change":"2024-03-28T00:54:36.822103+0000","last_active":"2024-03-28T00:56:38.284987+0000","last_peered":"2024-03-28T00:56:38.284987+0000","last_clean":"2024-03-28T00:56:38.284987+0000","last_became_active":"2024-03-28T00:54:36.821695+0000","last_became_peered":"2024-03-28T00:54:36.821695+0000","last_unstale":"2024-03-28T00:56:38.284987+0000","last_undegraded":"2024-03-28T00:56:38.284987+0000","last_fullsized":"2024-03-28T00:56:38.284987+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:45:00.613955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285112+0000","last_change":"2024-03-28T00:54:32.251523+0000","last_active":"2024-03-28T00:56:38.285112+0000","last_peered":"2024-03-28T00:56:38.285112+0000","last_clean":"2024-03-28T00:56:38.285112+0000","last_became_active":"2024-03-28T00:54:32.251223+0000","last_became_peered":"2024-03-28T00:54:32.251223+0000","last_unstale":"2024-03-28T00:56:38.285112+0000","last_undegraded":"2024-03-28T00:56:38.285112+0000","last_fullsized":"2024-03-28T00:56:38.285112+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:08:20.489848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284729+0000","last_change":"2024-03-28T00:54:32.817641+0000","last_active":"2024-03-28T00:56:38.284729+0000","last_peered":"2024-03-28T00:56:38.284729+0000","last_clean":"2024-03-28T00:56:38.284729+0000","last_became_active":"2024-03-28T00:54:32.817252+0000","last_became_peered":"2024-03-28T00:54:32.817252+0000","last_unstale":"2024-03-28T00:56:38.284729+0000","last_undegraded":"2024-03-28T00:56:38.284729+0000","last_fullsized":"2024-03-28T00:56:38.284729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:54:48.275005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285020+0000","last_change":"2024-03-28T00:54:37.638990+0000","last_active":"2024-03-28T00:56:38.285020+0000","last_peered":"2024-03-28T00:56:38.285020+0000","last_clean":"2024-03-28T00:56:38.285020+0000","last_became_active":"2024-03-28T00:54:37.638538+0000","last_became_peered":"2024-03-28T00:54:37.638538+0000","last_unstale":"2024-03-28T00:56:38.285020+0000","last_undegraded":"2024-03-28T00:56:38.285020+0000","last_fullsized":"2024-03-28T00:56:38.285020+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:28:10.791805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308482+0000","last_change":"2024-03-28T00:54:32.253254+0000","last_active":"2024-03-28T00:56:38.308482+0000","last_peered":"2024-03-28T00:56:38.308482+0000","last_clean":"2024-03-28T00:56:38.308482+0000","last_became_active":"2024-03-28T00:54:32.252879+0000","last_became_peered":"2024-03-28T00:54:32.252879+0000","last_unstale":"2024-03-28T00:56:38.308482+0000","last_undegraded":"2024-03-28T00:56:38.308482+0000","last_fullsized":"2024-03-28T00:56:38.308482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:59:51.039215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308081+0000","last_change":"2024-03-28T00:54:32.811288+0000","last_active":"2024-03-28T00:56:38.308081+0000","last_peered":"2024-03-28T00:56:38.308081+0000","last_clean":"2024-03-28T00:56:38.308081+0000","last_became_active":"2024-03-28T00:54:32.810968+0000","last_became_peered":"2024-03-28T00:54:32.810968+0000","last_unstale":"2024-03-28T00:56:38.308081+0000","last_undegraded":"2024-03-28T00:56:38.308081+0000","last_fullsized":"2024-03-28T00:56:38.308081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:16:56.113046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307049+0000","last_change":"2024-03-28T00:54:36.830714+0000","last_active":"2024-03-28T00:56:38.307049+0000","last_peered":"2024-03-28T00:56:38.307049+0000","last_clean":"2024-03-28T00:56:38.307049+0000","last_became_active":"2024-03-28T00:54:36.830253+0000","last_became_peered":"2024-03-28T00:54:36.830253+0000","last_unstale":"2024-03-28T00:56:38.307049+0000","last_undegraded":"2024-03-28T00:56:38.307049+0000","last_fullsized":"2024-03-28T00:56:38.307049+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:24:43.696150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285475+0000","last_change":"2024-03-28T00:54:32.254009+0000","last_active":"2024-03-28T00:56:38.285475+0000","last_peered":"2024-03-28T00:56:38.285475+0000","last_clean":"2024-03-28T00:56:38.285475+0000","last_became_active":"2024-03-28T00:54:32.253697+0000","last_became_peered":"2024-03-28T00:54:32.253697+0000","last_unstale":"2024-03-28T00:56:38.285475+0000","last_undegraded":"2024-03-28T00:56:38.285475+0000","last_fullsized":"2024-03-28T00:56:38.285475+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:17:44.629367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308505+0000","last_change":"2024-03-28T00:54:32.812828+0000","last_active":"2024-03-28T00:56:38.308505+0000","last_peered":"2024-03-28T00:56:38.308505+0000","last_clean":"2024-03-28T00:56:38.308505+0000","last_became_active":"2024-03-28T00:54:32.811591+0000","last_became_peered":"2024-03-28T00:54:32.811591+0000","last_unstale":"2024-03-28T00:56:38.308505+0000","last_undegraded":"2024-03-28T00:56:38.308505+0000","last_fullsized":"2024-03-28T00:56:38.308505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:04.613623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303210+0000","last_change":"2024-03-28T00:54:36.823284+0000","last_active":"2024-03-28T00:56:38.303210+0000","last_peered":"2024-03-28T00:56:38.303210+0000","last_clean":"2024-03-28T00:56:38.303210+0000","last_became_active":"2024-03-28T00:54:36.822883+0000","last_became_peered":"2024-03-28T00:54:36.822883+0000","last_unstale":"2024-03-28T00:56:38.303210+0000","last_undegraded":"2024-03-28T00:56:38.303210+0000","last_fullsized":"2024-03-28T00:56:38.303210+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:37:10.266063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308755+0000","last_change":"2024-03-28T00:54:32.253946+0000","last_active":"2024-03-28T00:56:38.308755+0000","last_peered":"2024-03-28T00:56:38.308755+0000","last_clean":"2024-03-28T00:56:38.308755+0000","last_became_active":"2024-03-28T00:54:32.253650+0000","last_became_peered":"2024-03-28T00:54:32.253650+0000","last_unstale":"2024-03-28T00:56:38.308755+0000","last_undegraded":"2024-03-28T00:56:38.308755+0000","last_fullsized":"2024-03-28T00:56:38.308755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:11:44.930074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308286+0000","last_change":"2024-03-28T00:54:32.815410+0000","last_active":"2024-03-28T00:56:38.308286+0000","last_peered":"2024-03-28T00:56:38.308286+0000","last_clean":"2024-03-28T00:56:38.308286+0000","last_became_active":"2024-03-28T00:54:32.815088+0000","last_became_peered":"2024-03-28T00:54:32.815088+0000","last_unstale":"2024-03-28T00:56:38.308286+0000","last_undegraded":"2024-03-28T00:56:38.308286+0000","last_fullsized":"2024-03-28T00:56:38.308286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:16:24.862822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307742+0000","last_change":"2024-03-28T00:54:36.823182+0000","last_active":"2024-03-28T00:56:38.307742+0000","last_peered":"2024-03-28T00:56:38.307742+0000","last_clean":"2024-03-28T00:56:38.307742+0000","last_became_active":"2024-03-28T00:54:36.821446+0000","last_became_peered":"2024-03-28T00:54:36.821446+0000","last_unstale":"2024-03-28T00:56:38.307742+0000","last_undegraded":"2024-03-28T00:56:38.307742+0000","last_fullsized":"2024-03-28T00:56:38.307742+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:45:20.030213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350942+0000","last_change":"2024-03-28T00:54:32.257412+0000","last_active":"2024-03-28T00:56:38.350942+0000","last_peered":"2024-03-28T00:56:38.350942+0000","last_clean":"2024-03-28T00:56:38.350942+0000","last_became_active":"2024-03-28T00:54:32.257115+0000","last_became_peered":"2024-03-28T00:54:32.257115+0000","last_unstale":"2024-03-28T00:56:38.350942+0000","last_undegraded":"2024-03-28T00:56:38.350942+0000","last_fullsized":"2024-03-28T00:56:38.350942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:04:29.850867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308522+0000","last_change":"2024-03-28T00:54:32.805879+0000","last_active":"2024-03-28T00:56:38.308522+0000","last_peered":"2024-03-28T00:56:38.308522+0000","last_clean":"2024-03-28T00:56:38.308522+0000","last_became_active":"2024-03-28T00:54:32.805507+0000","last_became_peered":"2024-03-28T00:54:32.805507+0000","last_unstale":"2024-03-28T00:56:38.308522+0000","last_undegraded":"2024-03-28T00:56:38.308522+0000","last_fullsized":"2024-03-28T00:56:38.308522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:51:51.229085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308515+0000","last_change":"2024-03-28T00:54:32.807219+0000","last_active":"2024-03-28T00:56:38.308515+0000","last_peered":"2024-03-28T00:56:38.308515+0000","last_clean":"2024-03-28T00:56:38.308515+0000","last_became_active":"2024-03-28T00:54:32.806856+0000","last_became_peered":"2024-03-28T00:54:32.806856+0000","last_unstale":"2024-03-28T00:56:38.308515+0000","last_undegraded":"2024-03-28T00:56:38.308515+0000","last_fullsized":"2024-03-28T00:56:38.308515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:05:49.914636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285057+0000","last_change":"2024-03-28T00:55:37.569156+0000","last_active":"2024-03-28T00:56:38.285057+0000","last_peered":"2024-03-28T00:56:38.285057+0000","last_clean":"2024-03-28T00:56:38.285057+0000","last_became_active":"2024-03-28T00:55:37.568775+0000","last_became_peered":"2024-03-28T00:55:37.568775+0000","last_unstale":"2024-03-28T00:56:38.285057+0000","last_undegraded":"2024-03-28T00:56:38.285057+0000","last_fullsized":"2024-03-28T00:56:38.285057+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:00:21.458550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285006+0000","last_change":"2024-03-28T00:54:32.813921+0000","last_active":"2024-03-28T00:56:38.285006+0000","last_peered":"2024-03-28T00:56:38.285006+0000","last_clean":"2024-03-28T00:56:38.285006+0000","last_became_active":"2024-03-28T00:54:32.812877+0000","last_became_peered":"2024-03-28T00:54:32.812877+0000","last_unstale":"2024-03-28T00:56:38.285006+0000","last_undegraded":"2024-03-28T00:56:38.285006+0000","last_fullsized":"2024-03-28T00:56:38.285006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:26:54.305074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303553+0000","last_change":"2024-03-28T00:54:32.815725+0000","last_active":"2024-03-28T00:56:38.303553+0000","last_peered":"2024-03-28T00:56:38.303553+0000","last_clean":"2024-03-28T00:56:38.303553+0000","last_became_active":"2024-03-28T00:54:32.815444+0000","last_became_peered":"2024-03-28T00:54:32.815444+0000","last_unstale":"2024-03-28T00:56:38.303553+0000","last_undegraded":"2024-03-28T00:56:38.303553+0000","last_fullsized":"2024-03-28T00:56:38.303553+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:23:17.326589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.351026+0000","last_change":"2024-03-28T00:54:32.806595+0000","last_active":"2024-03-28T00:56:38.351026+0000","last_peered":"2024-03-28T00:56:38.351026+0000","last_clean":"2024-03-28T00:56:38.351026+0000","last_became_active":"2024-03-28T00:54:32.806256+0000","last_became_peered":"2024-03-28T00:54:32.806256+0000","last_unstale":"2024-03-28T00:56:38.351026+0000","last_undegraded":"2024-03-28T00:56:38.351026+0000","last_fullsized":"2024-03-28T00:56:38.351026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:04:22.551511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308462+0000","last_change":"2024-03-28T00:54:32.805000+0000","last_active":"2024-03-28T00:56:38.308462+0000","last_peered":"2024-03-28T00:56:38.308462+0000","last_clean":"2024-03-28T00:56:38.308462+0000","last_became_active":"2024-03-28T00:54:32.804637+0000","last_became_peered":"2024-03-28T00:54:32.804637+0000","last_unstale":"2024-03-28T00:56:38.308462+0000","last_undegraded":"2024-03-28T00:56:38.308462+0000","last_fullsized":"2024-03-28T00:56:38.308462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:59:46.888014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285395+0000","last_change":"2024-03-28T00:54:32.802783+0000","last_active":"2024-03-28T00:56:38.285395+0000","last_peered":"2024-03-28T00:56:38.285395+0000","last_clean":"2024-03-28T00:56:38.285395+0000","last_became_active":"2024-03-28T00:54:32.802457+0000","last_became_peered":"2024-03-28T00:54:32.802457+0000","last_unstale":"2024-03-28T00:56:38.285395+0000","last_undegraded":"2024-03-28T00:56:38.285395+0000","last_fullsized":"2024-03-28T00:56:38.285395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:28:24.146047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285531+0000","last_change":"2024-03-28T00:54:32.810179+0000","last_active":"2024-03-28T00:56:38.285531+0000","last_peered":"2024-03-28T00:56:38.285531+0000","last_clean":"2024-03-28T00:56:38.285531+0000","last_became_active":"2024-03-28T00:54:32.809009+0000","last_became_peered":"2024-03-28T00:54:32.809009+0000","last_unstale":"2024-03-28T00:56:38.285531+0000","last_undegraded":"2024-03-28T00:56:38.285531+0000","last_fullsized":"2024-03-28T00:56:38.285531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:08:44.184910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308081+0000","last_change":"2024-03-28T00:54:32.810084+0000","last_active":"2024-03-28T00:56:38.308081+0000","last_peered":"2024-03-28T00:56:38.308081+0000","last_clean":"2024-03-28T00:56:38.308081+0000","last_became_active":"2024-03-28T00:54:32.809386+0000","last_became_peered":"2024-03-28T00:54:32.809386+0000","last_unstale":"2024-03-28T00:56:38.308081+0000","last_undegraded":"2024-03-28T00:56:38.308081+0000","last_fullsized":"2024-03-28T00:56:38.308081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:49:13.754797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284532+0000","last_change":"2024-03-28T00:54:32.809933+0000","last_active":"2024-03-28T00:56:38.284532+0000","last_peered":"2024-03-28T00:56:38.284532+0000","last_clean":"2024-03-28T00:56:38.284532+0000","last_became_active":"2024-03-28T00:54:32.808974+0000","last_became_peered":"2024-03-28T00:54:32.808974+0000","last_unstale":"2024-03-28T00:56:38.284532+0000","last_undegraded":"2024-03-28T00:56:38.284532+0000","last_fullsized":"2024-03-28T00:56:38.284532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:23:59.317887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307220+0000","last_change":"2024-03-28T00:54:32.808764+0000","last_active":"2024-03-28T00:56:38.307220+0000","last_peered":"2024-03-28T00:56:38.307220+0000","last_clean":"2024-03-28T00:56:38.307220+0000","last_became_active":"2024-03-28T00:54:32.808034+0000","last_became_peered":"2024-03-28T00:54:32.808034+0000","last_unstale":"2024-03-28T00:56:38.307220+0000","last_undegraded":"2024-03-28T00:56:38.307220+0000","last_fullsized":"2024-03-28T00:56:38.307220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:27:12.762520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350903+0000","last_change":"2024-03-28T00:54:32.805470+0000","last_active":"2024-03-28T00:56:38.350903+0000","last_peered":"2024-03-28T00:56:38.350903+0000","last_clean":"2024-03-28T00:56:38.350903+0000","last_became_active":"2024-03-28T00:54:32.805146+0000","last_became_peered":"2024-03-28T00:54:32.805146+0000","last_unstale":"2024-03-28T00:56:38.350903+0000","last_undegraded":"2024-03-28T00:56:38.350903+0000","last_fullsized":"2024-03-28T00:56:38.350903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:24:14.297966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308078+0000","last_change":"2024-03-28T00:55:37.566696+0000","last_active":"2024-03-28T00:56:38.308078+0000","last_peered":"2024-03-28T00:56:38.308078+0000","last_clean":"2024-03-28T00:56:38.308078+0000","last_became_active":"2024-03-28T00:55:37.566334+0000","last_became_peered":"2024-03-28T00:55:37.566334+0000","last_unstale":"2024-03-28T00:56:38.308078+0000","last_undegraded":"2024-03-28T00:56:38.308078+0000","last_fullsized":"2024-03-28T00:56:38.308078+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:41:37.752201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282084,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":77906,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":716,"num_read_kb":439,"num_write":175,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450560,"data_stored":226947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153,"ondisk_log_size":153,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383720,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":529042,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645358,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32336,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712816,"statfs":{"total":95995035648,"available":95961923584,"internally_reserved":0,"allocated":1187840,"data_stored":1189353,"data_compressed":65432,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808882,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":533352,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070520,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":529154,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332156,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1097728,"data_stored":1125226,"data_compressed":65432,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495681,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":524432,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":591335,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1052672,"data_stored":525820,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":599988,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182480,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1040384,"data_stored":524626,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920844,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":529082,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":1121122,"data_compressed":65432,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34218,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34218,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34218,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":66993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":75646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":74643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T00:56:49.681 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-28T00:56:49.682 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T00:56:49.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.817+0000 7f47c6bb3640 1 -- 172.21.15.169:0/1003845753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47c0107430 msgr2=0x7f47c0109820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:49.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.817+0000 7f47c6bb3640 1 --2- 172.21.15.169:0/1003845753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47c0107430 0x7f47c0109820 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f47b0009900 tx=0x7f47b002bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:49.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.817+0000 7f47c6bb3640 1 -- 172.21.15.169:0/1003845753 shutdown_connections 2024-03-28T00:56:49.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.817+0000 7f47c6bb3640 1 --2- 172.21.15.169:0/1003845753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47c0107430 0x7f47c0109820 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.817+0000 7f47c6bb3640 1 --2- 172.21.15.169:0/1003845753 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47c0104b00 0x7f47c0106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.817+0000 7f47c6bb3640 1 --2- 172.21.15.169:0/1003845753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f47c01021d0 0x7f47c01045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.817+0000 7f47c6bb3640 1 -- 172.21.15.169:0/1003845753 >> 172.21.15.169:0/1003845753 conn(0x7f47c00fbb80 msgr2=0x7f47c00fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:49.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.817+0000 7f47c6bb3640 1 -- 172.21.15.169:0/1003845753 shutdown_connections 2024-03-28T00:56:49.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.817+0000 7f47c6bb3640 1 -- 172.21.15.169:0/1003845753 wait complete. 2024-03-28T00:56:49.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.817+0000 7f47c6bb3640 1 Processor -- start 2024-03-28T00:56:49.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.817+0000 7f47c6bb3640 1 -- start start 2024-03-28T00:56:49.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.817+0000 7f47c6bb3640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47c01021d0 0x7f47c0197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:49.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.817+0000 7f47c6bb3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f47c0104b00 0x7f47c0197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:49.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.817+0000 7f47c6bb3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47c0107430 0x7f47c019ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:49.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.817+0000 7f47c6bb3640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c010d350 con 0x7f47c01021d0 2024-03-28T00:56:49.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.817+0000 7f47c6bb3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c010d1d0 con 0x7f47c0104b00 2024-03-28T00:56:49.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.817+0000 7f47c6bb3640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c010d4d0 con 0x7f47c0107430 2024-03-28T00:56:49.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.818+0000 7f47c494f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47c01021d0 0x7f47c0197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:49.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.818+0000 7f47c494f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47c01021d0 0x7f47c0197950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34838/0 (socket says 172.21.15.169:34838) 2024-03-28T00:56:49.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.818+0000 7f47c494f640 1 -- 172.21.15.169:0/254412601 learned_addr learned my addr 172.21.15.169:0/254412601 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:49.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.818+0000 7f47c5150640 1 --2- 172.21.15.169:0/254412601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47c0107430 0x7f47c019ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:49.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.818+0000 7f47bffff640 1 --2- 172.21.15.169:0/254412601 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f47c0104b00 0x7f47c0197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:49.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.818+0000 7f47c494f640 1 -- 172.21.15.169:0/254412601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47c0107430 msgr2=0x7f47c019ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:49.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.818+0000 7f47c494f640 1 --2- 172.21.15.169:0/254412601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47c0107430 0x7f47c019ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.818+0000 7f47c494f640 1 -- 172.21.15.169:0/254412601 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f47c0104b00 msgr2=0x7f47c0197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:49.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.818+0000 7f47c494f640 1 --2- 172.21.15.169:0/254412601 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f47c0104b00 0x7f47c0197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:49.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.818+0000 7f47c494f640 1 -- 172.21.15.169:0/254412601 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c019f570 con 0x7f47c01021d0 2024-03-28T00:56:49.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.818+0000 7f47c494f640 1 --2- 172.21.15.169:0/254412601 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47c01021d0 0x7f47c0197950 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f47b800ebc0 tx=0x7f47b800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:49.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.818+0000 7f47bdffb640 1 -- 172.21.15.169:0/254412601 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b8010630 con 0x7f47c01021d0 2024-03-28T00:56:49.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.818+0000 7f47c6bb3640 1 -- 172.21.15.169:0/254412601 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47c019f860 con 0x7f47c01021d0 2024-03-28T00:56:49.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.818+0000 7f47bdffb640 1 -- 172.21.15.169:0/254412601 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47b8010cb0 con 0x7f47c01021d0 2024-03-28T00:56:49.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.818+0000 7f47bdffb640 1 -- 172.21.15.169:0/254412601 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b8018bd0 con 0x7f47c01021d0 2024-03-28T00:56:49.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.818+0000 7f47c6bb3640 1 -- 172.21.15.169:0/254412601 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47c019fe00 con 0x7f47c01021d0 2024-03-28T00:56:49.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.819+0000 7f47c6bb3640 1 -- 172.21.15.169:0/254412601 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47c00688e0 con 0x7f47c01021d0 2024-03-28T00:56:49.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.820+0000 7f47bdffb640 1 -- 172.21.15.169:0/254412601 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f47b80107d0 con 0x7f47c01021d0 2024-03-28T00:56:49.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.821+0000 7f47bdffb640 1 --2- 172.21.15.169:0/254412601 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f47a80a2890 0x7f47a80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:49.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.821+0000 7f47bffff640 1 --2- 172.21.15.169:0/254412601 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f47a80a2890 0x7f47a80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:49.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.821+0000 7f47bdffb640 1 -- 172.21.15.169:0/254412601 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f47b80cf570 con 0x7f47c01021d0 2024-03-28T00:56:49.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.821+0000 7f47bffff640 1 --2- 172.21.15.169:0/254412601 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f47a80a2890 0x7f47a80a4d50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f47b40095f0 tx=0x7f47b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:49.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.825+0000 7f47bdffb640 1 -- 172.21.15.169:0/254412601 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f47b809bac0 con 0x7f47c01021d0 2024-03-28T00:56:49.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:49.996+0000 7f47c6bb3640 1 -- 172.21.15.169:0/254412601 --> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f47c0061ae0 con 0x7f47a80a2890 2024-03-28T00:56:50.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.006+0000 7f47bdffb640 1 -- 172.21.15.169:0/254412601 <== mgr.14152 v2:172.21.15.169:6800/1069722251 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340752 (secure 0 0 0) 0x7f47c0061ae0 con 0x7f47a80a2890 2024-03-28T00:56:50.012 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:50.014 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-03-28T00:56:50.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.009+0000 7f47c6bb3640 1 -- 172.21.15.169:0/254412601 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f47a80a2890 msgr2=0x7f47a80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:50.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.009+0000 7f47c6bb3640 1 --2- 172.21.15.169:0/254412601 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f47a80a2890 0x7f47a80a4d50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f47b40095f0 tx=0x7f47b4005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:50.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.009+0000 7f47c6bb3640 1 -- 172.21.15.169:0/254412601 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47c01021d0 msgr2=0x7f47c0197950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:50.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.009+0000 7f47c6bb3640 1 --2- 172.21.15.169:0/254412601 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47c01021d0 0x7f47c0197950 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f47b800ebc0 tx=0x7f47b800c6a0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.009+0000 7f47c6bb3640 1 -- 172.21.15.169:0/254412601 shutdown_connections 2024-03-28T00:56:50.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.009+0000 7f47c6bb3640 1 --2- 172.21.15.169:0/254412601 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f47a80a2890 0x7f47a80a4d50 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.009+0000 7f47c6bb3640 1 --2- 172.21.15.169:0/254412601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47c0107430 0x7f47c019ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.009+0000 7f47c6bb3640 1 --2- 172.21.15.169:0/254412601 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f47c0104b00 0x7f47c0197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.009+0000 7f47c6bb3640 1 --2- 172.21.15.169:0/254412601 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47c01021d0 0x7f47c0197950 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.009+0000 7f47c6bb3640 1 -- 172.21.15.169:0/254412601 >> 172.21.15.169:0/254412601 conn(0x7f47c00fbb80 msgr2=0x7f47c0102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:50.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.009+0000 7f47c6bb3640 1 -- 172.21.15.169:0/254412601 shutdown_connections 2024-03-28T00:56:50.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.009+0000 7f47c6bb3640 1 -- 172.21.15.169:0/254412601 wait complete. 2024-03-28T00:56:50.023 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-03-28T00:56:48.116387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":668508,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":842,"num_read_kb":548,"num_write":358,"num_write_kb":3074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":792,"ondisk_log_size":792,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357676,"kb_used_data":12616,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584148,"statfs":{"total":1151940427776,"available":1151574167552,"internally_reserved":0,"allocated":12918784,"data_stored":8322532,"data_compressed":462210,"data_compressed_allocated":626688,"data_compressed_original":2752512,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6771,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":63,"num_read_kb":38,"num_write":8,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002486"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308755+0000","last_change":"2024-03-28T00:54:32.809951+0000","last_active":"2024-03-28T00:56:38.308755+0000","last_peered":"2024-03-28T00:56:38.308755+0000","last_clean":"2024-03-28T00:56:38.308755+0000","last_became_active":"2024-03-28T00:54:32.808871+0000","last_became_peered":"2024-03-28T00:54:32.808871+0000","last_unstale":"2024-03-28T00:56:38.308755+0000","last_undegraded":"2024-03-28T00:56:38.308755+0000","last_fullsized":"2024-03-28T00:56:38.308755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:15:17.152487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303536+0000","last_change":"2024-03-28T00:54:32.816707+0000","last_active":"2024-03-28T00:56:38.303536+0000","last_peered":"2024-03-28T00:56:38.303536+0000","last_clean":"2024-03-28T00:56:38.303536+0000","last_became_active":"2024-03-28T00:54:32.816000+0000","last_became_peered":"2024-03-28T00:54:32.816000+0000","last_unstale":"2024-03-28T00:56:38.303536+0000","last_undegraded":"2024-03-28T00:56:38.303536+0000","last_fullsized":"2024-03-28T00:56:38.303536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:56:19.532808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303346+0000","last_change":"2024-03-28T00:54:32.817023+0000","last_active":"2024-03-28T00:56:38.303346+0000","last_peered":"2024-03-28T00:56:38.303346+0000","last_clean":"2024-03-28T00:56:38.303346+0000","last_became_active":"2024-03-28T00:54:32.816712+0000","last_became_peered":"2024-03-28T00:54:32.816712+0000","last_unstale":"2024-03-28T00:56:38.303346+0000","last_undegraded":"2024-03-28T00:56:38.303346+0000","last_fullsized":"2024-03-28T00:56:38.303346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:13:25.334294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285135+0000","last_change":"2024-03-28T00:54:32.807259+0000","last_active":"2024-03-28T00:56:38.285135+0000","last_peered":"2024-03-28T00:56:38.285135+0000","last_clean":"2024-03-28T00:56:38.285135+0000","last_became_active":"2024-03-28T00:54:32.806944+0000","last_became_peered":"2024-03-28T00:54:32.806944+0000","last_unstale":"2024-03-28T00:56:38.285135+0000","last_undegraded":"2024-03-28T00:56:38.285135+0000","last_fullsized":"2024-03-28T00:56:38.285135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:46:04.852351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308386+0000","last_change":"2024-03-28T00:54:32.807613+0000","last_active":"2024-03-28T00:56:38.308386+0000","last_peered":"2024-03-28T00:56:38.308386+0000","last_clean":"2024-03-28T00:56:38.308386+0000","last_became_active":"2024-03-28T00:54:32.807280+0000","last_became_peered":"2024-03-28T00:54:32.807280+0000","last_unstale":"2024-03-28T00:56:38.308386+0000","last_undegraded":"2024-03-28T00:56:38.308386+0000","last_fullsized":"2024-03-28T00:56:38.308386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:56:37.133675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286708+0000","last_change":"2024-03-28T00:54:32.806804+0000","last_active":"2024-03-28T00:56:38.286708+0000","last_peered":"2024-03-28T00:56:38.286708+0000","last_clean":"2024-03-28T00:56:38.286708+0000","last_became_active":"2024-03-28T00:54:32.805343+0000","last_became_peered":"2024-03-28T00:54:32.805343+0000","last_unstale":"2024-03-28T00:56:38.286708+0000","last_undegraded":"2024-03-28T00:56:38.286708+0000","last_fullsized":"2024-03-28T00:56:38.286708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:31:44.849926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307907+0000","last_change":"2024-03-28T00:55:37.563371+0000","last_active":"2024-03-28T00:56:38.307907+0000","last_peered":"2024-03-28T00:56:38.307907+0000","last_clean":"2024-03-28T00:56:38.307907+0000","last_became_active":"2024-03-28T00:55:37.563000+0000","last_became_peered":"2024-03-28T00:55:37.563000+0000","last_unstale":"2024-03-28T00:56:38.307907+0000","last_undegraded":"2024-03-28T00:56:38.307907+0000","last_fullsized":"2024-03-28T00:56:38.307907+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:56:58.782502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.309667+0000","last_change":"2024-03-28T00:56:38.309328+0000","last_active":"2024-03-28T00:56:38.309667+0000","last_peered":"2024-03-28T00:56:38.309667+0000","last_clean":"2024-03-28T00:56:38.309667+0000","last_became_active":"2024-03-28T00:56:38.308978+0000","last_became_peered":"2024-03-28T00:56:38.308978+0000","last_unstale":"2024-03-28T00:56:38.309667+0000","last_undegraded":"2024-03-28T00:56:38.309667+0000","last_fullsized":"2024-03-28T00:56:38.309667+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:45:52.643890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307485+0000","last_change":"2024-03-28T00:54:32.800667+0000","last_active":"2024-03-28T00:56:38.307485+0000","last_peered":"2024-03-28T00:56:38.307485+0000","last_clean":"2024-03-28T00:56:38.307485+0000","last_became_active":"2024-03-28T00:54:32.800332+0000","last_became_peered":"2024-03-28T00:54:32.800332+0000","last_unstale":"2024-03-28T00:56:38.307485+0000","last_undegraded":"2024-03-28T00:56:38.307485+0000","last_fullsized":"2024-03-28T00:56:38.307485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:41:27.577871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.288488+0000","last_change":"2024-03-28T00:56:38.288488+0000","last_active":"2024-03-28T00:56:38.288488+0000","last_peered":"2024-03-28T00:56:38.288488+0000","last_clean":"2024-03-28T00:56:38.288488+0000","last_became_active":"2024-03-28T00:56:38.288164+0000","last_became_peered":"2024-03-28T00:56:38.288164+0000","last_unstale":"2024-03-28T00:56:38.288488+0000","last_undegraded":"2024-03-28T00:56:38.288488+0000","last_fullsized":"2024-03-28T00:56:38.288488+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:50:33.312420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308456+0000","last_change":"2024-03-28T00:54:32.811543+0000","last_active":"2024-03-28T00:56:38.308456+0000","last_peered":"2024-03-28T00:56:38.308456+0000","last_clean":"2024-03-28T00:56:38.308456+0000","last_became_active":"2024-03-28T00:54:32.811210+0000","last_became_peered":"2024-03-28T00:54:32.811210+0000","last_unstale":"2024-03-28T00:56:38.308456+0000","last_undegraded":"2024-03-28T00:56:38.308456+0000","last_fullsized":"2024-03-28T00:56:38.308456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:47:19.539107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285098+0000","last_change":"2024-03-28T00:54:32.812648+0000","last_active":"2024-03-28T00:56:38.285098+0000","last_peered":"2024-03-28T00:56:38.285098+0000","last_clean":"2024-03-28T00:56:38.285098+0000","last_became_active":"2024-03-28T00:54:32.812340+0000","last_became_peered":"2024-03-28T00:54:32.812340+0000","last_unstale":"2024-03-28T00:56:38.285098+0000","last_undegraded":"2024-03-28T00:56:38.285098+0000","last_fullsized":"2024-03-28T00:56:38.285098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:20:45.449953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350843+0000","last_change":"2024-03-28T00:54:32.801807+0000","last_active":"2024-03-28T00:56:38.350843+0000","last_peered":"2024-03-28T00:56:38.350843+0000","last_clean":"2024-03-28T00:56:38.350843+0000","last_became_active":"2024-03-28T00:54:32.801498+0000","last_became_peered":"2024-03-28T00:54:32.801498+0000","last_unstale":"2024-03-28T00:56:38.350843+0000","last_undegraded":"2024-03-28T00:56:38.350843+0000","last_fullsized":"2024-03-28T00:56:38.350843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:35:35.827856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303105+0000","last_change":"2024-03-28T00:54:32.818033+0000","last_active":"2024-03-28T00:56:38.303105+0000","last_peered":"2024-03-28T00:56:38.303105+0000","last_clean":"2024-03-28T00:56:38.303105+0000","last_became_active":"2024-03-28T00:54:32.817309+0000","last_became_peered":"2024-03-28T00:54:32.817309+0000","last_unstale":"2024-03-28T00:56:38.303105+0000","last_undegraded":"2024-03-28T00:56:38.303105+0000","last_fullsized":"2024-03-28T00:56:38.303105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:39:42.682780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303139+0000","last_change":"2024-03-28T00:54:32.815224+0000","last_active":"2024-03-28T00:56:38.303139+0000","last_peered":"2024-03-28T00:56:38.303139+0000","last_clean":"2024-03-28T00:56:38.303139+0000","last_became_active":"2024-03-28T00:54:32.814936+0000","last_became_peered":"2024-03-28T00:54:32.814936+0000","last_unstale":"2024-03-28T00:56:38.303139+0000","last_undegraded":"2024-03-28T00:56:38.303139+0000","last_fullsized":"2024-03-28T00:56:38.303139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:08:40.200393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.306895+0000","last_change":"2024-03-28T00:54:32.807342+0000","last_active":"2024-03-28T00:56:38.306895+0000","last_peered":"2024-03-28T00:56:38.306895+0000","last_clean":"2024-03-28T00:56:38.306895+0000","last_became_active":"2024-03-28T00:54:32.806978+0000","last_became_peered":"2024-03-28T00:54:32.806978+0000","last_unstale":"2024-03-28T00:56:38.306895+0000","last_undegraded":"2024-03-28T00:56:38.306895+0000","last_fullsized":"2024-03-28T00:56:38.306895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:17:58.599693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308639+0000","last_change":"2024-03-28T00:54:32.810036+0000","last_active":"2024-03-28T00:56:38.308639+0000","last_peered":"2024-03-28T00:56:38.308639+0000","last_clean":"2024-03-28T00:56:38.308639+0000","last_became_active":"2024-03-28T00:54:32.809438+0000","last_became_peered":"2024-03-28T00:54:32.809438+0000","last_unstale":"2024-03-28T00:56:38.308639+0000","last_undegraded":"2024-03-28T00:56:38.308639+0000","last_fullsized":"2024-03-28T00:56:38.308639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:18:50.447474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286485+0000","last_change":"2024-03-28T00:54:32.815398+0000","last_active":"2024-03-28T00:56:38.286485+0000","last_peered":"2024-03-28T00:56:38.286485+0000","last_clean":"2024-03-28T00:56:38.286485+0000","last_became_active":"2024-03-28T00:54:32.815045+0000","last_became_peered":"2024-03-28T00:54:32.815045+0000","last_unstale":"2024-03-28T00:56:38.286485+0000","last_undegraded":"2024-03-28T00:56:38.286485+0000","last_fullsized":"2024-03-28T00:56:38.286485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:40:29.060217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285333+0000","last_change":"2024-03-28T00:54:32.805736+0000","last_active":"2024-03-28T00:56:38.285333+0000","last_peered":"2024-03-28T00:56:38.285333+0000","last_clean":"2024-03-28T00:56:38.285333+0000","last_became_active":"2024-03-28T00:54:32.805435+0000","last_became_peered":"2024-03-28T00:54:32.805435+0000","last_unstale":"2024-03-28T00:56:38.285333+0000","last_undegraded":"2024-03-28T00:56:38.285333+0000","last_fullsized":"2024-03-28T00:56:38.285333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:24:30.893583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285426+0000","last_change":"2024-03-28T00:54:36.824531+0000","last_active":"2024-03-28T00:56:38.285426+0000","last_peered":"2024-03-28T00:56:38.285426+0000","last_clean":"2024-03-28T00:56:38.285426+0000","last_became_active":"2024-03-28T00:54:36.824123+0000","last_became_peered":"2024-03-28T00:54:36.824123+0000","last_unstale":"2024-03-28T00:56:38.285426+0000","last_undegraded":"2024-03-28T00:56:38.285426+0000","last_fullsized":"2024-03-28T00:56:38.285426+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:10:47.503171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286759+0000","last_change":"2024-03-28T00:54:32.256226+0000","last_active":"2024-03-28T00:56:38.286759+0000","last_peered":"2024-03-28T00:56:38.286759+0000","last_clean":"2024-03-28T00:56:38.286759+0000","last_became_active":"2024-03-28T00:54:32.255921+0000","last_became_peered":"2024-03-28T00:54:32.255921+0000","last_unstale":"2024-03-28T00:56:38.286759+0000","last_undegraded":"2024-03-28T00:56:38.286759+0000","last_fullsized":"2024-03-28T00:56:38.286759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:56:04.196311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284800+0000","last_change":"2024-03-28T00:54:32.808248+0000","last_active":"2024-03-28T00:56:38.284800+0000","last_peered":"2024-03-28T00:56:38.284800+0000","last_clean":"2024-03-28T00:56:38.284800+0000","last_became_active":"2024-03-28T00:54:32.807923+0000","last_became_peered":"2024-03-28T00:54:32.807923+0000","last_unstale":"2024-03-28T00:56:38.284800+0000","last_undegraded":"2024-03-28T00:56:38.284800+0000","last_fullsized":"2024-03-28T00:56:38.284800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:59:56.002564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284703+0000","last_change":"2024-03-28T00:54:36.830556+0000","last_active":"2024-03-28T00:56:38.284703+0000","last_peered":"2024-03-28T00:56:38.284703+0000","last_clean":"2024-03-28T00:56:38.284703+0000","last_became_active":"2024-03-28T00:54:36.830151+0000","last_became_peered":"2024-03-28T00:54:36.830151+0000","last_unstale":"2024-03-28T00:56:38.284703+0000","last_undegraded":"2024-03-28T00:56:38.284703+0000","last_fullsized":"2024-03-28T00:56:38.284703+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:19:23.073410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285141+0000","last_change":"2024-03-28T00:54:32.260635+0000","last_active":"2024-03-28T00:56:38.285141+0000","last_peered":"2024-03-28T00:56:38.285141+0000","last_clean":"2024-03-28T00:56:38.285141+0000","last_became_active":"2024-03-28T00:54:32.260084+0000","last_became_peered":"2024-03-28T00:54:32.260084+0000","last_unstale":"2024-03-28T00:56:38.285141+0000","last_undegraded":"2024-03-28T00:56:38.285141+0000","last_fullsized":"2024-03-28T00:56:38.285141+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:06.858519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303455+0000","last_change":"2024-03-28T00:54:32.809901+0000","last_active":"2024-03-28T00:56:38.303455+0000","last_peered":"2024-03-28T00:56:38.303455+0000","last_clean":"2024-03-28T00:56:38.303455+0000","last_became_active":"2024-03-28T00:54:32.809557+0000","last_became_peered":"2024-03-28T00:54:32.809557+0000","last_unstale":"2024-03-28T00:56:38.303455+0000","last_undegraded":"2024-03-28T00:56:38.303455+0000","last_fullsized":"2024-03-28T00:56:38.303455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:05:31.363487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284824+0000","last_change":"2024-03-28T00:54:36.824661+0000","last_active":"2024-03-28T00:56:38.284824+0000","last_peered":"2024-03-28T00:56:38.284824+0000","last_clean":"2024-03-28T00:56:38.284824+0000","last_became_active":"2024-03-28T00:54:36.824223+0000","last_became_peered":"2024-03-28T00:54:36.824223+0000","last_unstale":"2024-03-28T00:56:38.284824+0000","last_undegraded":"2024-03-28T00:56:38.284824+0000","last_fullsized":"2024-03-28T00:56:38.284824+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:02:25.509248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308563+0000","last_change":"2024-03-28T00:54:32.256436+0000","last_active":"2024-03-28T00:56:38.308563+0000","last_peered":"2024-03-28T00:56:38.308563+0000","last_clean":"2024-03-28T00:56:38.308563+0000","last_became_active":"2024-03-28T00:54:32.256069+0000","last_became_peered":"2024-03-28T00:54:32.256069+0000","last_unstale":"2024-03-28T00:56:38.308563+0000","last_undegraded":"2024-03-28T00:56:38.308563+0000","last_fullsized":"2024-03-28T00:56:38.308563+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:43.688192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307212+0000","last_change":"2024-03-28T00:54:32.799269+0000","last_active":"2024-03-28T00:56:38.307212+0000","last_peered":"2024-03-28T00:56:38.307212+0000","last_clean":"2024-03-28T00:56:38.307212+0000","last_became_active":"2024-03-28T00:54:32.798848+0000","last_became_peered":"2024-03-28T00:54:32.798848+0000","last_unstale":"2024-03-28T00:56:38.307212+0000","last_undegraded":"2024-03-28T00:56:38.307212+0000","last_fullsized":"2024-03-28T00:56:38.307212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:56:20.318466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284703+0000","last_change":"2024-03-28T00:54:36.823153+0000","last_active":"2024-03-28T00:56:38.284703+0000","last_peered":"2024-03-28T00:56:38.284703+0000","last_clean":"2024-03-28T00:56:38.284703+0000","last_became_active":"2024-03-28T00:54:36.822817+0000","last_became_peered":"2024-03-28T00:54:36.822817+0000","last_unstale":"2024-03-28T00:56:38.284703+0000","last_undegraded":"2024-03-28T00:56:38.284703+0000","last_fullsized":"2024-03-28T00:56:38.284703+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:20:23.001599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308631+0000","last_change":"2024-03-28T00:54:32.250439+0000","last_active":"2024-03-28T00:56:38.308631+0000","last_peered":"2024-03-28T00:56:38.308631+0000","last_clean":"2024-03-28T00:56:38.308631+0000","last_became_active":"2024-03-28T00:54:32.250006+0000","last_became_peered":"2024-03-28T00:54:32.250006+0000","last_unstale":"2024-03-28T00:56:38.308631+0000","last_undegraded":"2024-03-28T00:56:38.308631+0000","last_fullsized":"2024-03-28T00:56:38.308631+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:31:38.957429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308661+0000","last_change":"2024-03-28T00:54:32.806888+0000","last_active":"2024-03-28T00:56:38.308661+0000","last_peered":"2024-03-28T00:56:38.308661+0000","last_clean":"2024-03-28T00:56:38.308661+0000","last_became_active":"2024-03-28T00:54:32.806581+0000","last_became_peered":"2024-03-28T00:54:32.806581+0000","last_unstale":"2024-03-28T00:56:38.308661+0000","last_undegraded":"2024-03-28T00:56:38.308661+0000","last_fullsized":"2024-03-28T00:56:38.308661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:25:52.331819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285199+0000","last_change":"2024-03-28T00:54:36.824041+0000","last_active":"2024-03-28T00:56:38.285199+0000","last_peered":"2024-03-28T00:56:38.285199+0000","last_clean":"2024-03-28T00:56:38.285199+0000","last_became_active":"2024-03-28T00:54:36.823658+0000","last_became_peered":"2024-03-28T00:54:36.823658+0000","last_unstale":"2024-03-28T00:56:38.285199+0000","last_undegraded":"2024-03-28T00:56:38.285199+0000","last_fullsized":"2024-03-28T00:56:38.285199+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:11:03.236009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'9","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:45.395923+0000","last_change":"2024-03-28T00:54:32.257534+0000","last_active":"2024-03-28T00:56:45.395923+0000","last_peered":"2024-03-28T00:56:45.395923+0000","last_clean":"2024-03-28T00:56:45.395923+0000","last_became_active":"2024-03-28T00:54:32.256404+0000","last_became_peered":"2024-03-28T00:54:32.256404+0000","last_unstale":"2024-03-28T00:56:45.395923+0000","last_undegraded":"2024-03-28T00:56:45.395923+0000","last_fullsized":"2024-03-28T00:56:45.395923+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:39:38.329799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1853,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308137+0000","last_change":"2024-03-28T00:54:32.806099+0000","last_active":"2024-03-28T00:56:38.308137+0000","last_peered":"2024-03-28T00:56:38.308137+0000","last_clean":"2024-03-28T00:56:38.308137+0000","last_became_active":"2024-03-28T00:54:32.805811+0000","last_became_peered":"2024-03-28T00:54:32.805811+0000","last_unstale":"2024-03-28T00:56:38.308137+0000","last_undegraded":"2024-03-28T00:56:38.308137+0000","last_fullsized":"2024-03-28T00:56:38.308137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:42:32.820245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285325+0000","last_change":"2024-03-28T00:54:36.827198+0000","last_active":"2024-03-28T00:56:38.285325+0000","last_peered":"2024-03-28T00:56:38.285325+0000","last_clean":"2024-03-28T00:56:38.285325+0000","last_became_active":"2024-03-28T00:54:36.826804+0000","last_became_peered":"2024-03-28T00:54:36.826804+0000","last_unstale":"2024-03-28T00:56:38.285325+0000","last_undegraded":"2024-03-28T00:56:38.285325+0000","last_fullsized":"2024-03-28T00:56:38.285325+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:38:11.030404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286690+0000","last_change":"2024-03-28T00:54:32.249394+0000","last_active":"2024-03-28T00:56:38.286690+0000","last_peered":"2024-03-28T00:56:38.286690+0000","last_clean":"2024-03-28T00:56:38.286690+0000","last_became_active":"2024-03-28T00:54:32.249089+0000","last_became_peered":"2024-03-28T00:54:32.249089+0000","last_unstale":"2024-03-28T00:56:38.286690+0000","last_undegraded":"2024-03-28T00:56:38.286690+0000","last_fullsized":"2024-03-28T00:56:38.286690+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:05:01.720053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307883+0000","last_change":"2024-03-28T00:54:32.809133+0000","last_active":"2024-03-28T00:56:38.307883+0000","last_peered":"2024-03-28T00:56:38.307883+0000","last_clean":"2024-03-28T00:56:38.307883+0000","last_became_active":"2024-03-28T00:54:32.808841+0000","last_became_peered":"2024-03-28T00:54:32.808841+0000","last_unstale":"2024-03-28T00:56:38.307883+0000","last_undegraded":"2024-03-28T00:56:38.307883+0000","last_fullsized":"2024-03-28T00:56:38.307883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:19:37.859542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308573+0000","last_change":"2024-03-28T00:54:36.824483+0000","last_active":"2024-03-28T00:56:38.308573+0000","last_peered":"2024-03-28T00:56:38.308573+0000","last_clean":"2024-03-28T00:56:38.308573+0000","last_became_active":"2024-03-28T00:54:36.823973+0000","last_became_peered":"2024-03-28T00:54:36.823973+0000","last_unstale":"2024-03-28T00:56:38.308573+0000","last_undegraded":"2024-03-28T00:56:38.308573+0000","last_fullsized":"2024-03-28T00:56:38.308573+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:01:06.251119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350674+0000","last_change":"2024-03-28T00:54:32.251287+0000","last_active":"2024-03-28T00:56:38.350674+0000","last_peered":"2024-03-28T00:56:38.350674+0000","last_clean":"2024-03-28T00:56:38.350674+0000","last_became_active":"2024-03-28T00:54:32.250904+0000","last_became_peered":"2024-03-28T00:54:32.250904+0000","last_unstale":"2024-03-28T00:56:38.350674+0000","last_undegraded":"2024-03-28T00:56:38.350674+0000","last_fullsized":"2024-03-28T00:56:38.350674+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:18:09.051056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284458+0000","last_change":"2024-03-28T00:54:32.803124+0000","last_active":"2024-03-28T00:56:38.284458+0000","last_peered":"2024-03-28T00:56:38.284458+0000","last_clean":"2024-03-28T00:56:38.284458+0000","last_became_active":"2024-03-28T00:54:32.802825+0000","last_became_peered":"2024-03-28T00:54:32.802825+0000","last_unstale":"2024-03-28T00:56:38.284458+0000","last_undegraded":"2024-03-28T00:56:38.284458+0000","last_fullsized":"2024-03-28T00:56:38.284458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:34:21.825314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303447+0000","last_change":"2024-03-28T00:54:36.832151+0000","last_active":"2024-03-28T00:56:38.303447+0000","last_peered":"2024-03-28T00:56:38.303447+0000","last_clean":"2024-03-28T00:56:38.303447+0000","last_became_active":"2024-03-28T00:54:36.831748+0000","last_became_peered":"2024-03-28T00:54:36.831748+0000","last_unstale":"2024-03-28T00:56:38.303447+0000","last_undegraded":"2024-03-28T00:56:38.303447+0000","last_fullsized":"2024-03-28T00:56:38.303447+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:45:32.284386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307890+0000","last_change":"2024-03-28T00:54:32.259048+0000","last_active":"2024-03-28T00:56:38.307890+0000","last_peered":"2024-03-28T00:56:38.307890+0000","last_clean":"2024-03-28T00:56:38.307890+0000","last_became_active":"2024-03-28T00:54:32.258038+0000","last_became_peered":"2024-03-28T00:54:32.258038+0000","last_unstale":"2024-03-28T00:56:38.307890+0000","last_undegraded":"2024-03-28T00:56:38.307890+0000","last_fullsized":"2024-03-28T00:56:38.307890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:45:23.525516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286245+0000","last_change":"2024-03-28T00:54:32.809771+0000","last_active":"2024-03-28T00:56:38.286245+0000","last_peered":"2024-03-28T00:56:38.286245+0000","last_clean":"2024-03-28T00:56:38.286245+0000","last_became_active":"2024-03-28T00:54:32.809467+0000","last_became_peered":"2024-03-28T00:54:32.809467+0000","last_unstale":"2024-03-28T00:56:38.286245+0000","last_undegraded":"2024-03-28T00:56:38.286245+0000","last_fullsized":"2024-03-28T00:56:38.286245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:30:15.440202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284418+0000","last_change":"2024-03-28T00:54:36.823313+0000","last_active":"2024-03-28T00:56:38.284418+0000","last_peered":"2024-03-28T00:56:38.284418+0000","last_clean":"2024-03-28T00:56:38.284418+0000","last_became_active":"2024-03-28T00:54:36.822923+0000","last_became_peered":"2024-03-28T00:54:36.822923+0000","last_unstale":"2024-03-28T00:56:38.284418+0000","last_undegraded":"2024-03-28T00:56:38.284418+0000","last_fullsized":"2024-03-28T00:56:38.284418+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:22:21.125028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'5","reported_seq":153,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:45.395588+0000","last_change":"2024-03-28T00:54:32.255084+0000","last_active":"2024-03-28T00:56:45.395588+0000","last_peered":"2024-03-28T00:56:45.395588+0000","last_clean":"2024-03-28T00:56:45.395588+0000","last_became_active":"2024-03-28T00:54:32.254744+0000","last_became_peered":"2024-03-28T00:54:32.254744+0000","last_unstale":"2024-03-28T00:56:45.395588+0000","last_undegraded":"2024-03-28T00:56:45.395588+0000","last_fullsized":"2024-03-28T00:56:45.395588+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:32:03.906826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":115,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307394+0000","last_change":"2024-03-28T00:54:32.810050+0000","last_active":"2024-03-28T00:56:38.307394+0000","last_peered":"2024-03-28T00:56:38.307394+0000","last_clean":"2024-03-28T00:56:38.307394+0000","last_became_active":"2024-03-28T00:54:32.809731+0000","last_became_peered":"2024-03-28T00:54:32.809731+0000","last_unstale":"2024-03-28T00:56:38.307394+0000","last_undegraded":"2024-03-28T00:56:38.307394+0000","last_fullsized":"2024-03-28T00:56:38.307394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:20:11.876731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307671+0000","last_change":"2024-03-28T00:55:37.566900+0000","last_active":"2024-03-28T00:56:38.307671+0000","last_peered":"2024-03-28T00:56:38.307671+0000","last_clean":"2024-03-28T00:56:38.307671+0000","last_became_active":"2024-03-28T00:55:37.566491+0000","last_became_peered":"2024-03-28T00:55:37.566491+0000","last_unstale":"2024-03-28T00:56:38.307671+0000","last_undegraded":"2024-03-28T00:56:38.307671+0000","last_fullsized":"2024-03-28T00:56:38.307671+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:31:11.489908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308398+0000","last_change":"2024-03-28T00:54:32.259159+0000","last_active":"2024-03-28T00:56:38.308398+0000","last_peered":"2024-03-28T00:56:38.308398+0000","last_clean":"2024-03-28T00:56:38.308398+0000","last_became_active":"2024-03-28T00:54:32.258708+0000","last_became_peered":"2024-03-28T00:54:32.258708+0000","last_unstale":"2024-03-28T00:56:38.308398+0000","last_undegraded":"2024-03-28T00:56:38.308398+0000","last_fullsized":"2024-03-28T00:56:38.308398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:37:01.524878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303253+0000","last_change":"2024-03-28T00:54:32.814358+0000","last_active":"2024-03-28T00:56:38.303253+0000","last_peered":"2024-03-28T00:56:38.303253+0000","last_clean":"2024-03-28T00:56:38.303253+0000","last_became_active":"2024-03-28T00:54:32.814018+0000","last_became_peered":"2024-03-28T00:54:32.814018+0000","last_unstale":"2024-03-28T00:56:38.303253+0000","last_undegraded":"2024-03-28T00:56:38.303253+0000","last_fullsized":"2024-03-28T00:56:38.303253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:24:38.287853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284939+0000","last_change":"2024-03-28T00:54:36.822161+0000","last_active":"2024-03-28T00:56:38.284939+0000","last_peered":"2024-03-28T00:56:38.284939+0000","last_clean":"2024-03-28T00:56:38.284939+0000","last_became_active":"2024-03-28T00:54:36.821750+0000","last_became_peered":"2024-03-28T00:54:36.821750+0000","last_unstale":"2024-03-28T00:56:38.284939+0000","last_undegraded":"2024-03-28T00:56:38.284939+0000","last_fullsized":"2024-03-28T00:56:38.284939+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:13:36.892453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308562+0000","last_change":"2024-03-28T00:54:32.257038+0000","last_active":"2024-03-28T00:56:38.308562+0000","last_peered":"2024-03-28T00:56:38.308562+0000","last_clean":"2024-03-28T00:56:38.308562+0000","last_became_active":"2024-03-28T00:54:32.256702+0000","last_became_peered":"2024-03-28T00:54:32.256702+0000","last_unstale":"2024-03-28T00:56:38.308562+0000","last_undegraded":"2024-03-28T00:56:38.308562+0000","last_fullsized":"2024-03-28T00:56:38.308562+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:35:17.294116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307329+0000","last_change":"2024-03-28T00:54:32.813514+0000","last_active":"2024-03-28T00:56:38.307329+0000","last_peered":"2024-03-28T00:56:38.307329+0000","last_clean":"2024-03-28T00:56:38.307329+0000","last_became_active":"2024-03-28T00:54:32.813192+0000","last_became_peered":"2024-03-28T00:54:32.813192+0000","last_unstale":"2024-03-28T00:56:38.307329+0000","last_undegraded":"2024-03-28T00:56:38.307329+0000","last_fullsized":"2024-03-28T00:56:38.307329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:55:03.410818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286147+0000","last_change":"2024-03-28T00:54:36.820995+0000","last_active":"2024-03-28T00:56:38.286147+0000","last_peered":"2024-03-28T00:56:38.286147+0000","last_clean":"2024-03-28T00:56:38.286147+0000","last_became_active":"2024-03-28T00:54:36.820628+0000","last_became_peered":"2024-03-28T00:54:36.820628+0000","last_unstale":"2024-03-28T00:56:38.286147+0000","last_undegraded":"2024-03-28T00:56:38.286147+0000","last_fullsized":"2024-03-28T00:56:38.286147+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:18:58.771527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'40","reported_seq":313,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:46.396459+0000","last_change":"2024-03-28T00:54:32.251092+0000","last_active":"2024-03-28T00:56:46.396459+0000","last_peered":"2024-03-28T00:56:46.396459+0000","last_clean":"2024-03-28T00:56:46.396459+0000","last_became_active":"2024-03-28T00:54:32.250759+0000","last_became_peered":"2024-03-28T00:54:32.250759+0000","last_unstale":"2024-03-28T00:56:46.396459+0000","last_undegraded":"2024-03-28T00:56:46.396459+0000","last_fullsized":"2024-03-28T00:56:46.396459+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:57:30.283365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73065,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":240,"num_read_kb":191,"num_write":41,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308272+0000","last_change":"2024-03-28T00:54:32.809580+0000","last_active":"2024-03-28T00:56:38.308272+0000","last_peered":"2024-03-28T00:56:38.308272+0000","last_clean":"2024-03-28T00:56:38.308272+0000","last_became_active":"2024-03-28T00:54:32.809238+0000","last_became_peered":"2024-03-28T00:54:32.809238+0000","last_unstale":"2024-03-28T00:56:38.308272+0000","last_undegraded":"2024-03-28T00:56:38.308272+0000","last_fullsized":"2024-03-28T00:56:38.308272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:20:17.397594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308935+0000","last_change":"2024-03-28T00:54:36.826851+0000","last_active":"2024-03-28T00:56:38.308935+0000","last_peered":"2024-03-28T00:56:38.308935+0000","last_clean":"2024-03-28T00:56:38.308935+0000","last_became_active":"2024-03-28T00:54:36.826359+0000","last_became_peered":"2024-03-28T00:54:36.826359+0000","last_unstale":"2024-03-28T00:56:38.308935+0000","last_undegraded":"2024-03-28T00:56:38.308935+0000","last_fullsized":"2024-03-28T00:56:38.308935+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:27:48.857368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285191+0000","last_change":"2024-03-28T00:54:32.251701+0000","last_active":"2024-03-28T00:56:38.285191+0000","last_peered":"2024-03-28T00:56:38.285191+0000","last_clean":"2024-03-28T00:56:38.285191+0000","last_became_active":"2024-03-28T00:54:32.251403+0000","last_became_peered":"2024-03-28T00:54:32.251403+0000","last_unstale":"2024-03-28T00:56:38.285191+0000","last_undegraded":"2024-03-28T00:56:38.285191+0000","last_fullsized":"2024-03-28T00:56:38.285191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:28:16.127643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350770+0000","last_change":"2024-03-28T00:54:32.806810+0000","last_active":"2024-03-28T00:56:38.350770+0000","last_peered":"2024-03-28T00:56:38.350770+0000","last_clean":"2024-03-28T00:56:38.350770+0000","last_became_active":"2024-03-28T00:54:32.806468+0000","last_became_peered":"2024-03-28T00:54:32.806468+0000","last_unstale":"2024-03-28T00:56:38.350770+0000","last_undegraded":"2024-03-28T00:56:38.350770+0000","last_fullsized":"2024-03-28T00:56:38.350770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:54:16.196032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307882+0000","last_change":"2024-03-28T00:55:37.919873+0000","last_active":"2024-03-28T00:56:38.307882+0000","last_peered":"2024-03-28T00:56:38.307882+0000","last_clean":"2024-03-28T00:56:38.307882+0000","last_became_active":"2024-03-28T00:55:37.919396+0000","last_became_peered":"2024-03-28T00:55:37.919396+0000","last_unstale":"2024-03-28T00:56:38.307882+0000","last_undegraded":"2024-03-28T00:56:38.307882+0000","last_fullsized":"2024-03-28T00:56:38.307882+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:05:59.524285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307902+0000","last_change":"2024-03-28T00:54:32.259440+0000","last_active":"2024-03-28T00:56:38.307902+0000","last_peered":"2024-03-28T00:56:38.307902+0000","last_clean":"2024-03-28T00:56:38.307902+0000","last_became_active":"2024-03-28T00:54:32.258225+0000","last_became_peered":"2024-03-28T00:54:32.258225+0000","last_unstale":"2024-03-28T00:56:38.307902+0000","last_undegraded":"2024-03-28T00:56:38.307902+0000","last_fullsized":"2024-03-28T00:56:38.307902+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:07:18.398302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308098+0000","last_change":"2024-03-28T00:54:32.806853+0000","last_active":"2024-03-28T00:56:38.308098+0000","last_peered":"2024-03-28T00:56:38.308098+0000","last_clean":"2024-03-28T00:56:38.308098+0000","last_became_active":"2024-03-28T00:54:32.806551+0000","last_became_peered":"2024-03-28T00:54:32.806551+0000","last_unstale":"2024-03-28T00:56:38.308098+0000","last_undegraded":"2024-03-28T00:56:38.308098+0000","last_fullsized":"2024-03-28T00:56:38.308098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:49:45.774252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286620+0000","last_change":"2024-03-28T00:54:36.827542+0000","last_active":"2024-03-28T00:56:38.286620+0000","last_peered":"2024-03-28T00:56:38.286620+0000","last_clean":"2024-03-28T00:56:38.286620+0000","last_became_active":"2024-03-28T00:54:36.827163+0000","last_became_peered":"2024-03-28T00:54:36.827163+0000","last_unstale":"2024-03-28T00:56:38.286620+0000","last_undegraded":"2024-03-28T00:56:38.286620+0000","last_fullsized":"2024-03-28T00:56:38.286620+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:36:15.278111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'3","reported_seq":152,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:44.395209+0000","last_change":"2024-03-28T00:54:32.253231+0000","last_active":"2024-03-28T00:56:44.395209+0000","last_peered":"2024-03-28T00:56:44.395209+0000","last_clean":"2024-03-28T00:56:44.395209+0000","last_became_active":"2024-03-28T00:54:32.252731+0000","last_became_peered":"2024-03-28T00:54:32.252731+0000","last_unstale":"2024-03-28T00:56:44.395209+0000","last_undegraded":"2024-03-28T00:56:44.395209+0000","last_fullsized":"2024-03-28T00:56:44.395209+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:05:01.716654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308421+0000","last_change":"2024-03-28T00:54:32.805073+0000","last_active":"2024-03-28T00:56:38.308421+0000","last_peered":"2024-03-28T00:56:38.308421+0000","last_clean":"2024-03-28T00:56:38.308421+0000","last_became_active":"2024-03-28T00:54:32.803690+0000","last_became_peered":"2024-03-28T00:54:32.803690+0000","last_unstale":"2024-03-28T00:56:38.308421+0000","last_undegraded":"2024-03-28T00:56:38.308421+0000","last_fullsized":"2024-03-28T00:56:38.308421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:41:16.586168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350844+0000","last_change":"2024-03-28T00:54:36.835463+0000","last_active":"2024-03-28T00:56:38.350844+0000","last_peered":"2024-03-28T00:56:38.350844+0000","last_clean":"2024-03-28T00:56:38.350844+0000","last_became_active":"2024-03-28T00:54:36.834982+0000","last_became_peered":"2024-03-28T00:54:36.834982+0000","last_unstale":"2024-03-28T00:56:38.350844+0000","last_undegraded":"2024-03-28T00:56:38.350844+0000","last_fullsized":"2024-03-28T00:56:38.350844+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:27:32.259586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307377+0000","last_change":"2024-03-28T00:54:32.248461+0000","last_active":"2024-03-28T00:56:38.307377+0000","last_peered":"2024-03-28T00:56:38.307377+0000","last_clean":"2024-03-28T00:56:38.307377+0000","last_became_active":"2024-03-28T00:54:32.248140+0000","last_became_peered":"2024-03-28T00:54:32.248140+0000","last_unstale":"2024-03-28T00:56:38.307377+0000","last_undegraded":"2024-03-28T00:56:38.307377+0000","last_fullsized":"2024-03-28T00:56:38.307377+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:13:38.784288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307746+0000","last_change":"2024-03-28T00:54:32.806651+0000","last_active":"2024-03-28T00:56:38.307746+0000","last_peered":"2024-03-28T00:56:38.307746+0000","last_clean":"2024-03-28T00:56:38.307746+0000","last_became_active":"2024-03-28T00:54:32.806329+0000","last_became_peered":"2024-03-28T00:54:32.806329+0000","last_unstale":"2024-03-28T00:56:38.307746+0000","last_undegraded":"2024-03-28T00:56:38.307746+0000","last_fullsized":"2024-03-28T00:56:38.307746+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:28:32.815396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307991+0000","last_change":"2024-03-28T00:54:36.827016+0000","last_active":"2024-03-28T00:56:38.307991+0000","last_peered":"2024-03-28T00:56:38.307991+0000","last_clean":"2024-03-28T00:56:38.307991+0000","last_became_active":"2024-03-28T00:54:36.826428+0000","last_became_peered":"2024-03-28T00:54:36.826428+0000","last_unstale":"2024-03-28T00:56:38.307991+0000","last_undegraded":"2024-03-28T00:56:38.307991+0000","last_fullsized":"2024-03-28T00:56:38.307991+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:33:01.593990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308142+0000","last_change":"2024-03-28T00:55:37.573420+0000","last_active":"2024-03-28T00:56:38.308142+0000","last_peered":"2024-03-28T00:56:38.308142+0000","last_clean":"2024-03-28T00:56:38.308142+0000","last_became_active":"2024-03-28T00:55:37.568657+0000","last_became_peered":"2024-03-28T00:55:37.568657+0000","last_unstale":"2024-03-28T00:56:38.308142+0000","last_undegraded":"2024-03-28T00:56:38.308142+0000","last_fullsized":"2024-03-28T00:56:38.308142+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:03:21.483054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308301+0000","last_change":"2024-03-28T00:55:37.919698+0000","last_active":"2024-03-28T00:56:38.308301+0000","last_peered":"2024-03-28T00:56:38.308301+0000","last_clean":"2024-03-28T00:56:38.308301+0000","last_became_active":"2024-03-28T00:55:37.919276+0000","last_became_peered":"2024-03-28T00:55:37.919276+0000","last_unstale":"2024-03-28T00:56:38.308301+0000","last_undegraded":"2024-03-28T00:56:38.308301+0000","last_fullsized":"2024-03-28T00:56:38.308301+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:12:39.042250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308070+0000","last_change":"2024-03-28T00:55:37.918600+0000","last_active":"2024-03-28T00:56:38.308070+0000","last_peered":"2024-03-28T00:56:38.308070+0000","last_clean":"2024-03-28T00:56:38.308070+0000","last_became_active":"2024-03-28T00:55:37.918088+0000","last_became_peered":"2024-03-28T00:55:37.918088+0000","last_unstale":"2024-03-28T00:56:38.308070+0000","last_undegraded":"2024-03-28T00:56:38.308070+0000","last_fullsized":"2024-03-28T00:56:38.308070+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:29:23.492261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286573+0000","last_change":"2024-03-28T00:54:32.264038+0000","last_active":"2024-03-28T00:56:38.286573+0000","last_peered":"2024-03-28T00:56:38.286573+0000","last_clean":"2024-03-28T00:56:38.286573+0000","last_became_active":"2024-03-28T00:54:32.263706+0000","last_became_peered":"2024-03-28T00:54:32.263706+0000","last_unstale":"2024-03-28T00:56:38.286573+0000","last_undegraded":"2024-03-28T00:56:38.286573+0000","last_fullsized":"2024-03-28T00:56:38.286573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:03.244556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286955+0000","last_change":"2024-03-28T00:54:32.804727+0000","last_active":"2024-03-28T00:56:38.286955+0000","last_peered":"2024-03-28T00:56:38.286955+0000","last_clean":"2024-03-28T00:56:38.286955+0000","last_became_active":"2024-03-28T00:54:32.804422+0000","last_became_peered":"2024-03-28T00:54:32.804422+0000","last_unstale":"2024-03-28T00:56:38.286955+0000","last_undegraded":"2024-03-28T00:56:38.286955+0000","last_fullsized":"2024-03-28T00:56:38.286955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:29:57.983722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.351075+0000","last_change":"2024-03-28T00:54:36.833416+0000","last_active":"2024-03-28T00:56:38.351075+0000","last_peered":"2024-03-28T00:56:38.351075+0000","last_clean":"2024-03-28T00:56:38.351075+0000","last_became_active":"2024-03-28T00:54:36.832985+0000","last_became_peered":"2024-03-28T00:54:36.832985+0000","last_unstale":"2024-03-28T00:56:38.351075+0000","last_undegraded":"2024-03-28T00:56:38.351075+0000","last_fullsized":"2024-03-28T00:56:38.351075+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:47:30.376032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308027+0000","last_change":"2024-03-28T00:54:32.256417+0000","last_active":"2024-03-28T00:56:38.308027+0000","last_peered":"2024-03-28T00:56:38.308027+0000","last_clean":"2024-03-28T00:56:38.308027+0000","last_became_active":"2024-03-28T00:54:32.256088+0000","last_became_peered":"2024-03-28T00:54:32.256088+0000","last_unstale":"2024-03-28T00:56:38.308027+0000","last_undegraded":"2024-03-28T00:56:38.308027+0000","last_fullsized":"2024-03-28T00:56:38.308027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:13:58.661597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286728+0000","last_change":"2024-03-28T00:54:32.817604+0000","last_active":"2024-03-28T00:56:38.286728+0000","last_peered":"2024-03-28T00:56:38.286728+0000","last_clean":"2024-03-28T00:56:38.286728+0000","last_became_active":"2024-03-28T00:54:32.817211+0000","last_became_peered":"2024-03-28T00:54:32.817211+0000","last_unstale":"2024-03-28T00:56:38.286728+0000","last_undegraded":"2024-03-28T00:56:38.286728+0000","last_fullsized":"2024-03-28T00:56:38.286728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:02:28.335075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350947+0000","last_change":"2024-03-28T00:54:37.638968+0000","last_active":"2024-03-28T00:56:38.350947+0000","last_peered":"2024-03-28T00:56:38.350947+0000","last_clean":"2024-03-28T00:56:38.350947+0000","last_became_active":"2024-03-28T00:54:37.638446+0000","last_became_peered":"2024-03-28T00:54:37.638446+0000","last_unstale":"2024-03-28T00:56:38.350947+0000","last_undegraded":"2024-03-28T00:56:38.350947+0000","last_fullsized":"2024-03-28T00:56:38.350947+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:54:56.348760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308145+0000","last_change":"2024-03-28T00:54:32.809822+0000","last_active":"2024-03-28T00:56:38.308145+0000","last_peered":"2024-03-28T00:56:38.308145+0000","last_clean":"2024-03-28T00:56:38.308145+0000","last_became_active":"2024-03-28T00:54:32.808284+0000","last_became_peered":"2024-03-28T00:54:32.808284+0000","last_unstale":"2024-03-28T00:56:38.308145+0000","last_undegraded":"2024-03-28T00:56:38.308145+0000","last_fullsized":"2024-03-28T00:56:38.308145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:49:11.651960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":151,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:43.393776+0000","last_change":"2024-03-28T00:54:32.261806+0000","last_active":"2024-03-28T00:56:43.393776+0000","last_peered":"2024-03-28T00:56:43.393776+0000","last_clean":"2024-03-28T00:56:43.393776+0000","last_became_active":"2024-03-28T00:54:32.261493+0000","last_became_peered":"2024-03-28T00:54:32.261493+0000","last_unstale":"2024-03-28T00:56:43.393776+0000","last_undegraded":"2024-03-28T00:56:43.393776+0000","last_fullsized":"2024-03-28T00:56:43.393776+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:36:04.283408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308358+0000","last_change":"2024-03-28T00:54:36.819332+0000","last_active":"2024-03-28T00:56:38.308358+0000","last_peered":"2024-03-28T00:56:38.308358+0000","last_clean":"2024-03-28T00:56:38.308358+0000","last_became_active":"2024-03-28T00:54:36.818932+0000","last_became_peered":"2024-03-28T00:54:36.818932+0000","last_unstale":"2024-03-28T00:56:38.308358+0000","last_undegraded":"2024-03-28T00:56:38.308358+0000","last_fullsized":"2024-03-28T00:56:38.308358+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:00:39.446738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284570+0000","last_change":"2024-03-28T00:54:32.806093+0000","last_active":"2024-03-28T00:56:38.284570+0000","last_peered":"2024-03-28T00:56:38.284570+0000","last_clean":"2024-03-28T00:56:38.284570+0000","last_became_active":"2024-03-28T00:54:32.805734+0000","last_became_peered":"2024-03-28T00:54:32.805734+0000","last_unstale":"2024-03-28T00:56:38.284570+0000","last_undegraded":"2024-03-28T00:56:38.284570+0000","last_fullsized":"2024-03-28T00:56:38.284570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:51:41.066225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284968+0000","last_change":"2024-03-28T00:54:32.245182+0000","last_active":"2024-03-28T00:56:38.284968+0000","last_peered":"2024-03-28T00:56:38.284968+0000","last_clean":"2024-03-28T00:56:38.284968+0000","last_became_active":"2024-03-28T00:54:32.244855+0000","last_became_peered":"2024-03-28T00:54:32.244855+0000","last_unstale":"2024-03-28T00:56:38.284968+0000","last_undegraded":"2024-03-28T00:56:38.284968+0000","last_fullsized":"2024-03-28T00:56:38.284968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:39:09.997199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308243+0000","last_change":"2024-03-28T00:53:20.355628+0000","last_active":"2024-03-28T00:56:38.308243+0000","last_peered":"2024-03-28T00:56:38.308243+0000","last_clean":"2024-03-28T00:56:38.308243+0000","last_became_active":"2024-03-28T00:53:20.355175+0000","last_became_peered":"2024-03-28T00:53:20.355175+0000","last_unstale":"2024-03-28T00:56:38.308243+0000","last_undegraded":"2024-03-28T00:56:38.308243+0000","last_fullsized":"2024-03-28T00:56:38.308243+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:49:57.054435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:49:57.054435+0000","last_clean_scrub_stamp":"2024-03-28T00:49:57.054435+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:55:46.753799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308032+0000","last_change":"2024-03-28T00:54:37.639760+0000","last_active":"2024-03-28T00:56:38.308032+0000","last_peered":"2024-03-28T00:56:38.308032+0000","last_clean":"2024-03-28T00:56:38.308032+0000","last_became_active":"2024-03-28T00:54:37.639287+0000","last_became_peered":"2024-03-28T00:54:37.639287+0000","last_unstale":"2024-03-28T00:56:38.308032+0000","last_undegraded":"2024-03-28T00:56:38.308032+0000","last_fullsized":"2024-03-28T00:56:38.308032+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:50:02.840312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350597+0000","last_change":"2024-03-28T00:54:32.807764+0000","last_active":"2024-03-28T00:56:38.350597+0000","last_peered":"2024-03-28T00:56:38.350597+0000","last_clean":"2024-03-28T00:56:38.350597+0000","last_became_active":"2024-03-28T00:54:32.807441+0000","last_became_peered":"2024-03-28T00:54:32.807441+0000","last_unstale":"2024-03-28T00:56:38.350597+0000","last_undegraded":"2024-03-28T00:56:38.350597+0000","last_fullsized":"2024-03-28T00:56:38.350597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:37:40.540514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308361+0000","last_change":"2024-03-28T00:54:32.259192+0000","last_active":"2024-03-28T00:56:38.308361+0000","last_peered":"2024-03-28T00:56:38.308361+0000","last_clean":"2024-03-28T00:56:38.308361+0000","last_became_active":"2024-03-28T00:54:32.258880+0000","last_became_peered":"2024-03-28T00:54:32.258880+0000","last_unstale":"2024-03-28T00:56:38.308361+0000","last_undegraded":"2024-03-28T00:56:38.308361+0000","last_fullsized":"2024-03-28T00:56:38.308361+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:20:11.384633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307815+0000","last_change":"2024-03-28T00:54:36.822268+0000","last_active":"2024-03-28T00:56:38.307815+0000","last_peered":"2024-03-28T00:56:38.307815+0000","last_clean":"2024-03-28T00:56:38.307815+0000","last_became_active":"2024-03-28T00:54:36.821773+0000","last_became_peered":"2024-03-28T00:54:36.821773+0000","last_unstale":"2024-03-28T00:56:38.307815+0000","last_undegraded":"2024-03-28T00:56:38.307815+0000","last_fullsized":"2024-03-28T00:56:38.307815+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:56:11.206975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308433+0000","last_change":"2024-03-28T00:54:32.256133+0000","last_active":"2024-03-28T00:56:38.308433+0000","last_peered":"2024-03-28T00:56:38.308433+0000","last_clean":"2024-03-28T00:56:38.308433+0000","last_became_active":"2024-03-28T00:54:32.250535+0000","last_became_peered":"2024-03-28T00:54:32.250535+0000","last_unstale":"2024-03-28T00:56:38.308433+0000","last_undegraded":"2024-03-28T00:56:38.308433+0000","last_fullsized":"2024-03-28T00:56:38.308433+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:18:50.874181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.351152+0000","last_change":"2024-03-28T00:54:32.808883+0000","last_active":"2024-03-28T00:56:38.351152+0000","last_peered":"2024-03-28T00:56:38.351152+0000","last_clean":"2024-03-28T00:56:38.351152+0000","last_became_active":"2024-03-28T00:54:32.808587+0000","last_became_peered":"2024-03-28T00:54:32.808587+0000","last_unstale":"2024-03-28T00:56:38.351152+0000","last_undegraded":"2024-03-28T00:56:38.351152+0000","last_fullsized":"2024-03-28T00:56:38.351152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:09:55.164876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284641+0000","last_change":"2024-03-28T00:54:37.637743+0000","last_active":"2024-03-28T00:56:38.284641+0000","last_peered":"2024-03-28T00:56:38.284641+0000","last_clean":"2024-03-28T00:56:38.284641+0000","last_became_active":"2024-03-28T00:54:37.637280+0000","last_became_peered":"2024-03-28T00:54:37.637280+0000","last_unstale":"2024-03-28T00:56:38.284641+0000","last_undegraded":"2024-03-28T00:56:38.284641+0000","last_fullsized":"2024-03-28T00:56:38.284641+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:40:51.431757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'9","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:45.255094+0000","last_change":"2024-03-28T00:54:32.246444+0000","last_active":"2024-03-28T00:56:45.255094+0000","last_peered":"2024-03-28T00:56:45.255094+0000","last_clean":"2024-03-28T00:56:45.255094+0000","last_became_active":"2024-03-28T00:54:32.245106+0000","last_became_peered":"2024-03-28T00:54:32.245106+0000","last_unstale":"2024-03-28T00:56:45.255094+0000","last_undegraded":"2024-03-28T00:56:45.255094+0000","last_fullsized":"2024-03-28T00:56:45.255094+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:13:45.123625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285364+0000","last_change":"2024-03-28T00:54:32.812851+0000","last_active":"2024-03-28T00:56:38.285364+0000","last_peered":"2024-03-28T00:56:38.285364+0000","last_clean":"2024-03-28T00:56:38.285364+0000","last_became_active":"2024-03-28T00:54:32.812518+0000","last_became_peered":"2024-03-28T00:54:32.812518+0000","last_unstale":"2024-03-28T00:56:38.285364+0000","last_undegraded":"2024-03-28T00:56:38.285364+0000","last_fullsized":"2024-03-28T00:56:38.285364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:08:41.518888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284865+0000","last_change":"2024-03-28T00:54:36.818709+0000","last_active":"2024-03-28T00:56:38.284865+0000","last_peered":"2024-03-28T00:56:38.284865+0000","last_clean":"2024-03-28T00:56:38.284865+0000","last_became_active":"2024-03-28T00:54:36.818304+0000","last_became_peered":"2024-03-28T00:54:36.818304+0000","last_unstale":"2024-03-28T00:56:38.284865+0000","last_undegraded":"2024-03-28T00:56:38.284865+0000","last_fullsized":"2024-03-28T00:56:38.284865+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:03:07.642685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285251+0000","last_change":"2024-03-28T00:54:32.256031+0000","last_active":"2024-03-28T00:56:38.285251+0000","last_peered":"2024-03-28T00:56:38.285251+0000","last_clean":"2024-03-28T00:56:38.285251+0000","last_became_active":"2024-03-28T00:54:32.255597+0000","last_became_peered":"2024-03-28T00:54:32.255597+0000","last_unstale":"2024-03-28T00:56:38.285251+0000","last_undegraded":"2024-03-28T00:56:38.285251+0000","last_fullsized":"2024-03-28T00:56:38.285251+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:31:10.837212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303584+0000","last_change":"2024-03-28T00:54:32.816961+0000","last_active":"2024-03-28T00:56:38.303584+0000","last_peered":"2024-03-28T00:56:38.303584+0000","last_clean":"2024-03-28T00:56:38.303584+0000","last_became_active":"2024-03-28T00:54:32.816408+0000","last_became_peered":"2024-03-28T00:54:32.816408+0000","last_unstale":"2024-03-28T00:56:38.303584+0000","last_undegraded":"2024-03-28T00:56:38.303584+0000","last_fullsized":"2024-03-28T00:56:38.303584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:08:57.441492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285039+0000","last_change":"2024-03-28T00:55:37.918692+0000","last_active":"2024-03-28T00:56:38.285039+0000","last_peered":"2024-03-28T00:56:38.285039+0000","last_clean":"2024-03-28T00:56:38.285039+0000","last_became_active":"2024-03-28T00:55:37.918195+0000","last_became_peered":"2024-03-28T00:55:37.918195+0000","last_unstale":"2024-03-28T00:56:38.285039+0000","last_undegraded":"2024-03-28T00:56:38.285039+0000","last_fullsized":"2024-03-28T00:56:38.285039+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:01:06.012325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.306972+0000","last_change":"2024-03-28T00:55:37.570700+0000","last_active":"2024-03-28T00:56:38.306972+0000","last_peered":"2024-03-28T00:56:38.306972+0000","last_clean":"2024-03-28T00:56:38.306972+0000","last_became_active":"2024-03-28T00:55:37.566298+0000","last_became_peered":"2024-03-28T00:55:37.566298+0000","last_unstale":"2024-03-28T00:56:38.306972+0000","last_undegraded":"2024-03-28T00:56:38.306972+0000","last_fullsized":"2024-03-28T00:56:38.306972+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:46:48.352817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307628+0000","last_change":"2024-03-28T00:54:32.811592+0000","last_active":"2024-03-28T00:56:38.307628+0000","last_peered":"2024-03-28T00:56:38.307628+0000","last_clean":"2024-03-28T00:56:38.307628+0000","last_became_active":"2024-03-28T00:54:32.811228+0000","last_became_peered":"2024-03-28T00:54:32.811228+0000","last_unstale":"2024-03-28T00:56:38.307628+0000","last_undegraded":"2024-03-28T00:56:38.307628+0000","last_fullsized":"2024-03-28T00:56:38.307628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:14:19.585237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.286681+0000","last_change":"2024-03-28T00:54:36.822501+0000","last_active":"2024-03-28T00:56:38.286681+0000","last_peered":"2024-03-28T00:56:38.286681+0000","last_clean":"2024-03-28T00:56:38.286681+0000","last_became_active":"2024-03-28T00:54:36.821630+0000","last_became_peered":"2024-03-28T00:54:36.821630+0000","last_unstale":"2024-03-28T00:56:38.286681+0000","last_undegraded":"2024-03-28T00:56:38.286681+0000","last_fullsized":"2024-03-28T00:56:38.286681+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:20:52.247055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284556+0000","last_change":"2024-03-28T00:54:32.247328+0000","last_active":"2024-03-28T00:56:38.284556+0000","last_peered":"2024-03-28T00:56:38.284556+0000","last_clean":"2024-03-28T00:56:38.284556+0000","last_became_active":"2024-03-28T00:54:32.247003+0000","last_became_peered":"2024-03-28T00:54:32.247003+0000","last_unstale":"2024-03-28T00:56:38.284556+0000","last_undegraded":"2024-03-28T00:56:38.284556+0000","last_fullsized":"2024-03-28T00:56:38.284556+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:33:09.764946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308401+0000","last_change":"2024-03-28T00:54:32.805977+0000","last_active":"2024-03-28T00:56:38.308401+0000","last_peered":"2024-03-28T00:56:38.308401+0000","last_clean":"2024-03-28T00:56:38.308401+0000","last_became_active":"2024-03-28T00:54:32.805636+0000","last_became_peered":"2024-03-28T00:54:32.805636+0000","last_unstale":"2024-03-28T00:56:38.308401+0000","last_undegraded":"2024-03-28T00:56:38.308401+0000","last_fullsized":"2024-03-28T00:56:38.308401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:04:42.901732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284987+0000","last_change":"2024-03-28T00:54:36.822103+0000","last_active":"2024-03-28T00:56:38.284987+0000","last_peered":"2024-03-28T00:56:38.284987+0000","last_clean":"2024-03-28T00:56:38.284987+0000","last_became_active":"2024-03-28T00:54:36.821695+0000","last_became_peered":"2024-03-28T00:54:36.821695+0000","last_unstale":"2024-03-28T00:56:38.284987+0000","last_undegraded":"2024-03-28T00:56:38.284987+0000","last_fullsized":"2024-03-28T00:56:38.284987+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:45:00.613955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285112+0000","last_change":"2024-03-28T00:54:32.251523+0000","last_active":"2024-03-28T00:56:38.285112+0000","last_peered":"2024-03-28T00:56:38.285112+0000","last_clean":"2024-03-28T00:56:38.285112+0000","last_became_active":"2024-03-28T00:54:32.251223+0000","last_became_peered":"2024-03-28T00:54:32.251223+0000","last_unstale":"2024-03-28T00:56:38.285112+0000","last_undegraded":"2024-03-28T00:56:38.285112+0000","last_fullsized":"2024-03-28T00:56:38.285112+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:08:20.489848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284729+0000","last_change":"2024-03-28T00:54:32.817641+0000","last_active":"2024-03-28T00:56:38.284729+0000","last_peered":"2024-03-28T00:56:38.284729+0000","last_clean":"2024-03-28T00:56:38.284729+0000","last_became_active":"2024-03-28T00:54:32.817252+0000","last_became_peered":"2024-03-28T00:54:32.817252+0000","last_unstale":"2024-03-28T00:56:38.284729+0000","last_undegraded":"2024-03-28T00:56:38.284729+0000","last_fullsized":"2024-03-28T00:56:38.284729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:54:48.275005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285020+0000","last_change":"2024-03-28T00:54:37.638990+0000","last_active":"2024-03-28T00:56:38.285020+0000","last_peered":"2024-03-28T00:56:38.285020+0000","last_clean":"2024-03-28T00:56:38.285020+0000","last_became_active":"2024-03-28T00:54:37.638538+0000","last_became_peered":"2024-03-28T00:54:37.638538+0000","last_unstale":"2024-03-28T00:56:38.285020+0000","last_undegraded":"2024-03-28T00:56:38.285020+0000","last_fullsized":"2024-03-28T00:56:38.285020+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:28:10.791805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308482+0000","last_change":"2024-03-28T00:54:32.253254+0000","last_active":"2024-03-28T00:56:38.308482+0000","last_peered":"2024-03-28T00:56:38.308482+0000","last_clean":"2024-03-28T00:56:38.308482+0000","last_became_active":"2024-03-28T00:54:32.252879+0000","last_became_peered":"2024-03-28T00:54:32.252879+0000","last_unstale":"2024-03-28T00:56:38.308482+0000","last_undegraded":"2024-03-28T00:56:38.308482+0000","last_fullsized":"2024-03-28T00:56:38.308482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:59:51.039215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308081+0000","last_change":"2024-03-28T00:54:32.811288+0000","last_active":"2024-03-28T00:56:38.308081+0000","last_peered":"2024-03-28T00:56:38.308081+0000","last_clean":"2024-03-28T00:56:38.308081+0000","last_became_active":"2024-03-28T00:54:32.810968+0000","last_became_peered":"2024-03-28T00:54:32.810968+0000","last_unstale":"2024-03-28T00:56:38.308081+0000","last_undegraded":"2024-03-28T00:56:38.308081+0000","last_fullsized":"2024-03-28T00:56:38.308081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:16:56.113046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307049+0000","last_change":"2024-03-28T00:54:36.830714+0000","last_active":"2024-03-28T00:56:38.307049+0000","last_peered":"2024-03-28T00:56:38.307049+0000","last_clean":"2024-03-28T00:56:38.307049+0000","last_became_active":"2024-03-28T00:54:36.830253+0000","last_became_peered":"2024-03-28T00:54:36.830253+0000","last_unstale":"2024-03-28T00:56:38.307049+0000","last_undegraded":"2024-03-28T00:56:38.307049+0000","last_fullsized":"2024-03-28T00:56:38.307049+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:24:43.696150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285475+0000","last_change":"2024-03-28T00:54:32.254009+0000","last_active":"2024-03-28T00:56:38.285475+0000","last_peered":"2024-03-28T00:56:38.285475+0000","last_clean":"2024-03-28T00:56:38.285475+0000","last_became_active":"2024-03-28T00:54:32.253697+0000","last_became_peered":"2024-03-28T00:54:32.253697+0000","last_unstale":"2024-03-28T00:56:38.285475+0000","last_undegraded":"2024-03-28T00:56:38.285475+0000","last_fullsized":"2024-03-28T00:56:38.285475+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:17:44.629367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308505+0000","last_change":"2024-03-28T00:54:32.812828+0000","last_active":"2024-03-28T00:56:38.308505+0000","last_peered":"2024-03-28T00:56:38.308505+0000","last_clean":"2024-03-28T00:56:38.308505+0000","last_became_active":"2024-03-28T00:54:32.811591+0000","last_became_peered":"2024-03-28T00:54:32.811591+0000","last_unstale":"2024-03-28T00:56:38.308505+0000","last_undegraded":"2024-03-28T00:56:38.308505+0000","last_fullsized":"2024-03-28T00:56:38.308505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:04.613623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303210+0000","last_change":"2024-03-28T00:54:36.823284+0000","last_active":"2024-03-28T00:56:38.303210+0000","last_peered":"2024-03-28T00:56:38.303210+0000","last_clean":"2024-03-28T00:56:38.303210+0000","last_became_active":"2024-03-28T00:54:36.822883+0000","last_became_peered":"2024-03-28T00:54:36.822883+0000","last_unstale":"2024-03-28T00:56:38.303210+0000","last_undegraded":"2024-03-28T00:56:38.303210+0000","last_fullsized":"2024-03-28T00:56:38.303210+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:37:10.266063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308755+0000","last_change":"2024-03-28T00:54:32.253946+0000","last_active":"2024-03-28T00:56:38.308755+0000","last_peered":"2024-03-28T00:56:38.308755+0000","last_clean":"2024-03-28T00:56:38.308755+0000","last_became_active":"2024-03-28T00:54:32.253650+0000","last_became_peered":"2024-03-28T00:54:32.253650+0000","last_unstale":"2024-03-28T00:56:38.308755+0000","last_undegraded":"2024-03-28T00:56:38.308755+0000","last_fullsized":"2024-03-28T00:56:38.308755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:11:44.930074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308286+0000","last_change":"2024-03-28T00:54:32.815410+0000","last_active":"2024-03-28T00:56:38.308286+0000","last_peered":"2024-03-28T00:56:38.308286+0000","last_clean":"2024-03-28T00:56:38.308286+0000","last_became_active":"2024-03-28T00:54:32.815088+0000","last_became_peered":"2024-03-28T00:54:32.815088+0000","last_unstale":"2024-03-28T00:56:38.308286+0000","last_undegraded":"2024-03-28T00:56:38.308286+0000","last_fullsized":"2024-03-28T00:56:38.308286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:16:24.862822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307742+0000","last_change":"2024-03-28T00:54:36.823182+0000","last_active":"2024-03-28T00:56:38.307742+0000","last_peered":"2024-03-28T00:56:38.307742+0000","last_clean":"2024-03-28T00:56:38.307742+0000","last_became_active":"2024-03-28T00:54:36.821446+0000","last_became_peered":"2024-03-28T00:54:36.821446+0000","last_unstale":"2024-03-28T00:56:38.307742+0000","last_undegraded":"2024-03-28T00:56:38.307742+0000","last_fullsized":"2024-03-28T00:56:38.307742+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:35.792977+0000","last_clean_scrub_stamp":"2024-03-28T00:54:35.792977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:45:20.030213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350942+0000","last_change":"2024-03-28T00:54:32.257412+0000","last_active":"2024-03-28T00:56:38.350942+0000","last_peered":"2024-03-28T00:56:38.350942+0000","last_clean":"2024-03-28T00:56:38.350942+0000","last_became_active":"2024-03-28T00:54:32.257115+0000","last_became_peered":"2024-03-28T00:54:32.257115+0000","last_unstale":"2024-03-28T00:56:38.350942+0000","last_undegraded":"2024-03-28T00:56:38.350942+0000","last_fullsized":"2024-03-28T00:56:38.350942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:31.232696+0000","last_clean_scrub_stamp":"2024-03-28T00:54:31.232696+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:04:29.850867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308522+0000","last_change":"2024-03-28T00:54:32.805879+0000","last_active":"2024-03-28T00:56:38.308522+0000","last_peered":"2024-03-28T00:56:38.308522+0000","last_clean":"2024-03-28T00:56:38.308522+0000","last_became_active":"2024-03-28T00:54:32.805507+0000","last_became_peered":"2024-03-28T00:54:32.805507+0000","last_unstale":"2024-03-28T00:56:38.308522+0000","last_undegraded":"2024-03-28T00:56:38.308522+0000","last_fullsized":"2024-03-28T00:56:38.308522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:51:51.229085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308515+0000","last_change":"2024-03-28T00:54:32.807219+0000","last_active":"2024-03-28T00:56:38.308515+0000","last_peered":"2024-03-28T00:56:38.308515+0000","last_clean":"2024-03-28T00:56:38.308515+0000","last_became_active":"2024-03-28T00:54:32.806856+0000","last_became_peered":"2024-03-28T00:54:32.806856+0000","last_unstale":"2024-03-28T00:56:38.308515+0000","last_undegraded":"2024-03-28T00:56:38.308515+0000","last_fullsized":"2024-03-28T00:56:38.308515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:05:49.914636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285057+0000","last_change":"2024-03-28T00:55:37.569156+0000","last_active":"2024-03-28T00:56:38.285057+0000","last_peered":"2024-03-28T00:56:38.285057+0000","last_clean":"2024-03-28T00:56:38.285057+0000","last_became_active":"2024-03-28T00:55:37.568775+0000","last_became_peered":"2024-03-28T00:55:37.568775+0000","last_unstale":"2024-03-28T00:56:38.285057+0000","last_undegraded":"2024-03-28T00:56:38.285057+0000","last_fullsized":"2024-03-28T00:56:38.285057+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:00:21.458550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285006+0000","last_change":"2024-03-28T00:54:32.813921+0000","last_active":"2024-03-28T00:56:38.285006+0000","last_peered":"2024-03-28T00:56:38.285006+0000","last_clean":"2024-03-28T00:56:38.285006+0000","last_became_active":"2024-03-28T00:54:32.812877+0000","last_became_peered":"2024-03-28T00:54:32.812877+0000","last_unstale":"2024-03-28T00:56:38.285006+0000","last_undegraded":"2024-03-28T00:56:38.285006+0000","last_fullsized":"2024-03-28T00:56:38.285006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:26:54.305074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.303553+0000","last_change":"2024-03-28T00:54:32.815725+0000","last_active":"2024-03-28T00:56:38.303553+0000","last_peered":"2024-03-28T00:56:38.303553+0000","last_clean":"2024-03-28T00:56:38.303553+0000","last_became_active":"2024-03-28T00:54:32.815444+0000","last_became_peered":"2024-03-28T00:54:32.815444+0000","last_unstale":"2024-03-28T00:56:38.303553+0000","last_undegraded":"2024-03-28T00:56:38.303553+0000","last_fullsized":"2024-03-28T00:56:38.303553+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:23:17.326589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.351026+0000","last_change":"2024-03-28T00:54:32.806595+0000","last_active":"2024-03-28T00:56:38.351026+0000","last_peered":"2024-03-28T00:56:38.351026+0000","last_clean":"2024-03-28T00:56:38.351026+0000","last_became_active":"2024-03-28T00:54:32.806256+0000","last_became_peered":"2024-03-28T00:54:32.806256+0000","last_unstale":"2024-03-28T00:56:38.351026+0000","last_undegraded":"2024-03-28T00:56:38.351026+0000","last_fullsized":"2024-03-28T00:56:38.351026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:04:22.551511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308462+0000","last_change":"2024-03-28T00:54:32.805000+0000","last_active":"2024-03-28T00:56:38.308462+0000","last_peered":"2024-03-28T00:56:38.308462+0000","last_clean":"2024-03-28T00:56:38.308462+0000","last_became_active":"2024-03-28T00:54:32.804637+0000","last_became_peered":"2024-03-28T00:54:32.804637+0000","last_unstale":"2024-03-28T00:56:38.308462+0000","last_undegraded":"2024-03-28T00:56:38.308462+0000","last_fullsized":"2024-03-28T00:56:38.308462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:59:46.888014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285395+0000","last_change":"2024-03-28T00:54:32.802783+0000","last_active":"2024-03-28T00:56:38.285395+0000","last_peered":"2024-03-28T00:56:38.285395+0000","last_clean":"2024-03-28T00:56:38.285395+0000","last_became_active":"2024-03-28T00:54:32.802457+0000","last_became_peered":"2024-03-28T00:54:32.802457+0000","last_unstale":"2024-03-28T00:56:38.285395+0000","last_undegraded":"2024-03-28T00:56:38.285395+0000","last_fullsized":"2024-03-28T00:56:38.285395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:28:24.146047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.285531+0000","last_change":"2024-03-28T00:54:32.810179+0000","last_active":"2024-03-28T00:56:38.285531+0000","last_peered":"2024-03-28T00:56:38.285531+0000","last_clean":"2024-03-28T00:56:38.285531+0000","last_became_active":"2024-03-28T00:54:32.809009+0000","last_became_peered":"2024-03-28T00:54:32.809009+0000","last_unstale":"2024-03-28T00:56:38.285531+0000","last_undegraded":"2024-03-28T00:56:38.285531+0000","last_fullsized":"2024-03-28T00:56:38.285531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:08:44.184910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308081+0000","last_change":"2024-03-28T00:54:32.810084+0000","last_active":"2024-03-28T00:56:38.308081+0000","last_peered":"2024-03-28T00:56:38.308081+0000","last_clean":"2024-03-28T00:56:38.308081+0000","last_became_active":"2024-03-28T00:54:32.809386+0000","last_became_peered":"2024-03-28T00:54:32.809386+0000","last_unstale":"2024-03-28T00:56:38.308081+0000","last_undegraded":"2024-03-28T00:56:38.308081+0000","last_fullsized":"2024-03-28T00:56:38.308081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:49:13.754797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.284532+0000","last_change":"2024-03-28T00:54:32.809933+0000","last_active":"2024-03-28T00:56:38.284532+0000","last_peered":"2024-03-28T00:56:38.284532+0000","last_clean":"2024-03-28T00:56:38.284532+0000","last_became_active":"2024-03-28T00:54:32.808974+0000","last_became_peered":"2024-03-28T00:54:32.808974+0000","last_unstale":"2024-03-28T00:56:38.284532+0000","last_undegraded":"2024-03-28T00:56:38.284532+0000","last_fullsized":"2024-03-28T00:56:38.284532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:23:59.317887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.307220+0000","last_change":"2024-03-28T00:54:32.808764+0000","last_active":"2024-03-28T00:56:38.307220+0000","last_peered":"2024-03-28T00:56:38.307220+0000","last_clean":"2024-03-28T00:56:38.307220+0000","last_became_active":"2024-03-28T00:54:32.808034+0000","last_became_peered":"2024-03-28T00:54:32.808034+0000","last_unstale":"2024-03-28T00:56:38.307220+0000","last_undegraded":"2024-03-28T00:56:38.307220+0000","last_fullsized":"2024-03-28T00:56:38.307220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:27:12.762520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.350903+0000","last_change":"2024-03-28T00:54:32.805470+0000","last_active":"2024-03-28T00:56:38.350903+0000","last_peered":"2024-03-28T00:56:38.350903+0000","last_clean":"2024-03-28T00:56:38.350903+0000","last_became_active":"2024-03-28T00:54:32.805146+0000","last_became_peered":"2024-03-28T00:54:32.805146+0000","last_unstale":"2024-03-28T00:56:38.350903+0000","last_undegraded":"2024-03-28T00:56:38.350903+0000","last_fullsized":"2024-03-28T00:56:38.350903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:24:14.297966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T00:56:38.308078+0000","last_change":"2024-03-28T00:55:37.566696+0000","last_active":"2024-03-28T00:56:38.308078+0000","last_peered":"2024-03-28T00:56:38.308078+0000","last_clean":"2024-03-28T00:56:38.308078+0000","last_became_active":"2024-03-28T00:55:37.566334+0000","last_became_peered":"2024-03-28T00:55:37.566334+0000","last_unstale":"2024-03-28T00:56:38.308078+0000","last_undegraded":"2024-03-28T00:56:38.308078+0000","last_fullsized":"2024-03-28T00:56:38.308078+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T00:54:32.234173+0000","last_clean_scrub_stamp":"2024-03-28T00:54:32.234173+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:41:37.752201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282084,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":77906,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":716,"num_read_kb":439,"num_write":175,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450560,"data_stored":226947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153,"ondisk_log_size":153,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383720,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":529042,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645358,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32336,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712816,"statfs":{"total":95995035648,"available":95961923584,"internally_reserved":0,"allocated":1187840,"data_stored":1189353,"data_compressed":65432,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808882,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":533352,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070520,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":529154,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332156,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1097728,"data_stored":1125226,"data_compressed":65432,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495681,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":524432,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":591335,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1052672,"data_stored":525820,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":599988,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182480,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1040384,"data_stored":524626,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920844,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":529082,"data_compressed":29546,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":1121122,"data_compressed":65432,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34218,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34218,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34218,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35886,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":66993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":75646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":74643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T00:56:50.029 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T00:56:50.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.164+0000 7fd0fc53b640 1 -- 172.21.15.169:0/3539116214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0f41028b0 msgr2=0x7fd0f4102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:50.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.164+0000 7fd0fc53b640 1 --2- 172.21.15.169:0/3539116214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0f41028b0 0x7fd0f4102d30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0009900 tx=0x7fd0f002bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:50.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.164+0000 7fd0fc53b640 1 -- 172.21.15.169:0/3539116214 shutdown_connections 2024-03-28T00:56:50.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.164+0000 7fd0fc53b640 1 --2- 172.21.15.169:0/3539116214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0f4103270 0x7fd0f4109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.164+0000 7fd0fc53b640 1 --2- 172.21.15.169:0/3539116214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0f41028b0 0x7fd0f4102d30 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.164+0000 7fd0fc53b640 1 --2- 172.21.15.169:0/3539116214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0f41016c0 0x7fd0f4101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.164+0000 7fd0fc53b640 1 -- 172.21.15.169:0/3539116214 >> 172.21.15.169:0/3539116214 conn(0x7fd0f40fcc50 msgr2=0x7fd0f40ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:50.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.164+0000 7fd0fc53b640 1 -- 172.21.15.169:0/3539116214 shutdown_connections 2024-03-28T00:56:50.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.164+0000 7fd0fc53b640 1 -- 172.21.15.169:0/3539116214 wait complete. 2024-03-28T00:56:50.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.164+0000 7fd0fc53b640 1 Processor -- start 2024-03-28T00:56:50.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0fc53b640 1 -- start start 2024-03-28T00:56:50.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0fc53b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0f41016c0 0x7fd0f4075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:50.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0fc53b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0f41028b0 0x7fd0f4073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:50.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0fc53b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0f4103270 0x7fd0f4073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:50.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0fc53b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f410d320 con 0x7fd0f41016c0 2024-03-28T00:56:50.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0fc53b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f410d1a0 con 0x7fd0f41028b0 2024-03-28T00:56:50.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0fc53b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f410d4a0 con 0x7fd0f4103270 2024-03-28T00:56:50.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0fa2d7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0f41016c0 0x7fd0f4075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:50.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0fa2d7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0f41016c0 0x7fd0f4075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34854/0 (socket says 172.21.15.169:34854) 2024-03-28T00:56:50.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0fa2d7640 1 -- 172.21.15.169:0/1832829248 learned_addr learned my addr 172.21.15.169:0/1832829248 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:50.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0faad8640 1 --2- 172.21.15.169:0/1832829248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0f4103270 0x7fd0f4073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:50.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0f9ad6640 1 --2- 172.21.15.169:0/1832829248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0f41028b0 0x7fd0f4073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:50.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0fa2d7640 1 -- 172.21.15.169:0/1832829248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0f4103270 msgr2=0x7fd0f4073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:50.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0fa2d7640 1 --2- 172.21.15.169:0/1832829248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0f4103270 0x7fd0f4073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0fa2d7640 1 -- 172.21.15.169:0/1832829248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0f41028b0 msgr2=0x7fd0f4073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:50.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0fa2d7640 1 --2- 172.21.15.169:0/1832829248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0f41028b0 0x7fd0f4073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0fa2d7640 1 -- 172.21.15.169:0/1832829248 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f4074860 con 0x7fd0f41016c0 2024-03-28T00:56:50.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0f9ad6640 1 --2- 172.21.15.169:0/1832829248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0f41028b0 0x7fd0f4073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:50.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0fa2d7640 1 --2- 172.21.15.169:0/1832829248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0f41016c0 0x7fd0f4075450 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400e9c0 tx=0x7fd0e400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:50.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0eb7fe640 1 -- 172.21.15.169:0/1832829248 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e400cd80 con 0x7fd0f41016c0 2024-03-28T00:56:50.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0eb7fe640 1 -- 172.21.15.169:0/1832829248 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd0e4010910 con 0x7fd0f41016c0 2024-03-28T00:56:50.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0eb7fe640 1 -- 172.21.15.169:0/1832829248 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4018950 con 0x7fd0f41016c0 2024-03-28T00:56:50.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.165+0000 7fd0fc53b640 1 -- 172.21.15.169:0/1832829248 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f41a1b10 con 0x7fd0f41016c0 2024-03-28T00:56:50.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.166+0000 7fd0fc53b640 1 -- 172.21.15.169:0/1832829248 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f41a2030 con 0x7fd0f41016c0 2024-03-28T00:56:50.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.167+0000 7fd0fc53b640 1 -- 172.21.15.169:0/1832829248 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0f4108a30 con 0x7fd0f41016c0 2024-03-28T00:56:50.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.168+0000 7fd0eb7fe640 1 -- 172.21.15.169:0/1832829248 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fd0e4010ac0 con 0x7fd0f41016c0 2024-03-28T00:56:50.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.168+0000 7fd0eb7fe640 1 --2- 172.21.15.169:0/1832829248 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd0e00a2890 0x7fd0e00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:50.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.168+0000 7fd0f9ad6640 1 --2- 172.21.15.169:0/1832829248 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd0e00a2890 0x7fd0e00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:50.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.168+0000 7fd0eb7fe640 1 -- 172.21.15.169:0/1832829248 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fd0e40cf3b0 con 0x7fd0f41016c0 2024-03-28T00:56:50.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.169+0000 7fd0f9ad6640 1 --2- 172.21.15.169:0/1832829248 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd0e00a2890 0x7fd0e00a4d50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd0f002c7b0 tx=0x7fd0f0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:50.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.173+0000 7fd0eb7fe640 1 -- 172.21.15.169:0/1832829248 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd0e4053740 con 0x7fd0f41016c0 2024-03-28T00:56:50.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.350+0000 7fd0fc53b640 1 -- 172.21.15.169:0/1832829248 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0f40622a0 con 0x7fd0f41016c0 2024-03-28T00:56:50.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.351+0000 7fd0eb7fe640 1 -- 172.21.15.169:0/1832829248 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+24823 (secure 0 0 0) 0x7fd0e409bf50 con 0x7fd0f41016c0 2024-03-28T00:56:50.354 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:50.355 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":94,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:56:38.281826+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9},{"from":5,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:56:50.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.353+0000 7fd0fc53b640 1 -- 172.21.15.169:0/1832829248 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd0e00a2890 msgr2=0x7fd0e00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:50.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.353+0000 7fd0fc53b640 1 --2- 172.21.15.169:0/1832829248 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd0e00a2890 0x7fd0e00a4d50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd0f002c7b0 tx=0x7fd0f0005fd0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.354+0000 7fd0fc53b640 1 -- 172.21.15.169:0/1832829248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0f41016c0 msgr2=0x7fd0f4075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:50.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.354+0000 7fd0fc53b640 1 --2- 172.21.15.169:0/1832829248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0f41016c0 0x7fd0f4075450 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400e9c0 tx=0x7fd0e400edc0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.354+0000 7fd0fc53b640 1 -- 172.21.15.169:0/1832829248 shutdown_connections 2024-03-28T00:56:50.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.354+0000 7fd0fc53b640 1 --2- 172.21.15.169:0/1832829248 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd0e00a2890 0x7fd0e00a4d50 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.354+0000 7fd0fc53b640 1 --2- 172.21.15.169:0/1832829248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0f4103270 0x7fd0f4073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.354+0000 7fd0fc53b640 1 --2- 172.21.15.169:0/1832829248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0f41028b0 0x7fd0f4073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.354+0000 7fd0fc53b640 1 --2- 172.21.15.169:0/1832829248 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0f41016c0 0x7fd0f4075450 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.354+0000 7fd0fc53b640 1 -- 172.21.15.169:0/1832829248 >> 172.21.15.169:0/1832829248 conn(0x7fd0f40fcc50 msgr2=0x7fd0f4105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:50.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.354+0000 7fd0fc53b640 1 -- 172.21.15.169:0/1832829248 shutdown_connections 2024-03-28T00:56:50.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.354+0000 7fd0fc53b640 1 -- 172.21.15.169:0/1832829248 wait complete. 2024-03-28T00:56:50.368 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T00:49:56.339525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T00:54:31.182562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T00:54:31.780429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T00:54:35.035724+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T00:56:50.368 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T00:56:50.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.471+0000 7f9bfad1b640 1 -- 172.21.15.169:0/2490529501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bf4103260 msgr2=0x7f9bf4109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:50.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.471+0000 7f9bfad1b640 1 --2- 172.21.15.169:0/2490529501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bf4103260 0x7f9bf4109af0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009860 tx=0x7f9be402bb70 comp rx=0 tx=0).stop 2024-03-28T00:56:50.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.471+0000 7f9bfad1b640 1 -- 172.21.15.169:0/2490529501 shutdown_connections 2024-03-28T00:56:50.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.471+0000 7f9bfad1b640 1 --2- 172.21.15.169:0/2490529501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bf4103260 0x7f9bf4109af0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.471+0000 7f9bfad1b640 1 --2- 172.21.15.169:0/2490529501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bf41028a0 0x7f9bf4102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.471+0000 7f9bfad1b640 1 --2- 172.21.15.169:0/2490529501 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bf41016b0 0x7f9bf4101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.471+0000 7f9bfad1b640 1 -- 172.21.15.169:0/2490529501 >> 172.21.15.169:0/2490529501 conn(0x7f9bf40fcc40 msgr2=0x7f9bf40ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:50.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.471+0000 7f9bfad1b640 1 -- 172.21.15.169:0/2490529501 shutdown_connections 2024-03-28T00:56:50.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.471+0000 7f9bfad1b640 1 -- 172.21.15.169:0/2490529501 wait complete. 2024-03-28T00:56:50.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.471+0000 7f9bfad1b640 1 Processor -- start 2024-03-28T00:56:50.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.471+0000 7f9bfad1b640 1 -- start start 2024-03-28T00:56:50.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.471+0000 7f9bfad1b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bf41016b0 0x7f9bf41979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:50.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.471+0000 7f9bfad1b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bf41028a0 0x7f9bf4197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:50.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.471+0000 7f9bfad1b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bf4103260 0x7f9bf419ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:50.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.471+0000 7f9bfad1b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf410d330 con 0x7f9bf41016b0 2024-03-28T00:56:50.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.471+0000 7f9bfad1b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf410d1b0 con 0x7f9bf4103260 2024-03-28T00:56:50.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.471+0000 7f9bfad1b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf410d4b0 con 0x7f9bf41028a0 2024-03-28T00:56:50.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.471+0000 7f9bf8ab7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bf41016b0 0x7f9bf41979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:50.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.471+0000 7f9bf8ab7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bf41016b0 0x7f9bf41979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34872/0 (socket says 172.21.15.169:34872) 2024-03-28T00:56:50.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.471+0000 7f9bf8ab7640 1 -- 172.21.15.169:0/1514535393 learned_addr learned my addr 172.21.15.169:0/1514535393 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:50.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.472+0000 7f9bf3fff640 1 --2- 172.21.15.169:0/1514535393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bf41028a0 0x7f9bf4197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:50.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.472+0000 7f9bf92b8640 1 --2- 172.21.15.169:0/1514535393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bf4103260 0x7f9bf419ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:50.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.472+0000 7f9bf8ab7640 1 -- 172.21.15.169:0/1514535393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bf41028a0 msgr2=0x7f9bf4197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:50.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.472+0000 7f9bf8ab7640 1 --2- 172.21.15.169:0/1514535393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bf41028a0 0x7f9bf4197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.472+0000 7f9bf8ab7640 1 -- 172.21.15.169:0/1514535393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bf4103260 msgr2=0x7f9bf419ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:50.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.472+0000 7f9bf8ab7640 1 --2- 172.21.15.169:0/1514535393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bf4103260 0x7f9bf419ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.472+0000 7f9bf8ab7640 1 -- 172.21.15.169:0/1514535393 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf419f5c0 con 0x7f9bf41016b0 2024-03-28T00:56:50.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.472+0000 7f9bf8ab7640 1 --2- 172.21.15.169:0/1514535393 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bf41016b0 0x7f9bf41979a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9bec00c3a0 tx=0x7f9bec00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:50.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.472+0000 7f9bf1ffb640 1 -- 172.21.15.169:0/1514535393 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec003970 con 0x7f9bf41016b0 2024-03-28T00:56:50.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.472+0000 7f9bfad1b640 1 -- 172.21.15.169:0/1514535393 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf419f8b0 con 0x7f9bf41016b0 2024-03-28T00:56:50.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.472+0000 7f9bf1ffb640 1 -- 172.21.15.169:0/1514535393 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9bec004020 con 0x7f9bf41016b0 2024-03-28T00:56:50.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.472+0000 7f9bf1ffb640 1 -- 172.21.15.169:0/1514535393 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec00b9f0 con 0x7f9bf41016b0 2024-03-28T00:56:50.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.472+0000 7f9bfad1b640 1 -- 172.21.15.169:0/1514535393 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf419fe50 con 0x7f9bf41016b0 2024-03-28T00:56:50.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.473+0000 7f9bfad1b640 1 -- 172.21.15.169:0/1514535393 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf41089d0 con 0x7f9bf41016b0 2024-03-28T00:56:50.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.474+0000 7f9bf1ffb640 1 -- 172.21.15.169:0/1514535393 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f9bec01d020 con 0x7f9bf41016b0 2024-03-28T00:56:50.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.475+0000 7f9bf1ffb640 1 --2- 172.21.15.169:0/1514535393 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9be00a2890 0x7f9be00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:50.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.475+0000 7f9bf3fff640 1 --2- 172.21.15.169:0/1514535393 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9be00a2890 0x7f9be00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:50.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.475+0000 7f9bf1ffb640 1 -- 172.21.15.169:0/1514535393 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f9bec0cf030 con 0x7f9bf41016b0 2024-03-28T00:56:50.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.475+0000 7f9bf3fff640 1 --2- 172.21.15.169:0/1514535393 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9be00a2890 0x7f9be00a4d50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9bf4198eb0 tx=0x7f9bdc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:50.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.479+0000 7f9bf1ffb640 1 -- 172.21.15.169:0/1514535393 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9bec09bbd0 con 0x7f9bf41016b0 2024-03-28T00:56:50.656 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.653+0000 7f9bfad1b640 1 -- 172.21.15.169:0/1514535393 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9bf402d070 con 0x7f9bf41016b0 2024-03-28T00:56:50.656 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.654+0000 7f9bf1ffb640 1 -- 172.21.15.169:0/1514535393 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9bec09b4d0 con 0x7f9bf41016b0 2024-03-28T00:56:50.657 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 1 2024-03-28T00:56:50.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.656+0000 7f9bfad1b640 1 -- 172.21.15.169:0/1514535393 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9be00a2890 msgr2=0x7f9be00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:50.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.656+0000 7f9bfad1b640 1 --2- 172.21.15.169:0/1514535393 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9be00a2890 0x7f9be00a4d50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9bf4198eb0 tx=0x7f9bdc005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:50.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.656+0000 7f9bfad1b640 1 -- 172.21.15.169:0/1514535393 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bf41016b0 msgr2=0x7f9bf41979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:50.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.656+0000 7f9bfad1b640 1 --2- 172.21.15.169:0/1514535393 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bf41016b0 0x7f9bf41979a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9bec00c3a0 tx=0x7f9bec00c860 comp rx=0 tx=0).stop 2024-03-28T00:56:50.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.656+0000 7f9bfad1b640 1 -- 172.21.15.169:0/1514535393 shutdown_connections 2024-03-28T00:56:50.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.656+0000 7f9bfad1b640 1 --2- 172.21.15.169:0/1514535393 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9be00a2890 0x7f9be00a4d50 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.656+0000 7f9bfad1b640 1 --2- 172.21.15.169:0/1514535393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bf4103260 0x7f9bf419ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.656+0000 7f9bfad1b640 1 --2- 172.21.15.169:0/1514535393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bf41028a0 0x7f9bf4197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.656+0000 7f9bfad1b640 1 --2- 172.21.15.169:0/1514535393 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9bf41016b0 0x7f9bf41979a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.656+0000 7f9bfad1b640 1 -- 172.21.15.169:0/1514535393 >> 172.21.15.169:0/1514535393 conn(0x7f9bf40fcc40 msgr2=0x7f9bf4105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:50.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.656+0000 7f9bfad1b640 1 -- 172.21.15.169:0/1514535393 shutdown_connections 2024-03-28T00:56:50.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.656+0000 7f9bfad1b640 1 -- 172.21.15.169:0/1514535393 wait complete. 2024-03-28T00:56:50.670 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T00:56:50.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.773+0000 7f94db6e8640 1 -- 172.21.15.169:0/585497723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94d41016b0 msgr2=0x7f94d4101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:50.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.773+0000 7f94db6e8640 1 --2- 172.21.15.169:0/585497723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94d41016b0 0x7f94d4101ad0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f94d0009900 tx=0x7f94d002bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:50.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.773+0000 7f94db6e8640 1 -- 172.21.15.169:0/585497723 shutdown_connections 2024-03-28T00:56:50.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.773+0000 7f94db6e8640 1 --2- 172.21.15.169:0/585497723 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94d4103260 0x7f94d4109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.773+0000 7f94db6e8640 1 --2- 172.21.15.169:0/585497723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94d41028a0 0x7f94d4102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.773+0000 7f94db6e8640 1 --2- 172.21.15.169:0/585497723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94d41016b0 0x7f94d4101ad0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.773+0000 7f94db6e8640 1 -- 172.21.15.169:0/585497723 >> 172.21.15.169:0/585497723 conn(0x7f94d40fcc40 msgr2=0x7f94d40ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:50.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.773+0000 7f94db6e8640 1 -- 172.21.15.169:0/585497723 shutdown_connections 2024-03-28T00:56:50.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.773+0000 7f94db6e8640 1 -- 172.21.15.169:0/585497723 wait complete. 2024-03-28T00:56:50.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.774+0000 7f94db6e8640 1 Processor -- start 2024-03-28T00:56:50.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.774+0000 7f94db6e8640 1 -- start start 2024-03-28T00:56:50.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.774+0000 7f94db6e8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94d41016b0 0x7f94d4197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:50.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.774+0000 7f94db6e8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94d41028a0 0x7f94d4197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:50.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.774+0000 7f94db6e8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94d4103260 0x7f94d419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:50.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.774+0000 7f94db6e8640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d410d350 con 0x7f94d4103260 2024-03-28T00:56:50.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.774+0000 7f94db6e8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d410d1d0 con 0x7f94d41028a0 2024-03-28T00:56:50.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.774+0000 7f94db6e8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d410d4d0 con 0x7f94d41016b0 2024-03-28T00:56:50.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.774+0000 7f94d9484640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94d41016b0 0x7f94d4197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:50.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.774+0000 7f94d9c85640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94d4103260 0x7f94d419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:50.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.774+0000 7f94d9484640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94d41016b0 0x7f94d4197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:41448/0 (socket says 172.21.15.169:41448) 2024-03-28T00:56:50.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.774+0000 7f94d8c83640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94d41028a0 0x7f94d4197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:50.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.774+0000 7f94d9c85640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94d4103260 0x7f94d419efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34886/0 (socket says 172.21.15.169:34886) 2024-03-28T00:56:50.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.774+0000 7f94d9484640 1 -- 172.21.15.169:0/1833621852 learned_addr learned my addr 172.21.15.169:0/1833621852 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:50.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.775+0000 7f94d9484640 1 -- 172.21.15.169:0/1833621852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94d41028a0 msgr2=0x7f94d4197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:50.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.775+0000 7f94d9484640 1 --2- 172.21.15.169:0/1833621852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94d41028a0 0x7f94d4197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.775+0000 7f94d9484640 1 -- 172.21.15.169:0/1833621852 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94d4103260 msgr2=0x7f94d419efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:50.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.775+0000 7f94d9484640 1 --2- 172.21.15.169:0/1833621852 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94d4103260 0x7f94d419efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.775+0000 7f94d9484640 1 -- 172.21.15.169:0/1833621852 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d419f640 con 0x7f94d41016b0 2024-03-28T00:56:50.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.775+0000 7f94d9c85640 1 --2- 172.21.15.169:0/1833621852 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94d4103260 0x7f94d419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:50.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.775+0000 7f94d8c83640 1 --2- 172.21.15.169:0/1833621852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94d41028a0 0x7f94d4197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:50.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.775+0000 7f94d9484640 1 --2- 172.21.15.169:0/1833621852 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94d41016b0 0x7f94d4197a20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f94d0000c00 tx=0x7f94d002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:50.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.775+0000 7f94ca7fc640 1 -- 172.21.15.169:0/1833621852 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d0043020 con 0x7f94d41016b0 2024-03-28T00:56:50.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.775+0000 7f94ca7fc640 1 -- 172.21.15.169:0/1833621852 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94d00054e0 con 0x7f94d41016b0 2024-03-28T00:56:50.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.775+0000 7f94db6e8640 1 -- 172.21.15.169:0/1833621852 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d419f8d0 con 0x7f94d41016b0 2024-03-28T00:56:50.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.775+0000 7f94ca7fc640 1 -- 172.21.15.169:0/1833621852 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d0003c40 con 0x7f94d41016b0 2024-03-28T00:56:50.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.775+0000 7f94db6e8640 1 -- 172.21.15.169:0/1833621852 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d419fe20 con 0x7f94d41016b0 2024-03-28T00:56:50.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.776+0000 7f94db6e8640 1 -- 172.21.15.169:0/1833621852 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d4108a20 con 0x7f94d41016b0 2024-03-28T00:56:50.782 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.777+0000 7f94ca7fc640 1 -- 172.21.15.169:0/1833621852 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f94d0046020 con 0x7f94d41016b0 2024-03-28T00:56:50.782 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.777+0000 7f94ca7fc640 1 --2- 172.21.15.169:0/1833621852 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f94c00a2890 0x7f94c00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:50.783 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.778+0000 7f94d8c83640 1 --2- 172.21.15.169:0/1833621852 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f94c00a2890 0x7f94c00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:50.783 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.778+0000 7f94ca7fc640 1 -- 172.21.15.169:0/1833621852 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f94d00f1240 con 0x7f94d41016b0 2024-03-28T00:56:50.783 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.778+0000 7f94d8c83640 1 --2- 172.21.15.169:0/1833621852 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f94c00a2890 0x7f94c00a4d50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f94d4198f30 tx=0x7f94c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:50.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.781+0000 7f94ca7fc640 1 -- 172.21.15.169:0/1833621852 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f94d004d020 con 0x7f94d41016b0 2024-03-28T00:56:50.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.954+0000 7f94db6e8640 1 -- 172.21.15.169:0/1833621852 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f94d402d290 con 0x7f94d41016b0 2024-03-28T00:56:50.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.955+0000 7f94ca7fc640 1 -- 172.21.15.169:0/1833621852 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) v1 ==== 104+0+11 (secure 0 0 0) 0x7f94d00bdef0 con 0x7f94d41016b0 2024-03-28T00:56:50.958 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T00:56:50.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.957+0000 7f94db6e8640 1 -- 172.21.15.169:0/1833621852 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f94c00a2890 msgr2=0x7f94c00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:50.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.957+0000 7f94db6e8640 1 --2- 172.21.15.169:0/1833621852 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f94c00a2890 0x7f94c00a4d50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f94d4198f30 tx=0x7f94c4005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:50.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.957+0000 7f94db6e8640 1 -- 172.21.15.169:0/1833621852 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94d41016b0 msgr2=0x7f94d4197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:50.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.957+0000 7f94db6e8640 1 --2- 172.21.15.169:0/1833621852 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94d41016b0 0x7f94d4197a20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f94d0000c00 tx=0x7f94d002c650 comp rx=0 tx=0).stop 2024-03-28T00:56:50.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.957+0000 7f94db6e8640 1 -- 172.21.15.169:0/1833621852 shutdown_connections 2024-03-28T00:56:50.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.957+0000 7f94db6e8640 1 --2- 172.21.15.169:0/1833621852 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f94c00a2890 0x7f94c00a4d50 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.957+0000 7f94db6e8640 1 --2- 172.21.15.169:0/1833621852 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94d4103260 0x7f94d419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.957+0000 7f94db6e8640 1 --2- 172.21.15.169:0/1833621852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94d41028a0 0x7f94d4197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.957+0000 7f94db6e8640 1 --2- 172.21.15.169:0/1833621852 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94d41016b0 0x7f94d4197a20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:50.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.957+0000 7f94db6e8640 1 -- 172.21.15.169:0/1833621852 >> 172.21.15.169:0/1833621852 conn(0x7f94d40fcc40 msgr2=0x7f94d4105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:50.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.957+0000 7f94db6e8640 1 -- 172.21.15.169:0/1833621852 shutdown_connections 2024-03-28T00:56:50.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:50.957+0000 7f94db6e8640 1 -- 172.21.15.169:0/1833621852 wait complete. 2024-03-28T00:56:50.971 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T00:56:51.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.074+0000 7ffae404d640 1 -- 172.21.15.169:0/3036297059 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffadc101790 msgr2=0x7ffadc101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:51.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.074+0000 7ffae404d640 1 --2- 172.21.15.169:0/3036297059 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffadc101790 0x7ffadc101c10 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ffacc009860 tx=0x7ffacc02bb70 comp rx=0 tx=0).stop 2024-03-28T00:56:51.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.074+0000 7ffae404d640 1 -- 172.21.15.169:0/3036297059 shutdown_connections 2024-03-28T00:56:51.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.074+0000 7ffae404d640 1 --2- 172.21.15.169:0/3036297059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffadc102150 0x7ffadc1089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.074+0000 7ffae404d640 1 --2- 172.21.15.169:0/3036297059 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffadc101790 0x7ffadc101c10 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.074+0000 7ffae404d640 1 --2- 172.21.15.169:0/3036297059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffadc1005a0 0x7ffadc1009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.074+0000 7ffae404d640 1 -- 172.21.15.169:0/3036297059 >> 172.21.15.169:0/3036297059 conn(0x7ffadc0fbb30 msgr2=0x7ffadc0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:51.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.074+0000 7ffae404d640 1 -- 172.21.15.169:0/3036297059 shutdown_connections 2024-03-28T00:56:51.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.074+0000 7ffae404d640 1 -- 172.21.15.169:0/3036297059 wait complete. 2024-03-28T00:56:51.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.075+0000 7ffae404d640 1 Processor -- start 2024-03-28T00:56:51.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.075+0000 7ffae404d640 1 -- start start 2024-03-28T00:56:51.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.075+0000 7ffae404d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffadc1005a0 0x7ffadc069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:51.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.075+0000 7ffae404d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffadc101790 0x7ffadc19c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:51.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.075+0000 7ffae404d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffadc102150 0x7ffadc19ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:51.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.075+0000 7ffae404d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffadc10c230 con 0x7ffadc102150 2024-03-28T00:56:51.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.075+0000 7ffae404d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffadc10c0b0 con 0x7ffadc1005a0 2024-03-28T00:56:51.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.075+0000 7ffae404d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffadc10c3b0 con 0x7ffadc101790 2024-03-28T00:56:51.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.075+0000 7ffae25ea640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffadc102150 0x7ffadc19ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:51.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.075+0000 7ffae25ea640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffadc102150 0x7ffadc19ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34912/0 (socket says 172.21.15.169:34912) 2024-03-28T00:56:51.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.075+0000 7ffae25ea640 1 -- 172.21.15.169:0/502228291 learned_addr learned my addr 172.21.15.169:0/502228291 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:51.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.075+0000 7ffae15e8640 1 --2- 172.21.15.169:0/502228291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffadc101790 0x7ffadc19c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:51.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.075+0000 7ffae1de9640 1 --2- 172.21.15.169:0/502228291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffadc1005a0 0x7ffadc069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:51.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.075+0000 7ffae25ea640 1 -- 172.21.15.169:0/502228291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffadc101790 msgr2=0x7ffadc19c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:51.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.075+0000 7ffae25ea640 1 --2- 172.21.15.169:0/502228291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffadc101790 0x7ffadc19c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.075+0000 7ffae25ea640 1 -- 172.21.15.169:0/502228291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffadc1005a0 msgr2=0x7ffadc069390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:51.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.075+0000 7ffae25ea640 1 --2- 172.21.15.169:0/502228291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffadc1005a0 0x7ffadc069390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.075+0000 7ffae25ea640 1 -- 172.21.15.169:0/502228291 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffadc19f620 con 0x7ffadc102150 2024-03-28T00:56:51.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.076+0000 7ffae25ea640 1 --2- 172.21.15.169:0/502228291 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffadc102150 0x7ffadc19ed60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ffad400efd0 tx=0x7ffad400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:51.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.076+0000 7ffad2ffd640 1 -- 172.21.15.169:0/502228291 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad4003b70 con 0x7ffadc102150 2024-03-28T00:56:51.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.076+0000 7ffad2ffd640 1 -- 172.21.15.169:0/502228291 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffad4003d10 con 0x7ffadc102150 2024-03-28T00:56:51.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.076+0000 7ffae404d640 1 -- 172.21.15.169:0/502228291 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffadc19f910 con 0x7ffadc102150 2024-03-28T00:56:51.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.076+0000 7ffad2ffd640 1 -- 172.21.15.169:0/502228291 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad40109f0 con 0x7ffadc102150 2024-03-28T00:56:51.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.076+0000 7ffae404d640 1 -- 172.21.15.169:0/502228291 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffadc19fde0 con 0x7ffadc102150 2024-03-28T00:56:51.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.077+0000 7ffae404d640 1 -- 172.21.15.169:0/502228291 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffadc107960 con 0x7ffadc102150 2024-03-28T00:56:51.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.078+0000 7ffad2ffd640 1 -- 172.21.15.169:0/502228291 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7ffad4004020 con 0x7ffadc102150 2024-03-28T00:56:51.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.078+0000 7ffad2ffd640 1 --2- 172.21.15.169:0/502228291 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffac80a2890 0x7ffac80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:51.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.078+0000 7ffae1de9640 1 --2- 172.21.15.169:0/502228291 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffac80a2890 0x7ffac80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:51.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.078+0000 7ffad2ffd640 1 -- 172.21.15.169:0/502228291 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ffad40cf360 con 0x7ffadc102150 2024-03-28T00:56:51.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.079+0000 7ffae1de9640 1 --2- 172.21.15.169:0/502228291 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffac80a2890 0x7ffac80a4d50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ffad8009530 tx=0x7ffad8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:51.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.083+0000 7ffad2ffd640 1 -- 172.21.15.169:0/502228291 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffad409bfa0 con 0x7ffadc102150 2024-03-28T00:56:51.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.261+0000 7ffae404d640 1 -- 172.21.15.169:0/502228291 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ffadc02d280 con 0x7ffadc102150 2024-03-28T00:56:51.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.261+0000 7ffad2ffd640 1 -- 172.21.15.169:0/502228291 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) v1 ==== 100+0+11 (secure 0 0 0) 0x7ffad409b8a0 con 0x7ffadc102150 2024-03-28T00:56:51.264 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 64 2024-03-28T00:56:51.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.263+0000 7ffae404d640 1 -- 172.21.15.169:0/502228291 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffac80a2890 msgr2=0x7ffac80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:51.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.263+0000 7ffae404d640 1 --2- 172.21.15.169:0/502228291 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffac80a2890 0x7ffac80a4d50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ffad8009530 tx=0x7ffad8009260 comp rx=0 tx=0).stop 2024-03-28T00:56:51.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.263+0000 7ffae404d640 1 -- 172.21.15.169:0/502228291 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffadc102150 msgr2=0x7ffadc19ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:51.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.263+0000 7ffae404d640 1 --2- 172.21.15.169:0/502228291 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffadc102150 0x7ffadc19ed60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ffad400efd0 tx=0x7ffad400c6a0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.263+0000 7ffae404d640 1 -- 172.21.15.169:0/502228291 shutdown_connections 2024-03-28T00:56:51.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.263+0000 7ffae404d640 1 --2- 172.21.15.169:0/502228291 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffac80a2890 0x7ffac80a4d50 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.263+0000 7ffae404d640 1 --2- 172.21.15.169:0/502228291 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffadc102150 0x7ffadc19ed60 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.263+0000 7ffae404d640 1 --2- 172.21.15.169:0/502228291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffadc101790 0x7ffadc19c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.263+0000 7ffae404d640 1 --2- 172.21.15.169:0/502228291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffadc1005a0 0x7ffadc069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.263+0000 7ffae404d640 1 -- 172.21.15.169:0/502228291 >> 172.21.15.169:0/502228291 conn(0x7ffadc0fbb30 msgr2=0x7ffadc104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:51.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.264+0000 7ffae404d640 1 -- 172.21.15.169:0/502228291 shutdown_connections 2024-03-28T00:56:51.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.264+0000 7ffae404d640 1 -- 172.21.15.169:0/502228291 wait complete. 2024-03-28T00:56:51.276 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:51 smithi169 ceph-mon[32274]: from='client.15531 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:56:51.276 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:51 smithi169 ceph-mon[32274]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:56:51.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:51 smithi169 ceph-mon[32274]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:56:51.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:51 smithi169 ceph-mon[32274]: pgmap v361: 129 pgs: 129 active+clean; 653 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.0 KiB/s wr, 5 op/s 2024-03-28T00:56:51.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:51 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1832829248' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:51.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:51 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1514535393' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:56:51.277 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:51 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1833621852' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:56:51.278 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T00:56:51.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.381+0000 7f762e019640 1 -- 172.21.15.169:0/1835026066 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76281005a0 msgr2=0x7f76281009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:51.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.381+0000 7f762e019640 1 --2- 172.21.15.169:0/1835026066 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76281005a0 0x7f76281009c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7618009900 tx=0x7f761802bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:51.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.381+0000 7f762e019640 1 -- 172.21.15.169:0/1835026066 shutdown_connections 2024-03-28T00:56:51.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.381+0000 7f762e019640 1 --2- 172.21.15.169:0/1835026066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7628102150 0x7f76281089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.381+0000 7f762e019640 1 --2- 172.21.15.169:0/1835026066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7628101790 0x7f7628101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.381+0000 7f762e019640 1 --2- 172.21.15.169:0/1835026066 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76281005a0 0x7f76281009c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.381+0000 7f762e019640 1 -- 172.21.15.169:0/1835026066 >> 172.21.15.169:0/1835026066 conn(0x7f76280fbb30 msgr2=0x7f76280fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:51.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.381+0000 7f762e019640 1 -- 172.21.15.169:0/1835026066 shutdown_connections 2024-03-28T00:56:51.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.381+0000 7f762e019640 1 -- 172.21.15.169:0/1835026066 wait complete. 2024-03-28T00:56:51.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.381+0000 7f762e019640 1 Processor -- start 2024-03-28T00:56:51.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.382+0000 7f762e019640 1 -- start start 2024-03-28T00:56:51.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.382+0000 7f762e019640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76281005a0 0x7f762806a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:51.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.382+0000 7f762e019640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7628101790 0x7f762819c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:51.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.382+0000 7f762e019640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7628102150 0x7f762819ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:51.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.382+0000 7f762e019640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762810c230 con 0x7f7628102150 2024-03-28T00:56:51.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.382+0000 7f762e019640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762810c0b0 con 0x7f7628101790 2024-03-28T00:56:51.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.382+0000 7f762e019640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762810c3b0 con 0x7f76281005a0 2024-03-28T00:56:51.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.382+0000 7f7627fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7628102150 0x7f762819ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:51.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.382+0000 7f76277fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76281005a0 0x7f762806a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:51.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.382+0000 7f7627fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7628102150 0x7f762819ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34932/0 (socket says 172.21.15.169:34932) 2024-03-28T00:56:51.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.382+0000 7f7627fff640 1 -- 172.21.15.169:0/1126474931 learned_addr learned my addr 172.21.15.169:0/1126474931 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:51.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.382+0000 7f7626ffd640 1 --2- 172.21.15.169:0/1126474931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7628101790 0x7f762819c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:51.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.382+0000 7f76277fe640 1 -- 172.21.15.169:0/1126474931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7628101790 msgr2=0x7f762819c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:51.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.382+0000 7f76277fe640 1 --2- 172.21.15.169:0/1126474931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7628101790 0x7f762819c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.382+0000 7f76277fe640 1 -- 172.21.15.169:0/1126474931 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7628102150 msgr2=0x7f762819ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:51.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.382+0000 7f76277fe640 1 --2- 172.21.15.169:0/1126474931 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7628102150 0x7f762819ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.382+0000 7f76277fe640 1 -- 172.21.15.169:0/1126474931 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762819f5e0 con 0x7f76281005a0 2024-03-28T00:56:51.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.382+0000 7f7627fff640 1 --2- 172.21.15.169:0/1126474931 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7628102150 0x7f762819ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:51.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.382+0000 7f76277fe640 1 --2- 172.21.15.169:0/1126474931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76281005a0 0x7f762806a880 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f761802c170 tx=0x7f761802c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:51.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.383+0000 7f7624ff9640 1 -- 172.21.15.169:0/1126474931 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7618041070 con 0x7f76281005a0 2024-03-28T00:56:51.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.383+0000 7f7624ff9640 1 -- 172.21.15.169:0/1126474931 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7618004450 con 0x7f76281005a0 2024-03-28T00:56:51.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.383+0000 7f762e019640 1 -- 172.21.15.169:0/1126474931 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762819f870 con 0x7f76281005a0 2024-03-28T00:56:51.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.383+0000 7f7624ff9640 1 -- 172.21.15.169:0/1126474931 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7618003e50 con 0x7f76281005a0 2024-03-28T00:56:51.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.383+0000 7f762e019640 1 -- 172.21.15.169:0/1126474931 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762819fd80 con 0x7f76281005a0 2024-03-28T00:56:51.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.384+0000 7f762e019640 1 -- 172.21.15.169:0/1126474931 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76281078c0 con 0x7f76281005a0 2024-03-28T00:56:51.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.385+0000 7f7624ff9640 1 -- 172.21.15.169:0/1126474931 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f7618044070 con 0x7f76281005a0 2024-03-28T00:56:51.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.385+0000 7f7624ff9640 1 --2- 172.21.15.169:0/1126474931 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f76100a2ab0 0x7f76100a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:51.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.385+0000 7f7626ffd640 1 --2- 172.21.15.169:0/1126474931 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f76100a2ab0 0x7f76100a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:51.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.385+0000 7f7624ff9640 1 -- 172.21.15.169:0/1126474931 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f76180f0860 con 0x7f76281005a0 2024-03-28T00:56:51.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.386+0000 7f7626ffd640 1 --2- 172.21.15.169:0/1126474931 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f76100a2ab0 0x7f76100a4f70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f761c0095f0 tx=0x7f761c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:51.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.390+0000 7f7624ff9640 1 -- 172.21.15.169:0/1126474931 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f76180bcec0 con 0x7f76281005a0 2024-03-28T00:56:51.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.563+0000 7f762e019640 1 -- 172.21.15.169:0/1126474931 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f762802d2c0 con 0x7f76281005a0 2024-03-28T00:56:51.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.564+0000 7f7624ff9640 1 -- 172.21.15.169:0/1126474931 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) v1 ==== 103+0+11 (secure 0 0 0) 0x7f76180c0f60 con 0x7f76281005a0 2024-03-28T00:56:51.567 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T00:56:51.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.566+0000 7f762e019640 1 -- 172.21.15.169:0/1126474931 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f76100a2ab0 msgr2=0x7f76100a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:51.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.566+0000 7f762e019640 1 --2- 172.21.15.169:0/1126474931 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f76100a2ab0 0x7f76100a4f70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f761c0095f0 tx=0x7f761c005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:51.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.566+0000 7f762e019640 1 -- 172.21.15.169:0/1126474931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76281005a0 msgr2=0x7f762806a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:51.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.566+0000 7f762e019640 1 --2- 172.21.15.169:0/1126474931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76281005a0 0x7f762806a880 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f761802c170 tx=0x7f761802c650 comp rx=0 tx=0).stop 2024-03-28T00:56:51.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.566+0000 7f762e019640 1 -- 172.21.15.169:0/1126474931 shutdown_connections 2024-03-28T00:56:51.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.566+0000 7f762e019640 1 --2- 172.21.15.169:0/1126474931 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f76100a2ab0 0x7f76100a4f70 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.566+0000 7f762e019640 1 --2- 172.21.15.169:0/1126474931 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7628102150 0x7f762819ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.566+0000 7f762e019640 1 --2- 172.21.15.169:0/1126474931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7628101790 0x7f762819c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.566+0000 7f762e019640 1 --2- 172.21.15.169:0/1126474931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76281005a0 0x7f762806a880 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.566+0000 7f762e019640 1 -- 172.21.15.169:0/1126474931 >> 172.21.15.169:0/1126474931 conn(0x7f76280fbb30 msgr2=0x7f7628104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:51.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.566+0000 7f762e019640 1 -- 172.21.15.169:0/1126474931 shutdown_connections 2024-03-28T00:56:51.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.566+0000 7f762e019640 1 -- 172.21.15.169:0/1126474931 wait complete. 2024-03-28T00:56:51.580 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:56:51.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:51 smithi174 ceph-mon[36060]: from='client.15531 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:56:51.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:51 smithi174 ceph-mon[36060]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:56:51.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:51 smithi174 ceph-mon[36060]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:56:51.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:51 smithi174 ceph-mon[36060]: pgmap v361: 129 pgs: 129 active+clean; 653 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.0 KiB/s wr, 5 op/s 2024-03-28T00:56:51.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:51 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1832829248' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:51.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:51 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1514535393' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:56:51.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:51 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1833621852' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:56:51.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:51 smithi177 ceph-mon[36206]: from='client.15531 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:56:51.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:51 smithi177 ceph-mon[36206]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:56:51.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:51 smithi177 ceph-mon[36206]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T00:56:51.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:51 smithi177 ceph-mon[36206]: pgmap v361: 129 pgs: 129 active+clean; 653 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.0 KiB/s wr, 5 op/s 2024-03-28T00:56:51.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:51 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1832829248' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:51.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:51 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1514535393' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T00:56:51.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:51 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1833621852' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T00:56:51.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.680+0000 7faa2d91b640 1 -- 172.21.15.169:0/2932420453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa281016e0 msgr2=0x7faa28101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:51.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.680+0000 7faa2d91b640 1 --2- 172.21.15.169:0/2932420453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa281016e0 0x7faa28101b00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009900 tx=0x7faa1c02bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:51.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.680+0000 7faa2d91b640 1 -- 172.21.15.169:0/2932420453 shutdown_connections 2024-03-28T00:56:51.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.680+0000 7faa2d91b640 1 --2- 172.21.15.169:0/2932420453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa28103290 0x7faa28109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.680+0000 7faa2d91b640 1 --2- 172.21.15.169:0/2932420453 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa281028d0 0x7faa28102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.680+0000 7faa2d91b640 1 --2- 172.21.15.169:0/2932420453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa281016e0 0x7faa28101b00 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.680+0000 7faa2d91b640 1 -- 172.21.15.169:0/2932420453 >> 172.21.15.169:0/2932420453 conn(0x7faa280fcc50 msgr2=0x7faa280ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:51.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.680+0000 7faa2d91b640 1 -- 172.21.15.169:0/2932420453 shutdown_connections 2024-03-28T00:56:51.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.680+0000 7faa2d91b640 1 -- 172.21.15.169:0/2932420453 wait complete. 2024-03-28T00:56:51.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.680+0000 7faa2d91b640 1 Processor -- start 2024-03-28T00:56:51.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.680+0000 7faa2d91b640 1 -- start start 2024-03-28T00:56:51.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa2d91b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa281016e0 0x7faa28197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:51.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa2d91b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa281028d0 0x7faa28197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:51.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa2d91b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa28103290 0x7faa2819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:51.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa2d91b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2810d330 con 0x7faa281016e0 2024-03-28T00:56:51.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa2d91b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2810d1b0 con 0x7faa28103290 2024-03-28T00:56:51.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa2d91b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2810d4b0 con 0x7faa281028d0 2024-03-28T00:56:51.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa26ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa281016e0 0x7faa28197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:51.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa26ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa281016e0 0x7faa28197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34956/0 (socket says 172.21.15.169:34956) 2024-03-28T00:56:51.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa26ffd640 1 -- 172.21.15.169:0/4055977596 learned_addr learned my addr 172.21.15.169:0/4055977596 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:51.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa267fc640 1 --2- 172.21.15.169:0/4055977596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa281028d0 0x7faa28197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:51.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa277fe640 1 --2- 172.21.15.169:0/4055977596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa28103290 0x7faa2819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:51.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa26ffd640 1 -- 172.21.15.169:0/4055977596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa281028d0 msgr2=0x7faa28197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:51.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa26ffd640 1 --2- 172.21.15.169:0/4055977596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa281028d0 0x7faa28197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa26ffd640 1 -- 172.21.15.169:0/4055977596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa28103290 msgr2=0x7faa2819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:51.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa26ffd640 1 --2- 172.21.15.169:0/4055977596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa28103290 0x7faa2819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa26ffd640 1 -- 172.21.15.169:0/4055977596 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa2819f660 con 0x7faa281016e0 2024-03-28T00:56:51.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa267fc640 1 --2- 172.21.15.169:0/4055977596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa281028d0 0x7faa28197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:51.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa26ffd640 1 --2- 172.21.15.169:0/4055977596 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa281016e0 0x7faa28197a40 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7faa1c000c00 tx=0x7faa1c02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:51.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa2c919640 1 -- 172.21.15.169:0/4055977596 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1c041070 con 0x7faa281016e0 2024-03-28T00:56:51.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa2d91b640 1 -- 172.21.15.169:0/4055977596 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa2819f8f0 con 0x7faa281016e0 2024-03-28T00:56:51.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa2c919640 1 -- 172.21.15.169:0/4055977596 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa1c004450 con 0x7faa281016e0 2024-03-28T00:56:51.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa2c919640 1 -- 172.21.15.169:0/4055977596 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1c003df0 con 0x7faa281016e0 2024-03-28T00:56:51.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.681+0000 7faa2d91b640 1 -- 172.21.15.169:0/4055977596 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa2819fde0 con 0x7faa281016e0 2024-03-28T00:56:51.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.682+0000 7faa2d91b640 1 -- 172.21.15.169:0/4055977596 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa28108af0 con 0x7faa281016e0 2024-03-28T00:56:51.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.683+0000 7faa2c919640 1 -- 172.21.15.169:0/4055977596 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7faa1c044070 con 0x7faa281016e0 2024-03-28T00:56:51.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.684+0000 7faa2c919640 1 --2- 172.21.15.169:0/4055977596 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faa100a2890 0x7faa100a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:51.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.684+0000 7faa267fc640 1 --2- 172.21.15.169:0/4055977596 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faa100a2890 0x7faa100a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:51.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.684+0000 7faa2c919640 1 -- 172.21.15.169:0/4055977596 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7faa1c0f0950 con 0x7faa281016e0 2024-03-28T00:56:51.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.684+0000 7faa267fc640 1 --2- 172.21.15.169:0/4055977596 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faa100a2890 0x7faa100a4d50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7faa28198f50 tx=0x7faa18005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:51.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.688+0000 7faa2c919640 1 -- 172.21.15.169:0/4055977596 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faa1c0bd690 con 0x7faa281016e0 2024-03-28T00:56:51.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.899+0000 7faa2d91b640 1 -- 172.21.15.169:0/4055977596 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa2802d070 con 0x7faa281016e0 2024-03-28T00:56:51.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.900+0000 7faa2c919640 1 -- 172.21.15.169:0/4055977596 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7faa1c0bcf90 con 0x7faa281016e0 2024-03-28T00:56:51.903 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:51.903 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T00:56:51.903 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T00:56:51.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.902+0000 7faa2d91b640 1 -- 172.21.15.169:0/4055977596 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faa100a2890 msgr2=0x7faa100a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:51.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.902+0000 7faa2d91b640 1 --2- 172.21.15.169:0/4055977596 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faa100a2890 0x7faa100a4d50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7faa28198f50 tx=0x7faa18005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:51.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.902+0000 7faa2d91b640 1 -- 172.21.15.169:0/4055977596 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa281016e0 msgr2=0x7faa28197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:51.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.902+0000 7faa2d91b640 1 --2- 172.21.15.169:0/4055977596 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa281016e0 0x7faa28197a40 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7faa1c000c00 tx=0x7faa1c02c640 comp rx=0 tx=0).stop 2024-03-28T00:56:51.906 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.902+0000 7faa2d91b640 1 -- 172.21.15.169:0/4055977596 shutdown_connections 2024-03-28T00:56:51.906 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.902+0000 7faa2d91b640 1 --2- 172.21.15.169:0/4055977596 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faa100a2890 0x7faa100a4d50 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.906 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.902+0000 7faa2d91b640 1 --2- 172.21.15.169:0/4055977596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa28103290 0x7faa2819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.906 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.902+0000 7faa2d91b640 1 --2- 172.21.15.169:0/4055977596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa281028d0 0x7faa28197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.906 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.902+0000 7faa2d91b640 1 --2- 172.21.15.169:0/4055977596 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faa281016e0 0x7faa28197a40 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:51.906 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.902+0000 7faa2d91b640 1 -- 172.21.15.169:0/4055977596 >> 172.21.15.169:0/4055977596 conn(0x7faa280fcc50 msgr2=0x7faa28105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:51.906 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.902+0000 7faa2d91b640 1 -- 172.21.15.169:0/4055977596 shutdown_connections 2024-03-28T00:56:51.906 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:51.902+0000 7faa2d91b640 1 -- 172.21.15.169:0/4055977596 wait complete. 2024-03-28T00:56:51.916 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T00:56:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.016+0000 7fb5471c4640 1 -- 172.21.15.169:0/2092637635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5401005e0 msgr2=0x7fb540100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.016+0000 7fb5471c4640 1 --2- 172.21.15.169:0/2092637635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5401005e0 0x7fb540100a00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb534009900 tx=0x7fb53402bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.016+0000 7fb5471c4640 1 -- 172.21.15.169:0/2092637635 shutdown_connections 2024-03-28T00:56:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.016+0000 7fb5471c4640 1 --2- 172.21.15.169:0/2092637635 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb540102190 0x7fb540108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.016+0000 7fb5471c4640 1 --2- 172.21.15.169:0/2092637635 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5401017d0 0x7fb540101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.016+0000 7fb5471c4640 1 --2- 172.21.15.169:0/2092637635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5401005e0 0x7fb540100a00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.016+0000 7fb5471c4640 1 -- 172.21.15.169:0/2092637635 >> 172.21.15.169:0/2092637635 conn(0x7fb5400fbb50 msgr2=0x7fb5400fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.016+0000 7fb5471c4640 1 -- 172.21.15.169:0/2092637635 shutdown_connections 2024-03-28T00:56:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.016+0000 7fb5471c4640 1 -- 172.21.15.169:0/2092637635 wait complete. 2024-03-28T00:56:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.016+0000 7fb5471c4640 1 Processor -- start 2024-03-28T00:56:52.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.016+0000 7fb5471c4640 1 -- start start 2024-03-28T00:56:52.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.016+0000 7fb5471c4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5401005e0 0x7fb540197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.016+0000 7fb5471c4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5401017d0 0x7fb540197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.016+0000 7fb5471c4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb540102190 0x7fb54019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.016+0000 7fb5471c4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54010c250 con 0x7fb5401017d0 2024-03-28T00:56:52.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.016+0000 7fb5471c4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54010c0d0 con 0x7fb5401005e0 2024-03-28T00:56:52.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.016+0000 7fb5471c4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54010c3d0 con 0x7fb540102190 2024-03-28T00:56:52.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.017+0000 7fb53ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5401017d0 0x7fb540197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.017+0000 7fb545761640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb540102190 0x7fb54019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.017+0000 7fb53ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5401017d0 0x7fb540197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34976/0 (socket says 172.21.15.169:34976) 2024-03-28T00:56:52.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.017+0000 7fb544f60640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5401005e0 0x7fb540197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.017+0000 7fb545761640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb540102190 0x7fb54019efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:41508/0 (socket says 172.21.15.169:41508) 2024-03-28T00:56:52.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.017+0000 7fb53ffff640 1 -- 172.21.15.169:0/1880281113 learned_addr learned my addr 172.21.15.169:0/1880281113 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:52.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.017+0000 7fb53ffff640 1 -- 172.21.15.169:0/1880281113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb540102190 msgr2=0x7fb54019efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.017+0000 7fb53ffff640 1 --2- 172.21.15.169:0/1880281113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb540102190 0x7fb54019efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.017+0000 7fb53ffff640 1 -- 172.21.15.169:0/1880281113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5401005e0 msgr2=0x7fb540197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.017+0000 7fb53ffff640 1 --2- 172.21.15.169:0/1880281113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5401005e0 0x7fb540197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.017+0000 7fb53ffff640 1 -- 172.21.15.169:0/1880281113 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54019f630 con 0x7fb5401017d0 2024-03-28T00:56:52.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.017+0000 7fb545761640 1 --2- 172.21.15.169:0/1880281113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb540102190 0x7fb54019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:52.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.017+0000 7fb544f60640 1 --2- 172.21.15.169:0/1880281113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5401005e0 0x7fb540197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:52.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.017+0000 7fb53ffff640 1 --2- 172.21.15.169:0/1880281113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5401017d0 0x7fb540197f50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb52c00e9f0 tx=0x7fb52c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:52.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.017+0000 7fb53dffb640 1 -- 172.21.15.169:0/1880281113 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52c00cd80 con 0x7fb5401017d0 2024-03-28T00:56:52.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.017+0000 7fb5471c4640 1 -- 172.21.15.169:0/1880281113 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb54019f920 con 0x7fb5401017d0 2024-03-28T00:56:52.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.017+0000 7fb53dffb640 1 -- 172.21.15.169:0/1880281113 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb52c010910 con 0x7fb5401017d0 2024-03-28T00:56:52.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.017+0000 7fb53dffb640 1 -- 172.21.15.169:0/1880281113 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52c0188f0 con 0x7fb5401017d0 2024-03-28T00:56:52.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.017+0000 7fb5471c4640 1 -- 172.21.15.169:0/1880281113 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb54019fec0 con 0x7fb5401017d0 2024-03-28T00:56:52.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.018+0000 7fb5471c4640 1 -- 172.21.15.169:0/1880281113 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5401a0090 con 0x7fb5401017d0 2024-03-28T00:56:52.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.019+0000 7fb53dffb640 1 -- 172.21.15.169:0/1880281113 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fb52c010ac0 con 0x7fb5401017d0 2024-03-28T00:56:52.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.020+0000 7fb53dffb640 1 --2- 172.21.15.169:0/1880281113 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb5300a2890 0x7fb5300a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.020+0000 7fb53dffb640 1 -- 172.21.15.169:0/1880281113 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb52c0cf340 con 0x7fb5401017d0 2024-03-28T00:56:52.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.020+0000 7fb544f60640 1 --2- 172.21.15.169:0/1880281113 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb5300a2890 0x7fb5300a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.021+0000 7fb544f60640 1 --2- 172.21.15.169:0/1880281113 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb5300a2890 0x7fb5300a4d50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb534000c00 tx=0x7fb534005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:52.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.025+0000 7fb53dffb640 1 -- 172.21.15.169:0/1880281113 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb52c09b890 con 0x7fb5401017d0 2024-03-28T00:56:52.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.202+0000 7fb5471c4640 1 -- 172.21.15.169:0/1880281113 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5401a0380 con 0x7fb5401017d0 2024-03-28T00:56:52.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.204+0000 7fb53dffb640 1 -- 172.21.15.169:0/1880281113 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+24823 (secure 0 0 0) 0x7fb52c084e20 con 0x7fb5401017d0 2024-03-28T00:56:52.207 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:56:52.207 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":94,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:56:38.281826+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9},{"from":5,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T00:56:52.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.206+0000 7fb5471c4640 1 -- 172.21.15.169:0/1880281113 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb5300a2890 msgr2=0x7fb5300a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.206+0000 7fb5471c4640 1 --2- 172.21.15.169:0/1880281113 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb5300a2890 0x7fb5300a4d50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb534000c00 tx=0x7fb534005fd0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.206+0000 7fb5471c4640 1 -- 172.21.15.169:0/1880281113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5401017d0 msgr2=0x7fb540197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.206+0000 7fb5471c4640 1 --2- 172.21.15.169:0/1880281113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5401017d0 0x7fb540197f50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb52c00e9f0 tx=0x7fb52c00eeb0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.206+0000 7fb5471c4640 1 -- 172.21.15.169:0/1880281113 shutdown_connections 2024-03-28T00:56:52.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.206+0000 7fb5471c4640 1 --2- 172.21.15.169:0/1880281113 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb5300a2890 0x7fb5300a4d50 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.206+0000 7fb5471c4640 1 --2- 172.21.15.169:0/1880281113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb540102190 0x7fb54019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.206+0000 7fb5471c4640 1 --2- 172.21.15.169:0/1880281113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5401017d0 0x7fb540197f50 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.206+0000 7fb5471c4640 1 --2- 172.21.15.169:0/1880281113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5401005e0 0x7fb540197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.206+0000 7fb5471c4640 1 -- 172.21.15.169:0/1880281113 >> 172.21.15.169:0/1880281113 conn(0x7fb5400fbb50 msgr2=0x7fb540069cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:52.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.206+0000 7fb5471c4640 1 -- 172.21.15.169:0/1880281113 shutdown_connections 2024-03-28T00:56:52.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.206+0000 7fb5471c4640 1 -- 172.21.15.169:0/1880281113 wait complete. 2024-03-28T00:56:52.220 DEBUG:tasks.fwd_scrub:Yielding 2024-03-28T00:56:52.221 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-28T00:56:52.231 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi169.front.sepia.ceph.com 2024-03-28T00:56:52.232 DEBUG:teuthology.orchestra.run.smithi169:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-28T00:56:52.232 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-28T00:56:52.232 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T00:56:52.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:56:52.232 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T00:56:52.235 INFO:teuthology.run_tasks:Running task workunit... 2024-03-28T00:56:52.249 INFO:tasks.workunit:Pulling workunits from ref 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T00:56:52.249 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-28T00:56:52.249 INFO:tasks.workunit:timeout=3h 2024-03-28T00:56:52.249 INFO:tasks.workunit:cleanup=True 2024-03-28T00:56:52.250 DEBUG:teuthology.orchestra.run.smithi169:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-28T00:56:52.265 INFO:teuthology.orchestra.run.smithi169.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-28T00:56:52.265 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-28T00:56:52.265 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-03-28T00:56:52.265 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T00:56:52.265 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-28T00:56:52.265 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-28 00:54:53.667849769 +0000 2024-03-28T00:56:52.265 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-28 00:54:53.667849769 +0000 2024-03-28T00:56:52.265 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-28 00:56:44.426371150 +0000 2024-03-28T00:56:52.265 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-28 00:54:53.667849769 +0000 2024-03-28T00:56:52.266 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-28T00:56:52.266 DEBUG:teuthology.orchestra.run.smithi169:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-28T00:56:52.337 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T00:56:52.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd082e30640 1 -- 172.21.15.169:0/3968641233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd07c101790 msgr2=0x7fd07c101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd082e30640 1 --2- 172.21.15.169:0/3968641233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd07c101790 0x7fd07c101c10 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd068009900 tx=0x7fd06802bc30 comp rx=0 tx=0).stop 2024-03-28T00:56:52.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd082e30640 1 -- 172.21.15.169:0/3968641233 shutdown_connections 2024-03-28T00:56:52.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd082e30640 1 --2- 172.21.15.169:0/3968641233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd07c102150 0x7fd07c1089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd082e30640 1 --2- 172.21.15.169:0/3968641233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd07c101790 0x7fd07c101c10 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd082e30640 1 --2- 172.21.15.169:0/3968641233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c1005a0 0x7fd07c1009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd082e30640 1 -- 172.21.15.169:0/3968641233 >> 172.21.15.169:0/3968641233 conn(0x7fd07c0fbb30 msgr2=0x7fd07c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:52.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd082e30640 1 -- 172.21.15.169:0/3968641233 shutdown_connections 2024-03-28T00:56:52.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd082e30640 1 -- 172.21.15.169:0/3968641233 wait complete. 2024-03-28T00:56:52.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd082e30640 1 Processor -- start 2024-03-28T00:56:52.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd082e30640 1 -- start start 2024-03-28T00:56:52.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd082e30640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c1005a0 0x7fd07c069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd082e30640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd07c101790 0x7fd07c19c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd082e30640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd07c102150 0x7fd07c19ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd082e30640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c10c270 con 0x7fd07c102150 2024-03-28T00:56:52.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd082e30640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c10c0f0 con 0x7fd07c1005a0 2024-03-28T00:56:52.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd082e30640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c10c3f0 con 0x7fd07c101790 2024-03-28T00:56:52.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd0813cd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd07c102150 0x7fd07c19ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd080bcc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c1005a0 0x7fd07c069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd0813cd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd07c102150 0x7fd07c19ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34990/0 (socket says 172.21.15.169:34990) 2024-03-28T00:56:52.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd07bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd07c101790 0x7fd07c19c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.339+0000 7fd0813cd640 1 -- 172.21.15.169:0/436753305 learned_addr learned my addr 172.21.15.169:0/436753305 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:52.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.340+0000 7fd080bcc640 1 -- 172.21.15.169:0/436753305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd07c101790 msgr2=0x7fd07c19c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.340+0000 7fd080bcc640 1 --2- 172.21.15.169:0/436753305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd07c101790 0x7fd07c19c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.340+0000 7fd080bcc640 1 -- 172.21.15.169:0/436753305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd07c102150 msgr2=0x7fd07c19ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.340+0000 7fd080bcc640 1 --2- 172.21.15.169:0/436753305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd07c102150 0x7fd07c19ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.340+0000 7fd080bcc640 1 -- 172.21.15.169:0/436753305 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c19f660 con 0x7fd07c1005a0 2024-03-28T00:56:52.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.340+0000 7fd0813cd640 1 --2- 172.21.15.169:0/436753305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd07c102150 0x7fd07c19ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:52.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.340+0000 7fd080bcc640 1 --2- 172.21.15.169:0/436753305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c1005a0 0x7fd07c069370 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd07000e9c0 tx=0x7fd07000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:52.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.340+0000 7fd079ffb640 1 -- 172.21.15.169:0/436753305 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07000ccc0 con 0x7fd07c1005a0 2024-03-28T00:56:52.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.340+0000 7fd082e30640 1 -- 172.21.15.169:0/436753305 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd07c19f8f0 con 0x7fd07c1005a0 2024-03-28T00:56:52.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.340+0000 7fd079ffb640 1 -- 172.21.15.169:0/436753305 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd0700035f0 con 0x7fd07c1005a0 2024-03-28T00:56:52.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.340+0000 7fd079ffb640 1 -- 172.21.15.169:0/436753305 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07000ce60 con 0x7fd07c1005a0 2024-03-28T00:56:52.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.340+0000 7fd082e30640 1 -- 172.21.15.169:0/436753305 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd07c19fe60 con 0x7fd07c1005a0 2024-03-28T00:56:52.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.342+0000 7fd079ffb640 1 -- 172.21.15.169:0/436753305 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fd0700037a0 con 0x7fd07c1005a0 2024-03-28T00:56:52.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.343+0000 7fd085a16740 1 Processor -- start 2024-03-28T00:56:52.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.343+0000 7fd079ffb640 1 --2- 172.21.15.169:0/436753305 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd06c0a2890 0x7fd06c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.343+0000 7fd085a16740 1 -- start start 2024-03-28T00:56:52.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.343+0000 7fd0813cd640 1 --2- 172.21.15.169:0/436753305 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd06c0a2890 0x7fd06c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.343+0000 7fd085a16740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe4960 0x561e2afe4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.343+0000 7fd085a16740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afe5350 0x561e2afe9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.343+0000 7fd079ffb640 1 -- 172.21.15.169:0/436753305 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fd0700cf410 con 0x7fd07c1005a0 2024-03-28T00:56:52.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.343+0000 7fd085a16740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afea4c0 0x561e2afec950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.343+0000 7fd085a16740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e2af24c90 con 0x561e2afe5350 2024-03-28T00:56:52.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.343+0000 7fd085a16740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e2af4a070 con 0x561e2afe4960 2024-03-28T00:56:52.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.343+0000 7fd085a16740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e2aef2c80 con 0x561e2afea4c0 2024-03-28T00:56:52.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.343+0000 7fd080bcc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe4960 0x561e2afe4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.343+0000 7fd0813cd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afea4c0 0x561e2afec950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.343+0000 7fd080bcc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe4960 0x561e2afe4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33692/0 (socket says 172.21.15.169:33692) 2024-03-28T00:56:52.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.343+0000 7fd080bcc640 1 -- 172.21.15.169:0/1850301616 learned_addr learned my addr 172.21.15.169:0/1850301616 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:52.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.343+0000 7fd07bfff640 1 --2- 172.21.15.169:0/1850301616 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afe5350 0x561e2afe9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.344+0000 7fd080bcc640 1 -- 172.21.15.169:0/1850301616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afea4c0 msgr2=0x561e2afec950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:52.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.344+0000 7fd080bcc640 1 --2- 172.21.15.169:0/1850301616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afea4c0 0x561e2afec950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.344+0000 7fd080bcc640 1 -- 172.21.15.169:0/1850301616 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afe5350 msgr2=0x561e2afe9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:52.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.344+0000 7fd080bcc640 1 --2- 172.21.15.169:0/1850301616 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afe5350 0x561e2afe9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.344+0000 7fd080bcc640 1 -- 172.21.15.169:0/1850301616 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e2aeead10 con 0x561e2afe4960 2024-03-28T00:56:52.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.344+0000 7fd0813cd640 1 --2- 172.21.15.169:0/436753305 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd06c0a2890 0x7fd06c0a4d50 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd074009fd0 tx=0x7fd074011810 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:52.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.344+0000 7fd080bcc640 1 --2- 172.21.15.169:0/1850301616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe4960 0x561e2afe4d80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd07009b7e0 tx=0x7fd07009c010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53cd666bd4ca3bc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:52.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.344+0000 7fd066ffd640 1 -- 172.21.15.169:0/1850301616 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0700a3ff0 con 0x561e2afe4960 2024-03-28T00:56:52.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.344+0000 7fd066ffd640 1 -- 172.21.15.169:0/1850301616 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd070099b50 con 0x561e2afe4960 2024-03-28T00:56:52.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.344+0000 7fd066ffd640 1 -- 172.21.15.169:0/1850301616 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0700a3dc0 con 0x561e2afe4960 2024-03-28T00:56:52.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.345+0000 7fd085a16740 1 -- 172.21.15.169:0/1850301616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe4960 msgr2=0x561e2afe4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.345+0000 7fd085a16740 1 --2- 172.21.15.169:0/1850301616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe4960 0x561e2afe4d80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd07009b7e0 tx=0x7fd07009c010 comp rx=0 tx=0).stop 2024-03-28T00:56:52.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.345+0000 7fd085a16740 1 -- 172.21.15.169:0/1850301616 shutdown_connections 2024-03-28T00:56:52.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.345+0000 7fd085a16740 1 --2- 172.21.15.169:0/1850301616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afea4c0 0x561e2afec950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.345+0000 7fd085a16740 1 --2- 172.21.15.169:0/1850301616 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afe5350 0x561e2afe9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.345+0000 7fd085a16740 1 --2- 172.21.15.169:0/1850301616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe4960 0x561e2afe4d80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.345+0000 7fd085a16740 1 -- 172.21.15.169:0/1850301616 >> 172.21.15.169:0/1850301616 conn(0x561e2afad1a0 msgr2=0x561e2afe02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:52.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.345+0000 7fd085a16740 1 -- 172.21.15.169:0/1850301616 shutdown_connections 2024-03-28T00:56:52.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.345+0000 7fd085a16740 1 -- 172.21.15.169:0/1850301616 wait complete. 2024-03-28T00:56:52.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.345+0000 7fd085a16740 1 -- start start 2024-03-28T00:56:52.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.345+0000 7fd085a16740 1 Processor -- start 2024-03-28T00:56:52.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.345+0000 7fd085a16740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe4960 0x561e2afeed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.345+0000 7fd085a16740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afe5350 0x561e2afef290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.345+0000 7fd085a16740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afea4c0 0x561e2aff62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.345+0000 7fd085a16740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e2af060d0 con 0x561e2afea4c0 2024-03-28T00:56:52.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.346+0000 7fd085a16740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e2af46110 con 0x561e2afe4960 2024-03-28T00:56:52.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.346+0000 7fd085a16740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e2afed470 con 0x561e2afe5350 2024-03-28T00:56:52.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.346+0000 7fd085a16740 10 client.0 _ll_register_callbacks cb 0x561e2af41e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:56:52.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.346+0000 7fd0813cd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afea4c0 0x561e2aff62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.346+0000 7fd07bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afe5350 0x561e2afef290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.346+0000 7fd0813cd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afea4c0 0x561e2aff62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:35008/0 (socket says 172.21.15.169:35008) 2024-03-28T00:56:52.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.346+0000 7fd0813cd640 1 -- 172.21.15.169:0/439074523 learned_addr learned my addr 172.21.15.169:0/439074523 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:52.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.346+0000 7fd080bcc640 1 --2- 172.21.15.169:0/439074523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe4960 0x561e2afeed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.346+0000 7fd0813cd640 1 -- 172.21.15.169:0/439074523 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afe5350 msgr2=0x561e2afef290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:52.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.346+0000 7fd0813cd640 1 --2- 172.21.15.169:0/439074523 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afe5350 0x561e2afef290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.346+0000 7fd0813cd640 1 -- 172.21.15.169:0/439074523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe4960 msgr2=0x561e2afeed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:52.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.346+0000 7fd0813cd640 1 --2- 172.21.15.169:0/439074523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe4960 0x561e2afeed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.346+0000 7fd0813cd640 1 -- 172.21.15.169:0/439074523 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e2ad267e0 con 0x561e2afea4c0 2024-03-28T00:56:52.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.346+0000 7fd080bcc640 1 --2- 172.21.15.169:0/439074523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe4960 0x561e2afeed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:52.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.346+0000 7fd07bfff640 1 --2- 172.21.15.169:0/439074523 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afe5350 0x561e2afef290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:52.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.346+0000 7fd0813cd640 1 --2- 172.21.15.169:0/439074523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afea4c0 0x561e2aff62d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd07400f9b0 tx=0x7fd074004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f5779383255e2f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:52.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.346+0000 7fd05bfff640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:56:52.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.346+0000 7fd05bfff640 1 -- 172.21.15.169:0/439074523 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd074024070 con 0x561e2afea4c0 2024-03-28T00:56:52.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.346+0000 7fd05bfff640 1 -- 172.21.15.169:0/439074523 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fd07401fab0 con 0x561e2afea4c0 2024-03-28T00:56:52.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.346+0000 7fd085a16740 1 -- 172.21.15.169:0/439074523 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd070099b50 con 0x561e2afea4c0 2024-03-28T00:56:52.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.347+0000 7fd05bfff640 1 -- 172.21.15.169:0/439074523 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd07401a040 con 0x561e2afea4c0 2024-03-28T00:56:52.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.347+0000 7fd085a16740 10 client.15597 fetch_fsmap learned FSMap version 25 2024-03-28T00:56:52.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.347+0000 7fd085a16740 1 -- 172.21.15.169:0/439074523 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561e2ad86f70 con 0x561e2afea4c0 2024-03-28T00:56:52.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.347+0000 7fd05bfff640 1 -- 172.21.15.169:0/439074523 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd070099b50 con 0x561e2afea4c0 2024-03-28T00:56:52.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.347+0000 7fd05bfff640 1 -- 172.21.15.169:0/439074523 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd07401d030 con 0x561e2afea4c0 2024-03-28T00:56:52.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.347+0000 7fd085a16740 10 client.15597 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:56:52.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.347+0000 7fd085a16740 10 client.15597 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:56:52.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.347+0000 7fd085a16740 20 client.15597 populate_metadata read hostname 'smithi169' 2024-03-28T00:56:52.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.347+0000 7fd085a16740 1 --2- 172.21.15.169:0/439074523 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561e2affabe0 0x561e2affd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.347+0000 7fd085a16740 4 client.15597 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:56:52.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.347+0000 7fd085a16740 1 -- 172.21.15.169:0/439074523 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561e2af575b0 con 0x561e2affabe0 2024-03-28T00:56:52.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.347+0000 7fd080bcc640 1 --2- 172.21.15.169:0/439074523 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561e2affabe0 0x561e2affd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.348+0000 7fd080bcc640 1 --2- 172.21.15.169:0/439074523 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561e2affabe0 0x561e2affd040 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=392dde4b8003a246 server_cookie=28ebb14eefdd5b1c in_seq=0 out_seq=0 2024-03-28T00:56:52.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.348+0000 7fd05bfff640 10 client.15597 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:56:52.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.348+0000 7fd05bfff640 1 -- 172.21.15.169:0/439074523 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561e2af575b0 con 0x561e2affabe0 2024-03-28T00:56:52.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.348+0000 7fd05bfff640 10 client.15597 handle_command_reply: tid=0 2024-03-28T00:56:52.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.348+0000 7fd085a16740 1 client.15597 shutdown 2024-03-28T00:56:52.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.348+0000 7fd085a16740 1 -- 172.21.15.169:0/439074523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afea4c0 msgr2=0x561e2aff62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.348+0000 7fd085a16740 1 --2- 172.21.15.169:0/439074523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afea4c0 0x561e2aff62d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd07400f9b0 tx=0x7fd074004ab0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.348+0000 7fd085a16740 1 -- 172.21.15.169:0/439074523 shutdown_connections 2024-03-28T00:56:52.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.348+0000 7fd085a16740 1 --2- 172.21.15.169:0/439074523 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561e2affabe0 0x561e2affd040 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.348+0000 7fd085a16740 1 --2- 172.21.15.169:0/439074523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afea4c0 0x561e2aff62d0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.348+0000 7fd085a16740 1 --2- 172.21.15.169:0/439074523 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afe5350 0x561e2afef290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.348+0000 7fd085a16740 1 --2- 172.21.15.169:0/439074523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe4960 0x561e2afeed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.348+0000 7fd085a16740 1 -- 172.21.15.169:0/439074523 >> 172.21.15.169:0/439074523 conn(0x561e2afad1a0 msgr2=0x561e2afdf530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:52.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.348+0000 7fd05bfff640 0 client.15597 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:56:52.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.349+0000 7fd085a16740 1 -- 172.21.15.169:0/439074523 shutdown_connections 2024-03-28T00:56:52.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.349+0000 7fd085a16740 1 -- 172.21.15.169:0/439074523 wait complete. 2024-03-28T00:56:52.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.349+0000 7fd085a16740 20 client.15597 trim_cache size 0 max 16384 2024-03-28T00:56:52.359 INFO:tasks.workunit.client.0.smithi169.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-28T00:56:52.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.366+0000 7fd085a16740 1 Processor -- start 2024-03-28T00:56:52.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.366+0000 7fd085a16740 1 -- start start 2024-03-28T00:56:52.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd085a16740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afe4960 0x561e2afe6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd085a16740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe5350 0x561e2afe6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd085a16740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afea4c0 0x561e2aff6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd085a16740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c1aa560 con 0x561e2afea4c0 2024-03-28T00:56:52.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd085a16740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c1a8060 con 0x561e2afe5350 2024-03-28T00:56:52.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd085a16740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c1a6100 con 0x561e2afe4960 2024-03-28T00:56:52.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd0813cd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afea4c0 0x561e2aff6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd0813cd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afea4c0 0x561e2aff6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:35014/0 (socket says 172.21.15.169:35014) 2024-03-28T00:56:52.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd0813cd640 1 -- 172.21.15.169:0/4000745513 learned_addr learned my addr 172.21.15.169:0/4000745513 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:52.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd080bcc640 1 --2- 172.21.15.169:0/4000745513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afe4960 0x561e2afe6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd07bfff640 1 --2- 172.21.15.169:0/4000745513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe5350 0x561e2afe6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd0813cd640 1 -- 172.21.15.169:0/4000745513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afe4960 msgr2=0x561e2afe6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:52.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd0813cd640 1 --2- 172.21.15.169:0/4000745513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afe4960 0x561e2afe6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd0813cd640 1 -- 172.21.15.169:0/4000745513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe5350 msgr2=0x561e2afe6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:52.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd0813cd640 1 --2- 172.21.15.169:0/4000745513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe5350 0x561e2afe6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd0813cd640 1 -- 172.21.15.169:0/4000745513 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e2ac1e170 con 0x561e2afea4c0 2024-03-28T00:56:52.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd080bcc640 1 --2- 172.21.15.169:0/4000745513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afe4960 0x561e2afe6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:52.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd07bfff640 1 --2- 172.21.15.169:0/4000745513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe5350 0x561e2afe6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:52.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd0813cd640 1 --2- 172.21.15.169:0/4000745513 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afea4c0 0x561e2aff6290 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fd074000f80 tx=0x7fd07400fa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48c80e43e9223ce9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:52.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd0667fc640 1 -- 172.21.15.169:0/4000745513 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07403b020 con 0x561e2afea4c0 2024-03-28T00:56:52.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd0667fc640 1 -- 172.21.15.169:0/4000745513 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd074011db0 con 0x561e2afea4c0 2024-03-28T00:56:52.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.367+0000 7fd0667fc640 1 -- 172.21.15.169:0/4000745513 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd074024070 con 0x561e2afea4c0 2024-03-28T00:56:52.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.368+0000 7fd085a16740 1 -- 172.21.15.169:0/4000745513 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afea4c0 msgr2=0x561e2aff6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.368+0000 7fd085a16740 1 --2- 172.21.15.169:0/4000745513 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afea4c0 0x561e2aff6290 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fd074000f80 tx=0x7fd07400fa60 comp rx=0 tx=0).stop 2024-03-28T00:56:52.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.368+0000 7fd085a16740 1 -- 172.21.15.169:0/4000745513 shutdown_connections 2024-03-28T00:56:52.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.368+0000 7fd085a16740 1 --2- 172.21.15.169:0/4000745513 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afea4c0 0x561e2aff6290 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.368+0000 7fd085a16740 1 --2- 172.21.15.169:0/4000745513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe5350 0x561e2afe6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.368+0000 7fd085a16740 1 --2- 172.21.15.169:0/4000745513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afe4960 0x561e2afe6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.368+0000 7fd085a16740 1 -- 172.21.15.169:0/4000745513 >> 172.21.15.169:0/4000745513 conn(0x561e2afad1a0 msgr2=0x561e2afed810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:52.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.368+0000 7fd085a16740 1 -- 172.21.15.169:0/4000745513 shutdown_connections 2024-03-28T00:56:52.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.368+0000 7fd085a16740 1 -- 172.21.15.169:0/4000745513 wait complete. 2024-03-28T00:56:52.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.368+0000 7fd085a16740 1 -- start start 2024-03-28T00:56:52.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.368+0000 7fd085a16740 1 Processor -- start 2024-03-28T00:56:52.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.369+0000 7fd085a16740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afe4960 0x561e2aff3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.369+0000 7fd085a16740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe5350 0x561e2aff4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.369+0000 7fd085a16740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afea4c0 0x561e2aff5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.369+0000 7fd085a16740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c1a6700 con 0x561e2afea4c0 2024-03-28T00:56:52.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.369+0000 7fd085a16740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c1a7a00 con 0x561e2afe5350 2024-03-28T00:56:52.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.369+0000 7fd085a16740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c1a6d40 con 0x561e2afe4960 2024-03-28T00:56:52.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.369+0000 7fd085a16740 10 client.0 _ll_register_callbacks cb 0x7fd07400f900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:56:52.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.369+0000 7fd0813cd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afea4c0 0x561e2aff5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.369+0000 7fd0813cd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afea4c0 0x561e2aff5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:35026/0 (socket says 172.21.15.169:35026) 2024-03-28T00:56:52.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.369+0000 7fd0813cd640 1 -- 172.21.15.169:0/2370569680 learned_addr learned my addr 172.21.15.169:0/2370569680 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:52.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.369+0000 7fd080bcc640 1 --2- 172.21.15.169:0/2370569680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afe4960 0x561e2aff3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.369+0000 7fd07bfff640 1 --2- 172.21.15.169:0/2370569680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe5350 0x561e2aff4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.369+0000 7fd0813cd640 1 -- 172.21.15.169:0/2370569680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afe4960 msgr2=0x561e2aff3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:52.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.369+0000 7fd0813cd640 1 --2- 172.21.15.169:0/2370569680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afe4960 0x561e2aff3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.369+0000 7fd0813cd640 1 -- 172.21.15.169:0/2370569680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe5350 msgr2=0x561e2aff4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:52.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.369+0000 7fd0813cd640 1 --2- 172.21.15.169:0/2370569680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe5350 0x561e2aff4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.369+0000 7fd0813cd640 1 -- 172.21.15.169:0/2370569680 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e2aff5640 con 0x561e2afea4c0 2024-03-28T00:56:52.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.369+0000 7fd080bcc640 1 --2- 172.21.15.169:0/2370569680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afe4960 0x561e2aff3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:52.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.369+0000 7fd07bfff640 1 --2- 172.21.15.169:0/2370569680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe5350 0x561e2aff4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:52.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.369+0000 7fd0813cd640 1 --2- 172.21.15.169:0/2370569680 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afea4c0 0x561e2aff5050 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fd074011d30 tx=0x7fd074005e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6066b9006e9233d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:52.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.369+0000 7fd05bfff640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:56:52.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.370+0000 7fd05bfff640 1 -- 172.21.15.169:0/2370569680 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07403b030 con 0x561e2afea4c0 2024-03-28T00:56:52.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.370+0000 7fd05bfff640 1 -- 172.21.15.169:0/2370569680 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fd074028400 con 0x561e2afea4c0 2024-03-28T00:56:52.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.370+0000 7fd085a16740 1 -- 172.21.15.169:0/2370569680 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd074011db0 con 0x561e2afea4c0 2024-03-28T00:56:52.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.370+0000 7fd05bfff640 1 -- 172.21.15.169:0/2370569680 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd07404f050 con 0x561e2afea4c0 2024-03-28T00:56:52.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.370+0000 7fd085a16740 10 client.15603 fetch_fsmap learned FSMap version 25 2024-03-28T00:56:52.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.370+0000 7fd085a16740 1 -- 172.21.15.169:0/2370569680 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561e2b0565f0 con 0x561e2afea4c0 2024-03-28T00:56:52.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.370+0000 7fd05bfff640 1 -- 172.21.15.169:0/2370569680 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd074011db0 con 0x561e2afea4c0 2024-03-28T00:56:52.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.370+0000 7fd05bfff640 1 -- 172.21.15.169:0/2370569680 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd0740054b0 con 0x561e2afea4c0 2024-03-28T00:56:52.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.370+0000 7fd085a16740 10 client.15603 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:56:52.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.370+0000 7fd085a16740 10 client.15603 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:56:52.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.370+0000 7fd085a16740 20 client.15603 populate_metadata read hostname 'smithi169' 2024-03-28T00:56:52.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.370+0000 7fd085a16740 1 --2- 172.21.15.169:0/2370569680 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561e2b056ee0 0x561e2b059300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.370+0000 7fd085a16740 4 client.15603 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T00:56:52.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.370+0000 7fd085a16740 1 -- 172.21.15.169:0/2370569680 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x561e2af51900 con 0x561e2b056ee0 2024-03-28T00:56:52.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.370+0000 7fd080bcc640 1 --2- 172.21.15.169:0/2370569680 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561e2b056ee0 0x561e2b059300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.371+0000 7fd05bfff640 10 client.15603 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:56:52.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.371+0000 7fd080bcc640 1 --2- 172.21.15.169:0/2370569680 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561e2b056ee0 0x561e2b059300 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f462d1033593c79 server_cookie=ea0689fe7e84e141 in_seq=0 out_seq=0 2024-03-28T00:56:52.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.372+0000 7fd05bfff640 1 -- 172.21.15.169:0/2370569680 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x561e2af51900 con 0x561e2b056ee0 2024-03-28T00:56:52.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.372+0000 7fd05bfff640 10 client.15603 handle_command_reply: tid=0 2024-03-28T00:56:52.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.372+0000 7fd085a16740 1 client.15603 shutdown 2024-03-28T00:56:52.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.372+0000 7fd085a16740 1 -- 172.21.15.169:0/2370569680 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afea4c0 msgr2=0x561e2aff5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.372+0000 7fd085a16740 1 --2- 172.21.15.169:0/2370569680 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afea4c0 0x561e2aff5050 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fd074011d30 tx=0x7fd074005e90 comp rx=0 tx=0).stop 2024-03-28T00:56:52.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.372+0000 7fd085a16740 1 -- 172.21.15.169:0/2370569680 shutdown_connections 2024-03-28T00:56:52.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.372+0000 7fd085a16740 1 --2- 172.21.15.169:0/2370569680 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561e2b056ee0 0x561e2b059300 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.372+0000 7fd085a16740 1 --2- 172.21.15.169:0/2370569680 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2afea4c0 0x561e2aff5050 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.372+0000 7fd085a16740 1 --2- 172.21.15.169:0/2370569680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2afe5350 0x561e2aff4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.372+0000 7fd085a16740 1 --2- 172.21.15.169:0/2370569680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2afe4960 0x561e2aff3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.372+0000 7fd05bfff640 0 client.15603 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:56:52.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.372+0000 7fd085a16740 1 -- 172.21.15.169:0/2370569680 >> 172.21.15.169:0/2370569680 conn(0x561e2afad1a0 msgr2=0x561e2afeed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:52.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.372+0000 7fd085a16740 1 -- 172.21.15.169:0/2370569680 shutdown_connections 2024-03-28T00:56:52.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.372+0000 7fd085a16740 1 -- 172.21.15.169:0/2370569680 wait complete. 2024-03-28T00:56:52.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.372+0000 7fd085a16740 20 client.15603 trim_cache size 0 max 16384 2024-03-28T00:56:52.381 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:56:52.381 INFO:teuthology.orchestra.run.smithi169.stdout: "return_code": 0, 2024-03-28T00:56:52.381 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_tag": "b3f8678a-8fc1-4d2f-9c76-e53112883012", 2024-03-28T00:56:52.381 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": "asynchronous" 2024-03-28T00:56:52.381 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:56:52.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.373+0000 7fd082e30640 1 -- 172.21.15.169:0/436753305 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd06c0a2890 msgr2=0x7fd06c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.373+0000 7fd082e30640 1 --2- 172.21.15.169:0/436753305 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd06c0a2890 0x7fd06c0a4d50 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd074009fd0 tx=0x7fd074011810 comp rx=0 tx=0).stop 2024-03-28T00:56:52.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.373+0000 7fd082e30640 1 -- 172.21.15.169:0/436753305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c1005a0 msgr2=0x7fd07c069370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.373+0000 7fd082e30640 1 --2- 172.21.15.169:0/436753305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c1005a0 0x7fd07c069370 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd07000e9c0 tx=0x7fd07000ee80 comp rx=0 tx=0).stop 2024-03-28T00:56:52.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.373+0000 7fd082e30640 1 -- 172.21.15.169:0/436753305 shutdown_connections 2024-03-28T00:56:52.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.373+0000 7fd082e30640 1 --2- 172.21.15.169:0/436753305 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd06c0a2890 0x7fd06c0a4d50 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.373+0000 7fd082e30640 1 --2- 172.21.15.169:0/436753305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd07c102150 0x7fd07c19ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.373+0000 7fd082e30640 1 --2- 172.21.15.169:0/436753305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd07c101790 0x7fd07c19c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.373+0000 7fd082e30640 1 --2- 172.21.15.169:0/436753305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c1005a0 0x7fd07c069370 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.373+0000 7fd082e30640 1 -- 172.21.15.169:0/436753305 >> 172.21.15.169:0/436753305 conn(0x7fd07c0fbb30 msgr2=0x7fd07c104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:52.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.373+0000 7fd082e30640 1 -- 172.21.15.169:0/436753305 shutdown_connections 2024-03-28T00:56:52.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.373+0000 7fd082e30640 1 -- 172.21.15.169:0/436753305 wait complete. 2024-03-28T00:56:52.386 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T00:56:52.387 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T00:56:52.450 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:52 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/502228291' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:56:52.451 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:52 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1126474931' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:56:52.451 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:52 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4055977596' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:56:52.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.533+0000 7fa22a6ac640 1 -- 172.21.15.169:0/2696179699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa224102150 msgr2=0x7fa2241089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.533+0000 7fa22a6ac640 1 --2- 172.21.15.169:0/2696179699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa224102150 0x7fa2241089e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa21c009ca0 tx=0x7fa21c02c0f0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.533+0000 7fa22a6ac640 1 -- 172.21.15.169:0/2696179699 shutdown_connections 2024-03-28T00:56:52.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.533+0000 7fa22a6ac640 1 --2- 172.21.15.169:0/2696179699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa224102150 0x7fa2241089e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.533+0000 7fa22a6ac640 1 --2- 172.21.15.169:0/2696179699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa224101790 0x7fa224101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.533+0000 7fa22a6ac640 1 --2- 172.21.15.169:0/2696179699 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2241005a0 0x7fa2241009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.533+0000 7fa22a6ac640 1 -- 172.21.15.169:0/2696179699 >> 172.21.15.169:0/2696179699 conn(0x7fa2240fbb30 msgr2=0x7fa2240fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:52.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.533+0000 7fa22a6ac640 1 -- 172.21.15.169:0/2696179699 shutdown_connections 2024-03-28T00:56:52.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.534+0000 7fa22a6ac640 1 -- 172.21.15.169:0/2696179699 wait complete. 2024-03-28T00:56:52.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.534+0000 7fa22a6ac640 1 Processor -- start 2024-03-28T00:56:52.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.534+0000 7fa22a6ac640 1 -- start start 2024-03-28T00:56:52.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.534+0000 7fa22a6ac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2241005a0 0x7fa224069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.534+0000 7fa22a6ac640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa224101790 0x7fa22419c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.534+0000 7fa22a6ac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa224102150 0x7fa22419ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.534+0000 7fa22a6ac640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22410c230 con 0x7fa2241005a0 2024-03-28T00:56:52.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.534+0000 7fa22a6ac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22410c0b0 con 0x7fa224102150 2024-03-28T00:56:52.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.534+0000 7fa22a6ac640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22410c3b0 con 0x7fa224101790 2024-03-28T00:56:52.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.534+0000 7fa223fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2241005a0 0x7fa224069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.534+0000 7fa223fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2241005a0 0x7fa224069370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:35042/0 (socket says 172.21.15.169:35042) 2024-03-28T00:56:52.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.534+0000 7fa223fff640 1 -- 172.21.15.169:0/3984508621 learned_addr learned my addr 172.21.15.169:0/3984508621 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:52.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.535+0000 7fa2237fe640 1 --2- 172.21.15.169:0/3984508621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa224101790 0x7fa22419c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.535+0000 7fa228c49640 1 --2- 172.21.15.169:0/3984508621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa224102150 0x7fa22419ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.535+0000 7fa223fff640 1 -- 172.21.15.169:0/3984508621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa224101790 msgr2=0x7fa22419c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.535+0000 7fa223fff640 1 --2- 172.21.15.169:0/3984508621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa224101790 0x7fa22419c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.535+0000 7fa223fff640 1 -- 172.21.15.169:0/3984508621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa224102150 msgr2=0x7fa22419ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.535+0000 7fa223fff640 1 --2- 172.21.15.169:0/3984508621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa224102150 0x7fa22419ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.535+0000 7fa223fff640 1 -- 172.21.15.169:0/3984508621 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa22419f5e0 con 0x7fa2241005a0 2024-03-28T00:56:52.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.535+0000 7fa223fff640 1 --2- 172.21.15.169:0/3984508621 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2241005a0 0x7fa224069370 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa21800e9c0 tx=0x7fa21800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:52.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.535+0000 7fa2217fa640 1 -- 172.21.15.169:0/3984508621 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21800cd80 con 0x7fa2241005a0 2024-03-28T00:56:52.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.535+0000 7fa22a6ac640 1 -- 172.21.15.169:0/3984508621 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa22419f870 con 0x7fa2241005a0 2024-03-28T00:56:52.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.535+0000 7fa2217fa640 1 -- 172.21.15.169:0/3984508621 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2180035f0 con 0x7fa2241005a0 2024-03-28T00:56:52.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.535+0000 7fa2217fa640 1 -- 172.21.15.169:0/3984508621 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa218010690 con 0x7fa2241005a0 2024-03-28T00:56:52.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.535+0000 7fa22a6ac640 1 -- 172.21.15.169:0/3984508621 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa22419fdc0 con 0x7fa2241005a0 2024-03-28T00:56:52.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.537+0000 7fa2217fa640 1 -- 172.21.15.169:0/3984508621 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fa2180037a0 con 0x7fa2241005a0 2024-03-28T00:56:52.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.538+0000 7fa2217fa640 1 --2- 172.21.15.169:0/3984508621 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa20c0a2890 0x7fa20c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.538+0000 7fa22d372740 1 Processor -- start 2024-03-28T00:56:52.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.538+0000 7fa22d372740 1 -- start start 2024-03-28T00:56:52.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.538+0000 7fa2237fe640 1 --2- 172.21.15.169:0/3984508621 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa20c0a2890 0x7fa20c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.538+0000 7fa2217fa640 1 -- 172.21.15.169:0/3984508621 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa2180cf340 con 0x7fa2241005a0 2024-03-28T00:56:52.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.538+0000 7fa22d372740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb27da0 0x55771eb281c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.538+0000 7fa22d372740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55771eb28790 0x55771eb2d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.538+0000 7fa22d372740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55771eb2d900 0x55771eb2fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.538+0000 7fa22d372740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55771ea8d3d0 con 0x55771eb27da0 2024-03-28T00:56:52.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.538+0000 7fa22d372740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55771ea35ad0 con 0x55771eb28790 2024-03-28T00:56:52.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.538+0000 7fa22d372740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55771ea89470 con 0x55771eb2d900 2024-03-28T00:56:52.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.538+0000 7fa223fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb27da0 0x55771eb281c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.538+0000 7fa223fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb27da0 0x55771eb281c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:35046/0 (socket says 172.21.15.169:35046) 2024-03-28T00:56:52.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.538+0000 7fa223fff640 1 -- 172.21.15.169:0/497583319 learned_addr learned my addr 172.21.15.169:0/497583319 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:52.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.538+0000 7fa2237fe640 1 --2- 172.21.15.169:0/497583319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55771eb28790 0x55771eb2d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.538+0000 7fa228c49640 1 --2- 172.21.15.169:0/497583319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55771eb2d900 0x55771eb2fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.538+0000 7fa2237fe640 1 --2- 172.21.15.169:0/3984508621 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa20c0a2890 0x7fa20c0a4d50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa2140095f0 tx=0x7fa214005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:52.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.539+0000 7fa223fff640 1 -- 172.21.15.169:0/497583319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55771eb2d900 msgr2=0x55771eb2fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:52.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.539+0000 7fa223fff640 1 --2- 172.21.15.169:0/497583319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55771eb2d900 0x55771eb2fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.539+0000 7fa223fff640 1 -- 172.21.15.169:0/497583319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55771eb28790 msgr2=0x55771eb2d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:52.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.539+0000 7fa223fff640 1 --2- 172.21.15.169:0/497583319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55771eb28790 0x55771eb2d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.539+0000 7fa223fff640 1 -- 172.21.15.169:0/497583319 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55771e86b7d0 con 0x55771eb27da0 2024-03-28T00:56:52.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.539+0000 7fa228c49640 1 --2- 172.21.15.169:0/497583319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55771eb2d900 0x55771eb2fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:52.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.539+0000 7fa223fff640 1 --2- 172.21.15.169:0/497583319 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb27da0 0x55771eb281c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa218009830 tx=0x7fa21809bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd0068d877d0f14 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:52.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.539+0000 7fa20a7fc640 1 -- 172.21.15.169:0/497583319 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2180b4070 con 0x55771eb27da0 2024-03-28T00:56:52.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.539+0000 7fa20a7fc640 1 -- 172.21.15.169:0/497583319 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa21809f930 con 0x55771eb27da0 2024-03-28T00:56:52.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.539+0000 7fa20a7fc640 1 -- 172.21.15.169:0/497583319 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa218097e50 con 0x55771eb27da0 2024-03-28T00:56:52.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.539+0000 7fa22d372740 1 -- 172.21.15.169:0/497583319 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb27da0 msgr2=0x55771eb281c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.539+0000 7fa22d372740 1 --2- 172.21.15.169:0/497583319 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb27da0 0x55771eb281c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa218009830 tx=0x7fa21809bee0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.540+0000 7fa22d372740 1 -- 172.21.15.169:0/497583319 shutdown_connections 2024-03-28T00:56:52.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.540+0000 7fa22d372740 1 --2- 172.21.15.169:0/497583319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55771eb2d900 0x55771eb2fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.540+0000 7fa22d372740 1 --2- 172.21.15.169:0/497583319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55771eb28790 0x55771eb2d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.540+0000 7fa22d372740 1 --2- 172.21.15.169:0/497583319 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb27da0 0x55771eb281c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.540+0000 7fa22d372740 1 -- 172.21.15.169:0/497583319 >> 172.21.15.169:0/497583319 conn(0x55771eb23310 msgr2=0x55771eb23710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:52.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.540+0000 7fa22d372740 1 -- 172.21.15.169:0/497583319 shutdown_connections 2024-03-28T00:56:52.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.540+0000 7fa22d372740 1 -- 172.21.15.169:0/497583319 wait complete. 2024-03-28T00:56:52.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.540+0000 7fa22d372740 1 -- start start 2024-03-28T00:56:52.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.540+0000 7fa22d372740 1 Processor -- start 2024-03-28T00:56:52.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.540+0000 7fa22d372740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55771eb27da0 0x55771eb32260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.540+0000 7fa22d372740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb28790 0x55771eb327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.540+0000 7fa22d372740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55771eb2d900 0x55771eb397e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.540+0000 7fa22d372740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55771eb308b0 con 0x55771eb28790 2024-03-28T00:56:52.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.540+0000 7fa22d372740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55771ea49060 con 0x55771eb27da0 2024-03-28T00:56:52.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.540+0000 7fa22d372740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55771eb30a30 con 0x55771eb2d900 2024-03-28T00:56:52.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.540+0000 7fa22d372740 10 client.0 _ll_register_callbacks cb 0x55771ea2b6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:56:52.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.540+0000 7fa2237fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb28790 0x55771eb327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.540+0000 7fa223fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55771eb27da0 0x55771eb32260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.540+0000 7fa2237fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb28790 0x55771eb327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:35054/0 (socket says 172.21.15.169:35054) 2024-03-28T00:56:52.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.540+0000 7fa228c49640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55771eb2d900 0x55771eb397e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.540+0000 7fa2237fe640 1 -- 172.21.15.169:0/2682206670 learned_addr learned my addr 172.21.15.169:0/2682206670 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:52.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.541+0000 7fa2237fe640 1 -- 172.21.15.169:0/2682206670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55771eb2d900 msgr2=0x55771eb397e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:52.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.541+0000 7fa2237fe640 1 --2- 172.21.15.169:0/2682206670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55771eb2d900 0x55771eb397e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.541+0000 7fa2237fe640 1 -- 172.21.15.169:0/2682206670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55771eb27da0 msgr2=0x55771eb32260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:52.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.541+0000 7fa2237fe640 1 --2- 172.21.15.169:0/2682206670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55771eb27da0 0x55771eb32260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.541+0000 7fa2237fe640 1 -- 172.21.15.169:0/2682206670 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55771e7a13f0 con 0x55771eb28790 2024-03-28T00:56:52.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.541+0000 7fa223fff640 1 --2- 172.21.15.169:0/2682206670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55771eb27da0 0x55771eb32260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:56:52.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.541+0000 7fa2237fe640 1 --2- 172.21.15.169:0/2682206670 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb28790 0x55771eb327a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa21401c9d0 tx=0x7fa214004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49fe65af919385ba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:52.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.541+0000 7fa2037fe640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:56:52.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.541+0000 7fa2037fe640 1 -- 172.21.15.169:0/2682206670 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21402a060 con 0x55771eb28790 2024-03-28T00:56:52.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.541+0000 7fa2037fe640 1 -- 172.21.15.169:0/2682206670 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa21401bae0 con 0x55771eb28790 2024-03-28T00:56:52.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.541+0000 7fa22d372740 1 -- 172.21.15.169:0/2682206670 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa21809f930 con 0x55771eb28790 2024-03-28T00:56:52.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.541+0000 7fa2037fe640 1 -- 172.21.15.169:0/2682206670 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa214031050 con 0x55771eb28790 2024-03-28T00:56:52.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.541+0000 7fa22d372740 10 client.15615 fetch_fsmap learned FSMap version 25 2024-03-28T00:56:52.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.541+0000 7fa22d372740 1 -- 172.21.15.169:0/2682206670 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55771e7610f0 con 0x55771eb28790 2024-03-28T00:56:52.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.541+0000 7fa2037fe640 1 -- 172.21.15.169:0/2682206670 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa21809f930 con 0x55771eb28790 2024-03-28T00:56:52.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.542+0000 7fa2037fe640 1 -- 172.21.15.169:0/2682206670 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa214024070 con 0x55771eb28790 2024-03-28T00:56:52.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.542+0000 7fa22d372740 10 client.15615 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:56:52.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.542+0000 7fa22d372740 10 client.15615 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:56:52.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.542+0000 7fa22d372740 20 client.15615 populate_metadata read hostname 'smithi169' 2024-03-28T00:56:52.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.542+0000 7fa22d372740 1 --2- 172.21.15.169:0/2682206670 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55771eb3dcc0 0x55771eb40120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.542+0000 7fa22d372740 4 client.15615 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:56:52.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.542+0000 7fa22d372740 1 -- 172.21.15.169:0/2682206670 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55771ea9a910 con 0x55771eb3dcc0 2024-03-28T00:56:52.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.542+0000 7fa223fff640 1 --2- 172.21.15.169:0/2682206670 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55771eb3dcc0 0x55771eb40120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.542+0000 7fa223fff640 1 --2- 172.21.15.169:0/2682206670 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55771eb3dcc0 0x55771eb40120 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72c23b7190920144 server_cookie=1df666463f900444 in_seq=0 out_seq=0 2024-03-28T00:56:52.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.542+0000 7fa2037fe640 10 client.15615 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:56:52.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.543+0000 7fa2037fe640 1 -- 172.21.15.169:0/2682206670 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55771ea9a910 con 0x55771eb3dcc0 2024-03-28T00:56:52.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.543+0000 7fa2037fe640 10 client.15615 handle_command_reply: tid=0 2024-03-28T00:56:52.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.543+0000 7fa22d372740 1 client.15615 shutdown 2024-03-28T00:56:52.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.543+0000 7fa22d372740 1 -- 172.21.15.169:0/2682206670 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb28790 msgr2=0x55771eb327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.543+0000 7fa22d372740 1 --2- 172.21.15.169:0/2682206670 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb28790 0x55771eb327a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa21401c9d0 tx=0x7fa214004730 comp rx=0 tx=0).stop 2024-03-28T00:56:52.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.543+0000 7fa22d372740 1 -- 172.21.15.169:0/2682206670 shutdown_connections 2024-03-28T00:56:52.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.543+0000 7fa22d372740 1 --2- 172.21.15.169:0/2682206670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55771eb2d900 0x55771eb397e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.543+0000 7fa22d372740 1 --2- 172.21.15.169:0/2682206670 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55771eb3dcc0 0x55771eb40120 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.543+0000 7fa22d372740 1 --2- 172.21.15.169:0/2682206670 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb28790 0x55771eb327a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.543+0000 7fa22d372740 1 --2- 172.21.15.169:0/2682206670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55771eb27da0 0x55771eb32260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.543+0000 7fa2037fe640 0 client.15615 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:56:52.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.543+0000 7fa22d372740 1 -- 172.21.15.169:0/2682206670 >> 172.21.15.169:0/2682206670 conn(0x55771eb23310 msgr2=0x55771eb22530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:52.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.543+0000 7fa22d372740 1 -- 172.21.15.169:0/2682206670 shutdown_connections 2024-03-28T00:56:52.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.543+0000 7fa22d372740 1 -- 172.21.15.169:0/2682206670 wait complete. 2024-03-28T00:56:52.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.543+0000 7fa22d372740 20 client.15615 trim_cache size 0 max 16384 2024-03-28T00:56:52.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.561+0000 7fa22d372740 1 Processor -- start 2024-03-28T00:56:52.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.561+0000 7fa22d372740 1 -- start start 2024-03-28T00:56:52.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.561+0000 7fa22d372740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb27da0 0x55771eb32260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.561+0000 7fa22d372740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55771eb28790 0x55771eb397a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.561+0000 7fa22d372740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55771eb2d900 0x55771eb327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.561+0000 7fa22d372740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2241ac760 con 0x55771eb27da0 2024-03-28T00:56:52.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.561+0000 7fa22d372740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2241aa260 con 0x55771eb2d900 2024-03-28T00:56:52.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.561+0000 7fa22d372740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2241a8300 con 0x55771eb28790 2024-03-28T00:56:52.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.561+0000 7fa223fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb27da0 0x55771eb32260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.561+0000 7fa223fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb27da0 0x55771eb32260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:35058/0 (socket says 172.21.15.169:35058) 2024-03-28T00:56:52.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.561+0000 7fa223fff640 1 -- 172.21.15.169:0/3891672450 learned_addr learned my addr 172.21.15.169:0/3891672450 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:52.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.561+0000 7fa228c49640 1 --2- 172.21.15.169:0/3891672450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55771eb2d900 0x55771eb327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.561+0000 7fa2237fe640 1 --2- 172.21.15.169:0/3891672450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55771eb28790 0x55771eb397a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.562+0000 7fa223fff640 1 -- 172.21.15.169:0/3891672450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55771eb28790 msgr2=0x55771eb397a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:52.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.562+0000 7fa223fff640 1 --2- 172.21.15.169:0/3891672450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55771eb28790 0x55771eb397a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.562+0000 7fa223fff640 1 -- 172.21.15.169:0/3891672450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55771eb2d900 msgr2=0x55771eb327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:52.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.562+0000 7fa223fff640 1 --2- 172.21.15.169:0/3891672450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55771eb2d900 0x55771eb327a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.562+0000 7fa223fff640 1 -- 172.21.15.169:0/3891672450 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55771eb23140 con 0x55771eb27da0 2024-03-28T00:56:52.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.562+0000 7fa223fff640 1 --2- 172.21.15.169:0/3891672450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb27da0 0x55771eb32260 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa2180058c0 tx=0x7fa218097b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c37dcdefe2dbd1f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:52.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.562+0000 7fa209ffb640 1 -- 172.21.15.169:0/3891672450 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2180b9070 con 0x55771eb27da0 2024-03-28T00:56:52.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.562+0000 7fa209ffb640 1 -- 172.21.15.169:0/3891672450 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa21809f930 con 0x55771eb27da0 2024-03-28T00:56:52.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.562+0000 7fa209ffb640 1 -- 172.21.15.169:0/3891672450 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2180b4040 con 0x55771eb27da0 2024-03-28T00:56:52.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.562+0000 7fa22d372740 1 -- 172.21.15.169:0/3891672450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb27da0 msgr2=0x55771eb32260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.562+0000 7fa22d372740 1 --2- 172.21.15.169:0/3891672450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb27da0 0x55771eb32260 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa2180058c0 tx=0x7fa218097b00 comp rx=0 tx=0).stop 2024-03-28T00:56:52.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.562+0000 7fa22d372740 1 -- 172.21.15.169:0/3891672450 shutdown_connections 2024-03-28T00:56:52.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.562+0000 7fa22d372740 1 --2- 172.21.15.169:0/3891672450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55771eb2d900 0x55771eb327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.562+0000 7fa22d372740 1 --2- 172.21.15.169:0/3891672450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55771eb28790 0x55771eb397a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.562+0000 7fa22d372740 1 --2- 172.21.15.169:0/3891672450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb27da0 0x55771eb32260 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.562+0000 7fa22d372740 1 -- 172.21.15.169:0/3891672450 >> 172.21.15.169:0/3891672450 conn(0x55771eb23310 msgr2=0x55771eb29c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:52.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.562+0000 7fa22d372740 1 -- 172.21.15.169:0/3891672450 shutdown_connections 2024-03-28T00:56:52.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.562+0000 7fa22d372740 1 -- 172.21.15.169:0/3891672450 wait complete. 2024-03-28T00:56:52.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.563+0000 7fa22d372740 1 -- start start 2024-03-28T00:56:52.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.563+0000 7fa22d372740 1 Processor -- start 2024-03-28T00:56:52.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.563+0000 7fa22d372740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb27da0 0x55771eb36ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.563+0000 7fa22d372740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55771eb28790 0x55771eb37220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.563+0000 7fa22d372740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55771eb2d900 0x55771eb38230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.563+0000 7fa22d372740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2241a8900 con 0x55771eb27da0 2024-03-28T00:56:52.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.563+0000 7fa22d372740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2241a9c00 con 0x55771eb28790 2024-03-28T00:56:52.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.563+0000 7fa22d372740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2241a8f40 con 0x55771eb2d900 2024-03-28T00:56:52.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.563+0000 7fa22d372740 10 client.0 _ll_register_callbacks cb 0x7fa214012d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:56:52.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.563+0000 7fa223fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb27da0 0x55771eb36ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.563+0000 7fa223fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb27da0 0x55771eb36ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:35070/0 (socket says 172.21.15.169:35070) 2024-03-28T00:56:52.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.563+0000 7fa223fff640 1 -- 172.21.15.169:0/2340811054 learned_addr learned my addr 172.21.15.169:0/2340811054 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:56:52.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.563+0000 7fa2237fe640 1 --2- 172.21.15.169:0/2340811054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55771eb28790 0x55771eb37220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.563+0000 7fa228c49640 1 --2- 172.21.15.169:0/2340811054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55771eb2d900 0x55771eb38230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.564+0000 7fa223fff640 1 -- 172.21.15.169:0/2340811054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55771eb2d900 msgr2=0x55771eb38230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:52.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.564+0000 7fa223fff640 1 --2- 172.21.15.169:0/2340811054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55771eb2d900 0x55771eb38230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.564+0000 7fa223fff640 1 -- 172.21.15.169:0/2340811054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55771eb28790 msgr2=0x55771eb37220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:56:52.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.564+0000 7fa223fff640 1 --2- 172.21.15.169:0/2340811054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55771eb28790 0x55771eb37220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.564+0000 7fa223fff640 1 -- 172.21.15.169:0/2340811054 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55771eb38820 con 0x55771eb27da0 2024-03-28T00:56:52.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.564+0000 7fa223fff640 1 --2- 172.21.15.169:0/2340811054 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb27da0 0x55771eb36ce0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa2180cc9d0 tx=0x7fa2180097c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d038766694ec1333 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:56:52.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.564+0000 7fa2037fe640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:56:52.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.564+0000 7fa2037fe640 1 -- 172.21.15.169:0/2340811054 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21807f060 con 0x55771eb27da0 2024-03-28T00:56:52.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.564+0000 7fa2037fe640 1 -- 172.21.15.169:0/2340811054 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa218097e50 con 0x55771eb27da0 2024-03-28T00:56:52.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.564+0000 7fa22d372740 1 -- 172.21.15.169:0/2340811054 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa21809f930 con 0x55771eb27da0 2024-03-28T00:56:52.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.564+0000 7fa2037fe640 1 -- 172.21.15.169:0/2340811054 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2180b4030 con 0x55771eb27da0 2024-03-28T00:56:52.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.564+0000 7fa22d372740 10 client.15621 fetch_fsmap learned FSMap version 25 2024-03-28T00:56:52.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.564+0000 7fa22d372740 1 -- 172.21.15.169:0/2340811054 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55771eb995f0 con 0x55771eb27da0 2024-03-28T00:56:52.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.564+0000 7fa2037fe640 1 -- 172.21.15.169:0/2340811054 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa21809f930 con 0x55771eb27da0 2024-03-28T00:56:52.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.564+0000 7fa2037fe640 1 -- 172.21.15.169:0/2340811054 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa218097530 con 0x55771eb27da0 2024-03-28T00:56:52.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.564+0000 7fa22d372740 10 client.15621 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:56:52.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.564+0000 7fa22d372740 10 client.15621 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:56:52.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.564+0000 7fa22d372740 20 client.15621 populate_metadata read hostname 'smithi169' 2024-03-28T00:56:52.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.564+0000 7fa22d372740 1 --2- 172.21.15.169:0/2340811054 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55771eb99910 0x55771eb9bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:56:52.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.564+0000 7fa22d372740 4 client.15621 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T00:56:52.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.564+0000 7fa22d372740 1 -- 172.21.15.169:0/2340811054 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55771ea94c60 con 0x55771eb99910 2024-03-28T00:56:52.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.565+0000 7fa2237fe640 1 --2- 172.21.15.169:0/2340811054 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55771eb99910 0x55771eb9bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:56:52.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.565+0000 7fa2037fe640 10 client.15621 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:56:52.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.565+0000 7fa2237fe640 1 --2- 172.21.15.169:0/2340811054 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55771eb99910 0x55771eb9bd00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed0034881cf48c8d server_cookie=d9ca3aa4f07ad295 in_seq=0 out_seq=0 2024-03-28T00:56:52.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.565+0000 7fa2037fe640 1 -- 172.21.15.169:0/2340811054 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55771ea94c60 con 0x55771eb99910 2024-03-28T00:56:52.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.565+0000 7fa2037fe640 10 client.15621 handle_command_reply: tid=0 2024-03-28T00:56:52.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.565+0000 7fa22d372740 1 client.15621 shutdown 2024-03-28T00:56:52.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.565+0000 7fa22d372740 1 -- 172.21.15.169:0/2340811054 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb27da0 msgr2=0x55771eb36ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.565+0000 7fa22d372740 1 --2- 172.21.15.169:0/2340811054 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb27da0 0x55771eb36ce0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa2180cc9d0 tx=0x7fa2180097c0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.565+0000 7fa22d372740 1 -- 172.21.15.169:0/2340811054 shutdown_connections 2024-03-28T00:56:52.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.565+0000 7fa22d372740 1 --2- 172.21.15.169:0/2340811054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55771eb2d900 0x55771eb38230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.565+0000 7fa22d372740 1 --2- 172.21.15.169:0/2340811054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55771eb28790 0x55771eb37220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.565+0000 7fa22d372740 1 --2- 172.21.15.169:0/2340811054 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55771eb99910 0x55771eb9bd00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.565+0000 7fa22d372740 1 --2- 172.21.15.169:0/2340811054 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55771eb27da0 0x55771eb36ce0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.565+0000 7fa2037fe640 0 client.15621 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:56:52.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.565+0000 7fa22d372740 1 -- 172.21.15.169:0/2340811054 >> 172.21.15.169:0/2340811054 conn(0x55771eb23310 msgr2=0x55771eb29c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:52.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.565+0000 7fa22d372740 1 -- 172.21.15.169:0/2340811054 shutdown_connections 2024-03-28T00:56:52.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.565+0000 7fa22d372740 1 -- 172.21.15.169:0/2340811054 wait complete. 2024-03-28T00:56:52.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.565+0000 7fa22d372740 20 client.15621 trim_cache size 0 max 16384 2024-03-28T00:56:52.576 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:56:52.576 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T00:56:52.576 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": { 2024-03-28T00:56:52.576 INFO:teuthology.orchestra.run.smithi169.stdout: "b3f8678a-8fc1-4d2f-9c76-e53112883012": { 2024-03-28T00:56:52.576 INFO:teuthology.orchestra.run.smithi169.stdout: "path": "/", 2024-03-28T00:56:52.576 INFO:teuthology.orchestra.run.smithi169.stdout: "tag": "b3f8678a-8fc1-4d2f-9c76-e53112883012", 2024-03-28T00:56:52.576 INFO:teuthology.orchestra.run.smithi169.stdout: "options": "recursive,force" 2024-03-28T00:56:52.577 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T00:56:52.577 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T00:56:52.577 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:56:52.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.566+0000 7fa22a6ac640 1 -- 172.21.15.169:0/3984508621 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa20c0a2890 msgr2=0x7fa20c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.566+0000 7fa22a6ac640 1 --2- 172.21.15.169:0/3984508621 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa20c0a2890 0x7fa20c0a4d50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa2140095f0 tx=0x7fa214005f90 comp rx=0 tx=0).stop 2024-03-28T00:56:52.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.566+0000 7fa22a6ac640 1 -- 172.21.15.169:0/3984508621 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2241005a0 msgr2=0x7fa224069370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:56:52.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.566+0000 7fa22a6ac640 1 --2- 172.21.15.169:0/3984508621 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2241005a0 0x7fa224069370 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa21800e9c0 tx=0x7fa21800ee80 comp rx=0 tx=0).stop 2024-03-28T00:56:52.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.566+0000 7fa22a6ac640 1 -- 172.21.15.169:0/3984508621 shutdown_connections 2024-03-28T00:56:52.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.566+0000 7fa22a6ac640 1 --2- 172.21.15.169:0/3984508621 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa20c0a2890 0x7fa20c0a4d50 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.566+0000 7fa22a6ac640 1 --2- 172.21.15.169:0/3984508621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa224102150 0x7fa22419ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.566+0000 7fa22a6ac640 1 --2- 172.21.15.169:0/3984508621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa224101790 0x7fa22419c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.566+0000 7fa22a6ac640 1 --2- 172.21.15.169:0/3984508621 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa2241005a0 0x7fa224069370 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:56:52.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.566+0000 7fa22a6ac640 1 -- 172.21.15.169:0/3984508621 >> 172.21.15.169:0/3984508621 conn(0x7fa2240fbb30 msgr2=0x7fa224104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:56:52.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.566+0000 7fa22a6ac640 1 -- 172.21.15.169:0/3984508621 shutdown_connections 2024-03-28T00:56:52.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:56:52.566+0000 7fa22a6ac640 1 -- 172.21.15.169:0/3984508621 wait complete. 2024-03-28T00:56:52.580 INFO:tasks.cephfs.filesystem:scrub status for tag:b3f8678a-8fc1-4d2f-9c76-e53112883012 - {'path': '/', 'tag': 'b3f8678a-8fc1-4d2f-9c76-e53112883012', 'options': 'recursive,force'} 2024-03-28T00:56:52.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:52 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/502228291' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:56:52.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:52 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1126474931' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:56:52.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:52 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/4055977596' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:56:52.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:52 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/502228291' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T00:56:52.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:52 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1126474931' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T00:56:52.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:52 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/4055977596' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:56:53.535 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:53 smithi169 ceph-mon[32274]: pgmap v362: 129 pgs: 129 active+clean; 653 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-28T00:56:53.535 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:53 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1880281113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:53.535 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:53 smithi169 ceph-mon[32274]: scrub queued for path: / 2024-03-28T00:56:53.535 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:53 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T00:56:53.535 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:53 smithi169 ceph-mon[32274]: scrub summary: active paths [/] 2024-03-28T00:56:53.535 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:53 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T00:56:53.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:53 smithi174 ceph-mon[36060]: pgmap v362: 129 pgs: 129 active+clean; 653 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-28T00:56:53.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:53 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1880281113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:53.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:53 smithi174 ceph-mon[36060]: scrub queued for path: / 2024-03-28T00:56:53.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:53 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T00:56:53.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:53 smithi174 ceph-mon[36060]: scrub summary: active paths [/] 2024-03-28T00:56:53.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:53 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T00:56:53.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:53 smithi177 ceph-mon[36206]: pgmap v362: 129 pgs: 129 active+clean; 653 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-28T00:56:53.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:53 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1880281113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T00:56:53.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:53 smithi177 ceph-mon[36206]: scrub queued for path: / 2024-03-28T00:56:53.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:53 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T00:56:53.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:53 smithi177 ceph-mon[36206]: scrub summary: active paths [/] 2024-03-28T00:56:53.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:53 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T00:56:55.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:55 smithi169 ceph-mon[32274]: pgmap v363: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-03-28T00:56:55.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:55 smithi174 ceph-mon[36060]: pgmap v363: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-03-28T00:56:55.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:55 smithi177 ceph-mon[36206]: pgmap v363: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-03-28T00:56:57.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:57 smithi169 ceph-mon[32274]: pgmap v364: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-28T00:56:57.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:57 smithi174 ceph-mon[36060]: pgmap v364: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-28T00:56:57.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:57 smithi177 ceph-mon[36206]: pgmap v364: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-28T00:56:59.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:56:59 smithi169 ceph-mon[32274]: pgmap v365: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-28T00:56:59.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:56:59 smithi174 ceph-mon[36060]: pgmap v365: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-28T00:56:59.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:56:59 smithi177 ceph-mon[36206]: pgmap v365: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-28T00:57:01.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:01 smithi169 ceph-mon[32274]: pgmap v366: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-28T00:57:01.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:01 smithi174 ceph-mon[36060]: pgmap v366: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-28T00:57:01.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:01 smithi177 ceph-mon[36206]: pgmap v366: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-28T00:57:02.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:02 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:57:02.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:02 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:57:02.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:02 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:57:03.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:03 smithi169 ceph-mon[32274]: pgmap v367: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-03-28T00:57:03.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:03 smithi177 ceph-mon[36206]: pgmap v367: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-03-28T00:57:03.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:03 smithi174 ceph-mon[36060]: pgmap v367: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-03-28T00:57:05.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:04 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:57:05.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:04 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:57:05.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:04 smithi169 ceph-mon[32274]: scrub summary: idle 2024-03-28T00:57:05.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:04 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:57:05.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:04 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:57:05.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:04 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:57:05.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:04 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:57:05.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:04 smithi169 ceph-mon[32274]: pgmap v368: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-28T00:57:05.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:04 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:57:05.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:04 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:57:05.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:04 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:57:05.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:04 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:57:05.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:04 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:57:05.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:04 smithi174 ceph-mon[36060]: scrub summary: idle 2024-03-28T00:57:05.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:04 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:57:05.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:04 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:57:05.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:04 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:57:05.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:04 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:57:05.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:04 smithi174 ceph-mon[36060]: pgmap v368: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-28T00:57:05.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:04 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:57:05.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:04 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:57:05.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:04 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:57:05.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:04 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:57:05.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:04 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:57:05.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:04 smithi177 ceph-mon[36206]: scrub summary: idle 2024-03-28T00:57:05.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:04 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:57:05.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:04 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:57:05.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:04 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:57:05.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:04 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:57:05.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:04 smithi177 ceph-mon[36206]: pgmap v368: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-28T00:57:05.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:04 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:57:05.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:04 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:57:05.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:04 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:57:05.377 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.372+0000 7fae9c597640 1 -- 172.21.15.169:0/2236591953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae94102190 msgr2=0x7fae94104580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:05.377 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.372+0000 7fae9c597640 1 --2- 172.21.15.169:0/2236591953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae94102190 0x7fae94104580 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fae90009860 tx=0x7fae9002bb70 comp rx=0 tx=0).stop 2024-03-28T00:57:05.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.372+0000 7fae9c597640 1 -- 172.21.15.169:0/2236591953 shutdown_connections 2024-03-28T00:57:05.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.372+0000 7fae9c597640 1 --2- 172.21.15.169:0/2236591953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae941073f0 0x7fae94109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.372+0000 7fae9c597640 1 --2- 172.21.15.169:0/2236591953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae94104ac0 0x7fae94106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.372+0000 7fae9c597640 1 --2- 172.21.15.169:0/2236591953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae94102190 0x7fae94104580 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.372+0000 7fae9c597640 1 -- 172.21.15.169:0/2236591953 >> 172.21.15.169:0/2236591953 conn(0x7fae940fbb80 msgr2=0x7fae940fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:05.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.372+0000 7fae9c597640 1 -- 172.21.15.169:0/2236591953 shutdown_connections 2024-03-28T00:57:05.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.372+0000 7fae9c597640 1 -- 172.21.15.169:0/2236591953 wait complete. 2024-03-28T00:57:05.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.373+0000 7fae9c597640 1 Processor -- start 2024-03-28T00:57:05.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.373+0000 7fae9c597640 1 -- start start 2024-03-28T00:57:05.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.373+0000 7fae9c597640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae94102190 0x7fae941979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:05.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.373+0000 7fae9c597640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae94104ac0 0x7fae94197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:05.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.373+0000 7fae9c597640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae941073f0 0x7fae9419eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:05.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.373+0000 7fae9c597640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9410d3b0 con 0x7fae94102190 2024-03-28T00:57:05.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.373+0000 7fae9c597640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9410d230 con 0x7fae94104ac0 2024-03-28T00:57:05.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.373+0000 7fae9c597640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9410d530 con 0x7fae941073f0 2024-03-28T00:57:05.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.373+0000 7fae9a333640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae94102190 0x7fae941979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:05.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.373+0000 7fae99b32640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae94104ac0 0x7fae94197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:05.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.373+0000 7fae99b32640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae94104ac0 0x7fae94197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:38764/0 (socket says 172.21.15.169:38764) 2024-03-28T00:57:05.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.373+0000 7fae9a333640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae94102190 0x7fae941979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53024/0 (socket says 172.21.15.169:53024) 2024-03-28T00:57:05.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.373+0000 7fae99b32640 1 -- 172.21.15.169:0/2048108999 learned_addr learned my addr 172.21.15.169:0/2048108999 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:05.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.373+0000 7fae9ab34640 1 --2- 172.21.15.169:0/2048108999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae941073f0 0x7fae9419eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:05.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.374+0000 7fae99b32640 1 -- 172.21.15.169:0/2048108999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae941073f0 msgr2=0x7fae9419eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:05.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.374+0000 7fae99b32640 1 --2- 172.21.15.169:0/2048108999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae941073f0 0x7fae9419eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.374+0000 7fae99b32640 1 -- 172.21.15.169:0/2048108999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae94102190 msgr2=0x7fae941979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:05.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.374+0000 7fae99b32640 1 --2- 172.21.15.169:0/2048108999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae94102190 0x7fae941979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.374+0000 7fae99b32640 1 -- 172.21.15.169:0/2048108999 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9419f550 con 0x7fae94104ac0 2024-03-28T00:57:05.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.374+0000 7fae99b32640 1 --2- 172.21.15.169:0/2048108999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae94104ac0 0x7fae94197f30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fae8400e9a0 tx=0x7fae8400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:05.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.374+0000 7fae8b7fe640 1 -- 172.21.15.169:0/2048108999 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8400cd80 con 0x7fae94104ac0 2024-03-28T00:57:05.380 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.374+0000 7fae8b7fe640 1 -- 172.21.15.169:0/2048108999 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae840035f0 con 0x7fae94104ac0 2024-03-28T00:57:05.380 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.374+0000 7fae9c597640 1 -- 172.21.15.169:0/2048108999 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9419f840 con 0x7fae94104ac0 2024-03-28T00:57:05.380 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.374+0000 7fae8b7fe640 1 -- 172.21.15.169:0/2048108999 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae84010670 con 0x7fae94104ac0 2024-03-28T00:57:05.380 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.374+0000 7fae9c597640 1 -- 172.21.15.169:0/2048108999 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9419fde0 con 0x7fae94104ac0 2024-03-28T00:57:05.386 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.376+0000 7fae8b7fe640 1 -- 172.21.15.169:0/2048108999 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fae840037a0 con 0x7fae94104ac0 2024-03-28T00:57:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.377+0000 7fae9f186740 1 Processor -- start 2024-03-28T00:57:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.377+0000 7fae8b7fe640 1 --2- 172.21.15.169:0/2048108999 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fae800a29c0 0x7fae800a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.377+0000 7fae9f186740 1 -- start start 2024-03-28T00:57:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.377+0000 7fae9f186740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f3920 0x55933c3f3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.377+0000 7fae8b7fe640 1 -- 172.21.15.169:0/2048108999 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fae840cf180 con 0x7fae94104ac0 2024-03-28T00:57:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.377+0000 7fae9f186740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f4310 0x55933c3f8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.377+0000 7fae9ab34640 1 --2- 172.21.15.169:0/2048108999 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fae800a29c0 0x7fae800a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.377+0000 7fae9f186740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55933c3f9480 0x55933c3fb910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.377+0000 7fae9f186740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55933c358b10 con 0x55933c3f3920 2024-03-28T00:57:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.377+0000 7fae9f186740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55933c301900 con 0x55933c3f4310 2024-03-28T00:57:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.377+0000 7fae9f186740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55933c354bb0 con 0x55933c3f9480 2024-03-28T00:57:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.377+0000 7fae9a333640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f3920 0x55933c3f3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.377+0000 7fae9a333640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f3920 0x55933c3f3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53030/0 (socket says 172.21.15.169:53030) 2024-03-28T00:57:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.377+0000 7fae9ab34640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55933c3f9480 0x55933c3fb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.377+0000 7fae9a333640 1 -- 172.21.15.169:0/3735959227 learned_addr learned my addr 172.21.15.169:0/3735959227 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.377+0000 7fae99b32640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f4310 0x55933c3f8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.378+0000 7fae9a333640 1 -- 172.21.15.169:0/3735959227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55933c3f9480 msgr2=0x55933c3fb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.378+0000 7fae9a333640 1 --2- 172.21.15.169:0/3735959227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55933c3f9480 0x55933c3fb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.378+0000 7fae9a333640 1 -- 172.21.15.169:0/3735959227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f4310 msgr2=0x55933c3f8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.378+0000 7fae9a333640 1 --2- 172.21.15.169:0/3735959227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f4310 0x55933c3f8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.378+0000 7fae9a333640 1 -- 172.21.15.169:0/3735959227 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55933c06d210 con 0x55933c3f3920 2024-03-28T00:57:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.378+0000 7fae99b32640 1 --2- 172.21.15.169:0/3735959227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f4310 0x55933c3f8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.378+0000 7fae9ab34640 1 --2- 172.21.15.169:0/2048108999 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fae800a29c0 0x7fae800a4e80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fae8c005fd0 tx=0x7fae8c009e00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.378+0000 7fae9a333640 1 --2- 172.21.15.169:0/3735959227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f3920 0x55933c3f3d40 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fae9002c180 tx=0x7fae90005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0e72092487cd270 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.378+0000 7fae88bfe640 1 -- 172.21.15.169:0/3735959227 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae90043020 con 0x55933c3f3920 2024-03-28T00:57:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.378+0000 7fae88bfe640 1 -- 172.21.15.169:0/3735959227 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae900323c0 con 0x55933c3f3920 2024-03-28T00:57:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.378+0000 7fae88bfe640 1 -- 172.21.15.169:0/3735959227 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae90039040 con 0x55933c3f3920 2024-03-28T00:57:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.379+0000 7fae9f186740 1 -- 172.21.15.169:0/3735959227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f3920 msgr2=0x55933c3f3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.379+0000 7fae9f186740 1 --2- 172.21.15.169:0/3735959227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f3920 0x55933c3f3d40 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fae9002c180 tx=0x7fae90005b20 comp rx=0 tx=0).stop 2024-03-28T00:57:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.379+0000 7fae9f186740 1 -- 172.21.15.169:0/3735959227 shutdown_connections 2024-03-28T00:57:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.379+0000 7fae9f186740 1 --2- 172.21.15.169:0/3735959227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55933c3f9480 0x55933c3fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.379+0000 7fae9f186740 1 --2- 172.21.15.169:0/3735959227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f4310 0x55933c3f8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.379+0000 7fae9f186740 1 --2- 172.21.15.169:0/3735959227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f3920 0x55933c3f3d40 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.379+0000 7fae9f186740 1 -- 172.21.15.169:0/3735959227 >> 172.21.15.169:0/3735959227 conn(0x55933c3eee90 msgr2=0x55933c3ef290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.379+0000 7fae9f186740 1 -- 172.21.15.169:0/3735959227 shutdown_connections 2024-03-28T00:57:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.379+0000 7fae9f186740 1 -- 172.21.15.169:0/3735959227 wait complete. 2024-03-28T00:57:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.379+0000 7fae9f186740 1 -- start start 2024-03-28T00:57:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.379+0000 7fae9f186740 1 Processor -- start 2024-03-28T00:57:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae9f186740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f3920 0x55933c401f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae9f186740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55933c3f4310 0x55933c4024d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae9f186740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f9480 0x55933c409510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae9f186740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55933c4007e0 con 0x55933c3f3920 2024-03-28T00:57:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae9f186740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55933c314d50 con 0x55933c3f9480 2024-03-28T00:57:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae9f186740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55933c400960 con 0x55933c3f4310 2024-03-28T00:57:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae9f186740 10 client.0 _ll_register_callbacks cb 0x55933c314720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae9a333640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f3920 0x55933c401f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae99b32640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55933c3f4310 0x55933c4024d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae9a333640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f3920 0x55933c401f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53036/0 (socket says 172.21.15.169:53036) 2024-03-28T00:57:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae9a333640 1 -- 172.21.15.169:0/767893795 learned_addr learned my addr 172.21.15.169:0/767893795 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae9ab34640 1 --2- 172.21.15.169:0/767893795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f9480 0x55933c409510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae9ab34640 1 -- 172.21.15.169:0/767893795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55933c3f4310 msgr2=0x55933c4024d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae9ab34640 1 --2- 172.21.15.169:0/767893795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55933c3f4310 0x55933c4024d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae9ab34640 1 -- 172.21.15.169:0/767893795 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f3920 msgr2=0x55933c401f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae9ab34640 1 --2- 172.21.15.169:0/767893795 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f3920 0x55933c401f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae9ab34640 1 -- 172.21.15.169:0/767893795 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55933c139620 con 0x55933c3f9480 2024-03-28T00:57:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae9a333640 1 --2- 172.21.15.169:0/767893795 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f3920 0x55933c401f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae9ab34640 1 --2- 172.21.15.169:0/767893795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f9480 0x55933c409510 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fae8c009dd0 tx=0x7fae8c01d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81ab0f480448e24 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:57:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae7dffb640 1 -- 172.21.15.169:0/767893795 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c004b20 con 0x55933c3f9480 2024-03-28T00:57:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae7dffb640 1 -- 172.21.15.169:0/767893795 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fae8c01c040 con 0x55933c3f9480 2024-03-28T00:57:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.380+0000 7fae9f186740 1 -- 172.21.15.169:0/767893795 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fae900323c0 con 0x55933c3f9480 2024-03-28T00:57:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.381+0000 7fae7dffb640 1 -- 172.21.15.169:0/767893795 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae8c031050 con 0x55933c3f9480 2024-03-28T00:57:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.381+0000 7fae9f186740 10 client.25457 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.381+0000 7fae9f186740 1 -- 172.21.15.169:0/767893795 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55933c40c8f0 con 0x55933c3f9480 2024-03-28T00:57:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.381+0000 7fae7dffb640 1 -- 172.21.15.169:0/767893795 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae900323c0 con 0x55933c3f9480 2024-03-28T00:57:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.381+0000 7fae7dffb640 1 -- 172.21.15.169:0/767893795 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fae8c028070 con 0x55933c3f9480 2024-03-28T00:57:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.381+0000 7fae9f186740 10 client.25457 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.381+0000 7fae9f186740 10 client.25457 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.381+0000 7fae9f186740 20 client.25457 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.381+0000 7fae9f186740 1 --2- 172.21.15.169:0/767893795 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55933c40df60 0x55933c4103d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.381+0000 7fae9f186740 4 client.25457 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:57:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.381+0000 7fae9f186740 1 -- 172.21.15.169:0/767893795 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55933c366050 con 0x55933c40df60 2024-03-28T00:57:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.381+0000 7fae9a333640 1 --2- 172.21.15.169:0/767893795 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55933c40df60 0x55933c4103d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.382+0000 7fae9a333640 1 --2- 172.21.15.169:0/767893795 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55933c40df60 0x55933c4103d0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55105e28f6eb90b4 server_cookie=8fec234ef0560102 in_seq=0 out_seq=0 2024-03-28T00:57:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.382+0000 7fae7dffb640 10 client.25457 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.382+0000 7fae7dffb640 1 -- 172.21.15.169:0/767893795 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55933c366050 con 0x55933c40df60 2024-03-28T00:57:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.382+0000 7fae7dffb640 10 client.25457 handle_command_reply: tid=0 2024-03-28T00:57:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.382+0000 7fae9f186740 1 client.25457 shutdown 2024-03-28T00:57:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.382+0000 7fae9f186740 1 -- 172.21.15.169:0/767893795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f9480 msgr2=0x55933c409510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.382+0000 7fae9f186740 1 --2- 172.21.15.169:0/767893795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f9480 0x55933c409510 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fae8c009dd0 tx=0x7fae8c01d000 comp rx=0 tx=0).stop 2024-03-28T00:57:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.383+0000 7fae9f186740 1 -- 172.21.15.169:0/767893795 shutdown_connections 2024-03-28T00:57:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.383+0000 7fae9f186740 1 --2- 172.21.15.169:0/767893795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f9480 0x55933c409510 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.383+0000 7fae9f186740 1 --2- 172.21.15.169:0/767893795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55933c3f4310 0x55933c4024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.383+0000 7fae9f186740 1 --2- 172.21.15.169:0/767893795 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55933c40df60 0x55933c4103d0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.383+0000 7fae9f186740 1 --2- 172.21.15.169:0/767893795 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f3920 0x55933c401f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.383+0000 7fae9f186740 1 -- 172.21.15.169:0/767893795 >> 172.21.15.169:0/767893795 conn(0x55933c3eee90 msgr2=0x55933c3ee020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.383+0000 7fae7dffb640 0 client.25457 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.383+0000 7fae9f186740 1 -- 172.21.15.169:0/767893795 shutdown_connections 2024-03-28T00:57:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.383+0000 7fae9f186740 1 -- 172.21.15.169:0/767893795 wait complete. 2024-03-28T00:57:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.383+0000 7fae9f186740 20 client.25457 trim_cache size 0 max 16384 2024-03-28T00:57:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.396+0000 7fae9f186740 1 Processor -- start 2024-03-28T00:57:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.396+0000 7fae9f186740 1 -- start start 2024-03-28T00:57:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.397+0000 7fae9f186740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f3920 0x55933c401f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.397+0000 7fae9f186740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f4310 0x55933c4024d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.397+0000 7fae9f186740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55933c3f9480 0x55933c4094d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.397+0000 7fae9f186740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae941aa4b0 con 0x55933c3f4310 2024-03-28T00:57:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.397+0000 7fae9f186740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae941a7fb0 con 0x55933c3f3920 2024-03-28T00:57:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.397+0000 7fae9f186740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae941a6050 con 0x55933c3f9480 2024-03-28T00:57:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.397+0000 7fae99b32640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f4310 0x55933c4024d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.397+0000 7fae99b32640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f4310 0x55933c4024d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53040/0 (socket says 172.21.15.169:53040) 2024-03-28T00:57:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.397+0000 7fae99b32640 1 -- 172.21.15.169:0/4018653059 learned_addr learned my addr 172.21.15.169:0/4018653059 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.397+0000 7fae9a333640 1 --2- 172.21.15.169:0/4018653059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f3920 0x55933c401f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.397+0000 7fae9ab34640 1 --2- 172.21.15.169:0/4018653059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55933c3f9480 0x55933c4094d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.397+0000 7fae99b32640 1 -- 172.21.15.169:0/4018653059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55933c3f9480 msgr2=0x55933c4094d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.397+0000 7fae99b32640 1 --2- 172.21.15.169:0/4018653059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55933c3f9480 0x55933c4094d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.397+0000 7fae99b32640 1 -- 172.21.15.169:0/4018653059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f3920 msgr2=0x55933c401f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.397+0000 7fae99b32640 1 --2- 172.21.15.169:0/4018653059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f3920 0x55933c401f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.397+0000 7fae99b32640 1 -- 172.21.15.169:0/4018653059 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55933c3f3d00 con 0x55933c3f4310 2024-03-28T00:57:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.397+0000 7fae99b32640 1 --2- 172.21.15.169:0/4018653059 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f4310 0x55933c4024d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fae8409b320 tx=0x7fae84086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f5ee2193f4bef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.397+0000 7fae7ffff640 1 -- 172.21.15.169:0/4018653059 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae840b4070 con 0x55933c3f4310 2024-03-28T00:57:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.397+0000 7fae7ffff640 1 -- 172.21.15.169:0/4018653059 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae8409b620 con 0x55933c3f4310 2024-03-28T00:57:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.397+0000 7fae7ffff640 1 -- 172.21.15.169:0/4018653059 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae84084bb0 con 0x55933c3f4310 2024-03-28T00:57:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.398+0000 7fae9f186740 1 -- 172.21.15.169:0/4018653059 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f4310 msgr2=0x55933c4024d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.398+0000 7fae9f186740 1 --2- 172.21.15.169:0/4018653059 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f4310 0x55933c4024d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fae8409b320 tx=0x7fae84086040 comp rx=0 tx=0).stop 2024-03-28T00:57:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.398+0000 7fae9f186740 1 -- 172.21.15.169:0/4018653059 shutdown_connections 2024-03-28T00:57:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.398+0000 7fae9f186740 1 --2- 172.21.15.169:0/4018653059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55933c3f9480 0x55933c4094d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.398+0000 7fae9f186740 1 --2- 172.21.15.169:0/4018653059 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f4310 0x55933c4024d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.398+0000 7fae9f186740 1 --2- 172.21.15.169:0/4018653059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f3920 0x55933c401f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.398+0000 7fae9f186740 1 -- 172.21.15.169:0/4018653059 >> 172.21.15.169:0/4018653059 conn(0x55933c3eee90 msgr2=0x55933c3ef270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.398+0000 7fae9f186740 1 -- 172.21.15.169:0/4018653059 shutdown_connections 2024-03-28T00:57:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.398+0000 7fae9f186740 1 -- 172.21.15.169:0/4018653059 wait complete. 2024-03-28T00:57:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.398+0000 7fae9f186740 1 -- start start 2024-03-28T00:57:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.398+0000 7fae9f186740 1 Processor -- start 2024-03-28T00:57:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae9f186740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f3920 0x55933c4066a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae9f186740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f4310 0x55933c406be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae9f186740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55933c3f9480 0x55933c407b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae9f186740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae941a6650 con 0x55933c3f4310 2024-03-28T00:57:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae9f186740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae941a7950 con 0x55933c3f3920 2024-03-28T00:57:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae9f186740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae941a6c90 con 0x55933c3f9480 2024-03-28T00:57:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae9f186740 10 client.0 _ll_register_callbacks cb 0x7fae8c005bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae99b32640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f4310 0x55933c406be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae9a333640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f3920 0x55933c4066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae9ab34640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55933c3f9480 0x55933c407b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae99b32640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f4310 0x55933c406be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53048/0 (socket says 172.21.15.169:53048) 2024-03-28T00:57:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae99b32640 1 -- 172.21.15.169:0/2365820557 learned_addr learned my addr 172.21.15.169:0/2365820557 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae9ab34640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55933c3f9480 0x55933c407b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:37458/0 (socket says 172.21.15.169:37458) 2024-03-28T00:57:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae9a333640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f3920 0x55933c4066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:38792/0 (socket says 172.21.15.169:38792) 2024-03-28T00:57:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae99b32640 1 -- 172.21.15.169:0/2365820557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55933c3f9480 msgr2=0x55933c407b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae99b32640 1 --2- 172.21.15.169:0/2365820557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55933c3f9480 0x55933c407b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae99b32640 1 -- 172.21.15.169:0/2365820557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f3920 msgr2=0x55933c4066a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae99b32640 1 --2- 172.21.15.169:0/2365820557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f3920 0x55933c4066a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae99b32640 1 -- 172.21.15.169:0/2365820557 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55933c408170 con 0x55933c3f4310 2024-03-28T00:57:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae9a333640 1 --2- 172.21.15.169:0/2365820557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f3920 0x55933c4066a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae99b32640 1 --2- 172.21.15.169:0/2365820557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f4310 0x55933c406be0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fae8409b5b0 tx=0x7fae8400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=917d2bc79b981ef2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:57:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae7dffb640 1 -- 172.21.15.169:0/2365820557 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae840b4040 con 0x55933c3f4310 2024-03-28T00:57:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae7dffb640 1 -- 172.21.15.169:0/2365820557 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fae84084db0 con 0x55933c3f4310 2024-03-28T00:57:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae9f186740 1 -- 172.21.15.169:0/2365820557 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fae8409b620 con 0x55933c3f4310 2024-03-28T00:57:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae7dffb640 1 -- 172.21.15.169:0/2365820557 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae8409f770 con 0x55933c3f4310 2024-03-28T00:57:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae9f186740 10 client.15639 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.399+0000 7fae9f186740 1 -- 172.21.15.169:0/2365820557 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55933c409480 con 0x55933c3f4310 2024-03-28T00:57:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.400+0000 7fae7dffb640 1 -- 172.21.15.169:0/2365820557 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae8409b620 con 0x55933c3f4310 2024-03-28T00:57:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.400+0000 7fae7dffb640 1 -- 172.21.15.169:0/2365820557 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fae8409a4e0 con 0x55933c3f4310 2024-03-28T00:57:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.400+0000 7fae9f186740 10 client.15639 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.400+0000 7fae9f186740 10 client.15639 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.400+0000 7fae9f186740 20 client.15639 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.400+0000 7fae9f186740 1 --2- 172.21.15.169:0/2365820557 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55933c469c60 0x55933c46c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.400+0000 7fae9f186740 4 client.15639 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T00:57:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.400+0000 7fae9f186740 1 -- 172.21.15.169:0/2365820557 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55933c3603a0 con 0x55933c469c60 2024-03-28T00:57:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.400+0000 7fae9a333640 1 --2- 172.21.15.169:0/2365820557 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55933c469c60 0x55933c46c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.400+0000 7fae7dffb640 10 client.15639 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.400+0000 7fae9a333640 1 --2- 172.21.15.169:0/2365820557 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55933c469c60 0x55933c46c080 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2516a752502bfd0b server_cookie=9ca1c2df8d7d60f6 in_seq=0 out_seq=0 2024-03-28T00:57:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.411+0000 7fae7dffb640 1 -- 172.21.15.169:0/2365820557 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55933c3603a0 con 0x55933c469c60 2024-03-28T00:57:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.411+0000 7fae7dffb640 10 client.15639 handle_command_reply: tid=0 2024-03-28T00:57:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.411+0000 7fae9f186740 1 client.15639 shutdown 2024-03-28T00:57:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae9f186740 1 -- 172.21.15.169:0/2365820557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f4310 msgr2=0x55933c406be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae9f186740 1 --2- 172.21.15.169:0/2365820557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f4310 0x55933c406be0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fae8409b5b0 tx=0x7fae8400ef90 comp rx=0 tx=0).stop 2024-03-28T00:57:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae9f186740 1 -- 172.21.15.169:0/2365820557 shutdown_connections 2024-03-28T00:57:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae9f186740 1 --2- 172.21.15.169:0/2365820557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55933c3f9480 0x55933c407b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae9f186740 1 --2- 172.21.15.169:0/2365820557 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55933c469c60 0x55933c46c080 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae9f186740 1 --2- 172.21.15.169:0/2365820557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55933c3f4310 0x55933c406be0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae9f186740 1 --2- 172.21.15.169:0/2365820557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55933c3f3920 0x55933c4066a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae7dffb640 0 client.15639 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae9f186740 1 -- 172.21.15.169:0/2365820557 >> 172.21.15.169:0/2365820557 conn(0x55933c3eee90 msgr2=0x55933c401f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae9f186740 1 -- 172.21.15.169:0/2365820557 shutdown_connections 2024-03-28T00:57:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae9f186740 1 -- 172.21.15.169:0/2365820557 wait complete. 2024-03-28T00:57:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae9f186740 20 client.15639 trim_cache size 0 max 16384 2024-03-28T00:57:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T00:57:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T00:57:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T00:57:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T00:57:05.417 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae9c597640 1 -- 172.21.15.169:0/2048108999 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fae800a29c0 msgr2=0x7fae800a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:05.417 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae9c597640 1 --2- 172.21.15.169:0/2048108999 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fae800a29c0 0x7fae800a4e80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fae8c005fd0 tx=0x7fae8c009e00 comp rx=0 tx=0).stop 2024-03-28T00:57:05.417 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae9c597640 1 -- 172.21.15.169:0/2048108999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae94104ac0 msgr2=0x7fae94197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:05.417 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae9c597640 1 --2- 172.21.15.169:0/2048108999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae94104ac0 0x7fae94197f30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fae8400e9a0 tx=0x7fae8400ee60 comp rx=0 tx=0).stop 2024-03-28T00:57:05.418 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae9c597640 1 -- 172.21.15.169:0/2048108999 shutdown_connections 2024-03-28T00:57:05.418 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae9c597640 1 --2- 172.21.15.169:0/2048108999 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fae800a29c0 0x7fae800a4e80 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.418 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae9c597640 1 --2- 172.21.15.169:0/2048108999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae941073f0 0x7fae9419eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.418 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae9c597640 1 --2- 172.21.15.169:0/2048108999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae94104ac0 0x7fae94197f30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.418 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae9c597640 1 --2- 172.21.15.169:0/2048108999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fae94102190 0x7fae941979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:05.418 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.412+0000 7fae9c597640 1 -- 172.21.15.169:0/2048108999 >> 172.21.15.169:0/2048108999 conn(0x7fae940fbb80 msgr2=0x7fae94102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:05.418 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.413+0000 7fae9c597640 1 -- 172.21.15.169:0/2048108999 shutdown_connections 2024-03-28T00:57:05.418 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:05.413+0000 7fae9c597640 1 -- 172.21.15.169:0/2048108999 wait complete. 2024-03-28T00:57:07.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:07 smithi169 ceph-mon[32274]: pgmap v369: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 938 B/s wr, 9 op/s 2024-03-28T00:57:07.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:07 smithi174 ceph-mon[36060]: pgmap v369: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 938 B/s wr, 9 op/s 2024-03-28T00:57:07.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:07 smithi177 ceph-mon[36206]: pgmap v369: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 938 B/s wr, 9 op/s 2024-03-28T00:57:09.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:09 smithi169 ceph-mon[32274]: pgmap v370: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-03-28T00:57:09.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:09 smithi174 ceph-mon[36060]: pgmap v370: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-03-28T00:57:09.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:09 smithi177 ceph-mon[36206]: pgmap v370: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-03-28T00:57:11.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:11 smithi169 ceph-mon[32274]: pgmap v371: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-28T00:57:11.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:11 smithi174 ceph-mon[36060]: pgmap v371: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-28T00:57:11.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:11 smithi177 ceph-mon[36206]: pgmap v371: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-28T00:57:13.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:13 smithi169 ceph-mon[32274]: pgmap v372: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-28T00:57:13.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:13 smithi174 ceph-mon[36060]: pgmap v372: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-28T00:57:13.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:13 smithi177 ceph-mon[36206]: pgmap v372: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-28T00:57:15.468 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:15 smithi177 ceph-mon[36206]: pgmap v373: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-28T00:57:15.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:15 smithi169 ceph-mon[32274]: pgmap v373: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-28T00:57:15.609 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:15 smithi174 ceph-mon[36060]: pgmap v373: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-28T00:57:17.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:17 smithi169 ceph-mon[32274]: pgmap v374: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-28T00:57:17.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:17 smithi174 ceph-mon[36060]: pgmap v374: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-28T00:57:17.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:17 smithi177 ceph-mon[36206]: pgmap v374: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-28T00:57:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.505+0000 7f34cf212640 1 -- 172.21.15.169:0/3457968072 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34c8107400 msgr2=0x7f34c8109810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.505+0000 7f34cf212640 1 --2- 172.21.15.169:0/3457968072 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34c8107400 0x7f34c8109810 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f34c0009ca0 tx=0x7f34c002c0f0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.505+0000 7f34cf212640 1 -- 172.21.15.169:0/3457968072 shutdown_connections 2024-03-28T00:57:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.505+0000 7f34cf212640 1 --2- 172.21.15.169:0/3457968072 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34c8107400 0x7f34c8109810 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.505+0000 7f34cf212640 1 --2- 172.21.15.169:0/3457968072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34c8104ad0 0x7f34c8106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.505+0000 7f34cf212640 1 --2- 172.21.15.169:0/3457968072 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34c81021a0 0x7f34c8104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.505+0000 7f34cf212640 1 -- 172.21.15.169:0/3457968072 >> 172.21.15.169:0/3457968072 conn(0x7f34c80fbb50 msgr2=0x7f34c80fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.506+0000 7f34cf212640 1 -- 172.21.15.169:0/3457968072 shutdown_connections 2024-03-28T00:57:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.506+0000 7f34cf212640 1 -- 172.21.15.169:0/3457968072 wait complete. 2024-03-28T00:57:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.506+0000 7f34cf212640 1 Processor -- start 2024-03-28T00:57:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.506+0000 7f34cf212640 1 -- start start 2024-03-28T00:57:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.506+0000 7f34cf212640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34c81021a0 0x7f34c8197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.506+0000 7f34cf212640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34c8104ad0 0x7f34c8197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.506+0000 7f34cf212640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34c8107400 0x7f34c819ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.506+0000 7f34cf212640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c810d360 con 0x7f34c81021a0 2024-03-28T00:57:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.506+0000 7f34cf212640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c810d1e0 con 0x7f34c8107400 2024-03-28T00:57:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.506+0000 7f34cf212640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c810d4e0 con 0x7f34c8104ad0 2024-03-28T00:57:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.506+0000 7f34ccfae640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34c81021a0 0x7f34c8197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.506+0000 7f34ccfae640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34c81021a0 0x7f34c8197920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60412/0 (socket says 172.21.15.169:60412) 2024-03-28T00:57:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.506+0000 7f34ccfae640 1 -- 172.21.15.169:0/1872690173 learned_addr learned my addr 172.21.15.169:0/1872690173 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.506+0000 7f34c7fff640 1 --2- 172.21.15.169:0/1872690173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34c8104ad0 0x7f34c8197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.506+0000 7f34cd7af640 1 --2- 172.21.15.169:0/1872690173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34c8107400 0x7f34c819ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.507+0000 7f34ccfae640 1 -- 172.21.15.169:0/1872690173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34c8104ad0 msgr2=0x7f34c8197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.507+0000 7f34ccfae640 1 --2- 172.21.15.169:0/1872690173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34c8104ad0 0x7f34c8197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.507+0000 7f34ccfae640 1 -- 172.21.15.169:0/1872690173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34c8107400 msgr2=0x7f34c819ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.507+0000 7f34ccfae640 1 --2- 172.21.15.169:0/1872690173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34c8107400 0x7f34c819ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.507+0000 7f34ccfae640 1 -- 172.21.15.169:0/1872690173 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c819f4f0 con 0x7f34c81021a0 2024-03-28T00:57:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.507+0000 7f34ccfae640 1 --2- 172.21.15.169:0/1872690173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34c81021a0 0x7f34c8197920 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00ea60 tx=0x7f34bc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.507+0000 7f34c5ffb640 1 -- 172.21.15.169:0/1872690173 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc00cde0 con 0x7f34c81021a0 2024-03-28T00:57:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.507+0000 7f34c5ffb640 1 -- 172.21.15.169:0/1872690173 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34bc0035f0 con 0x7f34c81021a0 2024-03-28T00:57:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.507+0000 7f34c5ffb640 1 -- 172.21.15.169:0/1872690173 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc010670 con 0x7f34c81021a0 2024-03-28T00:57:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.507+0000 7f34cf212640 1 -- 172.21.15.169:0/1872690173 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34c819f7e0 con 0x7f34c81021a0 2024-03-28T00:57:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.507+0000 7f34cf212640 1 -- 172.21.15.169:0/1872690173 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c819fd30 con 0x7f34c81021a0 2024-03-28T00:57:18.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.509+0000 7f34c5ffb640 1 -- 172.21.15.169:0/1872690173 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f34bc0037a0 con 0x7f34c81021a0 2024-03-28T00:57:18.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.510+0000 7f34d1dff740 1 Processor -- start 2024-03-28T00:57:18.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.510+0000 7f34d1dff740 1 -- start start 2024-03-28T00:57:18.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.510+0000 7f34c5ffb640 1 --2- 172.21.15.169:0/1872690173 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f34b80a2940 0x7f34b80a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:18.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.510+0000 7f34d1dff740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d3920 0x5635891d3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:18.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.510+0000 7f34d1dff740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d4310 0x5635891d8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:18.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.510+0000 7f34d1dff740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635891d9480 0x5635891db910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:18.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.510+0000 7f34d1dff740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x563589138b10 con 0x5635891d3920 2024-03-28T00:57:18.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.510+0000 7f34d1dff740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635890e1900 con 0x5635891d9480 2024-03-28T00:57:18.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.510+0000 7f34d1dff740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x563589134bb0 con 0x5635891d4310 2024-03-28T00:57:18.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.510+0000 7f34ccfae640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d3920 0x5635891d3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.510+0000 7f34cd7af640 1 --2- 172.21.15.169:0/1872690173 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f34b80a2940 0x7f34b80a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.510+0000 7f34ccfae640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d3920 0x5635891d3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60416/0 (socket says 172.21.15.169:60416) 2024-03-28T00:57:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.510+0000 7f34c7fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d4310 0x5635891d8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.510+0000 7f34ccfae640 1 -- 172.21.15.169:0/2280346205 learned_addr learned my addr 172.21.15.169:0/2280346205 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.510+0000 7f34cd7af640 1 --2- 172.21.15.169:0/2280346205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635891d9480 0x5635891db910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.510+0000 7f34c5ffb640 1 -- 172.21.15.169:0/1872690173 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f34bc0cf280 con 0x7f34c81021a0 2024-03-28T00:57:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.511+0000 7f34c7fff640 1 -- 172.21.15.169:0/2280346205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635891d9480 msgr2=0x5635891db910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.511+0000 7f34c7fff640 1 --2- 172.21.15.169:0/2280346205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635891d9480 0x5635891db910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.511+0000 7f34c7fff640 1 -- 172.21.15.169:0/2280346205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d3920 msgr2=0x5635891d3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.511+0000 7f34c7fff640 1 --2- 172.21.15.169:0/2280346205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d3920 0x5635891d3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.511+0000 7f34c7fff640 1 -- 172.21.15.169:0/2280346205 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563588e4d210 con 0x5635891d4310 2024-03-28T00:57:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.511+0000 7f34cd7af640 1 --2- 172.21.15.169:0/1872690173 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f34b80a2940 0x7f34b80a4e00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f34c0005e30 tx=0x7f34c002e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.511+0000 7f34c7fff640 1 --2- 172.21.15.169:0/2280346205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d4310 0x5635891d8e10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f34c806a9c0 tx=0x7f34b4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee7692051b028b2f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.511+0000 7f34b2ffd640 1 -- 172.21.15.169:0/2280346205 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b4004050 con 0x5635891d4310 2024-03-28T00:57:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.511+0000 7f34b2ffd640 1 -- 172.21.15.169:0/2280346205 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34b4005500 con 0x5635891d4310 2024-03-28T00:57:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.511+0000 7f34b2ffd640 1 -- 172.21.15.169:0/2280346205 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b4004d90 con 0x5635891d4310 2024-03-28T00:57:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.512+0000 7f34d1dff740 1 -- 172.21.15.169:0/2280346205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d4310 msgr2=0x5635891d8e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.512+0000 7f34d1dff740 1 --2- 172.21.15.169:0/2280346205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d4310 0x5635891d8e10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f34c806a9c0 tx=0x7f34b4011810 comp rx=0 tx=0).stop 2024-03-28T00:57:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.512+0000 7f34d1dff740 1 -- 172.21.15.169:0/2280346205 shutdown_connections 2024-03-28T00:57:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.512+0000 7f34d1dff740 1 --2- 172.21.15.169:0/2280346205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635891d9480 0x5635891db910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.512+0000 7f34d1dff740 1 --2- 172.21.15.169:0/2280346205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d4310 0x5635891d8e10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.512+0000 7f34d1dff740 1 --2- 172.21.15.169:0/2280346205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d3920 0x5635891d3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.512+0000 7f34d1dff740 1 -- 172.21.15.169:0/2280346205 >> 172.21.15.169:0/2280346205 conn(0x5635891cee90 msgr2=0x5635891cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.512+0000 7f34d1dff740 1 -- 172.21.15.169:0/2280346205 shutdown_connections 2024-03-28T00:57:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.512+0000 7f34d1dff740 1 -- 172.21.15.169:0/2280346205 wait complete. 2024-03-28T00:57:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.512+0000 7f34d1dff740 1 -- start start 2024-03-28T00:57:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.512+0000 7f34d1dff740 1 Processor -- start 2024-03-28T00:57:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34d1dff740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635891d3920 0x5635891ddbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34d1dff740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d4310 0x5635891de130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34d1dff740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d9480 0x5635891e5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34d1dff740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635891dc430 con 0x5635891d9480 2024-03-28T00:57:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34d1dff740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635890f4d50 con 0x5635891d3920 2024-03-28T00:57:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34d1dff740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635891dc5b0 con 0x5635891d4310 2024-03-28T00:57:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34d1dff740 10 client.0 _ll_register_callbacks cb 0x5635890f4720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34cd7af640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d9480 0x5635891e5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34c7fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d4310 0x5635891de130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34cd7af640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d9480 0x5635891e5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60430/0 (socket says 172.21.15.169:60430) 2024-03-28T00:57:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34cd7af640 1 -- 172.21.15.169:0/1360619333 learned_addr learned my addr 172.21.15.169:0/1360619333 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34ccfae640 1 --2- 172.21.15.169:0/1360619333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635891d3920 0x5635891ddbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34c7fff640 1 -- 172.21.15.169:0/1360619333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635891d3920 msgr2=0x5635891ddbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34c7fff640 1 --2- 172.21.15.169:0/1360619333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635891d3920 0x5635891ddbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34c7fff640 1 -- 172.21.15.169:0/1360619333 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d9480 msgr2=0x5635891e5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34c7fff640 1 --2- 172.21.15.169:0/1360619333 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d9480 0x5635891e5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34c7fff640 1 -- 172.21.15.169:0/1360619333 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563588f19620 con 0x5635891d4310 2024-03-28T00:57:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34ccfae640 1 --2- 172.21.15.169:0/1360619333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635891d3920 0x5635891ddbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34c7fff640 1 --2- 172.21.15.169:0/1360619333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d4310 0x5635891de130 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x5635891cfe00 tx=0x7f34b40056b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e58b8a0741fb389d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:57:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34a7fff640 1 -- 172.21.15.169:0/1360619333 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b4029070 con 0x5635891d4310 2024-03-28T00:57:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34a7fff640 1 -- 172.21.15.169:0/1360619333 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f34b4004f20 con 0x5635891d4310 2024-03-28T00:57:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.513+0000 7f34d1dff740 1 -- 172.21.15.169:0/1360619333 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f34b4005500 con 0x5635891d4310 2024-03-28T00:57:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.514+0000 7f34a7fff640 1 -- 172.21.15.169:0/1360619333 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f34b4035050 con 0x5635891d4310 2024-03-28T00:57:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.514+0000 7f34d1dff740 10 client.25447 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.514+0000 7f34d1dff740 1 -- 172.21.15.169:0/1360619333 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5635891e84e0 con 0x5635891d4310 2024-03-28T00:57:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.514+0000 7f34a7fff640 1 -- 172.21.15.169:0/1360619333 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34b4005500 con 0x5635891d4310 2024-03-28T00:57:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.514+0000 7f34a7fff640 1 -- 172.21.15.169:0/1360619333 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f34b4023430 con 0x5635891d4310 2024-03-28T00:57:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.514+0000 7f34d1dff740 10 client.25447 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.514+0000 7f34d1dff740 10 client.25447 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.514+0000 7f34d1dff740 20 client.25447 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.514+0000 7f34d1dff740 1 --2- 172.21.15.169:0/1360619333 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5635891e9940 0x5635891ebd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.514+0000 7f34d1dff740 4 client.25447 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:57:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.514+0000 7f34d1dff740 1 -- 172.21.15.169:0/1360619333 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563589146050 con 0x5635891e9940 2024-03-28T00:57:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.514+0000 7f34ccfae640 1 --2- 172.21.15.169:0/1360619333 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5635891e9940 0x5635891ebd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.514+0000 7f34a7fff640 10 client.25447 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.514+0000 7f34ccfae640 1 --2- 172.21.15.169:0/1360619333 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5635891e9940 0x5635891ebd30 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f297ea7fdced13fa server_cookie=4b09a1c81e936f61 in_seq=0 out_seq=0 2024-03-28T00:57:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.515+0000 7f34a7fff640 1 -- 172.21.15.169:0/1360619333 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563589146050 con 0x5635891e9940 2024-03-28T00:57:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.515+0000 7f34a7fff640 10 client.25447 handle_command_reply: tid=0 2024-03-28T00:57:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.515+0000 7f34d1dff740 1 client.25447 shutdown 2024-03-28T00:57:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.515+0000 7f34d1dff740 1 -- 172.21.15.169:0/1360619333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d4310 msgr2=0x5635891de130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.515+0000 7f34d1dff740 1 --2- 172.21.15.169:0/1360619333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d4310 0x5635891de130 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x5635891cfe00 tx=0x7f34b40056b0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.515+0000 7f34d1dff740 1 -- 172.21.15.169:0/1360619333 shutdown_connections 2024-03-28T00:57:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.515+0000 7f34d1dff740 1 --2- 172.21.15.169:0/1360619333 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5635891e9940 0x5635891ebd30 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.515+0000 7f34d1dff740 1 --2- 172.21.15.169:0/1360619333 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d9480 0x5635891e5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.515+0000 7f34a7fff640 0 client.25447 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.515+0000 7f34d1dff740 1 --2- 172.21.15.169:0/1360619333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d4310 0x5635891de130 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.515+0000 7f34d1dff740 1 --2- 172.21.15.169:0/1360619333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635891d3920 0x5635891ddbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.515+0000 7f34d1dff740 1 -- 172.21.15.169:0/1360619333 >> 172.21.15.169:0/1360619333 conn(0x5635891cee90 msgr2=0x5635891ce020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.515+0000 7f34d1dff740 1 -- 172.21.15.169:0/1360619333 shutdown_connections 2024-03-28T00:57:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.515+0000 7f34d1dff740 1 -- 172.21.15.169:0/1360619333 wait complete. 2024-03-28T00:57:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.515+0000 7f34d1dff740 20 client.25447 trim_cache size 0 max 16384 2024-03-28T00:57:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.529+0000 7f34d1dff740 1 Processor -- start 2024-03-28T00:57:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.529+0000 7f34d1dff740 1 -- start start 2024-03-28T00:57:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.529+0000 7f34d1dff740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d3920 0x5635891db870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.529+0000 7f34d1dff740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d4310 0x5635891e5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.529+0000 7f34d1dff740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635891d9480 0x5635891cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.529+0000 7f34d1dff740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c81ac690 con 0x5635891d3920 2024-03-28T00:57:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.529+0000 7f34d1dff740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c81aa190 con 0x5635891d9480 2024-03-28T00:57:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.529+0000 7f34d1dff740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c81a8230 con 0x5635891d4310 2024-03-28T00:57:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.529+0000 7f34ccfae640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d3920 0x5635891db870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.529+0000 7f34ccfae640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d3920 0x5635891db870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60432/0 (socket says 172.21.15.169:60432) 2024-03-28T00:57:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.530+0000 7f34ccfae640 1 -- 172.21.15.169:0/1755577163 learned_addr learned my addr 172.21.15.169:0/1755577163 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.530+0000 7f34cd7af640 1 --2- 172.21.15.169:0/1755577163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635891d9480 0x5635891cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.530+0000 7f34c7fff640 1 --2- 172.21.15.169:0/1755577163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d4310 0x5635891e5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.530+0000 7f34ccfae640 1 -- 172.21.15.169:0/1755577163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d4310 msgr2=0x5635891e5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.530+0000 7f34ccfae640 1 --2- 172.21.15.169:0/1755577163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d4310 0x5635891e5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.530+0000 7f34ccfae640 1 -- 172.21.15.169:0/1755577163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635891d9480 msgr2=0x5635891cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.530+0000 7f34ccfae640 1 --2- 172.21.15.169:0/1755577163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635891d9480 0x5635891cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.530+0000 7f34ccfae640 1 -- 172.21.15.169:0/1755577163 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635891cfc00 con 0x5635891d3920 2024-03-28T00:57:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.530+0000 7f34ccfae640 1 --2- 172.21.15.169:0/1755577163 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d3920 0x5635891db870 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00d010 tx=0x7f34bc096060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2a420592001a928 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.530+0000 7f34b27fc640 1 -- 172.21.15.169:0/1755577163 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc09b720 con 0x5635891d3920 2024-03-28T00:57:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.530+0000 7f34b27fc640 1 -- 172.21.15.169:0/1755577163 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34bc09f870 con 0x5635891d3920 2024-03-28T00:57:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.530+0000 7f34b27fc640 1 -- 172.21.15.169:0/1755577163 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc003b30 con 0x5635891d3920 2024-03-28T00:57:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.531+0000 7f34d1dff740 1 -- 172.21.15.169:0/1755577163 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d3920 msgr2=0x5635891db870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.531+0000 7f34d1dff740 1 --2- 172.21.15.169:0/1755577163 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d3920 0x5635891db870 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00d010 tx=0x7f34bc096060 comp rx=0 tx=0).stop 2024-03-28T00:57:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.531+0000 7f34d1dff740 1 -- 172.21.15.169:0/1755577163 shutdown_connections 2024-03-28T00:57:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.531+0000 7f34d1dff740 1 --2- 172.21.15.169:0/1755577163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635891d9480 0x5635891cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.531+0000 7f34d1dff740 1 --2- 172.21.15.169:0/1755577163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d4310 0x5635891e5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.531+0000 7f34d1dff740 1 --2- 172.21.15.169:0/1755577163 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d3920 0x5635891db870 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.531+0000 7f34d1dff740 1 -- 172.21.15.169:0/1755577163 >> 172.21.15.169:0/1755577163 conn(0x5635891cee90 msgr2=0x5635891c9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.531+0000 7f34d1dff740 1 -- 172.21.15.169:0/1755577163 shutdown_connections 2024-03-28T00:57:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.531+0000 7f34d1dff740 1 -- 172.21.15.169:0/1755577163 wait complete. 2024-03-28T00:57:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.531+0000 7f34d1dff740 1 -- start start 2024-03-28T00:57:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.531+0000 7f34d1dff740 1 Processor -- start 2024-03-28T00:57:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.531+0000 7f34d1dff740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635891d3920 0x5635891e2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.531+0000 7f34d1dff740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d4310 0x5635891e2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.531+0000 7f34d1dff740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d9480 0x5635891e36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.531+0000 7f34d1dff740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c81a8830 con 0x5635891d4310 2024-03-28T00:57:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.531+0000 7f34d1dff740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c81a9b30 con 0x5635891d3920 2024-03-28T00:57:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.531+0000 7f34d1dff740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c81a8e70 con 0x5635891d9480 2024-03-28T00:57:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.532+0000 7f34d1dff740 10 client.0 _ll_register_callbacks cb 0x7f34b4009a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.532+0000 7f34c7fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d4310 0x5635891e2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.532+0000 7f34c7fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d4310 0x5635891e2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60444/0 (socket says 172.21.15.169:60444) 2024-03-28T00:57:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.532+0000 7f34c7fff640 1 -- 172.21.15.169:0/2906029494 learned_addr learned my addr 172.21.15.169:0/2906029494 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.532+0000 7f34cd7af640 1 --2- 172.21.15.169:0/2906029494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d9480 0x5635891e36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.532+0000 7f34ccfae640 1 --2- 172.21.15.169:0/2906029494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635891d3920 0x5635891e2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.532+0000 7f34c7fff640 1 -- 172.21.15.169:0/2906029494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d9480 msgr2=0x5635891e36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.532+0000 7f34c7fff640 1 --2- 172.21.15.169:0/2906029494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d9480 0x5635891e36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.532+0000 7f34c7fff640 1 -- 172.21.15.169:0/2906029494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635891d3920 msgr2=0x5635891e2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.532+0000 7f34c7fff640 1 --2- 172.21.15.169:0/2906029494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635891d3920 0x5635891e2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.532+0000 7f34c7fff640 1 -- 172.21.15.169:0/2906029494 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5635891e3c90 con 0x5635891d4310 2024-03-28T00:57:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.532+0000 7f34c7fff640 1 --2- 172.21.15.169:0/2906029494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d4310 0x5635891e2690 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f34b4011850 tx=0x7f34b4005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e6c53e3fa3fb5d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.532+0000 7f34a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:57:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.532+0000 7f34a7fff640 1 -- 172.21.15.169:0/2906029494 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b403e060 con 0x5635891d4310 2024-03-28T00:57:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.532+0000 7f34a7fff640 1 -- 172.21.15.169:0/2906029494 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f34b4004570 con 0x5635891d4310 2024-03-28T00:57:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.532+0000 7f34d1dff740 1 -- 172.21.15.169:0/2906029494 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f34bc09f870 con 0x5635891d4310 2024-03-28T00:57:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.532+0000 7f34a7fff640 1 -- 172.21.15.169:0/2906029494 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f34b4029040 con 0x5635891d4310 2024-03-28T00:57:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.532+0000 7f34d1dff740 10 client.15657 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.532+0000 7f34d1dff740 1 -- 172.21.15.169:0/2906029494 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5635891e4fa0 con 0x5635891d4310 2024-03-28T00:57:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.532+0000 7f34a7fff640 1 -- 172.21.15.169:0/2906029494 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34bc09f870 con 0x5635891d4310 2024-03-28T00:57:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.533+0000 7f34a7fff640 1 -- 172.21.15.169:0/2906029494 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f34b4005320 con 0x5635891d4310 2024-03-28T00:57:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.533+0000 7f34d1dff740 10 client.15657 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.533+0000 7f34d1dff740 10 client.15657 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.533+0000 7f34d1dff740 20 client.15657 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.533+0000 7f34d1dff740 1 --2- 172.21.15.169:0/2906029494 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5635891e9940 0x5635892473c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:18.544 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.533+0000 7f34d1dff740 4 client.15657 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T00:57:18.544 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.533+0000 7f34d1dff740 1 -- 172.21.15.169:0/2906029494 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5635891403a0 con 0x5635891e9940 2024-03-28T00:57:18.544 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.533+0000 7f34ccfae640 1 --2- 172.21.15.169:0/2906029494 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5635891e9940 0x5635892473c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:18.544 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.533+0000 7f34a7fff640 10 client.15657 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:18.544 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.533+0000 7f34ccfae640 1 --2- 172.21.15.169:0/2906029494 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5635891e9940 0x5635892473c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6913f2c194fee476 server_cookie=452fd1c8f82dccab in_seq=0 out_seq=0 2024-03-28T00:57:18.545 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.538+0000 7f34a7fff640 1 -- 172.21.15.169:0/2906029494 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5635891403a0 con 0x5635891e9940 2024-03-28T00:57:18.545 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.538+0000 7f34a7fff640 10 client.15657 handle_command_reply: tid=0 2024-03-28T00:57:18.545 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.538+0000 7f34d1dff740 1 client.15657 shutdown 2024-03-28T00:57:18.545 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.538+0000 7f34d1dff740 1 -- 172.21.15.169:0/2906029494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d4310 msgr2=0x5635891e2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:18.545 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.538+0000 7f34d1dff740 1 --2- 172.21.15.169:0/2906029494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d4310 0x5635891e2690 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f34b4011850 tx=0x7f34b4005db0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.545 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.538+0000 7f34d1dff740 1 -- 172.21.15.169:0/2906029494 shutdown_connections 2024-03-28T00:57:18.545 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.538+0000 7f34d1dff740 1 --2- 172.21.15.169:0/2906029494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5635891d9480 0x5635891e36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.546 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.538+0000 7f34d1dff740 1 --2- 172.21.15.169:0/2906029494 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5635891e9940 0x5635892473c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.546 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.538+0000 7f34d1dff740 1 --2- 172.21.15.169:0/2906029494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5635891d4310 0x5635891e2690 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.546 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.538+0000 7f34d1dff740 1 --2- 172.21.15.169:0/2906029494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635891d3920 0x5635891e2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.546 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.538+0000 7f34a7fff640 0 client.15657 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:18.546 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.538+0000 7f34d1dff740 1 -- 172.21.15.169:0/2906029494 >> 172.21.15.169:0/2906029494 conn(0x5635891cee90 msgr2=0x5635891cf270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:18.546 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.538+0000 7f34d1dff740 1 -- 172.21.15.169:0/2906029494 shutdown_connections 2024-03-28T00:57:18.546 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.538+0000 7f34d1dff740 1 -- 172.21.15.169:0/2906029494 wait complete. 2024-03-28T00:57:18.546 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.539+0000 7f34d1dff740 20 client.15657 trim_cache size 0 max 16384 2024-03-28T00:57:18.546 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T00:57:18.547 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T00:57:18.547 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T00:57:18.547 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T00:57:18.547 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.539+0000 7f34cf212640 1 -- 172.21.15.169:0/1872690173 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f34b80a2940 msgr2=0x7f34b80a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:18.547 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.539+0000 7f34cf212640 1 --2- 172.21.15.169:0/1872690173 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f34b80a2940 0x7f34b80a4e00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f34c0005e30 tx=0x7f34c002e040 comp rx=0 tx=0).stop 2024-03-28T00:57:18.548 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.539+0000 7f34cf212640 1 -- 172.21.15.169:0/1872690173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34c81021a0 msgr2=0x7f34c8197920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:18.548 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.539+0000 7f34cf212640 1 --2- 172.21.15.169:0/1872690173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34c81021a0 0x7f34c8197920 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00ea60 tx=0x7f34bc00ef20 comp rx=0 tx=0).stop 2024-03-28T00:57:18.548 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.539+0000 7f34cf212640 1 -- 172.21.15.169:0/1872690173 shutdown_connections 2024-03-28T00:57:18.548 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.539+0000 7f34cf212640 1 --2- 172.21.15.169:0/1872690173 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f34b80a2940 0x7f34b80a4e00 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.548 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.539+0000 7f34cf212640 1 --2- 172.21.15.169:0/1872690173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34c8107400 0x7f34c819ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.548 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.539+0000 7f34cf212640 1 --2- 172.21.15.169:0/1872690173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34c8104ad0 0x7f34c8197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.548 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.539+0000 7f34cf212640 1 --2- 172.21.15.169:0/1872690173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34c81021a0 0x7f34c8197920 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:18.548 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.539+0000 7f34cf212640 1 -- 172.21.15.169:0/1872690173 >> 172.21.15.169:0/1872690173 conn(0x7f34c80fbb50 msgr2=0x7f34c8102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:18.548 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.539+0000 7f34cf212640 1 -- 172.21.15.169:0/1872690173 shutdown_connections 2024-03-28T00:57:18.548 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:18.539+0000 7f34cf212640 1 -- 172.21.15.169:0/1872690173 wait complete. 2024-03-28T00:57:19.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:19 smithi169 ceph-mon[32274]: pgmap v375: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 8 op/s 2024-03-28T00:57:19.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:19 smithi174 ceph-mon[36060]: pgmap v375: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 8 op/s 2024-03-28T00:57:19.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:19 smithi177 ceph-mon[36206]: pgmap v375: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 8 op/s 2024-03-28T00:57:21.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:21 smithi169 ceph-mon[32274]: pgmap v376: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-28T00:57:21.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:21 smithi174 ceph-mon[36060]: pgmap v376: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-28T00:57:21.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:21 smithi177 ceph-mon[36206]: pgmap v376: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-28T00:57:22.581 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T00:57:22.581 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T00:57:22.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.697+0000 7f147d3d4640 1 -- 172.21.15.169:0/838166555 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1478102150 msgr2=0x7f14781089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:22.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.697+0000 7f147d3d4640 1 --2- 172.21.15.169:0/838166555 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1478102150 0x7f14781089e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f1470009ca0 tx=0x7f147002c0f0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.698+0000 7f147d3d4640 1 -- 172.21.15.169:0/838166555 shutdown_connections 2024-03-28T00:57:22.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.698+0000 7f147d3d4640 1 --2- 172.21.15.169:0/838166555 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1478102150 0x7f14781089e0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.698+0000 7f147d3d4640 1 --2- 172.21.15.169:0/838166555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1478101790 0x7f1478101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.698+0000 7f147d3d4640 1 --2- 172.21.15.169:0/838166555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14781005a0 0x7f14781009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.698+0000 7f147d3d4640 1 -- 172.21.15.169:0/838166555 >> 172.21.15.169:0/838166555 conn(0x7f14780fbb30 msgr2=0x7f14780fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:22.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.698+0000 7f147d3d4640 1 -- 172.21.15.169:0/838166555 shutdown_connections 2024-03-28T00:57:22.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.698+0000 7f147d3d4640 1 -- 172.21.15.169:0/838166555 wait complete. 2024-03-28T00:57:22.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.698+0000 7f147d3d4640 1 Processor -- start 2024-03-28T00:57:22.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.698+0000 7f147d3d4640 1 -- start start 2024-03-28T00:57:22.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.698+0000 7f147d3d4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14781005a0 0x7f14781979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f147d3d4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1478101790 0x7f1478197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f147d3d4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1478102150 0x7f147819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f147d3d4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147810c230 con 0x7f1478102150 2024-03-28T00:57:22.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f147d3d4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147810c0b0 con 0x7f14781005a0 2024-03-28T00:57:22.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f147d3d4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147810c3b0 con 0x7f1478101790 2024-03-28T00:57:22.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f14777fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1478102150 0x7f147819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f14767fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1478101790 0x7f1478197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f1476ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14781005a0 0x7f14781979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f14777fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1478102150 0x7f147819ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60464/0 (socket says 172.21.15.169:60464) 2024-03-28T00:57:22.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f14777fe640 1 -- 172.21.15.169:0/3242262668 learned_addr learned my addr 172.21.15.169:0/3242262668 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:22.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f14767fc640 1 -- 172.21.15.169:0/3242262668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14781005a0 msgr2=0x7f14781979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:22.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f14767fc640 1 --2- 172.21.15.169:0/3242262668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14781005a0 0x7f14781979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f14767fc640 1 -- 172.21.15.169:0/3242262668 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1478102150 msgr2=0x7f147819ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:22.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f14767fc640 1 --2- 172.21.15.169:0/3242262668 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1478102150 0x7f147819ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f14767fc640 1 -- 172.21.15.169:0/3242262668 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147819f5c0 con 0x7f1478101790 2024-03-28T00:57:22.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f1476ffd640 1 --2- 172.21.15.169:0/3242262668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14781005a0 0x7f14781979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:22.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f14777fe640 1 --2- 172.21.15.169:0/3242262668 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1478102150 0x7f147819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:22.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f14767fc640 1 --2- 172.21.15.169:0/3242262668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1478101790 0x7f1478197ee0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f146c00e9f0 tx=0x7f146c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:22.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f145ffff640 1 -- 172.21.15.169:0/3242262668 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c00ccc0 con 0x7f1478101790 2024-03-28T00:57:22.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f147d3d4640 1 -- 172.21.15.169:0/3242262668 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147819f8b0 con 0x7f1478101790 2024-03-28T00:57:22.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f145ffff640 1 -- 172.21.15.169:0/3242262668 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f146c005020 con 0x7f1478101790 2024-03-28T00:57:22.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f145ffff640 1 -- 172.21.15.169:0/3242262668 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c0049e0 con 0x7f1478101790 2024-03-28T00:57:22.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.699+0000 7f147d3d4640 1 -- 172.21.15.169:0/3242262668 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147819fe00 con 0x7f1478101790 2024-03-28T00:57:22.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f145ffff640 1 -- 172.21.15.169:0/3242262668 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f146c005560 con 0x7f1478101790 2024-03-28T00:57:22.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f147ffc1740 1 Processor -- start 2024-03-28T00:57:22.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f147ffc1740 1 -- start start 2024-03-28T00:57:22.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f147ffc1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5561b9c1ada0 0x5561b9c1b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f147ffc1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5561b9c1b790 0x5561b9c20290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f147ffc1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c20900 0x5561b9c22d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f147ffc1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561b9b803d0 con 0x5561b9c20900 2024-03-28T00:57:22.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f147ffc1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561b9b28ad0 con 0x5561b9c1b790 2024-03-28T00:57:22.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f147ffc1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561b9b7c470 con 0x5561b9c1ada0 2024-03-28T00:57:22.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f14767fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c20900 0x5561b9c22d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f145ffff640 1 --2- 172.21.15.169:0/3242262668 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f14600a2890 0x7f14600a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f14777fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5561b9c1ada0 0x5561b9c1b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f14767fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c20900 0x5561b9c22d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60480/0 (socket says 172.21.15.169:60480) 2024-03-28T00:57:22.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f14767fc640 1 -- 172.21.15.169:0/1074516071 learned_addr learned my addr 172.21.15.169:0/1074516071 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:22.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f14777fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5561b9c1ada0 0x5561b9c1b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:34322/0 (socket says 172.21.15.169:34322) 2024-03-28T00:57:22.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f1476ffd640 1 --2- 172.21.15.169:0/1074516071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5561b9c1b790 0x5561b9c20290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f14777fe640 1 --2- 172.21.15.169:0/3242262668 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f14600a2890 0x7f14600a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f145ffff640 1 -- 172.21.15.169:0/3242262668 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f146c0cef40 con 0x7f1478101790 2024-03-28T00:57:22.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f1476ffd640 1 -- 172.21.15.169:0/1074516071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5561b9c1ada0 msgr2=0x5561b9c1b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:22.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f1476ffd640 1 --2- 172.21.15.169:0/1074516071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5561b9c1ada0 0x5561b9c1b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f1476ffd640 1 -- 172.21.15.169:0/1074516071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c20900 msgr2=0x5561b9c22d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:22.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f1476ffd640 1 --2- 172.21.15.169:0/1074516071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c20900 0x5561b9c22d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f1476ffd640 1 -- 172.21.15.169:0/1074516071 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561b995e7d0 con 0x5561b9c1b790 2024-03-28T00:57:22.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.702+0000 7f14767fc640 1 --2- 172.21.15.169:0/1074516071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c20900 0x5561b9c22d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:22.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.703+0000 7f1476ffd640 1 --2- 172.21.15.169:0/1074516071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5561b9c1b790 0x5561b9c20290 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1468009fa0 tx=0x7f14680118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f40b2e4043ea2c9c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:22.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.703+0000 7f145d7fa640 1 -- 172.21.15.169:0/1074516071 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468004310 con 0x5561b9c1b790 2024-03-28T00:57:22.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.703+0000 7f14777fe640 1 --2- 172.21.15.169:0/3242262668 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f14600a2890 0x7f14600a4d50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x5561b9c1bff0 tx=0x7f147002c0f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:22.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.703+0000 7f145d7fa640 1 -- 172.21.15.169:0/1074516071 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14680044b0 con 0x5561b9c1b790 2024-03-28T00:57:22.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.703+0000 7f145d7fa640 1 -- 172.21.15.169:0/1074516071 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14680035f0 con 0x5561b9c1b790 2024-03-28T00:57:22.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.703+0000 7f147ffc1740 1 -- 172.21.15.169:0/1074516071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5561b9c1b790 msgr2=0x5561b9c20290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:22.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.703+0000 7f147ffc1740 1 --2- 172.21.15.169:0/1074516071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5561b9c1b790 0x5561b9c20290 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1468009fa0 tx=0x7f14680118d0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.703+0000 7f147ffc1740 1 -- 172.21.15.169:0/1074516071 shutdown_connections 2024-03-28T00:57:22.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.703+0000 7f147ffc1740 1 --2- 172.21.15.169:0/1074516071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c20900 0x5561b9c22d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.703+0000 7f147ffc1740 1 --2- 172.21.15.169:0/1074516071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5561b9c1b790 0x5561b9c20290 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.703+0000 7f147ffc1740 1 --2- 172.21.15.169:0/1074516071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5561b9c1ada0 0x5561b9c1b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.703+0000 7f147ffc1740 1 -- 172.21.15.169:0/1074516071 >> 172.21.15.169:0/1074516071 conn(0x5561b9c16310 msgr2=0x5561b9c16710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:22.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.704+0000 7f147ffc1740 1 -- 172.21.15.169:0/1074516071 shutdown_connections 2024-03-28T00:57:22.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.704+0000 7f147ffc1740 1 -- 172.21.15.169:0/1074516071 wait complete. 2024-03-28T00:57:22.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.704+0000 7f147ffc1740 1 -- start start 2024-03-28T00:57:22.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.704+0000 7f147ffc1740 1 Processor -- start 2024-03-28T00:57:22.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.704+0000 7f147ffc1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c1ada0 0x5561b9c25250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.704+0000 7f147ffc1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5561b9c1b790 0x5561b9c25790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.704+0000 7f147ffc1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5561b9c20900 0x5561b9c2c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.704+0000 7f147ffc1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561b9c238b0 con 0x5561b9c1ada0 2024-03-28T00:57:22.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.704+0000 7f147ffc1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561b9b3c060 con 0x5561b9c20900 2024-03-28T00:57:22.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.704+0000 7f147ffc1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561b9c23a30 con 0x5561b9c1b790 2024-03-28T00:57:22.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.704+0000 7f147ffc1740 10 client.0 _ll_register_callbacks cb 0x5561b9b1e6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:22.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.704+0000 7f1476ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c1ada0 0x5561b9c25250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.704+0000 7f1476ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c1ada0 0x5561b9c25250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60496/0 (socket says 172.21.15.169:60496) 2024-03-28T00:57:22.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.704+0000 7f1476ffd640 1 -- 172.21.15.169:0/544751297 learned_addr learned my addr 172.21.15.169:0/544751297 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:22.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.704+0000 7f14777fe640 1 --2- 172.21.15.169:0/544751297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5561b9c20900 0x5561b9c2c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.704+0000 7f14767fc640 1 --2- 172.21.15.169:0/544751297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5561b9c1b790 0x5561b9c25790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.704+0000 7f1476ffd640 1 -- 172.21.15.169:0/544751297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5561b9c1b790 msgr2=0x5561b9c25790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:22.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.704+0000 7f1476ffd640 1 --2- 172.21.15.169:0/544751297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5561b9c1b790 0x5561b9c25790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.704+0000 7f1476ffd640 1 -- 172.21.15.169:0/544751297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5561b9c20900 msgr2=0x5561b9c2c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:22.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.704+0000 7f1476ffd640 1 --2- 172.21.15.169:0/544751297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5561b9c20900 0x5561b9c2c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.704+0000 7f1476ffd640 1 -- 172.21.15.169:0/544751297 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5561b98943f0 con 0x5561b9c1ada0 2024-03-28T00:57:22.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.705+0000 7f1476ffd640 1 --2- 172.21.15.169:0/544751297 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c1ada0 0x5561b9c25250 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f146801b040 tx=0x7f1468009d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27091e99f34c51db server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:22.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.705+0000 7f145aff5640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:57:22.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.705+0000 7f145aff5640 1 -- 172.21.15.169:0/544751297 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468028070 con 0x5561b9c1ada0 2024-03-28T00:57:22.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.705+0000 7f145aff5640 1 -- 172.21.15.169:0/544751297 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f1468005000 con 0x5561b9c1ada0 2024-03-28T00:57:22.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.705+0000 7f147ffc1740 1 -- 172.21.15.169:0/544751297 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f14680044b0 con 0x5561b9c1ada0 2024-03-28T00:57:22.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.705+0000 7f145aff5640 1 -- 172.21.15.169:0/544751297 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1468034050 con 0x5561b9c1ada0 2024-03-28T00:57:22.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.705+0000 7f147ffc1740 10 client.15669 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:22.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.705+0000 7f147ffc1740 1 -- 172.21.15.169:0/544751297 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5561b98540f0 con 0x5561b9c1ada0 2024-03-28T00:57:22.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.705+0000 7f145aff5640 1 -- 172.21.15.169:0/544751297 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14680044b0 con 0x5561b9c1ada0 2024-03-28T00:57:22.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.705+0000 7f145aff5640 1 -- 172.21.15.169:0/544751297 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f146800f4e0 con 0x5561b9c1ada0 2024-03-28T00:57:22.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.705+0000 7f147ffc1740 10 client.15669 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:22.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.705+0000 7f147ffc1740 10 client.15669 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:22.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.705+0000 7f147ffc1740 20 client.15669 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:22.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.705+0000 7f147ffc1740 1 --2- 172.21.15.169:0/544751297 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5561b9c30a00 0x5561b9c32e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.705+0000 7f147ffc1740 4 client.15669 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:57:22.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.705+0000 7f147ffc1740 1 -- 172.21.15.169:0/544751297 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5561b9b8d910 con 0x5561b9c30a00 2024-03-28T00:57:22.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.706+0000 7f14767fc640 1 --2- 172.21.15.169:0/544751297 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5561b9c30a00 0x5561b9c32e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.706+0000 7f145aff5640 10 client.15669 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:22.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.706+0000 7f14767fc640 1 --2- 172.21.15.169:0/544751297 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5561b9c30a00 0x5561b9c32e10 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29428559777805fe server_cookie=52e15e34485b744a in_seq=0 out_seq=0 2024-03-28T00:57:22.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.706+0000 7f145aff5640 1 -- 172.21.15.169:0/544751297 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5561b9b8d910 con 0x5561b9c30a00 2024-03-28T00:57:22.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.706+0000 7f145aff5640 10 client.15669 handle_command_reply: tid=0 2024-03-28T00:57:22.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.707+0000 7f147ffc1740 1 client.15669 shutdown 2024-03-28T00:57:22.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.707+0000 7f147ffc1740 1 -- 172.21.15.169:0/544751297 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c1ada0 msgr2=0x5561b9c25250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:22.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.707+0000 7f147ffc1740 1 --2- 172.21.15.169:0/544751297 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c1ada0 0x5561b9c25250 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f146801b040 tx=0x7f1468009d00 comp rx=0 tx=0).stop 2024-03-28T00:57:22.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.707+0000 7f147ffc1740 1 -- 172.21.15.169:0/544751297 shutdown_connections 2024-03-28T00:57:22.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.707+0000 7f147ffc1740 1 --2- 172.21.15.169:0/544751297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5561b9c20900 0x5561b9c2c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.707+0000 7f147ffc1740 1 --2- 172.21.15.169:0/544751297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5561b9c1b790 0x5561b9c25790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.707+0000 7f147ffc1740 1 --2- 172.21.15.169:0/544751297 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5561b9c30a00 0x5561b9c32e10 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.707+0000 7f147ffc1740 1 --2- 172.21.15.169:0/544751297 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c1ada0 0x5561b9c25250 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.707+0000 7f145aff5640 0 client.15669 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:22.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.707+0000 7f147ffc1740 1 -- 172.21.15.169:0/544751297 >> 172.21.15.169:0/544751297 conn(0x5561b9c16310 msgr2=0x5561b9c15530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:22.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.707+0000 7f147ffc1740 1 -- 172.21.15.169:0/544751297 shutdown_connections 2024-03-28T00:57:22.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.707+0000 7f147ffc1740 1 -- 172.21.15.169:0/544751297 wait complete. 2024-03-28T00:57:22.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.707+0000 7f147ffc1740 20 client.15669 trim_cache size 0 max 16384 2024-03-28T00:57:22.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.723+0000 7f147ffc1740 1 Processor -- start 2024-03-28T00:57:22.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.723+0000 7f147ffc1740 1 -- start start 2024-03-28T00:57:22.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.723+0000 7f147ffc1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c1ada0 0x5561b9c14ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.723+0000 7f147ffc1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5561b9c1b790 0x5561b9c2c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.723+0000 7f147ffc1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5561b9c20900 0x5561b9c37de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.723+0000 7f147ffc1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14781ac7a0 con 0x5561b9c1ada0 2024-03-28T00:57:22.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.723+0000 7f147ffc1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14781aa2a0 con 0x5561b9c1b790 2024-03-28T00:57:22.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.723+0000 7f147ffc1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14781a8340 con 0x5561b9c20900 2024-03-28T00:57:22.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.723+0000 7f1476ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c1ada0 0x5561b9c14ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.723+0000 7f1476ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c1ada0 0x5561b9c14ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60504/0 (socket says 172.21.15.169:60504) 2024-03-28T00:57:22.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.723+0000 7f1476ffd640 1 -- 172.21.15.169:0/3238020150 learned_addr learned my addr 172.21.15.169:0/3238020150 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:22.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.723+0000 7f14767fc640 1 --2- 172.21.15.169:0/3238020150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5561b9c1b790 0x5561b9c2c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.723+0000 7f14777fe640 1 --2- 172.21.15.169:0/3238020150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5561b9c20900 0x5561b9c37de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.723+0000 7f1476ffd640 1 -- 172.21.15.169:0/3238020150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5561b9c20900 msgr2=0x5561b9c37de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:22.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.723+0000 7f1476ffd640 1 --2- 172.21.15.169:0/3238020150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5561b9c20900 0x5561b9c37de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.723+0000 7f1476ffd640 1 -- 172.21.15.169:0/3238020150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5561b9c1b790 msgr2=0x5561b9c2c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:22.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.723+0000 7f1476ffd640 1 --2- 172.21.15.169:0/3238020150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5561b9c1b790 0x5561b9c2c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.723+0000 7f1476ffd640 1 -- 172.21.15.169:0/3238020150 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561b9c15c60 con 0x5561b9c1ada0 2024-03-28T00:57:22.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.723+0000 7f1476ffd640 1 --2- 172.21.15.169:0/3238020150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c1ada0 0x5561b9c14ca0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1468009730 tx=0x7f1468005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af01200d29bcab75 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:22.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.723+0000 7f145cff9640 1 -- 172.21.15.169:0/3238020150 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468031070 con 0x5561b9c1ada0 2024-03-28T00:57:22.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.724+0000 7f145cff9640 1 -- 172.21.15.169:0/3238020150 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f146800fc20 con 0x5561b9c1ada0 2024-03-28T00:57:22.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.724+0000 7f145cff9640 1 -- 172.21.15.169:0/3238020150 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468028070 con 0x5561b9c1ada0 2024-03-28T00:57:22.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.724+0000 7f147ffc1740 1 -- 172.21.15.169:0/3238020150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c1ada0 msgr2=0x5561b9c14ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:22.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.724+0000 7f147ffc1740 1 --2- 172.21.15.169:0/3238020150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c1ada0 0x5561b9c14ca0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1468009730 tx=0x7f1468005970 comp rx=0 tx=0).stop 2024-03-28T00:57:22.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.724+0000 7f147ffc1740 1 -- 172.21.15.169:0/3238020150 shutdown_connections 2024-03-28T00:57:22.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.724+0000 7f147ffc1740 1 --2- 172.21.15.169:0/3238020150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5561b9c20900 0x5561b9c37de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.724+0000 7f147ffc1740 1 --2- 172.21.15.169:0/3238020150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5561b9c1b790 0x5561b9c2c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.724+0000 7f147ffc1740 1 --2- 172.21.15.169:0/3238020150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c1ada0 0x5561b9c14ca0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.724+0000 7f147ffc1740 1 -- 172.21.15.169:0/3238020150 >> 172.21.15.169:0/3238020150 conn(0x5561b9c16310 msgr2=0x5561b9c376d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:22.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.724+0000 7f147ffc1740 1 -- 172.21.15.169:0/3238020150 shutdown_connections 2024-03-28T00:57:22.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.724+0000 7f147ffc1740 1 -- 172.21.15.169:0/3238020150 wait complete. 2024-03-28T00:57:22.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.724+0000 7f147ffc1740 1 -- start start 2024-03-28T00:57:22.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f147ffc1740 1 Processor -- start 2024-03-28T00:57:22.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f147ffc1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c1ada0 0x5561b9c2a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f147ffc1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5561b9c1b790 0x5561b9c2aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f147ffc1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5561b9c20900 0x5561b9c2ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f147ffc1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14781a8940 con 0x5561b9c1ada0 2024-03-28T00:57:22.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f147ffc1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14781a9c40 con 0x5561b9c20900 2024-03-28T00:57:22.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f147ffc1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14781a8f80 con 0x5561b9c1b790 2024-03-28T00:57:22.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f147ffc1740 10 client.0 _ll_register_callbacks cb 0x7f146c09b1b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:22.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f1476ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c1ada0 0x5561b9c2a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f1476ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c1ada0 0x5561b9c2a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60516/0 (socket says 172.21.15.169:60516) 2024-03-28T00:57:22.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f1476ffd640 1 -- 172.21.15.169:0/3537970625 learned_addr learned my addr 172.21.15.169:0/3537970625 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:22.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f14767fc640 1 --2- 172.21.15.169:0/3537970625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5561b9c1b790 0x5561b9c2aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f14777fe640 1 --2- 172.21.15.169:0/3537970625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5561b9c20900 0x5561b9c2ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f1476ffd640 1 -- 172.21.15.169:0/3537970625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5561b9c1b790 msgr2=0x5561b9c2aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:22.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f1476ffd640 1 --2- 172.21.15.169:0/3537970625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5561b9c1b790 0x5561b9c2aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f1476ffd640 1 -- 172.21.15.169:0/3537970625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5561b9c20900 msgr2=0x5561b9c2ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:22.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f1476ffd640 1 --2- 172.21.15.169:0/3537970625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5561b9c20900 0x5561b9c2ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f1476ffd640 1 -- 172.21.15.169:0/3537970625 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5561b9c2c040 con 0x5561b9c1ada0 2024-03-28T00:57:22.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f1476ffd640 1 --2- 172.21.15.169:0/3537970625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c1ada0 0x5561b9c2a500 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f146800f830 tx=0x7f1468005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4941511499603f2a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:22.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f145aff5640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:57:22.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f145aff5640 1 -- 172.21.15.169:0/3537970625 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468046060 con 0x5561b9c1ada0 2024-03-28T00:57:22.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f145aff5640 1 -- 172.21.15.169:0/3537970625 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f1468004f50 con 0x5561b9c1ada0 2024-03-28T00:57:22.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.725+0000 7f147ffc1740 1 -- 172.21.15.169:0/3537970625 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f146800fc20 con 0x5561b9c1ada0 2024-03-28T00:57:22.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.726+0000 7f145aff5640 1 -- 172.21.15.169:0/3537970625 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1468031030 con 0x5561b9c1ada0 2024-03-28T00:57:22.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.726+0000 7f147ffc1740 10 client.15675 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:22.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.726+0000 7f147ffc1740 1 -- 172.21.15.169:0/3537970625 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5561b9c8c850 con 0x5561b9c1ada0 2024-03-28T00:57:22.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.726+0000 7f145aff5640 1 -- 172.21.15.169:0/3537970625 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f146800fc20 con 0x5561b9c1ada0 2024-03-28T00:57:22.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.726+0000 7f145aff5640 1 -- 172.21.15.169:0/3537970625 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f1468003680 con 0x5561b9c1ada0 2024-03-28T00:57:22.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.726+0000 7f147ffc1740 10 client.15675 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:22.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.726+0000 7f147ffc1740 10 client.15675 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:22.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.726+0000 7f147ffc1740 20 client.15675 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:22.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.726+0000 7f147ffc1740 1 --2- 172.21.15.169:0/3537970625 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5561b9c8d010 0x5561b9c8f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.726+0000 7f147ffc1740 4 client.15675 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T00:57:22.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.726+0000 7f147ffc1740 1 -- 172.21.15.169:0/3537970625 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5561b9b87c60 con 0x5561b9c8d010 2024-03-28T00:57:22.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.726+0000 7f14767fc640 1 --2- 172.21.15.169:0/3537970625 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5561b9c8d010 0x5561b9c8f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.726+0000 7f145aff5640 10 client.15675 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:22.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.726+0000 7f14767fc640 1 --2- 172.21.15.169:0/3537970625 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5561b9c8d010 0x5561b9c8f420 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ad3194ac6f81471 server_cookie=33eaeefa3a33fc77 in_seq=0 out_seq=0 2024-03-28T00:57:22.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f145aff5640 1 -- 172.21.15.169:0/3537970625 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5561b9b87c60 con 0x5561b9c8d010 2024-03-28T00:57:22.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f145aff5640 10 client.15675 handle_command_reply: tid=0 2024-03-28T00:57:22.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147ffc1740 1 client.15675 shutdown 2024-03-28T00:57:22.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147ffc1740 1 -- 172.21.15.169:0/3537970625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c1ada0 msgr2=0x5561b9c2a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:22.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147ffc1740 1 --2- 172.21.15.169:0/3537970625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c1ada0 0x5561b9c2a500 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f146800f830 tx=0x7f1468005fd0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147ffc1740 1 -- 172.21.15.169:0/3537970625 shutdown_connections 2024-03-28T00:57:22.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147ffc1740 1 --2- 172.21.15.169:0/3537970625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5561b9c20900 0x5561b9c2ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147ffc1740 1 --2- 172.21.15.169:0/3537970625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5561b9c1b790 0x5561b9c2aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147ffc1740 1 --2- 172.21.15.169:0/3537970625 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5561b9c8d010 0x5561b9c8f420 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147ffc1740 1 --2- 172.21.15.169:0/3537970625 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5561b9c1ada0 0x5561b9c2a500 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f145aff5640 0 client.15675 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:22.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147ffc1740 1 -- 172.21.15.169:0/3537970625 >> 172.21.15.169:0/3537970625 conn(0x5561b9c16310 msgr2=0x5561b9c37f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:22.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147ffc1740 1 -- 172.21.15.169:0/3537970625 shutdown_connections 2024-03-28T00:57:22.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147ffc1740 1 -- 172.21.15.169:0/3537970625 wait complete. 2024-03-28T00:57:22.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147ffc1740 20 client.15675 trim_cache size 0 max 16384 2024-03-28T00:57:22.737 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:57:22.737 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "no active scrubs running", 2024-03-28T00:57:22.737 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": {} 2024-03-28T00:57:22.737 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:57:22.738 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147d3d4640 1 -- 172.21.15.169:0/3242262668 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f14600a2890 msgr2=0x7f14600a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:22.738 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147d3d4640 1 --2- 172.21.15.169:0/3242262668 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f14600a2890 0x7f14600a4d50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x5561b9c1bff0 tx=0x7f147002c0f0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147d3d4640 1 -- 172.21.15.169:0/3242262668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1478101790 msgr2=0x7f1478197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:22.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147d3d4640 1 --2- 172.21.15.169:0/3242262668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1478101790 0x7f1478197ee0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f146c00e9f0 tx=0x7f146c00eeb0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147d3d4640 1 -- 172.21.15.169:0/3242262668 shutdown_connections 2024-03-28T00:57:22.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147d3d4640 1 --2- 172.21.15.169:0/3242262668 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f14600a2890 0x7f14600a4d50 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147d3d4640 1 --2- 172.21.15.169:0/3242262668 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1478102150 0x7f147819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147d3d4640 1 --2- 172.21.15.169:0/3242262668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1478101790 0x7f1478197ee0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147d3d4640 1 --2- 172.21.15.169:0/3242262668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14781005a0 0x7f14781979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147d3d4640 1 -- 172.21.15.169:0/3242262668 >> 172.21.15.169:0/3242262668 conn(0x7f14780fbb30 msgr2=0x7f14780fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:22.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147d3d4640 1 -- 172.21.15.169:0/3242262668 shutdown_connections 2024-03-28T00:57:22.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.727+0000 7f147d3d4640 1 -- 172.21.15.169:0/3242262668 wait complete. 2024-03-28T00:57:22.741 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T00:57:22.741 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:57:22.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.841+0000 7fdbca09a640 1 -- 172.21.15.169:0/975236452 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbc4102170 msgr2=0x7fdbc4108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:22.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.841+0000 7fdbca09a640 1 --2- 172.21.15.169:0/975236452 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbc4102170 0x7fdbc4108a00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc009ca0 tx=0x7fdbbc02c0f0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.841+0000 7fdbca09a640 1 -- 172.21.15.169:0/975236452 shutdown_connections 2024-03-28T00:57:22.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.841+0000 7fdbca09a640 1 --2- 172.21.15.169:0/975236452 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbc4102170 0x7fdbc4108a00 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.841+0000 7fdbca09a640 1 --2- 172.21.15.169:0/975236452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc41017b0 0x7fdbc4101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.841+0000 7fdbca09a640 1 --2- 172.21.15.169:0/975236452 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdbc41005c0 0x7fdbc41009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.841+0000 7fdbca09a640 1 -- 172.21.15.169:0/975236452 >> 172.21.15.169:0/975236452 conn(0x7fdbc40fbb30 msgr2=0x7fdbc40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:22.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.841+0000 7fdbca09a640 1 -- 172.21.15.169:0/975236452 shutdown_connections 2024-03-28T00:57:22.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.841+0000 7fdbca09a640 1 -- 172.21.15.169:0/975236452 wait complete. 2024-03-28T00:57:22.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.842+0000 7fdbca09a640 1 Processor -- start 2024-03-28T00:57:22.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.842+0000 7fdbca09a640 1 -- start start 2024-03-28T00:57:22.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.842+0000 7fdbca09a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc41005c0 0x7fdbc4196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.842+0000 7fdbca09a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbc41017b0 0x7fdbc4196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.842+0000 7fdbca09a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdbc4102170 0x7fdbc419de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.842+0000 7fdbca09a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc410c240 con 0x7fdbc41017b0 2024-03-28T00:57:22.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.842+0000 7fdbca09a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc410c0c0 con 0x7fdbc4102170 2024-03-28T00:57:22.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.842+0000 7fdbca09a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc410c3c0 con 0x7fdbc41005c0 2024-03-28T00:57:22.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.842+0000 7fdbc2ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbc41017b0 0x7fdbc4196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.842+0000 7fdbc37fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc41005c0 0x7fdbc4196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.842+0000 7fdbc2ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbc41017b0 0x7fdbc4196da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60544/0 (socket says 172.21.15.169:60544) 2024-03-28T00:57:22.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.842+0000 7fdbc2ffd640 1 -- 172.21.15.169:0/1485735252 learned_addr learned my addr 172.21.15.169:0/1485735252 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:22.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.842+0000 7fdbc3fff640 1 --2- 172.21.15.169:0/1485735252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdbc4102170 0x7fdbc419de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.843+0000 7fdbc2ffd640 1 -- 172.21.15.169:0/1485735252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc41005c0 msgr2=0x7fdbc4196860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:22.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.843+0000 7fdbc2ffd640 1 --2- 172.21.15.169:0/1485735252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc41005c0 0x7fdbc4196860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.843+0000 7fdbc2ffd640 1 -- 172.21.15.169:0/1485735252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdbc4102170 msgr2=0x7fdbc419de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:22.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.843+0000 7fdbc2ffd640 1 --2- 172.21.15.169:0/1485735252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdbc4102170 0x7fdbc419de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:22.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.843+0000 7fdbc2ffd640 1 -- 172.21.15.169:0/1485735252 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc419e480 con 0x7fdbc41017b0 2024-03-28T00:57:22.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.843+0000 7fdbc37fe640 1 --2- 172.21.15.169:0/1485735252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc41005c0 0x7fdbc4196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:22.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.843+0000 7fdbc3fff640 1 --2- 172.21.15.169:0/1485735252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdbc4102170 0x7fdbc419de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:22.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.843+0000 7fdbc2ffd640 1 --2- 172.21.15.169:0/1485735252 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbc41017b0 0x7fdbc4196da0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800e9f0 tx=0x7fdbb800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:22.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.843+0000 7fdbc0ff9640 1 -- 172.21.15.169:0/1485735252 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb800cd80 con 0x7fdbc41017b0 2024-03-28T00:57:22.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.843+0000 7fdbc0ff9640 1 -- 172.21.15.169:0/1485735252 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbb8005500 con 0x7fdbc41017b0 2024-03-28T00:57:22.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.843+0000 7fdbca09a640 1 -- 172.21.15.169:0/1485735252 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc419e770 con 0x7fdbc41017b0 2024-03-28T00:57:22.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.843+0000 7fdbc0ff9640 1 -- 172.21.15.169:0/1485735252 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8010690 con 0x7fdbc41017b0 2024-03-28T00:57:22.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.843+0000 7fdbca09a640 1 -- 172.21.15.169:0/1485735252 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc419ec70 con 0x7fdbc41017b0 2024-03-28T00:57:22.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.844+0000 7fdbca09a640 1 -- 172.21.15.169:0/1485735252 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc4107880 con 0x7fdbc41017b0 2024-03-28T00:57:22.850 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.845+0000 7fdbc0ff9640 1 -- 172.21.15.169:0/1485735252 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fdbb80046c0 con 0x7fdbc41017b0 2024-03-28T00:57:22.850 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.846+0000 7fdbc0ff9640 1 --2- 172.21.15.169:0/1485735252 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdbac0a2890 0x7fdbac0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:22.850 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.846+0000 7fdbc37fe640 1 --2- 172.21.15.169:0/1485735252 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdbac0a2890 0x7fdbac0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:22.850 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.846+0000 7fdbc0ff9640 1 -- 172.21.15.169:0/1485735252 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fdbb80cf380 con 0x7fdbc41017b0 2024-03-28T00:57:22.850 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.846+0000 7fdbc37fe640 1 --2- 172.21.15.169:0/1485735252 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdbac0a2890 0x7fdbac0a4d50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4101610 tx=0x7fdbb4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:22.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:22.850+0000 7fdbc0ff9640 1 -- 172.21.15.169:0/1485735252 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdbb809bf20 con 0x7fdbc41017b0 2024-03-28T00:57:23.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.062+0000 7fdbca09a640 1 -- 172.21.15.169:0/1485735252 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbc402d070 con 0x7fdbc41017b0 2024-03-28T00:57:23.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.062+0000 7fdbc0ff9640 1 -- 172.21.15.169:0/1485735252 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7fdbb809b820 con 0x7fdbc41017b0 2024-03-28T00:57:23.065 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:57:23.065 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T00:57:23.066 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T00:57:23.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.064+0000 7fdbca09a640 1 -- 172.21.15.169:0/1485735252 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdbac0a2890 msgr2=0x7fdbac0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.064+0000 7fdbca09a640 1 --2- 172.21.15.169:0/1485735252 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdbac0a2890 0x7fdbac0a4d50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4101610 tx=0x7fdbb4005f90 comp rx=0 tx=0).stop 2024-03-28T00:57:23.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.064+0000 7fdbca09a640 1 -- 172.21.15.169:0/1485735252 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbc41017b0 msgr2=0x7fdbc4196da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.064+0000 7fdbca09a640 1 --2- 172.21.15.169:0/1485735252 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbc41017b0 0x7fdbc4196da0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800e9f0 tx=0x7fdbb800edf0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.064+0000 7fdbca09a640 1 -- 172.21.15.169:0/1485735252 shutdown_connections 2024-03-28T00:57:23.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.064+0000 7fdbca09a640 1 --2- 172.21.15.169:0/1485735252 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdbac0a2890 0x7fdbac0a4d50 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.064+0000 7fdbca09a640 1 --2- 172.21.15.169:0/1485735252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdbc4102170 0x7fdbc419de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.064+0000 7fdbca09a640 1 --2- 172.21.15.169:0/1485735252 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbc41017b0 0x7fdbc4196da0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.064+0000 7fdbca09a640 1 --2- 172.21.15.169:0/1485735252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc41005c0 0x7fdbc4196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.065+0000 7fdbca09a640 1 -- 172.21.15.169:0/1485735252 >> 172.21.15.169:0/1485735252 conn(0x7fdbc40fbb30 msgr2=0x7fdbc4104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:23.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.065+0000 7fdbca09a640 1 -- 172.21.15.169:0/1485735252 shutdown_connections 2024-03-28T00:57:23.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.065+0000 7fdbca09a640 1 -- 172.21.15.169:0/1485735252 wait complete. 2024-03-28T00:57:23.078 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T00:57:23.079 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T00:57:23.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.181+0000 7f150ec34640 1 -- 172.21.15.169:0/2643775998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15081021e0 msgr2=0x7f15081045d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.181+0000 7f150ec34640 1 --2- 172.21.15.169:0/2643775998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15081021e0 0x7f15081045d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f1500009900 tx=0x7f150002bc30 comp rx=0 tx=0).stop 2024-03-28T00:57:23.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.181+0000 7f150ec34640 1 -- 172.21.15.169:0/2643775998 shutdown_connections 2024-03-28T00:57:23.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.181+0000 7f150ec34640 1 --2- 172.21.15.169:0/2643775998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1508107440 0x7f1508109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.181+0000 7f150ec34640 1 --2- 172.21.15.169:0/2643775998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1508104b10 0x7f1508106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.181+0000 7f150ec34640 1 --2- 172.21.15.169:0/2643775998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15081021e0 0x7f15081045d0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.181+0000 7f150ec34640 1 -- 172.21.15.169:0/2643775998 >> 172.21.15.169:0/2643775998 conn(0x7f15080fbb50 msgr2=0x7f15080fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:23.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.181+0000 7f150ec34640 1 -- 172.21.15.169:0/2643775998 shutdown_connections 2024-03-28T00:57:23.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.181+0000 7f150ec34640 1 -- 172.21.15.169:0/2643775998 wait complete. 2024-03-28T00:57:23.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.181+0000 7f150ec34640 1 Processor -- start 2024-03-28T00:57:23.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.181+0000 7f150ec34640 1 -- start start 2024-03-28T00:57:23.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f150ec34640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15081021e0 0x7f1508197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f150ec34640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1508104b10 0x7f1508197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f150ec34640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1508107440 0x7f150819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f150ec34640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150810d360 con 0x7f15081021e0 2024-03-28T00:57:23.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f150ec34640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150810d1e0 con 0x7f1508104b10 2024-03-28T00:57:23.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f150ec34640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150810d4e0 con 0x7f1508107440 2024-03-28T00:57:23.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f150c9d0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15081021e0 0x7f1508197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f150c9d0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15081021e0 0x7f1508197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60568/0 (socket says 172.21.15.169:60568) 2024-03-28T00:57:23.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f150c9d0640 1 -- 172.21.15.169:0/3153852420 learned_addr learned my addr 172.21.15.169:0/3153852420 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:23.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f150d1d1640 1 --2- 172.21.15.169:0/3153852420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1508107440 0x7f150819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f1507fff640 1 --2- 172.21.15.169:0/3153852420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1508104b10 0x7f1508197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f150c9d0640 1 -- 172.21.15.169:0/3153852420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1508107440 msgr2=0x7f150819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f150c9d0640 1 --2- 172.21.15.169:0/3153852420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1508107440 0x7f150819f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f150c9d0640 1 -- 172.21.15.169:0/3153852420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1508104b10 msgr2=0x7f1508197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f150c9d0640 1 --2- 172.21.15.169:0/3153852420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1508104b10 0x7f1508197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f150c9d0640 1 -- 172.21.15.169:0/3153852420 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150819f690 con 0x7f15081021e0 2024-03-28T00:57:23.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f150d1d1640 1 --2- 172.21.15.169:0/3153852420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1508107440 0x7f150819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:23.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f150c9d0640 1 --2- 172.21.15.169:0/3153852420 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15081021e0 0x7f1508197a70 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1500009230 tx=0x7f150002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:23.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f1505ffb640 1 -- 172.21.15.169:0/3153852420 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1500041070 con 0x7f15081021e0 2024-03-28T00:57:23.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f1505ffb640 1 -- 172.21.15.169:0/3153852420 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1500004450 con 0x7f15081021e0 2024-03-28T00:57:23.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f1505ffb640 1 -- 172.21.15.169:0/3153852420 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1500003db0 con 0x7f15081021e0 2024-03-28T00:57:23.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.182+0000 7f150ec34640 1 -- 172.21.15.169:0/3153852420 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f150819f920 con 0x7f15081021e0 2024-03-28T00:57:23.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.183+0000 7f150ec34640 1 -- 172.21.15.169:0/3153852420 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f150819fdc0 con 0x7f15081021e0 2024-03-28T00:57:23.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.185+0000 7f1505ffb640 1 -- 172.21.15.169:0/3153852420 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f1500044070 con 0x7f15081021e0 2024-03-28T00:57:23.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.185+0000 7f151181a740 1 Processor -- start 2024-03-28T00:57:23.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.185+0000 7f151181a740 1 -- start start 2024-03-28T00:57:23.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f151181a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edb690 0x55ba06edbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f151181a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06edc080 0x55ba06ee0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f151181a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06ee11f0 0x55ba06ee3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f151181a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba06e9e610 con 0x55ba06ee11f0 2024-03-28T00:57:23.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f151181a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba06e413c0 con 0x55ba06edb690 2024-03-28T00:57:23.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f151181a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba06de9d60 con 0x55ba06edc080 2024-03-28T00:57:23.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f1507fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06ee11f0 0x55ba06ee3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f150c9d0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06edc080 0x55ba06ee0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f150d1d1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edb690 0x55ba06edbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f150c9d0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06edc080 0x55ba06ee0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:34388/0 (socket says 172.21.15.169:34388) 2024-03-28T00:57:23.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f150d1d1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edb690 0x55ba06edbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57792/0 (socket says 172.21.15.169:57792) 2024-03-28T00:57:23.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f1507fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06ee11f0 0x55ba06ee3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60570/0 (socket says 172.21.15.169:60570) 2024-03-28T00:57:23.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f150c9d0640 1 -- 172.21.15.169:0/1439951925 learned_addr learned my addr 172.21.15.169:0/1439951925 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:23.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f1505ffb640 1 --2- 172.21.15.169:0/3153852420 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f14f40a2890 0x7f14f40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f150c9d0640 1 -- 172.21.15.169:0/1439951925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edb690 msgr2=0x55ba06edbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f150c9d0640 1 --2- 172.21.15.169:0/1439951925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edb690 0x55ba06edbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f150c9d0640 1 -- 172.21.15.169:0/1439951925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06ee11f0 msgr2=0x55ba06ee3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f150c9d0640 1 --2- 172.21.15.169:0/1439951925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06ee11f0 0x55ba06ee3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f150c9d0640 1 -- 172.21.15.169:0/1439951925 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba06b553f0 con 0x55ba06edc080 2024-03-28T00:57:23.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f1507fff640 1 --2- 172.21.15.169:0/1439951925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06ee11f0 0x55ba06ee3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:23.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f150d1d1640 1 --2- 172.21.15.169:0/1439951925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edb690 0x55ba06edbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:23.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f150d1d1640 1 --2- 172.21.15.169:0/3153852420 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f14f40a2890 0x7f14f40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f1505ffb640 1 -- 172.21.15.169:0/3153852420 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f15000f0a30 con 0x7f15081021e0 2024-03-28T00:57:23.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.186+0000 7f150c9d0640 1 --2- 172.21.15.169:0/1439951925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06edc080 0x55ba06ee0b80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f15000ee090 tx=0x7f1500005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b78703dfa4058832 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:23.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.187+0000 7f14feffd640 1 -- 172.21.15.169:0/1439951925 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15000bcdb0 con 0x55ba06edc080 2024-03-28T00:57:23.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.187+0000 7f150d1d1640 1 --2- 172.21.15.169:0/3153852420 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f14f40a2890 0x7f14f40a4d50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f14f0009fd0 tx=0x7f14f00046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:23.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.187+0000 7f14feffd640 1 -- 172.21.15.169:0/1439951925 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f150004de50 con 0x55ba06edc080 2024-03-28T00:57:23.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.187+0000 7f14feffd640 1 -- 172.21.15.169:0/1439951925 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150004da20 con 0x55ba06edc080 2024-03-28T00:57:23.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.187+0000 7f151181a740 1 -- 172.21.15.169:0/1439951925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06edc080 msgr2=0x55ba06ee0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.187+0000 7f151181a740 1 --2- 172.21.15.169:0/1439951925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06edc080 0x55ba06ee0b80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f15000ee090 tx=0x7f1500005e50 comp rx=0 tx=0).stop 2024-03-28T00:57:23.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.187+0000 7f151181a740 1 -- 172.21.15.169:0/1439951925 shutdown_connections 2024-03-28T00:57:23.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.187+0000 7f151181a740 1 --2- 172.21.15.169:0/1439951925 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06ee11f0 0x55ba06ee3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.187+0000 7f151181a740 1 --2- 172.21.15.169:0/1439951925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06edc080 0x55ba06ee0b80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.187+0000 7f151181a740 1 --2- 172.21.15.169:0/1439951925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edb690 0x55ba06edbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.187+0000 7f151181a740 1 -- 172.21.15.169:0/1439951925 >> 172.21.15.169:0/1439951925 conn(0x55ba06ed6c00 msgr2=0x55ba06ed7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:23.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.187+0000 7f151181a740 1 -- 172.21.15.169:0/1439951925 shutdown_connections 2024-03-28T00:57:23.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.187+0000 7f151181a740 1 -- 172.21.15.169:0/1439951925 wait complete. 2024-03-28T00:57:23.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.188+0000 7f151181a740 1 -- start start 2024-03-28T00:57:23.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.188+0000 7f151181a740 1 Processor -- start 2024-03-28T00:57:23.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.188+0000 7f151181a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edb690 0x55ba06ee56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.188+0000 7f151181a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06edc080 0x55ba06ee5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.188+0000 7f151181a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06ee11f0 0x55ba06eecc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.188+0000 7f151181a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba06dfd240 con 0x55ba06ee11f0 2024-03-28T00:57:23.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.188+0000 7f151181a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba06e3d460 con 0x55ba06edb690 2024-03-28T00:57:23.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.188+0000 7f151181a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba06ee3f60 con 0x55ba06edc080 2024-03-28T00:57:23.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.188+0000 7f151181a740 10 client.0 _ll_register_callbacks cb 0x55ba06dfcc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:23.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.188+0000 7f150d1d1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06ee11f0 0x55ba06eecc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.188+0000 7f150c9d0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edb690 0x55ba06ee56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.188+0000 7f150d1d1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06ee11f0 0x55ba06eecc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60576/0 (socket says 172.21.15.169:60576) 2024-03-28T00:57:23.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.188+0000 7f150d1d1640 1 -- 172.21.15.169:0/1506140643 learned_addr learned my addr 172.21.15.169:0/1506140643 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:23.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.188+0000 7f1507fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06edc080 0x55ba06ee5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.188+0000 7f150c9d0640 1 -- 172.21.15.169:0/1506140643 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06edc080 msgr2=0x55ba06ee5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.188+0000 7f150c9d0640 1 --2- 172.21.15.169:0/1506140643 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06edc080 0x55ba06ee5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.188+0000 7f150c9d0640 1 -- 172.21.15.169:0/1506140643 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06ee11f0 msgr2=0x55ba06eecc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.188+0000 7f150c9d0640 1 --2- 172.21.15.169:0/1506140643 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06ee11f0 0x55ba06eecc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.188+0000 7f150c9d0640 1 -- 172.21.15.169:0/1506140643 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ba06c1f7d0 con 0x55ba06edb690 2024-03-28T00:57:23.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.188+0000 7f150d1d1640 1 --2- 172.21.15.169:0/1506140643 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06ee11f0 0x55ba06eecc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:23.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.189+0000 7f150c9d0640 1 --2- 172.21.15.169:0/1506140643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edb690 0x55ba06ee56a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f15000b9440 tx=0x7f1500009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6a3fcaf95b897f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:23.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.189+0000 7f14e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:57:23.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.189+0000 7f14e7fff640 1 -- 172.21.15.169:0/1506140643 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150004b070 con 0x55ba06edb690 2024-03-28T00:57:23.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.189+0000 7f14e7fff640 1 -- 172.21.15.169:0/1506140643 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f1500004c60 con 0x55ba06edb690 2024-03-28T00:57:23.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.189+0000 7f151181a740 1 -- 172.21.15.169:0/1506140643 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f150004de50 con 0x55ba06edb690 2024-03-28T00:57:23.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.189+0000 7f14e7fff640 1 -- 172.21.15.169:0/1506140643 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f150004dc80 con 0x55ba06edb690 2024-03-28T00:57:23.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.189+0000 7f151181a740 10 client.25517 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:23.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.189+0000 7f151181a740 1 -- 172.21.15.169:0/1506140643 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ba06eeff90 con 0x55ba06edb690 2024-03-28T00:57:23.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.189+0000 7f14e7fff640 1 -- 172.21.15.169:0/1506140643 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f150004de50 con 0x55ba06edb690 2024-03-28T00:57:23.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.189+0000 7f14e7fff640 1 -- 172.21.15.169:0/1506140643 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f15000ee200 con 0x55ba06edb690 2024-03-28T00:57:23.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.189+0000 7f151181a740 10 client.25517 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:23.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.189+0000 7f151181a740 10 client.25517 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T00:57:23.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.189+0000 7f151181a740 20 client.25517 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:23.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.189+0000 7f151181a740 1 --2- 172.21.15.169:0/1506140643 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55ba06ef12c0 0x55ba06ef36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.189+0000 7f151181a740 4 client.25517 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:57:23.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.189+0000 7f151181a740 1 -- 172.21.15.169:0/1506140643 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ba06e4e900 con 0x55ba06ef12c0 2024-03-28T00:57:23.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.189+0000 7f1507fff640 1 --2- 172.21.15.169:0/1506140643 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55ba06ef12c0 0x55ba06ef36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.190+0000 7f1507fff640 1 --2- 172.21.15.169:0/1506140643 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55ba06ef12c0 0x55ba06ef36b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=712b5c7ae6dd68e2 server_cookie=fa2777b601ae8f57 in_seq=0 out_seq=0 2024-03-28T00:57:23.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.190+0000 7f14e7fff640 10 client.25517 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T00:57:23.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.190+0000 7f14e7fff640 1 -- 172.21.15.169:0/1506140643 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ba06e4e900 con 0x55ba06ef12c0 2024-03-28T00:57:23.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.190+0000 7f14e7fff640 10 client.25517 handle_command_reply: tid=0 2024-03-28T00:57:23.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.190+0000 7f151181a740 1 client.25517 shutdown 2024-03-28T00:57:23.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.190+0000 7f151181a740 1 -- 172.21.15.169:0/1506140643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edb690 msgr2=0x55ba06ee56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.191+0000 7f151181a740 1 --2- 172.21.15.169:0/1506140643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edb690 0x55ba06ee56a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f15000b9440 tx=0x7f1500009260 comp rx=0 tx=0).stop 2024-03-28T00:57:23.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.191+0000 7f151181a740 1 -- 172.21.15.169:0/1506140643 shutdown_connections 2024-03-28T00:57:23.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.191+0000 7f151181a740 1 --2- 172.21.15.169:0/1506140643 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55ba06ef12c0 0x55ba06ef36b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.191+0000 7f151181a740 1 --2- 172.21.15.169:0/1506140643 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06ee11f0 0x55ba06eecc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.191+0000 7f14e7fff640 0 client.25517 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T00:57:23.207 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.191+0000 7f151181a740 1 --2- 172.21.15.169:0/1506140643 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06edc080 0x55ba06ee5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.207 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.191+0000 7f151181a740 1 --2- 172.21.15.169:0/1506140643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edb690 0x55ba06ee56a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.207 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.191+0000 7f151181a740 1 -- 172.21.15.169:0/1506140643 >> 172.21.15.169:0/1506140643 conn(0x55ba06ed6c00 msgr2=0x55ba06ed5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:23.207 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.191+0000 7f151181a740 1 -- 172.21.15.169:0/1506140643 shutdown_connections 2024-03-28T00:57:23.207 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.191+0000 7f151181a740 1 -- 172.21.15.169:0/1506140643 wait complete. 2024-03-28T00:57:23.207 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.191+0000 7f151181a740 20 client.25517 trim_cache size 0 max 16384 2024-03-28T00:57:23.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.203+0000 7f151181a740 1 Processor -- start 2024-03-28T00:57:23.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.203+0000 7f151181a740 1 -- start start 2024-03-28T00:57:23.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.203+0000 7f151181a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06edb690 0x55ba06ee5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.203+0000 7f151181a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edc080 0x55ba06eecbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.203+0000 7f151181a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06ee11f0 0x55ba06eda570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.203+0000 7f151181a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15081ac6a0 con 0x55ba06edb690 2024-03-28T00:57:23.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.203+0000 7f151181a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15081aa1a0 con 0x55ba06edc080 2024-03-28T00:57:23.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.203+0000 7f151181a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15081a8240 con 0x55ba06ee11f0 2024-03-28T00:57:23.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.203+0000 7f150c9d0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06edb690 0x55ba06ee5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.203+0000 7f150c9d0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06edb690 0x55ba06ee5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60590/0 (socket says 172.21.15.169:60590) 2024-03-28T00:57:23.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.203+0000 7f150c9d0640 1 -- 172.21.15.169:0/3298131730 learned_addr learned my addr 172.21.15.169:0/3298131730 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:23.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.203+0000 7f150d1d1640 1 --2- 172.21.15.169:0/3298131730 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06ee11f0 0x55ba06eda570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.203+0000 7f1507fff640 1 --2- 172.21.15.169:0/3298131730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edc080 0x55ba06eecbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.204+0000 7f150c9d0640 1 -- 172.21.15.169:0/3298131730 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06ee11f0 msgr2=0x55ba06eda570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.204+0000 7f150c9d0640 1 --2- 172.21.15.169:0/3298131730 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06ee11f0 0x55ba06eda570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.204+0000 7f150c9d0640 1 -- 172.21.15.169:0/3298131730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edc080 msgr2=0x55ba06eecbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.204+0000 7f150c9d0640 1 --2- 172.21.15.169:0/3298131730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edc080 0x55ba06eecbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.204+0000 7f150c9d0640 1 -- 172.21.15.169:0/3298131730 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba06ee4780 con 0x55ba06edb690 2024-03-28T00:57:23.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.204+0000 7f150c9d0640 1 --2- 172.21.15.169:0/3298131730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06edb690 0x55ba06ee5b40 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1500009430 tx=0x7f15000d1290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1312f9c3394b923b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:23.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.204+0000 7f14fe7fc640 1 -- 172.21.15.169:0/3298131730 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150004b020 con 0x55ba06edb690 2024-03-28T00:57:23.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.204+0000 7f14fe7fc640 1 -- 172.21.15.169:0/3298131730 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15000bd6e0 con 0x55ba06edb690 2024-03-28T00:57:23.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.204+0000 7f14fe7fc640 1 -- 172.21.15.169:0/3298131730 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1500087e10 con 0x55ba06edb690 2024-03-28T00:57:23.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.205+0000 7f151181a740 1 -- 172.21.15.169:0/3298131730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06edb690 msgr2=0x55ba06ee5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.205+0000 7f151181a740 1 --2- 172.21.15.169:0/3298131730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06edb690 0x55ba06ee5b40 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1500009430 tx=0x7f15000d1290 comp rx=0 tx=0).stop 2024-03-28T00:57:23.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.205+0000 7f151181a740 1 -- 172.21.15.169:0/3298131730 shutdown_connections 2024-03-28T00:57:23.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.205+0000 7f151181a740 1 --2- 172.21.15.169:0/3298131730 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06ee11f0 0x55ba06eda570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.205+0000 7f151181a740 1 --2- 172.21.15.169:0/3298131730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edc080 0x55ba06eecbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.205+0000 7f151181a740 1 --2- 172.21.15.169:0/3298131730 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06edb690 0x55ba06ee5b40 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.205+0000 7f151181a740 1 -- 172.21.15.169:0/3298131730 >> 172.21.15.169:0/3298131730 conn(0x55ba06ed6c00 msgr2=0x55ba06ed9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:23.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.205+0000 7f151181a740 1 -- 172.21.15.169:0/3298131730 shutdown_connections 2024-03-28T00:57:23.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.205+0000 7f151181a740 1 -- 172.21.15.169:0/3298131730 wait complete. 2024-03-28T00:57:23.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.205+0000 7f151181a740 1 -- start start 2024-03-28T00:57:23.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.205+0000 7f151181a740 1 Processor -- start 2024-03-28T00:57:23.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.205+0000 7f151181a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06edb690 0x55ba06eea7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.205+0000 7f151181a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edc080 0x55ba06eead00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.205+0000 7f151181a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06ee11f0 0x55ba06eebd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.205+0000 7f151181a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15081a8840 con 0x55ba06edb690 2024-03-28T00:57:23.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.205+0000 7f151181a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15081a9b40 con 0x55ba06edc080 2024-03-28T00:57:23.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.205+0000 7f151181a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15081a8e80 con 0x55ba06ee11f0 2024-03-28T00:57:23.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.206+0000 7f151181a740 10 client.0 _ll_register_callbacks cb 0x7f15000b9310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:23.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.206+0000 7f150c9d0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06edb690 0x55ba06eea7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.206+0000 7f150c9d0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06edb690 0x55ba06eea7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60596/0 (socket says 172.21.15.169:60596) 2024-03-28T00:57:23.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.206+0000 7f150c9d0640 1 -- 172.21.15.169:0/2478695429 learned_addr learned my addr 172.21.15.169:0/2478695429 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:23.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.206+0000 7f1507fff640 1 --2- 172.21.15.169:0/2478695429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edc080 0x55ba06eead00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.206+0000 7f150d1d1640 1 --2- 172.21.15.169:0/2478695429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06ee11f0 0x55ba06eebd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.206+0000 7f150c9d0640 1 -- 172.21.15.169:0/2478695429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06ee11f0 msgr2=0x55ba06eebd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.206+0000 7f150c9d0640 1 --2- 172.21.15.169:0/2478695429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06ee11f0 0x55ba06eebd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.206+0000 7f150c9d0640 1 -- 172.21.15.169:0/2478695429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edc080 msgr2=0x55ba06eead00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.206+0000 7f150c9d0640 1 --2- 172.21.15.169:0/2478695429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edc080 0x55ba06eead00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.206+0000 7f150c9d0640 1 -- 172.21.15.169:0/2478695429 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ba06eec300 con 0x55ba06edb690 2024-03-28T00:57:23.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.206+0000 7f150c9d0640 1 --2- 172.21.15.169:0/2478695429 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06edb690 0x55ba06eea7c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f15000b99b0 tx=0x7f15000f0e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de493539b7698544 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:23.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.206+0000 7f14e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:57:23.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.206+0000 7f14e7fff640 1 -- 172.21.15.169:0/2478695429 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150004b070 con 0x55ba06edb690 2024-03-28T00:57:23.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.206+0000 7f14e7fff640 1 -- 172.21.15.169:0/2478695429 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f15000c5500 con 0x55ba06edb690 2024-03-28T00:57:23.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.206+0000 7f151181a740 1 -- 172.21.15.169:0/2478695429 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f15000bd6e0 con 0x55ba06edb690 2024-03-28T00:57:23.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.206+0000 7f14e7fff640 1 -- 172.21.15.169:0/2478695429 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15000bc940 con 0x55ba06edb690 2024-03-28T00:57:23.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.206+0000 7f151181a740 10 client.15699 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:23.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.206+0000 7f151181a740 1 -- 172.21.15.169:0/2478695429 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ba06eec300 con 0x55ba06edb690 2024-03-28T00:57:23.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.207+0000 7f14e7fff640 1 -- 172.21.15.169:0/2478695429 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15000bd6e0 con 0x55ba06edb690 2024-03-28T00:57:23.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.207+0000 7f14e7fff640 1 -- 172.21.15.169:0/2478695429 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f15000b7b50 con 0x55ba06edb690 2024-03-28T00:57:23.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.207+0000 7f151181a740 10 client.15699 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:23.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.207+0000 7f151181a740 10 client.15699 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T00:57:23.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.207+0000 7f151181a740 20 client.15699 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:23.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.207+0000 7f151181a740 1 --2- 172.21.15.169:0/2478695429 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55ba06ef12c0 0x55ba06f4f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.207+0000 7f151181a740 4 client.15699 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T00:57:23.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.207+0000 7f151181a740 1 -- 172.21.15.169:0/2478695429 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ba06e48c50 con 0x55ba06ef12c0 2024-03-28T00:57:23.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.207+0000 7f1507fff640 1 --2- 172.21.15.169:0/2478695429 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55ba06ef12c0 0x55ba06f4f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.207+0000 7f14e7fff640 10 client.15699 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T00:57:23.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.207+0000 7f1507fff640 1 --2- 172.21.15.169:0/2478695429 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55ba06ef12c0 0x55ba06f4f300 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9ed91a1d0320d1dd server_cookie=b3f387f2a2f9295f in_seq=0 out_seq=0 2024-03-28T00:57:23.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.207+0000 7f14e7fff640 1 -- 172.21.15.169:0/2478695429 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ba06e48c50 con 0x55ba06ef12c0 2024-03-28T00:57:23.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.207+0000 7f14e7fff640 10 client.15699 handle_command_reply: tid=0 2024-03-28T00:57:23.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.207+0000 7f151181a740 1 client.15699 shutdown 2024-03-28T00:57:23.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f151181a740 1 -- 172.21.15.169:0/2478695429 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06edb690 msgr2=0x55ba06eea7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f151181a740 1 --2- 172.21.15.169:0/2478695429 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06edb690 0x55ba06eea7c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f15000b99b0 tx=0x7f15000f0e30 comp rx=0 tx=0).stop 2024-03-28T00:57:23.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f151181a740 1 -- 172.21.15.169:0/2478695429 shutdown_connections 2024-03-28T00:57:23.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f151181a740 1 --2- 172.21.15.169:0/2478695429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba06ee11f0 0x55ba06eebd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f151181a740 1 --2- 172.21.15.169:0/2478695429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba06edc080 0x55ba06eead00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f151181a740 1 --2- 172.21.15.169:0/2478695429 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55ba06ef12c0 0x55ba06f4f300 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f151181a740 1 --2- 172.21.15.169:0/2478695429 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ba06edb690 0x55ba06eea7c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f151181a740 1 -- 172.21.15.169:0/2478695429 >> 172.21.15.169:0/2478695429 conn(0x55ba06ed6c00 msgr2=0x55ba06ee5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:23.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f14e7fff640 0 client.15699 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T00:57:23.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f151181a740 1 -- 172.21.15.169:0/2478695429 shutdown_connections 2024-03-28T00:57:23.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f151181a740 1 -- 172.21.15.169:0/2478695429 wait complete. 2024-03-28T00:57:23.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f151181a740 20 client.15699 trim_cache size 0 max 16384 2024-03-28T00:57:23.222 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T00:57:23.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f150ec34640 1 -- 172.21.15.169:0/3153852420 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f14f40a2890 msgr2=0x7f14f40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f150ec34640 1 --2- 172.21.15.169:0/3153852420 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f14f40a2890 0x7f14f40a4d50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f14f0009fd0 tx=0x7f14f00046c0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f150ec34640 1 -- 172.21.15.169:0/3153852420 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15081021e0 msgr2=0x7f1508197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f150ec34640 1 --2- 172.21.15.169:0/3153852420 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15081021e0 0x7f1508197a70 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1500009230 tx=0x7f150002c650 comp rx=0 tx=0).stop 2024-03-28T00:57:23.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f150ec34640 1 -- 172.21.15.169:0/3153852420 shutdown_connections 2024-03-28T00:57:23.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f150ec34640 1 --2- 172.21.15.169:0/3153852420 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f14f40a2890 0x7f14f40a4d50 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f150ec34640 1 --2- 172.21.15.169:0/3153852420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1508107440 0x7f150819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f150ec34640 1 --2- 172.21.15.169:0/3153852420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1508104b10 0x7f1508197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f150ec34640 1 --2- 172.21.15.169:0/3153852420 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15081021e0 0x7f1508197a70 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f150ec34640 1 -- 172.21.15.169:0/3153852420 >> 172.21.15.169:0/3153852420 conn(0x7f15080fbb50 msgr2=0x7f1508102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:23.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f150ec34640 1 -- 172.21.15.169:0/3153852420 shutdown_connections 2024-03-28T00:57:23.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.208+0000 7f150ec34640 1 -- 172.21.15.169:0/3153852420 wait complete. 2024-03-28T00:57:23.225 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T00:57:23.225 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T00:57:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.327+0000 7f232b174640 1 -- 172.21.15.169:0/3107608676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23241028a0 msgr2=0x7f2324102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.327+0000 7f232b174640 1 --2- 172.21.15.169:0/3107608676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23241028a0 0x7f2324102d20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2310009900 tx=0x7f231002bc30 comp rx=0 tx=0).stop 2024-03-28T00:57:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.327+0000 7f232b174640 1 -- 172.21.15.169:0/3107608676 shutdown_connections 2024-03-28T00:57:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.327+0000 7f232b174640 1 --2- 172.21.15.169:0/3107608676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2324103260 0x7f2324109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.327+0000 7f232b174640 1 --2- 172.21.15.169:0/3107608676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23241028a0 0x7f2324102d20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.327+0000 7f232b174640 1 --2- 172.21.15.169:0/3107608676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23241016b0 0x7f2324101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.327+0000 7f232b174640 1 -- 172.21.15.169:0/3107608676 >> 172.21.15.169:0/3107608676 conn(0x7f23240fcc40 msgr2=0x7f23240ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.327+0000 7f232b174640 1 -- 172.21.15.169:0/3107608676 shutdown_connections 2024-03-28T00:57:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.327+0000 7f232b174640 1 -- 172.21.15.169:0/3107608676 wait complete. 2024-03-28T00:57:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.327+0000 7f232b174640 1 Processor -- start 2024-03-28T00:57:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.327+0000 7f232b174640 1 -- start start 2024-03-28T00:57:23.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.327+0000 7f232b174640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23241016b0 0x7f2324197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.327+0000 7f232b174640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23241028a0 0x7f2324197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.327+0000 7f232b174640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2324103260 0x7f232419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.327+0000 7f232b174640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232410d310 con 0x7f2324103260 2024-03-28T00:57:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.327+0000 7f232b174640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232410d190 con 0x7f23241028a0 2024-03-28T00:57:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.327+0000 7f232b174640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232410d490 con 0x7f23241016b0 2024-03-28T00:57:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.328+0000 7f2329711640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2324103260 0x7f232419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.328+0000 7f2328f10640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23241016b0 0x7f2324197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.328+0000 7f2323fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23241028a0 0x7f2324197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.328+0000 7f2328f10640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23241016b0 0x7f2324197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:34434/0 (socket says 172.21.15.169:34434) 2024-03-28T00:57:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.328+0000 7f2328f10640 1 -- 172.21.15.169:0/2854021664 learned_addr learned my addr 172.21.15.169:0/2854021664 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.328+0000 7f2323fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23241028a0 0x7f2324197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57846/0 (socket says 172.21.15.169:57846) 2024-03-28T00:57:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.328+0000 7f2329711640 1 -- 172.21.15.169:0/2854021664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23241016b0 msgr2=0x7f2324197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.328+0000 7f2329711640 1 --2- 172.21.15.169:0/2854021664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23241016b0 0x7f2324197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.328+0000 7f2329711640 1 -- 172.21.15.169:0/2854021664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23241028a0 msgr2=0x7f2324197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.328+0000 7f2329711640 1 --2- 172.21.15.169:0/2854021664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23241028a0 0x7f2324197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.328+0000 7f2329711640 1 -- 172.21.15.169:0/2854021664 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232419f680 con 0x7f2324103260 2024-03-28T00:57:23.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.328+0000 7f2323fff640 1 --2- 172.21.15.169:0/2854021664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23241028a0 0x7f2324197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:23.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.328+0000 7f2328f10640 1 --2- 172.21.15.169:0/2854021664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23241016b0 0x7f2324197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:23.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.328+0000 7f2329711640 1 --2- 172.21.15.169:0/2854021664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2324103260 0x7f232419f010 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f231c00ec50 tx=0x7f231c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:23.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.328+0000 7f2321ffb640 1 -- 172.21.15.169:0/2854021664 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c003b70 con 0x7f2324103260 2024-03-28T00:57:23.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.328+0000 7f232b174640 1 -- 172.21.15.169:0/2854021664 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f232419f970 con 0x7f2324103260 2024-03-28T00:57:23.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.328+0000 7f2321ffb640 1 -- 172.21.15.169:0/2854021664 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f231c003d10 con 0x7f2324103260 2024-03-28T00:57:23.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.328+0000 7f2321ffb640 1 -- 172.21.15.169:0/2854021664 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c010830 con 0x7f2324103260 2024-03-28T00:57:23.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.328+0000 7f232b174640 1 -- 172.21.15.169:0/2854021664 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f232419fe90 con 0x7f2324103260 2024-03-28T00:57:23.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.330+0000 7f2321ffb640 1 -- 172.21.15.169:0/2854021664 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f231c003fe0 con 0x7f2324103260 2024-03-28T00:57:23.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.331+0000 7f232de37740 1 Processor -- start 2024-03-28T00:57:23.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.331+0000 7f232de37740 1 -- start start 2024-03-28T00:57:23.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.331+0000 7f2321ffb640 1 --2- 172.21.15.169:0/2854021664 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f23140a2890 0x7f23140a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.331+0000 7f232de37740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd361aaa0 0x563bd361aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.331+0000 7f232de37740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563bd361b490 0x563bd361ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.331+0000 7f232de37740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd3620600 0x563bd3622a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.331+0000 7f232de37740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bd35d9610 con 0x563bd361aaa0 2024-03-28T00:57:23.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.331+0000 7f232de37740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bd357c3c0 con 0x563bd3620600 2024-03-28T00:57:23.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.331+0000 7f232de37740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bd3524d60 con 0x563bd361b490 2024-03-28T00:57:23.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.331+0000 7f2328f10640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd361aaa0 0x563bd361aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.331+0000 7f2328f10640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd361aaa0 0x563bd361aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60626/0 (socket says 172.21.15.169:60626) 2024-03-28T00:57:23.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.331+0000 7f2323fff640 1 --2- 172.21.15.169:0/2854021664 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f23140a2890 0x7f23140a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.331+0000 7f2328f10640 1 -- 172.21.15.169:0/4050956879 learned_addr learned my addr 172.21.15.169:0/4050956879 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:23.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.331+0000 7f2321ffb640 1 -- 172.21.15.169:0/2854021664 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f231c0cf490 con 0x7f2324103260 2024-03-28T00:57:23.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.331+0000 7f2329711640 1 --2- 172.21.15.169:0/4050956879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd3620600 0x563bd3622a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.331+0000 7f2323fff640 1 --2- 172.21.15.169:0/4050956879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563bd361b490 0x563bd361ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.331+0000 7f2328f10640 1 -- 172.21.15.169:0/4050956879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563bd361b490 msgr2=0x563bd361ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.331+0000 7f2328f10640 1 --2- 172.21.15.169:0/4050956879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563bd361b490 0x563bd361ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.331+0000 7f2328f10640 1 -- 172.21.15.169:0/4050956879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd3620600 msgr2=0x563bd3622a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.332+0000 7f2328f10640 1 --2- 172.21.15.169:0/4050956879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd3620600 0x563bd3622a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.332+0000 7f2328f10640 1 -- 172.21.15.169:0/4050956879 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563bd32903f0 con 0x563bd361aaa0 2024-03-28T00:57:23.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.332+0000 7f2329711640 1 --2- 172.21.15.169:0/4050956879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd3620600 0x563bd3622a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:23.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.332+0000 7f2323fff640 1 --2- 172.21.15.169:0/2854021664 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f23140a2890 0x7f23140a4d50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f231002c7b0 tx=0x7f231002c630 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:23.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.332+0000 7f2328f10640 1 --2- 172.21.15.169:0/4050956879 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd361aaa0 0x563bd361aec0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f23180115c0 tx=0x7f2318011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f41b49aa869e785 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:23.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.332+0000 7f230effd640 1 -- 172.21.15.169:0/4050956879 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2318005020 con 0x563bd361aaa0 2024-03-28T00:57:23.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.332+0000 7f230effd640 1 -- 172.21.15.169:0/4050956879 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23180047a0 con 0x563bd361aaa0 2024-03-28T00:57:23.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.332+0000 7f230effd640 1 -- 172.21.15.169:0/4050956879 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231800f9a0 con 0x563bd361aaa0 2024-03-28T00:57:23.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.333+0000 7f232de37740 1 -- 172.21.15.169:0/4050956879 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd361aaa0 msgr2=0x563bd361aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.333+0000 7f232de37740 1 --2- 172.21.15.169:0/4050956879 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd361aaa0 0x563bd361aec0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f23180115c0 tx=0x7f2318011a80 comp rx=0 tx=0).stop 2024-03-28T00:57:23.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.333+0000 7f232de37740 1 -- 172.21.15.169:0/4050956879 shutdown_connections 2024-03-28T00:57:23.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.333+0000 7f232de37740 1 --2- 172.21.15.169:0/4050956879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd3620600 0x563bd3622a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.333+0000 7f232de37740 1 --2- 172.21.15.169:0/4050956879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563bd361b490 0x563bd361ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.333+0000 7f232de37740 1 --2- 172.21.15.169:0/4050956879 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd361aaa0 0x563bd361aec0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.333+0000 7f232de37740 1 -- 172.21.15.169:0/4050956879 >> 172.21.15.169:0/4050956879 conn(0x563bd3611c00 msgr2=0x563bd3612000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:23.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.333+0000 7f232de37740 1 -- 172.21.15.169:0/4050956879 shutdown_connections 2024-03-28T00:57:23.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.333+0000 7f232de37740 1 -- 172.21.15.169:0/4050956879 wait complete. 2024-03-28T00:57:23.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.333+0000 7f232de37740 1 -- start start 2024-03-28T00:57:23.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.333+0000 7f232de37740 1 Processor -- start 2024-03-28T00:57:23.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.333+0000 7f232de37740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd361aaa0 0x563bd3624e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.333+0000 7f232de37740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563bd361b490 0x563bd3625350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.333+0000 7f232de37740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd3620600 0x563bd362c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.333+0000 7f232de37740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bd3538240 con 0x563bd361aaa0 2024-03-28T00:57:23.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.333+0000 7f232de37740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bd3578460 con 0x563bd3620600 2024-03-28T00:57:23.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.333+0000 7f232de37740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bd36235b0 con 0x563bd361b490 2024-03-28T00:57:23.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.334+0000 7f232de37740 10 client.0 _ll_register_callbacks cb 0x563bd3537c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:23.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.334+0000 7f2328f10640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd361aaa0 0x563bd3624e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.334+0000 7f2328f10640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd361aaa0 0x563bd3624e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60634/0 (socket says 172.21.15.169:60634) 2024-03-28T00:57:23.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.334+0000 7f2323fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563bd361b490 0x563bd3625350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.334+0000 7f2328f10640 1 -- 172.21.15.169:0/1428712455 learned_addr learned my addr 172.21.15.169:0/1428712455 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:23.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.334+0000 7f2329711640 1 --2- 172.21.15.169:0/1428712455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd3620600 0x563bd362c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.334+0000 7f2328f10640 1 -- 172.21.15.169:0/1428712455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563bd361b490 msgr2=0x563bd3625350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.334+0000 7f2328f10640 1 --2- 172.21.15.169:0/1428712455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563bd361b490 0x563bd3625350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.334+0000 7f2328f10640 1 -- 172.21.15.169:0/1428712455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd3620600 msgr2=0x563bd362c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.334+0000 7f2328f10640 1 --2- 172.21.15.169:0/1428712455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd3620600 0x563bd362c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.334+0000 7f2328f10640 1 -- 172.21.15.169:0/1428712455 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563bd335a7d0 con 0x563bd361aaa0 2024-03-28T00:57:23.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.334+0000 7f2328f10640 1 --2- 172.21.15.169:0/1428712455 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd361aaa0 0x563bd3624e10 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f231801b040 tx=0x7f23180052b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239e907e6fc6e641 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:23.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.334+0000 7f2303fff640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:57:23.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.334+0000 7f2303fff640 1 -- 172.21.15.169:0/1428712455 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231801d050 con 0x563bd361aaa0 2024-03-28T00:57:23.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.334+0000 7f2303fff640 1 -- 172.21.15.169:0/1428712455 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f23180054a0 con 0x563bd361aaa0 2024-03-28T00:57:23.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.334+0000 7f232de37740 1 -- 172.21.15.169:0/1428712455 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f23180047a0 con 0x563bd361aaa0 2024-03-28T00:57:23.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.334+0000 7f2303fff640 1 -- 172.21.15.169:0/1428712455 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2318032050 con 0x563bd361aaa0 2024-03-28T00:57:23.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.334+0000 7f232de37740 10 client.15711 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:23.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.334+0000 7f232de37740 1 -- 172.21.15.169:0/1428712455 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563bd362f750 con 0x563bd361aaa0 2024-03-28T00:57:23.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.335+0000 7f2303fff640 1 -- 172.21.15.169:0/1428712455 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23180047a0 con 0x563bd361aaa0 2024-03-28T00:57:23.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.335+0000 7f2303fff640 1 -- 172.21.15.169:0/1428712455 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f23180035f0 con 0x563bd361aaa0 2024-03-28T00:57:23.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.335+0000 7f232de37740 10 client.15711 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:23.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.335+0000 7f232de37740 10 client.15711 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T00:57:23.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.335+0000 7f232de37740 20 client.15711 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:23.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.335+0000 7f232de37740 1 --2- 172.21.15.169:0/1428712455 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x563bd3630ca0 0x563bd36330f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.335+0000 7f232de37740 4 client.15711 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:57:23.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.335+0000 7f232de37740 1 -- 172.21.15.169:0/1428712455 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563bd3589900 con 0x563bd3630ca0 2024-03-28T00:57:23.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.335+0000 7f2323fff640 1 --2- 172.21.15.169:0/1428712455 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x563bd3630ca0 0x563bd36330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.335+0000 7f2303fff640 10 client.15711 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T00:57:23.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.335+0000 7f2323fff640 1 --2- 172.21.15.169:0/1428712455 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x563bd3630ca0 0x563bd36330f0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6aa25673652bcd83 server_cookie=544d999f140c2208 in_seq=0 out_seq=0 2024-03-28T00:57:23.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.336+0000 7f2303fff640 1 -- 172.21.15.169:0/1428712455 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563bd3589900 con 0x563bd3630ca0 2024-03-28T00:57:23.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.336+0000 7f2303fff640 10 client.15711 handle_command_reply: tid=0 2024-03-28T00:57:23.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.336+0000 7f232de37740 1 client.15711 shutdown 2024-03-28T00:57:23.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.336+0000 7f232de37740 1 -- 172.21.15.169:0/1428712455 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd361aaa0 msgr2=0x563bd3624e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.336+0000 7f232de37740 1 --2- 172.21.15.169:0/1428712455 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd361aaa0 0x563bd3624e10 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f231801b040 tx=0x7f23180052b0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.336+0000 7f232de37740 1 -- 172.21.15.169:0/1428712455 shutdown_connections 2024-03-28T00:57:23.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.336+0000 7f232de37740 1 --2- 172.21.15.169:0/1428712455 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x563bd3630ca0 0x563bd36330f0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.336+0000 7f232de37740 1 --2- 172.21.15.169:0/1428712455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd3620600 0x563bd362c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.336+0000 7f2303fff640 0 client.15711 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T00:57:23.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.336+0000 7f232de37740 1 --2- 172.21.15.169:0/1428712455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563bd361b490 0x563bd3625350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.336+0000 7f232de37740 1 --2- 172.21.15.169:0/1428712455 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd361aaa0 0x563bd3624e10 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.336+0000 7f232de37740 1 -- 172.21.15.169:0/1428712455 >> 172.21.15.169:0/1428712455 conn(0x563bd3611c00 msgr2=0x563bd3610e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:23.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.336+0000 7f232de37740 1 -- 172.21.15.169:0/1428712455 shutdown_connections 2024-03-28T00:57:23.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.336+0000 7f232de37740 1 -- 172.21.15.169:0/1428712455 wait complete. 2024-03-28T00:57:23.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.336+0000 7f232de37740 20 client.15711 trim_cache size 0 max 16384 2024-03-28T00:57:23.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.348+0000 7f232de37740 1 Processor -- start 2024-03-28T00:57:23.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.348+0000 7f232de37740 1 -- start start 2024-03-28T00:57:23.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.348+0000 7f232de37740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd361aaa0 0x563bd3610ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.348+0000 7f232de37740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563bd361b490 0x563bd36111e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.348+0000 7f232de37740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd3620600 0x563bd362c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.348+0000 7f232de37740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23241aa540 con 0x563bd3620600 2024-03-28T00:57:23.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.348+0000 7f232de37740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23241a8040 con 0x563bd361aaa0 2024-03-28T00:57:23.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.348+0000 7f232de37740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23241a60e0 con 0x563bd361b490 2024-03-28T00:57:23.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.348+0000 7f2329711640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd3620600 0x563bd362c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.348+0000 7f2329711640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd3620600 0x563bd362c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60646/0 (socket says 172.21.15.169:60646) 2024-03-28T00:57:23.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.349+0000 7f2329711640 1 -- 172.21.15.169:0/1817347827 learned_addr learned my addr 172.21.15.169:0/1817347827 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:23.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.349+0000 7f2328f10640 1 --2- 172.21.15.169:0/1817347827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd361aaa0 0x563bd3610ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.349+0000 7f2323fff640 1 --2- 172.21.15.169:0/1817347827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563bd361b490 0x563bd36111e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.349+0000 7f2329711640 1 -- 172.21.15.169:0/1817347827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563bd361b490 msgr2=0x563bd36111e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.349+0000 7f2329711640 1 --2- 172.21.15.169:0/1817347827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563bd361b490 0x563bd36111e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.349+0000 7f2329711640 1 -- 172.21.15.169:0/1817347827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd361aaa0 msgr2=0x563bd3610ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.349+0000 7f2329711640 1 --2- 172.21.15.169:0/1817347827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd361aaa0 0x563bd3610ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.349+0000 7f2329711640 1 -- 172.21.15.169:0/1817347827 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563bd361c450 con 0x563bd3620600 2024-03-28T00:57:23.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.349+0000 7f2328f10640 1 --2- 172.21.15.169:0/1817347827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd361aaa0 0x563bd3610ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:23.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.349+0000 7f2329711640 1 --2- 172.21.15.169:0/1817347827 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd3620600 0x563bd362c2b0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f231c098300 tx=0x7f231c09b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c0c23b632064e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:23.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.349+0000 7f230e7fc640 1 -- 172.21.15.169:0/1817347827 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c013190 con 0x563bd3620600 2024-03-28T00:57:23.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.349+0000 7f230e7fc640 1 -- 172.21.15.169:0/1817347827 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f231c09b930 con 0x563bd3620600 2024-03-28T00:57:23.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.349+0000 7f230e7fc640 1 -- 172.21.15.169:0/1817347827 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c097fa0 con 0x563bd3620600 2024-03-28T00:57:23.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f232de37740 1 -- 172.21.15.169:0/1817347827 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd3620600 msgr2=0x563bd362c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f232de37740 1 --2- 172.21.15.169:0/1817347827 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd3620600 0x563bd362c2b0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f231c098300 tx=0x7f231c09b7c0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f232de37740 1 -- 172.21.15.169:0/1817347827 shutdown_connections 2024-03-28T00:57:23.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f232de37740 1 --2- 172.21.15.169:0/1817347827 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd3620600 0x563bd362c2b0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f232de37740 1 --2- 172.21.15.169:0/1817347827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563bd361b490 0x563bd36111e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f232de37740 1 --2- 172.21.15.169:0/1817347827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd361aaa0 0x563bd3610ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f232de37740 1 -- 172.21.15.169:0/1817347827 >> 172.21.15.169:0/1817347827 conn(0x563bd3611c00 msgr2=0x563bd3610590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:23.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f232de37740 1 -- 172.21.15.169:0/1817347827 shutdown_connections 2024-03-28T00:57:23.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f232de37740 1 -- 172.21.15.169:0/1817347827 wait complete. 2024-03-28T00:57:23.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f232de37740 1 -- start start 2024-03-28T00:57:23.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f232de37740 1 Processor -- start 2024-03-28T00:57:23.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f232de37740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd361aaa0 0x563bd362a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f232de37740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd361b490 0x563bd362a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f232de37740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563bd3620600 0x563bd362b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f232de37740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23241a66e0 con 0x563bd361aaa0 2024-03-28T00:57:23.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f232de37740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23241a79e0 con 0x563bd361b490 2024-03-28T00:57:23.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f232de37740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23241a6d20 con 0x563bd3620600 2024-03-28T00:57:23.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f232de37740 10 client.0 _ll_register_callbacks cb 0x7f2318011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:23.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f2328f10640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd361aaa0 0x563bd362a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f2328f10640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd361aaa0 0x563bd362a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60660/0 (socket says 172.21.15.169:60660) 2024-03-28T00:57:23.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f2328f10640 1 -- 172.21.15.169:0/1679985047 learned_addr learned my addr 172.21.15.169:0/1679985047 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:23.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f2329711640 1 --2- 172.21.15.169:0/1679985047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563bd3620600 0x563bd362b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.350+0000 7f2323fff640 1 --2- 172.21.15.169:0/1679985047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd361b490 0x563bd362a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.351+0000 7f2328f10640 1 -- 172.21.15.169:0/1679985047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563bd3620600 msgr2=0x563bd362b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.351+0000 7f2328f10640 1 --2- 172.21.15.169:0/1679985047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563bd3620600 0x563bd362b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.351+0000 7f2328f10640 1 -- 172.21.15.169:0/1679985047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd361b490 msgr2=0x563bd362a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.351+0000 7f2328f10640 1 --2- 172.21.15.169:0/1679985047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd361b490 0x563bd362a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.351+0000 7f2328f10640 1 -- 172.21.15.169:0/1679985047 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563bd362bb00 con 0x563bd361aaa0 2024-03-28T00:57:23.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.351+0000 7f2328f10640 1 --2- 172.21.15.169:0/1679985047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd361aaa0 0x563bd362a060 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2318003990 tx=0x7f2318009e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d384306948ff6473 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:23.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.351+0000 7f2303fff640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:57:23.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.351+0000 7f2303fff640 1 -- 172.21.15.169:0/1679985047 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231801d040 con 0x563bd361aaa0 2024-03-28T00:57:23.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.351+0000 7f2303fff640 1 -- 172.21.15.169:0/1679985047 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2318005320 con 0x563bd361aaa0 2024-03-28T00:57:23.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.351+0000 7f232de37740 1 -- 172.21.15.169:0/1679985047 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f231c09b930 con 0x563bd361aaa0 2024-03-28T00:57:23.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.351+0000 7f2303fff640 1 -- 172.21.15.169:0/1679985047 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2318004d10 con 0x563bd361aaa0 2024-03-28T00:57:23.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.351+0000 7f232de37740 10 client.15717 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:23.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.351+0000 7f232de37740 1 -- 172.21.15.169:0/1679985047 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563bd368ca00 con 0x563bd361aaa0 2024-03-28T00:57:23.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.351+0000 7f2303fff640 1 -- 172.21.15.169:0/1679985047 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f231c09b930 con 0x563bd361aaa0 2024-03-28T00:57:23.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.351+0000 7f2303fff640 1 -- 172.21.15.169:0/1679985047 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f231800f8d0 con 0x563bd361aaa0 2024-03-28T00:57:23.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.351+0000 7f232de37740 10 client.15717 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:23.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.351+0000 7f232de37740 10 client.15717 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T00:57:23.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.351+0000 7f232de37740 20 client.15717 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:23.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.351+0000 7f232de37740 1 --2- 172.21.15.169:0/1679985047 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x563bd368d370 0x563bd368f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.351+0000 7f232de37740 4 client.15717 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T00:57:23.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.352+0000 7f232de37740 1 -- 172.21.15.169:0/1679985047 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563bd3583c50 con 0x563bd368d370 2024-03-28T00:57:23.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.352+0000 7f2323fff640 1 --2- 172.21.15.169:0/1679985047 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x563bd368d370 0x563bd368f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.352+0000 7f2303fff640 10 client.15717 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T00:57:23.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.352+0000 7f2323fff640 1 --2- 172.21.15.169:0/1679985047 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x563bd368d370 0x563bd368f790 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=34cdf48827ab00c6 server_cookie=b29b11c87aef81da in_seq=0 out_seq=0 2024-03-28T00:57:23.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.352+0000 7f2303fff640 1 -- 172.21.15.169:0/1679985047 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563bd3583c50 con 0x563bd368d370 2024-03-28T00:57:23.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.352+0000 7f2303fff640 10 client.15717 handle_command_reply: tid=0 2024-03-28T00:57:23.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.352+0000 7f232de37740 1 client.15717 shutdown 2024-03-28T00:57:23.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.352+0000 7f232de37740 1 -- 172.21.15.169:0/1679985047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd361aaa0 msgr2=0x563bd362a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.352+0000 7f232de37740 1 --2- 172.21.15.169:0/1679985047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd361aaa0 0x563bd362a060 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2318003990 tx=0x7f2318009e40 comp rx=0 tx=0).stop 2024-03-28T00:57:23.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.352+0000 7f232de37740 1 -- 172.21.15.169:0/1679985047 shutdown_connections 2024-03-28T00:57:23.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.352+0000 7f232de37740 1 --2- 172.21.15.169:0/1679985047 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x563bd368d370 0x563bd368f790 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.352+0000 7f232de37740 1 --2- 172.21.15.169:0/1679985047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563bd3620600 0x563bd362b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.352+0000 7f2303fff640 0 client.15717 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T00:57:23.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.352+0000 7f232de37740 1 --2- 172.21.15.169:0/1679985047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bd361b490 0x563bd362a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.352+0000 7f232de37740 1 --2- 172.21.15.169:0/1679985047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563bd361aaa0 0x563bd362a060 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.352+0000 7f232de37740 1 -- 172.21.15.169:0/1679985047 >> 172.21.15.169:0/1679985047 conn(0x563bd3611c00 msgr2=0x563bd3610e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:23.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.352+0000 7f232de37740 1 -- 172.21.15.169:0/1679985047 shutdown_connections 2024-03-28T00:57:23.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.352+0000 7f232de37740 1 -- 172.21.15.169:0/1679985047 wait complete. 2024-03-28T00:57:23.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.352+0000 7f232de37740 20 client.15717 trim_cache size 0 max 16384 2024-03-28T00:57:23.363 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T00:57:23.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.353+0000 7f232b174640 1 -- 172.21.15.169:0/2854021664 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f23140a2890 msgr2=0x7f23140a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.353+0000 7f232b174640 1 --2- 172.21.15.169:0/2854021664 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f23140a2890 0x7f23140a4d50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f231002c7b0 tx=0x7f231002c630 comp rx=0 tx=0).stop 2024-03-28T00:57:23.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.353+0000 7f232b174640 1 -- 172.21.15.169:0/2854021664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2324103260 msgr2=0x7f232419f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.353+0000 7f232b174640 1 --2- 172.21.15.169:0/2854021664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2324103260 0x7f232419f010 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f231c00ec50 tx=0x7f231c00c6a0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.353+0000 7f232b174640 1 -- 172.21.15.169:0/2854021664 shutdown_connections 2024-03-28T00:57:23.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.353+0000 7f232b174640 1 --2- 172.21.15.169:0/2854021664 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f23140a2890 0x7f23140a4d50 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.353+0000 7f232b174640 1 --2- 172.21.15.169:0/2854021664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2324103260 0x7f232419f010 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.353+0000 7f232b174640 1 --2- 172.21.15.169:0/2854021664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23241028a0 0x7f2324197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.353+0000 7f232b174640 1 --2- 172.21.15.169:0/2854021664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23241016b0 0x7f2324197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.353+0000 7f232b174640 1 -- 172.21.15.169:0/2854021664 >> 172.21.15.169:0/2854021664 conn(0x7f23240fcc40 msgr2=0x7f2324105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:23.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.353+0000 7f232b174640 1 -- 172.21.15.169:0/2854021664 shutdown_connections 2024-03-28T00:57:23.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.353+0000 7f232b174640 1 -- 172.21.15.169:0/2854021664 wait complete. 2024-03-28T00:57:23.365 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:23 smithi169 ceph-mon[32274]: pgmap v377: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-28T00:57:23.365 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:23 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1485735252' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:57:23.367 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T00:57:23.367 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T00:57:23.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.469+0000 7fad9f532640 1 -- 172.21.15.169:0/2471700620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad98103280 msgr2=0x7fad98109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.469+0000 7fad9f532640 1 --2- 172.21.15.169:0/2471700620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad98103280 0x7fad98109b10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fad90009be0 tx=0x7fad9002bef0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.469+0000 7fad9f532640 1 -- 172.21.15.169:0/2471700620 shutdown_connections 2024-03-28T00:57:23.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.469+0000 7fad9f532640 1 --2- 172.21.15.169:0/2471700620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad98103280 0x7fad98109b10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.469+0000 7fad9f532640 1 --2- 172.21.15.169:0/2471700620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad981028c0 0x7fad98102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.469+0000 7fad9f532640 1 --2- 172.21.15.169:0/2471700620 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad981016d0 0x7fad98101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.469+0000 7fad9f532640 1 -- 172.21.15.169:0/2471700620 >> 172.21.15.169:0/2471700620 conn(0x7fad980fcc40 msgr2=0x7fad980ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:23.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.469+0000 7fad9f532640 1 -- 172.21.15.169:0/2471700620 shutdown_connections 2024-03-28T00:57:23.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.469+0000 7fad9f532640 1 -- 172.21.15.169:0/2471700620 wait complete. 2024-03-28T00:57:23.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.469+0000 7fad9f532640 1 Processor -- start 2024-03-28T00:57:23.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.469+0000 7fad9f532640 1 -- start start 2024-03-28T00:57:23.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad9f532640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad981016d0 0x7fad981979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad9f532640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad981028c0 0x7fad98197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad9f532640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad98103280 0x7fad9819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad9f532640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9810d350 con 0x7fad981028c0 2024-03-28T00:57:23.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad9f532640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9810d1d0 con 0x7fad981016d0 2024-03-28T00:57:23.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad9f532640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9810d4d0 con 0x7fad98103280 2024-03-28T00:57:23.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad9cacd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad981028c0 0x7fad98197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad9cacd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad981028c0 0x7fad98197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60682/0 (socket says 172.21.15.169:60682) 2024-03-28T00:57:23.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad9dacf640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad98103280 0x7fad9819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad9cacd640 1 -- 172.21.15.169:0/2266725125 learned_addr learned my addr 172.21.15.169:0/2266725125 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:23.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad9d2ce640 1 --2- 172.21.15.169:0/2266725125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad981016d0 0x7fad981979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad9cacd640 1 -- 172.21.15.169:0/2266725125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad98103280 msgr2=0x7fad9819ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad9cacd640 1 --2- 172.21.15.169:0/2266725125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad98103280 0x7fad9819ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad9cacd640 1 -- 172.21.15.169:0/2266725125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad981016d0 msgr2=0x7fad981979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad9cacd640 1 --2- 172.21.15.169:0/2266725125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad981016d0 0x7fad981979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad9cacd640 1 -- 172.21.15.169:0/2266725125 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9819f5c0 con 0x7fad981028c0 2024-03-28T00:57:23.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad9dacf640 1 --2- 172.21.15.169:0/2266725125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad98103280 0x7fad9819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:23.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad9d2ce640 1 --2- 172.21.15.169:0/2266725125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad981016d0 0x7fad981979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:23.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad9cacd640 1 --2- 172.21.15.169:0/2266725125 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad981028c0 0x7fad98197ee0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fad9400e9f0 tx=0x7fad9400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:23.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad8e7fc640 1 -- 172.21.15.169:0/2266725125 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9400cd80 con 0x7fad981028c0 2024-03-28T00:57:23.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad9f532640 1 -- 172.21.15.169:0/2266725125 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad9819f8b0 con 0x7fad981028c0 2024-03-28T00:57:23.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad8e7fc640 1 -- 172.21.15.169:0/2266725125 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad94005500 con 0x7fad981028c0 2024-03-28T00:57:23.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad8e7fc640 1 -- 172.21.15.169:0/2266725125 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad94010690 con 0x7fad981028c0 2024-03-28T00:57:23.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.470+0000 7fad9f532640 1 -- 172.21.15.169:0/2266725125 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad9819fdb0 con 0x7fad981028c0 2024-03-28T00:57:23.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.473+0000 7fad8e7fc640 1 -- 172.21.15.169:0/2266725125 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fad940046c0 con 0x7fad981028c0 2024-03-28T00:57:23.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.473+0000 7fada21f3740 1 Processor -- start 2024-03-28T00:57:23.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.473+0000 7fada21f3740 1 -- start start 2024-03-28T00:57:23.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.473+0000 7fada21f3740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c11ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.473+0000 7fada21f3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567e1c12080 0x5567e1c16b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.473+0000 7fada21f3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c171f0 0x5567e1c19680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.473+0000 7fada21f3740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567e1bd4610 con 0x5567e1c11690 2024-03-28T00:57:23.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.473+0000 7fada21f3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567e1b773c0 con 0x5567e1c171f0 2024-03-28T00:57:23.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.473+0000 7fada21f3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567e1b1fd60 con 0x5567e1c12080 2024-03-28T00:57:23.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.473+0000 7fad9dacf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c11ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.473+0000 7fad9cacd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c171f0 0x5567e1c19680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.473+0000 7fad9dacf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c11ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60692/0 (socket says 172.21.15.169:60692) 2024-03-28T00:57:23.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.473+0000 7fad9dacf640 1 -- 172.21.15.169:0/3409404514 learned_addr learned my addr 172.21.15.169:0/3409404514 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:23.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.473+0000 7fad9d2ce640 1 --2- 172.21.15.169:0/3409404514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567e1c12080 0x5567e1c16b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.474+0000 7fad8e7fc640 1 --2- 172.21.15.169:0/2266725125 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fad840a2890 0x7fad840a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.474+0000 7fad9cacd640 1 -- 172.21.15.169:0/3409404514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567e1c12080 msgr2=0x5567e1c16b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.474+0000 7fad9cacd640 1 --2- 172.21.15.169:0/3409404514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567e1c12080 0x5567e1c16b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.474+0000 7fad9cacd640 1 -- 172.21.15.169:0/3409404514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 msgr2=0x5567e1c11ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.474+0000 7fad9cacd640 1 --2- 172.21.15.169:0/3409404514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c11ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.474+0000 7fad9cacd640 1 -- 172.21.15.169:0/3409404514 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5567e188b3f0 con 0x5567e1c171f0 2024-03-28T00:57:23.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.474+0000 7fad9dacf640 1 --2- 172.21.15.169:0/2266725125 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fad840a2890 0x7fad840a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.474+0000 7fad8e7fc640 1 -- 172.21.15.169:0/2266725125 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fad940cf380 con 0x7fad981028c0 2024-03-28T00:57:23.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.474+0000 7fad9cacd640 1 --2- 172.21.15.169:0/3409404514 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c171f0 0x5567e1c19680 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fad940cc990 tx=0x7fad9406d560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b16ef3eb10e6861f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:23.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.474+0000 7fad837fe640 1 -- 172.21.15.169:0/3409404514 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9409b360 con 0x5567e1c171f0 2024-03-28T00:57:23.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.474+0000 7fad9dacf640 1 --2- 172.21.15.169:0/2266725125 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fad840a2890 0x7fad840a4d50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x5567e1c128e0 tx=0x7fad90009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:23.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.474+0000 7fad837fe640 1 -- 172.21.15.169:0/3409404514 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad940a97e0 con 0x5567e1c171f0 2024-03-28T00:57:23.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.474+0000 7fad837fe640 1 -- 172.21.15.169:0/3409404514 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9409b180 con 0x5567e1c171f0 2024-03-28T00:57:23.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.475+0000 7fada21f3740 1 -- 172.21.15.169:0/3409404514 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c171f0 msgr2=0x5567e1c19680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.475+0000 7fada21f3740 1 --2- 172.21.15.169:0/3409404514 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c171f0 0x5567e1c19680 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fad940cc990 tx=0x7fad9406d560 comp rx=0 tx=0).stop 2024-03-28T00:57:23.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.475+0000 7fada21f3740 1 -- 172.21.15.169:0/3409404514 shutdown_connections 2024-03-28T00:57:23.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.475+0000 7fada21f3740 1 --2- 172.21.15.169:0/3409404514 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c171f0 0x5567e1c19680 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.475+0000 7fada21f3740 1 --2- 172.21.15.169:0/3409404514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567e1c12080 0x5567e1c16b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.475+0000 7fada21f3740 1 --2- 172.21.15.169:0/3409404514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c11ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.475+0000 7fada21f3740 1 -- 172.21.15.169:0/3409404514 >> 172.21.15.169:0/3409404514 conn(0x5567e1c0cc00 msgr2=0x5567e1c0d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:23.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.475+0000 7fada21f3740 1 -- 172.21.15.169:0/3409404514 shutdown_connections 2024-03-28T00:57:23.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.475+0000 7fada21f3740 1 -- 172.21.15.169:0/3409404514 wait complete. 2024-03-28T00:57:23.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.475+0000 7fada21f3740 1 -- start start 2024-03-28T00:57:23.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.475+0000 7fada21f3740 1 Processor -- start 2024-03-28T00:57:23.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fada21f3740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c1b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fada21f3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c12080 0x5567e1c1bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fada21f3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567e1c171f0 0x5567e1c22f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fada21f3740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567e1b33240 con 0x5567e1c11690 2024-03-28T00:57:23.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fada21f3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567e1b73460 con 0x5567e1c12080 2024-03-28T00:57:23.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fada21f3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567e1c1a140 con 0x5567e1c171f0 2024-03-28T00:57:23.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fada21f3740 10 client.0 _ll_register_callbacks cb 0x5567e1b32c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:23.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fad9d2ce640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c1b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fad9cacd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c12080 0x5567e1c1bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fad9d2ce640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c1b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60702/0 (socket says 172.21.15.169:60702) 2024-03-28T00:57:23.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fad9cacd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c12080 0x5567e1c1bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57908/0 (socket says 172.21.15.169:57908) 2024-03-28T00:57:23.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fad9d2ce640 1 -- 172.21.15.169:0/4209263041 learned_addr learned my addr 172.21.15.169:0/4209263041 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:23.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fad9dacf640 1 --2- 172.21.15.169:0/4209263041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567e1c171f0 0x5567e1c22f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fad9d2ce640 1 -- 172.21.15.169:0/4209263041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567e1c171f0 msgr2=0x5567e1c22f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fad9d2ce640 1 --2- 172.21.15.169:0/4209263041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567e1c171f0 0x5567e1c22f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fad9d2ce640 1 -- 172.21.15.169:0/4209263041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c12080 msgr2=0x5567e1c1bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fad9d2ce640 1 --2- 172.21.15.169:0/4209263041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c12080 0x5567e1c1bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fad9d2ce640 1 -- 172.21.15.169:0/4209263041 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5567e19557d0 con 0x5567e1c11690 2024-03-28T00:57:23.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fad9cacd640 1 --2- 172.21.15.169:0/4209263041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c12080 0x5567e1c1bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:23.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fad9d2ce640 1 --2- 172.21.15.169:0/4209263041 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c1b990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fad88005fd0 tx=0x7fad8800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55094938552871a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:23.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fad80ff9640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:57:23.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fad80ff9640 1 -- 172.21.15.169:0/4209263041 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad880120d0 con 0x5567e1c11690 2024-03-28T00:57:23.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fad80ff9640 1 -- 172.21.15.169:0/4209263041 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fad88010430 con 0x5567e1c11690 2024-03-28T00:57:23.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.476+0000 7fada21f3740 1 -- 172.21.15.169:0/4209263041 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fad940a97e0 con 0x5567e1c11690 2024-03-28T00:57:23.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.477+0000 7fad80ff9640 1 -- 172.21.15.169:0/4209263041 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad88012270 con 0x5567e1c11690 2024-03-28T00:57:23.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.477+0000 7fada21f3740 10 client.15729 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:23.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.477+0000 7fada21f3740 1 -- 172.21.15.169:0/4209263041 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5567e1c26350 con 0x5567e1c11690 2024-03-28T00:57:23.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.477+0000 7fad80ff9640 1 -- 172.21.15.169:0/4209263041 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad940a97e0 con 0x5567e1c11690 2024-03-28T00:57:23.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.477+0000 7fad80ff9640 1 -- 172.21.15.169:0/4209263041 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fad8801c720 con 0x5567e1c11690 2024-03-28T00:57:23.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.477+0000 7fada21f3740 10 client.15729 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:23.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.477+0000 7fada21f3740 10 client.15729 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:23.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.477+0000 7fada21f3740 20 client.15729 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:23.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.477+0000 7fada21f3740 1 --2- 172.21.15.169:0/4209263041 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5567e1c27a50 0x5567e1c29ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.477+0000 7fada21f3740 4 client.15729 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:57:23.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.477+0000 7fada21f3740 1 -- 172.21.15.169:0/4209263041 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5567e1b84900 con 0x5567e1c27a50 2024-03-28T00:57:23.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.477+0000 7fad9cacd640 1 --2- 172.21.15.169:0/4209263041 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5567e1c27a50 0x5567e1c29ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.477+0000 7fad9cacd640 1 --2- 172.21.15.169:0/4209263041 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5567e1c27a50 0x5567e1c29ed0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9e8785a92ccdbba server_cookie=a7b8c06485ddc3c8 in_seq=0 out_seq=0 2024-03-28T00:57:23.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.477+0000 7fad80ff9640 10 client.15729 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:23.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.478+0000 7fad80ff9640 1 -- 172.21.15.169:0/4209263041 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5567e1b84900 con 0x5567e1c27a50 2024-03-28T00:57:23.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.478+0000 7fad80ff9640 10 client.15729 handle_command_reply: tid=0 2024-03-28T00:57:23.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.478+0000 7fada21f3740 1 client.15729 shutdown 2024-03-28T00:57:23.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.479+0000 7fada21f3740 1 -- 172.21.15.169:0/4209263041 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 msgr2=0x5567e1c1b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.479+0000 7fada21f3740 1 --2- 172.21.15.169:0/4209263041 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c1b990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fad88005fd0 tx=0x7fad8800f790 comp rx=0 tx=0).stop 2024-03-28T00:57:23.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.479+0000 7fada21f3740 1 -- 172.21.15.169:0/4209263041 shutdown_connections 2024-03-28T00:57:23.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.479+0000 7fada21f3740 1 --2- 172.21.15.169:0/4209263041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567e1c171f0 0x5567e1c22f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.479+0000 7fada21f3740 1 --2- 172.21.15.169:0/4209263041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c12080 0x5567e1c1bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.479+0000 7fada21f3740 1 --2- 172.21.15.169:0/4209263041 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5567e1c27a50 0x5567e1c29ed0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.479+0000 7fada21f3740 1 --2- 172.21.15.169:0/4209263041 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c1b990 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.479+0000 7fad80ff9640 0 client.15729 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:23.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.479+0000 7fada21f3740 1 -- 172.21.15.169:0/4209263041 >> 172.21.15.169:0/4209263041 conn(0x5567e1c0cc00 msgr2=0x5567e1c0be20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:23.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.479+0000 7fada21f3740 1 -- 172.21.15.169:0/4209263041 shutdown_connections 2024-03-28T00:57:23.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.479+0000 7fada21f3740 1 -- 172.21.15.169:0/4209263041 wait complete. 2024-03-28T00:57:23.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.479+0000 7fada21f3740 20 client.15729 trim_cache size 0 max 16384 2024-03-28T00:57:23.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.491+0000 7fada21f3740 1 Processor -- start 2024-03-28T00:57:23.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.491+0000 7fada21f3740 1 -- start start 2024-03-28T00:57:23.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.491+0000 7fada21f3740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c10930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.491+0000 7fada21f3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567e1c12080 0x5567e1c22ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.491+0000 7fada21f3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c171f0 0x5567e1c144e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.491+0000 7fada21f3740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad981ac790 con 0x5567e1c11690 2024-03-28T00:57:23.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.491+0000 7fada21f3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad981aa290 con 0x5567e1c171f0 2024-03-28T00:57:23.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.491+0000 7fada21f3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad981a8330 con 0x5567e1c12080 2024-03-28T00:57:23.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.491+0000 7fad9d2ce640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c10930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.491+0000 7fad9dacf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c171f0 0x5567e1c144e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.491+0000 7fad9cacd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567e1c12080 0x5567e1c22ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.491+0000 7fad9d2ce640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c10930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60712/0 (socket says 172.21.15.169:60712) 2024-03-28T00:57:23.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.491+0000 7fad9d2ce640 1 -- 172.21.15.169:0/4218453588 learned_addr learned my addr 172.21.15.169:0/4218453588 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:23.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.492+0000 7fad9dacf640 1 -- 172.21.15.169:0/4218453588 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567e1c12080 msgr2=0x5567e1c22ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.492+0000 7fad9dacf640 1 --2- 172.21.15.169:0/4218453588 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567e1c12080 0x5567e1c22ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.492+0000 7fad9dacf640 1 -- 172.21.15.169:0/4218453588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 msgr2=0x5567e1c10930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.492+0000 7fad9dacf640 1 --2- 172.21.15.169:0/4218453588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c10930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.492+0000 7fad9dacf640 1 -- 172.21.15.169:0/4218453588 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5567e1c12460 con 0x5567e1c171f0 2024-03-28T00:57:23.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.492+0000 7fad9cacd640 1 --2- 172.21.15.169:0/4218453588 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567e1c12080 0x5567e1c22ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:23.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.492+0000 7fad9d2ce640 1 --2- 172.21.15.169:0/4218453588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c10930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:23.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.492+0000 7fad9dacf640 1 --2- 172.21.15.169:0/4218453588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c171f0 0x5567e1c144e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fad90036040 tx=0x7fad90042000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db98174fcde81edd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:23.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.492+0000 7fad82ffd640 1 -- 172.21.15.169:0/4218453588 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90004040 con 0x5567e1c171f0 2024-03-28T00:57:23.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.492+0000 7fad82ffd640 1 -- 172.21.15.169:0/4218453588 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad90052070 con 0x5567e1c171f0 2024-03-28T00:57:23.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.492+0000 7fad82ffd640 1 -- 172.21.15.169:0/4218453588 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90034b40 con 0x5567e1c171f0 2024-03-28T00:57:23.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fada21f3740 1 -- 172.21.15.169:0/4218453588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c171f0 msgr2=0x5567e1c144e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fada21f3740 1 --2- 172.21.15.169:0/4218453588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c171f0 0x5567e1c144e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fad90036040 tx=0x7fad90042000 comp rx=0 tx=0).stop 2024-03-28T00:57:23.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fada21f3740 1 -- 172.21.15.169:0/4218453588 shutdown_connections 2024-03-28T00:57:23.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fada21f3740 1 --2- 172.21.15.169:0/4218453588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c171f0 0x5567e1c144e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fada21f3740 1 --2- 172.21.15.169:0/4218453588 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567e1c12080 0x5567e1c22ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fada21f3740 1 --2- 172.21.15.169:0/4218453588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c10930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fada21f3740 1 -- 172.21.15.169:0/4218453588 >> 172.21.15.169:0/4218453588 conn(0x5567e1c0cc00 msgr2=0x5567e1c0be20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:23.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fada21f3740 1 -- 172.21.15.169:0/4218453588 shutdown_connections 2024-03-28T00:57:23.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fada21f3740 1 -- 172.21.15.169:0/4218453588 wait complete. 2024-03-28T00:57:23.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fada21f3740 1 -- start start 2024-03-28T00:57:23.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fada21f3740 1 Processor -- start 2024-03-28T00:57:23.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fada21f3740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c209d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fada21f3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567e1c12080 0x5567e1c20f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fada21f3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c171f0 0x5567e1c21f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fada21f3740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad981a8930 con 0x5567e1c11690 2024-03-28T00:57:23.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fada21f3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad981a9c30 con 0x5567e1c171f0 2024-03-28T00:57:23.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fada21f3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad981a8f70 con 0x5567e1c12080 2024-03-28T00:57:23.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fada21f3740 10 client.0 _ll_register_callbacks cb 0x7fad880127f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:23.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fad9d2ce640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c209d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fad9d2ce640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c209d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60722/0 (socket says 172.21.15.169:60722) 2024-03-28T00:57:23.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fad9d2ce640 1 -- 172.21.15.169:0/2606777602 learned_addr learned my addr 172.21.15.169:0/2606777602 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:23.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.493+0000 7fad9dacf640 1 --2- 172.21.15.169:0/2606777602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c171f0 0x5567e1c21f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.494+0000 7fad9cacd640 1 --2- 172.21.15.169:0/2606777602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567e1c12080 0x5567e1c20f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.494+0000 7fad9d2ce640 1 -- 172.21.15.169:0/2606777602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567e1c12080 msgr2=0x5567e1c20f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.494+0000 7fad9d2ce640 1 --2- 172.21.15.169:0/2606777602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567e1c12080 0x5567e1c20f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.494+0000 7fad9d2ce640 1 -- 172.21.15.169:0/2606777602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c171f0 msgr2=0x5567e1c21f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:23.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.494+0000 7fad9d2ce640 1 --2- 172.21.15.169:0/2606777602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c171f0 0x5567e1c21f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.494+0000 7fad9d2ce640 1 -- 172.21.15.169:0/2606777602 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5567e1c22510 con 0x5567e1c11690 2024-03-28T00:57:23.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.494+0000 7fad9d2ce640 1 --2- 172.21.15.169:0/2606777602 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c209d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fad88009530 tx=0x7fad88005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=592b3407621e42bb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:23.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.494+0000 7fad80ff9640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:57:23.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.494+0000 7fad80ff9640 1 -- 172.21.15.169:0/2606777602 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8802f020 con 0x5567e1c11690 2024-03-28T00:57:23.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.494+0000 7fad80ff9640 1 -- 172.21.15.169:0/2606777602 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fad8801b6c0 con 0x5567e1c11690 2024-03-28T00:57:23.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.494+0000 7fada21f3740 1 -- 172.21.15.169:0/2606777602 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fad90052070 con 0x5567e1c11690 2024-03-28T00:57:23.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.494+0000 7fad80ff9640 1 -- 172.21.15.169:0/2606777602 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad880120d0 con 0x5567e1c11690 2024-03-28T00:57:23.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.494+0000 7fada21f3740 10 client.15735 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:23.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.494+0000 7fada21f3740 1 -- 172.21.15.169:0/2606777602 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5567e1c83780 con 0x5567e1c11690 2024-03-28T00:57:23.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.494+0000 7fad80ff9640 1 -- 172.21.15.169:0/2606777602 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad90052070 con 0x5567e1c11690 2024-03-28T00:57:23.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.494+0000 7fad80ff9640 1 -- 172.21.15.169:0/2606777602 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fad8802a720 con 0x5567e1c11690 2024-03-28T00:57:23.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.495+0000 7fada21f3740 10 client.15735 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:23.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.495+0000 7fada21f3740 10 client.15735 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:23.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.495+0000 7fada21f3740 20 client.15735 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:23.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.495+0000 7fada21f3740 1 --2- 172.21.15.169:0/2606777602 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5567e1c84150 0x5567e1c86570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:23.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.495+0000 7fada21f3740 4 client.15735 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T00:57:23.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.495+0000 7fada21f3740 1 -- 172.21.15.169:0/2606777602 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5567e1b7ec50 con 0x5567e1c84150 2024-03-28T00:57:23.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.495+0000 7fad9cacd640 1 --2- 172.21.15.169:0/2606777602 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5567e1c84150 0x5567e1c86570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:23.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.495+0000 7fad80ff9640 10 client.15735 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:23.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.495+0000 7fad9cacd640 1 --2- 172.21.15.169:0/2606777602 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5567e1c84150 0x5567e1c86570 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0d72cf6da38790c server_cookie=ecb833f02fd69853 in_seq=0 out_seq=0 2024-03-28T00:57:23.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.495+0000 7fad80ff9640 1 -- 172.21.15.169:0/2606777602 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5567e1b7ec50 con 0x5567e1c84150 2024-03-28T00:57:23.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.495+0000 7fad80ff9640 10 client.15735 handle_command_reply: tid=0 2024-03-28T00:57:23.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.495+0000 7fada21f3740 1 client.15735 shutdown 2024-03-28T00:57:23.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.495+0000 7fada21f3740 1 -- 172.21.15.169:0/2606777602 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 msgr2=0x5567e1c209d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.495+0000 7fada21f3740 1 --2- 172.21.15.169:0/2606777602 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c209d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fad88009530 tx=0x7fad88005880 comp rx=0 tx=0).stop 2024-03-28T00:57:23.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.495+0000 7fada21f3740 1 -- 172.21.15.169:0/2606777602 shutdown_connections 2024-03-28T00:57:23.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.495+0000 7fada21f3740 1 --2- 172.21.15.169:0/2606777602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567e1c171f0 0x5567e1c21f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.495+0000 7fada21f3740 1 --2- 172.21.15.169:0/2606777602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567e1c12080 0x5567e1c20f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.495+0000 7fada21f3740 1 --2- 172.21.15.169:0/2606777602 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5567e1c84150 0x5567e1c86570 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.495+0000 7fada21f3740 1 --2- 172.21.15.169:0/2606777602 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5567e1c11690 0x5567e1c209d0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.495+0000 7fad80ff9640 0 client.15735 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:23.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.495+0000 7fada21f3740 1 -- 172.21.15.169:0/2606777602 >> 172.21.15.169:0/2606777602 conn(0x5567e1c0cc00 msgr2=0x5567e1c2e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:23.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.496+0000 7fada21f3740 1 -- 172.21.15.169:0/2606777602 shutdown_connections 2024-03-28T00:57:23.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.496+0000 7fada21f3740 1 -- 172.21.15.169:0/2606777602 wait complete. 2024-03-28T00:57:23.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.496+0000 7fada21f3740 20 client.15735 trim_cache size 0 max 16384 2024-03-28T00:57:23.508 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T00:57:23.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.496+0000 7fad9f532640 1 -- 172.21.15.169:0/2266725125 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fad840a2890 msgr2=0x7fad840a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.496+0000 7fad9f532640 1 --2- 172.21.15.169:0/2266725125 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fad840a2890 0x7fad840a4d50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x5567e1c128e0 tx=0x7fad90009500 comp rx=0 tx=0).stop 2024-03-28T00:57:23.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.496+0000 7fad9f532640 1 -- 172.21.15.169:0/2266725125 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad981028c0 msgr2=0x7fad98197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:23.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.496+0000 7fad9f532640 1 --2- 172.21.15.169:0/2266725125 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad981028c0 0x7fad98197ee0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fad9400e9f0 tx=0x7fad9400edf0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.496+0000 7fad9f532640 1 -- 172.21.15.169:0/2266725125 shutdown_connections 2024-03-28T00:57:23.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.496+0000 7fad9f532640 1 --2- 172.21.15.169:0/2266725125 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fad840a2890 0x7fad840a4d50 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.496+0000 7fad9f532640 1 --2- 172.21.15.169:0/2266725125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad98103280 0x7fad9819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.496+0000 7fad9f532640 1 --2- 172.21.15.169:0/2266725125 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fad981028c0 0x7fad98197ee0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.496+0000 7fad9f532640 1 --2- 172.21.15.169:0/2266725125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad981016d0 0x7fad981979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:23.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.496+0000 7fad9f532640 1 -- 172.21.15.169:0/2266725125 >> 172.21.15.169:0/2266725125 conn(0x7fad980fcc40 msgr2=0x7fad98105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:23.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.496+0000 7fad9f532640 1 -- 172.21.15.169:0/2266725125 shutdown_connections 2024-03-28T00:57:23.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:23.496+0000 7fad9f532640 1 -- 172.21.15.169:0/2266725125 wait complete. 2024-03-28T00:57:23.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:23 smithi174 ceph-mon[36060]: pgmap v377: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-28T00:57:23.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:23 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1485735252' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:57:23.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:23 smithi177 ceph-mon[36206]: pgmap v377: 129 pgs: 129 active+clean; 664 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-28T00:57:23.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:23 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1485735252' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:57:24.511 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T00:57:24.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:57:24.512 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T00:57:24.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.627+0000 7f6daa59d640 1 -- 172.21.15.169:0/269448180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6da4107260 msgr2=0x7f6da41096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:24.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.627+0000 7f6daa59d640 1 --2- 172.21.15.169:0/269448180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6da4107260 0x7f6da41096b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009be0 tx=0x7f6d9c02bef0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.627+0000 7f6daa59d640 1 -- 172.21.15.169:0/269448180 shutdown_connections 2024-03-28T00:57:24.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.627+0000 7f6daa59d640 1 --2- 172.21.15.169:0/269448180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6da4107260 0x7f6da41096b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.627+0000 7f6daa59d640 1 --2- 172.21.15.169:0/269448180 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6da4104930 0x7f6da4106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.627+0000 7f6daa59d640 1 --2- 172.21.15.169:0/269448180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6da4102000 0x7f6da41043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.627+0000 7f6daa59d640 1 -- 172.21.15.169:0/269448180 >> 172.21.15.169:0/269448180 conn(0x7f6da40fba10 msgr2=0x7f6da40fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:24.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.627+0000 7f6daa59d640 1 -- 172.21.15.169:0/269448180 shutdown_connections 2024-03-28T00:57:24.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.627+0000 7f6daa59d640 1 -- 172.21.15.169:0/269448180 wait complete. 2024-03-28T00:57:24.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.627+0000 7f6daa59d640 1 Processor -- start 2024-03-28T00:57:24.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.627+0000 7f6daa59d640 1 -- start start 2024-03-28T00:57:24.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6daa59d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6da4102000 0x7f6da4197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6daa59d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6da4104930 0x7f6da4197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6daa59d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6da4107260 0x7f6da419edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6daa59d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da410d210 con 0x7f6da4104930 2024-03-28T00:57:24.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6daa59d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da410d090 con 0x7f6da4107260 2024-03-28T00:57:24.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6daa59d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da410d390 con 0x7f6da4102000 2024-03-28T00:57:24.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6da8d9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6da4104930 0x7f6da4197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6da959b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6da4102000 0x7f6da4197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6da8d9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6da4104930 0x7f6da4197d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60734/0 (socket says 172.21.15.169:60734) 2024-03-28T00:57:24.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6da8d9a640 1 -- 172.21.15.169:0/3346558688 learned_addr learned my addr 172.21.15.169:0/3346558688 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:24.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6da9d9c640 1 --2- 172.21.15.169:0/3346558688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6da4107260 0x7f6da419edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6da8d9a640 1 -- 172.21.15.169:0/3346558688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6da4102000 msgr2=0x7f6da4197820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:24.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6da8d9a640 1 --2- 172.21.15.169:0/3346558688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6da4102000 0x7f6da4197820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6da8d9a640 1 -- 172.21.15.169:0/3346558688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6da4107260 msgr2=0x7f6da419edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:24.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6da8d9a640 1 --2- 172.21.15.169:0/3346558688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6da4107260 0x7f6da419edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6da8d9a640 1 -- 172.21.15.169:0/3346558688 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da419f440 con 0x7f6da4104930 2024-03-28T00:57:24.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6da959b640 1 --2- 172.21.15.169:0/3346558688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6da4102000 0x7f6da4197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:24.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6da8d9a640 1 --2- 172.21.15.169:0/3346558688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6da4104930 0x7f6da4197d60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6da000e9f0 tx=0x7f6da000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:24.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6d9a7fc640 1 -- 172.21.15.169:0/3346558688 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da000ccc0 con 0x7f6da4104930 2024-03-28T00:57:24.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6daa59d640 1 -- 172.21.15.169:0/3346558688 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6da419f730 con 0x7f6da4104930 2024-03-28T00:57:24.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6d9a7fc640 1 -- 172.21.15.169:0/3346558688 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6da0010910 con 0x7f6da4104930 2024-03-28T00:57:24.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6d9a7fc640 1 -- 172.21.15.169:0/3346558688 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da0018960 con 0x7f6da4104930 2024-03-28T00:57:24.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.628+0000 7f6daa59d640 1 -- 172.21.15.169:0/3346558688 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6da419fc80 con 0x7f6da4104930 2024-03-28T00:57:24.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.631+0000 7f6d9a7fc640 1 -- 172.21.15.169:0/3346558688 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f6da0010ac0 con 0x7f6da4104930 2024-03-28T00:57:24.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.631+0000 7f6dae3ec740 1 Processor -- start 2024-03-28T00:57:24.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.631+0000 7f6dae3ec740 1 -- start start 2024-03-28T00:57:24.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.631+0000 7f6dae3ec740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b2f40 0x56449f7b3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.631+0000 7f6dae3ec740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b3930 0x56449f7b8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.631+0000 7f6dae3ec740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b8aa0 0x56449f7baf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.631+0000 7f6dae3ec740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56449f6eec90 con 0x56449f7b8aa0 2024-03-28T00:57:24.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.631+0000 7f6dae3ec740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x56449f714070 con 0x56449f7b3930 2024-03-28T00:57:24.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.631+0000 7f6dae3ec740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56449f6bcc80 con 0x56449f7b2f40 2024-03-28T00:57:24.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.632+0000 7f6d9a7fc640 1 --2- 172.21.15.169:0/3346558688 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6d900a2890 0x7f6d900a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.632+0000 7f6da8d9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b8aa0 0x56449f7baf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.632+0000 7f6da959b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b3930 0x56449f7b8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.632+0000 7f6da9d9c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b2f40 0x56449f7b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.632+0000 7f6da959b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b3930 0x56449f7b8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57940/0 (socket says 172.21.15.169:57940) 2024-03-28T00:57:24.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.632+0000 7f6da9d9c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b2f40 0x56449f7b3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:34532/0 (socket says 172.21.15.169:34532) 2024-03-28T00:57:24.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.632+0000 7f6da959b640 1 -- 172.21.15.169:0/3378605529 learned_addr learned my addr 172.21.15.169:0/3378605529 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:24.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.632+0000 7f6da9d9c640 1 --2- 172.21.15.169:0/3346558688 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6d900a2890 0x7f6d900a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.632+0000 7f6d9a7fc640 1 -- 172.21.15.169:0/3346558688 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f6da00cf280 con 0x7f6da4104930 2024-03-28T00:57:24.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.632+0000 7f6da959b640 1 -- 172.21.15.169:0/3378605529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b2f40 msgr2=0x56449f7b3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:24.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.632+0000 7f6da959b640 1 --2- 172.21.15.169:0/3378605529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b2f40 0x56449f7b3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.632+0000 7f6da959b640 1 -- 172.21.15.169:0/3378605529 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b8aa0 msgr2=0x56449f7baf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:24.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.632+0000 7f6da959b640 1 --2- 172.21.15.169:0/3378605529 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b8aa0 0x56449f7baf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.632+0000 7f6da959b640 1 -- 172.21.15.169:0/3378605529 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56449f6b4d10 con 0x56449f7b3930 2024-03-28T00:57:24.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.632+0000 7f6da9d9c640 1 --2- 172.21.15.169:0/3378605529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b2f40 0x56449f7b3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:24.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.632+0000 7f6da9d9c640 1 --2- 172.21.15.169:0/3346558688 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6d900a2890 0x7f6d900a4d50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c004ab0 tx=0x7f6d9c004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:24.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.632+0000 7f6da959b640 1 --2- 172.21.15.169:0/3378605529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b3930 0x56449f7b8430 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6d940115c0 tx=0x7f6d94011a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c474873a6afb2c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:24.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.632+0000 7f6d8f7fe640 1 -- 172.21.15.169:0/3378605529 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9400f640 con 0x56449f7b3930 2024-03-28T00:57:24.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.632+0000 7f6d8f7fe640 1 -- 172.21.15.169:0/3378605529 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d9400fcc0 con 0x56449f7b3930 2024-03-28T00:57:24.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.632+0000 7f6d8f7fe640 1 -- 172.21.15.169:0/3378605529 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d94019d50 con 0x56449f7b3930 2024-03-28T00:57:24.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.633+0000 7f6dae3ec740 1 -- 172.21.15.169:0/3378605529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b3930 msgr2=0x56449f7b8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:24.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.633+0000 7f6dae3ec740 1 --2- 172.21.15.169:0/3378605529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b3930 0x56449f7b8430 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6d940115c0 tx=0x7f6d94011a80 comp rx=0 tx=0).stop 2024-03-28T00:57:24.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.633+0000 7f6dae3ec740 1 -- 172.21.15.169:0/3378605529 shutdown_connections 2024-03-28T00:57:24.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.633+0000 7f6dae3ec740 1 --2- 172.21.15.169:0/3378605529 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b8aa0 0x56449f7baf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.633+0000 7f6dae3ec740 1 --2- 172.21.15.169:0/3378605529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b3930 0x56449f7b8430 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.633+0000 7f6dae3ec740 1 --2- 172.21.15.169:0/3378605529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b2f40 0x56449f7b3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.633+0000 7f6dae3ec740 1 -- 172.21.15.169:0/3378605529 >> 172.21.15.169:0/3378605529 conn(0x56449f7771a0 msgr2=0x56449f7aa4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:24.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.633+0000 7f6dae3ec740 1 -- 172.21.15.169:0/3378605529 shutdown_connections 2024-03-28T00:57:24.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.633+0000 7f6dae3ec740 1 -- 172.21.15.169:0/3378605529 wait complete. 2024-03-28T00:57:24.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.634+0000 7f6dae3ec740 1 -- start start 2024-03-28T00:57:24.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.634+0000 7f6dae3ec740 1 Processor -- start 2024-03-28T00:57:24.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.634+0000 7f6dae3ec740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b2f40 0x56449f7bd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.634+0000 7f6dae3ec740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b3930 0x56449f7bd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.634+0000 7f6dae3ec740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b8aa0 0x56449f7c4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.634+0000 7f6dae3ec740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56449f6d00d0 con 0x56449f7b2f40 2024-03-28T00:57:24.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.634+0000 7f6dae3ec740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x56449f710110 con 0x56449f7b8aa0 2024-03-28T00:57:24.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.634+0000 7f6dae3ec740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56449f7bba50 con 0x56449f7b3930 2024-03-28T00:57:24.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.634+0000 7f6dae3ec740 10 client.0 _ll_register_callbacks cb 0x56449f70be60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:24.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.634+0000 7f6da959b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b2f40 0x56449f7bd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.634+0000 7f6da959b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b2f40 0x56449f7bd2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60758/0 (socket says 172.21.15.169:60758) 2024-03-28T00:57:24.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.634+0000 7f6da959b640 1 -- 172.21.15.169:0/3718348982 learned_addr learned my addr 172.21.15.169:0/3718348982 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:24.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.634+0000 7f6da9d9c640 1 --2- 172.21.15.169:0/3718348982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b8aa0 0x56449f7c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.634+0000 7f6da8d9a640 1 --2- 172.21.15.169:0/3718348982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b3930 0x56449f7bd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.634+0000 7f6da959b640 1 -- 172.21.15.169:0/3718348982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b3930 msgr2=0x56449f7bd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:24.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.634+0000 7f6da959b640 1 --2- 172.21.15.169:0/3718348982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b3930 0x56449f7bd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.634+0000 7f6da959b640 1 -- 172.21.15.169:0/3718348982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b8aa0 msgr2=0x56449f7c4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:24.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.634+0000 7f6da959b640 1 --2- 172.21.15.169:0/3718348982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b8aa0 0x56449f7c4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.634+0000 7f6da959b640 1 -- 172.21.15.169:0/3718348982 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56449f4f07e0 con 0x56449f7b2f40 2024-03-28T00:57:24.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.635+0000 7f6da959b640 1 --2- 172.21.15.169:0/3718348982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b2f40 0x56449f7bd2d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f6d9401b040 tx=0x7f6d94005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44790e7d3d6973f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:24.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.635+0000 7f6d8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:57:24.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.635+0000 7f6d8cff9640 1 -- 172.21.15.169:0/3718348982 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d94004040 con 0x56449f7b2f40 2024-03-28T00:57:24.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.635+0000 7f6d8cff9640 1 -- 172.21.15.169:0/3718348982 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f6d940041e0 con 0x56449f7b2f40 2024-03-28T00:57:24.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.635+0000 7f6dae3ec740 1 -- 172.21.15.169:0/3718348982 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6d9400fcc0 con 0x56449f7b2f40 2024-03-28T00:57:24.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.635+0000 7f6d8cff9640 1 -- 172.21.15.169:0/3718348982 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d94030050 con 0x56449f7b2f40 2024-03-28T00:57:24.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.635+0000 7f6dae3ec740 10 client.15747 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:24.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.635+0000 7f6dae3ec740 1 -- 172.21.15.169:0/3718348982 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56449f550f70 con 0x56449f7b2f40 2024-03-28T00:57:24.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.635+0000 7f6d8cff9640 1 -- 172.21.15.169:0/3718348982 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d9400fcc0 con 0x56449f7b2f40 2024-03-28T00:57:24.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.635+0000 7f6d8cff9640 1 -- 172.21.15.169:0/3718348982 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f6d94005350 con 0x56449f7b2f40 2024-03-28T00:57:24.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.635+0000 7f6dae3ec740 10 client.15747 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:24.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.635+0000 7f6dae3ec740 10 client.15747 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:24.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.635+0000 7f6dae3ec740 20 client.15747 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:24.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.635+0000 7f6dae3ec740 1 --2- 172.21.15.169:0/3718348982 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56449f7c8ea0 0x56449f7cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.635+0000 7f6dae3ec740 4 client.15747 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:57:24.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.635+0000 7f6dae3ec740 1 -- 172.21.15.169:0/3718348982 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56449f7215b0 con 0x56449f7c8ea0 2024-03-28T00:57:24.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.635+0000 7f6da8d9a640 1 --2- 172.21.15.169:0/3718348982 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56449f7c8ea0 0x56449f7cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.636+0000 7f6d8cff9640 10 client.15747 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:24.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.636+0000 7f6da8d9a640 1 --2- 172.21.15.169:0/3718348982 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56449f7c8ea0 0x56449f7cb2b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3818142e8980fcf server_cookie=8ca2ea98052842a in_seq=0 out_seq=0 2024-03-28T00:57:24.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.636+0000 7f6d8cff9640 1 -- 172.21.15.169:0/3718348982 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56449f7215b0 con 0x56449f7c8ea0 2024-03-28T00:57:24.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.636+0000 7f6d8cff9640 10 client.15747 handle_command_reply: tid=0 2024-03-28T00:57:24.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.636+0000 7f6dae3ec740 1 client.15747 shutdown 2024-03-28T00:57:24.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.636+0000 7f6dae3ec740 1 -- 172.21.15.169:0/3718348982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b2f40 msgr2=0x56449f7bd2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:24.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.637+0000 7f6dae3ec740 1 --2- 172.21.15.169:0/3718348982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b2f40 0x56449f7bd2d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f6d9401b040 tx=0x7f6d94005750 comp rx=0 tx=0).stop 2024-03-28T00:57:24.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.637+0000 7f6dae3ec740 1 -- 172.21.15.169:0/3718348982 shutdown_connections 2024-03-28T00:57:24.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.637+0000 7f6dae3ec740 1 --2- 172.21.15.169:0/3718348982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b8aa0 0x56449f7c4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.637+0000 7f6dae3ec740 1 --2- 172.21.15.169:0/3718348982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b3930 0x56449f7bd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.637+0000 7f6dae3ec740 1 --2- 172.21.15.169:0/3718348982 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56449f7c8ea0 0x56449f7cb2b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.637+0000 7f6dae3ec740 1 --2- 172.21.15.169:0/3718348982 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b2f40 0x56449f7bd2d0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.637+0000 7f6d8cff9640 0 client.15747 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:24.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.637+0000 7f6dae3ec740 1 -- 172.21.15.169:0/3718348982 >> 172.21.15.169:0/3718348982 conn(0x56449f7771a0 msgr2=0x56449f7a97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:24.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.637+0000 7f6dae3ec740 1 -- 172.21.15.169:0/3718348982 shutdown_connections 2024-03-28T00:57:24.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.637+0000 7f6dae3ec740 1 -- 172.21.15.169:0/3718348982 wait complete. 2024-03-28T00:57:24.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.637+0000 7f6dae3ec740 20 client.15747 trim_cache size 0 max 16384 2024-03-28T00:57:24.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.653+0000 7f6dae3ec740 1 Processor -- start 2024-03-28T00:57:24.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.653+0000 7f6dae3ec740 1 -- start start 2024-03-28T00:57:24.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.653+0000 7f6dae3ec740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b2f40 0x56449f7a9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.653+0000 7f6dae3ec740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b3930 0x56449f7c4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.653+0000 7f6dae3ec740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b8aa0 0x56449f7bc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.653+0000 7f6dae3ec740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da41aa290 con 0x56449f7b3930 2024-03-28T00:57:24.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.653+0000 7f6dae3ec740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da41a7d90 con 0x56449f7b8aa0 2024-03-28T00:57:24.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.653+0000 7f6dae3ec740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da41a5e30 con 0x56449f7b2f40 2024-03-28T00:57:24.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.653+0000 7f6da8d9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b3930 0x56449f7c4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.653+0000 7f6da8d9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b3930 0x56449f7c4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60772/0 (socket says 172.21.15.169:60772) 2024-03-28T00:57:24.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.653+0000 7f6da8d9a640 1 -- 172.21.15.169:0/3195452776 learned_addr learned my addr 172.21.15.169:0/3195452776 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:24.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.653+0000 7f6da959b640 1 --2- 172.21.15.169:0/3195452776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b2f40 0x56449f7a9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.653+0000 7f6da9d9c640 1 --2- 172.21.15.169:0/3195452776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b8aa0 0x56449f7bc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.653+0000 7f6da8d9a640 1 -- 172.21.15.169:0/3195452776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b2f40 msgr2=0x56449f7a9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:24.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.653+0000 7f6da8d9a640 1 --2- 172.21.15.169:0/3195452776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b2f40 0x56449f7a9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.653+0000 7f6da8d9a640 1 -- 172.21.15.169:0/3195452776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b8aa0 msgr2=0x56449f7bc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:24.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.653+0000 7f6da8d9a640 1 --2- 172.21.15.169:0/3195452776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b8aa0 0x56449f7bc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.653+0000 7f6da8d9a640 1 -- 172.21.15.169:0/3195452776 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56449f3e8170 con 0x56449f7b3930 2024-03-28T00:57:24.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.653+0000 7f6da8d9a640 1 --2- 172.21.15.169:0/3195452776 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b3930 0x56449f7c4770 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6da0000c00 tx=0x7f6da0086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b43f81c299cab5c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:24.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.654+0000 7f6d8effd640 1 -- 172.21.15.169:0/3195452776 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da00a3ec0 con 0x56449f7b3930 2024-03-28T00:57:24.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.654+0000 7f6d8effd640 1 -- 172.21.15.169:0/3195452776 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6da009b7d0 con 0x56449f7b3930 2024-03-28T00:57:24.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.654+0000 7f6d8effd640 1 -- 172.21.15.169:0/3195452776 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da0097e40 con 0x56449f7b3930 2024-03-28T00:57:24.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.654+0000 7f6dae3ec740 1 -- 172.21.15.169:0/3195452776 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b3930 msgr2=0x56449f7c4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:24.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.654+0000 7f6dae3ec740 1 --2- 172.21.15.169:0/3195452776 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b3930 0x56449f7c4770 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6da0000c00 tx=0x7f6da0086040 comp rx=0 tx=0).stop 2024-03-28T00:57:24.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.654+0000 7f6dae3ec740 1 -- 172.21.15.169:0/3195452776 shutdown_connections 2024-03-28T00:57:24.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.654+0000 7f6dae3ec740 1 --2- 172.21.15.169:0/3195452776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b8aa0 0x56449f7bc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.654+0000 7f6dae3ec740 1 --2- 172.21.15.169:0/3195452776 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b3930 0x56449f7c4770 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.654+0000 7f6dae3ec740 1 --2- 172.21.15.169:0/3195452776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b2f40 0x56449f7a9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.654+0000 7f6dae3ec740 1 -- 172.21.15.169:0/3195452776 >> 172.21.15.169:0/3195452776 conn(0x56449f7771a0 msgr2=0x56449f7a8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:24.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.654+0000 7f6dae3ec740 1 -- 172.21.15.169:0/3195452776 shutdown_connections 2024-03-28T00:57:24.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.654+0000 7f6dae3ec740 1 -- 172.21.15.169:0/3195452776 wait complete. 2024-03-28T00:57:24.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.654+0000 7f6dae3ec740 1 -- start start 2024-03-28T00:57:24.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.654+0000 7f6dae3ec740 1 Processor -- start 2024-03-28T00:57:24.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.655+0000 7f6dae3ec740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b2f40 0x56449f7c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.655+0000 7f6dae3ec740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b3930 0x56449f7c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.655+0000 7f6dae3ec740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b8aa0 0x56449f7c3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.655+0000 7f6dae3ec740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da41a6430 con 0x56449f7b8aa0 2024-03-28T00:57:24.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.655+0000 7f6dae3ec740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da41a7730 con 0x56449f7b2f40 2024-03-28T00:57:24.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.655+0000 7f6dae3ec740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da41a6a70 con 0x56449f7b3930 2024-03-28T00:57:24.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.655+0000 7f6dae3ec740 10 client.0 _ll_register_callbacks cb 0x7f6da0009600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:24.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.655+0000 7f6da9d9c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b8aa0 0x56449f7c3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.655+0000 7f6da8d9a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b3930 0x56449f7c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.655+0000 7f6da9d9c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b8aa0 0x56449f7c3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60780/0 (socket says 172.21.15.169:60780) 2024-03-28T00:57:24.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.655+0000 7f6da9d9c640 1 -- 172.21.15.169:0/2193002864 learned_addr learned my addr 172.21.15.169:0/2193002864 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:24.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.655+0000 7f6da959b640 1 --2- 172.21.15.169:0/2193002864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b2f40 0x56449f7c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.655+0000 7f6da9d9c640 1 -- 172.21.15.169:0/2193002864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b3930 msgr2=0x56449f7c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:24.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.655+0000 7f6da9d9c640 1 --2- 172.21.15.169:0/2193002864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b3930 0x56449f7c2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.655+0000 7f6da9d9c640 1 -- 172.21.15.169:0/2193002864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b2f40 msgr2=0x56449f7c2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:24.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.655+0000 7f6da9d9c640 1 --2- 172.21.15.169:0/2193002864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b2f40 0x56449f7c2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.655+0000 7f6da9d9c640 1 -- 172.21.15.169:0/2193002864 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56449f7c4240 con 0x56449f7b8aa0 2024-03-28T00:57:24.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.655+0000 7f6da8d9a640 1 --2- 172.21.15.169:0/2193002864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b3930 0x56449f7c2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:24.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.655+0000 7f6da959b640 1 --2- 172.21.15.169:0/2193002864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b2f40 0x56449f7c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:24.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.655+0000 7f6da9d9c640 1 --2- 172.21.15.169:0/2193002864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b8aa0 0x56449f7c3c50 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c036040 tx=0x7f6d9c009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdcb0b34c3702c31 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:24.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.655+0000 7f6d8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:57:24.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.656+0000 7f6d8cff9640 1 -- 172.21.15.169:0/2193002864 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c055030 con 0x56449f7b8aa0 2024-03-28T00:57:24.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.656+0000 7f6d8cff9640 1 -- 172.21.15.169:0/2193002864 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f6d9c04c040 con 0x56449f7b8aa0 2024-03-28T00:57:24.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.656+0000 7f6dae3ec740 1 -- 172.21.15.169:0/2193002864 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6da009b7d0 con 0x56449f7b8aa0 2024-03-28T00:57:24.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.656+0000 7f6d8cff9640 1 -- 172.21.15.169:0/2193002864 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d9c062050 con 0x56449f7b8aa0 2024-03-28T00:57:24.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.656+0000 7f6dae3ec740 10 client.15753 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:24.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.656+0000 7f6dae3ec740 1 -- 172.21.15.169:0/2193002864 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56449f824f30 con 0x56449f7b8aa0 2024-03-28T00:57:24.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.656+0000 7f6d8cff9640 1 -- 172.21.15.169:0/2193002864 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6da009b7d0 con 0x56449f7b8aa0 2024-03-28T00:57:24.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.656+0000 7f6d8cff9640 1 -- 172.21.15.169:0/2193002864 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f6d9c034850 con 0x56449f7b8aa0 2024-03-28T00:57:24.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.656+0000 7f6dae3ec740 10 client.15753 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:24.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.656+0000 7f6dae3ec740 10 client.15753 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:24.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.656+0000 7f6dae3ec740 20 client.15753 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:24.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.656+0000 7f6dae3ec740 1 --2- 172.21.15.169:0/2193002864 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56449f825910 0x56449f827d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.656+0000 7f6dae3ec740 4 client.15753 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T00:57:24.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.656+0000 7f6dae3ec740 1 -- 172.21.15.169:0/2193002864 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56449f71b900 con 0x56449f825910 2024-03-28T00:57:24.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.656+0000 7f6da959b640 1 --2- 172.21.15.169:0/2193002864 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56449f825910 0x56449f827d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.657+0000 7f6d8cff9640 10 client.15753 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:24.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.657+0000 7f6da959b640 1 --2- 172.21.15.169:0/2193002864 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56449f825910 0x56449f827d30 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70a87160c2b724d8 server_cookie=d9400b179233aaa0 in_seq=0 out_seq=0 2024-03-28T00:57:24.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.657+0000 7f6d8cff9640 1 -- 172.21.15.169:0/2193002864 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56449f71b900 con 0x56449f825910 2024-03-28T00:57:24.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.657+0000 7f6d8cff9640 10 client.15753 handle_command_reply: tid=0 2024-03-28T00:57:24.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.657+0000 7f6dae3ec740 1 client.15753 shutdown 2024-03-28T00:57:24.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.658+0000 7f6dae3ec740 1 -- 172.21.15.169:0/2193002864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b8aa0 msgr2=0x56449f7c3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:24.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.658+0000 7f6dae3ec740 1 --2- 172.21.15.169:0/2193002864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b8aa0 0x56449f7c3c50 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c036040 tx=0x7f6d9c009500 comp rx=0 tx=0).stop 2024-03-28T00:57:24.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.658+0000 7f6dae3ec740 1 -- 172.21.15.169:0/2193002864 shutdown_connections 2024-03-28T00:57:24.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.658+0000 7f6dae3ec740 1 --2- 172.21.15.169:0/2193002864 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56449f825910 0x56449f827d30 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.658+0000 7f6dae3ec740 1 --2- 172.21.15.169:0/2193002864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56449f7b8aa0 0x56449f7c3c50 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.658+0000 7f6dae3ec740 1 --2- 172.21.15.169:0/2193002864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56449f7b3930 0x56449f7c2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.658+0000 7f6dae3ec740 1 --2- 172.21.15.169:0/2193002864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56449f7b2f40 0x56449f7c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.658+0000 7f6d8cff9640 0 client.15753 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:24.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.658+0000 7f6dae3ec740 1 -- 172.21.15.169:0/2193002864 >> 172.21.15.169:0/2193002864 conn(0x56449f7771a0 msgr2=0x56449f7a97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:24.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.658+0000 7f6dae3ec740 1 -- 172.21.15.169:0/2193002864 shutdown_connections 2024-03-28T00:57:24.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.658+0000 7f6dae3ec740 1 -- 172.21.15.169:0/2193002864 wait complete. 2024-03-28T00:57:24.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.658+0000 7f6dae3ec740 20 client.15753 trim_cache size 0 max 16384 2024-03-28T00:57:24.672 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:57:24.672 INFO:teuthology.orchestra.run.smithi169.stdout: "return_code": 0, 2024-03-28T00:57:24.672 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_tag": "43b0d6b7-4eb2-48a3-8af9-1e7d56051cc3", 2024-03-28T00:57:24.672 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": "asynchronous" 2024-03-28T00:57:24.673 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:57:24.675 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.658+0000 7f6daa59d640 1 -- 172.21.15.169:0/3346558688 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6d900a2890 msgr2=0x7f6d900a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:24.675 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.658+0000 7f6daa59d640 1 --2- 172.21.15.169:0/3346558688 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6d900a2890 0x7f6d900a4d50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c004ab0 tx=0x7f6d9c004a40 comp rx=0 tx=0).stop 2024-03-28T00:57:24.675 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.658+0000 7f6daa59d640 1 -- 172.21.15.169:0/3346558688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6da4104930 msgr2=0x7f6da4197d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:24.675 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.658+0000 7f6daa59d640 1 --2- 172.21.15.169:0/3346558688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6da4104930 0x7f6da4197d60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6da000e9f0 tx=0x7f6da000eeb0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.675 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.658+0000 7f6daa59d640 1 -- 172.21.15.169:0/3346558688 shutdown_connections 2024-03-28T00:57:24.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.658+0000 7f6daa59d640 1 --2- 172.21.15.169:0/3346558688 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6d900a2890 0x7f6d900a4d50 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.658+0000 7f6daa59d640 1 --2- 172.21.15.169:0/3346558688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6da4107260 0x7f6da419edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.658+0000 7f6daa59d640 1 --2- 172.21.15.169:0/3346558688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6da4104930 0x7f6da4197d60 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.658+0000 7f6daa59d640 1 --2- 172.21.15.169:0/3346558688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6da4102000 0x7f6da4197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.659+0000 7f6daa59d640 1 -- 172.21.15.169:0/3346558688 >> 172.21.15.169:0/3346558688 conn(0x7f6da40fba10 msgr2=0x7f6da4102900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:24.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.659+0000 7f6daa59d640 1 -- 172.21.15.169:0/3346558688 shutdown_connections 2024-03-28T00:57:24.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.659+0000 7f6daa59d640 1 -- 172.21.15.169:0/3346558688 wait complete. 2024-03-28T00:57:24.677 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T00:57:24.677 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T00:57:24.783 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.778+0000 7f1d3f4f8640 1 -- 172.21.15.169:0/2363648873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d381028a0 msgr2=0x7f1d38102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:24.783 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.778+0000 7f1d3f4f8640 1 --2- 172.21.15.169:0/2363648873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d381028a0 0x7f1d38102d20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1d34009900 tx=0x7f1d3402bc30 comp rx=0 tx=0).stop 2024-03-28T00:57:24.783 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.778+0000 7f1d3f4f8640 1 -- 172.21.15.169:0/2363648873 shutdown_connections 2024-03-28T00:57:24.783 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.778+0000 7f1d3f4f8640 1 --2- 172.21.15.169:0/2363648873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d38103260 0x7f1d38109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.783 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.778+0000 7f1d3f4f8640 1 --2- 172.21.15.169:0/2363648873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d381028a0 0x7f1d38102d20 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.783 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.778+0000 7f1d3f4f8640 1 --2- 172.21.15.169:0/2363648873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d381016b0 0x7f1d38101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.778+0000 7f1d3f4f8640 1 -- 172.21.15.169:0/2363648873 >> 172.21.15.169:0/2363648873 conn(0x7f1d380fcc40 msgr2=0x7f1d380ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:24.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.778+0000 7f1d3f4f8640 1 -- 172.21.15.169:0/2363648873 shutdown_connections 2024-03-28T00:57:24.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.778+0000 7f1d3f4f8640 1 -- 172.21.15.169:0/2363648873 wait complete. 2024-03-28T00:57:24.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.778+0000 7f1d3f4f8640 1 Processor -- start 2024-03-28T00:57:24.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.778+0000 7f1d3f4f8640 1 -- start start 2024-03-28T00:57:24.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d3f4f8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d381016b0 0x7f1d381979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d3f4f8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d381028a0 0x7f1d38197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d3f4f8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d38103260 0x7f1d3819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d3f4f8640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3810d330 con 0x7f1d381016b0 2024-03-28T00:57:24.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d3f4f8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3810d1b0 con 0x7f1d38103260 2024-03-28T00:57:24.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d3f4f8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3810d4b0 con 0x7f1d381028a0 2024-03-28T00:57:24.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d3d294640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d381016b0 0x7f1d381979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d3d294640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d381016b0 0x7f1d381979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60788/0 (socket says 172.21.15.169:60788) 2024-03-28T00:57:24.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d3ca93640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d381028a0 0x7f1d38197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d3d294640 1 -- 172.21.15.169:0/4187281567 learned_addr learned my addr 172.21.15.169:0/4187281567 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:24.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d3da95640 1 --2- 172.21.15.169:0/4187281567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d38103260 0x7f1d3819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d3d294640 1 -- 172.21.15.169:0/4187281567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d381028a0 msgr2=0x7f1d38197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:24.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d3d294640 1 --2- 172.21.15.169:0/4187281567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d381028a0 0x7f1d38197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d3d294640 1 -- 172.21.15.169:0/4187281567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d38103260 msgr2=0x7f1d3819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:24.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d3d294640 1 --2- 172.21.15.169:0/4187281567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d38103260 0x7f1d3819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d3d294640 1 -- 172.21.15.169:0/4187281567 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3819f600 con 0x7f1d381016b0 2024-03-28T00:57:24.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d3d294640 1 --2- 172.21.15.169:0/4187281567 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d381016b0 0x7f1d381979e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800e9c0 tx=0x7f1d2800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:24.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d2e7fc640 1 -- 172.21.15.169:0/4187281567 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2800cd80 con 0x7f1d381016b0 2024-03-28T00:57:24.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d3f4f8640 1 -- 172.21.15.169:0/4187281567 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d3819f8f0 con 0x7f1d381016b0 2024-03-28T00:57:24.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d3f4f8640 1 -- 172.21.15.169:0/4187281567 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d3819fe10 con 0x7f1d381016b0 2024-03-28T00:57:24.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d2e7fc640 1 -- 172.21.15.169:0/4187281567 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d280035f0 con 0x7f1d381016b0 2024-03-28T00:57:24.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.779+0000 7f1d2e7fc640 1 -- 172.21.15.169:0/4187281567 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d28010670 con 0x7f1d381016b0 2024-03-28T00:57:24.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.781+0000 7f1d2e7fc640 1 -- 172.21.15.169:0/4187281567 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f1d280037a0 con 0x7f1d381016b0 2024-03-28T00:57:24.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.782+0000 7f1d2e7fc640 1 --2- 172.21.15.169:0/4187281567 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1d240a2890 0x7f1d240a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.782+0000 7f1d421c1740 1 Processor -- start 2024-03-28T00:57:24.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.782+0000 7f1d421c1740 1 -- start start 2024-03-28T00:57:24.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.782+0000 7f1d3ca93640 1 --2- 172.21.15.169:0/4187281567 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1d240a2890 0x7f1d240a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.782+0000 7f1d2e7fc640 1 -- 172.21.15.169:0/4187281567 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f1d280cf390 con 0x7f1d381016b0 2024-03-28T00:57:24.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.782+0000 7f1d421c1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac232da0 0x55e9ac2331c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.782+0000 7f1d421c1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac233790 0x55e9ac238290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.782+0000 7f1d421c1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac238900 0x55e9ac23ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.782+0000 7f1d421c1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9ac1983d0 con 0x55e9ac233790 2024-03-28T00:57:24.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.782+0000 7f1d421c1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9ac140ad0 con 0x55e9ac238900 2024-03-28T00:57:24.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.782+0000 7f1d421c1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9ac194470 con 0x55e9ac232da0 2024-03-28T00:57:24.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.782+0000 7f1d3ca93640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac233790 0x55e9ac238290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.783+0000 7f1d3d294640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac232da0 0x55e9ac2331c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.783+0000 7f1d3ca93640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac233790 0x55e9ac238290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60796/0 (socket says 172.21.15.169:60796) 2024-03-28T00:57:24.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.783+0000 7f1d3ca93640 1 -- 172.21.15.169:0/327835020 learned_addr learned my addr 172.21.15.169:0/327835020 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:24.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.783+0000 7f1d3da95640 1 --2- 172.21.15.169:0/327835020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac238900 0x55e9ac23ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.783+0000 7f1d3da95640 1 -- 172.21.15.169:0/327835020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac232da0 msgr2=0x55e9ac2331c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:24.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.783+0000 7f1d3da95640 1 --2- 172.21.15.169:0/327835020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac232da0 0x55e9ac2331c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.783+0000 7f1d3da95640 1 -- 172.21.15.169:0/327835020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac233790 msgr2=0x55e9ac238290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:24.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.783+0000 7f1d3da95640 1 --2- 172.21.15.169:0/327835020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac233790 0x55e9ac238290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.783+0000 7f1d3da95640 1 -- 172.21.15.169:0/327835020 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e9abf767d0 con 0x55e9ac238900 2024-03-28T00:57:24.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.783+0000 7f1d3ca93640 1 --2- 172.21.15.169:0/4187281567 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1d240a2890 0x7f1d240a4d50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1d3402c710 tx=0x7f1d3402cbc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:24.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.783+0000 7f1d3da95640 1 --2- 172.21.15.169:0/327835020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac238900 0x55e9ac23ad90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1d30009f00 tx=0x7f1d30011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f02232d0e9fdbb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:24.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.783+0000 7f1d237fe640 1 -- 172.21.15.169:0/327835020 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d30004d20 con 0x55e9ac238900 2024-03-28T00:57:24.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.783+0000 7f1d237fe640 1 -- 172.21.15.169:0/327835020 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d30004050 con 0x55e9ac238900 2024-03-28T00:57:24.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.783+0000 7f1d237fe640 1 -- 172.21.15.169:0/327835020 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d30005260 con 0x55e9ac238900 2024-03-28T00:57:24.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.784+0000 7f1d421c1740 1 -- 172.21.15.169:0/327835020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac238900 msgr2=0x55e9ac23ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:24.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.784+0000 7f1d421c1740 1 --2- 172.21.15.169:0/327835020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac238900 0x55e9ac23ad90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1d30009f00 tx=0x7f1d30011810 comp rx=0 tx=0).stop 2024-03-28T00:57:24.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.784+0000 7f1d421c1740 1 -- 172.21.15.169:0/327835020 shutdown_connections 2024-03-28T00:57:24.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.784+0000 7f1d421c1740 1 --2- 172.21.15.169:0/327835020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac238900 0x55e9ac23ad90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.784+0000 7f1d421c1740 1 --2- 172.21.15.169:0/327835020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac233790 0x55e9ac238290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.784+0000 7f1d421c1740 1 --2- 172.21.15.169:0/327835020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac232da0 0x55e9ac2331c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.784+0000 7f1d421c1740 1 -- 172.21.15.169:0/327835020 >> 172.21.15.169:0/327835020 conn(0x55e9ac22e310 msgr2=0x55e9ac22e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:24.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.784+0000 7f1d421c1740 1 -- 172.21.15.169:0/327835020 shutdown_connections 2024-03-28T00:57:24.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.784+0000 7f1d421c1740 1 -- 172.21.15.169:0/327835020 wait complete. 2024-03-28T00:57:24.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.784+0000 7f1d421c1740 1 -- start start 2024-03-28T00:57:24.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.784+0000 7f1d421c1740 1 Processor -- start 2024-03-28T00:57:24.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d421c1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac232da0 0x55e9ac23d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d421c1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac233790 0x55e9ac23d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d421c1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac238900 0x55e9ac2446f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d421c1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9ac23b8b0 con 0x55e9ac232da0 2024-03-28T00:57:24.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d421c1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9ac154060 con 0x55e9ac233790 2024-03-28T00:57:24.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d421c1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9ac23ba30 con 0x55e9ac238900 2024-03-28T00:57:24.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d3d294640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac232da0 0x55e9ac23d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d421c1740 10 client.0 _ll_register_callbacks cb 0x55e9ac1366a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:24.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d3ca93640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac233790 0x55e9ac23d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d3d294640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac232da0 0x55e9ac23d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60798/0 (socket says 172.21.15.169:60798) 2024-03-28T00:57:24.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d3da95640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac238900 0x55e9ac2446f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d3d294640 1 -- 172.21.15.169:0/864801380 learned_addr learned my addr 172.21.15.169:0/864801380 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:24.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d3ca93640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac233790 0x55e9ac23d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:58004/0 (socket says 172.21.15.169:58004) 2024-03-28T00:57:24.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d3da95640 1 -- 172.21.15.169:0/864801380 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac233790 msgr2=0x55e9ac23d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:24.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d3da95640 1 --2- 172.21.15.169:0/864801380 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac233790 0x55e9ac23d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d3da95640 1 -- 172.21.15.169:0/864801380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac232da0 msgr2=0x55e9ac23d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:24.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d3da95640 1 --2- 172.21.15.169:0/864801380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac232da0 0x55e9ac23d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d3da95640 1 -- 172.21.15.169:0/864801380 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e9abeac3f0 con 0x55e9ac238900 2024-03-28T00:57:24.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d3d294640 1 --2- 172.21.15.169:0/864801380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac232da0 0x55e9ac23d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:24.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d3ca93640 1 --2- 172.21.15.169:0/864801380 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac233790 0x55e9ac23d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:24.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d3da95640 1 --2- 172.21.15.169:0/864801380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac238900 0x55e9ac2446f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1d3001b040 tx=0x7f1d30003c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=658c7e493f3d8c82 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:24.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d20ff9640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:57:24.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d20ff9640 1 -- 172.21.15.169:0/864801380 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d30029020 con 0x55e9ac238900 2024-03-28T00:57:24.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d421c1740 1 -- 172.21.15.169:0/864801380 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1d30004050 con 0x55e9ac238900 2024-03-28T00:57:24.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.785+0000 7f1d20ff9640 1 -- 172.21.15.169:0/864801380 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f1d30004610 con 0x55e9ac238900 2024-03-28T00:57:24.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.786+0000 7f1d20ff9640 1 -- 172.21.15.169:0/864801380 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d30034050 con 0x55e9ac238900 2024-03-28T00:57:24.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.786+0000 7f1d421c1740 10 client.25561 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:24.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.786+0000 7f1d421c1740 1 -- 172.21.15.169:0/864801380 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55e9abe6c0f0 con 0x55e9ac238900 2024-03-28T00:57:24.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.786+0000 7f1d20ff9640 1 -- 172.21.15.169:0/864801380 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d30004050 con 0x55e9ac238900 2024-03-28T00:57:24.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.786+0000 7f1d20ff9640 1 -- 172.21.15.169:0/864801380 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f1d300196d0 con 0x55e9ac238900 2024-03-28T00:57:24.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.786+0000 7f1d421c1740 10 client.25561 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:24.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.786+0000 7f1d421c1740 10 client.25561 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:24.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.786+0000 7f1d421c1740 20 client.25561 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:24.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.786+0000 7f1d421c1740 1 --2- 172.21.15.169:0/864801380 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e9ac248a70 0x55e9ac24ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.786+0000 7f1d421c1740 4 client.25561 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:57:24.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.786+0000 7f1d421c1740 1 -- 172.21.15.169:0/864801380 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e9ac1a5910 con 0x55e9ac248a70 2024-03-28T00:57:24.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.786+0000 7f1d3d294640 1 --2- 172.21.15.169:0/864801380 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e9ac248a70 0x55e9ac24ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.786+0000 7f1d20ff9640 10 client.25561 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:24.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.786+0000 7f1d3d294640 1 --2- 172.21.15.169:0/864801380 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e9ac248a70 0x55e9ac24ae80 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e28a891a6037aa06 server_cookie=40a9ebd924cf80a7 in_seq=0 out_seq=0 2024-03-28T00:57:24.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.787+0000 7f1d20ff9640 1 -- 172.21.15.169:0/864801380 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e9ac1a5910 con 0x55e9ac248a70 2024-03-28T00:57:24.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.787+0000 7f1d20ff9640 10 client.25561 handle_command_reply: tid=0 2024-03-28T00:57:24.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.787+0000 7f1d421c1740 1 client.25561 shutdown 2024-03-28T00:57:24.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.787+0000 7f1d421c1740 1 -- 172.21.15.169:0/864801380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac238900 msgr2=0x55e9ac2446f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:24.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.787+0000 7f1d421c1740 1 --2- 172.21.15.169:0/864801380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac238900 0x55e9ac2446f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1d3001b040 tx=0x7f1d30003c60 comp rx=0 tx=0).stop 2024-03-28T00:57:24.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.787+0000 7f1d421c1740 1 -- 172.21.15.169:0/864801380 shutdown_connections 2024-03-28T00:57:24.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.787+0000 7f1d421c1740 1 --2- 172.21.15.169:0/864801380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac238900 0x55e9ac2446f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.787+0000 7f1d421c1740 1 --2- 172.21.15.169:0/864801380 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac233790 0x55e9ac23d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.787+0000 7f1d421c1740 1 --2- 172.21.15.169:0/864801380 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e9ac248a70 0x55e9ac24ae80 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.787+0000 7f1d421c1740 1 --2- 172.21.15.169:0/864801380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac232da0 0x55e9ac23d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.787+0000 7f1d20ff9640 0 client.25561 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:24.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.787+0000 7f1d421c1740 1 -- 172.21.15.169:0/864801380 >> 172.21.15.169:0/864801380 conn(0x55e9ac22e310 msgr2=0x55e9ac22d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:24.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.787+0000 7f1d421c1740 1 -- 172.21.15.169:0/864801380 shutdown_connections 2024-03-28T00:57:24.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.787+0000 7f1d421c1740 1 -- 172.21.15.169:0/864801380 wait complete. 2024-03-28T00:57:24.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.787+0000 7f1d421c1740 20 client.25561 trim_cache size 0 max 16384 2024-03-28T00:57:24.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.803+0000 7f1d421c1740 1 Processor -- start 2024-03-28T00:57:24.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.803+0000 7f1d421c1740 1 -- start start 2024-03-28T00:57:24.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.803+0000 7f1d421c1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac232da0 0x55e9ac2382a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.803+0000 7f1d421c1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac233790 0x55e9ac2446b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.803+0000 7f1d421c1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac238900 0x55e9ac22cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.803+0000 7f1d421c1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d381ac6f0 con 0x55e9ac232da0 2024-03-28T00:57:24.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.803+0000 7f1d421c1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d381aa1f0 con 0x55e9ac238900 2024-03-28T00:57:24.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.803+0000 7f1d421c1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d381a8290 con 0x55e9ac233790 2024-03-28T00:57:24.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.804+0000 7f1d3d294640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac232da0 0x55e9ac2382a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.804+0000 7f1d3d294640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac232da0 0x55e9ac2382a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60804/0 (socket says 172.21.15.169:60804) 2024-03-28T00:57:24.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.804+0000 7f1d3d294640 1 -- 172.21.15.169:0/784332658 learned_addr learned my addr 172.21.15.169:0/784332658 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:24.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.804+0000 7f1d3da95640 1 --2- 172.21.15.169:0/784332658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac238900 0x55e9ac22cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.804+0000 7f1d3ca93640 1 --2- 172.21.15.169:0/784332658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac233790 0x55e9ac2446b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.804+0000 7f1d3d294640 1 -- 172.21.15.169:0/784332658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac233790 msgr2=0x55e9ac2446b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:24.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.804+0000 7f1d3d294640 1 --2- 172.21.15.169:0/784332658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac233790 0x55e9ac2446b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.804+0000 7f1d3d294640 1 -- 172.21.15.169:0/784332658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac238900 msgr2=0x55e9ac22cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:24.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.804+0000 7f1d3d294640 1 --2- 172.21.15.169:0/784332658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac238900 0x55e9ac22cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.804+0000 7f1d3d294640 1 -- 172.21.15.169:0/784332658 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e9ac233b70 con 0x55e9ac232da0 2024-03-28T00:57:24.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.804+0000 7f1d3d294640 1 --2- 172.21.15.169:0/784332658 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac232da0 0x55e9ac2382a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1d28097c90 tx=0x7f1d280cf040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf57634927147345 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:24.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.804+0000 7f1d22ffd640 1 -- 172.21.15.169:0/784332658 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d280a3fd0 con 0x55e9ac232da0 2024-03-28T00:57:24.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.804+0000 7f1d22ffd640 1 -- 172.21.15.169:0/784332658 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d28099b30 con 0x55e9ac232da0 2024-03-28T00:57:24.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.804+0000 7f1d22ffd640 1 -- 172.21.15.169:0/784332658 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d28003af0 con 0x55e9ac232da0 2024-03-28T00:57:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.805+0000 7f1d421c1740 1 -- 172.21.15.169:0/784332658 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac232da0 msgr2=0x55e9ac2382a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.805+0000 7f1d421c1740 1 --2- 172.21.15.169:0/784332658 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac232da0 0x55e9ac2382a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1d28097c90 tx=0x7f1d280cf040 comp rx=0 tx=0).stop 2024-03-28T00:57:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.805+0000 7f1d421c1740 1 -- 172.21.15.169:0/784332658 shutdown_connections 2024-03-28T00:57:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.805+0000 7f1d421c1740 1 --2- 172.21.15.169:0/784332658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac238900 0x55e9ac22cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.805+0000 7f1d421c1740 1 --2- 172.21.15.169:0/784332658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac233790 0x55e9ac2446b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.805+0000 7f1d421c1740 1 --2- 172.21.15.169:0/784332658 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac232da0 0x55e9ac2382a0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.805+0000 7f1d421c1740 1 -- 172.21.15.169:0/784332658 >> 172.21.15.169:0/784332658 conn(0x55e9ac22e310 msgr2=0x55e9ac23d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.805+0000 7f1d421c1740 1 -- 172.21.15.169:0/784332658 shutdown_connections 2024-03-28T00:57:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.805+0000 7f1d421c1740 1 -- 172.21.15.169:0/784332658 wait complete. 2024-03-28T00:57:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.805+0000 7f1d421c1740 1 -- start start 2024-03-28T00:57:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.805+0000 7f1d421c1740 1 Processor -- start 2024-03-28T00:57:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.805+0000 7f1d421c1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac232da0 0x55e9ac2412f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.805+0000 7f1d421c1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac233790 0x55e9ac241830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.805+0000 7f1d421c1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac238900 0x55e9ac2a2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.805+0000 7f1d421c1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d381a8890 con 0x55e9ac233790 2024-03-28T00:57:24.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.805+0000 7f1d421c1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d381a9b90 con 0x55e9ac232da0 2024-03-28T00:57:24.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.805+0000 7f1d421c1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d381a8ed0 con 0x55e9ac238900 2024-03-28T00:57:24.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.806+0000 7f1d421c1740 10 client.0 _ll_register_callbacks cb 0x7f1d28005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:24.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.806+0000 7f1d3ca93640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac233790 0x55e9ac241830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.806+0000 7f1d3d294640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac232da0 0x55e9ac2412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.806+0000 7f1d3ca93640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac233790 0x55e9ac241830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60818/0 (socket says 172.21.15.169:60818) 2024-03-28T00:57:24.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.806+0000 7f1d3ca93640 1 -- 172.21.15.169:0/429762136 learned_addr learned my addr 172.21.15.169:0/429762136 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:24.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.806+0000 7f1d3da95640 1 --2- 172.21.15.169:0/429762136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac238900 0x55e9ac2a2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.806+0000 7f1d3ca93640 1 -- 172.21.15.169:0/429762136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac238900 msgr2=0x55e9ac2a2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:24.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.806+0000 7f1d3ca93640 1 --2- 172.21.15.169:0/429762136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac238900 0x55e9ac2a2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.806+0000 7f1d3ca93640 1 -- 172.21.15.169:0/429762136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac232da0 msgr2=0x55e9ac2412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:24.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.806+0000 7f1d3ca93640 1 --2- 172.21.15.169:0/429762136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac232da0 0x55e9ac2412f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.806+0000 7f1d3ca93640 1 -- 172.21.15.169:0/429762136 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e9ac244900 con 0x55e9ac233790 2024-03-28T00:57:24.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.806+0000 7f1d3da95640 1 --2- 172.21.15.169:0/429762136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac238900 0x55e9ac2a2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:57:24.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.806+0000 7f1d3d294640 1 --2- 172.21.15.169:0/429762136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac232da0 0x55e9ac2412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:24.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.806+0000 7f1d3ca93640 1 --2- 172.21.15.169:0/429762136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac233790 0x55e9ac241830 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f1d34009230 tx=0x7f1d34044040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f29ce208e907bc99 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:24.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.806+0000 7f1d20ff9640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:57:24.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.806+0000 7f1d20ff9640 1 -- 172.21.15.169:0/429762136 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d34004c50 con 0x55e9ac233790 2024-03-28T00:57:24.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.806+0000 7f1d421c1740 1 -- 172.21.15.169:0/429762136 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1d28099b30 con 0x55e9ac233790 2024-03-28T00:57:24.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.806+0000 7f1d20ff9640 1 -- 172.21.15.169:0/429762136 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f1d34004df0 con 0x55e9ac233790 2024-03-28T00:57:24.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.806+0000 7f1d20ff9640 1 -- 172.21.15.169:0/429762136 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d3405e050 con 0x55e9ac233790 2024-03-28T00:57:24.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.806+0000 7f1d421c1740 10 client.15771 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:24.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.806+0000 7f1d421c1740 1 -- 172.21.15.169:0/429762136 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55e9ac2a38d0 con 0x55e9ac233790 2024-03-28T00:57:24.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.807+0000 7f1d20ff9640 1 -- 172.21.15.169:0/429762136 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d28099b30 con 0x55e9ac233790 2024-03-28T00:57:24.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.807+0000 7f1d20ff9640 1 -- 172.21.15.169:0/429762136 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f1d34034600 con 0x55e9ac233790 2024-03-28T00:57:24.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.807+0000 7f1d421c1740 10 client.15771 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:24.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.807+0000 7f1d421c1740 10 client.15771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:24.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.807+0000 7f1d421c1740 20 client.15771 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:24.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.807+0000 7f1d421c1740 1 --2- 172.21.15.169:0/429762136 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e9ac2a4300 0x55e9ac2a6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:24.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.807+0000 7f1d421c1740 4 client.15771 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T00:57:24.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.807+0000 7f1d421c1740 1 -- 172.21.15.169:0/429762136 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e9ac19fc60 con 0x55e9ac2a4300 2024-03-28T00:57:24.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.807+0000 7f1d3d294640 1 --2- 172.21.15.169:0/429762136 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e9ac2a4300 0x55e9ac2a6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:24.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.807+0000 7f1d3d294640 1 --2- 172.21.15.169:0/429762136 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e9ac2a4300 0x55e9ac2a6720 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c9b03b597b69a33 server_cookie=24e94bc78d37a754 in_seq=0 out_seq=0 2024-03-28T00:57:24.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.807+0000 7f1d20ff9640 10 client.15771 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:24.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.807+0000 7f1d20ff9640 1 -- 172.21.15.169:0/429762136 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55e9ac19fc60 con 0x55e9ac2a4300 2024-03-28T00:57:24.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.807+0000 7f1d20ff9640 10 client.15771 handle_command_reply: tid=0 2024-03-28T00:57:24.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.807+0000 7f1d421c1740 1 client.15771 shutdown 2024-03-28T00:57:24.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.807+0000 7f1d421c1740 1 -- 172.21.15.169:0/429762136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac233790 msgr2=0x55e9ac241830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:24.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.807+0000 7f1d421c1740 1 --2- 172.21.15.169:0/429762136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac233790 0x55e9ac241830 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f1d34009230 tx=0x7f1d34044040 comp rx=0 tx=0).stop 2024-03-28T00:57:24.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d421c1740 1 -- 172.21.15.169:0/429762136 shutdown_connections 2024-03-28T00:57:24.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d421c1740 1 --2- 172.21.15.169:0/429762136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9ac238900 0x55e9ac2a2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d421c1740 1 --2- 172.21.15.169:0/429762136 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e9ac2a4300 0x55e9ac2a6720 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d421c1740 1 --2- 172.21.15.169:0/429762136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e9ac233790 0x55e9ac241830 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d20ff9640 0 client.15771 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:24.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d421c1740 1 --2- 172.21.15.169:0/429762136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9ac232da0 0x55e9ac2412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d421c1740 1 -- 172.21.15.169:0/429762136 >> 172.21.15.169:0/429762136 conn(0x55e9ac22e310 msgr2=0x55e9ac231570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:24.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d421c1740 1 -- 172.21.15.169:0/429762136 shutdown_connections 2024-03-28T00:57:24.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d421c1740 1 -- 172.21.15.169:0/429762136 wait complete. 2024-03-28T00:57:24.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d421c1740 20 client.15771 trim_cache size 0 max 16384 2024-03-28T00:57:24.818 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:57:24.818 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T00:57:24.818 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": { 2024-03-28T00:57:24.819 INFO:teuthology.orchestra.run.smithi169.stdout: "43b0d6b7-4eb2-48a3-8af9-1e7d56051cc3": { 2024-03-28T00:57:24.819 INFO:teuthology.orchestra.run.smithi169.stdout: "path": "/", 2024-03-28T00:57:24.819 INFO:teuthology.orchestra.run.smithi169.stdout: "tag": "43b0d6b7-4eb2-48a3-8af9-1e7d56051cc3", 2024-03-28T00:57:24.819 INFO:teuthology.orchestra.run.smithi169.stdout: "options": "recursive,force" 2024-03-28T00:57:24.819 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T00:57:24.819 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T00:57:24.819 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:57:24.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d3f4f8640 1 -- 172.21.15.169:0/4187281567 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1d240a2890 msgr2=0x7f1d240a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:24.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d3f4f8640 1 --2- 172.21.15.169:0/4187281567 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1d240a2890 0x7f1d240a4d50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1d3402c710 tx=0x7f1d3402cbc0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d3f4f8640 1 -- 172.21.15.169:0/4187281567 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d381016b0 msgr2=0x7f1d381979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:24.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d3f4f8640 1 --2- 172.21.15.169:0/4187281567 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d381016b0 0x7f1d381979e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800e9c0 tx=0x7f1d2800ee80 comp rx=0 tx=0).stop 2024-03-28T00:57:24.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d3f4f8640 1 -- 172.21.15.169:0/4187281567 shutdown_connections 2024-03-28T00:57:24.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d3f4f8640 1 --2- 172.21.15.169:0/4187281567 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1d240a2890 0x7f1d240a4d50 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d3f4f8640 1 --2- 172.21.15.169:0/4187281567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d38103260 0x7f1d3819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d3f4f8640 1 --2- 172.21.15.169:0/4187281567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d381028a0 0x7f1d38197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d3f4f8640 1 --2- 172.21.15.169:0/4187281567 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d381016b0 0x7f1d381979e0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:24.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d3f4f8640 1 -- 172.21.15.169:0/4187281567 >> 172.21.15.169:0/4187281567 conn(0x7f1d380fcc40 msgr2=0x7f1d38105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:24.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d3f4f8640 1 -- 172.21.15.169:0/4187281567 shutdown_connections 2024-03-28T00:57:24.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:24.808+0000 7f1d3f4f8640 1 -- 172.21.15.169:0/4187281567 wait complete. 2024-03-28T00:57:24.822 INFO:tasks.cephfs.filesystem:scrub status for tag:43b0d6b7-4eb2-48a3-8af9-1e7d56051cc3 - {'path': '/', 'tag': '43b0d6b7-4eb2-48a3-8af9-1e7d56051cc3', 'options': 'recursive,force'} 2024-03-28T00:57:25.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:25 smithi169 ceph-mon[32274]: pgmap v378: 129 pgs: 129 active+clean; 665 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-28T00:57:25.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:25 smithi169 ceph-mon[32274]: scrub queued for path: / 2024-03-28T00:57:25.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:25 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T00:57:25.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:25 smithi169 ceph-mon[32274]: scrub summary: active paths [/] 2024-03-28T00:57:25.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:25 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T00:57:25.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:25 smithi174 ceph-mon[36060]: pgmap v378: 129 pgs: 129 active+clean; 665 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-28T00:57:25.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:25 smithi174 ceph-mon[36060]: scrub queued for path: / 2024-03-28T00:57:25.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:25 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T00:57:25.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:25 smithi174 ceph-mon[36060]: scrub summary: active paths [/] 2024-03-28T00:57:25.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:25 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T00:57:25.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:25 smithi177 ceph-mon[36206]: pgmap v378: 129 pgs: 129 active+clean; 665 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-28T00:57:25.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:25 smithi177 ceph-mon[36206]: scrub queued for path: / 2024-03-28T00:57:25.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:25 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T00:57:25.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:25 smithi177 ceph-mon[36206]: scrub summary: active paths [/] 2024-03-28T00:57:25.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:25 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T00:57:27.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:27 smithi169 ceph-mon[32274]: pgmap v379: 129 pgs: 129 active+clean; 665 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T00:57:27.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:27 smithi174 ceph-mon[36060]: pgmap v379: 129 pgs: 129 active+clean; 665 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T00:57:27.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:27 smithi177 ceph-mon[36206]: pgmap v379: 129 pgs: 129 active+clean; 665 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T00:57:29.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:29 smithi169 ceph-mon[32274]: pgmap v380: 129 pgs: 129 active+clean; 665 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T00:57:29.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:29 smithi169 ceph-mon[32274]: scrub summary: idle 2024-03-28T00:57:29.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:29 smithi174 ceph-mon[36060]: pgmap v380: 129 pgs: 129 active+clean; 665 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T00:57:29.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:29 smithi174 ceph-mon[36060]: scrub summary: idle 2024-03-28T00:57:29.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:29 smithi177 ceph-mon[36206]: pgmap v380: 129 pgs: 129 active+clean; 665 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T00:57:29.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:29 smithi177 ceph-mon[36206]: scrub summary: idle 2024-03-28T00:57:31.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:31 smithi169 ceph-mon[32274]: pgmap v381: 129 pgs: 129 active+clean; 665 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-28T00:57:31.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:31 smithi174 ceph-mon[36060]: pgmap v381: 129 pgs: 129 active+clean; 665 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-28T00:57:31.633 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.628+0000 7fdbf8497640 1 -- 172.21.15.169:0/1454458588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbf01017b0 msgr2=0x7fdbf0101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:31.633 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.628+0000 7fdbf8497640 1 --2- 172.21.15.169:0/1454458588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbf01017b0 0x7fdbf0101c30 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0009900 tx=0x7fdbe002bc30 comp rx=0 tx=0).stop 2024-03-28T00:57:31.633 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.628+0000 7fdbf8497640 1 -- 172.21.15.169:0/1454458588 shutdown_connections 2024-03-28T00:57:31.634 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.628+0000 7fdbf8497640 1 --2- 172.21.15.169:0/1454458588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdbf0102170 0x7fdbf0108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.634 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.628+0000 7fdbf8497640 1 --2- 172.21.15.169:0/1454458588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbf01017b0 0x7fdbf0101c30 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.634 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.628+0000 7fdbf8497640 1 --2- 172.21.15.169:0/1454458588 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf01005c0 0x7fdbf01009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.634 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.628+0000 7fdbf8497640 1 -- 172.21.15.169:0/1454458588 >> 172.21.15.169:0/1454458588 conn(0x7fdbf00fbb30 msgr2=0x7fdbf00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:31.634 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.628+0000 7fdbf8497640 1 -- 172.21.15.169:0/1454458588 shutdown_connections 2024-03-28T00:57:31.634 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.628+0000 7fdbf8497640 1 -- 172.21.15.169:0/1454458588 wait complete. 2024-03-28T00:57:31.634 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.629+0000 7fdbf8497640 1 Processor -- start 2024-03-28T00:57:31.634 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.629+0000 7fdbf8497640 1 -- start start 2024-03-28T00:57:31.634 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.629+0000 7fdbf8497640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdbf01005c0 0x7fdbf0104500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:31.634 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.629+0000 7fdbf8497640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbf01017b0 0x7fdbf0102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:31.634 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.629+0000 7fdbf8497640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf0102170 0x7fdbf01030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:31.634 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.629+0000 7fdbf8497640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf010c280 con 0x7fdbf01017b0 2024-03-28T00:57:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.629+0000 7fdbf8497640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf010c100 con 0x7fdbf01005c0 2024-03-28T00:57:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.629+0000 7fdbf8497640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf010c400 con 0x7fdbf0102170 2024-03-28T00:57:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.629+0000 7fdbf5a32640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbf01017b0 0x7fdbf0102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.629+0000 7fdbf6a34640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf0102170 0x7fdbf01030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.629+0000 7fdbf6a34640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf0102170 0x7fdbf01030d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:39568/0 (socket says 172.21.15.169:39568) 2024-03-28T00:57:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.629+0000 7fdbf5a32640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbf01017b0 0x7fdbf0102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34094/0 (socket says 172.21.15.169:34094) 2024-03-28T00:57:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.629+0000 7fdbf6a34640 1 -- 172.21.15.169:0/1060561532 learned_addr learned my addr 172.21.15.169:0/1060561532 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.629+0000 7fdbf6233640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdbf01005c0 0x7fdbf0104500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.629+0000 7fdbf6a34640 1 -- 172.21.15.169:0/1060561532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdbf01005c0 msgr2=0x7fdbf0104500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.629+0000 7fdbf6a34640 1 --2- 172.21.15.169:0/1060561532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdbf01005c0 0x7fdbf0104500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.629+0000 7fdbf6a34640 1 -- 172.21.15.169:0/1060561532 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbf01017b0 msgr2=0x7fdbf0102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.629+0000 7fdbf6a34640 1 --2- 172.21.15.169:0/1060561532 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbf01017b0 0x7fdbf0102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.629+0000 7fdbf6a34640 1 -- 172.21.15.169:0/1060561532 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf0103990 con 0x7fdbf0102170 2024-03-28T00:57:31.636 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.629+0000 7fdbf5a32640 1 --2- 172.21.15.169:0/1060561532 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbf01017b0 0x7fdbf0102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:31.636 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.629+0000 7fdbf6233640 1 --2- 172.21.15.169:0/1060561532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdbf01005c0 0x7fdbf0104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:31.636 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.630+0000 7fdbf6a34640 1 --2- 172.21.15.169:0/1060561532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf0102170 0x7fdbf01030d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdbe800eb10 tx=0x7fdbe800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:31.636 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.630+0000 7fdbe77fe640 1 -- 172.21.15.169:0/1060561532 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe8004b20 con 0x7fdbf0102170 2024-03-28T00:57:31.636 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.630+0000 7fdbf8497640 1 -- 172.21.15.169:0/1060561532 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf01a1ae0 con 0x7fdbf0102170 2024-03-28T00:57:31.636 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.630+0000 7fdbe77fe640 1 -- 172.21.15.169:0/1060561532 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbe8004cc0 con 0x7fdbf0102170 2024-03-28T00:57:31.636 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.630+0000 7fdbe77fe640 1 -- 172.21.15.169:0/1060561532 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe8010760 con 0x7fdbf0102170 2024-03-28T00:57:31.636 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.630+0000 7fdbf8497640 1 -- 172.21.15.169:0/1060561532 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf01a2000 con 0x7fdbf0102170 2024-03-28T00:57:31.644 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.632+0000 7fdbe77fe640 1 -- 172.21.15.169:0/1060561532 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fdbe8005020 con 0x7fdbf0102170 2024-03-28T00:57:31.644 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.632+0000 7fdbe77fe640 1 --2- 172.21.15.169:0/1060561532 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdbdc0a2940 0x7fdbdc0a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:31.644 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.633+0000 7fdbf6233640 1 --2- 172.21.15.169:0/1060561532 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdbdc0a2940 0x7fdbdc0a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:31.644 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.633+0000 7fdbe77fe640 1 -- 172.21.15.169:0/1060561532 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fdbe80cf500 con 0x7fdbf0102170 2024-03-28T00:57:31.644 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.633+0000 7fdbf6233640 1 --2- 172.21.15.169:0/1060561532 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdbdc0a2940 0x7fdbdc0a4e00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0101610 tx=0x7fdbec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.633+0000 7fdbfad22740 1 Processor -- start 2024-03-28T00:57:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.634+0000 7fdbfad22740 1 -- start start 2024-03-28T00:57:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.634+0000 7fdbfad22740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad26920 0x55cd3ad26d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.634+0000 7fdbfad22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad27310 0x55cd3ad2be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.634+0000 7fdbfad22740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cd3ad2c480 0x55cd3ad2e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.634+0000 7fdbfad22740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd3ac8bb10 con 0x55cd3ad26920 2024-03-28T00:57:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.634+0000 7fdbfad22740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd3ac34900 con 0x55cd3ad2c480 2024-03-28T00:57:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.634+0000 7fdbfad22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd3ac87bb0 con 0x55cd3ad27310 2024-03-28T00:57:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.634+0000 7fdbf6233640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad26920 0x55cd3ad26d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.634+0000 7fdbf5a32640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad27310 0x55cd3ad2be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.634+0000 7fdbf5a32640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad27310 0x55cd3ad2be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:39572/0 (socket says 172.21.15.169:39572) 2024-03-28T00:57:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.634+0000 7fdbf6233640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad26920 0x55cd3ad26d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34110/0 (socket says 172.21.15.169:34110) 2024-03-28T00:57:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.634+0000 7fdbf5a32640 1 -- 172.21.15.169:0/1960121480 learned_addr learned my addr 172.21.15.169:0/1960121480 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.634+0000 7fdbf6a34640 1 --2- 172.21.15.169:0/1960121480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cd3ad2c480 0x55cd3ad2e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.634+0000 7fdbf5a32640 1 -- 172.21.15.169:0/1960121480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cd3ad2c480 msgr2=0x55cd3ad2e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.634+0000 7fdbf5a32640 1 --2- 172.21.15.169:0/1960121480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cd3ad2c480 0x55cd3ad2e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.634+0000 7fdbf5a32640 1 -- 172.21.15.169:0/1960121480 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad26920 msgr2=0x55cd3ad26d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.634+0000 7fdbf5a32640 1 --2- 172.21.15.169:0/1960121480 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad26920 0x55cd3ad26d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.634+0000 7fdbf5a32640 1 -- 172.21.15.169:0/1960121480 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd3a9a0210 con 0x55cd3ad27310 2024-03-28T00:57:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.634+0000 7fdbf5a32640 1 --2- 172.21.15.169:0/1960121480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad27310 0x55cd3ad2be10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0005ab0 tx=0x7fdbe0009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c19816efc9b686cc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.635+0000 7fdbe4bfe640 1 -- 172.21.15.169:0/1960121480 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe0045070 con 0x55cd3ad27310 2024-03-28T00:57:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.635+0000 7fdbe4bfe640 1 -- 172.21.15.169:0/1960121480 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbe002cd30 con 0x55cd3ad27310 2024-03-28T00:57:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.635+0000 7fdbe4bfe640 1 -- 172.21.15.169:0/1960121480 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe0005110 con 0x55cd3ad27310 2024-03-28T00:57:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.635+0000 7fdbfad22740 1 -- 172.21.15.169:0/1960121480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad27310 msgr2=0x55cd3ad2be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.635+0000 7fdbfad22740 1 --2- 172.21.15.169:0/1960121480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad27310 0x55cd3ad2be10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0005ab0 tx=0x7fdbe0009560 comp rx=0 tx=0).stop 2024-03-28T00:57:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.635+0000 7fdbfad22740 1 -- 172.21.15.169:0/1960121480 shutdown_connections 2024-03-28T00:57:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.635+0000 7fdbfad22740 1 --2- 172.21.15.169:0/1960121480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cd3ad2c480 0x55cd3ad2e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.635+0000 7fdbfad22740 1 --2- 172.21.15.169:0/1960121480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad27310 0x55cd3ad2be10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.635+0000 7fdbfad22740 1 --2- 172.21.15.169:0/1960121480 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad26920 0x55cd3ad26d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.635+0000 7fdbfad22740 1 -- 172.21.15.169:0/1960121480 >> 172.21.15.169:0/1960121480 conn(0x55cd3ad21e90 msgr2=0x55cd3ad22290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbfad22740 1 -- 172.21.15.169:0/1960121480 shutdown_connections 2024-03-28T00:57:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbfad22740 1 -- 172.21.15.169:0/1960121480 wait complete. 2024-03-28T00:57:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbfad22740 1 -- start start 2024-03-28T00:57:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbfad22740 1 Processor -- start 2024-03-28T00:57:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbfad22740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad26920 0x55cd3ad30c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbfad22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad27310 0x55cd3ad31150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbfad22740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cd3ad2c480 0x55cd3ad38120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbfad22740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd3ad2f3d0 con 0x55cd3ad26920 2024-03-28T00:57:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbfad22740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd3ac47d50 con 0x55cd3ad2c480 2024-03-28T00:57:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbfad22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd3ad2f550 con 0x55cd3ad27310 2024-03-28T00:57:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbf6233640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad26920 0x55cd3ad30c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbf6233640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad26920 0x55cd3ad30c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34122/0 (socket says 172.21.15.169:34122) 2024-03-28T00:57:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbfad22740 10 client.0 _ll_register_callbacks cb 0x55cd3ac47720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbf6233640 1 -- 172.21.15.169:0/1462865991 learned_addr learned my addr 172.21.15.169:0/1462865991 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbf5a32640 1 --2- 172.21.15.169:0/1462865991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad27310 0x55cd3ad31150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbf6a34640 1 --2- 172.21.15.169:0/1462865991 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cd3ad2c480 0x55cd3ad38120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbf6233640 1 -- 172.21.15.169:0/1462865991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad27310 msgr2=0x55cd3ad31150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbf6233640 1 --2- 172.21.15.169:0/1462865991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad27310 0x55cd3ad31150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbf6233640 1 -- 172.21.15.169:0/1462865991 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cd3ad2c480 msgr2=0x55cd3ad38120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbf6233640 1 --2- 172.21.15.169:0/1462865991 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cd3ad2c480 0x55cd3ad38120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbf6233640 1 -- 172.21.15.169:0/1462865991 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd3aa6c620 con 0x55cd3ad26920 2024-03-28T00:57:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.636+0000 7fdbf5a32640 1 --2- 172.21.15.169:0/1462865991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad27310 0x55cd3ad31150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.637+0000 7fdbf6233640 1 --2- 172.21.15.169:0/1462865991 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad26920 0x55cd3ad30c10 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fdbec003760 tx=0x7fdbec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=204118d7a288192d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.637+0000 7fdbd9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:57:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.637+0000 7fdbd9ffb640 1 -- 172.21.15.169:0/1462865991 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec02b060 con 0x55cd3ad26920 2024-03-28T00:57:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.637+0000 7fdbd9ffb640 1 -- 172.21.15.169:0/1462865991 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fdbec00f690 con 0x55cd3ad26920 2024-03-28T00:57:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.637+0000 7fdbfad22740 1 -- 172.21.15.169:0/1462865991 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdbe002cd30 con 0x55cd3ad26920 2024-03-28T00:57:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.637+0000 7fdbd9ffb640 1 -- 172.21.15.169:0/1462865991 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdbec032050 con 0x55cd3ad26920 2024-03-28T00:57:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.637+0000 7fdbfad22740 10 client.15783 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.637+0000 7fdbfad22740 1 -- 172.21.15.169:0/1462865991 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55cd3ad3b550 con 0x55cd3ad26920 2024-03-28T00:57:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.637+0000 7fdbd9ffb640 1 -- 172.21.15.169:0/1462865991 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbe002cd30 con 0x55cd3ad26920 2024-03-28T00:57:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.637+0000 7fdbd9ffb640 1 -- 172.21.15.169:0/1462865991 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fdbec01d9b0 con 0x55cd3ad26920 2024-03-28T00:57:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.637+0000 7fdbfad22740 10 client.15783 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.637+0000 7fdbfad22740 10 client.15783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.637+0000 7fdbfad22740 20 client.15783 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:31.650 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.637+0000 7fdbfad22740 1 --2- 172.21.15.169:0/1462865991 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55cd3ad3cc70 0x55cd3ad3f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:31.650 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.638+0000 7fdbfad22740 4 client.15783 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:57:31.650 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.638+0000 7fdbfad22740 1 -- 172.21.15.169:0/1462865991 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cd3ac99050 con 0x55cd3ad3cc70 2024-03-28T00:57:31.650 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.638+0000 7fdbf5a32640 1 --2- 172.21.15.169:0/1462865991 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55cd3ad3cc70 0x55cd3ad3f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:31.650 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.638+0000 7fdbf5a32640 1 --2- 172.21.15.169:0/1462865991 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55cd3ad3cc70 0x55cd3ad3f0e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=347cc70f05eef079 server_cookie=e3632d1d7af492 in_seq=0 out_seq=0 2024-03-28T00:57:31.650 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.638+0000 7fdbd9ffb640 10 client.15783 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:31.650 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.639+0000 7fdbd9ffb640 1 -- 172.21.15.169:0/1462865991 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cd3ac99050 con 0x55cd3ad3cc70 2024-03-28T00:57:31.650 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.639+0000 7fdbd9ffb640 10 client.15783 handle_command_reply: tid=0 2024-03-28T00:57:31.650 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.639+0000 7fdbfad22740 1 client.15783 shutdown 2024-03-28T00:57:31.650 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.639+0000 7fdbfad22740 1 -- 172.21.15.169:0/1462865991 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad26920 msgr2=0x55cd3ad30c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:31.650 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.639+0000 7fdbfad22740 1 --2- 172.21.15.169:0/1462865991 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad26920 0x55cd3ad30c10 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fdbec003760 tx=0x7fdbec004730 comp rx=0 tx=0).stop 2024-03-28T00:57:31.650 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.639+0000 7fdbfad22740 1 -- 172.21.15.169:0/1462865991 shutdown_connections 2024-03-28T00:57:31.650 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.639+0000 7fdbfad22740 1 --2- 172.21.15.169:0/1462865991 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cd3ad2c480 0x55cd3ad38120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.650 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.639+0000 7fdbfad22740 1 --2- 172.21.15.169:0/1462865991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad27310 0x55cd3ad31150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.651 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.639+0000 7fdbfad22740 1 --2- 172.21.15.169:0/1462865991 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55cd3ad3cc70 0x55cd3ad3f0e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.651 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.639+0000 7fdbfad22740 1 --2- 172.21.15.169:0/1462865991 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad26920 0x55cd3ad30c10 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.651 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.639+0000 7fdbd9ffb640 0 client.15783 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:31.651 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.639+0000 7fdbfad22740 1 -- 172.21.15.169:0/1462865991 >> 172.21.15.169:0/1462865991 conn(0x55cd3ad21e90 msgr2=0x55cd3ad21020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:31.651 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.639+0000 7fdbfad22740 1 -- 172.21.15.169:0/1462865991 shutdown_connections 2024-03-28T00:57:31.651 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.639+0000 7fdbfad22740 1 -- 172.21.15.169:0/1462865991 wait complete. 2024-03-28T00:57:31.651 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.639+0000 7fdbfad22740 20 client.15783 trim_cache size 0 max 16384 2024-03-28T00:57:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.653+0000 7fdbfad22740 1 Processor -- start 2024-03-28T00:57:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.653+0000 7fdbfad22740 1 -- start start 2024-03-28T00:57:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.653+0000 7fdbfad22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad26920 0x55cd3ad22270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.653+0000 7fdbfad22740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cd3ad27310 0x55cd3ad380e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.653+0000 7fdbfad22740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad2c480 0x55cd3ad30820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.653+0000 7fdbfad22740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf01ae7c0 con 0x55cd3ad2c480 2024-03-28T00:57:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.653+0000 7fdbfad22740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf01ac2c0 con 0x55cd3ad27310 2024-03-28T00:57:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.653+0000 7fdbfad22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf01aa360 con 0x55cd3ad26920 2024-03-28T00:57:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.653+0000 7fdbf6233640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad26920 0x55cd3ad22270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.653+0000 7fdbf6233640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad26920 0x55cd3ad22270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:39598/0 (socket says 172.21.15.169:39598) 2024-03-28T00:57:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.653+0000 7fdbf6233640 1 -- 172.21.15.169:0/313646514 learned_addr learned my addr 172.21.15.169:0/313646514 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.653+0000 7fdbf5a32640 1 --2- 172.21.15.169:0/313646514 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cd3ad27310 0x55cd3ad380e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.653+0000 7fdbf6a34640 1 --2- 172.21.15.169:0/313646514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad2c480 0x55cd3ad30820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.653+0000 7fdbf6233640 1 -- 172.21.15.169:0/313646514 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cd3ad27310 msgr2=0x55cd3ad380e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.653+0000 7fdbf6233640 1 --2- 172.21.15.169:0/313646514 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cd3ad27310 0x55cd3ad380e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.653+0000 7fdbf6233640 1 -- 172.21.15.169:0/313646514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad2c480 msgr2=0x55cd3ad30820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.653+0000 7fdbf6233640 1 --2- 172.21.15.169:0/313646514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad2c480 0x55cd3ad30820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.653+0000 7fdbf6233640 1 -- 172.21.15.169:0/313646514 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd3ad395a0 con 0x55cd3ad26920 2024-03-28T00:57:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.654+0000 7fdbf5a32640 1 --2- 172.21.15.169:0/313646514 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cd3ad27310 0x55cd3ad380e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.654+0000 7fdbf6a34640 1 --2- 172.21.15.169:0/313646514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad2c480 0x55cd3ad30820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:57:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.654+0000 7fdbf6233640 1 --2- 172.21.15.169:0/313646514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad26920 0x55cd3ad22270 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdbec009e80 tx=0x7fdbec003ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=950636f6c1eeacce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.654+0000 7fdbdbfff640 1 -- 172.21.15.169:0/313646514 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec02b040 con 0x55cd3ad26920 2024-03-28T00:57:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.654+0000 7fdbdbfff640 1 -- 172.21.15.169:0/313646514 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbec01d900 con 0x55cd3ad26920 2024-03-28T00:57:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.654+0000 7fdbdbfff640 1 -- 172.21.15.169:0/313646514 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec00fb10 con 0x55cd3ad26920 2024-03-28T00:57:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbfad22740 1 -- 172.21.15.169:0/313646514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad26920 msgr2=0x55cd3ad22270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbfad22740 1 --2- 172.21.15.169:0/313646514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad26920 0x55cd3ad22270 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdbec009e80 tx=0x7fdbec003ca0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbfad22740 1 -- 172.21.15.169:0/313646514 shutdown_connections 2024-03-28T00:57:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbfad22740 1 --2- 172.21.15.169:0/313646514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad2c480 0x55cd3ad30820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbfad22740 1 --2- 172.21.15.169:0/313646514 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cd3ad27310 0x55cd3ad380e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbfad22740 1 --2- 172.21.15.169:0/313646514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad26920 0x55cd3ad22270 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbfad22740 1 -- 172.21.15.169:0/313646514 >> 172.21.15.169:0/313646514 conn(0x55cd3ad21e90 msgr2=0x55cd3ad1ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbfad22740 1 -- 172.21.15.169:0/313646514 shutdown_connections 2024-03-28T00:57:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbfad22740 1 -- 172.21.15.169:0/313646514 wait complete. 2024-03-28T00:57:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbfad22740 1 -- start start 2024-03-28T00:57:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbfad22740 1 Processor -- start 2024-03-28T00:57:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbfad22740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad26920 0x55cd3ad355d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbfad22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad27310 0x55cd3ad35b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbfad22740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cd3ad2c480 0x55cd3ad36ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbfad22740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf01aa960 con 0x55cd3ad26920 2024-03-28T00:57:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbfad22740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf01abc60 con 0x55cd3ad2c480 2024-03-28T00:57:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbfad22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf01aafa0 con 0x55cd3ad27310 2024-03-28T00:57:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbfad22740 10 client.0 _ll_register_callbacks cb 0x7fdbec0035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbf6233640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad26920 0x55cd3ad355d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbf5a32640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad27310 0x55cd3ad35b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbf6233640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad26920 0x55cd3ad355d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34140/0 (socket says 172.21.15.169:34140) 2024-03-28T00:57:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.655+0000 7fdbf6233640 1 -- 172.21.15.169:0/1477134803 learned_addr learned my addr 172.21.15.169:0/1477134803 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.656+0000 7fdbf6a34640 1 --2- 172.21.15.169:0/1477134803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cd3ad2c480 0x55cd3ad36ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.656+0000 7fdbf5a32640 1 -- 172.21.15.169:0/1477134803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cd3ad2c480 msgr2=0x55cd3ad36ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.656+0000 7fdbf5a32640 1 --2- 172.21.15.169:0/1477134803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cd3ad2c480 0x55cd3ad36ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.656+0000 7fdbf5a32640 1 -- 172.21.15.169:0/1477134803 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad26920 msgr2=0x55cd3ad355d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.656+0000 7fdbf5a32640 1 --2- 172.21.15.169:0/1477134803 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad26920 0x55cd3ad355d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.656+0000 7fdbf5a32640 1 -- 172.21.15.169:0/1477134803 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd3ad370c0 con 0x55cd3ad27310 2024-03-28T00:57:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.656+0000 7fdbf5a32640 1 --2- 172.21.15.169:0/1477134803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad27310 0x55cd3ad35b10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x55cd3ad22e00 tx=0x7fdbe0005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d0b973153b86841 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.656+0000 7fdbd9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:57:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.656+0000 7fdbd9ffb640 1 -- 172.21.15.169:0/1477134803 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe0056060 con 0x55cd3ad27310 2024-03-28T00:57:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.656+0000 7fdbfad22740 1 -- 172.21.15.169:0/1477134803 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdbec01d900 con 0x55cd3ad27310 2024-03-28T00:57:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.656+0000 7fdbd9ffb640 1 -- 172.21.15.169:0/1477134803 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fdbe0005070 con 0x55cd3ad27310 2024-03-28T00:57:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.656+0000 7fdbd9ffb640 1 -- 172.21.15.169:0/1477134803 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdbe005d050 con 0x55cd3ad27310 2024-03-28T00:57:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.656+0000 7fdbfad22740 10 client.25585 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.656+0000 7fdbfad22740 1 -- 172.21.15.169:0/1477134803 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55cd3ad383d0 con 0x55cd3ad27310 2024-03-28T00:57:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.656+0000 7fdbd9ffb640 1 -- 172.21.15.169:0/1477134803 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbec01d900 con 0x55cd3ad27310 2024-03-28T00:57:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.657+0000 7fdbd9ffb640 1 -- 172.21.15.169:0/1477134803 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fdbe004a9d0 con 0x55cd3ad27310 2024-03-28T00:57:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.657+0000 7fdbfad22740 10 client.25585 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.657+0000 7fdbfad22740 10 client.25585 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.657+0000 7fdbfad22740 20 client.25585 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.657+0000 7fdbfad22740 1 --2- 172.21.15.169:0/1477134803 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55cd3ad98990 0x55cd3ad9ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.657+0000 7fdbfad22740 4 client.25585 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T00:57:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.657+0000 7fdbfad22740 1 -- 172.21.15.169:0/1477134803 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55cd3ac933a0 con 0x55cd3ad98990 2024-03-28T00:57:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.657+0000 7fdbf6233640 1 --2- 172.21.15.169:0/1477134803 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55cd3ad98990 0x55cd3ad9ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.657+0000 7fdbd9ffb640 10 client.25585 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.657+0000 7fdbf6233640 1 --2- 172.21.15.169:0/1477134803 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55cd3ad98990 0x55cd3ad9ad80 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce58b0b608139ac2 server_cookie=bd93f43902c09320 in_seq=0 out_seq=0 2024-03-28T00:57:31.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:31 smithi177 ceph-mon[36206]: pgmap v381: 129 pgs: 129 active+clean; 665 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-28T00:57:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.663+0000 7fdbd9ffb640 1 -- 172.21.15.169:0/1477134803 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55cd3ac933a0 con 0x55cd3ad98990 2024-03-28T00:57:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.663+0000 7fdbd9ffb640 10 client.25585 handle_command_reply: tid=0 2024-03-28T00:57:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.663+0000 7fdbfad22740 1 client.25585 shutdown 2024-03-28T00:57:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.663+0000 7fdbfad22740 1 -- 172.21.15.169:0/1477134803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad27310 msgr2=0x55cd3ad35b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.663+0000 7fdbfad22740 1 --2- 172.21.15.169:0/1477134803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad27310 0x55cd3ad35b10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x55cd3ad22e00 tx=0x7fdbe0005c00 comp rx=0 tx=0).stop 2024-03-28T00:57:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.663+0000 7fdbfad22740 1 -- 172.21.15.169:0/1477134803 shutdown_connections 2024-03-28T00:57:31.669 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.663+0000 7fdbfad22740 1 --2- 172.21.15.169:0/1477134803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cd3ad2c480 0x55cd3ad36ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.669 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.663+0000 7fdbfad22740 1 --2- 172.21.15.169:0/1477134803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cd3ad27310 0x55cd3ad35b10 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.669 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.663+0000 7fdbfad22740 1 --2- 172.21.15.169:0/1477134803 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55cd3ad98990 0x55cd3ad9ad80 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.669 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.663+0000 7fdbfad22740 1 --2- 172.21.15.169:0/1477134803 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cd3ad26920 0x55cd3ad355d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.669 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.663+0000 7fdbd9ffb640 0 client.25585 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:31.669 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.663+0000 7fdbfad22740 1 -- 172.21.15.169:0/1477134803 >> 172.21.15.169:0/1477134803 conn(0x55cd3ad21e90 msgr2=0x55cd3ad1ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:31.669 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.663+0000 7fdbfad22740 1 -- 172.21.15.169:0/1477134803 shutdown_connections 2024-03-28T00:57:31.669 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.663+0000 7fdbfad22740 1 -- 172.21.15.169:0/1477134803 wait complete. 2024-03-28T00:57:31.669 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.663+0000 7fdbfad22740 20 client.25585 trim_cache size 0 max 16384 2024-03-28T00:57:31.669 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T00:57:31.669 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T00:57:31.670 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T00:57:31.670 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T00:57:31.671 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.664+0000 7fdbf8497640 1 -- 172.21.15.169:0/1060561532 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdbdc0a2940 msgr2=0x7fdbdc0a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:31.671 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.664+0000 7fdbf8497640 1 --2- 172.21.15.169:0/1060561532 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdbdc0a2940 0x7fdbdc0a4e00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0101610 tx=0x7fdbec005f90 comp rx=0 tx=0).stop 2024-03-28T00:57:31.671 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.664+0000 7fdbf8497640 1 -- 172.21.15.169:0/1060561532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf0102170 msgr2=0x7fdbf01030d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:31.671 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.664+0000 7fdbf8497640 1 --2- 172.21.15.169:0/1060561532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf0102170 0x7fdbf01030d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdbe800eb10 tx=0x7fdbe800c2d0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.671 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.664+0000 7fdbf8497640 1 -- 172.21.15.169:0/1060561532 shutdown_connections 2024-03-28T00:57:31.671 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.664+0000 7fdbf8497640 1 --2- 172.21.15.169:0/1060561532 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdbdc0a2940 0x7fdbdc0a4e00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.671 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.664+0000 7fdbf8497640 1 --2- 172.21.15.169:0/1060561532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf0102170 0x7fdbf01030d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.671 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.664+0000 7fdbf8497640 1 --2- 172.21.15.169:0/1060561532 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbf01017b0 0x7fdbf0102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.671 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.664+0000 7fdbf8497640 1 --2- 172.21.15.169:0/1060561532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdbf01005c0 0x7fdbf0104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:31.671 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.664+0000 7fdbf8497640 1 -- 172.21.15.169:0/1060561532 >> 172.21.15.169:0/1060561532 conn(0x7fdbf00fbb30 msgr2=0x7fdbf00fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:31.671 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.664+0000 7fdbf8497640 1 -- 172.21.15.169:0/1060561532 shutdown_connections 2024-03-28T00:57:31.671 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:31.664+0000 7fdbf8497640 1 -- 172.21.15.169:0/1060561532 wait complete. 2024-03-28T00:57:33.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:33 smithi169 ceph-mon[32274]: pgmap v382: 129 pgs: 129 active+clean; 665 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-28T00:57:33.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:33 smithi174 ceph-mon[36060]: pgmap v382: 129 pgs: 129 active+clean; 665 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-28T00:57:33.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:33 smithi177 ceph-mon[36206]: pgmap v382: 129 pgs: 129 active+clean; 665 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-28T00:57:35.554 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:35 smithi169 ceph-mon[32274]: pgmap v383: 129 pgs: 129 active+clean; 676 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-03-28T00:57:35.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:35 smithi174 ceph-mon[36060]: pgmap v383: 129 pgs: 129 active+clean; 676 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-03-28T00:57:35.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:35 smithi177 ceph-mon[36206]: pgmap v383: 129 pgs: 129 active+clean; 676 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-03-28T00:57:37.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:37 smithi169 ceph-mon[32274]: pgmap v384: 129 pgs: 129 active+clean; 676 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-28T00:57:37.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:37 smithi174 ceph-mon[36060]: pgmap v384: 129 pgs: 129 active+clean; 676 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-28T00:57:37.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:37 smithi177 ceph-mon[36206]: pgmap v384: 129 pgs: 129 active+clean; 676 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-28T00:57:39.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:39 smithi169 ceph-mon[32274]: pgmap v385: 129 pgs: 129 active+clean; 676 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-28T00:57:39.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:39 smithi174 ceph-mon[36060]: pgmap v385: 129 pgs: 129 active+clean; 676 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-28T00:57:39.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:39 smithi177 ceph-mon[36206]: pgmap v385: 129 pgs: 129 active+clean; 676 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-28T00:57:41.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:41 smithi169 ceph-mon[32274]: pgmap v386: 129 pgs: 129 active+clean; 676 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-28T00:57:41.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:41 smithi174 ceph-mon[36060]: pgmap v386: 129 pgs: 129 active+clean; 676 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-28T00:57:41.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:41 smithi177 ceph-mon[36206]: pgmap v386: 129 pgs: 129 active+clean; 676 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-28T00:57:43.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:43 smithi169 ceph-mon[32274]: pgmap v387: 129 pgs: 129 active+clean; 676 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-28T00:57:43.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:43 smithi174 ceph-mon[36060]: pgmap v387: 129 pgs: 129 active+clean; 676 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-28T00:57:43.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:43 smithi177 ceph-mon[36206]: pgmap v387: 129 pgs: 129 active+clean; 676 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-28T00:57:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.771+0000 7f0eb8ce8640 1 -- 172.21.15.169:0/601299396 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0eb41016f0 msgr2=0x7f0eb4101b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:44.775 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.771+0000 7f0eb8ce8640 1 --2- 172.21.15.169:0/601299396 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0eb41016f0 0x7f0eb4101b10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4009900 tx=0x7f0ea402bc30 comp rx=0 tx=0).stop 2024-03-28T00:57:44.775 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.771+0000 7f0eb8ce8640 1 -- 172.21.15.169:0/601299396 shutdown_connections 2024-03-28T00:57:44.775 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.771+0000 7f0eb8ce8640 1 --2- 172.21.15.169:0/601299396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eb41032a0 0x7f0eb4109b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.775 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.771+0000 7f0eb8ce8640 1 --2- 172.21.15.169:0/601299396 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0eb41028e0 0x7f0eb4102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.775 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.771+0000 7f0eb8ce8640 1 --2- 172.21.15.169:0/601299396 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0eb41016f0 0x7f0eb4101b10 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.775 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.771+0000 7f0eb8ce8640 1 -- 172.21.15.169:0/601299396 >> 172.21.15.169:0/601299396 conn(0x7f0eb40fcc60 msgr2=0x7f0eb40ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:44.775 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.771+0000 7f0eb8ce8640 1 -- 172.21.15.169:0/601299396 shutdown_connections 2024-03-28T00:57:44.775 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.771+0000 7f0eb8ce8640 1 -- 172.21.15.169:0/601299396 wait complete. 2024-03-28T00:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.772+0000 7f0eb8ce8640 1 Processor -- start 2024-03-28T00:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.772+0000 7f0eb8ce8640 1 -- start start 2024-03-28T00:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.772+0000 7f0eb8ce8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0eb41016f0 0x7f0eb4197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.772+0000 7f0eb8ce8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eb41028e0 0x7f0eb4197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.773+0000 7f0eb8ce8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0eb41032a0 0x7f0eb419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.773+0000 7f0eb8ce8640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb410d370 con 0x7f0eb41032a0 2024-03-28T00:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.773+0000 7f0eb8ce8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb410d1f0 con 0x7f0eb41016f0 2024-03-28T00:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.773+0000 7f0eb8ce8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb410d4f0 con 0x7f0eb41028e0 2024-03-28T00:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.773+0000 7f0eb2d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0eb41032a0 0x7f0eb419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.773+0000 7f0eabfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eb41028e0 0x7f0eb4197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.773+0000 7f0eabfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eb41028e0 0x7f0eb4197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:35264/0 (socket says 172.21.15.169:35264) 2024-03-28T00:57:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.773+0000 7f0eabfff640 1 -- 172.21.15.169:0/2875584190 learned_addr learned my addr 172.21.15.169:0/2875584190 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.773+0000 7f0eb259c640 1 --2- 172.21.15.169:0/2875584190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0eb41016f0 0x7f0eb4197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.773+0000 7f0eb259c640 1 -- 172.21.15.169:0/2875584190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eb41028e0 msgr2=0x7f0eb4197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.773+0000 7f0eb259c640 1 --2- 172.21.15.169:0/2875584190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eb41028e0 0x7f0eb4197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.773+0000 7f0eb259c640 1 -- 172.21.15.169:0/2875584190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0eb41032a0 msgr2=0x7f0eb419ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.773+0000 7f0eb259c640 1 --2- 172.21.15.169:0/2875584190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0eb41032a0 0x7f0eb419ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.773+0000 7f0eb259c640 1 -- 172.21.15.169:0/2875584190 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb40738d0 con 0x7f0eb41016f0 2024-03-28T00:57:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.773+0000 7f0eabfff640 1 --2- 172.21.15.169:0/2875584190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eb41028e0 0x7f0eb4197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.773+0000 7f0eb259c640 1 --2- 172.21.15.169:0/2875584190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0eb41016f0 0x7f0eb4197960 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0ea402c170 tx=0x7f0ea402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.773+0000 7f0eab7fe640 1 -- 172.21.15.169:0/2875584190 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea4041070 con 0x7f0eb41016f0 2024-03-28T00:57:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.773+0000 7f0eb8ce8640 1 -- 172.21.15.169:0/2875584190 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb4073b60 con 0x7f0eb41016f0 2024-03-28T00:57:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.773+0000 7f0eab7fe640 1 -- 172.21.15.169:0/2875584190 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ea4004450 con 0x7f0eb41016f0 2024-03-28T00:57:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.773+0000 7f0eab7fe640 1 -- 172.21.15.169:0/2875584190 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea4003e50 con 0x7f0eb41016f0 2024-03-28T00:57:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.774+0000 7f0eb8ce8640 1 -- 172.21.15.169:0/2875584190 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb4074070 con 0x7f0eb41016f0 2024-03-28T00:57:44.779 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.776+0000 7f0eab7fe640 1 -- 172.21.15.169:0/2875584190 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f0ea4044070 con 0x7f0eb41016f0 2024-03-28T00:57:44.779 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.776+0000 7f0eab7fe640 1 --2- 172.21.15.169:0/2875584190 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0ea00a2ae0 0x7f0ea00a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:44.779 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.776+0000 7f0eabfff640 1 --2- 172.21.15.169:0/2875584190 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0ea00a2ae0 0x7f0ea00a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:44.779 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.776+0000 7f0eab7fe640 1 -- 172.21.15.169:0/2875584190 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f0ea40f0920 con 0x7f0eb41016f0 2024-03-28T00:57:44.779 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.777+0000 7f0eabfff640 1 --2- 172.21.15.169:0/2875584190 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0ea00a2ae0 0x7f0ea00a4fa0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4198e70 tx=0x7f0e9c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:44.780 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.777+0000 7f0ebb9ab740 1 Processor -- start 2024-03-28T00:57:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.777+0000 7f0ebb9ab740 1 -- start start 2024-03-28T00:57:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.777+0000 7f0ebb9ab740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f1920 0x55e05e4f1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.777+0000 7f0ebb9ab740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f2310 0x55e05e4f6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.777+0000 7f0ebb9ab740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f7480 0x55e05e4f9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.777+0000 7f0ebb9ab740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05e456b10 con 0x55e05e4f2310 2024-03-28T00:57:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.777+0000 7f0ebb9ab740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05e3ff900 con 0x55e05e4f7480 2024-03-28T00:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.777+0000 7f0ebb9ab740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05e452bb0 con 0x55e05e4f1920 2024-03-28T00:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.777+0000 7f0eb259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f1920 0x55e05e4f1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.777+0000 7f0eabfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f2310 0x55e05e4f6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.777+0000 7f0eb259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f1920 0x55e05e4f1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:35280/0 (socket says 172.21.15.169:35280) 2024-03-28T00:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.777+0000 7f0eb259c640 1 -- 172.21.15.169:0/2038047505 learned_addr learned my addr 172.21.15.169:0/2038047505 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.777+0000 7f0eb2d9d640 1 --2- 172.21.15.169:0/2038047505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f7480 0x55e05e4f9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.778+0000 7f0eb259c640 1 -- 172.21.15.169:0/2038047505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f7480 msgr2=0x55e05e4f9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.778+0000 7f0eb259c640 1 --2- 172.21.15.169:0/2038047505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f7480 0x55e05e4f9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.778+0000 7f0eb259c640 1 -- 172.21.15.169:0/2038047505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f2310 msgr2=0x55e05e4f6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.778+0000 7f0eb259c640 1 --2- 172.21.15.169:0/2038047505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f2310 0x55e05e4f6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.778+0000 7f0eb259c640 1 -- 172.21.15.169:0/2038047505 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e05e16b210 con 0x55e05e4f1920 2024-03-28T00:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.778+0000 7f0eabfff640 1 --2- 172.21.15.169:0/2038047505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f2310 0x55e05e4f6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.778+0000 7f0eb259c640 1 --2- 172.21.15.169:0/2038047505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f1920 0x55e05e4f1d40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4006010 tx=0x7f0ea40bcac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34fac62ca3e6c801 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.778+0000 7f0ea8ff9640 1 -- 172.21.15.169:0/2038047505 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea404b040 con 0x55e05e4f1920 2024-03-28T00:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.778+0000 7f0ea8ff9640 1 -- 172.21.15.169:0/2038047505 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ea40c1020 con 0x55e05e4f1920 2024-03-28T00:57:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.778+0000 7f0ea8ff9640 1 -- 172.21.15.169:0/2038047505 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea40c51e0 con 0x55e05e4f1920 2024-03-28T00:57:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.779+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/2038047505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f1920 msgr2=0x55e05e4f1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.779+0000 7f0ebb9ab740 1 --2- 172.21.15.169:0/2038047505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f1920 0x55e05e4f1d40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4006010 tx=0x7f0ea40bcac0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.779+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/2038047505 shutdown_connections 2024-03-28T00:57:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.779+0000 7f0ebb9ab740 1 --2- 172.21.15.169:0/2038047505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f7480 0x55e05e4f9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.779+0000 7f0ebb9ab740 1 --2- 172.21.15.169:0/2038047505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f2310 0x55e05e4f6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.779+0000 7f0ebb9ab740 1 --2- 172.21.15.169:0/2038047505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f1920 0x55e05e4f1d40 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.779+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/2038047505 >> 172.21.15.169:0/2038047505 conn(0x55e05e4ece90 msgr2=0x55e05e4ed290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.779+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/2038047505 shutdown_connections 2024-03-28T00:57:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.780+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/2038047505 wait complete. 2024-03-28T00:57:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.780+0000 7f0ebb9ab740 1 -- start start 2024-03-28T00:57:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.780+0000 7f0ebb9ab740 1 Processor -- start 2024-03-28T00:57:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.780+0000 7f0ebb9ab740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f1920 0x55e05e4fb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.780+0000 7f0ebb9ab740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f2310 0x55e05e4fbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0ebb9ab740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f7480 0x55e05e502f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0eb259c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f1920 0x55e05e4fb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0ebb9ab740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05e4fa1f0 con 0x55e05e4f1920 2024-03-28T00:57:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0ebb9ab740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05e412d50 con 0x55e05e4f7480 2024-03-28T00:57:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0eb259c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f1920 0x55e05e4fb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40682/0 (socket says 172.21.15.169:40682) 2024-03-28T00:57:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0eb259c640 1 -- 172.21.15.169:0/2904793868 learned_addr learned my addr 172.21.15.169:0/2904793868 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/2904793868 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05e4fa370 con 0x55e05e4f2310 2024-03-28T00:57:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0eabfff640 1 --2- 172.21.15.169:0/2904793868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f2310 0x55e05e4fbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0ebb9ab740 10 client.0 _ll_register_callbacks cb 0x55e05e412720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0eb2d9d640 1 --2- 172.21.15.169:0/2904793868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f7480 0x55e05e502f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0eb259c640 1 -- 172.21.15.169:0/2904793868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f2310 msgr2=0x55e05e4fbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0eb259c640 1 --2- 172.21.15.169:0/2904793868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f2310 0x55e05e4fbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0eb259c640 1 -- 172.21.15.169:0/2904793868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f7480 msgr2=0x55e05e502f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0eb259c640 1 --2- 172.21.15.169:0/2904793868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f7480 0x55e05e502f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0eb259c640 1 -- 172.21.15.169:0/2904793868 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e05e237620 con 0x55e05e4f1920 2024-03-28T00:57:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0eabfff640 1 --2- 172.21.15.169:0/2904793868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f2310 0x55e05e4fbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0eb2d9d640 1 --2- 172.21.15.169:0/2904793868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f7480 0x55e05e502f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:57:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0eb259c640 1 --2- 172.21.15.169:0/2904793868 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f1920 0x55e05e4fb9b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0ea40c5140 tx=0x7f0ea4009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b158fcb9d0ae48 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0e99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:57:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0e99ffb640 1 -- 172.21.15.169:0/2904793868 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea40d9020 con 0x55e05e4f1920 2024-03-28T00:57:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0e99ffb640 1 -- 172.21.15.169:0/2904793868 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f0ea40bc5e0 con 0x55e05e4f1920 2024-03-28T00:57:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.781+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/2904793868 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ea40c1020 con 0x55e05e4f1920 2024-03-28T00:57:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.782+0000 7f0e99ffb640 1 -- 172.21.15.169:0/2904793868 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ea40a4050 con 0x55e05e4f1920 2024-03-28T00:57:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.782+0000 7f0ebb9ab740 10 client.15801 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.782+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/2904793868 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55e05e4fa1f0 con 0x55e05e4f1920 2024-03-28T00:57:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.782+0000 7f0e99ffb640 1 -- 172.21.15.169:0/2904793868 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ea40c5670 con 0x55e05e4f1920 2024-03-28T00:57:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.782+0000 7f0e99ffb640 1 -- 172.21.15.169:0/2904793868 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f0ea40c1f40 con 0x55e05e4f1920 2024-03-28T00:57:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.782+0000 7f0ebb9ab740 10 client.15801 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.782+0000 7f0ebb9ab740 10 client.15801 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.782+0000 7f0ebb9ab740 20 client.15801 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.782+0000 7f0ebb9ab740 1 --2- 172.21.15.169:0/2904793868 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e05e507470 0x55e05e509860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.782+0000 7f0ebb9ab740 4 client.15801 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:57:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.782+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/2904793868 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ea40c1020 con 0x55e05e507470 2024-03-28T00:57:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.782+0000 7f0eabfff640 1 --2- 172.21.15.169:0/2904793868 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e05e507470 0x55e05e509860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.783+0000 7f0e99ffb640 10 client.15801 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.783+0000 7f0eabfff640 1 --2- 172.21.15.169:0/2904793868 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e05e507470 0x55e05e509860 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6be6f0215da4451c server_cookie=615d04683e262a2d in_seq=0 out_seq=0 2024-03-28T00:57:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.783+0000 7f0e99ffb640 1 -- 172.21.15.169:0/2904793868 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7f0ea40c1020 con 0x55e05e507470 2024-03-28T00:57:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.783+0000 7f0e99ffb640 10 client.15801 handle_command_reply: tid=0 2024-03-28T00:57:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.783+0000 7f0ebb9ab740 1 client.15801 shutdown 2024-03-28T00:57:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.783+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/2904793868 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f1920 msgr2=0x55e05e4fb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.783+0000 7f0ebb9ab740 1 --2- 172.21.15.169:0/2904793868 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f1920 0x55e05e4fb9b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0ea40c5140 tx=0x7f0ea4009680 comp rx=0 tx=0).stop 2024-03-28T00:57:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.784+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/2904793868 shutdown_connections 2024-03-28T00:57:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.784+0000 7f0ebb9ab740 1 --2- 172.21.15.169:0/2904793868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f7480 0x55e05e502f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.784+0000 7f0ebb9ab740 1 --2- 172.21.15.169:0/2904793868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f2310 0x55e05e4fbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.784+0000 7f0ebb9ab740 1 --2- 172.21.15.169:0/2904793868 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e05e507470 0x55e05e509860 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.784+0000 7f0ebb9ab740 1 --2- 172.21.15.169:0/2904793868 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f1920 0x55e05e4fb9b0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.784+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/2904793868 >> 172.21.15.169:0/2904793868 conn(0x55e05e4ece90 msgr2=0x55e05e4ec020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.784+0000 7f0e99ffb640 0 client.15801 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.784+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/2904793868 shutdown_connections 2024-03-28T00:57:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.784+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/2904793868 wait complete. 2024-03-28T00:57:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.784+0000 7f0ebb9ab740 20 client.15801 trim_cache size 0 max 16384 2024-03-28T00:57:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.809+0000 7f0ebb9ab740 1 Processor -- start 2024-03-28T00:57:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.810+0000 7f0ebb9ab740 1 -- start start 2024-03-28T00:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.810+0000 7f0ebb9ab740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f1920 0x55e05e4f0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.810+0000 7f0ebb9ab740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f2310 0x55e05e502ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.810+0000 7f0ebb9ab740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f7480 0x55e05e4fb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.810+0000 7f0ebb9ab740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb41ae8c0 con 0x55e05e4f1920 2024-03-28T00:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.810+0000 7f0ebb9ab740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb41ac3c0 con 0x55e05e4f2310 2024-03-28T00:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.810+0000 7f0ebb9ab740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb41aa460 con 0x55e05e4f7480 2024-03-28T00:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.810+0000 7f0eabfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f2310 0x55e05e502ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.810+0000 7f0eabfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f2310 0x55e05e502ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33012/0 (socket says 172.21.15.169:33012) 2024-03-28T00:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.810+0000 7f0eabfff640 1 -- 172.21.15.169:0/2101306249 learned_addr learned my addr 172.21.15.169:0/2101306249 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.810+0000 7f0eb259c640 1 --2- 172.21.15.169:0/2101306249 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f1920 0x55e05e4f0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.810+0000 7f0eb2d9d640 1 --2- 172.21.15.169:0/2101306249 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f7480 0x55e05e4fb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.810+0000 7f0eabfff640 1 -- 172.21.15.169:0/2101306249 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f7480 msgr2=0x55e05e4fb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.810+0000 7f0eabfff640 1 --2- 172.21.15.169:0/2101306249 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f7480 0x55e05e4fb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.810+0000 7f0eabfff640 1 -- 172.21.15.169:0/2101306249 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f1920 msgr2=0x55e05e4f0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.810+0000 7f0eabfff640 1 --2- 172.21.15.169:0/2101306249 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f1920 0x55e05e4f0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.810+0000 7f0eabfff640 1 -- 172.21.15.169:0/2101306249 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e05e4ed8e0 con 0x55e05e4f2310 2024-03-28T00:57:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.811+0000 7f0eabfff640 1 --2- 172.21.15.169:0/2101306249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f2310 0x55e05e502ec0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c009fd0 tx=0x7f0e9c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d628fe46e21f288 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.811+0000 7f0e9bfff640 1 -- 172.21.15.169:0/2101306249 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c030070 con 0x55e05e4f2310 2024-03-28T00:57:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.811+0000 7f0e9bfff640 1 -- 172.21.15.169:0/2101306249 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e9c01cdc0 con 0x55e05e4f2310 2024-03-28T00:57:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.811+0000 7f0e9bfff640 1 -- 172.21.15.169:0/2101306249 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c028410 con 0x55e05e4f2310 2024-03-28T00:57:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.812+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/2101306249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f2310 msgr2=0x55e05e502ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.812+0000 7f0ebb9ab740 1 --2- 172.21.15.169:0/2101306249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f2310 0x55e05e502ec0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c009fd0 tx=0x7f0e9c014000 comp rx=0 tx=0).stop 2024-03-28T00:57:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.812+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/2101306249 shutdown_connections 2024-03-28T00:57:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.812+0000 7f0ebb9ab740 1 --2- 172.21.15.169:0/2101306249 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f7480 0x55e05e4fb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.812+0000 7f0ebb9ab740 1 --2- 172.21.15.169:0/2101306249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f2310 0x55e05e502ec0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.812+0000 7f0ebb9ab740 1 --2- 172.21.15.169:0/2101306249 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f1920 0x55e05e4f0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.812+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/2101306249 >> 172.21.15.169:0/2101306249 conn(0x55e05e4ece90 msgr2=0x55e05e4f00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.812+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/2101306249 shutdown_connections 2024-03-28T00:57:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.812+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/2101306249 wait complete. 2024-03-28T00:57:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.813+0000 7f0ebb9ab740 1 -- start start 2024-03-28T00:57:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.813+0000 7f0ebb9ab740 1 Processor -- start 2024-03-28T00:57:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.813+0000 7f0ebb9ab740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f1920 0x55e05e5008c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.813+0000 7f0ebb9ab740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f2310 0x55e05e500e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.813+0000 7f0ebb9ab740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f7480 0x55e05e501e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.813+0000 7f0ebb9ab740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb41aaa60 con 0x55e05e4f1920 2024-03-28T00:57:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.813+0000 7f0ebb9ab740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb41abd60 con 0x55e05e4f2310 2024-03-28T00:57:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.813+0000 7f0ebb9ab740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb41ab0a0 con 0x55e05e4f7480 2024-03-28T00:57:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.813+0000 7f0ebb9ab740 10 client.0 _ll_register_callbacks cb 0x7f0ea40b92f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.813+0000 7f0eb259c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f1920 0x55e05e5008c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.813+0000 7f0eb259c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f1920 0x55e05e5008c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40688/0 (socket says 172.21.15.169:40688) 2024-03-28T00:57:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.813+0000 7f0eabfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f2310 0x55e05e500e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.813+0000 7f0eb259c640 1 -- 172.21.15.169:0/4234841809 learned_addr learned my addr 172.21.15.169:0/4234841809 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.813+0000 7f0eb2d9d640 1 --2- 172.21.15.169:0/4234841809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f7480 0x55e05e501e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.814+0000 7f0eb259c640 1 -- 172.21.15.169:0/4234841809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f7480 msgr2=0x55e05e501e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.814+0000 7f0eb259c640 1 --2- 172.21.15.169:0/4234841809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f7480 0x55e05e501e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.814+0000 7f0eb259c640 1 -- 172.21.15.169:0/4234841809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f2310 msgr2=0x55e05e500e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.814+0000 7f0eb259c640 1 --2- 172.21.15.169:0/4234841809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f2310 0x55e05e500e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.814+0000 7f0eb259c640 1 -- 172.21.15.169:0/4234841809 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e05e502400 con 0x55e05e4f1920 2024-03-28T00:57:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.814+0000 7f0eb259c640 1 --2- 172.21.15.169:0/4234841809 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f1920 0x55e05e5008c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0ea40ee4b0 tx=0x7f0ea4005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c62b1f23d9698685 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.814+0000 7f0e99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:57:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.814+0000 7f0e99ffb640 1 -- 172.21.15.169:0/4234841809 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea40d9040 con 0x55e05e4f1920 2024-03-28T00:57:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.814+0000 7f0e99ffb640 1 -- 172.21.15.169:0/4234841809 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f0ea40035f0 con 0x55e05e4f1920 2024-03-28T00:57:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.814+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/4234841809 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0e9c01cdc0 con 0x55e05e4f1920 2024-03-28T00:57:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.814+0000 7f0e99ffb640 1 -- 172.21.15.169:0/4234841809 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ea404b070 con 0x55e05e4f1920 2024-03-28T00:57:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.814+0000 7f0ebb9ab740 10 client.15807 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.814+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/4234841809 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55e05e502400 con 0x55e05e4f1920 2024-03-28T00:57:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.814+0000 7f0e99ffb640 1 -- 172.21.15.169:0/4234841809 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e9c01cdc0 con 0x55e05e4f1920 2024-03-28T00:57:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.815+0000 7f0e99ffb640 1 -- 172.21.15.169:0/4234841809 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f0ea40bbe40 con 0x55e05e4f1920 2024-03-28T00:57:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.815+0000 7f0ebb9ab740 10 client.15807 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.815+0000 7f0ebb9ab740 10 client.15807 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.815+0000 7f0ebb9ab740 20 client.15807 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.815+0000 7f0ebb9ab740 1 --2- 172.21.15.169:0/4234841809 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e05e563370 0x55e05e565760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.815+0000 7f0ebb9ab740 4 client.15807 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T00:57:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.815+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/4234841809 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e05e464050 con 0x55e05e563370 2024-03-28T00:57:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.815+0000 7f0eabfff640 1 --2- 172.21.15.169:0/4234841809 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e05e563370 0x55e05e565760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.815+0000 7f0e99ffb640 10 client.15807 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.815+0000 7f0eabfff640 1 --2- 172.21.15.169:0/4234841809 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e05e563370 0x55e05e565760 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8571fa57660f3692 server_cookie=a8834607e2cb105b in_seq=0 out_seq=0 2024-03-28T00:57:44.823 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.818+0000 7f0e99ffb640 1 -- 172.21.15.169:0/4234841809 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e05e464050 con 0x55e05e563370 2024-03-28T00:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.818+0000 7f0e99ffb640 10 client.15807 handle_command_reply: tid=0 2024-03-28T00:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.819+0000 7f0ebb9ab740 1 client.15807 shutdown 2024-03-28T00:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.819+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/4234841809 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f1920 msgr2=0x55e05e5008c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.819+0000 7f0ebb9ab740 1 --2- 172.21.15.169:0/4234841809 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f1920 0x55e05e5008c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0ea40ee4b0 tx=0x7f0ea4005b50 comp rx=0 tx=0).stop 2024-03-28T00:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.819+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/4234841809 shutdown_connections 2024-03-28T00:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.819+0000 7f0ebb9ab740 1 --2- 172.21.15.169:0/4234841809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e05e4f7480 0x55e05e501e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.819+0000 7f0ebb9ab740 1 --2- 172.21.15.169:0/4234841809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e05e4f2310 0x55e05e500e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.819+0000 7f0ebb9ab740 1 --2- 172.21.15.169:0/4234841809 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e05e563370 0x55e05e565760 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.819+0000 7f0ebb9ab740 1 --2- 172.21.15.169:0/4234841809 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e05e4f1920 0x55e05e5008c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.819+0000 7f0e99ffb640 0 client.15807 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.819+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/4234841809 >> 172.21.15.169:0/4234841809 conn(0x55e05e4ece90 msgr2=0x55e05e4f00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:44.827 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.819+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/4234841809 shutdown_connections 2024-03-28T00:57:44.827 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.819+0000 7f0ebb9ab740 1 -- 172.21.15.169:0/4234841809 wait complete. 2024-03-28T00:57:44.827 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.819+0000 7f0ebb9ab740 20 client.15807 trim_cache size 0 max 16384 2024-03-28T00:57:44.827 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T00:57:44.827 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T00:57:44.827 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T00:57:44.827 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T00:57:44.828 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.820+0000 7f0eb8ce8640 1 -- 172.21.15.169:0/2875584190 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0ea00a2ae0 msgr2=0x7f0ea00a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:44.828 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.820+0000 7f0eb8ce8640 1 --2- 172.21.15.169:0/2875584190 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0ea00a2ae0 0x7f0ea00a4fa0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4198e70 tx=0x7f0e9c005f90 comp rx=0 tx=0).stop 2024-03-28T00:57:44.828 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.820+0000 7f0eb8ce8640 1 -- 172.21.15.169:0/2875584190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0eb41016f0 msgr2=0x7f0eb4197960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:44.828 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.820+0000 7f0eb8ce8640 1 --2- 172.21.15.169:0/2875584190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0eb41016f0 0x7f0eb4197960 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0ea402c170 tx=0x7f0ea402c650 comp rx=0 tx=0).stop 2024-03-28T00:57:44.828 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.820+0000 7f0eb8ce8640 1 -- 172.21.15.169:0/2875584190 shutdown_connections 2024-03-28T00:57:44.828 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.820+0000 7f0eb8ce8640 1 --2- 172.21.15.169:0/2875584190 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0ea00a2ae0 0x7f0ea00a4fa0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.828 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.820+0000 7f0eb8ce8640 1 --2- 172.21.15.169:0/2875584190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0eb41032a0 0x7f0eb419ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.829 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.820+0000 7f0eb8ce8640 1 --2- 172.21.15.169:0/2875584190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eb41028e0 0x7f0eb4197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.829 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.820+0000 7f0eb8ce8640 1 --2- 172.21.15.169:0/2875584190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0eb41016f0 0x7f0eb4197960 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:44.829 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.820+0000 7f0eb8ce8640 1 -- 172.21.15.169:0/2875584190 >> 172.21.15.169:0/2875584190 conn(0x7f0eb40fcc60 msgr2=0x7f0eb4105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:44.829 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.820+0000 7f0eb8ce8640 1 -- 172.21.15.169:0/2875584190 shutdown_connections 2024-03-28T00:57:44.829 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:44.820+0000 7f0eb8ce8640 1 -- 172.21.15.169:0/2875584190 wait complete. 2024-03-28T00:57:45.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:45 smithi169 ceph-mon[32274]: pgmap v388: 129 pgs: 129 active+clean; 676 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-03-28T00:57:45.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:45 smithi174 ceph-mon[36060]: pgmap v388: 129 pgs: 129 active+clean; 676 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-03-28T00:57:45.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:45 smithi177 ceph-mon[36206]: pgmap v388: 129 pgs: 129 active+clean; 676 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-03-28T00:57:47.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:47 smithi169 ceph-mon[32274]: pgmap v389: 129 pgs: 129 active+clean; 676 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-28T00:57:47.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:47 smithi174 ceph-mon[36060]: pgmap v389: 129 pgs: 129 active+clean; 676 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-28T00:57:47.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:47 smithi177 ceph-mon[36206]: pgmap v389: 129 pgs: 129 active+clean; 676 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-28T00:57:49.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:49 smithi169 ceph-mon[32274]: pgmap v390: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-28T00:57:49.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:49 smithi174 ceph-mon[36060]: pgmap v390: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-28T00:57:49.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:49 smithi177 ceph-mon[36206]: pgmap v390: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-28T00:57:51.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:51 smithi169 ceph-mon[32274]: pgmap v391: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T00:57:51.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:51 smithi174 ceph-mon[36060]: pgmap v391: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T00:57:51.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:51 smithi177 ceph-mon[36206]: pgmap v391: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-28T00:57:53.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:53 smithi169 ceph-mon[32274]: pgmap v392: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T00:57:53.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:53 smithi174 ceph-mon[36060]: pgmap v392: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T00:57:53.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:53 smithi177 ceph-mon[36206]: pgmap v392: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T00:57:54.823 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T00:57:54.823 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T00:57:54.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.932+0000 7f2fd1c4b640 1 -- 172.21.15.169:0/2458950651 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2fcc1005b0 msgr2=0x7f2fcc1009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:54.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.932+0000 7f2fd1c4b640 1 --2- 172.21.15.169:0/2458950651 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2fcc1005b0 0x7f2fcc1009d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc00d960 tx=0x7f2fbc02b8e0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.932+0000 7f2fd1c4b640 1 -- 172.21.15.169:0/2458950651 shutdown_connections 2024-03-28T00:57:54.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.932+0000 7f2fd1c4b640 1 --2- 172.21.15.169:0/2458950651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2fcc102160 0x7f2fcc1089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.932+0000 7f2fd1c4b640 1 --2- 172.21.15.169:0/2458950651 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fcc1017a0 0x7f2fcc101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.932+0000 7f2fd1c4b640 1 --2- 172.21.15.169:0/2458950651 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2fcc1005b0 0x7f2fcc1009d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.932+0000 7f2fd1c4b640 1 -- 172.21.15.169:0/2458950651 >> 172.21.15.169:0/2458950651 conn(0x7f2fcc0fbb40 msgr2=0x7f2fcc0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:54.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.932+0000 7f2fd1c4b640 1 -- 172.21.15.169:0/2458950651 shutdown_connections 2024-03-28T00:57:54.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.932+0000 7f2fd1c4b640 1 -- 172.21.15.169:0/2458950651 wait complete. 2024-03-28T00:57:54.938 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.933+0000 7f2fd1c4b640 1 Processor -- start 2024-03-28T00:57:54.938 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.933+0000 7f2fd1c4b640 1 -- start start 2024-03-28T00:57:54.938 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.933+0000 7f2fd1c4b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2fcc1005b0 0x7f2fcc1957e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:54.938 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.933+0000 7f2fd1c4b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2fcc1017a0 0x7f2fcc195d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:54.938 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.933+0000 7f2fd1c4b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fcc102160 0x7f2fcc19cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:54.938 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.933+0000 7f2fd1c4b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc10b180 con 0x7f2fcc102160 2024-03-28T00:57:54.938 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.933+0000 7f2fd1c4b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc10b000 con 0x7f2fcc1017a0 2024-03-28T00:57:54.938 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.933+0000 7f2fd1c4b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc10b300 con 0x7f2fcc1005b0 2024-03-28T00:57:54.938 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.933+0000 7f2fcaffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2fcc1017a0 0x7f2fcc195d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:54.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.933+0000 7f2fcb7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2fcc1005b0 0x7f2fcc1957e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:54.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.933+0000 7f2fcbfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fcc102160 0x7f2fcc19cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:54.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.933+0000 7f2fcaffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2fcc1017a0 0x7f2fcc195d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:52976/0 (socket says 172.21.15.169:52976) 2024-03-28T00:57:54.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.933+0000 7f2fcb7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2fcc1005b0 0x7f2fcc1957e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:33552/0 (socket says 172.21.15.169:33552) 2024-03-28T00:57:54.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.933+0000 7f2fcaffd640 1 -- 172.21.15.169:0/3385426098 learned_addr learned my addr 172.21.15.169:0/3385426098 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:54.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.934+0000 7f2fcb7fe640 1 -- 172.21.15.169:0/3385426098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2fcc1017a0 msgr2=0x7f2fcc195d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:54.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.934+0000 7f2fcb7fe640 1 --2- 172.21.15.169:0/3385426098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2fcc1017a0 0x7f2fcc195d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.934+0000 7f2fcb7fe640 1 -- 172.21.15.169:0/3385426098 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fcc102160 msgr2=0x7f2fcc19cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:54.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.934+0000 7f2fcb7fe640 1 --2- 172.21.15.169:0/3385426098 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fcc102160 0x7f2fcc19cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.934+0000 7f2fcb7fe640 1 -- 172.21.15.169:0/3385426098 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc19d400 con 0x7f2fcc1005b0 2024-03-28T00:57:54.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.934+0000 7f2fcb7fe640 1 --2- 172.21.15.169:0/3385426098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2fcc1005b0 0x7f2fcc1957e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc030650 tx=0x7f2fbc0052f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:54.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.934+0000 7f2fc8ff9640 1 -- 172.21.15.169:0/3385426098 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc035040 con 0x7f2fcc1005b0 2024-03-28T00:57:54.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.934+0000 7f2fc8ff9640 1 -- 172.21.15.169:0/3385426098 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2fbc03f070 con 0x7f2fcc1005b0 2024-03-28T00:57:54.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.934+0000 7f2fc8ff9640 1 -- 172.21.15.169:0/3385426098 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc004920 con 0x7f2fcc1005b0 2024-03-28T00:57:54.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.934+0000 7f2fd1c4b640 1 -- 172.21.15.169:0/3385426098 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fcc19d690 con 0x7f2fcc1005b0 2024-03-28T00:57:54.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.934+0000 7f2fd1c4b640 1 -- 172.21.15.169:0/3385426098 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fcc19db80 con 0x7f2fcc1005b0 2024-03-28T00:57:54.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.937+0000 7f2fd483c740 1 Processor -- start 2024-03-28T00:57:54.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.937+0000 7f2fd483c740 1 -- start start 2024-03-28T00:57:54.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.937+0000 7f2fc8ff9640 1 -- 172.21.15.169:0/3385426098 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f2fbc038020 con 0x7f2fcc1005b0 2024-03-28T00:57:54.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.937+0000 7f2fd483c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f02b966da0 0x55f02b9671c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:54.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.937+0000 7f2fd483c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b967790 0x55f02b96c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:54.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.937+0000 7f2fd483c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b96c900 0x55f02b96ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:54.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.937+0000 7f2fd483c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f02b8cc3d0 con 0x55f02b966da0 2024-03-28T00:57:54.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.937+0000 7f2fd483c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f02b874ad0 con 0x55f02b96c900 2024-03-28T00:57:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.937+0000 7f2fd483c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f02b8c8470 con 0x55f02b967790 2024-03-28T00:57:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.937+0000 7f2fcb7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b967790 0x55f02b96c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.937+0000 7f2fcb7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b967790 0x55f02b96c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:33566/0 (socket says 172.21.15.169:33566) 2024-03-28T00:57:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.937+0000 7f2fcb7fe640 1 -- 172.21.15.169:0/1394019808 learned_addr learned my addr 172.21.15.169:0/1394019808 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.937+0000 7f2fcbfff640 1 --2- 172.21.15.169:0/1394019808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f02b966da0 0x55f02b9671c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.937+0000 7f2fcaffd640 1 --2- 172.21.15.169:0/1394019808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b96c900 0x55f02b96ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.938+0000 7f2fcb7fe640 1 -- 172.21.15.169:0/1394019808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b96c900 msgr2=0x55f02b96ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.938+0000 7f2fcb7fe640 1 --2- 172.21.15.169:0/1394019808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b96c900 0x55f02b96ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.938+0000 7f2fcb7fe640 1 -- 172.21.15.169:0/1394019808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f02b966da0 msgr2=0x55f02b9671c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.938+0000 7f2fcb7fe640 1 --2- 172.21.15.169:0/1394019808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f02b966da0 0x55f02b9671c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.938+0000 7f2fcb7fe640 1 -- 172.21.15.169:0/1394019808 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f02b6aa7d0 con 0x55f02b967790 2024-03-28T00:57:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.938+0000 7f2fcb7fe640 1 --2- 172.21.15.169:0/1394019808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b967790 0x55f02b96c290 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc0edee0 tx=0x7f2fbc0f6ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98a7a56eea200737 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.938+0000 7f2fc8ff9640 1 --2- 172.21.15.169:0/3385426098 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2fb40a2910 0x7f2fb40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.938+0000 7f2fb1ffb640 1 -- 172.21.15.169:0/1394019808 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc0ff070 con 0x55f02b967790 2024-03-28T00:57:54.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.938+0000 7f2fcbfff640 1 --2- 172.21.15.169:0/3385426098 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2fb40a2910 0x7f2fb40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:54.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.938+0000 7f2fb1ffb640 1 -- 172.21.15.169:0/1394019808 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2fbc0bd530 con 0x55f02b967790 2024-03-28T00:57:54.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.938+0000 7f2fc8ff9640 1 -- 172.21.15.169:0/3385426098 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2fbc0f0880 con 0x7f2fcc1005b0 2024-03-28T00:57:54.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.938+0000 7f2fb1ffb640 1 -- 172.21.15.169:0/1394019808 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc0bce30 con 0x55f02b967790 2024-03-28T00:57:54.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.939+0000 7f2fd483c740 1 -- 172.21.15.169:0/1394019808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b967790 msgr2=0x55f02b96c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:54.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.939+0000 7f2fd483c740 1 --2- 172.21.15.169:0/1394019808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b967790 0x55f02b96c290 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc0edee0 tx=0x7f2fbc0f6ef0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.939+0000 7f2fcbfff640 1 --2- 172.21.15.169:0/3385426098 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2fb40a2910 0x7f2fb40a4dd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x55f02b967ff0 tx=0x7f2fc4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:54.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.939+0000 7f2fd483c740 1 -- 172.21.15.169:0/1394019808 shutdown_connections 2024-03-28T00:57:54.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.939+0000 7f2fd483c740 1 --2- 172.21.15.169:0/1394019808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b96c900 0x55f02b96ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.939+0000 7f2fd483c740 1 --2- 172.21.15.169:0/1394019808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b967790 0x55f02b96c290 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.939+0000 7f2fd483c740 1 --2- 172.21.15.169:0/1394019808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f02b966da0 0x55f02b9671c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.939+0000 7f2fd483c740 1 -- 172.21.15.169:0/1394019808 >> 172.21.15.169:0/1394019808 conn(0x55f02b962310 msgr2=0x55f02b962710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:54.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.939+0000 7f2fd483c740 1 -- 172.21.15.169:0/1394019808 shutdown_connections 2024-03-28T00:57:54.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.939+0000 7f2fd483c740 1 -- 172.21.15.169:0/1394019808 wait complete. 2024-03-28T00:57:54.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.939+0000 7f2fd483c740 1 -- start start 2024-03-28T00:57:54.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.940+0000 7f2fd483c740 1 Processor -- start 2024-03-28T00:57:54.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.940+0000 7f2fd483c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b966da0 0x55f02b970f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:54.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.940+0000 7f2fd483c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f02b967790 0x55f02b971480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:54.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.940+0000 7f2fd483c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b96c900 0x55f02b9784c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:54.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.940+0000 7f2fd483c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f02b96f670 con 0x55f02b967790 2024-03-28T00:57:54.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.940+0000 7f2fd483c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f02b888060 con 0x55f02b966da0 2024-03-28T00:57:54.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.940+0000 7f2fd483c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f02b96f7f0 con 0x55f02b96c900 2024-03-28T00:57:54.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.940+0000 7f2fd483c740 10 client.0 _ll_register_callbacks cb 0x55f02b86a6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:54.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.940+0000 7f2fcbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b96c900 0x55f02b9784c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:54.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.940+0000 7f2fcaffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f02b967790 0x55f02b971480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:54.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.940+0000 7f2fcbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b96c900 0x55f02b9784c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:33578/0 (socket says 172.21.15.169:33578) 2024-03-28T00:57:54.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.940+0000 7f2fcbfff640 1 -- 172.21.15.169:0/1366801253 learned_addr learned my addr 172.21.15.169:0/1366801253 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:54.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.940+0000 7f2fcb7fe640 1 --2- 172.21.15.169:0/1366801253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b966da0 0x55f02b970f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:54.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.940+0000 7f2fcbfff640 1 -- 172.21.15.169:0/1366801253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b966da0 msgr2=0x55f02b970f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:54.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.940+0000 7f2fcbfff640 1 --2- 172.21.15.169:0/1366801253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b966da0 0x55f02b970f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.940+0000 7f2fcbfff640 1 -- 172.21.15.169:0/1366801253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f02b967790 msgr2=0x55f02b971480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:54.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.940+0000 7f2fcbfff640 1 --2- 172.21.15.169:0/1366801253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f02b967790 0x55f02b971480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.940+0000 7f2fcbfff640 1 -- 172.21.15.169:0/1366801253 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f02b5e03f0 con 0x55f02b96c900 2024-03-28T00:57:54.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.941+0000 7f2fcbfff640 1 --2- 172.21.15.169:0/1366801253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b96c900 0x55f02b9784c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4009cd0 tx=0x7f2fc4004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=183f898805beb3d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:54.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.941+0000 7f2faaffd640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:57:54.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.941+0000 7f2faaffd640 1 -- 172.21.15.169:0/1366801253 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc400f830 con 0x55f02b96c900 2024-03-28T00:57:54.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.941+0000 7f2faaffd640 1 -- 172.21.15.169:0/1366801253 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2fc400f9d0 con 0x55f02b96c900 2024-03-28T00:57:54.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.941+0000 7f2fd483c740 1 -- 172.21.15.169:0/1366801253 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2fbc0bd530 con 0x55f02b96c900 2024-03-28T00:57:54.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.941+0000 7f2faaffd640 1 -- 172.21.15.169:0/1366801253 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2fc400fbd0 con 0x55f02b96c900 2024-03-28T00:57:54.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.941+0000 7f2fd483c740 10 client.25615 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:54.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.941+0000 7f2fd483c740 1 -- 172.21.15.169:0/1366801253 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f02b5a00f0 con 0x55f02b96c900 2024-03-28T00:57:54.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.941+0000 7f2faaffd640 1 -- 172.21.15.169:0/1366801253 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2fbc0bd530 con 0x55f02b96c900 2024-03-28T00:57:54.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.942+0000 7f2faaffd640 1 -- 172.21.15.169:0/1366801253 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2fc4026070 con 0x55f02b96c900 2024-03-28T00:57:54.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.942+0000 7f2fd483c740 10 client.25615 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:54.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.942+0000 7f2fd483c740 10 client.25615 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:54.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.942+0000 7f2fd483c740 20 client.25615 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:54.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.942+0000 7f2fd483c740 1 --2- 172.21.15.169:0/1366801253 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f02b97cb40 0x55f02b97efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:54.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.942+0000 7f2fd483c740 4 client.25615 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:57:54.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.942+0000 7f2fd483c740 1 -- 172.21.15.169:0/1366801253 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f02b8d9910 con 0x55f02b97cb40 2024-03-28T00:57:54.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.942+0000 7f2fcb7fe640 1 --2- 172.21.15.169:0/1366801253 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f02b97cb40 0x55f02b97efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:54.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.942+0000 7f2fcb7fe640 1 --2- 172.21.15.169:0/1366801253 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f02b97cb40 0x55f02b97efa0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42a5b890115a5496 server_cookie=1c3eb3f14034f6ef in_seq=0 out_seq=0 2024-03-28T00:57:54.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.942+0000 7f2faaffd640 10 client.25615 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:54.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.943+0000 7f2faaffd640 1 -- 172.21.15.169:0/1366801253 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f02b8d9910 con 0x55f02b97cb40 2024-03-28T00:57:54.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.943+0000 7f2faaffd640 10 client.25615 handle_command_reply: tid=0 2024-03-28T00:57:54.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.943+0000 7f2fd483c740 1 client.25615 shutdown 2024-03-28T00:57:54.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.944+0000 7f2fd483c740 1 -- 172.21.15.169:0/1366801253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b96c900 msgr2=0x55f02b9784c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:54.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.944+0000 7f2fd483c740 1 --2- 172.21.15.169:0/1366801253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b96c900 0x55f02b9784c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4009cd0 tx=0x7f2fc4004ab0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.944+0000 7f2fd483c740 1 -- 172.21.15.169:0/1366801253 shutdown_connections 2024-03-28T00:57:54.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.944+0000 7f2fd483c740 1 --2- 172.21.15.169:0/1366801253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b96c900 0x55f02b9784c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.944+0000 7f2fd483c740 1 --2- 172.21.15.169:0/1366801253 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f02b97cb40 0x55f02b97efa0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.944+0000 7f2faaffd640 0 client.25615 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:54.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.944+0000 7f2fd483c740 1 --2- 172.21.15.169:0/1366801253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f02b967790 0x55f02b971480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.944+0000 7f2fd483c740 1 --2- 172.21.15.169:0/1366801253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b966da0 0x55f02b970f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.944+0000 7f2fd483c740 1 -- 172.21.15.169:0/1366801253 >> 172.21.15.169:0/1366801253 conn(0x55f02b962310 msgr2=0x55f02b961530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:54.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.944+0000 7f2fd483c740 1 -- 172.21.15.169:0/1366801253 shutdown_connections 2024-03-28T00:57:54.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.944+0000 7f2fd483c740 1 -- 172.21.15.169:0/1366801253 wait complete. 2024-03-28T00:57:54.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.944+0000 7f2fd483c740 20 client.25615 trim_cache size 0 max 16384 2024-03-28T00:57:54.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.960+0000 7f2fd483c740 1 Processor -- start 2024-03-28T00:57:54.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.960+0000 7f2fd483c740 1 -- start start 2024-03-28T00:57:54.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.960+0000 7f2fd483c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b966da0 0x55f02b970560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:54.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.960+0000 7f2fd483c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f02b967790 0x55f02b978480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:54.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.960+0000 7f2fd483c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b96c900 0x55f02b960ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:54.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.960+0000 7f2fd483c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc1aa440 con 0x55f02b967790 2024-03-28T00:57:54.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.960+0000 7f2fd483c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc1a7f40 con 0x55f02b966da0 2024-03-28T00:57:54.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.960+0000 7f2fd483c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc1a5fe0 con 0x55f02b96c900 2024-03-28T00:57:54.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.960+0000 7f2fcb7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b966da0 0x55f02b970560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:54.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.960+0000 7f2fcaffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f02b967790 0x55f02b978480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:54.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.960+0000 7f2fcb7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b966da0 0x55f02b970560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:53008/0 (socket says 172.21.15.169:53008) 2024-03-28T00:57:54.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.960+0000 7f2fcb7fe640 1 -- 172.21.15.169:0/1226568831 learned_addr learned my addr 172.21.15.169:0/1226568831 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:54.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.960+0000 7f2fcbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b96c900 0x55f02b960ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:54.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.960+0000 7f2fcb7fe640 1 -- 172.21.15.169:0/1226568831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b96c900 msgr2=0x55f02b960ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:54.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.960+0000 7f2fcb7fe640 1 --2- 172.21.15.169:0/1226568831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b96c900 0x55f02b960ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.960+0000 7f2fcb7fe640 1 -- 172.21.15.169:0/1226568831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f02b967790 msgr2=0x55f02b978480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:54.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.960+0000 7f2fcb7fe640 1 --2- 172.21.15.169:0/1226568831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f02b967790 0x55f02b978480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.960+0000 7f2fcb7fe640 1 -- 172.21.15.169:0/1226568831 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f02b978fb0 con 0x55f02b966da0 2024-03-28T00:57:54.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.961+0000 7f2fcb7fe640 1 --2- 172.21.15.169:0/1226568831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b966da0 0x55f02b970560 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc0f8730 tx=0x7f2fbc0bc970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7c2dc87f530b4b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:54.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.961+0000 7f2fb17fa640 1 -- 172.21.15.169:0/1226568831 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc0ff040 con 0x55f02b966da0 2024-03-28T00:57:54.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.961+0000 7f2fb17fa640 1 -- 172.21.15.169:0/1226568831 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2fbc0044b0 con 0x55f02b966da0 2024-03-28T00:57:54.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.961+0000 7f2fb17fa640 1 -- 172.21.15.169:0/1226568831 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc0bd530 con 0x55f02b966da0 2024-03-28T00:57:54.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.961+0000 7f2fd483c740 1 -- 172.21.15.169:0/1226568831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b966da0 msgr2=0x55f02b970560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:54.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.961+0000 7f2fd483c740 1 --2- 172.21.15.169:0/1226568831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b966da0 0x55f02b970560 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc0f8730 tx=0x7f2fbc0bc970 comp rx=0 tx=0).stop 2024-03-28T00:57:54.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.961+0000 7f2fd483c740 1 -- 172.21.15.169:0/1226568831 shutdown_connections 2024-03-28T00:57:54.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.961+0000 7f2fd483c740 1 --2- 172.21.15.169:0/1226568831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b96c900 0x55f02b960ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.961+0000 7f2fd483c740 1 --2- 172.21.15.169:0/1226568831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f02b967790 0x55f02b978480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.961+0000 7f2fd483c740 1 --2- 172.21.15.169:0/1226568831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b966da0 0x55f02b970560 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.962+0000 7f2fd483c740 1 -- 172.21.15.169:0/1226568831 >> 172.21.15.169:0/1226568831 conn(0x55f02b962310 msgr2=0x55f02b970f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:54.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.962+0000 7f2fd483c740 1 -- 172.21.15.169:0/1226568831 shutdown_connections 2024-03-28T00:57:54.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.962+0000 7f2fd483c740 1 -- 172.21.15.169:0/1226568831 wait complete. 2024-03-28T00:57:54.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.962+0000 7f2fd483c740 1 -- start start 2024-03-28T00:57:54.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.962+0000 7f2fd483c740 1 Processor -- start 2024-03-28T00:57:54.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.962+0000 7f2fd483c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b966da0 0x55f02b976100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:54.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.962+0000 7f2fd483c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b967790 0x55f02b976640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:54.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.962+0000 7f2fd483c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f02b96c900 0x55f02b977600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:54.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.962+0000 7f2fd483c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc1a65e0 con 0x55f02b96c900 2024-03-28T00:57:54.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.962+0000 7f2fd483c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc1a78e0 con 0x55f02b966da0 2024-03-28T00:57:54.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.962+0000 7f2fd483c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc1a6c20 con 0x55f02b967790 2024-03-28T00:57:54.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.962+0000 7f2fd483c740 10 client.0 _ll_register_callbacks cb 0x7f2fc4009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:54.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.963+0000 7f2fcbfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f02b96c900 0x55f02b977600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:54.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.963+0000 7f2fcb7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b966da0 0x55f02b976100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:54.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.963+0000 7f2fcbfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f02b96c900 0x55f02b977600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33106/0 (socket says 172.21.15.169:33106) 2024-03-28T00:57:54.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.963+0000 7f2fcbfff640 1 -- 172.21.15.169:0/351469525 learned_addr learned my addr 172.21.15.169:0/351469525 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:54.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.963+0000 7f2fcaffd640 1 --2- 172.21.15.169:0/351469525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b967790 0x55f02b976640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:54.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.963+0000 7f2fcb7fe640 1 -- 172.21.15.169:0/351469525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b967790 msgr2=0x55f02b976640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:54.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.963+0000 7f2fcb7fe640 1 --2- 172.21.15.169:0/351469525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b967790 0x55f02b976640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.963+0000 7f2fcb7fe640 1 -- 172.21.15.169:0/351469525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f02b96c900 msgr2=0x55f02b977600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:54.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.963+0000 7f2fcb7fe640 1 --2- 172.21.15.169:0/351469525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f02b96c900 0x55f02b977600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.963+0000 7f2fcb7fe640 1 -- 172.21.15.169:0/351469525 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f02b977bf0 con 0x55f02b966da0 2024-03-28T00:57:54.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.963+0000 7f2fcb7fe640 1 --2- 172.21.15.169:0/351469525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b966da0 0x55f02b976100 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc02be20 tx=0x7f2fbc032e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cd5274aa8b97e61 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:54.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.963+0000 7f2faaffd640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:57:54.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.963+0000 7f2faaffd640 1 -- 172.21.15.169:0/351469525 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc0b90d0 con 0x55f02b966da0 2024-03-28T00:57:54.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.963+0000 7f2faaffd640 1 -- 172.21.15.169:0/351469525 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2fbc0f6300 con 0x55f02b966da0 2024-03-28T00:57:54.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.963+0000 7f2fd483c740 1 -- 172.21.15.169:0/351469525 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2fbc0044b0 con 0x55f02b966da0 2024-03-28T00:57:54.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.963+0000 7f2faaffd640 1 -- 172.21.15.169:0/351469525 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2fbc0bd530 con 0x55f02b966da0 2024-03-28T00:57:54.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.963+0000 7f2fd483c740 10 client.25649 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:54.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.963+0000 7f2fd483c740 1 -- 172.21.15.169:0/351469525 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f02b977bf0 con 0x55f02b966da0 2024-03-28T00:57:54.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.963+0000 7f2faaffd640 1 -- 172.21.15.169:0/351469525 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2fbc0c0f80 con 0x55f02b966da0 2024-03-28T00:57:54.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.964+0000 7f2faaffd640 1 -- 172.21.15.169:0/351469525 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2fbc04b020 con 0x55f02b966da0 2024-03-28T00:57:54.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.964+0000 7f2fd483c740 10 client.25649 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:54.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.964+0000 7f2fd483c740 10 client.25649 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:54.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.964+0000 7f2fd483c740 20 client.25649 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:54.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.964+0000 7f2fd483c740 1 --2- 172.21.15.169:0/351469525 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f02b9d8d30 0x55f02b9db120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:54.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.964+0000 7f2fd483c740 4 client.25649 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T00:57:54.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.964+0000 7f2fd483c740 1 -- 172.21.15.169:0/351469525 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f2fbc0044b0 con 0x55f02b9d8d30 2024-03-28T00:57:54.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.964+0000 7f2fcaffd640 1 --2- 172.21.15.169:0/351469525 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f02b9d8d30 0x55f02b9db120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:54.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.964+0000 7f2faaffd640 10 client.25649 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:54.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.964+0000 7f2fcaffd640 1 --2- 172.21.15.169:0/351469525 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f02b9d8d30 0x55f02b9db120 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=522d40bb69360397 server_cookie=8962aedcb603b9b4 in_seq=0 out_seq=0 2024-03-28T00:57:54.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.965+0000 7f2faaffd640 1 -- 172.21.15.169:0/351469525 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x7f2fbc0044b0 con 0x55f02b9d8d30 2024-03-28T00:57:54.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.965+0000 7f2faaffd640 10 client.25649 handle_command_reply: tid=0 2024-03-28T00:57:54.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.965+0000 7f2fd483c740 1 client.25649 shutdown 2024-03-28T00:57:54.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.965+0000 7f2fd483c740 1 -- 172.21.15.169:0/351469525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b966da0 msgr2=0x55f02b976100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:54.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.965+0000 7f2fd483c740 1 --2- 172.21.15.169:0/351469525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b966da0 0x55f02b976100 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc02be20 tx=0x7f2fbc032e40 comp rx=0 tx=0).stop 2024-03-28T00:57:54.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.965+0000 7f2fd483c740 1 -- 172.21.15.169:0/351469525 shutdown_connections 2024-03-28T00:57:54.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.965+0000 7f2fd483c740 1 --2- 172.21.15.169:0/351469525 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f02b9d8d30 0x55f02b9db120 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.965+0000 7f2fd483c740 1 --2- 172.21.15.169:0/351469525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f02b96c900 0x55f02b977600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.965+0000 7f2faaffd640 0 client.25649 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:54.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.965+0000 7f2fd483c740 1 --2- 172.21.15.169:0/351469525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f02b967790 0x55f02b976640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.965+0000 7f2fd483c740 1 --2- 172.21.15.169:0/351469525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f02b966da0 0x55f02b976100 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.965+0000 7f2fd483c740 1 -- 172.21.15.169:0/351469525 >> 172.21.15.169:0/351469525 conn(0x55f02b962310 msgr2=0x55f02b970f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:54.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.965+0000 7f2fd483c740 1 -- 172.21.15.169:0/351469525 shutdown_connections 2024-03-28T00:57:54.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.965+0000 7f2fd483c740 1 -- 172.21.15.169:0/351469525 wait complete. 2024-03-28T00:57:54.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.965+0000 7f2fd483c740 20 client.25649 trim_cache size 0 max 16384 2024-03-28T00:57:54.973 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:57:54.973 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "no active scrubs running", 2024-03-28T00:57:54.973 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": {} 2024-03-28T00:57:54.973 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:57:54.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.966+0000 7f2fd1c4b640 1 -- 172.21.15.169:0/3385426098 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2fb40a2910 msgr2=0x7f2fb40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:54.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.966+0000 7f2fd1c4b640 1 --2- 172.21.15.169:0/3385426098 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2fb40a2910 0x7f2fb40a4dd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x55f02b967ff0 tx=0x7f2fc4009500 comp rx=0 tx=0).stop 2024-03-28T00:57:54.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.966+0000 7f2fd1c4b640 1 -- 172.21.15.169:0/3385426098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2fcc1005b0 msgr2=0x7f2fcc1957e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:54.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.966+0000 7f2fd1c4b640 1 --2- 172.21.15.169:0/3385426098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2fcc1005b0 0x7f2fcc1957e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc030650 tx=0x7f2fbc0052f0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.966+0000 7f2fd1c4b640 1 -- 172.21.15.169:0/3385426098 shutdown_connections 2024-03-28T00:57:54.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.966+0000 7f2fd1c4b640 1 --2- 172.21.15.169:0/3385426098 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2fb40a2910 0x7f2fb40a4dd0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.966+0000 7f2fd1c4b640 1 --2- 172.21.15.169:0/3385426098 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fcc102160 0x7f2fcc19cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.966+0000 7f2fd1c4b640 1 --2- 172.21.15.169:0/3385426098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2fcc1017a0 0x7f2fcc195d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.966+0000 7f2fd1c4b640 1 --2- 172.21.15.169:0/3385426098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2fcc1005b0 0x7f2fcc1957e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:54.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.966+0000 7f2fd1c4b640 1 -- 172.21.15.169:0/3385426098 >> 172.21.15.169:0/3385426098 conn(0x7f2fcc0fbb40 msgr2=0x7f2fcc104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:54.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.966+0000 7f2fd1c4b640 1 -- 172.21.15.169:0/3385426098 shutdown_connections 2024-03-28T00:57:54.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:54.966+0000 7f2fd1c4b640 1 -- 172.21.15.169:0/3385426098 wait complete. 2024-03-28T00:57:54.980 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T00:57:54.980 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:57:55.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.084+0000 7f4a71fdf640 1 -- 172.21.15.169:0/3082200087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a6c102140 msgr2=0x7f4a6c104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.084+0000 7f4a71fdf640 1 --2- 172.21.15.169:0/3082200087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a6c102140 0x7f4a6c104530 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c009900 tx=0x7f4a5c02bc30 comp rx=0 tx=0).stop 2024-03-28T00:57:55.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.084+0000 7f4a71fdf640 1 -- 172.21.15.169:0/3082200087 shutdown_connections 2024-03-28T00:57:55.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.084+0000 7f4a71fdf640 1 --2- 172.21.15.169:0/3082200087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a6c1073a0 0x7f4a6c1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.084+0000 7f4a71fdf640 1 --2- 172.21.15.169:0/3082200087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a6c104a70 0x7f4a6c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.084+0000 7f4a71fdf640 1 --2- 172.21.15.169:0/3082200087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a6c102140 0x7f4a6c104530 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.084+0000 7f4a71fdf640 1 -- 172.21.15.169:0/3082200087 >> 172.21.15.169:0/3082200087 conn(0x7f4a6c0fbb30 msgr2=0x7f4a6c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:55.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.084+0000 7f4a71fdf640 1 -- 172.21.15.169:0/3082200087 shutdown_connections 2024-03-28T00:57:55.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.084+0000 7f4a71fdf640 1 -- 172.21.15.169:0/3082200087 wait complete. 2024-03-28T00:57:55.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.085+0000 7f4a71fdf640 1 Processor -- start 2024-03-28T00:57:55.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.085+0000 7f4a71fdf640 1 -- start start 2024-03-28T00:57:55.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.085+0000 7f4a71fdf640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a6c102140 0x7f4a6c1979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.085+0000 7f4a71fdf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a6c104a70 0x7f4a6c197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.085+0000 7f4a71fdf640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a6c1073a0 0x7f4a6c19ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.085+0000 7f4a6b7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a6c102140 0x7f4a6c1979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.085+0000 7f4a71fdf640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c10d3c0 con 0x7f4a6c102140 2024-03-28T00:57:55.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.085+0000 7f4a71fdf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c10d240 con 0x7f4a6c104a70 2024-03-28T00:57:55.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.085+0000 7f4a71fdf640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c10d540 con 0x7f4a6c1073a0 2024-03-28T00:57:55.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.085+0000 7f4a6affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a6c104a70 0x7f4a6c197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.085+0000 7f4a6affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a6c104a70 0x7f4a6c197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:53044/0 (socket says 172.21.15.169:53044) 2024-03-28T00:57:55.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.085+0000 7f4a6b7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a6c102140 0x7f4a6c1979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33124/0 (socket says 172.21.15.169:33124) 2024-03-28T00:57:55.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.085+0000 7f4a6affd640 1 -- 172.21.15.169:0/3211638895 learned_addr learned my addr 172.21.15.169:0/3211638895 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:55.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.085+0000 7f4a6bfff640 1 --2- 172.21.15.169:0/3211638895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a6c1073a0 0x7f4a6c19ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.086+0000 7f4a6affd640 1 -- 172.21.15.169:0/3211638895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a6c1073a0 msgr2=0x7f4a6c19ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.086+0000 7f4a6affd640 1 --2- 172.21.15.169:0/3211638895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a6c1073a0 0x7f4a6c19ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.086+0000 7f4a6affd640 1 -- 172.21.15.169:0/3211638895 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a6c102140 msgr2=0x7f4a6c1979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.086+0000 7f4a6affd640 1 --2- 172.21.15.169:0/3211638895 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a6c102140 0x7f4a6c1979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.086+0000 7f4a6affd640 1 -- 172.21.15.169:0/3211638895 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6c19f600 con 0x7f4a6c104a70 2024-03-28T00:57:55.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.086+0000 7f4a6affd640 1 --2- 172.21.15.169:0/3211638895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a6c104a70 0x7f4a6c197f20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000e9f0 tx=0x7f4a6000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:55.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.086+0000 7f4a68ff9640 1 -- 172.21.15.169:0/3211638895 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6000ccc0 con 0x7f4a6c104a70 2024-03-28T00:57:55.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.086+0000 7f4a68ff9640 1 -- 172.21.15.169:0/3211638895 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a60005500 con 0x7f4a6c104a70 2024-03-28T00:57:55.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.086+0000 7f4a68ff9640 1 -- 172.21.15.169:0/3211638895 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a60010690 con 0x7f4a6c104a70 2024-03-28T00:57:55.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.086+0000 7f4a71fdf640 1 -- 172.21.15.169:0/3211638895 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6c0fff90 con 0x7f4a6c104a70 2024-03-28T00:57:55.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.086+0000 7f4a71fdf640 1 -- 172.21.15.169:0/3211638895 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6c100490 con 0x7f4a6c104a70 2024-03-28T00:57:55.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.087+0000 7f4a71fdf640 1 -- 172.21.15.169:0/3211638895 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a6c0fd620 con 0x7f4a6c104a70 2024-03-28T00:57:55.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.089+0000 7f4a68ff9640 1 -- 172.21.15.169:0/3211638895 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f4a600046c0 con 0x7f4a6c104a70 2024-03-28T00:57:55.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.090+0000 7f4a68ff9640 1 --2- 172.21.15.169:0/3211638895 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4a540a2910 0x7f4a540a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.090+0000 7f4a68ff9640 1 -- 172.21.15.169:0/3211638895 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f4a600cf450 con 0x7f4a6c104a70 2024-03-28T00:57:55.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.090+0000 7f4a6b7fe640 1 --2- 172.21.15.169:0/3211638895 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4a540a2910 0x7f4a540a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.090+0000 7f4a6b7fe640 1 --2- 172.21.15.169:0/3211638895 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4a540a2910 0x7f4a540a4dd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c02c240 tx=0x7f4a5c03f040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:55.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.093+0000 7f4a68ff9640 1 -- 172.21.15.169:0/3211638895 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4a60053740 con 0x7f4a6c104a70 2024-03-28T00:57:55.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.319+0000 7f4a71fdf640 1 -- 172.21.15.169:0/3211638895 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a6c02d070 con 0x7f4a6c104a70 2024-03-28T00:57:55.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.320+0000 7f4a68ff9640 1 -- 172.21.15.169:0/3211638895 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7f4a6009bff0 con 0x7f4a6c104a70 2024-03-28T00:57:55.323 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:57:55.323 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T00:57:55.324 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T00:57:55.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.323+0000 7f4a71fdf640 1 -- 172.21.15.169:0/3211638895 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4a540a2910 msgr2=0x7f4a540a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.323+0000 7f4a71fdf640 1 --2- 172.21.15.169:0/3211638895 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4a540a2910 0x7f4a540a4dd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c02c240 tx=0x7f4a5c03f040 comp rx=0 tx=0).stop 2024-03-28T00:57:55.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.323+0000 7f4a71fdf640 1 -- 172.21.15.169:0/3211638895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a6c104a70 msgr2=0x7f4a6c197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.323+0000 7f4a71fdf640 1 --2- 172.21.15.169:0/3211638895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a6c104a70 0x7f4a6c197f20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000e9f0 tx=0x7f4a6000eeb0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.323+0000 7f4a71fdf640 1 -- 172.21.15.169:0/3211638895 shutdown_connections 2024-03-28T00:57:55.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.323+0000 7f4a71fdf640 1 --2- 172.21.15.169:0/3211638895 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4a540a2910 0x7f4a540a4dd0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.323+0000 7f4a71fdf640 1 --2- 172.21.15.169:0/3211638895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a6c1073a0 0x7f4a6c19ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.323+0000 7f4a71fdf640 1 --2- 172.21.15.169:0/3211638895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a6c104a70 0x7f4a6c197f20 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.323+0000 7f4a71fdf640 1 --2- 172.21.15.169:0/3211638895 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a6c102140 0x7f4a6c1979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.323+0000 7f4a71fdf640 1 -- 172.21.15.169:0/3211638895 >> 172.21.15.169:0/3211638895 conn(0x7f4a6c0fbb30 msgr2=0x7f4a6c102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:55.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.323+0000 7f4a71fdf640 1 -- 172.21.15.169:0/3211638895 shutdown_connections 2024-03-28T00:57:55.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.323+0000 7f4a71fdf640 1 -- 172.21.15.169:0/3211638895 wait complete. 2024-03-28T00:57:55.337 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T00:57:55.337 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T00:57:55.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.442+0000 7f58489e1640 1 -- 172.21.15.169:0/1709262698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5844107400 msgr2=0x7f58441097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.442+0000 7f58489e1640 1 --2- 172.21.15.169:0/1709262698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5844107400 0x7f58441097f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f583c009be0 tx=0x7f583c02bef0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.442+0000 7f58489e1640 1 -- 172.21.15.169:0/1709262698 shutdown_connections 2024-03-28T00:57:55.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.442+0000 7f58489e1640 1 --2- 172.21.15.169:0/1709262698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5844107400 0x7f58441097f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.442+0000 7f58489e1640 1 --2- 172.21.15.169:0/1709262698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5844104ad0 0x7f5844106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.442+0000 7f58489e1640 1 --2- 172.21.15.169:0/1709262698 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58441021a0 0x7f5844104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.442+0000 7f58489e1640 1 -- 172.21.15.169:0/1709262698 >> 172.21.15.169:0/1709262698 conn(0x7f58440fbb30 msgr2=0x7f58440fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:55.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.442+0000 7f58489e1640 1 -- 172.21.15.169:0/1709262698 shutdown_connections 2024-03-28T00:57:55.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.442+0000 7f58489e1640 1 -- 172.21.15.169:0/1709262698 wait complete. 2024-03-28T00:57:55.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.443+0000 7f58489e1640 1 Processor -- start 2024-03-28T00:57:55.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.443+0000 7f58489e1640 1 -- start start 2024-03-28T00:57:55.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.443+0000 7f58489e1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58441021a0 0x7f58441979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.443+0000 7f58489e1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5844104ad0 0x7f5844197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.443+0000 7f58489e1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5844107400 0x7f584419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.443+0000 7f58489e1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584410d330 con 0x7f5844104ad0 2024-03-28T00:57:55.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.443+0000 7f5841d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5844104ad0 0x7f5844197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.443+0000 7f58489e1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584410d1b0 con 0x7f58441021a0 2024-03-28T00:57:55.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.443+0000 7f5841d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5844104ad0 0x7f5844197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33140/0 (socket says 172.21.15.169:33140) 2024-03-28T00:57:55.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.443+0000 7f5841d9b640 1 -- 172.21.15.169:0/2848815808 learned_addr learned my addr 172.21.15.169:0/2848815808 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:55.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.443+0000 7f5842d9d640 1 --2- 172.21.15.169:0/2848815808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5844107400 0x7f584419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.443+0000 7f58489e1640 1 -- 172.21.15.169:0/2848815808 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584410d4b0 con 0x7f5844107400 2024-03-28T00:57:55.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.443+0000 7f584259c640 1 --2- 172.21.15.169:0/2848815808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58441021a0 0x7f58441979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.444+0000 7f5842d9d640 1 -- 172.21.15.169:0/2848815808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58441021a0 msgr2=0x7f58441979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.444+0000 7f5842d9d640 1 --2- 172.21.15.169:0/2848815808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58441021a0 0x7f58441979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.444+0000 7f5842d9d640 1 -- 172.21.15.169:0/2848815808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5844104ad0 msgr2=0x7f5844197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.444+0000 7f5842d9d640 1 --2- 172.21.15.169:0/2848815808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5844104ad0 0x7f5844197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.444+0000 7f5842d9d640 1 -- 172.21.15.169:0/2848815808 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584419f5f0 con 0x7f5844107400 2024-03-28T00:57:55.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.444+0000 7f584259c640 1 --2- 172.21.15.169:0/2848815808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58441021a0 0x7f58441979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:57:55.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.444+0000 7f5842d9d640 1 --2- 172.21.15.169:0/2848815808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5844107400 0x7f584419ef80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f583c02c430 tx=0x7f583c02cb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:55.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.444+0000 7f58337fe640 1 -- 172.21.15.169:0/2848815808 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c043020 con 0x7f5844107400 2024-03-28T00:57:55.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.444+0000 7f58337fe640 1 -- 172.21.15.169:0/2848815808 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f583c003e50 con 0x7f5844107400 2024-03-28T00:57:55.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.444+0000 7f58489e1640 1 -- 172.21.15.169:0/2848815808 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f584419f880 con 0x7f5844107400 2024-03-28T00:57:55.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.444+0000 7f58337fe640 1 -- 172.21.15.169:0/2848815808 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c004860 con 0x7f5844107400 2024-03-28T00:57:55.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.444+0000 7f58489e1640 1 -- 172.21.15.169:0/2848815808 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f584419fd60 con 0x7f5844107400 2024-03-28T00:57:55.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.447+0000 7f58337fe640 1 -- 172.21.15.169:0/2848815808 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f583c046020 con 0x7f5844107400 2024-03-28T00:57:55.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.447+0000 7f584b5c5740 1 Processor -- start 2024-03-28T00:57:55.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.448+0000 7f584b5c5740 1 -- start start 2024-03-28T00:57:55.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.448+0000 7f58337fe640 1 --2- 172.21.15.169:0/2848815808 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f582c0a2990 0x7f582c0a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.448+0000 7f584b5c5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971c9690 0x5640971c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.448+0000 7f584b5c5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640971ca080 0x5640971ceb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.448+0000 7f584b5c5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971cf1f0 0x5640971d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.448+0000 7f584b5c5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56409718c610 con 0x5640971c9690 2024-03-28T00:57:55.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.448+0000 7f584b5c5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x56409712f3c0 con 0x5640971ca080 2024-03-28T00:57:55.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.448+0000 7f584b5c5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640970d7d60 con 0x5640971cf1f0 2024-03-28T00:57:55.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.448+0000 7f58337fe640 1 -- 172.21.15.169:0/2848815808 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f583c0f1460 con 0x7f5844107400 2024-03-28T00:57:55.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.448+0000 7f584259c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971c9690 0x5640971c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.448+0000 7f584259c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971c9690 0x5640971c9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33144/0 (socket says 172.21.15.169:33144) 2024-03-28T00:57:55.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.448+0000 7f584259c640 1 -- 172.21.15.169:0/2393365528 learned_addr learned my addr 172.21.15.169:0/2393365528 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:55.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.448+0000 7f5841d9b640 1 --2- 172.21.15.169:0/2848815808 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f582c0a2990 0x7f582c0a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.448+0000 7f5842d9d640 1 --2- 172.21.15.169:0/2393365528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971cf1f0 0x5640971d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.448+0000 7f5842d9d640 1 -- 172.21.15.169:0/2393365528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640971ca080 msgr2=0x5640971ceb80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T00:57:55.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.449+0000 7f5842d9d640 1 --2- 172.21.15.169:0/2393365528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640971ca080 0x5640971ceb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.449+0000 7f5842d9d640 1 -- 172.21.15.169:0/2393365528 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971c9690 msgr2=0x5640971c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.449+0000 7f5842d9d640 1 --2- 172.21.15.169:0/2393365528 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971c9690 0x5640971c9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.449+0000 7f5842d9d640 1 -- 172.21.15.169:0/2393365528 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564096e433f0 con 0x5640971cf1f0 2024-03-28T00:57:55.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.449+0000 7f584259c640 1 --2- 172.21.15.169:0/2393365528 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971c9690 0x5640971c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:55.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.449+0000 7f5842d9d640 1 --2- 172.21.15.169:0/2393365528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971cf1f0 0x5640971d1680 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f583c0eeb20 tx=0x7f583c009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cddd733cd3feea4a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:55.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.449+0000 7f5830ff9640 1 -- 172.21.15.169:0/2393365528 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c004040 con 0x5640971cf1f0 2024-03-28T00:57:55.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.449+0000 7f5841d9b640 1 --2- 172.21.15.169:0/2848815808 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f582c0a2990 0x7f582c0a4e50 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5844198ee0 tx=0x7f5838005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:55.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.449+0000 7f5830ff9640 1 -- 172.21.15.169:0/2393365528 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f583c0be0b0 con 0x5640971cf1f0 2024-03-28T00:57:55.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.449+0000 7f5830ff9640 1 -- 172.21.15.169:0/2393365528 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c0eec60 con 0x5640971cf1f0 2024-03-28T00:57:55.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.450+0000 7f584b5c5740 1 -- 172.21.15.169:0/2393365528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971cf1f0 msgr2=0x5640971d1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.450+0000 7f584b5c5740 1 --2- 172.21.15.169:0/2393365528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971cf1f0 0x5640971d1680 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f583c0eeb20 tx=0x7f583c009500 comp rx=0 tx=0).stop 2024-03-28T00:57:55.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.450+0000 7f584b5c5740 1 -- 172.21.15.169:0/2393365528 shutdown_connections 2024-03-28T00:57:55.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.450+0000 7f584b5c5740 1 --2- 172.21.15.169:0/2393365528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971cf1f0 0x5640971d1680 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.450+0000 7f584b5c5740 1 --2- 172.21.15.169:0/2393365528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640971ca080 0x5640971ceb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.450+0000 7f584b5c5740 1 --2- 172.21.15.169:0/2393365528 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971c9690 0x5640971c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.450+0000 7f584b5c5740 1 -- 172.21.15.169:0/2393365528 >> 172.21.15.169:0/2393365528 conn(0x5640971c4c00 msgr2=0x5640971c5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:55.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.450+0000 7f584b5c5740 1 -- 172.21.15.169:0/2393365528 shutdown_connections 2024-03-28T00:57:55.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.450+0000 7f584b5c5740 1 -- 172.21.15.169:0/2393365528 wait complete. 2024-03-28T00:57:55.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.450+0000 7f584b5c5740 1 -- start start 2024-03-28T00:57:55.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.450+0000 7f584b5c5740 1 Processor -- start 2024-03-28T00:57:55.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.451+0000 7f584b5c5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971c9690 0x5640971d3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.451+0000 7f584b5c5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971ca080 0x5640971d3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.451+0000 7f584b5c5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640971cf1f0 0x5640971dad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.451+0000 7f584b5c5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640970eb240 con 0x5640971c9690 2024-03-28T00:57:55.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.451+0000 7f584b5c5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x56409712b460 con 0x5640971cf1f0 2024-03-28T00:57:55.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.451+0000 7f584b5c5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640970d7d60 con 0x5640971ca080 2024-03-28T00:57:55.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.451+0000 7f584b5c5740 10 client.0 _ll_register_callbacks cb 0x5640970eac10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:55.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.451+0000 7f584259c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971c9690 0x5640971d3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.451+0000 7f584259c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971c9690 0x5640971d3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33146/0 (socket says 172.21.15.169:33146) 2024-03-28T00:57:55.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.451+0000 7f584259c640 1 -- 172.21.15.169:0/1916638893 learned_addr learned my addr 172.21.15.169:0/1916638893 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:55.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.451+0000 7f5842d9d640 1 --2- 172.21.15.169:0/1916638893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640971cf1f0 0x5640971dad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.451+0000 7f5841d9b640 1 --2- 172.21.15.169:0/1916638893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971ca080 0x5640971d3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.451+0000 7f584259c640 1 -- 172.21.15.169:0/1916638893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971ca080 msgr2=0x5640971d3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.451+0000 7f584259c640 1 --2- 172.21.15.169:0/1916638893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971ca080 0x5640971d3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.451+0000 7f584259c640 1 -- 172.21.15.169:0/1916638893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640971cf1f0 msgr2=0x5640971dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.451+0000 7f584259c640 1 --2- 172.21.15.169:0/1916638893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640971cf1f0 0x5640971dad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.451+0000 7f584259c640 1 -- 172.21.15.169:0/1916638893 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564096f0d7d0 con 0x5640971c9690 2024-03-28T00:57:55.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.451+0000 7f584259c640 1 --2- 172.21.15.169:0/1916638893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971c9690 0x5640971d3780 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f58340126b0 tx=0x7f5834012b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41016df585afb2be server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:55.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.451+0000 7f5829ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:57:55.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.452+0000 7f5829ffb640 1 -- 172.21.15.169:0/1916638893 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5834011b50 con 0x5640971c9690 2024-03-28T00:57:55.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.452+0000 7f5829ffb640 1 -- 172.21.15.169:0/1916638893 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f5834011cf0 con 0x5640971c9690 2024-03-28T00:57:55.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.452+0000 7f584b5c5740 1 -- 172.21.15.169:0/1916638893 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f583c0be0b0 con 0x5640971c9690 2024-03-28T00:57:55.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.452+0000 7f5829ffb640 1 -- 172.21.15.169:0/1916638893 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5834025050 con 0x5640971c9690 2024-03-28T00:57:55.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.452+0000 7f584b5c5740 10 client.15843 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:55.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.452+0000 7f584b5c5740 1 -- 172.21.15.169:0/1916638893 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5640971de140 con 0x5640971c9690 2024-03-28T00:57:55.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.452+0000 7f5829ffb640 1 -- 172.21.15.169:0/1916638893 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f583c0be0b0 con 0x5640971c9690 2024-03-28T00:57:55.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.452+0000 7f5829ffb640 1 -- 172.21.15.169:0/1916638893 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f583400f660 con 0x5640971c9690 2024-03-28T00:57:55.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.452+0000 7f584b5c5740 10 client.15843 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:55.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.452+0000 7f584b5c5740 10 client.15843 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T00:57:55.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.452+0000 7f584b5c5740 20 client.15843 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:55.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.452+0000 7f584b5c5740 1 --2- 172.21.15.169:0/1916638893 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x5640971df7f0 0x5640971e1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.452+0000 7f584b5c5740 4 client.15843 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:57:55.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.452+0000 7f584b5c5740 1 -- 172.21.15.169:0/1916638893 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56409713c900 con 0x5640971df7f0 2024-03-28T00:57:55.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.453+0000 7f5841d9b640 1 --2- 172.21.15.169:0/1916638893 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x5640971df7f0 0x5640971e1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.453+0000 7f5841d9b640 1 --2- 172.21.15.169:0/1916638893 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x5640971df7f0 0x5640971e1c30 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=76f096957e9bc9f6 server_cookie=222d2daf7f70a996 in_seq=0 out_seq=0 2024-03-28T00:57:55.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.453+0000 7f5829ffb640 10 client.15843 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T00:57:55.461 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.454+0000 7f5829ffb640 1 -- 172.21.15.169:0/1916638893 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56409713c900 con 0x5640971df7f0 2024-03-28T00:57:55.461 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.454+0000 7f5829ffb640 10 client.15843 handle_command_reply: tid=0 2024-03-28T00:57:55.461 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.454+0000 7f584b5c5740 1 client.15843 shutdown 2024-03-28T00:57:55.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.454+0000 7f584b5c5740 1 -- 172.21.15.169:0/1916638893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971c9690 msgr2=0x5640971d3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.454+0000 7f584b5c5740 1 --2- 172.21.15.169:0/1916638893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971c9690 0x5640971d3780 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f58340126b0 tx=0x7f5834012b70 comp rx=0 tx=0).stop 2024-03-28T00:57:55.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.454+0000 7f584b5c5740 1 -- 172.21.15.169:0/1916638893 shutdown_connections 2024-03-28T00:57:55.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.454+0000 7f584b5c5740 1 --2- 172.21.15.169:0/1916638893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640971cf1f0 0x5640971dad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.454+0000 7f584b5c5740 1 --2- 172.21.15.169:0/1916638893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971ca080 0x5640971d3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.454+0000 7f584b5c5740 1 --2- 172.21.15.169:0/1916638893 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x5640971df7f0 0x5640971e1c30 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.454+0000 7f584b5c5740 1 --2- 172.21.15.169:0/1916638893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971c9690 0x5640971d3780 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.454+0000 7f5829ffb640 0 client.15843 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T00:57:55.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.454+0000 7f584b5c5740 1 -- 172.21.15.169:0/1916638893 >> 172.21.15.169:0/1916638893 conn(0x5640971c4c00 msgr2=0x5640971c3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:55.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.454+0000 7f584b5c5740 1 -- 172.21.15.169:0/1916638893 shutdown_connections 2024-03-28T00:57:55.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.454+0000 7f584b5c5740 1 -- 172.21.15.169:0/1916638893 wait complete. 2024-03-28T00:57:55.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.455+0000 7f584b5c5740 20 client.15843 trim_cache size 0 max 16384 2024-03-28T00:57:55.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.467+0000 7f584b5c5740 1 Processor -- start 2024-03-28T00:57:55.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.467+0000 7f584b5c5740 1 -- start start 2024-03-28T00:57:55.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.467+0000 7f584b5c5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971c9690 0x5640971c8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.467+0000 7f584b5c5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640971ca080 0x5640971dac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.467+0000 7f584b5c5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971cf1f0 0x5640971e64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.467+0000 7f584b5c5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58441aa550 con 0x5640971cf1f0 2024-03-28T00:57:55.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.467+0000 7f584b5c5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58441a8050 con 0x5640971ca080 2024-03-28T00:57:55.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.467+0000 7f584b5c5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58441a60f0 con 0x5640971c9690 2024-03-28T00:57:55.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.467+0000 7f584259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971c9690 0x5640971c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.467+0000 7f5841d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640971ca080 0x5640971dac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.467+0000 7f584259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971c9690 0x5640971c8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:33660/0 (socket says 172.21.15.169:33660) 2024-03-28T00:57:55.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.467+0000 7f5841d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640971ca080 0x5640971dac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:53080/0 (socket says 172.21.15.169:53080) 2024-03-28T00:57:55.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.467+0000 7f584259c640 1 -- 172.21.15.169:0/3695268737 learned_addr learned my addr 172.21.15.169:0/3695268737 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:55.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.467+0000 7f5842d9d640 1 --2- 172.21.15.169:0/3695268737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971cf1f0 0x5640971e64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.467+0000 7f584259c640 1 -- 172.21.15.169:0/3695268737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640971ca080 msgr2=0x5640971dac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.467+0000 7f584259c640 1 --2- 172.21.15.169:0/3695268737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640971ca080 0x5640971dac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.468+0000 7f584259c640 1 -- 172.21.15.169:0/3695268737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971cf1f0 msgr2=0x5640971e64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.468+0000 7f584259c640 1 --2- 172.21.15.169:0/3695268737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971cf1f0 0x5640971e64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.468+0000 7f584259c640 1 -- 172.21.15.169:0/3695268737 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5640971c0670 con 0x5640971c9690 2024-03-28T00:57:55.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.468+0000 7f5842d9d640 1 --2- 172.21.15.169:0/3695268737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971cf1f0 0x5640971e64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:55.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.468+0000 7f584259c640 1 --2- 172.21.15.169:0/3695268737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971c9690 0x5640971c8570 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5834012680 tx=0x7f58340092d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbcf969004100376 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:55.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.468+0000 7f582bfff640 1 -- 172.21.15.169:0/3695268737 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583400fc40 con 0x5640971c9690 2024-03-28T00:57:55.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.468+0000 7f582bfff640 1 -- 172.21.15.169:0/3695268737 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f583402e070 con 0x5640971c9690 2024-03-28T00:57:55.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.468+0000 7f582bfff640 1 -- 172.21.15.169:0/3695268737 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58340042d0 con 0x5640971c9690 2024-03-28T00:57:55.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.469+0000 7f584b5c5740 1 -- 172.21.15.169:0/3695268737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971c9690 msgr2=0x5640971c8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.469+0000 7f584b5c5740 1 --2- 172.21.15.169:0/3695268737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971c9690 0x5640971c8570 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5834012680 tx=0x7f58340092d0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.469+0000 7f584b5c5740 1 -- 172.21.15.169:0/3695268737 shutdown_connections 2024-03-28T00:57:55.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.469+0000 7f584b5c5740 1 --2- 172.21.15.169:0/3695268737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971cf1f0 0x5640971e64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.469+0000 7f584b5c5740 1 --2- 172.21.15.169:0/3695268737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640971ca080 0x5640971dac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.469+0000 7f584b5c5740 1 --2- 172.21.15.169:0/3695268737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971c9690 0x5640971c8570 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.469+0000 7f584b5c5740 1 -- 172.21.15.169:0/3695268737 >> 172.21.15.169:0/3695268737 conn(0x5640971c4c00 msgr2=0x5640971c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:55.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.469+0000 7f584b5c5740 1 -- 172.21.15.169:0/3695268737 shutdown_connections 2024-03-28T00:57:55.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.469+0000 7f584b5c5740 1 -- 172.21.15.169:0/3695268737 wait complete. 2024-03-28T00:57:55.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.469+0000 7f584b5c5740 1 -- start start 2024-03-28T00:57:55.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.469+0000 7f584b5c5740 1 Processor -- start 2024-03-28T00:57:55.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f584b5c5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971c9690 0x5640971d85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f584b5c5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971ca080 0x5640971d8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f584b5c5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640971cf1f0 0x5640971d9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f584b5c5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58441a66f0 con 0x5640971ca080 2024-03-28T00:57:55.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f584b5c5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58441a79f0 con 0x5640971cf1f0 2024-03-28T00:57:55.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f584b5c5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58441a6d30 con 0x5640971c9690 2024-03-28T00:57:55.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f584259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971c9690 0x5640971d85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f584b5c5740 10 client.0 _ll_register_callbacks cb 0x7f5834009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:55.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f584259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971c9690 0x5640971d85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:33662/0 (socket says 172.21.15.169:33662) 2024-03-28T00:57:55.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f584259c640 1 -- 172.21.15.169:0/3536059664 learned_addr learned my addr 172.21.15.169:0/3536059664 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:55.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f5841d9b640 1 --2- 172.21.15.169:0/3536059664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971ca080 0x5640971d8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f584259c640 1 -- 172.21.15.169:0/3536059664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640971cf1f0 msgr2=0x5640971d9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f584259c640 1 --2- 172.21.15.169:0/3536059664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640971cf1f0 0x5640971d9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f584259c640 1 -- 172.21.15.169:0/3536059664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971ca080 msgr2=0x5640971d8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f584259c640 1 --2- 172.21.15.169:0/3536059664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971ca080 0x5640971d8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f584259c640 1 -- 172.21.15.169:0/3536059664 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5640971da0f0 con 0x5640971c9690 2024-03-28T00:57:55.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f5841d9b640 1 --2- 172.21.15.169:0/3536059664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971ca080 0x5640971d8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:57:55.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f584259c640 1 --2- 172.21.15.169:0/3536059664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971c9690 0x5640971d85b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5834009f80 tx=0x7f5834004400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20ff234d98d1820c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:55.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f5829ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:57:55.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f5829ffb640 1 -- 172.21.15.169:0/3536059664 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583400f670 con 0x5640971c9690 2024-03-28T00:57:55.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f5829ffb640 1 -- 172.21.15.169:0/3536059664 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f5834018040 con 0x5640971c9690 2024-03-28T00:57:55.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.470+0000 7f584b5c5740 1 -- 172.21.15.169:0/3536059664 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f583402e070 con 0x5640971c9690 2024-03-28T00:57:55.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.471+0000 7f5829ffb640 1 -- 172.21.15.169:0/3536059664 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5834011b50 con 0x5640971c9690 2024-03-28T00:57:55.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.471+0000 7f584b5c5740 10 client.25645 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:55.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.471+0000 7f584b5c5740 1 -- 172.21.15.169:0/3536059664 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56409723b340 con 0x5640971c9690 2024-03-28T00:57:55.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.471+0000 7f5829ffb640 1 -- 172.21.15.169:0/3536059664 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f583402e070 con 0x5640971c9690 2024-03-28T00:57:55.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.471+0000 7f5829ffb640 1 -- 172.21.15.169:0/3536059664 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f5834004810 con 0x5640971c9690 2024-03-28T00:57:55.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.471+0000 7f584b5c5740 10 client.25645 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:55.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.471+0000 7f584b5c5740 10 client.25645 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T00:57:55.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.471+0000 7f584b5c5740 20 client.25645 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:55.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.471+0000 7f584b5c5740 1 --2- 172.21.15.169:0/3536059664 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x5640971df7f0 0x56409723dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.471+0000 7f584b5c5740 4 client.25645 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T00:57:55.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.471+0000 7f584b5c5740 1 -- 172.21.15.169:0/3536059664 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564097136c50 con 0x5640971df7f0 2024-03-28T00:57:55.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.471+0000 7f5841d9b640 1 --2- 172.21.15.169:0/3536059664 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x5640971df7f0 0x56409723dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.472+0000 7f5829ffb640 10 client.25645 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T00:57:55.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.472+0000 7f5841d9b640 1 --2- 172.21.15.169:0/3536059664 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x5640971df7f0 0x56409723dc70 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f7a4e457768697f3 server_cookie=c5fcbd9890173328 in_seq=0 out_seq=0 2024-03-28T00:57:55.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.472+0000 7f5829ffb640 1 -- 172.21.15.169:0/3536059664 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564097136c50 con 0x5640971df7f0 2024-03-28T00:57:55.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.472+0000 7f5829ffb640 10 client.25645 handle_command_reply: tid=0 2024-03-28T00:57:55.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.472+0000 7f584b5c5740 1 client.25645 shutdown 2024-03-28T00:57:55.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.472+0000 7f584b5c5740 1 -- 172.21.15.169:0/3536059664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971c9690 msgr2=0x5640971d85b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.472+0000 7f584b5c5740 1 --2- 172.21.15.169:0/3536059664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971c9690 0x5640971d85b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5834009f80 tx=0x7f5834004400 comp rx=0 tx=0).stop 2024-03-28T00:57:55.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.472+0000 7f584b5c5740 1 -- 172.21.15.169:0/3536059664 shutdown_connections 2024-03-28T00:57:55.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.472+0000 7f584b5c5740 1 --2- 172.21.15.169:0/3536059664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640971cf1f0 0x5640971d9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.472+0000 7f584b5c5740 1 --2- 172.21.15.169:0/3536059664 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x5640971df7f0 0x56409723dc70 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.472+0000 7f584b5c5740 1 --2- 172.21.15.169:0/3536059664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5640971ca080 0x5640971d8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.472+0000 7f5829ffb640 0 client.25645 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T00:57:55.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.472+0000 7f584b5c5740 1 --2- 172.21.15.169:0/3536059664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640971c9690 0x5640971d85b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.472+0000 7f584b5c5740 1 -- 172.21.15.169:0/3536059664 >> 172.21.15.169:0/3536059664 conn(0x5640971c4c00 msgr2=0x5640971c3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:55.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.472+0000 7f584b5c5740 1 -- 172.21.15.169:0/3536059664 shutdown_connections 2024-03-28T00:57:55.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.472+0000 7f584b5c5740 1 -- 172.21.15.169:0/3536059664 wait complete. 2024-03-28T00:57:55.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.472+0000 7f584b5c5740 20 client.25645 trim_cache size 0 max 16384 2024-03-28T00:57:55.480 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T00:57:55.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.473+0000 7f58489e1640 1 -- 172.21.15.169:0/2848815808 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f582c0a2990 msgr2=0x7f582c0a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.473+0000 7f58489e1640 1 --2- 172.21.15.169:0/2848815808 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f582c0a2990 0x7f582c0a4e50 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5844198ee0 tx=0x7f5838005f90 comp rx=0 tx=0).stop 2024-03-28T00:57:55.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.473+0000 7f58489e1640 1 -- 172.21.15.169:0/2848815808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5844107400 msgr2=0x7f584419ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.473+0000 7f58489e1640 1 --2- 172.21.15.169:0/2848815808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5844107400 0x7f584419ef80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f583c02c430 tx=0x7f583c02cb90 comp rx=0 tx=0).stop 2024-03-28T00:57:55.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.473+0000 7f58489e1640 1 -- 172.21.15.169:0/2848815808 shutdown_connections 2024-03-28T00:57:55.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.473+0000 7f58489e1640 1 --2- 172.21.15.169:0/2848815808 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f582c0a2990 0x7f582c0a4e50 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.473+0000 7f58489e1640 1 --2- 172.21.15.169:0/2848815808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5844107400 0x7f584419ef80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.473+0000 7f58489e1640 1 --2- 172.21.15.169:0/2848815808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5844104ad0 0x7f5844197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.473+0000 7f58489e1640 1 --2- 172.21.15.169:0/2848815808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58441021a0 0x7f58441979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.473+0000 7f58489e1640 1 -- 172.21.15.169:0/2848815808 >> 172.21.15.169:0/2848815808 conn(0x7f58440fbb30 msgr2=0x7f5844102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:55.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.473+0000 7f58489e1640 1 -- 172.21.15.169:0/2848815808 shutdown_connections 2024-03-28T00:57:55.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.473+0000 7f58489e1640 1 -- 172.21.15.169:0/2848815808 wait complete. 2024-03-28T00:57:55.486 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:55 smithi169 ceph-mon[32274]: pgmap v393: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T00:57:55.487 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T00:57:55.487 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T00:57:55.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.591+0000 7f2042f74640 1 -- 172.21.15.169:0/1883341341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f203c107380 msgr2=0x7f203c1097d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.591+0000 7f2042f74640 1 --2- 172.21.15.169:0/1883341341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f203c107380 0x7f203c1097d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2034009ca0 tx=0x7f203402c0f0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.591+0000 7f2042f74640 1 -- 172.21.15.169:0/1883341341 shutdown_connections 2024-03-28T00:57:55.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.591+0000 7f2042f74640 1 --2- 172.21.15.169:0/1883341341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f203c107380 0x7f203c1097d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.591+0000 7f2042f74640 1 --2- 172.21.15.169:0/1883341341 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f203c104a50 0x7f203c106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.591+0000 7f2042f74640 1 --2- 172.21.15.169:0/1883341341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f203c102120 0x7f203c104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.591+0000 7f2042f74640 1 -- 172.21.15.169:0/1883341341 >> 172.21.15.169:0/1883341341 conn(0x7f203c0fbb30 msgr2=0x7f203c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:55.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.591+0000 7f2042f74640 1 -- 172.21.15.169:0/1883341341 shutdown_connections 2024-03-28T00:57:55.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.591+0000 7f2042f74640 1 -- 172.21.15.169:0/1883341341 wait complete. 2024-03-28T00:57:55.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.592+0000 7f2042f74640 1 Processor -- start 2024-03-28T00:57:55.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.592+0000 7f2042f74640 1 -- start start 2024-03-28T00:57:55.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.592+0000 7f2042f74640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f203c102120 0x7f203c1979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.592+0000 7f2042f74640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f203c104a50 0x7f203c197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.592+0000 7f2042f74640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f203c107380 0x7f203c19ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.592+0000 7f2042f74640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c10d320 con 0x7f203c102120 2024-03-28T00:57:55.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.592+0000 7f2042f74640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c10d1a0 con 0x7f203c107380 2024-03-28T00:57:55.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.592+0000 7f2042f74640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c10d4a0 con 0x7f203c104a50 2024-03-28T00:57:55.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.592+0000 7f203bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f203c104a50 0x7f203c197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.592+0000 7f203bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f203c104a50 0x7f203c197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:33684/0 (socket says 172.21.15.169:33684) 2024-03-28T00:57:55.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.592+0000 7f203bfff640 1 -- 172.21.15.169:0/506589265 learned_addr learned my addr 172.21.15.169:0/506589265 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:55.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.592+0000 7f2040d10640 1 --2- 172.21.15.169:0/506589265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f203c102120 0x7f203c1979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.592+0000 7f2041511640 1 --2- 172.21.15.169:0/506589265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f203c107380 0x7f203c19ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.592+0000 7f203bfff640 1 -- 172.21.15.169:0/506589265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f203c107380 msgr2=0x7f203c19ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.592+0000 7f203bfff640 1 --2- 172.21.15.169:0/506589265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f203c107380 0x7f203c19ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.592+0000 7f203bfff640 1 -- 172.21.15.169:0/506589265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f203c102120 msgr2=0x7f203c1979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.592+0000 7f203bfff640 1 --2- 172.21.15.169:0/506589265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f203c102120 0x7f203c1979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.592+0000 7f203bfff640 1 -- 172.21.15.169:0/506589265 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203c19f590 con 0x7f203c104a50 2024-03-28T00:57:55.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.592+0000 7f2041511640 1 --2- 172.21.15.169:0/506589265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f203c107380 0x7f203c19ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:57:55.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.592+0000 7f2040d10640 1 --2- 172.21.15.169:0/506589265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f203c102120 0x7f203c1979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:57:55.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.592+0000 7f203bfff640 1 --2- 172.21.15.169:0/506589265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f203c104a50 0x7f203c197f00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f202800e9c0 tx=0x7f202800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:55.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.593+0000 7f2039ffb640 1 -- 172.21.15.169:0/506589265 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202800cd80 con 0x7f203c104a50 2024-03-28T00:57:55.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.593+0000 7f2039ffb640 1 -- 172.21.15.169:0/506589265 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2028005500 con 0x7f203c104a50 2024-03-28T00:57:55.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.593+0000 7f2039ffb640 1 -- 172.21.15.169:0/506589265 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2028010690 con 0x7f203c104a50 2024-03-28T00:57:55.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.593+0000 7f2042f74640 1 -- 172.21.15.169:0/506589265 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f203c19f880 con 0x7f203c104a50 2024-03-28T00:57:55.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.593+0000 7f2042f74640 1 -- 172.21.15.169:0/506589265 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f203c19fd80 con 0x7f203c104a50 2024-03-28T00:57:55.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.595+0000 7f2039ffb640 1 -- 172.21.15.169:0/506589265 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f20280046c0 con 0x7f203c104a50 2024-03-28T00:57:55.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.596+0000 7f2045c37740 1 Processor -- start 2024-03-28T00:57:55.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.596+0000 7f2045c37740 1 -- start start 2024-03-28T00:57:55.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.596+0000 7f2039ffb640 1 --2- 172.21.15.169:0/506589265 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f202c0a2990 0x7f202c0a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.596+0000 7f2045c37740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c73690 0x564595c73ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.596+0000 7f2045c37740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c74080 0x564595c78b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.596+0000 7f2045c37740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c791f0 0x564595c7b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.596+0000 7f2045c37740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x564595c36610 con 0x564595c74080 2024-03-28T00:57:55.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.596+0000 7f2045c37740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x564595bd93c0 con 0x564595c791f0 2024-03-28T00:57:55.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.596+0000 7f2045c37740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x564595b81d60 con 0x564595c73690 2024-03-28T00:57:55.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.596+0000 7f2041511640 1 --2- 172.21.15.169:0/506589265 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f202c0a2990 0x7f202c0a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.596+0000 7f2039ffb640 1 -- 172.21.15.169:0/506589265 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f20280cf440 con 0x7f203c104a50 2024-03-28T00:57:55.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.597+0000 7f2040d10640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c73690 0x564595c73ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.597+0000 7f2041511640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c791f0 0x564595c7b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.597+0000 7f2040d10640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c73690 0x564595c73ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:33690/0 (socket says 172.21.15.169:33690) 2024-03-28T00:57:55.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.597+0000 7f2040d10640 1 -- 172.21.15.169:0/1103159748 learned_addr learned my addr 172.21.15.169:0/1103159748 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:55.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.597+0000 7f2040d10640 1 -- 172.21.15.169:0/1103159748 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c791f0 msgr2=0x564595c7b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.597+0000 7f2040d10640 1 --2- 172.21.15.169:0/1103159748 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c791f0 0x564595c7b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.597+0000 7f2040d10640 1 -- 172.21.15.169:0/1103159748 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c74080 msgr2=0x564595c78b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.597+0000 7f2041511640 1 --2- 172.21.15.169:0/506589265 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f202c0a2990 0x7f202c0a4e50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f203402c630 tx=0x7f2034009910 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:55.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.597+0000 7f203bfff640 1 --2- 172.21.15.169:0/1103159748 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c74080 0x564595c78b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.597+0000 7f2040d10640 1 --2- 172.21.15.169:0/1103159748 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c74080 0x564595c78b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.597+0000 7f2040d10640 1 -- 172.21.15.169:0/1103159748 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5645958ed3f0 con 0x564595c73690 2024-03-28T00:57:55.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.597+0000 7f203bfff640 1 --2- 172.21.15.169:0/1103159748 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c74080 0x564595c78b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:57:55.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.597+0000 7f2040d10640 1 --2- 172.21.15.169:0/1103159748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c73690 0x564595c73ab0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2030009cd0 tx=0x7f20300117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9109d4ba74b54dba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:55.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.597+0000 7f2026ffd640 1 -- 172.21.15.169:0/1103159748 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2030012450 con 0x564595c73690 2024-03-28T00:57:55.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.598+0000 7f2026ffd640 1 -- 172.21.15.169:0/1103159748 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2030012ad0 con 0x564595c73690 2024-03-28T00:57:55.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.598+0000 7f2026ffd640 1 -- 172.21.15.169:0/1103159748 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203000fc40 con 0x564595c73690 2024-03-28T00:57:55.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.598+0000 7f2045c37740 1 -- 172.21.15.169:0/1103159748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c73690 msgr2=0x564595c73ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.598+0000 7f2045c37740 1 --2- 172.21.15.169:0/1103159748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c73690 0x564595c73ab0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2030009cd0 tx=0x7f20300117b0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.598+0000 7f2045c37740 1 -- 172.21.15.169:0/1103159748 shutdown_connections 2024-03-28T00:57:55.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.598+0000 7f2045c37740 1 --2- 172.21.15.169:0/1103159748 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c791f0 0x564595c7b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.598+0000 7f2045c37740 1 --2- 172.21.15.169:0/1103159748 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c74080 0x564595c78b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.598+0000 7f2045c37740 1 --2- 172.21.15.169:0/1103159748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c73690 0x564595c73ab0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.598+0000 7f2045c37740 1 -- 172.21.15.169:0/1103159748 >> 172.21.15.169:0/1103159748 conn(0x564595c6ec00 msgr2=0x564595c6f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:55.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.598+0000 7f2045c37740 1 -- 172.21.15.169:0/1103159748 shutdown_connections 2024-03-28T00:57:55.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.599+0000 7f2045c37740 1 -- 172.21.15.169:0/1103159748 wait complete. 2024-03-28T00:57:55.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.599+0000 7f2045c37740 1 -- start start 2024-03-28T00:57:55.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.599+0000 7f2045c37740 1 Processor -- start 2024-03-28T00:57:55.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.599+0000 7f2045c37740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c73690 0x564595c81e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.599+0000 7f2045c37740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c74080 0x564595c823b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.599+0000 7f2045c37740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c791f0 0x564595c893f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.599+0000 7f2045c37740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x564595b95240 con 0x564595c791f0 2024-03-28T00:57:55.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.599+0000 7f2045c37740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x564595bd5460 con 0x564595c73690 2024-03-28T00:57:55.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.599+0000 7f2045c37740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x564595c80600 con 0x564595c74080 2024-03-28T00:57:55.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.599+0000 7f2045c37740 10 client.0 _ll_register_callbacks cb 0x564595b94c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:55.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.599+0000 7f2041511640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c791f0 0x564595c893f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.599+0000 7f2040d10640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c73690 0x564595c81e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.599+0000 7f2041511640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c791f0 0x564595c893f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33212/0 (socket says 172.21.15.169:33212) 2024-03-28T00:57:55.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.599+0000 7f203bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c74080 0x564595c823b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.599+0000 7f2041511640 1 -- 172.21.15.169:0/2810886490 learned_addr learned my addr 172.21.15.169:0/2810886490 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:55.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.600+0000 7f2040d10640 1 -- 172.21.15.169:0/2810886490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c74080 msgr2=0x564595c823b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.600+0000 7f2040d10640 1 --2- 172.21.15.169:0/2810886490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c74080 0x564595c823b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.600+0000 7f2040d10640 1 -- 172.21.15.169:0/2810886490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c791f0 msgr2=0x564595c893f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.600+0000 7f2040d10640 1 --2- 172.21.15.169:0/2810886490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c791f0 0x564595c893f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.600+0000 7f2040d10640 1 -- 172.21.15.169:0/2810886490 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5645959b77d0 con 0x564595c73690 2024-03-28T00:57:55.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.600+0000 7f203bfff640 1 --2- 172.21.15.169:0/2810886490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c74080 0x564595c823b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:55.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.600+0000 7f2041511640 1 --2- 172.21.15.169:0/2810886490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c791f0 0x564595c893f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:55.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.600+0000 7f2040d10640 1 --2- 172.21.15.169:0/2810886490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c73690 0x564595c81e70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2030009ca0 tx=0x7f2030004540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d30deb50a1739fbf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:55.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.600+0000 7f201bfff640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:57:55.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.600+0000 7f201bfff640 1 -- 172.21.15.169:0/2810886490 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2030027020 con 0x564595c73690 2024-03-28T00:57:55.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.600+0000 7f201bfff640 1 -- 172.21.15.169:0/2810886490 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2030012450 con 0x564595c73690 2024-03-28T00:57:55.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.600+0000 7f2045c37740 1 -- 172.21.15.169:0/2810886490 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2030012ad0 con 0x564595c73690 2024-03-28T00:57:55.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.600+0000 7f201bfff640 1 -- 172.21.15.169:0/2810886490 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2030032050 con 0x564595c73690 2024-03-28T00:57:55.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.601+0000 7f2045c37740 10 client.25679 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:55.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.601+0000 7f2045c37740 1 -- 172.21.15.169:0/2810886490 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x564595c7bda0 con 0x564595c73690 2024-03-28T00:57:55.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.601+0000 7f201bfff640 1 -- 172.21.15.169:0/2810886490 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2030012ad0 con 0x564595c73690 2024-03-28T00:57:55.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.601+0000 7f201bfff640 1 -- 172.21.15.169:0/2810886490 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2030005150 con 0x564595c73690 2024-03-28T00:57:55.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.601+0000 7f2045c37740 10 client.25679 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:55.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.601+0000 7f2045c37740 10 client.25679 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T00:57:55.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.601+0000 7f2045c37740 20 client.25679 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:55.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.601+0000 7f2045c37740 1 --2- 172.21.15.169:0/2810886490 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x564595c7d200 0x564595c7f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.601+0000 7f2045c37740 4 client.25679 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:57:55.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.601+0000 7f2045c37740 1 -- 172.21.15.169:0/2810886490 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564595be6900 con 0x564595c7d200 2024-03-28T00:57:55.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.601+0000 7f203bfff640 1 --2- 172.21.15.169:0/2810886490 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x564595c7d200 0x564595c7f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.602+0000 7f201bfff640 10 client.25679 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T00:57:55.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.602+0000 7f203bfff640 1 --2- 172.21.15.169:0/2810886490 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x564595c7d200 0x564595c7f610 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1e2ee97abb0c1f server_cookie=58b949507617969 in_seq=0 out_seq=0 2024-03-28T00:57:55.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.602+0000 7f201bfff640 1 -- 172.21.15.169:0/2810886490 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564595be6900 con 0x564595c7d200 2024-03-28T00:57:55.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.602+0000 7f201bfff640 10 client.25679 handle_command_reply: tid=0 2024-03-28T00:57:55.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.602+0000 7f2045c37740 1 client.25679 shutdown 2024-03-28T00:57:55.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.603+0000 7f2045c37740 1 -- 172.21.15.169:0/2810886490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c73690 msgr2=0x564595c81e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.603+0000 7f2045c37740 1 --2- 172.21.15.169:0/2810886490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c73690 0x564595c81e70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2030009ca0 tx=0x7f2030004540 comp rx=0 tx=0).stop 2024-03-28T00:57:55.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.603+0000 7f2045c37740 1 -- 172.21.15.169:0/2810886490 shutdown_connections 2024-03-28T00:57:55.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.603+0000 7f2045c37740 1 --2- 172.21.15.169:0/2810886490 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x564595c7d200 0x564595c7f610 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.603+0000 7f201bfff640 0 client.25679 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T00:57:55.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.603+0000 7f2045c37740 1 --2- 172.21.15.169:0/2810886490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c791f0 0x564595c893f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.603+0000 7f2045c37740 1 --2- 172.21.15.169:0/2810886490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c74080 0x564595c823b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.603+0000 7f2045c37740 1 --2- 172.21.15.169:0/2810886490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c73690 0x564595c81e70 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.603+0000 7f2045c37740 1 -- 172.21.15.169:0/2810886490 >> 172.21.15.169:0/2810886490 conn(0x564595c6ec00 msgr2=0x564595c6de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:55.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.603+0000 7f2045c37740 1 -- 172.21.15.169:0/2810886490 shutdown_connections 2024-03-28T00:57:55.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.603+0000 7f2045c37740 1 -- 172.21.15.169:0/2810886490 wait complete. 2024-03-28T00:57:55.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.603+0000 7f2045c37740 20 client.25679 trim_cache size 0 max 16384 2024-03-28T00:57:55.611 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:55 smithi174 ceph-mon[36060]: pgmap v393: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T00:57:55.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.615+0000 7f2045c37740 1 Processor -- start 2024-03-28T00:57:55.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.615+0000 7f2045c37740 1 -- start start 2024-03-28T00:57:55.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.615+0000 7f2045c37740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c73690 0x564595c6d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.615+0000 7f2045c37740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c74080 0x564595c893b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.615+0000 7f2045c37740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c791f0 0x564595c6dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.615+0000 7f2045c37740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c1aa4e0 con 0x564595c73690 2024-03-28T00:57:55.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.615+0000 7f2045c37740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c1a7fe0 con 0x564595c791f0 2024-03-28T00:57:55.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.615+0000 7f2045c37740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c1a6080 con 0x564595c74080 2024-03-28T00:57:55.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.615+0000 7f2040d10640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c73690 0x564595c6d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.615+0000 7f2040d10640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c73690 0x564595c6d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33218/0 (socket says 172.21.15.169:33218) 2024-03-28T00:57:55.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.615+0000 7f2040d10640 1 -- 172.21.15.169:0/2950777454 learned_addr learned my addr 172.21.15.169:0/2950777454 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:55.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.615+0000 7f203bfff640 1 --2- 172.21.15.169:0/2950777454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c74080 0x564595c893b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.616+0000 7f2041511640 1 --2- 172.21.15.169:0/2950777454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c791f0 0x564595c6dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.616+0000 7f203bfff640 1 -- 172.21.15.169:0/2950777454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c791f0 msgr2=0x564595c6dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.616+0000 7f203bfff640 1 --2- 172.21.15.169:0/2950777454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c791f0 0x564595c6dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.616+0000 7f203bfff640 1 -- 172.21.15.169:0/2950777454 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c73690 msgr2=0x564595c6d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.616+0000 7f203bfff640 1 --2- 172.21.15.169:0/2950777454 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c73690 0x564595c6d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.616+0000 7f203bfff640 1 -- 172.21.15.169:0/2950777454 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564595c74460 con 0x564595c74080 2024-03-28T00:57:55.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.616+0000 7f2041511640 1 --2- 172.21.15.169:0/2950777454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c791f0 0x564595c6dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:57:55.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.616+0000 7f2040d10640 1 --2- 172.21.15.169:0/2950777454 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c73690 0x564595c6d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:55.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.616+0000 7f203bfff640 1 --2- 172.21.15.169:0/2950777454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c74080 0x564595c893b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f20280cc9f0 tx=0x7f2028004b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af071947b9ffa5e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:55.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.616+0000 7f20267fc640 1 -- 172.21.15.169:0/2950777454 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202809b880 con 0x564595c74080 2024-03-28T00:57:55.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.616+0000 7f20267fc640 1 -- 172.21.15.169:0/2950777454 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f202809f9d0 con 0x564595c74080 2024-03-28T00:57:55.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.616+0000 7f20267fc640 1 -- 172.21.15.169:0/2950777454 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20280ccdd0 con 0x564595c74080 2024-03-28T00:57:55.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.617+0000 7f2045c37740 1 -- 172.21.15.169:0/2950777454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c74080 msgr2=0x564595c893b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.617+0000 7f2045c37740 1 --2- 172.21.15.169:0/2950777454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c74080 0x564595c893b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f20280cc9f0 tx=0x7f2028004b30 comp rx=0 tx=0).stop 2024-03-28T00:57:55.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.617+0000 7f2045c37740 1 -- 172.21.15.169:0/2950777454 shutdown_connections 2024-03-28T00:57:55.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.617+0000 7f2045c37740 1 --2- 172.21.15.169:0/2950777454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c791f0 0x564595c6dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.617+0000 7f2045c37740 1 --2- 172.21.15.169:0/2950777454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c74080 0x564595c893b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.617+0000 7f2045c37740 1 --2- 172.21.15.169:0/2950777454 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c73690 0x564595c6d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.617+0000 7f2045c37740 1 -- 172.21.15.169:0/2950777454 >> 172.21.15.169:0/2950777454 conn(0x564595c6ec00 msgr2=0x564595c75610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:55.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.617+0000 7f2045c37740 1 -- 172.21.15.169:0/2950777454 shutdown_connections 2024-03-28T00:57:55.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.617+0000 7f2045c37740 1 -- 172.21.15.169:0/2950777454 wait complete. 2024-03-28T00:57:55.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.617+0000 7f2045c37740 1 -- start start 2024-03-28T00:57:55.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.617+0000 7f2045c37740 1 Processor -- start 2024-03-28T00:57:55.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f2045c37740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c73690 0x564595c863c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f2045c37740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c74080 0x564595c86900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f2045c37740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c791f0 0x564595cebac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f2045c37740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c1a6680 con 0x564595c73690 2024-03-28T00:57:55.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f2045c37740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c1a7980 con 0x564595c791f0 2024-03-28T00:57:55.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f2045c37740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c1a6cc0 con 0x564595c74080 2024-03-28T00:57:55.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f2040d10640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c73690 0x564595c863c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f2045c37740 10 client.0 _ll_register_callbacks cb 0x7f2030004400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:55.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f2040d10640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c73690 0x564595c863c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33222/0 (socket says 172.21.15.169:33222) 2024-03-28T00:57:55.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f203bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c74080 0x564595c86900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f2040d10640 1 -- 172.21.15.169:0/3267246069 learned_addr learned my addr 172.21.15.169:0/3267246069 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:55.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f2041511640 1 --2- 172.21.15.169:0/3267246069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c791f0 0x564595cebac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f203bfff640 1 -- 172.21.15.169:0/3267246069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c791f0 msgr2=0x564595cebac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f203bfff640 1 --2- 172.21.15.169:0/3267246069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c791f0 0x564595cebac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f203bfff640 1 -- 172.21.15.169:0/3267246069 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c73690 msgr2=0x564595c863c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f203bfff640 1 --2- 172.21.15.169:0/3267246069 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c73690 0x564595c863c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f203bfff640 1 -- 172.21.15.169:0/3267246069 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564595cec0b0 con 0x564595c74080 2024-03-28T00:57:55.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f2041511640 1 --2- 172.21.15.169:0/3267246069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c791f0 0x564595cebac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:57:55.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f2040d10640 1 --2- 172.21.15.169:0/3267246069 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c73690 0x564595c863c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:55.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f203bfff640 1 --2- 172.21.15.169:0/3267246069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c74080 0x564595c86900 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x564595c8a090 tx=0x7f2028005a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd4a35bef0f7b9b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:55.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f201bfff640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:57:55.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f201bfff640 1 -- 172.21.15.169:0/3267246069 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20280b3070 con 0x564595c74080 2024-03-28T00:57:55.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f2045c37740 1 -- 172.21.15.169:0/3267246069 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f202809f9d0 con 0x564595c74080 2024-03-28T00:57:55.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f201bfff640 1 -- 172.21.15.169:0/3267246069 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f202809bce0 con 0x564595c74080 2024-03-28T00:57:55.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f201bfff640 1 -- 172.21.15.169:0/3267246069 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f202809b880 con 0x564595c74080 2024-03-28T00:57:55.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f2045c37740 10 client.25663 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:55.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.618+0000 7f2045c37740 1 -- 172.21.15.169:0/3267246069 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x564595ced350 con 0x564595c74080 2024-03-28T00:57:55.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.619+0000 7f201bfff640 1 -- 172.21.15.169:0/3267246069 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f202809f9d0 con 0x564595c74080 2024-03-28T00:57:55.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.619+0000 7f201bfff640 1 -- 172.21.15.169:0/3267246069 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f202809ccd0 con 0x564595c74080 2024-03-28T00:57:55.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.619+0000 7f2045c37740 10 client.25663 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:55.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.619+0000 7f2045c37740 10 client.25663 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T00:57:55.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.619+0000 7f2045c37740 20 client.25663 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:55.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.619+0000 7f2045c37740 1 --2- 172.21.15.169:0/3267246069 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x564595cedbf0 0x564595cf0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.619+0000 7f2045c37740 4 client.25663 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T00:57:55.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.619+0000 7f2045c37740 1 -- 172.21.15.169:0/3267246069 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564595be0c50 con 0x564595cedbf0 2024-03-28T00:57:55.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.619+0000 7f2040d10640 1 --2- 172.21.15.169:0/3267246069 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x564595cedbf0 0x564595cf0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.619+0000 7f201bfff640 10 client.25663 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T00:57:55.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.619+0000 7f2040d10640 1 --2- 172.21.15.169:0/3267246069 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x564595cedbf0 0x564595cf0010 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f5601e521d508788 server_cookie=86348930ad65ceb9 in_seq=0 out_seq=0 2024-03-28T00:57:55.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.620+0000 7f201bfff640 1 -- 172.21.15.169:0/3267246069 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564595be0c50 con 0x564595cedbf0 2024-03-28T00:57:55.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.620+0000 7f201bfff640 10 client.25663 handle_command_reply: tid=0 2024-03-28T00:57:55.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.620+0000 7f2045c37740 1 client.25663 shutdown 2024-03-28T00:57:55.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.620+0000 7f2045c37740 1 -- 172.21.15.169:0/3267246069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c74080 msgr2=0x564595c86900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.620+0000 7f2045c37740 1 --2- 172.21.15.169:0/3267246069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c74080 0x564595c86900 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x564595c8a090 tx=0x7f2028005a90 comp rx=0 tx=0).stop 2024-03-28T00:57:55.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.620+0000 7f2045c37740 1 -- 172.21.15.169:0/3267246069 shutdown_connections 2024-03-28T00:57:55.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.620+0000 7f2045c37740 1 --2- 172.21.15.169:0/3267246069 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x564595cedbf0 0x564595cf0010 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.620+0000 7f201bfff640 0 client.25663 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T00:57:55.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.620+0000 7f2045c37740 1 --2- 172.21.15.169:0/3267246069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564595c791f0 0x564595cebac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.620+0000 7f2045c37740 1 --2- 172.21.15.169:0/3267246069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564595c74080 0x564595c86900 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.620+0000 7f2045c37740 1 --2- 172.21.15.169:0/3267246069 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564595c73690 0x564595c863c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.620+0000 7f2045c37740 1 -- 172.21.15.169:0/3267246069 >> 172.21.15.169:0/3267246069 conn(0x564595c6ec00 msgr2=0x564595c726f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:55.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.620+0000 7f2045c37740 1 -- 172.21.15.169:0/3267246069 shutdown_connections 2024-03-28T00:57:55.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.620+0000 7f2045c37740 1 -- 172.21.15.169:0/3267246069 wait complete. 2024-03-28T00:57:55.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.620+0000 7f2045c37740 20 client.25663 trim_cache size 0 max 16384 2024-03-28T00:57:55.629 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T00:57:55.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.621+0000 7f2042f74640 1 -- 172.21.15.169:0/506589265 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f202c0a2990 msgr2=0x7f202c0a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.621+0000 7f2042f74640 1 --2- 172.21.15.169:0/506589265 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f202c0a2990 0x7f202c0a4e50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f203402c630 tx=0x7f2034009910 comp rx=0 tx=0).stop 2024-03-28T00:57:55.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.621+0000 7f2042f74640 1 -- 172.21.15.169:0/506589265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f203c104a50 msgr2=0x7f203c197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.621+0000 7f2042f74640 1 --2- 172.21.15.169:0/506589265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f203c104a50 0x7f203c197f00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f202800e9c0 tx=0x7f202800ee80 comp rx=0 tx=0).stop 2024-03-28T00:57:55.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.621+0000 7f2042f74640 1 -- 172.21.15.169:0/506589265 shutdown_connections 2024-03-28T00:57:55.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.621+0000 7f2042f74640 1 --2- 172.21.15.169:0/506589265 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f202c0a2990 0x7f202c0a4e50 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.621+0000 7f2042f74640 1 --2- 172.21.15.169:0/506589265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f203c107380 0x7f203c19ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.621+0000 7f2042f74640 1 --2- 172.21.15.169:0/506589265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f203c104a50 0x7f203c197f00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.621+0000 7f2042f74640 1 --2- 172.21.15.169:0/506589265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f203c102120 0x7f203c1979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.621+0000 7f2042f74640 1 -- 172.21.15.169:0/506589265 >> 172.21.15.169:0/506589265 conn(0x7f203c0fbb30 msgr2=0x7f203c102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:55.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.621+0000 7f2042f74640 1 -- 172.21.15.169:0/506589265 shutdown_connections 2024-03-28T00:57:55.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.621+0000 7f2042f74640 1 -- 172.21.15.169:0/506589265 wait complete. 2024-03-28T00:57:55.635 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T00:57:55.635 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T00:57:55.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:55 smithi177 ceph-mon[36206]: pgmap v393: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T00:57:55.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.739+0000 7f8ebc64b640 1 -- 172.21.15.169:0/815626966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8eb4104a50 msgr2=0x7f8eb4106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.739+0000 7f8ebc64b640 1 --2- 172.21.15.169:0/815626966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8eb4104a50 0x7f8eb4106e40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4009900 tx=0x7f8ea402bc30 comp rx=0 tx=0).stop 2024-03-28T00:57:55.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.739+0000 7f8ebc64b640 1 -- 172.21.15.169:0/815626966 shutdown_connections 2024-03-28T00:57:55.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.739+0000 7f8ebc64b640 1 --2- 172.21.15.169:0/815626966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8eb4107380 0x7f8eb41097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.739+0000 7f8ebc64b640 1 --2- 172.21.15.169:0/815626966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8eb4104a50 0x7f8eb4106e40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.739+0000 7f8ebc64b640 1 --2- 172.21.15.169:0/815626966 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8eb4102120 0x7f8eb4104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.739+0000 7f8ebc64b640 1 -- 172.21.15.169:0/815626966 >> 172.21.15.169:0/815626966 conn(0x7f8eb40fbb30 msgr2=0x7f8eb40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:55.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.739+0000 7f8ebc64b640 1 -- 172.21.15.169:0/815626966 shutdown_connections 2024-03-28T00:57:55.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.739+0000 7f8ebc64b640 1 -- 172.21.15.169:0/815626966 wait complete. 2024-03-28T00:57:55.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.740+0000 7f8ebc64b640 1 Processor -- start 2024-03-28T00:57:55.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.740+0000 7f8ebc64b640 1 -- start start 2024-03-28T00:57:55.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.740+0000 7f8ebc64b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8eb4102120 0x7f8eb41957e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.740+0000 7f8ebc64b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8eb4104a50 0x7f8eb4195d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.740+0000 7f8ebc64b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8eb4107380 0x7f8eb419cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.740+0000 7f8ebc64b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb410b160 con 0x7f8eb4102120 2024-03-28T00:57:55.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.740+0000 7f8ebc64b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb410afe0 con 0x7f8eb4107380 2024-03-28T00:57:55.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.740+0000 7f8eb9be6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8eb4104a50 0x7f8eb4195d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.740+0000 7f8ebc64b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb410b2e0 con 0x7f8eb4104a50 2024-03-28T00:57:55.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.740+0000 7f8eb9be6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8eb4104a50 0x7f8eb4195d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:33726/0 (socket says 172.21.15.169:33726) 2024-03-28T00:57:55.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.740+0000 7f8eb9be6640 1 -- 172.21.15.169:0/2365040017 learned_addr learned my addr 172.21.15.169:0/2365040017 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:55.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.740+0000 7f8eba3e7640 1 --2- 172.21.15.169:0/2365040017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8eb4102120 0x7f8eb41957e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.740+0000 7f8ebabe8640 1 --2- 172.21.15.169:0/2365040017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8eb4107380 0x7f8eb419cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.740+0000 7f8eb9be6640 1 -- 172.21.15.169:0/2365040017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8eb4107380 msgr2=0x7f8eb419cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.740+0000 7f8eb9be6640 1 --2- 172.21.15.169:0/2365040017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8eb4107380 0x7f8eb419cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.740+0000 7f8eb9be6640 1 -- 172.21.15.169:0/2365040017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8eb4102120 msgr2=0x7f8eb41957e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.740+0000 7f8eb9be6640 1 --2- 172.21.15.169:0/2365040017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8eb4102120 0x7f8eb41957e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.740+0000 7f8eb9be6640 1 -- 172.21.15.169:0/2365040017 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb419d400 con 0x7f8eb4104a50 2024-03-28T00:57:55.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.741+0000 7f8eb9be6640 1 --2- 172.21.15.169:0/2365040017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8eb4104a50 0x7f8eb4195d20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4000c00 tx=0x7f8ea402c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:55.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.741+0000 7f8eab7fe640 1 -- 172.21.15.169:0/2365040017 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4041070 con 0x7f8eb4104a50 2024-03-28T00:57:55.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.741+0000 7f8ebc64b640 1 -- 172.21.15.169:0/2365040017 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb419d690 con 0x7f8eb4104a50 2024-03-28T00:57:55.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.741+0000 7f8eab7fe640 1 -- 172.21.15.169:0/2365040017 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ea4004450 con 0x7f8eb4104a50 2024-03-28T00:57:55.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.741+0000 7f8eab7fe640 1 -- 172.21.15.169:0/2365040017 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4034440 con 0x7f8eb4104a50 2024-03-28T00:57:55.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.741+0000 7f8ebc64b640 1 -- 172.21.15.169:0/2365040017 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb419dbd0 con 0x7f8eb4104a50 2024-03-28T00:57:55.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.744+0000 7f8eab7fe640 1 -- 172.21.15.169:0/2365040017 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f8ea4044070 con 0x7f8eb4104a50 2024-03-28T00:57:55.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.744+0000 7f8ebf237740 1 Processor -- start 2024-03-28T00:57:55.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.744+0000 7f8ebf237740 1 -- start start 2024-03-28T00:57:55.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.744+0000 7f8ebf237740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56340262e690 0x56340262eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.744+0000 7f8ebf237740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56340262f080 0x563402633b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.744+0000 7f8ebf237740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5634026341f0 0x563402636680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.745+0000 7f8ebf237740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634025f1610 con 0x56340262f080 2024-03-28T00:57:55.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.745+0000 7f8ebf237740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634025943c0 con 0x56340262e690 2024-03-28T00:57:55.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.745+0000 7f8ebf237740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56340253cd60 con 0x5634026341f0 2024-03-28T00:57:55.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.745+0000 7f8ebabe8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56340262e690 0x56340262eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.745+0000 7f8ebabe8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56340262e690 0x56340262eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:53158/0 (socket says 172.21.15.169:53158) 2024-03-28T00:57:55.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.745+0000 7f8ebabe8640 1 -- 172.21.15.169:0/2946540710 learned_addr learned my addr 172.21.15.169:0/2946540710 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:55.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.745+0000 7f8eba3e7640 1 --2- 172.21.15.169:0/2946540710 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56340262f080 0x563402633b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.745+0000 7f8eb9be6640 1 --2- 172.21.15.169:0/2946540710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5634026341f0 0x563402636680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.745+0000 7f8eab7fe640 1 --2- 172.21.15.169:0/2365040017 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8ea00a2910 0x7f8ea00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.745+0000 7f8ebabe8640 1 -- 172.21.15.169:0/2946540710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5634026341f0 msgr2=0x563402636680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.745+0000 7f8ebabe8640 1 --2- 172.21.15.169:0/2946540710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5634026341f0 0x563402636680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.745+0000 7f8ebabe8640 1 -- 172.21.15.169:0/2946540710 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56340262f080 msgr2=0x563402633b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.745+0000 7f8ebabe8640 1 --2- 172.21.15.169:0/2946540710 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56340262f080 0x563402633b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.745+0000 7f8ebabe8640 1 -- 172.21.15.169:0/2946540710 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634022a83f0 con 0x56340262e690 2024-03-28T00:57:55.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.745+0000 7f8eba3e7640 1 --2- 172.21.15.169:0/2946540710 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56340262f080 0x563402633b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:55.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.745+0000 7f8eab7fe640 1 -- 172.21.15.169:0/2365040017 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f8ea40f08c0 con 0x7f8eb4104a50 2024-03-28T00:57:55.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.745+0000 7f8ebabe8640 1 --2- 172.21.15.169:0/2365040017 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8ea00a2910 0x7f8ea00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.746+0000 7f8ebabe8640 1 --2- 172.21.15.169:0/2946540710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56340262e690 0x56340262eab0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8eac009990 tx=0x7f8eac00eaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c3c9aee0cceee30 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:55.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.746+0000 7f8ea8bfe640 1 -- 172.21.15.169:0/2946540710 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eac021040 con 0x56340262e690 2024-03-28T00:57:55.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.746+0000 7f8ea8bfe640 1 -- 172.21.15.169:0/2946540710 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8eac015bc0 con 0x56340262e690 2024-03-28T00:57:55.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.746+0000 7f8ea8bfe640 1 -- 172.21.15.169:0/2946540710 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eac01da40 con 0x56340262e690 2024-03-28T00:57:55.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.747+0000 7f8ebf237740 1 -- 172.21.15.169:0/2946540710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56340262e690 msgr2=0x56340262eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.747+0000 7f8ebf237740 1 --2- 172.21.15.169:0/2946540710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56340262e690 0x56340262eab0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8eac009990 tx=0x7f8eac00eaf0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.747+0000 7f8ebabe8640 1 --2- 172.21.15.169:0/2365040017 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8ea00a2910 0x7f8ea00a4dd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8eac027000 tx=0x7f8eac028000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:55.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.747+0000 7f8ebf237740 1 -- 172.21.15.169:0/2946540710 shutdown_connections 2024-03-28T00:57:55.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.747+0000 7f8ebf237740 1 --2- 172.21.15.169:0/2946540710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5634026341f0 0x563402636680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.747+0000 7f8ebf237740 1 --2- 172.21.15.169:0/2946540710 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56340262f080 0x563402633b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.747+0000 7f8ebf237740 1 --2- 172.21.15.169:0/2946540710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56340262e690 0x56340262eab0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.747+0000 7f8ebf237740 1 -- 172.21.15.169:0/2946540710 >> 172.21.15.169:0/2946540710 conn(0x563402629c00 msgr2=0x56340262a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:55.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.747+0000 7f8ebf237740 1 -- 172.21.15.169:0/2946540710 shutdown_connections 2024-03-28T00:57:55.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.747+0000 7f8ebf237740 1 -- 172.21.15.169:0/2946540710 wait complete. 2024-03-28T00:57:55.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.747+0000 7f8ebf237740 1 -- start start 2024-03-28T00:57:55.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.747+0000 7f8ebf237740 1 Processor -- start 2024-03-28T00:57:55.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.747+0000 7f8ebf237740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56340262e690 0x563402638a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.747+0000 7f8ebf237740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56340262f080 0x563402638f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.747+0000 7f8ebf237740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5634026341f0 0x56340263ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.747+0000 7f8ebf237740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x563402550240 con 0x56340262f080 2024-03-28T00:57:55.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.747+0000 7f8ebf237740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x563402590460 con 0x56340262e690 2024-03-28T00:57:55.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.747+0000 7f8ebf237740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634026371a0 con 0x5634026341f0 2024-03-28T00:57:55.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.747+0000 7f8ebf237740 10 client.0 _ll_register_callbacks cb 0x56340254fc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:55.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.747+0000 7f8eb9be6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56340262f080 0x563402638f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.748+0000 7f8eb9be6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56340262f080 0x563402638f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33254/0 (socket says 172.21.15.169:33254) 2024-03-28T00:57:55.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.748+0000 7f8eb9be6640 1 -- 172.21.15.169:0/539221783 learned_addr learned my addr 172.21.15.169:0/539221783 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:55.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.748+0000 7f8eba3e7640 1 --2- 172.21.15.169:0/539221783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56340262e690 0x563402638a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.748+0000 7f8eb9be6640 1 -- 172.21.15.169:0/539221783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5634026341f0 msgr2=0x56340263ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.748+0000 7f8eb9be6640 1 --2- 172.21.15.169:0/539221783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5634026341f0 0x56340263ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.748+0000 7f8eb9be6640 1 -- 172.21.15.169:0/539221783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56340262e690 msgr2=0x563402638a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.748+0000 7f8eb9be6640 1 --2- 172.21.15.169:0/539221783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56340262e690 0x563402638a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.748+0000 7f8eb9be6640 1 -- 172.21.15.169:0/539221783 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634023727d0 con 0x56340262f080 2024-03-28T00:57:55.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.748+0000 7f8eba3e7640 1 --2- 172.21.15.169:0/539221783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56340262e690 0x563402638a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:57:55.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.748+0000 7f8eb9be6640 1 --2- 172.21.15.169:0/539221783 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56340262f080 0x563402638f90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8ea40b9410 tx=0x7f8ea402c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8dbf0653bb5469 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:55.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.748+0000 7f8e9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:57:55.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.748+0000 7f8e9dffb640 1 -- 172.21.15.169:0/539221783 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea404b030 con 0x56340262f080 2024-03-28T00:57:55.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.748+0000 7f8e9dffb640 1 -- 172.21.15.169:0/539221783 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f8ea40c30b0 con 0x56340262f080 2024-03-28T00:57:55.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.748+0000 7f8ebf237740 1 -- 172.21.15.169:0/539221783 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8eac015bc0 con 0x56340262f080 2024-03-28T00:57:55.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.748+0000 7f8e9dffb640 1 -- 172.21.15.169:0/539221783 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ea40bcaf0 con 0x56340262f080 2024-03-28T00:57:55.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.748+0000 7f8ebf237740 10 client.15873 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:55.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.748+0000 7f8ebf237740 1 -- 172.21.15.169:0/539221783 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563402643340 con 0x56340262f080 2024-03-28T00:57:55.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.749+0000 7f8e9dffb640 1 -- 172.21.15.169:0/539221783 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ea4004c80 con 0x56340262f080 2024-03-28T00:57:55.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.749+0000 7f8e9dffb640 1 -- 172.21.15.169:0/539221783 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f8ea40c5180 con 0x56340262f080 2024-03-28T00:57:55.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.749+0000 7f8ebf237740 10 client.15873 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:55.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.749+0000 7f8ebf237740 10 client.15873 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:55.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.749+0000 7f8ebf237740 20 client.15873 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:55.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.749+0000 7f8ebf237740 1 --2- 172.21.15.169:0/539221783 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563402644a20 0x563402646e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.749+0000 7f8ebf237740 4 client.15873 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:57:55.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.749+0000 7f8ebf237740 1 -- 172.21.15.169:0/539221783 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8eac015bc0 con 0x563402644a20 2024-03-28T00:57:55.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.749+0000 7f8eba3e7640 1 --2- 172.21.15.169:0/539221783 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563402644a20 0x563402646e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.750+0000 7f8eba3e7640 1 --2- 172.21.15.169:0/539221783 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563402644a20 0x563402646e70 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca70cdcabded7c2d server_cookie=ec13e39ceb3ce6ad in_seq=0 out_seq=0 2024-03-28T00:57:55.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.750+0000 7f8e9dffb640 10 client.15873 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:55.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.750+0000 7f8e9dffb640 1 -- 172.21.15.169:0/539221783 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7f8eac015bc0 con 0x563402644a20 2024-03-28T00:57:55.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.750+0000 7f8e9dffb640 10 client.15873 handle_command_reply: tid=0 2024-03-28T00:57:55.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.750+0000 7f8ebf237740 1 client.15873 shutdown 2024-03-28T00:57:55.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.750+0000 7f8ebf237740 1 -- 172.21.15.169:0/539221783 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56340262f080 msgr2=0x563402638f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.750+0000 7f8ebf237740 1 --2- 172.21.15.169:0/539221783 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56340262f080 0x563402638f90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8ea40b9410 tx=0x7f8ea402c880 comp rx=0 tx=0).stop 2024-03-28T00:57:55.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.750+0000 7f8ebf237740 1 -- 172.21.15.169:0/539221783 shutdown_connections 2024-03-28T00:57:55.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.750+0000 7f8ebf237740 1 --2- 172.21.15.169:0/539221783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5634026341f0 0x56340263ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.750+0000 7f8ebf237740 1 --2- 172.21.15.169:0/539221783 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563402644a20 0x563402646e70 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.750+0000 7f8ebf237740 1 --2- 172.21.15.169:0/539221783 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56340262f080 0x563402638f90 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.750+0000 7f8ebf237740 1 --2- 172.21.15.169:0/539221783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56340262e690 0x563402638a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.750+0000 7f8ebf237740 1 -- 172.21.15.169:0/539221783 >> 172.21.15.169:0/539221783 conn(0x563402629c00 msgr2=0x563402628e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:55.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.750+0000 7f8e9dffb640 0 client.15873 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:55.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.751+0000 7f8ebf237740 1 -- 172.21.15.169:0/539221783 shutdown_connections 2024-03-28T00:57:55.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.751+0000 7f8ebf237740 1 -- 172.21.15.169:0/539221783 wait complete. 2024-03-28T00:57:55.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.751+0000 7f8ebf237740 20 client.15873 trim_cache size 0 max 16384 2024-03-28T00:57:55.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.763+0000 7f8ebf237740 1 Processor -- start 2024-03-28T00:57:55.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.763+0000 7f8ebf237740 1 -- start start 2024-03-28T00:57:55.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.763+0000 7f8ebf237740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56340262e690 0x5634026305e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.763+0000 7f8ebf237740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5634026341f0 0x56340263ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.763+0000 7f8ebf237740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563402630b20 0x563402628590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.763+0000 7f8ebf237740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb41aa550 con 0x56340262e690 2024-03-28T00:57:55.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.763+0000 7f8ebf237740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb41a8050 con 0x563402630b20 2024-03-28T00:57:55.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.763+0000 7f8ebf237740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb41a60f0 con 0x5634026341f0 2024-03-28T00:57:55.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.763+0000 7f8eba3e7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56340262e690 0x5634026305e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.763+0000 7f8eb9be6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5634026341f0 0x56340263ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.763+0000 7f8eb9be6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5634026341f0 0x56340263ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:33742/0 (socket says 172.21.15.169:33742) 2024-03-28T00:57:55.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.763+0000 7f8eba3e7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56340262e690 0x5634026305e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33270/0 (socket says 172.21.15.169:33270) 2024-03-28T00:57:55.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.763+0000 7f8eb9be6640 1 -- 172.21.15.169:0/3890122236 learned_addr learned my addr 172.21.15.169:0/3890122236 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:55.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.763+0000 7f8ebabe8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563402630b20 0x563402628590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.764+0000 7f8eb9be6640 1 -- 172.21.15.169:0/3890122236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563402630b20 msgr2=0x563402628590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.764+0000 7f8eb9be6640 1 --2- 172.21.15.169:0/3890122236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563402630b20 0x563402628590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.764+0000 7f8eb9be6640 1 -- 172.21.15.169:0/3890122236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56340262e690 msgr2=0x5634026305e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.764+0000 7f8eb9be6640 1 --2- 172.21.15.169:0/3890122236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56340262e690 0x5634026305e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.764+0000 7f8eb9be6640 1 -- 172.21.15.169:0/3890122236 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634026474c0 con 0x5634026341f0 2024-03-28T00:57:55.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.764+0000 7f8ebabe8640 1 --2- 172.21.15.169:0/3890122236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563402630b20 0x563402628590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:55.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.764+0000 7f8eba3e7640 1 --2- 172.21.15.169:0/3890122236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56340262e690 0x5634026305e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:55.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.764+0000 7f8eb9be6640 1 --2- 172.21.15.169:0/3890122236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5634026341f0 0x56340263ff90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8ea40ee060 tx=0x7f8ea4009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d1ea40d98cffc2a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:55.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.764+0000 7f8e9ffff640 1 -- 172.21.15.169:0/3890122236 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea40bc910 con 0x5634026341f0 2024-03-28T00:57:55.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.764+0000 7f8e9ffff640 1 -- 172.21.15.169:0/3890122236 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ea4004c80 con 0x5634026341f0 2024-03-28T00:57:55.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.764+0000 7f8e9ffff640 1 -- 172.21.15.169:0/3890122236 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea40a60f0 con 0x5634026341f0 2024-03-28T00:57:55.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.765+0000 7f8ebf237740 1 -- 172.21.15.169:0/3890122236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5634026341f0 msgr2=0x56340263ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.765+0000 7f8ebf237740 1 --2- 172.21.15.169:0/3890122236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5634026341f0 0x56340263ff90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8ea40ee060 tx=0x7f8ea4009680 comp rx=0 tx=0).stop 2024-03-28T00:57:55.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.765+0000 7f8ebf237740 1 -- 172.21.15.169:0/3890122236 shutdown_connections 2024-03-28T00:57:55.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.765+0000 7f8ebf237740 1 --2- 172.21.15.169:0/3890122236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563402630b20 0x563402628590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.765+0000 7f8ebf237740 1 --2- 172.21.15.169:0/3890122236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5634026341f0 0x56340263ff90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.765+0000 7f8ebf237740 1 --2- 172.21.15.169:0/3890122236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56340262e690 0x5634026305e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.765+0000 7f8ebf237740 1 -- 172.21.15.169:0/3890122236 >> 172.21.15.169:0/3890122236 conn(0x563402629c00 msgr2=0x563402629fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:55.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.765+0000 7f8ebf237740 1 -- 172.21.15.169:0/3890122236 shutdown_connections 2024-03-28T00:57:55.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.765+0000 7f8ebf237740 1 -- 172.21.15.169:0/3890122236 wait complete. 2024-03-28T00:57:55.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.765+0000 7f8ebf237740 1 -- start start 2024-03-28T00:57:55.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.765+0000 7f8ebf237740 1 Processor -- start 2024-03-28T00:57:55.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.765+0000 7f8ebf237740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56340262e690 0x56340263fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.765+0000 7f8ebf237740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563402630b20 0x56340263e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.765+0000 7f8ebf237740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5634026341f0 0x56340263e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.765+0000 7f8ebf237740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb41a66f0 con 0x5634026341f0 2024-03-28T00:57:55.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.765+0000 7f8ebf237740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb41a79f0 con 0x56340262e690 2024-03-28T00:57:55.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.766+0000 7f8ebf237740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb41a6d30 con 0x563402630b20 2024-03-28T00:57:55.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.766+0000 7f8ebf237740 10 client.0 _ll_register_callbacks cb 0x7f8ea40f6220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:55.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.766+0000 7f8eba3e7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56340262e690 0x56340263fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.766+0000 7f8ebabe8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5634026341f0 0x56340263e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.766+0000 7f8eba3e7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56340262e690 0x56340263fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:53202/0 (socket says 172.21.15.169:53202) 2024-03-28T00:57:55.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.766+0000 7f8eba3e7640 1 -- 172.21.15.169:0/2166662763 learned_addr learned my addr 172.21.15.169:0/2166662763 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:55.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.766+0000 7f8eb9be6640 1 --2- 172.21.15.169:0/2166662763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563402630b20 0x56340263e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.766+0000 7f8eba3e7640 1 -- 172.21.15.169:0/2166662763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563402630b20 msgr2=0x56340263e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.766+0000 7f8eba3e7640 1 --2- 172.21.15.169:0/2166662763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563402630b20 0x56340263e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.766+0000 7f8eba3e7640 1 -- 172.21.15.169:0/2166662763 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5634026341f0 msgr2=0x56340263e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:55.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.766+0000 7f8eba3e7640 1 --2- 172.21.15.169:0/2166662763 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5634026341f0 0x56340263e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.766+0000 7f8eba3e7640 1 -- 172.21.15.169:0/2166662763 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634026372e0 con 0x56340262e690 2024-03-28T00:57:55.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.766+0000 7f8ebabe8640 1 --2- 172.21.15.169:0/2166662763 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5634026341f0 0x56340263e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:55.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.766+0000 7f8eba3e7640 1 --2- 172.21.15.169:0/2166662763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56340262e690 0x56340263fe30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0009c40 tx=0x7f8eb001fb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbc9dcf6b46f6447 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:55.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.766+0000 7f8e9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:57:55.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.766+0000 7f8e9dffb640 1 -- 172.21.15.169:0/2166662763 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb002c060 con 0x56340262e690 2024-03-28T00:57:55.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.766+0000 7f8ebf237740 1 -- 172.21.15.169:0/2166662763 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ea4004c80 con 0x56340262e690 2024-03-28T00:57:55.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.766+0000 7f8e9dffb640 1 -- 172.21.15.169:0/2166662763 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f8eb001e860 con 0x56340262e690 2024-03-28T00:57:55.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.767+0000 7f8e9dffb640 1 -- 172.21.15.169:0/2166662763 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8eb0033050 con 0x56340262e690 2024-03-28T00:57:55.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.767+0000 7f8ebf237740 10 client.25700 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:55.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.767+0000 7f8ebf237740 1 -- 172.21.15.169:0/2166662763 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5634026a1190 con 0x56340262e690 2024-03-28T00:57:55.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.767+0000 7f8e9dffb640 1 -- 172.21.15.169:0/2166662763 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ea4004c80 con 0x56340262e690 2024-03-28T00:57:55.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.767+0000 7f8e9dffb640 1 -- 172.21.15.169:0/2166662763 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f8eb0010aa0 con 0x56340262e690 2024-03-28T00:57:55.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.767+0000 7f8ebf237740 10 client.25700 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:55.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.767+0000 7f8ebf237740 10 client.25700 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:55.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.767+0000 7f8ebf237740 20 client.25700 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:55.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.767+0000 7f8ebf237740 1 --2- 172.21.15.169:0/2166662763 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5634026a1ad0 0x5634026a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:55.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.767+0000 7f8ebf237740 4 client.25700 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T00:57:55.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.767+0000 7f8ebf237740 1 -- 172.21.15.169:0/2166662763 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5634025a1900 con 0x5634026a1ad0 2024-03-28T00:57:55.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.767+0000 7f8eb9be6640 1 --2- 172.21.15.169:0/2166662763 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5634026a1ad0 0x5634026a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:55.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.767+0000 7f8e9dffb640 10 client.25700 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:55.776 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.767+0000 7f8eb9be6640 1 --2- 172.21.15.169:0/2166662763 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5634026a1ad0 0x5634026a3ef0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4697cfcc1739caa server_cookie=555850b15cd385f3 in_seq=0 out_seq=0 2024-03-28T00:57:55.776 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.768+0000 7f8e9dffb640 1 -- 172.21.15.169:0/2166662763 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5634025a1900 con 0x5634026a1ad0 2024-03-28T00:57:55.776 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.768+0000 7f8e9dffb640 10 client.25700 handle_command_reply: tid=0 2024-03-28T00:57:55.776 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.768+0000 7f8ebf237740 1 client.25700 shutdown 2024-03-28T00:57:55.776 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.768+0000 7f8ebf237740 1 -- 172.21.15.169:0/2166662763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56340262e690 msgr2=0x56340263fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.776 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.768+0000 7f8ebf237740 1 --2- 172.21.15.169:0/2166662763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56340262e690 0x56340263fe30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0009c40 tx=0x7f8eb001fb50 comp rx=0 tx=0).stop 2024-03-28T00:57:55.776 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.768+0000 7f8ebf237740 1 -- 172.21.15.169:0/2166662763 shutdown_connections 2024-03-28T00:57:55.776 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.768+0000 7f8ebf237740 1 --2- 172.21.15.169:0/2166662763 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5634026a1ad0 0x5634026a3ef0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.776 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.768+0000 7f8ebf237740 1 --2- 172.21.15.169:0/2166662763 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5634026341f0 0x56340263e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.776 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.768+0000 7f8e9dffb640 0 client.25700 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:55.776 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.768+0000 7f8ebf237740 1 --2- 172.21.15.169:0/2166662763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563402630b20 0x56340263e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.776 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.768+0000 7f8ebf237740 1 --2- 172.21.15.169:0/2166662763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56340262e690 0x56340263fe30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.776 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.768+0000 7f8ebf237740 1 -- 172.21.15.169:0/2166662763 >> 172.21.15.169:0/2166662763 conn(0x563402629c00 msgr2=0x56340262ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:55.776 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.768+0000 7f8ebf237740 1 -- 172.21.15.169:0/2166662763 shutdown_connections 2024-03-28T00:57:55.777 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.768+0000 7f8ebf237740 1 -- 172.21.15.169:0/2166662763 wait complete. 2024-03-28T00:57:55.777 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.768+0000 7f8ebf237740 20 client.25700 trim_cache size 0 max 16384 2024-03-28T00:57:55.777 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T00:57:55.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.768+0000 7f8ebc64b640 1 -- 172.21.15.169:0/2365040017 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8ea00a2910 msgr2=0x7f8ea00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.768+0000 7f8ebc64b640 1 --2- 172.21.15.169:0/2365040017 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8ea00a2910 0x7f8ea00a4dd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8eac027000 tx=0x7f8eac028000 comp rx=0 tx=0).stop 2024-03-28T00:57:55.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.768+0000 7f8ebc64b640 1 -- 172.21.15.169:0/2365040017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8eb4104a50 msgr2=0x7f8eb4195d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:55.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.768+0000 7f8ebc64b640 1 --2- 172.21.15.169:0/2365040017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8eb4104a50 0x7f8eb4195d20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4000c00 tx=0x7f8ea402c640 comp rx=0 tx=0).stop 2024-03-28T00:57:55.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.769+0000 7f8ebc64b640 1 -- 172.21.15.169:0/2365040017 shutdown_connections 2024-03-28T00:57:55.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.769+0000 7f8ebc64b640 1 --2- 172.21.15.169:0/2365040017 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8ea00a2910 0x7f8ea00a4dd0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.769+0000 7f8ebc64b640 1 --2- 172.21.15.169:0/2365040017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8eb4107380 0x7f8eb419cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.769+0000 7f8ebc64b640 1 --2- 172.21.15.169:0/2365040017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8eb4104a50 0x7f8eb4195d20 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.769+0000 7f8ebc64b640 1 --2- 172.21.15.169:0/2365040017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8eb4102120 0x7f8eb41957e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:55.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.769+0000 7f8ebc64b640 1 -- 172.21.15.169:0/2365040017 >> 172.21.15.169:0/2365040017 conn(0x7f8eb40fbb30 msgr2=0x7f8eb4102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:55.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.769+0000 7f8ebc64b640 1 -- 172.21.15.169:0/2365040017 shutdown_connections 2024-03-28T00:57:55.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:55.769+0000 7f8ebc64b640 1 -- 172.21.15.169:0/2365040017 wait complete. 2024-03-28T00:57:56.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:56 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3211638895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:57:56.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:56 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3211638895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:57:56.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:56 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3211638895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:57:56.784 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T00:57:56.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:57:56.784 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T00:57:56.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.937+0000 7f4ab3675640 1 -- 172.21.15.169:0/3438150940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aac1017b0 msgr2=0x7f4aac101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:56.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.937+0000 7f4ab3675640 1 --2- 172.21.15.169:0/3438150940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aac1017b0 0x7f4aac101c30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009900 tx=0x7f4a9c02bc50 comp rx=0 tx=0).stop 2024-03-28T00:57:56.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.937+0000 7f4ab3675640 1 -- 172.21.15.169:0/3438150940 shutdown_connections 2024-03-28T00:57:56.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.937+0000 7f4ab3675640 1 --2- 172.21.15.169:0/3438150940 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4aac102170 0x7f4aac108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.937+0000 7f4ab3675640 1 --2- 172.21.15.169:0/3438150940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aac1017b0 0x7f4aac101c30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.937+0000 7f4ab3675640 1 --2- 172.21.15.169:0/3438150940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4aac1005c0 0x7f4aac1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.937+0000 7f4ab3675640 1 -- 172.21.15.169:0/3438150940 >> 172.21.15.169:0/3438150940 conn(0x7f4aac0fbb30 msgr2=0x7f4aac0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:56.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.937+0000 7f4ab3675640 1 -- 172.21.15.169:0/3438150940 shutdown_connections 2024-03-28T00:57:56.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.937+0000 7f4ab3675640 1 -- 172.21.15.169:0/3438150940 wait complete. 2024-03-28T00:57:56.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.937+0000 7f4ab3675640 1 Processor -- start 2024-03-28T00:57:56.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.937+0000 7f4ab3675640 1 -- start start 2024-03-28T00:57:56.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.938+0000 7f4ab3675640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aac1005c0 0x7f4aac197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:56.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.938+0000 7f4ab3675640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4aac1017b0 0x7f4aac197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:56.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.938+0000 7f4ab3675640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4aac102170 0x7f4aac19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:56.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.938+0000 7f4ab3675640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac10c280 con 0x7f4aac102170 2024-03-28T00:57:56.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.938+0000 7f4ab3675640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac10c100 con 0x7f4aac1017b0 2024-03-28T00:57:56.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.938+0000 7f4ab3675640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac10c400 con 0x7f4aac1005c0 2024-03-28T00:57:56.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.938+0000 7f4ab1411640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aac1005c0 0x7f4aac197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:56.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.938+0000 7f4ab1c12640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4aac102170 0x7f4aac19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:56.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.938+0000 7f4ab1411640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aac1005c0 0x7f4aac197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:58758/0 (socket says 172.21.15.169:58758) 2024-03-28T00:57:56.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.938+0000 7f4ab1411640 1 -- 172.21.15.169:0/944412887 learned_addr learned my addr 172.21.15.169:0/944412887 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:56.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.938+0000 7f4ab0c10640 1 --2- 172.21.15.169:0/944412887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4aac1017b0 0x7f4aac197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:56.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.938+0000 7f4ab1411640 1 -- 172.21.15.169:0/944412887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4aac1017b0 msgr2=0x7f4aac197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:56.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.938+0000 7f4ab1411640 1 --2- 172.21.15.169:0/944412887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4aac1017b0 0x7f4aac197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.938+0000 7f4ab1411640 1 -- 172.21.15.169:0/944412887 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4aac102170 msgr2=0x7f4aac19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:56.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.938+0000 7f4ab1411640 1 --2- 172.21.15.169:0/944412887 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4aac102170 0x7f4aac19f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.938+0000 7f4ab1411640 1 -- 172.21.15.169:0/944412887 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac19f670 con 0x7f4aac1005c0 2024-03-28T00:57:56.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.938+0000 7f4ab0c10640 1 --2- 172.21.15.169:0/944412887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4aac1017b0 0x7f4aac197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:57:56.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.938+0000 7f4ab1c12640 1 --2- 172.21.15.169:0/944412887 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4aac102170 0x7f4aac19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:56.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.938+0000 7f4ab1411640 1 --2- 172.21.15.169:0/944412887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aac1005c0 0x7f4aac197a50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4aa800d770 tx=0x7f4aa800db70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:56.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.939+0000 7f4aa27fc640 1 -- 172.21.15.169:0/944412887 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa8017070 con 0x7f4aac1005c0 2024-03-28T00:57:56.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.939+0000 7f4ab3675640 1 -- 172.21.15.169:0/944412887 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aac19f960 con 0x7f4aac1005c0 2024-03-28T00:57:56.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.939+0000 7f4aa27fc640 1 -- 172.21.15.169:0/944412887 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4aa8009d00 con 0x7f4aac1005c0 2024-03-28T00:57:56.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.939+0000 7f4aa27fc640 1 -- 172.21.15.169:0/944412887 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa8003d30 con 0x7f4aac1005c0 2024-03-28T00:57:56.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.939+0000 7f4ab3675640 1 -- 172.21.15.169:0/944412887 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aac19feb0 con 0x7f4aac1005c0 2024-03-28T00:57:56.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.941+0000 7f4aa27fc640 1 -- 172.21.15.169:0/944412887 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f4aa8009820 con 0x7f4aac1005c0 2024-03-28T00:57:56.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.941+0000 7f4aa27fc640 1 --2- 172.21.15.169:0/944412887 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4a980a2890 0x7f4a980a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:56.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.941+0000 7f4aa27fc640 1 -- 172.21.15.169:0/944412887 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f4aa80ce910 con 0x7f4aac1005c0 2024-03-28T00:57:56.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.942+0000 7f4ab0c10640 1 --2- 172.21.15.169:0/944412887 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4a980a2890 0x7f4a980a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:56.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.942+0000 7f4ab0c10640 1 --2- 172.21.15.169:0/944412887 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4a980a2890 0x7f4a980a4d50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4aac198f60 tx=0x7f4a9c004cb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:56.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.943+0000 7f4ab625a740 1 Processor -- start 2024-03-28T00:57:56.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.943+0000 7f4ab625a740 1 -- start start 2024-03-28T00:57:56.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.943+0000 7f4ab625a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5e960 0x55d630c5ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:56.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.943+0000 7f4ab625a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d630c5f350 0x55d630c63e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:56.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.943+0000 7f4ab625a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c644c0 0x55d630c66950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:56.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.943+0000 7f4ab625a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d630b9ec90 con 0x55d630c5e960 2024-03-28T00:57:56.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.943+0000 7f4ab625a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d630bc4070 con 0x55d630c5f350 2024-03-28T00:57:56.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.943+0000 7f4ab625a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d630b6cc80 con 0x55d630c644c0 2024-03-28T00:57:56.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.943+0000 7f4ab1411640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5e960 0x55d630c5ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:56.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.943+0000 7f4ab1c12640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c644c0 0x55d630c66950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:56.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.943+0000 7f4ab1411640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5e960 0x55d630c5ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39292/0 (socket says 172.21.15.169:39292) 2024-03-28T00:57:56.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.943+0000 7f4ab0c10640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d630c5f350 0x55d630c63e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:56.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.943+0000 7f4ab1c12640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c644c0 0x55d630c66950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:58768/0 (socket says 172.21.15.169:58768) 2024-03-28T00:57:56.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.943+0000 7f4ab1411640 1 -- 172.21.15.169:0/3657530793 learned_addr learned my addr 172.21.15.169:0/3657530793 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:56.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.944+0000 7f4ab1c12640 1 -- 172.21.15.169:0/3657530793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d630c5f350 msgr2=0x55d630c63e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:56.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.944+0000 7f4ab1c12640 1 --2- 172.21.15.169:0/3657530793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d630c5f350 0x55d630c63e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.944+0000 7f4ab1c12640 1 -- 172.21.15.169:0/3657530793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5e960 msgr2=0x55d630c5ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:56.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.944+0000 7f4ab1c12640 1 --2- 172.21.15.169:0/3657530793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5e960 0x55d630c5ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.944+0000 7f4ab1c12640 1 -- 172.21.15.169:0/3657530793 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d630b64d10 con 0x55d630c644c0 2024-03-28T00:57:56.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.944+0000 7f4ab1c12640 1 --2- 172.21.15.169:0/3657530793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c644c0 0x55d630c66950 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4011760 tx=0x7f4aa4011c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2370b6406bd81626 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:56.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.944+0000 7f4a977fe640 1 -- 172.21.15.169:0/3657530793 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4004d20 con 0x55d630c644c0 2024-03-28T00:57:56.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.944+0000 7f4a977fe640 1 -- 172.21.15.169:0/3657530793 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4aa4003970 con 0x55d630c644c0 2024-03-28T00:57:56.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.944+0000 7f4a977fe640 1 -- 172.21.15.169:0/3657530793 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4004580 con 0x55d630c644c0 2024-03-28T00:57:56.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.945+0000 7f4ab625a740 1 -- 172.21.15.169:0/3657530793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c644c0 msgr2=0x55d630c66950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:56.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.945+0000 7f4ab625a740 1 --2- 172.21.15.169:0/3657530793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c644c0 0x55d630c66950 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4011760 tx=0x7f4aa4011c20 comp rx=0 tx=0).stop 2024-03-28T00:57:56.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.945+0000 7f4ab625a740 1 -- 172.21.15.169:0/3657530793 shutdown_connections 2024-03-28T00:57:56.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.945+0000 7f4ab625a740 1 --2- 172.21.15.169:0/3657530793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c644c0 0x55d630c66950 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.945+0000 7f4ab625a740 1 --2- 172.21.15.169:0/3657530793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d630c5f350 0x55d630c63e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.945+0000 7f4ab625a740 1 --2- 172.21.15.169:0/3657530793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5e960 0x55d630c5ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.945+0000 7f4ab625a740 1 -- 172.21.15.169:0/3657530793 >> 172.21.15.169:0/3657530793 conn(0x55d630c271a0 msgr2=0x55d630c5a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:56.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.945+0000 7f4ab625a740 1 -- 172.21.15.169:0/3657530793 shutdown_connections 2024-03-28T00:57:56.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.945+0000 7f4ab625a740 1 -- 172.21.15.169:0/3657530793 wait complete. 2024-03-28T00:57:56.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.945+0000 7f4ab625a740 1 -- start start 2024-03-28T00:57:56.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.946+0000 7f4ab625a740 1 Processor -- start 2024-03-28T00:57:56.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.946+0000 7f4ab625a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5e960 0x55d630c68cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:56.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.946+0000 7f4ab625a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c5f350 0x55d630c69200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:56.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.946+0000 7f4ab625a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d630c644c0 0x55d630c70240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:56.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.946+0000 7f4ab1411640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5e960 0x55d630c68cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:56.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.946+0000 7f4ab1411640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5e960 0x55d630c68cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39300/0 (socket says 172.21.15.169:39300) 2024-03-28T00:57:56.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.946+0000 7f4ab1411640 1 -- 172.21.15.169:0/4281616668 learned_addr learned my addr 172.21.15.169:0/4281616668 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:56.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.946+0000 7f4ab625a740 1 -- 172.21.15.169:0/4281616668 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d630b800d0 con 0x55d630c5e960 2024-03-28T00:57:56.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.946+0000 7f4ab625a740 1 -- 172.21.15.169:0/4281616668 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d630bc0110 con 0x55d630c644c0 2024-03-28T00:57:56.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.946+0000 7f4ab625a740 1 -- 172.21.15.169:0/4281616668 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d630c67470 con 0x55d630c5f350 2024-03-28T00:57:56.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.946+0000 7f4ab0c10640 1 --2- 172.21.15.169:0/4281616668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c5f350 0x55d630c69200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:56.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.946+0000 7f4ab1c12640 1 --2- 172.21.15.169:0/4281616668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d630c644c0 0x55d630c70240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:56.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.946+0000 7f4ab625a740 10 client.0 _ll_register_callbacks cb 0x55d630bbbe60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:56.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.946+0000 7f4ab1411640 1 -- 172.21.15.169:0/4281616668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c5f350 msgr2=0x55d630c69200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:56.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.946+0000 7f4ab1411640 1 --2- 172.21.15.169:0/4281616668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c5f350 0x55d630c69200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.946+0000 7f4ab1411640 1 -- 172.21.15.169:0/4281616668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d630c644c0 msgr2=0x55d630c70240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:56.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.946+0000 7f4ab1411640 1 --2- 172.21.15.169:0/4281616668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d630c644c0 0x55d630c70240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.946+0000 7f4ab1411640 1 -- 172.21.15.169:0/4281616668 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d6309a07e0 con 0x55d630c5e960 2024-03-28T00:57:56.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.947+0000 7f4ab0c10640 1 --2- 172.21.15.169:0/4281616668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c5f350 0x55d630c69200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:56.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.947+0000 7f4ab625a740 1 -- 172.21.15.169:0/4281616668 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4aa4003970 con 0x55d630c5e960 2024-03-28T00:57:56.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.947+0000 7f4ab1411640 1 --2- 172.21.15.169:0/4281616668 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5e960 0x55d630c68cc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4aa809aab0 tx=0x7f4aa809a9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd3bb2183f67da79 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:56.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.947+0000 7f4a94ff9640 10 client.15891 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:57:56.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.947+0000 7f4a94ff9640 1 -- 172.21.15.169:0/4281616668 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa8020050 con 0x55d630c5e960 2024-03-28T00:57:56.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.947+0000 7f4a94ff9640 1 -- 172.21.15.169:0/4281616668 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f4aa8097420 con 0x55d630c5e960 2024-03-28T00:57:56.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.947+0000 7f4a94ff9640 1 -- 172.21.15.169:0/4281616668 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4aa809ef00 con 0x55d630c5e960 2024-03-28T00:57:56.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.947+0000 7f4ab625a740 10 client.15891 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:56.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.947+0000 7f4ab625a740 1 -- 172.21.15.169:0/4281616668 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x55d630a00f70 con 0x55d630c5e960 2024-03-28T00:57:56.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.947+0000 7f4a94ff9640 1 -- 172.21.15.169:0/4281616668 <== mon.0 v2:172.21.15.169:3300/0 4 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f4aa809a200 con 0x55d630c5e960 2024-03-28T00:57:56.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.947+0000 7f4ab625a740 10 client.15891 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:56.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.947+0000 7f4ab625a740 10 client.15891 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:56.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.947+0000 7f4ab625a740 20 client.15891 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:56.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.947+0000 7f4ab625a740 1 --2- 172.21.15.169:0/4281616668 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d630c749e0 0x55d630c76e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:56.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.948+0000 7f4ab625a740 4 client.15891 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:57:56.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.948+0000 7f4ab625a740 1 -- 172.21.15.169:0/4281616668 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d630bd15b0 con 0x55d630c749e0 2024-03-28T00:57:56.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.948+0000 7f4ab0c10640 1 --2- 172.21.15.169:0/4281616668 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d630c749e0 0x55d630c76e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:56.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.948+0000 7f4a94ff9640 10 client.15891 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:56.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.948+0000 7f4ab0c10640 1 --2- 172.21.15.169:0/4281616668 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d630c749e0 0x55d630c76e60 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5786abaab6c780cf server_cookie=6ee2362dcf1eed48 in_seq=0 out_seq=0 2024-03-28T00:57:56.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.949+0000 7f4a94ff9640 1 -- 172.21.15.169:0/4281616668 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d630bd15b0 con 0x55d630c749e0 2024-03-28T00:57:56.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.949+0000 7f4a94ff9640 10 client.15891 handle_command_reply: tid=0 2024-03-28T00:57:56.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.949+0000 7f4ab625a740 1 client.15891 shutdown 2024-03-28T00:57:56.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.949+0000 7f4ab625a740 1 -- 172.21.15.169:0/4281616668 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5e960 msgr2=0x55d630c68cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:56.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.949+0000 7f4ab625a740 1 --2- 172.21.15.169:0/4281616668 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5e960 0x55d630c68cc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4aa809aab0 tx=0x7f4aa809a9b0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.949+0000 7f4ab625a740 1 -- 172.21.15.169:0/4281616668 shutdown_connections 2024-03-28T00:57:56.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.949+0000 7f4ab625a740 1 --2- 172.21.15.169:0/4281616668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d630c644c0 0x55d630c70240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.949+0000 7f4ab625a740 1 --2- 172.21.15.169:0/4281616668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c5f350 0x55d630c69200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.949+0000 7f4ab625a740 1 --2- 172.21.15.169:0/4281616668 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d630c749e0 0x55d630c76e60 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.949+0000 7f4a94ff9640 0 client.15891 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:56.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.949+0000 7f4ab625a740 1 --2- 172.21.15.169:0/4281616668 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5e960 0x55d630c68cc0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.949+0000 7f4ab625a740 1 -- 172.21.15.169:0/4281616668 >> 172.21.15.169:0/4281616668 conn(0x55d630c271a0 msgr2=0x55d630c59530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:56.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.949+0000 7f4ab625a740 1 -- 172.21.15.169:0/4281616668 shutdown_connections 2024-03-28T00:57:56.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.949+0000 7f4ab625a740 1 -- 172.21.15.169:0/4281616668 wait complete. 2024-03-28T00:57:56.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.949+0000 7f4ab625a740 20 client.15891 trim_cache size 0 max 16384 2024-03-28T00:57:56.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.980+0000 7f4ab625a740 1 Processor -- start 2024-03-28T00:57:56.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.981+0000 7f4ab625a740 1 -- start start 2024-03-28T00:57:56.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.981+0000 7f4ab625a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c5e960 0x55d630c7b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:56.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.981+0000 7f4ab625a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5f350 0x55d630c70200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:56.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.981+0000 7f4ab625a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d630c644c0 0x55d630c7bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:56.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.981+0000 7f4ab625a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac1aa570 con 0x55d630c5f350 2024-03-28T00:57:56.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.981+0000 7f4ab625a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac1a8070 con 0x55d630c644c0 2024-03-28T00:57:56.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.981+0000 7f4ab625a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac1a6110 con 0x55d630c5e960 2024-03-28T00:57:56.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.981+0000 7f4ab0c10640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5f350 0x55d630c70200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:56.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.981+0000 7f4ab1411640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c5e960 0x55d630c7b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:56.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.981+0000 7f4ab1411640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c5e960 0x55d630c7b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:58780/0 (socket says 172.21.15.169:58780) 2024-03-28T00:57:56.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.981+0000 7f4ab1411640 1 -- 172.21.15.169:0/4021164738 learned_addr learned my addr 172.21.15.169:0/4021164738 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:56.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.981+0000 7f4ab1c12640 1 --2- 172.21.15.169:0/4021164738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d630c644c0 0x55d630c7bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:56.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.981+0000 7f4ab1411640 1 -- 172.21.15.169:0/4021164738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d630c644c0 msgr2=0x55d630c7bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:56.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.981+0000 7f4ab1411640 1 --2- 172.21.15.169:0/4021164738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d630c644c0 0x55d630c7bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.981+0000 7f4ab1411640 1 -- 172.21.15.169:0/4021164738 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5f350 msgr2=0x55d630c70200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:56.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.981+0000 7f4ab1411640 1 --2- 172.21.15.169:0/4021164738 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5f350 0x55d630c70200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.981+0000 7f4ab1411640 1 -- 172.21.15.169:0/4021164738 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d630898170 con 0x55d630c5e960 2024-03-28T00:57:56.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.981+0000 7f4ab0c10640 1 --2- 172.21.15.169:0/4021164738 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5f350 0x55d630c70200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:56.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.982+0000 7f4ab1411640 1 --2- 172.21.15.169:0/4021164738 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c5e960 0x55d630c7b720 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4aa80cebf0 tx=0x7f4aa80ce000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98775203a1782918 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:56.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.982+0000 7f4a96ffd640 1 -- 172.21.15.169:0/4021164738 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa80a3550 con 0x55d630c5e960 2024-03-28T00:57:56.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.982+0000 7f4a96ffd640 1 -- 172.21.15.169:0/4021164738 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4aa4003970 con 0x55d630c5e960 2024-03-28T00:57:56.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.982+0000 7f4a96ffd640 1 -- 172.21.15.169:0/4021164738 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa8097420 con 0x55d630c5e960 2024-03-28T00:57:56.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.982+0000 7f4ab625a740 1 -- 172.21.15.169:0/4021164738 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c5e960 msgr2=0x55d630c7b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:56.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.982+0000 7f4ab625a740 1 --2- 172.21.15.169:0/4021164738 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c5e960 0x55d630c7b720 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4aa80cebf0 tx=0x7f4aa80ce000 comp rx=0 tx=0).stop 2024-03-28T00:57:56.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.982+0000 7f4ab625a740 1 -- 172.21.15.169:0/4021164738 shutdown_connections 2024-03-28T00:57:56.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.982+0000 7f4ab625a740 1 --2- 172.21.15.169:0/4021164738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d630c644c0 0x55d630c7bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.983+0000 7f4ab625a740 1 --2- 172.21.15.169:0/4021164738 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5f350 0x55d630c70200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.983+0000 7f4ab625a740 1 --2- 172.21.15.169:0/4021164738 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c5e960 0x55d630c7b720 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.983+0000 7f4ab625a740 1 -- 172.21.15.169:0/4021164738 >> 172.21.15.169:0/4021164738 conn(0x55d630c271a0 msgr2=0x55d630c59530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:56.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.983+0000 7f4ab625a740 1 -- 172.21.15.169:0/4021164738 shutdown_connections 2024-03-28T00:57:56.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.983+0000 7f4ab625a740 1 -- 172.21.15.169:0/4021164738 wait complete. 2024-03-28T00:57:56.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.983+0000 7f4ab625a740 1 -- start start 2024-03-28T00:57:56.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.983+0000 7f4ab625a740 1 Processor -- start 2024-03-28T00:57:56.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.983+0000 7f4ab625a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d630c5e960 0x55d630c6f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:56.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.983+0000 7f4ab625a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5f350 0x55d630c6ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:56.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.983+0000 7f4ab625a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c644c0 0x55d630c6dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:56.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.983+0000 7f4ab625a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac1a6710 con 0x55d630c5f350 2024-03-28T00:57:56.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.983+0000 7f4ab625a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac1a7a10 con 0x55d630c5e960 2024-03-28T00:57:56.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.983+0000 7f4ab625a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac1a6d50 con 0x55d630c644c0 2024-03-28T00:57:56.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.983+0000 7f4ab625a740 10 client.0 _ll_register_callbacks cb 0x7f4aa800d6c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:56.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.984+0000 7f4ab0c10640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5f350 0x55d630c6ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:56.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.984+0000 7f4ab1411640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d630c5e960 0x55d630c6f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:56.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.984+0000 7f4ab0c10640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5f350 0x55d630c6ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39304/0 (socket says 172.21.15.169:39304) 2024-03-28T00:57:56.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.984+0000 7f4ab0c10640 1 -- 172.21.15.169:0/2505166632 learned_addr learned my addr 172.21.15.169:0/2505166632 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:56.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.984+0000 7f4ab1c12640 1 --2- 172.21.15.169:0/2505166632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c644c0 0x55d630c6dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:56.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.984+0000 7f4ab0c10640 1 -- 172.21.15.169:0/2505166632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c644c0 msgr2=0x55d630c6dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:56.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.984+0000 7f4ab0c10640 1 --2- 172.21.15.169:0/2505166632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c644c0 0x55d630c6dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.984+0000 7f4ab0c10640 1 -- 172.21.15.169:0/2505166632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d630c5e960 msgr2=0x55d630c6f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:56.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.984+0000 7f4ab0c10640 1 --2- 172.21.15.169:0/2505166632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d630c5e960 0x55d630c6f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.984+0000 7f4ab0c10640 1 -- 172.21.15.169:0/2505166632 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d630c6e620 con 0x55d630c5f350 2024-03-28T00:57:56.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.984+0000 7f4ab1411640 1 --2- 172.21.15.169:0/2505166632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d630c5e960 0x55d630c6f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:56.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.984+0000 7f4ab0c10640 1 --2- 172.21.15.169:0/2505166632 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5f350 0x55d630c6ff10 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009230 tx=0x7f4a9c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e4c374c425588f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:56.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.984+0000 7f4a94ff9640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:57:56.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.984+0000 7f4a94ff9640 1 -- 172.21.15.169:0/2505166632 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c05b060 con 0x55d630c5f350 2024-03-28T00:57:56.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.984+0000 7f4ab625a740 1 -- 172.21.15.169:0/2505166632 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4aa4003970 con 0x55d630c5f350 2024-03-28T00:57:56.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.984+0000 7f4a94ff9640 1 -- 172.21.15.169:0/2505166632 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f4a9c003980 con 0x55d630c5f350 2024-03-28T00:57:56.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.984+0000 7f4a94ff9640 1 -- 172.21.15.169:0/2505166632 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a9c062050 con 0x55d630c5f350 2024-03-28T00:57:56.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.984+0000 7f4ab625a740 10 client.15897 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:56.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.984+0000 7f4ab625a740 1 -- 172.21.15.169:0/2505166632 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d630cd0d40 con 0x55d630c5f350 2024-03-28T00:57:56.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.985+0000 7f4a94ff9640 1 -- 172.21.15.169:0/2505166632 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4aa4003970 con 0x55d630c5f350 2024-03-28T00:57:56.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.985+0000 7f4a94ff9640 1 -- 172.21.15.169:0/2505166632 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f4a9c004d90 con 0x55d630c5f350 2024-03-28T00:57:56.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.985+0000 7f4ab625a740 10 client.15897 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:56.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.985+0000 7f4ab625a740 10 client.15897 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:56.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.985+0000 7f4ab625a740 20 client.15897 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:56.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.985+0000 7f4ab625a740 1 --2- 172.21.15.169:0/2505166632 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d630c749e0 0x55d630cd36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:56.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.985+0000 7f4ab625a740 4 client.15897 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T00:57:56.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.985+0000 7f4ab625a740 1 -- 172.21.15.169:0/2505166632 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55d630bcb900 con 0x55d630c749e0 2024-03-28T00:57:56.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.985+0000 7f4ab1411640 1 --2- 172.21.15.169:0/2505166632 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d630c749e0 0x55d630cd36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:56.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.986+0000 7f4a94ff9640 10 client.15897 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:56.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.986+0000 7f4ab1411640 1 --2- 172.21.15.169:0/2505166632 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d630c749e0 0x55d630cd36e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=857c236a1c03cf1b server_cookie=3a9ccae52d26d643 in_seq=0 out_seq=0 2024-03-28T00:57:56.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.986+0000 7f4a94ff9640 1 -- 172.21.15.169:0/2505166632 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55d630bcb900 con 0x55d630c749e0 2024-03-28T00:57:56.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.986+0000 7f4a94ff9640 10 client.15897 handle_command_reply: tid=0 2024-03-28T00:57:56.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.987+0000 7f4ab625a740 1 client.15897 shutdown 2024-03-28T00:57:56.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.987+0000 7f4ab625a740 1 -- 172.21.15.169:0/2505166632 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5f350 msgr2=0x55d630c6ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:56.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.987+0000 7f4ab625a740 1 --2- 172.21.15.169:0/2505166632 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5f350 0x55d630c6ff10 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009230 tx=0x7f4a9c009260 comp rx=0 tx=0).stop 2024-03-28T00:57:56.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.987+0000 7f4ab625a740 1 -- 172.21.15.169:0/2505166632 shutdown_connections 2024-03-28T00:57:56.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.987+0000 7f4ab625a740 1 --2- 172.21.15.169:0/2505166632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d630c644c0 0x55d630c6dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.987+0000 7f4ab625a740 1 --2- 172.21.15.169:0/2505166632 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d630c749e0 0x55d630cd36e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.987+0000 7f4ab625a740 1 --2- 172.21.15.169:0/2505166632 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d630c5f350 0x55d630c6ff10 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.987+0000 7f4ab625a740 1 --2- 172.21.15.169:0/2505166632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d630c5e960 0x55d630c6f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.987+0000 7f4a94ff9640 0 client.15897 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:56.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.987+0000 7f4ab625a740 1 -- 172.21.15.169:0/2505166632 >> 172.21.15.169:0/2505166632 conn(0x55d630c271a0 msgr2=0x55d630c58ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:56.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.987+0000 7f4ab625a740 1 -- 172.21.15.169:0/2505166632 shutdown_connections 2024-03-28T00:57:56.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.987+0000 7f4ab625a740 1 -- 172.21.15.169:0/2505166632 wait complete. 2024-03-28T00:57:56.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.988+0000 7f4ab625a740 20 client.15897 trim_cache size 0 max 16384 2024-03-28T00:57:56.995 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:57:56.995 INFO:teuthology.orchestra.run.smithi169.stdout: "return_code": 0, 2024-03-28T00:57:56.995 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_tag": "83cb7ad5-2b46-42e7-91bd-2a9351262c8b", 2024-03-28T00:57:56.995 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": "asynchronous" 2024-03-28T00:57:56.995 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:57:56.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.988+0000 7f4ab3675640 1 -- 172.21.15.169:0/944412887 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4a980a2890 msgr2=0x7f4a980a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:56.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.988+0000 7f4ab3675640 1 --2- 172.21.15.169:0/944412887 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4a980a2890 0x7f4a980a4d50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4aac198f60 tx=0x7f4a9c004cb0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.988+0000 7f4ab3675640 1 -- 172.21.15.169:0/944412887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aac1005c0 msgr2=0x7f4aac197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:56.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.988+0000 7f4ab3675640 1 --2- 172.21.15.169:0/944412887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aac1005c0 0x7f4aac197a50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4aa800d770 tx=0x7f4aa800db70 comp rx=0 tx=0).stop 2024-03-28T00:57:56.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.988+0000 7f4ab3675640 1 -- 172.21.15.169:0/944412887 shutdown_connections 2024-03-28T00:57:56.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.988+0000 7f4ab3675640 1 --2- 172.21.15.169:0/944412887 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4a980a2890 0x7f4a980a4d50 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.988+0000 7f4ab3675640 1 --2- 172.21.15.169:0/944412887 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4aac102170 0x7f4aac19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.988+0000 7f4ab3675640 1 --2- 172.21.15.169:0/944412887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4aac1017b0 0x7f4aac197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.988+0000 7f4ab3675640 1 --2- 172.21.15.169:0/944412887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aac1005c0 0x7f4aac197a50 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:56.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.988+0000 7f4ab3675640 1 -- 172.21.15.169:0/944412887 >> 172.21.15.169:0/944412887 conn(0x7f4aac0fbb30 msgr2=0x7f4aac0fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:56.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.988+0000 7f4ab3675640 1 -- 172.21.15.169:0/944412887 shutdown_connections 2024-03-28T00:57:56.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:56.988+0000 7f4ab3675640 1 -- 172.21.15.169:0/944412887 wait complete. 2024-03-28T00:57:57.002 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T00:57:57.002 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T00:57:57.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.106+0000 7f28a09b2640 1 -- 172.21.15.169:0/1048195031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f289c1005a0 msgr2=0x7f289c1009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:57.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.106+0000 7f28a09b2640 1 --2- 172.21.15.169:0/1048195031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f289c1005a0 0x7f289c1009c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f288c009860 tx=0x7f288c02bb70 comp rx=0 tx=0).stop 2024-03-28T00:57:57.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.106+0000 7f28a09b2640 1 -- 172.21.15.169:0/1048195031 shutdown_connections 2024-03-28T00:57:57.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.106+0000 7f28a09b2640 1 --2- 172.21.15.169:0/1048195031 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f289c102150 0x7f289c1089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.106+0000 7f28a09b2640 1 --2- 172.21.15.169:0/1048195031 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f289c101790 0x7f289c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.106+0000 7f28a09b2640 1 --2- 172.21.15.169:0/1048195031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f289c1005a0 0x7f289c1009c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.106+0000 7f28a09b2640 1 -- 172.21.15.169:0/1048195031 >> 172.21.15.169:0/1048195031 conn(0x7f289c0fbb30 msgr2=0x7f289c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:57.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.106+0000 7f28a09b2640 1 -- 172.21.15.169:0/1048195031 shutdown_connections 2024-03-28T00:57:57.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.106+0000 7f28a09b2640 1 -- 172.21.15.169:0/1048195031 wait complete. 2024-03-28T00:57:57.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.107+0000 7f28a09b2640 1 Processor -- start 2024-03-28T00:57:57.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.107+0000 7f28a09b2640 1 -- start start 2024-03-28T00:57:57.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.107+0000 7f28a09b2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f289c1005a0 0x7f289c1968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.107+0000 7f28a09b2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f289c101790 0x7f289c196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.107+0000 7f28a09b2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f289c102150 0x7f289c19de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.107+0000 7f28a09b2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289c10c1f0 con 0x7f289c101790 2024-03-28T00:57:57.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.107+0000 7f28a09b2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289c10c070 con 0x7f289c102150 2024-03-28T00:57:57.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.107+0000 7f28a09b2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289c10c370 con 0x7f289c1005a0 2024-03-28T00:57:57.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.107+0000 7f2899d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f289c101790 0x7f289c196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.107+0000 7f289a59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f289c1005a0 0x7f289c1968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.107+0000 7f289ad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f289c102150 0x7f289c19de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.107+0000 7f2899d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f289c101790 0x7f289c196e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39328/0 (socket says 172.21.15.169:39328) 2024-03-28T00:57:57.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.107+0000 7f289a59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f289c1005a0 0x7f289c1968c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:58808/0 (socket says 172.21.15.169:58808) 2024-03-28T00:57:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.107+0000 7f2899d9b640 1 -- 172.21.15.169:0/3362819000 learned_addr learned my addr 172.21.15.169:0/3362819000 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.107+0000 7f2899d9b640 1 -- 172.21.15.169:0/3362819000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f289c1005a0 msgr2=0x7f289c1968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.107+0000 7f2899d9b640 1 --2- 172.21.15.169:0/3362819000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f289c1005a0 0x7f289c1968c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.107+0000 7f2899d9b640 1 -- 172.21.15.169:0/3362819000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f289c102150 msgr2=0x7f289c19de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.107+0000 7f2899d9b640 1 --2- 172.21.15.169:0/3362819000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f289c102150 0x7f289c19de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.107+0000 7f2899d9b640 1 -- 172.21.15.169:0/3362819000 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289c068800 con 0x7f289c101790 2024-03-28T00:57:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.108+0000 7f289a59c640 1 --2- 172.21.15.169:0/3362819000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f289c1005a0 0x7f289c1968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.108+0000 7f289ad9d640 1 --2- 172.21.15.169:0/3362819000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f289c102150 0x7f289c19de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.108+0000 7f2899d9b640 1 --2- 172.21.15.169:0/3362819000 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f289c101790 0x7f289c196e00 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f289000e9c0 tx=0x7f289000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.108+0000 7f288b7fe640 1 -- 172.21.15.169:0/3362819000 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289000cde0 con 0x7f289c101790 2024-03-28T00:57:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.108+0000 7f28a09b2640 1 -- 172.21.15.169:0/3362819000 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f289c068af0 con 0x7f289c101790 2024-03-28T00:57:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.108+0000 7f288b7fe640 1 -- 172.21.15.169:0/3362819000 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28900035f0 con 0x7f289c101790 2024-03-28T00:57:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.108+0000 7f288b7fe640 1 -- 172.21.15.169:0/3362819000 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2890010650 con 0x7f289c101790 2024-03-28T00:57:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.108+0000 7f28a09b2640 1 -- 172.21.15.169:0/3362819000 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f289c068fc0 con 0x7f289c101790 2024-03-28T00:57:57.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.111+0000 7f288b7fe640 1 -- 172.21.15.169:0/3362819000 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f28900037a0 con 0x7f289c101790 2024-03-28T00:57:57.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.111+0000 7f28a35a0740 1 Processor -- start 2024-03-28T00:57:57.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.111+0000 7f28a35a0740 1 -- start start 2024-03-28T00:57:57.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.111+0000 7f28a35a0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73dd55b0 0x557c73dd59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.111+0000 7f28a35a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73dd5fa0 0x557c73ddaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.111+0000 7f28a35a0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73ddb110 0x557c73ddd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.111+0000 7f28a35a0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c73d323d0 con 0x557c73ddb110 2024-03-28T00:57:57.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.111+0000 7f28a35a0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c73cdaad0 con 0x557c73dd55b0 2024-03-28T00:57:57.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.112+0000 7f28a35a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c73d2e470 con 0x557c73dd5fa0 2024-03-28T00:57:57.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.112+0000 7f289a59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73dd5fa0 0x557c73ddaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.112+0000 7f288b7fe640 1 --2- 172.21.15.169:0/3362819000 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f28840a2890 0x7f28840a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.112+0000 7f289a59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73dd5fa0 0x557c73ddaaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:58822/0 (socket says 172.21.15.169:58822) 2024-03-28T00:57:57.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.112+0000 7f289a59c640 1 -- 172.21.15.169:0/3770205467 learned_addr learned my addr 172.21.15.169:0/3770205467 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:57.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.112+0000 7f2899d9b640 1 --2- 172.21.15.169:0/3770205467 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73ddb110 0x557c73ddd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.112+0000 7f289ad9d640 1 --2- 172.21.15.169:0/3770205467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73dd55b0 0x557c73dd59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.112+0000 7f289ad9d640 1 --2- 172.21.15.169:0/3362819000 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f28840a2890 0x7f28840a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.112+0000 7f288b7fe640 1 -- 172.21.15.169:0/3362819000 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f28900cf270 con 0x7f289c101790 2024-03-28T00:57:57.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.112+0000 7f289ad9d640 1 -- 172.21.15.169:0/3770205467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73dd5fa0 msgr2=0x557c73ddaaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:57.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.112+0000 7f289ad9d640 1 --2- 172.21.15.169:0/3770205467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73dd5fa0 0x557c73ddaaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.112+0000 7f289ad9d640 1 -- 172.21.15.169:0/3770205467 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73ddb110 msgr2=0x557c73ddd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:57.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.112+0000 7f289ad9d640 1 --2- 172.21.15.169:0/3770205467 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73ddb110 0x557c73ddd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.112+0000 7f289ad9d640 1 -- 172.21.15.169:0/3770205467 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c73b107d0 con 0x557c73dd55b0 2024-03-28T00:57:57.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.112+0000 7f2899d9b640 1 --2- 172.21.15.169:0/3770205467 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73ddb110 0x557c73ddd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:57.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.113+0000 7f289ad9d640 1 --2- 172.21.15.169:0/3770205467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73dd55b0 0x557c73dd59d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2894005fd0 tx=0x7f289400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24c3bc07c9ccd9ad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:57.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.113+0000 7f289ad9d640 1 --2- 172.21.15.169:0/3362819000 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f28840a2890 0x7f28840a4d50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f289401d000 tx=0x7f289401f000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:57.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.113+0000 7f2888ff9640 1 -- 172.21.15.169:0/3770205467 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2894013130 con 0x557c73dd55b0 2024-03-28T00:57:57.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.113+0000 7f2888ff9640 1 -- 172.21.15.169:0/3770205467 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2894022070 con 0x557c73dd55b0 2024-03-28T00:57:57.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.113+0000 7f2888ff9640 1 -- 172.21.15.169:0/3770205467 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289401e400 con 0x557c73dd55b0 2024-03-28T00:57:57.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.114+0000 7f28a35a0740 1 -- 172.21.15.169:0/3770205467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73dd55b0 msgr2=0x557c73dd59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:57.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.114+0000 7f28a35a0740 1 --2- 172.21.15.169:0/3770205467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73dd55b0 0x557c73dd59d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2894005fd0 tx=0x7f289400f790 comp rx=0 tx=0).stop 2024-03-28T00:57:57.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.114+0000 7f28a35a0740 1 -- 172.21.15.169:0/3770205467 shutdown_connections 2024-03-28T00:57:57.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.114+0000 7f28a35a0740 1 --2- 172.21.15.169:0/3770205467 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73ddb110 0x557c73ddd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.114+0000 7f28a35a0740 1 --2- 172.21.15.169:0/3770205467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73dd5fa0 0x557c73ddaaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.114+0000 7f28a35a0740 1 --2- 172.21.15.169:0/3770205467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73dd55b0 0x557c73dd59d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.114+0000 7f28a35a0740 1 -- 172.21.15.169:0/3770205467 >> 172.21.15.169:0/3770205467 conn(0x557c73dd0b20 msgr2=0x557c73dd0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:57.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.114+0000 7f28a35a0740 1 -- 172.21.15.169:0/3770205467 shutdown_connections 2024-03-28T00:57:57.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.114+0000 7f28a35a0740 1 -- 172.21.15.169:0/3770205467 wait complete. 2024-03-28T00:57:57.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.114+0000 7f28a35a0740 1 -- start start 2024-03-28T00:57:57.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.114+0000 7f28a35a0740 1 Processor -- start 2024-03-28T00:57:57.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.114+0000 7f28a35a0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73dd55b0 0x557c73ddfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.114+0000 7f28a35a0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73dd5fa0 0x557c73ddff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.114+0000 7f28a35a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73ddb110 0x557c73de6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.114+0000 7f28a35a0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c73dde060 con 0x557c73dd55b0 2024-03-28T00:57:57.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.114+0000 7f289a59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73dd55b0 0x557c73ddfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.115+0000 7f289a59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73dd55b0 0x557c73ddfa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39354/0 (socket says 172.21.15.169:39354) 2024-03-28T00:57:57.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.115+0000 7f289a59c640 1 -- 172.21.15.169:0/2887879017 learned_addr learned my addr 172.21.15.169:0/2887879017 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:57.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.115+0000 7f2899d9b640 1 --2- 172.21.15.169:0/2887879017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73dd5fa0 0x557c73ddff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.115+0000 7f28a35a0740 1 -- 172.21.15.169:0/2887879017 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c73cee060 con 0x557c73dd5fa0 2024-03-28T00:57:57.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.115+0000 7f28a35a0740 1 -- 172.21.15.169:0/2887879017 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c73dde1e0 con 0x557c73ddb110 2024-03-28T00:57:57.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.115+0000 7f28a35a0740 10 client.0 _ll_register_callbacks cb 0x557c73cd06a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:57.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.115+0000 7f289ad9d640 1 --2- 172.21.15.169:0/2887879017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73ddb110 0x557c73de6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.115+0000 7f2899d9b640 1 -- 172.21.15.169:0/2887879017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73ddb110 msgr2=0x557c73de6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:57.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.115+0000 7f2899d9b640 1 --2- 172.21.15.169:0/2887879017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73ddb110 0x557c73de6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.115+0000 7f2899d9b640 1 -- 172.21.15.169:0/2887879017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73dd55b0 msgr2=0x557c73ddfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:57.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.115+0000 7f2899d9b640 1 --2- 172.21.15.169:0/2887879017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73dd55b0 0x557c73ddfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.115+0000 7f2899d9b640 1 -- 172.21.15.169:0/2887879017 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c73a463f0 con 0x557c73dd5fa0 2024-03-28T00:57:57.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.115+0000 7f289a59c640 1 --2- 172.21.15.169:0/2887879017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73dd55b0 0x557c73ddfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:57.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.115+0000 7f2899d9b640 1 --2- 172.21.15.169:0/2887879017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73dd5fa0 0x557c73ddff60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f28900cc880 tx=0x7f289009b160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64cf870f561b66fc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:57.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.115+0000 7f2881ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:57:57.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.115+0000 7f2881ffb640 1 -- 172.21.15.169:0/2887879017 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28900a3eb0 con 0x557c73dd5fa0 2024-03-28T00:57:57.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.115+0000 7f2881ffb640 1 -- 172.21.15.169:0/2887879017 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2890003b10 con 0x557c73dd5fa0 2024-03-28T00:57:57.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.115+0000 7f28a35a0740 1 -- 172.21.15.169:0/2887879017 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2894022070 con 0x557c73dd5fa0 2024-03-28T00:57:57.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.115+0000 7f2881ffb640 1 -- 172.21.15.169:0/2887879017 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28900a3a20 con 0x557c73dd5fa0 2024-03-28T00:57:57.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.115+0000 7f28a35a0740 10 client.25730 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:57.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.115+0000 7f28a35a0740 1 -- 172.21.15.169:0/2887879017 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557c73a060f0 con 0x557c73dd5fa0 2024-03-28T00:57:57.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.116+0000 7f2881ffb640 1 -- 172.21.15.169:0/2887879017 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2894022070 con 0x557c73dd5fa0 2024-03-28T00:57:57.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.116+0000 7f2881ffb640 1 -- 172.21.15.169:0/2887879017 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2890097460 con 0x557c73dd5fa0 2024-03-28T00:57:57.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.116+0000 7f28a35a0740 10 client.25730 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:57.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.116+0000 7f28a35a0740 10 client.25730 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:57.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.116+0000 7f28a35a0740 20 client.25730 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:57.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.116+0000 7f28a35a0740 1 --2- 172.21.15.169:0/2887879017 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557c73deb380 0x557c73ded7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.116+0000 7f28a35a0740 4 client.25730 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:57:57.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.116+0000 7f28a35a0740 1 -- 172.21.15.169:0/2887879017 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557c73d3f910 con 0x557c73deb380 2024-03-28T00:57:57.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.116+0000 7f289a59c640 1 --2- 172.21.15.169:0/2887879017 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557c73deb380 0x557c73ded7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.116+0000 7f289a59c640 1 --2- 172.21.15.169:0/2887879017 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557c73deb380 0x557c73ded7f0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33a7d0cdce5d013 server_cookie=8025e6486fd006b in_seq=0 out_seq=0 2024-03-28T00:57:57.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.116+0000 7f2881ffb640 10 client.25730 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:57.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.117+0000 7f2881ffb640 1 -- 172.21.15.169:0/2887879017 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557c73d3f910 con 0x557c73deb380 2024-03-28T00:57:57.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.117+0000 7f2881ffb640 10 client.25730 handle_command_reply: tid=0 2024-03-28T00:57:57.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.117+0000 7f28a35a0740 1 client.25730 shutdown 2024-03-28T00:57:57.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.117+0000 7f28a35a0740 1 -- 172.21.15.169:0/2887879017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73dd5fa0 msgr2=0x557c73ddff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:57.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.117+0000 7f28a35a0740 1 --2- 172.21.15.169:0/2887879017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73dd5fa0 0x557c73ddff60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f28900cc880 tx=0x7f289009b160 comp rx=0 tx=0).stop 2024-03-28T00:57:57.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.117+0000 7f28a35a0740 1 -- 172.21.15.169:0/2887879017 shutdown_connections 2024-03-28T00:57:57.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.117+0000 7f28a35a0740 1 --2- 172.21.15.169:0/2887879017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73ddb110 0x557c73de6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.117+0000 7f28a35a0740 1 --2- 172.21.15.169:0/2887879017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73dd5fa0 0x557c73ddff60 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.117+0000 7f28a35a0740 1 --2- 172.21.15.169:0/2887879017 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557c73deb380 0x557c73ded7f0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.117+0000 7f28a35a0740 1 --2- 172.21.15.169:0/2887879017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73dd55b0 0x557c73ddfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.117+0000 7f2881ffb640 0 client.25730 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:57.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.117+0000 7f28a35a0740 1 -- 172.21.15.169:0/2887879017 >> 172.21.15.169:0/2887879017 conn(0x557c73dd0b20 msgr2=0x557c73dc7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:57.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.117+0000 7f28a35a0740 1 -- 172.21.15.169:0/2887879017 shutdown_connections 2024-03-28T00:57:57.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.117+0000 7f28a35a0740 1 -- 172.21.15.169:0/2887879017 wait complete. 2024-03-28T00:57:57.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.117+0000 7f28a35a0740 20 client.25730 trim_cache size 0 max 16384 2024-03-28T00:57:57.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.133+0000 7f28a35a0740 1 Processor -- start 2024-03-28T00:57:57.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.133+0000 7f28a35a0740 1 -- start start 2024-03-28T00:57:57.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.133+0000 7f28a35a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73dd55b0 0x557c73ddfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.133+0000 7f28a35a0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73dd5fa0 0x557c73de6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.133+0000 7f28a35a0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73ddb110 0x557c73ddff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.133+0000 7f28a35a0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289c1aa500 con 0x557c73dd5fa0 2024-03-28T00:57:57.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.133+0000 7f28a35a0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289c1a8000 con 0x557c73ddb110 2024-03-28T00:57:57.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.133+0000 7f28a35a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289c1a60a0 con 0x557c73dd55b0 2024-03-28T00:57:57.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.134+0000 7f289a59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73dd55b0 0x557c73ddfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.134+0000 7f289ad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73ddb110 0x557c73ddff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.134+0000 7f2899d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73dd5fa0 0x557c73de6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.134+0000 7f289ad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73ddb110 0x557c73ddff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:59130/0 (socket says 172.21.15.169:59130) 2024-03-28T00:57:57.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.134+0000 7f289a59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73dd55b0 0x557c73ddfa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:58834/0 (socket says 172.21.15.169:58834) 2024-03-28T00:57:57.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.134+0000 7f289ad9d640 1 -- 172.21.15.169:0/1742304137 learned_addr learned my addr 172.21.15.169:0/1742304137 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:57.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.134+0000 7f289a59c640 1 -- 172.21.15.169:0/1742304137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73ddb110 msgr2=0x557c73ddff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:57.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.134+0000 7f289a59c640 1 --2- 172.21.15.169:0/1742304137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73ddb110 0x557c73ddff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.134+0000 7f289a59c640 1 -- 172.21.15.169:0/1742304137 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73dd5fa0 msgr2=0x557c73de6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:57.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.134+0000 7f289a59c640 1 --2- 172.21.15.169:0/1742304137 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73dd5fa0 0x557c73de6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.134+0000 7f289a59c640 1 -- 172.21.15.169:0/1742304137 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c73dc2910 con 0x557c73dd55b0 2024-03-28T00:57:57.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.134+0000 7f289ad9d640 1 --2- 172.21.15.169:0/1742304137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73ddb110 0x557c73ddff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:57.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.134+0000 7f2899d9b640 1 --2- 172.21.15.169:0/1742304137 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73dd5fa0 0x557c73de6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:57.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.134+0000 7f289a59c640 1 --2- 172.21.15.169:0/1742304137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73dd55b0 0x557c73ddfa20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f288c037040 tx=0x7f288c02e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f31bbc217764661 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:57.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.134+0000 7f2883fff640 1 -- 172.21.15.169:0/1742304137 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288c039040 con 0x557c73dd55b0 2024-03-28T00:57:57.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.134+0000 7f2883fff640 1 -- 172.21.15.169:0/1742304137 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f288c02c510 con 0x557c73dd55b0 2024-03-28T00:57:57.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.134+0000 7f2883fff640 1 -- 172.21.15.169:0/1742304137 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288c032880 con 0x557c73dd55b0 2024-03-28T00:57:57.142 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.135+0000 7f28a35a0740 1 -- 172.21.15.169:0/1742304137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73dd55b0 msgr2=0x557c73ddfa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:57.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.135+0000 7f28a35a0740 1 --2- 172.21.15.169:0/1742304137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73dd55b0 0x557c73ddfa20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f288c037040 tx=0x7f288c02e040 comp rx=0 tx=0).stop 2024-03-28T00:57:57.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.135+0000 7f28a35a0740 1 -- 172.21.15.169:0/1742304137 shutdown_connections 2024-03-28T00:57:57.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.135+0000 7f28a35a0740 1 --2- 172.21.15.169:0/1742304137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73ddb110 0x557c73ddff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.135+0000 7f28a35a0740 1 --2- 172.21.15.169:0/1742304137 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73dd5fa0 0x557c73de6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.135+0000 7f28a35a0740 1 --2- 172.21.15.169:0/1742304137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73dd55b0 0x557c73ddfa20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.135+0000 7f28a35a0740 1 -- 172.21.15.169:0/1742304137 >> 172.21.15.169:0/1742304137 conn(0x557c73dd0b20 msgr2=0x557c73dd7500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:57.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.135+0000 7f28a35a0740 1 -- 172.21.15.169:0/1742304137 shutdown_connections 2024-03-28T00:57:57.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.135+0000 7f28a35a0740 1 -- 172.21.15.169:0/1742304137 wait complete. 2024-03-28T00:57:57.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.135+0000 7f28a35a0740 1 -- start start 2024-03-28T00:57:57.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.135+0000 7f28a35a0740 1 Processor -- start 2024-03-28T00:57:57.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.136+0000 7f28a35a0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73dd55b0 0x557c73de45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.136+0000 7f28a35a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73dd5fa0 0x557c73de4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.136+0000 7f28a35a0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73ddb110 0x557c73de5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.136+0000 7f28a35a0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289c1a66a0 con 0x557c73dd55b0 2024-03-28T00:57:57.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.136+0000 7f289a59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73dd55b0 0x557c73de45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.136+0000 7f28a35a0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289c1a79a0 con 0x557c73ddb110 2024-03-28T00:57:57.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.136+0000 7f28a35a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289c1a6ce0 con 0x557c73dd5fa0 2024-03-28T00:57:57.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.136+0000 7f2899d9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73dd5fa0 0x557c73de4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.136+0000 7f2899d9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73dd5fa0 0x557c73de4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:58846/0 (socket says 172.21.15.169:58846) 2024-03-28T00:57:57.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.136+0000 7f2899d9b640 1 -- 172.21.15.169:0/523064943 learned_addr learned my addr 172.21.15.169:0/523064943 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:57.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.136+0000 7f28a35a0740 10 client.0 _ll_register_callbacks cb 0x7f2890003e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:57.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.136+0000 7f289ad9d640 1 --2- 172.21.15.169:0/523064943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73ddb110 0x557c73de5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.136+0000 7f2899d9b640 1 -- 172.21.15.169:0/523064943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73ddb110 msgr2=0x557c73de5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:57.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.136+0000 7f2899d9b640 1 --2- 172.21.15.169:0/523064943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73ddb110 0x557c73de5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.136+0000 7f2899d9b640 1 -- 172.21.15.169:0/523064943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73dd55b0 msgr2=0x557c73de45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:57.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.136+0000 7f2899d9b640 1 --2- 172.21.15.169:0/523064943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73dd55b0 0x557c73de45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.136+0000 7f2899d9b640 1 -- 172.21.15.169:0/523064943 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c73de6110 con 0x557c73dd5fa0 2024-03-28T00:57:57.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.136+0000 7f2899d9b640 1 --2- 172.21.15.169:0/523064943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73dd5fa0 0x557c73de4b10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f289009b410 tx=0x7f2890097870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d1dc1cfa4c7d5a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:57.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.136+0000 7f2881ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:57:57.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.137+0000 7f2881ffb640 1 -- 172.21.15.169:0/523064943 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28900a3eb0 con 0x557c73dd5fa0 2024-03-28T00:57:57.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.137+0000 7f2881ffb640 1 -- 172.21.15.169:0/523064943 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f28900afaf0 con 0x557c73dd5fa0 2024-03-28T00:57:57.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.137+0000 7f28a35a0740 1 -- 172.21.15.169:0/523064943 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f288c02c510 con 0x557c73dd5fa0 2024-03-28T00:57:57.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.137+0000 7f2881ffb640 1 -- 172.21.15.169:0/523064943 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2890083050 con 0x557c73dd5fa0 2024-03-28T00:57:57.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.137+0000 7f28a35a0740 10 client.25714 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:57.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.137+0000 7f28a35a0740 1 -- 172.21.15.169:0/523064943 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557c73e46c80 con 0x557c73dd5fa0 2024-03-28T00:57:57.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.137+0000 7f2881ffb640 1 -- 172.21.15.169:0/523064943 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f288c02c510 con 0x557c73dd5fa0 2024-03-28T00:57:57.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.137+0000 7f2881ffb640 1 -- 172.21.15.169:0/523064943 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f28900a0980 con 0x557c73dd5fa0 2024-03-28T00:57:57.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.137+0000 7f28a35a0740 10 client.25714 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:57.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.137+0000 7f28a35a0740 10 client.25714 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:57.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.137+0000 7f28a35a0740 20 client.25714 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:57.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.137+0000 7f28a35a0740 1 --2- 172.21.15.169:0/523064943 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557c73deb380 0x557c73e49640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.137+0000 7f28a35a0740 4 client.25714 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T00:57:57.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.137+0000 7f28a35a0740 1 -- 172.21.15.169:0/523064943 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557c73d39c60 con 0x557c73deb380 2024-03-28T00:57:57.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.137+0000 7f289a59c640 1 --2- 172.21.15.169:0/523064943 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557c73deb380 0x557c73e49640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.138+0000 7f289a59c640 1 --2- 172.21.15.169:0/523064943 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557c73deb380 0x557c73e49640 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8351740aa426a0e server_cookie=41ae249a1912ac7c in_seq=0 out_seq=0 2024-03-28T00:57:57.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.138+0000 7f2881ffb640 10 client.25714 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:57.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.138+0000 7f2881ffb640 1 -- 172.21.15.169:0/523064943 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x557c73d39c60 con 0x557c73deb380 2024-03-28T00:57:57.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.138+0000 7f2881ffb640 10 client.25714 handle_command_reply: tid=0 2024-03-28T00:57:57.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.138+0000 7f28a35a0740 1 client.25714 shutdown 2024-03-28T00:57:57.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.138+0000 7f28a35a0740 1 -- 172.21.15.169:0/523064943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73dd5fa0 msgr2=0x557c73de4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:57.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.138+0000 7f28a35a0740 1 --2- 172.21.15.169:0/523064943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73dd5fa0 0x557c73de4b10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f289009b410 tx=0x7f2890097870 comp rx=0 tx=0).stop 2024-03-28T00:57:57.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.138+0000 7f28a35a0740 1 -- 172.21.15.169:0/523064943 shutdown_connections 2024-03-28T00:57:57.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.138+0000 7f28a35a0740 1 --2- 172.21.15.169:0/523064943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c73ddb110 0x557c73de5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.138+0000 7f28a35a0740 1 --2- 172.21.15.169:0/523064943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c73dd5fa0 0x557c73de4b10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.138+0000 7f28a35a0740 1 --2- 172.21.15.169:0/523064943 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557c73deb380 0x557c73e49640 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.138+0000 7f28a35a0740 1 --2- 172.21.15.169:0/523064943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557c73dd55b0 0x557c73de45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.138+0000 7f2881ffb640 0 client.25714 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:57.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.138+0000 7f28a35a0740 1 -- 172.21.15.169:0/523064943 >> 172.21.15.169:0/523064943 conn(0x557c73dd0b20 msgr2=0x557c73ddf960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:57.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.138+0000 7f28a35a0740 1 -- 172.21.15.169:0/523064943 shutdown_connections 2024-03-28T00:57:57.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.138+0000 7f28a35a0740 1 -- 172.21.15.169:0/523064943 wait complete. 2024-03-28T00:57:57.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.138+0000 7f28a35a0740 20 client.25714 trim_cache size 0 max 16384 2024-03-28T00:57:57.147 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:57:57.147 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T00:57:57.147 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": { 2024-03-28T00:57:57.147 INFO:teuthology.orchestra.run.smithi169.stdout: "83cb7ad5-2b46-42e7-91bd-2a9351262c8b": { 2024-03-28T00:57:57.147 INFO:teuthology.orchestra.run.smithi169.stdout: "path": "/", 2024-03-28T00:57:57.147 INFO:teuthology.orchestra.run.smithi169.stdout: "tag": "83cb7ad5-2b46-42e7-91bd-2a9351262c8b", 2024-03-28T00:57:57.147 INFO:teuthology.orchestra.run.smithi169.stdout: "options": "recursive,force" 2024-03-28T00:57:57.147 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T00:57:57.147 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T00:57:57.148 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:57:57.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.139+0000 7f28a09b2640 1 -- 172.21.15.169:0/3362819000 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f28840a2890 msgr2=0x7f28840a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:57.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.139+0000 7f28a09b2640 1 --2- 172.21.15.169:0/3362819000 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f28840a2890 0x7f28840a4d50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f289401d000 tx=0x7f289401f000 comp rx=0 tx=0).stop 2024-03-28T00:57:57.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.139+0000 7f28a09b2640 1 -- 172.21.15.169:0/3362819000 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f289c101790 msgr2=0x7f289c196e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:57.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.139+0000 7f28a09b2640 1 --2- 172.21.15.169:0/3362819000 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f289c101790 0x7f289c196e00 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f289000e9c0 tx=0x7f289000ee80 comp rx=0 tx=0).stop 2024-03-28T00:57:57.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.139+0000 7f28a09b2640 1 -- 172.21.15.169:0/3362819000 shutdown_connections 2024-03-28T00:57:57.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.139+0000 7f28a09b2640 1 --2- 172.21.15.169:0/3362819000 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f28840a2890 0x7f28840a4d50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.139+0000 7f28a09b2640 1 --2- 172.21.15.169:0/3362819000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f289c102150 0x7f289c19de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.139+0000 7f28a09b2640 1 --2- 172.21.15.169:0/3362819000 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f289c101790 0x7f289c196e00 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.139+0000 7f28a09b2640 1 --2- 172.21.15.169:0/3362819000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f289c1005a0 0x7f289c1968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.139+0000 7f28a09b2640 1 -- 172.21.15.169:0/3362819000 >> 172.21.15.169:0/3362819000 conn(0x7f289c0fbb30 msgr2=0x7f289c104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:57.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.139+0000 7f28a09b2640 1 -- 172.21.15.169:0/3362819000 shutdown_connections 2024-03-28T00:57:57.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:57:57.139+0000 7f28a09b2640 1 -- 172.21.15.169:0/3362819000 wait complete. 2024-03-28T00:57:57.153 INFO:tasks.cephfs.filesystem:scrub status for tag:83cb7ad5-2b46-42e7-91bd-2a9351262c8b - {'path': '/', 'tag': '83cb7ad5-2b46-42e7-91bd-2a9351262c8b', 'options': 'recursive,force'} 2024-03-28T00:57:57.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:57 smithi169 ceph-mon[32274]: pgmap v394: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T00:57:57.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:57 smithi174 ceph-mon[36060]: pgmap v394: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T00:57:57.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:57 smithi177 ceph-mon[36206]: pgmap v394: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T00:57:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.932+0000 7fe7678f1640 1 -- 172.21.15.169:0/1745965973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7601008f0 msgr2=0x7fe760100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.932+0000 7fe7678f1640 1 --2- 172.21.15.169:0/1745965973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7601008f0 0x7fe760100d50 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe750009900 tx=0x7fe75002bc30 comp rx=0 tx=0).stop 2024-03-28T00:57:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.932+0000 7fe7678f1640 1 -- 172.21.15.169:0/1745965973 shutdown_connections 2024-03-28T00:57:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.932+0000 7fe7678f1640 1 --2- 172.21.15.169:0/1745965973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe760101290 0x7fe760101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.932+0000 7fe7678f1640 1 --2- 172.21.15.169:0/1745965973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7601008f0 0x7fe760100d50 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.932+0000 7fe7678f1640 1 --2- 172.21.15.169:0/1745965973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7600fff90 0x7fe7601003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.932+0000 7fe7678f1640 1 -- 172.21.15.169:0/1745965973 >> 172.21.15.169:0/1745965973 conn(0x7fe7600fbb30 msgr2=0x7fe7600fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.933+0000 7fe7678f1640 1 -- 172.21.15.169:0/1745965973 shutdown_connections 2024-03-28T00:57:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.933+0000 7fe7678f1640 1 -- 172.21.15.169:0/1745965973 wait complete. 2024-03-28T00:57:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.933+0000 7fe7678f1640 1 Processor -- start 2024-03-28T00:57:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.933+0000 7fe7678f1640 1 -- start start 2024-03-28T00:57:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.933+0000 7fe7678f1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7600fff90 0x7fe760199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.933+0000 7fe7678f1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7601008f0 0x7fe76019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.933+0000 7fe7678f1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe760101290 0x7fe7601a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.933+0000 7fe7678f1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76010f550 con 0x7fe7600fff90 2024-03-28T00:57:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.933+0000 7fe7678f1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76010f3d0 con 0x7fe760101290 2024-03-28T00:57:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.933+0000 7fe7678f1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76010f6d0 con 0x7fe7601008f0 2024-03-28T00:57:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.933+0000 7fe76568d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7600fff90 0x7fe760199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.934+0000 7fe764e8c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7601008f0 0x7fe76019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.934+0000 7fe764e8c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7601008f0 0x7fe76019a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:58860/0 (socket says 172.21.15.169:58860) 2024-03-28T00:57:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.934+0000 7fe764e8c640 1 -- 172.21.15.169:0/2702760953 learned_addr learned my addr 172.21.15.169:0/2702760953 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.934+0000 7fe76568d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7600fff90 0x7fe760199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39388/0 (socket says 172.21.15.169:39388) 2024-03-28T00:57:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.934+0000 7fe765e8e640 1 --2- 172.21.15.169:0/2702760953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe760101290 0x7fe7601a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.934+0000 7fe764e8c640 1 -- 172.21.15.169:0/2702760953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe760101290 msgr2=0x7fe7601a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.934+0000 7fe764e8c640 1 --2- 172.21.15.169:0/2702760953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe760101290 0x7fe7601a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.934+0000 7fe764e8c640 1 -- 172.21.15.169:0/2702760953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7600fff90 msgr2=0x7fe760199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.934+0000 7fe764e8c640 1 --2- 172.21.15.169:0/2702760953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7600fff90 0x7fe760199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.934+0000 7fe764e8c640 1 -- 172.21.15.169:0/2702760953 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7601a1790 con 0x7fe7601008f0 2024-03-28T00:57:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.934+0000 7fe76568d640 1 --2- 172.21.15.169:0/2702760953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7600fff90 0x7fe760199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.934+0000 7fe764e8c640 1 --2- 172.21.15.169:0/2702760953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7601008f0 0x7fe76019a100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe750000c00 tx=0x7fe75002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.934+0000 7fe7567fc640 1 -- 172.21.15.169:0/2702760953 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe750043020 con 0x7fe7601008f0 2024-03-28T00:57:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.934+0000 7fe7678f1640 1 -- 172.21.15.169:0/2702760953 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7601a1a20 con 0x7fe7601008f0 2024-03-28T00:57:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.934+0000 7fe7567fc640 1 -- 172.21.15.169:0/2702760953 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe7500054e0 con 0x7fe7601008f0 2024-03-28T00:57:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.934+0000 7fe7567fc640 1 -- 172.21.15.169:0/2702760953 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe750003bd0 con 0x7fe7601008f0 2024-03-28T00:57:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.934+0000 7fe7678f1640 1 -- 172.21.15.169:0/2702760953 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7601a1f70 con 0x7fe7601008f0 2024-03-28T00:57:57.942 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.937+0000 7fe76a5b0740 1 Processor -- start 2024-03-28T00:57:57.942 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.937+0000 7fe76a5b0740 1 -- start start 2024-03-28T00:57:57.942 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.937+0000 7fe76a5b0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa1a920 0x5556caa1ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.942 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.937+0000 7fe76a5b0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa1b310 0x5556caa1fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.942 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.937+0000 7fe76a5b0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556caa20480 0x5556caa22910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.942 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.937+0000 7fe76a5b0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556ca97fb10 con 0x5556caa20480 2024-03-28T00:57:57.942 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.937+0000 7fe76a5b0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556ca928900 con 0x5556caa1b310 2024-03-28T00:57:57.942 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.937+0000 7fe76a5b0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556ca97bbb0 con 0x5556caa1a920 2024-03-28T00:57:57.943 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.938+0000 7fe765e8e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa1a920 0x5556caa1ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.943 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.938+0000 7fe765e8e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa1a920 0x5556caa1ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:58864/0 (socket says 172.21.15.169:58864) 2024-03-28T00:57:57.943 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.938+0000 7fe76568d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa1b310 0x5556caa1fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.943 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.938+0000 7fe765e8e640 1 -- 172.21.15.169:0/1467287344 learned_addr learned my addr 172.21.15.169:0/1467287344 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:57.943 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.938+0000 7fe7567fc640 1 -- 172.21.15.169:0/2702760953 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fe750046020 con 0x7fe7601008f0 2024-03-28T00:57:57.943 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.938+0000 7fe765e8e640 1 -- 172.21.15.169:0/1467287344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa1b310 msgr2=0x5556caa1fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:57.943 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.938+0000 7fe765e8e640 1 --2- 172.21.15.169:0/1467287344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa1b310 0x5556caa1fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.943 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.938+0000 7fe765e8e640 1 -- 172.21.15.169:0/1467287344 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556caa20480 msgr2=0x5556caa22910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T00:57:57.943 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.938+0000 7fe765e8e640 1 --2- 172.21.15.169:0/1467287344 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556caa20480 0x5556caa22910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.943 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.938+0000 7fe765e8e640 1 -- 172.21.15.169:0/1467287344 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556ca694210 con 0x5556caa1a920 2024-03-28T00:57:57.943 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.938+0000 7fe765e8e640 1 --2- 172.21.15.169:0/1467287344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa1a920 0x5556caa1ad40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe758011440 tx=0x7fe758011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9405fcc45a1601ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:57.943 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.938+0000 7fe74b7fe640 1 -- 172.21.15.169:0/1467287344 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe758003c30 con 0x5556caa1a920 2024-03-28T00:57:57.943 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.938+0000 7fe74b7fe640 1 -- 172.21.15.169:0/1467287344 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe758003dd0 con 0x5556caa1a920 2024-03-28T00:57:57.943 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.938+0000 7fe74b7fe640 1 -- 172.21.15.169:0/1467287344 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75800fac0 con 0x5556caa1a920 2024-03-28T00:57:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.938+0000 7fe7567fc640 1 --2- 172.21.15.169:0/2702760953 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe74c0a2940 0x7fe74c0a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.939+0000 7fe7567fc640 1 -- 172.21.15.169:0/2702760953 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fe7500f0ed0 con 0x7fe7601008f0 2024-03-28T00:57:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.939+0000 7fe76568d640 1 --2- 172.21.15.169:0/2702760953 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe74c0a2940 0x7fe74c0a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.939+0000 7fe76a5b0740 1 -- 172.21.15.169:0/1467287344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa1a920 msgr2=0x5556caa1ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.939+0000 7fe76a5b0740 1 --2- 172.21.15.169:0/1467287344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa1a920 0x5556caa1ad40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe758011440 tx=0x7fe758011900 comp rx=0 tx=0).stop 2024-03-28T00:57:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.939+0000 7fe76a5b0740 1 -- 172.21.15.169:0/1467287344 shutdown_connections 2024-03-28T00:57:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.939+0000 7fe76a5b0740 1 --2- 172.21.15.169:0/1467287344 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556caa20480 0x5556caa22910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.939+0000 7fe76a5b0740 1 --2- 172.21.15.169:0/1467287344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa1b310 0x5556caa1fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.939+0000 7fe76a5b0740 1 --2- 172.21.15.169:0/1467287344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa1a920 0x5556caa1ad40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.939+0000 7fe76a5b0740 1 -- 172.21.15.169:0/1467287344 >> 172.21.15.169:0/1467287344 conn(0x5556caa15e90 msgr2=0x5556caa16290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.939+0000 7fe76568d640 1 --2- 172.21.15.169:0/2702760953 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe74c0a2940 0x7fe74c0a4e00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x5556caa1b710 tx=0x7fe75c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.939+0000 7fe76a5b0740 1 -- 172.21.15.169:0/1467287344 shutdown_connections 2024-03-28T00:57:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.939+0000 7fe76a5b0740 1 -- 172.21.15.169:0/1467287344 wait complete. 2024-03-28T00:57:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.940+0000 7fe76a5b0740 1 -- start start 2024-03-28T00:57:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.940+0000 7fe76a5b0740 1 Processor -- start 2024-03-28T00:57:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.940+0000 7fe76a5b0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa1a920 0x5556caa24d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.940+0000 7fe76a5b0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556caa1b310 0x5556caa25240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.940+0000 7fe76a5b0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa20480 0x5556caa2c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.940+0000 7fe76a5b0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556caa23430 con 0x5556caa1b310 2024-03-28T00:57:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.940+0000 7fe76a5b0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556ca93bd50 con 0x5556caa1a920 2024-03-28T00:57:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.940+0000 7fe76a5b0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556caa235b0 con 0x5556caa20480 2024-03-28T00:57:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.940+0000 7fe76a5b0740 10 client.0 _ll_register_callbacks cb 0x5556ca93b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.940+0000 7fe76568d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa1a920 0x5556caa24d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.940+0000 7fe76568d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa1a920 0x5556caa24d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:59178/0 (socket says 172.21.15.169:59178) 2024-03-28T00:57:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.940+0000 7fe76568d640 1 -- 172.21.15.169:0/1631575038 learned_addr learned my addr 172.21.15.169:0/1631575038 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.941+0000 7fe764e8c640 1 --2- 172.21.15.169:0/1631575038 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556caa1b310 0x5556caa25240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.941+0000 7fe765e8e640 1 --2- 172.21.15.169:0/1631575038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa20480 0x5556caa2c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.941+0000 7fe76568d640 1 -- 172.21.15.169:0/1631575038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa20480 msgr2=0x5556caa2c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.941+0000 7fe76568d640 1 --2- 172.21.15.169:0/1631575038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa20480 0x5556caa2c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.941+0000 7fe76568d640 1 -- 172.21.15.169:0/1631575038 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556caa1b310 msgr2=0x5556caa25240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.941+0000 7fe76568d640 1 --2- 172.21.15.169:0/1631575038 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556caa1b310 0x5556caa25240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.941+0000 7fe76568d640 1 -- 172.21.15.169:0/1631575038 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5556ca760620 con 0x5556caa1a920 2024-03-28T00:57:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.941+0000 7fe764e8c640 1 --2- 172.21.15.169:0/1631575038 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556caa1b310 0x5556caa25240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:57:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.941+0000 7fe76568d640 1 --2- 172.21.15.169:0/1631575038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa1a920 0x5556caa24d00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe75c009560 tx=0x7fe75c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96de2e544df10839 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.941+0000 7fe748ff9640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:57:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.941+0000 7fe748ff9640 1 -- 172.21.15.169:0/1631575038 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c004020 con 0x5556caa1a920 2024-03-28T00:57:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.941+0000 7fe76a5b0740 1 -- 172.21.15.169:0/1631575038 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe758003dd0 con 0x5556caa1a920 2024-03-28T00:57:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.941+0000 7fe748ff9640 1 -- 172.21.15.169:0/1631575038 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fe75c018030 con 0x5556caa1a920 2024-03-28T00:57:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.941+0000 7fe748ff9640 1 -- 172.21.15.169:0/1631575038 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe75c034050 con 0x5556caa1a920 2024-03-28T00:57:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.941+0000 7fe76a5b0740 10 client.25748 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.941+0000 7fe76a5b0740 1 -- 172.21.15.169:0/1631575038 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5556caa2f6b0 con 0x5556caa1a920 2024-03-28T00:57:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.942+0000 7fe748ff9640 1 -- 172.21.15.169:0/1631575038 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe758003dd0 con 0x5556caa1a920 2024-03-28T00:57:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.942+0000 7fe748ff9640 1 -- 172.21.15.169:0/1631575038 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fe75c003680 con 0x5556caa1a920 2024-03-28T00:57:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.942+0000 7fe76a5b0740 10 client.25748 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.942+0000 7fe76a5b0740 10 client.25748 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.942+0000 7fe76a5b0740 20 client.25748 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.942+0000 7fe76a5b0740 1 --2- 172.21.15.169:0/1631575038 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5556caa30cd0 0x5556caa330e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.942+0000 7fe76a5b0740 4 client.25748 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:57:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.942+0000 7fe76a5b0740 1 -- 172.21.15.169:0/1631575038 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5556ca98d050 con 0x5556caa30cd0 2024-03-28T00:57:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.942+0000 7fe764e8c640 1 --2- 172.21.15.169:0/1631575038 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5556caa30cd0 0x5556caa330e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.942+0000 7fe748ff9640 10 client.25748 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.942+0000 7fe764e8c640 1 --2- 172.21.15.169:0/1631575038 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5556caa30cd0 0x5556caa330e0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54155c4f3c6e0fd server_cookie=ca5ab35467ff73fb in_seq=0 out_seq=0 2024-03-28T00:57:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.943+0000 7fe748ff9640 1 -- 172.21.15.169:0/1631575038 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5556ca98d050 con 0x5556caa30cd0 2024-03-28T00:57:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.943+0000 7fe748ff9640 10 client.25748 handle_command_reply: tid=0 2024-03-28T00:57:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.943+0000 7fe76a5b0740 1 client.25748 shutdown 2024-03-28T00:57:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.943+0000 7fe76a5b0740 1 -- 172.21.15.169:0/1631575038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa1a920 msgr2=0x5556caa24d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.943+0000 7fe76a5b0740 1 --2- 172.21.15.169:0/1631575038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa1a920 0x5556caa24d00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe75c009560 tx=0x7fe75c004730 comp rx=0 tx=0).stop 2024-03-28T00:57:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.943+0000 7fe76a5b0740 1 -- 172.21.15.169:0/1631575038 shutdown_connections 2024-03-28T00:57:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.943+0000 7fe76a5b0740 1 --2- 172.21.15.169:0/1631575038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa20480 0x5556caa2c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.943+0000 7fe76a5b0740 1 --2- 172.21.15.169:0/1631575038 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5556caa30cd0 0x5556caa330e0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.943+0000 7fe76a5b0740 1 --2- 172.21.15.169:0/1631575038 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556caa1b310 0x5556caa25240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.943+0000 7fe748ff9640 0 client.25748 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.943+0000 7fe76a5b0740 1 --2- 172.21.15.169:0/1631575038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa1a920 0x5556caa24d00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.943+0000 7fe76a5b0740 1 -- 172.21.15.169:0/1631575038 >> 172.21.15.169:0/1631575038 conn(0x5556caa15e90 msgr2=0x5556caa15020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.943+0000 7fe76a5b0740 1 -- 172.21.15.169:0/1631575038 shutdown_connections 2024-03-28T00:57:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.943+0000 7fe76a5b0740 1 -- 172.21.15.169:0/1631575038 wait complete. 2024-03-28T00:57:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.943+0000 7fe76a5b0740 20 client.25748 trim_cache size 0 max 16384 2024-03-28T00:57:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.957+0000 7fe76a5b0740 1 Processor -- start 2024-03-28T00:57:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.957+0000 7fe76a5b0740 1 -- start start 2024-03-28T00:57:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.957+0000 7fe76a5b0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa1a920 0x5556caa330c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.957+0000 7fe76a5b0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556caa1b310 0x5556caa2c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.957+0000 7fe76a5b0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa20480 0x5556caa1fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.957+0000 7fe76a5b0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7601ae8d0 con 0x5556caa1b310 2024-03-28T00:57:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.957+0000 7fe76a5b0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7601ac3d0 con 0x5556caa20480 2024-03-28T00:57:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.957+0000 7fe76a5b0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7601aa470 con 0x5556caa1a920 2024-03-28T00:57:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.958+0000 7fe76568d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa1a920 0x5556caa330c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.958+0000 7fe76568d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa1a920 0x5556caa330c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:58878/0 (socket says 172.21.15.169:58878) 2024-03-28T00:57:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.958+0000 7fe76568d640 1 -- 172.21.15.169:0/3784194618 learned_addr learned my addr 172.21.15.169:0/3784194618 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.958+0000 7fe764e8c640 1 --2- 172.21.15.169:0/3784194618 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556caa1b310 0x5556caa2c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.958+0000 7fe765e8e640 1 --2- 172.21.15.169:0/3784194618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa20480 0x5556caa1fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.958+0000 7fe76568d640 1 -- 172.21.15.169:0/3784194618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa20480 msgr2=0x5556caa1fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.958+0000 7fe76568d640 1 --2- 172.21.15.169:0/3784194618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa20480 0x5556caa1fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.958+0000 7fe76568d640 1 -- 172.21.15.169:0/3784194618 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556caa1b310 msgr2=0x5556caa2c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.958+0000 7fe76568d640 1 --2- 172.21.15.169:0/3784194618 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556caa1b310 0x5556caa2c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.958+0000 7fe76568d640 1 -- 172.21.15.169:0/3784194618 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556caa11520 con 0x5556caa1a920 2024-03-28T00:57:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.958+0000 7fe764e8c640 1 --2- 172.21.15.169:0/3784194618 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556caa1b310 0x5556caa2c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:57:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.958+0000 7fe76568d640 1 --2- 172.21.15.169:0/3784194618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa1a920 0x5556caa330c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe75c009730 tx=0x7fe75c003930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b364fb8c87f2474 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.959+0000 7fe74affd640 1 -- 172.21.15.169:0/3784194618 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c03d070 con 0x5556caa1a920 2024-03-28T00:57:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.959+0000 7fe74affd640 1 -- 172.21.15.169:0/3784194618 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe75c01e040 con 0x5556caa1a920 2024-03-28T00:57:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.959+0000 7fe74affd640 1 -- 172.21.15.169:0/3784194618 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c01e650 con 0x5556caa1a920 2024-03-28T00:57:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.959+0000 7fe76a5b0740 1 -- 172.21.15.169:0/3784194618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa1a920 msgr2=0x5556caa330c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.959+0000 7fe76a5b0740 1 --2- 172.21.15.169:0/3784194618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa1a920 0x5556caa330c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe75c009730 tx=0x7fe75c003930 comp rx=0 tx=0).stop 2024-03-28T00:57:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.959+0000 7fe76a5b0740 1 -- 172.21.15.169:0/3784194618 shutdown_connections 2024-03-28T00:57:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.959+0000 7fe76a5b0740 1 --2- 172.21.15.169:0/3784194618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa20480 0x5556caa1fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.959+0000 7fe76a5b0740 1 --2- 172.21.15.169:0/3784194618 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556caa1b310 0x5556caa2c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.959+0000 7fe76a5b0740 1 --2- 172.21.15.169:0/3784194618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa1a920 0x5556caa330c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.959+0000 7fe76a5b0740 1 -- 172.21.15.169:0/3784194618 >> 172.21.15.169:0/3784194618 conn(0x5556caa15e90 msgr2=0x5556caa16270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.959+0000 7fe76a5b0740 1 -- 172.21.15.169:0/3784194618 shutdown_connections 2024-03-28T00:57:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.959+0000 7fe76a5b0740 1 -- 172.21.15.169:0/3784194618 wait complete. 2024-03-28T00:57:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.960+0000 7fe76a5b0740 1 -- start start 2024-03-28T00:57:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.960+0000 7fe76a5b0740 1 Processor -- start 2024-03-28T00:57:57.970 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.960+0000 7fe76a5b0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa1a920 0x5556caa29a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.970 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.960+0000 7fe76a5b0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556caa1b310 0x5556caa29f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.970 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.960+0000 7fe76a5b0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa20480 0x5556caa2af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.970 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.960+0000 7fe76a5b0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7601aaa70 con 0x5556caa1b310 2024-03-28T00:57:57.970 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.960+0000 7fe76a5b0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7601abd70 con 0x5556caa20480 2024-03-28T00:57:57.971 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.960+0000 7fe76a5b0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7601ab0b0 con 0x5556caa1a920 2024-03-28T00:57:57.971 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.960+0000 7fe76a5b0740 10 client.0 _ll_register_callbacks cb 0x7fe75c00f9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:57:57.971 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.960+0000 7fe76568d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa1a920 0x5556caa29a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.971 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.960+0000 7fe76568d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa1a920 0x5556caa29a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:58880/0 (socket says 172.21.15.169:58880) 2024-03-28T00:57:57.971 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.960+0000 7fe76568d640 1 -- 172.21.15.169:0/678142520 learned_addr learned my addr 172.21.15.169:0/678142520 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:57:57.971 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.960+0000 7fe764e8c640 1 --2- 172.21.15.169:0/678142520 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556caa1b310 0x5556caa29f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.971 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.960+0000 7fe765e8e640 1 --2- 172.21.15.169:0/678142520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa20480 0x5556caa2af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.971 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.961+0000 7fe76568d640 1 -- 172.21.15.169:0/678142520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa20480 msgr2=0x5556caa2af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:57.971 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.961+0000 7fe76568d640 1 --2- 172.21.15.169:0/678142520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa20480 0x5556caa2af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.971 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.961+0000 7fe76568d640 1 -- 172.21.15.169:0/678142520 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556caa1b310 msgr2=0x5556caa29f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:57:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.961+0000 7fe76568d640 1 --2- 172.21.15.169:0/678142520 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556caa1b310 0x5556caa29f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.961+0000 7fe76568d640 1 -- 172.21.15.169:0/678142520 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5556caa2b580 con 0x5556caa1a920 2024-03-28T00:57:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.961+0000 7fe765e8e640 1 --2- 172.21.15.169:0/678142520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa20480 0x5556caa2af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:57:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.961+0000 7fe76568d640 1 --2- 172.21.15.169:0/678142520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa1a920 0x5556caa29a40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe75c019040 tx=0x7fe75c0124c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e4ebdef6df6d33f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:57:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.961+0000 7fe748ff9640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:57:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.961+0000 7fe748ff9640 1 -- 172.21.15.169:0/678142520 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c03d050 con 0x5556caa1a920 2024-03-28T00:57:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.961+0000 7fe748ff9640 1 -- 172.21.15.169:0/678142520 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fe75c004280 con 0x5556caa1a920 2024-03-28T00:57:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.961+0000 7fe76a5b0740 1 -- 172.21.15.169:0/678142520 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe75c01e040 con 0x5556caa1a920 2024-03-28T00:57:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.961+0000 7fe748ff9640 1 -- 172.21.15.169:0/678142520 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe75c04c050 con 0x5556caa1a920 2024-03-28T00:57:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.961+0000 7fe76a5b0740 10 client.25732 fetch_fsmap learned FSMap version 25 2024-03-28T00:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.961+0000 7fe76a5b0740 1 -- 172.21.15.169:0/678142520 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5556caa94ee0 con 0x5556caa1a920 2024-03-28T00:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.961+0000 7fe748ff9640 1 -- 172.21.15.169:0/678142520 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe75c01e040 con 0x5556caa1a920 2024-03-28T00:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.961+0000 7fe748ff9640 1 -- 172.21.15.169:0/678142520 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fe75c004b00 con 0x5556caa1a920 2024-03-28T00:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.962+0000 7fe76a5b0740 10 client.25732 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.962+0000 7fe76a5b0740 10 client.25732 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.962+0000 7fe76a5b0740 20 client.25732 populate_metadata read hostname 'smithi169' 2024-03-28T00:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.962+0000 7fe76a5b0740 1 --2- 172.21.15.169:0/678142520 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5556caa95750 0x5556caa97b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.962+0000 7fe76a5b0740 4 client.25732 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T00:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.962+0000 7fe76a5b0740 1 -- 172.21.15.169:0/678142520 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5556ca9873a0 con 0x5556caa95750 2024-03-28T00:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.962+0000 7fe764e8c640 1 --2- 172.21.15.169:0/678142520 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5556caa95750 0x5556caa97b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:57:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.962+0000 7fe748ff9640 10 client.25732 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.962+0000 7fe764e8c640 1 --2- 172.21.15.169:0/678142520 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5556caa95750 0x5556caa97b70 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bdcc41a062a5bbf server_cookie=d5430e4875b4a5b3 in_seq=0 out_seq=0 2024-03-28T00:57:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.965+0000 7fe748ff9640 1 -- 172.21.15.169:0/678142520 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5556ca9873a0 con 0x5556caa95750 2024-03-28T00:57:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.965+0000 7fe748ff9640 10 client.25732 handle_command_reply: tid=0 2024-03-28T00:57:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.965+0000 7fe76a5b0740 1 client.25732 shutdown 2024-03-28T00:57:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.965+0000 7fe76a5b0740 1 -- 172.21.15.169:0/678142520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa1a920 msgr2=0x5556caa29a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.965+0000 7fe76a5b0740 1 --2- 172.21.15.169:0/678142520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa1a920 0x5556caa29a40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe75c019040 tx=0x7fe75c0124c0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.965+0000 7fe76a5b0740 1 -- 172.21.15.169:0/678142520 shutdown_connections 2024-03-28T00:57:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.965+0000 7fe76a5b0740 1 --2- 172.21.15.169:0/678142520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556caa20480 0x5556caa2af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.965+0000 7fe76a5b0740 1 --2- 172.21.15.169:0/678142520 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5556caa95750 0x5556caa97b70 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.965+0000 7fe76a5b0740 1 --2- 172.21.15.169:0/678142520 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556caa1b310 0x5556caa29f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.965+0000 7fe76a5b0740 1 --2- 172.21.15.169:0/678142520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556caa1a920 0x5556caa29a40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.965+0000 7fe748ff9640 0 client.25732 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:57:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.965+0000 7fe76a5b0740 1 -- 172.21.15.169:0/678142520 >> 172.21.15.169:0/678142520 conn(0x5556caa15e90 msgr2=0x5556caa1fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.965+0000 7fe76a5b0740 1 -- 172.21.15.169:0/678142520 shutdown_connections 2024-03-28T00:57:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.965+0000 7fe76a5b0740 1 -- 172.21.15.169:0/678142520 wait complete. 2024-03-28T00:57:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.965+0000 7fe76a5b0740 20 client.25732 trim_cache size 0 max 16384 2024-03-28T00:57:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T00:57:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T00:57:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T00:57:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T00:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.966+0000 7fe7678f1640 1 -- 172.21.15.169:0/2702760953 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe74c0a2940 msgr2=0x7fe74c0a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.966+0000 7fe7678f1640 1 --2- 172.21.15.169:0/2702760953 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe74c0a2940 0x7fe74c0a4e00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x5556caa1b710 tx=0x7fe75c005f90 comp rx=0 tx=0).stop 2024-03-28T00:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.966+0000 7fe7678f1640 1 -- 172.21.15.169:0/2702760953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7601008f0 msgr2=0x7fe76019a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.966+0000 7fe7678f1640 1 --2- 172.21.15.169:0/2702760953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7601008f0 0x7fe76019a100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe750000c00 tx=0x7fe75002c650 comp rx=0 tx=0).stop 2024-03-28T00:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.966+0000 7fe7678f1640 1 -- 172.21.15.169:0/2702760953 shutdown_connections 2024-03-28T00:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.966+0000 7fe7678f1640 1 --2- 172.21.15.169:0/2702760953 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe74c0a2940 0x7fe74c0a4e00 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.966+0000 7fe7678f1640 1 --2- 172.21.15.169:0/2702760953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe760101290 0x7fe7601a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.966+0000 7fe7678f1640 1 --2- 172.21.15.169:0/2702760953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7601008f0 0x7fe76019a100 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.966+0000 7fe7678f1640 1 --2- 172.21.15.169:0/2702760953 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7600fff90 0x7fe760199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.966+0000 7fe7678f1640 1 -- 172.21.15.169:0/2702760953 >> 172.21.15.169:0/2702760953 conn(0x7fe7600fbb30 msgr2=0x7fe7600fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.966+0000 7fe7678f1640 1 -- 172.21.15.169:0/2702760953 shutdown_connections 2024-03-28T00:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:57:57.966+0000 7fe7678f1640 1 -- 172.21.15.169:0/2702760953 wait complete. 2024-03-28T00:57:58.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:58 smithi169 ceph-mon[32274]: scrub queued for path: / 2024-03-28T00:57:58.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:58 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T00:57:58.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:58 smithi169 ceph-mon[32274]: scrub summary: active paths [/] 2024-03-28T00:57:58.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:58 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T00:57:58.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:58 smithi174 ceph-mon[36060]: scrub queued for path: / 2024-03-28T00:57:58.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:58 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T00:57:58.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:58 smithi174 ceph-mon[36060]: scrub summary: active paths [/] 2024-03-28T00:57:58.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:58 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T00:57:58.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:58 smithi177 ceph-mon[36206]: scrub queued for path: / 2024-03-28T00:57:58.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:58 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T00:57:58.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:58 smithi177 ceph-mon[36206]: scrub summary: active paths [/] 2024-03-28T00:57:58.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:58 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T00:57:59.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:59 smithi169 ceph-mon[32274]: pgmap v395: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T00:57:59.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:57:59 smithi169 ceph-mon[32274]: scrub summary: idle 2024-03-28T00:57:59.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:59 smithi174 ceph-mon[36060]: pgmap v395: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T00:57:59.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:57:59 smithi174 ceph-mon[36060]: scrub summary: idle 2024-03-28T00:57:59.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:59 smithi177 ceph-mon[36206]: pgmap v395: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T00:57:59.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:57:59 smithi177 ceph-mon[36206]: scrub summary: idle 2024-03-28T00:58:01.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:01 smithi169 ceph-mon[32274]: pgmap v396: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T00:58:01.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:01 smithi174 ceph-mon[36060]: pgmap v396: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T00:58:01.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:01 smithi177 ceph-mon[36206]: pgmap v396: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T00:58:03.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:03 smithi169 ceph-mon[32274]: pgmap v397: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T00:58:03.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:03 smithi174 ceph-mon[36060]: pgmap v397: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T00:58:03.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:03 smithi177 ceph-mon[36206]: pgmap v397: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T00:58:05.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:05 smithi169 ceph-mon[32274]: pgmap v398: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-28T00:58:05.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:05 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:58:05.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:05 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:58:05.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:05 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:58:05.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:05 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:58:05.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:05 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:58:05.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:05 smithi174 ceph-mon[36060]: pgmap v398: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-28T00:58:05.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:05 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:58:05.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:05 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:58:05.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:05 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:58:05.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:05 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:58:05.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:05 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:58:05.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:05 smithi177 ceph-mon[36206]: pgmap v398: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-28T00:58:05.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:05 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:58:05.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:05 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:58:05.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:05 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:58:05.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:05 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:58:05.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:05 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:58:06.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:06 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:58:06.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:06 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:58:06.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:06 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:58:06.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:06 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:58:06.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:06 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:58:06.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:06 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:58:06.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:06 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:58:06.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:06 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:58:06.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:06 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:58:07.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:07 smithi169 ceph-mon[32274]: pgmap v399: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-28T00:58:07.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:07 smithi174 ceph-mon[36060]: pgmap v399: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-28T00:58:07.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:07 smithi177 ceph-mon[36206]: pgmap v399: 129 pgs: 129 active+clean; 677 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-28T00:58:09.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:09 smithi169 ceph-mon[32274]: pgmap v400: 129 pgs: 129 active+clean; 678 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-28T00:58:09.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:09 smithi169 ceph-mon[32274]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-28T00:58:09.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:09 smithi174 ceph-mon[36060]: pgmap v400: 129 pgs: 129 active+clean; 678 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-28T00:58:09.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:09 smithi174 ceph-mon[36060]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-28T00:58:09.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:09 smithi177 ceph-mon[36206]: pgmap v400: 129 pgs: 129 active+clean; 678 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-28T00:58:09.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:09 smithi177 ceph-mon[36206]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-28T00:58:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.056+0000 7f7dfc4e8640 1 -- 172.21.15.169:0/397475476 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7df41005c0 msgr2=0x7f7df41009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.056+0000 7f7dfc4e8640 1 --2- 172.21.15.169:0/397475476 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7df41005c0 0x7f7df41009e0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7de4009900 tx=0x7f7de402bc30 comp rx=0 tx=0).stop 2024-03-28T00:58:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.056+0000 7f7dfc4e8640 1 -- 172.21.15.169:0/397475476 shutdown_connections 2024-03-28T00:58:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.056+0000 7f7dfc4e8640 1 --2- 172.21.15.169:0/397475476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7df4102170 0x7f7df4108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.056+0000 7f7dfc4e8640 1 --2- 172.21.15.169:0/397475476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7df41017b0 0x7f7df4101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.056+0000 7f7dfc4e8640 1 --2- 172.21.15.169:0/397475476 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7df41005c0 0x7f7df41009e0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.056+0000 7f7dfc4e8640 1 -- 172.21.15.169:0/397475476 >> 172.21.15.169:0/397475476 conn(0x7f7df40fbb30 msgr2=0x7f7df40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.056+0000 7f7dfc4e8640 1 -- 172.21.15.169:0/397475476 shutdown_connections 2024-03-28T00:58:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.056+0000 7f7dfc4e8640 1 -- 172.21.15.169:0/397475476 wait complete. 2024-03-28T00:58:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.056+0000 7f7dfc4e8640 1 Processor -- start 2024-03-28T00:58:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.056+0000 7f7dfc4e8640 1 -- start start 2024-03-28T00:58:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.056+0000 7f7dfc4e8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7df41005c0 0x7f7df4199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.056+0000 7f7dfc4e8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7df41017b0 0x7f7df419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.056+0000 7f7dfc4e8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7df4102170 0x7f7df41a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.056+0000 7f7dfc4e8640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df410c290 con 0x7f7df41017b0 2024-03-28T00:58:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.056+0000 7f7dfc4e8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df410c110 con 0x7f7df4102170 2024-03-28T00:58:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.056+0000 7f7dfc4e8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df410c410 con 0x7f7df41005c0 2024-03-28T00:58:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.057+0000 7f7dfa284640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7df41005c0 0x7f7df4199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.057+0000 7f7df9a83640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7df41017b0 0x7f7df419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.057+0000 7f7df9a83640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7df41017b0 0x7f7df419a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:35344/0 (socket says 172.21.15.169:35344) 2024-03-28T00:58:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.057+0000 7f7dfa284640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7df41005c0 0x7f7df4199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56928/0 (socket says 172.21.15.169:56928) 2024-03-28T00:58:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.057+0000 7f7df9a83640 1 -- 172.21.15.169:0/1184446709 learned_addr learned my addr 172.21.15.169:0/1184446709 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.057+0000 7f7dfaa85640 1 --2- 172.21.15.169:0/1184446709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7df4102170 0x7f7df41a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.057+0000 7f7df9a83640 1 -- 172.21.15.169:0/1184446709 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7df41005c0 msgr2=0x7f7df4199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.057+0000 7f7df9a83640 1 --2- 172.21.15.169:0/1184446709 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7df41005c0 0x7f7df4199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.057+0000 7f7df9a83640 1 -- 172.21.15.169:0/1184446709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7df4102170 msgr2=0x7f7df41a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.057+0000 7f7df9a83640 1 --2- 172.21.15.169:0/1184446709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7df4102170 0x7f7df41a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.057+0000 7f7df9a83640 1 -- 172.21.15.169:0/1184446709 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df41a17c0 con 0x7f7df41017b0 2024-03-28T00:58:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.057+0000 7f7df9a83640 1 --2- 172.21.15.169:0/1184446709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7df41017b0 0x7f7df419a0e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f7df000e9f0 tx=0x7f7df000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.057+0000 7f7deb7fe640 1 -- 172.21.15.169:0/1184446709 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df000cd80 con 0x7f7df41017b0 2024-03-28T00:58:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.057+0000 7f7deb7fe640 1 -- 172.21.15.169:0/1184446709 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7df0005500 con 0x7f7df41017b0 2024-03-28T00:58:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.057+0000 7f7dfc4e8640 1 -- 172.21.15.169:0/1184446709 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df41a1ab0 con 0x7f7df41017b0 2024-03-28T00:58:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.057+0000 7f7deb7fe640 1 -- 172.21.15.169:0/1184446709 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df0010690 con 0x7f7df41017b0 2024-03-28T00:58:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.057+0000 7f7dfc4e8640 1 -- 172.21.15.169:0/1184446709 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df41a1fb0 con 0x7f7df41017b0 2024-03-28T00:58:11.070 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.059+0000 7f7deb7fe640 1 -- 172.21.15.169:0/1184446709 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7df00046c0 con 0x7f7df41017b0 2024-03-28T00:58:11.070 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.060+0000 7f7deb7fe640 1 --2- 172.21.15.169:0/1184446709 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7de00a28a0 0x7f7de00a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:11.070 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.060+0000 7f7dff1ab740 1 Processor -- start 2024-03-28T00:58:11.070 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.060+0000 7f7deb7fe640 1 -- 172.21.15.169:0/1184446709 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f7df00cf1e0 con 0x7f7df41017b0 2024-03-28T00:58:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.060+0000 7f7dfa284640 1 --2- 172.21.15.169:0/1184446709 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7de00a28a0 0x7f7de00a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.060+0000 7f7dff1ab740 1 -- start start 2024-03-28T00:58:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.060+0000 7f7dff1ab740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757ca4920 0x564757ca4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.060+0000 7f7dff1ab740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757ca5310 0x564757ca9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.060+0000 7f7dff1ab740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757caa480 0x564757cac910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.060+0000 7f7dff1ab740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x564757c09b10 con 0x564757caa480 2024-03-28T00:58:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.060+0000 7f7dff1ab740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x564757bb2900 con 0x564757ca5310 2024-03-28T00:58:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.060+0000 7f7dff1ab740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x564757c05bb0 con 0x564757ca4920 2024-03-28T00:58:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.061+0000 7f7dfaa85640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757caa480 0x564757cac910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.061+0000 7f7dfa284640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757ca4920 0x564757ca4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.061+0000 7f7df9a83640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757ca5310 0x564757ca9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.061+0000 7f7df9a83640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757ca5310 0x564757ca9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:39346/0 (socket says 172.21.15.169:39346) 2024-03-28T00:58:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.061+0000 7f7dfaa85640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757caa480 0x564757cac910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:35358/0 (socket says 172.21.15.169:35358) 2024-03-28T00:58:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.061+0000 7f7df9a83640 1 -- 172.21.15.169:0/3721191200 learned_addr learned my addr 172.21.15.169:0/3721191200 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.061+0000 7f7dfa284640 1 --2- 172.21.15.169:0/1184446709 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7de00a28a0 0x7f7de00a4d60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7de402c170 tx=0x7f7de402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:11.072 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.061+0000 7f7df9a83640 1 -- 172.21.15.169:0/3721191200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757ca4920 msgr2=0x564757ca4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:11.072 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.061+0000 7f7df9a83640 1 --2- 172.21.15.169:0/3721191200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757ca4920 0x564757ca4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.072 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.061+0000 7f7df9a83640 1 -- 172.21.15.169:0/3721191200 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757caa480 msgr2=0x564757cac910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:11.072 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.061+0000 7f7df9a83640 1 --2- 172.21.15.169:0/3721191200 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757caa480 0x564757cac910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.072 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.061+0000 7f7df9a83640 1 -- 172.21.15.169:0/3721191200 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56475791e210 con 0x564757ca5310 2024-03-28T00:58:11.072 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.061+0000 7f7df9a83640 1 --2- 172.21.15.169:0/3721191200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757ca5310 0x564757ca9e10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7df0097c40 tx=0x7f7df009b320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=599549234995b140 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:11.072 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.061+0000 7f7de8bfe640 1 -- 172.21.15.169:0/3721191200 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df009b7e0 con 0x564757ca5310 2024-03-28T00:58:11.072 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.061+0000 7f7de8bfe640 1 -- 172.21.15.169:0/3721191200 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7df009f930 con 0x564757ca5310 2024-03-28T00:58:11.072 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.061+0000 7f7de8bfe640 1 -- 172.21.15.169:0/3721191200 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df00a3af0 con 0x564757ca5310 2024-03-28T00:58:11.072 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.062+0000 7f7dff1ab740 1 -- 172.21.15.169:0/3721191200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757ca5310 msgr2=0x564757ca9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:11.072 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.062+0000 7f7dff1ab740 1 --2- 172.21.15.169:0/3721191200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757ca5310 0x564757ca9e10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7df0097c40 tx=0x7f7df009b320 comp rx=0 tx=0).stop 2024-03-28T00:58:11.072 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.062+0000 7f7dff1ab740 1 -- 172.21.15.169:0/3721191200 shutdown_connections 2024-03-28T00:58:11.072 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.062+0000 7f7dff1ab740 1 --2- 172.21.15.169:0/3721191200 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757caa480 0x564757cac910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.072 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.062+0000 7f7dff1ab740 1 --2- 172.21.15.169:0/3721191200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757ca5310 0x564757ca9e10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.072 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.062+0000 7f7dff1ab740 1 --2- 172.21.15.169:0/3721191200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757ca4920 0x564757ca4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.073 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.062+0000 7f7dff1ab740 1 -- 172.21.15.169:0/3721191200 >> 172.21.15.169:0/3721191200 conn(0x564757c9fe90 msgr2=0x564757ca0290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:11.073 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.062+0000 7f7dff1ab740 1 -- 172.21.15.169:0/3721191200 shutdown_connections 2024-03-28T00:58:11.073 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.062+0000 7f7dff1ab740 1 -- 172.21.15.169:0/3721191200 wait complete. 2024-03-28T00:58:11.073 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.062+0000 7f7dff1ab740 1 -- start start 2024-03-28T00:58:11.073 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7dff1ab740 1 Processor -- start 2024-03-28T00:58:11.073 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7dff1ab740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757ca4920 0x564757caec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:11.073 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7dff1ab740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757ca5310 0x564757caf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:11.073 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7dff1ab740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757caa480 0x564757cb61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:11.073 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7dff1ab740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x564757cad430 con 0x564757ca5310 2024-03-28T00:58:11.073 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7dff1ab740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x564757bc5d50 con 0x564757caa480 2024-03-28T00:58:11.073 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7dff1ab740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x564757cad5b0 con 0x564757ca4920 2024-03-28T00:58:11.073 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7dff1ab740 10 client.0 _ll_register_callbacks cb 0x564757bc5720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:58:11.073 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7dfaa85640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757caa480 0x564757cb61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:11.073 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7df9a83640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757ca5310 0x564757caf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7dfa284640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757ca4920 0x564757caec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7dfaa85640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757caa480 0x564757cb61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:39350/0 (socket says 172.21.15.169:39350) 2024-03-28T00:58:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7df9a83640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757ca5310 0x564757caf190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:35372/0 (socket says 172.21.15.169:35372) 2024-03-28T00:58:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7dfaa85640 1 -- 172.21.15.169:0/2248088573 learned_addr learned my addr 172.21.15.169:0/2248088573 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7dfa284640 1 -- 172.21.15.169:0/2248088573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757caa480 msgr2=0x564757cb61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7dfa284640 1 --2- 172.21.15.169:0/2248088573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757caa480 0x564757cb61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7dfa284640 1 -- 172.21.15.169:0/2248088573 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757ca5310 msgr2=0x564757caf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7dfa284640 1 --2- 172.21.15.169:0/2248088573 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757ca5310 0x564757caf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7dfa284640 1 -- 172.21.15.169:0/2248088573 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5647579ea620 con 0x564757ca4920 2024-03-28T00:58:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7dfaa85640 1 --2- 172.21.15.169:0/2248088573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757caa480 0x564757cb61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:58:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7dfa284640 1 --2- 172.21.15.169:0/2248088573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757ca4920 0x564757caec50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7de4009230 tx=0x7f7de402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0b0d4238fae3f31 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.063+0000 7f7dddffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:58:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.064+0000 7f7dddffb640 1 -- 172.21.15.169:0/2248088573 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de40038c0 con 0x564757ca4920 2024-03-28T00:58:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.064+0000 7f7dff1ab740 1 -- 172.21.15.169:0/2248088573 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7df009f930 con 0x564757ca4920 2024-03-28T00:58:11.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.064+0000 7f7dddffb640 1 -- 172.21.15.169:0/2248088573 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f7de4003a60 con 0x564757ca4920 2024-03-28T00:58:11.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.064+0000 7f7dddffb640 1 -- 172.21.15.169:0/2248088573 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7de405d050 con 0x564757ca4920 2024-03-28T00:58:11.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.064+0000 7f7dff1ab740 10 client.25744 fetch_fsmap learned FSMap version 25 2024-03-28T00:58:11.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.064+0000 7f7dff1ab740 1 -- 172.21.15.169:0/2248088573 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x564757cb95b0 con 0x564757ca4920 2024-03-28T00:58:11.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.064+0000 7f7dddffb640 1 -- 172.21.15.169:0/2248088573 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7df009f930 con 0x564757ca4920 2024-03-28T00:58:11.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.064+0000 7f7dddffb640 1 -- 172.21.15.169:0/2248088573 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f7de4034440 con 0x564757ca4920 2024-03-28T00:58:11.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.064+0000 7f7dff1ab740 10 client.25744 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:58:11.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.064+0000 7f7dff1ab740 10 client.25744 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:58:11.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.064+0000 7f7dff1ab740 20 client.25744 populate_metadata read hostname 'smithi169' 2024-03-28T00:58:11.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.064+0000 7f7dff1ab740 1 --2- 172.21.15.169:0/2248088573 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564757cbac80 0x564757cbd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:11.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.064+0000 7f7dff1ab740 4 client.25744 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:58:11.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.064+0000 7f7dff1ab740 1 -- 172.21.15.169:0/2248088573 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564757c17050 con 0x564757cbac80 2024-03-28T00:58:11.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.064+0000 7f7df9a83640 1 --2- 172.21.15.169:0/2248088573 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564757cbac80 0x564757cbd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:11.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.065+0000 7f7dddffb640 10 client.25744 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:11.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.065+0000 7f7df9a83640 1 --2- 172.21.15.169:0/2248088573 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564757cbac80 0x564757cbd090 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81698865e94fae20 server_cookie=2f7c0d8434c72829 in_seq=0 out_seq=0 2024-03-28T00:58:11.076 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.065+0000 7f7dddffb640 1 -- 172.21.15.169:0/2248088573 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564757c17050 con 0x564757cbac80 2024-03-28T00:58:11.076 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.065+0000 7f7dddffb640 10 client.25744 handle_command_reply: tid=0 2024-03-28T00:58:11.076 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.065+0000 7f7dff1ab740 1 client.25744 shutdown 2024-03-28T00:58:11.076 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.065+0000 7f7dff1ab740 1 -- 172.21.15.169:0/2248088573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757ca4920 msgr2=0x564757caec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:11.076 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.065+0000 7f7dff1ab740 1 --2- 172.21.15.169:0/2248088573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757ca4920 0x564757caec50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7de4009230 tx=0x7f7de402bc30 comp rx=0 tx=0).stop 2024-03-28T00:58:11.076 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.066+0000 7f7dff1ab740 1 -- 172.21.15.169:0/2248088573 shutdown_connections 2024-03-28T00:58:11.076 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.066+0000 7f7dff1ab740 1 --2- 172.21.15.169:0/2248088573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757caa480 0x564757cb61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.076 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.066+0000 7f7dff1ab740 1 --2- 172.21.15.169:0/2248088573 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564757cbac80 0x564757cbd090 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.076 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.066+0000 7f7dff1ab740 1 --2- 172.21.15.169:0/2248088573 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757ca5310 0x564757caf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.076 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.066+0000 7f7dff1ab740 1 --2- 172.21.15.169:0/2248088573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757ca4920 0x564757caec50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.076 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.066+0000 7f7dddffb640 0 client.25744 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:11.076 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.066+0000 7f7dff1ab740 1 -- 172.21.15.169:0/2248088573 >> 172.21.15.169:0/2248088573 conn(0x564757c9fe90 msgr2=0x564757c9f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:11.076 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.066+0000 7f7dff1ab740 1 -- 172.21.15.169:0/2248088573 shutdown_connections 2024-03-28T00:58:11.076 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.066+0000 7f7dff1ab740 1 -- 172.21.15.169:0/2248088573 wait complete. 2024-03-28T00:58:11.076 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.066+0000 7f7dff1ab740 20 client.25744 trim_cache size 0 max 16384 2024-03-28T00:58:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.080+0000 7f7dff1ab740 1 Processor -- start 2024-03-28T00:58:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.080+0000 7f7dff1ab740 1 -- start start 2024-03-28T00:58:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.080+0000 7f7dff1ab740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757ca4920 0x564757cb6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.080+0000 7f7dff1ab740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757ca5310 0x564757cb6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.080+0000 7f7dff1ab740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757caa480 0x564757ca0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.080+0000 7f7dff1ab740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df41ae8e0 con 0x564757ca4920 2024-03-28T00:58:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.080+0000 7f7dff1ab740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df41ac3e0 con 0x564757caa480 2024-03-28T00:58:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.080+0000 7f7dff1ab740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df41aa480 con 0x564757ca5310 2024-03-28T00:58:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.080+0000 7f7df9a83640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757ca5310 0x564757cb6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.080+0000 7f7dfa284640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757ca4920 0x564757cb6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.080+0000 7f7df9a83640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757ca5310 0x564757cb6190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56948/0 (socket says 172.21.15.169:56948) 2024-03-28T00:58:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.080+0000 7f7dfa284640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757ca4920 0x564757cb6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:35388/0 (socket says 172.21.15.169:35388) 2024-03-28T00:58:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.080+0000 7f7df9a83640 1 -- 172.21.15.169:0/2701151123 learned_addr learned my addr 172.21.15.169:0/2701151123 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.080+0000 7f7dfaa85640 1 --2- 172.21.15.169:0/2701151123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757caa480 0x564757ca0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.080+0000 7f7df9a83640 1 -- 172.21.15.169:0/2701151123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757caa480 msgr2=0x564757ca0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.080+0000 7f7df9a83640 1 --2- 172.21.15.169:0/2701151123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757caa480 0x564757ca0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.080+0000 7f7df9a83640 1 -- 172.21.15.169:0/2701151123 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757ca4920 msgr2=0x564757cb6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.080+0000 7f7df9a83640 1 --2- 172.21.15.169:0/2701151123 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757ca4920 0x564757cb6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.080+0000 7f7df9a83640 1 -- 172.21.15.169:0/2701151123 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564757cafc40 con 0x564757ca5310 2024-03-28T00:58:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.080+0000 7f7dfa284640 1 --2- 172.21.15.169:0/2701151123 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757ca4920 0x564757cb6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:58:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.080+0000 7f7df9a83640 1 --2- 172.21.15.169:0/2701151123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757ca5310 0x564757cb6190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7df000c6a0 tx=0x7f7df009b450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c367fffd3411cf0f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.081+0000 7f7ddffff640 1 -- 172.21.15.169:0/2701151123 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df00b9070 con 0x564757ca5310 2024-03-28T00:58:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.081+0000 7f7ddffff640 1 -- 172.21.15.169:0/2701151123 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7df00ae070 con 0x564757ca5310 2024-03-28T00:58:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.081+0000 7f7ddffff640 1 -- 172.21.15.169:0/2701151123 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df00860b0 con 0x564757ca5310 2024-03-28T00:58:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.081+0000 7f7dff1ab740 1 -- 172.21.15.169:0/2701151123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757ca5310 msgr2=0x564757cb6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.081+0000 7f7dff1ab740 1 --2- 172.21.15.169:0/2701151123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757ca5310 0x564757cb6190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7df000c6a0 tx=0x7f7df009b450 comp rx=0 tx=0).stop 2024-03-28T00:58:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.081+0000 7f7dff1ab740 1 -- 172.21.15.169:0/2701151123 shutdown_connections 2024-03-28T00:58:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.081+0000 7f7dff1ab740 1 --2- 172.21.15.169:0/2701151123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757caa480 0x564757ca0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.081+0000 7f7dff1ab740 1 --2- 172.21.15.169:0/2701151123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757ca5310 0x564757cb6190 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.081+0000 7f7dff1ab740 1 --2- 172.21.15.169:0/2701151123 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757ca4920 0x564757cb6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.081+0000 7f7dff1ab740 1 -- 172.21.15.169:0/2701151123 >> 172.21.15.169:0/2701151123 conn(0x564757c9fe90 msgr2=0x564757ca9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.081+0000 7f7dff1ab740 1 -- 172.21.15.169:0/2701151123 shutdown_connections 2024-03-28T00:58:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.081+0000 7f7dff1ab740 1 -- 172.21.15.169:0/2701151123 wait complete. 2024-03-28T00:58:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.082+0000 7f7dff1ab740 1 -- start start 2024-03-28T00:58:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.082+0000 7f7dff1ab740 1 Processor -- start 2024-03-28T00:58:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.082+0000 7f7dff1ab740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757ca4920 0x564757cb3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.082+0000 7f7dff1ab740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757ca5310 0x564757cb40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.082+0000 7f7dff1ab740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757caa480 0x564757cb50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.082+0000 7f7dff1ab740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df41aaa80 con 0x564757ca4920 2024-03-28T00:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.082+0000 7f7dff1ab740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df41abd80 con 0x564757ca5310 2024-03-28T00:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.082+0000 7f7dff1ab740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df41ab0c0 con 0x564757caa480 2024-03-28T00:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.082+0000 7f7dff1ab740 10 client.0 _ll_register_callbacks cb 0x7f7de4004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.082+0000 7f7dfa284640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757ca4920 0x564757cb3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.082+0000 7f7dfa284640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757ca4920 0x564757cb3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:35402/0 (socket says 172.21.15.169:35402) 2024-03-28T00:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.082+0000 7f7dfa284640 1 -- 172.21.15.169:0/2320743033 learned_addr learned my addr 172.21.15.169:0/2320743033 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.082+0000 7f7df9a83640 1 --2- 172.21.15.169:0/2320743033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757ca5310 0x564757cb40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.082+0000 7f7dfaa85640 1 --2- 172.21.15.169:0/2320743033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757caa480 0x564757cb50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.082+0000 7f7dfa284640 1 -- 172.21.15.169:0/2320743033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757caa480 msgr2=0x564757cb50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.082+0000 7f7dfa284640 1 --2- 172.21.15.169:0/2320743033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757caa480 0x564757cb50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.082+0000 7f7dfa284640 1 -- 172.21.15.169:0/2320743033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757ca5310 msgr2=0x564757cb40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.082+0000 7f7dfa284640 1 --2- 172.21.15.169:0/2320743033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757ca5310 0x564757cb40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.082+0000 7f7dfa284640 1 -- 172.21.15.169:0/2320743033 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564757cb56b0 con 0x564757ca4920 2024-03-28T00:58:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.082+0000 7f7df9a83640 1 --2- 172.21.15.169:0/2320743033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757ca5310 0x564757cb40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:58:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.083+0000 7f7dfa284640 1 --2- 172.21.15.169:0/2320743033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757ca4920 0x564757cb3b70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7de402c240 tx=0x7f7de40346f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13bfc65157eb5df8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.083+0000 7f7dddffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:58:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.083+0000 7f7dddffb640 1 -- 172.21.15.169:0/2320743033 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de4034dc0 con 0x564757ca4920 2024-03-28T00:58:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.083+0000 7f7dddffb640 1 -- 172.21.15.169:0/2320743033 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f7de40038c0 con 0x564757ca4920 2024-03-28T00:58:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.083+0000 7f7dff1ab740 1 -- 172.21.15.169:0/2320743033 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7df00ae070 con 0x564757ca4920 2024-03-28T00:58:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.083+0000 7f7dddffb640 1 -- 172.21.15.169:0/2320743033 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7de4003ac0 con 0x564757ca4920 2024-03-28T00:58:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.083+0000 7f7dff1ab740 10 client.15945 fetch_fsmap learned FSMap version 25 2024-03-28T00:58:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.083+0000 7f7dff1ab740 1 -- 172.21.15.169:0/2320743033 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x564757d16740 con 0x564757ca4920 2024-03-28T00:58:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.083+0000 7f7dddffb640 1 -- 172.21.15.169:0/2320743033 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7df00ae070 con 0x564757ca4920 2024-03-28T00:58:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.083+0000 7f7dddffb640 1 -- 172.21.15.169:0/2320743033 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f7de40046c0 con 0x564757ca4920 2024-03-28T00:58:11.092 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.083+0000 7f7dff1ab740 10 client.15945 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:58:11.092 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.083+0000 7f7dff1ab740 10 client.15945 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:58:11.092 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.083+0000 7f7dff1ab740 20 client.15945 populate_metadata read hostname 'smithi169' 2024-03-28T00:58:11.092 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.083+0000 7f7dff1ab740 1 --2- 172.21.15.169:0/2320743033 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564757d170e0 0x564757d19500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:11.092 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.083+0000 7f7dff1ab740 4 client.15945 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T00:58:11.092 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.083+0000 7f7dff1ab740 1 -- 172.21.15.169:0/2320743033 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564757c113a0 con 0x564757d170e0 2024-03-28T00:58:11.092 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.084+0000 7f7df9a83640 1 --2- 172.21.15.169:0/2320743033 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564757d170e0 0x564757d19500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:11.092 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.084+0000 7f7df9a83640 1 --2- 172.21.15.169:0/2320743033 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564757d170e0 0x564757d19500 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c16ec6e599d41e8 server_cookie=c730ff5526d5aa61 in_seq=0 out_seq=0 2024-03-28T00:58:11.092 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.084+0000 7f7dddffb640 10 client.15945 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:11.093 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.088+0000 7f7dddffb640 1 -- 172.21.15.169:0/2320743033 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564757c113a0 con 0x564757d170e0 2024-03-28T00:58:11.093 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.088+0000 7f7dddffb640 10 client.15945 handle_command_reply: tid=0 2024-03-28T00:58:11.093 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.088+0000 7f7dff1ab740 1 client.15945 shutdown 2024-03-28T00:58:11.093 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.088+0000 7f7dff1ab740 1 -- 172.21.15.169:0/2320743033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757ca4920 msgr2=0x564757cb3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:11.093 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.088+0000 7f7dff1ab740 1 --2- 172.21.15.169:0/2320743033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757ca4920 0x564757cb3b70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7de402c240 tx=0x7f7de40346f0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.088+0000 7f7dff1ab740 1 -- 172.21.15.169:0/2320743033 shutdown_connections 2024-03-28T00:58:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.088+0000 7f7dff1ab740 1 --2- 172.21.15.169:0/2320743033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564757caa480 0x564757cb50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.088+0000 7f7dff1ab740 1 --2- 172.21.15.169:0/2320743033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564757ca5310 0x564757cb40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.088+0000 7f7dff1ab740 1 --2- 172.21.15.169:0/2320743033 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564757d170e0 0x564757d19500 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.088+0000 7f7dff1ab740 1 --2- 172.21.15.169:0/2320743033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564757ca4920 0x564757cb3b70 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.088+0000 7f7dddffb640 0 client.15945 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.088+0000 7f7dff1ab740 1 -- 172.21.15.169:0/2320743033 >> 172.21.15.169:0/2320743033 conn(0x564757c9fe90 msgr2=0x564757ca0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.088+0000 7f7dff1ab740 1 -- 172.21.15.169:0/2320743033 shutdown_connections 2024-03-28T00:58:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.088+0000 7f7dff1ab740 1 -- 172.21.15.169:0/2320743033 wait complete. 2024-03-28T00:58:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.089+0000 7f7dff1ab740 20 client.15945 trim_cache size 0 max 16384 2024-03-28T00:58:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T00:58:11.094 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T00:58:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T00:58:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T00:58:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.089+0000 7f7dfc4e8640 1 -- 172.21.15.169:0/1184446709 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7de00a28a0 msgr2=0x7f7de00a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.089+0000 7f7dfc4e8640 1 --2- 172.21.15.169:0/1184446709 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7de00a28a0 0x7f7de00a4d60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7de402c170 tx=0x7f7de402df90 comp rx=0 tx=0).stop 2024-03-28T00:58:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.089+0000 7f7dfc4e8640 1 -- 172.21.15.169:0/1184446709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7df41017b0 msgr2=0x7f7df419a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.089+0000 7f7dfc4e8640 1 --2- 172.21.15.169:0/1184446709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7df41017b0 0x7f7df419a0e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f7df000e9f0 tx=0x7f7df000edf0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.089+0000 7f7dfc4e8640 1 -- 172.21.15.169:0/1184446709 shutdown_connections 2024-03-28T00:58:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.089+0000 7f7dfc4e8640 1 --2- 172.21.15.169:0/1184446709 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7de00a28a0 0x7f7de00a4d60 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.089+0000 7f7dfc4e8640 1 --2- 172.21.15.169:0/1184446709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7df4102170 0x7f7df41a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.089+0000 7f7dfc4e8640 1 --2- 172.21.15.169:0/1184446709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7df41017b0 0x7f7df419a0e0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.089+0000 7f7dfc4e8640 1 --2- 172.21.15.169:0/1184446709 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7df41005c0 0x7f7df4199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.089+0000 7f7dfc4e8640 1 -- 172.21.15.169:0/1184446709 >> 172.21.15.169:0/1184446709 conn(0x7f7df40fbb30 msgr2=0x7f7df4068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.089+0000 7f7dfc4e8640 1 -- 172.21.15.169:0/1184446709 shutdown_connections 2024-03-28T00:58:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:11.089+0000 7f7dfc4e8640 1 -- 172.21.15.169:0/1184446709 wait complete. 2024-03-28T00:58:11.195 INFO:tasks.workunit.client.0.smithi169.stderr:Updating files: 99% (12188/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-28T00:58:11.476 INFO:tasks.workunit.client.0.smithi169.stderr:Note: switching to '3be9a5c9bc793e11e2800a8c0c696e8b46742033'. 2024-03-28T00:58:11.476 INFO:tasks.workunit.client.0.smithi169.stderr: 2024-03-28T00:58:11.476 INFO:tasks.workunit.client.0.smithi169.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-28T00:58:11.476 INFO:tasks.workunit.client.0.smithi169.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-28T00:58:11.477 INFO:tasks.workunit.client.0.smithi169.stderr:state without impacting any branches by switching back to a branch. 2024-03-28T00:58:11.477 INFO:tasks.workunit.client.0.smithi169.stderr: 2024-03-28T00:58:11.477 INFO:tasks.workunit.client.0.smithi169.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-28T00:58:11.477 INFO:tasks.workunit.client.0.smithi169.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-28T00:58:11.477 INFO:tasks.workunit.client.0.smithi169.stderr: 2024-03-28T00:58:11.477 INFO:tasks.workunit.client.0.smithi169.stderr: git switch -c 2024-03-28T00:58:11.477 INFO:tasks.workunit.client.0.smithi169.stderr: 2024-03-28T00:58:11.477 INFO:tasks.workunit.client.0.smithi169.stderr:Or undo this operation with: 2024-03-28T00:58:11.477 INFO:tasks.workunit.client.0.smithi169.stderr: 2024-03-28T00:58:11.477 INFO:tasks.workunit.client.0.smithi169.stderr: git switch - 2024-03-28T00:58:11.477 INFO:tasks.workunit.client.0.smithi169.stderr: 2024-03-28T00:58:11.477 INFO:tasks.workunit.client.0.smithi169.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-28T00:58:11.477 INFO:tasks.workunit.client.0.smithi169.stderr: 2024-03-28T00:58:11.477 INFO:tasks.workunit.client.0.smithi169.stderr:HEAD is now at 3be9a5c9bc7 dummy commit #1 2024-03-28T00:58:11.481 DEBUG:teuthology.orchestra.run.smithi169:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-28T00:58:11.540 INFO:tasks.workunit.client.0.smithi169.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-28T00:58:11.542 INFO:tasks.workunit.client.0.smithi169.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T00:58:11.543 INFO:tasks.workunit.client.0.smithi169.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-28T00:58:11.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:11 smithi169 ceph-mon[32274]: pgmap v401: 129 pgs: 129 active+clean; 678 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:58:11.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:11 smithi174 ceph-mon[36060]: pgmap v401: 129 pgs: 129 active+clean; 678 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:58:11.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:11 smithi177 ceph-mon[36206]: pgmap v401: 129 pgs: 129 active+clean; 678 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-28T00:58:12.285 INFO:tasks.workunit.client.0.smithi169.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-28T00:58:12.345 INFO:tasks.workunit.client.0.smithi169.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-28T00:58:12.385 INFO:tasks.workunit.client.0.smithi169.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T00:58:12.387 INFO:tasks.workunit.client.0.smithi169.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T00:58:12.387 INFO:tasks.workunit.client.0.smithi169.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-28T00:58:12.426 INFO:tasks.workunit.client.0.smithi169.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T00:58:12.429 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-28T00:58:12.429 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-28T00:58:12.485 INFO:tasks.workunit:Running workunits matching suites/iogen.sh on client.0... 2024-03-28T00:58:12.486 INFO:tasks.workunit:Running workunit suites/iogen.sh... 2024-03-28T00:58:12.487 DEBUG:teuthology.orchestra.run.smithi169:workunit test suites/iogen.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iogen.sh 2024-03-28T00:58:12.546 INFO:tasks.workunit.client.0.smithi169.stdout:getting iogen 2024-03-28T00:58:12.546 INFO:tasks.workunit.client.0.smithi169.stderr:+ echo 'getting iogen' 2024-03-28T00:58:12.546 INFO:tasks.workunit.client.0.smithi169.stderr:+ wget http://download.ceph.com/qa/iogen_3.1p0.tar 2024-03-28T00:58:12.570 INFO:tasks.workunit.client.0.smithi169.stderr:--2024-03-28 00:58:12-- http://download.ceph.com/qa/iogen_3.1p0.tar 2024-03-28T00:58:12.570 INFO:tasks.workunit.client.0.smithi169.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-03-28T00:58:12.591 INFO:tasks.workunit.client.0.smithi169.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-03-28T00:58:12.613 INFO:tasks.workunit.client.0.smithi169.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-28T00:58:12.613 INFO:tasks.workunit.client.0.smithi169.stderr:Length: 8071 (7.9K) [application/octet-stream] 2024-03-28T00:58:12.614 INFO:tasks.workunit.client.0.smithi169.stderr:Saving to: ‘iogen_3.1p0.tar’ 2024-03-28T00:58:12.617 INFO:tasks.workunit.client.0.smithi169.stderr: 2024-03-28T00:58:12.617 INFO:tasks.workunit.client.0.smithi169.stderr: 0K ....... 100% 342M=0s 2024-03-28T00:58:12.617 INFO:tasks.workunit.client.0.smithi169.stderr: 2024-03-28T00:58:12.617 INFO:tasks.workunit.client.0.smithi169.stderr:2024-03-28 00:58:12 (342 MB/s) - ‘iogen_3.1p0.tar’ saved [8071/8071] 2024-03-28T00:58:12.617 INFO:tasks.workunit.client.0.smithi169.stderr: 2024-03-28T00:58:12.617 INFO:tasks.workunit.client.0.smithi169.stderr:+ tar -xvzf iogen_3.1p0.tar 2024-03-28T00:58:12.628 INFO:tasks.workunit.client.0.smithi169.stdout:iogen_3.1p0 2024-03-28T00:58:12.632 INFO:tasks.workunit.client.0.smithi169.stdout:iogen_3.1p0/src 2024-03-28T00:58:12.633 INFO:tasks.workunit.client.0.smithi169.stdout:iogen_3.1p0/src/iogen.8 2024-03-28T00:58:12.634 INFO:tasks.workunit.client.0.smithi169.stdout:iogen_3.1p0/src/iogen.c 2024-03-28T00:58:12.637 INFO:tasks.workunit.client.0.smithi169.stdout:iogen_3.1p0/README 2024-03-28T00:58:12.641 INFO:tasks.workunit.client.0.smithi169.stdout:iogen_3.1p0/Makefile.bsd 2024-03-28T00:58:12.645 INFO:tasks.workunit.client.0.smithi169.stdout:iogen_3.1p0/Makefile 2024-03-28T00:58:12.651 INFO:tasks.workunit.client.0.smithi169.stderr:+ cd iogen_3.1p0 2024-03-28T00:58:12.651 INFO:tasks.workunit.client.0.smithi169.stdout:making iogen 2024-03-28T00:58:12.651 INFO:tasks.workunit.client.0.smithi169.stderr:+ echo 'making iogen' 2024-03-28T00:58:12.651 INFO:tasks.workunit.client.0.smithi169.stderr:+ make 2024-03-28T00:58:12.653 INFO:tasks.workunit.client.0.smithi169.stdout:gcc -D_FILE_OFFSET_BITS=64 -DVERSION=\"3.1p0\" -Dstrlcpy=strncpy -Darc4random=rand -O2 src/iogen.c -o iogen 2024-03-28T00:58:12.883 INFO:tasks.workunit.client.0.smithi169.stdout:nroff -mandoc -Tascii src/iogen.8 > iogen.cat8 2024-03-28T00:58:12.996 INFO:tasks.workunit.client.0.smithi169.stdout:running iogen 2024-03-28T00:58:12.996 INFO:tasks.workunit.client.0.smithi169.stderr:+ echo 'running iogen' 2024-03-28T00:58:12.996 INFO:tasks.workunit.client.0.smithi169.stderr:+ ./iogen -n 5 -s 2g 2024-03-28T00:58:12.998 INFO:tasks.workunit.client.0.smithi169.stderr:child spawned 87457 2024-03-28T00:58:12.998 INFO:tasks.workunit.client.0.smithi169.stderr:child spawned 87458 2024-03-28T00:58:12.998 INFO:tasks.workunit.client.0.smithi169.stderr:child spawned 87459 2024-03-28T00:58:12.998 INFO:tasks.workunit.client.0.smithi169.stderr:child spawned 87460 2024-03-28T00:58:12.998 INFO:tasks.workunit.client.0.smithi169.stderr:child spawned 87461 2024-03-28T00:58:12.998 INFO:tasks.workunit.client.0.smithi169.stdout:sleep for 10 min 2024-03-28T00:58:12.999 INFO:tasks.workunit.client.0.smithi169.stderr:+ echo 'sleep for 10 min' 2024-03-28T00:58:12.999 INFO:tasks.workunit.client.0.smithi169.stderr:+ sleep 600 2024-03-28T00:58:13.265 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:13 smithi169 ceph-mon[32274]: pgmap v402: 129 pgs: 129 active+clean; 678 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-28T00:58:13.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:13 smithi174 ceph-mon[36060]: pgmap v402: 129 pgs: 129 active+clean; 678 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-28T00:58:13.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:13 smithi177 ceph-mon[36206]: pgmap v402: 129 pgs: 129 active+clean; 678 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-28T00:58:15.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:15 smithi169 ceph-mon[32274]: pgmap v403: 129 pgs: 129 active+clean; 13 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.0 MiB/s wr, 8 op/s 2024-03-28T00:58:15.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:15 smithi174 ceph-mon[36060]: pgmap v403: 129 pgs: 129 active+clean; 13 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.0 MiB/s wr, 8 op/s 2024-03-28T00:58:15.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:15 smithi177 ceph-mon[36206]: pgmap v403: 129 pgs: 129 active+clean; 13 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.0 MiB/s wr, 8 op/s 2024-03-28T00:58:16.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:16 smithi174 ceph-mon[36060]: pgmap v404: 129 pgs: 129 active+clean; 1.5 GiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 132 MiB/s wr, 41 op/s 2024-03-28T00:58:16.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:16 smithi177 ceph-mon[36206]: pgmap v404: 129 pgs: 129 active+clean; 1.5 GiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 132 MiB/s wr, 41 op/s 2024-03-28T00:58:16.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:16 smithi169 ceph-mon[32274]: pgmap v404: 129 pgs: 129 active+clean; 1.5 GiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 132 MiB/s wr, 41 op/s 2024-03-28T00:58:19.521 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:19 smithi169 ceph-mon[32274]: pgmap v405: 129 pgs: 129 active+clean; 2.0 GiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 169 MiB/s wr, 124 op/s 2024-03-28T00:58:19.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:19 smithi174 ceph-mon[36060]: pgmap v405: 129 pgs: 129 active+clean; 2.0 GiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 169 MiB/s wr, 124 op/s 2024-03-28T00:58:19.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:19 smithi177 ceph-mon[36206]: pgmap v405: 129 pgs: 129 active+clean; 2.0 GiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 169 MiB/s wr, 124 op/s 2024-03-28T00:58:21.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:21 smithi169 ceph-mon[32274]: pgmap v406: 129 pgs: 129 active+clean; 2.7 GiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 233 MiB/s wr, 139 op/s 2024-03-28T00:58:21.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:21 smithi174 ceph-mon[36060]: pgmap v406: 129 pgs: 129 active+clean; 2.7 GiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 233 MiB/s wr, 139 op/s 2024-03-28T00:58:21.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:21 smithi177 ceph-mon[36206]: pgmap v406: 129 pgs: 129 active+clean; 2.7 GiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 233 MiB/s wr, 139 op/s 2024-03-28T00:58:23.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:23 smithi169 ceph-mon[32274]: pgmap v407: 129 pgs: 129 active+clean; 6.8 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 593 MiB/s wr, 555 op/s 2024-03-28T00:58:23.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:23 smithi174 ceph-mon[36060]: pgmap v407: 129 pgs: 129 active+clean; 6.8 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 593 MiB/s wr, 555 op/s 2024-03-28T00:58:23.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:23 smithi177 ceph-mon[36206]: pgmap v407: 129 pgs: 129 active+clean; 6.8 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 593 MiB/s wr, 555 op/s 2024-03-28T00:58:24.373 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.365+0000 7fde2ef45640 1 -- 172.21.15.169:0/4225171723 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde28078df0 msgr2=0x7fde2807b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:24.373 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.365+0000 7fde2ef45640 1 --2- 172.21.15.169:0/4225171723 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde28078df0 0x7fde2807b280 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fde20009ca0 tx=0x7fde2002c140 comp rx=0 tx=0).stop 2024-03-28T00:58:24.373 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.365+0000 7fde2ef45640 1 -- 172.21.15.169:0/4225171723 shutdown_connections 2024-03-28T00:58:24.373 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.365+0000 7fde2ef45640 1 --2- 172.21.15.169:0/4225171723 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde28078df0 0x7fde2807b280 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.374 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.365+0000 7fde2ef45640 1 --2- 172.21.15.169:0/4225171723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde28076440 0x7fde280788b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.374 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.365+0000 7fde2ef45640 1 --2- 172.21.15.169:0/4225171723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde28073560 0x7fde28073980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.374 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.365+0000 7fde2ef45640 1 -- 172.21.15.169:0/4225171723 >> 172.21.15.169:0/4225171723 conn(0x7fde2806e6e0 msgr2=0x7fde28070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:24.382 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.374+0000 7fde2ef45640 1 -- 172.21.15.169:0/4225171723 shutdown_connections 2024-03-28T00:58:24.382 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.378+0000 7fde2ef45640 1 -- 172.21.15.169:0/4225171723 wait complete. 2024-03-28T00:58:24.383 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.380+0000 7fde2ef45640 1 Processor -- start 2024-03-28T00:58:24.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.381+0000 7fde2ef45640 1 -- start start 2024-03-28T00:58:24.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.382+0000 7fde2ef45640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde28073560 0x7fde2812c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:24.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.383+0000 7fde2ef45640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde28076440 0x7fde2812ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:24.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.383+0000 7fde2ef45640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde2812f370 0x7fde28131830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:24.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.383+0000 7fde2ef45640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde28082f90 con 0x7fde28073560 2024-03-28T00:58:24.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.383+0000 7fde2ef45640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde28082e10 con 0x7fde2812f370 2024-03-28T00:58:24.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.383+0000 7fde2ef45640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde28083110 con 0x7fde28076440 2024-03-28T00:58:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.387+0000 7fde27fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde28076440 0x7fde2812ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.388+0000 7fde27fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde28076440 0x7fde2812ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:59522/0 (socket says 172.21.15.169:59522) 2024-03-28T00:58:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.388+0000 7fde27fff640 1 -- 172.21.15.169:0/3860147881 learned_addr learned my addr 172.21.15.169:0/3860147881 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.388+0000 7fde2cce1640 1 --2- 172.21.15.169:0/3860147881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde28073560 0x7fde2812c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.388+0000 7fde2d4e2640 1 --2- 172.21.15.169:0/3860147881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde2812f370 0x7fde28131830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.395+0000 7fde2cce1640 1 -- 172.21.15.169:0/3860147881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde28076440 msgr2=0x7fde2812ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.395+0000 7fde2cce1640 1 --2- 172.21.15.169:0/3860147881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde28076440 0x7fde2812ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.395+0000 7fde2cce1640 1 -- 172.21.15.169:0/3860147881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde2812f370 msgr2=0x7fde28131830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.395+0000 7fde2cce1640 1 --2- 172.21.15.169:0/3860147881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde2812f370 0x7fde28131830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.395+0000 7fde2cce1640 1 -- 172.21.15.169:0/3860147881 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde28132140 con 0x7fde28073560 2024-03-28T00:58:24.403 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.400+0000 7fde2cce1640 1 --2- 172.21.15.169:0/3860147881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde28073560 0x7fde2812c8e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fde1800e9f0 tx=0x7fde1800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:24.413 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.410+0000 7fde25ffb640 1 -- 172.21.15.169:0/3860147881 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1800ccc0 con 0x7fde28073560 2024-03-28T00:58:24.413 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.410+0000 7fde25ffb640 1 -- 172.21.15.169:0/3860147881 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde18010910 con 0x7fde28073560 2024-03-28T00:58:24.413 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.410+0000 7fde25ffb640 1 -- 172.21.15.169:0/3860147881 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde18018960 con 0x7fde28073560 2024-03-28T00:58:24.413 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.410+0000 7fde2ef45640 1 -- 172.21.15.169:0/3860147881 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde281323d0 con 0x7fde28073560 2024-03-28T00:58:24.413 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.410+0000 7fde2ef45640 1 -- 172.21.15.169:0/3860147881 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde281328d0 con 0x7fde28073560 2024-03-28T00:58:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.418+0000 7fde25ffb640 1 -- 172.21.15.169:0/3860147881 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fde18010ac0 con 0x7fde28073560 2024-03-28T00:58:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.419+0000 7fde25ffb640 1 --2- 172.21.15.169:0/3860147881 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fde100a2ac0 0x7fde100a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.419+0000 7fde25ffb640 1 -- 172.21.15.169:0/3860147881 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fde180cf1c0 con 0x7fde28073560 2024-03-28T00:58:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.425+0000 7fde27fff640 1 --2- 172.21.15.169:0/3860147881 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fde100a2ac0 0x7fde100a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:24.432 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.429+0000 7fde31c0d740 1 Processor -- start 2024-03-28T00:58:24.432 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.430+0000 7fde31c0d740 1 -- start start 2024-03-28T00:58:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.430+0000 7fde31c0d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557478ee9920 0x557478ee9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.431+0000 7fde31c0d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478eea310 0x557478eeee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.431+0000 7fde31c0d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557478eef480 0x557478ef1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.431+0000 7fde31c0d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x557478e4eb10 con 0x557478ee9920 2024-03-28T00:58:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.431+0000 7fde31c0d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x557478df7900 con 0x557478eea310 2024-03-28T00:58:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.431+0000 7fde31c0d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557478e4abb0 con 0x557478eef480 2024-03-28T00:58:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.431+0000 7fde27fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478eea310 0x557478eeee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.431+0000 7fde27fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478eea310 0x557478eeee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:43290/0 (socket says 172.21.15.169:43290) 2024-03-28T00:58:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.431+0000 7fde27fff640 1 -- 172.21.15.169:0/4022832368 learned_addr learned my addr 172.21.15.169:0/4022832368 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.432+0000 7fde2d4e2640 1 --2- 172.21.15.169:0/4022832368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557478eef480 0x557478ef1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:24.437 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.434+0000 7fde2d4e2640 1 -- 172.21.15.169:0/4022832368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478eea310 msgr2=0x557478eeee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:24.437 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.434+0000 7fde2d4e2640 1 --2- 172.21.15.169:0/4022832368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478eea310 0x557478eeee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.437 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.434+0000 7fde2d4e2640 1 -- 172.21.15.169:0/4022832368 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557478ee9920 msgr2=0x557478ee9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:24.437 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.434+0000 7fde2d4e2640 1 --2- 172.21.15.169:0/4022832368 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557478ee9920 0x557478ee9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.437 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.434+0000 7fde2d4e2640 1 -- 172.21.15.169:0/4022832368 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557478b63210 con 0x557478eef480 2024-03-28T00:58:24.438 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.436+0000 7fde27fff640 1 --2- 172.21.15.169:0/4022832368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478eea310 0x557478eeee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:58:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.437+0000 7fde2d4e2640 1 --2- 172.21.15.169:0/4022832368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557478eef480 0x557478ef1910 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fde20009bf0 tx=0x7fde20009280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f791f64d413eaafc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.442+0000 7fde27fff640 1 --2- 172.21.15.169:0/3860147881 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fde100a2ac0 0x7fde100a4f80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x557478eea710 tx=0x7fde1c00b660 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.443+0000 7fde0effd640 1 -- 172.21.15.169:0/4022832368 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde20039050 con 0x557478eef480 2024-03-28T00:58:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.444+0000 7fde0effd640 1 -- 172.21.15.169:0/4022832368 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde20045070 con 0x557478eef480 2024-03-28T00:58:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.444+0000 7fde0effd640 1 -- 172.21.15.169:0/4022832368 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde20040870 con 0x557478eef480 2024-03-28T00:58:24.451 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.448+0000 7fde31c0d740 1 -- 172.21.15.169:0/4022832368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557478eef480 msgr2=0x557478ef1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:24.451 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.448+0000 7fde31c0d740 1 --2- 172.21.15.169:0/4022832368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557478eef480 0x557478ef1910 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fde20009bf0 tx=0x7fde20009280 comp rx=0 tx=0).stop 2024-03-28T00:58:24.452 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.450+0000 7fde31c0d740 1 -- 172.21.15.169:0/4022832368 shutdown_connections 2024-03-28T00:58:24.452 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.450+0000 7fde31c0d740 1 --2- 172.21.15.169:0/4022832368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557478eef480 0x557478ef1910 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.452 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.450+0000 7fde31c0d740 1 --2- 172.21.15.169:0/4022832368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478eea310 0x557478eeee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.452 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.450+0000 7fde31c0d740 1 --2- 172.21.15.169:0/4022832368 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557478ee9920 0x557478ee9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.453 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.450+0000 7fde31c0d740 1 -- 172.21.15.169:0/4022832368 >> 172.21.15.169:0/4022832368 conn(0x557478ee4e90 msgr2=0x557478ee5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:24.453 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.450+0000 7fde31c0d740 1 -- 172.21.15.169:0/4022832368 shutdown_connections 2024-03-28T00:58:24.453 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.451+0000 7fde31c0d740 1 -- 172.21.15.169:0/4022832368 wait complete. 2024-03-28T00:58:24.454 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.451+0000 7fde31c0d740 1 -- start start 2024-03-28T00:58:24.454 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.452+0000 7fde31c0d740 1 Processor -- start 2024-03-28T00:58:24.462 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.457+0000 7fde31c0d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557478ee9920 0x557478ef3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:24.462 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.457+0000 7fde31c0d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478eea310 0x557478ef41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:24.462 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.457+0000 7fde31c0d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557478eef480 0x557478efb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:24.463 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.457+0000 7fde31c0d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x557478ef23d0 con 0x557478eef480 2024-03-28T00:58:24.463 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.457+0000 7fde31c0d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x557478e0ad50 con 0x557478eea310 2024-03-28T00:58:24.463 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.457+0000 7fde31c0d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557478ef2550 con 0x557478ee9920 2024-03-28T00:58:24.463 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.457+0000 7fde31c0d740 10 client.0 _ll_register_callbacks cb 0x557478e0a720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:58:24.464 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.460+0000 7fde27fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478eea310 0x557478ef41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:24.464 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.461+0000 7fde27fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478eea310 0x557478ef41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:43304/0 (socket says 172.21.15.169:43304) 2024-03-28T00:58:24.464 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.461+0000 7fde27fff640 1 -- 172.21.15.169:0/3204012564 learned_addr learned my addr 172.21.15.169:0/3204012564 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:24.464 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.460+0000 7fde2cce1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557478ee9920 0x557478ef3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:24.465 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.461+0000 7fde2d4e2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557478eef480 0x557478efb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:24.468 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.463+0000 7fde27fff640 1 -- 172.21.15.169:0/3204012564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557478ee9920 msgr2=0x557478ef3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:24.468 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.463+0000 7fde27fff640 1 --2- 172.21.15.169:0/3204012564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557478ee9920 0x557478ef3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.468 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.463+0000 7fde27fff640 1 -- 172.21.15.169:0/3204012564 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557478eef480 msgr2=0x557478efb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:24.468 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.463+0000 7fde27fff640 1 --2- 172.21.15.169:0/3204012564 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557478eef480 0x557478efb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.468 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.463+0000 7fde27fff640 1 -- 172.21.15.169:0/3204012564 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557478c2f620 con 0x557478eea310 2024-03-28T00:58:24.469 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.466+0000 7fde27fff640 1 --2- 172.21.15.169:0/3204012564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478eea310 0x557478ef41d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fde1c0116f0 tx=0x7fde1c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e18fac27b8a3b36c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.470+0000 7fde07fff640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:58:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.470+0000 7fde07fff640 1 -- 172.21.15.169:0/3204012564 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1c004230 con 0x557478eea310 2024-03-28T00:58:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.470+0000 7fde07fff640 1 -- 172.21.15.169:0/3204012564 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fde1c0043d0 con 0x557478eea310 2024-03-28T00:58:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.470+0000 7fde31c0d740 1 -- 172.21.15.169:0/3204012564 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fde20045070 con 0x557478eea310 2024-03-28T00:58:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.471+0000 7fde07fff640 1 -- 172.21.15.169:0/3204012564 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde1c02e050 con 0x557478eea310 2024-03-28T00:58:24.477 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.474+0000 7fde31c0d740 10 client.25781 fetch_fsmap learned FSMap version 25 2024-03-28T00:58:24.477 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.474+0000 7fde31c0d740 1 -- 172.21.15.169:0/3204012564 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557478efe590 con 0x557478eea310 2024-03-28T00:58:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.477+0000 7fde07fff640 1 -- 172.21.15.169:0/3204012564 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde20045070 con 0x557478eea310 2024-03-28T00:58:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.479+0000 7fde07fff640 1 -- 172.21.15.169:0/3204012564 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fde1c003c70 con 0x557478eea310 2024-03-28T00:58:24.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.480+0000 7fde31c0d740 10 client.25781 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:58:24.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.480+0000 7fde31c0d740 10 client.25781 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:58:24.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.480+0000 7fde31c0d740 20 client.25781 populate_metadata read hostname 'smithi169' 2024-03-28T00:58:24.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.480+0000 7fde31c0d740 1 --2- 172.21.15.169:0/3204012564 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557478effcb0 0x557478f020c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:24.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.480+0000 7fde31c0d740 4 client.25781 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:58:24.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.480+0000 7fde31c0d740 1 -- 172.21.15.169:0/3204012564 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557478e5c050 con 0x557478effcb0 2024-03-28T00:58:24.488 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.485+0000 7fde2cce1640 1 --2- 172.21.15.169:0/3204012564 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557478effcb0 0x557478f020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:24.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.488+0000 7fde2cce1640 1 --2- 172.21.15.169:0/3204012564 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557478effcb0 0x557478f020c0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dca60e1c413c1c8f server_cookie=63f17fd3f83a1e0b in_seq=0 out_seq=0 2024-03-28T00:58:24.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.488+0000 7fde07fff640 10 client.25781 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:24.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.494+0000 7fde07fff640 1 -- 172.21.15.169:0/3204012564 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557478e5c050 con 0x557478effcb0 2024-03-28T00:58:24.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.494+0000 7fde07fff640 10 client.25781 handle_command_reply: tid=0 2024-03-28T00:58:24.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.494+0000 7fde31c0d740 1 client.25781 shutdown 2024-03-28T00:58:24.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.497+0000 7fde31c0d740 1 -- 172.21.15.169:0/3204012564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478eea310 msgr2=0x557478ef41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:24.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.497+0000 7fde31c0d740 1 --2- 172.21.15.169:0/3204012564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478eea310 0x557478ef41d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fde1c0116f0 tx=0x7fde1c0046c0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.497+0000 7fde31c0d740 1 -- 172.21.15.169:0/3204012564 shutdown_connections 2024-03-28T00:58:24.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.497+0000 7fde31c0d740 1 --2- 172.21.15.169:0/3204012564 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557478effcb0 0x557478f020c0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.497+0000 7fde31c0d740 1 --2- 172.21.15.169:0/3204012564 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557478eef480 0x557478efb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.497+0000 7fde31c0d740 1 --2- 172.21.15.169:0/3204012564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478eea310 0x557478ef41d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.497+0000 7fde31c0d740 1 --2- 172.21.15.169:0/3204012564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557478ee9920 0x557478ef3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.497+0000 7fde07fff640 0 client.25781 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:24.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.497+0000 7fde31c0d740 1 -- 172.21.15.169:0/3204012564 >> 172.21.15.169:0/3204012564 conn(0x557478ee4e90 msgr2=0x557478ee4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:24.502 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.498+0000 7fde31c0d740 1 -- 172.21.15.169:0/3204012564 shutdown_connections 2024-03-28T00:58:24.502 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.499+0000 7fde31c0d740 1 -- 172.21.15.169:0/3204012564 wait complete. 2024-03-28T00:58:24.503 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.499+0000 7fde31c0d740 20 client.25781 trim_cache size 0 max 16384 2024-03-28T00:58:24.578 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.575+0000 7fde31c0d740 1 Processor -- start 2024-03-28T00:58:24.582 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.580+0000 7fde31c0d740 1 -- start start 2024-03-28T00:58:24.583 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.580+0000 7fde31c0d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478ee9920 0x557478eeb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:24.583 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.580+0000 7fde31c0d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557478eea310 0x557478efb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:24.584 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.581+0000 7fde31c0d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557478eef480 0x557478eebbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:24.584 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.581+0000 7fde31c0d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde14011f50 con 0x557478eea310 2024-03-28T00:58:24.584 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.581+0000 7fde31c0d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1400fa50 con 0x557478ee9920 2024-03-28T00:58:24.584 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.581+0000 7fde31c0d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1400dba0 con 0x557478eef480 2024-03-28T00:58:24.585 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.582+0000 7fde27fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557478eea310 0x557478efb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:24.585 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.582+0000 7fde27fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557478eea310 0x557478efb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40998/0 (socket says 172.21.15.169:40998) 2024-03-28T00:58:24.585 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.582+0000 7fde27fff640 1 -- 172.21.15.169:0/3015227440 learned_addr learned my addr 172.21.15.169:0/3015227440 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.583+0000 7fde27fff640 1 -- 172.21.15.169:0/3015227440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557478eef480 msgr2=0x557478eebbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T00:58:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.583+0000 7fde27fff640 1 --2- 172.21.15.169:0/3015227440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557478eef480 0x557478eebbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.583+0000 7fde27fff640 1 -- 172.21.15.169:0/3015227440 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478ee9920 msgr2=0x557478eeb670 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T00:58:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.583+0000 7fde27fff640 1 --2- 172.21.15.169:0/3015227440 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478ee9920 0x557478eeb670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.583+0000 7fde27fff640 1 -- 172.21.15.169:0/3015227440 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557478eea0b0 con 0x557478eea310 2024-03-28T00:58:24.587 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.584+0000 7fde27fff640 1 --2- 172.21.15.169:0/3015227440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557478eea310 0x557478efb1d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fde1c00b4a0 tx=0x7fde1c005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74e571213362a6b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.590+0000 7fde0e7fc640 1 -- 172.21.15.169:0/3015227440 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1c012740 con 0x557478eea310 2024-03-28T00:58:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.590+0000 7fde0e7fc640 1 -- 172.21.15.169:0/3015227440 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde1c036070 con 0x557478eea310 2024-03-28T00:58:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.590+0000 7fde0e7fc640 1 -- 172.21.15.169:0/3015227440 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1c03b860 con 0x557478eea310 2024-03-28T00:58:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.593+0000 7fde31c0d740 1 -- 172.21.15.169:0/3015227440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557478eea310 msgr2=0x557478efb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.593+0000 7fde31c0d740 1 --2- 172.21.15.169:0/3015227440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557478eea310 0x557478efb1d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fde1c00b4a0 tx=0x7fde1c005f70 comp rx=0 tx=0).stop 2024-03-28T00:58:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.593+0000 7fde31c0d740 1 -- 172.21.15.169:0/3015227440 shutdown_connections 2024-03-28T00:58:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.593+0000 7fde31c0d740 1 --2- 172.21.15.169:0/3015227440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557478eef480 0x557478eebbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.599 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.593+0000 7fde31c0d740 1 --2- 172.21.15.169:0/3015227440 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557478eea310 0x557478efb1d0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.599 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.593+0000 7fde31c0d740 1 --2- 172.21.15.169:0/3015227440 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478ee9920 0x557478eeb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.599 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.593+0000 7fde31c0d740 1 -- 172.21.15.169:0/3015227440 >> 172.21.15.169:0/3015227440 conn(0x557478ee4e90 msgr2=0x557478ee5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:24.605 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.603+0000 7fde31c0d740 1 -- 172.21.15.169:0/3015227440 shutdown_connections 2024-03-28T00:58:24.606 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.603+0000 7fde31c0d740 1 -- 172.21.15.169:0/3015227440 wait complete. 2024-03-28T00:58:24.606 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.603+0000 7fde31c0d740 1 -- start start 2024-03-28T00:58:24.606 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.604+0000 7fde31c0d740 1 Processor -- start 2024-03-28T00:58:24.608 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.605+0000 7fde31c0d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478ee9920 0x557478ef7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:24.608 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.605+0000 7fde31c0d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557478eea310 0x557478ef84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:24.609 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.605+0000 7fde31c0d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557478eef480 0x557478f59580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:24.609 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.605+0000 7fde31c0d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1400e110 con 0x557478eef480 2024-03-28T00:58:24.609 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.605+0000 7fde31c0d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1400f3f0 con 0x557478ee9920 2024-03-28T00:58:24.609 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.605+0000 7fde31c0d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1400e730 con 0x557478eea310 2024-03-28T00:58:24.609 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.605+0000 7fde31c0d740 10 client.0 _ll_register_callbacks cb 0x7fde1c00fe60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:58:24.610 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.606+0000 7fde2d4e2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557478eef480 0x557478f59580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:24.610 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.606+0000 7fde2cce1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478ee9920 0x557478ef7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:24.610 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.606+0000 7fde2cce1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478ee9920 0x557478ef7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:43326/0 (socket says 172.21.15.169:43326) 2024-03-28T00:58:24.610 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.606+0000 7fde2cce1640 1 -- 172.21.15.169:0/4121935735 learned_addr learned my addr 172.21.15.169:0/4121935735 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:24.610 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.606+0000 7fde27fff640 1 --2- 172.21.15.169:0/4121935735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557478eea310 0x557478ef84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:24.611 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.607+0000 7fde27fff640 1 -- 172.21.15.169:0/4121935735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478ee9920 msgr2=0x557478ef7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:24.611 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.607+0000 7fde27fff640 1 --2- 172.21.15.169:0/4121935735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478ee9920 0x557478ef7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.611 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.607+0000 7fde27fff640 1 -- 172.21.15.169:0/4121935735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557478eef480 msgr2=0x557478f59580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:24.611 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.607+0000 7fde27fff640 1 --2- 172.21.15.169:0/4121935735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557478eef480 0x557478f59580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:24.611 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.607+0000 7fde27fff640 1 -- 172.21.15.169:0/4121935735 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557478ef8b40 con 0x557478eea310 2024-03-28T00:58:24.616 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.609+0000 7fde27fff640 1 --2- 172.21.15.169:0/4121935735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557478eea310 0x557478ef84a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fde1c00fc30 tx=0x7fde1c012970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7e8d0c9ac26302b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:24.616 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.609+0000 7fde07fff640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:58:24.616 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.613+0000 7fde07fff640 1 -- 172.21.15.169:0/4121935735 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1c041030 con 0x557478eea310 2024-03-28T00:58:24.616 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.613+0000 7fde31c0d740 1 -- 172.21.15.169:0/4121935735 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fde1c036070 con 0x557478eea310 2024-03-28T00:58:24.616 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.613+0000 7fde07fff640 1 -- 172.21.15.169:0/4121935735 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fde1c004020 con 0x557478eea310 2024-03-28T00:58:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.620+0000 7fde07fff640 1 -- 172.21.15.169:0/4121935735 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde1c012d70 con 0x557478eea310 2024-03-28T00:58:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.623+0000 7fde31c0d740 10 client.25765 fetch_fsmap learned FSMap version 25 2024-03-28T00:58:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.623+0000 7fde31c0d740 1 -- 172.21.15.169:0/4121935735 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557478f5abe0 con 0x557478eea310 2024-03-28T00:58:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.626+0000 7fde07fff640 1 -- 172.21.15.169:0/4121935735 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde1c036070 con 0x557478eea310 2024-03-28T00:58:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.627+0000 7fde07fff640 1 -- 172.21.15.169:0/4121935735 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fde1c03bb80 con 0x557478eea310 2024-03-28T00:58:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.627+0000 7fde31c0d740 10 client.25765 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:58:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.627+0000 7fde31c0d740 10 client.25765 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:58:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.627+0000 7fde31c0d740 20 client.25765 populate_metadata read hostname 'smithi169' 2024-03-28T00:58:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.627+0000 7fde31c0d740 1 --2- 172.21.15.169:0/4121935735 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557478f5b400 0x557478f5d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.627+0000 7fde31c0d740 4 client.25765 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T00:58:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.627+0000 7fde31c0d740 1 -- 172.21.15.169:0/4121935735 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557478e563a0 con 0x557478f5b400 2024-03-28T00:58:24.634 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.629+0000 7fde2cce1640 1 --2- 172.21.15.169:0/4121935735 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557478f5b400 0x557478f5d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:24.644 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.641+0000 7fde2cce1640 1 --2- 172.21.15.169:0/4121935735 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557478f5b400 0x557478f5d820 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78dcb52c6083e895 server_cookie=977384e9e46a9089 in_seq=0 out_seq=0 2024-03-28T00:58:24.644 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:24.642+0000 7fde07fff640 10 client.25765 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:25.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:25 smithi169 ceph-mon[32274]: pgmap v408: 129 pgs: 129 active+clean; 7.5 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 656 MiB/s wr, 692 op/s 2024-03-28T00:58:25.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:25 smithi174 ceph-mon[36060]: pgmap v408: 129 pgs: 129 active+clean; 7.5 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 656 MiB/s wr, 692 op/s 2024-03-28T00:58:25.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:25 smithi177 ceph-mon[36206]: pgmap v408: 129 pgs: 129 active+clean; 7.5 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 656 MiB/s wr, 692 op/s 2024-03-28T00:58:27.154 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T00:58:27.154 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T00:58:27.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.477+0000 7fa70cbab640 1 -- 172.21.15.169:0/4099290685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa708109870 msgr2=0x7fa708110100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:27.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.477+0000 7fa70cbab640 1 --2- 172.21.15.169:0/4099290685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa708109870 0x7fa708110100 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fa700009ca0 tx=0x7fa70002c140 comp rx=0 tx=0).stop 2024-03-28T00:58:27.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.479+0000 7fa70cbab640 1 -- 172.21.15.169:0/4099290685 shutdown_connections 2024-03-28T00:58:27.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.479+0000 7fa70cbab640 1 --2- 172.21.15.169:0/4099290685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa708109870 0x7fa708110100 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:27.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.479+0000 7fa70cbab640 1 --2- 172.21.15.169:0/4099290685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa708108eb0 0x7fa708109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:27.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.479+0000 7fa70cbab640 1 --2- 172.21.15.169:0/4099290685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa708107cc0 0x7fa7081080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:27.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.479+0000 7fa70cbab640 1 -- 172.21.15.169:0/4099290685 >> 172.21.15.169:0/4099290685 conn(0x7fa708076c60 msgr2=0x7fa7080790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:27.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.484+0000 7fa70cbab640 1 -- 172.21.15.169:0/4099290685 shutdown_connections 2024-03-28T00:58:27.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.485+0000 7fa70cbab640 1 -- 172.21.15.169:0/4099290685 wait complete. 2024-03-28T00:58:27.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.486+0000 7fa70cbab640 1 Processor -- start 2024-03-28T00:58:27.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.487+0000 7fa70cbab640 1 -- start start 2024-03-28T00:58:27.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.487+0000 7fa70cbab640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa708107cc0 0x7fa708070230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:27.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.487+0000 7fa70cbab640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa708108eb0 0x7fa708070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:27.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.487+0000 7fa70cbab640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa708109870 0x7fa70807de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:27.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.487+0000 7fa70cbab640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa708113a50 con 0x7fa708108eb0 2024-03-28T00:58:27.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.487+0000 7fa70cbab640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7081138d0 con 0x7fa708109870 2024-03-28T00:58:27.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.487+0000 7fa70cbab640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa708113bd0 con 0x7fa708107cc0 2024-03-28T00:58:27.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.490+0000 7fa70659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa708107cc0 0x7fa708070230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:27.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.490+0000 7fa70659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa708107cc0 0x7fa708070230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:44620/0 (socket says 172.21.15.169:44620) 2024-03-28T00:58:27.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.490+0000 7fa70659c640 1 -- 172.21.15.169:0/3315086611 learned_addr learned my addr 172.21.15.169:0/3315086611 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:27.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.490+0000 7fa705d9b640 1 --2- 172.21.15.169:0/3315086611 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa708108eb0 0x7fa708070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:27.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.491+0000 7fa706d9d640 1 --2- 172.21.15.169:0/3315086611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa708109870 0x7fa70807de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:27.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.492+0000 7fa705d9b640 1 -- 172.21.15.169:0/3315086611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa708107cc0 msgr2=0x7fa708070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:27.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.492+0000 7fa705d9b640 1 --2- 172.21.15.169:0/3315086611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa708107cc0 0x7fa708070230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:27.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.493+0000 7fa705d9b640 1 -- 172.21.15.169:0/3315086611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa708109870 msgr2=0x7fa70807de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:27.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.493+0000 7fa705d9b640 1 --2- 172.21.15.169:0/3315086611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa708109870 0x7fa70807de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:27.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.493+0000 7fa705d9b640 1 -- 172.21.15.169:0/3315086611 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa70807e3d0 con 0x7fa708108eb0 2024-03-28T00:58:27.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.497+0000 7fa705d9b640 1 --2- 172.21.15.169:0/3315086611 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa708108eb0 0x7fa708070770 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc009fd0 tx=0x7fa6fc00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:27.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.499+0000 7fa6f77fe640 1 -- 172.21.15.169:0/3315086611 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc015020 con 0x7fa708108eb0 2024-03-28T00:58:27.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.499+0000 7fa70cbab640 1 -- 172.21.15.169:0/3315086611 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa70807e6c0 con 0x7fa708108eb0 2024-03-28T00:58:27.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.499+0000 7fa70cbab640 1 -- 172.21.15.169:0/3315086611 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa70807ec10 con 0x7fa708108eb0 2024-03-28T00:58:27.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.499+0000 7fa6f77fe640 1 -- 172.21.15.169:0/3315086611 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6fc011d10 con 0x7fa708108eb0 2024-03-28T00:58:27.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.499+0000 7fa6f77fe640 1 -- 172.21.15.169:0/3315086611 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc00f2e0 con 0x7fa708108eb0 2024-03-28T00:58:27.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.504+0000 7fa70f78b740 1 Processor -- start 2024-03-28T00:58:27.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.508+0000 7fa6f77fe640 1 -- 172.21.15.169:0/3315086611 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa6fc018020 con 0x7fa708108eb0 2024-03-28T00:58:27.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.509+0000 7fa6f77fe640 1 --2- 172.21.15.169:0/3315086611 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa6f00a2a00 0x7fa6f00a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:27.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.510+0000 7fa6f77fe640 1 -- 172.21.15.169:0/3315086611 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa6fc0cf6c0 con 0x7fa708108eb0 2024-03-28T00:58:27.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.511+0000 7fa706d9d640 1 --2- 172.21.15.169:0/3315086611 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa6f00a2a00 0x7fa6f00a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:27.520 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.517+0000 7fa706d9d640 1 --2- 172.21.15.169:0/3315086611 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa6f00a2a00 0x7fa6f00a4ec0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa700006010 tx=0x7fa700009800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:27.520 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.518+0000 7fa70f78b740 1 -- start start 2024-03-28T00:58:27.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.519+0000 7fa70f78b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a41da0 0x5571d5a421c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:27.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.519+0000 7fa70f78b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571d5a42790 0x5571d5a47290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:27.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.519+0000 7fa70f78b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5571d5a47900 0x5571d5a49d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:27.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.519+0000 7fa70f78b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571d59a73d0 con 0x5571d5a47900 2024-03-28T00:58:27.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.519+0000 7fa70f78b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571d594fad0 con 0x5571d5a42790 2024-03-28T00:58:27.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.519+0000 7fa70f78b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571d59a3470 con 0x5571d5a41da0 2024-03-28T00:58:27.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.520+0000 7fa705d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571d5a42790 0x5571d5a47290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:27.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.520+0000 7fa705d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571d5a42790 0x5571d5a47290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:42046/0 (socket says 172.21.15.169:42046) 2024-03-28T00:58:27.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.520+0000 7fa705d9b640 1 -- 172.21.15.169:0/3403524151 learned_addr learned my addr 172.21.15.169:0/3403524151 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:27.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.523+0000 7fa705d9b640 1 -- 172.21.15.169:0/3403524151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a41da0 msgr2=0x5571d5a421c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T00:58:27.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.523+0000 7fa705d9b640 1 --2- 172.21.15.169:0/3403524151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a41da0 0x5571d5a421c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:27.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.523+0000 7fa705d9b640 1 -- 172.21.15.169:0/3403524151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5571d5a47900 msgr2=0x5571d5a49d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:27.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.523+0000 7fa705d9b640 1 --2- 172.21.15.169:0/3403524151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5571d5a47900 0x5571d5a49d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:27.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.523+0000 7fa705d9b640 1 -- 172.21.15.169:0/3403524151 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5571d57857d0 con 0x5571d5a42790 2024-03-28T00:58:27.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.525+0000 7fa705d9b640 1 --2- 172.21.15.169:0/3403524151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571d5a42790 0x5571d5a47290 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0a3ed0 tx=0x7fa6fc009300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c98695701f62e194 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:27.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.526+0000 7fa6f4ff9640 1 -- 172.21.15.169:0/3403524151 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc09bc60 con 0x5571d5a42790 2024-03-28T00:58:27.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.526+0000 7fa6f4ff9640 1 -- 172.21.15.169:0/3403524151 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6fc09fdb0 con 0x5571d5a42790 2024-03-28T00:58:27.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.526+0000 7fa6f4ff9640 1 -- 172.21.15.169:0/3403524151 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc0982d0 con 0x5571d5a42790 2024-03-28T00:58:27.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.527+0000 7fa70f78b740 1 -- 172.21.15.169:0/3403524151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571d5a42790 msgr2=0x5571d5a47290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:27.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.527+0000 7fa70f78b740 1 --2- 172.21.15.169:0/3403524151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571d5a42790 0x5571d5a47290 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0a3ed0 tx=0x7fa6fc009300 comp rx=0 tx=0).stop 2024-03-28T00:58:27.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.533+0000 7fa70f78b740 1 -- 172.21.15.169:0/3403524151 shutdown_connections 2024-03-28T00:58:27.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.533+0000 7fa70f78b740 1 --2- 172.21.15.169:0/3403524151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5571d5a47900 0x5571d5a49d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:27.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.533+0000 7fa70f78b740 1 --2- 172.21.15.169:0/3403524151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571d5a42790 0x5571d5a47290 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:27.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.533+0000 7fa70f78b740 1 --2- 172.21.15.169:0/3403524151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a41da0 0x5571d5a421c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:27.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.533+0000 7fa70f78b740 1 -- 172.21.15.169:0/3403524151 >> 172.21.15.169:0/3403524151 conn(0x5571d5a3d310 msgr2=0x5571d5a3d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:27.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.543+0000 7fa70f78b740 1 -- 172.21.15.169:0/3403524151 shutdown_connections 2024-03-28T00:58:27.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:27 smithi169 ceph-mon[32274]: pgmap v409: 129 pgs: 129 active+clean; 9.5 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 909 MiB/s wr, 1.26k op/s 2024-03-28T00:58:27.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.550+0000 7fa70f78b740 1 -- 172.21.15.169:0/3403524151 wait complete. 2024-03-28T00:58:27.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.550+0000 7fa70f78b740 1 -- start start 2024-03-28T00:58:27.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.550+0000 7fa70f78b740 1 Processor -- start 2024-03-28T00:58:27.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.551+0000 7fa70f78b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a41da0 0x5571d5a4c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:27.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.551+0000 7fa70f78b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571d5a42790 0x5571d5a4c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:27.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.551+0000 7fa70f78b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5571d5a47900 0x5571d5a537f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:27.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.551+0000 7fa70f78b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571d5a4a7f0 con 0x5571d5a47900 2024-03-28T00:58:27.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.551+0000 7fa70f78b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571d5963060 con 0x5571d5a42790 2024-03-28T00:58:27.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.551+0000 7fa70f78b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571d5a4a970 con 0x5571d5a41da0 2024-03-28T00:58:27.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.551+0000 7fa70f78b740 10 client.0 _ll_register_callbacks cb 0x5571d59456a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:58:27.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.552+0000 7fa70659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a41da0 0x5571d5a4c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:27.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.552+0000 7fa70659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a41da0 0x5571d5a4c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:44634/0 (socket says 172.21.15.169:44634) 2024-03-28T00:58:27.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.552+0000 7fa70659c640 1 -- 172.21.15.169:0/864061792 learned_addr learned my addr 172.21.15.169:0/864061792 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:27.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.552+0000 7fa705d9b640 1 --2- 172.21.15.169:0/864061792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571d5a42790 0x5571d5a4c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:27.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.553+0000 7fa70659c640 1 -- 172.21.15.169:0/864061792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571d5a42790 msgr2=0x5571d5a4c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:27.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.553+0000 7fa70659c640 1 --2- 172.21.15.169:0/864061792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571d5a42790 0x5571d5a4c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:27.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.553+0000 7fa70659c640 1 -- 172.21.15.169:0/864061792 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5571d5a47900 msgr2=0x5571d5a537f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:27.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.553+0000 7fa70659c640 1 --2- 172.21.15.169:0/864061792 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5571d5a47900 0x5571d5a537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:27.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.553+0000 7fa70659c640 1 -- 172.21.15.169:0/864061792 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5571d56bb3f0 con 0x5571d5a41da0 2024-03-28T00:58:27.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.554+0000 7fa70659c640 1 --2- 172.21.15.169:0/864061792 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a41da0 0x5571d5a4c270 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8009fd0 tx=0x7fa6f800d670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=658ee0bff850e7eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:27.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.558+0000 7fa6edffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:58:27.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.558+0000 7fa6edffb640 1 -- 172.21.15.169:0/864061792 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f800de40 con 0x5571d5a41da0 2024-03-28T00:58:27.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.558+0000 7fa6edffb640 1 -- 172.21.15.169:0/864061792 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa6f8012630 con 0x5571d5a41da0 2024-03-28T00:58:27.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.558+0000 7fa70f78b740 1 -- 172.21.15.169:0/864061792 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa6fc09fdb0 con 0x5571d5a41da0 2024-03-28T00:58:27.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.559+0000 7fa6edffb640 1 -- 172.21.15.169:0/864061792 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6f8021050 con 0x5571d5a41da0 2024-03-28T00:58:27.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.559+0000 7fa70f78b740 10 client.25771 fetch_fsmap learned FSMap version 25 2024-03-28T00:58:27.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.559+0000 7fa70f78b740 1 -- 172.21.15.169:0/864061792 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5571d567b0f0 con 0x5571d5a41da0 2024-03-28T00:58:27.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.559+0000 7fa6edffb640 1 -- 172.21.15.169:0/864061792 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6fc09fdb0 con 0x5571d5a41da0 2024-03-28T00:58:27.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.560+0000 7fa6edffb640 1 -- 172.21.15.169:0/864061792 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa6f801c030 con 0x5571d5a41da0 2024-03-28T00:58:27.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.561+0000 7fa70f78b740 10 client.25771 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:58:27.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.561+0000 7fa70f78b740 10 client.25771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:58:27.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.561+0000 7fa70f78b740 20 client.25771 populate_metadata read hostname 'smithi169' 2024-03-28T00:58:27.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.561+0000 7fa70f78b740 1 --2- 172.21.15.169:0/864061792 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5571d5a57df0 0x5571d5a5a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:27.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.561+0000 7fa70f78b740 4 client.25771 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:58:27.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.561+0000 7fa70f78b740 1 -- 172.21.15.169:0/864061792 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5571d59b4910 con 0x5571d5a57df0 2024-03-28T00:58:27.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.562+0000 7fa705d9b640 1 --2- 172.21.15.169:0/864061792 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5571d5a57df0 0x5571d5a5a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:27.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.564+0000 7fa705d9b640 1 --2- 172.21.15.169:0/864061792 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5571d5a57df0 0x5571d5a5a250 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8548a0d8b14f9c57 server_cookie=bdfe02416696fb82 in_seq=0 out_seq=0 2024-03-28T00:58:27.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:27.565+0000 7fa6edffb640 10 client.25771 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:27.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:27 smithi174 ceph-mon[36060]: pgmap v409: 129 pgs: 129 active+clean; 9.5 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 909 MiB/s wr, 1.26k op/s 2024-03-28T00:58:27.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:27 smithi177 ceph-mon[36206]: pgmap v409: 129 pgs: 129 active+clean; 9.5 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 909 MiB/s wr, 1.26k op/s 2024-03-28T00:58:29.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:29 smithi169 ceph-mon[32274]: pgmap v410: 129 pgs: 129 active+clean; 9.8 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 838 MiB/s wr, 1.39k op/s 2024-03-28T00:58:29.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:29 smithi174 ceph-mon[36060]: pgmap v410: 129 pgs: 129 active+clean; 9.8 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 838 MiB/s wr, 1.39k op/s 2024-03-28T00:58:29.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:29 smithi177 ceph-mon[36206]: pgmap v410: 129 pgs: 129 active+clean; 9.8 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 838 MiB/s wr, 1.39k op/s 2024-03-28T00:58:31.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:31 smithi169 ceph-mon[32274]: pgmap v411: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 858 MiB/s wr, 1.40k op/s 2024-03-28T00:58:31.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:31 smithi174 ceph-mon[36060]: pgmap v411: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 858 MiB/s wr, 1.40k op/s 2024-03-28T00:58:31.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:31 smithi177 ceph-mon[36206]: pgmap v411: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 858 MiB/s wr, 1.40k op/s 2024-03-28T00:58:33.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:33 smithi174 ceph-mon[36060]: pgmap v412: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 GiB/s wr, 2.18k op/s 2024-03-28T00:58:33.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:33 smithi177 ceph-mon[36206]: pgmap v412: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 GiB/s wr, 2.18k op/s 2024-03-28T00:58:33.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:33 smithi169 ceph-mon[32274]: pgmap v412: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 GiB/s wr, 2.18k op/s 2024-03-28T00:58:35.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:35 smithi174 ceph-mon[36060]: pgmap v413: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 774 MiB/s wr, 1.90k op/s 2024-03-28T00:58:35.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:35 smithi177 ceph-mon[36206]: pgmap v413: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 774 MiB/s wr, 1.90k op/s 2024-03-28T00:58:35.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:35 smithi169 ceph-mon[32274]: pgmap v413: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 774 MiB/s wr, 1.90k op/s 2024-03-28T00:58:36.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:36 smithi174 ceph-mon[36060]: pgmap v414: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 940 MiB/s wr, 2.10k op/s 2024-03-28T00:58:36.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:36 smithi177 ceph-mon[36206]: pgmap v414: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 940 MiB/s wr, 2.10k op/s 2024-03-28T00:58:36.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:36 smithi169 ceph-mon[32274]: pgmap v414: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 940 MiB/s wr, 2.10k op/s 2024-03-28T00:58:39.559 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:39 smithi169 ceph-mon[32274]: pgmap v415: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 774 MiB/s wr, 1.65k op/s 2024-03-28T00:58:39.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:39 smithi174 ceph-mon[36060]: pgmap v415: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 774 MiB/s wr, 1.65k op/s 2024-03-28T00:58:39.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:39 smithi177 ceph-mon[36206]: pgmap v415: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 774 MiB/s wr, 1.65k op/s 2024-03-28T00:58:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.805+0000 7fde07fff640 1 -- 172.21.15.169:0/4121935735 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557478e563a0 con 0x557478f5b400 2024-03-28T00:58:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.805+0000 7fde07fff640 10 client.25765 handle_command_reply: tid=0 2024-03-28T00:58:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.805+0000 7fde31c0d740 1 client.25765 shutdown 2024-03-28T00:58:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.806+0000 7fde31c0d740 1 -- 172.21.15.169:0/4121935735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557478eea310 msgr2=0x557478ef84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.806+0000 7fde31c0d740 1 --2- 172.21.15.169:0/4121935735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557478eea310 0x557478ef84a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fde1c00fc30 tx=0x7fde1c012970 comp rx=0 tx=0).stop 2024-03-28T00:58:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.807+0000 7fde31c0d740 1 -- 172.21.15.169:0/4121935735 shutdown_connections 2024-03-28T00:58:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.807+0000 7fde31c0d740 1 --2- 172.21.15.169:0/4121935735 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557478f5b400 0x557478f5d820 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.807+0000 7fde31c0d740 1 --2- 172.21.15.169:0/4121935735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557478eef480 0x557478f59580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.807+0000 7fde31c0d740 1 --2- 172.21.15.169:0/4121935735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557478eea310 0x557478ef84a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.807+0000 7fde31c0d740 1 --2- 172.21.15.169:0/4121935735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557478ee9920 0x557478ef7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.807+0000 7fde31c0d740 1 -- 172.21.15.169:0/4121935735 >> 172.21.15.169:0/4121935735 conn(0x557478ee4e90 msgr2=0x557478ee80f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.807+0000 7fde07fff640 0 client.25765 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:40.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.807+0000 7fa6edffb640 1 -- 172.21.15.169:0/864061792 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5571d59b4910 con 0x5571d5a57df0 2024-03-28T00:58:40.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.807+0000 7fa6edffb640 10 client.25771 handle_command_reply: tid=0 2024-03-28T00:58:40.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.807+0000 7fa70f78b740 1 client.25771 shutdown 2024-03-28T00:58:40.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.808+0000 7fa70f78b740 1 -- 172.21.15.169:0/864061792 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a41da0 msgr2=0x5571d5a4c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:40.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.808+0000 7fa70f78b740 1 --2- 172.21.15.169:0/864061792 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a41da0 0x5571d5a4c270 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8009fd0 tx=0x7fa6f800d670 comp rx=0 tx=0).stop 2024-03-28T00:58:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.808+0000 7fde31c0d740 1 -- 172.21.15.169:0/4121935735 shutdown_connections 2024-03-28T00:58:40.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.808+0000 7fa70f78b740 1 -- 172.21.15.169:0/864061792 shutdown_connections 2024-03-28T00:58:40.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.808+0000 7fa70f78b740 1 --2- 172.21.15.169:0/864061792 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5571d5a57df0 0x5571d5a5a250 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.808+0000 7fa70f78b740 1 --2- 172.21.15.169:0/864061792 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5571d5a47900 0x5571d5a537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.808+0000 7fa70f78b740 1 --2- 172.21.15.169:0/864061792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571d5a42790 0x5571d5a4c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.808+0000 7fa70f78b740 1 --2- 172.21.15.169:0/864061792 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a41da0 0x5571d5a4c270 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8009fd0 tx=0x7fa6f800d670 comp rx=0 tx=0).stop 2024-03-28T00:58:40.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.808+0000 7fa70f78b740 1 -- 172.21.15.169:0/864061792 >> 172.21.15.169:0/864061792 conn(0x5571d5a3d310 msgr2=0x5571d5a3c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:40.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.808+0000 7fa6edffb640 0 client.25771 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:40.814 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.808+0000 7fde31c0d740 1 -- 172.21.15.169:0/4121935735 wait complete. 2024-03-28T00:58:40.814 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.809+0000 7fde31c0d740 20 client.25765 trim_cache size 0 max 16384 2024-03-28T00:58:40.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.809+0000 7fa70f78b740 1 -- 172.21.15.169:0/864061792 shutdown_connections 2024-03-28T00:58:40.815 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T00:58:40.815 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T00:58:40.815 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T00:58:40.815 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T00:58:40.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.809+0000 7fa70f78b740 1 -- 172.21.15.169:0/864061792 wait complete. 2024-03-28T00:58:40.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.811+0000 7fa70f78b740 20 client.25771 trim_cache size 0 max 16384 2024-03-28T00:58:40.824 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.821+0000 7fde2ef45640 1 -- 172.21.15.169:0/3860147881 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fde100a2ac0 msgr2=0x7fde100a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:40.824 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.821+0000 7fde2ef45640 1 --2- 172.21.15.169:0/3860147881 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fde100a2ac0 0x7fde100a4f80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x557478eea710 tx=0x7fde1c00b660 comp rx=0 tx=0).stop 2024-03-28T00:58:40.824 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.821+0000 7fde2ef45640 1 -- 172.21.15.169:0/3860147881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde28073560 msgr2=0x7fde2812c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.821+0000 7fde2ef45640 1 --2- 172.21.15.169:0/3860147881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde28073560 0x7fde2812c8e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fde1800e9f0 tx=0x7fde1800eeb0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.821+0000 7fde2ef45640 1 -- 172.21.15.169:0/3860147881 shutdown_connections 2024-03-28T00:58:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.821+0000 7fde2ef45640 1 --2- 172.21.15.169:0/3860147881 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fde100a2ac0 0x7fde100a4f80 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.821+0000 7fde2ef45640 1 --2- 172.21.15.169:0/3860147881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde2812f370 0x7fde28131830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.821+0000 7fde2ef45640 1 --2- 172.21.15.169:0/3860147881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde28076440 0x7fde2812ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.821+0000 7fde2ef45640 1 --2- 172.21.15.169:0/3860147881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde28073560 0x7fde2812c8e0 secure :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fde1800e9f0 tx=0x7fde1800eeb0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.821+0000 7fde2ef45640 1 -- 172.21.15.169:0/3860147881 >> 172.21.15.169:0/3860147881 conn(0x7fde2806e6e0 msgr2=0x7fde2806f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:40.827 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.824+0000 7fde2ef45640 1 -- 172.21.15.169:0/3860147881 shutdown_connections 2024-03-28T00:58:40.827 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:40.824+0000 7fde2ef45640 1 -- 172.21.15.169:0/3860147881 wait complete. 2024-03-28T00:58:40.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.867+0000 7fa70f78b740 1 Processor -- start 2024-03-28T00:58:40.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.869+0000 7fa70f78b740 1 -- start start 2024-03-28T00:58:40.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.869+0000 7fa70f78b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5571d5a42790 0x5571d5a38390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:40.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.869+0000 7fa70f78b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571d5a47900 0x5571d5a537b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:40.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.869+0000 7fa70f78b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a40c80 0x5571d5a4b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:40.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.869+0000 7fa70f78b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e8012090 con 0x5571d5a42790 2024-03-28T00:58:40.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.869+0000 7fa70f78b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e800fb90 con 0x5571d5a47900 2024-03-28T00:58:40.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.869+0000 7fa70f78b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e800dc30 con 0x5571d5a40c80 2024-03-28T00:58:40.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.870+0000 7fa70659c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5571d5a42790 0x5571d5a38390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:40.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.870+0000 7fa70659c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5571d5a42790 0x5571d5a38390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46642/0 (socket says 172.21.15.169:46642) 2024-03-28T00:58:40.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.870+0000 7fa70659c640 1 -- 172.21.15.169:0/4166397206 learned_addr learned my addr 172.21.15.169:0/4166397206 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:40.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.870+0000 7fa70659c640 1 -- 172.21.15.169:0/4166397206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a40c80 msgr2=0x5571d5a4b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:40.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.871+0000 7fa70659c640 1 --2- 172.21.15.169:0/4166397206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a40c80 0x5571d5a4b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.871+0000 7fa70659c640 1 -- 172.21.15.169:0/4166397206 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571d5a47900 msgr2=0x5571d5a537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:40.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.871+0000 7fa70659c640 1 --2- 172.21.15.169:0/4166397206 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571d5a47900 0x5571d5a537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.871+0000 7fa70659c640 1 -- 172.21.15.169:0/4166397206 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5571d5a38b60 con 0x5571d5a42790 2024-03-28T00:58:40.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.872+0000 7fa70659c640 1 --2- 172.21.15.169:0/4166397206 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5571d5a42790 0x5571d5a38390 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa6f800db70 tx=0x7fa6f801e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d0ea6242569b8d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:40.876 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.872+0000 7fa6effff640 1 -- 172.21.15.169:0/4166397206 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f800d030 con 0x5571d5a42790 2024-03-28T00:58:40.876 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.873+0000 7fa6effff640 1 -- 172.21.15.169:0/4166397206 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6f8029030 con 0x5571d5a42790 2024-03-28T00:58:40.877 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.874+0000 7fa70f78b740 1 -- 172.21.15.169:0/4166397206 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5571d5a42790 msgr2=0x5571d5a38390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:40.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.874+0000 7fa70f78b740 1 --2- 172.21.15.169:0/4166397206 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5571d5a42790 0x5571d5a38390 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa6f800db70 tx=0x7fa6f801e000 comp rx=0 tx=0).stop 2024-03-28T00:58:40.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.875+0000 7fa70f78b740 1 -- 172.21.15.169:0/4166397206 shutdown_connections 2024-03-28T00:58:40.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.875+0000 7fa70f78b740 1 --2- 172.21.15.169:0/4166397206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a40c80 0x5571d5a4b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.875+0000 7fa70f78b740 1 --2- 172.21.15.169:0/4166397206 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571d5a47900 0x5571d5a537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.875+0000 7fa70f78b740 1 --2- 172.21.15.169:0/4166397206 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5571d5a42790 0x5571d5a38390 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.875+0000 7fa70f78b740 1 -- 172.21.15.169:0/4166397206 >> 172.21.15.169:0/4166397206 conn(0x5571d5a3d310 msgr2=0x5571d5a40570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:40.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.889+0000 7fa70f78b740 1 -- 172.21.15.169:0/4166397206 shutdown_connections 2024-03-28T00:58:40.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.890+0000 7fa70f78b740 1 -- 172.21.15.169:0/4166397206 wait complete. 2024-03-28T00:58:40.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.890+0000 7fa70f78b740 1 -- start start 2024-03-28T00:58:40.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.890+0000 7fa70f78b740 1 Processor -- start 2024-03-28T00:58:40.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.891+0000 7fa70f78b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a40c80 0x5571d5a511a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:40.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.891+0000 7fa70f78b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571d5a42790 0x5571d5a516e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:40.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.891+0000 7fa70f78b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5571d5a47900 0x5571d5a526f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:40.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.891+0000 7fa70f78b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e800e230 con 0x5571d5a47900 2024-03-28T00:58:40.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.891+0000 7fa70f78b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e800f530 con 0x5571d5a42790 2024-03-28T00:58:40.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.891+0000 7fa70f78b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e800e870 con 0x5571d5a40c80 2024-03-28T00:58:40.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.891+0000 7fa70f78b740 10 client.0 _ll_register_callbacks cb 0x7fa6f80091e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:58:40.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.892+0000 7fa70659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a40c80 0x5571d5a511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:40.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.892+0000 7fa70659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a40c80 0x5571d5a511a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:51668/0 (socket says 172.21.15.169:51668) 2024-03-28T00:58:40.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.892+0000 7fa70659c640 1 -- 172.21.15.169:0/4172300818 learned_addr learned my addr 172.21.15.169:0/4172300818 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:40.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.892+0000 7fa70659c640 1 -- 172.21.15.169:0/4172300818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571d5a42790 msgr2=0x5571d5a516e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T00:58:40.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.892+0000 7fa70659c640 1 --2- 172.21.15.169:0/4172300818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571d5a42790 0x5571d5a516e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.892+0000 7fa70659c640 1 -- 172.21.15.169:0/4172300818 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5571d5a47900 msgr2=0x5571d5a526f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T00:58:40.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.892+0000 7fa70659c640 1 --2- 172.21.15.169:0/4172300818 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5571d5a47900 0x5571d5a526f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.893+0000 7fa70659c640 1 -- 172.21.15.169:0/4172300818 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5571d5a52ce0 con 0x5571d5a40c80 2024-03-28T00:58:40.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.893+0000 7fa70659c640 1 --2- 172.21.15.169:0/4172300818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a40c80 0x5571d5a511a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80144d0 tx=0x7fa6f8006980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81f9a53a51b8c1a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:40.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.899+0000 7fa6edffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:58:40.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.899+0000 7fa6edffb640 1 -- 172.21.15.169:0/4172300818 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f8012a40 con 0x5571d5a40c80 2024-03-28T00:58:40.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.904+0000 7fa6edffb640 1 -- 172.21.15.169:0/4172300818 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa6f8012be0 con 0x5571d5a40c80 2024-03-28T00:58:40.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.904+0000 7fa70f78b740 1 -- 172.21.15.169:0/4172300818 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa6f8029030 con 0x5571d5a40c80 2024-03-28T00:58:40.912 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.910+0000 7fa6edffb640 1 -- 172.21.15.169:0/4172300818 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6f8042050 con 0x5571d5a40c80 2024-03-28T00:58:40.913 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.910+0000 7fa70f78b740 10 client.25774 fetch_fsmap learned FSMap version 25 2024-03-28T00:58:40.913 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.910+0000 7fa70f78b740 1 -- 172.21.15.169:0/4172300818 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5571d5ab39e0 con 0x5571d5a40c80 2024-03-28T00:58:40.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.913+0000 7fa6edffb640 1 -- 172.21.15.169:0/4172300818 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6f8029030 con 0x5571d5a40c80 2024-03-28T00:58:40.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.913+0000 7fa6edffb640 1 -- 172.21.15.169:0/4172300818 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa6f801c050 con 0x5571d5a40c80 2024-03-28T00:58:40.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.913+0000 7fa70f78b740 10 client.25774 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:58:40.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.913+0000 7fa70f78b740 10 client.25774 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:58:40.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.913+0000 7fa70f78b740 20 client.25774 populate_metadata read hostname 'smithi169' 2024-03-28T00:58:40.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.913+0000 7fa70f78b740 1 --2- 172.21.15.169:0/4172300818 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5571d5ab4210 0x5571d5ab6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:40.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.913+0000 7fa70f78b740 4 client.25774 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T00:58:40.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.913+0000 7fa70f78b740 1 -- 172.21.15.169:0/4172300818 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5571d59aec60 con 0x5571d5ab4210 2024-03-28T00:58:40.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.914+0000 7fa705d9b640 1 --2- 172.21.15.169:0/4172300818 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5571d5ab4210 0x5571d5ab6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:40.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.922+0000 7fa6edffb640 10 client.25774 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:40.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.922+0000 7fa705d9b640 1 --2- 172.21.15.169:0/4172300818 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5571d5ab4210 0x5571d5ab6620 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5afd4e0d6c2adc55 server_cookie=671b6b00d5b2723 in_seq=0 out_seq=0 2024-03-28T00:58:40.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.926+0000 7fa6edffb640 1 -- 172.21.15.169:0/4172300818 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5571d59aec60 con 0x5571d5ab4210 2024-03-28T00:58:40.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.926+0000 7fa6edffb640 10 client.25774 handle_command_reply: tid=0 2024-03-28T00:58:40.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.926+0000 7fa70f78b740 1 client.25774 shutdown 2024-03-28T00:58:40.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.932+0000 7fa70f78b740 1 -- 172.21.15.169:0/4172300818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a40c80 msgr2=0x5571d5a511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:40.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.932+0000 7fa70f78b740 1 --2- 172.21.15.169:0/4172300818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a40c80 0x5571d5a511a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80144d0 tx=0x7fa6f8006980 comp rx=0 tx=0).stop 2024-03-28T00:58:40.938 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.936+0000 7fa70f78b740 1 -- 172.21.15.169:0/4172300818 shutdown_connections 2024-03-28T00:58:40.938 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.936+0000 7fa70f78b740 1 --2- 172.21.15.169:0/4172300818 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5571d5ab4210 0x5571d5ab6620 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.936+0000 7fa70f78b740 1 --2- 172.21.15.169:0/4172300818 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5571d5a47900 0x5571d5a526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.936+0000 7fa70f78b740 1 --2- 172.21.15.169:0/4172300818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571d5a42790 0x5571d5a516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.936+0000 7fa70f78b740 1 --2- 172.21.15.169:0/4172300818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5571d5a40c80 0x5571d5a511a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.936+0000 7fa70f78b740 1 -- 172.21.15.169:0/4172300818 >> 172.21.15.169:0/4172300818 conn(0x5571d5a3d310 msgr2=0x5571d5a3c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:40.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.936+0000 7fa6edffb640 0 client.25774 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:40.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.942+0000 7fa70f78b740 1 -- 172.21.15.169:0/4172300818 shutdown_connections 2024-03-28T00:58:40.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.942+0000 7fa70f78b740 1 -- 172.21.15.169:0/4172300818 wait complete. 2024-03-28T00:58:40.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.948+0000 7fa70f78b740 20 client.25774 trim_cache size 0 max 16384 2024-03-28T00:58:40.951 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:58:40.951 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "no active scrubs running", 2024-03-28T00:58:40.951 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": {} 2024-03-28T00:58:40.951 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:58:40.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.958+0000 7fa70cbab640 1 -- 172.21.15.169:0/3315086611 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa6f00a2a00 msgr2=0x7fa6f00a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:40.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.958+0000 7fa70cbab640 1 --2- 172.21.15.169:0/3315086611 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa6f00a2a00 0x7fa6f00a4ec0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa700006010 tx=0x7fa700009800 comp rx=0 tx=0).stop 2024-03-28T00:58:40.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.958+0000 7fa70cbab640 1 -- 172.21.15.169:0/3315086611 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa708108eb0 msgr2=0x7fa708070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:40.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.958+0000 7fa70cbab640 1 --2- 172.21.15.169:0/3315086611 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa708108eb0 0x7fa708070770 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc009fd0 tx=0x7fa6fc00b870 comp rx=0 tx=0).stop 2024-03-28T00:58:40.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.959+0000 7fa70cbab640 1 -- 172.21.15.169:0/3315086611 shutdown_connections 2024-03-28T00:58:40.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.959+0000 7fa70cbab640 1 --2- 172.21.15.169:0/3315086611 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa6f00a2a00 0x7fa6f00a4ec0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.959+0000 7fa70cbab640 1 --2- 172.21.15.169:0/3315086611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa708109870 0x7fa70807de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.959+0000 7fa70cbab640 1 --2- 172.21.15.169:0/3315086611 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa708108eb0 0x7fa708070770 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.959+0000 7fa70cbab640 1 --2- 172.21.15.169:0/3315086611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa708107cc0 0x7fa708070230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:40.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.959+0000 7fa70cbab640 1 -- 172.21.15.169:0/3315086611 >> 172.21.15.169:0/3315086611 conn(0x7fa708076c60 msgr2=0x7fa70810c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:40.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.959+0000 7fa70cbab640 1 -- 172.21.15.169:0/3315086611 shutdown_connections 2024-03-28T00:58:40.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:40.959+0000 7fa70cbab640 1 -- 172.21.15.169:0/3315086611 wait complete. 2024-03-28T00:58:41.023 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T00:58:41.024 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:58:41.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:41 smithi169 ceph-mon[32274]: pgmap v416: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 771 MiB/s wr, 1.58k op/s 2024-03-28T00:58:41.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.389+0000 7fe59acd9640 1 -- 172.21.15.169:0/1855782453 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe594078c00 msgr2=0x7fe59407b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:41.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.389+0000 7fe59acd9640 1 --2- 172.21.15.169:0/1855782453 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe594078c00 0x7fe59407b090 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fe584009900 tx=0x7fe58402bc30 comp rx=0 tx=0).stop 2024-03-28T00:58:41.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.404+0000 7fe59acd9640 1 -- 172.21.15.169:0/1855782453 shutdown_connections 2024-03-28T00:58:41.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.404+0000 7fe59acd9640 1 --2- 172.21.15.169:0/1855782453 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe594078c00 0x7fe59407b090 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:41.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.404+0000 7fe59acd9640 1 --2- 172.21.15.169:0/1855782453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe594076250 0x7fe5940786c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:41.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.404+0000 7fe59acd9640 1 --2- 172.21.15.169:0/1855782453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe594073370 0x7fe594073790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:41.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.404+0000 7fe59acd9640 1 -- 172.21.15.169:0/1855782453 >> 172.21.15.169:0/1855782453 conn(0x7fe59406e3d0 msgr2=0x7fe594070810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:41.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.406+0000 7fe59acd9640 1 -- 172.21.15.169:0/1855782453 shutdown_connections 2024-03-28T00:58:41.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.411+0000 7fe59acd9640 1 -- 172.21.15.169:0/1855782453 wait complete. 2024-03-28T00:58:41.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.412+0000 7fe59acd9640 1 Processor -- start 2024-03-28T00:58:41.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.412+0000 7fe59acd9640 1 -- start start 2024-03-28T00:58:41.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.413+0000 7fe59acd9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe594073370 0x7fe59412c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:41.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.413+0000 7fe59acd9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe594076250 0x7fe59412ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:41.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.413+0000 7fe59acd9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe594078c00 0x7fe5941313a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:41.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.413+0000 7fe59acd9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe594082da0 con 0x7fe594076250 2024-03-28T00:58:41.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.413+0000 7fe59acd9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe594082c20 con 0x7fe594073370 2024-03-28T00:58:41.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.413+0000 7fe59acd9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe594082f20 con 0x7fe594078c00 2024-03-28T00:58:41.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.415+0000 7fe59a4d8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe594078c00 0x7fe5941313a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:41.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.417+0000 7fe59a4d8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe594078c00 0x7fe5941313a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:51676/0 (socket says 172.21.15.169:51676) 2024-03-28T00:58:41.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.417+0000 7fe59a4d8640 1 -- 172.21.15.169:0/3842540113 learned_addr learned my addr 172.21.15.169:0/3842540113 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:41.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.418+0000 7fe599cd7640 1 --2- 172.21.15.169:0/3842540113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe594073370 0x7fe59412c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:41.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.418+0000 7fe5994d6640 1 --2- 172.21.15.169:0/3842540113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe594076250 0x7fe59412ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:41.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.423+0000 7fe59a4d8640 1 -- 172.21.15.169:0/3842540113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe594073370 msgr2=0x7fe59412c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:41.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.423+0000 7fe59a4d8640 1 --2- 172.21.15.169:0/3842540113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe594073370 0x7fe59412c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:41.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.423+0000 7fe59a4d8640 1 -- 172.21.15.169:0/3842540113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe594076250 msgr2=0x7fe59412ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:41.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.423+0000 7fe59a4d8640 1 --2- 172.21.15.169:0/3842540113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe594076250 0x7fe59412ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:41.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.423+0000 7fe59a4d8640 1 -- 172.21.15.169:0/3842540113 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe594131c60 con 0x7fe594078c00 2024-03-28T00:58:41.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.424+0000 7fe599cd7640 1 --2- 172.21.15.169:0/3842540113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe594073370 0x7fe59412c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:58:41.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.424+0000 7fe5994d6640 1 --2- 172.21.15.169:0/3842540113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe594076250 0x7fe59412ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:58:41.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.429+0000 7fe59a4d8640 1 --2- 172.21.15.169:0/3842540113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe594078c00 0x7fe5941313a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe58402c7b0 tx=0x7fe5840045c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:41.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.434+0000 7fe58affd640 1 -- 172.21.15.169:0/3842540113 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe584042020 con 0x7fe594078c00 2024-03-28T00:58:41.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.434+0000 7fe58affd640 1 -- 172.21.15.169:0/3842540113 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe584005410 con 0x7fe594078c00 2024-03-28T00:58:41.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.434+0000 7fe58affd640 1 -- 172.21.15.169:0/3842540113 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58403e7e0 con 0x7fe594078c00 2024-03-28T00:58:41.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.436+0000 7fe59acd9640 1 -- 172.21.15.169:0/3842540113 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe594131ef0 con 0x7fe594078c00 2024-03-28T00:58:41.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.436+0000 7fe59acd9640 1 -- 172.21.15.169:0/3842540113 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe594132360 con 0x7fe594078c00 2024-03-28T00:58:41.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.439+0000 7fe59acd9640 1 -- 172.21.15.169:0/3842540113 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe59406f930 con 0x7fe594078c00 2024-03-28T00:58:41.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.446+0000 7fe58affd640 1 -- 172.21.15.169:0/3842540113 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe584045020 con 0x7fe594078c00 2024-03-28T00:58:41.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.448+0000 7fe58affd640 1 --2- 172.21.15.169:0/3842540113 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe5800a2a00 0x7fe5800a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:41.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.448+0000 7fe58affd640 1 -- 172.21.15.169:0/3842540113 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fe5840f0a80 con 0x7fe594078c00 2024-03-28T00:58:41.454 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.451+0000 7fe599cd7640 1 --2- 172.21.15.169:0/3842540113 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe5800a2a00 0x7fe5800a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:41.461 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.459+0000 7fe599cd7640 1 --2- 172.21.15.169:0/3842540113 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe5800a2a00 0x7fe5800a4ec0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe58c005fd0 tx=0x7fe58c009520 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:41.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:41.459+0000 7fe58affd640 1 -- 172.21.15.169:0/3842540113 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe58404c020 con 0x7fe594078c00 2024-03-28T00:58:41.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:41 smithi174 ceph-mon[36060]: pgmap v416: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 771 MiB/s wr, 1.58k op/s 2024-03-28T00:58:41.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:41 smithi177 ceph-mon[36206]: pgmap v416: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 771 MiB/s wr, 1.58k op/s 2024-03-28T00:58:42.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.083+0000 7fe59acd9640 1 -- 172.21.15.169:0/3842540113 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe59406fb30 con 0x7fe594078c00 2024-03-28T00:58:42.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.086+0000 7fe58affd640 1 -- 172.21.15.169:0/3842540113 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7fe5840bd830 con 0x7fe594078c00 2024-03-28T00:58:42.090 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:58:42.090 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T00:58:42.090 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T00:58:42.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.093+0000 7fe59acd9640 1 -- 172.21.15.169:0/3842540113 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe5800a2a00 msgr2=0x7fe5800a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:42.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.093+0000 7fe59acd9640 1 --2- 172.21.15.169:0/3842540113 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe5800a2a00 0x7fe5800a4ec0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe58c005fd0 tx=0x7fe58c009520 comp rx=0 tx=0).stop 2024-03-28T00:58:42.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.094+0000 7fe59acd9640 1 -- 172.21.15.169:0/3842540113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe594078c00 msgr2=0x7fe5941313a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:42.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.094+0000 7fe59acd9640 1 --2- 172.21.15.169:0/3842540113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe594078c00 0x7fe5941313a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe58402c7b0 tx=0x7fe5840045c0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.098+0000 7fe59acd9640 1 -- 172.21.15.169:0/3842540113 shutdown_connections 2024-03-28T00:58:42.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.098+0000 7fe59acd9640 1 --2- 172.21.15.169:0/3842540113 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe5800a2a00 0x7fe5800a4ec0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.098+0000 7fe59acd9640 1 --2- 172.21.15.169:0/3842540113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe594078c00 0x7fe5941313a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.098+0000 7fe59acd9640 1 --2- 172.21.15.169:0/3842540113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe594076250 0x7fe59412ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.098+0000 7fe59acd9640 1 --2- 172.21.15.169:0/3842540113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe594073370 0x7fe59412c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.098+0000 7fe59acd9640 1 -- 172.21.15.169:0/3842540113 >> 172.21.15.169:0/3842540113 conn(0x7fe59406e3d0 msgr2=0x7fe594076c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:42.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.103+0000 7fe59acd9640 1 -- 172.21.15.169:0/3842540113 shutdown_connections 2024-03-28T00:58:42.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.104+0000 7fe59acd9640 1 -- 172.21.15.169:0/3842540113 wait complete. 2024-03-28T00:58:42.134 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T00:58:42.134 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T00:58:42.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.476+0000 7f5652cd0640 1 -- 172.21.15.169:0/3623474322 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56440a6160 msgr2=0x7f56440a6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:42.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.476+0000 7f5652cd0640 1 --2- 172.21.15.169:0/3623474322 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56440a6160 0x7f56440a6580 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f563c008570 tx=0x7f563c02fbc0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.476+0000 7f5652cd0640 1 -- 172.21.15.169:0/3623474322 shutdown_connections 2024-03-28T00:58:42.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.476+0000 7f5652cd0640 1 --2- 172.21.15.169:0/3623474322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56440a5150 0x7f564400bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.476+0000 7f5652cd0640 1 --2- 172.21.15.169:0/3623474322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56440a4790 0x7f56440a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.476+0000 7f5652cd0640 1 --2- 172.21.15.169:0/3623474322 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56440a6160 0x7f56440a6580 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.476+0000 7f5652cd0640 1 -- 172.21.15.169:0/3623474322 >> 172.21.15.169:0/3623474322 conn(0x7f56440a00d0 msgr2=0x7f56440a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:42.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.482+0000 7f5652cd0640 1 -- 172.21.15.169:0/3623474322 shutdown_connections 2024-03-28T00:58:42.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.483+0000 7f5652cd0640 1 -- 172.21.15.169:0/3623474322 wait complete. 2024-03-28T00:58:42.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.484+0000 7f5652cd0640 1 Processor -- start 2024-03-28T00:58:42.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.484+0000 7f5652cd0640 1 -- start start 2024-03-28T00:58:42.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.484+0000 7f5652cd0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56440a4790 0x7f5644015390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:42.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.484+0000 7f5652cd0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56440a5150 0x7f56440158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:42.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.484+0000 7f5652cd0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56440a6160 0x7f56440139d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:42.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.484+0000 7f5652cd0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564400e2a0 con 0x7f56440a4790 2024-03-28T00:58:42.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.484+0000 7f5652cd0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564400e120 con 0x7f56440a6160 2024-03-28T00:58:42.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.484+0000 7f5652cd0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564400e420 con 0x7f56440a5150 2024-03-28T00:58:42.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.485+0000 7f56524cf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56440a6160 0x7f56440139d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:42.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.485+0000 7f56524cf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56440a6160 0x7f56440139d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:54752/0 (socket says 172.21.15.169:54752) 2024-03-28T00:58:42.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.485+0000 7f56524cf640 1 -- 172.21.15.169:0/539023382 learned_addr learned my addr 172.21.15.169:0/539023382 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:42.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.486+0000 7f56524cf640 1 -- 172.21.15.169:0/539023382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56440a5150 msgr2=0x7f56440158d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:42.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.486+0000 7f56524cf640 1 --2- 172.21.15.169:0/539023382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56440a5150 0x7f56440158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.486+0000 7f56524cf640 1 -- 172.21.15.169:0/539023382 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56440a4790 msgr2=0x7f5644015390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:42.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.486+0000 7f56524cf640 1 --2- 172.21.15.169:0/539023382 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56440a4790 0x7f5644015390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.486+0000 7f56524cf640 1 -- 172.21.15.169:0/539023382 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56440141a0 con 0x7f56440a6160 2024-03-28T00:58:42.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.487+0000 7f56524cf640 1 --2- 172.21.15.169:0/539023382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56440a6160 0x7f56440139d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f564800a910 tx=0x7f564800add0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:42.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.492+0000 7f5642ffd640 1 -- 172.21.15.169:0/539023382 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5648007a70 con 0x7f56440a6160 2024-03-28T00:58:42.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.492+0000 7f5642ffd640 1 -- 172.21.15.169:0/539023382 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5648007c10 con 0x7f56440a6160 2024-03-28T00:58:42.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.493+0000 7f5642ffd640 1 -- 172.21.15.169:0/539023382 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5648014410 con 0x7f56440a6160 2024-03-28T00:58:42.497 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.494+0000 7f5652cd0640 1 -- 172.21.15.169:0/539023382 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5644014490 con 0x7f56440a6160 2024-03-28T00:58:42.497 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.494+0000 7f5652cd0640 1 -- 172.21.15.169:0/539023382 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56440cf650 con 0x7f56440a6160 2024-03-28T00:58:42.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.498+0000 7f5656bf5740 1 Processor -- start 2024-03-28T00:58:42.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.499+0000 7f5656bf5740 1 -- start start 2024-03-28T00:58:42.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.499+0000 7f5656bf5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556677a8b860 0x556677a8bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:42.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.499+0000 7f5656bf5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a8c250 0x556677a90d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:42.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.501+0000 7f5642ffd640 1 -- 172.21.15.169:0/539023382 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f56480035f0 con 0x7f56440a6160 2024-03-28T00:58:42.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.502+0000 7f56514cd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556677a8b860 0x556677a8bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:42.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.502+0000 7f56514cd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556677a8b860 0x556677a8bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46690/0 (socket says 172.21.15.169:46690) 2024-03-28T00:58:42.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.503+0000 7f5642ffd640 1 --2- 172.21.15.169:0/539023382 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f56380a28f0 0x7f56380a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:42.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.504+0000 7f5642ffd640 1 -- 172.21.15.169:0/539023382 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f56480ce350 con 0x7f56440a6160 2024-03-28T00:58:42.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.504+0000 7f5656bf5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556677a913c0 0x556677a93850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:42.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.504+0000 7f5656bf5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x556677a4e610 con 0x556677a8b860 2024-03-28T00:58:42.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.504+0000 7f5656bf5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566779f13c0 con 0x556677a913c0 2024-03-28T00:58:42.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.504+0000 7f5656bf5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x556677999d60 con 0x556677a8c250 2024-03-28T00:58:42.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.505+0000 7f5651cce640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a8c250 0x556677a90d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:42.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.505+0000 7f5651cce640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a8c250 0x556677a90d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:51710/0 (socket says 172.21.15.169:51710) 2024-03-28T00:58:42.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.505+0000 7f5651cce640 1 -- 172.21.15.169:0/1957815140 learned_addr learned my addr 172.21.15.169:0/1957815140 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:42.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.505+0000 7f56524cf640 1 --2- 172.21.15.169:0/1957815140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556677a913c0 0x556677a93850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:42.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.506+0000 7f5651cce640 1 -- 172.21.15.169:0/1957815140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556677a913c0 msgr2=0x556677a93850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:42.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.506+0000 7f5651cce640 1 --2- 172.21.15.169:0/1957815140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556677a913c0 0x556677a93850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.506+0000 7f5651cce640 1 -- 172.21.15.169:0/1957815140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556677a8b860 msgr2=0x556677a8bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:42.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.506+0000 7f5651cce640 1 --2- 172.21.15.169:0/1957815140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556677a8b860 0x556677a8bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.506+0000 7f5651cce640 1 -- 172.21.15.169:0/1957815140 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566777053f0 con 0x556677a8c250 2024-03-28T00:58:42.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.506+0000 7f5651cce640 1 --2- 172.21.15.169:0/1957815140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a8c250 0x556677a90d50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f563c005f30 tx=0x7f563c008280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81eaeb534455cfcc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:42.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.507+0000 7f5637fff640 1 -- 172.21.15.169:0/1957815140 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c00a820 con 0x556677a8c250 2024-03-28T00:58:42.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.507+0000 7f5637fff640 1 -- 172.21.15.169:0/1957815140 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f563c00f290 con 0x556677a8c250 2024-03-28T00:58:42.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.507+0000 7f5637fff640 1 -- 172.21.15.169:0/1957815140 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c0055a0 con 0x556677a8c250 2024-03-28T00:58:42.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.507+0000 7f56514cd640 1 --2- 172.21.15.169:0/539023382 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f56380a28f0 0x7f56380a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:42.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.509+0000 7f5656bf5740 1 -- 172.21.15.169:0/1957815140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a8c250 msgr2=0x556677a90d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:42.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.509+0000 7f5656bf5740 1 --2- 172.21.15.169:0/1957815140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a8c250 0x556677a90d50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f563c005f30 tx=0x7f563c008280 comp rx=0 tx=0).stop 2024-03-28T00:58:42.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.509+0000 7f5656bf5740 1 -- 172.21.15.169:0/1957815140 shutdown_connections 2024-03-28T00:58:42.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.509+0000 7f5656bf5740 1 --2- 172.21.15.169:0/1957815140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556677a913c0 0x556677a93850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.509+0000 7f5656bf5740 1 --2- 172.21.15.169:0/1957815140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a8c250 0x556677a90d50 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.509+0000 7f5656bf5740 1 --2- 172.21.15.169:0/1957815140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556677a8b860 0x556677a8bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.509+0000 7f5656bf5740 1 -- 172.21.15.169:0/1957815140 >> 172.21.15.169:0/1957815140 conn(0x556677a86dd0 msgr2=0x556677a871d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:42.514 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.511+0000 7f5656bf5740 1 -- 172.21.15.169:0/1957815140 shutdown_connections 2024-03-28T00:58:42.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.512+0000 7f5656bf5740 1 -- 172.21.15.169:0/1957815140 wait complete. 2024-03-28T00:58:42.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.512+0000 7f56514cd640 1 --2- 172.21.15.169:0/539023382 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f56380a28f0 0x7f56380a4db0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f564c050d00 tx=0x7f564c06b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:42.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.513+0000 7f5656bf5740 1 -- start start 2024-03-28T00:58:42.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.513+0000 7f5656bf5740 1 Processor -- start 2024-03-28T00:58:42.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.514+0000 7f5656bf5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556677a8b860 0x556677a95b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:42.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.514+0000 7f5656bf5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556677a8c250 0x556677a96090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:42.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.514+0000 7f5656bf5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a913c0 0x556677a9d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:42.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.514+0000 7f5656bf5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566779ad240 con 0x556677a8b860 2024-03-28T00:58:42.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.514+0000 7f5656bf5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566779ed460 con 0x556677a8c250 2024-03-28T00:58:42.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.514+0000 7f5656bf5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x556677a94310 con 0x556677a913c0 2024-03-28T00:58:42.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.514+0000 7f5656bf5740 10 client.0 _ll_register_callbacks cb 0x5566779acc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:58:42.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.514+0000 7f56514cd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556677a8c250 0x556677a96090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:42.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.514+0000 7f56514cd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556677a8c250 0x556677a96090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:54768/0 (socket says 172.21.15.169:54768) 2024-03-28T00:58:42.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.514+0000 7f56514cd640 1 -- 172.21.15.169:0/2939425861 learned_addr learned my addr 172.21.15.169:0/2939425861 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:42.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.514+0000 7f5651cce640 1 --2- 172.21.15.169:0/2939425861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556677a8b860 0x556677a95b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:42.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.515+0000 7f56524cf640 1 --2- 172.21.15.169:0/2939425861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a913c0 0x556677a9d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:42.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.518+0000 7f56514cd640 1 -- 172.21.15.169:0/2939425861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a913c0 msgr2=0x556677a9d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:42.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.518+0000 7f56514cd640 1 --2- 172.21.15.169:0/2939425861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a913c0 0x556677a9d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.518+0000 7f56514cd640 1 -- 172.21.15.169:0/2939425861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556677a8b860 msgr2=0x556677a95b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:42.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.518+0000 7f56514cd640 1 --2- 172.21.15.169:0/2939425861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556677a8b860 0x556677a95b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.518+0000 7f56514cd640 1 -- 172.21.15.169:0/2939425861 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5566777cf7d0 con 0x556677a8c250 2024-03-28T00:58:42.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.518+0000 7f56524cf640 1 --2- 172.21.15.169:0/2939425861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a913c0 0x556677a9d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:58:42.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.518+0000 7f56514cd640 1 --2- 172.21.15.169:0/2939425861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556677a8c250 0x556677a96090 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f564c0674b0 tx=0x7f564c067540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c0064c3cbadcb08 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:42.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.518+0000 7f56357fa640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:58:42.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.519+0000 7f56357fa640 1 -- 172.21.15.169:0/2939425861 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c08a060 con 0x556677a8c250 2024-03-28T00:58:42.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.519+0000 7f56357fa640 1 -- 172.21.15.169:0/2939425861 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f564c079d30 con 0x556677a8c250 2024-03-28T00:58:42.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.520+0000 7f5656bf5740 1 -- 172.21.15.169:0/2939425861 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5566779fe900 con 0x556677a8c250 2024-03-28T00:58:42.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.522+0000 7f56357fa640 1 -- 172.21.15.169:0/2939425861 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f564c07c0d0 con 0x556677a8c250 2024-03-28T00:58:42.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.522+0000 7f5656bf5740 10 client.25805 fetch_fsmap learned FSMap version 25 2024-03-28T00:58:42.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.522+0000 7f5656bf5740 1 -- 172.21.15.169:0/2939425861 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556677aa04b0 con 0x556677a8c250 2024-03-28T00:58:42.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.523+0000 7f56357fa640 1 -- 172.21.15.169:0/2939425861 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5566779fe900 con 0x556677a8c250 2024-03-28T00:58:42.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.524+0000 7f56357fa640 1 -- 172.21.15.169:0/2939425861 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f564c0855b0 con 0x556677a8c250 2024-03-28T00:58:42.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.524+0000 7f5656bf5740 10 client.25805 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:58:42.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.524+0000 7f5656bf5740 10 client.25805 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T00:58:42.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.524+0000 7f5656bf5740 20 client.25805 populate_metadata read hostname 'smithi169' 2024-03-28T00:58:42.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.524+0000 7f5656bf5740 1 --2- 172.21.15.169:0/2939425861 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x556677aa1ec0 0x556677aa4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:42.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.524+0000 7f5656bf5740 4 client.25805 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:58:42.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.524+0000 7f5656bf5740 1 -- 172.21.15.169:0/2939425861 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5566779f8c50 con 0x556677aa1ec0 2024-03-28T00:58:42.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.525+0000 7f5651cce640 1 --2- 172.21.15.169:0/2939425861 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x556677aa1ec0 0x556677aa4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:42.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.525+0000 7f5651cce640 1 --2- 172.21.15.169:0/2939425861 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x556677aa1ec0 0x556677aa4320 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=54ee58dc845fe2f4 server_cookie=3312281d0dc72711 in_seq=0 out_seq=0 2024-03-28T00:58:42.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.526+0000 7f56357fa640 10 client.25805 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T00:58:42.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.526+0000 7f56357fa640 1 -- 172.21.15.169:0/2939425861 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5566779f8c50 con 0x556677aa1ec0 2024-03-28T00:58:42.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.526+0000 7f56357fa640 10 client.25805 handle_command_reply: tid=0 2024-03-28T00:58:42.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.527+0000 7f5656bf5740 1 client.25805 shutdown 2024-03-28T00:58:42.534 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.532+0000 7f5656bf5740 1 -- 172.21.15.169:0/2939425861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556677a8c250 msgr2=0x556677a96090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:42.534 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.532+0000 7f5656bf5740 1 --2- 172.21.15.169:0/2939425861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556677a8c250 0x556677a96090 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f564c0674b0 tx=0x7f564c067540 comp rx=0 tx=0).stop 2024-03-28T00:58:42.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.534+0000 7f5656bf5740 1 -- 172.21.15.169:0/2939425861 shutdown_connections 2024-03-28T00:58:42.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.534+0000 7f5656bf5740 1 --2- 172.21.15.169:0/2939425861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a913c0 0x556677a9d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.534+0000 7f5656bf5740 1 --2- 172.21.15.169:0/2939425861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556677a8c250 0x556677a96090 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.534+0000 7f5656bf5740 1 --2- 172.21.15.169:0/2939425861 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x556677aa1ec0 0x556677aa4320 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.534+0000 7f5656bf5740 1 --2- 172.21.15.169:0/2939425861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556677a8b860 0x556677a95b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.534+0000 7f5656bf5740 1 -- 172.21.15.169:0/2939425861 >> 172.21.15.169:0/2939425861 conn(0x556677a86dd0 msgr2=0x556677a86070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:42.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.535+0000 7f56357fa640 0 client.25805 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T00:58:42.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.539+0000 7f5656bf5740 1 -- 172.21.15.169:0/2939425861 shutdown_connections 2024-03-28T00:58:42.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.540+0000 7f5656bf5740 1 -- 172.21.15.169:0/2939425861 wait complete. 2024-03-28T00:58:42.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.542+0000 7f5656bf5740 20 client.25805 trim_cache size 0 max 16384 2024-03-28T00:58:42.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:42 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3842540113' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:58:42.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.597+0000 7f5656bf5740 1 Processor -- start 2024-03-28T00:58:42.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.598+0000 7f5656bf5740 1 -- start start 2024-03-28T00:58:42.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.599+0000 7f5656bf5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556677a8b860 0x556677a871b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:42.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.600+0000 7f5656bf5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a8c250 0x556677a9d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:42.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.600+0000 7f5656bf5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556677a913c0 0x556677a95b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:42.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.601+0000 7f5656bf5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5644182890 con 0x556677a8b860 2024-03-28T00:58:42.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.601+0000 7f5656bf5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56440d96a0 con 0x556677a913c0 2024-03-28T00:58:42.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.601+0000 7f5656bf5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5644005170 con 0x556677a8c250 2024-03-28T00:58:42.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.601+0000 7f56514cd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a8c250 0x556677a9d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:42.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.601+0000 7f56514cd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a8c250 0x556677a9d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:51728/0 (socket says 172.21.15.169:51728) 2024-03-28T00:58:42.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.601+0000 7f56514cd640 1 -- 172.21.15.169:0/2028416875 learned_addr learned my addr 172.21.15.169:0/2028416875 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:42.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.602+0000 7f5651cce640 1 --2- 172.21.15.169:0/2028416875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556677a8b860 0x556677a871b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:42.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.602+0000 7f56524cf640 1 --2- 172.21.15.169:0/2028416875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556677a913c0 0x556677a95b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:42.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.603+0000 7f56514cd640 1 -- 172.21.15.169:0/2028416875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556677a913c0 msgr2=0x556677a95b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:42.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.603+0000 7f56514cd640 1 --2- 172.21.15.169:0/2028416875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556677a913c0 0x556677a95b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.603+0000 7f56514cd640 1 -- 172.21.15.169:0/2028416875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556677a8b860 msgr2=0x556677a871b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:42.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.603+0000 7f56514cd640 1 --2- 172.21.15.169:0/2028416875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556677a8b860 0x556677a871b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.603+0000 7f56514cd640 1 -- 172.21.15.169:0/2028416875 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556677a90a60 con 0x556677a8c250 2024-03-28T00:58:42.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:42 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3842540113' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:58:42.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.611+0000 7f56514cd640 1 --2- 172.21.15.169:0/2028416875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a8c250 0x556677a9d090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f564c050670 tx=0x7f564c04f9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5b7c208af6bcb41 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:42.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.615+0000 7f56377fe640 1 -- 172.21.15.169:0/2028416875 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c08a070 con 0x556677a8c250 2024-03-28T00:58:42.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.615+0000 7f56377fe640 1 -- 172.21.15.169:0/2028416875 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f564c073040 con 0x556677a8c250 2024-03-28T00:58:42.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.615+0000 7f56377fe640 1 -- 172.21.15.169:0/2028416875 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c079d60 con 0x556677a8c250 2024-03-28T00:58:42.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.618+0000 7f5656bf5740 1 -- 172.21.15.169:0/2028416875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a8c250 msgr2=0x556677a9d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:42.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.618+0000 7f5656bf5740 1 --2- 172.21.15.169:0/2028416875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a8c250 0x556677a9d090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f564c050670 tx=0x7f564c04f9a0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.619+0000 7f5656bf5740 1 -- 172.21.15.169:0/2028416875 shutdown_connections 2024-03-28T00:58:42.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.619+0000 7f5656bf5740 1 --2- 172.21.15.169:0/2028416875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556677a913c0 0x556677a95b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.619+0000 7f5656bf5740 1 --2- 172.21.15.169:0/2028416875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a8c250 0x556677a9d090 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.619+0000 7f5656bf5740 1 --2- 172.21.15.169:0/2028416875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556677a8b860 0x556677a871b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.619+0000 7f5656bf5740 1 -- 172.21.15.169:0/2028416875 >> 172.21.15.169:0/2028416875 conn(0x556677a86dd0 msgr2=0x556677a94630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:42.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.622+0000 7f5656bf5740 1 -- 172.21.15.169:0/2028416875 shutdown_connections 2024-03-28T00:58:42.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.623+0000 7f5656bf5740 1 -- 172.21.15.169:0/2028416875 wait complete. 2024-03-28T00:58:42.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.624+0000 7f5656bf5740 1 -- start start 2024-03-28T00:58:42.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.625+0000 7f5656bf5740 1 Processor -- start 2024-03-28T00:58:42.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.625+0000 7f5656bf5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a8b860 0x556677a99f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:42.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.626+0000 7f5656bf5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556677a8c250 0x556677a9a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:42.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.626+0000 7f5656bf5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556677a913c0 0x556677afb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:42.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.626+0000 7f5656bf5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56440d7d60 con 0x556677a913c0 2024-03-28T00:58:42.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.626+0000 7f5656bf5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56440d9040 con 0x556677a8c250 2024-03-28T00:58:42.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.626+0000 7f5656bf5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56440d8380 con 0x556677a8b860 2024-03-28T00:58:42.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.627+0000 7f5656bf5740 10 client.0 _ll_register_callbacks cb 0x7f564c07cae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:58:42.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.627+0000 7f5651cce640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a8b860 0x556677a99f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:42.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.627+0000 7f5651cce640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a8b860 0x556677a99f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:51740/0 (socket says 172.21.15.169:51740) 2024-03-28T00:58:42.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.627+0000 7f5651cce640 1 -- 172.21.15.169:0/1818028536 learned_addr learned my addr 172.21.15.169:0/1818028536 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:42.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.627+0000 7f56524cf640 1 --2- 172.21.15.169:0/1818028536 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556677a913c0 0x556677afb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:42.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.628+0000 7f5651cce640 1 -- 172.21.15.169:0/1818028536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556677a8c250 msgr2=0x556677a9a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:42.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.628+0000 7f5651cce640 1 --2- 172.21.15.169:0/1818028536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556677a8c250 0x556677a9a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.628+0000 7f5651cce640 1 -- 172.21.15.169:0/1818028536 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556677a913c0 msgr2=0x556677afb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:42.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.628+0000 7f5651cce640 1 --2- 172.21.15.169:0/1818028536 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556677a913c0 0x556677afb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.628+0000 7f5651cce640 1 -- 172.21.15.169:0/1818028536 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556677afbd20 con 0x556677a8b860 2024-03-28T00:58:42.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.633+0000 7f5651cce640 1 --2- 172.21.15.169:0/1818028536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a8b860 0x556677a99f90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f563c005f30 tx=0x7f563c00a690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90971b459ea4d272 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:42.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.634+0000 7f56357fa640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:58:42.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.635+0000 7f56357fa640 1 -- 172.21.15.169:0/1818028536 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c04e080 con 0x556677a8b860 2024-03-28T00:58:42.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.636+0000 7f5656bf5740 1 -- 172.21.15.169:0/1818028536 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f564c073040 con 0x556677a8b860 2024-03-28T00:58:42.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.636+0000 7f56357fa640 1 -- 172.21.15.169:0/1818028536 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f563c04e220 con 0x556677a8b860 2024-03-28T00:58:42.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.639+0000 7f56357fa640 1 -- 172.21.15.169:0/1818028536 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f563c005040 con 0x556677a8b860 2024-03-28T00:58:42.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.639+0000 7f5656bf5740 10 client.25795 fetch_fsmap learned FSMap version 25 2024-03-28T00:58:42.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.639+0000 7f5656bf5740 1 -- 172.21.15.169:0/1818028536 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556677afd0a0 con 0x556677a8b860 2024-03-28T00:58:42.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.647+0000 7f56357fa640 1 -- 172.21.15.169:0/1818028536 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f563c0046c0 con 0x556677a8b860 2024-03-28T00:58:42.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.647+0000 7f56357fa640 1 -- 172.21.15.169:0/1818028536 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f563c04b030 con 0x556677a8b860 2024-03-28T00:58:42.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.648+0000 7f5656bf5740 10 client.25795 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:58:42.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.648+0000 7f5656bf5740 10 client.25795 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T00:58:42.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.648+0000 7f5656bf5740 20 client.25795 populate_metadata read hostname 'smithi169' 2024-03-28T00:58:42.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.648+0000 7f5656bf5740 1 --2- 172.21.15.169:0/1818028536 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x556677afda30 0x556677affe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:42.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.648+0000 7f5656bf5740 4 client.25795 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T00:58:42.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.648+0000 7f5656bf5740 1 -- 172.21.15.169:0/1818028536 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f564c073040 con 0x556677afda30 2024-03-28T00:58:42.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.650+0000 7f56514cd640 1 --2- 172.21.15.169:0/1818028536 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x556677afda30 0x556677affe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:42.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:42 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3842540113' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:58:42.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.682+0000 7f56514cd640 1 --2- 172.21.15.169:0/1818028536 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x556677afda30 0x556677affe50 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ec3098ab242f66d6 server_cookie=9161106cb178a587 in_seq=0 out_seq=0 2024-03-28T00:58:42.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.683+0000 7f56357fa640 10 client.25795 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T00:58:42.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.684+0000 7f56357fa640 1 -- 172.21.15.169:0/1818028536 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f564c073040 con 0x556677afda30 2024-03-28T00:58:42.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.684+0000 7f56357fa640 10 client.25795 handle_command_reply: tid=0 2024-03-28T00:58:42.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.685+0000 7f5656bf5740 1 client.25795 shutdown 2024-03-28T00:58:42.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.699+0000 7f5656bf5740 1 -- 172.21.15.169:0/1818028536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a8b860 msgr2=0x556677a99f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:42.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.699+0000 7f5656bf5740 1 --2- 172.21.15.169:0/1818028536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a8b860 0x556677a99f90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f563c005f30 tx=0x7f563c00a690 comp rx=0 tx=0).stop 2024-03-28T00:58:42.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.705+0000 7f5656bf5740 1 -- 172.21.15.169:0/1818028536 shutdown_connections 2024-03-28T00:58:42.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.705+0000 7f5656bf5740 1 --2- 172.21.15.169:0/1818028536 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x556677afda30 0x556677affe50 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.705+0000 7f5656bf5740 1 --2- 172.21.15.169:0/1818028536 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556677a913c0 0x556677afb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.705+0000 7f5656bf5740 1 --2- 172.21.15.169:0/1818028536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556677a8c250 0x556677a9a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.705+0000 7f5656bf5740 1 --2- 172.21.15.169:0/1818028536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556677a8b860 0x556677a99f90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.705+0000 7f5656bf5740 1 -- 172.21.15.169:0/1818028536 >> 172.21.15.169:0/1818028536 conn(0x556677a86dd0 msgr2=0x556677a871b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:42.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.705+0000 7f56357fa640 0 client.25795 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T00:58:42.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.713+0000 7f5656bf5740 1 -- 172.21.15.169:0/1818028536 shutdown_connections 2024-03-28T00:58:42.720 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.716+0000 7f5656bf5740 1 -- 172.21.15.169:0/1818028536 wait complete. 2024-03-28T00:58:42.720 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.717+0000 7f5656bf5740 20 client.25795 trim_cache size 0 max 16384 2024-03-28T00:58:42.722 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T00:58:42.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.721+0000 7f5652cd0640 1 -- 172.21.15.169:0/539023382 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f56380a28f0 msgr2=0x7f56380a4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:42.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.721+0000 7f5652cd0640 1 --2- 172.21.15.169:0/539023382 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f56380a28f0 0x7f56380a4db0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f564c050d00 tx=0x7f564c06b040 comp rx=0 tx=0).stop 2024-03-28T00:58:42.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.721+0000 7f5652cd0640 1 -- 172.21.15.169:0/539023382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56440a6160 msgr2=0x7f56440139d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:42.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.721+0000 7f5652cd0640 1 --2- 172.21.15.169:0/539023382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56440a6160 0x7f56440139d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f564800a910 tx=0x7f564800add0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.722+0000 7f5652cd0640 1 -- 172.21.15.169:0/539023382 shutdown_connections 2024-03-28T00:58:42.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.722+0000 7f5652cd0640 1 --2- 172.21.15.169:0/539023382 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f56380a28f0 0x7f56380a4db0 secure :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f564c050d00 tx=0x7f564c06b040 comp rx=0 tx=0).stop 2024-03-28T00:58:42.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.722+0000 7f5652cd0640 1 --2- 172.21.15.169:0/539023382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56440a6160 0x7f56440139d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.722+0000 7f5652cd0640 1 --2- 172.21.15.169:0/539023382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56440a5150 0x7f56440158d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.722+0000 7f5652cd0640 1 --2- 172.21.15.169:0/539023382 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56440a4790 0x7f5644015390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:42.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.722+0000 7f5652cd0640 1 -- 172.21.15.169:0/539023382 >> 172.21.15.169:0/539023382 conn(0x7f56440a00d0 msgr2=0x7f56440078a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:42.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.725+0000 7f5652cd0640 1 -- 172.21.15.169:0/539023382 shutdown_connections 2024-03-28T00:58:42.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:42.728+0000 7f5652cd0640 1 -- 172.21.15.169:0/539023382 wait complete. 2024-03-28T00:58:42.788 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T00:58:42.788 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T00:58:43.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.141+0000 7f28cf9cd640 1 -- 172.21.15.169:0/1174573462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28c81096b0 msgr2=0x7f28c810ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:43.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.141+0000 7f28cf9cd640 1 --2- 172.21.15.169:0/1174573462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28c81096b0 0x7f28c810ff40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f28c0009ca0 tx=0x7f28c002c140 comp rx=0 tx=0).stop 2024-03-28T00:58:43.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.144+0000 7f28cf9cd640 1 -- 172.21.15.169:0/1174573462 shutdown_connections 2024-03-28T00:58:43.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.144+0000 7f28cf9cd640 1 --2- 172.21.15.169:0/1174573462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28c81096b0 0x7f28c810ff40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.144+0000 7f28cf9cd640 1 --2- 172.21.15.169:0/1174573462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28c8108cf0 0x7f28c8109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.144+0000 7f28cf9cd640 1 --2- 172.21.15.169:0/1174573462 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28c8107b00 0x7f28c8107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.144+0000 7f28cf9cd640 1 -- 172.21.15.169:0/1174573462 >> 172.21.15.169:0/1174573462 conn(0x7f28c8076aa0 msgr2=0x7f28c8078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:43.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.152+0000 7f28cf9cd640 1 -- 172.21.15.169:0/1174573462 shutdown_connections 2024-03-28T00:58:43.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.152+0000 7f28cf9cd640 1 -- 172.21.15.169:0/1174573462 wait complete. 2024-03-28T00:58:43.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.154+0000 7f28cf9cd640 1 Processor -- start 2024-03-28T00:58:43.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.154+0000 7f28cf9cd640 1 -- start start 2024-03-28T00:58:43.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.156+0000 7f28cf9cd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28c8107b00 0x7f28c8081b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.157+0000 7f28cf9cd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28c8108cf0 0x7f28c8082060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.157+0000 7f28cf9cd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28c81096b0 0x7f28c8080160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.157+0000 7f28cf9cd640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c807a940 con 0x7f28c8108cf0 2024-03-28T00:58:43.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.157+0000 7f28cf9cd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c807a7c0 con 0x7f28c81096b0 2024-03-28T00:58:43.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.157+0000 7f28cf9cd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c807aac0 con 0x7f28c8107b00 2024-03-28T00:58:43.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.158+0000 7f28cd769640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28c8107b00 0x7f28c8081b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.158+0000 7f28cdf6a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28c81096b0 0x7f28c8080160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.158+0000 7f28cd769640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28c8107b00 0x7f28c8081b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:51760/0 (socket says 172.21.15.169:51760) 2024-03-28T00:58:43.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.159+0000 7f28cd769640 1 -- 172.21.15.169:0/2628116575 learned_addr learned my addr 172.21.15.169:0/2628116575 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:43.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.159+0000 7f28ccf68640 1 --2- 172.21.15.169:0/2628116575 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28c8108cf0 0x7f28c8082060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.161+0000 7f28cd769640 1 -- 172.21.15.169:0/2628116575 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28c81096b0 msgr2=0x7f28c8080160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:43.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.161+0000 7f28cd769640 1 --2- 172.21.15.169:0/2628116575 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28c81096b0 0x7f28c8080160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.161+0000 7f28cd769640 1 -- 172.21.15.169:0/2628116575 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28c8108cf0 msgr2=0x7f28c8082060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:43.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.161+0000 7f28cd769640 1 --2- 172.21.15.169:0/2628116575 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28c8108cf0 0x7f28c8082060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.161+0000 7f28cd769640 1 -- 172.21.15.169:0/2628116575 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c80809f0 con 0x7f28c8107b00 2024-03-28T00:58:43.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.164+0000 7f28cd769640 1 --2- 172.21.15.169:0/2628116575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28c8107b00 0x7f28c8081b20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f28c400b480 tx=0x7f28c400b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:43.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.168+0000 7f28be7fc640 1 -- 172.21.15.169:0/2628116575 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c4014070 con 0x7f28c8107b00 2024-03-28T00:58:43.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.168+0000 7f28be7fc640 1 -- 172.21.15.169:0/2628116575 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28c4010d50 con 0x7f28c8107b00 2024-03-28T00:58:43.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.168+0000 7f28be7fc640 1 -- 172.21.15.169:0/2628116575 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c4017360 con 0x7f28c8107b00 2024-03-28T00:58:43.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.168+0000 7f28cf9cd640 1 -- 172.21.15.169:0/2628116575 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c8080ce0 con 0x7f28c8107b00 2024-03-28T00:58:43.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.168+0000 7f28cf9cd640 1 -- 172.21.15.169:0/2628116575 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c806b320 con 0x7f28c8107b00 2024-03-28T00:58:43.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.178+0000 7f28be7fc640 1 -- 172.21.15.169:0/2628116575 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f28c4017070 con 0x7f28c8107b00 2024-03-28T00:58:43.182 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.179+0000 7f28d25ba740 1 Processor -- start 2024-03-28T00:58:43.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.180+0000 7f28d25ba740 1 -- start start 2024-03-28T00:58:43.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.181+0000 7f28d25ba740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b194860 0x561c0b194c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.181+0000 7f28d25ba740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c0b195250 0x561c0b199d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.182+0000 7f28d25ba740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c0b19a3c0 0x561c0b19c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.182+0000 7f28d25ba740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c0b157610 con 0x561c0b19a3c0 2024-03-28T00:58:43.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.182+0000 7f28cdf6a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b194860 0x561c0b194c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.182+0000 7f28cdf6a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b194860 0x561c0b194c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:54810/0 (socket says 172.21.15.169:54810) 2024-03-28T00:58:43.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.182+0000 7f28cdf6a640 1 -- 172.21.15.169:0/2737415140 learned_addr learned my addr 172.21.15.169:0/2737415140 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:43.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.182+0000 7f28d25ba740 1 -- 172.21.15.169:0/2737415140 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c0b0fa3c0 con 0x561c0b194860 2024-03-28T00:58:43.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.182+0000 7f28d25ba740 1 -- 172.21.15.169:0/2737415140 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c0b0a2d60 con 0x561c0b195250 2024-03-28T00:58:43.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.182+0000 7f28be7fc640 1 --2- 172.21.15.169:0/2628116575 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f28b40a2a00 0x7f28b40a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.184+0000 7f28be7fc640 1 -- 172.21.15.169:0/2628116575 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f28c40ce350 con 0x7f28c8107b00 2024-03-28T00:58:43.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.184+0000 7f28cd769640 1 --2- 172.21.15.169:0/2737415140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c0b195250 0x561c0b199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.185+0000 7f28ccf68640 1 --2- 172.21.15.169:0/2737415140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c0b19a3c0 0x561c0b19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.186+0000 7f28cdf6a640 1 --2- 172.21.15.169:0/2628116575 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f28b40a2a00 0x7f28b40a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.187+0000 7f28cdf6a640 1 -- 172.21.15.169:0/2737415140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c0b195250 msgr2=0x561c0b199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:43.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.187+0000 7f28cdf6a640 1 --2- 172.21.15.169:0/2737415140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c0b195250 0x561c0b199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.187+0000 7f28cdf6a640 1 -- 172.21.15.169:0/2737415140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c0b19a3c0 msgr2=0x561c0b19c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:43.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.187+0000 7f28cdf6a640 1 --2- 172.21.15.169:0/2737415140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c0b19a3c0 0x561c0b19c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.187+0000 7f28cdf6a640 1 -- 172.21.15.169:0/2737415140 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c0ae0e3f0 con 0x561c0b194860 2024-03-28T00:58:43.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.192+0000 7f28cdf6a640 1 --2- 172.21.15.169:0/2737415140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b194860 0x561c0b194c80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f28c0032910 tx=0x7f28c0032940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74f65359dbdce63c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:43.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.196+0000 7f28b37fe640 1 -- 172.21.15.169:0/2737415140 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c0039030 con 0x561c0b194860 2024-03-28T00:58:43.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.196+0000 7f28b37fe640 1 -- 172.21.15.169:0/2737415140 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28c0034aa0 con 0x561c0b194860 2024-03-28T00:58:43.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.198+0000 7f28d25ba740 1 -- 172.21.15.169:0/2737415140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b194860 msgr2=0x561c0b194c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:43.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.198+0000 7f28d25ba740 1 --2- 172.21.15.169:0/2737415140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b194860 0x561c0b194c80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f28c0032910 tx=0x7f28c0032940 comp rx=0 tx=0).stop 2024-03-28T00:58:43.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.208+0000 7f28d25ba740 1 -- 172.21.15.169:0/2737415140 shutdown_connections 2024-03-28T00:58:43.214 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.208+0000 7f28d25ba740 1 --2- 172.21.15.169:0/2737415140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c0b19a3c0 0x561c0b19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.214 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.208+0000 7f28d25ba740 1 --2- 172.21.15.169:0/2737415140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c0b195250 0x561c0b199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.214 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.208+0000 7f28d25ba740 1 --2- 172.21.15.169:0/2737415140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b194860 0x561c0b194c80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.214 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.208+0000 7f28d25ba740 1 -- 172.21.15.169:0/2737415140 >> 172.21.15.169:0/2737415140 conn(0x561c0b18fdd0 msgr2=0x561c0b1901d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:43.214 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.208+0000 7f28cdf6a640 1 --2- 172.21.15.169:0/2628116575 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f28b40a2a00 0x7f28b40a4ec0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f28c0032410 tx=0x7f28c0048000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:43.214 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.211+0000 7f28d25ba740 1 -- 172.21.15.169:0/2737415140 shutdown_connections 2024-03-28T00:58:43.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.213+0000 7f28d25ba740 1 -- 172.21.15.169:0/2737415140 wait complete. 2024-03-28T00:58:43.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.213+0000 7f28d25ba740 1 -- start start 2024-03-28T00:58:43.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.213+0000 7f28d25ba740 1 Processor -- start 2024-03-28T00:58:43.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.216+0000 7f28d25ba740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c0b194860 0x561c0b19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.216+0000 7f28d25ba740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b195250 0x561c0b19f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.216+0000 7f28d25ba740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c0b19a3c0 0x561c0b1a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.216+0000 7f28d25ba740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c0b0b6240 con 0x561c0b194860 2024-03-28T00:58:43.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.216+0000 7f28d25ba740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c0b0f6460 con 0x561c0b195250 2024-03-28T00:58:43.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.216+0000 7f28d25ba740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c0b19d360 con 0x561c0b19a3c0 2024-03-28T00:58:43.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.216+0000 7f28d25ba740 10 client.0 _ll_register_callbacks cb 0x561c0b0b5c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:58:43.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.217+0000 7f28ccf68640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b195250 0x561c0b19f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.217+0000 7f28ccf68640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b195250 0x561c0b19f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:54812/0 (socket says 172.21.15.169:54812) 2024-03-28T00:58:43.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.217+0000 7f28ccf68640 1 -- 172.21.15.169:0/2856263523 learned_addr learned my addr 172.21.15.169:0/2856263523 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:43.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.220+0000 7f28cd769640 1 --2- 172.21.15.169:0/2856263523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c0b194860 0x561c0b19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.222+0000 7f28cdf6a640 1 --2- 172.21.15.169:0/2856263523 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c0b19a3c0 0x561c0b1a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.222+0000 7f28ccf68640 1 -- 172.21.15.169:0/2856263523 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c0b19a3c0 msgr2=0x561c0b1a60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:43.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.222+0000 7f28ccf68640 1 --2- 172.21.15.169:0/2856263523 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c0b19a3c0 0x561c0b1a60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.222+0000 7f28ccf68640 1 -- 172.21.15.169:0/2856263523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c0b194860 msgr2=0x561c0b19ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:43.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.222+0000 7f28ccf68640 1 --2- 172.21.15.169:0/2856263523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c0b194860 0x561c0b19ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.222+0000 7f28ccf68640 1 -- 172.21.15.169:0/2856263523 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c0aed87d0 con 0x561c0b195250 2024-03-28T00:58:43.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.223+0000 7f28ccf68640 1 --2- 172.21.15.169:0/2856263523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b195250 0x561c0b19f120 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f28b8011570 tx=0x7f28b8011a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c3184eadd570847 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:43.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.230+0000 7f28b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:58:43.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.230+0000 7f28b0ff9640 1 -- 172.21.15.169:0/2856263523 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b8013730 con 0x561c0b195250 2024-03-28T00:58:43.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.231+0000 7f28d25ba740 1 -- 172.21.15.169:0/2856263523 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f28c0034aa0 con 0x561c0b195250 2024-03-28T00:58:43.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.231+0000 7f28b0ff9640 1 -- 172.21.15.169:0/2856263523 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f28b80138d0 con 0x561c0b195250 2024-03-28T00:58:43.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.232+0000 7f28b0ff9640 1 -- 172.21.15.169:0/2856263523 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28b8026050 con 0x561c0b195250 2024-03-28T00:58:43.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.236+0000 7f28d25ba740 10 client.25814 fetch_fsmap learned FSMap version 25 2024-03-28T00:58:43.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.236+0000 7f28d25ba740 1 -- 172.21.15.169:0/2856263523 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561c0b1a9500 con 0x561c0b195250 2024-03-28T00:58:43.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.239+0000 7f28b0ff9640 1 -- 172.21.15.169:0/2856263523 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28c0034aa0 con 0x561c0b195250 2024-03-28T00:58:43.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.240+0000 7f28b0ff9640 1 -- 172.21.15.169:0/2856263523 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f28b801d400 con 0x561c0b195250 2024-03-28T00:58:43.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.243+0000 7f28d25ba740 10 client.25814 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:58:43.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.243+0000 7f28d25ba740 10 client.25814 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T00:58:43.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.243+0000 7f28d25ba740 20 client.25814 populate_metadata read hostname 'smithi169' 2024-03-28T00:58:43.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.243+0000 7f28d25ba740 1 --2- 172.21.15.169:0/2856263523 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x561c0b1aab60 0x561c0b1acfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.243+0000 7f28d25ba740 4 client.25814 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:58:43.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.243+0000 7f28d25ba740 1 -- 172.21.15.169:0/2856263523 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561c0b107900 con 0x561c0b1aab60 2024-03-28T00:58:43.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.244+0000 7f28cd769640 1 --2- 172.21.15.169:0/2856263523 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x561c0b1aab60 0x561c0b1acfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.249+0000 7f28cd769640 1 --2- 172.21.15.169:0/2856263523 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x561c0b1aab60 0x561c0b1acfe0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1eebc70697007e65 server_cookie=48c712bd6823b487 in_seq=0 out_seq=0 2024-03-28T00:58:43.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.252+0000 7f28b0ff9640 10 client.25814 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T00:58:43.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.252+0000 7f28b0ff9640 1 -- 172.21.15.169:0/2856263523 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561c0b107900 con 0x561c0b1aab60 2024-03-28T00:58:43.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.252+0000 7f28b0ff9640 10 client.25814 handle_command_reply: tid=0 2024-03-28T00:58:43.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.253+0000 7f28d25ba740 1 client.25814 shutdown 2024-03-28T00:58:43.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.254+0000 7f28d25ba740 1 -- 172.21.15.169:0/2856263523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b195250 msgr2=0x561c0b19f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:43.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.254+0000 7f28d25ba740 1 --2- 172.21.15.169:0/2856263523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b195250 0x561c0b19f120 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f28b8011570 tx=0x7f28b8011a30 comp rx=0 tx=0).stop 2024-03-28T00:58:43.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.254+0000 7f28d25ba740 1 -- 172.21.15.169:0/2856263523 shutdown_connections 2024-03-28T00:58:43.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.254+0000 7f28d25ba740 1 --2- 172.21.15.169:0/2856263523 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x561c0b1aab60 0x561c0b1acfe0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.254+0000 7f28d25ba740 1 --2- 172.21.15.169:0/2856263523 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c0b19a3c0 0x561c0b1a60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.254+0000 7f28d25ba740 1 --2- 172.21.15.169:0/2856263523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b195250 0x561c0b19f120 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.254+0000 7f28d25ba740 1 --2- 172.21.15.169:0/2856263523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c0b194860 0x561c0b19ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.254+0000 7f28b0ff9640 0 client.25814 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T00:58:43.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.254+0000 7f28d25ba740 1 -- 172.21.15.169:0/2856263523 >> 172.21.15.169:0/2856263523 conn(0x561c0b18fdd0 msgr2=0x561c0b18f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:43.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.255+0000 7f28d25ba740 1 -- 172.21.15.169:0/2856263523 shutdown_connections 2024-03-28T00:58:43.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.255+0000 7f28d25ba740 1 -- 172.21.15.169:0/2856263523 wait complete. 2024-03-28T00:58:43.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.255+0000 7f28d25ba740 20 client.25814 trim_cache size 0 max 16384 2024-03-28T00:58:43.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.307+0000 7f28d25ba740 1 Processor -- start 2024-03-28T00:58:43.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.308+0000 7f28d25ba740 1 -- start start 2024-03-28T00:58:43.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.308+0000 7f28d25ba740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c0b194860 0x561c0b196750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.308+0000 7f28d25ba740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c0b195250 0x561c0b196c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.308+0000 7f28d25ba740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b19a3c0 0x561c0b1a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.308+0000 7f28d25ba740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ac011f90 con 0x561c0b195250 2024-03-28T00:58:43.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.308+0000 7f28d25ba740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ac00fa90 con 0x561c0b19a3c0 2024-03-28T00:58:43.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.308+0000 7f28d25ba740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ac00dbe0 con 0x561c0b194860 2024-03-28T00:58:43.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.309+0000 7f28cd769640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c0b194860 0x561c0b196750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.309+0000 7f28cd769640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c0b194860 0x561c0b196750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:51792/0 (socket says 172.21.15.169:51792) 2024-03-28T00:58:43.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.309+0000 7f28cd769640 1 -- 172.21.15.169:0/555504964 learned_addr learned my addr 172.21.15.169:0/555504964 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:43.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.309+0000 7f28cdf6a640 1 --2- 172.21.15.169:0/555504964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b19a3c0 0x561c0b1a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.310+0000 7f28cdf6a640 1 -- 172.21.15.169:0/555504964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c0b194860 msgr2=0x561c0b196750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:43.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.310+0000 7f28cdf6a640 1 --2- 172.21.15.169:0/555504964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c0b194860 0x561c0b196750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.310+0000 7f28cdf6a640 1 -- 172.21.15.169:0/555504964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c0b195250 msgr2=0x561c0b196c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:43.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.310+0000 7f28cdf6a640 1 --2- 172.21.15.169:0/555504964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c0b195250 0x561c0b196c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.310+0000 7f28cdf6a640 1 -- 172.21.15.169:0/555504964 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c0b19fa80 con 0x561c0b19a3c0 2024-03-28T00:58:43.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.311+0000 7f28cdf6a640 1 --2- 172.21.15.169:0/555504964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b19a3c0 0x561c0b1a60b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f28c0009bf0 tx=0x7f28c0032940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a97a40170c2506ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:43.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.314+0000 7f28b2ffd640 1 -- 172.21.15.169:0/555504964 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c005b070 con 0x561c0b19a3c0 2024-03-28T00:58:43.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.314+0000 7f28b2ffd640 1 -- 172.21.15.169:0/555504964 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28c0052050 con 0x561c0b19a3c0 2024-03-28T00:58:43.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.315+0000 7f28d25ba740 1 -- 172.21.15.169:0/555504964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b19a3c0 msgr2=0x561c0b1a60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:43.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.315+0000 7f28d25ba740 1 --2- 172.21.15.169:0/555504964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b19a3c0 0x561c0b1a60b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f28c0009bf0 tx=0x7f28c0032940 comp rx=0 tx=0).stop 2024-03-28T00:58:43.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.315+0000 7f28d25ba740 1 -- 172.21.15.169:0/555504964 shutdown_connections 2024-03-28T00:58:43.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.315+0000 7f28d25ba740 1 --2- 172.21.15.169:0/555504964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b19a3c0 0x561c0b1a60b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.315+0000 7f28d25ba740 1 --2- 172.21.15.169:0/555504964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c0b195250 0x561c0b196c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.315+0000 7f28d25ba740 1 --2- 172.21.15.169:0/555504964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c0b194860 0x561c0b196750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.315+0000 7f28d25ba740 1 -- 172.21.15.169:0/555504964 >> 172.21.15.169:0/555504964 conn(0x561c0b18fdd0 msgr2=0x561c0b19ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:43.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.323+0000 7f28d25ba740 1 -- 172.21.15.169:0/555504964 shutdown_connections 2024-03-28T00:58:43.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.323+0000 7f28d25ba740 1 -- 172.21.15.169:0/555504964 wait complete. 2024-03-28T00:58:43.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.323+0000 7f28d25ba740 1 -- start start 2024-03-28T00:58:43.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.324+0000 7f28d25ba740 1 Processor -- start 2024-03-28T00:58:43.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.327+0000 7f28d25ba740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b194860 0x561c0b1a3e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.327+0000 7f28d25ba740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c0b195250 0x561c0b1a4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.327+0000 7f28d25ba740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c0b1a53a0 0x561c0b1a5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.328+0000 7f28d25ba740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ac00e150 con 0x561c0b1a53a0 2024-03-28T00:58:43.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.328+0000 7f28cd769640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b194860 0x561c0b1a3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.328+0000 7f28cd769640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b194860 0x561c0b1a3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:54824/0 (socket says 172.21.15.169:54824) 2024-03-28T00:58:43.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.328+0000 7f28cd769640 1 -- 172.21.15.169:0/3277347227 learned_addr learned my addr 172.21.15.169:0/3277347227 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:43.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.328+0000 7f28d25ba740 1 -- 172.21.15.169:0/3277347227 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ac00f430 con 0x561c0b194860 2024-03-28T00:58:43.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.328+0000 7f28ccf68640 1 --2- 172.21.15.169:0/3277347227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c0b195250 0x561c0b1a4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.328+0000 7f28d25ba740 1 -- 172.21.15.169:0/3277347227 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ac00e770 con 0x561c0b195250 2024-03-28T00:58:43.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.328+0000 7f28d25ba740 10 client.0 _ll_register_callbacks cb 0x7f28b8009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:58:43.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.329+0000 7f28cdf6a640 1 --2- 172.21.15.169:0/3277347227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c0b1a53a0 0x561c0b1a5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.329+0000 7f28cd769640 1 -- 172.21.15.169:0/3277347227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c0b195250 msgr2=0x561c0b1a4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:43.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.329+0000 7f28cd769640 1 --2- 172.21.15.169:0/3277347227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c0b195250 0x561c0b1a4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.329+0000 7f28cd769640 1 -- 172.21.15.169:0/3277347227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c0b1a53a0 msgr2=0x561c0b1a5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:43.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.329+0000 7f28cd769640 1 --2- 172.21.15.169:0/3277347227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c0b1a53a0 0x561c0b1a5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.329+0000 7f28cd769640 1 -- 172.21.15.169:0/3277347227 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c0b1a5d60 con 0x561c0b194860 2024-03-28T00:58:43.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.330+0000 7f28cd769640 1 --2- 172.21.15.169:0/3277347227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b194860 0x561c0b1a3e50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f28c400fa40 tx=0x7f28c40cd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a75f20310efb1a09 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:43.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.331+0000 7f28cdf6a640 1 --2- 172.21.15.169:0/3277347227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c0b1a53a0 0x561c0b1a5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:58:43.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.332+0000 7f28b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:58:43.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.332+0000 7f28b0ff9640 1 -- 172.21.15.169:0/3277347227 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c401e040 con 0x561c0b194860 2024-03-28T00:58:43.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.332+0000 7f28b0ff9640 1 -- 172.21.15.169:0/3277347227 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f28c4010870 con 0x561c0b194860 2024-03-28T00:58:43.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.332+0000 7f28d25ba740 1 -- 172.21.15.169:0/3277347227 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561c0b196260 con 0x561c0b194860 2024-03-28T00:58:43.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.337+0000 7f28b0ff9640 1 -- 172.21.15.169:0/3277347227 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28c40a3090 con 0x561c0b194860 2024-03-28T00:58:43.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.338+0000 7f28d25ba740 10 client.25820 fetch_fsmap learned FSMap version 25 2024-03-28T00:58:43.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.338+0000 7f28d25ba740 1 -- 172.21.15.169:0/3277347227 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561c0b207090 con 0x561c0b194860 2024-03-28T00:58:43.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.342+0000 7f28b0ff9640 1 -- 172.21.15.169:0/3277347227 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x561c0b196260 con 0x561c0b194860 2024-03-28T00:58:43.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.342+0000 7f28b0ff9640 1 -- 172.21.15.169:0/3277347227 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f28c40b0ce0 con 0x561c0b194860 2024-03-28T00:58:43.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.342+0000 7f28d25ba740 10 client.25820 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:58:43.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.342+0000 7f28d25ba740 10 client.25820 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T00:58:43.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.342+0000 7f28d25ba740 20 client.25820 populate_metadata read hostname 'smithi169' 2024-03-28T00:58:43.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.342+0000 7f28d25ba740 1 --2- 172.21.15.169:0/3277347227 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x561c0b2079c0 0x561c0b209de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.342+0000 7f28d25ba740 4 client.25820 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T00:58:43.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.342+0000 7f28d25ba740 1 -- 172.21.15.169:0/3277347227 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561c0b101c50 con 0x561c0b2079c0 2024-03-28T00:58:43.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.344+0000 7f28ccf68640 1 --2- 172.21.15.169:0/3277347227 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x561c0b2079c0 0x561c0b209de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.347+0000 7f28ccf68640 1 --2- 172.21.15.169:0/3277347227 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x561c0b2079c0 0x561c0b209de0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3bb5bbf8c1b98186 server_cookie=1f39cb4619ba3790 in_seq=0 out_seq=0 2024-03-28T00:58:43.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.347+0000 7f28b0ff9640 10 client.25820 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T00:58:43.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.348+0000 7f28b0ff9640 1 -- 172.21.15.169:0/3277347227 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561c0b101c50 con 0x561c0b2079c0 2024-03-28T00:58:43.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.348+0000 7f28b0ff9640 10 client.25820 handle_command_reply: tid=0 2024-03-28T00:58:43.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.351+0000 7f28d25ba740 1 client.25820 shutdown 2024-03-28T00:58:43.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.352+0000 7f28d25ba740 1 -- 172.21.15.169:0/3277347227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b194860 msgr2=0x561c0b1a3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:43.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.352+0000 7f28d25ba740 1 --2- 172.21.15.169:0/3277347227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b194860 0x561c0b1a3e50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f28c400fa40 tx=0x7f28c40cd040 comp rx=0 tx=0).stop 2024-03-28T00:58:43.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.364+0000 7f28d25ba740 1 -- 172.21.15.169:0/3277347227 shutdown_connections 2024-03-28T00:58:43.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.364+0000 7f28d25ba740 1 --2- 172.21.15.169:0/3277347227 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x561c0b2079c0 0x561c0b209de0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.364+0000 7f28d25ba740 1 --2- 172.21.15.169:0/3277347227 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c0b1a53a0 0x561c0b1a5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.364+0000 7f28d25ba740 1 --2- 172.21.15.169:0/3277347227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c0b195250 0x561c0b1a4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.364+0000 7f28d25ba740 1 --2- 172.21.15.169:0/3277347227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c0b194860 0x561c0b1a3e50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.364+0000 7f28b0ff9640 0 client.25820 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T00:58:43.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.364+0000 7f28d25ba740 1 -- 172.21.15.169:0/3277347227 >> 172.21.15.169:0/3277347227 conn(0x561c0b18fdd0 msgr2=0x561c0b19eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:43.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.366+0000 7f28d25ba740 1 -- 172.21.15.169:0/3277347227 shutdown_connections 2024-03-28T00:58:43.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.368+0000 7f28d25ba740 1 -- 172.21.15.169:0/3277347227 wait complete. 2024-03-28T00:58:43.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.371+0000 7f28d25ba740 20 client.25820 trim_cache size 0 max 16384 2024-03-28T00:58:43.376 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T00:58:43.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.381+0000 7f28cf9cd640 1 -- 172.21.15.169:0/2628116575 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f28b40a2a00 msgr2=0x7f28b40a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:43.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.381+0000 7f28cf9cd640 1 --2- 172.21.15.169:0/2628116575 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f28b40a2a00 0x7f28b40a4ec0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f28c0032410 tx=0x7f28c0048000 comp rx=0 tx=0).stop 2024-03-28T00:58:43.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.381+0000 7f28cf9cd640 1 -- 172.21.15.169:0/2628116575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28c8107b00 msgr2=0x7f28c8081b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:43.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.381+0000 7f28cf9cd640 1 --2- 172.21.15.169:0/2628116575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28c8107b00 0x7f28c8081b20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f28c400b480 tx=0x7f28c400b940 comp rx=0 tx=0).stop 2024-03-28T00:58:43.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.384+0000 7f28cf9cd640 1 -- 172.21.15.169:0/2628116575 shutdown_connections 2024-03-28T00:58:43.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.384+0000 7f28cf9cd640 1 --2- 172.21.15.169:0/2628116575 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f28b40a2a00 0x7f28b40a4ec0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.384+0000 7f28cf9cd640 1 --2- 172.21.15.169:0/2628116575 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28c81096b0 0x7f28c8080160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.384+0000 7f28cf9cd640 1 --2- 172.21.15.169:0/2628116575 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28c8108cf0 0x7f28c8082060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.384+0000 7f28cf9cd640 1 --2- 172.21.15.169:0/2628116575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28c8107b00 0x7f28c8081b20 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.384+0000 7f28cf9cd640 1 -- 172.21.15.169:0/2628116575 >> 172.21.15.169:0/2628116575 conn(0x7f28c8076aa0 msgr2=0x7f28c810c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:43.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.384+0000 7f28cf9cd640 1 -- 172.21.15.169:0/2628116575 shutdown_connections 2024-03-28T00:58:43.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.384+0000 7f28cf9cd640 1 -- 172.21.15.169:0/2628116575 wait complete. 2024-03-28T00:58:43.436 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T00:58:43.436 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T00:58:43.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:43 smithi169 ceph-mon[32274]: pgmap v417: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 980 MiB/s wr, 2.10k op/s 2024-03-28T00:58:43.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:43 smithi174 ceph-mon[36060]: pgmap v417: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 980 MiB/s wr, 2.10k op/s 2024-03-28T00:58:43.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:43 smithi177 ceph-mon[36206]: pgmap v417: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 980 MiB/s wr, 2.10k op/s 2024-03-28T00:58:43.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.791+0000 7f96b9ef1640 1 -- 172.21.15.169:0/1549970547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96b41017d0 msgr2=0x7f96b4101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:43.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.791+0000 7f96b9ef1640 1 --2- 172.21.15.169:0/1549970547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96b41017d0 0x7f96b4101c50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f96a80108e0 tx=0x7f96a802fb30 comp rx=0 tx=0).stop 2024-03-28T00:58:43.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.797+0000 7f96b9ef1640 1 -- 172.21.15.169:0/1549970547 shutdown_connections 2024-03-28T00:58:43.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.797+0000 7f96b9ef1640 1 --2- 172.21.15.169:0/1549970547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96b4102190 0x7f96b4108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.797+0000 7f96b9ef1640 1 --2- 172.21.15.169:0/1549970547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96b41017d0 0x7f96b4101c50 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.797+0000 7f96b9ef1640 1 --2- 172.21.15.169:0/1549970547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96b41005e0 0x7f96b4100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.797+0000 7f96b9ef1640 1 -- 172.21.15.169:0/1549970547 >> 172.21.15.169:0/1549970547 conn(0x7f96b40fbb50 msgr2=0x7f96b40fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:43.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.806+0000 7f96b9ef1640 1 -- 172.21.15.169:0/1549970547 shutdown_connections 2024-03-28T00:58:43.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.812+0000 7f96b9ef1640 1 -- 172.21.15.169:0/1549970547 wait complete. 2024-03-28T00:58:43.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.813+0000 7f96b9ef1640 1 Processor -- start 2024-03-28T00:58:43.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.813+0000 7f96b9ef1640 1 -- start start 2024-03-28T00:58:43.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.814+0000 7f96b9ef1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96b41005e0 0x7f96b4070130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.814+0000 7f96b9ef1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96b41017d0 0x7f96b4070670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.814+0000 7f96b9ef1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96b4102190 0x7f96b41a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.814+0000 7f96b9ef1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b410b1c0 con 0x7f96b41005e0 2024-03-28T00:58:43.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.814+0000 7f96b9ef1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b410b040 con 0x7f96b41017d0 2024-03-28T00:58:43.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.814+0000 7f96b9ef1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b410b340 con 0x7f96b4102190 2024-03-28T00:58:43.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.816+0000 7f96b2ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96b41017d0 0x7f96b4070670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.816+0000 7f96b3fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96b4102190 0x7f96b41a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.816+0000 7f96b2ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96b41017d0 0x7f96b4070670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:54846/0 (socket says 172.21.15.169:54846) 2024-03-28T00:58:43.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.816+0000 7f96b2ffd640 1 -- 172.21.15.169:0/2678101848 learned_addr learned my addr 172.21.15.169:0/2678101848 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:43.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.819+0000 7f96b2ffd640 1 -- 172.21.15.169:0/2678101848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96b4102190 msgr2=0x7f96b41a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:43.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.819+0000 7f96b2ffd640 1 --2- 172.21.15.169:0/2678101848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96b4102190 0x7f96b41a58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.819+0000 7f96b2ffd640 1 -- 172.21.15.169:0/2678101848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96b41005e0 msgr2=0x7f96b4070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:43.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.819+0000 7f96b2ffd640 1 --2- 172.21.15.169:0/2678101848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96b41005e0 0x7f96b4070130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.819+0000 7f96b2ffd640 1 -- 172.21.15.169:0/2678101848 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b41a5e10 con 0x7f96b41017d0 2024-03-28T00:58:43.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.821+0000 7f96b2ffd640 1 --2- 172.21.15.169:0/2678101848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96b41017d0 0x7f96b4070670 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f96a8000c00 tx=0x7f96a8031920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:43.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.829+0000 7f96b0ff9640 1 -- 172.21.15.169:0/2678101848 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a8048020 con 0x7f96b41017d0 2024-03-28T00:58:43.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.829+0000 7f96b0ff9640 1 -- 172.21.15.169:0/2678101848 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96a8005630 con 0x7f96b41017d0 2024-03-28T00:58:43.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.829+0000 7f96b0ff9640 1 -- 172.21.15.169:0/2678101848 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a8004b90 con 0x7f96b41017d0 2024-03-28T00:58:43.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.830+0000 7f96b9ef1640 1 -- 172.21.15.169:0/2678101848 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b41a5fe0 con 0x7f96b41017d0 2024-03-28T00:58:43.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.831+0000 7f96b9ef1640 1 -- 172.21.15.169:0/2678101848 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b41a6470 con 0x7f96b41017d0 2024-03-28T00:58:43.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.835+0000 7f96bcbb9740 1 Processor -- start 2024-03-28T00:58:43.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.835+0000 7f96bcbb9740 1 -- start start 2024-03-28T00:58:43.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.835+0000 7f96bcbb9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592ac9a3ea0 0x5592ac9a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.835+0000 7f96bcbb9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a4890 0x5592ac9a9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.835+0000 7f96bcbb9740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a9a00 0x5592ac9abe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.835+0000 7f96bcbb9740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592ac95e610 con 0x5592ac9a9a00 2024-03-28T00:58:43.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.835+0000 7f96bcbb9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592ac9013c0 con 0x5592ac9a4890 2024-03-28T00:58:43.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.835+0000 7f96bcbb9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592ac8a9d60 con 0x5592ac9a3ea0 2024-03-28T00:58:43.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.837+0000 7f96b0ff9640 1 -- 172.21.15.169:0/2678101848 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f96a804b020 con 0x7f96b41017d0 2024-03-28T00:58:43.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.837+0000 7f96b2ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a9a00 0x5592ac9abe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.837+0000 7f96b2ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a9a00 0x5592ac9abe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46796/0 (socket says 172.21.15.169:46796) 2024-03-28T00:58:43.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.837+0000 7f96b2ffd640 1 -- 172.21.15.169:0/1080790463 learned_addr learned my addr 172.21.15.169:0/1080790463 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:43.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.837+0000 7f96b37fe640 1 --2- 172.21.15.169:0/1080790463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a4890 0x5592ac9a9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.839+0000 7f96b0ff9640 1 --2- 172.21.15.169:0/2678101848 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f969c0a2a00 0x7f969c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.839+0000 7f96b2ffd640 1 -- 172.21.15.169:0/1080790463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592ac9a3ea0 msgr2=0x5592ac9a42c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T00:58:43.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.839+0000 7f96b2ffd640 1 --2- 172.21.15.169:0/1080790463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592ac9a3ea0 0x5592ac9a42c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.839+0000 7f96b2ffd640 1 -- 172.21.15.169:0/1080790463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a4890 msgr2=0x5592ac9a9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:43.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.839+0000 7f96b2ffd640 1 --2- 172.21.15.169:0/1080790463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a4890 0x5592ac9a9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.839+0000 7f96b2ffd640 1 -- 172.21.15.169:0/1080790463 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592ac6153f0 con 0x5592ac9a9a00 2024-03-28T00:58:43.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.843+0000 7f96b3fff640 1 --2- 172.21.15.169:0/2678101848 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f969c0a2a00 0x7f969c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.843+0000 7f96b2ffd640 1 --2- 172.21.15.169:0/1080790463 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a9a00 0x5592ac9abe90 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f96a8095c10 tx=0x7f96a80040d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af6660951706eb2d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:43.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.844+0000 7f9699ffb640 1 -- 172.21.15.169:0/1080790463 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a80bcdd0 con 0x5592ac9a9a00 2024-03-28T00:58:43.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.845+0000 7f9699ffb640 1 -- 172.21.15.169:0/1080790463 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96a809e750 con 0x5592ac9a9a00 2024-03-28T00:58:43.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.845+0000 7f9699ffb640 1 -- 172.21.15.169:0/1080790463 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a80f43c0 con 0x5592ac9a9a00 2024-03-28T00:58:43.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.848+0000 7f96bcbb9740 1 -- 172.21.15.169:0/1080790463 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a9a00 msgr2=0x5592ac9abe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:43.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.848+0000 7f96bcbb9740 1 --2- 172.21.15.169:0/1080790463 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a9a00 0x5592ac9abe90 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f96a8095c10 tx=0x7f96a80040d0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.848+0000 7f96b0ff9640 1 -- 172.21.15.169:0/2678101848 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f96a80f0cf0 con 0x7f96b41017d0 2024-03-28T00:58:43.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.850+0000 7f96b3fff640 1 --2- 172.21.15.169:0/2678101848 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f969c0a2a00 0x7f969c0a4ec0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f96a4005dc0 tx=0x7f96a4005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:43.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.855+0000 7f96bcbb9740 1 -- 172.21.15.169:0/1080790463 shutdown_connections 2024-03-28T00:58:43.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.855+0000 7f96bcbb9740 1 --2- 172.21.15.169:0/1080790463 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a9a00 0x5592ac9abe90 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.855+0000 7f96bcbb9740 1 --2- 172.21.15.169:0/1080790463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a4890 0x5592ac9a9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.855+0000 7f96bcbb9740 1 --2- 172.21.15.169:0/1080790463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592ac9a3ea0 0x5592ac9a42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.855+0000 7f96bcbb9740 1 -- 172.21.15.169:0/1080790463 >> 172.21.15.169:0/1080790463 conn(0x5592ac996c00 msgr2=0x5592ac997000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:43.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.856+0000 7f96bcbb9740 1 -- 172.21.15.169:0/1080790463 shutdown_connections 2024-03-28T00:58:43.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.857+0000 7f96bcbb9740 1 -- 172.21.15.169:0/1080790463 wait complete. 2024-03-28T00:58:43.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.858+0000 7f96bcbb9740 1 -- start start 2024-03-28T00:58:43.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.858+0000 7f96bcbb9740 1 Processor -- start 2024-03-28T00:58:43.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.861+0000 7f96bcbb9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592ac9a3ea0 0x5592ac99ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.861+0000 7f96bcbb9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a4890 0x5592ac99cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.861+0000 7f96bcbb9740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a9a00 0x5592ac9a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.861+0000 7f96bcbb9740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592ac8bd240 con 0x5592ac9a9a00 2024-03-28T00:58:43.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.861+0000 7f96bcbb9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592ac8fd460 con 0x5592ac9a4890 2024-03-28T00:58:43.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.861+0000 7f96bcbb9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592ac95e610 con 0x5592ac9a3ea0 2024-03-28T00:58:43.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.861+0000 7f96bcbb9740 10 client.0 _ll_register_callbacks cb 0x5592ac8bcc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:58:43.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.862+0000 7f96b3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a9a00 0x5592ac9a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.862+0000 7f96b3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a9a00 0x5592ac9a1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46802/0 (socket says 172.21.15.169:46802) 2024-03-28T00:58:43.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.862+0000 7f96b3fff640 1 -- 172.21.15.169:0/3192883428 learned_addr learned my addr 172.21.15.169:0/3192883428 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:43.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.863+0000 7f96b2ffd640 1 --2- 172.21.15.169:0/3192883428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a4890 0x5592ac99cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.864+0000 7f96b3fff640 1 -- 172.21.15.169:0/3192883428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592ac9a3ea0 msgr2=0x5592ac99ca10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T00:58:43.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.864+0000 7f96b3fff640 1 --2- 172.21.15.169:0/3192883428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592ac9a3ea0 0x5592ac99ca10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.864+0000 7f96b3fff640 1 -- 172.21.15.169:0/3192883428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a4890 msgr2=0x5592ac99cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:43.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.864+0000 7f96b3fff640 1 --2- 172.21.15.169:0/3192883428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a4890 0x5592ac99cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.864+0000 7f96b3fff640 1 -- 172.21.15.169:0/3192883428 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5592ac6df7d0 con 0x5592ac9a9a00 2024-03-28T00:58:43.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.866+0000 7f96b3fff640 1 --2- 172.21.15.169:0/3192883428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a9a00 0x5592ac9a1f80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f96a4011720 tx=0x7f96a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ebc54fa81ee8a27 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:43.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.869+0000 7f96977f6640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:58:43.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.869+0000 7f96977f6640 1 -- 172.21.15.169:0/3192883428 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a4022020 con 0x5592ac9a9a00 2024-03-28T00:58:43.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.869+0000 7f96977f6640 1 -- 172.21.15.169:0/3192883428 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f96a4010690 con 0x5592ac9a9a00 2024-03-28T00:58:43.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.870+0000 7f96bcbb9740 1 -- 172.21.15.169:0/3192883428 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5592ac90e900 con 0x5592ac9a9a00 2024-03-28T00:58:43.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.872+0000 7f96977f6640 1 -- 172.21.15.169:0/3192883428 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96a4010890 con 0x5592ac9a9a00 2024-03-28T00:58:43.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.873+0000 7f96bcbb9740 10 client.16011 fetch_fsmap learned FSMap version 25 2024-03-28T00:58:43.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.873+0000 7f96bcbb9740 1 -- 172.21.15.169:0/3192883428 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5592ac9aff00 con 0x5592ac9a9a00 2024-03-28T00:58:43.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.876+0000 7f96977f6640 1 -- 172.21.15.169:0/3192883428 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5592ac90e900 con 0x5592ac9a9a00 2024-03-28T00:58:43.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.877+0000 7f96977f6640 1 -- 172.21.15.169:0/3192883428 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f96a402f020 con 0x5592ac9a9a00 2024-03-28T00:58:43.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.878+0000 7f96bcbb9740 10 client.16011 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:58:43.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.878+0000 7f96bcbb9740 10 client.16011 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:58:43.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.878+0000 7f96bcbb9740 20 client.16011 populate_metadata read hostname 'smithi169' 2024-03-28T00:58:43.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.878+0000 7f96bcbb9740 1 --2- 172.21.15.169:0/3192883428 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5592ac9b1a30 0x5592ac9b3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.878+0000 7f96bcbb9740 4 client.16011 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:58:43.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.878+0000 7f96bcbb9740 1 -- 172.21.15.169:0/3192883428 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5592ac908c50 con 0x5592ac9b1a30 2024-03-28T00:58:43.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.894+0000 7f96b37fe640 1 --2- 172.21.15.169:0/3192883428 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5592ac9b1a30 0x5592ac9b3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.921 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.919+0000 7f96b37fe640 1 --2- 172.21.15.169:0/3192883428 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5592ac9b1a30 0x5592ac9b3eb0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68da99b22bd83740 server_cookie=6a2c79e4733cdeb4 in_seq=0 out_seq=0 2024-03-28T00:58:43.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.919+0000 7f96977f6640 10 client.16011 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:43.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.922+0000 7f96977f6640 1 -- 172.21.15.169:0/3192883428 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5592ac908c50 con 0x5592ac9b1a30 2024-03-28T00:58:43.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.922+0000 7f96977f6640 10 client.16011 handle_command_reply: tid=0 2024-03-28T00:58:43.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.925+0000 7f96bcbb9740 1 client.16011 shutdown 2024-03-28T00:58:43.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.927+0000 7f96bcbb9740 1 -- 172.21.15.169:0/3192883428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a9a00 msgr2=0x5592ac9a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:43.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.927+0000 7f96bcbb9740 1 --2- 172.21.15.169:0/3192883428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a9a00 0x5592ac9a1f80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f96a4011720 tx=0x7f96a40046c0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.930+0000 7f96bcbb9740 1 -- 172.21.15.169:0/3192883428 shutdown_connections 2024-03-28T00:58:43.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.930+0000 7f96bcbb9740 1 --2- 172.21.15.169:0/3192883428 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5592ac9b1a30 0x5592ac9b3eb0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.930+0000 7f96bcbb9740 1 --2- 172.21.15.169:0/3192883428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a9a00 0x5592ac9a1f80 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.930+0000 7f96bcbb9740 1 --2- 172.21.15.169:0/3192883428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a4890 0x5592ac99cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.930+0000 7f96bcbb9740 1 --2- 172.21.15.169:0/3192883428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592ac9a3ea0 0x5592ac99ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.930+0000 7f96bcbb9740 1 -- 172.21.15.169:0/3192883428 >> 172.21.15.169:0/3192883428 conn(0x5592ac996c00 msgr2=0x5592ac995e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:43.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.931+0000 7f96977f6640 0 client.16011 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:43.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.938+0000 7f96bcbb9740 1 -- 172.21.15.169:0/3192883428 shutdown_connections 2024-03-28T00:58:43.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.942+0000 7f96bcbb9740 1 -- 172.21.15.169:0/3192883428 wait complete. 2024-03-28T00:58:43.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.942+0000 7f96bcbb9740 20 client.16011 trim_cache size 0 max 16384 2024-03-28T00:58:43.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.974+0000 7f96bcbb9740 1 Processor -- start 2024-03-28T00:58:43.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.974+0000 7f96bcbb9740 1 -- start start 2024-03-28T00:58:43.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.974+0000 7f96bcbb9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a3ea0 0x5592ac99ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.974+0000 7f96bcbb9740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a4890 0x5592ac99cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.974+0000 7f96bcbb9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592ac9a9a00 0x5592ac9a5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.974+0000 7f96bcbb9740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b41b2d90 con 0x5592ac9a4890 2024-03-28T00:58:43.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.974+0000 7f96bcbb9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b41b0890 con 0x5592ac9a3ea0 2024-03-28T00:58:43.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.974+0000 7f96bcbb9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b41ae9e0 con 0x5592ac9a9a00 2024-03-28T00:58:43.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.975+0000 7f96b3fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592ac9a9a00 0x5592ac9a5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.975+0000 7f96b37fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a3ea0 0x5592ac99ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.975+0000 7f96b37fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a3ea0 0x5592ac99ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:54864/0 (socket says 172.21.15.169:54864) 2024-03-28T00:58:43.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.975+0000 7f96b37fe640 1 -- 172.21.15.169:0/2449434132 learned_addr learned my addr 172.21.15.169:0/2449434132 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:43.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.975+0000 7f96b2ffd640 1 --2- 172.21.15.169:0/2449434132 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a4890 0x5592ac99cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.976+0000 7f96b2ffd640 1 -- 172.21.15.169:0/2449434132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592ac9a9a00 msgr2=0x5592ac9a5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:43.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.976+0000 7f96b2ffd640 1 --2- 172.21.15.169:0/2449434132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592ac9a9a00 0x5592ac9a5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.976+0000 7f96b2ffd640 1 -- 172.21.15.169:0/2449434132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a3ea0 msgr2=0x5592ac99ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:43.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.976+0000 7f96b2ffd640 1 --2- 172.21.15.169:0/2449434132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a3ea0 0x5592ac99ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.976+0000 7f96b2ffd640 1 -- 172.21.15.169:0/2449434132 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592ac9b4530 con 0x5592ac9a4890 2024-03-28T00:58:43.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.979+0000 7f96b2ffd640 1 --2- 172.21.15.169:0/2449434132 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a4890 0x5592ac99cf50 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f96a8095c10 tx=0x7f96a8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e8dc8e682d57e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:43.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.981+0000 7f96997fa640 1 -- 172.21.15.169:0/2449434132 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a809dde0 con 0x5592ac9a4890 2024-03-28T00:58:43.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.981+0000 7f96997fa640 1 -- 172.21.15.169:0/2449434132 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96a80f45e0 con 0x5592ac9a4890 2024-03-28T00:58:43.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.981+0000 7f96997fa640 1 -- 172.21.15.169:0/2449434132 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a80bcbf0 con 0x5592ac9a4890 2024-03-28T00:58:43.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.982+0000 7f96bcbb9740 1 -- 172.21.15.169:0/2449434132 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a4890 msgr2=0x5592ac99cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:43.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.982+0000 7f96bcbb9740 1 --2- 172.21.15.169:0/2449434132 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a4890 0x5592ac99cf50 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f96a8095c10 tx=0x7f96a8005f90 comp rx=0 tx=0).stop 2024-03-28T00:58:43.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.982+0000 7f96bcbb9740 1 -- 172.21.15.169:0/2449434132 shutdown_connections 2024-03-28T00:58:43.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.982+0000 7f96bcbb9740 1 --2- 172.21.15.169:0/2449434132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592ac9a9a00 0x5592ac9a5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.982+0000 7f96bcbb9740 1 --2- 172.21.15.169:0/2449434132 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a4890 0x5592ac99cf50 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.982+0000 7f96bcbb9740 1 --2- 172.21.15.169:0/2449434132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a3ea0 0x5592ac99ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.982+0000 7f96bcbb9740 1 -- 172.21.15.169:0/2449434132 >> 172.21.15.169:0/2449434132 conn(0x5592ac996c00 msgr2=0x5592ac996fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:43.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.983+0000 7f96bcbb9740 1 -- 172.21.15.169:0/2449434132 shutdown_connections 2024-03-28T00:58:43.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.983+0000 7f96bcbb9740 1 -- 172.21.15.169:0/2449434132 wait complete. 2024-03-28T00:58:43.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.983+0000 7f96bcbb9740 1 -- start start 2024-03-28T00:58:43.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.983+0000 7f96bcbb9740 1 Processor -- start 2024-03-28T00:58:43.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.984+0000 7f96bcbb9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a3ea0 0x5592ac9c3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.984+0000 7f96bcbb9740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a4890 0x5592ac9c3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.984+0000 7f96bcbb9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592ac9a9a00 0x5592ac9c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.984+0000 7f96bcbb9740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b41aef50 con 0x5592ac9a4890 2024-03-28T00:58:43.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.984+0000 7f96bcbb9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b41b0230 con 0x5592ac9a3ea0 2024-03-28T00:58:43.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.984+0000 7f96bcbb9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b41af570 con 0x5592ac9a9a00 2024-03-28T00:58:43.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.984+0000 7f96bcbb9740 10 client.0 _ll_register_callbacks cb 0x7f96a400ece0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:58:43.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.984+0000 7f96b3fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592ac9a9a00 0x5592ac9c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.984+0000 7f96b3fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592ac9a9a00 0x5592ac9c1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:51832/0 (socket says 172.21.15.169:51832) 2024-03-28T00:58:43.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.984+0000 7f96b3fff640 1 -- 172.21.15.169:0/4234278581 learned_addr learned my addr 172.21.15.169:0/4234278581 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:43.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.984+0000 7f96b37fe640 1 --2- 172.21.15.169:0/4234278581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a3ea0 0x5592ac9c3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.985+0000 7f96b2ffd640 1 --2- 172.21.15.169:0/4234278581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a4890 0x5592ac9c3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:43.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.986+0000 7f96b37fe640 1 -- 172.21.15.169:0/4234278581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592ac9a9a00 msgr2=0x5592ac9c1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:43.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.986+0000 7f96b37fe640 1 --2- 172.21.15.169:0/4234278581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592ac9a9a00 0x5592ac9c1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.986+0000 7f96b37fe640 1 -- 172.21.15.169:0/4234278581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a4890 msgr2=0x5592ac9c3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:43.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.986+0000 7f96b37fe640 1 --2- 172.21.15.169:0/4234278581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a4890 0x5592ac9c3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:43.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.986+0000 7f96b37fe640 1 -- 172.21.15.169:0/4234278581 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5592ac9c2640 con 0x5592ac9a3ea0 2024-03-28T00:58:43.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.987+0000 7f96b37fe640 1 --2- 172.21.15.169:0/4234278581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a3ea0 0x5592ac9c3940 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f96ac000f80 tx=0x7f96ac02aa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3fdf2b55a9d7cb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:43.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.987+0000 7f96977f6640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:58:43.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.987+0000 7f96977f6640 1 -- 172.21.15.169:0/4234278581 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac01ca90 con 0x5592ac9a3ea0 2024-03-28T00:58:43.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.988+0000 7f96bcbb9740 1 -- 172.21.15.169:0/4234278581 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f96a80f45e0 con 0x5592ac9a3ea0 2024-03-28T00:58:43.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.988+0000 7f96977f6640 1 -- 172.21.15.169:0/4234278581 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f96ac01cc30 con 0x5592ac9a3ea0 2024-03-28T00:58:43.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.988+0000 7f96977f6640 1 -- 172.21.15.169:0/4234278581 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96ac03a050 con 0x5592ac9a3ea0 2024-03-28T00:58:43.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.988+0000 7f96bcbb9740 10 client.25838 fetch_fsmap learned FSMap version 25 2024-03-28T00:58:43.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.988+0000 7f96bcbb9740 1 -- 172.21.15.169:0/4234278581 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5592aca0d2e0 con 0x5592ac9a3ea0 2024-03-28T00:58:43.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.989+0000 7f96977f6640 1 -- 172.21.15.169:0/4234278581 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96a80f45e0 con 0x5592ac9a3ea0 2024-03-28T00:58:43.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.990+0000 7f96977f6640 1 -- 172.21.15.169:0/4234278581 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f96ac032380 con 0x5592ac9a3ea0 2024-03-28T00:58:43.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.990+0000 7f96bcbb9740 10 client.25838 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:58:43.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.990+0000 7f96bcbb9740 10 client.25838 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:58:43.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.990+0000 7f96bcbb9740 20 client.25838 populate_metadata read hostname 'smithi169' 2024-03-28T00:58:43.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.990+0000 7f96bcbb9740 1 --2- 172.21.15.169:0/4234278581 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5592aca0dc40 0x5592aca10060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:43.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.990+0000 7f96bcbb9740 4 client.25838 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T00:58:43.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.990+0000 7f96bcbb9740 1 -- 172.21.15.169:0/4234278581 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5592ac886640 con 0x5592aca0dc40 2024-03-28T00:58:43.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.993+0000 7f96b2ffd640 1 --2- 172.21.15.169:0/4234278581 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5592aca0dc40 0x5592aca10060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:44.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.999+0000 7f96977f6640 10 client.25838 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:44.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:43.999+0000 7f96b2ffd640 1 --2- 172.21.15.169:0/4234278581 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5592aca0dc40 0x5592aca10060 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78e5a85077882d9 server_cookie=f5f53f2b983b5380 in_seq=0 out_seq=0 2024-03-28T00:58:44.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.003+0000 7f96977f6640 1 -- 172.21.15.169:0/4234278581 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5592ac886640 con 0x5592aca0dc40 2024-03-28T00:58:44.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.003+0000 7f96977f6640 10 client.25838 handle_command_reply: tid=0 2024-03-28T00:58:44.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.003+0000 7f96bcbb9740 1 client.25838 shutdown 2024-03-28T00:58:44.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.003+0000 7f96bcbb9740 1 -- 172.21.15.169:0/4234278581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a3ea0 msgr2=0x5592ac9c3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:44.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.003+0000 7f96bcbb9740 1 --2- 172.21.15.169:0/4234278581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a3ea0 0x5592ac9c3940 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f96ac000f80 tx=0x7f96ac02aa70 comp rx=0 tx=0).stop 2024-03-28T00:58:44.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.003+0000 7f96bcbb9740 1 -- 172.21.15.169:0/4234278581 shutdown_connections 2024-03-28T00:58:44.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.003+0000 7f96bcbb9740 1 --2- 172.21.15.169:0/4234278581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592ac9a9a00 0x5592ac9c1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:44.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.003+0000 7f96bcbb9740 1 --2- 172.21.15.169:0/4234278581 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5592aca0dc40 0x5592aca10060 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:44.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.003+0000 7f96bcbb9740 1 --2- 172.21.15.169:0/4234278581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5592ac9a4890 0x5592ac9c3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:44.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.003+0000 7f96bcbb9740 1 --2- 172.21.15.169:0/4234278581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592ac9a3ea0 0x5592ac9c3940 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:44.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.003+0000 7f96bcbb9740 1 -- 172.21.15.169:0/4234278581 >> 172.21.15.169:0/4234278581 conn(0x5592ac996c00 msgr2=0x5592ac9a5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:44.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.004+0000 7f96977f6640 0 client.25838 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:44.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.004+0000 7f96bcbb9740 1 -- 172.21.15.169:0/4234278581 shutdown_connections 2024-03-28T00:58:44.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.004+0000 7f96bcbb9740 1 -- 172.21.15.169:0/4234278581 wait complete. 2024-03-28T00:58:44.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.004+0000 7f96bcbb9740 20 client.25838 trim_cache size 0 max 16384 2024-03-28T00:58:44.008 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T00:58:44.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.006+0000 7f96b9ef1640 1 -- 172.21.15.169:0/2678101848 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f969c0a2a00 msgr2=0x7f969c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:44.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.006+0000 7f96b9ef1640 1 --2- 172.21.15.169:0/2678101848 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f969c0a2a00 0x7f969c0a4ec0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f96a4005dc0 tx=0x7f96a4005d50 comp rx=0 tx=0).stop 2024-03-28T00:58:44.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.006+0000 7f96b9ef1640 1 -- 172.21.15.169:0/2678101848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96b41017d0 msgr2=0x7f96b4070670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:44.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.006+0000 7f96b9ef1640 1 --2- 172.21.15.169:0/2678101848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96b41017d0 0x7f96b4070670 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f96a8000c00 tx=0x7f96a8031920 comp rx=0 tx=0).stop 2024-03-28T00:58:44.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.006+0000 7f96b9ef1640 1 -- 172.21.15.169:0/2678101848 shutdown_connections 2024-03-28T00:58:44.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.006+0000 7f96b9ef1640 1 --2- 172.21.15.169:0/2678101848 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f969c0a2a00 0x7f969c0a4ec0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:44.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.006+0000 7f96b9ef1640 1 --2- 172.21.15.169:0/2678101848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96b4102190 0x7f96b41a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:44.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.006+0000 7f96b9ef1640 1 --2- 172.21.15.169:0/2678101848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96b41017d0 0x7f96b4070670 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:44.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.006+0000 7f96b9ef1640 1 --2- 172.21.15.169:0/2678101848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96b41005e0 0x7f96b4070130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:44.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.006+0000 7f96b9ef1640 1 -- 172.21.15.169:0/2678101848 >> 172.21.15.169:0/2678101848 conn(0x7f96b40fbb50 msgr2=0x7f96b4104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:44.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.007+0000 7f96b9ef1640 1 -- 172.21.15.169:0/2678101848 shutdown_connections 2024-03-28T00:58:44.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:44.008+0000 7f96b9ef1640 1 -- 172.21.15.169:0/2678101848 wait complete. 2024-03-28T00:58:45.036 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T00:58:45.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:58:45.036 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T00:58:45.400 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:45 smithi169 ceph-mon[32274]: pgmap v418: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 773 MiB/s wr, 1.50k op/s 2024-03-28T00:58:45.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.398+0000 7f8da4db9640 1 -- 172.21.15.169:0/1804946103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8da0102120 msgr2=0x7f8da0104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:45.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.398+0000 7f8da4db9640 1 --2- 172.21.15.169:0/1804946103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8da0102120 0x7f8da0104510 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8d90009900 tx=0x7f8d9002bc80 comp rx=0 tx=0).stop 2024-03-28T00:58:45.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.398+0000 7f8da4db9640 1 -- 172.21.15.169:0/1804946103 shutdown_connections 2024-03-28T00:58:45.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.398+0000 7f8da4db9640 1 --2- 172.21.15.169:0/1804946103 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8da0107380 0x7f8da01097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.398+0000 7f8da4db9640 1 --2- 172.21.15.169:0/1804946103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8da0104a50 0x7f8da0106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.398+0000 7f8da4db9640 1 --2- 172.21.15.169:0/1804946103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8da0102120 0x7f8da0104510 secure :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8d90009900 tx=0x7f8d9002bc80 comp rx=0 tx=0).stop 2024-03-28T00:58:45.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.398+0000 7f8da4db9640 1 -- 172.21.15.169:0/1804946103 >> 172.21.15.169:0/1804946103 conn(0x7f8da00fbb30 msgr2=0x7f8da00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:45.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.399+0000 7f8da4db9640 1 -- 172.21.15.169:0/1804946103 shutdown_connections 2024-03-28T00:58:45.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.399+0000 7f8da4db9640 1 -- 172.21.15.169:0/1804946103 wait complete. 2024-03-28T00:58:45.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.401+0000 7f8da4db9640 1 Processor -- start 2024-03-28T00:58:45.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.401+0000 7f8da4db9640 1 -- start start 2024-03-28T00:58:45.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.402+0000 7f8da4db9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8da0104a50 0x7f8da0070260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:45.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.402+0000 7f8da4db9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8da0107380 0x7f8da00707a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:45.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.402+0000 7f8da4db9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8da010c710 0x7f8da010cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:45.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.402+0000 7f8da4db9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da010f680 con 0x7f8da0107380 2024-03-28T00:58:45.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.402+0000 7f8da4db9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da010f500 con 0x7f8da010c710 2024-03-28T00:58:45.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.402+0000 7f8da4db9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da010f800 con 0x7f8da0104a50 2024-03-28T00:58:45.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.403+0000 7f8d9e59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8da0104a50 0x7f8da0070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:45.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.403+0000 7f8d9e59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8da0104a50 0x7f8da0070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:51844/0 (socket says 172.21.15.169:51844) 2024-03-28T00:58:45.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.403+0000 7f8d9e59c640 1 -- 172.21.15.169:0/3033851893 learned_addr learned my addr 172.21.15.169:0/3033851893 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:45.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.403+0000 7f8d9dd9b640 1 --2- 172.21.15.169:0/3033851893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8da0107380 0x7f8da00707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:45.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.403+0000 7f8d9ed9d640 1 --2- 172.21.15.169:0/3033851893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8da010c710 0x7f8da010cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:45.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.404+0000 7f8d9e59c640 1 -- 172.21.15.169:0/3033851893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8da010c710 msgr2=0x7f8da010cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:45.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.404+0000 7f8d9e59c640 1 --2- 172.21.15.169:0/3033851893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8da010c710 0x7f8da010cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.404+0000 7f8d9e59c640 1 -- 172.21.15.169:0/3033851893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8da0107380 msgr2=0x7f8da00707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:45.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.404+0000 7f8d9e59c640 1 --2- 172.21.15.169:0/3033851893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8da0107380 0x7f8da00707a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.404+0000 7f8d9e59c640 1 -- 172.21.15.169:0/3033851893 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da010d100 con 0x7f8da0104a50 2024-03-28T00:58:45.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.405+0000 7f8d9dd9b640 1 --2- 172.21.15.169:0/3033851893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8da0107380 0x7f8da00707a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:58:45.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.406+0000 7f8d9e59c640 1 --2- 172.21.15.169:0/3033851893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8da0104a50 0x7f8da0070260 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8d9002c210 tx=0x7f8d90004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:45.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.419+0000 7f8d8f7fe640 1 -- 172.21.15.169:0/3033851893 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d90041070 con 0x7f8da0104a50 2024-03-28T00:58:45.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.419+0000 7f8d8f7fe640 1 -- 172.21.15.169:0/3033851893 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d900054e0 con 0x7f8da0104a50 2024-03-28T00:58:45.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.421+0000 7f8d8f7fe640 1 -- 172.21.15.169:0/3033851893 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d90034440 con 0x7f8da0104a50 2024-03-28T00:58:45.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.421+0000 7f8da4db9640 1 -- 172.21.15.169:0/3033851893 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da010d390 con 0x7f8da0104a50 2024-03-28T00:58:45.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.421+0000 7f8da4db9640 1 -- 172.21.15.169:0/3033851893 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da010d800 con 0x7f8da0104a50 2024-03-28T00:58:45.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.428+0000 7f8da79a4740 1 Processor -- start 2024-03-28T00:58:45.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.430+0000 7f8da79a4740 1 -- start start 2024-03-28T00:58:45.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.430+0000 7f8da79a4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560910140960 0x560910140d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:45.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.430+0000 7f8da79a4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560910141350 0x560910145e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:45.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.430+0000 7f8da79a4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5609101464c0 0x560910148950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:45.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.430+0000 7f8da79a4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x560910080c90 con 0x560910141350 2024-03-28T00:58:45.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.430+0000 7f8da79a4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609100a6070 con 0x5609101464c0 2024-03-28T00:58:45.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.430+0000 7f8da79a4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56091004ec80 con 0x560910140960 2024-03-28T00:58:45.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.431+0000 7f8d8f7fe640 1 -- 172.21.15.169:0/3033851893 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8d90044070 con 0x7f8da0104a50 2024-03-28T00:58:45.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.431+0000 7f8d9ed9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560910140960 0x560910140d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:45.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.431+0000 7f8d9ed9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560910140960 0x560910140d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:51856/0 (socket says 172.21.15.169:51856) 2024-03-28T00:58:45.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.431+0000 7f8d9ed9d640 1 -- 172.21.15.169:0/1681450467 learned_addr learned my addr 172.21.15.169:0/1681450467 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:45.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.432+0000 7f8d8f7fe640 1 --2- 172.21.15.169:0/3033851893 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8d880a28f0 0x7f8d880a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:45.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.432+0000 7f8d9ed9d640 1 -- 172.21.15.169:0/1681450467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5609101464c0 msgr2=0x560910148950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T00:58:45.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.432+0000 7f8d9ed9d640 1 --2- 172.21.15.169:0/1681450467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5609101464c0 0x560910148950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.432+0000 7f8d9ed9d640 1 -- 172.21.15.169:0/1681450467 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560910141350 msgr2=0x560910145e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:45.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.432+0000 7f8d9ed9d640 1 --2- 172.21.15.169:0/1681450467 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560910141350 0x560910145e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.432+0000 7f8d9ed9d640 1 -- 172.21.15.169:0/1681450467 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560910046d10 con 0x560910140960 2024-03-28T00:58:45.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.433+0000 7f8d9ed9d640 1 --2- 172.21.15.169:0/3033851893 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8d880a28f0 0x7f8d880a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:45.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.433+0000 7f8d8f7fe640 1 -- 172.21.15.169:0/3033851893 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f8d900f04c0 con 0x7f8da0104a50 2024-03-28T00:58:45.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.434+0000 7f8d9ed9d640 1 --2- 172.21.15.169:0/1681450467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560910140960 0x560910140d80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8d9800c570 tx=0x7f8d98009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c00609d718cb329b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:45.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.435+0000 7f8d9ed9d640 1 --2- 172.21.15.169:0/3033851893 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8d880a28f0 0x7f8d880a4db0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8d98016000 tx=0x7f8d98018000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:45.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.441+0000 7f8d8cff9640 1 -- 172.21.15.169:0/1681450467 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d98018640 con 0x560910140960 2024-03-28T00:58:45.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.441+0000 7f8d8cff9640 1 -- 172.21.15.169:0/1681450467 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d98018cc0 con 0x560910140960 2024-03-28T00:58:45.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.442+0000 7f8da79a4740 1 -- 172.21.15.169:0/1681450467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560910140960 msgr2=0x560910140d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:45.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.442+0000 7f8da79a4740 1 --2- 172.21.15.169:0/1681450467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560910140960 0x560910140d80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8d9800c570 tx=0x7f8d98009f90 comp rx=0 tx=0).stop 2024-03-28T00:58:45.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.446+0000 7f8da79a4740 1 -- 172.21.15.169:0/1681450467 shutdown_connections 2024-03-28T00:58:45.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.446+0000 7f8da79a4740 1 --2- 172.21.15.169:0/1681450467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5609101464c0 0x560910148950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.446+0000 7f8da79a4740 1 --2- 172.21.15.169:0/1681450467 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560910141350 0x560910145e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.446+0000 7f8da79a4740 1 --2- 172.21.15.169:0/1681450467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560910140960 0x560910140d80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.446+0000 7f8da79a4740 1 -- 172.21.15.169:0/1681450467 >> 172.21.15.169:0/1681450467 conn(0x5609101091a0 msgr2=0x56091013c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:45.454 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.451+0000 7f8da79a4740 1 -- 172.21.15.169:0/1681450467 shutdown_connections 2024-03-28T00:58:45.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.453+0000 7f8da79a4740 1 -- 172.21.15.169:0/1681450467 wait complete. 2024-03-28T00:58:45.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.453+0000 7f8da79a4740 1 -- start start 2024-03-28T00:58:45.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.453+0000 7f8da79a4740 1 Processor -- start 2024-03-28T00:58:45.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.454+0000 7f8da79a4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560910140960 0x56091014afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:45.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.454+0000 7f8da79a4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560910141350 0x56091014b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:45.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.454+0000 7f8da79a4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609101464c0 0x560910152560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:45.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.454+0000 7f8da79a4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609100620d0 con 0x5609101464c0 2024-03-28T00:58:45.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.454+0000 7f8da79a4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609100a2110 con 0x560910140960 2024-03-28T00:58:45.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.454+0000 7f8da79a4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609101494c0 con 0x560910141350 2024-03-28T00:58:45.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.454+0000 7f8da79a4740 10 client.0 _ll_register_callbacks cb 0x56091009de60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:58:45.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.455+0000 7f8d9ed9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609101464c0 0x560910152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:45.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.455+0000 7f8d9ed9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609101464c0 0x560910152560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46866/0 (socket says 172.21.15.169:46866) 2024-03-28T00:58:45.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.455+0000 7f8d9ed9d640 1 -- 172.21.15.169:0/4192722140 learned_addr learned my addr 172.21.15.169:0/4192722140 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:45.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.456+0000 7f8d9e59c640 1 --2- 172.21.15.169:0/4192722140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560910140960 0x56091014afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:45.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.456+0000 7f8d9ed9d640 1 -- 172.21.15.169:0/4192722140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560910141350 msgr2=0x56091014b520 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T00:58:45.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.456+0000 7f8d9ed9d640 1 --2- 172.21.15.169:0/4192722140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560910141350 0x56091014b520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.456+0000 7f8d9ed9d640 1 -- 172.21.15.169:0/4192722140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560910140960 msgr2=0x56091014afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:45.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.456+0000 7f8d9ed9d640 1 --2- 172.21.15.169:0/4192722140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560910140960 0x56091014afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.456+0000 7f8d9ed9d640 1 -- 172.21.15.169:0/4192722140 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56090fe827e0 con 0x5609101464c0 2024-03-28T00:58:45.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.459+0000 7f8d9ed9d640 1 --2- 172.21.15.169:0/4192722140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609101464c0 0x560910152560 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8d980098f0 tx=0x7f8d98018670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=387e2cb265590058 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:45.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.461+0000 7f8d85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:58:45.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.465+0000 7f8d85ffb640 1 -- 172.21.15.169:0/4192722140 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d98018c70 con 0x5609101464c0 2024-03-28T00:58:45.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.466+0000 7f8da79a4740 1 -- 172.21.15.169:0/4192722140 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5609100b35b0 con 0x5609101464c0 2024-03-28T00:58:45.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.466+0000 7f8d85ffb640 1 -- 172.21.15.169:0/4192722140 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f8d98029430 con 0x5609101464c0 2024-03-28T00:58:45.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.467+0000 7f8d85ffb640 1 -- 172.21.15.169:0/4192722140 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d98009c60 con 0x5609101464c0 2024-03-28T00:58:45.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.468+0000 7f8da79a4740 10 client.16023 fetch_fsmap learned FSMap version 25 2024-03-28T00:58:45.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.468+0000 7f8da79a4740 1 -- 172.21.15.169:0/4192722140 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56090fee2f70 con 0x5609101464c0 2024-03-28T00:58:45.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.471+0000 7f8d85ffb640 1 -- 172.21.15.169:0/4192722140 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5609100b35b0 con 0x5609101464c0 2024-03-28T00:58:45.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.471+0000 7f8d85ffb640 1 -- 172.21.15.169:0/4192722140 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f8d98028070 con 0x5609101464c0 2024-03-28T00:58:45.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.472+0000 7f8da79a4740 10 client.16023 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:58:45.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.472+0000 7f8da79a4740 10 client.16023 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:58:45.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.472+0000 7f8da79a4740 20 client.16023 populate_metadata read hostname 'smithi169' 2024-03-28T00:58:45.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.472+0000 7f8da79a4740 1 --2- 172.21.15.169:0/4192722140 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560910167cd0 0x56091016a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:45.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.472+0000 7f8da79a4740 4 client.16023 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:58:45.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.472+0000 7f8da79a4740 1 -- 172.21.15.169:0/4192722140 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5609100ad900 con 0x560910167cd0 2024-03-28T00:58:45.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.473+0000 7f8d9e59c640 1 --2- 172.21.15.169:0/4192722140 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560910167cd0 0x56091016a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:45.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.478+0000 7f8d9e59c640 1 --2- 172.21.15.169:0/4192722140 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560910167cd0 0x56091016a0c0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7570a18e95775a9 server_cookie=89375cad7e6feaf2 in_seq=0 out_seq=0 2024-03-28T00:58:45.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.481+0000 7f8d85ffb640 10 client.16023 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:45.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.484+0000 7f8d85ffb640 1 -- 172.21.15.169:0/4192722140 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5609100ad900 con 0x560910167cd0 2024-03-28T00:58:45.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.484+0000 7f8d85ffb640 10 client.16023 handle_command_reply: tid=0 2024-03-28T00:58:45.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.484+0000 7f8da79a4740 1 client.16023 shutdown 2024-03-28T00:58:45.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.491+0000 7f8da79a4740 1 -- 172.21.15.169:0/4192722140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609101464c0 msgr2=0x560910152560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:45.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.491+0000 7f8da79a4740 1 --2- 172.21.15.169:0/4192722140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609101464c0 0x560910152560 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8d980098f0 tx=0x7f8d98018670 comp rx=0 tx=0).stop 2024-03-28T00:58:45.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.492+0000 7f8da79a4740 1 -- 172.21.15.169:0/4192722140 shutdown_connections 2024-03-28T00:58:45.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.492+0000 7f8da79a4740 1 --2- 172.21.15.169:0/4192722140 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560910167cd0 0x56091016a0c0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.492+0000 7f8da79a4740 1 --2- 172.21.15.169:0/4192722140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5609101464c0 0x560910152560 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.492+0000 7f8da79a4740 1 --2- 172.21.15.169:0/4192722140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560910141350 0x56091014b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.492+0000 7f8da79a4740 1 --2- 172.21.15.169:0/4192722140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560910140960 0x56091014afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.492+0000 7f8da79a4740 1 -- 172.21.15.169:0/4192722140 >> 172.21.15.169:0/4192722140 conn(0x5609101091a0 msgr2=0x56091013b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:45.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.493+0000 7f8d85ffb640 0 client.16023 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:45.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.493+0000 7f8da79a4740 1 -- 172.21.15.169:0/4192722140 shutdown_connections 2024-03-28T00:58:45.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.493+0000 7f8da79a4740 1 -- 172.21.15.169:0/4192722140 wait complete. 2024-03-28T00:58:45.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.493+0000 7f8da79a4740 20 client.16023 trim_cache size 0 max 16384 2024-03-28T00:58:45.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.533+0000 7f8da79a4740 1 Processor -- start 2024-03-28T00:58:45.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.534+0000 7f8da79a4740 1 -- start start 2024-03-28T00:58:45.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.534+0000 7f8da79a4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560910140960 0x560910149c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:45.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.534+0000 7f8da79a4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560910141350 0x560910152520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:45.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.534+0000 7f8da79a4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5609101464c0 0x56091013f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:45.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.534+0000 7f8da79a4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da01b32d0 con 0x560910141350 2024-03-28T00:58:45.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.534+0000 7f8da79a4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da01b0dd0 con 0x5609101464c0 2024-03-28T00:58:45.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.534+0000 7f8da79a4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da01ace50 con 0x560910140960 2024-03-28T00:58:45.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.535+0000 7f8d9dd9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560910141350 0x560910152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:45.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.535+0000 7f8d9dd9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560910141350 0x560910152520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46868/0 (socket says 172.21.15.169:46868) 2024-03-28T00:58:45.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.535+0000 7f8d9dd9b640 1 -- 172.21.15.169:0/2437113434 learned_addr learned my addr 172.21.15.169:0/2437113434 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:45.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.536+0000 7f8d9e59c640 1 --2- 172.21.15.169:0/2437113434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560910140960 0x560910149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:45.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.536+0000 7f8d9ed9d640 1 --2- 172.21.15.169:0/2437113434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5609101464c0 0x56091013f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:45.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.539+0000 7f8d9dd9b640 1 -- 172.21.15.169:0/2437113434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560910140960 msgr2=0x560910149c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:45.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.539+0000 7f8d9dd9b640 1 --2- 172.21.15.169:0/2437113434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560910140960 0x560910149c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.539+0000 7f8d9dd9b640 1 -- 172.21.15.169:0/2437113434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5609101464c0 msgr2=0x56091013f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:45.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.539+0000 7f8d9dd9b640 1 --2- 172.21.15.169:0/2437113434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5609101464c0 0x56091013f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.539+0000 7f8d9dd9b640 1 -- 172.21.15.169:0/2437113434 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56091014add0 con 0x560910141350 2024-03-28T00:58:45.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.541+0000 7f8d9dd9b640 1 --2- 172.21.15.169:0/2437113434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560910141350 0x560910152520 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400a6c0 tx=0x7f8d9400ab80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=584347bd712a1669 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:45.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.541+0000 7f8d87fff640 1 -- 172.21.15.169:0/2437113434 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d94015070 con 0x560910141350 2024-03-28T00:58:45.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.542+0000 7f8d87fff640 1 -- 172.21.15.169:0/2437113434 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d9400ce10 con 0x560910141350 2024-03-28T00:58:45.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.542+0000 7f8d87fff640 1 -- 172.21.15.169:0/2437113434 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d940183c0 con 0x560910141350 2024-03-28T00:58:45.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.544+0000 7f8da79a4740 1 -- 172.21.15.169:0/2437113434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560910141350 msgr2=0x560910152520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:45.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.544+0000 7f8da79a4740 1 --2- 172.21.15.169:0/2437113434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560910141350 0x560910152520 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400a6c0 tx=0x7f8d9400ab80 comp rx=0 tx=0).stop 2024-03-28T00:58:45.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.548+0000 7f8da79a4740 1 -- 172.21.15.169:0/2437113434 shutdown_connections 2024-03-28T00:58:45.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.548+0000 7f8da79a4740 1 --2- 172.21.15.169:0/2437113434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5609101464c0 0x56091013f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.548+0000 7f8da79a4740 1 --2- 172.21.15.169:0/2437113434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560910141350 0x560910152520 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.548+0000 7f8da79a4740 1 --2- 172.21.15.169:0/2437113434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560910140960 0x560910149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.548+0000 7f8da79a4740 1 -- 172.21.15.169:0/2437113434 >> 172.21.15.169:0/2437113434 conn(0x5609101091a0 msgr2=0x560910149510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:45.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.554+0000 7f8da79a4740 1 -- 172.21.15.169:0/2437113434 shutdown_connections 2024-03-28T00:58:45.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.554+0000 7f8da79a4740 1 -- 172.21.15.169:0/2437113434 wait complete. 2024-03-28T00:58:45.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.554+0000 7f8da79a4740 1 -- start start 2024-03-28T00:58:45.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.555+0000 7f8da79a4740 1 Processor -- start 2024-03-28T00:58:45.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.555+0000 7f8da79a4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560910140960 0x560910150050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:45.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.556+0000 7f8da79a4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560910141350 0x560910150590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:45.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.556+0000 7f8da79a4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609101464c0 0x5609101515a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:45.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.556+0000 7f8da79a4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da01ad450 con 0x560910140960 2024-03-28T00:58:45.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.556+0000 7f8da79a4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da01b0770 con 0x560910141350 2024-03-28T00:58:45.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.556+0000 7f8da79a4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da01ada90 con 0x5609101464c0 2024-03-28T00:58:45.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.556+0000 7f8da79a4740 10 client.0 _ll_register_callbacks cb 0x7f8d9800c420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:58:45.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.559+0000 7f8d9ed9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609101464c0 0x5609101515a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:45.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.559+0000 7f8d9dd9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560910141350 0x560910150590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:45.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.561+0000 7f8d9ed9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609101464c0 0x5609101515a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:51874/0 (socket says 172.21.15.169:51874) 2024-03-28T00:58:45.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.561+0000 7f8d9ed9d640 1 -- 172.21.15.169:0/4273024131 learned_addr learned my addr 172.21.15.169:0/4273024131 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:45.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.562+0000 7f8d9e59c640 1 --2- 172.21.15.169:0/4273024131 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560910140960 0x560910150050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:45.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.564+0000 7f8d9ed9d640 1 -- 172.21.15.169:0/4273024131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560910141350 msgr2=0x560910150590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:45.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.564+0000 7f8d9ed9d640 1 --2- 172.21.15.169:0/4273024131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560910141350 0x560910150590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.564+0000 7f8d9ed9d640 1 -- 172.21.15.169:0/4273024131 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560910140960 msgr2=0x560910150050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:45.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.564+0000 7f8d9ed9d640 1 --2- 172.21.15.169:0/4273024131 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560910140960 0x560910150050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.564+0000 7f8d9ed9d640 1 -- 172.21.15.169:0/4273024131 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56090fd7a170 con 0x5609101464c0 2024-03-28T00:58:45.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.568+0000 7f8d9ed9d640 1 --2- 172.21.15.169:0/4273024131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609101464c0 0x5609101515a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8d98009fd0 tx=0x7f8d98018ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ef1f1188051050a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:45.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.569+0000 7f8d85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:58:45.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.570+0000 7f8d85ffb640 1 -- 172.21.15.169:0/4273024131 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d98040060 con 0x5609101464c0 2024-03-28T00:58:45.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.570+0000 7f8da79a4740 1 -- 172.21.15.169:0/4273024131 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8d9400ce10 con 0x5609101464c0 2024-03-28T00:58:45.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.570+0000 7f8d85ffb640 1 -- 172.21.15.169:0/4273024131 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f8d98029630 con 0x5609101464c0 2024-03-28T00:58:45.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.570+0000 7f8d85ffb640 1 -- 172.21.15.169:0/4273024131 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d98029830 con 0x5609101464c0 2024-03-28T00:58:45.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.570+0000 7f8da79a4740 10 client.25831 fetch_fsmap learned FSMap version 25 2024-03-28T00:58:45.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.570+0000 7f8da79a4740 1 -- 172.21.15.169:0/4273024131 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5609101c37c0 con 0x5609101464c0 2024-03-28T00:58:45.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.573+0000 7f8d85ffb640 1 -- 172.21.15.169:0/4273024131 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d9400ce10 con 0x5609101464c0 2024-03-28T00:58:45.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.573+0000 7f8d85ffb640 1 -- 172.21.15.169:0/4273024131 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f8d98028070 con 0x5609101464c0 2024-03-28T00:58:45.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.573+0000 7f8da79a4740 10 client.25831 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:58:45.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.573+0000 7f8da79a4740 10 client.25831 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:58:45.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.573+0000 7f8da79a4740 20 client.25831 populate_metadata read hostname 'smithi169' 2024-03-28T00:58:45.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.573+0000 7f8da79a4740 1 --2- 172.21.15.169:0/4273024131 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5609101c4140 0x5609101c6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:45.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.573+0000 7f8da79a4740 4 client.25831 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T00:58:45.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.573+0000 7f8da79a4740 1 -- 172.21.15.169:0/4273024131 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56091002bb80 con 0x5609101c4140 2024-03-28T00:58:45.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.574+0000 7f8d9e59c640 1 --2- 172.21.15.169:0/4273024131 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5609101c4140 0x5609101c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:45.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.581+0000 7f8d9e59c640 1 --2- 172.21.15.169:0/4273024131 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5609101c4140 0x5609101c6560 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c10c0b51f7495090 server_cookie=4c69e9fdc95bb0c1 in_seq=0 out_seq=0 2024-03-28T00:58:45.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.581+0000 7f8d85ffb640 10 client.25831 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:45.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:45 smithi174 ceph-mon[36060]: pgmap v418: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 773 MiB/s wr, 1.50k op/s 2024-03-28T00:58:45.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:45 smithi177 ceph-mon[36206]: pgmap v418: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 773 MiB/s wr, 1.50k op/s 2024-03-28T00:58:45.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.836+0000 7f8d85ffb640 1 -- 172.21.15.169:0/4273024131 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56091002bb80 con 0x5609101c4140 2024-03-28T00:58:45.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.836+0000 7f8d85ffb640 10 client.25831 handle_command_reply: tid=0 2024-03-28T00:58:45.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.836+0000 7f8da79a4740 1 client.25831 shutdown 2024-03-28T00:58:45.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.836+0000 7f8da79a4740 1 -- 172.21.15.169:0/4273024131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609101464c0 msgr2=0x5609101515a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:45.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.836+0000 7f8da79a4740 1 --2- 172.21.15.169:0/4273024131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609101464c0 0x5609101515a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8d98009fd0 tx=0x7f8d98018ca0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.836+0000 7f8da79a4740 1 -- 172.21.15.169:0/4273024131 shutdown_connections 2024-03-28T00:58:45.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.836+0000 7f8da79a4740 1 --2- 172.21.15.169:0/4273024131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609101464c0 0x5609101515a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.836+0000 7f8da79a4740 1 --2- 172.21.15.169:0/4273024131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560910141350 0x560910150590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.836+0000 7f8da79a4740 1 --2- 172.21.15.169:0/4273024131 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5609101c4140 0x5609101c6560 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.837+0000 7f8da79a4740 1 --2- 172.21.15.169:0/4273024131 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560910140960 0x560910150050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.837+0000 7f8da79a4740 1 -- 172.21.15.169:0/4273024131 >> 172.21.15.169:0/4273024131 conn(0x5609101091a0 msgr2=0x560910149350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:45.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.837+0000 7f8d85ffb640 0 client.25831 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:45.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.837+0000 7f8da79a4740 1 -- 172.21.15.169:0/4273024131 shutdown_connections 2024-03-28T00:58:45.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.837+0000 7f8da79a4740 1 -- 172.21.15.169:0/4273024131 wait complete. 2024-03-28T00:58:45.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.837+0000 7f8da79a4740 20 client.25831 trim_cache size 0 max 16384 2024-03-28T00:58:45.841 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:58:45.841 INFO:teuthology.orchestra.run.smithi169.stdout: "return_code": 0, 2024-03-28T00:58:45.842 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_tag": "bdf5bf72-032f-4049-825d-9fbb21aacecf", 2024-03-28T00:58:45.842 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": "asynchronous" 2024-03-28T00:58:45.842 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:58:45.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.842+0000 7f8da4db9640 1 -- 172.21.15.169:0/3033851893 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8d880a28f0 msgr2=0x7f8d880a4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:45.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.842+0000 7f8da4db9640 1 --2- 172.21.15.169:0/3033851893 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8d880a28f0 0x7f8d880a4db0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8d98016000 tx=0x7f8d98018000 comp rx=0 tx=0).stop 2024-03-28T00:58:45.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.842+0000 7f8da4db9640 1 -- 172.21.15.169:0/3033851893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8da0104a50 msgr2=0x7f8da0070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:45.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.842+0000 7f8da4db9640 1 --2- 172.21.15.169:0/3033851893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8da0104a50 0x7f8da0070260 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8d9002c210 tx=0x7f8d90004040 comp rx=0 tx=0).stop 2024-03-28T00:58:45.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.842+0000 7f8da4db9640 1 -- 172.21.15.169:0/3033851893 shutdown_connections 2024-03-28T00:58:45.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.842+0000 7f8da4db9640 1 --2- 172.21.15.169:0/3033851893 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f8d880a28f0 0x7f8d880a4db0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.842+0000 7f8da4db9640 1 --2- 172.21.15.169:0/3033851893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8da010c710 0x7f8da010cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.842+0000 7f8da4db9640 1 --2- 172.21.15.169:0/3033851893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8da0107380 0x7f8da00707a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.842+0000 7f8da4db9640 1 --2- 172.21.15.169:0/3033851893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8da0104a50 0x7f8da0070260 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:45.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.842+0000 7f8da4db9640 1 -- 172.21.15.169:0/3033851893 >> 172.21.15.169:0/3033851893 conn(0x7f8da00fbb30 msgr2=0x7f8da01029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:45.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.842+0000 7f8da4db9640 1 -- 172.21.15.169:0/3033851893 shutdown_connections 2024-03-28T00:58:45.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:45.842+0000 7f8da4db9640 1 -- 172.21.15.169:0/3033851893 wait complete. 2024-03-28T00:58:45.917 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T00:58:45.917 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T00:58:46.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.241+0000 7ff4cd353640 1 -- 172.21.15.169:0/3071351662 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c402ccd0 con 0x7ff4c8109730 2024-03-28T00:58:46.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.242+0000 7ff4cf357640 1 -- 172.21.15.169:0/3071351662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4c8109730 msgr2=0x7ff4c810dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:46.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.242+0000 7ff4cf357640 1 --2- 172.21.15.169:0/3071351662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4c8109730 0x7ff4c810dfd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4009900 tx=0x7ff4c402bc80 comp rx=0 tx=0).stop 2024-03-28T00:58:46.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.245+0000 7ff4cf357640 1 -- 172.21.15.169:0/3071351662 shutdown_connections 2024-03-28T00:58:46.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.245+0000 7ff4cf357640 1 --2- 172.21.15.169:0/3071351662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4c8109730 0x7ff4c810dfd0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.245+0000 7ff4cf357640 1 --2- 172.21.15.169:0/3071351662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4c8108d70 0x7ff4c81091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.245+0000 7ff4cf357640 1 --2- 172.21.15.169:0/3071351662 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4c8072ab0 0x7ff4c8072ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.245+0000 7ff4cf357640 1 -- 172.21.15.169:0/3071351662 >> 172.21.15.169:0/3071351662 conn(0x7ff4c806e580 msgr2=0x7ff4c80709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:46.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.251+0000 7ff4cf357640 1 -- 172.21.15.169:0/3071351662 shutdown_connections 2024-03-28T00:58:46.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.251+0000 7ff4cf357640 1 -- 172.21.15.169:0/3071351662 wait complete. 2024-03-28T00:58:46.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.252+0000 7ff4cf357640 1 Processor -- start 2024-03-28T00:58:46.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.252+0000 7ff4cf357640 1 -- start start 2024-03-28T00:58:46.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.253+0000 7ff4cf357640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4c8072ab0 0x7ff4c8114cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:46.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.253+0000 7ff4cf357640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4c8108d70 0x7ff4c8115230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:46.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.253+0000 7ff4cf357640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4c8109730 0x7ff4c81b6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:46.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.253+0000 7ff4cf357640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c811c110 con 0x7ff4c8108d70 2024-03-28T00:58:46.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.253+0000 7ff4cf357640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c811bf90 con 0x7ff4c8109730 2024-03-28T00:58:46.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.253+0000 7ff4cf357640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c811c290 con 0x7ff4c8072ab0 2024-03-28T00:58:46.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.253+0000 7ff4ce355640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4c8072ab0 0x7ff4c8114cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:46.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.253+0000 7ff4ceb56640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4c8109730 0x7ff4c81b6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:46.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.254+0000 7ff4ceb56640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4c8109730 0x7ff4c81b6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:54922/0 (socket says 172.21.15.169:54922) 2024-03-28T00:58:46.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.254+0000 7ff4ce355640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4c8072ab0 0x7ff4c8114cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:51876/0 (socket says 172.21.15.169:51876) 2024-03-28T00:58:46.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.254+0000 7ff4ceb56640 1 -- 172.21.15.169:0/3352836762 learned_addr learned my addr 172.21.15.169:0/3352836762 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:46.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.254+0000 7ff4cdb54640 1 --2- 172.21.15.169:0/3352836762 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4c8108d70 0x7ff4c8115230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:46.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.254+0000 7ff4ceb56640 1 -- 172.21.15.169:0/3352836762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4c8072ab0 msgr2=0x7ff4c8114cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:46.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.254+0000 7ff4ceb56640 1 --2- 172.21.15.169:0/3352836762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4c8072ab0 0x7ff4c8114cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.255+0000 7ff4ceb56640 1 -- 172.21.15.169:0/3352836762 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4c8108d70 msgr2=0x7ff4c8115230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:46.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.255+0000 7ff4ceb56640 1 --2- 172.21.15.169:0/3352836762 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4c8108d70 0x7ff4c8115230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.255+0000 7ff4ceb56640 1 -- 172.21.15.169:0/3352836762 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c81158d0 con 0x7ff4c8109730 2024-03-28T00:58:46.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.255+0000 7ff4ce355640 1 --2- 172.21.15.169:0/3352836762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4c8072ab0 0x7ff4c8114cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:58:46.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.255+0000 7ff4cdb54640 1 --2- 172.21.15.169:0/3352836762 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4c8108d70 0x7ff4c8115230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T00:58:46.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.257+0000 7ff4ceb56640 1 --2- 172.21.15.169:0/3352836762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4c8109730 0x7ff4c81b6820 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff4c402c930 tx=0x7ff4c402c6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:46.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.261+0000 7ff4bf7fe640 1 -- 172.21.15.169:0/3352836762 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4041070 con 0x7ff4c8109730 2024-03-28T00:58:46.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.261+0000 7ff4cf357640 1 -- 172.21.15.169:0/3352836762 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c81b6e20 con 0x7ff4c8109730 2024-03-28T00:58:46.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.261+0000 7ff4cf357640 1 -- 172.21.15.169:0/3352836762 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c81b7220 con 0x7ff4c8109730 2024-03-28T00:58:46.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.262+0000 7ff4bf7fe640 1 -- 172.21.15.169:0/3352836762 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4c40054e0 con 0x7ff4c8109730 2024-03-28T00:58:46.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.262+0000 7ff4bf7fe640 1 -- 172.21.15.169:0/3352836762 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4034440 con 0x7ff4c8109730 2024-03-28T00:58:46.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.266+0000 7ff4d31a4740 1 Processor -- start 2024-03-28T00:58:46.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.267+0000 7ff4bf7fe640 1 -- 172.21.15.169:0/3352836762 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff4c4044070 con 0x7ff4c8109730 2024-03-28T00:58:46.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.268+0000 7ff4bf7fe640 1 --2- 172.21.15.169:0/3352836762 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff4b40a2980 0x7ff4b40a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:46.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.268+0000 7ff4bf7fe640 1 -- 172.21.15.169:0/3352836762 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ff4c40f0b00 con 0x7ff4c8109730 2024-03-28T00:58:46.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.269+0000 7ff4d31a4740 1 -- start start 2024-03-28T00:58:46.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.270+0000 7ff4d31a4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867ceff80 0x555867cf03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:46.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.270+0000 7ff4d31a4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867cf0970 0x555867cf5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:46.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.270+0000 7ff4d31a4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867cf5ae0 0x555867cf7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:46.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.270+0000 7ff4d31a4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x555867c553d0 con 0x555867cf5ae0 2024-03-28T00:58:46.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.270+0000 7ff4d31a4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x555867bfdad0 con 0x555867ceff80 2024-03-28T00:58:46.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.270+0000 7ff4d31a4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x555867c51470 con 0x555867cf0970 2024-03-28T00:58:46.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.271+0000 7ff4ceb56640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867cf5ae0 0x555867cf7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:46.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.271+0000 7ff4ceb56640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867cf5ae0 0x555867cf7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46904/0 (socket says 172.21.15.169:46904) 2024-03-28T00:58:46.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.271+0000 7ff4ceb56640 1 -- 172.21.15.169:0/1901012004 learned_addr learned my addr 172.21.15.169:0/1901012004 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:46.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.271+0000 7ff4cdb54640 1 --2- 172.21.15.169:0/3352836762 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff4b40a2980 0x7ff4b40a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:46.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.271+0000 7ff4ce355640 1 --2- 172.21.15.169:0/1901012004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867ceff80 0x555867cf03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:46.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.271+0000 7ff4cdb54640 1 --2- 172.21.15.169:0/1901012004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867cf0970 0x555867cf5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:46.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.272+0000 7ff4ceb56640 1 -- 172.21.15.169:0/1901012004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867cf0970 msgr2=0x555867cf5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:46.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.272+0000 7ff4ceb56640 1 --2- 172.21.15.169:0/1901012004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867cf0970 0x555867cf5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.272+0000 7ff4ceb56640 1 -- 172.21.15.169:0/1901012004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867ceff80 msgr2=0x555867cf03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:46.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.272+0000 7ff4ceb56640 1 --2- 172.21.15.169:0/1901012004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867ceff80 0x555867cf03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.272+0000 7ff4ceb56640 1 -- 172.21.15.169:0/1901012004 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555867a337d0 con 0x555867cf5ae0 2024-03-28T00:58:46.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.272+0000 7ff4ceb56640 1 --2- 172.21.15.169:0/1901012004 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867cf5ae0 0x555867cf7f70 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40b9560 tx=0x7ff4c40bcf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be096f26c29de665 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:46.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.275+0000 7ff4bcbfe640 1 -- 172.21.15.169:0/1901012004 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c404b040 con 0x555867cf5ae0 2024-03-28T00:58:46.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.275+0000 7ff4bcbfe640 1 -- 172.21.15.169:0/1901012004 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4c40bd100 con 0x555867cf5ae0 2024-03-28T00:58:46.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.275+0000 7ff4bcbfe640 1 -- 172.21.15.169:0/1901012004 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4004cf0 con 0x555867cf5ae0 2024-03-28T00:58:46.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.276+0000 7ff4cdb54640 1 --2- 172.21.15.169:0/3352836762 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff4b40a2980 0x7ff4b40a4e40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x555867cf0d70 tx=0x7ff4c0009960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:46.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.277+0000 7ff4d31a4740 1 -- 172.21.15.169:0/1901012004 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867cf5ae0 msgr2=0x555867cf7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:46.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.277+0000 7ff4d31a4740 1 --2- 172.21.15.169:0/1901012004 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867cf5ae0 0x555867cf7f70 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40b9560 tx=0x7ff4c40bcf90 comp rx=0 tx=0).stop 2024-03-28T00:58:46.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.277+0000 7ff4d31a4740 1 -- 172.21.15.169:0/1901012004 shutdown_connections 2024-03-28T00:58:46.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.277+0000 7ff4d31a4740 1 --2- 172.21.15.169:0/1901012004 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867cf5ae0 0x555867cf7f70 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.277+0000 7ff4d31a4740 1 --2- 172.21.15.169:0/1901012004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867cf0970 0x555867cf5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.277+0000 7ff4d31a4740 1 --2- 172.21.15.169:0/1901012004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867ceff80 0x555867cf03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.277+0000 7ff4d31a4740 1 -- 172.21.15.169:0/1901012004 >> 172.21.15.169:0/1901012004 conn(0x555867ceb4f0 msgr2=0x555867ceb8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:46.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.278+0000 7ff4d31a4740 1 -- 172.21.15.169:0/1901012004 shutdown_connections 2024-03-28T00:58:46.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.280+0000 7ff4d31a4740 1 -- 172.21.15.169:0/1901012004 wait complete. 2024-03-28T00:58:46.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.280+0000 7ff4d31a4740 1 -- start start 2024-03-28T00:58:46.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.281+0000 7ff4d31a4740 1 Processor -- start 2024-03-28T00:58:46.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.281+0000 7ff4d31a4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867ceff80 0x555867cfa040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:46.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.281+0000 7ff4d31a4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867cf0970 0x555867cfa580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:46.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.281+0000 7ff4d31a4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867cf5ae0 0x555867d015c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:46.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.281+0000 7ff4d31a4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x555867cf8850 con 0x555867ceff80 2024-03-28T00:58:46.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.281+0000 7ff4d31a4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x555867c11060 con 0x555867cf0970 2024-03-28T00:58:46.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.281+0000 7ff4d31a4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x555867cf89d0 con 0x555867cf5ae0 2024-03-28T00:58:46.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.281+0000 7ff4d31a4740 10 client.0 _ll_register_callbacks cb 0x555867bf36a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:58:46.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.282+0000 7ff4ceb56640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867cf5ae0 0x555867d015c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:46.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.282+0000 7ff4cdb54640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867cf0970 0x555867cfa580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:46.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.282+0000 7ff4ceb56640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867cf5ae0 0x555867d015c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:51888/0 (socket says 172.21.15.169:51888) 2024-03-28T00:58:46.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.282+0000 7ff4ceb56640 1 -- 172.21.15.169:0/527022263 learned_addr learned my addr 172.21.15.169:0/527022263 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:46.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.282+0000 7ff4ce355640 1 --2- 172.21.15.169:0/527022263 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867ceff80 0x555867cfa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:46.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.283+0000 7ff4ce355640 1 -- 172.21.15.169:0/527022263 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867cf5ae0 msgr2=0x555867d015c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:46.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.283+0000 7ff4ce355640 1 --2- 172.21.15.169:0/527022263 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867cf5ae0 0x555867d015c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.283+0000 7ff4ce355640 1 -- 172.21.15.169:0/527022263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867cf0970 msgr2=0x555867cfa580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:46.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.283+0000 7ff4ce355640 1 --2- 172.21.15.169:0/527022263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867cf0970 0x555867cfa580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.283+0000 7ff4ce355640 1 -- 172.21.15.169:0/527022263 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558679693f0 con 0x555867ceff80 2024-03-28T00:58:46.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.284+0000 7ff4ce355640 1 --2- 172.21.15.169:0/527022263 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867ceff80 0x555867cfa040 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x555867cf11d0 tx=0x7ff4b8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47ee9b8be492eada server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:46.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.284+0000 7ff4b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:58:46.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.286+0000 7ff4b1ffb640 1 -- 172.21.15.169:0/527022263 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8019070 con 0x555867ceff80 2024-03-28T00:58:46.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.286+0000 7ff4b1ffb640 1 -- 172.21.15.169:0/527022263 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ff4b8015870 con 0x555867ceff80 2024-03-28T00:58:46.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.286+0000 7ff4d31a4740 1 -- 172.21.15.169:0/527022263 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff4c40bd100 con 0x555867ceff80 2024-03-28T00:58:46.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.290+0000 7ff4b1ffb640 1 -- 172.21.15.169:0/527022263 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4b8023050 con 0x555867ceff80 2024-03-28T00:58:46.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.291+0000 7ff4d31a4740 10 client.16038 fetch_fsmap learned FSMap version 25 2024-03-28T00:58:46.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.291+0000 7ff4d31a4740 1 -- 172.21.15.169:0/527022263 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5558679290f0 con 0x555867ceff80 2024-03-28T00:58:46.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.294+0000 7ff4b1ffb640 1 -- 172.21.15.169:0/527022263 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4c40bd100 con 0x555867ceff80 2024-03-28T00:58:46.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.295+0000 7ff4b1ffb640 1 -- 172.21.15.169:0/527022263 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ff4b8013400 con 0x555867ceff80 2024-03-28T00:58:46.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.295+0000 7ff4d31a4740 10 client.16038 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:58:46.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.295+0000 7ff4d31a4740 10 client.16038 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:58:46.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.295+0000 7ff4d31a4740 20 client.16038 populate_metadata read hostname 'smithi169' 2024-03-28T00:58:46.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.295+0000 7ff4d31a4740 1 --2- 172.21.15.169:0/527022263 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555867d05ba0 0x555867d08000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:46.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.295+0000 7ff4d31a4740 4 client.16038 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:58:46.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.295+0000 7ff4d31a4740 1 -- 172.21.15.169:0/527022263 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555867c62910 con 0x555867d05ba0 2024-03-28T00:58:46.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.296+0000 7ff4cdb54640 1 --2- 172.21.15.169:0/527022263 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555867d05ba0 0x555867d08000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:46.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.297+0000 7ff4cdb54640 1 --2- 172.21.15.169:0/527022263 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555867d05ba0 0x555867d08000 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89874adffc6e9d5d server_cookie=3e6099ae76917b39 in_seq=0 out_seq=0 2024-03-28T00:58:46.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.297+0000 7ff4b1ffb640 10 client.16038 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:46.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.301+0000 7ff4b1ffb640 1 -- 172.21.15.169:0/527022263 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555867c62910 con 0x555867d05ba0 2024-03-28T00:58:46.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.301+0000 7ff4b1ffb640 10 client.16038 handle_command_reply: tid=0 2024-03-28T00:58:46.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.302+0000 7ff4d31a4740 1 client.16038 shutdown 2024-03-28T00:58:46.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.302+0000 7ff4d31a4740 1 -- 172.21.15.169:0/527022263 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867ceff80 msgr2=0x555867cfa040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:46.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.302+0000 7ff4d31a4740 1 --2- 172.21.15.169:0/527022263 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867ceff80 0x555867cfa040 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x555867cf11d0 tx=0x7ff4b8009f90 comp rx=0 tx=0).stop 2024-03-28T00:58:46.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.306+0000 7ff4d31a4740 1 -- 172.21.15.169:0/527022263 shutdown_connections 2024-03-28T00:58:46.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.306+0000 7ff4d31a4740 1 --2- 172.21.15.169:0/527022263 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867cf5ae0 0x555867d015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.306+0000 7ff4d31a4740 1 --2- 172.21.15.169:0/527022263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867cf0970 0x555867cfa580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.306+0000 7ff4d31a4740 1 --2- 172.21.15.169:0/527022263 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555867d05ba0 0x555867d08000 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.306+0000 7ff4d31a4740 1 --2- 172.21.15.169:0/527022263 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867ceff80 0x555867cfa040 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.306+0000 7ff4d31a4740 1 -- 172.21.15.169:0/527022263 >> 172.21.15.169:0/527022263 conn(0x555867ceb4f0 msgr2=0x555867cea7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:46.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.308+0000 7ff4b1ffb640 0 client.16038 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:46.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.308+0000 7ff4d31a4740 1 -- 172.21.15.169:0/527022263 shutdown_connections 2024-03-28T00:58:46.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.311+0000 7ff4d31a4740 1 -- 172.21.15.169:0/527022263 wait complete. 2024-03-28T00:58:46.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.311+0000 7ff4d31a4740 20 client.16038 trim_cache size 0 max 16384 2024-03-28T00:58:46.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.371+0000 7ff4d31a4740 1 Processor -- start 2024-03-28T00:58:46.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.374+0000 7ff4d31a4740 1 -- start start 2024-03-28T00:58:46.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.374+0000 7ff4d31a4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867ceff80 0x555867ceedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:46.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.375+0000 7ff4d31a4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867cf0970 0x555867d014e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:46.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.375+0000 7ff4d31a4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867cf5ae0 0x555867cf8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:46.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.375+0000 7ff4d31a4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c81c3c00 con 0x555867ceff80 2024-03-28T00:58:46.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.375+0000 7ff4d31a4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c81c1700 con 0x555867cf0970 2024-03-28T00:58:46.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.375+0000 7ff4d31a4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c81bf7a0 con 0x555867cf5ae0 2024-03-28T00:58:46.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.377+0000 7ff4cdb54640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867cf0970 0x555867d014e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:46.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.377+0000 7ff4cdb54640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867cf0970 0x555867d014e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:54954/0 (socket says 172.21.15.169:54954) 2024-03-28T00:58:46.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.377+0000 7ff4cdb54640 1 -- 172.21.15.169:0/605627578 learned_addr learned my addr 172.21.15.169:0/605627578 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:46.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.377+0000 7ff4ceb56640 1 --2- 172.21.15.169:0/605627578 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867cf5ae0 0x555867cf8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:46.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.379+0000 7ff4cdb54640 1 -- 172.21.15.169:0/605627578 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867cf5ae0 msgr2=0x555867cf8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:46.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.379+0000 7ff4cdb54640 1 --2- 172.21.15.169:0/605627578 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867cf5ae0 0x555867cf8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.379+0000 7ff4cdb54640 1 -- 172.21.15.169:0/605627578 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867ceff80 msgr2=0x555867ceedc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T00:58:46.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.379+0000 7ff4cdb54640 1 --2- 172.21.15.169:0/605627578 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867ceff80 0x555867ceedc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.379+0000 7ff4cdb54640 1 -- 172.21.15.169:0/605627578 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555867ce6f60 con 0x555867cf0970 2024-03-28T00:58:46.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.384+0000 7ff4cdb54640 1 --2- 172.21.15.169:0/605627578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867cf0970 0x555867d014e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0007be0 tx=0x7ff4c0004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8ea3281ab8d9863 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:46.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.392+0000 7ff4b3fff640 1 -- 172.21.15.169:0/605627578 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c002b070 con 0x555867cf0970 2024-03-28T00:58:46.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.393+0000 7ff4b3fff640 1 -- 172.21.15.169:0/605627578 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4c0016040 con 0x555867cf0970 2024-03-28T00:58:46.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.394+0000 7ff4d31a4740 1 -- 172.21.15.169:0/605627578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867cf0970 msgr2=0x555867d014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:46.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.394+0000 7ff4d31a4740 1 --2- 172.21.15.169:0/605627578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867cf0970 0x555867d014e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0007be0 tx=0x7ff4c0004a80 comp rx=0 tx=0).stop 2024-03-28T00:58:46.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.396+0000 7ff4d31a4740 1 -- 172.21.15.169:0/605627578 shutdown_connections 2024-03-28T00:58:46.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.396+0000 7ff4d31a4740 1 --2- 172.21.15.169:0/605627578 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867cf5ae0 0x555867cf8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.396+0000 7ff4d31a4740 1 --2- 172.21.15.169:0/605627578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867cf0970 0x555867d014e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.396+0000 7ff4d31a4740 1 --2- 172.21.15.169:0/605627578 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867ceff80 0x555867ceedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.396+0000 7ff4d31a4740 1 -- 172.21.15.169:0/605627578 >> 172.21.15.169:0/605627578 conn(0x555867ceb4f0 msgr2=0x555867cea7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:46.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.403+0000 7ff4d31a4740 1 -- 172.21.15.169:0/605627578 shutdown_connections 2024-03-28T00:58:46.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.407+0000 7ff4d31a4740 1 -- 172.21.15.169:0/605627578 wait complete. 2024-03-28T00:58:46.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.407+0000 7ff4d31a4740 1 -- start start 2024-03-28T00:58:46.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.407+0000 7ff4d31a4740 1 Processor -- start 2024-03-28T00:58:46.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.408+0000 7ff4d31a4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867ceff80 0x555867cff080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:46.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.408+0000 7ff4d31a4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867cf5ae0 0x555867cff5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:46.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.408+0000 7ff4d31a4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867d005d0 0x555867d00a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:46.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.408+0000 7ff4d31a4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c81bfda0 con 0x555867ceff80 2024-03-28T00:58:46.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.408+0000 7ff4d31a4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c81c10a0 con 0x555867cf5ae0 2024-03-28T00:58:46.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.408+0000 7ff4d31a4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c81c03e0 con 0x555867d005d0 2024-03-28T00:58:46.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.408+0000 7ff4d31a4740 10 client.0 _ll_register_callbacks cb 0x7ff4b8009e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:58:46.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.411+0000 7ff4cdb54640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867cf5ae0 0x555867cff5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:46.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.411+0000 7ff4ceb56640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867d005d0 0x555867d00a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:46.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.416+0000 7ff4ceb56640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867d005d0 0x555867d00a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:51896/0 (socket says 172.21.15.169:51896) 2024-03-28T00:58:46.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.416+0000 7ff4ceb56640 1 -- 172.21.15.169:0/3637143617 learned_addr learned my addr 172.21.15.169:0/3637143617 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:46.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.416+0000 7ff4ce355640 1 --2- 172.21.15.169:0/3637143617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867ceff80 0x555867cff080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:46.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.419+0000 7ff4cdb54640 1 -- 172.21.15.169:0/3637143617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867d005d0 msgr2=0x555867d00a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:46.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.419+0000 7ff4cdb54640 1 --2- 172.21.15.169:0/3637143617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867d005d0 0x555867d00a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.419+0000 7ff4cdb54640 1 -- 172.21.15.169:0/3637143617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867ceff80 msgr2=0x555867cff080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:46.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.419+0000 7ff4cdb54640 1 --2- 172.21.15.169:0/3637143617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867ceff80 0x555867cff080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.419+0000 7ff4cdb54640 1 -- 172.21.15.169:0/3637143617 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555867d00f90 con 0x555867cf5ae0 2024-03-28T00:58:46.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.422+0000 7ff4cdb54640 1 --2- 172.21.15.169:0/3637143617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867cf5ae0 0x555867cff5c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0029040 tx=0x7ff4c001c630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2421c7b3baf53cbf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:46.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.422+0000 7ff4b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:58:46.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.427+0000 7ff4b1ffb640 1 -- 172.21.15.169:0/3637143617 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c002b070 con 0x555867cf5ae0 2024-03-28T00:58:46.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.427+0000 7ff4d31a4740 1 -- 172.21.15.169:0/3637143617 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555867c5cc60 con 0x555867cf5ae0 2024-03-28T00:58:46.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.427+0000 7ff4b1ffb640 1 -- 172.21.15.169:0/3637143617 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ff4c001ca30 con 0x555867cf5ae0 2024-03-28T00:58:46.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.430+0000 7ff4b1ffb640 1 -- 172.21.15.169:0/3637143617 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4c0010030 con 0x555867cf5ae0 2024-03-28T00:58:46.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.432+0000 7ff4d31a4740 10 client.25865 fetch_fsmap learned FSMap version 25 2024-03-28T00:58:46.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.433+0000 7ff4d31a4740 1 -- 172.21.15.169:0/3637143617 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x555867d61e20 con 0x555867cf5ae0 2024-03-28T00:58:46.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.440+0000 7ff4b1ffb640 1 -- 172.21.15.169:0/3637143617 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x555867c5cc60 con 0x555867cf5ae0 2024-03-28T00:58:46.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.443+0000 7ff4b1ffb640 1 -- 172.21.15.169:0/3637143617 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ff4c001bc40 con 0x555867cf5ae0 2024-03-28T00:58:46.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.443+0000 7ff4d31a4740 10 client.25865 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:58:46.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.443+0000 7ff4d31a4740 10 client.25865 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:58:46.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.443+0000 7ff4d31a4740 20 client.25865 populate_metadata read hostname 'smithi169' 2024-03-28T00:58:46.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.443+0000 7ff4d31a4740 1 --2- 172.21.15.169:0/3637143617 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555867d626e0 0x555867d64b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:46.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.443+0000 7ff4d31a4740 4 client.25865 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T00:58:46.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.443+0000 7ff4d31a4740 1 -- 172.21.15.169:0/3637143617 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555867bda4e0 con 0x555867d626e0 2024-03-28T00:58:46.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.449+0000 7ff4ce355640 1 --2- 172.21.15.169:0/3637143617 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555867d626e0 0x555867d64b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:46.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.468+0000 7ff4ce355640 1 --2- 172.21.15.169:0/3637143617 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555867d626e0 0x555867d64b60 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cde3335cc703158 server_cookie=62baa55b8c934c77 in_seq=0 out_seq=0 2024-03-28T00:58:46.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.468+0000 7ff4b1ffb640 10 client.25865 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:46.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.472+0000 7ff4b1ffb640 1 -- 172.21.15.169:0/3637143617 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+266 (crc 0 0 0) 0x555867bda4e0 con 0x555867d626e0 2024-03-28T00:58:46.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.472+0000 7ff4b1ffb640 10 client.25865 handle_command_reply: tid=0 2024-03-28T00:58:46.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.472+0000 7ff4d31a4740 1 client.25865 shutdown 2024-03-28T00:58:46.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.481+0000 7ff4d31a4740 1 -- 172.21.15.169:0/3637143617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867cf5ae0 msgr2=0x555867cff5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:46.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.481+0000 7ff4d31a4740 1 --2- 172.21.15.169:0/3637143617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867cf5ae0 0x555867cff5c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0029040 tx=0x7ff4c001c630 comp rx=0 tx=0).stop 2024-03-28T00:58:46.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.483+0000 7ff4d31a4740 1 -- 172.21.15.169:0/3637143617 shutdown_connections 2024-03-28T00:58:46.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.483+0000 7ff4d31a4740 1 --2- 172.21.15.169:0/3637143617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555867d005d0 0x555867d00a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.483+0000 7ff4d31a4740 1 --2- 172.21.15.169:0/3637143617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555867cf5ae0 0x555867cff5c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.483+0000 7ff4d31a4740 1 --2- 172.21.15.169:0/3637143617 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555867d626e0 0x555867d64b60 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.483+0000 7ff4d31a4740 1 --2- 172.21.15.169:0/3637143617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555867ceff80 0x555867cff080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.483+0000 7ff4d31a4740 1 -- 172.21.15.169:0/3637143617 >> 172.21.15.169:0/3637143617 conn(0x555867ceb4f0 msgr2=0x555867d0c900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:46.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.485+0000 7ff4b1ffb640 0 client.25865 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:46.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.489+0000 7ff4d31a4740 1 -- 172.21.15.169:0/3637143617 shutdown_connections 2024-03-28T00:58:46.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.491+0000 7ff4d31a4740 1 -- 172.21.15.169:0/3637143617 wait complete. 2024-03-28T00:58:46.499 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:58:46.499 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "scrub active (10 inodes in the stack)", 2024-03-28T00:58:46.499 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": { 2024-03-28T00:58:46.499 INFO:teuthology.orchestra.run.smithi169.stdout: "bdf5bf72-032f-4049-825d-9fbb21aacecf": { 2024-03-28T00:58:46.499 INFO:teuthology.orchestra.run.smithi169.stdout: "path": "/", 2024-03-28T00:58:46.499 INFO:teuthology.orchestra.run.smithi169.stdout: "tag": "bdf5bf72-032f-4049-825d-9fbb21aacecf", 2024-03-28T00:58:46.499 INFO:teuthology.orchestra.run.smithi169.stdout: "options": "recursive,force" 2024-03-28T00:58:46.499 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T00:58:46.499 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T00:58:46.499 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:58:46.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.494+0000 7ff4d31a4740 20 client.25865 trim_cache size 0 max 16384 2024-03-28T00:58:46.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.504+0000 7ff4cf357640 1 -- 172.21.15.169:0/3352836762 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff4b40a2980 msgr2=0x7ff4b40a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:46.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.504+0000 7ff4cf357640 1 --2- 172.21.15.169:0/3352836762 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff4b40a2980 0x7ff4b40a4e40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x555867cf0d70 tx=0x7ff4c0009960 comp rx=0 tx=0).stop 2024-03-28T00:58:46.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.504+0000 7ff4cf357640 1 -- 172.21.15.169:0/3352836762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4c8109730 msgr2=0x7ff4c81b6820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:46.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.504+0000 7ff4cf357640 1 --2- 172.21.15.169:0/3352836762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4c8109730 0x7ff4c81b6820 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff4c402c930 tx=0x7ff4c402c6f0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.505+0000 7ff4cf357640 1 -- 172.21.15.169:0/3352836762 shutdown_connections 2024-03-28T00:58:46.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.505+0000 7ff4cf357640 1 --2- 172.21.15.169:0/3352836762 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff4b40a2980 0x7ff4b40a4e40 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.505+0000 7ff4cf357640 1 --2- 172.21.15.169:0/3352836762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4c8109730 0x7ff4c81b6820 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.505+0000 7ff4cf357640 1 --2- 172.21.15.169:0/3352836762 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff4c8108d70 0x7ff4c8115230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.505+0000 7ff4cf357640 1 --2- 172.21.15.169:0/3352836762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4c8072ab0 0x7ff4c8114cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:46.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.505+0000 7ff4cf357640 1 -- 172.21.15.169:0/3352836762 >> 172.21.15.169:0/3352836762 conn(0x7ff4c806e580 msgr2=0x7ff4c8070130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:46.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.506+0000 7ff4cf357640 1 -- 172.21.15.169:0/3352836762 shutdown_connections 2024-03-28T00:58:46.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:58:46.506+0000 7ff4cf357640 1 -- 172.21.15.169:0/3352836762 wait complete. 2024-03-28T00:58:46.542 INFO:tasks.cephfs.filesystem:scrub status for tag:bdf5bf72-032f-4049-825d-9fbb21aacecf - {'path': '/', 'tag': 'bdf5bf72-032f-4049-825d-9fbb21aacecf', 'options': 'recursive,force'} 2024-03-28T00:58:46.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:46 smithi177 ceph-mon[36206]: pgmap v419: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 902 MiB/s wr, 1.67k op/s 2024-03-28T00:58:46.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:46 smithi169 ceph-mon[32274]: pgmap v419: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 902 MiB/s wr, 1.67k op/s 2024-03-28T00:58:46.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:46 smithi174 ceph-mon[36060]: pgmap v419: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 902 MiB/s wr, 1.67k op/s 2024-03-28T00:58:47.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:47 smithi177 ceph-mon[36206]: scrub queued for path: / 2024-03-28T00:58:47.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:47 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T00:58:47.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:47 smithi177 ceph-mon[36206]: scrub summary: active paths [/] 2024-03-28T00:58:47.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:47 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T00:58:47.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:47 smithi169 ceph-mon[32274]: scrub queued for path: / 2024-03-28T00:58:47.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:47 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T00:58:47.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:47 smithi169 ceph-mon[32274]: scrub summary: active paths [/] 2024-03-28T00:58:47.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:47 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T00:58:47.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:47 smithi174 ceph-mon[36060]: scrub queued for path: / 2024-03-28T00:58:47.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:47 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T00:58:47.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:47 smithi174 ceph-mon[36060]: scrub summary: active paths [/] 2024-03-28T00:58:47.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:47 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T00:58:48.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:48 smithi177 ceph-mon[36206]: pgmap v420: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 785 MiB/s wr, 1.59k op/s 2024-03-28T00:58:48.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:48 smithi169 ceph-mon[32274]: pgmap v420: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 785 MiB/s wr, 1.59k op/s 2024-03-28T00:58:48.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:48 smithi174 ceph-mon[36060]: pgmap v420: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 785 MiB/s wr, 1.59k op/s 2024-03-28T00:58:49.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:49 smithi169 ceph-mon[32274]: scrub summary: idle 2024-03-28T00:58:49.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:49 smithi174 ceph-mon[36060]: scrub summary: idle 2024-03-28T00:58:49.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:49 smithi177 ceph-mon[36206]: scrub summary: idle 2024-03-28T00:58:50.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:50 smithi169 ceph-mon[32274]: pgmap v421: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 754 MiB/s wr, 1.62k op/s 2024-03-28T00:58:50.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:50 smithi174 ceph-mon[36060]: pgmap v421: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 754 MiB/s wr, 1.62k op/s 2024-03-28T00:58:50.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:50 smithi177 ceph-mon[36206]: pgmap v421: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 754 MiB/s wr, 1.62k op/s 2024-03-28T00:58:53.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:53 smithi169 ceph-mon[32274]: pgmap v422: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 948 MiB/s wr, 2.06k op/s 2024-03-28T00:58:53.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:53 smithi174 ceph-mon[36060]: pgmap v422: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 948 MiB/s wr, 2.06k op/s 2024-03-28T00:58:53.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:53 smithi177 ceph-mon[36206]: pgmap v422: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 948 MiB/s wr, 2.06k op/s 2024-03-28T00:58:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.078+0000 7fee06162640 1 -- 172.21.15.169:0/390077246 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fee00072950 msgr2=0x7fee00072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.078+0000 7fee06162640 1 --2- 172.21.15.169:0/390077246 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fee00072950 0x7fee00072d70 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009860 tx=0x7fedf002bb90 comp rx=0 tx=0).stop 2024-03-28T00:58:54.085 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.082+0000 7fee06162640 1 -- 172.21.15.169:0/390077246 shutdown_connections 2024-03-28T00:58:54.085 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.082+0000 7fee06162640 1 --2- 172.21.15.169:0/390077246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee00109390 0x7fee0010e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.086 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.082+0000 7fee06162640 1 --2- 172.21.15.169:0/390077246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee000732b0 0x7fee00108cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.086 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.082+0000 7fee06162640 1 --2- 172.21.15.169:0/390077246 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fee00072950 0x7fee00072d70 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.086 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.082+0000 7fee06162640 1 -- 172.21.15.169:0/390077246 >> 172.21.15.169:0/390077246 conn(0x7fee0006e510 msgr2=0x7fee00070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:54.098 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.095+0000 7fee06162640 1 -- 172.21.15.169:0/390077246 shutdown_connections 2024-03-28T00:58:54.100 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.095+0000 7fee06162640 1 -- 172.21.15.169:0/390077246 wait complete. 2024-03-28T00:58:54.100 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.097+0000 7fee06162640 1 Processor -- start 2024-03-28T00:58:54.100 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.097+0000 7fee06162640 1 -- start start 2024-03-28T00:58:54.101 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.098+0000 7fee06162640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee00072950 0x7fee00115030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:54.101 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.098+0000 7fee06162640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee00109390 0x7fee00115570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:54.101 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.098+0000 7fee06162640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fee001b4730 0x7fee001b6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:54.101 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.098+0000 7fee06162640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0011c030 con 0x7fee001b4730 2024-03-28T00:58:54.101 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.098+0000 7fee06162640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0011beb0 con 0x7fee00072950 2024-03-28T00:58:54.101 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.098+0000 7fee06162640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0011c1b0 con 0x7fee00109390 2024-03-28T00:58:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.100+0000 7fee05961640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fee001b4730 0x7fee001b6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.100+0000 7fee05961640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fee001b4730 0x7fee001b6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:42764/0 (socket says 172.21.15.169:42764) 2024-03-28T00:58:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.100+0000 7fee05160640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee00072950 0x7fee00115030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.100+0000 7fee05961640 1 -- 172.21.15.169:0/3571764958 learned_addr learned my addr 172.21.15.169:0/3571764958 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:54.106 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.100+0000 7fee05160640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee00072950 0x7fee00115030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:45618/0 (socket says 172.21.15.169:45618) 2024-03-28T00:58:54.107 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.101+0000 7fee0495f640 1 --2- 172.21.15.169:0/3571764958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee00109390 0x7fee00115570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:54.112 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.105+0000 7fee05160640 1 -- 172.21.15.169:0/3571764958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee00109390 msgr2=0x7fee00115570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:54.112 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.105+0000 7fee05160640 1 --2- 172.21.15.169:0/3571764958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee00109390 0x7fee00115570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.112 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.105+0000 7fee05160640 1 -- 172.21.15.169:0/3571764958 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fee001b4730 msgr2=0x7fee001b6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:54.112 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.105+0000 7fee05160640 1 --2- 172.21.15.169:0/3571764958 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fee001b4730 0x7fee001b6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.112 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.105+0000 7fee05160640 1 -- 172.21.15.169:0/3571764958 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee001b7220 con 0x7fee00072950 2024-03-28T00:58:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.110+0000 7fee05160640 1 --2- 172.21.15.169:0/3571764958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee00072950 0x7fee00115030 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009780 tx=0x7fedf00349e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.112+0000 7fedf67fc640 1 -- 172.21.15.169:0/3571764958 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0039040 con 0x7fee00072950 2024-03-28T00:58:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.112+0000 7fee06162640 1 -- 172.21.15.169:0/3571764958 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee001b74b0 con 0x7fee00072950 2024-03-28T00:58:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.112+0000 7fee06162640 1 -- 172.21.15.169:0/3571764958 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee001b7940 con 0x7fee00072950 2024-03-28T00:58:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.112+0000 7fedf67fc640 1 -- 172.21.15.169:0/3571764958 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedf0032b20 con 0x7fee00072950 2024-03-28T00:58:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.112+0000 7fedf67fc640 1 -- 172.21.15.169:0/3571764958 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0003e60 con 0x7fee00072950 2024-03-28T00:58:54.121 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.118+0000 7fedf67fc640 1 -- 172.21.15.169:0/3571764958 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fedf0044020 con 0x7fee00072950 2024-03-28T00:58:54.121 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.119+0000 7fedf67fc640 1 --2- 172.21.15.169:0/3571764958 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fedec0a2a00 0x7fedec0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:54.123 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.119+0000 7fedf67fc640 1 -- 172.21.15.169:0/3571764958 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fedf00f0a30 con 0x7fee00072950 2024-03-28T00:58:54.128 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.125+0000 7fee09fac740 1 Processor -- start 2024-03-28T00:58:54.129 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.126+0000 7fee0495f640 1 --2- 172.21.15.169:0/3571764958 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fedec0a2a00 0x7fedec0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:54.129 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.126+0000 7fee09fac740 1 -- start start 2024-03-28T00:58:54.130 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.126+0000 7fee09fac740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f1b20 0x564e787f1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:54.130 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.127+0000 7fee09fac740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e787f2510 0x564e787f7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:54.130 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.127+0000 7fee09fac740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e787f7680 0x564e787f9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:54.130 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.127+0000 7fee09fac740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e78756b10 con 0x564e787f7680 2024-03-28T00:58:54.130 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.127+0000 7fee09fac740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e786ff900 con 0x564e787f1b20 2024-03-28T00:58:54.130 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.127+0000 7fee09fac740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e78752bb0 con 0x564e787f2510 2024-03-28T00:58:54.131 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.127+0000 7fee0495f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e787f2510 0x564e787f7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:54.131 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.127+0000 7fee05160640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f1b20 0x564e787f1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:54.131 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.128+0000 7fee05160640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f1b20 0x564e787f1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:45632/0 (socket says 172.21.15.169:45632) 2024-03-28T00:58:54.131 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.128+0000 7fee05160640 1 -- 172.21.15.169:0/1324924685 learned_addr learned my addr 172.21.15.169:0/1324924685 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:54.131 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.128+0000 7fee0495f640 1 --2- 172.21.15.169:0/3571764958 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fedec0a2a00 0x7fedec0a4ec0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fee00116540 tx=0x7fedf800d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.128+0000 7fee05160640 1 -- 172.21.15.169:0/1324924685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e787f2510 msgr2=0x564e787f7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.128+0000 7fee05160640 1 --2- 172.21.15.169:0/1324924685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e787f2510 0x564e787f7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.128+0000 7fee05160640 1 -- 172.21.15.169:0/1324924685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e787f7680 msgr2=0x564e787f9b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T00:58:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.128+0000 7fee05160640 1 --2- 172.21.15.169:0/1324924685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e787f7680 0x564e787f9b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.128+0000 7fee05160640 1 -- 172.21.15.169:0/1324924685 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e7846b210 con 0x564e787f1b20 2024-03-28T00:58:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.134+0000 7fee05160640 1 --2- 172.21.15.169:0/1324924685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f1b20 0x564e787f1f40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fedf002c1a0 tx=0x7fedf00bc930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21563ef1c5246bcd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.138+0000 7fedeb7fe640 1 -- 172.21.15.169:0/1324924685 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf00bcfd0 con 0x564e787f1b20 2024-03-28T00:58:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.138+0000 7fedeb7fe640 1 -- 172.21.15.169:0/1324924685 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedf00c1120 con 0x564e787f1b20 2024-03-28T00:58:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.139+0000 7fee09fac740 1 -- 172.21.15.169:0/1324924685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f1b20 msgr2=0x564e787f1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.139+0000 7fee09fac740 1 --2- 172.21.15.169:0/1324924685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f1b20 0x564e787f1f40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fedf002c1a0 tx=0x7fedf00bc930 comp rx=0 tx=0).stop 2024-03-28T00:58:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.143+0000 7fee09fac740 1 -- 172.21.15.169:0/1324924685 shutdown_connections 2024-03-28T00:58:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.143+0000 7fee09fac740 1 --2- 172.21.15.169:0/1324924685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e787f7680 0x564e787f9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.143+0000 7fee09fac740 1 --2- 172.21.15.169:0/1324924685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e787f2510 0x564e787f7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.143+0000 7fee09fac740 1 --2- 172.21.15.169:0/1324924685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f1b20 0x564e787f1f40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.143+0000 7fee09fac740 1 -- 172.21.15.169:0/1324924685 >> 172.21.15.169:0/1324924685 conn(0x564e787ed090 msgr2=0x564e787ed490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:54.153 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.147+0000 7fee09fac740 1 -- 172.21.15.169:0/1324924685 shutdown_connections 2024-03-28T00:58:54.153 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.150+0000 7fee09fac740 1 -- 172.21.15.169:0/1324924685 wait complete. 2024-03-28T00:58:54.153 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.150+0000 7fee09fac740 1 -- start start 2024-03-28T00:58:54.153 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.150+0000 7fee09fac740 1 Processor -- start 2024-03-28T00:58:54.154 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.151+0000 7fee09fac740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e787f1b20 0x564e787fbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.151+0000 7fee09fac740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f2510 0x564e787fc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.151+0000 7fee09fac740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e787f7680 0x564e78803040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.151+0000 7fee09fac740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e787fa3f0 con 0x564e787f1b20 2024-03-28T00:58:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.151+0000 7fee09fac740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e78712d50 con 0x564e787f2510 2024-03-28T00:58:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.151+0000 7fee09fac740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e787fa570 con 0x564e787f7680 2024-03-28T00:58:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.151+0000 7fee09fac740 10 client.0 _ll_register_callbacks cb 0x564e78712720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:58:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.152+0000 7fee05160640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e787f1b20 0x564e787fbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.152+0000 7fee05160640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e787f1b20 0x564e787fbac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:42780/0 (socket says 172.21.15.169:42780) 2024-03-28T00:58:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.152+0000 7fee05160640 1 -- 172.21.15.169:0/4257101295 learned_addr learned my addr 172.21.15.169:0/4257101295 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:54.156 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.153+0000 7fee05160640 1 -- 172.21.15.169:0/4257101295 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e787f7680 msgr2=0x564e78803040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:54.156 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.153+0000 7fee05160640 1 --2- 172.21.15.169:0/4257101295 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e787f7680 0x564e78803040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.156 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.153+0000 7fee05160640 1 -- 172.21.15.169:0/4257101295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f2510 msgr2=0x564e787fc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:54.156 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.153+0000 7fee05160640 1 --2- 172.21.15.169:0/4257101295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f2510 0x564e787fc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.156 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.153+0000 7fee05160640 1 -- 172.21.15.169:0/4257101295 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564e78537620 con 0x564e787f1b20 2024-03-28T00:58:54.157 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.154+0000 7fee05160640 1 --2- 172.21.15.169:0/4257101295 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e787f1b20 0x564e787fbac0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fedf00b99a0 tx=0x7fedf0032ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8571447588f3d72 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:54.157 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.154+0000 7fede8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:58:54.157 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.154+0000 7fede8ff9640 1 -- 172.21.15.169:0/4257101295 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf00c5770 con 0x564e787f1b20 2024-03-28T00:58:54.157 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.155+0000 7fee09fac740 1 -- 172.21.15.169:0/4257101295 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fedf00c1120 con 0x564e787f1b20 2024-03-28T00:58:54.158 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.155+0000 7fede8ff9640 1 -- 172.21.15.169:0/4257101295 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fedf00ee1b0 con 0x564e787f1b20 2024-03-28T00:58:54.171 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.167+0000 7fede8ff9640 1 -- 172.21.15.169:0/4257101295 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedf00bcfd0 con 0x564e787f1b20 2024-03-28T00:58:54.171 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.168+0000 7fee09fac740 10 client.16050 fetch_fsmap learned FSMap version 25 2024-03-28T00:58:54.171 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.168+0000 7fee09fac740 1 -- 172.21.15.169:0/4257101295 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x564e787fa3f0 con 0x564e787f1b20 2024-03-28T00:58:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.169+0000 7fede8ff9640 1 -- 172.21.15.169:0/4257101295 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedf00c1120 con 0x564e787f1b20 2024-03-28T00:58:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.169+0000 7fede8ff9640 1 -- 172.21.15.169:0/4257101295 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fedf003c070 con 0x564e787f1b20 2024-03-28T00:58:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.170+0000 7fee09fac740 10 client.16050 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:58:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.170+0000 7fee09fac740 10 client.16050 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:58:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.170+0000 7fee09fac740 20 client.16050 populate_metadata read hostname 'smithi169' 2024-03-28T00:58:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.170+0000 7fee09fac740 1 --2- 172.21.15.169:0/4257101295 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564e78807a70 0x564e78809f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.170+0000 7fee09fac740 4 client.16050 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:58:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.170+0000 7fee09fac740 1 -- 172.21.15.169:0/4257101295 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564e78764050 con 0x564e78807a70 2024-03-28T00:58:54.185 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.182+0000 7fee0495f640 1 --2- 172.21.15.169:0/4257101295 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564e78807a70 0x564e78809f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:54.210 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.206+0000 7fee0495f640 1 --2- 172.21.15.169:0/4257101295 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564e78807a70 0x564e78809f30 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9b61140397777c5 server_cookie=981cfcdacdc78fe1 in_seq=0 out_seq=0 2024-03-28T00:58:54.211 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.206+0000 7fede8ff9640 10 client.16050 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:54.219 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.213+0000 7fede8ff9640 1 -- 172.21.15.169:0/4257101295 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564e78764050 con 0x564e78807a70 2024-03-28T00:58:54.219 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.213+0000 7fede8ff9640 10 client.16050 handle_command_reply: tid=0 2024-03-28T00:58:54.219 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.213+0000 7fee09fac740 1 client.16050 shutdown 2024-03-28T00:58:54.219 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.214+0000 7fee09fac740 1 -- 172.21.15.169:0/4257101295 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e787f1b20 msgr2=0x564e787fbac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:54.219 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.214+0000 7fee09fac740 1 --2- 172.21.15.169:0/4257101295 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e787f1b20 0x564e787fbac0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fedf00b99a0 tx=0x7fedf0032ef0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.219 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.215+0000 7fee09fac740 1 -- 172.21.15.169:0/4257101295 shutdown_connections 2024-03-28T00:58:54.219 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.215+0000 7fee09fac740 1 --2- 172.21.15.169:0/4257101295 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e787f7680 0x564e78803040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.219 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.215+0000 7fee09fac740 1 --2- 172.21.15.169:0/4257101295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f2510 0x564e787fc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.219 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.215+0000 7fee09fac740 1 --2- 172.21.15.169:0/4257101295 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564e78807a70 0x564e78809f30 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.219 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.215+0000 7fee09fac740 1 --2- 172.21.15.169:0/4257101295 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e787f1b20 0x564e787fbac0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.219 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.215+0000 7fee09fac740 1 -- 172.21.15.169:0/4257101295 >> 172.21.15.169:0/4257101295 conn(0x564e787ed090 msgr2=0x564e787ec290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:54.219 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.216+0000 7fede8ff9640 0 client.16050 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:54.220 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.217+0000 7fee09fac740 1 -- 172.21.15.169:0/4257101295 shutdown_connections 2024-03-28T00:58:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.218+0000 7fee09fac740 1 -- 172.21.15.169:0/4257101295 wait complete. 2024-03-28T00:58:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.220+0000 7fee09fac740 20 client.16050 trim_cache size 0 max 16384 2024-03-28T00:58:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.279+0000 7fee09fac740 1 Processor -- start 2024-03-28T00:58:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.279+0000 7fee09fac740 1 -- start start 2024-03-28T00:58:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.279+0000 7fee09fac740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e787f1b20 0x564e787f02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.280+0000 7fee09fac740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e787f2510 0x564e787f0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.280+0000 7fee09fac740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f7680 0x564e78803000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.280+0000 7fee09fac740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede4012030 con 0x564e787f2510 2024-03-28T00:58:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.280+0000 7fee09fac740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede400fb30 con 0x564e787f7680 2024-03-28T00:58:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.280+0000 7fee09fac740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede400dbd0 con 0x564e787f1b20 2024-03-28T00:58:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.285+0000 7fee05160640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e787f1b20 0x564e787f02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.285+0000 7fee05961640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f7680 0x564e78803000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:54.289 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.286+0000 7fee05961640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f7680 0x564e78803000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:45644/0 (socket says 172.21.15.169:45644) 2024-03-28T00:58:54.289 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.286+0000 7fee05961640 1 -- 172.21.15.169:0/3858259831 learned_addr learned my addr 172.21.15.169:0/3858259831 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:54.290 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.287+0000 7fee0495f640 1 --2- 172.21.15.169:0/3858259831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e787f2510 0x564e787f0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:54.292 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.289+0000 7fee05160640 1 -- 172.21.15.169:0/3858259831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f7680 msgr2=0x564e78803000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:54.292 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.289+0000 7fee05160640 1 --2- 172.21.15.169:0/3858259831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f7680 0x564e78803000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.293 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.289+0000 7fee05160640 1 -- 172.21.15.169:0/3858259831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e787f2510 msgr2=0x564e787f0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:54.293 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.290+0000 7fee05160640 1 --2- 172.21.15.169:0/3858259831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e787f2510 0x564e787f0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.293 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.290+0000 7fee05160640 1 -- 172.21.15.169:0/3858259831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e787fa6b0 con 0x564e787f1b20 2024-03-28T00:58:54.295 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.293+0000 7fee05160640 1 --2- 172.21.15.169:0/3858259831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e787f1b20 0x564e787f02f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fedf002e040 tx=0x7fedf00bc930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf72118f55f8e742 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.296+0000 7fedeaffd640 1 -- 172.21.15.169:0/3858259831 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf00c5770 con 0x564e787f1b20 2024-03-28T00:58:54.300 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.297+0000 7fedeaffd640 1 -- 172.21.15.169:0/3858259831 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedf00c1120 con 0x564e787f1b20 2024-03-28T00:58:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.298+0000 7fedeaffd640 1 -- 172.21.15.169:0/3858259831 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf00b7b40 con 0x564e787f1b20 2024-03-28T00:58:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.299+0000 7fee09fac740 1 -- 172.21.15.169:0/3858259831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e787f1b20 msgr2=0x564e787f02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:58:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.299+0000 7fee09fac740 1 --2- 172.21.15.169:0/3858259831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e787f1b20 0x564e787f02f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fedf002e040 tx=0x7fedf00bc930 comp rx=0 tx=0).stop 2024-03-28T00:58:54.303 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.300+0000 7fee09fac740 1 -- 172.21.15.169:0/3858259831 shutdown_connections 2024-03-28T00:58:54.303 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.300+0000 7fee09fac740 1 --2- 172.21.15.169:0/3858259831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f7680 0x564e78803000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.303 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.300+0000 7fee09fac740 1 --2- 172.21.15.169:0/3858259831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e787f2510 0x564e787f0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.303 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.300+0000 7fee09fac740 1 --2- 172.21.15.169:0/3858259831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e787f1b20 0x564e787f02f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.303 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.300+0000 7fee09fac740 1 -- 172.21.15.169:0/3858259831 >> 172.21.15.169:0/3858259831 conn(0x564e787ed090 msgr2=0x564e787ed470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:58:54.303 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.300+0000 7fee09fac740 1 -- 172.21.15.169:0/3858259831 shutdown_connections 2024-03-28T00:58:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.300+0000 7fee09fac740 1 -- 172.21.15.169:0/3858259831 wait complete. 2024-03-28T00:58:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.301+0000 7fee09fac740 1 -- start start 2024-03-28T00:58:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.301+0000 7fee09fac740 1 Processor -- start 2024-03-28T00:58:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.302+0000 7fee09fac740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e787f2510 0x564e78800010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.302+0000 7fee09fac740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f7680 0x564e78800550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.302+0000 7fee09fac740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e7885f3a0 0x564e78861790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.302+0000 7fee09fac740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede400e1d0 con 0x564e787f2510 2024-03-28T00:58:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.302+0000 7fee09fac740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede400f4d0 con 0x564e787f7680 2024-03-28T00:58:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.302+0000 7fee09fac740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede400e810 con 0x564e7885f3a0 2024-03-28T00:58:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.302+0000 7fee09fac740 10 client.0 _ll_register_callbacks cb 0x7fedf00bce70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:58:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.303+0000 7fee05961640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e7885f3a0 0x564e78861790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.303+0000 7fee05961640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e7885f3a0 0x564e78861790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:34060/0 (socket says 172.21.15.169:34060) 2024-03-28T00:58:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.303+0000 7fee05961640 1 -- 172.21.15.169:0/962273427 learned_addr learned my addr 172.21.15.169:0/962273427 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:58:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.304+0000 7fee0495f640 1 --2- 172.21.15.169:0/962273427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f7680 0x564e78800550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.304+0000 7fee05160640 1 --2- 172.21.15.169:0/962273427 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e787f2510 0x564e78800010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.304+0000 7fee05961640 1 -- 172.21.15.169:0/962273427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f7680 msgr2=0x564e78800550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.304+0000 7fee05961640 1 --2- 172.21.15.169:0/962273427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f7680 0x564e78800550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.304+0000 7fee05961640 1 -- 172.21.15.169:0/962273427 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e787f2510 msgr2=0x564e78800010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:58:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.304+0000 7fee05961640 1 --2- 172.21.15.169:0/962273427 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e787f2510 0x564e78800010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:58:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.304+0000 7fee05961640 1 -- 172.21.15.169:0/962273427 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564e78861d00 con 0x564e7885f3a0 2024-03-28T00:58:54.312 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.309+0000 7fee05961640 1 --2- 172.21.15.169:0/962273427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e7885f3a0 0x564e78861790 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fedfc012840 tx=0x7fedfc012d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c17e400689452c58 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:58:54.312 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.309+0000 7fede8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:58:54.313 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.309+0000 7fede8ff9640 1 -- 172.21.15.169:0/962273427 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc00f820 con 0x564e7885f3a0 2024-03-28T00:58:54.313 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.310+0000 7fee09fac740 1 -- 172.21.15.169:0/962273427 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fedf00c1120 con 0x564e7885f3a0 2024-03-28T00:58:54.313 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.310+0000 7fede8ff9640 1 -- 172.21.15.169:0/962273427 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fedfc00f9c0 con 0x564e7885f3a0 2024-03-28T00:58:54.315 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.312+0000 7fede8ff9640 1 -- 172.21.15.169:0/962273427 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedfc00fbc0 con 0x564e7885f3a0 2024-03-28T00:58:54.315 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.313+0000 7fee09fac740 10 client.25861 fetch_fsmap learned FSMap version 25 2024-03-28T00:58:54.315 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.313+0000 7fee09fac740 1 -- 172.21.15.169:0/962273427 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x564e78863130 con 0x564e7885f3a0 2024-03-28T00:58:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.313+0000 7fede8ff9640 1 -- 172.21.15.169:0/962273427 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedf00c1120 con 0x564e7885f3a0 2024-03-28T00:58:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.316+0000 7fede8ff9640 1 -- 172.21.15.169:0/962273427 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fedfc01c420 con 0x564e7885f3a0 2024-03-28T00:58:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.316+0000 7fee09fac740 10 client.25861 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:58:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.316+0000 7fee09fac740 10 client.25861 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:58:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.316+0000 7fee09fac740 20 client.25861 populate_metadata read hostname 'smithi169' 2024-03-28T00:58:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.316+0000 7fee09fac740 1 --2- 172.21.15.169:0/962273427 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564e78863a90 0x564e78865eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:58:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.316+0000 7fee09fac740 4 client.25861 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T00:58:54.320 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.316+0000 7fee09fac740 1 -- 172.21.15.169:0/962273427 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564e7875e3a0 con 0x564e78863a90 2024-03-28T00:58:54.322 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.319+0000 7fee05160640 1 --2- 172.21.15.169:0/962273427 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564e78863a90 0x564e78865eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:58:54.328 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.325+0000 7fede8ff9640 10 client.25861 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:58:54.328 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:58:54.325+0000 7fee05160640 1 --2- 172.21.15.169:0/962273427 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564e78863a90 0x564e78865eb0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff75ffe53a124104 server_cookie=90755b17cf8bfca9 in_seq=0 out_seq=0 2024-03-28T00:58:55.555 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:55 smithi169 ceph-mon[32274]: pgmap v423: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 767 MiB/s wr, 1.54k op/s 2024-03-28T00:58:55.610 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:55 smithi174 ceph-mon[36060]: pgmap v423: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 767 MiB/s wr, 1.54k op/s 2024-03-28T00:58:55.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:55 smithi177 ceph-mon[36206]: pgmap v423: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 767 MiB/s wr, 1.54k op/s 2024-03-28T00:58:57.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:57 smithi169 ceph-mon[32274]: pgmap v424: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 865 MiB/s wr, 1.71k op/s 2024-03-28T00:58:57.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:57 smithi174 ceph-mon[36060]: pgmap v424: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 865 MiB/s wr, 1.71k op/s 2024-03-28T00:58:57.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:57 smithi177 ceph-mon[36206]: pgmap v424: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 865 MiB/s wr, 1.71k op/s 2024-03-28T00:58:59.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:58:59 smithi174 ceph-mon[36060]: pgmap v425: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 801 MiB/s wr, 1.63k op/s 2024-03-28T00:58:59.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:58:59 smithi177 ceph-mon[36206]: pgmap v425: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 801 MiB/s wr, 1.63k op/s 2024-03-28T00:58:59.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:58:59 smithi169 ceph-mon[32274]: pgmap v425: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 801 MiB/s wr, 1.63k op/s 2024-03-28T00:59:00.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:00 smithi177 ceph-mon[36206]: pgmap v426: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 743 MiB/s wr, 1.50k op/s 2024-03-28T00:59:00.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:00 smithi169 ceph-mon[32274]: pgmap v426: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 743 MiB/s wr, 1.50k op/s 2024-03-28T00:59:00.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:00 smithi174 ceph-mon[36060]: pgmap v426: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 743 MiB/s wr, 1.50k op/s 2024-03-28T00:59:01.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.757+0000 7fede8ff9640 1 -- 172.21.15.169:0/962273427 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564e7875e3a0 con 0x564e78863a90 2024-03-28T00:59:01.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.757+0000 7fede8ff9640 10 client.25861 handle_command_reply: tid=0 2024-03-28T00:59:01.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.757+0000 7fee09fac740 1 client.25861 shutdown 2024-03-28T00:59:01.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.757+0000 7fee09fac740 1 -- 172.21.15.169:0/962273427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e7885f3a0 msgr2=0x564e78861790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:01.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.757+0000 7fee09fac740 1 --2- 172.21.15.169:0/962273427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e7885f3a0 0x564e78861790 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fedfc012840 tx=0x7fedfc012d00 comp rx=0 tx=0).stop 2024-03-28T00:59:01.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.758+0000 7fee09fac740 1 -- 172.21.15.169:0/962273427 shutdown_connections 2024-03-28T00:59:01.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.758+0000 7fee09fac740 1 --2- 172.21.15.169:0/962273427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e7885f3a0 0x564e78861790 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:01.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.758+0000 7fee09fac740 1 --2- 172.21.15.169:0/962273427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e787f7680 0x564e78800550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:01.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.758+0000 7fee09fac740 1 --2- 172.21.15.169:0/962273427 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564e78863a90 0x564e78865eb0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.758+0000 7fee09fac740 1 --2- 172.21.15.169:0/962273427 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564e787f2510 0x564e78800010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.758+0000 7fee09fac740 1 -- 172.21.15.169:0/962273427 >> 172.21.15.169:0/962273427 conn(0x564e787ed090 msgr2=0x564e787f3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.758+0000 7fede8ff9640 0 client.25861 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.758+0000 7fee09fac740 1 -- 172.21.15.169:0/962273427 shutdown_connections 2024-03-28T00:59:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.758+0000 7fee09fac740 1 -- 172.21.15.169:0/962273427 wait complete. 2024-03-28T00:59:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.758+0000 7fee09fac740 20 client.25861 trim_cache size 0 max 16384 2024-03-28T00:59:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T00:59:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T00:59:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T00:59:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T00:59:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.767+0000 7fee06162640 1 -- 172.21.15.169:0/3571764958 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fedec0a2a00 msgr2=0x7fedec0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.767+0000 7fee06162640 1 --2- 172.21.15.169:0/3571764958 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fedec0a2a00 0x7fedec0a4ec0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fee00116540 tx=0x7fedf800d040 comp rx=0 tx=0).stop 2024-03-28T00:59:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.767+0000 7fee06162640 1 -- 172.21.15.169:0/3571764958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee00072950 msgr2=0x7fee00115030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.767+0000 7fee06162640 1 --2- 172.21.15.169:0/3571764958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee00072950 0x7fee00115030 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009780 tx=0x7fedf00349e0 comp rx=0 tx=0).stop 2024-03-28T00:59:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.767+0000 7fee06162640 1 -- 172.21.15.169:0/3571764958 shutdown_connections 2024-03-28T00:59:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.767+0000 7fee06162640 1 --2- 172.21.15.169:0/3571764958 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fedec0a2a00 0x7fedec0a4ec0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.767+0000 7fee06162640 1 --2- 172.21.15.169:0/3571764958 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fee001b4730 0x7fee001b6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.767+0000 7fee06162640 1 --2- 172.21.15.169:0/3571764958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee00109390 0x7fee00115570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.767+0000 7fee06162640 1 --2- 172.21.15.169:0/3571764958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee00072950 0x7fee00115030 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.767+0000 7fee06162640 1 -- 172.21.15.169:0/3571764958 >> 172.21.15.169:0/3571764958 conn(0x7fee0006e510 msgr2=0x7fee00070360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:01.771 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.768+0000 7fee06162640 1 -- 172.21.15.169:0/3571764958 shutdown_connections 2024-03-28T00:59:01.771 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:01.768+0000 7fee06162640 1 -- 172.21.15.169:0/3571764958 wait complete. 2024-03-28T00:59:03.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:03 smithi169 ceph-mon[32274]: pgmap v427: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 892 MiB/s wr, 1.86k op/s 2024-03-28T00:59:03.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:03 smithi174 ceph-mon[36060]: pgmap v427: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 892 MiB/s wr, 1.86k op/s 2024-03-28T00:59:03.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:03 smithi177 ceph-mon[36206]: pgmap v427: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 892 MiB/s wr, 1.86k op/s 2024-03-28T00:59:05.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:05 smithi169 ceph-mon[32274]: pgmap v428: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 774 MiB/s wr, 1.71k op/s 2024-03-28T00:59:05.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:05 smithi174 ceph-mon[36060]: pgmap v428: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 774 MiB/s wr, 1.71k op/s 2024-03-28T00:59:05.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:05 smithi177 ceph-mon[36206]: pgmap v428: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 774 MiB/s wr, 1.71k op/s 2024-03-28T00:59:06.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:06 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:59:06.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:06 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:59:06.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:06 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T00:59:07.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:07 smithi174 ceph-mon[36060]: pgmap v429: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 864 MiB/s wr, 1.93k op/s 2024-03-28T00:59:07.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:07 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:59:07.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:07 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:59:07.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:07 smithi177 ceph-mon[36206]: pgmap v429: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 864 MiB/s wr, 1.93k op/s 2024-03-28T00:59:07.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:07 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:59:07.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:07 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:59:07.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:07 smithi169 ceph-mon[32274]: pgmap v429: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 864 MiB/s wr, 1.93k op/s 2024-03-28T00:59:07.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:07 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:59:07.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:07 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:59:09.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:09 smithi174 ceph-mon[36060]: pgmap v430: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 772 MiB/s wr, 1.68k op/s 2024-03-28T00:59:09.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:09 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:59:09.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:09 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:59:09.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:09 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:59:09.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:09 smithi177 ceph-mon[36206]: pgmap v430: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 772 MiB/s wr, 1.68k op/s 2024-03-28T00:59:09.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:09 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:59:09.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:09 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:59:09.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:09 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:59:09.806 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:09 smithi169 ceph-mon[32274]: pgmap v430: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 772 MiB/s wr, 1.68k op/s 2024-03-28T00:59:09.806 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:09 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T00:59:09.806 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:09 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T00:59:09.806 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:09 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T00:59:10.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:10 smithi177 ceph-mon[36206]: pgmap v431: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 748 MiB/s wr, 1.55k op/s 2024-03-28T00:59:10.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:10 smithi169 ceph-mon[32274]: pgmap v431: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 748 MiB/s wr, 1.55k op/s 2024-03-28T00:59:10.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:10 smithi174 ceph-mon[36060]: pgmap v431: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 748 MiB/s wr, 1.55k op/s 2024-03-28T00:59:13.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:13 smithi169 ceph-mon[32274]: pgmap v432: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 863 MiB/s wr, 1.79k op/s 2024-03-28T00:59:13.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:13 smithi174 ceph-mon[36060]: pgmap v432: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 863 MiB/s wr, 1.79k op/s 2024-03-28T00:59:13.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:13 smithi177 ceph-mon[36206]: pgmap v432: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 863 MiB/s wr, 1.79k op/s 2024-03-28T00:59:15.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.111+0000 7fa6c7fff640 1 -- 172.21.15.169:0/3533106502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6c806ef70 msgr2=0x7fa6c806f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:15.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.111+0000 7fa6c7fff640 1 --2- 172.21.15.169:0/3533106502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6c806ef70 0x7fa6c806f810 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0009900 tx=0x7fa6c002bc40 comp rx=0 tx=0).stop 2024-03-28T00:59:15.121 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.119+0000 7fa6c7fff640 1 -- 172.21.15.169:0/3533106502 shutdown_connections 2024-03-28T00:59:15.121 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.119+0000 7fa6c7fff640 1 --2- 172.21.15.169:0/3533106502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6c806ef70 0x7fa6c806f810 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.121 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.119+0000 7fa6c7fff640 1 --2- 172.21.15.169:0/3533106502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6c806e5d0 0x7fa6c806ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.121 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.119+0000 7fa6c7fff640 1 --2- 172.21.15.169:0/3533106502 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6c806d390 0x7fa6c806d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.119+0000 7fa6c7fff640 1 -- 172.21.15.169:0/3533106502 >> 172.21.15.169:0/3533106502 conn(0x7fa6c806bcf0 msgr2=0x7fa6c806c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.125+0000 7fa6c7fff640 1 -- 172.21.15.169:0/3533106502 shutdown_connections 2024-03-28T00:59:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.125+0000 7fa6c7fff640 1 -- 172.21.15.169:0/3533106502 wait complete. 2024-03-28T00:59:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.127+0000 7fa6c7fff640 1 Processor -- start 2024-03-28T00:59:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.130+0000 7fa6c7fff640 1 -- start start 2024-03-28T00:59:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.131+0000 7fa6c7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6c806d390 0x7fa6c81a6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.131+0000 7fa6c7fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6c806e5d0 0x7fa6c81a6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.131+0000 7fa6c7fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6c806ef70 0x7fa6c81adc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.131+0000 7fa6c7fff640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c811bfe0 con 0x7fa6c806e5d0 2024-03-28T00:59:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.131+0000 7fa6c7fff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c811be60 con 0x7fa6c806d390 2024-03-28T00:59:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.131+0000 7fa6c7fff640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c811c160 con 0x7fa6c806ef70 2024-03-28T00:59:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.132+0000 7fa6c6ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6c806d390 0x7fa6c81a6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.132+0000 7fa6c6ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6c806d390 0x7fa6c81a6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:44584/0 (socket says 172.21.15.169:44584) 2024-03-28T00:59:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.132+0000 7fa6c6ffd640 1 -- 172.21.15.169:0/3706155071 learned_addr learned my addr 172.21.15.169:0/3706155071 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:15.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.133+0000 7fa6c67fc640 1 --2- 172.21.15.169:0/3706155071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6c806e5d0 0x7fa6c81a6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:15.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.133+0000 7fa6c6ffd640 1 -- 172.21.15.169:0/3706155071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6c806ef70 msgr2=0x7fa6c81adc00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T00:59:15.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.133+0000 7fa6c6ffd640 1 --2- 172.21.15.169:0/3706155071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6c806ef70 0x7fa6c81adc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.133+0000 7fa6c6ffd640 1 -- 172.21.15.169:0/3706155071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6c806e5d0 msgr2=0x7fa6c81a6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:15.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.133+0000 7fa6c6ffd640 1 --2- 172.21.15.169:0/3706155071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6c806e5d0 0x7fa6c81a6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.133+0000 7fa6c6ffd640 1 -- 172.21.15.169:0/3706155071 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c81ae270 con 0x7fa6c806d390 2024-03-28T00:59:15.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.133+0000 7fa6c6ffd640 1 --2- 172.21.15.169:0/3706155071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6c806d390 0x7fa6c81a6650 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa6b800ba50 tx=0x7fa6b800bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.134+0000 7fa6cc8f6640 1 -- 172.21.15.169:0/3706155071 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8019070 con 0x7fa6c806d390 2024-03-28T00:59:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.134+0000 7fa6c7fff640 1 -- 172.21.15.169:0/3706155071 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c81ae560 con 0x7fa6c806d390 2024-03-28T00:59:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.134+0000 7fa6c7fff640 1 -- 172.21.15.169:0/3706155071 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c81aeb00 con 0x7fa6c806d390 2024-03-28T00:59:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.137+0000 7fa6cc8f6640 1 -- 172.21.15.169:0/3706155071 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6b800cbe0 con 0x7fa6c806d390 2024-03-28T00:59:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.137+0000 7fa6cc8f6640 1 -- 172.21.15.169:0/3706155071 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8019070 con 0x7fa6c806d390 2024-03-28T00:59:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.138+0000 7fa6d05bf740 1 Processor -- start 2024-03-28T00:59:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.138+0000 7fa6d05bf740 1 -- start start 2024-03-28T00:59:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.139+0000 7fa6d05bf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46ab20 0x55ae6a46af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.139+0000 7fa6d05bf740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ae6a46b510 0x55ae6a470010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.139+0000 7fa6d05bf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a470680 0x55ae6a472b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.139+0000 7fa6d05bf740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae6a3cfb10 con 0x55ae6a46b510 2024-03-28T00:59:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.139+0000 7fa6d05bf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae6a378900 con 0x55ae6a46ab20 2024-03-28T00:59:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.139+0000 7fa6d05bf740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae6a3cbbb0 con 0x55ae6a470680 2024-03-28T00:59:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.140+0000 7fa6c67fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a470680 0x55ae6a472b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.140+0000 7fa6c67fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a470680 0x55ae6a472b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:37098/0 (socket says 172.21.15.169:37098) 2024-03-28T00:59:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.140+0000 7fa6c67fc640 1 -- 172.21.15.169:0/2001906233 learned_addr learned my addr 172.21.15.169:0/2001906233 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:15.149 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.140+0000 7fa6c77fe640 1 --2- 172.21.15.169:0/2001906233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46ab20 0x55ae6a46af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:15.149 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.143+0000 7fa6c67fc640 1 -- 172.21.15.169:0/2001906233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46ab20 msgr2=0x55ae6a46af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:15.149 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.143+0000 7fa6c67fc640 1 --2- 172.21.15.169:0/2001906233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46ab20 0x55ae6a46af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.149 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.143+0000 7fa6c67fc640 1 -- 172.21.15.169:0/2001906233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ae6a46b510 msgr2=0x55ae6a470010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T00:59:15.149 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.143+0000 7fa6c67fc640 1 --2- 172.21.15.169:0/2001906233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ae6a46b510 0x55ae6a470010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.143+0000 7fa6c67fc640 1 -- 172.21.15.169:0/2001906233 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae6a0e4210 con 0x55ae6a470680 2024-03-28T00:59:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.146+0000 7fa6c67fc640 1 --2- 172.21.15.169:0/2001906233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a470680 0x55ae6a472b10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa6b000bc80 tx=0x7fa6b000f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f225ff8770d3f1a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.147+0000 7fa6bdffb640 1 -- 172.21.15.169:0/2001906233 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b0015070 con 0x55ae6a470680 2024-03-28T00:59:15.150 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.148+0000 7fa6bdffb640 1 -- 172.21.15.169:0/2001906233 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6b001a940 con 0x55ae6a470680 2024-03-28T00:59:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.149+0000 7fa6d05bf740 1 -- 172.21.15.169:0/2001906233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a470680 msgr2=0x55ae6a472b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.149+0000 7fa6d05bf740 1 --2- 172.21.15.169:0/2001906233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a470680 0x55ae6a472b10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa6b000bc80 tx=0x7fa6b000f730 comp rx=0 tx=0).stop 2024-03-28T00:59:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.152+0000 7fa6cc8f6640 1 -- 172.21.15.169:0/3706155071 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa6b801c070 con 0x7fa6c806d390 2024-03-28T00:59:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.154+0000 7fa6d05bf740 1 -- 172.21.15.169:0/2001906233 shutdown_connections 2024-03-28T00:59:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.154+0000 7fa6d05bf740 1 --2- 172.21.15.169:0/2001906233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a470680 0x55ae6a472b10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.154+0000 7fa6d05bf740 1 --2- 172.21.15.169:0/2001906233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ae6a46b510 0x55ae6a470010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.154+0000 7fa6d05bf740 1 --2- 172.21.15.169:0/2001906233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46ab20 0x55ae6a46af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.154+0000 7fa6d05bf740 1 -- 172.21.15.169:0/2001906233 >> 172.21.15.169:0/2001906233 conn(0x55ae6a466090 msgr2=0x55ae6a466490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:15.160 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.157+0000 7fa6d05bf740 1 -- 172.21.15.169:0/2001906233 shutdown_connections 2024-03-28T00:59:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.160+0000 7fa6d05bf740 1 -- 172.21.15.169:0/2001906233 wait complete. 2024-03-28T00:59:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.160+0000 7fa6cc8f6640 1 --2- 172.21.15.169:0/3706155071 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa6b40a2a00 0x7fa6b40a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:15.171 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.160+0000 7fa6d05bf740 1 -- start start 2024-03-28T00:59:15.171 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.160+0000 7fa6cc8f6640 1 -- 172.21.15.169:0/3706155071 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa6b80ce050 con 0x7fa6c806d390 2024-03-28T00:59:15.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.161+0000 7fa6c67fc640 1 --2- 172.21.15.169:0/3706155071 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa6b40a2a00 0x7fa6b40a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:15.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.161+0000 7fa6d05bf740 1 Processor -- start 2024-03-28T00:59:15.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.161+0000 7fa6c67fc640 1 --2- 172.21.15.169:0/3706155071 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa6b40a2a00 0x7fa6b40a4ec0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0011560 tx=0x7fa6b0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:15.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.165+0000 7fa6d05bf740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ae6a46ab20 0x55ae6a485f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:15.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.165+0000 7fa6d05bf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46b510 0x55ae6a486450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:15.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.165+0000 7fa6d05bf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a470680 0x55ae6a48d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:15.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.165+0000 7fa6d05bf740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae6a484690 con 0x55ae6a46ab20 2024-03-28T00:59:15.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.165+0000 7fa6d05bf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae6a38bd50 con 0x55ae6a46b510 2024-03-28T00:59:15.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.165+0000 7fa6d05bf740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae6a484810 con 0x55ae6a470680 2024-03-28T00:59:15.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.165+0000 7fa6d05bf740 10 client.0 _ll_register_callbacks cb 0x55ae6a38b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:59:15.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.168+0000 7fa6c77fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a470680 0x55ae6a48d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:15.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.168+0000 7fa6c77fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a470680 0x55ae6a48d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:37102/0 (socket says 172.21.15.169:37102) 2024-03-28T00:59:15.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.168+0000 7fa6c77fe640 1 -- 172.21.15.169:0/2117289991 learned_addr learned my addr 172.21.15.169:0/2117289991 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:15.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.168+0000 7fa6c67fc640 1 --2- 172.21.15.169:0/2117289991 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46b510 0x55ae6a486450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.168+0000 7fa6c77fe640 1 -- 172.21.15.169:0/2117289991 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46b510 msgr2=0x55ae6a486450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.168+0000 7fa6c77fe640 1 --2- 172.21.15.169:0/2117289991 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46b510 0x55ae6a486450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.168+0000 7fa6c77fe640 1 -- 172.21.15.169:0/2117289991 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ae6a46ab20 msgr2=0x55ae6a485f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T00:59:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.168+0000 7fa6c77fe640 1 --2- 172.21.15.169:0/2117289991 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ae6a46ab20 0x55ae6a485f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.168+0000 7fa6c77fe640 1 -- 172.21.15.169:0/2117289991 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ae6a1b0620 con 0x55ae6a470680 2024-03-28T00:59:15.176 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.170+0000 7fa6c77fe640 1 --2- 172.21.15.169:0/2117289991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a470680 0x55ae6a48d420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa6c002efd0 tx=0x7fa6c0004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e0a5660f9320219 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:15.176 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.170+0000 7fa6a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:59:15.176 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.173+0000 7fa6a6ffd640 1 -- 172.21.15.169:0/2117289991 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0042070 con 0x55ae6a470680 2024-03-28T00:59:15.176 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.173+0000 7fa6a6ffd640 1 -- 172.21.15.169:0/2117289991 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa6c0005200 con 0x55ae6a470680 2024-03-28T00:59:15.177 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.173+0000 7fa6d05bf740 1 -- 172.21.15.169:0/2117289991 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa6b001a940 con 0x55ae6a470680 2024-03-28T00:59:15.177 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.173+0000 7fa6a6ffd640 1 -- 172.21.15.169:0/2117289991 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6c0050050 con 0x55ae6a470680 2024-03-28T00:59:15.177 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.173+0000 7fa6d05bf740 10 client.25870 fetch_fsmap learned FSMap version 25 2024-03-28T00:59:15.177 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.173+0000 7fa6d05bf740 1 -- 172.21.15.169:0/2117289991 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ae6a490620 con 0x55ae6a470680 2024-03-28T00:59:15.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.177+0000 7fa6a6ffd640 1 -- 172.21.15.169:0/2117289991 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6b001a940 con 0x55ae6a470680 2024-03-28T00:59:15.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.178+0000 7fa6a6ffd640 1 -- 172.21.15.169:0/2117289991 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa6c0003800 con 0x55ae6a470680 2024-03-28T00:59:15.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.178+0000 7fa6d05bf740 10 client.25870 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:59:15.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.178+0000 7fa6d05bf740 10 client.25870 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:59:15.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.178+0000 7fa6d05bf740 20 client.25870 populate_metadata read hostname 'smithi169' 2024-03-28T00:59:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.178+0000 7fa6d05bf740 1 --2- 172.21.15.169:0/2117289991 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ae6a491cf0 0x55ae6a494140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.178+0000 7fa6d05bf740 4 client.25870 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:59:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.178+0000 7fa6d05bf740 1 -- 172.21.15.169:0/2117289991 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ae6a3dd050 con 0x55ae6a491cf0 2024-03-28T00:59:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.184+0000 7fa6c6ffd640 1 --2- 172.21.15.169:0/2117289991 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ae6a491cf0 0x55ae6a494140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.192+0000 7fa6c6ffd640 1 --2- 172.21.15.169:0/2117289991 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ae6a491cf0 0x55ae6a494140 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cdb9f993a2914b8 server_cookie=2a80837e8c706cf6 in_seq=0 out_seq=0 2024-03-28T00:59:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.192+0000 7fa6a6ffd640 10 client.25870 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.197+0000 7fa6a6ffd640 1 -- 172.21.15.169:0/2117289991 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ae6a3dd050 con 0x55ae6a491cf0 2024-03-28T00:59:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.197+0000 7fa6a6ffd640 10 client.25870 handle_command_reply: tid=0 2024-03-28T00:59:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.197+0000 7fa6d05bf740 1 client.25870 shutdown 2024-03-28T00:59:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.198+0000 7fa6d05bf740 1 -- 172.21.15.169:0/2117289991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a470680 msgr2=0x55ae6a48d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.198+0000 7fa6d05bf740 1 --2- 172.21.15.169:0/2117289991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a470680 0x55ae6a48d420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa6c002efd0 tx=0x7fa6c0004270 comp rx=0 tx=0).stop 2024-03-28T00:59:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.199+0000 7fa6d05bf740 1 -- 172.21.15.169:0/2117289991 shutdown_connections 2024-03-28T00:59:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.199+0000 7fa6d05bf740 1 --2- 172.21.15.169:0/2117289991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a470680 0x55ae6a48d420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.199+0000 7fa6d05bf740 1 --2- 172.21.15.169:0/2117289991 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46b510 0x55ae6a486450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.199+0000 7fa6d05bf740 1 --2- 172.21.15.169:0/2117289991 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ae6a491cf0 0x55ae6a494140 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.199+0000 7fa6d05bf740 1 --2- 172.21.15.169:0/2117289991 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ae6a46ab20 0x55ae6a485f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.199+0000 7fa6d05bf740 1 -- 172.21.15.169:0/2117289991 >> 172.21.15.169:0/2117289991 conn(0x55ae6a466090 msgr2=0x55ae6a465290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:15.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.202+0000 7fa6a6ffd640 0 client.25870 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:15.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.202+0000 7fa6d05bf740 1 -- 172.21.15.169:0/2117289991 shutdown_connections 2024-03-28T00:59:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.202+0000 7fa6d05bf740 1 -- 172.21.15.169:0/2117289991 wait complete. 2024-03-28T00:59:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.203+0000 7fa6d05bf740 20 client.25870 trim_cache size 0 max 16384 2024-03-28T00:59:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.263+0000 7fa6d05bf740 1 Processor -- start 2024-03-28T00:59:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.264+0000 7fa6d05bf740 1 -- start start 2024-03-28T00:59:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.265+0000 7fa6d05bf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46ab20 0x55ae6a4692f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.265+0000 7fa6d05bf740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ae6a46b510 0x55ae6a48d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.265+0000 7fa6d05bf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a470680 0x55ae6a469830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.265+0000 7fa6d05bf740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a8012120 con 0x55ae6a46b510 2024-03-28T00:59:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.265+0000 7fa6d05bf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a800fc20 con 0x55ae6a46ab20 2024-03-28T00:59:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.265+0000 7fa6d05bf740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a800dcc0 con 0x55ae6a470680 2024-03-28T00:59:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.265+0000 7fa6c6ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46ab20 0x55ae6a4692f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.265+0000 7fa6c6ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46ab20 0x55ae6a4692f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:44602/0 (socket says 172.21.15.169:44602) 2024-03-28T00:59:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.266+0000 7fa6c67fc640 1 --2- 172.21.15.169:0/2735509296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ae6a46b510 0x55ae6a48d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.266+0000 7fa6c77fe640 1 --2- 172.21.15.169:0/2735509296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a470680 0x55ae6a469830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.265+0000 7fa6c6ffd640 1 -- 172.21.15.169:0/2735509296 learned_addr learned my addr 172.21.15.169:0/2735509296 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:15.270 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.267+0000 7fa6c67fc640 1 -- 172.21.15.169:0/2735509296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a470680 msgr2=0x55ae6a469830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:15.270 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.267+0000 7fa6c67fc640 1 --2- 172.21.15.169:0/2735509296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a470680 0x55ae6a469830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.270 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.267+0000 7fa6c67fc640 1 -- 172.21.15.169:0/2735509296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46ab20 msgr2=0x55ae6a4692f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:15.270 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.267+0000 7fa6c67fc640 1 --2- 172.21.15.169:0/2735509296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46ab20 0x55ae6a4692f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.270 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.267+0000 7fa6c67fc640 1 -- 172.21.15.169:0/2735509296 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae6a466e00 con 0x55ae6a46b510 2024-03-28T00:59:15.271 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.268+0000 7fa6c67fc640 1 --2- 172.21.15.169:0/2735509296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ae6a46b510 0x55ae6a48d3e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fa6b00058c0 tx=0x7fa6b001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71483b31b52d2d40 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:15.280 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.277+0000 7fa6bd7fa640 1 -- 172.21.15.169:0/2735509296 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b0015050 con 0x55ae6a46b510 2024-03-28T00:59:15.280 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.278+0000 7fa6bd7fa640 1 -- 172.21.15.169:0/2735509296 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6b000be30 con 0x55ae6a46b510 2024-03-28T00:59:15.282 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.279+0000 7fa6d05bf740 1 -- 172.21.15.169:0/2735509296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ae6a46b510 msgr2=0x55ae6a48d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:15.282 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.279+0000 7fa6d05bf740 1 --2- 172.21.15.169:0/2735509296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ae6a46b510 0x55ae6a48d3e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fa6b00058c0 tx=0x7fa6b001f000 comp rx=0 tx=0).stop 2024-03-28T00:59:15.286 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.284+0000 7fa6d05bf740 1 -- 172.21.15.169:0/2735509296 shutdown_connections 2024-03-28T00:59:15.287 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.284+0000 7fa6d05bf740 1 --2- 172.21.15.169:0/2735509296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a470680 0x55ae6a469830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.287 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.284+0000 7fa6d05bf740 1 --2- 172.21.15.169:0/2735509296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ae6a46b510 0x55ae6a48d3e0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.287 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.284+0000 7fa6d05bf740 1 --2- 172.21.15.169:0/2735509296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46ab20 0x55ae6a4692f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.287 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.284+0000 7fa6d05bf740 1 -- 172.21.15.169:0/2735509296 >> 172.21.15.169:0/2735509296 conn(0x55ae6a466090 msgr2=0x55ae6a46c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:15.290 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.287+0000 7fa6d05bf740 1 -- 172.21.15.169:0/2735509296 shutdown_connections 2024-03-28T00:59:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.287+0000 7fa6d05bf740 1 -- 172.21.15.169:0/2735509296 wait complete. 2024-03-28T00:59:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.288+0000 7fa6d05bf740 1 -- start start 2024-03-28T00:59:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.288+0000 7fa6d05bf740 1 Processor -- start 2024-03-28T00:59:15.292 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.289+0000 7fa6d05bf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46ab20 0x55ae6a48aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:15.292 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.289+0000 7fa6d05bf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a46b510 0x55ae6a48b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:15.292 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.289+0000 7fa6d05bf740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ae6a470680 0x55ae6a48c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:15.292 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.289+0000 7fa6d05bf740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a800e2c0 con 0x55ae6a470680 2024-03-28T00:59:15.293 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.289+0000 7fa6d05bf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a800f5c0 con 0x55ae6a46ab20 2024-03-28T00:59:15.293 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.289+0000 7fa6d05bf740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a800e900 con 0x55ae6a46b510 2024-03-28T00:59:15.293 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.289+0000 7fa6d05bf740 10 client.0 _ll_register_callbacks cb 0x7fa6c002cd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:59:15.293 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.290+0000 7fa6c6ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46ab20 0x55ae6a48aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:15.293 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.290+0000 7fa6c6ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46ab20 0x55ae6a48aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:44618/0 (socket says 172.21.15.169:44618) 2024-03-28T00:59:15.293 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.290+0000 7fa6c6ffd640 1 -- 172.21.15.169:0/3136137604 learned_addr learned my addr 172.21.15.169:0/3136137604 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:15.294 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.291+0000 7fa6c6ffd640 1 -- 172.21.15.169:0/3136137604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a46b510 msgr2=0x55ae6a48b400 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T00:59:15.294 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.291+0000 7fa6c6ffd640 1 --2- 172.21.15.169:0/3136137604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a46b510 0x55ae6a48b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.294 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.291+0000 7fa6c6ffd640 1 -- 172.21.15.169:0/3136137604 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ae6a470680 msgr2=0x55ae6a48c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:15.294 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.291+0000 7fa6c6ffd640 1 --2- 172.21.15.169:0/3136137604 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ae6a470680 0x55ae6a48c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:15.294 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.291+0000 7fa6c6ffd640 1 -- 172.21.15.169:0/3136137604 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ae6a48c9b0 con 0x55ae6a46ab20 2024-03-28T00:59:15.294 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.292+0000 7fa6c6ffd640 1 --2- 172.21.15.169:0/3136137604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46ab20 0x55ae6a48aec0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa6b80cb720 tx=0x7fa6b8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c76cca5051202bb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:15.296 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.292+0000 7fa6a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:59:15.296 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.293+0000 7fa6a6ffd640 1 -- 172.21.15.169:0/3136137604 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b80b8060 con 0x55ae6a46ab20 2024-03-28T00:59:15.296 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.293+0000 7fa6a6ffd640 1 -- 172.21.15.169:0/3136137604 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa6b8096c60 con 0x55ae6a46ab20 2024-03-28T00:59:15.296 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.293+0000 7fa6d05bf740 1 -- 172.21.15.169:0/3136137604 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa6b000be30 con 0x55ae6a46ab20 2024-03-28T00:59:15.302 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.299+0000 7fa6a6ffd640 1 -- 172.21.15.169:0/3136137604 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6b8080050 con 0x55ae6a46ab20 2024-03-28T00:59:15.303 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.299+0000 7fa6d05bf740 10 client.25895 fetch_fsmap learned FSMap version 25 2024-03-28T00:59:15.303 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.299+0000 7fa6d05bf740 1 -- 172.21.15.169:0/3136137604 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ae6a4eda30 con 0x55ae6a46ab20 2024-03-28T00:59:15.304 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.300+0000 7fa6a6ffd640 1 -- 172.21.15.169:0/3136137604 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6b000be30 con 0x55ae6a46ab20 2024-03-28T00:59:15.306 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.302+0000 7fa6a6ffd640 1 -- 172.21.15.169:0/3136137604 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa6b80a2900 con 0x55ae6a46ab20 2024-03-28T00:59:15.306 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.303+0000 7fa6d05bf740 10 client.25895 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:59:15.306 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.303+0000 7fa6d05bf740 10 client.25895 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:59:15.306 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.303+0000 7fa6d05bf740 20 client.25895 populate_metadata read hostname 'smithi169' 2024-03-28T00:59:15.306 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.303+0000 7fa6d05bf740 1 --2- 172.21.15.169:0/3136137604 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ae6a4ee460 0x55ae6a4f0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:15.306 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.303+0000 7fa6d05bf740 4 client.25895 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T00:59:15.306 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.303+0000 7fa6d05bf740 1 -- 172.21.15.169:0/3136137604 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ae6a3d73a0 con 0x55ae6a4ee460 2024-03-28T00:59:15.313 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.309+0000 7fa6c67fc640 1 --2- 172.21.15.169:0/3136137604 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ae6a4ee460 0x55ae6a4f0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:15.320 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.317+0000 7fa6a6ffd640 10 client.25895 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:15.320 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:15.317+0000 7fa6c67fc640 1 --2- 172.21.15.169:0/3136137604 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ae6a4ee460 0x55ae6a4f0880 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75c5d9b5c99ad1a9 server_cookie=f8b00d39f9ac93b in_seq=0 out_seq=0 2024-03-28T00:59:15.553 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:15 smithi169 ceph-mon[32274]: pgmap v433: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 819 MiB/s wr, 1.74k op/s 2024-03-28T00:59:15.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:15 smithi174 ceph-mon[36060]: pgmap v433: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 819 MiB/s wr, 1.74k op/s 2024-03-28T00:59:15.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:15 smithi177 ceph-mon[36206]: pgmap v433: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 819 MiB/s wr, 1.74k op/s 2024-03-28T00:59:16.542 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T00:59:16.543 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T00:59:16.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.937+0000 7f9d7c710640 1 -- 172.21.15.169:0/2335941408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d74078de0 msgr2=0x7f9d7407b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:16.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.937+0000 7f9d7c710640 1 --2- 172.21.15.169:0/2335941408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d74078de0 0x7f9d7407b270 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c009ca0 tx=0x7f9d6c02c140 comp rx=0 tx=0).stop 2024-03-28T00:59:16.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.940+0000 7f9d7c710640 1 -- 172.21.15.169:0/2335941408 shutdown_connections 2024-03-28T00:59:16.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.940+0000 7f9d7c710640 1 --2- 172.21.15.169:0/2335941408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d74078de0 0x7f9d7407b270 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:16.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.940+0000 7f9d7c710640 1 --2- 172.21.15.169:0/2335941408 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d74076430 0x7f9d740788a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:16.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.940+0000 7f9d7c710640 1 --2- 172.21.15.169:0/2335941408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d74073550 0x7f9d74073970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:16.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.940+0000 7f9d7c710640 1 -- 172.21.15.169:0/2335941408 >> 172.21.15.169:0/2335941408 conn(0x7f9d7406e500 msgr2=0x7f9d74070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:16.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.949+0000 7f9d7c710640 1 -- 172.21.15.169:0/2335941408 shutdown_connections 2024-03-28T00:59:16.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.951+0000 7f9d7c710640 1 -- 172.21.15.169:0/2335941408 wait complete. 2024-03-28T00:59:16.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.952+0000 7f9d7c710640 1 Processor -- start 2024-03-28T00:59:16.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.952+0000 7f9d7c710640 1 -- start start 2024-03-28T00:59:16.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.952+0000 7f9d7c710640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d74073550 0x7f9d741af5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:16.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.952+0000 7f9d7c710640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d74076430 0x7f9d741afae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:16.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.952+0000 7f9d7c710640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d74078de0 0x7f9d741b6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:16.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.952+0000 7f9d7c710640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d74080ef0 con 0x7f9d74078de0 2024-03-28T00:59:16.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.952+0000 7f9d7c710640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d74080d70 con 0x7f9d74076430 2024-03-28T00:59:16.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.952+0000 7f9d7c710640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d74081070 con 0x7f9d74073550 2024-03-28T00:59:16.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.953+0000 7f9d7a4ac640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d74073550 0x7f9d741af5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:16.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.953+0000 7f9d79cab640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d74076430 0x7f9d741afae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:16.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.953+0000 7f9d79cab640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d74076430 0x7f9d741afae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49024/0 (socket says 172.21.15.169:49024) 2024-03-28T00:59:16.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.953+0000 7f9d79cab640 1 -- 172.21.15.169:0/2572982363 learned_addr learned my addr 172.21.15.169:0/2572982363 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:16.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.953+0000 7f9d79cab640 1 -- 172.21.15.169:0/2572982363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d74073550 msgr2=0x7f9d741af5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:16.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.953+0000 7f9d79cab640 1 --2- 172.21.15.169:0/2572982363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d74073550 0x7f9d741af5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:16.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.953+0000 7f9d79cab640 1 -- 172.21.15.169:0/2572982363 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d74078de0 msgr2=0x7f9d741b6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:16.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.953+0000 7f9d79cab640 1 --2- 172.21.15.169:0/2572982363 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d74078de0 0x7f9d741b6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:16.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.953+0000 7f9d79cab640 1 -- 172.21.15.169:0/2572982363 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d741b6e10 con 0x7f9d74076430 2024-03-28T00:59:16.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.953+0000 7f9d79cab640 1 --2- 172.21.15.169:0/2572982363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d74076430 0x7f9d741afae0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9d6400b450 tx=0x7f9d6400b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:16.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.954+0000 7f9d6b7fe640 1 -- 172.21.15.169:0/2572982363 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d64015020 con 0x7f9d74076430 2024-03-28T00:59:16.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.954+0000 7f9d6b7fe640 1 -- 172.21.15.169:0/2572982363 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d64011d70 con 0x7f9d74076430 2024-03-28T00:59:16.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.954+0000 7f9d7c710640 1 -- 172.21.15.169:0/2572982363 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d741b7100 con 0x7f9d74076430 2024-03-28T00:59:16.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.954+0000 7f9d6b7fe640 1 -- 172.21.15.169:0/2572982363 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6400f340 con 0x7f9d74076430 2024-03-28T00:59:16.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.955+0000 7f9d7c710640 1 -- 172.21.15.169:0/2572982363 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d741b76d0 con 0x7f9d74076430 2024-03-28T00:59:16.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.964+0000 7f9d6b7fe640 1 -- 172.21.15.169:0/2572982363 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9d64018020 con 0x7f9d74076430 2024-03-28T00:59:16.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.966+0000 7f9d7f3de740 1 Processor -- start 2024-03-28T00:59:16.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.967+0000 7f9d7f3de740 1 -- start start 2024-03-28T00:59:16.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.967+0000 7f9d7f3de740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b85fcda0 0x5572b85fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:16.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.967+0000 7f9d7f3de740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b85fd790 0x5572b8602290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:16.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.967+0000 7f9d7f3de740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5572b8602900 0x5572b8604d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:16.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.967+0000 7f9d7f3de740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572b85623d0 con 0x5572b85fd790 2024-03-28T00:59:16.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.967+0000 7f9d7f3de740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572b850aad0 con 0x5572b85fcda0 2024-03-28T00:59:16.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.967+0000 7f9d7f3de740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572b855e470 con 0x5572b8602900 2024-03-28T00:59:16.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.968+0000 7f9d6b7fe640 1 --2- 172.21.15.169:0/2572982363 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9d600a2a20 0x7f9d600a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:16.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.968+0000 7f9d6b7fe640 1 -- 172.21.15.169:0/2572982363 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f9d640ce820 con 0x7f9d74076430 2024-03-28T00:59:16.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.969+0000 7f9d7acad640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b85fcda0 0x5572b85fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:16.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.969+0000 7f9d7acad640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b85fcda0 0x5572b85fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49034/0 (socket says 172.21.15.169:49034) 2024-03-28T00:59:16.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.969+0000 7f9d7acad640 1 -- 172.21.15.169:0/3756928251 learned_addr learned my addr 172.21.15.169:0/3756928251 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:16.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.969+0000 7f9d7acad640 1 --2- 172.21.15.169:0/2572982363 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9d600a2a20 0x7f9d600a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:16.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.969+0000 7f9d7a4ac640 1 --2- 172.21.15.169:0/3756928251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b85fd790 0x5572b8602290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:16.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.970+0000 7f9d7a4ac640 1 -- 172.21.15.169:0/3756928251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5572b8602900 msgr2=0x5572b8604d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:16.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.970+0000 7f9d7a4ac640 1 --2- 172.21.15.169:0/3756928251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5572b8602900 0x5572b8604d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:16.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.970+0000 7f9d7a4ac640 1 -- 172.21.15.169:0/3756928251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b85fcda0 msgr2=0x5572b85fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:16.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.970+0000 7f9d7a4ac640 1 --2- 172.21.15.169:0/3756928251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b85fcda0 0x5572b85fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:16.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.970+0000 7f9d7a4ac640 1 -- 172.21.15.169:0/3756928251 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572b83407d0 con 0x5572b85fd790 2024-03-28T00:59:16.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.970+0000 7f9d7acad640 1 --2- 172.21.15.169:0/3756928251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b85fcda0 0x5572b85fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:59:16.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.970+0000 7f9d7a4ac640 1 --2- 172.21.15.169:0/3756928251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b85fd790 0x5572b8602290 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9d7000a7d0 tx=0x7f9d7000ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5d42c3281f55563 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:16.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.970+0000 7f9d7acad640 1 --2- 172.21.15.169:0/2572982363 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9d600a2a20 0x7f9d600a4ee0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c032630 tx=0x7f9d6c005eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:16.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.972+0000 7f9d68ff9640 1 -- 172.21.15.169:0/3756928251 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d70015070 con 0x5572b85fd790 2024-03-28T00:59:16.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.972+0000 7f9d68ff9640 1 -- 172.21.15.169:0/3756928251 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d7000cb40 con 0x5572b85fd790 2024-03-28T00:59:16.976 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.973+0000 7f9d7f3de740 1 -- 172.21.15.169:0/3756928251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b85fd790 msgr2=0x5572b8602290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:16.976 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.973+0000 7f9d7f3de740 1 --2- 172.21.15.169:0/3756928251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b85fd790 0x5572b8602290 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9d7000a7d0 tx=0x7f9d7000ac90 comp rx=0 tx=0).stop 2024-03-28T00:59:16.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.974+0000 7f9d7f3de740 1 -- 172.21.15.169:0/3756928251 shutdown_connections 2024-03-28T00:59:16.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.974+0000 7f9d7f3de740 1 --2- 172.21.15.169:0/3756928251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5572b8602900 0x5572b8604d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:16.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.974+0000 7f9d7f3de740 1 --2- 172.21.15.169:0/3756928251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b85fd790 0x5572b8602290 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:16.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.974+0000 7f9d7f3de740 1 --2- 172.21.15.169:0/3756928251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b85fcda0 0x5572b85fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:16.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.974+0000 7f9d7f3de740 1 -- 172.21.15.169:0/3756928251 >> 172.21.15.169:0/3756928251 conn(0x5572b85f8310 msgr2=0x5572b85f8710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:16.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.975+0000 7f9d7f3de740 1 -- 172.21.15.169:0/3756928251 shutdown_connections 2024-03-28T00:59:16.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.975+0000 7f9d7f3de740 1 -- 172.21.15.169:0/3756928251 wait complete. 2024-03-28T00:59:16.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.975+0000 7f9d7f3de740 1 -- start start 2024-03-28T00:59:16.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.975+0000 7f9d7f3de740 1 Processor -- start 2024-03-28T00:59:16.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.976+0000 7f9d7f3de740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b85fcda0 0x5572b8607320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:16.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.976+0000 7f9d7f3de740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5572b85fd790 0x5572b8607860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:16.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.976+0000 7f9d7f3de740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b8602900 0x5572b860e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:16.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.976+0000 7f9d7f3de740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572b8605950 con 0x5572b85fcda0 2024-03-28T00:59:16.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.976+0000 7f9d7f3de740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572b851e060 con 0x5572b8602900 2024-03-28T00:59:16.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.976+0000 7f9d7f3de740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572b8605ad0 con 0x5572b85fd790 2024-03-28T00:59:16.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.976+0000 7f9d7f3de740 10 client.0 _ll_register_callbacks cb 0x5572b85006a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:59:16.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.978+0000 7f9d7a4ac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b85fcda0 0x5572b8607320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:16.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.978+0000 7f9d7acad640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b8602900 0x5572b860e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:16.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.978+0000 7f9d7a4ac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b85fcda0 0x5572b8607320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60966/0 (socket says 172.21.15.169:60966) 2024-03-28T00:59:16.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.978+0000 7f9d7a4ac640 1 -- 172.21.15.169:0/3302950919 learned_addr learned my addr 172.21.15.169:0/3302950919 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:16.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.978+0000 7f9d79cab640 1 --2- 172.21.15.169:0/3302950919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5572b85fd790 0x5572b8607860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:16.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.979+0000 7f9d7acad640 1 -- 172.21.15.169:0/3302950919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5572b85fd790 msgr2=0x5572b8607860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:16.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.979+0000 7f9d7acad640 1 --2- 172.21.15.169:0/3302950919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5572b85fd790 0x5572b8607860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:16.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.979+0000 7f9d7acad640 1 -- 172.21.15.169:0/3302950919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b85fcda0 msgr2=0x5572b8607320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:16.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.979+0000 7f9d7acad640 1 --2- 172.21.15.169:0/3302950919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b85fcda0 0x5572b8607320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:16.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.979+0000 7f9d7acad640 1 -- 172.21.15.169:0/3302950919 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5572b82763f0 con 0x5572b8602900 2024-03-28T00:59:16.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.979+0000 7f9d7acad640 1 --2- 172.21.15.169:0/3302950919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b8602900 0x5572b860e8a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c009ca0 tx=0x7f9d6c032cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9227fd4ab62eb87f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:16.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.980+0000 7f9d5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:59:16.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.980+0000 7f9d5dffb640 1 -- 172.21.15.169:0/3302950919 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c039040 con 0x5572b8602900 2024-03-28T00:59:16.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.980+0000 7f9d7f3de740 1 -- 172.21.15.169:0/3302950919 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d7000cb40 con 0x5572b8602900 2024-03-28T00:59:16.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.980+0000 7f9d5dffb640 1 -- 172.21.15.169:0/3302950919 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f9d6c004040 con 0x5572b8602900 2024-03-28T00:59:16.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.980+0000 7f9d5dffb640 1 -- 172.21.15.169:0/3302950919 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d6c059050 con 0x5572b8602900 2024-03-28T00:59:16.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.980+0000 7f9d7f3de740 10 client.25907 fetch_fsmap learned FSMap version 25 2024-03-28T00:59:16.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.980+0000 7f9d7f3de740 1 -- 172.21.15.169:0/3302950919 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5572b82360f0 con 0x5572b8602900 2024-03-28T00:59:16.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.982+0000 7f9d5dffb640 1 -- 172.21.15.169:0/3302950919 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d7000cb40 con 0x5572b8602900 2024-03-28T00:59:16.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.982+0000 7f9d5dffb640 1 -- 172.21.15.169:0/3302950919 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9d6c03f040 con 0x5572b8602900 2024-03-28T00:59:16.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.987+0000 7f9d7f3de740 10 client.25907 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:59:16.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.987+0000 7f9d7f3de740 10 client.25907 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:59:16.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.987+0000 7f9d7f3de740 20 client.25907 populate_metadata read hostname 'smithi169' 2024-03-28T00:59:16.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.987+0000 7f9d7f3de740 1 --2- 172.21.15.169:0/3302950919 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5572b8612de0 0x5572b8615220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:16.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.987+0000 7f9d7f3de740 4 client.25907 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:59:16.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.987+0000 7f9d7f3de740 1 -- 172.21.15.169:0/3302950919 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5572b856f910 con 0x5572b8612de0 2024-03-28T00:59:16.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.987+0000 7f9d7a4ac640 1 --2- 172.21.15.169:0/3302950919 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5572b8612de0 0x5572b8615220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:16.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.987+0000 7f9d7a4ac640 1 --2- 172.21.15.169:0/3302950919 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5572b8612de0 0x5572b8615220 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6d0ae9c81af470a server_cookie=1923129eb6ab1a66 in_seq=0 out_seq=0 2024-03-28T00:59:16.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:16.988+0000 7f9d5dffb640 10 client.25907 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:17.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:17 smithi169 ceph-mon[32274]: pgmap v434: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 847 MiB/s wr, 1.79k op/s 2024-03-28T00:59:17.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:17 smithi174 ceph-mon[36060]: pgmap v434: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 847 MiB/s wr, 1.79k op/s 2024-03-28T00:59:17.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:17 smithi177 ceph-mon[36206]: pgmap v434: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 847 MiB/s wr, 1.79k op/s 2024-03-28T00:59:19.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:19 smithi174 ceph-mon[36060]: pgmap v435: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 743 MiB/s wr, 1.64k op/s 2024-03-28T00:59:19.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:19 smithi177 ceph-mon[36206]: pgmap v435: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 743 MiB/s wr, 1.64k op/s 2024-03-28T00:59:19.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:19 smithi169 ceph-mon[32274]: pgmap v435: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 743 MiB/s wr, 1.64k op/s 2024-03-28T00:59:21.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:21 smithi174 ceph-mon[36060]: pgmap v436: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 733 MiB/s wr, 1.62k op/s 2024-03-28T00:59:21.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:21 smithi177 ceph-mon[36206]: pgmap v436: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 733 MiB/s wr, 1.62k op/s 2024-03-28T00:59:21.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:21 smithi169 ceph-mon[32274]: pgmap v436: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 733 MiB/s wr, 1.62k op/s 2024-03-28T00:59:23.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:23 smithi174 ceph-mon[36060]: pgmap v437: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 822 MiB/s wr, 1.68k op/s 2024-03-28T00:59:23.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:23 smithi177 ceph-mon[36206]: pgmap v437: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 822 MiB/s wr, 1.68k op/s 2024-03-28T00:59:23.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:23 smithi169 ceph-mon[32274]: pgmap v437: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 822 MiB/s wr, 1.68k op/s 2024-03-28T00:59:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.881+0000 7fa6a6ffd640 1 -- 172.21.15.169:0/3136137604 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ae6a3d73a0 con 0x55ae6a4ee460 2024-03-28T00:59:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.881+0000 7fa6a6ffd640 10 client.25895 handle_command_reply: tid=0 2024-03-28T00:59:23.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.881+0000 7f9d5dffb640 1 -- 172.21.15.169:0/3302950919 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5572b856f910 con 0x5572b8612de0 2024-03-28T00:59:23.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.881+0000 7f9d5dffb640 10 client.25907 handle_command_reply: tid=0 2024-03-28T00:59:23.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.881+0000 7f9d7f3de740 1 client.25907 shutdown 2024-03-28T00:59:23.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.881+0000 7f9d7f3de740 1 -- 172.21.15.169:0/3302950919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b8602900 msgr2=0x5572b860e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:23.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.881+0000 7f9d7f3de740 1 --2- 172.21.15.169:0/3302950919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b8602900 0x5572b860e8a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c009ca0 tx=0x7f9d6c032cd0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.881+0000 7fa6d05bf740 1 client.25895 shutdown 2024-03-28T00:59:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.882+0000 7fa6d05bf740 1 -- 172.21.15.169:0/3136137604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46ab20 msgr2=0x55ae6a48aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.882+0000 7fa6d05bf740 1 --2- 172.21.15.169:0/3136137604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46ab20 0x55ae6a48aec0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa6b80cb720 tx=0x7fa6b8004a40 comp rx=0 tx=0).stop 2024-03-28T00:59:23.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.882+0000 7f9d7f3de740 1 -- 172.21.15.169:0/3302950919 shutdown_connections 2024-03-28T00:59:23.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.882+0000 7f9d7f3de740 1 --2- 172.21.15.169:0/3302950919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b8602900 0x5572b860e8a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.882+0000 7f9d7f3de740 1 --2- 172.21.15.169:0/3302950919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5572b85fd790 0x5572b8607860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.882+0000 7f9d7f3de740 1 --2- 172.21.15.169:0/3302950919 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5572b8612de0 0x5572b8615220 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.882+0000 7f9d7f3de740 1 --2- 172.21.15.169:0/3302950919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b85fcda0 0x5572b8607320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.882+0000 7f9d7f3de740 1 -- 172.21.15.169:0/3302950919 >> 172.21.15.169:0/3302950919 conn(0x5572b85f8310 msgr2=0x5572b85f7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:23.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.882+0000 7f9d5dffb640 0 client.25907 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.882+0000 7fa6d05bf740 1 -- 172.21.15.169:0/3136137604 shutdown_connections 2024-03-28T00:59:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.882+0000 7fa6d05bf740 1 --2- 172.21.15.169:0/3136137604 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ae6a4ee460 0x55ae6a4f0880 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.882+0000 7fa6d05bf740 1 --2- 172.21.15.169:0/3136137604 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ae6a470680 0x55ae6a48c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.882+0000 7fa6d05bf740 1 --2- 172.21.15.169:0/3136137604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ae6a46b510 0x55ae6a48b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.882+0000 7fa6d05bf740 1 --2- 172.21.15.169:0/3136137604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae6a46ab20 0x55ae6a48aec0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.882+0000 7fa6d05bf740 1 -- 172.21.15.169:0/3136137604 >> 172.21.15.169:0/3136137604 conn(0x55ae6a466090 msgr2=0x55ae6a4692f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.882+0000 7fa6a6ffd640 0 client.25895 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:23.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.882+0000 7f9d7f3de740 1 -- 172.21.15.169:0/3302950919 shutdown_connections 2024-03-28T00:59:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.882+0000 7fa6d05bf740 1 -- 172.21.15.169:0/3136137604 shutdown_connections 2024-03-28T00:59:23.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.882+0000 7f9d7f3de740 1 -- 172.21.15.169:0/3302950919 wait complete. 2024-03-28T00:59:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.882+0000 7fa6d05bf740 1 -- 172.21.15.169:0/3136137604 wait complete. 2024-03-28T00:59:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.883+0000 7fa6d05bf740 20 client.25895 trim_cache size 0 max 16384 2024-03-28T00:59:23.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.883+0000 7f9d7f3de740 20 client.25907 trim_cache size 0 max 16384 2024-03-28T00:59:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T00:59:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T00:59:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T00:59:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T00:59:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.886+0000 7fa6c7fff640 1 -- 172.21.15.169:0/3706155071 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa6b40a2a00 msgr2=0x7fa6b40a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.886+0000 7fa6c7fff640 1 --2- 172.21.15.169:0/3706155071 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa6b40a2a00 0x7fa6b40a4ec0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0011560 tx=0x7fa6b0005850 comp rx=0 tx=0).stop 2024-03-28T00:59:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.886+0000 7fa6c7fff640 1 -- 172.21.15.169:0/3706155071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6c806d390 msgr2=0x7fa6c81a6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.886+0000 7fa6c7fff640 1 --2- 172.21.15.169:0/3706155071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6c806d390 0x7fa6c81a6650 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa6b800ba50 tx=0x7fa6b800bf10 comp rx=0 tx=0).stop 2024-03-28T00:59:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.887+0000 7fa6c7fff640 1 -- 172.21.15.169:0/3706155071 shutdown_connections 2024-03-28T00:59:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.887+0000 7fa6c7fff640 1 --2- 172.21.15.169:0/3706155071 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa6b40a2a00 0x7fa6b40a4ec0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.887+0000 7fa6c7fff640 1 --2- 172.21.15.169:0/3706155071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6c806ef70 0x7fa6c81adc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.887+0000 7fa6c7fff640 1 --2- 172.21.15.169:0/3706155071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6c806e5d0 0x7fa6c81a6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.887+0000 7fa6c7fff640 1 --2- 172.21.15.169:0/3706155071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6c806d390 0x7fa6c81a6650 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.887+0000 7fa6c7fff640 1 -- 172.21.15.169:0/3706155071 >> 172.21.15.169:0/3706155071 conn(0x7fa6c806bcf0 msgr2=0x7fa6c810a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.893+0000 7fa6c7fff640 1 -- 172.21.15.169:0/3706155071 shutdown_connections 2024-03-28T00:59:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:23.893+0000 7fa6c7fff640 1 -- 172.21.15.169:0/3706155071 wait complete. 2024-03-28T00:59:23.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.940+0000 7f9d7f3de740 1 Processor -- start 2024-03-28T00:59:23.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.940+0000 7f9d7f3de740 1 -- start start 2024-03-28T00:59:23.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.941+0000 7f9d7f3de740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b85fcda0 0x5572b8605f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:23.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.941+0000 7f9d7f3de740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5572b85fd790 0x5572b860e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:23.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.941+0000 7f9d7f3de740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b8602900 0x5572b85febf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:23.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.942+0000 7f9d7a4ac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b85fcda0 0x5572b8605f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:23.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.942+0000 7f9d7a4ac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b85fcda0 0x5572b8605f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60982/0 (socket says 172.21.15.169:60982) 2024-03-28T00:59:23.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.942+0000 7f9d79cab640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5572b85fd790 0x5572b860e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:23.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.942+0000 7f9d79cab640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5572b85fd790 0x5572b860e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:53736/0 (socket says 172.21.15.169:53736) 2024-03-28T00:59:23.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.942+0000 7f9d7acad640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b8602900 0x5572b85febf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:23.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.942+0000 7f9d7acad640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b8602900 0x5572b85febf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49052/0 (socket says 172.21.15.169:49052) 2024-03-28T00:59:23.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.942+0000 7f9d7a4ac640 1 -- 172.21.15.169:0/120557241 learned_addr learned my addr 172.21.15.169:0/120557241 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:23.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.942+0000 7f9d7f3de740 1 -- 172.21.15.169:0/120557241 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d580120f0 con 0x5572b85fcda0 2024-03-28T00:59:23.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.942+0000 7f9d7f3de740 1 -- 172.21.15.169:0/120557241 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5800fbf0 con 0x5572b8602900 2024-03-28T00:59:23.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.942+0000 7f9d7f3de740 1 -- 172.21.15.169:0/120557241 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5800dc90 con 0x5572b85fd790 2024-03-28T00:59:23.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.943+0000 7f9d7a4ac640 1 -- 172.21.15.169:0/120557241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5572b85fd790 msgr2=0x5572b860e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:23.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.943+0000 7f9d7a4ac640 1 --2- 172.21.15.169:0/120557241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5572b85fd790 0x5572b860e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.943+0000 7f9d7a4ac640 1 -- 172.21.15.169:0/120557241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b8602900 msgr2=0x5572b85febf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:23.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.943+0000 7f9d7a4ac640 1 --2- 172.21.15.169:0/120557241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b8602900 0x5572b85febf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.943+0000 7f9d7a4ac640 1 -- 172.21.15.169:0/120557241 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572b85fdb70 con 0x5572b85fcda0 2024-03-28T00:59:23.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.944+0000 7f9d7a4ac640 1 --2- 172.21.15.169:0/120557241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b85fcda0 0x5572b8605f50 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9d70008010 tx=0x7f9d7000a4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1206af7f21bd0bda server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:23.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.950+0000 7f9d5ffff640 1 -- 172.21.15.169:0/120557241 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7002d070 con 0x5572b85fcda0 2024-03-28T00:59:23.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.950+0000 7f9d5ffff640 1 -- 172.21.15.169:0/120557241 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d70004520 con 0x5572b85fcda0 2024-03-28T00:59:23.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.950+0000 7f9d5ffff640 1 -- 172.21.15.169:0/120557241 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d70015040 con 0x5572b85fcda0 2024-03-28T00:59:23.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.952+0000 7f9d7f3de740 1 -- 172.21.15.169:0/120557241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b85fcda0 msgr2=0x5572b8605f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:23.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.952+0000 7f9d7f3de740 1 --2- 172.21.15.169:0/120557241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b85fcda0 0x5572b8605f50 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9d70008010 tx=0x7f9d7000a4a0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.952+0000 7f9d7f3de740 1 -- 172.21.15.169:0/120557241 shutdown_connections 2024-03-28T00:59:23.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.952+0000 7f9d7f3de740 1 --2- 172.21.15.169:0/120557241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b8602900 0x5572b85febf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.952+0000 7f9d7f3de740 1 --2- 172.21.15.169:0/120557241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5572b85fd790 0x5572b860e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.952+0000 7f9d7f3de740 1 --2- 172.21.15.169:0/120557241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b85fcda0 0x5572b8605f50 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.952+0000 7f9d7f3de740 1 -- 172.21.15.169:0/120557241 >> 172.21.15.169:0/120557241 conn(0x5572b85f8310 msgr2=0x5572b8607320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:23.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.954+0000 7f9d7f3de740 1 -- 172.21.15.169:0/120557241 shutdown_connections 2024-03-28T00:59:23.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.954+0000 7f9d7f3de740 1 -- 172.21.15.169:0/120557241 wait complete. 2024-03-28T00:59:23.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.955+0000 7f9d7f3de740 1 -- start start 2024-03-28T00:59:23.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.955+0000 7f9d7f3de740 1 Processor -- start 2024-03-28T00:59:23.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.956+0000 7f9d7f3de740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5572b85fcda0 0x5572b860c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:23.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.956+0000 7f9d7f3de740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b85fd790 0x5572b860c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:23.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.956+0000 7f9d7f3de740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b8602900 0x5572b860d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:23.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.956+0000 7f9d7f3de740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5800e290 con 0x5572b8602900 2024-03-28T00:59:23.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.956+0000 7f9d7f3de740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5800f590 con 0x5572b85fd790 2024-03-28T00:59:23.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.956+0000 7f9d7f3de740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5800e8d0 con 0x5572b85fcda0 2024-03-28T00:59:23.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.956+0000 7f9d7f3de740 10 client.0 _ll_register_callbacks cb 0x7f9d6c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:59:23.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.957+0000 7f9d7acad640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b8602900 0x5572b860d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:23.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.957+0000 7f9d7acad640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b8602900 0x5572b860d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60998/0 (socket says 172.21.15.169:60998) 2024-03-28T00:59:23.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.957+0000 7f9d7acad640 1 -- 172.21.15.169:0/2519307450 learned_addr learned my addr 172.21.15.169:0/2519307450 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:23.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.957+0000 7f9d7acad640 1 -- 172.21.15.169:0/2519307450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5572b85fcda0 msgr2=0x5572b860c1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T00:59:23.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.957+0000 7f9d7acad640 1 --2- 172.21.15.169:0/2519307450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5572b85fcda0 0x5572b860c1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.957+0000 7f9d7acad640 1 -- 172.21.15.169:0/2519307450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b85fd790 msgr2=0x5572b860c730 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T00:59:23.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.957+0000 7f9d7acad640 1 --2- 172.21.15.169:0/2519307450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b85fd790 0x5572b860c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.957+0000 7f9d7acad640 1 -- 172.21.15.169:0/2519307450 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5572b860dd30 con 0x5572b8602900 2024-03-28T00:59:23.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.958+0000 7f9d7acad640 1 --2- 172.21.15.169:0/2519307450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b8602900 0x5572b860d740 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c004af0 tx=0x7f9d6c04b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=604b297a4bf51648 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:23.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.958+0000 7f9d5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:59:23.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.958+0000 7f9d5dffb640 1 -- 172.21.15.169:0/2519307450 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c063020 con 0x5572b8602900 2024-03-28T00:59:23.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.958+0000 7f9d5dffb640 1 -- 172.21.15.169:0/2519307450 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f9d6c034850 con 0x5572b8602900 2024-03-28T00:59:23.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.958+0000 7f9d7f3de740 1 -- 172.21.15.169:0/2519307450 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d70004520 con 0x5572b8602900 2024-03-28T00:59:23.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.959+0000 7f9d5dffb640 1 -- 172.21.15.169:0/2519307450 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d6c039040 con 0x5572b8602900 2024-03-28T00:59:23.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.960+0000 7f9d7f3de740 10 client.16074 fetch_fsmap learned FSMap version 25 2024-03-28T00:59:23.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.960+0000 7f9d7f3de740 1 -- 172.21.15.169:0/2519307450 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5572b866e9f0 con 0x5572b8602900 2024-03-28T00:59:23.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.961+0000 7f9d5dffb640 1 -- 172.21.15.169:0/2519307450 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d70004520 con 0x5572b8602900 2024-03-28T00:59:23.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.961+0000 7f9d5dffb640 1 -- 172.21.15.169:0/2519307450 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9d6c03f040 con 0x5572b8602900 2024-03-28T00:59:23.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.961+0000 7f9d7f3de740 10 client.16074 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:59:23.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.961+0000 7f9d7f3de740 10 client.16074 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:59:23.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.961+0000 7f9d7f3de740 20 client.16074 populate_metadata read hostname 'smithi169' 2024-03-28T00:59:23.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.961+0000 7f9d7f3de740 1 --2- 172.21.15.169:0/2519307450 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5572b866f340 0x5572b8671730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:23.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.961+0000 7f9d7f3de740 4 client.16074 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T00:59:23.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.961+0000 7f9d7f3de740 1 -- 172.21.15.169:0/2519307450 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5572b8569c60 con 0x5572b866f340 2024-03-28T00:59:23.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.965+0000 7f9d7a4ac640 1 --2- 172.21.15.169:0/2519307450 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5572b866f340 0x5572b8671730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:23.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.977+0000 7f9d7a4ac640 1 --2- 172.21.15.169:0/2519307450 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5572b866f340 0x5572b8671730 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d757ce92ae1a5cdf server_cookie=74acb20819074e3c in_seq=0 out_seq=0 2024-03-28T00:59:23.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.977+0000 7f9d5dffb640 10 client.16074 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:23.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.980+0000 7f9d5dffb640 1 -- 172.21.15.169:0/2519307450 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5572b8569c60 con 0x5572b866f340 2024-03-28T00:59:23.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.980+0000 7f9d5dffb640 10 client.16074 handle_command_reply: tid=0 2024-03-28T00:59:23.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.980+0000 7f9d7f3de740 1 client.16074 shutdown 2024-03-28T00:59:23.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.984+0000 7f9d7f3de740 1 -- 172.21.15.169:0/2519307450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b8602900 msgr2=0x5572b860d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:23.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.984+0000 7f9d7f3de740 1 --2- 172.21.15.169:0/2519307450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b8602900 0x5572b860d740 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c004af0 tx=0x7f9d6c04b700 comp rx=0 tx=0).stop 2024-03-28T00:59:23.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.984+0000 7f9d7f3de740 1 -- 172.21.15.169:0/2519307450 shutdown_connections 2024-03-28T00:59:23.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.984+0000 7f9d7f3de740 1 --2- 172.21.15.169:0/2519307450 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5572b866f340 0x5572b8671730 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.984+0000 7f9d7f3de740 1 --2- 172.21.15.169:0/2519307450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5572b8602900 0x5572b860d740 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.984+0000 7f9d7f3de740 1 --2- 172.21.15.169:0/2519307450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5572b85fd790 0x5572b860c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.984+0000 7f9d7f3de740 1 --2- 172.21.15.169:0/2519307450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5572b85fcda0 0x5572b860c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:23.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.984+0000 7f9d7f3de740 1 -- 172.21.15.169:0/2519307450 >> 172.21.15.169:0/2519307450 conn(0x5572b85f8310 msgr2=0x5572b8607320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:23.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.986+0000 7f9d5dffb640 0 client.16074 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:23.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.988+0000 7f9d7f3de740 1 -- 172.21.15.169:0/2519307450 shutdown_connections 2024-03-28T00:59:23.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.990+0000 7f9d7f3de740 1 -- 172.21.15.169:0/2519307450 wait complete. 2024-03-28T00:59:23.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.991+0000 7f9d7f3de740 20 client.16074 trim_cache size 0 max 16384 2024-03-28T00:59:23.996 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:59:23.996 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "no active scrubs running", 2024-03-28T00:59:23.996 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": {} 2024-03-28T00:59:23.997 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:59:24.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.997+0000 7f9d7c710640 1 -- 172.21.15.169:0/2572982363 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9d600a2a20 msgr2=0x7f9d600a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:24.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.997+0000 7f9d7c710640 1 --2- 172.21.15.169:0/2572982363 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9d600a2a20 0x7f9d600a4ee0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c032630 tx=0x7f9d6c005eb0 comp rx=0 tx=0).stop 2024-03-28T00:59:24.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.998+0000 7f9d7c710640 1 -- 172.21.15.169:0/2572982363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d74076430 msgr2=0x7f9d741afae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:24.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.998+0000 7f9d7c710640 1 --2- 172.21.15.169:0/2572982363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d74076430 0x7f9d741afae0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9d6400b450 tx=0x7f9d6400b910 comp rx=0 tx=0).stop 2024-03-28T00:59:24.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.998+0000 7f9d7c710640 1 -- 172.21.15.169:0/2572982363 shutdown_connections 2024-03-28T00:59:24.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.998+0000 7f9d7c710640 1 --2- 172.21.15.169:0/2572982363 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9d600a2a20 0x7f9d600a4ee0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:24.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.998+0000 7f9d7c710640 1 --2- 172.21.15.169:0/2572982363 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d74078de0 0x7f9d741b6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:24.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.998+0000 7f9d7c710640 1 --2- 172.21.15.169:0/2572982363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d74076430 0x7f9d741afae0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:24.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.998+0000 7f9d7c710640 1 --2- 172.21.15.169:0/2572982363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d74073550 0x7f9d741af5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:24.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.999+0000 7f9d7c710640 1 -- 172.21.15.169:0/2572982363 >> 172.21.15.169:0/2572982363 conn(0x7f9d7406e500 msgr2=0x7f9d74076890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:24.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:23.999+0000 7f9d7c710640 1 -- 172.21.15.169:0/2572982363 shutdown_connections 2024-03-28T00:59:24.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.000+0000 7f9d7c710640 1 -- 172.21.15.169:0/2572982363 wait complete. 2024-03-28T00:59:24.040 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T00:59:24.041 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T00:59:24.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.233+0000 7f4ba5f52640 1 -- 172.21.15.169:0/4137579949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ba0100100 msgr2=0x7f4ba01005b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:24.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.233+0000 7f4ba5f52640 1 --2- 172.21.15.169:0/4137579949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ba0100100 0x7f4ba01005b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f4b98009ca0 tx=0x7f4b9802c0f0 comp rx=0 tx=0).stop 2024-03-28T00:59:24.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.233+0000 7f4ba5f52640 1 -- 172.21.15.169:0/4137579949 shutdown_connections 2024-03-28T00:59:24.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.233+0000 7f4ba5f52640 1 --2- 172.21.15.169:0/4137579949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ba0100100 0x7f4ba01005b0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:24.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.233+0000 7f4ba5f52640 1 --2- 172.21.15.169:0/4137579949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ba00ff740 0x7f4ba00ffbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:24.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.233+0000 7f4ba5f52640 1 --2- 172.21.15.169:0/4137579949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ba0101c50 0x7f4ba0102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:24.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.233+0000 7f4ba5f52640 1 -- 172.21.15.169:0/4137579949 >> 172.21.15.169:0/4137579949 conn(0x7f4ba00fb2c0 msgr2=0x7f4ba00fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:24.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.233+0000 7f4ba5f52640 1 -- 172.21.15.169:0/4137579949 shutdown_connections 2024-03-28T00:59:24.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.233+0000 7f4ba5f52640 1 -- 172.21.15.169:0/4137579949 wait complete. 2024-03-28T00:59:24.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.234+0000 7f4ba5f52640 1 Processor -- start 2024-03-28T00:59:24.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.234+0000 7f4ba5f52640 1 -- start start 2024-03-28T00:59:24.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.235+0000 7f4ba5f52640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ba00ff740 0x7f4ba0199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:24.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.235+0000 7f4ba5f52640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ba0100100 0x7f4ba019a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:24.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.235+0000 7f4ba5f52640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ba0101c50 0x7f4ba01a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:24.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.235+0000 7f4ba5f52640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba010f650 con 0x7f4ba0100100 2024-03-28T00:59:24.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.235+0000 7f4ba5f52640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba010f4d0 con 0x7f4ba00ff740 2024-03-28T00:59:24.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.235+0000 7f4ba5f52640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba010f7d0 con 0x7f4ba0101c50 2024-03-28T00:59:24.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.235+0000 7f4b9effd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ba0100100 0x7f4ba019a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:24.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.235+0000 7f4b9effd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ba0100100 0x7f4ba019a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:32774/0 (socket says 172.21.15.169:32774) 2024-03-28T00:59:24.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.235+0000 7f4b9effd640 1 -- 172.21.15.169:0/2443351750 learned_addr learned my addr 172.21.15.169:0/2443351750 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:24.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.235+0000 7f4b9ffff640 1 --2- 172.21.15.169:0/2443351750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ba0101c50 0x7f4ba01a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:24.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.235+0000 7f4b9f7fe640 1 --2- 172.21.15.169:0/2443351750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ba00ff740 0x7f4ba0199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:24.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.236+0000 7f4b9effd640 1 -- 172.21.15.169:0/2443351750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ba0101c50 msgr2=0x7f4ba01a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:24.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.236+0000 7f4b9effd640 1 --2- 172.21.15.169:0/2443351750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ba0101c50 0x7f4ba01a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:24.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.236+0000 7f4b9effd640 1 -- 172.21.15.169:0/2443351750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ba00ff740 msgr2=0x7f4ba0199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:24.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.236+0000 7f4b9effd640 1 --2- 172.21.15.169:0/2443351750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ba00ff740 0x7f4ba0199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:24.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.236+0000 7f4b9effd640 1 -- 172.21.15.169:0/2443351750 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba01a18b0 con 0x7f4ba0100100 2024-03-28T00:59:24.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.236+0000 7f4b9effd640 1 --2- 172.21.15.169:0/2443351750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ba0100100 0x7f4ba019a200 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f4b9400b450 tx=0x7f4b9400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:24.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.236+0000 7f4b9cff9640 1 -- 172.21.15.169:0/2443351750 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b94015020 con 0x7f4ba0100100 2024-03-28T00:59:24.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.236+0000 7f4b9cff9640 1 -- 172.21.15.169:0/2443351750 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b94011d70 con 0x7f4ba0100100 2024-03-28T00:59:24.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.236+0000 7f4ba5f52640 1 -- 172.21.15.169:0/2443351750 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba01a1ba0 con 0x7f4ba0100100 2024-03-28T00:59:24.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.236+0000 7f4b9cff9640 1 -- 172.21.15.169:0/2443351750 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9400f340 con 0x7f4ba0100100 2024-03-28T00:59:24.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.236+0000 7f4ba5f52640 1 -- 172.21.15.169:0/2443351750 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba01a2140 con 0x7f4ba0100100 2024-03-28T00:59:24.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.238+0000 7f4ba5f52640 1 -- 172.21.15.169:0/2443351750 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba010b560 con 0x7f4ba0100100 2024-03-28T00:59:24.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.240+0000 7f4b9cff9640 1 -- 172.21.15.169:0/2443351750 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4b94018020 con 0x7f4ba0100100 2024-03-28T00:59:24.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.241+0000 7f4b9cff9640 1 --2- 172.21.15.169:0/2443351750 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4b880a2900 0x7f4b880a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:24.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.241+0000 7f4b9f7fe640 1 --2- 172.21.15.169:0/2443351750 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4b880a2900 0x7f4b880a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:24.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.242+0000 7f4b9cff9640 1 -- 172.21.15.169:0/2443351750 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f4b940cede0 con 0x7f4ba0100100 2024-03-28T00:59:24.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.242+0000 7f4b9f7fe640 1 --2- 172.21.15.169:0/2443351750 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4b880a2900 0x7f4b880a4dc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4ba00fe8b0 tx=0x7f4b90008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:24.249 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.246+0000 7f4b9cff9640 1 -- 172.21.15.169:0/2443351750 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4b9401f020 con 0x7f4ba0100100 2024-03-28T00:59:24.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.721+0000 7f4ba5f52640 1 -- 172.21.15.169:0/2443351750 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ba002d2c0 con 0x7f4ba0100100 2024-03-28T00:59:24.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.723+0000 7f4b9cff9640 1 -- 172.21.15.169:0/2443351750 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7f4b9400d040 con 0x7f4ba0100100 2024-03-28T00:59:24.726 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T00:59:24.726 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T00:59:24.726 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T00:59:24.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.727+0000 7f4ba5f52640 1 -- 172.21.15.169:0/2443351750 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4b880a2900 msgr2=0x7f4b880a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:24.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.727+0000 7f4ba5f52640 1 --2- 172.21.15.169:0/2443351750 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4b880a2900 0x7f4b880a4dc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4ba00fe8b0 tx=0x7f4b90008040 comp rx=0 tx=0).stop 2024-03-28T00:59:24.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.727+0000 7f4ba5f52640 1 -- 172.21.15.169:0/2443351750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ba0100100 msgr2=0x7f4ba019a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:24.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.727+0000 7f4ba5f52640 1 --2- 172.21.15.169:0/2443351750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ba0100100 0x7f4ba019a200 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f4b9400b450 tx=0x7f4b9400b910 comp rx=0 tx=0).stop 2024-03-28T00:59:24.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.729+0000 7f4ba5f52640 1 -- 172.21.15.169:0/2443351750 shutdown_connections 2024-03-28T00:59:24.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.729+0000 7f4ba5f52640 1 --2- 172.21.15.169:0/2443351750 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4b880a2900 0x7f4b880a4dc0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:24.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.729+0000 7f4ba5f52640 1 --2- 172.21.15.169:0/2443351750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ba0101c50 0x7f4ba01a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:24.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.729+0000 7f4ba5f52640 1 --2- 172.21.15.169:0/2443351750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ba0100100 0x7f4ba019a200 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:24.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.729+0000 7f4ba5f52640 1 --2- 172.21.15.169:0/2443351750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ba00ff740 0x7f4ba0199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:24.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.729+0000 7f4ba5f52640 1 -- 172.21.15.169:0/2443351750 >> 172.21.15.169:0/2443351750 conn(0x7f4ba00fb2c0 msgr2=0x7f4ba00fcdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:24.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.730+0000 7f4ba5f52640 1 -- 172.21.15.169:0/2443351750 shutdown_connections 2024-03-28T00:59:24.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:24.730+0000 7f4ba5f52640 1 -- 172.21.15.169:0/2443351750 wait complete. 2024-03-28T00:59:24.760 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T00:59:24.760 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T00:59:25.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.177+0000 7fbd1f044640 1 -- 172.21.15.169:0/1860390399 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd18072a10 msgr2=0x7fbd18072e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:25.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.177+0000 7fbd1f044640 1 --2- 172.21.15.169:0/1860390399 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd18072a10 0x7fbd18072e30 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fbd10009860 tx=0x7fbd1002bb70 comp rx=0 tx=0).stop 2024-03-28T00:59:25.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.177+0000 7fbd1f044640 1 -- 172.21.15.169:0/1860390399 shutdown_connections 2024-03-28T00:59:25.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.177+0000 7fbd1f044640 1 --2- 172.21.15.169:0/1860390399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd181096a0 0x7fbd1810df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.177+0000 7fbd1f044640 1 --2- 172.21.15.169:0/1860390399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd18108ce0 0x7fbd18109160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.177+0000 7fbd1f044640 1 --2- 172.21.15.169:0/1860390399 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd18072a10 0x7fbd18072e30 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.177+0000 7fbd1f044640 1 -- 172.21.15.169:0/1860390399 >> 172.21.15.169:0/1860390399 conn(0x7fbd1806e4c0 msgr2=0x7fbd18070900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:25.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.177+0000 7fbd1f044640 1 -- 172.21.15.169:0/1860390399 shutdown_connections 2024-03-28T00:59:25.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.177+0000 7fbd1f044640 1 -- 172.21.15.169:0/1860390399 wait complete. 2024-03-28T00:59:25.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.179+0000 7fbd1f044640 1 Processor -- start 2024-03-28T00:59:25.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.179+0000 7fbd1f044640 1 -- start start 2024-03-28T00:59:25.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.179+0000 7fbd1f044640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd18072a10 0x7fbd18114d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.179+0000 7fbd1f044640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd18108ce0 0x7fbd18115250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.179+0000 7fbd1f044640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd181096a0 0x7fbd181a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.179+0000 7fbd1f044640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1811bff0 con 0x7fbd181096a0 2024-03-28T00:59:25.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.179+0000 7fbd1f044640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1811be70 con 0x7fbd18072a10 2024-03-28T00:59:25.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.179+0000 7fbd1f044640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1811c170 con 0x7fbd18108ce0 2024-03-28T00:59:25.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.179+0000 7fbd1cde0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd18072a10 0x7fbd18114d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.179+0000 7fbd1cde0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd18072a10 0x7fbd18114d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49106/0 (socket says 172.21.15.169:49106) 2024-03-28T00:59:25.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.179+0000 7fbd1cde0640 1 -- 172.21.15.169:0/1241514966 learned_addr learned my addr 172.21.15.169:0/1241514966 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:25.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.180+0000 7fbd1cde0640 1 -- 172.21.15.169:0/1241514966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd18108ce0 msgr2=0x7fbd18115250 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T00:59:25.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.180+0000 7fbd1cde0640 1 --2- 172.21.15.169:0/1241514966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd18108ce0 0x7fbd18115250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.180+0000 7fbd1cde0640 1 -- 172.21.15.169:0/1241514966 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd181096a0 msgr2=0x7fbd181a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:25.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.180+0000 7fbd1cde0640 1 --2- 172.21.15.169:0/1241514966 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd181096a0 0x7fbd181a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.180+0000 7fbd1cde0640 1 -- 172.21.15.169:0/1241514966 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd181a5c10 con 0x7fbd18072a10 2024-03-28T00:59:25.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.181+0000 7fbd1cde0640 1 --2- 172.21.15.169:0/1241514966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd18072a10 0x7fbd18114d10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbd10000c00 tx=0x7fbd10032460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:25.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.181+0000 7fbd15ffb640 1 -- 172.21.15.169:0/1241514966 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd10041020 con 0x7fbd18072a10 2024-03-28T00:59:25.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.181+0000 7fbd15ffb640 1 -- 172.21.15.169:0/1241514966 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd1002c9f0 con 0x7fbd18072a10 2024-03-28T00:59:25.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.181+0000 7fbd15ffb640 1 -- 172.21.15.169:0/1241514966 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd10039040 con 0x7fbd18072a10 2024-03-28T00:59:25.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.182+0000 7fbd1f044640 1 -- 172.21.15.169:0/1241514966 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd181a5ea0 con 0x7fbd18072a10 2024-03-28T00:59:25.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.183+0000 7fbd1f044640 1 -- 172.21.15.169:0/1241514966 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd181a6340 con 0x7fbd18072a10 2024-03-28T00:59:25.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.190+0000 7fbd21c2f740 1 Processor -- start 2024-03-28T00:59:25.196 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.193+0000 7fbd21c2f740 1 -- start start 2024-03-28T00:59:25.196 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.193+0000 7fbd21c2f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfd860 0x561c47cfdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.196 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.193+0000 7fbd21c2f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cfe250 0x561c47d02d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.196 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.193+0000 7fbd21c2f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c47d033c0 0x561c47d05850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.196 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.193+0000 7fbd1d5e1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfd860 0x561c47cfdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.196 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.193+0000 7fbd21c2f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c47cc0610 con 0x561c47d033c0 2024-03-28T00:59:25.196 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.193+0000 7fbd21c2f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c47c633c0 con 0x561c47cfd860 2024-03-28T00:59:25.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.193+0000 7fbd15ffb640 1 -- 172.21.15.169:0/1241514966 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbd10044020 con 0x7fbd18072a10 2024-03-28T00:59:25.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.193+0000 7fbd1d5e1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfd860 0x561c47cfdc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49112/0 (socket says 172.21.15.169:49112) 2024-03-28T00:59:25.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.193+0000 7fbd1d5e1640 1 -- 172.21.15.169:0/1918707211 learned_addr learned my addr 172.21.15.169:0/1918707211 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:25.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.194+0000 7fbd15ffb640 1 --2- 172.21.15.169:0/1241514966 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbd040a2a00 0x7fbd040a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.194+0000 7fbd1cde0640 1 --2- 172.21.15.169:0/1918707211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cfe250 0x561c47d02d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.194+0000 7fbd21c2f740 1 -- 172.21.15.169:0/1918707211 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c47c0bd60 con 0x561c47cfe250 2024-03-28T00:59:25.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.195+0000 7fbd1cde0640 1 -- 172.21.15.169:0/1918707211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfd860 msgr2=0x561c47cfdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:25.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.195+0000 7fbd1cde0640 1 --2- 172.21.15.169:0/1918707211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfd860 0x561c47cfdc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.195+0000 7fbd1cde0640 1 -- 172.21.15.169:0/1918707211 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c47d033c0 msgr2=0x561c47d05850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T00:59:25.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.195+0000 7fbd1cde0640 1 --2- 172.21.15.169:0/1918707211 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c47d033c0 0x561c47d05850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.195+0000 7fbd1cde0640 1 -- 172.21.15.169:0/1918707211 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c479773f0 con 0x561c47cfe250 2024-03-28T00:59:25.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.195+0000 7fbd1cde0640 1 --2- 172.21.15.169:0/1918707211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cfe250 0x561c47d02d50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbd100edf90 tx=0x7fbd100f6b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb4861222ca48f5d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:25.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.196+0000 7fbd15ffb640 1 -- 172.21.15.169:0/1241514966 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fbd100f08f0 con 0x7fbd18072a10 2024-03-28T00:59:25.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.196+0000 7fbd0effd640 1 -- 172.21.15.169:0/1918707211 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd100b90a0 con 0x561c47cfe250 2024-03-28T00:59:25.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.196+0000 7fbd0effd640 1 -- 172.21.15.169:0/1918707211 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd100bd590 con 0x561c47cfe250 2024-03-28T00:59:25.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.196+0000 7fbd1d5e1640 1 --2- 172.21.15.169:0/1241514966 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbd040a2a00 0x7fbd040a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.197+0000 7fbd21c2f740 1 -- 172.21.15.169:0/1918707211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cfe250 msgr2=0x561c47d02d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:25.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.197+0000 7fbd21c2f740 1 --2- 172.21.15.169:0/1918707211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cfe250 0x561c47d02d50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbd100edf90 tx=0x7fbd100f6b20 comp rx=0 tx=0).stop 2024-03-28T00:59:25.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.200+0000 7fbd1d5e1640 1 --2- 172.21.15.169:0/1241514966 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbd040a2a00 0x7fbd040a4ec0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x561c47cfeab0 tx=0x7fbd000046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:25.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.200+0000 7fbd21c2f740 1 -- 172.21.15.169:0/1918707211 shutdown_connections 2024-03-28T00:59:25.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.200+0000 7fbd21c2f740 1 --2- 172.21.15.169:0/1918707211 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c47d033c0 0x561c47d05850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.200+0000 7fbd21c2f740 1 --2- 172.21.15.169:0/1918707211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cfe250 0x561c47d02d50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.200+0000 7fbd21c2f740 1 --2- 172.21.15.169:0/1918707211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfd860 0x561c47cfdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.200+0000 7fbd21c2f740 1 -- 172.21.15.169:0/1918707211 >> 172.21.15.169:0/1918707211 conn(0x561c47cf8dd0 msgr2=0x561c47cf91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:25.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.202+0000 7fbd21c2f740 1 -- 172.21.15.169:0/1918707211 shutdown_connections 2024-03-28T00:59:25.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.203+0000 7fbd21c2f740 1 -- 172.21.15.169:0/1918707211 wait complete. 2024-03-28T00:59:25.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.203+0000 7fbd21c2f740 1 -- start start 2024-03-28T00:59:25.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.204+0000 7fbd21c2f740 1 Processor -- start 2024-03-28T00:59:25.207 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.204+0000 7fbd21c2f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cfd860 0x561c47d07910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.207 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.204+0000 7fbd21c2f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfe250 0x561c47d07e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.207 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.204+0000 7fbd21c2f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c47d033c0 0x561c47d0ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.204+0000 7fbd21c2f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c47c1f240 con 0x561c47d033c0 2024-03-28T00:59:25.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.204+0000 7fbd21c2f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c47c5f460 con 0x561c47cfe250 2024-03-28T00:59:25.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.204+0000 7fbd21c2f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c47d06130 con 0x561c47cfd860 2024-03-28T00:59:25.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.204+0000 7fbd21c2f740 10 client.0 _ll_register_callbacks cb 0x561c47c1ec10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:59:25.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.206+0000 7fbd1d5e1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c47d033c0 0x561c47d0ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.206+0000 7fbd1d5e1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c47d033c0 0x561c47d0ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:32792/0 (socket says 172.21.15.169:32792) 2024-03-28T00:59:25.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.206+0000 7fbd1d5e1640 1 -- 172.21.15.169:0/2311446917 learned_addr learned my addr 172.21.15.169:0/2311446917 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:25.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.206+0000 7fbd1cde0640 1 --2- 172.21.15.169:0/2311446917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cfd860 0x561c47d07910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.206+0000 7fbd17fff640 1 --2- 172.21.15.169:0/2311446917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfe250 0x561c47d07e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.206+0000 7fbd1cde0640 1 -- 172.21.15.169:0/2311446917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfe250 msgr2=0x561c47d07e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:25.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.206+0000 7fbd1cde0640 1 --2- 172.21.15.169:0/2311446917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfe250 0x561c47d07e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.206+0000 7fbd1cde0640 1 -- 172.21.15.169:0/2311446917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c47d033c0 msgr2=0x561c47d0ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:25.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.206+0000 7fbd1cde0640 1 --2- 172.21.15.169:0/2311446917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c47d033c0 0x561c47d0ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.206+0000 7fbd1cde0640 1 -- 172.21.15.169:0/2311446917 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c47a417d0 con 0x561c47cfd860 2024-03-28T00:59:25.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.207+0000 7fbd1cde0640 1 --2- 172.21.15.169:0/2311446917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cfd860 0x561c47d07910 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x561c47cfecc0 tx=0x7fbd100f64b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=513d4162d42be98c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:25.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.208+0000 7fbcf7fff640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:59:25.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.208+0000 7fbcf7fff640 1 -- 172.21.15.169:0/2311446917 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd100cbfd0 con 0x561c47cfd860 2024-03-28T00:59:25.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.208+0000 7fbd21c2f740 1 -- 172.21.15.169:0/2311446917 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbd100bd590 con 0x561c47cfd860 2024-03-28T00:59:25.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.208+0000 7fbcf7fff640 1 -- 172.21.15.169:0/2311446917 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fbd100bb190 con 0x561c47cfd860 2024-03-28T00:59:25.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.209+0000 7fbcf7fff640 1 -- 172.21.15.169:0/2311446917 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd100cd070 con 0x561c47cfd860 2024-03-28T00:59:25.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.209+0000 7fbd21c2f740 10 client.25891 fetch_fsmap learned FSMap version 25 2024-03-28T00:59:25.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.209+0000 7fbd21c2f740 1 -- 172.21.15.169:0/2311446917 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561c47d12200 con 0x561c47cfd860 2024-03-28T00:59:25.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.209+0000 7fbcf7fff640 1 -- 172.21.15.169:0/2311446917 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd100bd590 con 0x561c47cfd860 2024-03-28T00:59:25.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.210+0000 7fbcf7fff640 1 -- 172.21.15.169:0/2311446917 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fbd1003c030 con 0x561c47cfd860 2024-03-28T00:59:25.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.210+0000 7fbd21c2f740 10 client.25891 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:59:25.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.210+0000 7fbd21c2f740 10 client.25891 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T00:59:25.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.210+0000 7fbd21c2f740 20 client.25891 populate_metadata read hostname 'smithi169' 2024-03-28T00:59:25.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.210+0000 7fbd21c2f740 1 --2- 172.21.15.169:0/2311446917 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x561c47d13520 0x561c47d15950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.210+0000 7fbd21c2f740 4 client.25891 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:59:25.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.210+0000 7fbd21c2f740 1 -- 172.21.15.169:0/2311446917 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561c47c70900 con 0x561c47d13520 2024-03-28T00:59:25.214 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.211+0000 7fbd17fff640 1 --2- 172.21.15.169:0/2311446917 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x561c47d13520 0x561c47d15950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.213+0000 7fbd17fff640 1 --2- 172.21.15.169:0/2311446917 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x561c47d13520 0x561c47d15950 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4d94523652f2fd6e server_cookie=e03b98545030bc6c in_seq=0 out_seq=0 2024-03-28T00:59:25.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.213+0000 7fbcf7fff640 10 client.25891 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T00:59:25.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.214+0000 7fbcf7fff640 1 -- 172.21.15.169:0/2311446917 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561c47c70900 con 0x561c47d13520 2024-03-28T00:59:25.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.215+0000 7fbcf7fff640 10 client.25891 handle_command_reply: tid=0 2024-03-28T00:59:25.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.215+0000 7fbd21c2f740 1 client.25891 shutdown 2024-03-28T00:59:25.223 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.220+0000 7fbd21c2f740 1 -- 172.21.15.169:0/2311446917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cfd860 msgr2=0x561c47d07910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:25.223 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.220+0000 7fbd21c2f740 1 --2- 172.21.15.169:0/2311446917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cfd860 0x561c47d07910 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x561c47cfecc0 tx=0x7fbd100f64b0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.226+0000 7fbd21c2f740 1 -- 172.21.15.169:0/2311446917 shutdown_connections 2024-03-28T00:59:25.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.226+0000 7fbd21c2f740 1 --2- 172.21.15.169:0/2311446917 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x561c47d13520 0x561c47d15950 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.226+0000 7fbd21c2f740 1 --2- 172.21.15.169:0/2311446917 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c47d033c0 0x561c47d0ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.226+0000 7fbd21c2f740 1 --2- 172.21.15.169:0/2311446917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfe250 0x561c47d07e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.226+0000 7fbd21c2f740 1 --2- 172.21.15.169:0/2311446917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cfd860 0x561c47d07910 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.226+0000 7fbd21c2f740 1 -- 172.21.15.169:0/2311446917 >> 172.21.15.169:0/2311446917 conn(0x561c47cf8dd0 msgr2=0x561c47cf8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:25.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.227+0000 7fbcf7fff640 0 client.25891 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T00:59:25.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.229+0000 7fbd21c2f740 1 -- 172.21.15.169:0/2311446917 shutdown_connections 2024-03-28T00:59:25.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.232+0000 7fbd21c2f740 1 -- 172.21.15.169:0/2311446917 wait complete. 2024-03-28T00:59:25.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.234+0000 7fbd21c2f740 20 client.25891 trim_cache size 0 max 16384 2024-03-28T00:59:25.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.264+0000 7fbd21c2f740 1 Processor -- start 2024-03-28T00:59:25.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.264+0000 7fbd21c2f740 1 -- start start 2024-03-28T00:59:25.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.264+0000 7fbd21c2f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c47cf8dd0 0x561c47cf3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.264+0000 7fbd21c2f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cfd860 0x561c47d0ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.264+0000 7fbd21c2f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfe250 0x561c47d07c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.264+0000 7fbd21c2f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd181d4ce0 con 0x561c47cf8dd0 2024-03-28T00:59:25.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.264+0000 7fbd21c2f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd181b0850 con 0x561c47cfe250 2024-03-28T00:59:25.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.264+0000 7fbd21c2f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd181ae8f0 con 0x561c47cfd860 2024-03-28T00:59:25.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.268+0000 7fbd17fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cfd860 0x561c47d0ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.268+0000 7fbd17fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cfd860 0x561c47d0ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:53814/0 (socket says 172.21.15.169:53814) 2024-03-28T00:59:25.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.268+0000 7fbd17fff640 1 -- 172.21.15.169:0/407345994 learned_addr learned my addr 172.21.15.169:0/407345994 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:25.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.269+0000 7fbd1d5e1640 1 --2- 172.21.15.169:0/407345994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfe250 0x561c47d07c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.271+0000 7fbd17fff640 1 -- 172.21.15.169:0/407345994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfe250 msgr2=0x561c47d07c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:25.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.271+0000 7fbd17fff640 1 --2- 172.21.15.169:0/407345994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfe250 0x561c47d07c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.271+0000 7fbd17fff640 1 -- 172.21.15.169:0/407345994 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c47cf8dd0 msgr2=0x561c47cf3f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T00:59:25.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.271+0000 7fbd17fff640 1 --2- 172.21.15.169:0/407345994 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c47cf8dd0 0x561c47cf3f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.271+0000 7fbd17fff640 1 -- 172.21.15.169:0/407345994 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c47d029a0 con 0x561c47cfd860 2024-03-28T00:59:25.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.271+0000 7fbd17fff640 1 --2- 172.21.15.169:0/407345994 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cfd860 0x561c47d0ee20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbd08009fd0 tx=0x7fbd08009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5633f2ab80334718 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:25.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.271+0000 7fbd0e7fc640 1 -- 172.21.15.169:0/407345994 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd080120d0 con 0x561c47cfd860 2024-03-28T00:59:25.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.272+0000 7fbd0e7fc640 1 -- 172.21.15.169:0/407345994 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd08012270 con 0x561c47cfd860 2024-03-28T00:59:25.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.273+0000 7fbd21c2f740 1 -- 172.21.15.169:0/407345994 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cfd860 msgr2=0x561c47d0ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:25.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.273+0000 7fbd21c2f740 1 --2- 172.21.15.169:0/407345994 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cfd860 0x561c47d0ee20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbd08009fd0 tx=0x7fbd08009500 comp rx=0 tx=0).stop 2024-03-28T00:59:25.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.274+0000 7fbd21c2f740 1 -- 172.21.15.169:0/407345994 shutdown_connections 2024-03-28T00:59:25.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.274+0000 7fbd21c2f740 1 --2- 172.21.15.169:0/407345994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfe250 0x561c47d07c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.274+0000 7fbd21c2f740 1 --2- 172.21.15.169:0/407345994 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cfd860 0x561c47d0ee20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.274+0000 7fbd21c2f740 1 --2- 172.21.15.169:0/407345994 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c47cf8dd0 0x561c47cf3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.274+0000 7fbd21c2f740 1 -- 172.21.15.169:0/407345994 >> 172.21.15.169:0/407345994 conn(0x561c47d067e0 msgr2=0x561c47d07520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:25.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.282+0000 7fbd21c2f740 1 -- 172.21.15.169:0/407345994 shutdown_connections 2024-03-28T00:59:25.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.287+0000 7fbd21c2f740 1 -- 172.21.15.169:0/407345994 wait complete. 2024-03-28T00:59:25.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.287+0000 7fbd21c2f740 1 -- start start 2024-03-28T00:59:25.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.287+0000 7fbd21c2f740 1 Processor -- start 2024-03-28T00:59:25.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.288+0000 7fbd21c2f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cf8dd0 0x561c47d0cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.288+0000 7fbd21c2f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c47cfd860 0x561c47d0d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.288+0000 7fbd21c2f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfe250 0x561c47d0e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.288+0000 7fbd21c2f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd181aeef0 con 0x561c47cfd860 2024-03-28T00:59:25.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.288+0000 7fbd21c2f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd181b01f0 con 0x561c47cfe250 2024-03-28T00:59:25.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.288+0000 7fbd21c2f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd181af530 con 0x561c47cf8dd0 2024-03-28T00:59:25.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.288+0000 7fbd21c2f740 10 client.0 _ll_register_callbacks cb 0x7fbd100edfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:59:25.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.289+0000 7fbd1d5e1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfe250 0x561c47d0e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.289+0000 7fbd1d5e1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfe250 0x561c47d0e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49126/0 (socket says 172.21.15.169:49126) 2024-03-28T00:59:25.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.289+0000 7fbd1d5e1640 1 -- 172.21.15.169:0/3285683124 learned_addr learned my addr 172.21.15.169:0/3285683124 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:25.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.289+0000 7fbd1d5e1640 1 -- 172.21.15.169:0/3285683124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cf8dd0 msgr2=0x561c47d0cc80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T00:59:25.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.289+0000 7fbd1d5e1640 1 --2- 172.21.15.169:0/3285683124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cf8dd0 0x561c47d0cc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.290+0000 7fbd1d5e1640 1 -- 172.21.15.169:0/3285683124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c47cfd860 msgr2=0x561c47d0d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:25.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.290+0000 7fbd1d5e1640 1 --2- 172.21.15.169:0/3285683124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c47cfd860 0x561c47d0d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.290+0000 7fbd1d5e1640 1 -- 172.21.15.169:0/3285683124 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c47d0e770 con 0x561c47cfe250 2024-03-28T00:59:25.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.290+0000 7fbd1d5e1640 1 --2- 172.21.15.169:0/3285683124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfe250 0x561c47d0e180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbd0001fbf0 tx=0x7fbd00009600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67531aa9e4ac655f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:25.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.290+0000 7fbcf7fff640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:59:25.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.290+0000 7fbcf7fff640 1 -- 172.21.15.169:0/3285683124 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0000fc10 con 0x561c47cfe250 2024-03-28T00:59:25.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.290+0000 7fbcf7fff640 1 -- 172.21.15.169:0/3285683124 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fbd0000fdb0 con 0x561c47cfe250 2024-03-28T00:59:25.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.290+0000 7fbd21c2f740 1 -- 172.21.15.169:0/3285683124 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbd08012270 con 0x561c47cfe250 2024-03-28T00:59:25.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.291+0000 7fbcf7fff640 1 -- 172.21.15.169:0/3285683124 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd00029420 con 0x561c47cfe250 2024-03-28T00:59:25.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.291+0000 7fbd21c2f740 10 client.25931 fetch_fsmap learned FSMap version 25 2024-03-28T00:59:25.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.291+0000 7fbd21c2f740 1 -- 172.21.15.169:0/3285683124 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561c47d6f540 con 0x561c47cfe250 2024-03-28T00:59:25.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.291+0000 7fbcf7fff640 1 -- 172.21.15.169:0/3285683124 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd08012270 con 0x561c47cfe250 2024-03-28T00:59:25.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.292+0000 7fbcf7fff640 1 -- 172.21.15.169:0/3285683124 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fbd0001fce0 con 0x561c47cfe250 2024-03-28T00:59:25.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.293+0000 7fbd21c2f740 10 client.25931 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:59:25.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.293+0000 7fbd21c2f740 10 client.25931 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T00:59:25.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.293+0000 7fbd21c2f740 20 client.25931 populate_metadata read hostname 'smithi169' 2024-03-28T00:59:25.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.293+0000 7fbd21c2f740 1 --2- 172.21.15.169:0/3285683124 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x561c47d033c0 0x561c47d71fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.293+0000 7fbd21c2f740 4 client.25931 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T00:59:25.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.293+0000 7fbd21c2f740 1 -- 172.21.15.169:0/3285683124 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561c47c6ac50 con 0x561c47d033c0 2024-03-28T00:59:25.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.294+0000 7fbd1cde0640 1 --2- 172.21.15.169:0/3285683124 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x561c47d033c0 0x561c47d71fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.297+0000 7fbd1cde0640 1 --2- 172.21.15.169:0/3285683124 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x561c47d033c0 0x561c47d71fd0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a171ceebe7cb055d server_cookie=fe493c1a7af0c621 in_seq=0 out_seq=0 2024-03-28T00:59:25.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.298+0000 7fbcf7fff640 10 client.25931 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T00:59:25.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.301+0000 7fbcf7fff640 1 -- 172.21.15.169:0/3285683124 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561c47c6ac50 con 0x561c47d033c0 2024-03-28T00:59:25.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.302+0000 7fbcf7fff640 10 client.25931 handle_command_reply: tid=0 2024-03-28T00:59:25.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.302+0000 7fbd21c2f740 1 client.25931 shutdown 2024-03-28T00:59:25.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.302+0000 7fbd21c2f740 1 -- 172.21.15.169:0/3285683124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfe250 msgr2=0x561c47d0e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:25.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.302+0000 7fbd21c2f740 1 --2- 172.21.15.169:0/3285683124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfe250 0x561c47d0e180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbd0001fbf0 tx=0x7fbd00009600 comp rx=0 tx=0).stop 2024-03-28T00:59:25.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.305+0000 7fbd21c2f740 1 -- 172.21.15.169:0/3285683124 shutdown_connections 2024-03-28T00:59:25.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.305+0000 7fbd21c2f740 1 --2- 172.21.15.169:0/3285683124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c47cfe250 0x561c47d0e180 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.305+0000 7fbd21c2f740 1 --2- 172.21.15.169:0/3285683124 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x561c47d033c0 0x561c47d71fd0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.305+0000 7fbd21c2f740 1 --2- 172.21.15.169:0/3285683124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c47cfd860 0x561c47d0d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.305+0000 7fbd21c2f740 1 --2- 172.21.15.169:0/3285683124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c47cf8dd0 0x561c47d0cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.305+0000 7fbd21c2f740 1 -- 172.21.15.169:0/3285683124 >> 172.21.15.169:0/3285683124 conn(0x561c47d067e0 msgr2=0x561c47cfc030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:25.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.306+0000 7fbcf7fff640 0 client.25931 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T00:59:25.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.308+0000 7fbd21c2f740 1 -- 172.21.15.169:0/3285683124 shutdown_connections 2024-03-28T00:59:25.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.309+0000 7fbd21c2f740 1 -- 172.21.15.169:0/3285683124 wait complete. 2024-03-28T00:59:25.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.309+0000 7fbd21c2f740 20 client.25931 trim_cache size 0 max 16384 2024-03-28T00:59:25.315 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T00:59:25.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.317+0000 7fbd1f044640 1 -- 172.21.15.169:0/1241514966 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbd040a2a00 msgr2=0x7fbd040a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:25.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.318+0000 7fbd1f044640 1 --2- 172.21.15.169:0/1241514966 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbd040a2a00 0x7fbd040a4ec0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x561c47cfeab0 tx=0x7fbd000046c0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.318+0000 7fbd1f044640 1 -- 172.21.15.169:0/1241514966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd18072a10 msgr2=0x7fbd18114d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:25.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.318+0000 7fbd1f044640 1 --2- 172.21.15.169:0/1241514966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd18072a10 0x7fbd18114d10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbd10000c00 tx=0x7fbd10032460 comp rx=0 tx=0).stop 2024-03-28T00:59:25.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.318+0000 7fbd1f044640 1 -- 172.21.15.169:0/1241514966 shutdown_connections 2024-03-28T00:59:25.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.318+0000 7fbd1f044640 1 --2- 172.21.15.169:0/1241514966 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbd040a2a00 0x7fbd040a4ec0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.318+0000 7fbd1f044640 1 --2- 172.21.15.169:0/1241514966 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd181096a0 0x7fbd181a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.318+0000 7fbd1f044640 1 --2- 172.21.15.169:0/1241514966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd18108ce0 0x7fbd18115250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.318+0000 7fbd1f044640 1 --2- 172.21.15.169:0/1241514966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd18072a10 0x7fbd18114d10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.318+0000 7fbd1f044640 1 -- 172.21.15.169:0/1241514966 >> 172.21.15.169:0/1241514966 conn(0x7fbd1806e4c0 msgr2=0x7fbd180700e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:25.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.318+0000 7fbd1f044640 1 -- 172.21.15.169:0/1241514966 shutdown_connections 2024-03-28T00:59:25.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.318+0000 7fbd1f044640 1 -- 172.21.15.169:0/1241514966 wait complete. 2024-03-28T00:59:25.385 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T00:59:25.385 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T00:59:25.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:25 smithi174 ceph-mon[36060]: pgmap v438: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 790 MiB/s wr, 1.56k op/s 2024-03-28T00:59:25.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:25 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2443351750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:59:25.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:25 smithi177 ceph-mon[36206]: pgmap v438: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 790 MiB/s wr, 1.56k op/s 2024-03-28T00:59:25.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:25 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2443351750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:59:25.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.728+0000 7f2a0c5e6640 1 -- 172.21.15.169:0/2709073379 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a04101030 msgr2=0x7f2a041058f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:25.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.728+0000 7f2a0c5e6640 1 --2- 172.21.15.169:0/2709073379 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a04101030 0x7f2a041058f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f29f4009900 tx=0x7f29f402bc80 comp rx=0 tx=0).stop 2024-03-28T00:59:25.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.736+0000 7f2a0c5e6640 1 -- 172.21.15.169:0/2709073379 shutdown_connections 2024-03-28T00:59:25.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.736+0000 7f2a0c5e6640 1 --2- 172.21.15.169:0/2709073379 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a04101030 0x7f2a041058f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.736+0000 7f2a0c5e6640 1 --2- 172.21.15.169:0/2709073379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a04100690 0x7f2a04100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.736+0000 7f2a0c5e6640 1 --2- 172.21.15.169:0/2709073379 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a040ff560 0x7f2a040ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.736+0000 7f2a0c5e6640 1 -- 172.21.15.169:0/2709073379 >> 172.21.15.169:0/2709073379 conn(0x7f2a040fae60 msgr2=0x7f2a040fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:25.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.742+0000 7f2a0c5e6640 1 -- 172.21.15.169:0/2709073379 shutdown_connections 2024-03-28T00:59:25.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.742+0000 7f2a0c5e6640 1 -- 172.21.15.169:0/2709073379 wait complete. 2024-03-28T00:59:25.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.743+0000 7f2a0c5e6640 1 Processor -- start 2024-03-28T00:59:25.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.743+0000 7f2a0c5e6640 1 -- start start 2024-03-28T00:59:25.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.744+0000 7f2a0c5e6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a040ff560 0x7f2a0419df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.744+0000 7f2a0c5e6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a04100690 0x7f2a0419e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.744+0000 7f2a0c5e6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a04101030 0x7f2a041a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.744+0000 7f2a0c5e6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a04113940 con 0x7f2a040ff560 2024-03-28T00:59:25.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.744+0000 7f2a0c5e6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a041137c0 con 0x7f2a04101030 2024-03-28T00:59:25.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.744+0000 7f2a0c5e6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a04113ac0 con 0x7f2a04100690 2024-03-28T00:59:25.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.744+0000 7f2a0a382640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a040ff560 0x7f2a0419df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.744+0000 7f2a0a382640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a040ff560 0x7f2a0419df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:32818/0 (socket says 172.21.15.169:32818) 2024-03-28T00:59:25.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.744+0000 7f2a0a382640 1 -- 172.21.15.169:0/1789345958 learned_addr learned my addr 172.21.15.169:0/1789345958 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:25.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.744+0000 7f2a0a382640 1 -- 172.21.15.169:0/1789345958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a04100690 msgr2=0x7f2a0419e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:25.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.744+0000 7f2a0a382640 1 --2- 172.21.15.169:0/1789345958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a04100690 0x7f2a0419e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.744+0000 7f2a0a382640 1 -- 172.21.15.169:0/1789345958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a04101030 msgr2=0x7f2a041a5540 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T00:59:25.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.744+0000 7f2a0a382640 1 --2- 172.21.15.169:0/1789345958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a04101030 0x7f2a041a5540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.745+0000 7f2a0a382640 1 -- 172.21.15.169:0/1789345958 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a041a5bb0 con 0x7f2a040ff560 2024-03-28T00:59:25.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.745+0000 7f2a0a382640 1 --2- 172.21.15.169:0/1789345958 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a040ff560 0x7f2a0419df90 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f29fc00eab0 tx=0x7f29fc00ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:25.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.760+0000 7f29fb7fe640 1 -- 172.21.15.169:0/1789345958 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29fc00ce40 con 0x7f2a040ff560 2024-03-28T00:59:25.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.761+0000 7f29fb7fe640 1 -- 172.21.15.169:0/1789345958 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29fc011950 con 0x7f2a040ff560 2024-03-28T00:59:25.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.761+0000 7f29fb7fe640 1 -- 172.21.15.169:0/1789345958 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29fc010a70 con 0x7f2a040ff560 2024-03-28T00:59:25.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.763+0000 7f2a0c5e6640 1 -- 172.21.15.169:0/1789345958 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a041a5ea0 con 0x7f2a040ff560 2024-03-28T00:59:25.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.763+0000 7f2a0c5e6640 1 -- 172.21.15.169:0/1789345958 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a041a6310 con 0x7f2a040ff560 2024-03-28T00:59:25.777 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.772+0000 7f29fb7fe640 1 -- 172.21.15.169:0/1789345958 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f29fc010c10 con 0x7f2a040ff560 2024-03-28T00:59:25.777 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.773+0000 7f29fb7fe640 1 --2- 172.21.15.169:0/1789345958 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f29f00a2980 0x7f29f00a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.777 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.774+0000 7f29fb7fe640 1 -- 172.21.15.169:0/1789345958 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f29fc0d0340 con 0x7f2a040ff560 2024-03-28T00:59:25.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.775+0000 7f2a09b81640 1 --2- 172.21.15.169:0/1789345958 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f29f00a2980 0x7f29f00a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.776+0000 7f2a0f1d5740 1 Processor -- start 2024-03-28T00:59:25.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.776+0000 7f2a0f1d5740 1 -- start start 2024-03-28T00:59:25.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.776+0000 7f2a09b81640 1 --2- 172.21.15.169:0/1789345958 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f29f00a2980 0x7f29f00a4e40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2a0419f4a0 tx=0x7f2a00005bf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:25.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.777+0000 7f2a0f1d5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a819465690 0x55a819465ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.777+0000 7f2a0f1d5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819466080 0x55a81946ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.777+0000 7f2a0f1d5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a81946b1f0 0x55a81946d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.777+0000 7f2a0f1d5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a819428610 con 0x55a819466080 2024-03-28T00:59:25.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.777+0000 7f2a0f1d5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8193cb3c0 con 0x55a81946b1f0 2024-03-28T00:59:25.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.777+0000 7f2a0f1d5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a819373d60 con 0x55a819465690 2024-03-28T00:59:25.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.777+0000 7f2a09b81640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819466080 0x55a81946ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.777+0000 7f2a09b81640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819466080 0x55a81946ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:32830/0 (socket says 172.21.15.169:32830) 2024-03-28T00:59:25.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.777+0000 7f2a09b81640 1 -- 172.21.15.169:0/4015614999 learned_addr learned my addr 172.21.15.169:0/4015614999 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:25.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.778+0000 7f2a09b81640 1 -- 172.21.15.169:0/4015614999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a819465690 msgr2=0x55a819465ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:25.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.778+0000 7f2a09b81640 1 --2- 172.21.15.169:0/4015614999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a819465690 0x55a819465ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.778+0000 7f2a09b81640 1 -- 172.21.15.169:0/4015614999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a81946b1f0 msgr2=0x55a81946d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:25.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.778+0000 7f2a09b81640 1 --2- 172.21.15.169:0/4015614999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a81946b1f0 0x55a81946d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.778+0000 7f2a09b81640 1 -- 172.21.15.169:0/4015614999 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8190df3f0 con 0x55a819466080 2024-03-28T00:59:25.789 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.785+0000 7f2a09b81640 1 --2- 172.21.15.169:0/4015614999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819466080 0x55a81946ab80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2a0000afd0 tx=0x7f2a0000f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54fa43e021149c83 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:25.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.787+0000 7f29f8bfe640 1 -- 172.21.15.169:0/4015614999 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00020070 con 0x55a819466080 2024-03-28T00:59:25.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.787+0000 7f29f8bfe640 1 -- 172.21.15.169:0/4015614999 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a00016070 con 0x55a819466080 2024-03-28T00:59:25.792 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.788+0000 7f29f8bfe640 1 -- 172.21.15.169:0/4015614999 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0000e750 con 0x55a819466080 2024-03-28T00:59:25.792 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.788+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/4015614999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819466080 msgr2=0x55a81946ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:25.792 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.788+0000 7f2a0f1d5740 1 --2- 172.21.15.169:0/4015614999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819466080 0x55a81946ab80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2a0000afd0 tx=0x7f2a0000f000 comp rx=0 tx=0).stop 2024-03-28T00:59:25.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.789+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/4015614999 shutdown_connections 2024-03-28T00:59:25.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.789+0000 7f2a0f1d5740 1 --2- 172.21.15.169:0/4015614999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a81946b1f0 0x55a81946d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.789+0000 7f2a0f1d5740 1 --2- 172.21.15.169:0/4015614999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819466080 0x55a81946ab80 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.789+0000 7f2a0f1d5740 1 --2- 172.21.15.169:0/4015614999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a819465690 0x55a819465ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.789+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/4015614999 >> 172.21.15.169:0/4015614999 conn(0x55a819460c00 msgr2=0x55a819461000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:25.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.789+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/4015614999 shutdown_connections 2024-03-28T00:59:25.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.789+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/4015614999 wait complete. 2024-03-28T00:59:25.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.790+0000 7f2a0f1d5740 1 -- start start 2024-03-28T00:59:25.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.790+0000 7f2a0f1d5740 1 Processor -- start 2024-03-28T00:59:25.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.793+0000 7f2a0f1d5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819465690 0x55a81946fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.793+0000 7f2a0f1d5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a819466080 0x55a819470070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.793+0000 7f2a0f1d5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a81946b1f0 0x55a8194770b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.793+0000 7f2a0f1d5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a819387240 con 0x55a819465690 2024-03-28T00:59:25.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.793+0000 7f2a0f1d5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8193c7460 con 0x55a81946b1f0 2024-03-28T00:59:25.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.793+0000 7f2a0f1d5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a81946e1f0 con 0x55a819466080 2024-03-28T00:59:25.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.793+0000 7f2a0f1d5740 10 client.0 _ll_register_callbacks cb 0x55a819386c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:59:25.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:25 smithi169 ceph-mon[32274]: pgmap v438: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 790 MiB/s wr, 1.56k op/s 2024-03-28T00:59:25.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:25 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2443351750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T00:59:25.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.797+0000 7f2a0a382640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819465690 0x55a81946fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.797+0000 7f2a0a382640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819465690 0x55a81946fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:32832/0 (socket says 172.21.15.169:32832) 2024-03-28T00:59:25.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.797+0000 7f2a0a382640 1 -- 172.21.15.169:0/1152322779 learned_addr learned my addr 172.21.15.169:0/1152322779 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:25.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.798+0000 7f2a0a382640 1 -- 172.21.15.169:0/1152322779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a819466080 msgr2=0x55a819470070 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T00:59:25.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.798+0000 7f2a0a382640 1 --2- 172.21.15.169:0/1152322779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a819466080 0x55a819470070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.798+0000 7f2a0a382640 1 -- 172.21.15.169:0/1152322779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a81946b1f0 msgr2=0x55a8194770b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T00:59:25.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.798+0000 7f2a0a382640 1 --2- 172.21.15.169:0/1152322779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a81946b1f0 0x55a8194770b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.798+0000 7f2a0a382640 1 -- 172.21.15.169:0/1152322779 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a8191a97d0 con 0x55a819465690 2024-03-28T00:59:25.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.799+0000 7f2a0a382640 1 --2- 172.21.15.169:0/1152322779 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819465690 0x55a81946fb30 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f29fc09c6e0 tx=0x7f29fc09c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe3c9f85832b4402 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:25.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.799+0000 7f29edffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:59:25.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.801+0000 7f29edffb640 1 -- 172.21.15.169:0/1152322779 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29fc0d0160 con 0x55a819465690 2024-03-28T00:59:25.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.801+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/1152322779 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2a00016070 con 0x55a819465690 2024-03-28T00:59:25.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.801+0000 7f29edffb640 1 -- 172.21.15.169:0/1152322779 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f29fc020d90 con 0x55a819465690 2024-03-28T00:59:25.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.805+0000 7f29edffb640 1 -- 172.21.15.169:0/1152322779 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29fc09c9e0 con 0x55a819465690 2024-03-28T00:59:25.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.805+0000 7f2a0f1d5740 10 client.16098 fetch_fsmap learned FSMap version 25 2024-03-28T00:59:25.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.805+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/1152322779 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a81947a490 con 0x55a819465690 2024-03-28T00:59:25.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.806+0000 7f29edffb640 1 -- 172.21.15.169:0/1152322779 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a00016070 con 0x55a819465690 2024-03-28T00:59:25.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.806+0000 7f29edffb640 1 -- 172.21.15.169:0/1152322779 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f29fc00f040 con 0x55a819465690 2024-03-28T00:59:25.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.807+0000 7f2a0f1d5740 10 client.16098 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:59:25.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.807+0000 7f2a0f1d5740 10 client.16098 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T00:59:25.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.807+0000 7f2a0f1d5740 20 client.16098 populate_metadata read hostname 'smithi169' 2024-03-28T00:59:25.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.807+0000 7f2a0f1d5740 1 --2- 172.21.15.169:0/1152322779 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55a81947c070 0x55a81947e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.807+0000 7f2a0f1d5740 4 client.16098 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:59:25.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.807+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/1152322779 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a8193d8900 con 0x55a81947c070 2024-03-28T00:59:25.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.810+0000 7f2a09b81640 1 --2- 172.21.15.169:0/1152322779 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55a81947c070 0x55a81947e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.820+0000 7f2a09b81640 1 --2- 172.21.15.169:0/1152322779 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55a81947c070 0x55a81947e490 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b4830603b9f68f82 server_cookie=93ea2a6b15676732 in_seq=0 out_seq=0 2024-03-28T00:59:25.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.820+0000 7f29edffb640 10 client.16098 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T00:59:25.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.823+0000 7f29edffb640 1 -- 172.21.15.169:0/1152322779 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a8193d8900 con 0x55a81947c070 2024-03-28T00:59:25.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.823+0000 7f29edffb640 10 client.16098 handle_command_reply: tid=0 2024-03-28T00:59:25.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.824+0000 7f2a0f1d5740 1 client.16098 shutdown 2024-03-28T00:59:25.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.829+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/1152322779 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819465690 msgr2=0x55a81946fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:25.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.829+0000 7f2a0f1d5740 1 --2- 172.21.15.169:0/1152322779 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819465690 0x55a81946fb30 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f29fc09c6e0 tx=0x7f29fc09c520 comp rx=0 tx=0).stop 2024-03-28T00:59:25.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.829+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/1152322779 shutdown_connections 2024-03-28T00:59:25.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.829+0000 7f2a0f1d5740 1 --2- 172.21.15.169:0/1152322779 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55a81947c070 0x55a81947e490 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.829+0000 7f2a0f1d5740 1 --2- 172.21.15.169:0/1152322779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a81946b1f0 0x55a8194770b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.829+0000 7f2a0f1d5740 1 --2- 172.21.15.169:0/1152322779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a819466080 0x55a819470070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.829+0000 7f2a0f1d5740 1 --2- 172.21.15.169:0/1152322779 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819465690 0x55a81946fb30 secure :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f29fc09c6e0 tx=0x7f29fc09c520 comp rx=0 tx=0).stop 2024-03-28T00:59:25.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.829+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/1152322779 >> 172.21.15.169:0/1152322779 conn(0x55a819460c00 msgr2=0x55a81945fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:25.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.829+0000 7f29edffb640 0 client.16098 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T00:59:25.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.829+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/1152322779 shutdown_connections 2024-03-28T00:59:25.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.829+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/1152322779 wait complete. 2024-03-28T00:59:25.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.832+0000 7f2a0f1d5740 20 client.16098 trim_cache size 0 max 16384 2024-03-28T00:59:25.876 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.873+0000 7f2a0f1d5740 1 Processor -- start 2024-03-28T00:59:25.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.877+0000 7f2a0f1d5740 1 -- start start 2024-03-28T00:59:25.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.877+0000 7f2a0f1d5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a819466080 0x55a81946abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.877+0000 7f2a0f1d5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a81946b1f0 0x55a819477070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.877+0000 7f2a0f1d5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819461440 0x55a81946fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.877+0000 7f2a0f1d5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e8012090 con 0x55a819461440 2024-03-28T00:59:25.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.878+0000 7f2a0f1d5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e800fb90 con 0x55a81946b1f0 2024-03-28T00:59:25.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.878+0000 7f2a0f1d5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e800dc30 con 0x55a819466080 2024-03-28T00:59:25.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.878+0000 7f2a0ab83640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819461440 0x55a81946fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.878+0000 7f2a09b81640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a81946b1f0 0x55a819477070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.878+0000 7f2a09b81640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a81946b1f0 0x55a819477070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49152/0 (socket says 172.21.15.169:49152) 2024-03-28T00:59:25.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.878+0000 7f2a09b81640 1 -- 172.21.15.169:0/2502546742 learned_addr learned my addr 172.21.15.169:0/2502546742 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:25.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.879+0000 7f2a0a382640 1 --2- 172.21.15.169:0/2502546742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a819466080 0x55a81946abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.879+0000 7f2a0ab83640 1 -- 172.21.15.169:0/2502546742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a819466080 msgr2=0x55a81946abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:25.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.879+0000 7f2a0ab83640 1 --2- 172.21.15.169:0/2502546742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a819466080 0x55a81946abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.879+0000 7f2a0ab83640 1 -- 172.21.15.169:0/2502546742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a81946b1f0 msgr2=0x55a819477070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:25.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.879+0000 7f2a0ab83640 1 --2- 172.21.15.169:0/2502546742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a81946b1f0 0x55a819477070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.879+0000 7f2a0ab83640 1 -- 172.21.15.169:0/2502546742 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a81945c670 con 0x55a819461440 2024-03-28T00:59:25.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.880+0000 7f2a0ab83640 1 --2- 172.21.15.169:0/2502546742 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819461440 0x55a81946fe50 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f29f4009850 tx=0x7f29f4009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adce17f2b3d7881 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:25.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.881+0000 7f29effff640 1 -- 172.21.15.169:0/2502546742 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f4039040 con 0x55a819461440 2024-03-28T00:59:25.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.881+0000 7f29effff640 1 -- 172.21.15.169:0/2502546742 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29f4004780 con 0x55a819461440 2024-03-28T00:59:25.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.881+0000 7f29effff640 1 -- 172.21.15.169:0/2502546742 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f40035f0 con 0x55a819461440 2024-03-28T00:59:25.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.885+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/2502546742 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819461440 msgr2=0x55a81946fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:25.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.885+0000 7f2a0f1d5740 1 --2- 172.21.15.169:0/2502546742 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819461440 0x55a81946fe50 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f29f4009850 tx=0x7f29f4009510 comp rx=0 tx=0).stop 2024-03-28T00:59:25.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.885+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/2502546742 shutdown_connections 2024-03-28T00:59:25.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.885+0000 7f2a0f1d5740 1 --2- 172.21.15.169:0/2502546742 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819461440 0x55a81946fe50 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.885+0000 7f2a0f1d5740 1 --2- 172.21.15.169:0/2502546742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a81946b1f0 0x55a819477070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.885+0000 7f2a0f1d5740 1 --2- 172.21.15.169:0/2502546742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a819466080 0x55a81946abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.885+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/2502546742 >> 172.21.15.169:0/2502546742 conn(0x55a819460c00 msgr2=0x55a81946f740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:25.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.888+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/2502546742 shutdown_connections 2024-03-28T00:59:25.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.888+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/2502546742 wait complete. 2024-03-28T00:59:25.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.889+0000 7f2a0f1d5740 1 -- start start 2024-03-28T00:59:25.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.889+0000 7f2a0f1d5740 1 Processor -- start 2024-03-28T00:59:25.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.889+0000 7f2a0f1d5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819461440 0x55a819474a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.889+0000 7f2a0f1d5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a819466080 0x55a819474fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.889+0000 7f2a0f1d5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a81946b1f0 0x55a819475f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.890+0000 7f2a0f1d5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e800e230 con 0x55a819461440 2024-03-28T00:59:25.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.890+0000 7f2a0f1d5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e800f530 con 0x55a819466080 2024-03-28T00:59:25.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.890+0000 7f2a0f1d5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e800e870 con 0x55a81946b1f0 2024-03-28T00:59:25.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.890+0000 7f2a0f1d5740 10 client.0 _ll_register_callbacks cb 0x7f29fc098ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:59:25.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.890+0000 7f2a0a382640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819461440 0x55a819474a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.890+0000 7f2a0a382640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819461440 0x55a819474a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:32856/0 (socket says 172.21.15.169:32856) 2024-03-28T00:59:25.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.890+0000 7f2a0a382640 1 -- 172.21.15.169:0/3786891475 learned_addr learned my addr 172.21.15.169:0/3786891475 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:25.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.890+0000 7f2a09b81640 1 --2- 172.21.15.169:0/3786891475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a819466080 0x55a819474fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.890+0000 7f2a0ab83640 1 --2- 172.21.15.169:0/3786891475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a81946b1f0 0x55a819475f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.891+0000 7f2a0ab83640 1 -- 172.21.15.169:0/3786891475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a819466080 msgr2=0x55a819474fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:25.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.891+0000 7f2a0ab83640 1 --2- 172.21.15.169:0/3786891475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a819466080 0x55a819474fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.891+0000 7f2a0ab83640 1 -- 172.21.15.169:0/3786891475 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819461440 msgr2=0x55a819474a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:25.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.891+0000 7f2a0ab83640 1 --2- 172.21.15.169:0/3786891475 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819461440 0x55a819474a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.891+0000 7f2a0ab83640 1 -- 172.21.15.169:0/3786891475 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a819476560 con 0x55a81946b1f0 2024-03-28T00:59:25.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.896+0000 7f2a0ab83640 1 --2- 172.21.15.169:0/3786891475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a81946b1f0 0x55a819475f70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f29f402bcc0 tx=0x7f29f402c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c4b1658da25c877 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:25.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.897+0000 7f29edffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:59:25.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.897+0000 7f29edffb640 1 -- 172.21.15.169:0/3786891475 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f4049020 con 0x55a81946b1f0 2024-03-28T00:59:25.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.897+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/3786891475 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f29f4004780 con 0x55a81946b1f0 2024-03-28T00:59:25.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.897+0000 7f29edffb640 1 -- 172.21.15.169:0/3786891475 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f29f4004f70 con 0x55a81946b1f0 2024-03-28T00:59:25.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.898+0000 7f29edffb640 1 -- 172.21.15.169:0/3786891475 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29f4054050 con 0x55a81946b1f0 2024-03-28T00:59:25.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.900+0000 7f2a0f1d5740 10 client.25903 fetch_fsmap learned FSMap version 25 2024-03-28T00:59:25.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.901+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/3786891475 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a8194e8c50 con 0x55a81946b1f0 2024-03-28T00:59:25.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.901+0000 7f29edffb640 1 -- 172.21.15.169:0/3786891475 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29f4004780 con 0x55a81946b1f0 2024-03-28T00:59:25.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.902+0000 7f29edffb640 1 -- 172.21.15.169:0/3786891475 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f29f403c070 con 0x55a81946b1f0 2024-03-28T00:59:25.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.904+0000 7f2a0f1d5740 10 client.25903 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:59:25.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.904+0000 7f2a0f1d5740 10 client.25903 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T00:59:25.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.904+0000 7f2a0f1d5740 20 client.25903 populate_metadata read hostname 'smithi169' 2024-03-28T00:59:25.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.904+0000 7f2a0f1d5740 1 --2- 172.21.15.169:0/3786891475 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55a8194e94e0 0x55a8194eb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:25.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.904+0000 7f2a0f1d5740 4 client.25903 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T00:59:25.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.904+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/3786891475 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a8193d2c50 con 0x55a8194e94e0 2024-03-28T00:59:25.913 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.910+0000 7f2a0a382640 1 --2- 172.21.15.169:0/3786891475 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55a8194e94e0 0x55a8194eb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:25.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.928+0000 7f2a0a382640 1 --2- 172.21.15.169:0/3786891475 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55a8194e94e0 0x55a8194eb900 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e825e2db7e0da636 server_cookie=2ac19602ebb99506 in_seq=0 out_seq=0 2024-03-28T00:59:25.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.928+0000 7f29edffb640 10 client.25903 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T00:59:25.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.929+0000 7f29edffb640 1 -- 172.21.15.169:0/3786891475 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a8193d2c50 con 0x55a8194e94e0 2024-03-28T00:59:25.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.929+0000 7f29edffb640 10 client.25903 handle_command_reply: tid=0 2024-03-28T00:59:25.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.931+0000 7f2a0f1d5740 1 client.25903 shutdown 2024-03-28T00:59:25.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.936+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/3786891475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a81946b1f0 msgr2=0x55a819475f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:25.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.936+0000 7f2a0f1d5740 1 --2- 172.21.15.169:0/3786891475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a81946b1f0 0x55a819475f70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f29f402bcc0 tx=0x7f29f402c850 comp rx=0 tx=0).stop 2024-03-28T00:59:25.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.936+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/3786891475 shutdown_connections 2024-03-28T00:59:25.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.936+0000 7f2a0f1d5740 1 --2- 172.21.15.169:0/3786891475 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55a8194e94e0 0x55a8194eb900 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.936+0000 7f2a0f1d5740 1 --2- 172.21.15.169:0/3786891475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a81946b1f0 0x55a819475f70 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.936+0000 7f2a0f1d5740 1 --2- 172.21.15.169:0/3786891475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a819466080 0x55a819474fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.936+0000 7f2a0f1d5740 1 --2- 172.21.15.169:0/3786891475 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a819461440 0x55a819474a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.936+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/3786891475 >> 172.21.15.169:0/3786891475 conn(0x55a819460c00 msgr2=0x55a81946f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:25.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.937+0000 7f29edffb640 0 client.25903 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T00:59:25.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.938+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/3786891475 shutdown_connections 2024-03-28T00:59:25.949 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T00:59:25.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.943+0000 7f2a0f1d5740 1 -- 172.21.15.169:0/3786891475 wait complete. 2024-03-28T00:59:25.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.943+0000 7f2a0f1d5740 20 client.25903 trim_cache size 0 max 16384 2024-03-28T00:59:25.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.959+0000 7f2a0c5e6640 1 -- 172.21.15.169:0/1789345958 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f29f00a2980 msgr2=0x7f29f00a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:25.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.959+0000 7f2a0c5e6640 1 --2- 172.21.15.169:0/1789345958 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f29f00a2980 0x7f29f00a4e40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2a0419f4a0 tx=0x7f2a00005bf0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.959+0000 7f2a0c5e6640 1 -- 172.21.15.169:0/1789345958 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a040ff560 msgr2=0x7f2a0419df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:25.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.959+0000 7f2a0c5e6640 1 --2- 172.21.15.169:0/1789345958 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a040ff560 0x7f2a0419df90 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f29fc00eab0 tx=0x7f29fc00ef70 comp rx=0 tx=0).stop 2024-03-28T00:59:25.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.964+0000 7f2a0c5e6640 1 -- 172.21.15.169:0/1789345958 shutdown_connections 2024-03-28T00:59:25.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.964+0000 7f2a0c5e6640 1 --2- 172.21.15.169:0/1789345958 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f29f00a2980 0x7f29f00a4e40 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.964+0000 7f2a0c5e6640 1 --2- 172.21.15.169:0/1789345958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a04101030 0x7f2a041a5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.964+0000 7f2a0c5e6640 1 --2- 172.21.15.169:0/1789345958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a04100690 0x7f2a0419e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.964+0000 7f2a0c5e6640 1 --2- 172.21.15.169:0/1789345958 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a040ff560 0x7f2a0419df90 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:25.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.964+0000 7f2a0c5e6640 1 -- 172.21.15.169:0/1789345958 >> 172.21.15.169:0/1789345958 conn(0x7f2a040fae60 msgr2=0x7f2a041018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:25.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.965+0000 7f2a0c5e6640 1 -- 172.21.15.169:0/1789345958 shutdown_connections 2024-03-28T00:59:25.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:25.965+0000 7f2a0c5e6640 1 -- 172.21.15.169:0/1789345958 wait complete. 2024-03-28T00:59:26.020 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T00:59:26.020 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T00:59:26.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.315+0000 7fb3ad137640 1 -- 172.21.15.169:0/3558918536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3a00a4c60 msgr2=0x7fb3a00a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:26.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.315+0000 7fb3ad137640 1 --2- 172.21.15.169:0/3558918536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3a00a4c60 0x7fb3a00a5080 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8067370 tx=0x7fb3a808f060 comp rx=0 tx=0).stop 2024-03-28T00:59:26.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.316+0000 7fb3ad137640 1 -- 172.21.15.169:0/3558918536 shutdown_connections 2024-03-28T00:59:26.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.316+0000 7fb3ad137640 1 --2- 172.21.15.169:0/3558918536 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3a00a6730 0x7fb3a00aaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.316+0000 7fb3ad137640 1 --2- 172.21.15.169:0/3558918536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3a00a5d90 0x7fb3a00a61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.316+0000 7fb3ad137640 1 --2- 172.21.15.169:0/3558918536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3a00a4c60 0x7fb3a00a5080 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.316+0000 7fb3ad137640 1 -- 172.21.15.169:0/3558918536 >> 172.21.15.169:0/3558918536 conn(0x7fb3a00a00f0 msgr2=0x7fb3a00a2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:26.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.324+0000 7fb3ad137640 1 -- 172.21.15.169:0/3558918536 shutdown_connections 2024-03-28T00:59:26.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.325+0000 7fb3ad137640 1 -- 172.21.15.169:0/3558918536 wait complete. 2024-03-28T00:59:26.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.326+0000 7fb3ad137640 1 Processor -- start 2024-03-28T00:59:26.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.330+0000 7fb3ad137640 1 -- start start 2024-03-28T00:59:26.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.330+0000 7fb3ad137640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3a00a4c60 0x7fb3a00b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:26.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.330+0000 7fb3ad137640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3a00a5d90 0x7fb3a00b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:26.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.330+0000 7fb3ad137640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3a00a6730 0x7fb3a0153680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:26.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.331+0000 7fb3ad137640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a00b8fb0 con 0x7fb3a00a6730 2024-03-28T00:59:26.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.331+0000 7fb3ad137640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a00b8e30 con 0x7fb3a00a4c60 2024-03-28T00:59:26.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.331+0000 7fb3ad137640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a00b9130 con 0x7fb3a00a5d90 2024-03-28T00:59:26.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.332+0000 7fb3a6d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3a00a4c60 0x7fb3a00b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:26.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.332+0000 7fb3a659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3a00a5d90 0x7fb3a00b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:26.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.333+0000 7fb3a6d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3a00a4c60 0x7fb3a00b38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49176/0 (socket says 172.21.15.169:49176) 2024-03-28T00:59:26.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.333+0000 7fb3a6d9d640 1 -- 172.21.15.169:0/1655535321 learned_addr learned my addr 172.21.15.169:0/1655535321 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:26.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.335+0000 7fb3a759e640 1 --2- 172.21.15.169:0/1655535321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3a00a6730 0x7fb3a0153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:26.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.336+0000 7fb3a6d9d640 1 -- 172.21.15.169:0/1655535321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3a00a5d90 msgr2=0x7fb3a00b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:26.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.337+0000 7fb3a6d9d640 1 --2- 172.21.15.169:0/1655535321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3a00a5d90 0x7fb3a00b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.337+0000 7fb3a6d9d640 1 -- 172.21.15.169:0/1655535321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3a00a6730 msgr2=0x7fb3a0153680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:26.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.337+0000 7fb3a6d9d640 1 --2- 172.21.15.169:0/1655535321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3a00a6730 0x7fb3a0153680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.337+0000 7fb3a6d9d640 1 -- 172.21.15.169:0/1655535321 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a0153bc0 con 0x7fb3a00a4c60 2024-03-28T00:59:26.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.338+0000 7fb3a759e640 1 --2- 172.21.15.169:0/1655535321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3a00a6730 0x7fb3a0153680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:59:26.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.338+0000 7fb3a6d9d640 1 --2- 172.21.15.169:0/1655535321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3a00a4c60 0x7fb3a00b38a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8064b90 tx=0x7fb3a80693d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:26.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.342+0000 7fb38ffff640 1 -- 172.21.15.169:0/1655535321 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a80a6070 con 0x7fb3a00a4c60 2024-03-28T00:59:26.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.342+0000 7fb38ffff640 1 -- 172.21.15.169:0/1655535321 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3a808fcd0 con 0x7fb3a00a4c60 2024-03-28T00:59:26.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.342+0000 7fb38ffff640 1 -- 172.21.15.169:0/1655535321 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a80a1780 con 0x7fb3a00a4c60 2024-03-28T00:59:26.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.342+0000 7fb3ad137640 1 -- 172.21.15.169:0/1655535321 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a0153d90 con 0x7fb3a00a4c60 2024-03-28T00:59:26.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.342+0000 7fb3ad137640 1 -- 172.21.15.169:0/1655535321 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a0154140 con 0x7fb3a00a4c60 2024-03-28T00:59:26.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.351+0000 7fb3afdfb740 1 Processor -- start 2024-03-28T00:59:26.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.351+0000 7fb3afdfb740 1 -- start start 2024-03-28T00:59:26.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.351+0000 7fb3afdfb740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b80860 0x55ca06b80c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:26.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.351+0000 7fb3afdfb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b81250 0x55ca06b85d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:26.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.351+0000 7fb3afdfb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca06b863c0 0x55ca06b88850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:26.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.351+0000 7fb3afdfb740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca06b43610 con 0x55ca06b80860 2024-03-28T00:59:26.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.351+0000 7fb3afdfb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca06ae63c0 con 0x55ca06b81250 2024-03-28T00:59:26.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.351+0000 7fb3afdfb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca06a8ed60 con 0x55ca06b863c0 2024-03-28T00:59:26.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.351+0000 7fb3a6d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b81250 0x55ca06b85d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:26.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.351+0000 7fb3a6d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b81250 0x55ca06b85d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49192/0 (socket says 172.21.15.169:49192) 2024-03-28T00:59:26.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.351+0000 7fb3a6d9d640 1 -- 172.21.15.169:0/2940271671 learned_addr learned my addr 172.21.15.169:0/2940271671 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:26.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.352+0000 7fb3a659c640 1 --2- 172.21.15.169:0/2940271671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca06b863c0 0x55ca06b88850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:26.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.352+0000 7fb3a6d9d640 1 -- 172.21.15.169:0/2940271671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca06b863c0 msgr2=0x55ca06b88850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:26.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.352+0000 7fb3a6d9d640 1 --2- 172.21.15.169:0/2940271671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca06b863c0 0x55ca06b88850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.352+0000 7fb3a6d9d640 1 -- 172.21.15.169:0/2940271671 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b80860 msgr2=0x55ca06b80c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:26.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.352+0000 7fb3a6d9d640 1 --2- 172.21.15.169:0/2940271671 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b80860 0x55ca06b80c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.352+0000 7fb3a6d9d640 1 -- 172.21.15.169:0/2940271671 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca067fa3f0 con 0x55ca06b81250 2024-03-28T00:59:26.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.353+0000 7fb3a6d9d640 1 --2- 172.21.15.169:0/2940271671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b81250 0x55ca06b85d50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8066f60 tx=0x7fb3a80a1cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a629819f456986a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:26.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.358+0000 7fb38d7fa640 1 -- 172.21.15.169:0/2940271671 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a8098bd0 con 0x55ca06b81250 2024-03-28T00:59:26.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.358+0000 7fb38d7fa640 1 -- 172.21.15.169:0/2940271671 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3a8093030 con 0x55ca06b81250 2024-03-28T00:59:26.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.358+0000 7fb38d7fa640 1 -- 172.21.15.169:0/2940271671 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a80d7700 con 0x55ca06b81250 2024-03-28T00:59:26.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.359+0000 7fb3afdfb740 1 -- 172.21.15.169:0/2940271671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b81250 msgr2=0x55ca06b85d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:26.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.359+0000 7fb3afdfb740 1 --2- 172.21.15.169:0/2940271671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b81250 0x55ca06b85d50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8066f60 tx=0x7fb3a80a1cf0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.361+0000 7fb3afdfb740 1 -- 172.21.15.169:0/2940271671 shutdown_connections 2024-03-28T00:59:26.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.362+0000 7fb3afdfb740 1 --2- 172.21.15.169:0/2940271671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca06b863c0 0x55ca06b88850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.362+0000 7fb3afdfb740 1 --2- 172.21.15.169:0/2940271671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b81250 0x55ca06b85d50 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.362+0000 7fb3afdfb740 1 --2- 172.21.15.169:0/2940271671 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b80860 0x55ca06b80c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.362+0000 7fb3afdfb740 1 -- 172.21.15.169:0/2940271671 >> 172.21.15.169:0/2940271671 conn(0x55ca06b7bdd0 msgr2=0x55ca06b7c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:26.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.364+0000 7fb3afdfb740 1 -- 172.21.15.169:0/2940271671 shutdown_connections 2024-03-28T00:59:26.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.364+0000 7fb3afdfb740 1 -- 172.21.15.169:0/2940271671 wait complete. 2024-03-28T00:59:26.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.364+0000 7fb3afdfb740 1 -- start start 2024-03-28T00:59:26.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.364+0000 7fb3afdfb740 1 Processor -- start 2024-03-28T00:59:26.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.365+0000 7fb3afdfb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b80860 0x55ca06b8a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:26.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.365+0000 7fb3afdfb740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b81250 0x55ca06b8ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:26.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.365+0000 7fb3afdfb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca06b863c0 0x55ca06b91e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:26.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.365+0000 7fb3afdfb740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca06aa2240 con 0x55ca06b81250 2024-03-28T00:59:26.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.365+0000 7fb3afdfb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca06ae2460 con 0x55ca06b80860 2024-03-28T00:59:26.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.365+0000 7fb3afdfb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca06ae63c0 con 0x55ca06b863c0 2024-03-28T00:59:26.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.365+0000 7fb3afdfb740 10 client.0 _ll_register_callbacks cb 0x55ca06aa1c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:59:26.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.366+0000 7fb3a759e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b80860 0x55ca06b8a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:26.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.367+0000 7fb3a759e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b80860 0x55ca06b8a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49196/0 (socket says 172.21.15.169:49196) 2024-03-28T00:59:26.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.367+0000 7fb3a759e640 1 -- 172.21.15.169:0/4236564126 learned_addr learned my addr 172.21.15.169:0/4236564126 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:26.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.368+0000 7fb3a6d9d640 1 --2- 172.21.15.169:0/4236564126 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b81250 0x55ca06b8ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:26.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.368+0000 7fb3a659c640 1 --2- 172.21.15.169:0/4236564126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca06b863c0 0x55ca06b91e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:26.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.368+0000 7fb3a759e640 1 -- 172.21.15.169:0/4236564126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca06b863c0 msgr2=0x55ca06b91e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:26.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.368+0000 7fb3a759e640 1 --2- 172.21.15.169:0/4236564126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca06b863c0 0x55ca06b91e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.368+0000 7fb3a759e640 1 -- 172.21.15.169:0/4236564126 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b81250 msgr2=0x55ca06b8ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:26.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.368+0000 7fb3a759e640 1 --2- 172.21.15.169:0/4236564126 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b81250 0x55ca06b8ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.368+0000 7fb3a759e640 1 -- 172.21.15.169:0/4236564126 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ca068c47d0 con 0x55ca06b80860 2024-03-28T00:59:26.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.368+0000 7fb3a659c640 1 --2- 172.21.15.169:0/4236564126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca06b863c0 0x55ca06b91e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:59:26.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.370+0000 7fb38ffff640 1 -- 172.21.15.169:0/1655535321 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb3a80d7960 con 0x7fb3a00a4c60 2024-03-28T00:59:26.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.371+0000 7fb3a759e640 1 --2- 172.21.15.169:0/4236564126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b80860 0x55ca06b8a8c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb39c004700 tx=0x7fb39c0039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12293fc25b8d4a01 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:26.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.371+0000 7fb38aff5640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T00:59:26.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.371+0000 7fb38ffff640 1 --2- 172.21.15.169:0/1655535321 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb3900a28e0 0x7fb3900a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:26.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.372+0000 7fb38ffff640 1 -- 172.21.15.169:0/1655535321 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb3a8098bd0 con 0x7fb3a00a4c60 2024-03-28T00:59:26.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.372+0000 7fb38aff5640 1 -- 172.21.15.169:0/4236564126 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c011850 con 0x55ca06b80860 2024-03-28T00:59:26.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.373+0000 7fb3afdfb740 1 -- 172.21.15.169:0/4236564126 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb3a8093030 con 0x55ca06b80860 2024-03-28T00:59:26.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.373+0000 7fb38aff5640 1 -- 172.21.15.169:0/4236564126 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb39c0119f0 con 0x55ca06b80860 2024-03-28T00:59:26.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.374+0000 7fb38aff5640 1 -- 172.21.15.169:0/4236564126 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb39c023050 con 0x55ca06b80860 2024-03-28T00:59:26.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.374+0000 7fb3afdfb740 10 client.25949 fetch_fsmap learned FSMap version 25 2024-03-28T00:59:26.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.374+0000 7fb3afdfb740 1 -- 172.21.15.169:0/4236564126 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ca06b95220 con 0x55ca06b80860 2024-03-28T00:59:26.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.376+0000 7fb38aff5640 1 -- 172.21.15.169:0/4236564126 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3a8093030 con 0x55ca06b80860 2024-03-28T00:59:26.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.376+0000 7fb38aff5640 1 -- 172.21.15.169:0/4236564126 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb39c010440 con 0x55ca06b80860 2024-03-28T00:59:26.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.377+0000 7fb3afdfb740 10 client.25949 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:59:26.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.377+0000 7fb3afdfb740 10 client.25949 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:59:26.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.377+0000 7fb3afdfb740 20 client.25949 populate_metadata read hostname 'smithi169' 2024-03-28T00:59:26.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.377+0000 7fb3afdfb740 1 --2- 172.21.15.169:0/4236564126 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ca06b96d30 0x55ca06b99190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:26.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.377+0000 7fb3afdfb740 4 client.25949 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:59:26.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.377+0000 7fb3afdfb740 1 -- 172.21.15.169:0/4236564126 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ca06af3900 con 0x55ca06b96d30 2024-03-28T00:59:26.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.377+0000 7fb3a6d9d640 1 --2- 172.21.15.169:0/1655535321 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb3900a28e0 0x7fb3900a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:26.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.380+0000 7fb3a659c640 1 --2- 172.21.15.169:0/4236564126 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ca06b96d30 0x55ca06b99190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:26.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.389+0000 7fb3a6d9d640 1 --2- 172.21.15.169:0/1655535321 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb3900a28e0 0x7fb3900a4da0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x55ca06b81650 tx=0x7fb3a80a1cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:26.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.396+0000 7fb3a659c640 1 --2- 172.21.15.169:0/4236564126 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ca06b96d30 0x55ca06b99190 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e5232cab3c3709 server_cookie=3bac6ae1fa894bd6 in_seq=0 out_seq=0 2024-03-28T00:59:26.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.399+0000 7fb38aff5640 10 client.25949 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:26.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.399+0000 7fb38aff5640 1 -- 172.21.15.169:0/4236564126 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ca06af3900 con 0x55ca06b96d30 2024-03-28T00:59:26.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.399+0000 7fb38aff5640 10 client.25949 handle_command_reply: tid=0 2024-03-28T00:59:26.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.399+0000 7fb3afdfb740 1 client.25949 shutdown 2024-03-28T00:59:26.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.400+0000 7fb3afdfb740 1 -- 172.21.15.169:0/4236564126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b80860 msgr2=0x55ca06b8a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:26.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.400+0000 7fb3afdfb740 1 --2- 172.21.15.169:0/4236564126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b80860 0x55ca06b8a8c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb39c004700 tx=0x7fb39c0039c0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.404+0000 7fb3afdfb740 1 -- 172.21.15.169:0/4236564126 shutdown_connections 2024-03-28T00:59:26.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.404+0000 7fb3afdfb740 1 --2- 172.21.15.169:0/4236564126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca06b863c0 0x55ca06b91e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.404+0000 7fb3afdfb740 1 --2- 172.21.15.169:0/4236564126 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ca06b96d30 0x55ca06b99190 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.404+0000 7fb3afdfb740 1 --2- 172.21.15.169:0/4236564126 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b81250 0x55ca06b8ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.404+0000 7fb3afdfb740 1 --2- 172.21.15.169:0/4236564126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b80860 0x55ca06b8a8c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.404+0000 7fb3afdfb740 1 -- 172.21.15.169:0/4236564126 >> 172.21.15.169:0/4236564126 conn(0x55ca06b7bdd0 msgr2=0x55ca06b7b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:26.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.404+0000 7fb38aff5640 0 client.25949 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:26.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.407+0000 7fb3afdfb740 1 -- 172.21.15.169:0/4236564126 shutdown_connections 2024-03-28T00:59:26.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.407+0000 7fb3afdfb740 1 -- 172.21.15.169:0/4236564126 wait complete. 2024-03-28T00:59:26.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.408+0000 7fb3afdfb740 20 client.25949 trim_cache size 0 max 16384 2024-03-28T00:59:26.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.439+0000 7fb3afdfb740 1 Processor -- start 2024-03-28T00:59:26.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.442+0000 7fb3afdfb740 1 -- start start 2024-03-28T00:59:26.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.442+0000 7fb3afdfb740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b80860 0x55ca06b827b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:26.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.443+0000 7fb3afdfb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b81250 0x55ca06b82cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:26.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.443+0000 7fb3a759e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b80860 0x55ca06b827b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:26.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.443+0000 7fb3a759e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b80860 0x55ca06b827b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:37070/0 (socket says 172.21.15.169:37070) 2024-03-28T00:59:26.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.443+0000 7fb3a759e640 1 -- 172.21.15.169:0/818850056 learned_addr learned my addr 172.21.15.169:0/818850056 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:26.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.443+0000 7fb3afdfb740 1 --2- 172.21.15.169:0/818850056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca06b863c0 0x55ca06b91e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:26.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.443+0000 7fb3afdfb740 1 -- 172.21.15.169:0/818850056 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3940120e0 con 0x55ca06b80860 2024-03-28T00:59:26.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.443+0000 7fb3afdfb740 1 -- 172.21.15.169:0/818850056 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39400fbe0 con 0x55ca06b81250 2024-03-28T00:59:26.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.443+0000 7fb3afdfb740 1 -- 172.21.15.169:0/818850056 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39400dc80 con 0x55ca06b863c0 2024-03-28T00:59:26.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.444+0000 7fb3a759e640 1 -- 172.21.15.169:0/818850056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca06b863c0 msgr2=0x55ca06b91e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:26.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.444+0000 7fb3a759e640 1 --2- 172.21.15.169:0/818850056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca06b863c0 0x55ca06b91e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.444+0000 7fb3a759e640 1 -- 172.21.15.169:0/818850056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b81250 msgr2=0x55ca06b82cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T00:59:26.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.444+0000 7fb3a759e640 1 --2- 172.21.15.169:0/818850056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b81250 0x55ca06b82cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.444+0000 7fb3a759e640 1 -- 172.21.15.169:0/818850056 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca06b85a50 con 0x55ca06b80860 2024-03-28T00:59:26.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.445+0000 7fb3a759e640 1 --2- 172.21.15.169:0/818850056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b80860 0x55ca06b827b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb39c0107e0 tx=0x7fb39c004000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fee5487902617a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:26.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.447+0000 7fb38cff9640 1 -- 172.21.15.169:0/818850056 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c010e00 con 0x55ca06b80860 2024-03-28T00:59:26.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.447+0000 7fb38cff9640 1 -- 172.21.15.169:0/818850056 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb39c015070 con 0x55ca06b80860 2024-03-28T00:59:26.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.447+0000 7fb38cff9640 1 -- 172.21.15.169:0/818850056 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c004730 con 0x55ca06b80860 2024-03-28T00:59:26.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.449+0000 7fb3afdfb740 1 -- 172.21.15.169:0/818850056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b80860 msgr2=0x55ca06b827b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:26.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.449+0000 7fb3afdfb740 1 --2- 172.21.15.169:0/818850056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b80860 0x55ca06b827b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb39c0107e0 tx=0x7fb39c004000 comp rx=0 tx=0).stop 2024-03-28T00:59:26.453 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.450+0000 7fb3afdfb740 1 -- 172.21.15.169:0/818850056 shutdown_connections 2024-03-28T00:59:26.453 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.450+0000 7fb3afdfb740 1 --2- 172.21.15.169:0/818850056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca06b863c0 0x55ca06b91e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.453 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.450+0000 7fb3afdfb740 1 --2- 172.21.15.169:0/818850056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b81250 0x55ca06b82cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.453 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.450+0000 7fb3afdfb740 1 --2- 172.21.15.169:0/818850056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b80860 0x55ca06b827b0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.453 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.450+0000 7fb3afdfb740 1 -- 172.21.15.169:0/818850056 >> 172.21.15.169:0/818850056 conn(0x55ca06b7bdd0 msgr2=0x55ca06b98d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:26.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.459+0000 7fb3afdfb740 1 -- 172.21.15.169:0/818850056 shutdown_connections 2024-03-28T00:59:26.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.462+0000 7fb3afdfb740 1 -- 172.21.15.169:0/818850056 wait complete. 2024-03-28T00:59:26.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.463+0000 7fb3afdfb740 1 -- start start 2024-03-28T00:59:26.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.463+0000 7fb3afdfb740 1 Processor -- start 2024-03-28T00:59:26.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.465+0000 7fb3afdfb740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b80860 0x55ca06b8f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:26.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.465+0000 7fb3afdfb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca06b81250 0x55ca06b8f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:26.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.465+0000 7fb3afdfb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b863c0 0x55ca06b905b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:26.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.465+0000 7fb3afdfb740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39400e280 con 0x55ca06b80860 2024-03-28T00:59:26.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.465+0000 7fb3afdfb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39400f580 con 0x55ca06b863c0 2024-03-28T00:59:26.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.465+0000 7fb3afdfb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39400e8c0 con 0x55ca06b81250 2024-03-28T00:59:26.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.465+0000 7fb3afdfb740 10 client.0 _ll_register_callbacks cb 0x7fb39c004540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:59:26.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.466+0000 7fb3a759e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b80860 0x55ca06b8f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:26.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.466+0000 7fb3a759e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b80860 0x55ca06b8f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:37082/0 (socket says 172.21.15.169:37082) 2024-03-28T00:59:26.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.466+0000 7fb3a759e640 1 -- 172.21.15.169:0/223475193 learned_addr learned my addr 172.21.15.169:0/223475193 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:26.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.466+0000 7fb3a659c640 1 --2- 172.21.15.169:0/223475193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca06b81250 0x55ca06b8f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:26.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.467+0000 7fb3a659c640 1 -- 172.21.15.169:0/223475193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b863c0 msgr2=0x55ca06b905b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:26.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.467+0000 7fb3a659c640 1 --2- 172.21.15.169:0/223475193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b863c0 0x55ca06b905b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.467+0000 7fb3a659c640 1 -- 172.21.15.169:0/223475193 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b80860 msgr2=0x55ca06b8f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:26.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.467+0000 7fb3a659c640 1 --2- 172.21.15.169:0/223475193 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b80860 0x55ca06b8f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.467+0000 7fb3a659c640 1 -- 172.21.15.169:0/223475193 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ca06b90ba0 con 0x55ca06b81250 2024-03-28T00:59:26.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.468+0000 7fb3a659c640 1 --2- 172.21.15.169:0/223475193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca06b81250 0x55ca06b8f5f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x55ca06b77920 tx=0x7fb39801fc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85df94f79bde0ae8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:26.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.468+0000 7fb38aff5640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:59:26.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.470+0000 7fb38aff5640 1 -- 172.21.15.169:0/223475193 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39801f2d0 con 0x55ca06b81250 2024-03-28T00:59:26.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.470+0000 7fb38aff5640 1 -- 172.21.15.169:0/223475193 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb3980128a0 con 0x55ca06b81250 2024-03-28T00:59:26.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.471+0000 7fb3afdfb740 1 -- 172.21.15.169:0/223475193 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb39c015070 con 0x55ca06b81250 2024-03-28T00:59:26.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.473+0000 7fb38aff5640 1 -- 172.21.15.169:0/223475193 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb398012aa0 con 0x55ca06b81250 2024-03-28T00:59:26.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.474+0000 7fb3afdfb740 10 client.25915 fetch_fsmap learned FSMap version 25 2024-03-28T00:59:26.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.474+0000 7fb3afdfb740 1 -- 172.21.15.169:0/223475193 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ca06b91f70 con 0x55ca06b81250 2024-03-28T00:59:26.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.477+0000 7fb38aff5640 1 -- 172.21.15.169:0/223475193 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb39c015070 con 0x55ca06b81250 2024-03-28T00:59:26.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.477+0000 7fb38aff5640 1 -- 172.21.15.169:0/223475193 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb398004900 con 0x55ca06b81250 2024-03-28T00:59:26.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.478+0000 7fb3afdfb740 10 client.25915 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:59:26.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.478+0000 7fb3afdfb740 10 client.25915 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:59:26.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.478+0000 7fb3afdfb740 20 client.25915 populate_metadata read hostname 'smithi169' 2024-03-28T00:59:26.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.478+0000 7fb3afdfb740 1 --2- 172.21.15.169:0/223475193 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ca06bf2b40 0x55ca06bf4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:26.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.478+0000 7fb3afdfb740 4 client.25915 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T00:59:26.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.478+0000 7fb3afdfb740 1 -- 172.21.15.169:0/223475193 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ca06aedc50 con 0x55ca06bf2b40 2024-03-28T00:59:26.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.485+0000 7fb3a759e640 1 --2- 172.21.15.169:0/223475193 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ca06bf2b40 0x55ca06bf4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:26.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.491+0000 7fb38aff5640 10 client.25915 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:26.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.491+0000 7fb3a759e640 1 --2- 172.21.15.169:0/223475193 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ca06bf2b40 0x55ca06bf4f60 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db12a2e75cc2fe9f server_cookie=62530fa86d072ba9 in_seq=0 out_seq=0 2024-03-28T00:59:26.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.497+0000 7fb38aff5640 1 -- 172.21.15.169:0/223475193 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ca06aedc50 con 0x55ca06bf2b40 2024-03-28T00:59:26.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.497+0000 7fb38aff5640 10 client.25915 handle_command_reply: tid=0 2024-03-28T00:59:26.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.497+0000 7fb3afdfb740 1 client.25915 shutdown 2024-03-28T00:59:26.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.501+0000 7fb3afdfb740 1 -- 172.21.15.169:0/223475193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca06b81250 msgr2=0x55ca06b8f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:26.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.501+0000 7fb3afdfb740 1 --2- 172.21.15.169:0/223475193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca06b81250 0x55ca06b8f5f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x55ca06b77920 tx=0x7fb39801fc10 comp rx=0 tx=0).stop 2024-03-28T00:59:26.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.502+0000 7fb3afdfb740 1 -- 172.21.15.169:0/223475193 shutdown_connections 2024-03-28T00:59:26.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.502+0000 7fb3afdfb740 1 --2- 172.21.15.169:0/223475193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca06b863c0 0x55ca06b905b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.502+0000 7fb3afdfb740 1 --2- 172.21.15.169:0/223475193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca06b81250 0x55ca06b8f5f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.502+0000 7fb3afdfb740 1 --2- 172.21.15.169:0/223475193 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ca06bf2b40 0x55ca06bf4f60 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.502+0000 7fb3afdfb740 1 --2- 172.21.15.169:0/223475193 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ca06b80860 0x55ca06b8f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.502+0000 7fb3afdfb740 1 -- 172.21.15.169:0/223475193 >> 172.21.15.169:0/223475193 conn(0x55ca06b7bdd0 msgr2=0x55ca06b7f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:26.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.502+0000 7fb38aff5640 0 client.25915 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:26.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.503+0000 7fb3afdfb740 1 -- 172.21.15.169:0/223475193 shutdown_connections 2024-03-28T00:59:26.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.507+0000 7fb3afdfb740 1 -- 172.21.15.169:0/223475193 wait complete. 2024-03-28T00:59:26.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.513+0000 7fb3afdfb740 20 client.25915 trim_cache size 0 max 16384 2024-03-28T00:59:26.517 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T00:59:26.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.517+0000 7fb3ad137640 1 -- 172.21.15.169:0/1655535321 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb3900a28e0 msgr2=0x7fb3900a4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:26.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.517+0000 7fb3ad137640 1 --2- 172.21.15.169:0/1655535321 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb3900a28e0 0x7fb3900a4da0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x55ca06b81650 tx=0x7fb3a80a1cf0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.518+0000 7fb3ad137640 1 -- 172.21.15.169:0/1655535321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3a00a4c60 msgr2=0x7fb3a00b38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:26.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.518+0000 7fb3ad137640 1 --2- 172.21.15.169:0/1655535321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3a00a4c60 0x7fb3a00b38a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8064b90 tx=0x7fb3a80693d0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.518+0000 7fb3ad137640 1 -- 172.21.15.169:0/1655535321 shutdown_connections 2024-03-28T00:59:26.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.518+0000 7fb3ad137640 1 --2- 172.21.15.169:0/1655535321 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb3900a28e0 0x7fb3900a4da0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.518+0000 7fb3ad137640 1 --2- 172.21.15.169:0/1655535321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3a00a6730 0x7fb3a0153680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.518+0000 7fb3ad137640 1 --2- 172.21.15.169:0/1655535321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3a00a5d90 0x7fb3a00b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.518+0000 7fb3ad137640 1 --2- 172.21.15.169:0/1655535321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3a00a4c60 0x7fb3a00b38a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:26.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.518+0000 7fb3ad137640 1 -- 172.21.15.169:0/1655535321 >> 172.21.15.169:0/1655535321 conn(0x7fb3a00a00f0 msgr2=0x7fb3a00a9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:26.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.520+0000 7fb3ad137640 1 -- 172.21.15.169:0/1655535321 shutdown_connections 2024-03-28T00:59:26.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:26.520+0000 7fb3ad137640 1 -- 172.21.15.169:0/1655535321 wait complete. 2024-03-28T00:59:27.557 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T00:59:27.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T00:59:27.558 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T00:59:27.630 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:27 smithi169 ceph-mon[32274]: pgmap v439: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 836 MiB/s wr, 1.57k op/s 2024-03-28T00:59:27.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:27 smithi177 ceph-mon[36206]: pgmap v439: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 836 MiB/s wr, 1.57k op/s 2024-03-28T00:59:27.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:27 smithi174 ceph-mon[36060]: pgmap v439: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 836 MiB/s wr, 1.57k op/s 2024-03-28T00:59:27.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.924+0000 7f6a28748640 1 -- 172.21.15.169:0/589227535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a201097b0 msgr2=0x7f6a2010bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:27.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.924+0000 7f6a28748640 1 --2- 172.21.15.169:0/589227535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a201097b0 0x7f6a2010bba0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6a10009900 tx=0x7f6a1002bc60 comp rx=0 tx=0).stop 2024-03-28T00:59:27.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.926+0000 7f6a28748640 1 -- 172.21.15.169:0/589227535 shutdown_connections 2024-03-28T00:59:27.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.926+0000 7f6a28748640 1 --2- 172.21.15.169:0/589227535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6a2010ea60 0x7f6a20110ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:27.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.926+0000 7f6a28748640 1 --2- 172.21.15.169:0/589227535 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a2010c0e0 0x7f6a2010e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:27.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.926+0000 7f6a28748640 1 --2- 172.21.15.169:0/589227535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a201097b0 0x7f6a2010bba0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:27.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.926+0000 7f6a28748640 1 -- 172.21.15.169:0/589227535 >> 172.21.15.169:0/589227535 conn(0x7f6a2006e500 msgr2=0x7f6a20070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:27.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.927+0000 7f6a28748640 1 -- 172.21.15.169:0/589227535 shutdown_connections 2024-03-28T00:59:27.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.927+0000 7f6a28748640 1 -- 172.21.15.169:0/589227535 wait complete. 2024-03-28T00:59:27.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.927+0000 7f6a28748640 1 Processor -- start 2024-03-28T00:59:27.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.927+0000 7f6a28748640 1 -- start start 2024-03-28T00:59:27.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.928+0000 7f6a28748640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6a201097b0 0x7f6a20078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:27.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.928+0000 7f6a28748640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a2010c0e0 0x7f6a20078e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:27.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.928+0000 7f6a28748640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a2010ea60 0x7f6a2007feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:27.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.928+0000 7f6a28748640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a20113a30 con 0x7f6a201097b0 2024-03-28T00:59:27.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.928+0000 7f6a28748640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a201138b0 con 0x7f6a2010c0e0 2024-03-28T00:59:27.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.928+0000 7f6a28748640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a20113bb0 con 0x7f6a2010ea60 2024-03-28T00:59:27.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.928+0000 7f6a26ce5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a2010ea60 0x7f6a2007feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:27.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.929+0000 7f6a25ce3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a2010c0e0 0x7f6a20078e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:27.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.929+0000 7f6a25ce3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a2010c0e0 0x7f6a20078e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:39712/0 (socket says 172.21.15.169:39712) 2024-03-28T00:59:27.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.929+0000 7f6a25ce3640 1 -- 172.21.15.169:0/4023969892 learned_addr learned my addr 172.21.15.169:0/4023969892 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:27.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.928+0000 7f6a26ce5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a2010ea60 0x7f6a2007feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48982/0 (socket says 172.21.15.169:48982) 2024-03-28T00:59:27.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.929+0000 7f6a264e4640 1 --2- 172.21.15.169:0/4023969892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6a201097b0 0x7f6a20078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:27.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.930+0000 7f6a25ce3640 1 -- 172.21.15.169:0/4023969892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a2010ea60 msgr2=0x7f6a2007feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:27.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.930+0000 7f6a25ce3640 1 --2- 172.21.15.169:0/4023969892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a2010ea60 0x7f6a2007feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:27.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.930+0000 7f6a25ce3640 1 -- 172.21.15.169:0/4023969892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6a201097b0 msgr2=0x7f6a20078950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:27.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.930+0000 7f6a25ce3640 1 --2- 172.21.15.169:0/4023969892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6a201097b0 0x7f6a20078950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:27.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.930+0000 7f6a25ce3640 1 -- 172.21.15.169:0/4023969892 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a201bb170 con 0x7f6a2010c0e0 2024-03-28T00:59:27.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.930+0000 7f6a25ce3640 1 --2- 172.21.15.169:0/4023969892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a2010c0e0 0x7f6a20078e90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6a0800b9e0 tx=0x7f6a0800bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:27.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.932+0000 7f6a1f7fe640 1 -- 172.21.15.169:0/4023969892 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a08019070 con 0x7f6a2010c0e0 2024-03-28T00:59:27.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.932+0000 7f6a1f7fe640 1 -- 172.21.15.169:0/4023969892 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a0800cba0 con 0x7f6a2010c0e0 2024-03-28T00:59:27.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.933+0000 7f6a28748640 1 -- 172.21.15.169:0/4023969892 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a20080450 con 0x7f6a2010c0e0 2024-03-28T00:59:27.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.933+0000 7f6a28748640 1 -- 172.21.15.169:0/4023969892 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a20080940 con 0x7f6a2010c0e0 2024-03-28T00:59:27.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.944+0000 7f6a2b40c740 1 Processor -- start 2024-03-28T00:59:27.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.945+0000 7f6a2b40c740 1 -- start start 2024-03-28T00:59:27.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.945+0000 7f6a2b40c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556eebb6970 0x5556eebb6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:27.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.945+0000 7f6a2b40c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebb7360 0x5556eebbbe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:27.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.945+0000 7f6a2b40c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebbc4d0 0x5556eebbe960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:27.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.945+0000 7f6a2b40c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556eeae5c90 con 0x5556eebb7360 2024-03-28T00:59:27.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.945+0000 7f6a2b40c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556eeb0b070 con 0x5556eebb6970 2024-03-28T00:59:27.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.945+0000 7f6a2b40c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556eeab3c80 con 0x5556eebbc4d0 2024-03-28T00:59:27.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.947+0000 7f6a25ce3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebbc4d0 0x5556eebbe960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:27.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.947+0000 7f6a25ce3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebbc4d0 0x5556eebbe960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48988/0 (socket says 172.21.15.169:48988) 2024-03-28T00:59:27.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.947+0000 7f6a25ce3640 1 -- 172.21.15.169:0/1445610741 learned_addr learned my addr 172.21.15.169:0/1445610741 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:27.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.947+0000 7f6a1f7fe640 1 -- 172.21.15.169:0/4023969892 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a08012740 con 0x7f6a2010c0e0 2024-03-28T00:59:27.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.947+0000 7f6a25ce3640 1 -- 172.21.15.169:0/1445610741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556eebb6970 msgr2=0x5556eebb6d90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T00:59:27.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.947+0000 7f6a1f7fe640 1 -- 172.21.15.169:0/4023969892 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6a0801c070 con 0x7f6a2010c0e0 2024-03-28T00:59:27.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.947+0000 7f6a25ce3640 1 --2- 172.21.15.169:0/1445610741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556eebb6970 0x5556eebb6d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:27.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.947+0000 7f6a25ce3640 1 -- 172.21.15.169:0/1445610741 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebb7360 msgr2=0x5556eebbbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:27.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.947+0000 7f6a25ce3640 1 --2- 172.21.15.169:0/1445610741 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebb7360 0x5556eebbbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:27.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.947+0000 7f6a25ce3640 1 -- 172.21.15.169:0/1445610741 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556eeaabd10 con 0x5556eebbc4d0 2024-03-28T00:59:27.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.948+0000 7f6a25ce3640 1 --2- 172.21.15.169:0/1445610741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebbc4d0 0x5556eebbe960 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6a080d62e0 tx=0x7f6a080d63d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e2744efefa2769c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:27.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.948+0000 7f6a1cff9640 1 -- 172.21.15.169:0/1445610741 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a080e1070 con 0x5556eebbc4d0 2024-03-28T00:59:27.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.948+0000 7f6a1cff9640 1 -- 172.21.15.169:0/1445610741 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a080d05b0 con 0x5556eebbc4d0 2024-03-28T00:59:27.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.949+0000 7f6a1f7fe640 1 --2- 172.21.15.169:0/4023969892 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6a0c0a2890 0x7f6a0c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:27.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.949+0000 7f6a1f7fe640 1 -- 172.21.15.169:0/4023969892 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f6a08014520 con 0x7f6a2010c0e0 2024-03-28T00:59:27.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.949+0000 7f6a26ce5640 1 --2- 172.21.15.169:0/4023969892 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6a0c0a2890 0x7f6a0c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:27.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.950+0000 7f6a2b40c740 1 -- 172.21.15.169:0/1445610741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebbc4d0 msgr2=0x5556eebbe960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:27.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.950+0000 7f6a2b40c740 1 --2- 172.21.15.169:0/1445610741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebbc4d0 0x5556eebbe960 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6a080d62e0 tx=0x7f6a080d63d0 comp rx=0 tx=0).stop 2024-03-28T00:59:27.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.950+0000 7f6a2b40c740 1 -- 172.21.15.169:0/1445610741 shutdown_connections 2024-03-28T00:59:27.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.950+0000 7f6a2b40c740 1 --2- 172.21.15.169:0/1445610741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebbc4d0 0x5556eebbe960 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:27.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.950+0000 7f6a2b40c740 1 --2- 172.21.15.169:0/1445610741 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebb7360 0x5556eebbbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:27.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.950+0000 7f6a2b40c740 1 --2- 172.21.15.169:0/1445610741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556eebb6970 0x5556eebb6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:27.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.950+0000 7f6a2b40c740 1 -- 172.21.15.169:0/1445610741 >> 172.21.15.169:0/1445610741 conn(0x5556eeb6e1a0 msgr2=0x5556eeba12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:27.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.953+0000 7f6a2b40c740 1 -- 172.21.15.169:0/1445610741 shutdown_connections 2024-03-28T00:59:27.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.965+0000 7f6a2b40c740 1 -- 172.21.15.169:0/1445610741 wait complete. 2024-03-28T00:59:27.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.966+0000 7f6a2b40c740 1 -- start start 2024-03-28T00:59:27.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.966+0000 7f6a26ce5640 1 --2- 172.21.15.169:0/4023969892 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6a0c0a2890 0x7f6a0c0a4d50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x5556eebb7bc0 tx=0x7f6a180090e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:27.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.969+0000 7f6a2b40c740 1 Processor -- start 2024-03-28T00:59:27.976 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.973+0000 7f6a2b40c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebb6970 0x5556eebc0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:27.976 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.973+0000 7f6a2b40c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556eebb7360 0x5556eebc11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:27.976 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.973+0000 7f6a2b40c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebbc4d0 0x5556eebc81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:27.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.973+0000 7f6a2b40c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556eeac70d0 con 0x5556eebb6970 2024-03-28T00:59:27.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.973+0000 7f6a2b40c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556eeb07110 con 0x5556eebb7360 2024-03-28T00:59:27.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.973+0000 7f6a2b40c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556eebbf520 con 0x5556eebbc4d0 2024-03-28T00:59:27.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.973+0000 7f6a2b40c740 10 client.0 _ll_register_callbacks cb 0x5556eeb02e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:59:27.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.975+0000 7f6a25ce3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556eebb7360 0x5556eebc11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:27.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.975+0000 7f6a264e4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebb6970 0x5556eebc0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:27.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.975+0000 7f6a264e4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebb6970 0x5556eebc0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:37118/0 (socket says 172.21.15.169:37118) 2024-03-28T00:59:27.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.975+0000 7f6a264e4640 1 -- 172.21.15.169:0/404314913 learned_addr learned my addr 172.21.15.169:0/404314913 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:27.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.975+0000 7f6a26ce5640 1 --2- 172.21.15.169:0/404314913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebbc4d0 0x5556eebc81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:27.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.978+0000 7f6a26ce5640 1 -- 172.21.15.169:0/404314913 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556eebb7360 msgr2=0x5556eebc11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:27.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.978+0000 7f6a26ce5640 1 --2- 172.21.15.169:0/404314913 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556eebb7360 0x5556eebc11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:27.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.978+0000 7f6a26ce5640 1 -- 172.21.15.169:0/404314913 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebb6970 msgr2=0x5556eebc0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:27.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.978+0000 7f6a26ce5640 1 --2- 172.21.15.169:0/404314913 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebb6970 0x5556eebc0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:27.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.978+0000 7f6a26ce5640 1 -- 172.21.15.169:0/404314913 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5556ee8e77e0 con 0x5556eebbc4d0 2024-03-28T00:59:27.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.980+0000 7f6a26ce5640 1 --2- 172.21.15.169:0/404314913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebbc4d0 0x5556eebc81f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6a18015700 tx=0x7f6a180047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5a90e9685580288 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:27.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.982+0000 7f6a15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:59:27.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.982+0000 7f6a15ffb640 1 -- 172.21.15.169:0/404314913 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a18023020 con 0x5556eebbc4d0 2024-03-28T00:59:27.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.982+0000 7f6a15ffb640 1 -- 172.21.15.169:0/404314913 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f6a18013510 con 0x5556eebbc4d0 2024-03-28T00:59:27.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.982+0000 7f6a2b40c740 1 -- 172.21.15.169:0/404314913 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5556eeb185b0 con 0x5556eebbc4d0 2024-03-28T00:59:27.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.986+0000 7f6a15ffb640 1 -- 172.21.15.169:0/404314913 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a1801b040 con 0x5556eebbc4d0 2024-03-28T00:59:27.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.987+0000 7f6a2b40c740 10 client.25927 fetch_fsmap learned FSMap version 25 2024-03-28T00:59:27.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.987+0000 7f6a2b40c740 1 -- 172.21.15.169:0/404314913 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5556ee947f70 con 0x5556eebbc4d0 2024-03-28T00:59:27.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.991+0000 7f6a15ffb640 1 -- 172.21.15.169:0/404314913 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5556eeb185b0 con 0x5556eebbc4d0 2024-03-28T00:59:27.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.991+0000 7f6a15ffb640 1 -- 172.21.15.169:0/404314913 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f6a18030020 con 0x5556eebbc4d0 2024-03-28T00:59:27.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.991+0000 7f6a2b40c740 10 client.25927 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:59:27.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.991+0000 7f6a2b40c740 10 client.25927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:59:27.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.991+0000 7f6a2b40c740 20 client.25927 populate_metadata read hostname 'smithi169' 2024-03-28T00:59:27.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.991+0000 7f6a2b40c740 1 --2- 172.21.15.169:0/404314913 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5556eebccfd0 0x5556eebcf450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:27.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.991+0000 7f6a2b40c740 4 client.25927 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:59:27.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.991+0000 7f6a2b40c740 1 -- 172.21.15.169:0/404314913 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5556eeb12900 con 0x5556eebccfd0 2024-03-28T00:59:28.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:27.999+0000 7f6a264e4640 1 --2- 172.21.15.169:0/404314913 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5556eebccfd0 0x5556eebcf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:28.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.017+0000 7f6a264e4640 1 --2- 172.21.15.169:0/404314913 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5556eebccfd0 0x5556eebcf450 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a370e47a838f6fc0 server_cookie=d41bb8c6ff94a895 in_seq=0 out_seq=0 2024-03-28T00:59:28.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.019+0000 7f6a15ffb640 10 client.25927 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:28.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.033+0000 7f6a15ffb640 1 -- 172.21.15.169:0/404314913 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5556eeb12900 con 0x5556eebccfd0 2024-03-28T00:59:28.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.033+0000 7f6a15ffb640 10 client.25927 handle_command_reply: tid=0 2024-03-28T00:59:28.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.033+0000 7f6a2b40c740 1 client.25927 shutdown 2024-03-28T00:59:28.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.039+0000 7f6a2b40c740 1 -- 172.21.15.169:0/404314913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebbc4d0 msgr2=0x5556eebc81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:28.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.039+0000 7f6a2b40c740 1 --2- 172.21.15.169:0/404314913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebbc4d0 0x5556eebc81f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6a18015700 tx=0x7f6a180047d0 comp rx=0 tx=0).stop 2024-03-28T00:59:28.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.041+0000 7f6a2b40c740 1 -- 172.21.15.169:0/404314913 shutdown_connections 2024-03-28T00:59:28.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.041+0000 7f6a2b40c740 1 --2- 172.21.15.169:0/404314913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebbc4d0 0x5556eebc81f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:28.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.041+0000 7f6a2b40c740 1 --2- 172.21.15.169:0/404314913 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556eebb7360 0x5556eebc11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:28.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.041+0000 7f6a2b40c740 1 --2- 172.21.15.169:0/404314913 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5556eebccfd0 0x5556eebcf450 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:28.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.041+0000 7f6a2b40c740 1 --2- 172.21.15.169:0/404314913 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebb6970 0x5556eebc0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:28.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.041+0000 7f6a2b40c740 1 -- 172.21.15.169:0/404314913 >> 172.21.15.169:0/404314913 conn(0x5556eeb6e1a0 msgr2=0x5556eeba0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:28.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.041+0000 7f6a15ffb640 0 client.25927 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:28.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.044+0000 7f6a2b40c740 1 -- 172.21.15.169:0/404314913 shutdown_connections 2024-03-28T00:59:28.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.047+0000 7f6a2b40c740 1 -- 172.21.15.169:0/404314913 wait complete. 2024-03-28T00:59:28.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.047+0000 7f6a2b40c740 20 client.25927 trim_cache size 0 max 16384 2024-03-28T00:59:28.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.100+0000 7f6a2b40c740 1 Processor -- start 2024-03-28T00:59:28.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.101+0000 7f6a2b40c740 1 -- start start 2024-03-28T00:59:28.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.101+0000 7f6a2b40c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556eebb6970 0x5556eeba1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:28.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.101+0000 7f6a2b40c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebb7360 0x5556eebc81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:28.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.101+0000 7f6a2b40c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebbc4d0 0x5556eeba4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:28.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.101+0000 7f6a2b40c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a201c3e20 con 0x5556eebbc4d0 2024-03-28T00:59:28.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.101+0000 7f6a2b40c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a201c1920 con 0x5556eebb6970 2024-03-28T00:59:28.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.101+0000 7f6a2b40c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a201bf9c0 con 0x5556eebb7360 2024-03-28T00:59:28.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.103+0000 7f6a264e4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556eebb6970 0x5556eeba1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:28.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.103+0000 7f6a264e4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556eebb6970 0x5556eeba1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:39744/0 (socket says 172.21.15.169:39744) 2024-03-28T00:59:28.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.103+0000 7f6a264e4640 1 -- 172.21.15.169:0/2754690320 learned_addr learned my addr 172.21.15.169:0/2754690320 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:28.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.104+0000 7f6a26ce5640 1 --2- 172.21.15.169:0/2754690320 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebbc4d0 0x5556eeba4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:28.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.106+0000 7f6a25ce3640 1 --2- 172.21.15.169:0/2754690320 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebb7360 0x5556eebc81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:28.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.106+0000 7f6a264e4640 1 -- 172.21.15.169:0/2754690320 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebb7360 msgr2=0x5556eebc81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:28.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.106+0000 7f6a264e4640 1 --2- 172.21.15.169:0/2754690320 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebb7360 0x5556eebc81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:28.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.106+0000 7f6a264e4640 1 -- 172.21.15.169:0/2754690320 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebbc4d0 msgr2=0x5556eeba4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:28.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.106+0000 7f6a264e4640 1 --2- 172.21.15.169:0/2754690320 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebbc4d0 0x5556eeba4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:28.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.106+0000 7f6a264e4640 1 -- 172.21.15.169:0/2754690320 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556ee7df170 con 0x5556eebb6970 2024-03-28T00:59:28.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.109+0000 7f6a264e4640 1 --2- 172.21.15.169:0/2754690320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556eebb6970 0x5556eeba1500 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6a1002e010 tx=0x7f6a1002e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a786ae715834f8b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:28.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.110+0000 7f6a17fff640 1 -- 172.21.15.169:0/2754690320 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1002c9b0 con 0x5556eebb6970 2024-03-28T00:59:28.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.110+0000 7f6a17fff640 1 -- 172.21.15.169:0/2754690320 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a1004b070 con 0x5556eebb6970 2024-03-28T00:59:28.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.110+0000 7f6a17fff640 1 -- 172.21.15.169:0/2754690320 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a10004be0 con 0x5556eebb6970 2024-03-28T00:59:28.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.111+0000 7f6a2b40c740 1 -- 172.21.15.169:0/2754690320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556eebb6970 msgr2=0x5556eeba1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:28.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.111+0000 7f6a2b40c740 1 --2- 172.21.15.169:0/2754690320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556eebb6970 0x5556eeba1500 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6a1002e010 tx=0x7f6a1002e040 comp rx=0 tx=0).stop 2024-03-28T00:59:28.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.112+0000 7f6a2b40c740 1 -- 172.21.15.169:0/2754690320 shutdown_connections 2024-03-28T00:59:28.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.112+0000 7f6a2b40c740 1 --2- 172.21.15.169:0/2754690320 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebbc4d0 0x5556eeba4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:28.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.112+0000 7f6a2b40c740 1 --2- 172.21.15.169:0/2754690320 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebb7360 0x5556eebc81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:28.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.112+0000 7f6a2b40c740 1 --2- 172.21.15.169:0/2754690320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556eebb6970 0x5556eeba1500 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:28.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.112+0000 7f6a2b40c740 1 -- 172.21.15.169:0/2754690320 >> 172.21.15.169:0/2754690320 conn(0x5556eeb6e1a0 msgr2=0x5556eeba0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:28.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.114+0000 7f6a2b40c740 1 -- 172.21.15.169:0/2754690320 shutdown_connections 2024-03-28T00:59:28.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.114+0000 7f6a2b40c740 1 -- 172.21.15.169:0/2754690320 wait complete. 2024-03-28T00:59:28.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.115+0000 7f6a2b40c740 1 -- start start 2024-03-28T00:59:28.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.115+0000 7f6a2b40c740 1 Processor -- start 2024-03-28T00:59:28.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.115+0000 7f6a2b40c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebb6970 0x5556eebc60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:28.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.115+0000 7f6a2b40c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556eebb7360 0x5556eebc65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:28.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.115+0000 7f6a2b40c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebbc4d0 0x5556eebc7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:28.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.116+0000 7f6a2b40c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a201bffc0 con 0x5556eebbc4d0 2024-03-28T00:59:28.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.116+0000 7f6a2b40c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a201c12c0 con 0x5556eebb7360 2024-03-28T00:59:28.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.116+0000 7f6a2b40c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a201c0600 con 0x5556eebb6970 2024-03-28T00:59:28.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.116+0000 7f6a2b40c740 10 client.0 _ll_register_callbacks cb 0x7f6a18009eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:59:28.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.116+0000 7f6a26ce5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebbc4d0 0x5556eebc7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:28.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.116+0000 7f6a26ce5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebbc4d0 0x5556eebc7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:37138/0 (socket says 172.21.15.169:37138) 2024-03-28T00:59:28.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.116+0000 7f6a26ce5640 1 -- 172.21.15.169:0/3273459195 learned_addr learned my addr 172.21.15.169:0/3273459195 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:28.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.117+0000 7f6a25ce3640 1 --2- 172.21.15.169:0/3273459195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556eebb7360 0x5556eebc65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:28.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.117+0000 7f6a264e4640 1 --2- 172.21.15.169:0/3273459195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebb6970 0x5556eebc60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:28.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.118+0000 7f6a26ce5640 1 -- 172.21.15.169:0/3273459195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebb6970 msgr2=0x5556eebc60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:28.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.118+0000 7f6a26ce5640 1 --2- 172.21.15.169:0/3273459195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebb6970 0x5556eebc60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:28.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.118+0000 7f6a26ce5640 1 -- 172.21.15.169:0/3273459195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556eebb7360 msgr2=0x5556eebc65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:28.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.118+0000 7f6a26ce5640 1 --2- 172.21.15.169:0/3273459195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556eebb7360 0x5556eebc65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:28.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.118+0000 7f6a26ce5640 1 -- 172.21.15.169:0/3273459195 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5556eebc7bf0 con 0x5556eebbc4d0 2024-03-28T00:59:28.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.123+0000 7f6a26ce5640 1 --2- 172.21.15.169:0/3273459195 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebbc4d0 0x5556eebc7600 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6a1800f010 tx=0x7f6a18028620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47e8af505af9ed7b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:28.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.123+0000 7f6a15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:59:28.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.124+0000 7f6a15ffb640 1 -- 172.21.15.169:0/3273459195 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a18005bc0 con 0x5556eebbc4d0 2024-03-28T00:59:28.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.125+0000 7f6a2b40c740 1 -- 172.21.15.169:0/3273459195 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6a1004b070 con 0x5556eebbc4d0 2024-03-28T00:59:28.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.125+0000 7f6a15ffb640 1 -- 172.21.15.169:0/3273459195 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f6a18005d60 con 0x5556eebbc4d0 2024-03-28T00:59:28.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.129+0000 7f6a15ffb640 1 -- 172.21.15.169:0/3273459195 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a18041050 con 0x5556eebbc4d0 2024-03-28T00:59:28.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.129+0000 7f6a2b40c740 10 client.16131 fetch_fsmap learned FSMap version 25 2024-03-28T00:59:28.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.129+0000 7f6a2b40c740 1 -- 172.21.15.169:0/3273459195 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5556eec29160 con 0x5556eebbc4d0 2024-03-28T00:59:28.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.130+0000 7f6a15ffb640 1 -- 172.21.15.169:0/3273459195 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a1004b070 con 0x5556eebbc4d0 2024-03-28T00:59:28.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.130+0000 7f6a15ffb640 1 -- 172.21.15.169:0/3273459195 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f6a18030070 con 0x5556eebbc4d0 2024-03-28T00:59:28.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.130+0000 7f6a2b40c740 10 client.16131 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:59:28.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.130+0000 7f6a2b40c740 10 client.16131 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:59:28.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.130+0000 7f6a2b40c740 20 client.16131 populate_metadata read hostname 'smithi169' 2024-03-28T00:59:28.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.130+0000 7f6a2b40c740 1 --2- 172.21.15.169:0/3273459195 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5556eec29ae0 0x5556eec2bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:28.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.130+0000 7f6a2b40c740 4 client.16131 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T00:59:28.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.130+0000 7f6a2b40c740 1 -- 172.21.15.169:0/3273459195 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5556eea90b80 con 0x5556eec29ae0 2024-03-28T00:59:28.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.138+0000 7f6a264e4640 1 --2- 172.21.15.169:0/3273459195 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5556eec29ae0 0x5556eec2bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:28.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.145+0000 7f6a264e4640 1 --2- 172.21.15.169:0/3273459195 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5556eec29ae0 0x5556eec2bf00 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c92af1892023041 server_cookie=25c1dd941e6d72cf in_seq=0 out_seq=0 2024-03-28T00:59:28.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:28.145+0000 7f6a15ffb640 10 client.16131 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:29.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.292+0000 7f6a15ffb640 1 -- 172.21.15.169:0/3273459195 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5556eea90b80 con 0x5556eec29ae0 2024-03-28T00:59:29.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.292+0000 7f6a15ffb640 10 client.16131 handle_command_reply: tid=0 2024-03-28T00:59:29.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.292+0000 7f6a2b40c740 1 client.16131 shutdown 2024-03-28T00:59:29.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.293+0000 7f6a2b40c740 1 -- 172.21.15.169:0/3273459195 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebbc4d0 msgr2=0x5556eebc7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:29.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.293+0000 7f6a2b40c740 1 --2- 172.21.15.169:0/3273459195 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebbc4d0 0x5556eebc7600 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6a1800f010 tx=0x7f6a18028620 comp rx=0 tx=0).stop 2024-03-28T00:59:29.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.294+0000 7f6a2b40c740 1 -- 172.21.15.169:0/3273459195 shutdown_connections 2024-03-28T00:59:29.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.294+0000 7f6a2b40c740 1 --2- 172.21.15.169:0/3273459195 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5556eec29ae0 0x5556eec2bf00 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.294+0000 7f6a2b40c740 1 --2- 172.21.15.169:0/3273459195 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556eebbc4d0 0x5556eebc7600 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.294+0000 7f6a2b40c740 1 --2- 172.21.15.169:0/3273459195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556eebb7360 0x5556eebc65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.294+0000 7f6a2b40c740 1 --2- 172.21.15.169:0/3273459195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556eebb6970 0x5556eebc60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.294+0000 7f6a2b40c740 1 -- 172.21.15.169:0/3273459195 >> 172.21.15.169:0/3273459195 conn(0x5556eeb6e1a0 msgr2=0x5556eeba4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:29.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.294+0000 7f6a15ffb640 0 client.16131 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:29.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.294+0000 7f6a2b40c740 1 -- 172.21.15.169:0/3273459195 shutdown_connections 2024-03-28T00:59:29.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.294+0000 7f6a2b40c740 1 -- 172.21.15.169:0/3273459195 wait complete. 2024-03-28T00:59:29.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.294+0000 7f6a2b40c740 20 client.16131 trim_cache size 0 max 16384 2024-03-28T00:59:29.298 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:59:29.298 INFO:teuthology.orchestra.run.smithi169.stdout: "return_code": 0, 2024-03-28T00:59:29.298 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_tag": "b887080d-c0f2-4257-975e-e4f3f2016cf3", 2024-03-28T00:59:29.298 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": "asynchronous" 2024-03-28T00:59:29.298 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:59:29.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.300+0000 7f6a28748640 1 -- 172.21.15.169:0/4023969892 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6a0c0a2890 msgr2=0x7f6a0c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:29.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.300+0000 7f6a28748640 1 --2- 172.21.15.169:0/4023969892 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6a0c0a2890 0x7f6a0c0a4d50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x5556eebb7bc0 tx=0x7f6a180090e0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.300+0000 7f6a28748640 1 -- 172.21.15.169:0/4023969892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a2010c0e0 msgr2=0x7f6a20078e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:29.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.300+0000 7f6a28748640 1 --2- 172.21.15.169:0/4023969892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a2010c0e0 0x7f6a20078e90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6a0800b9e0 tx=0x7f6a0800bea0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.300+0000 7f6a28748640 1 -- 172.21.15.169:0/4023969892 shutdown_connections 2024-03-28T00:59:29.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.300+0000 7f6a28748640 1 --2- 172.21.15.169:0/4023969892 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6a0c0a2890 0x7f6a0c0a4d50 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.300+0000 7f6a28748640 1 --2- 172.21.15.169:0/4023969892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a2010ea60 0x7f6a2007feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.300+0000 7f6a28748640 1 --2- 172.21.15.169:0/4023969892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a2010c0e0 0x7f6a20078e90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.300+0000 7f6a28748640 1 --2- 172.21.15.169:0/4023969892 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6a201097b0 0x7f6a20078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.300+0000 7f6a28748640 1 -- 172.21.15.169:0/4023969892 >> 172.21.15.169:0/4023969892 conn(0x7f6a2006e500 msgr2=0x7f6a2010ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:29.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.301+0000 7f6a28748640 1 -- 172.21.15.169:0/4023969892 shutdown_connections 2024-03-28T00:59:29.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.301+0000 7f6a28748640 1 -- 172.21.15.169:0/4023969892 wait complete. 2024-03-28T00:59:29.339 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T00:59:29.339 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T00:59:29.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:29 smithi177 ceph-mon[36206]: pgmap v440: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 716 MiB/s wr, 1.23k op/s 2024-03-28T00:59:29.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.728+0000 7fc35b59e640 1 -- 172.21.15.169:0/392312694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc35c072ab0 msgr2=0x7fc35c072ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:29.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.728+0000 7fc35b59e640 1 --2- 172.21.15.169:0/392312694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc35c072ab0 0x7fc35c072ed0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc354009be0 tx=0x7fc35402bef0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.738 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.736+0000 7fc35b59e640 1 -- 172.21.15.169:0/392312694 shutdown_connections 2024-03-28T00:59:29.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.736+0000 7fc35b59e640 1 --2- 172.21.15.169:0/392312694 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc35c109730 0x7fc35c10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.736+0000 7fc35b59e640 1 --2- 172.21.15.169:0/392312694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc35c108d70 0x7fc35c1091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.736+0000 7fc35b59e640 1 --2- 172.21.15.169:0/392312694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc35c072ab0 0x7fc35c072ed0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.736+0000 7fc35b59e640 1 -- 172.21.15.169:0/392312694 >> 172.21.15.169:0/392312694 conn(0x7fc35c06e580 msgr2=0x7fc35c0709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:29.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.738+0000 7fc35b59e640 1 -- 172.21.15.169:0/392312694 shutdown_connections 2024-03-28T00:59:29.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.741+0000 7fc35b59e640 1 -- 172.21.15.169:0/392312694 wait complete. 2024-03-28T00:59:29.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.743+0000 7fc35b59e640 1 Processor -- start 2024-03-28T00:59:29.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.744+0000 7fc35b59e640 1 -- start start 2024-03-28T00:59:29.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.744+0000 7fc35b59e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc35c072ab0 0x7fc35c1a65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:29.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.744+0000 7fc35b59e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc35c108d70 0x7fc35c1a6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:29.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.744+0000 7fc35b59e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc35c109730 0x7fc35c1adba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:29.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.744+0000 7fc35b59e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c11c080 con 0x7fc35c109730 2024-03-28T00:59:29.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.744+0000 7fc35b59e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c11bf00 con 0x7fc35c072ab0 2024-03-28T00:59:29.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.744+0000 7fc35b59e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c11c200 con 0x7fc35c108d70 2024-03-28T00:59:29.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.745+0000 7fc359d9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc35c108d70 0x7fc35c1a6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:29.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.745+0000 7fc359d9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc35c108d70 0x7fc35c1a6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:49036/0 (socket says 172.21.15.169:49036) 2024-03-28T00:59:29.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.745+0000 7fc359d9b640 1 -- 172.21.15.169:0/1398738692 learned_addr learned my addr 172.21.15.169:0/1398738692 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:29.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.745+0000 7fc359d9b640 1 -- 172.21.15.169:0/1398738692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc35c072ab0 msgr2=0x7fc35c1a65f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T00:59:29.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.745+0000 7fc359d9b640 1 --2- 172.21.15.169:0/1398738692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc35c072ab0 0x7fc35c1a65f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.745+0000 7fc359d9b640 1 -- 172.21.15.169:0/1398738692 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc35c109730 msgr2=0x7fc35c1adba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:29.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.745+0000 7fc359d9b640 1 --2- 172.21.15.169:0/1398738692 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc35c109730 0x7fc35c1adba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.745+0000 7fc359d9b640 1 -- 172.21.15.169:0/1398738692 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c10ff00 con 0x7fc35c108d70 2024-03-28T00:59:29.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.746+0000 7fc359d9b640 1 --2- 172.21.15.169:0/1398738692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc35c108d70 0x7fc35c1a6b30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc35000e9c0 tx=0x7fc35000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:29.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.746+0000 7fc34b7fe640 1 -- 172.21.15.169:0/1398738692 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35000ccc0 con 0x7fc35c108d70 2024-03-28T00:59:29.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.746+0000 7fc34b7fe640 1 -- 172.21.15.169:0/1398738692 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc350005500 con 0x7fc35c108d70 2024-03-28T00:59:29.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.746+0000 7fc34b7fe640 1 -- 172.21.15.169:0/1398738692 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc350010430 con 0x7fc35c108d70 2024-03-28T00:59:29.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.747+0000 7fc35b59e640 1 -- 172.21.15.169:0/1398738692 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc35c1101f0 con 0x7fc35c108d70 2024-03-28T00:59:29.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.747+0000 7fc35b59e640 1 -- 172.21.15.169:0/1398738692 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc35c1106f0 con 0x7fc35c108d70 2024-03-28T00:59:29.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.755+0000 7fc363bee740 1 Processor -- start 2024-03-28T00:59:29.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.755+0000 7fc34b7fe640 1 -- 172.21.15.169:0/1398738692 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc350005020 con 0x7fc35c108d70 2024-03-28T00:59:29.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.757+0000 7fc363bee740 1 -- start start 2024-03-28T00:59:29.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.762+0000 7fc363bee740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d49f80 0x563316d4a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:29.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.763+0000 7fc363bee740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d4a970 0x563316d4f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:29.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.763+0000 7fc363bee740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563316d4fae0 0x563316d51f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:29.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.763+0000 7fc363bee740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x563316caf3d0 con 0x563316d4fae0 2024-03-28T00:59:29.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.763+0000 7fc363bee740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x563316c57ad0 con 0x563316d49f80 2024-03-28T00:59:29.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.763+0000 7fc363bee740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x563316cab470 con 0x563316d4a970 2024-03-28T00:59:29.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.763+0000 7fc35ad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d49f80 0x563316d4a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:29.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.763+0000 7fc35ad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d49f80 0x563316d4a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:39776/0 (socket says 172.21.15.169:39776) 2024-03-28T00:59:29.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.763+0000 7fc35ad9d640 1 -- 172.21.15.169:0/3283041506 learned_addr learned my addr 172.21.15.169:0/3283041506 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:29.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.764+0000 7fc34b7fe640 1 --2- 172.21.15.169:0/1398738692 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc3440a2a10 0x7fc3440a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:29.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.764+0000 7fc35a59c640 1 --2- 172.21.15.169:0/3283041506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d4a970 0x563316d4f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:29.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.764+0000 7fc34b7fe640 1 -- 172.21.15.169:0/1398738692 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc3500cf0b0 con 0x7fc35c108d70 2024-03-28T00:59:29.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.764+0000 7fc35ad9d640 1 -- 172.21.15.169:0/3283041506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d4a970 msgr2=0x563316d4f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:29.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.764+0000 7fc35ad9d640 1 --2- 172.21.15.169:0/3283041506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d4a970 0x563316d4f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.764+0000 7fc35ad9d640 1 -- 172.21.15.169:0/3283041506 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563316d4fae0 msgr2=0x563316d51f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T00:59:29.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.764+0000 7fc35ad9d640 1 --2- 172.21.15.169:0/3283041506 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563316d4fae0 0x563316d51f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.764+0000 7fc35ad9d640 1 -- 172.21.15.169:0/3283041506 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563316a8d7d0 con 0x563316d49f80 2024-03-28T00:59:29.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.764+0000 7fc35ad9d640 1 --2- 172.21.15.169:0/1398738692 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc3440a2a10 0x7fc3440a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:29.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.765+0000 7fc35ad9d640 1 --2- 172.21.15.169:0/3283041506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d49f80 0x563316d4a3a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc34c00bbb0 tx=0x7fc34c00f6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=907bbea31654e3dd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:29.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.766+0000 7fc348ff9640 1 -- 172.21.15.169:0/3283041506 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34c018070 con 0x563316d49f80 2024-03-28T00:59:29.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.766+0000 7fc348ff9640 1 -- 172.21.15.169:0/3283041506 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc34c012d60 con 0x563316d49f80 2024-03-28T00:59:29.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.767+0000 7fc363bee740 1 -- 172.21.15.169:0/3283041506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d49f80 msgr2=0x563316d4a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:29.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.767+0000 7fc363bee740 1 --2- 172.21.15.169:0/3283041506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d49f80 0x563316d4a3a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc34c00bbb0 tx=0x7fc34c00f6d0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.768+0000 7fc35ad9d640 1 --2- 172.21.15.169:0/1398738692 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc3440a2a10 0x7fc3440a4ed0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc34c01e000 tx=0x7fc34c011b70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:29.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.768+0000 7fc363bee740 1 -- 172.21.15.169:0/3283041506 shutdown_connections 2024-03-28T00:59:29.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.768+0000 7fc363bee740 1 --2- 172.21.15.169:0/3283041506 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563316d4fae0 0x563316d51f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.768+0000 7fc363bee740 1 --2- 172.21.15.169:0/3283041506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d4a970 0x563316d4f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.768+0000 7fc363bee740 1 --2- 172.21.15.169:0/3283041506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d49f80 0x563316d4a3a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.768+0000 7fc363bee740 1 -- 172.21.15.169:0/3283041506 >> 172.21.15.169:0/3283041506 conn(0x563316d454f0 msgr2=0x563316d458f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:29.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.768+0000 7fc363bee740 1 -- 172.21.15.169:0/3283041506 shutdown_connections 2024-03-28T00:59:29.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.770+0000 7fc363bee740 1 -- 172.21.15.169:0/3283041506 wait complete. 2024-03-28T00:59:29.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.770+0000 7fc363bee740 1 -- start start 2024-03-28T00:59:29.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.770+0000 7fc363bee740 1 Processor -- start 2024-03-28T00:59:29.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.772+0000 7fc363bee740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d49f80 0x563316d546b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:29.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.772+0000 7fc363bee740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563316d4a970 0x563316d54bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:29.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.772+0000 7fc363bee740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d4fae0 0x563316d5bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:29.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.772+0000 7fc363bee740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x563316d52ae0 con 0x563316d4a970 2024-03-28T00:59:29.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.772+0000 7fc363bee740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x563316c6b060 con 0x563316d4fae0 2024-03-28T00:59:29.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.772+0000 7fc363bee740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x563316d52c60 con 0x563316d49f80 2024-03-28T00:59:29.776 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.772+0000 7fc363bee740 10 client.0 _ll_register_callbacks cb 0x563316c4d6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:59:29.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.773+0000 7fc35a59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d49f80 0x563316d546b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:29.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.775+0000 7fc359d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563316d4a970 0x563316d54bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:29.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.775+0000 7fc359d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563316d4a970 0x563316d54bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:37152/0 (socket says 172.21.15.169:37152) 2024-03-28T00:59:29.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.775+0000 7fc359d9b640 1 -- 172.21.15.169:0/831430904 learned_addr learned my addr 172.21.15.169:0/831430904 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:29.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.775+0000 7fc35ad9d640 1 --2- 172.21.15.169:0/831430904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d4fae0 0x563316d5bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:29.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.776+0000 7fc35a59c640 1 -- 172.21.15.169:0/831430904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d4fae0 msgr2=0x563316d5bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:29.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.776+0000 7fc35a59c640 1 --2- 172.21.15.169:0/831430904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d4fae0 0x563316d5bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.776+0000 7fc35a59c640 1 -- 172.21.15.169:0/831430904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563316d4a970 msgr2=0x563316d54bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:29.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.776+0000 7fc35a59c640 1 --2- 172.21.15.169:0/831430904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563316d4a970 0x563316d54bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.776+0000 7fc35a59c640 1 -- 172.21.15.169:0/831430904 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5633169c33f0 con 0x563316d49f80 2024-03-28T00:59:29.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.776+0000 7fc35a59c640 1 --2- 172.21.15.169:0/831430904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d49f80 0x563316d546b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x563316d4ad70 tx=0x7fc354033d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3e29a92349d0fae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:29.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.776+0000 7fc341ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:59:29.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.778+0000 7fc341ffb640 1 -- 172.21.15.169:0/831430904 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354042070 con 0x563316d49f80 2024-03-28T00:59:29.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.778+0000 7fc341ffb640 1 -- 172.21.15.169:0/831430904 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc354003d80 con 0x563316d49f80 2024-03-28T00:59:29.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.778+0000 7fc363bee740 1 -- 172.21.15.169:0/831430904 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563316cbc910 con 0x563316d49f80 2024-03-28T00:59:29.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.779+0000 7fc341ffb640 1 -- 172.21.15.169:0/831430904 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc35404c050 con 0x563316d49f80 2024-03-28T00:59:29.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.779+0000 7fc363bee740 10 client.25939 fetch_fsmap learned FSMap version 25 2024-03-28T00:59:29.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.779+0000 7fc363bee740 1 -- 172.21.15.169:0/831430904 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5633169830f0 con 0x563316d49f80 2024-03-28T00:59:29.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.781+0000 7fc341ffb640 1 -- 172.21.15.169:0/831430904 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563316cbc910 con 0x563316d49f80 2024-03-28T00:59:29.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.781+0000 7fc341ffb640 1 -- 172.21.15.169:0/831430904 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fc354004420 con 0x563316d49f80 2024-03-28T00:59:29.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.781+0000 7fc363bee740 10 client.25939 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:59:29.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.781+0000 7fc363bee740 10 client.25939 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:59:29.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.781+0000 7fc363bee740 20 client.25939 populate_metadata read hostname 'smithi169' 2024-03-28T00:59:29.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.781+0000 7fc363bee740 1 --2- 172.21.15.169:0/831430904 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563316d60180 0x563316d625c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:29.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.781+0000 7fc363bee740 4 client.25939 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:59:29.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.781+0000 7fc363bee740 1 -- 172.21.15.169:0/831430904 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563316cb6c60 con 0x563316d60180 2024-03-28T00:59:29.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.782+0000 7fc359d9b640 1 --2- 172.21.15.169:0/831430904 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563316d60180 0x563316d625c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:29.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.784+0000 7fc359d9b640 1 --2- 172.21.15.169:0/831430904 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563316d60180 0x563316d625c0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89076dbfe951519a server_cookie=243f83a2fab91e99 in_seq=0 out_seq=0 2024-03-28T00:59:29.789 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.785+0000 7fc341ffb640 10 client.25939 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:29.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.788+0000 7fc341ffb640 1 -- 172.21.15.169:0/831430904 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563316cb6c60 con 0x563316d60180 2024-03-28T00:59:29.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.788+0000 7fc341ffb640 10 client.25939 handle_command_reply: tid=0 2024-03-28T00:59:29.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.788+0000 7fc363bee740 1 client.25939 shutdown 2024-03-28T00:59:29.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.791+0000 7fc363bee740 1 -- 172.21.15.169:0/831430904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d49f80 msgr2=0x563316d546b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:29.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.791+0000 7fc363bee740 1 --2- 172.21.15.169:0/831430904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d49f80 0x563316d546b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x563316d4ad70 tx=0x7fc354033d70 comp rx=0 tx=0).stop 2024-03-28T00:59:29.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.793+0000 7fc363bee740 1 -- 172.21.15.169:0/831430904 shutdown_connections 2024-03-28T00:59:29.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.793+0000 7fc363bee740 1 --2- 172.21.15.169:0/831430904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d4fae0 0x563316d5bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.793+0000 7fc363bee740 1 --2- 172.21.15.169:0/831430904 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563316d60180 0x563316d625c0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.793+0000 7fc363bee740 1 --2- 172.21.15.169:0/831430904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563316d4a970 0x563316d54bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.793+0000 7fc363bee740 1 --2- 172.21.15.169:0/831430904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d49f80 0x563316d546b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.793+0000 7fc363bee740 1 -- 172.21.15.169:0/831430904 >> 172.21.15.169:0/831430904 conn(0x563316d454f0 msgr2=0x563316d447b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:29.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.795+0000 7fc341ffb640 0 client.25939 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:29.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.796+0000 7fc363bee740 1 -- 172.21.15.169:0/831430904 shutdown_connections 2024-03-28T00:59:29.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.797+0000 7fc363bee740 1 -- 172.21.15.169:0/831430904 wait complete. 2024-03-28T00:59:29.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.798+0000 7fc363bee740 20 client.25939 trim_cache size 0 max 16384 2024-03-28T00:59:29.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:29 smithi169 ceph-mon[32274]: pgmap v440: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 716 MiB/s wr, 1.23k op/s 2024-03-28T00:59:29.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.841+0000 7fc363bee740 1 Processor -- start 2024-03-28T00:59:29.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.841+0000 7fc363bee740 1 -- start start 2024-03-28T00:59:29.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.842+0000 7fc363bee740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d49f80 0x563316d48dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:29.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.842+0000 7fc363bee740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d4a970 0x563316d5bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:29.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.842+0000 7fc35a59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d49f80 0x563316d48dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:29.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.842+0000 7fc35a59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d49f80 0x563316d48dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:39790/0 (socket says 172.21.15.169:39790) 2024-03-28T00:59:29.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.842+0000 7fc35a59c640 1 -- 172.21.15.169:0/1119597537 learned_addr learned my addr 172.21.15.169:0/1119597537 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:29.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.842+0000 7fc363bee740 1 --2- 172.21.15.169:0/1119597537 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563316d4fae0 0x563316d4cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:29.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.842+0000 7fc359d9b640 1 --2- 172.21.15.169:0/1119597537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d4a970 0x563316d5bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:29.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.842+0000 7fc363bee740 1 -- 172.21.15.169:0/1119597537 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c0120d0 con 0x563316d4fae0 2024-03-28T00:59:29.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.842+0000 7fc363bee740 1 -- 172.21.15.169:0/1119597537 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c00fbd0 con 0x563316d49f80 2024-03-28T00:59:29.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.842+0000 7fc363bee740 1 -- 172.21.15.169:0/1119597537 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c00dc70 con 0x563316d4a970 2024-03-28T00:59:29.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.843+0000 7fc35ad9d640 1 --2- 172.21.15.169:0/1119597537 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563316d4fae0 0x563316d4cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:29.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.843+0000 7fc359d9b640 1 -- 172.21.15.169:0/1119597537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d49f80 msgr2=0x563316d48dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:29.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.843+0000 7fc359d9b640 1 --2- 172.21.15.169:0/1119597537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d49f80 0x563316d48dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.843+0000 7fc359d9b640 1 -- 172.21.15.169:0/1119597537 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563316d4fae0 msgr2=0x563316d4cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:29.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.843+0000 7fc359d9b640 1 --2- 172.21.15.169:0/1119597537 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563316d4fae0 0x563316d4cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.843+0000 7fc359d9b640 1 -- 172.21.15.169:0/1119597537 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563316d40f60 con 0x563316d4a970 2024-03-28T00:59:29.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.844+0000 7fc359d9b640 1 --2- 172.21.15.169:0/1119597537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d4a970 0x563316d5bb80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc350097ab0 tx=0x7fc350009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=561018491d15befa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:29.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.844+0000 7fc343fff640 1 -- 172.21.15.169:0/1119597537 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3500ae070 con 0x563316d4a970 2024-03-28T00:59:29.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.844+0000 7fc343fff640 1 -- 172.21.15.169:0/1119597537 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3500a3df0 con 0x563316d4a970 2024-03-28T00:59:29.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.844+0000 7fc343fff640 1 -- 172.21.15.169:0/1119597537 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35009bb10 con 0x563316d4a970 2024-03-28T00:59:29.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.846+0000 7fc363bee740 1 -- 172.21.15.169:0/1119597537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d4a970 msgr2=0x563316d5bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:29.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.846+0000 7fc363bee740 1 --2- 172.21.15.169:0/1119597537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d4a970 0x563316d5bb80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc350097ab0 tx=0x7fc350009f90 comp rx=0 tx=0).stop 2024-03-28T00:59:29.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.846+0000 7fc363bee740 1 -- 172.21.15.169:0/1119597537 shutdown_connections 2024-03-28T00:59:29.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.846+0000 7fc363bee740 1 --2- 172.21.15.169:0/1119597537 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563316d4fae0 0x563316d4cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.846+0000 7fc363bee740 1 --2- 172.21.15.169:0/1119597537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d4a970 0x563316d5bb80 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.846+0000 7fc363bee740 1 --2- 172.21.15.169:0/1119597537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d49f80 0x563316d48dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.846+0000 7fc363bee740 1 -- 172.21.15.169:0/1119597537 >> 172.21.15.169:0/1119597537 conn(0x563316d454f0 msgr2=0x563316d447b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:29.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.847+0000 7fc363bee740 1 -- 172.21.15.169:0/1119597537 shutdown_connections 2024-03-28T00:59:29.850 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.847+0000 7fc363bee740 1 -- 172.21.15.169:0/1119597537 wait complete. 2024-03-28T00:59:29.850 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.847+0000 7fc363bee740 1 -- start start 2024-03-28T00:59:29.850 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.847+0000 7fc363bee740 1 Processor -- start 2024-03-28T00:59:29.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.848+0000 7fc363bee740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563316d49f80 0x563316db5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:29.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.848+0000 7fc363bee740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d4a970 0x563316db7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:29.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.848+0000 7fc363bee740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d4fae0 0x563316dba4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:29.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.848+0000 7fc363bee740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c00e270 con 0x563316d49f80 2024-03-28T00:59:29.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.848+0000 7fc363bee740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c00f570 con 0x563316d4a970 2024-03-28T00:59:29.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.848+0000 7fc363bee740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c00e8b0 con 0x563316d4fae0 2024-03-28T00:59:29.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.848+0000 7fc363bee740 10 client.0 _ll_register_callbacks cb 0x7fc350005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:59:29.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.848+0000 7fc359d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d4a970 0x563316db7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:29.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.848+0000 7fc359d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d4a970 0x563316db7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:39792/0 (socket says 172.21.15.169:39792) 2024-03-28T00:59:29.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.848+0000 7fc359d9b640 1 -- 172.21.15.169:0/2496320881 learned_addr learned my addr 172.21.15.169:0/2496320881 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:29.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.849+0000 7fc35ad9d640 1 --2- 172.21.15.169:0/2496320881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d4fae0 0x563316dba4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:29.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.849+0000 7fc35a59c640 1 --2- 172.21.15.169:0/2496320881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563316d49f80 0x563316db5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:29.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.850+0000 7fc35a59c640 1 -- 172.21.15.169:0/2496320881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d4fae0 msgr2=0x563316dba4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:29.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.850+0000 7fc35a59c640 1 --2- 172.21.15.169:0/2496320881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d4fae0 0x563316dba4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.850+0000 7fc35a59c640 1 -- 172.21.15.169:0/2496320881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d4a970 msgr2=0x563316db7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:29.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.850+0000 7fc35a59c640 1 --2- 172.21.15.169:0/2496320881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d4a970 0x563316db7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.850+0000 7fc35a59c640 1 -- 172.21.15.169:0/2496320881 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563316dbaab0 con 0x563316d49f80 2024-03-28T00:59:29.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.850+0000 7fc35a59c640 1 --2- 172.21.15.169:0/2496320881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563316d49f80 0x563316db5a20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fc354002230 tx=0x7fc354033480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c16d946c7ca3c3c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:29.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:29 smithi174 ceph-mon[36060]: pgmap v440: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 716 MiB/s wr, 1.23k op/s 2024-03-28T00:59:29.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.856+0000 7fc341ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:59:29.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.856+0000 7fc341ffb640 1 -- 172.21.15.169:0/2496320881 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354042040 con 0x563316d49f80 2024-03-28T00:59:29.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.856+0000 7fc341ffb640 1 -- 172.21.15.169:0/2496320881 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc354004d40 con 0x563316d49f80 2024-03-28T00:59:29.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.856+0000 7fc363bee740 1 -- 172.21.15.169:0/2496320881 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc3500a3df0 con 0x563316d49f80 2024-03-28T00:59:29.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.860+0000 7fc341ffb640 1 -- 172.21.15.169:0/2496320881 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc35402ce60 con 0x563316d49f80 2024-03-28T00:59:29.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.860+0000 7fc363bee740 10 client.16143 fetch_fsmap learned FSMap version 25 2024-03-28T00:59:29.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.860+0000 7fc363bee740 1 -- 172.21.15.169:0/2496320881 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563316dbbe30 con 0x563316d49f80 2024-03-28T00:59:29.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.861+0000 7fc341ffb640 1 -- 172.21.15.169:0/2496320881 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3500a3df0 con 0x563316d49f80 2024-03-28T00:59:29.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.861+0000 7fc341ffb640 1 -- 172.21.15.169:0/2496320881 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fc35405b590 con 0x563316d49f80 2024-03-28T00:59:29.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.862+0000 7fc363bee740 10 client.16143 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:59:29.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.862+0000 7fc363bee740 10 client.16143 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T00:59:29.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.862+0000 7fc363bee740 20 client.16143 populate_metadata read hostname 'smithi169' 2024-03-28T00:59:29.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.862+0000 7fc363bee740 1 --2- 172.21.15.169:0/2496320881 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563316dbc6a0 0x563316dbeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:29.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.862+0000 7fc363bee740 4 client.16143 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T00:59:29.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.862+0000 7fc363bee740 1 -- 172.21.15.169:0/2496320881 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563316c344e0 con 0x563316dbc6a0 2024-03-28T00:59:29.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.863+0000 7fc359d9b640 1 --2- 172.21.15.169:0/2496320881 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563316dbc6a0 0x563316dbeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:29.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.863+0000 7fc359d9b640 1 --2- 172.21.15.169:0/2496320881 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563316dbc6a0 0x563316dbeb00 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c568d3f7b4cb193 server_cookie=5be86c7d80b4f9a6 in_seq=0 out_seq=0 2024-03-28T00:59:29.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.864+0000 7fc341ffb640 10 client.16143 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:29.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.866+0000 7fc341ffb640 1 -- 172.21.15.169:0/2496320881 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x563316c344e0 con 0x563316dbc6a0 2024-03-28T00:59:29.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.866+0000 7fc341ffb640 10 client.16143 handle_command_reply: tid=0 2024-03-28T00:59:29.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.867+0000 7fc363bee740 1 client.16143 shutdown 2024-03-28T00:59:29.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.870+0000 7fc363bee740 1 -- 172.21.15.169:0/2496320881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563316d49f80 msgr2=0x563316db5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:29.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.870+0000 7fc363bee740 1 --2- 172.21.15.169:0/2496320881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563316d49f80 0x563316db5a20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fc354002230 tx=0x7fc354033480 comp rx=0 tx=0).stop 2024-03-28T00:59:29.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.871+0000 7fc363bee740 1 -- 172.21.15.169:0/2496320881 shutdown_connections 2024-03-28T00:59:29.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.871+0000 7fc363bee740 1 --2- 172.21.15.169:0/2496320881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563316d4fae0 0x563316dba4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.871+0000 7fc363bee740 1 --2- 172.21.15.169:0/2496320881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563316d4a970 0x563316db7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.871+0000 7fc363bee740 1 --2- 172.21.15.169:0/2496320881 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563316dbc6a0 0x563316dbeb00 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.871+0000 7fc363bee740 1 --2- 172.21.15.169:0/2496320881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563316d49f80 0x563316db5a20 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.871+0000 7fc363bee740 1 -- 172.21.15.169:0/2496320881 >> 172.21.15.169:0/2496320881 conn(0x563316d454f0 msgr2=0x563316d66e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:29.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.872+0000 7fc341ffb640 0 client.16143 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:29.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.872+0000 7fc363bee740 1 -- 172.21.15.169:0/2496320881 shutdown_connections 2024-03-28T00:59:29.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.873+0000 7fc363bee740 1 -- 172.21.15.169:0/2496320881 wait complete. 2024-03-28T00:59:29.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.873+0000 7fc363bee740 20 client.16143 trim_cache size 0 max 16384 2024-03-28T00:59:29.880 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T00:59:29.880 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "scrub active (4 inodes in the stack)", 2024-03-28T00:59:29.880 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": { 2024-03-28T00:59:29.880 INFO:teuthology.orchestra.run.smithi169.stdout: "b887080d-c0f2-4257-975e-e4f3f2016cf3": { 2024-03-28T00:59:29.880 INFO:teuthology.orchestra.run.smithi169.stdout: "path": "/", 2024-03-28T00:59:29.880 INFO:teuthology.orchestra.run.smithi169.stdout: "tag": "b887080d-c0f2-4257-975e-e4f3f2016cf3", 2024-03-28T00:59:29.880 INFO:teuthology.orchestra.run.smithi169.stdout: "options": "recursive,force" 2024-03-28T00:59:29.880 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T00:59:29.880 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T00:59:29.880 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T00:59:29.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.883+0000 7fc35b59e640 1 -- 172.21.15.169:0/1398738692 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc3440a2a10 msgr2=0x7fc3440a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:29.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.883+0000 7fc35b59e640 1 --2- 172.21.15.169:0/1398738692 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc3440a2a10 0x7fc3440a4ed0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc34c01e000 tx=0x7fc34c011b70 comp rx=0 tx=0).stop 2024-03-28T00:59:29.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.884+0000 7fc35b59e640 1 -- 172.21.15.169:0/1398738692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc35c108d70 msgr2=0x7fc35c1a6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:29.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.884+0000 7fc35b59e640 1 --2- 172.21.15.169:0/1398738692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc35c108d70 0x7fc35c1a6b30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc35000e9c0 tx=0x7fc35000ee80 comp rx=0 tx=0).stop 2024-03-28T00:59:29.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.886+0000 7fc35b59e640 1 -- 172.21.15.169:0/1398738692 shutdown_connections 2024-03-28T00:59:29.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.886+0000 7fc35b59e640 1 --2- 172.21.15.169:0/1398738692 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc3440a2a10 0x7fc3440a4ed0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.886+0000 7fc35b59e640 1 --2- 172.21.15.169:0/1398738692 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc35c109730 0x7fc35c1adba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.886+0000 7fc35b59e640 1 --2- 172.21.15.169:0/1398738692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc35c108d70 0x7fc35c1a6b30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.886+0000 7fc35b59e640 1 --2- 172.21.15.169:0/1398738692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc35c072ab0 0x7fc35c1a65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:29.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.886+0000 7fc35b59e640 1 -- 172.21.15.169:0/1398738692 >> 172.21.15.169:0/1398738692 conn(0x7fc35c06e580 msgr2=0x7fc35c070120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:29.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.888+0000 7fc35b59e640 1 -- 172.21.15.169:0/1398738692 shutdown_connections 2024-03-28T00:59:29.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T00:59:29.888+0000 7fc35b59e640 1 -- 172.21.15.169:0/1398738692 wait complete. 2024-03-28T00:59:29.929 INFO:tasks.cephfs.filesystem:scrub status for tag:b887080d-c0f2-4257-975e-e4f3f2016cf3 - {'path': '/', 'tag': 'b887080d-c0f2-4257-975e-e4f3f2016cf3', 'options': 'recursive,force'} 2024-03-28T00:59:30.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:30 smithi169 ceph-mon[32274]: scrub queued for path: / 2024-03-28T00:59:30.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:30 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T00:59:30.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:30 smithi169 ceph-mon[32274]: scrub summary: active paths [/] 2024-03-28T00:59:30.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:30 smithi169 ceph-mon[32274]: pgmap v441: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 759 MiB/s wr, 1.26k op/s 2024-03-28T00:59:30.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:30 smithi174 ceph-mon[36060]: scrub queued for path: / 2024-03-28T00:59:30.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:30 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T00:59:30.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:30 smithi174 ceph-mon[36060]: scrub summary: active paths [/] 2024-03-28T00:59:30.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:30 smithi174 ceph-mon[36060]: pgmap v441: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 759 MiB/s wr, 1.26k op/s 2024-03-28T00:59:30.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:30 smithi177 ceph-mon[36206]: scrub queued for path: / 2024-03-28T00:59:30.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:30 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T00:59:30.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:30 smithi177 ceph-mon[36206]: scrub summary: active paths [/] 2024-03-28T00:59:30.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:30 smithi177 ceph-mon[36206]: pgmap v441: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 759 MiB/s wr, 1.26k op/s 2024-03-28T00:59:33.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:33 smithi169 ceph-mon[32274]: pgmap v442: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 855 MiB/s wr, 1.47k op/s 2024-03-28T00:59:33.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:33 smithi174 ceph-mon[36060]: pgmap v442: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 855 MiB/s wr, 1.47k op/s 2024-03-28T00:59:33.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:33 smithi177 ceph-mon[36206]: pgmap v442: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 855 MiB/s wr, 1.47k op/s 2024-03-28T00:59:35.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:35 smithi169 ceph-mon[32274]: pgmap v443: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 806 MiB/s wr, 1.50k op/s 2024-03-28T00:59:35.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:35 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T00:59:35.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:35 smithi174 ceph-mon[36060]: pgmap v443: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 806 MiB/s wr, 1.50k op/s 2024-03-28T00:59:35.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:35 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T00:59:35.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:35 smithi177 ceph-mon[36206]: pgmap v443: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 806 MiB/s wr, 1.50k op/s 2024-03-28T00:59:35.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:35 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T00:59:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.179+0000 7ffb43fff640 1 -- 172.21.15.169:0/2180704548 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb44072950 msgr2=0x7ffb44072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.179+0000 7ffb43fff640 1 --2- 172.21.15.169:0/2180704548 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb44072950 0x7ffb44072d70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffb38009900 tx=0x7ffb3802bc80 comp rx=0 tx=0).stop 2024-03-28T00:59:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.179+0000 7ffb43fff640 1 -- 172.21.15.169:0/2180704548 shutdown_connections 2024-03-28T00:59:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.179+0000 7ffb43fff640 1 --2- 172.21.15.169:0/2180704548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb44109390 0x7ffb4410e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.179+0000 7ffb43fff640 1 --2- 172.21.15.169:0/2180704548 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb440732b0 0x7ffb44108cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.179+0000 7ffb43fff640 1 --2- 172.21.15.169:0/2180704548 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb44072950 0x7ffb44072d70 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.179+0000 7ffb43fff640 1 -- 172.21.15.169:0/2180704548 >> 172.21.15.169:0/2180704548 conn(0x7ffb4406e510 msgr2=0x7ffb44070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:37.185 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.180+0000 7ffb43fff640 1 -- 172.21.15.169:0/2180704548 shutdown_connections 2024-03-28T00:59:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.180+0000 7ffb43fff640 1 -- 172.21.15.169:0/2180704548 wait complete. 2024-03-28T00:59:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.181+0000 7ffb43fff640 1 Processor -- start 2024-03-28T00:59:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.181+0000 7ffb43fff640 1 -- start start 2024-03-28T00:59:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.181+0000 7ffb43fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb44072950 0x7ffb441a67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.181+0000 7ffb43fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb440732b0 0x7ffb441a6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.181+0000 7ffb43fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb44109390 0x7ffb44079900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.181+0000 7ffb43fff640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4411c0b0 con 0x7ffb44072950 2024-03-28T00:59:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.181+0000 7ffb43fff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4411bf30 con 0x7ffb44109390 2024-03-28T00:59:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.181+0000 7ffb43fff640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4411c230 con 0x7ffb440732b0 2024-03-28T00:59:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.182+0000 7ffb437fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb44109390 0x7ffb44079900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.182+0000 7ffb437fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb44109390 0x7ffb44079900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:53992/0 (socket says 172.21.15.169:53992) 2024-03-28T00:59:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.182+0000 7ffb437fe640 1 -- 172.21.15.169:0/150500750 learned_addr learned my addr 172.21.15.169:0/150500750 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.183+0000 7ffb437fe640 1 -- 172.21.15.169:0/150500750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb440732b0 msgr2=0x7ffb441a6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.183+0000 7ffb437fe640 1 --2- 172.21.15.169:0/150500750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb440732b0 0x7ffb441a6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.187 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.183+0000 7ffb437fe640 1 -- 172.21.15.169:0/150500750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb44072950 msgr2=0x7ffb441a67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:37.187 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.183+0000 7ffb42ffd640 1 --2- 172.21.15.169:0/150500750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb44072950 0x7ffb441a67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:37.187 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.183+0000 7ffb437fe640 1 --2- 172.21.15.169:0/150500750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb44072950 0x7ffb441a67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.187 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.183+0000 7ffb437fe640 1 -- 172.21.15.169:0/150500750 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4407a000 con 0x7ffb44109390 2024-03-28T00:59:37.187 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.184+0000 7ffb437fe640 1 --2- 172.21.15.169:0/150500750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb44109390 0x7ffb44079900 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c00b970 tx=0x7ffb3c00be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:37.190 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.187+0000 7ffb42ffd640 1 --2- 172.21.15.169:0/150500750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb44072950 0x7ffb441a67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T00:59:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.190+0000 7ffb48920640 1 -- 172.21.15.169:0/150500750 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c019070 con 0x7ffb44109390 2024-03-28T00:59:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.190+0000 7ffb48920640 1 -- 172.21.15.169:0/150500750 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb3c00cb40 con 0x7ffb44109390 2024-03-28T00:59:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.190+0000 7ffb48920640 1 -- 172.21.15.169:0/150500750 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c0076e0 con 0x7ffb44109390 2024-03-28T00:59:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.191+0000 7ffb43fff640 1 -- 172.21.15.169:0/150500750 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4407a2f0 con 0x7ffb44109390 2024-03-28T00:59:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.191+0000 7ffb43fff640 1 -- 172.21.15.169:0/150500750 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb4407a870 con 0x7ffb44109390 2024-03-28T00:59:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.196+0000 7ffb4c5ee740 1 Processor -- start 2024-03-28T00:59:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.196+0000 7ffb48920640 1 -- 172.21.15.169:0/150500750 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ffb3c01c070 con 0x7ffb44109390 2024-03-28T00:59:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.196+0000 7ffb4c5ee740 1 -- start start 2024-03-28T00:59:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.196+0000 7ffb4c5ee740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b79140ab20 0x55b79140af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.196+0000 7ffb4c5ee740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b79140b510 0x55b791410010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.196+0000 7ffb4c5ee740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b791410680 0x55b791412b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.196+0000 7ffb4c5ee740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b79136fb10 con 0x55b79140b510 2024-03-28T00:59:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.196+0000 7ffb4c5ee740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b791318900 con 0x55b791410680 2024-03-28T00:59:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.196+0000 7ffb4c5ee740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b79136bbb0 con 0x55b79140ab20 2024-03-28T00:59:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.197+0000 7ffb48920640 1 --2- 172.21.15.169:0/150500750 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffb2c0a2a00 0x7ffb2c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:37.201 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.197+0000 7ffb48920640 1 -- 172.21.15.169:0/150500750 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ffb3c0ce3e0 con 0x7ffb44109390 2024-03-28T00:59:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.197+0000 7ffb427fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b79140ab20 0x55b79140af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.200+0000 7ffb427fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b79140ab20 0x55b79140af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:39632/0 (socket says 172.21.15.169:39632) 2024-03-28T00:59:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.200+0000 7ffb427fc640 1 -- 172.21.15.169:0/801996319 learned_addr learned my addr 172.21.15.169:0/801996319 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.200+0000 7ffb427fc640 1 --2- 172.21.15.169:0/801996319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b791410680 0x55b791412b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.200+0000 7ffb42ffd640 1 --2- 172.21.15.169:0/801996319 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b79140b510 0x55b791410010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.201+0000 7ffb427fc640 1 -- 172.21.15.169:0/801996319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b791410680 msgr2=0x55b791412b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.201+0000 7ffb427fc640 1 --2- 172.21.15.169:0/801996319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b791410680 0x55b791412b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.201+0000 7ffb427fc640 1 -- 172.21.15.169:0/801996319 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b79140b510 msgr2=0x55b791410010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.201+0000 7ffb427fc640 1 --2- 172.21.15.169:0/801996319 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b79140b510 0x55b791410010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.201+0000 7ffb427fc640 1 -- 172.21.15.169:0/801996319 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b791084210 con 0x55b79140ab20 2024-03-28T00:59:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.202+0000 7ffb437fe640 1 --2- 172.21.15.169:0/150500750 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffb2c0a2a00 0x7ffb2c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:37.206 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.202+0000 7ffb427fc640 1 --2- 172.21.15.169:0/801996319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b79140ab20 0x55b79140af40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ffb340019b0 tx=0x7ffb34007dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cab1c781908a5dd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:37.209 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.206+0000 7ffb29ffb640 1 -- 172.21.15.169:0/801996319 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb34006ae0 con 0x55b79140ab20 2024-03-28T00:59:37.209 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.206+0000 7ffb29ffb640 1 -- 172.21.15.169:0/801996319 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb34011040 con 0x55b79140ab20 2024-03-28T00:59:37.209 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.207+0000 7ffb29ffb640 1 -- 172.21.15.169:0/801996319 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb34007610 con 0x55b79140ab20 2024-03-28T00:59:37.210 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.208+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/801996319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b79140ab20 msgr2=0x55b79140af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:37.210 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.208+0000 7ffb4c5ee740 1 --2- 172.21.15.169:0/801996319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b79140ab20 0x55b79140af40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ffb340019b0 tx=0x7ffb34007dd0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.213 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.210+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/801996319 shutdown_connections 2024-03-28T00:59:37.213 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.210+0000 7ffb4c5ee740 1 --2- 172.21.15.169:0/801996319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b791410680 0x55b791412b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.213 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.210+0000 7ffb4c5ee740 1 --2- 172.21.15.169:0/801996319 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b79140b510 0x55b791410010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.213 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.210+0000 7ffb4c5ee740 1 --2- 172.21.15.169:0/801996319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b79140ab20 0x55b79140af40 secure :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ffb340019b0 tx=0x7ffb34007dd0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.213 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.210+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/801996319 >> 172.21.15.169:0/801996319 conn(0x55b791406090 msgr2=0x55b791406490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:37.214 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.211+0000 7ffb437fe640 1 --2- 172.21.15.169:0/150500750 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffb2c0a2a00 0x7ffb2c0a4ec0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c0a2bf0 tx=0x7ffb3c014b90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:37.214 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.211+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/801996319 shutdown_connections 2024-03-28T00:59:37.222 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.219+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/801996319 wait complete. 2024-03-28T00:59:37.222 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.219+0000 7ffb4c5ee740 1 -- start start 2024-03-28T00:59:37.223 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.220+0000 7ffb4c5ee740 1 Processor -- start 2024-03-28T00:59:37.224 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.222+0000 7ffb4c5ee740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b79140b510 0x55b791415060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:37.225 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.222+0000 7ffb4c5ee740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b791410680 0x55b7914155a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:37.225 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.222+0000 7ffb4c5ee740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b79141a560 0x55b79141c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:37.225 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.222+0000 7ffb4c5ee740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b791413630 con 0x55b79140b510 2024-03-28T00:59:37.225 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.222+0000 7ffb4c5ee740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b79132bd50 con 0x55b791410680 2024-03-28T00:59:37.225 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.222+0000 7ffb4c5ee740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7914137b0 con 0x55b79141a560 2024-03-28T00:59:37.225 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.222+0000 7ffb4c5ee740 10 client.0 _ll_register_callbacks cb 0x55b79132b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:59:37.233 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.230+0000 7ffb427fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b79141a560 0x55b79141c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:37.233 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.230+0000 7ffb42ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b791410680 0x55b7914155a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:37.233 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.230+0000 7ffb427fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b79141a560 0x55b79141c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:39634/0 (socket says 172.21.15.169:39634) 2024-03-28T00:59:37.233 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.230+0000 7ffb427fc640 1 -- 172.21.15.169:0/4031310919 learned_addr learned my addr 172.21.15.169:0/4031310919 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:37.233 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.230+0000 7ffb427fc640 1 --2- 172.21.15.169:0/4031310919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b79140b510 0x55b791415060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:37.240 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.237+0000 7ffb427fc640 1 -- 172.21.15.169:0/4031310919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b791410680 msgr2=0x55b7914155a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:37.240 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.237+0000 7ffb427fc640 1 --2- 172.21.15.169:0/4031310919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b791410680 0x55b7914155a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.240 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.237+0000 7ffb427fc640 1 -- 172.21.15.169:0/4031310919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b79140b510 msgr2=0x55b791415060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:37.240 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.237+0000 7ffb427fc640 1 --2- 172.21.15.169:0/4031310919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b79140b510 0x55b791415060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.240 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.237+0000 7ffb427fc640 1 -- 172.21.15.169:0/4031310919 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b791150620 con 0x55b79141a560 2024-03-28T00:59:37.248 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.246+0000 7ffb427fc640 1 --2- 172.21.15.169:0/4031310919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b79141a560 0x55b79141c950 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ffb3400a040 tx=0x7ffb34007a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28a58c0af684c92c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:37.249 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.246+0000 7ffb22ffd640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T00:59:37.253 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.249+0000 7ffb22ffd640 1 -- 172.21.15.169:0/4031310919 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb34006b70 con 0x55b79141a560 2024-03-28T00:59:37.253 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.250+0000 7ffb22ffd640 1 -- 172.21.15.169:0/4031310919 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ffb34006d10 con 0x55b79141a560 2024-03-28T00:59:37.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.250+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/4031310919 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb34011040 con 0x55b79141a560 2024-03-28T00:59:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.257+0000 7ffb22ffd640 1 -- 172.21.15.169:0/4031310919 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb340107c0 con 0x55b79141a560 2024-03-28T00:59:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.259+0000 7ffb4c5ee740 10 client.25954 fetch_fsmap learned FSMap version 25 2024-03-28T00:59:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.259+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/4031310919 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55b79141fcc0 con 0x55b79141a560 2024-03-28T00:59:37.269 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.266+0000 7ffb22ffd640 1 -- 172.21.15.169:0/4031310919 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb34011040 con 0x55b79141a560 2024-03-28T00:59:37.269 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.266+0000 7ffb22ffd640 1 -- 172.21.15.169:0/4031310919 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ffb34021040 con 0x55b79141a560 2024-03-28T00:59:37.272 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.266+0000 7ffb4c5ee740 10 client.25954 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:59:37.272 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.266+0000 7ffb4c5ee740 10 client.25954 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:59:37.272 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.266+0000 7ffb4c5ee740 20 client.25954 populate_metadata read hostname 'smithi169' 2024-03-28T00:59:37.273 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.266+0000 7ffb4c5ee740 1 --2- 172.21.15.169:0/4031310919 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b791421670 0x55b791423a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:37.273 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.266+0000 7ffb4c5ee740 4 client.25954 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:59:37.273 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.266+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/4031310919 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b79137d050 con 0x55b791421670 2024-03-28T00:59:37.275 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.270+0000 7ffb42ffd640 1 --2- 172.21.15.169:0/4031310919 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b791421670 0x55b791423a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:37.286 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.281+0000 7ffb42ffd640 1 --2- 172.21.15.169:0/4031310919 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b791421670 0x55b791423a60 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d88fca36ae0e0787 server_cookie=b58055d5a307d794 in_seq=0 out_seq=0 2024-03-28T00:59:37.286 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.281+0000 7ffb22ffd640 10 client.25954 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:37.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.287+0000 7ffb22ffd640 1 -- 172.21.15.169:0/4031310919 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b79137d050 con 0x55b791421670 2024-03-28T00:59:37.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.287+0000 7ffb22ffd640 10 client.25954 handle_command_reply: tid=0 2024-03-28T00:59:37.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.287+0000 7ffb4c5ee740 1 client.25954 shutdown 2024-03-28T00:59:37.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.288+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/4031310919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b79141a560 msgr2=0x55b79141c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:37.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.288+0000 7ffb4c5ee740 1 --2- 172.21.15.169:0/4031310919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b79141a560 0x55b79141c950 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ffb3400a040 tx=0x7ffb34007a20 comp rx=0 tx=0).stop 2024-03-28T00:59:37.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.288+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/4031310919 shutdown_connections 2024-03-28T00:59:37.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.288+0000 7ffb4c5ee740 1 --2- 172.21.15.169:0/4031310919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b79141a560 0x55b79141c950 secure :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ffb3400a040 tx=0x7ffb34007a20 comp rx=0 tx=0).stop 2024-03-28T00:59:37.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.288+0000 7ffb4c5ee740 1 --2- 172.21.15.169:0/4031310919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b791410680 0x55b7914155a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.288+0000 7ffb4c5ee740 1 --2- 172.21.15.169:0/4031310919 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b791421670 0x55b791423a60 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.288+0000 7ffb4c5ee740 1 --2- 172.21.15.169:0/4031310919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b79140b510 0x55b791415060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.292 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.288+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/4031310919 >> 172.21.15.169:0/4031310919 conn(0x55b791406090 msgr2=0x55b791405290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:37.292 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.288+0000 7ffb22ffd640 0 client.25954 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:37.294 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.292+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/4031310919 shutdown_connections 2024-03-28T00:59:37.298 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.292+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/4031310919 wait complete. 2024-03-28T00:59:37.298 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.292+0000 7ffb4c5ee740 20 client.25954 trim_cache size 0 max 16384 2024-03-28T00:59:37.334 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.331+0000 7ffb4c5ee740 1 Processor -- start 2024-03-28T00:59:37.334 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.331+0000 7ffb4c5ee740 1 -- start start 2024-03-28T00:59:37.334 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.331+0000 7ffb4c5ee740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b79140b510 0x55b791413b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:37.339 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.334+0000 7ffb4c5ee740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b791410680 0x55b791415060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:37.339 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.334+0000 7ffb4c5ee740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b7914155a0 0x55b79140ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:37.339 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.334+0000 7ffb4c5ee740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb441cc660 con 0x55b79140b510 2024-03-28T00:59:37.339 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.334+0000 7ffb4c5ee740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb441ca160 con 0x55b7914155a0 2024-03-28T00:59:37.339 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.334+0000 7ffb4c5ee740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb44082da0 con 0x55b791410680 2024-03-28T00:59:37.343 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.339+0000 7ffb42ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b791410680 0x55b791415060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:37.343 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.340+0000 7ffb427fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b79140b510 0x55b791413b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:37.343 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.340+0000 7ffb427fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b79140b510 0x55b791413b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:41478/0 (socket says 172.21.15.169:41478) 2024-03-28T00:59:37.343 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.340+0000 7ffb42ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b791410680 0x55b791415060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:39646/0 (socket says 172.21.15.169:39646) 2024-03-28T00:59:37.343 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.340+0000 7ffb42ffd640 1 -- 172.21.15.169:0/1615093753 learned_addr learned my addr 172.21.15.169:0/1615093753 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:37.343 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.340+0000 7ffb427fc640 1 --2- 172.21.15.169:0/1615093753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b7914155a0 0x55b79140ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:37.348 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.345+0000 7ffb42ffd640 1 -- 172.21.15.169:0/1615093753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b7914155a0 msgr2=0x55b79140ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:37.348 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.345+0000 7ffb42ffd640 1 --2- 172.21.15.169:0/1615093753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b7914155a0 0x55b79140ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.348 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.345+0000 7ffb42ffd640 1 -- 172.21.15.169:0/1615093753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b79140b510 msgr2=0x55b791413b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:37.348 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.345+0000 7ffb42ffd640 1 --2- 172.21.15.169:0/1615093753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b79140b510 0x55b791413b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.348 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.345+0000 7ffb42ffd640 1 -- 172.21.15.169:0/1615093753 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b791423dd0 con 0x55b791410680 2024-03-28T00:59:37.348 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.345+0000 7ffb427fc640 1 --2- 172.21.15.169:0/1615093753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b79140b510 0x55b791413b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T00:59:37.354 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.352+0000 7ffb42ffd640 1 --2- 172.21.15.169:0/1615093753 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b791410680 0x55b791415060 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ffb3802c290 tx=0x7ffb3802bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f6023fd44378ecd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.353+0000 7ffb297fa640 1 -- 172.21.15.169:0/1615093753 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3804b070 con 0x55b791410680 2024-03-28T00:59:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.353+0000 7ffb297fa640 1 -- 172.21.15.169:0/1615093753 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb380043f0 con 0x55b791410680 2024-03-28T00:59:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.353+0000 7ffb297fa640 1 -- 172.21.15.169:0/1615093753 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb380046c0 con 0x55b791410680 2024-03-28T00:59:37.359 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.355+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/1615093753 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b791410680 msgr2=0x55b791415060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:37.360 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.355+0000 7ffb4c5ee740 1 --2- 172.21.15.169:0/1615093753 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b791410680 0x55b791415060 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ffb3802c290 tx=0x7ffb3802bc80 comp rx=0 tx=0).stop 2024-03-28T00:59:37.360 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.355+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/1615093753 shutdown_connections 2024-03-28T00:59:37.360 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.355+0000 7ffb4c5ee740 1 --2- 172.21.15.169:0/1615093753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b7914155a0 0x55b79140ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.360 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.355+0000 7ffb4c5ee740 1 --2- 172.21.15.169:0/1615093753 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b791410680 0x55b791415060 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.360 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.355+0000 7ffb4c5ee740 1 --2- 172.21.15.169:0/1615093753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b79140b510 0x55b791413b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.360 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.355+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/1615093753 >> 172.21.15.169:0/1615093753 conn(0x55b791406090 msgr2=0x55b791406470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.360+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/1615093753 shutdown_connections 2024-03-28T00:59:37.369 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.362+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/1615093753 wait complete. 2024-03-28T00:59:37.369 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.362+0000 7ffb4c5ee740 1 -- start start 2024-03-28T00:59:37.370 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.363+0000 7ffb4c5ee740 1 Processor -- start 2024-03-28T00:59:37.370 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.364+0000 7ffb4c5ee740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b79140b510 0x55b791433440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:37.370 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.364+0000 7ffb4c5ee740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b791410680 0x55b791433980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:37.370 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.365+0000 7ffb4c5ee740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b7914155a0 0x55b791431a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:37.370 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.365+0000 7ffb4c5ee740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb440833a0 con 0x55b791410680 2024-03-28T00:59:37.370 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.365+0000 7ffb4c5ee740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb440846a0 con 0x55b7914155a0 2024-03-28T00:59:37.370 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.365+0000 7ffb4c5ee740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb440839e0 con 0x55b79140b510 2024-03-28T00:59:37.370 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.365+0000 7ffb4c5ee740 10 client.0 _ll_register_callbacks cb 0x7ffb340077e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:59:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.367+0000 7ffb42ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b791410680 0x55b791433980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.367+0000 7ffb42ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b791410680 0x55b791433980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:41480/0 (socket says 172.21.15.169:41480) 2024-03-28T00:59:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.367+0000 7ffb42ffd640 1 -- 172.21.15.169:0/2365827815 learned_addr learned my addr 172.21.15.169:0/2365827815 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.371+0000 7ffb42ffd640 1 -- 172.21.15.169:0/2365827815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b79140b510 msgr2=0x55b791433440 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T00:59:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.371+0000 7ffb42ffd640 1 --2- 172.21.15.169:0/2365827815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b79140b510 0x55b791433440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.371+0000 7ffb42ffd640 1 -- 172.21.15.169:0/2365827815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b7914155a0 msgr2=0x55b791431a50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T00:59:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.371+0000 7ffb42ffd640 1 --2- 172.21.15.169:0/2365827815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b7914155a0 0x55b791431a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.371+0000 7ffb42ffd640 1 -- 172.21.15.169:0/2365827815 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b791432090 con 0x55b791410680 2024-03-28T00:59:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.377+0000 7ffb42ffd640 1 --2- 172.21.15.169:0/2365827815 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b791410680 0x55b791433980 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ffb38004650 tx=0x7ffb38009590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8b08b8d3a748a0a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.378+0000 7ffb22ffd640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:59:37.385 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.379+0000 7ffb22ffd640 1 -- 172.21.15.169:0/2365827815 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3804b040 con 0x55b791410680 2024-03-28T00:59:37.385 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.379+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/2365827815 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb380043f0 con 0x55b791410680 2024-03-28T00:59:37.385 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.379+0000 7ffb22ffd640 1 -- 172.21.15.169:0/2365827815 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ffb3802ca70 con 0x55b791410680 2024-03-28T00:59:37.385 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.379+0000 7ffb22ffd640 1 -- 172.21.15.169:0/2365827815 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb38005020 con 0x55b791410680 2024-03-28T00:59:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.384+0000 7ffb4c5ee740 10 client.16155 fetch_fsmap learned FSMap version 25 2024-03-28T00:59:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.384+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/2365827815 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55b79147cd60 con 0x55b791410680 2024-03-28T00:59:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.385+0000 7ffb22ffd640 1 -- 172.21.15.169:0/2365827815 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb380043f0 con 0x55b791410680 2024-03-28T00:59:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.385+0000 7ffb22ffd640 1 -- 172.21.15.169:0/2365827815 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ffb3804e070 con 0x55b791410680 2024-03-28T00:59:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.387+0000 7ffb4c5ee740 10 client.16155 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:59:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.387+0000 7ffb4c5ee740 10 client.16155 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:59:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.387+0000 7ffb4c5ee740 20 client.16155 populate_metadata read hostname 'smithi169' 2024-03-28T00:59:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.387+0000 7ffb4c5ee740 1 --2- 172.21.15.169:0/2365827815 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b79147d5b0 0x55b79147f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.387+0000 7ffb4c5ee740 4 client.16155 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T00:59:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.387+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/2365827815 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b7913773a0 con 0x55b79147d5b0 2024-03-28T00:59:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.392+0000 7ffb427fc640 1 --2- 172.21.15.169:0/2365827815 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b79147d5b0 0x55b79147f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:37.409 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.407+0000 7ffb427fc640 1 --2- 172.21.15.169:0/2365827815 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b79147d5b0 0x55b79147f9d0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bc6053bdbec9c14 server_cookie=8b161a5003dce556 in_seq=0 out_seq=0 2024-03-28T00:59:37.411 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:37.407+0000 7ffb22ffd640 10 client.16155 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:37.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:37 smithi169 ceph-mon[32274]: pgmap v444: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 819 MiB/s wr, 1.47k op/s 2024-03-28T00:59:37.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:37 smithi174 ceph-mon[36060]: pgmap v444: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 819 MiB/s wr, 1.47k op/s 2024-03-28T00:59:37.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:37 smithi177 ceph-mon[36206]: pgmap v444: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 819 MiB/s wr, 1.47k op/s 2024-03-28T00:59:39.562 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:39 smithi169 ceph-mon[32274]: pgmap v445: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 732 MiB/s wr, 1.17k op/s 2024-03-28T00:59:39.562 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:39 smithi169 ceph-mon[32274]: scrub summary: idle 2024-03-28T00:59:39.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:39 smithi174 ceph-mon[36060]: pgmap v445: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 732 MiB/s wr, 1.17k op/s 2024-03-28T00:59:39.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:39 smithi174 ceph-mon[36060]: scrub summary: idle 2024-03-28T00:59:39.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:39 smithi177 ceph-mon[36206]: pgmap v445: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 732 MiB/s wr, 1.17k op/s 2024-03-28T00:59:39.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:39 smithi177 ceph-mon[36206]: scrub summary: idle 2024-03-28T00:59:41.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:41 smithi169 ceph-mon[32274]: pgmap v446: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 779 MiB/s wr, 1.23k op/s 2024-03-28T00:59:41.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:41 smithi174 ceph-mon[36060]: pgmap v446: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 779 MiB/s wr, 1.23k op/s 2024-03-28T00:59:41.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:41 smithi177 ceph-mon[36206]: pgmap v446: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 779 MiB/s wr, 1.23k op/s 2024-03-28T00:59:42.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.494+0000 7ffb22ffd640 1 -- 172.21.15.169:0/2365827815 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b7913773a0 con 0x55b79147d5b0 2024-03-28T00:59:42.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.494+0000 7ffb22ffd640 10 client.16155 handle_command_reply: tid=0 2024-03-28T00:59:42.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.494+0000 7ffb4c5ee740 1 client.16155 shutdown 2024-03-28T00:59:42.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.494+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/2365827815 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b791410680 msgr2=0x55b791433980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:42.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.494+0000 7ffb4c5ee740 1 --2- 172.21.15.169:0/2365827815 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b791410680 0x55b791433980 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ffb38004650 tx=0x7ffb38009590 comp rx=0 tx=0).stop 2024-03-28T00:59:42.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.494+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/2365827815 shutdown_connections 2024-03-28T00:59:42.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.494+0000 7ffb4c5ee740 1 --2- 172.21.15.169:0/2365827815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b7914155a0 0x55b791431a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:42.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.494+0000 7ffb4c5ee740 1 --2- 172.21.15.169:0/2365827815 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b79147d5b0 0x55b79147f9d0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:42.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.494+0000 7ffb4c5ee740 1 --2- 172.21.15.169:0/2365827815 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b791410680 0x55b791433980 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:42.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.494+0000 7ffb4c5ee740 1 --2- 172.21.15.169:0/2365827815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b79140b510 0x55b791433440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:42.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.494+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/2365827815 >> 172.21.15.169:0/2365827815 conn(0x55b791406090 msgr2=0x55b791413b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:42.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.495+0000 7ffb22ffd640 0 client.16155 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:42.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.495+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/2365827815 shutdown_connections 2024-03-28T00:59:42.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.495+0000 7ffb4c5ee740 1 -- 172.21.15.169:0/2365827815 wait complete. 2024-03-28T00:59:42.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.495+0000 7ffb4c5ee740 20 client.16155 trim_cache size 0 max 16384 2024-03-28T00:59:42.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T00:59:42.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T00:59:42.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T00:59:42.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T00:59:42.502 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.498+0000 7ffb43fff640 1 -- 172.21.15.169:0/150500750 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffb2c0a2a00 msgr2=0x7ffb2c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:42.502 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.498+0000 7ffb43fff640 1 --2- 172.21.15.169:0/150500750 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffb2c0a2a00 0x7ffb2c0a4ec0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c0a2bf0 tx=0x7ffb3c014b90 comp rx=0 tx=0).stop 2024-03-28T00:59:42.502 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.498+0000 7ffb43fff640 1 -- 172.21.15.169:0/150500750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb44109390 msgr2=0x7ffb44079900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:42.502 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.498+0000 7ffb43fff640 1 --2- 172.21.15.169:0/150500750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb44109390 0x7ffb44079900 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c00b970 tx=0x7ffb3c00be30 comp rx=0 tx=0).stop 2024-03-28T00:59:42.502 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.498+0000 7ffb43fff640 1 -- 172.21.15.169:0/150500750 shutdown_connections 2024-03-28T00:59:42.502 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.498+0000 7ffb43fff640 1 --2- 172.21.15.169:0/150500750 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffb2c0a2a00 0x7ffb2c0a4ec0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:42.502 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.498+0000 7ffb43fff640 1 --2- 172.21.15.169:0/150500750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb44109390 0x7ffb44079900 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:42.502 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.498+0000 7ffb43fff640 1 --2- 172.21.15.169:0/150500750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb440732b0 0x7ffb441a6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:42.502 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.498+0000 7ffb43fff640 1 --2- 172.21.15.169:0/150500750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb44072950 0x7ffb441a67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:42.502 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.498+0000 7ffb43fff640 1 -- 172.21.15.169:0/150500750 >> 172.21.15.169:0/150500750 conn(0x7ffb4406e510 msgr2=0x7ffb44070260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:42.503 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.499+0000 7ffb43fff640 1 -- 172.21.15.169:0/150500750 shutdown_connections 2024-03-28T00:59:42.503 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:42.499+0000 7ffb43fff640 1 -- 172.21.15.169:0/150500750 wait complete. 2024-03-28T00:59:43.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:43 smithi174 ceph-mon[36060]: pgmap v447: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 872 MiB/s wr, 1.39k op/s 2024-03-28T00:59:43.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:43 smithi177 ceph-mon[36206]: pgmap v447: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 872 MiB/s wr, 1.39k op/s 2024-03-28T00:59:43.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:43 smithi169 ceph-mon[32274]: pgmap v447: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 872 MiB/s wr, 1.39k op/s 2024-03-28T00:59:45.604 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:45 smithi169 ceph-mon[32274]: pgmap v448: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 831 MiB/s wr, 1.34k op/s 2024-03-28T00:59:45.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:45 smithi174 ceph-mon[36060]: pgmap v448: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 831 MiB/s wr, 1.34k op/s 2024-03-28T00:59:45.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:45 smithi177 ceph-mon[36206]: pgmap v448: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 831 MiB/s wr, 1.34k op/s 2024-03-28T00:59:47.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:47 smithi174 ceph-mon[36060]: pgmap v449: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 847 MiB/s wr, 1.39k op/s 2024-03-28T00:59:47.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:47 smithi177 ceph-mon[36206]: pgmap v449: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 847 MiB/s wr, 1.39k op/s 2024-03-28T00:59:47.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:47 smithi169 ceph-mon[32274]: pgmap v449: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 847 MiB/s wr, 1.39k op/s 2024-03-28T00:59:49.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:49 smithi174 ceph-mon[36060]: pgmap v450: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 817 MiB/s wr, 1.37k op/s 2024-03-28T00:59:49.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:49 smithi177 ceph-mon[36206]: pgmap v450: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 817 MiB/s wr, 1.37k op/s 2024-03-28T00:59:49.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:49 smithi169 ceph-mon[32274]: pgmap v450: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 817 MiB/s wr, 1.37k op/s 2024-03-28T00:59:51.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:51 smithi174 ceph-mon[36060]: pgmap v451: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 790 MiB/s wr, 1.34k op/s 2024-03-28T00:59:51.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:51 smithi177 ceph-mon[36206]: pgmap v451: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 790 MiB/s wr, 1.34k op/s 2024-03-28T00:59:51.804 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:51 smithi169 ceph-mon[32274]: pgmap v451: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 790 MiB/s wr, 1.34k op/s 2024-03-28T00:59:53.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:53 smithi174 ceph-mon[36060]: pgmap v452: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 916 MiB/s wr, 1.57k op/s 2024-03-28T00:59:53.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:53 smithi177 ceph-mon[36206]: pgmap v452: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 916 MiB/s wr, 1.57k op/s 2024-03-28T00:59:53.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:53 smithi169 ceph-mon[32274]: pgmap v452: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 916 MiB/s wr, 1.57k op/s 2024-03-28T00:59:55.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:55 smithi174 ceph-mon[36060]: pgmap v453: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 799 MiB/s wr, 1.36k op/s 2024-03-28T00:59:55.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:55 smithi177 ceph-mon[36206]: pgmap v453: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 799 MiB/s wr, 1.36k op/s 2024-03-28T00:59:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.721+0000 7fafb3c54640 1 -- 172.21.15.169:0/284254495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafac10d9c0 msgr2=0x7fafac10de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.721+0000 7fafb3c54640 1 --2- 172.21.15.169:0/284254495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafac10d9c0 0x7fafac10de40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fafa4009be0 tx=0x7fafa402bf10 comp rx=0 tx=0).stop 2024-03-28T00:59:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.722+0000 7fafb3c54640 1 -- 172.21.15.169:0/284254495 shutdown_connections 2024-03-28T00:59:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.722+0000 7fafb3c54640 1 --2- 172.21.15.169:0/284254495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafac10d9c0 0x7fafac10de40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.722+0000 7fafb3c54640 1 --2- 172.21.15.169:0/284254495 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafac10b010 0x7fafac10d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.722+0000 7fafb3c54640 1 --2- 172.21.15.169:0/284254495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafac1086a0 0x7fafac10aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.722+0000 7fafb3c54640 1 -- 172.21.15.169:0/284254495 >> 172.21.15.169:0/284254495 conn(0x7fafac06e6e0 msgr2=0x7fafac070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.722+0000 7fafb3c54640 1 -- 172.21.15.169:0/284254495 shutdown_connections 2024-03-28T00:59:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.722+0000 7fafb3c54640 1 -- 172.21.15.169:0/284254495 wait complete. 2024-03-28T00:59:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.723+0000 7fafb3c54640 1 Processor -- start 2024-03-28T00:59:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.723+0000 7fafb3c54640 1 -- start start 2024-03-28T00:59:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.723+0000 7fafb3c54640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafac1086a0 0x7fafac1a66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.723+0000 7fafb3c54640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafac10b010 0x7fafac1a6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.723+0000 7fafb3c54640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafac10d9c0 0x7fafac1adc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.723+0000 7fafb3c54640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafac11c190 con 0x7fafac10d9c0 2024-03-28T00:59:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.723+0000 7fafb3c54640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafac11c010 con 0x7fafac10b010 2024-03-28T00:59:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.723+0000 7fafb3c54640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafac11c310 con 0x7fafac1086a0 2024-03-28T00:59:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.723+0000 7fafb19f0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafac1086a0 0x7fafac1a66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.723+0000 7fafb19f0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafac1086a0 0x7fafac1a66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:60082/0 (socket says 172.21.15.169:60082) 2024-03-28T00:59:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.723+0000 7fafb19f0640 1 -- 172.21.15.169:0/1554366365 learned_addr learned my addr 172.21.15.169:0/1554366365 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.723+0000 7fafb21f1640 1 --2- 172.21.15.169:0/1554366365 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafac10d9c0 0x7fafac1adc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.724+0000 7fafb21f1640 1 -- 172.21.15.169:0/1554366365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafac1086a0 msgr2=0x7fafac1a66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:55.728 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.724+0000 7fafb21f1640 1 --2- 172.21.15.169:0/1554366365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafac1086a0 0x7fafac1a66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.728 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.724+0000 7fafb21f1640 1 -- 172.21.15.169:0/1554366365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafac10b010 msgr2=0x7fafac1a6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:55.728 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.724+0000 7fafb21f1640 1 --2- 172.21.15.169:0/1554366365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafac10b010 0x7fafac1a6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.728 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.724+0000 7fafb21f1640 1 -- 172.21.15.169:0/1554366365 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafac1ae2b0 con 0x7fafac10d9c0 2024-03-28T00:59:55.729 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.724+0000 7fafb21f1640 1 --2- 172.21.15.169:0/1554366365 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafac10d9c0 0x7fafac1adc40 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fafa402c450 tx=0x7fafa40338c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:55.729 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.725+0000 7fafa2ffd640 1 -- 172.21.15.169:0/1554366365 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4043020 con 0x7fafac10d9c0 2024-03-28T00:59:55.729 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.725+0000 7fafb3c54640 1 -- 172.21.15.169:0/1554366365 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafac1ae540 con 0x7fafac10d9c0 2024-03-28T00:59:55.729 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.725+0000 7fafb3c54640 1 -- 172.21.15.169:0/1554366365 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafac1aea60 con 0x7fafac10d9c0 2024-03-28T00:59:55.729 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.725+0000 7fafa2ffd640 1 -- 172.21.15.169:0/1554366365 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafa4003b80 con 0x7fafac10d9c0 2024-03-28T00:59:55.729 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.725+0000 7fafa2ffd640 1 -- 172.21.15.169:0/1554366365 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa40347a0 con 0x7fafac10d9c0 2024-03-28T00:59:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.729+0000 7fafb683d740 1 Processor -- start 2024-03-28T00:59:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.731+0000 7fafa2ffd640 1 -- 172.21.15.169:0/1554366365 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fafa4046020 con 0x7fafac10d9c0 2024-03-28T00:59:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.732+0000 7fafa2ffd640 1 --2- 172.21.15.169:0/1554366365 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faf980a2a90 0x7faf980a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.733+0000 7fafa2ffd640 1 -- 172.21.15.169:0/1554366365 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fafa40f16a0 con 0x7fafac10d9c0 2024-03-28T00:59:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.734+0000 7fafb11ef640 1 --2- 172.21.15.169:0/1554366365 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faf980a2a90 0x7faf980a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.734+0000 7fafb683d740 1 -- start start 2024-03-28T00:59:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.735+0000 7fafb683d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bbc920 0x55f414bbcd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.735+0000 7fafb683d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f414bbd310 0x55f414bc1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.735+0000 7fafb683d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bc2480 0x55f414bc4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.735+0000 7fafb683d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f414b21b10 con 0x55f414bc2480 2024-03-28T00:59:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.735+0000 7fafb683d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f414aca900 con 0x55f414bbd310 2024-03-28T00:59:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.735+0000 7fafb683d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f414b1dbb0 con 0x55f414bbc920 2024-03-28T00:59:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.735+0000 7fafb19f0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bbc920 0x55f414bbcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.735+0000 7fafb19f0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bbc920 0x55f414bbcd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:60086/0 (socket says 172.21.15.169:60086) 2024-03-28T00:59:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.735+0000 7fafb19f0640 1 -- 172.21.15.169:0/3475929527 learned_addr learned my addr 172.21.15.169:0/3475929527 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.735+0000 7fafb21f1640 1 --2- 172.21.15.169:0/3475929527 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bc2480 0x55f414bc4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.735+0000 7fafb11ef640 1 --2- 172.21.15.169:0/3475929527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f414bbd310 0x55f414bc1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.736+0000 7fafb19f0640 1 -- 172.21.15.169:0/3475929527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f414bbd310 msgr2=0x55f414bc1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.736+0000 7fafb19f0640 1 --2- 172.21.15.169:0/3475929527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f414bbd310 0x55f414bc1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.736+0000 7fafb19f0640 1 -- 172.21.15.169:0/3475929527 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bc2480 msgr2=0x55f414bc4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.736+0000 7fafb19f0640 1 --2- 172.21.15.169:0/3475929527 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bc2480 0x55f414bc4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.736+0000 7fafb19f0640 1 -- 172.21.15.169:0/3475929527 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f414836210 con 0x55f414bbc920 2024-03-28T00:59:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.736+0000 7fafb19f0640 1 --2- 172.21.15.169:0/3475929527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bbc920 0x55f414bbcd40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fafa800bda0 tx=0x7fafa80107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0b534c22e63381d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.737+0000 7fafb11ef640 1 --2- 172.21.15.169:0/1554366365 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faf980a2a90 0x7faf980a4f50 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7faf9c0058c0 tx=0x7faf9c0127c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.738+0000 7faf97fff640 1 -- 172.21.15.169:0/3475929527 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa801c680 con 0x55f414bbc920 2024-03-28T00:59:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.738+0000 7faf97fff640 1 -- 172.21.15.169:0/3475929527 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafa801cd00 con 0x55f414bbc920 2024-03-28T00:59:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.739+0000 7fafb683d740 1 -- 172.21.15.169:0/3475929527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bbc920 msgr2=0x55f414bbcd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.739+0000 7fafb683d740 1 --2- 172.21.15.169:0/3475929527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bbc920 0x55f414bbcd40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fafa800bda0 tx=0x7fafa80107a0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.743+0000 7fafb683d740 1 -- 172.21.15.169:0/3475929527 shutdown_connections 2024-03-28T00:59:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.743+0000 7fafb683d740 1 --2- 172.21.15.169:0/3475929527 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bc2480 0x55f414bc4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.743+0000 7fafb683d740 1 --2- 172.21.15.169:0/3475929527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f414bbd310 0x55f414bc1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.743+0000 7fafb683d740 1 --2- 172.21.15.169:0/3475929527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bbc920 0x55f414bbcd40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.743+0000 7fafb683d740 1 -- 172.21.15.169:0/3475929527 >> 172.21.15.169:0/3475929527 conn(0x55f414bb7e90 msgr2=0x55f414bb8290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.745+0000 7fafb683d740 1 -- 172.21.15.169:0/3475929527 shutdown_connections 2024-03-28T00:59:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.749+0000 7fafb683d740 1 -- 172.21.15.169:0/3475929527 wait complete. 2024-03-28T00:59:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.749+0000 7fafb683d740 1 -- start start 2024-03-28T00:59:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.749+0000 7fafb683d740 1 Processor -- start 2024-03-28T00:59:55.753 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.750+0000 7fafb683d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f414bbc920 0x55f414bc6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:55.753 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.750+0000 7fafb683d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bbd310 0x55f414bc7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:55.753 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.750+0000 7fafb683d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bc2480 0x55f414bce540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:55.753 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.750+0000 7fafb683d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f414bc5480 con 0x55f414bbd310 2024-03-28T00:59:55.753 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.750+0000 7fafb683d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f414addd50 con 0x55f414bbc920 2024-03-28T00:59:55.753 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.750+0000 7fafb683d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f414bc5600 con 0x55f414bc2480 2024-03-28T00:59:55.753 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.750+0000 7fafb683d740 10 client.0 _ll_register_callbacks cb 0x55f414add720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:59:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.751+0000 7fafb11ef640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bbd310 0x55f414bc7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.752+0000 7fafb11ef640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bbd310 0x55f414bc7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:54194/0 (socket says 172.21.15.169:54194) 2024-03-28T00:59:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.752+0000 7fafb11ef640 1 -- 172.21.15.169:0/329946176 learned_addr learned my addr 172.21.15.169:0/329946176 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.752+0000 7fafb21f1640 1 --2- 172.21.15.169:0/329946176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bc2480 0x55f414bce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.753+0000 7fafb19f0640 1 --2- 172.21.15.169:0/329946176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f414bbc920 0x55f414bc6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.756+0000 7fafb11ef640 1 -- 172.21.15.169:0/329946176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bc2480 msgr2=0x55f414bce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.756+0000 7fafb11ef640 1 --2- 172.21.15.169:0/329946176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bc2480 0x55f414bce540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.756+0000 7fafb11ef640 1 -- 172.21.15.169:0/329946176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f414bbc920 msgr2=0x55f414bc6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.756+0000 7fafb11ef640 1 --2- 172.21.15.169:0/329946176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f414bbc920 0x55f414bc6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.756+0000 7fafb11ef640 1 -- 172.21.15.169:0/329946176 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f414902620 con 0x55f414bbd310 2024-03-28T00:59:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.757+0000 7fafb11ef640 1 --2- 172.21.15.169:0/329946176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bbd310 0x55f414bc7500 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7faf9c0108d0 tx=0x7faf9c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9382e804a093c1a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.757+0000 7faf957fa640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:59:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.758+0000 7faf957fa640 1 -- 172.21.15.169:0/329946176 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9c025070 con 0x55f414bbd310 2024-03-28T00:59:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.758+0000 7fafb683d740 1 -- 172.21.15.169:0/329946176 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f414b2f050 con 0x55f414bbd310 2024-03-28T00:59:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.758+0000 7faf957fa640 1 -- 172.21.15.169:0/329946176 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7faf9c00ead0 con 0x55f414bbd310 2024-03-28T00:59:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.758+0000 7faf957fa640 1 -- 172.21.15.169:0/329946176 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7faf9c031050 con 0x55f414bbd310 2024-03-28T00:59:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.758+0000 7fafb683d740 10 client.16167 fetch_fsmap learned FSMap version 25 2024-03-28T00:59:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.758+0000 7fafb683d740 1 -- 172.21.15.169:0/329946176 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f414bd19e0 con 0x55f414bbd310 2024-03-28T00:59:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.759+0000 7faf957fa640 1 -- 172.21.15.169:0/329946176 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f414b2f050 con 0x55f414bbd310 2024-03-28T00:59:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.759+0000 7faf957fa640 1 -- 172.21.15.169:0/329946176 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7faf9c01d380 con 0x55f414bbd310 2024-03-28T00:59:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.759+0000 7fafb683d740 10 client.16167 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:59:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.759+0000 7fafb683d740 10 client.16167 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:59:55.763 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.760+0000 7fafb683d740 20 client.16167 populate_metadata read hostname 'smithi169' 2024-03-28T00:59:55.763 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.760+0000 7fafb683d740 1 --2- 172.21.15.169:0/329946176 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f414bd35a0 0x55f414bd5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:55.763 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.760+0000 7fafb683d740 4 client.16167 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T00:59:55.763 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.760+0000 7fafb683d740 1 -- 172.21.15.169:0/329946176 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f414b293a0 con 0x55f414bd35a0 2024-03-28T00:59:55.763 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.760+0000 7fafb19f0640 1 --2- 172.21.15.169:0/329946176 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f414bd35a0 0x55f414bd5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.764+0000 7faf957fa640 10 client.16167 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.764+0000 7fafb19f0640 1 --2- 172.21.15.169:0/329946176 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f414bd35a0 0x55f414bd5a00 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec07477973b199dd server_cookie=b02052bb73b09e8d in_seq=0 out_seq=0 2024-03-28T00:59:55.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.767+0000 7faf957fa640 1 -- 172.21.15.169:0/329946176 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f414b293a0 con 0x55f414bd35a0 2024-03-28T00:59:55.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.767+0000 7faf957fa640 10 client.16167 handle_command_reply: tid=0 2024-03-28T00:59:55.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.767+0000 7fafb683d740 1 client.16167 shutdown 2024-03-28T00:59:55.774 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.771+0000 7fafb683d740 1 -- 172.21.15.169:0/329946176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bbd310 msgr2=0x55f414bc7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:55.774 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.771+0000 7fafb683d740 1 --2- 172.21.15.169:0/329946176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bbd310 0x55f414bc7500 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7faf9c0108d0 tx=0x7faf9c009260 comp rx=0 tx=0).stop 2024-03-28T00:59:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.772+0000 7fafb683d740 1 -- 172.21.15.169:0/329946176 shutdown_connections 2024-03-28T00:59:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.772+0000 7fafb683d740 1 --2- 172.21.15.169:0/329946176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bc2480 0x55f414bce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.772+0000 7fafb683d740 1 --2- 172.21.15.169:0/329946176 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f414bd35a0 0x55f414bd5a00 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.772+0000 7fafb683d740 1 --2- 172.21.15.169:0/329946176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bbd310 0x55f414bc7500 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.772+0000 7fafb683d740 1 --2- 172.21.15.169:0/329946176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f414bbc920 0x55f414bc6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.772+0000 7fafb683d740 1 -- 172.21.15.169:0/329946176 >> 172.21.15.169:0/329946176 conn(0x55f414bb7e90 msgr2=0x55f414bb7090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.772+0000 7faf957fa640 0 client.16167 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.772+0000 7fafb683d740 1 -- 172.21.15.169:0/329946176 shutdown_connections 2024-03-28T00:59:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.772+0000 7fafb683d740 1 -- 172.21.15.169:0/329946176 wait complete. 2024-03-28T00:59:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.772+0000 7fafb683d740 20 client.16167 trim_cache size 0 max 16384 2024-03-28T00:59:55.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:55 smithi169 ceph-mon[32274]: pgmap v453: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 799 MiB/s wr, 1.36k op/s 2024-03-28T00:59:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.829+0000 7fafb683d740 1 Processor -- start 2024-03-28T00:59:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.830+0000 7fafb683d740 1 -- start start 2024-03-28T00:59:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.830+0000 7fafb683d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bbc920 0x55f414bc6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:55.834 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.831+0000 7fafb683d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f414bbd310 0x55f414bce500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:55.834 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.831+0000 7fafb683d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bc2480 0x55f414bc67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:55.834 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.831+0000 7fafb683d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf900120d0 con 0x55f414bbc920 2024-03-28T00:59:55.835 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.831+0000 7fafb683d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9000fbd0 con 0x55f414bbd310 2024-03-28T00:59:55.835 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.831+0000 7fafb683d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9000dc70 con 0x55f414bc2480 2024-03-28T00:59:55.837 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.834+0000 7fafb11ef640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f414bbd310 0x55f414bce500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:55.837 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.834+0000 7fafb19f0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bbc920 0x55f414bc6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:55.837 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.834+0000 7fafb19f0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bbc920 0x55f414bc6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:54196/0 (socket says 172.21.15.169:54196) 2024-03-28T00:59:55.837 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.834+0000 7fafb19f0640 1 -- 172.21.15.169:0/2340472010 learned_addr learned my addr 172.21.15.169:0/2340472010 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:55.837 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.834+0000 7fafb21f1640 1 --2- 172.21.15.169:0/2340472010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bc2480 0x55f414bc67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.837+0000 7fafb19f0640 1 -- 172.21.15.169:0/2340472010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bc2480 msgr2=0x55f414bc67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.837+0000 7fafb19f0640 1 --2- 172.21.15.169:0/2340472010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bc2480 0x55f414bc67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.837+0000 7fafb19f0640 1 -- 172.21.15.169:0/2340472010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f414bbd310 msgr2=0x55f414bce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.837+0000 7fafb19f0640 1 --2- 172.21.15.169:0/2340472010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f414bbd310 0x55f414bce500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.838+0000 7fafb19f0640 1 -- 172.21.15.169:0/2340472010 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f414bbd6f0 con 0x55f414bbc920 2024-03-28T00:59:55.845 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.841+0000 7fafb19f0640 1 --2- 172.21.15.169:0/2340472010 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bbc920 0x55f414bc6290 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fafa8010ca0 tx=0x7fafa8005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af92a279c98a268d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:55.858 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.853+0000 7faf977fe640 1 -- 172.21.15.169:0/2340472010 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa802e070 con 0x55f414bbc920 2024-03-28T00:59:55.858 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.854+0000 7faf977fe640 1 -- 172.21.15.169:0/2340472010 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafa8029050 con 0x55f414bbc920 2024-03-28T00:59:55.859 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.855+0000 7fafb683d740 1 -- 172.21.15.169:0/2340472010 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bbc920 msgr2=0x55f414bc6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T00:59:55.859 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.855+0000 7fafb683d740 1 --2- 172.21.15.169:0/2340472010 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bbc920 0x55f414bc6290 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fafa8010ca0 tx=0x7fafa8005040 comp rx=0 tx=0).stop 2024-03-28T00:59:55.865 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.861+0000 7fafb683d740 1 -- 172.21.15.169:0/2340472010 shutdown_connections 2024-03-28T00:59:55.865 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.861+0000 7fafb683d740 1 --2- 172.21.15.169:0/2340472010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bc2480 0x55f414bc67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.865 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.861+0000 7fafb683d740 1 --2- 172.21.15.169:0/2340472010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f414bbd310 0x55f414bce500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.865 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.861+0000 7fafb683d740 1 --2- 172.21.15.169:0/2340472010 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bbc920 0x55f414bc6290 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.865 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.861+0000 7fafb683d740 1 -- 172.21.15.169:0/2340472010 >> 172.21.15.169:0/2340472010 conn(0x55f414bb7e90 msgr2=0x55f414bbe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T00:59:55.865 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.862+0000 7fafb683d740 1 -- 172.21.15.169:0/2340472010 shutdown_connections 2024-03-28T00:59:55.868 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.865+0000 7fafb683d740 1 -- 172.21.15.169:0/2340472010 wait complete. 2024-03-28T00:59:55.869 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.865+0000 7fafb683d740 1 -- start start 2024-03-28T00:59:55.869 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.866+0000 7fafb683d740 1 Processor -- start 2024-03-28T00:59:55.869 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.866+0000 7fafb683d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f414bbc920 0x55f414bce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:55.870 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.866+0000 7fafb683d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bbd310 0x55f414bcae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:55.870 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.866+0000 7fafb683d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bc2480 0x55f414bcb350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:55.870 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.866+0000 7fafb683d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9000e270 con 0x55f414bc2480 2024-03-28T00:59:55.870 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.866+0000 7fafb683d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9000f570 con 0x55f414bbc920 2024-03-28T00:59:55.870 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.866+0000 7fafb683d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9000e8b0 con 0x55f414bbd310 2024-03-28T00:59:55.870 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.866+0000 7fafb683d740 10 client.0 _ll_register_callbacks cb 0x7faf9c0107b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T00:59:55.870 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.867+0000 7fafb19f0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f414bbc920 0x55f414bce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:55.870 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.867+0000 7fafb11ef640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bbd310 0x55f414bcae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:55.871 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.867+0000 7fafb11ef640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bbd310 0x55f414bcae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:60112/0 (socket says 172.21.15.169:60112) 2024-03-28T00:59:55.871 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.867+0000 7fafb19f0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f414bbc920 0x55f414bce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:48710/0 (socket says 172.21.15.169:48710) 2024-03-28T00:59:55.871 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.867+0000 7fafb11ef640 1 -- 172.21.15.169:0/1965526875 learned_addr learned my addr 172.21.15.169:0/1965526875 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T00:59:55.871 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.867+0000 7fafb21f1640 1 --2- 172.21.15.169:0/1965526875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bc2480 0x55f414bcb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:55.871 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.868+0000 7fafb21f1640 1 -- 172.21.15.169:0/1965526875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bbd310 msgr2=0x55f414bcae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:55.871 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.868+0000 7fafb21f1640 1 --2- 172.21.15.169:0/1965526875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bbd310 0x55f414bcae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.871 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.868+0000 7fafb21f1640 1 -- 172.21.15.169:0/1965526875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f414bbc920 msgr2=0x55f414bce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T00:59:55.871 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.868+0000 7fafb21f1640 1 --2- 172.21.15.169:0/1965526875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f414bbc920 0x55f414bce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T00:59:55.871 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.868+0000 7fafb21f1640 1 -- 172.21.15.169:0/1965526875 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f414c2cec0 con 0x55f414bc2480 2024-03-28T00:59:55.872 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.869+0000 7fafb21f1640 1 --2- 172.21.15.169:0/1965526875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bc2480 0x55f414bcb350 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fafa40eeda0 tx=0x7fafa4005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b337471b1b0d46 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T00:59:55.872 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.869+0000 7faf957fa640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T00:59:55.872 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.869+0000 7faf957fa640 1 -- 172.21.15.169:0/1965526875 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa40da070 con 0x55f414bc2480 2024-03-28T00:59:55.872 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.870+0000 7fafb683d740 1 -- 172.21.15.169:0/1965526875 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fafa8029050 con 0x55f414bc2480 2024-03-28T00:59:55.873 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.870+0000 7faf957fa640 1 -- 172.21.15.169:0/1965526875 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fafa40f12e0 con 0x55f414bc2480 2024-03-28T00:59:55.873 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.870+0000 7faf957fa640 1 -- 172.21.15.169:0/1965526875 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafa40bdc40 con 0x55f414bc2480 2024-03-28T00:59:55.873 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.870+0000 7fafb683d740 10 client.16173 fetch_fsmap learned FSMap version 25 2024-03-28T00:59:55.873 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.870+0000 7fafb683d740 1 -- 172.21.15.169:0/1965526875 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f414c2e1b0 con 0x55f414bc2480 2024-03-28T00:59:55.874 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.871+0000 7faf957fa640 1 -- 172.21.15.169:0/1965526875 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafa8029050 con 0x55f414bc2480 2024-03-28T00:59:55.874 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.871+0000 7faf957fa640 1 -- 172.21.15.169:0/1965526875 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fafa4050060 con 0x55f414bc2480 2024-03-28T00:59:55.874 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.871+0000 7fafb683d740 10 client.16173 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T00:59:55.874 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.871+0000 7fafb683d740 10 client.16173 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T00:59:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.871+0000 7fafb683d740 20 client.16173 populate_metadata read hostname 'smithi169' 2024-03-28T00:59:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.871+0000 7fafb683d740 1 --2- 172.21.15.169:0/1965526875 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f414c2ea70 0x55f414c30e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T00:59:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.871+0000 7fafb683d740 4 client.16173 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T00:59:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.872+0000 7fafb683d740 1 -- 172.21.15.169:0/1965526875 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f414aa7710 con 0x55f414c2ea70 2024-03-28T00:59:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.872+0000 7fafb19f0640 1 --2- 172.21.15.169:0/1965526875 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f414c2ea70 0x55f414c30e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T00:59:55.879 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.876+0000 7fafb19f0640 1 --2- 172.21.15.169:0/1965526875 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f414c2ea70 0x55f414c30e90 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c653b8ed2bd8176e server_cookie=bef8486ac0d68102 in_seq=0 out_seq=0 2024-03-28T00:59:55.879 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T00:59:55.876+0000 7faf957fa640 10 client.16173 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T00:59:57.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:57 smithi177 ceph-mon[36206]: pgmap v454: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 813 MiB/s wr, 1.34k op/s 2024-03-28T00:59:57.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:57 smithi169 ceph-mon[32274]: pgmap v454: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 813 MiB/s wr, 1.34k op/s 2024-03-28T00:59:57.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:57 smithi174 ceph-mon[36060]: pgmap v454: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 813 MiB/s wr, 1.34k op/s 2024-03-28T00:59:59.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 00:59:59 smithi177 ceph-mon[36206]: pgmap v455: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 793 MiB/s wr, 1.29k op/s 2024-03-28T00:59:59.809 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 00:59:59 smithi169 ceph-mon[32274]: pgmap v455: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 793 MiB/s wr, 1.29k op/s 2024-03-28T00:59:59.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 00:59:59 smithi174 ceph-mon[36060]: pgmap v455: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 793 MiB/s wr, 1.29k op/s 2024-03-28T00:59:59.931 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T00:59:59.931 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:00:00.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.234+0000 7ffade16f640 1 -- 172.21.15.169:0/588500893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffad8107cc0 msgr2=0x7ffad81080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:00.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.234+0000 7ffade16f640 1 --2- 172.21.15.169:0/588500893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffad8107cc0 0x7ffad81080e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ffac8009900 tx=0x7ffac802bc60 comp rx=0 tx=0).stop 2024-03-28T01:00:00.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.234+0000 7ffade16f640 1 -- 172.21.15.169:0/588500893 shutdown_connections 2024-03-28T01:00:00.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.234+0000 7ffade16f640 1 --2- 172.21.15.169:0/588500893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffad8109870 0x7ffad8110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.234+0000 7ffade16f640 1 --2- 172.21.15.169:0/588500893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffad8108eb0 0x7ffad8109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.234+0000 7ffade16f640 1 --2- 172.21.15.169:0/588500893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffad8107cc0 0x7ffad81080e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.234+0000 7ffade16f640 1 -- 172.21.15.169:0/588500893 >> 172.21.15.169:0/588500893 conn(0x7ffad8076c60 msgr2=0x7ffad80790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:00.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.237+0000 7ffade16f640 1 -- 172.21.15.169:0/588500893 shutdown_connections 2024-03-28T01:00:00.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.242+0000 7ffade16f640 1 -- 172.21.15.169:0/588500893 wait complete. 2024-03-28T01:00:00.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.243+0000 7ffade16f640 1 Processor -- start 2024-03-28T01:00:00.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.246+0000 7ffade16f640 1 -- start start 2024-03-28T01:00:00.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.247+0000 7ffade16f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffad8107cc0 0x7ffad8070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:00.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.247+0000 7ffade16f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffad8108eb0 0x7ffad8070570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:00.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.247+0000 7ffade16f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffad8109870 0x7ffad81ae190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:00.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.247+0000 7ffade16f640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad8113a80 con 0x7ffad8109870 2024-03-28T01:00:00.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.247+0000 7ffade16f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad8113900 con 0x7ffad8107cc0 2024-03-28T01:00:00.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.247+0000 7ffade16f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad8113c00 con 0x7ffad8108eb0 2024-03-28T01:00:00.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.249+0000 7ffad77fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffad8107cc0 0x7ffad8070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:00.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.249+0000 7ffad77fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffad8107cc0 0x7ffad8070030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:41192/0 (socket says 172.21.15.169:41192) 2024-03-28T01:00:00.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.249+0000 7ffad77fe640 1 -- 172.21.15.169:0/4283033176 learned_addr learned my addr 172.21.15.169:0/4283033176 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:00.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.249+0000 7ffad6ffd640 1 --2- 172.21.15.169:0/4283033176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffad8108eb0 0x7ffad8070570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:00.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.250+0000 7ffad77fe640 1 -- 172.21.15.169:0/4283033176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffad8108eb0 msgr2=0x7ffad8070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:00.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.250+0000 7ffad77fe640 1 --2- 172.21.15.169:0/4283033176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffad8108eb0 0x7ffad8070570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.250+0000 7ffad77fe640 1 -- 172.21.15.169:0/4283033176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffad8109870 msgr2=0x7ffad81ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:00.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.250+0000 7ffad77fe640 1 --2- 172.21.15.169:0/4283033176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffad8109870 0x7ffad81ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.250+0000 7ffad77fe640 1 -- 172.21.15.169:0/4283033176 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad8070c40 con 0x7ffad8107cc0 2024-03-28T01:00:00.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.253+0000 7ffad77fe640 1 --2- 172.21.15.169:0/4283033176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffad8107cc0 0x7ffad8070030 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ffac802c270 tx=0x7ffac802c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:00.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.253+0000 7ffad4ff9640 1 -- 172.21.15.169:0/4283033176 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac8041070 con 0x7ffad8107cc0 2024-03-28T01:00:00.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.253+0000 7ffade16f640 1 -- 172.21.15.169:0/4283033176 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad8079fb0 con 0x7ffad8107cc0 2024-03-28T01:00:00.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.253+0000 7ffade16f640 1 -- 172.21.15.169:0/4283033176 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad807a4a0 con 0x7ffad8107cc0 2024-03-28T01:00:00.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.253+0000 7ffad4ff9640 1 -- 172.21.15.169:0/4283033176 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffac80054e0 con 0x7ffad8107cc0 2024-03-28T01:00:00.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.254+0000 7ffad4ff9640 1 -- 172.21.15.169:0/4283033176 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac8034440 con 0x7ffad8107cc0 2024-03-28T01:00:00.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.257+0000 7ffae0e30740 1 Processor -- start 2024-03-28T01:00:00.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.258+0000 7ffae0e30740 1 -- start start 2024-03-28T01:00:00.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.259+0000 7ffae0e30740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bd9da0 0x55afe6bda1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:00.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.259+0000 7ffae0e30740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55afe6bda790 0x55afe6bdf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:00.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.259+0000 7ffae0e30740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afe6bdf900 0x55afe6be1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:00.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.259+0000 7ffae0e30740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afe6b3f3d0 con 0x55afe6bdf900 2024-03-28T01:00:00.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.259+0000 7ffae0e30740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afe6ae7ad0 con 0x55afe6bda790 2024-03-28T01:00:00.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.259+0000 7ffae0e30740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afe6b3b470 con 0x55afe6bd9da0 2024-03-28T01:00:00.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.259+0000 7ffad6ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afe6bdf900 0x55afe6be1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:00.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.259+0000 7ffad6ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afe6bdf900 0x55afe6be1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47092/0 (socket says 172.21.15.169:47092) 2024-03-28T01:00:00.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.259+0000 7ffad6ffd640 1 -- 172.21.15.169:0/2850431184 learned_addr learned my addr 172.21.15.169:0/2850431184 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:00.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.259+0000 7ffad7fff640 1 --2- 172.21.15.169:0/2850431184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bd9da0 0x55afe6bda1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:00.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.260+0000 7ffad4ff9640 1 -- 172.21.15.169:0/4283033176 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ffac8044070 con 0x7ffad8107cc0 2024-03-28T01:00:00.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.260+0000 7ffad6ffd640 1 -- 172.21.15.169:0/2850431184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bd9da0 msgr2=0x55afe6bda1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:00.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.260+0000 7ffad6ffd640 1 --2- 172.21.15.169:0/2850431184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bd9da0 0x55afe6bda1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.260+0000 7ffad6ffd640 1 -- 172.21.15.169:0/2850431184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55afe6bda790 msgr2=0x55afe6bdf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:00.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.260+0000 7ffad6ffd640 1 --2- 172.21.15.169:0/2850431184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55afe6bda790 0x55afe6bdf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.260+0000 7ffad6ffd640 1 -- 172.21.15.169:0/2850431184 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55afe691d7d0 con 0x55afe6bdf900 2024-03-28T01:00:00.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.260+0000 7ffad6ffd640 1 --2- 172.21.15.169:0/2850431184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afe6bdf900 0x55afe6be1d90 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ffacc00a8a0 tx=0x7ffacc00ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0af7ffdc13d7062 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:00.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.261+0000 7ffad4ff9640 1 --2- 172.21.15.169:0/4283033176 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffac00a27f0 0x7ffac00a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:00.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.261+0000 7ffad4ff9640 1 -- 172.21.15.169:0/4283033176 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ffac80f0940 con 0x7ffad8107cc0 2024-03-28T01:00:00.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.262+0000 7ffabdffb640 1 -- 172.21.15.169:0/2850431184 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffacc015070 con 0x55afe6bdf900 2024-03-28T01:00:00.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.262+0000 7ffabdffb640 1 -- 172.21.15.169:0/2850431184 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffacc00cb40 con 0x55afe6bdf900 2024-03-28T01:00:00.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.262+0000 7ffabdffb640 1 -- 172.21.15.169:0/2850431184 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffacc0185c0 con 0x55afe6bdf900 2024-03-28T01:00:00.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.263+0000 7ffad7fff640 1 --2- 172.21.15.169:0/2850431184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bd9da0 0x55afe6bda1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:00:00.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.265+0000 7ffae0e30740 1 -- 172.21.15.169:0/2850431184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afe6bdf900 msgr2=0x55afe6be1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:00.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.265+0000 7ffae0e30740 1 --2- 172.21.15.169:0/2850431184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afe6bdf900 0x55afe6be1d90 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ffacc00a8a0 tx=0x7ffacc00ad60 comp rx=0 tx=0).stop 2024-03-28T01:00:00.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.265+0000 7ffad7fff640 1 --2- 172.21.15.169:0/4283033176 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffac00a27f0 0x7ffac00a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:00.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.265+0000 7ffad7fff640 1 --2- 172.21.15.169:0/4283033176 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffac00a27f0 0x7ffac00a4cb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x55afe6bdaff0 tx=0x7ffad0009450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:00.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.266+0000 7ffae0e30740 1 -- 172.21.15.169:0/2850431184 shutdown_connections 2024-03-28T01:00:00.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.266+0000 7ffae0e30740 1 --2- 172.21.15.169:0/2850431184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afe6bdf900 0x55afe6be1d90 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.266+0000 7ffae0e30740 1 --2- 172.21.15.169:0/2850431184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55afe6bda790 0x55afe6bdf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.266+0000 7ffae0e30740 1 --2- 172.21.15.169:0/2850431184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bd9da0 0x55afe6bda1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.266+0000 7ffae0e30740 1 -- 172.21.15.169:0/2850431184 >> 172.21.15.169:0/2850431184 conn(0x55afe6bd5310 msgr2=0x55afe6bd5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:00.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.268+0000 7ffae0e30740 1 -- 172.21.15.169:0/2850431184 shutdown_connections 2024-03-28T01:00:00.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.271+0000 7ffae0e30740 1 -- 172.21.15.169:0/2850431184 wait complete. 2024-03-28T01:00:00.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.271+0000 7ffae0e30740 1 -- start start 2024-03-28T01:00:00.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.272+0000 7ffae0e30740 1 Processor -- start 2024-03-28T01:00:00.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.272+0000 7ffae0e30740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afe6bd9da0 0x55afe6be42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:00.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.272+0000 7ffae0e30740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bda790 0x55afe6be4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:00.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.272+0000 7ffae0e30740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55afe6bdf900 0x55afe6beb840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:00.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.272+0000 7ffae0e30740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afe6be28b0 con 0x55afe6bd9da0 2024-03-28T01:00:00.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.272+0000 7ffae0e30740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afe6afb060 con 0x55afe6bdf900 2024-03-28T01:00:00.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.272+0000 7ffae0e30740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afe6be2a30 con 0x55afe6bda790 2024-03-28T01:00:00.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.272+0000 7ffae0e30740 10 client.0 _ll_register_callbacks cb 0x55afe6add6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:00.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.273+0000 7ffad6ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bda790 0x55afe6be4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:00.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.273+0000 7ffad6ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bda790 0x55afe6be4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:45854/0 (socket says 172.21.15.169:45854) 2024-03-28T01:00:00.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.273+0000 7ffad6ffd640 1 -- 172.21.15.169:0/4110872285 learned_addr learned my addr 172.21.15.169:0/4110872285 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:00.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.273+0000 7ffad7fff640 1 --2- 172.21.15.169:0/4110872285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55afe6bdf900 0x55afe6beb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:00.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.273+0000 7ffad7fff640 1 -- 172.21.15.169:0/4110872285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bda790 msgr2=0x55afe6be4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:00.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.273+0000 7ffad7fff640 1 --2- 172.21.15.169:0/4110872285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bda790 0x55afe6be4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.273+0000 7ffad7fff640 1 -- 172.21.15.169:0/4110872285 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afe6bd9da0 msgr2=0x55afe6be42c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:00:00.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.273+0000 7ffad7fff640 1 --2- 172.21.15.169:0/4110872285 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afe6bd9da0 0x55afe6be42c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.273+0000 7ffad7fff640 1 -- 172.21.15.169:0/4110872285 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55afe68533f0 con 0x55afe6bdf900 2024-03-28T01:00:00.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.274+0000 7ffad7fff640 1 --2- 172.21.15.169:0/4110872285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55afe6bdf900 0x55afe6beb840 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ffad000cc80 tx=0x7ffad0004bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56bddccf3ae1620e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:00.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.277+0000 7ffab6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:00:00.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.277+0000 7ffab6ffd640 1 -- 172.21.15.169:0/4110872285 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad0028060 con 0x55afe6bdf900 2024-03-28T01:00:00.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.278+0000 7ffae0e30740 1 -- 172.21.15.169:0/4110872285 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffacc00cb40 con 0x55afe6bdf900 2024-03-28T01:00:00.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.278+0000 7ffab6ffd640 1 -- 172.21.15.169:0/4110872285 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ffad0012030 con 0x55afe6bdf900 2024-03-28T01:00:00.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.278+0000 7ffab6ffd640 1 -- 172.21.15.169:0/4110872285 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffad002f050 con 0x55afe6bdf900 2024-03-28T01:00:00.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.282+0000 7ffae0e30740 10 client.26006 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:00.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.282+0000 7ffae0e30740 1 -- 172.21.15.169:0/4110872285 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55afe68130f0 con 0x55afe6bdf900 2024-03-28T01:00:00.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.283+0000 7ffab6ffd640 1 -- 172.21.15.169:0/4110872285 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffacc00cb40 con 0x55afe6bdf900 2024-03-28T01:00:00.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.283+0000 7ffab6ffd640 1 -- 172.21.15.169:0/4110872285 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ffad0023820 con 0x55afe6bdf900 2024-03-28T01:00:00.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.283+0000 7ffae0e30740 10 client.26006 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:00.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.283+0000 7ffae0e30740 10 client.26006 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:00.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.283+0000 7ffae0e30740 20 client.26006 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:00.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.283+0000 7ffae0e30740 1 --2- 172.21.15.169:0/4110872285 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55afe6bf02b0 0x55afe6bf2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:00.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.283+0000 7ffae0e30740 4 client.26006 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:00:00.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.283+0000 7ffae0e30740 1 -- 172.21.15.169:0/4110872285 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55afe6b4c910 con 0x55afe6bf02b0 2024-03-28T01:00:00.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.291+0000 7ffad77fe640 1 --2- 172.21.15.169:0/4110872285 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55afe6bf02b0 0x55afe6bf2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:00.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.296+0000 7ffad77fe640 1 --2- 172.21.15.169:0/4110872285 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55afe6bf02b0 0x55afe6bf2710 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=947ee7ca5d4e03e6 server_cookie=d240d33af2d7a836 in_seq=0 out_seq=0 2024-03-28T01:00:00.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.296+0000 7ffab6ffd640 10 client.26006 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:00.371 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.368+0000 7faf957fa640 1 -- 172.21.15.169:0/1965526875 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f414aa7710 con 0x55f414c2ea70 2024-03-28T01:00:00.371 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.368+0000 7faf957fa640 10 client.16173 handle_command_reply: tid=0 2024-03-28T01:00:00.372 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.368+0000 7fafb683d740 1 client.16173 shutdown 2024-03-28T01:00:00.372 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.369+0000 7fafb683d740 1 -- 172.21.15.169:0/1965526875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bc2480 msgr2=0x55f414bcb350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:00.372 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.369+0000 7fafb683d740 1 --2- 172.21.15.169:0/1965526875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bc2480 0x55f414bcb350 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fafa40eeda0 tx=0x7fafa4005bf0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.369+0000 7fafb683d740 1 -- 172.21.15.169:0/1965526875 shutdown_connections 2024-03-28T01:00:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.369+0000 7fafb683d740 1 --2- 172.21.15.169:0/1965526875 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f414c2ea70 0x55f414c30e90 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.369+0000 7fafb683d740 1 --2- 172.21.15.169:0/1965526875 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f414bc2480 0x55f414bcb350 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.369+0000 7fafb683d740 1 --2- 172.21.15.169:0/1965526875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f414bbd310 0x55f414bcae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.369+0000 7fafb683d740 1 --2- 172.21.15.169:0/1965526875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f414bbc920 0x55f414bce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.369+0000 7fafb683d740 1 -- 172.21.15.169:0/1965526875 >> 172.21.15.169:0/1965526875 conn(0x55f414bb7e90 msgr2=0x55f414bbb980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.369+0000 7faf957fa640 0 client.16173 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.369+0000 7fafb683d740 1 -- 172.21.15.169:0/1965526875 shutdown_connections 2024-03-28T01:00:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.369+0000 7fafb683d740 1 -- 172.21.15.169:0/1965526875 wait complete. 2024-03-28T01:00:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.370+0000 7fafb683d740 20 client.16173 trim_cache size 0 max 16384 2024-03-28T01:00:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:00:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:00:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:00:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:00:00.382 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.379+0000 7fafb3c54640 1 -- 172.21.15.169:0/1554366365 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faf980a2a90 msgr2=0x7faf980a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:00.383 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.379+0000 7fafb3c54640 1 --2- 172.21.15.169:0/1554366365 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faf980a2a90 0x7faf980a4f50 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7faf9c0058c0 tx=0x7faf9c0127c0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.383 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.379+0000 7fafb3c54640 1 -- 172.21.15.169:0/1554366365 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafac10d9c0 msgr2=0x7fafac1adc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:00.383 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.379+0000 7fafb3c54640 1 --2- 172.21.15.169:0/1554366365 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafac10d9c0 0x7fafac1adc40 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fafa402c450 tx=0x7fafa40338c0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.379+0000 7ffab6ffd640 1 -- 172.21.15.169:0/4110872285 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55afe6b4c910 con 0x55afe6bf02b0 2024-03-28T01:00:00.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.379+0000 7ffab6ffd640 10 client.26006 handle_command_reply: tid=0 2024-03-28T01:00:00.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.379+0000 7ffae0e30740 1 client.26006 shutdown 2024-03-28T01:00:00.384 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.381+0000 7fafb3c54640 1 -- 172.21.15.169:0/1554366365 shutdown_connections 2024-03-28T01:00:00.384 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.382+0000 7fafb3c54640 1 --2- 172.21.15.169:0/1554366365 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faf980a2a90 0x7faf980a4f50 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.384 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.382+0000 7fafb3c54640 1 --2- 172.21.15.169:0/1554366365 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafac10d9c0 0x7fafac1adc40 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.385 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.382+0000 7fafb3c54640 1 --2- 172.21.15.169:0/1554366365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafac10b010 0x7fafac1a6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.385 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.382+0000 7fafb3c54640 1 --2- 172.21.15.169:0/1554366365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafac1086a0 0x7fafac1a66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.382+0000 7ffae0e30740 1 -- 172.21.15.169:0/4110872285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55afe6bdf900 msgr2=0x55afe6beb840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:00.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.382+0000 7ffae0e30740 1 --2- 172.21.15.169:0/4110872285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55afe6bdf900 0x55afe6beb840 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ffad000cc80 tx=0x7ffad0004bc0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.386 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.382+0000 7fafb3c54640 1 -- 172.21.15.169:0/1554366365 >> 172.21.15.169:0/1554366365 conn(0x7fafac06e6e0 msgr2=0x7fafac109460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:00.386 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.382+0000 7fafb3c54640 1 -- 172.21.15.169:0/1554366365 shutdown_connections 2024-03-28T01:00:00.386 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:00.382+0000 7fafb3c54640 1 -- 172.21.15.169:0/1554366365 wait complete. 2024-03-28T01:00:00.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.384+0000 7ffae0e30740 1 -- 172.21.15.169:0/4110872285 shutdown_connections 2024-03-28T01:00:00.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.384+0000 7ffae0e30740 1 --2- 172.21.15.169:0/4110872285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55afe6bdf900 0x55afe6beb840 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.384+0000 7ffae0e30740 1 --2- 172.21.15.169:0/4110872285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bda790 0x55afe6be4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.384+0000 7ffae0e30740 1 --2- 172.21.15.169:0/4110872285 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55afe6bf02b0 0x55afe6bf2710 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.384+0000 7ffae0e30740 1 --2- 172.21.15.169:0/4110872285 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afe6bd9da0 0x55afe6be42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.384+0000 7ffae0e30740 1 -- 172.21.15.169:0/4110872285 >> 172.21.15.169:0/4110872285 conn(0x55afe6bd5310 msgr2=0x55afe6bd4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:00.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.384+0000 7ffab6ffd640 0 client.26006 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:00.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.385+0000 7ffae0e30740 1 -- 172.21.15.169:0/4110872285 shutdown_connections 2024-03-28T01:00:00.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.385+0000 7ffae0e30740 1 -- 172.21.15.169:0/4110872285 wait complete. 2024-03-28T01:00:00.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.385+0000 7ffae0e30740 20 client.26006 trim_cache size 0 max 16384 2024-03-28T01:00:00.440 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.437+0000 7ffae0e30740 1 Processor -- start 2024-03-28T01:00:00.440 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.437+0000 7ffae0e30740 1 -- start start 2024-03-28T01:00:00.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.438+0000 7ffae0e30740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55afe6bd9da0 0x55afe6be38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:00.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.438+0000 7ffae0e30740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bda790 0x55afe6beb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:00.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.438+0000 7ffae0e30740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afe6bdf900 0x55afe6bd3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:00.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.439+0000 7ffae0e30740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac4012090 con 0x55afe6bdf900 2024-03-28T01:00:00.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.439+0000 7ffae0e30740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac400fb90 con 0x55afe6bd9da0 2024-03-28T01:00:00.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.439+0000 7ffae0e30740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac400dc30 con 0x55afe6bda790 2024-03-28T01:00:00.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.439+0000 7ffad77fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55afe6bd9da0 0x55afe6be38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:00.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.440+0000 7ffad7fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afe6bdf900 0x55afe6bd3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:00.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.440+0000 7ffad7fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afe6bdf900 0x55afe6bd3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47094/0 (socket says 172.21.15.169:47094) 2024-03-28T01:00:00.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.440+0000 7ffad7fff640 1 -- 172.21.15.169:0/163160721 learned_addr learned my addr 172.21.15.169:0/163160721 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:00.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.440+0000 7ffad6ffd640 1 --2- 172.21.15.169:0/163160721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bda790 0x55afe6beb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:00.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.442+0000 7ffad77fe640 1 -- 172.21.15.169:0/163160721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bda790 msgr2=0x55afe6beb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:00.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.442+0000 7ffad77fe640 1 --2- 172.21.15.169:0/163160721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bda790 0x55afe6beb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.442+0000 7ffad77fe640 1 -- 172.21.15.169:0/163160721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afe6bdf900 msgr2=0x55afe6bd3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:00.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.442+0000 7ffad77fe640 1 --2- 172.21.15.169:0/163160721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afe6bdf900 0x55afe6bd3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.442+0000 7ffad77fe640 1 -- 172.21.15.169:0/163160721 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55afe6bdb020 con 0x55afe6bd9da0 2024-03-28T01:00:00.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.443+0000 7ffad77fe640 1 --2- 172.21.15.169:0/163160721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55afe6bd9da0 0x55afe6be38e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x55afe6bdb200 tx=0x7ffac802c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2be23b66f3cb9116 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:00.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.445+0000 7ffabd7fa640 1 -- 172.21.15.169:0/163160721 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac80bcf70 con 0x55afe6bd9da0 2024-03-28T01:00:00.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.445+0000 7ffabd7fa640 1 -- 172.21.15.169:0/163160721 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffac80c10c0 con 0x55afe6bd9da0 2024-03-28T01:00:00.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.445+0000 7ffabd7fa640 1 -- 172.21.15.169:0/163160721 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac8005000 con 0x55afe6bd9da0 2024-03-28T01:00:00.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.447+0000 7ffae0e30740 1 -- 172.21.15.169:0/163160721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55afe6bd9da0 msgr2=0x55afe6be38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:00.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.447+0000 7ffae0e30740 1 --2- 172.21.15.169:0/163160721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55afe6bd9da0 0x55afe6be38e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x55afe6bdb200 tx=0x7ffac802c940 comp rx=0 tx=0).stop 2024-03-28T01:00:00.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.447+0000 7ffae0e30740 1 -- 172.21.15.169:0/163160721 shutdown_connections 2024-03-28T01:00:00.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.447+0000 7ffae0e30740 1 --2- 172.21.15.169:0/163160721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afe6bdf900 0x55afe6bd3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.447+0000 7ffae0e30740 1 --2- 172.21.15.169:0/163160721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bda790 0x55afe6beb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.447+0000 7ffae0e30740 1 --2- 172.21.15.169:0/163160721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55afe6bd9da0 0x55afe6be38e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.447+0000 7ffae0e30740 1 -- 172.21.15.169:0/163160721 >> 172.21.15.169:0/163160721 conn(0x55afe6bd5310 msgr2=0x55afe6be42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:00.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.455+0000 7ffae0e30740 1 -- 172.21.15.169:0/163160721 shutdown_connections 2024-03-28T01:00:00.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.459+0000 7ffae0e30740 1 -- 172.21.15.169:0/163160721 wait complete. 2024-03-28T01:00:00.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.460+0000 7ffae0e30740 1 -- start start 2024-03-28T01:00:00.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.460+0000 7ffae0e30740 1 Processor -- start 2024-03-28T01:00:00.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.462+0000 7ffae0e30740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bd9da0 0x55afe6beb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:00.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.462+0000 7ffae0e30740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afe6bda790 0x55afe6beb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:00.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.462+0000 7ffae0e30740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55afe6bdf900 0x55afe6be9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:00.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.462+0000 7ffae0e30740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac400e230 con 0x55afe6bda790 2024-03-28T01:00:00.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.463+0000 7ffae0e30740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac400f530 con 0x55afe6bdf900 2024-03-28T01:00:00.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.463+0000 7ffae0e30740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac400e870 con 0x55afe6bd9da0 2024-03-28T01:00:00.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.463+0000 7ffad77fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bd9da0 0x55afe6beb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:00.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.463+0000 7ffad77fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bd9da0 0x55afe6beb230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:45884/0 (socket says 172.21.15.169:45884) 2024-03-28T01:00:00.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.463+0000 7ffad77fe640 1 -- 172.21.15.169:0/1407404132 learned_addr learned my addr 172.21.15.169:0/1407404132 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:00.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.463+0000 7ffad7fff640 1 --2- 172.21.15.169:0/1407404132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55afe6bdf900 0x55afe6be9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:00.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.463+0000 7ffae0e30740 10 client.0 _ll_register_callbacks cb 0x7ffad000ceb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:00.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.467+0000 7ffad77fe640 1 -- 172.21.15.169:0/1407404132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55afe6bdf900 msgr2=0x55afe6be9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:00.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.467+0000 7ffad77fe640 1 --2- 172.21.15.169:0/1407404132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55afe6bdf900 0x55afe6be9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.467+0000 7ffad77fe640 1 -- 172.21.15.169:0/1407404132 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afe6bda790 msgr2=0x55afe6beb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:00.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.467+0000 7ffad77fe640 1 --2- 172.21.15.169:0/1407404132 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afe6bda790 0x55afe6beb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.467+0000 7ffad77fe640 1 -- 172.21.15.169:0/1407404132 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55afe6bea070 con 0x55afe6bd9da0 2024-03-28T01:00:00.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.471+0000 7ffad77fe640 1 --2- 172.21.15.169:0/1407404132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bd9da0 0x55afe6beb230 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffac80b9940 tx=0x7ffac802c180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d60cfd7c859654a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:00.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.471+0000 7ffab6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:00:00.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.473+0000 7ffab6ffd640 1 -- 172.21.15.169:0/1407404132 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac80bd3d0 con 0x55afe6bd9da0 2024-03-28T01:00:00.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.474+0000 7ffae0e30740 1 -- 172.21.15.169:0/1407404132 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffac80c10c0 con 0x55afe6bd9da0 2024-03-28T01:00:00.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.474+0000 7ffab6ffd640 1 -- 172.21.15.169:0/1407404132 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ffac8005000 con 0x55afe6bd9da0 2024-03-28T01:00:00.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.479+0000 7ffab6ffd640 1 -- 172.21.15.169:0/1407404132 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffac80bcf70 con 0x55afe6bd9da0 2024-03-28T01:00:00.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.479+0000 7ffae0e30740 10 client.25984 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:00.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.479+0000 7ffae0e30740 1 -- 172.21.15.169:0/1407404132 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55afe6c4c6e0 con 0x55afe6bd9da0 2024-03-28T01:00:00.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.480+0000 7ffab6ffd640 1 -- 172.21.15.169:0/1407404132 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffac80c10c0 con 0x55afe6bd9da0 2024-03-28T01:00:00.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.481+0000 7ffab6ffd640 1 -- 172.21.15.169:0/1407404132 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ffac80bc3c0 con 0x55afe6bd9da0 2024-03-28T01:00:00.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.481+0000 7ffae0e30740 10 client.25984 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:00.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.481+0000 7ffae0e30740 10 client.25984 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:00.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.481+0000 7ffae0e30740 20 client.25984 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:00.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.481+0000 7ffae0e30740 1 --2- 172.21.15.169:0/1407404132 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55afe6c4cfe0 0x55afe6c4f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:00.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.481+0000 7ffae0e30740 4 client.25984 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:00:00.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.481+0000 7ffae0e30740 1 -- 172.21.15.169:0/1407404132 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55afe6be40b0 con 0x55afe6c4cfe0 2024-03-28T01:00:00.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.482+0000 7ffad6ffd640 1 --2- 172.21.15.169:0/1407404132 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55afe6c4cfe0 0x55afe6c4f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:00.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.491+0000 7ffad6ffd640 1 --2- 172.21.15.169:0/1407404132 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55afe6c4cfe0 0x55afe6c4f400 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a2a2ceda53029f server_cookie=a8c5393f35508384 in_seq=0 out_seq=0 2024-03-28T01:00:00.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.491+0000 7ffab6ffd640 10 client.25984 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:00.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.493+0000 7ffab6ffd640 1 -- 172.21.15.169:0/1407404132 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55afe6be40b0 con 0x55afe6c4cfe0 2024-03-28T01:00:00.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.493+0000 7ffab6ffd640 10 client.25984 handle_command_reply: tid=0 2024-03-28T01:00:00.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.494+0000 7ffae0e30740 1 client.25984 shutdown 2024-03-28T01:00:00.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.499+0000 7ffae0e30740 1 -- 172.21.15.169:0/1407404132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bd9da0 msgr2=0x55afe6beb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:00.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.499+0000 7ffae0e30740 1 --2- 172.21.15.169:0/1407404132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bd9da0 0x55afe6beb230 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffac80b9940 tx=0x7ffac802c180 comp rx=0 tx=0).stop 2024-03-28T01:00:00.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.499+0000 7ffae0e30740 1 -- 172.21.15.169:0/1407404132 shutdown_connections 2024-03-28T01:00:00.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.499+0000 7ffae0e30740 1 --2- 172.21.15.169:0/1407404132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55afe6bdf900 0x55afe6be9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.499+0000 7ffae0e30740 1 --2- 172.21.15.169:0/1407404132 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55afe6c4cfe0 0x55afe6c4f400 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.499+0000 7ffae0e30740 1 --2- 172.21.15.169:0/1407404132 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55afe6bda790 0x55afe6beb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.499+0000 7ffae0e30740 1 --2- 172.21.15.169:0/1407404132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afe6bd9da0 0x55afe6beb230 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.499+0000 7ffae0e30740 1 -- 172.21.15.169:0/1407404132 >> 172.21.15.169:0/1407404132 conn(0x55afe6bd5310 msgr2=0x55afe6bd8570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:00.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.499+0000 7ffab6ffd640 0 client.25984 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:00.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.507+0000 7ffae0e30740 1 -- 172.21.15.169:0/1407404132 shutdown_connections 2024-03-28T01:00:00.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.509+0000 7ffae0e30740 1 -- 172.21.15.169:0/1407404132 wait complete. 2024-03-28T01:00:00.514 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.511+0000 7ffae0e30740 20 client.25984 trim_cache size 0 max 16384 2024-03-28T01:00:00.517 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:00:00.517 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "no active scrubs running", 2024-03-28T01:00:00.517 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": {} 2024-03-28T01:00:00.517 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:00:00.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.517+0000 7ffade16f640 1 -- 172.21.15.169:0/4283033176 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffac00a27f0 msgr2=0x7ffac00a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:00.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.517+0000 7ffade16f640 1 --2- 172.21.15.169:0/4283033176 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffac00a27f0 0x7ffac00a4cb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x55afe6bdaff0 tx=0x7ffad0009450 comp rx=0 tx=0).stop 2024-03-28T01:00:00.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.517+0000 7ffade16f640 1 -- 172.21.15.169:0/4283033176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffad8107cc0 msgr2=0x7ffad8070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:00.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.517+0000 7ffade16f640 1 --2- 172.21.15.169:0/4283033176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffad8107cc0 0x7ffad8070030 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ffac802c270 tx=0x7ffac802c670 comp rx=0 tx=0).stop 2024-03-28T01:00:00.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.521+0000 7ffade16f640 1 -- 172.21.15.169:0/4283033176 shutdown_connections 2024-03-28T01:00:00.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.521+0000 7ffade16f640 1 --2- 172.21.15.169:0/4283033176 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffac00a27f0 0x7ffac00a4cb0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.521+0000 7ffade16f640 1 --2- 172.21.15.169:0/4283033176 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffad8109870 0x7ffad81ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.521+0000 7ffade16f640 1 --2- 172.21.15.169:0/4283033176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffad8108eb0 0x7ffad8070570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.521+0000 7ffade16f640 1 --2- 172.21.15.169:0/4283033176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffad8107cc0 0x7ffad8070030 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.521+0000 7ffade16f640 1 -- 172.21.15.169:0/4283033176 >> 172.21.15.169:0/4283033176 conn(0x7ffad8076c60 msgr2=0x7ffad810c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:00.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.527+0000 7ffade16f640 1 -- 172.21.15.169:0/4283033176 shutdown_connections 2024-03-28T01:00:00.531 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.528+0000 7ffade16f640 1 -- 172.21.15.169:0/4283033176 wait complete. 2024-03-28T01:00:00.592 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T01:00:00.592 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T01:00:00.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:00 smithi177 ceph-mon[36206]: overall HEALTH_OK 2024-03-28T01:00:00.677 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:00 smithi169 ceph-mon[32274]: overall HEALTH_OK 2024-03-28T01:00:00.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:00 smithi174 ceph-mon[36060]: overall HEALTH_OK 2024-03-28T01:00:00.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.987+0000 7f33bd4a7640 1 -- 172.21.15.169:0/874343029 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33b810d760 msgr2=0x7f33b81185e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:00.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.987+0000 7f33bd4a7640 1 --2- 172.21.15.169:0/874343029 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33b810d760 0x7f33b81185e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f33ac009900 tx=0x7f33ac02bc50 comp rx=0 tx=0).stop 2024-03-28T01:00:00.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.987+0000 7f33bd4a7640 1 -- 172.21.15.169:0/874343029 shutdown_connections 2024-03-28T01:00:00.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.987+0000 7f33bd4a7640 1 --2- 172.21.15.169:0/874343029 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33b810d760 0x7f33b81185e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.987+0000 7f33bd4a7640 1 --2- 172.21.15.169:0/874343029 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33b810ac80 0x7f33b810d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.987+0000 7f33bd4a7640 1 --2- 172.21.15.169:0/874343029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33b80734a0 0x7f33b810a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:00.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.987+0000 7f33bd4a7640 1 -- 172.21.15.169:0/874343029 >> 172.21.15.169:0/874343029 conn(0x7f33b806e3d0 msgr2=0x7f33b8070810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:00.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.992+0000 7f33bd4a7640 1 -- 172.21.15.169:0/874343029 shutdown_connections 2024-03-28T01:00:00.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.993+0000 7f33bd4a7640 1 -- 172.21.15.169:0/874343029 wait complete. 2024-03-28T01:00:00.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:00.995+0000 7f33bd4a7640 1 Processor -- start 2024-03-28T01:00:01.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.000+0000 7f33bd4a7640 1 -- start start 2024-03-28T01:00:01.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.001+0000 7f33bd4a7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33b80734a0 0x7f33b81151f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:01.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.001+0000 7f33bd4a7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33b810ac80 0x7f33b8115730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:01.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.001+0000 7f33bd4a7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33b810d760 0x7f33b81b6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:01.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.001+0000 7f33bd4a7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b811bf60 con 0x7f33b810d760 2024-03-28T01:00:01.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.001+0000 7f33bd4a7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b811bde0 con 0x7f33b80734a0 2024-03-28T01:00:01.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.001+0000 7f33bd4a7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b811c0e0 con 0x7f33b810ac80 2024-03-28T01:00:01.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.002+0000 7f33b7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33b80734a0 0x7f33b81151f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:01.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.002+0000 7f33b77fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33b810ac80 0x7f33b8115730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:01.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.002+0000 7f33b7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33b80734a0 0x7f33b81151f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:41256/0 (socket says 172.21.15.169:41256) 2024-03-28T01:00:01.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.002+0000 7f33b77fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33b810ac80 0x7f33b8115730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:45900/0 (socket says 172.21.15.169:45900) 2024-03-28T01:00:01.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.002+0000 7f33b77fe640 1 -- 172.21.15.169:0/1819539336 learned_addr learned my addr 172.21.15.169:0/1819539336 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:01.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.003+0000 7f33b77fe640 1 -- 172.21.15.169:0/1819539336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33b80734a0 msgr2=0x7f33b81151f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:01.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.003+0000 7f33b77fe640 1 --2- 172.21.15.169:0/1819539336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33b80734a0 0x7f33b81151f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:01.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.003+0000 7f33b77fe640 1 -- 172.21.15.169:0/1819539336 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33b810d760 msgr2=0x7f33b81b6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:01.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.003+0000 7f33b77fe640 1 --2- 172.21.15.169:0/1819539336 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33b810d760 0x7f33b81b6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:01.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.003+0000 7f33b77fe640 1 -- 172.21.15.169:0/1819539336 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b81188b0 con 0x7f33b810ac80 2024-03-28T01:00:01.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.004+0000 7f33b77fe640 1 --2- 172.21.15.169:0/1819539336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33b810ac80 0x7f33b8115730 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f33a800c920 tx=0x7f33a800cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:01.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.008+0000 7f33b57fa640 1 -- 172.21.15.169:0/1819539336 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a8007c70 con 0x7f33b810ac80 2024-03-28T01:00:01.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.008+0000 7f33bd4a7640 1 -- 172.21.15.169:0/1819539336 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b81b6c70 con 0x7f33b810ac80 2024-03-28T01:00:01.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.008+0000 7f33bd4a7640 1 -- 172.21.15.169:0/1819539336 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b81b7190 con 0x7f33b810ac80 2024-03-28T01:00:01.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.008+0000 7f33b57fa640 1 -- 172.21.15.169:0/1819539336 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33a8007e10 con 0x7f33b810ac80 2024-03-28T01:00:01.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.008+0000 7f33b57fa640 1 -- 172.21.15.169:0/1819539336 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a8018820 con 0x7f33b810ac80 2024-03-28T01:00:01.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.024+0000 7f33b57fa640 1 -- 172.21.15.169:0/1819539336 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f33a8018ae0 con 0x7f33b810ac80 2024-03-28T01:00:01.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.027+0000 7f33bd4a7640 1 -- 172.21.15.169:0/1819539336 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33a40052c0 con 0x7f33b810ac80 2024-03-28T01:00:01.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.029+0000 7f33b57fa640 1 --2- 172.21.15.169:0/1819539336 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f33a00a2a00 0x7f33a00a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:01.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.029+0000 7f33b57fa640 1 -- 172.21.15.169:0/1819539336 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f33a80ced70 con 0x7f33b810ac80 2024-03-28T01:00:01.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.030+0000 7f33b7fff640 1 --2- 172.21.15.169:0/1819539336 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f33a00a2a00 0x7f33a00a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:01.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.039+0000 7f33b57fa640 1 -- 172.21.15.169:0/1819539336 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f33a8014030 con 0x7f33b810ac80 2024-03-28T01:00:01.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.039+0000 7f33b7fff640 1 --2- 172.21.15.169:0/1819539336 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f33a00a2a00 0x7f33a00a4ec0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f33b80738a0 tx=0x7f33b0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:01.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:01 smithi177 ceph-mon[36206]: pgmap v456: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 792 MiB/s wr, 1.29k op/s 2024-03-28T01:00:01.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.709+0000 7f33bd4a7640 1 -- 172.21.15.169:0/1819539336 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33a40055b0 con 0x7f33b810ac80 2024-03-28T01:00:01.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.711+0000 7f33b57fa640 1 -- 172.21.15.169:0/1819539336 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7f33a809ba10 con 0x7f33b810ac80 2024-03-28T01:00:01.715 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T01:00:01.715 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T01:00:01.715 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T01:00:01.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.720+0000 7f33bd4a7640 1 -- 172.21.15.169:0/1819539336 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f33a00a2a00 msgr2=0x7f33a00a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:01.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.720+0000 7f33bd4a7640 1 --2- 172.21.15.169:0/1819539336 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f33a00a2a00 0x7f33a00a4ec0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f33b80738a0 tx=0x7f33b0009500 comp rx=0 tx=0).stop 2024-03-28T01:00:01.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.720+0000 7f33bd4a7640 1 -- 172.21.15.169:0/1819539336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33b810ac80 msgr2=0x7f33b8115730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:01.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.720+0000 7f33bd4a7640 1 --2- 172.21.15.169:0/1819539336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33b810ac80 0x7f33b8115730 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f33a800c920 tx=0x7f33a800cde0 comp rx=0 tx=0).stop 2024-03-28T01:00:01.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.722+0000 7f33bd4a7640 1 -- 172.21.15.169:0/1819539336 shutdown_connections 2024-03-28T01:00:01.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.722+0000 7f33bd4a7640 1 --2- 172.21.15.169:0/1819539336 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f33a00a2a00 0x7f33a00a4ec0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:01.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.722+0000 7f33bd4a7640 1 --2- 172.21.15.169:0/1819539336 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33b810d760 0x7f33b81b6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:01.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.722+0000 7f33bd4a7640 1 --2- 172.21.15.169:0/1819539336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33b810ac80 0x7f33b8115730 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:01.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.722+0000 7f33bd4a7640 1 --2- 172.21.15.169:0/1819539336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33b80734a0 0x7f33b81151f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:01.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.723+0000 7f33bd4a7640 1 -- 172.21.15.169:0/1819539336 >> 172.21.15.169:0/1819539336 conn(0x7f33b806e3d0 msgr2=0x7f33b81099d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:01.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.723+0000 7f33bd4a7640 1 -- 172.21.15.169:0/1819539336 shutdown_connections 2024-03-28T01:00:01.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:01.723+0000 7f33bd4a7640 1 -- 172.21.15.169:0/1819539336 wait complete. 2024-03-28T01:00:01.767 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:00:01.767 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T01:00:01.768 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:01 smithi169 ceph-mon[32274]: pgmap v456: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 792 MiB/s wr, 1.29k op/s 2024-03-28T01:00:01.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:01 smithi174 ceph-mon[36060]: pgmap v456: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 792 MiB/s wr, 1.29k op/s 2024-03-28T01:00:02.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.079+0000 7f26a96ac640 1 -- 172.21.15.169:0/812444687 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26a4107cc0 msgr2=0x7f26a41080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:02.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.079+0000 7f26a96ac640 1 --2- 172.21.15.169:0/812444687 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26a4107cc0 0x7f26a41080e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f2698009900 tx=0x7f269802bc30 comp rx=0 tx=0).stop 2024-03-28T01:00:02.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.082+0000 7f26a96ac640 1 -- 172.21.15.169:0/812444687 shutdown_connections 2024-03-28T01:00:02.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.082+0000 7f26a96ac640 1 --2- 172.21.15.169:0/812444687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26a4109870 0x7f26a4110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.082+0000 7f26a96ac640 1 --2- 172.21.15.169:0/812444687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26a4108eb0 0x7f26a4109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.082+0000 7f26a96ac640 1 --2- 172.21.15.169:0/812444687 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26a4107cc0 0x7f26a41080e0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.082+0000 7f26a96ac640 1 -- 172.21.15.169:0/812444687 >> 172.21.15.169:0/812444687 conn(0x7f26a4076c60 msgr2=0x7f26a40790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:02.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.085+0000 7f26a96ac640 1 -- 172.21.15.169:0/812444687 shutdown_connections 2024-03-28T01:00:02.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.085+0000 7f26a96ac640 1 -- 172.21.15.169:0/812444687 wait complete. 2024-03-28T01:00:02.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.086+0000 7f26a96ac640 1 Processor -- start 2024-03-28T01:00:02.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.087+0000 7f26a96ac640 1 -- start start 2024-03-28T01:00:02.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.087+0000 7f26a96ac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26a4107cc0 0x7f26a4072c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.087+0000 7f26a96ac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26a4108eb0 0x7f26a4073170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.087+0000 7f26a96ac640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26a4109870 0x7f26a40712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.087+0000 7f26a96ac640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a406bb10 con 0x7f26a4107cc0 2024-03-28T01:00:02.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.087+0000 7f26a96ac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a406b990 con 0x7f26a4108eb0 2024-03-28T01:00:02.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.087+0000 7f26a96ac640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a406bc90 con 0x7f26a4109870 2024-03-28T01:00:02.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.090+0000 7f26a27fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26a4108eb0 0x7f26a4073170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.090+0000 7f26a27fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26a4108eb0 0x7f26a4073170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:41278/0 (socket says 172.21.15.169:41278) 2024-03-28T01:00:02.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.090+0000 7f26a27fc640 1 -- 172.21.15.169:0/4096976641 learned_addr learned my addr 172.21.15.169:0/4096976641 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:02.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.091+0000 7f26a27fc640 1 -- 172.21.15.169:0/4096976641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26a4109870 msgr2=0x7f26a40712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:02.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.091+0000 7f26a27fc640 1 --2- 172.21.15.169:0/4096976641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26a4109870 0x7f26a40712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.091+0000 7f26a27fc640 1 -- 172.21.15.169:0/4096976641 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26a4107cc0 msgr2=0x7f26a4072c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:02.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.091+0000 7f26a27fc640 1 --2- 172.21.15.169:0/4096976641 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26a4107cc0 0x7f26a4072c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.091+0000 7f26a27fc640 1 -- 172.21.15.169:0/4096976641 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a4071ad0 con 0x7f26a4108eb0 2024-03-28T01:00:02.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.091+0000 7f26a27fc640 1 --2- 172.21.15.169:0/4096976641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26a4108eb0 0x7f26a4073170 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f269c00cce0 tx=0x7f269c00a620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:02.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.113+0000 7f268bfff640 1 -- 172.21.15.169:0/4096976641 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c004b20 con 0x7f26a4108eb0 2024-03-28T01:00:02.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.113+0000 7f268bfff640 1 -- 172.21.15.169:0/4096976641 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f269c004cc0 con 0x7f26a4108eb0 2024-03-28T01:00:02.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.113+0000 7f268bfff640 1 -- 172.21.15.169:0/4096976641 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c00f8c0 con 0x7f26a4108eb0 2024-03-28T01:00:02.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.114+0000 7f26a96ac640 1 -- 172.21.15.169:0/4096976641 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a4071dc0 con 0x7f26a4108eb0 2024-03-28T01:00:02.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.114+0000 7f26a96ac640 1 -- 172.21.15.169:0/4096976641 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a407ff20 con 0x7f26a4108eb0 2024-03-28T01:00:02.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.124+0000 7f268bfff640 1 -- 172.21.15.169:0/4096976641 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f269c00fa60 con 0x7f26a4108eb0 2024-03-28T01:00:02.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.134+0000 7f26ac372740 1 Processor -- start 2024-03-28T01:00:02.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.134+0000 7f268bfff640 1 --2- 172.21.15.169:0/4096976641 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f268c0a2a00 0x7f268c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.134+0000 7f268bfff640 1 -- 172.21.15.169:0/4096976641 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f269c0cff90 con 0x7f26a4108eb0 2024-03-28T01:00:02.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.137+0000 7f26ac372740 1 -- start start 2024-03-28T01:00:02.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.137+0000 7f26a2ffd640 1 --2- 172.21.15.169:0/4096976641 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f268c0a2a00 0x7f268c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.137+0000 7f26ac372740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c7608e690 0x563c7608eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.137+0000 7f26ac372740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c7608f080 0x563c76093b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.137+0000 7f26ac372740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c760941f0 0x563c76096680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.137+0000 7f26ac372740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c76051610 con 0x563c7608e690 2024-03-28T01:00:02.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.137+0000 7f26ac372740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c75ff43c0 con 0x563c760941f0 2024-03-28T01:00:02.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.137+0000 7f26ac372740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c75f9cd60 con 0x563c7608f080 2024-03-28T01:00:02.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.137+0000 7f26a2ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c7608e690 0x563c7608eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.137+0000 7f26a2ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c7608e690 0x563c7608eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47142/0 (socket says 172.21.15.169:47142) 2024-03-28T01:00:02.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.137+0000 7f26a2ffd640 1 -- 172.21.15.169:0/759258796 learned_addr learned my addr 172.21.15.169:0/759258796 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:02.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.138+0000 7f26a2ffd640 1 -- 172.21.15.169:0/759258796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c7608f080 msgr2=0x563c76093b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:02.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.138+0000 7f26a2ffd640 1 --2- 172.21.15.169:0/759258796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c7608f080 0x563c76093b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.138+0000 7f26a2ffd640 1 -- 172.21.15.169:0/759258796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c760941f0 msgr2=0x563c76096680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:02.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.138+0000 7f26a2ffd640 1 --2- 172.21.15.169:0/759258796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c760941f0 0x563c76096680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.138+0000 7f26a2ffd640 1 -- 172.21.15.169:0/759258796 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c75d083f0 con 0x563c7608e690 2024-03-28T01:00:02.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.139+0000 7f26a2ffd640 1 --2- 172.21.15.169:0/759258796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c7608e690 0x563c7608eab0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f2698009230 tx=0x7f26980041c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78ce67420a82633d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:02.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.139+0000 7f26a2ffd640 1 --2- 172.21.15.169:0/4096976641 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f268c0a2a00 0x7f268c0a4ec0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2698005ab0 tx=0x7f269802e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:02.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.141+0000 7f26897fa640 1 -- 172.21.15.169:0/759258796 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2698050070 con 0x563c7608e690 2024-03-28T01:00:02.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.141+0000 7f26897fa640 1 -- 172.21.15.169:0/759258796 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2698003bc0 con 0x563c7608e690 2024-03-28T01:00:02.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.142+0000 7f26ac372740 1 -- 172.21.15.169:0/759258796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c7608e690 msgr2=0x563c7608eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:02.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.142+0000 7f26ac372740 1 --2- 172.21.15.169:0/759258796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c7608e690 0x563c7608eab0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f2698009230 tx=0x7f26980041c0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.143+0000 7f26ac372740 1 -- 172.21.15.169:0/759258796 shutdown_connections 2024-03-28T01:00:02.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.143+0000 7f26ac372740 1 --2- 172.21.15.169:0/759258796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c760941f0 0x563c76096680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.143+0000 7f26ac372740 1 --2- 172.21.15.169:0/759258796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c7608f080 0x563c76093b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.143+0000 7f26ac372740 1 --2- 172.21.15.169:0/759258796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c7608e690 0x563c7608eab0 secure :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f2698009230 tx=0x7f26980041c0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.143+0000 7f26ac372740 1 -- 172.21.15.169:0/759258796 >> 172.21.15.169:0/759258796 conn(0x563c76089c00 msgr2=0x563c7608a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:02.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.146+0000 7f26ac372740 1 -- 172.21.15.169:0/759258796 shutdown_connections 2024-03-28T01:00:02.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.149+0000 7f26ac372740 1 -- 172.21.15.169:0/759258796 wait complete. 2024-03-28T01:00:02.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.149+0000 7f26ac372740 1 -- start start 2024-03-28T01:00:02.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.150+0000 7f26ac372740 1 Processor -- start 2024-03-28T01:00:02.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.154+0000 7f26ac372740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c7608f080 0x563c76098cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.154+0000 7f26ac372740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c760941f0 0x563c76099210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.154+0000 7f26ac372740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c7609e240 0x563c760a0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.154+0000 7f26ac372740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c75fb0240 con 0x563c760941f0 2024-03-28T01:00:02.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.154+0000 7f26ac372740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c75ff0460 con 0x563c7608f080 2024-03-28T01:00:02.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.154+0000 7f26ac372740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c76097190 con 0x563c7609e240 2024-03-28T01:00:02.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.154+0000 7f26ac372740 10 client.0 _ll_register_callbacks cb 0x563c75fafc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:02.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.154+0000 7f26a2ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c7608f080 0x563c76098cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.155+0000 7f26a2ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c7608f080 0x563c76098cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:41284/0 (socket says 172.21.15.169:41284) 2024-03-28T01:00:02.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.155+0000 7f26a2ffd640 1 -- 172.21.15.169:0/817214393 learned_addr learned my addr 172.21.15.169:0/817214393 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:02.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.158+0000 7f26a27fc640 1 --2- 172.21.15.169:0/817214393 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c760941f0 0x563c76099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.158+0000 7f26a37fe640 1 --2- 172.21.15.169:0/817214393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c7609e240 0x563c760a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.159+0000 7f26a2ffd640 1 -- 172.21.15.169:0/817214393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c7609e240 msgr2=0x563c760a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:02.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.159+0000 7f26a2ffd640 1 --2- 172.21.15.169:0/817214393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c7609e240 0x563c760a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.159+0000 7f26a2ffd640 1 -- 172.21.15.169:0/817214393 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c760941f0 msgr2=0x563c76099210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:02.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.159+0000 7f26a2ffd640 1 --2- 172.21.15.169:0/817214393 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c760941f0 0x563c76099210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.159+0000 7f26a2ffd640 1 -- 172.21.15.169:0/817214393 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c75dd27d0 con 0x563c7608f080 2024-03-28T01:00:02.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.160+0000 7f26a2ffd640 1 --2- 172.21.15.169:0/817214393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c7608f080 0x563c76098cd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2698009900 tx=0x7f2698004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a93fa13efbea9402 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:02.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.160+0000 7f2686ff5640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:00:02.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.161+0000 7f2686ff5640 1 -- 172.21.15.169:0/817214393 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2698050040 con 0x563c7608f080 2024-03-28T01:00:02.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.161+0000 7f26ac372740 1 -- 172.21.15.169:0/817214393 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2698003bc0 con 0x563c7608f080 2024-03-28T01:00:02.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.165+0000 7f2686ff5640 1 -- 172.21.15.169:0/817214393 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f269804b450 con 0x563c7608f080 2024-03-28T01:00:02.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.165+0000 7f2686ff5640 1 -- 172.21.15.169:0/817214393 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f269804b650 con 0x563c7608f080 2024-03-28T01:00:02.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.165+0000 7f26ac372740 10 client.26024 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:02.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.165+0000 7f26ac372740 1 -- 172.21.15.169:0/817214393 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563c760a39a0 con 0x563c7608f080 2024-03-28T01:00:02.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.169+0000 7f2686ff5640 1 -- 172.21.15.169:0/817214393 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2698003bc0 con 0x563c7608f080 2024-03-28T01:00:02.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.169+0000 7f2686ff5640 1 -- 172.21.15.169:0/817214393 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2698053070 con 0x563c7608f080 2024-03-28T01:00:02.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.169+0000 7f26ac372740 10 client.26024 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:02.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.169+0000 7f26ac372740 10 client.26024 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:00:02.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.169+0000 7f26ac372740 20 client.26024 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:02.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.169+0000 7f26ac372740 1 --2- 172.21.15.169:0/817214393 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x563c760a5150 0x563c760a7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.169+0000 7f26ac372740 4 client.26024 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:00:02.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.169+0000 7f26ac372740 1 -- 172.21.15.169:0/817214393 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563c76001900 con 0x563c760a5150 2024-03-28T01:00:02.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.171+0000 7f26a27fc640 1 --2- 172.21.15.169:0/817214393 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x563c760a5150 0x563c760a7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.173+0000 7f26a27fc640 1 --2- 172.21.15.169:0/817214393 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x563c760a5150 0x563c760a7570 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9b818b32bdafc0cc server_cookie=bb7d48494e374f41 in_seq=0 out_seq=0 2024-03-28T01:00:02.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.175+0000 7f2686ff5640 10 client.26024 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:00:02.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.176+0000 7f2686ff5640 1 -- 172.21.15.169:0/817214393 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563c76001900 con 0x563c760a5150 2024-03-28T01:00:02.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.176+0000 7f2686ff5640 10 client.26024 handle_command_reply: tid=0 2024-03-28T01:00:02.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.176+0000 7f26ac372740 1 client.26024 shutdown 2024-03-28T01:00:02.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.182+0000 7f26ac372740 1 -- 172.21.15.169:0/817214393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c7608f080 msgr2=0x563c76098cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:02.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.182+0000 7f26ac372740 1 --2- 172.21.15.169:0/817214393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c7608f080 0x563c76098cd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2698009900 tx=0x7f2698004610 comp rx=0 tx=0).stop 2024-03-28T01:00:02.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.186+0000 7f26ac372740 1 -- 172.21.15.169:0/817214393 shutdown_connections 2024-03-28T01:00:02.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.186+0000 7f26ac372740 1 --2- 172.21.15.169:0/817214393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c7609e240 0x563c760a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.186+0000 7f26ac372740 1 --2- 172.21.15.169:0/817214393 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x563c760a5150 0x563c760a7570 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.186+0000 7f26ac372740 1 --2- 172.21.15.169:0/817214393 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c760941f0 0x563c76099210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.186+0000 7f26ac372740 1 --2- 172.21.15.169:0/817214393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c7608f080 0x563c76098cd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.186+0000 7f26ac372740 1 -- 172.21.15.169:0/817214393 >> 172.21.15.169:0/817214393 conn(0x563c76089c00 msgr2=0x563c76088e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:02.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.186+0000 7f2686ff5640 0 client.26024 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:00:02.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.188+0000 7f26ac372740 1 -- 172.21.15.169:0/817214393 shutdown_connections 2024-03-28T01:00:02.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.188+0000 7f26ac372740 1 -- 172.21.15.169:0/817214393 wait complete. 2024-03-28T01:00:02.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.189+0000 7f26ac372740 20 client.26024 trim_cache size 0 max 16384 2024-03-28T01:00:02.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.219+0000 7f26ac372740 1 Processor -- start 2024-03-28T01:00:02.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.225+0000 7f26ac372740 1 -- start start 2024-03-28T01:00:02.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.225+0000 7f26ac372740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c7608f080 0x563c7608fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.225+0000 7f26ac372740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c760941f0 0x563c76090760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.225+0000 7f26ac372740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c7609e240 0x563c76090ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.225+0000 7f26ac372740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26900120d0 con 0x563c7609e240 2024-03-28T01:00:02.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.225+0000 7f26ac372740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269000fbd0 con 0x563c7608f080 2024-03-28T01:00:02.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.225+0000 7f26ac372740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269000dc70 con 0x563c760941f0 2024-03-28T01:00:02.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.225+0000 7f26a2ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c7608f080 0x563c7608fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.225+0000 7f26a2ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c7608f080 0x563c7608fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:41296/0 (socket says 172.21.15.169:41296) 2024-03-28T01:00:02.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.226+0000 7f26a27fc640 1 --2- 172.21.15.169:0/459503228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c760941f0 0x563c76090760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.226+0000 7f26a37fe640 1 --2- 172.21.15.169:0/459503228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c7609e240 0x563c76090ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.226+0000 7f26a2ffd640 1 -- 172.21.15.169:0/459503228 learned_addr learned my addr 172.21.15.169:0/459503228 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:02.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.230+0000 7f26a37fe640 1 -- 172.21.15.169:0/459503228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c760941f0 msgr2=0x563c76090760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:02.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.230+0000 7f26a37fe640 1 --2- 172.21.15.169:0/459503228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c760941f0 0x563c76090760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.230+0000 7f26a37fe640 1 -- 172.21.15.169:0/459503228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c7608f080 msgr2=0x563c7608fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:02.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.230+0000 7f26a37fe640 1 --2- 172.21.15.169:0/459503228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c7608f080 0x563c7608fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.230+0000 7f26a37fe640 1 -- 172.21.15.169:0/459503228 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c76093a30 con 0x563c7609e240 2024-03-28T01:00:02.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.234+0000 7f26a37fe640 1 --2- 172.21.15.169:0/459503228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c7609e240 0x563c76090ca0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2694005b00 tx=0x7f269400cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21ae0fff4e80c28e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:02.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.237+0000 7f2688ff9640 1 -- 172.21.15.169:0/459503228 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269400ec90 con 0x563c7609e240 2024-03-28T01:00:02.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.244+0000 7f2688ff9640 1 -- 172.21.15.169:0/459503228 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f269400ee30 con 0x563c7609e240 2024-03-28T01:00:02.249 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.246+0000 7f26ac372740 1 -- 172.21.15.169:0/459503228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c7609e240 msgr2=0x563c76090ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:02.249 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.246+0000 7f26ac372740 1 --2- 172.21.15.169:0/459503228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c7609e240 0x563c76090ca0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2694005b00 tx=0x7f269400cd20 comp rx=0 tx=0).stop 2024-03-28T01:00:02.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.250+0000 7f26ac372740 1 -- 172.21.15.169:0/459503228 shutdown_connections 2024-03-28T01:00:02.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.251+0000 7f26ac372740 1 --2- 172.21.15.169:0/459503228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c7609e240 0x563c76090ca0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.251+0000 7f26ac372740 1 --2- 172.21.15.169:0/459503228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c760941f0 0x563c76090760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.251+0000 7f26ac372740 1 --2- 172.21.15.169:0/459503228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c7608f080 0x563c7608fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.251+0000 7f26ac372740 1 -- 172.21.15.169:0/459503228 >> 172.21.15.169:0/459503228 conn(0x563c76089c00 msgr2=0x563c76084cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:02.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.255+0000 7f26ac372740 1 -- 172.21.15.169:0/459503228 shutdown_connections 2024-03-28T01:00:02.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.256+0000 7f26ac372740 1 -- 172.21.15.169:0/459503228 wait complete. 2024-03-28T01:00:02.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.257+0000 7f26ac372740 1 -- start start 2024-03-28T01:00:02.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.257+0000 7f26ac372740 1 Processor -- start 2024-03-28T01:00:02.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.257+0000 7f26ac372740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c7608f080 0x563c760b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.257+0000 7f26ac372740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c760941f0 0x563c760b57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.258+0000 7f26ac372740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c7609e240 0x563c760b67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.258+0000 7f26ac372740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269000e270 con 0x563c760941f0 2024-03-28T01:00:02.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.258+0000 7f26ac372740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269000f570 con 0x563c7609e240 2024-03-28T01:00:02.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.258+0000 7f26ac372740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269000e8b0 con 0x563c7608f080 2024-03-28T01:00:02.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.258+0000 7f26ac372740 10 client.0 _ll_register_callbacks cb 0x7f269802cc90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:02.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.258+0000 7f26a2ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c7608f080 0x563c760b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.258+0000 7f26a2ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c7608f080 0x563c760b5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:45934/0 (socket says 172.21.15.169:45934) 2024-03-28T01:00:02.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.258+0000 7f26a2ffd640 1 -- 172.21.15.169:0/2381486096 learned_addr learned my addr 172.21.15.169:0/2381486096 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:02.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.261+0000 7f26a37fe640 1 --2- 172.21.15.169:0/2381486096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c7609e240 0x563c760b67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.261+0000 7f26a27fc640 1 --2- 172.21.15.169:0/2381486096 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c760941f0 0x563c760b57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.264+0000 7f26a2ffd640 1 -- 172.21.15.169:0/2381486096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c7609e240 msgr2=0x563c760b67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:02.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.264+0000 7f26a2ffd640 1 --2- 172.21.15.169:0/2381486096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c7609e240 0x563c760b67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.264+0000 7f26a2ffd640 1 -- 172.21.15.169:0/2381486096 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c760941f0 msgr2=0x563c760b57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:02.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.264+0000 7f26a2ffd640 1 --2- 172.21.15.169:0/2381486096 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c760941f0 0x563c760b57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.264+0000 7f26a2ffd640 1 -- 172.21.15.169:0/2381486096 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c760b6db0 con 0x563c7608f080 2024-03-28T01:00:02.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.264+0000 7f26a37fe640 1 --2- 172.21.15.169:0/2381486096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c7609e240 0x563c760b67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:00:02.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.266+0000 7f26a2ffd640 1 --2- 172.21.15.169:0/2381486096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c7608f080 0x563c760b5270 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f269802c7b0 tx=0x7f2698034c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81d3cf28ea3debeb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:02.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.267+0000 7f2686ff5640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:00:02.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.270+0000 7f2686ff5640 1 -- 172.21.15.169:0/2381486096 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2698050040 con 0x563c7608f080 2024-03-28T01:00:02.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.270+0000 7f26ac372740 1 -- 172.21.15.169:0/2381486096 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f269400ee30 con 0x563c7608f080 2024-03-28T01:00:02.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.271+0000 7f2686ff5640 1 -- 172.21.15.169:0/2381486096 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f269804b450 con 0x563c7608f080 2024-03-28T01:00:02.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.272+0000 7f2686ff5640 1 -- 172.21.15.169:0/2381486096 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f269806c050 con 0x563c7608f080 2024-03-28T01:00:02.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.272+0000 7f26ac372740 10 client.25999 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:02.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.272+0000 7f26ac372740 1 -- 172.21.15.169:0/2381486096 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563c76100330 con 0x563c7608f080 2024-03-28T01:00:02.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.274+0000 7f2686ff5640 1 -- 172.21.15.169:0/2381486096 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f269400ee30 con 0x563c7608f080 2024-03-28T01:00:02.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.274+0000 7f2686ff5640 1 -- 172.21.15.169:0/2381486096 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2698053040 con 0x563c7608f080 2024-03-28T01:00:02.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.274+0000 7f26ac372740 10 client.25999 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:02.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.274+0000 7f26ac372740 10 client.25999 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:00:02.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.274+0000 7f26ac372740 20 client.25999 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:02.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.274+0000 7f26ac372740 1 --2- 172.21.15.169:0/2381486096 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x563c76100ce0 0x563c76103100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.274+0000 7f26ac372740 4 client.25999 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:00:02.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.274+0000 7f26ac372740 1 -- 172.21.15.169:0/2381486096 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563c75ffbc50 con 0x563c76100ce0 2024-03-28T01:00:02.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.275+0000 7f26a27fc640 1 --2- 172.21.15.169:0/2381486096 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x563c76100ce0 0x563c76103100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.277+0000 7f26a27fc640 1 --2- 172.21.15.169:0/2381486096 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x563c76100ce0 0x563c76103100 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=58d46ba3607d297a server_cookie=26d7dc373016780a in_seq=0 out_seq=0 2024-03-28T01:00:02.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.278+0000 7f2686ff5640 10 client.25999 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:00:02.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.278+0000 7f2686ff5640 1 -- 172.21.15.169:0/2381486096 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563c75ffbc50 con 0x563c76100ce0 2024-03-28T01:00:02.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.278+0000 7f2686ff5640 10 client.25999 handle_command_reply: tid=0 2024-03-28T01:00:02.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.278+0000 7f26ac372740 1 client.25999 shutdown 2024-03-28T01:00:02.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.278+0000 7f26ac372740 1 -- 172.21.15.169:0/2381486096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c7608f080 msgr2=0x563c760b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:02.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.278+0000 7f26ac372740 1 --2- 172.21.15.169:0/2381486096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c7608f080 0x563c760b5270 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f269802c7b0 tx=0x7f2698034c30 comp rx=0 tx=0).stop 2024-03-28T01:00:02.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.278+0000 7f26ac372740 1 -- 172.21.15.169:0/2381486096 shutdown_connections 2024-03-28T01:00:02.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.278+0000 7f26ac372740 1 --2- 172.21.15.169:0/2381486096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563c7609e240 0x563c760b67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.278+0000 7f26ac372740 1 --2- 172.21.15.169:0/2381486096 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x563c76100ce0 0x563c76103100 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.278+0000 7f26ac372740 1 --2- 172.21.15.169:0/2381486096 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563c760941f0 0x563c760b57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.278+0000 7f26ac372740 1 --2- 172.21.15.169:0/2381486096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c7608f080 0x563c760b5270 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.278+0000 7f26ac372740 1 -- 172.21.15.169:0/2381486096 >> 172.21.15.169:0/2381486096 conn(0x563c76089c00 msgr2=0x563c76084cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:02.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.278+0000 7f2686ff5640 0 client.25999 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:00:02.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.279+0000 7f26ac372740 1 -- 172.21.15.169:0/2381486096 shutdown_connections 2024-03-28T01:00:02.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.279+0000 7f26ac372740 1 -- 172.21.15.169:0/2381486096 wait complete. 2024-03-28T01:00:02.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.279+0000 7f26ac372740 20 client.25999 trim_cache size 0 max 16384 2024-03-28T01:00:02.283 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:00:02.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.281+0000 7f26a96ac640 1 -- 172.21.15.169:0/4096976641 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f268c0a2a00 msgr2=0x7f268c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:02.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.282+0000 7f26a96ac640 1 --2- 172.21.15.169:0/4096976641 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f268c0a2a00 0x7f268c0a4ec0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2698005ab0 tx=0x7f269802e000 comp rx=0 tx=0).stop 2024-03-28T01:00:02.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.282+0000 7f26a96ac640 1 -- 172.21.15.169:0/4096976641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26a4108eb0 msgr2=0x7f26a4073170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:02.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.282+0000 7f26a96ac640 1 --2- 172.21.15.169:0/4096976641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26a4108eb0 0x7f26a4073170 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f269c00cce0 tx=0x7f269c00a620 comp rx=0 tx=0).stop 2024-03-28T01:00:02.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.283+0000 7f26a96ac640 1 -- 172.21.15.169:0/4096976641 shutdown_connections 2024-03-28T01:00:02.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.284+0000 7f26a96ac640 1 --2- 172.21.15.169:0/4096976641 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f268c0a2a00 0x7f268c0a4ec0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.284+0000 7f26a96ac640 1 --2- 172.21.15.169:0/4096976641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26a4109870 0x7f26a40712b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.284+0000 7f26a96ac640 1 --2- 172.21.15.169:0/4096976641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26a4108eb0 0x7f26a4073170 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.284+0000 7f26a96ac640 1 --2- 172.21.15.169:0/4096976641 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26a4107cc0 0x7f26a4072c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.284+0000 7f26a96ac640 1 -- 172.21.15.169:0/4096976641 >> 172.21.15.169:0/4096976641 conn(0x7f26a4076c60 msgr2=0x7f26a410c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:02.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.286+0000 7f26a96ac640 1 -- 172.21.15.169:0/4096976641 shutdown_connections 2024-03-28T01:00:02.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.287+0000 7f26a96ac640 1 -- 172.21.15.169:0/4096976641 wait complete. 2024-03-28T01:00:02.336 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:00:02.336 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T01:00:02.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.644+0000 7f5d6ef47640 1 -- 172.21.15.169:0/124223712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5d68109690 msgr2=0x7f5d6810ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:02.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.644+0000 7f5d6ef47640 1 --2- 172.21.15.169:0/124223712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5d68109690 0x7f5d6810ff20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5d64009900 tx=0x7f5d6402bc30 comp rx=0 tx=0).stop 2024-03-28T01:00:02.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.644+0000 7f5d6ef47640 1 -- 172.21.15.169:0/124223712 shutdown_connections 2024-03-28T01:00:02.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.644+0000 7f5d6ef47640 1 --2- 172.21.15.169:0/124223712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5d68109690 0x7f5d6810ff20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.644+0000 7f5d6ef47640 1 --2- 172.21.15.169:0/124223712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d68108cd0 0x7f5d68109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.644+0000 7f5d6ef47640 1 --2- 172.21.15.169:0/124223712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d68107ae0 0x7f5d68107f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.644+0000 7f5d6ef47640 1 -- 172.21.15.169:0/124223712 >> 172.21.15.169:0/124223712 conn(0x7f5d68076a80 msgr2=0x7f5d68078ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:02.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.647+0000 7f5d6ef47640 1 -- 172.21.15.169:0/124223712 shutdown_connections 2024-03-28T01:00:02.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.647+0000 7f5d6ef47640 1 -- 172.21.15.169:0/124223712 wait complete. 2024-03-28T01:00:02.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.648+0000 7f5d6ef47640 1 Processor -- start 2024-03-28T01:00:02.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.648+0000 7f5d6ef47640 1 -- start start 2024-03-28T01:00:02.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.648+0000 7f5d6ef47640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5d68107ae0 0x7f5d68070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.648+0000 7f5d6ef47640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d68108cd0 0x7f5d68071500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.648+0000 7f5d6ef47640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d68109690 0x7f5d68072540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.648+0000 7f5d6ef47640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6806b6d0 con 0x7f5d68108cd0 2024-03-28T01:00:02.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.648+0000 7f5d6ef47640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6806b550 con 0x7f5d68107ae0 2024-03-28T01:00:02.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.648+0000 7f5d6ef47640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6806b850 con 0x7f5d68109690 2024-03-28T01:00:02.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.649+0000 7f5d6df45640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5d68107ae0 0x7f5d68070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.649+0000 7f5d6e746640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d68109690 0x7f5d68072540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.649+0000 7f5d6e746640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d68109690 0x7f5d68072540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:45940/0 (socket says 172.21.15.169:45940) 2024-03-28T01:00:02.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.649+0000 7f5d6e746640 1 -- 172.21.15.169:0/2425744091 learned_addr learned my addr 172.21.15.169:0/2425744091 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:02.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.649+0000 7f5d6e746640 1 -- 172.21.15.169:0/2425744091 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5d68107ae0 msgr2=0x7f5d68070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:02.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.649+0000 7f5d6e746640 1 --2- 172.21.15.169:0/2425744091 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5d68107ae0 0x7f5d68070fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.649+0000 7f5d6e746640 1 -- 172.21.15.169:0/2425744091 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d68108cd0 msgr2=0x7f5d68071500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:02.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.649+0000 7f5d6e746640 1 --2- 172.21.15.169:0/2425744091 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d68108cd0 0x7f5d68071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.649+0000 7f5d6e746640 1 -- 172.21.15.169:0/2425744091 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d68072bb0 con 0x7f5d68109690 2024-03-28T01:00:02.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.650+0000 7f5d6e746640 1 --2- 172.21.15.169:0/2425744091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d68109690 0x7f5d68072540 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5d64006010 tx=0x7f5d64004e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:02.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.650+0000 7f5d5effd640 1 -- 172.21.15.169:0/2425744091 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d64041070 con 0x7f5d68109690 2024-03-28T01:00:02.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.650+0000 7f5d5effd640 1 -- 172.21.15.169:0/2425744091 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5d64005410 con 0x7f5d68109690 2024-03-28T01:00:02.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.650+0000 7f5d5effd640 1 -- 172.21.15.169:0/2425744091 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d64034860 con 0x7f5d68109690 2024-03-28T01:00:02.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.650+0000 7f5d6ef47640 1 -- 172.21.15.169:0/2425744091 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d68072e40 con 0x7f5d68109690 2024-03-28T01:00:02.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.650+0000 7f5d6ef47640 1 -- 172.21.15.169:0/2425744091 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d680732e0 con 0x7f5d68109690 2024-03-28T01:00:02.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.656+0000 7f5d72d90740 1 Processor -- start 2024-03-28T01:00:02.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.658+0000 7f5d72d90740 1 -- start start 2024-03-28T01:00:02.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.658+0000 7f5d72d90740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f0bd428a0 0x564f0bd42cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.658+0000 7f5d72d90740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564f0bd43290 0x564f0bd47d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.658+0000 7f5d72d90740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd48400 0x564f0bd4a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.658+0000 7f5d72d90740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f0bd05610 con 0x564f0bd43290 2024-03-28T01:00:02.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.658+0000 7f5d72d90740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f0bca83c0 con 0x564f0bd48400 2024-03-28T01:00:02.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.658+0000 7f5d72d90740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f0bc50d60 con 0x564f0bd428a0 2024-03-28T01:00:02.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.660+0000 7f5d5effd640 1 -- 172.21.15.169:0/2425744091 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5d64044070 con 0x7f5d68109690 2024-03-28T01:00:02.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.660+0000 7f5d6df45640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564f0bd43290 0x564f0bd47d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.660+0000 7f5d6df45640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564f0bd43290 0x564f0bd47d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47212/0 (socket says 172.21.15.169:47212) 2024-03-28T01:00:02.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.660+0000 7f5d6df45640 1 -- 172.21.15.169:0/1340381942 learned_addr learned my addr 172.21.15.169:0/1340381942 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:02.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.661+0000 7f5d5effd640 1 --2- 172.21.15.169:0/2425744091 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5d540a2a10 0x7f5d540a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.661+0000 7f5d5effd640 1 -- 172.21.15.169:0/2425744091 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f5d640f0900 con 0x7f5d68109690 2024-03-28T01:00:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.662+0000 7f5d6df45640 1 -- 172.21.15.169:0/1340381942 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f0bd428a0 msgr2=0x564f0bd42cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:00:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.662+0000 7f5d6df45640 1 --2- 172.21.15.169:0/1340381942 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f0bd428a0 0x564f0bd42cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.662+0000 7f5d6df45640 1 -- 172.21.15.169:0/1340381942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd48400 msgr2=0x564f0bd4a890 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:00:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.662+0000 7f5d6df45640 1 --2- 172.21.15.169:0/1340381942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd48400 0x564f0bd4a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.662+0000 7f5d6df45640 1 -- 172.21.15.169:0/1340381942 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564f0b9bc3f0 con 0x564f0bd43290 2024-03-28T01:00:02.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.663+0000 7f5d6e746640 1 --2- 172.21.15.169:0/2425744091 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5d540a2a10 0x7f5d540a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.663+0000 7f5d6df45640 1 --2- 172.21.15.169:0/1340381942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564f0bd43290 0x564f0bd47d90 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5d60011470 tx=0x7f5d60011930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=805a4a114e026b1d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:02.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:02 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1819539336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:00:02.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.664+0000 7f5d53fff640 1 -- 172.21.15.169:0/1340381942 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6001e070 con 0x564f0bd43290 2024-03-28T01:00:02.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.664+0000 7f5d6e746640 1 --2- 172.21.15.169:0/2425744091 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5d540a2a10 0x7f5d540a4ed0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5d640c5110 tx=0x7f5d6402c6d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:02.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.664+0000 7f5d53fff640 1 -- 172.21.15.169:0/1340381942 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5d60004030 con 0x564f0bd43290 2024-03-28T01:00:02.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.665+0000 7f5d72d90740 1 -- 172.21.15.169:0/1340381942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564f0bd43290 msgr2=0x564f0bd47d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:02.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.665+0000 7f5d72d90740 1 --2- 172.21.15.169:0/1340381942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564f0bd43290 0x564f0bd47d90 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5d60011470 tx=0x7f5d60011930 comp rx=0 tx=0).stop 2024-03-28T01:00:02.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.667+0000 7f5d72d90740 1 -- 172.21.15.169:0/1340381942 shutdown_connections 2024-03-28T01:00:02.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.667+0000 7f5d72d90740 1 --2- 172.21.15.169:0/1340381942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd48400 0x564f0bd4a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.667+0000 7f5d72d90740 1 --2- 172.21.15.169:0/1340381942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564f0bd43290 0x564f0bd47d90 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.667+0000 7f5d72d90740 1 --2- 172.21.15.169:0/1340381942 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f0bd428a0 0x564f0bd42cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.667+0000 7f5d72d90740 1 -- 172.21.15.169:0/1340381942 >> 172.21.15.169:0/1340381942 conn(0x564f0bd3de10 msgr2=0x564f0bd3e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:02.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.672+0000 7f5d72d90740 1 -- 172.21.15.169:0/1340381942 shutdown_connections 2024-03-28T01:00:02.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.675+0000 7f5d72d90740 1 -- 172.21.15.169:0/1340381942 wait complete. 2024-03-28T01:00:02.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.676+0000 7f5d72d90740 1 -- start start 2024-03-28T01:00:02.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.676+0000 7f5d72d90740 1 Processor -- start 2024-03-28T01:00:02.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.676+0000 7f5d72d90740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f0bd428a0 0x564f0bd4cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.676+0000 7f5d72d90740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd43290 0x564f0bd4d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.676+0000 7f5d72d90740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564f0bd48400 0x564f0bd54220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.676+0000 7f5d72d90740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f0bc64240 con 0x564f0bd48400 2024-03-28T01:00:02.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.676+0000 7f5d72d90740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f0bca4460 con 0x564f0bd43290 2024-03-28T01:00:02.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.676+0000 7f5d72d90740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f0bd4b400 con 0x564f0bd428a0 2024-03-28T01:00:02.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.676+0000 7f5d72d90740 10 client.0 _ll_register_callbacks cb 0x564f0bc63c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:02.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.677+0000 7f5d6df45640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd43290 0x564f0bd4d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.682+0000 7f5d6df45640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd43290 0x564f0bd4d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:41360/0 (socket says 172.21.15.169:41360) 2024-03-28T01:00:02.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.682+0000 7f5d6df45640 1 -- 172.21.15.169:0/3267596728 learned_addr learned my addr 172.21.15.169:0/3267596728 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:02.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.685+0000 7f5d6df45640 1 -- 172.21.15.169:0/3267596728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f0bd428a0 msgr2=0x564f0bd4cca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:00:02.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.685+0000 7f5d6df45640 1 --2- 172.21.15.169:0/3267596728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f0bd428a0 0x564f0bd4cca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.685+0000 7f5d6df45640 1 -- 172.21.15.169:0/3267596728 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564f0bd48400 msgr2=0x564f0bd54220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:00:02.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.685+0000 7f5d6df45640 1 --2- 172.21.15.169:0/3267596728 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564f0bd48400 0x564f0bd54220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.685+0000 7f5d6df45640 1 -- 172.21.15.169:0/3267596728 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564f0ba867d0 con 0x564f0bd43290 2024-03-28T01:00:02.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.686+0000 7f5d6df45640 1 --2- 172.21.15.169:0/3267596728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd43290 0x564f0bd4d1e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5d6001c040 tx=0x7f5d60004410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8034f4d2289889c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:02.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.686+0000 7f5d517fa640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:00:02.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.686+0000 7f5d517fa640 1 -- 172.21.15.169:0/3267596728 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6001e040 con 0x564f0bd43290 2024-03-28T01:00:02.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.686+0000 7f5d517fa640 1 -- 172.21.15.169:0/3267596728 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f5d6000fa50 con 0x564f0bd43290 2024-03-28T01:00:02.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.687+0000 7f5d72d90740 1 -- 172.21.15.169:0/3267596728 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5d60004030 con 0x564f0bd43290 2024-03-28T01:00:02.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.687+0000 7f5d517fa640 1 -- 172.21.15.169:0/3267596728 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d60009c00 con 0x564f0bd43290 2024-03-28T01:00:02.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.687+0000 7f5d72d90740 10 client.26039 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:02.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.687+0000 7f5d72d90740 1 -- 172.21.15.169:0/3267596728 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x564f0bd685d0 con 0x564f0bd43290 2024-03-28T01:00:02.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.687+0000 7f5d517fa640 1 -- 172.21.15.169:0/3267596728 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5d60004030 con 0x564f0bd43290 2024-03-28T01:00:02.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.688+0000 7f5d517fa640 1 -- 172.21.15.169:0/3267596728 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f5d60021030 con 0x564f0bd43290 2024-03-28T01:00:02.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.688+0000 7f5d72d90740 10 client.26039 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:02.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.688+0000 7f5d72d90740 10 client.26039 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:00:02.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.688+0000 7f5d72d90740 20 client.26039 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:02.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.688+0000 7f5d72d90740 1 --2- 172.21.15.169:0/3267596728 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x564f0bd69a50 0x564f0bd6bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.688+0000 7f5d72d90740 4 client.26039 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:00:02.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.688+0000 7f5d72d90740 1 -- 172.21.15.169:0/3267596728 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564f0bcb5900 con 0x564f0bd69a50 2024-03-28T01:00:02.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.727+0000 7f5d6d744640 1 --2- 172.21.15.169:0/3267596728 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x564f0bd69a50 0x564f0bd6bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.744+0000 7f5d6d744640 1 --2- 172.21.15.169:0/3267596728 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x564f0bd69a50 0x564f0bd6bee0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a63055a045cc57e8 server_cookie=2c0d646716406b42 in_seq=0 out_seq=0 2024-03-28T01:00:02.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.744+0000 7f5d517fa640 10 client.26039 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:00:02.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.753+0000 7f5d517fa640 1 -- 172.21.15.169:0/3267596728 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564f0bcb5900 con 0x564f0bd69a50 2024-03-28T01:00:02.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.753+0000 7f5d517fa640 10 client.26039 handle_command_reply: tid=0 2024-03-28T01:00:02.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.753+0000 7f5d72d90740 1 client.26039 shutdown 2024-03-28T01:00:02.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.753+0000 7f5d72d90740 1 -- 172.21.15.169:0/3267596728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd43290 msgr2=0x564f0bd4d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:02.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.753+0000 7f5d72d90740 1 --2- 172.21.15.169:0/3267596728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd43290 0x564f0bd4d1e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5d6001c040 tx=0x7f5d60004410 comp rx=0 tx=0).stop 2024-03-28T01:00:02.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.754+0000 7f5d72d90740 1 -- 172.21.15.169:0/3267596728 shutdown_connections 2024-03-28T01:00:02.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.754+0000 7f5d72d90740 1 --2- 172.21.15.169:0/3267596728 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x564f0bd69a50 0x564f0bd6bee0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.754+0000 7f5d72d90740 1 --2- 172.21.15.169:0/3267596728 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564f0bd48400 0x564f0bd54220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.754+0000 7f5d72d90740 1 --2- 172.21.15.169:0/3267596728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd43290 0x564f0bd4d1e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.754+0000 7f5d72d90740 1 --2- 172.21.15.169:0/3267596728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f0bd428a0 0x564f0bd4cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.754+0000 7f5d72d90740 1 -- 172.21.15.169:0/3267596728 >> 172.21.15.169:0/3267596728 conn(0x564f0bd3de10 msgr2=0x564f0bd3d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:02.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.755+0000 7f5d517fa640 0 client.26039 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:00:02.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.755+0000 7f5d72d90740 1 -- 172.21.15.169:0/3267596728 shutdown_connections 2024-03-28T01:00:02.761 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.756+0000 7f5d72d90740 1 -- 172.21.15.169:0/3267596728 wait complete. 2024-03-28T01:00:02.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.761+0000 7f5d72d90740 20 client.26039 trim_cache size 0 max 16384 2024-03-28T01:00:02.812 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:02 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1819539336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:00:02.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.811+0000 7f5d72d90740 1 Processor -- start 2024-03-28T01:00:02.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.813+0000 7f5d72d90740 1 -- start start 2024-03-28T01:00:02.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.814+0000 7f5d72d90740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd428a0 0x564f0bd41780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.814+0000 7f5d72d90740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564f0bd43290 0x564f0bd541e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.815+0000 7f5d72d90740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f0bd48400 0x564f0bd707e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.815+0000 7f5d72d90740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c0120f0 con 0x564f0bd43290 2024-03-28T01:00:02.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.815+0000 7f5d72d90740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c00fbf0 con 0x564f0bd428a0 2024-03-28T01:00:02.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.815+0000 7f5d72d90740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c00dc90 con 0x564f0bd48400 2024-03-28T01:00:02.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.816+0000 7f5d6d744640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f0bd48400 0x564f0bd707e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.817+0000 7f5d6d744640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f0bd48400 0x564f0bd707e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:45968/0 (socket says 172.21.15.169:45968) 2024-03-28T01:00:02.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.817+0000 7f5d6d744640 1 -- 172.21.15.169:0/3095501190 learned_addr learned my addr 172.21.15.169:0/3095501190 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:02.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.818+0000 7f5d6d744640 1 --2- 172.21.15.169:0/3095501190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd428a0 0x564f0bd41780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.818+0000 7f5d6d744640 1 -- 172.21.15.169:0/3095501190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd428a0 msgr2=0x564f0bd41780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:02.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.818+0000 7f5d6d744640 1 --2- 172.21.15.169:0/3095501190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd428a0 0x564f0bd41780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.818+0000 7f5d6d744640 1 -- 172.21.15.169:0/3095501190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564f0bd43290 msgr2=0x564f0bd541e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:02.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.818+0000 7f5d6d744640 1 --2- 172.21.15.169:0/3095501190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564f0bd43290 0x564f0bd541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.819+0000 7f5d6d744640 1 -- 172.21.15.169:0/3095501190 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564f0bd6c560 con 0x564f0bd48400 2024-03-28T01:00:02.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.822+0000 7f5d6d744640 1 --2- 172.21.15.169:0/3095501190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f0bd48400 0x564f0bd707e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5d5800b6c0 tx=0x7f5d58005ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b7d95b1f5e9225e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:02.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.826+0000 7f5d537fe640 1 -- 172.21.15.169:0/3095501190 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d58017c30 con 0x564f0bd48400 2024-03-28T01:00:02.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.826+0000 7f5d537fe640 1 -- 172.21.15.169:0/3095501190 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5d58005500 con 0x564f0bd48400 2024-03-28T01:00:02.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.826+0000 7f5d537fe640 1 -- 172.21.15.169:0/3095501190 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d58010690 con 0x564f0bd48400 2024-03-28T01:00:02.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.828+0000 7f5d72d90740 1 -- 172.21.15.169:0/3095501190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f0bd48400 msgr2=0x564f0bd707e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:02.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.828+0000 7f5d72d90740 1 --2- 172.21.15.169:0/3095501190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f0bd48400 0x564f0bd707e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5d5800b6c0 tx=0x7f5d58005ce0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.837+0000 7f5d72d90740 1 -- 172.21.15.169:0/3095501190 shutdown_connections 2024-03-28T01:00:02.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.837+0000 7f5d72d90740 1 --2- 172.21.15.169:0/3095501190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f0bd48400 0x564f0bd707e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.837+0000 7f5d72d90740 1 --2- 172.21.15.169:0/3095501190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564f0bd43290 0x564f0bd541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.837+0000 7f5d72d90740 1 --2- 172.21.15.169:0/3095501190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd428a0 0x564f0bd41780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.837+0000 7f5d72d90740 1 -- 172.21.15.169:0/3095501190 >> 172.21.15.169:0/3095501190 conn(0x564f0bd3de10 msgr2=0x564f0bd41070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:02.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.840+0000 7f5d72d90740 1 -- 172.21.15.169:0/3095501190 shutdown_connections 2024-03-28T01:00:02.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.841+0000 7f5d72d90740 1 -- 172.21.15.169:0/3095501190 wait complete. 2024-03-28T01:00:02.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.841+0000 7f5d72d90740 1 -- start start 2024-03-28T01:00:02.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.842+0000 7f5d72d90740 1 Processor -- start 2024-03-28T01:00:02.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.842+0000 7f5d72d90740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f0bd428a0 0x564f0bd51c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.843+0000 7f5d72d90740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564f0bd43290 0x564f0bd521d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.843+0000 7f5d72d90740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd48400 0x564f0bd53190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.843+0000 7f5d72d90740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c00e290 con 0x564f0bd43290 2024-03-28T01:00:02.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.843+0000 7f5d72d90740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c00f590 con 0x564f0bd48400 2024-03-28T01:00:02.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.843+0000 7f5d72d90740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c00e8d0 con 0x564f0bd428a0 2024-03-28T01:00:02.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.843+0000 7f5d72d90740 10 client.0 _ll_register_callbacks cb 0x7f5d60003dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:02.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.844+0000 7f5d6d744640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd48400 0x564f0bd53190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.844+0000 7f5d6d744640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd48400 0x564f0bd53190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:41374/0 (socket says 172.21.15.169:41374) 2024-03-28T01:00:02.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.844+0000 7f5d6d744640 1 -- 172.21.15.169:0/1885719447 learned_addr learned my addr 172.21.15.169:0/1885719447 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:02.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.844+0000 7f5d6d744640 1 --2- 172.21.15.169:0/1885719447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f0bd428a0 0x564f0bd51c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.844+0000 7f5d6df45640 1 --2- 172.21.15.169:0/1885719447 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564f0bd43290 0x564f0bd521d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.850 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.847+0000 7f5d6df45640 1 -- 172.21.15.169:0/1885719447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f0bd428a0 msgr2=0x564f0bd51c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:02.850 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.847+0000 7f5d6df45640 1 --2- 172.21.15.169:0/1885719447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f0bd428a0 0x564f0bd51c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.850 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.847+0000 7f5d6df45640 1 -- 172.21.15.169:0/1885719447 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd48400 msgr2=0x564f0bd53190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:02.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.847+0000 7f5d6df45640 1 --2- 172.21.15.169:0/1885719447 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd48400 0x564f0bd53190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.847+0000 7f5d6df45640 1 -- 172.21.15.169:0/1885719447 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564f0bd3d9b0 con 0x564f0bd43290 2024-03-28T01:00:02.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.848+0000 7f5d6df45640 1 --2- 172.21.15.169:0/1885719447 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564f0bd43290 0x564f0bd521d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x564f0bca44d0 tx=0x7f5d600043f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74f088f5639ef7a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:02.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.848+0000 7f5d517fa640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:00:02.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.852+0000 7f5d517fa640 1 -- 172.21.15.169:0/1885719447 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d60016060 con 0x564f0bd43290 2024-03-28T01:00:02.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.852+0000 7f5d72d90740 1 -- 172.21.15.169:0/1885719447 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5d58005500 con 0x564f0bd43290 2024-03-28T01:00:02.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.852+0000 7f5d517fa640 1 -- 172.21.15.169:0/1885719447 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f5d6000fbd0 con 0x564f0bd43290 2024-03-28T01:00:02.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.853+0000 7f5d517fa640 1 -- 172.21.15.169:0/1885719447 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d6001e040 con 0x564f0bd43290 2024-03-28T01:00:02.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.854+0000 7f5d72d90740 10 client.16203 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:02.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.854+0000 7f5d72d90740 1 -- 172.21.15.169:0/1885719447 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x564f0bdc5530 con 0x564f0bd43290 2024-03-28T01:00:02.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:02 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1819539336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:00:02.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.856+0000 7f5d517fa640 1 -- 172.21.15.169:0/1885719447 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5d58005500 con 0x564f0bd43290 2024-03-28T01:00:02.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.856+0000 7f5d517fa640 1 -- 172.21.15.169:0/1885719447 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f5d60036030 con 0x564f0bd43290 2024-03-28T01:00:02.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.856+0000 7f5d72d90740 10 client.16203 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:02.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.856+0000 7f5d72d90740 10 client.16203 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:00:02.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.856+0000 7f5d72d90740 20 client.16203 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:02.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.856+0000 7f5d72d90740 1 --2- 172.21.15.169:0/1885719447 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x564f0bdc5e00 0x564f0bdc8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:02.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.856+0000 7f5d72d90740 4 client.16203 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:00:02.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.856+0000 7f5d72d90740 1 -- 172.21.15.169:0/1885719447 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564f0bd44200 con 0x564f0bdc5e00 2024-03-28T01:00:02.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.857+0000 7f5d6d744640 1 --2- 172.21.15.169:0/1885719447 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x564f0bdc5e00 0x564f0bdc8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:02.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.861+0000 7f5d517fa640 10 client.16203 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:00:02.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.861+0000 7f5d6d744640 1 --2- 172.21.15.169:0/1885719447 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x564f0bdc5e00 0x564f0bdc8220 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7192d052f50c6dbc server_cookie=4118f221951e6b8e in_seq=0 out_seq=0 2024-03-28T01:00:02.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.865+0000 7f5d517fa640 1 -- 172.21.15.169:0/1885719447 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564f0bd44200 con 0x564f0bdc5e00 2024-03-28T01:00:02.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.866+0000 7f5d517fa640 10 client.16203 handle_command_reply: tid=0 2024-03-28T01:00:02.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.866+0000 7f5d72d90740 1 client.16203 shutdown 2024-03-28T01:00:02.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.870+0000 7f5d72d90740 1 -- 172.21.15.169:0/1885719447 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564f0bd43290 msgr2=0x564f0bd521d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:02.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.870+0000 7f5d72d90740 1 --2- 172.21.15.169:0/1885719447 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564f0bd43290 0x564f0bd521d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x564f0bca44d0 tx=0x7f5d600043f0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.870+0000 7f5d72d90740 1 -- 172.21.15.169:0/1885719447 shutdown_connections 2024-03-28T01:00:02.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.870+0000 7f5d72d90740 1 --2- 172.21.15.169:0/1885719447 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x564f0bdc5e00 0x564f0bdc8220 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.870+0000 7f5d72d90740 1 --2- 172.21.15.169:0/1885719447 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f0bd48400 0x564f0bd53190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.870+0000 7f5d72d90740 1 --2- 172.21.15.169:0/1885719447 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564f0bd43290 0x564f0bd521d0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.870+0000 7f5d72d90740 1 --2- 172.21.15.169:0/1885719447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f0bd428a0 0x564f0bd51c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.870+0000 7f5d72d90740 1 -- 172.21.15.169:0/1885719447 >> 172.21.15.169:0/1885719447 conn(0x564f0bd3de10 msgr2=0x564f0bd41900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:02.876 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.870+0000 7f5d517fa640 0 client.16203 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:00:02.876 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.871+0000 7f5d72d90740 1 -- 172.21.15.169:0/1885719447 shutdown_connections 2024-03-28T01:00:02.876 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.871+0000 7f5d72d90740 1 -- 172.21.15.169:0/1885719447 wait complete. 2024-03-28T01:00:02.877 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.874+0000 7f5d72d90740 20 client.16203 trim_cache size 0 max 16384 2024-03-28T01:00:02.881 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:00:02.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.883+0000 7f5d6ef47640 1 -- 172.21.15.169:0/2425744091 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5d540a2a10 msgr2=0x7f5d540a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:02.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.883+0000 7f5d6ef47640 1 --2- 172.21.15.169:0/2425744091 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5d540a2a10 0x7f5d540a4ed0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5d640c5110 tx=0x7f5d6402c6d0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.883+0000 7f5d6ef47640 1 -- 172.21.15.169:0/2425744091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d68109690 msgr2=0x7f5d68072540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:02.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.883+0000 7f5d6ef47640 1 --2- 172.21.15.169:0/2425744091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d68109690 0x7f5d68072540 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5d64006010 tx=0x7f5d64004e30 comp rx=0 tx=0).stop 2024-03-28T01:00:02.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.888+0000 7f5d6ef47640 1 -- 172.21.15.169:0/2425744091 shutdown_connections 2024-03-28T01:00:02.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.888+0000 7f5d6ef47640 1 --2- 172.21.15.169:0/2425744091 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5d540a2a10 0x7f5d540a4ed0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.888+0000 7f5d6ef47640 1 --2- 172.21.15.169:0/2425744091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d68109690 0x7f5d68072540 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.888+0000 7f5d6ef47640 1 --2- 172.21.15.169:0/2425744091 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5d68108cd0 0x7f5d68071500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.888+0000 7f5d6ef47640 1 --2- 172.21.15.169:0/2425744091 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5d68107ae0 0x7f5d68070fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:02.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.888+0000 7f5d6ef47640 1 -- 172.21.15.169:0/2425744091 >> 172.21.15.169:0/2425744091 conn(0x7f5d68076a80 msgr2=0x7f5d6810bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:02.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.890+0000 7f5d6ef47640 1 -- 172.21.15.169:0/2425744091 shutdown_connections 2024-03-28T01:00:02.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:02.892+0000 7f5d6ef47640 1 -- 172.21.15.169:0/2425744091 wait complete. 2024-03-28T01:00:02.941 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:00:02.941 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T01:00:03.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.247+0000 7f252ebbd640 1 -- 172.21.15.169:0/2655361807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2528109730 msgr2=0x7f252810dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:03.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.247+0000 7f252ebbd640 1 --2- 172.21.15.169:0/2655361807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2528109730 0x7f252810dfd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2520009ca0 tx=0x7f252002c100 comp rx=0 tx=0).stop 2024-03-28T01:00:03.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.247+0000 7f252ebbd640 1 -- 172.21.15.169:0/2655361807 shutdown_connections 2024-03-28T01:00:03.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.247+0000 7f252ebbd640 1 --2- 172.21.15.169:0/2655361807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2528109730 0x7f252810dfd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.247+0000 7f252ebbd640 1 --2- 172.21.15.169:0/2655361807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2528108d70 0x7f25281091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.247+0000 7f252ebbd640 1 --2- 172.21.15.169:0/2655361807 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2528072ab0 0x7f2528072ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.247+0000 7f252ebbd640 1 -- 172.21.15.169:0/2655361807 >> 172.21.15.169:0/2655361807 conn(0x7f252806e580 msgr2=0x7f25280709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:03.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.253+0000 7f252ebbd640 1 -- 172.21.15.169:0/2655361807 shutdown_connections 2024-03-28T01:00:03.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.254+0000 7f252ebbd640 1 -- 172.21.15.169:0/2655361807 wait complete. 2024-03-28T01:00:03.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.255+0000 7f252ebbd640 1 Processor -- start 2024-03-28T01:00:03.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.255+0000 7f252ebbd640 1 -- start start 2024-03-28T01:00:03.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.255+0000 7f252ebbd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2528072ab0 0x7f25281a64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:03.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.255+0000 7f252ebbd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2528108d70 0x7f25281a6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:03.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.255+0000 7f252ebbd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25281aba60 0x7f25281ade50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:03.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.255+0000 7f252ebbd640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252811c080 con 0x7f2528072ab0 2024-03-28T01:00:03.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.255+0000 7f252ebbd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252811bf00 con 0x7f25281aba60 2024-03-28T01:00:03.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.255+0000 7f252ebbd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252811c200 con 0x7f2528108d70 2024-03-28T01:00:03.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.256+0000 7f252dbbb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2528072ab0 0x7f25281a64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:03.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.256+0000 7f252e3bc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25281aba60 0x7f25281ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:03.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.256+0000 7f252e3bc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25281aba60 0x7f25281ade50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:41402/0 (socket says 172.21.15.169:41402) 2024-03-28T01:00:03.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.256+0000 7f252e3bc640 1 -- 172.21.15.169:0/405850151 learned_addr learned my addr 172.21.15.169:0/405850151 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:03.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.257+0000 7f252e3bc640 1 -- 172.21.15.169:0/405850151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2528108d70 msgr2=0x7f25281a6a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:00:03.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.257+0000 7f252e3bc640 1 --2- 172.21.15.169:0/405850151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2528108d70 0x7f25281a6a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.257+0000 7f252e3bc640 1 -- 172.21.15.169:0/405850151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2528072ab0 msgr2=0x7f25281a64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:03.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.257+0000 7f252e3bc640 1 --2- 172.21.15.169:0/405850151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2528072ab0 0x7f25281a64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.257+0000 7f252e3bc640 1 -- 172.21.15.169:0/405850151 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25281ae550 con 0x7f25281aba60 2024-03-28T01:00:03.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.259+0000 7f252e3bc640 1 --2- 172.21.15.169:0/405850151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25281aba60 0x7f25281ade50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f252002c690 tx=0x7f2520003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:03.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.264+0000 7f251effd640 1 -- 172.21.15.169:0/405850151 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2520041020 con 0x7f25281aba60 2024-03-28T01:00:03.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.264+0000 7f252ebbd640 1 -- 172.21.15.169:0/405850151 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25281ae7e0 con 0x7f25281aba60 2024-03-28T01:00:03.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.265+0000 7f252ebbd640 1 -- 172.21.15.169:0/405850151 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25281aecd0 con 0x7f25281aba60 2024-03-28T01:00:03.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.270+0000 7f2532a05740 1 Processor -- start 2024-03-28T01:00:03.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.270+0000 7f251effd640 1 -- 172.21.15.169:0/405850151 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2520004520 con 0x7f25281aba60 2024-03-28T01:00:03.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.270+0000 7f251effd640 1 -- 172.21.15.169:0/405850151 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2520039040 con 0x7f25281aba60 2024-03-28T01:00:03.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.272+0000 7f2532a05740 1 -- start start 2024-03-28T01:00:03.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.273+0000 7f251effd640 1 -- 172.21.15.169:0/405850151 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2520044020 con 0x7f25281aba60 2024-03-28T01:00:03.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.274+0000 7f251effd640 1 --2- 172.21.15.169:0/405850151 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f25140a2a10 0x7f25140a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:03.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.274+0000 7f251effd640 1 -- 172.21.15.169:0/405850151 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f25200f1d50 con 0x7f25281aba60 2024-03-28T01:00:03.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.285+0000 7f252d3ba640 1 --2- 172.21.15.169:0/405850151 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f25140a2a10 0x7f25140a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:03.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.286+0000 7f2532a05740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad3860 0x555931ad3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:03.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.286+0000 7f2532a05740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555931ad4250 0x555931ad8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:03.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.286+0000 7f2532a05740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad93c0 0x555931adb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:03.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.286+0000 7f2532a05740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x555931a96610 con 0x555931ad4250 2024-03-28T01:00:03.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.286+0000 7f2532a05740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x555931a393c0 con 0x555931ad93c0 2024-03-28T01:00:03.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.286+0000 7f2532a05740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559319e1d60 con 0x555931ad3860 2024-03-28T01:00:03.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.289+0000 7f252e3bc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad93c0 0x555931adb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:03.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.289+0000 7f252dbbb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad3860 0x555931ad3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:03.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.291+0000 7f252dbbb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad3860 0x555931ad3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:46000/0 (socket says 172.21.15.169:46000) 2024-03-28T01:00:03.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.291+0000 7f252dbbb640 1 -- 172.21.15.169:0/4084494124 learned_addr learned my addr 172.21.15.169:0/4084494124 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:03.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.292+0000 7f252d3ba640 1 --2- 172.21.15.169:0/4084494124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555931ad4250 0x555931ad8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:03.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.295+0000 7f252dbbb640 1 -- 172.21.15.169:0/4084494124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad93c0 msgr2=0x555931adb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:03.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.295+0000 7f252dbbb640 1 --2- 172.21.15.169:0/4084494124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad93c0 0x555931adb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.295+0000 7f252dbbb640 1 -- 172.21.15.169:0/4084494124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555931ad4250 msgr2=0x555931ad8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:03.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.295+0000 7f252dbbb640 1 --2- 172.21.15.169:0/4084494124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555931ad4250 0x555931ad8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.295+0000 7f252dbbb640 1 -- 172.21.15.169:0/4084494124 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55593174d3f0 con 0x555931ad3860 2024-03-28T01:00:03.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.298+0000 7f252dbbb640 1 --2- 172.21.15.169:0/4084494124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad3860 0x555931ad3c80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2524009bf0 tx=0x7f252400e6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff1cea358a392d6e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:03.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.299+0000 7f2513fff640 1 -- 172.21.15.169:0/4084494124 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2524016070 con 0x555931ad3860 2024-03-28T01:00:03.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.299+0000 7f2513fff640 1 -- 172.21.15.169:0/4084494124 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2524011950 con 0x555931ad3860 2024-03-28T01:00:03.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.299+0000 7f2513fff640 1 -- 172.21.15.169:0/4084494124 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25240108b0 con 0x555931ad3860 2024-03-28T01:00:03.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.300+0000 7f2532a05740 1 -- 172.21.15.169:0/4084494124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad3860 msgr2=0x555931ad3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:03.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.300+0000 7f2532a05740 1 --2- 172.21.15.169:0/4084494124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad3860 0x555931ad3c80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2524009bf0 tx=0x7f252400e6c0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.303+0000 7f2532a05740 1 -- 172.21.15.169:0/4084494124 shutdown_connections 2024-03-28T01:00:03.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.303+0000 7f2532a05740 1 --2- 172.21.15.169:0/4084494124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad93c0 0x555931adb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.303+0000 7f2532a05740 1 --2- 172.21.15.169:0/4084494124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555931ad4250 0x555931ad8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.303+0000 7f2532a05740 1 --2- 172.21.15.169:0/4084494124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad3860 0x555931ad3c80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.303+0000 7f2532a05740 1 -- 172.21.15.169:0/4084494124 >> 172.21.15.169:0/4084494124 conn(0x555931acedd0 msgr2=0x555931acf1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:03.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.305+0000 7f2532a05740 1 -- 172.21.15.169:0/4084494124 shutdown_connections 2024-03-28T01:00:03.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.306+0000 7f2532a05740 1 -- 172.21.15.169:0/4084494124 wait complete. 2024-03-28T01:00:03.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.306+0000 7f2532a05740 1 -- start start 2024-03-28T01:00:03.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.310+0000 7f252d3ba640 1 --2- 172.21.15.169:0/405850151 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f25140a2a10 0x7f25140a4ed0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f25281a79d0 tx=0x7f25180137d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:03.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.315+0000 7f2532a05740 1 Processor -- start 2024-03-28T01:00:03.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.316+0000 7f2532a05740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad3860 0x555931addc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:03.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.316+0000 7f2532a05740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555931ad4250 0x555931ade170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:03.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.316+0000 7f2532a05740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad93c0 0x555931ae51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:03.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.316+0000 7f2532a05740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559319f5240 con 0x555931ad4250 2024-03-28T01:00:03.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.316+0000 7f2532a05740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x555931a35460 con 0x555931ad93c0 2024-03-28T01:00:03.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.316+0000 7f2532a05740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x555931adc370 con 0x555931ad3860 2024-03-28T01:00:03.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.316+0000 7f2532a05740 10 client.0 _ll_register_callbacks cb 0x5559319f4c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:03.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.317+0000 7f252dbbb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad3860 0x555931addc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:03.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.317+0000 7f252dbbb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad3860 0x555931addc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:46006/0 (socket says 172.21.15.169:46006) 2024-03-28T01:00:03.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.317+0000 7f252dbbb640 1 -- 172.21.15.169:0/1173210501 learned_addr learned my addr 172.21.15.169:0/1173210501 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:03.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.318+0000 7f252dbbb640 1 -- 172.21.15.169:0/1173210501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad93c0 msgr2=0x555931ae51b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:00:03.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.318+0000 7f252dbbb640 1 --2- 172.21.15.169:0/1173210501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad93c0 0x555931ae51b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.318+0000 7f252dbbb640 1 -- 172.21.15.169:0/1173210501 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555931ad4250 msgr2=0x555931ade170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:03.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.318+0000 7f252dbbb640 1 --2- 172.21.15.169:0/1173210501 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555931ad4250 0x555931ade170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.318+0000 7f252dbbb640 1 -- 172.21.15.169:0/1173210501 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5559318177d0 con 0x555931ad3860 2024-03-28T01:00:03.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.321+0000 7f252dbbb640 1 --2- 172.21.15.169:0/1173210501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad3860 0x555931addc30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2524009370 tx=0x7f25240097f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a54ea61b07be8b8f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:03.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.323+0000 7f25117fa640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:00:03.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.323+0000 7f25117fa640 1 -- 172.21.15.169:0/1173210501 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2524016050 con 0x555931ad3860 2024-03-28T01:00:03.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.324+0000 7f25117fa640 1 -- 172.21.15.169:0/1173210501 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2524005040 con 0x555931ad3860 2024-03-28T01:00:03.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.326+0000 7f2532a05740 1 -- 172.21.15.169:0/1173210501 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2524011950 con 0x555931ad3860 2024-03-28T01:00:03.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.326+0000 7f25117fa640 1 -- 172.21.15.169:0/1173210501 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2524005240 con 0x555931ad3860 2024-03-28T01:00:03.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.327+0000 7f2532a05740 10 client.26017 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:03.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.327+0000 7f2532a05740 1 -- 172.21.15.169:0/1173210501 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x555931ae8550 con 0x555931ad3860 2024-03-28T01:00:03.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.328+0000 7f25117fa640 1 -- 172.21.15.169:0/1173210501 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2524011950 con 0x555931ad3860 2024-03-28T01:00:03.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.330+0000 7f25117fa640 1 -- 172.21.15.169:0/1173210501 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f252400f040 con 0x555931ad3860 2024-03-28T01:00:03.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.330+0000 7f2532a05740 10 client.26017 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:03.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.330+0000 7f2532a05740 10 client.26017 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:03.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.330+0000 7f2532a05740 20 client.26017 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:03.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.330+0000 7f2532a05740 1 --2- 172.21.15.169:0/1173210501 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555931aea050 0x555931aec4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:03.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.330+0000 7f2532a05740 4 client.26017 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:00:03.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.330+0000 7f2532a05740 1 -- 172.21.15.169:0/1173210501 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555931a46900 con 0x555931aea050 2024-03-28T01:00:03.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.336+0000 7f252d3ba640 1 --2- 172.21.15.169:0/1173210501 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555931aea050 0x555931aec4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:03.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.341+0000 7f25117fa640 10 client.26017 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:03.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.341+0000 7f252d3ba640 1 --2- 172.21.15.169:0/1173210501 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555931aea050 0x555931aec4b0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21c23c0626dd11a9 server_cookie=cbd34ce10b6ebd9a in_seq=0 out_seq=0 2024-03-28T01:00:03.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.351+0000 7f25117fa640 1 -- 172.21.15.169:0/1173210501 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555931a46900 con 0x555931aea050 2024-03-28T01:00:03.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.351+0000 7f25117fa640 10 client.26017 handle_command_reply: tid=0 2024-03-28T01:00:03.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.351+0000 7f2532a05740 1 client.26017 shutdown 2024-03-28T01:00:03.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.355+0000 7f2532a05740 1 -- 172.21.15.169:0/1173210501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad3860 msgr2=0x555931addc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:03.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.355+0000 7f2532a05740 1 --2- 172.21.15.169:0/1173210501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad3860 0x555931addc30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2524009370 tx=0x7f25240097f0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.356+0000 7f2532a05740 1 -- 172.21.15.169:0/1173210501 shutdown_connections 2024-03-28T01:00:03.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.356+0000 7f2532a05740 1 --2- 172.21.15.169:0/1173210501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad93c0 0x555931ae51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.356+0000 7f2532a05740 1 --2- 172.21.15.169:0/1173210501 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555931aea050 0x555931aec4b0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.356+0000 7f2532a05740 1 --2- 172.21.15.169:0/1173210501 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555931ad4250 0x555931ade170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.356+0000 7f2532a05740 1 --2- 172.21.15.169:0/1173210501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad3860 0x555931addc30 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.356+0000 7f2532a05740 1 -- 172.21.15.169:0/1173210501 >> 172.21.15.169:0/1173210501 conn(0x555931acedd0 msgr2=0x555931ace070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:03.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.356+0000 7f25117fa640 0 client.26017 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:03.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.356+0000 7f2532a05740 1 -- 172.21.15.169:0/1173210501 shutdown_connections 2024-03-28T01:00:03.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.357+0000 7f2532a05740 1 -- 172.21.15.169:0/1173210501 wait complete. 2024-03-28T01:00:03.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.357+0000 7f2532a05740 20 client.26017 trim_cache size 0 max 16384 2024-03-28T01:00:03.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.415+0000 7f2532a05740 1 Processor -- start 2024-03-28T01:00:03.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.415+0000 7f2532a05740 1 -- start start 2024-03-28T01:00:03.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.415+0000 7f2532a05740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad3860 0x555931af0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:03.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.415+0000 7f2532a05740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555931ad4250 0x555931ae50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:03.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.416+0000 7f2532a05740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad93c0 0x555931af12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:03.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.416+0000 7f252d3ba640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555931ad4250 0x555931ae50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:03.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.416+0000 7f252d3ba640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555931ad4250 0x555931ae50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47282/0 (socket says 172.21.15.169:47282) 2024-03-28T01:00:03.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.416+0000 7f252d3ba640 1 -- 172.21.15.169:0/3314195759 learned_addr learned my addr 172.21.15.169:0/3314195759 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:03.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.416+0000 7f2532a05740 1 -- 172.21.15.169:0/3314195759 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c011ec0 con 0x555931ad4250 2024-03-28T01:00:03.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.416+0000 7f252dbbb640 1 --2- 172.21.15.169:0/3314195759 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad3860 0x555931af0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:03.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.416+0000 7f2532a05740 1 -- 172.21.15.169:0/3314195759 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c00f9c0 con 0x555931ad3860 2024-03-28T01:00:03.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.416+0000 7f2532a05740 1 -- 172.21.15.169:0/3314195759 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c00db10 con 0x555931ad93c0 2024-03-28T01:00:03.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.417+0000 7f252dbbb640 1 -- 172.21.15.169:0/3314195759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad93c0 msgr2=0x555931af12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:03.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.417+0000 7f252dbbb640 1 --2- 172.21.15.169:0/3314195759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad93c0 0x555931af12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.417+0000 7f252dbbb640 1 -- 172.21.15.169:0/3314195759 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555931ad4250 msgr2=0x555931ae50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:03.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.417+0000 7f252dbbb640 1 --2- 172.21.15.169:0/3314195759 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555931ad4250 0x555931ae50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.417+0000 7f252dbbb640 1 -- 172.21.15.169:0/3314195759 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555931aecb30 con 0x555931ad3860 2024-03-28T01:00:03.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.418+0000 7f252dbbb640 1 --2- 172.21.15.169:0/3314195759 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad3860 0x555931af0db0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2524000c00 tx=0x7f2524001e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd43cca86757fd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:03.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.429+0000 7f25137fe640 1 -- 172.21.15.169:0/3314195759 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2524039020 con 0x555931ad3860 2024-03-28T01:00:03.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.429+0000 7f25137fe640 1 -- 172.21.15.169:0/3314195759 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2524011930 con 0x555931ad3860 2024-03-28T01:00:03.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.429+0000 7f25137fe640 1 -- 172.21.15.169:0/3314195759 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2524016050 con 0x555931ad3860 2024-03-28T01:00:03.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.433+0000 7f2532a05740 1 -- 172.21.15.169:0/3314195759 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad3860 msgr2=0x555931af0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:03.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.433+0000 7f2532a05740 1 --2- 172.21.15.169:0/3314195759 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad3860 0x555931af0db0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2524000c00 tx=0x7f2524001e50 comp rx=0 tx=0).stop 2024-03-28T01:00:03.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.439+0000 7f2532a05740 1 -- 172.21.15.169:0/3314195759 shutdown_connections 2024-03-28T01:00:03.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.439+0000 7f2532a05740 1 --2- 172.21.15.169:0/3314195759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad93c0 0x555931af12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.439+0000 7f2532a05740 1 --2- 172.21.15.169:0/3314195759 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555931ad4250 0x555931ae50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.439+0000 7f2532a05740 1 --2- 172.21.15.169:0/3314195759 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad3860 0x555931af0db0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.439+0000 7f2532a05740 1 -- 172.21.15.169:0/3314195759 >> 172.21.15.169:0/3314195759 conn(0x555931acedd0 msgr2=0x555931ad2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:03.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.442+0000 7f2532a05740 1 -- 172.21.15.169:0/3314195759 shutdown_connections 2024-03-28T01:00:03.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.442+0000 7f2532a05740 1 -- 172.21.15.169:0/3314195759 wait complete. 2024-03-28T01:00:03.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.443+0000 7f2532a05740 1 -- start start 2024-03-28T01:00:03.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.443+0000 7f2532a05740 1 Processor -- start 2024-03-28T01:00:03.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.444+0000 7f2532a05740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad3860 0x555931ae2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:03.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.444+0000 7f2532a05740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad4250 0x555931ae2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:03.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.444+0000 7f2532a05740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555931ad93c0 0x555931ae3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:03.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.444+0000 7f2532a05740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c00e080 con 0x555931ad93c0 2024-03-28T01:00:03.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.444+0000 7f2532a05740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c00f360 con 0x555931ad4250 2024-03-28T01:00:03.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.444+0000 7f2532a05740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c00e6a0 con 0x555931ad3860 2024-03-28T01:00:03.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.444+0000 7f2532a05740 10 client.0 _ll_register_callbacks cb 0x7f25240092a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:03.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.444+0000 7f252d3ba640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad4250 0x555931ae2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:03.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.444+0000 7f252d3ba640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad4250 0x555931ae2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:41440/0 (socket says 172.21.15.169:41440) 2024-03-28T01:00:03.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.444+0000 7f252d3ba640 1 -- 172.21.15.169:0/2115162204 learned_addr learned my addr 172.21.15.169:0/2115162204 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:03.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.445+0000 7f252dbbb640 1 --2- 172.21.15.169:0/2115162204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad3860 0x555931ae2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:03.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.445+0000 7f252d3ba640 1 -- 172.21.15.169:0/2115162204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad3860 msgr2=0x555931ae2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:03.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.445+0000 7f252d3ba640 1 --2- 172.21.15.169:0/2115162204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad3860 0x555931ae2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.445+0000 7f252d3ba640 1 -- 172.21.15.169:0/2115162204 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555931ad93c0 msgr2=0x555931ae3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:03.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.445+0000 7f252d3ba640 1 --2- 172.21.15.169:0/2115162204 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555931ad93c0 0x555931ae3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.445+0000 7f252d3ba640 1 -- 172.21.15.169:0/2115162204 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555931ae43d0 con 0x555931ad4250 2024-03-28T01:00:03.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.446+0000 7f252d3ba640 1 --2- 172.21.15.169:0/2115162204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad4250 0x555931ae2dd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x555931ad4cc0 tx=0x7f25180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b841cc2aee887a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:03.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.453+0000 7f25117fa640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:00:03.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.453+0000 7f25117fa640 1 -- 172.21.15.169:0/2115162204 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2518020920 con 0x555931ad4250 2024-03-28T01:00:03.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.453+0000 7f2532a05740 1 -- 172.21.15.169:0/2115162204 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2524011930 con 0x555931ad4250 2024-03-28T01:00:03.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.453+0000 7f25117fa640 1 -- 172.21.15.169:0/2115162204 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2518020ac0 con 0x555931ad4250 2024-03-28T01:00:03.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.454+0000 7f25117fa640 1 -- 172.21.15.169:0/2115162204 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2518020cc0 con 0x555931ad4250 2024-03-28T01:00:03.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.454+0000 7f2532a05740 10 client.26060 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:03.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.454+0000 7f2532a05740 1 -- 172.21.15.169:0/2115162204 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x555931b45980 con 0x555931ad4250 2024-03-28T01:00:03.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.455+0000 7f25117fa640 1 -- 172.21.15.169:0/2115162204 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2524011930 con 0x555931ad4250 2024-03-28T01:00:03.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.455+0000 7f25117fa640 1 -- 172.21.15.169:0/2115162204 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f251801c040 con 0x555931ad4250 2024-03-28T01:00:03.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.455+0000 7f2532a05740 10 client.26060 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:03.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.455+0000 7f2532a05740 10 client.26060 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:03.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.455+0000 7f2532a05740 20 client.26060 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:03.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.455+0000 7f2532a05740 1 --2- 172.21.15.169:0/2115162204 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555931b46300 0x555931b48720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:03.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.455+0000 7f2532a05740 4 client.26060 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:00:03.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.455+0000 7f2532a05740 1 -- 172.21.15.169:0/2115162204 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555931a40c50 con 0x555931b46300 2024-03-28T01:00:03.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.462+0000 7f252dbbb640 1 --2- 172.21.15.169:0/2115162204 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555931b46300 0x555931b48720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:03.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.465+0000 7f252dbbb640 1 --2- 172.21.15.169:0/2115162204 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555931b46300 0x555931b48720 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e645492004985ce server_cookie=1033abf89ff0319a in_seq=0 out_seq=0 2024-03-28T01:00:03.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.465+0000 7f25117fa640 10 client.26060 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:03.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.474+0000 7f25117fa640 1 -- 172.21.15.169:0/2115162204 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555931a40c50 con 0x555931b46300 2024-03-28T01:00:03.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.474+0000 7f25117fa640 10 client.26060 handle_command_reply: tid=0 2024-03-28T01:00:03.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.474+0000 7f2532a05740 1 client.26060 shutdown 2024-03-28T01:00:03.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.476+0000 7f2532a05740 1 -- 172.21.15.169:0/2115162204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad4250 msgr2=0x555931ae2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:03.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.476+0000 7f2532a05740 1 --2- 172.21.15.169:0/2115162204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad4250 0x555931ae2dd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x555931ad4cc0 tx=0x7f25180046c0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.482+0000 7f2532a05740 1 -- 172.21.15.169:0/2115162204 shutdown_connections 2024-03-28T01:00:03.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.482+0000 7f2532a05740 1 --2- 172.21.15.169:0/2115162204 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555931b46300 0x555931b48720 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.482+0000 7f2532a05740 1 --2- 172.21.15.169:0/2115162204 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555931ad93c0 0x555931ae3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.482+0000 7f2532a05740 1 --2- 172.21.15.169:0/2115162204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555931ad4250 0x555931ae2dd0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.482+0000 7f2532a05740 1 --2- 172.21.15.169:0/2115162204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555931ad3860 0x555931ae2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.482+0000 7f2532a05740 1 -- 172.21.15.169:0/2115162204 >> 172.21.15.169:0/2115162204 conn(0x555931acedd0 msgr2=0x555931acd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:03.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.482+0000 7f25117fa640 0 client.26060 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:03.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.485+0000 7f2532a05740 1 -- 172.21.15.169:0/2115162204 shutdown_connections 2024-03-28T01:00:03.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.491+0000 7f2532a05740 1 -- 172.21.15.169:0/2115162204 wait complete. 2024-03-28T01:00:03.500 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:00:03.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.495+0000 7f2532a05740 20 client.26060 trim_cache size 0 max 16384 2024-03-28T01:00:03.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.504+0000 7f252ebbd640 1 -- 172.21.15.169:0/405850151 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f25140a2a10 msgr2=0x7f25140a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:03.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.504+0000 7f252ebbd640 1 --2- 172.21.15.169:0/405850151 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f25140a2a10 0x7f25140a4ed0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f25281a79d0 tx=0x7f25180137d0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.505+0000 7f252ebbd640 1 -- 172.21.15.169:0/405850151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25281aba60 msgr2=0x7f25281ade50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:03.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.505+0000 7f252ebbd640 1 --2- 172.21.15.169:0/405850151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25281aba60 0x7f25281ade50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f252002c690 tx=0x7f2520003ba0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.506+0000 7f252ebbd640 1 -- 172.21.15.169:0/405850151 shutdown_connections 2024-03-28T01:00:03.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.506+0000 7f252ebbd640 1 --2- 172.21.15.169:0/405850151 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f25140a2a10 0x7f25140a4ed0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.506+0000 7f252ebbd640 1 --2- 172.21.15.169:0/405850151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25281aba60 0x7f25281ade50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.506+0000 7f252ebbd640 1 --2- 172.21.15.169:0/405850151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2528108d70 0x7f25281a6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.506+0000 7f252ebbd640 1 --2- 172.21.15.169:0/405850151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2528072ab0 0x7f25281a64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:03.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.506+0000 7f252ebbd640 1 -- 172.21.15.169:0/405850151 >> 172.21.15.169:0/405850151 conn(0x7f252806e580 msgr2=0x7f2528109f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:03.514 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.510+0000 7f252ebbd640 1 -- 172.21.15.169:0/405850151 shutdown_connections 2024-03-28T01:00:03.514 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:03.510+0000 7f252ebbd640 1 -- 172.21.15.169:0/405850151 wait complete. 2024-03-28T01:00:03.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:03 smithi177 ceph-mon[36206]: pgmap v457: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 907 MiB/s wr, 1.49k op/s 2024-03-28T01:00:03.804 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:03 smithi169 ceph-mon[32274]: pgmap v457: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 907 MiB/s wr, 1.49k op/s 2024-03-28T01:00:03.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:03 smithi174 ceph-mon[36060]: pgmap v457: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 907 MiB/s wr, 1.49k op/s 2024-03-28T01:00:04.558 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T01:00:04.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T01:00:04.559 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T01:00:04.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.963+0000 7f22a991c640 1 -- 172.21.15.169:0/2563427680 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22a4072ab0 msgr2=0x7f22a4072ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:04.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.963+0000 7f22a991c640 1 --2- 172.21.15.169:0/2563427680 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22a4072ab0 0x7f22a4072ed0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f229c01acb0 tx=0x7f229c03d150 comp rx=0 tx=0).stop 2024-03-28T01:00:04.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.967+0000 7f22a991c640 1 -- 172.21.15.169:0/2563427680 shutdown_connections 2024-03-28T01:00:04.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.967+0000 7f22a991c640 1 --2- 172.21.15.169:0/2563427680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22a4109730 0x7f22a410dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:04.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.967+0000 7f22a991c640 1 --2- 172.21.15.169:0/2563427680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22a4108d70 0x7f22a41091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:04.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.967+0000 7f22a991c640 1 --2- 172.21.15.169:0/2563427680 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22a4072ab0 0x7f22a4072ed0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:04.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.967+0000 7f22a991c640 1 -- 172.21.15.169:0/2563427680 >> 172.21.15.169:0/2563427680 conn(0x7f22a406e580 msgr2=0x7f22a40709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:04.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.970+0000 7f22a991c640 1 -- 172.21.15.169:0/2563427680 shutdown_connections 2024-03-28T01:00:04.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.973+0000 7f22a991c640 1 -- 172.21.15.169:0/2563427680 wait complete. 2024-03-28T01:00:04.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.974+0000 7f22a991c640 1 Processor -- start 2024-03-28T01:00:04.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.974+0000 7f22a991c640 1 -- start start 2024-03-28T01:00:04.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.974+0000 7f22a991c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22a4072ab0 0x7f22a419dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:04.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.974+0000 7f22a991c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22a4108d70 0x7f22a419e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:04.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.974+0000 7f22a991c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22a4109730 0x7f22a4079900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:04.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.974+0000 7f22a991c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a41119d0 con 0x7f22a4109730 2024-03-28T01:00:04.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.974+0000 7f22a991c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a4111850 con 0x7f22a4072ab0 2024-03-28T01:00:04.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.974+0000 7f22a991c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a4111b50 con 0x7f22a4108d70 2024-03-28T01:00:04.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.975+0000 7f22a891a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22a4072ab0 0x7f22a419dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:04.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.975+0000 7f22a891a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22a4072ab0 0x7f22a419dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:41458/0 (socket says 172.21.15.169:41458) 2024-03-28T01:00:04.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.975+0000 7f22a891a640 1 -- 172.21.15.169:0/3776152405 learned_addr learned my addr 172.21.15.169:0/3776152405 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:04.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.977+0000 7f22a3fff640 1 --2- 172.21.15.169:0/3776152405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22a4108d70 0x7f22a419e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:04.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.978+0000 7f22a911b640 1 --2- 172.21.15.169:0/3776152405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22a4109730 0x7f22a4079900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:04.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.978+0000 7f22a891a640 1 -- 172.21.15.169:0/3776152405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22a4108d70 msgr2=0x7f22a419e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:04.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.978+0000 7f22a891a640 1 --2- 172.21.15.169:0/3776152405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22a4108d70 0x7f22a419e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:04.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.978+0000 7f22a891a640 1 -- 172.21.15.169:0/3776152405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22a4109730 msgr2=0x7f22a4079900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:04.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.978+0000 7f22a891a640 1 --2- 172.21.15.169:0/3776152405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22a4109730 0x7f22a4079900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:04.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.978+0000 7f22a891a640 1 -- 172.21.15.169:0/3776152405 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a4079fa0 con 0x7f22a4072ab0 2024-03-28T01:00:04.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.981+0000 7f22a891a640 1 --2- 172.21.15.169:0/3776152405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22a4072ab0 0x7f22a419dee0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f229c000f80 tx=0x7f229c01a880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:04.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.981+0000 7f22a1ffb640 1 -- 172.21.15.169:0/3776152405 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229c052020 con 0x7f22a4072ab0 2024-03-28T01:00:04.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.981+0000 7f22a1ffb640 1 -- 172.21.15.169:0/3776152405 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f229c045a20 con 0x7f22a4072ab0 2024-03-28T01:00:04.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.982+0000 7f22a991c640 1 -- 172.21.15.169:0/3776152405 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a407a230 con 0x7f22a4072ab0 2024-03-28T01:00:04.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.982+0000 7f22a991c640 1 -- 172.21.15.169:0/3776152405 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a407a720 con 0x7f22a4072ab0 2024-03-28T01:00:04.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.982+0000 7f22a1ffb640 1 -- 172.21.15.169:0/3776152405 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229c04a040 con 0x7f22a4072ab0 2024-03-28T01:00:04.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.988+0000 7f22a1ffb640 1 -- 172.21.15.169:0/3776152405 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f229c055020 con 0x7f22a4072ab0 2024-03-28T01:00:04.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.988+0000 7f22ad769740 1 Processor -- start 2024-03-28T01:00:04.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.988+0000 7f22ad769740 1 -- start start 2024-03-28T01:00:04.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.989+0000 7f22ad769740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd846db30 0x559fd846df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:04.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.989+0000 7f22ad769740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fd846e520 0x559fd8473020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:04.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.989+0000 7f22ad769740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd8473690 0x559fd8475b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:04.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.989+0000 7f22ad769740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fd83adc90 con 0x559fd846db30 2024-03-28T01:00:04.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.989+0000 7f22ad769740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fd83d3070 con 0x559fd846e520 2024-03-28T01:00:04.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.989+0000 7f22ad769740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fd837bc80 con 0x559fd8473690 2024-03-28T01:00:04.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.989+0000 7f22a3fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd8473690 0x559fd8475b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:04.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.989+0000 7f22a3fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd8473690 0x559fd8475b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:46058/0 (socket says 172.21.15.169:46058) 2024-03-28T01:00:04.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.989+0000 7f22a3fff640 1 -- 172.21.15.169:0/3432711065 learned_addr learned my addr 172.21.15.169:0/3432711065 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:04.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.989+0000 7f22a911b640 1 --2- 172.21.15.169:0/3432711065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd846db30 0x559fd846df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:04.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.989+0000 7f22a1ffb640 1 --2- 172.21.15.169:0/3776152405 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f228c0a2980 0x7f228c0a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:04.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.989+0000 7f22a3fff640 1 -- 172.21.15.169:0/3432711065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fd846e520 msgr2=0x559fd8473020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:00:04.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.989+0000 7f22a3fff640 1 --2- 172.21.15.169:0/3432711065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fd846e520 0x559fd8473020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:04.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.989+0000 7f22a3fff640 1 -- 172.21.15.169:0/3432711065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd846db30 msgr2=0x559fd846df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:04.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.989+0000 7f22a3fff640 1 --2- 172.21.15.169:0/3432711065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd846db30 0x559fd846df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:04.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.989+0000 7f22a3fff640 1 -- 172.21.15.169:0/3432711065 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559fd8373d10 con 0x559fd8473690 2024-03-28T01:00:04.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.990+0000 7f22a1ffb640 1 -- 172.21.15.169:0/3776152405 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f229c101b50 con 0x7f22a4072ab0 2024-03-28T01:00:04.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.990+0000 7f22a3fff640 1 --2- 172.21.15.169:0/3432711065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd8473690 0x559fd8475b20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2298009b90 tx=0x7f229800e6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=820f3190dec39638 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:04.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.990+0000 7f228affd640 1 -- 172.21.15.169:0/3432711065 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298016070 con 0x559fd8473690 2024-03-28T01:00:04.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.991+0000 7f228affd640 1 -- 172.21.15.169:0/3432711065 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2298011950 con 0x559fd8473690 2024-03-28T01:00:04.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.992+0000 7f22a911b640 1 --2- 172.21.15.169:0/3776152405 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f228c0a2980 0x7f228c0a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:04.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.994+0000 7f22a911b640 1 --2- 172.21.15.169:0/3776152405 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f228c0a2980 0x7f228c0a4e40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2294005fd0 tx=0x7f22940046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:05.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.996+0000 7f22ad769740 1 -- 172.21.15.169:0/3432711065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd8473690 msgr2=0x559fd8475b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:05.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.996+0000 7f22ad769740 1 --2- 172.21.15.169:0/3432711065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd8473690 0x559fd8475b20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2298009b90 tx=0x7f229800e6c0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.998+0000 7f22ad769740 1 -- 172.21.15.169:0/3432711065 shutdown_connections 2024-03-28T01:00:05.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.998+0000 7f22ad769740 1 --2- 172.21.15.169:0/3432711065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd8473690 0x559fd8475b20 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.998+0000 7f22ad769740 1 --2- 172.21.15.169:0/3432711065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fd846e520 0x559fd8473020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.998+0000 7f22ad769740 1 --2- 172.21.15.169:0/3432711065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd846db30 0x559fd846df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:04.998+0000 7f22ad769740 1 -- 172.21.15.169:0/3432711065 >> 172.21.15.169:0/3432711065 conn(0x559fd84361a0 msgr2=0x559fd84694a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:05.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.000+0000 7f22ad769740 1 -- 172.21.15.169:0/3432711065 shutdown_connections 2024-03-28T01:00:05.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.000+0000 7f22ad769740 1 -- 172.21.15.169:0/3432711065 wait complete. 2024-03-28T01:00:05.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.001+0000 7f22ad769740 1 -- start start 2024-03-28T01:00:05.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.001+0000 7f22ad769740 1 Processor -- start 2024-03-28T01:00:05.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.001+0000 7f22ad769740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd846db30 0x559fd8477f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.001+0000 7f22ad769740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fd846e520 0x559fd84784b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.001+0000 7f22ad769740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd8473690 0x559fd847f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.001+0000 7f22ad769740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fd838f0d0 con 0x559fd846db30 2024-03-28T01:00:05.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.001+0000 7f22ad769740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fd83cf110 con 0x559fd846e520 2024-03-28T01:00:05.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.001+0000 7f22ad769740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fd8476690 con 0x559fd8473690 2024-03-28T01:00:05.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.001+0000 7f22ad769740 10 client.0 _ll_register_callbacks cb 0x559fd83cae60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:05.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.002+0000 7f22a3fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fd846e520 0x559fd84784b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.002+0000 7f22a3fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fd846e520 0x559fd84784b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:41460/0 (socket says 172.21.15.169:41460) 2024-03-28T01:00:05.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.002+0000 7f22a3fff640 1 -- 172.21.15.169:0/3684496599 learned_addr learned my addr 172.21.15.169:0/3684496599 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:05.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.002+0000 7f22a911b640 1 --2- 172.21.15.169:0/3684496599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd8473690 0x559fd847f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.002+0000 7f22a3fff640 1 -- 172.21.15.169:0/3684496599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd8473690 msgr2=0x559fd847f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:05.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.002+0000 7f22a891a640 1 --2- 172.21.15.169:0/3684496599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd846db30 0x559fd8477f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.002+0000 7f22a3fff640 1 --2- 172.21.15.169:0/3684496599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd8473690 0x559fd847f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.002+0000 7f22a3fff640 1 -- 172.21.15.169:0/3684496599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd846db30 msgr2=0x559fd8477f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:05.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.002+0000 7f22a3fff640 1 --2- 172.21.15.169:0/3684496599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd846db30 0x559fd8477f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.002+0000 7f22a3fff640 1 -- 172.21.15.169:0/3684496599 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559fd81af7e0 con 0x559fd846e520 2024-03-28T01:00:05.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.002+0000 7f22a891a640 1 --2- 172.21.15.169:0/3684496599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd846db30 0x559fd8477f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:00:05.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.003+0000 7f22a3fff640 1 --2- 172.21.15.169:0/3684496599 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fd846e520 0x559fd84784b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2298009f20 tx=0x7f2298011b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd80537871d199e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:05.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.008+0000 7f22887f8640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:00:05.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.008+0000 7f22887f8640 1 -- 172.21.15.169:0/3684496599 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298016050 con 0x559fd846e520 2024-03-28T01:00:05.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.008+0000 7f22ad769740 1 -- 172.21.15.169:0/3684496599 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2298011950 con 0x559fd846e520 2024-03-28T01:00:05.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.008+0000 7f22887f8640 1 -- 172.21.15.169:0/3684496599 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2298010a30 con 0x559fd846e520 2024-03-28T01:00:05.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.008+0000 7f22887f8640 1 -- 172.21.15.169:0/3684496599 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2298030050 con 0x559fd846e520 2024-03-28T01:00:05.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.009+0000 7f22ad769740 10 client.26066 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:05.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.009+0000 7f22ad769740 1 -- 172.21.15.169:0/3684496599 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x559fd820ff70 con 0x559fd846e520 2024-03-28T01:00:05.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.009+0000 7f22887f8640 1 -- 172.21.15.169:0/3684496599 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2298011950 con 0x559fd846e520 2024-03-28T01:00:05.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.010+0000 7f22887f8640 1 -- 172.21.15.169:0/3684496599 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f229802b030 con 0x559fd846e520 2024-03-28T01:00:05.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.013+0000 7f22ad769740 10 client.26066 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:05.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.013+0000 7f22ad769740 10 client.26066 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:05.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.013+0000 7f22ad769740 20 client.26066 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:05.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.013+0000 7f22ad769740 1 --2- 172.21.15.169:0/3684496599 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559fd84839e0 0x559fd8485dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.013+0000 7f22ad769740 4 client.26066 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:00:05.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.013+0000 7f22ad769740 1 -- 172.21.15.169:0/3684496599 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559fd83e05b0 con 0x559fd84839e0 2024-03-28T01:00:05.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.013+0000 7f22a891a640 1 --2- 172.21.15.169:0/3684496599 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559fd84839e0 0x559fd8485dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.014+0000 7f22a891a640 1 --2- 172.21.15.169:0/3684496599 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559fd84839e0 0x559fd8485dd0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8f1a9cdffc23a08 server_cookie=e9512cbffb62394 in_seq=0 out_seq=0 2024-03-28T01:00:05.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.014+0000 7f22887f8640 10 client.26066 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:05.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.016+0000 7f22887f8640 1 -- 172.21.15.169:0/3684496599 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559fd83e05b0 con 0x559fd84839e0 2024-03-28T01:00:05.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.016+0000 7f22887f8640 10 client.26066 handle_command_reply: tid=0 2024-03-28T01:00:05.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.016+0000 7f22ad769740 1 client.26066 shutdown 2024-03-28T01:00:05.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.016+0000 7f22ad769740 1 -- 172.21.15.169:0/3684496599 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fd846e520 msgr2=0x559fd84784b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:05.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.016+0000 7f22ad769740 1 --2- 172.21.15.169:0/3684496599 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fd846e520 0x559fd84784b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2298009f20 tx=0x7f2298011b00 comp rx=0 tx=0).stop 2024-03-28T01:00:05.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.018+0000 7f22ad769740 1 -- 172.21.15.169:0/3684496599 shutdown_connections 2024-03-28T01:00:05.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.018+0000 7f22ad769740 1 --2- 172.21.15.169:0/3684496599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd8473690 0x559fd847f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.018+0000 7f22ad769740 1 --2- 172.21.15.169:0/3684496599 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fd846e520 0x559fd84784b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.018+0000 7f22ad769740 1 --2- 172.21.15.169:0/3684496599 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559fd84839e0 0x559fd8485dd0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.018+0000 7f22ad769740 1 --2- 172.21.15.169:0/3684496599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd846db30 0x559fd8477f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.018+0000 7f22ad769740 1 -- 172.21.15.169:0/3684496599 >> 172.21.15.169:0/3684496599 conn(0x559fd84361a0 msgr2=0x559fd84687e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:05.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.018+0000 7f22887f8640 0 client.26066 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:05.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.018+0000 7f22ad769740 1 -- 172.21.15.169:0/3684496599 shutdown_connections 2024-03-28T01:00:05.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.022+0000 7f22ad769740 1 -- 172.21.15.169:0/3684496599 wait complete. 2024-03-28T01:00:05.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.026+0000 7f22ad769740 20 client.26066 trim_cache size 0 max 16384 2024-03-28T01:00:05.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.076+0000 7f22ad769740 1 Processor -- start 2024-03-28T01:00:05.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.076+0000 7f22ad769740 1 -- start start 2024-03-28T01:00:05.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.076+0000 7f22ad769740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd846db30 0x559fd846c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.076+0000 7f22ad769740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd846e520 0x559fd847f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.076+0000 7f22ad769740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fd8473690 0x559fd846cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.077+0000 7f22ad769740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a41c3cb0 con 0x559fd846e520 2024-03-28T01:00:05.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.077+0000 7f22ad769740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a41c17b0 con 0x559fd8473690 2024-03-28T01:00:05.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.077+0000 7f22ad769740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a4082cd0 con 0x559fd846db30 2024-03-28T01:00:05.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.077+0000 7f22a3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd846e520 0x559fd847f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.077+0000 7f22a891a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd846db30 0x559fd846c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.077+0000 7f22a3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd846e520 0x559fd847f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47330/0 (socket says 172.21.15.169:47330) 2024-03-28T01:00:05.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.077+0000 7f22a3fff640 1 -- 172.21.15.169:0/1078553470 learned_addr learned my addr 172.21.15.169:0/1078553470 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:05.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.077+0000 7f22a911b640 1 --2- 172.21.15.169:0/1078553470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fd8473690 0x559fd846cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.077+0000 7f22a891a640 1 -- 172.21.15.169:0/1078553470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fd8473690 msgr2=0x559fd846cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:05.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.077+0000 7f22a891a640 1 --2- 172.21.15.169:0/1078553470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fd8473690 0x559fd846cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.077+0000 7f22a891a640 1 -- 172.21.15.169:0/1078553470 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd846e520 msgr2=0x559fd847f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:05.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.077+0000 7f22a891a640 1 --2- 172.21.15.169:0/1078553470 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd846e520 0x559fd847f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.077+0000 7f22a891a640 1 -- 172.21.15.169:0/1078553470 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559fd80a7170 con 0x559fd846db30 2024-03-28T01:00:05.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.078+0000 7f22a891a640 1 --2- 172.21.15.169:0/1078553470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd846db30 0x559fd846c6e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x559fd846ef90 tx=0x7f229c0ff180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dc8835a372da7dd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:05.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.078+0000 7f228a7fc640 1 -- 172.21.15.169:0/1078553470 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229c0d22a0 con 0x559fd846db30 2024-03-28T01:00:05.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.078+0000 7f228a7fc640 1 -- 172.21.15.169:0/1078553470 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f229c0d68f0 con 0x559fd846db30 2024-03-28T01:00:05.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.078+0000 7f228a7fc640 1 -- 172.21.15.169:0/1078553470 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229c0ca950 con 0x559fd846db30 2024-03-28T01:00:05.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.080+0000 7f22ad769740 1 -- 172.21.15.169:0/1078553470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd846db30 msgr2=0x559fd846c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:05.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.080+0000 7f22ad769740 1 --2- 172.21.15.169:0/1078553470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd846db30 0x559fd846c6e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x559fd846ef90 tx=0x7f229c0ff180 comp rx=0 tx=0).stop 2024-03-28T01:00:05.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.080+0000 7f22ad769740 1 -- 172.21.15.169:0/1078553470 shutdown_connections 2024-03-28T01:00:05.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.080+0000 7f22ad769740 1 --2- 172.21.15.169:0/1078553470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fd8473690 0x559fd846cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.080+0000 7f22ad769740 1 --2- 172.21.15.169:0/1078553470 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd846e520 0x559fd847f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.080+0000 7f22ad769740 1 --2- 172.21.15.169:0/1078553470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd846db30 0x559fd846c6e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.080+0000 7f22ad769740 1 -- 172.21.15.169:0/1078553470 >> 172.21.15.169:0/1078553470 conn(0x559fd84361a0 msgr2=0x559fd846fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:05.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.080+0000 7f22ad769740 1 -- 172.21.15.169:0/1078553470 shutdown_connections 2024-03-28T01:00:05.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.080+0000 7f22ad769740 1 -- 172.21.15.169:0/1078553470 wait complete. 2024-03-28T01:00:05.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.080+0000 7f22ad769740 1 -- start start 2024-03-28T01:00:05.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.081+0000 7f22ad769740 1 Processor -- start 2024-03-28T01:00:05.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.081+0000 7f22ad769740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd846db30 0x559fd847f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.081+0000 7f22ad769740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fd846e520 0x559fd847d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.081+0000 7f22ad769740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd8473690 0x559fd847dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.081+0000 7f22ad769740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a40832a0 con 0x559fd8473690 2024-03-28T01:00:05.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.081+0000 7f22ad769740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a4084580 con 0x559fd846e520 2024-03-28T01:00:05.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.081+0000 7f22ad769740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a40838c0 con 0x559fd846db30 2024-03-28T01:00:05.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.081+0000 7f22ad769740 10 client.0 _ll_register_callbacks cb 0x7f22980099e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:05.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.081+0000 7f22a911b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd8473690 0x559fd847dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.081+0000 7f22a911b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd8473690 0x559fd847dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47344/0 (socket says 172.21.15.169:47344) 2024-03-28T01:00:05.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.081+0000 7f22a911b640 1 -- 172.21.15.169:0/4159901613 learned_addr learned my addr 172.21.15.169:0/4159901613 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:05.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.082+0000 7f22a3fff640 1 --2- 172.21.15.169:0/4159901613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fd846e520 0x559fd847d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.082+0000 7f22a911b640 1 -- 172.21.15.169:0/4159901613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd846db30 msgr2=0x559fd847f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:05.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.082+0000 7f22a911b640 1 --2- 172.21.15.169:0/4159901613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd846db30 0x559fd847f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.082+0000 7f22a911b640 1 -- 172.21.15.169:0/4159901613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fd846e520 msgr2=0x559fd847d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:05.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.082+0000 7f22a911b640 1 --2- 172.21.15.169:0/4159901613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fd846e520 0x559fd847d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.082+0000 7f22a911b640 1 -- 172.21.15.169:0/4159901613 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559fd847e140 con 0x559fd8473690 2024-03-28T01:00:05.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.082+0000 7f22a911b640 1 --2- 172.21.15.169:0/4159901613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd8473690 0x559fd847dc00 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f2294010880 tx=0x7f2294012070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=524ed659a27b1d65 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:05.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.082+0000 7f22887f8640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:00:05.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.082+0000 7f22887f8640 1 -- 172.21.15.169:0/4159901613 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229402b060 con 0x559fd8473690 2024-03-28T01:00:05.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.082+0000 7f22887f8640 1 -- 172.21.15.169:0/4159901613 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f229401cb20 con 0x559fd8473690 2024-03-28T01:00:05.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.082+0000 7f22ad769740 1 -- 172.21.15.169:0/4159901613 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f229c0d68f0 con 0x559fd8473690 2024-03-28T01:00:05.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.083+0000 7f22887f8640 1 -- 172.21.15.169:0/4159901613 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2294032050 con 0x559fd8473690 2024-03-28T01:00:05.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.084+0000 7f22ad769740 10 client.16224 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:05.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.084+0000 7f22ad769740 1 -- 172.21.15.169:0/4159901613 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x559fd84dfed0 con 0x559fd8473690 2024-03-28T01:00:05.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.084+0000 7f22887f8640 1 -- 172.21.15.169:0/4159901613 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f229c0d68f0 con 0x559fd8473690 2024-03-28T01:00:05.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.084+0000 7f22887f8640 1 -- 172.21.15.169:0/4159901613 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f229401a490 con 0x559fd8473690 2024-03-28T01:00:05.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.084+0000 7f22ad769740 10 client.16224 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:05.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.084+0000 7f22ad769740 10 client.16224 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:05.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.084+0000 7f22ad769740 20 client.16224 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:05.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.084+0000 7f22ad769740 1 --2- 172.21.15.169:0/4159901613 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559fd84e07d0 0x559fd84e2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.084+0000 7f22ad769740 4 client.16224 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T01:00:05.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.084+0000 7f22ad769740 1 -- 172.21.15.169:0/4159901613 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x559fd83da900 con 0x559fd84e07d0 2024-03-28T01:00:05.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.086+0000 7f22a891a640 1 --2- 172.21.15.169:0/4159901613 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559fd84e07d0 0x559fd84e2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.087+0000 7f22a891a640 1 --2- 172.21.15.169:0/4159901613 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559fd84e07d0 0x559fd84e2c90 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5601d870ce4bc27c server_cookie=303290424e706f76 in_seq=0 out_seq=0 2024-03-28T01:00:05.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.087+0000 7f22887f8640 10 client.16224 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:05.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.210+0000 7f22887f8640 1 -- 172.21.15.169:0/4159901613 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x559fd83da900 con 0x559fd84e07d0 2024-03-28T01:00:05.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.210+0000 7f22887f8640 10 client.16224 handle_command_reply: tid=0 2024-03-28T01:00:05.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.210+0000 7f22ad769740 1 client.16224 shutdown 2024-03-28T01:00:05.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.210+0000 7f22ad769740 1 -- 172.21.15.169:0/4159901613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd8473690 msgr2=0x559fd847dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:05.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.210+0000 7f22ad769740 1 --2- 172.21.15.169:0/4159901613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd8473690 0x559fd847dc00 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f2294010880 tx=0x7f2294012070 comp rx=0 tx=0).stop 2024-03-28T01:00:05.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.210+0000 7f22ad769740 1 -- 172.21.15.169:0/4159901613 shutdown_connections 2024-03-28T01:00:05.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.210+0000 7f22ad769740 1 --2- 172.21.15.169:0/4159901613 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559fd84e07d0 0x559fd84e2c90 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.210+0000 7f22ad769740 1 --2- 172.21.15.169:0/4159901613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559fd8473690 0x559fd847dc00 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.210+0000 7f22ad769740 1 --2- 172.21.15.169:0/4159901613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fd846e520 0x559fd847d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.210+0000 7f22ad769740 1 --2- 172.21.15.169:0/4159901613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd846db30 0x559fd847f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.210+0000 7f22ad769740 1 -- 172.21.15.169:0/4159901613 >> 172.21.15.169:0/4159901613 conn(0x559fd84361a0 msgr2=0x559fd8467f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:05.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.210+0000 7f22887f8640 0 client.16224 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:05.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.211+0000 7f22ad769740 1 -- 172.21.15.169:0/4159901613 shutdown_connections 2024-03-28T01:00:05.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.211+0000 7f22ad769740 1 -- 172.21.15.169:0/4159901613 wait complete. 2024-03-28T01:00:05.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.211+0000 7f22ad769740 20 client.16224 trim_cache size 0 max 16384 2024-03-28T01:00:05.216 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:00:05.216 INFO:teuthology.orchestra.run.smithi169.stdout: "return_code": 0, 2024-03-28T01:00:05.216 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_tag": "bda6fa8e-1a01-4cfd-83bd-268cef84c120", 2024-03-28T01:00:05.216 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": "asynchronous" 2024-03-28T01:00:05.217 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:00:05.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.213+0000 7f22a991c640 1 -- 172.21.15.169:0/3776152405 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f228c0a2980 msgr2=0x7f228c0a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:05.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.213+0000 7f22a991c640 1 --2- 172.21.15.169:0/3776152405 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f228c0a2980 0x7f228c0a4e40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2294005fd0 tx=0x7f22940046c0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.213+0000 7f22a991c640 1 -- 172.21.15.169:0/3776152405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22a4072ab0 msgr2=0x7f22a419dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:05.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.213+0000 7f22a991c640 1 --2- 172.21.15.169:0/3776152405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22a4072ab0 0x7f22a419dee0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f229c000f80 tx=0x7f229c01a880 comp rx=0 tx=0).stop 2024-03-28T01:00:05.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.213+0000 7f22a991c640 1 -- 172.21.15.169:0/3776152405 shutdown_connections 2024-03-28T01:00:05.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.213+0000 7f22a991c640 1 --2- 172.21.15.169:0/3776152405 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f228c0a2980 0x7f228c0a4e40 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.213+0000 7f22a991c640 1 --2- 172.21.15.169:0/3776152405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22a4109730 0x7f22a4079900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.213+0000 7f22a991c640 1 --2- 172.21.15.169:0/3776152405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22a4108d70 0x7f22a419e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.213+0000 7f22a991c640 1 --2- 172.21.15.169:0/3776152405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22a4072ab0 0x7f22a419dee0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.213+0000 7f22a991c640 1 -- 172.21.15.169:0/3776152405 >> 172.21.15.169:0/3776152405 conn(0x7f22a406e580 msgr2=0x7f22a410bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:05.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.213+0000 7f22a991c640 1 -- 172.21.15.169:0/3776152405 shutdown_connections 2024-03-28T01:00:05.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.213+0000 7f22a991c640 1 -- 172.21.15.169:0/3776152405 wait complete. 2024-03-28T01:00:05.246 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:00:05.247 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:00:05.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.480+0000 7f06ef362640 1 -- 172.21.15.169:0/2930947882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06e0099ab0 msgr2=0x7f06e00a03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:05.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.480+0000 7f06ef362640 1 --2- 172.21.15.169:0/2930947882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06e0099ab0 0x7f06e00a03d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f06e806b090 tx=0x7f06e80973a0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.481+0000 7f06ef362640 1 -- 172.21.15.169:0/2930947882 shutdown_connections 2024-03-28T01:00:05.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.481+0000 7f06ef362640 1 --2- 172.21.15.169:0/2930947882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06e0099ab0 0x7f06e00a03d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.481+0000 7f06ef362640 1 --2- 172.21.15.169:0/2930947882 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06e00990f0 0x7f06e0099570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.481+0000 7f06ef362640 1 --2- 172.21.15.169:0/2930947882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06e0097f20 0x7f06e0098360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.481+0000 7f06ef362640 1 -- 172.21.15.169:0/2930947882 >> 172.21.15.169:0/2930947882 conn(0x7f06e00934e0 msgr2=0x7f06e0095920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:05.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.481+0000 7f06ef362640 1 -- 172.21.15.169:0/2930947882 shutdown_connections 2024-03-28T01:00:05.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.481+0000 7f06ef362640 1 -- 172.21.15.169:0/2930947882 wait complete. 2024-03-28T01:00:05.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.482+0000 7f06ef362640 1 Processor -- start 2024-03-28T01:00:05.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.482+0000 7f06ef362640 1 -- start start 2024-03-28T01:00:05.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.482+0000 7f06ef362640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06e0097f20 0x7f06e00a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.482+0000 7f06ef362640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06e00990f0 0x7f06e00a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.483+0000 7f06ef362640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06e0099ab0 0x7f06e00aa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.483+0000 7f06ef362640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e0009b60 con 0x7f06e0099ab0 2024-03-28T01:00:05.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.483+0000 7f06ef362640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e00099e0 con 0x7f06e0097f20 2024-03-28T01:00:05.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.483+0000 7f06ef362640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e0009ce0 con 0x7f06e00990f0 2024-03-28T01:00:05.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.483+0000 7f06eeb61640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06e0099ab0 0x7f06e00aa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.483+0000 7f06eeb61640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06e0099ab0 0x7f06e00aa6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47360/0 (socket says 172.21.15.169:47360) 2024-03-28T01:00:05.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.483+0000 7f06eeb61640 1 -- 172.21.15.169:0/2966491820 learned_addr learned my addr 172.21.15.169:0/2966491820 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:05.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.483+0000 7f06ee360640 1 --2- 172.21.15.169:0/2966491820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06e0097f20 0x7f06e00a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.484+0000 7f06edb5f640 1 --2- 172.21.15.169:0/2966491820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06e00990f0 0x7f06e00a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.484+0000 7f06eeb61640 1 -- 172.21.15.169:0/2966491820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06e00990f0 msgr2=0x7f06e00a5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:05.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.484+0000 7f06eeb61640 1 --2- 172.21.15.169:0/2966491820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06e00990f0 0x7f06e00a5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.484+0000 7f06eeb61640 1 -- 172.21.15.169:0/2966491820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06e0097f20 msgr2=0x7f06e00a5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:05.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.484+0000 7f06eeb61640 1 --2- 172.21.15.169:0/2966491820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06e0097f20 0x7f06e00a5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.484+0000 7f06eeb61640 1 -- 172.21.15.169:0/2966491820 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e00aad30 con 0x7f06e0099ab0 2024-03-28T01:00:05.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.484+0000 7f06edb5f640 1 --2- 172.21.15.169:0/2966491820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06e00990f0 0x7f06e00a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:00:05.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.484+0000 7f06eeb61640 1 --2- 172.21.15.169:0/2966491820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06e0099ab0 0x7f06e00aa6c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f06e8050670 tx=0x7f06e809df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:05.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.485+0000 7f06df7fe640 1 -- 172.21.15.169:0/2966491820 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e80ad020 con 0x7f06e0099ab0 2024-03-28T01:00:05.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.485+0000 7f06df7fe640 1 -- 172.21.15.169:0/2966491820 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f06e80696a0 con 0x7f06e0099ab0 2024-03-28T01:00:05.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.485+0000 7f06ef362640 1 -- 172.21.15.169:0/2966491820 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06e00aafc0 con 0x7f06e0099ab0 2024-03-28T01:00:05.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.485+0000 7f06ef362640 1 -- 172.21.15.169:0/2966491820 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06e00ab560 con 0x7f06e0099ab0 2024-03-28T01:00:05.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.485+0000 7f06df7fe640 1 -- 172.21.15.169:0/2966491820 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e80a98a0 con 0x7f06e0099ab0 2024-03-28T01:00:05.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.490+0000 7f06f31ae740 1 Processor -- start 2024-03-28T01:00:05.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.491+0000 7f06df7fe640 1 -- 172.21.15.169:0/2966491820 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f06e80b0020 con 0x7f06e0099ab0 2024-03-28T01:00:05.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.492+0000 7f06f31ae740 1 -- start start 2024-03-28T01:00:05.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.493+0000 7f06df7fe640 1 --2- 172.21.15.169:0/2966491820 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f06d40a29d0 0x7f06d40a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.494+0000 7f06df7fe640 1 -- 172.21.15.169:0/2966491820 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f06e815cb40 con 0x7f06e0099ab0 2024-03-28T01:00:05.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.494+0000 7f06f31ae740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d3fa0 0x56439e6d43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.494+0000 7f06f31ae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d4990 0x56439e6d9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.494+0000 7f06f31ae740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 0x56439e6dbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.495+0000 7f06f31ae740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56439e6393d0 con 0x56439e6d9b00 2024-03-28T01:00:05.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.495+0000 7f06f31ae740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x56439e5e1ad0 con 0x56439e6d3fa0 2024-03-28T01:00:05.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.495+0000 7f06f31ae740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56439e635470 con 0x56439e6d4990 2024-03-28T01:00:05.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.496+0000 7f06eeb61640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 0x56439e6dbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.496+0000 7f06edb5f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d4990 0x56439e6d9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.496+0000 7f06eeb61640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 0x56439e6dbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47376/0 (socket says 172.21.15.169:47376) 2024-03-28T01:00:05.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.496+0000 7f06edb5f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d4990 0x56439e6d9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:46120/0 (socket says 172.21.15.169:46120) 2024-03-28T01:00:05.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.496+0000 7f06eeb61640 1 -- 172.21.15.169:0/2584706272 learned_addr learned my addr 172.21.15.169:0/2584706272 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:05.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.496+0000 7f06edb5f640 1 --2- 172.21.15.169:0/2966491820 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f06d40a29d0 0x7f06d40a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.498+0000 7f06ee360640 1 --2- 172.21.15.169:0/2584706272 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d3fa0 0x56439e6d43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.499+0000 7f06eeb61640 1 -- 172.21.15.169:0/2584706272 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d4990 msgr2=0x56439e6d9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:05.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.499+0000 7f06eeb61640 1 --2- 172.21.15.169:0/2584706272 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d4990 0x56439e6d9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.499+0000 7f06eeb61640 1 -- 172.21.15.169:0/2584706272 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d3fa0 msgr2=0x56439e6d43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:05.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.499+0000 7f06eeb61640 1 --2- 172.21.15.169:0/2584706272 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d3fa0 0x56439e6d43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.499+0000 7f06eeb61640 1 -- 172.21.15.169:0/2584706272 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56439e4177d0 con 0x56439e6d9b00 2024-03-28T01:00:05.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.500+0000 7f06eeb61640 1 --2- 172.21.15.169:0/2584706272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 0x56439e6dbf90 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f06e8129070 tx=0x7f06e8067540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84973fb63954e028 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:05.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.500+0000 7f06dcbfe640 1 -- 172.21.15.169:0/2584706272 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e812d230 con 0x56439e6d9b00 2024-03-28T01:00:05.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.500+0000 7f06dcbfe640 1 -- 172.21.15.169:0/2584706272 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f06e8131880 con 0x56439e6d9b00 2024-03-28T01:00:05.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.500+0000 7f06dcbfe640 1 -- 172.21.15.169:0/2584706272 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e80a5340 con 0x56439e6d9b00 2024-03-28T01:00:05.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.503+0000 7f06edb5f640 1 --2- 172.21.15.169:0/2966491820 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f06d40a29d0 0x7f06d40a4e90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f06d8005d00 tx=0x7f06d800b6c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:05.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.506+0000 7f06f31ae740 1 -- 172.21.15.169:0/2584706272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 msgr2=0x56439e6dbf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:05.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.506+0000 7f06f31ae740 1 --2- 172.21.15.169:0/2584706272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 0x56439e6dbf90 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f06e8129070 tx=0x7f06e8067540 comp rx=0 tx=0).stop 2024-03-28T01:00:05.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.506+0000 7f06f31ae740 1 -- 172.21.15.169:0/2584706272 shutdown_connections 2024-03-28T01:00:05.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.506+0000 7f06f31ae740 1 --2- 172.21.15.169:0/2584706272 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 0x56439e6dbf90 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.506+0000 7f06f31ae740 1 --2- 172.21.15.169:0/2584706272 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d4990 0x56439e6d9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.506+0000 7f06f31ae740 1 --2- 172.21.15.169:0/2584706272 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d3fa0 0x56439e6d43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.506+0000 7f06f31ae740 1 -- 172.21.15.169:0/2584706272 >> 172.21.15.169:0/2584706272 conn(0x56439e6cf510 msgr2=0x56439e6cf910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:05.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.508+0000 7f06f31ae740 1 -- 172.21.15.169:0/2584706272 shutdown_connections 2024-03-28T01:00:05.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.512+0000 7f06f31ae740 1 -- 172.21.15.169:0/2584706272 wait complete. 2024-03-28T01:00:05.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.512+0000 7f06f31ae740 1 -- start start 2024-03-28T01:00:05.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.512+0000 7f06f31ae740 1 Processor -- start 2024-03-28T01:00:05.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.513+0000 7f06f31ae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d3fa0 0x56439e6dde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.513+0000 7f06f31ae740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d4990 0x56439e6de340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.513+0000 7f06f31ae740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 0x56439e6e5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.513+0000 7f06f31ae740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56439e6393d0 con 0x56439e6d9b00 2024-03-28T01:00:05.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.513+0000 7f06f31ae740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x56439e5f5060 con 0x56439e6d4990 2024-03-28T01:00:05.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.513+0000 7f06f31ae740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56439e6dc870 con 0x56439e6d3fa0 2024-03-28T01:00:05.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.514+0000 7f06f31ae740 10 client.0 _ll_register_callbacks cb 0x56439e5d76a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:05.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.515+0000 7f06edb5f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d4990 0x56439e6de340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.517+0000 7f06eeb61640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 0x56439e6e5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.517+0000 7f06eeb61640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 0x56439e6e5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47388/0 (socket says 172.21.15.169:47388) 2024-03-28T01:00:05.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.517+0000 7f06eeb61640 1 -- 172.21.15.169:0/3984889301 learned_addr learned my addr 172.21.15.169:0/3984889301 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:05.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.518+0000 7f06ee360640 1 --2- 172.21.15.169:0/3984889301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d3fa0 0x56439e6dde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.518+0000 7f06eeb61640 1 -- 172.21.15.169:0/3984889301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d3fa0 msgr2=0x56439e6dde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:05.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.518+0000 7f06eeb61640 1 --2- 172.21.15.169:0/3984889301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d3fa0 0x56439e6dde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.518+0000 7f06eeb61640 1 -- 172.21.15.169:0/3984889301 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d4990 msgr2=0x56439e6de340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:05.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.518+0000 7f06eeb61640 1 --2- 172.21.15.169:0/3984889301 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d4990 0x56439e6de340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.518+0000 7f06eeb61640 1 -- 172.21.15.169:0/3984889301 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56439e34d3f0 con 0x56439e6d9b00 2024-03-28T01:00:05.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.519+0000 7f06eeb61640 1 --2- 172.21.15.169:0/3984889301 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 0x56439e6e5380 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f06e8066ec0 tx=0x7f06e80973a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7ceef8e266a467d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:05.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.522+0000 7f06d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:00:05.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.522+0000 7f06d1ffb640 1 -- 172.21.15.169:0/3984889301 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e80a5550 con 0x56439e6d9b00 2024-03-28T01:00:05.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.522+0000 7f06d1ffb640 1 -- 172.21.15.169:0/3984889301 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f06e8147be0 con 0x56439e6d9b00 2024-03-28T01:00:05.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.525+0000 7f06f31ae740 1 -- 172.21.15.169:0/3984889301 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f06e8131880 con 0x56439e6d9b00 2024-03-28T01:00:05.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.531+0000 7f06d1ffb640 1 -- 172.21.15.169:0/3984889301 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f06e812d230 con 0x56439e6d9b00 2024-03-28T01:00:05.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.531+0000 7f06f31ae740 10 client.16236 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:05.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.531+0000 7f06f31ae740 1 -- 172.21.15.169:0/3984889301 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56439e30d0f0 con 0x56439e6d9b00 2024-03-28T01:00:05.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.533+0000 7f06d1ffb640 1 -- 172.21.15.169:0/3984889301 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f06e8131880 con 0x56439e6d9b00 2024-03-28T01:00:05.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.533+0000 7f06d1ffb640 1 -- 172.21.15.169:0/3984889301 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f06e8128530 con 0x56439e6d9b00 2024-03-28T01:00:05.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.535+0000 7f06f31ae740 10 client.16236 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:05.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.535+0000 7f06f31ae740 10 client.16236 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:05.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.535+0000 7f06f31ae740 20 client.16236 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:05.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.535+0000 7f06f31ae740 1 --2- 172.21.15.169:0/3984889301 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56439e6e9580 0x56439e6eb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.535+0000 7f06f31ae740 4 client.16236 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:00:05.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.535+0000 7f06f31ae740 1 -- 172.21.15.169:0/3984889301 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56439e646910 con 0x56439e6e9580 2024-03-28T01:00:05.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.536+0000 7f06ee360640 1 --2- 172.21.15.169:0/3984889301 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56439e6e9580 0x56439e6eb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.538+0000 7f06ee360640 1 --2- 172.21.15.169:0/3984889301 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56439e6e9580 0x56439e6eb970 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd7deb01b5aac424 server_cookie=e87538c32df1f1f2 in_seq=0 out_seq=0 2024-03-28T01:00:05.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.538+0000 7f06d1ffb640 10 client.16236 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:05.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.542+0000 7f06d1ffb640 1 -- 172.21.15.169:0/3984889301 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56439e646910 con 0x56439e6e9580 2024-03-28T01:00:05.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.542+0000 7f06d1ffb640 10 client.16236 handle_command_reply: tid=0 2024-03-28T01:00:05.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.542+0000 7f06f31ae740 1 client.16236 shutdown 2024-03-28T01:00:05.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.544+0000 7f06f31ae740 1 -- 172.21.15.169:0/3984889301 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 msgr2=0x56439e6e5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:05.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.544+0000 7f06f31ae740 1 --2- 172.21.15.169:0/3984889301 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 0x56439e6e5380 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f06e8066ec0 tx=0x7f06e80973a0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.544+0000 7f06f31ae740 1 -- 172.21.15.169:0/3984889301 shutdown_connections 2024-03-28T01:00:05.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.544+0000 7f06f31ae740 1 --2- 172.21.15.169:0/3984889301 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56439e6e9580 0x56439e6eb970 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.544+0000 7f06f31ae740 1 --2- 172.21.15.169:0/3984889301 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 0x56439e6e5380 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.544+0000 7f06f31ae740 1 --2- 172.21.15.169:0/3984889301 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d4990 0x56439e6de340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.544+0000 7f06f31ae740 1 --2- 172.21.15.169:0/3984889301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d3fa0 0x56439e6dde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.544+0000 7f06f31ae740 1 -- 172.21.15.169:0/3984889301 >> 172.21.15.169:0/3984889301 conn(0x56439e6cf510 msgr2=0x56439e6ce7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:05.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.545+0000 7f06d1ffb640 0 client.16236 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:05.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.548+0000 7f06f31ae740 1 -- 172.21.15.169:0/3984889301 shutdown_connections 2024-03-28T01:00:05.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.549+0000 7f06f31ae740 1 -- 172.21.15.169:0/3984889301 wait complete. 2024-03-28T01:00:05.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.549+0000 7f06f31ae740 20 client.16236 trim_cache size 0 max 16384 2024-03-28T01:00:05.554 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:05 smithi169 ceph-mon[32274]: pgmap v458: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 789 MiB/s wr, 1.28k op/s 2024-03-28T01:00:05.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.580+0000 7f06f31ae740 1 Processor -- start 2024-03-28T01:00:05.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.580+0000 7f06f31ae740 1 -- start start 2024-03-28T01:00:05.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.580+0000 7f06f31ae740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d3fa0 0x56439e6d5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.580+0000 7f06f31ae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d4990 0x56439e6d61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.580+0000 7f06f31ae740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 0x56439e6e5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.580+0000 7f06ee360640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d3fa0 0x56439e6d5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.580+0000 7f06edb5f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d4990 0x56439e6d61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.580+0000 7f06ee360640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d3fa0 0x56439e6d5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:41522/0 (socket says 172.21.15.169:41522) 2024-03-28T01:00:05.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.580+0000 7f06eeb61640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 0x56439e6e5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.580+0000 7f06ee360640 1 -- 172.21.15.169:0/319552368 learned_addr learned my addr 172.21.15.169:0/319552368 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:05.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.580+0000 7f06f31ae740 1 -- 172.21.15.169:0/319552368 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e013ecc0 con 0x56439e6d9b00 2024-03-28T01:00:05.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.581+0000 7f06f31ae740 1 -- 172.21.15.169:0/319552368 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e013c7c0 con 0x56439e6d3fa0 2024-03-28T01:00:05.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.581+0000 7f06f31ae740 1 -- 172.21.15.169:0/319552368 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e013a890 con 0x56439e6d4990 2024-03-28T01:00:05.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.581+0000 7f06edb5f640 1 -- 172.21.15.169:0/319552368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d3fa0 msgr2=0x56439e6d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:05.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.581+0000 7f06edb5f640 1 --2- 172.21.15.169:0/319552368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d3fa0 0x56439e6d5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.581+0000 7f06edb5f640 1 -- 172.21.15.169:0/319552368 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 msgr2=0x56439e6e5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:05.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.581+0000 7f06edb5f640 1 --2- 172.21.15.169:0/319552368 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 0x56439e6e5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.581+0000 7f06edb5f640 1 -- 172.21.15.169:0/319552368 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56439e6dbf50 con 0x56439e6d4990 2024-03-28T01:00:05.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.582+0000 7f06edb5f640 1 --2- 172.21.15.169:0/319552368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d4990 0x56439e6d61b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x56439e6d4d90 tx=0x7f06d801e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a40d3a2c309f2b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:05.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.583+0000 7f06d3fff640 1 -- 172.21.15.169:0/319552368 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d801f8c0 con 0x56439e6d4990 2024-03-28T01:00:05.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.583+0000 7f06d3fff640 1 -- 172.21.15.169:0/319552368 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f06d80293f0 con 0x56439e6d4990 2024-03-28T01:00:05.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.583+0000 7f06d3fff640 1 -- 172.21.15.169:0/319552368 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d8028600 con 0x56439e6d4990 2024-03-28T01:00:05.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.584+0000 7f06f31ae740 1 -- 172.21.15.169:0/319552368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d4990 msgr2=0x56439e6d61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:05.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.584+0000 7f06f31ae740 1 --2- 172.21.15.169:0/319552368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d4990 0x56439e6d61b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x56439e6d4d90 tx=0x7f06d801e000 comp rx=0 tx=0).stop 2024-03-28T01:00:05.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.585+0000 7f06f31ae740 1 -- 172.21.15.169:0/319552368 shutdown_connections 2024-03-28T01:00:05.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.585+0000 7f06f31ae740 1 --2- 172.21.15.169:0/319552368 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 0x56439e6e5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.585+0000 7f06f31ae740 1 --2- 172.21.15.169:0/319552368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d4990 0x56439e6d61b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.585+0000 7f06f31ae740 1 --2- 172.21.15.169:0/319552368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d3fa0 0x56439e6d5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.585+0000 7f06f31ae740 1 -- 172.21.15.169:0/319552368 >> 172.21.15.169:0/319552368 conn(0x56439e6cf510 msgr2=0x56439e6dde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:05.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.586+0000 7f06f31ae740 1 -- 172.21.15.169:0/319552368 shutdown_connections 2024-03-28T01:00:05.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.586+0000 7f06f31ae740 1 -- 172.21.15.169:0/319552368 wait complete. 2024-03-28T01:00:05.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.587+0000 7f06f31ae740 1 -- start start 2024-03-28T01:00:05.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.587+0000 7f06f31ae740 1 Processor -- start 2024-03-28T01:00:05.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.588+0000 7f06f31ae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d3fa0 0x56439e6e2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.588+0000 7f06f31ae740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d4990 0x56439e6e30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.588+0000 7f06f31ae740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 0x56439e6e40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.588+0000 7f06edb5f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d4990 0x56439e6e30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.588+0000 7f06edb5f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d4990 0x56439e6e30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:41526/0 (socket says 172.21.15.169:41526) 2024-03-28T01:00:05.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.588+0000 7f06edb5f640 1 -- 172.21.15.169:0/2582991659 learned_addr learned my addr 172.21.15.169:0/2582991659 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:05.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.588+0000 7f06f31ae740 1 -- 172.21.15.169:0/2582991659 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e013ae60 con 0x56439e6d9b00 2024-03-28T01:00:05.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.588+0000 7f06f31ae740 1 -- 172.21.15.169:0/2582991659 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e013c160 con 0x56439e6d4990 2024-03-28T01:00:05.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.588+0000 7f06f31ae740 1 -- 172.21.15.169:0/2582991659 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e013b4a0 con 0x56439e6d3fa0 2024-03-28T01:00:05.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.588+0000 7f06eeb61640 1 --2- 172.21.15.169:0/2582991659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 0x56439e6e40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.588+0000 7f06f31ae740 10 client.0 _ll_register_callbacks cb 0x7f06d8013400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:05.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.588+0000 7f06ee360640 1 --2- 172.21.15.169:0/2582991659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d3fa0 0x56439e6e2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.588+0000 7f06edb5f640 1 -- 172.21.15.169:0/2582991659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d3fa0 msgr2=0x56439e6e2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:05.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.589+0000 7f06edb5f640 1 --2- 172.21.15.169:0/2582991659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d3fa0 0x56439e6e2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.589+0000 7f06edb5f640 1 -- 172.21.15.169:0/2582991659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 msgr2=0x56439e6e40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:05.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.589+0000 7f06edb5f640 1 --2- 172.21.15.169:0/2582991659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 0x56439e6e40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.589+0000 7f06edb5f640 1 -- 172.21.15.169:0/2582991659 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56439e6e46e0 con 0x56439e6d4990 2024-03-28T01:00:05.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.589+0000 7f06edb5f640 1 --2- 172.21.15.169:0/2582991659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d4990 0x56439e6e30e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f06d8029fd0 tx=0x7f06d801faf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d47328ea573d3db2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:05.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.589+0000 7f06d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:00:05.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.589+0000 7f06d1ffb640 1 -- 172.21.15.169:0/2582991659 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d8039020 con 0x56439e6d4990 2024-03-28T01:00:05.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.589+0000 7f06d1ffb640 1 -- 172.21.15.169:0/2582991659 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f06d80297a0 con 0x56439e6d4990 2024-03-28T01:00:05.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.589+0000 7f06f31ae740 1 -- 172.21.15.169:0/2582991659 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f06d80293f0 con 0x56439e6d4990 2024-03-28T01:00:05.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.590+0000 7f06d1ffb640 1 -- 172.21.15.169:0/2582991659 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f06d8044050 con 0x56439e6d4990 2024-03-28T01:00:05.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.590+0000 7f06f31ae740 10 client.26090 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:05.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.590+0000 7f06f31ae740 1 -- 172.21.15.169:0/2582991659 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56439e744f80 con 0x56439e6d4990 2024-03-28T01:00:05.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.590+0000 7f06d1ffb640 1 -- 172.21.15.169:0/2582991659 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f06d80293f0 con 0x56439e6d4990 2024-03-28T01:00:05.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.590+0000 7f06d1ffb640 1 -- 172.21.15.169:0/2582991659 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f06d800f040 con 0x56439e6d4990 2024-03-28T01:00:05.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.590+0000 7f06f31ae740 10 client.26090 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:05.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.590+0000 7f06f31ae740 10 client.26090 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:05.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.590+0000 7f06f31ae740 20 client.26090 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:05.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.590+0000 7f06f31ae740 1 --2- 172.21.15.169:0/2582991659 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56439e745820 0x56439e747c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:05.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.590+0000 7f06f31ae740 4 client.26090 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:00:05.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.590+0000 7f06f31ae740 1 -- 172.21.15.169:0/2582991659 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56439e640c60 con 0x56439e745820 2024-03-28T01:00:05.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.591+0000 7f06ee360640 1 --2- 172.21.15.169:0/2582991659 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56439e745820 0x56439e747c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:05.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.591+0000 7f06d1ffb640 10 client.26090 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:05.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.591+0000 7f06ee360640 1 --2- 172.21.15.169:0/2582991659 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56439e745820 0x56439e747c40 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15d7b897b3d17bff server_cookie=47cd40aac96f0ca in_seq=0 out_seq=0 2024-03-28T01:00:05.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.594+0000 7f06d1ffb640 1 -- 172.21.15.169:0/2582991659 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x56439e640c60 con 0x56439e745820 2024-03-28T01:00:05.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.594+0000 7f06d1ffb640 10 client.26090 handle_command_reply: tid=0 2024-03-28T01:00:05.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.594+0000 7f06f31ae740 1 client.26090 shutdown 2024-03-28T01:00:05.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.594+0000 7f06f31ae740 1 -- 172.21.15.169:0/2582991659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d4990 msgr2=0x56439e6e30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:05.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.594+0000 7f06f31ae740 1 --2- 172.21.15.169:0/2582991659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d4990 0x56439e6e30e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f06d8029fd0 tx=0x7f06d801faf0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.594+0000 7f06f31ae740 1 -- 172.21.15.169:0/2582991659 shutdown_connections 2024-03-28T01:00:05.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.594+0000 7f06f31ae740 1 --2- 172.21.15.169:0/2582991659 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56439e745820 0x56439e747c40 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.594+0000 7f06f31ae740 1 --2- 172.21.15.169:0/2582991659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56439e6d9b00 0x56439e6e40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.594+0000 7f06f31ae740 1 --2- 172.21.15.169:0/2582991659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56439e6d4990 0x56439e6e30e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.594+0000 7f06f31ae740 1 --2- 172.21.15.169:0/2582991659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56439e6d3fa0 0x56439e6e2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.594+0000 7f06f31ae740 1 -- 172.21.15.169:0/2582991659 >> 172.21.15.169:0/2582991659 conn(0x56439e6cf510 msgr2=0x56439e6ddd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:05.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.594+0000 7f06d1ffb640 0 client.26090 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:05.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.594+0000 7f06f31ae740 1 -- 172.21.15.169:0/2582991659 shutdown_connections 2024-03-28T01:00:05.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.594+0000 7f06f31ae740 1 -- 172.21.15.169:0/2582991659 wait complete. 2024-03-28T01:00:05.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.594+0000 7f06f31ae740 20 client.26090 trim_cache size 0 max 16384 2024-03-28T01:00:05.600 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:00:05.600 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "scrub active (7 inodes in the stack)", 2024-03-28T01:00:05.600 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": { 2024-03-28T01:00:05.600 INFO:teuthology.orchestra.run.smithi169.stdout: "bda6fa8e-1a01-4cfd-83bd-268cef84c120": { 2024-03-28T01:00:05.600 INFO:teuthology.orchestra.run.smithi169.stdout: "path": "/", 2024-03-28T01:00:05.600 INFO:teuthology.orchestra.run.smithi169.stdout: "tag": "bda6fa8e-1a01-4cfd-83bd-268cef84c120", 2024-03-28T01:00:05.600 INFO:teuthology.orchestra.run.smithi169.stdout: "options": "recursive,force" 2024-03-28T01:00:05.600 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:00:05.600 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:00:05.600 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:00:05.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.595+0000 7f06ef362640 1 -- 172.21.15.169:0/2966491820 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f06d40a29d0 msgr2=0x7f06d40a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:05.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.595+0000 7f06ef362640 1 --2- 172.21.15.169:0/2966491820 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f06d40a29d0 0x7f06d40a4e90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f06d8005d00 tx=0x7f06d800b6c0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.595+0000 7f06ef362640 1 -- 172.21.15.169:0/2966491820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06e0099ab0 msgr2=0x7f06e00aa6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:05.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.595+0000 7f06ef362640 1 --2- 172.21.15.169:0/2966491820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06e0099ab0 0x7f06e00aa6c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f06e8050670 tx=0x7f06e809df80 comp rx=0 tx=0).stop 2024-03-28T01:00:05.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.595+0000 7f06ef362640 1 -- 172.21.15.169:0/2966491820 shutdown_connections 2024-03-28T01:00:05.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.595+0000 7f06ef362640 1 --2- 172.21.15.169:0/2966491820 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f06d40a29d0 0x7f06d40a4e90 secure :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f06d8005d00 tx=0x7f06d800b6c0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.595+0000 7f06ef362640 1 --2- 172.21.15.169:0/2966491820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06e0099ab0 0x7f06e00aa6c0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.595+0000 7f06ef362640 1 --2- 172.21.15.169:0/2966491820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06e00990f0 0x7f06e00a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.595+0000 7f06ef362640 1 --2- 172.21.15.169:0/2966491820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06e0097f20 0x7f06e00a5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:05.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.595+0000 7f06ef362640 1 -- 172.21.15.169:0/2966491820 >> 172.21.15.169:0/2966491820 conn(0x7f06e00934e0 msgr2=0x7f06e000c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:05.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.595+0000 7f06ef362640 1 -- 172.21.15.169:0/2966491820 shutdown_connections 2024-03-28T01:00:05.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:05.595+0000 7f06ef362640 1 -- 172.21.15.169:0/2966491820 wait complete. 2024-03-28T01:00:05.622 INFO:tasks.cephfs.filesystem:scrub status for tag:bda6fa8e-1a01-4cfd-83bd-268cef84c120 - {'path': '/', 'tag': 'bda6fa8e-1a01-4cfd-83bd-268cef84c120', 'options': 'recursive,force'} 2024-03-28T01:00:05.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:05 smithi174 ceph-mon[36060]: pgmap v458: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 789 MiB/s wr, 1.28k op/s 2024-03-28T01:00:05.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:05 smithi177 ceph-mon[36206]: pgmap v458: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 789 MiB/s wr, 1.28k op/s 2024-03-28T01:00:06.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:06 smithi169 ceph-mon[32274]: scrub queued for path: / 2024-03-28T01:00:06.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:06 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:00:06.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:06 smithi169 ceph-mon[32274]: scrub summary: active paths [/] 2024-03-28T01:00:06.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:06 smithi169 ceph-mon[32274]: pgmap v459: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 835 MiB/s wr, 1.35k op/s 2024-03-28T01:00:06.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:06 smithi174 ceph-mon[36060]: scrub queued for path: / 2024-03-28T01:00:06.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:06 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:00:06.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:06 smithi174 ceph-mon[36060]: scrub summary: active paths [/] 2024-03-28T01:00:06.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:06 smithi174 ceph-mon[36060]: pgmap v459: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 835 MiB/s wr, 1.35k op/s 2024-03-28T01:00:06.904 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:06 smithi177 ceph-mon[36206]: scrub queued for path: / 2024-03-28T01:00:06.904 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:06 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:00:06.904 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:06 smithi177 ceph-mon[36206]: scrub summary: active paths [/] 2024-03-28T01:00:06.905 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:06 smithi177 ceph-mon[36206]: pgmap v459: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 835 MiB/s wr, 1.35k op/s 2024-03-28T01:00:08.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:07 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:00:08.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:07 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:00:08.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:07 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:00:09.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:09 smithi169 ceph-mon[32274]: pgmap v460: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 794 MiB/s wr, 1.31k op/s 2024-03-28T01:00:09.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:09 smithi169 ceph-mon[32274]: scrub summary: idle 2024-03-28T01:00:09.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:09 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:00:09.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:09 smithi174 ceph-mon[36060]: pgmap v460: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 794 MiB/s wr, 1.31k op/s 2024-03-28T01:00:09.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:09 smithi174 ceph-mon[36060]: scrub summary: idle 2024-03-28T01:00:09.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:09 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:00:09.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:09 smithi177 ceph-mon[36206]: pgmap v460: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 794 MiB/s wr, 1.31k op/s 2024-03-28T01:00:09.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:09 smithi177 ceph-mon[36206]: scrub summary: idle 2024-03-28T01:00:09.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:09 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:00:11.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:11 smithi169 ceph-mon[32274]: pgmap v461: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 753 MiB/s wr, 1.25k op/s 2024-03-28T01:00:11.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:11 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:00:11.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:11 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:00:11.551 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:11 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:00:11.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:11 smithi174 ceph-mon[36060]: pgmap v461: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 753 MiB/s wr, 1.25k op/s 2024-03-28T01:00:11.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:11 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:00:11.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:11 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:00:11.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:11 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:00:11.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:11 smithi177 ceph-mon[36206]: pgmap v461: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 753 MiB/s wr, 1.25k op/s 2024-03-28T01:00:11.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:11 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:00:11.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:11 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:00:11.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:11 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:00:13.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:13 smithi174 ceph-mon[36060]: pgmap v462: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 879 MiB/s wr, 1.42k op/s 2024-03-28T01:00:13.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:13 smithi177 ceph-mon[36206]: pgmap v462: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 879 MiB/s wr, 1.42k op/s 2024-03-28T01:00:13.672 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.668+0000 7f1af3fff640 1 -- 172.21.15.169:0/1675214139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af4108d30 msgr2=0x7f1af41091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:13.672 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.668+0000 7f1af3fff640 1 --2- 172.21.15.169:0/1675214139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af4108d30 0x7f1af41091b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8009860 tx=0x7f1ae802bbb0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.677+0000 7f1af3fff640 1 -- 172.21.15.169:0/1675214139 shutdown_connections 2024-03-28T01:00:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.677+0000 7f1af3fff640 1 --2- 172.21.15.169:0/1675214139 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1af41096f0 0x7f1af410ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.677+0000 7f1af3fff640 1 --2- 172.21.15.169:0/1675214139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af4108d30 0x7f1af41091b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.677+0000 7f1af3fff640 1 --2- 172.21.15.169:0/1675214139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1af4107b40 0x7f1af4107f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.677+0000 7f1af3fff640 1 -- 172.21.15.169:0/1675214139 >> 172.21.15.169:0/1675214139 conn(0x7f1af4076ad0 msgr2=0x7f1af4078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:13.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.683+0000 7f1af3fff640 1 -- 172.21.15.169:0/1675214139 shutdown_connections 2024-03-28T01:00:13.688 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.686+0000 7f1af3fff640 1 -- 172.21.15.169:0/1675214139 wait complete. 2024-03-28T01:00:13.689 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.687+0000 7f1af3fff640 1 Processor -- start 2024-03-28T01:00:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.687+0000 7f1af3fff640 1 -- start start 2024-03-28T01:00:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.687+0000 7f1af3fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af4107b40 0x7f1af407ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.687+0000 7f1af3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1af4108d30 0x7f1af407f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.687+0000 7f1af3fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1af41096f0 0x7f1af41adfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.687+0000 7f1af3fff640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af4113890 con 0x7f1af4108d30 2024-03-28T01:00:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.687+0000 7f1af3fff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af4113710 con 0x7f1af41096f0 2024-03-28T01:00:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.687+0000 7f1af3fff640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af4113a10 con 0x7f1af4107b40 2024-03-28T01:00:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.688+0000 7f1af27fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1af4108d30 0x7f1af407f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.688+0000 7f1af27fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1af4108d30 0x7f1af407f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39792/0 (socket says 172.21.15.169:39792) 2024-03-28T01:00:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.688+0000 7f1af27fc640 1 -- 172.21.15.169:0/1430859052 learned_addr learned my addr 172.21.15.169:0/1430859052 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.689+0000 7f1af37fe640 1 --2- 172.21.15.169:0/1430859052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1af41096f0 0x7f1af41adfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.689+0000 7f1af2ffd640 1 --2- 172.21.15.169:0/1430859052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af4107b40 0x7f1af407ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.691+0000 7f1af2ffd640 1 -- 172.21.15.169:0/1430859052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1af41096f0 msgr2=0x7f1af41adfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.691+0000 7f1af2ffd640 1 --2- 172.21.15.169:0/1430859052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1af41096f0 0x7f1af41adfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.691+0000 7f1af2ffd640 1 -- 172.21.15.169:0/1430859052 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1af4108d30 msgr2=0x7f1af407f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.691+0000 7f1af2ffd640 1 --2- 172.21.15.169:0/1430859052 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1af4108d30 0x7f1af407f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.691+0000 7f1af2ffd640 1 -- 172.21.15.169:0/1430859052 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af41ae4e0 con 0x7f1af4107b40 2024-03-28T01:00:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.693+0000 7f1af2ffd640 1 --2- 172.21.15.169:0/1430859052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af4107b40 0x7f1af407ee80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1aec005fd0 tx=0x7f1aec00c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.698+0000 7f1af88c4640 1 -- 172.21.15.169:0/1430859052 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aec003970 con 0x7f1af4107b40 2024-03-28T01:00:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.698+0000 7f1af3fff640 1 -- 172.21.15.169:0/1430859052 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af41ae6b0 con 0x7f1af4107b40 2024-03-28T01:00:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.698+0000 7f1af3fff640 1 -- 172.21.15.169:0/1430859052 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af41aebf0 con 0x7f1af4107b40 2024-03-28T01:00:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.699+0000 7f1af88c4640 1 -- 172.21.15.169:0/1430859052 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1aec004020 con 0x7f1af4107b40 2024-03-28T01:00:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.699+0000 7f1af88c4640 1 -- 172.21.15.169:0/1430859052 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aec00b990 con 0x7f1af4107b40 2024-03-28T01:00:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.704+0000 7f1af88c4640 1 -- 172.21.15.169:0/1430859052 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1aec01d020 con 0x7f1af4107b40 2024-03-28T01:00:13.716 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.712+0000 7f1afc58f740 1 Processor -- start 2024-03-28T01:00:13.717 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.714+0000 7f1af88c4640 1 --2- 172.21.15.169:0/1430859052 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1adc0a2a00 0x7f1adc0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.715+0000 7f1af88c4640 1 -- 172.21.15.169:0/1430859052 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f1aec0cecc0 con 0x7f1af4107b40 2024-03-28T01:00:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.718+0000 7f1afc58f740 1 -- start start 2024-03-28T01:00:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.720+0000 7f1af37fe640 1 --2- 172.21.15.169:0/1430859052 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1adc0a2a00 0x7f1adc0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.720+0000 7f1afc58f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c597b20 0x55bd3c597f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.720+0000 7f1afc58f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c598510 0x55bd3c59d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.721+0000 7f1afc58f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c59d680 0x55bd3c59fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.721+0000 7f1afc58f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd3c4fcb10 con 0x55bd3c598510 2024-03-28T01:00:13.724 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.721+0000 7f1afc58f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd3c4a5900 con 0x55bd3c597b20 2024-03-28T01:00:13.724 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.721+0000 7f1afc58f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd3c4f8bb0 con 0x55bd3c59d680 2024-03-28T01:00:13.725 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.722+0000 7f1af2ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c597b20 0x55bd3c597f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:13.725 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.722+0000 7f1af2ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c597b20 0x55bd3c597f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:32978/0 (socket says 172.21.15.169:32978) 2024-03-28T01:00:13.725 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.722+0000 7f1af2ffd640 1 -- 172.21.15.169:0/567840872 learned_addr learned my addr 172.21.15.169:0/567840872 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:13.726 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.723+0000 7f1af2ffd640 1 -- 172.21.15.169:0/567840872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c59d680 msgr2=0x55bd3c59fb10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:00:13.726 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.723+0000 7f1af2ffd640 1 --2- 172.21.15.169:0/567840872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c59d680 0x55bd3c59fb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.726 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.723+0000 7f1af2ffd640 1 -- 172.21.15.169:0/567840872 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c598510 msgr2=0x55bd3c59d010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:00:13.727 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.723+0000 7f1af2ffd640 1 --2- 172.21.15.169:0/567840872 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c598510 0x55bd3c59d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.727 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.723+0000 7f1af2ffd640 1 -- 172.21.15.169:0/567840872 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bd3c211210 con 0x55bd3c597b20 2024-03-28T01:00:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.730+0000 7f1af2ffd640 1 --2- 172.21.15.169:0/567840872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c597b20 0x55bd3c597f40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1aec0049d0 tx=0x7f1aec09acf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86b95fd74da30ee2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.732+0000 7f1ad9ffb640 1 -- 172.21.15.169:0/567840872 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aec0a3a00 con 0x55bd3c597b20 2024-03-28T01:00:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.733+0000 7f1ad9ffb640 1 -- 172.21.15.169:0/567840872 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1aec09f160 con 0x55bd3c597b20 2024-03-28T01:00:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.733+0000 7f1ad9ffb640 1 -- 172.21.15.169:0/567840872 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aec0041d0 con 0x55bd3c597b20 2024-03-28T01:00:13.739 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.736+0000 7f1afc58f740 1 -- 172.21.15.169:0/567840872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c597b20 msgr2=0x55bd3c597f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:13.739 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.736+0000 7f1afc58f740 1 --2- 172.21.15.169:0/567840872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c597b20 0x55bd3c597f40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1aec0049d0 tx=0x7f1aec09acf0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.744+0000 7f1afc58f740 1 -- 172.21.15.169:0/567840872 shutdown_connections 2024-03-28T01:00:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.744+0000 7f1afc58f740 1 --2- 172.21.15.169:0/567840872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c59d680 0x55bd3c59fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.744+0000 7f1afc58f740 1 --2- 172.21.15.169:0/567840872 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c598510 0x55bd3c59d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.744+0000 7f1afc58f740 1 --2- 172.21.15.169:0/567840872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c597b20 0x55bd3c597f40 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.744+0000 7f1afc58f740 1 -- 172.21.15.169:0/567840872 >> 172.21.15.169:0/567840872 conn(0x55bd3c593090 msgr2=0x55bd3c593490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.744+0000 7f1af37fe640 1 --2- 172.21.15.169:0/1430859052 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1adc0a2a00 0x7f1adc0a4ec0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4006fd0 tx=0x7f1ae4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.744+0000 7f1afc58f740 1 -- 172.21.15.169:0/567840872 shutdown_connections 2024-03-28T01:00:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.751+0000 7f1afc58f740 1 -- 172.21.15.169:0/567840872 wait complete. 2024-03-28T01:00:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.752+0000 7f1afc58f740 1 -- start start 2024-03-28T01:00:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.753+0000 7f1afc58f740 1 Processor -- start 2024-03-28T01:00:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.754+0000 7f1afc58f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c597b20 0x55bd3c5a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.754+0000 7f1afc58f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c598510 0x55bd3c5a2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.754+0000 7f1afc58f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c59d680 0x55bd3c5a93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.754+0000 7f1afc58f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd3c5a0630 con 0x55bd3c597b20 2024-03-28T01:00:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.754+0000 7f1afc58f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd3c4b8d50 con 0x55bd3c598510 2024-03-28T01:00:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.754+0000 7f1afc58f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd3c5a07b0 con 0x55bd3c59d680 2024-03-28T01:00:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.754+0000 7f1afc58f740 10 client.0 _ll_register_callbacks cb 0x55bd3c4b8720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:13.766 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.759+0000 7f1af2ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c597b20 0x55bd3c5a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.759+0000 7f1af2ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c597b20 0x55bd3c5a1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39806/0 (socket says 172.21.15.169:39806) 2024-03-28T01:00:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.759+0000 7f1af2ffd640 1 -- 172.21.15.169:0/3145191911 learned_addr learned my addr 172.21.15.169:0/3145191911 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.759+0000 7f1af27fc640 1 --2- 172.21.15.169:0/3145191911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c598510 0x55bd3c5a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.759+0000 7f1af37fe640 1 --2- 172.21.15.169:0/3145191911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c59d680 0x55bd3c5a93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.764+0000 7f1af2ffd640 1 -- 172.21.15.169:0/3145191911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c59d680 msgr2=0x55bd3c5a93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.764+0000 7f1af2ffd640 1 --2- 172.21.15.169:0/3145191911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c59d680 0x55bd3c5a93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.764+0000 7f1af2ffd640 1 -- 172.21.15.169:0/3145191911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c598510 msgr2=0x55bd3c5a2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.764+0000 7f1af2ffd640 1 --2- 172.21.15.169:0/3145191911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c598510 0x55bd3c5a2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.764+0000 7f1af2ffd640 1 -- 172.21.15.169:0/3145191911 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bd3c2dd620 con 0x55bd3c597b20 2024-03-28T01:00:13.774 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.771+0000 7f1af2ffd640 1 --2- 172.21.15.169:0/3145191911 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c597b20 0x55bd3c5a1e50 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f1aec009250 tx=0x7f1aec0bb300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f11b79f8c175c5e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:13.774 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.772+0000 7f1ad2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:00:13.780 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.777+0000 7f1ad2ffd640 1 -- 172.21.15.169:0/3145191911 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aec0041d0 con 0x55bd3c597b20 2024-03-28T01:00:13.780 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.777+0000 7f1afc58f740 1 -- 172.21.15.169:0/3145191911 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1aec09f160 con 0x55bd3c597b20 2024-03-28T01:00:13.780 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.777+0000 7f1ad2ffd640 1 -- 172.21.15.169:0/3145191911 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f1aec09b4c0 con 0x55bd3c597b20 2024-03-28T01:00:13.785 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.783+0000 7f1ad2ffd640 1 -- 172.21.15.169:0/3145191911 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1aec0ad050 con 0x55bd3c597b20 2024-03-28T01:00:13.789 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.786+0000 7f1afc58f740 10 client.16248 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:13.789 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.786+0000 7f1afc58f740 1 -- 172.21.15.169:0/3145191911 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55bd3c5a0630 con 0x55bd3c597b20 2024-03-28T01:00:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.790+0000 7f1ad2ffd640 1 -- 172.21.15.169:0/3145191911 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1aec09f160 con 0x55bd3c597b20 2024-03-28T01:00:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.790+0000 7f1ad2ffd640 1 -- 172.21.15.169:0/3145191911 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f1aec00b040 con 0x55bd3c597b20 2024-03-28T01:00:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.790+0000 7f1afc58f740 10 client.16248 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.790+0000 7f1afc58f740 10 client.16248 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.790+0000 7f1afc58f740 20 client.16248 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.790+0000 7f1afc58f740 1 --2- 172.21.15.169:0/3145191911 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bd3c5add40 0x55bd3c5b0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.790+0000 7f1afc58f740 4 client.16248 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:00:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.790+0000 7f1afc58f740 1 -- 172.21.15.169:0/3145191911 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bd3c50a050 con 0x55bd3c5add40 2024-03-28T01:00:13.796 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.793+0000 7f1af27fc640 1 --2- 172.21.15.169:0/3145191911 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bd3c5add40 0x55bd3c5b0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:13.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:13 smithi169 ceph-mon[32274]: pgmap v462: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 879 MiB/s wr, 1.42k op/s 2024-03-28T01:00:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.798+0000 7f1af27fc640 1 --2- 172.21.15.169:0/3145191911 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bd3c5add40 0x55bd3c5b0160 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=261fd4494ed56281 server_cookie=7769973d8cebf31 in_seq=0 out_seq=0 2024-03-28T01:00:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.799+0000 7f1ad2ffd640 10 client.16248 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.807+0000 7f1ad2ffd640 1 -- 172.21.15.169:0/3145191911 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bd3c50a050 con 0x55bd3c5add40 2024-03-28T01:00:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.807+0000 7f1ad2ffd640 10 client.16248 handle_command_reply: tid=0 2024-03-28T01:00:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.807+0000 7f1afc58f740 1 client.16248 shutdown 2024-03-28T01:00:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.811+0000 7f1afc58f740 1 -- 172.21.15.169:0/3145191911 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c597b20 msgr2=0x55bd3c5a1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.811+0000 7f1afc58f740 1 --2- 172.21.15.169:0/3145191911 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c597b20 0x55bd3c5a1e50 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f1aec009250 tx=0x7f1aec0bb300 comp rx=0 tx=0).stop 2024-03-28T01:00:13.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.817+0000 7f1afc58f740 1 -- 172.21.15.169:0/3145191911 shutdown_connections 2024-03-28T01:00:13.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.817+0000 7f1afc58f740 1 --2- 172.21.15.169:0/3145191911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c59d680 0x55bd3c5a93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.817+0000 7f1afc58f740 1 --2- 172.21.15.169:0/3145191911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c598510 0x55bd3c5a2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.817+0000 7f1afc58f740 1 --2- 172.21.15.169:0/3145191911 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bd3c5add40 0x55bd3c5b0160 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.817+0000 7f1afc58f740 1 --2- 172.21.15.169:0/3145191911 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c597b20 0x55bd3c5a1e50 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.817+0000 7f1afc58f740 1 -- 172.21.15.169:0/3145191911 >> 172.21.15.169:0/3145191911 conn(0x55bd3c593090 msgr2=0x55bd3c592290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:13.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.817+0000 7f1ad2ffd640 0 client.16248 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:13.824 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.821+0000 7f1afc58f740 1 -- 172.21.15.169:0/3145191911 shutdown_connections 2024-03-28T01:00:13.824 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.821+0000 7f1afc58f740 1 -- 172.21.15.169:0/3145191911 wait complete. 2024-03-28T01:00:13.829 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.826+0000 7f1afc58f740 20 client.16248 trim_cache size 0 max 16384 2024-03-28T01:00:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.887+0000 7f1afc58f740 1 Processor -- start 2024-03-28T01:00:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.887+0000 7f1afc58f740 1 -- start start 2024-03-28T01:00:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.888+0000 7f1afc58f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c597b20 0x55bd3c5997d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.888+0000 7f1afc58f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c598510 0x55bd3c5a9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:13.896 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.888+0000 7f1afc58f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c59d680 0x55bd3c599d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:13.896 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.889+0000 7f1af27fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c598510 0x55bd3c5a9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:13.896 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.889+0000 7f1af2ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c597b20 0x55bd3c5997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:13.896 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.890+0000 7f1af27fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c598510 0x55bd3c5a9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:37820/0 (socket says 172.21.15.169:37820) 2024-03-28T01:00:13.896 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.890+0000 7f1af2ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c597b20 0x55bd3c5997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39820/0 (socket says 172.21.15.169:39820) 2024-03-28T01:00:13.896 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.890+0000 7f1af27fc640 1 -- 172.21.15.169:0/497195493 learned_addr learned my addr 172.21.15.169:0/497195493 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:13.896 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.890+0000 7f1afc58f740 1 -- 172.21.15.169:0/497195493 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae00120e0 con 0x55bd3c597b20 2024-03-28T01:00:13.896 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.890+0000 7f1afc58f740 1 -- 172.21.15.169:0/497195493 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae000fbe0 con 0x55bd3c59d680 2024-03-28T01:00:13.897 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.890+0000 7f1afc58f740 1 -- 172.21.15.169:0/497195493 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae000dc80 con 0x55bd3c598510 2024-03-28T01:00:13.897 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.894+0000 7f1af37fe640 1 --2- 172.21.15.169:0/497195493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c59d680 0x55bd3c599d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:13.897 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.895+0000 7f1af27fc640 1 -- 172.21.15.169:0/497195493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c59d680 msgr2=0x55bd3c599d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:13.898 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.895+0000 7f1af27fc640 1 --2- 172.21.15.169:0/497195493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c59d680 0x55bd3c599d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.898 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.895+0000 7f1af27fc640 1 -- 172.21.15.169:0/497195493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c597b20 msgr2=0x55bd3c5997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:13.898 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.895+0000 7f1af27fc640 1 --2- 172.21.15.169:0/497195493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c597b20 0x55bd3c5997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.898 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.895+0000 7f1af27fc640 1 -- 172.21.15.169:0/497195493 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bd3c592a10 con 0x55bd3c598510 2024-03-28T01:00:13.901 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.897+0000 7f1af27fc640 1 --2- 172.21.15.169:0/497195493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c598510 0x55bd3c5a9390 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1ae80095f0 tx=0x7f1ae8035be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fee73ad801ffd87 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:13.902 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.899+0000 7f1ad97fa640 1 -- 172.21.15.169:0/497195493 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8049070 con 0x55bd3c598510 2024-03-28T01:00:13.902 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.900+0000 7f1ad97fa640 1 -- 172.21.15.169:0/497195493 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ae8043030 con 0x55bd3c598510 2024-03-28T01:00:13.902 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.900+0000 7f1ad97fa640 1 -- 172.21.15.169:0/497195493 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8039040 con 0x55bd3c598510 2024-03-28T01:00:13.903 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.901+0000 7f1afc58f740 1 -- 172.21.15.169:0/497195493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c598510 msgr2=0x55bd3c5a9390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:13.904 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.901+0000 7f1afc58f740 1 --2- 172.21.15.169:0/497195493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c598510 0x55bd3c5a9390 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1ae80095f0 tx=0x7f1ae8035be0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.905 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.902+0000 7f1afc58f740 1 -- 172.21.15.169:0/497195493 shutdown_connections 2024-03-28T01:00:13.905 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.902+0000 7f1afc58f740 1 --2- 172.21.15.169:0/497195493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c59d680 0x55bd3c599d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.905 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.902+0000 7f1afc58f740 1 --2- 172.21.15.169:0/497195493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c598510 0x55bd3c5a9390 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.905 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.902+0000 7f1afc58f740 1 --2- 172.21.15.169:0/497195493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c597b20 0x55bd3c5997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.905 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.902+0000 7f1afc58f740 1 -- 172.21.15.169:0/497195493 >> 172.21.15.169:0/497195493 conn(0x55bd3c593090 msgr2=0x55bd3c593470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:13.910 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.903+0000 7f1afc58f740 1 -- 172.21.15.169:0/497195493 shutdown_connections 2024-03-28T01:00:13.910 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.903+0000 7f1afc58f740 1 -- 172.21.15.169:0/497195493 wait complete. 2024-03-28T01:00:13.910 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.904+0000 7f1afc58f740 1 -- start start 2024-03-28T01:00:13.910 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.904+0000 7f1afc58f740 1 Processor -- start 2024-03-28T01:00:13.910 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.905+0000 7f1afc58f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c597b20 0x55bd3c5a6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:13.911 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.905+0000 7f1afc58f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c598510 0x55bd3c5a69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:13.911 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.905+0000 7f1afc58f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c59d680 0x55bd3c5a79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:13.911 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.905+0000 7f1afc58f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae000e280 con 0x55bd3c59d680 2024-03-28T01:00:13.911 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.905+0000 7f1afc58f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae000f580 con 0x55bd3c598510 2024-03-28T01:00:13.911 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.905+0000 7f1afc58f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae000e8c0 con 0x55bd3c597b20 2024-03-28T01:00:13.911 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.906+0000 7f1afc58f740 10 client.0 _ll_register_callbacks cb 0x7f1aec0cc330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:13.911 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.906+0000 7f1af2ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c597b20 0x55bd3c5a6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:13.911 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.906+0000 7f1af2ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c597b20 0x55bd3c5a6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:37836/0 (socket says 172.21.15.169:37836) 2024-03-28T01:00:13.911 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.906+0000 7f1af2ffd640 1 -- 172.21.15.169:0/1989811250 learned_addr learned my addr 172.21.15.169:0/1989811250 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:13.911 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.907+0000 7f1af37fe640 1 --2- 172.21.15.169:0/1989811250 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c59d680 0x55bd3c5a79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:13.911 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.907+0000 7f1af27fc640 1 --2- 172.21.15.169:0/1989811250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c598510 0x55bd3c5a69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:13.912 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.909+0000 7f1af2ffd640 1 -- 172.21.15.169:0/1989811250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c598510 msgr2=0x55bd3c5a69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:13.912 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.909+0000 7f1af2ffd640 1 --2- 172.21.15.169:0/1989811250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c598510 0x55bd3c5a69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.912 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.909+0000 7f1af2ffd640 1 -- 172.21.15.169:0/1989811250 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c59d680 msgr2=0x55bd3c5a79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:13.913 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.909+0000 7f1af2ffd640 1 --2- 172.21.15.169:0/1989811250 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c59d680 0x55bd3c5a79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:13.913 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.909+0000 7f1af2ffd640 1 -- 172.21.15.169:0/1989811250 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bd3c5a7fc0 con 0x55bd3c597b20 2024-03-28T01:00:13.913 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.911+0000 7f1af2ffd640 1 --2- 172.21.15.169:0/1989811250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c597b20 0x55bd3c5a6480 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1aec097be0 tx=0x7f1aec004da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9db4537f5cdcceaa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:13.914 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.911+0000 7f1ad2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:00:13.914 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.911+0000 7f1ad2ffd640 1 -- 172.21.15.169:0/1989811250 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aec097500 con 0x55bd3c597b20 2024-03-28T01:00:13.914 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.911+0000 7f1afc58f740 1 -- 172.21.15.169:0/1989811250 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1ae8043030 con 0x55bd3c597b20 2024-03-28T01:00:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.918+0000 7f1ad2ffd640 1 -- 172.21.15.169:0/1989811250 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f1aec0cc3e0 con 0x55bd3c597b20 2024-03-28T01:00:13.921 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.919+0000 7f1ad2ffd640 1 -- 172.21.15.169:0/1989811250 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1aec0bb0b0 con 0x55bd3c597b20 2024-03-28T01:00:13.925 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.920+0000 7f1afc58f740 10 client.26053 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:13.925 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.920+0000 7f1afc58f740 1 -- 172.21.15.169:0/1989811250 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55bd3c5a92d0 con 0x55bd3c597b20 2024-03-28T01:00:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.925+0000 7f1ad2ffd640 1 -- 172.21.15.169:0/1989811250 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ae8043030 con 0x55bd3c597b20 2024-03-28T01:00:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.926+0000 7f1ad2ffd640 1 -- 172.21.15.169:0/1989811250 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f1aec0aa020 con 0x55bd3c597b20 2024-03-28T01:00:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.927+0000 7f1afc58f740 10 client.26053 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.927+0000 7f1afc58f740 10 client.26053 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.927+0000 7f1afc58f740 20 client.26053 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.927+0000 7f1afc58f740 1 --2- 172.21.15.169:0/1989811250 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bd3c6097d0 0x55bd3c60bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.927+0000 7f1afc58f740 4 client.26053 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:00:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.927+0000 7f1afc58f740 1 -- 172.21.15.169:0/1989811250 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55bd3c5043a0 con 0x55bd3c6097d0 2024-03-28T01:00:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.929+0000 7f1af27fc640 1 --2- 172.21.15.169:0/1989811250 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bd3c6097d0 0x55bd3c60bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:13.939 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.936+0000 7f1ad2ffd640 10 client.26053 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:13.940 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:13.936+0000 7f1af27fc640 1 --2- 172.21.15.169:0/1989811250 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bd3c6097d0 0x55bd3c60bbc0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ae38f14b7ff784e server_cookie=fa31ac06bd6721cc in_seq=0 out_seq=0 2024-03-28T01:00:15.607 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:15 smithi169 ceph-mon[32274]: pgmap v463: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 768 MiB/s wr, 1.22k op/s 2024-03-28T01:00:15.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:15 smithi174 ceph-mon[36060]: pgmap v463: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 768 MiB/s wr, 1.22k op/s 2024-03-28T01:00:15.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:15 smithi177 ceph-mon[36206]: pgmap v463: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 768 MiB/s wr, 1.22k op/s 2024-03-28T01:00:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.481+0000 7f1ad2ffd640 1 -- 172.21.15.169:0/1989811250 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55bd3c5043a0 con 0x55bd3c6097d0 2024-03-28T01:00:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.481+0000 7f1ad2ffd640 10 client.26053 handle_command_reply: tid=0 2024-03-28T01:00:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.481+0000 7f1afc58f740 1 client.26053 shutdown 2024-03-28T01:00:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.482+0000 7f1afc58f740 1 -- 172.21.15.169:0/1989811250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c597b20 msgr2=0x55bd3c5a6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.482+0000 7f1afc58f740 1 --2- 172.21.15.169:0/1989811250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c597b20 0x55bd3c5a6480 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1aec097be0 tx=0x7f1aec004da0 comp rx=0 tx=0).stop 2024-03-28T01:00:17.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.483+0000 7f1afc58f740 1 -- 172.21.15.169:0/1989811250 shutdown_connections 2024-03-28T01:00:17.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.483+0000 7f1afc58f740 1 --2- 172.21.15.169:0/1989811250 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bd3c6097d0 0x55bd3c60bbc0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:17.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.483+0000 7f1afc58f740 1 --2- 172.21.15.169:0/1989811250 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bd3c59d680 0x55bd3c5a79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:17.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.483+0000 7f1afc58f740 1 --2- 172.21.15.169:0/1989811250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd3c598510 0x55bd3c5a69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:17.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.483+0000 7f1afc58f740 1 --2- 172.21.15.169:0/1989811250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bd3c597b20 0x55bd3c5a6480 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:17.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.483+0000 7f1afc58f740 1 -- 172.21.15.169:0/1989811250 >> 172.21.15.169:0/1989811250 conn(0x55bd3c593090 msgr2=0x55bd3c5997d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:17.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.483+0000 7f1ad2ffd640 0 client.26053 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:17.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.484+0000 7f1afc58f740 1 -- 172.21.15.169:0/1989811250 shutdown_connections 2024-03-28T01:00:17.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.484+0000 7f1afc58f740 1 -- 172.21.15.169:0/1989811250 wait complete. 2024-03-28T01:00:17.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.484+0000 7f1afc58f740 20 client.26053 trim_cache size 0 max 16384 2024-03-28T01:00:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:00:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:00:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:00:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:00:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.490+0000 7f1af3fff640 1 -- 172.21.15.169:0/1430859052 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1adc0a2a00 msgr2=0x7f1adc0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.490+0000 7f1af3fff640 1 --2- 172.21.15.169:0/1430859052 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1adc0a2a00 0x7f1adc0a4ec0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4006fd0 tx=0x7f1ae4008040 comp rx=0 tx=0).stop 2024-03-28T01:00:17.494 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.490+0000 7f1af3fff640 1 -- 172.21.15.169:0/1430859052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af4107b40 msgr2=0x7f1af407ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.490+0000 7f1af3fff640 1 --2- 172.21.15.169:0/1430859052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af4107b40 0x7f1af407ee80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1aec005fd0 tx=0x7f1aec00c800 comp rx=0 tx=0).stop 2024-03-28T01:00:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.491+0000 7f1af3fff640 1 -- 172.21.15.169:0/1430859052 shutdown_connections 2024-03-28T01:00:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.491+0000 7f1af3fff640 1 --2- 172.21.15.169:0/1430859052 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1adc0a2a00 0x7f1adc0a4ec0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.491+0000 7f1af3fff640 1 --2- 172.21.15.169:0/1430859052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1af41096f0 0x7f1af41adfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.491+0000 7f1af3fff640 1 --2- 172.21.15.169:0/1430859052 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1af4108d30 0x7f1af407f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.491+0000 7f1af3fff640 1 --2- 172.21.15.169:0/1430859052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af4107b40 0x7f1af407ee80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.491+0000 7f1af3fff640 1 -- 172.21.15.169:0/1430859052 >> 172.21.15.169:0/1430859052 conn(0x7f1af4076ad0 msgr2=0x7f1af410c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.491+0000 7f1af3fff640 1 -- 172.21.15.169:0/1430859052 shutdown_connections 2024-03-28T01:00:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:17.491+0000 7f1af3fff640 1 -- 172.21.15.169:0/1430859052 wait complete. 2024-03-28T01:00:17.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:17 smithi174 ceph-mon[36060]: pgmap v464: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 812 MiB/s wr, 1.29k op/s 2024-03-28T01:00:17.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:17 smithi177 ceph-mon[36206]: pgmap v464: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 812 MiB/s wr, 1.29k op/s 2024-03-28T01:00:17.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:17 smithi169 ceph-mon[32274]: pgmap v464: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 812 MiB/s wr, 1.29k op/s 2024-03-28T01:00:19.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:19 smithi174 ceph-mon[36060]: pgmap v465: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 784 MiB/s wr, 1.23k op/s 2024-03-28T01:00:19.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:19 smithi177 ceph-mon[36206]: pgmap v465: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 784 MiB/s wr, 1.23k op/s 2024-03-28T01:00:19.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:19 smithi169 ceph-mon[32274]: pgmap v465: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 784 MiB/s wr, 1.23k op/s 2024-03-28T01:00:21.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:21 smithi174 ceph-mon[36060]: pgmap v466: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 774 MiB/s wr, 1.22k op/s 2024-03-28T01:00:21.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:21 smithi177 ceph-mon[36206]: pgmap v466: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 774 MiB/s wr, 1.22k op/s 2024-03-28T01:00:21.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:21 smithi169 ceph-mon[32274]: pgmap v466: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 774 MiB/s wr, 1.22k op/s 2024-03-28T01:00:23.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:23 smithi177 ceph-mon[36206]: pgmap v467: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 883 MiB/s wr, 1.60k op/s 2024-03-28T01:00:23.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:23 smithi169 ceph-mon[32274]: pgmap v467: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 883 MiB/s wr, 1.60k op/s 2024-03-28T01:00:23.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:23 smithi174 ceph-mon[36060]: pgmap v467: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 883 MiB/s wr, 1.60k op/s 2024-03-28T01:00:25.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:25 smithi177 ceph-mon[36206]: pgmap v468: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 766 MiB/s wr, 1.52k op/s 2024-03-28T01:00:25.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:25 smithi169 ceph-mon[32274]: pgmap v468: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 766 MiB/s wr, 1.52k op/s 2024-03-28T01:00:25.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:25 smithi174 ceph-mon[36060]: pgmap v468: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 766 MiB/s wr, 1.52k op/s 2024-03-28T01:00:27.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:27 smithi177 ceph-mon[36206]: pgmap v469: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 811 MiB/s wr, 1.50k op/s 2024-03-28T01:00:27.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:27 smithi169 ceph-mon[32274]: pgmap v469: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 811 MiB/s wr, 1.50k op/s 2024-03-28T01:00:27.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:27 smithi174 ceph-mon[36060]: pgmap v469: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 811 MiB/s wr, 1.50k op/s 2024-03-28T01:00:29.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:29 smithi177 ceph-mon[36206]: pgmap v470: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 748 MiB/s wr, 1.36k op/s 2024-03-28T01:00:29.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:29 smithi169 ceph-mon[32274]: pgmap v470: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 748 MiB/s wr, 1.36k op/s 2024-03-28T01:00:29.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:29 smithi174 ceph-mon[36060]: pgmap v470: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 748 MiB/s wr, 1.36k op/s 2024-03-28T01:00:30.746 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.743+0000 7f46bb59e640 1 -- 172.21.15.169:0/3217987182 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46bc0732b0 msgr2=0x7f46bc108cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:30.747 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.743+0000 7f46bb59e640 1 --2- 172.21.15.169:0/3217987182 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46bc0732b0 0x7f46bc108cd0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f46b4009ca0 tx=0x7f46b402c140 comp rx=0 tx=0).stop 2024-03-28T01:00:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.744+0000 7f46bb59e640 1 -- 172.21.15.169:0/3217987182 shutdown_connections 2024-03-28T01:00:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.744+0000 7f46bb59e640 1 --2- 172.21.15.169:0/3217987182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46bc109390 0x7f46bc10e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.744+0000 7f46bb59e640 1 --2- 172.21.15.169:0/3217987182 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46bc0732b0 0x7f46bc108cd0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.744+0000 7f46bb59e640 1 --2- 172.21.15.169:0/3217987182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46bc072950 0x7f46bc072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.744+0000 7f46bb59e640 1 -- 172.21.15.169:0/3217987182 >> 172.21.15.169:0/3217987182 conn(0x7f46bc06e510 msgr2=0x7f46bc070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:30.754 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.751+0000 7f46bb59e640 1 -- 172.21.15.169:0/3217987182 shutdown_connections 2024-03-28T01:00:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.755+0000 7f46bb59e640 1 -- 172.21.15.169:0/3217987182 wait complete. 2024-03-28T01:00:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.756+0000 7f46bb59e640 1 Processor -- start 2024-03-28T01:00:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.756+0000 7f46bb59e640 1 -- start start 2024-03-28T01:00:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.756+0000 7f46bb59e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46bc072950 0x7f46bc1a6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.756+0000 7f46bb59e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46bc0732b0 0x7f46bc1a6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.756+0000 7f46bb59e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46bc109390 0x7f46bc1adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.756+0000 7f46bb59e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc11c0d0 con 0x7f46bc072950 2024-03-28T01:00:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.756+0000 7f46bb59e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc11bf50 con 0x7f46bc0732b0 2024-03-28T01:00:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.756+0000 7f46bb59e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc11c250 con 0x7f46bc109390 2024-03-28T01:00:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.757+0000 7f46bad9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46bc109390 0x7f46bc1adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.758+0000 7f46b9d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46bc0732b0 0x7f46bc1a6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.758+0000 7f46b9d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46bc0732b0 0x7f46bc1a6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:40134/0 (socket says 172.21.15.169:40134) 2024-03-28T01:00:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.758+0000 7f46b9d9b640 1 -- 172.21.15.169:0/349907742 learned_addr learned my addr 172.21.15.169:0/349907742 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.761+0000 7f46b9d9b640 1 -- 172.21.15.169:0/349907742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46bc109390 msgr2=0x7f46bc1adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.761+0000 7f46b9d9b640 1 --2- 172.21.15.169:0/349907742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46bc109390 0x7f46bc1adde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.761+0000 7f46b9d9b640 1 -- 172.21.15.169:0/349907742 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46bc072950 msgr2=0x7f46bc1a6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.761+0000 7f46b9d9b640 1 --2- 172.21.15.169:0/349907742 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46bc072950 0x7f46bc1a6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.761+0000 7f46b9d9b640 1 -- 172.21.15.169:0/349907742 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46bc10ff00 con 0x7f46bc0732b0 2024-03-28T01:00:30.769 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.766+0000 7f46b9d9b640 1 --2- 172.21.15.169:0/349907742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46bc0732b0 0x7f46bc1a6d70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f46b4037040 tx=0x7f46b4035e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.767+0000 7f46ab7fe640 1 -- 172.21.15.169:0/349907742 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b4039040 con 0x7f46bc0732b0 2024-03-28T01:00:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.767+0000 7f46bb59e640 1 -- 172.21.15.169:0/349907742 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46bc110190 con 0x7f46bc0732b0 2024-03-28T01:00:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.767+0000 7f46bb59e640 1 -- 172.21.15.169:0/349907742 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46bc110750 con 0x7f46bc0732b0 2024-03-28T01:00:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.768+0000 7f46ab7fe640 1 -- 172.21.15.169:0/349907742 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46b4003e50 con 0x7f46bc0732b0 2024-03-28T01:00:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.768+0000 7f46ab7fe640 1 -- 172.21.15.169:0/349907742 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b403f440 con 0x7f46bc0732b0 2024-03-28T01:00:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.773+0000 7f46c3462740 1 Processor -- start 2024-03-28T01:00:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.773+0000 7f46c3462740 1 -- start start 2024-03-28T01:00:30.777 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.774+0000 7f46c3462740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab36bb20 0x5621ab36bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:30.777 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.774+0000 7f46c3462740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621ab36c510 0x5621ab371010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:30.777 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.774+0000 7f46c3462740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab371680 0x5621ab373b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:30.777 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.774+0000 7f46c3462740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621ab2d0b10 con 0x5621ab371680 2024-03-28T01:00:30.778 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.775+0000 7f46c3462740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621ab279900 con 0x5621ab36bb20 2024-03-28T01:00:30.778 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.775+0000 7f46c3462740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621ab2ccbb0 con 0x5621ab36c510 2024-03-28T01:00:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.779+0000 7f46ba59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621ab36c510 0x5621ab371010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.779+0000 7f46bad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab36bb20 0x5621ab36bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.779+0000 7f46bad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab36bb20 0x5621ab36bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:40142/0 (socket says 172.21.15.169:40142) 2024-03-28T01:00:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.779+0000 7f46bad9d640 1 -- 172.21.15.169:0/3579556492 learned_addr learned my addr 172.21.15.169:0/3579556492 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:30.787 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.783+0000 7f46bad9d640 1 -- 172.21.15.169:0/3579556492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621ab36c510 msgr2=0x5621ab371010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:30.787 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.783+0000 7f46bad9d640 1 --2- 172.21.15.169:0/3579556492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621ab36c510 0x5621ab371010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.787 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.783+0000 7f46bad9d640 1 -- 172.21.15.169:0/3579556492 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab371680 msgr2=0x5621ab373b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:00:30.787 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.783+0000 7f46bad9d640 1 --2- 172.21.15.169:0/3579556492 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab371680 0x5621ab373b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.787 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.783+0000 7f46bad9d640 1 -- 172.21.15.169:0/3579556492 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5621aafe5210 con 0x5621ab36bb20 2024-03-28T01:00:30.787 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.784+0000 7f46bad9d640 1 --2- 172.21.15.169:0/3579556492 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab36bb20 0x5621ab36bf40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f46ac00b7a0 tx=0x7f46ac00bc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e996cb29af9c841 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:30.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.787+0000 7f46ab7fe640 1 -- 172.21.15.169:0/349907742 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f46b403f5e0 con 0x7f46bc0732b0 2024-03-28T01:00:30.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.787+0000 7f46a8ff9640 1 -- 172.21.15.169:0/3579556492 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac00d950 con 0x5621ab36bb20 2024-03-28T01:00:30.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.787+0000 7f46a8ff9640 1 -- 172.21.15.169:0/3579556492 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46ac012420 con 0x5621ab36bb20 2024-03-28T01:00:30.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.787+0000 7f46a8ff9640 1 -- 172.21.15.169:0/3579556492 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac014630 con 0x5621ab36bb20 2024-03-28T01:00:30.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.788+0000 7f46ab7fe640 1 --2- 172.21.15.169:0/349907742 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f46a40a2930 0x7f46a40a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:30.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.788+0000 7f46c3462740 1 -- 172.21.15.169:0/3579556492 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab36bb20 msgr2=0x5621ab36bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:30.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.788+0000 7f46c3462740 1 --2- 172.21.15.169:0/3579556492 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab36bb20 0x5621ab36bf40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f46ac00b7a0 tx=0x7f46ac00bc60 comp rx=0 tx=0).stop 2024-03-28T01:00:30.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.789+0000 7f46ab7fe640 1 -- 172.21.15.169:0/349907742 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f46b40f1e60 con 0x7f46bc0732b0 2024-03-28T01:00:30.795 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.793+0000 7f46ba59c640 1 --2- 172.21.15.169:0/349907742 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f46a40a2930 0x7f46a40a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.793+0000 7f46c3462740 1 -- 172.21.15.169:0/3579556492 shutdown_connections 2024-03-28T01:00:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.793+0000 7f46c3462740 1 --2- 172.21.15.169:0/3579556492 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab371680 0x5621ab373b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.793+0000 7f46c3462740 1 --2- 172.21.15.169:0/3579556492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621ab36c510 0x5621ab371010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.793+0000 7f46c3462740 1 --2- 172.21.15.169:0/3579556492 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab36bb20 0x5621ab36bf40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.793+0000 7f46c3462740 1 -- 172.21.15.169:0/3579556492 >> 172.21.15.169:0/3579556492 conn(0x5621ab367090 msgr2=0x5621ab367490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.797+0000 7f46c3462740 1 -- 172.21.15.169:0/3579556492 shutdown_connections 2024-03-28T01:00:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.799+0000 7f46c3462740 1 -- 172.21.15.169:0/3579556492 wait complete. 2024-03-28T01:00:30.803 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.799+0000 7f46c3462740 1 -- start start 2024-03-28T01:00:30.803 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.800+0000 7f46c3462740 1 Processor -- start 2024-03-28T01:00:30.809 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.806+0000 7f46ba59c640 1 --2- 172.21.15.169:0/349907742 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f46a40a2930 0x7f46a40a4df0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x5621ab368000 tx=0x7f46b0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.807+0000 7f46c3462740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab36bb20 0x5621ab375fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.807+0000 7f46c3462740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab36c510 0x5621ab376510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.807+0000 7f46c3462740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621ab371680 0x5621ab37d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.807+0000 7f46c3462740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621ab374630 con 0x5621ab36c510 2024-03-28T01:00:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.807+0000 7f46c3462740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621ab28cd50 con 0x5621ab36bb20 2024-03-28T01:00:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.807+0000 7f46c3462740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621ab3747b0 con 0x5621ab371680 2024-03-28T01:00:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.807+0000 7f46c3462740 10 client.0 _ll_register_callbacks cb 0x5621ab28c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.810+0000 7f46b9d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab36c510 0x5621ab376510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.810+0000 7f46b9d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab36c510 0x5621ab376510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:59088/0 (socket says 172.21.15.169:59088) 2024-03-28T01:00:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.810+0000 7f46b9d9b640 1 -- 172.21.15.169:0/780175927 learned_addr learned my addr 172.21.15.169:0/780175927 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.810+0000 7f46ba59c640 1 --2- 172.21.15.169:0/780175927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab36bb20 0x5621ab375fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:30.818 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.815+0000 7f46b9d9b640 1 -- 172.21.15.169:0/780175927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621ab371680 msgr2=0x5621ab37d4e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:00:30.818 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.815+0000 7f46b9d9b640 1 --2- 172.21.15.169:0/780175927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621ab371680 0x5621ab37d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.818 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.815+0000 7f46b9d9b640 1 -- 172.21.15.169:0/780175927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab36bb20 msgr2=0x5621ab375fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:30.818 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.815+0000 7f46b9d9b640 1 --2- 172.21.15.169:0/780175927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab36bb20 0x5621ab375fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.818 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.815+0000 7f46b9d9b640 1 -- 172.21.15.169:0/780175927 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5621ab0b1620 con 0x5621ab36c510 2024-03-28T01:00:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.816+0000 7f46b9d9b640 1 --2- 172.21.15.169:0/780175927 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab36c510 0x5621ab376510 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f46b40ba8f0 tx=0x7f46b40be320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5635d4b6779b3a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.816+0000 7f46a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:00:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.816+0000 7f46a1ffb640 1 -- 172.21.15.169:0/780175927 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b40beb90 con 0x5621ab36c510 2024-03-28T01:00:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.816+0000 7f46a1ffb640 1 -- 172.21.15.169:0/780175927 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f46b4004040 con 0x5621ab36c510 2024-03-28T01:00:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.819+0000 7f46c3462740 1 -- 172.21.15.169:0/780175927 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f46ac012420 con 0x5621ab36c510 2024-03-28T01:00:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.820+0000 7f46a1ffb640 1 -- 172.21.15.169:0/780175927 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46b4004240 con 0x5621ab36c510 2024-03-28T01:00:30.825 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.822+0000 7f46c3462740 10 client.16260 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:30.825 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.822+0000 7f46c3462740 1 -- 172.21.15.169:0/780175927 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5621ab380850 con 0x5621ab36c510 2024-03-28T01:00:30.825 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.823+0000 7f46a1ffb640 1 -- 172.21.15.169:0/780175927 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46ac012420 con 0x5621ab36c510 2024-03-28T01:00:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.823+0000 7f46a1ffb640 1 -- 172.21.15.169:0/780175927 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f46b403c030 con 0x5621ab36c510 2024-03-28T01:00:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.823+0000 7f46c3462740 10 client.16260 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.823+0000 7f46c3462740 10 client.16260 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.823+0000 7f46c3462740 20 client.16260 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.824+0000 7f46c3462740 1 --2- 172.21.15.169:0/780175927 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5621ab382270 0x5621ab384690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.824+0000 7f46c3462740 4 client.16260 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:00:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.824+0000 7f46c3462740 1 -- 172.21.15.169:0/780175927 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5621ab2de050 con 0x5621ab382270 2024-03-28T01:00:30.830 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.826+0000 7f46ba59c640 1 --2- 172.21.15.169:0/780175927 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5621ab382270 0x5621ab384690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.832+0000 7f46ba59c640 1 --2- 172.21.15.169:0/780175927 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5621ab382270 0x5621ab384690 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37faa162ef090c56 server_cookie=1daf028122df5e09 in_seq=0 out_seq=0 2024-03-28T01:00:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.832+0000 7f46a1ffb640 10 client.16260 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:30.847 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.844+0000 7f46a1ffb640 1 -- 172.21.15.169:0/780175927 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5621ab2de050 con 0x5621ab382270 2024-03-28T01:00:30.847 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.844+0000 7f46a1ffb640 10 client.16260 handle_command_reply: tid=0 2024-03-28T01:00:30.847 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.845+0000 7f46c3462740 1 client.16260 shutdown 2024-03-28T01:00:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.847+0000 7f46c3462740 1 -- 172.21.15.169:0/780175927 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab36c510 msgr2=0x5621ab376510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.847+0000 7f46c3462740 1 --2- 172.21.15.169:0/780175927 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab36c510 0x5621ab376510 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f46b40ba8f0 tx=0x7f46b40be320 comp rx=0 tx=0).stop 2024-03-28T01:00:30.857 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.854+0000 7f46c3462740 1 -- 172.21.15.169:0/780175927 shutdown_connections 2024-03-28T01:00:30.857 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.854+0000 7f46c3462740 1 --2- 172.21.15.169:0/780175927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621ab371680 0x5621ab37d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.857 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.854+0000 7f46c3462740 1 --2- 172.21.15.169:0/780175927 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5621ab382270 0x5621ab384690 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.857 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.854+0000 7f46c3462740 1 --2- 172.21.15.169:0/780175927 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab36c510 0x5621ab376510 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.857 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.854+0000 7f46c3462740 1 --2- 172.21.15.169:0/780175927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab36bb20 0x5621ab375fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.857 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.854+0000 7f46c3462740 1 -- 172.21.15.169:0/780175927 >> 172.21.15.169:0/780175927 conn(0x5621ab367090 msgr2=0x5621ab366290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:30.858 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.854+0000 7f46a1ffb640 0 client.16260 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:30.858 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.855+0000 7f46c3462740 1 -- 172.21.15.169:0/780175927 shutdown_connections 2024-03-28T01:00:30.858 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.855+0000 7f46c3462740 1 -- 172.21.15.169:0/780175927 wait complete. 2024-03-28T01:00:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.858+0000 7f46c3462740 20 client.16260 trim_cache size 0 max 16384 2024-03-28T01:00:30.909 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.906+0000 7f46c3462740 1 Processor -- start 2024-03-28T01:00:30.910 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.907+0000 7f46c3462740 1 -- start start 2024-03-28T01:00:30.910 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.907+0000 7f46c3462740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621ab36bb20 0x5621ab3759b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:30.910 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.907+0000 7f46c3462740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab36c510 0x5621ab37d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:30.910 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.907+0000 7f46c3462740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab371680 0x5621ab365a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:30.910 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.907+0000 7f46c3462740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c012090 con 0x5621ab371680 2024-03-28T01:00:30.911 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.907+0000 7f46c3462740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c00fb90 con 0x5621ab36c510 2024-03-28T01:00:30.911 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.907+0000 7f46c3462740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c00dc30 con 0x5621ab36bb20 2024-03-28T01:00:30.911 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.908+0000 7f46bad9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab371680 0x5621ab365a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:30.911 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.908+0000 7f46bad9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab371680 0x5621ab365a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:59102/0 (socket says 172.21.15.169:59102) 2024-03-28T01:00:30.911 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.908+0000 7f46bad9d640 1 -- 172.21.15.169:0/859866342 learned_addr learned my addr 172.21.15.169:0/859866342 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:30.911 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.908+0000 7f46b9d9b640 1 --2- 172.21.15.169:0/859866342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab36c510 0x5621ab37d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:30.915 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.912+0000 7f46ba59c640 1 --2- 172.21.15.169:0/859866342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621ab36bb20 0x5621ab3759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:30.915 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.912+0000 7f46bad9d640 1 -- 172.21.15.169:0/859866342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621ab36bb20 msgr2=0x5621ab3759b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:30.915 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.912+0000 7f46bad9d640 1 --2- 172.21.15.169:0/859866342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621ab36bb20 0x5621ab3759b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.915 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.912+0000 7f46bad9d640 1 -- 172.21.15.169:0/859866342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab36c510 msgr2=0x5621ab37d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:30.915 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.912+0000 7f46bad9d640 1 --2- 172.21.15.169:0/859866342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab36c510 0x5621ab37d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.915 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.912+0000 7f46bad9d640 1 -- 172.21.15.169:0/859866342 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5621ab367e00 con 0x5621ab371680 2024-03-28T01:00:30.915 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.912+0000 7f46ba59c640 1 --2- 172.21.15.169:0/859866342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621ab36bb20 0x5621ab3759b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:00:30.915 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.912+0000 7f46b9d9b640 1 --2- 172.21.15.169:0/859866342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab36c510 0x5621ab37d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:00:30.916 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.912+0000 7f46bad9d640 1 --2- 172.21.15.169:0/859866342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab371680 0x5621ab365a00 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f46ac00dbf0 tx=0x7f46ac00dc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=208f46ffbabfe90d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:30.916 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.913+0000 7f46a3fff640 1 -- 172.21.15.169:0/859866342 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac021020 con 0x5621ab371680 2024-03-28T01:00:30.916 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.913+0000 7f46a3fff640 1 -- 172.21.15.169:0/859866342 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46ac004510 con 0x5621ab371680 2024-03-28T01:00:30.916 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.913+0000 7f46a3fff640 1 -- 172.21.15.169:0/859866342 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac003d10 con 0x5621ab371680 2024-03-28T01:00:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.917+0000 7f46c3462740 1 -- 172.21.15.169:0/859866342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab371680 msgr2=0x5621ab365a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.917+0000 7f46c3462740 1 --2- 172.21.15.169:0/859866342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab371680 0x5621ab365a00 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f46ac00dbf0 tx=0x7f46ac00dc20 comp rx=0 tx=0).stop 2024-03-28T01:00:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.921+0000 7f46c3462740 1 -- 172.21.15.169:0/859866342 shutdown_connections 2024-03-28T01:00:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.921+0000 7f46c3462740 1 --2- 172.21.15.169:0/859866342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab371680 0x5621ab365a00 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.921+0000 7f46c3462740 1 --2- 172.21.15.169:0/859866342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab36c510 0x5621ab37d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.921+0000 7f46c3462740 1 --2- 172.21.15.169:0/859866342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621ab36bb20 0x5621ab3759b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.921+0000 7f46c3462740 1 -- 172.21.15.169:0/859866342 >> 172.21.15.169:0/859866342 conn(0x5621ab367090 msgr2=0x5621ab3752a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.921+0000 7f46c3462740 1 -- 172.21.15.169:0/859866342 shutdown_connections 2024-03-28T01:00:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.922+0000 7f46c3462740 1 -- 172.21.15.169:0/859866342 wait complete. 2024-03-28T01:00:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.922+0000 7f46c3462740 1 -- start start 2024-03-28T01:00:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.922+0000 7f46c3462740 1 Processor -- start 2024-03-28T01:00:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.923+0000 7f46c3462740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab36bb20 0x5621ab37a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.923+0000 7f46c3462740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621ab36c510 0x5621ab37aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.923+0000 7f46c3462740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab371680 0x5621ab37baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.923+0000 7f46c3462740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c00e230 con 0x5621ab36bb20 2024-03-28T01:00:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.923+0000 7f46c3462740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c00f530 con 0x5621ab371680 2024-03-28T01:00:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.923+0000 7f46c3462740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c00e870 con 0x5621ab36c510 2024-03-28T01:00:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.923+0000 7f46c3462740 10 client.0 _ll_register_callbacks cb 0x5621ab28c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:30.928 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.924+0000 7f46bad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab371680 0x5621ab37baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:30.928 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.924+0000 7f46bad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab371680 0x5621ab37baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:40180/0 (socket says 172.21.15.169:40180) 2024-03-28T01:00:30.928 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.924+0000 7f46bad9d640 1 -- 172.21.15.169:0/1365733705 learned_addr learned my addr 172.21.15.169:0/1365733705 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:30.928 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.925+0000 7f46b9d9b640 1 --2- 172.21.15.169:0/1365733705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621ab36c510 0x5621ab37aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:30.928 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.925+0000 7f46ba59c640 1 --2- 172.21.15.169:0/1365733705 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab36bb20 0x5621ab37a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.926+0000 7f46ba59c640 1 -- 172.21.15.169:0/1365733705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621ab36c510 msgr2=0x5621ab37aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.926+0000 7f46ba59c640 1 --2- 172.21.15.169:0/1365733705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621ab36c510 0x5621ab37aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.926+0000 7f46ba59c640 1 -- 172.21.15.169:0/1365733705 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab371680 msgr2=0x5621ab37baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.926+0000 7f46ba59c640 1 --2- 172.21.15.169:0/1365733705 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab371680 0x5621ab37baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.926+0000 7f46ba59c640 1 -- 172.21.15.169:0/1365733705 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5621ab37c0e0 con 0x5621ab36bb20 2024-03-28T01:00:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.926+0000 7f46ba59c640 1 --2- 172.21.15.169:0/1365733705 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab36bb20 0x5621ab37a5a0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f46b40bc740 tx=0x7f46b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edd710d44dcb6107 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.926+0000 7f46a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:00:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.927+0000 7f46a1ffb640 1 -- 172.21.15.169:0/1365733705 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b00252f0 con 0x5621ab36bb20 2024-03-28T01:00:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.927+0000 7f46a1ffb640 1 -- 172.21.15.169:0/1365733705 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f46b000f040 con 0x5621ab36bb20 2024-03-28T01:00:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.927+0000 7f46c3462740 1 -- 172.21.15.169:0/1365733705 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f46ac004510 con 0x5621ab36bb20 2024-03-28T01:00:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.927+0000 7f46a1ffb640 1 -- 172.21.15.169:0/1365733705 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46b0025e10 con 0x5621ab36bb20 2024-03-28T01:00:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.932+0000 7f46c3462740 10 client.16266 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.932+0000 7f46c3462740 1 -- 172.21.15.169:0/1365733705 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5621ab37d430 con 0x5621ab36bb20 2024-03-28T01:00:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.933+0000 7f46a1ffb640 1 -- 172.21.15.169:0/1365733705 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46ac004510 con 0x5621ab36bb20 2024-03-28T01:00:30.937 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.934+0000 7f46a1ffb640 1 -- 172.21.15.169:0/1365733705 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f46b0013070 con 0x5621ab36bb20 2024-03-28T01:00:30.939 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.935+0000 7f46c3462740 10 client.16266 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:30.939 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.935+0000 7f46c3462740 10 client.16266 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:30.939 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.935+0000 7f46c3462740 20 client.16266 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:30.939 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.935+0000 7f46c3462740 1 --2- 172.21.15.169:0/1365733705 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5621ab3ddf00 0x5621ab3e02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:30.939 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.935+0000 7f46c3462740 4 client.16266 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:00:30.939 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.935+0000 7f46c3462740 1 -- 172.21.15.169:0/1365733705 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5621ab2d83a0 con 0x5621ab3ddf00 2024-03-28T01:00:30.940 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.937+0000 7f46b9d9b640 1 --2- 172.21.15.169:0/1365733705 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5621ab3ddf00 0x5621ab3e02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.944+0000 7f46b9d9b640 1 --2- 172.21.15.169:0/1365733705 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5621ab3ddf00 0x5621ab3e02f0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd419d62088417c5 server_cookie=6064525bb3264c11 in_seq=0 out_seq=0 2024-03-28T01:00:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:30.946+0000 7f46a1ffb640 10 client.16266 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:31.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:31 smithi177 ceph-mon[36206]: pgmap v471: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 763 MiB/s wr, 1.40k op/s 2024-03-28T01:00:31.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:31 smithi169 ceph-mon[32274]: pgmap v471: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 763 MiB/s wr, 1.40k op/s 2024-03-28T01:00:31.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:31 smithi174 ceph-mon[36060]: pgmap v471: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 763 MiB/s wr, 1.40k op/s 2024-03-28T01:00:33.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:33 smithi177 ceph-mon[36206]: pgmap v472: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 870 MiB/s wr, 1.64k op/s 2024-03-28T01:00:33.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:33 smithi169 ceph-mon[32274]: pgmap v472: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 870 MiB/s wr, 1.64k op/s 2024-03-28T01:00:33.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:33 smithi174 ceph-mon[36060]: pgmap v472: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 870 MiB/s wr, 1.64k op/s 2024-03-28T01:00:35.623 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:00:35.623 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:00:35.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:35 smithi169 ceph-mon[32274]: pgmap v473: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 758 MiB/s wr, 1.26k op/s 2024-03-28T01:00:35.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:35 smithi174 ceph-mon[36060]: pgmap v473: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 758 MiB/s wr, 1.26k op/s 2024-03-28T01:00:35.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:35 smithi177 ceph-mon[36206]: pgmap v473: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 758 MiB/s wr, 1.26k op/s 2024-03-28T01:00:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.310+0000 7f46a1ffb640 1 -- 172.21.15.169:0/1365733705 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5621ab2d83a0 con 0x5621ab3ddf00 2024-03-28T01:00:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.310+0000 7f46a1ffb640 10 client.16266 handle_command_reply: tid=0 2024-03-28T01:00:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.310+0000 7f46c3462740 1 client.16266 shutdown 2024-03-28T01:00:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.310+0000 7f46c3462740 1 -- 172.21.15.169:0/1365733705 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab36bb20 msgr2=0x5621ab37a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:36.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.310+0000 7f46c3462740 1 --2- 172.21.15.169:0/1365733705 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab36bb20 0x5621ab37a5a0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f46b40bc740 tx=0x7f46b0004750 comp rx=0 tx=0).stop 2024-03-28T01:00:36.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.311+0000 7f46c3462740 1 -- 172.21.15.169:0/1365733705 shutdown_connections 2024-03-28T01:00:36.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.311+0000 7f46c3462740 1 --2- 172.21.15.169:0/1365733705 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621ab371680 0x5621ab37baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:36.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.311+0000 7f46c3462740 1 --2- 172.21.15.169:0/1365733705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621ab36c510 0x5621ab37aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:36.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.311+0000 7f46c3462740 1 --2- 172.21.15.169:0/1365733705 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5621ab3ddf00 0x5621ab3e02f0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:36.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.311+0000 7f46c3462740 1 --2- 172.21.15.169:0/1365733705 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5621ab36bb20 0x5621ab37a5a0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:36.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.311+0000 7f46c3462740 1 -- 172.21.15.169:0/1365733705 >> 172.21.15.169:0/1365733705 conn(0x5621ab367090 msgr2=0x5621ab36a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:36.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.311+0000 7f46a1ffb640 0 client.16266 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:36.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.311+0000 7f46c3462740 1 -- 172.21.15.169:0/1365733705 shutdown_connections 2024-03-28T01:00:36.315 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.311+0000 7f46c3462740 1 -- 172.21.15.169:0/1365733705 wait complete. 2024-03-28T01:00:36.315 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.311+0000 7f46c3462740 20 client.16266 trim_cache size 0 max 16384 2024-03-28T01:00:36.315 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:00:36.315 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:00:36.315 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:00:36.315 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:00:36.317 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.314+0000 7f46bb59e640 1 -- 172.21.15.169:0/349907742 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f46a40a2930 msgr2=0x7f46a40a4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:36.317 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.314+0000 7f46bb59e640 1 --2- 172.21.15.169:0/349907742 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f46a40a2930 0x7f46a40a4df0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x5621ab368000 tx=0x7f46b0008040 comp rx=0 tx=0).stop 2024-03-28T01:00:36.317 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.314+0000 7f46bb59e640 1 -- 172.21.15.169:0/349907742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46bc0732b0 msgr2=0x7f46bc1a6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:36.317 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.314+0000 7f46bb59e640 1 --2- 172.21.15.169:0/349907742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46bc0732b0 0x7f46bc1a6d70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f46b4037040 tx=0x7f46b4035e50 comp rx=0 tx=0).stop 2024-03-28T01:00:36.317 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.314+0000 7f46bb59e640 1 -- 172.21.15.169:0/349907742 shutdown_connections 2024-03-28T01:00:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.314+0000 7f46bb59e640 1 --2- 172.21.15.169:0/349907742 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f46a40a2930 0x7f46a40a4df0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.314+0000 7f46bb59e640 1 --2- 172.21.15.169:0/349907742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46bc109390 0x7f46bc1adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.314+0000 7f46bb59e640 1 --2- 172.21.15.169:0/349907742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46bc0732b0 0x7f46bc1a6d70 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.314+0000 7f46bb59e640 1 --2- 172.21.15.169:0/349907742 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f46bc072950 0x7f46bc1a6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.314+0000 7f46bb59e640 1 -- 172.21.15.169:0/349907742 >> 172.21.15.169:0/349907742 conn(0x7f46bc06e510 msgr2=0x7f46bc06ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.314+0000 7f46bb59e640 1 -- 172.21.15.169:0/349907742 shutdown_connections 2024-03-28T01:00:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:36.314+0000 7f46bb59e640 1 -- 172.21.15.169:0/349907742 wait complete. 2024-03-28T01:00:37.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.647+0000 7fa4738a1640 1 -- 172.21.15.169:0/51028106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa46c072a10 msgr2=0x7fa46c072e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:37.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.647+0000 7fa4738a1640 1 --2- 172.21.15.169:0/51028106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa46c072a10 0x7fa46c072e30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa45c009900 tx=0x7fa45c02bc60 comp rx=0 tx=0).stop 2024-03-28T01:00:37.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.654+0000 7fa4738a1640 1 -- 172.21.15.169:0/51028106 shutdown_connections 2024-03-28T01:00:37.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.654+0000 7fa4738a1640 1 --2- 172.21.15.169:0/51028106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa46c1096a0 0x7fa46c10df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.654+0000 7fa4738a1640 1 --2- 172.21.15.169:0/51028106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa46c108ce0 0x7fa46c109160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.654+0000 7fa4738a1640 1 --2- 172.21.15.169:0/51028106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa46c072a10 0x7fa46c072e30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.654+0000 7fa4738a1640 1 -- 172.21.15.169:0/51028106 >> 172.21.15.169:0/51028106 conn(0x7fa46c06e4c0 msgr2=0x7fa46c070900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:37.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.662+0000 7fa4738a1640 1 -- 172.21.15.169:0/51028106 shutdown_connections 2024-03-28T01:00:37.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.662+0000 7fa4738a1640 1 -- 172.21.15.169:0/51028106 wait complete. 2024-03-28T01:00:37.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.664+0000 7fa4738a1640 1 Processor -- start 2024-03-28T01:00:37.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.664+0000 7fa4738a1640 1 -- start start 2024-03-28T01:00:37.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.664+0000 7fa4738a1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa46c072a10 0x7fa46c114cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:37.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.664+0000 7fa4738a1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa46c108ce0 0x7fa46c115230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:37.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.664+0000 7fa4738a1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa46c1096a0 0x7fa46c1b67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:37.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.664+0000 7fa4738a1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa46c11c0d0 con 0x7fa46c108ce0 2024-03-28T01:00:37.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.664+0000 7fa4738a1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa46c11bf50 con 0x7fa46c072a10 2024-03-28T01:00:37.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.664+0000 7fa4738a1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa46c11c250 con 0x7fa46c1096a0 2024-03-28T01:00:37.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.665+0000 7fa470e3c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa46c108ce0 0x7fa46c115230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:37.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.666+0000 7fa470e3c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa46c108ce0 0x7fa46c115230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36446/0 (socket says 172.21.15.169:36446) 2024-03-28T01:00:37.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.666+0000 7fa470e3c640 1 -- 172.21.15.169:0/2064159087 learned_addr learned my addr 172.21.15.169:0/2064159087 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:37.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.667+0000 7fa471e3e640 1 --2- 172.21.15.169:0/2064159087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa46c1096a0 0x7fa46c1b67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:37.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.671+0000 7fa47163d640 1 --2- 172.21.15.169:0/2064159087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa46c072a10 0x7fa46c114cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:37.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.674+0000 7fa470e3c640 1 -- 172.21.15.169:0/2064159087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa46c1096a0 msgr2=0x7fa46c1b67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:37.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.674+0000 7fa470e3c640 1 --2- 172.21.15.169:0/2064159087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa46c1096a0 0x7fa46c1b67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.674+0000 7fa470e3c640 1 -- 172.21.15.169:0/2064159087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa46c072a10 msgr2=0x7fa46c114cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:37.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.674+0000 7fa470e3c640 1 --2- 172.21.15.169:0/2064159087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa46c072a10 0x7fa46c114cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.674+0000 7fa470e3c640 1 -- 172.21.15.169:0/2064159087 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa46c1158d0 con 0x7fa46c108ce0 2024-03-28T01:00:37.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.676+0000 7fa470e3c640 1 --2- 172.21.15.169:0/2064159087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa46c108ce0 0x7fa46c115230 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa46800b470 tx=0x7fa46800b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:37.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.679+0000 7fa4627fc640 1 -- 172.21.15.169:0/2064159087 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa468015020 con 0x7fa46c108ce0 2024-03-28T01:00:37.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.679+0000 7fa4738a1640 1 -- 172.21.15.169:0/2064159087 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa46c1b6e40 con 0x7fa46c108ce0 2024-03-28T01:00:37.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.679+0000 7fa4738a1640 1 -- 172.21.15.169:0/2064159087 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa46c1b72a0 con 0x7fa46c108ce0 2024-03-28T01:00:37.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.680+0000 7fa4627fc640 1 -- 172.21.15.169:0/2064159087 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa468011d70 con 0x7fa46c108ce0 2024-03-28T01:00:37.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.681+0000 7fa4627fc640 1 -- 172.21.15.169:0/2064159087 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46800f340 con 0x7fa46c108ce0 2024-03-28T01:00:37.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.688+0000 7fa4627fc640 1 -- 172.21.15.169:0/2064159087 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa468018020 con 0x7fa46c108ce0 2024-03-28T01:00:37.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.690+0000 7fa4627fc640 1 --2- 172.21.15.169:0/2064159087 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa4580a2a00 0x7fa4580a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:37.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.690+0000 7fa47656a740 1 Processor -- start 2024-03-28T01:00:37.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.691+0000 7fa47656a740 1 -- start start 2024-03-28T01:00:37.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.691+0000 7fa47656a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602519daf80 0x5602519db3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:37.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.691+0000 7fa47656a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602519db970 0x5602519e0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:37.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.691+0000 7fa47656a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519e0ae0 0x5602519e2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:37.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.691+0000 7fa47656a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602519403d0 con 0x5602519db970 2024-03-28T01:00:37.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.691+0000 7fa47656a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602518e8ad0 con 0x5602519e0ae0 2024-03-28T01:00:37.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.691+0000 7fa47656a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56025193c470 con 0x5602519daf80 2024-03-28T01:00:37.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.694+0000 7fa471e3e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519e0ae0 0x5602519e2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:37.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.696+0000 7fa471e3e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519e0ae0 0x5602519e2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:55944/0 (socket says 172.21.15.169:55944) 2024-03-28T01:00:37.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.696+0000 7fa471e3e640 1 -- 172.21.15.169:0/1264683074 learned_addr learned my addr 172.21.15.169:0/1264683074 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:37.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.696+0000 7fa47163d640 1 --2- 172.21.15.169:0/2064159087 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa4580a2a00 0x7fa4580a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:37.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.697+0000 7fa4627fc640 1 -- 172.21.15.169:0/2064159087 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa4680ce820 con 0x7fa46c108ce0 2024-03-28T01:00:37.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.697+0000 7fa47163d640 1 --2- 172.21.15.169:0/1264683074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602519daf80 0x5602519db3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:37.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.699+0000 7fa470e3c640 1 --2- 172.21.15.169:0/1264683074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602519db970 0x5602519e0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:37.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.700+0000 7fa47163d640 1 -- 172.21.15.169:0/1264683074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519e0ae0 msgr2=0x5602519e2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:37.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.700+0000 7fa47163d640 1 --2- 172.21.15.169:0/1264683074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519e0ae0 0x5602519e2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.700+0000 7fa47163d640 1 -- 172.21.15.169:0/1264683074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602519db970 msgr2=0x5602519e0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:37.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.700+0000 7fa47163d640 1 --2- 172.21.15.169:0/1264683074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602519db970 0x5602519e0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.700+0000 7fa47163d640 1 -- 172.21.15.169:0/1264683074 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56025171e7d0 con 0x5602519daf80 2024-03-28T01:00:37.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.701+0000 7fa47163d640 1 --2- 172.21.15.169:0/1264683074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602519daf80 0x5602519db3a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa45c036040 tx=0x7fa45c0042c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=771eb0e2a364cb67 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:37.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.701+0000 7fa47163d640 1 --2- 172.21.15.169:0/2064159087 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa4580a2a00 0x7fa4580a4ec0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa45c02c740 tx=0x7fa45c03b000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:37.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.701+0000 7fa4577fe640 1 -- 172.21.15.169:0/1264683074 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45c005000 con 0x5602519daf80 2024-03-28T01:00:37.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.702+0000 7fa4577fe640 1 -- 172.21.15.169:0/1264683074 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa45c0035f0 con 0x5602519daf80 2024-03-28T01:00:37.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.702+0000 7fa4577fe640 1 -- 172.21.15.169:0/1264683074 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45c0446e0 con 0x5602519daf80 2024-03-28T01:00:37.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.702+0000 7fa471e3e640 1 --2- 172.21.15.169:0/1264683074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519e0ae0 0x5602519e2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:00:37.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.703+0000 7fa47656a740 1 -- 172.21.15.169:0/1264683074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602519daf80 msgr2=0x5602519db3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:37.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.703+0000 7fa47656a740 1 --2- 172.21.15.169:0/1264683074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602519daf80 0x5602519db3a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa45c036040 tx=0x7fa45c0042c0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.703+0000 7fa47656a740 1 -- 172.21.15.169:0/1264683074 shutdown_connections 2024-03-28T01:00:37.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.703+0000 7fa47656a740 1 --2- 172.21.15.169:0/1264683074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519e0ae0 0x5602519e2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.703+0000 7fa47656a740 1 --2- 172.21.15.169:0/1264683074 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602519db970 0x5602519e0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.703+0000 7fa47656a740 1 --2- 172.21.15.169:0/1264683074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602519daf80 0x5602519db3a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.703+0000 7fa47656a740 1 -- 172.21.15.169:0/1264683074 >> 172.21.15.169:0/1264683074 conn(0x5602519d64f0 msgr2=0x5602519d68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:37.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.706+0000 7fa47656a740 1 -- 172.21.15.169:0/1264683074 shutdown_connections 2024-03-28T01:00:37.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.706+0000 7fa47656a740 1 -- 172.21.15.169:0/1264683074 wait complete. 2024-03-28T01:00:37.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.707+0000 7fa47656a740 1 -- start start 2024-03-28T01:00:37.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.707+0000 7fa47656a740 1 Processor -- start 2024-03-28T01:00:37.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.710+0000 7fa47656a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519daf80 0x5602519e5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:37.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.711+0000 7fa47656a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602519db970 0x5602519e5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:37.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.711+0000 7fa47656a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602519e0ae0 0x5602519ec940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:37.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.711+0000 7fa47656a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602519e39d0 con 0x5602519e0ae0 2024-03-28T01:00:37.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.711+0000 7fa47656a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602518fc060 con 0x5602519daf80 2024-03-28T01:00:37.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.711+0000 7fa47656a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602519e3b50 con 0x5602519db970 2024-03-28T01:00:37.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.711+0000 7fa47656a740 10 client.0 _ll_register_callbacks cb 0x5602518de6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:37.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.712+0000 7fa47163d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519daf80 0x5602519e5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:37.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.712+0000 7fa47163d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519daf80 0x5602519e5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:55948/0 (socket says 172.21.15.169:55948) 2024-03-28T01:00:37.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.712+0000 7fa47163d640 1 -- 172.21.15.169:0/55620988 learned_addr learned my addr 172.21.15.169:0/55620988 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:37.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.712+0000 7fa47163d640 1 -- 172.21.15.169:0/55620988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602519db970 msgr2=0x5602519e5970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:00:37.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.712+0000 7fa47163d640 1 --2- 172.21.15.169:0/55620988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602519db970 0x5602519e5970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.712+0000 7fa47163d640 1 -- 172.21.15.169:0/55620988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602519e0ae0 msgr2=0x5602519ec940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:00:37.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.712+0000 7fa47163d640 1 --2- 172.21.15.169:0/55620988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602519e0ae0 0x5602519ec940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.712+0000 7fa47163d640 1 -- 172.21.15.169:0/55620988 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5602516543f0 con 0x5602519daf80 2024-03-28T01:00:37.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.713+0000 7fa47163d640 1 --2- 172.21.15.169:0/55620988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519daf80 0x5602519e5430 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa45c009900 tx=0x7fa45c004f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be104c9f47ee94c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:37.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.713+0000 7fa454ff9640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:00:37.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.713+0000 7fa454ff9640 1 -- 172.21.15.169:0/55620988 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45c02ca50 con 0x5602519daf80 2024-03-28T01:00:37.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.713+0000 7fa47656a740 1 -- 172.21.15.169:0/55620988 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa45c0035f0 con 0x5602519daf80 2024-03-28T01:00:37.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.714+0000 7fa454ff9640 1 -- 172.21.15.169:0/55620988 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa45c004b50 con 0x5602519daf80 2024-03-28T01:00:37.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.714+0000 7fa454ff9640 1 -- 172.21.15.169:0/55620988 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa45c005210 con 0x5602519daf80 2024-03-28T01:00:37.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.715+0000 7fa47656a740 10 client.26135 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:37.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.715+0000 7fa47656a740 1 -- 172.21.15.169:0/55620988 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5602516140f0 con 0x5602519daf80 2024-03-28T01:00:37.720 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.717+0000 7fa454ff9640 1 -- 172.21.15.169:0/55620988 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa45c0035f0 con 0x5602519daf80 2024-03-28T01:00:37.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.718+0000 7fa454ff9640 1 -- 172.21.15.169:0/55620988 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa45c04b070 con 0x5602519daf80 2024-03-28T01:00:37.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.719+0000 7fa47656a740 10 client.26135 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:37.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.719+0000 7fa47656a740 10 client.26135 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:37.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.719+0000 7fa47656a740 20 client.26135 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:37.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.719+0000 7fa47656a740 1 --2- 172.21.15.169:0/55620988 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5602519f0dc0 0x5602519f3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:37.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.719+0000 7fa47656a740 4 client.26135 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:00:37.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.719+0000 7fa47656a740 1 -- 172.21.15.169:0/55620988 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56025194d910 con 0x5602519f0dc0 2024-03-28T01:00:37.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.719+0000 7fa470e3c640 1 --2- 172.21.15.169:0/55620988 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5602519f0dc0 0x5602519f3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:37.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.742+0000 7fa454ff9640 10 client.26135 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:37.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.742+0000 7fa470e3c640 1 --2- 172.21.15.169:0/55620988 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5602519f0dc0 0x5602519f3280 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b07ab13ecc2d3b6d server_cookie=668890cd8594b56 in_seq=0 out_seq=0 2024-03-28T01:00:37.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.750+0000 7fa454ff9640 1 -- 172.21.15.169:0/55620988 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56025194d910 con 0x5602519f0dc0 2024-03-28T01:00:37.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.750+0000 7fa454ff9640 10 client.26135 handle_command_reply: tid=0 2024-03-28T01:00:37.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.750+0000 7fa47656a740 1 client.26135 shutdown 2024-03-28T01:00:37.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.754+0000 7fa47656a740 1 -- 172.21.15.169:0/55620988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519daf80 msgr2=0x5602519e5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:37.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.754+0000 7fa47656a740 1 --2- 172.21.15.169:0/55620988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519daf80 0x5602519e5430 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa45c009900 tx=0x7fa45c004f10 comp rx=0 tx=0).stop 2024-03-28T01:00:37.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.754+0000 7fa47656a740 1 -- 172.21.15.169:0/55620988 shutdown_connections 2024-03-28T01:00:37.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.754+0000 7fa47656a740 1 --2- 172.21.15.169:0/55620988 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5602519f0dc0 0x5602519f3280 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.754+0000 7fa47656a740 1 --2- 172.21.15.169:0/55620988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602519e0ae0 0x5602519ec940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.754+0000 7fa47656a740 1 --2- 172.21.15.169:0/55620988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602519db970 0x5602519e5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.754+0000 7fa47656a740 1 --2- 172.21.15.169:0/55620988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519daf80 0x5602519e5430 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.754+0000 7fa47656a740 1 -- 172.21.15.169:0/55620988 >> 172.21.15.169:0/55620988 conn(0x5602519d64f0 msgr2=0x5602519d57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:37.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.754+0000 7fa454ff9640 0 client.26135 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:37.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.754+0000 7fa47656a740 1 -- 172.21.15.169:0/55620988 shutdown_connections 2024-03-28T01:00:37.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.754+0000 7fa47656a740 1 -- 172.21.15.169:0/55620988 wait complete. 2024-03-28T01:00:37.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.755+0000 7fa47656a740 20 client.26135 trim_cache size 0 max 16384 2024-03-28T01:00:37.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.791+0000 7fa47656a740 1 Processor -- start 2024-03-28T01:00:37.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.791+0000 7fa47656a740 1 -- start start 2024-03-28T01:00:37.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.792+0000 7fa47656a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602519daf80 0x5602519e5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:37.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.792+0000 7fa47656a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602519db970 0x5602519ec900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:37.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.792+0000 7fa47656a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519e0ae0 0x5602519d4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:37.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.792+0000 7fa47656a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa450011e20 con 0x5602519daf80 2024-03-28T01:00:37.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.792+0000 7fa47656a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45000f920 con 0x5602519e0ae0 2024-03-28T01:00:37.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.792+0000 7fa47656a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45000da70 con 0x5602519db970 2024-03-28T01:00:37.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.793+0000 7fa47163d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602519daf80 0x5602519e5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:37.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.793+0000 7fa47163d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602519daf80 0x5602519e5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36472/0 (socket says 172.21.15.169:36472) 2024-03-28T01:00:37.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.793+0000 7fa47163d640 1 -- 172.21.15.169:0/3823636447 learned_addr learned my addr 172.21.15.169:0/3823636447 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:37.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.793+0000 7fa471e3e640 1 --2- 172.21.15.169:0/3823636447 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519e0ae0 0x5602519d4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:37.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.794+0000 7fa470e3c640 1 --2- 172.21.15.169:0/3823636447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602519db970 0x5602519ec900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:37.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.794+0000 7fa471e3e640 1 -- 172.21.15.169:0/3823636447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602519db970 msgr2=0x5602519ec900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:37.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.794+0000 7fa471e3e640 1 --2- 172.21.15.169:0/3823636447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602519db970 0x5602519ec900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.794+0000 7fa471e3e640 1 -- 172.21.15.169:0/3823636447 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602519daf80 msgr2=0x5602519e5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:37.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.794+0000 7fa471e3e640 1 --2- 172.21.15.169:0/3823636447 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602519daf80 0x5602519e5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.794+0000 7fa471e3e640 1 -- 172.21.15.169:0/3823636447 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602519e3c60 con 0x5602519e0ae0 2024-03-28T01:00:37.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.795+0000 7fa471e3e640 1 --2- 172.21.15.169:0/3823636447 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519e0ae0 0x5602519d4f20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x5602519d7460 tx=0x7fa464009870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b17638c2cb4be17 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:37.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.799+0000 7fa456ffd640 1 -- 172.21.15.169:0/3823636447 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46401c070 con 0x5602519e0ae0 2024-03-28T01:00:37.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.799+0000 7fa456ffd640 1 -- 172.21.15.169:0/3823636447 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa4640159c0 con 0x5602519e0ae0 2024-03-28T01:00:37.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.800+0000 7fa47656a740 1 -- 172.21.15.169:0/3823636447 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519e0ae0 msgr2=0x5602519d4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:37.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.800+0000 7fa47656a740 1 --2- 172.21.15.169:0/3823636447 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519e0ae0 0x5602519d4f20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x5602519d7460 tx=0x7fa464009870 comp rx=0 tx=0).stop 2024-03-28T01:00:37.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.801+0000 7fa47656a740 1 -- 172.21.15.169:0/3823636447 shutdown_connections 2024-03-28T01:00:37.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.801+0000 7fa47656a740 1 --2- 172.21.15.169:0/3823636447 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519e0ae0 0x5602519d4f20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.801+0000 7fa47656a740 1 --2- 172.21.15.169:0/3823636447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602519db970 0x5602519ec900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.801+0000 7fa47656a740 1 --2- 172.21.15.169:0/3823636447 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602519daf80 0x5602519e5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.801+0000 7fa47656a740 1 -- 172.21.15.169:0/3823636447 >> 172.21.15.169:0/3823636447 conn(0x5602519d64f0 msgr2=0x5602519e5040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:37.805 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:37 smithi169 ceph-mon[32274]: pgmap v474: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 813 MiB/s wr, 1.25k op/s 2024-03-28T01:00:37.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.802+0000 7fa47656a740 1 -- 172.21.15.169:0/3823636447 shutdown_connections 2024-03-28T01:00:37.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.802+0000 7fa47656a740 1 -- 172.21.15.169:0/3823636447 wait complete. 2024-03-28T01:00:37.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.802+0000 7fa47656a740 1 -- start start 2024-03-28T01:00:37.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.802+0000 7fa47656a740 1 Processor -- start 2024-03-28T01:00:37.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.803+0000 7fa47656a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519daf80 0x5602519ea050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:37.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.803+0000 7fa47656a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602519db970 0x5602519ea590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:37.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.803+0000 7fa47656a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602519e0ae0 0x5602519eb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:37.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.803+0000 7fa47656a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45000dfe0 con 0x5602519e0ae0 2024-03-28T01:00:37.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.803+0000 7fa47656a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45000f2c0 con 0x5602519daf80 2024-03-28T01:00:37.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.803+0000 7fa47656a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45000e600 con 0x5602519db970 2024-03-28T01:00:37.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.803+0000 7fa47656a740 10 client.0 _ll_register_callbacks cb 0x7fa468009320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:37.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.804+0000 7fa47163d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519daf80 0x5602519ea050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:37.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.804+0000 7fa47163d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519daf80 0x5602519ea050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:55972/0 (socket says 172.21.15.169:55972) 2024-03-28T01:00:37.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.804+0000 7fa47163d640 1 -- 172.21.15.169:0/2534993954 learned_addr learned my addr 172.21.15.169:0/2534993954 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:37.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.804+0000 7fa470e3c640 1 --2- 172.21.15.169:0/2534993954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602519db970 0x5602519ea590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:37.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.805+0000 7fa47163d640 1 -- 172.21.15.169:0/2534993954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602519db970 msgr2=0x5602519ea590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:37.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.805+0000 7fa47163d640 1 --2- 172.21.15.169:0/2534993954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602519db970 0x5602519ea590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.805+0000 7fa47163d640 1 -- 172.21.15.169:0/2534993954 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602519e0ae0 msgr2=0x5602519eb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:37.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.805+0000 7fa47163d640 1 --2- 172.21.15.169:0/2534993954 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602519e0ae0 0x5602519eb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.805+0000 7fa47163d640 1 -- 172.21.15.169:0/2534993954 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5602519ebb40 con 0x5602519daf80 2024-03-28T01:00:37.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.807+0000 7fa47163d640 1 --2- 172.21.15.169:0/2534993954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519daf80 0x5602519ea050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa45c009900 tx=0x7fa45c02c200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9a49fe3476a1bd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:37.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.807+0000 7fa454ff9640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:00:37.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.807+0000 7fa454ff9640 1 -- 172.21.15.169:0/2534993954 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45c0055c0 con 0x5602519daf80 2024-03-28T01:00:37.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.807+0000 7fa454ff9640 1 -- 172.21.15.169:0/2534993954 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa45c04daa0 con 0x5602519daf80 2024-03-28T01:00:37.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.808+0000 7fa47656a740 1 -- 172.21.15.169:0/2534993954 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa4640159c0 con 0x5602519daf80 2024-03-28T01:00:37.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.809+0000 7fa454ff9640 1 -- 172.21.15.169:0/2534993954 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa45c004240 con 0x5602519daf80 2024-03-28T01:00:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.812+0000 7fa47656a740 10 client.26141 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.812+0000 7fa47656a740 1 -- 172.21.15.169:0/2534993954 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x560251a4c7b0 con 0x5602519daf80 2024-03-28T01:00:37.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.814+0000 7fa454ff9640 1 -- 172.21.15.169:0/2534993954 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa4640159c0 con 0x5602519daf80 2024-03-28T01:00:37.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.814+0000 7fa454ff9640 1 -- 172.21.15.169:0/2534993954 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa45c04b070 con 0x5602519daf80 2024-03-28T01:00:37.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.814+0000 7fa47656a740 10 client.26141 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:37.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.814+0000 7fa47656a740 10 client.26141 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:37.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.814+0000 7fa47656a740 20 client.26141 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:37.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.814+0000 7fa47656a740 1 --2- 172.21.15.169:0/2534993954 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560251a4cfb0 0x560251a4f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:37.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.814+0000 7fa47656a740 4 client.26141 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:00:37.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.814+0000 7fa47656a740 1 -- 172.21.15.169:0/2534993954 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560251947c60 con 0x560251a4cfb0 2024-03-28T01:00:37.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.815+0000 7fa470e3c640 1 --2- 172.21.15.169:0/2534993954 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560251a4cfb0 0x560251a4f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:37.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.819+0000 7fa470e3c640 1 --2- 172.21.15.169:0/2534993954 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560251a4cfb0 0x560251a4f3a0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd03d47c18daf728 server_cookie=47d380e77e1b09b1 in_seq=0 out_seq=0 2024-03-28T01:00:37.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.820+0000 7fa454ff9640 10 client.26141 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:37.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.824+0000 7fa454ff9640 1 -- 172.21.15.169:0/2534993954 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x560251947c60 con 0x560251a4cfb0 2024-03-28T01:00:37.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.824+0000 7fa454ff9640 10 client.26141 handle_command_reply: tid=0 2024-03-28T01:00:37.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.824+0000 7fa47656a740 1 client.26141 shutdown 2024-03-28T01:00:37.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.827+0000 7fa47656a740 1 -- 172.21.15.169:0/2534993954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519daf80 msgr2=0x5602519ea050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:37.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.827+0000 7fa47656a740 1 --2- 172.21.15.169:0/2534993954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519daf80 0x5602519ea050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa45c009900 tx=0x7fa45c02c200 comp rx=0 tx=0).stop 2024-03-28T01:00:37.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.828+0000 7fa47656a740 1 -- 172.21.15.169:0/2534993954 shutdown_connections 2024-03-28T01:00:37.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.828+0000 7fa47656a740 1 --2- 172.21.15.169:0/2534993954 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560251a4cfb0 0x560251a4f3a0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.828+0000 7fa47656a740 1 --2- 172.21.15.169:0/2534993954 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5602519e0ae0 0x5602519eb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.828+0000 7fa47656a740 1 --2- 172.21.15.169:0/2534993954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602519db970 0x5602519ea590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.828+0000 7fa47656a740 1 --2- 172.21.15.169:0/2534993954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602519daf80 0x5602519ea050 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.828+0000 7fa47656a740 1 -- 172.21.15.169:0/2534993954 >> 172.21.15.169:0/2534993954 conn(0x5602519d64f0 msgr2=0x5602519d9750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:37.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.829+0000 7fa454ff9640 0 client.26141 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:37.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.830+0000 7fa47656a740 1 -- 172.21.15.169:0/2534993954 shutdown_connections 2024-03-28T01:00:37.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.833+0000 7fa47656a740 1 -- 172.21.15.169:0/2534993954 wait complete. 2024-03-28T01:00:37.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.836+0000 7fa47656a740 20 client.26141 trim_cache size 0 max 16384 2024-03-28T01:00:37.846 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:00:37.847 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "no active scrubs running", 2024-03-28T01:00:37.847 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": {} 2024-03-28T01:00:37.847 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:00:37.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.850+0000 7fa4738a1640 1 -- 172.21.15.169:0/2064159087 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa4580a2a00 msgr2=0x7fa4580a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:37.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.850+0000 7fa4738a1640 1 --2- 172.21.15.169:0/2064159087 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa4580a2a00 0x7fa4580a4ec0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa45c02c740 tx=0x7fa45c03b000 comp rx=0 tx=0).stop 2024-03-28T01:00:37.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.850+0000 7fa4738a1640 1 -- 172.21.15.169:0/2064159087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa46c108ce0 msgr2=0x7fa46c115230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:37.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.850+0000 7fa4738a1640 1 --2- 172.21.15.169:0/2064159087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa46c108ce0 0x7fa46c115230 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa46800b470 tx=0x7fa46800b930 comp rx=0 tx=0).stop 2024-03-28T01:00:37.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.852+0000 7fa4738a1640 1 -- 172.21.15.169:0/2064159087 shutdown_connections 2024-03-28T01:00:37.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.852+0000 7fa4738a1640 1 --2- 172.21.15.169:0/2064159087 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa4580a2a00 0x7fa4580a4ec0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.852+0000 7fa4738a1640 1 --2- 172.21.15.169:0/2064159087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa46c1096a0 0x7fa46c1b67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.852+0000 7fa4738a1640 1 --2- 172.21.15.169:0/2064159087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa46c108ce0 0x7fa46c115230 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.853+0000 7fa4738a1640 1 --2- 172.21.15.169:0/2064159087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa46c072a10 0x7fa46c114cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:37.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.853+0000 7fa4738a1640 1 -- 172.21.15.169:0/2064159087 >> 172.21.15.169:0/2064159087 conn(0x7fa46c06e4c0 msgr2=0x7fa46c070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:37.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:37 smithi174 ceph-mon[36060]: pgmap v474: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 813 MiB/s wr, 1.25k op/s 2024-03-28T01:00:37.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.856+0000 7fa4738a1640 1 -- 172.21.15.169:0/2064159087 shutdown_connections 2024-03-28T01:00:37.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:37.856+0000 7fa4738a1640 1 -- 172.21.15.169:0/2064159087 wait complete. 2024-03-28T01:00:37.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:37 smithi177 ceph-mon[36206]: pgmap v474: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 813 MiB/s wr, 1.25k op/s 2024-03-28T01:00:37.965 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T01:00:37.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T01:00:38.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:38 smithi169 ceph-mon[32274]: pgmap v475: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 744 MiB/s wr, 1.22k op/s 2024-03-28T01:00:38.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:38 smithi174 ceph-mon[36060]: pgmap v475: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 744 MiB/s wr, 1.22k op/s 2024-03-28T01:00:38.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:38 smithi177 ceph-mon[36206]: pgmap v475: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 744 MiB/s wr, 1.22k op/s 2024-03-28T01:00:39.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.224+0000 7f69e28d5640 1 -- 172.21.15.169:0/3673928683 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69dc0ffd60 msgr2=0x7f69dc102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:39.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.224+0000 7f69e28d5640 1 --2- 172.21.15.169:0/3673928683 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69dc0ffd60 0x7f69dc102190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009ca0 tx=0x7f69d402c120 comp rx=0 tx=0).stop 2024-03-28T01:00:39.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.224+0000 7f69e28d5640 1 -- 172.21.15.169:0/3673928683 shutdown_connections 2024-03-28T01:00:39.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.225+0000 7f69e28d5640 1 --2- 172.21.15.169:0/3673928683 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69dc105080 0x7f69dc105500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:39.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.225+0000 7f69e28d5640 1 --2- 172.21.15.169:0/3673928683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69dc1026d0 0x7f69dc104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:39.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.225+0000 7f69e28d5640 1 --2- 172.21.15.169:0/3673928683 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69dc0ffd60 0x7f69dc102190 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:39.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.225+0000 7f69e28d5640 1 -- 172.21.15.169:0/3673928683 >> 172.21.15.169:0/3673928683 conn(0x7f69dc0fb980 msgr2=0x7f69dc0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:39.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.225+0000 7f69e28d5640 1 -- 172.21.15.169:0/3673928683 shutdown_connections 2024-03-28T01:00:39.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.225+0000 7f69e28d5640 1 -- 172.21.15.169:0/3673928683 wait complete. 2024-03-28T01:00:39.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.226+0000 7f69e28d5640 1 Processor -- start 2024-03-28T01:00:39.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.226+0000 7f69e28d5640 1 -- start start 2024-03-28T01:00:39.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.226+0000 7f69e28d5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69dc1026d0 0x7f69dc19e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:39.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.226+0000 7f69e28d5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69dc105080 0x7f69dc19e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:39.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.226+0000 7f69e28d5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69dc1a3630 0x7f69dc1a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:39.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.226+0000 7f69e28d5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc113810 con 0x7f69dc1026d0 2024-03-28T01:00:39.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.226+0000 7f69e28d5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc113690 con 0x7f69dc1a3630 2024-03-28T01:00:39.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.226+0000 7f69e28d5640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc113990 con 0x7f69dc105080 2024-03-28T01:00:39.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.226+0000 7f69e20d4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69dc1a3630 0x7f69dc1a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:39.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.226+0000 7f69e10d2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69dc105080 0x7f69dc19e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:39.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.226+0000 7f69e20d4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69dc1a3630 0x7f69dc1a5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:55976/0 (socket says 172.21.15.169:55976) 2024-03-28T01:00:39.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.226+0000 7f69e20d4640 1 -- 172.21.15.169:0/4188778190 learned_addr learned my addr 172.21.15.169:0/4188778190 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:39.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.226+0000 7f69e18d3640 1 --2- 172.21.15.169:0/4188778190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69dc1026d0 0x7f69dc19e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:39.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.227+0000 7f69e20d4640 1 -- 172.21.15.169:0/4188778190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69dc105080 msgr2=0x7f69dc19e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:39.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.227+0000 7f69e20d4640 1 --2- 172.21.15.169:0/4188778190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69dc105080 0x7f69dc19e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:39.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.227+0000 7f69e20d4640 1 -- 172.21.15.169:0/4188778190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69dc1026d0 msgr2=0x7f69dc19e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:39.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.227+0000 7f69e20d4640 1 --2- 172.21.15.169:0/4188778190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69dc1026d0 0x7f69dc19e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:39.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.227+0000 7f69e20d4640 1 -- 172.21.15.169:0/4188778190 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69dc1a6090 con 0x7f69dc1a3630 2024-03-28T01:00:39.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.228+0000 7f69e20d4640 1 --2- 172.21.15.169:0/4188778190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69dc1a3630 0x7f69dc1a5a20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f69d8009fd0 tx=0x7f69d800b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:39.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.228+0000 7f69d2ffd640 1 -- 172.21.15.169:0/4188778190 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d8011830 con 0x7f69dc1a3630 2024-03-28T01:00:39.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.228+0000 7f69d2ffd640 1 -- 172.21.15.169:0/4188778190 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69d8016070 con 0x7f69dc1a3630 2024-03-28T01:00:39.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.228+0000 7f69d2ffd640 1 -- 172.21.15.169:0/4188778190 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d800f2e0 con 0x7f69dc1a3630 2024-03-28T01:00:39.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.229+0000 7f69e28d5640 1 -- 172.21.15.169:0/4188778190 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69dc1a6380 con 0x7f69dc1a3630 2024-03-28T01:00:39.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.229+0000 7f69e28d5640 1 -- 172.21.15.169:0/4188778190 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69dc1a6900 con 0x7f69dc1a3630 2024-03-28T01:00:39.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.231+0000 7f69e28d5640 1 -- 172.21.15.169:0/4188778190 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69dc0fd470 con 0x7f69dc1a3630 2024-03-28T01:00:39.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.233+0000 7f69d2ffd640 1 -- 172.21.15.169:0/4188778190 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f69d80119d0 con 0x7f69dc1a3630 2024-03-28T01:00:39.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.234+0000 7f69d2ffd640 1 --2- 172.21.15.169:0/4188778190 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f69c80a28d0 0x7f69c80a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:39.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.234+0000 7f69d2ffd640 1 -- 172.21.15.169:0/4188778190 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f69d80ce390 con 0x7f69dc1a3630 2024-03-28T01:00:39.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.235+0000 7f69e18d3640 1 --2- 172.21.15.169:0/4188778190 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f69c80a28d0 0x7f69c80a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:39.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.238+0000 7f69e18d3640 1 --2- 172.21.15.169:0/4188778190 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f69c80a28d0 0x7f69c80a4d90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009bf0 tx=0x7f69d402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:39.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.247+0000 7f69d2ffd640 1 -- 172.21.15.169:0/4188778190 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f69d809b130 con 0x7f69dc1a3630 2024-03-28T01:00:39.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.884+0000 7f69e28d5640 1 -- 172.21.15.169:0/4188778190 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69dc02d250 con 0x7f69dc1a3630 2024-03-28T01:00:39.894 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T01:00:39.894 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T01:00:39.895 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T01:00:39.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.890+0000 7f69d2ffd640 1 -- 172.21.15.169:0/4188778190 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7f69d809aa30 con 0x7f69dc1a3630 2024-03-28T01:00:39.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.908+0000 7f69e28d5640 1 -- 172.21.15.169:0/4188778190 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f69c80a28d0 msgr2=0x7f69c80a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:39.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.908+0000 7f69e28d5640 1 --2- 172.21.15.169:0/4188778190 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f69c80a28d0 0x7f69c80a4d90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009bf0 tx=0x7f69d402e040 comp rx=0 tx=0).stop 2024-03-28T01:00:39.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.908+0000 7f69e28d5640 1 -- 172.21.15.169:0/4188778190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69dc1a3630 msgr2=0x7f69dc1a5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:39.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.908+0000 7f69e28d5640 1 --2- 172.21.15.169:0/4188778190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69dc1a3630 0x7f69dc1a5a20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f69d8009fd0 tx=0x7f69d800b8e0 comp rx=0 tx=0).stop 2024-03-28T01:00:39.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.914+0000 7f69e28d5640 1 -- 172.21.15.169:0/4188778190 shutdown_connections 2024-03-28T01:00:39.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.914+0000 7f69e28d5640 1 --2- 172.21.15.169:0/4188778190 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f69c80a28d0 0x7f69c80a4d90 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:39.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.914+0000 7f69e28d5640 1 --2- 172.21.15.169:0/4188778190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69dc1a3630 0x7f69dc1a5a20 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:39.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.914+0000 7f69e28d5640 1 --2- 172.21.15.169:0/4188778190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69dc105080 0x7f69dc19e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:39.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.914+0000 7f69e28d5640 1 --2- 172.21.15.169:0/4188778190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69dc1026d0 0x7f69dc19e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:39.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.914+0000 7f69e28d5640 1 -- 172.21.15.169:0/4188778190 >> 172.21.15.169:0/4188778190 conn(0x7f69dc0fb980 msgr2=0x7f69dc103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:39.920 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.918+0000 7f69e28d5640 1 -- 172.21.15.169:0/4188778190 shutdown_connections 2024-03-28T01:00:39.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:39.921+0000 7f69e28d5640 1 -- 172.21.15.169:0/4188778190 wait complete. 2024-03-28T01:00:39.975 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:00:39.975 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T01:00:40.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.290+0000 7f88fe2fe640 1 -- 172.21.15.169:0/4044287046 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88f81097b0 msgr2=0x7f88f810bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:40.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.290+0000 7f88fe2fe640 1 --2- 172.21.15.169:0/4044287046 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88f81097b0 0x7f88f810bba0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f88f0009ca0 tx=0x7f88f002c0f0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.290+0000 7f88fe2fe640 1 -- 172.21.15.169:0/4044287046 shutdown_connections 2024-03-28T01:00:40.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.290+0000 7f88fe2fe640 1 --2- 172.21.15.169:0/4044287046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88f810ea60 0x7f88f8110ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.290+0000 7f88fe2fe640 1 --2- 172.21.15.169:0/4044287046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88f810c0e0 0x7f88f810e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.290+0000 7f88fe2fe640 1 --2- 172.21.15.169:0/4044287046 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88f81097b0 0x7f88f810bba0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.290+0000 7f88fe2fe640 1 -- 172.21.15.169:0/4044287046 >> 172.21.15.169:0/4044287046 conn(0x7f88f806e500 msgr2=0x7f88f8070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:40.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.291+0000 7f88fe2fe640 1 -- 172.21.15.169:0/4044287046 shutdown_connections 2024-03-28T01:00:40.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.291+0000 7f88fe2fe640 1 -- 172.21.15.169:0/4044287046 wait complete. 2024-03-28T01:00:40.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.292+0000 7f88fe2fe640 1 Processor -- start 2024-03-28T01:00:40.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.292+0000 7f88fe2fe640 1 -- start start 2024-03-28T01:00:40.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.292+0000 7f88fe2fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88f81097b0 0x7f88f8117e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:40.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.293+0000 7f88fe2fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88f810c0e0 0x7f88f81183a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:40.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.293+0000 7f88fe2fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88f810ea60 0x7f88f8119390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:40.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.293+0000 7f88fe2fe640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f8112610 con 0x7f88f81097b0 2024-03-28T01:00:40.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.293+0000 7f88f7fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88f81097b0 0x7f88f8117e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:40.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.293+0000 7f88f7fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88f81097b0 0x7f88f8117e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36516/0 (socket says 172.21.15.169:36516) 2024-03-28T01:00:40.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.293+0000 7f88f7fff640 1 -- 172.21.15.169:0/2952286010 learned_addr learned my addr 172.21.15.169:0/2952286010 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:40.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.293+0000 7f88fe2fe640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f8112490 con 0x7f88f810ea60 2024-03-28T01:00:40.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.293+0000 7f88fe2fe640 1 -- 172.21.15.169:0/2952286010 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f8112790 con 0x7f88f810c0e0 2024-03-28T01:00:40.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.293+0000 7f88f77fe640 1 --2- 172.21.15.169:0/2952286010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88f810c0e0 0x7f88f81183a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:40.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.294+0000 7f88f7fff640 1 -- 172.21.15.169:0/2952286010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88f810c0e0 msgr2=0x7f88f81183a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:40.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.294+0000 7f88f7fff640 1 --2- 172.21.15.169:0/2952286010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88f810c0e0 0x7f88f81183a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.294+0000 7f88f7fff640 1 -- 172.21.15.169:0/2952286010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88f810ea60 msgr2=0x7f88f8119390 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:00:40.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.294+0000 7f88f7fff640 1 --2- 172.21.15.169:0/2952286010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88f810ea60 0x7f88f8119390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.294+0000 7f88f7fff640 1 -- 172.21.15.169:0/2952286010 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f8119a00 con 0x7f88f81097b0 2024-03-28T01:00:40.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.294+0000 7f88f7fff640 1 --2- 172.21.15.169:0/2952286010 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88f81097b0 0x7f88f8117e60 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f88f0037040 tx=0x7f88f0035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:40.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.296+0000 7f88f57fa640 1 -- 172.21.15.169:0/2952286010 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0039040 con 0x7f88f81097b0 2024-03-28T01:00:40.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.296+0000 7f88f57fa640 1 -- 172.21.15.169:0/2952286010 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88f0044070 con 0x7f88f81097b0 2024-03-28T01:00:40.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.297+0000 7f88f57fa640 1 -- 172.21.15.169:0/2952286010 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0004280 con 0x7f88f81097b0 2024-03-28T01:00:40.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.297+0000 7f88fe2fe640 1 -- 172.21.15.169:0/2952286010 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f8119c90 con 0x7f88f81097b0 2024-03-28T01:00:40.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.297+0000 7f88fe2fe640 1 -- 172.21.15.169:0/2952286010 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f811a100 con 0x7f88f81097b0 2024-03-28T01:00:40.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:39 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4188778190' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:00:40.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.301+0000 7f8900fbd740 1 Processor -- start 2024-03-28T01:00:40.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.301+0000 7f8900fbd740 1 -- start start 2024-03-28T01:00:40.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.301+0000 7f8900fbd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e9996690 0x55a5e9996ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:40.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.301+0000 7f8900fbd740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a5e9997080 0x55a5e999bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:40.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.301+0000 7f8900fbd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e999c1f0 0x55a5e999e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:40.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.301+0000 7f8900fbd740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5e9959610 con 0x55a5e9997080 2024-03-28T01:00:40.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.301+0000 7f8900fbd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5e98fc3c0 con 0x55a5e999c1f0 2024-03-28T01:00:40.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.301+0000 7f8900fbd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5e98a4d60 con 0x55a5e9996690 2024-03-28T01:00:40.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.302+0000 7f88fc89b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e9996690 0x55a5e9996ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:40.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.302+0000 7f88fc89b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e9996690 0x55a5e9996ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:33054/0 (socket says 172.21.15.169:33054) 2024-03-28T01:00:40.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.302+0000 7f88fc89b640 1 -- 172.21.15.169:0/230120611 learned_addr learned my addr 172.21.15.169:0/230120611 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:40.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.302+0000 7f88fc89b640 1 -- 172.21.15.169:0/230120611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e999c1f0 msgr2=0x55a5e999e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:40.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.302+0000 7f88fc89b640 1 --2- 172.21.15.169:0/230120611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e999c1f0 0x55a5e999e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.302+0000 7f88fc89b640 1 -- 172.21.15.169:0/230120611 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a5e9997080 msgr2=0x55a5e999bb80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:00:40.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.302+0000 7f88fc89b640 1 --2- 172.21.15.169:0/230120611 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a5e9997080 0x55a5e999bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.302+0000 7f88fc89b640 1 -- 172.21.15.169:0/230120611 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5e96103f0 con 0x55a5e9996690 2024-03-28T01:00:40.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.303+0000 7f88fc89b640 1 --2- 172.21.15.169:0/230120611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e9996690 0x55a5e9996ab0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f88ec009b70 tx=0x7f88ec00e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0d7482bc82d29e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:40.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.303+0000 7f88de7fc640 1 -- 172.21.15.169:0/230120611 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec009e60 con 0x55a5e9996690 2024-03-28T01:00:40.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.303+0000 7f88de7fc640 1 -- 172.21.15.169:0/230120611 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88ec005500 con 0x55a5e9996690 2024-03-28T01:00:40.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.303+0000 7f88de7fc640 1 -- 172.21.15.169:0/230120611 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec010690 con 0x55a5e9996690 2024-03-28T01:00:40.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.303+0000 7f88f57fa640 1 -- 172.21.15.169:0/2952286010 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f88f0004420 con 0x7f88f81097b0 2024-03-28T01:00:40.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.304+0000 7f8900fbd740 1 -- 172.21.15.169:0/230120611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e9996690 msgr2=0x55a5e9996ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:40.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.304+0000 7f8900fbd740 1 --2- 172.21.15.169:0/230120611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e9996690 0x55a5e9996ab0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f88ec009b70 tx=0x7f88ec00e720 comp rx=0 tx=0).stop 2024-03-28T01:00:40.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.305+0000 7f88f57fa640 1 --2- 172.21.15.169:0/2952286010 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f88e00a2720 0x7f88e00a4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:40.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.305+0000 7f88f57fa640 1 -- 172.21.15.169:0/2952286010 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f88f00f0b30 con 0x7f88f81097b0 2024-03-28T01:00:40.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.306+0000 7f88fc89b640 1 --2- 172.21.15.169:0/2952286010 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f88e00a2720 0x7f88e00a4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:40.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.307+0000 7f8900fbd740 1 -- 172.21.15.169:0/230120611 shutdown_connections 2024-03-28T01:00:40.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.307+0000 7f8900fbd740 1 --2- 172.21.15.169:0/230120611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e999c1f0 0x55a5e999e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.307+0000 7f8900fbd740 1 --2- 172.21.15.169:0/230120611 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a5e9997080 0x55a5e999bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.307+0000 7f8900fbd740 1 --2- 172.21.15.169:0/230120611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e9996690 0x55a5e9996ab0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.307+0000 7f8900fbd740 1 -- 172.21.15.169:0/230120611 >> 172.21.15.169:0/230120611 conn(0x55a5e9991c00 msgr2=0x55a5e9992000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:40.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.307+0000 7f8900fbd740 1 -- 172.21.15.169:0/230120611 shutdown_connections 2024-03-28T01:00:40.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.307+0000 7f8900fbd740 1 -- 172.21.15.169:0/230120611 wait complete. 2024-03-28T01:00:40.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.307+0000 7f8900fbd740 1 -- start start 2024-03-28T01:00:40.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.308+0000 7f8900fbd740 1 Processor -- start 2024-03-28T01:00:40.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.309+0000 7f8900fbd740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a5e9996690 0x55a5e99a0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:40.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.309+0000 7f8900fbd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e9997080 0x55a5e99a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:40.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.309+0000 7f8900fbd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e999c1f0 0x55a5e99a7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:40.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.309+0000 7f8900fbd740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5e98b8240 con 0x55a5e9996690 2024-03-28T01:00:40.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.309+0000 7f8900fbd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5e98f8460 con 0x55a5e999c1f0 2024-03-28T01:00:40.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.309+0000 7f8900fbd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5e999f1a0 con 0x55a5e9997080 2024-03-28T01:00:40.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.309+0000 7f8900fbd740 10 client.0 _ll_register_callbacks cb 0x55a5e98b7c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:40.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.310+0000 7f88fc89b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e999c1f0 0x55a5e99a7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:40.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.310+0000 7f88fc89b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e999c1f0 0x55a5e99a7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:56014/0 (socket says 172.21.15.169:56014) 2024-03-28T01:00:40.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.310+0000 7f88fc89b640 1 -- 172.21.15.169:0/4047450428 learned_addr learned my addr 172.21.15.169:0/4047450428 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:40.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.310+0000 7f88fc89b640 1 -- 172.21.15.169:0/4047450428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e9997080 msgr2=0x55a5e99a0db0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:00:40.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.310+0000 7f88fc89b640 1 --2- 172.21.15.169:0/4047450428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e9997080 0x55a5e99a0db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.310+0000 7f88fc89b640 1 -- 172.21.15.169:0/4047450428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a5e9996690 msgr2=0x55a5e99a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:40.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.310+0000 7f88fc89b640 1 --2- 172.21.15.169:0/4047450428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a5e9996690 0x55a5e99a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.310+0000 7f88fc89b640 1 -- 172.21.15.169:0/4047450428 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a5e96da7d0 con 0x55a5e999c1f0 2024-03-28T01:00:40.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.311+0000 7f88fc89b640 1 --2- 172.21.15.169:0/4047450428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e999c1f0 0x55a5e99a7df0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f88ec0057c0 tx=0x7f88ec014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb8f8b2f4bbaa9e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:40.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.311+0000 7f88d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:00:40.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.311+0000 7f88d77fe640 1 -- 172.21.15.169:0/4047450428 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec010aa0 con 0x55a5e999c1f0 2024-03-28T01:00:40.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.311+0000 7f8900fbd740 1 -- 172.21.15.169:0/4047450428 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f88ec005500 con 0x55a5e999c1f0 2024-03-28T01:00:40.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.311+0000 7f88d77fe640 1 -- 172.21.15.169:0/4047450428 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f88ec010c40 con 0x55a5e999c1f0 2024-03-28T01:00:40.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.311+0000 7f88d77fe640 1 -- 172.21.15.169:0/4047450428 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88ec03a050 con 0x55a5e999c1f0 2024-03-28T01:00:40.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.311+0000 7f8900fbd740 10 client.26153 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:40.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.311+0000 7f8900fbd740 1 -- 172.21.15.169:0/4047450428 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a5e99ab2a0 con 0x55a5e999c1f0 2024-03-28T01:00:40.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.314+0000 7f88d77fe640 1 -- 172.21.15.169:0/4047450428 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88ec005500 con 0x55a5e999c1f0 2024-03-28T01:00:40.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.314+0000 7f88d77fe640 1 -- 172.21.15.169:0/4047450428 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f88ec018430 con 0x55a5e999c1f0 2024-03-28T01:00:40.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.317+0000 7f8900fbd740 10 client.26153 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:40.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.317+0000 7f8900fbd740 10 client.26153 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:00:40.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.318+0000 7f8900fbd740 20 client.26153 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:40.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.318+0000 7f8900fbd740 1 --2- 172.21.15.169:0/4047450428 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55a5e99ac780 0x55a5e99aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:40.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.318+0000 7f8900fbd740 4 client.26153 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:00:40.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.318+0000 7f8900fbd740 1 -- 172.21.15.169:0/4047450428 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a5e9909900 con 0x55a5e99ac780 2024-03-28T01:00:40.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.322+0000 7f88f7fff640 1 --2- 172.21.15.169:0/4047450428 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55a5e99ac780 0x55a5e99aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:40.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.323+0000 7f88fc89b640 1 --2- 172.21.15.169:0/2952286010 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f88e00a2720 0x7f88e00a4be0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00fe50 tx=0x7f88ec00fe80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:40.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.325+0000 7f88f7fff640 1 --2- 172.21.15.169:0/4047450428 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55a5e99ac780 0x55a5e99aeb70 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7a7c22a8eda294a server_cookie=7e50edaf6d074d0b in_seq=0 out_seq=0 2024-03-28T01:00:40.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.325+0000 7f88d77fe640 10 client.26153 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:00:40.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.326+0000 7f88d77fe640 1 -- 172.21.15.169:0/4047450428 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a5e9909900 con 0x55a5e99ac780 2024-03-28T01:00:40.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.326+0000 7f88d77fe640 10 client.26153 handle_command_reply: tid=0 2024-03-28T01:00:40.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.326+0000 7f8900fbd740 1 client.26153 shutdown 2024-03-28T01:00:40.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.327+0000 7f8900fbd740 1 -- 172.21.15.169:0/4047450428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e999c1f0 msgr2=0x55a5e99a7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:40.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.327+0000 7f8900fbd740 1 --2- 172.21.15.169:0/4047450428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e999c1f0 0x55a5e99a7df0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f88ec0057c0 tx=0x7f88ec014000 comp rx=0 tx=0).stop 2024-03-28T01:00:40.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.330+0000 7f8900fbd740 1 -- 172.21.15.169:0/4047450428 shutdown_connections 2024-03-28T01:00:40.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.330+0000 7f8900fbd740 1 --2- 172.21.15.169:0/4047450428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e999c1f0 0x55a5e99a7df0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.330+0000 7f8900fbd740 1 --2- 172.21.15.169:0/4047450428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e9997080 0x55a5e99a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.330+0000 7f8900fbd740 1 --2- 172.21.15.169:0/4047450428 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55a5e99ac780 0x55a5e99aeb70 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.330+0000 7f8900fbd740 1 --2- 172.21.15.169:0/4047450428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a5e9996690 0x55a5e99a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.330+0000 7f8900fbd740 1 -- 172.21.15.169:0/4047450428 >> 172.21.15.169:0/4047450428 conn(0x55a5e9991c00 msgr2=0x55a5e9990e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:40.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.330+0000 7f88d77fe640 0 client.26153 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:00:40.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.331+0000 7f8900fbd740 1 -- 172.21.15.169:0/4047450428 shutdown_connections 2024-03-28T01:00:40.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.331+0000 7f8900fbd740 1 -- 172.21.15.169:0/4047450428 wait complete. 2024-03-28T01:00:40.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.331+0000 7f8900fbd740 20 client.26153 trim_cache size 0 max 16384 2024-03-28T01:00:40.357 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:39 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/4188778190' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:00:40.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.367+0000 7f8900fbd740 1 Processor -- start 2024-03-28T01:00:40.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.369+0000 7f8900fbd740 1 -- start start 2024-03-28T01:00:40.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.369+0000 7f8900fbd740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a5e9996690 0x55a5e99ae790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:40.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.369+0000 7f8900fbd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e9997080 0x55a5e99a7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:40.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.369+0000 7f8900fbd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e999c1f0 0x55a5e99aecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:40.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.369+0000 7f8900fbd740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f81c3a10 con 0x55a5e9996690 2024-03-28T01:00:40.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.369+0000 7f8900fbd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f81c1510 con 0x55a5e9997080 2024-03-28T01:00:40.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.369+0000 7f8900fbd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f81bf5b0 con 0x55a5e999c1f0 2024-03-28T01:00:40.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.371+0000 7f88f7fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a5e9996690 0x55a5e99ae790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:40.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.371+0000 7f88f7fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a5e9996690 0x55a5e99ae790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36538/0 (socket says 172.21.15.169:36538) 2024-03-28T01:00:40.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.371+0000 7f88f7fff640 1 -- 172.21.15.169:0/3066881274 learned_addr learned my addr 172.21.15.169:0/3066881274 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:40.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.371+0000 7f88fc89b640 1 --2- 172.21.15.169:0/3066881274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e999c1f0 0x55a5e99aecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:40.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.372+0000 7f88f77fe640 1 --2- 172.21.15.169:0/3066881274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e9997080 0x55a5e99a7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:40.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.372+0000 7f88fc89b640 1 -- 172.21.15.169:0/3066881274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e9997080 msgr2=0x55a5e99a7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:40.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.372+0000 7f88fc89b640 1 --2- 172.21.15.169:0/3066881274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e9997080 0x55a5e99a7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.372+0000 7f88fc89b640 1 -- 172.21.15.169:0/3066881274 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a5e9996690 msgr2=0x55a5e99ae790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:40.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.372+0000 7f88fc89b640 1 --2- 172.21.15.169:0/3066881274 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a5e9996690 0x55a5e99ae790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.372+0000 7f88fc89b640 1 -- 172.21.15.169:0/3066881274 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5e998ccb0 con 0x55a5e999c1f0 2024-03-28T01:00:40.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.373+0000 7f88fc89b640 1 --2- 172.21.15.169:0/3066881274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e999c1f0 0x55a5e99aecd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f88ec004fd0 tx=0x7f88ec041000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a47955b7a1f26371 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:40.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.373+0000 7f88ddffb640 1 -- 172.21.15.169:0/3066881274 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec01d050 con 0x55a5e999c1f0 2024-03-28T01:00:40.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.373+0000 7f88ddffb640 1 -- 172.21.15.169:0/3066881274 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88ec01e040 con 0x55a5e999c1f0 2024-03-28T01:00:40.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.373+0000 7f88ddffb640 1 -- 172.21.15.169:0/3066881274 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec004030 con 0x55a5e999c1f0 2024-03-28T01:00:40.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.375+0000 7f8900fbd740 1 -- 172.21.15.169:0/3066881274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e999c1f0 msgr2=0x55a5e99aecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:40.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.375+0000 7f8900fbd740 1 --2- 172.21.15.169:0/3066881274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e999c1f0 0x55a5e99aecd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f88ec004fd0 tx=0x7f88ec041000 comp rx=0 tx=0).stop 2024-03-28T01:00:40.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.382+0000 7f8900fbd740 1 -- 172.21.15.169:0/3066881274 shutdown_connections 2024-03-28T01:00:40.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.382+0000 7f8900fbd740 1 --2- 172.21.15.169:0/3066881274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e999c1f0 0x55a5e99aecd0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.382+0000 7f8900fbd740 1 --2- 172.21.15.169:0/3066881274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e9997080 0x55a5e99a7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.382+0000 7f8900fbd740 1 --2- 172.21.15.169:0/3066881274 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a5e9996690 0x55a5e99ae790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.382+0000 7f8900fbd740 1 -- 172.21.15.169:0/3066881274 >> 172.21.15.169:0/3066881274 conn(0x55a5e9991c00 msgr2=0x55a5e99a8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:40.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.395+0000 7f8900fbd740 1 -- 172.21.15.169:0/3066881274 shutdown_connections 2024-03-28T01:00:40.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.402+0000 7f8900fbd740 1 -- 172.21.15.169:0/3066881274 wait complete. 2024-03-28T01:00:40.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.402+0000 7f8900fbd740 1 -- start start 2024-03-28T01:00:40.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.403+0000 7f8900fbd740 1 Processor -- start 2024-03-28T01:00:40.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.403+0000 7f8900fbd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e999c1f0 0x55a5e99a5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:40.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.403+0000 7f8900fbd740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a5e99a5890 0x55a5e99a68a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:40.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.403+0000 7f8900fbd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e99a5d10 0x55a5e99a6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:40.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.403+0000 7f8900fbd740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f81bfbb0 con 0x55a5e99a5890 2024-03-28T01:00:40.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.403+0000 7f8900fbd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f81c0eb0 con 0x55a5e999c1f0 2024-03-28T01:00:40.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.403+0000 7f8900fbd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f81c01f0 con 0x55a5e99a5d10 2024-03-28T01:00:40.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.403+0000 7f8900fbd740 10 client.0 _ll_register_callbacks cb 0x7f88ec010430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:40.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.407+0000 7f88f7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e999c1f0 0x55a5e99a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:40.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.407+0000 7f88f7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e999c1f0 0x55a5e99a5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:56028/0 (socket says 172.21.15.169:56028) 2024-03-28T01:00:40.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.407+0000 7f88f7fff640 1 -- 172.21.15.169:0/4048966012 learned_addr learned my addr 172.21.15.169:0/4048966012 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:40.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.407+0000 7f88fc89b640 1 --2- 172.21.15.169:0/4048966012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e99a5d10 0x55a5e99a6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:40.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.407+0000 7f88f77fe640 1 --2- 172.21.15.169:0/4048966012 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a5e99a5890 0x55a5e99a68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:40.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.411+0000 7f88fc89b640 1 -- 172.21.15.169:0/4048966012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e999c1f0 msgr2=0x55a5e99a5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:40.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.411+0000 7f88fc89b640 1 --2- 172.21.15.169:0/4048966012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e999c1f0 0x55a5e99a5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.411+0000 7f88fc89b640 1 -- 172.21.15.169:0/4048966012 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a5e99a5890 msgr2=0x55a5e99a68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:40.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.411+0000 7f88fc89b640 1 --2- 172.21.15.169:0/4048966012 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a5e99a5890 0x55a5e99a68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.411+0000 7f88fc89b640 1 -- 172.21.15.169:0/4048966012 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a5e99a7650 con 0x55a5e99a5d10 2024-03-28T01:00:40.416 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:39 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/4188778190' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:00:40.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.417+0000 7f88fc89b640 1 --2- 172.21.15.169:0/4048966012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e99a5d10 0x55a5e99a6e20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x55a5e98b8300 tx=0x7f88ec00fd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a34ff0971d55ad8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:40.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.421+0000 7f88d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:00:40.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.421+0000 7f88d77fe640 1 -- 172.21.15.169:0/4048966012 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec0108a0 con 0x55a5e99a5d10 2024-03-28T01:00:40.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.421+0000 7f88d77fe640 1 -- 172.21.15.169:0/4048966012 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f88ec010a40 con 0x55a5e99a5d10 2024-03-28T01:00:40.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.422+0000 7f8900fbd740 1 -- 172.21.15.169:0/4048966012 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f88ec01e040 con 0x55a5e99a5d10 2024-03-28T01:00:40.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.424+0000 7f88d77fe640 1 -- 172.21.15.169:0/4048966012 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88ec010c40 con 0x55a5e99a5d10 2024-03-28T01:00:40.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.429+0000 7f8900fbd740 10 client.26095 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:40.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.429+0000 7f8900fbd740 1 -- 172.21.15.169:0/4048966012 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a5e9a085e0 con 0x55a5e99a5d10 2024-03-28T01:00:40.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.433+0000 7f88d77fe640 1 -- 172.21.15.169:0/4048966012 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88ec01e040 con 0x55a5e99a5d10 2024-03-28T01:00:40.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.434+0000 7f88d77fe640 1 -- 172.21.15.169:0/4048966012 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f88ec03d070 con 0x55a5e99a5d10 2024-03-28T01:00:40.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.434+0000 7f8900fbd740 10 client.26095 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:40.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.434+0000 7f8900fbd740 10 client.26095 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:00:40.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.434+0000 7f8900fbd740 20 client.26095 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:40.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.434+0000 7f8900fbd740 1 --2- 172.21.15.169:0/4048966012 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55a5e9a08e40 0x55a5e9a0b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:40.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.434+0000 7f8900fbd740 4 client.26095 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:00:40.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.434+0000 7f8900fbd740 1 -- 172.21.15.169:0/4048966012 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a5e9903c50 con 0x55a5e9a08e40 2024-03-28T01:00:40.440 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.437+0000 7f88f7fff640 1 --2- 172.21.15.169:0/4048966012 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55a5e9a08e40 0x55a5e9a0b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:40.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.450+0000 7f88d77fe640 10 client.26095 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:00:40.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.470+0000 7f88f7fff640 1 --2- 172.21.15.169:0/4048966012 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55a5e9a08e40 0x55a5e9a0b260 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=65ef92d60f65a4aa server_cookie=bbfd086952753904 in_seq=0 out_seq=0 2024-03-28T01:00:40.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.475+0000 7f88d77fe640 1 -- 172.21.15.169:0/4048966012 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a5e9903c50 con 0x55a5e9a08e40 2024-03-28T01:00:40.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.475+0000 7f88d77fe640 10 client.26095 handle_command_reply: tid=0 2024-03-28T01:00:40.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.475+0000 7f8900fbd740 1 client.26095 shutdown 2024-03-28T01:00:40.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.477+0000 7f8900fbd740 1 -- 172.21.15.169:0/4048966012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e99a5d10 msgr2=0x55a5e99a6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:40.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.477+0000 7f8900fbd740 1 --2- 172.21.15.169:0/4048966012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e99a5d10 0x55a5e99a6e20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x55a5e98b8300 tx=0x7f88ec00fd10 comp rx=0 tx=0).stop 2024-03-28T01:00:40.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.483+0000 7f8900fbd740 1 -- 172.21.15.169:0/4048966012 shutdown_connections 2024-03-28T01:00:40.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.483+0000 7f8900fbd740 1 --2- 172.21.15.169:0/4048966012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5e99a5d10 0x55a5e99a6e20 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.483+0000 7f8900fbd740 1 --2- 172.21.15.169:0/4048966012 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55a5e9a08e40 0x55a5e9a0b260 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.483+0000 7f8900fbd740 1 --2- 172.21.15.169:0/4048966012 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a5e99a5890 0x55a5e99a68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.483+0000 7f8900fbd740 1 --2- 172.21.15.169:0/4048966012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5e999c1f0 0x55a5e99a5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.483+0000 7f8900fbd740 1 -- 172.21.15.169:0/4048966012 >> 172.21.15.169:0/4048966012 conn(0x55a5e9991c00 msgr2=0x55a5e99a0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:40.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.483+0000 7f88d77fe640 0 client.26095 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:00:40.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.483+0000 7f8900fbd740 1 -- 172.21.15.169:0/4048966012 shutdown_connections 2024-03-28T01:00:40.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.485+0000 7f8900fbd740 1 -- 172.21.15.169:0/4048966012 wait complete. 2024-03-28T01:00:40.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.486+0000 7f8900fbd740 20 client.26095 trim_cache size 0 max 16384 2024-03-28T01:00:40.489 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:00:40.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.488+0000 7f88fe2fe640 1 -- 172.21.15.169:0/2952286010 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f88e00a2720 msgr2=0x7f88e00a4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:40.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.488+0000 7f88fe2fe640 1 --2- 172.21.15.169:0/2952286010 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f88e00a2720 0x7f88e00a4be0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00fe50 tx=0x7f88ec00fe80 comp rx=0 tx=0).stop 2024-03-28T01:00:40.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.488+0000 7f88fe2fe640 1 -- 172.21.15.169:0/2952286010 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88f81097b0 msgr2=0x7f88f8117e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:40.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.488+0000 7f88fe2fe640 1 --2- 172.21.15.169:0/2952286010 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88f81097b0 0x7f88f8117e60 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f88f0037040 tx=0x7f88f0035d10 comp rx=0 tx=0).stop 2024-03-28T01:00:40.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.496+0000 7f88fe2fe640 1 -- 172.21.15.169:0/2952286010 shutdown_connections 2024-03-28T01:00:40.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.496+0000 7f88fe2fe640 1 --2- 172.21.15.169:0/2952286010 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f88e00a2720 0x7f88e00a4be0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.496+0000 7f88fe2fe640 1 --2- 172.21.15.169:0/2952286010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88f810ea60 0x7f88f8119390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.496+0000 7f88fe2fe640 1 --2- 172.21.15.169:0/2952286010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88f810c0e0 0x7f88f81183a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.496+0000 7f88fe2fe640 1 --2- 172.21.15.169:0/2952286010 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88f81097b0 0x7f88f8117e60 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:40.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.496+0000 7f88fe2fe640 1 -- 172.21.15.169:0/2952286010 >> 172.21.15.169:0/2952286010 conn(0x7f88f806e500 msgr2=0x7f88f810e250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:40.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.505+0000 7f88fe2fe640 1 -- 172.21.15.169:0/2952286010 shutdown_connections 2024-03-28T01:00:40.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:40.505+0000 7f88fe2fe640 1 -- 172.21.15.169:0/2952286010 wait complete. 2024-03-28T01:00:40.553 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:00:40.554 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T01:00:41.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:40 smithi169 ceph-mon[32274]: pgmap v476: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 794 MiB/s wr, 1.29k op/s 2024-03-28T01:00:41.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:40 smithi174 ceph-mon[36060]: pgmap v476: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 794 MiB/s wr, 1.29k op/s 2024-03-28T01:00:41.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.388+0000 7f9fb02cc640 1 -- 172.21.15.169:0/2905219795 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fa81086a0 msgr2=0x7f9fa810aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:41.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.388+0000 7f9fb02cc640 1 --2- 172.21.15.169:0/2905219795 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fa81086a0 0x7f9fa810aad0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f9f98009900 tx=0x7f9f9802bc80 comp rx=0 tx=0).stop 2024-03-28T01:00:41.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.392+0000 7f9fb02cc640 1 -- 172.21.15.169:0/2905219795 shutdown_connections 2024-03-28T01:00:41.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.392+0000 7f9fb02cc640 1 --2- 172.21.15.169:0/2905219795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fa810d9c0 0x7f9fa810de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.392+0000 7f9fb02cc640 1 --2- 172.21.15.169:0/2905219795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9fa810b010 0x7f9fa810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.392+0000 7f9fb02cc640 1 --2- 172.21.15.169:0/2905219795 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fa81086a0 0x7f9fa810aad0 secure :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f9f98009900 tx=0x7f9f9802bc80 comp rx=0 tx=0).stop 2024-03-28T01:00:41.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.392+0000 7f9fb02cc640 1 -- 172.21.15.169:0/2905219795 >> 172.21.15.169:0/2905219795 conn(0x7f9fa806e500 msgr2=0x7f9fa8070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:41.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.402+0000 7f9fb02cc640 1 -- 172.21.15.169:0/2905219795 shutdown_connections 2024-03-28T01:00:41.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.407+0000 7f9fb02cc640 1 -- 172.21.15.169:0/2905219795 wait complete. 2024-03-28T01:00:41.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.407+0000 7f9fb02cc640 1 Processor -- start 2024-03-28T01:00:41.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.408+0000 7f9fb02cc640 1 -- start start 2024-03-28T01:00:41.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.408+0000 7f9fb02cc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fa810b010 0x7f9fa8115060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:41.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.408+0000 7f9fb02cc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9fa810d9c0 0x7f9fa81155a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:41.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.408+0000 7f9fb02cc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fa81b4840 0x7f9fa81b6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:41.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.408+0000 7f9fb02cc640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa811c160 con 0x7f9fa810b010 2024-03-28T01:00:41.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.408+0000 7f9fb02cc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa811bfe0 con 0x7f9fa81b4840 2024-03-28T01:00:41.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.408+0000 7f9fb02cc640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa811c2e0 con 0x7f9fa810d9c0 2024-03-28T01:00:41.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.411+0000 7f9fae869640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fa81b4840 0x7f9fa81b6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:41.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.411+0000 7f9fae869640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fa81b4840 0x7f9fa81b6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:56042/0 (socket says 172.21.15.169:56042) 2024-03-28T01:00:41.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.411+0000 7f9fae869640 1 -- 172.21.15.169:0/1534874101 learned_addr learned my addr 172.21.15.169:0/1534874101 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:41.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.413+0000 7f9fae869640 1 -- 172.21.15.169:0/1534874101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9fa810d9c0 msgr2=0x7f9fa81155a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:00:41.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.413+0000 7f9fae869640 1 --2- 172.21.15.169:0/1534874101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9fa810d9c0 0x7f9fa81155a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.413+0000 7f9fae869640 1 -- 172.21.15.169:0/1534874101 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fa810b010 msgr2=0x7f9fa8115060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:41.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.413+0000 7f9fae869640 1 --2- 172.21.15.169:0/1534874101 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fa810b010 0x7f9fa8115060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.413+0000 7f9fae869640 1 -- 172.21.15.169:0/1534874101 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa81b7220 con 0x7f9fa81b4840 2024-03-28T01:00:41.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:40 smithi177 ceph-mon[36206]: pgmap v476: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 794 MiB/s wr, 1.29k op/s 2024-03-28T01:00:41.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.414+0000 7f9fae869640 1 --2- 172.21.15.169:0/1534874101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fa81b4840 0x7f9fa81b6c30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9fa400e970 tx=0x7f9fa400ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:41.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.417+0000 7f9f9f7fe640 1 -- 172.21.15.169:0/1534874101 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa400ccc0 con 0x7f9fa81b4840 2024-03-28T01:00:41.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.417+0000 7f9f9f7fe640 1 -- 172.21.15.169:0/1534874101 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9fa4005500 con 0x7f9fa81b4840 2024-03-28T01:00:41.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.417+0000 7f9fb02cc640 1 -- 172.21.15.169:0/1534874101 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa81b7510 con 0x7f9fa81b4840 2024-03-28T01:00:41.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.417+0000 7f9fb02cc640 1 -- 172.21.15.169:0/1534874101 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa81b7a60 con 0x7f9fa81b4840 2024-03-28T01:00:41.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.418+0000 7f9f9f7fe640 1 -- 172.21.15.169:0/1534874101 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4010690 con 0x7f9fa81b4840 2024-03-28T01:00:41.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.422+0000 7f9fb2f8a740 1 Processor -- start 2024-03-28T01:00:41.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.425+0000 7f9f9f7fe640 1 -- 172.21.15.169:0/1534874101 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9fa4004e80 con 0x7f9fa81b4840 2024-03-28T01:00:41.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.427+0000 7f9f9f7fe640 1 --2- 172.21.15.169:0/1534874101 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9f940a2a00 0x7f9f940a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:41.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.428+0000 7f9fb2f8a740 1 -- start start 2024-03-28T01:00:41.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.428+0000 7f9fb2f8a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adef690 0x55de2adefab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:41.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.428+0000 7f9fb2f8a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de2adf0080 0x55de2adf4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:41.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.428+0000 7f9fb2f8a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adf51f0 0x55de2adf7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:41.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.428+0000 7f9fb2f8a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2adb2610 con 0x55de2adf51f0 2024-03-28T01:00:41.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.428+0000 7f9fb2f8a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2ad553c0 con 0x55de2adf0080 2024-03-28T01:00:41.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.428+0000 7f9fb2f8a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2acfdd60 con 0x55de2adef690 2024-03-28T01:00:41.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.428+0000 7f9f9f7fe640 1 -- 172.21.15.169:0/1534874101 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f9fa40cf220 con 0x7f9fa81b4840 2024-03-28T01:00:41.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.429+0000 7f9fae068640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adef690 0x55de2adefab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:41.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.429+0000 7f9fae068640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adef690 0x55de2adefab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:33110/0 (socket says 172.21.15.169:33110) 2024-03-28T01:00:41.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.429+0000 7f9fae068640 1 -- 172.21.15.169:0/1298084560 learned_addr learned my addr 172.21.15.169:0/1298084560 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:41.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.429+0000 7f9fae869640 1 --2- 172.21.15.169:0/1298084560 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adf51f0 0x55de2adf7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:41.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.431+0000 7f9fae068640 1 -- 172.21.15.169:0/1298084560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de2adf0080 msgr2=0x55de2adf4b80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:00:41.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.431+0000 7f9fae068640 1 --2- 172.21.15.169:0/1298084560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de2adf0080 0x55de2adf4b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.431+0000 7f9fae068640 1 -- 172.21.15.169:0/1298084560 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adf51f0 msgr2=0x55de2adf7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:41.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.431+0000 7f9fae068640 1 --2- 172.21.15.169:0/1298084560 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adf51f0 0x55de2adf7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.431+0000 7f9fae068640 1 -- 172.21.15.169:0/1298084560 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de2aa693f0 con 0x55de2adef690 2024-03-28T01:00:41.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.432+0000 7f9fae068640 1 --2- 172.21.15.169:0/1298084560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adef690 0x55de2adefab0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9f98004b90 tx=0x7f9f98009510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a05d569e1e747c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:41.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.433+0000 7f9fad867640 1 --2- 172.21.15.169:0/1534874101 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9f940a2a00 0x7f9f940a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:41.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.435+0000 7f9fad867640 1 --2- 172.21.15.169:0/1534874101 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9f940a2a00 0x7f9f940a4ec0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0005fd0 tx=0x7f9fa0005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:41.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.446+0000 7f9f9cbfe640 1 -- 172.21.15.169:0/1298084560 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f98039040 con 0x55de2adef690 2024-03-28T01:00:41.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.446+0000 7f9f9cbfe640 1 -- 172.21.15.169:0/1298084560 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f980048c0 con 0x55de2adef690 2024-03-28T01:00:41.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.447+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/1298084560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adef690 msgr2=0x55de2adefab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:41.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.447+0000 7f9fb2f8a740 1 --2- 172.21.15.169:0/1298084560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adef690 0x55de2adefab0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9f98004b90 tx=0x7f9f98009510 comp rx=0 tx=0).stop 2024-03-28T01:00:41.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.449+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/1298084560 shutdown_connections 2024-03-28T01:00:41.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.449+0000 7f9fb2f8a740 1 --2- 172.21.15.169:0/1298084560 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adf51f0 0x55de2adf7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.449+0000 7f9fb2f8a740 1 --2- 172.21.15.169:0/1298084560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de2adf0080 0x55de2adf4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.449+0000 7f9fb2f8a740 1 --2- 172.21.15.169:0/1298084560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adef690 0x55de2adefab0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.449+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/1298084560 >> 172.21.15.169:0/1298084560 conn(0x55de2adeac00 msgr2=0x55de2adeb000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:41.455 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.452+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/1298084560 shutdown_connections 2024-03-28T01:00:41.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.453+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/1298084560 wait complete. 2024-03-28T01:00:41.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.453+0000 7f9fb2f8a740 1 -- start start 2024-03-28T01:00:41.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.453+0000 7f9fb2f8a740 1 Processor -- start 2024-03-28T01:00:41.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.454+0000 7f9fb2f8a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adef690 0x55de2adf9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:41.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.454+0000 7f9fb2f8a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de2adf0080 0x55de2adf9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:41.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.454+0000 7f9fb2f8a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adf51f0 0x55de2ae00fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:41.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.454+0000 7f9fb2f8a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2ad11240 con 0x55de2adf51f0 2024-03-28T01:00:41.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.454+0000 7f9fb2f8a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2ad51460 con 0x55de2adf0080 2024-03-28T01:00:41.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.454+0000 7f9fb2f8a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2adf8190 con 0x55de2adef690 2024-03-28T01:00:41.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.454+0000 7f9fb2f8a740 10 client.0 _ll_register_callbacks cb 0x55de2ad10c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:41.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.455+0000 7f9fae869640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adf51f0 0x55de2ae00fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:41.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.455+0000 7f9fae869640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adf51f0 0x55de2ae00fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36602/0 (socket says 172.21.15.169:36602) 2024-03-28T01:00:41.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.455+0000 7f9fae869640 1 -- 172.21.15.169:0/944017971 learned_addr learned my addr 172.21.15.169:0/944017971 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:41.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.455+0000 7f9fad867640 1 --2- 172.21.15.169:0/944017971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de2adf0080 0x55de2adf9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:41.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.455+0000 7f9fae068640 1 --2- 172.21.15.169:0/944017971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adef690 0x55de2adf9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:41.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.456+0000 7f9fae869640 1 -- 172.21.15.169:0/944017971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adef690 msgr2=0x55de2adf9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:41.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.456+0000 7f9fae869640 1 --2- 172.21.15.169:0/944017971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adef690 0x55de2adf9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.456+0000 7f9fae869640 1 -- 172.21.15.169:0/944017971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de2adf0080 msgr2=0x55de2adf9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:41.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.456+0000 7f9fae869640 1 --2- 172.21.15.169:0/944017971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de2adf0080 0x55de2adf9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.456+0000 7f9fae869640 1 -- 172.21.15.169:0/944017971 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55de2ab337d0 con 0x55de2adf51f0 2024-03-28T01:00:41.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.456+0000 7f9fae869640 1 --2- 172.21.15.169:0/944017971 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adf51f0 0x55de2ae00fb0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f9fa409b750 tx=0x7f9fa409b250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c05729e933a7ae22 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:41.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.456+0000 7f9f91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:00:41.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.456+0000 7f9f91ffb640 1 -- 172.21.15.169:0/944017971 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa40b9060 con 0x55de2adf51f0 2024-03-28T01:00:41.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.456+0000 7f9f91ffb640 1 -- 172.21.15.169:0/944017971 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f9fa4005170 con 0x55de2adf51f0 2024-03-28T01:00:41.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.457+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/944017971 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9f980048c0 con 0x55de2adf51f0 2024-03-28T01:00:41.461 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.457+0000 7f9f91ffb640 1 -- 172.21.15.169:0/944017971 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9fa40b3030 con 0x55de2adf51f0 2024-03-28T01:00:41.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.461+0000 7f9fb2f8a740 10 client.16299 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:41.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.461+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/944017971 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55de2ae043f0 con 0x55de2adf51f0 2024-03-28T01:00:41.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.468+0000 7f9f91ffb640 1 -- 172.21.15.169:0/944017971 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f980048c0 con 0x55de2adf51f0 2024-03-28T01:00:41.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.469+0000 7f9f91ffb640 1 -- 172.21.15.169:0/944017971 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9fa409ba20 con 0x55de2adf51f0 2024-03-28T01:00:41.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.469+0000 7f9fb2f8a740 10 client.16299 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:41.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.469+0000 7f9fb2f8a740 10 client.16299 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:00:41.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.470+0000 7f9fb2f8a740 20 client.16299 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:41.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.470+0000 7f9fb2f8a740 1 --2- 172.21.15.169:0/944017971 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55de2ae06020 0x55de2ae08440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:41.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.470+0000 7f9fb2f8a740 4 client.16299 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:00:41.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.470+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/944017971 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55de2ad62900 con 0x55de2ae06020 2024-03-28T01:00:41.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.474+0000 7f9fae068640 1 --2- 172.21.15.169:0/944017971 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55de2ae06020 0x55de2ae08440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:41.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.487+0000 7f9fae068640 1 --2- 172.21.15.169:0/944017971 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55de2ae06020 0x55de2ae08440 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fcbf30716511eab3 server_cookie=4965b9424535a4f6 in_seq=0 out_seq=0 2024-03-28T01:00:41.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.488+0000 7f9f91ffb640 10 client.16299 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:00:41.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.491+0000 7f9f91ffb640 1 -- 172.21.15.169:0/944017971 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55de2ad62900 con 0x55de2ae06020 2024-03-28T01:00:41.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.491+0000 7f9f91ffb640 10 client.16299 handle_command_reply: tid=0 2024-03-28T01:00:41.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.491+0000 7f9fb2f8a740 1 client.16299 shutdown 2024-03-28T01:00:41.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.498+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/944017971 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adf51f0 msgr2=0x55de2ae00fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:41.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.498+0000 7f9fb2f8a740 1 --2- 172.21.15.169:0/944017971 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adf51f0 0x55de2ae00fb0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f9fa409b750 tx=0x7f9fa409b250 comp rx=0 tx=0).stop 2024-03-28T01:00:41.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.498+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/944017971 shutdown_connections 2024-03-28T01:00:41.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.498+0000 7f9fb2f8a740 1 --2- 172.21.15.169:0/944017971 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55de2ae06020 0x55de2ae08440 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.498+0000 7f9fb2f8a740 1 --2- 172.21.15.169:0/944017971 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adf51f0 0x55de2ae00fb0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.498+0000 7f9fb2f8a740 1 --2- 172.21.15.169:0/944017971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de2adf0080 0x55de2adf9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.498+0000 7f9fb2f8a740 1 --2- 172.21.15.169:0/944017971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adef690 0x55de2adf9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.498+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/944017971 >> 172.21.15.169:0/944017971 conn(0x55de2adeac00 msgr2=0x55de2ade9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:41.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.498+0000 7f9f91ffb640 0 client.16299 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:00:41.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.501+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/944017971 shutdown_connections 2024-03-28T01:00:41.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.501+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/944017971 wait complete. 2024-03-28T01:00:41.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.502+0000 7f9fb2f8a740 20 client.16299 trim_cache size 0 max 16384 2024-03-28T01:00:41.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.536+0000 7f9fb2f8a740 1 Processor -- start 2024-03-28T01:00:41.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.540+0000 7f9fb2f8a740 1 -- start start 2024-03-28T01:00:41.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.540+0000 7f9fb2f8a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adef690 0x55de2adf8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:41.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.540+0000 7f9fb2f8a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adf0080 0x55de2ae00f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:41.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.540+0000 7f9fb2f8a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de2adf51f0 0x55de2adf9240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:41.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.540+0000 7f9fb2f8a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c011f40 con 0x55de2adf0080 2024-03-28T01:00:41.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.540+0000 7f9fb2f8a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c00fa40 con 0x55de2adf51f0 2024-03-28T01:00:41.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.540+0000 7f9fb2f8a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c00db90 con 0x55de2adef690 2024-03-28T01:00:41.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.541+0000 7f9fad867640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adf0080 0x55de2ae00f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:41.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.541+0000 7f9fad867640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adf0080 0x55de2ae00f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36616/0 (socket says 172.21.15.169:36616) 2024-03-28T01:00:41.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.541+0000 7f9fad867640 1 -- 172.21.15.169:0/3916361617 learned_addr learned my addr 172.21.15.169:0/3916361617 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:41.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.544+0000 7f9fae869640 1 --2- 172.21.15.169:0/3916361617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de2adf51f0 0x55de2adf9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:41.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.544+0000 7f9fae068640 1 --2- 172.21.15.169:0/3916361617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adef690 0x55de2adf8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:41.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.547+0000 7f9fae869640 1 -- 172.21.15.169:0/3916361617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adef690 msgr2=0x55de2adf8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:41.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.547+0000 7f9fae869640 1 --2- 172.21.15.169:0/3916361617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adef690 0x55de2adf8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.547+0000 7f9fae869640 1 -- 172.21.15.169:0/3916361617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adf0080 msgr2=0x55de2ae00f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:41.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.547+0000 7f9fae869640 1 --2- 172.21.15.169:0/3916361617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adf0080 0x55de2ae00f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.547+0000 7f9fae869640 1 -- 172.21.15.169:0/3916361617 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de2ade6670 con 0x55de2adf51f0 2024-03-28T01:00:41.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.552+0000 7f9fae869640 1 --2- 172.21.15.169:0/3916361617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de2adf51f0 0x55de2adf9240 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4099a70 tx=0x7f9fa409c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab7ecbedf50479e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:41.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.554+0000 7f9f93fff640 1 -- 172.21.15.169:0/3916361617 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4097a60 con 0x55de2adf51f0 2024-03-28T01:00:41.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.554+0000 7f9f93fff640 1 -- 172.21.15.169:0/3916361617 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9fa409f910 con 0x55de2adf51f0 2024-03-28T01:00:41.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.555+0000 7f9f93fff640 1 -- 172.21.15.169:0/3916361617 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa40046c0 con 0x55de2adf51f0 2024-03-28T01:00:41.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.556+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/3916361617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de2adf51f0 msgr2=0x55de2adf9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:41.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.556+0000 7f9fb2f8a740 1 --2- 172.21.15.169:0/3916361617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de2adf51f0 0x55de2adf9240 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4099a70 tx=0x7f9fa409c000 comp rx=0 tx=0).stop 2024-03-28T01:00:41.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.557+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/3916361617 shutdown_connections 2024-03-28T01:00:41.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.557+0000 7f9fb2f8a740 1 --2- 172.21.15.169:0/3916361617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de2adf51f0 0x55de2adf9240 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.557+0000 7f9fb2f8a740 1 --2- 172.21.15.169:0/3916361617 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adf0080 0x55de2ae00f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.557+0000 7f9fb2f8a740 1 --2- 172.21.15.169:0/3916361617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adef690 0x55de2adf8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.557+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/3916361617 >> 172.21.15.169:0/3916361617 conn(0x55de2adeac00 msgr2=0x55de2adeafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:41.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.559+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/3916361617 shutdown_connections 2024-03-28T01:00:41.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.560+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/3916361617 wait complete. 2024-03-28T01:00:41.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.560+0000 7f9fb2f8a740 1 -- start start 2024-03-28T01:00:41.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.561+0000 7f9fb2f8a740 1 Processor -- start 2024-03-28T01:00:41.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.561+0000 7f9fb2f8a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adef690 0x55de2adfe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:41.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.562+0000 7f9fb2f8a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de2adf0080 0x55de2adfece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:41.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.562+0000 7f9fb2f8a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adf51f0 0x55de2adffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:41.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.562+0000 7f9fb2f8a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c00e100 con 0x55de2adef690 2024-03-28T01:00:41.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.562+0000 7f9fb2f8a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c00f3e0 con 0x55de2adf0080 2024-03-28T01:00:41.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.562+0000 7f9fb2f8a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c00e720 con 0x55de2adf51f0 2024-03-28T01:00:41.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.562+0000 7f9fb2f8a740 10 client.0 _ll_register_callbacks cb 0x7f9fa409b660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:41.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.563+0000 7f9fae869640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adf51f0 0x55de2adffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:41.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.563+0000 7f9fae869640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adf51f0 0x55de2adffcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:33130/0 (socket says 172.21.15.169:33130) 2024-03-28T01:00:41.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.563+0000 7f9fae869640 1 -- 172.21.15.169:0/3007988390 learned_addr learned my addr 172.21.15.169:0/3007988390 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:41.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.564+0000 7f9fad867640 1 --2- 172.21.15.169:0/3007988390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de2adf0080 0x55de2adfece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:41.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.565+0000 7f9fae068640 1 --2- 172.21.15.169:0/3007988390 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adef690 0x55de2adfe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:41.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.566+0000 7f9fae869640 1 -- 172.21.15.169:0/3007988390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de2adf0080 msgr2=0x55de2adfece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:41.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.566+0000 7f9fae869640 1 --2- 172.21.15.169:0/3007988390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de2adf0080 0x55de2adfece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.566+0000 7f9fae869640 1 -- 172.21.15.169:0/3007988390 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adef690 msgr2=0x55de2adfe7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:41.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.566+0000 7f9fae869640 1 --2- 172.21.15.169:0/3007988390 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adef690 0x55de2adfe7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.566+0000 7f9fae869640 1 -- 172.21.15.169:0/3007988390 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55de2ae002e0 con 0x55de2adf51f0 2024-03-28T01:00:41.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.575+0000 7f9fae869640 1 --2- 172.21.15.169:0/3007988390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adf51f0 0x55de2adffcf0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9fa409b4c0 tx=0x7f9fa40048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9f454b807fe153e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:41.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.575+0000 7f9f91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:00:41.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.582+0000 7f9f91ffb640 1 -- 172.21.15.169:0/3007988390 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa40b9020 con 0x55de2adf51f0 2024-03-28T01:00:41.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.582+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/3007988390 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9fa409f910 con 0x55de2adf51f0 2024-03-28T01:00:41.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.582+0000 7f9f91ffb640 1 -- 172.21.15.169:0/3007988390 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f9fa40af040 con 0x55de2adf51f0 2024-03-28T01:00:41.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.586+0000 7f9f91ffb640 1 -- 172.21.15.169:0/3007988390 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9fa4097a60 con 0x55de2adf51f0 2024-03-28T01:00:41.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.587+0000 7f9fb2f8a740 10 client.26107 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:41.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.587+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/3007988390 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55de2ae61a00 con 0x55de2adf51f0 2024-03-28T01:00:41.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.589+0000 7f9f91ffb640 1 -- 172.21.15.169:0/3007988390 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9fa409f910 con 0x55de2adf51f0 2024-03-28T01:00:41.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.589+0000 7f9f91ffb640 1 -- 172.21.15.169:0/3007988390 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9fa40afcb0 con 0x55de2adf51f0 2024-03-28T01:00:41.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.589+0000 7f9fb2f8a740 10 client.26107 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:41.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.589+0000 7f9fb2f8a740 10 client.26107 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:00:41.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.590+0000 7f9fb2f8a740 20 client.26107 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:41.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.590+0000 7f9fb2f8a740 1 --2- 172.21.15.169:0/3007988390 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55de2ae62110 0x55de2ae64530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:41.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.590+0000 7f9fb2f8a740 4 client.26107 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:00:41.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.590+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/3007988390 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55de2ad5cc50 con 0x55de2ae62110 2024-03-28T01:00:41.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.591+0000 7f9fae068640 1 --2- 172.21.15.169:0/3007988390 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55de2ae62110 0x55de2ae64530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:41.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.593+0000 7f9fae068640 1 --2- 172.21.15.169:0/3007988390 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55de2ae62110 0x55de2ae64530 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=76ab1bdadcb8e046 server_cookie=76c5581fd1259b53 in_seq=0 out_seq=0 2024-03-28T01:00:41.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.593+0000 7f9f91ffb640 10 client.26107 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:00:41.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.595+0000 7f9f91ffb640 1 -- 172.21.15.169:0/3007988390 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55de2ad5cc50 con 0x55de2ae62110 2024-03-28T01:00:41.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.595+0000 7f9f91ffb640 10 client.26107 handle_command_reply: tid=0 2024-03-28T01:00:41.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.596+0000 7f9fb2f8a740 1 client.26107 shutdown 2024-03-28T01:00:41.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.600+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/3007988390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adf51f0 msgr2=0x55de2adffcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:41.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.600+0000 7f9fb2f8a740 1 --2- 172.21.15.169:0/3007988390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adf51f0 0x55de2adffcf0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9fa409b4c0 tx=0x7f9fa40048f0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.601+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/3007988390 shutdown_connections 2024-03-28T01:00:41.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.601+0000 7f9fb2f8a740 1 --2- 172.21.15.169:0/3007988390 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55de2ae62110 0x55de2ae64530 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.601+0000 7f9fb2f8a740 1 --2- 172.21.15.169:0/3007988390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55de2adf51f0 0x55de2adffcf0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.601+0000 7f9fb2f8a740 1 --2- 172.21.15.169:0/3007988390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de2adf0080 0x55de2adfece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.601+0000 7f9fb2f8a740 1 --2- 172.21.15.169:0/3007988390 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55de2adef690 0x55de2adfe7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.601+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/3007988390 >> 172.21.15.169:0/3007988390 conn(0x55de2adeac00 msgr2=0x55de2adf8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:41.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.601+0000 7f9f91ffb640 0 client.26107 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:00:41.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.601+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/3007988390 shutdown_connections 2024-03-28T01:00:41.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.601+0000 7f9fb2f8a740 1 -- 172.21.15.169:0/3007988390 wait complete. 2024-03-28T01:00:41.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.605+0000 7f9fb2f8a740 20 client.26107 trim_cache size 0 max 16384 2024-03-28T01:00:41.615 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:00:41.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.615+0000 7f9fb02cc640 1 -- 172.21.15.169:0/1534874101 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9f940a2a00 msgr2=0x7f9f940a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:41.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.615+0000 7f9fb02cc640 1 --2- 172.21.15.169:0/1534874101 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9f940a2a00 0x7f9f940a4ec0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0005fd0 tx=0x7f9fa0005ee0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.615+0000 7f9fb02cc640 1 -- 172.21.15.169:0/1534874101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fa81b4840 msgr2=0x7f9fa81b6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:41.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.615+0000 7f9fb02cc640 1 --2- 172.21.15.169:0/1534874101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fa81b4840 0x7f9fa81b6c30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9fa400e970 tx=0x7f9fa400ee30 comp rx=0 tx=0).stop 2024-03-28T01:00:41.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.622+0000 7f9fb02cc640 1 -- 172.21.15.169:0/1534874101 shutdown_connections 2024-03-28T01:00:41.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.622+0000 7f9fb02cc640 1 --2- 172.21.15.169:0/1534874101 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9f940a2a00 0x7f9f940a4ec0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.622+0000 7f9fb02cc640 1 --2- 172.21.15.169:0/1534874101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fa81b4840 0x7f9fa81b6c30 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.622+0000 7f9fb02cc640 1 --2- 172.21.15.169:0/1534874101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9fa810d9c0 0x7f9fa81155a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.622+0000 7f9fb02cc640 1 --2- 172.21.15.169:0/1534874101 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fa810b010 0x7f9fa8115060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:41.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.622+0000 7f9fb02cc640 1 -- 172.21.15.169:0/1534874101 >> 172.21.15.169:0/1534874101 conn(0x7f9fa806e500 msgr2=0x7f9fa810b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:41.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.628+0000 7f9fb02cc640 1 -- 172.21.15.169:0/1534874101 shutdown_connections 2024-03-28T01:00:41.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:41.632+0000 7f9fb02cc640 1 -- 172.21.15.169:0/1534874101 wait complete. 2024-03-28T01:00:41.678 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:00:41.678 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T01:00:42.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.146+0000 7f661f40b640 1 -- 172.21.15.169:0/1137271478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f661810d9c0 msgr2=0x7f661810de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:42.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.146+0000 7f661f40b640 1 --2- 172.21.15.169:0/1137271478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f661810d9c0 0x7f661810de40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6614009900 tx=0x7f661402bc60 comp rx=0 tx=0).stop 2024-03-28T01:00:42.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.157+0000 7f661f40b640 1 -- 172.21.15.169:0/1137271478 shutdown_connections 2024-03-28T01:00:42.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.157+0000 7f661f40b640 1 --2- 172.21.15.169:0/1137271478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f661810d9c0 0x7f661810de40 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.157+0000 7f661f40b640 1 --2- 172.21.15.169:0/1137271478 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f661810b010 0x7f661810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.157+0000 7f661f40b640 1 --2- 172.21.15.169:0/1137271478 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66181086a0 0x7f661810aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.157+0000 7f661f40b640 1 -- 172.21.15.169:0/1137271478 >> 172.21.15.169:0/1137271478 conn(0x7f661806e500 msgr2=0x7f6618070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:42.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.163+0000 7f661f40b640 1 -- 172.21.15.169:0/1137271478 shutdown_connections 2024-03-28T01:00:42.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.165+0000 7f661f40b640 1 -- 172.21.15.169:0/1137271478 wait complete. 2024-03-28T01:00:42.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.166+0000 7f661f40b640 1 Processor -- start 2024-03-28T01:00:42.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.166+0000 7f661f40b640 1 -- start start 2024-03-28T01:00:42.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.166+0000 7f661f40b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66181086a0 0x7f66181198f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:42.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.166+0000 7f661f40b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f661810b010 0x7f6618119e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:42.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.166+0000 7f661f40b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f661810d9c0 0x7f6618117f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:42.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.166+0000 7f661f40b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6618110800 con 0x7f661810b010 2024-03-28T01:00:42.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.166+0000 7f661f40b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6618110680 con 0x7f66181086a0 2024-03-28T01:00:42.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.166+0000 7f661f40b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6618110980 con 0x7f661810d9c0 2024-03-28T01:00:42.174 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.167+0000 7f661c9a6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f661810b010 0x7f6618119e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:42.174 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.167+0000 7f661c9a6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f661810b010 0x7f6618119e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36644/0 (socket says 172.21.15.169:36644) 2024-03-28T01:00:42.174 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.167+0000 7f661c9a6640 1 -- 172.21.15.169:0/51066547 learned_addr learned my addr 172.21.15.169:0/51066547 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:42.174 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.167+0000 7f661c9a6640 1 -- 172.21.15.169:0/51066547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f661810d9c0 msgr2=0x7f6618117f30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:00:42.174 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.167+0000 7f661c9a6640 1 --2- 172.21.15.169:0/51066547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f661810d9c0 0x7f6618117f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.174 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.167+0000 7f661c9a6640 1 -- 172.21.15.169:0/51066547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66181086a0 msgr2=0x7f66181198f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:00:42.174 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.167+0000 7f661c9a6640 1 --2- 172.21.15.169:0/51066547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66181086a0 0x7f66181198f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.174 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.167+0000 7f661c9a6640 1 -- 172.21.15.169:0/51066547 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66181186b0 con 0x7f661810b010 2024-03-28T01:00:42.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.168+0000 7f661c9a6640 1 --2- 172.21.15.169:0/51066547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f661810b010 0x7f6618119e30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f660800d950 tx=0x7f660800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:42.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.171+0000 7f660e7fc640 1 -- 172.21.15.169:0/51066547 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6608017070 con 0x7f661810b010 2024-03-28T01:00:42.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.171+0000 7f661f40b640 1 -- 172.21.15.169:0/51066547 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66181189a0 con 0x7f661810b010 2024-03-28T01:00:42.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.171+0000 7f661f40b640 1 -- 172.21.15.169:0/51066547 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66181b7490 con 0x7f661810b010 2024-03-28T01:00:42.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.173+0000 7f660e7fc640 1 -- 172.21.15.169:0/51066547 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6608009bb0 con 0x7f661810b010 2024-03-28T01:00:42.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.173+0000 7f660e7fc640 1 -- 172.21.15.169:0/51066547 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660800f600 con 0x7f661810b010 2024-03-28T01:00:42.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.185+0000 7f6621ff9740 1 Processor -- start 2024-03-28T01:00:42.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.186+0000 7f6621ff9740 1 -- start start 2024-03-28T01:00:42.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.186+0000 7f6621ff9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee48690 0x55e6dee48ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:42.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.186+0000 7f6621ff9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee49080 0x55e6dee4db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:42.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.186+0000 7f6621ff9740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e6dee4e1f0 0x55e6dee50680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:42.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.186+0000 7f6621ff9740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6dee0b610 con 0x55e6dee4e1f0 2024-03-28T01:00:42.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.186+0000 7f6621ff9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6dedae3c0 con 0x55e6dee49080 2024-03-28T01:00:42.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.186+0000 7f6621ff9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6ded56d60 con 0x55e6dee48690 2024-03-28T01:00:42.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.185+0000 7f660e7fc640 1 -- 172.21.15.169:0/51066547 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f660800f7a0 con 0x7f661810b010 2024-03-28T01:00:42.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.187+0000 7f661d9a8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee48690 0x55e6dee48ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:42.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.187+0000 7f661d9a8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee48690 0x55e6dee48ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:33138/0 (socket says 172.21.15.169:33138) 2024-03-28T01:00:42.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.187+0000 7f661d9a8640 1 -- 172.21.15.169:0/344663256 learned_addr learned my addr 172.21.15.169:0/344663256 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:42.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.187+0000 7f661d1a7640 1 --2- 172.21.15.169:0/344663256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee49080 0x55e6dee4db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:42.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.188+0000 7f661d9a8640 1 -- 172.21.15.169:0/344663256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee49080 msgr2=0x55e6dee4db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:42.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.188+0000 7f661d9a8640 1 --2- 172.21.15.169:0/344663256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee49080 0x55e6dee4db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.188+0000 7f661d9a8640 1 -- 172.21.15.169:0/344663256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e6dee4e1f0 msgr2=0x55e6dee50680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:42.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.188+0000 7f661d9a8640 1 --2- 172.21.15.169:0/344663256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e6dee4e1f0 0x55e6dee50680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.188+0000 7f661d9a8640 1 -- 172.21.15.169:0/344663256 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e6deac23f0 con 0x55e6dee48690 2024-03-28T01:00:42.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.188+0000 7f660e7fc640 1 --2- 172.21.15.169:0/51066547 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f66040a2a00 0x7f66040a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:42.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.188+0000 7f661d9a8640 1 --2- 172.21.15.169:0/344663256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee48690 0x55e6dee48ab0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f661402c270 tx=0x7f66140094d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d80bb5f6f1836c7f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:42.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.188+0000 7f660e7fc640 1 -- 172.21.15.169:0/51066547 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f66080ce8c0 con 0x7f661810b010 2024-03-28T01:00:42.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.190+0000 7f66037fe640 1 -- 172.21.15.169:0/344663256 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6614043020 con 0x55e6dee48690 2024-03-28T01:00:42.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.190+0000 7f66037fe640 1 -- 172.21.15.169:0/344663256 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6614004040 con 0x55e6dee48690 2024-03-28T01:00:42.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.190+0000 7f66037fe640 1 -- 172.21.15.169:0/344663256 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6614005530 con 0x55e6dee48690 2024-03-28T01:00:42.196 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.191+0000 7f6621ff9740 1 -- 172.21.15.169:0/344663256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee48690 msgr2=0x55e6dee48ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:42.196 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.191+0000 7f6621ff9740 1 --2- 172.21.15.169:0/344663256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee48690 0x55e6dee48ab0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f661402c270 tx=0x7f66140094d0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.194+0000 7f661d1a7640 1 --2- 172.21.15.169:0/51066547 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f66040a2a00 0x7f66040a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:42.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.195+0000 7f6621ff9740 1 -- 172.21.15.169:0/344663256 shutdown_connections 2024-03-28T01:00:42.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.195+0000 7f6621ff9740 1 --2- 172.21.15.169:0/344663256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e6dee4e1f0 0x55e6dee50680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.195+0000 7f6621ff9740 1 --2- 172.21.15.169:0/344663256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee49080 0x55e6dee4db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.195+0000 7f6621ff9740 1 --2- 172.21.15.169:0/344663256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee48690 0x55e6dee48ab0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.195+0000 7f6621ff9740 1 -- 172.21.15.169:0/344663256 >> 172.21.15.169:0/344663256 conn(0x55e6dee43c00 msgr2=0x55e6dee44000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:42.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.195+0000 7f6621ff9740 1 -- 172.21.15.169:0/344663256 shutdown_connections 2024-03-28T01:00:42.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.195+0000 7f6621ff9740 1 -- 172.21.15.169:0/344663256 wait complete. 2024-03-28T01:00:42.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.196+0000 7f6621ff9740 1 -- start start 2024-03-28T01:00:42.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.196+0000 7f6621ff9740 1 Processor -- start 2024-03-28T01:00:42.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.196+0000 7f661d1a7640 1 --2- 172.21.15.169:0/51066547 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f66040a2a00 0x7f66040a4ec0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x55e6dee44b70 tx=0x7f6610009410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:42.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.196+0000 7f6621ff9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee48690 0x55e6dee52ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:42.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.196+0000 7f6621ff9740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e6dee49080 0x55e6dee53000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:42.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.196+0000 7f6621ff9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee4e1f0 0x55e6dee5a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:42.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.197+0000 7f6621ff9740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6ded6a240 con 0x55e6dee49080 2024-03-28T01:00:42.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.197+0000 7f6621ff9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6dedaa460 con 0x55e6dee48690 2024-03-28T01:00:42.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.197+0000 7f6621ff9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6dee51140 con 0x55e6dee4e1f0 2024-03-28T01:00:42.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.197+0000 7f6621ff9740 10 client.0 _ll_register_callbacks cb 0x55e6ded69c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:42.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.197+0000 7f661d1a7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee48690 0x55e6dee52ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:42.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.197+0000 7f661d1a7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee48690 0x55e6dee52ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:56098/0 (socket says 172.21.15.169:56098) 2024-03-28T01:00:42.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.197+0000 7f661d1a7640 1 -- 172.21.15.169:0/2239000818 learned_addr learned my addr 172.21.15.169:0/2239000818 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:42.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.197+0000 7f661d9a8640 1 --2- 172.21.15.169:0/2239000818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee4e1f0 0x55e6dee5a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:42.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.197+0000 7f661d1a7640 1 -- 172.21.15.169:0/2239000818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee4e1f0 msgr2=0x55e6dee5a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:42.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.197+0000 7f661d1a7640 1 --2- 172.21.15.169:0/2239000818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee4e1f0 0x55e6dee5a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.198+0000 7f661d1a7640 1 -- 172.21.15.169:0/2239000818 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e6dee49080 msgr2=0x55e6dee53000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:42.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.198+0000 7f661d1a7640 1 --2- 172.21.15.169:0/2239000818 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e6dee49080 0x55e6dee53000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.198+0000 7f661d1a7640 1 -- 172.21.15.169:0/2239000818 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e6deb8c7d0 con 0x55e6dee48690 2024-03-28T01:00:42.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.199+0000 7f661d1a7640 1 --2- 172.21.15.169:0/2239000818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee48690 0x55e6dee52ac0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6610017bd0 tx=0x7f6610004b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afc1aa7268369e12 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:42.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.200+0000 7f6600ff9640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:00:42.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.200+0000 7f6600ff9640 1 -- 172.21.15.169:0/2239000818 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6610027060 con 0x55e6dee48690 2024-03-28T01:00:42.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.200+0000 7f6621ff9740 1 -- 172.21.15.169:0/2239000818 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6614004040 con 0x55e6dee48690 2024-03-28T01:00:42.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.200+0000 7f6600ff9640 1 -- 172.21.15.169:0/2239000818 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f6610012030 con 0x55e6dee48690 2024-03-28T01:00:42.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.201+0000 7f6600ff9640 1 -- 172.21.15.169:0/2239000818 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6610017cc0 con 0x55e6dee48690 2024-03-28T01:00:42.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.202+0000 7f6621ff9740 10 client.26183 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:42.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.202+0000 7f6621ff9740 1 -- 172.21.15.169:0/2239000818 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55e6dee5d450 con 0x55e6dee48690 2024-03-28T01:00:42.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.205+0000 7f6600ff9640 1 -- 172.21.15.169:0/2239000818 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6614004040 con 0x55e6dee48690 2024-03-28T01:00:42.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.205+0000 7f6600ff9640 1 -- 172.21.15.169:0/2239000818 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f661002b9b0 con 0x55e6dee48690 2024-03-28T01:00:42.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.205+0000 7f6621ff9740 10 client.26183 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:42.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.205+0000 7f6621ff9740 10 client.26183 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:42.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.205+0000 7f6621ff9740 20 client.26183 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:42.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.205+0000 7f6621ff9740 1 --2- 172.21.15.169:0/2239000818 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e6dee5eb50 0x55e6dee60fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:42.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.205+0000 7f6621ff9740 4 client.26183 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:00:42.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.205+0000 7f6621ff9740 1 -- 172.21.15.169:0/2239000818 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e6dedbb900 con 0x55e6dee5eb50 2024-03-28T01:00:42.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.207+0000 7f661c9a6640 1 --2- 172.21.15.169:0/2239000818 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e6dee5eb50 0x55e6dee60fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:42.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.218+0000 7f661c9a6640 1 --2- 172.21.15.169:0/2239000818 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e6dee5eb50 0x55e6dee60fb0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a0121b43893886a server_cookie=ea1559f61738a55b in_seq=0 out_seq=0 2024-03-28T01:00:42.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.218+0000 7f6600ff9640 10 client.26183 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:42.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.225+0000 7f6600ff9640 1 -- 172.21.15.169:0/2239000818 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e6dedbb900 con 0x55e6dee5eb50 2024-03-28T01:00:42.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.225+0000 7f6600ff9640 10 client.26183 handle_command_reply: tid=0 2024-03-28T01:00:42.228 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.225+0000 7f6621ff9740 1 client.26183 shutdown 2024-03-28T01:00:42.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.228+0000 7f6621ff9740 1 -- 172.21.15.169:0/2239000818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee48690 msgr2=0x55e6dee52ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:42.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.228+0000 7f6621ff9740 1 --2- 172.21.15.169:0/2239000818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee48690 0x55e6dee52ac0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6610017bd0 tx=0x7f6610004b50 comp rx=0 tx=0).stop 2024-03-28T01:00:42.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.229+0000 7f6621ff9740 1 -- 172.21.15.169:0/2239000818 shutdown_connections 2024-03-28T01:00:42.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.229+0000 7f6621ff9740 1 --2- 172.21.15.169:0/2239000818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee4e1f0 0x55e6dee5a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.229+0000 7f6621ff9740 1 --2- 172.21.15.169:0/2239000818 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e6dee5eb50 0x55e6dee60fb0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.229+0000 7f6621ff9740 1 --2- 172.21.15.169:0/2239000818 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e6dee49080 0x55e6dee53000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.229+0000 7f6621ff9740 1 --2- 172.21.15.169:0/2239000818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee48690 0x55e6dee52ac0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.229+0000 7f6621ff9740 1 -- 172.21.15.169:0/2239000818 >> 172.21.15.169:0/2239000818 conn(0x55e6dee43c00 msgr2=0x55e6dee42e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:42.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.229+0000 7f6600ff9640 0 client.26183 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:42.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.231+0000 7f6621ff9740 1 -- 172.21.15.169:0/2239000818 shutdown_connections 2024-03-28T01:00:42.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.231+0000 7f6621ff9740 1 -- 172.21.15.169:0/2239000818 wait complete. 2024-03-28T01:00:42.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.231+0000 7f6621ff9740 20 client.26183 trim_cache size 0 max 16384 2024-03-28T01:00:42.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.269+0000 7f6621ff9740 1 Processor -- start 2024-03-28T01:00:42.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.271+0000 7f6621ff9740 1 -- start start 2024-03-28T01:00:42.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.271+0000 7f6621ff9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee48690 0x55e6dee52de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:42.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.271+0000 7f6621ff9740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e6dee49080 0x55e6dee5a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:42.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.271+0000 7f6621ff9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee4e1f0 0x55e6dee51570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:42.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.271+0000 7f6621ff9740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc0120e0 con 0x55e6dee49080 2024-03-28T01:00:42.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.271+0000 7f6621ff9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc00fbe0 con 0x55e6dee48690 2024-03-28T01:00:42.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.271+0000 7f6621ff9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc00dc80 con 0x55e6dee4e1f0 2024-03-28T01:00:42.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.271+0000 7f661d1a7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee48690 0x55e6dee52de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:42.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.272+0000 7f661d1a7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee48690 0x55e6dee52de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:56110/0 (socket says 172.21.15.169:56110) 2024-03-28T01:00:42.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.272+0000 7f661d1a7640 1 -- 172.21.15.169:0/4083136174 learned_addr learned my addr 172.21.15.169:0/4083136174 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:42.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.272+0000 7f661d9a8640 1 --2- 172.21.15.169:0/4083136174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee4e1f0 0x55e6dee51570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:42.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.272+0000 7f661d1a7640 1 -- 172.21.15.169:0/4083136174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee4e1f0 msgr2=0x55e6dee51570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:42.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.272+0000 7f661d1a7640 1 --2- 172.21.15.169:0/4083136174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee4e1f0 0x55e6dee51570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.272+0000 7f661d1a7640 1 -- 172.21.15.169:0/4083136174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e6dee49080 msgr2=0x55e6dee5a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:42.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.272+0000 7f661d1a7640 1 --2- 172.21.15.169:0/4083136174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e6dee49080 0x55e6dee5a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.272+0000 7f661d1a7640 1 -- 172.21.15.169:0/4083136174 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e6dee44970 con 0x55e6dee48690 2024-03-28T01:00:42.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.272+0000 7f661d1a7640 1 --2- 172.21.15.169:0/4083136174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee48690 0x55e6dee52de0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6610009510 tx=0x7f661002bf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5b8dc53e03217ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:42.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.273+0000 7f6602ffd640 1 -- 172.21.15.169:0/4083136174 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6610031080 con 0x55e6dee48690 2024-03-28T01:00:42.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.273+0000 7f6602ffd640 1 -- 172.21.15.169:0/4083136174 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f661002a410 con 0x55e6dee48690 2024-03-28T01:00:42.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.274+0000 7f6602ffd640 1 -- 172.21.15.169:0/4083136174 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661002ae40 con 0x55e6dee48690 2024-03-28T01:00:42.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.274+0000 7f6621ff9740 1 -- 172.21.15.169:0/4083136174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee48690 msgr2=0x55e6dee52de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:42.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.274+0000 7f6621ff9740 1 --2- 172.21.15.169:0/4083136174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee48690 0x55e6dee52de0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6610009510 tx=0x7f661002bf70 comp rx=0 tx=0).stop 2024-03-28T01:00:42.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.274+0000 7f6621ff9740 1 -- 172.21.15.169:0/4083136174 shutdown_connections 2024-03-28T01:00:42.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.274+0000 7f6621ff9740 1 --2- 172.21.15.169:0/4083136174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee4e1f0 0x55e6dee51570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.274+0000 7f6621ff9740 1 --2- 172.21.15.169:0/4083136174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e6dee49080 0x55e6dee5a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.274+0000 7f6621ff9740 1 --2- 172.21.15.169:0/4083136174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee48690 0x55e6dee52de0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.274+0000 7f6621ff9740 1 -- 172.21.15.169:0/4083136174 >> 172.21.15.169:0/4083136174 conn(0x55e6dee43c00 msgr2=0x55e6dee526d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:42.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.275+0000 7f6621ff9740 1 -- 172.21.15.169:0/4083136174 shutdown_connections 2024-03-28T01:00:42.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.275+0000 7f6621ff9740 1 -- 172.21.15.169:0/4083136174 wait complete. 2024-03-28T01:00:42.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.275+0000 7f6621ff9740 1 -- start start 2024-03-28T01:00:42.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.275+0000 7f6621ff9740 1 Processor -- start 2024-03-28T01:00:42.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.276+0000 7f6621ff9740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e6dee48690 0x55e6dee57820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:42.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.276+0000 7f6621ff9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee49080 0x55e6dee57d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:42.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.276+0000 7f6621ff9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee4e1f0 0x55e6dee58d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:42.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.276+0000 7f6621ff9740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc00e280 con 0x55e6dee48690 2024-03-28T01:00:42.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.276+0000 7f6621ff9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc00f580 con 0x55e6dee4e1f0 2024-03-28T01:00:42.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.276+0000 7f6621ff9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc00e8c0 con 0x55e6dee49080 2024-03-28T01:00:42.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.276+0000 7f6621ff9740 10 client.0 _ll_register_callbacks cb 0x7f6610017880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:42.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.276+0000 7f661c9a6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee49080 0x55e6dee57d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:42.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.276+0000 7f661c9a6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee49080 0x55e6dee57d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:33158/0 (socket says 172.21.15.169:33158) 2024-03-28T01:00:42.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.276+0000 7f661c9a6640 1 -- 172.21.15.169:0/2645227628 learned_addr learned my addr 172.21.15.169:0/2645227628 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:42.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.276+0000 7f661d9a8640 1 --2- 172.21.15.169:0/2645227628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee4e1f0 0x55e6dee58d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:42.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.277+0000 7f661c9a6640 1 -- 172.21.15.169:0/2645227628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee4e1f0 msgr2=0x55e6dee58d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:42.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.277+0000 7f661c9a6640 1 --2- 172.21.15.169:0/2645227628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee4e1f0 0x55e6dee58d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.277+0000 7f661c9a6640 1 -- 172.21.15.169:0/2645227628 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e6dee48690 msgr2=0x55e6dee57820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:42.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.277+0000 7f661c9a6640 1 --2- 172.21.15.169:0/2645227628 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e6dee48690 0x55e6dee57820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.277+0000 7f661c9a6640 1 -- 172.21.15.169:0/2645227628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e6dee592f0 con 0x55e6dee49080 2024-03-28T01:00:42.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.278+0000 7f661c9a6640 1 --2- 172.21.15.169:0/2645227628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee49080 0x55e6dee57d60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6608000c00 tx=0x7f660809b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2706f69fd83793ef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:42.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.278+0000 7f6600ff9640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:00:42.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.278+0000 7f6600ff9640 1 -- 172.21.15.169:0/2645227628 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66080b8060 con 0x55e6dee49080 2024-03-28T01:00:42.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.278+0000 7f6600ff9640 1 -- 172.21.15.169:0/2645227628 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f6608009d60 con 0x55e6dee49080 2024-03-28T01:00:42.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.278+0000 7f6621ff9740 1 -- 172.21.15.169:0/2645227628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f661002a410 con 0x55e6dee49080 2024-03-28T01:00:42.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.278+0000 7f6600ff9640 1 -- 172.21.15.169:0/2645227628 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6608020040 con 0x55e6dee49080 2024-03-28T01:00:42.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.278+0000 7f6621ff9740 10 client.26122 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:42.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.278+0000 7f6621ff9740 1 -- 172.21.15.169:0/2645227628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55e6deeba570 con 0x55e6dee49080 2024-03-28T01:00:42.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.278+0000 7f6600ff9640 1 -- 172.21.15.169:0/2645227628 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f661002a410 con 0x55e6dee49080 2024-03-28T01:00:42.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.278+0000 7f6600ff9640 1 -- 172.21.15.169:0/2645227628 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f6608099d20 con 0x55e6dee49080 2024-03-28T01:00:42.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.278+0000 7f6621ff9740 10 client.26122 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:42.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.278+0000 7f6621ff9740 10 client.26122 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:42.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.278+0000 7f6621ff9740 20 client.26122 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:42.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.278+0000 7f6621ff9740 1 --2- 172.21.15.169:0/2645227628 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e6deebafa0 0x55e6deebd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:42.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.279+0000 7f6621ff9740 4 client.26122 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:00:42.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.279+0000 7f6621ff9740 1 -- 172.21.15.169:0/2645227628 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e6dedb5c50 con 0x55e6deebafa0 2024-03-28T01:00:42.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.279+0000 7f661d1a7640 1 --2- 172.21.15.169:0/2645227628 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e6deebafa0 0x55e6deebd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:42.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.281+0000 7f661d1a7640 1 --2- 172.21.15.169:0/2645227628 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e6deebafa0 0x55e6deebd3c0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7049bc6ab0bcb1b5 server_cookie=1836aa4cf8f2fbbf in_seq=0 out_seq=0 2024-03-28T01:00:42.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.281+0000 7f6600ff9640 10 client.26122 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:42.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.284+0000 7f6600ff9640 1 -- 172.21.15.169:0/2645227628 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e6dedb5c50 con 0x55e6deebafa0 2024-03-28T01:00:42.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.284+0000 7f6600ff9640 10 client.26122 handle_command_reply: tid=0 2024-03-28T01:00:42.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.284+0000 7f6621ff9740 1 client.26122 shutdown 2024-03-28T01:00:42.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.284+0000 7f6621ff9740 1 -- 172.21.15.169:0/2645227628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee49080 msgr2=0x55e6dee57d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:42.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.284+0000 7f6621ff9740 1 --2- 172.21.15.169:0/2645227628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee49080 0x55e6dee57d60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6608000c00 tx=0x7f660809b670 comp rx=0 tx=0).stop 2024-03-28T01:00:42.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.285+0000 7f6621ff9740 1 -- 172.21.15.169:0/2645227628 shutdown_connections 2024-03-28T01:00:42.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.285+0000 7f6621ff9740 1 --2- 172.21.15.169:0/2645227628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6dee4e1f0 0x55e6dee58d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.285+0000 7f6621ff9740 1 --2- 172.21.15.169:0/2645227628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e6dee49080 0x55e6dee57d60 secure :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6608000c00 tx=0x7f660809b670 comp rx=0 tx=0).stop 2024-03-28T01:00:42.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.285+0000 7f6621ff9740 1 --2- 172.21.15.169:0/2645227628 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e6deebafa0 0x55e6deebd3c0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.285+0000 7f6621ff9740 1 --2- 172.21.15.169:0/2645227628 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e6dee48690 0x55e6dee57820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.285+0000 7f6621ff9740 1 -- 172.21.15.169:0/2645227628 >> 172.21.15.169:0/2645227628 conn(0x55e6dee43c00 msgr2=0x55e6dee46e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:42.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.285+0000 7f6600ff9640 0 client.26122 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:42.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.285+0000 7f6621ff9740 1 -- 172.21.15.169:0/2645227628 shutdown_connections 2024-03-28T01:00:42.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.285+0000 7f6621ff9740 1 -- 172.21.15.169:0/2645227628 wait complete. 2024-03-28T01:00:42.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.285+0000 7f6621ff9740 20 client.26122 trim_cache size 0 max 16384 2024-03-28T01:00:42.290 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:00:42.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.288+0000 7f661f40b640 1 -- 172.21.15.169:0/51066547 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f66040a2a00 msgr2=0x7f66040a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:42.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.288+0000 7f661f40b640 1 --2- 172.21.15.169:0/51066547 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f66040a2a00 0x7f66040a4ec0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x55e6dee44b70 tx=0x7f6610009410 comp rx=0 tx=0).stop 2024-03-28T01:00:42.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.288+0000 7f661f40b640 1 -- 172.21.15.169:0/51066547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f661810b010 msgr2=0x7f6618119e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:42.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.288+0000 7f661f40b640 1 --2- 172.21.15.169:0/51066547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f661810b010 0x7f6618119e30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f660800d950 tx=0x7f660800de10 comp rx=0 tx=0).stop 2024-03-28T01:00:42.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.288+0000 7f661f40b640 1 -- 172.21.15.169:0/51066547 shutdown_connections 2024-03-28T01:00:42.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.288+0000 7f661f40b640 1 --2- 172.21.15.169:0/51066547 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f66040a2a00 0x7f66040a4ec0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.288+0000 7f661f40b640 1 --2- 172.21.15.169:0/51066547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f661810d9c0 0x7f6618117f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.288+0000 7f661f40b640 1 --2- 172.21.15.169:0/51066547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f661810b010 0x7f6618119e30 secure :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f660800d950 tx=0x7f660800de10 comp rx=0 tx=0).stop 2024-03-28T01:00:42.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.288+0000 7f661f40b640 1 --2- 172.21.15.169:0/51066547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66181086a0 0x7f66181198f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:42.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.288+0000 7f661f40b640 1 -- 172.21.15.169:0/51066547 >> 172.21.15.169:0/51066547 conn(0x7f661806e500 msgr2=0x7f6618108ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:42.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.292+0000 7f661f40b640 1 -- 172.21.15.169:0/51066547 shutdown_connections 2024-03-28T01:00:42.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:42.293+0000 7f661f40b640 1 -- 172.21.15.169:0/51066547 wait complete. 2024-03-28T01:00:43.333 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T01:00:43.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T01:00:43.333 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T01:00:43.568 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:43 smithi169 ceph-mon[32274]: pgmap v477: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 879 MiB/s wr, 1.36k op/s 2024-03-28T01:00:43.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:43 smithi174 ceph-mon[36060]: pgmap v477: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 879 MiB/s wr, 1.36k op/s 2024-03-28T01:00:43.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:43 smithi177 ceph-mon[36206]: pgmap v477: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 879 MiB/s wr, 1.36k op/s 2024-03-28T01:00:45.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:45 smithi174 ceph-mon[36060]: pgmap v478: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 727 MiB/s wr, 1.06k op/s 2024-03-28T01:00:45.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:45 smithi177 ceph-mon[36206]: pgmap v478: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 727 MiB/s wr, 1.06k op/s 2024-03-28T01:00:45.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:45 smithi169 ceph-mon[32274]: pgmap v478: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 727 MiB/s wr, 1.06k op/s 2024-03-28T01:00:46.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.555+0000 7f44cffff640 1 -- 172.21.15.169:0/1806868130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44d0100550 msgr2=0x7f44d01009b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:46.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.555+0000 7f44cffff640 1 --2- 172.21.15.169:0/1806868130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44d0100550 0x7f44d01009b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f44c4009900 tx=0x7f44c402bc80 comp rx=0 tx=0).stop 2024-03-28T01:00:46.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.558+0000 7f44cffff640 1 -- 172.21.15.169:0/1806868130 shutdown_connections 2024-03-28T01:00:46.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.558+0000 7f44cffff640 1 --2- 172.21.15.169:0/1806868130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44d0100ef0 0x7f44d01057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:46.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.558+0000 7f44cffff640 1 --2- 172.21.15.169:0/1806868130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44d0100550 0x7f44d01009b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:46.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.558+0000 7f44cffff640 1 --2- 172.21.15.169:0/1806868130 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44d00ff420 0x7f44d00ff840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:46.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.558+0000 7f44cffff640 1 -- 172.21.15.169:0/1806868130 >> 172.21.15.169:0/1806868130 conn(0x7f44d00fad20 msgr2=0x7f44d00fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:46.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.559+0000 7f44cffff640 1 -- 172.21.15.169:0/1806868130 shutdown_connections 2024-03-28T01:00:46.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.563+0000 7f44cffff640 1 -- 172.21.15.169:0/1806868130 wait complete. 2024-03-28T01:00:46.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.615+0000 7f44cffff640 1 Processor -- start 2024-03-28T01:00:46.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.615+0000 7f44cffff640 1 -- start start 2024-03-28T01:00:46.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.616+0000 7f44cffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44d00ff420 0x7f44d010be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:46.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.616+0000 7f44cffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44d0100550 0x7f44d010c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:46.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.616+0000 7f44cffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44d0100ef0 0x7f44d01adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:46.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.616+0000 7f44cffff640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d01138a0 con 0x7f44d0100550 2024-03-28T01:00:46.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.616+0000 7f44cffff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d0113720 con 0x7f44d00ff420 2024-03-28T01:00:46.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.616+0000 7f44cffff640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d0113a20 con 0x7f44d0100ef0 2024-03-28T01:00:46.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.616+0000 7f44ce7fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44d0100550 0x7f44d010c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:46.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.616+0000 7f44ce7fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44d0100550 0x7f44d010c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:58336/0 (socket says 172.21.15.169:58336) 2024-03-28T01:00:46.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.616+0000 7f44ce7fc640 1 -- 172.21.15.169:0/2193389423 learned_addr learned my addr 172.21.15.169:0/2193389423 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:46.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.617+0000 7f44ceffd640 1 --2- 172.21.15.169:0/2193389423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44d00ff420 0x7f44d010be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:46.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.617+0000 7f44cf7fe640 1 --2- 172.21.15.169:0/2193389423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44d0100ef0 0x7f44d01adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:46.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.617+0000 7f44ceffd640 1 -- 172.21.15.169:0/2193389423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44d0100ef0 msgr2=0x7f44d01adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:46.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.617+0000 7f44ceffd640 1 --2- 172.21.15.169:0/2193389423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44d0100ef0 0x7f44d01adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:46.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.617+0000 7f44ceffd640 1 -- 172.21.15.169:0/2193389423 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44d0100550 msgr2=0x7f44d010c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:46.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.617+0000 7f44ceffd640 1 --2- 172.21.15.169:0/2193389423 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44d0100550 0x7f44d010c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:46.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.617+0000 7f44ceffd640 1 -- 172.21.15.169:0/2193389423 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d010f520 con 0x7f44d00ff420 2024-03-28T01:00:46.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.618+0000 7f44ceffd640 1 --2- 172.21.15.169:0/2193389423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44d00ff420 0x7f44d010be60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f44c000c970 tx=0x7f44c000ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:46.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.630+0000 7f44d4a09640 1 -- 172.21.15.169:0/2193389423 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c0007bb0 con 0x7f44d00ff420 2024-03-28T01:00:46.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.630+0000 7f44d4a09640 1 -- 172.21.15.169:0/2193389423 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44c0007d50 con 0x7f44d00ff420 2024-03-28T01:00:46.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.630+0000 7f44d4a09640 1 -- 172.21.15.169:0/2193389423 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c0018820 con 0x7f44d00ff420 2024-03-28T01:00:46.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.630+0000 7f44cffff640 1 -- 172.21.15.169:0/2193389423 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d006b0c0 con 0x7f44d00ff420 2024-03-28T01:00:46.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.631+0000 7f44cffff640 1 -- 172.21.15.169:0/2193389423 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d006b5c0 con 0x7f44d00ff420 2024-03-28T01:00:46.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.639+0000 7f44d4a09640 1 -- 172.21.15.169:0/2193389423 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f44c00189c0 con 0x7f44d00ff420 2024-03-28T01:00:46.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.646+0000 7f44d4a09640 1 --2- 172.21.15.169:0/2193389423 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f44b80a2990 0x7f44b80a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:46.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.646+0000 7f44d4a09640 1 -- 172.21.15.169:0/2193389423 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f44c00cfe30 con 0x7f44d00ff420 2024-03-28T01:00:46.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.651+0000 7f44ce7fc640 1 --2- 172.21.15.169:0/2193389423 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f44b80a2990 0x7f44b80a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:46.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.859+0000 7f44ce7fc640 1 --2- 172.21.15.169:0/2193389423 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f44b80a2990 0x7f44b80a4e50 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f44c402c8f0 tx=0x7f44c4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:46.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.866+0000 7f44d85fa740 1 Processor -- start 2024-03-28T01:00:46.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.867+0000 7f44d85fa740 1 -- start start 2024-03-28T01:00:46.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.867+0000 7f44d85fa740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559402f02b30 0x559402f02f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:46.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.867+0000 7f44d85fa740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f03520 0x559402f08020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:46.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.867+0000 7f44d85fa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559402f08690 0x559402f0ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:46.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.867+0000 7f44d85fa740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x559402e42c90 con 0x559402f02b30 2024-03-28T01:00:46.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.867+0000 7f44d85fa740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x559402e68070 con 0x559402f03520 2024-03-28T01:00:46.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.868+0000 7f44d85fa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x559402e10c80 con 0x559402f08690 2024-03-28T01:00:46.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.868+0000 7f44ce7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f03520 0x559402f08020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:46.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.868+0000 7f44ce7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f03520 0x559402f08020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:50248/0 (socket says 172.21.15.169:50248) 2024-03-28T01:00:46.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.868+0000 7f44ce7fc640 1 -- 172.21.15.169:0/1378251281 learned_addr learned my addr 172.21.15.169:0/1378251281 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:46.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.868+0000 7f44ce7fc640 1 -- 172.21.15.169:0/1378251281 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559402f08690 msgr2=0x559402f0ab20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:00:46.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.868+0000 7f44ce7fc640 1 --2- 172.21.15.169:0/1378251281 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559402f08690 0x559402f0ab20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:46.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.868+0000 7f44ce7fc640 1 -- 172.21.15.169:0/1378251281 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559402f02b30 msgr2=0x559402f02f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:46.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.868+0000 7f44ce7fc640 1 --2- 172.21.15.169:0/1378251281 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559402f02b30 0x559402f02f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:46.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.868+0000 7f44ce7fc640 1 -- 172.21.15.169:0/1378251281 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559402e08d10 con 0x559402f03520 2024-03-28T01:00:46.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.869+0000 7f44ce7fc640 1 --2- 172.21.15.169:0/1378251281 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f03520 0x559402f08020 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f44c402cb40 tx=0x7f44c403e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f6ee5e3d5b4d0b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:46.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.875+0000 7f44b5ffb640 1 -- 172.21.15.169:0/1378251281 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c404f070 con 0x559402f03520 2024-03-28T01:00:46.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.875+0000 7f44b5ffb640 1 -- 172.21.15.169:0/1378251281 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44c4045070 con 0x559402f03520 2024-03-28T01:00:46.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.875+0000 7f44b5ffb640 1 -- 172.21.15.169:0/1378251281 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c4003830 con 0x559402f03520 2024-03-28T01:00:46.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.877+0000 7f44d85fa740 1 -- 172.21.15.169:0/1378251281 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f03520 msgr2=0x559402f08020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:46.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.877+0000 7f44d85fa740 1 --2- 172.21.15.169:0/1378251281 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f03520 0x559402f08020 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f44c402cb40 tx=0x7f44c403e000 comp rx=0 tx=0).stop 2024-03-28T01:00:46.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.877+0000 7f44d85fa740 1 -- 172.21.15.169:0/1378251281 shutdown_connections 2024-03-28T01:00:46.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.877+0000 7f44d85fa740 1 --2- 172.21.15.169:0/1378251281 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559402f08690 0x559402f0ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:46.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.877+0000 7f44d85fa740 1 --2- 172.21.15.169:0/1378251281 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f03520 0x559402f08020 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:46.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.877+0000 7f44d85fa740 1 --2- 172.21.15.169:0/1378251281 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559402f02b30 0x559402f02f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:46.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.877+0000 7f44d85fa740 1 -- 172.21.15.169:0/1378251281 >> 172.21.15.169:0/1378251281 conn(0x559402ecb1a0 msgr2=0x559402efe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:46.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.877+0000 7f44d85fa740 1 -- 172.21.15.169:0/1378251281 shutdown_connections 2024-03-28T01:00:46.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.877+0000 7f44d85fa740 1 -- 172.21.15.169:0/1378251281 wait complete. 2024-03-28T01:00:46.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.957+0000 7f44d85fa740 1 -- start start 2024-03-28T01:00:46.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.989+0000 7f44d85fa740 1 Processor -- start 2024-03-28T01:00:46.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.992+0000 7f44d85fa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559402f02b30 0x559402f0cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:46.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.992+0000 7f44d85fa740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559402f03520 0x559402f0d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:46.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.992+0000 7f44d85fa740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f08690 0x559402f14500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:46.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.992+0000 7f44d85fa740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x559402e240d0 con 0x559402f03520 2024-03-28T01:00:46.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.992+0000 7f44d85fa740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x559402e64110 con 0x559402f08690 2024-03-28T01:00:46.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.992+0000 7f44d85fa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x559402f0b5e0 con 0x559402f02b30 2024-03-28T01:00:46.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.994+0000 7f44cf7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f08690 0x559402f14500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:46.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.994+0000 7f44cf7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f08690 0x559402f14500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:50258/0 (socket says 172.21.15.169:50258) 2024-03-28T01:00:46.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.994+0000 7f44cf7fe640 1 -- 172.21.15.169:0/3628536672 learned_addr learned my addr 172.21.15.169:0/3628536672 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:47.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.996+0000 7f44ceffd640 1 --2- 172.21.15.169:0/3628536672 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559402f02b30 0x559402f0cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:47.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.997+0000 7f44ce7fc640 1 --2- 172.21.15.169:0/3628536672 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559402f03520 0x559402f0d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:47.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.997+0000 7f44cf7fe640 1 -- 172.21.15.169:0/3628536672 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559402f02b30 msgr2=0x559402f0cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:47.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.997+0000 7f44cf7fe640 1 --2- 172.21.15.169:0/3628536672 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559402f02b30 0x559402f0cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.997+0000 7f44cf7fe640 1 -- 172.21.15.169:0/3628536672 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559402f03520 msgr2=0x559402f0d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:47.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.997+0000 7f44cf7fe640 1 --2- 172.21.15.169:0/3628536672 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559402f03520 0x559402f0d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.997+0000 7f44cf7fe640 1 -- 172.21.15.169:0/3628536672 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559402c447e0 con 0x559402f08690 2024-03-28T01:00:47.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.997+0000 7f44cf7fe640 1 --2- 172.21.15.169:0/3628536672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f08690 0x559402f14500 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f44c800aaf0 tx=0x7f44c800e0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f1e1dd527f9a442 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:47.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.998+0000 7f44aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:00:47.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:46.998+0000 7f44aeffd640 1 -- 172.21.15.169:0/3628536672 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c8018020 con 0x559402f08690 2024-03-28T01:00:47.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.001+0000 7f44aeffd640 1 -- 172.21.15.169:0/3628536672 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f44c800ec80 con 0x559402f08690 2024-03-28T01:00:47.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.029+0000 7f44d85fa740 10 client.0 _ll_register_callbacks cb 0x559402e5fe60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:47.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.037+0000 7f44d85fa740 1 -- 172.21.15.169:0/3628536672 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f44c4045070 con 0x559402f08690 2024-03-28T01:00:47.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.045+0000 7f44aeffd640 1 -- 172.21.15.169:0/3628536672 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44c8023050 con 0x559402f08690 2024-03-28T01:00:47.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.045+0000 7f44d85fa740 10 client.26201 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:47.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.045+0000 7f44d85fa740 1 -- 172.21.15.169:0/3628536672 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x559402ca4f70 con 0x559402f08690 2024-03-28T01:00:47.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.049+0000 7f44aeffd640 1 -- 172.21.15.169:0/3628536672 <== mon.2 v2:172.21.15.174:3300/0 4 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f44c8014400 con 0x559402f08690 2024-03-28T01:00:47.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.049+0000 7f44d85fa740 10 client.26201 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:47.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.049+0000 7f44d85fa740 10 client.26201 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:47.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.049+0000 7f44d85fa740 20 client.26201 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:47.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.049+0000 7f44d85fa740 1 --2- 172.21.15.169:0/3628536672 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559402f18fd0 0x559402f1b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:47.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.049+0000 7f44d85fa740 4 client.26201 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:00:47.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.049+0000 7f44d85fa740 1 -- 172.21.15.169:0/3628536672 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559402e755b0 con 0x559402f18fd0 2024-03-28T01:00:47.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.058+0000 7f44ceffd640 1 --2- 172.21.15.169:0/3628536672 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559402f18fd0 0x559402f1b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:47.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.068+0000 7f44ceffd640 1 --2- 172.21.15.169:0/3628536672 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559402f18fd0 0x559402f1b490 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a417430dd60bffe8 server_cookie=a074ac4095e17dec in_seq=0 out_seq=0 2024-03-28T01:00:47.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.068+0000 7f44aeffd640 10 client.26201 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:47.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.076+0000 7f44aeffd640 1 -- 172.21.15.169:0/3628536672 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559402e755b0 con 0x559402f18fd0 2024-03-28T01:00:47.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.076+0000 7f44aeffd640 10 client.26201 handle_command_reply: tid=0 2024-03-28T01:00:47.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.076+0000 7f44d85fa740 1 client.26201 shutdown 2024-03-28T01:00:47.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.079+0000 7f44d85fa740 1 -- 172.21.15.169:0/3628536672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f08690 msgr2=0x559402f14500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:47.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.079+0000 7f44d85fa740 1 --2- 172.21.15.169:0/3628536672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f08690 0x559402f14500 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f44c800aaf0 tx=0x7f44c800e0c0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.082+0000 7f44d85fa740 1 -- 172.21.15.169:0/3628536672 shutdown_connections 2024-03-28T01:00:47.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.082+0000 7f44d85fa740 1 --2- 172.21.15.169:0/3628536672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f08690 0x559402f14500 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.082+0000 7f44d85fa740 1 --2- 172.21.15.169:0/3628536672 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559402f18fd0 0x559402f1b490 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.082+0000 7f44d85fa740 1 --2- 172.21.15.169:0/3628536672 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559402f03520 0x559402f0d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.082+0000 7f44d85fa740 1 --2- 172.21.15.169:0/3628536672 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559402f02b30 0x559402f0cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.082+0000 7f44aeffd640 0 client.26201 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:47.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.082+0000 7f44d85fa740 1 -- 172.21.15.169:0/3628536672 >> 172.21.15.169:0/3628536672 conn(0x559402ecb1a0 msgr2=0x559402efd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:47.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.086+0000 7f44d85fa740 1 -- 172.21.15.169:0/3628536672 shutdown_connections 2024-03-28T01:00:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.086+0000 7f44d85fa740 1 -- 172.21.15.169:0/3628536672 wait complete. 2024-03-28T01:00:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.086+0000 7f44d85fa740 20 client.26201 trim_cache size 0 max 16384 2024-03-28T01:00:47.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.134+0000 7f44d85fa740 1 Processor -- start 2024-03-28T01:00:47.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.134+0000 7f44d85fa740 1 -- start start 2024-03-28T01:00:47.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.134+0000 7f44d85fa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559402f02b30 0x559402f0d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:47.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.135+0000 7f44d85fa740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f03520 0x559402f144c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:47.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.135+0000 7f44d85fa740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559402f08690 0x559402f016e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:47.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.135+0000 7f44d85fa740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d01c3950 con 0x559402f08690 2024-03-28T01:00:47.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.135+0000 7f44cf7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559402f08690 0x559402f016e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:47.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.135+0000 7f44cf7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559402f08690 0x559402f016e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:58358/0 (socket says 172.21.15.169:58358) 2024-03-28T01:00:47.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.135+0000 7f44cf7fe640 1 -- 172.21.15.169:0/3981040295 learned_addr learned my addr 172.21.15.169:0/3981040295 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:47.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.135+0000 7f44d85fa740 1 -- 172.21.15.169:0/3981040295 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d01c13e0 con 0x559402f03520 2024-03-28T01:00:47.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.135+0000 7f44d85fa740 1 -- 172.21.15.169:0/3981040295 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d01bf500 con 0x559402f02b30 2024-03-28T01:00:47.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.136+0000 7f44ceffd640 1 --2- 172.21.15.169:0/3981040295 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559402f02b30 0x559402f0d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:47.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.136+0000 7f44cf7fe640 1 -- 172.21.15.169:0/3981040295 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559402f02b30 msgr2=0x559402f0d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:47.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.136+0000 7f44cf7fe640 1 --2- 172.21.15.169:0/3981040295 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559402f02b30 0x559402f0d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.136+0000 7f44cf7fe640 1 -- 172.21.15.169:0/3981040295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f03520 msgr2=0x559402f144c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:00:47.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.136+0000 7f44cf7fe640 1 --2- 172.21.15.169:0/3981040295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f03520 0x559402f144c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.136+0000 7f44cf7fe640 1 -- 172.21.15.169:0/3981040295 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559402b3c170 con 0x559402f08690 2024-03-28T01:00:47.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.137+0000 7f44cf7fe640 1 --2- 172.21.15.169:0/3981040295 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559402f08690 0x559402f016e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f44c80146c0 tx=0x7f44c8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc8f060fb19dc70 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:47.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.137+0000 7f44b57fa640 1 -- 172.21.15.169:0/3981040295 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c8018050 con 0x559402f08690 2024-03-28T01:00:47.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.138+0000 7f44b57fa640 1 -- 172.21.15.169:0/3981040295 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44c4045070 con 0x559402f08690 2024-03-28T01:00:47.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.138+0000 7f44b57fa640 1 -- 172.21.15.169:0/3981040295 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c8004020 con 0x559402f08690 2024-03-28T01:00:47.142 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.139+0000 7f44d85fa740 1 -- 172.21.15.169:0/3981040295 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559402f08690 msgr2=0x559402f016e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:47.142 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.139+0000 7f44d85fa740 1 --2- 172.21.15.169:0/3981040295 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559402f08690 0x559402f016e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f44c80146c0 tx=0x7f44c8013000 comp rx=0 tx=0).stop 2024-03-28T01:00:47.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.143+0000 7f44d85fa740 1 -- 172.21.15.169:0/3981040295 shutdown_connections 2024-03-28T01:00:47.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.143+0000 7f44d85fa740 1 --2- 172.21.15.169:0/3981040295 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559402f08690 0x559402f016e0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.143+0000 7f44d85fa740 1 --2- 172.21.15.169:0/3981040295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f03520 0x559402f144c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.143+0000 7f44d85fa740 1 --2- 172.21.15.169:0/3981040295 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559402f02b30 0x559402f0d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.143+0000 7f44d85fa740 1 -- 172.21.15.169:0/3981040295 >> 172.21.15.169:0/3981040295 conn(0x559402ecb1a0 msgr2=0x559402f0cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:47.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.144+0000 7f44d85fa740 1 -- 172.21.15.169:0/3981040295 shutdown_connections 2024-03-28T01:00:47.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.144+0000 7f44d85fa740 1 -- 172.21.15.169:0/3981040295 wait complete. 2024-03-28T01:00:47.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.144+0000 7f44d85fa740 1 -- start start 2024-03-28T01:00:47.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.145+0000 7f44d85fa740 1 Processor -- start 2024-03-28T01:00:47.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.145+0000 7f44d85fa740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559402f02b30 0x559402f11e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:47.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.146+0000 7f44d85fa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559402f03520 0x559402f12350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:47.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.146+0000 7f44d85fa740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f08690 0x559402f13310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:47.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.146+0000 7f44d85fa740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d01bfa80 con 0x559402f02b30 2024-03-28T01:00:47.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.146+0000 7f44d85fa740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d01c0d80 con 0x559402f08690 2024-03-28T01:00:47.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.146+0000 7f44d85fa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d01c00c0 con 0x559402f03520 2024-03-28T01:00:47.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.146+0000 7f44cf7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f08690 0x559402f13310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:47.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.146+0000 7f44d85fa740 10 client.0 _ll_register_callbacks cb 0x7f44c8009a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:47.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.146+0000 7f44cf7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f08690 0x559402f13310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:50264/0 (socket says 172.21.15.169:50264) 2024-03-28T01:00:47.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.146+0000 7f44cf7fe640 1 -- 172.21.15.169:0/1567754436 learned_addr learned my addr 172.21.15.169:0/1567754436 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:47.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.147+0000 7f44ceffd640 1 --2- 172.21.15.169:0/1567754436 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559402f02b30 0x559402f11e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:47.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.148+0000 7f44ce7fc640 1 --2- 172.21.15.169:0/1567754436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559402f03520 0x559402f12350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:47.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.149+0000 7f44ce7fc640 1 -- 172.21.15.169:0/1567754436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f08690 msgr2=0x559402f13310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:47.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.149+0000 7f44ce7fc640 1 --2- 172.21.15.169:0/1567754436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f08690 0x559402f13310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.149+0000 7f44ce7fc640 1 -- 172.21.15.169:0/1567754436 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559402f02b30 msgr2=0x559402f11e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:47.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.149+0000 7f44ce7fc640 1 --2- 172.21.15.169:0/1567754436 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559402f02b30 0x559402f11e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.149+0000 7f44ce7fc640 1 -- 172.21.15.169:0/1567754436 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559402f13900 con 0x559402f03520 2024-03-28T01:00:47.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.150+0000 7f44ce7fc640 1 --2- 172.21.15.169:0/1567754436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559402f03520 0x559402f12350 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f44c40044d0 tx=0x7f44c40048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86ce4a2e0799511e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:47.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.151+0000 7f44aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:00:47.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.154+0000 7f44aeffd640 1 -- 172.21.15.169:0/1567754436 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c404f040 con 0x559402f03520 2024-03-28T01:00:47.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.154+0000 7f44d85fa740 1 -- 172.21.15.169:0/1567754436 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f44c4045070 con 0x559402f03520 2024-03-28T01:00:47.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.154+0000 7f44aeffd640 1 -- 172.21.15.169:0/1567754436 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f44c40035f0 con 0x559402f03520 2024-03-28T01:00:47.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.156+0000 7f44aeffd640 1 -- 172.21.15.169:0/1567754436 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44c4066050 con 0x559402f03520 2024-03-28T01:00:47.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.158+0000 7f44d85fa740 10 client.26131 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:47.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.158+0000 7f44d85fa740 1 -- 172.21.15.169:0/1567754436 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x559402f74bf0 con 0x559402f03520 2024-03-28T01:00:47.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.160+0000 7f44aeffd640 1 -- 172.21.15.169:0/1567754436 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44c4045070 con 0x559402f03520 2024-03-28T01:00:47.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.160+0000 7f44aeffd640 1 -- 172.21.15.169:0/1567754436 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f44c404a070 con 0x559402f03520 2024-03-28T01:00:47.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.160+0000 7f44d85fa740 10 client.26131 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:47.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.160+0000 7f44d85fa740 10 client.26131 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:47.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.160+0000 7f44d85fa740 20 client.26131 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:47.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.160+0000 7f44d85fa740 1 --2- 172.21.15.169:0/1567754436 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559402f75070 0x559402f77460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:47.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.160+0000 7f44d85fa740 4 client.26131 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T01:00:47.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.160+0000 7f44d85fa740 1 -- 172.21.15.169:0/1567754436 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x559402e6f900 con 0x559402f75070 2024-03-28T01:00:47.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.161+0000 7f44ceffd640 1 --2- 172.21.15.169:0/1567754436 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559402f75070 0x559402f77460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:47.174 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.171+0000 7f44ceffd640 1 --2- 172.21.15.169:0/1567754436 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559402f75070 0x559402f77460 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c88729262123b52a server_cookie=b4d94cba7916641b in_seq=0 out_seq=0 2024-03-28T01:00:47.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.171+0000 7f44aeffd640 10 client.26131 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:47.607 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:47 smithi174 ceph-mon[36060]: pgmap v479: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 787 MiB/s wr, 1.31k op/s 2024-03-28T01:00:47.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.659+0000 7f44aeffd640 1 -- 172.21.15.169:0/1567754436 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x559402e6f900 con 0x559402f75070 2024-03-28T01:00:47.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.659+0000 7f44aeffd640 10 client.26131 handle_command_reply: tid=0 2024-03-28T01:00:47.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.660+0000 7f44d85fa740 1 client.26131 shutdown 2024-03-28T01:00:47.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.660+0000 7f44d85fa740 1 -- 172.21.15.169:0/1567754436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559402f03520 msgr2=0x559402f12350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:47.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.660+0000 7f44d85fa740 1 --2- 172.21.15.169:0/1567754436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559402f03520 0x559402f12350 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f44c40044d0 tx=0x7f44c40048f0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.660+0000 7f44d85fa740 1 -- 172.21.15.169:0/1567754436 shutdown_connections 2024-03-28T01:00:47.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.660+0000 7f44d85fa740 1 --2- 172.21.15.169:0/1567754436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559402f08690 0x559402f13310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.660+0000 7f44d85fa740 1 --2- 172.21.15.169:0/1567754436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559402f03520 0x559402f12350 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.660+0000 7f44d85fa740 1 --2- 172.21.15.169:0/1567754436 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559402f75070 0x559402f77460 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.660+0000 7f44d85fa740 1 --2- 172.21.15.169:0/1567754436 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559402f02b30 0x559402f11e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.660+0000 7f44d85fa740 1 -- 172.21.15.169:0/1567754436 >> 172.21.15.169:0/1567754436 conn(0x559402ecb1a0 msgr2=0x559402efcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:47.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.660+0000 7f44aeffd640 0 client.26131 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:47.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.660+0000 7f44d85fa740 1 -- 172.21.15.169:0/1567754436 shutdown_connections 2024-03-28T01:00:47.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.660+0000 7f44d85fa740 1 -- 172.21.15.169:0/1567754436 wait complete. 2024-03-28T01:00:47.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.661+0000 7f44d85fa740 20 client.26131 trim_cache size 0 max 16384 2024-03-28T01:00:47.665 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:00:47.665 INFO:teuthology.orchestra.run.smithi169.stdout: "return_code": 0, 2024-03-28T01:00:47.665 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_tag": "d6f9488b-8563-4e0d-aba7-2ee84b2ad59c", 2024-03-28T01:00:47.665 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": "asynchronous" 2024-03-28T01:00:47.665 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:00:47.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.663+0000 7f44cffff640 1 -- 172.21.15.169:0/2193389423 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f44b80a2990 msgr2=0x7f44b80a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:47.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.663+0000 7f44cffff640 1 --2- 172.21.15.169:0/2193389423 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f44b80a2990 0x7f44b80a4e50 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f44c402c8f0 tx=0x7f44c4005fd0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.663+0000 7f44cffff640 1 -- 172.21.15.169:0/2193389423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44d00ff420 msgr2=0x7f44d010be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:47.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.663+0000 7f44cffff640 1 --2- 172.21.15.169:0/2193389423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44d00ff420 0x7f44d010be60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f44c000c970 tx=0x7f44c000ce30 comp rx=0 tx=0).stop 2024-03-28T01:00:47.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:47 smithi177 ceph-mon[36206]: pgmap v479: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 787 MiB/s wr, 1.31k op/s 2024-03-28T01:00:47.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.666+0000 7f44cffff640 1 -- 172.21.15.169:0/2193389423 shutdown_connections 2024-03-28T01:00:47.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.666+0000 7f44cffff640 1 --2- 172.21.15.169:0/2193389423 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f44b80a2990 0x7f44b80a4e50 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.666+0000 7f44cffff640 1 --2- 172.21.15.169:0/2193389423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44d0100ef0 0x7f44d01adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.666+0000 7f44cffff640 1 --2- 172.21.15.169:0/2193389423 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44d0100550 0x7f44d010c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.666+0000 7f44cffff640 1 --2- 172.21.15.169:0/2193389423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44d00ff420 0x7f44d010be60 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:47.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.666+0000 7f44cffff640 1 -- 172.21.15.169:0/2193389423 >> 172.21.15.169:0/2193389423 conn(0x7f44d00fad20 msgr2=0x7f44d0101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:47.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.669+0000 7f44cffff640 1 -- 172.21.15.169:0/2193389423 shutdown_connections 2024-03-28T01:00:47.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:47.669+0000 7f44cffff640 1 -- 172.21.15.169:0/2193389423 wait complete. 2024-03-28T01:00:47.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:47 smithi169 ceph-mon[32274]: pgmap v479: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 787 MiB/s wr, 1.31k op/s 2024-03-28T01:00:47.826 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:00:47.826 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:00:48.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.304+0000 7f3e2d20b640 1 -- 172.21.15.169:0/938028553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e2810d9e0 msgr2=0x7f3e2810de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:48.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.304+0000 7f3e2d20b640 1 --2- 172.21.15.169:0/938028553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e2810d9e0 0x7f3e2810de90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3e18009860 tx=0x7f3e1802bb90 comp rx=0 tx=0).stop 2024-03-28T01:00:48.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.304+0000 7f3e2d20b640 1 -- 172.21.15.169:0/938028553 shutdown_connections 2024-03-28T01:00:48.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.304+0000 7f3e2d20b640 1 --2- 172.21.15.169:0/938028553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e2810d9e0 0x7f3e2810de90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.304+0000 7f3e2d20b640 1 --2- 172.21.15.169:0/938028553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e2810b010 0x7f3e2810d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.304+0000 7f3e2d20b640 1 --2- 172.21.15.169:0/938028553 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e281086a0 0x7f3e2810aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.304+0000 7f3e2d20b640 1 -- 172.21.15.169:0/938028553 >> 172.21.15.169:0/938028553 conn(0x7f3e2806eb30 msgr2=0x7f3e28070f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:48.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.305+0000 7f3e2d20b640 1 -- 172.21.15.169:0/938028553 shutdown_connections 2024-03-28T01:00:48.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.305+0000 7f3e2d20b640 1 -- 172.21.15.169:0/938028553 wait complete. 2024-03-28T01:00:48.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.306+0000 7f3e2d20b640 1 Processor -- start 2024-03-28T01:00:48.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.308+0000 7f3e2d20b640 1 -- start start 2024-03-28T01:00:48.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.308+0000 7f3e2d20b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e281086a0 0x7f3e28116cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:48.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.309+0000 7f3e2d20b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e2810b010 0x7f3e281171f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:48.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.309+0000 7f3e2d20b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e2810d9e0 0x7f3e281b6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:48.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.309+0000 7f3e2d20b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2811c190 con 0x7f3e2810d9e0 2024-03-28T01:00:48.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.309+0000 7f3e2d20b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2811c010 con 0x7f3e2810b010 2024-03-28T01:00:48.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.309+0000 7f3e2d20b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2811c310 con 0x7f3e281086a0 2024-03-28T01:00:48.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.309+0000 7f3e26d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e281086a0 0x7f3e28116cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:48.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.309+0000 7f3e26d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e281086a0 0x7f3e28116cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:52802/0 (socket says 172.21.15.169:52802) 2024-03-28T01:00:48.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.309+0000 7f3e26d9d640 1 -- 172.21.15.169:0/2008122593 learned_addr learned my addr 172.21.15.169:0/2008122593 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:48.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.311+0000 7f3e2659c640 1 --2- 172.21.15.169:0/2008122593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e2810b010 0x7f3e281171f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:48.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.311+0000 7f3e2759e640 1 --2- 172.21.15.169:0/2008122593 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e2810d9e0 0x7f3e281b6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:48.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.312+0000 7f3e26d9d640 1 -- 172.21.15.169:0/2008122593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e2810b010 msgr2=0x7f3e281171f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:48.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.312+0000 7f3e26d9d640 1 --2- 172.21.15.169:0/2008122593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e2810b010 0x7f3e281171f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.312+0000 7f3e26d9d640 1 -- 172.21.15.169:0/2008122593 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e2810d9e0 msgr2=0x7f3e281b6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:48.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.312+0000 7f3e26d9d640 1 --2- 172.21.15.169:0/2008122593 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e2810d9e0 0x7f3e281b6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.312+0000 7f3e26d9d640 1 -- 172.21.15.169:0/2008122593 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2811a370 con 0x7f3e281086a0 2024-03-28T01:00:48.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.313+0000 7f3e26d9d640 1 --2- 172.21.15.169:0/2008122593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e281086a0 0x7f3e28116cb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c00e970 tx=0x7f3e1c00ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:48.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.315+0000 7f3e0ffff640 1 -- 172.21.15.169:0/2008122593 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c00cd80 con 0x7f3e281086a0 2024-03-28T01:00:48.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.315+0000 7f3e0ffff640 1 -- 172.21.15.169:0/2008122593 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e1c005500 con 0x7f3e281086a0 2024-03-28T01:00:48.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.315+0000 7f3e0ffff640 1 -- 172.21.15.169:0/2008122593 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c0326a0 con 0x7f3e281086a0 2024-03-28T01:00:48.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.317+0000 7f3e2d20b640 1 -- 172.21.15.169:0/2008122593 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e28073930 con 0x7f3e281086a0 2024-03-28T01:00:48.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.317+0000 7f3e2d20b640 1 -- 172.21.15.169:0/2008122593 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e28073da0 con 0x7f3e281086a0 2024-03-28T01:00:48.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.321+0000 7f3e0ffff640 1 -- 172.21.15.169:0/2008122593 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3e1c0046c0 con 0x7f3e281086a0 2024-03-28T01:00:48.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.331+0000 7f3e2fdf3740 1 Processor -- start 2024-03-28T01:00:48.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.331+0000 7f3e2fdf3740 1 -- start start 2024-03-28T01:00:48.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.332+0000 7f3e0ffff640 1 --2- 172.21.15.169:0/2008122593 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3e100a2a00 0x7f3e100a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:48.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.332+0000 7f3e2fdf3740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60afda0 0x55a6d60b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:48.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.332+0000 7f3e2fdf3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a6d60b0790 0x55a6d60b5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:48.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.332+0000 7f3e2759e640 1 --2- 172.21.15.169:0/2008122593 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3e100a2a00 0x7f3e100a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:48.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.332+0000 7f3e2fdf3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6d60b5900 0x55a6d60b7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:48.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.332+0000 7f3e2fdf3740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d60153d0 con 0x55a6d60afda0 2024-03-28T01:00:48.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.332+0000 7f3e26d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60afda0 0x55a6d60b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:48.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.332+0000 7f3e2fdf3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d5fbdad0 con 0x55a6d60b0790 2024-03-28T01:00:48.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.332+0000 7f3e2fdf3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d6011470 con 0x55a6d60b5900 2024-03-28T01:00:48.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.332+0000 7f3e26d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60afda0 0x55a6d60b01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:58390/0 (socket says 172.21.15.169:58390) 2024-03-28T01:00:48.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.332+0000 7f3e26d9d640 1 -- 172.21.15.169:0/3324402632 learned_addr learned my addr 172.21.15.169:0/3324402632 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:48.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.332+0000 7f3e0ffff640 1 -- 172.21.15.169:0/2008122593 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f3e1c0f1270 con 0x7f3e281086a0 2024-03-28T01:00:48.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.333+0000 7f3e2659c640 1 --2- 172.21.15.169:0/3324402632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a6d60b0790 0x55a6d60b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:48.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.333+0000 7f3e2759e640 1 --2- 172.21.15.169:0/3324402632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6d60b5900 0x55a6d60b7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:48.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.333+0000 7f3e26d9d640 1 -- 172.21.15.169:0/3324402632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6d60b5900 msgr2=0x55a6d60b7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:48.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.333+0000 7f3e26d9d640 1 --2- 172.21.15.169:0/3324402632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6d60b5900 0x55a6d60b7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.333+0000 7f3e26d9d640 1 -- 172.21.15.169:0/3324402632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a6d60b0790 msgr2=0x55a6d60b5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:48.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.333+0000 7f3e26d9d640 1 --2- 172.21.15.169:0/3324402632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a6d60b0790 0x55a6d60b5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.333+0000 7f3e26d9d640 1 -- 172.21.15.169:0/3324402632 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6d5df37d0 con 0x55a6d60afda0 2024-03-28T01:00:48.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.333+0000 7f3e2759e640 1 --2- 172.21.15.169:0/2008122593 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3e100a2a00 0x7f3e100a4ec0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3e18037040 tx=0x7f3e18005b20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:48.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.333+0000 7f3e26d9d640 1 --2- 172.21.15.169:0/3324402632 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60afda0 0x55a6d60b01c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c0bd7a0 tx=0x7f3e1c0bd5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d1e5499700f6064 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:48.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.334+0000 7f3e0d7fa640 1 -- 172.21.15.169:0/3324402632 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c0bdf10 con 0x55a6d60afda0 2024-03-28T01:00:48.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.334+0000 7f3e0d7fa640 1 -- 172.21.15.169:0/3324402632 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e1c0bd810 con 0x55a6d60afda0 2024-03-28T01:00:48.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.334+0000 7f3e0d7fa640 1 -- 172.21.15.169:0/3324402632 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c0b07e0 con 0x55a6d60afda0 2024-03-28T01:00:48.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.335+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/3324402632 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60afda0 msgr2=0x55a6d60b01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:48.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.335+0000 7f3e2fdf3740 1 --2- 172.21.15.169:0/3324402632 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60afda0 0x55a6d60b01c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c0bd7a0 tx=0x7f3e1c0bd5e0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.336+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/3324402632 shutdown_connections 2024-03-28T01:00:48.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.336+0000 7f3e2fdf3740 1 --2- 172.21.15.169:0/3324402632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6d60b5900 0x55a6d60b7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.336+0000 7f3e2fdf3740 1 --2- 172.21.15.169:0/3324402632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a6d60b0790 0x55a6d60b5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.336+0000 7f3e2fdf3740 1 --2- 172.21.15.169:0/3324402632 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60afda0 0x55a6d60b01c0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.336+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/3324402632 >> 172.21.15.169:0/3324402632 conn(0x55a6d60ab310 msgr2=0x55a6d60ab710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.336+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/3324402632 shutdown_connections 2024-03-28T01:00:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.336+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/3324402632 wait complete. 2024-03-28T01:00:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.337+0000 7f3e2fdf3740 1 -- start start 2024-03-28T01:00:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.337+0000 7f3e2fdf3740 1 Processor -- start 2024-03-28T01:00:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.337+0000 7f3e2fdf3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a6d60afda0 0x55a6d60ba010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.337+0000 7f3e2fdf3740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60b0790 0x55a6d60ba550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.337+0000 7f3e2fdf3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6d60b5900 0x55a6d60c1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.338+0000 7f3e2fdf3740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d60b87f0 con 0x55a6d60b0790 2024-03-28T01:00:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.338+0000 7f3e2fdf3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d5fd1060 con 0x55a6d60afda0 2024-03-28T01:00:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.338+0000 7f3e2fdf3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d60b8970 con 0x55a6d60b5900 2024-03-28T01:00:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.338+0000 7f3e2fdf3740 10 client.0 _ll_register_callbacks cb 0x55a6d5fb36a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.338+0000 7f3e2659c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60b0790 0x55a6d60ba550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.338+0000 7f3e2659c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60b0790 0x55a6d60ba550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:58396/0 (socket says 172.21.15.169:58396) 2024-03-28T01:00:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.338+0000 7f3e2659c640 1 -- 172.21.15.169:0/654715505 learned_addr learned my addr 172.21.15.169:0/654715505 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.339+0000 7f3e2659c640 1 -- 172.21.15.169:0/654715505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6d60b5900 msgr2=0x55a6d60c1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.339+0000 7f3e2659c640 1 --2- 172.21.15.169:0/654715505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6d60b5900 0x55a6d60c1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.339+0000 7f3e2659c640 1 -- 172.21.15.169:0/654715505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a6d60afda0 msgr2=0x55a6d60ba010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.339+0000 7f3e2659c640 1 --2- 172.21.15.169:0/654715505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a6d60afda0 0x55a6d60ba010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.339+0000 7f3e2659c640 1 -- 172.21.15.169:0/654715505 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6d5d293f0 con 0x55a6d60b0790 2024-03-28T01:00:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.340+0000 7f3e2659c640 1 --2- 172.21.15.169:0/654715505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60b0790 0x55a6d60ba550 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f3e20010b40 tx=0x7f3e20007f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4769f6f5de2eb08 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.340+0000 7f3e0aff5640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:00:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.340+0000 7f3e0aff5640 1 -- 172.21.15.169:0/654715505 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2000fdd0 con 0x55a6d60b0790 2024-03-28T01:00:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.340+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/654715505 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3e1c0bd810 con 0x55a6d60b0790 2024-03-28T01:00:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.340+0000 7f3e0aff5640 1 -- 172.21.15.169:0/654715505 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f3e20003970 con 0x55a6d60b0790 2024-03-28T01:00:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.341+0000 7f3e0aff5640 1 -- 172.21.15.169:0/654715505 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e20023050 con 0x55a6d60b0790 2024-03-28T01:00:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.341+0000 7f3e2fdf3740 10 client.16332 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:48.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.341+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/654715505 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a6d5ce90f0 con 0x55a6d60b0790 2024-03-28T01:00:48.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.341+0000 7f3e0aff5640 1 -- 172.21.15.169:0/654715505 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e20003b70 con 0x55a6d60b0790 2024-03-28T01:00:48.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.341+0000 7f3e0aff5640 1 -- 172.21.15.169:0/654715505 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f3e2001a620 con 0x55a6d60b0790 2024-03-28T01:00:48.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.341+0000 7f3e2fdf3740 10 client.16332 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:48.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.342+0000 7f3e2fdf3740 10 client.16332 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:48.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.342+0000 7f3e2fdf3740 20 client.16332 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:48.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.342+0000 7f3e2fdf3740 1 --2- 172.21.15.169:0/654715505 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a6d60c6020 0x55a6d60c8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:48.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.342+0000 7f3e2fdf3740 4 client.16332 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:00:48.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.342+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/654715505 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e1c0bd810 con 0x55a6d60c6020 2024-03-28T01:00:48.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.343+0000 7f3e26d9d640 1 --2- 172.21.15.169:0/654715505 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a6d60c6020 0x55a6d60c8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:48.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.347+0000 7f3e26d9d640 1 --2- 172.21.15.169:0/654715505 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a6d60c6020 0x55a6d60c8480 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fa3b6a7ebcd33db server_cookie=a09607e7b16c7154 in_seq=0 out_seq=0 2024-03-28T01:00:48.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.348+0000 7f3e0aff5640 10 client.16332 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:48.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.349+0000 7f3e0aff5640 1 -- 172.21.15.169:0/654715505 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7f3e1c0bd810 con 0x55a6d60c6020 2024-03-28T01:00:48.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.349+0000 7f3e0aff5640 10 client.16332 handle_command_reply: tid=0 2024-03-28T01:00:48.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.349+0000 7f3e2fdf3740 1 client.16332 shutdown 2024-03-28T01:00:48.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.353+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/654715505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60b0790 msgr2=0x55a6d60ba550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:48.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.353+0000 7f3e2fdf3740 1 --2- 172.21.15.169:0/654715505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60b0790 0x55a6d60ba550 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f3e20010b40 tx=0x7f3e20007f80 comp rx=0 tx=0).stop 2024-03-28T01:00:48.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.359+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/654715505 shutdown_connections 2024-03-28T01:00:48.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.359+0000 7f3e2fdf3740 1 --2- 172.21.15.169:0/654715505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6d60b5900 0x55a6d60c1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.359+0000 7f3e2fdf3740 1 --2- 172.21.15.169:0/654715505 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a6d60c6020 0x55a6d60c8480 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.359+0000 7f3e2fdf3740 1 --2- 172.21.15.169:0/654715505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60b0790 0x55a6d60ba550 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.359+0000 7f3e2fdf3740 1 --2- 172.21.15.169:0/654715505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a6d60afda0 0x55a6d60ba010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.359+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/654715505 >> 172.21.15.169:0/654715505 conn(0x55a6d60ab310 msgr2=0x55a6d60aa530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:48.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.360+0000 7f3e0aff5640 0 client.16332 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:48.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.370+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/654715505 shutdown_connections 2024-03-28T01:00:48.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.370+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/654715505 wait complete. 2024-03-28T01:00:48.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.370+0000 7f3e2fdf3740 20 client.16332 trim_cache size 0 max 16384 2024-03-28T01:00:48.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.469+0000 7f3e2fdf3740 1 Processor -- start 2024-03-28T01:00:48.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.469+0000 7f3e2fdf3740 1 -- start start 2024-03-28T01:00:48.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.470+0000 7f3e2fdf3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6d60afda0 0x55a6d60a9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:48.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.470+0000 7f3e2fdf3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a6d60b0790 0x55a6d60c1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:48.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.470+0000 7f3e2fdf3740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60b5900 0x55a6d60aa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:48.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.471+0000 7f3e2fdf3740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e280807d0 con 0x55a6d60b5900 2024-03-28T01:00:48.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.471+0000 7f3e2fdf3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2807e2d0 con 0x55a6d60b0790 2024-03-28T01:00:48.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.471+0000 7f3e2fdf3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2807c3c0 con 0x55a6d60afda0 2024-03-28T01:00:48.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.472+0000 7f3e26d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6d60afda0 0x55a6d60a9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:48.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.472+0000 7f3e26d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6d60afda0 0x55a6d60a9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:52832/0 (socket says 172.21.15.169:52832) 2024-03-28T01:00:48.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.472+0000 7f3e26d9d640 1 -- 172.21.15.169:0/2890520656 learned_addr learned my addr 172.21.15.169:0/2890520656 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:48.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.473+0000 7f3e2659c640 1 --2- 172.21.15.169:0/2890520656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a6d60b0790 0x55a6d60c1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:48.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.474+0000 7f3e26d9d640 1 -- 172.21.15.169:0/2890520656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a6d60b0790 msgr2=0x55a6d60c1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:48.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.474+0000 7f3e26d9d640 1 --2- 172.21.15.169:0/2890520656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a6d60b0790 0x55a6d60c1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.474+0000 7f3e26d9d640 1 -- 172.21.15.169:0/2890520656 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60b5900 msgr2=0x55a6d60aa1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:48.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.474+0000 7f3e26d9d640 1 --2- 172.21.15.169:0/2890520656 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60b5900 0x55a6d60aa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.474+0000 7f3e26d9d640 1 -- 172.21.15.169:0/2890520656 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6d60ac080 con 0x55a6d60afda0 2024-03-28T01:00:48.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.475+0000 7f3e26d9d640 1 --2- 172.21.15.169:0/2890520656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6d60afda0 0x55a6d60a9ca0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c00c6a0 tx=0x7f3e1c0f0040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce6e1d7404ea43c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:48.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.492+0000 7f3e0cff9640 1 -- 172.21.15.169:0/2890520656 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c0db070 con 0x55a6d60afda0 2024-03-28T01:00:48.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.493+0000 7f3e0cff9640 1 -- 172.21.15.169:0/2890520656 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e1c0bdf10 con 0x55a6d60afda0 2024-03-28T01:00:48.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.493+0000 7f3e0cff9640 1 -- 172.21.15.169:0/2890520656 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c0beb40 con 0x55a6d60afda0 2024-03-28T01:00:48.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.496+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/2890520656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6d60afda0 msgr2=0x55a6d60a9ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:48.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.496+0000 7f3e2fdf3740 1 --2- 172.21.15.169:0/2890520656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6d60afda0 0x55a6d60a9ca0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c00c6a0 tx=0x7f3e1c0f0040 comp rx=0 tx=0).stop 2024-03-28T01:00:48.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.496+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/2890520656 shutdown_connections 2024-03-28T01:00:48.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.496+0000 7f3e2fdf3740 1 --2- 172.21.15.169:0/2890520656 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60b5900 0x55a6d60aa1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.496+0000 7f3e2fdf3740 1 --2- 172.21.15.169:0/2890520656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a6d60b0790 0x55a6d60c1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.496+0000 7f3e2fdf3740 1 --2- 172.21.15.169:0/2890520656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6d60afda0 0x55a6d60a9ca0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.496+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/2890520656 >> 172.21.15.169:0/2890520656 conn(0x55a6d60ab310 msgr2=0x55a6d60b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:48.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.497+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/2890520656 shutdown_connections 2024-03-28T01:00:48.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.497+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/2890520656 wait complete. 2024-03-28T01:00:48.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.497+0000 7f3e2fdf3740 1 -- start start 2024-03-28T01:00:48.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.497+0000 7f3e2fdf3740 1 Processor -- start 2024-03-28T01:00:48.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.498+0000 7f3e2fdf3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a6d60afda0 0x55a6d60bef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:48.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.498+0000 7f3e2fdf3740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60b0790 0x55a6d60bf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:48.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.498+0000 7f3e2fdf3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6d60b5900 0x55a6d60c04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:48.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.498+0000 7f3e2fdf3740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2807c990 con 0x55a6d60b0790 2024-03-28T01:00:48.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.498+0000 7f3e2fdf3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2807dc70 con 0x55a6d60afda0 2024-03-28T01:00:48.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.498+0000 7f3e2fdf3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2807cfb0 con 0x55a6d60b5900 2024-03-28T01:00:48.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.498+0000 7f3e2fdf3740 10 client.0 _ll_register_callbacks cb 0x7f3e20007af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:48.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.499+0000 7f3e2659c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60b0790 0x55a6d60bf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:48.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.499+0000 7f3e2659c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60b0790 0x55a6d60bf4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:58422/0 (socket says 172.21.15.169:58422) 2024-03-28T01:00:48.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.499+0000 7f3e2659c640 1 -- 172.21.15.169:0/2689205180 learned_addr learned my addr 172.21.15.169:0/2689205180 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:48.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.501+0000 7f3e2759e640 1 --2- 172.21.15.169:0/2689205180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6d60b5900 0x55a6d60c04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:48.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.502+0000 7f3e26d9d640 1 --2- 172.21.15.169:0/2689205180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a6d60afda0 0x55a6d60bef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:48.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.505+0000 7f3e26d9d640 1 -- 172.21.15.169:0/2689205180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6d60b5900 msgr2=0x55a6d60c04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:48.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.505+0000 7f3e26d9d640 1 --2- 172.21.15.169:0/2689205180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6d60b5900 0x55a6d60c04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.505+0000 7f3e26d9d640 1 -- 172.21.15.169:0/2689205180 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60b0790 msgr2=0x55a6d60bf4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:48.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.505+0000 7f3e26d9d640 1 --2- 172.21.15.169:0/2689205180 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60b0790 0x55a6d60bf4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.505+0000 7f3e26d9d640 1 -- 172.21.15.169:0/2689205180 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6d60c0ac0 con 0x55a6d60afda0 2024-03-28T01:00:48.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.510+0000 7f3e26d9d640 1 --2- 172.21.15.169:0/2689205180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a6d60afda0 0x55a6d60bef80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c00c6a0 tx=0x7f3e1c004aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=252934f954056464 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:48.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.510+0000 7f3e0aff5640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:00:48.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.511+0000 7f3e0aff5640 1 -- 172.21.15.169:0/2689205180 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c0db040 con 0x55a6d60afda0 2024-03-28T01:00:48.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.511+0000 7f3e0aff5640 1 -- 172.21.15.169:0/2689205180 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f3e1c0beb40 con 0x55a6d60afda0 2024-03-28T01:00:48.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.513+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/2689205180 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3e1c0bdf10 con 0x55a6d60afda0 2024-03-28T01:00:48.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.516+0000 7f3e0aff5640 1 -- 172.21.15.169:0/2689205180 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e1c0b5050 con 0x55a6d60afda0 2024-03-28T01:00:48.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.522+0000 7f3e2fdf3740 10 client.26216 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:48.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.522+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/2689205180 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a6d6121c50 con 0x55a6d60afda0 2024-03-28T01:00:48.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.524+0000 7f3e0aff5640 1 -- 172.21.15.169:0/2689205180 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e1c0bdf10 con 0x55a6d60afda0 2024-03-28T01:00:48.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.524+0000 7f3e0aff5640 1 -- 172.21.15.169:0/2689205180 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f3e1c0c9080 con 0x55a6d60afda0 2024-03-28T01:00:48.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.524+0000 7f3e2fdf3740 10 client.26216 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:48.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.524+0000 7f3e2fdf3740 10 client.26216 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:48.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.524+0000 7f3e2fdf3740 20 client.26216 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:48.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.524+0000 7f3e2fdf3740 1 --2- 172.21.15.169:0/2689205180 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a6d61222f0 0x55a6d61246e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:48.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.524+0000 7f3e2fdf3740 4 client.26216 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:00:48.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.524+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/2689205180 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a6d6022910 con 0x55a6d61222f0 2024-03-28T01:00:48.531 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.528+0000 7f3e2659c640 1 --2- 172.21.15.169:0/2689205180 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a6d61222f0 0x55a6d61246e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:48.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.542+0000 7f3e2659c640 1 --2- 172.21.15.169:0/2689205180 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a6d61222f0 0x55a6d61246e0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf9b2fc2cc4f6919 server_cookie=5f2e65e7419053a6 in_seq=0 out_seq=0 2024-03-28T01:00:48.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.542+0000 7f3e0aff5640 10 client.26216 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:48.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.547+0000 7f3e0aff5640 1 -- 172.21.15.169:0/2689205180 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+266 (crc 0 0 0) 0x55a6d6022910 con 0x55a6d61222f0 2024-03-28T01:00:48.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.547+0000 7f3e0aff5640 10 client.26216 handle_command_reply: tid=0 2024-03-28T01:00:48.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.547+0000 7f3e2fdf3740 1 client.26216 shutdown 2024-03-28T01:00:48.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.548+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/2689205180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a6d60afda0 msgr2=0x55a6d60bef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:48.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.548+0000 7f3e2fdf3740 1 --2- 172.21.15.169:0/2689205180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a6d60afda0 0x55a6d60bef80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c00c6a0 tx=0x7f3e1c004aa0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.549+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/2689205180 shutdown_connections 2024-03-28T01:00:48.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.549+0000 7f3e2fdf3740 1 --2- 172.21.15.169:0/2689205180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6d60b5900 0x55a6d60c04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.549+0000 7f3e2fdf3740 1 --2- 172.21.15.169:0/2689205180 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a6d61222f0 0x55a6d61246e0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.549+0000 7f3e2fdf3740 1 --2- 172.21.15.169:0/2689205180 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a6d60b0790 0x55a6d60bf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.549+0000 7f3e2fdf3740 1 --2- 172.21.15.169:0/2689205180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a6d60afda0 0x55a6d60bef80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.549+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/2689205180 >> 172.21.15.169:0/2689205180 conn(0x55a6d60ab310 msgr2=0x55a6d60ae570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:48.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.550+0000 7f3e0aff5640 0 client.26216 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:48.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.550+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/2689205180 shutdown_connections 2024-03-28T01:00:48.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.554+0000 7f3e2fdf3740 1 -- 172.21.15.169:0/2689205180 wait complete. 2024-03-28T01:00:48.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.555+0000 7f3e2fdf3740 20 client.26216 trim_cache size 0 max 16384 2024-03-28T01:00:48.565 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:00:48.566 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "scrub active (15 inodes in the stack)", 2024-03-28T01:00:48.566 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": { 2024-03-28T01:00:48.566 INFO:teuthology.orchestra.run.smithi169.stdout: "d6f9488b-8563-4e0d-aba7-2ee84b2ad59c": { 2024-03-28T01:00:48.566 INFO:teuthology.orchestra.run.smithi169.stdout: "path": "/", 2024-03-28T01:00:48.566 INFO:teuthology.orchestra.run.smithi169.stdout: "tag": "d6f9488b-8563-4e0d-aba7-2ee84b2ad59c", 2024-03-28T01:00:48.566 INFO:teuthology.orchestra.run.smithi169.stdout: "options": "recursive,force" 2024-03-28T01:00:48.566 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:00:48.566 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:00:48.566 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:00:48.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.568+0000 7f3e2d20b640 1 -- 172.21.15.169:0/2008122593 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3e100a2a00 msgr2=0x7f3e100a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:48.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.568+0000 7f3e2d20b640 1 --2- 172.21.15.169:0/2008122593 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3e100a2a00 0x7f3e100a4ec0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3e18037040 tx=0x7f3e18005b20 comp rx=0 tx=0).stop 2024-03-28T01:00:48.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.568+0000 7f3e2d20b640 1 -- 172.21.15.169:0/2008122593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e281086a0 msgr2=0x7f3e28116cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:48.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.568+0000 7f3e2d20b640 1 --2- 172.21.15.169:0/2008122593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e281086a0 0x7f3e28116cb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c00e970 tx=0x7f3e1c00ee30 comp rx=0 tx=0).stop 2024-03-28T01:00:48.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.568+0000 7f3e2d20b640 1 -- 172.21.15.169:0/2008122593 shutdown_connections 2024-03-28T01:00:48.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.568+0000 7f3e2d20b640 1 --2- 172.21.15.169:0/2008122593 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3e100a2a00 0x7f3e100a4ec0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.569+0000 7f3e2d20b640 1 --2- 172.21.15.169:0/2008122593 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e2810d9e0 0x7f3e281b6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.569+0000 7f3e2d20b640 1 --2- 172.21.15.169:0/2008122593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e2810b010 0x7f3e281171f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.569+0000 7f3e2d20b640 1 --2- 172.21.15.169:0/2008122593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e281086a0 0x7f3e28116cb0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:48.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.569+0000 7f3e2d20b640 1 -- 172.21.15.169:0/2008122593 >> 172.21.15.169:0/2008122593 conn(0x7f3e2806eb30 msgr2=0x7f3e2810b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:48.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.570+0000 7f3e2d20b640 1 -- 172.21.15.169:0/2008122593 shutdown_connections 2024-03-28T01:00:48.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:00:48.570+0000 7f3e2d20b640 1 -- 172.21.15.169:0/2008122593 wait complete. 2024-03-28T01:00:48.615 INFO:tasks.cephfs.filesystem:scrub status for tag:d6f9488b-8563-4e0d-aba7-2ee84b2ad59c - {'path': '/', 'tag': 'd6f9488b-8563-4e0d-aba7-2ee84b2ad59c', 'options': 'recursive,force'} 2024-03-28T01:00:48.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:48 smithi177 ceph-mon[36206]: scrub queued for path: / 2024-03-28T01:00:48.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:48 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:00:48.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:48 smithi177 ceph-mon[36206]: scrub summary: active paths [/] 2024-03-28T01:00:48.809 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:48 smithi169 ceph-mon[32274]: scrub queued for path: / 2024-03-28T01:00:48.809 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:48 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:00:48.809 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:48 smithi169 ceph-mon[32274]: scrub summary: active paths [/] 2024-03-28T01:00:48.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:48 smithi174 ceph-mon[36060]: scrub queued for path: / 2024-03-28T01:00:48.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:48 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:00:48.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:48 smithi174 ceph-mon[36060]: scrub summary: active paths [/] 2024-03-28T01:00:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.483+0000 7fe79e99d640 1 -- 172.21.15.169:0/1953114741 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7980740a0 msgr2=0x7fe798074500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:49.488 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.483+0000 7fe79e99d640 1 --2- 172.21.15.169:0/1953114741 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7980740a0 0x7fe798074500 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe78c009900 tx=0x7fe78c02bcd0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.488 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.483+0000 7fe79e99d640 1 -- 172.21.15.169:0/1953114741 shutdown_connections 2024-03-28T01:00:49.488 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.483+0000 7fe79e99d640 1 --2- 172.21.15.169:0/1953114741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe798109580 0x7fe79810ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.488 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.483+0000 7fe79e99d640 1 --2- 172.21.15.169:0/1953114741 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7980740a0 0x7fe798074500 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.488 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.483+0000 7fe79e99d640 1 --2- 172.21.15.169:0/1953114741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe798075680 0x7fe798073ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.488 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.483+0000 7fe79e99d640 1 -- 172.21.15.169:0/1953114741 >> 172.21.15.169:0/1953114741 conn(0x7fe7980fcc60 msgr2=0x7fe7980ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:49.488 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.483+0000 7fe79e99d640 1 -- 172.21.15.169:0/1953114741 shutdown_connections 2024-03-28T01:00:49.488 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.483+0000 7fe79e99d640 1 -- 172.21.15.169:0/1953114741 wait complete. 2024-03-28T01:00:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.484+0000 7fe79e99d640 1 Processor -- start 2024-03-28T01:00:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.484+0000 7fe79e99d640 1 -- start start 2024-03-28T01:00:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.485+0000 7fe79e99d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7980740a0 0x7fe798199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.485+0000 7fe79e99d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe798075680 0x7fe79819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.485+0000 7fe79e99d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe798109580 0x7fe7981a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.485+0000 7fe79e99d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79810f570 con 0x7fe798075680 2024-03-28T01:00:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.485+0000 7fe79e99d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79810f3f0 con 0x7fe7980740a0 2024-03-28T01:00:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.485+0000 7fe79e99d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79810f6f0 con 0x7fe798109580 2024-03-28T01:00:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.485+0000 7fe7977fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe798075680 0x7fe79819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.485+0000 7fe7977fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe798075680 0x7fe79819a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:58444/0 (socket says 172.21.15.169:58444) 2024-03-28T01:00:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.485+0000 7fe7977fe640 1 -- 172.21.15.169:0/1387195473 learned_addr learned my addr 172.21.15.169:0/1387195473 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.485+0000 7fe79cf3a640 1 --2- 172.21.15.169:0/1387195473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe798109580 0x7fe7981a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.486+0000 7fe797fff640 1 --2- 172.21.15.169:0/1387195473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7980740a0 0x7fe798199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.486+0000 7fe79cf3a640 1 -- 172.21.15.169:0/1387195473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7980740a0 msgr2=0x7fe798199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.486+0000 7fe79cf3a640 1 --2- 172.21.15.169:0/1387195473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7980740a0 0x7fe798199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.486+0000 7fe79cf3a640 1 -- 172.21.15.169:0/1387195473 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe798075680 msgr2=0x7fe79819a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.486+0000 7fe79cf3a640 1 --2- 172.21.15.169:0/1387195473 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe798075680 0x7fe79819a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.486+0000 7fe79cf3a640 1 -- 172.21.15.169:0/1387195473 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7981a1750 con 0x7fe798109580 2024-03-28T01:00:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.486+0000 7fe79cf3a640 1 --2- 172.21.15.169:0/1387195473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe798109580 0x7fe7981a10e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe790009fd0 tx=0x7fe79000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.486+0000 7fe7957fa640 1 -- 172.21.15.169:0/1387195473 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79000ce20 con 0x7fe798109580 2024-03-28T01:00:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.487+0000 7fe7957fa640 1 -- 172.21.15.169:0/1387195473 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe790005020 con 0x7fe798109580 2024-03-28T01:00:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.487+0000 7fe7957fa640 1 -- 172.21.15.169:0/1387195473 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe790010720 con 0x7fe798109580 2024-03-28T01:00:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.487+0000 7fe79e99d640 1 -- 172.21.15.169:0/1387195473 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7981a1a40 con 0x7fe798109580 2024-03-28T01:00:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.487+0000 7fe79e99d640 1 -- 172.21.15.169:0/1387195473 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7981a1fb0 con 0x7fe798109580 2024-03-28T01:00:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.490+0000 7fe7957fa640 1 -- 172.21.15.169:0/1387195473 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe790005540 con 0x7fe798109580 2024-03-28T01:00:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.490+0000 7fe7a1581740 1 Processor -- start 2024-03-28T01:00:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.490+0000 7fe7a1581740 1 -- start start 2024-03-28T01:00:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.491+0000 7fe7a1581740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd2920 0x55fa7bcd2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.491+0000 7fe7a1581740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa7bcd3310 0x55fa7bcd7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.491+0000 7fe7a1581740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa7bcd8480 0x55fa7bcda910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.491+0000 7fe7a1581740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa7bc37b10 con 0x55fa7bcd2920 2024-03-28T01:00:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.491+0000 7fe7a1581740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa7bbe0900 con 0x55fa7bcd8480 2024-03-28T01:00:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.491+0000 7fe7a1581740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa7bc33bb0 con 0x55fa7bcd3310 2024-03-28T01:00:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.491+0000 7fe7977fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd2920 0x55fa7bcd2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.491+0000 7fe7977fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd2920 0x55fa7bcd2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:58452/0 (socket says 172.21.15.169:58452) 2024-03-28T01:00:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.491+0000 7fe7977fe640 1 -- 172.21.15.169:0/2395028777 learned_addr learned my addr 172.21.15.169:0/2395028777 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:49.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.491+0000 7fe797fff640 1 --2- 172.21.15.169:0/2395028777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa7bcd3310 0x55fa7bcd7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:49.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.491+0000 7fe7977fe640 1 -- 172.21.15.169:0/2395028777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa7bcd3310 msgr2=0x55fa7bcd7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:49.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.491+0000 7fe7977fe640 1 --2- 172.21.15.169:0/2395028777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa7bcd3310 0x55fa7bcd7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.491+0000 7fe7957fa640 1 --2- 172.21.15.169:0/1387195473 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe7800a2920 0x7fe7800a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:49.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.491+0000 7fe7977fe640 1 -- 172.21.15.169:0/2395028777 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa7bcd8480 msgr2=0x55fa7bcda910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:49.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.491+0000 7fe7977fe640 1 --2- 172.21.15.169:0/2395028777 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa7bcd8480 0x55fa7bcda910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.491+0000 7fe7977fe640 1 -- 172.21.15.169:0/2395028777 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa7b94c210 con 0x55fa7bcd2920 2024-03-28T01:00:49.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.491+0000 7fe79cf3a640 1 --2- 172.21.15.169:0/1387195473 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe7800a2920 0x7fe7800a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:49.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.491+0000 7fe7977fe640 1 --2- 172.21.15.169:0/2395028777 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd2920 0x55fa7bcd2d40 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe78c005c40 tx=0x7fe78c02c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddb5ae6e0eca6b5a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:49.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.492+0000 7fe77e7fc640 1 -- 172.21.15.169:0/2395028777 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78c044020 con 0x55fa7bcd2920 2024-03-28T01:00:49.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.492+0000 7fe77e7fc640 1 -- 172.21.15.169:0/2395028777 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe78c02cc20 con 0x55fa7bcd2920 2024-03-28T01:00:49.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.492+0000 7fe77e7fc640 1 -- 172.21.15.169:0/2395028777 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78c005150 con 0x55fa7bcd2920 2024-03-28T01:00:49.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.492+0000 7fe7957fa640 1 -- 172.21.15.169:0/1387195473 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fe7900cf3f0 con 0x7fe798109580 2024-03-28T01:00:49.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.493+0000 7fe79cf3a640 1 --2- 172.21.15.169:0/1387195473 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe7800a2920 0x7fe7800a4de0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe790084fc0 tx=0x7fe790096450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:49.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.493+0000 7fe7a1581740 1 -- 172.21.15.169:0/2395028777 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd2920 msgr2=0x55fa7bcd2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:49.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.493+0000 7fe7a1581740 1 --2- 172.21.15.169:0/2395028777 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd2920 0x55fa7bcd2d40 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe78c005c40 tx=0x7fe78c02c770 comp rx=0 tx=0).stop 2024-03-28T01:00:49.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.493+0000 7fe7a1581740 1 -- 172.21.15.169:0/2395028777 shutdown_connections 2024-03-28T01:00:49.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.493+0000 7fe7a1581740 1 --2- 172.21.15.169:0/2395028777 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa7bcd8480 0x55fa7bcda910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.493+0000 7fe7a1581740 1 --2- 172.21.15.169:0/2395028777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa7bcd3310 0x55fa7bcd7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.493+0000 7fe7a1581740 1 --2- 172.21.15.169:0/2395028777 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd2920 0x55fa7bcd2d40 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.493+0000 7fe7a1581740 1 -- 172.21.15.169:0/2395028777 >> 172.21.15.169:0/2395028777 conn(0x55fa7bccde90 msgr2=0x55fa7bcce290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:49.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.493+0000 7fe7a1581740 1 -- 172.21.15.169:0/2395028777 shutdown_connections 2024-03-28T01:00:49.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.493+0000 7fe7a1581740 1 -- 172.21.15.169:0/2395028777 wait complete. 2024-03-28T01:00:49.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.493+0000 7fe7a1581740 1 -- start start 2024-03-28T01:00:49.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.494+0000 7fe7a1581740 1 Processor -- start 2024-03-28T01:00:49.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.494+0000 7fe7a1581740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa7bcd2920 0x55fa7bcdccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:49.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.494+0000 7fe7a1581740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa7bcd3310 0x55fa7bcdd230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:49.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.494+0000 7fe7a1581740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd8480 0x55fa7bce4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:49.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.494+0000 7fe7a1581740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa7bcdb3d0 con 0x55fa7bcd8480 2024-03-28T01:00:49.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.494+0000 7fe7a1581740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa7bbf3d50 con 0x55fa7bcd2920 2024-03-28T01:00:49.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.494+0000 7fe7a1581740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa7bcdb550 con 0x55fa7bcd3310 2024-03-28T01:00:49.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.494+0000 7fe7a1581740 10 client.0 _ll_register_callbacks cb 0x55fa7bbf3720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:49.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.495+0000 7fe7977fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa7bcd2920 0x55fa7bcdccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:49.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.495+0000 7fe7977fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd8480 0x55fa7bce4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:49.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.495+0000 7fe7977fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd8480 0x55fa7bce4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:58454/0 (socket says 172.21.15.169:58454) 2024-03-28T01:00:49.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.495+0000 7fe7977fe640 1 -- 172.21.15.169:0/2262218831 learned_addr learned my addr 172.21.15.169:0/2262218831 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:49.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.495+0000 7fe797fff640 1 --2- 172.21.15.169:0/2262218831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa7bcd3310 0x55fa7bcdd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:49.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.495+0000 7fe797fff640 1 -- 172.21.15.169:0/2262218831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa7bcd2920 msgr2=0x55fa7bcdccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:49.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.495+0000 7fe797fff640 1 --2- 172.21.15.169:0/2262218831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa7bcd2920 0x55fa7bcdccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.495+0000 7fe797fff640 1 -- 172.21.15.169:0/2262218831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd8480 msgr2=0x55fa7bce4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:49.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.495+0000 7fe797fff640 1 --2- 172.21.15.169:0/2262218831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd8480 0x55fa7bce4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.495+0000 7fe797fff640 1 -- 172.21.15.169:0/2262218831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fa7ba18620 con 0x55fa7bcd3310 2024-03-28T01:00:49.502 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.498+0000 7fe797fff640 1 --2- 172.21.15.169:0/2262218831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa7bcd3310 0x55fa7bcdd230 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x55fa7bcd3710 tx=0x7fe788013b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69eb21d57049bd1b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:49.503 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.498+0000 7fe7777fe640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:00:49.506 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.502+0000 7fe7777fe640 1 -- 172.21.15.169:0/2262218831 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe788011d40 con 0x55fa7bcd3310 2024-03-28T01:00:49.507 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.503+0000 7fe7a1581740 1 -- 172.21.15.169:0/2262218831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe78c02cc20 con 0x55fa7bcd3310 2024-03-28T01:00:49.507 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.503+0000 7fe7777fe640 1 -- 172.21.15.169:0/2262218831 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fe788004030 con 0x55fa7bcd3310 2024-03-28T01:00:49.508 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.505+0000 7fe7777fe640 1 -- 172.21.15.169:0/2262218831 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe788004270 con 0x55fa7bcd3310 2024-03-28T01:00:49.509 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.506+0000 7fe7a1581740 10 client.26158 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:49.510 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.506+0000 7fe7a1581740 1 -- 172.21.15.169:0/2262218831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55fa7bcefe50 con 0x55fa7bcd3310 2024-03-28T01:00:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.510+0000 7fe7777fe640 1 -- 172.21.15.169:0/2262218831 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe788004410 con 0x55fa7bcd3310 2024-03-28T01:00:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.510+0000 7fe7777fe640 1 -- 172.21.15.169:0/2262218831 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fe78801a070 con 0x55fa7bcd3310 2024-03-28T01:00:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.510+0000 7fe7a1581740 10 client.26158 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.510+0000 7fe7a1581740 10 client.26158 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.510+0000 7fe7a1581740 20 client.26158 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.510+0000 7fe7a1581740 1 --2- 172.21.15.169:0/2262218831 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55fa7bcf1a10 0x55fa7bcf3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.510+0000 7fe7a1581740 4 client.26158 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:00:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.510+0000 7fe7a1581740 1 -- 172.21.15.169:0/2262218831 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe78c02cc20 con 0x55fa7bcf1a10 2024-03-28T01:00:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.513+0000 7fe7977fe640 1 --2- 172.21.15.169:0/2262218831 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55fa7bcf1a10 0x55fa7bcf3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:49.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.520+0000 7fe7977fe640 1 --2- 172.21.15.169:0/2262218831 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55fa7bcf1a10 0x55fa7bcf3e30 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88184724c36e9531 server_cookie=fad0c967d76e4379 in_seq=0 out_seq=0 2024-03-28T01:00:49.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.520+0000 7fe7777fe640 10 client.26158 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.528+0000 7fe7777fe640 1 -- 172.21.15.169:0/2262218831 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7fe78c02cc20 con 0x55fa7bcf1a10 2024-03-28T01:00:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.528+0000 7fe7777fe640 10 client.26158 handle_command_reply: tid=0 2024-03-28T01:00:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.528+0000 7fe7a1581740 1 client.26158 shutdown 2024-03-28T01:00:49.535 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.532+0000 7fe7a1581740 1 -- 172.21.15.169:0/2262218831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa7bcd3310 msgr2=0x55fa7bcdd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:49.536 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.532+0000 7fe7a1581740 1 --2- 172.21.15.169:0/2262218831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa7bcd3310 0x55fa7bcdd230 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x55fa7bcd3710 tx=0x7fe788013b70 comp rx=0 tx=0).stop 2024-03-28T01:00:49.539 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.536+0000 7fe7a1581740 1 -- 172.21.15.169:0/2262218831 shutdown_connections 2024-03-28T01:00:49.540 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.536+0000 7fe7a1581740 1 --2- 172.21.15.169:0/2262218831 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55fa7bcf1a10 0x55fa7bcf3e30 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.540 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.536+0000 7fe7a1581740 1 --2- 172.21.15.169:0/2262218831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd8480 0x55fa7bce4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.540 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.536+0000 7fe7a1581740 1 --2- 172.21.15.169:0/2262218831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa7bcd3310 0x55fa7bcdd230 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.540 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.536+0000 7fe7a1581740 1 --2- 172.21.15.169:0/2262218831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa7bcd2920 0x55fa7bcdccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.540 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.536+0000 7fe7a1581740 1 -- 172.21.15.169:0/2262218831 >> 172.21.15.169:0/2262218831 conn(0x55fa7bccde90 msgr2=0x55fa7bccd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:49.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.539+0000 7fe7777fe640 0 client.26158 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:49.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.540+0000 7fe7a1581740 1 -- 172.21.15.169:0/2262218831 shutdown_connections 2024-03-28T01:00:49.547 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.544+0000 7fe7a1581740 1 -- 172.21.15.169:0/2262218831 wait complete. 2024-03-28T01:00:49.548 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.544+0000 7fe7a1581740 20 client.26158 trim_cache size 0 max 16384 2024-03-28T01:00:49.596 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.590+0000 7fe7a1581740 1 Processor -- start 2024-03-28T01:00:49.596 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.591+0000 7fe7a1581740 1 -- start start 2024-03-28T01:00:49.596 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.591+0000 7fe7a1581740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa7bcd2920 0x55fa7bcce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:49.597 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.591+0000 7fe7a1581740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd3310 0x55fa7bce41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:49.597 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.591+0000 7fe7a1581740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa7bcd8480 0x55fa7bcdccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:49.597 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.591+0000 7fe7a1581740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7981aa420 con 0x55fa7bcd3310 2024-03-28T01:00:49.597 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.591+0000 7fe7a1581740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7981a7f20 con 0x55fa7bcd2920 2024-03-28T01:00:49.597 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.591+0000 7fe7a1581740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7981a5fc0 con 0x55fa7bcd8480 2024-03-28T01:00:49.600 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.595+0000 7fe7977fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa7bcd2920 0x55fa7bcce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:49.600 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.595+0000 7fe7977fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa7bcd8480 0x55fa7bcdccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:49.603 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.596+0000 7fe797fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd3310 0x55fa7bce41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:49.603 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.596+0000 7fe797fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd3310 0x55fa7bce41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:58470/0 (socket says 172.21.15.169:58470) 2024-03-28T01:00:49.603 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.596+0000 7fe797fff640 1 -- 172.21.15.169:0/3039452174 learned_addr learned my addr 172.21.15.169:0/3039452174 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:49.608 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.601+0000 7fe797fff640 1 -- 172.21.15.169:0/3039452174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa7bcd8480 msgr2=0x55fa7bcdccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:49.608 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.601+0000 7fe797fff640 1 --2- 172.21.15.169:0/3039452174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa7bcd8480 0x55fa7bcdccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.608 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.601+0000 7fe797fff640 1 -- 172.21.15.169:0/3039452174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa7bcd2920 msgr2=0x55fa7bcce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:49.608 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.601+0000 7fe797fff640 1 --2- 172.21.15.169:0/3039452174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa7bcd2920 0x55fa7bcce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.608 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.601+0000 7fe797fff640 1 -- 172.21.15.169:0/3039452174 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa7bccec00 con 0x55fa7bcd3310 2024-03-28T01:00:49.609 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.605+0000 7fe797fff640 1 --2- 172.21.15.169:0/3039452174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd3310 0x55fa7bce41c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe78800fef0 tx=0x7fe78800ff20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcc57c2f1fa609a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:49.611 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.607+0000 7fe77dffb640 1 -- 172.21.15.169:0/3039452174 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe788000e60 con 0x55fa7bcd3310 2024-03-28T01:00:49.612 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.608+0000 7fe77dffb640 1 -- 172.21.15.169:0/3039452174 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe78800f450 con 0x55fa7bcd3310 2024-03-28T01:00:49.612 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.608+0000 7fe77dffb640 1 -- 172.21.15.169:0/3039452174 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78802a6a0 con 0x55fa7bcd3310 2024-03-28T01:00:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.610+0000 7fe7a1581740 1 -- 172.21.15.169:0/3039452174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd3310 msgr2=0x55fa7bce41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.610+0000 7fe7a1581740 1 --2- 172.21.15.169:0/3039452174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd3310 0x55fa7bce41c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe78800fef0 tx=0x7fe78800ff20 comp rx=0 tx=0).stop 2024-03-28T01:00:49.624 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.620+0000 7fe7a1581740 1 -- 172.21.15.169:0/3039452174 shutdown_connections 2024-03-28T01:00:49.624 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.620+0000 7fe7a1581740 1 --2- 172.21.15.169:0/3039452174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa7bcd8480 0x55fa7bcdccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.624 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.620+0000 7fe7a1581740 1 --2- 172.21.15.169:0/3039452174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd3310 0x55fa7bce41c0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.624 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.620+0000 7fe7a1581740 1 --2- 172.21.15.169:0/3039452174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa7bcd2920 0x55fa7bcce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.624 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.620+0000 7fe7a1581740 1 -- 172.21.15.169:0/3039452174 >> 172.21.15.169:0/3039452174 conn(0x55fa7bccde90 msgr2=0x55fa7bcc8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:49.632 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.629+0000 7fe7a1581740 1 -- 172.21.15.169:0/3039452174 shutdown_connections 2024-03-28T01:00:49.634 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.631+0000 7fe7a1581740 1 -- 172.21.15.169:0/3039452174 wait complete. 2024-03-28T01:00:49.635 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.631+0000 7fe7a1581740 1 -- start start 2024-03-28T01:00:49.637 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.632+0000 7fe7a1581740 1 Processor -- start 2024-03-28T01:00:49.637 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.632+0000 7fe7a1581740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa7bcd2920 0x55fa7bce15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:49.637 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.632+0000 7fe7a1581740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa7bcd3310 0x55fa7bce1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:49.637 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.632+0000 7fe7a1581740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd8480 0x55fa7bce2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:49.637 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.632+0000 7fe7a1581740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7981a65c0 con 0x55fa7bcd8480 2024-03-28T01:00:49.637 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.632+0000 7fe7a1581740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7981a78c0 con 0x55fa7bcd2920 2024-03-28T01:00:49.637 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.632+0000 7fe7a1581740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7981a6c00 con 0x55fa7bcd3310 2024-03-28T01:00:49.637 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.632+0000 7fe7a1581740 10 client.0 _ll_register_callbacks cb 0x7fe788009d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:00:49.637 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.633+0000 7fe797fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa7bcd3310 0x55fa7bce1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.634+0000 7fe7977fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa7bcd2920 0x55fa7bce15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.634+0000 7fe7977fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa7bcd2920 0x55fa7bce15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:50372/0 (socket says 172.21.15.169:50372) 2024-03-28T01:00:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.634+0000 7fe7977fe640 1 -- 172.21.15.169:0/2500530678 learned_addr learned my addr 172.21.15.169:0/2500530678 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:00:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.636+0000 7fe7977fe640 1 --2- 172.21.15.169:0/2500530678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd8480 0x55fa7bce2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.637+0000 7fe7977fe640 1 -- 172.21.15.169:0/2500530678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa7bcd3310 msgr2=0x55fa7bce1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.637+0000 7fe7977fe640 1 --2- 172.21.15.169:0/2500530678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa7bcd3310 0x55fa7bce1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.637+0000 7fe7977fe640 1 -- 172.21.15.169:0/2500530678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd8480 msgr2=0x55fa7bce2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:00:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.637+0000 7fe7977fe640 1 --2- 172.21.15.169:0/2500530678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd8480 0x55fa7bce2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:49.643 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.637+0000 7fe7977fe640 1 -- 172.21.15.169:0/2500530678 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fa7bce3120 con 0x55fa7bcd2920 2024-03-28T01:00:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.639+0000 7fe7977fe640 1 --2- 172.21.15.169:0/2500530678 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa7bcd2920 0x55fa7bce15e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe78c005c40 tx=0x7fe78c04c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fce6ca223429eb06 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:00:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.644+0000 7fe7777fe640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:00:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.644+0000 7fe7777fe640 1 -- 172.21.15.169:0/2500530678 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78c044040 con 0x55fa7bcd2920 2024-03-28T01:00:49.650 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.644+0000 7fe7777fe640 1 -- 172.21.15.169:0/2500530678 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fe78c0047a0 con 0x55fa7bcd2920 2024-03-28T01:00:49.650 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.645+0000 7fe7a1581740 1 -- 172.21.15.169:0/2500530678 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe78800f450 con 0x55fa7bcd2920 2024-03-28T01:00:49.652 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.649+0000 7fe7777fe640 1 -- 172.21.15.169:0/2500530678 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe78c0049a0 con 0x55fa7bcd2920 2024-03-28T01:00:49.653 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.649+0000 7fe7a1581740 10 client.26231 fetch_fsmap learned FSMap version 25 2024-03-28T01:00:49.653 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.649+0000 7fe7a1581740 1 -- 172.21.15.169:0/2500530678 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55fa7bce4430 con 0x55fa7bcd2920 2024-03-28T01:00:49.658 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.655+0000 7fe7777fe640 1 -- 172.21.15.169:0/2500530678 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe78800f450 con 0x55fa7bcd2920 2024-03-28T01:00:49.659 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.655+0000 7fe7777fe640 1 -- 172.21.15.169:0/2500530678 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fe78c005460 con 0x55fa7bcd2920 2024-03-28T01:00:49.660 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.657+0000 7fe7a1581740 10 client.26231 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:00:49.660 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.657+0000 7fe7a1581740 10 client.26231 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:00:49.661 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.657+0000 7fe7a1581740 20 client.26231 populate_metadata read hostname 'smithi169' 2024-03-28T01:00:49.661 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.657+0000 7fe7a1581740 1 --2- 172.21.15.169:0/2500530678 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55fa7bce4840 0x55fa7bce6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:00:49.661 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.657+0000 7fe7a1581740 4 client.26231 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:00:49.661 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.657+0000 7fe7a1581740 1 -- 172.21.15.169:0/2500530678 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55fa7bc45050 con 0x55fa7bce4840 2024-03-28T01:00:49.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:49 smithi177 ceph-mon[36206]: pgmap v480: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 698 MiB/s wr, 1.28k op/s 2024-03-28T01:00:49.669 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.665+0000 7fe797fff640 1 --2- 172.21.15.169:0/2500530678 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55fa7bce4840 0x55fa7bce6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:00:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.675+0000 7fe7777fe640 10 client.26231 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:49.675+0000 7fe797fff640 1 --2- 172.21.15.169:0/2500530678 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55fa7bce4840 0x55fa7bce6c30 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28e30de281bd9836 server_cookie=2d1b501583e0d848 in_seq=0 out_seq=0 2024-03-28T01:00:49.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:49 smithi169 ceph-mon[32274]: pgmap v480: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 698 MiB/s wr, 1.28k op/s 2024-03-28T01:00:49.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:49 smithi174 ceph-mon[36060]: pgmap v480: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 698 MiB/s wr, 1.28k op/s 2024-03-28T01:00:51.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:51 smithi169 ceph-mon[32274]: pgmap v481: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 749 MiB/s wr, 1.36k op/s 2024-03-28T01:00:51.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:51 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:00:51.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:51 smithi174 ceph-mon[36060]: pgmap v481: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 749 MiB/s wr, 1.36k op/s 2024-03-28T01:00:51.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:51 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:00:51.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:51 smithi177 ceph-mon[36206]: pgmap v481: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 749 MiB/s wr, 1.36k op/s 2024-03-28T01:00:51.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:51 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:00:53.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:53 smithi169 ceph-mon[32274]: pgmap v482: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 818 MiB/s wr, 1.37k op/s 2024-03-28T01:00:53.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:53 smithi174 ceph-mon[36060]: pgmap v482: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 818 MiB/s wr, 1.37k op/s 2024-03-28T01:00:53.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:53 smithi177 ceph-mon[36206]: pgmap v482: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 818 MiB/s wr, 1.37k op/s 2024-03-28T01:00:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.249+0000 7fe7777fe640 1 -- 172.21.15.169:0/2500530678 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55fa7bc45050 con 0x55fa7bce4840 2024-03-28T01:00:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.249+0000 7fe7777fe640 10 client.26231 handle_command_reply: tid=0 2024-03-28T01:00:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.249+0000 7fe7a1581740 1 client.26231 shutdown 2024-03-28T01:00:54.255 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.251+0000 7fe7a1581740 1 -- 172.21.15.169:0/2500530678 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa7bcd2920 msgr2=0x55fa7bce15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:54.255 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.251+0000 7fe7a1581740 1 --2- 172.21.15.169:0/2500530678 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa7bcd2920 0x55fa7bce15e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe78c005c40 tx=0x7fe78c04c040 comp rx=0 tx=0).stop 2024-03-28T01:00:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.252+0000 7fe7a1581740 1 -- 172.21.15.169:0/2500530678 shutdown_connections 2024-03-28T01:00:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.252+0000 7fe7a1581740 1 --2- 172.21.15.169:0/2500530678 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55fa7bce4840 0x55fa7bce6c30 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.252+0000 7fe7a1581740 1 --2- 172.21.15.169:0/2500530678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fa7bcd8480 0x55fa7bce2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.252+0000 7fe7a1581740 1 --2- 172.21.15.169:0/2500530678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fa7bcd3310 0x55fa7bce1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.252+0000 7fe7a1581740 1 --2- 172.21.15.169:0/2500530678 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa7bcd2920 0x55fa7bce15e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.252+0000 7fe7a1581740 1 -- 172.21.15.169:0/2500530678 >> 172.21.15.169:0/2500530678 conn(0x55fa7bccde90 msgr2=0x55fa7bcce270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.252+0000 7fe7777fe640 0 client.26231 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:00:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.252+0000 7fe7a1581740 1 -- 172.21.15.169:0/2500530678 shutdown_connections 2024-03-28T01:00:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.252+0000 7fe7a1581740 1 -- 172.21.15.169:0/2500530678 wait complete. 2024-03-28T01:00:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.252+0000 7fe7a1581740 20 client.26231 trim_cache size 0 max 16384 2024-03-28T01:00:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:00:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:00:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:00:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:00:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.256+0000 7fe79e99d640 1 -- 172.21.15.169:0/1387195473 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe7800a2920 msgr2=0x7fe7800a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.256+0000 7fe79e99d640 1 --2- 172.21.15.169:0/1387195473 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe7800a2920 0x7fe7800a4de0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe790084fc0 tx=0x7fe790096450 comp rx=0 tx=0).stop 2024-03-28T01:00:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.256+0000 7fe79e99d640 1 -- 172.21.15.169:0/1387195473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe798109580 msgr2=0x7fe7981a10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:00:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.256+0000 7fe79e99d640 1 --2- 172.21.15.169:0/1387195473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe798109580 0x7fe7981a10e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe790009fd0 tx=0x7fe79000ef10 comp rx=0 tx=0).stop 2024-03-28T01:00:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.260+0000 7fe79e99d640 1 -- 172.21.15.169:0/1387195473 shutdown_connections 2024-03-28T01:00:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.260+0000 7fe79e99d640 1 --2- 172.21.15.169:0/1387195473 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe7800a2920 0x7fe7800a4de0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.260+0000 7fe79e99d640 1 --2- 172.21.15.169:0/1387195473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe798109580 0x7fe7981a10e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.260+0000 7fe79e99d640 1 --2- 172.21.15.169:0/1387195473 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe798075680 0x7fe79819a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.260+0000 7fe79e99d640 1 --2- 172.21.15.169:0/1387195473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7980740a0 0x7fe798199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:00:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.260+0000 7fe79e99d640 1 -- 172.21.15.169:0/1387195473 >> 172.21.15.169:0/1387195473 conn(0x7fe7980fcc60 msgr2=0x7fe798105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:00:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.260+0000 7fe79e99d640 1 -- 172.21.15.169:0/1387195473 shutdown_connections 2024-03-28T01:00:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:00:54.260+0000 7fe79e99d640 1 -- 172.21.15.169:0/1387195473 wait complete. 2024-03-28T01:00:54.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:54 smithi169 ceph-mon[32274]: scrub summary: idle 2024-03-28T01:00:54.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:54 smithi174 ceph-mon[36060]: scrub summary: idle 2024-03-28T01:00:54.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:54 smithi177 ceph-mon[36206]: scrub summary: idle 2024-03-28T01:00:55.806 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:55 smithi169 ceph-mon[32274]: pgmap v483: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 652 MiB/s wr, 1.15k op/s 2024-03-28T01:00:55.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:55 smithi174 ceph-mon[36060]: pgmap v483: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 652 MiB/s wr, 1.15k op/s 2024-03-28T01:00:55.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:55 smithi177 ceph-mon[36206]: pgmap v483: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 652 MiB/s wr, 1.15k op/s 2024-03-28T01:00:57.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:57 smithi169 ceph-mon[32274]: pgmap v484: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 769 MiB/s wr, 1.30k op/s 2024-03-28T01:00:57.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:57 smithi174 ceph-mon[36060]: pgmap v484: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 769 MiB/s wr, 1.30k op/s 2024-03-28T01:00:57.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:57 smithi177 ceph-mon[36206]: pgmap v484: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 769 MiB/s wr, 1.30k op/s 2024-03-28T01:00:58.811 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:00:58 smithi169 ceph-mon[32274]: pgmap v485: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 732 MiB/s wr, 1.09k op/s 2024-03-28T01:00:58.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:00:58 smithi174 ceph-mon[36060]: pgmap v485: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 732 MiB/s wr, 1.09k op/s 2024-03-28T01:00:58.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:00:58 smithi177 ceph-mon[36206]: pgmap v485: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 732 MiB/s wr, 1.09k op/s 2024-03-28T01:01:01.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:01 smithi177 ceph-mon[36206]: pgmap v486: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 769 MiB/s wr, 1.08k op/s 2024-03-28T01:01:01.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:01 smithi174 ceph-mon[36060]: pgmap v486: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 769 MiB/s wr, 1.08k op/s 2024-03-28T01:01:01.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:01 smithi169 ceph-mon[32274]: pgmap v486: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 769 MiB/s wr, 1.08k op/s 2024-03-28T01:01:03.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:03 smithi174 ceph-mon[36060]: pgmap v487: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 869 MiB/s wr, 1.27k op/s 2024-03-28T01:01:03.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:03 smithi177 ceph-mon[36206]: pgmap v487: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 869 MiB/s wr, 1.27k op/s 2024-03-28T01:01:03.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:03 smithi169 ceph-mon[32274]: pgmap v487: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 869 MiB/s wr, 1.27k op/s 2024-03-28T01:01:05.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:05 smithi174 ceph-mon[36060]: pgmap v488: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 721 MiB/s wr, 1.19k op/s 2024-03-28T01:01:05.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:05 smithi177 ceph-mon[36206]: pgmap v488: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 721 MiB/s wr, 1.19k op/s 2024-03-28T01:01:05.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:05 smithi169 ceph-mon[32274]: pgmap v488: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 721 MiB/s wr, 1.19k op/s 2024-03-28T01:01:07.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:07 smithi177 ceph-mon[36206]: pgmap v489: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 845 MiB/s wr, 1.41k op/s 2024-03-28T01:01:07.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:07 smithi169 ceph-mon[32274]: pgmap v489: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 845 MiB/s wr, 1.41k op/s 2024-03-28T01:01:07.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:07 smithi174 ceph-mon[36060]: pgmap v489: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 845 MiB/s wr, 1.41k op/s 2024-03-28T01:01:08.270 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.266+0000 7ff1dfe97640 1 -- 172.21.15.169:0/513863540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1d81086a0 msgr2=0x7ff1d810aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:08.270 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.266+0000 7ff1dfe97640 1 --2- 172.21.15.169:0/513863540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1d81086a0 0x7ff1d810aad0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff1d401a910 tx=0x7ff1d403cca0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.272+0000 7ff1dfe97640 1 -- 172.21.15.169:0/513863540 shutdown_connections 2024-03-28T01:01:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.272+0000 7ff1dfe97640 1 --2- 172.21.15.169:0/513863540 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff1d810d9c0 0x7ff1d810de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.272+0000 7ff1dfe97640 1 --2- 172.21.15.169:0/513863540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1d810b010 0x7ff1d810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.272+0000 7ff1dfe97640 1 --2- 172.21.15.169:0/513863540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1d81086a0 0x7ff1d810aad0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.272+0000 7ff1dfe97640 1 -- 172.21.15.169:0/513863540 >> 172.21.15.169:0/513863540 conn(0x7ff1d806e6e0 msgr2=0x7ff1d8070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:08.284 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.282+0000 7ff1dfe97640 1 -- 172.21.15.169:0/513863540 shutdown_connections 2024-03-28T01:01:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.284+0000 7ff1dfe97640 1 -- 172.21.15.169:0/513863540 wait complete. 2024-03-28T01:01:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.285+0000 7ff1dfe97640 1 Processor -- start 2024-03-28T01:01:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.288+0000 7ff1dfe97640 1 -- start start 2024-03-28T01:01:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.288+0000 7ff1dfe97640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1d81086a0 0x7ff1d819df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.288+0000 7ff1dfe97640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1d810b010 0x7ff1d819e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.288+0000 7ff1dfe97640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff1d810d9c0 0x7ff1d81a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.288+0000 7ff1dfe97640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d81119d0 con 0x7ff1d810d9c0 2024-03-28T01:01:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.288+0000 7ff1dfe97640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d8111850 con 0x7ff1d81086a0 2024-03-28T01:01:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.288+0000 7ff1dfe97640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d8111b50 con 0x7ff1d810b010 2024-03-28T01:01:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.288+0000 7ff1dd432640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1d810b010 0x7ff1d819e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.288+0000 7ff1dd432640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1d810b010 0x7ff1d819e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:55622/0 (socket says 172.21.15.169:55622) 2024-03-28T01:01:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.288+0000 7ff1dd432640 1 -- 172.21.15.169:0/3903550990 learned_addr learned my addr 172.21.15.169:0/3903550990 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.288+0000 7ff1ddc33640 1 --2- 172.21.15.169:0/3903550990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1d81086a0 0x7ff1d819df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.289+0000 7ff1dd432640 1 -- 172.21.15.169:0/3903550990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1d81086a0 msgr2=0x7ff1d819df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.289+0000 7ff1dd432640 1 --2- 172.21.15.169:0/3903550990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1d81086a0 0x7ff1d819df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.289+0000 7ff1dd432640 1 -- 172.21.15.169:0/3903550990 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff1d810d9c0 msgr2=0x7ff1d81a5510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:01:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.289+0000 7ff1dd432640 1 --2- 172.21.15.169:0/3903550990 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff1d810d9c0 0x7ff1d81a5510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.289+0000 7ff1dd432640 1 -- 172.21.15.169:0/3903550990 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d81a5b80 con 0x7ff1d810b010 2024-03-28T01:01:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.289+0000 7ff1dd432640 1 --2- 172.21.15.169:0/3903550990 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1d810b010 0x7ff1d819e4a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800d950 tx=0x7ff1c800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.291+0000 7ff1ceffd640 1 -- 172.21.15.169:0/3903550990 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8017070 con 0x7ff1d810b010 2024-03-28T01:01:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.292+0000 7ff1dfe97640 1 -- 172.21.15.169:0/3903550990 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1d81a5e70 con 0x7ff1d810b010 2024-03-28T01:01:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.292+0000 7ff1dfe97640 1 -- 172.21.15.169:0/3903550990 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1d81a63b0 con 0x7ff1d810b010 2024-03-28T01:01:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.293+0000 7ff1ceffd640 1 -- 172.21.15.169:0/3903550990 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1c8009c30 con 0x7ff1d810b010 2024-03-28T01:01:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.293+0000 7ff1ceffd640 1 -- 172.21.15.169:0/3903550990 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c800f690 con 0x7ff1d810b010 2024-03-28T01:01:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.299+0000 7ff1e2a81740 1 Processor -- start 2024-03-28T01:01:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.299+0000 7ff1ceffd640 1 -- 172.21.15.169:0/3903550990 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff1c800f870 con 0x7ff1d810b010 2024-03-28T01:01:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.301+0000 7ff1e2a81740 1 -- start start 2024-03-28T01:01:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.303+0000 7ff1e2a81740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e10c920 0x562f9e10cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.304+0000 7ff1e2a81740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562f9e10d310 0x562f9e111e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.304+0000 7ff1e2a81740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f9e112480 0x562f9e114910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.304+0000 7ff1e2a81740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f9e071b10 con 0x562f9e112480 2024-03-28T01:01:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.304+0000 7ff1e2a81740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f9e01a900 con 0x562f9e10d310 2024-03-28T01:01:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.304+0000 7ff1e2a81740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f9e06dbb0 con 0x562f9e10c920 2024-03-28T01:01:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.304+0000 7ff1de434640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e10c920 0x562f9e10cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.306+0000 7ff1ceffd640 1 --2- 172.21.15.169:0/3903550990 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff1c40a2ae0 0x7ff1c40a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.306+0000 7ff1ceffd640 1 -- 172.21.15.169:0/3903550990 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ff1c80cfaa0 con 0x7ff1d810b010 2024-03-28T01:01:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.307+0000 7ff1de434640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e10c920 0x562f9e10cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:55638/0 (socket says 172.21.15.169:55638) 2024-03-28T01:01:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.307+0000 7ff1de434640 1 -- 172.21.15.169:0/3530407972 learned_addr learned my addr 172.21.15.169:0/3530407972 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.308+0000 7ff1dd432640 1 --2- 172.21.15.169:0/3530407972 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f9e112480 0x562f9e114910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:08.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.311+0000 7ff1dd432640 1 -- 172.21.15.169:0/3530407972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e10c920 msgr2=0x562f9e10cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:08.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.311+0000 7ff1dd432640 1 --2- 172.21.15.169:0/3530407972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e10c920 0x562f9e10cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.311+0000 7ff1dd432640 1 -- 172.21.15.169:0/3530407972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562f9e10d310 msgr2=0x562f9e111e10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:01:08.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.311+0000 7ff1dd432640 1 --2- 172.21.15.169:0/3530407972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562f9e10d310 0x562f9e111e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.311+0000 7ff1dd432640 1 -- 172.21.15.169:0/3530407972 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f9dd86210 con 0x562f9e112480 2024-03-28T01:01:08.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.312+0000 7ff1de434640 1 --2- 172.21.15.169:0/3903550990 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff1c40a2ae0 0x7ff1c40a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.315+0000 7ff1dd432640 1 --2- 172.21.15.169:0/3530407972 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f9e112480 0x562f9e114910 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff1c809bd40 tx=0x7ff1c8009de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae7e9b7cba2c67aa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.317+0000 7ff1c3fff640 1 -- 172.21.15.169:0/3530407972 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8020070 con 0x562f9e112480 2024-03-28T01:01:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.319+0000 7ff1de434640 1 --2- 172.21.15.169:0/3903550990 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff1c40a2ae0 0x7ff1c40a4fa0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x562f9e10db70 tx=0x7ff1d0004af0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.323+0000 7ff1c3fff640 1 -- 172.21.15.169:0/3530407972 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1c809c040 con 0x562f9e112480 2024-03-28T01:01:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.324+0000 7ff1e2a81740 1 -- 172.21.15.169:0/3530407972 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f9e112480 msgr2=0x562f9e114910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.325+0000 7ff1e2a81740 1 --2- 172.21.15.169:0/3530407972 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f9e112480 0x562f9e114910 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff1c809bd40 tx=0x7ff1c8009de0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.326+0000 7ff1e2a81740 1 -- 172.21.15.169:0/3530407972 shutdown_connections 2024-03-28T01:01:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.326+0000 7ff1e2a81740 1 --2- 172.21.15.169:0/3530407972 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f9e112480 0x562f9e114910 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.326+0000 7ff1e2a81740 1 --2- 172.21.15.169:0/3530407972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562f9e10d310 0x562f9e111e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.326+0000 7ff1e2a81740 1 --2- 172.21.15.169:0/3530407972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e10c920 0x562f9e10cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.326+0000 7ff1e2a81740 1 -- 172.21.15.169:0/3530407972 >> 172.21.15.169:0/3530407972 conn(0x562f9e107e90 msgr2=0x562f9e108290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.329+0000 7ff1e2a81740 1 -- 172.21.15.169:0/3530407972 shutdown_connections 2024-03-28T01:01:08.336 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.333+0000 7ff1e2a81740 1 -- 172.21.15.169:0/3530407972 wait complete. 2024-03-28T01:01:08.336 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.333+0000 7ff1e2a81740 1 -- start start 2024-03-28T01:01:08.336 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.333+0000 7ff1e2a81740 1 Processor -- start 2024-03-28T01:01:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.334+0000 7ff1e2a81740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f9e10c920 0x562f9e116db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.334+0000 7ff1e2a81740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562f9e10d310 0x562f9e1172f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.334+0000 7ff1e2a81740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e112480 0x562f9e11e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.334+0000 7ff1e2a81740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f9e115470 con 0x562f9e10c920 2024-03-28T01:01:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.334+0000 7ff1e2a81740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f9e02dd50 con 0x562f9e10d310 2024-03-28T01:01:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.334+0000 7ff1e2a81740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f9e1155f0 con 0x562f9e112480 2024-03-28T01:01:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.334+0000 7ff1e2a81740 10 client.0 _ll_register_callbacks cb 0x562f9e02d720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.336+0000 7ff1de434640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e112480 0x562f9e11e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.336+0000 7ff1dd432640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562f9e10d310 0x562f9e1172f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.337+0000 7ff1dd432640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562f9e10d310 0x562f9e1172f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:47366/0 (socket says 172.21.15.169:47366) 2024-03-28T01:01:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.337+0000 7ff1dd432640 1 -- 172.21.15.169:0/4023818695 learned_addr learned my addr 172.21.15.169:0/4023818695 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:08.343 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.339+0000 7ff1dd432640 1 -- 172.21.15.169:0/4023818695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e112480 msgr2=0x562f9e11e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:08.343 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.339+0000 7ff1dd432640 1 --2- 172.21.15.169:0/4023818695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e112480 0x562f9e11e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.343 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.339+0000 7ff1dd432640 1 -- 172.21.15.169:0/4023818695 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f9e10c920 msgr2=0x562f9e116db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:08.343 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.339+0000 7ff1dd432640 1 --2- 172.21.15.169:0/4023818695 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f9e10c920 0x562f9e116db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.343 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.339+0000 7ff1dd432640 1 -- 172.21.15.169:0/4023818695 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562f9de52620 con 0x562f9e10d310 2024-03-28T01:01:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.342+0000 7ff1dd432640 1 --2- 172.21.15.169:0/4023818695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562f9e10d310 0x562f9e1172f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80a42b0 tx=0x7ff1c809c0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc8cf39221652e72 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.343+0000 7ff1c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:01:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.345+0000 7ff1c17fa640 1 -- 172.21.15.169:0/4023818695 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c80b4020 con 0x562f9e10d310 2024-03-28T01:01:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.345+0000 7ff1e2a81740 1 -- 172.21.15.169:0/4023818695 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562f9e07f050 con 0x562f9e10d310 2024-03-28T01:01:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.345+0000 7ff1c17fa640 1 -- 172.21.15.169:0/4023818695 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ff1c80bab40 con 0x562f9e10d310 2024-03-28T01:01:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.345+0000 7ff1c17fa640 1 -- 172.21.15.169:0/4023818695 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1c8020070 con 0x562f9e10d310 2024-03-28T01:01:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.346+0000 7ff1e2a81740 10 client.26240 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.346+0000 7ff1e2a81740 1 -- 172.21.15.169:0/4023818695 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x562f9e121730 con 0x562f9e10d310 2024-03-28T01:01:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.346+0000 7ff1c17fa640 1 -- 172.21.15.169:0/4023818695 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x562f9e07f050 con 0x562f9e10d310 2024-03-28T01:01:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.347+0000 7ff1c17fa640 1 -- 172.21.15.169:0/4023818695 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ff1c8097d90 con 0x562f9e10d310 2024-03-28T01:01:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.347+0000 7ff1e2a81740 10 client.26240 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.347+0000 7ff1e2a81740 10 client.26240 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:01:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.347+0000 7ff1e2a81740 20 client.26240 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.347+0000 7ff1e2a81740 1 --2- 172.21.15.169:0/4023818695 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562f9e123080 0x562f9e1254e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.347+0000 7ff1e2a81740 4 client.26240 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:01:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.347+0000 7ff1e2a81740 1 -- 172.21.15.169:0/4023818695 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562f9e0793a0 con 0x562f9e123080 2024-03-28T01:01:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.350+0000 7ff1ddc33640 1 --2- 172.21.15.169:0/4023818695 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562f9e123080 0x562f9e1254e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:08.356 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.352+0000 7ff1ddc33640 1 --2- 172.21.15.169:0/4023818695 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562f9e123080 0x562f9e1254e0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0433dc9dc1202fa server_cookie=2820b7185726e165 in_seq=0 out_seq=0 2024-03-28T01:01:08.356 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.352+0000 7ff1c17fa640 10 client.26240 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:08.361 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.354+0000 7ff1c17fa640 1 -- 172.21.15.169:0/4023818695 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562f9e0793a0 con 0x562f9e123080 2024-03-28T01:01:08.361 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.354+0000 7ff1c17fa640 10 client.26240 handle_command_reply: tid=0 2024-03-28T01:01:08.361 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.355+0000 7ff1e2a81740 1 client.26240 shutdown 2024-03-28T01:01:08.361 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.356+0000 7ff1e2a81740 1 -- 172.21.15.169:0/4023818695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562f9e10d310 msgr2=0x562f9e1172f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.356+0000 7ff1e2a81740 1 --2- 172.21.15.169:0/4023818695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562f9e10d310 0x562f9e1172f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80a42b0 tx=0x7ff1c809c0c0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.357+0000 7ff1e2a81740 1 -- 172.21.15.169:0/4023818695 shutdown_connections 2024-03-28T01:01:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.357+0000 7ff1e2a81740 1 --2- 172.21.15.169:0/4023818695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e112480 0x562f9e11e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.357+0000 7ff1e2a81740 1 --2- 172.21.15.169:0/4023818695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562f9e10d310 0x562f9e1172f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.357+0000 7ff1e2a81740 1 --2- 172.21.15.169:0/4023818695 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562f9e123080 0x562f9e1254e0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.357+0000 7ff1e2a81740 1 --2- 172.21.15.169:0/4023818695 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f9e10c920 0x562f9e116db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.357+0000 7ff1e2a81740 1 -- 172.21.15.169:0/4023818695 >> 172.21.15.169:0/4023818695 conn(0x562f9e107e90 msgr2=0x562f9e107090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.358+0000 7ff1c17fa640 0 client.26240 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.360+0000 7ff1e2a81740 1 -- 172.21.15.169:0/4023818695 shutdown_connections 2024-03-28T01:01:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.364+0000 7ff1e2a81740 1 -- 172.21.15.169:0/4023818695 wait complete. 2024-03-28T01:01:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.366+0000 7ff1e2a81740 20 client.26240 trim_cache size 0 max 16384 2024-03-28T01:01:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.397+0000 7ff1e2a81740 1 Processor -- start 2024-03-28T01:01:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.397+0000 7ff1e2a81740 1 -- start start 2024-03-28T01:01:08.400 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.397+0000 7ff1e2a81740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e10c920 0x562f9e108270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:08.400 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.398+0000 7ff1e2a81740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562f9e10d310 0x562f9e11e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.398+0000 7ff1ddc33640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e10c920 0x562f9e108270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.398+0000 7ff1ddc33640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e10c920 0x562f9e108270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:55652/0 (socket says 172.21.15.169:55652) 2024-03-28T01:01:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.398+0000 7ff1ddc33640 1 -- 172.21.15.169:0/3055030150 learned_addr learned my addr 172.21.15.169:0/3055030150 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.398+0000 7ff1e2a81740 1 --2- 172.21.15.169:0/3055030150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f9e112480 0x562f9e116db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.398+0000 7ff1dd432640 1 --2- 172.21.15.169:0/3055030150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562f9e10d310 0x562f9e11e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.398+0000 7ff1e2a81740 1 -- 172.21.15.169:0/3055030150 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc0120d0 con 0x562f9e112480 2024-03-28T01:01:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.398+0000 7ff1e2a81740 1 -- 172.21.15.169:0/3055030150 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc00fbd0 con 0x562f9e10d310 2024-03-28T01:01:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.398+0000 7ff1e2a81740 1 -- 172.21.15.169:0/3055030150 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc00dc70 con 0x562f9e10c920 2024-03-28T01:01:08.402 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.399+0000 7ff1de434640 1 --2- 172.21.15.169:0/3055030150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f9e112480 0x562f9e116db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:08.402 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.399+0000 7ff1dd432640 1 -- 172.21.15.169:0/3055030150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e10c920 msgr2=0x562f9e108270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:08.402 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.399+0000 7ff1dd432640 1 --2- 172.21.15.169:0/3055030150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e10c920 0x562f9e108270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.402 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.399+0000 7ff1dd432640 1 -- 172.21.15.169:0/3055030150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f9e112480 msgr2=0x562f9e116db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:08.402 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.399+0000 7ff1dd432640 1 --2- 172.21.15.169:0/3055030150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f9e112480 0x562f9e116db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.403 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.399+0000 7ff1dd432640 1 -- 172.21.15.169:0/3055030150 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f9e10d6f0 con 0x562f9e10d310 2024-03-28T01:01:08.404 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.401+0000 7ff1dd432640 1 --2- 172.21.15.169:0/3055030150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562f9e10d310 0x562f9e11e2f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff1c809bfd0 tx=0x7ff1c800df40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d0a905d0cef5c09 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.406+0000 7ff1c37fe640 1 -- 172.21.15.169:0/3055030150 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c80b4070 con 0x562f9e10d310 2024-03-28T01:01:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.407+0000 7ff1c37fe640 1 -- 172.21.15.169:0/3055030150 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1c809b490 con 0x562f9e10d310 2024-03-28T01:01:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.407+0000 7ff1c37fe640 1 -- 172.21.15.169:0/3055030150 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c80bab40 con 0x562f9e10d310 2024-03-28T01:01:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.408+0000 7ff1e2a81740 1 -- 172.21.15.169:0/3055030150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562f9e10d310 msgr2=0x562f9e11e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.408+0000 7ff1e2a81740 1 --2- 172.21.15.169:0/3055030150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562f9e10d310 0x562f9e11e2f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff1c809bfd0 tx=0x7ff1c800df40 comp rx=0 tx=0).stop 2024-03-28T01:01:08.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.412+0000 7ff1e2a81740 1 -- 172.21.15.169:0/3055030150 shutdown_connections 2024-03-28T01:01:08.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.412+0000 7ff1e2a81740 1 --2- 172.21.15.169:0/3055030150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f9e112480 0x562f9e116db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.412+0000 7ff1e2a81740 1 --2- 172.21.15.169:0/3055030150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562f9e10d310 0x562f9e11e2f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.412+0000 7ff1e2a81740 1 --2- 172.21.15.169:0/3055030150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e10c920 0x562f9e108270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.412+0000 7ff1e2a81740 1 -- 172.21.15.169:0/3055030150 >> 172.21.15.169:0/3055030150 conn(0x562f9e107e90 msgr2=0x562f9e10dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:08.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.412+0000 7ff1e2a81740 1 -- 172.21.15.169:0/3055030150 shutdown_connections 2024-03-28T01:01:08.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.412+0000 7ff1e2a81740 1 -- 172.21.15.169:0/3055030150 wait complete. 2024-03-28T01:01:08.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.413+0000 7ff1e2a81740 1 -- start start 2024-03-28T01:01:08.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.413+0000 7ff1e2a81740 1 Processor -- start 2024-03-28T01:01:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.414+0000 7ff1e2a81740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f9e10c920 0x562f9e11dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.414+0000 7ff1e2a81740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e10d310 0x562f9e11aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.414+0000 7ff1e2a81740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562f9e112480 0x562f9e11afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.414+0000 7ff1e2a81740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc00e270 con 0x562f9e10c920 2024-03-28T01:01:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.414+0000 7ff1dd432640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e10d310 0x562f9e11aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.414+0000 7ff1dd432640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e10d310 0x562f9e11aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:55658/0 (socket says 172.21.15.169:55658) 2024-03-28T01:01:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.414+0000 7ff1dd432640 1 -- 172.21.15.169:0/1635197378 learned_addr learned my addr 172.21.15.169:0/1635197378 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.414+0000 7ff1e2a81740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc00f570 con 0x562f9e112480 2024-03-28T01:01:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.414+0000 7ff1e2a81740 1 -- 172.21.15.169:0/1635197378 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc00e8b0 con 0x562f9e10d310 2024-03-28T01:01:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.415+0000 7ff1e2a81740 10 client.0 _ll_register_callbacks cb 0x7ff1c809bc30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.415+0000 7ff1dd432640 1 -- 172.21.15.169:0/1635197378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562f9e112480 msgr2=0x562f9e11afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:08.420 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.415+0000 7ff1dd432640 1 --2- 172.21.15.169:0/1635197378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562f9e112480 0x562f9e11afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.420 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.415+0000 7ff1dd432640 1 -- 172.21.15.169:0/1635197378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f9e10c920 msgr2=0x562f9e11dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:08.420 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.415+0000 7ff1dd432640 1 --2- 172.21.15.169:0/1635197378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f9e10c920 0x562f9e11dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:08.420 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.415+0000 7ff1dd432640 1 -- 172.21.15.169:0/1635197378 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562f9e11b560 con 0x562f9e10d310 2024-03-28T01:01:08.420 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.417+0000 7ff1dd432640 1 --2- 172.21.15.169:0/1635197378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e10d310 0x562f9e11aa60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8009610 tx=0x7ff1c8098410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecd38cc63633c84b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:08.424 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.422+0000 7ff1c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:01:08.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.422+0000 7ff1c17fa640 1 -- 172.21.15.169:0/1635197378 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c80b4070 con 0x562f9e10d310 2024-03-28T01:01:08.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.422+0000 7ff1c17fa640 1 -- 172.21.15.169:0/1635197378 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ff1c80bad40 con 0x562f9e10d310 2024-03-28T01:01:08.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.423+0000 7ff1e2a81740 1 -- 172.21.15.169:0/1635197378 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff1c809b490 con 0x562f9e10d310 2024-03-28T01:01:08.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.430+0000 7ff1c17fa640 1 -- 172.21.15.169:0/1635197378 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1c8097fa0 con 0x562f9e10d310 2024-03-28T01:01:08.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.430+0000 7ff1e2a81740 10 client.26182 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:08.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.430+0000 7ff1e2a81740 1 -- 172.21.15.169:0/1635197378 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x562f9e17d990 con 0x562f9e10d310 2024-03-28T01:01:08.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.431+0000 7ff1c17fa640 1 -- 172.21.15.169:0/1635197378 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1c80a10b0 con 0x562f9e10d310 2024-03-28T01:01:08.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.431+0000 7ff1c17fa640 1 -- 172.21.15.169:0/1635197378 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ff1c80a4a30 con 0x562f9e10d310 2024-03-28T01:01:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.432+0000 7ff1e2a81740 10 client.26182 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.432+0000 7ff1e2a81740 10 client.26182 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:01:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.432+0000 7ff1e2a81740 20 client.26182 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.432+0000 7ff1e2a81740 1 --2- 172.21.15.169:0/1635197378 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562f9e17e060 0x562f9e180480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.432+0000 7ff1e2a81740 4 client.26182 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:01:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.432+0000 7ff1e2a81740 1 -- 172.21.15.169:0/1635197378 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x7ff1c809b490 con 0x562f9e17e060 2024-03-28T01:01:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.432+0000 7ff1ddc33640 1 --2- 172.21.15.169:0/1635197378 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562f9e17e060 0x562f9e180480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:08.436 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.433+0000 7ff1ddc33640 1 --2- 172.21.15.169:0/1635197378 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562f9e17e060 0x562f9e180480 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10a39072023269ac server_cookie=5ff11e769e4b1c69 in_seq=0 out_seq=0 2024-03-28T01:01:08.437 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:08.434+0000 7ff1c17fa640 10 client.26182 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:09.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:09 smithi177 ceph-mon[36206]: pgmap v490: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 791 MiB/s wr, 1.34k op/s 2024-03-28T01:01:09.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:09 smithi169 ceph-mon[32274]: pgmap v490: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 791 MiB/s wr, 1.34k op/s 2024-03-28T01:01:09.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:09 smithi174 ceph-mon[36060]: pgmap v490: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 791 MiB/s wr, 1.34k op/s 2024-03-28T01:01:11.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:11 smithi177 ceph-mon[36206]: pgmap v491: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 801 MiB/s wr, 1.47k op/s 2024-03-28T01:01:11.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:11 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:01:11.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:11 smithi169 ceph-mon[32274]: pgmap v491: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 801 MiB/s wr, 1.47k op/s 2024-03-28T01:01:11.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:11 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:01:11.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:11 smithi174 ceph-mon[36060]: pgmap v491: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 801 MiB/s wr, 1.47k op/s 2024-03-28T01:01:11.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:11 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:01:13.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:13 smithi177 ceph-mon[36206]: pgmap v492: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 831 MiB/s wr, 1.69k op/s 2024-03-28T01:01:13.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:13 smithi169 ceph-mon[32274]: pgmap v492: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 831 MiB/s wr, 1.69k op/s 2024-03-28T01:01:13.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:13 smithi174 ceph-mon[36060]: pgmap v492: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 831 MiB/s wr, 1.69k op/s 2024-03-28T01:01:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.003+0000 7ff1c17fa640 1 -- 172.21.15.169:0/1635197378 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x7ff1c809b490 con 0x562f9e17e060 2024-03-28T01:01:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.003+0000 7ff1c17fa640 10 client.26182 handle_command_reply: tid=0 2024-03-28T01:01:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.003+0000 7ff1e2a81740 1 client.26182 shutdown 2024-03-28T01:01:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.005+0000 7ff1e2a81740 1 -- 172.21.15.169:0/1635197378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e10d310 msgr2=0x562f9e11aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.005+0000 7ff1e2a81740 1 --2- 172.21.15.169:0/1635197378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e10d310 0x562f9e11aa60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8009610 tx=0x7ff1c8098410 comp rx=0 tx=0).stop 2024-03-28T01:01:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.006+0000 7ff1e2a81740 1 -- 172.21.15.169:0/1635197378 shutdown_connections 2024-03-28T01:01:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.006+0000 7ff1e2a81740 1 --2- 172.21.15.169:0/1635197378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562f9e112480 0x562f9e11afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.006+0000 7ff1e2a81740 1 --2- 172.21.15.169:0/1635197378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562f9e10d310 0x562f9e11aa60 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.006+0000 7ff1e2a81740 1 --2- 172.21.15.169:0/1635197378 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562f9e17e060 0x562f9e180480 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.006+0000 7ff1e2a81740 1 --2- 172.21.15.169:0/1635197378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562f9e10c920 0x562f9e11dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.006+0000 7ff1e2a81740 1 -- 172.21.15.169:0/1635197378 >> 172.21.15.169:0/1635197378 conn(0x562f9e107e90 msgr2=0x562f9e108270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.006+0000 7ff1c17fa640 0 client.26182 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:15.010 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.006+0000 7ff1e2a81740 1 -- 172.21.15.169:0/1635197378 shutdown_connections 2024-03-28T01:01:15.010 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.006+0000 7ff1e2a81740 1 -- 172.21.15.169:0/1635197378 wait complete. 2024-03-28T01:01:15.010 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.006+0000 7ff1e2a81740 20 client.26182 trim_cache size 0 max 16384 2024-03-28T01:01:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:01:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:01:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:01:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:01:15.021 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.016+0000 7ff1dfe97640 1 -- 172.21.15.169:0/3903550990 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff1c40a2ae0 msgr2=0x7ff1c40a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.016+0000 7ff1dfe97640 1 --2- 172.21.15.169:0/3903550990 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff1c40a2ae0 0x7ff1c40a4fa0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x562f9e10db70 tx=0x7ff1d0004af0 comp rx=0 tx=0).stop 2024-03-28T01:01:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.016+0000 7ff1dfe97640 1 -- 172.21.15.169:0/3903550990 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1d810b010 msgr2=0x7ff1d819e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.016+0000 7ff1dfe97640 1 --2- 172.21.15.169:0/3903550990 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1d810b010 0x7ff1d819e4a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800d950 tx=0x7ff1c800de10 comp rx=0 tx=0).stop 2024-03-28T01:01:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.018+0000 7ff1dfe97640 1 -- 172.21.15.169:0/3903550990 shutdown_connections 2024-03-28T01:01:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.018+0000 7ff1dfe97640 1 --2- 172.21.15.169:0/3903550990 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff1c40a2ae0 0x7ff1c40a4fa0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.018+0000 7ff1dfe97640 1 --2- 172.21.15.169:0/3903550990 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff1d810d9c0 0x7ff1d81a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.018+0000 7ff1dfe97640 1 --2- 172.21.15.169:0/3903550990 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1d810b010 0x7ff1d819e4a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.018+0000 7ff1dfe97640 1 --2- 172.21.15.169:0/3903550990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1d81086a0 0x7ff1d819df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:15.022 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.018+0000 7ff1dfe97640 1 -- 172.21.15.169:0/3903550990 >> 172.21.15.169:0/3903550990 conn(0x7ff1d806e6e0 msgr2=0x7ff1d8108d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.020+0000 7ff1dfe97640 1 -- 172.21.15.169:0/3903550990 shutdown_connections 2024-03-28T01:01:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:15.020+0000 7ff1dfe97640 1 -- 172.21.15.169:0/3903550990 wait complete. 2024-03-28T01:01:15.659 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:15 smithi174 ceph-mon[36060]: pgmap v493: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 684 MiB/s wr, 1.55k op/s 2024-03-28T01:01:15.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:15 smithi177 ceph-mon[36206]: pgmap v493: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 684 MiB/s wr, 1.55k op/s 2024-03-28T01:01:15.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:15 smithi169 ceph-mon[32274]: pgmap v493: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 684 MiB/s wr, 1.55k op/s 2024-03-28T01:01:17.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:17 smithi174 ceph-mon[36060]: pgmap v494: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 789 MiB/s wr, 1.73k op/s 2024-03-28T01:01:17.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:17 smithi177 ceph-mon[36206]: pgmap v494: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 789 MiB/s wr, 1.73k op/s 2024-03-28T01:01:17.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:17 smithi169 ceph-mon[32274]: pgmap v494: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 789 MiB/s wr, 1.73k op/s 2024-03-28T01:01:18.616 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:01:18.616 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:01:19.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.238+0000 7f88bf7fe640 1 -- 172.21.15.169:0/2580419409 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac004ce0 con 0x7f88c01006c0 2024-03-28T01:01:19.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.239+0000 7f88c6d9d640 1 -- 172.21.15.169:0/2580419409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88c01006c0 msgr2=0x7f88c0100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:19.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.239+0000 7f88c6d9d640 1 --2- 172.21.15.169:0/2580419409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88c01006c0 0x7f88c0100b40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f88ac01a910 tx=0x7f88ac03cca0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.239+0000 7f88c6d9d640 1 -- 172.21.15.169:0/2580419409 shutdown_connections 2024-03-28T01:01:19.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.239+0000 7f88c6d9d640 1 --2- 172.21.15.169:0/2580419409 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88c0101080 0x7f88c0105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.239+0000 7f88c6d9d640 1 --2- 172.21.15.169:0/2580419409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88c01006c0 0x7f88c0100b40 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.239+0000 7f88c6d9d640 1 --2- 172.21.15.169:0/2580419409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88c00ff960 0x7f88c00ffd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.239+0000 7f88c6d9d640 1 -- 172.21.15.169:0/2580419409 >> 172.21.15.169:0/2580419409 conn(0x7f88c00fb290 msgr2=0x7f88c00fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:19.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.253+0000 7f88c6d9d640 1 -- 172.21.15.169:0/2580419409 shutdown_connections 2024-03-28T01:01:19.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.254+0000 7f88c6d9d640 1 -- 172.21.15.169:0/2580419409 wait complete. 2024-03-28T01:01:19.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.255+0000 7f88c6d9d640 1 Processor -- start 2024-03-28T01:01:19.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.255+0000 7f88c6d9d640 1 -- start start 2024-03-28T01:01:19.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.255+0000 7f88c6d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88c00ff960 0x7f88c006d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.255+0000 7f88c6d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88c01006c0 0x7f88c006f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.255+0000 7f88c6d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88c0101080 0x7f88c0071b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.255+0000 7f88c6d9d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c0109280 con 0x7f88c01006c0 2024-03-28T01:01:19.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.255+0000 7f88c6d9d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c0109100 con 0x7f88c00ff960 2024-03-28T01:01:19.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.255+0000 7f88c6d9d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c0109400 con 0x7f88c0101080 2024-03-28T01:01:19.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.256+0000 7f88bffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88c01006c0 0x7f88c006f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:19.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.256+0000 7f88bffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88c01006c0 0x7f88c006f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36144/0 (socket says 172.21.15.169:36144) 2024-03-28T01:01:19.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.256+0000 7f88bffff640 1 -- 172.21.15.169:0/1292103096 learned_addr learned my addr 172.21.15.169:0/1292103096 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:19.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.256+0000 7f88bffff640 1 -- 172.21.15.169:0/1292103096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88c0101080 msgr2=0x7f88c0071b90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:01:19.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.257+0000 7f88bffff640 1 --2- 172.21.15.169:0/1292103096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88c0101080 0x7f88c0071b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.257+0000 7f88bffff640 1 -- 172.21.15.169:0/1292103096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88c00ff960 msgr2=0x7f88c006d0d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:01:19.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.257+0000 7f88bffff640 1 --2- 172.21.15.169:0/1292103096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88c00ff960 0x7f88c006d0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.257+0000 7f88bffff640 1 -- 172.21.15.169:0/1292103096 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c0072450 con 0x7f88c01006c0 2024-03-28T01:01:19.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.257+0000 7f88bffff640 1 --2- 172.21.15.169:0/1292103096 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88c01006c0 0x7f88c006f620 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f88ac047040 tx=0x7f88ac004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:19.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.259+0000 7f88bdffb640 1 -- 172.21.15.169:0/1292103096 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac03d850 con 0x7f88c01006c0 2024-03-28T01:01:19.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.259+0000 7f88bdffb640 1 -- 172.21.15.169:0/1292103096 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88ac053070 con 0x7f88c01006c0 2024-03-28T01:01:19.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.260+0000 7f88c6d9d640 1 -- 172.21.15.169:0/1292103096 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c00726e0 con 0x7f88c01006c0 2024-03-28T01:01:19.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.260+0000 7f88c6d9d640 1 -- 172.21.15.169:0/1292103096 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c0072c50 con 0x7f88c01006c0 2024-03-28T01:01:19.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.260+0000 7f88bdffb640 1 -- 172.21.15.169:0/1292103096 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac04e480 con 0x7f88c01006c0 2024-03-28T01:01:19.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.265+0000 7f88c9981740 1 Processor -- start 2024-03-28T01:01:19.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.265+0000 7f88c9981740 1 -- start start 2024-03-28T01:01:19.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.267+0000 7f88c9981740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d7c4d9da0 0x559d7c4da1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.269+0000 7f88c9981740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4da790 0x559d7c4df290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.285+0000 7f88c533a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d7c4d9da0 0x559d7c4da1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:19.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.285+0000 7f88c9981740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4df900 0x559d7c4e1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.285+0000 7f88c9981740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d7c43f3d0 con 0x559d7c4da790 2024-03-28T01:01:19.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.285+0000 7f88c9981740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d7c3e7ad0 con 0x559d7c4df900 2024-03-28T01:01:19.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.285+0000 7f88c9981740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d7c43b470 con 0x559d7c4d9da0 2024-03-28T01:01:19.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.285+0000 7f88c4b39640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4da790 0x559d7c4df290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:19.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.286+0000 7f88c4b39640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4da790 0x559d7c4df290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36152/0 (socket says 172.21.15.169:36152) 2024-03-28T01:01:19.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.286+0000 7f88c4b39640 1 -- 172.21.15.169:0/794100786 learned_addr learned my addr 172.21.15.169:0/794100786 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:19.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.286+0000 7f88bdffb640 1 -- 172.21.15.169:0/1292103096 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f88ac0035f0 con 0x7f88c01006c0 2024-03-28T01:01:19.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.286+0000 7f88c4b39640 1 -- 172.21.15.169:0/794100786 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d7c4d9da0 msgr2=0x559d7c4da1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:19.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.286+0000 7f88c4b39640 1 --2- 172.21.15.169:0/794100786 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d7c4d9da0 0x559d7c4da1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.286+0000 7f88c4b39640 1 -- 172.21.15.169:0/794100786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4df900 msgr2=0x559d7c4e1d90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:01:19.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.286+0000 7f88c4b39640 1 --2- 172.21.15.169:0/794100786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4df900 0x559d7c4e1d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.286+0000 7f88c4b39640 1 -- 172.21.15.169:0/794100786 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d7c21d7d0 con 0x559d7c4da790 2024-03-28T01:01:19.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.287+0000 7f88c4b39640 1 --2- 172.21.15.169:0/794100786 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4da790 0x559d7c4df290 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f88b4009770 tx=0x7f88b4009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5270f2d7a0ebd3f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:19.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.287+0000 7f88bdffb640 1 --2- 172.21.15.169:0/1292103096 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f88b00a2a00 0x7f88b00a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.288+0000 7f88bdffb640 1 -- 172.21.15.169:0/1292103096 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f88ac101410 con 0x7f88c01006c0 2024-03-28T01:01:19.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.293+0000 7f88c533a640 1 --2- 172.21.15.169:0/1292103096 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f88b00a2a00 0x7f88b00a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:19.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.293+0000 7f88aaffd640 1 -- 172.21.15.169:0/794100786 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b4015020 con 0x559d7c4da790 2024-03-28T01:01:19.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.293+0000 7f88aaffd640 1 -- 172.21.15.169:0/794100786 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88b4011420 con 0x559d7c4da790 2024-03-28T01:01:19.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.293+0000 7f88aaffd640 1 -- 172.21.15.169:0/794100786 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b40105e0 con 0x559d7c4da790 2024-03-28T01:01:19.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.296+0000 7f88c9981740 1 -- 172.21.15.169:0/794100786 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4da790 msgr2=0x559d7c4df290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:19.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.296+0000 7f88c9981740 1 --2- 172.21.15.169:0/794100786 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4da790 0x559d7c4df290 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f88b4009770 tx=0x7f88b4009c30 comp rx=0 tx=0).stop 2024-03-28T01:01:19.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.296+0000 7f88c533a640 1 --2- 172.21.15.169:0/1292103096 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f88b00a2a00 0x7f88b00a4ec0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x559d7c4daff0 tx=0x7f88b8007250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:19.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.300+0000 7f88c9981740 1 -- 172.21.15.169:0/794100786 shutdown_connections 2024-03-28T01:01:19.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.300+0000 7f88c9981740 1 --2- 172.21.15.169:0/794100786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4df900 0x559d7c4e1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.300+0000 7f88c9981740 1 --2- 172.21.15.169:0/794100786 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4da790 0x559d7c4df290 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.300+0000 7f88c9981740 1 --2- 172.21.15.169:0/794100786 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d7c4d9da0 0x559d7c4da1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.300+0000 7f88c9981740 1 -- 172.21.15.169:0/794100786 >> 172.21.15.169:0/794100786 conn(0x559d7c4d5310 msgr2=0x559d7c4d5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:19.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.301+0000 7f88c9981740 1 -- 172.21.15.169:0/794100786 shutdown_connections 2024-03-28T01:01:19.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.304+0000 7f88c9981740 1 -- 172.21.15.169:0/794100786 wait complete. 2024-03-28T01:01:19.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.304+0000 7f88c9981740 1 -- start start 2024-03-28T01:01:19.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.304+0000 7f88c9981740 1 Processor -- start 2024-03-28T01:01:19.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.307+0000 7f88c9981740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4d9da0 0x559d7c4e42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.307+0000 7f88c9981740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4da790 0x559d7c4e4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.307+0000 7f88c9981740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d7c4df900 0x559d7c4eb840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.307+0000 7f88c9981740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d7c4e28b0 con 0x559d7c4da790 2024-03-28T01:01:19.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.307+0000 7f88c9981740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d7c3fb060 con 0x559d7c4d9da0 2024-03-28T01:01:19.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.307+0000 7f88c9981740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d7c4e2a30 con 0x559d7c4df900 2024-03-28T01:01:19.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.307+0000 7f88c9981740 10 client.0 _ll_register_callbacks cb 0x559d7c3dd6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:19.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.309+0000 7f88bffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4da790 0x559d7c4e4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:19.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.309+0000 7f88bffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4da790 0x559d7c4e4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36168/0 (socket says 172.21.15.169:36168) 2024-03-28T01:01:19.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.309+0000 7f88bffff640 1 -- 172.21.15.169:0/4137727046 learned_addr learned my addr 172.21.15.169:0/4137727046 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:19.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.309+0000 7f88c533a640 1 --2- 172.21.15.169:0/4137727046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d7c4df900 0x559d7c4eb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:19.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.309+0000 7f88c4b39640 1 --2- 172.21.15.169:0/4137727046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4d9da0 0x559d7c4e42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:19.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.314+0000 7f88c4b39640 1 -- 172.21.15.169:0/4137727046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d7c4df900 msgr2=0x559d7c4eb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:19.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.314+0000 7f88c4b39640 1 --2- 172.21.15.169:0/4137727046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d7c4df900 0x559d7c4eb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.314+0000 7f88c4b39640 1 -- 172.21.15.169:0/4137727046 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4da790 msgr2=0x559d7c4e4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:19.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.314+0000 7f88c4b39640 1 --2- 172.21.15.169:0/4137727046 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4da790 0x559d7c4e4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.314+0000 7f88c4b39640 1 -- 172.21.15.169:0/4137727046 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d7c1533f0 con 0x559d7c4d9da0 2024-03-28T01:01:19.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.314+0000 7f88c4b39640 1 --2- 172.21.15.169:0/4137727046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4d9da0 0x559d7c4e42c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x559d7c4d6280 tx=0x7f88b4011800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2431103f117f97a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:19.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.320+0000 7f889ffff640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:01:19.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.320+0000 7f889ffff640 1 -- 172.21.15.169:0/4137727046 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b400bad0 con 0x559d7c4d9da0 2024-03-28T01:01:19.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.320+0000 7f889ffff640 1 -- 172.21.15.169:0/4137727046 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f88b400bc70 con 0x559d7c4d9da0 2024-03-28T01:01:19.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.320+0000 7f88c9981740 1 -- 172.21.15.169:0/4137727046 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f88b4011420 con 0x559d7c4d9da0 2024-03-28T01:01:19.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.323+0000 7f889ffff640 1 -- 172.21.15.169:0/4137727046 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88b400be70 con 0x559d7c4d9da0 2024-03-28T01:01:19.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.324+0000 7f88c9981740 10 client.26246 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:19.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.324+0000 7f88c9981740 1 -- 172.21.15.169:0/4137727046 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x559d7c1130f0 con 0x559d7c4d9da0 2024-03-28T01:01:19.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.327+0000 7f889ffff640 1 -- 172.21.15.169:0/4137727046 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88b4011420 con 0x559d7c4d9da0 2024-03-28T01:01:19.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.327+0000 7f889ffff640 1 -- 172.21.15.169:0/4137727046 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f88b402c480 con 0x559d7c4d9da0 2024-03-28T01:01:19.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.329+0000 7f88c9981740 10 client.26246 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:19.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.329+0000 7f88c9981740 10 client.26246 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:01:19.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.329+0000 7f88c9981740 20 client.26246 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:19.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.329+0000 7f88c9981740 1 --2- 172.21.15.169:0/4137727046 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559d7c4ef9a0 0x559d7c4f1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.329+0000 7f88c9981740 4 client.26246 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:01:19.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.329+0000 7f88c9981740 1 -- 172.21.15.169:0/4137727046 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559d7c44c910 con 0x559d7c4ef9a0 2024-03-28T01:01:19.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.334+0000 7f88bffff640 1 --2- 172.21.15.169:0/4137727046 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559d7c4ef9a0 0x559d7c4f1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:19.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.339+0000 7f88bffff640 1 --2- 172.21.15.169:0/4137727046 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559d7c4ef9a0 0x559d7c4f1d90 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd2a460df014ccfc server_cookie=4d9a3b445aeb4a8 in_seq=0 out_seq=0 2024-03-28T01:01:19.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.339+0000 7f889ffff640 10 client.26246 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:19.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.343+0000 7f889ffff640 1 -- 172.21.15.169:0/4137727046 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559d7c44c910 con 0x559d7c4ef9a0 2024-03-28T01:01:19.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.343+0000 7f889ffff640 10 client.26246 handle_command_reply: tid=0 2024-03-28T01:01:19.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.344+0000 7f88c9981740 1 client.26246 shutdown 2024-03-28T01:01:19.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.344+0000 7f88c9981740 1 -- 172.21.15.169:0/4137727046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4d9da0 msgr2=0x559d7c4e42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:19.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.344+0000 7f88c9981740 1 --2- 172.21.15.169:0/4137727046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4d9da0 0x559d7c4e42c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x559d7c4d6280 tx=0x7f88b4011800 comp rx=0 tx=0).stop 2024-03-28T01:01:19.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.344+0000 7f88c9981740 1 -- 172.21.15.169:0/4137727046 shutdown_connections 2024-03-28T01:01:19.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.344+0000 7f88c9981740 1 --2- 172.21.15.169:0/4137727046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d7c4df900 0x559d7c4eb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.344+0000 7f88c9981740 1 --2- 172.21.15.169:0/4137727046 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559d7c4ef9a0 0x559d7c4f1d90 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.344+0000 7f88c9981740 1 --2- 172.21.15.169:0/4137727046 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4da790 0x559d7c4e4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.344+0000 7f88c9981740 1 --2- 172.21.15.169:0/4137727046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4d9da0 0x559d7c4e42c0 secure :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0x559d7c4d6280 tx=0x7f88b4011800 comp rx=0 tx=0).stop 2024-03-28T01:01:19.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.344+0000 7f88c9981740 1 -- 172.21.15.169:0/4137727046 >> 172.21.15.169:0/4137727046 conn(0x559d7c4d5310 msgr2=0x559d7c4d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:19.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.344+0000 7f889ffff640 0 client.26246 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:19.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.346+0000 7f88c9981740 1 -- 172.21.15.169:0/4137727046 shutdown_connections 2024-03-28T01:01:19.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.347+0000 7f88c9981740 1 -- 172.21.15.169:0/4137727046 wait complete. 2024-03-28T01:01:19.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.347+0000 7f88c9981740 20 client.26246 trim_cache size 0 max 16384 2024-03-28T01:01:19.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.387+0000 7f88c9981740 1 Processor -- start 2024-03-28T01:01:19.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.388+0000 7f88c9981740 1 -- start start 2024-03-28T01:01:19.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.388+0000 7f88c9981740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d7c4da790 0x559d7c4e3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.388+0000 7f88c9981740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4df900 0x559d7c4eb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.388+0000 7f88c9981740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4e3440 0x559d7c4e3e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.388+0000 7f88c9981740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c01b2fb0 con 0x559d7c4e3440 2024-03-28T01:01:19.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.388+0000 7f88c9981740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c01b0ab0 con 0x559d7c4df900 2024-03-28T01:01:19.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.388+0000 7f88c9981740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c01aeb50 con 0x559d7c4da790 2024-03-28T01:01:19.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.389+0000 7f88bffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4df900 0x559d7c4eb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:19.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.389+0000 7f88bffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4df900 0x559d7c4eb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33798/0 (socket says 172.21.15.169:33798) 2024-03-28T01:01:19.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.389+0000 7f88bffff640 1 -- 172.21.15.169:0/1461725988 learned_addr learned my addr 172.21.15.169:0/1461725988 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:19.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.390+0000 7f88bffff640 1 -- 172.21.15.169:0/1461725988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d7c4da790 msgr2=0x559d7c4e3910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:01:19.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.390+0000 7f88bffff640 1 --2- 172.21.15.169:0/1461725988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d7c4da790 0x559d7c4e3910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.390+0000 7f88bffff640 1 -- 172.21.15.169:0/1461725988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4e3440 msgr2=0x559d7c4e3e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:01:19.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.390+0000 7f88bffff640 1 --2- 172.21.15.169:0/1461725988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4e3440 0x559d7c4e3e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.390+0000 7f88bffff640 1 -- 172.21.15.169:0/1461725988 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d7c4d6080 con 0x559d7c4df900 2024-03-28T01:01:19.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.391+0000 7f88bffff640 1 --2- 172.21.15.169:0/1461725988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4df900 0x559d7c4eb7d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f88ac0039d0 tx=0x7f88ac050040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e1128d318da788b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:19.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.394+0000 7f88aa7fc640 1 -- 172.21.15.169:0/1461725988 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac0eb070 con 0x559d7c4df900 2024-03-28T01:01:19.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.394+0000 7f88aa7fc640 1 -- 172.21.15.169:0/1461725988 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88ac0ce130 con 0x559d7c4df900 2024-03-28T01:01:19.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.394+0000 7f88aa7fc640 1 -- 172.21.15.169:0/1461725988 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac0cbdc0 con 0x559d7c4df900 2024-03-28T01:01:19.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.395+0000 7f88c9981740 1 -- 172.21.15.169:0/1461725988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4df900 msgr2=0x559d7c4eb7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:19.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.395+0000 7f88c9981740 1 --2- 172.21.15.169:0/1461725988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4df900 0x559d7c4eb7d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f88ac0039d0 tx=0x7f88ac050040 comp rx=0 tx=0).stop 2024-03-28T01:01:19.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.395+0000 7f88c9981740 1 -- 172.21.15.169:0/1461725988 shutdown_connections 2024-03-28T01:01:19.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.395+0000 7f88c9981740 1 --2- 172.21.15.169:0/1461725988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4e3440 0x559d7c4e3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.395+0000 7f88c9981740 1 --2- 172.21.15.169:0/1461725988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4df900 0x559d7c4eb7d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.395+0000 7f88c9981740 1 --2- 172.21.15.169:0/1461725988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d7c4da790 0x559d7c4e3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.395+0000 7f88c9981740 1 -- 172.21.15.169:0/1461725988 >> 172.21.15.169:0/1461725988 conn(0x559d7c4d5310 msgr2=0x559d7c4d8570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:19.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.397+0000 7f88c9981740 1 -- 172.21.15.169:0/1461725988 shutdown_connections 2024-03-28T01:01:19.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.397+0000 7f88c9981740 1 -- 172.21.15.169:0/1461725988 wait complete. 2024-03-28T01:01:19.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.397+0000 7f88c9981740 1 -- start start 2024-03-28T01:01:19.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.398+0000 7f88c9981740 1 Processor -- start 2024-03-28T01:01:19.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.399+0000 7f88c9981740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4da790 0x559d7c4e9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.399+0000 7f88c9981740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4df900 0x559d7c4e9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.399+0000 7f88c9981740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d7c4e3440 0x559d7c4eaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.399+0000 7f88c9981740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c01af150 con 0x559d7c4da790 2024-03-28T01:01:19.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.399+0000 7f88c9981740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c01b0450 con 0x559d7c4df900 2024-03-28T01:01:19.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.399+0000 7f88c9981740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c01af790 con 0x559d7c4e3440 2024-03-28T01:01:19.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.399+0000 7f88c9981740 10 client.0 _ll_register_callbacks cb 0x7f88b40095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:19.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.400+0000 7f88bffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4df900 0x559d7c4e9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:19.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.400+0000 7f88c4b39640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4da790 0x559d7c4e9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:19.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.400+0000 7f88bffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4df900 0x559d7c4e9ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33814/0 (socket says 172.21.15.169:33814) 2024-03-28T01:01:19.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.400+0000 7f88bffff640 1 -- 172.21.15.169:0/1315081691 learned_addr learned my addr 172.21.15.169:0/1315081691 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:19.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.400+0000 7f88c533a640 1 --2- 172.21.15.169:0/1315081691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d7c4e3440 0x559d7c4eaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:19.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.400+0000 7f88bffff640 1 -- 172.21.15.169:0/1315081691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d7c4e3440 msgr2=0x559d7c4eaad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:19.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.400+0000 7f88bffff640 1 --2- 172.21.15.169:0/1315081691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d7c4e3440 0x559d7c4eaad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.400+0000 7f88bffff640 1 -- 172.21.15.169:0/1315081691 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4da790 msgr2=0x559d7c4e9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:19.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.400+0000 7f88bffff640 1 --2- 172.21.15.169:0/1315081691 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4da790 0x559d7c4e9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.400+0000 7f88bffff640 1 -- 172.21.15.169:0/1315081691 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d7c4eb0c0 con 0x559d7c4df900 2024-03-28T01:01:19.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.401+0000 7f88bffff640 1 --2- 172.21.15.169:0/1315081691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4df900 0x559d7c4e9ac0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f88ac05a040 tx=0x7f88ac005680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=893aa6152df31960 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:19.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.401+0000 7f889ffff640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:01:19.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.401+0000 7f889ffff640 1 -- 172.21.15.169:0/1315081691 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac0eb060 con 0x559d7c4df900 2024-03-28T01:01:19.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.401+0000 7f889ffff640 1 -- 172.21.15.169:0/1315081691 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f88ac0fec20 con 0x559d7c4df900 2024-03-28T01:01:19.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.401+0000 7f88c9981740 1 -- 172.21.15.169:0/1315081691 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f88ac0ce130 con 0x559d7c4df900 2024-03-28T01:01:19.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.407+0000 7f889ffff640 1 -- 172.21.15.169:0/1315081691 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88ac0ca0a0 con 0x559d7c4df900 2024-03-28T01:01:19.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.408+0000 7f88c9981740 10 client.26252 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:19.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.408+0000 7f88c9981740 1 -- 172.21.15.169:0/1315081691 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x559d7c54b860 con 0x559d7c4df900 2024-03-28T01:01:19.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.409+0000 7f889ffff640 1 -- 172.21.15.169:0/1315081691 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88ac0d5d40 con 0x559d7c4df900 2024-03-28T01:01:19.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.409+0000 7f889ffff640 1 -- 172.21.15.169:0/1315081691 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f88ac0c9780 con 0x559d7c4df900 2024-03-28T01:01:19.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.410+0000 7f88c9981740 10 client.26252 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:19.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.410+0000 7f88c9981740 10 client.26252 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:01:19.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.410+0000 7f88c9981740 20 client.26252 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:19.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.410+0000 7f88c9981740 1 --2- 172.21.15.169:0/1315081691 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559d7c54c060 0x559d7c54e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.410+0000 7f88c9981740 4 client.26252 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:01:19.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.410+0000 7f88c9981740 1 -- 172.21.15.169:0/1315081691 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f88ac0ce130 con 0x559d7c54c060 2024-03-28T01:01:19.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.411+0000 7f88c4b39640 1 --2- 172.21.15.169:0/1315081691 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559d7c54c060 0x559d7c54e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:19.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.415+0000 7f88c4b39640 1 --2- 172.21.15.169:0/1315081691 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559d7c54c060 0x559d7c54e480 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d9a81739ee8278d server_cookie=102aba7c09c7ae14 in_seq=0 out_seq=0 2024-03-28T01:01:19.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.415+0000 7f889ffff640 10 client.26252 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:19.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.416+0000 7f889ffff640 1 -- 172.21.15.169:0/1315081691 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x7f88ac0ce130 con 0x559d7c54c060 2024-03-28T01:01:19.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.416+0000 7f889ffff640 10 client.26252 handle_command_reply: tid=0 2024-03-28T01:01:19.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.416+0000 7f88c9981740 1 client.26252 shutdown 2024-03-28T01:01:19.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.417+0000 7f88c9981740 1 -- 172.21.15.169:0/1315081691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4df900 msgr2=0x559d7c4e9ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:19.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.417+0000 7f88c9981740 1 --2- 172.21.15.169:0/1315081691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4df900 0x559d7c4e9ac0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f88ac05a040 tx=0x7f88ac005680 comp rx=0 tx=0).stop 2024-03-28T01:01:19.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.418+0000 7f88c9981740 1 -- 172.21.15.169:0/1315081691 shutdown_connections 2024-03-28T01:01:19.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.418+0000 7f88c9981740 1 --2- 172.21.15.169:0/1315081691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d7c4e3440 0x559d7c4eaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.418+0000 7f88c9981740 1 --2- 172.21.15.169:0/1315081691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d7c4df900 0x559d7c4e9ac0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.418+0000 7f88c9981740 1 --2- 172.21.15.169:0/1315081691 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x559d7c54c060 0x559d7c54e480 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.418+0000 7f88c9981740 1 --2- 172.21.15.169:0/1315081691 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x559d7c4da790 0x559d7c4e9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.418+0000 7f889ffff640 0 client.26252 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:19.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.418+0000 7f88c9981740 1 -- 172.21.15.169:0/1315081691 >> 172.21.15.169:0/1315081691 conn(0x559d7c4d5310 msgr2=0x559d7c4d3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:19.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.418+0000 7f88c9981740 1 -- 172.21.15.169:0/1315081691 shutdown_connections 2024-03-28T01:01:19.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.418+0000 7f88c9981740 1 -- 172.21.15.169:0/1315081691 wait complete. 2024-03-28T01:01:19.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.418+0000 7f88c9981740 20 client.26252 trim_cache size 0 max 16384 2024-03-28T01:01:19.422 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:01:19.422 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "no active scrubs running", 2024-03-28T01:01:19.422 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": {} 2024-03-28T01:01:19.422 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:01:19.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.420+0000 7f88c6d9d640 1 -- 172.21.15.169:0/1292103096 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f88b00a2a00 msgr2=0x7f88b00a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:19.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.420+0000 7f88c6d9d640 1 --2- 172.21.15.169:0/1292103096 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f88b00a2a00 0x7f88b00a4ec0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x559d7c4daff0 tx=0x7f88b8007250 comp rx=0 tx=0).stop 2024-03-28T01:01:19.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.420+0000 7f88c6d9d640 1 -- 172.21.15.169:0/1292103096 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88c01006c0 msgr2=0x7f88c006f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:19.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.420+0000 7f88c6d9d640 1 --2- 172.21.15.169:0/1292103096 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88c01006c0 0x7f88c006f620 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f88ac047040 tx=0x7f88ac004270 comp rx=0 tx=0).stop 2024-03-28T01:01:19.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.421+0000 7f88c6d9d640 1 -- 172.21.15.169:0/1292103096 shutdown_connections 2024-03-28T01:01:19.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.421+0000 7f88c6d9d640 1 --2- 172.21.15.169:0/1292103096 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f88b00a2a00 0x7f88b00a4ec0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.421+0000 7f88c6d9d640 1 --2- 172.21.15.169:0/1292103096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88c0101080 0x7f88c0071b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.421+0000 7f88c6d9d640 1 --2- 172.21.15.169:0/1292103096 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88c01006c0 0x7f88c006f620 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.421+0000 7f88c6d9d640 1 --2- 172.21.15.169:0/1292103096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88c00ff960 0x7f88c006d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.421+0000 7f88c6d9d640 1 -- 172.21.15.169:0/1292103096 >> 172.21.15.169:0/1292103096 conn(0x7f88c00fb290 msgr2=0x7f88c0103b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:19.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.421+0000 7f88c6d9d640 1 -- 172.21.15.169:0/1292103096 shutdown_connections 2024-03-28T01:01:19.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.421+0000 7f88c6d9d640 1 -- 172.21.15.169:0/1292103096 wait complete. 2024-03-28T01:01:19.452 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T01:01:19.452 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T01:01:19.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:19 smithi177 ceph-mon[36206]: pgmap v495: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 683 MiB/s wr, 1.55k op/s 2024-03-28T01:01:19.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.784+0000 7f9876345640 1 -- 172.21.15.169:0/2136592046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9870108910 msgr2=0x7f987010ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:19.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.784+0000 7f9876345640 1 --2- 172.21.15.169:0/2136592046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9870108910 0x7f987010ad00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f986400fb60 tx=0x7f986402fd10 comp rx=0 tx=0).stop 2024-03-28T01:01:19.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.794+0000 7f9876345640 1 -- 172.21.15.169:0/2136592046 shutdown_connections 2024-03-28T01:01:19.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.794+0000 7f9876345640 1 --2- 172.21.15.169:0/2136592046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f987010dd70 0x7f9870110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.794+0000 7f9876345640 1 --2- 172.21.15.169:0/2136592046 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f987010b2d0 0x7f987010d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.794+0000 7f9876345640 1 --2- 172.21.15.169:0/2136592046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9870108910 0x7f987010ad00 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.794+0000 7f9876345640 1 -- 172.21.15.169:0/2136592046 >> 172.21.15.169:0/2136592046 conn(0x7f987006e8b0 msgr2=0x7f9870070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:19.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.807+0000 7f9876345640 1 -- 172.21.15.169:0/2136592046 shutdown_connections 2024-03-28T01:01:19.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.808+0000 7f9876345640 1 -- 172.21.15.169:0/2136592046 wait complete. 2024-03-28T01:01:19.811 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:19 smithi169 ceph-mon[32274]: pgmap v495: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 683 MiB/s wr, 1.55k op/s 2024-03-28T01:01:19.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.810+0000 7f9876345640 1 Processor -- start 2024-03-28T01:01:19.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.810+0000 7f9876345640 1 -- start start 2024-03-28T01:01:19.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.811+0000 7f9876345640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9870108910 0x7f987019e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.811+0000 7f9876345640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f987010b2d0 0x7f987019e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.811+0000 7f9876345640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f987010dd70 0x7f98701a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.811+0000 7f9876345640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9870113a30 con 0x7f987010dd70 2024-03-28T01:01:19.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.811+0000 7f9876345640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98701138b0 con 0x7f987010b2d0 2024-03-28T01:01:19.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.811+0000 7f9876345640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9870113bb0 con 0x7f9870108910 2024-03-28T01:01:19.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.812+0000 7f986f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f987010b2d0 0x7f987019e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:19.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.812+0000 7f986f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f987010b2d0 0x7f987019e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33826/0 (socket says 172.21.15.169:33826) 2024-03-28T01:01:19.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.812+0000 7f986f7fe640 1 -- 172.21.15.169:0/4217217159 learned_addr learned my addr 172.21.15.169:0/4217217159 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:19.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.812+0000 7f986f7fe640 1 -- 172.21.15.169:0/4217217159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9870108910 msgr2=0x7f987019e0d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:01:19.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.812+0000 7f986f7fe640 1 --2- 172.21.15.169:0/4217217159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9870108910 0x7f987019e0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.812+0000 7f986f7fe640 1 -- 172.21.15.169:0/4217217159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f987010dd70 msgr2=0x7f98701a5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:19.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.812+0000 7f986f7fe640 1 --2- 172.21.15.169:0/4217217159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f987010dd70 0x7f98701a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:19.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.812+0000 7f986f7fe640 1 -- 172.21.15.169:0/4217217159 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98701a5cf0 con 0x7f987010b2d0 2024-03-28T01:01:19.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.813+0000 7f986f7fe640 1 --2- 172.21.15.169:0/4217217159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f987010b2d0 0x7f987019e610 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f986000e9c0 tx=0x7f986000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:19.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.814+0000 7f986d7fa640 1 -- 172.21.15.169:0/4217217159 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986000ccc0 con 0x7f987010b2d0 2024-03-28T01:01:19.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.814+0000 7f986d7fa640 1 -- 172.21.15.169:0/4217217159 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9860010910 con 0x7f987010b2d0 2024-03-28T01:01:19.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.814+0000 7f986d7fa640 1 -- 172.21.15.169:0/4217217159 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9860018900 con 0x7f987010b2d0 2024-03-28T01:01:19.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.816+0000 7f9876345640 1 -- 172.21.15.169:0/4217217159 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98701a5fe0 con 0x7f987010b2d0 2024-03-28T01:01:19.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.816+0000 7f9876345640 1 -- 172.21.15.169:0/4217217159 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98701a6410 con 0x7f987010b2d0 2024-03-28T01:01:19.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.826+0000 7f986d7fa640 1 -- 172.21.15.169:0/4217217159 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f986001f020 con 0x7f987010b2d0 2024-03-28T01:01:19.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.827+0000 7f986d7fa640 1 --2- 172.21.15.169:0/4217217159 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f98580a2980 0x7f98580a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:19.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.827+0000 7f986d7fa640 1 -- 172.21.15.169:0/4217217159 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f98600cf020 con 0x7f987010b2d0 2024-03-28T01:01:19.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.833+0000 7f986ffff640 1 --2- 172.21.15.169:0/4217217159 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f98580a2980 0x7f98580a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:19.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.831+0000 7f9876345640 1 -- 172.21.15.169:0/4217217159 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98700703a0 con 0x7f987010b2d0 2024-03-28T01:01:19.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:19 smithi174 ceph-mon[36060]: pgmap v495: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 683 MiB/s wr, 1.55k op/s 2024-03-28T01:01:19.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.863+0000 7f986ffff640 1 --2- 172.21.15.169:0/4217217159 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f98580a2980 0x7f98580a4e40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9864005b60 tx=0x7f9864000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:19.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:19.864+0000 7f986d7fa640 1 -- 172.21.15.169:0/4217217159 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9860053740 con 0x7f987010b2d0 2024-03-28T01:01:20.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.372+0000 7f9876345640 1 -- 172.21.15.169:0/4217217159 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9870062340 con 0x7f987010b2d0 2024-03-28T01:01:20.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.377+0000 7f986d7fa640 1 -- 172.21.15.169:0/4217217159 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7f986009bd20 con 0x7f987010b2d0 2024-03-28T01:01:20.380 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T01:01:20.380 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T01:01:20.381 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T01:01:20.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.385+0000 7f9856ffd640 1 -- 172.21.15.169:0/4217217159 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f98580a2980 msgr2=0x7f98580a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:20.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.385+0000 7f9856ffd640 1 --2- 172.21.15.169:0/4217217159 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f98580a2980 0x7f98580a4e40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9864005b60 tx=0x7f9864000bc0 comp rx=0 tx=0).stop 2024-03-28T01:01:20.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.385+0000 7f9856ffd640 1 -- 172.21.15.169:0/4217217159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f987010b2d0 msgr2=0x7f987019e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:20.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.385+0000 7f9856ffd640 1 --2- 172.21.15.169:0/4217217159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f987010b2d0 0x7f987019e610 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f986000e9c0 tx=0x7f986000ee80 comp rx=0 tx=0).stop 2024-03-28T01:01:20.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.387+0000 7f9856ffd640 1 -- 172.21.15.169:0/4217217159 shutdown_connections 2024-03-28T01:01:20.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.387+0000 7f9856ffd640 1 --2- 172.21.15.169:0/4217217159 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f98580a2980 0x7f98580a4e40 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:20.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.387+0000 7f9856ffd640 1 --2- 172.21.15.169:0/4217217159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f987010dd70 0x7f98701a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:20.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.387+0000 7f9856ffd640 1 --2- 172.21.15.169:0/4217217159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f987010b2d0 0x7f987019e610 secure :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f986000e9c0 tx=0x7f986000ee80 comp rx=0 tx=0).stop 2024-03-28T01:01:20.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.387+0000 7f9856ffd640 1 --2- 172.21.15.169:0/4217217159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9870108910 0x7f987019e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:20.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.387+0000 7f9856ffd640 1 -- 172.21.15.169:0/4217217159 >> 172.21.15.169:0/4217217159 conn(0x7f987006e8b0 msgr2=0x7f9870109270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:20.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.390+0000 7f9856ffd640 1 -- 172.21.15.169:0/4217217159 shutdown_connections 2024-03-28T01:01:20.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.392+0000 7f9856ffd640 1 -- 172.21.15.169:0/4217217159 wait complete. 2024-03-28T01:01:20.437 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:01:20.437 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T01:01:20.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:20 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/4217217159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:01:20.807 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:20 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4217217159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:01:20.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:20 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/4217217159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:01:20.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.879+0000 7fa7fbeff640 1 -- 172.21.15.169:0/1494305659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7f410d9c0 msgr2=0x7fa7f410de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:20.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.879+0000 7fa7fbeff640 1 --2- 172.21.15.169:0/1494305659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7f410d9c0 0x7fa7f410de40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0009900 tx=0x7fa7f002bc60 comp rx=0 tx=0).stop 2024-03-28T01:01:20.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.882+0000 7fa7fbeff640 1 -- 172.21.15.169:0/1494305659 shutdown_connections 2024-03-28T01:01:20.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.882+0000 7fa7fbeff640 1 --2- 172.21.15.169:0/1494305659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7f410d9c0 0x7fa7f410de40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:20.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.882+0000 7fa7fbeff640 1 --2- 172.21.15.169:0/1494305659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa7f410b010 0x7fa7f410d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:20.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.882+0000 7fa7fbeff640 1 --2- 172.21.15.169:0/1494305659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7f41086a0 0x7fa7f410aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:20.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.882+0000 7fa7fbeff640 1 -- 172.21.15.169:0/1494305659 >> 172.21.15.169:0/1494305659 conn(0x7fa7f406e500 msgr2=0x7fa7f4070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:20.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.883+0000 7fa7fbeff640 1 -- 172.21.15.169:0/1494305659 shutdown_connections 2024-03-28T01:01:20.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.883+0000 7fa7fbeff640 1 -- 172.21.15.169:0/1494305659 wait complete. 2024-03-28T01:01:20.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.884+0000 7fa7fbeff640 1 Processor -- start 2024-03-28T01:01:20.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.884+0000 7fa7fbeff640 1 -- start start 2024-03-28T01:01:20.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.884+0000 7fa7fbeff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7f41086a0 0x7fa7f4116d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:20.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.884+0000 7fa7fbeff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa7f410b010 0x7fa7f41172d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:20.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.884+0000 7fa7fbeff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7f410d9c0 0x7fa7f41b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:20.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.884+0000 7fa7fbeff640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f411c190 con 0x7fa7f410b010 2024-03-28T01:01:20.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.884+0000 7fa7fbeff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f411c010 con 0x7fa7f41086a0 2024-03-28T01:01:20.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.884+0000 7fa7fbeff640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f411c310 con 0x7fa7f410d9c0 2024-03-28T01:01:20.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.885+0000 7fa7fa49c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7f410d9c0 0x7fa7f41b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:20.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.885+0000 7fa7fa49c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7f410d9c0 0x7fa7f41b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:44460/0 (socket says 172.21.15.169:44460) 2024-03-28T01:01:20.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.885+0000 7fa7fa49c640 1 -- 172.21.15.169:0/2428728412 learned_addr learned my addr 172.21.15.169:0/2428728412 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:20.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.885+0000 7fa7f9c9b640 1 --2- 172.21.15.169:0/2428728412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7f41086a0 0x7fa7f4116d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:20.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.886+0000 7fa7f949a640 1 --2- 172.21.15.169:0/2428728412 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa7f410b010 0x7fa7f41172d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:20.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.886+0000 7fa7f9c9b640 1 -- 172.21.15.169:0/2428728412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7f410d9c0 msgr2=0x7fa7f41b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:20.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.886+0000 7fa7f9c9b640 1 --2- 172.21.15.169:0/2428728412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7f410d9c0 0x7fa7f41b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:20.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.886+0000 7fa7f9c9b640 1 -- 172.21.15.169:0/2428728412 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa7f410b010 msgr2=0x7fa7f41172d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:20.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.886+0000 7fa7f9c9b640 1 --2- 172.21.15.169:0/2428728412 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa7f410b010 0x7fa7f41172d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:20.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.886+0000 7fa7f9c9b640 1 -- 172.21.15.169:0/2428728412 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f41b6de0 con 0x7fa7f41086a0 2024-03-28T01:01:20.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.886+0000 7fa7f9c9b640 1 --2- 172.21.15.169:0/2428728412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7f41086a0 0x7fa7f4116d90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4009fd0 tx=0x7fa7e400b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:20.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.887+0000 7fa7eaffd640 1 -- 172.21.15.169:0/2428728412 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4015020 con 0x7fa7f41086a0 2024-03-28T01:01:20.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.887+0000 7fa7eaffd640 1 -- 172.21.15.169:0/2428728412 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7e4011d10 con 0x7fa7f41086a0 2024-03-28T01:01:20.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.887+0000 7fa7fbeff640 1 -- 172.21.15.169:0/2428728412 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7f41b6fb0 con 0x7fa7f41086a0 2024-03-28T01:01:20.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.887+0000 7fa7fbeff640 1 -- 172.21.15.169:0/2428728412 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7f41b7410 con 0x7fa7f41086a0 2024-03-28T01:01:20.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.887+0000 7fa7eaffd640 1 -- 172.21.15.169:0/2428728412 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e400f2e0 con 0x7fa7f41086a0 2024-03-28T01:01:20.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.890+0000 7fa7eaffd640 1 -- 172.21.15.169:0/2428728412 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa7e4018020 con 0x7fa7f41086a0 2024-03-28T01:01:20.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.891+0000 7fa7eaffd640 1 --2- 172.21.15.169:0/2428728412 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa7e00a2950 0x7fa7e00a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:20.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.892+0000 7fa7f949a640 1 --2- 172.21.15.169:0/2428728412 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa7e00a2950 0x7fa7e00a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:20.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.892+0000 7fa7eaffd640 1 -- 172.21.15.169:0/2428728412 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa7e40cf5f0 con 0x7fa7f41086a0 2024-03-28T01:01:20.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.896+0000 7fa7f949a640 1 --2- 172.21.15.169:0/2428728412 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa7e00a2950 0x7fa7e00a4e10 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa7f41182a0 tx=0x7fa7ec009870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:20.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.899+0000 7fa7febc8740 1 Processor -- start 2024-03-28T01:01:20.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.907+0000 7fa7febc8740 1 -- start start 2024-03-28T01:01:20.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.907+0000 7fa7febc8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b3690 0x55b4fd1b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:20.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.907+0000 7fa7febc8740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1b4080 0x55b4fd1b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:20.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.907+0000 7fa7febc8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4fd1b91f0 0x55b4fd1bb680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:20.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.907+0000 7fa7febc8740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4fd176610 con 0x55b4fd1b4080 2024-03-28T01:01:20.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.907+0000 7fa7febc8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4fd1193c0 con 0x55b4fd1b3690 2024-03-28T01:01:20.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.907+0000 7fa7febc8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4fd0c1d60 con 0x55b4fd1b91f0 2024-03-28T01:01:20.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.908+0000 7fa7f949a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1b4080 0x55b4fd1b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:20.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.908+0000 7fa7f949a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1b4080 0x55b4fd1b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36222/0 (socket says 172.21.15.169:36222) 2024-03-28T01:01:20.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.908+0000 7fa7f949a640 1 -- 172.21.15.169:0/2009562735 learned_addr learned my addr 172.21.15.169:0/2009562735 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:20.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.908+0000 7fa7f9c9b640 1 --2- 172.21.15.169:0/2009562735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b3690 0x55b4fd1b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:20.912 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.909+0000 7fa7f9c9b640 1 -- 172.21.15.169:0/2009562735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4fd1b91f0 msgr2=0x55b4fd1bb680 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:01:20.912 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.909+0000 7fa7f9c9b640 1 --2- 172.21.15.169:0/2009562735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4fd1b91f0 0x55b4fd1bb680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:20.912 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.909+0000 7fa7f9c9b640 1 -- 172.21.15.169:0/2009562735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1b4080 msgr2=0x55b4fd1b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:20.912 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.909+0000 7fa7f9c9b640 1 --2- 172.21.15.169:0/2009562735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1b4080 0x55b4fd1b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:20.912 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.909+0000 7fa7f9c9b640 1 -- 172.21.15.169:0/2009562735 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b4fce2d3f0 con 0x55b4fd1b3690 2024-03-28T01:01:20.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.912+0000 7fa7f9c9b640 1 --2- 172.21.15.169:0/2009562735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b3690 0x55b4fd1b3ab0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4097ff0 tx=0x7fa7e409c350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d506d5d2136d540 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:20.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.913+0000 7fa7dffff640 1 -- 172.21.15.169:0/2009562735 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e40b4070 con 0x55b4fd1b3690 2024-03-28T01:01:20.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.913+0000 7fa7dffff640 1 -- 172.21.15.169:0/2009562735 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7e409fce0 con 0x55b4fd1b3690 2024-03-28T01:01:20.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.914+0000 7fa7febc8740 1 -- 172.21.15.169:0/2009562735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b3690 msgr2=0x55b4fd1b3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:20.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.914+0000 7fa7febc8740 1 --2- 172.21.15.169:0/2009562735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b3690 0x55b4fd1b3ab0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4097ff0 tx=0x7fa7e409c350 comp rx=0 tx=0).stop 2024-03-28T01:01:20.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.914+0000 7fa7febc8740 1 -- 172.21.15.169:0/2009562735 shutdown_connections 2024-03-28T01:01:20.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.914+0000 7fa7febc8740 1 --2- 172.21.15.169:0/2009562735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4fd1b91f0 0x55b4fd1bb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:20.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.914+0000 7fa7febc8740 1 --2- 172.21.15.169:0/2009562735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1b4080 0x55b4fd1b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:20.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.914+0000 7fa7febc8740 1 --2- 172.21.15.169:0/2009562735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b3690 0x55b4fd1b3ab0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:20.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.914+0000 7fa7febc8740 1 -- 172.21.15.169:0/2009562735 >> 172.21.15.169:0/2009562735 conn(0x55b4fd1aec00 msgr2=0x55b4fd1af000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:20.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.915+0000 7fa7febc8740 1 -- 172.21.15.169:0/2009562735 shutdown_connections 2024-03-28T01:01:20.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.918+0000 7fa7febc8740 1 -- 172.21.15.169:0/2009562735 wait complete. 2024-03-28T01:01:20.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.918+0000 7fa7febc8740 1 -- start start 2024-03-28T01:01:20.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.919+0000 7fa7febc8740 1 Processor -- start 2024-03-28T01:01:20.927 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.923+0000 7fa7febc8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b3690 0x55b4fd1bdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:20.927 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.923+0000 7fa7febc8740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1b4080 0x55b4fd1be000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:20.927 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.923+0000 7fa7febc8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4fd1b91f0 0x55b4fd1c4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:20.927 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.923+0000 7fa7febc8740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4fd0d5240 con 0x55b4fd1b4080 2024-03-28T01:01:20.927 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.923+0000 7fa7febc8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4fd115460 con 0x55b4fd1b3690 2024-03-28T01:01:20.927 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.923+0000 7fa7febc8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4fd1bc1f0 con 0x55b4fd1b91f0 2024-03-28T01:01:20.927 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.924+0000 7fa7febc8740 10 client.0 _ll_register_callbacks cb 0x55b4fd0d4c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:20.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.928+0000 7fa7f9c9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b3690 0x55b4fd1bdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:20.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.928+0000 7fa7f9c9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b3690 0x55b4fd1bdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33858/0 (socket says 172.21.15.169:33858) 2024-03-28T01:01:20.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.928+0000 7fa7f9c9b640 1 -- 172.21.15.169:0/632979309 learned_addr learned my addr 172.21.15.169:0/632979309 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:20.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.930+0000 7fa7f9c9b640 1 -- 172.21.15.169:0/632979309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4fd1b91f0 msgr2=0x55b4fd1c4fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:01:20.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.930+0000 7fa7f9c9b640 1 --2- 172.21.15.169:0/632979309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4fd1b91f0 0x55b4fd1c4fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:20.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.930+0000 7fa7f9c9b640 1 -- 172.21.15.169:0/632979309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1b4080 msgr2=0x55b4fd1be000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:20.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.930+0000 7fa7f9c9b640 1 --2- 172.21.15.169:0/632979309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1b4080 0x55b4fd1be000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:20.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.930+0000 7fa7f9c9b640 1 -- 172.21.15.169:0/632979309 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b4fcef77d0 con 0x55b4fd1b3690 2024-03-28T01:01:20.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.930+0000 7fa7f9c9b640 1 --2- 172.21.15.169:0/632979309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b3690 0x55b4fd1bdac0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa7e40091c0 tx=0x7fa7e400bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4296a73a57204faa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:20.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.932+0000 7fa7dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:01:20.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.932+0000 7fa7dd7fa640 1 -- 172.21.15.169:0/632979309 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e40b8020 con 0x55b4fd1b3690 2024-03-28T01:01:20.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.932+0000 7fa7dd7fa640 1 -- 172.21.15.169:0/632979309 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa7e409afe0 con 0x55b4fd1b3690 2024-03-28T01:01:20.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.932+0000 7fa7febc8740 1 -- 172.21.15.169:0/632979309 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa7e409fce0 con 0x55b4fd1b3690 2024-03-28T01:01:20.937 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.934+0000 7fa7dd7fa640 1 -- 172.21.15.169:0/632979309 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7e40b4070 con 0x55b4fd1b3690 2024-03-28T01:01:20.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.940+0000 7fa7febc8740 10 client.26267 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:20.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.940+0000 7fa7febc8740 1 -- 172.21.15.169:0/632979309 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55b4fd1c8340 con 0x55b4fd1b3690 2024-03-28T01:01:20.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.943+0000 7fa7dd7fa640 1 -- 172.21.15.169:0/632979309 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7e409fce0 con 0x55b4fd1b3690 2024-03-28T01:01:20.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.943+0000 7fa7dd7fa640 1 -- 172.21.15.169:0/632979309 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa7e40978e0 con 0x55b4fd1b3690 2024-03-28T01:01:20.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.946+0000 7fa7febc8740 10 client.26267 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:20.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.946+0000 7fa7febc8740 10 client.26267 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:01:20.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.946+0000 7fa7febc8740 20 client.26267 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:20.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.946+0000 7fa7febc8740 1 --2- 172.21.15.169:0/632979309 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b4fd1c9690 0x55b4fd1cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:20.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.946+0000 7fa7febc8740 4 client.26267 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:01:20.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.946+0000 7fa7febc8740 1 -- 172.21.15.169:0/632979309 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b4fd126900 con 0x55b4fd1c9690 2024-03-28T01:01:20.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.948+0000 7fa7f949a640 1 --2- 172.21.15.169:0/632979309 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b4fd1c9690 0x55b4fd1cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:20.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.959+0000 7fa7f949a640 1 --2- 172.21.15.169:0/632979309 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b4fd1c9690 0x55b4fd1cba80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e66d5ce9922c167e server_cookie=3f77bebf9f3a27e7 in_seq=0 out_seq=0 2024-03-28T01:01:20.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.959+0000 7fa7dd7fa640 10 client.26267 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:01:20.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.965+0000 7fa7dd7fa640 1 -- 172.21.15.169:0/632979309 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b4fd126900 con 0x55b4fd1c9690 2024-03-28T01:01:20.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.965+0000 7fa7dd7fa640 10 client.26267 handle_command_reply: tid=0 2024-03-28T01:01:20.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.965+0000 7fa7febc8740 1 client.26267 shutdown 2024-03-28T01:01:20.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.966+0000 7fa7febc8740 1 -- 172.21.15.169:0/632979309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b3690 msgr2=0x55b4fd1bdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:20.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.966+0000 7fa7febc8740 1 --2- 172.21.15.169:0/632979309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b3690 0x55b4fd1bdac0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa7e40091c0 tx=0x7fa7e400bf20 comp rx=0 tx=0).stop 2024-03-28T01:01:20.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.966+0000 7fa7febc8740 1 -- 172.21.15.169:0/632979309 shutdown_connections 2024-03-28T01:01:20.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.966+0000 7fa7febc8740 1 --2- 172.21.15.169:0/632979309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4fd1b91f0 0x55b4fd1c4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:20.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.966+0000 7fa7febc8740 1 --2- 172.21.15.169:0/632979309 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b4fd1c9690 0x55b4fd1cba80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:20.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.966+0000 7fa7febc8740 1 --2- 172.21.15.169:0/632979309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1b4080 0x55b4fd1be000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:20.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.966+0000 7fa7febc8740 1 --2- 172.21.15.169:0/632979309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b3690 0x55b4fd1bdac0 secure :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa7e40091c0 tx=0x7fa7e400bf20 comp rx=0 tx=0).stop 2024-03-28T01:01:20.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.966+0000 7fa7febc8740 1 -- 172.21.15.169:0/632979309 >> 172.21.15.169:0/632979309 conn(0x55b4fd1aec00 msgr2=0x55b4fd1ade20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:20.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.967+0000 7fa7dd7fa640 0 client.26267 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:01:20.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.969+0000 7fa7febc8740 1 -- 172.21.15.169:0/632979309 shutdown_connections 2024-03-28T01:01:20.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.971+0000 7fa7febc8740 1 -- 172.21.15.169:0/632979309 wait complete. 2024-03-28T01:01:20.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:20.973+0000 7fa7febc8740 20 client.26267 trim_cache size 0 max 16384 2024-03-28T01:01:21.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.014+0000 7fa7febc8740 1 Processor -- start 2024-03-28T01:01:21.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.014+0000 7fa7febc8740 1 -- start start 2024-03-28T01:01:21.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.014+0000 7fa7febc8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b4080 0x55b4fd1bd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.014+0000 7fa7febc8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4fd1b91f0 0x55b4fd1c4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.014+0000 7fa7febc8740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1bdc10 0x55b4fd1ad590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.014+0000 7fa7febc8740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d8012080 con 0x55b4fd1bdc10 2024-03-28T01:01:21.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.014+0000 7fa7febc8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d800fb80 con 0x55b4fd1b4080 2024-03-28T01:01:21.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.014+0000 7fa7febc8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d800dc20 con 0x55b4fd1b91f0 2024-03-28T01:01:21.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.015+0000 7fa7fa49c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1bdc10 0x55b4fd1ad590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:21.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.015+0000 7fa7fa49c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1bdc10 0x55b4fd1ad590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36252/0 (socket says 172.21.15.169:36252) 2024-03-28T01:01:21.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.015+0000 7fa7fa49c640 1 -- 172.21.15.169:0/2450837164 learned_addr learned my addr 172.21.15.169:0/2450837164 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:21.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.015+0000 7fa7fa49c640 1 -- 172.21.15.169:0/2450837164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4fd1b91f0 msgr2=0x55b4fd1c4f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:01:21.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.015+0000 7fa7fa49c640 1 --2- 172.21.15.169:0/2450837164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4fd1b91f0 0x55b4fd1c4f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.015+0000 7fa7fa49c640 1 -- 172.21.15.169:0/2450837164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b4080 msgr2=0x55b4fd1bd6d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:01:21.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.015+0000 7fa7fa49c640 1 --2- 172.21.15.169:0/2450837164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b4080 0x55b4fd1bd6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.015+0000 7fa7fa49c640 1 -- 172.21.15.169:0/2450837164 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b4fd1b87f0 con 0x55b4fd1bdc10 2024-03-28T01:01:21.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.016+0000 7fa7fa49c640 1 --2- 172.21.15.169:0/2450837164 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1bdc10 0x55b4fd1ad590 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0009230 tx=0x7fa7f002c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9be3bf7c9b0ad87 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:21.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.016+0000 7fa7df7fe640 1 -- 172.21.15.169:0/2450837164 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f0042070 con 0x55b4fd1bdc10 2024-03-28T01:01:21.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.016+0000 7fa7df7fe640 1 -- 172.21.15.169:0/2450837164 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7f00046c0 con 0x55b4fd1bdc10 2024-03-28T01:01:21.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.018+0000 7fa7febc8740 1 -- 172.21.15.169:0/2450837164 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1bdc10 msgr2=0x55b4fd1ad590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:21.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.018+0000 7fa7febc8740 1 --2- 172.21.15.169:0/2450837164 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1bdc10 0x55b4fd1ad590 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0009230 tx=0x7fa7f002c6f0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.036+0000 7fa7febc8740 1 -- 172.21.15.169:0/2450837164 shutdown_connections 2024-03-28T01:01:21.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.036+0000 7fa7febc8740 1 --2- 172.21.15.169:0/2450837164 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1bdc10 0x55b4fd1ad590 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.036+0000 7fa7febc8740 1 --2- 172.21.15.169:0/2450837164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4fd1b91f0 0x55b4fd1c4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.036+0000 7fa7febc8740 1 --2- 172.21.15.169:0/2450837164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b4080 0x55b4fd1bd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.036+0000 7fa7febc8740 1 -- 172.21.15.169:0/2450837164 >> 172.21.15.169:0/2450837164 conn(0x55b4fd1aec00 msgr2=0x55b4fd1aefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:21.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.036+0000 7fa7febc8740 1 -- 172.21.15.169:0/2450837164 shutdown_connections 2024-03-28T01:01:21.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.036+0000 7fa7febc8740 1 -- 172.21.15.169:0/2450837164 wait complete. 2024-03-28T01:01:21.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.036+0000 7fa7febc8740 1 -- start start 2024-03-28T01:01:21.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.037+0000 7fa7febc8740 1 Processor -- start 2024-03-28T01:01:21.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.037+0000 7fa7febc8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b4080 0x55b4fd1c2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.037+0000 7fa7febc8740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1b91f0 0x55b4fd1c34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.037+0000 7fa7febc8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4fd1bdc10 0x55b4fd1c4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.037+0000 7fa7febc8740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d800e220 con 0x55b4fd1b91f0 2024-03-28T01:01:21.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.037+0000 7fa7febc8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d800f520 con 0x55b4fd1b4080 2024-03-28T01:01:21.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.037+0000 7fa7febc8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d800e860 con 0x55b4fd1bdc10 2024-03-28T01:01:21.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.037+0000 7fa7febc8740 10 client.0 _ll_register_callbacks cb 0x7fa7e409b4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:21.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.040+0000 7fa7f949a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1b91f0 0x55b4fd1c34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:21.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.040+0000 7fa7f949a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1b91f0 0x55b4fd1c34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36256/0 (socket says 172.21.15.169:36256) 2024-03-28T01:01:21.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.040+0000 7fa7f949a640 1 -- 172.21.15.169:0/849819998 learned_addr learned my addr 172.21.15.169:0/849819998 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:21.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.041+0000 7fa7f9c9b640 1 --2- 172.21.15.169:0/849819998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b4080 0x55b4fd1c2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:21.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.041+0000 7fa7fa49c640 1 --2- 172.21.15.169:0/849819998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4fd1bdc10 0x55b4fd1c4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:21.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.047+0000 7fa7f949a640 1 -- 172.21.15.169:0/849819998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4fd1bdc10 msgr2=0x55b4fd1c4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:21.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.047+0000 7fa7f949a640 1 --2- 172.21.15.169:0/849819998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4fd1bdc10 0x55b4fd1c4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.047+0000 7fa7f949a640 1 -- 172.21.15.169:0/849819998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b4080 msgr2=0x55b4fd1c2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:21.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.047+0000 7fa7f949a640 1 --2- 172.21.15.169:0/849819998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b4080 0x55b4fd1c2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.047+0000 7fa7f949a640 1 -- 172.21.15.169:0/849819998 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b4fd1c4a60 con 0x55b4fd1b91f0 2024-03-28T01:01:21.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.058+0000 7fa7f949a640 1 --2- 172.21.15.169:0/849819998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1b91f0 0x55b4fd1c34d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x55b4fd1b4480 tx=0x7fa7ec004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaa263ffec20aad0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:21.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.058+0000 7fa7dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:01:21.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.065+0000 7fa7dd7fa640 1 -- 172.21.15.169:0/849819998 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec02c020 con 0x55b4fd1b91f0 2024-03-28T01:01:21.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.065+0000 7fa7febc8740 1 -- 172.21.15.169:0/849819998 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa7f00046c0 con 0x55b4fd1b91f0 2024-03-28T01:01:21.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.065+0000 7fa7dd7fa640 1 -- 172.21.15.169:0/849819998 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa7ec028590 con 0x55b4fd1b91f0 2024-03-28T01:01:21.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.067+0000 7fa7dd7fa640 1 -- 172.21.15.169:0/849819998 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7ec037050 con 0x55b4fd1b91f0 2024-03-28T01:01:21.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.067+0000 7fa7febc8740 10 client.16389 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:21.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.067+0000 7fa7febc8740 1 -- 172.21.15.169:0/849819998 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55b4fd225720 con 0x55b4fd1b91f0 2024-03-28T01:01:21.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.071+0000 7fa7dd7fa640 1 -- 172.21.15.169:0/849819998 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7f00046c0 con 0x55b4fd1b91f0 2024-03-28T01:01:21.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.071+0000 7fa7dd7fa640 1 -- 172.21.15.169:0/849819998 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa7ec00f040 con 0x55b4fd1b91f0 2024-03-28T01:01:21.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.071+0000 7fa7febc8740 10 client.16389 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:21.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.071+0000 7fa7febc8740 10 client.16389 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:01:21.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.071+0000 7fa7febc8740 20 client.16389 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:21.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.071+0000 7fa7febc8740 1 --2- 172.21.15.169:0/849819998 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b4fd2260b0 0x55b4fd2284d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.071+0000 7fa7febc8740 4 client.16389 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:01:21.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.071+0000 7fa7febc8740 1 -- 172.21.15.169:0/849819998 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b4fd120c50 con 0x55b4fd2260b0 2024-03-28T01:01:21.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.076+0000 7fa7f9c9b640 1 --2- 172.21.15.169:0/849819998 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b4fd2260b0 0x55b4fd2284d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:21.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.092+0000 7fa7f9c9b640 1 --2- 172.21.15.169:0/849819998 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b4fd2260b0 0x55b4fd2284d0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e8449c90bff6f3a7 server_cookie=ec963f8a45f5c0b4 in_seq=0 out_seq=0 2024-03-28T01:01:21.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.094+0000 7fa7dd7fa640 10 client.16389 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:01:21.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.095+0000 7fa7dd7fa640 1 -- 172.21.15.169:0/849819998 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b4fd120c50 con 0x55b4fd2260b0 2024-03-28T01:01:21.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.095+0000 7fa7dd7fa640 10 client.16389 handle_command_reply: tid=0 2024-03-28T01:01:21.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.095+0000 7fa7febc8740 1 client.16389 shutdown 2024-03-28T01:01:21.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.096+0000 7fa7febc8740 1 -- 172.21.15.169:0/849819998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1b91f0 msgr2=0x55b4fd1c34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:21.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.096+0000 7fa7febc8740 1 --2- 172.21.15.169:0/849819998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1b91f0 0x55b4fd1c34d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x55b4fd1b4480 tx=0x7fa7ec004ab0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.099+0000 7fa7febc8740 1 -- 172.21.15.169:0/849819998 shutdown_connections 2024-03-28T01:01:21.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.099+0000 7fa7febc8740 1 --2- 172.21.15.169:0/849819998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4fd1bdc10 0x55b4fd1c4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.099+0000 7fa7febc8740 1 --2- 172.21.15.169:0/849819998 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b4fd2260b0 0x55b4fd2284d0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.099+0000 7fa7febc8740 1 --2- 172.21.15.169:0/849819998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4fd1b91f0 0x55b4fd1c34d0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.099+0000 7fa7febc8740 1 --2- 172.21.15.169:0/849819998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4fd1b4080 0x55b4fd1c2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.099+0000 7fa7febc8740 1 -- 172.21.15.169:0/849819998 >> 172.21.15.169:0/849819998 conn(0x55b4fd1aec00 msgr2=0x55b4fd1b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:21.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.100+0000 7fa7dd7fa640 0 client.16389 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:01:21.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.106+0000 7fa7febc8740 1 -- 172.21.15.169:0/849819998 shutdown_connections 2024-03-28T01:01:21.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.112+0000 7fa7febc8740 1 -- 172.21.15.169:0/849819998 wait complete. 2024-03-28T01:01:21.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.112+0000 7fa7febc8740 20 client.16389 trim_cache size 0 max 16384 2024-03-28T01:01:21.118 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:01:21.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.118+0000 7fa7fbeff640 1 -- 172.21.15.169:0/2428728412 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa7e00a2950 msgr2=0x7fa7e00a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:21.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.118+0000 7fa7fbeff640 1 --2- 172.21.15.169:0/2428728412 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa7e00a2950 0x7fa7e00a4e10 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa7f41182a0 tx=0x7fa7ec009870 comp rx=0 tx=0).stop 2024-03-28T01:01:21.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.118+0000 7fa7fbeff640 1 -- 172.21.15.169:0/2428728412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7f41086a0 msgr2=0x7fa7f4116d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:21.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.118+0000 7fa7fbeff640 1 --2- 172.21.15.169:0/2428728412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7f41086a0 0x7fa7f4116d90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4009fd0 tx=0x7fa7e400b870 comp rx=0 tx=0).stop 2024-03-28T01:01:21.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.120+0000 7fa7fbeff640 1 -- 172.21.15.169:0/2428728412 shutdown_connections 2024-03-28T01:01:21.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.120+0000 7fa7fbeff640 1 --2- 172.21.15.169:0/2428728412 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa7e00a2950 0x7fa7e00a4e10 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.120+0000 7fa7fbeff640 1 --2- 172.21.15.169:0/2428728412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7f410d9c0 0x7fa7f41b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.120+0000 7fa7fbeff640 1 --2- 172.21.15.169:0/2428728412 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa7f410b010 0x7fa7f41172d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.120+0000 7fa7fbeff640 1 --2- 172.21.15.169:0/2428728412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7f41086a0 0x7fa7f4116d90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.120+0000 7fa7fbeff640 1 -- 172.21.15.169:0/2428728412 >> 172.21.15.169:0/2428728412 conn(0x7fa7f406e500 msgr2=0x7fa7f410b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:21.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.123+0000 7fa7fbeff640 1 -- 172.21.15.169:0/2428728412 shutdown_connections 2024-03-28T01:01:21.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.123+0000 7fa7fbeff640 1 -- 172.21.15.169:0/2428728412 wait complete. 2024-03-28T01:01:21.169 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:01:21.170 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T01:01:21.431 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:21 smithi169 ceph-mon[32274]: pgmap v496: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 680 MiB/s wr, 1.46k op/s 2024-03-28T01:01:21.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.571+0000 7fc6e0085640 1 -- 172.21.15.169:0/1228620737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6d8078de0 msgr2=0x7fc6d807b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:21.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.571+0000 7fc6e0085640 1 --2- 172.21.15.169:0/1228620737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6d8078de0 0x7fc6d807b270 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009be0 tx=0x7fc6d002bef0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.573+0000 7fc6e0085640 1 -- 172.21.15.169:0/1228620737 shutdown_connections 2024-03-28T01:01:21.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.573+0000 7fc6e0085640 1 --2- 172.21.15.169:0/1228620737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6d8078de0 0x7fc6d807b270 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.573+0000 7fc6e0085640 1 --2- 172.21.15.169:0/1228620737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d8076430 0x7fc6d80788a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.573+0000 7fc6e0085640 1 --2- 172.21.15.169:0/1228620737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6d8073550 0x7fc6d8073970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.573+0000 7fc6e0085640 1 -- 172.21.15.169:0/1228620737 >> 172.21.15.169:0/1228620737 conn(0x7fc6d806e500 msgr2=0x7fc6d8070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:21.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.575+0000 7fc6e0085640 1 -- 172.21.15.169:0/1228620737 shutdown_connections 2024-03-28T01:01:21.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.575+0000 7fc6e0085640 1 -- 172.21.15.169:0/1228620737 wait complete. 2024-03-28T01:01:21.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.576+0000 7fc6e0085640 1 Processor -- start 2024-03-28T01:01:21.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.576+0000 7fc6e0085640 1 -- start start 2024-03-28T01:01:21.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.576+0000 7fc6e0085640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6d8073550 0x7fc6d81af5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.576+0000 7fc6e0085640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d8076430 0x7fc6d81afb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.576+0000 7fc6e0085640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6d8078de0 0x7fc6d81b6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.576+0000 7fc6e0085640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d807ec20 con 0x7fc6d8078de0 2024-03-28T01:01:21.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.576+0000 7fc6e0085640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d807eaa0 con 0x7fc6d8073550 2024-03-28T01:01:21.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.576+0000 7fc6e0085640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d807eda0 con 0x7fc6d8076430 2024-03-28T01:01:21.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.577+0000 7fc6dde21640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6d8073550 0x7fc6d81af5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:21.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.578+0000 7fc6dd620640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d8076430 0x7fc6d81afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:21.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.578+0000 7fc6dd620640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d8076430 0x7fc6d81afb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:44514/0 (socket says 172.21.15.169:44514) 2024-03-28T01:01:21.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.578+0000 7fc6dd620640 1 -- 172.21.15.169:0/1746460087 learned_addr learned my addr 172.21.15.169:0/1746460087 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:21.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.578+0000 7fc6de622640 1 --2- 172.21.15.169:0/1746460087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6d8078de0 0x7fc6d81b6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:21.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.578+0000 7fc6dd620640 1 -- 172.21.15.169:0/1746460087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6d8073550 msgr2=0x7fc6d81af5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:21.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.578+0000 7fc6dd620640 1 --2- 172.21.15.169:0/1746460087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6d8073550 0x7fc6d81af5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.578+0000 7fc6dd620640 1 -- 172.21.15.169:0/1746460087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6d8078de0 msgr2=0x7fc6d81b6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:21.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.578+0000 7fc6dd620640 1 --2- 172.21.15.169:0/1746460087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6d8078de0 0x7fc6d81b6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.578+0000 7fc6dd620640 1 -- 172.21.15.169:0/1746460087 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d81b6ba0 con 0x7fc6d8076430 2024-03-28T01:01:21.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.582+0000 7fc6dd620640 1 --2- 172.21.15.169:0/1746460087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d8076430 0x7fc6d81afb20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc6c800c890 tx=0x7fc6c800cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:21.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.584+0000 7fc6ceffd640 1 -- 172.21.15.169:0/1746460087 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8004220 con 0x7fc6d8076430 2024-03-28T01:01:21.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.584+0000 7fc6ceffd640 1 -- 172.21.15.169:0/1746460087 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6c80043c0 con 0x7fc6d8076430 2024-03-28T01:01:21.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.585+0000 7fc6ceffd640 1 -- 172.21.15.169:0/1746460087 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8004a00 con 0x7fc6d8076430 2024-03-28T01:01:21.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.586+0000 7fc6e0085640 1 -- 172.21.15.169:0/1746460087 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d81b6e90 con 0x7fc6d8076430 2024-03-28T01:01:21.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.586+0000 7fc6e0085640 1 -- 172.21.15.169:0/1746460087 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d81b7240 con 0x7fc6d8076430 2024-03-28T01:01:21.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.590+0000 7fc6e2c74740 1 Processor -- start 2024-03-28T01:01:21.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.590+0000 7fc6e2c74740 1 -- start start 2024-03-28T01:01:21.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.591+0000 7fc6e2c74740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652f9a09690 0x5652f9a09ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.591+0000 7fc6e2c74740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a0a080 0x5652f9a0eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.591+0000 7fc6e2c74740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652f9a0f1f0 0x5652f9a11680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.591+0000 7fc6e2c74740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652f99cc610 con 0x5652f9a0f1f0 2024-03-28T01:01:21.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.591+0000 7fc6e2c74740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652f996f3c0 con 0x5652f9a0a080 2024-03-28T01:01:21.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.591+0000 7fc6e2c74740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652f9917d60 con 0x5652f9a09690 2024-03-28T01:01:21.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.591+0000 7fc6dde21640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a0a080 0x5652f9a0eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:21.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.591+0000 7fc6dde21640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a0a080 0x5652f9a0eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33900/0 (socket says 172.21.15.169:33900) 2024-03-28T01:01:21.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.591+0000 7fc6dde21640 1 -- 172.21.15.169:0/2101577912 learned_addr learned my addr 172.21.15.169:0/2101577912 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:21.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.591+0000 7fc6de622640 1 --2- 172.21.15.169:0/2101577912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652f9a09690 0x5652f9a09ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:21.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.594+0000 7fc6ceffd640 1 -- 172.21.15.169:0/1746460087 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc6c80079b0 con 0x7fc6d8076430 2024-03-28T01:01:21.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.595+0000 7fc6ceffd640 1 --2- 172.21.15.169:0/1746460087 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc6c40a6c00 0x7fc6c40a90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.595+0000 7fc6dde21640 1 -- 172.21.15.169:0/2101577912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652f9a09690 msgr2=0x5652f9a09ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:21.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.595+0000 7fc6dde21640 1 --2- 172.21.15.169:0/2101577912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652f9a09690 0x5652f9a09ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.595+0000 7fc6dde21640 1 -- 172.21.15.169:0/2101577912 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652f9a0f1f0 msgr2=0x5652f9a11680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:01:21.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.595+0000 7fc6dde21640 1 --2- 172.21.15.169:0/2101577912 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652f9a0f1f0 0x5652f9a11680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.595+0000 7fc6dde21640 1 -- 172.21.15.169:0/2101577912 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652f96833f0 con 0x5652f9a0a080 2024-03-28T01:01:21.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.596+0000 7fc6ceffd640 1 -- 172.21.15.169:0/1746460087 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc6c8099310 con 0x7fc6d8076430 2024-03-28T01:01:21.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.597+0000 7fc6de622640 1 --2- 172.21.15.169:0/1746460087 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc6c40a6c00 0x7fc6c40a90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:21.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.600+0000 7fc6dde21640 1 --2- 172.21.15.169:0/2101577912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a0a080 0x5652f9a0eb80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4009b80 tx=0x7fc6d400e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6de37d262cce57f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:21.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.602+0000 7fc6c3fff640 1 -- 172.21.15.169:0/2101577912 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4016070 con 0x5652f9a0a080 2024-03-28T01:01:21.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.602+0000 7fc6c3fff640 1 -- 172.21.15.169:0/2101577912 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6d4011950 con 0x5652f9a0a080 2024-03-28T01:01:21.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.602+0000 7fc6c3fff640 1 -- 172.21.15.169:0/2101577912 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4010870 con 0x5652f9a0a080 2024-03-28T01:01:21.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.603+0000 7fc6e2c74740 1 -- 172.21.15.169:0/2101577912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a0a080 msgr2=0x5652f9a0eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:21.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.603+0000 7fc6e2c74740 1 --2- 172.21.15.169:0/2101577912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a0a080 0x5652f9a0eb80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4009b80 tx=0x7fc6d400e6c0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.605+0000 7fc6e2c74740 1 -- 172.21.15.169:0/2101577912 shutdown_connections 2024-03-28T01:01:21.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.605+0000 7fc6e2c74740 1 --2- 172.21.15.169:0/2101577912 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652f9a0f1f0 0x5652f9a11680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.605+0000 7fc6e2c74740 1 --2- 172.21.15.169:0/2101577912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a0a080 0x5652f9a0eb80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.605+0000 7fc6e2c74740 1 --2- 172.21.15.169:0/2101577912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652f9a09690 0x5652f9a09ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.605+0000 7fc6e2c74740 1 -- 172.21.15.169:0/2101577912 >> 172.21.15.169:0/2101577912 conn(0x5652f9a04c00 msgr2=0x5652f9a05000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:21.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.606+0000 7fc6de622640 1 --2- 172.21.15.169:0/1746460087 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc6c40a6c00 0x7fc6c40a90c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0034fd0 tx=0x7fc6d0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:21.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.608+0000 7fc6e2c74740 1 -- 172.21.15.169:0/2101577912 shutdown_connections 2024-03-28T01:01:21.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.610+0000 7fc6e2c74740 1 -- 172.21.15.169:0/2101577912 wait complete. 2024-03-28T01:01:21.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.611+0000 7fc6e2c74740 1 -- start start 2024-03-28T01:01:21.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.611+0000 7fc6e2c74740 1 Processor -- start 2024-03-28T01:01:21.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.611+0000 7fc6e2c74740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652f9a09690 0x5652f9a1c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.611+0000 7fc6e2c74740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652f9a0a080 0x5652f9a1c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.612+0000 7fc6e2c74740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a0f1f0 0x5652f9a23820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.612+0000 7fc6e2c74740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652f992b240 con 0x5652f9a09690 2024-03-28T01:01:21.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.612+0000 7fc6e2c74740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652f996b460 con 0x5652f9a0f1f0 2024-03-28T01:01:21.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.612+0000 7fc6e2c74740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652f9a121a0 con 0x5652f9a0a080 2024-03-28T01:01:21.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.612+0000 7fc6e2c74740 10 client.0 _ll_register_callbacks cb 0x5652f992ac10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:21.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.612+0000 7fc6dde21640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652f9a09690 0x5652f9a1c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:21.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.612+0000 7fc6dde21640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652f9a09690 0x5652f9a1c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36274/0 (socket says 172.21.15.169:36274) 2024-03-28T01:01:21.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.612+0000 7fc6dde21640 1 -- 172.21.15.169:0/102884271 learned_addr learned my addr 172.21.15.169:0/102884271 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:21.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.612+0000 7fc6dde21640 1 -- 172.21.15.169:0/102884271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652f9a0a080 msgr2=0x5652f9a1c850 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:01:21.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.612+0000 7fc6dde21640 1 --2- 172.21.15.169:0/102884271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652f9a0a080 0x5652f9a1c850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.612+0000 7fc6dde21640 1 -- 172.21.15.169:0/102884271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a0f1f0 msgr2=0x5652f9a23820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:21.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.612+0000 7fc6dde21640 1 --2- 172.21.15.169:0/102884271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a0f1f0 0x5652f9a23820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.612+0000 7fc6dde21640 1 -- 172.21.15.169:0/102884271 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652f974d7d0 con 0x5652f9a09690 2024-03-28T01:01:21.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.613+0000 7fc6dde21640 1 --2- 172.21.15.169:0/102884271 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652f9a09690 0x5652f9a1c310 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4000c00 tx=0x7fc6d40116a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f09455eb744deb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:21.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.613+0000 7fc6c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:01:21.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.616+0000 7fc6c17fa640 1 -- 172.21.15.169:0/102884271 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4016050 con 0x5652f9a09690 2024-03-28T01:01:21.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.616+0000 7fc6e2c74740 1 -- 172.21.15.169:0/102884271 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc6d4011950 con 0x5652f9a09690 2024-03-28T01:01:21.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.616+0000 7fc6c17fa640 1 -- 172.21.15.169:0/102884271 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc6d4005040 con 0x5652f9a09690 2024-03-28T01:01:21.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.620+0000 7fc6c17fa640 1 -- 172.21.15.169:0/102884271 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6d4005240 con 0x5652f9a09690 2024-03-28T01:01:21.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.621+0000 7fc6e2c74740 10 client.16398 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:21.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.621+0000 7fc6e2c74740 1 -- 172.21.15.169:0/102884271 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5652f9a16600 con 0x5652f9a09690 2024-03-28T01:01:21.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.623+0000 7fc6c17fa640 1 -- 172.21.15.169:0/102884271 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6d4011950 con 0x5652f9a09690 2024-03-28T01:01:21.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.624+0000 7fc6c17fa640 1 -- 172.21.15.169:0/102884271 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fc6d402b030 con 0x5652f9a09690 2024-03-28T01:01:21.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.624+0000 7fc6e2c74740 10 client.16398 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:21.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.624+0000 7fc6e2c74740 10 client.16398 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:01:21.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.624+0000 7fc6e2c74740 20 client.16398 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:21.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.624+0000 7fc6e2c74740 1 --2- 172.21.15.169:0/102884271 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5652f9a17ff0 0x5652f9a1a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.624+0000 7fc6e2c74740 4 client.16398 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:01:21.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.624+0000 7fc6e2c74740 1 -- 172.21.15.169:0/102884271 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5652f997c900 con 0x5652f9a17ff0 2024-03-28T01:01:21.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.626+0000 7fc6dd620640 1 --2- 172.21.15.169:0/102884271 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5652f9a17ff0 0x5652f9a1a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:21.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.629+0000 7fc6dd620640 1 --2- 172.21.15.169:0/102884271 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5652f9a17ff0 0x5652f9a1a430 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=11945b83fff6ac server_cookie=cee6d3d8ed24ac72 in_seq=0 out_seq=0 2024-03-28T01:01:21.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.629+0000 7fc6c17fa640 10 client.16398 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:01:21.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.631+0000 7fc6c17fa640 1 -- 172.21.15.169:0/102884271 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5652f997c900 con 0x5652f9a17ff0 2024-03-28T01:01:21.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.631+0000 7fc6c17fa640 10 client.16398 handle_command_reply: tid=0 2024-03-28T01:01:21.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.631+0000 7fc6e2c74740 1 client.16398 shutdown 2024-03-28T01:01:21.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.634+0000 7fc6e2c74740 1 -- 172.21.15.169:0/102884271 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652f9a09690 msgr2=0x5652f9a1c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:21.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.634+0000 7fc6e2c74740 1 --2- 172.21.15.169:0/102884271 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652f9a09690 0x5652f9a1c310 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4000c00 tx=0x7fc6d40116a0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.634+0000 7fc6e2c74740 1 -- 172.21.15.169:0/102884271 shutdown_connections 2024-03-28T01:01:21.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.634+0000 7fc6e2c74740 1 --2- 172.21.15.169:0/102884271 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5652f9a17ff0 0x5652f9a1a430 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.634+0000 7fc6e2c74740 1 --2- 172.21.15.169:0/102884271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a0f1f0 0x5652f9a23820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.634+0000 7fc6e2c74740 1 --2- 172.21.15.169:0/102884271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652f9a0a080 0x5652f9a1c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.634+0000 7fc6e2c74740 1 --2- 172.21.15.169:0/102884271 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652f9a09690 0x5652f9a1c310 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.634+0000 7fc6e2c74740 1 -- 172.21.15.169:0/102884271 >> 172.21.15.169:0/102884271 conn(0x5652f9a04c00 msgr2=0x5652f9a03e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:21.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.635+0000 7fc6c17fa640 0 client.16398 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:01:21.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.635+0000 7fc6e2c74740 1 -- 172.21.15.169:0/102884271 shutdown_connections 2024-03-28T01:01:21.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.635+0000 7fc6e2c74740 1 -- 172.21.15.169:0/102884271 wait complete. 2024-03-28T01:01:21.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.641+0000 7fc6e2c74740 20 client.16398 trim_cache size 0 max 16384 2024-03-28T01:01:21.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.676+0000 7fc6e2c74740 1 Processor -- start 2024-03-28T01:01:21.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.677+0000 7fc6e2c74740 1 -- start start 2024-03-28T01:01:21.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.677+0000 7fc6e2c74740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652f9a09690 0x5652f9a08570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.677+0000 7fc6e2c74740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652f9a0a080 0x5652f9a237e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.677+0000 7fc6e2c74740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a0f1f0 0x5652f9a38d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.677+0000 7fc6e2c74740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d81c3bb0 con 0x5652f9a0a080 2024-03-28T01:01:21.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.677+0000 7fc6e2c74740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d81c16b0 con 0x5652f9a0f1f0 2024-03-28T01:01:21.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.677+0000 7fc6e2c74740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d81bf800 con 0x5652f9a09690 2024-03-28T01:01:21.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.678+0000 7fc6de622640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a0f1f0 0x5652f9a38d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:21.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.678+0000 7fc6dd620640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652f9a0a080 0x5652f9a237e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:21.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.678+0000 7fc6de622640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a0f1f0 0x5652f9a38d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33908/0 (socket says 172.21.15.169:33908) 2024-03-28T01:01:21.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.678+0000 7fc6dd620640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652f9a0a080 0x5652f9a237e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36290/0 (socket says 172.21.15.169:36290) 2024-03-28T01:01:21.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.678+0000 7fc6de622640 1 -- 172.21.15.169:0/2061003419 learned_addr learned my addr 172.21.15.169:0/2061003419 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:21.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.678+0000 7fc6dde21640 1 --2- 172.21.15.169:0/2061003419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652f9a09690 0x5652f9a08570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:21.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.678+0000 7fc6de622640 1 -- 172.21.15.169:0/2061003419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652f9a09690 msgr2=0x5652f9a08570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:21.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.678+0000 7fc6de622640 1 --2- 172.21.15.169:0/2061003419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652f9a09690 0x5652f9a08570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.678+0000 7fc6de622640 1 -- 172.21.15.169:0/2061003419 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652f9a0a080 msgr2=0x5652f9a237e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:21.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.678+0000 7fc6de622640 1 --2- 172.21.15.169:0/2061003419 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652f9a0a080 0x5652f9a237e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.678+0000 7fc6de622640 1 -- 172.21.15.169:0/2061003419 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652f9a14710 con 0x5652f9a0f1f0 2024-03-28T01:01:21.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.679+0000 7fc6de622640 1 --2- 172.21.15.169:0/2061003419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a0f1f0 0x5652f9a38d90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009be0 tx=0x7fc6d002bef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=295cb47265236e53 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:21.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.681+0000 7fc6c37fe640 1 -- 172.21.15.169:0/2061003419 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d002cda0 con 0x5652f9a0f1f0 2024-03-28T01:01:21.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.682+0000 7fc6c37fe640 1 -- 172.21.15.169:0/2061003419 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6d0034910 con 0x5652f9a0f1f0 2024-03-28T01:01:21.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.683+0000 7fc6e2c74740 1 -- 172.21.15.169:0/2061003419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a0f1f0 msgr2=0x5652f9a38d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:21.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.683+0000 7fc6e2c74740 1 --2- 172.21.15.169:0/2061003419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a0f1f0 0x5652f9a38d90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009be0 tx=0x7fc6d002bef0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.684+0000 7fc6e2c74740 1 -- 172.21.15.169:0/2061003419 shutdown_connections 2024-03-28T01:01:21.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.684+0000 7fc6e2c74740 1 --2- 172.21.15.169:0/2061003419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a0f1f0 0x5652f9a38d90 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.684+0000 7fc6e2c74740 1 --2- 172.21.15.169:0/2061003419 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652f9a0a080 0x5652f9a237e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.684+0000 7fc6e2c74740 1 --2- 172.21.15.169:0/2061003419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652f9a09690 0x5652f9a08570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.684+0000 7fc6e2c74740 1 -- 172.21.15.169:0/2061003419 >> 172.21.15.169:0/2061003419 conn(0x5652f9a04c00 msgr2=0x5652f9a07e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:21.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.686+0000 7fc6e2c74740 1 -- 172.21.15.169:0/2061003419 shutdown_connections 2024-03-28T01:01:21.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.690+0000 7fc6e2c74740 1 -- 172.21.15.169:0/2061003419 wait complete. 2024-03-28T01:01:21.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.690+0000 7fc6e2c74740 1 -- start start 2024-03-28T01:01:21.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.690+0000 7fc6e2c74740 1 Processor -- start 2024-03-28T01:01:21.694 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.691+0000 7fc6e2c74740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a09690 0x5652f9a233b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.694 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.691+0000 7fc6e2c74740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652f9a0a080 0x5652f9a219c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.694 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.691+0000 7fc6e2c74740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652f9a0f1f0 0x5652f9a21f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.694 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.691+0000 7fc6e2c74740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d81bfd70 con 0x5652f9a0a080 2024-03-28T01:01:21.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.691+0000 7fc6e2c74740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d81c1050 con 0x5652f9a09690 2024-03-28T01:01:21.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.691+0000 7fc6e2c74740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d81c0390 con 0x5652f9a0f1f0 2024-03-28T01:01:21.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.691+0000 7fc6e2c74740 10 client.0 _ll_register_callbacks cb 0x7fc6d40097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:21.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.692+0000 7fc6de622640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652f9a0f1f0 0x5652f9a21f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:21.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.692+0000 7fc6de622640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652f9a0f1f0 0x5652f9a21f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:44554/0 (socket says 172.21.15.169:44554) 2024-03-28T01:01:21.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.692+0000 7fc6de622640 1 -- 172.21.15.169:0/363617973 learned_addr learned my addr 172.21.15.169:0/363617973 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:21.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.692+0000 7fc6dde21640 1 --2- 172.21.15.169:0/363617973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a09690 0x5652f9a233b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:21.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.696+0000 7fc6de622640 1 -- 172.21.15.169:0/363617973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a09690 msgr2=0x5652f9a233b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:21.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.696+0000 7fc6de622640 1 --2- 172.21.15.169:0/363617973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a09690 0x5652f9a233b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.696+0000 7fc6de622640 1 -- 172.21.15.169:0/363617973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652f9a0a080 msgr2=0x5652f9a219c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:21.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.696+0000 7fc6de622640 1 --2- 172.21.15.169:0/363617973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652f9a0a080 0x5652f9a219c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.696+0000 7fc6de622640 1 -- 172.21.15.169:0/363617973 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652f9a22440 con 0x5652f9a0f1f0 2024-03-28T01:01:21.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.696+0000 7fc6de622640 1 --2- 172.21.15.169:0/363617973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652f9a0f1f0 0x5652f9a21f00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0004af0 tx=0x7fc6d0033fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67dad3bcff9c9e68 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:21.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.699+0000 7fc6c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:01:21.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.699+0000 7fc6c17fa640 1 -- 172.21.15.169:0/363617973 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d005c070 con 0x5652f9a0f1f0 2024-03-28T01:01:21.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.699+0000 7fc6c17fa640 1 -- 172.21.15.169:0/363617973 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc6d0034ac0 con 0x5652f9a0f1f0 2024-03-28T01:01:21.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.699+0000 7fc6e2c74740 1 -- 172.21.15.169:0/363617973 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc6d0034910 con 0x5652f9a0f1f0 2024-03-28T01:01:21.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.700+0000 7fc6c17fa640 1 -- 172.21.15.169:0/363617973 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6d0066050 con 0x5652f9a0f1f0 2024-03-28T01:01:21.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.700+0000 7fc6e2c74740 10 client.26212 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:21.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.700+0000 7fc6e2c74740 1 -- 172.21.15.169:0/363617973 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5652f9a8c670 con 0x5652f9a0f1f0 2024-03-28T01:01:21.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.700+0000 7fc6c17fa640 1 -- 172.21.15.169:0/363617973 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6d0034910 con 0x5652f9a0f1f0 2024-03-28T01:01:21.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.700+0000 7fc6c17fa640 1 -- 172.21.15.169:0/363617973 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fc6d004c420 con 0x5652f9a0f1f0 2024-03-28T01:01:21.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.702+0000 7fc6e2c74740 10 client.26212 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:21.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.702+0000 7fc6e2c74740 10 client.26212 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:01:21.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.702+0000 7fc6e2c74740 20 client.26212 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:21.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.702+0000 7fc6e2c74740 1 --2- 172.21.15.169:0/363617973 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5652f9a8cfb0 0x5652f9a8f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:21.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.702+0000 7fc6e2c74740 4 client.26212 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:01:21.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.702+0000 7fc6e2c74740 1 -- 172.21.15.169:0/363617973 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5652f9976c50 con 0x5652f9a8cfb0 2024-03-28T01:01:21.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.707+0000 7fc6dde21640 1 --2- 172.21.15.169:0/363617973 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5652f9a8cfb0 0x5652f9a8f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:21.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.763+0000 7fc6dde21640 1 --2- 172.21.15.169:0/363617973 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5652f9a8cfb0 0x5652f9a8f3d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d1c8b3da58103456 server_cookie=c80ac3775525c314 in_seq=0 out_seq=0 2024-03-28T01:01:21.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.763+0000 7fc6c17fa640 10 client.26212 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:01:21.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.764+0000 7fc6c17fa640 1 -- 172.21.15.169:0/363617973 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5652f9976c50 con 0x5652f9a8cfb0 2024-03-28T01:01:21.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.764+0000 7fc6c17fa640 10 client.26212 handle_command_reply: tid=0 2024-03-28T01:01:21.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.764+0000 7fc6e2c74740 1 client.26212 shutdown 2024-03-28T01:01:21.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.764+0000 7fc6e2c74740 1 -- 172.21.15.169:0/363617973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652f9a0f1f0 msgr2=0x5652f9a21f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:21.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.764+0000 7fc6e2c74740 1 --2- 172.21.15.169:0/363617973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652f9a0f1f0 0x5652f9a21f00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0004af0 tx=0x7fc6d0033fa0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.764+0000 7fc6e2c74740 1 -- 172.21.15.169:0/363617973 shutdown_connections 2024-03-28T01:01:21.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.764+0000 7fc6e2c74740 1 --2- 172.21.15.169:0/363617973 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5652f9a8cfb0 0x5652f9a8f3d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.764+0000 7fc6e2c74740 1 --2- 172.21.15.169:0/363617973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652f9a0f1f0 0x5652f9a21f00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.764+0000 7fc6e2c74740 1 --2- 172.21.15.169:0/363617973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652f9a0a080 0x5652f9a219c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.764+0000 7fc6e2c74740 1 --2- 172.21.15.169:0/363617973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652f9a09690 0x5652f9a233b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.764+0000 7fc6e2c74740 1 -- 172.21.15.169:0/363617973 >> 172.21.15.169:0/363617973 conn(0x5652f9a04c00 msgr2=0x5652f9a086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:21.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.765+0000 7fc6c17fa640 0 client.26212 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:01:21.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.766+0000 7fc6e2c74740 1 -- 172.21.15.169:0/363617973 shutdown_connections 2024-03-28T01:01:21.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.766+0000 7fc6e2c74740 1 -- 172.21.15.169:0/363617973 wait complete. 2024-03-28T01:01:21.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.766+0000 7fc6e2c74740 20 client.26212 trim_cache size 0 max 16384 2024-03-28T01:01:21.772 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:01:21.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.770+0000 7fc6e0085640 1 -- 172.21.15.169:0/1746460087 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc6c40a6c00 msgr2=0x7fc6c40a90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:21.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.770+0000 7fc6e0085640 1 --2- 172.21.15.169:0/1746460087 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc6c40a6c00 0x7fc6c40a90c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0034fd0 tx=0x7fc6d0009500 comp rx=0 tx=0).stop 2024-03-28T01:01:21.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.770+0000 7fc6e0085640 1 -- 172.21.15.169:0/1746460087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d8076430 msgr2=0x7fc6d81afb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:21.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.770+0000 7fc6e0085640 1 --2- 172.21.15.169:0/1746460087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d8076430 0x7fc6d81afb20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc6c800c890 tx=0x7fc6c800cd50 comp rx=0 tx=0).stop 2024-03-28T01:01:21.777 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.774+0000 7fc6e0085640 1 -- 172.21.15.169:0/1746460087 shutdown_connections 2024-03-28T01:01:21.777 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.774+0000 7fc6e0085640 1 --2- 172.21.15.169:0/1746460087 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc6c40a6c00 0x7fc6c40a90c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.777 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.774+0000 7fc6e0085640 1 --2- 172.21.15.169:0/1746460087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6d8078de0 0x7fc6d81b6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.777 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.774+0000 7fc6e0085640 1 --2- 172.21.15.169:0/1746460087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d8076430 0x7fc6d81afb20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.777 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.774+0000 7fc6e0085640 1 --2- 172.21.15.169:0/1746460087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6d8073550 0x7fc6d81af5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:21.777 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.774+0000 7fc6e0085640 1 -- 172.21.15.169:0/1746460087 >> 172.21.15.169:0/1746460087 conn(0x7fc6d806e500 msgr2=0x7fc6d8078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:21.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.775+0000 7fc6e0085640 1 -- 172.21.15.169:0/1746460087 shutdown_connections 2024-03-28T01:01:21.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:21.777+0000 7fc6e0085640 1 -- 172.21.15.169:0/1746460087 wait complete. 2024-03-28T01:01:21.824 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:01:21.824 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T01:01:21.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:21 smithi174 ceph-mon[36060]: pgmap v496: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 680 MiB/s wr, 1.46k op/s 2024-03-28T01:01:21.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:21 smithi177 ceph-mon[36206]: pgmap v496: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 680 MiB/s wr, 1.46k op/s 2024-03-28T01:01:22.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.016+0000 7f370b5cd640 1 -- 172.21.15.169:0/2319536263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37041005b0 msgr2=0x7f37041009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:22.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.016+0000 7f370b5cd640 1 --2- 172.21.15.169:0/2319536263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37041005b0 0x7f37041009d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3700009900 tx=0x7f370002bc30 comp rx=0 tx=0).stop 2024-03-28T01:01:22.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.016+0000 7f370b5cd640 1 -- 172.21.15.169:0/2319536263 shutdown_connections 2024-03-28T01:01:22.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.016+0000 7f370b5cd640 1 --2- 172.21.15.169:0/2319536263 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3704102160 0x7f37041089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.016+0000 7f370b5cd640 1 --2- 172.21.15.169:0/2319536263 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37041017a0 0x7f3704101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.016+0000 7f370b5cd640 1 --2- 172.21.15.169:0/2319536263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37041005b0 0x7f37041009d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.016+0000 7f370b5cd640 1 -- 172.21.15.169:0/2319536263 >> 172.21.15.169:0/2319536263 conn(0x7f37040fbb40 msgr2=0x7f37040fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:22.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.016+0000 7f370b5cd640 1 -- 172.21.15.169:0/2319536263 shutdown_connections 2024-03-28T01:01:22.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.016+0000 7f370b5cd640 1 -- 172.21.15.169:0/2319536263 wait complete. 2024-03-28T01:01:22.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.017+0000 7f370b5cd640 1 Processor -- start 2024-03-28T01:01:22.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.017+0000 7f370b5cd640 1 -- start start 2024-03-28T01:01:22.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.017+0000 7f370b5cd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37041005b0 0x7f370406a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:22.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.017+0000 7f370b5cd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37041017a0 0x7f370419c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:22.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.017+0000 7f370b5cd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3704102160 0x7f370419ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:22.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.017+0000 7f370b5cd640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370410c270 con 0x7f37041017a0 2024-03-28T01:01:22.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.017+0000 7f370b5cd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370410c0f0 con 0x7f37041005b0 2024-03-28T01:01:22.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.017+0000 7f370b5cd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370410c3f0 con 0x7f3704102160 2024-03-28T01:01:22.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.018+0000 7f3709369640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37041005b0 0x7f370406a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:22.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.018+0000 7f3709b6a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3704102160 0x7f370419ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:22.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.018+0000 7f3709369640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37041005b0 0x7f370406a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33938/0 (socket says 172.21.15.169:33938) 2024-03-28T01:01:22.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.018+0000 7f3709369640 1 -- 172.21.15.169:0/1795226934 learned_addr learned my addr 172.21.15.169:0/1795226934 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:22.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.018+0000 7f3708b68640 1 --2- 172.21.15.169:0/1795226934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37041017a0 0x7f370419c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:22.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.018+0000 7f3709369640 1 -- 172.21.15.169:0/1795226934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3704102160 msgr2=0x7f370419ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:22.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.018+0000 7f3709369640 1 --2- 172.21.15.169:0/1795226934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3704102160 0x7f370419ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.018+0000 7f3709369640 1 -- 172.21.15.169:0/1795226934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37041017a0 msgr2=0x7f370419c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:22.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.018+0000 7f3709369640 1 --2- 172.21.15.169:0/1795226934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37041017a0 0x7f370419c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.018+0000 7f3709369640 1 -- 172.21.15.169:0/1795226934 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f370419f690 con 0x7f37041005b0 2024-03-28T01:01:22.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.018+0000 7f3709369640 1 --2- 172.21.15.169:0/1795226934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37041005b0 0x7f370406a780 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3700000c00 tx=0x7f370002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:22.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.018+0000 7f36fa7fc640 1 -- 172.21.15.169:0/1795226934 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3700043020 con 0x7f37041005b0 2024-03-28T01:01:22.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.018+0000 7f36fa7fc640 1 -- 172.21.15.169:0/1795226934 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f37000054e0 con 0x7f37041005b0 2024-03-28T01:01:22.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.018+0000 7f370b5cd640 1 -- 172.21.15.169:0/1795226934 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f370419f8c0 con 0x7f37041005b0 2024-03-28T01:01:22.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.018+0000 7f36fa7fc640 1 -- 172.21.15.169:0/1795226934 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3700003c40 con 0x7f37041005b0 2024-03-28T01:01:22.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.018+0000 7f370b5cd640 1 -- 172.21.15.169:0/1795226934 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f370419fe10 con 0x7f37041005b0 2024-03-28T01:01:22.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.022+0000 7f36fa7fc640 1 -- 172.21.15.169:0/1795226934 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3700046020 con 0x7f37041005b0 2024-03-28T01:01:22.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.022+0000 7f370e1bc740 1 Processor -- start 2024-03-28T01:01:22.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.022+0000 7f370e1bc740 1 -- start start 2024-03-28T01:01:22.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.022+0000 7f370e1bc740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5078aa0 0x555ac5078ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:22.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.022+0000 7f370e1bc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5079490 0x555ac507df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:22.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.022+0000 7f370e1bc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ac507e600 0x555ac5080a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:22.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.022+0000 7f370e1bc740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac5037610 con 0x555ac5078aa0 2024-03-28T01:01:22.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.022+0000 7f370e1bc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac4fda3c0 con 0x555ac507e600 2024-03-28T01:01:22.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.022+0000 7f370e1bc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac4f82d60 con 0x555ac5079490 2024-03-28T01:01:22.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.022+0000 7f3709b6a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5078aa0 0x555ac5078ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:22.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.022+0000 7f3709b6a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5078aa0 0x555ac5078ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36332/0 (socket says 172.21.15.169:36332) 2024-03-28T01:01:22.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.022+0000 7f3709b6a640 1 -- 172.21.15.169:0/220190583 learned_addr learned my addr 172.21.15.169:0/220190583 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:22.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.022+0000 7f3709369640 1 --2- 172.21.15.169:0/220190583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5079490 0x555ac507df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:22.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.022+0000 7f3708b68640 1 --2- 172.21.15.169:0/220190583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ac507e600 0x555ac5080a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:22.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.023+0000 7f3709b6a640 1 -- 172.21.15.169:0/220190583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5079490 msgr2=0x555ac507df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:22.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.023+0000 7f3709b6a640 1 --2- 172.21.15.169:0/220190583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5079490 0x555ac507df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.023+0000 7f3709b6a640 1 -- 172.21.15.169:0/220190583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ac507e600 msgr2=0x555ac5080a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:22.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.023+0000 7f3709b6a640 1 --2- 172.21.15.169:0/220190583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ac507e600 0x555ac5080a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.023+0000 7f3709b6a640 1 -- 172.21.15.169:0/220190583 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ac4cee3f0 con 0x555ac5078aa0 2024-03-28T01:01:22.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.023+0000 7f3709369640 1 --2- 172.21.15.169:0/220190583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5079490 0x555ac507df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:01:22.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.023+0000 7f36fa7fc640 1 --2- 172.21.15.169:0/1795226934 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36f00a2800 0x7f36f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:22.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.023+0000 7f3709369640 1 --2- 172.21.15.169:0/1795226934 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36f00a2800 0x7f36f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:22.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.023+0000 7f3709b6a640 1 --2- 172.21.15.169:0/220190583 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5078aa0 0x555ac5078ec0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f36fc009fd0 tx=0x7f36fc0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea7022e2a647bf95 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:22.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.023+0000 7f36fa7fc640 1 -- 172.21.15.169:0/1795226934 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f37000f0e40 con 0x7f37041005b0 2024-03-28T01:01:22.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.023+0000 7f36ef7fe640 1 -- 172.21.15.169:0/220190583 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36fc004d20 con 0x555ac5078aa0 2024-03-28T01:01:22.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.023+0000 7f36ef7fe640 1 -- 172.21.15.169:0/220190583 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36fc004050 con 0x555ac5078aa0 2024-03-28T01:01:22.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.023+0000 7f36ef7fe640 1 -- 172.21.15.169:0/220190583 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36fc005240 con 0x555ac5078aa0 2024-03-28T01:01:22.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.024+0000 7f370e1bc740 1 -- 172.21.15.169:0/220190583 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5078aa0 msgr2=0x555ac5078ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:22.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.024+0000 7f370e1bc740 1 --2- 172.21.15.169:0/220190583 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5078aa0 0x555ac5078ec0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f36fc009fd0 tx=0x7f36fc0118d0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.024+0000 7f3709369640 1 --2- 172.21.15.169:0/1795226934 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36f00a2800 0x7f36f00a4cc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f37000ee910 tx=0x7f37000f4920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:22.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.024+0000 7f370e1bc740 1 -- 172.21.15.169:0/220190583 shutdown_connections 2024-03-28T01:01:22.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.024+0000 7f370e1bc740 1 --2- 172.21.15.169:0/220190583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ac507e600 0x555ac5080a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.024+0000 7f370e1bc740 1 --2- 172.21.15.169:0/220190583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5079490 0x555ac507df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.024+0000 7f370e1bc740 1 --2- 172.21.15.169:0/220190583 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5078aa0 0x555ac5078ec0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.024+0000 7f370e1bc740 1 -- 172.21.15.169:0/220190583 >> 172.21.15.169:0/220190583 conn(0x555ac5074010 msgr2=0x555ac5074410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:22.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.024+0000 7f370e1bc740 1 -- 172.21.15.169:0/220190583 shutdown_connections 2024-03-28T01:01:22.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.024+0000 7f370e1bc740 1 -- 172.21.15.169:0/220190583 wait complete. 2024-03-28T01:01:22.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.024+0000 7f370e1bc740 1 -- start start 2024-03-28T01:01:22.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.024+0000 7f370e1bc740 1 Processor -- start 2024-03-28T01:01:22.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.025+0000 7f370e1bc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5078aa0 0x555ac5082e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:22.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.025+0000 7f370e1bc740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5079490 0x555ac5083350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:22.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.025+0000 7f370e1bc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ac507e600 0x555ac508a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:22.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.025+0000 7f370e1bc740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac4f96240 con 0x555ac5079490 2024-03-28T01:01:22.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.025+0000 7f370e1bc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac4fd6460 con 0x555ac507e600 2024-03-28T01:01:22.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.025+0000 7f370e1bc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac50815b0 con 0x555ac5078aa0 2024-03-28T01:01:22.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.025+0000 7f370e1bc740 10 client.0 _ll_register_callbacks cb 0x555ac4f95c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:22.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.025+0000 7f3709b6a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5078aa0 0x555ac5082e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:22.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.025+0000 7f3708b68640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5079490 0x555ac5083350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:22.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.025+0000 7f3709b6a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5078aa0 0x555ac5082e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:44586/0 (socket says 172.21.15.169:44586) 2024-03-28T01:01:22.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.025+0000 7f3709b6a640 1 -- 172.21.15.169:0/3557138910 learned_addr learned my addr 172.21.15.169:0/3557138910 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:22.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.025+0000 7f3709b6a640 1 --2- 172.21.15.169:0/3557138910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ac507e600 0x555ac508a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:22.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.025+0000 7f3709b6a640 1 -- 172.21.15.169:0/3557138910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ac507e600 msgr2=0x555ac508a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:22.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.025+0000 7f3709b6a640 1 --2- 172.21.15.169:0/3557138910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ac507e600 0x555ac508a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.025+0000 7f3709b6a640 1 -- 172.21.15.169:0/3557138910 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5079490 msgr2=0x555ac5083350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:22.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.025+0000 7f3709b6a640 1 --2- 172.21.15.169:0/3557138910 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5079490 0x555ac5083350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.025+0000 7f3709b6a640 1 -- 172.21.15.169:0/3557138910 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555ac4db87d0 con 0x555ac5078aa0 2024-03-28T01:01:22.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.025+0000 7f3709b6a640 1 --2- 172.21.15.169:0/3557138910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5078aa0 0x555ac5082e10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f36fc011dd0 tx=0x7f36fc000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=633a799c1b23a91f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:22.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.026+0000 7f36ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:01:22.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.026+0000 7f36ecff9640 1 -- 172.21.15.169:0/3557138910 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36fc02c020 con 0x555ac5078aa0 2024-03-28T01:01:22.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.026+0000 7f370e1bc740 1 -- 172.21.15.169:0/3557138910 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36fc004050 con 0x555ac5078aa0 2024-03-28T01:01:22.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.026+0000 7f36ecff9640 1 -- 172.21.15.169:0/3557138910 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f36fc00f6f0 con 0x555ac5078aa0 2024-03-28T01:01:22.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.026+0000 7f36ecff9640 1 -- 172.21.15.169:0/3557138910 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36fc037050 con 0x555ac5078aa0 2024-03-28T01:01:22.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.026+0000 7f370e1bc740 10 client.26224 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:22.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.026+0000 7f370e1bc740 1 -- 172.21.15.169:0/3557138910 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x555ac508d750 con 0x555ac5078aa0 2024-03-28T01:01:22.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.026+0000 7f36ecff9640 1 -- 172.21.15.169:0/3557138910 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36fc004050 con 0x555ac5078aa0 2024-03-28T01:01:22.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.026+0000 7f36ecff9640 1 -- 172.21.15.169:0/3557138910 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f36fc028750 con 0x555ac5078aa0 2024-03-28T01:01:22.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.026+0000 7f370e1bc740 10 client.26224 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:22.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.026+0000 7f370e1bc740 10 client.26224 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:01:22.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.026+0000 7f370e1bc740 20 client.26224 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:22.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.026+0000 7f370e1bc740 1 --2- 172.21.15.169:0/3557138910 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555ac508ec50 0x555ac5091040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:22.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.026+0000 7f370e1bc740 4 client.26224 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:01:22.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.026+0000 7f370e1bc740 1 -- 172.21.15.169:0/3557138910 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555ac4fe7900 con 0x555ac508ec50 2024-03-28T01:01:22.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.027+0000 7f3708b68640 1 --2- 172.21.15.169:0/3557138910 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555ac508ec50 0x555ac5091040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:22.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.027+0000 7f36ecff9640 10 client.26224 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:22.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.027+0000 7f3708b68640 1 --2- 172.21.15.169:0/3557138910 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555ac508ec50 0x555ac5091040 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b873570e56f0606 server_cookie=74ba784e29787086 in_seq=0 out_seq=0 2024-03-28T01:01:22.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.028+0000 7f36ecff9640 1 -- 172.21.15.169:0/3557138910 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555ac4fe7900 con 0x555ac508ec50 2024-03-28T01:01:22.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.028+0000 7f36ecff9640 10 client.26224 handle_command_reply: tid=0 2024-03-28T01:01:22.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.028+0000 7f370e1bc740 1 client.26224 shutdown 2024-03-28T01:01:22.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.028+0000 7f370e1bc740 1 -- 172.21.15.169:0/3557138910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5078aa0 msgr2=0x555ac5082e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:22.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.028+0000 7f370e1bc740 1 --2- 172.21.15.169:0/3557138910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5078aa0 0x555ac5082e10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f36fc011dd0 tx=0x7f36fc000f40 comp rx=0 tx=0).stop 2024-03-28T01:01:22.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.028+0000 7f370e1bc740 1 -- 172.21.15.169:0/3557138910 shutdown_connections 2024-03-28T01:01:22.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.028+0000 7f370e1bc740 1 --2- 172.21.15.169:0/3557138910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ac507e600 0x555ac508a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.028+0000 7f370e1bc740 1 --2- 172.21.15.169:0/3557138910 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555ac508ec50 0x555ac5091040 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.028+0000 7f370e1bc740 1 --2- 172.21.15.169:0/3557138910 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5079490 0x555ac5083350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.028+0000 7f36ecff9640 0 client.26224 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:22.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.028+0000 7f370e1bc740 1 --2- 172.21.15.169:0/3557138910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5078aa0 0x555ac5082e10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.028+0000 7f370e1bc740 1 -- 172.21.15.169:0/3557138910 >> 172.21.15.169:0/3557138910 conn(0x555ac5074010 msgr2=0x555ac506ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:22.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.028+0000 7f370e1bc740 1 -- 172.21.15.169:0/3557138910 shutdown_connections 2024-03-28T01:01:22.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.028+0000 7f370e1bc740 1 -- 172.21.15.169:0/3557138910 wait complete. 2024-03-28T01:01:22.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.028+0000 7f370e1bc740 20 client.26224 trim_cache size 0 max 16384 2024-03-28T01:01:22.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.061+0000 7f370e1bc740 1 Processor -- start 2024-03-28T01:01:22.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.061+0000 7f370e1bc740 1 -- start start 2024-03-28T01:01:22.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.061+0000 7f370e1bc740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5078aa0 0x555ac50832b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:22.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.061+0000 7f370e1bc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5079490 0x555ac5077980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:22.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.061+0000 7f370e1bc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ac507e600 0x555ac50725b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:22.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.061+0000 7f370e1bc740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37041aa4d0 con 0x555ac5078aa0 2024-03-28T01:01:22.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.061+0000 7f370e1bc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37041a7fd0 con 0x555ac507e600 2024-03-28T01:01:22.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.061+0000 7f370e1bc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37041a6070 con 0x555ac5079490 2024-03-28T01:01:22.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.062+0000 7f3709b6a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5078aa0 0x555ac50832b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:22.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.062+0000 7f3709b6a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5078aa0 0x555ac50832b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36350/0 (socket says 172.21.15.169:36350) 2024-03-28T01:01:22.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.062+0000 7f3709b6a640 1 -- 172.21.15.169:0/3590224713 learned_addr learned my addr 172.21.15.169:0/3590224713 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:22.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.062+0000 7f3709b6a640 1 --2- 172.21.15.169:0/3590224713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ac507e600 0x555ac50725b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:22.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.062+0000 7f3709b6a640 1 -- 172.21.15.169:0/3590224713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5079490 msgr2=0x555ac5077980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:01:22.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.062+0000 7f3709b6a640 1 --2- 172.21.15.169:0/3590224713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5079490 0x555ac5077980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.062+0000 7f3709b6a640 1 -- 172.21.15.169:0/3590224713 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5078aa0 msgr2=0x555ac50832b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:22.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.062+0000 7f3709b6a640 1 --2- 172.21.15.169:0/3590224713 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5078aa0 0x555ac50832b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.062+0000 7f3709b6a640 1 -- 172.21.15.169:0/3590224713 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ac506b480 con 0x555ac507e600 2024-03-28T01:01:22.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.063+0000 7f3709b6a640 1 --2- 172.21.15.169:0/3590224713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ac507e600 0x555ac50725b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f36fc028bc0 tx=0x7f36fc0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=687d3d55ca982ea3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:22.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.064+0000 7f36eeffd640 1 -- 172.21.15.169:0/3590224713 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36fc040070 con 0x555ac507e600 2024-03-28T01:01:22.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.068+0000 7f36eeffd640 1 -- 172.21.15.169:0/3590224713 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36fc004550 con 0x555ac507e600 2024-03-28T01:01:22.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.069+0000 7f36eeffd640 1 -- 172.21.15.169:0/3590224713 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36fc02c040 con 0x555ac507e600 2024-03-28T01:01:22.073 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.070+0000 7f370e1bc740 1 -- 172.21.15.169:0/3590224713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ac507e600 msgr2=0x555ac50725b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:22.073 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.070+0000 7f370e1bc740 1 --2- 172.21.15.169:0/3590224713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ac507e600 0x555ac50725b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f36fc028bc0 tx=0x7f36fc0118d0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.073+0000 7f370e1bc740 1 -- 172.21.15.169:0/3590224713 shutdown_connections 2024-03-28T01:01:22.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.073+0000 7f370e1bc740 1 --2- 172.21.15.169:0/3590224713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ac507e600 0x555ac50725b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.073+0000 7f370e1bc740 1 --2- 172.21.15.169:0/3590224713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5079490 0x555ac5077980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.073+0000 7f370e1bc740 1 --2- 172.21.15.169:0/3590224713 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5078aa0 0x555ac50832b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.073+0000 7f370e1bc740 1 -- 172.21.15.169:0/3590224713 >> 172.21.15.169:0/3590224713 conn(0x555ac5074010 msgr2=0x555ac5077270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:22.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.078+0000 7f370e1bc740 1 -- 172.21.15.169:0/3590224713 shutdown_connections 2024-03-28T01:01:22.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.079+0000 7f370e1bc740 1 -- 172.21.15.169:0/3590224713 wait complete. 2024-03-28T01:01:22.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.079+0000 7f370e1bc740 1 -- start start 2024-03-28T01:01:22.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.080+0000 7f370e1bc740 1 Processor -- start 2024-03-28T01:01:22.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.082+0000 7f370e1bc740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5078aa0 0x555ac5086290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:22.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.082+0000 7f370e1bc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5079490 0x555ac50887e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:22.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.082+0000 7f370e1bc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ac507e600 0x555ac5088d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:22.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.082+0000 7f370e1bc740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37041a6670 con 0x555ac5078aa0 2024-03-28T01:01:22.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.082+0000 7f370e1bc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37041a7970 con 0x555ac507e600 2024-03-28T01:01:22.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.082+0000 7f370e1bc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37041a6cb0 con 0x555ac5079490 2024-03-28T01:01:22.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.082+0000 7f370e1bc740 10 client.0 _ll_register_callbacks cb 0x7f36fc004200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:22.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.082+0000 7f3708b68640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5079490 0x555ac50887e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:22.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.083+0000 7f3708b68640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5079490 0x555ac50887e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:44606/0 (socket says 172.21.15.169:44606) 2024-03-28T01:01:22.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.083+0000 7f3708b68640 1 -- 172.21.15.169:0/2043699703 learned_addr learned my addr 172.21.15.169:0/2043699703 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:22.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.083+0000 7f3709b6a640 1 --2- 172.21.15.169:0/2043699703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ac507e600 0x555ac5088d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:22.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.083+0000 7f3708b68640 1 -- 172.21.15.169:0/2043699703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ac507e600 msgr2=0x555ac5088d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:22.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.083+0000 7f3708b68640 1 --2- 172.21.15.169:0/2043699703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ac507e600 0x555ac5088d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.083+0000 7f3708b68640 1 -- 172.21.15.169:0/2043699703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5078aa0 msgr2=0x555ac5086290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:22.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.083+0000 7f3708b68640 1 --2- 172.21.15.169:0/2043699703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5078aa0 0x555ac5086290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.083+0000 7f3708b68640 1 -- 172.21.15.169:0/2043699703 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555ac50913b0 con 0x555ac5079490 2024-03-28T01:01:22.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.083+0000 7f3708b68640 1 --2- 172.21.15.169:0/2043699703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5079490 0x555ac50887e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x555ac50918b0 tx=0x7f36f4004560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ad998d5bc1c1d77 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:22.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.085+0000 7f36ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:01:22.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.085+0000 7f36ecff9640 1 -- 172.21.15.169:0/2043699703 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f40202f0 con 0x555ac5079490 2024-03-28T01:01:22.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.085+0000 7f36ecff9640 1 -- 172.21.15.169:0/2043699703 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f36f4020770 con 0x555ac5079490 2024-03-28T01:01:22.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.086+0000 7f370e1bc740 1 -- 172.21.15.169:0/2043699703 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36fc004550 con 0x555ac5079490 2024-03-28T01:01:22.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.088+0000 7f36ecff9640 1 -- 172.21.15.169:0/2043699703 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36f4020970 con 0x555ac5079490 2024-03-28T01:01:22.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.088+0000 7f370e1bc740 10 client.26227 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:22.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.088+0000 7f370e1bc740 1 -- 172.21.15.169:0/2043699703 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x555ac508a5d0 con 0x555ac5079490 2024-03-28T01:01:22.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.089+0000 7f36ecff9640 1 -- 172.21.15.169:0/2043699703 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36fc004550 con 0x555ac5079490 2024-03-28T01:01:22.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.089+0000 7f36ecff9640 1 -- 172.21.15.169:0/2043699703 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f36f4010430 con 0x555ac5079490 2024-03-28T01:01:22.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.090+0000 7f370e1bc740 10 client.26227 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:22.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.090+0000 7f370e1bc740 10 client.26227 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:01:22.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.090+0000 7f370e1bc740 20 client.26227 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:22.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.090+0000 7f370e1bc740 1 --2- 172.21.15.169:0/2043699703 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555ac50e6ad0 0x555ac50e8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:22.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.090+0000 7f370e1bc740 4 client.26227 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:01:22.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.090+0000 7f370e1bc740 1 -- 172.21.15.169:0/2043699703 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555ac4fe1c50 con 0x555ac50e6ad0 2024-03-28T01:01:22.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.092+0000 7f3709b6a640 1 --2- 172.21.15.169:0/2043699703 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555ac50e6ad0 0x555ac50e8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:22.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.092+0000 7f3709b6a640 1 --2- 172.21.15.169:0/2043699703 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555ac50e6ad0 0x555ac50e8ef0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef05da9b96928ac3 server_cookie=44a4ecafbccb863d in_seq=0 out_seq=0 2024-03-28T01:01:22.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.095+0000 7f36ecff9640 10 client.26227 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:22.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.095+0000 7f36ecff9640 1 -- 172.21.15.169:0/2043699703 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555ac4fe1c50 con 0x555ac50e6ad0 2024-03-28T01:01:22.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.096+0000 7f36ecff9640 10 client.26227 handle_command_reply: tid=0 2024-03-28T01:01:22.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.096+0000 7f370e1bc740 1 client.26227 shutdown 2024-03-28T01:01:22.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.099+0000 7f370e1bc740 1 -- 172.21.15.169:0/2043699703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5079490 msgr2=0x555ac50887e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:22.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.099+0000 7f370e1bc740 1 --2- 172.21.15.169:0/2043699703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5079490 0x555ac50887e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x555ac50918b0 tx=0x7f36f4004560 comp rx=0 tx=0).stop 2024-03-28T01:01:22.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.101+0000 7f370e1bc740 1 -- 172.21.15.169:0/2043699703 shutdown_connections 2024-03-28T01:01:22.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.101+0000 7f370e1bc740 1 --2- 172.21.15.169:0/2043699703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ac507e600 0x555ac5088d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.101+0000 7f370e1bc740 1 --2- 172.21.15.169:0/2043699703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ac5079490 0x555ac50887e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.101+0000 7f370e1bc740 1 --2- 172.21.15.169:0/2043699703 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x555ac50e6ad0 0x555ac50e8ef0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.101+0000 7f370e1bc740 1 --2- 172.21.15.169:0/2043699703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x555ac5078aa0 0x555ac5086290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.101+0000 7f370e1bc740 1 -- 172.21.15.169:0/2043699703 >> 172.21.15.169:0/2043699703 conn(0x555ac5074010 msgr2=0x555ac5077b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:22.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.102+0000 7f36ecff9640 0 client.26227 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:22.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.102+0000 7f370e1bc740 1 -- 172.21.15.169:0/2043699703 shutdown_connections 2024-03-28T01:01:22.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.102+0000 7f370e1bc740 1 -- 172.21.15.169:0/2043699703 wait complete. 2024-03-28T01:01:22.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.103+0000 7f370e1bc740 20 client.26227 trim_cache size 0 max 16384 2024-03-28T01:01:22.107 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:01:22.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.106+0000 7f370b5cd640 1 -- 172.21.15.169:0/1795226934 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36f00a2800 msgr2=0x7f36f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:22.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.106+0000 7f370b5cd640 1 --2- 172.21.15.169:0/1795226934 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36f00a2800 0x7f36f00a4cc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f37000ee910 tx=0x7f37000f4920 comp rx=0 tx=0).stop 2024-03-28T01:01:22.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.106+0000 7f370b5cd640 1 -- 172.21.15.169:0/1795226934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37041005b0 msgr2=0x7f370406a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:22.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.106+0000 7f370b5cd640 1 --2- 172.21.15.169:0/1795226934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37041005b0 0x7f370406a780 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3700000c00 tx=0x7f370002c650 comp rx=0 tx=0).stop 2024-03-28T01:01:22.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.106+0000 7f370b5cd640 1 -- 172.21.15.169:0/1795226934 shutdown_connections 2024-03-28T01:01:22.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.106+0000 7f370b5cd640 1 --2- 172.21.15.169:0/1795226934 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36f00a2800 0x7f36f00a4cc0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.106+0000 7f370b5cd640 1 --2- 172.21.15.169:0/1795226934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3704102160 0x7f370419ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.106+0000 7f370b5cd640 1 --2- 172.21.15.169:0/1795226934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37041017a0 0x7f370419c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.106+0000 7f370b5cd640 1 --2- 172.21.15.169:0/1795226934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37041005b0 0x7f370406a780 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:22.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.106+0000 7f370b5cd640 1 -- 172.21.15.169:0/1795226934 >> 172.21.15.169:0/1795226934 conn(0x7f37040fbb40 msgr2=0x7f3704104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:22.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.108+0000 7f370b5cd640 1 -- 172.21.15.169:0/1795226934 shutdown_connections 2024-03-28T01:01:22.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:22.108+0000 7f370b5cd640 1 -- 172.21.15.169:0/1795226934 wait complete. 2024-03-28T01:01:23.158 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T01:01:23.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T01:01:23.159 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T01:01:23.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:23 smithi169 ceph-mon[32274]: pgmap v497: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 743 MiB/s wr, 1.30k op/s 2024-03-28T01:01:23.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:23 smithi174 ceph-mon[36060]: pgmap v497: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 743 MiB/s wr, 1.30k op/s 2024-03-28T01:01:23.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:23 smithi177 ceph-mon[36206]: pgmap v497: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 743 MiB/s wr, 1.30k op/s 2024-03-28T01:01:23.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.957+0000 7f2ca0063640 1 -- 172.21.15.169:0/3132264904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c9810d9c0 msgr2=0x7f2c9810de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:23.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.957+0000 7f2ca0063640 1 --2- 172.21.15.169:0/3132264904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c9810d9c0 0x7f2c9810de40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c009900 tx=0x7f2c8c02bc60 comp rx=0 tx=0).stop 2024-03-28T01:01:23.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.958+0000 7f2ca0063640 1 -- 172.21.15.169:0/3132264904 shutdown_connections 2024-03-28T01:01:23.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.958+0000 7f2ca0063640 1 --2- 172.21.15.169:0/3132264904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c9810d9c0 0x7f2c9810de40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:23.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.958+0000 7f2ca0063640 1 --2- 172.21.15.169:0/3132264904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2c9810b010 0x7f2c9810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:23.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.958+0000 7f2ca0063640 1 --2- 172.21.15.169:0/3132264904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c981086a0 0x7f2c9810aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:23.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.958+0000 7f2ca0063640 1 -- 172.21.15.169:0/3132264904 >> 172.21.15.169:0/3132264904 conn(0x7f2c9806e500 msgr2=0x7f2c98070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:23.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.962+0000 7f2ca0063640 1 -- 172.21.15.169:0/3132264904 shutdown_connections 2024-03-28T01:01:23.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.964+0000 7f2ca0063640 1 -- 172.21.15.169:0/3132264904 wait complete. 2024-03-28T01:01:23.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.965+0000 7f2ca0063640 1 Processor -- start 2024-03-28T01:01:23.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.965+0000 7f2ca0063640 1 -- start start 2024-03-28T01:01:23.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.966+0000 7f2ca0063640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c981086a0 0x7f2c98118840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:23.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.966+0000 7f2ca0063640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c9810b010 0x7f2c98118d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:23.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.966+0000 7f2ca0063640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2c9810d9c0 0x7f2c981192c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:23.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.966+0000 7f2ca0063640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c981110d0 con 0x7f2c9810d9c0 2024-03-28T01:01:23.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.966+0000 7f2ca0063640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c98110f50 con 0x7f2c981086a0 2024-03-28T01:01:23.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.966+0000 7f2ca0063640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c98111250 con 0x7f2c9810b010 2024-03-28T01:01:23.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.972+0000 7f2c9ddff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c981086a0 0x7f2c98118840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:23.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.972+0000 7f2c9ddff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c981086a0 0x7f2c98118840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33982/0 (socket says 172.21.15.169:33982) 2024-03-28T01:01:23.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.972+0000 7f2c9ddff640 1 -- 172.21.15.169:0/3025032036 learned_addr learned my addr 172.21.15.169:0/3025032036 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:23.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.973+0000 7f2c9e600640 1 --2- 172.21.15.169:0/3025032036 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2c9810d9c0 0x7f2c981192c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:23.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.973+0000 7f2c9d5fe640 1 --2- 172.21.15.169:0/3025032036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c9810b010 0x7f2c98118d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:23.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.974+0000 7f2c9ddff640 1 -- 172.21.15.169:0/3025032036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c9810b010 msgr2=0x7f2c98118d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:23.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.974+0000 7f2c9ddff640 1 --2- 172.21.15.169:0/3025032036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c9810b010 0x7f2c98118d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:23.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.974+0000 7f2c9ddff640 1 -- 172.21.15.169:0/3025032036 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2c9810d9c0 msgr2=0x7f2c981192c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:23.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.974+0000 7f2c9ddff640 1 --2- 172.21.15.169:0/3025032036 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2c9810d9c0 0x7f2c981192c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:23.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.974+0000 7f2c9ddff640 1 -- 172.21.15.169:0/3025032036 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c98120f40 con 0x7f2c981086a0 2024-03-28T01:01:23.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.975+0000 7f2c9ddff640 1 --2- 172.21.15.169:0/3025032036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c981086a0 0x7f2c98118840 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2c8800c890 tx=0x7f2c8800cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:23.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.978+0000 7f2c96ffd640 1 -- 172.21.15.169:0/3025032036 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c88004220 con 0x7f2c981086a0 2024-03-28T01:01:23.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.979+0000 7f2ca0063640 1 -- 172.21.15.169:0/3025032036 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c98121230 con 0x7f2c981086a0 2024-03-28T01:01:23.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.979+0000 7f2ca0063640 1 -- 172.21.15.169:0/3025032036 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c98121780 con 0x7f2c981086a0 2024-03-28T01:01:23.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.979+0000 7f2c96ffd640 1 -- 172.21.15.169:0/3025032036 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c880043c0 con 0x7f2c981086a0 2024-03-28T01:01:23.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.979+0000 7f2c96ffd640 1 -- 172.21.15.169:0/3025032036 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c880049c0 con 0x7f2c981086a0 2024-03-28T01:01:23.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.987+0000 7f2ca2c49740 1 Processor -- start 2024-03-28T01:01:23.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.993+0000 7f2c96ffd640 1 -- 172.21.15.169:0/3025032036 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2c88004c80 con 0x7f2c981086a0 2024-03-28T01:01:23.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.994+0000 7f2ca2c49740 1 -- start start 2024-03-28T01:01:23.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.995+0000 7f2ca2c49740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632ece1a960 0x5632ece1ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:23.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.995+0000 7f2ca2c49740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece1b350 0x5632ece1fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:23.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.995+0000 7f2ca2c49740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5632ece204c0 0x5632ece22950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:23.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.995+0000 7f2ca2c49740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632ecd5ac90 con 0x5632ece1b350 2024-03-28T01:01:23.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.995+0000 7f2ca2c49740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632ecd80070 con 0x5632ece1a960 2024-03-28T01:01:23.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.995+0000 7f2ca2c49740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632ecd28c80 con 0x5632ece204c0 2024-03-28T01:01:24.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.996+0000 7f2c96ffd640 1 --2- 172.21.15.169:0/3025032036 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2c800a29d0 0x7f2c800a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:24.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.997+0000 7f2c96ffd640 1 -- 172.21.15.169:0/3025032036 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2c880cead0 con 0x7f2c981086a0 2024-03-28T01:01:24.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.997+0000 7f2c9e600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632ece1a960 0x5632ece1ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:24.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.997+0000 7f2c9e600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632ece1a960 0x5632ece1ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33986/0 (socket says 172.21.15.169:33986) 2024-03-28T01:01:24.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.997+0000 7f2c9e600640 1 -- 172.21.15.169:0/182111251 learned_addr learned my addr 172.21.15.169:0/182111251 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:24.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.997+0000 7f2c9d5fe640 1 --2- 172.21.15.169:0/182111251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5632ece204c0 0x5632ece22950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:24.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:23.998+0000 7f2c9ddff640 1 --2- 172.21.15.169:0/182111251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece1b350 0x5632ece1fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:24.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.002+0000 7f2c9e600640 1 --2- 172.21.15.169:0/3025032036 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2c800a29d0 0x7f2c800a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:24.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.003+0000 7f2c9ddff640 1 -- 172.21.15.169:0/182111251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5632ece204c0 msgr2=0x5632ece22950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:24.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.003+0000 7f2c9ddff640 1 --2- 172.21.15.169:0/182111251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5632ece204c0 0x5632ece22950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:24.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.003+0000 7f2c9ddff640 1 -- 172.21.15.169:0/182111251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632ece1a960 msgr2=0x5632ece1ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:24.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.003+0000 7f2c9ddff640 1 --2- 172.21.15.169:0/182111251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632ece1a960 0x5632ece1ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:24.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.003+0000 7f2c9ddff640 1 -- 172.21.15.169:0/182111251 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5632ecd20d10 con 0x5632ece1b350 2024-03-28T01:01:24.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.007+0000 7f2c9ddff640 1 --2- 172.21.15.169:0/182111251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece1b350 0x5632ece1fe50 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2c880055f0 tx=0x7f2c8809abb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb6de881ebc28906 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:24.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.008+0000 7f2c9e600640 1 --2- 172.21.15.169:0/3025032036 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2c800a29d0 0x7f2c800a4e90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c005ec0 tx=0x7f2c8c02bc60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:24.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.008+0000 7f2c7ffff640 1 -- 172.21.15.169:0/182111251 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8801d040 con 0x5632ece1b350 2024-03-28T01:01:24.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.008+0000 7f2c7ffff640 1 -- 172.21.15.169:0/182111251 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c8809b070 con 0x5632ece1b350 2024-03-28T01:01:24.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.010+0000 7f2ca2c49740 1 -- 172.21.15.169:0/182111251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece1b350 msgr2=0x5632ece1fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:24.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.010+0000 7f2ca2c49740 1 --2- 172.21.15.169:0/182111251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece1b350 0x5632ece1fe50 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2c880055f0 tx=0x7f2c8809abb0 comp rx=0 tx=0).stop 2024-03-28T01:01:24.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.010+0000 7f2ca2c49740 1 -- 172.21.15.169:0/182111251 shutdown_connections 2024-03-28T01:01:24.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.010+0000 7f2ca2c49740 1 --2- 172.21.15.169:0/182111251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5632ece204c0 0x5632ece22950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:24.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.010+0000 7f2ca2c49740 1 --2- 172.21.15.169:0/182111251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece1b350 0x5632ece1fe50 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:24.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.010+0000 7f2ca2c49740 1 --2- 172.21.15.169:0/182111251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632ece1a960 0x5632ece1ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:24.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.010+0000 7f2ca2c49740 1 -- 172.21.15.169:0/182111251 >> 172.21.15.169:0/182111251 conn(0x5632ecde31a0 msgr2=0x5632ece162d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:24.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.017+0000 7f2ca2c49740 1 -- 172.21.15.169:0/182111251 shutdown_connections 2024-03-28T01:01:24.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.017+0000 7f2ca2c49740 1 -- 172.21.15.169:0/182111251 wait complete. 2024-03-28T01:01:24.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.018+0000 7f2ca2c49740 1 -- start start 2024-03-28T01:01:24.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.018+0000 7f2ca2c49740 1 Processor -- start 2024-03-28T01:01:24.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.019+0000 7f2ca2c49740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5632ece1a960 0x5632ece249a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:24.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.019+0000 7f2ca2c49740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece1b350 0x5632ece24ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:24.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.019+0000 7f2ca2c49740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632ece204c0 0x5632ece2bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:24.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.019+0000 7f2ca2c49740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632ecd3c0d0 con 0x5632ece1b350 2024-03-28T01:01:24.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.019+0000 7f2ca2c49740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632ecd7c110 con 0x5632ece204c0 2024-03-28T01:01:24.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.019+0000 7f2ca2c49740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632ece23400 con 0x5632ece1a960 2024-03-28T01:01:24.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.019+0000 7f2ca2c49740 10 client.0 _ll_register_callbacks cb 0x5632ecd77e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:24.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.022+0000 7f2c9e600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632ece204c0 0x5632ece2bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:24.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.022+0000 7f2c9ddff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5632ece1a960 0x5632ece249a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:24.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.022+0000 7f2c9d5fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece1b350 0x5632ece24ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:24.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.022+0000 7f2c9d5fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece1b350 0x5632ece24ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36372/0 (socket says 172.21.15.169:36372) 2024-03-28T01:01:24.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.022+0000 7f2c9d5fe640 1 -- 172.21.15.169:0/3718561744 learned_addr learned my addr 172.21.15.169:0/3718561744 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:24.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.023+0000 7f2c9d5fe640 1 -- 172.21.15.169:0/3718561744 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5632ece1a960 msgr2=0x5632ece249a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:24.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.023+0000 7f2c9d5fe640 1 --2- 172.21.15.169:0/3718561744 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5632ece1a960 0x5632ece249a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:24.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.023+0000 7f2c9d5fe640 1 -- 172.21.15.169:0/3718561744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632ece204c0 msgr2=0x5632ece2bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:24.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.023+0000 7f2c9d5fe640 1 --2- 172.21.15.169:0/3718561744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632ece204c0 0x5632ece2bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:24.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.023+0000 7f2c9d5fe640 1 -- 172.21.15.169:0/3718561744 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5632ecb5c7e0 con 0x5632ece1b350 2024-03-28T01:01:24.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.025+0000 7f2c9d5fe640 1 --2- 172.21.15.169:0/3718561744 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece1b350 0x5632ece24ee0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2c9000f790 tx=0x7f2c9000fc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=467a88e49379ce71 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:24.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.026+0000 7f2c7d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:01:24.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.028+0000 7f2c7d7fa640 1 -- 172.21.15.169:0/3718561744 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c90017070 con 0x5632ece1b350 2024-03-28T01:01:24.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.028+0000 7f2ca2c49740 1 -- 172.21.15.169:0/3718561744 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2c8809b070 con 0x5632ece1b350 2024-03-28T01:01:24.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.028+0000 7f2c7d7fa640 1 -- 172.21.15.169:0/3718561744 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2c9000d970 con 0x5632ece1b350 2024-03-28T01:01:24.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.029+0000 7f2c7d7fa640 1 -- 172.21.15.169:0/3718561744 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c90023050 con 0x5632ece1b350 2024-03-28T01:01:24.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.034+0000 7f2ca2c49740 10 client.16425 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:24.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.034+0000 7f2ca2c49740 1 -- 172.21.15.169:0/3718561744 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5632ecbbcf70 con 0x5632ece1b350 2024-03-28T01:01:24.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.038+0000 7f2c7d7fa640 1 -- 172.21.15.169:0/3718561744 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c8809b070 con 0x5632ece1b350 2024-03-28T01:01:24.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.038+0000 7f2c7d7fa640 1 -- 172.21.15.169:0/3718561744 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2c900123e0 con 0x5632ece1b350 2024-03-28T01:01:24.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.038+0000 7f2ca2c49740 10 client.16425 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:24.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.038+0000 7f2ca2c49740 10 client.16425 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:01:24.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.038+0000 7f2ca2c49740 20 client.16425 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:24.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.038+0000 7f2ca2c49740 1 --2- 172.21.15.169:0/3718561744 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5632ece52cb0 0x5632ece550f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:24.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.038+0000 7f2ca2c49740 4 client.16425 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:01:24.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.038+0000 7f2ca2c49740 1 -- 172.21.15.169:0/3718561744 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5632ecd8d5b0 con 0x5632ece52cb0 2024-03-28T01:01:24.049 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.046+0000 7f2c9ddff640 1 --2- 172.21.15.169:0/3718561744 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5632ece52cb0 0x5632ece550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:24.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.057+0000 7f2c9ddff640 1 --2- 172.21.15.169:0/3718561744 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5632ece52cb0 0x5632ece550f0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdcd843008109c59 server_cookie=d97edbe6c936355c in_seq=0 out_seq=0 2024-03-28T01:01:24.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.058+0000 7f2c7d7fa640 10 client.16425 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:24.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.058+0000 7f2c7d7fa640 1 -- 172.21.15.169:0/3718561744 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5632ecd8d5b0 con 0x5632ece52cb0 2024-03-28T01:01:24.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.058+0000 7f2c7d7fa640 10 client.16425 handle_command_reply: tid=0 2024-03-28T01:01:24.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.058+0000 7f2ca2c49740 1 client.16425 shutdown 2024-03-28T01:01:24.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.065+0000 7f2ca2c49740 1 -- 172.21.15.169:0/3718561744 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece1b350 msgr2=0x5632ece24ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:24.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.065+0000 7f2ca2c49740 1 --2- 172.21.15.169:0/3718561744 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece1b350 0x5632ece24ee0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2c9000f790 tx=0x7f2c9000fc50 comp rx=0 tx=0).stop 2024-03-28T01:01:24.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.067+0000 7f2ca2c49740 1 -- 172.21.15.169:0/3718561744 shutdown_connections 2024-03-28T01:01:24.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.067+0000 7f2ca2c49740 1 --2- 172.21.15.169:0/3718561744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632ece204c0 0x5632ece2bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:24.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.067+0000 7f2ca2c49740 1 --2- 172.21.15.169:0/3718561744 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5632ece52cb0 0x5632ece550f0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:24.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.067+0000 7f2ca2c49740 1 --2- 172.21.15.169:0/3718561744 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece1b350 0x5632ece24ee0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:24.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.067+0000 7f2ca2c49740 1 --2- 172.21.15.169:0/3718561744 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5632ece1a960 0x5632ece249a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:24.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.067+0000 7f2ca2c49740 1 -- 172.21.15.169:0/3718561744 >> 172.21.15.169:0/3718561744 conn(0x5632ecde31a0 msgr2=0x5632ece15530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:24.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.067+0000 7f2c7d7fa640 0 client.16425 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:24.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.071+0000 7f2ca2c49740 1 -- 172.21.15.169:0/3718561744 shutdown_connections 2024-03-28T01:01:24.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.071+0000 7f2ca2c49740 1 -- 172.21.15.169:0/3718561744 wait complete. 2024-03-28T01:01:24.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.071+0000 7f2ca2c49740 20 client.16425 trim_cache size 0 max 16384 2024-03-28T01:01:24.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.118+0000 7f2ca2c49740 1 Processor -- start 2024-03-28T01:01:24.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.118+0000 7f2ca2c49740 1 -- start start 2024-03-28T01:01:24.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.118+0000 7f2ca2c49740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5632ece1a960 0x5632ece19c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:24.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.118+0000 7f2ca2c49740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632ece1b350 0x5632ece2bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:24.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.118+0000 7f2ca2c49740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece204c0 0x5632ece14ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:24.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.118+0000 7f2ca2c49740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c84012120 con 0x5632ece204c0 2024-03-28T01:01:24.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.118+0000 7f2ca2c49740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8400fc20 con 0x5632ece1b350 2024-03-28T01:01:24.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.118+0000 7f2ca2c49740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8400dcc0 con 0x5632ece1a960 2024-03-28T01:01:24.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.119+0000 7f2c9e600640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece204c0 0x5632ece14ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:24.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.119+0000 7f2c9e600640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece204c0 0x5632ece14ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36380/0 (socket says 172.21.15.169:36380) 2024-03-28T01:01:24.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.119+0000 7f2c9e600640 1 -- 172.21.15.169:0/364611916 learned_addr learned my addr 172.21.15.169:0/364611916 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:24.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.119+0000 7f2c9ddff640 1 --2- 172.21.15.169:0/364611916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5632ece1a960 0x5632ece19c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:24.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.119+0000 7f2c9e600640 1 -- 172.21.15.169:0/364611916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5632ece1a960 msgr2=0x5632ece19c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:24.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.119+0000 7f2c9e600640 1 --2- 172.21.15.169:0/364611916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5632ece1a960 0x5632ece19c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:24.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.119+0000 7f2c9e600640 1 -- 172.21.15.169:0/364611916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632ece1b350 msgr2=0x5632ece2bee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:01:24.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.119+0000 7f2c9e600640 1 --2- 172.21.15.169:0/364611916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632ece1b350 0x5632ece2bee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:24.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.119+0000 7f2c9e600640 1 -- 172.21.15.169:0/364611916 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5632eca54170 con 0x5632ece204c0 2024-03-28T01:01:24.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.120+0000 7f2c9e600640 1 --2- 172.21.15.169:0/364611916 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece204c0 0x5632ece14ca0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2c9806b390 tx=0x7f2c8c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26b151ebec670b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:24.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.122+0000 7f2c7f7fe640 1 -- 172.21.15.169:0/364611916 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c0044c0 con 0x5632ece204c0 2024-03-28T01:01:24.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.122+0000 7f2c7f7fe640 1 -- 172.21.15.169:0/364611916 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c8c054070 con 0x5632ece204c0 2024-03-28T01:01:24.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.122+0000 7f2c7f7fe640 1 -- 172.21.15.169:0/364611916 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c003bf0 con 0x5632ece204c0 2024-03-28T01:01:24.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.123+0000 7f2ca2c49740 1 -- 172.21.15.169:0/364611916 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece204c0 msgr2=0x5632ece14ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:24.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.123+0000 7f2ca2c49740 1 --2- 172.21.15.169:0/364611916 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece204c0 0x5632ece14ca0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2c9806b390 tx=0x7f2c8c02e070 comp rx=0 tx=0).stop 2024-03-28T01:01:24.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.123+0000 7f2ca2c49740 1 -- 172.21.15.169:0/364611916 shutdown_connections 2024-03-28T01:01:24.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.123+0000 7f2ca2c49740 1 --2- 172.21.15.169:0/364611916 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece204c0 0x5632ece14ca0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:24.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.123+0000 7f2ca2c49740 1 --2- 172.21.15.169:0/364611916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632ece1b350 0x5632ece2bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:24.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.123+0000 7f2ca2c49740 1 --2- 172.21.15.169:0/364611916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5632ece1a960 0x5632ece19c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:24.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.123+0000 7f2ca2c49740 1 -- 172.21.15.169:0/364611916 >> 172.21.15.169:0/364611916 conn(0x5632ecde31a0 msgr2=0x5632ece19510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:24.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.124+0000 7f2ca2c49740 1 -- 172.21.15.169:0/364611916 shutdown_connections 2024-03-28T01:01:24.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.125+0000 7f2ca2c49740 1 -- 172.21.15.169:0/364611916 wait complete. 2024-03-28T01:01:24.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.125+0000 7f2ca2c49740 1 -- start start 2024-03-28T01:01:24.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.125+0000 7f2ca2c49740 1 Processor -- start 2024-03-28T01:01:24.129 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.126+0000 7f2ca2c49740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5632ece1a960 0x5632ece2ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:24.129 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.126+0000 7f2ca2c49740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632ece1b350 0x5632ece2bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:24.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.126+0000 7f2ca2c49740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece204c0 0x5632ece2a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:24.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.126+0000 7f2c9ddff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5632ece1a960 0x5632ece2ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:24.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.127+0000 7f2c9ddff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5632ece1a960 0x5632ece2ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:44664/0 (socket says 172.21.15.169:44664) 2024-03-28T01:01:24.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.127+0000 7f2c9d5fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632ece1b350 0x5632ece2bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:24.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.127+0000 7f2c9ddff640 1 -- 172.21.15.169:0/1523156829 learned_addr learned my addr 172.21.15.169:0/1523156829 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:24.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.127+0000 7f2ca2c49740 1 -- 172.21.15.169:0/1523156829 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8400e2c0 con 0x5632ece204c0 2024-03-28T01:01:24.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.127+0000 7f2ca2c49740 1 -- 172.21.15.169:0/1523156829 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8400f5c0 con 0x5632ece1b350 2024-03-28T01:01:24.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.127+0000 7f2ca2c49740 1 -- 172.21.15.169:0/1523156829 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8400e900 con 0x5632ece1a960 2024-03-28T01:01:24.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.127+0000 7f2ca2c49740 10 client.0 _ll_register_callbacks cb 0x7f2c9000f3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:24.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.128+0000 7f2c9e600640 1 --2- 172.21.15.169:0/1523156829 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece204c0 0x5632ece2a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:24.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.129+0000 7f2c9e600640 1 -- 172.21.15.169:0/1523156829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5632ece1a960 msgr2=0x5632ece2ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:24.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.129+0000 7f2c9e600640 1 --2- 172.21.15.169:0/1523156829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5632ece1a960 0x5632ece2ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:24.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.129+0000 7f2c9e600640 1 -- 172.21.15.169:0/1523156829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632ece1b350 msgr2=0x5632ece2bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:24.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.129+0000 7f2c9e600640 1 --2- 172.21.15.169:0/1523156829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632ece1b350 0x5632ece2bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:24.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.129+0000 7f2c9e600640 1 -- 172.21.15.169:0/1523156829 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5632ece2a960 con 0x5632ece204c0 2024-03-28T01:01:24.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.130+0000 7f2c9e600640 1 --2- 172.21.15.169:0/1523156829 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece204c0 0x5632ece2a130 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c036040 tx=0x7f2c8c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e13b9a449b2f5e78 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:24.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.130+0000 7f2c7d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:01:24.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.130+0000 7f2c7d7fa640 1 -- 172.21.15.169:0/1523156829 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c05f030 con 0x5632ece204c0 2024-03-28T01:01:24.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.130+0000 7f2c7d7fa640 1 -- 172.21.15.169:0/1523156829 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2c8c005210 con 0x5632ece204c0 2024-03-28T01:01:24.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.130+0000 7f2ca2c49740 1 -- 172.21.15.169:0/1523156829 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2c8c054070 con 0x5632ece204c0 2024-03-28T01:01:24.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.130+0000 7f2c7d7fa640 1 -- 172.21.15.169:0/1523156829 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c8c069050 con 0x5632ece204c0 2024-03-28T01:01:24.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.131+0000 7f2ca2c49740 10 client.16431 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:24.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.131+0000 7f2ca2c49740 1 -- 172.21.15.169:0/1523156829 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5632eceaf200 con 0x5632ece204c0 2024-03-28T01:01:24.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.134+0000 7f2c7d7fa640 1 -- 172.21.15.169:0/1523156829 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c8c054070 con 0x5632ece204c0 2024-03-28T01:01:24.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.134+0000 7f2c7d7fa640 1 -- 172.21.15.169:0/1523156829 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2c8c04fa50 con 0x5632ece204c0 2024-03-28T01:01:24.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.135+0000 7f2ca2c49740 10 client.16431 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:24.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.135+0000 7f2ca2c49740 10 client.16431 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:01:24.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.135+0000 7f2ca2c49740 20 client.16431 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:24.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.135+0000 7f2ca2c49740 1 --2- 172.21.15.169:0/1523156829 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5632eceafb10 0x5632eceb1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:24.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.135+0000 7f2ca2c49740 4 client.16431 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T01:01:24.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.135+0000 7f2ca2c49740 1 -- 172.21.15.169:0/1523156829 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5632ecd87900 con 0x5632eceafb10 2024-03-28T01:01:24.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.136+0000 7f2c9ddff640 1 --2- 172.21.15.169:0/1523156829 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5632eceafb10 0x5632eceb1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:24.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.136+0000 7f2c9ddff640 1 --2- 172.21.15.169:0/1523156829 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5632eceafb10 0x5632eceb1f30 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34b6c8e745a542ca server_cookie=f2563908ac1fa0ab in_seq=0 out_seq=0 2024-03-28T01:01:24.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:24.143+0000 7f2c7d7fa640 10 client.16431 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:25.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.670+0000 7f2c7d7fa640 1 -- 172.21.15.169:0/1523156829 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5632ecd87900 con 0x5632eceafb10 2024-03-28T01:01:25.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.670+0000 7f2c7d7fa640 10 client.16431 handle_command_reply: tid=0 2024-03-28T01:01:25.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.670+0000 7f2ca2c49740 1 client.16431 shutdown 2024-03-28T01:01:25.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.670+0000 7f2ca2c49740 1 -- 172.21.15.169:0/1523156829 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece204c0 msgr2=0x5632ece2a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:25.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.670+0000 7f2ca2c49740 1 --2- 172.21.15.169:0/1523156829 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece204c0 0x5632ece2a130 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c036040 tx=0x7f2c8c009260 comp rx=0 tx=0).stop 2024-03-28T01:01:25.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.670+0000 7f2ca2c49740 1 -- 172.21.15.169:0/1523156829 shutdown_connections 2024-03-28T01:01:25.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.670+0000 7f2ca2c49740 1 --2- 172.21.15.169:0/1523156829 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5632eceafb10 0x5632eceb1f30 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:25.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.670+0000 7f2ca2c49740 1 --2- 172.21.15.169:0/1523156829 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5632ece204c0 0x5632ece2a130 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:25.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.670+0000 7f2ca2c49740 1 --2- 172.21.15.169:0/1523156829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632ece1b350 0x5632ece2bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:25.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.670+0000 7f2ca2c49740 1 --2- 172.21.15.169:0/1523156829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5632ece1a960 0x5632ece2ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:25.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.670+0000 7f2ca2c49740 1 -- 172.21.15.169:0/1523156829 >> 172.21.15.169:0/1523156829 conn(0x5632ecde31a0 msgr2=0x5632ece599f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:25.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.670+0000 7f2c7d7fa640 0 client.16431 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:25.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.670+0000 7f2ca2c49740 1 -- 172.21.15.169:0/1523156829 shutdown_connections 2024-03-28T01:01:25.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.670+0000 7f2ca2c49740 1 -- 172.21.15.169:0/1523156829 wait complete. 2024-03-28T01:01:25.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.670+0000 7f2ca2c49740 20 client.16431 trim_cache size 0 max 16384 2024-03-28T01:01:25.675 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:01:25.675 INFO:teuthology.orchestra.run.smithi169.stdout: "return_code": 0, 2024-03-28T01:01:25.675 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_tag": "3f89318b-5ce4-490e-8074-9a5c6e4c511c", 2024-03-28T01:01:25.675 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": "asynchronous" 2024-03-28T01:01:25.675 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:01:25.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.671+0000 7f2ca0063640 1 -- 172.21.15.169:0/3025032036 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2c800a29d0 msgr2=0x7f2c800a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:25.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.671+0000 7f2ca0063640 1 --2- 172.21.15.169:0/3025032036 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2c800a29d0 0x7f2c800a4e90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c005ec0 tx=0x7f2c8c02bc60 comp rx=0 tx=0).stop 2024-03-28T01:01:25.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.671+0000 7f2ca0063640 1 -- 172.21.15.169:0/3025032036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c981086a0 msgr2=0x7f2c98118840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:25.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.671+0000 7f2ca0063640 1 --2- 172.21.15.169:0/3025032036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c981086a0 0x7f2c98118840 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2c8800c890 tx=0x7f2c8800cd50 comp rx=0 tx=0).stop 2024-03-28T01:01:25.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.671+0000 7f2ca0063640 1 -- 172.21.15.169:0/3025032036 shutdown_connections 2024-03-28T01:01:25.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.671+0000 7f2ca0063640 1 --2- 172.21.15.169:0/3025032036 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2c800a29d0 0x7f2c800a4e90 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:25.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.671+0000 7f2ca0063640 1 --2- 172.21.15.169:0/3025032036 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2c9810d9c0 0x7f2c981192c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:25.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.671+0000 7f2ca0063640 1 --2- 172.21.15.169:0/3025032036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c9810b010 0x7f2c98118d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:25.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.671+0000 7f2ca0063640 1 --2- 172.21.15.169:0/3025032036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c981086a0 0x7f2c98118840 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:25.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.671+0000 7f2ca0063640 1 -- 172.21.15.169:0/3025032036 >> 172.21.15.169:0/3025032036 conn(0x7f2c9806e500 msgr2=0x7f2c98108e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:25.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.671+0000 7f2ca0063640 1 -- 172.21.15.169:0/3025032036 shutdown_connections 2024-03-28T01:01:25.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:25.671+0000 7f2ca0063640 1 -- 172.21.15.169:0/3025032036 wait complete. 2024-03-28T01:01:25.693 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:01:25.693 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:01:25.756 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:25 smithi169 ceph-mon[32274]: pgmap v498: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 668 MiB/s wr, 1.04k op/s 2024-03-28T01:01:25.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:25 smithi174 ceph-mon[36060]: pgmap v498: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 668 MiB/s wr, 1.04k op/s 2024-03-28T01:01:25.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:25 smithi177 ceph-mon[36206]: pgmap v498: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 668 MiB/s wr, 1.04k op/s 2024-03-28T01:01:26.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.049+0000 7f369fcc6640 1 -- 172.21.15.169:0/472941099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f369810d9c0 msgr2=0x7f369810de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:26.053 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.049+0000 7f369fcc6640 1 --2- 172.21.15.169:0/472941099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f369810d9c0 0x7f369810de40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3688009900 tx=0x7f368802bc80 comp rx=0 tx=0).stop 2024-03-28T01:01:26.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.055+0000 7f369fcc6640 1 -- 172.21.15.169:0/472941099 shutdown_connections 2024-03-28T01:01:26.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.055+0000 7f369fcc6640 1 --2- 172.21.15.169:0/472941099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f369810d9c0 0x7f369810de40 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.055+0000 7f369fcc6640 1 --2- 172.21.15.169:0/472941099 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f369810b010 0x7f369810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.055+0000 7f369fcc6640 1 --2- 172.21.15.169:0/472941099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36981086a0 0x7f369810aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.055+0000 7f369fcc6640 1 -- 172.21.15.169:0/472941099 >> 172.21.15.169:0/472941099 conn(0x7f369806e500 msgr2=0x7f3698070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:26.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.060+0000 7f369fcc6640 1 -- 172.21.15.169:0/472941099 shutdown_connections 2024-03-28T01:01:26.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.063+0000 7f369fcc6640 1 -- 172.21.15.169:0/472941099 wait complete. 2024-03-28T01:01:26.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.063+0000 7f369fcc6640 1 Processor -- start 2024-03-28T01:01:26.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.067+0000 7f369fcc6640 1 -- start start 2024-03-28T01:01:26.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.067+0000 7f369fcc6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36981086a0 0x7f3698116d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:26.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.067+0000 7f369fcc6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f369810b010 0x7f3698117290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:26.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.067+0000 7f369fcc6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f369810d9c0 0x7f36981b6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:26.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.067+0000 7f369fcc6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369811c150 con 0x7f369810d9c0 2024-03-28T01:01:26.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.067+0000 7f369fcc6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369811bfd0 con 0x7f36981086a0 2024-03-28T01:01:26.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.067+0000 7f369fcc6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369811c2d0 con 0x7f369810b010 2024-03-28T01:01:26.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.068+0000 7f369e263640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f369810d9c0 0x7f36981b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:26.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.068+0000 7f369e263640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f369810d9c0 0x7f36981b6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36414/0 (socket says 172.21.15.169:36414) 2024-03-28T01:01:26.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.068+0000 7f369e263640 1 -- 172.21.15.169:0/3626499033 learned_addr learned my addr 172.21.15.169:0/3626499033 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:26.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.069+0000 7f369d261640 1 --2- 172.21.15.169:0/3626499033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f369810b010 0x7f3698117290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:26.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.069+0000 7f369da62640 1 --2- 172.21.15.169:0/3626499033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36981086a0 0x7f3698116d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:26.078 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.074+0000 7f369da62640 1 -- 172.21.15.169:0/3626499033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f369810b010 msgr2=0x7f3698117290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:26.078 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.075+0000 7f369da62640 1 --2- 172.21.15.169:0/3626499033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f369810b010 0x7f3698117290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.078 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.075+0000 7f369da62640 1 -- 172.21.15.169:0/3626499033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f369810d9c0 msgr2=0x7f36981b6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:26.078 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.075+0000 7f369da62640 1 --2- 172.21.15.169:0/3626499033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f369810d9c0 0x7f36981b6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.078 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.075+0000 7f369da62640 1 -- 172.21.15.169:0/3626499033 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36981b6d40 con 0x7f36981086a0 2024-03-28T01:01:26.078 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.076+0000 7f369da62640 1 --2- 172.21.15.169:0/3626499033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36981086a0 0x7f3698116d50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f369000cce0 tx=0x7f36900075f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:26.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.077+0000 7f368effd640 1 -- 172.21.15.169:0/3626499033 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3690004b20 con 0x7f36981086a0 2024-03-28T01:01:26.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.077+0000 7f369fcc6640 1 -- 172.21.15.169:0/3626499033 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36981b6f10 con 0x7f36981086a0 2024-03-28T01:01:26.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.077+0000 7f369fcc6640 1 -- 172.21.15.169:0/3626499033 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36981b72c0 con 0x7f36981086a0 2024-03-28T01:01:26.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.077+0000 7f368effd640 1 -- 172.21.15.169:0/3626499033 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3690004cc0 con 0x7f36981086a0 2024-03-28T01:01:26.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.077+0000 7f368effd640 1 -- 172.21.15.169:0/3626499033 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369000f6b0 con 0x7f36981086a0 2024-03-28T01:01:26.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.084+0000 7f36a2993740 1 Processor -- start 2024-03-28T01:01:26.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.085+0000 7f368effd640 1 -- 172.21.15.169:0/3626499033 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f369000f950 con 0x7f36981086a0 2024-03-28T01:01:26.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.086+0000 7f36a2993740 1 -- start start 2024-03-28T01:01:26.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.089+0000 7f36a2993740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b6c4ddda0 0x558b6c4de1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:26.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.089+0000 7f36a2993740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4de790 0x558b6c4e3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:26.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.089+0000 7f36a2993740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4e3900 0x558b6c4e5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:26.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.089+0000 7f36a2993740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b6c4433d0 con 0x558b6c4de790 2024-03-28T01:01:26.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.089+0000 7f36a2993740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b6c3ebad0 con 0x558b6c4ddda0 2024-03-28T01:01:26.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.089+0000 7f36a2993740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b6c43f470 con 0x558b6c4e3900 2024-03-28T01:01:26.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.090+0000 7f369da62640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4de790 0x558b6c4e3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:26.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.090+0000 7f369da62640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4de790 0x558b6c4e3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36422/0 (socket says 172.21.15.169:36422) 2024-03-28T01:01:26.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.090+0000 7f369da62640 1 -- 172.21.15.169:0/2148064946 learned_addr learned my addr 172.21.15.169:0/2148064946 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:26.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.090+0000 7f369e263640 1 --2- 172.21.15.169:0/2148064946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b6c4ddda0 0x558b6c4de1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:26.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.090+0000 7f369d261640 1 --2- 172.21.15.169:0/2148064946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4e3900 0x558b6c4e5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:26.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.090+0000 7f368effd640 1 --2- 172.21.15.169:0/3626499033 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36840a2a00 0x7f36840a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:26.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.090+0000 7f369e263640 1 -- 172.21.15.169:0/2148064946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4e3900 msgr2=0x558b6c4e5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:26.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.090+0000 7f369e263640 1 --2- 172.21.15.169:0/2148064946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4e3900 0x558b6c4e5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.090+0000 7f369e263640 1 -- 172.21.15.169:0/2148064946 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4de790 msgr2=0x558b6c4e3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:26.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.090+0000 7f369e263640 1 --2- 172.21.15.169:0/2148064946 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4de790 0x558b6c4e3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.090+0000 7f369e263640 1 -- 172.21.15.169:0/2148064946 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b6c2217d0 con 0x558b6c4ddda0 2024-03-28T01:01:26.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.091+0000 7f368effd640 1 -- 172.21.15.169:0/3626499033 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f36900cec00 con 0x7f36981086a0 2024-03-28T01:01:26.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.091+0000 7f369e263640 1 --2- 172.21.15.169:0/3626499033 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36840a2a00 0x7f36840a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:26.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.091+0000 7f369e263640 1 --2- 172.21.15.169:0/2148064946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b6c4ddda0 0x558b6c4de1c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3688005ab0 tx=0x7f3688004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53bf08d8f6c0f63a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:26.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.092+0000 7f3683fff640 1 -- 172.21.15.169:0/2148064946 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3688045030 con 0x558b6c4ddda0 2024-03-28T01:01:26.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.092+0000 7f3683fff640 1 -- 172.21.15.169:0/2148064946 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3688005450 con 0x558b6c4ddda0 2024-03-28T01:01:26.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.092+0000 7f3683fff640 1 -- 172.21.15.169:0/2148064946 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368803f9e0 con 0x558b6c4ddda0 2024-03-28T01:01:26.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.093+0000 7f36a2993740 1 -- 172.21.15.169:0/2148064946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b6c4ddda0 msgr2=0x558b6c4de1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:26.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.093+0000 7f369e263640 1 --2- 172.21.15.169:0/3626499033 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36840a2a00 0x7f36840a4ec0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3688006010 tx=0x7f368804c000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:26.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.093+0000 7f36a2993740 1 --2- 172.21.15.169:0/2148064946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b6c4ddda0 0x558b6c4de1c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3688005ab0 tx=0x7f3688004770 comp rx=0 tx=0).stop 2024-03-28T01:01:26.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.093+0000 7f36a2993740 1 -- 172.21.15.169:0/2148064946 shutdown_connections 2024-03-28T01:01:26.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.093+0000 7f36a2993740 1 --2- 172.21.15.169:0/2148064946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4e3900 0x558b6c4e5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.093+0000 7f36a2993740 1 --2- 172.21.15.169:0/2148064946 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4de790 0x558b6c4e3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.093+0000 7f36a2993740 1 --2- 172.21.15.169:0/2148064946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b6c4ddda0 0x558b6c4de1c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.093+0000 7f36a2993740 1 -- 172.21.15.169:0/2148064946 >> 172.21.15.169:0/2148064946 conn(0x558b6c4d9310 msgr2=0x558b6c4d9710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:26.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.094+0000 7f36a2993740 1 -- 172.21.15.169:0/2148064946 shutdown_connections 2024-03-28T01:01:26.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.094+0000 7f36a2993740 1 -- 172.21.15.169:0/2148064946 wait complete. 2024-03-28T01:01:26.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.094+0000 7f36a2993740 1 -- start start 2024-03-28T01:01:26.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.094+0000 7f36a2993740 1 Processor -- start 2024-03-28T01:01:26.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.095+0000 7f36a2993740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4ddda0 0x558b6c4e8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:26.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.095+0000 7f36a2993740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4de790 0x558b6c4e87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:26.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.095+0000 7f36a2993740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b6c4e3900 0x558b6c4ef7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:26.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.095+0000 7f36a2993740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b6c4e6950 con 0x558b6c4ddda0 2024-03-28T01:01:26.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.095+0000 7f36a2993740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b6c3ff060 con 0x558b6c4e3900 2024-03-28T01:01:26.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.095+0000 7f36a2993740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b6c4e6ad0 con 0x558b6c4de790 2024-03-28T01:01:26.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.095+0000 7f36a2993740 10 client.0 _ll_register_callbacks cb 0x558b6c3e16a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:26.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.098+0000 7f369e263640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b6c4e3900 0x558b6c4ef7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:26.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.099+0000 7f369da62640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4ddda0 0x558b6c4e8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:26.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.099+0000 7f369da62640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4ddda0 0x558b6c4e8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36436/0 (socket says 172.21.15.169:36436) 2024-03-28T01:01:26.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.099+0000 7f369da62640 1 -- 172.21.15.169:0/2329063527 learned_addr learned my addr 172.21.15.169:0/2329063527 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:26.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.099+0000 7f369d261640 1 --2- 172.21.15.169:0/2329063527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4de790 0x558b6c4e87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:26.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.099+0000 7f369da62640 1 -- 172.21.15.169:0/2329063527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4de790 msgr2=0x558b6c4e87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:26.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.099+0000 7f369da62640 1 --2- 172.21.15.169:0/2329063527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4de790 0x558b6c4e87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.099+0000 7f369da62640 1 -- 172.21.15.169:0/2329063527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b6c4e3900 msgr2=0x558b6c4ef7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:26.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.099+0000 7f369da62640 1 --2- 172.21.15.169:0/2329063527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b6c4e3900 0x558b6c4ef7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.099+0000 7f369da62640 1 -- 172.21.15.169:0/2329063527 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558b6c1573f0 con 0x558b6c4ddda0 2024-03-28T01:01:26.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.099+0000 7f369d261640 1 --2- 172.21.15.169:0/2329063527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4de790 0x558b6c4e87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:01:26.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.100+0000 7f369da62640 1 --2- 172.21.15.169:0/2329063527 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4ddda0 0x558b6c4e8260 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3690000f80 tx=0x7f369000a7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93148bd607815045 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:26.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.100+0000 7f36817fa640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:01:26.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.100+0000 7f36817fa640 1 -- 172.21.15.169:0/2329063527 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369001d050 con 0x558b6c4ddda0 2024-03-28T01:01:26.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.100+0000 7f36817fa640 1 -- 172.21.15.169:0/2329063527 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f369009a6a0 con 0x558b6c4ddda0 2024-03-28T01:01:26.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.100+0000 7f36a2993740 1 -- 172.21.15.169:0/2329063527 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3688005450 con 0x558b6c4ddda0 2024-03-28T01:01:26.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.101+0000 7f36817fa640 1 -- 172.21.15.169:0/2329063527 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f369009abb0 con 0x558b6c4ddda0 2024-03-28T01:01:26.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.102+0000 7f36a2993740 10 client.16443 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:26.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.102+0000 7f36a2993740 1 -- 172.21.15.169:0/2329063527 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558b6c1170f0 con 0x558b6c4ddda0 2024-03-28T01:01:26.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.107+0000 7f36817fa640 1 -- 172.21.15.169:0/2329063527 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3688005450 con 0x558b6c4ddda0 2024-03-28T01:01:26.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.107+0000 7f36817fa640 1 -- 172.21.15.169:0/2329063527 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f369009a110 con 0x558b6c4ddda0 2024-03-28T01:01:26.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.108+0000 7f36a2993740 10 client.16443 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:26.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.108+0000 7f36a2993740 10 client.16443 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:01:26.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.108+0000 7f36a2993740 20 client.16443 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:26.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.108+0000 7f36a2993740 1 --2- 172.21.15.169:0/2329063527 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558b6c4f3c70 0x558b6c4f6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:26.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.108+0000 7f36a2993740 4 client.16443 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:01:26.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.108+0000 7f36a2993740 1 -- 172.21.15.169:0/2329063527 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558b6c450910 con 0x558b6c4f3c70 2024-03-28T01:01:26.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.109+0000 7f369d261640 1 --2- 172.21.15.169:0/2329063527 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558b6c4f3c70 0x558b6c4f6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:26.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.110+0000 7f369d261640 1 --2- 172.21.15.169:0/2329063527 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558b6c4f3c70 0x558b6c4f6110 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eac6616761356b5 server_cookie=998a2e299792fd88 in_seq=0 out_seq=0 2024-03-28T01:01:26.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.110+0000 7f36817fa640 10 client.16443 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:26.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.114+0000 7f36817fa640 1 -- 172.21.15.169:0/2329063527 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558b6c450910 con 0x558b6c4f3c70 2024-03-28T01:01:26.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.114+0000 7f36817fa640 10 client.16443 handle_command_reply: tid=0 2024-03-28T01:01:26.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.114+0000 7f36a2993740 1 client.16443 shutdown 2024-03-28T01:01:26.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.116+0000 7f36a2993740 1 -- 172.21.15.169:0/2329063527 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4ddda0 msgr2=0x558b6c4e8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:26.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.116+0000 7f36a2993740 1 --2- 172.21.15.169:0/2329063527 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4ddda0 0x558b6c4e8260 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3690000f80 tx=0x7f369000a7f0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.121+0000 7f36a2993740 1 -- 172.21.15.169:0/2329063527 shutdown_connections 2024-03-28T01:01:26.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.121+0000 7f36a2993740 1 --2- 172.21.15.169:0/2329063527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b6c4e3900 0x558b6c4ef7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.121+0000 7f36a2993740 1 --2- 172.21.15.169:0/2329063527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4de790 0x558b6c4e87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.121+0000 7f36a2993740 1 --2- 172.21.15.169:0/2329063527 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558b6c4f3c70 0x558b6c4f6110 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.121+0000 7f36a2993740 1 --2- 172.21.15.169:0/2329063527 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4ddda0 0x558b6c4e8260 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.121+0000 7f36a2993740 1 -- 172.21.15.169:0/2329063527 >> 172.21.15.169:0/2329063527 conn(0x558b6c4d9310 msgr2=0x558b6c4d8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:26.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.121+0000 7f36817fa640 0 client.16443 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:26.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.124+0000 7f36a2993740 1 -- 172.21.15.169:0/2329063527 shutdown_connections 2024-03-28T01:01:26.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.124+0000 7f36a2993740 1 -- 172.21.15.169:0/2329063527 wait complete. 2024-03-28T01:01:26.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.128+0000 7f36a2993740 20 client.16443 trim_cache size 0 max 16384 2024-03-28T01:01:26.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.169+0000 7f36a2993740 1 Processor -- start 2024-03-28T01:01:26.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.172+0000 7f36a2993740 1 -- start start 2024-03-28T01:01:26.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.172+0000 7f36a2993740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4ddda0 0x558b6c4e7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:26.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.172+0000 7f36a2993740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4de790 0x558b6c4ef770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:26.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.172+0000 7f36a2993740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b6c4e3900 0x558b6c4e83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:26.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.172+0000 7f36a2993740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367c012010 con 0x558b6c4de790 2024-03-28T01:01:26.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.172+0000 7f36a2993740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367c00fb10 con 0x558b6c4e3900 2024-03-28T01:01:26.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.172+0000 7f36a2993740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367c00dbb0 con 0x558b6c4ddda0 2024-03-28T01:01:26.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.173+0000 7f369da62640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4ddda0 0x558b6c4e7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:26.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.173+0000 7f369da62640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4ddda0 0x558b6c4e7e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:44710/0 (socket says 172.21.15.169:44710) 2024-03-28T01:01:26.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.173+0000 7f369da62640 1 -- 172.21.15.169:0/3087136881 learned_addr learned my addr 172.21.15.169:0/3087136881 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:26.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.174+0000 7f369da62640 1 -- 172.21.15.169:0/3087136881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b6c4e3900 msgr2=0x558b6c4e83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:26.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.174+0000 7f369da62640 1 --2- 172.21.15.169:0/3087136881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b6c4e3900 0x558b6c4e83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.174+0000 7f369da62640 1 -- 172.21.15.169:0/3087136881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4de790 msgr2=0x558b6c4ef770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:26.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.174+0000 7f369da62640 1 --2- 172.21.15.169:0/3087136881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4de790 0x558b6c4ef770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.174+0000 7f369da62640 1 -- 172.21.15.169:0/3087136881 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b6c4d8c60 con 0x558b6c4ddda0 2024-03-28T01:01:26.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.176+0000 7f369da62640 1 --2- 172.21.15.169:0/3087136881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4ddda0 0x558b6c4e7e70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3690099500 tx=0x7f3690004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5b5e37c19fefd6c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:26.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.178+0000 7f36837fe640 1 -- 172.21.15.169:0/3087136881 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369001d070 con 0x558b6c4ddda0 2024-03-28T01:01:26.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.179+0000 7f36837fe640 1 -- 172.21.15.169:0/3087136881 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36900d6040 con 0x558b6c4ddda0 2024-03-28T01:01:26.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.179+0000 7f36837fe640 1 -- 172.21.15.169:0/3087136881 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369009b250 con 0x558b6c4ddda0 2024-03-28T01:01:26.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.181+0000 7f36a2993740 1 -- 172.21.15.169:0/3087136881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4ddda0 msgr2=0x558b6c4e7e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:26.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.181+0000 7f36a2993740 1 --2- 172.21.15.169:0/3087136881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4ddda0 0x558b6c4e7e70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3690099500 tx=0x7f3690004ab0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.189+0000 7f36a2993740 1 -- 172.21.15.169:0/3087136881 shutdown_connections 2024-03-28T01:01:26.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.189+0000 7f36a2993740 1 --2- 172.21.15.169:0/3087136881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b6c4e3900 0x558b6c4e83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.189+0000 7f36a2993740 1 --2- 172.21.15.169:0/3087136881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4de790 0x558b6c4ef770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.189+0000 7f36a2993740 1 --2- 172.21.15.169:0/3087136881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4ddda0 0x558b6c4e7e70 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.189+0000 7f36a2993740 1 -- 172.21.15.169:0/3087136881 >> 172.21.15.169:0/3087136881 conn(0x558b6c4d9310 msgr2=0x558b6c4dfb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:26.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.192+0000 7f36a2993740 1 -- 172.21.15.169:0/3087136881 shutdown_connections 2024-03-28T01:01:26.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.193+0000 7f36a2993740 1 -- 172.21.15.169:0/3087136881 wait complete. 2024-03-28T01:01:26.196 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.193+0000 7f36a2993740 1 -- start start 2024-03-28T01:01:26.196 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.194+0000 7f36a2993740 1 Processor -- start 2024-03-28T01:01:26.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.195+0000 7f36a2993740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4ddda0 0x558b6c4ed000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:26.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.195+0000 7f36a2993740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b6c4de790 0x558b6c4ed540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:26.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.195+0000 7f36a2993740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4e3900 0x558b6c4ee550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:26.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.195+0000 7f36a2993740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367c00e1b0 con 0x558b6c4e3900 2024-03-28T01:01:26.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.195+0000 7f369da62640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4ddda0 0x558b6c4ed000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:26.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.195+0000 7f369da62640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4ddda0 0x558b6c4ed000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:44720/0 (socket says 172.21.15.169:44720) 2024-03-28T01:01:26.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.195+0000 7f369da62640 1 -- 172.21.15.169:0/2875346820 learned_addr learned my addr 172.21.15.169:0/2875346820 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:26.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.195+0000 7f369d261640 1 --2- 172.21.15.169:0/2875346820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b6c4de790 0x558b6c4ed540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:26.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.195+0000 7f36a2993740 1 -- 172.21.15.169:0/2875346820 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367c00f4b0 con 0x558b6c4de790 2024-03-28T01:01:26.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.195+0000 7f369e263640 1 --2- 172.21.15.169:0/2875346820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4e3900 0x558b6c4ee550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:26.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.195+0000 7f36a2993740 1 -- 172.21.15.169:0/2875346820 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367c00e7f0 con 0x558b6c4ddda0 2024-03-28T01:01:26.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.196+0000 7f36a2993740 10 client.0 _ll_register_callbacks cb 0x7f3690005b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:26.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.197+0000 7f369da62640 1 -- 172.21.15.169:0/2875346820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b6c4de790 msgr2=0x558b6c4ed540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:26.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.197+0000 7f369da62640 1 --2- 172.21.15.169:0/2875346820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b6c4de790 0x558b6c4ed540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.197+0000 7f369da62640 1 -- 172.21.15.169:0/2875346820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4e3900 msgr2=0x558b6c4ee550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:26.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.197+0000 7f369da62640 1 --2- 172.21.15.169:0/2875346820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4e3900 0x558b6c4ee550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.197+0000 7f369da62640 1 -- 172.21.15.169:0/2875346820 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558b6c4eeb40 con 0x558b6c4ddda0 2024-03-28T01:01:26.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.197+0000 7f369e263640 1 --2- 172.21.15.169:0/2875346820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4e3900 0x558b6c4ee550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:01:26.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.198+0000 7f369da62640 1 --2- 172.21.15.169:0/2875346820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4ddda0 0x558b6c4ed000 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f369009b1e0 tx=0x7f3690004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec52936c1152a5c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:26.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.198+0000 7f36817fa640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:01:26.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.198+0000 7f36817fa640 1 -- 172.21.15.169:0/2875346820 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369001d070 con 0x558b6c4ddda0 2024-03-28T01:01:26.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.198+0000 7f36817fa640 1 -- 172.21.15.169:0/2875346820 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f3690096fa0 con 0x558b6c4ddda0 2024-03-28T01:01:26.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.198+0000 7f36a2993740 1 -- 172.21.15.169:0/2875346820 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36900d6040 con 0x558b6c4ddda0 2024-03-28T01:01:26.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.198+0000 7f36817fa640 1 -- 172.21.15.169:0/2875346820 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f369009b020 con 0x558b6c4ddda0 2024-03-28T01:01:26.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.199+0000 7f36a2993740 10 client.26257 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:26.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.199+0000 7f36a2993740 1 -- 172.21.15.169:0/2875346820 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558b6c54f690 con 0x558b6c4ddda0 2024-03-28T01:01:26.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.200+0000 7f36817fa640 1 -- 172.21.15.169:0/2875346820 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f369009b250 con 0x558b6c4ddda0 2024-03-28T01:01:26.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.200+0000 7f36817fa640 1 -- 172.21.15.169:0/2875346820 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f36900af170 con 0x558b6c4ddda0 2024-03-28T01:01:26.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.200+0000 7f36a2993740 10 client.26257 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:26.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.200+0000 7f36a2993740 10 client.26257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:01:26.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.200+0000 7f36a2993740 20 client.26257 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:26.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.201+0000 7f36a2993740 1 --2- 172.21.15.169:0/2875346820 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558b6c54fda0 0x558b6c552190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:26.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.201+0000 7f36a2993740 4 client.26257 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:01:26.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.201+0000 7f36a2993740 1 -- 172.21.15.169:0/2875346820 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f36900d6040 con 0x558b6c54fda0 2024-03-28T01:01:26.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.201+0000 7f369d261640 1 --2- 172.21.15.169:0/2875346820 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558b6c54fda0 0x558b6c552190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:26.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.203+0000 7f369d261640 1 --2- 172.21.15.169:0/2875346820 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558b6c54fda0 0x558b6c552190 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b15fd770d0f5dccc server_cookie=535655a8c4cb99ef in_seq=0 out_seq=0 2024-03-28T01:01:26.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.203+0000 7f36817fa640 10 client.26257 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:26.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.207+0000 7f36817fa640 1 -- 172.21.15.169:0/2875346820 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x7f36900d6040 con 0x558b6c54fda0 2024-03-28T01:01:26.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.207+0000 7f36817fa640 10 client.26257 handle_command_reply: tid=0 2024-03-28T01:01:26.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.208+0000 7f36a2993740 1 client.26257 shutdown 2024-03-28T01:01:26.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.208+0000 7f36a2993740 1 -- 172.21.15.169:0/2875346820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4ddda0 msgr2=0x558b6c4ed000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:26.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.208+0000 7f36a2993740 1 --2- 172.21.15.169:0/2875346820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4ddda0 0x558b6c4ed000 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f369009b1e0 tx=0x7f3690004a40 comp rx=0 tx=0).stop 2024-03-28T01:01:26.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.209+0000 7f36a2993740 1 -- 172.21.15.169:0/2875346820 shutdown_connections 2024-03-28T01:01:26.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.209+0000 7f36a2993740 1 --2- 172.21.15.169:0/2875346820 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558b6c54fda0 0x558b6c552190 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.209+0000 7f36a2993740 1 --2- 172.21.15.169:0/2875346820 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b6c4e3900 0x558b6c4ee550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.209+0000 7f36817fa640 0 client.26257 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:26.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.209+0000 7f36a2993740 1 --2- 172.21.15.169:0/2875346820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b6c4de790 0x558b6c4ed540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.209+0000 7f36a2993740 1 --2- 172.21.15.169:0/2875346820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b6c4ddda0 0x558b6c4ed000 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.209+0000 7f36a2993740 1 -- 172.21.15.169:0/2875346820 >> 172.21.15.169:0/2875346820 conn(0x558b6c4d9310 msgr2=0x558b6c4e7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:26.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.209+0000 7f36a2993740 1 -- 172.21.15.169:0/2875346820 shutdown_connections 2024-03-28T01:01:26.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.209+0000 7f36a2993740 1 -- 172.21.15.169:0/2875346820 wait complete. 2024-03-28T01:01:26.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.209+0000 7f36a2993740 20 client.26257 trim_cache size 0 max 16384 2024-03-28T01:01:26.214 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:01:26.214 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "scrub active (0 inodes in the stack)", 2024-03-28T01:01:26.214 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": { 2024-03-28T01:01:26.214 INFO:teuthology.orchestra.run.smithi169.stdout: "3f89318b-5ce4-490e-8074-9a5c6e4c511c": { 2024-03-28T01:01:26.214 INFO:teuthology.orchestra.run.smithi169.stdout: "path": "/", 2024-03-28T01:01:26.214 INFO:teuthology.orchestra.run.smithi169.stdout: "tag": "3f89318b-5ce4-490e-8074-9a5c6e4c511c", 2024-03-28T01:01:26.214 INFO:teuthology.orchestra.run.smithi169.stdout: "options": "recursive,force" 2024-03-28T01:01:26.214 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:01:26.214 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:01:26.214 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:01:26.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.212+0000 7f369fcc6640 1 -- 172.21.15.169:0/3626499033 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36840a2a00 msgr2=0x7f36840a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:26.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.212+0000 7f369fcc6640 1 --2- 172.21.15.169:0/3626499033 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36840a2a00 0x7f36840a4ec0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3688006010 tx=0x7f368804c000 comp rx=0 tx=0).stop 2024-03-28T01:01:26.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.212+0000 7f369fcc6640 1 -- 172.21.15.169:0/3626499033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36981086a0 msgr2=0x7f3698116d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:26.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.212+0000 7f369fcc6640 1 --2- 172.21.15.169:0/3626499033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36981086a0 0x7f3698116d50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f369000cce0 tx=0x7f36900075f0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.212+0000 7f369fcc6640 1 -- 172.21.15.169:0/3626499033 shutdown_connections 2024-03-28T01:01:26.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.212+0000 7f369fcc6640 1 --2- 172.21.15.169:0/3626499033 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36840a2a00 0x7f36840a4ec0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.212+0000 7f369fcc6640 1 --2- 172.21.15.169:0/3626499033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f369810d9c0 0x7f36981b6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.212+0000 7f369fcc6640 1 --2- 172.21.15.169:0/3626499033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f369810b010 0x7f3698117290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.212+0000 7f369fcc6640 1 --2- 172.21.15.169:0/3626499033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36981086a0 0x7f3698116d50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:26.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.212+0000 7f369fcc6640 1 -- 172.21.15.169:0/3626499033 >> 172.21.15.169:0/3626499033 conn(0x7f369806e500 msgr2=0x7f369810b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:26.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.214+0000 7f369fcc6640 1 -- 172.21.15.169:0/3626499033 shutdown_connections 2024-03-28T01:01:26.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:26.215+0000 7f369fcc6640 1 -- 172.21.15.169:0/3626499033 wait complete. 2024-03-28T01:01:26.240 INFO:tasks.cephfs.filesystem:scrub status for tag:3f89318b-5ce4-490e-8074-9a5c6e4c511c - {'path': '/', 'tag': '3f89318b-5ce4-490e-8074-9a5c6e4c511c', 'options': 'recursive,force'} 2024-03-28T01:01:26.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:26 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:01:26.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:26 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:01:26.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:26 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:01:26.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:26 smithi169 ceph-mon[32274]: scrub queued for path: / 2024-03-28T01:01:26.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:26 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:01:26.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:26 smithi169 ceph-mon[32274]: scrub summary: active paths [/] 2024-03-28T01:01:26.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:26 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:01:26.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:26 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:01:26.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:26 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:01:26.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:26 smithi174 ceph-mon[36060]: scrub queued for path: / 2024-03-28T01:01:26.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:26 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:01:26.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:26 smithi174 ceph-mon[36060]: scrub summary: active paths [/] 2024-03-28T01:01:26.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:26 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:01:26.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:26 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:01:26.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:26 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:01:26.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:26 smithi177 ceph-mon[36206]: scrub queued for path: / 2024-03-28T01:01:26.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:26 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:01:26.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:26 smithi177 ceph-mon[36206]: scrub summary: active paths [/] 2024-03-28T01:01:27.710 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:27 smithi174 ceph-mon[36060]: pgmap v499: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 745 MiB/s wr, 1.10k op/s 2024-03-28T01:01:27.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:27 smithi169 ceph-mon[32274]: pgmap v499: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 745 MiB/s wr, 1.10k op/s 2024-03-28T01:01:27.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:27 smithi177 ceph-mon[36206]: pgmap v499: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 745 MiB/s wr, 1.10k op/s 2024-03-28T01:01:28.287 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.282+0000 7f947023d640 1 -- 172.21.15.169:0/3958081905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f946810b010 msgr2=0x7f946810d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:28.287 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.282+0000 7f947023d640 1 --2- 172.21.15.169:0/3958081905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f946810b010 0x7f946810d480 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9464009900 tx=0x7f946402bc40 comp rx=0 tx=0).stop 2024-03-28T01:01:28.287 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.284+0000 7f947023d640 1 -- 172.21.15.169:0/3958081905 shutdown_connections 2024-03-28T01:01:28.287 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.284+0000 7f947023d640 1 --2- 172.21.15.169:0/3958081905 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f946810d9c0 0x7f946810de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.287 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.284+0000 7f947023d640 1 --2- 172.21.15.169:0/3958081905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f946810b010 0x7f946810d480 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.287 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.284+0000 7f947023d640 1 --2- 172.21.15.169:0/3958081905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94681086a0 0x7f946810aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.288 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.284+0000 7f947023d640 1 -- 172.21.15.169:0/3958081905 >> 172.21.15.169:0/3958081905 conn(0x7f946806e6e0 msgr2=0x7f9468070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.287+0000 7f947023d640 1 -- 172.21.15.169:0/3958081905 shutdown_connections 2024-03-28T01:01:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.287+0000 7f947023d640 1 -- 172.21.15.169:0/3958081905 wait complete. 2024-03-28T01:01:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.288+0000 7f947023d640 1 Processor -- start 2024-03-28T01:01:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.291+0000 7f947023d640 1 -- start start 2024-03-28T01:01:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.291+0000 7f947023d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94681086a0 0x7f9468114cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.291+0000 7f947023d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f946810b010 0x7f94681151f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.291+0000 7f947023d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f946810d9c0 0x7f94681b6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.291+0000 7f947023d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946811c190 con 0x7f94681086a0 2024-03-28T01:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.291+0000 7f947023d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946811c010 con 0x7f946810b010 2024-03-28T01:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.291+0000 7f947023d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946811c310 con 0x7f946810d9c0 2024-03-28T01:01:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.292+0000 7f946d7d8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f946810b010 0x7f94681151f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.292+0000 7f946d7d8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f946810b010 0x7f94681151f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:35032/0 (socket says 172.21.15.169:35032) 2024-03-28T01:01:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.292+0000 7f946d7d8640 1 -- 172.21.15.169:0/3892568903 learned_addr learned my addr 172.21.15.169:0/3892568903 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.292+0000 7f946e7da640 1 --2- 172.21.15.169:0/3892568903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f946810d9c0 0x7f94681b6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.292+0000 7f946dfd9640 1 --2- 172.21.15.169:0/3892568903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94681086a0 0x7f9468114cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.293+0000 7f946e7da640 1 -- 172.21.15.169:0/3892568903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f946810b010 msgr2=0x7f94681151f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.293+0000 7f946e7da640 1 --2- 172.21.15.169:0/3892568903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f946810b010 0x7f94681151f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.293+0000 7f946e7da640 1 -- 172.21.15.169:0/3892568903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94681086a0 msgr2=0x7f9468114cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.293+0000 7f946e7da640 1 --2- 172.21.15.169:0/3892568903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94681086a0 0x7f9468114cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.293+0000 7f946e7da640 1 -- 172.21.15.169:0/3892568903 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9468118320 con 0x7f946810d9c0 2024-03-28T01:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.294+0000 7f946e7da640 1 --2- 172.21.15.169:0/3892568903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f946810d9c0 0x7f94681b6860 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f945800d680 tx=0x7f945800db40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.294+0000 7f946d7d8640 1 --2- 172.21.15.169:0/3892568903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f946810b010 0x7f94681151f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.294+0000 7f945effd640 1 -- 172.21.15.169:0/3892568903 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9458017070 con 0x7f946810d9c0 2024-03-28T01:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.294+0000 7f945effd640 1 -- 172.21.15.169:0/3892568903 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9458009820 con 0x7f946810d9c0 2024-03-28T01:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.294+0000 7f945effd640 1 -- 172.21.15.169:0/3892568903 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9458004d00 con 0x7f946810d9c0 2024-03-28T01:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.294+0000 7f947023d640 1 -- 172.21.15.169:0/3892568903 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94681b6e00 con 0x7f946810d9c0 2024-03-28T01:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.294+0000 7f947023d640 1 -- 172.21.15.169:0/3892568903 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94681b7210 con 0x7f946810d9c0 2024-03-28T01:01:28.300 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.297+0000 7f945effd640 1 -- 172.21.15.169:0/3892568903 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f94580099d0 con 0x7f946810d9c0 2024-03-28T01:01:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.298+0000 7f945effd640 1 --2- 172.21.15.169:0/3892568903 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f94540a29f0 0x7f94540a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.298+0000 7f945effd640 1 -- 172.21.15.169:0/3892568903 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f94580ce7c0 con 0x7f946810d9c0 2024-03-28T01:01:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.298+0000 7f946dfd9640 1 --2- 172.21.15.169:0/3892568903 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f94540a29f0 0x7f94540a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.300+0000 7f946dfd9640 1 --2- 172.21.15.169:0/3892568903 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f94540a29f0 0x7f94540a4eb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9460005c40 tx=0x7f946000a590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.301+0000 7f9472e21740 1 Processor -- start 2024-03-28T01:01:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.301+0000 7f9472e21740 1 -- start start 2024-03-28T01:01:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.301+0000 7f9472e21740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf4733a920 0x55bf4733ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.301+0000 7f9472e21740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bf4733b310 0x55bf4733fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.302+0000 7f9472e21740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf47340480 0x55bf47342910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.302+0000 7f9472e21740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf4729fb10 con 0x55bf4733b310 2024-03-28T01:01:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.302+0000 7f9472e21740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf47248900 con 0x55bf47340480 2024-03-28T01:01:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.302+0000 7f9472e21740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf4729bbb0 con 0x55bf4733a920 2024-03-28T01:01:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.302+0000 7f946e7da640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf47340480 0x55bf47342910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.302+0000 7f946e7da640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf47340480 0x55bf47342910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:35048/0 (socket says 172.21.15.169:35048) 2024-03-28T01:01:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.302+0000 7f946e7da640 1 -- 172.21.15.169:0/654974261 learned_addr learned my addr 172.21.15.169:0/654974261 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.302+0000 7f946e7da640 1 -- 172.21.15.169:0/654974261 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf4733a920 msgr2=0x55bf4733ad40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:01:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.302+0000 7f946e7da640 1 --2- 172.21.15.169:0/654974261 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf4733a920 0x55bf4733ad40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.302+0000 7f946e7da640 1 -- 172.21.15.169:0/654974261 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bf4733b310 msgr2=0x55bf4733fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.302+0000 7f946e7da640 1 --2- 172.21.15.169:0/654974261 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bf4733b310 0x55bf4733fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.302+0000 7f946e7da640 1 -- 172.21.15.169:0/654974261 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf46fb4210 con 0x55bf47340480 2024-03-28T01:01:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.303+0000 7f946e7da640 1 --2- 172.21.15.169:0/654974261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf47340480 0x55bf47342910 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f94580971c0 tx=0x7f9458009ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f9c49cc9cf19e56 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:28.307 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.303+0000 7f9453fff640 1 -- 172.21.15.169:0/654974261 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9458020040 con 0x55bf47340480 2024-03-28T01:01:28.307 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.304+0000 7f9453fff640 1 -- 172.21.15.169:0/654974261 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f945809eeb0 con 0x55bf47340480 2024-03-28T01:01:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.305+0000 7f9453fff640 1 -- 172.21.15.169:0/654974261 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945809a4a0 con 0x55bf47340480 2024-03-28T01:01:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.305+0000 7f9472e21740 1 -- 172.21.15.169:0/654974261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf47340480 msgr2=0x55bf47342910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.305+0000 7f9472e21740 1 --2- 172.21.15.169:0/654974261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf47340480 0x55bf47342910 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f94580971c0 tx=0x7f9458009ea0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.305+0000 7f9472e21740 1 -- 172.21.15.169:0/654974261 shutdown_connections 2024-03-28T01:01:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.305+0000 7f9472e21740 1 --2- 172.21.15.169:0/654974261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf47340480 0x55bf47342910 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.305+0000 7f9472e21740 1 --2- 172.21.15.169:0/654974261 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bf4733b310 0x55bf4733fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.305+0000 7f9472e21740 1 --2- 172.21.15.169:0/654974261 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf4733a920 0x55bf4733ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.305+0000 7f9472e21740 1 -- 172.21.15.169:0/654974261 >> 172.21.15.169:0/654974261 conn(0x55bf47335e90 msgr2=0x55bf47336290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.305+0000 7f9472e21740 1 -- 172.21.15.169:0/654974261 shutdown_connections 2024-03-28T01:01:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.305+0000 7f9472e21740 1 -- 172.21.15.169:0/654974261 wait complete. 2024-03-28T01:01:28.309 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.306+0000 7f9472e21740 1 -- start start 2024-03-28T01:01:28.309 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.306+0000 7f9472e21740 1 Processor -- start 2024-03-28T01:01:28.310 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.307+0000 7f9472e21740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bf4733b310 0x55bf47344d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:28.310 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.307+0000 7f9472e21740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf47340480 0x55bf473452d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:28.310 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.307+0000 7f9472e21740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf4734a300 0x55bf4734c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.307+0000 7f9472e21740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf473434d0 con 0x55bf4733b310 2024-03-28T01:01:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.307+0000 7f9472e21740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf4725bd50 con 0x55bf47340480 2024-03-28T01:01:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.307+0000 7f9472e21740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf47343650 con 0x55bf4734a300 2024-03-28T01:01:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.307+0000 7f9472e21740 10 client.0 _ll_register_callbacks cb 0x55bf4725b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.310+0000 7f946e7da640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf4734a300 0x55bf4734c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.310+0000 7f946e7da640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf4734a300 0x55bf4734c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:57302/0 (socket says 172.21.15.169:57302) 2024-03-28T01:01:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.311+0000 7f946e7da640 1 -- 172.21.15.169:0/3628151297 learned_addr learned my addr 172.21.15.169:0/3628151297 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.311+0000 7f946e7da640 1 -- 172.21.15.169:0/3628151297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf47340480 msgr2=0x55bf473452d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.311+0000 7f946dfd9640 1 --2- 172.21.15.169:0/3628151297 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bf4733b310 0x55bf47344d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.311+0000 7f946e7da640 1 --2- 172.21.15.169:0/3628151297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf47340480 0x55bf473452d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.315 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.311+0000 7f946e7da640 1 -- 172.21.15.169:0/3628151297 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bf4733b310 msgr2=0x55bf47344d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:28.315 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.311+0000 7f946e7da640 1 --2- 172.21.15.169:0/3628151297 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bf4733b310 0x55bf47344d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.315 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.311+0000 7f946e7da640 1 -- 172.21.15.169:0/3628151297 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bf47080620 con 0x55bf4734a300 2024-03-28T01:01:28.315 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.312+0000 7f946dfd9640 1 --2- 172.21.15.169:0/3628151297 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bf4733b310 0x55bf47344d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:01:28.316 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.312+0000 7f946e7da640 1 --2- 172.21.15.169:0/3628151297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf4734a300 0x55bf4734c6f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9458012040 tx=0x7f94580a3500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bef285094c31224d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:28.316 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.312+0000 7f94517fa640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:01:28.316 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.312+0000 7f94517fa640 1 -- 172.21.15.169:0/3628151297 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94580b7070 con 0x55bf4734a300 2024-03-28T01:01:28.316 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.312+0000 7f9472e21740 1 -- 172.21.15.169:0/3628151297 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f945809eeb0 con 0x55bf4734a300 2024-03-28T01:01:28.316 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.313+0000 7f94517fa640 1 -- 172.21.15.169:0/3628151297 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f94580b92c0 con 0x55bf4734a300 2024-03-28T01:01:28.316 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.313+0000 7f94517fa640 1 -- 172.21.15.169:0/3628151297 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9458020070 con 0x55bf4734a300 2024-03-28T01:01:28.319 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.316+0000 7f9472e21740 10 client.26266 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:28.319 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.316+0000 7f9472e21740 1 -- 172.21.15.169:0/3628151297 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55bf47343520 con 0x55bf4734a300 2024-03-28T01:01:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.320+0000 7f94517fa640 1 -- 172.21.15.169:0/3628151297 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f945809eeb0 con 0x55bf4734a300 2024-03-28T01:01:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.320+0000 7f94517fa640 1 -- 172.21.15.169:0/3628151297 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9458085040 con 0x55bf4734a300 2024-03-28T01:01:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.320+0000 7f9472e21740 10 client.26266 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.320+0000 7f9472e21740 10 client.26266 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:01:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.320+0000 7f9472e21740 20 client.26266 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.320+0000 7f9472e21740 1 --2- 172.21.15.169:0/3628151297 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bf47350d60 0x55bf47353150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:28.324 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.320+0000 7f9472e21740 4 client.26266 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:01:28.324 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.320+0000 7f9472e21740 1 -- 172.21.15.169:0/3628151297 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bf472ad050 con 0x55bf47350d60 2024-03-28T01:01:28.324 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.321+0000 7f946dfd9640 1 --2- 172.21.15.169:0/3628151297 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bf47350d60 0x55bf47353150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:28.324 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.321+0000 7f94517fa640 10 client.26266 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:28.324 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.322+0000 7f946dfd9640 1 --2- 172.21.15.169:0/3628151297 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bf47350d60 0x55bf47353150 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e681f41c217ea2ac server_cookie=81f2d82754d0c11c in_seq=0 out_seq=0 2024-03-28T01:01:28.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.325+0000 7f94517fa640 1 -- 172.21.15.169:0/3628151297 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bf472ad050 con 0x55bf47350d60 2024-03-28T01:01:28.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.325+0000 7f94517fa640 10 client.26266 handle_command_reply: tid=0 2024-03-28T01:01:28.328 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.325+0000 7f9472e21740 1 client.26266 shutdown 2024-03-28T01:01:28.330 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.328+0000 7f9472e21740 1 -- 172.21.15.169:0/3628151297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf4734a300 msgr2=0x55bf4734c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:28.330 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.328+0000 7f9472e21740 1 --2- 172.21.15.169:0/3628151297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf4734a300 0x55bf4734c6f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9458012040 tx=0x7f94580a3500 comp rx=0 tx=0).stop 2024-03-28T01:01:28.331 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.328+0000 7f9472e21740 1 -- 172.21.15.169:0/3628151297 shutdown_connections 2024-03-28T01:01:28.331 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.328+0000 7f9472e21740 1 --2- 172.21.15.169:0/3628151297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf4734a300 0x55bf4734c6f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.331 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.328+0000 7f9472e21740 1 --2- 172.21.15.169:0/3628151297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf47340480 0x55bf473452d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.331 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.328+0000 7f9472e21740 1 --2- 172.21.15.169:0/3628151297 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bf47350d60 0x55bf47353150 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.331 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.328+0000 7f9472e21740 1 --2- 172.21.15.169:0/3628151297 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bf4733b310 0x55bf47344d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.331 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.328+0000 7f9472e21740 1 -- 172.21.15.169:0/3628151297 >> 172.21.15.169:0/3628151297 conn(0x55bf47335e90 msgr2=0x55bf47335020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:28.332 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.328+0000 7f94517fa640 0 client.26266 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:28.332 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.328+0000 7f9472e21740 1 -- 172.21.15.169:0/3628151297 shutdown_connections 2024-03-28T01:01:28.332 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.328+0000 7f9472e21740 1 -- 172.21.15.169:0/3628151297 wait complete. 2024-03-28T01:01:28.332 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.328+0000 7f9472e21740 20 client.26266 trim_cache size 0 max 16384 2024-03-28T01:01:28.365 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.361+0000 7f9472e21740 1 Processor -- start 2024-03-28T01:01:28.365 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.361+0000 7f9472e21740 1 -- start start 2024-03-28T01:01:28.365 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.362+0000 7f9472e21740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf4733b310 0x55bf473443b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:28.365 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.362+0000 7f9472e21740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bf47340480 0x55bf47343a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:28.365 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.362+0000 7f9472e21740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf4734a300 0x55bf47344d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:28.365 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.362+0000 7f9472e21740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94681c3d70 con 0x55bf47340480 2024-03-28T01:01:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.362+0000 7f946dfd9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf4733b310 0x55bf473443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.362+0000 7f946dfd9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf4733b310 0x55bf473443b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:35060/0 (socket says 172.21.15.169:35060) 2024-03-28T01:01:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.362+0000 7f946dfd9640 1 -- 172.21.15.169:0/3023512794 learned_addr learned my addr 172.21.15.169:0/3023512794 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.362+0000 7f9472e21740 1 -- 172.21.15.169:0/3023512794 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94681c1870 con 0x55bf4733b310 2024-03-28T01:01:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.362+0000 7f946d7d8640 1 --2- 172.21.15.169:0/3023512794 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bf47340480 0x55bf47343a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.362+0000 7f9472e21740 1 -- 172.21.15.169:0/3023512794 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94681bf910 con 0x55bf4734a300 2024-03-28T01:01:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.362+0000 7f946e7da640 1 --2- 172.21.15.169:0/3023512794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf4734a300 0x55bf47344d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.363+0000 7f946dfd9640 1 -- 172.21.15.169:0/3023512794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf4734a300 msgr2=0x55bf47344d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.363+0000 7f946dfd9640 1 --2- 172.21.15.169:0/3023512794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf4734a300 0x55bf47344d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.363+0000 7f946dfd9640 1 -- 172.21.15.169:0/3023512794 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bf47340480 msgr2=0x55bf47343a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:28.367 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.363+0000 7f946dfd9640 1 --2- 172.21.15.169:0/3023512794 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bf47340480 0x55bf47343a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.367 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.363+0000 7f946dfd9640 1 -- 172.21.15.169:0/3023512794 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf4733bba0 con 0x55bf4733b310 2024-03-28T01:01:28.368 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.365+0000 7f946dfd9640 1 --2- 172.21.15.169:0/3023512794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf4733b310 0x55bf473443b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f946000fc20 tx=0x7f9460012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5f8ad3b7a8ae142 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:28.369 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.366+0000 7f94537fe640 1 -- 172.21.15.169:0/3023512794 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9460016070 con 0x55bf4733b310 2024-03-28T01:01:28.369 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.366+0000 7f94537fe640 1 -- 172.21.15.169:0/3023512794 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9460014070 con 0x55bf4733b310 2024-03-28T01:01:28.369 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.366+0000 7f94537fe640 1 -- 172.21.15.169:0/3023512794 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9460026980 con 0x55bf4733b310 2024-03-28T01:01:28.372 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.368+0000 7f9472e21740 1 -- 172.21.15.169:0/3023512794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf4733b310 msgr2=0x55bf473443b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:28.372 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.368+0000 7f9472e21740 1 --2- 172.21.15.169:0/3023512794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf4733b310 0x55bf473443b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f946000fc20 tx=0x7f9460012000 comp rx=0 tx=0).stop 2024-03-28T01:01:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.371+0000 7f9472e21740 1 -- 172.21.15.169:0/3023512794 shutdown_connections 2024-03-28T01:01:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.371+0000 7f9472e21740 1 --2- 172.21.15.169:0/3023512794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf4734a300 0x55bf47344d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.371+0000 7f9472e21740 1 --2- 172.21.15.169:0/3023512794 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bf47340480 0x55bf47343a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.371+0000 7f9472e21740 1 --2- 172.21.15.169:0/3023512794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf4733b310 0x55bf473443b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.371+0000 7f9472e21740 1 -- 172.21.15.169:0/3023512794 >> 172.21.15.169:0/3023512794 conn(0x55bf47335e90 msgr2=0x55bf47336270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.372+0000 7f9472e21740 1 -- 172.21.15.169:0/3023512794 shutdown_connections 2024-03-28T01:01:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.375+0000 7f9472e21740 1 -- 172.21.15.169:0/3023512794 wait complete. 2024-03-28T01:01:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.375+0000 7f9472e21740 1 -- start start 2024-03-28T01:01:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.376+0000 7f9472e21740 1 Processor -- start 2024-03-28T01:01:28.380 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.377+0000 7f9472e21740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bf4733b310 0x55bf473a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:28.380 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.377+0000 7f9472e21740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf47340480 0x55bf473a7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:28.380 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.377+0000 7f9472e21740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf4734a300 0x55bf473aa490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:28.380 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.377+0000 7f9472e21740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94681bff10 con 0x55bf4733b310 2024-03-28T01:01:28.380 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.377+0000 7f9472e21740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94681c1210 con 0x55bf4734a300 2024-03-28T01:01:28.380 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.377+0000 7f9472e21740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94681c0550 con 0x55bf47340480 2024-03-28T01:01:28.381 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.377+0000 7f9472e21740 10 client.0 _ll_register_callbacks cb 0x7f94580973d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:28.381 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.378+0000 7f946d7d8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf47340480 0x55bf473a7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:28.381 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.378+0000 7f946d7d8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf47340480 0x55bf473a7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:57316/0 (socket says 172.21.15.169:57316) 2024-03-28T01:01:28.381 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.378+0000 7f946d7d8640 1 -- 172.21.15.169:0/3397185723 learned_addr learned my addr 172.21.15.169:0/3397185723 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:28.381 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.378+0000 7f946e7da640 1 --2- 172.21.15.169:0/3397185723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf4734a300 0x55bf473aa490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:28.381 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.378+0000 7f946dfd9640 1 --2- 172.21.15.169:0/3397185723 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bf4733b310 0x55bf473a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:28.382 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.379+0000 7f946d7d8640 1 -- 172.21.15.169:0/3397185723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf4734a300 msgr2=0x55bf473aa490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:28.382 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.379+0000 7f946d7d8640 1 --2- 172.21.15.169:0/3397185723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf4734a300 0x55bf473aa490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.382 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.379+0000 7f946d7d8640 1 -- 172.21.15.169:0/3397185723 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bf4733b310 msgr2=0x55bf473a59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:28.382 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.379+0000 7f946d7d8640 1 --2- 172.21.15.169:0/3397185723 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bf4733b310 0x55bf473a59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:28.382 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.379+0000 7f946d7d8640 1 -- 172.21.15.169:0/3397185723 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bf473aa9d0 con 0x55bf47340480 2024-03-28T01:01:28.384 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.381+0000 7f946d7d8640 1 --2- 172.21.15.169:0/3397185723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf47340480 0x55bf473a7f40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9464009850 tx=0x7f9464004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25bc6e5154ec006b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:28.384 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.381+0000 7f94517fa640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:01:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.383+0000 7f94517fa640 1 -- 172.21.15.169:0/3397185723 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9464042070 con 0x55bf47340480 2024-03-28T01:01:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.383+0000 7f94517fa640 1 -- 172.21.15.169:0/3397185723 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f9464005230 con 0x55bf47340480 2024-03-28T01:01:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.383+0000 7f9472e21740 1 -- 172.21.15.169:0/3397185723 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9460014070 con 0x55bf47340480 2024-03-28T01:01:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.386+0000 7f94517fa640 1 -- 172.21.15.169:0/3397185723 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9464005430 con 0x55bf47340480 2024-03-28T01:01:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.386+0000 7f9472e21740 10 client.26272 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.386+0000 7f9472e21740 1 -- 172.21.15.169:0/3397185723 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55bf473abcc0 con 0x55bf47340480 2024-03-28T01:01:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.386+0000 7f94517fa640 1 -- 172.21.15.169:0/3397185723 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9460014070 con 0x55bf47340480 2024-03-28T01:01:28.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.388+0000 7f94517fa640 1 -- 172.21.15.169:0/3397185723 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f94640046c0 con 0x55bf47340480 2024-03-28T01:01:28.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.388+0000 7f9472e21740 10 client.26272 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:28.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.388+0000 7f9472e21740 10 client.26272 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:01:28.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.388+0000 7f9472e21740 20 client.26272 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:28.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.388+0000 7f9472e21740 1 --2- 172.21.15.169:0/3397185723 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bf473ac490 0x55bf473ae910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:28.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.388+0000 7f9472e21740 4 client.26272 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:01:28.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.388+0000 7f9472e21740 1 -- 172.21.15.169:0/3397185723 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55bf472a73a0 con 0x55bf473ac490 2024-03-28T01:01:28.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.389+0000 7f946dfd9640 1 --2- 172.21.15.169:0/3397185723 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bf473ac490 0x55bf473ae910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:28.393 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.391+0000 7f946dfd9640 1 --2- 172.21.15.169:0/3397185723 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bf473ac490 0x55bf473ae910 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2e4487527142938 server_cookie=28a7e4d1aaa0060f in_seq=0 out_seq=0 2024-03-28T01:01:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:28.391+0000 7f94517fa640 10 client.26272 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.254+0000 7f94517fa640 1 -- 172.21.15.169:0/3397185723 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55bf472a73a0 con 0x55bf473ac490 2024-03-28T01:01:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.254+0000 7f94517fa640 10 client.26272 handle_command_reply: tid=0 2024-03-28T01:01:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.254+0000 7f9472e21740 1 client.26272 shutdown 2024-03-28T01:01:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.254+0000 7f9472e21740 1 -- 172.21.15.169:0/3397185723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf47340480 msgr2=0x55bf473a7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.254+0000 7f9472e21740 1 --2- 172.21.15.169:0/3397185723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf47340480 0x55bf473a7f40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9464009850 tx=0x7f9464004270 comp rx=0 tx=0).stop 2024-03-28T01:01:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.257+0000 7f9472e21740 1 -- 172.21.15.169:0/3397185723 shutdown_connections 2024-03-28T01:01:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.257+0000 7f9472e21740 1 --2- 172.21.15.169:0/3397185723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf4734a300 0x55bf473aa490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.257+0000 7f9472e21740 1 --2- 172.21.15.169:0/3397185723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf47340480 0x55bf473a7f40 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.257+0000 7f9472e21740 1 --2- 172.21.15.169:0/3397185723 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bf473ac490 0x55bf473ae910 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.257+0000 7f9472e21740 1 --2- 172.21.15.169:0/3397185723 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bf4733b310 0x55bf473a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.257+0000 7f9472e21740 1 -- 172.21.15.169:0/3397185723 >> 172.21.15.169:0/3397185723 conn(0x55bf47335e90 msgr2=0x55bf4733c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.257+0000 7f94517fa640 0 client.26272 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.258+0000 7f9472e21740 1 -- 172.21.15.169:0/3397185723 shutdown_connections 2024-03-28T01:01:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.258+0000 7f9472e21740 1 -- 172.21.15.169:0/3397185723 wait complete. 2024-03-28T01:01:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.258+0000 7f9472e21740 20 client.26272 trim_cache size 0 max 16384 2024-03-28T01:01:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:01:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:01:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:01:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:01:29.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.261+0000 7f947023d640 1 -- 172.21.15.169:0/3892568903 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f94540a29f0 msgr2=0x7f94540a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:29.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.261+0000 7f947023d640 1 --2- 172.21.15.169:0/3892568903 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f94540a29f0 0x7f94540a4eb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9460005c40 tx=0x7f946000a590 comp rx=0 tx=0).stop 2024-03-28T01:01:29.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.261+0000 7f947023d640 1 -- 172.21.15.169:0/3892568903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f946810d9c0 msgr2=0x7f94681b6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:29.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.261+0000 7f947023d640 1 --2- 172.21.15.169:0/3892568903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f946810d9c0 0x7f94681b6860 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f945800d680 tx=0x7f945800db40 comp rx=0 tx=0).stop 2024-03-28T01:01:29.267 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.264+0000 7f947023d640 1 -- 172.21.15.169:0/3892568903 shutdown_connections 2024-03-28T01:01:29.267 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.264+0000 7f947023d640 1 --2- 172.21.15.169:0/3892568903 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f94540a29f0 0x7f94540a4eb0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:29.267 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.264+0000 7f947023d640 1 --2- 172.21.15.169:0/3892568903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f946810d9c0 0x7f94681b6860 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:29.267 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.264+0000 7f947023d640 1 --2- 172.21.15.169:0/3892568903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f946810b010 0x7f94681151f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:29.267 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.264+0000 7f947023d640 1 --2- 172.21.15.169:0/3892568903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94681086a0 0x7f9468114cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:29.267 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.264+0000 7f947023d640 1 -- 172.21.15.169:0/3892568903 >> 172.21.15.169:0/3892568903 conn(0x7f946806e6e0 msgr2=0x7f946810a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.271+0000 7f947023d640 1 -- 172.21.15.169:0/3892568903 shutdown_connections 2024-03-28T01:01:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:29.271+0000 7f947023d640 1 -- 172.21.15.169:0/3892568903 wait complete. 2024-03-28T01:01:29.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:29 smithi169 ceph-mon[32274]: pgmap v500: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 648 MiB/s wr, 954 op/s 2024-03-28T01:01:29.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:29 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:01:29.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:29 smithi174 ceph-mon[36060]: pgmap v500: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 648 MiB/s wr, 954 op/s 2024-03-28T01:01:29.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:29 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:01:29.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:29 smithi177 ceph-mon[36206]: pgmap v500: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 648 MiB/s wr, 954 op/s 2024-03-28T01:01:29.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:29 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:01:31.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:31 smithi169 ceph-mon[32274]: pgmap v501: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 661 MiB/s wr, 1.06k op/s 2024-03-28T01:01:31.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:31 smithi174 ceph-mon[36060]: pgmap v501: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 661 MiB/s wr, 1.06k op/s 2024-03-28T01:01:31.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:31 smithi177 ceph-mon[36206]: pgmap v501: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 661 MiB/s wr, 1.06k op/s 2024-03-28T01:01:33.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:33 smithi169 ceph-mon[32274]: pgmap v502: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 754 MiB/s wr, 1.48k op/s 2024-03-28T01:01:33.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:33 smithi174 ceph-mon[36060]: pgmap v502: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 754 MiB/s wr, 1.48k op/s 2024-03-28T01:01:33.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:33 smithi177 ceph-mon[36206]: pgmap v502: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 754 MiB/s wr, 1.48k op/s 2024-03-28T01:01:34.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:34 smithi169 ceph-mon[32274]: scrub summary: idle 2024-03-28T01:01:34.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:34 smithi174 ceph-mon[36060]: scrub summary: idle 2024-03-28T01:01:34.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:34 smithi177 ceph-mon[36206]: scrub summary: idle 2024-03-28T01:01:35.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:35 smithi169 ceph-mon[32274]: pgmap v503: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 653 MiB/s wr, 1.38k op/s 2024-03-28T01:01:35.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:35 smithi174 ceph-mon[36060]: pgmap v503: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 653 MiB/s wr, 1.38k op/s 2024-03-28T01:01:35.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:35 smithi177 ceph-mon[36206]: pgmap v503: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 653 MiB/s wr, 1.38k op/s 2024-03-28T01:01:37.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:37 smithi169 ceph-mon[32274]: pgmap v504: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 761 MiB/s wr, 1.43k op/s 2024-03-28T01:01:37.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:37 smithi174 ceph-mon[36060]: pgmap v504: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 761 MiB/s wr, 1.43k op/s 2024-03-28T01:01:37.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:37 smithi177 ceph-mon[36206]: pgmap v504: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 761 MiB/s wr, 1.43k op/s 2024-03-28T01:01:38.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:38 smithi174 ceph-mon[36060]: pgmap v505: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 684 MiB/s wr, 1.26k op/s 2024-03-28T01:01:38.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:38 smithi177 ceph-mon[36206]: pgmap v505: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 684 MiB/s wr, 1.26k op/s 2024-03-28T01:01:39.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:38 smithi169 ceph-mon[32274]: pgmap v505: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 684 MiB/s wr, 1.26k op/s 2024-03-28T01:01:41.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:41 smithi177 ceph-mon[36206]: pgmap v506: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 718 MiB/s wr, 1.36k op/s 2024-03-28T01:01:41.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:41 smithi169 ceph-mon[32274]: pgmap v506: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 718 MiB/s wr, 1.36k op/s 2024-03-28T01:01:41.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:41 smithi174 ceph-mon[36060]: pgmap v506: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 718 MiB/s wr, 1.36k op/s 2024-03-28T01:01:42.664 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.661+0000 7f7eb2e0b640 1 -- 172.21.15.169:0/3645503187 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7eac108a40 msgr2=0x7f7eac108e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:42.664 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.661+0000 7f7eb2e0b640 1 --2- 172.21.15.169:0/3645503187 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7eac108a40 0x7f7eac108e60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f7ea000aa10 tx=0x7f7ea002cfb0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.665+0000 7f7eb2e0b640 1 -- 172.21.15.169:0/3645503187 shutdown_connections 2024-03-28T01:01:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.665+0000 7f7eb2e0b640 1 --2- 172.21.15.169:0/3645503187 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7eac10a0f0 0x7f7eac1109a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.665+0000 7f7eb2e0b640 1 --2- 172.21.15.169:0/3645503187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7eac109750 0x7f7eac109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.665+0000 7f7eb2e0b640 1 --2- 172.21.15.169:0/3645503187 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7eac108a40 0x7f7eac108e60 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.665+0000 7f7eb2e0b640 1 -- 172.21.15.169:0/3645503187 >> 172.21.15.169:0/3645503187 conn(0x7f7eac06e8a0 msgr2=0x7f7eac070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:42.671 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.669+0000 7f7eb2e0b640 1 -- 172.21.15.169:0/3645503187 shutdown_connections 2024-03-28T01:01:42.672 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.670+0000 7f7eb2e0b640 1 -- 172.21.15.169:0/3645503187 wait complete. 2024-03-28T01:01:42.674 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.671+0000 7f7eb2e0b640 1 Processor -- start 2024-03-28T01:01:42.674 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.671+0000 7f7eb2e0b640 1 -- start start 2024-03-28T01:01:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.672+0000 7f7eb2e0b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7eac108a40 0x7f7eac19e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.672+0000 7f7eb2e0b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7eac109750 0x7f7eac19e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.672+0000 7f7eb2e0b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7eac10a0f0 0x7f7eac1a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.672+0000 7f7eb2e0b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac113a30 con 0x7f7eac10a0f0 2024-03-28T01:01:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.672+0000 7f7eb2e0b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac1138b0 con 0x7f7eac108a40 2024-03-28T01:01:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.672+0000 7f7eb2e0b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac113bb0 con 0x7f7eac109750 2024-03-28T01:01:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.674+0000 7f7eb13a8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7eac10a0f0 0x7f7eac1a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.674+0000 7f7eb13a8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7eac10a0f0 0x7f7eac1a5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48460/0 (socket says 172.21.15.169:48460) 2024-03-28T01:01:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.674+0000 7f7eb13a8640 1 -- 172.21.15.169:0/1904642362 learned_addr learned my addr 172.21.15.169:0/1904642362 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:42.679 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.675+0000 7f7eb0ba7640 1 --2- 172.21.15.169:0/1904642362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7eac108a40 0x7f7eac19e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:42.679 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.675+0000 7f7eabfff640 1 --2- 172.21.15.169:0/1904642362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7eac109750 0x7f7eac19e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:42.679 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.676+0000 7f7eb13a8640 1 -- 172.21.15.169:0/1904642362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7eac109750 msgr2=0x7f7eac19e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:42.679 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.676+0000 7f7eb13a8640 1 --2- 172.21.15.169:0/1904642362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7eac109750 0x7f7eac19e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.679 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.676+0000 7f7eb13a8640 1 -- 172.21.15.169:0/1904642362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7eac108a40 msgr2=0x7f7eac19e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:42.679 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.676+0000 7f7eb13a8640 1 --2- 172.21.15.169:0/1904642362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7eac108a40 0x7f7eac19e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.679 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.676+0000 7f7eb13a8640 1 -- 172.21.15.169:0/1904642362 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eac1a5cc0 con 0x7f7eac10a0f0 2024-03-28T01:01:42.682 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.679+0000 7f7eb13a8640 1 --2- 172.21.15.169:0/1904642362 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7eac10a0f0 0x7f7eac1a5650 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7ea400df40 tx=0x7f7ea400df70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:42.683 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.680+0000 7f7ea9ffb640 1 -- 172.21.15.169:0/1904642362 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea400e980 con 0x7f7eac10a0f0 2024-03-28T01:01:42.683 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.680+0000 7f7ea9ffb640 1 -- 172.21.15.169:0/1904642362 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ea4009580 con 0x7f7eac10a0f0 2024-03-28T01:01:42.683 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.680+0000 7f7ea9ffb640 1 -- 172.21.15.169:0/1904642362 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea4016650 con 0x7f7eac10a0f0 2024-03-28T01:01:42.683 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.681+0000 7f7eb2e0b640 1 -- 172.21.15.169:0/1904642362 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eac1a5fb0 con 0x7f7eac10a0f0 2024-03-28T01:01:42.684 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.681+0000 7f7eb2e0b640 1 -- 172.21.15.169:0/1904642362 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eac1a6440 con 0x7f7eac10a0f0 2024-03-28T01:01:42.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.687+0000 7f7eb59f6740 1 Processor -- start 2024-03-28T01:01:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.691+0000 7f7eb59f6740 1 -- start start 2024-03-28T01:01:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.691+0000 7f7ea9ffb640 1 -- 172.21.15.169:0/1904642362 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7ea401d020 con 0x7f7eac10a0f0 2024-03-28T01:01:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.691+0000 7f7eb59f6740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e2920 0x55f13a0e2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.691+0000 7f7eb59f6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f13a0e3310 0x55f13a0e7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.691+0000 7f7eb59f6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e8480 0x55f13a0ea910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.691+0000 7f7eb59f6740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f13a047b10 con 0x55f13a0e2920 2024-03-28T01:01:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.691+0000 7f7eb59f6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f139ff0900 con 0x55f13a0e8480 2024-03-28T01:01:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.691+0000 7f7eb59f6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f13a043bb0 con 0x55f13a0e3310 2024-03-28T01:01:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.692+0000 7f7ea9ffb640 1 --2- 172.21.15.169:0/1904642362 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7e940a2ae0 0x7f7e940a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.692+0000 7f7eabfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e2920 0x55f13a0e2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.692+0000 7f7eabfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e2920 0x55f13a0e2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48476/0 (socket says 172.21.15.169:48476) 2024-03-28T01:01:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.692+0000 7f7ea9ffb640 1 -- 172.21.15.169:0/1904642362 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f7ea40d03c0 con 0x7f7eac10a0f0 2024-03-28T01:01:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.692+0000 7f7eabfff640 1 -- 172.21.15.169:0/2814833912 learned_addr learned my addr 172.21.15.169:0/2814833912 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.692+0000 7f7eabfff640 1 --2- 172.21.15.169:0/2814833912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e8480 0x55f13a0ea910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.693+0000 7f7eb0ba7640 1 --2- 172.21.15.169:0/2814833912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f13a0e3310 0x55f13a0e7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.693+0000 7f7eabfff640 1 -- 172.21.15.169:0/2814833912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f13a0e3310 msgr2=0x55f13a0e7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.693+0000 7f7eabfff640 1 --2- 172.21.15.169:0/2814833912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f13a0e3310 0x55f13a0e7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.693+0000 7f7eabfff640 1 -- 172.21.15.169:0/2814833912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e8480 msgr2=0x55f13a0ea910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.693+0000 7f7eabfff640 1 --2- 172.21.15.169:0/2814833912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e8480 0x55f13a0ea910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.693+0000 7f7eabfff640 1 -- 172.21.15.169:0/2814833912 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f139d5c210 con 0x55f13a0e2920 2024-03-28T01:01:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.694+0000 7f7eabfff640 1 --2- 172.21.15.169:0/2814833912 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e2920 0x55f13a0e2d40 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c00aed0 tx=0x7f7e9c00a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f64ec05b7f4b42f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.694+0000 7f7eb13a8640 1 --2- 172.21.15.169:0/1904642362 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7e940a2ae0 0x7f7e940a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.694+0000 7f7e92ffd640 1 -- 172.21.15.169:0/2814833912 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c00f040 con 0x55f13a0e2920 2024-03-28T01:01:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.694+0000 7f7e92ffd640 1 -- 172.21.15.169:0/2814833912 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e9c0036b0 con 0x55f13a0e2920 2024-03-28T01:01:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.694+0000 7f7e92ffd640 1 -- 172.21.15.169:0/2814833912 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c03a6e0 con 0x55f13a0e2920 2024-03-28T01:01:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.695+0000 7f7eb13a8640 1 --2- 172.21.15.169:0/1904642362 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7e940a2ae0 0x7f7e940a4fa0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7ea4098dc0 tx=0x7f7ea400bf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.697+0000 7f7eb59f6740 1 -- 172.21.15.169:0/2814833912 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e2920 msgr2=0x55f13a0e2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.697+0000 7f7eb59f6740 1 --2- 172.21.15.169:0/2814833912 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e2920 0x55f13a0e2d40 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c00aed0 tx=0x7f7e9c00a0d0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.703 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.700+0000 7f7eb59f6740 1 -- 172.21.15.169:0/2814833912 shutdown_connections 2024-03-28T01:01:42.703 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.700+0000 7f7eb59f6740 1 --2- 172.21.15.169:0/2814833912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e8480 0x55f13a0ea910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.703 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.700+0000 7f7eb59f6740 1 --2- 172.21.15.169:0/2814833912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f13a0e3310 0x55f13a0e7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.703 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.700+0000 7f7eb59f6740 1 --2- 172.21.15.169:0/2814833912 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e2920 0x55f13a0e2d40 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.703 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.700+0000 7f7eb59f6740 1 -- 172.21.15.169:0/2814833912 >> 172.21.15.169:0/2814833912 conn(0x55f13a0dde90 msgr2=0x55f13a0de290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:42.703 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.700+0000 7f7eb59f6740 1 -- 172.21.15.169:0/2814833912 shutdown_connections 2024-03-28T01:01:42.704 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.700+0000 7f7eb59f6740 1 -- 172.21.15.169:0/2814833912 wait complete. 2024-03-28T01:01:42.704 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.701+0000 7f7eb59f6740 1 -- start start 2024-03-28T01:01:42.704 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.701+0000 7f7eb59f6740 1 Processor -- start 2024-03-28T01:01:42.705 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.702+0000 7f7eb59f6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f13a0e2920 0x55f13a0f0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:42.705 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.702+0000 7f7eb59f6740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e3310 0x55f13a0f0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:42.705 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.702+0000 7f7eb59f6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e8480 0x55f13a0ecbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:42.705 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.702+0000 7f7eb59f6740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f13a0eb3d0 con 0x55f13a0e3310 2024-03-28T01:01:42.705 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.702+0000 7f7eb59f6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f13a003d50 con 0x55f13a0e8480 2024-03-28T01:01:42.705 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.702+0000 7f7eb59f6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f13a0eb550 con 0x55f13a0e2920 2024-03-28T01:01:42.705 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.702+0000 7f7eb59f6740 10 client.0 _ll_register_callbacks cb 0x55f13a003720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.703+0000 7f7eb0ba7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e3310 0x55f13a0f0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.703+0000 7f7eabfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f13a0e2920 0x55f13a0f0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.703+0000 7f7eb0ba7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e3310 0x55f13a0f0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48492/0 (socket says 172.21.15.169:48492) 2024-03-28T01:01:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.703+0000 7f7eb0ba7640 1 -- 172.21.15.169:0/2053910851 learned_addr learned my addr 172.21.15.169:0/2053910851 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.703+0000 7f7eabfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e8480 0x55f13a0ecbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.703+0000 7f7eabfff640 1 -- 172.21.15.169:0/2053910851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e8480 msgr2=0x55f13a0ecbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.703+0000 7f7eabfff640 1 --2- 172.21.15.169:0/2053910851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e8480 0x55f13a0ecbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.703+0000 7f7eabfff640 1 -- 172.21.15.169:0/2053910851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e3310 msgr2=0x55f13a0f0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:42.707 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.703+0000 7f7eabfff640 1 --2- 172.21.15.169:0/2053910851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e3310 0x55f13a0f0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.707 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.703+0000 7f7eabfff640 1 -- 172.21.15.169:0/2053910851 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f139e28620 con 0x55f13a0e2920 2024-03-28T01:01:42.707 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.704+0000 7f7eb0ba7640 1 --2- 172.21.15.169:0/2053910851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e3310 0x55f13a0f0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:01:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.704+0000 7f7eabfff640 1 --2- 172.21.15.169:0/2053910851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f13a0e2920 0x55f13a0f0120 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c006a40 tx=0x7f7e9c003e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b24497d2a1936be server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.705+0000 7f7e8bfff640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:01:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.705+0000 7f7e8bfff640 1 -- 172.21.15.169:0/2053910851 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c004380 con 0x55f13a0e2920 2024-03-28T01:01:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.705+0000 7f7e8bfff640 1 -- 172.21.15.169:0/2053910851 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f7e9c004520 con 0x55f13a0e2920 2024-03-28T01:01:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.705+0000 7f7eb59f6740 1 -- 172.21.15.169:0/2053910851 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7e9c0036b0 con 0x55f13a0e2920 2024-03-28T01:01:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.705+0000 7f7e8bfff640 1 -- 172.21.15.169:0/2053910851 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e9c056050 con 0x55f13a0e2920 2024-03-28T01:01:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.705+0000 7f7eb59f6740 10 client.26275 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.705+0000 7f7eb59f6740 1 -- 172.21.15.169:0/2053910851 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f13a0f3560 con 0x55f13a0e2920 2024-03-28T01:01:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.705+0000 7f7e8bfff640 1 -- 172.21.15.169:0/2053910851 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e9c0036b0 con 0x55f13a0e2920 2024-03-28T01:01:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.706+0000 7f7e8bfff640 1 -- 172.21.15.169:0/2053910851 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f7e9c051030 con 0x55f13a0e2920 2024-03-28T01:01:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.706+0000 7f7eb59f6740 10 client.26275 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.706+0000 7f7eb59f6740 10 client.26275 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:01:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.706+0000 7f7eb59f6740 20 client.26275 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.706+0000 7f7eb59f6740 1 --2- 172.21.15.169:0/2053910851 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f13a0f4a20 0x55f13a0f6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.706+0000 7f7eb59f6740 4 client.26275 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:01:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.706+0000 7f7eb59f6740 1 -- 172.21.15.169:0/2053910851 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f13a055050 con 0x55f13a0f4a20 2024-03-28T01:01:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.707+0000 7f7eb0ba7640 1 --2- 172.21.15.169:0/2053910851 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f13a0f4a20 0x55f13a0f6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.708+0000 7f7eb0ba7640 1 --2- 172.21.15.169:0/2053910851 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f13a0f4a20 0x55f13a0f6e10 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad505a30b763a27c server_cookie=254026fb187e5f10 in_seq=0 out_seq=0 2024-03-28T01:01:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.708+0000 7f7e8bfff640 10 client.26275 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:42.713 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.710+0000 7f7e8bfff640 1 -- 172.21.15.169:0/2053910851 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f13a055050 con 0x55f13a0f4a20 2024-03-28T01:01:42.713 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.710+0000 7f7e8bfff640 10 client.26275 handle_command_reply: tid=0 2024-03-28T01:01:42.715 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.712+0000 7f7eb59f6740 1 client.26275 shutdown 2024-03-28T01:01:42.717 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.714+0000 7f7eb59f6740 1 -- 172.21.15.169:0/2053910851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f13a0e2920 msgr2=0x55f13a0f0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:42.717 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.714+0000 7f7eb59f6740 1 --2- 172.21.15.169:0/2053910851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f13a0e2920 0x55f13a0f0120 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c006a40 tx=0x7f7e9c003e20 comp rx=0 tx=0).stop 2024-03-28T01:01:42.719 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.716+0000 7f7eb59f6740 1 -- 172.21.15.169:0/2053910851 shutdown_connections 2024-03-28T01:01:42.719 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.716+0000 7f7eb59f6740 1 --2- 172.21.15.169:0/2053910851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e8480 0x55f13a0ecbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.720 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.716+0000 7f7eb59f6740 1 --2- 172.21.15.169:0/2053910851 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f13a0f4a20 0x55f13a0f6e10 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.720 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.716+0000 7f7eb59f6740 1 --2- 172.21.15.169:0/2053910851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e3310 0x55f13a0f0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.720 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.716+0000 7f7eb59f6740 1 --2- 172.21.15.169:0/2053910851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f13a0e2920 0x55f13a0f0120 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.720 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.716+0000 7f7e8bfff640 0 client.26275 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:42.720 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.716+0000 7f7eb59f6740 1 -- 172.21.15.169:0/2053910851 >> 172.21.15.169:0/2053910851 conn(0x55f13a0dde90 msgr2=0x55f13a0dd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:42.720 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.717+0000 7f7eb59f6740 1 -- 172.21.15.169:0/2053910851 shutdown_connections 2024-03-28T01:01:42.720 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.717+0000 7f7eb59f6740 1 -- 172.21.15.169:0/2053910851 wait complete. 2024-03-28T01:01:42.720 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.717+0000 7f7eb59f6740 20 client.26275 trim_cache size 0 max 16384 2024-03-28T01:01:42.759 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.755+0000 7f7eb59f6740 1 Processor -- start 2024-03-28T01:01:42.759 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.755+0000 7f7eb59f6740 1 -- start start 2024-03-28T01:01:42.759 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.755+0000 7f7eb59f6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e2920 0x55f13a0f0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:42.759 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.756+0000 7f7eb59f6740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e3310 0x55f13a0ea870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:42.759 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.756+0000 7f7eb59f6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f13a0e8480 0x55f13a0de270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:42.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.756+0000 7f7eb0ba7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e3310 0x55f13a0ea870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:42.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.756+0000 7f7eb0ba7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e3310 0x55f13a0ea870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48496/0 (socket says 172.21.15.169:48496) 2024-03-28T01:01:42.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.756+0000 7f7eabfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e2920 0x55f13a0f0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:42.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.756+0000 7f7eabfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e2920 0x55f13a0f0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:45854/0 (socket says 172.21.15.169:45854) 2024-03-28T01:01:42.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.756+0000 7f7eb0ba7640 1 -- 172.21.15.169:0/2138142217 learned_addr learned my addr 172.21.15.169:0/2138142217 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:42.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.756+0000 7f7eb59f6740 1 -- 172.21.15.169:0/2138142217 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac080000 con 0x55f13a0e3310 2024-03-28T01:01:42.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.756+0000 7f7eb59f6740 1 -- 172.21.15.169:0/2138142217 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac07db00 con 0x55f13a0e2920 2024-03-28T01:01:42.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.756+0000 7f7eb59f6740 1 -- 172.21.15.169:0/2138142217 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac07bba0 con 0x55f13a0e8480 2024-03-28T01:01:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.757+0000 7f7eb0ba7640 1 -- 172.21.15.169:0/2138142217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f13a0e8480 msgr2=0x55f13a0de270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.757+0000 7f7eb0ba7640 1 --2- 172.21.15.169:0/2138142217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f13a0e8480 0x55f13a0de270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.757+0000 7f7eb0ba7640 1 -- 172.21.15.169:0/2138142217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e2920 msgr2=0x55f13a0f0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.757+0000 7f7eb0ba7640 1 --2- 172.21.15.169:0/2138142217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e2920 0x55f13a0f0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.757+0000 7f7eb0ba7640 1 -- 172.21.15.169:0/2138142217 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f13a0edc30 con 0x55f13a0e3310 2024-03-28T01:01:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.757+0000 7f7eb0ba7640 1 --2- 172.21.15.169:0/2138142217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e3310 0x55f13a0ea870 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0004d80 tx=0x7f7ea000a370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14c509bed4f1e52b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.758+0000 7f7e927fc640 1 -- 172.21.15.169:0/2138142217 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0004240 con 0x55f13a0e3310 2024-03-28T01:01:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.758+0000 7f7e927fc640 1 -- 172.21.15.169:0/2138142217 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ea00043e0 con 0x55f13a0e3310 2024-03-28T01:01:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.758+0000 7f7e927fc640 1 -- 172.21.15.169:0/2138142217 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea00051b0 con 0x55f13a0e3310 2024-03-28T01:01:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.759+0000 7f7eb59f6740 1 -- 172.21.15.169:0/2138142217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e3310 msgr2=0x55f13a0ea870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.759+0000 7f7eb59f6740 1 --2- 172.21.15.169:0/2138142217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e3310 0x55f13a0ea870 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0004d80 tx=0x7f7ea000a370 comp rx=0 tx=0).stop 2024-03-28T01:01:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.759+0000 7f7eb59f6740 1 -- 172.21.15.169:0/2138142217 shutdown_connections 2024-03-28T01:01:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.759+0000 7f7eb59f6740 1 --2- 172.21.15.169:0/2138142217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f13a0e8480 0x55f13a0de270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.759+0000 7f7eb59f6740 1 --2- 172.21.15.169:0/2138142217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e3310 0x55f13a0ea870 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.759+0000 7f7eb59f6740 1 --2- 172.21.15.169:0/2138142217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e2920 0x55f13a0f0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.759+0000 7f7eb59f6740 1 -- 172.21.15.169:0/2138142217 >> 172.21.15.169:0/2138142217 conn(0x55f13a0dde90 msgr2=0x55f13a0d8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.760+0000 7f7eb59f6740 1 -- 172.21.15.169:0/2138142217 shutdown_connections 2024-03-28T01:01:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.760+0000 7f7eb59f6740 1 -- 172.21.15.169:0/2138142217 wait complete. 2024-03-28T01:01:42.764 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.760+0000 7f7eb59f6740 1 -- start start 2024-03-28T01:01:42.764 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.760+0000 7f7eb59f6740 1 Processor -- start 2024-03-28T01:01:42.764 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.761+0000 7f7eb59f6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f13a0e2920 0x55f13a0fa060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:42.765 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.762+0000 7f7eb59f6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e3310 0x55f13a0fa5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:42.765 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.762+0000 7f7eb59f6740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e8480 0x55f13a0fb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:42.765 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.762+0000 7f7eb0ba7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e3310 0x55f13a0fa5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:42.765 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.762+0000 7f7eb0ba7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e3310 0x55f13a0fa5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:45858/0 (socket says 172.21.15.169:45858) 2024-03-28T01:01:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.762+0000 7f7eabfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f13a0e2920 0x55f13a0fa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.762+0000 7f7eabfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f13a0e2920 0x55f13a0fa060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:58844/0 (socket says 172.21.15.169:58844) 2024-03-28T01:01:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.762+0000 7f7eb0ba7640 1 -- 172.21.15.169:0/947057086 learned_addr learned my addr 172.21.15.169:0/947057086 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.762+0000 7f7eb59f6740 1 -- 172.21.15.169:0/947057086 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac07c1a0 con 0x55f13a0e8480 2024-03-28T01:01:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.763+0000 7f7eb59f6740 1 -- 172.21.15.169:0/947057086 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac07d4a0 con 0x55f13a0e3310 2024-03-28T01:01:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.763+0000 7f7eb59f6740 1 -- 172.21.15.169:0/947057086 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac07c7e0 con 0x55f13a0e2920 2024-03-28T01:01:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.763+0000 7f7eabfff640 1 --2- 172.21.15.169:0/947057086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e8480 0x55f13a0fb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.763+0000 7f7eabfff640 1 -- 172.21.15.169:0/947057086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f13a0e2920 msgr2=0x55f13a0fa060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.763+0000 7f7eabfff640 1 --2- 172.21.15.169:0/947057086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f13a0e2920 0x55f13a0fa060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.763+0000 7f7eabfff640 1 -- 172.21.15.169:0/947057086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e3310 msgr2=0x55f13a0fa5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.763+0000 7f7eabfff640 1 --2- 172.21.15.169:0/947057086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e3310 0x55f13a0fa5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.763+0000 7f7eabfff640 1 -- 172.21.15.169:0/947057086 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f13a0fbba0 con 0x55f13a0e8480 2024-03-28T01:01:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.764+0000 7f7eabfff640 1 --2- 172.21.15.169:0/947057086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e8480 0x55f13a0fb5b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c049840 tx=0x7f7e9c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=394429ebef28a042 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.764+0000 7f7e8bfff640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:01:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.764+0000 7f7eb59f6740 10 client.0 _ll_register_callbacks cb 0x7f7e9c004020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.764+0000 7f7eb59f6740 1 -- 172.21.15.169:0/947057086 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ea00043e0 con 0x55f13a0e8480 2024-03-28T01:01:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.764+0000 7f7e8bfff640 1 -- 172.21.15.169:0/947057086 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c0036b0 con 0x55f13a0e8480 2024-03-28T01:01:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.764+0000 7f7e8bfff640 1 -- 172.21.15.169:0/947057086 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f7e9c0042e0 con 0x55f13a0e8480 2024-03-28T01:01:42.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.764+0000 7f7e8bfff640 1 -- 172.21.15.169:0/947057086 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e9c068050 con 0x55f13a0e8480 2024-03-28T01:01:42.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.766+0000 7f7eb59f6740 10 client.16476 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:42.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.766+0000 7f7eb59f6740 1 -- 172.21.15.169:0/947057086 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x55f13a0fcd00 con 0x55f13a0e8480 2024-03-28T01:01:42.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.767+0000 7f7e8bfff640 1 -- 172.21.15.169:0/947057086 <== mon.0 v2:172.21.15.169:3300/0 4 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f7e9c051040 con 0x55f13a0e8480 2024-03-28T01:01:42.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.768+0000 7f7eb59f6740 10 client.16476 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:42.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.768+0000 7f7eb59f6740 10 client.16476 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:01:42.771 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.768+0000 7f7eb59f6740 20 client.16476 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:42.771 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.768+0000 7f7eb59f6740 1 --2- 172.21.15.169:0/947057086 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f13a165140 0x55f13a167530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:42.771 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.768+0000 7f7eb59f6740 4 client.16476 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:01:42.771 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.768+0000 7f7eb59f6740 1 -- 172.21.15.169:0/947057086 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f13a04f3a0 con 0x55f13a165140 2024-03-28T01:01:42.774 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.772+0000 7f7eb0ba7640 1 --2- 172.21.15.169:0/947057086 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f13a165140 0x55f13a167530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.776+0000 7f7eb0ba7640 1 --2- 172.21.15.169:0/947057086 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f13a165140 0x55f13a167530 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f545e772aba88df8 server_cookie=40867ca278a8359c in_seq=0 out_seq=0 2024-03-28T01:01:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:42.776+0000 7f7e8bfff640 10 client.16476 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:43.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:43 smithi177 ceph-mon[36206]: pgmap v507: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 878 MiB/s wr, 1.65k op/s 2024-03-28T01:01:43.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:43 smithi169 ceph-mon[32274]: pgmap v507: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 878 MiB/s wr, 1.65k op/s 2024-03-28T01:01:43.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:43 smithi174 ceph-mon[36060]: pgmap v507: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 878 MiB/s wr, 1.65k op/s 2024-03-28T01:01:45.659 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:45 smithi174 ceph-mon[36060]: pgmap v508: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 725 MiB/s wr, 1.21k op/s 2024-03-28T01:01:45.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:45 smithi177 ceph-mon[36206]: pgmap v508: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 725 MiB/s wr, 1.21k op/s 2024-03-28T01:01:45.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:45 smithi169 ceph-mon[32274]: pgmap v508: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 725 MiB/s wr, 1.21k op/s 2024-03-28T01:01:47.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:47 smithi177 ceph-mon[36206]: pgmap v509: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 836 MiB/s wr, 1.45k op/s 2024-03-28T01:01:47.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:47 smithi169 ceph-mon[32274]: pgmap v509: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 836 MiB/s wr, 1.45k op/s 2024-03-28T01:01:47.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:47 smithi174 ceph-mon[36060]: pgmap v509: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 836 MiB/s wr, 1.45k op/s 2024-03-28T01:01:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.251+0000 7f7e8bfff640 1 -- 172.21.15.169:0/947057086 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f13a04f3a0 con 0x55f13a165140 2024-03-28T01:01:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.251+0000 7f7e8bfff640 10 client.16476 handle_command_reply: tid=0 2024-03-28T01:01:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.251+0000 7f7eb59f6740 1 client.16476 shutdown 2024-03-28T01:01:48.255 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.251+0000 7f7eb59f6740 1 -- 172.21.15.169:0/947057086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e8480 msgr2=0x55f13a0fb5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:48.255 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.251+0000 7f7eb59f6740 1 --2- 172.21.15.169:0/947057086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e8480 0x55f13a0fb5b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c049840 tx=0x7f7e9c000bc0 comp rx=0 tx=0).stop 2024-03-28T01:01:48.255 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.251+0000 7f7eb59f6740 1 -- 172.21.15.169:0/947057086 shutdown_connections 2024-03-28T01:01:48.255 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.251+0000 7f7eb59f6740 1 --2- 172.21.15.169:0/947057086 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f13a165140 0x55f13a167530 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:48.255 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.251+0000 7f7eb59f6740 1 --2- 172.21.15.169:0/947057086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f13a0e8480 0x55f13a0fb5b0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:48.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.251+0000 7f7eb59f6740 1 --2- 172.21.15.169:0/947057086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f13a0e3310 0x55f13a0fa5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:48.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.251+0000 7f7eb59f6740 1 --2- 172.21.15.169:0/947057086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f13a0e2920 0x55f13a0fa060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:48.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.251+0000 7f7eb59f6740 1 -- 172.21.15.169:0/947057086 >> 172.21.15.169:0/947057086 conn(0x55f13a0dde90 msgr2=0x55f13a0de270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:48.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.251+0000 7f7e8bfff640 0 client.16476 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:48.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.252+0000 7f7eb59f6740 1 -- 172.21.15.169:0/947057086 shutdown_connections 2024-03-28T01:01:48.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.252+0000 7f7eb59f6740 1 -- 172.21.15.169:0/947057086 wait complete. 2024-03-28T01:01:48.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.252+0000 7f7eb59f6740 20 client.16476 trim_cache size 0 max 16384 2024-03-28T01:01:48.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:01:48.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:01:48.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:01:48.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:01:48.258 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.254+0000 7f7eb2e0b640 1 -- 172.21.15.169:0/1904642362 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7e940a2ae0 msgr2=0x7f7e940a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:48.258 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.254+0000 7f7eb2e0b640 1 --2- 172.21.15.169:0/1904642362 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7e940a2ae0 0x7f7e940a4fa0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7ea4098dc0 tx=0x7f7ea400bf90 comp rx=0 tx=0).stop 2024-03-28T01:01:48.258 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.254+0000 7f7eb2e0b640 1 -- 172.21.15.169:0/1904642362 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7eac10a0f0 msgr2=0x7f7eac1a5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:48.258 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.254+0000 7f7eb2e0b640 1 --2- 172.21.15.169:0/1904642362 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7eac10a0f0 0x7f7eac1a5650 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7ea400df40 tx=0x7f7ea400df70 comp rx=0 tx=0).stop 2024-03-28T01:01:48.258 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.254+0000 7f7eb2e0b640 1 -- 172.21.15.169:0/1904642362 shutdown_connections 2024-03-28T01:01:48.258 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.254+0000 7f7eb2e0b640 1 --2- 172.21.15.169:0/1904642362 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7e940a2ae0 0x7f7e940a4fa0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:48.258 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.254+0000 7f7eb2e0b640 1 --2- 172.21.15.169:0/1904642362 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7eac10a0f0 0x7f7eac1a5650 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:48.259 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.255+0000 7f7eb2e0b640 1 --2- 172.21.15.169:0/1904642362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7eac109750 0x7f7eac19e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:48.259 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.255+0000 7f7eb2e0b640 1 --2- 172.21.15.169:0/1904642362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7eac108a40 0x7f7eac19e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:48.259 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.255+0000 7f7eb2e0b640 1 -- 172.21.15.169:0/1904642362 >> 172.21.15.169:0/1904642362 conn(0x7f7eac06e8a0 msgr2=0x7f7eac10ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:48.259 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.255+0000 7f7eb2e0b640 1 -- 172.21.15.169:0/1904642362 shutdown_connections 2024-03-28T01:01:48.259 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:01:48.255+0000 7f7eb2e0b640 1 -- 172.21.15.169:0/1904642362 wait complete. 2024-03-28T01:01:49.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:49 smithi177 ceph-mon[36206]: pgmap v510: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 748 MiB/s wr, 1.39k op/s 2024-03-28T01:01:49.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:49 smithi169 ceph-mon[32274]: pgmap v510: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 748 MiB/s wr, 1.39k op/s 2024-03-28T01:01:49.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:49 smithi174 ceph-mon[36060]: pgmap v510: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 748 MiB/s wr, 1.39k op/s 2024-03-28T01:01:51.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:51 smithi169 ceph-mon[32274]: pgmap v511: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 746 MiB/s wr, 1.51k op/s 2024-03-28T01:01:51.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:51 smithi174 ceph-mon[36060]: pgmap v511: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 746 MiB/s wr, 1.51k op/s 2024-03-28T01:01:51.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:51 smithi177 ceph-mon[36206]: pgmap v511: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 746 MiB/s wr, 1.51k op/s 2024-03-28T01:01:53.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:53 smithi169 ceph-mon[32274]: pgmap v512: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 862 MiB/s wr, 2.00k op/s 2024-03-28T01:01:53.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:53 smithi174 ceph-mon[36060]: pgmap v512: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 862 MiB/s wr, 2.00k op/s 2024-03-28T01:01:53.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:53 smithi177 ceph-mon[36206]: pgmap v512: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 862 MiB/s wr, 2.00k op/s 2024-03-28T01:01:55.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:55 smithi169 ceph-mon[32274]: pgmap v513: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 683 MiB/s wr, 1.55k op/s 2024-03-28T01:01:55.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:55 smithi174 ceph-mon[36060]: pgmap v513: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 683 MiB/s wr, 1.55k op/s 2024-03-28T01:01:55.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:55 smithi177 ceph-mon[36206]: pgmap v513: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 683 MiB/s wr, 1.55k op/s 2024-03-28T01:01:56.241 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:01:56.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:01:56.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.774+0000 7f55e498f640 1 -- 172.21.15.169:0/3068865370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55e0109640 msgr2=0x7f55e010df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:56.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.774+0000 7f55e498f640 1 --2- 172.21.15.169:0/3068865370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55e0109640 0x7f55e010df00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f55d8009be0 tx=0x7f55d802bf10 comp rx=0 tx=0).stop 2024-03-28T01:01:56.782 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.777+0000 7f55e498f640 1 -- 172.21.15.169:0/3068865370 shutdown_connections 2024-03-28T01:01:56.783 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.777+0000 7f55e498f640 1 --2- 172.21.15.169:0/3068865370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55e0109640 0x7f55e010df00 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.783 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.777+0000 7f55e498f640 1 --2- 172.21.15.169:0/3068865370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55e0108ca0 0x7f55e0109100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.783 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.777+0000 7f55e498f640 1 --2- 172.21.15.169:0/3068865370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55e00729f0 0x7f55e0072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.783 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.777+0000 7f55e498f640 1 -- 172.21.15.169:0/3068865370 >> 172.21.15.169:0/3068865370 conn(0x7f55e006e4c0 msgr2=0x7f55e0070900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:56.783 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.781+0000 7f55e498f640 1 -- 172.21.15.169:0/3068865370 shutdown_connections 2024-03-28T01:01:56.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.781+0000 7f55e498f640 1 -- 172.21.15.169:0/3068865370 wait complete. 2024-03-28T01:01:56.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.782+0000 7f55e498f640 1 Processor -- start 2024-03-28T01:01:56.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.782+0000 7f55e498f640 1 -- start start 2024-03-28T01:01:56.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.782+0000 7f55e498f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55e00729f0 0x7f55e019dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:56.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.782+0000 7f55e498f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55e0108ca0 0x7f55e019e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:56.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.782+0000 7f55e498f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55e0109640 0x7f55e01a52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:56.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.782+0000 7f55e498f640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e0111850 con 0x7f55e0108ca0 2024-03-28T01:01:56.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.782+0000 7f55e498f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e01116d0 con 0x7f55e0109640 2024-03-28T01:01:56.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.782+0000 7f55e498f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e01119d0 con 0x7f55e00729f0 2024-03-28T01:01:56.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.783+0000 7f55ded9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55e00729f0 0x7f55e019dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:56.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.783+0000 7f55ded9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55e00729f0 0x7f55e019dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:55826/0 (socket says 172.21.15.169:55826) 2024-03-28T01:01:56.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.783+0000 7f55ded9d640 1 -- 172.21.15.169:0/3398059375 learned_addr learned my addr 172.21.15.169:0/3398059375 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:56.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.784+0000 7f55ded9d640 1 -- 172.21.15.169:0/3398059375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55e0109640 msgr2=0x7f55e01a52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:56.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.784+0000 7f55ded9d640 1 --2- 172.21.15.169:0/3398059375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55e0109640 0x7f55e01a52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.788 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.784+0000 7f55ded9d640 1 -- 172.21.15.169:0/3398059375 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55e0108ca0 msgr2=0x7f55e019e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:56.788 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.784+0000 7f55ded9d640 1 --2- 172.21.15.169:0/3398059375 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55e0108ca0 0x7f55e019e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.788 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.784+0000 7f55ded9d640 1 -- 172.21.15.169:0/3398059375 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e01a5950 con 0x7f55e00729f0 2024-03-28T01:01:56.789 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.787+0000 7f55ded9d640 1 --2- 172.21.15.169:0/3398059375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55e00729f0 0x7f55e019dd40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f55d001fa60 tx=0x7f55d001ff20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:56.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.794+0000 7f55cffff640 1 -- 172.21.15.169:0/3398059375 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d001dc50 con 0x7f55e00729f0 2024-03-28T01:01:56.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.794+0000 7f55e498f640 1 -- 172.21.15.169:0/3398059375 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55e01a5c40 con 0x7f55e00729f0 2024-03-28T01:01:56.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.794+0000 7f55e498f640 1 -- 172.21.15.169:0/3398059375 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55e01a6190 con 0x7f55e00729f0 2024-03-28T01:01:56.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.794+0000 7f55cffff640 1 -- 172.21.15.169:0/3398059375 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f55d001ddf0 con 0x7f55e00729f0 2024-03-28T01:01:56.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.795+0000 7f55cffff640 1 -- 172.21.15.169:0/3398059375 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d00216a0 con 0x7f55e00729f0 2024-03-28T01:01:56.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.799+0000 7f55e7d7f740 1 Processor -- start 2024-03-28T01:01:56.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.799+0000 7f55e7d7f740 1 -- start start 2024-03-28T01:01:56.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.799+0000 7f55e7d7f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e84501fa0 0x558e845023c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:56.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.799+0000 7f55e7d7f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84502990 0x558e84507490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:56.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.799+0000 7f55e7d7f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84507b00 0x558e84509f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:56.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.799+0000 7f55e7d7f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e844673d0 con 0x558e84502990 2024-03-28T01:01:56.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.799+0000 7f55e7d7f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e8440fad0 con 0x558e84507b00 2024-03-28T01:01:56.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.799+0000 7f55e7d7f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e84463470 con 0x558e84501fa0 2024-03-28T01:01:56.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.804+0000 7f55de59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84507b00 0x558e84509f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:56.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.804+0000 7f55df59e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e84501fa0 0x558e845023c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:56.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.804+0000 7f55de59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84507b00 0x558e84509f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49230/0 (socket says 172.21.15.169:49230) 2024-03-28T01:01:56.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.804+0000 7f55de59c640 1 -- 172.21.15.169:0/3543073344 learned_addr learned my addr 172.21.15.169:0/3543073344 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:56.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.807+0000 7f55ded9d640 1 --2- 172.21.15.169:0/3543073344 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84502990 0x558e84507490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:56.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.807+0000 7f55df59e640 1 -- 172.21.15.169:0/3543073344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84507b00 msgr2=0x558e84509f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:56.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.807+0000 7f55df59e640 1 --2- 172.21.15.169:0/3543073344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84507b00 0x558e84509f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.807+0000 7f55df59e640 1 -- 172.21.15.169:0/3543073344 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84502990 msgr2=0x558e84507490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:56.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.807+0000 7f55df59e640 1 --2- 172.21.15.169:0/3543073344 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84502990 0x558e84507490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.807+0000 7f55df59e640 1 -- 172.21.15.169:0/3543073344 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e842457d0 con 0x558e84501fa0 2024-03-28T01:01:56.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.809+0000 7f55df59e640 1 --2- 172.21.15.169:0/3543073344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e84501fa0 0x558e845023c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f55d8036040 tx=0x7f55d8009940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58ecab9ad06e15 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:56.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.812+0000 7f55cd7fa640 1 -- 172.21.15.169:0/3543073344 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d8042020 con 0x558e84501fa0 2024-03-28T01:01:56.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.812+0000 7f55cd7fa640 1 -- 172.21.15.169:0/3543073344 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f55d8003e50 con 0x558e84501fa0 2024-03-28T01:01:56.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.813+0000 7f55e7d7f740 1 -- 172.21.15.169:0/3543073344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e84501fa0 msgr2=0x558e845023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:56.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.813+0000 7f55e7d7f740 1 --2- 172.21.15.169:0/3543073344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e84501fa0 0x558e845023c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f55d8036040 tx=0x7f55d8009940 comp rx=0 tx=0).stop 2024-03-28T01:01:56.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.814+0000 7f55e7d7f740 1 -- 172.21.15.169:0/3543073344 shutdown_connections 2024-03-28T01:01:56.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.814+0000 7f55e7d7f740 1 --2- 172.21.15.169:0/3543073344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84507b00 0x558e84509f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.814+0000 7f55e7d7f740 1 --2- 172.21.15.169:0/3543073344 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84502990 0x558e84507490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.814+0000 7f55e7d7f740 1 --2- 172.21.15.169:0/3543073344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e84501fa0 0x558e845023c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.814+0000 7f55e7d7f740 1 -- 172.21.15.169:0/3543073344 >> 172.21.15.169:0/3543073344 conn(0x558e844fd510 msgr2=0x558e844fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:56.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.817+0000 7f55cffff640 1 -- 172.21.15.169:0/3398059375 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f55d0004660 con 0x7f55e00729f0 2024-03-28T01:01:56.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.824+0000 7f55e7d7f740 1 -- 172.21.15.169:0/3543073344 shutdown_connections 2024-03-28T01:01:56.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.826+0000 7f55cffff640 1 --2- 172.21.15.169:0/3398059375 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f55c80a2730 0x7f55c80a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:56.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.827+0000 7f55cffff640 1 -- 172.21.15.169:0/3398059375 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f55d00e23f0 con 0x7f55e00729f0 2024-03-28T01:01:56.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.827+0000 7f55e7d7f740 1 -- 172.21.15.169:0/3543073344 wait complete. 2024-03-28T01:01:56.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.828+0000 7f55e7d7f740 1 -- start start 2024-03-28T01:01:56.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.828+0000 7f55e7d7f740 1 Processor -- start 2024-03-28T01:01:56.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.828+0000 7f55e7d7f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e84502990 0x558e8450c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:56.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.828+0000 7f55e7d7f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84507b00 0x558e8450c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:56.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.828+0000 7f55e7d7f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84511910 0x558e84513d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:56.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.828+0000 7f55e7d7f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e8450aaa0 con 0x558e84507b00 2024-03-28T01:01:56.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.828+0000 7f55e7d7f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e84423060 con 0x558e84511910 2024-03-28T01:01:56.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.828+0000 7f55e7d7f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e8450ac20 con 0x558e84502990 2024-03-28T01:01:56.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.829+0000 7f55e7d7f740 10 client.0 _ll_register_callbacks cb 0x558e844056a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:56.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.831+0000 7f55de59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84507b00 0x558e8450c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:56.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.831+0000 7f55de59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84507b00 0x558e8450c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:54522/0 (socket says 172.21.15.169:54522) 2024-03-28T01:01:56.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.831+0000 7f55de59c640 1 -- 172.21.15.169:0/830942672 learned_addr learned my addr 172.21.15.169:0/830942672 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:56.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.831+0000 7f55de59c640 1 --2- 172.21.15.169:0/3398059375 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f55c80a2730 0x7f55c80a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:56.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.831+0000 7f55df59e640 1 --2- 172.21.15.169:0/830942672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84511910 0x558e84513d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:56.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.832+0000 7f55ded9d640 1 --2- 172.21.15.169:0/830942672 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e84502990 0x558e8450c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:56.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.835+0000 7f55df59e640 1 -- 172.21.15.169:0/830942672 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e84502990 msgr2=0x558e8450c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:56.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.835+0000 7f55df59e640 1 --2- 172.21.15.169:0/830942672 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e84502990 0x558e8450c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.835+0000 7f55df59e640 1 -- 172.21.15.169:0/830942672 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84507b00 msgr2=0x558e8450c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:56.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.835+0000 7f55df59e640 1 --2- 172.21.15.169:0/830942672 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84507b00 0x558e8450c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.835+0000 7f55df59e640 1 -- 172.21.15.169:0/830942672 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558e8417b3f0 con 0x558e84511910 2024-03-28T01:01:56.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.836+0000 7f55de59c640 1 --2- 172.21.15.169:0/3398059375 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f55c80a2730 0x7f55c80a4bf0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f55d400cfd0 tx=0x7f55d400c730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:56.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.841+0000 7f55df59e640 1 --2- 172.21.15.169:0/830942672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84511910 0x558e84513d00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f55d80097e0 tx=0x7f55d80094f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36a53cc859dee6cf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:56.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.842+0000 7f55c67fc640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:01:56.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.843+0000 7f55c67fc640 1 -- 172.21.15.169:0/830942672 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d80333e0 con 0x558e84511910 2024-03-28T01:01:56.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.843+0000 7f55e7d7f740 1 -- 172.21.15.169:0/830942672 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558e84474910 con 0x558e84511910 2024-03-28T01:01:56.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.843+0000 7f55c67fc640 1 -- 172.21.15.169:0/830942672 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f55d8033580 con 0x558e84511910 2024-03-28T01:01:56.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.845+0000 7f55c67fc640 1 -- 172.21.15.169:0/830942672 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f55d8057050 con 0x558e84511910 2024-03-28T01:01:56.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.845+0000 7f55e7d7f740 10 client.26357 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:56.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.845+0000 7f55e7d7f740 1 -- 172.21.15.169:0/830942672 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558e8413b0f0 con 0x558e84511910 2024-03-28T01:01:56.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.850+0000 7f55c67fc640 1 -- 172.21.15.169:0/830942672 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558e84474910 con 0x558e84511910 2024-03-28T01:01:56.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.851+0000 7f55c67fc640 1 -- 172.21.15.169:0/830942672 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f55d8049460 con 0x558e84511910 2024-03-28T01:01:56.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.852+0000 7f55e7d7f740 10 client.26357 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:56.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.852+0000 7f55e7d7f740 10 client.26357 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:01:56.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.852+0000 7f55e7d7f740 20 client.26357 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:56.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.852+0000 7f55e7d7f740 1 --2- 172.21.15.169:0/830942672 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558e845182d0 0x558e8451a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:56.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.852+0000 7f55e7d7f740 4 client.26357 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:01:56.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.852+0000 7f55e7d7f740 1 -- 172.21.15.169:0/830942672 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558e8446ec60 con 0x558e845182d0 2024-03-28T01:01:56.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.854+0000 7f55ded9d640 1 --2- 172.21.15.169:0/830942672 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558e845182d0 0x558e8451a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:56.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.857+0000 7f55ded9d640 1 --2- 172.21.15.169:0/830942672 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558e845182d0 0x558e8451a710 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=882cb2ff3c8691ef server_cookie=d7bb4be0b3fb7fd1 in_seq=0 out_seq=0 2024-03-28T01:01:56.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.858+0000 7f55c67fc640 10 client.26357 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:56.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.868+0000 7f55c67fc640 1 -- 172.21.15.169:0/830942672 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558e8446ec60 con 0x558e845182d0 2024-03-28T01:01:56.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.868+0000 7f55c67fc640 10 client.26357 handle_command_reply: tid=0 2024-03-28T01:01:56.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.868+0000 7f55e7d7f740 1 client.26357 shutdown 2024-03-28T01:01:56.876 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.872+0000 7f55e7d7f740 1 -- 172.21.15.169:0/830942672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84511910 msgr2=0x558e84513d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:56.876 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.872+0000 7f55e7d7f740 1 --2- 172.21.15.169:0/830942672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84511910 0x558e84513d00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f55d80097e0 tx=0x7f55d80094f0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.877 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.873+0000 7f55e7d7f740 1 -- 172.21.15.169:0/830942672 shutdown_connections 2024-03-28T01:01:56.877 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.873+0000 7f55e7d7f740 1 --2- 172.21.15.169:0/830942672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84511910 0x558e84513d00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.873+0000 7f55e7d7f740 1 --2- 172.21.15.169:0/830942672 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558e845182d0 0x558e8451a710 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.873+0000 7f55e7d7f740 1 --2- 172.21.15.169:0/830942672 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84507b00 0x558e8450c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.873+0000 7f55e7d7f740 1 --2- 172.21.15.169:0/830942672 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e84502990 0x558e8450c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.873+0000 7f55c67fc640 0 client.26357 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:56.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.873+0000 7f55e7d7f740 1 -- 172.21.15.169:0/830942672 >> 172.21.15.169:0/830942672 conn(0x558e844fd510 msgr2=0x558e845021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:56.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.880+0000 7f55e7d7f740 1 -- 172.21.15.169:0/830942672 shutdown_connections 2024-03-28T01:01:56.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.883+0000 7f55e7d7f740 1 -- 172.21.15.169:0/830942672 wait complete. 2024-03-28T01:01:56.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.884+0000 7f55e7d7f740 20 client.26357 trim_cache size 0 max 16384 2024-03-28T01:01:56.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.945+0000 7f55e7d7f740 1 Processor -- start 2024-03-28T01:01:56.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.946+0000 7f55e7d7f740 1 -- start start 2024-03-28T01:01:56.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.947+0000 7f55e7d7f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84502990 0x558e8450c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:56.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.947+0000 7f55e7d7f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e84507b00 0x558e8450c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:56.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.947+0000 7f55e7d7f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84511910 0x558e84503cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:56.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.947+0000 7f55e7d7f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c0011f10 con 0x558e84511910 2024-03-28T01:01:56.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.947+0000 7f55e7d7f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c000fa10 con 0x558e84502990 2024-03-28T01:01:56.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.947+0000 7f55e7d7f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c000db60 con 0x558e84507b00 2024-03-28T01:01:56.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.948+0000 7f55df59e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84511910 0x558e84503cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:56.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.948+0000 7f55df59e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84511910 0x558e84503cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:54536/0 (socket says 172.21.15.169:54536) 2024-03-28T01:01:56.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.948+0000 7f55df59e640 1 -- 172.21.15.169:0/3500470219 learned_addr learned my addr 172.21.15.169:0/3500470219 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:56.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.949+0000 7f55ded9d640 1 --2- 172.21.15.169:0/3500470219 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84502990 0x558e8450c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:56.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.949+0000 7f55de59c640 1 --2- 172.21.15.169:0/3500470219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e84507b00 0x558e8450c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:56.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.951+0000 7f55de59c640 1 -- 172.21.15.169:0/3500470219 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84502990 msgr2=0x558e8450c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:56.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.951+0000 7f55de59c640 1 --2- 172.21.15.169:0/3500470219 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84502990 0x558e8450c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.951+0000 7f55de59c640 1 -- 172.21.15.169:0/3500470219 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84511910 msgr2=0x558e84503cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:56.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.951+0000 7f55de59c640 1 --2- 172.21.15.169:0/3500470219 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84511910 0x558e84503cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.951+0000 7f55de59c640 1 -- 172.21.15.169:0/3500470219 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e84506f80 con 0x558e84507b00 2024-03-28T01:01:56.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.953+0000 7f55de59c640 1 --2- 172.21.15.169:0/3500470219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e84507b00 0x558e8450c8e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f55d4005d00 tx=0x7f55d401f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=676a3b8f413116a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:56.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.956+0000 7f55ccff9640 1 -- 172.21.15.169:0/3500470219 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d4024070 con 0x558e84507b00 2024-03-28T01:01:56.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.956+0000 7f55ccff9640 1 -- 172.21.15.169:0/3500470219 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f55d4003cd0 con 0x558e84507b00 2024-03-28T01:01:56.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.956+0000 7f55ccff9640 1 -- 172.21.15.169:0/3500470219 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d4020c90 con 0x558e84507b00 2024-03-28T01:01:56.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.961+0000 7f55e7d7f740 1 -- 172.21.15.169:0/3500470219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e84507b00 msgr2=0x558e8450c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:56.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.961+0000 7f55e7d7f740 1 --2- 172.21.15.169:0/3500470219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e84507b00 0x558e8450c8e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f55d4005d00 tx=0x7f55d401f000 comp rx=0 tx=0).stop 2024-03-28T01:01:56.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.967+0000 7f55e7d7f740 1 -- 172.21.15.169:0/3500470219 shutdown_connections 2024-03-28T01:01:56.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.967+0000 7f55e7d7f740 1 --2- 172.21.15.169:0/3500470219 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84511910 0x558e84503cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.967+0000 7f55e7d7f740 1 --2- 172.21.15.169:0/3500470219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e84507b00 0x558e8450c8e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.967+0000 7f55e7d7f740 1 --2- 172.21.15.169:0/3500470219 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84502990 0x558e8450c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.967+0000 7f55e7d7f740 1 -- 172.21.15.169:0/3500470219 >> 172.21.15.169:0/3500470219 conn(0x558e844fd510 msgr2=0x558e844fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:56.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.969+0000 7f55e7d7f740 1 -- 172.21.15.169:0/3500470219 shutdown_connections 2024-03-28T01:01:56.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.969+0000 7f55e7d7f740 1 -- 172.21.15.169:0/3500470219 wait complete. 2024-03-28T01:01:56.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.970+0000 7f55e7d7f740 1 -- start start 2024-03-28T01:01:56.976 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.970+0000 7f55e7d7f740 1 Processor -- start 2024-03-28T01:01:56.976 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.971+0000 7f55e7d7f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84502990 0x558e8452a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:56.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.971+0000 7f55e7d7f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84507b00 0x558e8452a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:56.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.971+0000 7f55e7d7f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e84511910 0x558e845287c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:56.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.971+0000 7f55e7d7f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c000e0d0 con 0x558e84507b00 2024-03-28T01:01:56.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.971+0000 7f55e7d7f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c000f3b0 con 0x558e84502990 2024-03-28T01:01:56.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.971+0000 7f55e7d7f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c000e6f0 con 0x558e84511910 2024-03-28T01:01:56.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.971+0000 7f55e7d7f740 10 client.0 _ll_register_callbacks cb 0x7f55d8009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:56.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.972+0000 7f55ded9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84502990 0x558e8452a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:56.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.972+0000 7f55ded9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84502990 0x558e8452a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49240/0 (socket says 172.21.15.169:49240) 2024-03-28T01:01:56.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.972+0000 7f55ded9d640 1 -- 172.21.15.169:0/4064174246 learned_addr learned my addr 172.21.15.169:0/4064174246 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:56.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.972+0000 7f55de59c640 1 --2- 172.21.15.169:0/4064174246 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84507b00 0x558e8452a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:56.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.974+0000 7f55ded9d640 1 -- 172.21.15.169:0/4064174246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e84511910 msgr2=0x558e845287c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:56.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.974+0000 7f55ded9d640 1 --2- 172.21.15.169:0/4064174246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e84511910 0x558e845287c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.974+0000 7f55ded9d640 1 -- 172.21.15.169:0/4064174246 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84507b00 msgr2=0x558e8452a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:56.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.974+0000 7f55ded9d640 1 --2- 172.21.15.169:0/4064174246 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84507b00 0x558e8452a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:56.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.974+0000 7f55ded9d640 1 -- 172.21.15.169:0/4064174246 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558e84528f00 con 0x558e84502990 2024-03-28T01:01:56.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.980+0000 7f55ded9d640 1 --2- 172.21.15.169:0/4064174246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84502990 0x558e8452a1b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f55d00ae900 tx=0x7f55d00ae970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37708e901f7da1ca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:56.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.981+0000 7f55c67fc640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:01:56.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.982+0000 7f55c67fc640 1 -- 172.21.15.169:0/4064174246 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d00e20b0 con 0x558e84502990 2024-03-28T01:01:56.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.982+0000 7f55e7d7f740 1 -- 172.21.15.169:0/4064174246 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f55d4003cd0 con 0x558e84502990 2024-03-28T01:01:56.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.982+0000 7f55c67fc640 1 -- 172.21.15.169:0/4064174246 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f55d00aaf50 con 0x558e84502990 2024-03-28T01:01:56.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.986+0000 7f55c67fc640 1 -- 172.21.15.169:0/4064174246 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f55d00b6c80 con 0x558e84502990 2024-03-28T01:01:56.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.991+0000 7f55e7d7f740 10 client.26363 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:56.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.991+0000 7f55e7d7f740 1 -- 172.21.15.169:0/4064174246 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558e84573b70 con 0x558e84502990 2024-03-28T01:01:57.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.998+0000 7f55c67fc640 1 -- 172.21.15.169:0/4064174246 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f55d4003cd0 con 0x558e84502990 2024-03-28T01:01:57.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.998+0000 7f55c67fc640 1 -- 172.21.15.169:0/4064174246 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f55d00adf50 con 0x558e84502990 2024-03-28T01:01:57.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.998+0000 7f55e7d7f740 10 client.26363 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:57.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.998+0000 7f55e7d7f740 10 client.26363 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:01:57.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.998+0000 7f55e7d7f740 20 client.26363 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:57.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.998+0000 7f55e7d7f740 1 --2- 172.21.15.169:0/4064174246 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558e845745f0 0x558e84576a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:57.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.998+0000 7f55e7d7f740 4 client.26363 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:01:57.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:56.998+0000 7f55e7d7f740 1 -- 172.21.15.169:0/4064174246 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558e843ec4e0 con 0x558e845745f0 2024-03-28T01:01:57.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.000+0000 7f55de59c640 1 --2- 172.21.15.169:0/4064174246 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558e845745f0 0x558e84576a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:57.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.012+0000 7f55de59c640 1 --2- 172.21.15.169:0/4064174246 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558e845745f0 0x558e84576a10 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc38d284fc29b7b4 server_cookie=42534018c5de5b36 in_seq=0 out_seq=0 2024-03-28T01:01:57.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.012+0000 7f55c67fc640 10 client.26363 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:57.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.014+0000 7f55c67fc640 1 -- 172.21.15.169:0/4064174246 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x558e843ec4e0 con 0x558e845745f0 2024-03-28T01:01:57.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.014+0000 7f55c67fc640 10 client.26363 handle_command_reply: tid=0 2024-03-28T01:01:57.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.014+0000 7f55e7d7f740 1 client.26363 shutdown 2024-03-28T01:01:57.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.019+0000 7f55e7d7f740 1 -- 172.21.15.169:0/4064174246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84502990 msgr2=0x558e8452a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:57.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.019+0000 7f55e7d7f740 1 --2- 172.21.15.169:0/4064174246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84502990 0x558e8452a1b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f55d00ae900 tx=0x7f55d00ae970 comp rx=0 tx=0).stop 2024-03-28T01:01:57.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.023+0000 7f55e7d7f740 1 -- 172.21.15.169:0/4064174246 shutdown_connections 2024-03-28T01:01:57.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.023+0000 7f55e7d7f740 1 --2- 172.21.15.169:0/4064174246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e84511910 0x558e845287c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:57.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.023+0000 7f55e7d7f740 1 --2- 172.21.15.169:0/4064174246 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558e845745f0 0x558e84576a10 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:57.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.023+0000 7f55e7d7f740 1 --2- 172.21.15.169:0/4064174246 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e84507b00 0x558e8452a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:57.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.023+0000 7f55e7d7f740 1 --2- 172.21.15.169:0/4064174246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e84502990 0x558e8452a1b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:57.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.023+0000 7f55e7d7f740 1 -- 172.21.15.169:0/4064174246 >> 172.21.15.169:0/4064174246 conn(0x558e844fd510 msgr2=0x558e84503cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:57.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.024+0000 7f55c67fc640 0 client.26363 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:57.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.024+0000 7f55e7d7f740 1 -- 172.21.15.169:0/4064174246 shutdown_connections 2024-03-28T01:01:57.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.028+0000 7f55e7d7f740 1 -- 172.21.15.169:0/4064174246 wait complete. 2024-03-28T01:01:57.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.030+0000 7f55e7d7f740 20 client.26363 trim_cache size 0 max 16384 2024-03-28T01:01:57.036 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:01:57.036 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "no active scrubs running", 2024-03-28T01:01:57.036 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": {} 2024-03-28T01:01:57.037 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:01:57.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.040+0000 7f55e498f640 1 -- 172.21.15.169:0/3398059375 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f55c80a2730 msgr2=0x7f55c80a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:57.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.040+0000 7f55e498f640 1 --2- 172.21.15.169:0/3398059375 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f55c80a2730 0x7f55c80a4bf0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f55d400cfd0 tx=0x7f55d400c730 comp rx=0 tx=0).stop 2024-03-28T01:01:57.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.040+0000 7f55e498f640 1 -- 172.21.15.169:0/3398059375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55e00729f0 msgr2=0x7f55e019dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:57.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.040+0000 7f55e498f640 1 --2- 172.21.15.169:0/3398059375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55e00729f0 0x7f55e019dd40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f55d001fa60 tx=0x7f55d001ff20 comp rx=0 tx=0).stop 2024-03-28T01:01:57.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.042+0000 7f55e498f640 1 -- 172.21.15.169:0/3398059375 shutdown_connections 2024-03-28T01:01:57.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.042+0000 7f55e498f640 1 --2- 172.21.15.169:0/3398059375 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f55c80a2730 0x7f55c80a4bf0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:57.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.042+0000 7f55e498f640 1 --2- 172.21.15.169:0/3398059375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55e0109640 0x7f55e01a52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:57.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.042+0000 7f55e498f640 1 --2- 172.21.15.169:0/3398059375 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55e0108ca0 0x7f55e019e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:57.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.042+0000 7f55e498f640 1 --2- 172.21.15.169:0/3398059375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55e00729f0 0x7f55e019dd40 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:57.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.042+0000 7f55e498f640 1 -- 172.21.15.169:0/3398059375 >> 172.21.15.169:0/3398059375 conn(0x7f55e006e4c0 msgr2=0x7f55e010bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:57.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.045+0000 7f55e498f640 1 -- 172.21.15.169:0/3398059375 shutdown_connections 2024-03-28T01:01:57.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.047+0000 7f55e498f640 1 -- 172.21.15.169:0/3398059375 wait complete. 2024-03-28T01:01:57.097 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T01:01:57.097 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T01:01:57.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.388+0000 7efec091c640 1 -- 172.21.15.169:0/161516365 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efebc100ed0 msgr2=0x7efebc105770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:57.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.388+0000 7efec091c640 1 --2- 172.21.15.169:0/161516365 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efebc100ed0 0x7efebc105770 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7efeb4009ca0 tx=0x7efeb402c140 comp rx=0 tx=0).stop 2024-03-28T01:01:57.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.389+0000 7efec091c640 1 -- 172.21.15.169:0/161516365 shutdown_connections 2024-03-28T01:01:57.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.389+0000 7efec091c640 1 --2- 172.21.15.169:0/161516365 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efebc100ed0 0x7efebc105770 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:57.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.389+0000 7efec091c640 1 --2- 172.21.15.169:0/161516365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efebc100510 0x7efebc100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:57.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.389+0000 7efec091c640 1 --2- 172.21.15.169:0/161516365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efebc0ff7b0 0x7efebc0ffbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:57.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.389+0000 7efec091c640 1 -- 172.21.15.169:0/161516365 >> 172.21.15.169:0/161516365 conn(0x7efebc0fb0e0 msgr2=0x7efebc0fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:57.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.394+0000 7efec091c640 1 -- 172.21.15.169:0/161516365 shutdown_connections 2024-03-28T01:01:57.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.396+0000 7efec091c640 1 -- 172.21.15.169:0/161516365 wait complete. 2024-03-28T01:01:57.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.398+0000 7efec091c640 1 Processor -- start 2024-03-28T01:01:57.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.399+0000 7efec091c640 1 -- start start 2024-03-28T01:01:57.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.399+0000 7efec091c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efebc0ff7b0 0x7efebc10bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:57.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.399+0000 7efec091c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efebc100510 0x7efebc10c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:57.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.399+0000 7efec091c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efebc100ed0 0x7efebc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:57.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.399+0000 7efec091c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc113800 con 0x7efebc100ed0 2024-03-28T01:01:57.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.399+0000 7efec091c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc113680 con 0x7efebc100510 2024-03-28T01:01:57.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.400+0000 7efec091c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc113980 con 0x7efebc0ff7b0 2024-03-28T01:01:57.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.401+0000 7efebb59e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efebc100ed0 0x7efebc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:57.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.401+0000 7efebb59e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efebc100ed0 0x7efebc06f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:54546/0 (socket says 172.21.15.169:54546) 2024-03-28T01:01:57.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.401+0000 7efebb59e640 1 -- 172.21.15.169:0/1035009080 learned_addr learned my addr 172.21.15.169:0/1035009080 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:57.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.402+0000 7efebb59e640 1 -- 172.21.15.169:0/1035009080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efebc0ff7b0 msgr2=0x7efebc10bb30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:01:57.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.402+0000 7efebb59e640 1 --2- 172.21.15.169:0/1035009080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efebc0ff7b0 0x7efebc10bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:57.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.402+0000 7efebb59e640 1 -- 172.21.15.169:0/1035009080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efebc100510 msgr2=0x7efebc10c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:57.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.402+0000 7efebb59e640 1 --2- 172.21.15.169:0/1035009080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efebc100510 0x7efebc10c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:57.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.402+0000 7efebb59e640 1 -- 172.21.15.169:0/1035009080 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efebc06f620 con 0x7efebc100ed0 2024-03-28T01:01:57.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.403+0000 7efebb59e640 1 --2- 172.21.15.169:0/1035009080 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efebc100ed0 0x7efebc06f0e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7efeb4032650 tx=0x7efeb4035e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:57.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.404+0000 7efea3fff640 1 -- 172.21.15.169:0/1035009080 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb4039040 con 0x7efebc100ed0 2024-03-28T01:01:57.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.404+0000 7efec091c640 1 -- 172.21.15.169:0/1035009080 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efebc06f8b0 con 0x7efebc100ed0 2024-03-28T01:01:57.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.404+0000 7efea3fff640 1 -- 172.21.15.169:0/1035009080 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efeb4044070 con 0x7efebc100ed0 2024-03-28T01:01:57.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.404+0000 7efea3fff640 1 -- 172.21.15.169:0/1035009080 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb4004780 con 0x7efebc100ed0 2024-03-28T01:01:57.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.404+0000 7efec091c640 1 -- 172.21.15.169:0/1035009080 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efebc06fe70 con 0x7efebc100ed0 2024-03-28T01:01:57.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.407+0000 7efec091c640 1 -- 172.21.15.169:0/1035009080 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efebc10c770 con 0x7efebc100ed0 2024-03-28T01:01:57.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.411+0000 7efea3fff640 1 -- 172.21.15.169:0/1035009080 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7efeb4003970 con 0x7efebc100ed0 2024-03-28T01:01:57.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.418+0000 7efea3fff640 1 --2- 172.21.15.169:0/1035009080 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efea40a2950 0x7efea40a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:57.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.419+0000 7efea3fff640 1 -- 172.21.15.169:0/1035009080 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7efeb40f1e80 con 0x7efebc100ed0 2024-03-28T01:01:57.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.426+0000 7efebad9d640 1 --2- 172.21.15.169:0/1035009080 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efea40a2950 0x7efea40a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:57.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.428+0000 7efea3fff640 1 -- 172.21.15.169:0/1035009080 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efeb40beb20 con 0x7efebc100ed0 2024-03-28T01:01:57.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:57.430+0000 7efebad9d640 1 --2- 172.21.15.169:0/1035009080 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efea40a2950 0x7efea40a4e10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7efeb0009670 tx=0x7efeb0006bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:57.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:57 smithi169 ceph-mon[32274]: pgmap v514: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 812 MiB/s wr, 1.64k op/s 2024-03-28T01:01:57.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:57 smithi174 ceph-mon[36060]: pgmap v514: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 812 MiB/s wr, 1.64k op/s 2024-03-28T01:01:57.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:57 smithi177 ceph-mon[36206]: pgmap v514: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 812 MiB/s wr, 1.64k op/s 2024-03-28T01:01:58.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.014+0000 7efec091c640 1 -- 172.21.15.169:0/1035009080 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efebc0622a0 con 0x7efebc100ed0 2024-03-28T01:01:58.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.017+0000 7efea3fff640 1 -- 172.21.15.169:0/1035009080 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7efeb40be420 con 0x7efebc100ed0 2024-03-28T01:01:58.023 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T01:01:58.023 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T01:01:58.024 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T01:01:58.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.022+0000 7efea1ffb640 1 -- 172.21.15.169:0/1035009080 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efea40a2950 msgr2=0x7efea40a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:58.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.023+0000 7efea1ffb640 1 --2- 172.21.15.169:0/1035009080 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efea40a2950 0x7efea40a4e10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7efeb0009670 tx=0x7efeb0006bd0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.023+0000 7efea1ffb640 1 -- 172.21.15.169:0/1035009080 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efebc100ed0 msgr2=0x7efebc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:58.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.023+0000 7efea1ffb640 1 --2- 172.21.15.169:0/1035009080 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efebc100ed0 0x7efebc06f0e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7efeb4032650 tx=0x7efeb4035e50 comp rx=0 tx=0).stop 2024-03-28T01:01:58.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.025+0000 7efea1ffb640 1 -- 172.21.15.169:0/1035009080 shutdown_connections 2024-03-28T01:01:58.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.025+0000 7efea1ffb640 1 --2- 172.21.15.169:0/1035009080 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efea40a2950 0x7efea40a4e10 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.025+0000 7efea1ffb640 1 --2- 172.21.15.169:0/1035009080 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efebc100ed0 0x7efebc06f0e0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.025+0000 7efea1ffb640 1 --2- 172.21.15.169:0/1035009080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efebc100510 0x7efebc10c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.025+0000 7efea1ffb640 1 --2- 172.21.15.169:0/1035009080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efebc0ff7b0 0x7efebc10bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.025+0000 7efea1ffb640 1 -- 172.21.15.169:0/1035009080 >> 172.21.15.169:0/1035009080 conn(0x7efebc0fb0e0 msgr2=0x7efebc101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:58.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.026+0000 7efea1ffb640 1 -- 172.21.15.169:0/1035009080 shutdown_connections 2024-03-28T01:01:58.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.026+0000 7efea1ffb640 1 -- 172.21.15.169:0/1035009080 wait complete. 2024-03-28T01:01:58.071 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:01:58.071 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T01:01:58.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.445+0000 7face9192640 1 -- 172.21.15.169:0/357556325 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7face40ffc00 msgr2=0x7face4100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:58.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.445+0000 7face9192640 1 --2- 172.21.15.169:0/357556325 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7face40ffc00 0x7face4100020 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7facd8009900 tx=0x7facd8030090 comp rx=0 tx=0).stop 2024-03-28T01:01:58.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.446+0000 7face9192640 1 -- 172.21.15.169:0/357556325 shutdown_connections 2024-03-28T01:01:58.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.446+0000 7face9192640 1 --2- 172.21.15.169:0/357556325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7face41017b0 0x7face4108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.446+0000 7face9192640 1 --2- 172.21.15.169:0/357556325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7face4100df0 0x7face4101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.446+0000 7face9192640 1 --2- 172.21.15.169:0/357556325 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7face40ffc00 0x7face4100020 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.446+0000 7face9192640 1 -- 172.21.15.169:0/357556325 >> 172.21.15.169:0/357556325 conn(0x7face40fb190 msgr2=0x7face40fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:58.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.449+0000 7face9192640 1 -- 172.21.15.169:0/357556325 shutdown_connections 2024-03-28T01:01:58.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.452+0000 7face9192640 1 -- 172.21.15.169:0/357556325 wait complete. 2024-03-28T01:01:58.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.452+0000 7face9192640 1 Processor -- start 2024-03-28T01:01:58.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.456+0000 7face9192640 1 -- start start 2024-03-28T01:01:58.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.456+0000 7face9192640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7face40ffc00 0x7face419df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.456+0000 7face9192640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7face4100df0 0x7face419e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.456+0000 7face9192640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7face41017b0 0x7face41a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.456+0000 7face9192640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face41138d0 con 0x7face40ffc00 2024-03-28T01:01:58.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.456+0000 7face9192640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face4113750 con 0x7face4100df0 2024-03-28T01:01:58.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.456+0000 7face9192640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face4113a50 con 0x7face41017b0 2024-03-28T01:01:58.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.456+0000 7face3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7face40ffc00 0x7face419df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:58.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.456+0000 7face3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7face40ffc00 0x7face419df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:54568/0 (socket says 172.21.15.169:54568) 2024-03-28T01:01:58.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.457+0000 7face3fff640 1 -- 172.21.15.169:0/1333297623 learned_addr learned my addr 172.21.15.169:0/1333297623 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:58.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.458+0000 7face8991640 1 --2- 172.21.15.169:0/1333297623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7face41017b0 0x7face41a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:58.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.459+0000 7face8991640 1 -- 172.21.15.169:0/1333297623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7face4100df0 msgr2=0x7face419e4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:01:58.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.459+0000 7face37fe640 1 --2- 172.21.15.169:0/1333297623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7face4100df0 0x7face419e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:58.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.459+0000 7face8991640 1 --2- 172.21.15.169:0/1333297623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7face4100df0 0x7face419e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.459+0000 7face8991640 1 -- 172.21.15.169:0/1333297623 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7face40ffc00 msgr2=0x7face419df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:58.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.459+0000 7face8991640 1 --2- 172.21.15.169:0/1333297623 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7face40ffc00 0x7face419df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.459+0000 7face8991640 1 -- 172.21.15.169:0/1333297623 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face41a5b90 con 0x7face41017b0 2024-03-28T01:01:58.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.459+0000 7face3fff640 1 --2- 172.21.15.169:0/1333297623 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7face40ffc00 0x7face419df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:01:58.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.459+0000 7face8991640 1 --2- 172.21.15.169:0/1333297623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7face41017b0 0x7face41a5520 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7facdc00da10 tx=0x7facdc00ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:58.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.463+0000 7face17fa640 1 -- 172.21.15.169:0/1333297623 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc00bd70 con 0x7face41017b0 2024-03-28T01:01:58.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.463+0000 7face9192640 1 -- 172.21.15.169:0/1333297623 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face41a5e80 con 0x7face41017b0 2024-03-28T01:01:58.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.463+0000 7face9192640 1 -- 172.21.15.169:0/1333297623 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face41a6340 con 0x7face41017b0 2024-03-28T01:01:58.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.463+0000 7face17fa640 1 -- 172.21.15.169:0/1333297623 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7facdc003e50 con 0x7face41017b0 2024-03-28T01:01:58.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.463+0000 7face17fa640 1 -- 172.21.15.169:0/1333297623 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc0106c0 con 0x7face41017b0 2024-03-28T01:01:58.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.470+0000 7facecfe8740 1 Processor -- start 2024-03-28T01:01:58.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.474+0000 7face17fa640 1 -- 172.21.15.169:0/1333297623 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7facdc010860 con 0x7face41017b0 2024-03-28T01:01:58.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.474+0000 7facecfe8740 1 -- start start 2024-03-28T01:01:58.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.474+0000 7facecfe8740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56527cb08860 0x56527cb08c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.474+0000 7facecfe8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb09250 0x56527cb0dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.474+0000 7facecfe8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56527cb0e3c0 0x56527cb10850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.474+0000 7facecfe8740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56527cacb610 con 0x56527cb08860 2024-03-28T01:01:58.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.474+0000 7facecfe8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x56527ca6e3c0 con 0x56527cb0e3c0 2024-03-28T01:01:58.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.474+0000 7facecfe8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56527ca16d60 con 0x56527cb09250 2024-03-28T01:01:58.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.475+0000 7face3fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb09250 0x56527cb0dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:58.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.475+0000 7face3fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb09250 0x56527cb0dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:55920/0 (socket says 172.21.15.169:55920) 2024-03-28T01:01:58.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.475+0000 7face3fff640 1 -- 172.21.15.169:0/3785933350 learned_addr learned my addr 172.21.15.169:0/3785933350 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:58.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.475+0000 7face3fff640 1 -- 172.21.15.169:0/3785933350 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56527cb0e3c0 msgr2=0x56527cb10850 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:01:58.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.475+0000 7face3fff640 1 --2- 172.21.15.169:0/3785933350 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56527cb0e3c0 0x56527cb10850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.475+0000 7face3fff640 1 -- 172.21.15.169:0/3785933350 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56527cb08860 msgr2=0x56527cb08c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:58.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.475+0000 7face3fff640 1 --2- 172.21.15.169:0/3785933350 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56527cb08860 0x56527cb08c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.475+0000 7face3fff640 1 -- 172.21.15.169:0/3785933350 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56527c7823f0 con 0x56527cb09250 2024-03-28T01:01:58.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.477+0000 7face3fff640 1 --2- 172.21.15.169:0/3785933350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb09250 0x56527cb0dd50 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7facd8009230 tx=0x7facd80044a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b73f4fb35e18ad8b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:58.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.479+0000 7face17fa640 1 --2- 172.21.15.169:0/1333297623 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faccc0a2a00 0x7faccc0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.479+0000 7face17fa640 1 -- 172.21.15.169:0/1333297623 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7facdc0cf620 con 0x7face41017b0 2024-03-28T01:01:58.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.480+0000 7facca7fc640 1 -- 172.21.15.169:0/3785933350 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd8046070 con 0x56527cb09250 2024-03-28T01:01:58.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.480+0000 7facca7fc640 1 -- 172.21.15.169:0/3785933350 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7facd80104f0 con 0x56527cb09250 2024-03-28T01:01:58.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.480+0000 7facca7fc640 1 -- 172.21.15.169:0/3785933350 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd80416e0 con 0x56527cb09250 2024-03-28T01:01:58.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.482+0000 7facecfe8740 1 -- 172.21.15.169:0/3785933350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb09250 msgr2=0x56527cb0dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:58.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.482+0000 7facecfe8740 1 --2- 172.21.15.169:0/3785933350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb09250 0x56527cb0dd50 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7facd8009230 tx=0x7facd80044a0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.483+0000 7facecfe8740 1 -- 172.21.15.169:0/3785933350 shutdown_connections 2024-03-28T01:01:58.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.483+0000 7facecfe8740 1 --2- 172.21.15.169:0/3785933350 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56527cb0e3c0 0x56527cb10850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.483+0000 7facecfe8740 1 --2- 172.21.15.169:0/3785933350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb09250 0x56527cb0dd50 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.483+0000 7facecfe8740 1 --2- 172.21.15.169:0/3785933350 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56527cb08860 0x56527cb08c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.483+0000 7facecfe8740 1 -- 172.21.15.169:0/3785933350 >> 172.21.15.169:0/3785933350 conn(0x56527cb03dd0 msgr2=0x56527cb041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:58.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.485+0000 7face37fe640 1 --2- 172.21.15.169:0/1333297623 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faccc0a2a00 0x7faccc0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:58.577 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:58 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1035009080' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:01:58.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.487+0000 7facecfe8740 1 -- 172.21.15.169:0/3785933350 shutdown_connections 2024-03-28T01:01:58.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.487+0000 7facecfe8740 1 -- 172.21.15.169:0/3785933350 wait complete. 2024-03-28T01:01:58.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.488+0000 7facecfe8740 1 -- start start 2024-03-28T01:01:58.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.488+0000 7facecfe8740 1 Processor -- start 2024-03-28T01:01:58.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.491+0000 7facecfe8740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56527cb08860 0x56527cb12bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.491+0000 7facecfe8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb09250 0x56527cb13130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.491+0000 7facecfe8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56527cb0e3c0 0x56527cb1a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.491+0000 7facecfe8740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56527ca2a240 con 0x56527cb08860 2024-03-28T01:01:58.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.491+0000 7facecfe8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x56527ca6a460 con 0x56527cb0e3c0 2024-03-28T01:01:58.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.491+0000 7facecfe8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56527cb11310 con 0x56527cb09250 2024-03-28T01:01:58.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.491+0000 7facecfe8740 10 client.0 _ll_register_callbacks cb 0x56527ca29c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:58.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.492+0000 7face37fe640 1 --2- 172.21.15.169:0/1333297623 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faccc0a2a00 0x7faccc0a4ec0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x56527cb09ab0 tx=0x7facd4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:58.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.493+0000 7face3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56527cb08860 0x56527cb12bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:58.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.493+0000 7face3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56527cb08860 0x56527cb12bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:54598/0 (socket says 172.21.15.169:54598) 2024-03-28T01:01:58.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.494+0000 7face3fff640 1 -- 172.21.15.169:0/660989961 learned_addr learned my addr 172.21.15.169:0/660989961 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:58.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.494+0000 7face8991640 1 --2- 172.21.15.169:0/660989961 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56527cb0e3c0 0x56527cb1a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:58.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.494+0000 7face37fe640 1 --2- 172.21.15.169:0/660989961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb09250 0x56527cb13130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:58.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.496+0000 7face37fe640 1 -- 172.21.15.169:0/660989961 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56527cb0e3c0 msgr2=0x56527cb1a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:58.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.496+0000 7face37fe640 1 --2- 172.21.15.169:0/660989961 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56527cb0e3c0 0x56527cb1a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.496+0000 7face37fe640 1 -- 172.21.15.169:0/660989961 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56527cb08860 msgr2=0x56527cb12bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:58.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.496+0000 7face37fe640 1 --2- 172.21.15.169:0/660989961 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56527cb08860 0x56527cb12bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.496+0000 7face37fe640 1 -- 172.21.15.169:0/660989961 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56527c84c7d0 con 0x56527cb09250 2024-03-28T01:01:58.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.497+0000 7face37fe640 1 --2- 172.21.15.169:0/660989961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb09250 0x56527cb13130 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7facd400cb50 tx=0x7facd40046e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=209defea3f6cbc65 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:58.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.503+0000 7facc37fe640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:01:58.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.503+0000 7facc37fe640 1 -- 172.21.15.169:0/660989961 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd4021020 con 0x56527cb09250 2024-03-28T01:01:58.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.503+0000 7facc37fe640 1 -- 172.21.15.169:0/660989961 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7facd4013c10 con 0x56527cb09250 2024-03-28T01:01:58.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.505+0000 7facecfe8740 1 -- 172.21.15.169:0/660989961 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7facd80104f0 con 0x56527cb09250 2024-03-28T01:01:58.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.506+0000 7facc37fe640 1 -- 172.21.15.169:0/660989961 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7facd4019040 con 0x56527cb09250 2024-03-28T01:01:58.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.508+0000 7facecfe8740 10 client.26305 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:58.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.508+0000 7facecfe8740 1 -- 172.21.15.169:0/660989961 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56527cb2e5f0 con 0x56527cb09250 2024-03-28T01:01:58.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.508+0000 7facc37fe640 1 -- 172.21.15.169:0/660989961 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7facd80104f0 con 0x56527cb09250 2024-03-28T01:01:58.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.508+0000 7facc37fe640 1 -- 172.21.15.169:0/660989961 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7facd402e020 con 0x56527cb09250 2024-03-28T01:01:58.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.509+0000 7facecfe8740 10 client.26305 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:58.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.509+0000 7facecfe8740 10 client.26305 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:01:58.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.509+0000 7facecfe8740 20 client.26305 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:58.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.509+0000 7facecfe8740 1 --2- 172.21.15.169:0/660989961 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x56527cb2fc80 0x56527cb32100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.509+0000 7facecfe8740 4 client.26305 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:01:58.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.509+0000 7facecfe8740 1 -- 172.21.15.169:0/660989961 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56527ca7b900 con 0x56527cb2fc80 2024-03-28T01:01:58.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.510+0000 7face3fff640 1 --2- 172.21.15.169:0/660989961 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x56527cb2fc80 0x56527cb32100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:58.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.521+0000 7face3fff640 1 --2- 172.21.15.169:0/660989961 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x56527cb2fc80 0x56527cb32100 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ebc533f3109a2138 server_cookie=dbf60aec7cb99ffa in_seq=0 out_seq=0 2024-03-28T01:01:58.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.523+0000 7facc37fe640 10 client.26305 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:01:58.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.524+0000 7facc37fe640 1 -- 172.21.15.169:0/660989961 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56527ca7b900 con 0x56527cb2fc80 2024-03-28T01:01:58.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.524+0000 7facc37fe640 10 client.26305 handle_command_reply: tid=0 2024-03-28T01:01:58.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.524+0000 7facecfe8740 1 client.26305 shutdown 2024-03-28T01:01:58.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.528+0000 7facecfe8740 1 -- 172.21.15.169:0/660989961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb09250 msgr2=0x56527cb13130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:58.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.528+0000 7facecfe8740 1 --2- 172.21.15.169:0/660989961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb09250 0x56527cb13130 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7facd400cb50 tx=0x7facd40046e0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.529+0000 7facecfe8740 1 -- 172.21.15.169:0/660989961 shutdown_connections 2024-03-28T01:01:58.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.529+0000 7facecfe8740 1 --2- 172.21.15.169:0/660989961 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56527cb0e3c0 0x56527cb1a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.529+0000 7facecfe8740 1 --2- 172.21.15.169:0/660989961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb09250 0x56527cb13130 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.529+0000 7facecfe8740 1 --2- 172.21.15.169:0/660989961 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x56527cb2fc80 0x56527cb32100 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.529+0000 7facecfe8740 1 --2- 172.21.15.169:0/660989961 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56527cb08860 0x56527cb12bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.529+0000 7facecfe8740 1 -- 172.21.15.169:0/660989961 >> 172.21.15.169:0/660989961 conn(0x56527cb03dd0 msgr2=0x56527cb03070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:58.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.529+0000 7facc37fe640 0 client.26305 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:01:58.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.529+0000 7facecfe8740 1 -- 172.21.15.169:0/660989961 shutdown_connections 2024-03-28T01:01:58.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.531+0000 7facecfe8740 1 -- 172.21.15.169:0/660989961 wait complete. 2024-03-28T01:01:58.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.531+0000 7facecfe8740 20 client.26305 trim_cache size 0 max 16384 2024-03-28T01:01:58.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.560+0000 7facecfe8740 1 Processor -- start 2024-03-28T01:01:58.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.560+0000 7facecfe8740 1 -- start start 2024-03-28T01:01:58.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.560+0000 7facecfe8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb08860 0x56527cb027e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.560+0000 7facecfe8740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56527cb09250 0x56527cb1a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.560+0000 7face3fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb08860 0x56527cb027e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:58.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.560+0000 7facecfe8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56527cb0e3c0 0x56527cb02d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.560+0000 7face3fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb08860 0x56527cb027e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:55942/0 (socket says 172.21.15.169:55942) 2024-03-28T01:01:58.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.560+0000 7face3fff640 1 -- 172.21.15.169:0/1489241530 learned_addr learned my addr 172.21.15.169:0/1489241530 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:58.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.560+0000 7facecfe8740 1 -- 172.21.15.169:0/1489241530 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd00120e0 con 0x56527cb09250 2024-03-28T01:01:58.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.560+0000 7facecfe8740 1 -- 172.21.15.169:0/1489241530 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd000fbe0 con 0x56527cb0e3c0 2024-03-28T01:01:58.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.560+0000 7facecfe8740 1 -- 172.21.15.169:0/1489241530 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd000dc80 con 0x56527cb08860 2024-03-28T01:01:58.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.561+0000 7face8991640 1 --2- 172.21.15.169:0/1489241530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56527cb0e3c0 0x56527cb02d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:58.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.561+0000 7face37fe640 1 --2- 172.21.15.169:0/1489241530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56527cb09250 0x56527cb1a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:58.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.561+0000 7face8991640 1 -- 172.21.15.169:0/1489241530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb08860 msgr2=0x56527cb027e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:58.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.561+0000 7face8991640 1 --2- 172.21.15.169:0/1489241530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb08860 0x56527cb027e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.561+0000 7face8991640 1 -- 172.21.15.169:0/1489241530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56527cb09250 msgr2=0x56527cb1a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:58.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.561+0000 7face8991640 1 --2- 172.21.15.169:0/1489241530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56527cb09250 0x56527cb1a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.561+0000 7face8991640 1 -- 172.21.15.169:0/1489241530 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56527cb0da30 con 0x56527cb0e3c0 2024-03-28T01:01:58.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.561+0000 7face8991640 1 --2- 172.21.15.169:0/1489241530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56527cb0e3c0 0x56527cb02d20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7facdc0a3e30 tx=0x7facdc0ccef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b9dc4b0d76a62 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:58.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.562+0000 7facc9ffb640 1 -- 172.21.15.169:0/1489241530 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc010b50 con 0x56527cb0e3c0 2024-03-28T01:01:58.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.562+0000 7facc9ffb640 1 -- 172.21.15.169:0/1489241530 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7facdc0b4070 con 0x56527cb0e3c0 2024-03-28T01:01:58.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.562+0000 7facc9ffb640 1 -- 172.21.15.169:0/1489241530 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc09c0c0 con 0x56527cb0e3c0 2024-03-28T01:01:58.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.563+0000 7facecfe8740 1 -- 172.21.15.169:0/1489241530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56527cb0e3c0 msgr2=0x56527cb02d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:58.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.563+0000 7facecfe8740 1 --2- 172.21.15.169:0/1489241530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56527cb0e3c0 0x56527cb02d20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7facdc0a3e30 tx=0x7facdc0ccef0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.564+0000 7facecfe8740 1 -- 172.21.15.169:0/1489241530 shutdown_connections 2024-03-28T01:01:58.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.564+0000 7facecfe8740 1 --2- 172.21.15.169:0/1489241530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56527cb0e3c0 0x56527cb02d20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.564+0000 7facecfe8740 1 --2- 172.21.15.169:0/1489241530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56527cb09250 0x56527cb1a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.564+0000 7facecfe8740 1 --2- 172.21.15.169:0/1489241530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb08860 0x56527cb027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.564+0000 7facecfe8740 1 -- 172.21.15.169:0/1489241530 >> 172.21.15.169:0/1489241530 conn(0x56527cb03dd0 msgr2=0x56527cb12bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:58.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.567+0000 7facecfe8740 1 -- 172.21.15.169:0/1489241530 shutdown_connections 2024-03-28T01:01:58.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.567+0000 7facecfe8740 1 -- 172.21.15.169:0/1489241530 wait complete. 2024-03-28T01:01:58.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.568+0000 7facecfe8740 1 -- start start 2024-03-28T01:01:58.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.568+0000 7facecfe8740 1 Processor -- start 2024-03-28T01:01:58.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.569+0000 7facecfe8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56527cb08860 0x56527cb17970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.569+0000 7facecfe8740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56527cb09250 0x56527cb17eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.569+0000 7facecfe8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb0e3c0 0x56527cb18ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.569+0000 7facecfe8740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd000e280 con 0x56527cb09250 2024-03-28T01:01:58.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.569+0000 7facecfe8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd000f580 con 0x56527cb08860 2024-03-28T01:01:58.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.569+0000 7facecfe8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd000e8c0 con 0x56527cb0e3c0 2024-03-28T01:01:58.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.569+0000 7facecfe8740 10 client.0 _ll_register_callbacks cb 0x7facd400c800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:58.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.569+0000 7face8991640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb0e3c0 0x56527cb18ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:58.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.569+0000 7face8991640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb0e3c0 0x56527cb18ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:55948/0 (socket says 172.21.15.169:55948) 2024-03-28T01:01:58.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.569+0000 7face8991640 1 -- 172.21.15.169:0/1745124650 learned_addr learned my addr 172.21.15.169:0/1745124650 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:58.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.570+0000 7face37fe640 1 --2- 172.21.15.169:0/1745124650 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56527cb09250 0x56527cb17eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:58.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.570+0000 7face8991640 1 -- 172.21.15.169:0/1745124650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56527cb08860 msgr2=0x56527cb17970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:01:58.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.570+0000 7face8991640 1 --2- 172.21.15.169:0/1745124650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56527cb08860 0x56527cb17970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.570+0000 7face8991640 1 -- 172.21.15.169:0/1745124650 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56527cb09250 msgr2=0x56527cb17eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:58.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.570+0000 7face8991640 1 --2- 172.21.15.169:0/1745124650 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56527cb09250 0x56527cb17eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.570+0000 7face8991640 1 -- 172.21.15.169:0/1745124650 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56527cb194b0 con 0x56527cb0e3c0 2024-03-28T01:01:58.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.570+0000 7face37fe640 1 --2- 172.21.15.169:0/1745124650 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56527cb09250 0x56527cb17eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:01:58.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.575+0000 7face8991640 1 --2- 172.21.15.169:0/1745124650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb0e3c0 0x56527cb18ec0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7facdc0cf510 tx=0x7facdc0cf540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=746e1ed0819b5971 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:58.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.575+0000 7facc37fe640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:01:58.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.577+0000 7facc37fe640 1 -- 172.21.15.169:0/1745124650 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc0b9060 con 0x56527cb0e3c0 2024-03-28T01:01:58.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.578+0000 7facc37fe640 1 -- 172.21.15.169:0/1745124650 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7facdc010d50 con 0x56527cb0e3c0 2024-03-28T01:01:58.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.578+0000 7facecfe8740 1 -- 172.21.15.169:0/1745124650 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7facdc0b4070 con 0x56527cb0e3c0 2024-03-28T01:01:58.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.580+0000 7facc37fe640 1 -- 172.21.15.169:0/1745124650 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7facdc0ba250 con 0x56527cb0e3c0 2024-03-28T01:01:58.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.581+0000 7facecfe8740 10 client.26311 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:58.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.581+0000 7facecfe8740 1 -- 172.21.15.169:0/1745124650 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56527cb208a0 con 0x56527cb0e3c0 2024-03-28T01:01:58.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.583+0000 7facc37fe640 1 -- 172.21.15.169:0/1745124650 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7facdc0b4070 con 0x56527cb0e3c0 2024-03-28T01:01:58.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.583+0000 7facc37fe640 1 -- 172.21.15.169:0/1745124650 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7facdc024060 con 0x56527cb0e3c0 2024-03-28T01:01:58.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.586+0000 7facecfe8740 10 client.26311 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:58.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.586+0000 7facecfe8740 10 client.26311 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:01:58.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.586+0000 7facecfe8740 20 client.26311 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:58.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.586+0000 7facecfe8740 1 --2- 172.21.15.169:0/1745124650 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x56527cb21110 0x56527cb23530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.586+0000 7facecfe8740 4 client.26311 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:01:58.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.586+0000 7facecfe8740 1 -- 172.21.15.169:0/1745124650 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56527ca75c50 con 0x56527cb21110 2024-03-28T01:01:58.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.587+0000 7face3fff640 1 --2- 172.21.15.169:0/1745124650 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x56527cb21110 0x56527cb23530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:58.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.588+0000 7face3fff640 1 --2- 172.21.15.169:0/1745124650 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x56527cb21110 0x56527cb23530 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f4d62d2a5f099b10 server_cookie=cbc033b0c005a401 in_seq=0 out_seq=0 2024-03-28T01:01:58.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.588+0000 7facc37fe640 10 client.26311 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:01:58.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.589+0000 7facc37fe640 1 -- 172.21.15.169:0/1745124650 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56527ca75c50 con 0x56527cb21110 2024-03-28T01:01:58.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.589+0000 7facc37fe640 10 client.26311 handle_command_reply: tid=0 2024-03-28T01:01:58.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.589+0000 7facecfe8740 1 client.26311 shutdown 2024-03-28T01:01:58.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.591+0000 7facecfe8740 1 -- 172.21.15.169:0/1745124650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb0e3c0 msgr2=0x56527cb18ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:58.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.591+0000 7facecfe8740 1 --2- 172.21.15.169:0/1745124650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb0e3c0 0x56527cb18ec0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7facdc0cf510 tx=0x7facdc0cf540 comp rx=0 tx=0).stop 2024-03-28T01:01:58.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.591+0000 7facecfe8740 1 -- 172.21.15.169:0/1745124650 shutdown_connections 2024-03-28T01:01:58.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.591+0000 7facecfe8740 1 --2- 172.21.15.169:0/1745124650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56527cb0e3c0 0x56527cb18ec0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.591+0000 7facecfe8740 1 --2- 172.21.15.169:0/1745124650 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x56527cb21110 0x56527cb23530 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.591+0000 7facecfe8740 1 --2- 172.21.15.169:0/1745124650 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56527cb09250 0x56527cb17eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.591+0000 7facecfe8740 1 --2- 172.21.15.169:0/1745124650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56527cb08860 0x56527cb17970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.591+0000 7facecfe8740 1 -- 172.21.15.169:0/1745124650 >> 172.21.15.169:0/1745124650 conn(0x56527cb03dd0 msgr2=0x56527cb07030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:58.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.591+0000 7facc37fe640 0 client.26311 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:01:58.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.592+0000 7facecfe8740 1 -- 172.21.15.169:0/1745124650 shutdown_connections 2024-03-28T01:01:58.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.592+0000 7facecfe8740 1 -- 172.21.15.169:0/1745124650 wait complete. 2024-03-28T01:01:58.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.592+0000 7facecfe8740 20 client.26311 trim_cache size 0 max 16384 2024-03-28T01:01:58.612 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:01:58.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.597+0000 7face9192640 1 -- 172.21.15.169:0/1333297623 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faccc0a2a00 msgr2=0x7faccc0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:58.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.597+0000 7face9192640 1 --2- 172.21.15.169:0/1333297623 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faccc0a2a00 0x7faccc0a4ec0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x56527cb09ab0 tx=0x7facd4008040 comp rx=0 tx=0).stop 2024-03-28T01:01:58.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.597+0000 7face9192640 1 -- 172.21.15.169:0/1333297623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7face41017b0 msgr2=0x7face41a5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:58.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.597+0000 7face9192640 1 --2- 172.21.15.169:0/1333297623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7face41017b0 0x7face41a5520 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7facdc00da10 tx=0x7facdc00ded0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.599+0000 7face9192640 1 -- 172.21.15.169:0/1333297623 shutdown_connections 2024-03-28T01:01:58.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.599+0000 7face9192640 1 --2- 172.21.15.169:0/1333297623 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faccc0a2a00 0x7faccc0a4ec0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.599+0000 7face9192640 1 --2- 172.21.15.169:0/1333297623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7face41017b0 0x7face41a5520 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.599+0000 7face9192640 1 --2- 172.21.15.169:0/1333297623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7face4100df0 0x7face419e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.599+0000 7face9192640 1 --2- 172.21.15.169:0/1333297623 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7face40ffc00 0x7face419df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.600+0000 7face9192640 1 -- 172.21.15.169:0/1333297623 >> 172.21.15.169:0/1333297623 conn(0x7face40fb190 msgr2=0x7face41040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:58.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.600+0000 7face9192640 1 -- 172.21.15.169:0/1333297623 shutdown_connections 2024-03-28T01:01:58.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.600+0000 7face9192640 1 -- 172.21.15.169:0/1333297623 wait complete. 2024-03-28T01:01:58.632 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:01:58.632 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T01:01:58.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:58 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1035009080' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:01:58.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:58 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1035009080' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:01:58.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.946+0000 7fc54633c640 1 -- 172.21.15.169:0/2906684731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc540105050 msgr2=0x7fc5401054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:58.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.946+0000 7fc54633c640 1 --2- 172.21.15.169:0/2906684731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc540105050 0x7fc5401054d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc530009860 tx=0x7fc53002bb90 comp rx=0 tx=0).stop 2024-03-28T01:01:58.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.951+0000 7fc54633c640 1 -- 172.21.15.169:0/2906684731 shutdown_connections 2024-03-28T01:01:58.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.951+0000 7fc54633c640 1 --2- 172.21.15.169:0/2906684731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc540105050 0x7fc5401054d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.951+0000 7fc54633c640 1 --2- 172.21.15.169:0/2906684731 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5401026a0 0x7fc540104b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.951+0000 7fc54633c640 1 --2- 172.21.15.169:0/2906684731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5400ffd30 0x7fc540102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.951+0000 7fc54633c640 1 -- 172.21.15.169:0/2906684731 >> 172.21.15.169:0/2906684731 conn(0x7fc5400fb970 msgr2=0x7fc5400fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:58.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.952+0000 7fc54633c640 1 -- 172.21.15.169:0/2906684731 shutdown_connections 2024-03-28T01:01:58.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.954+0000 7fc54633c640 1 -- 172.21.15.169:0/2906684731 wait complete. 2024-03-28T01:01:58.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.955+0000 7fc54633c640 1 Processor -- start 2024-03-28T01:01:58.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.955+0000 7fc54633c640 1 -- start start 2024-03-28T01:01:58.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.956+0000 7fc54633c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5400ffd30 0x7fc540070020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.956+0000 7fc54633c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5401026a0 0x7fc540070560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.956+0000 7fc54633c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc540105050 0x7fc54019cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.956+0000 7fc54633c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5401136c0 con 0x7fc5400ffd30 2024-03-28T01:01:58.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.956+0000 7fc54633c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc540113540 con 0x7fc5401026a0 2024-03-28T01:01:58.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.956+0000 7fc54633c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc540113840 con 0x7fc540105050 2024-03-28T01:01:58.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.957+0000 7fc545b3b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc540105050 0x7fc54019cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:58.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.957+0000 7fc545b3b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc540105050 0x7fc54019cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:55966/0 (socket says 172.21.15.169:55966) 2024-03-28T01:01:58.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.957+0000 7fc545b3b640 1 -- 172.21.15.169:0/805263306 learned_addr learned my addr 172.21.15.169:0/805263306 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:58.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.957+0000 7fc545b3b640 1 -- 172.21.15.169:0/805263306 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5401026a0 msgr2=0x7fc540070560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:01:58.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.957+0000 7fc545b3b640 1 --2- 172.21.15.169:0/805263306 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5401026a0 0x7fc540070560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.957+0000 7fc545b3b640 1 -- 172.21.15.169:0/805263306 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5400ffd30 msgr2=0x7fc540070020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:01:58.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.957+0000 7fc545b3b640 1 --2- 172.21.15.169:0/805263306 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5400ffd30 0x7fc540070020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.957+0000 7fc545b3b640 1 -- 172.21.15.169:0/805263306 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc540070c60 con 0x7fc540105050 2024-03-28T01:01:58.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.959+0000 7fc545b3b640 1 --2- 172.21.15.169:0/805263306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc540105050 0x7fc54019cdc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc530009780 tx=0x7fc530004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:58.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.960+0000 7fc5367fc640 1 -- 172.21.15.169:0/805263306 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc530041020 con 0x7fc540105050 2024-03-28T01:01:58.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.961+0000 7fc54633c640 1 -- 172.21.15.169:0/805263306 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc54019d3c0 con 0x7fc540105050 2024-03-28T01:01:58.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.961+0000 7fc54633c640 1 -- 172.21.15.169:0/805263306 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc54019d900 con 0x7fc540105050 2024-03-28T01:01:58.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.961+0000 7fc5367fc640 1 -- 172.21.15.169:0/805263306 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc5300323d0 con 0x7fc540105050 2024-03-28T01:01:58.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.961+0000 7fc5367fc640 1 -- 172.21.15.169:0/805263306 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc530039040 con 0x7fc540105050 2024-03-28T01:01:58.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.968+0000 7fc54a188740 1 Processor -- start 2024-03-28T01:01:58.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.970+0000 7fc5367fc640 1 -- 172.21.15.169:0/805263306 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc530044020 con 0x7fc540105050 2024-03-28T01:01:58.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.970+0000 7fc54a188740 1 -- start start 2024-03-28T01:01:58.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.970+0000 7fc54a188740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d9137aa8b0 0x55d9137aacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.971+0000 7fc54a188740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9137ab2a0 0x55d9137afda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.971+0000 7fc54a188740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137b0410 0x55d9137b28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.971+0000 7fc54a188740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d91375c610 con 0x55d9137aa8b0 2024-03-28T01:01:58.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.971+0000 7fc54a188740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9136ff3c0 con 0x55d9137b0410 2024-03-28T01:01:58.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.971+0000 7fc54a188740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9136a7d60 con 0x55d9137ab2a0 2024-03-28T01:01:58.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.972+0000 7fc5367fc640 1 --2- 172.21.15.169:0/805263306 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc52c0a28d0 0x7fc52c0a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:58.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.972+0000 7fc544b39640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137b0410 0x55d9137b28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:58.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.972+0000 7fc544b39640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137b0410 0x55d9137b28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49324/0 (socket says 172.21.15.169:49324) 2024-03-28T01:01:58.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.972+0000 7fc544b39640 1 -- 172.21.15.169:0/398668938 learned_addr learned my addr 172.21.15.169:0/398668938 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:58.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.972+0000 7fc544b39640 1 --2- 172.21.15.169:0/398668938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d9137aa8b0 0x55d9137aacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:58.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.972+0000 7fc5367fc640 1 -- 172.21.15.169:0/805263306 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc5300f08c0 con 0x7fc540105050 2024-03-28T01:01:58.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.974+0000 7fc544b39640 1 -- 172.21.15.169:0/398668938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9137ab2a0 msgr2=0x55d9137afda0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:01:58.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.974+0000 7fc544b39640 1 --2- 172.21.15.169:0/398668938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9137ab2a0 0x55d9137afda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.974+0000 7fc544b39640 1 -- 172.21.15.169:0/398668938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137b0410 msgr2=0x55d9137b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:58.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.974+0000 7fc544b39640 1 --2- 172.21.15.169:0/398668938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137b0410 0x55d9137b28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.974+0000 7fc544b39640 1 -- 172.21.15.169:0/398668938 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d9134133f0 con 0x55d9137aa8b0 2024-03-28T01:01:58.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.977+0000 7fc544b39640 1 --2- 172.21.15.169:0/398668938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d9137aa8b0 0x55d9137aacd0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc53c005ab0 tx=0x7fc53c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9c791db735699d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:58.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.982+0000 7fc52b7fe640 1 -- 172.21.15.169:0/398668938 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c005220 con 0x55d9137aa8b0 2024-03-28T01:01:58.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.982+0000 7fc52b7fe640 1 -- 172.21.15.169:0/398668938 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc53c004c80 con 0x55d9137aa8b0 2024-03-28T01:01:58.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.983+0000 7fc54a188740 1 -- 172.21.15.169:0/398668938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d9137aa8b0 msgr2=0x55d9137aacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:58.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.983+0000 7fc54a188740 1 --2- 172.21.15.169:0/398668938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d9137aa8b0 0x55d9137aacd0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc53c005ab0 tx=0x7fc53c0046c0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.983+0000 7fc545b3b640 1 --2- 172.21.15.169:0/805263306 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc52c0a28d0 0x7fc52c0a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:58.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.986+0000 7fc545b3b640 1 --2- 172.21.15.169:0/805263306 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc52c0a28d0 0x7fc52c0a4d90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc5300c5130 tx=0x7fc530005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:58.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.991+0000 7fc54a188740 1 -- 172.21.15.169:0/398668938 shutdown_connections 2024-03-28T01:01:58.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.991+0000 7fc54a188740 1 --2- 172.21.15.169:0/398668938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137b0410 0x55d9137b28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.991+0000 7fc54a188740 1 --2- 172.21.15.169:0/398668938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9137ab2a0 0x55d9137afda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.991+0000 7fc54a188740 1 --2- 172.21.15.169:0/398668938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d9137aa8b0 0x55d9137aacd0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:58.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.991+0000 7fc54a188740 1 -- 172.21.15.169:0/398668938 >> 172.21.15.169:0/398668938 conn(0x55d9137a5e20 msgr2=0x55d9137a6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:58.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.993+0000 7fc54a188740 1 -- 172.21.15.169:0/398668938 shutdown_connections 2024-03-28T01:01:59.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.994+0000 7fc54a188740 1 -- 172.21.15.169:0/398668938 wait complete. 2024-03-28T01:01:59.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.994+0000 7fc54a188740 1 -- start start 2024-03-28T01:01:59.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.994+0000 7fc54a188740 1 Processor -- start 2024-03-28T01:01:59.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.995+0000 7fc54a188740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137aa8b0 0x55d9137b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.995+0000 7fc54a188740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9137ab2a0 0x55d9137b5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.995+0000 7fc54a188740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d9137b0410 0x55d9137bc280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.995+0000 7fc54a188740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9136bb240 con 0x55d9137b0410 2024-03-28T01:01:59.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.995+0000 7fc54a188740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9136fb460 con 0x55d9137aa8b0 2024-03-28T01:01:59.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.995+0000 7fc54a188740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9137b3460 con 0x55d9137ab2a0 2024-03-28T01:01:59.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.995+0000 7fc54a188740 10 client.0 _ll_register_callbacks cb 0x55d9136bac10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:59.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.995+0000 7fc544b39640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137aa8b0 0x55d9137b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.996+0000 7fc544b39640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137aa8b0 0x55d9137b4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49326/0 (socket says 172.21.15.169:49326) 2024-03-28T01:01:59.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.996+0000 7fc544b39640 1 -- 172.21.15.169:0/2593164112 learned_addr learned my addr 172.21.15.169:0/2593164112 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:59.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.996+0000 7fc544b39640 1 -- 172.21.15.169:0/2593164112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9137ab2a0 msgr2=0x55d9137b5240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:01:59.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.996+0000 7fc544b39640 1 --2- 172.21.15.169:0/2593164112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9137ab2a0 0x55d9137b5240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.996+0000 7fc544b39640 1 -- 172.21.15.169:0/2593164112 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d9137b0410 msgr2=0x55d9137bc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:59.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.996+0000 7fc544b39640 1 --2- 172.21.15.169:0/2593164112 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d9137b0410 0x55d9137bc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.996+0000 7fc544b39640 1 -- 172.21.15.169:0/2593164112 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d9134dd7d0 con 0x55d9137aa8b0 2024-03-28T01:01:59.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:58.996+0000 7fc544b39640 1 --2- 172.21.15.169:0/2593164112 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137aa8b0 0x55d9137b4d00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc53c00f760 tx=0x7fc53c00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37e7ec0f56d6e7a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:59.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.001+0000 7fc528ff9640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:01:59.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.001+0000 7fc528ff9640 1 -- 172.21.15.169:0/2593164112 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c004940 con 0x55d9137aa8b0 2024-03-28T01:01:59.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.001+0000 7fc54a188740 1 -- 172.21.15.169:0/2593164112 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc53c004c80 con 0x55d9137aa8b0 2024-03-28T01:01:59.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.001+0000 7fc528ff9640 1 -- 172.21.15.169:0/2593164112 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc53c012070 con 0x55d9137aa8b0 2024-03-28T01:01:59.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.006+0000 7fc528ff9640 1 -- 172.21.15.169:0/2593164112 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc53c02e050 con 0x55d9137aa8b0 2024-03-28T01:01:59.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.006+0000 7fc54a188740 10 client.26378 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:59.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.006+0000 7fc54a188740 1 -- 172.21.15.169:0/2593164112 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d913796090 con 0x55d9137aa8b0 2024-03-28T01:01:59.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.009+0000 7fc528ff9640 1 -- 172.21.15.169:0/2593164112 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc53c004c80 con 0x55d9137aa8b0 2024-03-28T01:01:59.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.011+0000 7fc528ff9640 1 -- 172.21.15.169:0/2593164112 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fc53c01d4b0 con 0x55d9137aa8b0 2024-03-28T01:01:59.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.011+0000 7fc54a188740 10 client.26378 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:59.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.011+0000 7fc54a188740 10 client.26378 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:01:59.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.011+0000 7fc54a188740 20 client.26378 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:59.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.011+0000 7fc54a188740 1 --2- 172.21.15.169:0/2593164112 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55d913797a60 0x55d913799ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.011+0000 7fc54a188740 4 client.26378 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:01:59.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.011+0000 7fc54a188740 1 -- 172.21.15.169:0/2593164112 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d91370c900 con 0x55d913797a60 2024-03-28T01:01:59.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.022+0000 7fc54533a640 1 --2- 172.21.15.169:0/2593164112 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55d913797a60 0x55d913799ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.039+0000 7fc54533a640 1 --2- 172.21.15.169:0/2593164112 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55d913797a60 0x55d913799ec0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4e5c94208690c9df server_cookie=2142243e75aad955 in_seq=0 out_seq=0 2024-03-28T01:01:59.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.041+0000 7fc528ff9640 10 client.26378 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:01:59.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.045+0000 7fc528ff9640 1 -- 172.21.15.169:0/2593164112 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d91370c900 con 0x55d913797a60 2024-03-28T01:01:59.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.045+0000 7fc528ff9640 10 client.26378 handle_command_reply: tid=0 2024-03-28T01:01:59.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.045+0000 7fc54a188740 1 client.26378 shutdown 2024-03-28T01:01:59.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.045+0000 7fc54a188740 1 -- 172.21.15.169:0/2593164112 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137aa8b0 msgr2=0x55d9137b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:59.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.045+0000 7fc54a188740 1 --2- 172.21.15.169:0/2593164112 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137aa8b0 0x55d9137b4d00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc53c00f760 tx=0x7fc53c00f790 comp rx=0 tx=0).stop 2024-03-28T01:01:59.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.046+0000 7fc54a188740 1 -- 172.21.15.169:0/2593164112 shutdown_connections 2024-03-28T01:01:59.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.046+0000 7fc54a188740 1 --2- 172.21.15.169:0/2593164112 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55d913797a60 0x55d913799ec0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.046+0000 7fc54a188740 1 --2- 172.21.15.169:0/2593164112 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d9137b0410 0x55d9137bc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.046+0000 7fc54a188740 1 --2- 172.21.15.169:0/2593164112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9137ab2a0 0x55d9137b5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.046+0000 7fc54a188740 1 --2- 172.21.15.169:0/2593164112 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137aa8b0 0x55d9137b4d00 secure :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc53c00f760 tx=0x7fc53c00f790 comp rx=0 tx=0).stop 2024-03-28T01:01:59.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.046+0000 7fc54a188740 1 -- 172.21.15.169:0/2593164112 >> 172.21.15.169:0/2593164112 conn(0x55d9137a5e20 msgr2=0x55d913794070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:59.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.046+0000 7fc528ff9640 0 client.26378 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:01:59.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.046+0000 7fc54a188740 1 -- 172.21.15.169:0/2593164112 shutdown_connections 2024-03-28T01:01:59.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.048+0000 7fc54a188740 1 -- 172.21.15.169:0/2593164112 wait complete. 2024-03-28T01:01:59.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.048+0000 7fc54a188740 20 client.26378 trim_cache size 0 max 16384 2024-03-28T01:01:59.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.082+0000 7fc54a188740 1 Processor -- start 2024-03-28T01:01:59.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.082+0000 7fc54a188740 1 -- start start 2024-03-28T01:01:59.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.082+0000 7fc54a188740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d9137ab2a0 0x55d9137b4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.083+0000 7fc54a188740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137b0410 0x55d9137bc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.083+0000 7fc54a188740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9137ace10 0x55d9137b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.083+0000 7fc54a188740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc524011f90 con 0x55d9137ab2a0 2024-03-28T01:01:59.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.083+0000 7fc54a188740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52400fa90 con 0x55d9137b0410 2024-03-28T01:01:59.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.083+0000 7fc54a188740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52400dbe0 con 0x55d9137ace10 2024-03-28T01:01:59.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.083+0000 7fc54533a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137b0410 0x55d9137bc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.083+0000 7fc54533a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137b0410 0x55d9137bc240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49336/0 (socket says 172.21.15.169:49336) 2024-03-28T01:01:59.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.083+0000 7fc54533a640 1 -- 172.21.15.169:0/87399762 learned_addr learned my addr 172.21.15.169:0/87399762 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:59.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.084+0000 7fc544b39640 1 --2- 172.21.15.169:0/87399762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9137ace10 0x55d9137b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.084+0000 7fc544b39640 1 --2- 172.21.15.169:0/87399762 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d9137ab2a0 0x55d9137b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.084+0000 7fc544b39640 1 -- 172.21.15.169:0/87399762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9137ace10 msgr2=0x55d9137b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:59.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.084+0000 7fc544b39640 1 --2- 172.21.15.169:0/87399762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9137ace10 0x55d9137b4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.084+0000 7fc544b39640 1 -- 172.21.15.169:0/87399762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137b0410 msgr2=0x55d9137bc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:59.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.084+0000 7fc544b39640 1 --2- 172.21.15.169:0/87399762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137b0410 0x55d9137bc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.084+0000 7fc544b39640 1 -- 172.21.15.169:0/87399762 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d913790870 con 0x55d9137ab2a0 2024-03-28T01:01:59.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.085+0000 7fc544b39640 1 --2- 172.21.15.169:0/87399762 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d9137ab2a0 0x55d9137b4910 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc53c013000 tx=0x7fc53c005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d399cab8dd3bdc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:59.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.085+0000 7fc52affd640 1 -- 172.21.15.169:0/87399762 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c00f800 con 0x55d9137ab2a0 2024-03-28T01:01:59.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.086+0000 7fc52affd640 1 -- 172.21.15.169:0/87399762 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc53c01d440 con 0x55d9137ab2a0 2024-03-28T01:01:59.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.086+0000 7fc52affd640 1 -- 172.21.15.169:0/87399762 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c033780 con 0x55d9137ab2a0 2024-03-28T01:01:59.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.088+0000 7fc54a188740 1 -- 172.21.15.169:0/87399762 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d9137ab2a0 msgr2=0x55d9137b4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:59.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.088+0000 7fc54a188740 1 --2- 172.21.15.169:0/87399762 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d9137ab2a0 0x55d9137b4910 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc53c013000 tx=0x7fc53c005a00 comp rx=0 tx=0).stop 2024-03-28T01:01:59.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.088+0000 7fc54a188740 1 -- 172.21.15.169:0/87399762 shutdown_connections 2024-03-28T01:01:59.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.088+0000 7fc54a188740 1 --2- 172.21.15.169:0/87399762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9137ace10 0x55d9137b4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.088+0000 7fc54a188740 1 --2- 172.21.15.169:0/87399762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137b0410 0x55d9137bc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.088+0000 7fc54a188740 1 --2- 172.21.15.169:0/87399762 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d9137ab2a0 0x55d9137b4910 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.088+0000 7fc54a188740 1 -- 172.21.15.169:0/87399762 >> 172.21.15.169:0/87399762 conn(0x55d9137a5e20 msgr2=0x55d9137ac700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:59.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.088+0000 7fc54a188740 1 -- 172.21.15.169:0/87399762 shutdown_connections 2024-03-28T01:01:59.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.089+0000 7fc54a188740 1 -- 172.21.15.169:0/87399762 wait complete. 2024-03-28T01:01:59.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.089+0000 7fc54a188740 1 -- start start 2024-03-28T01:01:59.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.089+0000 7fc54a188740 1 Processor -- start 2024-03-28T01:01:59.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.090+0000 7fc54a188740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137ab2a0 0x55d9137b9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.090+0000 7fc54a188740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d9137b0410 0x55d9137b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.090+0000 7fc54a188740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d913829fe0 0x55d91382c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.090+0000 7fc54a188740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52400e150 con 0x55d9137b0410 2024-03-28T01:01:59.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.090+0000 7fc54a188740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52400f430 con 0x55d9137ab2a0 2024-03-28T01:01:59.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.090+0000 7fc54a188740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52400e770 con 0x55d913829fe0 2024-03-28T01:01:59.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.090+0000 7fc54a188740 10 client.0 _ll_register_callbacks cb 0x7fc53c00db70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:59.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.090+0000 7fc544b39640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d913829fe0 0x55d91382c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.090+0000 7fc544b39640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d913829fe0 0x55d91382c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56004/0 (socket says 172.21.15.169:56004) 2024-03-28T01:01:59.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.090+0000 7fc544b39640 1 -- 172.21.15.169:0/469126073 learned_addr learned my addr 172.21.15.169:0/469126073 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:59.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.090+0000 7fc544b39640 1 --2- 172.21.15.169:0/469126073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137ab2a0 0x55d9137b9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.091+0000 7fc54533a640 1 --2- 172.21.15.169:0/469126073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d9137b0410 0x55d9137b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.091+0000 7fc544b39640 1 -- 172.21.15.169:0/469126073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137ab2a0 msgr2=0x55d9137b9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:59.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.091+0000 7fc544b39640 1 --2- 172.21.15.169:0/469126073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137ab2a0 0x55d9137b9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.091+0000 7fc544b39640 1 -- 172.21.15.169:0/469126073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d9137b0410 msgr2=0x55d9137b95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:59.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.091+0000 7fc544b39640 1 --2- 172.21.15.169:0/469126073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d9137b0410 0x55d9137b95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.091+0000 7fc544b39640 1 -- 172.21.15.169:0/469126073 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d91382c910 con 0x55d913829fe0 2024-03-28T01:01:59.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.092+0000 7fc544b39640 1 --2- 172.21.15.169:0/469126073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d913829fe0 0x55d91382c3d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc53c01dde0 tx=0x7fc53c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b34dcdf6a4e571dd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:59.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.092+0000 7fc528ff9640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:01:59.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.093+0000 7fc528ff9640 1 -- 172.21.15.169:0/469126073 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c00d660 con 0x55d913829fe0 2024-03-28T01:01:59.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.093+0000 7fc528ff9640 1 -- 172.21.15.169:0/469126073 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc53c012040 con 0x55d913829fe0 2024-03-28T01:01:59.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.093+0000 7fc54a188740 1 -- 172.21.15.169:0/469126073 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc53c01d440 con 0x55d913829fe0 2024-03-28T01:01:59.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.093+0000 7fc528ff9640 1 -- 172.21.15.169:0/469126073 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc53c012240 con 0x55d913829fe0 2024-03-28T01:01:59.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.093+0000 7fc54a188740 10 client.26323 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:59.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.093+0000 7fc54a188740 1 -- 172.21.15.169:0/469126073 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d91382db50 con 0x55d913829fe0 2024-03-28T01:01:59.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.094+0000 7fc528ff9640 1 -- 172.21.15.169:0/469126073 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc53c01d440 con 0x55d913829fe0 2024-03-28T01:01:59.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.094+0000 7fc528ff9640 1 -- 172.21.15.169:0/469126073 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fc53c042480 con 0x55d913829fe0 2024-03-28T01:01:59.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.095+0000 7fc54a188740 10 client.26323 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:59.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.095+0000 7fc54a188740 10 client.26323 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:01:59.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.095+0000 7fc54a188740 20 client.26323 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:59.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.095+0000 7fc54a188740 1 --2- 172.21.15.169:0/469126073 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55d91382e430 0x55d913830850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.095+0000 7fc54a188740 4 client.26323 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:01:59.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.095+0000 7fc54a188740 1 -- 172.21.15.169:0/469126073 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d913706c50 con 0x55d91382e430 2024-03-28T01:01:59.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.095+0000 7fc54533a640 1 --2- 172.21.15.169:0/469126073 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55d91382e430 0x55d913830850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.110+0000 7fc54533a640 1 --2- 172.21.15.169:0/469126073 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55d91382e430 0x55d913830850 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=de708e9be58681c0 server_cookie=395620efa5c4aaa in_seq=0 out_seq=0 2024-03-28T01:01:59.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.110+0000 7fc528ff9640 10 client.26323 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:01:59.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.110+0000 7fc528ff9640 1 -- 172.21.15.169:0/469126073 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d913706c50 con 0x55d91382e430 2024-03-28T01:01:59.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.110+0000 7fc528ff9640 10 client.26323 handle_command_reply: tid=0 2024-03-28T01:01:59.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.110+0000 7fc54a188740 1 client.26323 shutdown 2024-03-28T01:01:59.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.117+0000 7fc54a188740 1 -- 172.21.15.169:0/469126073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d913829fe0 msgr2=0x55d91382c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:59.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.117+0000 7fc54a188740 1 --2- 172.21.15.169:0/469126073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d913829fe0 0x55d91382c3d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc53c01dde0 tx=0x7fc53c000bc0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.118+0000 7fc54a188740 1 -- 172.21.15.169:0/469126073 shutdown_connections 2024-03-28T01:01:59.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.118+0000 7fc54a188740 1 --2- 172.21.15.169:0/469126073 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55d91382e430 0x55d913830850 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.118+0000 7fc54a188740 1 --2- 172.21.15.169:0/469126073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d913829fe0 0x55d91382c3d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.118+0000 7fc54a188740 1 --2- 172.21.15.169:0/469126073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d9137b0410 0x55d9137b95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.118+0000 7fc54a188740 1 --2- 172.21.15.169:0/469126073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9137ab2a0 0x55d9137b9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.118+0000 7fc54a188740 1 -- 172.21.15.169:0/469126073 >> 172.21.15.169:0/469126073 conn(0x55d9137a5e20 msgr2=0x55d9137b4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:59.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.118+0000 7fc528ff9640 0 client.26323 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:01:59.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.119+0000 7fc54a188740 1 -- 172.21.15.169:0/469126073 shutdown_connections 2024-03-28T01:01:59.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.119+0000 7fc54a188740 1 -- 172.21.15.169:0/469126073 wait complete. 2024-03-28T01:01:59.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.123+0000 7fc54a188740 20 client.26323 trim_cache size 0 max 16384 2024-03-28T01:01:59.128 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:01:59.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.134+0000 7fc54633c640 1 -- 172.21.15.169:0/805263306 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc52c0a28d0 msgr2=0x7fc52c0a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:59.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.134+0000 7fc54633c640 1 --2- 172.21.15.169:0/805263306 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc52c0a28d0 0x7fc52c0a4d90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc5300c5130 tx=0x7fc530005f90 comp rx=0 tx=0).stop 2024-03-28T01:01:59.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.134+0000 7fc54633c640 1 -- 172.21.15.169:0/805263306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc540105050 msgr2=0x7fc54019cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:59.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.134+0000 7fc54633c640 1 --2- 172.21.15.169:0/805263306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc540105050 0x7fc54019cdc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc530009780 tx=0x7fc530004270 comp rx=0 tx=0).stop 2024-03-28T01:01:59.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.147+0000 7fc54633c640 1 -- 172.21.15.169:0/805263306 shutdown_connections 2024-03-28T01:01:59.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.147+0000 7fc54633c640 1 --2- 172.21.15.169:0/805263306 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc52c0a28d0 0x7fc52c0a4d90 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.147+0000 7fc54633c640 1 --2- 172.21.15.169:0/805263306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc540105050 0x7fc54019cdc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.147+0000 7fc54633c640 1 --2- 172.21.15.169:0/805263306 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5401026a0 0x7fc540070560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.147+0000 7fc54633c640 1 --2- 172.21.15.169:0/805263306 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5400ffd30 0x7fc540070020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.147+0000 7fc54633c640 1 -- 172.21.15.169:0/805263306 >> 172.21.15.169:0/805263306 conn(0x7fc5400fb970 msgr2=0x7fc5401009e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:59.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.147+0000 7fc54633c640 1 -- 172.21.15.169:0/805263306 shutdown_connections 2024-03-28T01:01:59.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.147+0000 7fc54633c640 1 -- 172.21.15.169:0/805263306 wait complete. 2024-03-28T01:01:59.206 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:01:59.206 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T01:01:59.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.520+0000 7fc56e332640 1 -- 172.21.15.169:0/3101888529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc568101510 msgr2=0x7fc568101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:59.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.521+0000 7fc56e332640 1 --2- 172.21.15.169:0/3101888529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc568101510 0x7fc568101930 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc558009900 tx=0x7fc55802bc40 comp rx=0 tx=0).stop 2024-03-28T01:01:59.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.522+0000 7fc56e332640 1 -- 172.21.15.169:0/3101888529 shutdown_connections 2024-03-28T01:01:59.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.522+0000 7fc56e332640 1 --2- 172.21.15.169:0/3101888529 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5681030c0 0x7fc568109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.522+0000 7fc56e332640 1 --2- 172.21.15.169:0/3101888529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc568102700 0x7fc568102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.522+0000 7fc56e332640 1 --2- 172.21.15.169:0/3101888529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc568101510 0x7fc568101930 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.522+0000 7fc56e332640 1 -- 172.21.15.169:0/3101888529 >> 172.21.15.169:0/3101888529 conn(0x7fc5680fca80 msgr2=0x7fc5680feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:59.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.522+0000 7fc56e332640 1 -- 172.21.15.169:0/3101888529 shutdown_connections 2024-03-28T01:01:59.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.522+0000 7fc56e332640 1 -- 172.21.15.169:0/3101888529 wait complete. 2024-03-28T01:01:59.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.523+0000 7fc56e332640 1 Processor -- start 2024-03-28T01:01:59.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.524+0000 7fc56e332640 1 -- start start 2024-03-28T01:01:59.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.525+0000 7fc56e332640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc568101510 0x7fc568197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.525+0000 7fc56e332640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc568102700 0x7fc568197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.526+0000 7fc56e332640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5681030c0 0x7fc56819ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.526+0000 7fc5677fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc568102700 0x7fc568197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.526+0000 7fc5677fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc568102700 0x7fc568197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49374/0 (socket says 172.21.15.169:49374) 2024-03-28T01:01:59.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.526+0000 7fc567fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc568101510 0x7fc568197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.526+0000 7fc567fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc568101510 0x7fc568197620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56034/0 (socket says 172.21.15.169:56034) 2024-03-28T01:01:59.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.526+0000 7fc5677fe640 1 -- 172.21.15.169:0/2874042503 learned_addr learned my addr 172.21.15.169:0/2874042503 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:59.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.526+0000 7fc56e332640 1 -- 172.21.15.169:0/2874042503 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56810d1f0 con 0x7fc5681030c0 2024-03-28T01:01:59.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.526+0000 7fc56e332640 1 -- 172.21.15.169:0/2874042503 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56810d070 con 0x7fc568102700 2024-03-28T01:01:59.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.526+0000 7fc56e332640 1 -- 172.21.15.169:0/2874042503 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56810d370 con 0x7fc568101510 2024-03-28T01:01:59.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.527+0000 7fc5677fe640 1 -- 172.21.15.169:0/2874042503 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc568101510 msgr2=0x7fc568197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:59.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.527+0000 7fc5677fe640 1 --2- 172.21.15.169:0/2874042503 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc568101510 0x7fc568197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.527+0000 7fc5677fe640 1 -- 172.21.15.169:0/2874042503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5681030c0 msgr2=0x7fc56819ebd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:01:59.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.527+0000 7fc5677fe640 1 --2- 172.21.15.169:0/2874042503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5681030c0 0x7fc56819ebd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.527+0000 7fc5677fe640 1 -- 172.21.15.169:0/2874042503 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56819f240 con 0x7fc568102700 2024-03-28T01:01:59.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.528+0000 7fc5677fe640 1 --2- 172.21.15.169:0/2874042503 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc568102700 0x7fc568197b60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc55c00d950 tx=0x7fc55c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:59.534 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.531+0000 7fc5657fa640 1 -- 172.21.15.169:0/2874042503 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c017030 con 0x7fc568102700 2024-03-28T01:01:59.534 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.531+0000 7fc5657fa640 1 -- 172.21.15.169:0/2874042503 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc55c004020 con 0x7fc568102700 2024-03-28T01:01:59.534 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.531+0000 7fc5657fa640 1 -- 172.21.15.169:0/2874042503 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c003d60 con 0x7fc568102700 2024-03-28T01:01:59.534 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.531+0000 7fc56e332640 1 -- 172.21.15.169:0/2874042503 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc56819f530 con 0x7fc568102700 2024-03-28T01:01:59.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.531+0000 7fc56e332640 1 -- 172.21.15.169:0/2874042503 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc56819fa30 con 0x7fc568102700 2024-03-28T01:01:59.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.536+0000 7fc570ff5740 1 Processor -- start 2024-03-28T01:01:59.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.536+0000 7fc570ff5740 1 -- start start 2024-03-28T01:01:59.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.537+0000 7fc570ff5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae1860 0x55d07bae1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.537+0000 7fc570ff5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d07bae2250 0x55d07bae6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.537+0000 7fc570ff5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d07bae73c0 0x55d07bae9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.537+0000 7fc570ff5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d07baa4610 con 0x55d07bae2250 2024-03-28T01:01:59.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.537+0000 7fc570ff5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d07ba473c0 con 0x55d07bae1860 2024-03-28T01:01:59.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.537+0000 7fc570ff5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d07b9efd60 con 0x55d07bae73c0 2024-03-28T01:01:59.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.537+0000 7fc56c8cf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae1860 0x55d07bae1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.537+0000 7fc56c8cf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae1860 0x55d07bae1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49388/0 (socket says 172.21.15.169:49388) 2024-03-28T01:01:59.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.537+0000 7fc56c8cf640 1 -- 172.21.15.169:0/3105486265 learned_addr learned my addr 172.21.15.169:0/3105486265 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:59.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.538+0000 7fc567fff640 1 --2- 172.21.15.169:0/3105486265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d07bae2250 0x55d07bae6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.539+0000 7fc56c8cf640 1 -- 172.21.15.169:0/3105486265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d07bae73c0 msgr2=0x55d07bae9850 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:01:59.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.539+0000 7fc56c8cf640 1 --2- 172.21.15.169:0/3105486265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d07bae73c0 0x55d07bae9850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.539+0000 7fc56c8cf640 1 -- 172.21.15.169:0/3105486265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d07bae2250 msgr2=0x55d07bae6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:59.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.539+0000 7fc56c8cf640 1 --2- 172.21.15.169:0/3105486265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d07bae2250 0x55d07bae6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.539+0000 7fc56c8cf640 1 -- 172.21.15.169:0/3105486265 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d07b75b3f0 con 0x55d07bae1860 2024-03-28T01:01:59.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.539+0000 7fc5657fa640 1 -- 172.21.15.169:0/2874042503 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc55c01a070 con 0x7fc568102700 2024-03-28T01:01:59.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.539+0000 7fc56c8cf640 1 --2- 172.21.15.169:0/3105486265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae1860 0x55d07bae1c80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc560009db0 tx=0x7fc56000e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c224371f26b02397 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:59.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.539+0000 7fc54e7fc640 1 -- 172.21.15.169:0/3105486265 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc560004b20 con 0x55d07bae1860 2024-03-28T01:01:59.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.539+0000 7fc54e7fc640 1 -- 172.21.15.169:0/3105486265 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc560004cc0 con 0x55d07bae1860 2024-03-28T01:01:59.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.539+0000 7fc54e7fc640 1 -- 172.21.15.169:0/3105486265 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc560018930 con 0x55d07bae1860 2024-03-28T01:01:59.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.541+0000 7fc570ff5740 1 -- 172.21.15.169:0/3105486265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae1860 msgr2=0x55d07bae1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:59.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.541+0000 7fc570ff5740 1 --2- 172.21.15.169:0/3105486265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae1860 0x55d07bae1c80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc560009db0 tx=0x7fc56000e720 comp rx=0 tx=0).stop 2024-03-28T01:01:59.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.541+0000 7fc570ff5740 1 -- 172.21.15.169:0/3105486265 shutdown_connections 2024-03-28T01:01:59.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.541+0000 7fc570ff5740 1 --2- 172.21.15.169:0/3105486265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d07bae73c0 0x55d07bae9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.541+0000 7fc570ff5740 1 --2- 172.21.15.169:0/3105486265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d07bae2250 0x55d07bae6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.541+0000 7fc570ff5740 1 --2- 172.21.15.169:0/3105486265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae1860 0x55d07bae1c80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.541+0000 7fc570ff5740 1 -- 172.21.15.169:0/3105486265 >> 172.21.15.169:0/3105486265 conn(0x55d07badcdd0 msgr2=0x55d07badd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:59.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.541+0000 7fc5657fa640 1 --2- 172.21.15.169:0/2874042503 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc5500a6e10 0x7fc5500a92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.541+0000 7fc5657fa640 1 -- 172.21.15.169:0/2874042503 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc55c0cded0 con 0x7fc568102700 2024-03-28T01:01:59.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.541+0000 7fc567fff640 1 --2- 172.21.15.169:0/2874042503 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc5500a6e10 0x7fc5500a92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.542+0000 7fc570ff5740 1 -- 172.21.15.169:0/3105486265 shutdown_connections 2024-03-28T01:01:59.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.542+0000 7fc570ff5740 1 -- 172.21.15.169:0/3105486265 wait complete. 2024-03-28T01:01:59.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.542+0000 7fc570ff5740 1 -- start start 2024-03-28T01:01:59.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.543+0000 7fc570ff5740 1 Processor -- start 2024-03-28T01:01:59.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.543+0000 7fc570ff5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae1860 0x55d07baf4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.543+0000 7fc570ff5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d07bae2250 0x55d07baf49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.543+0000 7fc570ff5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d07bae73c0 0x55d07bafba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.543+0000 7fc570ff5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d07ba03240 con 0x55d07bae2250 2024-03-28T01:01:59.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.543+0000 7fc570ff5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d07ba43460 con 0x55d07bae1860 2024-03-28T01:01:59.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.543+0000 7fc570ff5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d07baea370 con 0x55d07bae73c0 2024-03-28T01:01:59.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.543+0000 7fc570ff5740 10 client.0 _ll_register_callbacks cb 0x55d07ba02c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:59.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.544+0000 7fc567fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae1860 0x55d07baf4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.544+0000 7fc567fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae1860 0x55d07baf4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49396/0 (socket says 172.21.15.169:49396) 2024-03-28T01:01:59.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.544+0000 7fc567fff640 1 -- 172.21.15.169:0/62024134 learned_addr learned my addr 172.21.15.169:0/62024134 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:59.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.544+0000 7fc567fff640 1 -- 172.21.15.169:0/62024134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d07bae73c0 msgr2=0x55d07bafba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:59.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.545+0000 7fc567fff640 1 --2- 172.21.15.169:0/62024134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d07bae73c0 0x55d07bafba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.545+0000 7fc567fff640 1 -- 172.21.15.169:0/62024134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d07bae2250 msgr2=0x55d07baf49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:59.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.545+0000 7fc567fff640 1 --2- 172.21.15.169:0/62024134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d07bae2250 0x55d07baf49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.545+0000 7fc567fff640 1 -- 172.21.15.169:0/62024134 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d07b8257d0 con 0x55d07bae1860 2024-03-28T01:01:59.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.545+0000 7fc567fff640 1 --2- 172.21.15.169:0/62024134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae1860 0x55d07baf4490 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc558009900 tx=0x7fc558004d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfc9ee7b456af41e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:59.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.546+0000 7fc5477fe640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:01:59.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.546+0000 7fc567fff640 1 --2- 172.21.15.169:0/2874042503 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc5500a6e10 0x7fc5500a92d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc5580096c0 tx=0x7fc558040000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:59.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.546+0000 7fc5477fe640 1 -- 172.21.15.169:0/62024134 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55802cb50 con 0x55d07bae1860 2024-03-28T01:01:59.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.547+0000 7fc570ff5740 1 -- 172.21.15.169:0/62024134 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc560004cc0 con 0x55d07bae1860 2024-03-28T01:01:59.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.547+0000 7fc5477fe640 1 -- 172.21.15.169:0/62024134 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc5580035f0 con 0x55d07bae1860 2024-03-28T01:01:59.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.547+0000 7fc5477fe640 1 -- 172.21.15.169:0/62024134 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc558059020 con 0x55d07bae1860 2024-03-28T01:01:59.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.547+0000 7fc570ff5740 10 client.26396 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:59.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.547+0000 7fc570ff5740 1 -- 172.21.15.169:0/62024134 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d07bb0fe00 con 0x55d07bae1860 2024-03-28T01:01:59.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.548+0000 7fc5477fe640 1 -- 172.21.15.169:0/62024134 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc560004cc0 con 0x55d07bae1860 2024-03-28T01:01:59.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.549+0000 7fc5477fe640 1 -- 172.21.15.169:0/62024134 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fc55804c030 con 0x55d07bae1860 2024-03-28T01:01:59.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.550+0000 7fc570ff5740 10 client.26396 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:59.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.550+0000 7fc570ff5740 10 client.26396 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:01:59.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.550+0000 7fc570ff5740 20 client.26396 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:59.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.550+0000 7fc570ff5740 1 --2- 172.21.15.169:0/62024134 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d07bb11480 0x55d07bb13920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.550+0000 7fc570ff5740 4 client.26396 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:01:59.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.550+0000 7fc570ff5740 1 -- 172.21.15.169:0/62024134 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d07ba54900 con 0x55d07bb11480 2024-03-28T01:01:59.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.555+0000 7fc5677fe640 1 --2- 172.21.15.169:0/62024134 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d07bb11480 0x55d07bb13920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.558+0000 7fc5677fe640 1 --2- 172.21.15.169:0/62024134 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d07bb11480 0x55d07bb13920 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=faca24a9ef356a86 server_cookie=5cb2411e02da925a in_seq=0 out_seq=0 2024-03-28T01:01:59.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.559+0000 7fc5477fe640 10 client.26396 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:59.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.565+0000 7fc5477fe640 1 -- 172.21.15.169:0/62024134 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d07ba54900 con 0x55d07bb11480 2024-03-28T01:01:59.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.565+0000 7fc5477fe640 10 client.26396 handle_command_reply: tid=0 2024-03-28T01:01:59.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.565+0000 7fc570ff5740 1 client.26396 shutdown 2024-03-28T01:01:59.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.565+0000 7fc570ff5740 1 -- 172.21.15.169:0/62024134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae1860 msgr2=0x55d07baf4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:59.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.565+0000 7fc570ff5740 1 --2- 172.21.15.169:0/62024134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae1860 0x55d07baf4490 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc558009900 tx=0x7fc558004d50 comp rx=0 tx=0).stop 2024-03-28T01:01:59.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.566+0000 7fc570ff5740 1 -- 172.21.15.169:0/62024134 shutdown_connections 2024-03-28T01:01:59.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.566+0000 7fc570ff5740 1 --2- 172.21.15.169:0/62024134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d07bae73c0 0x55d07bafba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.566+0000 7fc570ff5740 1 --2- 172.21.15.169:0/62024134 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d07bb11480 0x55d07bb13920 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.566+0000 7fc570ff5740 1 --2- 172.21.15.169:0/62024134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d07bae2250 0x55d07baf49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.566+0000 7fc570ff5740 1 --2- 172.21.15.169:0/62024134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae1860 0x55d07baf4490 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.566+0000 7fc570ff5740 1 -- 172.21.15.169:0/62024134 >> 172.21.15.169:0/62024134 conn(0x55d07badcdd0 msgr2=0x55d07badc070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:59.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.566+0000 7fc5477fe640 0 client.26396 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:59.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.566+0000 7fc570ff5740 1 -- 172.21.15.169:0/62024134 shutdown_connections 2024-03-28T01:01:59.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.566+0000 7fc570ff5740 1 -- 172.21.15.169:0/62024134 wait complete. 2024-03-28T01:01:59.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.566+0000 7fc570ff5740 20 client.26396 trim_cache size 0 max 16384 2024-03-28T01:01:59.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.584+0000 7fc570ff5740 1 Processor -- start 2024-03-28T01:01:59.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.585+0000 7fc570ff5740 1 -- start start 2024-03-28T01:01:59.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.585+0000 7fc570ff5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d07bae1860 0x55d07bb13490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.585+0000 7fc570ff5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae2250 0x55d07bafb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.585+0000 7fc570ff5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d07bae73c0 0x55d07baf47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.585+0000 7fc570ff5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5681a1880 con 0x55d07bae1860 2024-03-28T01:01:59.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.585+0000 7fc570ff5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc568075190 con 0x55d07bae2250 2024-03-28T01:01:59.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.585+0000 7fc570ff5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5680732e0 con 0x55d07bae73c0 2024-03-28T01:01:59.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.585+0000 7fc567fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d07bae1860 0x55d07bb13490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.585+0000 7fc567fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d07bae1860 0x55d07bb13490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:54708/0 (socket says 172.21.15.169:54708) 2024-03-28T01:01:59.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.585+0000 7fc567fff640 1 -- 172.21.15.169:0/2049751706 learned_addr learned my addr 172.21.15.169:0/2049751706 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:59.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.585+0000 7fc56c8cf640 1 --2- 172.21.15.169:0/2049751706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d07bae73c0 0x55d07baf47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.585+0000 7fc5677fe640 1 --2- 172.21.15.169:0/2049751706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae2250 0x55d07bafb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.586+0000 7fc567fff640 1 -- 172.21.15.169:0/2049751706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d07bae73c0 msgr2=0x55d07baf47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:59.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.586+0000 7fc567fff640 1 --2- 172.21.15.169:0/2049751706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d07bae73c0 0x55d07baf47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.586+0000 7fc567fff640 1 -- 172.21.15.169:0/2049751706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae2250 msgr2=0x55d07bafb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:59.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.586+0000 7fc567fff640 1 --2- 172.21.15.169:0/2049751706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae2250 0x55d07bafb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.586+0000 7fc567fff640 1 -- 172.21.15.169:0/2049751706 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d07baea4b0 con 0x55d07bae1860 2024-03-28T01:01:59.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.586+0000 7fc5677fe640 1 --2- 172.21.15.169:0/2049751706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae2250 0x55d07bafb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:01:59.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.586+0000 7fc567fff640 1 --2- 172.21.15.169:0/2049751706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d07bae1860 0x55d07bb13490 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc5580095d0 tx=0x7fc55802bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12fd787cb1ecde9e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:59.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.586+0000 7fc54dffb640 1 -- 172.21.15.169:0/2049751706 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558005550 con 0x55d07bae1860 2024-03-28T01:01:59.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.586+0000 7fc54dffb640 1 -- 172.21.15.169:0/2049751706 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc558034440 con 0x55d07bae1860 2024-03-28T01:01:59.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.586+0000 7fc54dffb640 1 -- 172.21.15.169:0/2049751706 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558003cd0 con 0x55d07bae1860 2024-03-28T01:01:59.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.587+0000 7fc570ff5740 1 -- 172.21.15.169:0/2049751706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d07bae1860 msgr2=0x55d07bb13490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:59.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.587+0000 7fc570ff5740 1 --2- 172.21.15.169:0/2049751706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d07bae1860 0x55d07bb13490 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc5580095d0 tx=0x7fc55802bc40 comp rx=0 tx=0).stop 2024-03-28T01:01:59.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.587+0000 7fc570ff5740 1 -- 172.21.15.169:0/2049751706 shutdown_connections 2024-03-28T01:01:59.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.587+0000 7fc570ff5740 1 --2- 172.21.15.169:0/2049751706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d07bae73c0 0x55d07baf47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.587+0000 7fc570ff5740 1 --2- 172.21.15.169:0/2049751706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae2250 0x55d07bafb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.587+0000 7fc570ff5740 1 --2- 172.21.15.169:0/2049751706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d07bae1860 0x55d07bb13490 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.588+0000 7fc570ff5740 1 -- 172.21.15.169:0/2049751706 >> 172.21.15.169:0/2049751706 conn(0x55d07badcdd0 msgr2=0x55d07baf40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:59.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.588+0000 7fc570ff5740 1 -- 172.21.15.169:0/2049751706 shutdown_connections 2024-03-28T01:01:59.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.588+0000 7fc570ff5740 1 -- 172.21.15.169:0/2049751706 wait complete. 2024-03-28T01:01:59.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.588+0000 7fc570ff5740 1 -- start start 2024-03-28T01:01:59.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.589+0000 7fc570ff5740 1 Processor -- start 2024-03-28T01:01:59.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.589+0000 7fc570ff5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d07bae1860 0x55d07baf95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.590+0000 7fc570ff5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d07bae2250 0x55d07baf9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.590+0000 7fc570ff5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae73c0 0x55d07bafaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.590+0000 7fc5677fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d07bae2250 0x55d07baf9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.590+0000 7fc5677fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d07bae2250 0x55d07baf9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56064/0 (socket says 172.21.15.169:56064) 2024-03-28T01:01:59.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.590+0000 7fc567fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d07bae1860 0x55d07baf95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.590+0000 7fc5677fe640 1 -- 172.21.15.169:0/4171941762 learned_addr learned my addr 172.21.15.169:0/4171941762 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:01:59.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.590+0000 7fc570ff5740 1 -- 172.21.15.169:0/4171941762 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc568073850 con 0x55d07bae1860 2024-03-28T01:01:59.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.590+0000 7fc570ff5740 1 -- 172.21.15.169:0/4171941762 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc568074b30 con 0x55d07bae73c0 2024-03-28T01:01:59.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.590+0000 7fc570ff5740 1 -- 172.21.15.169:0/4171941762 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc568073e70 con 0x55d07bae2250 2024-03-28T01:01:59.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.590+0000 7fc56c8cf640 1 --2- 172.21.15.169:0/4171941762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae73c0 0x55d07bafaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.590+0000 7fc570ff5740 10 client.0 _ll_register_callbacks cb 0x7fc5580048d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:01:59.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.590+0000 7fc5677fe640 1 -- 172.21.15.169:0/4171941762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae73c0 msgr2=0x55d07bafaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:59.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.590+0000 7fc5677fe640 1 --2- 172.21.15.169:0/4171941762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae73c0 0x55d07bafaa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.590+0000 7fc5677fe640 1 -- 172.21.15.169:0/4171941762 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d07bae1860 msgr2=0x55d07baf95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:01:59.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.590+0000 7fc5677fe640 1 --2- 172.21.15.169:0/4171941762 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d07bae1860 0x55d07baf95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.590+0000 7fc5677fe640 1 -- 172.21.15.169:0/4171941762 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d07bafb020 con 0x55d07bae2250 2024-03-28T01:01:59.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.590+0000 7fc5677fe640 1 --2- 172.21.15.169:0/4171941762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d07bae2250 0x55d07baf9ae0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc55c096f50 tx=0x7fc55c09a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1630c243dd814c12 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:01:59.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.590+0000 7fc5477fe640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:01:59.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.591+0000 7fc5477fe640 1 -- 172.21.15.169:0/4171941762 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c09e6d0 con 0x55d07bae2250 2024-03-28T01:01:59.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.591+0000 7fc5477fe640 1 -- 172.21.15.169:0/4171941762 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc55c096bf0 con 0x55d07bae2250 2024-03-28T01:01:59.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.591+0000 7fc570ff5740 1 -- 172.21.15.169:0/4171941762 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc558034440 con 0x55d07bae2250 2024-03-28T01:01:59.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.591+0000 7fc5477fe640 1 -- 172.21.15.169:0/4171941762 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc55c0a2c90 con 0x55d07bae2250 2024-03-28T01:01:59.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.591+0000 7fc570ff5740 10 client.26335 fetch_fsmap learned FSMap version 25 2024-03-28T01:01:59.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.591+0000 7fc570ff5740 1 -- 172.21.15.169:0/4171941762 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d07bb6d250 con 0x55d07bae2250 2024-03-28T01:01:59.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.591+0000 7fc5477fe640 1 -- 172.21.15.169:0/4171941762 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc558034440 con 0x55d07bae2250 2024-03-28T01:01:59.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.591+0000 7fc5477fe640 1 -- 172.21.15.169:0/4171941762 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fc55c0999d0 con 0x55d07bae2250 2024-03-28T01:01:59.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.591+0000 7fc570ff5740 10 client.26335 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:01:59.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.591+0000 7fc570ff5740 10 client.26335 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:01:59.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.591+0000 7fc570ff5740 20 client.26335 populate_metadata read hostname 'smithi169' 2024-03-28T01:01:59.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.591+0000 7fc570ff5740 1 --2- 172.21.15.169:0/4171941762 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d07bb6dcd0 0x55d07bb700f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:01:59.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.591+0000 7fc570ff5740 4 client.26335 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:01:59.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.591+0000 7fc570ff5740 1 -- 172.21.15.169:0/4171941762 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d07ba4ec50 con 0x55d07bb6dcd0 2024-03-28T01:01:59.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.591+0000 7fc567fff640 1 --2- 172.21.15.169:0/4171941762 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d07bb6dcd0 0x55d07bb700f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:01:59.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.592+0000 7fc5477fe640 10 client.26335 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:59.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.592+0000 7fc567fff640 1 --2- 172.21.15.169:0/4171941762 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d07bb6dcd0 0x55d07bb700f0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1b70a5509353f0d server_cookie=aac568f8e1ca7f56 in_seq=0 out_seq=0 2024-03-28T01:01:59.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.593+0000 7fc5477fe640 1 -- 172.21.15.169:0/4171941762 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d07ba4ec50 con 0x55d07bb6dcd0 2024-03-28T01:01:59.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.593+0000 7fc5477fe640 10 client.26335 handle_command_reply: tid=0 2024-03-28T01:01:59.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.593+0000 7fc570ff5740 1 client.26335 shutdown 2024-03-28T01:01:59.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.593+0000 7fc570ff5740 1 -- 172.21.15.169:0/4171941762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d07bae2250 msgr2=0x55d07baf9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:59.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.593+0000 7fc570ff5740 1 --2- 172.21.15.169:0/4171941762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d07bae2250 0x55d07baf9ae0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc55c096f50 tx=0x7fc55c09a180 comp rx=0 tx=0).stop 2024-03-28T01:01:59.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.593+0000 7fc570ff5740 1 -- 172.21.15.169:0/4171941762 shutdown_connections 2024-03-28T01:01:59.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.593+0000 7fc570ff5740 1 --2- 172.21.15.169:0/4171941762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d07bae73c0 0x55d07bafaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.593+0000 7fc570ff5740 1 --2- 172.21.15.169:0/4171941762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d07bae2250 0x55d07baf9ae0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.593+0000 7fc570ff5740 1 --2- 172.21.15.169:0/4171941762 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d07bb6dcd0 0x55d07bb700f0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.593+0000 7fc570ff5740 1 --2- 172.21.15.169:0/4171941762 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d07bae1860 0x55d07baf95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.593+0000 7fc5477fe640 0 client.26335 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:01:59.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.593+0000 7fc570ff5740 1 -- 172.21.15.169:0/4171941762 >> 172.21.15.169:0/4171941762 conn(0x55d07badcdd0 msgr2=0x55d07badb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:59.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.594+0000 7fc570ff5740 1 -- 172.21.15.169:0/4171941762 shutdown_connections 2024-03-28T01:01:59.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.594+0000 7fc570ff5740 1 -- 172.21.15.169:0/4171941762 wait complete. 2024-03-28T01:01:59.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.594+0000 7fc570ff5740 20 client.26335 trim_cache size 0 max 16384 2024-03-28T01:01:59.600 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:01:59.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.594+0000 7fc56e332640 1 -- 172.21.15.169:0/2874042503 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc5500a6e10 msgr2=0x7fc5500a92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:59.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.594+0000 7fc56e332640 1 --2- 172.21.15.169:0/2874042503 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc5500a6e10 0x7fc5500a92d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc5580096c0 tx=0x7fc558040000 comp rx=0 tx=0).stop 2024-03-28T01:01:59.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.594+0000 7fc56e332640 1 -- 172.21.15.169:0/2874042503 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc568102700 msgr2=0x7fc568197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:01:59.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.594+0000 7fc56e332640 1 --2- 172.21.15.169:0/2874042503 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc568102700 0x7fc568197b60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc55c00d950 tx=0x7fc55c00de10 comp rx=0 tx=0).stop 2024-03-28T01:01:59.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.594+0000 7fc56e332640 1 -- 172.21.15.169:0/2874042503 shutdown_connections 2024-03-28T01:01:59.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.594+0000 7fc56e332640 1 --2- 172.21.15.169:0/2874042503 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc5500a6e10 0x7fc5500a92d0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.594+0000 7fc56e332640 1 --2- 172.21.15.169:0/2874042503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5681030c0 0x7fc56819ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.594+0000 7fc56e332640 1 --2- 172.21.15.169:0/2874042503 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc568102700 0x7fc568197b60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.594+0000 7fc56e332640 1 --2- 172.21.15.169:0/2874042503 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc568101510 0x7fc568197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:01:59.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.594+0000 7fc56e332640 1 -- 172.21.15.169:0/2874042503 >> 172.21.15.169:0/2874042503 conn(0x7fc5680fca80 msgr2=0x7fc568105930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:01:59.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.594+0000 7fc56e332640 1 -- 172.21.15.169:0/2874042503 shutdown_connections 2024-03-28T01:01:59.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:01:59.594+0000 7fc56e332640 1 -- 172.21.15.169:0/2874042503 wait complete. 2024-03-28T01:01:59.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:01:59 smithi169 ceph-mon[32274]: pgmap v515: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 730 MiB/s wr, 1.49k op/s 2024-03-28T01:01:59.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:01:59 smithi174 ceph-mon[36060]: pgmap v515: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 730 MiB/s wr, 1.49k op/s 2024-03-28T01:01:59.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:01:59 smithi177 ceph-mon[36206]: pgmap v515: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 730 MiB/s wr, 1.49k op/s 2024-03-28T01:02:00.612 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T01:02:00.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T01:02:00.612 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T01:02:00.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.959+0000 7fd7c4f47640 1 -- 172.21.15.169:0/2695845055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7c0108d70 msgr2=0x7fd7c01091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:00.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.959+0000 7fd7c4f47640 1 --2- 172.21.15.169:0/2695845055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7c0108d70 0x7fd7c01091f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0009900 tx=0x7fd7b002bc50 comp rx=0 tx=0).stop 2024-03-28T01:02:00.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.960+0000 7fd7c4f47640 1 -- 172.21.15.169:0/2695845055 shutdown_connections 2024-03-28T01:02:00.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.960+0000 7fd7c4f47640 1 --2- 172.21.15.169:0/2695845055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7c0109730 0x7fd7c010dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:00.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.960+0000 7fd7c4f47640 1 --2- 172.21.15.169:0/2695845055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7c0108d70 0x7fd7c01091f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:00.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.960+0000 7fd7c4f47640 1 --2- 172.21.15.169:0/2695845055 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7c0072ab0 0x7fd7c0072ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:00.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.960+0000 7fd7c4f47640 1 -- 172.21.15.169:0/2695845055 >> 172.21.15.169:0/2695845055 conn(0x7fd7c006e580 msgr2=0x7fd7c00709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:00.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.962+0000 7fd7c4f47640 1 -- 172.21.15.169:0/2695845055 shutdown_connections 2024-03-28T01:02:00.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.962+0000 7fd7c4f47640 1 -- 172.21.15.169:0/2695845055 wait complete. 2024-03-28T01:02:00.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.963+0000 7fd7c4f47640 1 Processor -- start 2024-03-28T01:02:00.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.963+0000 7fd7c4f47640 1 -- start start 2024-03-28T01:02:00.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.964+0000 7fd7c4f47640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7c0108d70 0x7fd7c0114d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:00.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.964+0000 7fd7c4f47640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7c0109730 0x7fd7c0115270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:00.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.964+0000 7fd7c4f47640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7c01b47e0 0x7fd7c01b6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:00.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.964+0000 7fd7c4f47640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c011c0e0 con 0x7fd7c0108d70 2024-03-28T01:02:00.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.964+0000 7fd7c4f47640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c011bf60 con 0x7fd7c0109730 2024-03-28T01:02:00.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.964+0000 7fd7c4f47640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c011c260 con 0x7fd7c01b47e0 2024-03-28T01:02:00.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.964+0000 7fd7bffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7c01b47e0 0x7fd7c01b6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:00.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.965+0000 7fd7bffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7c01b47e0 0x7fd7c01b6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56084/0 (socket says 172.21.15.169:56084) 2024-03-28T01:02:00.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.965+0000 7fd7bffff640 1 -- 172.21.15.169:0/79633510 learned_addr learned my addr 172.21.15.169:0/79633510 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:00.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.965+0000 7fd7beffd640 1 --2- 172.21.15.169:0/79633510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7c0109730 0x7fd7c0115270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:00.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.966+0000 7fd7bffff640 1 -- 172.21.15.169:0/79633510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7c0109730 msgr2=0x7fd7c0115270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:00.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.966+0000 7fd7bffff640 1 --2- 172.21.15.169:0/79633510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7c0109730 0x7fd7c0115270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:00.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.966+0000 7fd7bffff640 1 -- 172.21.15.169:0/79633510 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7c0108d70 msgr2=0x7fd7c0114d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:00.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.966+0000 7fd7bffff640 1 --2- 172.21.15.169:0/79633510 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7c0108d70 0x7fd7c0114d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:00.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.966+0000 7fd7bffff640 1 -- 172.21.15.169:0/79633510 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c01b7110 con 0x7fd7c01b47e0 2024-03-28T01:02:00.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.967+0000 7fd7bffff640 1 --2- 172.21.15.169:0/79633510 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7c01b47e0 0x7fd7c01b6bd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd7b800eb10 tx=0x7fd7b800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:00.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.971+0000 7fd7bcff9640 1 -- 172.21.15.169:0/79633510 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8004b20 con 0x7fd7c01b47e0 2024-03-28T01:02:00.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.971+0000 7fd7c4f47640 1 -- 172.21.15.169:0/79633510 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c01b72e0 con 0x7fd7c01b47e0 2024-03-28T01:02:00.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.971+0000 7fd7c4f47640 1 -- 172.21.15.169:0/79633510 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c01b77d0 con 0x7fd7c01b47e0 2024-03-28T01:02:00.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.971+0000 7fd7bcff9640 1 -- 172.21.15.169:0/79633510 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd7b8004cc0 con 0x7fd7c01b47e0 2024-03-28T01:02:00.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.972+0000 7fd7bcff9640 1 -- 172.21.15.169:0/79633510 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8010760 con 0x7fd7c01b47e0 2024-03-28T01:02:00.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.976+0000 7fd7c8d8c740 1 Processor -- start 2024-03-28T01:02:00.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.977+0000 7fd7c8d8c740 1 -- start start 2024-03-28T01:02:00.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.977+0000 7fd7c8d8c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8b2845b30 0x55a8b2845f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:00.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.977+0000 7fd7c8d8c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b2846520 0x55a8b284b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:00.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.977+0000 7fd7c8d8c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b284b690 0x55a8b284db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:00.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.977+0000 7fd7c8d8c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8b2785c90 con 0x55a8b2846520 2024-03-28T01:02:00.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.977+0000 7fd7c8d8c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8b27ab070 con 0x55a8b2845b30 2024-03-28T01:02:00.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.977+0000 7fd7c8d8c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8b2753c80 con 0x55a8b284b690 2024-03-28T01:02:00.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.978+0000 7fd7beffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b284b690 0x55a8b284db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:00.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.978+0000 7fd7beffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8b2845b30 0x55a8b2845f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:00.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.982+0000 7fd7bcff9640 1 -- 172.21.15.169:0/79633510 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd7b8005020 con 0x7fd7c01b47e0 2024-03-28T01:02:00.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.982+0000 7fd7bf7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b2846520 0x55a8b284b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:00.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.982+0000 7fd7bf7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b2846520 0x55a8b284b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:54754/0 (socket says 172.21.15.169:54754) 2024-03-28T01:02:00.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.982+0000 7fd7bf7fe640 1 -- 172.21.15.169:0/2783249390 learned_addr learned my addr 172.21.15.169:0/2783249390 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:00.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.983+0000 7fd7bcff9640 1 --2- 172.21.15.169:0/79633510 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd7a80a2940 0x7fd7a80a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:00.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.984+0000 7fd7bcff9640 1 -- 172.21.15.169:0/79633510 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fd7b80cf550 con 0x7fd7c01b47e0 2024-03-28T01:02:00.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.985+0000 7fd7bf7fe640 1 -- 172.21.15.169:0/2783249390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b284b690 msgr2=0x55a8b284db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:00.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.985+0000 7fd7bf7fe640 1 --2- 172.21.15.169:0/2783249390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b284b690 0x55a8b284db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:00.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.985+0000 7fd7bf7fe640 1 -- 172.21.15.169:0/2783249390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8b2845b30 msgr2=0x55a8b2845f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:00.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.985+0000 7fd7bf7fe640 1 --2- 172.21.15.169:0/2783249390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8b2845b30 0x55a8b2845f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:00.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.986+0000 7fd7bf7fe640 1 -- 172.21.15.169:0/2783249390 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8b274bd10 con 0x55a8b2846520 2024-03-28T01:02:00.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.989+0000 7fd7bf7fe640 1 --2- 172.21.15.169:0/2783249390 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b2846520 0x55a8b284b020 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400b9d0 tx=0x7fd7b400be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49eb2360b113edd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:00.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.990+0000 7fd7bffff640 1 --2- 172.21.15.169:0/79633510 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd7a80a2940 0x7fd7a80a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.996+0000 7fd7a5ffb640 1 -- 172.21.15.169:0/2783249390 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4016070 con 0x55a8b2846520 2024-03-28T01:02:01.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.996+0000 7fd7a5ffb640 1 -- 172.21.15.169:0/2783249390 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd7b400dbe0 con 0x55a8b2846520 2024-03-28T01:02:01.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:00.996+0000 7fd7a5ffb640 1 -- 172.21.15.169:0/2783249390 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b400f690 con 0x55a8b2846520 2024-03-28T01:02:01.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.001+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/2783249390 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b2846520 msgr2=0x55a8b284b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:01.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.001+0000 7fd7c8d8c740 1 --2- 172.21.15.169:0/2783249390 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b2846520 0x55a8b284b020 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400b9d0 tx=0x7fd7b400be90 comp rx=0 tx=0).stop 2024-03-28T01:02:01.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.002+0000 7fd7bffff640 1 --2- 172.21.15.169:0/79633510 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd7a80a2940 0x7fd7a80a4e00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd7b809b7f0 tx=0x7fd7b8005b40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:01.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.010+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/2783249390 shutdown_connections 2024-03-28T01:02:01.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.010+0000 7fd7c8d8c740 1 --2- 172.21.15.169:0/2783249390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b284b690 0x55a8b284db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.010+0000 7fd7c8d8c740 1 --2- 172.21.15.169:0/2783249390 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b2846520 0x55a8b284b020 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.010+0000 7fd7c8d8c740 1 --2- 172.21.15.169:0/2783249390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8b2845b30 0x55a8b2845f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.010+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/2783249390 >> 172.21.15.169:0/2783249390 conn(0x55a8b280e1a0 msgr2=0x55a8b28414a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:01.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.015+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/2783249390 shutdown_connections 2024-03-28T01:02:01.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.019+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/2783249390 wait complete. 2024-03-28T01:02:01.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.019+0000 7fd7c8d8c740 1 -- start start 2024-03-28T01:02:01.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.020+0000 7fd7c8d8c740 1 Processor -- start 2024-03-28T01:02:01.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.022+0000 7fd7c8d8c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8b2845b30 0x55a8b284ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.022+0000 7fd7c8d8c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b2846520 0x55a8b2850440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.022+0000 7fd7c8d8c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b284b690 0x55a8b2857480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.022+0000 7fd7c8d8c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8b27670d0 con 0x55a8b284b690 2024-03-28T01:02:01.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.022+0000 7fd7c8d8c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8b27a7110 con 0x55a8b2845b30 2024-03-28T01:02:01.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.022+0000 7fd7c8d8c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8b284e640 con 0x55a8b2846520 2024-03-28T01:02:01.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.022+0000 7fd7c8d8c740 10 client.0 _ll_register_callbacks cb 0x55a8b27a2e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:01.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.024+0000 7fd7beffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8b2845b30 0x55a8b284ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.026+0000 7fd7bf7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b2846520 0x55a8b2850440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.026+0000 7fd7bf7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b2846520 0x55a8b2850440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56104/0 (socket says 172.21.15.169:56104) 2024-03-28T01:02:01.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.026+0000 7fd7bf7fe640 1 -- 172.21.15.169:0/736161139 learned_addr learned my addr 172.21.15.169:0/736161139 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:01.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.027+0000 7fd7beffd640 1 --2- 172.21.15.169:0/736161139 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b284b690 0x55a8b2857480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.032+0000 7fd7beffd640 1 -- 172.21.15.169:0/736161139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b2846520 msgr2=0x55a8b2850440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:01.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.032+0000 7fd7beffd640 1 --2- 172.21.15.169:0/736161139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b2846520 0x55a8b2850440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.032+0000 7fd7beffd640 1 -- 172.21.15.169:0/736161139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8b2845b30 msgr2=0x55a8b284ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:01.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.032+0000 7fd7beffd640 1 --2- 172.21.15.169:0/736161139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8b2845b30 0x55a8b284ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.032+0000 7fd7beffd640 1 -- 172.21.15.169:0/736161139 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a8b25877e0 con 0x55a8b284b690 2024-03-28T01:02:01.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.033+0000 7fd7beffd640 1 --2- 172.21.15.169:0/736161139 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b284b690 0x55a8b2857480 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd7b002c7d0 tx=0x7fd7b002bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c726818bc56f3197 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:01.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.035+0000 7fd79effd640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:02:01.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.037+0000 7fd79effd640 1 -- 172.21.15.169:0/736161139 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b0004d50 con 0x55a8b284b690 2024-03-28T01:02:01.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.037+0000 7fd79effd640 1 -- 172.21.15.169:0/736161139 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fd7b0042440 con 0x55a8b284b690 2024-03-28T01:02:01.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.039+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/736161139 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd7b400dbe0 con 0x55a8b284b690 2024-03-28T01:02:01.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.046+0000 7fd79effd640 1 -- 172.21.15.169:0/736161139 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7b0005640 con 0x55a8b284b690 2024-03-28T01:02:01.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.048+0000 7fd7c8d8c740 10 client.16542 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:01.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.048+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/736161139 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a8b25e7f70 con 0x55a8b284b690 2024-03-28T01:02:01.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.050+0000 7fd79effd640 1 -- 172.21.15.169:0/736161139 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd7b400dbe0 con 0x55a8b284b690 2024-03-28T01:02:01.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.050+0000 7fd79effd640 1 -- 172.21.15.169:0/736161139 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd7b004e070 con 0x55a8b284b690 2024-03-28T01:02:01.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.050+0000 7fd7c8d8c740 10 client.16542 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:01.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.050+0000 7fd7c8d8c740 10 client.16542 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:02:01.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.050+0000 7fd7c8d8c740 20 client.16542 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:01.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.050+0000 7fd7c8d8c740 1 --2- 172.21.15.169:0/736161139 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a8b285c140 0x55a8b285e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.050+0000 7fd7c8d8c740 4 client.16542 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:02:01.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.050+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/736161139 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a8b27b85b0 con 0x55a8b285c140 2024-03-28T01:02:01.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.054+0000 7fd7bf7fe640 1 --2- 172.21.15.169:0/736161139 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a8b285c140 0x55a8b285e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.066+0000 7fd7bf7fe640 1 --2- 172.21.15.169:0/736161139 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a8b285c140 0x55a8b285e580 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bd6527198253db2 server_cookie=d1c41603f9f5d0d0 in_seq=0 out_seq=0 2024-03-28T01:02:01.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.066+0000 7fd79effd640 10 client.16542 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:01.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.069+0000 7fd79effd640 1 -- 172.21.15.169:0/736161139 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a8b27b85b0 con 0x55a8b285c140 2024-03-28T01:02:01.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.069+0000 7fd79effd640 10 client.16542 handle_command_reply: tid=0 2024-03-28T01:02:01.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.069+0000 7fd7c8d8c740 1 client.16542 shutdown 2024-03-28T01:02:01.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.074+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/736161139 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b284b690 msgr2=0x55a8b2857480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:01.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.074+0000 7fd7c8d8c740 1 --2- 172.21.15.169:0/736161139 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b284b690 0x55a8b2857480 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd7b002c7d0 tx=0x7fd7b002bc50 comp rx=0 tx=0).stop 2024-03-28T01:02:01.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.079+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/736161139 shutdown_connections 2024-03-28T01:02:01.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.079+0000 7fd7c8d8c740 1 --2- 172.21.15.169:0/736161139 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a8b285c140 0x55a8b285e580 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.079+0000 7fd7c8d8c740 1 --2- 172.21.15.169:0/736161139 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b284b690 0x55a8b2857480 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.079+0000 7fd7c8d8c740 1 --2- 172.21.15.169:0/736161139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b2846520 0x55a8b2850440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.079+0000 7fd7c8d8c740 1 --2- 172.21.15.169:0/736161139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8b2845b30 0x55a8b284ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.079+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/736161139 >> 172.21.15.169:0/736161139 conn(0x55a8b280e1a0 msgr2=0x55a8b28407e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:01.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.079+0000 7fd79effd640 0 client.16542 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:01.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.082+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/736161139 shutdown_connections 2024-03-28T01:02:01.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.083+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/736161139 wait complete. 2024-03-28T01:02:01.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.084+0000 7fd7c8d8c740 20 client.16542 trim_cache size 0 max 16384 2024-03-28T01:02:01.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.123+0000 7fd7c8d8c740 1 Processor -- start 2024-03-28T01:02:01.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.123+0000 7fd7c8d8c740 1 -- start start 2024-03-28T01:02:01.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.124+0000 7fd7c8d8c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b2845b30 0x55a8b28446e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.124+0000 7fd7c8d8c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8b2846520 0x55a8b2857410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.124+0000 7fd7c8d8c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b284b690 0x55a8b2844c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.124+0000 7fd7c8d8c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac0120e0 con 0x55a8b2845b30 2024-03-28T01:02:01.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.124+0000 7fd7c8d8c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac00fbe0 con 0x55a8b2846520 2024-03-28T01:02:01.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.124+0000 7fd7c8d8c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac00dc80 con 0x55a8b284b690 2024-03-28T01:02:01.129 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.126+0000 7fd7beffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b2845b30 0x55a8b28446e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.129 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.126+0000 7fd7beffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b2845b30 0x55a8b28446e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:54764/0 (socket says 172.21.15.169:54764) 2024-03-28T01:02:01.129 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.126+0000 7fd7beffd640 1 -- 172.21.15.169:0/3986770284 learned_addr learned my addr 172.21.15.169:0/3986770284 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:01.129 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.126+0000 7fd7beffd640 1 --2- 172.21.15.169:0/3986770284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b284b690 0x55a8b2844c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.128+0000 7fd7bf7fe640 1 --2- 172.21.15.169:0/3986770284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8b2846520 0x55a8b2857410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.128+0000 7fd7beffd640 1 -- 172.21.15.169:0/3986770284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b284b690 msgr2=0x55a8b2844c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:01.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.128+0000 7fd7beffd640 1 --2- 172.21.15.169:0/3986770284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b284b690 0x55a8b2844c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.128+0000 7fd7beffd640 1 -- 172.21.15.169:0/3986770284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8b2846520 msgr2=0x55a8b2857410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:01.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.128+0000 7fd7beffd640 1 --2- 172.21.15.169:0/3986770284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8b2846520 0x55a8b2857410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.129+0000 7fd7beffd640 1 -- 172.21.15.169:0/3986770284 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8b247f170 con 0x55a8b2845b30 2024-03-28T01:02:01.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.133+0000 7fd7beffd640 1 --2- 172.21.15.169:0/3986770284 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b2845b30 0x55a8b28446e0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0005c40 tx=0x7fd7b00053a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23685c6da37767c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:01.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.140+0000 7fd7a57fa640 1 -- 172.21.15.169:0/3986770284 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b002cc80 con 0x55a8b2845b30 2024-03-28T01:02:01.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.145+0000 7fd7a57fa640 1 -- 172.21.15.169:0/3986770284 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd7b0005640 con 0x55a8b2845b30 2024-03-28T01:02:01.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.145+0000 7fd7a57fa640 1 -- 172.21.15.169:0/3986770284 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b00035f0 con 0x55a8b2845b30 2024-03-28T01:02:01.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.146+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/3986770284 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b2845b30 msgr2=0x55a8b28446e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:01.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.146+0000 7fd7c8d8c740 1 --2- 172.21.15.169:0/3986770284 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b2845b30 0x55a8b28446e0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0005c40 tx=0x7fd7b00053a0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.147+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/3986770284 shutdown_connections 2024-03-28T01:02:01.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.147+0000 7fd7c8d8c740 1 --2- 172.21.15.169:0/3986770284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b284b690 0x55a8b2844c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.147+0000 7fd7c8d8c740 1 --2- 172.21.15.169:0/3986770284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8b2846520 0x55a8b2857410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.147+0000 7fd7c8d8c740 1 --2- 172.21.15.169:0/3986770284 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b2845b30 0x55a8b28446e0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.147+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/3986770284 >> 172.21.15.169:0/3986770284 conn(0x55a8b280e1a0 msgr2=0x55a8b2847a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:01.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.147+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/3986770284 shutdown_connections 2024-03-28T01:02:01.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.147+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/3986770284 wait complete. 2024-03-28T01:02:01.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.148+0000 7fd7c8d8c740 1 -- start start 2024-03-28T01:02:01.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.148+0000 7fd7c8d8c740 1 Processor -- start 2024-03-28T01:02:01.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.149+0000 7fd7c8d8c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8b2845b30 0x55a8b2856e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.149+0000 7fd7c8d8c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b2846520 0x55a8b2857350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.149+0000 7fd7c8d8c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b284b690 0x55a8b28554c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.149+0000 7fd7c8d8c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac00e280 con 0x55a8b284b690 2024-03-28T01:02:01.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.149+0000 7fd7c8d8c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac00f580 con 0x55a8b2845b30 2024-03-28T01:02:01.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.149+0000 7fd7c8d8c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac00e8c0 con 0x55a8b2846520 2024-03-28T01:02:01.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.150+0000 7fd7bf7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b2846520 0x55a8b2857350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.150+0000 7fd7bf7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b2846520 0x55a8b2857350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56122/0 (socket says 172.21.15.169:56122) 2024-03-28T01:02:01.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.150+0000 7fd7bf7fe640 1 -- 172.21.15.169:0/1487321745 learned_addr learned my addr 172.21.15.169:0/1487321745 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:01.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.150+0000 7fd7beffd640 1 --2- 172.21.15.169:0/1487321745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8b2845b30 0x55a8b2856e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.150+0000 7fd7c8d8c740 10 client.0 _ll_register_callbacks cb 0x7fd7b0009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:01.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.150+0000 7fd7beffd640 1 --2- 172.21.15.169:0/1487321745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b284b690 0x55a8b28554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.151+0000 7fd7bf7fe640 1 -- 172.21.15.169:0/1487321745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8b2845b30 msgr2=0x55a8b2856e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:01.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.151+0000 7fd7bf7fe640 1 --2- 172.21.15.169:0/1487321745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8b2845b30 0x55a8b2856e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.151+0000 7fd7bf7fe640 1 -- 172.21.15.169:0/1487321745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b284b690 msgr2=0x55a8b28554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:01.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.151+0000 7fd7bf7fe640 1 --2- 172.21.15.169:0/1487321745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b284b690 0x55a8b28554c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.151+0000 7fd7bf7fe640 1 -- 172.21.15.169:0/1487321745 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a8b2855c00 con 0x55a8b2846520 2024-03-28T01:02:01.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.151+0000 7fd7bf7fe640 1 --2- 172.21.15.169:0/1487321745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b2846520 0x55a8b2857350 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4004700 tx=0x7fd7b400df00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62f6b6eddaee16e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:01.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.152+0000 7fd79effd640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:02:01.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.154+0000 7fd79effd640 1 -- 172.21.15.169:0/1487321745 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b402c070 con 0x55a8b2846520 2024-03-28T01:02:01.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.154+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/1487321745 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd7b0005640 con 0x55a8b2846520 2024-03-28T01:02:01.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.154+0000 7fd79effd640 1 -- 172.21.15.169:0/1487321745 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fd7b40286f0 con 0x55a8b2846520 2024-03-28T01:02:01.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.156+0000 7fd79effd640 1 -- 172.21.15.169:0/1487321745 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7b4035050 con 0x55a8b2846520 2024-03-28T01:02:01.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.156+0000 7fd7c8d8c740 10 client.26350 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:01.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.156+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/1487321745 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a8b28b84e0 con 0x55a8b2846520 2024-03-28T01:02:01.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.161+0000 7fd79effd640 1 -- 172.21.15.169:0/1487321745 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd7b0005640 con 0x55a8b2846520 2024-03-28T01:02:01.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.162+0000 7fd79effd640 1 -- 172.21.15.169:0/1487321745 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd7b402f020 con 0x55a8b2846520 2024-03-28T01:02:01.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.162+0000 7fd7c8d8c740 10 client.26350 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:01.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.162+0000 7fd7c8d8c740 10 client.26350 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:02:01.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.162+0000 7fd7c8d8c740 20 client.26350 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:01.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.162+0000 7fd7c8d8c740 1 --2- 172.21.15.169:0/1487321745 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a8b28b8e40 0x55a8b28bb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.162+0000 7fd7c8d8c740 4 client.26350 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T01:02:01.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.162+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/1487321745 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55a8b2847530 con 0x55a8b28b8e40 2024-03-28T01:02:01.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.164+0000 7fd7beffd640 1 --2- 172.21.15.169:0/1487321745 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a8b28b8e40 0x55a8b28bb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.174+0000 7fd7beffd640 1 --2- 172.21.15.169:0/1487321745 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a8b28b8e40 0x55a8b28bb260 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff4f2bfa5067bdee server_cookie=bcce450da0f8eee8 in_seq=0 out_seq=0 2024-03-28T01:02:01.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.174+0000 7fd79effd640 10 client.26350 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:01.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.183+0000 7fd79effd640 1 -- 172.21.15.169:0/1487321745 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a8b2847530 con 0x55a8b28b8e40 2024-03-28T01:02:01.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.183+0000 7fd79effd640 10 client.26350 handle_command_reply: tid=0 2024-03-28T01:02:01.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.184+0000 7fd7c8d8c740 1 client.26350 shutdown 2024-03-28T01:02:01.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.191+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/1487321745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b2846520 msgr2=0x55a8b2857350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:01.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.191+0000 7fd7c8d8c740 1 --2- 172.21.15.169:0/1487321745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b2846520 0x55a8b2857350 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4004700 tx=0x7fd7b400df00 comp rx=0 tx=0).stop 2024-03-28T01:02:01.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.194+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/1487321745 shutdown_connections 2024-03-28T01:02:01.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.194+0000 7fd7c8d8c740 1 --2- 172.21.15.169:0/1487321745 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a8b28b8e40 0x55a8b28bb260 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.194+0000 7fd7c8d8c740 1 --2- 172.21.15.169:0/1487321745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a8b284b690 0x55a8b28554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.194+0000 7fd7c8d8c740 1 --2- 172.21.15.169:0/1487321745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8b2846520 0x55a8b2857350 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.194+0000 7fd7c8d8c740 1 --2- 172.21.15.169:0/1487321745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8b2845b30 0x55a8b2856e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.194+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/1487321745 >> 172.21.15.169:0/1487321745 conn(0x55a8b280e1a0 msgr2=0x55a8b283ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:01.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.194+0000 7fd79effd640 0 client.26350 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:01.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.197+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/1487321745 shutdown_connections 2024-03-28T01:02:01.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.198+0000 7fd7c8d8c740 1 -- 172.21.15.169:0/1487321745 wait complete. 2024-03-28T01:02:01.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.198+0000 7fd7c8d8c740 20 client.26350 trim_cache size 0 max 16384 2024-03-28T01:02:01.203 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:02:01.203 INFO:teuthology.orchestra.run.smithi169.stdout: "return_code": 0, 2024-03-28T01:02:01.203 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_tag": "d53fe40b-7254-44b3-949f-5e2c733c431c", 2024-03-28T01:02:01.203 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": "asynchronous" 2024-03-28T01:02:01.203 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:02:01.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.201+0000 7fd7c4f47640 1 -- 172.21.15.169:0/79633510 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd7a80a2940 msgr2=0x7fd7a80a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:01.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.201+0000 7fd7c4f47640 1 --2- 172.21.15.169:0/79633510 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd7a80a2940 0x7fd7a80a4e00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd7b809b7f0 tx=0x7fd7b8005b40 comp rx=0 tx=0).stop 2024-03-28T01:02:01.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.201+0000 7fd7c4f47640 1 -- 172.21.15.169:0/79633510 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7c01b47e0 msgr2=0x7fd7c01b6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:01.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.201+0000 7fd7c4f47640 1 --2- 172.21.15.169:0/79633510 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7c01b47e0 0x7fd7c01b6bd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd7b800eb10 tx=0x7fd7b800c2d0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.201+0000 7fd7c4f47640 1 -- 172.21.15.169:0/79633510 shutdown_connections 2024-03-28T01:02:01.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.201+0000 7fd7c4f47640 1 --2- 172.21.15.169:0/79633510 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd7a80a2940 0x7fd7a80a4e00 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.201+0000 7fd7c4f47640 1 --2- 172.21.15.169:0/79633510 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7c01b47e0 0x7fd7c01b6bd0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.201+0000 7fd7c4f47640 1 --2- 172.21.15.169:0/79633510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7c0109730 0x7fd7c0115270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.201+0000 7fd7c4f47640 1 --2- 172.21.15.169:0/79633510 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7c0108d70 0x7fd7c0114d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.202+0000 7fd7c4f47640 1 -- 172.21.15.169:0/79633510 >> 172.21.15.169:0/79633510 conn(0x7fd7c006e580 msgr2=0x7fd7c0070170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:01.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.205+0000 7fd7c4f47640 1 -- 172.21.15.169:0/79633510 shutdown_connections 2024-03-28T01:02:01.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.206+0000 7fd7c4f47640 1 -- 172.21.15.169:0/79633510 wait complete. 2024-03-28T01:02:01.247 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:02:01.247 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:02:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.549+0000 7f15bf59e640 1 -- 172.21.15.169:0/1114211988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15c0076270 msgr2=0x7f15c00786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.549+0000 7f15bf59e640 1 --2- 172.21.15.169:0/1114211988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15c0076270 0x7f15c00786e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f15b800afd0 tx=0x7f15b80301f0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.551+0000 7f15bf59e640 1 -- 172.21.15.169:0/1114211988 shutdown_connections 2024-03-28T01:02:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.551+0000 7f15bf59e640 1 --2- 172.21.15.169:0/1114211988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15c0078c20 0x7f15c007b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.551+0000 7f15bf59e640 1 --2- 172.21.15.169:0/1114211988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15c0076270 0x7f15c00786e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.551+0000 7f15bf59e640 1 --2- 172.21.15.169:0/1114211988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15c0073390 0x7f15c00737b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.551+0000 7f15bf59e640 1 -- 172.21.15.169:0/1114211988 >> 172.21.15.169:0/1114211988 conn(0x7f15c006e510 msgr2=0x7f15c0070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.552+0000 7f15bf59e640 1 -- 172.21.15.169:0/1114211988 shutdown_connections 2024-03-28T01:02:01.559 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.555+0000 7f15bf59e640 1 -- 172.21.15.169:0/1114211988 wait complete. 2024-03-28T01:02:01.559 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.556+0000 7f15bf59e640 1 Processor -- start 2024-03-28T01:02:01.571 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.557+0000 7f15bf59e640 1 -- start start 2024-03-28T01:02:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.557+0000 7f15bf59e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15c0073390 0x7f15c01b7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.557+0000 7f15bf59e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15c0076270 0x7f15c01b8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.557+0000 7f15bf59e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15c0078c20 0x7f15c01becf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.557+0000 7f15bf59e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c007eb30 con 0x7f15c0078c20 2024-03-28T01:02:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.557+0000 7f15bf59e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c007e9b0 con 0x7f15c0073390 2024-03-28T01:02:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.557+0000 7f15bf59e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c007ecb0 con 0x7f15c0076270 2024-03-28T01:02:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.558+0000 7f15bed9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15c0078c20 0x7f15c01becf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.558+0000 7f15bed9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15c0078c20 0x7f15c01becf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:54790/0 (socket says 172.21.15.169:54790) 2024-03-28T01:02:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.558+0000 7f15bed9d640 1 -- 172.21.15.169:0/3751165343 learned_addr learned my addr 172.21.15.169:0/3751165343 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.559+0000 7f15bed9d640 1 -- 172.21.15.169:0/3751165343 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15c0076270 msgr2=0x7f15c01b8330 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.559+0000 7f15bed9d640 1 --2- 172.21.15.169:0/3751165343 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15c0076270 0x7f15c01b8330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.559+0000 7f15bed9d640 1 -- 172.21.15.169:0/3751165343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15c0073390 msgr2=0x7f15c01b7df0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.559+0000 7f15bed9d640 1 --2- 172.21.15.169:0/3751165343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15c0073390 0x7f15c01b7df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.559+0000 7f15bed9d640 1 -- 172.21.15.169:0/3751165343 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c01af4f0 con 0x7f15c0078c20 2024-03-28T01:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.560+0000 7f15bed9d640 1 --2- 172.21.15.169:0/3751165343 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15c0078c20 0x7f15c01becf0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f15b000b4a0 tx=0x7f15b000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.565+0000 7f15af7fe640 1 -- 172.21.15.169:0/3751165343 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b0014070 con 0x7f15c0078c20 2024-03-28T01:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.566+0000 7f15bf59e640 1 -- 172.21.15.169:0/3751165343 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c01af780 con 0x7f15c0078c20 2024-03-28T01:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.566+0000 7f15bf59e640 1 -- 172.21.15.169:0/3751165343 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c01afd50 con 0x7f15c0078c20 2024-03-28T01:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.566+0000 7f15af7fe640 1 -- 172.21.15.169:0/3751165343 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15b0010af0 con 0x7f15c0078c20 2024-03-28T01:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.566+0000 7f15af7fe640 1 -- 172.21.15.169:0/3751165343 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b0017cc0 con 0x7f15c0078c20 2024-03-28T01:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.576+0000 7f15c77c0740 1 Processor -- start 2024-03-28T01:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.578+0000 7f15c77c0740 1 -- start start 2024-03-28T01:02:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.578+0000 7f15c77c0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754326ab20 0x55754326af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.578+0000 7f15c77c0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55754326b510 0x557543270010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.578+0000 7f15c77c0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557543270680 0x557543272b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.578+0000 7f15c77c0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575431cfb10 con 0x557543270680 2024-03-28T01:02:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.578+0000 7f15c77c0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x557543178900 con 0x55754326b510 2024-03-28T01:02:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.578+0000 7f15c77c0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575431cbbb0 con 0x55754326ab20 2024-03-28T01:02:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.578+0000 7f15af7fe640 1 -- 172.21.15.169:0/3751165343 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f15b0017070 con 0x7f15c0078c20 2024-03-28T01:02:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.579+0000 7f15af7fe640 1 --2- 172.21.15.169:0/3751165343 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f15a80c4a10 0x7f15a80c6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.580+0000 7f15af7fe640 1 -- 172.21.15.169:0/3751165343 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f15b00ce6f0 con 0x7f15c0078c20 2024-03-28T01:02:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.582+0000 7f15bed9d640 1 --2- 172.21.15.169:0/3751165343 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f15a80c4a10 0x7f15a80c6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.582+0000 7f15bdd9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754326ab20 0x55754326af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.582+0000 7f15bdd9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557543270680 0x557543272b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.582+0000 7f15bdd9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557543270680 0x557543272b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:54810/0 (socket says 172.21.15.169:54810) 2024-03-28T01:02:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.582+0000 7f15bdd9b640 1 -- 172.21.15.169:0/388091647 learned_addr learned my addr 172.21.15.169:0/388091647 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.583+0000 7f15bdd9b640 1 -- 172.21.15.169:0/388091647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55754326b510 msgr2=0x557543270010 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:02:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.583+0000 7f15bdd9b640 1 --2- 172.21.15.169:0/388091647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55754326b510 0x557543270010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.583+0000 7f15bdd9b640 1 -- 172.21.15.169:0/388091647 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557543270680 msgr2=0x557543272b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.583+0000 7f15bdd9b640 1 --2- 172.21.15.169:0/388091647 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557543270680 0x557543272b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.583+0000 7f15bdd9b640 1 -- 172.21.15.169:0/388091647 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557542ee4210 con 0x55754326ab20 2024-03-28T01:02:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.583+0000 7f15bed9d640 1 --2- 172.21.15.169:0/3751165343 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f15a80c4a10 0x7f15a80c6ed0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f15b000be60 tx=0x7f15b00092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.587+0000 7f15bdd9b640 1 --2- 172.21.15.169:0/388091647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754326ab20 0x55754326af40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f15b8008a50 tx=0x7f15b80301f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d645e5f534e485bf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.588+0000 7f15acff9640 1 -- 172.21.15.169:0/388091647 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b8037a20 con 0x55754326ab20 2024-03-28T01:02:01.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.588+0000 7fb869827640 1 -- 172.21.15.169:0/1686747355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb8641086a0 msgr2=0x7fb86410aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:01.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.588+0000 7fb869827640 1 --2- 172.21.15.169:0/1686747355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb8641086a0 0x7fb86410aad0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb85401a910 tx=0x7fb85403cca0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.588+0000 7f15acff9640 1 -- 172.21.15.169:0/388091647 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15b8037bc0 con 0x55754326ab20 2024-03-28T01:02:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.589+0000 7f15c77c0740 1 -- 172.21.15.169:0/388091647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754326ab20 msgr2=0x55754326af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.589+0000 7f15c77c0740 1 --2- 172.21.15.169:0/388091647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754326ab20 0x55754326af40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f15b8008a50 tx=0x7f15b80301f0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.591+0000 7f15c77c0740 1 -- 172.21.15.169:0/388091647 shutdown_connections 2024-03-28T01:02:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.591+0000 7f15c77c0740 1 --2- 172.21.15.169:0/388091647 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557543270680 0x557543272b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.591+0000 7f15c77c0740 1 --2- 172.21.15.169:0/388091647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55754326b510 0x557543270010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.591+0000 7f15c77c0740 1 --2- 172.21.15.169:0/388091647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754326ab20 0x55754326af40 secure :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f15b8008a50 tx=0x7f15b80301f0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.591+0000 7f15c77c0740 1 -- 172.21.15.169:0/388091647 >> 172.21.15.169:0/388091647 conn(0x557543266090 msgr2=0x557543266490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.595+0000 7f15c77c0740 1 -- 172.21.15.169:0/388091647 shutdown_connections 2024-03-28T01:02:01.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.588+0000 7fb869827640 1 -- 172.21.15.169:0/1686747355 shutdown_connections 2024-03-28T01:02:01.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.588+0000 7fb869827640 1 --2- 172.21.15.169:0/1686747355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb86410d9c0 0x7fb86410de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.588+0000 7fb869827640 1 --2- 172.21.15.169:0/1686747355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb86410b010 0x7fb86410d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.588+0000 7fb869827640 1 --2- 172.21.15.169:0/1686747355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb8641086a0 0x7fb86410aad0 secure :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb85401a910 tx=0x7fb85403cca0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.588+0000 7fb869827640 1 -- 172.21.15.169:0/1686747355 >> 172.21.15.169:0/1686747355 conn(0x7fb86406e500 msgr2=0x7fb864070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:01.599 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.596+0000 7f15c77c0740 1 -- 172.21.15.169:0/388091647 wait complete. 2024-03-28T01:02:01.600 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.596+0000 7f15c77c0740 1 -- start start 2024-03-28T01:02:01.600 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.596+0000 7f15c77c0740 1 Processor -- start 2024-03-28T01:02:01.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.597+0000 7fb869827640 1 -- 172.21.15.169:0/1686747355 shutdown_connections 2024-03-28T01:02:01.600 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.597+0000 7f15c77c0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754326b510 0x557543274f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.600 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.597+0000 7f15c77c0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557543270680 0x5575432754c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.597+0000 7f15c77c0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55754327a4f0 0x55754327c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.597+0000 7f15c77c0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x557543273620 con 0x55754327a4f0 2024-03-28T01:02:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.597+0000 7f15c77c0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55754318bd50 con 0x557543270680 2024-03-28T01:02:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.597+0000 7f15c77c0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575432737a0 con 0x55754326b510 2024-03-28T01:02:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.597+0000 7f15c77c0740 10 client.0 _ll_register_callbacks cb 0x55754318b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.597+0000 7f15be59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557543270680 0x5575432754c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.597+0000 7f15be59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557543270680 0x5575432754c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49482/0 (socket says 172.21.15.169:49482) 2024-03-28T01:02:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.597+0000 7f15be59c640 1 -- 172.21.15.169:0/417920702 learned_addr learned my addr 172.21.15.169:0/417920702 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.598+0000 7f15be59c640 1 -- 172.21.15.169:0/417920702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754326b510 msgr2=0x557543274f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:02:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.598+0000 7f15be59c640 1 --2- 172.21.15.169:0/417920702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754326b510 0x557543274f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.598+0000 7f15be59c640 1 -- 172.21.15.169:0/417920702 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55754327a4f0 msgr2=0x55754327c8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:02:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.598+0000 7f15be59c640 1 --2- 172.21.15.169:0/417920702 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55754327a4f0 0x55754327c8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.598+0000 7f15be59c640 1 -- 172.21.15.169:0/417920702 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557542fb0620 con 0x557543270680 2024-03-28T01:02:01.602 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.598+0000 7f15be59c640 1 --2- 172.21.15.169:0/417920702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557543270680 0x5575432754c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009d80 tx=0x7f15b400b4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fed529cabe2666 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:01.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.601+0000 7fb869827640 1 -- 172.21.15.169:0/1686747355 wait complete. 2024-03-28T01:02:01.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.602+0000 7fb869827640 1 Processor -- start 2024-03-28T01:02:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.603+0000 7f15a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:02:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.603+0000 7f15a5ffb640 1 -- 172.21.15.169:0/417920702 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b4011420 con 0x557543270680 2024-03-28T01:02:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.603+0000 7f15c77c0740 1 -- 172.21.15.169:0/417920702 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f15b8037bc0 con 0x557543270680 2024-03-28T01:02:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.603+0000 7f15a5ffb640 1 -- 172.21.15.169:0/417920702 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f15b40115c0 con 0x557543270680 2024-03-28T01:02:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.607+0000 7f15a5ffb640 1 -- 172.21.15.169:0/417920702 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15b401f050 con 0x557543270680 2024-03-28T01:02:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.607+0000 7f15c77c0740 10 client.26423 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.607+0000 7f15c77c0740 1 -- 172.21.15.169:0/417920702 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55754327fcf0 con 0x557543270680 2024-03-28T01:02:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.609+0000 7f15a5ffb640 1 -- 172.21.15.169:0/417920702 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15b8037bc0 con 0x557543270680 2024-03-28T01:02:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.609+0000 7f15a5ffb640 1 -- 172.21.15.169:0/417920702 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f15b401aaa0 con 0x557543270680 2024-03-28T01:02:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.611+0000 7f15c77c0740 10 client.26423 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.611+0000 7f15c77c0740 10 client.26423 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:02:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.611+0000 7f15c77c0740 20 client.26423 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.611+0000 7f15c77c0740 1 --2- 172.21.15.169:0/417920702 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557543281480 0x557543283870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.611+0000 7f15c77c0740 4 client.26423 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:02:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.611+0000 7f15c77c0740 1 -- 172.21.15.169:0/417920702 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5575431dd050 con 0x557543281480 2024-03-28T01:02:01.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.609+0000 7fb869827640 1 -- start start 2024-03-28T01:02:01.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.610+0000 7fb869827640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb86410b010 0x7fb86419e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.610+0000 7fb869827640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb86410d9c0 0x7fb86419e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.610+0000 7fb869827640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8641a37b0 0x7fb8641a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.610+0000 7fb869827640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8641119c0 con 0x7fb86410d9c0 2024-03-28T01:02:01.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.610+0000 7fb869827640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb864111840 con 0x7fb86410b010 2024-03-28T01:02:01.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.610+0000 7fb869827640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb864111b40 con 0x7fb8641a37b0 2024-03-28T01:02:01.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.612+0000 7fb862ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb86410b010 0x7fb86419e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.612+0000 7fb862ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb86410b010 0x7fb86419e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49488/0 (socket says 172.21.15.169:49488) 2024-03-28T01:02:01.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.612+0000 7fb862ffd640 1 -- 172.21.15.169:0/268831551 learned_addr learned my addr 172.21.15.169:0/268831551 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:01.616 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.613+0000 7f15bdd9b640 1 --2- 172.21.15.169:0/417920702 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557543281480 0x557543283870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.613+0000 7fb862ffd640 1 -- 172.21.15.169:0/268831551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8641a37b0 msgr2=0x7fb8641a5ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:02:01.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.613+0000 7fb862ffd640 1 --2- 172.21.15.169:0/268831551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8641a37b0 0x7fb8641a5ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.613+0000 7fb862ffd640 1 -- 172.21.15.169:0/268831551 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb86410d9c0 msgr2=0x7fb86419e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:01.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.613+0000 7fb862ffd640 1 --2- 172.21.15.169:0/268831551 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb86410d9c0 0x7fb86419e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.613+0000 7fb862ffd640 1 -- 172.21.15.169:0/268831551 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8641a6210 con 0x7fb86410b010 2024-03-28T01:02:01.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.613+0000 7fb862ffd640 1 --2- 172.21.15.169:0/268831551 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb86410b010 0x7fb86419e210 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb85403d2b0 tx=0x7fb854004fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:01.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.614+0000 7fb868825640 1 -- 172.21.15.169:0/268831551 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85404a040 con 0x7fb86410b010 2024-03-28T01:02:01.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.614+0000 7fb868825640 1 -- 172.21.15.169:0/268831551 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb8540035f0 con 0x7fb86410b010 2024-03-28T01:02:01.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.615+0000 7fb868825640 1 -- 172.21.15.169:0/268831551 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb854045680 con 0x7fb86410b010 2024-03-28T01:02:01.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.615+0000 7fb869827640 1 -- 172.21.15.169:0/268831551 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8641a64a0 con 0x7fb86410b010 2024-03-28T01:02:01.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.615+0000 7fb869827640 1 -- 172.21.15.169:0/268831551 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8641a6940 con 0x7fb86410b010 2024-03-28T01:02:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.619+0000 7f15bdd9b640 1 --2- 172.21.15.169:0/417920702 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557543281480 0x557543283870 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fbda519184edc9e server_cookie=cfc845e60bac4c1b in_seq=0 out_seq=0 2024-03-28T01:02:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.619+0000 7f15a5ffb640 10 client.26423 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:01.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.624+0000 7fb86c412740 1 Processor -- start 2024-03-28T01:02:01.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.624+0000 7fb86c412740 1 -- start start 2024-03-28T01:02:01.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.624+0000 7fb868825640 1 -- 172.21.15.169:0/268831551 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb854055020 con 0x7fb86410b010 2024-03-28T01:02:01.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.625+0000 7fb86c412740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce131a9da0 0x55ce131aa1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.626+0000 7fb86c412740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131aa790 0x55ce131af290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.626+0000 7fb86c412740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131af900 0x55ce131b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.626+0000 7fb86c412740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce1310f3d0 con 0x55ce131a9da0 2024-03-28T01:02:01.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.626+0000 7fb862ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131aa790 0x55ce131af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.626+0000 7fb862ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131aa790 0x55ce131af290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49500/0 (socket says 172.21.15.169:49500) 2024-03-28T01:02:01.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.626+0000 7fb862ffd640 1 -- 172.21.15.169:0/3230199902 learned_addr learned my addr 172.21.15.169:0/3230199902 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:01.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.626+0000 7fb86c412740 1 -- 172.21.15.169:0/3230199902 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce130b7ad0 con 0x55ce131aa790 2024-03-28T01:02:01.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.626+0000 7fb8627fc640 1 --2- 172.21.15.169:0/3230199902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131af900 0x55ce131b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.627+0000 7fb86c412740 1 -- 172.21.15.169:0/3230199902 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce1310b470 con 0x55ce131af900 2024-03-28T01:02:01.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.628+0000 7fb868825640 1 --2- 172.21.15.169:0/268831551 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb84c0a2a10 0x7fb84c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.628+0000 7fb862ffd640 1 -- 172.21.15.169:0/3230199902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131af900 msgr2=0x55ce131b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:01.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.628+0000 7fb862ffd640 1 --2- 172.21.15.169:0/3230199902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131af900 0x55ce131b1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.628+0000 7fb862ffd640 1 -- 172.21.15.169:0/3230199902 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce131a9da0 msgr2=0x55ce131aa1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:01.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.628+0000 7fb862ffd640 1 --2- 172.21.15.169:0/3230199902 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce131a9da0 0x55ce131aa1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.628+0000 7fb862ffd640 1 -- 172.21.15.169:0/3230199902 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce12eed7d0 con 0x55ce131aa790 2024-03-28T01:02:01.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.628+0000 7fb8627fc640 1 --2- 172.21.15.169:0/3230199902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131af900 0x55ce131b1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:02:01.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.628+0000 7fb862ffd640 1 --2- 172.21.15.169:0/3230199902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131aa790 0x55ce131af290 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb854104c70 tx=0x7fb854109640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f1beb6ea78cf12f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:01.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.629+0000 7fb868825640 1 -- 172.21.15.169:0/268831551 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb854100c70 con 0x7fb86410b010 2024-03-28T01:02:01.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.629+0000 7fb8637fe640 1 --2- 172.21.15.169:0/268831551 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb84c0a2a10 0x7fb84c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.632+0000 7fb849ffb640 1 -- 172.21.15.169:0/3230199902 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb854110070 con 0x55ce131aa790 2024-03-28T01:02:01.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.632+0000 7fb849ffb640 1 -- 172.21.15.169:0/3230199902 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb8540cd910 con 0x55ce131aa790 2024-03-28T01:02:01.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.632+0000 7fb849ffb640 1 -- 172.21.15.169:0/3230199902 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8540c94b0 con 0x55ce131aa790 2024-03-28T01:02:01.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.632+0000 7fb8637fe640 1 --2- 172.21.15.169:0/268831551 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb84c0a2a10 0x7fb84c0a4ed0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x55ce131aaff0 tx=0x7fb858005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:01.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.634+0000 7fb86c412740 1 -- 172.21.15.169:0/3230199902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131aa790 msgr2=0x55ce131af290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:01.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.634+0000 7fb86c412740 1 --2- 172.21.15.169:0/3230199902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131aa790 0x55ce131af290 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb854104c70 tx=0x7fb854109640 comp rx=0 tx=0).stop 2024-03-28T01:02:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.634+0000 7f15a5ffb640 1 -- 172.21.15.169:0/417920702 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5575431dd050 con 0x557543281480 2024-03-28T01:02:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.634+0000 7f15a5ffb640 10 client.26423 handle_command_reply: tid=0 2024-03-28T01:02:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.634+0000 7f15c77c0740 1 client.26423 shutdown 2024-03-28T01:02:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.634+0000 7f15c77c0740 1 -- 172.21.15.169:0/417920702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557543270680 msgr2=0x5575432754c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.634+0000 7f15c77c0740 1 --2- 172.21.15.169:0/417920702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557543270680 0x5575432754c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009d80 tx=0x7f15b400b4b0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.641 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.635+0000 7f15c77c0740 1 -- 172.21.15.169:0/417920702 shutdown_connections 2024-03-28T01:02:01.641 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.635+0000 7f15c77c0740 1 --2- 172.21.15.169:0/417920702 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557543281480 0x557543283870 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.641 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.635+0000 7f15c77c0740 1 --2- 172.21.15.169:0/417920702 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55754327a4f0 0x55754327c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.641 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.635+0000 7f15c77c0740 1 --2- 172.21.15.169:0/417920702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557543270680 0x5575432754c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.641 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.635+0000 7f15c77c0740 1 --2- 172.21.15.169:0/417920702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754326b510 0x557543274f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.641 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.635+0000 7f15c77c0740 1 -- 172.21.15.169:0/417920702 >> 172.21.15.169:0/417920702 conn(0x557543266090 msgr2=0x557543265290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:01.643 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.638+0000 7f15a5ffb640 0 client.26423 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:01.643 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.638+0000 7f15c77c0740 1 -- 172.21.15.169:0/417920702 shutdown_connections 2024-03-28T01:02:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.641+0000 7f15c77c0740 1 -- 172.21.15.169:0/417920702 wait complete. 2024-03-28T01:02:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.641+0000 7f15c77c0740 20 client.26423 trim_cache size 0 max 16384 2024-03-28T01:02:01.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.641+0000 7fb86c412740 1 -- 172.21.15.169:0/3230199902 shutdown_connections 2024-03-28T01:02:01.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.641+0000 7fb86c412740 1 --2- 172.21.15.169:0/3230199902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131af900 0x55ce131b1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.641+0000 7fb86c412740 1 --2- 172.21.15.169:0/3230199902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131aa790 0x55ce131af290 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.641+0000 7fb86c412740 1 --2- 172.21.15.169:0/3230199902 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce131a9da0 0x55ce131aa1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.641+0000 7fb86c412740 1 -- 172.21.15.169:0/3230199902 >> 172.21.15.169:0/3230199902 conn(0x55ce131a5310 msgr2=0x55ce131a5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:01.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.641+0000 7fb86c412740 1 -- 172.21.15.169:0/3230199902 shutdown_connections 2024-03-28T01:02:01.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.641+0000 7fb86c412740 1 -- 172.21.15.169:0/3230199902 wait complete. 2024-03-28T01:02:01.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.642+0000 7fb86c412740 1 -- start start 2024-03-28T01:02:01.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.642+0000 7fb86c412740 1 Processor -- start 2024-03-28T01:02:01.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.643+0000 7fb86c412740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce131a9da0 0x55ce131b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.643+0000 7fb86c412740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131aa790 0x55ce131b4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.643+0000 7fb86c412740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131af900 0x55ce131bb420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.643+0000 7fb86c412740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce131b2670 con 0x55ce131a9da0 2024-03-28T01:02:01.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.643+0000 7fb86c412740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce130cb060 con 0x55ce131aa790 2024-03-28T01:02:01.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.643+0000 7fb86c412740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce131b27f0 con 0x55ce131af900 2024-03-28T01:02:01.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.643+0000 7fb86c412740 10 client.0 _ll_register_callbacks cb 0x55ce130ad6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:01.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.643+0000 7fb8627fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131aa790 0x55ce131b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.643+0000 7fb8627fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131aa790 0x55ce131b4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49506/0 (socket says 172.21.15.169:49506) 2024-03-28T01:02:01.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.643+0000 7fb8627fc640 1 -- 172.21.15.169:0/271871960 learned_addr learned my addr 172.21.15.169:0/271871960 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:01.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.644+0000 7fb8637fe640 1 --2- 172.21.15.169:0/271871960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131af900 0x55ce131bb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.644+0000 7fb8637fe640 1 -- 172.21.15.169:0/271871960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131aa790 msgr2=0x55ce131b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:01.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.644+0000 7fb8637fe640 1 --2- 172.21.15.169:0/271871960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131aa790 0x55ce131b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.644+0000 7fb8637fe640 1 -- 172.21.15.169:0/271871960 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce131a9da0 msgr2=0x55ce131b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:01.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.644+0000 7fb8637fe640 1 --2- 172.21.15.169:0/271871960 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce131a9da0 0x55ce131b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.644+0000 7fb8637fe640 1 -- 172.21.15.169:0/271871960 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ce12e233f0 con 0x55ce131af900 2024-03-28T01:02:01.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.645+0000 7fb8637fe640 1 --2- 172.21.15.169:0/271871960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131af900 0x55ce131bb420 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb86406b390 tx=0x7fb85800faa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5413ea627eb384d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:01.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.645+0000 7fb8477f6640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:02:01.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.645+0000 7fb8477f6640 1 -- 172.21.15.169:0/271871960 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb858020070 con 0x55ce131af900 2024-03-28T01:02:01.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.645+0000 7fb86c412740 1 -- 172.21.15.169:0/271871960 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb8540cd910 con 0x55ce131af900 2024-03-28T01:02:01.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.645+0000 7fb8477f6640 1 -- 172.21.15.169:0/271871960 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb858004220 con 0x55ce131af900 2024-03-28T01:02:01.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.645+0000 7fb8477f6640 1 -- 172.21.15.169:0/271871960 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb85802b050 con 0x55ce131af900 2024-03-28T01:02:01.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.646+0000 7fb86c412740 10 client.26362 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:01.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.646+0000 7fb86c412740 1 -- 172.21.15.169:0/271871960 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ce12de30f0 con 0x55ce131af900 2024-03-28T01:02:01.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.646+0000 7fb8477f6640 1 -- 172.21.15.169:0/271871960 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb8540cd910 con 0x55ce131af900 2024-03-28T01:02:01.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.647+0000 7fb8477f6640 1 -- 172.21.15.169:0/271871960 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb85801b070 con 0x55ce131af900 2024-03-28T01:02:01.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.647+0000 7fb86c412740 10 client.26362 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:01.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.647+0000 7fb86c412740 10 client.26362 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:02:01.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.647+0000 7fb86c412740 20 client.26362 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:01.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.647+0000 7fb86c412740 1 --2- 172.21.15.169:0/271871960 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ce131bfda0 0x55ce131c2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.647+0000 7fb86c412740 4 client.26362 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:02:01.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.647+0000 7fb86c412740 1 -- 172.21.15.169:0/271871960 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ce1311c910 con 0x55ce131bfda0 2024-03-28T01:02:01.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.648+0000 7fb862ffd640 1 --2- 172.21.15.169:0/271871960 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ce131bfda0 0x55ce131c2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.656 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.652+0000 7fb862ffd640 1 --2- 172.21.15.169:0/271871960 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ce131bfda0 0x55ce131c2200 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca0c1da91ab16d97 server_cookie=2a5c9808a090fab5 in_seq=0 out_seq=0 2024-03-28T01:02:01.656 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.652+0000 7fb8477f6640 10 client.26362 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:01.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.655+0000 7fb8477f6640 1 -- 172.21.15.169:0/271871960 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ce1311c910 con 0x55ce131bfda0 2024-03-28T01:02:01.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.655+0000 7fb8477f6640 10 client.26362 handle_command_reply: tid=0 2024-03-28T01:02:01.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.655+0000 7fb86c412740 1 client.26362 shutdown 2024-03-28T01:02:01.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.655+0000 7fb86c412740 1 -- 172.21.15.169:0/271871960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131af900 msgr2=0x55ce131bb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:01.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.655+0000 7fb86c412740 1 --2- 172.21.15.169:0/271871960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131af900 0x55ce131bb420 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb86406b390 tx=0x7fb85800faa0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.655+0000 7fb86c412740 1 -- 172.21.15.169:0/271871960 shutdown_connections 2024-03-28T01:02:01.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.655+0000 7fb86c412740 1 --2- 172.21.15.169:0/271871960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131af900 0x55ce131bb420 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.655+0000 7fb86c412740 1 --2- 172.21.15.169:0/271871960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131aa790 0x55ce131b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.655+0000 7fb86c412740 1 --2- 172.21.15.169:0/271871960 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ce131bfda0 0x55ce131c2200 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.655+0000 7fb86c412740 1 --2- 172.21.15.169:0/271871960 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce131a9da0 0x55ce131b3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.655+0000 7fb86c412740 1 -- 172.21.15.169:0/271871960 >> 172.21.15.169:0/271871960 conn(0x55ce131a5310 msgr2=0x55ce131a4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:01.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.655+0000 7fb8477f6640 0 client.26362 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:01.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.655+0000 7fb86c412740 1 -- 172.21.15.169:0/271871960 shutdown_connections 2024-03-28T01:02:01.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.655+0000 7fb86c412740 1 -- 172.21.15.169:0/271871960 wait complete. 2024-03-28T01:02:01.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.673+0000 7fb86c412740 20 client.26362 trim_cache size 0 max 16384 2024-03-28T01:02:01.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.682+0000 7f15c77c0740 1 Processor -- start 2024-03-28T01:02:01.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.682+0000 7f15c77c0740 1 -- start start 2024-03-28T01:02:01.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.682+0000 7f15c77c0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754326b510 0x557543274960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.682+0000 7f15c77c0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557543270680 0x557543264a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.682+0000 7f15c77c0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55754327a4f0 0x557543264f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.682+0000 7f15c77c0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a0012080 con 0x55754327a4f0 2024-03-28T01:02:01.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.682+0000 7f15c77c0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a000fb80 con 0x557543270680 2024-03-28T01:02:01.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.682+0000 7f15c77c0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a000dc20 con 0x55754326b510 2024-03-28T01:02:01.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.683+0000 7f15be59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557543270680 0x557543264a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.683+0000 7f15be59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557543270680 0x557543264a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49518/0 (socket says 172.21.15.169:49518) 2024-03-28T01:02:01.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.683+0000 7f15be59c640 1 -- 172.21.15.169:0/328677824 learned_addr learned my addr 172.21.15.169:0/328677824 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:01.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.683+0000 7f15bdd9b640 1 --2- 172.21.15.169:0/328677824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754326b510 0x557543274960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.687 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.683+0000 7f15bdd9b640 1 -- 172.21.15.169:0/328677824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557543270680 msgr2=0x557543264a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:01.687 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.683+0000 7f15bdd9b640 1 --2- 172.21.15.169:0/328677824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557543270680 0x557543264a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.687 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.683+0000 7f15bdd9b640 1 -- 172.21.15.169:0/328677824 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55754327a4f0 msgr2=0x557543264f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:01.687 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.683+0000 7f15bdd9b640 1 --2- 172.21.15.169:0/328677824 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55754327a4f0 0x557543264f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.687 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.683+0000 7f15bdd9b640 1 -- 172.21.15.169:0/328677824 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557543266e00 con 0x55754326b510 2024-03-28T01:02:01.687 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.683+0000 7f15bdd9b640 1 --2- 172.21.15.169:0/328677824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754326b510 0x557543274960 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55754326ba20 tx=0x7f15b8000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4240de395878b208 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:01.689 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.686+0000 7f15a7fff640 1 -- 172.21.15.169:0/328677824 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b8051310 con 0x55754326b510 2024-03-28T01:02:01.689 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.686+0000 7f15a7fff640 1 -- 172.21.15.169:0/328677824 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15b8056070 con 0x55754326b510 2024-03-28T01:02:01.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.687+0000 7f15c77c0740 1 -- 172.21.15.169:0/328677824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754326b510 msgr2=0x557543274960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:01.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.687+0000 7f15c77c0740 1 --2- 172.21.15.169:0/328677824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754326b510 0x557543274960 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55754326ba20 tx=0x7f15b8000f40 comp rx=0 tx=0).stop 2024-03-28T01:02:01.692 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.690+0000 7f15c77c0740 1 -- 172.21.15.169:0/328677824 shutdown_connections 2024-03-28T01:02:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.690+0000 7f15c77c0740 1 --2- 172.21.15.169:0/328677824 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55754327a4f0 0x557543264f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.690+0000 7f15c77c0740 1 --2- 172.21.15.169:0/328677824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557543270680 0x557543264a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.690+0000 7f15c77c0740 1 --2- 172.21.15.169:0/328677824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754326b510 0x557543274960 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.690+0000 7f15c77c0740 1 -- 172.21.15.169:0/328677824 >> 172.21.15.169:0/328677824 conn(0x557543266090 msgr2=0x557543274250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.690+0000 7f15c77c0740 1 -- 172.21.15.169:0/328677824 shutdown_connections 2024-03-28T01:02:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.690+0000 7f15c77c0740 1 -- 172.21.15.169:0/328677824 wait complete. 2024-03-28T01:02:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.690+0000 7f15c77c0740 1 -- start start 2024-03-28T01:02:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.691+0000 7f15c77c0740 1 Processor -- start 2024-03-28T01:02:01.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.691+0000 7f15c77c0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55754326b510 0x5575432936d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.691+0000 7f15c77c0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557543270680 0x557543291ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.691+0000 7f15c77c0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754327a4f0 0x557543292220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.691+0000 7f15c77c0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a000e220 con 0x55754326b510 2024-03-28T01:02:01.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.691+0000 7f15c77c0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a000f520 con 0x557543270680 2024-03-28T01:02:01.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.691+0000 7f15c77c0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a000e860 con 0x55754327a4f0 2024-03-28T01:02:01.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.691+0000 7f15c77c0740 10 client.0 _ll_register_callbacks cb 0x7f15b4009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:01.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.691+0000 7f15bdd9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754327a4f0 0x557543292220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.691+0000 7f15bdd9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754327a4f0 0x557543292220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56204/0 (socket says 172.21.15.169:56204) 2024-03-28T01:02:01.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.691+0000 7f15be59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557543270680 0x557543291ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.691+0000 7f15bdd9b640 1 -- 172.21.15.169:0/3319642938 learned_addr learned my addr 172.21.15.169:0/3319642938 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:01.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.692+0000 7f15bdd9b640 1 -- 172.21.15.169:0/3319642938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557543270680 msgr2=0x557543291ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:01.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.693+0000 7f15bdd9b640 1 --2- 172.21.15.169:0/3319642938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557543270680 0x557543291ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.693+0000 7f15bdd9b640 1 -- 172.21.15.169:0/3319642938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55754326b510 msgr2=0x5575432936d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:01.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.693+0000 7f15bdd9b640 1 --2- 172.21.15.169:0/3319642938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55754326b510 0x5575432936d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.693+0000 7f15bdd9b640 1 -- 172.21.15.169:0/3319642938 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557543275cb0 con 0x55754327a4f0 2024-03-28T01:02:01.701 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.699+0000 7f15bdd9b640 1 --2- 172.21.15.169:0/3319642938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754327a4f0 0x557543292220 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x557543267030 tx=0x7f15b80307a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7cf98f7dd4be96c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.700+0000 7f15a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:02:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.700+0000 7f15a5ffb640 1 -- 172.21.15.169:0/3319642938 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b8051a00 con 0x55754327a4f0 2024-03-28T01:02:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.700+0000 7f15a5ffb640 1 -- 172.21.15.169:0/3319642938 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f15b8051ba0 con 0x55754327a4f0 2024-03-28T01:02:01.704 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.702+0000 7f15c77c0740 1 -- 172.21.15.169:0/3319642938 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f15b8056070 con 0x55754327a4f0 2024-03-28T01:02:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.706+0000 7f15a5ffb640 1 -- 172.21.15.169:0/3319642938 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15b8051da0 con 0x55754327a4f0 2024-03-28T01:02:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.706+0000 7f15c77c0740 10 client.26368 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.706+0000 7f15c77c0740 1 -- 172.21.15.169:0/3319642938 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5575432dcde0 con 0x55754327a4f0 2024-03-28T01:02:01.716 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.709+0000 7f15a5ffb640 1 -- 172.21.15.169:0/3319642938 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15b8056070 con 0x55754327a4f0 2024-03-28T01:02:01.716 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.709+0000 7f15a5ffb640 1 -- 172.21.15.169:0/3319642938 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f15b804f640 con 0x55754327a4f0 2024-03-28T01:02:01.716 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.709+0000 7f15c77c0740 10 client.26368 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.709+0000 7f15c77c0740 10 client.26368 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:02:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.709+0000 7f15c77c0740 20 client.26368 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.709+0000 7f15c77c0740 1 --2- 172.21.15.169:0/3319642938 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5575432dd720 0x5575432dfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.709+0000 7f15c77c0740 4 client.26368 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:02:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.709+0000 7f15c77c0740 1 -- 172.21.15.169:0/3319642938 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5575431d73a0 con 0x5575432dd720 2024-03-28T01:02:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.715+0000 7f15be59c640 1 --2- 172.21.15.169:0/3319642938 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5575432dd720 0x5575432dfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.717+0000 7f15be59c640 1 --2- 172.21.15.169:0/3319642938 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5575432dd720 0x5575432dfb40 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1b1138dfa351b19 server_cookie=7d6ffda6912d9030 in_seq=0 out_seq=0 2024-03-28T01:02:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:01.717+0000 7f15a5ffb640 10 client.26368 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:01.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.720+0000 7fb86c412740 1 Processor -- start 2024-03-28T01:02:01.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.720+0000 7fb86c412740 1 -- start start 2024-03-28T01:02:01.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.720+0000 7fb86c412740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131a9da0 0x55ce131b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.720+0000 7fb86c412740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce131aa790 0x55ce131b4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.720+0000 7fb86c412740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131af900 0x55ce131bb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.720+0000 7fb86c412740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8641b3240 con 0x55ce131aa790 2024-03-28T01:02:01.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.720+0000 7fb86c412740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8641b0d40 con 0x55ce131af900 2024-03-28T01:02:01.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.720+0000 7fb86c412740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8641aee90 con 0x55ce131a9da0 2024-03-28T01:02:01.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.721+0000 7fb8637fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131af900 0x55ce131bb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.721+0000 7fb8637fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131af900 0x55ce131bb3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49530/0 (socket says 172.21.15.169:49530) 2024-03-28T01:02:01.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.721+0000 7fb8637fe640 1 -- 172.21.15.169:0/2477269654 learned_addr learned my addr 172.21.15.169:0/2477269654 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:01.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.721+0000 7fb862ffd640 1 --2- 172.21.15.169:0/2477269654 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131a9da0 0x55ce131b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.721+0000 7fb8627fc640 1 --2- 172.21.15.169:0/2477269654 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce131aa790 0x55ce131b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.722+0000 7fb8627fc640 1 -- 172.21.15.169:0/2477269654 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131a9da0 msgr2=0x55ce131b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:01.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.722+0000 7fb8627fc640 1 --2- 172.21.15.169:0/2477269654 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131a9da0 0x55ce131b3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.722+0000 7fb8627fc640 1 -- 172.21.15.169:0/2477269654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131af900 msgr2=0x55ce131bb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:01.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.722+0000 7fb8627fc640 1 --2- 172.21.15.169:0/2477269654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131af900 0x55ce131bb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.722+0000 7fb8627fc640 1 -- 172.21.15.169:0/2477269654 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce131a0d80 con 0x55ce131aa790 2024-03-28T01:02:01.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.724+0000 7fb8627fc640 1 --2- 172.21.15.169:0/2477269654 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce131aa790 0x55ce131b4450 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb85c009e70 tx=0x7fb85c009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e39bac527452960 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:01.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.729+0000 7fb8497fa640 1 -- 172.21.15.169:0/2477269654 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c01a020 con 0x55ce131aa790 2024-03-28T01:02:01.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.729+0000 7fb8497fa640 1 -- 172.21.15.169:0/2477269654 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb85c016d60 con 0x55ce131aa790 2024-03-28T01:02:01.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.730+0000 7fb86c412740 1 -- 172.21.15.169:0/2477269654 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce131aa790 msgr2=0x55ce131b4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:01.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.730+0000 7fb86c412740 1 --2- 172.21.15.169:0/2477269654 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce131aa790 0x55ce131b4450 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb85c009e70 tx=0x7fb85c009650 comp rx=0 tx=0).stop 2024-03-28T01:02:01.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.734+0000 7fb86c412740 1 -- 172.21.15.169:0/2477269654 shutdown_connections 2024-03-28T01:02:01.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.734+0000 7fb86c412740 1 --2- 172.21.15.169:0/2477269654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131af900 0x55ce131bb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.734+0000 7fb86c412740 1 --2- 172.21.15.169:0/2477269654 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce131aa790 0x55ce131b4450 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.734+0000 7fb86c412740 1 --2- 172.21.15.169:0/2477269654 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131a9da0 0x55ce131b3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.734+0000 7fb86c412740 1 -- 172.21.15.169:0/2477269654 >> 172.21.15.169:0/2477269654 conn(0x55ce131a5310 msgr2=0x55ce131a0390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:01.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.738+0000 7fb86c412740 1 -- 172.21.15.169:0/2477269654 shutdown_connections 2024-03-28T01:02:01.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.739+0000 7fb86c412740 1 -- 172.21.15.169:0/2477269654 wait complete. 2024-03-28T01:02:01.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.739+0000 7fb86c412740 1 -- start start 2024-03-28T01:02:01.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.739+0000 7fb86c412740 1 Processor -- start 2024-03-28T01:02:01.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.740+0000 7fb86c412740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce131a9da0 0x55ce131b8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.740+0000 7fb86c412740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131aa790 0x55ce131b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.740+0000 7fb86c412740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131af900 0x55ce132196c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.740+0000 7fb862ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce131a9da0 0x55ce131b8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.741+0000 7fb862ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce131a9da0 0x55ce131b8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:54878/0 (socket says 172.21.15.169:54878) 2024-03-28T01:02:01.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.741+0000 7fb8627fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131aa790 0x55ce131b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.741+0000 7fb8627fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131aa790 0x55ce131b8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56224/0 (socket says 172.21.15.169:56224) 2024-03-28T01:02:01.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.741+0000 7fb862ffd640 1 -- 172.21.15.169:0/112157051 learned_addr learned my addr 172.21.15.169:0/112157051 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:01.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.741+0000 7fb8637fe640 1 --2- 172.21.15.169:0/112157051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131af900 0x55ce132196c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.741+0000 7fb86c412740 1 -- 172.21.15.169:0/112157051 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8641af400 con 0x55ce131a9da0 2024-03-28T01:02:01.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.741+0000 7fb86c412740 1 -- 172.21.15.169:0/112157051 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8641b06e0 con 0x55ce131af900 2024-03-28T01:02:01.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.741+0000 7fb86c412740 1 -- 172.21.15.169:0/112157051 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8641afa20 con 0x55ce131aa790 2024-03-28T01:02:01.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.741+0000 7fb86c412740 10 client.0 _ll_register_callbacks cb 0x7fb85800a3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:01.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.743+0000 7fb862ffd640 1 -- 172.21.15.169:0/112157051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131aa790 msgr2=0x55ce131b8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:01.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.743+0000 7fb862ffd640 1 --2- 172.21.15.169:0/112157051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131aa790 0x55ce131b8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.743+0000 7fb862ffd640 1 -- 172.21.15.169:0/112157051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131af900 msgr2=0x55ce132196c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:01.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.743+0000 7fb862ffd640 1 --2- 172.21.15.169:0/112157051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131af900 0x55ce132196c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:01.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.743+0000 7fb862ffd640 1 -- 172.21.15.169:0/112157051 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ce13219cb0 con 0x55ce131a9da0 2024-03-28T01:02:01.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.744+0000 7fb862ffd640 1 --2- 172.21.15.169:0/112157051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce131a9da0 0x55ce131b8520 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb8540058c0 tx=0x7fb854000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=186dcf3e40cb3e47 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:01.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.744+0000 7fb8477f6640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:02:01.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.744+0000 7fb8477f6640 1 -- 172.21.15.169:0/112157051 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb854110040 con 0x55ce131a9da0 2024-03-28T01:02:01.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.745+0000 7fb8477f6640 1 -- 172.21.15.169:0/112157051 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb8540cb510 con 0x55ce131a9da0 2024-03-28T01:02:01.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.745+0000 7fb86c412740 1 -- 172.21.15.169:0/112157051 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb85c016d60 con 0x55ce131a9da0 2024-03-28T01:02:01.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.745+0000 7fb8477f6640 1 -- 172.21.15.169:0/112157051 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb8540047a0 con 0x55ce131a9da0 2024-03-28T01:02:01.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.745+0000 7fb86c412740 10 client.16563 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:01.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.745+0000 7fb86c412740 1 -- 172.21.15.169:0/112157051 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ce1321b030 con 0x55ce131a9da0 2024-03-28T01:02:01.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.749+0000 7fb8477f6640 1 -- 172.21.15.169:0/112157051 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb85c016d60 con 0x55ce131a9da0 2024-03-28T01:02:01.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.750+0000 7fb8477f6640 1 -- 172.21.15.169:0/112157051 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb85404d070 con 0x55ce131a9da0 2024-03-28T01:02:01.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.750+0000 7fb86c412740 10 client.16563 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:01.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.750+0000 7fb86c412740 10 client.16563 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:02:01.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.750+0000 7fb86c412740 20 client.16563 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:01.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.750+0000 7fb86c412740 1 --2- 172.21.15.169:0/112157051 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ce1321ba60 0x55ce1321de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:01.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.750+0000 7fb86c412740 4 client.16563 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:02:01.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.750+0000 7fb86c412740 1 -- 172.21.15.169:0/112157051 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ce13116c60 con 0x55ce1321ba60 2024-03-28T01:02:01.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.752+0000 7fb8627fc640 1 --2- 172.21.15.169:0/112157051 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ce1321ba60 0x55ce1321de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:01.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.761+0000 7fb8627fc640 1 --2- 172.21.15.169:0/112157051 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ce1321ba60 0x55ce1321de80 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5c0274b5da9c5f2 server_cookie=68806447686f469b in_seq=0 out_seq=0 2024-03-28T01:02:01.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:01.764+0000 7fb8477f6640 10 client.16563 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:01.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:01 smithi169 ceph-mon[32274]: pgmap v516: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 672 MiB/s wr, 1.42k op/s 2024-03-28T01:02:01.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:01 smithi174 ceph-mon[36060]: pgmap v516: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 672 MiB/s wr, 1.42k op/s 2024-03-28T01:02:01.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:01 smithi177 ceph-mon[36206]: pgmap v516: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 672 MiB/s wr, 1.42k op/s 2024-03-28T01:02:02.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:02 smithi169 ceph-mon[32274]: scrub queued for path: / 2024-03-28T01:02:02.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:02 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:02:02.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:02 smithi169 ceph-mon[32274]: scrub summary: active paths [/] 2024-03-28T01:02:02.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:02 smithi174 ceph-mon[36060]: scrub queued for path: / 2024-03-28T01:02:02.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:02 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:02:02.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:02 smithi174 ceph-mon[36060]: scrub summary: active paths [/] 2024-03-28T01:02:02.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:02 smithi177 ceph-mon[36206]: scrub queued for path: / 2024-03-28T01:02:02.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:02 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:02:02.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:02 smithi177 ceph-mon[36206]: scrub summary: active paths [/] 2024-03-28T01:02:03.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:03 smithi169 ceph-mon[32274]: pgmap v517: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 821 MiB/s wr, 1.65k op/s 2024-03-28T01:02:03.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:03 smithi174 ceph-mon[36060]: pgmap v517: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 821 MiB/s wr, 1.65k op/s 2024-03-28T01:02:03.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:03 smithi177 ceph-mon[36206]: pgmap v517: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 821 MiB/s wr, 1.65k op/s 2024-03-28T01:02:05.090 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.086+0000 7f15a5ffb640 1 -- 172.21.15.169:0/3319642938 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5575431d73a0 con 0x5575432dd720 2024-03-28T01:02:05.090 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.086+0000 7f15a5ffb640 10 client.26368 handle_command_reply: tid=0 2024-03-28T01:02:05.090 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.086+0000 7f15c77c0740 1 client.26368 shutdown 2024-03-28T01:02:05.090 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.087+0000 7f15c77c0740 1 -- 172.21.15.169:0/3319642938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754327a4f0 msgr2=0x557543292220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:05.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.087+0000 7f15c77c0740 1 --2- 172.21.15.169:0/3319642938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754327a4f0 0x557543292220 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x557543267030 tx=0x7f15b80307a0 comp rx=0 tx=0).stop 2024-03-28T01:02:05.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.087+0000 7f15c77c0740 1 -- 172.21.15.169:0/3319642938 shutdown_connections 2024-03-28T01:02:05.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.087+0000 7f15c77c0740 1 --2- 172.21.15.169:0/3319642938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55754327a4f0 0x557543292220 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:05.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.087+0000 7f15c77c0740 1 --2- 172.21.15.169:0/3319642938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557543270680 0x557543291ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:05.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.087+0000 7f15c77c0740 1 --2- 172.21.15.169:0/3319642938 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5575432dd720 0x5575432dfb40 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:05.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.087+0000 7f15c77c0740 1 --2- 172.21.15.169:0/3319642938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55754326b510 0x5575432936d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:05.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.087+0000 7f15c77c0740 1 -- 172.21.15.169:0/3319642938 >> 172.21.15.169:0/3319642938 conn(0x557543266090 msgr2=0x5575432692f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:05.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.087+0000 7f15a5ffb640 0 client.26368 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:05.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.087+0000 7f15c77c0740 1 -- 172.21.15.169:0/3319642938 shutdown_connections 2024-03-28T01:02:05.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.087+0000 7f15c77c0740 1 -- 172.21.15.169:0/3319642938 wait complete. 2024-03-28T01:02:05.091 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.087+0000 7f15c77c0740 20 client.26368 trim_cache size 0 max 16384 2024-03-28T01:02:05.092 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:02:05.092 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:02:05.092 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:02:05.092 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:02:05.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.088+0000 7fb8477f6640 1 -- 172.21.15.169:0/112157051 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ce13116c60 con 0x55ce1321ba60 2024-03-28T01:02:05.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.088+0000 7fb8477f6640 10 client.16563 handle_command_reply: tid=0 2024-03-28T01:02:05.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.088+0000 7fb86c412740 1 client.16563 shutdown 2024-03-28T01:02:05.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.089+0000 7fb86c412740 1 -- 172.21.15.169:0/112157051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce131a9da0 msgr2=0x55ce131b8520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:05.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.089+0000 7fb86c412740 1 --2- 172.21.15.169:0/112157051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce131a9da0 0x55ce131b8520 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb8540058c0 tx=0x7fb854000bc0 comp rx=0 tx=0).stop 2024-03-28T01:02:05.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.089+0000 7fb86c412740 1 -- 172.21.15.169:0/112157051 shutdown_connections 2024-03-28T01:02:05.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.089+0000 7fb86c412740 1 --2- 172.21.15.169:0/112157051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce131af900 0x55ce132196c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:05.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.089+0000 7fb86c412740 1 --2- 172.21.15.169:0/112157051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce131aa790 0x55ce131b8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:05.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.089+0000 7fb86c412740 1 --2- 172.21.15.169:0/112157051 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ce1321ba60 0x55ce1321de80 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:05.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.089+0000 7fb86c412740 1 --2- 172.21.15.169:0/112157051 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce131a9da0 0x55ce131b8520 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:05.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.089+0000 7fb86c412740 1 -- 172.21.15.169:0/112157051 >> 172.21.15.169:0/112157051 conn(0x55ce131a5310 msgr2=0x55ce131b3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:05.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.089+0000 7fb8477f6640 0 client.16563 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:05.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.089+0000 7fb86c412740 1 -- 172.21.15.169:0/112157051 shutdown_connections 2024-03-28T01:02:05.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.090+0000 7fb86c412740 1 -- 172.21.15.169:0/112157051 wait complete. 2024-03-28T01:02:05.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.090+0000 7fb86c412740 20 client.16563 trim_cache size 0 max 16384 2024-03-28T01:02:05.094 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:02:05.094 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T01:02:05.094 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": { 2024-03-28T01:02:05.094 INFO:teuthology.orchestra.run.smithi169.stdout: "d53fe40b-7254-44b3-949f-5e2c733c431c": { 2024-03-28T01:02:05.094 INFO:teuthology.orchestra.run.smithi169.stdout: "path": "/", 2024-03-28T01:02:05.094 INFO:teuthology.orchestra.run.smithi169.stdout: "tag": "d53fe40b-7254-44b3-949f-5e2c733c431c", 2024-03-28T01:02:05.094 INFO:teuthology.orchestra.run.smithi169.stdout: "options": "recursive,force" 2024-03-28T01:02:05.094 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:02:05.094 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:02:05.094 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:02:05.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.092+0000 7fb869827640 1 -- 172.21.15.169:0/268831551 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb84c0a2a10 msgr2=0x7fb84c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:05.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.092+0000 7fb869827640 1 --2- 172.21.15.169:0/268831551 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb84c0a2a10 0x7fb84c0a4ed0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x55ce131aaff0 tx=0x7fb858005be0 comp rx=0 tx=0).stop 2024-03-28T01:02:05.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.092+0000 7fb869827640 1 -- 172.21.15.169:0/268831551 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb86410b010 msgr2=0x7fb86419e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:05.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.092+0000 7fb869827640 1 --2- 172.21.15.169:0/268831551 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb86410b010 0x7fb86419e210 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb85403d2b0 tx=0x7fb854004fa0 comp rx=0 tx=0).stop 2024-03-28T01:02:05.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.092+0000 7fb869827640 1 -- 172.21.15.169:0/268831551 shutdown_connections 2024-03-28T01:02:05.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.092+0000 7fb869827640 1 --2- 172.21.15.169:0/268831551 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb84c0a2a10 0x7fb84c0a4ed0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:05.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.092+0000 7fb869827640 1 --2- 172.21.15.169:0/268831551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8641a37b0 0x7fb8641a5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:05.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.092+0000 7fb869827640 1 --2- 172.21.15.169:0/268831551 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb86410d9c0 0x7fb86419e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:05.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.093+0000 7fb869827640 1 --2- 172.21.15.169:0/268831551 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb86410b010 0x7fb86419e210 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:05.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.093+0000 7fb869827640 1 -- 172.21.15.169:0/268831551 >> 172.21.15.169:0/268831551 conn(0x7fb86406e500 msgr2=0x7fb864108f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:05.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.094+0000 7fb869827640 1 -- 172.21.15.169:0/268831551 shutdown_connections 2024-03-28T01:02:05.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:05.094+0000 7fb869827640 1 -- 172.21.15.169:0/268831551 wait complete. 2024-03-28T01:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.100+0000 7f15bf59e640 1 -- 172.21.15.169:0/3751165343 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f15a80c4a10 msgr2=0x7f15a80c6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.100+0000 7f15bf59e640 1 --2- 172.21.15.169:0/3751165343 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f15a80c4a10 0x7f15a80c6ed0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f15b000be60 tx=0x7f15b00092b0 comp rx=0 tx=0).stop 2024-03-28T01:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.100+0000 7f15bf59e640 1 -- 172.21.15.169:0/3751165343 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15c0078c20 msgr2=0x7f15c01becf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.100+0000 7f15bf59e640 1 --2- 172.21.15.169:0/3751165343 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15c0078c20 0x7f15c01becf0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f15b000b4a0 tx=0x7f15b000b960 comp rx=0 tx=0).stop 2024-03-28T01:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.100+0000 7f15bf59e640 1 -- 172.21.15.169:0/3751165343 shutdown_connections 2024-03-28T01:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.100+0000 7f15bf59e640 1 --2- 172.21.15.169:0/3751165343 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f15a80c4a10 0x7f15a80c6ed0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.100+0000 7f15bf59e640 1 --2- 172.21.15.169:0/3751165343 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15c0078c20 0x7f15c01becf0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.100+0000 7f15bf59e640 1 --2- 172.21.15.169:0/3751165343 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15c0076270 0x7f15c01b8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.100+0000 7f15bf59e640 1 --2- 172.21.15.169:0/3751165343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15c0073390 0x7f15c01b7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.100+0000 7f15bf59e640 1 -- 172.21.15.169:0/3751165343 >> 172.21.15.169:0/3751165343 conn(0x7f15c006e510 msgr2=0x7f15c0079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.100+0000 7f15bf59e640 1 -- 172.21.15.169:0/3751165343 shutdown_connections 2024-03-28T01:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:05.100+0000 7f15bf59e640 1 -- 172.21.15.169:0/3751165343 wait complete. 2024-03-28T01:02:05.144 INFO:tasks.cephfs.filesystem:scrub status for tag:d53fe40b-7254-44b3-949f-5e2c733c431c - {'path': '/', 'tag': 'd53fe40b-7254-44b3-949f-5e2c733c431c', 'options': 'recursive,force'} 2024-03-28T01:02:05.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:05 smithi169 ceph-mon[32274]: pgmap v518: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 697 MiB/s wr, 1.08k op/s 2024-03-28T01:02:05.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:05 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:02:05.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:05 smithi174 ceph-mon[36060]: pgmap v518: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 697 MiB/s wr, 1.08k op/s 2024-03-28T01:02:05.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:05 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:02:05.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:05 smithi177 ceph-mon[36206]: pgmap v518: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 697 MiB/s wr, 1.08k op/s 2024-03-28T01:02:05.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:05 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:02:07.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:07 smithi169 ceph-mon[32274]: pgmap v519: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 817 MiB/s wr, 1.30k op/s 2024-03-28T01:02:07.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:07 smithi174 ceph-mon[36060]: pgmap v519: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 817 MiB/s wr, 1.30k op/s 2024-03-28T01:02:07.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:07 smithi177 ceph-mon[36206]: pgmap v519: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 817 MiB/s wr, 1.30k op/s 2024-03-28T01:02:09.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:09 smithi174 ceph-mon[36060]: pgmap v520: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 743 MiB/s wr, 1.30k op/s 2024-03-28T01:02:09.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:09 smithi174 ceph-mon[36060]: scrub summary: idle 2024-03-28T01:02:09.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:09 smithi177 ceph-mon[36206]: pgmap v520: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 743 MiB/s wr, 1.30k op/s 2024-03-28T01:02:09.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:09 smithi177 ceph-mon[36206]: scrub summary: idle 2024-03-28T01:02:10.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:09 smithi169 ceph-mon[32274]: pgmap v520: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 743 MiB/s wr, 1.30k op/s 2024-03-28T01:02:10.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:09 smithi169 ceph-mon[32274]: scrub summary: idle 2024-03-28T01:02:10.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:10 smithi177 ceph-mon[36206]: pgmap v521: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 684 MiB/s wr, 1.24k op/s 2024-03-28T01:02:11.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:10 smithi169 ceph-mon[32274]: pgmap v521: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 684 MiB/s wr, 1.24k op/s 2024-03-28T01:02:11.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:10 smithi174 ceph-mon[36060]: pgmap v521: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 684 MiB/s wr, 1.24k op/s 2024-03-28T01:02:13.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:13 smithi169 ceph-mon[32274]: pgmap v522: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 896 MiB/s wr, 1.61k op/s 2024-03-28T01:02:13.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:13 smithi174 ceph-mon[36060]: pgmap v522: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 896 MiB/s wr, 1.61k op/s 2024-03-28T01:02:13.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:13 smithi177 ceph-mon[36206]: pgmap v522: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 896 MiB/s wr, 1.61k op/s 2024-03-28T01:02:15.815 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:15 smithi169 ceph-mon[32274]: pgmap v523: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 759 MiB/s wr, 1.28k op/s 2024-03-28T01:02:15.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:15 smithi174 ceph-mon[36060]: pgmap v523: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 759 MiB/s wr, 1.28k op/s 2024-03-28T01:02:15.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:15 smithi177 ceph-mon[36206]: pgmap v523: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 759 MiB/s wr, 1.28k op/s 2024-03-28T01:02:17.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:17 smithi169 ceph-mon[32274]: pgmap v524: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 822 MiB/s wr, 1.39k op/s 2024-03-28T01:02:17.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:17 smithi174 ceph-mon[36060]: pgmap v524: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 822 MiB/s wr, 1.39k op/s 2024-03-28T01:02:17.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:17 smithi177 ceph-mon[36206]: pgmap v524: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 822 MiB/s wr, 1.39k op/s 2024-03-28T01:02:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.359+0000 7fc9b0285640 1 -- 172.21.15.169:0/1715270737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9a810b010 msgr2=0x7fc9a810d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.359+0000 7fc9b0285640 1 --2- 172.21.15.169:0/1715270737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9a810b010 0x7fc9a810d480 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4009860 tx=0x7fc9a402bb70 comp rx=0 tx=0).stop 2024-03-28T01:02:18.366 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.362+0000 7fc9b0285640 1 -- 172.21.15.169:0/1715270737 shutdown_connections 2024-03-28T01:02:18.366 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.362+0000 7fc9b0285640 1 --2- 172.21.15.169:0/1715270737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9a810d9c0 0x7fc9a810de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.367 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.362+0000 7fc9b0285640 1 --2- 172.21.15.169:0/1715270737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9a810b010 0x7fc9a810d480 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.367 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.362+0000 7fc9b0285640 1 --2- 172.21.15.169:0/1715270737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9a81086a0 0x7fc9a810aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.367 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.362+0000 7fc9b0285640 1 -- 172.21.15.169:0/1715270737 >> 172.21.15.169:0/1715270737 conn(0x7fc9a806e6e0 msgr2=0x7fc9a8070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:18.371 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.366+0000 7fc9b0285640 1 -- 172.21.15.169:0/1715270737 shutdown_connections 2024-03-28T01:02:18.371 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.367+0000 7fc9b0285640 1 -- 172.21.15.169:0/1715270737 wait complete. 2024-03-28T01:02:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.368+0000 7fc9b0285640 1 Processor -- start 2024-03-28T01:02:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.368+0000 7fc9b0285640 1 -- start start 2024-03-28T01:02:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.368+0000 7fc9b0285640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9a81086a0 0x7fc9a819df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.368+0000 7fc9b0285640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9a810b010 0x7fc9a819e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.368+0000 7fc9b0285640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9a810d9c0 0x7fc9a81a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.368+0000 7fc9b0285640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a8111970 con 0x7fc9a810b010 2024-03-28T01:02:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.368+0000 7fc9b0285640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a81117f0 con 0x7fc9a81086a0 2024-03-28T01:02:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.368+0000 7fc9b0285640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a8111af0 con 0x7fc9a810d9c0 2024-03-28T01:02:18.376 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.372+0000 7fc9ae021640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9a81086a0 0x7fc9a819df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:18.376 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.372+0000 7fc9ae021640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9a81086a0 0x7fc9a819df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:44432/0 (socket says 172.21.15.169:44432) 2024-03-28T01:02:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.372+0000 7fc9ae021640 1 -- 172.21.15.169:0/826716607 learned_addr learned my addr 172.21.15.169:0/826716607 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.372+0000 7fc9ae822640 1 --2- 172.21.15.169:0/826716607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9a810d9c0 0x7fc9a81a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.372+0000 7fc9ad820640 1 --2- 172.21.15.169:0/826716607 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9a810b010 0x7fc9a819e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.372+0000 7fc9ad820640 1 -- 172.21.15.169:0/826716607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9a810d9c0 msgr2=0x7fc9a81a54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.372+0000 7fc9ad820640 1 --2- 172.21.15.169:0/826716607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9a810d9c0 0x7fc9a81a54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.372+0000 7fc9ad820640 1 -- 172.21.15.169:0/826716607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9a81086a0 msgr2=0x7fc9a819df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.372+0000 7fc9ad820640 1 --2- 172.21.15.169:0/826716607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9a81086a0 0x7fc9a819df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.372+0000 7fc9ad820640 1 -- 172.21.15.169:0/826716607 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a81a5b20 con 0x7fc9a810b010 2024-03-28T01:02:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.373+0000 7fc9ad820640 1 --2- 172.21.15.169:0/826716607 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9a810b010 0x7fc9a819e440 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4009780 tx=0x7fc9a4032440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.374+0000 7fc99f7fe640 1 -- 172.21.15.169:0/826716607 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a4039040 con 0x7fc9a810b010 2024-03-28T01:02:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.374+0000 7fc9b0285640 1 -- 172.21.15.169:0/826716607 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a81a5db0 con 0x7fc9a810b010 2024-03-28T01:02:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.374+0000 7fc9b0285640 1 -- 172.21.15.169:0/826716607 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a81a6260 con 0x7fc9a810b010 2024-03-28T01:02:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.374+0000 7fc99f7fe640 1 -- 172.21.15.169:0/826716607 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc9a402c9f0 con 0x7fc9a810b010 2024-03-28T01:02:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.374+0000 7fc99f7fe640 1 -- 172.21.15.169:0/826716607 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a40046c0 con 0x7fc9a810b010 2024-03-28T01:02:18.386 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.382+0000 7fc9b2e73740 1 Processor -- start 2024-03-28T01:02:18.386 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.383+0000 7fc9b2e73740 1 -- start start 2024-03-28T01:02:18.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.383+0000 7fc9b2e73740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557062c0b920 0x557062c0bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:18.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.383+0000 7fc9b2e73740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c0c310 0x557062c10e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:18.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.383+0000 7fc9b2e73740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557062c11480 0x557062c13910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:18.387 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.383+0000 7fc9b2e73740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x557062b70b10 con 0x557062c11480 2024-03-28T01:02:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.383+0000 7fc9b2e73740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x557062b19900 con 0x557062c0c310 2024-03-28T01:02:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.383+0000 7fc9b2e73740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557062b6cbb0 con 0x557062c0b920 2024-03-28T01:02:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.384+0000 7fc99f7fe640 1 -- 172.21.15.169:0/826716607 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc9a4044020 con 0x7fc9a810b010 2024-03-28T01:02:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.386+0000 7fc99f7fe640 1 --2- 172.21.15.169:0/826716607 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc9940a29f0 0x7fc9940a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:18.389 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.386+0000 7fc99f7fe640 1 -- 172.21.15.169:0/826716607 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc9a40efe10 con 0x7fc9a810b010 2024-03-28T01:02:18.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.386+0000 7fc9ae021640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c0c310 0x557062c10e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:18.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.386+0000 7fc9ae021640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c0c310 0x557062c10e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:44440/0 (socket says 172.21.15.169:44440) 2024-03-28T01:02:18.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.386+0000 7fc9ae021640 1 -- 172.21.15.169:0/3927011976 learned_addr learned my addr 172.21.15.169:0/3927011976 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:18.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.386+0000 7fc9ae822640 1 --2- 172.21.15.169:0/3927011976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557062c0b920 0x557062c0bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:18.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.387+0000 7fc9ae021640 1 -- 172.21.15.169:0/3927011976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557062c0b920 msgr2=0x557062c0bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:18.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.387+0000 7fc9ae021640 1 --2- 172.21.15.169:0/3927011976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557062c0b920 0x557062c0bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.387+0000 7fc9ae021640 1 -- 172.21.15.169:0/3927011976 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557062c11480 msgr2=0x557062c13910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:18.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.387+0000 7fc9ae021640 1 --2- 172.21.15.169:0/3927011976 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557062c11480 0x557062c13910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.387+0000 7fc9ae021640 1 -- 172.21.15.169:0/3927011976 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557062885210 con 0x557062c0c310 2024-03-28T01:02:18.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.387+0000 7fc9ae021640 1 --2- 172.21.15.169:0/3927011976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c0c310 0x557062c10e10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc998009d20 tx=0x7fc9980117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b232994be714f7f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:18.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.388+0000 7fc99cbfe640 1 -- 172.21.15.169:0/3927011976 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc998012650 con 0x557062c0c310 2024-03-28T01:02:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.388+0000 7fc99cbfe640 1 -- 172.21.15.169:0/3927011976 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc998009e10 con 0x557062c0c310 2024-03-28T01:02:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.389+0000 7fc9b2e73740 1 -- 172.21.15.169:0/3927011976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c0c310 msgr2=0x557062c10e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.390+0000 7fc9b2e73740 1 --2- 172.21.15.169:0/3927011976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c0c310 0x557062c10e10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc998009d20 tx=0x7fc9980117b0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.390+0000 7fc9b2e73740 1 -- 172.21.15.169:0/3927011976 shutdown_connections 2024-03-28T01:02:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.390+0000 7fc9b2e73740 1 --2- 172.21.15.169:0/3927011976 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557062c11480 0x557062c13910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.390+0000 7fc9b2e73740 1 --2- 172.21.15.169:0/3927011976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c0c310 0x557062c10e10 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.390+0000 7fc9b2e73740 1 --2- 172.21.15.169:0/3927011976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557062c0b920 0x557062c0bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.390+0000 7fc9b2e73740 1 -- 172.21.15.169:0/3927011976 >> 172.21.15.169:0/3927011976 conn(0x557062c06e90 msgr2=0x557062c07290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.391+0000 7fc9ae822640 1 --2- 172.21.15.169:0/826716607 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc9940a29f0 0x7fc9940a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.392+0000 7fc9ae822640 1 --2- 172.21.15.169:0/826716607 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc9940a29f0 0x7fc9940a4eb0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x557062c0cb70 tx=0x7fc9a0022040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.393+0000 7fc9b2e73740 1 -- 172.21.15.169:0/3927011976 shutdown_connections 2024-03-28T01:02:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.395+0000 7fc9b2e73740 1 -- 172.21.15.169:0/3927011976 wait complete. 2024-03-28T01:02:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.395+0000 7fc9b2e73740 1 -- start start 2024-03-28T01:02:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.395+0000 7fc9b2e73740 1 Processor -- start 2024-03-28T01:02:18.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.398+0000 7fc9b2e73740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557062c0b920 0x557062c15ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:18.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.398+0000 7fc9b2e73740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557062c0c310 0x557062c16410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:18.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.398+0000 7fc9b2e73740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c11480 0x557062c1d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:18.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.398+0000 7fc9b2e73740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x557062c14480 con 0x557062c0b920 2024-03-28T01:02:18.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.398+0000 7fc9b2e73740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x557062b2cd50 con 0x557062c11480 2024-03-28T01:02:18.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.398+0000 7fc9b2e73740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557062c14600 con 0x557062c0c310 2024-03-28T01:02:18.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.398+0000 7fc9b2e73740 10 client.0 _ll_register_callbacks cb 0x557062b2c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.399+0000 7fc9ad820640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557062c0c310 0x557062c16410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.399+0000 7fc9ae822640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c11480 0x557062c1d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.399+0000 7fc9ae822640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c11480 0x557062c1d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:44446/0 (socket says 172.21.15.169:44446) 2024-03-28T01:02:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.399+0000 7fc9ae822640 1 -- 172.21.15.169:0/3984875293 learned_addr learned my addr 172.21.15.169:0/3984875293 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.400+0000 7fc9ae822640 1 -- 172.21.15.169:0/3984875293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557062c0c310 msgr2=0x557062c16410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.400+0000 7fc9ae822640 1 --2- 172.21.15.169:0/3984875293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557062c0c310 0x557062c16410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.400+0000 7fc9ae822640 1 -- 172.21.15.169:0/3984875293 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557062c0b920 msgr2=0x557062c15ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:18.404 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.400+0000 7fc9ae822640 1 --2- 172.21.15.169:0/3984875293 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557062c0b920 0x557062c15ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.404 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.400+0000 7fc9ae822640 1 -- 172.21.15.169:0/3984875293 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557062951620 con 0x557062c11480 2024-03-28T01:02:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.402+0000 7fc9ae822640 1 --2- 172.21.15.169:0/3984875293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c11480 0x557062c1d450 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc9a002faf0 tx=0x7fc9a001a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa9134669bcc5547 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.403+0000 7fc991ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:02:18.410 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.407+0000 7fc991ffb640 1 -- 172.21.15.169:0/3984875293 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a001eb00 con 0x557062c11480 2024-03-28T01:02:18.410 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.408+0000 7fc991ffb640 1 -- 172.21.15.169:0/3984875293 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc9a0019040 con 0x557062c11480 2024-03-28T01:02:18.411 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.408+0000 7fc9b2e73740 1 -- 172.21.15.169:0/3984875293 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557062b7e050 con 0x557062c11480 2024-03-28T01:02:18.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.412+0000 7fc991ffb640 1 -- 172.21.15.169:0/3984875293 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9a001eca0 con 0x557062c11480 2024-03-28T01:02:18.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.412+0000 7fc9b2e73740 10 client.26456 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:18.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.413+0000 7fc9b2e73740 1 -- 172.21.15.169:0/3984875293 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557062c208c0 con 0x557062c11480 2024-03-28T01:02:18.418 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.415+0000 7fc991ffb640 1 -- 172.21.15.169:0/3984875293 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557062b7e050 con 0x557062c11480 2024-03-28T01:02:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.416+0000 7fc991ffb640 1 -- 172.21.15.169:0/3984875293 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fc9a0036070 con 0x557062c11480 2024-03-28T01:02:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.416+0000 7fc9b2e73740 10 client.26456 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.416+0000 7fc9b2e73740 10 client.26456 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:02:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.416+0000 7fc9b2e73740 20 client.26456 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.416+0000 7fc9b2e73740 1 --2- 172.21.15.169:0/3984875293 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557062c22370 0x557062c247f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.416+0000 7fc9b2e73740 4 client.26456 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:02:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.416+0000 7fc9b2e73740 1 -- 172.21.15.169:0/3984875293 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557062b783a0 con 0x557062c22370 2024-03-28T01:02:18.420 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.418+0000 7fc9ae021640 1 --2- 172.21.15.169:0/3984875293 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557062c22370 0x557062c247f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.421+0000 7fc9ae021640 1 --2- 172.21.15.169:0/3984875293 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557062c22370 0x557062c247f0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e08a22e417cc7a5d server_cookie=d54e6466460ce46b in_seq=0 out_seq=0 2024-03-28T01:02:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.421+0000 7fc991ffb640 10 client.26456 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.425+0000 7fc991ffb640 1 -- 172.21.15.169:0/3984875293 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557062b783a0 con 0x557062c22370 2024-03-28T01:02:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.425+0000 7fc991ffb640 10 client.26456 handle_command_reply: tid=0 2024-03-28T01:02:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.425+0000 7fc9b2e73740 1 client.26456 shutdown 2024-03-28T01:02:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.427+0000 7fc9b2e73740 1 -- 172.21.15.169:0/3984875293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c11480 msgr2=0x557062c1d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.427+0000 7fc9b2e73740 1 --2- 172.21.15.169:0/3984875293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c11480 0x557062c1d450 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc9a002faf0 tx=0x7fc9a001a5d0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.428+0000 7fc9b2e73740 1 -- 172.21.15.169:0/3984875293 shutdown_connections 2024-03-28T01:02:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.428+0000 7fc9b2e73740 1 --2- 172.21.15.169:0/3984875293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c11480 0x557062c1d450 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.428+0000 7fc9b2e73740 1 --2- 172.21.15.169:0/3984875293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557062c0c310 0x557062c16410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.428+0000 7fc9b2e73740 1 --2- 172.21.15.169:0/3984875293 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557062c22370 0x557062c247f0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.428+0000 7fc9b2e73740 1 --2- 172.21.15.169:0/3984875293 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557062c0b920 0x557062c15ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.428+0000 7fc9b2e73740 1 -- 172.21.15.169:0/3984875293 >> 172.21.15.169:0/3984875293 conn(0x557062c06e90 msgr2=0x557062c06020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.428+0000 7fc991ffb640 0 client.26456 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.428+0000 7fc9b2e73740 1 -- 172.21.15.169:0/3984875293 shutdown_connections 2024-03-28T01:02:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.428+0000 7fc9b2e73740 1 -- 172.21.15.169:0/3984875293 wait complete. 2024-03-28T01:02:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.428+0000 7fc9b2e73740 20 client.26456 trim_cache size 0 max 16384 2024-03-28T01:02:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.476+0000 7fc9b2e73740 1 Processor -- start 2024-03-28T01:02:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.476+0000 7fc9b2e73740 1 -- start start 2024-03-28T01:02:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.476+0000 7fc9b2e73740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557062c0b920 0x557062c07270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.476+0000 7fc9b2e73740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557062c0c310 0x557062c1d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.476+0000 7fc9b2e73740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c11480 0x557062c14a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.476+0000 7fc9b2e73740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c012120 con 0x557062c0b920 2024-03-28T01:02:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.476+0000 7fc9b2e73740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c00fc20 con 0x557062c11480 2024-03-28T01:02:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.476+0000 7fc9b2e73740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c00dcc0 con 0x557062c0c310 2024-03-28T01:02:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.477+0000 7fc9ae822640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c11480 0x557062c14a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.477+0000 7fc9ae822640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c11480 0x557062c14a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:44450/0 (socket says 172.21.15.169:44450) 2024-03-28T01:02:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.477+0000 7fc9ae822640 1 -- 172.21.15.169:0/3547366279 learned_addr learned my addr 172.21.15.169:0/3547366279 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.477+0000 7fc9ae822640 1 -- 172.21.15.169:0/3547366279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557062c0c310 msgr2=0x557062c1d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.477+0000 7fc9ae822640 1 --2- 172.21.15.169:0/3547366279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557062c0c310 0x557062c1d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.477+0000 7fc9ae822640 1 -- 172.21.15.169:0/3547366279 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557062c0b920 msgr2=0x557062c07270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.477+0000 7fc9ae822640 1 --2- 172.21.15.169:0/3547366279 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557062c0b920 0x557062c07270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.477+0000 7fc9ae822640 1 -- 172.21.15.169:0/3547366279 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557062c07c00 con 0x557062c11480 2024-03-28T01:02:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.478+0000 7fc9ae822640 1 --2- 172.21.15.169:0/3547366279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c11480 0x557062c14a80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0022010 tx=0x7fc9a001aea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6deb8a004db5dea4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.479+0000 7fc993fff640 1 -- 172.21.15.169:0/3547366279 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a0038440 con 0x557062c11480 2024-03-28T01:02:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.479+0000 7fc993fff640 1 -- 172.21.15.169:0/3547366279 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc9a0047070 con 0x557062c11480 2024-03-28T01:02:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.479+0000 7fc993fff640 1 -- 172.21.15.169:0/3547366279 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a0004610 con 0x557062c11480 2024-03-28T01:02:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.480+0000 7fc9b2e73740 1 -- 172.21.15.169:0/3547366279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c11480 msgr2=0x557062c14a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.480+0000 7fc9b2e73740 1 --2- 172.21.15.169:0/3547366279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c11480 0x557062c14a80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0022010 tx=0x7fc9a001aea0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.480+0000 7fc9b2e73740 1 -- 172.21.15.169:0/3547366279 shutdown_connections 2024-03-28T01:02:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.480+0000 7fc9b2e73740 1 --2- 172.21.15.169:0/3547366279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c11480 0x557062c14a80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.480+0000 7fc9b2e73740 1 --2- 172.21.15.169:0/3547366279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557062c0c310 0x557062c1d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.480+0000 7fc9b2e73740 1 --2- 172.21.15.169:0/3547366279 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557062c0b920 0x557062c07270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.480+0000 7fc9b2e73740 1 -- 172.21.15.169:0/3547366279 >> 172.21.15.169:0/3547366279 conn(0x557062c06e90 msgr2=0x557062c01eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:18.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.484+0000 7fc9b2e73740 1 -- 172.21.15.169:0/3547366279 shutdown_connections 2024-03-28T01:02:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.484+0000 7fc9b2e73740 1 -- 172.21.15.169:0/3547366279 wait complete. 2024-03-28T01:02:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.485+0000 7fc9b2e73740 1 -- start start 2024-03-28T01:02:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.485+0000 7fc9b2e73740 1 Processor -- start 2024-03-28T01:02:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.487+0000 7fc9b2e73740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557062c0b920 0x557062c1cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.487+0000 7fc9b2e73740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c0c310 0x557062c1d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.487+0000 7fc9b2e73740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557062c11480 0x557062c199b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.487+0000 7fc9b2e73740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c00e2c0 con 0x557062c0b920 2024-03-28T01:02:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.487+0000 7fc9b2e73740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c00f5c0 con 0x557062c0c310 2024-03-28T01:02:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.487+0000 7fc9b2e73740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c00e900 con 0x557062c11480 2024-03-28T01:02:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.487+0000 7fc9b2e73740 10 client.0 _ll_register_callbacks cb 0x7fc9a002f7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.487+0000 7fc9ae021640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557062c0b920 0x557062c1cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.487+0000 7fc9ae021640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557062c0b920 0x557062c1cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36822/0 (socket says 172.21.15.169:36822) 2024-03-28T01:02:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.487+0000 7fc9ae021640 1 -- 172.21.15.169:0/4076683908 learned_addr learned my addr 172.21.15.169:0/4076683908 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.487+0000 7fc9ad820640 1 --2- 172.21.15.169:0/4076683908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c0c310 0x557062c1d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.488+0000 7fc9ae822640 1 --2- 172.21.15.169:0/4076683908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557062c11480 0x557062c199b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.488+0000 7fc9ae021640 1 -- 172.21.15.169:0/4076683908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557062c11480 msgr2=0x557062c199b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.488+0000 7fc9ae021640 1 --2- 172.21.15.169:0/4076683908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557062c11480 0x557062c199b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.488+0000 7fc9ae021640 1 -- 172.21.15.169:0/4076683908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c0c310 msgr2=0x557062c1d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.488+0000 7fc9ae021640 1 --2- 172.21.15.169:0/4076683908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c0c310 0x557062c1d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.488+0000 7fc9ae021640 1 -- 172.21.15.169:0/4076683908 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557062c1a1e0 con 0x557062c0b920 2024-03-28T01:02:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.488+0000 7fc9ae822640 1 --2- 172.21.15.169:0/4076683908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557062c11480 0x557062c199b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:02:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.493+0000 7fc9ae021640 1 --2- 172.21.15.169:0/4076683908 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557062c0b920 0x557062c1cec0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc99801c040 tx=0x7fc9980059a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da495b4c22effa03 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.493+0000 7fc991ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:02:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.496+0000 7fc991ffb640 1 -- 172.21.15.169:0/4076683908 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99802e060 con 0x557062c0b920 2024-03-28T01:02:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.496+0000 7fc9b2e73740 1 -- 172.21.15.169:0/4076683908 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc9a0047070 con 0x557062c0b920 2024-03-28T01:02:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.496+0000 7fc991ffb640 1 -- 172.21.15.169:0/4076683908 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc998028170 con 0x557062c0b920 2024-03-28T01:02:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.497+0000 7fc991ffb640 1 -- 172.21.15.169:0/4076683908 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc998035050 con 0x557062c0b920 2024-03-28T01:02:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.497+0000 7fc9b2e73740 10 client.16572 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.497+0000 7fc9b2e73740 1 -- 172.21.15.169:0/4076683908 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557062c7caa0 con 0x557062c0b920 2024-03-28T01:02:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.498+0000 7fc991ffb640 1 -- 172.21.15.169:0/4076683908 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc9a0047070 con 0x557062c0b920 2024-03-28T01:02:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.498+0000 7fc991ffb640 1 -- 172.21.15.169:0/4076683908 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fc998028800 con 0x557062c0b920 2024-03-28T01:02:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.498+0000 7fc9b2e73740 10 client.16572 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.498+0000 7fc9b2e73740 10 client.16572 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:02:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.498+0000 7fc9b2e73740 20 client.16572 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.498+0000 7fc9b2e73740 1 --2- 172.21.15.169:0/4076683908 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557062c7d430 0x557062c7f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.498+0000 7fc9b2e73740 4 client.16572 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:02:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.498+0000 7fc9b2e73740 1 -- 172.21.15.169:0/4076683908 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557062af6710 con 0x557062c7d430 2024-03-28T01:02:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.499+0000 7fc9ad820640 1 --2- 172.21.15.169:0/4076683908 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557062c7d430 0x557062c7f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.501+0000 7fc9ad820640 1 --2- 172.21.15.169:0/4076683908 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557062c7d430 0x557062c7f850 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c63dff0e07733fe2 server_cookie=74f0ba8c7d0d924f in_seq=0 out_seq=0 2024-03-28T01:02:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:18.501+0000 7fc991ffb640 10 client.16572 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:19.362 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.357+0000 7fc991ffb640 1 -- 172.21.15.169:0/4076683908 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557062af6710 con 0x557062c7d430 2024-03-28T01:02:19.362 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.357+0000 7fc991ffb640 10 client.16572 handle_command_reply: tid=0 2024-03-28T01:02:19.363 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.357+0000 7fc9b2e73740 1 client.16572 shutdown 2024-03-28T01:02:19.363 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.357+0000 7fc9b2e73740 1 -- 172.21.15.169:0/4076683908 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557062c0b920 msgr2=0x557062c1cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:19.363 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.357+0000 7fc9b2e73740 1 --2- 172.21.15.169:0/4076683908 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557062c0b920 0x557062c1cec0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc99801c040 tx=0x7fc9980059a0 comp rx=0 tx=0).stop 2024-03-28T01:02:19.363 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.357+0000 7fc9b2e73740 1 -- 172.21.15.169:0/4076683908 shutdown_connections 2024-03-28T01:02:19.363 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.357+0000 7fc9b2e73740 1 --2- 172.21.15.169:0/4076683908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557062c11480 0x557062c199b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:19.363 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.357+0000 7fc9b2e73740 1 --2- 172.21.15.169:0/4076683908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557062c0c310 0x557062c1d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:19.364 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.357+0000 7fc9b2e73740 1 --2- 172.21.15.169:0/4076683908 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557062c7d430 0x557062c7f850 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:19.364 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.357+0000 7fc9b2e73740 1 --2- 172.21.15.169:0/4076683908 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557062c0b920 0x557062c1cec0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:19.364 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.357+0000 7fc9b2e73740 1 -- 172.21.15.169:0/4076683908 >> 172.21.15.169:0/4076683908 conn(0x557062c06e90 msgr2=0x557062c07270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:19.364 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.357+0000 7fc991ffb640 0 client.16572 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:19.364 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.358+0000 7fc9b2e73740 1 -- 172.21.15.169:0/4076683908 shutdown_connections 2024-03-28T01:02:19.364 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.358+0000 7fc9b2e73740 1 -- 172.21.15.169:0/4076683908 wait complete. 2024-03-28T01:02:19.364 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.358+0000 7fc9b2e73740 20 client.16572 trim_cache size 0 max 16384 2024-03-28T01:02:19.365 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:02:19.365 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:02:19.365 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:02:19.365 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:02:19.366 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.360+0000 7fc9b0285640 1 -- 172.21.15.169:0/826716607 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc9940a29f0 msgr2=0x7fc9940a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:19.366 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.360+0000 7fc9b0285640 1 --2- 172.21.15.169:0/826716607 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc9940a29f0 0x7fc9940a4eb0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x557062c0cb70 tx=0x7fc9a0022040 comp rx=0 tx=0).stop 2024-03-28T01:02:19.366 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.360+0000 7fc9b0285640 1 -- 172.21.15.169:0/826716607 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9a810b010 msgr2=0x7fc9a819e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:19.366 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.360+0000 7fc9b0285640 1 --2- 172.21.15.169:0/826716607 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9a810b010 0x7fc9a819e440 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4009780 tx=0x7fc9a4032440 comp rx=0 tx=0).stop 2024-03-28T01:02:19.366 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.362+0000 7fc9b0285640 1 -- 172.21.15.169:0/826716607 shutdown_connections 2024-03-28T01:02:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.362+0000 7fc9b0285640 1 --2- 172.21.15.169:0/826716607 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc9940a29f0 0x7fc9940a4eb0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.362+0000 7fc9b0285640 1 --2- 172.21.15.169:0/826716607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9a810d9c0 0x7fc9a81a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.362+0000 7fc9b0285640 1 --2- 172.21.15.169:0/826716607 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9a810b010 0x7fc9a819e440 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.362+0000 7fc9b0285640 1 --2- 172.21.15.169:0/826716607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9a81086a0 0x7fc9a819df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.362+0000 7fc9b0285640 1 -- 172.21.15.169:0/826716607 >> 172.21.15.169:0/826716607 conn(0x7fc9a806e6e0 msgr2=0x7fc9a810b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.363+0000 7fc9b0285640 1 -- 172.21.15.169:0/826716607 shutdown_connections 2024-03-28T01:02:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:19.363+0000 7fc9b0285640 1 -- 172.21.15.169:0/826716607 wait complete. 2024-03-28T01:02:19.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:19 smithi169 ceph-mon[32274]: pgmap v525: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 754 MiB/s wr, 1.29k op/s 2024-03-28T01:02:19.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:19 smithi174 ceph-mon[36060]: pgmap v525: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 754 MiB/s wr, 1.29k op/s 2024-03-28T01:02:19.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:19 smithi177 ceph-mon[36206]: pgmap v525: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 754 MiB/s wr, 1.29k op/s 2024-03-28T01:02:21.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:21 smithi169 ceph-mon[32274]: pgmap v526: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 742 MiB/s wr, 1.27k op/s 2024-03-28T01:02:21.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:21 smithi174 ceph-mon[36060]: pgmap v526: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 742 MiB/s wr, 1.27k op/s 2024-03-28T01:02:21.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:21 smithi177 ceph-mon[36206]: pgmap v526: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 742 MiB/s wr, 1.27k op/s 2024-03-28T01:02:23.806 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:23 smithi169 ceph-mon[32274]: pgmap v527: 129 pgs: 129 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 937 MiB/s wr, 1.89k op/s 2024-03-28T01:02:23.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:23 smithi174 ceph-mon[36060]: pgmap v527: 129 pgs: 129 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 937 MiB/s wr, 1.89k op/s 2024-03-28T01:02:23.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:23 smithi177 ceph-mon[36206]: pgmap v527: 129 pgs: 129 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 937 MiB/s wr, 1.89k op/s 2024-03-28T01:02:25.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:25 smithi169 ceph-mon[32274]: pgmap v528: 129 pgs: 129 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 728 MiB/s wr, 1.56k op/s 2024-03-28T01:02:25.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:25 smithi174 ceph-mon[36060]: pgmap v528: 129 pgs: 129 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 728 MiB/s wr, 1.56k op/s 2024-03-28T01:02:25.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:25 smithi177 ceph-mon[36206]: pgmap v528: 129 pgs: 129 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 728 MiB/s wr, 1.56k op/s 2024-03-28T01:02:27.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:27 smithi174 ceph-mon[36060]: pgmap v529: 129 pgs: 129 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 797 MiB/s wr, 1.84k op/s 2024-03-28T01:02:27.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:27 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:02:27.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:27 smithi177 ceph-mon[36206]: pgmap v529: 129 pgs: 129 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 797 MiB/s wr, 1.84k op/s 2024-03-28T01:02:27.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:27 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:02:28.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:27 smithi169 ceph-mon[32274]: pgmap v529: 129 pgs: 129 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 797 MiB/s wr, 1.84k op/s 2024-03-28T01:02:28.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:27 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:02:29.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:29 smithi174 ceph-mon[36060]: pgmap v530: 129 pgs: 129 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 759 MiB/s wr, 1.91k op/s 2024-03-28T01:02:29.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:29 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:02:29.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:29 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:02:29.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:29 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:02:29.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:29 smithi177 ceph-mon[36206]: pgmap v530: 129 pgs: 129 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 759 MiB/s wr, 1.91k op/s 2024-03-28T01:02:29.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:29 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:02:29.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:29 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:02:29.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:29 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:02:30.054 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:29 smithi169 ceph-mon[32274]: pgmap v530: 129 pgs: 129 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 759 MiB/s wr, 1.91k op/s 2024-03-28T01:02:30.054 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:29 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:02:30.054 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:29 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:02:30.054 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:29 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:02:30.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:30 smithi177 ceph-mon[36206]: pgmap v531: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 727 MiB/s wr, 1.93k op/s 2024-03-28T01:02:31.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:30 smithi169 ceph-mon[32274]: pgmap v531: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 727 MiB/s wr, 1.93k op/s 2024-03-28T01:02:31.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:30 smithi174 ceph-mon[36060]: pgmap v531: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 727 MiB/s wr, 1.93k op/s 2024-03-28T01:02:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.640+0000 7f011fc80640 1 -- 172.21.15.169:0/982506566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f011810d9c0 msgr2=0x7f011810de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.640+0000 7f011fc80640 1 --2- 172.21.15.169:0/982506566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f011810d9c0 0x7f011810de40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f010801a910 tx=0x7f010803cca0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.642+0000 7f011fc80640 1 -- 172.21.15.169:0/982506566 shutdown_connections 2024-03-28T01:02:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.642+0000 7f011fc80640 1 --2- 172.21.15.169:0/982506566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f011810d9c0 0x7f011810de40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.642+0000 7f011fc80640 1 --2- 172.21.15.169:0/982506566 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f011810b010 0x7f011810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.642+0000 7f011fc80640 1 --2- 172.21.15.169:0/982506566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01181086a0 0x7f011810aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.642+0000 7f011fc80640 1 -- 172.21.15.169:0/982506566 >> 172.21.15.169:0/982506566 conn(0x7f011806e6e0 msgr2=0x7f0118070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.650+0000 7f011fc80640 1 -- 172.21.15.169:0/982506566 shutdown_connections 2024-03-28T01:02:32.659 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.654+0000 7f011fc80640 1 -- 172.21.15.169:0/982506566 wait complete. 2024-03-28T01:02:32.660 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.655+0000 7f011fc80640 1 Processor -- start 2024-03-28T01:02:32.660 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.655+0000 7f011fc80640 1 -- start start 2024-03-28T01:02:32.660 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.655+0000 7f011fc80640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01181086a0 0x7f011819de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:32.660 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.655+0000 7f011fc80640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f011810b010 0x7f011819e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:32.660 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.655+0000 7f011fc80640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f011810d9c0 0x7f01181a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:32.660 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.655+0000 7f011fc80640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01181119d0 con 0x7f01181086a0 2024-03-28T01:02:32.660 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.655+0000 7f011fc80640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0118111850 con 0x7f011810d9c0 2024-03-28T01:02:32.660 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.655+0000 7f011fc80640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0118111b50 con 0x7f011810b010 2024-03-28T01:02:32.663 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.660+0000 7f011d21b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f011810b010 0x7f011819e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:32.664 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.661+0000 7f011d21b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f011810b010 0x7f011819e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:47298/0 (socket says 172.21.15.169:47298) 2024-03-28T01:02:32.664 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.661+0000 7f011d21b640 1 -- 172.21.15.169:0/2253799410 learned_addr learned my addr 172.21.15.169:0/2253799410 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:32.666 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.661+0000 7f011e21d640 1 --2- 172.21.15.169:0/2253799410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f011810d9c0 0x7f01181a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.664+0000 7f011da1c640 1 --2- 172.21.15.169:0/2253799410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01181086a0 0x7f011819de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.667+0000 7f011d21b640 1 -- 172.21.15.169:0/2253799410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f011810d9c0 msgr2=0x7f01181a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.667+0000 7f011d21b640 1 --2- 172.21.15.169:0/2253799410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f011810d9c0 0x7f01181a5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.667+0000 7f011d21b640 1 -- 172.21.15.169:0/2253799410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01181086a0 msgr2=0x7f011819de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.667+0000 7f011d21b640 1 --2- 172.21.15.169:0/2253799410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01181086a0 0x7f011819de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.667+0000 7f011d21b640 1 -- 172.21.15.169:0/2253799410 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01181a5ab0 con 0x7f011810b010 2024-03-28T01:02:32.678 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.675+0000 7f011d21b640 1 --2- 172.21.15.169:0/2253799410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f011810b010 0x7f011819e3d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f011000ba50 tx=0x7f011000bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.677+0000 7f010effd640 1 -- 172.21.15.169:0/2253799410 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0110019070 con 0x7f011810b010 2024-03-28T01:02:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.677+0000 7f010effd640 1 -- 172.21.15.169:0/2253799410 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f011000cbe0 con 0x7f011810b010 2024-03-28T01:02:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.677+0000 7f010effd640 1 -- 172.21.15.169:0/2253799410 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0110007630 con 0x7f011810b010 2024-03-28T01:02:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.678+0000 7f011fc80640 1 -- 172.21.15.169:0/2253799410 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01181a5da0 con 0x7f011810b010 2024-03-28T01:02:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.678+0000 7f011fc80640 1 -- 172.21.15.169:0/2253799410 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01181a61b0 con 0x7f011810b010 2024-03-28T01:02:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.684+0000 7f012285f740 1 Processor -- start 2024-03-28T01:02:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.686+0000 7f012285f740 1 -- start start 2024-03-28T01:02:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.686+0000 7f012285f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856482920 0x55f856482d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.687+0000 7f012285f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856483310 0x55f856487e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.687+0000 7f012285f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f856488480 0x55f85648a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.687+0000 7f012285f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8563e7b10 con 0x55f856483310 2024-03-28T01:02:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.687+0000 7f012285f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f856390900 con 0x55f856482920 2024-03-28T01:02:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.687+0000 7f012285f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8563e3bb0 con 0x55f856488480 2024-03-28T01:02:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.687+0000 7f011e21d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856482920 0x55f856482d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.687+0000 7f011e21d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856482920 0x55f856482d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49660/0 (socket says 172.21.15.169:49660) 2024-03-28T01:02:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.687+0000 7f011e21d640 1 -- 172.21.15.169:0/2466627499 learned_addr learned my addr 172.21.15.169:0/2466627499 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.688+0000 7f011da1c640 1 --2- 172.21.15.169:0/2466627499 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856483310 0x55f856487e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.688+0000 7f011e21d640 1 -- 172.21.15.169:0/2466627499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f856488480 msgr2=0x55f85648a910 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:02:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.688+0000 7f011e21d640 1 --2- 172.21.15.169:0/2466627499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f856488480 0x55f85648a910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.688+0000 7f011e21d640 1 -- 172.21.15.169:0/2466627499 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856483310 msgr2=0x55f856487e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.688+0000 7f011e21d640 1 --2- 172.21.15.169:0/2466627499 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856483310 0x55f856487e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.688+0000 7f011e21d640 1 -- 172.21.15.169:0/2466627499 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8560fc210 con 0x55f856482920 2024-03-28T01:02:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.688+0000 7f011da1c640 1 --2- 172.21.15.169:0/2466627499 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856483310 0x55f856487e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:02:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.689+0000 7f011e21d640 1 --2- 172.21.15.169:0/2466627499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856482920 0x55f856482d40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f010801a910 tx=0x7f010801a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7c9f1a317099f8a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.702+0000 7f0103fff640 1 -- 172.21.15.169:0/2466627499 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108004c10 con 0x55f856482920 2024-03-28T01:02:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.702+0000 7f0103fff640 1 -- 172.21.15.169:0/2466627499 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f010803d850 con 0x55f856482920 2024-03-28T01:02:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.702+0000 7f0103fff640 1 -- 172.21.15.169:0/2466627499 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108005370 con 0x55f856482920 2024-03-28T01:02:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.705+0000 7f012285f740 1 -- 172.21.15.169:0/2466627499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856482920 msgr2=0x55f856482d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.705+0000 7f012285f740 1 --2- 172.21.15.169:0/2466627499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856482920 0x55f856482d40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f010801a910 tx=0x7f010801a680 comp rx=0 tx=0).stop 2024-03-28T01:02:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.705+0000 7f010effd640 1 -- 172.21.15.169:0/2253799410 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0110014440 con 0x7f011810b010 2024-03-28T01:02:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.706+0000 7f010effd640 1 --2- 172.21.15.169:0/2253799410 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f01040a2ad0 0x7f01040a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.706+0000 7f010effd640 1 -- 172.21.15.169:0/2253799410 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f01100ce1e0 con 0x7f011810b010 2024-03-28T01:02:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.707+0000 7f011e21d640 1 --2- 172.21.15.169:0/2253799410 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f01040a2ad0 0x7f01040a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.710+0000 7f011e21d640 1 --2- 172.21.15.169:0/2253799410 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f01040a2ad0 0x7f01040a4f90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f010801a910 tx=0x7f01080055c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:32.717 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.714+0000 7f012285f740 1 -- 172.21.15.169:0/2466627499 shutdown_connections 2024-03-28T01:02:32.717 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.714+0000 7f012285f740 1 --2- 172.21.15.169:0/2466627499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f856488480 0x55f85648a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.717 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.714+0000 7f012285f740 1 --2- 172.21.15.169:0/2466627499 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856483310 0x55f856487e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.717 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.714+0000 7f012285f740 1 --2- 172.21.15.169:0/2466627499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856482920 0x55f856482d40 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.717 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.714+0000 7f012285f740 1 -- 172.21.15.169:0/2466627499 >> 172.21.15.169:0/2466627499 conn(0x55f85647de90 msgr2=0x55f85647e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:32.721 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.718+0000 7f012285f740 1 -- 172.21.15.169:0/2466627499 shutdown_connections 2024-03-28T01:02:32.723 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.720+0000 7f012285f740 1 -- 172.21.15.169:0/2466627499 wait complete. 2024-03-28T01:02:32.723 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.720+0000 7f012285f740 1 -- start start 2024-03-28T01:02:32.723 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.720+0000 7f012285f740 1 Processor -- start 2024-03-28T01:02:32.724 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.721+0000 7f012285f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856482920 0x55f85648cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:32.724 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.721+0000 7f012285f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f856483310 0x55f85648d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:32.724 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.721+0000 7f012285f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856488480 0x55f8564941a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:32.725 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.721+0000 7f012285f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f85648b3d0 con 0x55f856488480 2024-03-28T01:02:32.725 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.721+0000 7f012285f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8563a3d50 con 0x55f856482920 2024-03-28T01:02:32.725 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.721+0000 7f012285f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f85648b550 con 0x55f856483310 2024-03-28T01:02:32.725 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.721+0000 7f012285f740 10 client.0 _ll_register_callbacks cb 0x55f8563a3720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:32.728 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.725+0000 7f011da1c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856482920 0x55f85648cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:32.728 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.725+0000 7f011da1c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856482920 0x55f85648cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49666/0 (socket says 172.21.15.169:49666) 2024-03-28T01:02:32.728 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.725+0000 7f011da1c640 1 -- 172.21.15.169:0/1300974279 learned_addr learned my addr 172.21.15.169:0/1300974279 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:32.728 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.725+0000 7f011e21d640 1 --2- 172.21.15.169:0/1300974279 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856488480 0x55f8564941a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:32.730 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.727+0000 7f011da1c640 1 -- 172.21.15.169:0/1300974279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f856483310 msgr2=0x55f85648d160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:02:32.730 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.727+0000 7f011da1c640 1 --2- 172.21.15.169:0/1300974279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f856483310 0x55f85648d160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.730 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.727+0000 7f011da1c640 1 -- 172.21.15.169:0/1300974279 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856488480 msgr2=0x55f8564941a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:32.730 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.727+0000 7f011da1c640 1 --2- 172.21.15.169:0/1300974279 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856488480 0x55f8564941a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.730 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.727+0000 7f011da1c640 1 -- 172.21.15.169:0/1300974279 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f8561c8620 con 0x55f856482920 2024-03-28T01:02:32.736 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.733+0000 7f011da1c640 1 --2- 172.21.15.169:0/1300974279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856482920 0x55f85648cc20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f011400be60 tx=0x7f01140117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eff9d7efb7bbc205 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:32.736 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.733+0000 7f01017fa640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:02:32.738 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.735+0000 7f01017fa640 1 -- 172.21.15.169:0/1300974279 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0114012650 con 0x55f856482920 2024-03-28T01:02:32.738 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.735+0000 7f012285f740 1 -- 172.21.15.169:0/1300974279 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f010803d850 con 0x55f856482920 2024-03-28T01:02:32.738 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.735+0000 7f01017fa640 1 -- 172.21.15.169:0/1300974279 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f01140127f0 con 0x55f856482920 2024-03-28T01:02:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.738+0000 7f01017fa640 1 -- 172.21.15.169:0/1300974279 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f01140129f0 con 0x55f856482920 2024-03-28T01:02:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.739+0000 7f012285f740 10 client.26471 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.739+0000 7f012285f740 1 -- 172.21.15.169:0/1300974279 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f8564975d0 con 0x55f856482920 2024-03-28T01:02:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.740+0000 7f01017fa640 1 -- 172.21.15.169:0/1300974279 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f010803d850 con 0x55f856482920 2024-03-28T01:02:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.740+0000 7f01017fa640 1 -- 172.21.15.169:0/1300974279 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f011401a3d0 con 0x55f856482920 2024-03-28T01:02:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.740+0000 7f012285f740 10 client.26471 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.740+0000 7f012285f740 10 client.26471 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:02:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.740+0000 7f012285f740 20 client.26471 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.740+0000 7f012285f740 1 --2- 172.21.15.169:0/1300974279 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f856499130 0x55f85649b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.740+0000 7f012285f740 4 client.26471 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:02:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.740+0000 7f012285f740 1 -- 172.21.15.169:0/1300974279 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f8563f5050 con 0x55f856499130 2024-03-28T01:02:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.742+0000 7f011d21b640 1 --2- 172.21.15.169:0/1300974279 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f856499130 0x55f85649b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.752+0000 7f011d21b640 1 --2- 172.21.15.169:0/1300974279 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f856499130 0x55f85649b590 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3dc7a59705a421e server_cookie=71fb91342e075231 in_seq=0 out_seq=0 2024-03-28T01:02:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.752+0000 7f01017fa640 10 client.26471 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:32.758 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.755+0000 7f01017fa640 1 -- 172.21.15.169:0/1300974279 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f8563f5050 con 0x55f856499130 2024-03-28T01:02:32.758 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.755+0000 7f01017fa640 10 client.26471 handle_command_reply: tid=0 2024-03-28T01:02:32.759 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.756+0000 7f012285f740 1 client.26471 shutdown 2024-03-28T01:02:32.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.757+0000 7f012285f740 1 -- 172.21.15.169:0/1300974279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856482920 msgr2=0x55f85648cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:32.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.757+0000 7f012285f740 1 --2- 172.21.15.169:0/1300974279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856482920 0x55f85648cc20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f011400be60 tx=0x7f01140117b0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.767+0000 7f012285f740 1 -- 172.21.15.169:0/1300974279 shutdown_connections 2024-03-28T01:02:32.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.767+0000 7f012285f740 1 --2- 172.21.15.169:0/1300974279 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f856499130 0x55f85649b590 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.767+0000 7f012285f740 1 --2- 172.21.15.169:0/1300974279 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856488480 0x55f8564941a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.767+0000 7f012285f740 1 --2- 172.21.15.169:0/1300974279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f856483310 0x55f85648d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.767+0000 7f012285f740 1 --2- 172.21.15.169:0/1300974279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856482920 0x55f85648cc20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.767+0000 7f012285f740 1 -- 172.21.15.169:0/1300974279 >> 172.21.15.169:0/1300974279 conn(0x55f85647de90 msgr2=0x55f85647d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:32.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.767+0000 7f01017fa640 0 client.26471 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:32.770 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.767+0000 7f012285f740 1 -- 172.21.15.169:0/1300974279 shutdown_connections 2024-03-28T01:02:32.774 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.771+0000 7f012285f740 1 -- 172.21.15.169:0/1300974279 wait complete. 2024-03-28T01:02:32.775 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.773+0000 7f012285f740 20 client.26471 trim_cache size 0 max 16384 2024-03-28T01:02:32.823 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.821+0000 7f012285f740 1 Processor -- start 2024-03-28T01:02:32.823 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.821+0000 7f012285f740 1 -- start start 2024-03-28T01:02:32.824 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.821+0000 7f012285f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f856482920 0x55f85647e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:32.824 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.821+0000 7f012285f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856483310 0x55f856494160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:32.825 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.821+0000 7f012285f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856488480 0x55f8564810f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:32.825 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.822+0000 7f012285f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01181b2bc0 con 0x55f856483310 2024-03-28T01:02:32.825 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.822+0000 7f012285f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01181b06c0 con 0x55f856488480 2024-03-28T01:02:32.825 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.822+0000 7f012285f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01181ae760 con 0x55f856482920 2024-03-28T01:02:32.825 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.822+0000 7f011d21b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856483310 0x55f856494160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:32.825 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.822+0000 7f011d21b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856483310 0x55f856494160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39526/0 (socket says 172.21.15.169:39526) 2024-03-28T01:02:32.825 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.822+0000 7f011d21b640 1 -- 172.21.15.169:0/1541064588 learned_addr learned my addr 172.21.15.169:0/1541064588 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:32.825 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.822+0000 7f011e21d640 1 --2- 172.21.15.169:0/1541064588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856488480 0x55f8564810f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.823+0000 7f011d21b640 1 -- 172.21.15.169:0/1541064588 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f856482920 msgr2=0x55f85647e270 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:02:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.823+0000 7f011d21b640 1 --2- 172.21.15.169:0/1541064588 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f856482920 0x55f85647e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.823+0000 7f011d21b640 1 -- 172.21.15.169:0/1541064588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856488480 msgr2=0x55f8564810f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.823+0000 7f011d21b640 1 --2- 172.21.15.169:0/1541064588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856488480 0x55f8564810f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.823+0000 7f011d21b640 1 -- 172.21.15.169:0/1541064588 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f856482710 con 0x55f856483310 2024-03-28T01:02:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.824+0000 7f011d21b640 1 --2- 172.21.15.169:0/1541064588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856483310 0x55f856494160 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f0110014cb0 tx=0x7f0110014b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=958a5d4b87beff42 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.825+0000 7f01037fe640 1 -- 172.21.15.169:0/1541064588 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0110014730 con 0x55f856483310 2024-03-28T01:02:32.828 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.825+0000 7f01037fe640 1 -- 172.21.15.169:0/1541064588 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f011000cd90 con 0x55f856483310 2024-03-28T01:02:32.829 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.826+0000 7f012285f740 1 -- 172.21.15.169:0/1541064588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856483310 msgr2=0x55f856494160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:32.829 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.826+0000 7f012285f740 1 --2- 172.21.15.169:0/1541064588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856483310 0x55f856494160 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f0110014cb0 tx=0x7f0110014b60 comp rx=0 tx=0).stop 2024-03-28T01:02:32.829 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.826+0000 7f012285f740 1 -- 172.21.15.169:0/1541064588 shutdown_connections 2024-03-28T01:02:32.829 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.826+0000 7f012285f740 1 --2- 172.21.15.169:0/1541064588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856488480 0x55f8564810f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.830 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.826+0000 7f012285f740 1 --2- 172.21.15.169:0/1541064588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856483310 0x55f856494160 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.830 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.826+0000 7f012285f740 1 --2- 172.21.15.169:0/1541064588 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f856482920 0x55f85647e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.830 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.826+0000 7f012285f740 1 -- 172.21.15.169:0/1541064588 >> 172.21.15.169:0/1541064588 conn(0x55f85647de90 msgr2=0x55f856478eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:32.834 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.831+0000 7f012285f740 1 -- 172.21.15.169:0/1541064588 shutdown_connections 2024-03-28T01:02:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.833+0000 7f012285f740 1 -- 172.21.15.169:0/1541064588 wait complete. 2024-03-28T01:02:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.834+0000 7f012285f740 1 -- start start 2024-03-28T01:02:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.834+0000 7f012285f740 1 Processor -- start 2024-03-28T01:02:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.834+0000 7f012285f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856482920 0x55f856491550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.834+0000 7f012285f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f856488480 0x55f856491a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.834+0000 7f012285f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856492aa0 0x55f856492f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.834+0000 7f012285f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01181aed60 con 0x55f856482920 2024-03-28T01:02:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.834+0000 7f012285f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01181b0060 con 0x55f856492aa0 2024-03-28T01:02:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.834+0000 7f012285f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01181af3a0 con 0x55f856488480 2024-03-28T01:02:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.835+0000 7f012285f740 10 client.0 _ll_register_callbacks cb 0x7f011400b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.835+0000 7f011d21b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f856488480 0x55f856491a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.835+0000 7f011d21b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f856488480 0x55f856491a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:47330/0 (socket says 172.21.15.169:47330) 2024-03-28T01:02:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.835+0000 7f011d21b640 1 -- 172.21.15.169:0/536567870 learned_addr learned my addr 172.21.15.169:0/536567870 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.835+0000 7f011e21d640 1 --2- 172.21.15.169:0/536567870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856492aa0 0x55f856492f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.836+0000 7f011d21b640 1 -- 172.21.15.169:0/536567870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856492aa0 msgr2=0x55f856492f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.836+0000 7f011d21b640 1 --2- 172.21.15.169:0/536567870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856492aa0 0x55f856492f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.836+0000 7f011d21b640 1 -- 172.21.15.169:0/536567870 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856482920 msgr2=0x55f856491550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.836+0000 7f011d21b640 1 --2- 172.21.15.169:0/536567870 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856482920 0x55f856491550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.836+0000 7f011d21b640 1 -- 172.21.15.169:0/536567870 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f856493460 con 0x55f856488480 2024-03-28T01:02:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.836+0000 7f011d21b640 1 --2- 172.21.15.169:0/536567870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f856488480 0x55f856491a90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f01100a2a70 tx=0x7f01100a2fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c2672364f8f45c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:32.844 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.841+0000 7f01017fa640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:02:32.844 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.841+0000 7f01017fa640 1 -- 172.21.15.169:0/536567870 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01100b7020 con 0x55f856488480 2024-03-28T01:02:32.844 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.841+0000 7f01017fa640 1 -- 172.21.15.169:0/536567870 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f01100cb960 con 0x55f856488480 2024-03-28T01:02:32.844 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.841+0000 7f012285f740 1 -- 172.21.15.169:0/536567870 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f8563ef3a0 con 0x55f856488480 2024-03-28T01:02:32.844 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.842+0000 7f01017fa640 1 -- 172.21.15.169:0/536567870 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0110096f20 con 0x55f856488480 2024-03-28T01:02:32.845 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.842+0000 7f012285f740 10 client.26392 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:32.845 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.842+0000 7f012285f740 1 -- 172.21.15.169:0/536567870 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f8564f4c20 con 0x55f856488480 2024-03-28T01:02:32.845 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.842+0000 7f01017fa640 1 -- 172.21.15.169:0/536567870 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f8563ef3a0 con 0x55f856488480 2024-03-28T01:02:32.846 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.843+0000 7f01017fa640 1 -- 172.21.15.169:0/536567870 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f011009bb30 con 0x55f856488480 2024-03-28T01:02:32.847 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.843+0000 7f012285f740 10 client.26392 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:32.847 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.843+0000 7f012285f740 10 client.26392 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:02:32.847 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.843+0000 7f012285f740 20 client.26392 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:32.847 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.843+0000 7f012285f740 1 --2- 172.21.15.169:0/536567870 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f8564f5650 0x55f8564f7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:32.847 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.843+0000 7f012285f740 4 client.26392 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:02:32.847 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.843+0000 7f012285f740 1 -- 172.21.15.169:0/536567870 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f85636d710 con 0x55f8564f5650 2024-03-28T01:02:32.849 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.847+0000 7f011da1c640 1 --2- 172.21.15.169:0/536567870 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f8564f5650 0x55f8564f7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:32.856 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.853+0000 7f011da1c640 1 --2- 172.21.15.169:0/536567870 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f8564f5650 0x55f8564f7a70 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51ab3e653c6f5436 server_cookie=f7e471a98e3667e8 in_seq=0 out_seq=0 2024-03-28T01:02:32.857 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:32.854+0000 7f01017fa640 10 client.26392 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:33.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:33 smithi169 ceph-mon[32274]: pgmap v532: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 852 MiB/s wr, 2.10k op/s 2024-03-28T01:02:33.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:33 smithi174 ceph-mon[36060]: pgmap v532: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 852 MiB/s wr, 2.10k op/s 2024-03-28T01:02:33.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:33 smithi177 ceph-mon[36206]: pgmap v532: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 852 MiB/s wr, 2.10k op/s 2024-03-28T01:02:35.145 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:02:35.145 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:02:35.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.411+0000 7f01017fa640 1 -- 172.21.15.169:0/536567870 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f85636d710 con 0x55f8564f5650 2024-03-28T01:02:35.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.411+0000 7f01017fa640 10 client.26392 handle_command_reply: tid=0 2024-03-28T01:02:35.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.411+0000 7f012285f740 1 client.26392 shutdown 2024-03-28T01:02:35.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.412+0000 7f012285f740 1 -- 172.21.15.169:0/536567870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f856488480 msgr2=0x55f856491a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:35.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.412+0000 7f012285f740 1 --2- 172.21.15.169:0/536567870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f856488480 0x55f856491a90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f01100a2a70 tx=0x7f01100a2fa0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.412+0000 7f012285f740 1 -- 172.21.15.169:0/536567870 shutdown_connections 2024-03-28T01:02:35.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.412+0000 7f012285f740 1 --2- 172.21.15.169:0/536567870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f856492aa0 0x55f856492f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.412+0000 7f012285f740 1 --2- 172.21.15.169:0/536567870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f856488480 0x55f856491a90 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.412+0000 7f012285f740 1 --2- 172.21.15.169:0/536567870 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f8564f5650 0x55f8564f7a70 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.412+0000 7f012285f740 1 --2- 172.21.15.169:0/536567870 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f856482920 0x55f856491550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.412+0000 7f012285f740 1 -- 172.21.15.169:0/536567870 >> 172.21.15.169:0/536567870 conn(0x55f85647de90 msgr2=0x55f8564810f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:35.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.412+0000 7f01017fa640 0 client.26392 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:35.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.412+0000 7f012285f740 1 -- 172.21.15.169:0/536567870 shutdown_connections 2024-03-28T01:02:35.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.412+0000 7f012285f740 1 -- 172.21.15.169:0/536567870 wait complete. 2024-03-28T01:02:35.416 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.413+0000 7f012285f740 20 client.26392 trim_cache size 0 max 16384 2024-03-28T01:02:35.420 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:02:35.420 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:02:35.420 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:02:35.420 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:02:35.422 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.419+0000 7f011fc80640 1 -- 172.21.15.169:0/2253799410 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f01040a2ad0 msgr2=0x7f01040a4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:35.422 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.419+0000 7f011fc80640 1 --2- 172.21.15.169:0/2253799410 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f01040a2ad0 0x7f01040a4f90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f010801a910 tx=0x7f01080055c0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.422 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.419+0000 7f011fc80640 1 -- 172.21.15.169:0/2253799410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f011810b010 msgr2=0x7f011819e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:35.422 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.419+0000 7f011fc80640 1 --2- 172.21.15.169:0/2253799410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f011810b010 0x7f011819e3d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f011000ba50 tx=0x7f011000bf10 comp rx=0 tx=0).stop 2024-03-28T01:02:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.423+0000 7f011fc80640 1 -- 172.21.15.169:0/2253799410 shutdown_connections 2024-03-28T01:02:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.423+0000 7f011fc80640 1 --2- 172.21.15.169:0/2253799410 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f01040a2ad0 0x7f01040a4f90 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.423+0000 7f011fc80640 1 --2- 172.21.15.169:0/2253799410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f011810d9c0 0x7f01181a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.423+0000 7f011fc80640 1 --2- 172.21.15.169:0/2253799410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f011810b010 0x7f011819e3d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.423+0000 7f011fc80640 1 --2- 172.21.15.169:0/2253799410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f01181086a0 0x7f011819de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.423+0000 7f011fc80640 1 -- 172.21.15.169:0/2253799410 >> 172.21.15.169:0/2253799410 conn(0x7f011806e6e0 msgr2=0x7f0118108c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.424+0000 7f011fc80640 1 -- 172.21.15.169:0/2253799410 shutdown_connections 2024-03-28T01:02:35.429 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:35.425+0000 7f011fc80640 1 -- 172.21.15.169:0/2253799410 wait complete. 2024-03-28T01:02:35.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.480+0000 7ffb44a9a640 1 -- 172.21.15.169:0/4159434132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb401086a0 msgr2=0x7ffb4010aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:35.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.480+0000 7ffb44a9a640 1 --2- 172.21.15.169:0/4159434132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb401086a0 0x7ffb4010aad0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ffb30009900 tx=0x7ffb3002bc80 comp rx=0 tx=0).stop 2024-03-28T01:02:35.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.486+0000 7ffb44a9a640 1 -- 172.21.15.169:0/4159434132 shutdown_connections 2024-03-28T01:02:35.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.486+0000 7ffb44a9a640 1 --2- 172.21.15.169:0/4159434132 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb4010d9c0 0x7ffb4010de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.486+0000 7ffb44a9a640 1 --2- 172.21.15.169:0/4159434132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb4010b010 0x7ffb4010d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.486+0000 7ffb44a9a640 1 --2- 172.21.15.169:0/4159434132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb401086a0 0x7ffb4010aad0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.486+0000 7ffb44a9a640 1 -- 172.21.15.169:0/4159434132 >> 172.21.15.169:0/4159434132 conn(0x7ffb4006e500 msgr2=0x7ffb40070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:35.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.492+0000 7ffb44a9a640 1 -- 172.21.15.169:0/4159434132 shutdown_connections 2024-03-28T01:02:35.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.494+0000 7ffb44a9a640 1 -- 172.21.15.169:0/4159434132 wait complete. 2024-03-28T01:02:35.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.496+0000 7ffb44a9a640 1 Processor -- start 2024-03-28T01:02:35.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.499+0000 7ffb44a9a640 1 -- start start 2024-03-28T01:02:35.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.499+0000 7ffb44a9a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb401086a0 0x7ffb401a6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:35.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.499+0000 7ffb44a9a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb4010b010 0x7ffb401a6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:35.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.499+0000 7ffb44a9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb4010d9c0 0x7ffb401addb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:35.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.499+0000 7ffb44a9a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4011c1e0 con 0x7ffb4010d9c0 2024-03-28T01:02:35.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.499+0000 7ffb44a9a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4011c060 con 0x7ffb4010b010 2024-03-28T01:02:35.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.499+0000 7ffb44a9a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4011c360 con 0x7ffb401086a0 2024-03-28T01:02:35.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.502+0000 7ffb3e59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb401086a0 0x7ffb401a6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:35.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.502+0000 7ffb3e59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb401086a0 0x7ffb401a6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:47348/0 (socket says 172.21.15.169:47348) 2024-03-28T01:02:35.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.502+0000 7ffb3e59c640 1 -- 172.21.15.169:0/2988469383 learned_addr learned my addr 172.21.15.169:0/2988469383 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:35.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.502+0000 7ffb3e59c640 1 -- 172.21.15.169:0/2988469383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb4010b010 msgr2=0x7ffb401a6d40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:02:35.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.502+0000 7ffb3e59c640 1 --2- 172.21.15.169:0/2988469383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb4010b010 0x7ffb401a6d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.502+0000 7ffb3e59c640 1 -- 172.21.15.169:0/2988469383 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb4010d9c0 msgr2=0x7ffb401addb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:35.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.502+0000 7ffb3e59c640 1 --2- 172.21.15.169:0/2988469383 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb4010d9c0 0x7ffb401addb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.502+0000 7ffb3e59c640 1 -- 172.21.15.169:0/2988469383 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4010ff00 con 0x7ffb401086a0 2024-03-28T01:02:35.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.502+0000 7ffb3e59c640 1 --2- 172.21.15.169:0/2988469383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb401086a0 0x7ffb401a6800 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ffb3002c290 tx=0x7ffb300040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:35.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.504+0000 7ffb2f7fe640 1 -- 172.21.15.169:0/2988469383 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30043020 con 0x7ffb401086a0 2024-03-28T01:02:35.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.504+0000 7ffb2f7fe640 1 -- 172.21.15.169:0/2988469383 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb300044c0 con 0x7ffb401086a0 2024-03-28T01:02:35.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.504+0000 7ffb2f7fe640 1 -- 172.21.15.169:0/2988469383 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30003e10 con 0x7ffb401086a0 2024-03-28T01:02:35.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.506+0000 7ffb44a9a640 1 -- 172.21.15.169:0/2988469383 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb40110190 con 0x7ffb401086a0 2024-03-28T01:02:35.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.511+0000 7ffb44a9a640 1 -- 172.21.15.169:0/2988469383 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb40110690 con 0x7ffb401086a0 2024-03-28T01:02:35.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.516+0000 7ffb2f7fe640 1 -- 172.21.15.169:0/2988469383 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ffb30046020 con 0x7ffb401086a0 2024-03-28T01:02:35.519 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.517+0000 7ffb2f7fe640 1 --2- 172.21.15.169:0/2988469383 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffb280a29d0 0x7ffb280a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:35.520 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.517+0000 7ffb2f7fe640 1 -- 172.21.15.169:0/2988469383 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ffb300f0f10 con 0x7ffb401086a0 2024-03-28T01:02:35.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.520+0000 7ffb47765740 1 Processor -- start 2024-03-28T01:02:35.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.523+0000 7ffb47765740 1 -- start start 2024-03-28T01:02:35.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.523+0000 7ffb47765740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d172da0 0x55d24d1731c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:35.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.523+0000 7ffb47765740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d173790 0x55d24d178290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:35.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.523+0000 7ffb47765740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d24d178900 0x55d24d17ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:35.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.523+0000 7ffb47765740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d24d0d83d0 con 0x55d24d173790 2024-03-28T01:02:35.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.523+0000 7ffb47765740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d24d080ad0 con 0x55d24d178900 2024-03-28T01:02:35.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.523+0000 7ffb47765740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d24d0d4470 con 0x55d24d172da0 2024-03-28T01:02:35.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.523+0000 7ffb3e59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d172da0 0x55d24d1731c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:35.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.523+0000 7ffb3e59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d172da0 0x55d24d1731c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:47358/0 (socket says 172.21.15.169:47358) 2024-03-28T01:02:35.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.523+0000 7ffb3e59c640 1 -- 172.21.15.169:0/2147725498 learned_addr learned my addr 172.21.15.169:0/2147725498 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:35.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.523+0000 7ffb3e59c640 1 -- 172.21.15.169:0/2147725498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d24d178900 msgr2=0x55d24d17ad90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:02:35.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.523+0000 7ffb3e59c640 1 --2- 172.21.15.169:0/2147725498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d24d178900 0x55d24d17ad90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.523+0000 7ffb3e59c640 1 -- 172.21.15.169:0/2147725498 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d173790 msgr2=0x55d24d178290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:35.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.523+0000 7ffb3e59c640 1 --2- 172.21.15.169:0/2147725498 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d173790 0x55d24d178290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.523+0000 7ffb3e59c640 1 -- 172.21.15.169:0/2147725498 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d24ceb67d0 con 0x55d24d172da0 2024-03-28T01:02:35.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.524+0000 7ffb3e59c640 1 --2- 172.21.15.169:0/2147725498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d172da0 0x55d24d1731c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ffb300ee580 tx=0x7ffb3002bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4fcf977a4caf978 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:35.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.526+0000 7ffb3dd9b640 1 --2- 172.21.15.169:0/2988469383 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffb280a29d0 0x7ffb280a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:35.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.526+0000 7ffb2cff9640 1 -- 172.21.15.169:0/2147725498 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3003c350 con 0x55d24d172da0 2024-03-28T01:02:35.531 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.527+0000 7ffb2cff9640 1 -- 172.21.15.169:0/2147725498 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb300bdbb0 con 0x55d24d172da0 2024-03-28T01:02:35.531 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.527+0000 7ffb2cff9640 1 -- 172.21.15.169:0/2147725498 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb300c57c0 con 0x55d24d172da0 2024-03-28T01:02:35.531 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.528+0000 7ffb47765740 1 -- 172.21.15.169:0/2147725498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d172da0 msgr2=0x55d24d1731c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:35.531 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.528+0000 7ffb47765740 1 --2- 172.21.15.169:0/2147725498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d172da0 0x55d24d1731c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ffb300ee580 tx=0x7ffb3002bc80 comp rx=0 tx=0).stop 2024-03-28T01:02:35.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.529+0000 7ffb47765740 1 -- 172.21.15.169:0/2147725498 shutdown_connections 2024-03-28T01:02:35.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.529+0000 7ffb47765740 1 --2- 172.21.15.169:0/2147725498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d24d178900 0x55d24d17ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.529+0000 7ffb47765740 1 --2- 172.21.15.169:0/2147725498 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d173790 0x55d24d178290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.529+0000 7ffb47765740 1 --2- 172.21.15.169:0/2147725498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d172da0 0x55d24d1731c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.529+0000 7ffb47765740 1 -- 172.21.15.169:0/2147725498 >> 172.21.15.169:0/2147725498 conn(0x55d24d16e310 msgr2=0x55d24d16e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:35.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.531+0000 7ffb3dd9b640 1 --2- 172.21.15.169:0/2988469383 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffb280a29d0 0x7ffb280a4e90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x55d24d173b90 tx=0x7ffb34005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:35.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.533+0000 7ffb47765740 1 -- 172.21.15.169:0/2147725498 shutdown_connections 2024-03-28T01:02:35.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.538+0000 7ffb47765740 1 -- 172.21.15.169:0/2147725498 wait complete. 2024-03-28T01:02:35.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.538+0000 7ffb47765740 1 -- start start 2024-03-28T01:02:35.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.538+0000 7ffb47765740 1 Processor -- start 2024-03-28T01:02:35.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.539+0000 7ffb47765740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d172da0 0x55d24d17cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:35.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.539+0000 7ffb47765740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d24d173790 0x55d24d17d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:35.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.539+0000 7ffb47765740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d178900 0x55d24d184500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:35.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.539+0000 7ffb47765740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d24d17b670 con 0x55d24d178900 2024-03-28T01:02:35.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.539+0000 7ffb47765740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d24d094060 con 0x55d24d173790 2024-03-28T01:02:35.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.539+0000 7ffb47765740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d24d17b7f0 con 0x55d24d172da0 2024-03-28T01:02:35.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.539+0000 7ffb47765740 10 client.0 _ll_register_callbacks cb 0x55d24d0766a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:35.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.539+0000 7ffb3ed9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d178900 0x55d24d184500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:35.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.539+0000 7ffb3ed9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d178900 0x55d24d184500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39570/0 (socket says 172.21.15.169:39570) 2024-03-28T01:02:35.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.539+0000 7ffb3ed9d640 1 -- 172.21.15.169:0/792735557 learned_addr learned my addr 172.21.15.169:0/792735557 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:35.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.539+0000 7ffb3e59c640 1 --2- 172.21.15.169:0/792735557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d172da0 0x55d24d17cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:35.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.540+0000 7ffb3dd9b640 1 --2- 172.21.15.169:0/792735557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d24d173790 0x55d24d17d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:35.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.540+0000 7ffb3e59c640 1 -- 172.21.15.169:0/792735557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d24d173790 msgr2=0x55d24d17d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:35.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.540+0000 7ffb3e59c640 1 --2- 172.21.15.169:0/792735557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d24d173790 0x55d24d17d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.540+0000 7ffb3e59c640 1 -- 172.21.15.169:0/792735557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d178900 msgr2=0x55d24d184500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:35.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.540+0000 7ffb3e59c640 1 --2- 172.21.15.169:0/792735557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d178900 0x55d24d184500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.540+0000 7ffb3e59c640 1 -- 172.21.15.169:0/792735557 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d24cdec3f0 con 0x55d24d172da0 2024-03-28T01:02:35.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.540+0000 7ffb3ed9d640 1 --2- 172.21.15.169:0/792735557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d178900 0x55d24d184500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:02:35.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.540+0000 7ffb3e59c640 1 --2- 172.21.15.169:0/792735557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d172da0 0x55d24d17cf80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ffb300ee580 tx=0x7ffb300c5cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=703f911cf9a384c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:35.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.540+0000 7ffb25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:02:35.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.541+0000 7ffb25ffb640 1 -- 172.21.15.169:0/792735557 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb300bb7b0 con 0x55d24d172da0 2024-03-28T01:02:35.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.541+0000 7ffb47765740 1 -- 172.21.15.169:0/792735557 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb300bdbb0 con 0x55d24d172da0 2024-03-28T01:02:35.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.541+0000 7ffb25ffb640 1 -- 172.21.15.169:0/792735557 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ffb300c57c0 con 0x55d24d172da0 2024-03-28T01:02:35.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.541+0000 7ffb25ffb640 1 -- 172.21.15.169:0/792735557 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb300ce050 con 0x55d24d172da0 2024-03-28T01:02:35.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.541+0000 7ffb47765740 10 client.26404 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:35.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.541+0000 7ffb47765740 1 -- 172.21.15.169:0/792735557 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d24cdac0f0 con 0x55d24d172da0 2024-03-28T01:02:35.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.543+0000 7ffb25ffb640 1 -- 172.21.15.169:0/792735557 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb300bdbb0 con 0x55d24d172da0 2024-03-28T01:02:35.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.543+0000 7ffb25ffb640 1 -- 172.21.15.169:0/792735557 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ffb300c10d0 con 0x55d24d172da0 2024-03-28T01:02:35.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.543+0000 7ffb47765740 10 client.26404 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:35.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.543+0000 7ffb47765740 10 client.26404 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:02:35.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.543+0000 7ffb47765740 20 client.26404 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:35.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.543+0000 7ffb47765740 1 --2- 172.21.15.169:0/792735557 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d24d188660 0x55d24d18aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:35.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.543+0000 7ffb47765740 4 client.26404 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:02:35.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.543+0000 7ffb47765740 1 -- 172.21.15.169:0/792735557 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d24d0e5910 con 0x55d24d188660 2024-03-28T01:02:35.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.544+0000 7ffb3dd9b640 1 --2- 172.21.15.169:0/792735557 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d24d188660 0x55d24d18aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:35.552 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:35 smithi169 ceph-mon[32274]: pgmap v533: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 650 MiB/s wr, 1.48k op/s 2024-03-28T01:02:35.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.549+0000 7ffb3dd9b640 1 --2- 172.21.15.169:0/792735557 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d24d188660 0x55d24d18aa50 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81f42a3602302674 server_cookie=71d67918a7f84798 in_seq=0 out_seq=0 2024-03-28T01:02:35.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.551+0000 7ffb25ffb640 10 client.26404 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:35.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.554+0000 7ffb25ffb640 1 -- 172.21.15.169:0/792735557 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d24d0e5910 con 0x55d24d188660 2024-03-28T01:02:35.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.554+0000 7ffb25ffb640 10 client.26404 handle_command_reply: tid=0 2024-03-28T01:02:35.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.554+0000 7ffb47765740 1 client.26404 shutdown 2024-03-28T01:02:35.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.564+0000 7ffb47765740 1 -- 172.21.15.169:0/792735557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d172da0 msgr2=0x55d24d17cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:35.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.564+0000 7ffb47765740 1 --2- 172.21.15.169:0/792735557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d172da0 0x55d24d17cf80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ffb300ee580 tx=0x7ffb300c5cc0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.566+0000 7ffb47765740 1 -- 172.21.15.169:0/792735557 shutdown_connections 2024-03-28T01:02:35.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.566+0000 7ffb47765740 1 --2- 172.21.15.169:0/792735557 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d24d188660 0x55d24d18aa50 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.566+0000 7ffb47765740 1 --2- 172.21.15.169:0/792735557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d178900 0x55d24d184500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.566+0000 7ffb47765740 1 --2- 172.21.15.169:0/792735557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d24d173790 0x55d24d17d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.566+0000 7ffb47765740 1 --2- 172.21.15.169:0/792735557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d172da0 0x55d24d17cf80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.566+0000 7ffb47765740 1 -- 172.21.15.169:0/792735557 >> 172.21.15.169:0/792735557 conn(0x55d24d16e310 msgr2=0x55d24d16d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:35.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.566+0000 7ffb25ffb640 0 client.26404 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:35.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.568+0000 7ffb47765740 1 -- 172.21.15.169:0/792735557 shutdown_connections 2024-03-28T01:02:35.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.570+0000 7ffb47765740 1 -- 172.21.15.169:0/792735557 wait complete. 2024-03-28T01:02:35.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.570+0000 7ffb47765740 20 client.26404 trim_cache size 0 max 16384 2024-03-28T01:02:35.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.608+0000 7ffb47765740 1 Processor -- start 2024-03-28T01:02:35.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.608+0000 7ffb47765740 1 -- start start 2024-03-28T01:02:35.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.608+0000 7ffb47765740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d172da0 0x55d24d17d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:35.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.608+0000 7ffb47765740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d24d173790 0x55d24d1844c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:35.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.608+0000 7ffb47765740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d178900 0x55d24d18f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:35.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.608+0000 7ffb47765740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb401c3df0 con 0x55d24d178900 2024-03-28T01:02:35.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.608+0000 7ffb47765740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb401c18f0 con 0x55d24d173790 2024-03-28T01:02:35.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.608+0000 7ffb47765740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb40118350 con 0x55d24d172da0 2024-03-28T01:02:35.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.609+0000 7ffb3dd9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d24d173790 0x55d24d1844c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:35.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.609+0000 7ffb3ed9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d178900 0x55d24d18f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:35.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.609+0000 7ffb3e59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d172da0 0x55d24d17d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:35.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.609+0000 7ffb3dd9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d24d173790 0x55d24d1844c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:49712/0 (socket says 172.21.15.169:49712) 2024-03-28T01:02:35.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.609+0000 7ffb3dd9b640 1 -- 172.21.15.169:0/2227909103 learned_addr learned my addr 172.21.15.169:0/2227909103 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:35.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.609+0000 7ffb3dd9b640 1 -- 172.21.15.169:0/2227909103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d172da0 msgr2=0x55d24d17d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:35.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.609+0000 7ffb3dd9b640 1 --2- 172.21.15.169:0/2227909103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d172da0 0x55d24d17d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.609+0000 7ffb3dd9b640 1 -- 172.21.15.169:0/2227909103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d178900 msgr2=0x55d24d18f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:35.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.609+0000 7ffb3dd9b640 1 --2- 172.21.15.169:0/2227909103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d178900 0x55d24d18f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.609+0000 7ffb3dd9b640 1 -- 172.21.15.169:0/2227909103 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d24d169b60 con 0x55d24d173790 2024-03-28T01:02:35.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.610+0000 7ffb3dd9b640 1 --2- 172.21.15.169:0/2227909103 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d24d173790 0x55d24d1844c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ffb401a7d10 tx=0x7ffb34012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d68352b2b54aed45 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:35.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.611+0000 7ffb27fff640 1 -- 172.21.15.169:0/2227909103 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3400ee60 con 0x55d24d173790 2024-03-28T01:02:35.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.611+0000 7ffb27fff640 1 -- 172.21.15.169:0/2227909103 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb34029070 con 0x55d24d173790 2024-03-28T01:02:35.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.611+0000 7ffb27fff640 1 -- 172.21.15.169:0/2227909103 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3401c710 con 0x55d24d173790 2024-03-28T01:02:35.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.612+0000 7ffb47765740 1 -- 172.21.15.169:0/2227909103 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d24d173790 msgr2=0x55d24d1844c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:35.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.612+0000 7ffb47765740 1 --2- 172.21.15.169:0/2227909103 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d24d173790 0x55d24d1844c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ffb401a7d10 tx=0x7ffb34012000 comp rx=0 tx=0).stop 2024-03-28T01:02:35.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.617+0000 7ffb47765740 1 -- 172.21.15.169:0/2227909103 shutdown_connections 2024-03-28T01:02:35.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.617+0000 7ffb47765740 1 --2- 172.21.15.169:0/2227909103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d178900 0x55d24d18f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.617+0000 7ffb47765740 1 --2- 172.21.15.169:0/2227909103 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d24d173790 0x55d24d1844c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.617+0000 7ffb47765740 1 --2- 172.21.15.169:0/2227909103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d172da0 0x55d24d17d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.617+0000 7ffb47765740 1 -- 172.21.15.169:0/2227909103 >> 172.21.15.169:0/2227909103 conn(0x55d24d16e310 msgr2=0x55d24d171e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:35.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.621+0000 7ffb47765740 1 -- 172.21.15.169:0/2227909103 shutdown_connections 2024-03-28T01:02:35.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.621+0000 7ffb47765740 1 -- 172.21.15.169:0/2227909103 wait complete. 2024-03-28T01:02:35.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.622+0000 7ffb47765740 1 -- start start 2024-03-28T01:02:35.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.622+0000 7ffb47765740 1 Processor -- start 2024-03-28T01:02:35.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.623+0000 7ffb47765740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d24d172da0 0x55d24d1822f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:35.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.623+0000 7ffb47765740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d173790 0x55d24d182830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:35.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.623+0000 7ffb47765740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d178900 0x55d24d183840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:35.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.623+0000 7ffb47765740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb401bff90 con 0x55d24d178900 2024-03-28T01:02:35.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.623+0000 7ffb47765740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb401c1290 con 0x55d24d172da0 2024-03-28T01:02:35.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.623+0000 7ffb47765740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb401c05d0 con 0x55d24d173790 2024-03-28T01:02:35.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.623+0000 7ffb47765740 10 client.0 _ll_register_callbacks cb 0x7ffb300b9840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:35.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.623+0000 7ffb3ed9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d178900 0x55d24d183840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:35.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.623+0000 7ffb3ed9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d178900 0x55d24d183840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39588/0 (socket says 172.21.15.169:39588) 2024-03-28T01:02:35.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.624+0000 7ffb3ed9d640 1 -- 172.21.15.169:0/128699857 learned_addr learned my addr 172.21.15.169:0/128699857 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:35.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.624+0000 7ffb3dd9b640 1 --2- 172.21.15.169:0/128699857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d173790 0x55d24d182830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:35.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.624+0000 7ffb3e59c640 1 --2- 172.21.15.169:0/128699857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d24d172da0 0x55d24d1822f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:35.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.624+0000 7ffb3dd9b640 1 -- 172.21.15.169:0/128699857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d24d172da0 msgr2=0x55d24d1822f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:35.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.624+0000 7ffb3dd9b640 1 --2- 172.21.15.169:0/128699857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d24d172da0 0x55d24d1822f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.624+0000 7ffb3dd9b640 1 -- 172.21.15.169:0/128699857 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d178900 msgr2=0x55d24d183840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:35.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.624+0000 7ffb3dd9b640 1 --2- 172.21.15.169:0/128699857 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d178900 0x55d24d183840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.624+0000 7ffb3dd9b640 1 -- 172.21.15.169:0/128699857 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d24d183e30 con 0x55d24d173790 2024-03-28T01:02:35.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.625+0000 7ffb3dd9b640 1 --2- 172.21.15.169:0/128699857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d173790 0x55d24d182830 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ffb3400d460 tx=0x7ffb3400efc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=546c9dd2fc29f530 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:35.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.625+0000 7ffb25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:02:35.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.625+0000 7ffb25ffb640 1 -- 172.21.15.169:0/128699857 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb34005230 con 0x55d24d173790 2024-03-28T01:02:35.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.625+0000 7ffb25ffb640 1 -- 172.21.15.169:0/128699857 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ffb340053d0 con 0x55d24d173790 2024-03-28T01:02:35.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.626+0000 7ffb47765740 1 -- 172.21.15.169:0/128699857 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb34029070 con 0x55d24d173790 2024-03-28T01:02:35.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.628+0000 7ffb25ffb640 1 -- 172.21.15.169:0/128699857 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb3403b050 con 0x55d24d173790 2024-03-28T01:02:35.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.628+0000 7ffb47765740 10 client.26407 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:35.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.628+0000 7ffb47765740 1 -- 172.21.15.169:0/128699857 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d24d1e4520 con 0x55d24d173790 2024-03-28T01:02:35.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.629+0000 7ffb25ffb640 1 -- 172.21.15.169:0/128699857 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb34029070 con 0x55d24d173790 2024-03-28T01:02:35.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.629+0000 7ffb25ffb640 1 -- 172.21.15.169:0/128699857 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ffb34004a60 con 0x55d24d173790 2024-03-28T01:02:35.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.629+0000 7ffb47765740 10 client.26407 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:35.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.629+0000 7ffb47765740 10 client.26407 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:02:35.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.629+0000 7ffb47765740 20 client.26407 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:35.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.629+0000 7ffb47765740 1 --2- 172.21.15.169:0/128699857 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d24d1e4e00 0x55d24d1e7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:35.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.629+0000 7ffb47765740 4 client.26407 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:02:35.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.629+0000 7ffb47765740 1 -- 172.21.15.169:0/128699857 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d24d0dfc60 con 0x55d24d1e4e00 2024-03-28T01:02:35.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.632+0000 7ffb3e59c640 1 --2- 172.21.15.169:0/128699857 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d24d1e4e00 0x55d24d1e7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:35.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.633+0000 7ffb3e59c640 1 --2- 172.21.15.169:0/128699857 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d24d1e4e00 0x55d24d1e7220 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdb86cf937170303 server_cookie=f64cf45413496b0f in_seq=0 out_seq=0 2024-03-28T01:02:35.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.633+0000 7ffb25ffb640 10 client.26407 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:35.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.633+0000 7ffb25ffb640 1 -- 172.21.15.169:0/128699857 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55d24d0dfc60 con 0x55d24d1e4e00 2024-03-28T01:02:35.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.633+0000 7ffb25ffb640 10 client.26407 handle_command_reply: tid=0 2024-03-28T01:02:35.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.633+0000 7ffb47765740 1 client.26407 shutdown 2024-03-28T01:02:35.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.635+0000 7ffb47765740 1 -- 172.21.15.169:0/128699857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d173790 msgr2=0x55d24d182830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:35.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.635+0000 7ffb47765740 1 --2- 172.21.15.169:0/128699857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d173790 0x55d24d182830 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ffb3400d460 tx=0x7ffb3400efc0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.636+0000 7ffb47765740 1 -- 172.21.15.169:0/128699857 shutdown_connections 2024-03-28T01:02:35.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.636+0000 7ffb47765740 1 --2- 172.21.15.169:0/128699857 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d24d1e4e00 0x55d24d1e7220 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.636+0000 7ffb47765740 1 --2- 172.21.15.169:0/128699857 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d24d178900 0x55d24d183840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.636+0000 7ffb47765740 1 --2- 172.21.15.169:0/128699857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24d173790 0x55d24d182830 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.636+0000 7ffb47765740 1 --2- 172.21.15.169:0/128699857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d24d172da0 0x55d24d1822f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.636+0000 7ffb25ffb640 0 client.26407 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:35.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.636+0000 7ffb47765740 1 -- 172.21.15.169:0/128699857 >> 172.21.15.169:0/128699857 conn(0x55d24d16e310 msgr2=0x55d24d171e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:35.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.639+0000 7ffb47765740 1 -- 172.21.15.169:0/128699857 shutdown_connections 2024-03-28T01:02:35.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.639+0000 7ffb47765740 1 -- 172.21.15.169:0/128699857 wait complete. 2024-03-28T01:02:35.643 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:02:35.643 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "no active scrubs running", 2024-03-28T01:02:35.643 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": {} 2024-03-28T01:02:35.643 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:02:35.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.639+0000 7ffb47765740 20 client.26407 trim_cache size 0 max 16384 2024-03-28T01:02:35.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.641+0000 7ffb44a9a640 1 -- 172.21.15.169:0/2988469383 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffb280a29d0 msgr2=0x7ffb280a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:35.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.641+0000 7ffb44a9a640 1 --2- 172.21.15.169:0/2988469383 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffb280a29d0 0x7ffb280a4e90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x55d24d173b90 tx=0x7ffb34005e70 comp rx=0 tx=0).stop 2024-03-28T01:02:35.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.641+0000 7ffb44a9a640 1 -- 172.21.15.169:0/2988469383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb401086a0 msgr2=0x7ffb401a6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:35.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.641+0000 7ffb44a9a640 1 --2- 172.21.15.169:0/2988469383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb401086a0 0x7ffb401a6800 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ffb3002c290 tx=0x7ffb300040c0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.642+0000 7ffb44a9a640 1 -- 172.21.15.169:0/2988469383 shutdown_connections 2024-03-28T01:02:35.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.642+0000 7ffb44a9a640 1 --2- 172.21.15.169:0/2988469383 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffb280a29d0 0x7ffb280a4e90 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.643+0000 7ffb44a9a640 1 --2- 172.21.15.169:0/2988469383 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb4010d9c0 0x7ffb401addb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.643+0000 7ffb44a9a640 1 --2- 172.21.15.169:0/2988469383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb4010b010 0x7ffb401a6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.643+0000 7ffb44a9a640 1 --2- 172.21.15.169:0/2988469383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb401086a0 0x7ffb401a6800 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.643+0000 7ffb44a9a640 1 -- 172.21.15.169:0/2988469383 >> 172.21.15.169:0/2988469383 conn(0x7ffb4006e500 msgr2=0x7ffb4010a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:35.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.644+0000 7ffb44a9a640 1 -- 172.21.15.169:0/2988469383 shutdown_connections 2024-03-28T01:02:35.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.644+0000 7ffb44a9a640 1 -- 172.21.15.169:0/2988469383 wait complete. 2024-03-28T01:02:35.678 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T01:02:35.678 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T01:02:35.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:35 smithi174 ceph-mon[36060]: pgmap v533: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 650 MiB/s wr, 1.48k op/s 2024-03-28T01:02:35.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:35 smithi177 ceph-mon[36206]: pgmap v533: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 650 MiB/s wr, 1.48k op/s 2024-03-28T01:02:35.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.965+0000 7faeda064640 1 -- 172.21.15.169:0/4052149750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faed40ffd60 msgr2=0x7faed4102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:35.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.965+0000 7faeda064640 1 --2- 172.21.15.169:0/4052149750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faed40ffd60 0x7faed4102190 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7faec4009900 tx=0x7faec402bc80 comp rx=0 tx=0).stop 2024-03-28T01:02:35.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.970+0000 7faeda064640 1 -- 172.21.15.169:0/4052149750 shutdown_connections 2024-03-28T01:02:35.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.970+0000 7faeda064640 1 --2- 172.21.15.169:0/4052149750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faed4105080 0x7faed4105500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.970+0000 7faeda064640 1 --2- 172.21.15.169:0/4052149750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faed41026d0 0x7faed4104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.970+0000 7faeda064640 1 --2- 172.21.15.169:0/4052149750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faed40ffd60 0x7faed4102190 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.970+0000 7faeda064640 1 -- 172.21.15.169:0/4052149750 >> 172.21.15.169:0/4052149750 conn(0x7faed40fb980 msgr2=0x7faed40fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:35.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.978+0000 7faeda064640 1 -- 172.21.15.169:0/4052149750 shutdown_connections 2024-03-28T01:02:35.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.982+0000 7faeda064640 1 -- 172.21.15.169:0/4052149750 wait complete. 2024-03-28T01:02:35.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.982+0000 7faeda064640 1 Processor -- start 2024-03-28T01:02:35.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.988+0000 7faeda064640 1 -- start start 2024-03-28T01:02:35.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.989+0000 7faeda064640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faed40ffd60 0x7faed4070140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:35.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.989+0000 7faeda064640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faed41026d0 0x7faed4070680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:35.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.989+0000 7faeda064640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faed4105080 0x7faed41adfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:35.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.989+0000 7faed9062640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faed40ffd60 0x7faed4070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:35.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.989+0000 7faed9062640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faed40ffd60 0x7faed4070140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:47404/0 (socket says 172.21.15.169:47404) 2024-03-28T01:02:35.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.989+0000 7faed9062640 1 -- 172.21.15.169:0/58996841 learned_addr learned my addr 172.21.15.169:0/58996841 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:35.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.989+0000 7faeda064640 1 -- 172.21.15.169:0/58996841 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed41138c0 con 0x7faed41026d0 2024-03-28T01:02:35.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.989+0000 7faeda064640 1 -- 172.21.15.169:0/58996841 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed4113740 con 0x7faed4105080 2024-03-28T01:02:35.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.989+0000 7faed9863640 1 --2- 172.21.15.169:0/58996841 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faed4105080 0x7faed41adfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:35.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.989+0000 7faeda064640 1 -- 172.21.15.169:0/58996841 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed4113a40 con 0x7faed40ffd60 2024-03-28T01:02:35.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.990+0000 7faed9062640 1 -- 172.21.15.169:0/58996841 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faed4105080 msgr2=0x7faed41adfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:35.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.990+0000 7faed9062640 1 --2- 172.21.15.169:0/58996841 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faed4105080 0x7faed41adfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.990+0000 7faed9062640 1 -- 172.21.15.169:0/58996841 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faed41026d0 msgr2=0x7faed4070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:35.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.990+0000 7faed9062640 1 --2- 172.21.15.169:0/58996841 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faed41026d0 0x7faed4070680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:35.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.990+0000 7faed9062640 1 -- 172.21.15.169:0/58996841 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed4070d20 con 0x7faed40ffd60 2024-03-28T01:02:35.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.991+0000 7faed9062640 1 --2- 172.21.15.169:0/58996841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faed40ffd60 0x7faed4070140 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7faec4005ab0 tx=0x7faec4004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:35.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.991+0000 7faeca7fc640 1 -- 172.21.15.169:0/58996841 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec4041070 con 0x7faed40ffd60 2024-03-28T01:02:35.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.991+0000 7faeca7fc640 1 -- 172.21.15.169:0/58996841 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faec4004ac0 con 0x7faed40ffd60 2024-03-28T01:02:35.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.992+0000 7faeca7fc640 1 -- 172.21.15.169:0/58996841 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec40346a0 con 0x7faed40ffd60 2024-03-28T01:02:35.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.993+0000 7faeda064640 1 -- 172.21.15.169:0/58996841 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed41ae5d0 con 0x7faed40ffd60 2024-03-28T01:02:35.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.993+0000 7faeda064640 1 -- 172.21.15.169:0/58996841 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed41ae980 con 0x7faed40ffd60 2024-03-28T01:02:35.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.996+0000 7faeca7fc640 1 -- 172.21.15.169:0/58996841 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7faec4044070 con 0x7faed40ffd60 2024-03-28T01:02:36.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.998+0000 7faeca7fc640 1 --2- 172.21.15.169:0/58996841 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faec00a28f0 0x7faec00a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:36.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.998+0000 7faeca7fc640 1 -- 172.21.15.169:0/58996841 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7faec40f01d0 con 0x7faed40ffd60 2024-03-28T01:02:36.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:35.999+0000 7faebffff640 1 -- 172.21.15.169:0/58996841 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed40683b0 con 0x7faed40ffd60 2024-03-28T01:02:36.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.000+0000 7faed8861640 1 --2- 172.21.15.169:0/58996841 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faec00a28f0 0x7faec00a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:36.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.002+0000 7faed8861640 1 --2- 172.21.15.169:0/58996841 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faec00a28f0 0x7faec00a4db0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7faed4071650 tx=0x7faed0006c40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:36.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.011+0000 7faeca7fc640 1 -- 172.21.15.169:0/58996841 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faec404b060 con 0x7faed40ffd60 2024-03-28T01:02:36.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.567+0000 7faebffff640 1 -- 172.21.15.169:0/58996841 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faed40625e0 con 0x7faed40ffd60 2024-03-28T01:02:36.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.570+0000 7faeca7fc640 1 -- 172.21.15.169:0/58996841 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7faec40bcf00 con 0x7faed40ffd60 2024-03-28T01:02:36.580 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T01:02:36.580 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T01:02:36.580 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T01:02:36.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.587+0000 7faebffff640 1 -- 172.21.15.169:0/58996841 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faec00a28f0 msgr2=0x7faec00a4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:36.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.587+0000 7faebffff640 1 --2- 172.21.15.169:0/58996841 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faec00a28f0 0x7faec00a4db0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7faed4071650 tx=0x7faed0006c40 comp rx=0 tx=0).stop 2024-03-28T01:02:36.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.587+0000 7faebffff640 1 -- 172.21.15.169:0/58996841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faed40ffd60 msgr2=0x7faed4070140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:36.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.587+0000 7faebffff640 1 --2- 172.21.15.169:0/58996841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faed40ffd60 0x7faed4070140 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7faec4005ab0 tx=0x7faec4004270 comp rx=0 tx=0).stop 2024-03-28T01:02:36.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.594+0000 7faebffff640 1 -- 172.21.15.169:0/58996841 shutdown_connections 2024-03-28T01:02:36.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.594+0000 7faebffff640 1 --2- 172.21.15.169:0/58996841 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7faec00a28f0 0x7faec00a4db0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.594+0000 7faebffff640 1 --2- 172.21.15.169:0/58996841 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faed4105080 0x7faed41adfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.594+0000 7faebffff640 1 --2- 172.21.15.169:0/58996841 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faed41026d0 0x7faed4070680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.594+0000 7faebffff640 1 --2- 172.21.15.169:0/58996841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faed40ffd60 0x7faed4070140 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.594+0000 7faebffff640 1 -- 172.21.15.169:0/58996841 >> 172.21.15.169:0/58996841 conn(0x7faed40fb980 msgr2=0x7faed4104620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:36.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.598+0000 7faebffff640 1 -- 172.21.15.169:0/58996841 shutdown_connections 2024-03-28T01:02:36.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.602+0000 7faebffff640 1 -- 172.21.15.169:0/58996841 wait complete. 2024-03-28T01:02:36.660 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:02:36.660 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T01:02:36.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.905+0000 7f6d96cd3640 1 -- 172.21.15.169:0/3779920087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d90108ed0 msgr2=0x7f6d90109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:36.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.905+0000 7f6d96cd3640 1 --2- 172.21.15.169:0/3779920087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d90108ed0 0x7f6d90109350 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f6d88009ca0 tx=0x7f6d8802c0f0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.905+0000 7f6d96cd3640 1 -- 172.21.15.169:0/3779920087 shutdown_connections 2024-03-28T01:02:36.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.905+0000 7f6d96cd3640 1 --2- 172.21.15.169:0/3779920087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d90109890 0x7f6d90110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.905+0000 7f6d96cd3640 1 --2- 172.21.15.169:0/3779920087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d90108ed0 0x7f6d90109350 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.905+0000 7f6d96cd3640 1 --2- 172.21.15.169:0/3779920087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d90107ce0 0x7f6d90108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.905+0000 7f6d96cd3640 1 -- 172.21.15.169:0/3779920087 >> 172.21.15.169:0/3779920087 conn(0x7f6d90076c70 msgr2=0x7f6d900790b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:36.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.905+0000 7f6d96cd3640 1 -- 172.21.15.169:0/3779920087 shutdown_connections 2024-03-28T01:02:36.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.906+0000 7f6d96cd3640 1 -- 172.21.15.169:0/3779920087 wait complete. 2024-03-28T01:02:36.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.907+0000 7f6d96cd3640 1 Processor -- start 2024-03-28T01:02:36.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.907+0000 7f6d96cd3640 1 -- start start 2024-03-28T01:02:36.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.907+0000 7f6d96cd3640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d90107ce0 0x7f6d900700b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:36.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.907+0000 7f6d96cd3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d90108ed0 0x7f6d900705f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:36.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.907+0000 7f6d96cd3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d90109890 0x7f6d901ae050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:36.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.907+0000 7f6d96cd3640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d90113940 con 0x7f6d90107ce0 2024-03-28T01:02:36.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.908+0000 7f6d96cd3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d901137c0 con 0x7f6d90108ed0 2024-03-28T01:02:36.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.908+0000 7f6d94a6f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d90107ce0 0x7f6d900700b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:36.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.908+0000 7f6d94a6f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d90107ce0 0x7f6d900700b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38906/0 (socket says 172.21.15.169:38906) 2024-03-28T01:02:36.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.908+0000 7f6d94a6f640 1 -- 172.21.15.169:0/2023567538 learned_addr learned my addr 172.21.15.169:0/2023567538 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:36.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.908+0000 7f6d96cd3640 1 -- 172.21.15.169:0/2023567538 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d90113ac0 con 0x7f6d90109890 2024-03-28T01:02:36.913 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.910+0000 7f6d95270640 1 --2- 172.21.15.169:0/2023567538 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d90109890 0x7f6d901ae050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:36.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.912+0000 7f6d8ffff640 1 --2- 172.21.15.169:0/2023567538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d90108ed0 0x7f6d900705f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:36.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.912+0000 7f6d94a6f640 1 -- 172.21.15.169:0/2023567538 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d90109890 msgr2=0x7f6d901ae050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:36.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.912+0000 7f6d94a6f640 1 --2- 172.21.15.169:0/2023567538 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d90109890 0x7f6d901ae050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.912+0000 7f6d94a6f640 1 -- 172.21.15.169:0/2023567538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d90108ed0 msgr2=0x7f6d900705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:36.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.912+0000 7f6d94a6f640 1 --2- 172.21.15.169:0/2023567538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d90108ed0 0x7f6d900705f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.912+0000 7f6d94a6f640 1 -- 172.21.15.169:0/2023567538 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d901ae590 con 0x7f6d90107ce0 2024-03-28T01:02:36.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.913+0000 7f6d94a6f640 1 --2- 172.21.15.169:0/2023567538 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d90107ce0 0x7f6d900700b0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800d770 tx=0x7f6d7800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:36.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.914+0000 7f6d8dffb640 1 -- 172.21.15.169:0/2023567538 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d78017070 con 0x7f6d90107ce0 2024-03-28T01:02:36.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.914+0000 7f6d8dffb640 1 -- 172.21.15.169:0/2023567538 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d78009c30 con 0x7f6d90107ce0 2024-03-28T01:02:36.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.914+0000 7f6d96cd3640 1 -- 172.21.15.169:0/2023567538 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d901ae760 con 0x7f6d90107ce0 2024-03-28T01:02:36.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.914+0000 7f6d8dffb640 1 -- 172.21.15.169:0/2023567538 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d78003dd0 con 0x7f6d90107ce0 2024-03-28T01:02:36.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.915+0000 7f6d96cd3640 1 -- 172.21.15.169:0/2023567538 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d901aec80 con 0x7f6d90107ce0 2024-03-28T01:02:36.921 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.918+0000 7f6d99995740 1 Processor -- start 2024-03-28T01:02:36.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.918+0000 7f6d99995740 1 -- start start 2024-03-28T01:02:36.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.919+0000 7f6d99995740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d6f4a6690 0x556d6f4a6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:36.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.919+0000 7f6d99995740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4a7080 0x556d6f4abb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:36.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.919+0000 7f6d99995740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556d6f4ac1f0 0x556d6f4ae680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:36.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.919+0000 7f6d99995740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d6f469610 con 0x556d6f4ac1f0 2024-03-28T01:02:36.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.919+0000 7f6d99995740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d6f40c3c0 con 0x556d6f4a7080 2024-03-28T01:02:36.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.919+0000 7f6d99995740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d6f3b4d60 con 0x556d6f4a6690 2024-03-28T01:02:36.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.919+0000 7f6d8dffb640 1 -- 172.21.15.169:0/2023567538 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6d78004000 con 0x7f6d90107ce0 2024-03-28T01:02:36.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.919+0000 7f6d8ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556d6f4ac1f0 0x556d6f4ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:36.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.919+0000 7f6d8ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556d6f4ac1f0 0x556d6f4ae680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38912/0 (socket says 172.21.15.169:38912) 2024-03-28T01:02:36.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.919+0000 7f6d8ffff640 1 -- 172.21.15.169:0/2276259685 learned_addr learned my addr 172.21.15.169:0/2276259685 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:36.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.919+0000 7f6d95270640 1 --2- 172.21.15.169:0/2276259685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d6f4a6690 0x556d6f4a6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:36.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.919+0000 7f6d8ffff640 1 -- 172.21.15.169:0/2276259685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d6f4a6690 msgr2=0x556d6f4a6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:36.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.919+0000 7f6d8ffff640 1 --2- 172.21.15.169:0/2276259685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d6f4a6690 0x556d6f4a6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.919+0000 7f6d8ffff640 1 -- 172.21.15.169:0/2276259685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4a7080 msgr2=0x556d6f4abb80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:02:36.924 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.919+0000 7f6d8ffff640 1 --2- 172.21.15.169:0/2276259685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4a7080 0x556d6f4abb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.924 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.919+0000 7f6d8ffff640 1 -- 172.21.15.169:0/2276259685 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d6f1203f0 con 0x556d6f4ac1f0 2024-03-28T01:02:36.924 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.920+0000 7f6d8ffff640 1 --2- 172.21.15.169:0/2276259685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556d6f4ac1f0 0x556d6f4ae680 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6d88009bf0 tx=0x7f6d880323c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9cabc8f134f5e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:36.924 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.921+0000 7f6d86ffd640 1 -- 172.21.15.169:0/2276259685 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d88039050 con 0x556d6f4ac1f0 2024-03-28T01:02:36.924 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.921+0000 7f6d86ffd640 1 -- 172.21.15.169:0/2276259685 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d88045070 con 0x556d6f4ac1f0 2024-03-28T01:02:36.924 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.921+0000 7f6d86ffd640 1 -- 172.21.15.169:0/2276259685 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d880042a0 con 0x556d6f4ac1f0 2024-03-28T01:02:36.924 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.921+0000 7f6d8dffb640 1 --2- 172.21.15.169:0/2023567538 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6d7c0a29d0 0x7f6d7c0a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:36.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.922+0000 7f6d8dffb640 1 -- 172.21.15.169:0/2023567538 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f6d780ce6f0 con 0x7f6d90107ce0 2024-03-28T01:02:36.926 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.923+0000 7f6d95270640 1 --2- 172.21.15.169:0/2023567538 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6d7c0a29d0 0x7f6d7c0a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:36.926 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.923+0000 7f6d99995740 1 -- 172.21.15.169:0/2276259685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556d6f4ac1f0 msgr2=0x556d6f4ae680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:36.926 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.923+0000 7f6d99995740 1 --2- 172.21.15.169:0/2276259685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556d6f4ac1f0 0x556d6f4ae680 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6d88009bf0 tx=0x7f6d880323c0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.926 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.923+0000 7f6d99995740 1 -- 172.21.15.169:0/2276259685 shutdown_connections 2024-03-28T01:02:36.926 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.923+0000 7f6d99995740 1 --2- 172.21.15.169:0/2276259685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556d6f4ac1f0 0x556d6f4ae680 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.926 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.923+0000 7f6d99995740 1 --2- 172.21.15.169:0/2276259685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4a7080 0x556d6f4abb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.927 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.923+0000 7f6d99995740 1 --2- 172.21.15.169:0/2276259685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d6f4a6690 0x556d6f4a6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.927 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.923+0000 7f6d99995740 1 -- 172.21.15.169:0/2276259685 >> 172.21.15.169:0/2276259685 conn(0x556d6f4a1c00 msgr2=0x556d6f4a2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:36.927 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.923+0000 7f6d99995740 1 -- 172.21.15.169:0/2276259685 shutdown_connections 2024-03-28T01:02:36.927 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.923+0000 7f6d99995740 1 -- 172.21.15.169:0/2276259685 wait complete. 2024-03-28T01:02:36.927 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.924+0000 7f6d99995740 1 -- start start 2024-03-28T01:02:36.927 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.924+0000 7f6d95270640 1 --2- 172.21.15.169:0/2023567538 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6d7c0a29d0 0x7f6d7c0a4e90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x556d6f4a78e0 tx=0x7f6d80005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:36.927 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.924+0000 7f6d99995740 1 Processor -- start 2024-03-28T01:02:36.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.925+0000 7f6d99995740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4a6690 0x556d6f4c1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:36.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.925+0000 7f6d99995740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556d6f4a7080 0x556d6f4c2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:36.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.925+0000 7f6d94a6f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4a6690 0x556d6f4c1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:36.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.926+0000 7f6d94a6f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4a6690 0x556d6f4c1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57622/0 (socket says 172.21.15.169:57622) 2024-03-28T01:02:36.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.926+0000 7f6d94a6f640 1 -- 172.21.15.169:0/2518524156 learned_addr learned my addr 172.21.15.169:0/2518524156 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:36.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.926+0000 7f6d99995740 1 --2- 172.21.15.169:0/2518524156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d6f4ac1f0 0x556d6f4c9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:36.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.926+0000 7f6d99995740 1 -- 172.21.15.169:0/2518524156 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d6f3c8240 con 0x556d6f4a7080 2024-03-28T01:02:36.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.926+0000 7f6d8ffff640 1 --2- 172.21.15.169:0/2518524156 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556d6f4a7080 0x556d6f4c2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:36.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.926+0000 7f6d99995740 1 -- 172.21.15.169:0/2518524156 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d6f408460 con 0x556d6f4a6690 2024-03-28T01:02:36.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.926+0000 7f6d99995740 1 -- 172.21.15.169:0/2518524156 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d6f4c0150 con 0x556d6f4ac1f0 2024-03-28T01:02:36.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.926+0000 7f6d99995740 10 client.0 _ll_register_callbacks cb 0x556d6f3c7c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:36.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.927+0000 7f6d94a6f640 1 -- 172.21.15.169:0/2518524156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d6f4ac1f0 msgr2=0x556d6f4c9010 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:02:36.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.927+0000 7f6d94a6f640 1 --2- 172.21.15.169:0/2518524156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d6f4ac1f0 0x556d6f4c9010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.927+0000 7f6d94a6f640 1 -- 172.21.15.169:0/2518524156 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556d6f4a7080 msgr2=0x556d6f4c2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:36.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.927+0000 7f6d94a6f640 1 --2- 172.21.15.169:0/2518524156 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556d6f4a7080 0x556d6f4c2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.927+0000 7f6d94a6f640 1 -- 172.21.15.169:0/2518524156 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556d6f1ea7d0 con 0x556d6f4a6690 2024-03-28T01:02:36.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.927+0000 7f6d94a6f640 1 --2- 172.21.15.169:0/2518524156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4a6690 0x556d6f4c1b00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6d780970f0 tx=0x7f6d7809aa60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64f7bbc7f4b54fa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:36.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.927+0000 7f6d6ffff640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:02:36.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.927+0000 7f6d6ffff640 1 -- 172.21.15.169:0/2518524156 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d780b2020 con 0x556d6f4a6690 2024-03-28T01:02:36.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.927+0000 7f6d99995740 1 -- 172.21.15.169:0/2518524156 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6d88045070 con 0x556d6f4a6690 2024-03-28T01:02:36.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.927+0000 7f6d6ffff640 1 -- 172.21.15.169:0/2518524156 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f6d780ce3f0 con 0x556d6f4a6690 2024-03-28T01:02:36.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.927+0000 7f6d6ffff640 1 -- 172.21.15.169:0/2518524156 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d78020040 con 0x556d6f4a6690 2024-03-28T01:02:36.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.928+0000 7f6d99995740 10 client.26495 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:36.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.928+0000 7f6d99995740 1 -- 172.21.15.169:0/2518524156 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556d6f4cc440 con 0x556d6f4a6690 2024-03-28T01:02:36.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.928+0000 7f6d6ffff640 1 -- 172.21.15.169:0/2518524156 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d88045070 con 0x556d6f4a6690 2024-03-28T01:02:36.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.928+0000 7f6d6ffff640 1 -- 172.21.15.169:0/2518524156 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f6d78097300 con 0x556d6f4a6690 2024-03-28T01:02:36.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.928+0000 7f6d99995740 10 client.26495 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:36.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.928+0000 7f6d99995740 10 client.26495 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:02:36.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.928+0000 7f6d99995740 20 client.26495 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:36.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.928+0000 7f6d99995740 1 --2- 172.21.15.169:0/2518524156 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x556d6f4cde90 0x556d6f4d02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:36.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.928+0000 7f6d99995740 4 client.26495 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:02:36.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.928+0000 7f6d99995740 1 -- 172.21.15.169:0/2518524156 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556d6f419900 con 0x556d6f4cde90 2024-03-28T01:02:36.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.929+0000 7f6d8ffff640 1 --2- 172.21.15.169:0/2518524156 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x556d6f4cde90 0x556d6f4d02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:36.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.930+0000 7f6d8ffff640 1 --2- 172.21.15.169:0/2518524156 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x556d6f4cde90 0x556d6f4d02c0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=43d5baee7a71609f server_cookie=395ccd8f12045679 in_seq=0 out_seq=0 2024-03-28T01:02:36.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.931+0000 7f6d6ffff640 10 client.26495 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:02:36.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.932+0000 7f6d6ffff640 1 -- 172.21.15.169:0/2518524156 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556d6f419900 con 0x556d6f4cde90 2024-03-28T01:02:36.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.932+0000 7f6d6ffff640 10 client.26495 handle_command_reply: tid=0 2024-03-28T01:02:36.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.932+0000 7f6d99995740 1 client.26495 shutdown 2024-03-28T01:02:36.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.933+0000 7f6d99995740 1 -- 172.21.15.169:0/2518524156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4a6690 msgr2=0x556d6f4c1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:36.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.933+0000 7f6d99995740 1 --2- 172.21.15.169:0/2518524156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4a6690 0x556d6f4c1b00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6d780970f0 tx=0x7f6d7809aa60 comp rx=0 tx=0).stop 2024-03-28T01:02:36.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.933+0000 7f6d99995740 1 -- 172.21.15.169:0/2518524156 shutdown_connections 2024-03-28T01:02:36.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.933+0000 7f6d99995740 1 --2- 172.21.15.169:0/2518524156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d6f4ac1f0 0x556d6f4c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.933+0000 7f6d99995740 1 --2- 172.21.15.169:0/2518524156 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x556d6f4cde90 0x556d6f4d02c0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.933+0000 7f6d99995740 1 --2- 172.21.15.169:0/2518524156 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556d6f4a7080 0x556d6f4c2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.933+0000 7f6d99995740 1 --2- 172.21.15.169:0/2518524156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4a6690 0x556d6f4c1b00 secure :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6d780970f0 tx=0x7f6d7809aa60 comp rx=0 tx=0).stop 2024-03-28T01:02:36.937 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.933+0000 7f6d99995740 1 -- 172.21.15.169:0/2518524156 >> 172.21.15.169:0/2518524156 conn(0x556d6f4a1c00 msgr2=0x556d6f4a0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:36.937 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.933+0000 7f6d6ffff640 0 client.26495 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:02:36.937 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.933+0000 7f6d99995740 1 -- 172.21.15.169:0/2518524156 shutdown_connections 2024-03-28T01:02:36.937 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.933+0000 7f6d99995740 1 -- 172.21.15.169:0/2518524156 wait complete. 2024-03-28T01:02:36.937 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.933+0000 7f6d99995740 20 client.26495 trim_cache size 0 max 16384 2024-03-28T01:02:36.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.962+0000 7f6d99995740 1 Processor -- start 2024-03-28T01:02:36.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.962+0000 7f6d99995740 1 -- start start 2024-03-28T01:02:36.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.963+0000 7f6d99995740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4a7080 0x556d6f4a8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:36.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.963+0000 7f6d99995740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556d6f4ac1f0 0x556d6f4a8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:36.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.963+0000 7f6d99995740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d6f4c8fd0 0x556d6f4c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:36.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.963+0000 7f6d94a6f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4a7080 0x556d6f4a8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:36.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.963+0000 7f6d94a6f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4a7080 0x556d6f4a8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57638/0 (socket says 172.21.15.169:57638) 2024-03-28T01:02:36.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.964+0000 7f6d94a6f640 1 -- 172.21.15.169:0/131247481 learned_addr learned my addr 172.21.15.169:0/131247481 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:36.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.964+0000 7f6d99995740 1 -- 172.21.15.169:0/131247481 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d901b2cf0 con 0x556d6f4ac1f0 2024-03-28T01:02:36.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.964+0000 7f6d99995740 1 -- 172.21.15.169:0/131247481 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d901b07f0 con 0x556d6f4a7080 2024-03-28T01:02:36.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.964+0000 7f6d99995740 1 -- 172.21.15.169:0/131247481 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d900820b0 con 0x556d6f4c8fd0 2024-03-28T01:02:36.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.965+0000 7f6d94a6f640 1 -- 172.21.15.169:0/131247481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d6f4c8fd0 msgr2=0x556d6f4c1710 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:02:36.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.965+0000 7f6d94a6f640 1 --2- 172.21.15.169:0/131247481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d6f4c8fd0 0x556d6f4c1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.965+0000 7f6d94a6f640 1 -- 172.21.15.169:0/131247481 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556d6f4ac1f0 msgr2=0x556d6f4a8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:36.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.965+0000 7f6d94a6f640 1 --2- 172.21.15.169:0/131247481 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556d6f4ac1f0 0x556d6f4a8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.965+0000 7f6d94a6f640 1 -- 172.21.15.169:0/131247481 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d6f4a2970 con 0x556d6f4a7080 2024-03-28T01:02:36.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.965+0000 7f6d94a6f640 1 --2- 172.21.15.169:0/131247481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4a7080 0x556d6f4a8690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6d78012040 tx=0x7f6d78004500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a4fc0752a8b882c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:36.976 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.973+0000 7f6d867fc640 1 -- 172.21.15.169:0/131247481 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d780b2070 con 0x556d6f4a7080 2024-03-28T01:02:36.976 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.973+0000 7f6d867fc640 1 -- 172.21.15.169:0/131247481 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d780b8030 con 0x556d6f4a7080 2024-03-28T01:02:36.976 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.973+0000 7f6d867fc640 1 -- 172.21.15.169:0/131247481 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d780b94e0 con 0x556d6f4a7080 2024-03-28T01:02:36.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.975+0000 7f6d99995740 1 -- 172.21.15.169:0/131247481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4a7080 msgr2=0x556d6f4a8690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:36.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.975+0000 7f6d99995740 1 --2- 172.21.15.169:0/131247481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4a7080 0x556d6f4a8690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6d78012040 tx=0x7f6d78004500 comp rx=0 tx=0).stop 2024-03-28T01:02:36.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.975+0000 7f6d99995740 1 -- 172.21.15.169:0/131247481 shutdown_connections 2024-03-28T01:02:36.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.975+0000 7f6d99995740 1 --2- 172.21.15.169:0/131247481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d6f4c8fd0 0x556d6f4c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.975+0000 7f6d99995740 1 --2- 172.21.15.169:0/131247481 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556d6f4ac1f0 0x556d6f4a8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.975+0000 7f6d99995740 1 --2- 172.21.15.169:0/131247481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4a7080 0x556d6f4a8690 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:36.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.975+0000 7f6d99995740 1 -- 172.21.15.169:0/131247481 >> 172.21.15.169:0/131247481 conn(0x556d6f4a1c00 msgr2=0x556d6f4a1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:36.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.978+0000 7f6d99995740 1 -- 172.21.15.169:0/131247481 shutdown_connections 2024-03-28T01:02:36.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.979+0000 7f6d99995740 1 -- 172.21.15.169:0/131247481 wait complete. 2024-03-28T01:02:36.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.979+0000 7f6d99995740 1 -- start start 2024-03-28T01:02:36.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.979+0000 7f6d99995740 1 Processor -- start 2024-03-28T01:02:36.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.982+0000 7f6d99995740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d6f4a7080 0x556d6f4c6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:36.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.982+0000 7f6d99995740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556d6f4ac1f0 0x556d6f4c6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:36.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.982+0000 7f6d99995740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4c8fd0 0x556d6f4c74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:36.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.982+0000 7f6d99995740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d901aeee0 con 0x556d6f4ac1f0 2024-03-28T01:02:36.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.982+0000 7f6d99995740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d901b0190 con 0x556d6f4c8fd0 2024-03-28T01:02:36.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.982+0000 7f6d99995740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d901af4d0 con 0x556d6f4a7080 2024-03-28T01:02:36.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.982+0000 7f6d99995740 10 client.0 _ll_register_callbacks cb 0x7f6d7800dec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:36.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.984+0000 7f6d95270640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4c8fd0 0x556d6f4c74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:36.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.984+0000 7f6d94a6f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d6f4a7080 0x556d6f4c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:36.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.984+0000 7f6d95270640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4c8fd0 0x556d6f4c74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57648/0 (socket says 172.21.15.169:57648) 2024-03-28T01:02:36.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.984+0000 7f6d95270640 1 -- 172.21.15.169:0/3033720472 learned_addr learned my addr 172.21.15.169:0/3033720472 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:36.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.988+0000 7f6d8ffff640 1 --2- 172.21.15.169:0/3033720472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556d6f4ac1f0 0x556d6f4c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:37.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.991+0000 7f6d94a6f640 1 -- 172.21.15.169:0/3033720472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4c8fd0 msgr2=0x556d6f4c74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:37.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.991+0000 7f6d94a6f640 1 --2- 172.21.15.169:0/3033720472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4c8fd0 0x556d6f4c74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.991+0000 7f6d94a6f640 1 -- 172.21.15.169:0/3033720472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556d6f4ac1f0 msgr2=0x556d6f4c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:37.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.991+0000 7f6d94a6f640 1 --2- 172.21.15.169:0/3033720472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556d6f4ac1f0 0x556d6f4c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.991+0000 7f6d94a6f640 1 -- 172.21.15.169:0/3033720472 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556d6f4c7ae0 con 0x556d6f4a7080 2024-03-28T01:02:37.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.991+0000 7f6d94a6f640 1 --2- 172.21.15.169:0/3033720472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d6f4a7080 0x556d6f4c6010 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6d780970f0 tx=0x7f6d78004590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75cc8ca2b919d132 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:37.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:36.991+0000 7f6d6ffff640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:02:37.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.004+0000 7f6d6ffff640 1 -- 172.21.15.169:0/3033720472 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d780b2070 con 0x556d6f4a7080 2024-03-28T01:02:37.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.004+0000 7f6d6ffff640 1 -- 172.21.15.169:0/3033720472 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f6d78079d80 con 0x556d6f4a7080 2024-03-28T01:02:37.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.005+0000 7f6d99995740 1 -- 172.21.15.169:0/3033720472 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6d780b8030 con 0x556d6f4a7080 2024-03-28T01:02:37.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.005+0000 7f6d6ffff640 1 -- 172.21.15.169:0/3033720472 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d7809ac90 con 0x556d6f4a7080 2024-03-28T01:02:37.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.008+0000 7f6d99995740 10 client.26425 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:37.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.008+0000 7f6d99995740 1 -- 172.21.15.169:0/3033720472 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556d6f4c8de0 con 0x556d6f4a7080 2024-03-28T01:02:37.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.012+0000 7f6d6ffff640 1 -- 172.21.15.169:0/3033720472 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d780b8030 con 0x556d6f4a7080 2024-03-28T01:02:37.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.012+0000 7f6d6ffff640 1 -- 172.21.15.169:0/3033720472 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f6d7809fd00 con 0x556d6f4a7080 2024-03-28T01:02:37.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.012+0000 7f6d99995740 10 client.26425 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:37.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.012+0000 7f6d99995740 10 client.26425 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:02:37.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.012+0000 7f6d99995740 20 client.26425 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:37.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.012+0000 7f6d99995740 1 --2- 172.21.15.169:0/3033720472 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x556d6f529c90 0x556d6f52c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:37.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.012+0000 7f6d99995740 4 client.26425 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:02:37.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.012+0000 7f6d99995740 1 -- 172.21.15.169:0/3033720472 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556d6f413c50 con 0x556d6f529c90 2024-03-28T01:02:37.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.018+0000 7f6d8ffff640 1 --2- 172.21.15.169:0/3033720472 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x556d6f529c90 0x556d6f52c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:37.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.023+0000 7f6d8ffff640 1 --2- 172.21.15.169:0/3033720472 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x556d6f529c90 0x556d6f52c0b0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9ef22f37e02dbcc5 server_cookie=970ee95b4bbbad71 in_seq=0 out_seq=0 2024-03-28T01:02:37.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.024+0000 7f6d6ffff640 10 client.26425 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:02:37.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.025+0000 7f6d6ffff640 1 -- 172.21.15.169:0/3033720472 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556d6f413c50 con 0x556d6f529c90 2024-03-28T01:02:37.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.026+0000 7f6d6ffff640 10 client.26425 handle_command_reply: tid=0 2024-03-28T01:02:37.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.026+0000 7f6d99995740 1 client.26425 shutdown 2024-03-28T01:02:37.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.028+0000 7f6d99995740 1 -- 172.21.15.169:0/3033720472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d6f4a7080 msgr2=0x556d6f4c6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:37.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.028+0000 7f6d99995740 1 --2- 172.21.15.169:0/3033720472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d6f4a7080 0x556d6f4c6010 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6d780970f0 tx=0x7f6d78004590 comp rx=0 tx=0).stop 2024-03-28T01:02:37.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.031+0000 7f6d99995740 1 -- 172.21.15.169:0/3033720472 shutdown_connections 2024-03-28T01:02:37.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.031+0000 7f6d99995740 1 --2- 172.21.15.169:0/3033720472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d6f4c8fd0 0x556d6f4c74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.031+0000 7f6d99995740 1 --2- 172.21.15.169:0/3033720472 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x556d6f529c90 0x556d6f52c0b0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.031+0000 7f6d99995740 1 --2- 172.21.15.169:0/3033720472 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556d6f4ac1f0 0x556d6f4c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.031+0000 7f6d99995740 1 --2- 172.21.15.169:0/3033720472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d6f4a7080 0x556d6f4c6010 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.031+0000 7f6d99995740 1 -- 172.21.15.169:0/3033720472 >> 172.21.15.169:0/3033720472 conn(0x556d6f4a1c00 msgr2=0x556d6f4a8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:37.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.032+0000 7f6d6ffff640 0 client.26425 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:02:37.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.040+0000 7f6d99995740 1 -- 172.21.15.169:0/3033720472 shutdown_connections 2024-03-28T01:02:37.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.040+0000 7f6d99995740 1 -- 172.21.15.169:0/3033720472 wait complete. 2024-03-28T01:02:37.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.040+0000 7f6d99995740 20 client.26425 trim_cache size 0 max 16384 2024-03-28T01:02:37.044 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:02:37.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.042+0000 7f6d96cd3640 1 -- 172.21.15.169:0/2023567538 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6d7c0a29d0 msgr2=0x7f6d7c0a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:37.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.042+0000 7f6d96cd3640 1 --2- 172.21.15.169:0/2023567538 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6d7c0a29d0 0x7f6d7c0a4e90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x556d6f4a78e0 tx=0x7f6d80005f90 comp rx=0 tx=0).stop 2024-03-28T01:02:37.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.042+0000 7f6d96cd3640 1 -- 172.21.15.169:0/2023567538 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d90107ce0 msgr2=0x7f6d900700b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:37.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.042+0000 7f6d96cd3640 1 --2- 172.21.15.169:0/2023567538 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d90107ce0 0x7f6d900700b0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800d770 tx=0x7f6d7800dc30 comp rx=0 tx=0).stop 2024-03-28T01:02:37.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.044+0000 7f6d96cd3640 1 -- 172.21.15.169:0/2023567538 shutdown_connections 2024-03-28T01:02:37.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.044+0000 7f6d96cd3640 1 --2- 172.21.15.169:0/2023567538 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f6d7c0a29d0 0x7f6d7c0a4e90 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.044+0000 7f6d96cd3640 1 --2- 172.21.15.169:0/2023567538 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d90109890 0x7f6d901ae050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.044+0000 7f6d96cd3640 1 --2- 172.21.15.169:0/2023567538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d90108ed0 0x7f6d900705f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.045+0000 7f6d96cd3640 1 --2- 172.21.15.169:0/2023567538 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d90107ce0 0x7f6d900700b0 secure :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800d770 tx=0x7f6d7800dc30 comp rx=0 tx=0).stop 2024-03-28T01:02:37.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.045+0000 7f6d96cd3640 1 -- 172.21.15.169:0/2023567538 >> 172.21.15.169:0/2023567538 conn(0x7f6d90076c70 msgr2=0x7f6d9010c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:37.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.048+0000 7f6d96cd3640 1 -- 172.21.15.169:0/2023567538 shutdown_connections 2024-03-28T01:02:37.053 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.050+0000 7f6d96cd3640 1 -- 172.21.15.169:0/2023567538 wait complete. 2024-03-28T01:02:37.093 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:02:37.094 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T01:02:37.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.526+0000 7f9b04a7a640 1 -- 172.21.15.169:0/3872650690 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b000ffdf0 msgr2=0x7f9b00102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:37.531 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.526+0000 7f9b04a7a640 1 --2- 172.21.15.169:0/3872650690 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b000ffdf0 0x7f9b00102220 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9af4009900 tx=0x7f9af402bc80 comp rx=0 tx=0).stop 2024-03-28T01:02:37.534 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.531+0000 7f9b04a7a640 1 -- 172.21.15.169:0/3872650690 shutdown_connections 2024-03-28T01:02:37.534 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.531+0000 7f9b04a7a640 1 --2- 172.21.15.169:0/3872650690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b00105110 0x7f9b00105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.534 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.531+0000 7f9b04a7a640 1 --2- 172.21.15.169:0/3872650690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b00102760 0x7f9b00104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.531+0000 7f9b04a7a640 1 --2- 172.21.15.169:0/3872650690 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b000ffdf0 0x7f9b00102220 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.531+0000 7f9b04a7a640 1 -- 172.21.15.169:0/3872650690 >> 172.21.15.169:0/3872650690 conn(0x7f9b000fba10 msgr2=0x7f9b000fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:37.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.531+0000 7f9b04a7a640 1 -- 172.21.15.169:0/3872650690 shutdown_connections 2024-03-28T01:02:37.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.531+0000 7f9b04a7a640 1 -- 172.21.15.169:0/3872650690 wait complete. 2024-03-28T01:02:37.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.532+0000 7f9b04a7a640 1 Processor -- start 2024-03-28T01:02:37.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.532+0000 7f9b04a7a640 1 -- start start 2024-03-28T01:02:37.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.533+0000 7f9b04a7a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b000ffdf0 0x7f9b0019dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:37.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.533+0000 7f9b04a7a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b00102760 0x7f9b0019e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:37.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.533+0000 7f9aff7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b000ffdf0 0x7f9b0019dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:37.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.533+0000 7f9aff7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b000ffdf0 0x7f9b0019dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38950/0 (socket says 172.21.15.169:38950) 2024-03-28T01:02:37.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.534+0000 7f9b04a7a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b00105110 0x7f9b001a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:37.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.534+0000 7f9aff7fe640 1 -- 172.21.15.169:0/48535993 learned_addr learned my addr 172.21.15.169:0/48535993 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:37.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.534+0000 7f9b04a7a640 1 -- 172.21.15.169:0/48535993 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b00113950 con 0x7f9b000ffdf0 2024-03-28T01:02:37.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.534+0000 7f9b04a7a640 1 -- 172.21.15.169:0/48535993 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b001137d0 con 0x7f9b00102760 2024-03-28T01:02:37.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.534+0000 7f9b04a7a640 1 -- 172.21.15.169:0/48535993 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b00113ad0 con 0x7f9b00105110 2024-03-28T01:02:37.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.540+0000 7f9afeffd640 1 --2- 172.21.15.169:0/48535993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b00102760 0x7f9b0019e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:37.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.540+0000 7f9afffff640 1 --2- 172.21.15.169:0/48535993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b00105110 0x7f9b001a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:37.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.542+0000 7f9aff7fe640 1 -- 172.21.15.169:0/48535993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b00105110 msgr2=0x7f9b001a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:37.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.542+0000 7f9aff7fe640 1 --2- 172.21.15.169:0/48535993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b00105110 0x7f9b001a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.542+0000 7f9aff7fe640 1 -- 172.21.15.169:0/48535993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b00102760 msgr2=0x7f9b0019e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:37.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.542+0000 7f9aff7fe640 1 --2- 172.21.15.169:0/48535993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b00102760 0x7f9b0019e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.542+0000 7f9aff7fe640 1 -- 172.21.15.169:0/48535993 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b001a5be0 con 0x7f9b000ffdf0 2024-03-28T01:02:37.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.543+0000 7f9aff7fe640 1 --2- 172.21.15.169:0/48535993 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b000ffdf0 0x7f9b0019dfc0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9af402c930 tx=0x7f9af4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:37.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.549+0000 7f9afcff9640 1 -- 172.21.15.169:0/48535993 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4041070 con 0x7f9b000ffdf0 2024-03-28T01:02:37.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.549+0000 7f9b04a7a640 1 -- 172.21.15.169:0/48535993 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b001a5e70 con 0x7f9b000ffdf0 2024-03-28T01:02:37.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.550+0000 7f9b04a7a640 1 -- 172.21.15.169:0/48535993 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b001a6220 con 0x7f9b000ffdf0 2024-03-28T01:02:37.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.549+0000 7f9afcff9640 1 -- 172.21.15.169:0/48535993 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9af4004ac0 con 0x7f9b000ffdf0 2024-03-28T01:02:37.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.553+0000 7f9afcff9640 1 -- 172.21.15.169:0/48535993 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4003e60 con 0x7f9b000ffdf0 2024-03-28T01:02:37.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.557+0000 7f9b089a4740 1 Processor -- start 2024-03-28T01:02:37.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.559+0000 7f9b089a4740 1 -- start start 2024-03-28T01:02:37.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.559+0000 7f9b089a4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f310ebc870 0x55f310ebcc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:37.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.559+0000 7f9b089a4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ebd260 0x55f310ec1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:37.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.559+0000 7f9b089a4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ec23d0 0x55f310ec4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:37.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.559+0000 7f9b089a4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f310e6e610 con 0x55f310ebc870 2024-03-28T01:02:37.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.559+0000 7f9b089a4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f310e113c0 con 0x55f310ec23d0 2024-03-28T01:02:37.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.559+0000 7f9b089a4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f310db9d60 con 0x55f310ebd260 2024-03-28T01:02:37.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.560+0000 7f9aff7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ebd260 0x55f310ec1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:37.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.561+0000 7f9aff7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ebd260 0x55f310ec1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:41926/0 (socket says 172.21.15.169:41926) 2024-03-28T01:02:37.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.561+0000 7f9aff7fe640 1 -- 172.21.15.169:0/149344033 learned_addr learned my addr 172.21.15.169:0/149344033 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:37.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.561+0000 7f9afeffd640 1 --2- 172.21.15.169:0/149344033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ec23d0 0x55f310ec4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:37.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.562+0000 7f9afeffd640 1 -- 172.21.15.169:0/149344033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ebd260 msgr2=0x55f310ec1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:37.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.562+0000 7f9afeffd640 1 --2- 172.21.15.169:0/149344033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ebd260 0x55f310ec1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.562+0000 7f9afeffd640 1 -- 172.21.15.169:0/149344033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f310ebc870 msgr2=0x55f310ebcc90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:02:37.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.562+0000 7f9afeffd640 1 --2- 172.21.15.169:0/149344033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f310ebc870 0x55f310ebcc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.562+0000 7f9afeffd640 1 -- 172.21.15.169:0/149344033 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f310b253f0 con 0x55f310ec23d0 2024-03-28T01:02:37.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.562+0000 7f9afcff9640 1 -- 172.21.15.169:0/48535993 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9af4044070 con 0x7f9b000ffdf0 2024-03-28T01:02:37.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.562+0000 7f9afeffd640 1 --2- 172.21.15.169:0/149344033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ec23d0 0x55f310ec4860 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9af8009ef0 tx=0x7f9af800c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63fe2233f2f82dd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:37.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.563+0000 7f9afcff9640 1 --2- 172.21.15.169:0/48535993 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9ae80a2a00 0x7f9ae80a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:37.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.564+0000 7f9afcff9640 1 -- 172.21.15.169:0/48535993 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f9af40f0c70 con 0x7f9b000ffdf0 2024-03-28T01:02:37.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.565+0000 7f9ae5ffb640 1 -- 172.21.15.169:0/149344033 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af801a070 con 0x55f310ec23d0 2024-03-28T01:02:37.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.565+0000 7f9ae5ffb640 1 -- 172.21.15.169:0/149344033 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9af80158f0 con 0x55f310ec23d0 2024-03-28T01:02:37.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.566+0000 7f9afffff640 1 --2- 172.21.15.169:0/48535993 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9ae80a2a00 0x7f9ae80a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:37.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.566+0000 7f9b089a4740 1 -- 172.21.15.169:0/149344033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ec23d0 msgr2=0x55f310ec4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:37.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.566+0000 7f9b089a4740 1 --2- 172.21.15.169:0/149344033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ec23d0 0x55f310ec4860 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9af8009ef0 tx=0x7f9af800c7f0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.584 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.582+0000 7f9b089a4740 1 -- 172.21.15.169:0/149344033 shutdown_connections 2024-03-28T01:02:37.584 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.582+0000 7f9b089a4740 1 --2- 172.21.15.169:0/149344033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ec23d0 0x55f310ec4860 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.582+0000 7f9b089a4740 1 --2- 172.21.15.169:0/149344033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ebd260 0x55f310ec1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.582+0000 7f9b089a4740 1 --2- 172.21.15.169:0/149344033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f310ebc870 0x55f310ebcc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.582+0000 7f9b089a4740 1 -- 172.21.15.169:0/149344033 >> 172.21.15.169:0/149344033 conn(0x55f310ea6dd0 msgr2=0x55f310ea71d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:37.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.588+0000 7f9b089a4740 1 -- 172.21.15.169:0/149344033 shutdown_connections 2024-03-28T01:02:37.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.589+0000 7f9afffff640 1 --2- 172.21.15.169:0/48535993 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9ae80a2a00 0x7f9ae80a4ec0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x55f310ebdac0 tx=0x7f9af0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:37.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.593+0000 7f9b089a4740 1 -- 172.21.15.169:0/149344033 wait complete. 2024-03-28T01:02:37.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.593+0000 7f9b089a4740 1 -- start start 2024-03-28T01:02:37.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.593+0000 7f9b089a4740 1 Processor -- start 2024-03-28T01:02:37.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.595+0000 7f9b089a4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f310ebc870 0x55f310ec6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:37.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.595+0000 7f9b089a4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ebd260 0x55f310ec7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:37.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.595+0000 7f9b089a4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ec23d0 0x55f310ece1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:37.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.595+0000 7f9b089a4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f310dcd240 con 0x55f310ebc870 2024-03-28T01:02:37.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.595+0000 7f9b089a4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f310e0d460 con 0x55f310ebd260 2024-03-28T01:02:37.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.595+0000 7f9b089a4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f310ec53d0 con 0x55f310ec23d0 2024-03-28T01:02:37.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.595+0000 7f9b089a4740 10 client.0 _ll_register_callbacks cb 0x55f310dccc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:37.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.596+0000 7f9afeffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ebd260 0x55f310ec7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:37.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.596+0000 7f9afeffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ebd260 0x55f310ec7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57700/0 (socket says 172.21.15.169:57700) 2024-03-28T01:02:37.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.596+0000 7f9afeffd640 1 -- 172.21.15.169:0/896070531 learned_addr learned my addr 172.21.15.169:0/896070531 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:37.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.597+0000 7f9afeffd640 1 -- 172.21.15.169:0/896070531 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ec23d0 msgr2=0x55f310ece1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:37.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.597+0000 7f9afeffd640 1 --2- 172.21.15.169:0/896070531 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ec23d0 0x55f310ece1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.597+0000 7f9afeffd640 1 -- 172.21.15.169:0/896070531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f310ebc870 msgr2=0x55f310ec6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:37.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.597+0000 7f9afeffd640 1 --2- 172.21.15.169:0/896070531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f310ebc870 0x55f310ec6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.597+0000 7f9afeffd640 1 -- 172.21.15.169:0/896070531 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f310bef7d0 con 0x55f310ebd260 2024-03-28T01:02:37.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.599+0000 7f9afeffd640 1 --2- 172.21.15.169:0/896070531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ebd260 0x55f310ec7200 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9af8015cd0 tx=0x7f9af8015d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0d915db94f35cb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:37.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.600+0000 7f9ae37f6640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:02:37.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.600+0000 7f9ae37f6640 1 -- 172.21.15.169:0/896070531 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af801a050 con 0x55f310ebd260 2024-03-28T01:02:37.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.600+0000 7f9b089a4740 1 -- 172.21.15.169:0/896070531 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9af80158f0 con 0x55f310ebd260 2024-03-28T01:02:37.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.600+0000 7f9ae37f6640 1 -- 172.21.15.169:0/896070531 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f9af8015620 con 0x55f310ebd260 2024-03-28T01:02:37.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.601+0000 7f9ae37f6640 1 -- 172.21.15.169:0/896070531 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9af802f050 con 0x55f310ebd260 2024-03-28T01:02:37.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.601+0000 7f9b089a4740 10 client.26510 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:37.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.601+0000 7f9b089a4740 1 -- 172.21.15.169:0/896070531 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f310ee2580 con 0x55f310ebd260 2024-03-28T01:02:37.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.602+0000 7f9ae37f6640 1 -- 172.21.15.169:0/896070531 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9af80158f0 con 0x55f310ebd260 2024-03-28T01:02:37.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.603+0000 7f9ae37f6640 1 -- 172.21.15.169:0/896070531 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9af80044a0 con 0x55f310ebd260 2024-03-28T01:02:37.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.604+0000 7f9b089a4740 10 client.26510 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:37.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.604+0000 7f9b089a4740 10 client.26510 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:02:37.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.605+0000 7f9b089a4740 20 client.26510 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:37.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.605+0000 7f9b089a4740 1 --2- 172.21.15.169:0/896070531 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55f310ee3940 0x55f310ee5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:37.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.605+0000 7f9b089a4740 4 client.26510 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:02:37.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.605+0000 7f9b089a4740 1 -- 172.21.15.169:0/896070531 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f310e1e900 con 0x55f310ee3940 2024-03-28T01:02:37.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.607+0000 7f9aff7fe640 1 --2- 172.21.15.169:0/896070531 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55f310ee3940 0x55f310ee5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:37.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.618+0000 7f9aff7fe640 1 --2- 172.21.15.169:0/896070531 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55f310ee3940 0x55f310ee5d70 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4aa7963e5654bcbe server_cookie=d34c3711943948c9 in_seq=0 out_seq=0 2024-03-28T01:02:37.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.618+0000 7f9ae37f6640 10 client.26510 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:02:37.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.620+0000 7f9ae37f6640 1 -- 172.21.15.169:0/896070531 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f310e1e900 con 0x55f310ee3940 2024-03-28T01:02:37.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.620+0000 7f9ae37f6640 10 client.26510 handle_command_reply: tid=0 2024-03-28T01:02:37.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.621+0000 7f9b089a4740 1 client.26510 shutdown 2024-03-28T01:02:37.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.629+0000 7f9b089a4740 1 -- 172.21.15.169:0/896070531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ebd260 msgr2=0x55f310ec7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:37.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.629+0000 7f9b089a4740 1 --2- 172.21.15.169:0/896070531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ebd260 0x55f310ec7200 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9af8015cd0 tx=0x7f9af8015d00 comp rx=0 tx=0).stop 2024-03-28T01:02:37.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.631+0000 7f9b089a4740 1 -- 172.21.15.169:0/896070531 shutdown_connections 2024-03-28T01:02:37.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.631+0000 7f9b089a4740 1 --2- 172.21.15.169:0/896070531 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55f310ee3940 0x55f310ee5d70 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.631+0000 7f9b089a4740 1 --2- 172.21.15.169:0/896070531 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ec23d0 0x55f310ece1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.631+0000 7f9b089a4740 1 --2- 172.21.15.169:0/896070531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ebd260 0x55f310ec7200 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.631+0000 7f9b089a4740 1 --2- 172.21.15.169:0/896070531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f310ebc870 0x55f310ec6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.631+0000 7f9b089a4740 1 -- 172.21.15.169:0/896070531 >> 172.21.15.169:0/896070531 conn(0x55f310ea6dd0 msgr2=0x55f310ea6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:37.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.632+0000 7f9ae37f6640 0 client.26510 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:02:37.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.633+0000 7f9b089a4740 1 -- 172.21.15.169:0/896070531 shutdown_connections 2024-03-28T01:02:37.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.633+0000 7f9b089a4740 1 -- 172.21.15.169:0/896070531 wait complete. 2024-03-28T01:02:37.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.635+0000 7f9b089a4740 20 client.26510 trim_cache size 0 max 16384 2024-03-28T01:02:37.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.680+0000 7f9b089a4740 1 Processor -- start 2024-03-28T01:02:37.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.681+0000 7f9b089a4740 1 -- start start 2024-03-28T01:02:37.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.681+0000 7f9b089a4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ebc870 0x55f310ebe760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:37.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.681+0000 7f9b089a4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ebd260 0x55f310ece190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:37.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.681+0000 7f9b089a4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f310ec23d0 0x55f310ebeca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:37.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.681+0000 7f9b089a4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aec012120 con 0x55f310ec23d0 2024-03-28T01:02:37.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.681+0000 7f9b089a4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aec00fc20 con 0x55f310ebc870 2024-03-28T01:02:37.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.681+0000 7f9b089a4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aec00dcc0 con 0x55f310ebd260 2024-03-28T01:02:37.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.682+0000 7f9afeffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ebd260 0x55f310ece190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:37.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.682+0000 7f9aff7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ebc870 0x55f310ebe760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:37.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.682+0000 7f9aff7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ebc870 0x55f310ebe760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57702/0 (socket says 172.21.15.169:57702) 2024-03-28T01:02:37.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.682+0000 7f9aff7fe640 1 -- 172.21.15.169:0/1566967905 learned_addr learned my addr 172.21.15.169:0/1566967905 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:37.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.682+0000 7f9afffff640 1 --2- 172.21.15.169:0/1566967905 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f310ec23d0 0x55f310ebeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:37.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.682+0000 7f9afeffd640 1 -- 172.21.15.169:0/1566967905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ebc870 msgr2=0x55f310ebe760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:37.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.682+0000 7f9afeffd640 1 --2- 172.21.15.169:0/1566967905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ebc870 0x55f310ebe760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.682+0000 7f9afeffd640 1 -- 172.21.15.169:0/1566967905 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f310ec23d0 msgr2=0x55f310ebeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:37.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.682+0000 7f9afeffd640 1 --2- 172.21.15.169:0/1566967905 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f310ec23d0 0x55f310ebeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.682+0000 7f9afeffd640 1 -- 172.21.15.169:0/1566967905 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f310ebdaf0 con 0x55f310ebd260 2024-03-28T01:02:37.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.685+0000 7f9afeffd640 1 --2- 172.21.15.169:0/1566967905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ebd260 0x55f310ece190 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9af80048c0 tx=0x7f9af800c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16bc390ee39b844f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:37.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.686+0000 7f9ae57fa640 1 -- 172.21.15.169:0/1566967905 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af801a070 con 0x55f310ebd260 2024-03-28T01:02:37.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.686+0000 7f9ae57fa640 1 -- 172.21.15.169:0/1566967905 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9af8026070 con 0x55f310ebd260 2024-03-28T01:02:37.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.686+0000 7f9ae57fa640 1 -- 172.21.15.169:0/1566967905 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af8021bb0 con 0x55f310ebd260 2024-03-28T01:02:37.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.688+0000 7f9b089a4740 1 -- 172.21.15.169:0/1566967905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ebd260 msgr2=0x55f310ece190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:37.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.688+0000 7f9b089a4740 1 --2- 172.21.15.169:0/1566967905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ebd260 0x55f310ece190 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9af80048c0 tx=0x7f9af800c000 comp rx=0 tx=0).stop 2024-03-28T01:02:37.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.688+0000 7f9b089a4740 1 -- 172.21.15.169:0/1566967905 shutdown_connections 2024-03-28T01:02:37.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.688+0000 7f9b089a4740 1 --2- 172.21.15.169:0/1566967905 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f310ec23d0 0x55f310ebeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.688+0000 7f9b089a4740 1 --2- 172.21.15.169:0/1566967905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ebd260 0x55f310ece190 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.688+0000 7f9b089a4740 1 --2- 172.21.15.169:0/1566967905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ebc870 0x55f310ebe760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.688+0000 7f9b089a4740 1 -- 172.21.15.169:0/1566967905 >> 172.21.15.169:0/1566967905 conn(0x55f310ea6dd0 msgr2=0x55f310ec6cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:37.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.696+0000 7f9b089a4740 1 -- 172.21.15.169:0/1566967905 shutdown_connections 2024-03-28T01:02:37.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.696+0000 7f9b089a4740 1 -- 172.21.15.169:0/1566967905 wait complete. 2024-03-28T01:02:37.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.697+0000 7f9b089a4740 1 -- start start 2024-03-28T01:02:37.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.697+0000 7f9b089a4740 1 Processor -- start 2024-03-28T01:02:37.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.697+0000 7f9b089a4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ebc870 0x55f310ecb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:37.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.698+0000 7f9b089a4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ebd260 0x55f310ecbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:37.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.698+0000 7f9aff7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ebc870 0x55f310ecb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:37.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.698+0000 7f9aff7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ebc870 0x55f310ecb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:41952/0 (socket says 172.21.15.169:41952) 2024-03-28T01:02:37.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.698+0000 7f9aff7fe640 1 -- 172.21.15.169:0/366678541 learned_addr learned my addr 172.21.15.169:0/366678541 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:37.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.698+0000 7f9afeffd640 1 --2- 172.21.15.169:0/366678541 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ebd260 0x55f310ecbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:37.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.698+0000 7f9b089a4740 1 --2- 172.21.15.169:0/366678541 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f310ec23d0 0x55f310ecce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:37.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.698+0000 7f9b089a4740 1 -- 172.21.15.169:0/366678541 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aec00e2c0 con 0x55f310ec23d0 2024-03-28T01:02:37.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.699+0000 7f9b089a4740 1 -- 172.21.15.169:0/366678541 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aec00f5c0 con 0x55f310ebd260 2024-03-28T01:02:37.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.699+0000 7f9afffff640 1 --2- 172.21.15.169:0/366678541 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f310ec23d0 0x55f310ecce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:37.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.699+0000 7f9b089a4740 1 -- 172.21.15.169:0/366678541 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aec00e900 con 0x55f310ebc870 2024-03-28T01:02:37.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.699+0000 7f9aff7fe640 1 -- 172.21.15.169:0/366678541 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ebd260 msgr2=0x55f310ecbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:37.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.699+0000 7f9aff7fe640 1 --2- 172.21.15.169:0/366678541 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ebd260 0x55f310ecbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.699+0000 7f9aff7fe640 1 -- 172.21.15.169:0/366678541 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f310ec23d0 msgr2=0x55f310ecce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:37.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.700+0000 7f9aff7fe640 1 --2- 172.21.15.169:0/366678541 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f310ec23d0 0x55f310ecce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.700+0000 7f9aff7fe640 1 -- 172.21.15.169:0/366678541 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f310ecd450 con 0x55f310ebc870 2024-03-28T01:02:37.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.700+0000 7f9b089a4740 10 client.0 _ll_register_callbacks cb 0x7f9af8009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:37.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.700+0000 7f9aff7fe640 1 --2- 172.21.15.169:0/366678541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ebc870 0x55f310ecb960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9af40bd1a0 tx=0x7f9af4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7363d79af2326269 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:37.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.700+0000 7f9b089a4740 1 -- 172.21.15.169:0/366678541 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9af8026070 con 0x55f310ebc870 2024-03-28T01:02:37.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.700+0000 7f9ae37f6640 10 client.26437 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:02:37.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.701+0000 7f9ae37f6640 1 -- 172.21.15.169:0/366678541 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af404b040 con 0x55f310ebc870 2024-03-28T01:02:37.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.701+0000 7f9ae37f6640 1 -- 172.21.15.169:0/366678541 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f9af40dd0b0 con 0x55f310ebc870 2024-03-28T01:02:37.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.701+0000 7f9ae37f6640 1 -- 172.21.15.169:0/366678541 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9af40bcd50 con 0x55f310ebc870 2024-03-28T01:02:37.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.705+0000 7f9b089a4740 10 client.26437 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:37.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.705+0000 7f9b089a4740 1 -- 172.21.15.169:0/366678541 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x55f310f3f240 con 0x55f310ebc870 2024-03-28T01:02:37.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.711+0000 7f9ae37f6640 1 -- 172.21.15.169:0/366678541 <== mon.1 v2:172.21.15.177:3300/0 4 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9af40b97f0 con 0x55f310ebc870 2024-03-28T01:02:37.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.711+0000 7f9b089a4740 10 client.26437 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:37.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.711+0000 7f9b089a4740 10 client.26437 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:02:37.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.711+0000 7f9b089a4740 20 client.26437 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:37.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.711+0000 7f9b089a4740 1 --2- 172.21.15.169:0/366678541 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55f310f3fa40 0x55f310f41f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:37.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.711+0000 7f9b089a4740 4 client.26437 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:02:37.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.711+0000 7f9b089a4740 1 -- 172.21.15.169:0/366678541 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f9af8026070 con 0x55f310f3fa40 2024-03-28T01:02:37.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.714+0000 7f9afeffd640 1 --2- 172.21.15.169:0/366678541 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55f310f3fa40 0x55f310f41f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:37.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.719+0000 7f9afeffd640 1 --2- 172.21.15.169:0/366678541 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55f310f3fa40 0x55f310f41f00 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7c0ecdb86f8ea854 server_cookie=fdf61e44daa5e595 in_seq=0 out_seq=0 2024-03-28T01:02:37.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.719+0000 7f9ae37f6640 10 client.26437 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:02:37.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.723+0000 7f9ae37f6640 1 -- 172.21.15.169:0/366678541 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f9af8026070 con 0x55f310f3fa40 2024-03-28T01:02:37.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.723+0000 7f9ae37f6640 10 client.26437 handle_command_reply: tid=0 2024-03-28T01:02:37.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.723+0000 7f9b089a4740 1 client.26437 shutdown 2024-03-28T01:02:37.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.724+0000 7f9b089a4740 1 -- 172.21.15.169:0/366678541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ebc870 msgr2=0x55f310ecb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:37.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.724+0000 7f9b089a4740 1 --2- 172.21.15.169:0/366678541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ebc870 0x55f310ecb960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9af40bd1a0 tx=0x7f9af4000bc0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.724+0000 7f9b089a4740 1 -- 172.21.15.169:0/366678541 shutdown_connections 2024-03-28T01:02:37.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.724+0000 7f9b089a4740 1 --2- 172.21.15.169:0/366678541 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55f310f3fa40 0x55f310f41f00 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.724+0000 7f9b089a4740 1 --2- 172.21.15.169:0/366678541 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f310ec23d0 0x55f310ecce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.724+0000 7f9b089a4740 1 --2- 172.21.15.169:0/366678541 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f310ebd260 0x55f310ecbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.724+0000 7f9b089a4740 1 --2- 172.21.15.169:0/366678541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f310ebc870 0x55f310ecb960 secure :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9af40bd1a0 tx=0x7f9af4000bc0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.724+0000 7f9b089a4740 1 -- 172.21.15.169:0/366678541 >> 172.21.15.169:0/366678541 conn(0x55f310ea6dd0 msgr2=0x55f310ec6ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:37.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.724+0000 7f9ae37f6640 0 client.26437 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:02:37.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.728+0000 7f9b089a4740 1 -- 172.21.15.169:0/366678541 shutdown_connections 2024-03-28T01:02:37.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.728+0000 7f9b089a4740 1 -- 172.21.15.169:0/366678541 wait complete. 2024-03-28T01:02:37.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.731+0000 7f9b089a4740 20 client.26437 trim_cache size 0 max 16384 2024-03-28T01:02:37.736 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:02:37.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.743+0000 7f9b04a7a640 1 -- 172.21.15.169:0/48535993 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9ae80a2a00 msgr2=0x7f9ae80a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:37.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.743+0000 7f9b04a7a640 1 --2- 172.21.15.169:0/48535993 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9ae80a2a00 0x7f9ae80a4ec0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x55f310ebdac0 tx=0x7f9af0008040 comp rx=0 tx=0).stop 2024-03-28T01:02:37.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.743+0000 7f9b04a7a640 1 -- 172.21.15.169:0/48535993 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b000ffdf0 msgr2=0x7f9b0019dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:37.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.743+0000 7f9b04a7a640 1 --2- 172.21.15.169:0/48535993 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b000ffdf0 0x7f9b0019dfc0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9af402c930 tx=0x7f9af4004270 comp rx=0 tx=0).stop 2024-03-28T01:02:37.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.744+0000 7f9b04a7a640 1 -- 172.21.15.169:0/48535993 shutdown_connections 2024-03-28T01:02:37.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.744+0000 7f9b04a7a640 1 --2- 172.21.15.169:0/48535993 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9ae80a2a00 0x7f9ae80a4ec0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.744+0000 7f9b04a7a640 1 --2- 172.21.15.169:0/48535993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b00105110 0x7f9b001a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.744+0000 7f9b04a7a640 1 --2- 172.21.15.169:0/48535993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b00102760 0x7f9b0019e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.744+0000 7f9b04a7a640 1 --2- 172.21.15.169:0/48535993 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b000ffdf0 0x7f9b0019dfc0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:37.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.744+0000 7f9b04a7a640 1 -- 172.21.15.169:0/48535993 >> 172.21.15.169:0/48535993 conn(0x7f9b000fba10 msgr2=0x7f9b00103120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:37.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.744+0000 7f9b04a7a640 1 -- 172.21.15.169:0/48535993 shutdown_connections 2024-03-28T01:02:37.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:37.745+0000 7f9b04a7a640 1 -- 172.21.15.169:0/48535993 wait complete. 2024-03-28T01:02:37.779 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:37 smithi169 ceph-mon[32274]: pgmap v534: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 716 MiB/s wr, 1.74k op/s 2024-03-28T01:02:37.779 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:37 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/58996841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:02:37.782 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:02:37.782 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T01:02:37.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:37 smithi174 ceph-mon[36060]: pgmap v534: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 716 MiB/s wr, 1.74k op/s 2024-03-28T01:02:37.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:37 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/58996841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:02:37.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:37 smithi177 ceph-mon[36206]: pgmap v534: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 716 MiB/s wr, 1.74k op/s 2024-03-28T01:02:37.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:37 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/58996841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:02:38.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.172+0000 7f38b8737640 1 -- 172.21.15.169:0/1922654164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38b010b010 msgr2=0x7f38b010d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:38.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.172+0000 7f38b8737640 1 --2- 172.21.15.169:0/1922654164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38b010b010 0x7f38b010d480 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f38a0009900 tx=0x7f38a002bc80 comp rx=0 tx=0).stop 2024-03-28T01:02:38.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.183+0000 7f38b8737640 1 -- 172.21.15.169:0/1922654164 shutdown_connections 2024-03-28T01:02:38.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.183+0000 7f38b8737640 1 --2- 172.21.15.169:0/1922654164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38b010d9c0 0x7f38b010de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.183+0000 7f38b8737640 1 --2- 172.21.15.169:0/1922654164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38b010b010 0x7f38b010d480 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.183+0000 7f38b8737640 1 --2- 172.21.15.169:0/1922654164 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38b01086a0 0x7f38b010aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.183+0000 7f38b8737640 1 -- 172.21.15.169:0/1922654164 >> 172.21.15.169:0/1922654164 conn(0x7f38b006e500 msgr2=0x7f38b0070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:38.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.190+0000 7f38b8737640 1 -- 172.21.15.169:0/1922654164 shutdown_connections 2024-03-28T01:02:38.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.192+0000 7f38b8737640 1 -- 172.21.15.169:0/1922654164 wait complete. 2024-03-28T01:02:38.196 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.193+0000 7f38b8737640 1 Processor -- start 2024-03-28T01:02:38.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.197+0000 7f38b8737640 1 -- start start 2024-03-28T01:02:38.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.197+0000 7f38b8737640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38b01086a0 0x7f38b0116e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:38.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.197+0000 7f38b8737640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38b010b010 0x7f38b0117390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:38.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.197+0000 7f38b8737640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38b010d9c0 0x7f38b01b6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:38.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.197+0000 7f38b8737640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b011c280 con 0x7f38b010d9c0 2024-03-28T01:02:38.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.197+0000 7f38b8737640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b011c100 con 0x7f38b01086a0 2024-03-28T01:02:38.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.197+0000 7f38b8737640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b011c400 con 0x7f38b010b010 2024-03-28T01:02:38.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.198+0000 7f38b5cd2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38b010b010 0x7f38b0117390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:38.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.198+0000 7f38b5cd2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38b010b010 0x7f38b0117390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:41982/0 (socket says 172.21.15.169:41982) 2024-03-28T01:02:38.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.198+0000 7f38b5cd2640 1 -- 172.21.15.169:0/1031581883 learned_addr learned my addr 172.21.15.169:0/1031581883 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:38.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.198+0000 7f38b6cd4640 1 --2- 172.21.15.169:0/1031581883 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38b010d9c0 0x7f38b01b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:38.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.198+0000 7f38b64d3640 1 --2- 172.21.15.169:0/1031581883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38b01086a0 0x7f38b0116e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:38.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.198+0000 7f38b5cd2640 1 -- 172.21.15.169:0/1031581883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38b01086a0 msgr2=0x7f38b0116e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:38.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.198+0000 7f38b5cd2640 1 --2- 172.21.15.169:0/1031581883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38b01086a0 0x7f38b0116e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.198+0000 7f38b5cd2640 1 -- 172.21.15.169:0/1031581883 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38b010d9c0 msgr2=0x7f38b01b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:38.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.198+0000 7f38b5cd2640 1 --2- 172.21.15.169:0/1031581883 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38b010d9c0 0x7f38b01b6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.198+0000 7f38b5cd2640 1 -- 172.21.15.169:0/1031581883 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b01b6ed0 con 0x7f38b010b010 2024-03-28T01:02:38.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.198+0000 7f38b64d3640 1 --2- 172.21.15.169:0/1031581883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38b01086a0 0x7f38b0116e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:02:38.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.198+0000 7f38b6cd4640 1 --2- 172.21.15.169:0/1031581883 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38b010d9c0 0x7f38b01b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:02:38.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.198+0000 7f38b5cd2640 1 --2- 172.21.15.169:0/1031581883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38b010b010 0x7f38b0117390 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f38a002c290 tx=0x7f38a00040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:38.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.205+0000 7f38a77fe640 1 -- 172.21.15.169:0/1031581883 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a0041070 con 0x7f38b010b010 2024-03-28T01:02:38.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.205+0000 7f38a77fe640 1 -- 172.21.15.169:0/1031581883 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38a00035f0 con 0x7f38b010b010 2024-03-28T01:02:38.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.205+0000 7f38a77fe640 1 -- 172.21.15.169:0/1031581883 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a00346a0 con 0x7f38b010b010 2024-03-28T01:02:38.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.206+0000 7f38b8737640 1 -- 172.21.15.169:0/1031581883 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b01b70a0 con 0x7f38b010b010 2024-03-28T01:02:38.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.206+0000 7f38b8737640 1 -- 172.21.15.169:0/1031581883 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b01b7510 con 0x7f38b010b010 2024-03-28T01:02:38.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.218+0000 7f38bb3fc740 1 Processor -- start 2024-03-28T01:02:38.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.220+0000 7f38bb3fc740 1 -- start start 2024-03-28T01:02:38.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.220+0000 7f38bb3fc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e920690 0x55fe7e920ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:38.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.220+0000 7f38bb3fc740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe7e921080 0x55fe7e925b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:38.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.220+0000 7f38a77fe640 1 -- 172.21.15.169:0/1031581883 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f38a0044070 con 0x7f38b010b010 2024-03-28T01:02:38.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.220+0000 7f38bb3fc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e9261f0 0x55fe7e928680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:38.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.220+0000 7f38bb3fc740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe7e8e3610 con 0x55fe7e921080 2024-03-28T01:02:38.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.220+0000 7f38bb3fc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe7e8863c0 con 0x55fe7e9261f0 2024-03-28T01:02:38.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.220+0000 7f38bb3fc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe7e82ed60 con 0x55fe7e920690 2024-03-28T01:02:38.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.221+0000 7f38b6cd4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e920690 0x55fe7e920ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:38.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.221+0000 7f38b6cd4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e920690 0x55fe7e920ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:41988/0 (socket says 172.21.15.169:41988) 2024-03-28T01:02:38.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.221+0000 7f38b6cd4640 1 -- 172.21.15.169:0/2968402253 learned_addr learned my addr 172.21.15.169:0/2968402253 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:38.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.221+0000 7f38b6cd4640 1 -- 172.21.15.169:0/2968402253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e9261f0 msgr2=0x55fe7e928680 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:02:38.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.221+0000 7f38b6cd4640 1 --2- 172.21.15.169:0/2968402253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e9261f0 0x55fe7e928680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.221+0000 7f38b6cd4640 1 -- 172.21.15.169:0/2968402253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe7e921080 msgr2=0x55fe7e925b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:38.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.221+0000 7f38b6cd4640 1 --2- 172.21.15.169:0/2968402253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe7e921080 0x55fe7e925b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.221+0000 7f38b6cd4640 1 -- 172.21.15.169:0/2968402253 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe7e59a3f0 con 0x55fe7e920690 2024-03-28T01:02:38.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.222+0000 7f38b6cd4640 1 --2- 172.21.15.169:0/2968402253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e920690 0x55fe7e920ab0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f38ac009d60 tx=0x7f38ac00e5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1252c02fa44430ad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:38.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.222+0000 7f38a77fe640 1 --2- 172.21.15.169:0/1031581883 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f389c0a29d0 0x7f389c0a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:38.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.222+0000 7f38a77fe640 1 -- 172.21.15.169:0/1031581883 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f38a00f0c10 con 0x7f38b010b010 2024-03-28T01:02:38.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.223+0000 7f38b64d3640 1 --2- 172.21.15.169:0/1031581883 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f389c0a29d0 0x7f389c0a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:38.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.224+0000 7f38a4ff9640 1 -- 172.21.15.169:0/2968402253 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac017070 con 0x55fe7e920690 2024-03-28T01:02:38.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.224+0000 7f38b64d3640 1 --2- 172.21.15.169:0/1031581883 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f389c0a29d0 0x7f389c0a4e90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x55fe7e921480 tx=0x7f38a80095b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:38.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.224+0000 7f38a4ff9640 1 -- 172.21.15.169:0/2968402253 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38ac013950 con 0x55fe7e920690 2024-03-28T01:02:38.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.225+0000 7f38bb3fc740 1 -- 172.21.15.169:0/2968402253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e920690 msgr2=0x55fe7e920ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:38.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.225+0000 7f38bb3fc740 1 --2- 172.21.15.169:0/2968402253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e920690 0x55fe7e920ab0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f38ac009d60 tx=0x7f38ac00e5a0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.225+0000 7f38bb3fc740 1 -- 172.21.15.169:0/2968402253 shutdown_connections 2024-03-28T01:02:38.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.225+0000 7f38bb3fc740 1 --2- 172.21.15.169:0/2968402253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e9261f0 0x55fe7e928680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.225+0000 7f38bb3fc740 1 --2- 172.21.15.169:0/2968402253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe7e921080 0x55fe7e925b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.225+0000 7f38bb3fc740 1 --2- 172.21.15.169:0/2968402253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e920690 0x55fe7e920ab0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.225+0000 7f38bb3fc740 1 -- 172.21.15.169:0/2968402253 >> 172.21.15.169:0/2968402253 conn(0x55fe7e91bc00 msgr2=0x55fe7e91c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:38.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.227+0000 7f38bb3fc740 1 -- 172.21.15.169:0/2968402253 shutdown_connections 2024-03-28T01:02:38.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.230+0000 7f38bb3fc740 1 -- 172.21.15.169:0/2968402253 wait complete. 2024-03-28T01:02:38.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.231+0000 7f38bb3fc740 1 -- start start 2024-03-28T01:02:38.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.231+0000 7f38bb3fc740 1 Processor -- start 2024-03-28T01:02:38.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.231+0000 7f38bb3fc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e920690 0x55fe7e92aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:38.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.231+0000 7f38bb3fc740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe7e921080 0x55fe7e92b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:38.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.231+0000 7f38bb3fc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e9261f0 0x55fe7e932050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:38.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.231+0000 7f38bb3fc740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe7e842240 con 0x55fe7e921080 2024-03-28T01:02:38.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.231+0000 7f38bb3fc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe7e882460 con 0x55fe7e9261f0 2024-03-28T01:02:38.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.231+0000 7f38bb3fc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe7e9291f0 con 0x55fe7e920690 2024-03-28T01:02:38.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.231+0000 7f38bb3fc740 10 client.0 _ll_register_callbacks cb 0x55fe7e841c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:38.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.232+0000 7f38b64d3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e920690 0x55fe7e92aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:38.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.232+0000 7f38b64d3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e920690 0x55fe7e92aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:41998/0 (socket says 172.21.15.169:41998) 2024-03-28T01:02:38.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.232+0000 7f38b64d3640 1 -- 172.21.15.169:0/3006751305 learned_addr learned my addr 172.21.15.169:0/3006751305 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:38.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.232+0000 7f38b6cd4640 1 --2- 172.21.15.169:0/3006751305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e9261f0 0x55fe7e932050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:38.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.232+0000 7f38b64d3640 1 -- 172.21.15.169:0/3006751305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e9261f0 msgr2=0x55fe7e932050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:38.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.232+0000 7f38b64d3640 1 --2- 172.21.15.169:0/3006751305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e9261f0 0x55fe7e932050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.232+0000 7f38b64d3640 1 -- 172.21.15.169:0/3006751305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe7e921080 msgr2=0x55fe7e92b010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:02:38.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.232+0000 7f38b64d3640 1 --2- 172.21.15.169:0/3006751305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe7e921080 0x55fe7e92b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.232+0000 7f38b64d3640 1 -- 172.21.15.169:0/3006751305 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe7e6647d0 con 0x55fe7e920690 2024-03-28T01:02:38.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.233+0000 7f38b64d3640 1 --2- 172.21.15.169:0/3006751305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e920690 0x55fe7e92aad0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f38a8004b30 tx=0x7f38a8004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8c80fe8a9fc539e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:38.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.233+0000 7f3899ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:02:38.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.233+0000 7f3899ffb640 1 -- 172.21.15.169:0/3006751305 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a8018dc0 con 0x55fe7e920690 2024-03-28T01:02:38.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.233+0000 7f3899ffb640 1 -- 172.21.15.169:0/3006751305 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f38a8012030 con 0x55fe7e920690 2024-03-28T01:02:38.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.233+0000 7f38bb3fc740 1 -- 172.21.15.169:0/3006751305 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f38ac013950 con 0x55fe7e920690 2024-03-28T01:02:38.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.234+0000 7f3899ffb640 1 -- 172.21.15.169:0/3006751305 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38a8018030 con 0x55fe7e920690 2024-03-28T01:02:38.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.236+0000 7f38bb3fc740 10 client.26449 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:38.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.236+0000 7f38bb3fc740 1 -- 172.21.15.169:0/3006751305 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55fe7e9574d0 con 0x55fe7e920690 2024-03-28T01:02:38.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.237+0000 7f3899ffb640 1 -- 172.21.15.169:0/3006751305 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38ac013950 con 0x55fe7e920690 2024-03-28T01:02:38.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.237+0000 7f3899ffb640 1 -- 172.21.15.169:0/3006751305 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f38a802d850 con 0x55fe7e920690 2024-03-28T01:02:38.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.237+0000 7f38bb3fc740 10 client.26449 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:38.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.237+0000 7f38bb3fc740 10 client.26449 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:02:38.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.237+0000 7f38bb3fc740 20 client.26449 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:38.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.237+0000 7f38bb3fc740 1 --2- 172.21.15.169:0/3006751305 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55fe7e959080 0x55fe7e95b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:38.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.237+0000 7f38bb3fc740 4 client.26449 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:02:38.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.237+0000 7f38bb3fc740 1 -- 172.21.15.169:0/3006751305 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fe7e893900 con 0x55fe7e959080 2024-03-28T01:02:38.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.239+0000 7f38b5cd2640 1 --2- 172.21.15.169:0/3006751305 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55fe7e959080 0x55fe7e95b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:38.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.248+0000 7f3899ffb640 10 client.26449 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:38.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.248+0000 7f38b5cd2640 1 --2- 172.21.15.169:0/3006751305 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55fe7e959080 0x55fe7e95b4e0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b537f0c80266c62 server_cookie=317ce7bf38b27045 in_seq=0 out_seq=0 2024-03-28T01:02:38.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.250+0000 7f3899ffb640 1 -- 172.21.15.169:0/3006751305 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fe7e893900 con 0x55fe7e959080 2024-03-28T01:02:38.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.250+0000 7f3899ffb640 10 client.26449 handle_command_reply: tid=0 2024-03-28T01:02:38.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.250+0000 7f38bb3fc740 1 client.26449 shutdown 2024-03-28T01:02:38.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.255+0000 7f38bb3fc740 1 -- 172.21.15.169:0/3006751305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e920690 msgr2=0x55fe7e92aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:38.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.255+0000 7f38bb3fc740 1 --2- 172.21.15.169:0/3006751305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e920690 0x55fe7e92aad0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f38a8004b30 tx=0x7f38a8004a80 comp rx=0 tx=0).stop 2024-03-28T01:02:38.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.258+0000 7f38bb3fc740 1 -- 172.21.15.169:0/3006751305 shutdown_connections 2024-03-28T01:02:38.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.258+0000 7f38bb3fc740 1 --2- 172.21.15.169:0/3006751305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e9261f0 0x55fe7e932050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.258+0000 7f38bb3fc740 1 --2- 172.21.15.169:0/3006751305 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55fe7e959080 0x55fe7e95b4e0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.258+0000 7f38bb3fc740 1 --2- 172.21.15.169:0/3006751305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe7e921080 0x55fe7e92b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.258+0000 7f38bb3fc740 1 --2- 172.21.15.169:0/3006751305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e920690 0x55fe7e92aad0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.258+0000 7f3899ffb640 0 client.26449 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:38.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.258+0000 7f38bb3fc740 1 -- 172.21.15.169:0/3006751305 >> 172.21.15.169:0/3006751305 conn(0x55fe7e91bc00 msgr2=0x55fe7e91ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:38.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.259+0000 7f38bb3fc740 1 -- 172.21.15.169:0/3006751305 shutdown_connections 2024-03-28T01:02:38.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.259+0000 7f38bb3fc740 1 -- 172.21.15.169:0/3006751305 wait complete. 2024-03-28T01:02:38.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.260+0000 7f38bb3fc740 20 client.26449 trim_cache size 0 max 16384 2024-03-28T01:02:38.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.281+0000 7f38bb3fc740 1 Processor -- start 2024-03-28T01:02:38.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.282+0000 7f38bb3fc740 1 -- start start 2024-03-28T01:02:38.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.282+0000 7f38bb3fc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e920690 0x55fe7e92a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:38.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.282+0000 7f38bb3fc740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe7e921080 0x55fe7e932010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:38.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.282+0000 7f38bb3fc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e9261f0 0x55fe7e92ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:38.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.282+0000 7f38b64d3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e920690 0x55fe7e92a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:38.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.282+0000 7f38b64d3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e920690 0x55fe7e92a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57746/0 (socket says 172.21.15.169:57746) 2024-03-28T01:02:38.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.282+0000 7f38b64d3640 1 -- 172.21.15.169:0/3285690889 learned_addr learned my addr 172.21.15.169:0/3285690889 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:38.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.282+0000 7f38b5cd2640 1 --2- 172.21.15.169:0/3285690889 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe7e921080 0x55fe7e932010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:38.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.282+0000 7f38bb3fc740 1 -- 172.21.15.169:0/3285690889 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3894011ff0 con 0x55fe7e921080 2024-03-28T01:02:38.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.282+0000 7f38bb3fc740 1 -- 172.21.15.169:0/3285690889 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389400faf0 con 0x55fe7e920690 2024-03-28T01:02:38.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.283+0000 7f38bb3fc740 1 -- 172.21.15.169:0/3285690889 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389400db90 con 0x55fe7e9261f0 2024-03-28T01:02:38.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.283+0000 7f38b6cd4640 1 --2- 172.21.15.169:0/3285690889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e9261f0 0x55fe7e92ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:38.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.283+0000 7f38b6cd4640 1 -- 172.21.15.169:0/3285690889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e920690 msgr2=0x55fe7e92a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:38.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.283+0000 7f38b6cd4640 1 --2- 172.21.15.169:0/3285690889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e920690 0x55fe7e92a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.283+0000 7f38b6cd4640 1 -- 172.21.15.169:0/3285690889 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe7e921080 msgr2=0x55fe7e932010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:38.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.283+0000 7f38b6cd4640 1 --2- 172.21.15.169:0/3285690889 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe7e921080 0x55fe7e932010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.283+0000 7f38b6cd4640 1 -- 172.21.15.169:0/3285690889 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe7e91b550 con 0x55fe7e9261f0 2024-03-28T01:02:38.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.283+0000 7f38b5cd2640 1 --2- 172.21.15.169:0/3285690889 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe7e921080 0x55fe7e932010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:02:38.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.283+0000 7f38b64d3640 1 --2- 172.21.15.169:0/3285690889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e920690 0x55fe7e92a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:02:38.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.283+0000 7f38b6cd4640 1 --2- 172.21.15.169:0/3285690889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e9261f0 0x55fe7e92ac20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f38ac009d30 tx=0x7f38ac009f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6f385863fbbd95b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:38.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.284+0000 7f389bfff640 1 -- 172.21.15.169:0/3285690889 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac017070 con 0x55fe7e9261f0 2024-03-28T01:02:38.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.284+0000 7f389bfff640 1 -- 172.21.15.169:0/3285690889 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38ac012520 con 0x55fe7e9261f0 2024-03-28T01:02:38.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.284+0000 7f389bfff640 1 -- 172.21.15.169:0/3285690889 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac003ab0 con 0x55fe7e9261f0 2024-03-28T01:02:38.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.285+0000 7f38bb3fc740 1 -- 172.21.15.169:0/3285690889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e9261f0 msgr2=0x55fe7e92ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:38.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.285+0000 7f38bb3fc740 1 --2- 172.21.15.169:0/3285690889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e9261f0 0x55fe7e92ac20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f38ac009d30 tx=0x7f38ac009f70 comp rx=0 tx=0).stop 2024-03-28T01:02:38.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.285+0000 7f38bb3fc740 1 -- 172.21.15.169:0/3285690889 shutdown_connections 2024-03-28T01:02:38.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.285+0000 7f38bb3fc740 1 --2- 172.21.15.169:0/3285690889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e9261f0 0x55fe7e92ac20 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.285+0000 7f38bb3fc740 1 --2- 172.21.15.169:0/3285690889 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe7e921080 0x55fe7e932010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.285+0000 7f38bb3fc740 1 --2- 172.21.15.169:0/3285690889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e920690 0x55fe7e92a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.285+0000 7f38bb3fc740 1 -- 172.21.15.169:0/3285690889 >> 172.21.15.169:0/3285690889 conn(0x55fe7e91bc00 msgr2=0x55fe7e922580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:38.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.294+0000 7f38bb3fc740 1 -- 172.21.15.169:0/3285690889 shutdown_connections 2024-03-28T01:02:38.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.294+0000 7f38bb3fc740 1 -- 172.21.15.169:0/3285690889 wait complete. 2024-03-28T01:02:38.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.294+0000 7f38bb3fc740 1 -- start start 2024-03-28T01:02:38.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.294+0000 7f38bb3fc740 1 Processor -- start 2024-03-28T01:02:38.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.295+0000 7f38bb3fc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e920690 0x55fe7e92f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:38.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.295+0000 7f38bb3fc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e921080 0x55fe7e92feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:38.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.295+0000 7f38bb3fc740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe7e9261f0 0x55fe7e930ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:38.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.295+0000 7f38bb3fc740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389400e190 con 0x55fe7e9261f0 2024-03-28T01:02:38.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.295+0000 7f38bb3fc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389400f490 con 0x55fe7e921080 2024-03-28T01:02:38.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.295+0000 7f38bb3fc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389400e7d0 con 0x55fe7e920690 2024-03-28T01:02:38.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.295+0000 7f38bb3fc740 10 client.0 _ll_register_callbacks cb 0x7f38a80188f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:38.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.295+0000 7f38b5cd2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e921080 0x55fe7e92feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:38.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.295+0000 7f38b5cd2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e921080 0x55fe7e92feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57754/0 (socket says 172.21.15.169:57754) 2024-03-28T01:02:38.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.295+0000 7f38b5cd2640 1 -- 172.21.15.169:0/1035195191 learned_addr learned my addr 172.21.15.169:0/1035195191 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:38.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.295+0000 7f38b64d3640 1 --2- 172.21.15.169:0/1035195191 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e920690 0x55fe7e92f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:38.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.295+0000 7f38b6cd4640 1 --2- 172.21.15.169:0/1035195191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe7e9261f0 0x55fe7e930ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:38.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.295+0000 7f38b5cd2640 1 -- 172.21.15.169:0/1035195191 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e920690 msgr2=0x55fe7e92f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:38.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.296+0000 7f38b5cd2640 1 --2- 172.21.15.169:0/1035195191 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e920690 0x55fe7e92f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.296+0000 7f38b5cd2640 1 -- 172.21.15.169:0/1035195191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe7e9261f0 msgr2=0x55fe7e930ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:38.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.296+0000 7f38b5cd2640 1 --2- 172.21.15.169:0/1035195191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe7e9261f0 0x55fe7e930ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.296+0000 7f38b5cd2640 1 -- 172.21.15.169:0/1035195191 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe7e9314b0 con 0x55fe7e921080 2024-03-28T01:02:38.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.296+0000 7f38b6cd4640 1 --2- 172.21.15.169:0/1035195191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe7e9261f0 0x55fe7e930ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:02:38.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.298+0000 7f38b5cd2640 1 --2- 172.21.15.169:0/1035195191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e921080 0x55fe7e92feb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f38a00bd140 tx=0x7f38a002bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8a0e17eb1e8fce1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:38.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.298+0000 7f3899ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:02:38.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.299+0000 7f3899ffb640 1 -- 172.21.15.169:0/1035195191 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a004b040 con 0x55fe7e921080 2024-03-28T01:02:38.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.299+0000 7f3899ffb640 1 -- 172.21.15.169:0/1035195191 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f38a00dd040 con 0x55fe7e921080 2024-03-28T01:02:38.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.299+0000 7f38bb3fc740 1 -- 172.21.15.169:0/1035195191 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fe7e88dc50 con 0x55fe7e921080 2024-03-28T01:02:38.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.299+0000 7f3899ffb640 1 -- 172.21.15.169:0/1035195191 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38a0004c80 con 0x55fe7e921080 2024-03-28T01:02:38.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.299+0000 7f38bb3fc740 10 client.26528 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:38.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.299+0000 7f38bb3fc740 1 -- 172.21.15.169:0/1035195191 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55fe7e9b4c80 con 0x55fe7e921080 2024-03-28T01:02:38.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.303+0000 7f3899ffb640 1 -- 172.21.15.169:0/1035195191 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55fe7e88dc50 con 0x55fe7e921080 2024-03-28T01:02:38.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.303+0000 7f3899ffb640 1 -- 172.21.15.169:0/1035195191 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f38a00b9820 con 0x55fe7e921080 2024-03-28T01:02:38.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.304+0000 7f38bb3fc740 10 client.26528 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:38.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.304+0000 7f38bb3fc740 10 client.26528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:02:38.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.304+0000 7f38bb3fc740 20 client.26528 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:38.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.304+0000 7f38bb3fc740 1 --2- 172.21.15.169:0/1035195191 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55fe7e9b55d0 0x55fe7e9b79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:38.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.305+0000 7f38bb3fc740 4 client.26528 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:02:38.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.305+0000 7f38bb3fc740 1 -- 172.21.15.169:0/1035195191 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55fe7e80b640 con 0x55fe7e9b55d0 2024-03-28T01:02:38.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.305+0000 7f38b64d3640 1 --2- 172.21.15.169:0/1035195191 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55fe7e9b55d0 0x55fe7e9b79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:38.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.310+0000 7f38b64d3640 1 --2- 172.21.15.169:0/1035195191 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55fe7e9b55d0 0x55fe7e9b79f0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45f8734c5cd28656 server_cookie=ae3a22f21cb2afdb in_seq=0 out_seq=0 2024-03-28T01:02:38.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.310+0000 7f3899ffb640 10 client.26528 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:38.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.315+0000 7f3899ffb640 1 -- 172.21.15.169:0/1035195191 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55fe7e80b640 con 0x55fe7e9b55d0 2024-03-28T01:02:38.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.315+0000 7f3899ffb640 10 client.26528 handle_command_reply: tid=0 2024-03-28T01:02:38.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.315+0000 7f38bb3fc740 1 client.26528 shutdown 2024-03-28T01:02:38.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.315+0000 7f38bb3fc740 1 -- 172.21.15.169:0/1035195191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e921080 msgr2=0x55fe7e92feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:38.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.315+0000 7f38bb3fc740 1 --2- 172.21.15.169:0/1035195191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e921080 0x55fe7e92feb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f38a00bd140 tx=0x7f38a002bc80 comp rx=0 tx=0).stop 2024-03-28T01:02:38.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.315+0000 7f38bb3fc740 1 -- 172.21.15.169:0/1035195191 shutdown_connections 2024-03-28T01:02:38.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.315+0000 7f38bb3fc740 1 --2- 172.21.15.169:0/1035195191 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55fe7e9b55d0 0x55fe7e9b79f0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.315+0000 7f38bb3fc740 1 --2- 172.21.15.169:0/1035195191 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55fe7e9261f0 0x55fe7e930ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.315+0000 7f38bb3fc740 1 --2- 172.21.15.169:0/1035195191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7e921080 0x55fe7e92feb0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.315+0000 7f38bb3fc740 1 --2- 172.21.15.169:0/1035195191 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe7e920690 0x55fe7e92f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.315+0000 7f38bb3fc740 1 -- 172.21.15.169:0/1035195191 >> 172.21.15.169:0/1035195191 conn(0x55fe7e91bc00 msgr2=0x55fe7e92a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:38.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.315+0000 7f3899ffb640 0 client.26528 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:38.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.325+0000 7f38bb3fc740 1 -- 172.21.15.169:0/1035195191 shutdown_connections 2024-03-28T01:02:38.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.330+0000 7f38bb3fc740 1 -- 172.21.15.169:0/1035195191 wait complete. 2024-03-28T01:02:38.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.331+0000 7f38bb3fc740 20 client.26528 trim_cache size 0 max 16384 2024-03-28T01:02:38.334 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:02:38.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.341+0000 7f38b8737640 1 -- 172.21.15.169:0/1031581883 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f389c0a29d0 msgr2=0x7f389c0a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:38.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.341+0000 7f38b8737640 1 --2- 172.21.15.169:0/1031581883 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f389c0a29d0 0x7f389c0a4e90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x55fe7e921480 tx=0x7f38a80095b0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.341+0000 7f38b8737640 1 -- 172.21.15.169:0/1031581883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38b010b010 msgr2=0x7f38b0117390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:38.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.341+0000 7f38b8737640 1 --2- 172.21.15.169:0/1031581883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38b010b010 0x7f38b0117390 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f38a002c290 tx=0x7f38a00040c0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.344+0000 7f38b8737640 1 -- 172.21.15.169:0/1031581883 shutdown_connections 2024-03-28T01:02:38.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.344+0000 7f38b8737640 1 --2- 172.21.15.169:0/1031581883 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f389c0a29d0 0x7f389c0a4e90 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.344+0000 7f38b8737640 1 --2- 172.21.15.169:0/1031581883 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38b010d9c0 0x7f38b01b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.344+0000 7f38b8737640 1 --2- 172.21.15.169:0/1031581883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38b010b010 0x7f38b0117390 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.344+0000 7f38b8737640 1 --2- 172.21.15.169:0/1031581883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38b01086a0 0x7f38b0116e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:38.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.344+0000 7f38b8737640 1 -- 172.21.15.169:0/1031581883 >> 172.21.15.169:0/1031581883 conn(0x7f38b006e500 msgr2=0x7f38b010b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:38.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.345+0000 7f38b8737640 1 -- 172.21.15.169:0/1031581883 shutdown_connections 2024-03-28T01:02:38.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:38.346+0000 7f38b8737640 1 -- 172.21.15.169:0/1031581883 wait complete. 2024-03-28T01:02:39.411 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T01:02:39.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T01:02:39.412 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T01:02:39.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.756+0000 7f021b3d4640 1 -- 172.21.15.169:0/1763322028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0214108d70 msgr2=0x7f02141091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:39.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.756+0000 7f021b3d4640 1 --2- 172.21.15.169:0/1763322028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0214108d70 0x7f02141091f0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f020c009ca0 tx=0x7f020c02c140 comp rx=0 tx=0).stop 2024-03-28T01:02:39.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.761+0000 7f021b3d4640 1 -- 172.21.15.169:0/1763322028 shutdown_connections 2024-03-28T01:02:39.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.762+0000 7f021b3d4640 1 --2- 172.21.15.169:0/1763322028 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0214109730 0x7f021410dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.762+0000 7f021b3d4640 1 --2- 172.21.15.169:0/1763322028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0214108d70 0x7f02141091f0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.762+0000 7f021b3d4640 1 --2- 172.21.15.169:0/1763322028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0214072ab0 0x7f0214072ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.762+0000 7f021b3d4640 1 -- 172.21.15.169:0/1763322028 >> 172.21.15.169:0/1763322028 conn(0x7f021406e580 msgr2=0x7f02140709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:39.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.767+0000 7f021b3d4640 1 -- 172.21.15.169:0/1763322028 shutdown_connections 2024-03-28T01:02:39.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.783+0000 7f021b3d4640 1 -- 172.21.15.169:0/1763322028 wait complete. 2024-03-28T01:02:39.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.783+0000 7f021b3d4640 1 Processor -- start 2024-03-28T01:02:39.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.784+0000 7f021b3d4640 1 -- start start 2024-03-28T01:02:39.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.784+0000 7f021b3d4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0214072ab0 0x7f02141a66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:39.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.784+0000 7f021b3d4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0214108d70 0x7f02141a6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:39.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.784+0000 7f021b3d4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0214109730 0x7f02141adc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:39.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.784+0000 7f021b3d4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021411c040 con 0x7f0214108d70 2024-03-28T01:02:39.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.784+0000 7f021b3d4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021411bec0 con 0x7f0214109730 2024-03-28T01:02:39.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.784+0000 7f021b3d4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021411c1c0 con 0x7f0214072ab0 2024-03-28T01:02:39.788 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.785+0000 7f021abd3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0214109730 0x7f02141adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:39.788 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.785+0000 7f021abd3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0214109730 0x7f02141adc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57766/0 (socket says 172.21.15.169:57766) 2024-03-28T01:02:39.788 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.785+0000 7f021abd3640 1 -- 172.21.15.169:0/3793443048 learned_addr learned my addr 172.21.15.169:0/3793443048 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:39.788 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.785+0000 7f021a3d2640 1 --2- 172.21.15.169:0/3793443048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0214072ab0 0x7f02141a66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:39.788 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.785+0000 7f0219bd1640 1 --2- 172.21.15.169:0/3793443048 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0214108d70 0x7f02141a6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:39.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.791+0000 7f021abd3640 1 -- 172.21.15.169:0/3793443048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0214072ab0 msgr2=0x7f02141a66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:39.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.791+0000 7f021abd3640 1 --2- 172.21.15.169:0/3793443048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0214072ab0 0x7f02141a66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.791+0000 7f021abd3640 1 -- 172.21.15.169:0/3793443048 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0214108d70 msgr2=0x7f02141a6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:39.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.791+0000 7f021abd3640 1 --2- 172.21.15.169:0/3793443048 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0214108d70 0x7f02141a6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.791+0000 7f021abd3640 1 -- 172.21.15.169:0/3793443048 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02141ae2e0 con 0x7f0214109730 2024-03-28T01:02:39.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.792+0000 7f021abd3640 1 --2- 172.21.15.169:0/3793443048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0214109730 0x7f02141adc70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f020400d750 tx=0x7f020400dc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:39.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.795+0000 7f020b7fe640 1 -- 172.21.15.169:0/3793443048 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0204017070 con 0x7f0214109730 2024-03-28T01:02:39.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.795+0000 7f021b3d4640 1 -- 172.21.15.169:0/3793443048 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02141ae5d0 con 0x7f0214109730 2024-03-28T01:02:39.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.795+0000 7f021b3d4640 1 -- 172.21.15.169:0/3793443048 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02141aeaf0 con 0x7f0214109730 2024-03-28T01:02:39.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.795+0000 7f020b7fe640 1 -- 172.21.15.169:0/3793443048 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0204005500 con 0x7f0214109730 2024-03-28T01:02:39.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.795+0000 7f020b7fe640 1 -- 172.21.15.169:0/3793443048 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0204004d90 con 0x7f0214109730 2024-03-28T01:02:39.807 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:39 smithi169 ceph-mon[32274]: pgmap v535: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 669 MiB/s wr, 1.58k op/s 2024-03-28T01:02:39.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.803+0000 7f020b7fe640 1 -- 172.21.15.169:0/3793443048 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0204010460 con 0x7f0214109730 2024-03-28T01:02:39.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.804+0000 7f020b7fe640 1 --2- 172.21.15.169:0/3793443048 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f02000a2a00 0x7f02000a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:39.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.804+0000 7f021a3d2640 1 --2- 172.21.15.169:0/3793443048 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f02000a2a00 0x7f02000a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:39.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.805+0000 7f021a3d2640 1 --2- 172.21.15.169:0/3793443048 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f02000a2a00 0x7f02000a4ec0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f0214073800 tx=0x7f021000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:39.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.805+0000 7f020b7fe640 1 -- 172.21.15.169:0/3793443048 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f02040cf6c0 con 0x7f0214109730 2024-03-28T01:02:39.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.807+0000 7f021f226740 1 Processor -- start 2024-03-28T01:02:39.811 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.808+0000 7f021f226740 1 -- start start 2024-03-28T01:02:39.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.811+0000 7f021f226740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414db30 0x56177414df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:39.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.811+0000 7f021f226740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56177414e520 0x561774153020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:39.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.811+0000 7f021f226740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561774153690 0x561774155b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:39.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.811+0000 7f021f226740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56177408dc90 con 0x561774153690 2024-03-28T01:02:39.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.811+0000 7f021f226740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617740b3070 con 0x56177414db30 2024-03-28T01:02:39.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.811+0000 7f021f226740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56177405bc80 con 0x56177414e520 2024-03-28T01:02:39.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.812+0000 7f021a3d2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414db30 0x56177414df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:39.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.812+0000 7f021a3d2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414db30 0x56177414df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57778/0 (socket says 172.21.15.169:57778) 2024-03-28T01:02:39.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.812+0000 7f021a3d2640 1 -- 172.21.15.169:0/2620197195 learned_addr learned my addr 172.21.15.169:0/2620197195 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:39.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.812+0000 7f021abd3640 1 --2- 172.21.15.169:0/2620197195 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561774153690 0x561774155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:39.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.812+0000 7f0219bd1640 1 --2- 172.21.15.169:0/2620197195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56177414e520 0x561774153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:39.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.817+0000 7f0219bd1640 1 -- 172.21.15.169:0/2620197195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414db30 msgr2=0x56177414df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:39.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.817+0000 7f0219bd1640 1 --2- 172.21.15.169:0/2620197195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414db30 0x56177414df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.817+0000 7f0219bd1640 1 -- 172.21.15.169:0/2620197195 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561774153690 msgr2=0x561774155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:39.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.817+0000 7f0219bd1640 1 --2- 172.21.15.169:0/2620197195 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561774153690 0x561774155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.817+0000 7f0219bd1640 1 -- 172.21.15.169:0/2620197195 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561774053d10 con 0x56177414e520 2024-03-28T01:02:39.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.818+0000 7f0219bd1640 1 --2- 172.21.15.169:0/2620197195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56177414e520 0x561774153020 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f020c009bf0 tx=0x7f020c0323f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b01a1fe0c5e01bde server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:39.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.819+0000 7f0208bfe640 1 -- 172.21.15.169:0/2620197195 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020c039040 con 0x56177414e520 2024-03-28T01:02:39.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.819+0000 7f0208bfe640 1 -- 172.21.15.169:0/2620197195 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f020c032e70 con 0x56177414e520 2024-03-28T01:02:39.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.820+0000 7f021f226740 1 -- 172.21.15.169:0/2620197195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56177414e520 msgr2=0x561774153020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:39.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.820+0000 7f021f226740 1 --2- 172.21.15.169:0/2620197195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56177414e520 0x561774153020 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f020c009bf0 tx=0x7f020c0323f0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.820+0000 7f021f226740 1 -- 172.21.15.169:0/2620197195 shutdown_connections 2024-03-28T01:02:39.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.820+0000 7f021f226740 1 --2- 172.21.15.169:0/2620197195 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561774153690 0x561774155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.820+0000 7f021f226740 1 --2- 172.21.15.169:0/2620197195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56177414e520 0x561774153020 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.820+0000 7f021f226740 1 --2- 172.21.15.169:0/2620197195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414db30 0x56177414df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.820+0000 7f021f226740 1 -- 172.21.15.169:0/2620197195 >> 172.21.15.169:0/2620197195 conn(0x5617741161a0 msgr2=0x5617741494a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:39.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.822+0000 7f021f226740 1 -- 172.21.15.169:0/2620197195 shutdown_connections 2024-03-28T01:02:39.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.823+0000 7f021f226740 1 -- 172.21.15.169:0/2620197195 wait complete. 2024-03-28T01:02:39.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.823+0000 7f021f226740 1 -- start start 2024-03-28T01:02:39.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.824+0000 7f021f226740 1 Processor -- start 2024-03-28T01:02:39.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.824+0000 7f021f226740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414db30 0x561774157e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:39.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.824+0000 7f021f226740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56177414e520 0x5617741583a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:39.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.824+0000 7f021f226740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561774153690 0x56177415f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:39.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.824+0000 7f021f226740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56177406f0d0 con 0x56177414e520 2024-03-28T01:02:39.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.824+0000 7f021f226740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617740af110 con 0x56177414db30 2024-03-28T01:02:39.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.824+0000 7f021f226740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561774156680 con 0x561774153690 2024-03-28T01:02:39.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.824+0000 7f021f226740 10 client.0 _ll_register_callbacks cb 0x5617740aae60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:39.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.829+0000 7f021a3d2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414db30 0x561774157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:39.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.829+0000 7f0219bd1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56177414e520 0x5617741583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:39.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.829+0000 7f0219bd1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56177414e520 0x5617741583a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39058/0 (socket says 172.21.15.169:39058) 2024-03-28T01:02:39.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.829+0000 7f0219bd1640 1 -- 172.21.15.169:0/2499719922 learned_addr learned my addr 172.21.15.169:0/2499719922 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:39.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.829+0000 7f021abd3640 1 --2- 172.21.15.169:0/2499719922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561774153690 0x56177415f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:39.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.830+0000 7f021a3d2640 1 -- 172.21.15.169:0/2499719922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561774153690 msgr2=0x56177415f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:39.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.830+0000 7f021a3d2640 1 --2- 172.21.15.169:0/2499719922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561774153690 0x56177415f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.830+0000 7f021a3d2640 1 -- 172.21.15.169:0/2499719922 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56177414e520 msgr2=0x5617741583a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:39.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.830+0000 7f021a3d2640 1 --2- 172.21.15.169:0/2499719922 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56177414e520 0x5617741583a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.830+0000 7f021a3d2640 1 -- 172.21.15.169:0/2499719922 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561773e8f7e0 con 0x56177414db30 2024-03-28T01:02:39.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.831+0000 7f021a3d2640 1 --2- 172.21.15.169:0/2499719922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414db30 0x561774157e60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x56177414ed80 tx=0x7f0210004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56640d1aaba1fa05 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:39.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.831+0000 7f01fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:02:39.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.831+0000 7f01fdffb640 1 -- 172.21.15.169:0/2499719922 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021001fa80 con 0x56177414db30 2024-03-28T01:02:39.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.832+0000 7f01fdffb640 1 -- 172.21.15.169:0/2499719922 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f021001fc20 con 0x56177414db30 2024-03-28T01:02:39.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.832+0000 7f021f226740 1 -- 172.21.15.169:0/2499719922 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5617740c05b0 con 0x56177414db30 2024-03-28T01:02:39.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.833+0000 7f01fdffb640 1 -- 172.21.15.169:0/2499719922 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f021001fe20 con 0x56177414db30 2024-03-28T01:02:39.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.836+0000 7f021f226740 10 client.26540 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:39.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.836+0000 7f021f226740 1 -- 172.21.15.169:0/2499719922 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561773eeff70 con 0x56177414db30 2024-03-28T01:02:39.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.840+0000 7f01fdffb640 1 -- 172.21.15.169:0/2499719922 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5617740c05b0 con 0x56177414db30 2024-03-28T01:02:39.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.840+0000 7f01fdffb640 1 -- 172.21.15.169:0/2499719922 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f0210026070 con 0x56177414db30 2024-03-28T01:02:39.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.841+0000 7f021f226740 10 client.26540 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:39.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.841+0000 7f021f226740 10 client.26540 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:02:39.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.841+0000 7f021f226740 20 client.26540 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:39.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.841+0000 7f021f226740 1 --2- 172.21.15.169:0/2499719922 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561774163dc0 0x561774166220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:39.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.841+0000 7f021f226740 4 client.26540 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:02:39.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.841+0000 7f021f226740 1 -- 172.21.15.169:0/2499719922 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5617740ba900 con 0x561774163dc0 2024-03-28T01:02:39.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.846+0000 7f0219bd1640 1 --2- 172.21.15.169:0/2499719922 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561774163dc0 0x561774166220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:39.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:39 smithi174 ceph-mon[36060]: pgmap v535: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 669 MiB/s wr, 1.58k op/s 2024-03-28T01:02:39.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.853+0000 7f0219bd1640 1 --2- 172.21.15.169:0/2499719922 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561774163dc0 0x561774166220 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5563030a59306a5 server_cookie=8044871ea4f5f96e in_seq=0 out_seq=0 2024-03-28T01:02:39.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.853+0000 7f01fdffb640 10 client.26540 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:39.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.858+0000 7f01fdffb640 1 -- 172.21.15.169:0/2499719922 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5617740ba900 con 0x561774163dc0 2024-03-28T01:02:39.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.858+0000 7f01fdffb640 10 client.26540 handle_command_reply: tid=0 2024-03-28T01:02:39.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.859+0000 7f021f226740 1 client.26540 shutdown 2024-03-28T01:02:39.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.861+0000 7f021f226740 1 -- 172.21.15.169:0/2499719922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414db30 msgr2=0x561774157e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:39.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.861+0000 7f021f226740 1 --2- 172.21.15.169:0/2499719922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414db30 0x561774157e60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x56177414ed80 tx=0x7f0210004730 comp rx=0 tx=0).stop 2024-03-28T01:02:39.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.864+0000 7f021f226740 1 -- 172.21.15.169:0/2499719922 shutdown_connections 2024-03-28T01:02:39.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.864+0000 7f021f226740 1 --2- 172.21.15.169:0/2499719922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561774153690 0x56177415f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.864+0000 7f021f226740 1 --2- 172.21.15.169:0/2499719922 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561774163dc0 0x561774166220 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.864+0000 7f021f226740 1 --2- 172.21.15.169:0/2499719922 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56177414e520 0x5617741583a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.864+0000 7f021f226740 1 --2- 172.21.15.169:0/2499719922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414db30 0x561774157e60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.864+0000 7f021f226740 1 -- 172.21.15.169:0/2499719922 >> 172.21.15.169:0/2499719922 conn(0x5617741161a0 msgr2=0x5617741487e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:39.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.865+0000 7f01fdffb640 0 client.26540 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:39.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.870+0000 7f021f226740 1 -- 172.21.15.169:0/2499719922 shutdown_connections 2024-03-28T01:02:39.877 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.874+0000 7f021f226740 1 -- 172.21.15.169:0/2499719922 wait complete. 2024-03-28T01:02:39.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.875+0000 7f021f226740 20 client.26540 trim_cache size 0 max 16384 2024-03-28T01:02:39.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:39 smithi177 ceph-mon[36206]: pgmap v535: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 669 MiB/s wr, 1.58k op/s 2024-03-28T01:02:39.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.927+0000 7f021f226740 1 Processor -- start 2024-03-28T01:02:39.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.927+0000 7f021f226740 1 -- start start 2024-03-28T01:02:39.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.927+0000 7f021f226740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414db30 0x561774157360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:39.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.927+0000 7f021f226740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56177414e520 0x56177415f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:39.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.927+0000 7f021f226740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561774153690 0x561774148660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:39.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.928+0000 7f021f226740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f8012120 con 0x56177414e520 2024-03-28T01:02:39.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.928+0000 7f021f226740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f800fc20 con 0x56177414db30 2024-03-28T01:02:39.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.928+0000 7f021a3d2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414db30 0x561774157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:39.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.928+0000 7f021a3d2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414db30 0x561774157360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57802/0 (socket says 172.21.15.169:57802) 2024-03-28T01:02:39.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.928+0000 7f021a3d2640 1 -- 172.21.15.169:0/1655529671 learned_addr learned my addr 172.21.15.169:0/1655529671 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:39.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.928+0000 7f021f226740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f800dcc0 con 0x561774153690 2024-03-28T01:02:39.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.930+0000 7f021abd3640 1 --2- 172.21.15.169:0/1655529671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561774153690 0x561774148660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:39.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.930+0000 7f0219bd1640 1 --2- 172.21.15.169:0/1655529671 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56177414e520 0x56177415f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:39.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.931+0000 7f021a3d2640 1 -- 172.21.15.169:0/1655529671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561774153690 msgr2=0x561774148660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:39.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.931+0000 7f021a3d2640 1 --2- 172.21.15.169:0/1655529671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561774153690 0x561774148660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.931+0000 7f021a3d2640 1 -- 172.21.15.169:0/1655529671 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56177414e520 msgr2=0x56177415f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:39.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.931+0000 7f021a3d2640 1 --2- 172.21.15.169:0/1655529671 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56177414e520 0x56177415f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.931+0000 7f021a3d2640 1 -- 172.21.15.169:0/1655529671 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561773d87170 con 0x56177414db30 2024-03-28T01:02:39.937 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.933+0000 7f021a3d2640 1 --2- 172.21.15.169:0/1655529671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414db30 0x561774157360 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0210010910 tx=0x7f0210010a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3eed73deb2ef5f4f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:39.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.940+0000 7f01fffff640 1 -- 172.21.15.169:0/1655529671 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02100041a0 con 0x56177414db30 2024-03-28T01:02:39.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.940+0000 7f01fffff640 1 -- 172.21.15.169:0/1655529671 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f021003c070 con 0x56177414db30 2024-03-28T01:02:39.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.940+0000 7f01fffff640 1 -- 172.21.15.169:0/1655529671 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02100279b0 con 0x56177414db30 2024-03-28T01:02:39.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.942+0000 7f021f226740 1 -- 172.21.15.169:0/1655529671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414db30 msgr2=0x561774157360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:39.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.942+0000 7f021f226740 1 --2- 172.21.15.169:0/1655529671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414db30 0x561774157360 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0210010910 tx=0x7f0210010a50 comp rx=0 tx=0).stop 2024-03-28T01:02:39.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.942+0000 7f021f226740 1 -- 172.21.15.169:0/1655529671 shutdown_connections 2024-03-28T01:02:39.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.942+0000 7f021f226740 1 --2- 172.21.15.169:0/1655529671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561774153690 0x561774148660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.942+0000 7f021f226740 1 --2- 172.21.15.169:0/1655529671 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56177414e520 0x56177415f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.942+0000 7f021f226740 1 --2- 172.21.15.169:0/1655529671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414db30 0x561774157360 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.942+0000 7f021f226740 1 -- 172.21.15.169:0/1655529671 >> 172.21.15.169:0/1655529671 conn(0x5617741161a0 msgr2=0x561774147f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:39.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.942+0000 7f021f226740 1 -- 172.21.15.169:0/1655529671 shutdown_connections 2024-03-28T01:02:39.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.942+0000 7f021f226740 1 -- 172.21.15.169:0/1655529671 wait complete. 2024-03-28T01:02:39.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.943+0000 7f021f226740 1 -- start start 2024-03-28T01:02:39.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.943+0000 7f021f226740 1 Processor -- start 2024-03-28T01:02:39.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.944+0000 7f021f226740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56177414db30 0x56177415d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:39.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.944+0000 7f021f226740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414e520 0x56177415dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:39.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.944+0000 7f021a3d2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56177414db30 0x56177415d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:39.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.944+0000 7f021a3d2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56177414db30 0x56177415d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:42062/0 (socket says 172.21.15.169:42062) 2024-03-28T01:02:39.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.944+0000 7f021a3d2640 1 -- 172.21.15.169:0/3514406757 learned_addr learned my addr 172.21.15.169:0/3514406757 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:39.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.944+0000 7f0219bd1640 1 --2- 172.21.15.169:0/3514406757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414e520 0x56177415dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:39.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.944+0000 7f021f226740 1 --2- 172.21.15.169:0/3514406757 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561774153690 0x56177415e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:39.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.945+0000 7f021f226740 1 -- 172.21.15.169:0/3514406757 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f800e2c0 con 0x561774153690 2024-03-28T01:02:39.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.945+0000 7f021f226740 1 -- 172.21.15.169:0/3514406757 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f800f5c0 con 0x56177414e520 2024-03-28T01:02:39.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.945+0000 7f021f226740 1 -- 172.21.15.169:0/3514406757 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f800e900 con 0x56177414db30 2024-03-28T01:02:39.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.945+0000 7f021f226740 10 client.0 _ll_register_callbacks cb 0x7f0210011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:39.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.945+0000 7f0219bd1640 1 -- 172.21.15.169:0/3514406757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56177414db30 msgr2=0x56177415d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:39.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.945+0000 7f0219bd1640 1 --2- 172.21.15.169:0/3514406757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56177414db30 0x56177415d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.945+0000 7f0219bd1640 1 -- 172.21.15.169:0/3514406757 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561774153690 msgr2=0x56177415e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:39.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.945+0000 7f0219bd1640 1 --2- 172.21.15.169:0/3514406757 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561774153690 0x56177415e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:39.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.946+0000 7f0219bd1640 1 -- 172.21.15.169:0/3514406757 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56177415efd0 con 0x56177414e520 2024-03-28T01:02:39.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.946+0000 7f021f226740 1 -- 172.21.15.169:0/3514406757 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f021003c070 con 0x56177414e520 2024-03-28T01:02:39.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.946+0000 7f0219bd1640 1 --2- 172.21.15.169:0/3514406757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414e520 0x56177415dae0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f020c0324e0 tx=0x7f020c02c140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eb083f9929029ef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:39.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.946+0000 7f01fdffb640 10 client.26546 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:02:39.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.946+0000 7f01fdffb640 1 -- 172.21.15.169:0/3514406757 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020c039040 con 0x56177414e520 2024-03-28T01:02:39.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.947+0000 7f01fdffb640 1 -- 172.21.15.169:0/3514406757 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f020c052280 con 0x56177414e520 2024-03-28T01:02:39.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.947+0000 7f01fdffb640 1 -- 172.21.15.169:0/3514406757 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f020c0044f0 con 0x56177414e520 2024-03-28T01:02:39.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.949+0000 7f021f226740 10 client.26546 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:39.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.949+0000 7f021f226740 1 -- 172.21.15.169:0/3514406757 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x5617741bffe0 con 0x56177414e520 2024-03-28T01:02:39.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.949+0000 7f01fdffb640 1 -- 172.21.15.169:0/3514406757 <== mon.2 v2:172.21.15.174:3300/0 4 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f020c004690 con 0x56177414e520 2024-03-28T01:02:39.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.952+0000 7f021f226740 10 client.26546 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:39.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.952+0000 7f021f226740 10 client.26546 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:02:39.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.952+0000 7f021f226740 20 client.26546 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:39.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.952+0000 7f021f226740 1 --2- 172.21.15.169:0/3514406757 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5617741c0400 0x5617741c27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:39.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.952+0000 7f021f226740 4 client.26546 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T01:02:39.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.952+0000 7f021f226740 1 -- 172.21.15.169:0/3514406757 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x561774038b80 con 0x5617741c0400 2024-03-28T01:02:39.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.953+0000 7f021a3d2640 1 --2- 172.21.15.169:0/3514406757 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5617741c0400 0x5617741c27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:39.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.955+0000 7f021a3d2640 1 --2- 172.21.15.169:0/3514406757 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5617741c0400 0x5617741c27f0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=233697081d1167bb server_cookie=ff1ad8537a791a72 in_seq=0 out_seq=0 2024-03-28T01:02:39.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:39.955+0000 7f01fdffb640 10 client.26546 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:40.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.776+0000 7f01fdffb640 1 -- 172.21.15.169:0/3514406757 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x561774038b80 con 0x5617741c0400 2024-03-28T01:02:40.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.776+0000 7f01fdffb640 10 client.26546 handle_command_reply: tid=0 2024-03-28T01:02:40.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.776+0000 7f021f226740 1 client.26546 shutdown 2024-03-28T01:02:40.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.777+0000 7f021f226740 1 -- 172.21.15.169:0/3514406757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414e520 msgr2=0x56177415dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:40.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.777+0000 7f021f226740 1 --2- 172.21.15.169:0/3514406757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414e520 0x56177415dae0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f020c0324e0 tx=0x7f020c02c140 comp rx=0 tx=0).stop 2024-03-28T01:02:40.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.777+0000 7f021f226740 1 -- 172.21.15.169:0/3514406757 shutdown_connections 2024-03-28T01:02:40.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.777+0000 7f021f226740 1 --2- 172.21.15.169:0/3514406757 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5617741c0400 0x5617741c27f0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:40.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.777+0000 7f021f226740 1 --2- 172.21.15.169:0/3514406757 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561774153690 0x56177415e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:40.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.777+0000 7f021f226740 1 --2- 172.21.15.169:0/3514406757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56177414e520 0x56177415dae0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:40.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.777+0000 7f021f226740 1 --2- 172.21.15.169:0/3514406757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56177414db30 0x56177415d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:40.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.777+0000 7f021f226740 1 -- 172.21.15.169:0/3514406757 >> 172.21.15.169:0/3514406757 conn(0x5617741161a0 msgr2=0x561774147f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:40.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.777+0000 7f01fdffb640 0 client.26546 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:40.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.777+0000 7f021f226740 1 -- 172.21.15.169:0/3514406757 shutdown_connections 2024-03-28T01:02:40.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.777+0000 7f021f226740 1 -- 172.21.15.169:0/3514406757 wait complete. 2024-03-28T01:02:40.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.777+0000 7f021f226740 20 client.26546 trim_cache size 0 max 16384 2024-03-28T01:02:40.783 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:02:40.783 INFO:teuthology.orchestra.run.smithi169.stdout: "return_code": 0, 2024-03-28T01:02:40.783 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_tag": "e41d5635-a3bb-4cd0-8193-81bd9af97a5e", 2024-03-28T01:02:40.783 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": "asynchronous" 2024-03-28T01:02:40.783 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:02:40.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.781+0000 7f021b3d4640 1 -- 172.21.15.169:0/3793443048 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f02000a2a00 msgr2=0x7f02000a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:40.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.781+0000 7f021b3d4640 1 --2- 172.21.15.169:0/3793443048 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f02000a2a00 0x7f02000a4ec0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f0214073800 tx=0x7f021000b390 comp rx=0 tx=0).stop 2024-03-28T01:02:40.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.781+0000 7f021b3d4640 1 -- 172.21.15.169:0/3793443048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0214109730 msgr2=0x7f02141adc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:40.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.781+0000 7f021b3d4640 1 --2- 172.21.15.169:0/3793443048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0214109730 0x7f02141adc70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f020400d750 tx=0x7f020400dc10 comp rx=0 tx=0).stop 2024-03-28T01:02:40.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.784+0000 7f021b3d4640 1 -- 172.21.15.169:0/3793443048 shutdown_connections 2024-03-28T01:02:40.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.784+0000 7f021b3d4640 1 --2- 172.21.15.169:0/3793443048 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f02000a2a00 0x7f02000a4ec0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:40.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.784+0000 7f021b3d4640 1 --2- 172.21.15.169:0/3793443048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0214109730 0x7f02141adc70 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:40.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.784+0000 7f021b3d4640 1 --2- 172.21.15.169:0/3793443048 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0214108d70 0x7f02141a6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:40.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.784+0000 7f021b3d4640 1 --2- 172.21.15.169:0/3793443048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0214072ab0 0x7f02141a66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:40.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.784+0000 7f021b3d4640 1 -- 172.21.15.169:0/3793443048 >> 172.21.15.169:0/3793443048 conn(0x7f021406e580 msgr2=0x7f02140701d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:40.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.784+0000 7f021b3d4640 1 -- 172.21.15.169:0/3793443048 shutdown_connections 2024-03-28T01:02:40.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:40.784+0000 7f021b3d4640 1 -- 172.21.15.169:0/3793443048 wait complete. 2024-03-28T01:02:40.823 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:02:40.823 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:02:41.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.131+0000 7fb7e50dc640 1 -- 172.21.15.169:0/283397769 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7e01086a0 msgr2=0x7fb7e010aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:41.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.131+0000 7fb7e50dc640 1 --2- 172.21.15.169:0/283397769 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7e01086a0 0x7fb7e010aad0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0009900 tx=0x7fb7d002bc80 comp rx=0 tx=0).stop 2024-03-28T01:02:41.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.131+0000 7fb7e50dc640 1 -- 172.21.15.169:0/283397769 shutdown_connections 2024-03-28T01:02:41.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.131+0000 7fb7e50dc640 1 --2- 172.21.15.169:0/283397769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7e010d9c0 0x7fb7e010de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.131+0000 7fb7e50dc640 1 --2- 172.21.15.169:0/283397769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7e010b010 0x7fb7e010d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.131+0000 7fb7e50dc640 1 --2- 172.21.15.169:0/283397769 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7e01086a0 0x7fb7e010aad0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.131+0000 7fb7e50dc640 1 -- 172.21.15.169:0/283397769 >> 172.21.15.169:0/283397769 conn(0x7fb7e006e500 msgr2=0x7fb7e0070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:41.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.140+0000 7fb7e50dc640 1 -- 172.21.15.169:0/283397769 shutdown_connections 2024-03-28T01:02:41.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.140+0000 7fb7e50dc640 1 -- 172.21.15.169:0/283397769 wait complete. 2024-03-28T01:02:41.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.143+0000 7fb7e50dc640 1 Processor -- start 2024-03-28T01:02:41.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.143+0000 7fb7e50dc640 1 -- start start 2024-03-28T01:02:41.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.143+0000 7fb7e50dc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7e01086a0 0x7fb7e0116de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:41.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.143+0000 7fb7e50dc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7e010b010 0x7fb7e0117320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:41.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.143+0000 7fb7e50dc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7e010d9c0 0x7fb7e01b6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:41.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.143+0000 7fb7e50dc640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e011c260 con 0x7fb7e010b010 2024-03-28T01:02:41.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.143+0000 7fb7e50dc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e011c0e0 con 0x7fb7e010d9c0 2024-03-28T01:02:41.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.143+0000 7fb7e50dc640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e011c3e0 con 0x7fb7e01086a0 2024-03-28T01:02:41.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.145+0000 7fb7df59e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7e010d9c0 0x7fb7e01b6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:41.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.145+0000 7fb7de59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7e010b010 0x7fb7e0117320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:41.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.145+0000 7fb7de59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7e010b010 0x7fb7e0117320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39084/0 (socket says 172.21.15.169:39084) 2024-03-28T01:02:41.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.145+0000 7fb7de59c640 1 -- 172.21.15.169:0/3863308647 learned_addr learned my addr 172.21.15.169:0/3863308647 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:41.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.148+0000 7fb7ded9d640 1 --2- 172.21.15.169:0/3863308647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7e01086a0 0x7fb7e0116de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:41.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.151+0000 7fb7df59e640 1 -- 172.21.15.169:0/3863308647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7e01086a0 msgr2=0x7fb7e0116de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:41.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.151+0000 7fb7df59e640 1 --2- 172.21.15.169:0/3863308647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7e01086a0 0x7fb7e0116de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.151+0000 7fb7df59e640 1 -- 172.21.15.169:0/3863308647 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7e010b010 msgr2=0x7fb7e0117320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:41.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.151+0000 7fb7df59e640 1 --2- 172.21.15.169:0/3863308647 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7e010b010 0x7fb7e0117320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.151+0000 7fb7df59e640 1 -- 172.21.15.169:0/3863308647 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e01b6eb0 con 0x7fb7e010d9c0 2024-03-28T01:02:41.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.159+0000 7fb7df59e640 1 --2- 172.21.15.169:0/3863308647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7e010d9c0 0x7fb7e01b6970 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800bdf0 tx=0x7fb7d800bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:41.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.165+0000 7fb7c7fff640 1 -- 172.21.15.169:0/3863308647 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d8019070 con 0x7fb7e010d9c0 2024-03-28T01:02:41.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.165+0000 7fb7c7fff640 1 -- 172.21.15.169:0/3863308647 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7d8014920 con 0x7fb7e010d9c0 2024-03-28T01:02:41.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.165+0000 7fb7e50dc640 1 -- 172.21.15.169:0/3863308647 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e01b7080 con 0x7fb7e010d9c0 2024-03-28T01:02:41.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.165+0000 7fb7e50dc640 1 -- 172.21.15.169:0/3863308647 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e01b7570 con 0x7fb7e010d9c0 2024-03-28T01:02:41.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.166+0000 7fb7c7fff640 1 -- 172.21.15.169:0/3863308647 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d8012880 con 0x7fb7e010d9c0 2024-03-28T01:02:41.174 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.170+0000 7fb7e7da1740 1 Processor -- start 2024-03-28T01:02:41.174 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.171+0000 7fb7c7fff640 1 -- 172.21.15.169:0/3863308647 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb7d801c070 con 0x7fb7e010d9c0 2024-03-28T01:02:41.174 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.171+0000 7fb7e7da1740 1 -- start start 2024-03-28T01:02:41.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.172+0000 7fb7e7da1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800f4da0 0x558f800f51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:41.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.172+0000 7fb7e7da1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800f5790 0x558f800fa290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:41.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.172+0000 7fb7e7da1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f800fa900 0x558f800fcd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:41.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.172+0000 7fb7e7da1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f8005a3d0 con 0x558f800f4da0 2024-03-28T01:02:41.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.172+0000 7fb7e7da1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f80002ad0 con 0x558f800fa900 2024-03-28T01:02:41.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.172+0000 7fb7e7da1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f80056470 con 0x558f800f5790 2024-03-28T01:02:41.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.173+0000 7fb7c7fff640 1 --2- 172.21.15.169:0/3863308647 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb7c80a2a00 0x7fb7c80a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:41.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.173+0000 7fb7c7fff640 1 -- 172.21.15.169:0/3863308647 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb7d80cf5b0 con 0x7fb7e010d9c0 2024-03-28T01:02:41.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.174+0000 7fb7df59e640 1 --2- 172.21.15.169:0/3863308647 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb7c80a2a00 0x7fb7c80a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:41.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.177+0000 7fb7df59e640 1 --2- 172.21.15.169:0/3863308647 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb7c80a2a00 0x7fb7c80a4ec0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb7d809bb90 tx=0x7fb7d8004b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:41.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.177+0000 7fb7de59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f800fa900 0x558f800fcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:41.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.177+0000 7fb7de59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f800fa900 0x558f800fcd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57836/0 (socket says 172.21.15.169:57836) 2024-03-28T01:02:41.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.177+0000 7fb7de59c640 1 -- 172.21.15.169:0/3358278327 learned_addr learned my addr 172.21.15.169:0/3358278327 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:41.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.177+0000 7fb7ded9d640 1 --2- 172.21.15.169:0/3358278327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800f5790 0x558f800fa290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:41.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.180+0000 7fb7ded9d640 1 -- 172.21.15.169:0/3358278327 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f800fa900 msgr2=0x558f800fcd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:41.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.180+0000 7fb7ded9d640 1 --2- 172.21.15.169:0/3358278327 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f800fa900 0x558f800fcd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.180+0000 7fb7ded9d640 1 -- 172.21.15.169:0/3358278327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800f4da0 msgr2=0x558f800f51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:41.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.180+0000 7fb7ded9d640 1 --2- 172.21.15.169:0/3358278327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800f4da0 0x558f800f51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.180+0000 7fb7ded9d640 1 -- 172.21.15.169:0/3358278327 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f7fe387d0 con 0x558f800f5790 2024-03-28T01:02:41.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.181+0000 7fb7ded9d640 1 --2- 172.21.15.169:0/3358278327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800f5790 0x558f800fa290 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0009850 tx=0x7fb7d002c6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82e5b6a862e1165f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:41.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.193+0000 7fb7c57fa640 1 -- 172.21.15.169:0/3358278327 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0042030 con 0x558f800f5790 2024-03-28T01:02:41.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.194+0000 7fb7c57fa640 1 -- 172.21.15.169:0/3358278327 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7d002cbc0 con 0x558f800f5790 2024-03-28T01:02:41.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.194+0000 7fb7c57fa640 1 -- 172.21.15.169:0/3358278327 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0004920 con 0x558f800f5790 2024-03-28T01:02:41.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.196+0000 7fb7e7da1740 1 -- 172.21.15.169:0/3358278327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800f5790 msgr2=0x558f800fa290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:41.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.196+0000 7fb7e7da1740 1 --2- 172.21.15.169:0/3358278327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800f5790 0x558f800fa290 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0009850 tx=0x7fb7d002c6d0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.198+0000 7fb7e7da1740 1 -- 172.21.15.169:0/3358278327 shutdown_connections 2024-03-28T01:02:41.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.198+0000 7fb7e7da1740 1 --2- 172.21.15.169:0/3358278327 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f800fa900 0x558f800fcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.198+0000 7fb7e7da1740 1 --2- 172.21.15.169:0/3358278327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800f5790 0x558f800fa290 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.198+0000 7fb7e7da1740 1 --2- 172.21.15.169:0/3358278327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800f4da0 0x558f800f51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.198+0000 7fb7e7da1740 1 -- 172.21.15.169:0/3358278327 >> 172.21.15.169:0/3358278327 conn(0x558f800f0310 msgr2=0x558f800f0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:41.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.203+0000 7fb7e7da1740 1 -- 172.21.15.169:0/3358278327 shutdown_connections 2024-03-28T01:02:41.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.203+0000 7fb7e7da1740 1 -- 172.21.15.169:0/3358278327 wait complete. 2024-03-28T01:02:41.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.203+0000 7fb7e7da1740 1 -- start start 2024-03-28T01:02:41.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.204+0000 7fb7e7da1740 1 Processor -- start 2024-03-28T01:02:41.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.204+0000 7fb7e7da1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f800f4da0 0x558f800ff260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:41.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.204+0000 7fb7e7da1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800f5790 0x558f800ff7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:41.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.204+0000 7fb7e7da1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800fa900 0x558f801067e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:41.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.204+0000 7fb7e7da1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f800fd850 con 0x558f800f5790 2024-03-28T01:02:41.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.204+0000 7fb7e7da1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f80016060 con 0x558f800f4da0 2024-03-28T01:02:41.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.204+0000 7fb7e7da1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f800fd9d0 con 0x558f800fa900 2024-03-28T01:02:41.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.204+0000 7fb7e7da1740 10 client.0 _ll_register_callbacks cb 0x558f7fff86a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:41.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.205+0000 7fb7ded9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800f5790 0x558f800ff7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:41.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.205+0000 7fb7ded9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800f5790 0x558f800ff7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39114/0 (socket says 172.21.15.169:39114) 2024-03-28T01:02:41.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.205+0000 7fb7de59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800fa900 0x558f801067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:41.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.205+0000 7fb7ded9d640 1 -- 172.21.15.169:0/1631658559 learned_addr learned my addr 172.21.15.169:0/1631658559 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:41.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.206+0000 7fb7ded9d640 1 -- 172.21.15.169:0/1631658559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800fa900 msgr2=0x558f801067e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:41.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.206+0000 7fb7ded9d640 1 --2- 172.21.15.169:0/1631658559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800fa900 0x558f801067e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.206+0000 7fb7ded9d640 1 -- 172.21.15.169:0/1631658559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f800f4da0 msgr2=0x558f800ff260 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:02:41.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.206+0000 7fb7ded9d640 1 --2- 172.21.15.169:0/1631658559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f800f4da0 0x558f800ff260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.206+0000 7fb7ded9d640 1 -- 172.21.15.169:0/1631658559 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f7fd6e3f0 con 0x558f800f5790 2024-03-28T01:02:41.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.207+0000 7fb7ded9d640 1 --2- 172.21.15.169:0/1631658559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800f5790 0x558f800ff7a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0000c00 tx=0x7fb7d0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ea9ea206fed020f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:41.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.207+0000 7fb7c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:02:41.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.208+0000 7fb7c2ff5640 1 -- 172.21.15.169:0/1631658559 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0042070 con 0x558f800f5790 2024-03-28T01:02:41.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.208+0000 7fb7c2ff5640 1 -- 172.21.15.169:0/1631658559 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb7d00048c0 con 0x558f800f5790 2024-03-28T01:02:41.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.208+0000 7fb7e7da1740 1 -- 172.21.15.169:0/1631658559 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb7d002cbc0 con 0x558f800f5790 2024-03-28T01:02:41.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.212+0000 7fb7c2ff5640 1 -- 172.21.15.169:0/1631658559 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7d0055050 con 0x558f800f5790 2024-03-28T01:02:41.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.213+0000 7fb7e7da1740 10 client.16656 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:41.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.213+0000 7fb7e7da1740 1 -- 172.21.15.169:0/1631658559 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558f7fd2e0f0 con 0x558f800f5790 2024-03-28T01:02:41.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.214+0000 7fb7c2ff5640 1 -- 172.21.15.169:0/1631658559 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7d002cbc0 con 0x558f800f5790 2024-03-28T01:02:41.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.214+0000 7fb7c2ff5640 1 -- 172.21.15.169:0/1631658559 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb7d0003cd0 con 0x558f800f5790 2024-03-28T01:02:41.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.214+0000 7fb7e7da1740 10 client.16656 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:41.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.214+0000 7fb7e7da1740 10 client.16656 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:02:41.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.214+0000 7fb7e7da1740 20 client.16656 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:41.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.214+0000 7fb7e7da1740 1 --2- 172.21.15.169:0/1631658559 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558f8010ae30 0x558f8010d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:41.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.214+0000 7fb7e7da1740 4 client.16656 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:02:41.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.214+0000 7fb7e7da1740 1 -- 172.21.15.169:0/1631658559 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558f80067910 con 0x558f8010ae30 2024-03-28T01:02:41.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.214+0000 7fb7de59c640 1 --2- 172.21.15.169:0/1631658559 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558f8010ae30 0x558f8010d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:41.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.216+0000 7fb7c2ff5640 10 client.16656 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:41.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.217+0000 7fb7de59c640 1 --2- 172.21.15.169:0/1631658559 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558f8010ae30 0x558f8010d290 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee9b6e81827dfcd2 server_cookie=f7e6292a6f9c764 in_seq=0 out_seq=0 2024-03-28T01:02:41.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.227+0000 7fb7c2ff5640 1 -- 172.21.15.169:0/1631658559 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558f80067910 con 0x558f8010ae30 2024-03-28T01:02:41.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.227+0000 7fb7c2ff5640 10 client.16656 handle_command_reply: tid=0 2024-03-28T01:02:41.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.228+0000 7fb7e7da1740 1 client.16656 shutdown 2024-03-28T01:02:41.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.231+0000 7fb7e7da1740 1 -- 172.21.15.169:0/1631658559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800f5790 msgr2=0x558f800ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:41.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.231+0000 7fb7e7da1740 1 --2- 172.21.15.169:0/1631658559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800f5790 0x558f800ff7a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0000c00 tx=0x7fb7d0004270 comp rx=0 tx=0).stop 2024-03-28T01:02:41.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.231+0000 7fb7e7da1740 1 -- 172.21.15.169:0/1631658559 shutdown_connections 2024-03-28T01:02:41.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.231+0000 7fb7e7da1740 1 --2- 172.21.15.169:0/1631658559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800fa900 0x558f801067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.231+0000 7fb7e7da1740 1 --2- 172.21.15.169:0/1631658559 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558f8010ae30 0x558f8010d290 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.231+0000 7fb7e7da1740 1 --2- 172.21.15.169:0/1631658559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800f5790 0x558f800ff7a0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.231+0000 7fb7e7da1740 1 --2- 172.21.15.169:0/1631658559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f800f4da0 0x558f800ff260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.231+0000 7fb7e7da1740 1 -- 172.21.15.169:0/1631658559 >> 172.21.15.169:0/1631658559 conn(0x558f800f0310 msgr2=0x558f800ef530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:41.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.231+0000 7fb7c2ff5640 0 client.16656 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:41.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.232+0000 7fb7e7da1740 1 -- 172.21.15.169:0/1631658559 shutdown_connections 2024-03-28T01:02:41.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.234+0000 7fb7e7da1740 1 -- 172.21.15.169:0/1631658559 wait complete. 2024-03-28T01:02:41.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.236+0000 7fb7e7da1740 20 client.16656 trim_cache size 0 max 16384 2024-03-28T01:02:41.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.279+0000 7fb7e7da1740 1 Processor -- start 2024-03-28T01:02:41.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.281+0000 7fb7e7da1740 1 -- start start 2024-03-28T01:02:41.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.282+0000 7fb7e7da1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f800f4da0 0x558f800fddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:41.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.282+0000 7fb7e7da1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800f5790 0x558f801067a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:41.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.282+0000 7fb7e7da1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800fa900 0x558f800fe310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:41.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.282+0000 7fb7de59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f800f4da0 0x558f800fddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:41.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.282+0000 7fb7de59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f800f4da0 0x558f800fddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57854/0 (socket says 172.21.15.169:57854) 2024-03-28T01:02:41.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.282+0000 7fb7ded9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800f5790 0x558f801067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:41.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.282+0000 7fb7ded9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800f5790 0x558f801067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:42100/0 (socket says 172.21.15.169:42100) 2024-03-28T01:02:41.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.282+0000 7fb7de59c640 1 -- 172.21.15.169:0/3330056822 learned_addr learned my addr 172.21.15.169:0/3330056822 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:41.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.282+0000 7fb7e7da1740 1 -- 172.21.15.169:0/3330056822 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc0120e0 con 0x558f800fa900 2024-03-28T01:02:41.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.282+0000 7fb7e7da1740 1 -- 172.21.15.169:0/3330056822 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc00fbe0 con 0x558f800f4da0 2024-03-28T01:02:41.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.287+0000 7fb7e7da1740 1 -- 172.21.15.169:0/3330056822 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc00dc80 con 0x558f800f5790 2024-03-28T01:02:41.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.289+0000 7fb7ded9d640 1 -- 172.21.15.169:0/3330056822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f800f4da0 msgr2=0x558f800fddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:41.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.289+0000 7fb7ded9d640 1 --2- 172.21.15.169:0/3330056822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f800f4da0 0x558f800fddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.289+0000 7fb7ded9d640 1 -- 172.21.15.169:0/3330056822 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800fa900 msgr2=0x558f800fe310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:41.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.289+0000 7fb7ded9d640 1 --2- 172.21.15.169:0/3330056822 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800fa900 0x558f800fe310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.289+0000 7fb7ded9d640 1 -- 172.21.15.169:0/3330056822 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f800efc60 con 0x558f800f5790 2024-03-28T01:02:41.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.290+0000 7fb7ded9d640 1 --2- 172.21.15.169:0/3330056822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800f5790 0x558f801067a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0034520 tx=0x7fb7d002c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c57b50332fb8ca6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:41.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.290+0000 7fb7c4ff9640 1 -- 172.21.15.169:0/3330056822 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0042030 con 0x558f800f5790 2024-03-28T01:02:41.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.290+0000 7fb7c4ff9640 1 -- 172.21.15.169:0/3330056822 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7d004c070 con 0x558f800f5790 2024-03-28T01:02:41.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.290+0000 7fb7c4ff9640 1 -- 172.21.15.169:0/3330056822 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0004920 con 0x558f800f5790 2024-03-28T01:02:41.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.295+0000 7fb7e7da1740 1 -- 172.21.15.169:0/3330056822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800f5790 msgr2=0x558f801067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:41.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.295+0000 7fb7e7da1740 1 --2- 172.21.15.169:0/3330056822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800f5790 0x558f801067a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0034520 tx=0x7fb7d002c900 comp rx=0 tx=0).stop 2024-03-28T01:02:41.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.295+0000 7fb7e7da1740 1 -- 172.21.15.169:0/3330056822 shutdown_connections 2024-03-28T01:02:41.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.295+0000 7fb7e7da1740 1 --2- 172.21.15.169:0/3330056822 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800fa900 0x558f800fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.295+0000 7fb7e7da1740 1 --2- 172.21.15.169:0/3330056822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800f5790 0x558f801067a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.296+0000 7fb7e7da1740 1 --2- 172.21.15.169:0/3330056822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f800f4da0 0x558f800fddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.296+0000 7fb7e7da1740 1 -- 172.21.15.169:0/3330056822 >> 172.21.15.169:0/3330056822 conn(0x558f800f0310 msgr2=0x558f800eeca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:41.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.298+0000 7fb7e7da1740 1 -- 172.21.15.169:0/3330056822 shutdown_connections 2024-03-28T01:02:41.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.298+0000 7fb7e7da1740 1 -- 172.21.15.169:0/3330056822 wait complete. 2024-03-28T01:02:41.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.299+0000 7fb7e7da1740 1 -- start start 2024-03-28T01:02:41.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.299+0000 7fb7e7da1740 1 Processor -- start 2024-03-28T01:02:41.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.300+0000 7fb7e7da1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800f4da0 0x558f801044c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:41.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.300+0000 7fb7e7da1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f800f5790 0x558f80104a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:41.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.301+0000 7fb7e7da1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800fa900 0x558f801059c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:41.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.301+0000 7fb7de59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800f4da0 0x558f801044c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:41.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.301+0000 7fb7de59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800f4da0 0x558f801044c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39130/0 (socket says 172.21.15.169:39130) 2024-03-28T01:02:41.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.301+0000 7fb7de59c640 1 -- 172.21.15.169:0/2902149754 learned_addr learned my addr 172.21.15.169:0/2902149754 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:41.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.301+0000 7fb7e7da1740 1 -- 172.21.15.169:0/2902149754 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc00e280 con 0x558f800f4da0 2024-03-28T01:02:41.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.301+0000 7fb7e7da1740 1 -- 172.21.15.169:0/2902149754 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc00f580 con 0x558f800f5790 2024-03-28T01:02:41.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.301+0000 7fb7e7da1740 1 -- 172.21.15.169:0/2902149754 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc00e8c0 con 0x558f800fa900 2024-03-28T01:02:41.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.301+0000 7fb7e7da1740 10 client.0 _ll_register_callbacks cb 0x7fb7d40095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:41.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.301+0000 7fb7ded9d640 1 --2- 172.21.15.169:0/2902149754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f800f5790 0x558f80104a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:41.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.302+0000 7fb7de59c640 1 --2- 172.21.15.169:0/2902149754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800fa900 0x558f801059c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:41.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.302+0000 7fb7de59c640 1 -- 172.21.15.169:0/2902149754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800fa900 msgr2=0x558f801059c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:41.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.302+0000 7fb7de59c640 1 --2- 172.21.15.169:0/2902149754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800fa900 0x558f801059c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.302+0000 7fb7de59c640 1 -- 172.21.15.169:0/2902149754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f800f5790 msgr2=0x558f80104a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:41.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.302+0000 7fb7de59c640 1 --2- 172.21.15.169:0/2902149754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f800f5790 0x558f80104a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.302+0000 7fb7de59c640 1 -- 172.21.15.169:0/2902149754 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f80105fb0 con 0x558f800f4da0 2024-03-28T01:02:41.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.308+0000 7fb7de59c640 1 --2- 172.21.15.169:0/2902149754 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800f4da0 0x558f801044c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x558f800f5ff0 tx=0x7fb7d4009670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=623377c3b92b55fc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:41.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.308+0000 7fb7c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:02:41.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.315+0000 7fb7c2ff5640 1 -- 172.21.15.169:0/2902149754 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d40180d0 con 0x558f800f4da0 2024-03-28T01:02:41.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.315+0000 7fb7c2ff5640 1 -- 172.21.15.169:0/2902149754 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb7d400f040 con 0x558f800f4da0 2024-03-28T01:02:41.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.316+0000 7fb7e7da1740 1 -- 172.21.15.169:0/2902149754 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb7d004c070 con 0x558f800f4da0 2024-03-28T01:02:41.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.317+0000 7fb7c2ff5640 1 -- 172.21.15.169:0/2902149754 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7d4018270 con 0x558f800f4da0 2024-03-28T01:02:41.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.317+0000 7fb7e7da1740 10 client.16659 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:41.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.317+0000 7fb7e7da1740 1 -- 172.21.15.169:0/2902149754 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558f80166d50 con 0x558f800f4da0 2024-03-28T01:02:41.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.318+0000 7fb7c2ff5640 1 -- 172.21.15.169:0/2902149754 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7d4018c00 con 0x558f800f4da0 2024-03-28T01:02:41.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.318+0000 7fb7c2ff5640 1 -- 172.21.15.169:0/2902149754 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb7d40169a0 con 0x558f800f4da0 2024-03-28T01:02:41.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.318+0000 7fb7e7da1740 10 client.16659 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:41.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.318+0000 7fb7e7da1740 10 client.16659 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:02:41.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.318+0000 7fb7e7da1740 20 client.16659 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:41.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.318+0000 7fb7e7da1740 1 --2- 172.21.15.169:0/2902149754 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558f80167530 0x558f80169990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:41.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.318+0000 7fb7e7da1740 4 client.16659 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:02:41.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.319+0000 7fb7e7da1740 1 -- 172.21.15.169:0/2902149754 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7fb7d004c070 con 0x558f80167530 2024-03-28T01:02:41.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.320+0000 7fb7ded9d640 1 --2- 172.21.15.169:0/2902149754 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558f80167530 0x558f80169990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:41.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.326+0000 7fb7ded9d640 1 --2- 172.21.15.169:0/2902149754 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558f80167530 0x558f80169990 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14844d87a1eb165d server_cookie=6f26582340311d5f in_seq=0 out_seq=0 2024-03-28T01:02:41.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.326+0000 7fb7c2ff5640 10 client.16659 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:41.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.330+0000 7fb7c2ff5640 1 -- 172.21.15.169:0/2902149754 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x7fb7d004c070 con 0x558f80167530 2024-03-28T01:02:41.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.330+0000 7fb7c2ff5640 10 client.16659 handle_command_reply: tid=0 2024-03-28T01:02:41.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.330+0000 7fb7e7da1740 1 client.16659 shutdown 2024-03-28T01:02:41.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.330+0000 7fb7e7da1740 1 -- 172.21.15.169:0/2902149754 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800f4da0 msgr2=0x558f801044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:41.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.331+0000 7fb7e7da1740 1 --2- 172.21.15.169:0/2902149754 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800f4da0 0x558f801044c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x558f800f5ff0 tx=0x7fb7d4009670 comp rx=0 tx=0).stop 2024-03-28T01:02:41.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.331+0000 7fb7e7da1740 1 -- 172.21.15.169:0/2902149754 shutdown_connections 2024-03-28T01:02:41.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.331+0000 7fb7e7da1740 1 --2- 172.21.15.169:0/2902149754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f800fa900 0x558f801059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.331+0000 7fb7e7da1740 1 --2- 172.21.15.169:0/2902149754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f800f5790 0x558f80104a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.331+0000 7fb7e7da1740 1 --2- 172.21.15.169:0/2902149754 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558f80167530 0x558f80169990 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.331+0000 7fb7e7da1740 1 --2- 172.21.15.169:0/2902149754 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f800f4da0 0x558f801044c0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.332+0000 7fb7e7da1740 1 -- 172.21.15.169:0/2902149754 >> 172.21.15.169:0/2902149754 conn(0x558f800f0310 msgr2=0x558f800f3570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:41.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.332+0000 7fb7c2ff5640 0 client.16659 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:41.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.333+0000 7fb7e7da1740 1 -- 172.21.15.169:0/2902149754 shutdown_connections 2024-03-28T01:02:41.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.334+0000 7fb7e7da1740 1 -- 172.21.15.169:0/2902149754 wait complete. 2024-03-28T01:02:41.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.336+0000 7fb7e7da1740 20 client.16659 trim_cache size 0 max 16384 2024-03-28T01:02:41.342 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:02:41.342 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "scrub active (2 inodes in the stack)", 2024-03-28T01:02:41.342 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": { 2024-03-28T01:02:41.342 INFO:teuthology.orchestra.run.smithi169.stdout: "e41d5635-a3bb-4cd0-8193-81bd9af97a5e": { 2024-03-28T01:02:41.342 INFO:teuthology.orchestra.run.smithi169.stdout: "path": "/", 2024-03-28T01:02:41.342 INFO:teuthology.orchestra.run.smithi169.stdout: "tag": "e41d5635-a3bb-4cd0-8193-81bd9af97a5e", 2024-03-28T01:02:41.342 INFO:teuthology.orchestra.run.smithi169.stdout: "options": "recursive,force" 2024-03-28T01:02:41.342 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:02:41.342 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:02:41.342 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:02:41.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.342+0000 7fb7e50dc640 1 -- 172.21.15.169:0/3863308647 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb7c80a2a00 msgr2=0x7fb7c80a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:41.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.342+0000 7fb7e50dc640 1 --2- 172.21.15.169:0/3863308647 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb7c80a2a00 0x7fb7c80a4ec0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb7d809bb90 tx=0x7fb7d8004b50 comp rx=0 tx=0).stop 2024-03-28T01:02:41.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.342+0000 7fb7e50dc640 1 -- 172.21.15.169:0/3863308647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7e010d9c0 msgr2=0x7fb7e01b6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:41.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.342+0000 7fb7e50dc640 1 --2- 172.21.15.169:0/3863308647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7e010d9c0 0x7fb7e01b6970 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800bdf0 tx=0x7fb7d800bee0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.344+0000 7fb7e50dc640 1 -- 172.21.15.169:0/3863308647 shutdown_connections 2024-03-28T01:02:41.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.344+0000 7fb7e50dc640 1 --2- 172.21.15.169:0/3863308647 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb7c80a2a00 0x7fb7c80a4ec0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.344+0000 7fb7e50dc640 1 --2- 172.21.15.169:0/3863308647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7e010d9c0 0x7fb7e01b6970 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.344+0000 7fb7e50dc640 1 --2- 172.21.15.169:0/3863308647 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7e010b010 0x7fb7e0117320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.344+0000 7fb7e50dc640 1 --2- 172.21.15.169:0/3863308647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7e01086a0 0x7fb7e0116de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:41.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.344+0000 7fb7e50dc640 1 -- 172.21.15.169:0/3863308647 >> 172.21.15.169:0/3863308647 conn(0x7fb7e006e500 msgr2=0x7fb7e0108ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:41.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.346+0000 7fb7e50dc640 1 -- 172.21.15.169:0/3863308647 shutdown_connections 2024-03-28T01:02:41.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:02:41.347+0000 7fb7e50dc640 1 -- 172.21.15.169:0/3863308647 wait complete. 2024-03-28T01:02:41.382 INFO:tasks.cephfs.filesystem:scrub status for tag:e41d5635-a3bb-4cd0-8193-81bd9af97a5e - {'path': '/', 'tag': 'e41d5635-a3bb-4cd0-8193-81bd9af97a5e', 'options': 'recursive,force'} 2024-03-28T01:02:41.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:41 smithi169 ceph-mon[32274]: pgmap v536: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 599 MiB/s wr, 1.37k op/s 2024-03-28T01:02:41.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:41 smithi169 ceph-mon[32274]: scrub queued for path: / 2024-03-28T01:02:41.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:41 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:02:41.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:41 smithi169 ceph-mon[32274]: scrub summary: active paths [/] 2024-03-28T01:02:41.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:41 smithi174 ceph-mon[36060]: pgmap v536: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 599 MiB/s wr, 1.37k op/s 2024-03-28T01:02:41.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:41 smithi174 ceph-mon[36060]: scrub queued for path: / 2024-03-28T01:02:41.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:41 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:02:41.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:41 smithi174 ceph-mon[36060]: scrub summary: active paths [/] 2024-03-28T01:02:41.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:41 smithi177 ceph-mon[36206]: pgmap v536: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 599 MiB/s wr, 1.37k op/s 2024-03-28T01:02:41.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:41 smithi177 ceph-mon[36206]: scrub queued for path: / 2024-03-28T01:02:41.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:41 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:02:41.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:41 smithi177 ceph-mon[36206]: scrub summary: active paths [/] 2024-03-28T01:02:43.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:43 smithi169 ceph-mon[32274]: pgmap v537: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 763 MiB/s wr, 1.35k op/s 2024-03-28T01:02:43.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:43 smithi174 ceph-mon[36060]: pgmap v537: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 763 MiB/s wr, 1.35k op/s 2024-03-28T01:02:43.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:43 smithi177 ceph-mon[36206]: pgmap v537: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 763 MiB/s wr, 1.35k op/s 2024-03-28T01:02:45.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:45 smithi169 ceph-mon[32274]: pgmap v538: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 631 MiB/s wr, 1.09k op/s 2024-03-28T01:02:45.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:45 smithi174 ceph-mon[36060]: pgmap v538: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 631 MiB/s wr, 1.09k op/s 2024-03-28T01:02:45.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:45 smithi177 ceph-mon[36206]: pgmap v538: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 631 MiB/s wr, 1.09k op/s 2024-03-28T01:02:46.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:46 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:02:46.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:46 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:02:47.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:46 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:02:47.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:47 smithi174 ceph-mon[36060]: pgmap v539: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 730 MiB/s wr, 1.20k op/s 2024-03-28T01:02:47.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:47 smithi177 ceph-mon[36206]: pgmap v539: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 730 MiB/s wr, 1.20k op/s 2024-03-28T01:02:48.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:47 smithi169 ceph-mon[32274]: pgmap v539: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 730 MiB/s wr, 1.20k op/s 2024-03-28T01:02:48.782 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.778+0000 7f332a964640 1 -- 172.21.15.169:0/1168853630 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3324072950 msgr2=0x7f3324072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:48.782 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.778+0000 7f332a964640 1 --2- 172.21.15.169:0/1168853630 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3324072950 0x7f3324072d70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3318009900 tx=0x7f331802bc60 comp rx=0 tx=0).stop 2024-03-28T01:02:48.784 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.781+0000 7f332a964640 1 -- 172.21.15.169:0/1168853630 shutdown_connections 2024-03-28T01:02:48.784 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.781+0000 7f332a964640 1 --2- 172.21.15.169:0/1168853630 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3324109390 0x7f332410e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.785 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.781+0000 7f332a964640 1 --2- 172.21.15.169:0/1168853630 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33240732b0 0x7f3324108cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.785 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.781+0000 7f332a964640 1 --2- 172.21.15.169:0/1168853630 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3324072950 0x7f3324072d70 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.785 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.781+0000 7f332a964640 1 -- 172.21.15.169:0/1168853630 >> 172.21.15.169:0/1168853630 conn(0x7f332406e510 msgr2=0x7f3324070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:48.789 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.787+0000 7f332a964640 1 -- 172.21.15.169:0/1168853630 shutdown_connections 2024-03-28T01:02:48.793 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.791+0000 7f332a964640 1 -- 172.21.15.169:0/1168853630 wait complete. 2024-03-28T01:02:48.797 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.795+0000 7f332a964640 1 Processor -- start 2024-03-28T01:02:48.800 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.797+0000 7f332a964640 1 -- start start 2024-03-28T01:02:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.798+0000 7f332a964640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3324072950 0x7f33241a6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.798+0000 7f332a964640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33240732b0 0x7f33241a6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.798+0000 7f332a964640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3324109390 0x7f33241adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.798+0000 7f332a964640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332411c0b0 con 0x7f33240732b0 2024-03-28T01:02:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.799+0000 7f332a964640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332411bf30 con 0x7f3324072950 2024-03-28T01:02:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.799+0000 7f332a964640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332411c230 con 0x7f3324109390 2024-03-28T01:02:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.800+0000 7f332a163640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3324109390 0x7f33241adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.800+0000 7f332a163640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3324109390 0x7f33241adad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48530/0 (socket says 172.21.15.169:48530) 2024-03-28T01:02:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.800+0000 7f332a163640 1 -- 172.21.15.169:0/775036140 learned_addr learned my addr 172.21.15.169:0/775036140 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.800+0000 7f3329962640 1 --2- 172.21.15.169:0/775036140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3324072950 0x7f33241a6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.801+0000 7f332a163640 1 -- 172.21.15.169:0/775036140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3324072950 msgr2=0x7f33241a6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.801+0000 7f332a163640 1 --2- 172.21.15.169:0/775036140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3324072950 0x7f33241a6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.801+0000 7f332a163640 1 -- 172.21.15.169:0/775036140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33240732b0 msgr2=0x7f33241a6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.801+0000 7f332a163640 1 --2- 172.21.15.169:0/775036140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33240732b0 0x7f33241a6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.801+0000 7f332a163640 1 -- 172.21.15.169:0/775036140 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33241ae140 con 0x7f3324109390 2024-03-28T01:02:48.805 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.801+0000 7f332a163640 1 --2- 172.21.15.169:0/775036140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3324109390 0x7f33241adad0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f331400b970 tx=0x7f331400be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:48.807 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.803+0000 7f3322ffd640 1 -- 172.21.15.169:0/775036140 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3314019070 con 0x7f3324109390 2024-03-28T01:02:48.808 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.803+0000 7f3322ffd640 1 -- 172.21.15.169:0/775036140 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f331400cb40 con 0x7f3324109390 2024-03-28T01:02:48.808 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.804+0000 7f3322ffd640 1 -- 172.21.15.169:0/775036140 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3314007650 con 0x7f3324109390 2024-03-28T01:02:48.808 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.805+0000 7f332a964640 1 -- 172.21.15.169:0/775036140 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33241ae430 con 0x7f3324109390 2024-03-28T01:02:48.808 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.805+0000 7f332a964640 1 -- 172.21.15.169:0/775036140 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33241ae8a0 con 0x7f3324109390 2024-03-28T01:02:48.812 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.809+0000 7f332e7b1740 1 Processor -- start 2024-03-28T01:02:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.809+0000 7f332e7b1740 1 -- start start 2024-03-28T01:02:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.812+0000 7f332e7b1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562ad984de40 0x562ad984e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.812+0000 7f332e7b1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad984e830 0x562ad9853330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.812+0000 7f332e7b1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad98539a0 0x562ad9855e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.812+0000 7f332e7b1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ad97b2b10 con 0x562ad984de40 2024-03-28T01:02:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.813+0000 7f3329962640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad984e830 0x562ad9853330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.813+0000 7f3329962640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad984e830 0x562ad9853330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:35582/0 (socket says 172.21.15.169:35582) 2024-03-28T01:02:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.813+0000 7f3329962640 1 -- 172.21.15.169:0/3820489839 learned_addr learned my addr 172.21.15.169:0/3820489839 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.813+0000 7f332e7b1740 1 -- 172.21.15.169:0/3820489839 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ad975b900 con 0x562ad984e830 2024-03-28T01:02:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.813+0000 7f3322ffd640 1 -- 172.21.15.169:0/775036140 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f331400ccf0 con 0x7f3324109390 2024-03-28T01:02:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.813+0000 7f332e7b1740 1 -- 172.21.15.169:0/3820489839 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ad97aebb0 con 0x562ad98539a0 2024-03-28T01:02:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.815+0000 7f3322ffd640 1 --2- 172.21.15.169:0/775036140 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f330c0a28f0 0x7f330c0a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.816+0000 7f3322ffd640 1 -- 172.21.15.169:0/775036140 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f33140ce1f0 con 0x7f3324109390 2024-03-28T01:02:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.817+0000 7f3329962640 1 -- 172.21.15.169:0/3820489839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad98539a0 msgr2=0x562ad9855e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:02:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.817+0000 7f3329962640 1 --2- 172.21.15.169:0/3820489839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad98539a0 0x562ad9855e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.817+0000 7f3329962640 1 -- 172.21.15.169:0/3820489839 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562ad984de40 msgr2=0x562ad984e260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:02:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.817+0000 7f3329962640 1 --2- 172.21.15.169:0/3820489839 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562ad984de40 0x562ad984e260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.817+0000 7f3329962640 1 -- 172.21.15.169:0/3820489839 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ad94c7210 con 0x562ad984e830 2024-03-28T01:02:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.819+0000 7f332a163640 1 --2- 172.21.15.169:0/775036140 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f330c0a28f0 0x7f330c0a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.820+0000 7f3329962640 1 --2- 172.21.15.169:0/3820489839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad984e830 0x562ad9853330 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3318005c20 tx=0x7f3318009610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96d83f3842de961b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:48.829 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.825+0000 7f330b7fe640 1 -- 172.21.15.169:0/3820489839 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3318042070 con 0x562ad984e830 2024-03-28T01:02:48.829 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.825+0000 7f330b7fe640 1 -- 172.21.15.169:0/3820489839 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33180046c0 con 0x562ad984e830 2024-03-28T01:02:48.831 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.827+0000 7f332e7b1740 1 -- 172.21.15.169:0/3820489839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad984e830 msgr2=0x562ad9853330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:48.831 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.827+0000 7f332e7b1740 1 --2- 172.21.15.169:0/3820489839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad984e830 0x562ad9853330 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3318005c20 tx=0x7f3318009610 comp rx=0 tx=0).stop 2024-03-28T01:02:48.831 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.828+0000 7f332e7b1740 1 -- 172.21.15.169:0/3820489839 shutdown_connections 2024-03-28T01:02:48.831 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.828+0000 7f332e7b1740 1 --2- 172.21.15.169:0/3820489839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad98539a0 0x562ad9855e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.831 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.828+0000 7f332e7b1740 1 --2- 172.21.15.169:0/3820489839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad984e830 0x562ad9853330 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.831 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.828+0000 7f332e7b1740 1 --2- 172.21.15.169:0/3820489839 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562ad984de40 0x562ad984e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.831 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.828+0000 7f332e7b1740 1 -- 172.21.15.169:0/3820489839 >> 172.21.15.169:0/3820489839 conn(0x562ad98493b0 msgr2=0x562ad98497b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:48.834 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.831+0000 7f332e7b1740 1 -- 172.21.15.169:0/3820489839 shutdown_connections 2024-03-28T01:02:48.834 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.831+0000 7f332a163640 1 --2- 172.21.15.169:0/775036140 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f330c0a28f0 0x7f330c0a4db0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3314097210 tx=0x7f3314004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:48.839 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.833+0000 7f332e7b1740 1 -- 172.21.15.169:0/3820489839 wait complete. 2024-03-28T01:02:48.839 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.834+0000 7f332e7b1740 1 -- start start 2024-03-28T01:02:48.839 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.834+0000 7f332e7b1740 1 Processor -- start 2024-03-28T01:02:48.839 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.835+0000 7f332e7b1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad984de40 0x562ad9868e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:48.839 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.835+0000 7f332e7b1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad984e830 0x562ad98693b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.835+0000 7f332e7b1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562ad98539a0 0x562ad98703f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.835+0000 7f332e7b1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ad9856860 con 0x562ad98539a0 2024-03-28T01:02:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.835+0000 7f332e7b1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ad976ed50 con 0x562ad984de40 2024-03-28T01:02:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.835+0000 7f332e7b1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ad98569e0 con 0x562ad984e830 2024-03-28T01:02:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.835+0000 7f332e7b1740 10 client.0 _ll_register_callbacks cb 0x562ad976e720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.835+0000 7f3329161640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad984de40 0x562ad9868e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.836+0000 7f3329161640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad984de40 0x562ad9868e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:35586/0 (socket says 172.21.15.169:35586) 2024-03-28T01:02:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.836+0000 7f3329161640 1 -- 172.21.15.169:0/3593130460 learned_addr learned my addr 172.21.15.169:0/3593130460 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.836+0000 7f3329962640 1 --2- 172.21.15.169:0/3593130460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad984e830 0x562ad98693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.836+0000 7f3329161640 1 -- 172.21.15.169:0/3593130460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad984e830 msgr2=0x562ad98693b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.836+0000 7f3329161640 1 --2- 172.21.15.169:0/3593130460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad984e830 0x562ad98693b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.836+0000 7f3329161640 1 -- 172.21.15.169:0/3593130460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562ad98539a0 msgr2=0x562ad98703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.836+0000 7f3329161640 1 --2- 172.21.15.169:0/3593130460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562ad98539a0 0x562ad98703f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.836+0000 7f3329161640 1 -- 172.21.15.169:0/3593130460 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562ad9593620 con 0x562ad984de40 2024-03-28T01:02:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.836+0000 7f3329161640 1 --2- 172.21.15.169:0/3593130460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad984de40 0x562ad9868e70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f331c001e20 tx=0x7f331c001f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdcc5d2d70c50e1f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:48.845 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.839+0000 7f33097fa640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:02:48.845 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.839+0000 7f33097fa640 1 -- 172.21.15.169:0/3593130460 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331c0072d0 con 0x562ad984de40 2024-03-28T01:02:48.845 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.839+0000 7f33097fa640 1 -- 172.21.15.169:0/3593130460 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f331c007470 con 0x562ad984de40 2024-03-28T01:02:48.845 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.839+0000 7f332e7b1740 1 -- 172.21.15.169:0/3593130460 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f33180046c0 con 0x562ad984de40 2024-03-28T01:02:48.846 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.842+0000 7f33097fa640 1 -- 172.21.15.169:0/3593130460 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f331c008900 con 0x562ad984de40 2024-03-28T01:02:48.851 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.845+0000 7f332e7b1740 10 client.26567 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:48.851 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.845+0000 7f332e7b1740 1 -- 172.21.15.169:0/3593130460 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x562ad98593e0 con 0x562ad984de40 2024-03-28T01:02:48.851 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.846+0000 7f33097fa640 1 -- 172.21.15.169:0/3593130460 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33180046c0 con 0x562ad984de40 2024-03-28T01:02:48.852 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.846+0000 7f33097fa640 1 -- 172.21.15.169:0/3593130460 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f331c0137c0 con 0x562ad984de40 2024-03-28T01:02:48.852 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.846+0000 7f332e7b1740 10 client.26567 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:48.852 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.846+0000 7f332e7b1740 10 client.26567 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:02:48.852 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.846+0000 7f332e7b1740 20 client.26567 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:48.852 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.846+0000 7f332e7b1740 1 --2- 172.21.15.169:0/3593130460 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562ad985ae40 0x562ad985d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:48.852 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.846+0000 7f332e7b1740 4 client.26567 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:02:48.852 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.846+0000 7f332e7b1740 1 -- 172.21.15.169:0/3593130460 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562ad97c0050 con 0x562ad985ae40 2024-03-28T01:02:48.852 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.848+0000 7f3329962640 1 --2- 172.21.15.169:0/3593130460 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562ad985ae40 0x562ad985d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:48.852 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.848+0000 7f3329962640 1 --2- 172.21.15.169:0/3593130460 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562ad985ae40 0x562ad985d2e0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cda88e203376d8b1 server_cookie=e184aac913c1f290 in_seq=0 out_seq=0 2024-03-28T01:02:48.852 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.848+0000 7f33097fa640 10 client.26567 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:48.855 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.852+0000 7f33097fa640 1 -- 172.21.15.169:0/3593130460 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562ad97c0050 con 0x562ad985ae40 2024-03-28T01:02:48.855 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.852+0000 7f33097fa640 10 client.26567 handle_command_reply: tid=0 2024-03-28T01:02:48.855 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.852+0000 7f332e7b1740 1 client.26567 shutdown 2024-03-28T01:02:48.856 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.853+0000 7f332e7b1740 1 -- 172.21.15.169:0/3593130460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad984de40 msgr2=0x562ad9868e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:48.856 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.853+0000 7f332e7b1740 1 --2- 172.21.15.169:0/3593130460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad984de40 0x562ad9868e70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f331c001e20 tx=0x7f331c001f10 comp rx=0 tx=0).stop 2024-03-28T01:02:48.856 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.853+0000 7f332e7b1740 1 -- 172.21.15.169:0/3593130460 shutdown_connections 2024-03-28T01:02:48.856 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.853+0000 7f332e7b1740 1 --2- 172.21.15.169:0/3593130460 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562ad985ae40 0x562ad985d2e0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.856 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.853+0000 7f332e7b1740 1 --2- 172.21.15.169:0/3593130460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562ad98539a0 0x562ad98703f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.856 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.853+0000 7f332e7b1740 1 --2- 172.21.15.169:0/3593130460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad984e830 0x562ad98693b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.856 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.853+0000 7f332e7b1740 1 --2- 172.21.15.169:0/3593130460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad984de40 0x562ad9868e70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.856 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.853+0000 7f332e7b1740 1 -- 172.21.15.169:0/3593130460 >> 172.21.15.169:0/3593130460 conn(0x562ad98493b0 msgr2=0x562ad9848690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:48.856 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.853+0000 7f33097fa640 0 client.26567 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:48.856 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.853+0000 7f332e7b1740 1 -- 172.21.15.169:0/3593130460 shutdown_connections 2024-03-28T01:02:48.857 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.853+0000 7f332e7b1740 1 -- 172.21.15.169:0/3593130460 wait complete. 2024-03-28T01:02:48.857 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.854+0000 7f332e7b1740 20 client.26567 trim_cache size 0 max 16384 2024-03-28T01:02:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.883+0000 7f332e7b1740 1 Processor -- start 2024-03-28T01:02:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.883+0000 7f332e7b1740 1 -- start start 2024-03-28T01:02:48.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.884+0000 7f332e7b1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562ad984de40 0x562ad984faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.884+0000 7f332e7b1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad984e830 0x562ad9850030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.884+0000 7f332e7b1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad98539a0 0x562ad98703b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.885+0000 7f3329962640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad984e830 0x562ad9850030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.885+0000 7f3329962640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad984e830 0x562ad9850030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:35600/0 (socket says 172.21.15.169:35600) 2024-03-28T01:02:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.885+0000 7f3329962640 1 -- 172.21.15.169:0/2815569097 learned_addr learned my addr 172.21.15.169:0/2815569097 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:48.888 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.885+0000 7f332e7b1740 1 -- 172.21.15.169:0/2815569097 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33241b29c0 con 0x562ad984de40 2024-03-28T01:02:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.885+0000 7f3329161640 1 --2- 172.21.15.169:0/2815569097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad98539a0 0x562ad98703b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.885+0000 7f332e7b1740 1 -- 172.21.15.169:0/2815569097 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33241b04c0 con 0x562ad984e830 2024-03-28T01:02:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.885+0000 7f332e7b1740 1 -- 172.21.15.169:0/2815569097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3324118000 con 0x562ad98539a0 2024-03-28T01:02:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.886+0000 7f3329161640 1 -- 172.21.15.169:0/2815569097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad984e830 msgr2=0x562ad9850030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.886+0000 7f3329161640 1 --2- 172.21.15.169:0/2815569097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad984e830 0x562ad9850030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.889 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.886+0000 7f3329161640 1 -- 172.21.15.169:0/2815569097 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562ad984de40 msgr2=0x562ad984faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.886+0000 7f3329161640 1 --2- 172.21.15.169:0/2815569097 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562ad984de40 0x562ad984faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.886+0000 7f3329161640 1 -- 172.21.15.169:0/2815569097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ad984a120 con 0x562ad98539a0 2024-03-28T01:02:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.886+0000 7f3329161640 1 --2- 172.21.15.169:0/2815569097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad98539a0 0x562ad98703b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f331c013c10 tx=0x7f331c001160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cf5b5a53afba8b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.887+0000 7f330affd640 1 -- 172.21.15.169:0/2815569097 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331c0070d0 con 0x562ad98539a0 2024-03-28T01:02:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.887+0000 7f330affd640 1 -- 172.21.15.169:0/2815569097 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f331c027070 con 0x562ad98539a0 2024-03-28T01:02:48.890 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.887+0000 7f330affd640 1 -- 172.21.15.169:0/2815569097 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331c0049b0 con 0x562ad98539a0 2024-03-28T01:02:48.891 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.888+0000 7f332e7b1740 1 -- 172.21.15.169:0/2815569097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad98539a0 msgr2=0x562ad98703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:48.891 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.888+0000 7f332e7b1740 1 --2- 172.21.15.169:0/2815569097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad98539a0 0x562ad98703b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f331c013c10 tx=0x7f331c001160 comp rx=0 tx=0).stop 2024-03-28T01:02:48.898 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.896+0000 7f332e7b1740 1 -- 172.21.15.169:0/2815569097 shutdown_connections 2024-03-28T01:02:48.899 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.896+0000 7f332e7b1740 1 --2- 172.21.15.169:0/2815569097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad98539a0 0x562ad98703b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.899 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.896+0000 7f332e7b1740 1 --2- 172.21.15.169:0/2815569097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad984e830 0x562ad9850030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.899 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.896+0000 7f332e7b1740 1 --2- 172.21.15.169:0/2815569097 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562ad984de40 0x562ad984faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.899 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.896+0000 7f332e7b1740 1 -- 172.21.15.169:0/2815569097 >> 172.21.15.169:0/2815569097 conn(0x562ad98493b0 msgr2=0x562ad9849790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:48.899 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.897+0000 7f332e7b1740 1 -- 172.21.15.169:0/2815569097 shutdown_connections 2024-03-28T01:02:48.900 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.897+0000 7f332e7b1740 1 -- 172.21.15.169:0/2815569097 wait complete. 2024-03-28T01:02:48.900 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.897+0000 7f332e7b1740 1 -- start start 2024-03-28T01:02:48.900 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.898+0000 7f332e7b1740 1 Processor -- start 2024-03-28T01:02:48.902 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.898+0000 7f332e7b1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562ad984de40 0x562ad9870240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:48.902 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.898+0000 7f332e7b1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad984e830 0x562ad986cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:48.902 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.898+0000 7f332e7b1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad98539a0 0x562ad986d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:48.902 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.898+0000 7f332e7b1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33241aebb0 con 0x562ad984de40 2024-03-28T01:02:48.903 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.898+0000 7f332e7b1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33241afe60 con 0x562ad98539a0 2024-03-28T01:02:48.903 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.898+0000 7f332e7b1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33241af1a0 con 0x562ad984e830 2024-03-28T01:02:48.903 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.898+0000 7f332e7b1740 10 client.0 _ll_register_callbacks cb 0x7f331c001a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:02:48.903 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.899+0000 7f3329161640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad98539a0 0x562ad986d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:48.903 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.899+0000 7f3329962640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad984e830 0x562ad986cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:48.903 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.899+0000 7f3329962640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad984e830 0x562ad986cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48556/0 (socket says 172.21.15.169:48556) 2024-03-28T01:02:48.903 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.899+0000 7f3329962640 1 -- 172.21.15.169:0/2858801290 learned_addr learned my addr 172.21.15.169:0/2858801290 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:02:48.904 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.900+0000 7f3329161640 1 --2- 172.21.15.169:0/2858801290 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562ad984de40 0x562ad9870240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:48.904 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.900+0000 7f3329962640 1 -- 172.21.15.169:0/2858801290 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad98539a0 msgr2=0x562ad986d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:48.904 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.900+0000 7f3329962640 1 --2- 172.21.15.169:0/2858801290 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad98539a0 0x562ad986d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.904 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.900+0000 7f3329962640 1 -- 172.21.15.169:0/2858801290 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562ad984de40 msgr2=0x562ad9870240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:02:48.904 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.900+0000 7f3329962640 1 --2- 172.21.15.169:0/2858801290 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562ad984de40 0x562ad9870240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:48.904 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.900+0000 7f3329962640 1 -- 172.21.15.169:0/2858801290 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562ad98e0160 con 0x562ad984e830 2024-03-28T01:02:48.904 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.901+0000 7f3329962640 1 --2- 172.21.15.169:0/2858801290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad984e830 0x562ad986cce0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3318005740 tx=0x7f3318009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac08f818d18f3fe2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:02:48.904 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.901+0000 7f33097fa640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:02:48.905 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.901+0000 7f33097fa640 1 -- 172.21.15.169:0/2858801290 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3318056060 con 0x562ad984e830 2024-03-28T01:02:48.905 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.901+0000 7f33097fa640 1 -- 172.21.15.169:0/2858801290 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f3318004b50 con 0x562ad984e830 2024-03-28T01:02:48.905 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.902+0000 7f332e7b1740 1 -- 172.21.15.169:0/2858801290 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f331c027070 con 0x562ad984e830 2024-03-28T01:02:48.905 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.902+0000 7f33097fa640 1 -- 172.21.15.169:0/2858801290 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3318042040 con 0x562ad984e830 2024-03-28T01:02:48.905 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.902+0000 7f332e7b1740 10 client.26500 fetch_fsmap learned FSMap version 25 2024-03-28T01:02:48.905 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.902+0000 7f332e7b1740 1 -- 172.21.15.169:0/2858801290 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x562ad98e12e0 con 0x562ad984e830 2024-03-28T01:02:48.906 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.903+0000 7f33097fa640 1 -- 172.21.15.169:0/2858801290 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f331c027070 con 0x562ad984e830 2024-03-28T01:02:48.906 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.903+0000 7f33097fa640 1 -- 172.21.15.169:0/2858801290 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f3318051b00 con 0x562ad984e830 2024-03-28T01:02:48.906 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.903+0000 7f332e7b1740 10 client.26500 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:02:48.906 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.903+0000 7f332e7b1740 10 client.26500 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:02:48.906 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.903+0000 7f332e7b1740 20 client.26500 populate_metadata read hostname 'smithi169' 2024-03-28T01:02:48.906 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.903+0000 7f332e7b1740 1 --2- 172.21.15.169:0/2858801290 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562ad98e1700 0x562ad98e3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:02:48.906 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.903+0000 7f332e7b1740 4 client.26500 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:02:48.907 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.903+0000 7f332e7b1740 1 -- 172.21.15.169:0/2858801290 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562ad97ba3a0 con 0x562ad98e1700 2024-03-28T01:02:48.907 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.904+0000 7f3329161640 1 --2- 172.21.15.169:0/2858801290 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562ad98e1700 0x562ad98e3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:02:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.905+0000 7f3329161640 1 --2- 172.21.15.169:0/2858801290 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562ad98e1700 0x562ad98e3af0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be644eb85d49fb51 server_cookie=c03c2355577a0b3e in_seq=0 out_seq=0 2024-03-28T01:02:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:48.905+0000 7f33097fa640 10 client.26500 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:49.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:49 smithi174 ceph-mon[36060]: pgmap v540: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 744 MiB/s wr, 1.02k op/s 2024-03-28T01:02:49.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:49 smithi174 ceph-mon[36060]: scrub summary: idle 2024-03-28T01:02:49.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:49 smithi177 ceph-mon[36206]: pgmap v540: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 744 MiB/s wr, 1.02k op/s 2024-03-28T01:02:49.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:49 smithi177 ceph-mon[36206]: scrub summary: idle 2024-03-28T01:02:50.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:49 smithi169 ceph-mon[32274]: pgmap v540: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 744 MiB/s wr, 1.02k op/s 2024-03-28T01:02:50.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:49 smithi169 ceph-mon[32274]: scrub summary: idle 2024-03-28T01:02:50.907 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.903+0000 7f33097fa640 1 -- 172.21.15.169:0/2858801290 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562ad97ba3a0 con 0x562ad98e1700 2024-03-28T01:02:50.907 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.903+0000 7f33097fa640 10 client.26500 handle_command_reply: tid=0 2024-03-28T01:02:50.907 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.903+0000 7f332e7b1740 1 client.26500 shutdown 2024-03-28T01:02:50.907 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.903+0000 7f332e7b1740 1 -- 172.21.15.169:0/2858801290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad984e830 msgr2=0x562ad986cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:50.907 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.903+0000 7f332e7b1740 1 --2- 172.21.15.169:0/2858801290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad984e830 0x562ad986cce0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3318005740 tx=0x7f3318009680 comp rx=0 tx=0).stop 2024-03-28T01:02:50.907 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.904+0000 7f332e7b1740 1 -- 172.21.15.169:0/2858801290 shutdown_connections 2024-03-28T01:02:50.907 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.904+0000 7f332e7b1740 1 --2- 172.21.15.169:0/2858801290 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562ad98539a0 0x562ad986d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:50.907 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.904+0000 7f332e7b1740 1 --2- 172.21.15.169:0/2858801290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ad984e830 0x562ad986cce0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:50.907 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.904+0000 7f332e7b1740 1 --2- 172.21.15.169:0/2858801290 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562ad98e1700 0x562ad98e3af0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:50.908 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.904+0000 7f332e7b1740 1 --2- 172.21.15.169:0/2858801290 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562ad984de40 0x562ad9870240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:50.908 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.904+0000 7f332e7b1740 1 -- 172.21.15.169:0/2858801290 >> 172.21.15.169:0/2858801290 conn(0x562ad98493b0 msgr2=0x562ad984faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:50.908 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.904+0000 7f33097fa640 0 client.26500 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:02:50.908 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.904+0000 7f332e7b1740 1 -- 172.21.15.169:0/2858801290 shutdown_connections 2024-03-28T01:02:50.908 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.904+0000 7f332e7b1740 1 -- 172.21.15.169:0/2858801290 wait complete. 2024-03-28T01:02:50.908 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.905+0000 7f332e7b1740 20 client.26500 trim_cache size 0 max 16384 2024-03-28T01:02:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:02:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:02:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:02:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:02:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.908+0000 7f3320bff640 1 -- 172.21.15.169:0/775036140 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f330c0a28f0 msgr2=0x7f330c0a4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.908+0000 7f3320bff640 1 --2- 172.21.15.169:0/775036140 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f330c0a28f0 0x7f330c0a4db0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3314097210 tx=0x7f3314004a40 comp rx=0 tx=0).stop 2024-03-28T01:02:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.908+0000 7f3320bff640 1 -- 172.21.15.169:0/775036140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3324109390 msgr2=0x7f33241adad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:02:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.908+0000 7f3320bff640 1 --2- 172.21.15.169:0/775036140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3324109390 0x7f33241adad0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f331400b970 tx=0x7f331400be30 comp rx=0 tx=0).stop 2024-03-28T01:02:50.914 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.909+0000 7f3320bff640 1 -- 172.21.15.169:0/775036140 shutdown_connections 2024-03-28T01:02:50.914 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.909+0000 7f3320bff640 1 --2- 172.21.15.169:0/775036140 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f330c0a28f0 0x7f330c0a4db0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:50.914 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.909+0000 7f3320bff640 1 --2- 172.21.15.169:0/775036140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3324109390 0x7f33241adad0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:50.914 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.909+0000 7f3320bff640 1 --2- 172.21.15.169:0/775036140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33240732b0 0x7f33241a6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:50.914 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.909+0000 7f3320bff640 1 --2- 172.21.15.169:0/775036140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3324072950 0x7f33241a6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:02:50.914 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.909+0000 7f3320bff640 1 -- 172.21.15.169:0/775036140 >> 172.21.15.169:0/775036140 conn(0x7f332406e510 msgr2=0x7f3324109fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:02:50.915 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.909+0000 7f3320bff640 1 -- 172.21.15.169:0/775036140 shutdown_connections 2024-03-28T01:02:50.915 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:02:50.910+0000 7f3320bff640 1 -- 172.21.15.169:0/775036140 wait complete. 2024-03-28T01:02:51.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:51 smithi174 ceph-mon[36060]: pgmap v541: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 685 MiB/s wr, 811 op/s 2024-03-28T01:02:51.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:51 smithi177 ceph-mon[36206]: pgmap v541: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 685 MiB/s wr, 811 op/s 2024-03-28T01:02:52.053 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:51 smithi169 ceph-mon[32274]: pgmap v541: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 685 MiB/s wr, 811 op/s 2024-03-28T01:02:53.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:52 smithi169 ceph-mon[32274]: pgmap v542: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 872 MiB/s wr, 995 op/s 2024-03-28T01:02:53.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:52 smithi174 ceph-mon[36060]: pgmap v542: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 872 MiB/s wr, 995 op/s 2024-03-28T01:02:53.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:52 smithi177 ceph-mon[36206]: pgmap v542: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 872 MiB/s wr, 995 op/s 2024-03-28T01:02:55.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:55 smithi169 ceph-mon[32274]: pgmap v543: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 710 MiB/s wr, 929 op/s 2024-03-28T01:02:55.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:55 smithi174 ceph-mon[36060]: pgmap v543: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 710 MiB/s wr, 929 op/s 2024-03-28T01:02:55.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:55 smithi177 ceph-mon[36206]: pgmap v543: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 710 MiB/s wr, 929 op/s 2024-03-28T01:02:57.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:57 smithi169 ceph-mon[32274]: pgmap v544: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 675 MiB/s wr, 1.01k op/s 2024-03-28T01:02:57.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:57 smithi174 ceph-mon[36060]: pgmap v544: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 675 MiB/s wr, 1.01k op/s 2024-03-28T01:02:57.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:57 smithi177 ceph-mon[36206]: pgmap v544: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 675 MiB/s wr, 1.01k op/s 2024-03-28T01:02:59.794 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:02:59 smithi169 ceph-mon[32274]: pgmap v545: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 691 MiB/s wr, 1.30k op/s 2024-03-28T01:02:59.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:02:59 smithi174 ceph-mon[36060]: pgmap v545: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 691 MiB/s wr, 1.30k op/s 2024-03-28T01:02:59.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:02:59 smithi177 ceph-mon[36206]: pgmap v545: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 691 MiB/s wr, 1.30k op/s 2024-03-28T01:03:01.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:01 smithi169 ceph-mon[32274]: pgmap v546: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 642 MiB/s wr, 1.37k op/s 2024-03-28T01:03:01.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:01 smithi174 ceph-mon[36060]: pgmap v546: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 642 MiB/s wr, 1.37k op/s 2024-03-28T01:03:01.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:01 smithi177 ceph-mon[36206]: pgmap v546: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 642 MiB/s wr, 1.37k op/s 2024-03-28T01:03:03.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:03 smithi169 ceph-mon[32274]: pgmap v547: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s rd, 784 MiB/s wr, 1.62k op/s 2024-03-28T01:03:03.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:03 smithi174 ceph-mon[36060]: pgmap v547: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s rd, 784 MiB/s wr, 1.62k op/s 2024-03-28T01:03:03.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:03 smithi177 ceph-mon[36206]: pgmap v547: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s rd, 784 MiB/s wr, 1.62k op/s 2024-03-28T01:03:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.334+0000 7f77c7bc6640 1 -- 172.21.15.169:0/3714339493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77c01086a0 msgr2=0x7f77c010aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.334+0000 7f77c7bc6640 1 --2- 172.21.15.169:0/3714339493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77c01086a0 0x7f77c010aad0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f77b0009900 tx=0x7f77b002bc50 comp rx=0 tx=0).stop 2024-03-28T01:03:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.337+0000 7f77c7bc6640 1 -- 172.21.15.169:0/3714339493 shutdown_connections 2024-03-28T01:03:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.337+0000 7f77c7bc6640 1 --2- 172.21.15.169:0/3714339493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77c010d9c0 0x7f77c010de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.337+0000 7f77c7bc6640 1 --2- 172.21.15.169:0/3714339493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77c010b010 0x7f77c010d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.337+0000 7f77c7bc6640 1 --2- 172.21.15.169:0/3714339493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77c01086a0 0x7f77c010aad0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.337+0000 7f77c7bc6640 1 -- 172.21.15.169:0/3714339493 >> 172.21.15.169:0/3714339493 conn(0x7f77c006e6e0 msgr2=0x7f77c0070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.339+0000 7f77c7bc6640 1 -- 172.21.15.169:0/3714339493 shutdown_connections 2024-03-28T01:03:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.339+0000 7f77c7bc6640 1 -- 172.21.15.169:0/3714339493 wait complete. 2024-03-28T01:03:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.340+0000 7f77c7bc6640 1 Processor -- start 2024-03-28T01:03:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.340+0000 7f77c7bc6640 1 -- start start 2024-03-28T01:03:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.340+0000 7f77c7bc6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77c01086a0 0x7f77c0114d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.340+0000 7f77c7bc6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77c010b010 0x7f77c0115250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.340+0000 7f77c7bc6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77c010d9c0 0x7f77c01a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.340+0000 7f77c7bc6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c011c150 con 0x7f77c01086a0 2024-03-28T01:03:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.340+0000 7f77c7bc6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c011bfd0 con 0x7f77c010d9c0 2024-03-28T01:03:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.340+0000 7f77c7bc6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c011c2d0 con 0x7f77c010b010 2024-03-28T01:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.341+0000 7f77c5161640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77c010b010 0x7f77c0115250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.341+0000 7f77c5161640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77c010b010 0x7f77c0115250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:57598/0 (socket says 172.21.15.169:57598) 2024-03-28T01:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.341+0000 7f77c5161640 1 -- 172.21.15.169:0/178639286 learned_addr learned my addr 172.21.15.169:0/178639286 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.341+0000 7f77c6163640 1 --2- 172.21.15.169:0/178639286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77c010d9c0 0x7f77c01a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.342+0000 7f77c5161640 1 -- 172.21.15.169:0/178639286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77c010d9c0 msgr2=0x7f77c01a5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.342+0000 7f77c5161640 1 --2- 172.21.15.169:0/178639286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77c010d9c0 0x7f77c01a5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.342+0000 7f77c5161640 1 -- 172.21.15.169:0/178639286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77c01086a0 msgr2=0x7f77c0114d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.342+0000 7f77c5161640 1 --2- 172.21.15.169:0/178639286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77c01086a0 0x7f77c0114d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.342+0000 7f77c5161640 1 -- 172.21.15.169:0/178639286 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c01158f0 con 0x7f77c010b010 2024-03-28T01:03:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.343+0000 7f77c5161640 1 --2- 172.21.15.169:0/178639286 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77c010b010 0x7f77c0115250 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f77bc00c890 tx=0x7f77bc00cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:04.348 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.344+0000 7f77b6ffd640 1 -- 172.21.15.169:0/178639286 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77bc004220 con 0x7f77c010b010 2024-03-28T01:03:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.344+0000 7f77c7bc6640 1 -- 172.21.15.169:0/178639286 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c01a5e50 con 0x7f77c010b010 2024-03-28T01:03:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.344+0000 7f77c7bc6640 1 -- 172.21.15.169:0/178639286 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77c01a63f0 con 0x7f77c010b010 2024-03-28T01:03:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.345+0000 7f77b6ffd640 1 -- 172.21.15.169:0/178639286 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77bc0043c0 con 0x7f77c010b010 2024-03-28T01:03:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.345+0000 7f77b6ffd640 1 -- 172.21.15.169:0/178639286 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77bc0049e0 con 0x7f77c010b010 2024-03-28T01:03:04.353 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.348+0000 7f77ca7b2740 1 Processor -- start 2024-03-28T01:03:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.351+0000 7f77ca7b2740 1 -- start start 2024-03-28T01:03:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.352+0000 7f77ca7b2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a0920 0x55cc785a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.352+0000 7f77ca7b2740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a1310 0x55cc785a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.352+0000 7f77ca7b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc785a6480 0x55cc785a8910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.352+0000 7f77ca7b2740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc78505b10 con 0x55cc785a1310 2024-03-28T01:03:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.352+0000 7f77ca7b2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc784ae900 con 0x55cc785a0920 2024-03-28T01:03:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.352+0000 7f77ca7b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc78501bb0 con 0x55cc785a6480 2024-03-28T01:03:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.353+0000 7f77c6163640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a0920 0x55cc785a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.353+0000 7f77c6163640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a0920 0x55cc785a0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:46408/0 (socket says 172.21.15.169:46408) 2024-03-28T01:03:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.353+0000 7f77c6163640 1 -- 172.21.15.169:0/4122516526 learned_addr learned my addr 172.21.15.169:0/4122516526 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.355+0000 7f77c5161640 1 --2- 172.21.15.169:0/4122516526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc785a6480 0x55cc785a8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.355+0000 7f77b6ffd640 1 -- 172.21.15.169:0/178639286 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f77bc004c40 con 0x7f77c010b010 2024-03-28T01:03:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.357+0000 7f77b6ffd640 1 --2- 172.21.15.169:0/178639286 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f77ac0a29f0 0x7f77ac0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.357+0000 7f77b6ffd640 1 -- 172.21.15.169:0/178639286 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f77bc0cee70 con 0x7f77c010b010 2024-03-28T01:03:04.362 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.358+0000 7f77c6163640 1 --2- 172.21.15.169:0/178639286 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f77ac0a29f0 0x7f77ac0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:04.362 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.358+0000 7f77c6163640 1 -- 172.21.15.169:0/4122516526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc785a6480 msgr2=0x55cc785a8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:04.362 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.358+0000 7f77c6163640 1 --2- 172.21.15.169:0/4122516526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc785a6480 0x55cc785a8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.362 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.358+0000 7f77c6163640 1 -- 172.21.15.169:0/4122516526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a1310 msgr2=0x55cc785a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:04.362 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.358+0000 7f77c6163640 1 --2- 172.21.15.169:0/4122516526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a1310 0x55cc785a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.362 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.358+0000 7f77c6163640 1 -- 172.21.15.169:0/4122516526 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc7821a210 con 0x55cc785a0920 2024-03-28T01:03:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.361+0000 7f77c6163640 1 --2- 172.21.15.169:0/4122516526 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a0920 0x55cc785a0d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f77b80139c0 tx=0x7f77b8013e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50b40307f19751b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:04.364 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.362+0000 7f77c6163640 1 --2- 172.21.15.169:0/178639286 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f77ac0a29f0 0x7f77ac0a4eb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f77b8009000 tx=0x7f77b801d000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.364+0000 7f77abfff640 1 -- 172.21.15.169:0/4122516526 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b8013130 con 0x55cc785a0920 2024-03-28T01:03:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.364+0000 7f77abfff640 1 -- 172.21.15.169:0/4122516526 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77b8003e50 con 0x55cc785a0920 2024-03-28T01:03:04.370 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.365+0000 7f77ca7b2740 1 -- 172.21.15.169:0/4122516526 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a0920 msgr2=0x55cc785a0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:04.370 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.365+0000 7f77ca7b2740 1 --2- 172.21.15.169:0/4122516526 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a0920 0x55cc785a0d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f77b80139c0 tx=0x7f77b8013e80 comp rx=0 tx=0).stop 2024-03-28T01:03:04.371 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.368+0000 7f77ca7b2740 1 -- 172.21.15.169:0/4122516526 shutdown_connections 2024-03-28T01:03:04.371 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.368+0000 7f77ca7b2740 1 --2- 172.21.15.169:0/4122516526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc785a6480 0x55cc785a8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.371 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.368+0000 7f77ca7b2740 1 --2- 172.21.15.169:0/4122516526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a1310 0x55cc785a5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.372 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.368+0000 7f77ca7b2740 1 --2- 172.21.15.169:0/4122516526 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a0920 0x55cc785a0d40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.372 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.368+0000 7f77ca7b2740 1 -- 172.21.15.169:0/4122516526 >> 172.21.15.169:0/4122516526 conn(0x55cc7859be90 msgr2=0x55cc7859c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:04.373 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.370+0000 7f77ca7b2740 1 -- 172.21.15.169:0/4122516526 shutdown_connections 2024-03-28T01:03:04.373 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.370+0000 7f77ca7b2740 1 -- 172.21.15.169:0/4122516526 wait complete. 2024-03-28T01:03:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.371+0000 7f77ca7b2740 1 -- start start 2024-03-28T01:03:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.371+0000 7f77ca7b2740 1 Processor -- start 2024-03-28T01:03:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.371+0000 7f77ca7b2740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a0920 0x55cc785aac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.371+0000 7f77ca7b2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a1310 0x55cc785ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.371+0000 7f77ca7b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc785a6480 0x55cc785b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.371+0000 7f77ca7b2740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc785a9420 con 0x55cc785a0920 2024-03-28T01:03:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.371+0000 7f77ca7b2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc784c1d50 con 0x55cc785a1310 2024-03-28T01:03:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.371+0000 7f77ca7b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc785a95a0 con 0x55cc785a6480 2024-03-28T01:03:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.371+0000 7f77ca7b2740 10 client.0 _ll_register_callbacks cb 0x55cc784c1720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.373+0000 7f77c5161640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a1310 0x55cc785ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.373+0000 7f77c6163640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc785a6480 0x55cc785b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.373+0000 7f77c5962640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a0920 0x55cc785aac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.374+0000 7f77c5962640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a0920 0x55cc785aac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:52116/0 (socket says 172.21.15.169:52116) 2024-03-28T01:03:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.374+0000 7f77c5161640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a1310 0x55cc785ab1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:46422/0 (socket says 172.21.15.169:46422) 2024-03-28T01:03:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.374+0000 7f77c5161640 1 -- 172.21.15.169:0/2761783880 learned_addr learned my addr 172.21.15.169:0/2761783880 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.374+0000 7f77c5161640 1 -- 172.21.15.169:0/2761783880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc785a6480 msgr2=0x55cc785b21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.374+0000 7f77c5161640 1 --2- 172.21.15.169:0/2761783880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc785a6480 0x55cc785b21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.374+0000 7f77c5161640 1 -- 172.21.15.169:0/2761783880 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a0920 msgr2=0x55cc785aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.374+0000 7f77c5161640 1 --2- 172.21.15.169:0/2761783880 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a0920 0x55cc785aac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.375+0000 7f77c5161640 1 -- 172.21.15.169:0/2761783880 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cc782e6620 con 0x55cc785a1310 2024-03-28T01:03:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.375+0000 7f77c5161640 1 --2- 172.21.15.169:0/2761783880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a1310 0x55cc785ab1a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f77bc09b3a0 tx=0x7f77bc099550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4d53ee124d4bad7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.376+0000 7f77a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:03:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.376+0000 7f77a97fa640 1 -- 172.21.15.169:0/2761783880 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77bc01d040 con 0x55cc785a1310 2024-03-28T01:03:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.376+0000 7f77ca7b2740 1 -- 172.21.15.169:0/2761783880 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f77b8003e50 con 0x55cc785a1310 2024-03-28T01:03:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.376+0000 7f77a97fa640 1 -- 172.21.15.169:0/2761783880 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f77bc09f130 con 0x55cc785a1310 2024-03-28T01:03:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.377+0000 7f77a97fa640 1 -- 172.21.15.169:0/2761783880 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77bc09f330 con 0x55cc785a1310 2024-03-28T01:03:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.381+0000 7f77ca7b2740 10 client.26582 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:04.384 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.381+0000 7f77ca7b2740 1 -- 172.21.15.169:0/2761783880 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55cc785b5610 con 0x55cc785a1310 2024-03-28T01:03:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.387+0000 7f77a97fa640 1 -- 172.21.15.169:0/2761783880 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77b8003e50 con 0x55cc785a1310 2024-03-28T01:03:04.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.389+0000 7f77a97fa640 1 -- 172.21.15.169:0/2761783880 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f77bc09a860 con 0x55cc785a1310 2024-03-28T01:03:04.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.389+0000 7f77ca7b2740 10 client.26582 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:04.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.389+0000 7f77ca7b2740 10 client.26582 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:04.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.389+0000 7f77ca7b2740 20 client.26582 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:04.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.389+0000 7f77ca7b2740 1 --2- 172.21.15.169:0/2761783880 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55cc785b70e0 0x55cc785b9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:04.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.389+0000 7f77ca7b2740 4 client.26582 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:03:04.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.389+0000 7f77ca7b2740 1 -- 172.21.15.169:0/2761783880 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cc78513050 con 0x55cc785b70e0 2024-03-28T01:03:04.392 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.390+0000 7f77c5962640 1 --2- 172.21.15.169:0/2761783880 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55cc785b70e0 0x55cc785b9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:04.394 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.391+0000 7f77c5962640 1 --2- 172.21.15.169:0/2761783880 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55cc785b70e0 0x55cc785b9560 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39a74036403531b1 server_cookie=b151635a6760cabc in_seq=0 out_seq=0 2024-03-28T01:03:04.395 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.392+0000 7f77a97fa640 10 client.26582 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:04.399 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.393+0000 7f77a97fa640 1 -- 172.21.15.169:0/2761783880 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cc78513050 con 0x55cc785b70e0 2024-03-28T01:03:04.399 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.393+0000 7f77a97fa640 10 client.26582 handle_command_reply: tid=0 2024-03-28T01:03:04.399 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.394+0000 7f77ca7b2740 1 client.26582 shutdown 2024-03-28T01:03:04.399 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.394+0000 7f77ca7b2740 1 -- 172.21.15.169:0/2761783880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a1310 msgr2=0x55cc785ab1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:04.399 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.394+0000 7f77ca7b2740 1 --2- 172.21.15.169:0/2761783880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a1310 0x55cc785ab1a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f77bc09b3a0 tx=0x7f77bc099550 comp rx=0 tx=0).stop 2024-03-28T01:03:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.394+0000 7f77ca7b2740 1 -- 172.21.15.169:0/2761783880 shutdown_connections 2024-03-28T01:03:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.394+0000 7f77ca7b2740 1 --2- 172.21.15.169:0/2761783880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc785a6480 0x55cc785b21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.394+0000 7f77ca7b2740 1 --2- 172.21.15.169:0/2761783880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a1310 0x55cc785ab1a0 secure :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f77bc09b3a0 tx=0x7f77bc099550 comp rx=0 tx=0).stop 2024-03-28T01:03:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.394+0000 7f77ca7b2740 1 --2- 172.21.15.169:0/2761783880 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55cc785b70e0 0x55cc785b9560 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.394+0000 7f77ca7b2740 1 --2- 172.21.15.169:0/2761783880 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a0920 0x55cc785aac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.394+0000 7f77ca7b2740 1 -- 172.21.15.169:0/2761783880 >> 172.21.15.169:0/2761783880 conn(0x55cc7859be90 msgr2=0x55cc7859b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.394+0000 7f77a97fa640 0 client.26582 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.395+0000 7f77ca7b2740 1 -- 172.21.15.169:0/2761783880 shutdown_connections 2024-03-28T01:03:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.395+0000 7f77ca7b2740 1 -- 172.21.15.169:0/2761783880 wait complete. 2024-03-28T01:03:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.395+0000 7f77ca7b2740 20 client.26582 trim_cache size 0 max 16384 2024-03-28T01:03:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.428+0000 7f77ca7b2740 1 Processor -- start 2024-03-28T01:03:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.428+0000 7f77ca7b2740 1 -- start start 2024-03-28T01:03:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.429+0000 7f77ca7b2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a0920 0x55cc785a8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.429+0000 7f77ca7b2740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a6480 0x55cc785b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.429+0000 7f77ca7b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc785a5bc0 0x55cc7859c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.429+0000 7f77ca7b2740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a4011e70 con 0x55cc785a6480 2024-03-28T01:03:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.429+0000 7f77ca7b2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a400f970 con 0x55cc785a0920 2024-03-28T01:03:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.429+0000 7f77ca7b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a400dac0 con 0x55cc785a5bc0 2024-03-28T01:03:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.429+0000 7f77c5161640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a6480 0x55cc785b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.429+0000 7f77c5161640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a6480 0x55cc785b21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:52130/0 (socket says 172.21.15.169:52130) 2024-03-28T01:03:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.429+0000 7f77c5161640 1 -- 172.21.15.169:0/2081423493 learned_addr learned my addr 172.21.15.169:0/2081423493 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.429+0000 7f77c5161640 1 -- 172.21.15.169:0/2081423493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc785a5bc0 msgr2=0x55cc7859c270 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:03:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.429+0000 7f77c5161640 1 --2- 172.21.15.169:0/2081423493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc785a5bc0 0x55cc7859c270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.429+0000 7f77c5161640 1 -- 172.21.15.169:0/2081423493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a0920 msgr2=0x55cc785a8870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:03:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.429+0000 7f77c5161640 1 --2- 172.21.15.169:0/2081423493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a0920 0x55cc785a8870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.429+0000 7f77c5161640 1 -- 172.21.15.169:0/2081423493 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc785a5840 con 0x55cc785a6480 2024-03-28T01:03:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.430+0000 7f77c5161640 1 --2- 172.21.15.169:0/2081423493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a6480 0x55cc785b21a0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f77bc09abd0 tx=0x7f77bc099680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3049a973a703965 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.430+0000 7f77ab7fe640 1 -- 172.21.15.169:0/2081423493 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77bc01d070 con 0x55cc785a6480 2024-03-28T01:03:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.431+0000 7f77ab7fe640 1 -- 172.21.15.169:0/2081423493 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77bc0a3bb0 con 0x55cc785a6480 2024-03-28T01:03:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.431+0000 7f77ab7fe640 1 -- 172.21.15.169:0/2081423493 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77bc09a860 con 0x55cc785a6480 2024-03-28T01:03:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.433+0000 7f77ca7b2740 1 -- 172.21.15.169:0/2081423493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a6480 msgr2=0x55cc785b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.433+0000 7f77ca7b2740 1 --2- 172.21.15.169:0/2081423493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a6480 0x55cc785b21a0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f77bc09abd0 tx=0x7f77bc099680 comp rx=0 tx=0).stop 2024-03-28T01:03:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.433+0000 7f77ca7b2740 1 -- 172.21.15.169:0/2081423493 shutdown_connections 2024-03-28T01:03:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.433+0000 7f77ca7b2740 1 --2- 172.21.15.169:0/2081423493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc785a5bc0 0x55cc7859c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.433+0000 7f77ca7b2740 1 --2- 172.21.15.169:0/2081423493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a6480 0x55cc785b21a0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.433+0000 7f77ca7b2740 1 --2- 172.21.15.169:0/2081423493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a0920 0x55cc785a8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.433+0000 7f77ca7b2740 1 -- 172.21.15.169:0/2081423493 >> 172.21.15.169:0/2081423493 conn(0x55cc7859be90 msgr2=0x55cc78596eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.434+0000 7f77ca7b2740 1 -- 172.21.15.169:0/2081423493 shutdown_connections 2024-03-28T01:03:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.435+0000 7f77ca7b2740 1 -- 172.21.15.169:0/2081423493 wait complete. 2024-03-28T01:03:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.435+0000 7f77ca7b2740 1 -- start start 2024-03-28T01:03:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.436+0000 7f77ca7b2740 1 Processor -- start 2024-03-28T01:03:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.436+0000 7f77ca7b2740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a0920 0x55cc785af1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.437+0000 7f77ca7b2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a5bc0 0x55cc785af720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.437+0000 7f77ca7b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc785a6480 0x55cc785b0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.437+0000 7f77ca7b2740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a400e030 con 0x55cc785a0920 2024-03-28T01:03:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.437+0000 7f77ca7b2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a400f310 con 0x55cc785a5bc0 2024-03-28T01:03:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.437+0000 7f77ca7b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a400e650 con 0x55cc785a6480 2024-03-28T01:03:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.437+0000 7f77ca7b2740 10 client.0 _ll_register_callbacks cb 0x7f77bc09af50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.439+0000 7f77c5962640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a0920 0x55cc785af1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.439+0000 7f77c5962640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a0920 0x55cc785af1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:52136/0 (socket says 172.21.15.169:52136) 2024-03-28T01:03:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.439+0000 7f77c5962640 1 -- 172.21.15.169:0/1591109180 learned_addr learned my addr 172.21.15.169:0/1591109180 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.440+0000 7f77c6163640 1 --2- 172.21.15.169:0/1591109180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc785a6480 0x55cc785b0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.440+0000 7f77c5161640 1 --2- 172.21.15.169:0/1591109180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a5bc0 0x55cc785af720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.441+0000 7f77c5962640 1 -- 172.21.15.169:0/1591109180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc785a6480 msgr2=0x55cc785b0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.441+0000 7f77c5962640 1 --2- 172.21.15.169:0/1591109180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc785a6480 0x55cc785b0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.441+0000 7f77c5962640 1 -- 172.21.15.169:0/1591109180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a5bc0 msgr2=0x55cc785af720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.441+0000 7f77c5962640 1 --2- 172.21.15.169:0/1591109180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a5bc0 0x55cc785af720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.441+0000 7f77c5962640 1 -- 172.21.15.169:0/1591109180 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cc785b0d20 con 0x55cc785a0920 2024-03-28T01:03:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.441+0000 7f77c5962640 1 --2- 172.21.15.169:0/1591109180 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a0920 0x55cc785af1e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f77b002bc90 tx=0x7f77b002ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a408c9e9e9e5f8b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.441+0000 7f77a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:03:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.442+0000 7f77a97fa640 1 -- 172.21.15.169:0/1591109180 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b0047020 con 0x55cc785a0920 2024-03-28T01:03:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.442+0000 7f77ca7b2740 1 -- 172.21.15.169:0/1591109180 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f77bc0a3bb0 con 0x55cc785a0920 2024-03-28T01:03:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.442+0000 7f77a97fa640 1 -- 172.21.15.169:0/1591109180 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f77b0004040 con 0x55cc785a0920 2024-03-28T01:03:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.444+0000 7f77a97fa640 1 -- 172.21.15.169:0/1591109180 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77b0052050 con 0x55cc785a0920 2024-03-28T01:03:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.444+0000 7f77ca7b2740 10 client.16674 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.444+0000 7f77ca7b2740 1 -- 172.21.15.169:0/1591109180 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55cc785b20a0 con 0x55cc785a0920 2024-03-28T01:03:04.448 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.445+0000 7f77a97fa640 1 -- 172.21.15.169:0/1591109180 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77bc0a3bb0 con 0x55cc785a0920 2024-03-28T01:03:04.448 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.445+0000 7f77a97fa640 1 -- 172.21.15.169:0/1591109180 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f77b0005200 con 0x55cc785a0920 2024-03-28T01:03:04.448 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.445+0000 7f77ca7b2740 10 client.16674 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.445+0000 7f77ca7b2740 10 client.16674 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.445+0000 7f77ca7b2740 20 client.16674 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.445+0000 7f77ca7b2740 1 --2- 172.21.15.169:0/1591109180 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55cc78612da0 0x55cc786151c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.445+0000 7f77ca7b2740 4 client.16674 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:03:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.445+0000 7f77ca7b2740 1 -- 172.21.15.169:0/1591109180 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55cc7850d3a0 con 0x55cc78612da0 2024-03-28T01:03:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.450+0000 7f77c5161640 1 --2- 172.21.15.169:0/1591109180 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55cc78612da0 0x55cc786151c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.455+0000 7f77c5161640 1 --2- 172.21.15.169:0/1591109180 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55cc78612da0 0x55cc786151c0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b591d01c19c2f59 server_cookie=fb49b568db26ff3f in_seq=0 out_seq=0 2024-03-28T01:03:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.455+0000 7f77a97fa640 10 client.16674 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.561+0000 7f77a97fa640 1 -- 172.21.15.169:0/1591109180 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55cc7850d3a0 con 0x55cc78612da0 2024-03-28T01:03:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.561+0000 7f77a97fa640 10 client.16674 handle_command_reply: tid=0 2024-03-28T01:03:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.562+0000 7f77ca7b2740 1 client.16674 shutdown 2024-03-28T01:03:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.563+0000 7f77ca7b2740 1 -- 172.21.15.169:0/1591109180 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a0920 msgr2=0x55cc785af1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.563+0000 7f77ca7b2740 1 --2- 172.21.15.169:0/1591109180 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a0920 0x55cc785af1e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f77b002bc90 tx=0x7f77b002ca30 comp rx=0 tx=0).stop 2024-03-28T01:03:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.563+0000 7f77ca7b2740 1 -- 172.21.15.169:0/1591109180 shutdown_connections 2024-03-28T01:03:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.563+0000 7f77ca7b2740 1 --2- 172.21.15.169:0/1591109180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cc785a6480 0x55cc785b0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.563+0000 7f77ca7b2740 1 --2- 172.21.15.169:0/1591109180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc785a5bc0 0x55cc785af720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.563+0000 7f77ca7b2740 1 --2- 172.21.15.169:0/1591109180 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55cc78612da0 0x55cc786151c0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.564+0000 7f77ca7b2740 1 --2- 172.21.15.169:0/1591109180 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55cc785a0920 0x55cc785af1e0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.564+0000 7f77ca7b2740 1 -- 172.21.15.169:0/1591109180 >> 172.21.15.169:0/1591109180 conn(0x55cc7859be90 msgr2=0x55cc7859c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.564+0000 7f77a97fa640 0 client.16674 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.564+0000 7f77ca7b2740 1 -- 172.21.15.169:0/1591109180 shutdown_connections 2024-03-28T01:03:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.564+0000 7f77ca7b2740 1 -- 172.21.15.169:0/1591109180 wait complete. 2024-03-28T01:03:04.568 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.564+0000 7f77ca7b2740 20 client.16674 trim_cache size 0 max 16384 2024-03-28T01:03:04.568 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:03:04.568 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:03:04.568 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:03:04.568 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:03:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.565+0000 7f77c7bc6640 1 -- 172.21.15.169:0/178639286 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f77ac0a29f0 msgr2=0x7f77ac0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.565+0000 7f77c7bc6640 1 --2- 172.21.15.169:0/178639286 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f77ac0a29f0 0x7f77ac0a4eb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f77b8009000 tx=0x7f77b801d000 comp rx=0 tx=0).stop 2024-03-28T01:03:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.565+0000 7f77c7bc6640 1 -- 172.21.15.169:0/178639286 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77c010b010 msgr2=0x7f77c0115250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.565+0000 7f77c7bc6640 1 --2- 172.21.15.169:0/178639286 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77c010b010 0x7f77c0115250 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f77bc00c890 tx=0x7f77bc00cd50 comp rx=0 tx=0).stop 2024-03-28T01:03:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.565+0000 7f77c7bc6640 1 -- 172.21.15.169:0/178639286 shutdown_connections 2024-03-28T01:03:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.565+0000 7f77c7bc6640 1 --2- 172.21.15.169:0/178639286 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f77ac0a29f0 0x7f77ac0a4eb0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.565+0000 7f77c7bc6640 1 --2- 172.21.15.169:0/178639286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77c010d9c0 0x7f77c01a5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.565+0000 7f77c7bc6640 1 --2- 172.21.15.169:0/178639286 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77c010b010 0x7f77c0115250 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.565+0000 7f77c7bc6640 1 --2- 172.21.15.169:0/178639286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77c01086a0 0x7f77c0114d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.566+0000 7f77c7bc6640 1 -- 172.21.15.169:0/178639286 >> 172.21.15.169:0/178639286 conn(0x7f77c006e6e0 msgr2=0x7f77c010d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.566+0000 7f77c7bc6640 1 -- 172.21.15.169:0/178639286 shutdown_connections 2024-03-28T01:03:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:04.566+0000 7f77c7bc6640 1 -- 172.21.15.169:0/178639286 wait complete. 2024-03-28T01:03:05.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:05 smithi169 ceph-mon[32274]: pgmap v548: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s rd, 672 MiB/s wr, 1.70k op/s 2024-03-28T01:03:05.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:05 smithi174 ceph-mon[36060]: pgmap v548: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s rd, 672 MiB/s wr, 1.70k op/s 2024-03-28T01:03:05.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:05 smithi177 ceph-mon[36206]: pgmap v548: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s rd, 672 MiB/s wr, 1.70k op/s 2024-03-28T01:03:07.808 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:07 smithi169 ceph-mon[32274]: pgmap v549: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s rd, 633 MiB/s wr, 1.91k op/s 2024-03-28T01:03:07.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:07 smithi174 ceph-mon[36060]: pgmap v549: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s rd, 633 MiB/s wr, 1.91k op/s 2024-03-28T01:03:07.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:07 smithi177 ceph-mon[36206]: pgmap v549: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s rd, 633 MiB/s wr, 1.91k op/s 2024-03-28T01:03:09.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:09 smithi169 ceph-mon[32274]: pgmap v550: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s rd, 686 MiB/s wr, 2.72k op/s 2024-03-28T01:03:09.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:09 smithi174 ceph-mon[36060]: pgmap v550: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s rd, 686 MiB/s wr, 2.72k op/s 2024-03-28T01:03:09.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:09 smithi177 ceph-mon[36206]: pgmap v550: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s rd, 686 MiB/s wr, 2.72k op/s 2024-03-28T01:03:11.383 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:03:11.383 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:03:11.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:11 smithi169 ceph-mon[32274]: pgmap v551: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s rd, 576 MiB/s wr, 2.66k op/s 2024-03-28T01:03:11.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:11 smithi174 ceph-mon[36060]: pgmap v551: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s rd, 576 MiB/s wr, 2.66k op/s 2024-03-28T01:03:11.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:11 smithi177 ceph-mon[36206]: pgmap v551: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s rd, 576 MiB/s wr, 2.66k op/s 2024-03-28T01:03:13.804 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:13 smithi169 ceph-mon[32274]: pgmap v552: 129 pgs: 129 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s rd, 620 MiB/s wr, 3.57k op/s 2024-03-28T01:03:13.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:13 smithi174 ceph-mon[36060]: pgmap v552: 129 pgs: 129 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s rd, 620 MiB/s wr, 3.57k op/s 2024-03-28T01:03:13.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:13 smithi177 ceph-mon[36206]: pgmap v552: 129 pgs: 129 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s rd, 620 MiB/s wr, 3.57k op/s 2024-03-28T01:03:15.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.113+0000 7f98d0c68640 1 -- 172.21.15.169:0/2335849296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98cc10b010 msgr2=0x7f98cc10d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:15.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.113+0000 7f98d0c68640 1 --2- 172.21.15.169:0/2335849296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98cc10b010 0x7f98cc10d480 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f98b4009900 tx=0x7f98b402bc80 comp rx=0 tx=0).stop 2024-03-28T01:03:15.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.114+0000 7f98d0c68640 1 -- 172.21.15.169:0/2335849296 shutdown_connections 2024-03-28T01:03:15.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.115+0000 7f98d0c68640 1 --2- 172.21.15.169:0/2335849296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98cc10d9c0 0x7f98cc10de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.115+0000 7f98d0c68640 1 --2- 172.21.15.169:0/2335849296 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98cc10b010 0x7f98cc10d480 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.115+0000 7f98d0c68640 1 --2- 172.21.15.169:0/2335849296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98cc1086a0 0x7f98cc10aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.115+0000 7f98d0c68640 1 -- 172.21.15.169:0/2335849296 >> 172.21.15.169:0/2335849296 conn(0x7f98cc06e500 msgr2=0x7f98cc070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:15.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.121+0000 7f98d0c68640 1 -- 172.21.15.169:0/2335849296 shutdown_connections 2024-03-28T01:03:15.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.124+0000 7f98d0c68640 1 -- 172.21.15.169:0/2335849296 wait complete. 2024-03-28T01:03:15.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.125+0000 7f98d0c68640 1 Processor -- start 2024-03-28T01:03:15.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.125+0000 7f98d0c68640 1 -- start start 2024-03-28T01:03:15.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.125+0000 7f98d0c68640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98cc1086a0 0x7f98cc1a6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.125+0000 7f98d0c68640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98cc10b010 0x7f98cc1a6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.125+0000 7f98d0c68640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98cc10d9c0 0x7f98cc1adf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.125+0000 7f98d0c68640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc11c280 con 0x7f98cc1086a0 2024-03-28T01:03:15.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.125+0000 7f98d0c68640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc11c100 con 0x7f98cc10b010 2024-03-28T01:03:15.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.125+0000 7f98d0c68640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc11c400 con 0x7f98cc10d9c0 2024-03-28T01:03:15.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.127+0000 7f98c9d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98cc10b010 0x7f98cc1a6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:15.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.127+0000 7f98c9d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98cc10b010 0x7f98cc1a6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:43174/0 (socket says 172.21.15.169:43174) 2024-03-28T01:03:15.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.127+0000 7f98c9d9b640 1 -- 172.21.15.169:0/3942474337 learned_addr learned my addr 172.21.15.169:0/3942474337 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:15.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.128+0000 7f98cad9d640 1 --2- 172.21.15.169:0/3942474337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98cc10d9c0 0x7f98cc1adf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:15.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.133+0000 7f98cad9d640 1 -- 172.21.15.169:0/3942474337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98cc10b010 msgr2=0x7f98cc1a6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:15.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.133+0000 7f98cad9d640 1 --2- 172.21.15.169:0/3942474337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98cc10b010 0x7f98cc1a6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.133+0000 7f98cad9d640 1 -- 172.21.15.169:0/3942474337 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98cc1086a0 msgr2=0x7f98cc1a6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:15.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.133+0000 7f98cad9d640 1 --2- 172.21.15.169:0/3942474337 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98cc1086a0 0x7f98cc1a6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.133+0000 7f98cad9d640 1 -- 172.21.15.169:0/3942474337 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc10ff00 con 0x7f98cc10d9c0 2024-03-28T01:03:15.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.136+0000 7f98cad9d640 1 --2- 172.21.15.169:0/3942474337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98cc10d9c0 0x7f98cc1adf00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00bdf0 tx=0x7f98bc009ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:15.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.137+0000 7f98c37fe640 1 -- 172.21.15.169:0/3942474337 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc019070 con 0x7f98cc10d9c0 2024-03-28T01:03:15.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.137+0000 7f98c37fe640 1 -- 172.21.15.169:0/3942474337 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f98bc014920 con 0x7f98cc10d9c0 2024-03-28T01:03:15.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.137+0000 7f98c37fe640 1 -- 172.21.15.169:0/3942474337 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc0128e0 con 0x7f98cc10d9c0 2024-03-28T01:03:15.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.139+0000 7f98d0c68640 1 -- 172.21.15.169:0/3942474337 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98cc1101f0 con 0x7f98cc10d9c0 2024-03-28T01:03:15.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.139+0000 7f98d0c68640 1 -- 172.21.15.169:0/3942474337 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98cc1106c0 con 0x7f98cc10d9c0 2024-03-28T01:03:15.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.152+0000 7f98d384f740 1 Processor -- start 2024-03-28T01:03:15.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.152+0000 7f98d384f740 1 -- start start 2024-03-28T01:03:15.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.152+0000 7f98d384f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a52da0 0x5590c1a531c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.152+0000 7f98d384f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a53790 0x5590c1a58290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.152+0000 7f98d384f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590c1a58900 0x5590c1a5ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.152+0000 7f98d384f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590c19b83d0 con 0x5590c1a53790 2024-03-28T01:03:15.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.152+0000 7f98d384f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590c1960ad0 con 0x5590c1a52da0 2024-03-28T01:03:15.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.152+0000 7f98d384f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590c19b4470 con 0x5590c1a58900 2024-03-28T01:03:15.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.154+0000 7f98c9d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a52da0 0x5590c1a531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:15.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.154+0000 7f98c9d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a52da0 0x5590c1a531c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:43188/0 (socket says 172.21.15.169:43188) 2024-03-28T01:03:15.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.154+0000 7f98c9d9b640 1 -- 172.21.15.169:0/2908514605 learned_addr learned my addr 172.21.15.169:0/2908514605 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:15.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.154+0000 7f98c9d9b640 1 --2- 172.21.15.169:0/2908514605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590c1a58900 0x5590c1a5ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:15.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.154+0000 7f98ca59c640 1 --2- 172.21.15.169:0/2908514605 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a53790 0x5590c1a58290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:15.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.154+0000 7f98ca59c640 1 -- 172.21.15.169:0/2908514605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590c1a58900 msgr2=0x5590c1a5ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:15.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.154+0000 7f98ca59c640 1 --2- 172.21.15.169:0/2908514605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590c1a58900 0x5590c1a5ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.154+0000 7f98ca59c640 1 -- 172.21.15.169:0/2908514605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a52da0 msgr2=0x5590c1a531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:15.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.154+0000 7f98ca59c640 1 --2- 172.21.15.169:0/2908514605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a52da0 0x5590c1a531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.154+0000 7f98ca59c640 1 -- 172.21.15.169:0/2908514605 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5590c17967d0 con 0x5590c1a53790 2024-03-28T01:03:15.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.154+0000 7f98c37fe640 1 -- 172.21.15.169:0/3942474337 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f98bc014ad0 con 0x7f98cc10d9c0 2024-03-28T01:03:15.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.155+0000 7f98ca59c640 1 --2- 172.21.15.169:0/2908514605 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a53790 0x5590c1a58290 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f98c4009720 tx=0x7f98c4009be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75eca607af5fd785 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:15.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.155+0000 7f98c37fe640 1 --2- 172.21.15.169:0/3942474337 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f98b80a2a10 0x7f98b80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.155+0000 7f98c37fe640 1 -- 172.21.15.169:0/3942474337 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f98bc0ce580 con 0x7f98cc10d9c0 2024-03-28T01:03:15.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.156+0000 7f98c0ff9640 1 -- 172.21.15.169:0/2908514605 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4015020 con 0x5590c1a53790 2024-03-28T01:03:15.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.156+0000 7f98c0ff9640 1 -- 172.21.15.169:0/2908514605 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f98c4009e30 con 0x5590c1a53790 2024-03-28T01:03:15.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.157+0000 7f98d384f740 1 -- 172.21.15.169:0/2908514605 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a53790 msgr2=0x5590c1a58290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:15.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.157+0000 7f98d384f740 1 --2- 172.21.15.169:0/2908514605 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a53790 0x5590c1a58290 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f98c4009720 tx=0x7f98c4009be0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.158+0000 7f98c9d9b640 1 --2- 172.21.15.169:0/3942474337 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f98b80a2a10 0x7f98b80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:15.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.163+0000 7f98d384f740 1 -- 172.21.15.169:0/2908514605 shutdown_connections 2024-03-28T01:03:15.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.163+0000 7f98d384f740 1 --2- 172.21.15.169:0/2908514605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590c1a58900 0x5590c1a5ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.163+0000 7f98d384f740 1 --2- 172.21.15.169:0/2908514605 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a53790 0x5590c1a58290 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.163+0000 7f98d384f740 1 --2- 172.21.15.169:0/2908514605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a52da0 0x5590c1a531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.163+0000 7f98d384f740 1 -- 172.21.15.169:0/2908514605 >> 172.21.15.169:0/2908514605 conn(0x5590c1a4e310 msgr2=0x5590c1a4e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:15.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.170+0000 7f98d384f740 1 -- 172.21.15.169:0/2908514605 shutdown_connections 2024-03-28T01:03:15.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.175+0000 7f98d384f740 1 -- 172.21.15.169:0/2908514605 wait complete. 2024-03-28T01:03:15.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.175+0000 7f98d384f740 1 -- start start 2024-03-28T01:03:15.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.175+0000 7f98d384f740 1 Processor -- start 2024-03-28T01:03:15.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.178+0000 7f98c9d9b640 1 --2- 172.21.15.169:0/3942474337 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f98b80a2a10 0x7f98b80a4ed0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f98b40096c0 tx=0x7f98b4005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:15.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.180+0000 7f98d384f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a52da0 0x5590c1a5d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.180+0000 7f98d384f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a53790 0x5590c1a5d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.180+0000 7f98d384f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590c1a58900 0x5590c1a64720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.180+0000 7f98d384f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590c1a5b900 con 0x5590c1a52da0 2024-03-28T01:03:15.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.180+0000 7f98d384f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590c1974060 con 0x5590c1a53790 2024-03-28T01:03:15.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.180+0000 7f98d384f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590c1a5ba80 con 0x5590c1a58900 2024-03-28T01:03:15.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.180+0000 7f98d384f740 10 client.0 _ll_register_callbacks cb 0x5590c19566a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:15.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.183+0000 7f98c9d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a53790 0x5590c1a5d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:15.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.183+0000 7f98c9d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a53790 0x5590c1a5d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:43200/0 (socket says 172.21.15.169:43200) 2024-03-28T01:03:15.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.183+0000 7f98c9d9b640 1 -- 172.21.15.169:0/4194398144 learned_addr learned my addr 172.21.15.169:0/4194398144 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:15.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.184+0000 7f98ca59c640 1 --2- 172.21.15.169:0/4194398144 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a52da0 0x5590c1a5d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:15.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.185+0000 7f98c9d9b640 1 -- 172.21.15.169:0/4194398144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590c1a58900 msgr2=0x5590c1a64720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:03:15.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.185+0000 7f98c9d9b640 1 --2- 172.21.15.169:0/4194398144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590c1a58900 0x5590c1a64720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.185+0000 7f98c9d9b640 1 -- 172.21.15.169:0/4194398144 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a52da0 msgr2=0x5590c1a5d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:15.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.185+0000 7f98c9d9b640 1 --2- 172.21.15.169:0/4194398144 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a52da0 0x5590c1a5d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.185+0000 7f98c9d9b640 1 -- 172.21.15.169:0/4194398144 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5590c16cc3f0 con 0x5590c1a53790 2024-03-28T01:03:15.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.187+0000 7f98c9d9b640 1 --2- 172.21.15.169:0/4194398144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a53790 0x5590c1a5d6e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f98b4036040 tx=0x7f98b4000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=398fe087437cfc03 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:15.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.187+0000 7f98b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:03:15.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.188+0000 7f98b1ffb640 1 -- 172.21.15.169:0/4194398144 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b4052020 con 0x5590c1a53790 2024-03-28T01:03:15.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.188+0000 7f98b1ffb640 1 -- 172.21.15.169:0/4194398144 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f98b40046c0 con 0x5590c1a53790 2024-03-28T01:03:15.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.188+0000 7f98d384f740 1 -- 172.21.15.169:0/4194398144 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f98c4009e30 con 0x5590c1a53790 2024-03-28T01:03:15.196 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.192+0000 7f98b1ffb640 1 -- 172.21.15.169:0/4194398144 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98b4005590 con 0x5590c1a53790 2024-03-28T01:03:15.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.195+0000 7f98d384f740 10 client.26594 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:15.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.195+0000 7f98d384f740 1 -- 172.21.15.169:0/4194398144 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5590c168c0f0 con 0x5590c1a53790 2024-03-28T01:03:15.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.199+0000 7f98b1ffb640 1 -- 172.21.15.169:0/4194398144 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f98c4009e30 con 0x5590c1a53790 2024-03-28T01:03:15.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.199+0000 7f98b1ffb640 1 -- 172.21.15.169:0/4194398144 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f98b404e480 con 0x5590c1a53790 2024-03-28T01:03:15.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.200+0000 7f98d384f740 10 client.26594 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:15.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.200+0000 7f98d384f740 10 client.26594 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:15.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.200+0000 7f98d384f740 20 client.26594 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:15.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.200+0000 7f98d384f740 1 --2- 172.21.15.169:0/4194398144 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5590c1a68c00 0x5590c1a6b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.200+0000 7f98d384f740 4 client.26594 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:03:15.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.200+0000 7f98d384f740 1 -- 172.21.15.169:0/4194398144 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5590c19c5910 con 0x5590c1a68c00 2024-03-28T01:03:15.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.202+0000 7f98ca59c640 1 --2- 172.21.15.169:0/4194398144 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5590c1a68c00 0x5590c1a6b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:15.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.214+0000 7f98ca59c640 1 --2- 172.21.15.169:0/4194398144 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5590c1a68c00 0x5590c1a6b050 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afc400fd1b2c819d server_cookie=e6276273aec74e1a in_seq=0 out_seq=0 2024-03-28T01:03:15.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.214+0000 7f98b1ffb640 10 client.26594 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:15.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.215+0000 7f98b1ffb640 1 -- 172.21.15.169:0/4194398144 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5590c19c5910 con 0x5590c1a68c00 2024-03-28T01:03:15.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.215+0000 7f98b1ffb640 10 client.26594 handle_command_reply: tid=0 2024-03-28T01:03:15.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.217+0000 7f98d384f740 1 client.26594 shutdown 2024-03-28T01:03:15.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.222+0000 7f98d384f740 1 -- 172.21.15.169:0/4194398144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a53790 msgr2=0x5590c1a5d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:15.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.222+0000 7f98d384f740 1 --2- 172.21.15.169:0/4194398144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a53790 0x5590c1a5d6e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f98b4036040 tx=0x7f98b4000bc0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.227+0000 7f98d384f740 1 -- 172.21.15.169:0/4194398144 shutdown_connections 2024-03-28T01:03:15.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.227+0000 7f98d384f740 1 --2- 172.21.15.169:0/4194398144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590c1a58900 0x5590c1a64720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.227+0000 7f98d384f740 1 --2- 172.21.15.169:0/4194398144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a53790 0x5590c1a5d6e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.227+0000 7f98d384f740 1 --2- 172.21.15.169:0/4194398144 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5590c1a68c00 0x5590c1a6b050 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.227+0000 7f98d384f740 1 --2- 172.21.15.169:0/4194398144 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a52da0 0x5590c1a5d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.227+0000 7f98d384f740 1 -- 172.21.15.169:0/4194398144 >> 172.21.15.169:0/4194398144 conn(0x5590c1a4e310 msgr2=0x5590c1a4d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:15.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.228+0000 7f98b1ffb640 0 client.26594 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:15.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.232+0000 7f98d384f740 1 -- 172.21.15.169:0/4194398144 shutdown_connections 2024-03-28T01:03:15.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.236+0000 7f98d384f740 1 -- 172.21.15.169:0/4194398144 wait complete. 2024-03-28T01:03:15.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.238+0000 7f98d384f740 20 client.26594 trim_cache size 0 max 16384 2024-03-28T01:03:15.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.274+0000 7f98d384f740 1 Processor -- start 2024-03-28T01:03:15.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.274+0000 7f98d384f740 1 -- start start 2024-03-28T01:03:15.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.274+0000 7f98d384f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a52da0 0x5590c1a54ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.274+0000 7f98d384f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a53790 0x5590c1a54ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.274+0000 7f98d384f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590c1a58900 0x5590c1a646e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.274+0000 7f98d384f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc1c3de0 con 0x5590c1a53790 2024-03-28T01:03:15.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.274+0000 7f98d384f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc1c18e0 con 0x5590c1a52da0 2024-03-28T01:03:15.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.274+0000 7f98d384f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc118200 con 0x5590c1a58900 2024-03-28T01:03:15.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.275+0000 7f98c9d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a53790 0x5590c1a54ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:15.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.275+0000 7f98c9d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a53790 0x5590c1a54ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:59328/0 (socket says 172.21.15.169:59328) 2024-03-28T01:03:15.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.275+0000 7f98c9d9b640 1 -- 172.21.15.169:0/1705319743 learned_addr learned my addr 172.21.15.169:0/1705319743 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:15.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.275+0000 7f98cad9d640 1 --2- 172.21.15.169:0/1705319743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590c1a58900 0x5590c1a646e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:15.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.276+0000 7f98c9d9b640 1 -- 172.21.15.169:0/1705319743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590c1a58900 msgr2=0x5590c1a646e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:15.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.276+0000 7f98c9d9b640 1 --2- 172.21.15.169:0/1705319743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590c1a58900 0x5590c1a646e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.276+0000 7f98c9d9b640 1 -- 172.21.15.169:0/1705319743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a52da0 msgr2=0x5590c1a54ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:15.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.276+0000 7f98c9d9b640 1 --2- 172.21.15.169:0/1705319743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a52da0 0x5590c1a54ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.276+0000 7f98c9d9b640 1 -- 172.21.15.169:0/1705319743 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5590c1a57da0 con 0x5590c1a53790 2024-03-28T01:03:15.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.276+0000 7f98c9d9b640 1 --2- 172.21.15.169:0/1705319743 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a53790 0x5590c1a54ff0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f98b402cff0 tx=0x7f98b402c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb6935dc02fcf99d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:15.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.277+0000 7f98b3fff640 1 -- 172.21.15.169:0/1705319743 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b4052070 con 0x5590c1a53790 2024-03-28T01:03:15.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.277+0000 7f98b3fff640 1 -- 172.21.15.169:0/1705319743 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f98b404ea90 con 0x5590c1a53790 2024-03-28T01:03:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.278+0000 7f98d384f740 1 -- 172.21.15.169:0/1705319743 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a53790 msgr2=0x5590c1a54ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.278+0000 7f98d384f740 1 --2- 172.21.15.169:0/1705319743 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a53790 0x5590c1a54ff0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f98b402cff0 tx=0x7f98b402c300 comp rx=0 tx=0).stop 2024-03-28T01:03:15.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.281+0000 7f98d384f740 1 -- 172.21.15.169:0/1705319743 shutdown_connections 2024-03-28T01:03:15.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.281+0000 7f98d384f740 1 --2- 172.21.15.169:0/1705319743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590c1a58900 0x5590c1a646e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.281+0000 7f98d384f740 1 --2- 172.21.15.169:0/1705319743 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a53790 0x5590c1a54ff0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.281+0000 7f98d384f740 1 --2- 172.21.15.169:0/1705319743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a52da0 0x5590c1a54ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.281+0000 7f98d384f740 1 -- 172.21.15.169:0/1705319743 >> 172.21.15.169:0/1705319743 conn(0x5590c1a4e310 msgr2=0x5590c1a5d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:15.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.286+0000 7f98d384f740 1 -- 172.21.15.169:0/1705319743 shutdown_connections 2024-03-28T01:03:15.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.287+0000 7f98d384f740 1 -- 172.21.15.169:0/1705319743 wait complete. 2024-03-28T01:03:15.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.287+0000 7f98d384f740 1 -- start start 2024-03-28T01:03:15.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.287+0000 7f98d384f740 1 Processor -- start 2024-03-28T01:03:15.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.288+0000 7f98d384f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a52da0 0x5590c1a622b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.288+0000 7f98d384f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a53790 0x5590c1a627f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.288+0000 7f98d384f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590c1a58900 0x5590c1a63740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.288+0000 7f98d384f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc1bffa0 con 0x5590c1a52da0 2024-03-28T01:03:15.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.288+0000 7f98d384f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc1c1280 con 0x5590c1a53790 2024-03-28T01:03:15.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.288+0000 7f98d384f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc1c05c0 con 0x5590c1a58900 2024-03-28T01:03:15.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.288+0000 7f98d384f740 10 client.0 _ll_register_callbacks cb 0x7f98b4045070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:15.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.289+0000 7f98c9d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a53790 0x5590c1a627f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:15.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.289+0000 7f98cad9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590c1a58900 0x5590c1a63740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:15.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.290+0000 7f98cad9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590c1a58900 0x5590c1a63740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:35100/0 (socket says 172.21.15.169:35100) 2024-03-28T01:03:15.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.290+0000 7f98cad9d640 1 -- 172.21.15.169:0/2741459918 learned_addr learned my addr 172.21.15.169:0/2741459918 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:15.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.293+0000 7f98c9d9b640 1 -- 172.21.15.169:0/2741459918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590c1a58900 msgr2=0x5590c1a63740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:15.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.293+0000 7f98c9d9b640 1 --2- 172.21.15.169:0/2741459918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590c1a58900 0x5590c1a63740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.293+0000 7f98c9d9b640 1 -- 172.21.15.169:0/2741459918 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a52da0 msgr2=0x5590c1a622b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:15.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.293+0000 7f98c9d9b640 1 --2- 172.21.15.169:0/2741459918 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a52da0 0x5590c1a622b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.293+0000 7f98c9d9b640 1 -- 172.21.15.169:0/2741459918 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5590c1a63d30 con 0x5590c1a53790 2024-03-28T01:03:15.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.294+0000 7f98c9d9b640 1 --2- 172.21.15.169:0/2741459918 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a53790 0x5590c1a627f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f98b4006010 tx=0x7f98b404ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a669999724216289 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:15.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.294+0000 7f98b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:03:15.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.294+0000 7f98b1ffb640 1 -- 172.21.15.169:0/2741459918 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b406e070 con 0x5590c1a53790 2024-03-28T01:03:15.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.294+0000 7f98d384f740 1 -- 172.21.15.169:0/2741459918 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f98b404ea90 con 0x5590c1a53790 2024-03-28T01:03:15.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.295+0000 7f98b1ffb640 1 -- 172.21.15.169:0/2741459918 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f98b4003b80 con 0x5590c1a53790 2024-03-28T01:03:15.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.299+0000 7f98b1ffb640 1 -- 172.21.15.169:0/2741459918 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98b4052040 con 0x5590c1a53790 2024-03-28T01:03:15.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.300+0000 7f98d384f740 10 client.26597 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:15.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.300+0000 7f98d384f740 1 -- 172.21.15.169:0/2741459918 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5590c1ac4810 con 0x5590c1a53790 2024-03-28T01:03:15.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.302+0000 7f98b1ffb640 1 -- 172.21.15.169:0/2741459918 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f98b404ea90 con 0x5590c1a53790 2024-03-28T01:03:15.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.304+0000 7f98b1ffb640 1 -- 172.21.15.169:0/2741459918 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f98b4071020 con 0x5590c1a53790 2024-03-28T01:03:15.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.304+0000 7f98d384f740 10 client.26597 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:15.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.304+0000 7f98d384f740 10 client.26597 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:15.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.304+0000 7f98d384f740 20 client.26597 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:15.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.304+0000 7f98d384f740 1 --2- 172.21.15.169:0/2741459918 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5590c1ac4e70 0x5590c1ac7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.304+0000 7f98d384f740 4 client.26597 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:03:15.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.304+0000 7f98d384f740 1 -- 172.21.15.169:0/2741459918 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5590c19bfc60 con 0x5590c1ac4e70 2024-03-28T01:03:15.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.306+0000 7f98ca59c640 1 --2- 172.21.15.169:0/2741459918 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5590c1ac4e70 0x5590c1ac7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:15.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.310+0000 7f98ca59c640 1 --2- 172.21.15.169:0/2741459918 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5590c1ac4e70 0x5590c1ac7260 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a57be28e3ae4c4d server_cookie=7ff291831c962600 in_seq=0 out_seq=0 2024-03-28T01:03:15.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.311+0000 7f98b1ffb640 10 client.26597 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:15.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.311+0000 7f98b1ffb640 1 -- 172.21.15.169:0/2741459918 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5590c19bfc60 con 0x5590c1ac4e70 2024-03-28T01:03:15.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.311+0000 7f98b1ffb640 10 client.26597 handle_command_reply: tid=0 2024-03-28T01:03:15.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.313+0000 7f98d384f740 1 client.26597 shutdown 2024-03-28T01:03:15.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.317+0000 7f98d384f740 1 -- 172.21.15.169:0/2741459918 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a53790 msgr2=0x5590c1a627f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:15.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.317+0000 7f98d384f740 1 --2- 172.21.15.169:0/2741459918 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a53790 0x5590c1a627f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f98b4006010 tx=0x7f98b404ed30 comp rx=0 tx=0).stop 2024-03-28T01:03:15.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.317+0000 7f98d384f740 1 -- 172.21.15.169:0/2741459918 shutdown_connections 2024-03-28T01:03:15.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.317+0000 7f98d384f740 1 --2- 172.21.15.169:0/2741459918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590c1a58900 0x5590c1a63740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.317+0000 7f98d384f740 1 --2- 172.21.15.169:0/2741459918 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590c1a53790 0x5590c1a627f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.317+0000 7f98d384f740 1 --2- 172.21.15.169:0/2741459918 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5590c1ac4e70 0x5590c1ac7260 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.317+0000 7f98d384f740 1 --2- 172.21.15.169:0/2741459918 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5590c1a52da0 0x5590c1a622b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.317+0000 7f98d384f740 1 -- 172.21.15.169:0/2741459918 >> 172.21.15.169:0/2741459918 conn(0x5590c1a4e310 msgr2=0x5590c1a54ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:15.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.318+0000 7f98b1ffb640 0 client.26597 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:15.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.318+0000 7f98d384f740 1 -- 172.21.15.169:0/2741459918 shutdown_connections 2024-03-28T01:03:15.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.318+0000 7f98d384f740 1 -- 172.21.15.169:0/2741459918 wait complete. 2024-03-28T01:03:15.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.318+0000 7f98d384f740 20 client.26597 trim_cache size 0 max 16384 2024-03-28T01:03:15.322 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:03:15.322 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "no active scrubs running", 2024-03-28T01:03:15.322 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": {} 2024-03-28T01:03:15.322 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:03:15.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.321+0000 7f98d0c68640 1 -- 172.21.15.169:0/3942474337 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f98b80a2a10 msgr2=0x7f98b80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:15.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.321+0000 7f98d0c68640 1 --2- 172.21.15.169:0/3942474337 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f98b80a2a10 0x7f98b80a4ed0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f98b40096c0 tx=0x7f98b4005b50 comp rx=0 tx=0).stop 2024-03-28T01:03:15.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.321+0000 7f98d0c68640 1 -- 172.21.15.169:0/3942474337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98cc10d9c0 msgr2=0x7f98cc1adf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:15.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.321+0000 7f98d0c68640 1 --2- 172.21.15.169:0/3942474337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98cc10d9c0 0x7f98cc1adf00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00bdf0 tx=0x7f98bc009ef0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.321+0000 7f98d0c68640 1 -- 172.21.15.169:0/3942474337 shutdown_connections 2024-03-28T01:03:15.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.321+0000 7f98d0c68640 1 --2- 172.21.15.169:0/3942474337 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f98b80a2a10 0x7f98b80a4ed0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.322+0000 7f98d0c68640 1 --2- 172.21.15.169:0/3942474337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98cc10d9c0 0x7f98cc1adf00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.322+0000 7f98d0c68640 1 --2- 172.21.15.169:0/3942474337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98cc10b010 0x7f98cc1a6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.322+0000 7f98d0c68640 1 --2- 172.21.15.169:0/3942474337 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98cc1086a0 0x7f98cc1a6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.322+0000 7f98d0c68640 1 -- 172.21.15.169:0/3942474337 >> 172.21.15.169:0/3942474337 conn(0x7f98cc06e500 msgr2=0x7f98cc108ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:15.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.322+0000 7f98d0c68640 1 -- 172.21.15.169:0/3942474337 shutdown_connections 2024-03-28T01:03:15.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.322+0000 7f98d0c68640 1 -- 172.21.15.169:0/3942474337 wait complete. 2024-03-28T01:03:15.349 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T01:03:15.350 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T01:03:15.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.546+0000 7f87f9b77640 1 -- 172.21.15.169:0/3922818159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87f4101090 msgr2=0x7f87f4105930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:15.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.546+0000 7f87f9b77640 1 --2- 172.21.15.169:0/3922818159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87f4101090 0x7f87f4105930 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f87ec009ca0 tx=0x7f87ec02c140 comp rx=0 tx=0).stop 2024-03-28T01:03:15.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.546+0000 7f87f9b77640 1 -- 172.21.15.169:0/3922818159 shutdown_connections 2024-03-28T01:03:15.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.546+0000 7f87f9b77640 1 --2- 172.21.15.169:0/3922818159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87f4101090 0x7f87f4105930 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.546+0000 7f87f9b77640 1 --2- 172.21.15.169:0/3922818159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87f41006d0 0x7f87f4100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.546+0000 7f87f9b77640 1 --2- 172.21.15.169:0/3922818159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87f40ff970 0x7f87f40ffd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.546+0000 7f87f9b77640 1 -- 172.21.15.169:0/3922818159 >> 172.21.15.169:0/3922818159 conn(0x7f87f40fb2a0 msgr2=0x7f87f40fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:15.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.548+0000 7f87f9b77640 1 -- 172.21.15.169:0/3922818159 shutdown_connections 2024-03-28T01:03:15.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.548+0000 7f87f9b77640 1 -- 172.21.15.169:0/3922818159 wait complete. 2024-03-28T01:03:15.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.549+0000 7f87f9b77640 1 Processor -- start 2024-03-28T01:03:15.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.549+0000 7f87f9b77640 1 -- start start 2024-03-28T01:03:15.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.549+0000 7f87f9b77640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87f40ff970 0x7f87f410ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.549+0000 7f87f9b77640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87f41006d0 0x7f87f410f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.549+0000 7f87f9b77640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87f4101090 0x7f87f41101f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.549+0000 7f87f9b77640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f4107360 con 0x7f87f4101090 2024-03-28T01:03:15.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.549+0000 7f87f9b77640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f41071e0 con 0x7f87f41006d0 2024-03-28T01:03:15.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.549+0000 7f87f9b77640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f41074e0 con 0x7f87f40ff970 2024-03-28T01:03:15.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.550+0000 7f87f2ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87f41006d0 0x7f87f410f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:15.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.550+0000 7f87f2ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87f41006d0 0x7f87f410f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:43254/0 (socket says 172.21.15.169:43254) 2024-03-28T01:03:15.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.550+0000 7f87f3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87f4101090 0x7f87f41101f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:15.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.550+0000 7f87f3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87f4101090 0x7f87f41101f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:59342/0 (socket says 172.21.15.169:59342) 2024-03-28T01:03:15.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.550+0000 7f87f3fff640 1 -- 172.21.15.169:0/4077812698 learned_addr learned my addr 172.21.15.169:0/4077812698 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:15.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.551+0000 7f87f3fff640 1 -- 172.21.15.169:0/4077812698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87f40ff970 msgr2=0x7f87f410ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:15.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.551+0000 7f87f3fff640 1 --2- 172.21.15.169:0/4077812698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87f40ff970 0x7f87f410ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.551+0000 7f87f3fff640 1 -- 172.21.15.169:0/4077812698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87f41006d0 msgr2=0x7f87f410f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:15.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.551+0000 7f87f3fff640 1 --2- 172.21.15.169:0/4077812698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87f41006d0 0x7f87f410f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.551+0000 7f87f3fff640 1 -- 172.21.15.169:0/4077812698 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f4110860 con 0x7f87f4101090 2024-03-28T01:03:15.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.551+0000 7f87f3fff640 1 --2- 172.21.15.169:0/4077812698 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87f4101090 0x7f87f41101f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f87ec0098d0 tx=0x7f87ec035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:15.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.552+0000 7f87f0ff9640 1 -- 172.21.15.169:0/4077812698 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec039040 con 0x7f87f4101090 2024-03-28T01:03:15.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.552+0000 7f87f0ff9640 1 -- 172.21.15.169:0/4077812698 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87ec044070 con 0x7f87f4101090 2024-03-28T01:03:15.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.552+0000 7f87f0ff9640 1 -- 172.21.15.169:0/4077812698 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec03f6a0 con 0x7f87f4101090 2024-03-28T01:03:15.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.552+0000 7f87f9b77640 1 -- 172.21.15.169:0/4077812698 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f4110af0 con 0x7f87f4101090 2024-03-28T01:03:15.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.552+0000 7f87f9b77640 1 -- 172.21.15.169:0/4077812698 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f4111010 con 0x7f87f4101090 2024-03-28T01:03:15.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.555+0000 7f87f9b77640 1 -- 172.21.15.169:0/4077812698 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e00052c0 con 0x7f87f4101090 2024-03-28T01:03:15.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.557+0000 7f87f0ff9640 1 -- 172.21.15.169:0/4077812698 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f87ec003b70 con 0x7f87f4101090 2024-03-28T01:03:15.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.558+0000 7f87f0ff9640 1 --2- 172.21.15.169:0/4077812698 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f87dc0a2a10 0x7f87dc0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:15.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.558+0000 7f87f0ff9640 1 -- 172.21.15.169:0/4077812698 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f87ec0f11c0 con 0x7f87f4101090 2024-03-28T01:03:15.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.559+0000 7f87f37fe640 1 --2- 172.21.15.169:0/4077812698 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f87dc0a2a10 0x7f87dc0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:15.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.560+0000 7f87f37fe640 1 --2- 172.21.15.169:0/4077812698 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f87dc0a2a10 0x7f87dc0a4ed0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f87e80096f0 tx=0x7f87e8006c40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:15.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.570+0000 7f87f0ff9640 1 -- 172.21.15.169:0/4077812698 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f87ec04d020 con 0x7f87f4101090 2024-03-28T01:03:15.606 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:15 smithi169 ceph-mon[32274]: pgmap v553: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s rd, 492 MiB/s wr, 3.94k op/s 2024-03-28T01:03:15.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:15 smithi174 ceph-mon[36060]: pgmap v553: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s rd, 492 MiB/s wr, 3.94k op/s 2024-03-28T01:03:15.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:15 smithi177 ceph-mon[36206]: pgmap v553: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s rd, 492 MiB/s wr, 3.94k op/s 2024-03-28T01:03:15.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.937+0000 7f87f9b77640 1 -- 172.21.15.169:0/4077812698 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87e00055b0 con 0x7f87f4101090 2024-03-28T01:03:15.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.939+0000 7f87f0ff9640 1 -- 172.21.15.169:0/4077812698 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7f87ec0bde60 con 0x7f87f4101090 2024-03-28T01:03:15.945 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T01:03:15.945 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T01:03:15.945 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T01:03:15.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.945+0000 7f87f9b77640 1 -- 172.21.15.169:0/4077812698 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f87dc0a2a10 msgr2=0x7f87dc0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:15.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.945+0000 7f87f9b77640 1 --2- 172.21.15.169:0/4077812698 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f87dc0a2a10 0x7f87dc0a4ed0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f87e80096f0 tx=0x7f87e8006c40 comp rx=0 tx=0).stop 2024-03-28T01:03:15.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.945+0000 7f87f9b77640 1 -- 172.21.15.169:0/4077812698 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87f4101090 msgr2=0x7f87f41101f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:15.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.946+0000 7f87f9b77640 1 --2- 172.21.15.169:0/4077812698 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87f4101090 0x7f87f41101f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f87ec0098d0 tx=0x7f87ec035d10 comp rx=0 tx=0).stop 2024-03-28T01:03:15.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.946+0000 7f87f9b77640 1 -- 172.21.15.169:0/4077812698 shutdown_connections 2024-03-28T01:03:15.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.946+0000 7f87f9b77640 1 --2- 172.21.15.169:0/4077812698 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f87dc0a2a10 0x7f87dc0a4ed0 secure :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f87e80096f0 tx=0x7f87e8006c40 comp rx=0 tx=0).stop 2024-03-28T01:03:15.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.946+0000 7f87f9b77640 1 --2- 172.21.15.169:0/4077812698 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87f4101090 0x7f87f41101f0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.946+0000 7f87f9b77640 1 --2- 172.21.15.169:0/4077812698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87f41006d0 0x7f87f410f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.946+0000 7f87f9b77640 1 --2- 172.21.15.169:0/4077812698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87f40ff970 0x7f87f410ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:15.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.946+0000 7f87f9b77640 1 -- 172.21.15.169:0/4077812698 >> 172.21.15.169:0/4077812698 conn(0x7f87f40fb2a0 msgr2=0x7f87f4103990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:15.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.947+0000 7f87f9b77640 1 -- 172.21.15.169:0/4077812698 shutdown_connections 2024-03-28T01:03:15.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:15.947+0000 7f87f9b77640 1 -- 172.21.15.169:0/4077812698 wait complete. 2024-03-28T01:03:15.983 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:03:15.983 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T01:03:16.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.153+0000 7f0a4a777640 1 -- 172.21.15.169:0/2769685048 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a441017b0 msgr2=0x7f0a44101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.153+0000 7f0a4a777640 1 --2- 172.21.15.169:0/2769685048 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a441017b0 0x7f0a44101c30 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0a38009900 tx=0x7f0a3802bc30 comp rx=0 tx=0).stop 2024-03-28T01:03:16.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.153+0000 7f0a4a777640 1 -- 172.21.15.169:0/2769685048 shutdown_connections 2024-03-28T01:03:16.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.153+0000 7f0a4a777640 1 --2- 172.21.15.169:0/2769685048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a44102170 0x7f0a44108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.153+0000 7f0a4a777640 1 --2- 172.21.15.169:0/2769685048 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a441017b0 0x7f0a44101c30 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.153+0000 7f0a4a777640 1 --2- 172.21.15.169:0/2769685048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a441005c0 0x7f0a441009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.153+0000 7f0a4a777640 1 -- 172.21.15.169:0/2769685048 >> 172.21.15.169:0/2769685048 conn(0x7f0a440fbb30 msgr2=0x7f0a440fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:16.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.153+0000 7f0a4a777640 1 -- 172.21.15.169:0/2769685048 shutdown_connections 2024-03-28T01:03:16.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.153+0000 7f0a4a777640 1 -- 172.21.15.169:0/2769685048 wait complete. 2024-03-28T01:03:16.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.153+0000 7f0a4a777640 1 Processor -- start 2024-03-28T01:03:16.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.153+0000 7f0a4a777640 1 -- start start 2024-03-28T01:03:16.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.154+0000 7f0a4a777640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a441005c0 0x7f0a44197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.154+0000 7f0a4a777640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a441017b0 0x7f0a44197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.154+0000 7f0a4a777640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a44102170 0x7f0a4419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.154+0000 7f0a4a777640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4410c240 con 0x7f0a44102170 2024-03-28T01:03:16.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.154+0000 7f0a4a777640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4410c0c0 con 0x7f0a441005c0 2024-03-28T01:03:16.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.154+0000 7f0a4a777640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4410c3c0 con 0x7f0a441017b0 2024-03-28T01:03:16.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.154+0000 7f0a48d14640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a44102170 0x7f0a4419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.154+0000 7f0a48d14640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a44102170 0x7f0a4419f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:59370/0 (socket says 172.21.15.169:59370) 2024-03-28T01:03:16.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.154+0000 7f0a48d14640 1 -- 172.21.15.169:0/3782035517 learned_addr learned my addr 172.21.15.169:0/3782035517 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:16.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.154+0000 7f0a43fff640 1 --2- 172.21.15.169:0/3782035517 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a441005c0 0x7f0a44197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.154+0000 7f0a437fe640 1 --2- 172.21.15.169:0/3782035517 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a441017b0 0x7f0a44197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.154+0000 7f0a48d14640 1 -- 172.21.15.169:0/3782035517 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a441017b0 msgr2=0x7f0a44197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.154+0000 7f0a48d14640 1 --2- 172.21.15.169:0/3782035517 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a441017b0 0x7f0a44197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.154+0000 7f0a48d14640 1 -- 172.21.15.169:0/3782035517 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a441005c0 msgr2=0x7f0a44197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.154+0000 7f0a48d14640 1 --2- 172.21.15.169:0/3782035517 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a441005c0 0x7f0a44197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.154+0000 7f0a48d14640 1 -- 172.21.15.169:0/3782035517 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4419f690 con 0x7f0a44102170 2024-03-28T01:03:16.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.154+0000 7f0a48d14640 1 --2- 172.21.15.169:0/3782035517 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a44102170 0x7f0a4419f020 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c00eb10 tx=0x7f0a3c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:16.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.154+0000 7f0a417fa640 1 -- 172.21.15.169:0/3782035517 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c004b20 con 0x7f0a44102170 2024-03-28T01:03:16.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.155+0000 7f0a417fa640 1 -- 172.21.15.169:0/3782035517 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a3c004cc0 con 0x7f0a44102170 2024-03-28T01:03:16.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.155+0000 7f0a4a777640 1 -- 172.21.15.169:0/3782035517 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a4419f980 con 0x7f0a44102170 2024-03-28T01:03:16.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.155+0000 7f0a417fa640 1 -- 172.21.15.169:0/3782035517 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c010760 con 0x7f0a44102170 2024-03-28T01:03:16.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.155+0000 7f0a4a777640 1 -- 172.21.15.169:0/3782035517 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a4419fe50 con 0x7f0a44102170 2024-03-28T01:03:16.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.157+0000 7f0a417fa640 1 -- 172.21.15.169:0/3782035517 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0a3c005020 con 0x7f0a44102170 2024-03-28T01:03:16.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.157+0000 7f0a4cffd740 1 Processor -- start 2024-03-28T01:03:16.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.157+0000 7f0a4cffd740 1 -- start start 2024-03-28T01:03:16.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.157+0000 7f0a417fa640 1 --2- 172.21.15.169:0/3782035517 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0a2c0a2800 0x7f0a2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.157+0000 7f0a4cffd740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee59224690 0x55ee59224ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.157+0000 7f0a417fa640 1 -- 172.21.15.169:0/3782035517 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f0a3c0cf3d0 con 0x7f0a44102170 2024-03-28T01:03:16.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.157+0000 7f0a4cffd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee59225080 0x55ee59229b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.157+0000 7f0a4cffd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee5922a1f0 0x55ee5922c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.157+0000 7f0a437fe640 1 --2- 172.21.15.169:0/3782035517 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0a2c0a2800 0x7f0a2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.157+0000 7f0a4cffd740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee591e7610 con 0x55ee59224690 2024-03-28T01:03:16.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.157+0000 7f0a4cffd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee5918a3c0 con 0x55ee59225080 2024-03-28T01:03:16.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.157+0000 7f0a4cffd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee59132d60 con 0x55ee5922a1f0 2024-03-28T01:03:16.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.157+0000 7f0a43fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee59224690 0x55ee59224ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.157+0000 7f0a43fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee59224690 0x55ee59224ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:59382/0 (socket says 172.21.15.169:59382) 2024-03-28T01:03:16.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.157+0000 7f0a43fff640 1 -- 172.21.15.169:0/1840514115 learned_addr learned my addr 172.21.15.169:0/1840514115 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:16.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.158+0000 7f0a48d14640 1 --2- 172.21.15.169:0/1840514115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee5922a1f0 0x55ee5922c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.158+0000 7f0a437fe640 1 --2- 172.21.15.169:0/1840514115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee59225080 0x55ee59229b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.158+0000 7f0a43fff640 1 -- 172.21.15.169:0/1840514115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee5922a1f0 msgr2=0x55ee5922c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.158+0000 7f0a43fff640 1 --2- 172.21.15.169:0/1840514115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee5922a1f0 0x55ee5922c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.158+0000 7f0a43fff640 1 -- 172.21.15.169:0/1840514115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee59225080 msgr2=0x55ee59229b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.158+0000 7f0a43fff640 1 --2- 172.21.15.169:0/1840514115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee59225080 0x55ee59229b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.158+0000 7f0a43fff640 1 -- 172.21.15.169:0/1840514115 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee58e9e3f0 con 0x55ee59224690 2024-03-28T01:03:16.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.158+0000 7f0a437fe640 1 --2- 172.21.15.169:0/1840514115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee59225080 0x55ee59229b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:03:16.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.158+0000 7f0a48d14640 1 --2- 172.21.15.169:0/1840514115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee5922a1f0 0x55ee5922c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:03:16.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.158+0000 7f0a437fe640 1 --2- 172.21.15.169:0/3782035517 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0a2c0a2800 0x7f0a2c0a4cc0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0a38009230 tx=0x7f0a3802cae0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:16.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.158+0000 7f0a43fff640 1 --2- 172.21.15.169:0/1840514115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee59224690 0x55ee59224ab0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f0a34009ca0 tx=0x7f0a34011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7bf039bdc12a8d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:16.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.158+0000 7f0a2a7fc640 1 -- 172.21.15.169:0/1840514115 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34009e50 con 0x55ee59224690 2024-03-28T01:03:16.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.158+0000 7f0a2a7fc640 1 -- 172.21.15.169:0/1840514115 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a340035f0 con 0x55ee59224690 2024-03-28T01:03:16.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.158+0000 7f0a2a7fc640 1 -- 172.21.15.169:0/1840514115 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34019610 con 0x55ee59224690 2024-03-28T01:03:16.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.159+0000 7f0a4cffd740 1 -- 172.21.15.169:0/1840514115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee59224690 msgr2=0x55ee59224ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.159+0000 7f0a4cffd740 1 --2- 172.21.15.169:0/1840514115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee59224690 0x55ee59224ab0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f0a34009ca0 tx=0x7f0a34011810 comp rx=0 tx=0).stop 2024-03-28T01:03:16.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.159+0000 7f0a4cffd740 1 -- 172.21.15.169:0/1840514115 shutdown_connections 2024-03-28T01:03:16.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.159+0000 7f0a4cffd740 1 --2- 172.21.15.169:0/1840514115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee5922a1f0 0x55ee5922c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.159+0000 7f0a4cffd740 1 --2- 172.21.15.169:0/1840514115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee59225080 0x55ee59229b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.159+0000 7f0a4cffd740 1 --2- 172.21.15.169:0/1840514115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee59224690 0x55ee59224ab0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.159+0000 7f0a4cffd740 1 -- 172.21.15.169:0/1840514115 >> 172.21.15.169:0/1840514115 conn(0x55ee5921fc00 msgr2=0x55ee59220000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:16.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.159+0000 7f0a4cffd740 1 -- 172.21.15.169:0/1840514115 shutdown_connections 2024-03-28T01:03:16.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.159+0000 7f0a4cffd740 1 -- 172.21.15.169:0/1840514115 wait complete. 2024-03-28T01:03:16.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.159+0000 7f0a4cffd740 1 -- start start 2024-03-28T01:03:16.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.159+0000 7f0a4cffd740 1 Processor -- start 2024-03-28T01:03:16.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.159+0000 7f0a4cffd740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee59224690 0x55ee5922e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a4cffd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee59225080 0x55ee5922eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a4cffd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee5922a1f0 0x55ee59235f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a4cffd740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee59146240 con 0x55ee59224690 2024-03-28T01:03:16.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a4cffd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee59186460 con 0x55ee5922a1f0 2024-03-28T01:03:16.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a4cffd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee5922d1a0 con 0x55ee59225080 2024-03-28T01:03:16.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a4cffd740 10 client.0 _ll_register_callbacks cb 0x55ee59145c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:16.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a43fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee59224690 0x55ee5922e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a43fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee59224690 0x55ee5922e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:59392/0 (socket says 172.21.15.169:59392) 2024-03-28T01:03:16.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a43fff640 1 -- 172.21.15.169:0/2867855974 learned_addr learned my addr 172.21.15.169:0/2867855974 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:16.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a437fe640 1 --2- 172.21.15.169:0/2867855974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee59225080 0x55ee5922eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a48d14640 1 --2- 172.21.15.169:0/2867855974 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee5922a1f0 0x55ee59235f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a43fff640 1 -- 172.21.15.169:0/2867855974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee59225080 msgr2=0x55ee5922eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a43fff640 1 --2- 172.21.15.169:0/2867855974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee59225080 0x55ee5922eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a43fff640 1 -- 172.21.15.169:0/2867855974 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee5922a1f0 msgr2=0x55ee59235f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a43fff640 1 --2- 172.21.15.169:0/2867855974 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee5922a1f0 0x55ee59235f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a43fff640 1 -- 172.21.15.169:0/2867855974 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ee58f687d0 con 0x55ee59224690 2024-03-28T01:03:16.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a43fff640 1 --2- 172.21.15.169:0/2867855974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee59224690 0x55ee5922e9a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f0a3401b040 tx=0x7f0a340041c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f44cb497deb2a89 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:16.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a237fe640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:03:16.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a237fe640 1 -- 172.21.15.169:0/2867855974 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34028020 con 0x55ee59224690 2024-03-28T01:03:16.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a237fe640 1 -- 172.21.15.169:0/2867855974 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f0a34016040 con 0x55ee59224690 2024-03-28T01:03:16.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a4cffd740 1 -- 172.21.15.169:0/2867855974 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0a340035f0 con 0x55ee59224690 2024-03-28T01:03:16.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a237fe640 1 -- 172.21.15.169:0/2867855974 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a34033050 con 0x55ee59224690 2024-03-28T01:03:16.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a4cffd740 10 client.16701 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:16.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.160+0000 7f0a4cffd740 1 -- 172.21.15.169:0/2867855974 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ee59239290 con 0x55ee59224690 2024-03-28T01:03:16.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.161+0000 7f0a237fe640 1 -- 172.21.15.169:0/2867855974 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a340035f0 con 0x55ee59224690 2024-03-28T01:03:16.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.161+0000 7f0a237fe640 1 -- 172.21.15.169:0/2867855974 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f0a34004a90 con 0x55ee59224690 2024-03-28T01:03:16.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.161+0000 7f0a4cffd740 10 client.16701 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:16.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.161+0000 7f0a4cffd740 10 client.16701 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:03:16.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.161+0000 7f0a4cffd740 20 client.16701 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:16.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.161+0000 7f0a4cffd740 1 --2- 172.21.15.169:0/2867855974 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55ee5923a800 0x55ee5923cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.161+0000 7f0a4cffd740 4 client.16701 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:03:16.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.161+0000 7f0a4cffd740 1 -- 172.21.15.169:0/2867855974 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ee59197900 con 0x55ee5923a800 2024-03-28T01:03:16.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.161+0000 7f0a437fe640 1 --2- 172.21.15.169:0/2867855974 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55ee5923a800 0x55ee5923cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.161+0000 7f0a237fe640 10 client.16701 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:03:16.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.161+0000 7f0a437fe640 1 --2- 172.21.15.169:0/2867855974 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55ee5923a800 0x55ee5923cc10 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=21ee1583a3d5289d server_cookie=f57c97d537b9f229 in_seq=0 out_seq=0 2024-03-28T01:03:16.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.162+0000 7f0a237fe640 1 -- 172.21.15.169:0/2867855974 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ee59197900 con 0x55ee5923a800 2024-03-28T01:03:16.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.162+0000 7f0a237fe640 10 client.16701 handle_command_reply: tid=0 2024-03-28T01:03:16.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.162+0000 7f0a4cffd740 1 client.16701 shutdown 2024-03-28T01:03:16.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.162+0000 7f0a4cffd740 1 -- 172.21.15.169:0/2867855974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee59224690 msgr2=0x55ee5922e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.162+0000 7f0a4cffd740 1 --2- 172.21.15.169:0/2867855974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee59224690 0x55ee5922e9a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f0a3401b040 tx=0x7f0a340041c0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.162+0000 7f0a4cffd740 1 -- 172.21.15.169:0/2867855974 shutdown_connections 2024-03-28T01:03:16.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.162+0000 7f0a4cffd740 1 --2- 172.21.15.169:0/2867855974 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee5922a1f0 0x55ee59235f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.162+0000 7f0a4cffd740 1 --2- 172.21.15.169:0/2867855974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee59225080 0x55ee5922eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.162+0000 7f0a4cffd740 1 --2- 172.21.15.169:0/2867855974 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55ee5923a800 0x55ee5923cc10 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.162+0000 7f0a237fe640 0 client.16701 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:03:16.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.162+0000 7f0a4cffd740 1 --2- 172.21.15.169:0/2867855974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee59224690 0x55ee5922e9a0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.162+0000 7f0a4cffd740 1 -- 172.21.15.169:0/2867855974 >> 172.21.15.169:0/2867855974 conn(0x55ee5921fc00 msgr2=0x55ee5921ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:16.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.162+0000 7f0a4cffd740 1 -- 172.21.15.169:0/2867855974 shutdown_connections 2024-03-28T01:03:16.174 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.162+0000 7f0a4cffd740 1 -- 172.21.15.169:0/2867855974 wait complete. 2024-03-28T01:03:16.174 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.162+0000 7f0a4cffd740 20 client.16701 trim_cache size 0 max 16384 2024-03-28T01:03:16.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.174+0000 7f0a4cffd740 1 Processor -- start 2024-03-28T01:03:16.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.174+0000 7f0a4cffd740 1 -- start start 2024-03-28T01:03:16.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.175+0000 7f0a4cffd740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee5921fc00 0x55ee59223570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.175+0000 7f0a4cffd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee59224690 0x55ee59235ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.175+0000 7f0a4cffd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee59225080 0x55ee592414d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.175+0000 7f0a4cffd740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a441ac770 con 0x55ee5921fc00 2024-03-28T01:03:16.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.175+0000 7f0a4cffd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a441aa270 con 0x55ee59224690 2024-03-28T01:03:16.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.175+0000 7f0a4cffd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a441a8310 con 0x55ee59225080 2024-03-28T01:03:16.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.175+0000 7f0a43fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee5921fc00 0x55ee59223570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.175+0000 7f0a43fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee5921fc00 0x55ee59223570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:59402/0 (socket says 172.21.15.169:59402) 2024-03-28T01:03:16.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.175+0000 7f0a43fff640 1 -- 172.21.15.169:0/1070373128 learned_addr learned my addr 172.21.15.169:0/1070373128 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:16.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.175+0000 7f0a437fe640 1 --2- 172.21.15.169:0/1070373128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee59224690 0x55ee59235ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.175+0000 7f0a48d14640 1 --2- 172.21.15.169:0/1070373128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee59225080 0x55ee592414d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.175+0000 7f0a43fff640 1 -- 172.21.15.169:0/1070373128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee59225080 msgr2=0x55ee592414d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.175+0000 7f0a43fff640 1 --2- 172.21.15.169:0/1070373128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee59225080 0x55ee592414d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.175+0000 7f0a43fff640 1 -- 172.21.15.169:0/1070373128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee59224690 msgr2=0x55ee59235ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.175+0000 7f0a43fff640 1 --2- 172.21.15.169:0/1070373128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee59224690 0x55ee59235ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.175+0000 7f0a43fff640 1 -- 172.21.15.169:0/1070373128 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee59225460 con 0x55ee5921fc00 2024-03-28T01:03:16.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.175+0000 7f0a43fff640 1 --2- 172.21.15.169:0/1070373128 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee5921fc00 0x55ee59223570 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f0a340059f0 tx=0x7f0a340096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2836892d8833e63 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:16.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.175+0000 7f0a29ffb640 1 -- 172.21.15.169:0/1070373128 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34000e60 con 0x55ee5921fc00 2024-03-28T01:03:16.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.175+0000 7f0a29ffb640 1 -- 172.21.15.169:0/1070373128 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a34016030 con 0x55ee5921fc00 2024-03-28T01:03:16.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.175+0000 7f0a29ffb640 1 -- 172.21.15.169:0/1070373128 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34028040 con 0x55ee5921fc00 2024-03-28T01:03:16.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.176+0000 7f0a4cffd740 1 -- 172.21.15.169:0/1070373128 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee5921fc00 msgr2=0x55ee59223570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.176+0000 7f0a4cffd740 1 --2- 172.21.15.169:0/1070373128 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee5921fc00 0x55ee59223570 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f0a340059f0 tx=0x7f0a340096c0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.176+0000 7f0a4cffd740 1 -- 172.21.15.169:0/1070373128 shutdown_connections 2024-03-28T01:03:16.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.176+0000 7f0a4cffd740 1 --2- 172.21.15.169:0/1070373128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee59225080 0x55ee592414d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.176+0000 7f0a4cffd740 1 --2- 172.21.15.169:0/1070373128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee59224690 0x55ee59235ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.176+0000 7f0a4cffd740 1 --2- 172.21.15.169:0/1070373128 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee5921fc00 0x55ee59223570 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.176+0000 7f0a4cffd740 1 -- 172.21.15.169:0/1070373128 >> 172.21.15.169:0/1070373128 conn(0x55ee59225e00 msgr2=0x55ee59222e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:16.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.176+0000 7f0a4cffd740 1 -- 172.21.15.169:0/1070373128 shutdown_connections 2024-03-28T01:03:16.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.176+0000 7f0a4cffd740 1 -- 172.21.15.169:0/1070373128 wait complete. 2024-03-28T01:03:16.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.176+0000 7f0a4cffd740 1 -- start start 2024-03-28T01:03:16.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.176+0000 7f0a4cffd740 1 Processor -- start 2024-03-28T01:03:16.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.176+0000 7f0a4cffd740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee5921fc00 0x55ee59233890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.176+0000 7f0a4cffd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee59224690 0x55ee59233dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.176+0000 7f0a4cffd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee59225080 0x55ee59234de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.176+0000 7f0a4cffd740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a441a8910 con 0x55ee5921fc00 2024-03-28T01:03:16.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.176+0000 7f0a4cffd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a441a9c10 con 0x55ee59225080 2024-03-28T01:03:16.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.176+0000 7f0a4cffd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a441a8f50 con 0x55ee59224690 2024-03-28T01:03:16.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.177+0000 7f0a4cffd740 10 client.0 _ll_register_callbacks cb 0x7f0a34003bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:16.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.177+0000 7f0a43fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee5921fc00 0x55ee59233890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.177+0000 7f0a43fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee5921fc00 0x55ee59233890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:59410/0 (socket says 172.21.15.169:59410) 2024-03-28T01:03:16.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.177+0000 7f0a43fff640 1 -- 172.21.15.169:0/3115778965 learned_addr learned my addr 172.21.15.169:0/3115778965 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:16.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.177+0000 7f0a48d14640 1 --2- 172.21.15.169:0/3115778965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee59225080 0x55ee59234de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.177+0000 7f0a437fe640 1 --2- 172.21.15.169:0/3115778965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee59224690 0x55ee59233dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.177+0000 7f0a48d14640 1 -- 172.21.15.169:0/3115778965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee59224690 msgr2=0x55ee59233dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.177+0000 7f0a48d14640 1 --2- 172.21.15.169:0/3115778965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee59224690 0x55ee59233dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.177+0000 7f0a48d14640 1 -- 172.21.15.169:0/3115778965 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee5921fc00 msgr2=0x55ee59233890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.177+0000 7f0a48d14640 1 --2- 172.21.15.169:0/3115778965 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee5921fc00 0x55ee59233890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.177+0000 7f0a48d14640 1 -- 172.21.15.169:0/3115778965 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ee592353d0 con 0x55ee59225080 2024-03-28T01:03:16.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.177+0000 7f0a43fff640 1 --2- 172.21.15.169:0/3115778965 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee5921fc00 0x55ee59233890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:03:16.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.177+0000 7f0a48d14640 1 --2- 172.21.15.169:0/3115778965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee59225080 0x55ee59234de0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c0ccad0 tx=0x7f0a3c0058d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d17eea9600bdf8bd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:16.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.177+0000 7f0a237fe640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:03:16.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.177+0000 7f0a237fe640 1 -- 172.21.15.169:0/3115778965 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c0b3070 con 0x55ee59225080 2024-03-28T01:03:16.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.177+0000 7f0a4cffd740 1 -- 172.21.15.169:0/3115778965 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0a34016030 con 0x55ee59225080 2024-03-28T01:03:16.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.177+0000 7f0a237fe640 1 -- 172.21.15.169:0/3115778965 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f0a3c0a3c80 con 0x55ee59225080 2024-03-28T01:03:16.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.177+0000 7f0a237fe640 1 -- 172.21.15.169:0/3115778965 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a3c09fac0 con 0x55ee59225080 2024-03-28T01:03:16.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.177+0000 7f0a4cffd740 10 client.26618 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:16.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.177+0000 7f0a4cffd740 1 -- 172.21.15.169:0/3115778965 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ee59296390 con 0x55ee59225080 2024-03-28T01:03:16.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.177+0000 7f0a237fe640 1 -- 172.21.15.169:0/3115778965 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a34016030 con 0x55ee59225080 2024-03-28T01:03:16.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.178+0000 7f0a237fe640 1 -- 172.21.15.169:0/3115778965 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f0a3c005310 con 0x55ee59225080 2024-03-28T01:03:16.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.178+0000 7f0a4cffd740 10 client.26618 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:16.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.178+0000 7f0a4cffd740 10 client.26618 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:03:16.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.178+0000 7f0a4cffd740 20 client.26618 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:16.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.178+0000 7f0a4cffd740 1 --2- 172.21.15.169:0/3115778965 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55ee5922a1f0 0x55ee59298cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.178+0000 7f0a4cffd740 4 client.26618 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:03:16.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.178+0000 7f0a4cffd740 1 -- 172.21.15.169:0/3115778965 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ee59191c50 con 0x55ee5922a1f0 2024-03-28T01:03:16.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.178+0000 7f0a43fff640 1 --2- 172.21.15.169:0/3115778965 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55ee5922a1f0 0x55ee59298cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.178+0000 7f0a237fe640 10 client.26618 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:03:16.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.178+0000 7f0a43fff640 1 --2- 172.21.15.169:0/3115778965 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55ee5922a1f0 0x55ee59298cf0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b4684fac06eed1f6 server_cookie=73ae9225c5416fe4 in_seq=0 out_seq=0 2024-03-28T01:03:16.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.178+0000 7f0a237fe640 1 -- 172.21.15.169:0/3115778965 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ee59191c50 con 0x55ee5922a1f0 2024-03-28T01:03:16.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.178+0000 7f0a237fe640 10 client.26618 handle_command_reply: tid=0 2024-03-28T01:03:16.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.178+0000 7f0a4cffd740 1 client.26618 shutdown 2024-03-28T01:03:16.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.178+0000 7f0a4cffd740 1 -- 172.21.15.169:0/3115778965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee59225080 msgr2=0x55ee59234de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.178+0000 7f0a4cffd740 1 --2- 172.21.15.169:0/3115778965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee59225080 0x55ee59234de0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c0ccad0 tx=0x7f0a3c0058d0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.178+0000 7f0a4cffd740 1 -- 172.21.15.169:0/3115778965 shutdown_connections 2024-03-28T01:03:16.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.178+0000 7f0a4cffd740 1 --2- 172.21.15.169:0/3115778965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee59225080 0x55ee59234de0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.178+0000 7f0a4cffd740 1 --2- 172.21.15.169:0/3115778965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee59224690 0x55ee59233dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.178+0000 7f0a4cffd740 1 --2- 172.21.15.169:0/3115778965 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55ee5922a1f0 0x55ee59298cf0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.178+0000 7f0a4cffd740 1 --2- 172.21.15.169:0/3115778965 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ee5921fc00 0x55ee59233890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.178+0000 7f0a237fe640 0 client.26618 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:03:16.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.179+0000 7f0a4cffd740 1 -- 172.21.15.169:0/3115778965 >> 172.21.15.169:0/3115778965 conn(0x55ee59225e00 msgr2=0x55ee592236f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:16.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.179+0000 7f0a4cffd740 1 -- 172.21.15.169:0/3115778965 shutdown_connections 2024-03-28T01:03:16.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.179+0000 7f0a4cffd740 1 -- 172.21.15.169:0/3115778965 wait complete. 2024-03-28T01:03:16.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.179+0000 7f0a4cffd740 20 client.26618 trim_cache size 0 max 16384 2024-03-28T01:03:16.189 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:03:16.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.179+0000 7f0a4a777640 1 -- 172.21.15.169:0/3782035517 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0a2c0a2800 msgr2=0x7f0a2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.179+0000 7f0a4a777640 1 --2- 172.21.15.169:0/3782035517 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0a2c0a2800 0x7f0a2c0a4cc0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0a38009230 tx=0x7f0a3802cae0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.179+0000 7f0a4a777640 1 -- 172.21.15.169:0/3782035517 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a44102170 msgr2=0x7f0a4419f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.179+0000 7f0a4a777640 1 --2- 172.21.15.169:0/3782035517 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a44102170 0x7f0a4419f020 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c00eb10 tx=0x7f0a3c00c2d0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.179+0000 7f0a4a777640 1 -- 172.21.15.169:0/3782035517 shutdown_connections 2024-03-28T01:03:16.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.179+0000 7f0a4a777640 1 --2- 172.21.15.169:0/3782035517 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0a2c0a2800 0x7f0a2c0a4cc0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.179+0000 7f0a4a777640 1 --2- 172.21.15.169:0/3782035517 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a44102170 0x7f0a4419f020 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.179+0000 7f0a4a777640 1 --2- 172.21.15.169:0/3782035517 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a441017b0 0x7f0a44197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.179+0000 7f0a4a777640 1 --2- 172.21.15.169:0/3782035517 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a441005c0 0x7f0a44197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.179+0000 7f0a4a777640 1 -- 172.21.15.169:0/3782035517 >> 172.21.15.169:0/3782035517 conn(0x7f0a440fbb30 msgr2=0x7f0a44069940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:16.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.179+0000 7f0a4a777640 1 -- 172.21.15.169:0/3782035517 shutdown_connections 2024-03-28T01:03:16.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.179+0000 7f0a4a777640 1 -- 172.21.15.169:0/3782035517 wait complete. 2024-03-28T01:03:16.199 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:03:16.199 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T01:03:16.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.393+0000 7f2ccee61640 1 -- 172.21.15.169:0/2770025297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cc810d9c0 msgr2=0x7f2cc810de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.393+0000 7f2ccee61640 1 --2- 172.21.15.169:0/2770025297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cc810d9c0 0x7f2cc810de40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009860 tx=0x7f2cb802bbb0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.399+0000 7f2ccee61640 1 -- 172.21.15.169:0/2770025297 shutdown_connections 2024-03-28T01:03:16.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.399+0000 7f2ccee61640 1 --2- 172.21.15.169:0/2770025297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cc810d9c0 0x7f2cc810de40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.399+0000 7f2ccee61640 1 --2- 172.21.15.169:0/2770025297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cc810b010 0x7f2cc810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.399+0000 7f2ccee61640 1 --2- 172.21.15.169:0/2770025297 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2cc81086a0 0x7f2cc810aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.399+0000 7f2ccee61640 1 -- 172.21.15.169:0/2770025297 >> 172.21.15.169:0/2770025297 conn(0x7f2cc806e500 msgr2=0x7f2cc8070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:16.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.402+0000 7f2ccee61640 1 -- 172.21.15.169:0/2770025297 shutdown_connections 2024-03-28T01:03:16.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.402+0000 7f2ccee61640 1 -- 172.21.15.169:0/2770025297 wait complete. 2024-03-28T01:03:16.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.403+0000 7f2ccee61640 1 Processor -- start 2024-03-28T01:03:16.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.403+0000 7f2ccee61640 1 -- start start 2024-03-28T01:03:16.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.404+0000 7f2ccee61640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cc81086a0 0x7f2cc8114df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.404+0000 7f2ccee61640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2cc810b010 0x7f2cc8115330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.404+0000 7f2ccee61640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cc810d9c0 0x7f2cc81b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.404+0000 7f2ccee61640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc811c190 con 0x7f2cc810b010 2024-03-28T01:03:16.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.404+0000 7f2ccee61640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc811c010 con 0x7f2cc81086a0 2024-03-28T01:03:16.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.404+0000 7f2ccee61640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc811c310 con 0x7f2cc810d9c0 2024-03-28T01:03:16.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.405+0000 7f2cc7fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2cc810b010 0x7f2cc8115330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.405+0000 7f2cc7fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2cc810b010 0x7f2cc8115330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:59412/0 (socket says 172.21.15.169:59412) 2024-03-28T01:03:16.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.405+0000 7f2cc7fff640 1 -- 172.21.15.169:0/2111481158 learned_addr learned my addr 172.21.15.169:0/2111481158 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:16.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.405+0000 7f2ccd3fe640 1 --2- 172.21.15.169:0/2111481158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cc810d9c0 0x7f2cc81b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.406+0000 7f2cccbfd640 1 --2- 172.21.15.169:0/2111481158 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cc81086a0 0x7f2cc8114df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.407+0000 7f2cc7fff640 1 -- 172.21.15.169:0/2111481158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cc810d9c0 msgr2=0x7f2cc81b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.407+0000 7f2cc7fff640 1 --2- 172.21.15.169:0/2111481158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cc810d9c0 0x7f2cc81b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.407+0000 7f2cc7fff640 1 -- 172.21.15.169:0/2111481158 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cc81086a0 msgr2=0x7f2cc8114df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.407+0000 7f2cc7fff640 1 --2- 172.21.15.169:0/2111481158 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cc81086a0 0x7f2cc8114df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.407+0000 7f2cc7fff640 1 -- 172.21.15.169:0/2111481158 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc81b6de0 con 0x7f2cc810b010 2024-03-28T01:03:16.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.408+0000 7f2cc7fff640 1 --2- 172.21.15.169:0/2111481158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2cc810b010 0x7f2cc8115330 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2cc000efb0 tx=0x7f2cc000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:16.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.408+0000 7f2cc5ffb640 1 -- 172.21.15.169:0/2111481158 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc001b070 con 0x7f2cc810b010 2024-03-28T01:03:16.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.409+0000 7f2ccee61640 1 -- 172.21.15.169:0/2111481158 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc81b6fb0 con 0x7f2cc810b010 2024-03-28T01:03:16.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.409+0000 7f2ccee61640 1 -- 172.21.15.169:0/2111481158 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc81b7540 con 0x7f2cc810b010 2024-03-28T01:03:16.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.409+0000 7f2cc5ffb640 1 -- 172.21.15.169:0/2111481158 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2cc000ba50 con 0x7f2cc810b010 2024-03-28T01:03:16.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.409+0000 7f2cc5ffb640 1 -- 172.21.15.169:0/2111481158 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0016c00 con 0x7f2cc810b010 2024-03-28T01:03:16.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.414+0000 7f2cd1a54740 1 Processor -- start 2024-03-28T01:03:16.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.414+0000 7f2cd1a54740 1 -- start start 2024-03-28T01:03:16.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.415+0000 7f2cd1a54740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042a6690 0x5606042a6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.415+0000 7f2cd1a54740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042a7080 0x5606042abb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.415+0000 7f2cd1a54740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606042ac1f0 0x5606042ae680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.415+0000 7f2cd1a54740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x560604269610 con 0x5606042a6690 2024-03-28T01:03:16.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.415+0000 7f2cd1a54740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x56060420c3c0 con 0x5606042a7080 2024-03-28T01:03:16.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.416+0000 7f2cd1a54740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606041b4d60 con 0x5606042ac1f0 2024-03-28T01:03:16.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.417+0000 7f2ccd3fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042a6690 0x5606042a6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.417+0000 7f2ccd3fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042a6690 0x5606042a6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:59426/0 (socket says 172.21.15.169:59426) 2024-03-28T01:03:16.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.417+0000 7f2ccd3fe640 1 -- 172.21.15.169:0/2697289065 learned_addr learned my addr 172.21.15.169:0/2697289065 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:16.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.418+0000 7f2cccbfd640 1 --2- 172.21.15.169:0/2697289065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042a7080 0x5606042abb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.418+0000 7f2cc5ffb640 1 -- 172.21.15.169:0/2111481158 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2cc001e070 con 0x7f2cc810b010 2024-03-28T01:03:16.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.421+0000 7f2cc5ffb640 1 --2- 172.21.15.169:0/2111481158 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2cb00a28d0 0x7f2cb00a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.421+0000 7f2cc7fff640 1 --2- 172.21.15.169:0/2697289065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606042ac1f0 0x5606042ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.422+0000 7f2ccd3fe640 1 -- 172.21.15.169:0/2697289065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606042ac1f0 msgr2=0x5606042ae680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.422+0000 7f2ccd3fe640 1 --2- 172.21.15.169:0/2697289065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606042ac1f0 0x5606042ae680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.422+0000 7f2ccd3fe640 1 -- 172.21.15.169:0/2697289065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042a7080 msgr2=0x5606042abb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.422+0000 7f2ccd3fe640 1 --2- 172.21.15.169:0/2697289065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042a7080 0x5606042abb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.422+0000 7f2ccd3fe640 1 -- 172.21.15.169:0/2697289065 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560603f203f0 con 0x5606042a6690 2024-03-28T01:03:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.422+0000 7f2cc5ffb640 1 -- 172.21.15.169:0/2111481158 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2cc00cec10 con 0x7f2cc810b010 2024-03-28T01:03:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.422+0000 7f2ccd3fe640 1 --2- 172.21.15.169:0/2111481158 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2cb00a28d0 0x7f2cb00a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.423+0000 7f2ccd3fe640 1 --2- 172.21.15.169:0/2697289065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042a6690 0x5606042a6ab0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f2cc806b390 tx=0x7f2cb80323f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb590151b36a2511 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:16.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.424+0000 7f2ccd3fe640 1 --2- 172.21.15.169:0/2111481158 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2cb00a28d0 0x7f2cb00a4d90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009860 tx=0x7f2cb8045000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:16.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.427+0000 7f2caeffd640 1 -- 172.21.15.169:0/2697289065 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb804e070 con 0x5606042a6690 2024-03-28T01:03:16.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.427+0000 7f2caeffd640 1 -- 172.21.15.169:0/2697289065 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2cb8004380 con 0x5606042a6690 2024-03-28T01:03:16.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.429+0000 7f2cd1a54740 1 -- 172.21.15.169:0/2697289065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042a6690 msgr2=0x5606042a6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.429+0000 7f2cd1a54740 1 --2- 172.21.15.169:0/2697289065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042a6690 0x5606042a6ab0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f2cc806b390 tx=0x7f2cb80323f0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.431+0000 7f2cd1a54740 1 -- 172.21.15.169:0/2697289065 shutdown_connections 2024-03-28T01:03:16.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.431+0000 7f2cd1a54740 1 --2- 172.21.15.169:0/2697289065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606042ac1f0 0x5606042ae680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.431+0000 7f2cd1a54740 1 --2- 172.21.15.169:0/2697289065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042a7080 0x5606042abb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.431+0000 7f2cd1a54740 1 --2- 172.21.15.169:0/2697289065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042a6690 0x5606042a6ab0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.431+0000 7f2cd1a54740 1 -- 172.21.15.169:0/2697289065 >> 172.21.15.169:0/2697289065 conn(0x5606042a1c00 msgr2=0x5606042a2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:16.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.433+0000 7f2cd1a54740 1 -- 172.21.15.169:0/2697289065 shutdown_connections 2024-03-28T01:03:16.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.433+0000 7f2cd1a54740 1 -- 172.21.15.169:0/2697289065 wait complete. 2024-03-28T01:03:16.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.433+0000 7f2cd1a54740 1 -- start start 2024-03-28T01:03:16.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.434+0000 7f2cd1a54740 1 Processor -- start 2024-03-28T01:03:16.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.434+0000 7f2cd1a54740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606042a6690 0x5606042b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.434+0000 7f2cd1a54740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042a7080 0x5606042b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.434+0000 7f2cd1a54740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042ac1f0 0x5606042b7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.434+0000 7f2cd1a54740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606041c8240 con 0x5606042a7080 2024-03-28T01:03:16.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.434+0000 7f2cd1a54740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x560604208460 con 0x5606042ac1f0 2024-03-28T01:03:16.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.434+0000 7f2cd1a54740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606042af190 con 0x5606042a6690 2024-03-28T01:03:16.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.434+0000 7f2cd1a54740 10 client.0 _ll_register_callbacks cb 0x5606041c7c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:16.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.435+0000 7f2cccbfd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606042a6690 0x5606042b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.435+0000 7f2ccd3fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042ac1f0 0x5606042b7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.435+0000 7f2ccd3fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042ac1f0 0x5606042b7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:54484/0 (socket says 172.21.15.169:54484) 2024-03-28T01:03:16.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.435+0000 7f2ccd3fe640 1 -- 172.21.15.169:0/3604304536 learned_addr learned my addr 172.21.15.169:0/3604304536 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:16.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.436+0000 7f2cc7fff640 1 --2- 172.21.15.169:0/3604304536 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042a7080 0x5606042b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.436+0000 7f2cc7fff640 1 -- 172.21.15.169:0/3604304536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606042a6690 msgr2=0x5606042b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.436+0000 7f2cc7fff640 1 --2- 172.21.15.169:0/3604304536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606042a6690 0x5606042b0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.436+0000 7f2cc7fff640 1 -- 172.21.15.169:0/3604304536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042ac1f0 msgr2=0x5606042b7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.436+0000 7f2cc7fff640 1 --2- 172.21.15.169:0/3604304536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042ac1f0 0x5606042b7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.436+0000 7f2cc7fff640 1 -- 172.21.15.169:0/3604304536 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560603fea7d0 con 0x5606042a7080 2024-03-28T01:03:16.440 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.437+0000 7f2cc7fff640 1 --2- 172.21.15.169:0/3604304536 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042a7080 0x5606042b0f70 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f2cc00cc6e0 tx=0x7f2cc00077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b6fd2d927389d2b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:16.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.437+0000 7f2ca7fff640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:03:16.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.437+0000 7f2ca7fff640 1 -- 172.21.15.169:0/3604304536 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc009f360 con 0x5606042a7080 2024-03-28T01:03:16.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.438+0000 7f2cd1a54740 1 -- 172.21.15.169:0/3604304536 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2cb8004380 con 0x5606042a7080 2024-03-28T01:03:16.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.438+0000 7f2ca7fff640 1 -- 172.21.15.169:0/3604304536 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2cc0016410 con 0x5606042a7080 2024-03-28T01:03:16.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.438+0000 7f2ca7fff640 1 -- 172.21.15.169:0/3604304536 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2cc00b40d0 con 0x5606042a7080 2024-03-28T01:03:16.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.439+0000 7f2cd1a54740 10 client.16716 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:16.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.439+0000 7f2cd1a54740 1 -- 172.21.15.169:0/3604304536 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5606042bb3f0 con 0x5606042a7080 2024-03-28T01:03:16.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.440+0000 7f2ca7fff640 1 -- 172.21.15.169:0/3604304536 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2cb8004380 con 0x5606042a7080 2024-03-28T01:03:16.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.441+0000 7f2ca7fff640 1 -- 172.21.15.169:0/3604304536 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2cc000b040 con 0x5606042a7080 2024-03-28T01:03:16.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.441+0000 7f2cd1a54740 10 client.16716 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:16.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.441+0000 7f2cd1a54740 10 client.16716 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:03:16.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.441+0000 7f2cd1a54740 20 client.16716 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:16.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.441+0000 7f2cd1a54740 1 --2- 172.21.15.169:0/3604304536 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5606042bcf70 0x5606042bf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.441+0000 7f2cd1a54740 4 client.16716 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:03:16.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.441+0000 7f2cd1a54740 1 -- 172.21.15.169:0/3604304536 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560604219900 con 0x5606042bcf70 2024-03-28T01:03:16.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.445+0000 7f2cccbfd640 1 --2- 172.21.15.169:0/3604304536 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5606042bcf70 0x5606042bf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.455 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.452+0000 7f2cccbfd640 1 --2- 172.21.15.169:0/3604304536 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5606042bcf70 0x5606042bf3b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8b8a694d29e731ef server_cookie=6eddf24b1f2624a5 in_seq=0 out_seq=0 2024-03-28T01:03:16.455 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.452+0000 7f2ca7fff640 10 client.16716 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:03:16.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.455+0000 7f2ca7fff640 1 -- 172.21.15.169:0/3604304536 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560604219900 con 0x5606042bcf70 2024-03-28T01:03:16.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.455+0000 7f2ca7fff640 10 client.16716 handle_command_reply: tid=0 2024-03-28T01:03:16.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.455+0000 7f2cd1a54740 1 client.16716 shutdown 2024-03-28T01:03:16.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.455+0000 7f2cd1a54740 1 -- 172.21.15.169:0/3604304536 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042a7080 msgr2=0x5606042b0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.455+0000 7f2cd1a54740 1 --2- 172.21.15.169:0/3604304536 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042a7080 0x5606042b0f70 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f2cc00cc6e0 tx=0x7f2cc00077a0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.455+0000 7f2cd1a54740 1 -- 172.21.15.169:0/3604304536 shutdown_connections 2024-03-28T01:03:16.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.455+0000 7f2cd1a54740 1 --2- 172.21.15.169:0/3604304536 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5606042bcf70 0x5606042bf3b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.455+0000 7f2cd1a54740 1 --2- 172.21.15.169:0/3604304536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042ac1f0 0x5606042b7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.455+0000 7f2cd1a54740 1 --2- 172.21.15.169:0/3604304536 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042a7080 0x5606042b0f70 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.455+0000 7f2cd1a54740 1 --2- 172.21.15.169:0/3604304536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606042a6690 0x5606042b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.455+0000 7f2ca7fff640 0 client.16716 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:03:16.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.455+0000 7f2cd1a54740 1 -- 172.21.15.169:0/3604304536 >> 172.21.15.169:0/3604304536 conn(0x5606042a1c00 msgr2=0x5606042a0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:16.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.455+0000 7f2cd1a54740 1 -- 172.21.15.169:0/3604304536 shutdown_connections 2024-03-28T01:03:16.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.455+0000 7f2cd1a54740 1 -- 172.21.15.169:0/3604304536 wait complete. 2024-03-28T01:03:16.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.455+0000 7f2cd1a54740 20 client.16716 trim_cache size 0 max 16384 2024-03-28T01:03:16.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.470+0000 7f2cd1a54740 1 Processor -- start 2024-03-28T01:03:16.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.470+0000 7f2cd1a54740 1 -- start start 2024-03-28T01:03:16.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.470+0000 7f2cd1a54740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042a6690 0x5606042a0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.471+0000 7f2cd1a54740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042a7080 0x5606042b7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.471+0000 7f2cd1a54740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606042ac1f0 0x5606042a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.471+0000 7f2cccbfd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042a6690 0x5606042a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.471+0000 7f2cccbfd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042a6690 0x5606042a0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53104/0 (socket says 172.21.15.169:53104) 2024-03-28T01:03:16.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.471+0000 7f2cccbfd640 1 -- 172.21.15.169:0/1459575501 learned_addr learned my addr 172.21.15.169:0/1459575501 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:16.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.471+0000 7f2cd1a54740 1 -- 172.21.15.169:0/1459575501 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb4011f90 con 0x5606042a6690 2024-03-28T01:03:16.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.471+0000 7f2cd1a54740 1 -- 172.21.15.169:0/1459575501 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb400fa90 con 0x5606042a7080 2024-03-28T01:03:16.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.471+0000 7f2cd1a54740 1 -- 172.21.15.169:0/1459575501 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb400dbe0 con 0x5606042ac1f0 2024-03-28T01:03:16.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.471+0000 7f2cc7fff640 1 --2- 172.21.15.169:0/1459575501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042a7080 0x5606042b7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.471+0000 7f2ccd3fe640 1 --2- 172.21.15.169:0/1459575501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606042ac1f0 0x5606042a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.471+0000 7f2cc7fff640 1 -- 172.21.15.169:0/1459575501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606042ac1f0 msgr2=0x5606042a0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.471+0000 7f2cc7fff640 1 --2- 172.21.15.169:0/1459575501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606042ac1f0 0x5606042a0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.471+0000 7f2cc7fff640 1 -- 172.21.15.169:0/1459575501 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042a6690 msgr2=0x5606042a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.471+0000 7f2cc7fff640 1 --2- 172.21.15.169:0/1459575501 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042a6690 0x5606042a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.471+0000 7f2cc7fff640 1 -- 172.21.15.169:0/1459575501 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606042bfa30 con 0x5606042a7080 2024-03-28T01:03:16.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.471+0000 7f2cccbfd640 1 --2- 172.21.15.169:0/1459575501 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042a6690 0x5606042a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:03:16.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.472+0000 7f2cc7fff640 1 --2- 172.21.15.169:0/1459575501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042a7080 0x5606042b7f70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2cc009b480 tx=0x7f2cc0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8084c8b71955361 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:16.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.472+0000 7f2cae7fc640 1 -- 172.21.15.169:0/1459575501 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc009f360 con 0x5606042a7080 2024-03-28T01:03:16.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.472+0000 7f2cae7fc640 1 -- 172.21.15.169:0/1459575501 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2cc00a39b0 con 0x5606042a7080 2024-03-28T01:03:16.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.472+0000 7f2cae7fc640 1 -- 172.21.15.169:0/1459575501 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc000e0b0 con 0x5606042a7080 2024-03-28T01:03:16.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.473+0000 7f2cd1a54740 1 -- 172.21.15.169:0/1459575501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042a7080 msgr2=0x5606042b7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.473+0000 7f2cd1a54740 1 --2- 172.21.15.169:0/1459575501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042a7080 0x5606042b7f70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2cc009b480 tx=0x7f2cc0009780 comp rx=0 tx=0).stop 2024-03-28T01:03:16.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.473+0000 7f2cd1a54740 1 -- 172.21.15.169:0/1459575501 shutdown_connections 2024-03-28T01:03:16.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.473+0000 7f2cd1a54740 1 --2- 172.21.15.169:0/1459575501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606042ac1f0 0x5606042a0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.473+0000 7f2cd1a54740 1 --2- 172.21.15.169:0/1459575501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042a7080 0x5606042b7f70 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.473+0000 7f2cd1a54740 1 --2- 172.21.15.169:0/1459575501 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042a6690 0x5606042a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.473+0000 7f2cd1a54740 1 -- 172.21.15.169:0/1459575501 >> 172.21.15.169:0/1459575501 conn(0x5606042a1c00 msgr2=0x5606042a84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:16.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.473+0000 7f2cd1a54740 1 -- 172.21.15.169:0/1459575501 shutdown_connections 2024-03-28T01:03:16.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.474+0000 7f2cd1a54740 1 -- 172.21.15.169:0/1459575501 wait complete. 2024-03-28T01:03:16.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.474+0000 7f2cd1a54740 1 -- start start 2024-03-28T01:03:16.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.474+0000 7f2cd1a54740 1 Processor -- start 2024-03-28T01:03:16.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.475+0000 7f2cd1a54740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606042a6690 0x5606042b5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.475+0000 7f2cd1a54740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042a7080 0x5606042b5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.475+0000 7f2cd1a54740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042ac1f0 0x5606042b6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.475+0000 7f2cd1a54740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb400e150 con 0x5606042ac1f0 2024-03-28T01:03:16.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.475+0000 7f2cc7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042a7080 0x5606042b5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.475+0000 7f2cc7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042a7080 0x5606042b5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:54500/0 (socket says 172.21.15.169:54500) 2024-03-28T01:03:16.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.475+0000 7f2cc7fff640 1 -- 172.21.15.169:0/2507080028 learned_addr learned my addr 172.21.15.169:0/2507080028 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:16.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.475+0000 7f2ccd3fe640 1 --2- 172.21.15.169:0/2507080028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042ac1f0 0x5606042b6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.475+0000 7f2cd1a54740 1 -- 172.21.15.169:0/2507080028 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb400f430 con 0x5606042a7080 2024-03-28T01:03:16.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.475+0000 7f2cd1a54740 1 -- 172.21.15.169:0/2507080028 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb400e770 con 0x5606042a6690 2024-03-28T01:03:16.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.475+0000 7f2cccbfd640 1 --2- 172.21.15.169:0/2507080028 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606042a6690 0x5606042b5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.476+0000 7f2cd1a54740 10 client.0 _ll_register_callbacks cb 0x7f2cc009b0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:16.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.476+0000 7f2ccd3fe640 1 -- 172.21.15.169:0/2507080028 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606042a6690 msgr2=0x5606042b5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.476+0000 7f2ccd3fe640 1 --2- 172.21.15.169:0/2507080028 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606042a6690 0x5606042b5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.476+0000 7f2ccd3fe640 1 -- 172.21.15.169:0/2507080028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042a7080 msgr2=0x5606042b5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.476+0000 7f2ccd3fe640 1 --2- 172.21.15.169:0/2507080028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042a7080 0x5606042b5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.476+0000 7f2ccd3fe640 1 -- 172.21.15.169:0/2507080028 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5606042b7580 con 0x5606042ac1f0 2024-03-28T01:03:16.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.476+0000 7f2cc7fff640 1 --2- 172.21.15.169:0/2507080028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042a7080 0x5606042b5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:03:16.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.476+0000 7f2ccd3fe640 1 --2- 172.21.15.169:0/2507080028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042ac1f0 0x5606042b6f90 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009830 tx=0x7f2cb8034ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b5c7df03d326de server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:16.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.476+0000 7f2ca7fff640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:03:16.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.477+0000 7f2ca7fff640 1 -- 172.21.15.169:0/2507080028 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb8056070 con 0x5606042ac1f0 2024-03-28T01:03:16.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.477+0000 7f2ca7fff640 1 -- 172.21.15.169:0/2507080028 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2cb802c550 con 0x5606042ac1f0 2024-03-28T01:03:16.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.477+0000 7f2cd1a54740 1 -- 172.21.15.169:0/2507080028 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2cc00a39b0 con 0x5606042ac1f0 2024-03-28T01:03:16.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.477+0000 7f2ca7fff640 1 -- 172.21.15.169:0/2507080028 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2cb8062050 con 0x5606042ac1f0 2024-03-28T01:03:16.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.477+0000 7f2cd1a54740 10 client.16722 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:16.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.477+0000 7f2cd1a54740 1 -- 172.21.15.169:0/2507080028 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x560604318c90 con 0x5606042ac1f0 2024-03-28T01:03:16.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.478+0000 7f2ca7fff640 1 -- 172.21.15.169:0/2507080028 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2cb80035f0 con 0x5606042ac1f0 2024-03-28T01:03:16.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.478+0000 7f2ca7fff640 1 -- 172.21.15.169:0/2507080028 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2cb803f040 con 0x5606042ac1f0 2024-03-28T01:03:16.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.478+0000 7f2cd1a54740 10 client.16722 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:16.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.478+0000 7f2cd1a54740 10 client.16722 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:03:16.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.478+0000 7f2cd1a54740 20 client.16722 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:16.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.478+0000 7f2cd1a54740 1 --2- 172.21.15.169:0/2507080028 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5606043191d0 0x56060431b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.478+0000 7f2cd1a54740 4 client.16722 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:03:16.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.478+0000 7f2cd1a54740 1 -- 172.21.15.169:0/2507080028 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f2cc00a39b0 con 0x5606043191d0 2024-03-28T01:03:16.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.480+0000 7f2cccbfd640 1 --2- 172.21.15.169:0/2507080028 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5606043191d0 0x56060431b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.481+0000 7f2ca7fff640 10 client.16722 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:03:16.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.481+0000 7f2cccbfd640 1 --2- 172.21.15.169:0/2507080028 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5606043191d0 0x56060431b5f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cfc2f29750810be5 server_cookie=b21a826d8c6a8008 in_seq=0 out_seq=0 2024-03-28T01:03:16.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.482+0000 7f2ca7fff640 1 -- 172.21.15.169:0/2507080028 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f2cc00a39b0 con 0x5606043191d0 2024-03-28T01:03:16.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.482+0000 7f2ca7fff640 10 client.16722 handle_command_reply: tid=0 2024-03-28T01:03:16.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.482+0000 7f2cd1a54740 1 client.16722 shutdown 2024-03-28T01:03:16.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.482+0000 7f2cd1a54740 1 -- 172.21.15.169:0/2507080028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042ac1f0 msgr2=0x5606042b6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.482+0000 7f2cd1a54740 1 --2- 172.21.15.169:0/2507080028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042ac1f0 0x5606042b6f90 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009830 tx=0x7f2cb8034ad0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.483+0000 7f2cd1a54740 1 -- 172.21.15.169:0/2507080028 shutdown_connections 2024-03-28T01:03:16.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.483+0000 7f2cd1a54740 1 --2- 172.21.15.169:0/2507080028 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5606043191d0 0x56060431b5f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.483+0000 7f2cd1a54740 1 --2- 172.21.15.169:0/2507080028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606042ac1f0 0x5606042b6f90 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.483+0000 7f2cd1a54740 1 --2- 172.21.15.169:0/2507080028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606042a7080 0x5606042b5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.483+0000 7f2cd1a54740 1 --2- 172.21.15.169:0/2507080028 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606042a6690 0x5606042b5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.483+0000 7f2cd1a54740 1 -- 172.21.15.169:0/2507080028 >> 172.21.15.169:0/2507080028 conn(0x5606042a1c00 msgr2=0x5606042a4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:16.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.483+0000 7f2ca7fff640 0 client.16722 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:03:16.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.483+0000 7f2cd1a54740 1 -- 172.21.15.169:0/2507080028 shutdown_connections 2024-03-28T01:03:16.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.483+0000 7f2cd1a54740 1 -- 172.21.15.169:0/2507080028 wait complete. 2024-03-28T01:03:16.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.483+0000 7f2cd1a54740 20 client.16722 trim_cache size 0 max 16384 2024-03-28T01:03:16.487 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:03:16.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.485+0000 7f2ccee61640 1 -- 172.21.15.169:0/2111481158 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2cb00a28d0 msgr2=0x7f2cb00a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.485+0000 7f2ccee61640 1 --2- 172.21.15.169:0/2111481158 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2cb00a28d0 0x7f2cb00a4d90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009860 tx=0x7f2cb8045000 comp rx=0 tx=0).stop 2024-03-28T01:03:16.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.485+0000 7f2ccee61640 1 -- 172.21.15.169:0/2111481158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2cc810b010 msgr2=0x7f2cc8115330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.485+0000 7f2ccee61640 1 --2- 172.21.15.169:0/2111481158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2cc810b010 0x7f2cc8115330 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2cc000efb0 tx=0x7f2cc000c800 comp rx=0 tx=0).stop 2024-03-28T01:03:16.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.485+0000 7f2ccee61640 1 -- 172.21.15.169:0/2111481158 shutdown_connections 2024-03-28T01:03:16.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.485+0000 7f2ccee61640 1 --2- 172.21.15.169:0/2111481158 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2cb00a28d0 0x7f2cb00a4d90 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.485+0000 7f2ccee61640 1 --2- 172.21.15.169:0/2111481158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cc810d9c0 0x7f2cc81b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.485+0000 7f2ccee61640 1 --2- 172.21.15.169:0/2111481158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2cc810b010 0x7f2cc8115330 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.485+0000 7f2ccee61640 1 --2- 172.21.15.169:0/2111481158 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cc81086a0 0x7f2cc8114df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.485+0000 7f2ccee61640 1 -- 172.21.15.169:0/2111481158 >> 172.21.15.169:0/2111481158 conn(0x7f2cc806e500 msgr2=0x7f2cc8109ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:16.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.486+0000 7f2ccee61640 1 -- 172.21.15.169:0/2111481158 shutdown_connections 2024-03-28T01:03:16.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.486+0000 7f2ccee61640 1 -- 172.21.15.169:0/2111481158 wait complete. 2024-03-28T01:03:16.509 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:03:16.509 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T01:03:16.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.642+0000 7f172764b640 1 -- 172.21.15.169:0/3230260640 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1720104a50 msgr2=0x7f1720106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.642+0000 7f172764b640 1 --2- 172.21.15.169:0/3230260640 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1720104a50 0x7f1720106e40 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f1710009900 tx=0x7f171002bc30 comp rx=0 tx=0).stop 2024-03-28T01:03:16.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.642+0000 7f172764b640 1 -- 172.21.15.169:0/3230260640 shutdown_connections 2024-03-28T01:03:16.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.642+0000 7f172764b640 1 --2- 172.21.15.169:0/3230260640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1720107380 0x7f17201097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.642+0000 7f172764b640 1 --2- 172.21.15.169:0/3230260640 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1720104a50 0x7f1720106e40 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.642+0000 7f172764b640 1 --2- 172.21.15.169:0/3230260640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1720102120 0x7f1720104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.642+0000 7f172764b640 1 -- 172.21.15.169:0/3230260640 >> 172.21.15.169:0/3230260640 conn(0x7f17200fbb30 msgr2=0x7f17200fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:16.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.642+0000 7f172764b640 1 -- 172.21.15.169:0/3230260640 shutdown_connections 2024-03-28T01:03:16.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.642+0000 7f172764b640 1 -- 172.21.15.169:0/3230260640 wait complete. 2024-03-28T01:03:16.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.642+0000 7f172764b640 1 Processor -- start 2024-03-28T01:03:16.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.642+0000 7f172764b640 1 -- start start 2024-03-28T01:03:16.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.643+0000 7f172764b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1720102120 0x7f1720195800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.643+0000 7f172764b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1720104a50 0x7f1720195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.643+0000 7f172764b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1720107380 0x7f172019cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.643+0000 7f172764b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f172010b160 con 0x7f1720102120 2024-03-28T01:03:16.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.643+0000 7f172764b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f172010afe0 con 0x7f1720107380 2024-03-28T01:03:16.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.643+0000 7f172764b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f172010b2e0 con 0x7f1720104a50 2024-03-28T01:03:16.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.643+0000 7f17253e7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1720102120 0x7f1720195800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.643+0000 7f17253e7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1720102120 0x7f1720195800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53128/0 (socket says 172.21.15.169:53128) 2024-03-28T01:03:16.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.643+0000 7f17253e7640 1 -- 172.21.15.169:0/3374661218 learned_addr learned my addr 172.21.15.169:0/3374661218 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:16.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.643+0000 7f1724be6640 1 --2- 172.21.15.169:0/3374661218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1720104a50 0x7f1720195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.643+0000 7f17253e7640 1 -- 172.21.15.169:0/3374661218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1720104a50 msgr2=0x7f1720195d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.643+0000 7f17253e7640 1 --2- 172.21.15.169:0/3374661218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1720104a50 0x7f1720195d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.643+0000 7f17253e7640 1 -- 172.21.15.169:0/3374661218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1720107380 msgr2=0x7f172019cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.643+0000 7f17253e7640 1 --2- 172.21.15.169:0/3374661218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1720107380 0x7f172019cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.643+0000 7f17253e7640 1 -- 172.21.15.169:0/3374661218 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f172019d420 con 0x7f1720102120 2024-03-28T01:03:16.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.643+0000 7f17253e7640 1 --2- 172.21.15.169:0/3374661218 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1720102120 0x7f1720195800 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f171c009be0 tx=0x7f171c0071c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:16.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.644+0000 7f17167fc640 1 -- 172.21.15.169:0/3374661218 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171c007e00 con 0x7f1720102120 2024-03-28T01:03:16.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.644+0000 7f17167fc640 1 -- 172.21.15.169:0/3374661218 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f171c01e070 con 0x7f1720102120 2024-03-28T01:03:16.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.644+0000 7f17167fc640 1 -- 172.21.15.169:0/3374661218 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171c018890 con 0x7f1720102120 2024-03-28T01:03:16.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.644+0000 7f172764b640 1 -- 172.21.15.169:0/3374661218 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f172019d710 con 0x7f1720102120 2024-03-28T01:03:16.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.644+0000 7f172764b640 1 -- 172.21.15.169:0/3374661218 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f172019dc30 con 0x7f1720102120 2024-03-28T01:03:16.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.646+0000 7f17167fc640 1 -- 172.21.15.169:0/3374661218 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f171c019b60 con 0x7f1720102120 2024-03-28T01:03:16.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.647+0000 7f172a238740 1 Processor -- start 2024-03-28T01:03:16.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.647+0000 7f172a238740 1 -- start start 2024-03-28T01:03:16.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.647+0000 7f172a238740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d5fea0 0x560733d602c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.647+0000 7f172a238740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560733d60890 0x560733d65390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.647+0000 7f172a238740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d65a00 0x560733d67e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.647+0000 7f172a238740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x560733d1a610 con 0x560733d65a00 2024-03-28T01:03:16.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.647+0000 7f172a238740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x560733cbd3c0 con 0x560733d60890 2024-03-28T01:03:16.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.647+0000 7f172a238740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560733c65d60 con 0x560733d5fea0 2024-03-28T01:03:16.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.648+0000 7f1725be8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d5fea0 0x560733d602c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.648+0000 7f1724be6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d65a00 0x560733d67e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.648+0000 7f17167fc640 1 --2- 172.21.15.169:0/3374661218 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f170c0a2800 0x7f170c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.648+0000 7f1724be6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d65a00 0x560733d67e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53132/0 (socket says 172.21.15.169:53132) 2024-03-28T01:03:16.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.648+0000 7f1725be8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d5fea0 0x560733d602c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:46290/0 (socket says 172.21.15.169:46290) 2024-03-28T01:03:16.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.648+0000 7f1724be6640 1 -- 172.21.15.169:0/3250318910 learned_addr learned my addr 172.21.15.169:0/3250318910 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:16.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.648+0000 7f1725be8640 1 --2- 172.21.15.169:0/3374661218 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f170c0a2800 0x7f170c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.648+0000 7f17167fc640 1 -- 172.21.15.169:0/3374661218 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f171c0d2780 con 0x7f1720102120 2024-03-28T01:03:16.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.648+0000 7f1724be6640 1 -- 172.21.15.169:0/3250318910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d5fea0 msgr2=0x560733d602c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.648+0000 7f1724be6640 1 --2- 172.21.15.169:0/3250318910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d5fea0 0x560733d602c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.648+0000 7f1724be6640 1 -- 172.21.15.169:0/3250318910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560733d60890 msgr2=0x560733d65390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.648+0000 7f17253e7640 1 --2- 172.21.15.169:0/3250318910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560733d60890 0x560733d65390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.648+0000 7f1724be6640 1 --2- 172.21.15.169:0/3250318910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560733d60890 0x560733d65390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.648+0000 7f1724be6640 1 -- 172.21.15.169:0/3250318910 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607339d13f0 con 0x560733d65a00 2024-03-28T01:03:16.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.648+0000 7f1725be8640 1 --2- 172.21.15.169:0/3250318910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d5fea0 0x560733d602c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:03:16.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.649+0000 7f1725be8640 1 --2- 172.21.15.169:0/3374661218 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f170c0a2800 0x7f170c0a4cc0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1718005fd0 tx=0x7f1718009aa0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:16.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.649+0000 7f17253e7640 1 --2- 172.21.15.169:0/3250318910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560733d60890 0x560733d65390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:03:16.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.649+0000 7f1724be6640 1 --2- 172.21.15.169:0/3250318910 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d65a00 0x560733d67e90 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f1720069360 tx=0x7f1710009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac8a071101867c16 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:16.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.649+0000 7f170b7fe640 1 -- 172.21.15.169:0/3250318910 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171002cb80 con 0x560733d65a00 2024-03-28T01:03:16.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.649+0000 7f170b7fe640 1 -- 172.21.15.169:0/3250318910 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1710005040 con 0x560733d65a00 2024-03-28T01:03:16.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.649+0000 7f170b7fe640 1 -- 172.21.15.169:0/3250318910 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1710004920 con 0x560733d65a00 2024-03-28T01:03:16.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.650+0000 7f172a238740 1 -- 172.21.15.169:0/3250318910 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d65a00 msgr2=0x560733d67e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.650+0000 7f172a238740 1 --2- 172.21.15.169:0/3250318910 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d65a00 0x560733d67e90 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f1720069360 tx=0x7f1710009560 comp rx=0 tx=0).stop 2024-03-28T01:03:16.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.650+0000 7f172a238740 1 -- 172.21.15.169:0/3250318910 shutdown_connections 2024-03-28T01:03:16.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.650+0000 7f172a238740 1 --2- 172.21.15.169:0/3250318910 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d65a00 0x560733d67e90 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.650+0000 7f172a238740 1 --2- 172.21.15.169:0/3250318910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560733d60890 0x560733d65390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.650+0000 7f172a238740 1 --2- 172.21.15.169:0/3250318910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d5fea0 0x560733d602c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.650+0000 7f172a238740 1 -- 172.21.15.169:0/3250318910 >> 172.21.15.169:0/3250318910 conn(0x560733d5b410 msgr2=0x560733d5b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:16.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.651+0000 7f172a238740 1 -- 172.21.15.169:0/3250318910 shutdown_connections 2024-03-28T01:03:16.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.651+0000 7f172a238740 1 -- 172.21.15.169:0/3250318910 wait complete. 2024-03-28T01:03:16.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.651+0000 7f172a238740 1 -- start start 2024-03-28T01:03:16.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.651+0000 7f172a238740 1 Processor -- start 2024-03-28T01:03:16.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.652+0000 7f172a238740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d5fea0 0x560733d6a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.652+0000 7f172a238740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560733d60890 0x560733d6a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.652+0000 7f172a238740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d65a00 0x560733d71780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.652+0000 7f172a238740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x560733c79240 con 0x560733d5fea0 2024-03-28T01:03:16.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.652+0000 7f172a238740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x560733cb9460 con 0x560733d60890 2024-03-28T01:03:16.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.652+0000 7f172a238740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560733d68950 con 0x560733d65a00 2024-03-28T01:03:16.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.652+0000 7f172a238740 10 client.0 _ll_register_callbacks cb 0x560733c78c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:16.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.652+0000 7f17253e7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d5fea0 0x560733d6a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.652+0000 7f17253e7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d5fea0 0x560733d6a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53138/0 (socket says 172.21.15.169:53138) 2024-03-28T01:03:16.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.652+0000 7f17253e7640 1 -- 172.21.15.169:0/554403253 learned_addr learned my addr 172.21.15.169:0/554403253 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:16.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.652+0000 7f1725be8640 1 --2- 172.21.15.169:0/554403253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d65a00 0x560733d71780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.652+0000 7f17253e7640 1 -- 172.21.15.169:0/554403253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d65a00 msgr2=0x560733d71780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.652+0000 7f17253e7640 1 --2- 172.21.15.169:0/554403253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d65a00 0x560733d71780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.652+0000 7f17253e7640 1 -- 172.21.15.169:0/554403253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560733d60890 msgr2=0x560733d6a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.652+0000 7f17253e7640 1 --2- 172.21.15.169:0/554403253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560733d60890 0x560733d6a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.652+0000 7f17253e7640 1 -- 172.21.15.169:0/554403253 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560733a9b7d0 con 0x560733d5fea0 2024-03-28T01:03:16.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.652+0000 7f17253e7640 1 --2- 172.21.15.169:0/554403253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d5fea0 0x560733d6a200 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f171c09b1c0 tx=0x7f171c09f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ebb7c72b2f523a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:16.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.652+0000 7f1708ff9640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:03:16.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.653+0000 7f1708ff9640 1 -- 172.21.15.169:0/554403253 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171c09e3b0 con 0x560733d5fea0 2024-03-28T01:03:16.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.653+0000 7f172a238740 1 -- 172.21.15.169:0/554403253 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1710005040 con 0x560733d5fea0 2024-03-28T01:03:16.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.653+0000 7f1708ff9640 1 -- 172.21.15.169:0/554403253 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f171c09aab0 con 0x560733d5fea0 2024-03-28T01:03:16.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.653+0000 7f1708ff9640 1 -- 172.21.15.169:0/554403253 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f171c0a2eb0 con 0x560733d5fea0 2024-03-28T01:03:16.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.653+0000 7f172a238740 10 client.16734 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:16.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.653+0000 7f172a238740 1 -- 172.21.15.169:0/554403253 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x560733d74b80 con 0x560733d5fea0 2024-03-28T01:03:16.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.653+0000 7f1708ff9640 1 -- 172.21.15.169:0/554403253 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1710005040 con 0x560733d5fea0 2024-03-28T01:03:16.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.653+0000 7f1708ff9640 1 -- 172.21.15.169:0/554403253 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f171c089070 con 0x560733d5fea0 2024-03-28T01:03:16.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.653+0000 7f172a238740 10 client.16734 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:16.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.653+0000 7f172a238740 10 client.16734 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:16.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.654+0000 7f172a238740 20 client.16734 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:16.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.654+0000 7f172a238740 1 --2- 172.21.15.169:0/554403253 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560733d765f0 0x560733d78ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.654+0000 7f172a238740 4 client.16734 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:03:16.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.654+0000 7f172a238740 1 -- 172.21.15.169:0/554403253 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560733cca900 con 0x560733d765f0 2024-03-28T01:03:16.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.655+0000 7f1724be6640 1 --2- 172.21.15.169:0/554403253 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560733d765f0 0x560733d78ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.656+0000 7f1708ff9640 10 client.16734 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:16.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.656+0000 7f1724be6640 1 --2- 172.21.15.169:0/554403253 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560733d765f0 0x560733d78ab0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30243870735a96b3 server_cookie=56053fa38d166e98 in_seq=0 out_seq=0 2024-03-28T01:03:16.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.658+0000 7f1708ff9640 1 -- 172.21.15.169:0/554403253 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560733cca900 con 0x560733d765f0 2024-03-28T01:03:16.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.658+0000 7f1708ff9640 10 client.16734 handle_command_reply: tid=0 2024-03-28T01:03:16.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.658+0000 7f172a238740 1 client.16734 shutdown 2024-03-28T01:03:16.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.658+0000 7f172a238740 1 -- 172.21.15.169:0/554403253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d5fea0 msgr2=0x560733d6a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.658+0000 7f172a238740 1 --2- 172.21.15.169:0/554403253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d5fea0 0x560733d6a200 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f171c09b1c0 tx=0x7f171c09f460 comp rx=0 tx=0).stop 2024-03-28T01:03:16.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.658+0000 7f172a238740 1 -- 172.21.15.169:0/554403253 shutdown_connections 2024-03-28T01:03:16.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.658+0000 7f172a238740 1 --2- 172.21.15.169:0/554403253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d65a00 0x560733d71780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.658+0000 7f172a238740 1 --2- 172.21.15.169:0/554403253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560733d60890 0x560733d6a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.658+0000 7f172a238740 1 --2- 172.21.15.169:0/554403253 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560733d765f0 0x560733d78ab0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.658+0000 7f172a238740 1 --2- 172.21.15.169:0/554403253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d5fea0 0x560733d6a200 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.658+0000 7f172a238740 1 -- 172.21.15.169:0/554403253 >> 172.21.15.169:0/554403253 conn(0x560733d5b410 msgr2=0x560733d51e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:16.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.658+0000 7f1708ff9640 0 client.16734 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:16.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.658+0000 7f172a238740 1 -- 172.21.15.169:0/554403253 shutdown_connections 2024-03-28T01:03:16.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.658+0000 7f172a238740 1 -- 172.21.15.169:0/554403253 wait complete. 2024-03-28T01:03:16.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.659+0000 7f172a238740 20 client.16734 trim_cache size 0 max 16384 2024-03-28T01:03:16.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.683+0000 7f172a238740 1 Processor -- start 2024-03-28T01:03:16.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.683+0000 7f172a238740 1 -- start start 2024-03-28T01:03:16.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.683+0000 7f172a238740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560733d5fea0 0x560733d69e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.683+0000 7f172a238740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d60890 0x560733d71740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.683+0000 7f172a238740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d65a00 0x560733d6a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.683+0000 7f172a238740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17201aa450 con 0x560733d60890 2024-03-28T01:03:16.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.683+0000 7f172a238740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17201a7f50 con 0x560733d5fea0 2024-03-28T01:03:16.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.683+0000 7f172a238740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17201a5ff0 con 0x560733d65a00 2024-03-28T01:03:16.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.683+0000 7f1725be8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d65a00 0x560733d6a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.683+0000 7f1725be8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d65a00 0x560733d6a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:46306/0 (socket says 172.21.15.169:46306) 2024-03-28T01:03:16.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.683+0000 7f1725be8640 1 -- 172.21.15.169:0/3980219640 learned_addr learned my addr 172.21.15.169:0/3980219640 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:16.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.683+0000 7f1724be6640 1 --2- 172.21.15.169:0/3980219640 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d60890 0x560733d71740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.684+0000 7f17253e7640 1 --2- 172.21.15.169:0/3980219640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560733d5fea0 0x560733d69e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.684+0000 7f1725be8640 1 -- 172.21.15.169:0/3980219640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560733d5fea0 msgr2=0x560733d69e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.684+0000 7f1725be8640 1 --2- 172.21.15.169:0/3980219640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560733d5fea0 0x560733d69e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.684+0000 7f1725be8640 1 -- 172.21.15.169:0/3980219640 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d60890 msgr2=0x560733d71740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.684+0000 7f1725be8640 1 --2- 172.21.15.169:0/3980219640 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d60890 0x560733d71740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.684+0000 7f1725be8640 1 -- 172.21.15.169:0/3980219640 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560733d68250 con 0x560733d65a00 2024-03-28T01:03:16.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.684+0000 7f17253e7640 1 --2- 172.21.15.169:0/3980219640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560733d5fea0 0x560733d69e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:03:16.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.684+0000 7f1725be8640 1 --2- 172.21.15.169:0/3980219640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d65a00 0x560733d6a350 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1718010d80 tx=0x7f1718004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8320992915ce92e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:16.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.684+0000 7f170affd640 1 -- 172.21.15.169:0/3980219640 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1718010db0 con 0x560733d65a00 2024-03-28T01:03:16.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.684+0000 7f170affd640 1 -- 172.21.15.169:0/3980219640 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17180045c0 con 0x560733d65a00 2024-03-28T01:03:16.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.684+0000 7f170affd640 1 -- 172.21.15.169:0/3980219640 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1718005830 con 0x560733d65a00 2024-03-28T01:03:16.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.685+0000 7f172a238740 1 -- 172.21.15.169:0/3980219640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d65a00 msgr2=0x560733d6a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.685+0000 7f172a238740 1 --2- 172.21.15.169:0/3980219640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d65a00 0x560733d6a350 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1718010d80 tx=0x7f1718004a40 comp rx=0 tx=0).stop 2024-03-28T01:03:16.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.685+0000 7f172a238740 1 -- 172.21.15.169:0/3980219640 shutdown_connections 2024-03-28T01:03:16.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.685+0000 7f172a238740 1 --2- 172.21.15.169:0/3980219640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d65a00 0x560733d6a350 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.685+0000 7f172a238740 1 --2- 172.21.15.169:0/3980219640 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d60890 0x560733d71740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.685+0000 7f172a238740 1 --2- 172.21.15.169:0/3980219640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560733d5fea0 0x560733d69e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.685+0000 7f172a238740 1 -- 172.21.15.169:0/3980219640 >> 172.21.15.169:0/3980219640 conn(0x560733d5b410 msgr2=0x560733d61d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:16.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.685+0000 7f172a238740 1 -- 172.21.15.169:0/3980219640 shutdown_connections 2024-03-28T01:03:16.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.685+0000 7f172a238740 1 -- 172.21.15.169:0/3980219640 wait complete. 2024-03-28T01:03:16.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.686+0000 7f172a238740 1 -- start start 2024-03-28T01:03:16.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.686+0000 7f172a238740 1 Processor -- start 2024-03-28T01:03:16.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.686+0000 7f172a238740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d5fea0 0x560733d6e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.686+0000 7f172a238740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d60890 0x560733d6ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.687+0000 7f172a238740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560733d65a00 0x560733d6fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.687+0000 7f172a238740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17201a65f0 con 0x560733d60890 2024-03-28T01:03:16.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.687+0000 7f172a238740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17201a78f0 con 0x560733d65a00 2024-03-28T01:03:16.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.687+0000 7f172a238740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17201a6c30 con 0x560733d5fea0 2024-03-28T01:03:16.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.687+0000 7f172a238740 10 client.0 _ll_register_callbacks cb 0x7f171c09e990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:16.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.687+0000 7f1724be6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d60890 0x560733d6ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.687+0000 7f1724be6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d60890 0x560733d6ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53156/0 (socket says 172.21.15.169:53156) 2024-03-28T01:03:16.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.687+0000 7f1724be6640 1 -- 172.21.15.169:0/2052065007 learned_addr learned my addr 172.21.15.169:0/2052065007 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:16.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.687+0000 7f1724be6640 1 -- 172.21.15.169:0/2052065007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d5fea0 msgr2=0x560733d6e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:16.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.687+0000 7f1724be6640 1 --2- 172.21.15.169:0/2052065007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d5fea0 0x560733d6e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.687+0000 7f1724be6640 1 -- 172.21.15.169:0/2052065007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560733d65a00 msgr2=0x560733d6fd70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:03:16.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.687+0000 7f1724be6640 1 --2- 172.21.15.169:0/2052065007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560733d65a00 0x560733d6fd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.687+0000 7f1724be6640 1 -- 172.21.15.169:0/2052065007 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560733d70360 con 0x560733d60890 2024-03-28T01:03:16.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.688+0000 7f1724be6640 1 --2- 172.21.15.169:0/2052065007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d60890 0x560733d6ed60 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f1710009900 tx=0x7f171002c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1547e8a9f2d8969f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:16.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.688+0000 7f1708ff9640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:03:16.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.688+0000 7f1708ff9640 1 -- 172.21.15.169:0/2052065007 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1710005d50 con 0x560733d60890 2024-03-28T01:03:16.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.688+0000 7f1708ff9640 1 -- 172.21.15.169:0/2052065007 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f1710005240 con 0x560733d60890 2024-03-28T01:03:16.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.688+0000 7f172a238740 1 -- 172.21.15.169:0/2052065007 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f17180045c0 con 0x560733d60890 2024-03-28T01:03:16.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.689+0000 7f1708ff9640 1 -- 172.21.15.169:0/2052065007 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f171005b050 con 0x560733d60890 2024-03-28T01:03:16.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.689+0000 7f172a238740 10 client.16740 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:16.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.689+0000 7f172a238740 1 -- 172.21.15.169:0/2052065007 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x560733d71640 con 0x560733d60890 2024-03-28T01:03:16.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.689+0000 7f1708ff9640 1 -- 172.21.15.169:0/2052065007 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1710005480 con 0x560733d60890 2024-03-28T01:03:16.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.690+0000 7f1708ff9640 1 -- 172.21.15.169:0/2052065007 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f171004d590 con 0x560733d60890 2024-03-28T01:03:16.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.690+0000 7f172a238740 10 client.16740 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:16.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.690+0000 7f172a238740 10 client.16740 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:16.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.690+0000 7f172a238740 20 client.16740 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:16.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.690+0000 7f172a238740 1 --2- 172.21.15.169:0/2052065007 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560733dd1f80 0x560733dd4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:16.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.690+0000 7f172a238740 4 client.16740 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:03:16.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.690+0000 7f172a238740 1 -- 172.21.15.169:0/2052065007 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f17180045c0 con 0x560733dd1f80 2024-03-28T01:03:16.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.691+0000 7f17253e7640 1 --2- 172.21.15.169:0/2052065007 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560733dd1f80 0x560733dd4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:16.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.691+0000 7f1708ff9640 10 client.16740 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:16.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.691+0000 7f17253e7640 1 --2- 172.21.15.169:0/2052065007 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560733dd1f80 0x560733dd4370 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c237ae2ca2513ba server_cookie=18404af7e8f6c70 in_seq=0 out_seq=0 2024-03-28T01:03:16.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.692+0000 7f1708ff9640 1 -- 172.21.15.169:0/2052065007 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f17180045c0 con 0x560733dd1f80 2024-03-28T01:03:16.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.692+0000 7f1708ff9640 10 client.16740 handle_command_reply: tid=0 2024-03-28T01:03:16.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.692+0000 7f172a238740 1 client.16740 shutdown 2024-03-28T01:03:16.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.692+0000 7f172a238740 1 -- 172.21.15.169:0/2052065007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d60890 msgr2=0x560733d6ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.692+0000 7f172a238740 1 --2- 172.21.15.169:0/2052065007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d60890 0x560733d6ed60 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f1710009900 tx=0x7f171002c660 comp rx=0 tx=0).stop 2024-03-28T01:03:16.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.692+0000 7f172a238740 1 -- 172.21.15.169:0/2052065007 shutdown_connections 2024-03-28T01:03:16.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.692+0000 7f172a238740 1 --2- 172.21.15.169:0/2052065007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560733d65a00 0x560733d6fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.692+0000 7f172a238740 1 --2- 172.21.15.169:0/2052065007 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560733dd1f80 0x560733dd4370 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.692+0000 7f172a238740 1 --2- 172.21.15.169:0/2052065007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560733d60890 0x560733d6ed60 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.692+0000 7f1708ff9640 0 client.16740 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:16.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.692+0000 7f172a238740 1 --2- 172.21.15.169:0/2052065007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560733d5fea0 0x560733d6e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.692+0000 7f172a238740 1 -- 172.21.15.169:0/2052065007 >> 172.21.15.169:0/2052065007 conn(0x560733d5b410 msgr2=0x560733d69df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:16.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.692+0000 7f172a238740 1 -- 172.21.15.169:0/2052065007 shutdown_connections 2024-03-28T01:03:16.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.693+0000 7f172a238740 1 -- 172.21.15.169:0/2052065007 wait complete. 2024-03-28T01:03:16.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.693+0000 7f172a238740 20 client.16740 trim_cache size 0 max 16384 2024-03-28T01:03:16.698 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:03:16.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.694+0000 7f172764b640 1 -- 172.21.15.169:0/3374661218 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f170c0a2800 msgr2=0x7f170c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.694+0000 7f172764b640 1 --2- 172.21.15.169:0/3374661218 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f170c0a2800 0x7f170c0a4cc0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1718005fd0 tx=0x7f1718009aa0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.694+0000 7f172764b640 1 -- 172.21.15.169:0/3374661218 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1720102120 msgr2=0x7f1720195800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:16.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.694+0000 7f172764b640 1 --2- 172.21.15.169:0/3374661218 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1720102120 0x7f1720195800 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f171c009be0 tx=0x7f171c0071c0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.694+0000 7f172764b640 1 -- 172.21.15.169:0/3374661218 shutdown_connections 2024-03-28T01:03:16.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.694+0000 7f172764b640 1 --2- 172.21.15.169:0/3374661218 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f170c0a2800 0x7f170c0a4cc0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.694+0000 7f172764b640 1 --2- 172.21.15.169:0/3374661218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1720107380 0x7f172019cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.694+0000 7f172764b640 1 --2- 172.21.15.169:0/3374661218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1720104a50 0x7f1720195d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.694+0000 7f172764b640 1 --2- 172.21.15.169:0/3374661218 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1720102120 0x7f1720195800 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:16.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.694+0000 7f172764b640 1 -- 172.21.15.169:0/3374661218 >> 172.21.15.169:0/3374661218 conn(0x7f17200fbb30 msgr2=0x7f1720102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:16.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.694+0000 7f172764b640 1 -- 172.21.15.169:0/3374661218 shutdown_connections 2024-03-28T01:03:16.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:16.694+0000 7f172764b640 1 -- 172.21.15.169:0/3374661218 wait complete. 2024-03-28T01:03:16.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:16 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4077812698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:03:16.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:16 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/4077812698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:03:16.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:16 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/4077812698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:03:17.723 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T01:03:17.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T01:03:17.724 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T01:03:17.737 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.733+0000 7f242f459640 1 -- 172.21.15.169:0/34374378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24281028f0 msgr2=0x7f2428102d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:17.738 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.733+0000 7f242f459640 1 --2- 172.21.15.169:0/34374378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24281028f0 0x7f2428102d70 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f241801a910 tx=0x7f241803cca0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.738 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.733+0000 7f242f459640 1 -- 172.21.15.169:0/34374378 shutdown_connections 2024-03-28T01:03:17.738 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.733+0000 7f242f459640 1 --2- 172.21.15.169:0/34374378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24281032b0 0x7f2428109b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.738 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.733+0000 7f242f459640 1 --2- 172.21.15.169:0/34374378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24281028f0 0x7f2428102d70 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.738 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.733+0000 7f242f459640 1 --2- 172.21.15.169:0/34374378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2428101700 0x7f2428101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.738 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.733+0000 7f242f459640 1 -- 172.21.15.169:0/34374378 >> 172.21.15.169:0/34374378 conn(0x7f24280fcc70 msgr2=0x7f24280ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:17.739 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.736+0000 7f242f459640 1 -- 172.21.15.169:0/34374378 shutdown_connections 2024-03-28T01:03:17.739 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.736+0000 7f242f459640 1 -- 172.21.15.169:0/34374378 wait complete. 2024-03-28T01:03:17.740 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.737+0000 7f242f459640 1 Processor -- start 2024-03-28T01:03:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.737+0000 7f242f459640 1 -- start start 2024-03-28T01:03:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.738+0000 7f242f459640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2428101700 0x7f2428073400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.738+0000 7f242f459640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24281028f0 0x7f2428073940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.738+0000 7f242f459640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24281032b0 0x7f2428074980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.738+0000 7f242f459640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242806dd70 con 0x7f24281028f0 2024-03-28T01:03:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.738+0000 7f242f459640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242806dbf0 con 0x7f2428101700 2024-03-28T01:03:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.738+0000 7f242f459640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242806def0 con 0x7f24281032b0 2024-03-28T01:03:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.738+0000 7f242c9f4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24281028f0 0x7f2428073940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.738+0000 7f242c9f4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24281028f0 0x7f2428073940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53162/0 (socket says 172.21.15.169:53162) 2024-03-28T01:03:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.738+0000 7f242c9f4640 1 -- 172.21.15.169:0/2031692027 learned_addr learned my addr 172.21.15.169:0/2031692027 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.739+0000 7f242c9f4640 1 -- 172.21.15.169:0/2031692027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24281032b0 msgr2=0x7f2428074980 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:03:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.739+0000 7f242c9f4640 1 --2- 172.21.15.169:0/2031692027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24281032b0 0x7f2428074980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.739+0000 7f242c9f4640 1 -- 172.21.15.169:0/2031692027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2428101700 msgr2=0x7f2428073400 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:03:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.739+0000 7f242c9f4640 1 --2- 172.21.15.169:0/2031692027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2428101700 0x7f2428073400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.739+0000 7f242c9f4640 1 -- 172.21.15.169:0/2031692027 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2428074ff0 con 0x7f24281028f0 2024-03-28T01:03:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.740+0000 7f242c9f4640 1 --2- 172.21.15.169:0/2031692027 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24281028f0 0x7f2428073940 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f241803d2b0 tx=0x7f24180459f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.740+0000 7f24267fc640 1 -- 172.21.15.169:0/2031692027 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2418045e20 con 0x7f24281028f0 2024-03-28T01:03:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.741+0000 7f24267fc640 1 -- 172.21.15.169:0/2031692027 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2418052070 con 0x7f24281028f0 2024-03-28T01:03:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.741+0000 7f24267fc640 1 -- 172.21.15.169:0/2031692027 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24180046c0 con 0x7f24281028f0 2024-03-28T01:03:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.744+0000 7f242f459640 1 -- 172.21.15.169:0/2031692027 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2428075280 con 0x7f24281028f0 2024-03-28T01:03:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.744+0000 7f242f459640 1 -- 172.21.15.169:0/2031692027 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2428075770 con 0x7f24281028f0 2024-03-28T01:03:17.751 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.749+0000 7f2432042740 1 Processor -- start 2024-03-28T01:03:17.754 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.751+0000 7f24267fc640 1 -- 172.21.15.169:0/2031692027 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2418004050 con 0x7f24281028f0 2024-03-28T01:03:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.757+0000 7f24267fc640 1 --2- 172.21.15.169:0/2031692027 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f24100a2a10 0x7f24100a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.757+0000 7f24267fc640 1 -- 172.21.15.169:0/2031692027 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2418100f30 con 0x7f24281028f0 2024-03-28T01:03:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.758+0000 7f2432042740 1 -- start start 2024-03-28T01:03:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.758+0000 7f2432042740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa517920 0x55b5aa517d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.758+0000 7f2432042740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5aa518310 0x55b5aa51ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.758+0000 7f2432042740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5aa51d480 0x55b5aa51f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.758+0000 7f2432042740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5aa47cb10 con 0x55b5aa517920 2024-03-28T01:03:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.758+0000 7f2432042740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5aa425900 con 0x55b5aa51d480 2024-03-28T01:03:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.758+0000 7f2432042740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5aa478bb0 con 0x55b5aa518310 2024-03-28T01:03:17.762 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.759+0000 7f242c9f4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5aa518310 0x55b5aa51ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:17.762 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.759+0000 7f242c9f4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5aa518310 0x55b5aa51ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:46328/0 (socket says 172.21.15.169:46328) 2024-03-28T01:03:17.762 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.759+0000 7f242c9f4640 1 -- 172.21.15.169:0/3000352559 learned_addr learned my addr 172.21.15.169:0/3000352559 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:17.765 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.762+0000 7f242c9f4640 1 -- 172.21.15.169:0/3000352559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5aa51d480 msgr2=0x55b5aa51f910 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:03:17.765 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.762+0000 7f242c9f4640 1 --2- 172.21.15.169:0/3000352559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5aa51d480 0x55b5aa51f910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.765 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.762+0000 7f242c9f4640 1 -- 172.21.15.169:0/3000352559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa517920 msgr2=0x55b5aa517d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:17.765 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.762+0000 7f242c9f4640 1 --2- 172.21.15.169:0/3000352559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa517920 0x55b5aa517d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.765 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.762+0000 7f242c9f4640 1 -- 172.21.15.169:0/3000352559 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5aa191210 con 0x55b5aa518310 2024-03-28T01:03:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.764+0000 7f242c9f4640 1 --2- 172.21.15.169:0/3000352559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5aa518310 0x55b5aa51ce10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f24180fe5a0 tx=0x7f2418003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e0ab2bb688e5fa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.764+0000 7f242d9f6640 1 --2- 172.21.15.169:0/2031692027 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f24100a2a10 0x7f24100a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:17.768 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.765+0000 7f242d9f6640 1 --2- 172.21.15.169:0/2031692027 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f24100a2a10 0x7f24100a4ed0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f242000ab40 tx=0x7f2420009290 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:17.771 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.768+0000 7f240f7fe640 1 -- 172.21.15.169:0/3000352559 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24180cd4d0 con 0x55b5aa518310 2024-03-28T01:03:17.771 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.768+0000 7f240f7fe640 1 -- 172.21.15.169:0/3000352559 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24180d1620 con 0x55b5aa518310 2024-03-28T01:03:17.773 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.770+0000 7f2432042740 1 -- 172.21.15.169:0/3000352559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5aa518310 msgr2=0x55b5aa51ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:17.773 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.770+0000 7f2432042740 1 --2- 172.21.15.169:0/3000352559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5aa518310 0x55b5aa51ce10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f24180fe5a0 tx=0x7f2418003c10 comp rx=0 tx=0).stop 2024-03-28T01:03:17.778 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.775+0000 7f2432042740 1 -- 172.21.15.169:0/3000352559 shutdown_connections 2024-03-28T01:03:17.779 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.775+0000 7f2432042740 1 --2- 172.21.15.169:0/3000352559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5aa51d480 0x55b5aa51f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.779 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.775+0000 7f2432042740 1 --2- 172.21.15.169:0/3000352559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5aa518310 0x55b5aa51ce10 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.779 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.775+0000 7f2432042740 1 --2- 172.21.15.169:0/3000352559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa517920 0x55b5aa517d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.779 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.775+0000 7f2432042740 1 -- 172.21.15.169:0/3000352559 >> 172.21.15.169:0/3000352559 conn(0x55b5aa512e90 msgr2=0x55b5aa513290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:17.784 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.782+0000 7f2432042740 1 -- 172.21.15.169:0/3000352559 shutdown_connections 2024-03-28T01:03:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.782+0000 7f2432042740 1 -- 172.21.15.169:0/3000352559 wait complete. 2024-03-28T01:03:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.782+0000 7f2432042740 1 -- start start 2024-03-28T01:03:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.782+0000 7f2432042740 1 Processor -- start 2024-03-28T01:03:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.783+0000 7f2432042740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5aa517920 0x55b5aa5218d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.783+0000 7f2432042740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa518310 0x55b5aa521e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.783+0000 7f2432042740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5aa51d480 0x55b5aa528e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.783+0000 7f2432042740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5aa478bb0 con 0x55b5aa518310 2024-03-28T01:03:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.783+0000 7f2432042740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5aa438d50 con 0x55b5aa517920 2024-03-28T01:03:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.783+0000 7f2432042740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5aa5201f0 con 0x55b5aa51d480 2024-03-28T01:03:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.783+0000 7f2432042740 10 client.0 _ll_register_callbacks cb 0x55b5aa438720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:17.788 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.785+0000 7f242d9f6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5aa51d480 0x55b5aa528e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:17.788 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.785+0000 7f242d9f6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5aa51d480 0x55b5aa528e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:46340/0 (socket says 172.21.15.169:46340) 2024-03-28T01:03:17.788 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.785+0000 7f242d9f6640 1 -- 172.21.15.169:0/1139619750 learned_addr learned my addr 172.21.15.169:0/1139619750 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:17.789 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.785+0000 7f242c9f4640 1 --2- 172.21.15.169:0/1139619750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa518310 0x55b5aa521e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:17.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.786+0000 7f242d1f5640 1 --2- 172.21.15.169:0/1139619750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5aa517920 0x55b5aa5218d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.788+0000 7f242c9f4640 1 -- 172.21.15.169:0/1139619750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5aa51d480 msgr2=0x55b5aa528e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.788+0000 7f242c9f4640 1 --2- 172.21.15.169:0/1139619750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5aa51d480 0x55b5aa528e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.789+0000 7f242c9f4640 1 -- 172.21.15.169:0/1139619750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5aa517920 msgr2=0x55b5aa5218d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.789+0000 7f242c9f4640 1 --2- 172.21.15.169:0/1139619750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5aa517920 0x55b5aa5218d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.789+0000 7f242c9f4640 1 -- 172.21.15.169:0/1139619750 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b5aa25d620 con 0x55b5aa518310 2024-03-28T01:03:17.796 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.793+0000 7f242c9f4640 1 --2- 172.21.15.169:0/1139619750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa518310 0x55b5aa521e10 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x55b5aa518710 tx=0x7f241804ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7090d1e5a546d88c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:17.796 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.793+0000 7f240cff9640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:03:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.797+0000 7f240cff9640 1 -- 172.21.15.169:0/1139619750 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24180d5c70 con 0x55b5aa518310 2024-03-28T01:03:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.797+0000 7f240cff9640 1 -- 172.21.15.169:0/1139619750 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f24180d1330 con 0x55b5aa518310 2024-03-28T01:03:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.799+0000 7f2432042740 1 -- 172.21.15.169:0/1139619750 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f24180d1620 con 0x55b5aa518310 2024-03-28T01:03:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.801+0000 7f240cff9640 1 -- 172.21.15.169:0/1139619750 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24180cb570 con 0x55b5aa518310 2024-03-28T01:03:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.801+0000 7f2432042740 10 client.16749 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.801+0000 7f2432042740 1 -- 172.21.15.169:0/1139619750 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55b5aa52c1c0 con 0x55b5aa518310 2024-03-28T01:03:17.809 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.806+0000 7f240cff9640 1 -- 172.21.15.169:0/1139619750 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24180d1620 con 0x55b5aa518310 2024-03-28T01:03:17.810 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.807+0000 7f240cff9640 1 -- 172.21.15.169:0/1139619750 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f24180cc920 con 0x55b5aa518310 2024-03-28T01:03:17.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.808+0000 7f2432042740 10 client.16749 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:17.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.808+0000 7f2432042740 10 client.16749 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:17.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.808+0000 7f2432042740 20 client.16749 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:17.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.808+0000 7f2432042740 1 --2- 172.21.15.169:0/1139619750 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b5aa52d640 0x55b5aa52fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:17.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.808+0000 7f2432042740 4 client.16749 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:03:17.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.808+0000 7f2432042740 1 -- 172.21.15.169:0/1139619750 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b5aa48a050 con 0x55b5aa52d640 2024-03-28T01:03:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.813+0000 7f242d1f5640 1 --2- 172.21.15.169:0/1139619750 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b5aa52d640 0x55b5aa52fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:17.818 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.815+0000 7f242d1f5640 1 --2- 172.21.15.169:0/1139619750 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b5aa52d640 0x55b5aa52fa30 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cbbddf550e5255e server_cookie=1703697de5f851ae in_seq=0 out_seq=0 2024-03-28T01:03:17.820 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.817+0000 7f240cff9640 10 client.16749 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:17.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.818+0000 7f240cff9640 1 -- 172.21.15.169:0/1139619750 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b5aa48a050 con 0x55b5aa52d640 2024-03-28T01:03:17.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.818+0000 7f240cff9640 10 client.16749 handle_command_reply: tid=0 2024-03-28T01:03:17.821 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.818+0000 7f2432042740 1 client.16749 shutdown 2024-03-28T01:03:17.822 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.819+0000 7f2432042740 1 -- 172.21.15.169:0/1139619750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa518310 msgr2=0x55b5aa521e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:17.823 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.819+0000 7f2432042740 1 --2- 172.21.15.169:0/1139619750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa518310 0x55b5aa521e10 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x55b5aa518710 tx=0x7f241804ce40 comp rx=0 tx=0).stop 2024-03-28T01:03:17.823 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.819+0000 7f2432042740 1 -- 172.21.15.169:0/1139619750 shutdown_connections 2024-03-28T01:03:17.823 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.819+0000 7f2432042740 1 --2- 172.21.15.169:0/1139619750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5aa51d480 0x55b5aa528e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.823 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.819+0000 7f2432042740 1 --2- 172.21.15.169:0/1139619750 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b5aa52d640 0x55b5aa52fa30 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.823 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.819+0000 7f2432042740 1 --2- 172.21.15.169:0/1139619750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa518310 0x55b5aa521e10 secure :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0x55b5aa518710 tx=0x7f241804ce40 comp rx=0 tx=0).stop 2024-03-28T01:03:17.823 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.819+0000 7f2432042740 1 --2- 172.21.15.169:0/1139619750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5aa517920 0x55b5aa5218d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.823 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.819+0000 7f2432042740 1 -- 172.21.15.169:0/1139619750 >> 172.21.15.169:0/1139619750 conn(0x55b5aa512e90 msgr2=0x55b5aa512020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:17.823 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.819+0000 7f240cff9640 0 client.16749 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.819+0000 7f2432042740 1 -- 172.21.15.169:0/1139619750 shutdown_connections 2024-03-28T01:03:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.820+0000 7f2432042740 1 -- 172.21.15.169:0/1139619750 wait complete. 2024-03-28T01:03:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.820+0000 7f2432042740 20 client.16749 trim_cache size 0 max 16384 2024-03-28T01:03:17.833 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:17 smithi169 ceph-mon[32274]: pgmap v554: 129 pgs: 129 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s rd, 398 MiB/s wr, 3.81k op/s 2024-03-28T01:03:17.857 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.854+0000 7f2432042740 1 Processor -- start 2024-03-28T01:03:17.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:17 smithi174 ceph-mon[36060]: pgmap v554: 129 pgs: 129 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s rd, 398 MiB/s wr, 3.81k op/s 2024-03-28T01:03:17.860 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.855+0000 7f2432042740 1 -- start start 2024-03-28T01:03:17.860 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.856+0000 7f2432042740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5aa517920 0x55b5aa51f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:17.860 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.856+0000 7f2432042740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5aa51d480 0x55b5aa528e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:17.860 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.856+0000 7f2432042740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa5195d0 0x55b5aa519a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:17.860 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.856+0000 7f2432042740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24281aa2e0 con 0x55b5aa5195d0 2024-03-28T01:03:17.860 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.856+0000 7f2432042740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24281a7de0 con 0x55b5aa517920 2024-03-28T01:03:17.861 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.856+0000 7f2432042740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24281a5f30 con 0x55b5aa51d480 2024-03-28T01:03:17.861 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.856+0000 7f242d9f6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa5195d0 0x55b5aa519a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:17.861 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.856+0000 7f242d9f6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa5195d0 0x55b5aa519a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53182/0 (socket says 172.21.15.169:53182) 2024-03-28T01:03:17.861 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.856+0000 7f242d9f6640 1 -- 172.21.15.169:0/3388998320 learned_addr learned my addr 172.21.15.169:0/3388998320 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:17.861 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.857+0000 7f242d9f6640 1 -- 172.21.15.169:0/3388998320 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5aa51d480 msgr2=0x55b5aa528e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:03:17.861 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.857+0000 7f242d9f6640 1 --2- 172.21.15.169:0/3388998320 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5aa51d480 0x55b5aa528e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.861 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.857+0000 7f242d9f6640 1 -- 172.21.15.169:0/3388998320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5aa517920 msgr2=0x55b5aa51f870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:03:17.861 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.857+0000 7f242d9f6640 1 --2- 172.21.15.169:0/3388998320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5aa517920 0x55b5aa51f870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.861 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.857+0000 7f242d9f6640 1 -- 172.21.15.169:0/3388998320 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5aa522160 con 0x55b5aa5195d0 2024-03-28T01:03:17.861 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.857+0000 7f242d9f6640 1 --2- 172.21.15.169:0/3388998320 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa5195d0 0x55b5aa519a50 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f2420008330 tx=0x7f242000f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b218ce0e2cc364c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:17.861 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.857+0000 7f240effd640 1 -- 172.21.15.169:0/3388998320 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2420023630 con 0x55b5aa5195d0 2024-03-28T01:03:17.861 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.857+0000 7f240effd640 1 -- 172.21.15.169:0/3388998320 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f242001a910 con 0x55b5aa5195d0 2024-03-28T01:03:17.862 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.857+0000 7f240effd640 1 -- 172.21.15.169:0/3388998320 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2420022860 con 0x55b5aa5195d0 2024-03-28T01:03:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.858+0000 7f2432042740 1 -- 172.21.15.169:0/3388998320 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa5195d0 msgr2=0x55b5aa519a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.858+0000 7f2432042740 1 --2- 172.21.15.169:0/3388998320 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa5195d0 0x55b5aa519a50 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f2420008330 tx=0x7f242000f000 comp rx=0 tx=0).stop 2024-03-28T01:03:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.858+0000 7f2432042740 1 -- 172.21.15.169:0/3388998320 shutdown_connections 2024-03-28T01:03:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.858+0000 7f2432042740 1 --2- 172.21.15.169:0/3388998320 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa5195d0 0x55b5aa519a50 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.858+0000 7f2432042740 1 --2- 172.21.15.169:0/3388998320 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5aa51d480 0x55b5aa528e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.858+0000 7f2432042740 1 --2- 172.21.15.169:0/3388998320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5aa517920 0x55b5aa51f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.858+0000 7f2432042740 1 -- 172.21.15.169:0/3388998320 >> 172.21.15.169:0/3388998320 conn(0x55b5aa512e90 msgr2=0x55b5aa513270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.858+0000 7f2432042740 1 -- 172.21.15.169:0/3388998320 shutdown_connections 2024-03-28T01:03:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.859+0000 7f2432042740 1 -- 172.21.15.169:0/3388998320 wait complete. 2024-03-28T01:03:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.859+0000 7f2432042740 1 -- start start 2024-03-28T01:03:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.859+0000 7f2432042740 1 Processor -- start 2024-03-28T01:03:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.859+0000 7f2432042740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa517920 0x55b5aa526b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.859+0000 7f2432042740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5aa5195d0 0x55b5aa5270b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.859+0000 7f2432042740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5aa51d480 0x55b5aa528000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.859+0000 7f2432042740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24281a64a0 con 0x55b5aa517920 2024-03-28T01:03:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.859+0000 7f2432042740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24281a7780 con 0x55b5aa5195d0 2024-03-28T01:03:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.859+0000 7f2432042740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24281a6ac0 con 0x55b5aa51d480 2024-03-28T01:03:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.860+0000 7f2432042740 10 client.0 _ll_register_callbacks cb 0x7f24180045a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.860+0000 7f242d1f5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa517920 0x55b5aa526b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.860+0000 7f242d1f5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa517920 0x55b5aa526b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53198/0 (socket says 172.21.15.169:53198) 2024-03-28T01:03:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.860+0000 7f242d1f5640 1 -- 172.21.15.169:0/2604444685 learned_addr learned my addr 172.21.15.169:0/2604444685 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.860+0000 7f242d1f5640 1 -- 172.21.15.169:0/2604444685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5aa51d480 msgr2=0x55b5aa528000 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:03:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.860+0000 7f242d1f5640 1 --2- 172.21.15.169:0/2604444685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5aa51d480 0x55b5aa528000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.860+0000 7f242d1f5640 1 -- 172.21.15.169:0/2604444685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5aa5195d0 msgr2=0x55b5aa5270b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:03:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.860+0000 7f242d1f5640 1 --2- 172.21.15.169:0/2604444685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5aa5195d0 0x55b5aa5270b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.860+0000 7f242d1f5640 1 -- 172.21.15.169:0/2604444685 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b5aa5285f0 con 0x55b5aa517920 2024-03-28T01:03:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.861+0000 7f242d1f5640 1 --2- 172.21.15.169:0/2604444685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa517920 0x55b5aa526b70 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x55b5aa518b70 tx=0x7f241c010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=927b0a44cdd4a8be server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.861+0000 7f240cff9640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:03:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.864+0000 7f240cff9640 1 -- 172.21.15.169:0/2604444685 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241c005230 con 0x55b5aa517920 2024-03-28T01:03:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.864+0000 7f240cff9640 1 -- 172.21.15.169:0/2604444685 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f241c0053d0 con 0x55b5aa517920 2024-03-28T01:03:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.864+0000 7f2432042740 1 -- 172.21.15.169:0/2604444685 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f242001a910 con 0x55b5aa517920 2024-03-28T01:03:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.865+0000 7f240cff9640 1 -- 172.21.15.169:0/2604444685 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f241c0182d0 con 0x55b5aa517920 2024-03-28T01:03:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.865+0000 7f2432042740 10 client.16755 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.865+0000 7f2432042740 1 -- 172.21.15.169:0/2604444685 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55b5aa5894c0 con 0x55b5aa517920 2024-03-28T01:03:17.869 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.866+0000 7f240cff9640 1 -- 172.21.15.169:0/2604444685 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f242001a910 con 0x55b5aa517920 2024-03-28T01:03:17.870 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.867+0000 7f240cff9640 1 -- 172.21.15.169:0/2604444685 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f241c018930 con 0x55b5aa517920 2024-03-28T01:03:17.870 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.867+0000 7f2432042740 10 client.16755 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:17.870 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.867+0000 7f2432042740 10 client.16755 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:17.870 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.867+0000 7f2432042740 20 client.16755 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:17.870 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.867+0000 7f2432042740 1 --2- 172.21.15.169:0/2604444685 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b5aa589e20 0x55b5aa58c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:17.870 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.867+0000 7f2432042740 4 client.16755 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:03:17.871 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.867+0000 7f2432042740 1 -- 172.21.15.169:0/2604444685 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b5aa4843a0 con 0x55b5aa589e20 2024-03-28T01:03:17.871 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.868+0000 7f242c9f4640 1 --2- 172.21.15.169:0/2604444685 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b5aa589e20 0x55b5aa58c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:17.883 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.880+0000 7f242c9f4640 1 --2- 172.21.15.169:0/2604444685 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b5aa589e20 0x55b5aa58c240 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59aff22175c484c7 server_cookie=37967f4c94a5adee in_seq=0 out_seq=0 2024-03-28T01:03:17.883 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.880+0000 7f240cff9640 10 client.16755 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:17.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:17 smithi177 ceph-mon[36206]: pgmap v554: 129 pgs: 129 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s rd, 398 MiB/s wr, 3.81k op/s 2024-03-28T01:03:17.947 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.943+0000 7f240cff9640 1 -- 172.21.15.169:0/2604444685 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b5aa4843a0 con 0x55b5aa589e20 2024-03-28T01:03:17.947 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.943+0000 7f240cff9640 10 client.16755 handle_command_reply: tid=0 2024-03-28T01:03:17.947 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.943+0000 7f2432042740 1 client.16755 shutdown 2024-03-28T01:03:17.947 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.943+0000 7f2432042740 1 -- 172.21.15.169:0/2604444685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa517920 msgr2=0x55b5aa526b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:17.947 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.943+0000 7f2432042740 1 --2- 172.21.15.169:0/2604444685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa517920 0x55b5aa526b70 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x55b5aa518b70 tx=0x7f241c010800 comp rx=0 tx=0).stop 2024-03-28T01:03:17.947 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.943+0000 7f2432042740 1 -- 172.21.15.169:0/2604444685 shutdown_connections 2024-03-28T01:03:17.947 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.943+0000 7f2432042740 1 --2- 172.21.15.169:0/2604444685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5aa51d480 0x55b5aa528000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.947 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.943+0000 7f2432042740 1 --2- 172.21.15.169:0/2604444685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5aa5195d0 0x55b5aa5270b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.948 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.943+0000 7f2432042740 1 --2- 172.21.15.169:0/2604444685 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b5aa589e20 0x55b5aa58c240 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.948 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.944+0000 7f2432042740 1 --2- 172.21.15.169:0/2604444685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b5aa517920 0x55b5aa526b70 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.948 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.944+0000 7f2432042740 1 -- 172.21.15.169:0/2604444685 >> 172.21.15.169:0/2604444685 conn(0x55b5aa512e90 msgr2=0x55b5aa51c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:17.948 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.944+0000 7f240cff9640 0 client.16755 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:17.948 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.944+0000 7f2432042740 1 -- 172.21.15.169:0/2604444685 shutdown_connections 2024-03-28T01:03:17.948 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.944+0000 7f2432042740 1 -- 172.21.15.169:0/2604444685 wait complete. 2024-03-28T01:03:17.948 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.944+0000 7f2432042740 20 client.16755 trim_cache size 0 max 16384 2024-03-28T01:03:17.948 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:03:17.948 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:03:17.948 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:03:17.949 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:03:17.949 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.944+0000 7f242f459640 1 -- 172.21.15.169:0/2031692027 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f24100a2a10 msgr2=0x7f24100a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:17.949 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.944+0000 7f242f459640 1 --2- 172.21.15.169:0/2031692027 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f24100a2a10 0x7f24100a4ed0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f242000ab40 tx=0x7f2420009290 comp rx=0 tx=0).stop 2024-03-28T01:03:17.949 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.944+0000 7f242f459640 1 -- 172.21.15.169:0/2031692027 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24281028f0 msgr2=0x7f2428073940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:17.949 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.944+0000 7f242f459640 1 --2- 172.21.15.169:0/2031692027 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24281028f0 0x7f2428073940 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f241803d2b0 tx=0x7f24180459f0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.944+0000 7f242f459640 1 -- 172.21.15.169:0/2031692027 shutdown_connections 2024-03-28T01:03:17.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.944+0000 7f242f459640 1 --2- 172.21.15.169:0/2031692027 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f24100a2a10 0x7f24100a4ed0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.944+0000 7f242f459640 1 --2- 172.21.15.169:0/2031692027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24281032b0 0x7f2428074980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.944+0000 7f242f459640 1 --2- 172.21.15.169:0/2031692027 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24281028f0 0x7f2428073940 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.945+0000 7f242f459640 1 --2- 172.21.15.169:0/2031692027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2428101700 0x7f2428073400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.945+0000 7f242f459640 1 -- 172.21.15.169:0/2031692027 >> 172.21.15.169:0/2031692027 conn(0x7f24280fcc70 msgr2=0x7f2428100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:17.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.945+0000 7f242f459640 1 -- 172.21.15.169:0/2031692027 shutdown_connections 2024-03-28T01:03:17.950 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:17.945+0000 7f242f459640 1 -- 172.21.15.169:0/2031692027 wait complete. 2024-03-28T01:03:17.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.987+0000 7fdf0aa8c640 1 -- 172.21.15.169:0/1230375927 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf041000d0 msgr2=0x7fdf04109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:17.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.987+0000 7fdf0aa8c640 1 --2- 172.21.15.169:0/1230375927 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf041000d0 0x7fdf04109c80 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fdefc009be0 tx=0x7fdefc02bef0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.987+0000 7fdf0aa8c640 1 -- 172.21.15.169:0/1230375927 shutdown_connections 2024-03-28T01:03:17.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.987+0000 7fdf0aa8c640 1 --2- 172.21.15.169:0/1230375927 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf041000d0 0x7fdf04109c80 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.988+0000 7fdf0aa8c640 1 --2- 172.21.15.169:0/1230375927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf040ff710 0x7fdf040ffb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.988+0000 7fdf0aa8c640 1 --2- 172.21.15.169:0/1230375927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf04102ca0 0x7fdf041030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.988+0000 7fdf0aa8c640 1 -- 172.21.15.169:0/1230375927 >> 172.21.15.169:0/1230375927 conn(0x7fdf040fb290 msgr2=0x7fdf040fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:17.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.988+0000 7fdf0aa8c640 1 -- 172.21.15.169:0/1230375927 shutdown_connections 2024-03-28T01:03:17.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.988+0000 7fdf0aa8c640 1 -- 172.21.15.169:0/1230375927 wait complete. 2024-03-28T01:03:17.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.989+0000 7fdf0aa8c640 1 Processor -- start 2024-03-28T01:03:17.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.989+0000 7fdf0aa8c640 1 -- start start 2024-03-28T01:03:17.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.989+0000 7fdf0aa8c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf040ff710 0x7fdf041977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:17.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.989+0000 7fdf0aa8c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf041000d0 0x7fdf04197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:17.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.989+0000 7fdf0aa8c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf04102ca0 0x7fdf0419ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:17.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.990+0000 7fdf08828640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf040ff710 0x7fdf041977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:17.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.990+0000 7fdf03fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf041000d0 0x7fdf04197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:17.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.990+0000 7fdf08828640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf040ff710 0x7fdf041977a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:54626/0 (socket says 172.21.15.169:54626) 2024-03-28T01:03:17.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.990+0000 7fdf03fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf041000d0 0x7fdf04197ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:46370/0 (socket says 172.21.15.169:46370) 2024-03-28T01:03:17.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.990+0000 7fdf08828640 1 -- 172.21.15.169:0/2546474217 learned_addr learned my addr 172.21.15.169:0/2546474217 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:17.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.990+0000 7fdf0aa8c640 1 -- 172.21.15.169:0/2546474217 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0410d380 con 0x7fdf04102ca0 2024-03-28T01:03:17.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.990+0000 7fdf09029640 1 --2- 172.21.15.169:0/2546474217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf04102ca0 0x7fdf0419ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:17.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.990+0000 7fdf0aa8c640 1 -- 172.21.15.169:0/2546474217 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0410d200 con 0x7fdf040ff710 2024-03-28T01:03:17.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.990+0000 7fdf0aa8c640 1 -- 172.21.15.169:0/2546474217 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0410d500 con 0x7fdf041000d0 2024-03-28T01:03:17.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.990+0000 7fdf08828640 1 -- 172.21.15.169:0/2546474217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf041000d0 msgr2=0x7fdf04197ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:17.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.990+0000 7fdf08828640 1 --2- 172.21.15.169:0/2546474217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf041000d0 0x7fdf04197ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:17.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.990+0000 7fdf08828640 1 -- 172.21.15.169:0/2546474217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf04102ca0 msgr2=0x7fdf0419ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:18.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.990+0000 7fdf08828640 1 --2- 172.21.15.169:0/2546474217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf04102ca0 0x7fdf0419ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.990+0000 7fdf08828640 1 -- 172.21.15.169:0/2546474217 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0419f3c0 con 0x7fdf040ff710 2024-03-28T01:03:18.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.990+0000 7fdf09029640 1 --2- 172.21.15.169:0/2546474217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf04102ca0 0x7fdf0419ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:03:18.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.990+0000 7fdf08828640 1 --2- 172.21.15.169:0/2546474217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf040ff710 0x7fdf041977a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdef8009e10 tx=0x7fdef800edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:18.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.991+0000 7fdf01ffb640 1 -- 172.21.15.169:0/2546474217 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef800ccc0 con 0x7fdf040ff710 2024-03-28T01:03:18.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.991+0000 7fdf01ffb640 1 -- 172.21.15.169:0/2546474217 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdef80137d0 con 0x7fdf040ff710 2024-03-28T01:03:18.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.991+0000 7fdf0aa8c640 1 -- 172.21.15.169:0/2546474217 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf04100880 con 0x7fdf040ff710 2024-03-28T01:03:18.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.992+0000 7fdf01ffb640 1 -- 172.21.15.169:0/2546474217 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef8003850 con 0x7fdf040ff710 2024-03-28T01:03:18.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.992+0000 7fdf0aa8c640 1 -- 172.21.15.169:0/2546474217 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf04100d60 con 0x7fdf040ff710 2024-03-28T01:03:18.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.995+0000 7fdf01ffb640 1 -- 172.21.15.169:0/2546474217 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fdef8022020 con 0x7fdf040ff710 2024-03-28T01:03:18.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.996+0000 7fdf01ffb640 1 --2- 172.21.15.169:0/2546474217 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdef40a2900 0x7fdef40a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.996+0000 7fdf03fff640 1 --2- 172.21.15.169:0/2546474217 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdef40a2900 0x7fdef40a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.996+0000 7fdf0d672740 1 Processor -- start 2024-03-28T01:03:18.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.996+0000 7fdf01ffb640 1 -- 172.21.15.169:0/2546474217 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fdef80d30b0 con 0x7fdf040ff710 2024-03-28T01:03:18.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.996+0000 7fdf0d672740 1 -- start start 2024-03-28T01:03:18.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.997+0000 7fdf0d672740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88ca960 0x563aa88cad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.997+0000 7fdf0d672740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88cb350 0x563aa88cfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.997+0000 7fdf0d672740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88d04c0 0x563aa88d2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.997+0000 7fdf0d672740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x563aa880ac90 con 0x563aa88ca960 2024-03-28T01:03:18.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.997+0000 7fdf0d672740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x563aa8830070 con 0x563aa88d04c0 2024-03-28T01:03:18.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.997+0000 7fdf0d672740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x563aa87d8c80 con 0x563aa88cb350 2024-03-28T01:03:18.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.997+0000 7fdf03fff640 1 --2- 172.21.15.169:0/2546474217 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdef40a2900 0x7fdef40a4dc0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fdf04198cb0 tx=0x7fdef0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:18.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.997+0000 7fdf09029640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88d04c0 0x563aa88d2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.997+0000 7fdf09029640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88d04c0 0x563aa88d2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:54634/0 (socket says 172.21.15.169:54634) 2024-03-28T01:03:18.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.997+0000 7fdf09029640 1 -- 172.21.15.169:0/4033678507 learned_addr learned my addr 172.21.15.169:0/4033678507 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:18.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.997+0000 7fdf08828640 1 --2- 172.21.15.169:0/4033678507 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88ca960 0x563aa88cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.997+0000 7fdf03fff640 1 --2- 172.21.15.169:0/4033678507 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88cb350 0x563aa88cfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.997+0000 7fdf09029640 1 -- 172.21.15.169:0/4033678507 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88cb350 msgr2=0x563aa88cfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:18.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.997+0000 7fdf09029640 1 --2- 172.21.15.169:0/4033678507 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88cb350 0x563aa88cfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.997+0000 7fdf09029640 1 -- 172.21.15.169:0/4033678507 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88ca960 msgr2=0x563aa88cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:18.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.997+0000 7fdf09029640 1 --2- 172.21.15.169:0/4033678507 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88ca960 0x563aa88cad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.997+0000 7fdf09029640 1 -- 172.21.15.169:0/4033678507 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563aa87d0d10 con 0x563aa88d04c0 2024-03-28T01:03:18.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.997+0000 7fdf08828640 1 --2- 172.21.15.169:0/4033678507 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88ca960 0x563aa88cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:03:18.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.998+0000 7fdf09029640 1 --2- 172.21.15.169:0/4033678507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88d04c0 0x563aa88d2950 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fdefc004af0 tx=0x7fdefc009950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=133d4caf6df64e9e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:18.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.998+0000 7fdeeeffd640 1 -- 172.21.15.169:0/4033678507 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc043070 con 0x563aa88d04c0 2024-03-28T01:03:18.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.998+0000 7fdeeeffd640 1 -- 172.21.15.169:0/4033678507 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdefc033cc0 con 0x563aa88d04c0 2024-03-28T01:03:18.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.998+0000 7fdeeeffd640 1 -- 172.21.15.169:0/4033678507 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc02cc10 con 0x563aa88d04c0 2024-03-28T01:03:18.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.999+0000 7fdf0d672740 1 -- 172.21.15.169:0/4033678507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88d04c0 msgr2=0x563aa88d2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:18.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.999+0000 7fdf0d672740 1 --2- 172.21.15.169:0/4033678507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88d04c0 0x563aa88d2950 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fdefc004af0 tx=0x7fdefc009950 comp rx=0 tx=0).stop 2024-03-28T01:03:18.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.999+0000 7fdf0d672740 1 -- 172.21.15.169:0/4033678507 shutdown_connections 2024-03-28T01:03:18.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.999+0000 7fdf0d672740 1 --2- 172.21.15.169:0/4033678507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88d04c0 0x563aa88d2950 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.999+0000 7fdf0d672740 1 --2- 172.21.15.169:0/4033678507 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88cb350 0x563aa88cfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.999+0000 7fdf0d672740 1 --2- 172.21.15.169:0/4033678507 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88ca960 0x563aa88cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.999+0000 7fdf0d672740 1 -- 172.21.15.169:0/4033678507 >> 172.21.15.169:0/4033678507 conn(0x563aa88931a0 msgr2=0x563aa88c62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:18.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.999+0000 7fdf0d672740 1 -- 172.21.15.169:0/4033678507 shutdown_connections 2024-03-28T01:03:18.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.999+0000 7fdf0d672740 1 -- 172.21.15.169:0/4033678507 wait complete. 2024-03-28T01:03:18.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.999+0000 7fdf0d672740 1 -- start start 2024-03-28T01:03:18.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:17.999+0000 7fdf0d672740 1 Processor -- start 2024-03-28T01:03:18.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.000+0000 7fdf0d672740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88ca960 0x563aa88d4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.000+0000 7fdf0d672740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88cb350 0x563aa88d4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.000+0000 7fdf0d672740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88d04c0 0x563aa88dbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.000+0000 7fdf0d672740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x563aa87ec0d0 con 0x563aa88cb350 2024-03-28T01:03:18.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.000+0000 7fdf0d672740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x563aa882c110 con 0x563aa88d04c0 2024-03-28T01:03:18.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.000+0000 7fdf0d672740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x563aa88d3410 con 0x563aa88ca960 2024-03-28T01:03:18.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.000+0000 7fdf08828640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88ca960 0x563aa88d4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.000+0000 7fdf08828640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88ca960 0x563aa88d4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:46378/0 (socket says 172.21.15.169:46378) 2024-03-28T01:03:18.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.000+0000 7fdf08828640 1 -- 172.21.15.169:0/2544621178 learned_addr learned my addr 172.21.15.169:0/2544621178 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:18.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.000+0000 7fdf0d672740 10 client.0 _ll_register_callbacks cb 0x563aa8827e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:18.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.000+0000 7fdf03fff640 1 --2- 172.21.15.169:0/2544621178 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88cb350 0x563aa88d4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.001+0000 7fdf09029640 1 --2- 172.21.15.169:0/2544621178 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88d04c0 0x563aa88dbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.001+0000 7fdf03fff640 1 -- 172.21.15.169:0/2544621178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88ca960 msgr2=0x563aa88d4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:18.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.001+0000 7fdf03fff640 1 --2- 172.21.15.169:0/2544621178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88ca960 0x563aa88d4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.001+0000 7fdf03fff640 1 -- 172.21.15.169:0/2544621178 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88d04c0 msgr2=0x563aa88dbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:18.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.001+0000 7fdf03fff640 1 --2- 172.21.15.169:0/2544621178 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88d04c0 0x563aa88dbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.001+0000 7fdf03fff640 1 -- 172.21.15.169:0/2544621178 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563aa860c7e0 con 0x563aa88cb350 2024-03-28T01:03:18.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.002+0000 7fdf03fff640 1 --2- 172.21.15.169:0/2544621178 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88cb350 0x563aa88d4f90 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x563aa88cbdc0 tx=0x7fdef0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c1ef293f7164677 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:18.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.002+0000 7fdee3fff640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:03:18.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.002+0000 7fdee3fff640 1 -- 172.21.15.169:0/2544621178 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef001fd90 con 0x563aa88cb350 2024-03-28T01:03:18.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.002+0000 7fdee3fff640 1 -- 172.21.15.169:0/2544621178 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fdef00047a0 con 0x563aa88cb350 2024-03-28T01:03:18.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.002+0000 7fdf0d672740 1 -- 172.21.15.169:0/2544621178 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdefc033cc0 con 0x563aa88cb350 2024-03-28T01:03:18.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.002+0000 7fdee3fff640 1 -- 172.21.15.169:0/2544621178 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdef00049a0 con 0x563aa88cb350 2024-03-28T01:03:18.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.002+0000 7fdf0d672740 10 client.16767 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:18.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.002+0000 7fdf0d672740 1 -- 172.21.15.169:0/2544621178 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563aa866cf70 con 0x563aa88cb350 2024-03-28T01:03:18.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.003+0000 7fdee3fff640 1 -- 172.21.15.169:0/2544621178 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdefc033cc0 con 0x563aa88cb350 2024-03-28T01:03:18.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.003+0000 7fdee3fff640 1 -- 172.21.15.169:0/2544621178 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fdef0026070 con 0x563aa88cb350 2024-03-28T01:03:18.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.003+0000 7fdf0d672740 10 client.16767 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:18.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.003+0000 7fdf0d672740 10 client.16767 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:18.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.003+0000 7fdf0d672740 20 client.16767 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:18.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.003+0000 7fdf0d672740 1 --2- 172.21.15.169:0/2544621178 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563aa88e0990 0x563aa88e2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.003+0000 7fdf0d672740 4 client.16767 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:03:18.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.003+0000 7fdf0d672740 1 -- 172.21.15.169:0/2544621178 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563aa883d5b0 con 0x563aa88e0990 2024-03-28T01:03:18.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.003+0000 7fdf08828640 1 --2- 172.21.15.169:0/2544621178 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563aa88e0990 0x563aa88e2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.006+0000 7fdee3fff640 10 client.16767 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:18.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.006+0000 7fdf08828640 1 --2- 172.21.15.169:0/2544621178 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563aa88e0990 0x563aa88e2df0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8b16023a77b010c server_cookie=60e016c7ac7057c8 in_seq=0 out_seq=0 2024-03-28T01:03:18.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.007+0000 7fdee3fff640 1 -- 172.21.15.169:0/2544621178 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563aa883d5b0 con 0x563aa88e0990 2024-03-28T01:03:18.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.007+0000 7fdee3fff640 10 client.16767 handle_command_reply: tid=0 2024-03-28T01:03:18.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.008+0000 7fdf0d672740 1 client.16767 shutdown 2024-03-28T01:03:18.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.009+0000 7fdf0d672740 1 -- 172.21.15.169:0/2544621178 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88cb350 msgr2=0x563aa88d4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:18.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.009+0000 7fdf0d672740 1 --2- 172.21.15.169:0/2544621178 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88cb350 0x563aa88d4f90 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x563aa88cbdc0 tx=0x7fdef0004730 comp rx=0 tx=0).stop 2024-03-28T01:03:18.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.009+0000 7fdf0d672740 1 -- 172.21.15.169:0/2544621178 shutdown_connections 2024-03-28T01:03:18.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.009+0000 7fdf0d672740 1 --2- 172.21.15.169:0/2544621178 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88d04c0 0x563aa88dbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.009+0000 7fdf0d672740 1 --2- 172.21.15.169:0/2544621178 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563aa88e0990 0x563aa88e2df0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.009+0000 7fdee3fff640 0 client.16767 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:18.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.009+0000 7fdf0d672740 1 --2- 172.21.15.169:0/2544621178 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88cb350 0x563aa88d4f90 secure :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0x563aa88cbdc0 tx=0x7fdef0004730 comp rx=0 tx=0).stop 2024-03-28T01:03:18.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.009+0000 7fdf0d672740 1 --2- 172.21.15.169:0/2544621178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88ca960 0x563aa88d4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.009+0000 7fdf0d672740 1 -- 172.21.15.169:0/2544621178 >> 172.21.15.169:0/2544621178 conn(0x563aa88931a0 msgr2=0x563aa88c5530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:18.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.010+0000 7fdf0d672740 1 -- 172.21.15.169:0/2544621178 shutdown_connections 2024-03-28T01:03:18.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.010+0000 7fdf0d672740 1 -- 172.21.15.169:0/2544621178 wait complete. 2024-03-28T01:03:18.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.011+0000 7fdf0d672740 20 client.16767 trim_cache size 0 max 16384 2024-03-28T01:03:18.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.044+0000 7fdf0d672740 1 Processor -- start 2024-03-28T01:03:18.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.044+0000 7fdf0d672740 1 -- start start 2024-03-28T01:03:18.049 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.044+0000 7fdf0d672740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88ca960 0x563aa88dca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.049 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.045+0000 7fdf0d672740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88d04c0 0x563aa88dbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.049 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.045+0000 7fdf0d672740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88e86c0 0x563aa88dcf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.049 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.045+0000 7fdf0d672740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf041aa3e0 con 0x563aa88e86c0 2024-03-28T01:03:18.049 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.045+0000 7fdf0d672740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf041a7ee0 con 0x563aa88ca960 2024-03-28T01:03:18.049 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.045+0000 7fdf0d672740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf041a5f80 con 0x563aa88d04c0 2024-03-28T01:03:18.049 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.045+0000 7fdf03fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88d04c0 0x563aa88dbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.049 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.045+0000 7fdf03fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88d04c0 0x563aa88dbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:46388/0 (socket says 172.21.15.169:46388) 2024-03-28T01:03:18.049 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.045+0000 7fdf03fff640 1 -- 172.21.15.169:0/2572739621 learned_addr learned my addr 172.21.15.169:0/2572739621 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:18.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.045+0000 7fdf08828640 1 --2- 172.21.15.169:0/2572739621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88ca960 0x563aa88dca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.045+0000 7fdf09029640 1 --2- 172.21.15.169:0/2572739621 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88e86c0 0x563aa88dcf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.045+0000 7fdf03fff640 1 -- 172.21.15.169:0/2572739621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88ca960 msgr2=0x563aa88dca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:18.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.045+0000 7fdf03fff640 1 --2- 172.21.15.169:0/2572739621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88ca960 0x563aa88dca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.045+0000 7fdf03fff640 1 -- 172.21.15.169:0/2572739621 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88e86c0 msgr2=0x563aa88dcf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:18.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.045+0000 7fdf03fff640 1 --2- 172.21.15.169:0/2572739621 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88e86c0 0x563aa88dcf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.045+0000 7fdf03fff640 1 -- 172.21.15.169:0/2572739621 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563aa8504170 con 0x563aa88d04c0 2024-03-28T01:03:18.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.046+0000 7fdf03fff640 1 --2- 172.21.15.169:0/2572739621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88d04c0 0x563aa88dbf90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdef0027530 tx=0x7fdef0009a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f67eaab211fa3c6e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:18.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.046+0000 7fdeee7fc640 1 -- 172.21.15.169:0/2572739621 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef00047a0 con 0x563aa88d04c0 2024-03-28T01:03:18.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.046+0000 7fdeee7fc640 1 -- 172.21.15.169:0/2572739621 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdef0027c80 con 0x563aa88d04c0 2024-03-28T01:03:18.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.046+0000 7fdeee7fc640 1 -- 172.21.15.169:0/2572739621 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef0004da0 con 0x563aa88d04c0 2024-03-28T01:03:18.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.047+0000 7fdf0d672740 1 -- 172.21.15.169:0/2572739621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88d04c0 msgr2=0x563aa88dbf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:18.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.047+0000 7fdf0d672740 1 --2- 172.21.15.169:0/2572739621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88d04c0 0x563aa88dbf90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdef0027530 tx=0x7fdef0009a40 comp rx=0 tx=0).stop 2024-03-28T01:03:18.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.048+0000 7fdf0d672740 1 -- 172.21.15.169:0/2572739621 shutdown_connections 2024-03-28T01:03:18.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.048+0000 7fdf0d672740 1 --2- 172.21.15.169:0/2572739621 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88e86c0 0x563aa88dcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.048+0000 7fdf0d672740 1 --2- 172.21.15.169:0/2572739621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88d04c0 0x563aa88dbf90 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.048+0000 7fdf0d672740 1 --2- 172.21.15.169:0/2572739621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88ca960 0x563aa88dca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.048+0000 7fdf0d672740 1 -- 172.21.15.169:0/2572739621 >> 172.21.15.169:0/2572739621 conn(0x563aa88931a0 msgr2=0x563aa88c9510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:18.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.048+0000 7fdf0d672740 1 -- 172.21.15.169:0/2572739621 shutdown_connections 2024-03-28T01:03:18.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.048+0000 7fdf0d672740 1 -- 172.21.15.169:0/2572739621 wait complete. 2024-03-28T01:03:18.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.049+0000 7fdf0d672740 1 -- start start 2024-03-28T01:03:18.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.049+0000 7fdf0d672740 1 Processor -- start 2024-03-28T01:03:18.053 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.049+0000 7fdf0d672740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88ca960 0x563aa88dbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.053 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.049+0000 7fdf0d672740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88d04c0 0x563aa88da580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.053 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.049+0000 7fdf0d672740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88e86c0 0x563aa88daac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.053 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.050+0000 7fdf0d672740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf041a6580 con 0x563aa88e86c0 2024-03-28T01:03:18.053 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.050+0000 7fdf08828640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88ca960 0x563aa88dbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.050+0000 7fdf03fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88d04c0 0x563aa88da580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.050+0000 7fdf08828640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88ca960 0x563aa88dbf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:46390/0 (socket says 172.21.15.169:46390) 2024-03-28T01:03:18.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.050+0000 7fdf08828640 1 -- 172.21.15.169:0/2539117688 learned_addr learned my addr 172.21.15.169:0/2539117688 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:18.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.050+0000 7fdf0d672740 1 -- 172.21.15.169:0/2539117688 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf041a7880 con 0x563aa88d04c0 2024-03-28T01:03:18.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.050+0000 7fdf0d672740 1 -- 172.21.15.169:0/2539117688 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf041a6bc0 con 0x563aa88ca960 2024-03-28T01:03:18.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.050+0000 7fdf0d672740 10 client.0 _ll_register_callbacks cb 0x7fdef0004160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:18.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.050+0000 7fdf09029640 1 --2- 172.21.15.169:0/2539117688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88e86c0 0x563aa88daac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.050+0000 7fdf08828640 1 -- 172.21.15.169:0/2539117688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88d04c0 msgr2=0x563aa88da580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:18.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.050+0000 7fdf08828640 1 --2- 172.21.15.169:0/2539117688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88d04c0 0x563aa88da580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.050+0000 7fdf08828640 1 -- 172.21.15.169:0/2539117688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88e86c0 msgr2=0x563aa88daac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:18.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.050+0000 7fdf08828640 1 --2- 172.21.15.169:0/2539117688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88e86c0 0x563aa88daac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.050+0000 7fdf08828640 1 -- 172.21.15.169:0/2539117688 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563aa88db2f0 con 0x563aa88ca960 2024-03-28T01:03:18.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.050+0000 7fdf08828640 1 --2- 172.21.15.169:0/2539117688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88ca960 0x563aa88dbf00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fdef800c6a0 tx=0x7fdef80d2040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d8878097ee147d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:18.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.050+0000 7fdee3fff640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:03:18.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.051+0000 7fdee3fff640 1 -- 172.21.15.169:0/2539117688 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef80d08c0 con 0x563aa88ca960 2024-03-28T01:03:18.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.051+0000 7fdee3fff640 1 -- 172.21.15.169:0/2539117688 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fdef809cc10 con 0x563aa88ca960 2024-03-28T01:03:18.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.051+0000 7fdf0d672740 1 -- 172.21.15.169:0/2539117688 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdef0027c80 con 0x563aa88ca960 2024-03-28T01:03:18.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.051+0000 7fdee3fff640 1 -- 172.21.15.169:0/2539117688 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdef80bd050 con 0x563aa88ca960 2024-03-28T01:03:18.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.051+0000 7fdf0d672740 10 client.26599 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:18.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.051+0000 7fdf0d672740 1 -- 172.21.15.169:0/2539117688 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563aa893d340 con 0x563aa88ca960 2024-03-28T01:03:18.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.052+0000 7fdee3fff640 1 -- 172.21.15.169:0/2539117688 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdef0027c80 con 0x563aa88ca960 2024-03-28T01:03:18.057 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.054+0000 7fdee3fff640 1 -- 172.21.15.169:0/2539117688 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fdef8089cb0 con 0x563aa88ca960 2024-03-28T01:03:18.057 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.054+0000 7fdf0d672740 10 client.26599 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:18.057 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.054+0000 7fdf0d672740 10 client.26599 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:18.057 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.054+0000 7fdf0d672740 20 client.26599 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:18.057 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.054+0000 7fdf0d672740 1 --2- 172.21.15.169:0/2539117688 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563aa893dbd0 0x563aa893fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.057 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.054+0000 7fdf0d672740 4 client.26599 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T01:03:18.057 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.054+0000 7fdf0d672740 1 -- 172.21.15.169:0/2539117688 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x563aa88d4840 con 0x563aa893dbd0 2024-03-28T01:03:18.057 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.054+0000 7fdf03fff640 1 --2- 172.21.15.169:0/2539117688 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563aa893dbd0 0x563aa893fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.055+0000 7fdf03fff640 1 --2- 172.21.15.169:0/2539117688 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563aa893dbd0 0x563aa893fff0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f89b8aedf511c3f1 server_cookie=c0dc1d7de2bb1517 in_seq=0 out_seq=0 2024-03-28T01:03:18.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.055+0000 7fdee3fff640 10 client.26599 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:18.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.055+0000 7fdee3fff640 1 -- 172.21.15.169:0/2539117688 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x563aa88d4840 con 0x563aa893dbd0 2024-03-28T01:03:18.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.055+0000 7fdee3fff640 10 client.26599 handle_command_reply: tid=0 2024-03-28T01:03:18.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.058+0000 7fdf0d672740 1 client.26599 shutdown 2024-03-28T01:03:18.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.058+0000 7fdf0d672740 1 -- 172.21.15.169:0/2539117688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88ca960 msgr2=0x563aa88dbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:18.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.058+0000 7fdf0d672740 1 --2- 172.21.15.169:0/2539117688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88ca960 0x563aa88dbf00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fdef800c6a0 tx=0x7fdef80d2040 comp rx=0 tx=0).stop 2024-03-28T01:03:18.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.058+0000 7fdf0d672740 1 -- 172.21.15.169:0/2539117688 shutdown_connections 2024-03-28T01:03:18.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.058+0000 7fdf0d672740 1 --2- 172.21.15.169:0/2539117688 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563aa893dbd0 0x563aa893fff0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.058+0000 7fdf0d672740 1 --2- 172.21.15.169:0/2539117688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563aa88e86c0 0x563aa88daac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.058+0000 7fdf0d672740 1 --2- 172.21.15.169:0/2539117688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563aa88d04c0 0x563aa88da580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.058+0000 7fdf0d672740 1 --2- 172.21.15.169:0/2539117688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563aa88ca960 0x563aa88dbf00 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.058+0000 7fdf0d672740 1 -- 172.21.15.169:0/2539117688 >> 172.21.15.169:0/2539117688 conn(0x563aa88931a0 msgr2=0x563aa88c9510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:18.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.059+0000 7fdee3fff640 0 client.26599 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:18.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.059+0000 7fdf0d672740 1 -- 172.21.15.169:0/2539117688 shutdown_connections 2024-03-28T01:03:18.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.059+0000 7fdf0d672740 1 -- 172.21.15.169:0/2539117688 wait complete. 2024-03-28T01:03:18.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.059+0000 7fdf0d672740 20 client.26599 trim_cache size 0 max 16384 2024-03-28T01:03:18.063 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:03:18.063 INFO:teuthology.orchestra.run.smithi169.stdout: "return_code": 0, 2024-03-28T01:03:18.063 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_tag": "c51e7a9b-4ce5-4aae-a4c3-a9d71c3f45f9", 2024-03-28T01:03:18.063 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": "asynchronous" 2024-03-28T01:03:18.063 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:03:18.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.061+0000 7fdf0aa8c640 1 -- 172.21.15.169:0/2546474217 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdef40a2900 msgr2=0x7fdef40a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:18.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.061+0000 7fdf0aa8c640 1 --2- 172.21.15.169:0/2546474217 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdef40a2900 0x7fdef40a4dc0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fdf04198cb0 tx=0x7fdef0005f90 comp rx=0 tx=0).stop 2024-03-28T01:03:18.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.061+0000 7fdf0aa8c640 1 -- 172.21.15.169:0/2546474217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf040ff710 msgr2=0x7fdf041977a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:18.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.061+0000 7fdf0aa8c640 1 --2- 172.21.15.169:0/2546474217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf040ff710 0x7fdf041977a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdef8009e10 tx=0x7fdef800edc0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.062+0000 7fdf0aa8c640 1 -- 172.21.15.169:0/2546474217 shutdown_connections 2024-03-28T01:03:18.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.062+0000 7fdf0aa8c640 1 --2- 172.21.15.169:0/2546474217 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdef40a2900 0x7fdef40a4dc0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.062+0000 7fdf0aa8c640 1 --2- 172.21.15.169:0/2546474217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf04102ca0 0x7fdf0419ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.062+0000 7fdf0aa8c640 1 --2- 172.21.15.169:0/2546474217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf041000d0 0x7fdf04197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.062+0000 7fdf0aa8c640 1 --2- 172.21.15.169:0/2546474217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf040ff710 0x7fdf041977a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.062+0000 7fdf0aa8c640 1 -- 172.21.15.169:0/2546474217 >> 172.21.15.169:0/2546474217 conn(0x7fdf040fb290 msgr2=0x7fdf040fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:18.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.062+0000 7fdf0aa8c640 1 -- 172.21.15.169:0/2546474217 shutdown_connections 2024-03-28T01:03:18.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.062+0000 7fdf0aa8c640 1 -- 172.21.15.169:0/2546474217 wait complete. 2024-03-28T01:03:18.084 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:03:18.084 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:03:18.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.347+0000 7fbaabfff640 1 -- 172.21.15.169:0/3957479853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbaac102030 msgr2=0x7fbaac068800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:18.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.347+0000 7fbaabfff640 1 --2- 172.21.15.169:0/3957479853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbaac102030 0x7fbaac068800 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fba9c009860 tx=0x7fba9c02bb70 comp rx=0 tx=0).stop 2024-03-28T01:03:18.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.348+0000 7fbaabfff640 1 -- 172.21.15.169:0/3957479853 shutdown_connections 2024-03-28T01:03:18.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.348+0000 7fbaabfff640 1 --2- 172.21.15.169:0/3957479853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbaac102030 0x7fbaac068800 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.348+0000 7fbaabfff640 1 --2- 172.21.15.169:0/3957479853 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbaac101670 0x7fbaac101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.348+0000 7fbaabfff640 1 --2- 172.21.15.169:0/3957479853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbaac100480 0x7fbaac1008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.348+0000 7fbaabfff640 1 -- 172.21.15.169:0/3957479853 >> 172.21.15.169:0/3957479853 conn(0x7fbaac0fba10 msgr2=0x7fbaac0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:18.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.348+0000 7fbaabfff640 1 -- 172.21.15.169:0/3957479853 shutdown_connections 2024-03-28T01:03:18.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.348+0000 7fbaabfff640 1 -- 172.21.15.169:0/3957479853 wait complete. 2024-03-28T01:03:18.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.349+0000 7fbaabfff640 1 Processor -- start 2024-03-28T01:03:18.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.350+0000 7fbaabfff640 1 -- start start 2024-03-28T01:03:18.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.350+0000 7fbaabfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbaac100480 0x7fbaac19de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.350+0000 7fbaabfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbaac101670 0x7fbaac19e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.350+0000 7fbaabfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbaac102030 0x7fbaac1a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.350+0000 7fbaabfff640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaac10b030 con 0x7fbaac100480 2024-03-28T01:03:18.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.350+0000 7fbaabfff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaac10aeb0 con 0x7fbaac102030 2024-03-28T01:03:18.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.350+0000 7fbaabfff640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaac10b1b0 con 0x7fbaac101670 2024-03-28T01:03:18.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.350+0000 7fbaaaffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbaac100480 0x7fbaac19de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.350+0000 7fbaab7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbaac102030 0x7fbaac1a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.350+0000 7fbaab7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbaac102030 0x7fbaac1a5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:54674/0 (socket says 172.21.15.169:54674) 2024-03-28T01:03:18.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.350+0000 7fbaab7fe640 1 -- 172.21.15.169:0/1152528579 learned_addr learned my addr 172.21.15.169:0/1152528579 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:18.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.350+0000 7fbaaa7fc640 1 --2- 172.21.15.169:0/1152528579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbaac101670 0x7fbaac19e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.351+0000 7fbaab7fe640 1 -- 172.21.15.169:0/1152528579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbaac101670 msgr2=0x7fbaac19e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:18.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.351+0000 7fbaab7fe640 1 --2- 172.21.15.169:0/1152528579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbaac101670 0x7fbaac19e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.351+0000 7fbaab7fe640 1 -- 172.21.15.169:0/1152528579 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbaac100480 msgr2=0x7fbaac19de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:18.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.351+0000 7fbaab7fe640 1 --2- 172.21.15.169:0/1152528579 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbaac100480 0x7fbaac19de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.351+0000 7fbaab7fe640 1 -- 172.21.15.169:0/1152528579 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaac1a5aa0 con 0x7fbaac102030 2024-03-28T01:03:18.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.351+0000 7fbaab7fe640 1 --2- 172.21.15.169:0/1152528579 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbaac102030 0x7fbaac1a5430 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fba9c009780 tx=0x7fba9c0324a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:18.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.351+0000 7fbab08aa640 1 -- 172.21.15.169:0/1152528579 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9c041020 con 0x7fbaac102030 2024-03-28T01:03:18.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.351+0000 7fbab08aa640 1 -- 172.21.15.169:0/1152528579 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba9c02c9f0 con 0x7fbaac102030 2024-03-28T01:03:18.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.351+0000 7fbaabfff640 1 -- 172.21.15.169:0/1152528579 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaac1a5d30 con 0x7fbaac102030 2024-03-28T01:03:18.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.351+0000 7fbab08aa640 1 -- 172.21.15.169:0/1152528579 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9c039040 con 0x7fbaac102030 2024-03-28T01:03:18.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.351+0000 7fbaabfff640 1 -- 172.21.15.169:0/1152528579 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaac1a61d0 con 0x7fbaac102030 2024-03-28T01:03:18.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.353+0000 7fbab08aa640 1 -- 172.21.15.169:0/1152528579 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fba9c044020 con 0x7fbaac102030 2024-03-28T01:03:18.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.354+0000 7fbab4575740 1 Processor -- start 2024-03-28T01:03:18.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.354+0000 7fbab08aa640 1 --2- 172.21.15.169:0/1152528579 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fba940a2850 0x7fba940a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.354+0000 7fbab4575740 1 -- start start 2024-03-28T01:03:18.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.354+0000 7fbaaa7fc640 1 --2- 172.21.15.169:0/1152528579 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fba940a2850 0x7fba940a4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.355+0000 7fbab08aa640 1 -- 172.21.15.169:0/1152528579 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fba9c0f07b0 con 0x7fbaac102030 2024-03-28T01:03:18.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.354+0000 7fbab4575740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b9f437f80 0x558b9f4383a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.355+0000 7fbab4575740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f438970 0x558b9f43d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.355+0000 7fbab4575740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f43dae0 0x558b9f43ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.355+0000 7fbab4575740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b9f39d3d0 con 0x558b9f438970 2024-03-28T01:03:18.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.355+0000 7fbab4575740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b9f345ad0 con 0x558b9f43dae0 2024-03-28T01:03:18.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.355+0000 7fbab4575740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b9f399470 con 0x558b9f437f80 2024-03-28T01:03:18.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.355+0000 7fbaab7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f43dae0 0x558b9f43ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.355+0000 7fbaab7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f43dae0 0x558b9f43ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:54684/0 (socket says 172.21.15.169:54684) 2024-03-28T01:03:18.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.355+0000 7fbaab7fe640 1 -- 172.21.15.169:0/287971486 learned_addr learned my addr 172.21.15.169:0/287971486 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:18.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.355+0000 7fbaaaffd640 1 --2- 172.21.15.169:0/287971486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b9f437f80 0x558b9f4383a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.355+0000 7fbaaa7fc640 1 --2- 172.21.15.169:0/287971486 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f438970 0x558b9f43d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.355+0000 7fbaab7fe640 1 -- 172.21.15.169:0/287971486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b9f437f80 msgr2=0x558b9f4383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:18.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.355+0000 7fbaab7fe640 1 --2- 172.21.15.169:0/287971486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b9f437f80 0x558b9f4383a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.355+0000 7fbaab7fe640 1 -- 172.21.15.169:0/287971486 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f438970 msgr2=0x558b9f43d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:18.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.355+0000 7fbaab7fe640 1 --2- 172.21.15.169:0/287971486 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f438970 0x558b9f43d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.355+0000 7fbaab7fe640 1 -- 172.21.15.169:0/287971486 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b9f17b7d0 con 0x558b9f43dae0 2024-03-28T01:03:18.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.355+0000 7fbaaaffd640 1 --2- 172.21.15.169:0/287971486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b9f437f80 0x558b9f4383a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:03:18.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.355+0000 7fbaaa7fc640 1 --2- 172.21.15.169:0/1152528579 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fba940a2850 0x7fba940a4d10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbaa00095f0 tx=0x7fbaa0011810 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:18.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.355+0000 7fbaab7fe640 1 --2- 172.21.15.169:0/287971486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f43dae0 0x558b9f43ff70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fba9c0bd9c0 tx=0x7fba9c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd27b5a07f6e711c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:18.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.356+0000 7fba91ffb640 1 -- 172.21.15.169:0/287971486 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9c004400 con 0x558b9f43dae0 2024-03-28T01:03:18.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.356+0000 7fba91ffb640 1 -- 172.21.15.169:0/287971486 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba9c0bdcc0 con 0x558b9f43dae0 2024-03-28T01:03:18.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.356+0000 7fba91ffb640 1 -- 172.21.15.169:0/287971486 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9c0ba330 con 0x558b9f43dae0 2024-03-28T01:03:18.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.356+0000 7fbab4575740 1 -- 172.21.15.169:0/287971486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f43dae0 msgr2=0x558b9f43ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:18.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.356+0000 7fbab4575740 1 --2- 172.21.15.169:0/287971486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f43dae0 0x558b9f43ff70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fba9c0bd9c0 tx=0x7fba9c005f90 comp rx=0 tx=0).stop 2024-03-28T01:03:18.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.356+0000 7fbab4575740 1 -- 172.21.15.169:0/287971486 shutdown_connections 2024-03-28T01:03:18.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.356+0000 7fbab4575740 1 --2- 172.21.15.169:0/287971486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f43dae0 0x558b9f43ff70 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.356+0000 7fbab4575740 1 --2- 172.21.15.169:0/287971486 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f438970 0x558b9f43d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.356+0000 7fbab4575740 1 --2- 172.21.15.169:0/287971486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b9f437f80 0x558b9f4383a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.356+0000 7fbab4575740 1 -- 172.21.15.169:0/287971486 >> 172.21.15.169:0/287971486 conn(0x558b9f4334f0 msgr2=0x558b9f4338f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:18.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.357+0000 7fbab4575740 1 -- 172.21.15.169:0/287971486 shutdown_connections 2024-03-28T01:03:18.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.357+0000 7fbab4575740 1 -- 172.21.15.169:0/287971486 wait complete. 2024-03-28T01:03:18.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.357+0000 7fbab4575740 1 -- start start 2024-03-28T01:03:18.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.357+0000 7fbab4575740 1 Processor -- start 2024-03-28T01:03:18.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.357+0000 7fbab4575740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f437f80 0x558b9f442150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.357+0000 7fbab4575740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b9f438970 0x558b9f442690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.357+0000 7fbab4575740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f43dae0 0x558b9f4496d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.357+0000 7fbab4575740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b9f440850 con 0x558b9f437f80 2024-03-28T01:03:18.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.357+0000 7fbab4575740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b9f359060 con 0x558b9f43dae0 2024-03-28T01:03:18.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.357+0000 7fbab4575740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b9f4409d0 con 0x558b9f438970 2024-03-28T01:03:18.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.357+0000 7fbab4575740 10 client.0 _ll_register_callbacks cb 0x558b9f33b6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:18.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.358+0000 7fbaab7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f43dae0 0x558b9f4496d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.358+0000 7fbaaa7fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b9f438970 0x558b9f442690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.358+0000 7fbaab7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f43dae0 0x558b9f4496d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:54690/0 (socket says 172.21.15.169:54690) 2024-03-28T01:03:18.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.358+0000 7fbaab7fe640 1 -- 172.21.15.169:0/1550536441 learned_addr learned my addr 172.21.15.169:0/1550536441 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:18.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.358+0000 7fbaaaffd640 1 --2- 172.21.15.169:0/1550536441 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f437f80 0x558b9f442150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.358+0000 7fbaab7fe640 1 -- 172.21.15.169:0/1550536441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b9f438970 msgr2=0x558b9f442690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:18.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.358+0000 7fbaab7fe640 1 --2- 172.21.15.169:0/1550536441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b9f438970 0x558b9f442690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.358+0000 7fbaab7fe640 1 -- 172.21.15.169:0/1550536441 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f437f80 msgr2=0x558b9f442150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:18.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.358+0000 7fbaab7fe640 1 --2- 172.21.15.169:0/1550536441 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f437f80 0x558b9f442150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.358+0000 7fbaab7fe640 1 -- 172.21.15.169:0/1550536441 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558b9f0b13f0 con 0x558b9f43dae0 2024-03-28T01:03:18.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.358+0000 7fbaaaffd640 1 --2- 172.21.15.169:0/1550536441 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f437f80 0x558b9f442150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:03:18.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.358+0000 7fbaab7fe640 1 --2- 172.21.15.169:0/1550536441 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f43dae0 0x558b9f4496d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fba9c033000 tx=0x7fba9c0c65a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9ce6b9cd6f03a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:18.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.358+0000 7fba8affd640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:03:18.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.358+0000 7fba8affd640 1 -- 172.21.15.169:0/1550536441 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9c0da020 con 0x558b9f43dae0 2024-03-28T01:03:18.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.358+0000 7fba8affd640 1 -- 172.21.15.169:0/1550536441 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fba9c032640 con 0x558b9f43dae0 2024-03-28T01:03:18.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.358+0000 7fbab4575740 1 -- 172.21.15.169:0/1550536441 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fba9c0bdcc0 con 0x558b9f43dae0 2024-03-28T01:03:18.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.359+0000 7fba8affd640 1 -- 172.21.15.169:0/1550536441 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba9c0c1e10 con 0x558b9f43dae0 2024-03-28T01:03:18.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.359+0000 7fbab4575740 10 client.26672 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:18.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.359+0000 7fbab4575740 1 -- 172.21.15.169:0/1550536441 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558b9f0710f0 con 0x558b9f43dae0 2024-03-28T01:03:18.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.359+0000 7fba8affd640 1 -- 172.21.15.169:0/1550536441 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba9c0bdcc0 con 0x558b9f43dae0 2024-03-28T01:03:18.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.359+0000 7fba8affd640 1 -- 172.21.15.169:0/1550536441 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fba9c03c030 con 0x558b9f43dae0 2024-03-28T01:03:18.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.359+0000 7fbab4575740 10 client.26672 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:18.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.359+0000 7fbab4575740 10 client.26672 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:18.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.359+0000 7fbab4575740 20 client.26672 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:18.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.359+0000 7fbab4575740 1 --2- 172.21.15.169:0/1550536441 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558b9f44d990 0x558b9f44fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.359+0000 7fbab4575740 4 client.26672 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:03:18.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.359+0000 7fbab4575740 1 -- 172.21.15.169:0/1550536441 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558b9f3aa910 con 0x558b9f44d990 2024-03-28T01:03:18.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.359+0000 7fbaaaffd640 1 --2- 172.21.15.169:0/1550536441 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558b9f44d990 0x558b9f44fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.360+0000 7fba8affd640 10 client.26672 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:18.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.360+0000 7fbaaaffd640 1 --2- 172.21.15.169:0/1550536441 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558b9f44d990 0x558b9f44fdf0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2162737d878f4052 server_cookie=b86c06dd16bfdd3e in_seq=0 out_seq=0 2024-03-28T01:03:18.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.360+0000 7fba8affd640 1 -- 172.21.15.169:0/1550536441 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558b9f3aa910 con 0x558b9f44d990 2024-03-28T01:03:18.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.360+0000 7fba8affd640 10 client.26672 handle_command_reply: tid=0 2024-03-28T01:03:18.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.360+0000 7fbab4575740 1 client.26672 shutdown 2024-03-28T01:03:18.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.361+0000 7fbab4575740 1 -- 172.21.15.169:0/1550536441 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f43dae0 msgr2=0x558b9f4496d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:18.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.361+0000 7fbab4575740 1 --2- 172.21.15.169:0/1550536441 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f43dae0 0x558b9f4496d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fba9c033000 tx=0x7fba9c0c65a0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.361+0000 7fbab4575740 1 -- 172.21.15.169:0/1550536441 shutdown_connections 2024-03-28T01:03:18.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.361+0000 7fbab4575740 1 --2- 172.21.15.169:0/1550536441 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f43dae0 0x558b9f4496d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.361+0000 7fbab4575740 1 --2- 172.21.15.169:0/1550536441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b9f438970 0x558b9f442690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.361+0000 7fbab4575740 1 --2- 172.21.15.169:0/1550536441 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558b9f44d990 0x558b9f44fdf0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.361+0000 7fbab4575740 1 --2- 172.21.15.169:0/1550536441 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f437f80 0x558b9f442150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.361+0000 7fba8affd640 0 client.26672 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:18.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.361+0000 7fbab4575740 1 -- 172.21.15.169:0/1550536441 >> 172.21.15.169:0/1550536441 conn(0x558b9f4334f0 msgr2=0x558b9f4327b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:18.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.361+0000 7fbab4575740 1 -- 172.21.15.169:0/1550536441 shutdown_connections 2024-03-28T01:03:18.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.361+0000 7fbab4575740 1 -- 172.21.15.169:0/1550536441 wait complete. 2024-03-28T01:03:18.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.361+0000 7fbab4575740 20 client.26672 trim_cache size 0 max 16384 2024-03-28T01:03:18.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.379+0000 7fbab4575740 1 Processor -- start 2024-03-28T01:03:18.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.379+0000 7fbab4575740 1 -- start start 2024-03-28T01:03:18.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.379+0000 7fbab4575740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f437f80 0x558b9f439dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.379+0000 7fbab4575740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f438970 0x558b9f43a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.379+0000 7fbab4575740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b9f43dae0 0x558b9f449690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.379+0000 7fbab4575740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaac1b2bc0 con 0x558b9f437f80 2024-03-28T01:03:18.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.379+0000 7fbab4575740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaac1b06c0 con 0x558b9f438970 2024-03-28T01:03:18.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.379+0000 7fbab4575740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaac1ae760 con 0x558b9f43dae0 2024-03-28T01:03:18.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.379+0000 7fbaaaffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f437f80 0x558b9f439dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.379+0000 7fbaaa7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f438970 0x558b9f43a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.379+0000 7fbaaaffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f437f80 0x558b9f439dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53280/0 (socket says 172.21.15.169:53280) 2024-03-28T01:03:18.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.379+0000 7fbaaaffd640 1 -- 172.21.15.169:0/3266090500 learned_addr learned my addr 172.21.15.169:0/3266090500 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:18.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.379+0000 7fbaab7fe640 1 --2- 172.21.15.169:0/3266090500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b9f43dae0 0x558b9f449690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.379+0000 7fbaaaffd640 1 -- 172.21.15.169:0/3266090500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b9f43dae0 msgr2=0x558b9f449690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:18.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.379+0000 7fbaaaffd640 1 --2- 172.21.15.169:0/3266090500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b9f43dae0 0x558b9f449690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.379+0000 7fbaaaffd640 1 -- 172.21.15.169:0/3266090500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f438970 msgr2=0x558b9f43a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:18.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.379+0000 7fbaaaffd640 1 --2- 172.21.15.169:0/3266090500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f438970 0x558b9f43a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.379+0000 7fbaaaffd640 1 -- 172.21.15.169:0/3266090500 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b9f439200 con 0x558b9f437f80 2024-03-28T01:03:18.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.379+0000 7fbaaa7fc640 1 --2- 172.21.15.169:0/3266090500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f438970 0x558b9f43a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:03:18.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.380+0000 7fbaaaffd640 1 --2- 172.21.15.169:0/3266090500 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f437f80 0x558b9f439dd0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fbaac1014d0 tx=0x7fbaa40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5299c41f0fcfe949 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:18.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.380+0000 7fba917fa640 1 -- 172.21.15.169:0/3266090500 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa401bc40 con 0x558b9f437f80 2024-03-28T01:03:18.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.380+0000 7fba917fa640 1 -- 172.21.15.169:0/3266090500 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbaa4025070 con 0x558b9f437f80 2024-03-28T01:03:18.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.380+0000 7fba917fa640 1 -- 172.21.15.169:0/3266090500 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa4019960 con 0x558b9f437f80 2024-03-28T01:03:18.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.381+0000 7fbab4575740 1 -- 172.21.15.169:0/3266090500 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f437f80 msgr2=0x558b9f439dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:18.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.381+0000 7fbab4575740 1 --2- 172.21.15.169:0/3266090500 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f437f80 0x558b9f439dd0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fbaac1014d0 tx=0x7fbaa40095c0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.381+0000 7fbab4575740 1 -- 172.21.15.169:0/3266090500 shutdown_connections 2024-03-28T01:03:18.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.381+0000 7fbab4575740 1 --2- 172.21.15.169:0/3266090500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b9f43dae0 0x558b9f449690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.381+0000 7fbab4575740 1 --2- 172.21.15.169:0/3266090500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f438970 0x558b9f43a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.381+0000 7fbab4575740 1 --2- 172.21.15.169:0/3266090500 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f437f80 0x558b9f439dd0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.381+0000 7fbab4575740 1 -- 172.21.15.169:0/3266090500 >> 172.21.15.169:0/3266090500 conn(0x558b9f4334f0 msgr2=0x558b9f442150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:18.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.381+0000 7fbab4575740 1 -- 172.21.15.169:0/3266090500 shutdown_connections 2024-03-28T01:03:18.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.381+0000 7fbab4575740 1 -- 172.21.15.169:0/3266090500 wait complete. 2024-03-28T01:03:18.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.381+0000 7fbab4575740 1 -- start start 2024-03-28T01:03:18.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.381+0000 7fbab4575740 1 Processor -- start 2024-03-28T01:03:18.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.382+0000 7fbab4575740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f437f80 0x558b9f4471f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.382+0000 7fbab4575740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f438970 0x558b9f447730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.382+0000 7fbab4575740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b9f43dae0 0x558b9f4486d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.382+0000 7fbab4575740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaac1aed60 con 0x558b9f437f80 2024-03-28T01:03:18.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.382+0000 7fbab4575740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaac1b0060 con 0x558b9f438970 2024-03-28T01:03:18.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.382+0000 7fbab4575740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaac1af3a0 con 0x558b9f43dae0 2024-03-28T01:03:18.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.382+0000 7fbaaaffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f437f80 0x558b9f4471f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.382+0000 7fbaaaffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f437f80 0x558b9f4471f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53284/0 (socket says 172.21.15.169:53284) 2024-03-28T01:03:18.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.382+0000 7fbaaaffd640 1 -- 172.21.15.169:0/2799934721 learned_addr learned my addr 172.21.15.169:0/2799934721 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:18.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.382+0000 7fbab4575740 10 client.0 _ll_register_callbacks cb 0x7fba9c0ba050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:18.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.382+0000 7fbaab7fe640 1 --2- 172.21.15.169:0/2799934721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b9f43dae0 0x558b9f4486d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.382+0000 7fbaaa7fc640 1 --2- 172.21.15.169:0/2799934721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f438970 0x558b9f447730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.382+0000 7fbaaaffd640 1 -- 172.21.15.169:0/2799934721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b9f43dae0 msgr2=0x558b9f4486d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:18.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.382+0000 7fbaaaffd640 1 --2- 172.21.15.169:0/2799934721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b9f43dae0 0x558b9f4486d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.382+0000 7fbaaaffd640 1 -- 172.21.15.169:0/2799934721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f438970 msgr2=0x558b9f447730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:18.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.382+0000 7fbaaaffd640 1 --2- 172.21.15.169:0/2799934721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f438970 0x558b9f447730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.382+0000 7fbaaaffd640 1 -- 172.21.15.169:0/2799934721 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558b9f448cc0 con 0x558b9f437f80 2024-03-28T01:03:18.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.382+0000 7fbaaaffd640 1 --2- 172.21.15.169:0/2799934721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f437f80 0x558b9f4471f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fbaa400a010 tx=0x7fbaa401bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4001631136199e2a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:18.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.382+0000 7fba8affd640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:03:18.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.383+0000 7fba8affd640 1 -- 172.21.15.169:0/2799934721 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa402f070 con 0x558b9f437f80 2024-03-28T01:03:18.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.383+0000 7fba8affd640 1 -- 172.21.15.169:0/2799934721 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fbaa4020c70 con 0x558b9f437f80 2024-03-28T01:03:18.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.383+0000 7fbab4575740 1 -- 172.21.15.169:0/2799934721 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbaa4025070 con 0x558b9f437f80 2024-03-28T01:03:18.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.383+0000 7fba8affd640 1 -- 172.21.15.169:0/2799934721 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbaa403b050 con 0x558b9f437f80 2024-03-28T01:03:18.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.383+0000 7fbab4575740 10 client.16791 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:18.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.383+0000 7fbab4575740 1 -- 172.21.15.169:0/2799934721 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558b9f4a9670 con 0x558b9f437f80 2024-03-28T01:03:18.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.383+0000 7fba8affd640 1 -- 172.21.15.169:0/2799934721 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbaa4025070 con 0x558b9f437f80 2024-03-28T01:03:18.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.383+0000 7fba8affd640 1 -- 172.21.15.169:0/2799934721 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fbaa400b040 con 0x558b9f437f80 2024-03-28T01:03:18.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.383+0000 7fbab4575740 10 client.16791 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:18.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.383+0000 7fbab4575740 10 client.16791 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:18.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.383+0000 7fbab4575740 20 client.16791 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:18.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.383+0000 7fbab4575740 1 --2- 172.21.15.169:0/2799934721 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558b9f4a9f40 0x558b9f4ac330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:18.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.383+0000 7fbab4575740 4 client.16791 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:03:18.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.383+0000 7fbab4575740 1 -- 172.21.15.169:0/2799934721 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558b9f3a4c60 con 0x558b9f4a9f40 2024-03-28T01:03:18.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.384+0000 7fbaaa7fc640 1 --2- 172.21.15.169:0/2799934721 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558b9f4a9f40 0x558b9f4ac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:18.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.384+0000 7fba8affd640 10 client.16791 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:18.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.384+0000 7fbaaa7fc640 1 --2- 172.21.15.169:0/2799934721 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558b9f4a9f40 0x558b9f4ac330 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=531acc593c523a37 server_cookie=346311aed6b0629f in_seq=0 out_seq=0 2024-03-28T01:03:18.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.384+0000 7fba8affd640 1 -- 172.21.15.169:0/2799934721 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x558b9f3a4c60 con 0x558b9f4a9f40 2024-03-28T01:03:18.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.384+0000 7fba8affd640 10 client.16791 handle_command_reply: tid=0 2024-03-28T01:03:18.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.384+0000 7fbab4575740 1 client.16791 shutdown 2024-03-28T01:03:18.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.384+0000 7fbab4575740 1 -- 172.21.15.169:0/2799934721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f437f80 msgr2=0x558b9f4471f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:18.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.384+0000 7fbab4575740 1 --2- 172.21.15.169:0/2799934721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f437f80 0x558b9f4471f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fbaa400a010 tx=0x7fbaa401bcf0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.384+0000 7fbab4575740 1 -- 172.21.15.169:0/2799934721 shutdown_connections 2024-03-28T01:03:18.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.384+0000 7fbab4575740 1 --2- 172.21.15.169:0/2799934721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b9f43dae0 0x558b9f4486d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.384+0000 7fbab4575740 1 --2- 172.21.15.169:0/2799934721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558b9f438970 0x558b9f447730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.384+0000 7fbab4575740 1 --2- 172.21.15.169:0/2799934721 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558b9f4a9f40 0x558b9f4ac330 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.384+0000 7fbab4575740 1 --2- 172.21.15.169:0/2799934721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558b9f437f80 0x558b9f4471f0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.384+0000 7fba8affd640 0 client.16791 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:18.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.384+0000 7fbab4575740 1 -- 172.21.15.169:0/2799934721 >> 172.21.15.169:0/2799934721 conn(0x558b9f4334f0 msgr2=0x558b9f442150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:18.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.385+0000 7fbab4575740 1 -- 172.21.15.169:0/2799934721 shutdown_connections 2024-03-28T01:03:18.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.385+0000 7fbab4575740 1 -- 172.21.15.169:0/2799934721 wait complete. 2024-03-28T01:03:18.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.385+0000 7fbab4575740 20 client.16791 trim_cache size 0 max 16384 2024-03-28T01:03:18.394 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:03:18.394 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T01:03:18.394 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": { 2024-03-28T01:03:18.395 INFO:teuthology.orchestra.run.smithi169.stdout: "c51e7a9b-4ce5-4aae-a4c3-a9d71c3f45f9": { 2024-03-28T01:03:18.395 INFO:teuthology.orchestra.run.smithi169.stdout: "path": "/", 2024-03-28T01:03:18.395 INFO:teuthology.orchestra.run.smithi169.stdout: "tag": "c51e7a9b-4ce5-4aae-a4c3-a9d71c3f45f9", 2024-03-28T01:03:18.395 INFO:teuthology.orchestra.run.smithi169.stdout: "options": "recursive,force" 2024-03-28T01:03:18.395 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:03:18.395 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:03:18.395 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:03:18.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.385+0000 7fbaabfff640 1 -- 172.21.15.169:0/1152528579 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fba940a2850 msgr2=0x7fba940a4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:18.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.385+0000 7fbaabfff640 1 --2- 172.21.15.169:0/1152528579 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fba940a2850 0x7fba940a4d10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbaa00095f0 tx=0x7fbaa0011810 comp rx=0 tx=0).stop 2024-03-28T01:03:18.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.385+0000 7fbaabfff640 1 -- 172.21.15.169:0/1152528579 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbaac102030 msgr2=0x7fbaac1a5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:18.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.385+0000 7fbaabfff640 1 --2- 172.21.15.169:0/1152528579 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbaac102030 0x7fbaac1a5430 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fba9c009780 tx=0x7fba9c0324a0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.385+0000 7fbaabfff640 1 -- 172.21.15.169:0/1152528579 shutdown_connections 2024-03-28T01:03:18.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.385+0000 7fbaabfff640 1 --2- 172.21.15.169:0/1152528579 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fba940a2850 0x7fba940a4d10 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.385+0000 7fbaabfff640 1 --2- 172.21.15.169:0/1152528579 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbaac102030 0x7fbaac1a5430 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.385+0000 7fbaabfff640 1 --2- 172.21.15.169:0/1152528579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbaac101670 0x7fbaac19e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.385+0000 7fbaabfff640 1 --2- 172.21.15.169:0/1152528579 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbaac100480 0x7fbaac19de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:18.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.385+0000 7fbaabfff640 1 -- 172.21.15.169:0/1152528579 >> 172.21.15.169:0/1152528579 conn(0x7fbaac0fba10 msgr2=0x7fbaac10b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:18.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.385+0000 7fbaabfff640 1 -- 172.21.15.169:0/1152528579 shutdown_connections 2024-03-28T01:03:18.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:18.385+0000 7fbaabfff640 1 -- 172.21.15.169:0/1152528579 wait complete. 2024-03-28T01:03:18.400 INFO:tasks.cephfs.filesystem:scrub status for tag:c51e7a9b-4ce5-4aae-a4c3-a9d71c3f45f9 - {'path': '/', 'tag': 'c51e7a9b-4ce5-4aae-a4c3-a9d71c3f45f9', 'options': 'recursive,force'} 2024-03-28T01:03:19.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:19 smithi174 ceph-mon[36060]: scrub queued for path: / 2024-03-28T01:03:19.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:19 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:03:19.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:19 smithi174 ceph-mon[36060]: scrub summary: active paths [/] 2024-03-28T01:03:19.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:19 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:03:19.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:19 smithi174 ceph-mon[36060]: pgmap v555: 129 pgs: 129 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s rd, 434 MiB/s wr, 4.05k op/s 2024-03-28T01:03:19.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:19 smithi174 ceph-mon[36060]: scrub summary: idle 2024-03-28T01:03:19.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:19 smithi177 ceph-mon[36206]: scrub queued for path: / 2024-03-28T01:03:19.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:19 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:03:19.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:19 smithi177 ceph-mon[36206]: scrub summary: active paths [/] 2024-03-28T01:03:19.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:19 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:03:19.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:19 smithi177 ceph-mon[36206]: pgmap v555: 129 pgs: 129 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s rd, 434 MiB/s wr, 4.05k op/s 2024-03-28T01:03:19.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:19 smithi177 ceph-mon[36206]: scrub summary: idle 2024-03-28T01:03:20.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:19 smithi169 ceph-mon[32274]: scrub queued for path: / 2024-03-28T01:03:20.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:19 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:03:20.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:19 smithi169 ceph-mon[32274]: scrub summary: active paths [/] 2024-03-28T01:03:20.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:19 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:03:20.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:19 smithi169 ceph-mon[32274]: pgmap v555: 129 pgs: 129 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s rd, 434 MiB/s wr, 4.05k op/s 2024-03-28T01:03:20.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:19 smithi169 ceph-mon[32274]: scrub summary: idle 2024-03-28T01:03:21.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:21 smithi174 ceph-mon[36060]: pgmap v556: 129 pgs: 129 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s rd, 381 MiB/s wr, 3.25k op/s 2024-03-28T01:03:21.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:21 smithi177 ceph-mon[36206]: pgmap v556: 129 pgs: 129 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s rd, 381 MiB/s wr, 3.25k op/s 2024-03-28T01:03:22.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:21 smithi169 ceph-mon[32274]: pgmap v556: 129 pgs: 129 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s rd, 381 MiB/s wr, 3.25k op/s 2024-03-28T01:03:23.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:23 smithi174 ceph-mon[36060]: pgmap v557: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s rd, 458 MiB/s wr, 3.66k op/s 2024-03-28T01:03:23.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:23 smithi177 ceph-mon[36206]: pgmap v557: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s rd, 458 MiB/s wr, 3.66k op/s 2024-03-28T01:03:24.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:23 smithi169 ceph-mon[32274]: pgmap v557: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s rd, 458 MiB/s wr, 3.66k op/s 2024-03-28T01:03:25.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:25 smithi174 ceph-mon[36060]: pgmap v558: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s rd, 405 MiB/s wr, 3.12k op/s 2024-03-28T01:03:25.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:25 smithi177 ceph-mon[36206]: pgmap v558: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s rd, 405 MiB/s wr, 3.12k op/s 2024-03-28T01:03:26.053 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:25 smithi169 ceph-mon[32274]: pgmap v558: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s rd, 405 MiB/s wr, 3.12k op/s 2024-03-28T01:03:27.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:27 smithi174 ceph-mon[36060]: pgmap v559: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s rd, 376 MiB/s wr, 2.71k op/s 2024-03-28T01:03:27.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:27 smithi177 ceph-mon[36206]: pgmap v559: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s rd, 376 MiB/s wr, 2.71k op/s 2024-03-28T01:03:28.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:27 smithi169 ceph-mon[32274]: pgmap v559: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s rd, 376 MiB/s wr, 2.71k op/s 2024-03-28T01:03:29.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:29 smithi174 ceph-mon[36060]: pgmap v560: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 122 MiB/s rd, 467 MiB/s wr, 3.54k op/s 2024-03-28T01:03:29.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:29 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:03:29.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:29 smithi177 ceph-mon[36206]: pgmap v560: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 122 MiB/s rd, 467 MiB/s wr, 3.54k op/s 2024-03-28T01:03:29.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:29 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:03:30.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:29 smithi169 ceph-mon[32274]: pgmap v560: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 122 MiB/s rd, 467 MiB/s wr, 3.54k op/s 2024-03-28T01:03:30.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:29 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:03:31.109 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.106+0000 7f1be6434640 1 -- 172.21.15.169:0/2092047031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1be010b2d0 msgr2=0x7f1be010d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:31.110 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.106+0000 7f1be6434640 1 --2- 172.21.15.169:0/2092047031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1be010b2d0 0x7f1be010d6e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8005fd0 tx=0x7f1bd800aec0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.107+0000 7f1be6434640 1 -- 172.21.15.169:0/2092047031 shutdown_connections 2024-03-28T01:03:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.107+0000 7f1be6434640 1 --2- 172.21.15.169:0/2092047031 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1be010dd70 0x7f1be0110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.107+0000 7f1be6434640 1 --2- 172.21.15.169:0/2092047031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1be010b2d0 0x7f1be010d6e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.107+0000 7f1be6434640 1 --2- 172.21.15.169:0/2092047031 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1be0108910 0x7f1be010ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.107+0000 7f1be6434640 1 -- 172.21.15.169:0/2092047031 >> 172.21.15.169:0/2092047031 conn(0x7f1be006e6e0 msgr2=0x7f1be0070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.108+0000 7f1be6434640 1 -- 172.21.15.169:0/2092047031 shutdown_connections 2024-03-28T01:03:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.108+0000 7f1be6434640 1 -- 172.21.15.169:0/2092047031 wait complete. 2024-03-28T01:03:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.109+0000 7f1be6434640 1 Processor -- start 2024-03-28T01:03:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.109+0000 7f1be6434640 1 -- start start 2024-03-28T01:03:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.109+0000 7f1be6434640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1be0108910 0x7f1be01a6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.109+0000 7f1be6434640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1be010b2d0 0x7f1be01a6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.109+0000 7f1be6434640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1be010dd70 0x7f1be01adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.109+0000 7f1be6434640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be0117db0 con 0x7f1be010dd70 2024-03-28T01:03:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.109+0000 7f1be6434640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be0117c30 con 0x7f1be0108910 2024-03-28T01:03:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.109+0000 7f1be6434640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be0117f30 con 0x7f1be010b2d0 2024-03-28T01:03:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.110+0000 7f1bdf7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1be010b2d0 0x7f1be01a6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.110+0000 7f1bdf7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1be010b2d0 0x7f1be01a6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:43412/0 (socket says 172.21.15.169:43412) 2024-03-28T01:03:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.110+0000 7f1bdf7fe640 1 -- 172.21.15.169:0/703178830 learned_addr learned my addr 172.21.15.169:0/703178830 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.111+0000 7f1bdf7fe640 1 -- 172.21.15.169:0/703178830 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1be0108910 msgr2=0x7f1be01a6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.111+0000 7f1bdf7fe640 1 --2- 172.21.15.169:0/703178830 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1be0108910 0x7f1be01a6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.111+0000 7f1bdf7fe640 1 -- 172.21.15.169:0/703178830 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1be010dd70 msgr2=0x7f1be01adb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.111+0000 7f1bdf7fe640 1 --2- 172.21.15.169:0/703178830 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1be010dd70 0x7f1be01adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.111+0000 7f1bdf7fe640 1 -- 172.21.15.169:0/703178830 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be01ae190 con 0x7f1be010b2d0 2024-03-28T01:03:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.111+0000 7f1bdf7fe640 1 --2- 172.21.15.169:0/703178830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1be010b2d0 0x7f1be01a6ab0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1bd802ed00 tx=0x7f1bd802ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.113+0000 7f1bdd7fa640 1 -- 172.21.15.169:0/703178830 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd8068070 con 0x7f1be010b2d0 2024-03-28T01:03:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.113+0000 7f1bdd7fa640 1 -- 172.21.15.169:0/703178830 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1bd800a900 con 0x7f1be010b2d0 2024-03-28T01:03:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.113+0000 7f1bdd7fa640 1 -- 172.21.15.169:0/703178830 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd80042a0 con 0x7f1be010b2d0 2024-03-28T01:03:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.114+0000 7f1be6434640 1 -- 172.21.15.169:0/703178830 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1be00738d0 con 0x7f1be010b2d0 2024-03-28T01:03:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.114+0000 7f1be6434640 1 -- 172.21.15.169:0/703178830 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1be0073dd0 con 0x7f1be010b2d0 2024-03-28T01:03:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.120+0000 7f1be9019740 1 Processor -- start 2024-03-28T01:03:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.120+0000 7f1bdd7fa640 1 -- 172.21.15.169:0/703178830 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1bd806b070 con 0x7f1be010b2d0 2024-03-28T01:03:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.121+0000 7f1bdd7fa640 1 --2- 172.21.15.169:0/703178830 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1bc80a2ae0 0x7f1bc80a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.121+0000 7f1bdd7fa640 1 -- 172.21.15.169:0/703178830 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f1bd8113090 con 0x7f1be010b2d0 2024-03-28T01:03:31.126 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.123+0000 7f1be9019740 1 -- start start 2024-03-28T01:03:31.126 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.123+0000 7f1be9019740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586813fc920 0x5586813fcd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:31.126 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.123+0000 7f1be9019740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fd310 0x558681401e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:31.126 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.123+0000 7f1be9019740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558681402480 0x558681404910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.123+0000 7f1be9019740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x558681361b10 con 0x5586813fd310 2024-03-28T01:03:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.123+0000 7f1be9019740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55868130a900 con 0x558681402480 2024-03-28T01:03:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.123+0000 7f1be9019740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55868135dbb0 con 0x5586813fc920 2024-03-28T01:03:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.123+0000 7f1bdf7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fd310 0x558681401e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.123+0000 7f1bdffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586813fc920 0x5586813fcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.123+0000 7f1bdf7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fd310 0x558681401e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33366/0 (socket says 172.21.15.169:33366) 2024-03-28T01:03:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.123+0000 7f1be49d1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558681402480 0x558681404910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.123+0000 7f1bdf7fe640 1 -- 172.21.15.169:0/3049904538 learned_addr learned my addr 172.21.15.169:0/3049904538 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.123+0000 7f1be49d1640 1 --2- 172.21.15.169:0/703178830 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1bc80a2ae0 0x7f1bc80a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:31.128 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.124+0000 7f1bdffff640 1 -- 172.21.15.169:0/3049904538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558681402480 msgr2=0x558681404910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:31.128 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.124+0000 7f1bdffff640 1 --2- 172.21.15.169:0/3049904538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558681402480 0x558681404910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.128 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.124+0000 7f1bdffff640 1 -- 172.21.15.169:0/3049904538 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fd310 msgr2=0x558681401e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:31.128 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.124+0000 7f1bdffff640 1 --2- 172.21.15.169:0/3049904538 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fd310 0x558681401e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.128 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.124+0000 7f1bdffff640 1 -- 172.21.15.169:0/3049904538 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558681076210 con 0x5586813fc920 2024-03-28T01:03:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.124+0000 7f1be49d1640 1 --2- 172.21.15.169:0/703178830 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1bc80a2ae0 0x7f1bc80a4fa0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1bd40058c0 tx=0x7f1bd400b2e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.125+0000 7f1bdffff640 1 --2- 172.21.15.169:0/3049904538 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586813fc920 0x5586813fcd40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0009ae0 tx=0x7f1bd000e2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=159583e6bfb7ad91 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.125+0000 7f1bc67fc640 1 -- 172.21.15.169:0/3049904538 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd0016070 con 0x5586813fc920 2024-03-28T01:03:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.125+0000 7f1bc67fc640 1 -- 172.21.15.169:0/3049904538 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1bd0005500 con 0x5586813fc920 2024-03-28T01:03:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.125+0000 7f1bc67fc640 1 -- 172.21.15.169:0/3049904538 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd0011470 con 0x5586813fc920 2024-03-28T01:03:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.126+0000 7f1be9019740 1 -- 172.21.15.169:0/3049904538 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586813fc920 msgr2=0x5586813fcd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.126+0000 7f1be9019740 1 --2- 172.21.15.169:0/3049904538 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586813fc920 0x5586813fcd40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0009ae0 tx=0x7f1bd000e2f0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.126+0000 7f1be9019740 1 -- 172.21.15.169:0/3049904538 shutdown_connections 2024-03-28T01:03:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.126+0000 7f1be9019740 1 --2- 172.21.15.169:0/3049904538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558681402480 0x558681404910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.126+0000 7f1be9019740 1 --2- 172.21.15.169:0/3049904538 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fd310 0x558681401e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.126+0000 7f1be9019740 1 --2- 172.21.15.169:0/3049904538 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586813fc920 0x5586813fcd40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.126+0000 7f1be9019740 1 -- 172.21.15.169:0/3049904538 >> 172.21.15.169:0/3049904538 conn(0x5586813f7e90 msgr2=0x5586813f8290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.127+0000 7f1be9019740 1 -- 172.21.15.169:0/3049904538 shutdown_connections 2024-03-28T01:03:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.127+0000 7f1be9019740 1 -- 172.21.15.169:0/3049904538 wait complete. 2024-03-28T01:03:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.128+0000 7f1be9019740 1 -- start start 2024-03-28T01:03:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.128+0000 7f1be9019740 1 Processor -- start 2024-03-28T01:03:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.129+0000 7f1be9019740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fc920 0x558681406c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.129+0000 7f1be9019740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586813fd310 0x558681407180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.129+0000 7f1be9019740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558681402480 0x55868140e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.129+0000 7f1be9019740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x558681405430 con 0x5586813fc920 2024-03-28T01:03:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.129+0000 7f1bdf7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586813fd310 0x558681407180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.129+0000 7f1bdf7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586813fd310 0x558681407180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:44522/0 (socket says 172.21.15.169:44522) 2024-03-28T01:03:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.129+0000 7f1bdf7fe640 1 -- 172.21.15.169:0/3839540601 learned_addr learned my addr 172.21.15.169:0/3839540601 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.129+0000 7f1be9019740 1 -- 172.21.15.169:0/3839540601 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55868131dd50 con 0x5586813fd310 2024-03-28T01:03:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.129+0000 7f1be9019740 1 -- 172.21.15.169:0/3839540601 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586814055b0 con 0x558681402480 2024-03-28T01:03:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.129+0000 7f1be9019740 10 client.0 _ll_register_callbacks cb 0x55868131d720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.129+0000 7f1be49d1640 1 --2- 172.21.15.169:0/3839540601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558681402480 0x55868140e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.130+0000 7f1bdffff640 1 --2- 172.21.15.169:0/3839540601 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fc920 0x558681406c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.130+0000 7f1be49d1640 1 -- 172.21.15.169:0/3839540601 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586813fd310 msgr2=0x558681407180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.130+0000 7f1be49d1640 1 --2- 172.21.15.169:0/3839540601 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586813fd310 0x558681407180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.130+0000 7f1be49d1640 1 -- 172.21.15.169:0/3839540601 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fc920 msgr2=0x558681406c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.130+0000 7f1be49d1640 1 --2- 172.21.15.169:0/3839540601 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fc920 0x558681406c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.130+0000 7f1be49d1640 1 -- 172.21.15.169:0/3839540601 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558681142620 con 0x558681402480 2024-03-28T01:03:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.130+0000 7f1bdf7fe640 1 --2- 172.21.15.169:0/3839540601 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586813fd310 0x558681407180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:03:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.130+0000 7f1be49d1640 1 --2- 172.21.15.169:0/3839540601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558681402480 0x55868140e1c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1bd400b670 tx=0x7f1bd400b5c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85755945a09cb307 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.130+0000 7f1bbf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:03:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.130+0000 7f1bbf7fe640 1 -- 172.21.15.169:0/3839540601 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd4016040 con 0x558681402480 2024-03-28T01:03:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.130+0000 7f1bbf7fe640 1 -- 172.21.15.169:0/3839540601 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f1bd401c940 con 0x558681402480 2024-03-28T01:03:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.130+0000 7f1be9019740 1 -- 172.21.15.169:0/3839540601 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1bd0005500 con 0x558681402480 2024-03-28T01:03:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.131+0000 7f1bbf7fe640 1 -- 172.21.15.169:0/3839540601 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1bd402e050 con 0x558681402480 2024-03-28T01:03:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.131+0000 7f1be9019740 10 client.26629 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.131+0000 7f1be9019740 1 -- 172.21.15.169:0/3839540601 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5586814113c0 con 0x558681402480 2024-03-28T01:03:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.132+0000 7f1bbf7fe640 1 -- 172.21.15.169:0/3839540601 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1bd0005500 con 0x558681402480 2024-03-28T01:03:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.132+0000 7f1bbf7fe640 1 -- 172.21.15.169:0/3839540601 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f1bd40047a0 con 0x558681402480 2024-03-28T01:03:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.132+0000 7f1be9019740 10 client.26629 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.132+0000 7f1be9019740 10 client.26629 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:31.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.132+0000 7f1be9019740 20 client.26629 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:31.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.132+0000 7f1be9019740 1 --2- 172.21.15.169:0/3839540601 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558681412b00 0x558681414f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:31.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.132+0000 7f1be9019740 4 client.26629 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:03:31.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.132+0000 7f1be9019740 1 -- 172.21.15.169:0/3839540601 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55868136f050 con 0x558681412b00 2024-03-28T01:03:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.133+0000 7f1bdffff640 1 --2- 172.21.15.169:0/3839540601 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558681412b00 0x558681414f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.133+0000 7f1bbf7fe640 10 client.26629 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.133+0000 7f1bdffff640 1 --2- 172.21.15.169:0/3839540601 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558681412b00 0x558681414f20 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d490c9858921a76 server_cookie=4441b7d35c11689a in_seq=0 out_seq=0 2024-03-28T01:03:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.134+0000 7f1bbf7fe640 1 -- 172.21.15.169:0/3839540601 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55868136f050 con 0x558681412b00 2024-03-28T01:03:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.134+0000 7f1bbf7fe640 10 client.26629 handle_command_reply: tid=0 2024-03-28T01:03:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.134+0000 7f1be9019740 1 client.26629 shutdown 2024-03-28T01:03:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.135+0000 7f1be9019740 1 -- 172.21.15.169:0/3839540601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558681402480 msgr2=0x55868140e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.135+0000 7f1be9019740 1 --2- 172.21.15.169:0/3839540601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558681402480 0x55868140e1c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1bd400b670 tx=0x7f1bd400b5c0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.135+0000 7f1be9019740 1 -- 172.21.15.169:0/3839540601 shutdown_connections 2024-03-28T01:03:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.135+0000 7f1be9019740 1 --2- 172.21.15.169:0/3839540601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558681402480 0x55868140e1c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.135+0000 7f1be9019740 1 --2- 172.21.15.169:0/3839540601 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586813fd310 0x558681407180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.140 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.135+0000 7f1be9019740 1 --2- 172.21.15.169:0/3839540601 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558681412b00 0x558681414f20 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.140 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.135+0000 7f1be9019740 1 --2- 172.21.15.169:0/3839540601 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fc920 0x558681406c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.140 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.135+0000 7f1bbf7fe640 0 client.26629 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:31.140 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.135+0000 7f1be9019740 1 -- 172.21.15.169:0/3839540601 >> 172.21.15.169:0/3839540601 conn(0x5586813f7e90 msgr2=0x5586813f7020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:31.140 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.136+0000 7f1be9019740 1 -- 172.21.15.169:0/3839540601 shutdown_connections 2024-03-28T01:03:31.140 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.137+0000 7f1be9019740 1 -- 172.21.15.169:0/3839540601 wait complete. 2024-03-28T01:03:31.140 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.137+0000 7f1be9019740 20 client.26629 trim_cache size 0 max 16384 2024-03-28T01:03:31.167 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.164+0000 7f1be9019740 1 Processor -- start 2024-03-28T01:03:31.167 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.164+0000 7f1be9019740 1 -- start start 2024-03-28T01:03:31.168 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.165+0000 7f1be9019740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586813f7e90 0x5586813f8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:31.168 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.165+0000 7f1be9019740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fc920 0x55868140e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:31.169 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.165+0000 7f1be9019740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586813fd310 0x5586813fe630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:31.169 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.165+0000 7f1be9019740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc011fb0 con 0x5586813fc920 2024-03-28T01:03:31.169 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.166+0000 7f1be9019740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc00fab0 con 0x5586813fd310 2024-03-28T01:03:31.169 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.166+0000 7f1be9019740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc00db50 con 0x5586813f7e90 2024-03-28T01:03:31.169 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.166+0000 7f1bdffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586813f7e90 0x5586813f8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:31.169 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.166+0000 7f1bdffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586813f7e90 0x5586813f8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:43432/0 (socket says 172.21.15.169:43432) 2024-03-28T01:03:31.169 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.166+0000 7f1bdffff640 1 -- 172.21.15.169:0/3458048383 learned_addr learned my addr 172.21.15.169:0/3458048383 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:31.169 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.166+0000 7f1be49d1640 1 --2- 172.21.15.169:0/3458048383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586813fd310 0x5586813fe630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:31.171 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.168+0000 7f1be49d1640 1 -- 172.21.15.169:0/3458048383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586813f7e90 msgr2=0x5586813f8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:31.171 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.168+0000 7f1be49d1640 1 --2- 172.21.15.169:0/3458048383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586813f7e90 0x5586813f8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.171 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.168+0000 7f1be49d1640 1 -- 172.21.15.169:0/3458048383 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fc920 msgr2=0x55868140e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:31.171 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.168+0000 7f1be49d1640 1 --2- 172.21.15.169:0/3458048383 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fc920 0x55868140e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.171 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.168+0000 7f1be49d1640 1 -- 172.21.15.169:0/3458048383 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558681415290 con 0x5586813fd310 2024-03-28T01:03:31.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.169+0000 7f1be49d1640 1 --2- 172.21.15.169:0/3458048383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586813fd310 0x5586813fe630 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4004c40 tx=0x7f1bd4004650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c978cdc20e391f3a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:31.177 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.173+0000 7f1bc5ffb640 1 -- 172.21.15.169:0/3458048383 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd401c9c0 con 0x5586813fd310 2024-03-28T01:03:31.177 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.173+0000 7f1bc5ffb640 1 -- 172.21.15.169:0/3458048383 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1bd4021070 con 0x5586813fd310 2024-03-28T01:03:31.178 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.175+0000 7f1be9019740 1 -- 172.21.15.169:0/3458048383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586813fd310 msgr2=0x5586813fe630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:31.178 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.175+0000 7f1be9019740 1 --2- 172.21.15.169:0/3458048383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586813fd310 0x5586813fe630 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4004c40 tx=0x7f1bd4004650 comp rx=0 tx=0).stop 2024-03-28T01:03:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.179+0000 7f1be9019740 1 -- 172.21.15.169:0/3458048383 shutdown_connections 2024-03-28T01:03:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.179+0000 7f1be9019740 1 --2- 172.21.15.169:0/3458048383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586813fd310 0x5586813fe630 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.179+0000 7f1be9019740 1 --2- 172.21.15.169:0/3458048383 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fc920 0x55868140e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.179+0000 7f1be9019740 1 --2- 172.21.15.169:0/3458048383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586813f7e90 0x5586813f8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.179+0000 7f1be9019740 1 -- 172.21.15.169:0/3458048383 >> 172.21.15.169:0/3458048383 conn(0x5586813fdf10 msgr2=0x5586813f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:31.183 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.180+0000 7f1be9019740 1 -- 172.21.15.169:0/3458048383 shutdown_connections 2024-03-28T01:03:31.184 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.180+0000 7f1be9019740 1 -- 172.21.15.169:0/3458048383 wait complete. 2024-03-28T01:03:31.184 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.180+0000 7f1be9019740 1 -- start start 2024-03-28T01:03:31.184 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.181+0000 7f1be9019740 1 Processor -- start 2024-03-28T01:03:31.184 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.181+0000 7f1be9019740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586813f7e90 0x55868140b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:31.184 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.181+0000 7f1be9019740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fc920 0x55868140bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:31.184 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.181+0000 7f1be9019740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586813fd310 0x55868140ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:31.185 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.181+0000 7f1be9019740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc00e150 con 0x5586813fc920 2024-03-28T01:03:31.185 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.181+0000 7f1be9019740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc00f450 con 0x5586813f7e90 2024-03-28T01:03:31.185 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.181+0000 7f1be9019740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc00e790 con 0x5586813fd310 2024-03-28T01:03:31.185 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.181+0000 7f1be9019740 10 client.0 _ll_register_callbacks cb 0x7f1bd4004340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:31.185 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.182+0000 7f1bdf7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fc920 0x55868140bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:31.185 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.182+0000 7f1bdf7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fc920 0x55868140bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:33392/0 (socket says 172.21.15.169:33392) 2024-03-28T01:03:31.185 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.182+0000 7f1bdf7fe640 1 -- 172.21.15.169:0/2868850677 learned_addr learned my addr 172.21.15.169:0/2868850677 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:31.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.183+0000 7f1be49d1640 1 --2- 172.21.15.169:0/2868850677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586813fd310 0x55868140ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:31.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.183+0000 7f1bdffff640 1 --2- 172.21.15.169:0/2868850677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586813f7e90 0x55868140b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:31.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.183+0000 7f1bdf7fe640 1 -- 172.21.15.169:0/2868850677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586813fd310 msgr2=0x55868140ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:31.186 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.183+0000 7f1bdf7fe640 1 --2- 172.21.15.169:0/2868850677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586813fd310 0x55868140ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.187 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.183+0000 7f1bdf7fe640 1 -- 172.21.15.169:0/2868850677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586813f7e90 msgr2=0x55868140b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:31.187 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.183+0000 7f1bdf7fe640 1 --2- 172.21.15.169:0/2868850677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586813f7e90 0x55868140b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.187 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.183+0000 7f1bdf7fe640 1 -- 172.21.15.169:0/2868850677 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55868140d440 con 0x5586813fc920 2024-03-28T01:03:31.187 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.185+0000 7f1bdf7fe640 1 --2- 172.21.15.169:0/2868850677 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fc920 0x55868140bf00 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f1bd80dba90 tx=0x7f1bd80db9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fa00f1fec031786 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:31.187 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.185+0000 7f1bbf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:03:31.188 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.186+0000 7f1bbf7fe640 1 -- 172.21.15.169:0/2868850677 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd80f84a0 con 0x5586813fc920 2024-03-28T01:03:31.189 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.186+0000 7f1be9019740 1 -- 172.21.15.169:0/2868850677 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1bd4021070 con 0x5586813fc920 2024-03-28T01:03:31.189 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.186+0000 7f1bbf7fe640 1 -- 172.21.15.169:0/2868850677 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f1bd8004b40 con 0x5586813fc920 2024-03-28T01:03:31.190 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.187+0000 7f1bbf7fe640 1 -- 172.21.15.169:0/2868850677 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1bd80df630 con 0x5586813fc920 2024-03-28T01:03:31.192 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.187+0000 7f1be9019740 10 client.16803 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:31.192 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.187+0000 7f1be9019740 1 -- 172.21.15.169:0/2868850677 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55868147f4c0 con 0x5586813fc920 2024-03-28T01:03:31.193 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.190+0000 7f1bbf7fe640 1 -- 172.21.15.169:0/2868850677 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1bd4021070 con 0x5586813fc920 2024-03-28T01:03:31.193 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.190+0000 7f1bbf7fe640 1 -- 172.21.15.169:0/2868850677 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f1bd80dec20 con 0x5586813fc920 2024-03-28T01:03:31.194 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.191+0000 7f1be9019740 10 client.16803 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:31.194 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.191+0000 7f1be9019740 10 client.16803 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:31.194 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.191+0000 7f1be9019740 20 client.16803 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:31.194 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.191+0000 7f1be9019740 1 --2- 172.21.15.169:0/2868850677 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558681402480 0x558681481de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:31.194 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.191+0000 7f1be9019740 4 client.16803 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:03:31.194 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.191+0000 7f1be9019740 1 -- 172.21.15.169:0/2868850677 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5586813693a0 con 0x558681402480 2024-03-28T01:03:31.195 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.192+0000 7f1bdffff640 1 --2- 172.21.15.169:0/2868850677 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558681402480 0x558681481de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:31.197 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.194+0000 7f1bdffff640 1 --2- 172.21.15.169:0/2868850677 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558681402480 0x558681481de0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa3e8431d3bb9f47 server_cookie=c00e2b87c4624128 in_seq=0 out_seq=0 2024-03-28T01:03:31.201 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.198+0000 7f1bbf7fe640 10 client.16803 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:31.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.253+0000 7f1bbf7fe640 1 -- 172.21.15.169:0/2868850677 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5586813693a0 con 0x558681402480 2024-03-28T01:03:31.257 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.253+0000 7f1bbf7fe640 10 client.16803 handle_command_reply: tid=0 2024-03-28T01:03:31.257 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.253+0000 7f1be9019740 1 client.16803 shutdown 2024-03-28T01:03:31.260 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.257+0000 7f1be9019740 1 -- 172.21.15.169:0/2868850677 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fc920 msgr2=0x55868140bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:31.260 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.257+0000 7f1be9019740 1 --2- 172.21.15.169:0/2868850677 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fc920 0x55868140bf00 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f1bd80dba90 tx=0x7f1bd80db9f0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.260 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.257+0000 7f1be9019740 1 -- 172.21.15.169:0/2868850677 shutdown_connections 2024-03-28T01:03:31.260 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.257+0000 7f1be9019740 1 --2- 172.21.15.169:0/2868850677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586813fd310 0x55868140ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.260 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.257+0000 7f1be9019740 1 --2- 172.21.15.169:0/2868850677 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558681402480 0x558681481de0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.260 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.257+0000 7f1be9019740 1 --2- 172.21.15.169:0/2868850677 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5586813fc920 0x55868140bf00 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.260 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.257+0000 7f1be9019740 1 --2- 172.21.15.169:0/2868850677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586813f7e90 0x55868140b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.260 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.257+0000 7f1be9019740 1 -- 172.21.15.169:0/2868850677 >> 172.21.15.169:0/2868850677 conn(0x5586813fdf10 msgr2=0x5586813f8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:31.260 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.257+0000 7f1bbf7fe640 0 client.16803 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:31.262 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.259+0000 7f1be9019740 1 -- 172.21.15.169:0/2868850677 shutdown_connections 2024-03-28T01:03:31.262 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.259+0000 7f1be9019740 1 -- 172.21.15.169:0/2868850677 wait complete. 2024-03-28T01:03:31.263 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:03:31.263 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:03:31.263 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:03:31.263 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:03:31.265 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.260+0000 7f1be9019740 20 client.16803 trim_cache size 0 max 16384 2024-03-28T01:03:31.265 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.261+0000 7f1be6434640 1 -- 172.21.15.169:0/703178830 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1bc80a2ae0 msgr2=0x7f1bc80a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:31.265 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.261+0000 7f1be6434640 1 --2- 172.21.15.169:0/703178830 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1bc80a2ae0 0x7f1bc80a4fa0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1bd40058c0 tx=0x7f1bd400b2e0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.265 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.261+0000 7f1be6434640 1 -- 172.21.15.169:0/703178830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1be010b2d0 msgr2=0x7f1be01a6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:31.265 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.261+0000 7f1be6434640 1 --2- 172.21.15.169:0/703178830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1be010b2d0 0x7f1be01a6ab0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1bd802ed00 tx=0x7f1bd802ed30 comp rx=0 tx=0).stop 2024-03-28T01:03:31.265 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.261+0000 7f1be6434640 1 -- 172.21.15.169:0/703178830 shutdown_connections 2024-03-28T01:03:31.265 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.261+0000 7f1be6434640 1 --2- 172.21.15.169:0/703178830 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1bc80a2ae0 0x7f1bc80a4fa0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.265 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.261+0000 7f1be6434640 1 --2- 172.21.15.169:0/703178830 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1be010dd70 0x7f1be01adb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.265 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.261+0000 7f1be6434640 1 --2- 172.21.15.169:0/703178830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1be010b2d0 0x7f1be01a6ab0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.265 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.261+0000 7f1be6434640 1 --2- 172.21.15.169:0/703178830 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1be0108910 0x7f1be01a6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:31.265 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.261+0000 7f1be6434640 1 -- 172.21.15.169:0/703178830 >> 172.21.15.169:0/703178830 conn(0x7f1be006e6e0 msgr2=0x7f1be010c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:31.265 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.262+0000 7f1be6434640 1 -- 172.21.15.169:0/703178830 shutdown_connections 2024-03-28T01:03:31.265 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:31.262+0000 7f1be6434640 1 -- 172.21.15.169:0/703178830 wait complete. 2024-03-28T01:03:31.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:31 smithi174 ceph-mon[36060]: pgmap v561: 129 pgs: 129 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s rd, 438 MiB/s wr, 3.52k op/s 2024-03-28T01:03:31.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:31 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:31 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:31 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:31 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:31 smithi174 ceph-mon[36060]: Adjusting osd_memory_target on smithi174 to 174.1M 2024-03-28T01:03:31.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:31 smithi174 ceph-mon[36060]: Unable to set osd_memory_target on smithi174 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-28T01:03:31.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:31 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:31 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:31 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:31 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:31 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:03:31.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:31 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:03:31.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:31 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:03:31.893 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:31 smithi169 ceph-mon[32274]: pgmap v561: 129 pgs: 129 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s rd, 438 MiB/s wr, 3.52k op/s 2024-03-28T01:03:31.893 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:31 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.893 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:31 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.893 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:31 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.893 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:31 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.893 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:31 smithi169 ceph-mon[32274]: Adjusting osd_memory_target on smithi174 to 174.1M 2024-03-28T01:03:31.894 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:31 smithi169 ceph-mon[32274]: Unable to set osd_memory_target on smithi174 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-28T01:03:31.894 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:31 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.894 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:31 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.894 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:31 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.894 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:31 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.894 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:31 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:03:31.894 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:31 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:03:31.894 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:31 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:03:31.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:31 smithi177 ceph-mon[36206]: pgmap v561: 129 pgs: 129 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s rd, 438 MiB/s wr, 3.52k op/s 2024-03-28T01:03:31.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:31 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:31 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:31 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:31 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:31 smithi177 ceph-mon[36206]: Adjusting osd_memory_target on smithi174 to 174.1M 2024-03-28T01:03:31.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:31 smithi177 ceph-mon[36206]: Unable to set osd_memory_target on smithi174 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-28T01:03:31.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:31 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:31 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:31 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:31 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:03:31.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:31 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:03:31.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:31 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:03:31.918 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:31 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:03:32.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:32 smithi174 ceph-mon[36060]: Adjusting osd_memory_target on smithi169 to 176.2M 2024-03-28T01:03:32.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:32 smithi174 ceph-mon[36060]: Unable to set osd_memory_target on smithi169 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-03-28T01:03:32.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:32 smithi177 ceph-mon[36206]: Adjusting osd_memory_target on smithi169 to 176.2M 2024-03-28T01:03:32.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:32 smithi177 ceph-mon[36206]: Unable to set osd_memory_target on smithi169 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-03-28T01:03:33.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:32 smithi169 ceph-mon[32274]: Adjusting osd_memory_target on smithi169 to 176.2M 2024-03-28T01:03:33.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:32 smithi169 ceph-mon[32274]: Unable to set osd_memory_target on smithi169 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-03-28T01:03:33.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:33 smithi177 ceph-mon[36206]: pgmap v562: 129 pgs: 129 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 118 MiB/s rd, 464 MiB/s wr, 4.19k op/s 2024-03-28T01:03:34.063 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:33 smithi169 ceph-mon[32274]: pgmap v562: 129 pgs: 129 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 118 MiB/s rd, 464 MiB/s wr, 4.19k op/s 2024-03-28T01:03:34.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:33 smithi174 ceph-mon[36060]: pgmap v562: 129 pgs: 129 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 118 MiB/s rd, 464 MiB/s wr, 4.19k op/s 2024-03-28T01:03:35.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:35 smithi177 ceph-mon[36206]: pgmap v563: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s rd, 422 MiB/s wr, 4.12k op/s 2024-03-28T01:03:36.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:35 smithi169 ceph-mon[32274]: pgmap v563: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s rd, 422 MiB/s wr, 4.12k op/s 2024-03-28T01:03:36.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:35 smithi174 ceph-mon[36060]: pgmap v563: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s rd, 422 MiB/s wr, 4.12k op/s 2024-03-28T01:03:37.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:37 smithi177 ceph-mon[36206]: pgmap v564: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s rd, 383 MiB/s wr, 3.78k op/s 2024-03-28T01:03:38.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:37 smithi169 ceph-mon[32274]: pgmap v564: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s rd, 383 MiB/s wr, 3.78k op/s 2024-03-28T01:03:38.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:37 smithi174 ceph-mon[36060]: pgmap v564: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s rd, 383 MiB/s wr, 3.78k op/s 2024-03-28T01:03:39.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:39 smithi177 ceph-mon[36206]: pgmap v565: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 126 MiB/s rd, 449 MiB/s wr, 4.35k op/s 2024-03-28T01:03:40.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:39 smithi169 ceph-mon[32274]: pgmap v565: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 126 MiB/s rd, 449 MiB/s wr, 4.35k op/s 2024-03-28T01:03:40.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:39 smithi174 ceph-mon[36060]: pgmap v565: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 126 MiB/s rd, 449 MiB/s wr, 4.35k op/s 2024-03-28T01:03:41.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:40 smithi169 ceph-mon[32274]: pgmap v566: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 131 MiB/s rd, 380 MiB/s wr, 3.71k op/s 2024-03-28T01:03:41.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:40 smithi174 ceph-mon[36060]: pgmap v566: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 131 MiB/s rd, 380 MiB/s wr, 3.71k op/s 2024-03-28T01:03:41.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:40 smithi177 ceph-mon[36206]: pgmap v566: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 131 MiB/s rd, 380 MiB/s wr, 3.71k op/s 2024-03-28T01:03:43.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:43 smithi169 ceph-mon[32274]: pgmap v567: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 155 MiB/s rd, 401 MiB/s wr, 3.93k op/s 2024-03-28T01:03:43.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:43 smithi174 ceph-mon[36060]: pgmap v567: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 155 MiB/s rd, 401 MiB/s wr, 3.93k op/s 2024-03-28T01:03:43.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:43 smithi177 ceph-mon[36206]: pgmap v567: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 155 MiB/s rd, 401 MiB/s wr, 3.93k op/s 2024-03-28T01:03:44.450 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.447+0000 7f05fd1f4640 1 -- 172.21.15.169:0/2333368374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05f81017b0 msgr2=0x7f05f8101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:44.450 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.447+0000 7f05fd1f4640 1 --2- 172.21.15.169:0/2333368374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05f81017b0 0x7f05f8101c30 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f05f000d910 tx=0x7f05f002fde0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.451 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.448+0000 7f05fd1f4640 1 -- 172.21.15.169:0/2333368374 shutdown_connections 2024-03-28T01:03:44.451 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.448+0000 7f05fd1f4640 1 --2- 172.21.15.169:0/2333368374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05f8102170 0x7f05f8108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.451 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.448+0000 7f05fd1f4640 1 --2- 172.21.15.169:0/2333368374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05f81017b0 0x7f05f8101c30 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.451 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.448+0000 7f05fd1f4640 1 --2- 172.21.15.169:0/2333368374 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05f81005c0 0x7f05f81009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.451 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.448+0000 7f05fd1f4640 1 -- 172.21.15.169:0/2333368374 >> 172.21.15.169:0/2333368374 conn(0x7f05f80fbb30 msgr2=0x7f05f80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:44.457 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.452+0000 7f05fd1f4640 1 -- 172.21.15.169:0/2333368374 shutdown_connections 2024-03-28T01:03:44.457 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.454+0000 7f05fd1f4640 1 -- 172.21.15.169:0/2333368374 wait complete. 2024-03-28T01:03:44.459 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.455+0000 7f05fd1f4640 1 Processor -- start 2024-03-28T01:03:44.462 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.458+0000 7f05fd1f4640 1 -- start start 2024-03-28T01:03:44.462 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.458+0000 7f05fd1f4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05f81005c0 0x7f05f81957b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:44.463 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.458+0000 7f05fd1f4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05f81017b0 0x7f05f8195cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:44.463 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.458+0000 7f05fd1f4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05f8102170 0x7f05f819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:44.463 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.458+0000 7f05fd1f4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f810b1f0 con 0x7f05f81005c0 2024-03-28T01:03:44.463 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.458+0000 7f05fd1f4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f810b070 con 0x7f05f81017b0 2024-03-28T01:03:44.463 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.458+0000 7f05fd1f4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f810b370 con 0x7f05f8102170 2024-03-28T01:03:44.463 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.459+0000 7f05f6d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05f81005c0 0x7f05f81957b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:44.463 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.459+0000 7f05f6d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05f81005c0 0x7f05f81957b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:45876/0 (socket says 172.21.15.169:45876) 2024-03-28T01:03:44.463 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.459+0000 7f05f6d9d640 1 -- 172.21.15.169:0/3226248147 learned_addr learned my addr 172.21.15.169:0/3226248147 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:44.463 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.460+0000 7f05f759e640 1 --2- 172.21.15.169:0/3226248147 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05f8102170 0x7f05f819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:44.464 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.460+0000 7f05f659c640 1 --2- 172.21.15.169:0/3226248147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05f81017b0 0x7f05f8195cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:44.464 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.460+0000 7f05f759e640 1 -- 172.21.15.169:0/3226248147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05f81017b0 msgr2=0x7f05f8195cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:44.464 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.460+0000 7f05f759e640 1 --2- 172.21.15.169:0/3226248147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05f81017b0 0x7f05f8195cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.464 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.460+0000 7f05f759e640 1 -- 172.21.15.169:0/3226248147 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05f81005c0 msgr2=0x7f05f81957b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:44.464 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.460+0000 7f05f759e640 1 --2- 172.21.15.169:0/3226248147 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05f81005c0 0x7f05f81957b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.464 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.461+0000 7f05f759e640 1 -- 172.21.15.169:0/3226248147 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f806b1f0 con 0x7f05f8102170 2024-03-28T01:03:44.468 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.466+0000 7f05f759e640 1 --2- 172.21.15.169:0/3226248147 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05f8102170 0x7f05f819cd60 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f05e800ca30 tx=0x7f05e800cef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:44.472 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.469+0000 7f05effff640 1 -- 172.21.15.169:0/3226248147 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e8007d90 con 0x7f05f8102170 2024-03-28T01:03:44.472 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.470+0000 7f05effff640 1 -- 172.21.15.169:0/3226248147 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05e8005020 con 0x7f05f8102170 2024-03-28T01:03:44.473 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.470+0000 7f05effff640 1 -- 172.21.15.169:0/3226248147 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e800f780 con 0x7f05f8102170 2024-03-28T01:03:44.474 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.470+0000 7f05fd1f4640 1 -- 172.21.15.169:0/3226248147 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f806b4e0 con 0x7f05f8102170 2024-03-28T01:03:44.474 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.470+0000 7f05fd1f4640 1 -- 172.21.15.169:0/3226248147 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f806ba30 con 0x7f05f8102170 2024-03-28T01:03:44.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.474+0000 7f05ffde3740 1 Processor -- start 2024-03-28T01:03:44.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.476+0000 7f05ffde3740 1 -- start start 2024-03-28T01:03:44.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.477+0000 7f05ffde3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563225f7e920 0x563225f7ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:44.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.477+0000 7f05ffde3740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563225f7f310 0x563225f83e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:44.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.477+0000 7f05ffde3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f84480 0x563225f86910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:44.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.477+0000 7f05ffde3740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x563225ee3b10 con 0x563225f7f310 2024-03-28T01:03:44.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.477+0000 7f05ffde3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x563225e8c900 con 0x563225f7e920 2024-03-28T01:03:44.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.477+0000 7f05ffde3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x563225edfbb0 con 0x563225f84480 2024-03-28T01:03:44.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.477+0000 7f05effff640 1 -- 172.21.15.169:0/3226248147 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f05e800f920 con 0x7f05f8102170 2024-03-28T01:03:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.478+0000 7f05effff640 1 --2- 172.21.15.169:0/3226248147 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f05e40a2a90 0x7f05e40a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.478+0000 7f05effff640 1 -- 172.21.15.169:0/3226248147 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f05e80ceea0 con 0x7f05f8102170 2024-03-28T01:03:44.484 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.480+0000 7f05f659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f84480 0x563225f86910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:44.484 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.480+0000 7f05f659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f84480 0x563225f86910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:43880/0 (socket says 172.21.15.169:43880) 2024-03-28T01:03:44.484 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.480+0000 7f05f659c640 1 -- 172.21.15.169:0/1532464621 learned_addr learned my addr 172.21.15.169:0/1532464621 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:44.484 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.480+0000 7f05f659c640 1 --2- 172.21.15.169:0/1532464621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563225f7e920 0x563225f7ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:44.484 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.480+0000 7f05f6d9d640 1 --2- 172.21.15.169:0/1532464621 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563225f7f310 0x563225f83e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:44.485 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.481+0000 7f05f659c640 1 -- 172.21.15.169:0/1532464621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563225f7e920 msgr2=0x563225f7ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:44.485 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.481+0000 7f05f659c640 1 --2- 172.21.15.169:0/1532464621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563225f7e920 0x563225f7ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.485 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.481+0000 7f05f659c640 1 -- 172.21.15.169:0/1532464621 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563225f7f310 msgr2=0x563225f83e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:44.485 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.481+0000 7f05f659c640 1 --2- 172.21.15.169:0/1532464621 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563225f7f310 0x563225f83e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.481+0000 7f05f659c640 1 -- 172.21.15.169:0/1532464621 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563225bf8210 con 0x563225f84480 2024-03-28T01:03:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.484+0000 7f05f759e640 1 --2- 172.21.15.169:0/3226248147 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f05e40a2a90 0x7f05e40a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.485+0000 7f05f659c640 1 --2- 172.21.15.169:0/1532464621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f84480 0x563225f86910 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f05f000d1c0 tx=0x7f05f000d130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dba0d440c91e0f90 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.487+0000 7f05ed7fa640 1 -- 172.21.15.169:0/1532464621 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f0004030 con 0x563225f84480 2024-03-28T01:03:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.487+0000 7f05ed7fa640 1 -- 172.21.15.169:0/1532464621 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05f0034070 con 0x563225f84480 2024-03-28T01:03:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.487+0000 7f05ed7fa640 1 -- 172.21.15.169:0/1532464621 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f0005050 con 0x563225f84480 2024-03-28T01:03:44.492 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.489+0000 7f05ffde3740 1 -- 172.21.15.169:0/1532464621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f84480 msgr2=0x563225f86910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:44.492 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.489+0000 7f05ffde3740 1 --2- 172.21.15.169:0/1532464621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f84480 0x563225f86910 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f05f000d1c0 tx=0x7f05f000d130 comp rx=0 tx=0).stop 2024-03-28T01:03:44.496 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.490+0000 7f05f759e640 1 --2- 172.21.15.169:0/3226248147 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f05e40a2a90 0x7f05e40a4f50 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f05e809c310 tx=0x7f05e8098270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:44.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.494+0000 7f05ffde3740 1 -- 172.21.15.169:0/1532464621 shutdown_connections 2024-03-28T01:03:44.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.494+0000 7f05ffde3740 1 --2- 172.21.15.169:0/1532464621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f84480 0x563225f86910 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.494+0000 7f05ffde3740 1 --2- 172.21.15.169:0/1532464621 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563225f7f310 0x563225f83e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.494+0000 7f05ffde3740 1 --2- 172.21.15.169:0/1532464621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563225f7e920 0x563225f7ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.494+0000 7f05ffde3740 1 -- 172.21.15.169:0/1532464621 >> 172.21.15.169:0/1532464621 conn(0x563225f79e90 msgr2=0x563225f7a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:44.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.496+0000 7f05ffde3740 1 -- 172.21.15.169:0/1532464621 shutdown_connections 2024-03-28T01:03:44.503 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.500+0000 7f05ffde3740 1 -- 172.21.15.169:0/1532464621 wait complete. 2024-03-28T01:03:44.503 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.500+0000 7f05ffde3740 1 -- start start 2024-03-28T01:03:44.503 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.500+0000 7f05ffde3740 1 Processor -- start 2024-03-28T01:03:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.501+0000 7f05ffde3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f7e920 0x563225f91440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.501+0000 7f05ffde3740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563225f7f310 0x563225f91980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:44.506 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.501+0000 7f05ffde3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563225f84480 0x563225f989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:44.506 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.501+0000 7f05ffde3740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x563225f87370 con 0x563225f7f310 2024-03-28T01:03:44.506 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.501+0000 7f05ffde3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x563225e9fd50 con 0x563225f84480 2024-03-28T01:03:44.506 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.501+0000 7f05ffde3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x563225f874f0 con 0x563225f7e920 2024-03-28T01:03:44.506 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.501+0000 7f05ffde3740 10 client.0 _ll_register_callbacks cb 0x563225e9f720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:44.508 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.503+0000 7f05f6d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563225f7f310 0x563225f91980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:44.508 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.503+0000 7f05f6d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563225f7f310 0x563225f91980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:45898/0 (socket says 172.21.15.169:45898) 2024-03-28T01:03:44.508 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.503+0000 7f05f6d9d640 1 -- 172.21.15.169:0/784805426 learned_addr learned my addr 172.21.15.169:0/784805426 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:44.509 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.504+0000 7f05f659c640 1 --2- 172.21.15.169:0/784805426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f7e920 0x563225f91440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:44.509 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.504+0000 7f05f659c640 1 --2- 172.21.15.169:0/784805426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563225f84480 0x563225f989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:44.509 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.505+0000 7f05f6d9d640 1 -- 172.21.15.169:0/784805426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f7e920 msgr2=0x563225f91440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:44.509 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.505+0000 7f05f6d9d640 1 --2- 172.21.15.169:0/784805426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f7e920 0x563225f91440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.509 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.505+0000 7f05f6d9d640 1 -- 172.21.15.169:0/784805426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563225f84480 msgr2=0x563225f989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:44.509 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.505+0000 7f05f6d9d640 1 --2- 172.21.15.169:0/784805426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563225f84480 0x563225f989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.509 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.505+0000 7f05f6d9d640 1 -- 172.21.15.169:0/784805426 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563225cc4620 con 0x563225f7f310 2024-03-28T01:03:44.509 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.505+0000 7f05f6d9d640 1 --2- 172.21.15.169:0/784805426 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563225f7f310 0x563225f91980 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f05e0009ef0 tx=0x7f05e0011630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=353ffadfa1df9d06 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.506+0000 7f05de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:03:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.506+0000 7f05de7fc640 1 -- 172.21.15.169:0/784805426 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e0019070 con 0x563225f7f310 2024-03-28T01:03:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.506+0000 7f05de7fc640 1 -- 172.21.15.169:0/784805426 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f05e0015660 con 0x563225f7f310 2024-03-28T01:03:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.506+0000 7f05ffde3740 1 -- 172.21.15.169:0/784805426 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f05f0034070 con 0x563225f7f310 2024-03-28T01:03:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.510+0000 7f05de7fc640 1 -- 172.21.15.169:0/784805426 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05e0025050 con 0x563225f7f310 2024-03-28T01:03:44.519 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.516+0000 7f05ffde3740 10 client.16815 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:44.519 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.516+0000 7f05ffde3740 1 -- 172.21.15.169:0/784805426 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563225f8b830 con 0x563225f7f310 2024-03-28T01:03:44.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.518+0000 7f05de7fc640 1 -- 172.21.15.169:0/784805426 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05f0034070 con 0x563225f7f310 2024-03-28T01:03:44.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.518+0000 7f05de7fc640 1 -- 172.21.15.169:0/784805426 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f05e0014b70 con 0x563225f7f310 2024-03-28T01:03:44.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.520+0000 7f05ffde3740 10 client.16815 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:44.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.520+0000 7f05ffde3740 10 client.16815 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.520+0000 7f05ffde3740 20 client.16815 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.520+0000 7f05ffde3740 1 --2- 172.21.15.169:0/784805426 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563225f8cf50 0x563225f8f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.520+0000 7f05ffde3740 4 client.16815 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:03:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.520+0000 7f05ffde3740 1 -- 172.21.15.169:0/784805426 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563225ef1050 con 0x563225f8cf50 2024-03-28T01:03:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.524+0000 7f05f659c640 1 --2- 172.21.15.169:0/784805426 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563225f8cf50 0x563225f8f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.526+0000 7f05f659c640 1 --2- 172.21.15.169:0/784805426 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563225f8cf50 0x563225f8f390 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9df0bf0ad95c47bb server_cookie=33d3085c784878b2 in_seq=0 out_seq=0 2024-03-28T01:03:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.526+0000 7f05de7fc640 10 client.16815 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.533+0000 7f05de7fc640 1 -- 172.21.15.169:0/784805426 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563225ef1050 con 0x563225f8cf50 2024-03-28T01:03:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.533+0000 7f05de7fc640 10 client.16815 handle_command_reply: tid=0 2024-03-28T01:03:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.534+0000 7f05ffde3740 1 client.16815 shutdown 2024-03-28T01:03:44.538 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.536+0000 7f05ffde3740 1 -- 172.21.15.169:0/784805426 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563225f7f310 msgr2=0x563225f91980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:44.538 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.536+0000 7f05ffde3740 1 --2- 172.21.15.169:0/784805426 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563225f7f310 0x563225f91980 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f05e0009ef0 tx=0x7f05e0011630 comp rx=0 tx=0).stop 2024-03-28T01:03:44.539 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.536+0000 7f05ffde3740 1 -- 172.21.15.169:0/784805426 shutdown_connections 2024-03-28T01:03:44.539 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.536+0000 7f05ffde3740 1 --2- 172.21.15.169:0/784805426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563225f84480 0x563225f989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.539 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.536+0000 7f05ffde3740 1 --2- 172.21.15.169:0/784805426 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563225f8cf50 0x563225f8f390 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.539 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.536+0000 7f05ffde3740 1 --2- 172.21.15.169:0/784805426 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563225f7f310 0x563225f91980 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.539 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.536+0000 7f05ffde3740 1 --2- 172.21.15.169:0/784805426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f7e920 0x563225f91440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.539 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.536+0000 7f05ffde3740 1 -- 172.21.15.169:0/784805426 >> 172.21.15.169:0/784805426 conn(0x563225f79e90 msgr2=0x563225f79020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:44.539 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.536+0000 7f05de7fc640 0 client.16815 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:44.541 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.538+0000 7f05ffde3740 1 -- 172.21.15.169:0/784805426 shutdown_connections 2024-03-28T01:03:44.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.540+0000 7f05ffde3740 1 -- 172.21.15.169:0/784805426 wait complete. 2024-03-28T01:03:44.546 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.543+0000 7f05ffde3740 20 client.16815 trim_cache size 0 max 16384 2024-03-28T01:03:44.574 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.571+0000 7f05ffde3740 1 Processor -- start 2024-03-28T01:03:44.575 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.572+0000 7f05ffde3740 1 -- start start 2024-03-28T01:03:44.575 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.572+0000 7f05ffde3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f7e920 0x563225f7fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:44.575 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.572+0000 7f05ffde3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563225f7f310 0x563225f98980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:44.575 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.572+0000 7f05ffde3740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563225f84480 0x563225f7d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:44.575 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.572+0000 7f05ffde3740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d80120e0 con 0x563225f84480 2024-03-28T01:03:44.575 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.572+0000 7f05ffde3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d800fbe0 con 0x563225f7f310 2024-03-28T01:03:44.576 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.572+0000 7f05ffde3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d800dc80 con 0x563225f7e920 2024-03-28T01:03:44.577 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.573+0000 7f05f6d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563225f7f310 0x563225f98980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:44.577 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.573+0000 7f05f6d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563225f7f310 0x563225f98980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:59140/0 (socket says 172.21.15.169:59140) 2024-03-28T01:03:44.577 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.573+0000 7f05f6d9d640 1 -- 172.21.15.169:0/1978252201 learned_addr learned my addr 172.21.15.169:0/1978252201 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:44.578 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.575+0000 7f05f6d9d640 1 -- 172.21.15.169:0/1978252201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f7e920 msgr2=0x563225f7fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:44.578 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.575+0000 7f05f6d9d640 1 --2- 172.21.15.169:0/1978252201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f7e920 0x563225f7fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.579 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.575+0000 7f05f6d9d640 1 -- 172.21.15.169:0/1978252201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563225f84480 msgr2=0x563225f7d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:44.579 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.575+0000 7f05f6d9d640 1 --2- 172.21.15.169:0/1978252201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563225f84480 0x563225f7d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.579 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.575+0000 7f05f6d9d640 1 -- 172.21.15.169:0/1978252201 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563225f7f6f0 con 0x563225f7f310 2024-03-28T01:03:44.581 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.578+0000 7f05f6d9d640 1 --2- 172.21.15.169:0/1978252201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563225f7f310 0x563225f98980 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f05e0000ea0 tx=0x7f05e0006ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=108b05486be3f314 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:44.582 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.579+0000 7f05ecff9640 1 -- 172.21.15.169:0/1978252201 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e002e070 con 0x563225f7f310 2024-03-28T01:03:44.582 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.579+0000 7f05ecff9640 1 -- 172.21.15.169:0/1978252201 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05e001d6b0 con 0x563225f7f310 2024-03-28T01:03:44.583 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.579+0000 7f05ecff9640 1 -- 172.21.15.169:0/1978252201 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e0019040 con 0x563225f7f310 2024-03-28T01:03:44.583 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.580+0000 7f05ffde3740 1 -- 172.21.15.169:0/1978252201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563225f7f310 msgr2=0x563225f98980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:44.583 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.580+0000 7f05ffde3740 1 --2- 172.21.15.169:0/1978252201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563225f7f310 0x563225f98980 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f05e0000ea0 tx=0x7f05e0006ea0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.582+0000 7f05ffde3740 1 -- 172.21.15.169:0/1978252201 shutdown_connections 2024-03-28T01:03:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.582+0000 7f05ffde3740 1 --2- 172.21.15.169:0/1978252201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563225f84480 0x563225f7d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.582+0000 7f05ffde3740 1 --2- 172.21.15.169:0/1978252201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563225f7f310 0x563225f98980 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.582+0000 7f05ffde3740 1 --2- 172.21.15.169:0/1978252201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f7e920 0x563225f7fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.582+0000 7f05ffde3740 1 -- 172.21.15.169:0/1978252201 >> 172.21.15.169:0/1978252201 conn(0x563225f79e90 msgr2=0x563225f7a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:44.587 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.585+0000 7f05ffde3740 1 -- 172.21.15.169:0/1978252201 shutdown_connections 2024-03-28T01:03:44.588 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.585+0000 7f05ffde3740 1 -- 172.21.15.169:0/1978252201 wait complete. 2024-03-28T01:03:44.588 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.586+0000 7f05ffde3740 1 -- start start 2024-03-28T01:03:44.588 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.586+0000 7f05ffde3740 1 Processor -- start 2024-03-28T01:03:44.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.586+0000 7f05ffde3740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563225f7e920 0x563225f985d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:44.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.586+0000 7f05ffde3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f7f310 0x563225f95130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:44.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.586+0000 7f05ffde3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563225f84480 0x563225f95670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:44.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.586+0000 7f05ffde3740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d800e280 con 0x563225f7e920 2024-03-28T01:03:44.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.586+0000 7f05ffde3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d800f580 con 0x563225f84480 2024-03-28T01:03:44.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.586+0000 7f05ffde3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d800e8c0 con 0x563225f7f310 2024-03-28T01:03:44.589 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.586+0000 7f05ffde3740 10 client.0 _ll_register_callbacks cb 0x7f05e0009b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:44.590 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.586+0000 7f05f6d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f7f310 0x563225f95130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:44.590 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.586+0000 7f05f6d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f7f310 0x563225f95130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:43920/0 (socket says 172.21.15.169:43920) 2024-03-28T01:03:44.591 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.587+0000 7f05f6d9d640 1 -- 172.21.15.169:0/2595079346 learned_addr learned my addr 172.21.15.169:0/2595079346 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:44.592 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.589+0000 7f05f6d9d640 1 -- 172.21.15.169:0/2595079346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563225f84480 msgr2=0x563225f95670 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:03:44.592 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.589+0000 7f05f6d9d640 1 --2- 172.21.15.169:0/2595079346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563225f84480 0x563225f95670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.592 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.589+0000 7f05f6d9d640 1 -- 172.21.15.169:0/2595079346 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563225f7e920 msgr2=0x563225f985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.589+0000 7f05f6d9d640 1 --2- 172.21.15.169:0/2595079346 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563225f7e920 0x563225f985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.589+0000 7f05f6d9d640 1 -- 172.21.15.169:0/2595079346 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563225f95be0 con 0x563225f7f310 2024-03-28T01:03:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.589+0000 7f05f6d9d640 1 --2- 172.21.15.169:0/2595079346 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f7f310 0x563225f95130 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f05e0009c10 tx=0x7f05e0005c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98a6d0f87d29f975 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.589+0000 7f05de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:03:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.590+0000 7f05de7fc640 1 -- 172.21.15.169:0/2595079346 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e0019070 con 0x563225f7f310 2024-03-28T01:03:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.590+0000 7f05ffde3740 1 -- 172.21.15.169:0/2595079346 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f05e001d6b0 con 0x563225f7f310 2024-03-28T01:03:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.590+0000 7f05de7fc640 1 -- 172.21.15.169:0/2595079346 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f05e001da70 con 0x563225f7f310 2024-03-28T01:03:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.590+0000 7f05de7fc640 1 -- 172.21.15.169:0/2595079346 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05e002e070 con 0x563225f7f310 2024-03-28T01:03:44.597 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.594+0000 7f05ffde3740 10 client.26641 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:44.597 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.594+0000 7f05ffde3740 1 -- 172.21.15.169:0/2595079346 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563226001af0 con 0x563225f7f310 2024-03-28T01:03:44.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.597+0000 7f05de7fc640 1 -- 172.21.15.169:0/2595079346 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05e001d6b0 con 0x563225f7f310 2024-03-28T01:03:44.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.597+0000 7f05de7fc640 1 -- 172.21.15.169:0/2595079346 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f05e0014580 con 0x563225f7f310 2024-03-28T01:03:44.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.597+0000 7f05ffde3740 10 client.26641 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:44.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.597+0000 7f05ffde3740 10 client.26641 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:44.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.597+0000 7f05ffde3740 20 client.26641 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:44.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.597+0000 7f05ffde3740 1 --2- 172.21.15.169:0/2595079346 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5632260022c0 0x5632260046e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:44.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.597+0000 7f05ffde3740 4 client.26641 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:03:44.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.597+0000 7f05ffde3740 1 -- 172.21.15.169:0/2595079346 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563225eeb3a0 con 0x5632260022c0 2024-03-28T01:03:44.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.598+0000 7f05f659c640 1 --2- 172.21.15.169:0/2595079346 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5632260022c0 0x5632260046e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:44.607 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.603+0000 7f05f659c640 1 --2- 172.21.15.169:0/2595079346 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5632260022c0 0x5632260046e0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc246407c1b8749d server_cookie=77a6d37ef474e2b0 in_seq=0 out_seq=0 2024-03-28T01:03:44.607 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:44.604+0000 7f05de7fc640 10 client.26641 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:45.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:45 smithi169 ceph-mon[32274]: pgmap v568: 129 pgs: 129 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 168 MiB/s rd, 406 MiB/s wr, 4.08k op/s 2024-03-28T01:03:45.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:45 smithi174 ceph-mon[36060]: pgmap v568: 129 pgs: 129 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 168 MiB/s rd, 406 MiB/s wr, 4.08k op/s 2024-03-28T01:03:45.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:45 smithi177 ceph-mon[36206]: pgmap v568: 129 pgs: 129 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 168 MiB/s rd, 406 MiB/s wr, 4.08k op/s 2024-03-28T01:03:47.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:47 smithi169 ceph-mon[32274]: pgmap v569: 129 pgs: 129 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 161 MiB/s rd, 360 MiB/s wr, 3.72k op/s 2024-03-28T01:03:47.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:47 smithi174 ceph-mon[36060]: pgmap v569: 129 pgs: 129 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 161 MiB/s rd, 360 MiB/s wr, 3.72k op/s 2024-03-28T01:03:47.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:47 smithi177 ceph-mon[36206]: pgmap v569: 129 pgs: 129 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 161 MiB/s rd, 360 MiB/s wr, 3.72k op/s 2024-03-28T01:03:48.401 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:03:48.401 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:03:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.611+0000 7fb7dd401640 1 -- 172.21.15.169:0/1654571886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7d8072a80 msgr2=0x7fb7d8072ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.611+0000 7fb7dd401640 1 --2- 172.21.15.169:0/1654571886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7d8072a80 0x7fb7d8072ea0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb7c801a910 tx=0x7fb7c803cd40 comp rx=0 tx=0).stop 2024-03-28T01:03:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.611+0000 7fb7dd401640 1 -- 172.21.15.169:0/1654571886 shutdown_connections 2024-03-28T01:03:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.611+0000 7fb7dd401640 1 --2- 172.21.15.169:0/1654571886 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7d8109750 0x7fb7d810e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.611+0000 7fb7dd401640 1 --2- 172.21.15.169:0/1654571886 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7d8108db0 0x7fb7d8109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.611+0000 7fb7dd401640 1 --2- 172.21.15.169:0/1654571886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7d8072a80 0x7fb7d8072ea0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.611+0000 7fb7dd401640 1 -- 172.21.15.169:0/1654571886 >> 172.21.15.169:0/1654571886 conn(0x7fb7d806e580 msgr2=0x7fb7d80709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.611+0000 7fb7dd401640 1 -- 172.21.15.169:0/1654571886 shutdown_connections 2024-03-28T01:03:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.611+0000 7fb7dd401640 1 -- 172.21.15.169:0/1654571886 wait complete. 2024-03-28T01:03:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.612+0000 7fb7dd401640 1 Processor -- start 2024-03-28T01:03:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.612+0000 7fb7dd401640 1 -- start start 2024-03-28T01:03:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.613+0000 7fb7dd401640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7d8072a80 0x7fb7d819df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.613+0000 7fb7dd401640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7d8108db0 0x7fb7d819e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.613+0000 7fb7dd401640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7d8109750 0x7fb7d81a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.613+0000 7fb7dd401640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d81118f0 con 0x7fb7d8072a80 2024-03-28T01:03:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.613+0000 7fb7dd401640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d8111770 con 0x7fb7d8109750 2024-03-28T01:03:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.613+0000 7fb7dd401640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d8111a70 con 0x7fb7d8108db0 2024-03-28T01:03:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.613+0000 7fb7d6ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7d8072a80 0x7fb7d819df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.613+0000 7fb7d6ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7d8072a80 0x7fb7d819df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38228/0 (socket says 172.21.15.169:38228) 2024-03-28T01:03:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.613+0000 7fb7d6ffd640 1 -- 172.21.15.169:0/3756073070 learned_addr learned my addr 172.21.15.169:0/3756073070 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.613+0000 7fb7d77fe640 1 --2- 172.21.15.169:0/3756073070 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7d8109750 0x7fb7d81a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:48.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.613+0000 7fb7d67fc640 1 --2- 172.21.15.169:0/3756073070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7d8108db0 0x7fb7d819e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.614+0000 7fb7d77fe640 1 -- 172.21.15.169:0/3756073070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7d8108db0 msgr2=0x7fb7d819e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.614+0000 7fb7d77fe640 1 --2- 172.21.15.169:0/3756073070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7d8108db0 0x7fb7d819e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.614+0000 7fb7d77fe640 1 -- 172.21.15.169:0/3756073070 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7d8072a80 msgr2=0x7fb7d819df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:48.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.614+0000 7fb7d77fe640 1 --2- 172.21.15.169:0/3756073070 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7d8072a80 0x7fb7d819df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:48.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.614+0000 7fb7d77fe640 1 -- 172.21.15.169:0/3756073070 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d81a5b60 con 0x7fb7d8109750 2024-03-28T01:03:48.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.617+0000 7fb7d77fe640 1 --2- 172.21.15.169:0/3756073070 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7d8109750 0x7fb7d81a54f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb7d000be00 tx=0x7fb7d000bef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:48.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.620+0000 7fb7bffff640 1 -- 172.21.15.169:0/3756073070 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0019070 con 0x7fb7d8109750 2024-03-28T01:03:48.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.620+0000 7fb7bffff640 1 -- 172.21.15.169:0/3756073070 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7d000cd90 con 0x7fb7d8109750 2024-03-28T01:03:48.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.620+0000 7fb7bffff640 1 -- 172.21.15.169:0/3756073070 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d00128b0 con 0x7fb7d8109750 2024-03-28T01:03:48.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.621+0000 7fb7dd401640 1 -- 172.21.15.169:0/3756073070 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d81a5e50 con 0x7fb7d8109750 2024-03-28T01:03:48.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.621+0000 7fb7dd401640 1 -- 172.21.15.169:0/3756073070 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d81a6260 con 0x7fb7d8109750 2024-03-28T01:03:48.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.624+0000 7fb7dffea740 1 Processor -- start 2024-03-28T01:03:48.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.625+0000 7fb7dffea740 1 -- start start 2024-03-28T01:03:48.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.625+0000 7fb7dffea740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562b158baf80 0x562b158bb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:48.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.625+0000 7fb7dffea740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158bb970 0x562b158c0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:48.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.625+0000 7fb7dffea740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158c0ae0 0x562b158c2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:48.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.625+0000 7fb7dffea740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b158203d0 con 0x562b158c0ae0 2024-03-28T01:03:48.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.625+0000 7fb7dffea740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b157c8ad0 con 0x562b158bb970 2024-03-28T01:03:48.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.625+0000 7fb7dffea740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1581c470 con 0x562b158baf80 2024-03-28T01:03:48.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.628+0000 7fb7d67fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158c0ae0 0x562b158c2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:48.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.628+0000 7fb7d67fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158c0ae0 0x562b158c2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38244/0 (socket says 172.21.15.169:38244) 2024-03-28T01:03:48.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.628+0000 7fb7d67fc640 1 -- 172.21.15.169:0/1228153128 learned_addr learned my addr 172.21.15.169:0/1228153128 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:48.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.628+0000 7fb7d67fc640 1 --2- 172.21.15.169:0/1228153128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562b158baf80 0x562b158bb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:48.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.628+0000 7fb7d6ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158bb970 0x562b158c0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:48.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.632+0000 7fb7bffff640 1 -- 172.21.15.169:0/3756073070 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb7d0012a50 con 0x7fb7d8109750 2024-03-28T01:03:48.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.632+0000 7fb7d67fc640 1 -- 172.21.15.169:0/1228153128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562b158baf80 msgr2=0x562b158bb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:48.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.632+0000 7fb7d67fc640 1 --2- 172.21.15.169:0/1228153128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562b158baf80 0x562b158bb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:48.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.632+0000 7fb7d67fc640 1 -- 172.21.15.169:0/1228153128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158bb970 msgr2=0x562b158c0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:48.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.632+0000 7fb7d67fc640 1 --2- 172.21.15.169:0/1228153128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158bb970 0x562b158c0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:48.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.632+0000 7fb7d67fc640 1 -- 172.21.15.169:0/1228153128 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b155fe7d0 con 0x562b158c0ae0 2024-03-28T01:03:48.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.633+0000 7fb7bffff640 1 --2- 172.21.15.169:0/3756073070 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb7c00a2950 0x7fb7c00a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:48.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.633+0000 7fb7bffff640 1 -- 172.21.15.169:0/3756073070 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb7d0098b60 con 0x7fb7d8109750 2024-03-28T01:03:48.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.636+0000 7fb7d6ffd640 1 --2- 172.21.15.169:0/3756073070 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb7c00a2950 0x7fb7c00a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:48.640 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.638+0000 7fb7d67fc640 1 --2- 172.21.15.169:0/1228153128 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158c0ae0 0x562b158c2f70 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc009fd0 tx=0x7fb7cc005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=668822eacd39c8b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:48.640 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.638+0000 7fb7d6ffd640 1 --2- 172.21.15.169:0/3756073070 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb7c00a2950 0x7fb7c00a4e10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x562b158bc3e0 tx=0x7fb7c80053d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:48.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.640+0000 7fb7bd7fa640 1 -- 172.21.15.169:0/1228153128 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc004030 con 0x562b158c0ae0 2024-03-28T01:03:48.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.640+0000 7fb7bd7fa640 1 -- 172.21.15.169:0/1228153128 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7cc00fc70 con 0x562b158c0ae0 2024-03-28T01:03:48.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.640+0000 7fb7bd7fa640 1 -- 172.21.15.169:0/1228153128 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc0038e0 con 0x562b158c0ae0 2024-03-28T01:03:48.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.646+0000 7fb7dffea740 1 -- 172.21.15.169:0/1228153128 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158c0ae0 msgr2=0x562b158c2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:48.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.646+0000 7fb7dffea740 1 --2- 172.21.15.169:0/1228153128 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158c0ae0 0x562b158c2f70 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc009fd0 tx=0x7fb7cc005850 comp rx=0 tx=0).stop 2024-03-28T01:03:48.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.650+0000 7fb7dffea740 1 -- 172.21.15.169:0/1228153128 shutdown_connections 2024-03-28T01:03:48.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.650+0000 7fb7dffea740 1 --2- 172.21.15.169:0/1228153128 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158c0ae0 0x562b158c2f70 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:48.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.650+0000 7fb7dffea740 1 --2- 172.21.15.169:0/1228153128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158bb970 0x562b158c0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:48.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.650+0000 7fb7dffea740 1 --2- 172.21.15.169:0/1228153128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562b158baf80 0x562b158bb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:48.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.650+0000 7fb7dffea740 1 -- 172.21.15.169:0/1228153128 >> 172.21.15.169:0/1228153128 conn(0x562b158b64f0 msgr2=0x562b158b68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:48.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.654+0000 7fb7dffea740 1 -- 172.21.15.169:0/1228153128 shutdown_connections 2024-03-28T01:03:48.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.657+0000 7fb7dffea740 1 -- 172.21.15.169:0/1228153128 wait complete. 2024-03-28T01:03:48.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.657+0000 7fb7dffea740 1 -- start start 2024-03-28T01:03:48.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.657+0000 7fb7dffea740 1 Processor -- start 2024-03-28T01:03:48.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.660+0000 7fb7dffea740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158baf80 0x562b158c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:48.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.660+0000 7fb7dffea740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562b158bb970 0x562b158c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:48.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.660+0000 7fb7dffea740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158c0ae0 0x562b158cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:48.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.660+0000 7fb7dffea740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b158c3a90 con 0x562b158c0ae0 2024-03-28T01:03:48.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.660+0000 7fb7dffea740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b157dc060 con 0x562b158baf80 2024-03-28T01:03:48.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.660+0000 7fb7dffea740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b158c3c10 con 0x562b158bb970 2024-03-28T01:03:48.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.660+0000 7fb7dffea740 10 client.0 _ll_register_callbacks cb 0x562b157be6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:48.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.664+0000 7fb7d77fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158c0ae0 0x562b158cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:48.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.664+0000 7fb7d77fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158c0ae0 0x562b158cc9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38258/0 (socket says 172.21.15.169:38258) 2024-03-28T01:03:48.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.664+0000 7fb7d77fe640 1 -- 172.21.15.169:0/3646514497 learned_addr learned my addr 172.21.15.169:0/3646514497 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:48.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.664+0000 7fb7d67fc640 1 --2- 172.21.15.169:0/3646514497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562b158bb970 0x562b158c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:48.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.664+0000 7fb7d6ffd640 1 --2- 172.21.15.169:0/3646514497 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158baf80 0x562b158c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:48.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.665+0000 7fb7d67fc640 1 -- 172.21.15.169:0/3646514497 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158baf80 msgr2=0x562b158c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:48.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.665+0000 7fb7d67fc640 1 --2- 172.21.15.169:0/3646514497 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158baf80 0x562b158c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:48.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.665+0000 7fb7d67fc640 1 -- 172.21.15.169:0/3646514497 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158c0ae0 msgr2=0x562b158cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:48.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.665+0000 7fb7d67fc640 1 --2- 172.21.15.169:0/3646514497 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158c0ae0 0x562b158cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:48.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.665+0000 7fb7d67fc640 1 -- 172.21.15.169:0/3646514497 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562b155343f0 con 0x562b158bb970 2024-03-28T01:03:48.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.666+0000 7fb7d67fc640 1 --2- 172.21.15.169:0/3646514497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562b158bb970 0x562b158c59a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc01d000 tx=0x7fb7cc00fbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25e8eb2c25ac28e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:48.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.666+0000 7fb7baff5640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:03:48.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.667+0000 7fb7baff5640 1 -- 172.21.15.169:0/3646514497 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc0114c0 con 0x562b158bb970 2024-03-28T01:03:48.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.667+0000 7fb7baff5640 1 -- 172.21.15.169:0/3646514497 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb7cc005040 con 0x562b158bb970 2024-03-28T01:03:48.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.668+0000 7fb7dffea740 1 -- 172.21.15.169:0/3646514497 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb7cc00fc70 con 0x562b158bb970 2024-03-28T01:03:48.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.668+0000 7fb7baff5640 1 -- 172.21.15.169:0/3646514497 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7cc005260 con 0x562b158bb970 2024-03-28T01:03:48.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.669+0000 7fb7dffea740 10 client.26653 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:48.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.669+0000 7fb7dffea740 1 -- 172.21.15.169:0/3646514497 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x562b154f40f0 con 0x562b158bb970 2024-03-28T01:03:48.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.671+0000 7fb7baff5640 1 -- 172.21.15.169:0/3646514497 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7cc00fc70 con 0x562b158bb970 2024-03-28T01:03:48.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.671+0000 7fb7baff5640 1 -- 172.21.15.169:0/3646514497 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb7cc034690 con 0x562b158bb970 2024-03-28T01:03:48.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.675+0000 7fb7dffea740 10 client.26653 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:48.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.675+0000 7fb7dffea740 10 client.26653 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:48.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.675+0000 7fb7dffea740 20 client.26653 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:48.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.675+0000 7fb7dffea740 1 --2- 172.21.15.169:0/3646514497 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562b158d0c50 0x562b158d3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:48.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.675+0000 7fb7dffea740 4 client.26653 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:03:48.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.675+0000 7fb7dffea740 1 -- 172.21.15.169:0/3646514497 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562b1582d910 con 0x562b158d0c50 2024-03-28T01:03:48.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.679+0000 7fb7d6ffd640 1 --2- 172.21.15.169:0/3646514497 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562b158d0c50 0x562b158d3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:48.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.685+0000 7fb7d6ffd640 1 --2- 172.21.15.169:0/3646514497 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562b158d0c50 0x562b158d3040 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e63d8e600daa060e server_cookie=2aba9fd259bb997a in_seq=0 out_seq=0 2024-03-28T01:03:48.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:48.685+0000 7fb7baff5640 10 client.26653 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:49.262 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.259+0000 7f05de7fc640 1 -- 172.21.15.169:0/2595079346 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563225eeb3a0 con 0x5632260022c0 2024-03-28T01:03:49.262 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.259+0000 7f05de7fc640 10 client.26641 handle_command_reply: tid=0 2024-03-28T01:03:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:03:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:03:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:03:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:03:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.259+0000 7f05ffde3740 1 client.26641 shutdown 2024-03-28T01:03:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.259+0000 7f05ffde3740 1 -- 172.21.15.169:0/2595079346 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f7f310 msgr2=0x563225f95130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.259+0000 7f05ffde3740 1 --2- 172.21.15.169:0/2595079346 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f7f310 0x563225f95130 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f05e0009c10 tx=0x7f05e0005c10 comp rx=0 tx=0).stop 2024-03-28T01:03:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.259+0000 7f05ffde3740 1 -- 172.21.15.169:0/2595079346 shutdown_connections 2024-03-28T01:03:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.259+0000 7f05ffde3740 1 --2- 172.21.15.169:0/2595079346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563225f84480 0x563225f95670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.259+0000 7f05ffde3740 1 --2- 172.21.15.169:0/2595079346 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563225f7f310 0x563225f95130 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.259+0000 7f05ffde3740 1 --2- 172.21.15.169:0/2595079346 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5632260022c0 0x5632260046e0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.259+0000 7f05ffde3740 1 --2- 172.21.15.169:0/2595079346 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563225f7e920 0x563225f985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.259+0000 7f05ffde3740 1 -- 172.21.15.169:0/2595079346 >> 172.21.15.169:0/2595079346 conn(0x563225f79e90 msgr2=0x563225f806d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.259+0000 7f05de7fc640 0 client.26641 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.259+0000 7f05ffde3740 1 -- 172.21.15.169:0/2595079346 shutdown_connections 2024-03-28T01:03:49.265 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.259+0000 7f05ffde3740 1 -- 172.21.15.169:0/2595079346 wait complete. 2024-03-28T01:03:49.265 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.260+0000 7f05ffde3740 20 client.26641 trim_cache size 0 max 16384 2024-03-28T01:03:49.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.260+0000 7fb7baff5640 1 -- 172.21.15.169:0/3646514497 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562b1582d910 con 0x562b158d0c50 2024-03-28T01:03:49.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.260+0000 7fb7baff5640 10 client.26653 handle_command_reply: tid=0 2024-03-28T01:03:49.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.261+0000 7fb7dffea740 1 client.26653 shutdown 2024-03-28T01:03:49.266 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.261+0000 7f05fd1f4640 1 -- 172.21.15.169:0/3226248147 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f05e40a2a90 msgr2=0x7f05e40a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:49.266 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.261+0000 7f05fd1f4640 1 --2- 172.21.15.169:0/3226248147 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f05e40a2a90 0x7f05e40a4f50 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f05e809c310 tx=0x7f05e8098270 comp rx=0 tx=0).stop 2024-03-28T01:03:49.266 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.261+0000 7f05fd1f4640 1 -- 172.21.15.169:0/3226248147 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05f8102170 msgr2=0x7f05f819cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:49.266 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.261+0000 7f05fd1f4640 1 --2- 172.21.15.169:0/3226248147 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05f8102170 0x7f05f819cd60 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f05e800ca30 tx=0x7f05e800cef0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.261+0000 7fb7dffea740 1 -- 172.21.15.169:0/3646514497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562b158bb970 msgr2=0x562b158c59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:49.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.261+0000 7fb7dffea740 1 --2- 172.21.15.169:0/3646514497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562b158bb970 0x562b158c59a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc01d000 tx=0x7fb7cc00fbe0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.267 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.261+0000 7f05fd1f4640 1 -- 172.21.15.169:0/3226248147 shutdown_connections 2024-03-28T01:03:49.267 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.261+0000 7f05fd1f4640 1 --2- 172.21.15.169:0/3226248147 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f05e40a2a90 0x7f05e40a4f50 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.267 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.261+0000 7f05fd1f4640 1 --2- 172.21.15.169:0/3226248147 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05f8102170 0x7f05f819cd60 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.267 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.261+0000 7f05fd1f4640 1 --2- 172.21.15.169:0/3226248147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05f81017b0 0x7f05f8195cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.267 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.261+0000 7f05fd1f4640 1 --2- 172.21.15.169:0/3226248147 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05f81005c0 0x7f05f81957b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.261+0000 7fb7dffea740 1 -- 172.21.15.169:0/3646514497 shutdown_connections 2024-03-28T01:03:49.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.261+0000 7fb7dffea740 1 --2- 172.21.15.169:0/3646514497 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562b158d0c50 0x562b158d3040 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.261+0000 7fb7dffea740 1 --2- 172.21.15.169:0/3646514497 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158c0ae0 0x562b158cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.261+0000 7fb7dffea740 1 --2- 172.21.15.169:0/3646514497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562b158bb970 0x562b158c59a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.261+0000 7fb7dffea740 1 --2- 172.21.15.169:0/3646514497 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158baf80 0x562b158c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.261+0000 7fb7dffea740 1 -- 172.21.15.169:0/3646514497 >> 172.21.15.169:0/3646514497 conn(0x562b158b64f0 msgr2=0x562b158b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:49.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.261+0000 7fb7baff5640 0 client.26653 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:49.268 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.261+0000 7f05fd1f4640 1 -- 172.21.15.169:0/3226248147 >> 172.21.15.169:0/3226248147 conn(0x7f05f80fbb30 msgr2=0x7f05f8104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:49.269 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.261+0000 7f05fd1f4640 1 -- 172.21.15.169:0/3226248147 shutdown_connections 2024-03-28T01:03:49.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.261+0000 7fb7dffea740 1 -- 172.21.15.169:0/3646514497 shutdown_connections 2024-03-28T01:03:49.269 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:03:49.261+0000 7f05fd1f4640 1 -- 172.21.15.169:0/3226248147 wait complete. 2024-03-28T01:03:49.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.261+0000 7fb7dffea740 1 -- 172.21.15.169:0/3646514497 wait complete. 2024-03-28T01:03:49.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.262+0000 7fb7dffea740 20 client.26653 trim_cache size 0 max 16384 2024-03-28T01:03:49.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.301+0000 7fb7dffea740 1 Processor -- start 2024-03-28T01:03:49.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.301+0000 7fb7dffea740 1 -- start start 2024-03-28T01:03:49.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.302+0000 7fb7dffea740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158baf80 0x562b158b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:49.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.302+0000 7fb7dffea740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158bb970 0x562b158cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:49.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.302+0000 7fb7d6ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158baf80 0x562b158b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:49.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.302+0000 7fb7d6ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158baf80 0x562b158b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38266/0 (socket says 172.21.15.169:38266) 2024-03-28T01:03:49.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.302+0000 7fb7d6ffd640 1 -- 172.21.15.169:0/2277330593 learned_addr learned my addr 172.21.15.169:0/2277330593 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:49.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.302+0000 7fb7dffea740 1 --2- 172.21.15.169:0/2277330593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562b158c0ae0 0x562b158b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:49.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.303+0000 7fb7d67fc640 1 --2- 172.21.15.169:0/2277330593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158bb970 0x562b158cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:49.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.303+0000 7fb7dffea740 1 -- 172.21.15.169:0/2277330593 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d81b2c80 con 0x562b158baf80 2024-03-28T01:03:49.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.303+0000 7fb7dffea740 1 -- 172.21.15.169:0/2277330593 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d81b0780 con 0x562b158bb970 2024-03-28T01:03:49.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.303+0000 7fb7dffea740 1 -- 172.21.15.169:0/2277330593 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d81ae820 con 0x562b158c0ae0 2024-03-28T01:03:49.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.306+0000 7fb7d77fe640 1 --2- 172.21.15.169:0/2277330593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562b158c0ae0 0x562b158b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:49.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.307+0000 7fb7d67fc640 1 -- 172.21.15.169:0/2277330593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562b158c0ae0 msgr2=0x562b158b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:49.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.307+0000 7fb7d67fc640 1 --2- 172.21.15.169:0/2277330593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562b158c0ae0 0x562b158b5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.307+0000 7fb7d67fc640 1 -- 172.21.15.169:0/2277330593 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158baf80 msgr2=0x562b158b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:49.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.307+0000 7fb7d67fc640 1 --2- 172.21.15.169:0/2277330593 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158baf80 0x562b158b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.307+0000 7fb7d67fc640 1 -- 172.21.15.169:0/2277330593 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b158c3d50 con 0x562b158bb970 2024-03-28T01:03:49.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.309+0000 7fb7d67fc640 1 --2- 172.21.15.169:0/2277330593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158bb970 0x562b158cc9a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc034f80 tx=0x7fb7cc034d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d97084f76e9c78b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:49.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.309+0000 7fb7bcff9640 1 -- 172.21.15.169:0/2277330593 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc039070 con 0x562b158bb970 2024-03-28T01:03:49.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.310+0000 7fb7bcff9640 1 -- 172.21.15.169:0/2277330593 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7cc00fc50 con 0x562b158bb970 2024-03-28T01:03:49.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.310+0000 7fb7bcff9640 1 -- 172.21.15.169:0/2277330593 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc0112b0 con 0x562b158bb970 2024-03-28T01:03:49.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.314+0000 7fb7dffea740 1 -- 172.21.15.169:0/2277330593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158bb970 msgr2=0x562b158cc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:49.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.314+0000 7fb7dffea740 1 --2- 172.21.15.169:0/2277330593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158bb970 0x562b158cc9a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc034f80 tx=0x7fb7cc034d80 comp rx=0 tx=0).stop 2024-03-28T01:03:49.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.315+0000 7fb7dffea740 1 -- 172.21.15.169:0/2277330593 shutdown_connections 2024-03-28T01:03:49.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.315+0000 7fb7dffea740 1 --2- 172.21.15.169:0/2277330593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562b158c0ae0 0x562b158b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.315+0000 7fb7dffea740 1 --2- 172.21.15.169:0/2277330593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158bb970 0x562b158cc9a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.315+0000 7fb7dffea740 1 --2- 172.21.15.169:0/2277330593 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158baf80 0x562b158b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.315+0000 7fb7dffea740 1 -- 172.21.15.169:0/2277330593 >> 172.21.15.169:0/2277330593 conn(0x562b158b64f0 msgr2=0x562b158b68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:49.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.318+0000 7fb7dffea740 1 -- 172.21.15.169:0/2277330593 shutdown_connections 2024-03-28T01:03:49.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.318+0000 7fb7dffea740 1 -- 172.21.15.169:0/2277330593 wait complete. 2024-03-28T01:03:49.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.319+0000 7fb7dffea740 1 -- start start 2024-03-28T01:03:49.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.319+0000 7fb7dffea740 1 Processor -- start 2024-03-28T01:03:49.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.319+0000 7fb7dffea740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158baf80 0x562b158ca570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:49.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.319+0000 7fb7dffea740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562b158bb970 0x562b158caab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:49.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.319+0000 7fb7dffea740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158c0ae0 0x562b158cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:49.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.319+0000 7fb7dffea740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d81aee20 con 0x562b158c0ae0 2024-03-28T01:03:49.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.319+0000 7fb7dffea740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d81b0120 con 0x562b158baf80 2024-03-28T01:03:49.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.319+0000 7fb7dffea740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d81af460 con 0x562b158bb970 2024-03-28T01:03:49.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.319+0000 7fb7dffea740 10 client.0 _ll_register_callbacks cb 0x7fb7c801a790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:49.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.320+0000 7fb7d6ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158baf80 0x562b158ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:49.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.320+0000 7fb7d6ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158baf80 0x562b158ca570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:34416/0 (socket says 172.21.15.169:34416) 2024-03-28T01:03:49.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.320+0000 7fb7d6ffd640 1 -- 172.21.15.169:0/4106671333 learned_addr learned my addr 172.21.15.169:0/4106671333 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:49.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.320+0000 7fb7d67fc640 1 --2- 172.21.15.169:0/4106671333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562b158bb970 0x562b158caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:49.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.320+0000 7fb7d6ffd640 1 -- 172.21.15.169:0/4106671333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562b158bb970 msgr2=0x562b158caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:49.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.320+0000 7fb7d6ffd640 1 --2- 172.21.15.169:0/4106671333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562b158bb970 0x562b158caab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.320+0000 7fb7d6ffd640 1 -- 172.21.15.169:0/4106671333 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158c0ae0 msgr2=0x562b158cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:49.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.320+0000 7fb7d6ffd640 1 --2- 172.21.15.169:0/4106671333 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158c0ae0 0x562b158cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.320+0000 7fb7d6ffd640 1 -- 172.21.15.169:0/4106671333 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562b158cc0b0 con 0x562b158baf80 2024-03-28T01:03:49.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.321+0000 7fb7d6ffd640 1 --2- 172.21.15.169:0/4106671333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158baf80 0x562b158ca570 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8000c00 tx=0x7fb7c803cd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae2479d880e7c1e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:49.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.321+0000 7fb7baff5640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:03:49.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.321+0000 7fb7baff5640 1 -- 172.21.15.169:0/4106671333 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8010060 con 0x562b158baf80 2024-03-28T01:03:49.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.321+0000 7fb7dffea740 1 -- 172.21.15.169:0/4106671333 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb7cc00fc50 con 0x562b158baf80 2024-03-28T01:03:49.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.321+0000 7fb7baff5640 1 -- 172.21.15.169:0/4106671333 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb7c8003b00 con 0x562b158baf80 2024-03-28T01:03:49.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.321+0000 7fb7baff5640 1 -- 172.21.15.169:0/4106671333 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7c8017050 con 0x562b158baf80 2024-03-28T01:03:49.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.322+0000 7fb7dffea740 10 client.26714 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:49.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.322+0000 7fb7dffea740 1 -- 172.21.15.169:0/4106671333 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x562b1592c8f0 con 0x562b158baf80 2024-03-28T01:03:49.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.323+0000 7fb7baff5640 1 -- 172.21.15.169:0/4106671333 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7cc00fc50 con 0x562b158baf80 2024-03-28T01:03:49.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.323+0000 7fb7baff5640 1 -- 172.21.15.169:0/4106671333 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb7c8051430 con 0x562b158baf80 2024-03-28T01:03:49.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.324+0000 7fb7dffea740 10 client.26714 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:49.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.324+0000 7fb7dffea740 10 client.26714 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:49.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.324+0000 7fb7dffea740 20 client.26714 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:49.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.324+0000 7fb7dffea740 1 --2- 172.21.15.169:0/4106671333 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562b1592d1d0 0x562b1592f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:49.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.324+0000 7fb7dffea740 4 client.26714 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:03:49.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.324+0000 7fb7dffea740 1 -- 172.21.15.169:0/4106671333 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562b15827c60 con 0x562b1592d1d0 2024-03-28T01:03:49.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.325+0000 7fb7d67fc640 1 --2- 172.21.15.169:0/4106671333 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562b1592d1d0 0x562b1592f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:49.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.327+0000 7fb7d67fc640 1 --2- 172.21.15.169:0/4106671333 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562b1592d1d0 0x562b1592f5c0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a29a53f1f993cdef server_cookie=9e7f9b779d73c6d0 in_seq=0 out_seq=0 2024-03-28T01:03:49.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.327+0000 7fb7baff5640 10 client.26714 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:49.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.327+0000 7fb7baff5640 1 -- 172.21.15.169:0/4106671333 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x562b15827c60 con 0x562b1592d1d0 2024-03-28T01:03:49.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.327+0000 7fb7baff5640 10 client.26714 handle_command_reply: tid=0 2024-03-28T01:03:49.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.327+0000 7fb7dffea740 1 client.26714 shutdown 2024-03-28T01:03:49.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.332+0000 7fb7dffea740 1 -- 172.21.15.169:0/4106671333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158baf80 msgr2=0x562b158ca570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:49.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.332+0000 7fb7dffea740 1 --2- 172.21.15.169:0/4106671333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158baf80 0x562b158ca570 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8000c00 tx=0x7fb7c803cd40 comp rx=0 tx=0).stop 2024-03-28T01:03:49.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.332+0000 7fb7dffea740 1 -- 172.21.15.169:0/4106671333 shutdown_connections 2024-03-28T01:03:49.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.332+0000 7fb7dffea740 1 --2- 172.21.15.169:0/4106671333 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562b1592d1d0 0x562b1592f5c0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.332+0000 7fb7dffea740 1 --2- 172.21.15.169:0/4106671333 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562b158c0ae0 0x562b158cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.332+0000 7fb7dffea740 1 --2- 172.21.15.169:0/4106671333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562b158bb970 0x562b158caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.332+0000 7fb7dffea740 1 --2- 172.21.15.169:0/4106671333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b158baf80 0x562b158ca570 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.332+0000 7fb7dffea740 1 -- 172.21.15.169:0/4106671333 >> 172.21.15.169:0/4106671333 conn(0x562b158b64f0 msgr2=0x562b158b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:49.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.333+0000 7fb7baff5640 0 client.26714 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:49.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.334+0000 7fb7dffea740 1 -- 172.21.15.169:0/4106671333 shutdown_connections 2024-03-28T01:03:49.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.334+0000 7fb7dffea740 1 -- 172.21.15.169:0/4106671333 wait complete. 2024-03-28T01:03:49.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.335+0000 7fb7dffea740 20 client.26714 trim_cache size 0 max 16384 2024-03-28T01:03:49.340 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:03:49.340 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "no active scrubs running", 2024-03-28T01:03:49.340 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": {} 2024-03-28T01:03:49.341 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:03:49.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.338+0000 7fb7dd401640 1 -- 172.21.15.169:0/3756073070 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb7c00a2950 msgr2=0x7fb7c00a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:49.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.338+0000 7fb7dd401640 1 --2- 172.21.15.169:0/3756073070 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb7c00a2950 0x7fb7c00a4e10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x562b158bc3e0 tx=0x7fb7c80053d0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.338+0000 7fb7dd401640 1 -- 172.21.15.169:0/3756073070 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7d8109750 msgr2=0x7fb7d81a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:49.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.338+0000 7fb7dd401640 1 --2- 172.21.15.169:0/3756073070 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7d8109750 0x7fb7d81a54f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb7d000be00 tx=0x7fb7d000bef0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.339+0000 7fb7dd401640 1 -- 172.21.15.169:0/3756073070 shutdown_connections 2024-03-28T01:03:49.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.339+0000 7fb7dd401640 1 --2- 172.21.15.169:0/3756073070 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb7c00a2950 0x7fb7c00a4e10 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.339+0000 7fb7dd401640 1 --2- 172.21.15.169:0/3756073070 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7d8109750 0x7fb7d81a54f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.339+0000 7fb7dd401640 1 --2- 172.21.15.169:0/3756073070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7d8108db0 0x7fb7d819e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.339+0000 7fb7dd401640 1 --2- 172.21.15.169:0/3756073070 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb7d8072a80 0x7fb7d819df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.339+0000 7fb7dd401640 1 -- 172.21.15.169:0/3756073070 >> 172.21.15.169:0/3756073070 conn(0x7fb7d806e580 msgr2=0x7fb7d810c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:49.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.339+0000 7fb7dd401640 1 -- 172.21.15.169:0/3756073070 shutdown_connections 2024-03-28T01:03:49.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.339+0000 7fb7dd401640 1 -- 172.21.15.169:0/3756073070 wait complete. 2024-03-28T01:03:49.367 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T01:03:49.367 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T01:03:49.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.503+0000 7fc391d90640 1 -- 172.21.15.169:0/540460242 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc38c102170 msgr2=0x7fc38c108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:49.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.503+0000 7fc391d90640 1 --2- 172.21.15.169:0/540460242 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc38c102170 0x7fc38c108a00 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fc384009ca0 tx=0x7fc38402c0f0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.503+0000 7fc391d90640 1 -- 172.21.15.169:0/540460242 shutdown_connections 2024-03-28T01:03:49.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.503+0000 7fc391d90640 1 --2- 172.21.15.169:0/540460242 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc38c102170 0x7fc38c108a00 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.503+0000 7fc391d90640 1 --2- 172.21.15.169:0/540460242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc38c1017b0 0x7fc38c101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.503+0000 7fc391d90640 1 --2- 172.21.15.169:0/540460242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc38c1005c0 0x7fc38c1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.503+0000 7fc391d90640 1 -- 172.21.15.169:0/540460242 >> 172.21.15.169:0/540460242 conn(0x7fc38c0fbb30 msgr2=0x7fc38c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:49.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.503+0000 7fc391d90640 1 -- 172.21.15.169:0/540460242 shutdown_connections 2024-03-28T01:03:49.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.503+0000 7fc391d90640 1 -- 172.21.15.169:0/540460242 wait complete. 2024-03-28T01:03:49.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.504+0000 7fc391d90640 1 Processor -- start 2024-03-28T01:03:49.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.504+0000 7fc391d90640 1 -- start start 2024-03-28T01:03:49.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.504+0000 7fc391d90640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc38c1005c0 0x7fc38c104550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:49.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.504+0000 7fc391d90640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc38c1017b0 0x7fc38c102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:49.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.504+0000 7fc391d90640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc38c102170 0x7fc38c1030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:49.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.504+0000 7fc391d90640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c10c240 con 0x7fc38c102170 2024-03-28T01:03:49.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.504+0000 7fc391d90640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c10c0c0 con 0x7fc38c1005c0 2024-03-28T01:03:49.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.504+0000 7fc391d90640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c10c3c0 con 0x7fc38c1017b0 2024-03-28T01:03:49.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.504+0000 7fc38bfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc38c102170 0x7fc38c1030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:49.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.504+0000 7fc38bfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc38c102170 0x7fc38c1030d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38294/0 (socket says 172.21.15.169:38294) 2024-03-28T01:03:49.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.504+0000 7fc38bfff640 1 -- 172.21.15.169:0/3150672997 learned_addr learned my addr 172.21.15.169:0/3150672997 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:49.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.504+0000 7fc38affd640 1 --2- 172.21.15.169:0/3150672997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc38c1017b0 0x7fc38c102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:49.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.504+0000 7fc38b7fe640 1 --2- 172.21.15.169:0/3150672997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc38c1005c0 0x7fc38c104550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:49.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.504+0000 7fc38bfff640 1 -- 172.21.15.169:0/3150672997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc38c1017b0 msgr2=0x7fc38c102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:49.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.504+0000 7fc38bfff640 1 --2- 172.21.15.169:0/3150672997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc38c1017b0 0x7fc38c102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.504+0000 7fc38bfff640 1 -- 172.21.15.169:0/3150672997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc38c1005c0 msgr2=0x7fc38c104550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:49.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.504+0000 7fc38bfff640 1 --2- 172.21.15.169:0/3150672997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc38c1005c0 0x7fc38c104550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.505+0000 7fc38bfff640 1 -- 172.21.15.169:0/3150672997 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38c103960 con 0x7fc38c102170 2024-03-28T01:03:49.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.505+0000 7fc38b7fe640 1 --2- 172.21.15.169:0/3150672997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc38c1005c0 0x7fc38c104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:03:49.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.505+0000 7fc38bfff640 1 --2- 172.21.15.169:0/3150672997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc38c102170 0x7fc38c1030d0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc384037040 tx=0x7fc384032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:49.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.505+0000 7fc388ff9640 1 -- 172.21.15.169:0/3150672997 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc384041020 con 0x7fc38c102170 2024-03-28T01:03:49.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.505+0000 7fc391d90640 1 -- 172.21.15.169:0/3150672997 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc38c1a1a60 con 0x7fc38c102170 2024-03-28T01:03:49.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.505+0000 7fc391d90640 1 -- 172.21.15.169:0/3150672997 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc38c1a2000 con 0x7fc38c102170 2024-03-28T01:03:49.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.505+0000 7fc388ff9640 1 -- 172.21.15.169:0/3150672997 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc384034d90 con 0x7fc38c102170 2024-03-28T01:03:49.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.505+0000 7fc388ff9640 1 -- 172.21.15.169:0/3150672997 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc384039040 con 0x7fc38c102170 2024-03-28T01:03:49.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.506+0000 7fc391d90640 1 -- 172.21.15.169:0/3150672997 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc38c1a2230 con 0x7fc38c102170 2024-03-28T01:03:49.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.508+0000 7fc388ff9640 1 -- 172.21.15.169:0/3150672997 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc384044020 con 0x7fc38c102170 2024-03-28T01:03:49.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.508+0000 7fc388ff9640 1 --2- 172.21.15.169:0/3150672997 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc3740a2800 0x7fc3740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:49.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.508+0000 7fc388ff9640 1 -- 172.21.15.169:0/3150672997 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc3840f0d40 con 0x7fc38c102170 2024-03-28T01:03:49.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.508+0000 7fc38b7fe640 1 --2- 172.21.15.169:0/3150672997 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc3740a2800 0x7fc3740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:49.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.509+0000 7fc38b7fe640 1 --2- 172.21.15.169:0/3150672997 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc3740a2800 0x7fc3740a4cc0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc38c101610 tx=0x7fc37c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:49.519 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.516+0000 7fc388ff9640 1 -- 172.21.15.169:0/3150672997 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc38404b020 con 0x7fc38c102170 2024-03-28T01:03:49.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:49 smithi169 ceph-mon[32274]: pgmap v570: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 169 MiB/s rd, 426 MiB/s wr, 4.21k op/s 2024-03-28T01:03:49.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:49 smithi174 ceph-mon[36060]: pgmap v570: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 169 MiB/s rd, 426 MiB/s wr, 4.21k op/s 2024-03-28T01:03:49.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:49 smithi177 ceph-mon[36206]: pgmap v570: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 169 MiB/s rd, 426 MiB/s wr, 4.21k op/s 2024-03-28T01:03:49.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.940+0000 7fc391d90640 1 -- 172.21.15.169:0/3150672997 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc38c1a2400 con 0x7fc38c102170 2024-03-28T01:03:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T01:03:49.946 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T01:03:49.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.943+0000 7fc388ff9640 1 -- 172.21.15.169:0/3150672997 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7fc3840bda40 con 0x7fc38c102170 2024-03-28T01:03:49.946 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T01:03:49.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.947+0000 7fc391d90640 1 -- 172.21.15.169:0/3150672997 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc3740a2800 msgr2=0x7fc3740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:49.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.947+0000 7fc391d90640 1 --2- 172.21.15.169:0/3150672997 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc3740a2800 0x7fc3740a4cc0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc38c101610 tx=0x7fc37c005f90 comp rx=0 tx=0).stop 2024-03-28T01:03:49.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.947+0000 7fc391d90640 1 -- 172.21.15.169:0/3150672997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc38c102170 msgr2=0x7fc38c1030d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:49.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.947+0000 7fc391d90640 1 --2- 172.21.15.169:0/3150672997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc38c102170 0x7fc38c1030d0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc384037040 tx=0x7fc384032f20 comp rx=0 tx=0).stop 2024-03-28T01:03:49.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.947+0000 7fc391d90640 1 -- 172.21.15.169:0/3150672997 shutdown_connections 2024-03-28T01:03:49.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.947+0000 7fc391d90640 1 --2- 172.21.15.169:0/3150672997 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc3740a2800 0x7fc3740a4cc0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.947+0000 7fc391d90640 1 --2- 172.21.15.169:0/3150672997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc38c102170 0x7fc38c1030d0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.947+0000 7fc391d90640 1 --2- 172.21.15.169:0/3150672997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc38c1017b0 0x7fc38c102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.947+0000 7fc391d90640 1 --2- 172.21.15.169:0/3150672997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc38c1005c0 0x7fc38c104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:49.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.947+0000 7fc391d90640 1 -- 172.21.15.169:0/3150672997 >> 172.21.15.169:0/3150672997 conn(0x7fc38c0fbb30 msgr2=0x7fc38c0fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:49.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.948+0000 7fc391d90640 1 -- 172.21.15.169:0/3150672997 shutdown_connections 2024-03-28T01:03:49.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:49.948+0000 7fc391d90640 1 -- 172.21.15.169:0/3150672997 wait complete. 2024-03-28T01:03:49.975 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:03:49.976 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T01:03:50.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.209+0000 7f480b219640 1 -- 172.21.15.169:0/3543791373 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4804107cc0 msgr2=0x7f48041080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:50.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.209+0000 7f480b219640 1 --2- 172.21.15.169:0/3543791373 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4804107cc0 0x7f48041080e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f47ec009860 tx=0x7f47ec02bb90 comp rx=0 tx=0).stop 2024-03-28T01:03:50.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.211+0000 7f480b219640 1 -- 172.21.15.169:0/3543791373 shutdown_connections 2024-03-28T01:03:50.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.212+0000 7f480b219640 1 --2- 172.21.15.169:0/3543791373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4804109870 0x7f4804110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.212+0000 7f480b219640 1 --2- 172.21.15.169:0/3543791373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4804108eb0 0x7f4804109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.212+0000 7f480b219640 1 --2- 172.21.15.169:0/3543791373 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4804107cc0 0x7f48041080e0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.212+0000 7f480b219640 1 -- 172.21.15.169:0/3543791373 >> 172.21.15.169:0/3543791373 conn(0x7f4804076c60 msgr2=0x7f48040790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:50.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.214+0000 7f480b219640 1 -- 172.21.15.169:0/3543791373 shutdown_connections 2024-03-28T01:03:50.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.216+0000 7f480b219640 1 -- 172.21.15.169:0/3543791373 wait complete. 2024-03-28T01:03:50.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.217+0000 7f480b219640 1 Processor -- start 2024-03-28T01:03:50.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.217+0000 7f480b219640 1 -- start start 2024-03-28T01:03:50.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.217+0000 7f480b219640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4804107cc0 0x7f480419e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.218+0000 7f480b219640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4804108eb0 0x7f480419e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.218+0000 7f480b219640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4804109870 0x7f48041a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.218+0000 7f480b219640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4804113960 con 0x7f4804107cc0 2024-03-28T01:03:50.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.218+0000 7f480b219640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48041137e0 con 0x7f4804108eb0 2024-03-28T01:03:50.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.218+0000 7f480b219640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4804113ae0 con 0x7f4804109870 2024-03-28T01:03:50.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.218+0000 7f48097b6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4804109870 0x7f48041a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.218+0000 7f48097b6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4804109870 0x7f48041a5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:39542/0 (socket says 172.21.15.169:39542) 2024-03-28T01:03:50.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.218+0000 7f48097b6640 1 -- 172.21.15.169:0/4076797828 learned_addr learned my addr 172.21.15.169:0/4076797828 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:50.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.219+0000 7f4808fb5640 1 --2- 172.21.15.169:0/4076797828 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4804107cc0 0x7f480419e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.219+0000 7f48097b6640 1 -- 172.21.15.169:0/4076797828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4804108eb0 msgr2=0x7f480419e5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:03:50.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.219+0000 7f48097b6640 1 --2- 172.21.15.169:0/4076797828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4804108eb0 0x7f480419e5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.219+0000 7f48097b6640 1 -- 172.21.15.169:0/4076797828 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4804107cc0 msgr2=0x7f480419e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:50.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.219+0000 7f48097b6640 1 --2- 172.21.15.169:0/4076797828 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4804107cc0 0x7f480419e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.219+0000 7f48097b6640 1 -- 172.21.15.169:0/4076797828 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48041a5c80 con 0x7f4804109870 2024-03-28T01:03:50.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.219+0000 7f4808fb5640 1 --2- 172.21.15.169:0/4076797828 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4804107cc0 0x7f480419e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:03:50.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.220+0000 7f48097b6640 1 --2- 172.21.15.169:0/4076797828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4804109870 0x7f48041a5610 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f47fc00e970 tx=0x7f47fc00ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:50.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.221+0000 7f4801ffb640 1 -- 172.21.15.169:0/4076797828 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc00ccc0 con 0x7f4804109870 2024-03-28T01:03:50.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.221+0000 7f480b219640 1 -- 172.21.15.169:0/4076797828 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48041a5f70 con 0x7f4804109870 2024-03-28T01:03:50.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.222+0000 7f480b219640 1 -- 172.21.15.169:0/4076797828 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48041a6430 con 0x7f4804109870 2024-03-28T01:03:50.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.224+0000 7f4801ffb640 1 -- 172.21.15.169:0/4076797828 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47fc005500 con 0x7f4804109870 2024-03-28T01:03:50.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.224+0000 7f4801ffb640 1 -- 172.21.15.169:0/4076797828 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc010430 con 0x7f4804109870 2024-03-28T01:03:50.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.228+0000 7f480de0a740 1 Processor -- start 2024-03-28T01:03:50.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.235+0000 7f4801ffb640 1 -- 172.21.15.169:0/4076797828 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f47fc0105f0 con 0x7f4804109870 2024-03-28T01:03:50.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.236+0000 7f4801ffb640 1 --2- 172.21.15.169:0/4076797828 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f47f00a2950 0x7f47f00a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.236+0000 7f480de0a740 1 -- start start 2024-03-28T01:03:50.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.236+0000 7f480de0a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e71690 0x55b0a3e71ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.236+0000 7f480de0a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0a3e72080 0x55b0a3e76b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.236+0000 7f480de0a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e771f0 0x55b0a3e79680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.236+0000 7f480de0a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0a3e34610 con 0x55b0a3e71690 2024-03-28T01:03:50.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.236+0000 7f480de0a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0a3dd73c0 con 0x55b0a3e72080 2024-03-28T01:03:50.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.236+0000 7f480de0a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0a3d7fd60 con 0x55b0a3e771f0 2024-03-28T01:03:50.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.236+0000 7f4801ffb640 1 -- 172.21.15.169:0/4076797828 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f47fc005060 con 0x7f4804109870 2024-03-28T01:03:50.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.238+0000 7f4808fb5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e71690 0x55b0a3e71ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.238+0000 7f4808fb5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e71690 0x55b0a3e71ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38312/0 (socket says 172.21.15.169:38312) 2024-03-28T01:03:50.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.238+0000 7f4808fb5640 1 -- 172.21.15.169:0/2077679698 learned_addr learned my addr 172.21.15.169:0/2077679698 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:50.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.239+0000 7f4803fff640 1 --2- 172.21.15.169:0/2077679698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0a3e72080 0x55b0a3e76b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.239+0000 7f4803fff640 1 --2- 172.21.15.169:0/4076797828 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f47f00a2950 0x7f47f00a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.239+0000 7f48097b6640 1 --2- 172.21.15.169:0/2077679698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e771f0 0x55b0a3e79680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.239+0000 7f4808fb5640 1 -- 172.21.15.169:0/2077679698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e771f0 msgr2=0x55b0a3e79680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:50.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.239+0000 7f4808fb5640 1 --2- 172.21.15.169:0/2077679698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e771f0 0x55b0a3e79680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.239+0000 7f4808fb5640 1 -- 172.21.15.169:0/2077679698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0a3e72080 msgr2=0x55b0a3e76b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:50.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.239+0000 7f4808fb5640 1 --2- 172.21.15.169:0/2077679698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0a3e72080 0x55b0a3e76b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.239+0000 7f4808fb5640 1 -- 172.21.15.169:0/2077679698 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0a3aeb3f0 con 0x55b0a3e71690 2024-03-28T01:03:50.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.240+0000 7f4808fb5640 1 --2- 172.21.15.169:0/2077679698 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e71690 0x55b0a3e71ab0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f47ec000c00 tx=0x7f47ec009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b519f950a41ad55 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:50.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.240+0000 7f47faffd640 1 -- 172.21.15.169:0/2077679698 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec039040 con 0x55b0a3e71690 2024-03-28T01:03:50.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.240+0000 7f47faffd640 1 -- 172.21.15.169:0/2077679698 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47ec034db0 con 0x55b0a3e71690 2024-03-28T01:03:50.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.240+0000 7f47faffd640 1 -- 172.21.15.169:0/2077679698 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec02c770 con 0x55b0a3e71690 2024-03-28T01:03:50.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.241+0000 7f480de0a740 1 -- 172.21.15.169:0/2077679698 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e71690 msgr2=0x55b0a3e71ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:50.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.241+0000 7f480de0a740 1 --2- 172.21.15.169:0/2077679698 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e71690 0x55b0a3e71ab0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f47ec000c00 tx=0x7f47ec009520 comp rx=0 tx=0).stop 2024-03-28T01:03:50.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.242+0000 7f4803fff640 1 --2- 172.21.15.169:0/4076797828 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f47f00a2950 0x7f47f00a4e10 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x55b0a3e72480 tx=0x7f47f4004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:50.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.243+0000 7f480de0a740 1 -- 172.21.15.169:0/2077679698 shutdown_connections 2024-03-28T01:03:50.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.243+0000 7f480de0a740 1 --2- 172.21.15.169:0/2077679698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e771f0 0x55b0a3e79680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.243+0000 7f480de0a740 1 --2- 172.21.15.169:0/2077679698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0a3e72080 0x55b0a3e76b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.243+0000 7f480de0a740 1 --2- 172.21.15.169:0/2077679698 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e71690 0x55b0a3e71ab0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.243+0000 7f480de0a740 1 -- 172.21.15.169:0/2077679698 >> 172.21.15.169:0/2077679698 conn(0x55b0a3e6cc00 msgr2=0x55b0a3e6d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:50.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.248+0000 7f480de0a740 1 -- 172.21.15.169:0/2077679698 shutdown_connections 2024-03-28T01:03:50.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.253+0000 7f480de0a740 1 -- 172.21.15.169:0/2077679698 wait complete. 2024-03-28T01:03:50.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.253+0000 7f480de0a740 1 -- start start 2024-03-28T01:03:50.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.253+0000 7f480de0a740 1 Processor -- start 2024-03-28T01:03:50.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.257+0000 7f480de0a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e71690 0x55b0a3e7ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.257+0000 7f480de0a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0a3e72080 0x55b0a3e7bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.257+0000 7f480de0a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e771f0 0x55b0a3e82fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.257+0000 7f480de0a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0a3d93240 con 0x55b0a3e771f0 2024-03-28T01:03:50.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.257+0000 7f480de0a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0a3dd3460 con 0x55b0a3e72080 2024-03-28T01:03:50.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.257+0000 7f480de0a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0a3e7a140 con 0x55b0a3e71690 2024-03-28T01:03:50.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.258+0000 7f480de0a740 10 client.0 _ll_register_callbacks cb 0x55b0a3d92c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:50.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.258+0000 7f4808fb5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e71690 0x55b0a3e7ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.258+0000 7f4808fb5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e71690 0x55b0a3e7ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:39554/0 (socket says 172.21.15.169:39554) 2024-03-28T01:03:50.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.258+0000 7f4808fb5640 1 -- 172.21.15.169:0/2917471148 learned_addr learned my addr 172.21.15.169:0/2917471148 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:50.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.258+0000 7f48097b6640 1 --2- 172.21.15.169:0/2917471148 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e771f0 0x55b0a3e82fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.259+0000 7f48097b6640 1 -- 172.21.15.169:0/2917471148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e71690 msgr2=0x55b0a3e7ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:50.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.259+0000 7f48097b6640 1 --2- 172.21.15.169:0/2917471148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e71690 0x55b0a3e7ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.259+0000 7f48097b6640 1 -- 172.21.15.169:0/2917471148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0a3e72080 msgr2=0x55b0a3e7bfa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:03:50.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.259+0000 7f48097b6640 1 --2- 172.21.15.169:0/2917471148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0a3e72080 0x55b0a3e7bfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.259+0000 7f48097b6640 1 -- 172.21.15.169:0/2917471148 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b0a3bb57d0 con 0x55b0a3e771f0 2024-03-28T01:03:50.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.259+0000 7f48097b6640 1 --2- 172.21.15.169:0/2917471148 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e771f0 0x55b0a3e82fe0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f47fc0ccd90 tx=0x7f47fc005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa760b57ae19134 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:50.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.259+0000 7f47e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:03:50.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.260+0000 7f47e3fff640 1 -- 172.21.15.169:0/2917471148 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc0b9060 con 0x55b0a3e771f0 2024-03-28T01:03:50.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.261+0000 7f480de0a740 1 -- 172.21.15.169:0/2917471148 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f47ec034db0 con 0x55b0a3e771f0 2024-03-28T01:03:50.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.261+0000 7f47e3fff640 1 -- 172.21.15.169:0/2917471148 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f47fc0ccea0 con 0x55b0a3e771f0 2024-03-28T01:03:50.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.262+0000 7f47e3fff640 1 -- 172.21.15.169:0/2917471148 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47fc09bc30 con 0x55b0a3e771f0 2024-03-28T01:03:50.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.264+0000 7f480de0a740 10 client.16848 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:50.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.264+0000 7f480de0a740 1 -- 172.21.15.169:0/2917471148 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55b0a3e97430 con 0x55b0a3e771f0 2024-03-28T01:03:50.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.265+0000 7f47e3fff640 1 -- 172.21.15.169:0/2917471148 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47ec034db0 con 0x55b0a3e771f0 2024-03-28T01:03:50.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.265+0000 7f47e3fff640 1 -- 172.21.15.169:0/2917471148 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f47fc0ba250 con 0x55b0a3e771f0 2024-03-28T01:03:50.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.267+0000 7f480de0a740 10 client.16848 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:50.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.267+0000 7f480de0a740 10 client.16848 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:03:50.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.267+0000 7f480de0a740 20 client.16848 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:50.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.267+0000 7f480de0a740 1 --2- 172.21.15.169:0/2917471148 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b0a3e98a30 0x55b0a3e9ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.267+0000 7f480de0a740 4 client.16848 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:03:50.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.267+0000 7f480de0a740 1 -- 172.21.15.169:0/2917471148 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b0a3de4900 con 0x55b0a3e98a30 2024-03-28T01:03:50.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.272+0000 7f4808fb5640 1 --2- 172.21.15.169:0/2917471148 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b0a3e98a30 0x55b0a3e9ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.281+0000 7f4808fb5640 1 --2- 172.21.15.169:0/2917471148 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b0a3e98a30 0x55b0a3e9ae70 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b3a571db97190169 server_cookie=8283585ff032bb85 in_seq=0 out_seq=0 2024-03-28T01:03:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.281+0000 7f47e3fff640 10 client.16848 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:03:50.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.287+0000 7f47e3fff640 1 -- 172.21.15.169:0/2917471148 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b0a3de4900 con 0x55b0a3e98a30 2024-03-28T01:03:50.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.287+0000 7f47e3fff640 10 client.16848 handle_command_reply: tid=0 2024-03-28T01:03:50.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.287+0000 7f480de0a740 1 client.16848 shutdown 2024-03-28T01:03:50.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.289+0000 7f480de0a740 1 -- 172.21.15.169:0/2917471148 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e771f0 msgr2=0x55b0a3e82fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:50.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.289+0000 7f480de0a740 1 --2- 172.21.15.169:0/2917471148 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e771f0 0x55b0a3e82fe0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f47fc0ccd90 tx=0x7f47fc005770 comp rx=0 tx=0).stop 2024-03-28T01:03:50.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.295+0000 7f480de0a740 1 -- 172.21.15.169:0/2917471148 shutdown_connections 2024-03-28T01:03:50.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.295+0000 7f480de0a740 1 --2- 172.21.15.169:0/2917471148 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b0a3e98a30 0x55b0a3e9ae70 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.295+0000 7f480de0a740 1 --2- 172.21.15.169:0/2917471148 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e771f0 0x55b0a3e82fe0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.295+0000 7f480de0a740 1 --2- 172.21.15.169:0/2917471148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0a3e72080 0x55b0a3e7bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.295+0000 7f480de0a740 1 --2- 172.21.15.169:0/2917471148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e71690 0x55b0a3e7ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.295+0000 7f480de0a740 1 -- 172.21.15.169:0/2917471148 >> 172.21.15.169:0/2917471148 conn(0x55b0a3e6cc00 msgr2=0x55b0a3e6be20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:50.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.295+0000 7f47e3fff640 0 client.16848 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:03:50.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.298+0000 7f480de0a740 1 -- 172.21.15.169:0/2917471148 shutdown_connections 2024-03-28T01:03:50.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.300+0000 7f480de0a740 1 -- 172.21.15.169:0/2917471148 wait complete. 2024-03-28T01:03:50.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.304+0000 7f480de0a740 20 client.16848 trim_cache size 0 max 16384 2024-03-28T01:03:50.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.335+0000 7f480de0a740 1 Processor -- start 2024-03-28T01:03:50.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.335+0000 7f480de0a740 1 -- start start 2024-03-28T01:03:50.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.336+0000 7f480de0a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e71690 0x55b0a3e70570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.336+0000 7f480de0a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e72080 0x55b0a3e82fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.336+0000 7f480de0a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0a3e771f0 0x55b0a3e6b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.336+0000 7f480de0a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e4012090 con 0x55b0a3e72080 2024-03-28T01:03:50.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.336+0000 7f480de0a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e400fb90 con 0x55b0a3e771f0 2024-03-28T01:03:50.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.336+0000 7f480de0a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e400dc30 con 0x55b0a3e71690 2024-03-28T01:03:50.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.338+0000 7f4808fb5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e71690 0x55b0a3e70570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.338+0000 7f4808fb5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e71690 0x55b0a3e70570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:39556/0 (socket says 172.21.15.169:39556) 2024-03-28T01:03:50.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.338+0000 7f4808fb5640 1 -- 172.21.15.169:0/1004152469 learned_addr learned my addr 172.21.15.169:0/1004152469 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:50.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.339+0000 7f48097b6640 1 --2- 172.21.15.169:0/1004152469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0a3e771f0 0x55b0a3e6b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.341+0000 7f4808fb5640 1 -- 172.21.15.169:0/1004152469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0a3e771f0 msgr2=0x55b0a3e6b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:50.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.341+0000 7f4808fb5640 1 --2- 172.21.15.169:0/1004152469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0a3e771f0 0x55b0a3e6b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.342+0000 7f4808fb5640 1 -- 172.21.15.169:0/1004152469 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e72080 msgr2=0x55b0a3e82fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:50.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.342+0000 7f4808fb5640 1 --2- 172.21.15.169:0/1004152469 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e72080 0x55b0a3e82fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.342+0000 7f4808fb5640 1 -- 172.21.15.169:0/1004152469 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0a3e68480 con 0x55b0a3e71690 2024-03-28T01:03:50.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.342+0000 7f4808fb5640 1 --2- 172.21.15.169:0/1004152469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e71690 0x55b0a3e70570 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f47ec02e010 tx=0x7f47ec02e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc1954abf5d265 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:50.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.343+0000 7f47fa7fc640 1 -- 172.21.15.169:0/1004152469 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec039040 con 0x55b0a3e71690 2024-03-28T01:03:50.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.343+0000 7f47fa7fc640 1 -- 172.21.15.169:0/1004152469 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47ec005450 con 0x55b0a3e71690 2024-03-28T01:03:50.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.343+0000 7f47fa7fc640 1 -- 172.21.15.169:0/1004152469 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec02c790 con 0x55b0a3e71690 2024-03-28T01:03:50.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.344+0000 7f48097b6640 1 --2- 172.21.15.169:0/1004152469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0a3e771f0 0x55b0a3e6b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:03:50.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.344+0000 7f480de0a740 1 -- 172.21.15.169:0/1004152469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e71690 msgr2=0x55b0a3e70570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:50.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.344+0000 7f480de0a740 1 --2- 172.21.15.169:0/1004152469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e71690 0x55b0a3e70570 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f47ec02e010 tx=0x7f47ec02e040 comp rx=0 tx=0).stop 2024-03-28T01:03:50.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.344+0000 7f480de0a740 1 -- 172.21.15.169:0/1004152469 shutdown_connections 2024-03-28T01:03:50.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.344+0000 7f480de0a740 1 --2- 172.21.15.169:0/1004152469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0a3e771f0 0x55b0a3e6b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.344+0000 7f480de0a740 1 --2- 172.21.15.169:0/1004152469 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e72080 0x55b0a3e82fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.344+0000 7f480de0a740 1 --2- 172.21.15.169:0/1004152469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e71690 0x55b0a3e70570 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.344+0000 7f480de0a740 1 -- 172.21.15.169:0/1004152469 >> 172.21.15.169:0/1004152469 conn(0x55b0a3e6cc00 msgr2=0x55b0a3e6fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:50.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.349+0000 7f480de0a740 1 -- 172.21.15.169:0/1004152469 shutdown_connections 2024-03-28T01:03:50.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.351+0000 7f480de0a740 1 -- 172.21.15.169:0/1004152469 wait complete. 2024-03-28T01:03:50.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.352+0000 7f480de0a740 1 -- start start 2024-03-28T01:03:50.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.352+0000 7f480de0a740 1 Processor -- start 2024-03-28T01:03:50.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.352+0000 7f480de0a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e71690 0x55b0a3e80470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.352+0000 7f480de0a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e72080 0x55b0a3e809b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.352+0000 7f480de0a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0a3e771f0 0x55b0a3e81900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.352+0000 7f480de0a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e400e230 con 0x55b0a3e71690 2024-03-28T01:03:50.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.352+0000 7f480de0a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e400f530 con 0x55b0a3e771f0 2024-03-28T01:03:50.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.352+0000 7f480de0a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e400e870 con 0x55b0a3e72080 2024-03-28T01:03:50.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.353+0000 7f480de0a740 10 client.0 _ll_register_callbacks cb 0x7f47fc09ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:50.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.353+0000 7f4808fb5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e71690 0x55b0a3e80470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.353+0000 7f4803fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e72080 0x55b0a3e809b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.353+0000 7f4808fb5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e71690 0x55b0a3e80470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38342/0 (socket says 172.21.15.169:38342) 2024-03-28T01:03:50.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.353+0000 7f4803fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e72080 0x55b0a3e809b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:39558/0 (socket says 172.21.15.169:39558) 2024-03-28T01:03:50.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.353+0000 7f4808fb5640 1 -- 172.21.15.169:0/2777144331 learned_addr learned my addr 172.21.15.169:0/2777144331 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:50.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.354+0000 7f4808fb5640 1 -- 172.21.15.169:0/2777144331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e72080 msgr2=0x55b0a3e809b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:50.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.354+0000 7f4808fb5640 1 --2- 172.21.15.169:0/2777144331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e72080 0x55b0a3e809b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.354+0000 7f4808fb5640 1 -- 172.21.15.169:0/2777144331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0a3e771f0 msgr2=0x55b0a3e81900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:50.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.354+0000 7f4808fb5640 1 --2- 172.21.15.169:0/2777144331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0a3e771f0 0x55b0a3e81900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.354+0000 7f4808fb5640 1 -- 172.21.15.169:0/2777144331 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b0a3e81ef0 con 0x55b0a3e71690 2024-03-28T01:03:50.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.354+0000 7f4808fb5640 1 --2- 172.21.15.169:0/2777144331 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e71690 0x55b0a3e80470 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f47ec005d00 tx=0x7f47ec005690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef8ad8ad4b30bd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:50.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.355+0000 7f47e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:03:50.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.355+0000 7f47e3fff640 1 -- 172.21.15.169:0/2777144331 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec05d060 con 0x55b0a3e71690 2024-03-28T01:03:50.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.355+0000 7f480de0a740 1 -- 172.21.15.169:0/2777144331 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f47ec005450 con 0x55b0a3e71690 2024-03-28T01:03:50.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.355+0000 7f47e3fff640 1 -- 172.21.15.169:0/2777144331 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f47ec034640 con 0x55b0a3e71690 2024-03-28T01:03:50.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.355+0000 7f47e3fff640 1 -- 172.21.15.169:0/2777144331 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47ec039030 con 0x55b0a3e71690 2024-03-28T01:03:50.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.355+0000 7f480de0a740 10 client.16851 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:50.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.355+0000 7f480de0a740 1 -- 172.21.15.169:0/2777144331 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55b0a3e83160 con 0x55b0a3e71690 2024-03-28T01:03:50.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.356+0000 7f47e3fff640 1 -- 172.21.15.169:0/2777144331 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47ec005450 con 0x55b0a3e71690 2024-03-28T01:03:50.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.356+0000 7f47e3fff640 1 -- 172.21.15.169:0/2777144331 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f47ec0329f0 con 0x55b0a3e71690 2024-03-28T01:03:50.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.356+0000 7f480de0a740 10 client.16851 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:50.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.356+0000 7f480de0a740 10 client.16851 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:03:50.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.356+0000 7f480de0a740 20 client.16851 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:50.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.356+0000 7f480de0a740 1 --2- 172.21.15.169:0/2777144331 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b0a3ef4960 0x55b0a3ef6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.356+0000 7f480de0a740 4 client.16851 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:03:50.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.356+0000 7f480de0a740 1 -- 172.21.15.169:0/2777144331 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b0a3ddec50 con 0x55b0a3ef4960 2024-03-28T01:03:50.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.360+0000 7f4803fff640 1 --2- 172.21.15.169:0/2777144331 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b0a3ef4960 0x55b0a3ef6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.371+0000 7f4803fff640 1 --2- 172.21.15.169:0/2777144331 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b0a3ef4960 0x55b0a3ef6d80 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8d7bb6de3458d2d7 server_cookie=121b3a3baaab9bec in_seq=0 out_seq=0 2024-03-28T01:03:50.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.372+0000 7f47e3fff640 10 client.16851 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:03:50.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.373+0000 7f47e3fff640 1 -- 172.21.15.169:0/2777144331 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b0a3ddec50 con 0x55b0a3ef4960 2024-03-28T01:03:50.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.373+0000 7f47e3fff640 10 client.16851 handle_command_reply: tid=0 2024-03-28T01:03:50.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.374+0000 7f480de0a740 1 client.16851 shutdown 2024-03-28T01:03:50.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.377+0000 7f480de0a740 1 -- 172.21.15.169:0/2777144331 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e71690 msgr2=0x55b0a3e80470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:50.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.377+0000 7f480de0a740 1 --2- 172.21.15.169:0/2777144331 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e71690 0x55b0a3e80470 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f47ec005d00 tx=0x7f47ec005690 comp rx=0 tx=0).stop 2024-03-28T01:03:50.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.377+0000 7f480de0a740 1 -- 172.21.15.169:0/2777144331 shutdown_connections 2024-03-28T01:03:50.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.377+0000 7f480de0a740 1 --2- 172.21.15.169:0/2777144331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0a3e771f0 0x55b0a3e81900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.377+0000 7f480de0a740 1 --2- 172.21.15.169:0/2777144331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b0a3e72080 0x55b0a3e809b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.377+0000 7f480de0a740 1 --2- 172.21.15.169:0/2777144331 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b0a3ef4960 0x55b0a3ef6d80 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.377+0000 7f480de0a740 1 --2- 172.21.15.169:0/2777144331 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b0a3e71690 0x55b0a3e80470 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.377+0000 7f480de0a740 1 -- 172.21.15.169:0/2777144331 >> 172.21.15.169:0/2777144331 conn(0x55b0a3e6cc00 msgr2=0x55b0a3e706f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:50.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.378+0000 7f47e3fff640 0 client.16851 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:03:50.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.381+0000 7f480de0a740 1 -- 172.21.15.169:0/2777144331 shutdown_connections 2024-03-28T01:03:50.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.383+0000 7f480de0a740 1 -- 172.21.15.169:0/2777144331 wait complete. 2024-03-28T01:03:50.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.383+0000 7f480de0a740 20 client.16851 trim_cache size 0 max 16384 2024-03-28T01:03:50.388 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:03:50.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.387+0000 7f480b219640 1 -- 172.21.15.169:0/4076797828 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f47f00a2950 msgr2=0x7f47f00a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:50.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.387+0000 7f480b219640 1 --2- 172.21.15.169:0/4076797828 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f47f00a2950 0x7f47f00a4e10 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x55b0a3e72480 tx=0x7f47f4004a40 comp rx=0 tx=0).stop 2024-03-28T01:03:50.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.387+0000 7f480b219640 1 -- 172.21.15.169:0/4076797828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4804109870 msgr2=0x7f48041a5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:50.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.387+0000 7f480b219640 1 --2- 172.21.15.169:0/4076797828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4804109870 0x7f48041a5610 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f47fc00e970 tx=0x7f47fc00ee30 comp rx=0 tx=0).stop 2024-03-28T01:03:50.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.388+0000 7f480b219640 1 -- 172.21.15.169:0/4076797828 shutdown_connections 2024-03-28T01:03:50.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.388+0000 7f480b219640 1 --2- 172.21.15.169:0/4076797828 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f47f00a2950 0x7f47f00a4e10 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.388+0000 7f480b219640 1 --2- 172.21.15.169:0/4076797828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4804109870 0x7f48041a5610 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.388+0000 7f480b219640 1 --2- 172.21.15.169:0/4076797828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4804108eb0 0x7f480419e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.388+0000 7f480b219640 1 --2- 172.21.15.169:0/4076797828 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4804107cc0 0x7f480419e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.388+0000 7f480b219640 1 -- 172.21.15.169:0/4076797828 >> 172.21.15.169:0/4076797828 conn(0x7f4804076c60 msgr2=0x7f480410c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:50.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.390+0000 7f480b219640 1 -- 172.21.15.169:0/4076797828 shutdown_connections 2024-03-28T01:03:50.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.390+0000 7f480b219640 1 -- 172.21.15.169:0/4076797828 wait complete. 2024-03-28T01:03:50.420 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:03:50.420 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T01:03:50.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.612+0000 7f731f73b640 1 -- 172.21.15.169:0/4104334128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7318109870 msgr2=0x7f7318110100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:50.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.612+0000 7f731f73b640 1 --2- 172.21.15.169:0/4104334128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7318109870 0x7f7318110100 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7310009ca0 tx=0x7f731002c140 comp rx=0 tx=0).stop 2024-03-28T01:03:50.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.613+0000 7f731f73b640 1 -- 172.21.15.169:0/4104334128 shutdown_connections 2024-03-28T01:03:50.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.613+0000 7f731f73b640 1 --2- 172.21.15.169:0/4104334128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7318109870 0x7f7318110100 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.613+0000 7f731f73b640 1 --2- 172.21.15.169:0/4104334128 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7318108eb0 0x7f7318109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.613+0000 7f731f73b640 1 --2- 172.21.15.169:0/4104334128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7318107cc0 0x7f73181080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.613+0000 7f731f73b640 1 -- 172.21.15.169:0/4104334128 >> 172.21.15.169:0/4104334128 conn(0x7f7318076c60 msgr2=0x7f73180790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:50.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.614+0000 7f731f73b640 1 -- 172.21.15.169:0/4104334128 shutdown_connections 2024-03-28T01:03:50.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.614+0000 7f731f73b640 1 -- 172.21.15.169:0/4104334128 wait complete. 2024-03-28T01:03:50.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.615+0000 7f731f73b640 1 Processor -- start 2024-03-28T01:03:50.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.618+0000 7f731f73b640 1 -- start start 2024-03-28T01:03:50.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.618+0000 7f731f73b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7318107cc0 0x7f731807ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.618+0000 7f731f73b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7318108eb0 0x7f731807f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.618+0000 7f731f73b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7318109870 0x7f73181ae0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.618+0000 7f731f73b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73181139b0 con 0x7f7318107cc0 2024-03-28T01:03:50.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.618+0000 7f731f73b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7318113830 con 0x7f7318109870 2024-03-28T01:03:50.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.618+0000 7f731f73b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7318113b30 con 0x7f7318108eb0 2024-03-28T01:03:50.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.619+0000 7f731ccd6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7318108eb0 0x7f731807f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.619+0000 7f731ccd6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7318108eb0 0x7f731807f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:39574/0 (socket says 172.21.15.169:39574) 2024-03-28T01:03:50.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.619+0000 7f731ccd6640 1 -- 172.21.15.169:0/345936985 learned_addr learned my addr 172.21.15.169:0/345936985 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:50.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.619+0000 7f731dcd8640 1 --2- 172.21.15.169:0/345936985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7318109870 0x7f73181ae0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.619+0000 7f731d4d7640 1 --2- 172.21.15.169:0/345936985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7318107cc0 0x7f731807ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.620+0000 7f731dcd8640 1 -- 172.21.15.169:0/345936985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7318108eb0 msgr2=0x7f731807f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:50.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.620+0000 7f731dcd8640 1 --2- 172.21.15.169:0/345936985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7318108eb0 0x7f731807f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.620+0000 7f731dcd8640 1 -- 172.21.15.169:0/345936985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7318107cc0 msgr2=0x7f731807ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:50.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.620+0000 7f731dcd8640 1 --2- 172.21.15.169:0/345936985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7318107cc0 0x7f731807ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.620+0000 7f731dcd8640 1 -- 172.21.15.169:0/345936985 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7318082340 con 0x7f7318109870 2024-03-28T01:03:50.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.620+0000 7f731d4d7640 1 --2- 172.21.15.169:0/345936985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7318107cc0 0x7f731807ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:03:50.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.620+0000 7f731dcd8640 1 --2- 172.21.15.169:0/345936985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7318109870 0x7f73181ae0c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f731002c750 tx=0x7f7310034d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:50.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.620+0000 7f730e7fc640 1 -- 172.21.15.169:0/345936985 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7310041020 con 0x7f7318109870 2024-03-28T01:03:50.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.620+0000 7f731f73b640 1 -- 172.21.15.169:0/345936985 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731806b0c0 con 0x7f7318109870 2024-03-28T01:03:50.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.620+0000 7f731f73b640 1 -- 172.21.15.169:0/345936985 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f731806b650 con 0x7f7318109870 2024-03-28T01:03:50.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.620+0000 7f730e7fc640 1 -- 172.21.15.169:0/345936985 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73100327f0 con 0x7f7318109870 2024-03-28T01:03:50.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.620+0000 7f730e7fc640 1 -- 172.21.15.169:0/345936985 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7310039040 con 0x7f7318109870 2024-03-28T01:03:50.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.624+0000 7f730e7fc640 1 -- 172.21.15.169:0/345936985 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7310044020 con 0x7f7318109870 2024-03-28T01:03:50.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.625+0000 7f730e7fc640 1 --2- 172.21.15.169:0/345936985 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f73040a2900 0x7f73040a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.625+0000 7f730e7fc640 1 -- 172.21.15.169:0/345936985 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f73100f1090 con 0x7f7318109870 2024-03-28T01:03:50.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.625+0000 7f731d4d7640 1 --2- 172.21.15.169:0/345936985 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f73040a2900 0x7f73040a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.626+0000 7f731d4d7640 1 --2- 172.21.15.169:0/345936985 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f73040a2900 0x7f73040a4dc0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7318108d10 tx=0x7f7308008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:50.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.630+0000 7f7322400740 1 Processor -- start 2024-03-28T01:03:50.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.630+0000 7f7322400740 1 -- start start 2024-03-28T01:03:50.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.630+0000 7f7322400740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c5248b690 0x561c5248bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.630+0000 7f7322400740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c5248c080 0x561c52490b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.631+0000 7f7322400740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c524911f0 0x561c52493680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.631+0000 7f731ccd6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c5248c080 0x561c52490b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.631+0000 7f731ccd6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c5248c080 0x561c52490b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:39578/0 (socket says 172.21.15.169:39578) 2024-03-28T01:03:50.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.631+0000 7f731dcd8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c524911f0 0x561c52493680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.631+0000 7f731dcd8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c524911f0 0x561c52493680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38364/0 (socket says 172.21.15.169:38364) 2024-03-28T01:03:50.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.631+0000 7f731d4d7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c5248b690 0x561c5248bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.631+0000 7f731d4d7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c5248b690 0x561c5248bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:34502/0 (socket says 172.21.15.169:34502) 2024-03-28T01:03:50.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.631+0000 7f731ccd6640 1 -- 172.21.15.169:0/1526129220 learned_addr learned my addr 172.21.15.169:0/1526129220 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:50.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.631+0000 7f7322400740 1 -- 172.21.15.169:0/1526129220 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c5244e610 con 0x561c524911f0 2024-03-28T01:03:50.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.631+0000 7f7322400740 1 -- 172.21.15.169:0/1526129220 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c523f13c0 con 0x561c5248b690 2024-03-28T01:03:50.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.631+0000 7f7322400740 1 -- 172.21.15.169:0/1526129220 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c52399d60 con 0x561c5248c080 2024-03-28T01:03:50.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.632+0000 7f731ccd6640 1 -- 172.21.15.169:0/1526129220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c5248b690 msgr2=0x561c5248bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:50.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.632+0000 7f731ccd6640 1 --2- 172.21.15.169:0/1526129220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c5248b690 0x561c5248bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.632+0000 7f731ccd6640 1 -- 172.21.15.169:0/1526129220 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c524911f0 msgr2=0x561c52493680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:50.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.632+0000 7f731ccd6640 1 --2- 172.21.15.169:0/1526129220 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c524911f0 0x561c52493680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.632+0000 7f731ccd6640 1 -- 172.21.15.169:0/1526129220 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c521053f0 con 0x561c5248c080 2024-03-28T01:03:50.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.633+0000 7f731ccd6640 1 --2- 172.21.15.169:0/1526129220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c5248c080 0x561c52490b80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7314009d60 tx=0x7f7314011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a1a821c1c3e8e2b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:50.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.634+0000 7f73037fe640 1 -- 172.21.15.169:0/1526129220 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314004230 con 0x561c5248c080 2024-03-28T01:03:50.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.634+0000 7f73037fe640 1 -- 172.21.15.169:0/1526129220 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7314009e50 con 0x561c5248c080 2024-03-28T01:03:50.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.634+0000 7f73037fe640 1 -- 172.21.15.169:0/1526129220 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314004df0 con 0x561c5248c080 2024-03-28T01:03:50.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.635+0000 7f7322400740 1 -- 172.21.15.169:0/1526129220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c5248c080 msgr2=0x561c52490b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:50.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.635+0000 7f7322400740 1 --2- 172.21.15.169:0/1526129220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c5248c080 0x561c52490b80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7314009d60 tx=0x7f7314011810 comp rx=0 tx=0).stop 2024-03-28T01:03:50.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.641+0000 7f7322400740 1 -- 172.21.15.169:0/1526129220 shutdown_connections 2024-03-28T01:03:50.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.641+0000 7f7322400740 1 --2- 172.21.15.169:0/1526129220 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c524911f0 0x561c52493680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.641+0000 7f7322400740 1 --2- 172.21.15.169:0/1526129220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c5248c080 0x561c52490b80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.641+0000 7f7322400740 1 --2- 172.21.15.169:0/1526129220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c5248b690 0x561c5248bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.641+0000 7f7322400740 1 -- 172.21.15.169:0/1526129220 >> 172.21.15.169:0/1526129220 conn(0x561c52486c00 msgr2=0x561c52487000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:50.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.641+0000 7f7322400740 1 -- 172.21.15.169:0/1526129220 shutdown_connections 2024-03-28T01:03:50.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.641+0000 7f7322400740 1 -- 172.21.15.169:0/1526129220 wait complete. 2024-03-28T01:03:50.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.641+0000 7f7322400740 1 -- start start 2024-03-28T01:03:50.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.642+0000 7f7322400740 1 Processor -- start 2024-03-28T01:03:50.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.642+0000 7f7322400740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c5248b690 0x561c52495ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.642+0000 7f7322400740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c5248c080 0x561c52496020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.642+0000 7f7322400740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c524911f0 0x561c5249d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.642+0000 7f7322400740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c523ad240 con 0x561c5248b690 2024-03-28T01:03:50.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.642+0000 7f7322400740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c523ed460 con 0x561c524911f0 2024-03-28T01:03:50.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.642+0000 7f7322400740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c524941a0 con 0x561c5248c080 2024-03-28T01:03:50.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.642+0000 7f7322400740 10 client.0 _ll_register_callbacks cb 0x561c523acc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:50.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.643+0000 7f731dcd8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c524911f0 0x561c5249d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.643+0000 7f731d4d7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c5248b690 0x561c52495ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.643+0000 7f731d4d7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c5248b690 0x561c52495ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38378/0 (socket says 172.21.15.169:38378) 2024-03-28T01:03:50.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.643+0000 7f731d4d7640 1 -- 172.21.15.169:0/2741875355 learned_addr learned my addr 172.21.15.169:0/2741875355 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:50.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.643+0000 7f731dcd8640 1 -- 172.21.15.169:0/2741875355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c5248c080 msgr2=0x561c52496020 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:03:50.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.643+0000 7f731dcd8640 1 --2- 172.21.15.169:0/2741875355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c5248c080 0x561c52496020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.643+0000 7f731dcd8640 1 -- 172.21.15.169:0/2741875355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c5248b690 msgr2=0x561c52495ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:50.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.643+0000 7f731dcd8640 1 --2- 172.21.15.169:0/2741875355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c5248b690 0x561c52495ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.643+0000 7f731dcd8640 1 -- 172.21.15.169:0/2741875355 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c521cf7d0 con 0x561c524911f0 2024-03-28T01:03:50.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.644+0000 7f731dcd8640 1 --2- 172.21.15.169:0/2741875355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c524911f0 0x561c5249d060 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7310009ca0 tx=0x7f7310005df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bb7d8c6771497bd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:50.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.644+0000 7f7300ff9640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:03:50.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.646+0000 7f7300ff9640 1 -- 172.21.15.169:0/2741875355 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73100d6060 con 0x561c524911f0 2024-03-28T01:03:50.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.646+0000 7f7322400740 1 -- 172.21.15.169:0/2741875355 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7314009e50 con 0x561c524911f0 2024-03-28T01:03:50.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.646+0000 7f7300ff9640 1 -- 172.21.15.169:0/2741875355 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f73100b9ca0 con 0x561c524911f0 2024-03-28T01:03:50.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.646+0000 7f7300ff9640 1 -- 172.21.15.169:0/2741875355 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73100bdd30 con 0x561c524911f0 2024-03-28T01:03:50.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.646+0000 7f7322400740 10 client.26735 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:50.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.646+0000 7f7322400740 1 -- 172.21.15.169:0/2741875355 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561c524a03d0 con 0x561c524911f0 2024-03-28T01:03:50.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.649+0000 7f7300ff9640 1 -- 172.21.15.169:0/2741875355 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7314009e50 con 0x561c524911f0 2024-03-28T01:03:50.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.650+0000 7f7300ff9640 1 -- 172.21.15.169:0/2741875355 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f731003c030 con 0x561c524911f0 2024-03-28T01:03:50.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.650+0000 7f7322400740 10 client.26735 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:50.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.650+0000 7f7322400740 10 client.26735 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:03:50.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.650+0000 7f7322400740 20 client.26735 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:50.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.650+0000 7f7322400740 1 --2- 172.21.15.169:0/2741875355 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x561c524a1d20 0x561c524a4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.650+0000 7f7322400740 4 client.26735 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:03:50.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.650+0000 7f7322400740 1 -- 172.21.15.169:0/2741875355 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561c523fe900 con 0x561c524a1d20 2024-03-28T01:03:50.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.651+0000 7f731d4d7640 1 --2- 172.21.15.169:0/2741875355 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x561c524a1d20 0x561c524a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.657+0000 7f731d4d7640 1 --2- 172.21.15.169:0/2741875355 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x561c524a1d20 0x561c524a4180 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=341f304306b6f521 server_cookie=ddd2c86cfb812a88 in_seq=0 out_seq=0 2024-03-28T01:03:50.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.661+0000 7f7300ff9640 10 client.26735 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:03:50.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.661+0000 7f7300ff9640 1 -- 172.21.15.169:0/2741875355 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561c523fe900 con 0x561c524a1d20 2024-03-28T01:03:50.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.661+0000 7f7300ff9640 10 client.26735 handle_command_reply: tid=0 2024-03-28T01:03:50.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.661+0000 7f7322400740 1 client.26735 shutdown 2024-03-28T01:03:50.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.662+0000 7f7322400740 1 -- 172.21.15.169:0/2741875355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c524911f0 msgr2=0x561c5249d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:50.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.662+0000 7f7322400740 1 --2- 172.21.15.169:0/2741875355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c524911f0 0x561c5249d060 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7310009ca0 tx=0x7f7310005df0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.663+0000 7f7322400740 1 -- 172.21.15.169:0/2741875355 shutdown_connections 2024-03-28T01:03:50.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.663+0000 7f7322400740 1 --2- 172.21.15.169:0/2741875355 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x561c524a1d20 0x561c524a4180 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.663+0000 7f7322400740 1 --2- 172.21.15.169:0/2741875355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c524911f0 0x561c5249d060 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.663+0000 7f7322400740 1 --2- 172.21.15.169:0/2741875355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c5248c080 0x561c52496020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.663+0000 7f7322400740 1 --2- 172.21.15.169:0/2741875355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c5248b690 0x561c52495ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.663+0000 7f7322400740 1 -- 172.21.15.169:0/2741875355 >> 172.21.15.169:0/2741875355 conn(0x561c52486c00 msgr2=0x561c52485e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:50.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.663+0000 7f7300ff9640 0 client.26735 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:03:50.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.663+0000 7f7322400740 1 -- 172.21.15.169:0/2741875355 shutdown_connections 2024-03-28T01:03:50.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.663+0000 7f7322400740 1 -- 172.21.15.169:0/2741875355 wait complete. 2024-03-28T01:03:50.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.664+0000 7f7322400740 20 client.26735 trim_cache size 0 max 16384 2024-03-28T01:03:50.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.689+0000 7f7322400740 1 Processor -- start 2024-03-28T01:03:50.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.689+0000 7f7322400740 1 -- start start 2024-03-28T01:03:50.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.690+0000 7f7322400740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c5248b690 0x561c524a3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.690+0000 7f7322400740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c5248c080 0x561c5249d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.690+0000 7f7322400740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c524911f0 0x561c5248d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.694 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.690+0000 7f7322400740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc011f50 con 0x561c5248c080 2024-03-28T01:03:50.694 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.690+0000 7f7322400740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc00fa50 con 0x561c5248b690 2024-03-28T01:03:50.694 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.690+0000 7f7322400740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc00dba0 con 0x561c524911f0 2024-03-28T01:03:50.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.691+0000 7f731ccd6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c5248c080 0x561c5249d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.691+0000 7f731ccd6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c5248c080 0x561c5249d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38394/0 (socket says 172.21.15.169:38394) 2024-03-28T01:03:50.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.691+0000 7f731ccd6640 1 -- 172.21.15.169:0/1646218327 learned_addr learned my addr 172.21.15.169:0/1646218327 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:50.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.691+0000 7f731ccd6640 1 -- 172.21.15.169:0/1646218327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c524911f0 msgr2=0x561c5248d5c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:03:50.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.691+0000 7f731ccd6640 1 --2- 172.21.15.169:0/1646218327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c524911f0 0x561c5248d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.691+0000 7f731ccd6640 1 -- 172.21.15.169:0/1646218327 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c5248b690 msgr2=0x561c524a3d30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:03:50.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.691+0000 7f731ccd6640 1 --2- 172.21.15.169:0/1646218327 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c5248b690 0x561c524a3d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.691+0000 7f731ccd6640 1 -- 172.21.15.169:0/1646218327 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c52490870 con 0x561c5248c080 2024-03-28T01:03:50.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.692+0000 7f731ccd6640 1 --2- 172.21.15.169:0/1646218327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c5248c080 0x561c5249d020 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f7314009d60 tx=0x7f7314009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e78af5110c5199 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:50.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.694+0000 7f7302ffd640 1 -- 172.21.15.169:0/1646218327 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314009920 con 0x561c5248c080 2024-03-28T01:03:50.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.694+0000 7f7302ffd640 1 -- 172.21.15.169:0/1646218327 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73140055f0 con 0x561c5248c080 2024-03-28T01:03:50.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.694+0000 7f7302ffd640 1 -- 172.21.15.169:0/1646218327 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314019690 con 0x561c5248c080 2024-03-28T01:03:50.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.695+0000 7f7322400740 1 -- 172.21.15.169:0/1646218327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c5248c080 msgr2=0x561c5249d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:50.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.695+0000 7f7322400740 1 --2- 172.21.15.169:0/1646218327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c5248c080 0x561c5249d020 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f7314009d60 tx=0x7f7314009ed0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.696+0000 7f7322400740 1 -- 172.21.15.169:0/1646218327 shutdown_connections 2024-03-28T01:03:50.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.696+0000 7f7322400740 1 --2- 172.21.15.169:0/1646218327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c524911f0 0x561c5248d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.696+0000 7f7322400740 1 --2- 172.21.15.169:0/1646218327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c5248c080 0x561c5249d020 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.696+0000 7f7322400740 1 --2- 172.21.15.169:0/1646218327 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c5248b690 0x561c524a3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.696+0000 7f7322400740 1 -- 172.21.15.169:0/1646218327 >> 172.21.15.169:0/1646218327 conn(0x561c52486c00 msgr2=0x561c524945d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:50.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.696+0000 7f7322400740 1 -- 172.21.15.169:0/1646218327 shutdown_connections 2024-03-28T01:03:50.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.696+0000 7f7322400740 1 -- 172.21.15.169:0/1646218327 wait complete. 2024-03-28T01:03:50.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.697+0000 7f7322400740 1 -- start start 2024-03-28T01:03:50.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.697+0000 7f7322400740 1 Processor -- start 2024-03-28T01:03:50.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.698+0000 7f7322400740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c5248b690 0x561c5249af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.698+0000 7f7322400740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c5248c080 0x561c5249b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.698+0000 7f7322400740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c524911f0 0x561c5249c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.698+0000 7f7322400740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc00e110 con 0x561c5248b690 2024-03-28T01:03:50.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.698+0000 7f7322400740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc00f3f0 con 0x561c5248c080 2024-03-28T01:03:50.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.698+0000 7f7322400740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc00e730 con 0x561c524911f0 2024-03-28T01:03:50.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.699+0000 7f7322400740 10 client.0 _ll_register_callbacks cb 0x7f73100bd170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:50.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.699+0000 7f731ccd6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c5248c080 0x561c5249b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.699+0000 7f731dcd8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c524911f0 0x561c5249c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.699+0000 7f731ccd6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c5248c080 0x561c5249b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:34520/0 (socket says 172.21.15.169:34520) 2024-03-28T01:03:50.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.699+0000 7f731ccd6640 1 -- 172.21.15.169:0/186758737 learned_addr learned my addr 172.21.15.169:0/186758737 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:50.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.699+0000 7f731d4d7640 1 --2- 172.21.15.169:0/186758737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c5248b690 0x561c5249af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.702+0000 7f731ccd6640 1 -- 172.21.15.169:0/186758737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c524911f0 msgr2=0x561c5249c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:50.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.702+0000 7f731ccd6640 1 --2- 172.21.15.169:0/186758737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c524911f0 0x561c5249c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.702+0000 7f731ccd6640 1 -- 172.21.15.169:0/186758737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c5248b690 msgr2=0x561c5249af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:50.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.702+0000 7f731ccd6640 1 --2- 172.21.15.169:0/186758737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c5248b690 0x561c5249af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.702+0000 7f731ccd6640 1 -- 172.21.15.169:0/186758737 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c5249cad0 con 0x561c5248c080 2024-03-28T01:03:50.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.702+0000 7f731ccd6640 1 --2- 172.21.15.169:0/186758737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c5248c080 0x561c5249b4d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7314011d10 tx=0x7f73140035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ee700b2c4223e81 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:50.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.703+0000 7f7300ff9640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:03:50.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.705+0000 7f7300ff9640 1 -- 172.21.15.169:0/186758737 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314005110 con 0x561c5248c080 2024-03-28T01:03:50.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.705+0000 7f7300ff9640 1 -- 172.21.15.169:0/186758737 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f73140052b0 con 0x561c5248c080 2024-03-28T01:03:50.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.706+0000 7f7322400740 1 -- 172.21.15.169:0/186758737 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f73140055f0 con 0x561c5248c080 2024-03-28T01:03:50.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.707+0000 7f7300ff9640 1 -- 172.21.15.169:0/186758737 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7314019430 con 0x561c5248c080 2024-03-28T01:03:50.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.707+0000 7f7322400740 10 client.26738 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:50.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.707+0000 7f7322400740 1 -- 172.21.15.169:0/186758737 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561c524fded0 con 0x561c5248c080 2024-03-28T01:03:50.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.708+0000 7f7300ff9640 1 -- 172.21.15.169:0/186758737 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7314019ab0 con 0x561c5248c080 2024-03-28T01:03:50.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.709+0000 7f7300ff9640 1 -- 172.21.15.169:0/186758737 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f73140228c0 con 0x561c5248c080 2024-03-28T01:03:50.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.709+0000 7f7322400740 10 client.26738 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:50.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.709+0000 7f7322400740 10 client.26738 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:03:50.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.709+0000 7f7322400740 20 client.26738 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:50.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.709+0000 7f7322400740 1 --2- 172.21.15.169:0/186758737 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x561c524fe6a0 0x561c52500a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.709+0000 7f7322400740 4 client.26738 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:03:50.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.709+0000 7f7322400740 1 -- 172.21.15.169:0/186758737 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f73140055f0 con 0x561c524fe6a0 2024-03-28T01:03:50.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.710+0000 7f731d4d7640 1 --2- 172.21.15.169:0/186758737 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x561c524fe6a0 0x561c52500a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.710+0000 7f7300ff9640 10 client.26738 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:03:50.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.710+0000 7f731d4d7640 1 --2- 172.21.15.169:0/186758737 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x561c524fe6a0 0x561c52500a90 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e4793a3493d76d2b server_cookie=9772e5da12d765e8 in_seq=0 out_seq=0 2024-03-28T01:03:50.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.711+0000 7f7300ff9640 1 -- 172.21.15.169:0/186758737 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f73140055f0 con 0x561c524fe6a0 2024-03-28T01:03:50.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.711+0000 7f7300ff9640 10 client.26738 handle_command_reply: tid=0 2024-03-28T01:03:50.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.711+0000 7f7322400740 1 client.26738 shutdown 2024-03-28T01:03:50.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.711+0000 7f7322400740 1 -- 172.21.15.169:0/186758737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c5248c080 msgr2=0x561c5249b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:50.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.711+0000 7f7322400740 1 --2- 172.21.15.169:0/186758737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c5248c080 0x561c5249b4d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7314011d10 tx=0x7f73140035f0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.711+0000 7f7322400740 1 -- 172.21.15.169:0/186758737 shutdown_connections 2024-03-28T01:03:50.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.711+0000 7f7322400740 1 --2- 172.21.15.169:0/186758737 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x561c524fe6a0 0x561c52500a90 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.711+0000 7f7322400740 1 --2- 172.21.15.169:0/186758737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561c524911f0 0x561c5249c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.711+0000 7f7322400740 1 --2- 172.21.15.169:0/186758737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561c5248c080 0x561c5249b4d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.711+0000 7f7322400740 1 --2- 172.21.15.169:0/186758737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561c5248b690 0x561c5249af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.711+0000 7f7322400740 1 -- 172.21.15.169:0/186758737 >> 172.21.15.169:0/186758737 conn(0x561c52486c00 msgr2=0x561c5248d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:50.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.711+0000 7f7300ff9640 0 client.26738 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:03:50.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.712+0000 7f7322400740 1 -- 172.21.15.169:0/186758737 shutdown_connections 2024-03-28T01:03:50.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.712+0000 7f7322400740 1 -- 172.21.15.169:0/186758737 wait complete. 2024-03-28T01:03:50.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.712+0000 7f7322400740 20 client.26738 trim_cache size 0 max 16384 2024-03-28T01:03:50.716 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:03:50.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.713+0000 7f731f73b640 1 -- 172.21.15.169:0/345936985 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f73040a2900 msgr2=0x7f73040a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:50.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.713+0000 7f731f73b640 1 --2- 172.21.15.169:0/345936985 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f73040a2900 0x7f73040a4dc0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7318108d10 tx=0x7f7308008040 comp rx=0 tx=0).stop 2024-03-28T01:03:50.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.713+0000 7f731f73b640 1 -- 172.21.15.169:0/345936985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7318109870 msgr2=0x7f73181ae0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:50.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.713+0000 7f731f73b640 1 --2- 172.21.15.169:0/345936985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7318109870 0x7f73181ae0c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f731002c750 tx=0x7f7310034d70 comp rx=0 tx=0).stop 2024-03-28T01:03:50.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.713+0000 7f731f73b640 1 -- 172.21.15.169:0/345936985 shutdown_connections 2024-03-28T01:03:50.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.713+0000 7f731f73b640 1 --2- 172.21.15.169:0/345936985 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f73040a2900 0x7f73040a4dc0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.713+0000 7f731f73b640 1 --2- 172.21.15.169:0/345936985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7318109870 0x7f73181ae0c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.713+0000 7f731f73b640 1 --2- 172.21.15.169:0/345936985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7318108eb0 0x7f731807f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.713+0000 7f731f73b640 1 --2- 172.21.15.169:0/345936985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7318107cc0 0x7f731807ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.713+0000 7f731f73b640 1 -- 172.21.15.169:0/345936985 >> 172.21.15.169:0/345936985 conn(0x7f7318076c60 msgr2=0x7f731810c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:50.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.714+0000 7f731f73b640 1 -- 172.21.15.169:0/345936985 shutdown_connections 2024-03-28T01:03:50.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.714+0000 7f731f73b640 1 -- 172.21.15.169:0/345936985 wait complete. 2024-03-28T01:03:50.745 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:03:50.745 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T01:03:50.746 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:50 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3150672997' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:03:50.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:50 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3150672997' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:03:50.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:50 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3150672997' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:03:50.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.944+0000 7f86971e1640 1 -- 172.21.15.169:0/1638877346 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f869010db90 msgr2=0x7f8690110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:50.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.944+0000 7f86971e1640 1 --2- 172.21.15.169:0/1638877346 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f869010db90 0x7f8690110050 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8688005fd0 tx=0x7f868802ff70 comp rx=0 tx=0).stop 2024-03-28T01:03:50.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.947+0000 7f86971e1640 1 -- 172.21.15.169:0/1638877346 shutdown_connections 2024-03-28T01:03:50.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.947+0000 7f86971e1640 1 --2- 172.21.15.169:0/1638877346 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f869010db90 0x7f8690110050 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.947+0000 7f86971e1640 1 --2- 172.21.15.169:0/1638877346 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869010b0f0 0x7f869010d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.947+0000 7f86971e1640 1 --2- 172.21.15.169:0/1638877346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8690108730 0x7f869010ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.947+0000 7f86971e1640 1 -- 172.21.15.169:0/1638877346 >> 172.21.15.169:0/1638877346 conn(0x7f869006e6c0 msgr2=0x7f8690070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:50.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.951+0000 7f86971e1640 1 -- 172.21.15.169:0/1638877346 shutdown_connections 2024-03-28T01:03:50.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.951+0000 7f86971e1640 1 -- 172.21.15.169:0/1638877346 wait complete. 2024-03-28T01:03:50.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.952+0000 7f86971e1640 1 Processor -- start 2024-03-28T01:03:50.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.952+0000 7f86971e1640 1 -- start start 2024-03-28T01:03:50.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.952+0000 7f86971e1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8690108730 0x7f86901a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.952+0000 7f86971e1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f869010b0f0 0x7f86901a6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.952+0000 7f86971e1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869010db90 0x7f86901adae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.952+0000 7f86971e1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86901136e0 con 0x7f869010b0f0 2024-03-28T01:03:50.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.952+0000 7f86971e1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8690113560 con 0x7f8690108730 2024-03-28T01:03:50.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.952+0000 7f86971e1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8690113860 con 0x7f869010db90 2024-03-28T01:03:50.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.953+0000 7f868ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f869010b0f0 0x7f86901a6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.953+0000 7f868ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f869010b0f0 0x7f86901a6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38428/0 (socket says 172.21.15.169:38428) 2024-03-28T01:03:50.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.953+0000 7f868ffff640 1 -- 172.21.15.169:0/1183822505 learned_addr learned my addr 172.21.15.169:0/1183822505 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:50.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.954+0000 7f8694f7d640 1 --2- 172.21.15.169:0/1183822505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8690108730 0x7f86901a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.955+0000 7f868ffff640 1 -- 172.21.15.169:0/1183822505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869010db90 msgr2=0x7f86901adae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:03:50.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.955+0000 7f868ffff640 1 --2- 172.21.15.169:0/1183822505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869010db90 0x7f86901adae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.955+0000 7f868ffff640 1 -- 172.21.15.169:0/1183822505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8690108730 msgr2=0x7f86901a6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:50.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.955+0000 7f868ffff640 1 --2- 172.21.15.169:0/1183822505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8690108730 0x7f86901a6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.956+0000 7f868ffff640 1 -- 172.21.15.169:0/1183822505 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869019da70 con 0x7f869010b0f0 2024-03-28T01:03:50.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.959+0000 7f868ffff640 1 --2- 172.21.15.169:0/1183822505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f869010b0f0 0x7f86901a6a70 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f868400c970 tx=0x7f868400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:50.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.960+0000 7f868dffb640 1 -- 172.21.15.169:0/1183822505 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8684007b30 con 0x7f869010b0f0 2024-03-28T01:03:50.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.961+0000 7f86971e1640 1 -- 172.21.15.169:0/1183822505 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f869019dd60 con 0x7f869010b0f0 2024-03-28T01:03:50.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.961+0000 7f86971e1640 1 -- 172.21.15.169:0/1183822505 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f869019e2b0 con 0x7f869010b0f0 2024-03-28T01:03:50.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.961+0000 7f868dffb640 1 -- 172.21.15.169:0/1183822505 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8684005500 con 0x7f869010b0f0 2024-03-28T01:03:50.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.961+0000 7f868dffb640 1 -- 172.21.15.169:0/1183822505 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868400f450 con 0x7f869010b0f0 2024-03-28T01:03:50.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.967+0000 7f868dffb640 1 -- 172.21.15.169:0/1183822505 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f86840046c0 con 0x7f869010b0f0 2024-03-28T01:03:50.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.969+0000 7f8699dc5740 1 Processor -- start 2024-03-28T01:03:50.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.970+0000 7f868dffb640 1 --2- 172.21.15.169:0/1183822505 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f86780a2a00 0x7f86780a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.970+0000 7f868dffb640 1 -- 172.21.15.169:0/1183822505 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f86840cec90 con 0x7f869010b0f0 2024-03-28T01:03:50.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.973+0000 7f8699dc5740 1 -- start start 2024-03-28T01:03:50.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.974+0000 7f8699dc5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ec0daea860 0x55ec0daeac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.974+0000 7f8699dc5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daeb250 0x55ec0daefd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.974+0000 7f8699dc5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daf03c0 0x55ec0daf2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:50.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.974+0000 7f8699dc5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec0daad610 con 0x55ec0daf03c0 2024-03-28T01:03:50.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.974+0000 7f8699dc5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec0da503c0 con 0x55ec0daeb250 2024-03-28T01:03:50.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.974+0000 7f8699dc5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec0d9f8d60 con 0x55ec0daea860 2024-03-28T01:03:50.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.974+0000 7f869577e640 1 --2- 172.21.15.169:0/1183822505 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f86780a2a00 0x7f86780a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.975+0000 7f868ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daeb250 0x55ec0daefd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.976+0000 7f868ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daeb250 0x55ec0daefd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:34544/0 (socket says 172.21.15.169:34544) 2024-03-28T01:03:50.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.976+0000 7f868ffff640 1 -- 172.21.15.169:0/4094117434 learned_addr learned my addr 172.21.15.169:0/4094117434 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:50.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.978+0000 7f869577e640 1 --2- 172.21.15.169:0/4094117434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daf03c0 0x55ec0daf2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.979+0000 7f8694f7d640 1 --2- 172.21.15.169:0/4094117434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ec0daea860 0x55ec0daeac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:50.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.980+0000 7f868ffff640 1 -- 172.21.15.169:0/4094117434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ec0daea860 msgr2=0x55ec0daeac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:50.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.980+0000 7f868ffff640 1 --2- 172.21.15.169:0/4094117434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ec0daea860 0x55ec0daeac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.980+0000 7f868ffff640 1 -- 172.21.15.169:0/4094117434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daf03c0 msgr2=0x55ec0daf2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:50.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.980+0000 7f868ffff640 1 --2- 172.21.15.169:0/4094117434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daf03c0 0x55ec0daf2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.980+0000 7f868ffff640 1 -- 172.21.15.169:0/4094117434 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec0d7643f0 con 0x55ec0daeb250 2024-03-28T01:03:50.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.988+0000 7f868ffff640 1 --2- 172.21.15.169:0/4094117434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daeb250 0x55ec0daefd50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8684097690 tx=0x7f868409b9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fc739b0621dc9d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:50.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.990+0000 7f8676ffd640 1 -- 172.21.15.169:0/4094117434 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86840b4070 con 0x55ec0daeb250 2024-03-28T01:03:50.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.990+0000 7f8676ffd640 1 -- 172.21.15.169:0/4094117434 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f868409f170 con 0x55ec0daeb250 2024-03-28T01:03:50.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.992+0000 7f8699dc5740 1 -- 172.21.15.169:0/4094117434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daeb250 msgr2=0x55ec0daefd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:50.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.992+0000 7f8699dc5740 1 --2- 172.21.15.169:0/4094117434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daeb250 0x55ec0daefd50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8684097690 tx=0x7f868409b9f0 comp rx=0 tx=0).stop 2024-03-28T01:03:50.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.992+0000 7f8676ffd640 1 -- 172.21.15.169:0/4094117434 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868401d040 con 0x55ec0daeb250 2024-03-28T01:03:50.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.993+0000 7f869577e640 1 --2- 172.21.15.169:0/1183822505 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f86780a2a00 0x7f86780a4ec0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f868803a040 tx=0x7f8688004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:51.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.995+0000 7f8699dc5740 1 -- 172.21.15.169:0/4094117434 shutdown_connections 2024-03-28T01:03:51.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.995+0000 7f8699dc5740 1 --2- 172.21.15.169:0/4094117434 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daf03c0 0x55ec0daf2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.995+0000 7f8699dc5740 1 --2- 172.21.15.169:0/4094117434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daeb250 0x55ec0daefd50 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.995+0000 7f8699dc5740 1 --2- 172.21.15.169:0/4094117434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ec0daea860 0x55ec0daeac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.995+0000 7f8699dc5740 1 -- 172.21.15.169:0/4094117434 >> 172.21.15.169:0/4094117434 conn(0x55ec0dae5dd0 msgr2=0x55ec0dae61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:51.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.999+0000 7f8699dc5740 1 -- 172.21.15.169:0/4094117434 shutdown_connections 2024-03-28T01:03:51.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.999+0000 7f8699dc5740 1 -- 172.21.15.169:0/4094117434 wait complete. 2024-03-28T01:03:51.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:50.999+0000 7f8699dc5740 1 -- start start 2024-03-28T01:03:51.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.000+0000 7f8699dc5740 1 Processor -- start 2024-03-28T01:03:51.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.000+0000 7f8699dc5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ec0daea860 0x55ec0daf4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:51.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.001+0000 7f8699dc5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daeb250 0x55ec0daf5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:51.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.001+0000 7f8699dc5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daf03c0 0x55ec0dafc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:51.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.001+0000 7f8699dc5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec0da0c240 con 0x55ec0daeb250 2024-03-28T01:03:51.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.001+0000 7f8699dc5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec0da4c460 con 0x55ec0daf03c0 2024-03-28T01:03:51.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.001+0000 7f8699dc5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec0daf33c0 con 0x55ec0daea860 2024-03-28T01:03:51.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.001+0000 7f8699dc5740 10 client.0 _ll_register_callbacks cb 0x55ec0da0bc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:51.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.002+0000 7f868ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daeb250 0x55ec0daf5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:51.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.002+0000 7f868ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daeb250 0x55ec0daf5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38452/0 (socket says 172.21.15.169:38452) 2024-03-28T01:03:51.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.002+0000 7f868ffff640 1 -- 172.21.15.169:0/1095676535 learned_addr learned my addr 172.21.15.169:0/1095676535 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:51.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.003+0000 7f8694f7d640 1 --2- 172.21.15.169:0/1095676535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ec0daea860 0x55ec0daf4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:51.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.003+0000 7f869577e640 1 --2- 172.21.15.169:0/1095676535 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daf03c0 0x55ec0dafc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:51.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.004+0000 7f869577e640 1 -- 172.21.15.169:0/1095676535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ec0daea860 msgr2=0x55ec0daf4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:51.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.004+0000 7f869577e640 1 --2- 172.21.15.169:0/1095676535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ec0daea860 0x55ec0daf4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.004+0000 7f869577e640 1 -- 172.21.15.169:0/1095676535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daeb250 msgr2=0x55ec0daf5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:51.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.004+0000 7f869577e640 1 --2- 172.21.15.169:0/1095676535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daeb250 0x55ec0daf5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.004+0000 7f869577e640 1 -- 172.21.15.169:0/1095676535 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec0d82e7d0 con 0x55ec0daf03c0 2024-03-28T01:03:51.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.005+0000 7f869577e640 1 --2- 172.21.15.169:0/1095676535 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daf03c0 0x55ec0dafc2b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8688030580 tx=0x7f868800d040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ebfc961277e92e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:51.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.005+0000 7f866ffff640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:03:51.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.005+0000 7f866ffff640 1 -- 172.21.15.169:0/1095676535 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8688051060 con 0x55ec0daf03c0 2024-03-28T01:03:51.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.005+0000 7f8699dc5740 1 -- 172.21.15.169:0/1095676535 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f868409f170 con 0x55ec0daf03c0 2024-03-28T01:03:51.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.006+0000 7f866ffff640 1 -- 172.21.15.169:0/1095676535 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f868803f070 con 0x55ec0daf03c0 2024-03-28T01:03:51.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.006+0000 7f866ffff640 1 -- 172.21.15.169:0/1095676535 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8688058050 con 0x55ec0daf03c0 2024-03-28T01:03:51.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.006+0000 7f8699dc5740 10 client.26750 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:51.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.006+0000 7f8699dc5740 1 -- 172.21.15.169:0/1095676535 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ec0daff690 con 0x55ec0daf03c0 2024-03-28T01:03:51.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.008+0000 7f866ffff640 1 -- 172.21.15.169:0/1095676535 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f868409f170 con 0x55ec0daf03c0 2024-03-28T01:03:51.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.008+0000 7f866ffff640 1 -- 172.21.15.169:0/1095676535 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f868804b070 con 0x55ec0daf03c0 2024-03-28T01:03:51.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.008+0000 7f8699dc5740 10 client.26750 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:51.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.008+0000 7f8699dc5740 10 client.26750 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:51.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.009+0000 7f8699dc5740 20 client.26750 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:51.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.009+0000 7f8699dc5740 1 --2- 172.21.15.169:0/1095676535 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ec0db00d00 0x55ec0db030f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:51.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.009+0000 7f8699dc5740 4 client.26750 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:03:51.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.009+0000 7f8699dc5740 1 -- 172.21.15.169:0/1095676535 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ec0da5d900 con 0x55ec0db00d00 2024-03-28T01:03:51.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.009+0000 7f8694f7d640 1 --2- 172.21.15.169:0/1095676535 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ec0db00d00 0x55ec0db030f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:51.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.014+0000 7f8694f7d640 1 --2- 172.21.15.169:0/1095676535 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ec0db00d00 0x55ec0db030f0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc2d9068b6144791 server_cookie=83ad6ac14dc3f166 in_seq=0 out_seq=0 2024-03-28T01:03:51.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.014+0000 7f866ffff640 10 client.26750 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:51.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.016+0000 7f866ffff640 1 -- 172.21.15.169:0/1095676535 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ec0da5d900 con 0x55ec0db00d00 2024-03-28T01:03:51.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.016+0000 7f866ffff640 10 client.26750 handle_command_reply: tid=0 2024-03-28T01:03:51.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.016+0000 7f8699dc5740 1 client.26750 shutdown 2024-03-28T01:03:51.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.017+0000 7f8699dc5740 1 -- 172.21.15.169:0/1095676535 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daf03c0 msgr2=0x55ec0dafc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:51.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.017+0000 7f8699dc5740 1 --2- 172.21.15.169:0/1095676535 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daf03c0 0x55ec0dafc2b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8688030580 tx=0x7f868800d040 comp rx=0 tx=0).stop 2024-03-28T01:03:51.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.019+0000 7f8699dc5740 1 -- 172.21.15.169:0/1095676535 shutdown_connections 2024-03-28T01:03:51.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.019+0000 7f8699dc5740 1 --2- 172.21.15.169:0/1095676535 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daf03c0 0x55ec0dafc2b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.019+0000 7f8699dc5740 1 --2- 172.21.15.169:0/1095676535 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ec0db00d00 0x55ec0db030f0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.019+0000 7f8699dc5740 1 --2- 172.21.15.169:0/1095676535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daeb250 0x55ec0daf5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.019+0000 7f8699dc5740 1 --2- 172.21.15.169:0/1095676535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ec0daea860 0x55ec0daf4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.019+0000 7f8699dc5740 1 -- 172.21.15.169:0/1095676535 >> 172.21.15.169:0/1095676535 conn(0x55ec0dae5dd0 msgr2=0x55ec0dae5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:51.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.020+0000 7f866ffff640 0 client.26750 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:51.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.024+0000 7f8699dc5740 1 -- 172.21.15.169:0/1095676535 shutdown_connections 2024-03-28T01:03:51.028 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.025+0000 7f8699dc5740 1 -- 172.21.15.169:0/1095676535 wait complete. 2024-03-28T01:03:51.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.025+0000 7f8699dc5740 20 client.26750 trim_cache size 0 max 16384 2024-03-28T01:03:51.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.068+0000 7f8699dc5740 1 Processor -- start 2024-03-28T01:03:51.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.069+0000 7f8699dc5740 1 -- start start 2024-03-28T01:03:51.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.069+0000 7f8699dc5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daea860 0x55ec0dae47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:51.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.069+0000 7f8699dc5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daeb250 0x55ec0dafc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:51.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.069+0000 7f8699dc5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ec0daf03c0 0x55ec0dae4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:51.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.069+0000 7f8699dc5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c012120 con 0x55ec0daeb250 2024-03-28T01:03:51.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.069+0000 7f8699dc5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c00fc20 con 0x55ec0daea860 2024-03-28T01:03:51.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.069+0000 7f8699dc5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c00dcc0 con 0x55ec0daf03c0 2024-03-28T01:03:51.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.070+0000 7f869577e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ec0daf03c0 0x55ec0dae4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:51.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.070+0000 7f869577e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ec0daf03c0 0x55ec0dae4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:39664/0 (socket says 172.21.15.169:39664) 2024-03-28T01:03:51.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.070+0000 7f869577e640 1 -- 172.21.15.169:0/1442546577 learned_addr learned my addr 172.21.15.169:0/1442546577 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:51.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.071+0000 7f868ffff640 1 --2- 172.21.15.169:0/1442546577 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daeb250 0x55ec0dafc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:51.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.071+0000 7f868ffff640 1 -- 172.21.15.169:0/1442546577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ec0daf03c0 msgr2=0x55ec0dae4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:51.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.071+0000 7f868ffff640 1 --2- 172.21.15.169:0/1442546577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ec0daf03c0 0x55ec0dae4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.071+0000 7f868ffff640 1 -- 172.21.15.169:0/1442546577 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daea860 msgr2=0x55ec0dae47e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:03:51.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.071+0000 7f868ffff640 1 --2- 172.21.15.169:0/1442546577 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daea860 0x55ec0dae47e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.071+0000 7f868ffff640 1 -- 172.21.15.169:0/1442546577 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec0dae1840 con 0x55ec0daeb250 2024-03-28T01:03:51.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.071+0000 7f868ffff640 1 --2- 172.21.15.169:0/1442546577 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daeb250 0x55ec0dafc270 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f868409b170 tx=0x7f868400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4f3ee2ec167ff74 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:51.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.072+0000 7f86767fc640 1 -- 172.21.15.169:0/1442546577 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868401d040 con 0x55ec0daeb250 2024-03-28T01:03:51.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.072+0000 7f86767fc640 1 -- 172.21.15.169:0/1442546577 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f86840a39d0 con 0x55ec0daeb250 2024-03-28T01:03:51.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.073+0000 7f8699dc5740 1 -- 172.21.15.169:0/1442546577 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daeb250 msgr2=0x55ec0dafc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:51.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.073+0000 7f8699dc5740 1 --2- 172.21.15.169:0/1442546577 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daeb250 0x55ec0dafc270 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f868409b170 tx=0x7f868400c8c0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.073+0000 7f8699dc5740 1 -- 172.21.15.169:0/1442546577 shutdown_connections 2024-03-28T01:03:51.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.073+0000 7f8699dc5740 1 --2- 172.21.15.169:0/1442546577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ec0daf03c0 0x55ec0dae4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.073+0000 7f8699dc5740 1 --2- 172.21.15.169:0/1442546577 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daeb250 0x55ec0dafc270 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.073+0000 7f8699dc5740 1 --2- 172.21.15.169:0/1442546577 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daea860 0x55ec0dae47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.073+0000 7f8699dc5740 1 -- 172.21.15.169:0/1442546577 >> 172.21.15.169:0/1442546577 conn(0x55ec0dae5dd0 msgr2=0x55ec0daec7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:51.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.078+0000 7f8699dc5740 1 -- 172.21.15.169:0/1442546577 shutdown_connections 2024-03-28T01:03:51.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.079+0000 7f8699dc5740 1 -- 172.21.15.169:0/1442546577 wait complete. 2024-03-28T01:03:51.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.080+0000 7f8699dc5740 1 -- start start 2024-03-28T01:03:51.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.080+0000 7f8699dc5740 1 Processor -- start 2024-03-28T01:03:51.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.081+0000 7f8699dc5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daea860 0x55ec0daf96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:51.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.081+0000 7f8699dc5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daf03c0 0x55ec0daf9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:51.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.081+0000 7f8699dc5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ec0dafab00 0x55ec0dafaf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:51.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.081+0000 7f8699dc5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c00e2c0 con 0x55ec0daf03c0 2024-03-28T01:03:51.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.081+0000 7f8699dc5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c00f5c0 con 0x55ec0daea860 2024-03-28T01:03:51.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.081+0000 7f8699dc5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c00e900 con 0x55ec0dafab00 2024-03-28T01:03:51.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.081+0000 7f8699dc5740 10 client.0 _ll_register_callbacks cb 0x7f8688038da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:51.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.082+0000 7f868ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daf03c0 0x55ec0daf9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:51.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.082+0000 7f868ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daf03c0 0x55ec0daf9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38472/0 (socket says 172.21.15.169:38472) 2024-03-28T01:03:51.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.082+0000 7f868ffff640 1 -- 172.21.15.169:0/3257631329 learned_addr learned my addr 172.21.15.169:0/3257631329 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:51.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.082+0000 7f8694f7d640 1 --2- 172.21.15.169:0/3257631329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daea860 0x55ec0daf96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:51.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.082+0000 7f869577e640 1 --2- 172.21.15.169:0/3257631329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ec0dafab00 0x55ec0dafaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:51.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.084+0000 7f868ffff640 1 -- 172.21.15.169:0/3257631329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ec0dafab00 msgr2=0x55ec0dafaf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:51.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.084+0000 7f868ffff640 1 --2- 172.21.15.169:0/3257631329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ec0dafab00 0x55ec0dafaf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.084+0000 7f868ffff640 1 -- 172.21.15.169:0/3257631329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daea860 msgr2=0x55ec0daf96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:51.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.084+0000 7f868ffff640 1 --2- 172.21.15.169:0/3257631329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daea860 0x55ec0daf96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.084+0000 7f868ffff640 1 -- 172.21.15.169:0/3257631329 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec0dafb4c0 con 0x55ec0daf03c0 2024-03-28T01:03:51.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.087+0000 7f868ffff640 1 --2- 172.21.15.169:0/3257631329 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daf03c0 0x55ec0daf9be0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f868400c940 tx=0x7f86840bb2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79bedb665b72a441 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:51.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.088+0000 7f866ffff640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:03:51.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.092+0000 7f866ffff640 1 -- 172.21.15.169:0/3257631329 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868401d070 con 0x55ec0daf03c0 2024-03-28T01:03:51.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.092+0000 7f8699dc5740 1 -- 172.21.15.169:0/3257631329 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ec0da57c50 con 0x55ec0daf03c0 2024-03-28T01:03:51.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.092+0000 7f866ffff640 1 -- 172.21.15.169:0/3257631329 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f86840a3540 con 0x55ec0daf03c0 2024-03-28T01:03:51.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.093+0000 7f866ffff640 1 -- 172.21.15.169:0/3257631329 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86840b4040 con 0x55ec0daf03c0 2024-03-28T01:03:51.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.093+0000 7f8699dc5740 10 client.16884 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:51.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.093+0000 7f8699dc5740 1 -- 172.21.15.169:0/3257631329 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ec0db5c660 con 0x55ec0daf03c0 2024-03-28T01:03:51.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.094+0000 7f866ffff640 1 -- 172.21.15.169:0/3257631329 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ec0da57c50 con 0x55ec0daf03c0 2024-03-28T01:03:51.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.095+0000 7f866ffff640 1 -- 172.21.15.169:0/3257631329 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f86840b5020 con 0x55ec0daf03c0 2024-03-28T01:03:51.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.095+0000 7f8699dc5740 10 client.16884 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:51.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.095+0000 7f8699dc5740 10 client.16884 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:51.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.095+0000 7f8699dc5740 20 client.16884 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:51.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.095+0000 7f8699dc5740 1 --2- 172.21.15.169:0/3257631329 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ec0db5cfb0 0x55ec0db5f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:51.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.095+0000 7f8699dc5740 4 client.16884 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:03:51.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.095+0000 7f8699dc5740 1 -- 172.21.15.169:0/3257631329 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ec0d9d5640 con 0x55ec0db5cfb0 2024-03-28T01:03:51.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.096+0000 7f8694f7d640 1 --2- 172.21.15.169:0/3257631329 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ec0db5cfb0 0x55ec0db5f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:51.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.100+0000 7f8694f7d640 1 --2- 172.21.15.169:0/3257631329 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ec0db5cfb0 0x55ec0db5f3d0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8857b28249b476f9 server_cookie=87b20a5f54c7f331 in_seq=0 out_seq=0 2024-03-28T01:03:51.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.100+0000 7f866ffff640 10 client.16884 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:51.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.100+0000 7f866ffff640 1 -- 172.21.15.169:0/3257631329 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ec0d9d5640 con 0x55ec0db5cfb0 2024-03-28T01:03:51.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.100+0000 7f866ffff640 10 client.16884 handle_command_reply: tid=0 2024-03-28T01:03:51.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.100+0000 7f8699dc5740 1 client.16884 shutdown 2024-03-28T01:03:51.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.104+0000 7f8699dc5740 1 -- 172.21.15.169:0/3257631329 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daf03c0 msgr2=0x55ec0daf9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:51.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.104+0000 7f8699dc5740 1 --2- 172.21.15.169:0/3257631329 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daf03c0 0x55ec0daf9be0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f868400c940 tx=0x7f86840bb2e0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.109+0000 7f8699dc5740 1 -- 172.21.15.169:0/3257631329 shutdown_connections 2024-03-28T01:03:51.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.109+0000 7f8699dc5740 1 --2- 172.21.15.169:0/3257631329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ec0dafab00 0x55ec0dafaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.109+0000 7f8699dc5740 1 --2- 172.21.15.169:0/3257631329 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ec0db5cfb0 0x55ec0db5f3d0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.109+0000 7f8699dc5740 1 --2- 172.21.15.169:0/3257631329 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ec0daf03c0 0x55ec0daf9be0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.109+0000 7f8699dc5740 1 --2- 172.21.15.169:0/3257631329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec0daea860 0x55ec0daf96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.109+0000 7f8699dc5740 1 -- 172.21.15.169:0/3257631329 >> 172.21.15.169:0/3257631329 conn(0x55ec0dae5dd0 msgr2=0x55ec0dae9030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:51.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.109+0000 7f866ffff640 0 client.16884 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:51.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.113+0000 7f8699dc5740 1 -- 172.21.15.169:0/3257631329 shutdown_connections 2024-03-28T01:03:51.156 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:03:51.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.114+0000 7f8699dc5740 1 -- 172.21.15.169:0/3257631329 wait complete. 2024-03-28T01:03:51.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.114+0000 7f8699dc5740 20 client.16884 trim_cache size 0 max 16384 2024-03-28T01:03:51.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.125+0000 7f86971e1640 1 -- 172.21.15.169:0/1183822505 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f86780a2a00 msgr2=0x7f86780a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:51.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.125+0000 7f86971e1640 1 --2- 172.21.15.169:0/1183822505 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f86780a2a00 0x7f86780a4ec0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f868803a040 tx=0x7f8688004ab0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.126+0000 7f86971e1640 1 -- 172.21.15.169:0/1183822505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f869010b0f0 msgr2=0x7f86901a6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:51.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.126+0000 7f86971e1640 1 --2- 172.21.15.169:0/1183822505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f869010b0f0 0x7f86901a6a70 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f868400c970 tx=0x7f868400ce30 comp rx=0 tx=0).stop 2024-03-28T01:03:51.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.128+0000 7f86971e1640 1 -- 172.21.15.169:0/1183822505 shutdown_connections 2024-03-28T01:03:51.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.128+0000 7f86971e1640 1 --2- 172.21.15.169:0/1183822505 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f86780a2a00 0x7f86780a4ec0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.128+0000 7f86971e1640 1 --2- 172.21.15.169:0/1183822505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f869010db90 0x7f86901adae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.128+0000 7f86971e1640 1 --2- 172.21.15.169:0/1183822505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f869010b0f0 0x7f86901a6a70 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.128+0000 7f86971e1640 1 --2- 172.21.15.169:0/1183822505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8690108730 0x7f86901a6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:51.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.128+0000 7f86971e1640 1 -- 172.21.15.169:0/1183822505 >> 172.21.15.169:0/1183822505 conn(0x7f869006e6c0 msgr2=0x7f8690109070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:51.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.129+0000 7f86971e1640 1 -- 172.21.15.169:0/1183822505 shutdown_connections 2024-03-28T01:03:51.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:51.129+0000 7f86971e1640 1 -- 172.21.15.169:0/1183822505 wait complete. 2024-03-28T01:03:51.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:51 smithi174 ceph-mon[36060]: pgmap v571: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 139 MiB/s rd, 374 MiB/s wr, 3.79k op/s 2024-03-28T01:03:51.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:51 smithi177 ceph-mon[36206]: pgmap v571: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 139 MiB/s rd, 374 MiB/s wr, 3.79k op/s 2024-03-28T01:03:52.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:51 smithi169 ceph-mon[32274]: pgmap v571: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 139 MiB/s rd, 374 MiB/s wr, 3.79k op/s 2024-03-28T01:03:52.161 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T01:03:52.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T01:03:52.161 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T01:03:52.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.403+0000 7f5ebbdab640 1 -- 172.21.15.169:0/1495982279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5eb4101810 msgr2=0x7f5eb4101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:52.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.403+0000 7f5eb8b45640 1 -- 172.21.15.169:0/1495982279 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea4039070 con 0x7f5eb4101810 2024-03-28T01:03:52.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.403+0000 7f5ebbdab640 1 --2- 172.21.15.169:0/1495982279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5eb4101810 0x7f5eb4101c90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4009860 tx=0x7f5ea402bb70 comp rx=0 tx=0).stop 2024-03-28T01:03:52.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.404+0000 7f5ebbdab640 1 -- 172.21.15.169:0/1495982279 shutdown_connections 2024-03-28T01:03:52.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.404+0000 7f5ebbdab640 1 --2- 172.21.15.169:0/1495982279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5eb41021d0 0x7f5eb4108a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.404+0000 7f5ebbdab640 1 --2- 172.21.15.169:0/1495982279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5eb4101810 0x7f5eb4101c90 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.404+0000 7f5ebbdab640 1 --2- 172.21.15.169:0/1495982279 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5eb4100620 0x7f5eb4100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.404+0000 7f5ebbdab640 1 -- 172.21.15.169:0/1495982279 >> 172.21.15.169:0/1495982279 conn(0x7f5eb40fbbd0 msgr2=0x7f5eb40fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:52.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.404+0000 7f5ebbdab640 1 -- 172.21.15.169:0/1495982279 shutdown_connections 2024-03-28T01:03:52.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.404+0000 7f5ebbdab640 1 -- 172.21.15.169:0/1495982279 wait complete. 2024-03-28T01:03:52.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.405+0000 7f5ebbdab640 1 Processor -- start 2024-03-28T01:03:52.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.405+0000 7f5ebbdab640 1 -- start start 2024-03-28T01:03:52.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.405+0000 7f5ebbdab640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5eb4100620 0x7f5eb406a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.405+0000 7f5ebbdab640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5eb4101810 0x7f5eb419c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.405+0000 7f5ebbdab640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5eb41021d0 0x7f5eb419ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.405+0000 7f5ebbdab640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb410c260 con 0x7f5eb4100620 2024-03-28T01:03:52.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.406+0000 7f5ebbdab640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb410c0e0 con 0x7f5eb4101810 2024-03-28T01:03:52.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.406+0000 7f5ebbdab640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb410c3e0 con 0x7f5eb41021d0 2024-03-28T01:03:52.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.406+0000 7f5eb9346640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5eb4101810 0x7f5eb419c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.406+0000 7f5eba348640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5eb41021d0 0x7f5eb419ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.406+0000 7f5eba348640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5eb41021d0 0x7f5eb419ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:39696/0 (socket says 172.21.15.169:39696) 2024-03-28T01:03:52.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.406+0000 7f5eba348640 1 -- 172.21.15.169:0/1406685811 learned_addr learned my addr 172.21.15.169:0/1406685811 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:52.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.406+0000 7f5eb9b47640 1 --2- 172.21.15.169:0/1406685811 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5eb4100620 0x7f5eb406a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.406+0000 7f5eba348640 1 -- 172.21.15.169:0/1406685811 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5eb4101810 msgr2=0x7f5eb419c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:52.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.406+0000 7f5eba348640 1 --2- 172.21.15.169:0/1406685811 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5eb4101810 0x7f5eb419c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.406+0000 7f5eba348640 1 -- 172.21.15.169:0/1406685811 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5eb4100620 msgr2=0x7f5eb406a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:52.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.406+0000 7f5eba348640 1 --2- 172.21.15.169:0/1406685811 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5eb4100620 0x7f5eb406a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.406+0000 7f5eba348640 1 -- 172.21.15.169:0/1406685811 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb419f650 con 0x7f5eb41021d0 2024-03-28T01:03:52.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.407+0000 7f5eba348640 1 --2- 172.21.15.169:0/1406685811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5eb41021d0 0x7f5eb419ed40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5eac009fd0 tx=0x7f5eac00b630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:52.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.407+0000 7f5eaaffd640 1 -- 172.21.15.169:0/1406685811 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eac00be60 con 0x7f5eb41021d0 2024-03-28T01:03:52.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.407+0000 7f5eaaffd640 1 -- 172.21.15.169:0/1406685811 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5eac003b70 con 0x7f5eb41021d0 2024-03-28T01:03:52.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.407+0000 7f5eaaffd640 1 -- 172.21.15.169:0/1406685811 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eac01b900 con 0x7f5eb41021d0 2024-03-28T01:03:52.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.408+0000 7f5ebbdab640 1 -- 172.21.15.169:0/1406685811 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb4196550 con 0x7f5eb41021d0 2024-03-28T01:03:52.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.409+0000 7f5ebbdab640 1 -- 172.21.15.169:0/1406685811 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb4196a20 con 0x7f5eb41021d0 2024-03-28T01:03:52.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.412+0000 7f5ebe98a740 1 Processor -- start 2024-03-28T01:03:52.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.412+0000 7f5eaaffd640 1 -- 172.21.15.169:0/1406685811 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5eac01baa0 con 0x7f5eb41021d0 2024-03-28T01:03:52.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.413+0000 7f5ebe98a740 1 -- start start 2024-03-28T01:03:52.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.413+0000 7f5ebe98a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6ea421960 0x55b6ea421d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.414+0000 7f5ebe98a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 0x55b6ea426e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.414+0000 7f5ebe98a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6ea4274c0 0x55b6ea429950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.414+0000 7f5ebe98a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6ea361c90 con 0x55b6ea422350 2024-03-28T01:03:52.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.414+0000 7f5ebe98a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6ea387070 con 0x55b6ea421960 2024-03-28T01:03:52.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.414+0000 7f5ebe98a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6ea32fc80 con 0x55b6ea4274c0 2024-03-28T01:03:52.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.415+0000 7f5eb9b47640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 0x55b6ea426e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.415+0000 7f5eb9b47640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 0x55b6ea426e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38498/0 (socket says 172.21.15.169:38498) 2024-03-28T01:03:52.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.415+0000 7f5eb9b47640 1 -- 172.21.15.169:0/1810792173 learned_addr learned my addr 172.21.15.169:0/1810792173 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:52.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.415+0000 7f5eaaffd640 1 --2- 172.21.15.169:0/1406685811 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5ea00a2940 0x7f5ea00a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.415+0000 7f5eaaffd640 1 -- 172.21.15.169:0/1406685811 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f5eac0d2be0 con 0x7f5eb41021d0 2024-03-28T01:03:52.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.417+0000 7f5eba348640 1 --2- 172.21.15.169:0/1406685811 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5ea00a2940 0x7f5ea00a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.417+0000 7f5eb9b47640 1 -- 172.21.15.169:0/1810792173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6ea4274c0 msgr2=0x55b6ea429950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:03:52.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.417+0000 7f5eb9b47640 1 --2- 172.21.15.169:0/1810792173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6ea4274c0 0x55b6ea429950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.417+0000 7f5eb9b47640 1 -- 172.21.15.169:0/1810792173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6ea421960 msgr2=0x55b6ea421d80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:03:52.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.417+0000 7f5eb9b47640 1 --2- 172.21.15.169:0/1810792173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6ea421960 0x55b6ea421d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.417+0000 7f5eb9b47640 1 -- 172.21.15.169:0/1810792173 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6ea327d10 con 0x55b6ea422350 2024-03-28T01:03:52.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.417+0000 7f5eba348640 1 --2- 172.21.15.169:0/1406685811 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5ea00a2940 0x7f5ea00a4e00 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5eac09b5e0 tx=0x7f5eac027000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:52.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.418+0000 7f5eb9b47640 1 --2- 172.21.15.169:0/1810792173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 0x55b6ea426e50 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f5eb0009cf0 tx=0x7f5eb00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c176dd11bfa5eb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:52.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.418+0000 7f5e9ffff640 1 -- 172.21.15.169:0/1810792173 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb0012450 con 0x55b6ea422350 2024-03-28T01:03:52.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.419+0000 7f5e9ffff640 1 -- 172.21.15.169:0/1810792173 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5eb0012ad0 con 0x55b6ea422350 2024-03-28T01:03:52.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.419+0000 7f5e9ffff640 1 -- 172.21.15.169:0/1810792173 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb000fca0 con 0x55b6ea422350 2024-03-28T01:03:52.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.420+0000 7f5ebe98a740 1 -- 172.21.15.169:0/1810792173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 msgr2=0x55b6ea426e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:52.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.420+0000 7f5ebe98a740 1 --2- 172.21.15.169:0/1810792173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 0x55b6ea426e50 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f5eb0009cf0 tx=0x7f5eb00117b0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.424+0000 7f5ebe98a740 1 -- 172.21.15.169:0/1810792173 shutdown_connections 2024-03-28T01:03:52.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.424+0000 7f5ebe98a740 1 --2- 172.21.15.169:0/1810792173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6ea4274c0 0x55b6ea429950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.424+0000 7f5ebe98a740 1 --2- 172.21.15.169:0/1810792173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 0x55b6ea426e50 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.424+0000 7f5ebe98a740 1 --2- 172.21.15.169:0/1810792173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6ea421960 0x55b6ea421d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.424+0000 7f5ebe98a740 1 -- 172.21.15.169:0/1810792173 >> 172.21.15.169:0/1810792173 conn(0x55b6ea3ea1a0 msgr2=0x55b6ea41d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:52.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.427+0000 7f5ebe98a740 1 -- 172.21.15.169:0/1810792173 shutdown_connections 2024-03-28T01:03:52.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.428+0000 7f5ebe98a740 1 -- 172.21.15.169:0/1810792173 wait complete. 2024-03-28T01:03:52.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.429+0000 7f5ebe98a740 1 -- start start 2024-03-28T01:03:52.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.429+0000 7f5ebe98a740 1 Processor -- start 2024-03-28T01:03:52.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.430+0000 7f5ebe98a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6ea421960 0x55b6ea42bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.430+0000 7f5ebe98a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 0x55b6ea42c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.430+0000 7f5ebe98a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6ea4274c0 0x55b6ea433100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.430+0000 7f5eb9346640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6ea421960 0x55b6ea42bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.430+0000 7f5eb9346640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6ea421960 0x55b6ea42bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:39706/0 (socket says 172.21.15.169:39706) 2024-03-28T01:03:52.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.430+0000 7f5eb9346640 1 -- 172.21.15.169:0/1566449398 learned_addr learned my addr 172.21.15.169:0/1566449398 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:52.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.430+0000 7f5ebe98a740 1 -- 172.21.15.169:0/1566449398 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6ea3430d0 con 0x55b6ea422350 2024-03-28T01:03:52.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.430+0000 7f5ebe98a740 1 -- 172.21.15.169:0/1566449398 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6ea383110 con 0x55b6ea4274c0 2024-03-28T01:03:52.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.431+0000 7f5ebe98a740 1 -- 172.21.15.169:0/1566449398 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6ea42a4c0 con 0x55b6ea421960 2024-03-28T01:03:52.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.431+0000 7f5eb9346640 1 --2- 172.21.15.169:0/1566449398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6ea4274c0 0x55b6ea433100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.431+0000 7f5eb9346640 1 -- 172.21.15.169:0/1566449398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6ea4274c0 msgr2=0x55b6ea433100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:52.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.431+0000 7f5eb9346640 1 --2- 172.21.15.169:0/1566449398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6ea4274c0 0x55b6ea433100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.431+0000 7f5eb9346640 1 -- 172.21.15.169:0/1566449398 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 msgr2=0x55b6ea42c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:52.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.431+0000 7f5eb9346640 1 --2- 172.21.15.169:0/1566449398 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 0x55b6ea42c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.432+0000 7f5eb9346640 1 -- 172.21.15.169:0/1566449398 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b6ea1637e0 con 0x55b6ea421960 2024-03-28T01:03:52.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.431+0000 7f5ebe98a740 10 client.0 _ll_register_callbacks cb 0x55b6ea37ee60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:52.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.432+0000 7f5ebe98a740 1 -- 172.21.15.169:0/1566449398 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5eb0012ad0 con 0x55b6ea421960 2024-03-28T01:03:52.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.432+0000 7f5eb9346640 1 --2- 172.21.15.169:0/1566449398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6ea421960 0x55b6ea42bb80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4005940 tx=0x7f5ea402c100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b9c6a94a5c2830d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:52.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.432+0000 7f5e9d7fa640 10 client.26710 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:03:52.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.432+0000 7f5e9d7fa640 1 -- 172.21.15.169:0/1566449398 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea4049060 con 0x55b6ea421960 2024-03-28T01:03:52.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.433+0000 7f5e9d7fa640 1 -- 172.21.15.169:0/1566449398 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f5ea4005250 con 0x55b6ea421960 2024-03-28T01:03:52.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.433+0000 7f5e9d7fa640 1 -- 172.21.15.169:0/1566449398 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ea4039040 con 0x55b6ea421960 2024-03-28T01:03:52.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.433+0000 7f5ebe98a740 10 client.26710 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:52.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.433+0000 7f5ebe98a740 1 -- 172.21.15.169:0/1566449398 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x55b6ea1c3f70 con 0x55b6ea421960 2024-03-28T01:03:52.440 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.434+0000 7f5e9d7fa640 1 -- 172.21.15.169:0/1566449398 <== mon.1 v2:172.21.15.177:3300/0 4 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f5ea40054b0 con 0x55b6ea421960 2024-03-28T01:03:52.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.435+0000 7f5ebe98a740 10 client.26710 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:52.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.435+0000 7f5ebe98a740 10 client.26710 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:52.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.435+0000 7f5ebe98a740 20 client.26710 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:52.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.435+0000 7f5ebe98a740 1 --2- 172.21.15.169:0/1566449398 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b6ea437780 0x55b6ea439bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.435+0000 7f5ebe98a740 4 client.26710 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:03:52.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.435+0000 7f5ebe98a740 1 -- 172.21.15.169:0/1566449398 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5eb0012ad0 con 0x55b6ea437780 2024-03-28T01:03:52.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.438+0000 7f5eb9b47640 1 --2- 172.21.15.169:0/1566449398 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b6ea437780 0x55b6ea439bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.452+0000 7f5eb9b47640 1 --2- 172.21.15.169:0/1566449398 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b6ea437780 0x55b6ea439bd0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0571c3e8ca4e7a3 server_cookie=1cfb79bb0ffc8215 in_seq=0 out_seq=0 2024-03-28T01:03:52.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.452+0000 7f5e9d7fa640 10 client.26710 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:52.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.454+0000 7f5e9d7fa640 1 -- 172.21.15.169:0/1566449398 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7f5eb0012ad0 con 0x55b6ea437780 2024-03-28T01:03:52.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.454+0000 7f5e9d7fa640 10 client.26710 handle_command_reply: tid=0 2024-03-28T01:03:52.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.456+0000 7f5ebe98a740 1 client.26710 shutdown 2024-03-28T01:03:52.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.461+0000 7f5ebe98a740 1 -- 172.21.15.169:0/1566449398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6ea421960 msgr2=0x55b6ea42bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:52.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.461+0000 7f5ebe98a740 1 --2- 172.21.15.169:0/1566449398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6ea421960 0x55b6ea42bb80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4005940 tx=0x7f5ea402c100 comp rx=0 tx=0).stop 2024-03-28T01:03:52.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.461+0000 7f5ebe98a740 1 -- 172.21.15.169:0/1566449398 shutdown_connections 2024-03-28T01:03:52.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.461+0000 7f5ebe98a740 1 --2- 172.21.15.169:0/1566449398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6ea4274c0 0x55b6ea433100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.461+0000 7f5ebe98a740 1 --2- 172.21.15.169:0/1566449398 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b6ea437780 0x55b6ea439bd0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.461+0000 7f5ebe98a740 1 --2- 172.21.15.169:0/1566449398 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 0x55b6ea42c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.461+0000 7f5e9d7fa640 0 client.26710 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:52.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.462+0000 7f5ebe98a740 1 --2- 172.21.15.169:0/1566449398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6ea421960 0x55b6ea42bb80 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.462+0000 7f5ebe98a740 1 -- 172.21.15.169:0/1566449398 >> 172.21.15.169:0/1566449398 conn(0x55b6ea3ea1a0 msgr2=0x55b6ea41c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:52.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.465+0000 7f5ebe98a740 1 -- 172.21.15.169:0/1566449398 shutdown_connections 2024-03-28T01:03:52.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.466+0000 7f5ebe98a740 1 -- 172.21.15.169:0/1566449398 wait complete. 2024-03-28T01:03:52.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.469+0000 7f5ebe98a740 20 client.26710 trim_cache size 0 max 16384 2024-03-28T01:03:52.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.524+0000 7f5ebe98a740 1 Processor -- start 2024-03-28T01:03:52.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.524+0000 7f5ebe98a740 1 -- start start 2024-03-28T01:03:52.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.524+0000 7f5ebe98a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6ea421960 0x55b6ea420b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.524+0000 7f5ebe98a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 0x55b6ea4330c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.525+0000 7f5ebe98a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6ea4274c0 0x55b6ea42bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.525+0000 7f5ebe98a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb41aa5b0 con 0x55b6ea422350 2024-03-28T01:03:52.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.525+0000 7f5ebe98a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb41a80b0 con 0x55b6ea4274c0 2024-03-28T01:03:52.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.525+0000 7f5ebe98a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb41a6150 con 0x55b6ea421960 2024-03-28T01:03:52.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.525+0000 7f5eb9b47640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 0x55b6ea4330c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.525+0000 7f5eb9b47640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 0x55b6ea4330c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38516/0 (socket says 172.21.15.169:38516) 2024-03-28T01:03:52.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.525+0000 7f5eb9b47640 1 -- 172.21.15.169:0/3745052584 learned_addr learned my addr 172.21.15.169:0/3745052584 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:52.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.526+0000 7f5eb9b47640 1 -- 172.21.15.169:0/3745052584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6ea421960 msgr2=0x55b6ea420b80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:03:52.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.526+0000 7f5eb9b47640 1 --2- 172.21.15.169:0/3745052584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6ea421960 0x55b6ea420b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.531 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.526+0000 7f5eb9b47640 1 -- 172.21.15.169:0/3745052584 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6ea4274c0 msgr2=0x55b6ea42bea0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:03:52.531 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.526+0000 7f5eb9b47640 1 --2- 172.21.15.169:0/3745052584 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6ea4274c0 0x55b6ea42bea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.531 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.526+0000 7f5eb9b47640 1 -- 172.21.15.169:0/3745052584 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6ea05b170 con 0x55b6ea422350 2024-03-28T01:03:52.531 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.526+0000 7f5eb9b47640 1 --2- 172.21.15.169:0/3745052584 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 0x55b6ea4330c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f5eb001c040 tx=0x7f5eb000f660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a349fde611ec2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:52.531 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.527+0000 7f5e9f7fe640 1 -- 172.21.15.169:0/3745052584 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb002f070 con 0x55b6ea422350 2024-03-28T01:03:52.531 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.527+0000 7f5e9f7fe640 1 -- 172.21.15.169:0/3745052584 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5eb0028070 con 0x55b6ea422350 2024-03-28T01:03:52.531 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.527+0000 7f5e9f7fe640 1 -- 172.21.15.169:0/3745052584 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb0029c40 con 0x55b6ea422350 2024-03-28T01:03:52.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.528+0000 7f5ebe98a740 1 -- 172.21.15.169:0/3745052584 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 msgr2=0x55b6ea4330c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:52.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.528+0000 7f5ebe98a740 1 --2- 172.21.15.169:0/3745052584 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 0x55b6ea4330c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f5eb001c040 tx=0x7f5eb000f660 comp rx=0 tx=0).stop 2024-03-28T01:03:52.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.528+0000 7f5ebe98a740 1 -- 172.21.15.169:0/3745052584 shutdown_connections 2024-03-28T01:03:52.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.528+0000 7f5ebe98a740 1 --2- 172.21.15.169:0/3745052584 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6ea4274c0 0x55b6ea42bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.528+0000 7f5ebe98a740 1 --2- 172.21.15.169:0/3745052584 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 0x55b6ea4330c0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.528+0000 7f5ebe98a740 1 --2- 172.21.15.169:0/3745052584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6ea421960 0x55b6ea420b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.529+0000 7f5ebe98a740 1 -- 172.21.15.169:0/3745052584 >> 172.21.15.169:0/3745052584 conn(0x55b6ea3ea1a0 msgr2=0x55b6ea42b790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:52.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.531+0000 7f5ebe98a740 1 -- 172.21.15.169:0/3745052584 shutdown_connections 2024-03-28T01:03:52.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.532+0000 7f5ebe98a740 1 -- 172.21.15.169:0/3745052584 wait complete. 2024-03-28T01:03:52.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.532+0000 7f5ebe98a740 1 -- start start 2024-03-28T01:03:52.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.532+0000 7f5ebe98a740 1 Processor -- start 2024-03-28T01:03:52.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.533+0000 7f5ebe98a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6ea421960 0x55b6ea431720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.533+0000 7f5ebe98a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 0x55b6ea431c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.533+0000 7f5ebe98a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6ea4274c0 0x55b6ea4321a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.533+0000 7f5ebe98a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb41a6750 con 0x55b6ea422350 2024-03-28T01:03:52.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.533+0000 7f5ebe98a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb41a7a50 con 0x55b6ea4274c0 2024-03-28T01:03:52.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.533+0000 7f5ebe98a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb41a6d90 con 0x55b6ea421960 2024-03-28T01:03:52.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.533+0000 7f5ebe98a740 10 client.0 _ll_register_callbacks cb 0x7f5ea4037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:52.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.534+0000 7f5eb9b47640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 0x55b6ea431c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.534+0000 7f5eb9b47640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 0x55b6ea431c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38520/0 (socket says 172.21.15.169:38520) 2024-03-28T01:03:52.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.534+0000 7f5eb9b47640 1 -- 172.21.15.169:0/699691337 learned_addr learned my addr 172.21.15.169:0/699691337 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:52.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.534+0000 7f5eb9b47640 1 -- 172.21.15.169:0/699691337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6ea421960 msgr2=0x55b6ea431720 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:03:52.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.534+0000 7f5eb9b47640 1 --2- 172.21.15.169:0/699691337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6ea421960 0x55b6ea431720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.534+0000 7f5eb9b47640 1 -- 172.21.15.169:0/699691337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6ea4274c0 msgr2=0x55b6ea4321a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:03:52.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.534+0000 7f5eb9b47640 1 --2- 172.21.15.169:0/699691337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6ea4274c0 0x55b6ea4321a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.534+0000 7f5eb9b47640 1 -- 172.21.15.169:0/699691337 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b6ea4ac880 con 0x55b6ea422350 2024-03-28T01:03:52.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.535+0000 7f5eb9b47640 1 --2- 172.21.15.169:0/699691337 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 0x55b6ea431c60 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f5eb0000c00 tx=0x7f5eb0009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=685a0a25bd39d29f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:52.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.535+0000 7f5e9d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:03:52.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.536+0000 7f5e9d7fa640 1 -- 172.21.15.169:0/699691337 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb002f070 con 0x55b6ea422350 2024-03-28T01:03:52.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.536+0000 7f5ebe98a740 1 -- 172.21.15.169:0/699691337 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5eb0028070 con 0x55b6ea422350 2024-03-28T01:03:52.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.536+0000 7f5e9d7fa640 1 -- 172.21.15.169:0/699691337 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f5eb0029df0 con 0x55b6ea422350 2024-03-28T01:03:52.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.536+0000 7f5e9d7fa640 1 -- 172.21.15.169:0/699691337 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5eb003d050 con 0x55b6ea422350 2024-03-28T01:03:52.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.537+0000 7f5ebe98a740 10 client.16899 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:52.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.537+0000 7f5ebe98a740 1 -- 172.21.15.169:0/699691337 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55b6ea4ada60 con 0x55b6ea422350 2024-03-28T01:03:52.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.537+0000 7f5e9d7fa640 1 -- 172.21.15.169:0/699691337 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5eb0028070 con 0x55b6ea422350 2024-03-28T01:03:52.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.538+0000 7f5e9d7fa640 1 -- 172.21.15.169:0/699691337 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f5eb000f7a0 con 0x55b6ea422350 2024-03-28T01:03:52.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.538+0000 7f5ebe98a740 10 client.16899 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:52.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.538+0000 7f5ebe98a740 10 client.16899 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:52.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.538+0000 7f5ebe98a740 20 client.16899 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:52.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.538+0000 7f5ebe98a740 1 --2- 172.21.15.169:0/699691337 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b6ea437780 0x55b6ea4b0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.538+0000 7f5ebe98a740 4 client.16899 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T01:03:52.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.538+0000 7f5ebe98a740 1 -- 172.21.15.169:0/699691337 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55b6ea3945b0 con 0x55b6ea437780 2024-03-28T01:03:52.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.539+0000 7f5eb9346640 1 --2- 172.21.15.169:0/699691337 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b6ea437780 0x55b6ea4b0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.540+0000 7f5eb9346640 1 --2- 172.21.15.169:0/699691337 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b6ea437780 0x55b6ea4b0250 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=727488de37bb2977 server_cookie=223eb252fb60ab6b in_seq=0 out_seq=0 2024-03-28T01:03:52.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.540+0000 7f5e9d7fa640 10 client.16899 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:52.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.543+0000 7f5e9d7fa640 1 -- 172.21.15.169:0/699691337 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55b6ea3945b0 con 0x55b6ea437780 2024-03-28T01:03:52.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.543+0000 7f5e9d7fa640 10 client.16899 handle_command_reply: tid=0 2024-03-28T01:03:52.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.543+0000 7f5ebe98a740 1 client.16899 shutdown 2024-03-28T01:03:52.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.547+0000 7f5ebe98a740 1 -- 172.21.15.169:0/699691337 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 msgr2=0x55b6ea431c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:52.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.547+0000 7f5ebe98a740 1 --2- 172.21.15.169:0/699691337 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 0x55b6ea431c60 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f5eb0000c00 tx=0x7f5eb0009830 comp rx=0 tx=0).stop 2024-03-28T01:03:52.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.549+0000 7f5ebe98a740 1 -- 172.21.15.169:0/699691337 shutdown_connections 2024-03-28T01:03:52.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.549+0000 7f5ebe98a740 1 --2- 172.21.15.169:0/699691337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6ea4274c0 0x55b6ea4321a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.549+0000 7f5ebe98a740 1 --2- 172.21.15.169:0/699691337 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55b6ea437780 0x55b6ea4b0250 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.549+0000 7f5ebe98a740 1 --2- 172.21.15.169:0/699691337 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b6ea422350 0x55b6ea431c60 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.549+0000 7f5ebe98a740 1 --2- 172.21.15.169:0/699691337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6ea421960 0x55b6ea431720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.549+0000 7f5ebe98a740 1 -- 172.21.15.169:0/699691337 >> 172.21.15.169:0/699691337 conn(0x55b6ea3ea1a0 msgr2=0x55b6ea41bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:52.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.550+0000 7f5e9d7fa640 0 client.16899 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:52.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.551+0000 7f5ebe98a740 1 -- 172.21.15.169:0/699691337 shutdown_connections 2024-03-28T01:03:52.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.553+0000 7f5ebe98a740 1 -- 172.21.15.169:0/699691337 wait complete. 2024-03-28T01:03:52.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.554+0000 7f5ebe98a740 20 client.16899 trim_cache size 0 max 16384 2024-03-28T01:03:52.559 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:03:52.559 INFO:teuthology.orchestra.run.smithi169.stdout: "return_code": 0, 2024-03-28T01:03:52.559 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_tag": "83645ece-6202-46fc-8ea6-e5c584fd852c", 2024-03-28T01:03:52.559 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": "asynchronous" 2024-03-28T01:03:52.559 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:03:52.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.557+0000 7f5ebbdab640 1 -- 172.21.15.169:0/1406685811 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5ea00a2940 msgr2=0x7f5ea00a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:52.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.557+0000 7f5ebbdab640 1 --2- 172.21.15.169:0/1406685811 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5ea00a2940 0x7f5ea00a4e00 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5eac09b5e0 tx=0x7f5eac027000 comp rx=0 tx=0).stop 2024-03-28T01:03:52.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.557+0000 7f5ebbdab640 1 -- 172.21.15.169:0/1406685811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5eb41021d0 msgr2=0x7f5eb419ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:52.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.557+0000 7f5ebbdab640 1 --2- 172.21.15.169:0/1406685811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5eb41021d0 0x7f5eb419ed40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5eac009fd0 tx=0x7f5eac00b630 comp rx=0 tx=0).stop 2024-03-28T01:03:52.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.558+0000 7f5ebbdab640 1 -- 172.21.15.169:0/1406685811 shutdown_connections 2024-03-28T01:03:52.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.558+0000 7f5ebbdab640 1 --2- 172.21.15.169:0/1406685811 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5ea00a2940 0x7f5ea00a4e00 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.558+0000 7f5ebbdab640 1 --2- 172.21.15.169:0/1406685811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5eb41021d0 0x7f5eb419ed40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.558+0000 7f5ebbdab640 1 --2- 172.21.15.169:0/1406685811 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5eb4101810 0x7f5eb419c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.558+0000 7f5ebbdab640 1 --2- 172.21.15.169:0/1406685811 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5eb4100620 0x7f5eb406a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.558+0000 7f5ebbdab640 1 -- 172.21.15.169:0/1406685811 >> 172.21.15.169:0/1406685811 conn(0x7f5eb40fbbd0 msgr2=0x7f5eb4104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:52.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.562+0000 7f5ebbdab640 1 -- 172.21.15.169:0/1406685811 shutdown_connections 2024-03-28T01:03:52.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.562+0000 7f5ebbdab640 1 -- 172.21.15.169:0/1406685811 wait complete. 2024-03-28T01:03:52.593 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:03:52.593 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:03:52.788 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.784+0000 7f586559a640 1 -- 172.21.15.169:0/2454018130 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5858032340 con 0x7f5868100750 2024-03-28T01:03:52.789 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.785+0000 7f586759e640 1 -- 172.21.15.169:0/2454018130 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5868100750 msgr2=0x7f5868100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:52.789 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.785+0000 7f586759e640 1 --2- 172.21.15.169:0/2454018130 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5868100750 0x7f5868100bb0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f5858009860 tx=0x7f585802bb70 comp rx=0 tx=0).stop 2024-03-28T01:03:52.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.786+0000 7f586759e640 1 -- 172.21.15.169:0/2454018130 shutdown_connections 2024-03-28T01:03:52.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.786+0000 7f586759e640 1 --2- 172.21.15.169:0/2454018130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58681010f0 0x7f5868101570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.786+0000 7f586759e640 1 --2- 172.21.15.169:0/2454018130 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5868100750 0x7f5868100bb0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.786+0000 7f586759e640 1 --2- 172.21.15.169:0/2454018130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58680ffdf0 0x7f5868100210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.786+0000 7f586759e640 1 -- 172.21.15.169:0/2454018130 >> 172.21.15.169:0/2454018130 conn(0x7f58680fba10 msgr2=0x7f58680fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:52.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.787+0000 7f586759e640 1 -- 172.21.15.169:0/2454018130 shutdown_connections 2024-03-28T01:03:52.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.787+0000 7f586759e640 1 -- 172.21.15.169:0/2454018130 wait complete. 2024-03-28T01:03:52.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.788+0000 7f586759e640 1 Processor -- start 2024-03-28T01:03:52.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.788+0000 7f586759e640 1 -- start start 2024-03-28T01:03:52.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.788+0000 7f586759e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58680ffdf0 0x7f586819dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.788+0000 7f586759e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5868100750 0x7f586819e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.788+0000 7f586759e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58681010f0 0x7f58681a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.788+0000 7f586759e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586810f410 con 0x7f58681010f0 2024-03-28T01:03:52.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.788+0000 7f586759e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586810f290 con 0x7f58680ffdf0 2024-03-28T01:03:52.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.788+0000 7f586759e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586810f590 con 0x7f5868100750 2024-03-28T01:03:52.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.788+0000 7f5866d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58681010f0 0x7f58681a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.788+0000 7f586659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58680ffdf0 0x7f586819dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.788+0000 7f5866d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58681010f0 0x7f58681a5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38530/0 (socket says 172.21.15.169:38530) 2024-03-28T01:03:52.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.788+0000 7f5866d9d640 1 -- 172.21.15.169:0/3273381342 learned_addr learned my addr 172.21.15.169:0/3273381342 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:52.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.788+0000 7f5865d9b640 1 --2- 172.21.15.169:0/3273381342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5868100750 0x7f586819e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.789+0000 7f5866d9d640 1 -- 172.21.15.169:0/3273381342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5868100750 msgr2=0x7f586819e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:52.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.789+0000 7f5866d9d640 1 --2- 172.21.15.169:0/3273381342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5868100750 0x7f586819e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.789+0000 7f5866d9d640 1 -- 172.21.15.169:0/3273381342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58680ffdf0 msgr2=0x7f586819dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:52.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.789+0000 7f5866d9d640 1 --2- 172.21.15.169:0/3273381342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58680ffdf0 0x7f586819dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.789+0000 7f5866d9d640 1 -- 172.21.15.169:0/3273381342 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58681a5ae0 con 0x7f58681010f0 2024-03-28T01:03:52.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.789+0000 7f586659c640 1 --2- 172.21.15.169:0/3273381342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58680ffdf0 0x7f586819dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:03:52.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.789+0000 7f5866d9d640 1 --2- 172.21.15.169:0/3273381342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58681010f0 0x7f58681a5470 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f586000e940 tx=0x7f586000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:52.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.789+0000 7f585f7fe640 1 -- 172.21.15.169:0/3273381342 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586000cd80 con 0x7f58681010f0 2024-03-28T01:03:52.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.789+0000 7f585f7fe640 1 -- 172.21.15.169:0/3273381342 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58600035f0 con 0x7f58681010f0 2024-03-28T01:03:52.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.789+0000 7f586759e640 1 -- 172.21.15.169:0/3273381342 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58681a5dd0 con 0x7f58681010f0 2024-03-28T01:03:52.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.789+0000 7f585f7fe640 1 -- 172.21.15.169:0/3273381342 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5860010690 con 0x7f58681010f0 2024-03-28T01:03:52.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.789+0000 7f586759e640 1 -- 172.21.15.169:0/3273381342 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58681a6290 con 0x7f58681010f0 2024-03-28T01:03:52.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.792+0000 7f586f58f740 1 Processor -- start 2024-03-28T01:03:52.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.793+0000 7f585f7fe640 1 -- 172.21.15.169:0/3273381342 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f58600037a0 con 0x7f58681010f0 2024-03-28T01:03:52.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.793+0000 7f586f58f740 1 -- start start 2024-03-28T01:03:52.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.793+0000 7f586f58f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55567ac4bf80 0x55567ac4c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.793+0000 7f586f58f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac4c970 0x55567ac51470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.793+0000 7f586f58f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac51ae0 0x55567ac53f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.793+0000 7f586f58f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55567abb13d0 con 0x55567ac51ae0 2024-03-28T01:03:52.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.793+0000 7f586f58f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55567ab59ad0 con 0x55567ac4c970 2024-03-28T01:03:52.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.793+0000 7f586f58f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55567abad470 con 0x55567ac4bf80 2024-03-28T01:03:52.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.793+0000 7f586659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac4c970 0x55567ac51470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.793+0000 7f586659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac4c970 0x55567ac51470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:34658/0 (socket says 172.21.15.169:34658) 2024-03-28T01:03:52.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.793+0000 7f586659c640 1 -- 172.21.15.169:0/490242256 learned_addr learned my addr 172.21.15.169:0/490242256 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:52.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.793+0000 7f5865d9b640 1 --2- 172.21.15.169:0/490242256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac51ae0 0x55567ac53f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.793+0000 7f585f7fe640 1 --2- 172.21.15.169:0/3273381342 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f58480a2980 0x7f58480a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.794+0000 7f5865d9b640 1 --2- 172.21.15.169:0/490242256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55567ac4bf80 0x55567ac4c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.794+0000 7f5866d9d640 1 --2- 172.21.15.169:0/3273381342 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f58480a2980 0x7f58480a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.794+0000 7f586659c640 1 -- 172.21.15.169:0/490242256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55567ac4bf80 msgr2=0x55567ac4c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:52.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.794+0000 7f586659c640 1 --2- 172.21.15.169:0/490242256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55567ac4bf80 0x55567ac4c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.794+0000 7f586659c640 1 -- 172.21.15.169:0/490242256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac51ae0 msgr2=0x55567ac53f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:52.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.794+0000 7f586659c640 1 --2- 172.21.15.169:0/490242256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac51ae0 0x55567ac53f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.794+0000 7f586659c640 1 -- 172.21.15.169:0/490242256 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55567a98f7d0 con 0x55567ac4c970 2024-03-28T01:03:52.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.794+0000 7f585f7fe640 1 -- 172.21.15.169:0/3273381342 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f58600cf010 con 0x7f58681010f0 2024-03-28T01:03:52.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.794+0000 7f5865d9b640 1 --2- 172.21.15.169:0/490242256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55567ac4bf80 0x55567ac4c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:03:52.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.794+0000 7f586659c640 1 --2- 172.21.15.169:0/490242256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac4c970 0x55567ac51470 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f58500107e0 tx=0x7f5850010ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9eb2247b882c6e2c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:52.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.794+0000 7f585cff9640 1 -- 172.21.15.169:0/490242256 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5850016070 con 0x55567ac4c970 2024-03-28T01:03:52.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.794+0000 7f5866d9d640 1 --2- 172.21.15.169:0/3273381342 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f58480a2980 0x7f58480a4e40 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f586009b310 tx=0x7f5860007b90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:52.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.794+0000 7f585cff9640 1 -- 172.21.15.169:0/490242256 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f585001b460 con 0x55567ac4c970 2024-03-28T01:03:52.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.794+0000 7f585cff9640 1 -- 172.21.15.169:0/490242256 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58500126b0 con 0x55567ac4c970 2024-03-28T01:03:52.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.795+0000 7f586f58f740 1 -- 172.21.15.169:0/490242256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac4c970 msgr2=0x55567ac51470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:52.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.795+0000 7f586f58f740 1 --2- 172.21.15.169:0/490242256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac4c970 0x55567ac51470 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f58500107e0 tx=0x7f5850010ca0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.795+0000 7f586f58f740 1 -- 172.21.15.169:0/490242256 shutdown_connections 2024-03-28T01:03:52.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.795+0000 7f586f58f740 1 --2- 172.21.15.169:0/490242256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac51ae0 0x55567ac53f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.795+0000 7f586f58f740 1 --2- 172.21.15.169:0/490242256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac4c970 0x55567ac51470 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.795+0000 7f586f58f740 1 --2- 172.21.15.169:0/490242256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55567ac4bf80 0x55567ac4c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.795+0000 7f586f58f740 1 -- 172.21.15.169:0/490242256 >> 172.21.15.169:0/490242256 conn(0x55567ac474f0 msgr2=0x55567ac478f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:52.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.795+0000 7f586f58f740 1 -- 172.21.15.169:0/490242256 shutdown_connections 2024-03-28T01:03:52.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.795+0000 7f586f58f740 1 -- 172.21.15.169:0/490242256 wait complete. 2024-03-28T01:03:52.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.796+0000 7f586f58f740 1 -- start start 2024-03-28T01:03:52.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.796+0000 7f586f58f740 1 Processor -- start 2024-03-28T01:03:52.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.796+0000 7f586f58f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55567ac4bf80 0x55567ac56400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.796+0000 7f586f58f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac4c970 0x55567ac56940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.796+0000 7f586f58f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac51ae0 0x55567ac5d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.796+0000 7f586f58f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55567ac54a90 con 0x55567ac4c970 2024-03-28T01:03:52.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.796+0000 7f586f58f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55567ab6d060 con 0x55567ac51ae0 2024-03-28T01:03:52.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.796+0000 7f586f58f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55567ac54c10 con 0x55567ac4bf80 2024-03-28T01:03:52.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.796+0000 7f586f58f740 10 client.0 _ll_register_callbacks cb 0x55567ab4f6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:52.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.796+0000 7f586659c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac4c970 0x55567ac56940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.796+0000 7f586659c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac4c970 0x55567ac56940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38550/0 (socket says 172.21.15.169:38550) 2024-03-28T01:03:52.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.796+0000 7f586659c640 1 -- 172.21.15.169:0/171744184 learned_addr learned my addr 172.21.15.169:0/171744184 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:52.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.796+0000 7f5865d9b640 1 --2- 172.21.15.169:0/171744184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac51ae0 0x55567ac5d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.796+0000 7f5865d9b640 1 --2- 172.21.15.169:0/171744184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55567ac4bf80 0x55567ac56400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.797+0000 7f586659c640 1 -- 172.21.15.169:0/171744184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55567ac4bf80 msgr2=0x55567ac56400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:52.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.797+0000 7f586659c640 1 --2- 172.21.15.169:0/171744184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55567ac4bf80 0x55567ac56400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.797+0000 7f586659c640 1 -- 172.21.15.169:0/171744184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac51ae0 msgr2=0x55567ac5d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:52.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.797+0000 7f586659c640 1 --2- 172.21.15.169:0/171744184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac51ae0 0x55567ac5d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.797+0000 7f586659c640 1 -- 172.21.15.169:0/171744184 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55567a8c53f0 con 0x55567ac4c970 2024-03-28T01:03:52.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.797+0000 7f586659c640 1 --2- 172.21.15.169:0/171744184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac4c970 0x55567ac56940 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f5850006010 tx=0x7f585000eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dfd8e2c6b0e4bf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:52.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.797+0000 7f5855ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:03:52.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.797+0000 7f5855ffb640 1 -- 172.21.15.169:0/171744184 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5850029060 con 0x55567ac4c970 2024-03-28T01:03:52.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.797+0000 7f5855ffb640 1 -- 172.21.15.169:0/171744184 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f5850004a40 con 0x55567ac4c970 2024-03-28T01:03:52.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.797+0000 7f586f58f740 1 -- 172.21.15.169:0/171744184 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f585001b460 con 0x55567ac4c970 2024-03-28T01:03:52.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.797+0000 7f5855ffb640 1 -- 172.21.15.169:0/171744184 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5850030050 con 0x55567ac4c970 2024-03-28T01:03:52.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.797+0000 7f586f58f740 10 client.16911 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:52.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.797+0000 7f586f58f740 1 -- 172.21.15.169:0/171744184 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55567a8850f0 con 0x55567ac4c970 2024-03-28T01:03:52.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.797+0000 7f5855ffb640 1 -- 172.21.15.169:0/171744184 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f585001b460 con 0x55567ac4c970 2024-03-28T01:03:52.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.798+0000 7f5855ffb640 1 -- 172.21.15.169:0/171744184 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f585000f040 con 0x55567ac4c970 2024-03-28T01:03:52.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.798+0000 7f586f58f740 10 client.16911 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:52.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.798+0000 7f586f58f740 10 client.16911 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:52.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.798+0000 7f586f58f740 20 client.16911 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:52.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.798+0000 7f586f58f740 1 --2- 172.21.15.169:0/171744184 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55567ac61b90 0x55567ac63f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.798+0000 7f586f58f740 4 client.16911 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:03:52.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.798+0000 7f586f58f740 1 -- 172.21.15.169:0/171744184 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55567abbe910 con 0x55567ac61b90 2024-03-28T01:03:52.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.798+0000 7f5865d9b640 1 --2- 172.21.15.169:0/171744184 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55567ac61b90 0x55567ac63f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.798+0000 7f5855ffb640 10 client.16911 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:52.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.798+0000 7f5865d9b640 1 --2- 172.21.15.169:0/171744184 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55567ac61b90 0x55567ac63f80 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60150bdd0b292faf server_cookie=3ae1c2d30bf509b0 in_seq=0 out_seq=0 2024-03-28T01:03:52.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.799+0000 7f5855ffb640 1 -- 172.21.15.169:0/171744184 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55567abbe910 con 0x55567ac61b90 2024-03-28T01:03:52.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.799+0000 7f5855ffb640 10 client.16911 handle_command_reply: tid=0 2024-03-28T01:03:52.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.799+0000 7f586f58f740 1 client.16911 shutdown 2024-03-28T01:03:52.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.799+0000 7f586f58f740 1 -- 172.21.15.169:0/171744184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac4c970 msgr2=0x55567ac56940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:52.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.799+0000 7f586f58f740 1 --2- 172.21.15.169:0/171744184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac4c970 0x55567ac56940 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f5850006010 tx=0x7f585000eb00 comp rx=0 tx=0).stop 2024-03-28T01:03:52.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.799+0000 7f586f58f740 1 -- 172.21.15.169:0/171744184 shutdown_connections 2024-03-28T01:03:52.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.799+0000 7f586f58f740 1 --2- 172.21.15.169:0/171744184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac51ae0 0x55567ac5d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.799+0000 7f586f58f740 1 --2- 172.21.15.169:0/171744184 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55567ac61b90 0x55567ac63f80 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.799+0000 7f586f58f740 1 --2- 172.21.15.169:0/171744184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac4c970 0x55567ac56940 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.799+0000 7f586f58f740 1 --2- 172.21.15.169:0/171744184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55567ac4bf80 0x55567ac56400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.799+0000 7f586f58f740 1 -- 172.21.15.169:0/171744184 >> 172.21.15.169:0/171744184 conn(0x55567ac474f0 msgr2=0x55567ac467b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:52.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.799+0000 7f5855ffb640 0 client.16911 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:52.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.799+0000 7f586f58f740 1 -- 172.21.15.169:0/171744184 shutdown_connections 2024-03-28T01:03:52.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.799+0000 7f586f58f740 1 -- 172.21.15.169:0/171744184 wait complete. 2024-03-28T01:03:52.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.799+0000 7f586f58f740 20 client.16911 trim_cache size 0 max 16384 2024-03-28T01:03:52.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.828+0000 7f586f58f740 1 Processor -- start 2024-03-28T01:03:52.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.829+0000 7f586f58f740 1 -- start start 2024-03-28T01:03:52.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.830+0000 7f586f58f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55567ac4bf80 0x55567ac45f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.830+0000 7f586f58f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac4c970 0x55567ac5d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.830+0000 7f586f58f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac51ae0 0x55567ac46460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.830+0000 7f586f58f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58681b2b40 con 0x55567ac51ae0 2024-03-28T01:03:52.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.830+0000 7f586f58f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58681b0640 con 0x55567ac4c970 2024-03-28T01:03:52.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.830+0000 7f586f58f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58681ae6e0 con 0x55567ac4bf80 2024-03-28T01:03:52.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.837+0000 7f5865d9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55567ac4bf80 0x55567ac45f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.837+0000 7f5865d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac51ae0 0x55567ac46460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.837+0000 7f5865d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac51ae0 0x55567ac46460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38552/0 (socket says 172.21.15.169:38552) 2024-03-28T01:03:52.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.837+0000 7f5865d9b640 1 -- 172.21.15.169:0/4193629999 learned_addr learned my addr 172.21.15.169:0/4193629999 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:52.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.837+0000 7f586659c640 1 --2- 172.21.15.169:0/4193629999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac4c970 0x55567ac5d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.839+0000 7f5865d9b640 1 -- 172.21.15.169:0/4193629999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55567ac4bf80 msgr2=0x55567ac45f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:52.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.839+0000 7f5865d9b640 1 --2- 172.21.15.169:0/4193629999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55567ac4bf80 0x55567ac45f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.839+0000 7f5865d9b640 1 -- 172.21.15.169:0/4193629999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac4c970 msgr2=0x55567ac5d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:52.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.839+0000 7f5865d9b640 1 --2- 172.21.15.169:0/4193629999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac4c970 0x55567ac5d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.839+0000 7f5865d9b640 1 -- 172.21.15.169:0/4193629999 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55567ac510a0 con 0x55567ac51ae0 2024-03-28T01:03:52.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.839+0000 7f5865d9b640 1 --2- 172.21.15.169:0/4193629999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac51ae0 0x55567ac46460 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f5858009250 tx=0x7f585802bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad459654b23e6bc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:52.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.840+0000 7f5857fff640 1 -- 172.21.15.169:0/4193629999 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5858039040 con 0x55567ac51ae0 2024-03-28T01:03:52.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.840+0000 7f5857fff640 1 -- 172.21.15.169:0/4193629999 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5858005030 con 0x55567ac51ae0 2024-03-28T01:03:52.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.840+0000 7f5857fff640 1 -- 172.21.15.169:0/4193629999 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f585802c770 con 0x55567ac51ae0 2024-03-28T01:03:52.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.841+0000 7f586f58f740 1 -- 172.21.15.169:0/4193629999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac51ae0 msgr2=0x55567ac46460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:52.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.841+0000 7f586f58f740 1 --2- 172.21.15.169:0/4193629999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac51ae0 0x55567ac46460 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f5858009250 tx=0x7f585802bb70 comp rx=0 tx=0).stop 2024-03-28T01:03:52.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.842+0000 7f586f58f740 1 -- 172.21.15.169:0/4193629999 shutdown_connections 2024-03-28T01:03:52.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.842+0000 7f586f58f740 1 --2- 172.21.15.169:0/4193629999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac51ae0 0x55567ac46460 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.842+0000 7f586f58f740 1 --2- 172.21.15.169:0/4193629999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac4c970 0x55567ac5d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.842+0000 7f586f58f740 1 --2- 172.21.15.169:0/4193629999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55567ac4bf80 0x55567ac45f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.842+0000 7f586f58f740 1 -- 172.21.15.169:0/4193629999 >> 172.21.15.169:0/4193629999 conn(0x55567ac474f0 msgr2=0x55567ac63ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:52.850 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.846+0000 7f586f58f740 1 -- 172.21.15.169:0/4193629999 shutdown_connections 2024-03-28T01:03:52.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.847+0000 7f586f58f740 1 -- 172.21.15.169:0/4193629999 wait complete. 2024-03-28T01:03:52.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.847+0000 7f586f58f740 1 -- start start 2024-03-28T01:03:52.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.847+0000 7f586f58f740 1 Processor -- start 2024-03-28T01:03:52.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.847+0000 7f586f58f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac4bf80 0x55567ac5b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.848+0000 7f586f58f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac4c970 0x55567ac5bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.848+0000 7f586f58f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55567ac51ae0 0x55567ac5cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.848+0000 7f586f58f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58681aece0 con 0x55567ac4bf80 2024-03-28T01:03:52.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.848+0000 7f586f58f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58681affe0 con 0x55567ac4c970 2024-03-28T01:03:52.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.848+0000 7f586f58f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58681af320 con 0x55567ac51ae0 2024-03-28T01:03:52.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.848+0000 7f586f58f740 10 client.0 _ll_register_callbacks cb 0x7f5860005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:03:52.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.848+0000 7f5865d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac4bf80 0x55567ac5b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.848+0000 7f5865d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac4bf80 0x55567ac5b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38558/0 (socket says 172.21.15.169:38558) 2024-03-28T01:03:52.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.848+0000 7f5865d9b640 1 -- 172.21.15.169:0/205722442 learned_addr learned my addr 172.21.15.169:0/205722442 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:03:52.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.848+0000 7f5865d9b640 1 --2- 172.21.15.169:0/205722442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55567ac51ae0 0x55567ac5cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.848+0000 7f586659c640 1 --2- 172.21.15.169:0/205722442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac4c970 0x55567ac5bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.849+0000 7f5865d9b640 1 -- 172.21.15.169:0/205722442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55567ac51ae0 msgr2=0x55567ac5cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.849+0000 7f5865d9b640 1 --2- 172.21.15.169:0/205722442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55567ac51ae0 0x55567ac5cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.849+0000 7f5865d9b640 1 -- 172.21.15.169:0/205722442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac4c970 msgr2=0x55567ac5bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:03:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.849+0000 7f5865d9b640 1 --2- 172.21.15.169:0/205722442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac4c970 0x55567ac5bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.849+0000 7f5865d9b640 1 -- 172.21.15.169:0/205722442 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55567ac5d210 con 0x55567ac4bf80 2024-03-28T01:03:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.849+0000 7f5865d9b640 1 --2- 172.21.15.169:0/205722442 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac4bf80 0x55567ac5b6d0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f58580349e0 tx=0x7f585804e360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7590cf6d16391c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:03:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.849+0000 7f5855ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:03:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.849+0000 7f5855ffb640 1 -- 172.21.15.169:0/205722442 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5858039070 con 0x55567ac4bf80 2024-03-28T01:03:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.849+0000 7f5855ffb640 1 -- 172.21.15.169:0/205722442 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f585802c510 con 0x55567ac4bf80 2024-03-28T01:03:52.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.849+0000 7f586f58f740 1 -- 172.21.15.169:0/205722442 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5858005030 con 0x55567ac4bf80 2024-03-28T01:03:52.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.850+0000 7f5855ffb640 1 -- 172.21.15.169:0/205722442 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f585802c710 con 0x55567ac4bf80 2024-03-28T01:03:52.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.852+0000 7f586f58f740 10 client.16917 fetch_fsmap learned FSMap version 25 2024-03-28T01:03:52.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.852+0000 7f586f58f740 1 -- 172.21.15.169:0/205722442 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55567acce980 con 0x55567ac4bf80 2024-03-28T01:03:52.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.856+0000 7f5855ffb640 1 -- 172.21.15.169:0/205722442 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5858005030 con 0x55567ac4bf80 2024-03-28T01:03:52.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.856+0000 7f5855ffb640 1 -- 172.21.15.169:0/205722442 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f585804e620 con 0x55567ac4bf80 2024-03-28T01:03:52.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.856+0000 7f586f58f740 10 client.16917 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:03:52.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.856+0000 7f586f58f740 10 client.16917 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:03:52.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.856+0000 7f586f58f740 20 client.16917 populate_metadata read hostname 'smithi169' 2024-03-28T01:03:52.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.856+0000 7f586f58f740 1 --2- 172.21.15.169:0/205722442 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55567accf2d0 0x55567acd16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:03:52.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.856+0000 7f586f58f740 4 client.16917 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:03:52.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.856+0000 7f586f58f740 1 -- 172.21.15.169:0/205722442 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55567ac561f0 con 0x55567accf2d0 2024-03-28T01:03:52.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.857+0000 7f586659c640 1 --2- 172.21.15.169:0/205722442 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55567accf2d0 0x55567acd16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:03:52.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.868+0000 7f586659c640 1 --2- 172.21.15.169:0/205722442 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55567accf2d0 0x55567acd16f0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a326a729f0b94309 server_cookie=82dd49fb90d566fa in_seq=0 out_seq=0 2024-03-28T01:03:52.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.873+0000 7f5855ffb640 10 client.16917 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:52.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.875+0000 7f5855ffb640 1 -- 172.21.15.169:0/205722442 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55567ac561f0 con 0x55567accf2d0 2024-03-28T01:03:52.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.875+0000 7f5855ffb640 10 client.16917 handle_command_reply: tid=0 2024-03-28T01:03:52.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.876+0000 7f586f58f740 1 client.16917 shutdown 2024-03-28T01:03:52.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.880+0000 7f586f58f740 1 -- 172.21.15.169:0/205722442 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac4bf80 msgr2=0x55567ac5b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:52.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.880+0000 7f586f58f740 1 --2- 172.21.15.169:0/205722442 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac4bf80 0x55567ac5b6d0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f58580349e0 tx=0x7f585804e360 comp rx=0 tx=0).stop 2024-03-28T01:03:52.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.881+0000 7f586f58f740 1 -- 172.21.15.169:0/205722442 shutdown_connections 2024-03-28T01:03:52.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.881+0000 7f586f58f740 1 --2- 172.21.15.169:0/205722442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55567ac51ae0 0x55567ac5cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.881+0000 7f586f58f740 1 --2- 172.21.15.169:0/205722442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55567ac4c970 0x55567ac5bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.881+0000 7f586f58f740 1 --2- 172.21.15.169:0/205722442 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55567accf2d0 0x55567acd16f0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.881+0000 7f586f58f740 1 --2- 172.21.15.169:0/205722442 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55567ac4bf80 0x55567ac5b6d0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.881+0000 7f586f58f740 1 -- 172.21.15.169:0/205722442 >> 172.21.15.169:0/205722442 conn(0x55567ac474f0 msgr2=0x55567ac4a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:52.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.883+0000 7f5855ffb640 0 client.16917 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:03:52.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.885+0000 7f586f58f740 1 -- 172.21.15.169:0/205722442 shutdown_connections 2024-03-28T01:03:52.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.886+0000 7f586f58f740 1 -- 172.21.15.169:0/205722442 wait complete. 2024-03-28T01:03:52.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.887+0000 7f586f58f740 20 client.16917 trim_cache size 0 max 16384 2024-03-28T01:03:52.893 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:03:52.893 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T01:03:52.893 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": { 2024-03-28T01:03:52.894 INFO:teuthology.orchestra.run.smithi169.stdout: "83645ece-6202-46fc-8ea6-e5c584fd852c": { 2024-03-28T01:03:52.894 INFO:teuthology.orchestra.run.smithi169.stdout: "path": "/", 2024-03-28T01:03:52.894 INFO:teuthology.orchestra.run.smithi169.stdout: "tag": "83645ece-6202-46fc-8ea6-e5c584fd852c", 2024-03-28T01:03:52.894 INFO:teuthology.orchestra.run.smithi169.stdout: "options": "recursive,force" 2024-03-28T01:03:52.894 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:03:52.894 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:03:52.894 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:03:52.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.896+0000 7f586759e640 1 -- 172.21.15.169:0/3273381342 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f58480a2980 msgr2=0x7f58480a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:52.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.896+0000 7f586759e640 1 --2- 172.21.15.169:0/3273381342 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f58480a2980 0x7f58480a4e40 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f586009b310 tx=0x7f5860007b90 comp rx=0 tx=0).stop 2024-03-28T01:03:52.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.896+0000 7f586759e640 1 -- 172.21.15.169:0/3273381342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58681010f0 msgr2=0x7f58681a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:03:52.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.896+0000 7f586759e640 1 --2- 172.21.15.169:0/3273381342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58681010f0 0x7f58681a5470 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f586000e940 tx=0x7f586000ee00 comp rx=0 tx=0).stop 2024-03-28T01:03:52.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.896+0000 7f586759e640 1 -- 172.21.15.169:0/3273381342 shutdown_connections 2024-03-28T01:03:52.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.896+0000 7f586759e640 1 --2- 172.21.15.169:0/3273381342 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f58480a2980 0x7f58480a4e40 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.896+0000 7f586759e640 1 --2- 172.21.15.169:0/3273381342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58681010f0 0x7f58681a5470 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.896+0000 7f586759e640 1 --2- 172.21.15.169:0/3273381342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5868100750 0x7f586819e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.896+0000 7f586759e640 1 --2- 172.21.15.169:0/3273381342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58680ffdf0 0x7f586819dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:03:52.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.896+0000 7f586759e640 1 -- 172.21.15.169:0/3273381342 >> 172.21.15.169:0/3273381342 conn(0x7f58680fba10 msgr2=0x7f58680fd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:03:52.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.898+0000 7f586759e640 1 -- 172.21.15.169:0/3273381342 shutdown_connections 2024-03-28T01:03:52.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:03:52.899+0000 7f586759e640 1 -- 172.21.15.169:0/3273381342 wait complete. 2024-03-28T01:03:52.935 INFO:tasks.cephfs.filesystem:scrub status for tag:83645ece-6202-46fc-8ea6-e5c584fd852c - {'path': '/', 'tag': '83645ece-6202-46fc-8ea6-e5c584fd852c', 'options': 'recursive,force'} 2024-03-28T01:03:53.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:53 smithi174 ceph-mon[36060]: pgmap v572: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 115 MiB/s rd, 415 MiB/s wr, 4.14k op/s 2024-03-28T01:03:53.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:53 smithi174 ceph-mon[36060]: scrub queued for path: / 2024-03-28T01:03:53.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:53 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:03:53.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:53 smithi174 ceph-mon[36060]: scrub summary: active paths [/] 2024-03-28T01:03:53.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:53 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:03:53.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:53 smithi177 ceph-mon[36206]: pgmap v572: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 115 MiB/s rd, 415 MiB/s wr, 4.14k op/s 2024-03-28T01:03:53.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:53 smithi177 ceph-mon[36206]: scrub queued for path: / 2024-03-28T01:03:53.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:53 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:03:53.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:53 smithi177 ceph-mon[36206]: scrub summary: active paths [/] 2024-03-28T01:03:53.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:53 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:03:54.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:53 smithi169 ceph-mon[32274]: pgmap v572: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 115 MiB/s rd, 415 MiB/s wr, 4.14k op/s 2024-03-28T01:03:54.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:53 smithi169 ceph-mon[32274]: scrub queued for path: / 2024-03-28T01:03:54.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:53 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:03:54.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:53 smithi169 ceph-mon[32274]: scrub summary: active paths [/] 2024-03-28T01:03:54.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:53 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:03:54.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:54 smithi174 ceph-mon[36060]: scrub summary: idle 2024-03-28T01:03:54.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:54 smithi177 ceph-mon[36206]: scrub summary: idle 2024-03-28T01:03:55.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:54 smithi169 ceph-mon[32274]: scrub summary: idle 2024-03-28T01:03:55.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:55 smithi174 ceph-mon[36060]: pgmap v573: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s rd, 430 MiB/s wr, 4.07k op/s 2024-03-28T01:03:55.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:55 smithi177 ceph-mon[36206]: pgmap v573: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s rd, 430 MiB/s wr, 4.07k op/s 2024-03-28T01:03:56.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:55 smithi169 ceph-mon[32274]: pgmap v573: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s rd, 430 MiB/s wr, 4.07k op/s 2024-03-28T01:03:57.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:57 smithi174 ceph-mon[36060]: pgmap v574: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s rd, 438 MiB/s wr, 3.83k op/s 2024-03-28T01:03:57.921 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:57 smithi177 ceph-mon[36206]: pgmap v574: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s rd, 438 MiB/s wr, 3.83k op/s 2024-03-28T01:03:58.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:57 smithi169 ceph-mon[32274]: pgmap v574: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s rd, 438 MiB/s wr, 3.83k op/s 2024-03-28T01:03:59.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:03:59 smithi174 ceph-mon[36060]: pgmap v575: 129 pgs: 129 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s rd, 471 MiB/s wr, 4.32k op/s 2024-03-28T01:03:59.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:03:59 smithi177 ceph-mon[36206]: pgmap v575: 129 pgs: 129 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s rd, 471 MiB/s wr, 4.32k op/s 2024-03-28T01:04:00.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:03:59 smithi169 ceph-mon[32274]: pgmap v575: 129 pgs: 129 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s rd, 471 MiB/s wr, 4.32k op/s 2024-03-28T01:04:01.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:01 smithi177 ceph-mon[36206]: pgmap v576: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s rd, 419 MiB/s wr, 4.15k op/s 2024-03-28T01:04:02.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:01 smithi169 ceph-mon[32274]: pgmap v576: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s rd, 419 MiB/s wr, 4.15k op/s 2024-03-28T01:04:02.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:01 smithi174 ceph-mon[36060]: pgmap v576: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s rd, 419 MiB/s wr, 4.15k op/s 2024-03-28T01:04:03.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:03 smithi177 ceph-mon[36206]: pgmap v577: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s rd, 423 MiB/s wr, 4.29k op/s 2024-03-28T01:04:04.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:03 smithi169 ceph-mon[32274]: pgmap v577: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s rd, 423 MiB/s wr, 4.29k op/s 2024-03-28T01:04:04.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:03 smithi174 ceph-mon[36060]: pgmap v577: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s rd, 423 MiB/s wr, 4.29k op/s 2024-03-28T01:04:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.489+0000 7f0a54e85640 1 -- 172.21.15.169:0/1746684551 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a5010b010 msgr2=0x7f0a5010d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.489+0000 7f0a54e85640 1 --2- 172.21.15.169:0/1746684551 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a5010b010 0x7f0a5010d480 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f0a38009860 tx=0x7f0a3802bbb0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.494 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.490+0000 7f0a54e85640 1 -- 172.21.15.169:0/1746684551 shutdown_connections 2024-03-28T01:04:05.494 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.490+0000 7f0a54e85640 1 --2- 172.21.15.169:0/1746684551 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a5010d9c0 0x7f0a5010de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.494 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.490+0000 7f0a54e85640 1 --2- 172.21.15.169:0/1746684551 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a5010b010 0x7f0a5010d480 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.490+0000 7f0a54e85640 1 --2- 172.21.15.169:0/1746684551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a501086a0 0x7f0a5010aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.490+0000 7f0a54e85640 1 -- 172.21.15.169:0/1746684551 >> 172.21.15.169:0/1746684551 conn(0x7f0a5006e6e0 msgr2=0x7f0a50070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:05.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.494+0000 7f0a54e85640 1 -- 172.21.15.169:0/1746684551 shutdown_connections 2024-03-28T01:04:05.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.494+0000 7f0a54e85640 1 -- 172.21.15.169:0/1746684551 wait complete. 2024-03-28T01:04:05.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.495+0000 7f0a54e85640 1 Processor -- start 2024-03-28T01:04:05.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.495+0000 7f0a54e85640 1 -- start start 2024-03-28T01:04:05.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.495+0000 7f0a54e85640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a501086a0 0x7f0a50119810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:05.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.495+0000 7f0a54e85640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a5010b010 0x7f0a50119d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:05.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.495+0000 7f0a54e85640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a5010d9c0 0x7f0a50117e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:05.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.495+0000 7f0a54e85640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a501107e0 con 0x7f0a5010d9c0 2024-03-28T01:04:05.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.495+0000 7f0a54e85640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a50110660 con 0x7f0a501086a0 2024-03-28T01:04:05.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.495+0000 7f0a54e85640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a50110960 con 0x7f0a5010b010 2024-03-28T01:04:05.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.495+0000 7f0a4ed9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a5010d9c0 0x7f0a50117e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:05.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.495+0000 7f0a4ed9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a5010d9c0 0x7f0a50117e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:55532/0 (socket says 172.21.15.169:55532) 2024-03-28T01:04:05.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.495+0000 7f0a4ed9d640 1 -- 172.21.15.169:0/1282825386 learned_addr learned my addr 172.21.15.169:0/1282825386 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:05.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.495+0000 7f0a4e59c640 1 --2- 172.21.15.169:0/1282825386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a501086a0 0x7f0a50119810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:05.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.497+0000 7f0a4dd9b640 1 --2- 172.21.15.169:0/1282825386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a5010b010 0x7f0a50119d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:05.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.497+0000 7f0a4e59c640 1 -- 172.21.15.169:0/1282825386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a5010b010 msgr2=0x7f0a50119d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:05.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.497+0000 7f0a4e59c640 1 --2- 172.21.15.169:0/1282825386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a5010b010 0x7f0a50119d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.497+0000 7f0a4e59c640 1 -- 172.21.15.169:0/1282825386 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a5010d9c0 msgr2=0x7f0a50117e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:05.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.497+0000 7f0a4e59c640 1 --2- 172.21.15.169:0/1282825386 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a5010d9c0 0x7f0a50117e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.502 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.497+0000 7f0a4e59c640 1 -- 172.21.15.169:0/1282825386 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a501184f0 con 0x7f0a501086a0 2024-03-28T01:04:05.502 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.498+0000 7f0a4e59c640 1 --2- 172.21.15.169:0/1282825386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a501086a0 0x7f0a50119810 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0a4800e790 tx=0x7f0a4800ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:05.502 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.499+0000 7f0a477fe640 1 -- 172.21.15.169:0/1282825386 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4800eea0 con 0x7f0a501086a0 2024-03-28T01:04:05.503 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.499+0000 7f0a54e85640 1 -- 172.21.15.169:0/1282825386 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a501187e0 con 0x7f0a501086a0 2024-03-28T01:04:05.503 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.500+0000 7f0a54e85640 1 -- 172.21.15.169:0/1282825386 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a50118c90 con 0x7f0a501086a0 2024-03-28T01:04:05.503 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.500+0000 7f0a477fe640 1 -- 172.21.15.169:0/1282825386 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a4800ccc0 con 0x7f0a501086a0 2024-03-28T01:04:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.501+0000 7f0a477fe640 1 -- 172.21.15.169:0/1282825386 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a48018760 con 0x7f0a501086a0 2024-03-28T01:04:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.505+0000 7f0a477fe640 1 -- 172.21.15.169:0/1282825386 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0a48018900 con 0x7f0a501086a0 2024-03-28T01:04:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.506+0000 7f0a477fe640 1 --2- 172.21.15.169:0/1282825386 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0a3c0a28e0 0x7f0a3c0a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.507+0000 7f0a57a7c740 1 Processor -- start 2024-03-28T01:04:05.510 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.507+0000 7f0a57a7c740 1 -- start start 2024-03-28T01:04:05.510 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.506+0000 7f0a477fe640 1 -- 172.21.15.169:0/1282825386 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f0a480cefc0 con 0x7f0a501086a0 2024-03-28T01:04:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.508+0000 7f0a57a7c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5655208fd920 0x5655208fdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.508+0000 7f0a57a7c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5655208fe310 0x565520902e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.508+0000 7f0a57a7c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565520903480 0x565520905910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.508+0000 7f0a57a7c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x565520862b10 con 0x5655208fd920 2024-03-28T01:04:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.508+0000 7f0a4dd9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5655208fe310 0x565520902e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.508+0000 7f0a4dd9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5655208fe310 0x565520902e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:34572/0 (socket says 172.21.15.169:34572) 2024-03-28T01:04:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.508+0000 7f0a4dd9b640 1 -- 172.21.15.169:0/3047812 learned_addr learned my addr 172.21.15.169:0/3047812 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.509+0000 7f0a4ed9d640 1 --2- 172.21.15.169:0/3047812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565520903480 0x565520905910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.509+0000 7f0a57a7c740 1 -- 172.21.15.169:0/3047812 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x56552080b900 con 0x5655208fe310 2024-03-28T01:04:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.509+0000 7f0a57a7c740 1 -- 172.21.15.169:0/3047812 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56552085ebb0 con 0x565520903480 2024-03-28T01:04:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.509+0000 7f0a4e59c640 1 --2- 172.21.15.169:0/3047812 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5655208fd920 0x5655208fdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.509+0000 7f0a4dd9b640 1 --2- 172.21.15.169:0/1282825386 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0a3c0a28e0 0x7f0a3c0a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.510+0000 7f0a4ed9d640 1 -- 172.21.15.169:0/3047812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5655208fe310 msgr2=0x565520902e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.510+0000 7f0a4ed9d640 1 --2- 172.21.15.169:0/3047812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5655208fe310 0x565520902e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.510+0000 7f0a4ed9d640 1 -- 172.21.15.169:0/3047812 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5655208fd920 msgr2=0x5655208fdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.510+0000 7f0a4ed9d640 1 --2- 172.21.15.169:0/3047812 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5655208fd920 0x5655208fdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.510+0000 7f0a4ed9d640 1 -- 172.21.15.169:0/3047812 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565520577210 con 0x565520903480 2024-03-28T01:04:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.511+0000 7f0a4ed9d640 1 --2- 172.21.15.169:0/3047812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565520903480 0x565520905910 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0a40009f50 tx=0x7f0a40011870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=408052b2cf6a87db server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.511+0000 7f0a44ff9640 1 -- 172.21.15.169:0/3047812 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a40004d20 con 0x565520903480 2024-03-28T01:04:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.512+0000 7f0a44ff9640 1 -- 172.21.15.169:0/3047812 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a400129f0 con 0x565520903480 2024-03-28T01:04:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.512+0000 7f0a44ff9640 1 -- 172.21.15.169:0/3047812 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a40004d20 con 0x565520903480 2024-03-28T01:04:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.513+0000 7f0a57a7c740 1 -- 172.21.15.169:0/3047812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565520903480 msgr2=0x565520905910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.513+0000 7f0a57a7c740 1 --2- 172.21.15.169:0/3047812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565520903480 0x565520905910 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0a40009f50 tx=0x7f0a40011870 comp rx=0 tx=0).stop 2024-03-28T01:04:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.513+0000 7f0a4dd9b640 1 --2- 172.21.15.169:0/1282825386 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0a3c0a28e0 0x7f0a3c0a4da0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x5655208fe710 tx=0x7f0a380349b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.515+0000 7f0a57a7c740 1 -- 172.21.15.169:0/3047812 shutdown_connections 2024-03-28T01:04:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.515+0000 7f0a57a7c740 1 --2- 172.21.15.169:0/3047812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565520903480 0x565520905910 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.515+0000 7f0a57a7c740 1 --2- 172.21.15.169:0/3047812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5655208fe310 0x565520902e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.515+0000 7f0a57a7c740 1 --2- 172.21.15.169:0/3047812 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5655208fd920 0x5655208fdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.515+0000 7f0a57a7c740 1 -- 172.21.15.169:0/3047812 >> 172.21.15.169:0/3047812 conn(0x5655208f8e90 msgr2=0x5655208f9290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.517+0000 7f0a57a7c740 1 -- 172.21.15.169:0/3047812 shutdown_connections 2024-03-28T01:04:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.517+0000 7f0a57a7c740 1 -- 172.21.15.169:0/3047812 wait complete. 2024-03-28T01:04:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.517+0000 7f0a57a7c740 1 -- start start 2024-03-28T01:04:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.518+0000 7f0a57a7c740 1 Processor -- start 2024-03-28T01:04:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.518+0000 7f0a57a7c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5655208fd920 0x565520907b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.518+0000 7f0a57a7c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5655208fe310 0x5655209080d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.519+0000 7f0a57a7c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x565520903480 0x56552090f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.519+0000 7f0a57a7c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x565520906430 con 0x565520903480 2024-03-28T01:04:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.519+0000 7f0a4e59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5655208fd920 0x565520907b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.519+0000 7f0a4e59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5655208fd920 0x565520907b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:34582/0 (socket says 172.21.15.169:34582) 2024-03-28T01:04:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.519+0000 7f0a4e59c640 1 -- 172.21.15.169:0/837036265 learned_addr learned my addr 172.21.15.169:0/837036265 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.519+0000 7f0a57a7c740 1 -- 172.21.15.169:0/837036265 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x56552081ed50 con 0x5655208fd920 2024-03-28T01:04:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.519+0000 7f0a57a7c740 1 -- 172.21.15.169:0/837036265 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5655209065b0 con 0x5655208fe310 2024-03-28T01:04:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.519+0000 7f0a4dd9b640 1 --2- 172.21.15.169:0/837036265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5655208fe310 0x5655209080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.519+0000 7f0a57a7c740 10 client.0 _ll_register_callbacks cb 0x56552081e720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:04:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.520+0000 7f0a4ed9d640 1 --2- 172.21.15.169:0/837036265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x565520903480 0x56552090f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.520+0000 7f0a4ed9d640 1 -- 172.21.15.169:0/837036265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5655208fe310 msgr2=0x5655209080d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.521+0000 7f0a4ed9d640 1 --2- 172.21.15.169:0/837036265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5655208fe310 0x5655209080d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.521+0000 7f0a4ed9d640 1 -- 172.21.15.169:0/837036265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5655208fd920 msgr2=0x565520907b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.521+0000 7f0a4ed9d640 1 --2- 172.21.15.169:0/837036265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5655208fd920 0x565520907b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.521+0000 7f0a4ed9d640 1 -- 172.21.15.169:0/837036265 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565520643620 con 0x565520903480 2024-03-28T01:04:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.521+0000 7f0a4ed9d640 1 --2- 172.21.15.169:0/837036265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x565520903480 0x56552090f110 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f0a40009270 tx=0x7f0a400127c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74fa5c35ed8fc84 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.521+0000 7f0a35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:04:05.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.522+0000 7f0a35ffb640 1 -- 172.21.15.169:0/837036265 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a40028020 con 0x565520903480 2024-03-28T01:04:05.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.522+0000 7f0a35ffb640 1 -- 172.21.15.169:0/837036265 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f0a40005320 con 0x565520903480 2024-03-28T01:04:05.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.522+0000 7f0a4e59c640 1 --2- 172.21.15.169:0/837036265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5655208fd920 0x565520907b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:04:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.522+0000 7f0a4dd9b640 1 --2- 172.21.15.169:0/837036265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5655208fe310 0x5655209080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:04:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.523+0000 7f0a57a7c740 1 -- 172.21.15.169:0/837036265 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0a400129f0 con 0x565520903480 2024-03-28T01:04:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.523+0000 7f0a35ffb640 1 -- 172.21.15.169:0/837036265 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a40033050 con 0x565520903480 2024-03-28T01:04:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.524+0000 7f0a57a7c740 10 client.16929 fetch_fsmap learned FSMap version 25 2024-03-28T01:04:05.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.524+0000 7f0a57a7c740 1 -- 172.21.15.169:0/837036265 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x565520912270 con 0x565520903480 2024-03-28T01:04:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.526+0000 7f0a35ffb640 1 -- 172.21.15.169:0/837036265 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a40009b70 con 0x565520903480 2024-03-28T01:04:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.526+0000 7f0a35ffb640 1 -- 172.21.15.169:0/837036265 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f0a4000f040 con 0x565520903480 2024-03-28T01:04:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.526+0000 7f0a57a7c740 10 client.16929 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:04:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.527+0000 7f0a57a7c740 10 client.16929 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:04:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.527+0000 7f0a57a7c740 20 client.16929 populate_metadata read hostname 'smithi169' 2024-03-28T01:04:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.527+0000 7f0a57a7c740 1 --2- 172.21.15.169:0/837036265 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x565520913670 0x565520915a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.527+0000 7f0a57a7c740 4 client.16929 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:04:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.527+0000 7f0a57a7c740 1 -- 172.21.15.169:0/837036265 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0a400129f0 con 0x565520913670 2024-03-28T01:04:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.530+0000 7f0a4e59c640 1 --2- 172.21.15.169:0/837036265 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x565520913670 0x565520915a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.532+0000 7f0a4e59c640 1 --2- 172.21.15.169:0/837036265 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x565520913670 0x565520915a60 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8f12ed200b05d4f server_cookie=c375cdb7c855b686 in_seq=0 out_seq=0 2024-03-28T01:04:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.533+0000 7f0a35ffb640 10 client.16929 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:05.538 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.535+0000 7f0a35ffb640 1 -- 172.21.15.169:0/837036265 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7f0a400129f0 con 0x565520913670 2024-03-28T01:04:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.536+0000 7f0a35ffb640 10 client.16929 handle_command_reply: tid=0 2024-03-28T01:04:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.536+0000 7f0a57a7c740 1 client.16929 shutdown 2024-03-28T01:04:05.541 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.538+0000 7f0a57a7c740 1 -- 172.21.15.169:0/837036265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x565520903480 msgr2=0x56552090f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.538+0000 7f0a57a7c740 1 --2- 172.21.15.169:0/837036265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x565520903480 0x56552090f110 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f0a40009270 tx=0x7f0a400127c0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.539+0000 7f0a57a7c740 1 -- 172.21.15.169:0/837036265 shutdown_connections 2024-03-28T01:04:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.539+0000 7f0a57a7c740 1 --2- 172.21.15.169:0/837036265 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x565520913670 0x565520915a60 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.540+0000 7f0a57a7c740 1 --2- 172.21.15.169:0/837036265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x565520903480 0x56552090f110 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.540+0000 7f0a57a7c740 1 --2- 172.21.15.169:0/837036265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5655208fe310 0x5655209080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.540+0000 7f0a57a7c740 1 --2- 172.21.15.169:0/837036265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5655208fd920 0x565520907b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.540+0000 7f0a57a7c740 1 -- 172.21.15.169:0/837036265 >> 172.21.15.169:0/837036265 conn(0x5655208f8e90 msgr2=0x5655208f8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:05.546 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.541+0000 7f0a35ffb640 0 client.16929 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:05.546 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.543+0000 7f0a57a7c740 1 -- 172.21.15.169:0/837036265 shutdown_connections 2024-03-28T01:04:05.546 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.543+0000 7f0a57a7c740 1 -- 172.21.15.169:0/837036265 wait complete. 2024-03-28T01:04:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.544+0000 7f0a57a7c740 20 client.16929 trim_cache size 0 max 16384 2024-03-28T01:04:05.581 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.578+0000 7f0a57a7c740 1 Processor -- start 2024-03-28T01:04:05.581 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.578+0000 7f0a57a7c740 1 -- start start 2024-03-28T01:04:05.582 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.578+0000 7f0a57a7c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5655208fd920 0x5655208ff530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:05.582 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.578+0000 7f0a57a7c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5655208fe310 0x5655208ffa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:05.582 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.578+0000 7f0a57a7c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565520903480 0x56552090f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:05.582 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.578+0000 7f0a57a7c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a300120d0 con 0x5655208fd920 2024-03-28T01:04:05.582 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.578+0000 7f0a57a7c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3000fbd0 con 0x565520903480 2024-03-28T01:04:05.582 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.578+0000 7f0a57a7c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3000dc70 con 0x5655208fe310 2024-03-28T01:04:05.582 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.579+0000 7f0a4e59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5655208fd920 0x5655208ff530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:05.582 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.579+0000 7f0a4e59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5655208fd920 0x5655208ff530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:55560/0 (socket says 172.21.15.169:55560) 2024-03-28T01:04:05.582 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.579+0000 7f0a4e59c640 1 -- 172.21.15.169:0/794939146 learned_addr learned my addr 172.21.15.169:0/794939146 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:05.582 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.579+0000 7f0a4dd9b640 1 --2- 172.21.15.169:0/794939146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5655208fe310 0x5655208ffa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:05.583 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.580+0000 7f0a4ed9d640 1 --2- 172.21.15.169:0/794939146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565520903480 0x56552090f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:05.583 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.580+0000 7f0a4dd9b640 1 -- 172.21.15.169:0/794939146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565520903480 msgr2=0x56552090f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:05.583 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.580+0000 7f0a4dd9b640 1 --2- 172.21.15.169:0/794939146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565520903480 0x56552090f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.583 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.580+0000 7f0a4dd9b640 1 -- 172.21.15.169:0/794939146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5655208fd920 msgr2=0x5655208ff530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:05.583 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.580+0000 7f0a4dd9b640 1 --2- 172.21.15.169:0/794939146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5655208fd920 0x5655208ff530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.583 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.580+0000 7f0a4dd9b640 1 -- 172.21.15.169:0/794939146 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5655208fe970 con 0x5655208fe310 2024-03-28T01:04:05.584 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.582+0000 7f0a4dd9b640 1 --2- 172.21.15.169:0/794939146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5655208fe310 0x5655208ffa70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x5655208fed80 tx=0x7f0a3802e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8aa612560f65a89f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:05.587 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.584+0000 7f0a37fff640 1 -- 172.21.15.169:0/794939146 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a38053070 con 0x5655208fe310 2024-03-28T01:04:05.587 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.584+0000 7f0a37fff640 1 -- 172.21.15.169:0/794939146 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a3802ca30 con 0x5655208fe310 2024-03-28T01:04:05.587 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.584+0000 7f0a37fff640 1 -- 172.21.15.169:0/794939146 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a38039070 con 0x5655208fe310 2024-03-28T01:04:05.588 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.585+0000 7f0a57a7c740 1 -- 172.21.15.169:0/794939146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5655208fe310 msgr2=0x5655208ffa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:05.588 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.585+0000 7f0a57a7c740 1 --2- 172.21.15.169:0/794939146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5655208fe310 0x5655208ffa70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x5655208fed80 tx=0x7f0a3802e040 comp rx=0 tx=0).stop 2024-03-28T01:04:05.592 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.589+0000 7f0a57a7c740 1 -- 172.21.15.169:0/794939146 shutdown_connections 2024-03-28T01:04:05.592 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.589+0000 7f0a57a7c740 1 --2- 172.21.15.169:0/794939146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565520903480 0x56552090f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.592 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.589+0000 7f0a57a7c740 1 --2- 172.21.15.169:0/794939146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5655208fe310 0x5655208ffa70 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.592 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.589+0000 7f0a57a7c740 1 --2- 172.21.15.169:0/794939146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5655208fd920 0x5655208ff530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.592 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.589+0000 7f0a57a7c740 1 -- 172.21.15.169:0/794939146 >> 172.21.15.169:0/794939146 conn(0x5655208f8e90 msgr2=0x565520907b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:05.593 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.590+0000 7f0a57a7c740 1 -- 172.21.15.169:0/794939146 shutdown_connections 2024-03-28T01:04:05.596 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.593+0000 7f0a57a7c740 1 -- 172.21.15.169:0/794939146 wait complete. 2024-03-28T01:04:05.596 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.593+0000 7f0a57a7c740 1 -- start start 2024-03-28T01:04:05.596 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.593+0000 7f0a57a7c740 1 Processor -- start 2024-03-28T01:04:05.600 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.596+0000 7f0a57a7c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5655208fd920 0x56552090c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:05.600 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.596+0000 7f0a57a7c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5655208fe310 0x56552090cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:05.600 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.596+0000 7f0a57a7c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565520903480 0x56552090de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:05.600 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.596+0000 7f0a57a7c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3000e270 con 0x5655208fe310 2024-03-28T01:04:05.600 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.596+0000 7f0a57a7c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3000f570 con 0x5655208fd920 2024-03-28T01:04:05.600 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.596+0000 7f0a57a7c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3000e8b0 con 0x565520903480 2024-03-28T01:04:05.600 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.596+0000 7f0a57a7c740 10 client.0 _ll_register_callbacks cb 0x7f0a40012dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:04:05.600 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.596+0000 7f0a4ed9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565520903480 0x56552090de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:05.600 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.596+0000 7f0a4ed9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565520903480 0x56552090de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:39330/0 (socket says 172.21.15.169:39330) 2024-03-28T01:04:05.600 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.596+0000 7f0a4ed9d640 1 -- 172.21.15.169:0/487727930 learned_addr learned my addr 172.21.15.169:0/487727930 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:05.600 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.596+0000 7f0a4e59c640 1 --2- 172.21.15.169:0/487727930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5655208fd920 0x56552090c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:05.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.597+0000 7f0a4dd9b640 1 --2- 172.21.15.169:0/487727930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5655208fe310 0x56552090cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:05.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.598+0000 7f0a4dd9b640 1 -- 172.21.15.169:0/487727930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565520903480 msgr2=0x56552090de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:05.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.598+0000 7f0a4dd9b640 1 --2- 172.21.15.169:0/487727930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565520903480 0x56552090de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.598+0000 7f0a4dd9b640 1 -- 172.21.15.169:0/487727930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5655208fd920 msgr2=0x56552090c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:05.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.598+0000 7f0a4dd9b640 1 --2- 172.21.15.169:0/487727930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5655208fd920 0x56552090c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:05.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.598+0000 7f0a4dd9b640 1 -- 172.21.15.169:0/487727930 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56552090e3f0 con 0x5655208fe310 2024-03-28T01:04:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.601+0000 7f0a4dd9b640 1 --2- 172.21.15.169:0/487727930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5655208fe310 0x56552090cdf0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f0a38035fd0 tx=0x7f0a3802c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3d34dad5bd7a66c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.602+0000 7f0a35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:04:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.602+0000 7f0a35ffb640 1 -- 172.21.15.169:0/487727930 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a38053040 con 0x5655208fe310 2024-03-28T01:04:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.603+0000 7f0a35ffb640 1 -- 172.21.15.169:0/487727930 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f0a38032850 con 0x5655208fe310 2024-03-28T01:04:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.604+0000 7f0a57a7c740 1 -- 172.21.15.169:0/487727930 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0a3802ca30 con 0x5655208fe310 2024-03-28T01:04:05.608 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.605+0000 7f0a35ffb640 1 -- 172.21.15.169:0/487727930 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a38032a50 con 0x5655208fe310 2024-03-28T01:04:05.608 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.605+0000 7f0a57a7c740 10 client.16935 fetch_fsmap learned FSMap version 25 2024-03-28T01:04:05.608 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.605+0000 7f0a57a7c740 1 -- 172.21.15.169:0/487727930 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56552096eef0 con 0x5655208fe310 2024-03-28T01:04:05.609 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.606+0000 7f0a35ffb640 1 -- 172.21.15.169:0/487727930 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a3802ca30 con 0x5655208fe310 2024-03-28T01:04:05.609 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.606+0000 7f0a35ffb640 1 -- 172.21.15.169:0/487727930 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f0a38057b80 con 0x5655208fe310 2024-03-28T01:04:05.610 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.607+0000 7f0a57a7c740 10 client.16935 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:04:05.610 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.607+0000 7f0a57a7c740 10 client.16935 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:04:05.610 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.607+0000 7f0a57a7c740 20 client.16935 populate_metadata read hostname 'smithi169' 2024-03-28T01:04:05.610 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.607+0000 7f0a57a7c740 1 --2- 172.21.15.169:0/487727930 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56552096f7b0 0x565520971bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:05.610 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.607+0000 7f0a57a7c740 4 client.16935 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:04:05.610 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.607+0000 7f0a57a7c740 1 -- 172.21.15.169:0/487727930 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x565520870050 con 0x56552096f7b0 2024-03-28T01:04:05.614 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.611+0000 7f0a4e59c640 1 --2- 172.21.15.169:0/487727930 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56552096f7b0 0x565520971bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:05.615 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.612+0000 7f0a4e59c640 1 --2- 172.21.15.169:0/487727930 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56552096f7b0 0x565520971bd0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e75932f304ded2af server_cookie=6a865f718db060d2 in_seq=0 out_seq=0 2024-03-28T01:04:05.615 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:05.613+0000 7f0a35ffb640 10 client.16935 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:05.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:05 smithi177 ceph-mon[36206]: pgmap v578: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s rd, 395 MiB/s wr, 4.03k op/s 2024-03-28T01:04:06.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:05 smithi169 ceph-mon[32274]: pgmap v578: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s rd, 395 MiB/s wr, 4.03k op/s 2024-03-28T01:04:06.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:05 smithi174 ceph-mon[36060]: pgmap v578: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s rd, 395 MiB/s wr, 4.03k op/s 2024-03-28T01:04:07.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:06 smithi169 ceph-mon[32274]: pgmap v579: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s rd, 365 MiB/s wr, 3.87k op/s 2024-03-28T01:04:07.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:06 smithi174 ceph-mon[36060]: pgmap v579: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s rd, 365 MiB/s wr, 3.87k op/s 2024-03-28T01:04:07.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:06 smithi177 ceph-mon[36206]: pgmap v579: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s rd, 365 MiB/s wr, 3.87k op/s 2024-03-28T01:04:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.882+0000 7f0a35ffb640 1 -- 172.21.15.169:0/487727930 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x565520870050 con 0x56552096f7b0 2024-03-28T01:04:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.882+0000 7f0a35ffb640 10 client.16935 handle_command_reply: tid=0 2024-03-28T01:04:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.882+0000 7f0a57a7c740 1 client.16935 shutdown 2024-03-28T01:04:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.883+0000 7f0a57a7c740 1 -- 172.21.15.169:0/487727930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5655208fe310 msgr2=0x56552090cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.883+0000 7f0a57a7c740 1 --2- 172.21.15.169:0/487727930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5655208fe310 0x56552090cdf0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f0a38035fd0 tx=0x7f0a3802c9c0 comp rx=0 tx=0).stop 2024-03-28T01:04:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.883+0000 7f0a57a7c740 1 -- 172.21.15.169:0/487727930 shutdown_connections 2024-03-28T01:04:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.883+0000 7f0a57a7c740 1 --2- 172.21.15.169:0/487727930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565520903480 0x56552090de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.883+0000 7f0a57a7c740 1 --2- 172.21.15.169:0/487727930 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56552096f7b0 0x565520971bd0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.883+0000 7f0a57a7c740 1 --2- 172.21.15.169:0/487727930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5655208fe310 0x56552090cdf0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.883+0000 7f0a57a7c740 1 --2- 172.21.15.169:0/487727930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5655208fd920 0x56552090c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.883+0000 7f0a57a7c740 1 -- 172.21.15.169:0/487727930 >> 172.21.15.169:0/487727930 conn(0x5655208f8e90 msgr2=0x565520907b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.883+0000 7f0a35ffb640 0 client.16935 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.883+0000 7f0a57a7c740 1 -- 172.21.15.169:0/487727930 shutdown_connections 2024-03-28T01:04:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.883+0000 7f0a57a7c740 1 -- 172.21.15.169:0/487727930 wait complete. 2024-03-28T01:04:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.883+0000 7f0a57a7c740 20 client.16935 trim_cache size 0 max 16384 2024-03-28T01:04:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:04:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:04:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:04:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:04:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.883+0000 7f0a54e85640 1 -- 172.21.15.169:0/1282825386 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0a3c0a28e0 msgr2=0x7f0a3c0a4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.883+0000 7f0a54e85640 1 --2- 172.21.15.169:0/1282825386 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0a3c0a28e0 0x7f0a3c0a4da0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x5655208fe710 tx=0x7f0a380349b0 comp rx=0 tx=0).stop 2024-03-28T01:04:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.883+0000 7f0a54e85640 1 -- 172.21.15.169:0/1282825386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a501086a0 msgr2=0x7f0a50119810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.883+0000 7f0a54e85640 1 --2- 172.21.15.169:0/1282825386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a501086a0 0x7f0a50119810 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0a4800e790 tx=0x7f0a4800ec50 comp rx=0 tx=0).stop 2024-03-28T01:04:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.884+0000 7f0a54e85640 1 -- 172.21.15.169:0/1282825386 shutdown_connections 2024-03-28T01:04:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.884+0000 7f0a54e85640 1 --2- 172.21.15.169:0/1282825386 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0a3c0a28e0 0x7f0a3c0a4da0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.884+0000 7f0a54e85640 1 --2- 172.21.15.169:0/1282825386 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0a5010d9c0 0x7f0a50117e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.884+0000 7f0a54e85640 1 --2- 172.21.15.169:0/1282825386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a5010b010 0x7f0a50119d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.884+0000 7f0a54e85640 1 --2- 172.21.15.169:0/1282825386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a501086a0 0x7f0a50119810 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.884+0000 7f0a54e85640 1 -- 172.21.15.169:0/1282825386 >> 172.21.15.169:0/1282825386 conn(0x7f0a5006e6e0 msgr2=0x7f0a5010b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:08.890 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.884+0000 7f0a54e85640 1 -- 172.21.15.169:0/1282825386 shutdown_connections 2024-03-28T01:04:08.890 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:08.884+0000 7f0a54e85640 1 -- 172.21.15.169:0/1282825386 wait complete. 2024-03-28T01:04:09.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:09 smithi169 ceph-mon[32274]: pgmap v580: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 146 MiB/s rd, 367 MiB/s wr, 4.01k op/s 2024-03-28T01:04:09.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:09 smithi174 ceph-mon[36060]: pgmap v580: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 146 MiB/s rd, 367 MiB/s wr, 4.01k op/s 2024-03-28T01:04:09.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:09 smithi177 ceph-mon[36206]: pgmap v580: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 146 MiB/s rd, 367 MiB/s wr, 4.01k op/s 2024-03-28T01:04:11.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:11 smithi169 ceph-mon[32274]: pgmap v581: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 161 MiB/s rd, 350 MiB/s wr, 3.60k op/s 2024-03-28T01:04:11.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:11 smithi174 ceph-mon[36060]: pgmap v581: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 161 MiB/s rd, 350 MiB/s wr, 3.60k op/s 2024-03-28T01:04:11.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:11 smithi177 ceph-mon[36206]: pgmap v581: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 161 MiB/s rd, 350 MiB/s wr, 3.60k op/s 2024-03-28T01:04:13.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:13 smithi169 ceph-mon[32274]: pgmap v582: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 180 MiB/s rd, 371 MiB/s wr, 3.69k op/s 2024-03-28T01:04:13.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:13 smithi174 ceph-mon[36060]: pgmap v582: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 180 MiB/s rd, 371 MiB/s wr, 3.69k op/s 2024-03-28T01:04:13.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:13 smithi177 ceph-mon[36206]: pgmap v582: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 180 MiB/s rd, 371 MiB/s wr, 3.69k op/s 2024-03-28T01:04:15.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:15 smithi174 ceph-mon[36060]: pgmap v583: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 175 MiB/s rd, 367 MiB/s wr, 3.67k op/s 2024-03-28T01:04:15.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:15 smithi177 ceph-mon[36206]: pgmap v583: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 175 MiB/s rd, 367 MiB/s wr, 3.67k op/s 2024-03-28T01:04:16.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:15 smithi169 ceph-mon[32274]: pgmap v583: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 175 MiB/s rd, 367 MiB/s wr, 3.67k op/s 2024-03-28T01:04:17.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:17 smithi174 ceph-mon[36060]: pgmap v584: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 188 MiB/s rd, 365 MiB/s wr, 3.96k op/s 2024-03-28T01:04:17.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:17 smithi177 ceph-mon[36206]: pgmap v584: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 188 MiB/s rd, 365 MiB/s wr, 3.96k op/s 2024-03-28T01:04:18.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:17 smithi169 ceph-mon[32274]: pgmap v584: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 188 MiB/s rd, 365 MiB/s wr, 3.96k op/s 2024-03-28T01:04:19.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:19 smithi174 ceph-mon[36060]: pgmap v585: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 191 MiB/s rd, 361 MiB/s wr, 4.12k op/s 2024-03-28T01:04:19.919 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:19 smithi177 ceph-mon[36206]: pgmap v585: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 191 MiB/s rd, 361 MiB/s wr, 4.12k op/s 2024-03-28T01:04:20.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:19 smithi169 ceph-mon[32274]: pgmap v585: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 191 MiB/s rd, 361 MiB/s wr, 4.12k op/s 2024-03-28T01:04:21.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:21 smithi174 ceph-mon[36060]: pgmap v586: 129 pgs: 129 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 178 MiB/s rd, 355 MiB/s wr, 4.37k op/s 2024-03-28T01:04:21.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:21 smithi177 ceph-mon[36206]: pgmap v586: 129 pgs: 129 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 178 MiB/s rd, 355 MiB/s wr, 4.37k op/s 2024-03-28T01:04:22.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:21 smithi169 ceph-mon[32274]: pgmap v586: 129 pgs: 129 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 178 MiB/s rd, 355 MiB/s wr, 4.37k op/s 2024-03-28T01:04:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.461+0000 7fb219fa9640 1 -- 172.21.15.169:0/847591765 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb214101bd0 msgr2=0x7fb214101ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.461+0000 7fb219fa9640 1 --2- 172.21.15.169:0/847591765 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb214101bd0 0x7fb214101ff0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb204009900 tx=0x7fb20402bc30 comp rx=0 tx=0).stop 2024-03-28T01:04:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.461+0000 7fb219fa9640 1 -- 172.21.15.169:0/847591765 shutdown_connections 2024-03-28T01:04:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.461+0000 7fb219fa9640 1 --2- 172.21.15.169:0/847591765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2141011e0 0x7fb214108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.461+0000 7fb219fa9640 1 --2- 172.21.15.169:0/847591765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb214100820 0x7fb214100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.461+0000 7fb219fa9640 1 --2- 172.21.15.169:0/847591765 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb214101bd0 0x7fb214101ff0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.461+0000 7fb219fa9640 1 -- 172.21.15.169:0/847591765 >> 172.21.15.169:0/847591765 conn(0x7fb2140fb2f0 msgr2=0x7fb2140fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:22.467 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.461+0000 7fb219fa9640 1 -- 172.21.15.169:0/847591765 shutdown_connections 2024-03-28T01:04:22.467 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.461+0000 7fb219fa9640 1 -- 172.21.15.169:0/847591765 wait complete. 2024-03-28T01:04:22.467 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.461+0000 7fb219fa9640 1 Processor -- start 2024-03-28T01:04:22.467 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.461+0000 7fb219fa9640 1 -- start start 2024-03-28T01:04:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.461+0000 7fb219fa9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb214100820 0x7fb214196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.461+0000 7fb219fa9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2141011e0 0x7fb214196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.461+0000 7fb219fa9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb214101bd0 0x7fb21419de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.462+0000 7fb219fa9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21410c270 con 0x7fb214101bd0 2024-03-28T01:04:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.462+0000 7fb219fa9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21410c0f0 con 0x7fb2141011e0 2024-03-28T01:04:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.462+0000 7fb219fa9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21410c3f0 con 0x7fb214100820 2024-03-28T01:04:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.462+0000 7fb213fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb214101bd0 0x7fb21419de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.462+0000 7fb212ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2141011e0 0x7fb214196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.462+0000 7fb212ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2141011e0 0x7fb214196da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57440/0 (socket says 172.21.15.169:57440) 2024-03-28T01:04:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.462+0000 7fb213fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb214101bd0 0x7fb21419de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36084/0 (socket says 172.21.15.169:36084) 2024-03-28T01:04:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.462+0000 7fb212ffd640 1 -- 172.21.15.169:0/1942732422 learned_addr learned my addr 172.21.15.169:0/1942732422 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.462+0000 7fb2137fe640 1 --2- 172.21.15.169:0/1942732422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb214100820 0x7fb214196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.462+0000 7fb212ffd640 1 -- 172.21.15.169:0/1942732422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb214100820 msgr2=0x7fb214196860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.462+0000 7fb212ffd640 1 --2- 172.21.15.169:0/1942732422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb214100820 0x7fb214196860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.462+0000 7fb212ffd640 1 -- 172.21.15.169:0/1942732422 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb214101bd0 msgr2=0x7fb21419de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.462+0000 7fb212ffd640 1 --2- 172.21.15.169:0/1942732422 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb214101bd0 0x7fb21419de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.462+0000 7fb212ffd640 1 -- 172.21.15.169:0/1942732422 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21419e480 con 0x7fb2141011e0 2024-03-28T01:04:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.462+0000 7fb212ffd640 1 --2- 172.21.15.169:0/1942732422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2141011e0 0x7fb214196da0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb20800e9f0 tx=0x7fb20800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.462+0000 7fb210ff9640 1 -- 172.21.15.169:0/1942732422 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20801b070 con 0x7fb2141011e0 2024-03-28T01:04:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.462+0000 7fb219fa9640 1 -- 172.21.15.169:0/1942732422 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2140ff770 con 0x7fb2141011e0 2024-03-28T01:04:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.462+0000 7fb210ff9640 1 -- 172.21.15.169:0/1942732422 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb20800db90 con 0x7fb2141011e0 2024-03-28T01:04:22.470 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.462+0000 7fb210ff9640 1 -- 172.21.15.169:0/1942732422 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb208015610 con 0x7fb2141011e0 2024-03-28T01:04:22.470 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.462+0000 7fb219fa9640 1 -- 172.21.15.169:0/1942732422 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2140ffcc0 con 0x7fb2141011e0 2024-03-28T01:04:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.464+0000 7fb210ff9640 1 -- 172.21.15.169:0/1942732422 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb20801e070 con 0x7fb2141011e0 2024-03-28T01:04:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.465+0000 7fb210ff9640 1 --2- 172.21.15.169:0/1942732422 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1fc0a28a0 0x7fb1fc0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.465+0000 7fb2137fe640 1 --2- 172.21.15.169:0/1942732422 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1fc0a28a0 0x7fb1fc0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.465+0000 7fb210ff9640 1 -- 172.21.15.169:0/1942732422 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb2080ce620 con 0x7fb2141011e0 2024-03-28T01:04:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.465+0000 7fb21cb8f740 1 Processor -- start 2024-03-28T01:04:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.465+0000 7fb21cb8f740 1 -- start start 2024-03-28T01:04:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.465+0000 7fb21cb8f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454d920 0x557bf454dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb21cb8f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf454e310 0x557bf4552e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb21cb8f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf4553480 0x557bf4555910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb2137fe640 1 --2- 172.21.15.169:0/1942732422 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1fc0a28a0 0x7fb1fc0a4d60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb204000c00 tx=0x7fb204005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb21cb8f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bf44b2b10 con 0x557bf454e310 2024-03-28T01:04:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb21cb8f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bf445b900 con 0x557bf454d920 2024-03-28T01:04:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb21cb8f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bf44aebb0 con 0x557bf4553480 2024-03-28T01:04:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb212ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf454e310 0x557bf4552e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb212ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf454e310 0x557bf4552e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36092/0 (socket says 172.21.15.169:36092) 2024-03-28T01:04:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb212ffd640 1 -- 172.21.15.169:0/1325068884 learned_addr learned my addr 172.21.15.169:0/1325068884 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb2137fe640 1 --2- 172.21.15.169:0/1325068884 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454d920 0x557bf454dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb212ffd640 1 -- 172.21.15.169:0/1325068884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf4553480 msgr2=0x557bf4555910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb213fff640 1 --2- 172.21.15.169:0/1325068884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf4553480 0x557bf4555910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb212ffd640 1 --2- 172.21.15.169:0/1325068884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf4553480 0x557bf4555910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb212ffd640 1 -- 172.21.15.169:0/1325068884 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454d920 msgr2=0x557bf454dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb212ffd640 1 --2- 172.21.15.169:0/1325068884 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454d920 0x557bf454dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb212ffd640 1 -- 172.21.15.169:0/1325068884 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557bf41c7210 con 0x557bf454e310 2024-03-28T01:04:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb213fff640 1 --2- 172.21.15.169:0/1325068884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf4553480 0x557bf4555910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:04:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb2137fe640 1 --2- 172.21.15.169:0/1325068884 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454d920 0x557bf454dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:04:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb212ffd640 1 --2- 172.21.15.169:0/1325068884 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf454e310 0x557bf4552e10 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fb208097130 tx=0x7fb20809ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9617ecb2a10c2899 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb1f9ffb640 1 -- 172.21.15.169:0/1325068884 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb208098fd0 con 0x557bf454e310 2024-03-28T01:04:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb1f9ffb640 1 -- 172.21.15.169:0/1325068884 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb20800dd40 con 0x557bf454e310 2024-03-28T01:04:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.466+0000 7fb1f9ffb640 1 -- 172.21.15.169:0/1325068884 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb208099b90 con 0x557bf454e310 2024-03-28T01:04:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.467+0000 7fb21cb8f740 1 -- 172.21.15.169:0/1325068884 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf454e310 msgr2=0x557bf4552e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.467+0000 7fb21cb8f740 1 --2- 172.21.15.169:0/1325068884 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf454e310 0x557bf4552e10 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fb208097130 tx=0x7fb20809ed70 comp rx=0 tx=0).stop 2024-03-28T01:04:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.467+0000 7fb21cb8f740 1 -- 172.21.15.169:0/1325068884 shutdown_connections 2024-03-28T01:04:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.467+0000 7fb21cb8f740 1 --2- 172.21.15.169:0/1325068884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf4553480 0x557bf4555910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.467+0000 7fb21cb8f740 1 --2- 172.21.15.169:0/1325068884 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf454e310 0x557bf4552e10 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.467+0000 7fb21cb8f740 1 --2- 172.21.15.169:0/1325068884 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454d920 0x557bf454dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.467+0000 7fb21cb8f740 1 -- 172.21.15.169:0/1325068884 >> 172.21.15.169:0/1325068884 conn(0x557bf4548e90 msgr2=0x557bf4549290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.467+0000 7fb21cb8f740 1 -- 172.21.15.169:0/1325068884 shutdown_connections 2024-03-28T01:04:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.467+0000 7fb21cb8f740 1 -- 172.21.15.169:0/1325068884 wait complete. 2024-03-28T01:04:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.467+0000 7fb21cb8f740 1 -- start start 2024-03-28T01:04:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.468+0000 7fb21cb8f740 1 Processor -- start 2024-03-28T01:04:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.468+0000 7fb21cb8f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf454d920 0x557bf4557b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.468+0000 7fb21cb8f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454e310 0x557bf45580c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.468+0000 7fb21cb8f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf4553480 0x557bf455f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.468+0000 7fb21cb8f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bf4556370 con 0x557bf4553480 2024-03-28T01:04:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.468+0000 7fb21cb8f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bf446ed50 con 0x557bf454e310 2024-03-28T01:04:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.468+0000 7fb21cb8f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bf45564f0 con 0x557bf454d920 2024-03-28T01:04:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.468+0000 7fb21cb8f740 10 client.0 _ll_register_callbacks cb 0x557bf446e720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:04:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.468+0000 7fb213fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf4553480 0x557bf455f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.468+0000 7fb212ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454e310 0x557bf45580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.468+0000 7fb2137fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf454d920 0x557bf4557b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.468+0000 7fb213fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf4553480 0x557bf455f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36094/0 (socket says 172.21.15.169:36094) 2024-03-28T01:04:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.468+0000 7fb212ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454e310 0x557bf45580c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57460/0 (socket says 172.21.15.169:57460) 2024-03-28T01:04:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.468+0000 7fb213fff640 1 -- 172.21.15.169:0/3728452214 learned_addr learned my addr 172.21.15.169:0/3728452214 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.468+0000 7fb213fff640 1 -- 172.21.15.169:0/3728452214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf454d920 msgr2=0x557bf4557b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.468+0000 7fb213fff640 1 --2- 172.21.15.169:0/3728452214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf454d920 0x557bf4557b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.468+0000 7fb213fff640 1 -- 172.21.15.169:0/3728452214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454e310 msgr2=0x557bf45580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.468+0000 7fb213fff640 1 --2- 172.21.15.169:0/3728452214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454e310 0x557bf45580c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.468+0000 7fb213fff640 1 -- 172.21.15.169:0/3728452214 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557bf4293620 con 0x557bf4553480 2024-03-28T01:04:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.468+0000 7fb212ffd640 1 --2- 172.21.15.169:0/3728452214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454e310 0x557bf45580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:04:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.469+0000 7fb213fff640 1 --2- 172.21.15.169:0/3728452214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf4553480 0x557bf455f100 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb20c0042f0 tx=0x7fb20c003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0e2ab5d7f3913bf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.469+0000 7fb1f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:04:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.469+0000 7fb1f2ffd640 1 -- 172.21.15.169:0/3728452214 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20c009e30 con 0x557bf4553480 2024-03-28T01:04:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.469+0000 7fb1f2ffd640 1 -- 172.21.15.169:0/3728452214 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb20c01a410 con 0x557bf4553480 2024-03-28T01:04:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.469+0000 7fb21cb8f740 1 -- 172.21.15.169:0/3728452214 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb20800dd40 con 0x557bf4553480 2024-03-28T01:04:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.469+0000 7fb1f2ffd640 1 -- 172.21.15.169:0/3728452214 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb20c028050 con 0x557bf4553480 2024-03-28T01:04:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.469+0000 7fb21cb8f740 10 client.16947 fetch_fsmap learned FSMap version 25 2024-03-28T01:04:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.469+0000 7fb21cb8f740 1 -- 172.21.15.169:0/3728452214 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557bf45624b0 con 0x557bf4553480 2024-03-28T01:04:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.469+0000 7fb1f2ffd640 1 -- 172.21.15.169:0/3728452214 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb20800dd40 con 0x557bf4553480 2024-03-28T01:04:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.469+0000 7fb1f2ffd640 1 -- 172.21.15.169:0/3728452214 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb20c0235e0 con 0x557bf4553480 2024-03-28T01:04:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.469+0000 7fb21cb8f740 10 client.16947 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:04:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.469+0000 7fb21cb8f740 10 client.16947 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:04:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.469+0000 7fb21cb8f740 20 client.16947 populate_metadata read hostname 'smithi169' 2024-03-28T01:04:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.469+0000 7fb21cb8f740 1 --2- 172.21.15.169:0/3728452214 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557bf4563bd0 0x557bf4566000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.469+0000 7fb21cb8f740 4 client.16947 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:04:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.469+0000 7fb21cb8f740 1 -- 172.21.15.169:0/3728452214 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557bf44c0050 con 0x557bf4563bd0 2024-03-28T01:04:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.470+0000 7fb2137fe640 1 --2- 172.21.15.169:0/3728452214 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557bf4563bd0 0x557bf4566000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.470+0000 7fb1f2ffd640 10 client.16947 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.470+0000 7fb2137fe640 1 --2- 172.21.15.169:0/3728452214 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557bf4563bd0 0x557bf4566000 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4e6a57ed15e9a76 server_cookie=60727cde76ff5d58 in_seq=0 out_seq=0 2024-03-28T01:04:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.471+0000 7fb1f2ffd640 1 -- 172.21.15.169:0/3728452214 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557bf44c0050 con 0x557bf4563bd0 2024-03-28T01:04:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.471+0000 7fb1f2ffd640 10 client.16947 handle_command_reply: tid=0 2024-03-28T01:04:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.471+0000 7fb21cb8f740 1 client.16947 shutdown 2024-03-28T01:04:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.471+0000 7fb21cb8f740 1 -- 172.21.15.169:0/3728452214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf4553480 msgr2=0x557bf455f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.471+0000 7fb21cb8f740 1 --2- 172.21.15.169:0/3728452214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf4553480 0x557bf455f100 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb20c0042f0 tx=0x7fb20c003f90 comp rx=0 tx=0).stop 2024-03-28T01:04:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.471+0000 7fb21cb8f740 1 -- 172.21.15.169:0/3728452214 shutdown_connections 2024-03-28T01:04:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.471+0000 7fb21cb8f740 1 --2- 172.21.15.169:0/3728452214 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557bf4563bd0 0x557bf4566000 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.472+0000 7fb21cb8f740 1 --2- 172.21.15.169:0/3728452214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf4553480 0x557bf455f100 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.472+0000 7fb21cb8f740 1 --2- 172.21.15.169:0/3728452214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454e310 0x557bf45580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.472+0000 7fb1f2ffd640 0 client.16947 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.472+0000 7fb21cb8f740 1 --2- 172.21.15.169:0/3728452214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf454d920 0x557bf4557b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.472+0000 7fb21cb8f740 1 -- 172.21.15.169:0/3728452214 >> 172.21.15.169:0/3728452214 conn(0x557bf4548e90 msgr2=0x557bf4548020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.472+0000 7fb21cb8f740 1 -- 172.21.15.169:0/3728452214 shutdown_connections 2024-03-28T01:04:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.472+0000 7fb21cb8f740 1 -- 172.21.15.169:0/3728452214 wait complete. 2024-03-28T01:04:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.472+0000 7fb21cb8f740 20 client.16947 trim_cache size 0 max 16384 2024-03-28T01:04:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.489+0000 7fb21cb8f740 1 Processor -- start 2024-03-28T01:04:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.489+0000 7fb21cb8f740 1 -- start start 2024-03-28T01:04:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.489+0000 7fb21cb8f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454d920 0x557bf4555870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.489+0000 7fb21cb8f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf454e310 0x557bf455f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.489+0000 7fb21cb8f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf4553480 0x557bf4557b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.489+0000 7fb21cb8f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2141aa500 con 0x557bf454e310 2024-03-28T01:04:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.489+0000 7fb21cb8f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2141a8000 con 0x557bf454d920 2024-03-28T01:04:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.489+0000 7fb21cb8f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2141a60a0 con 0x557bf4553480 2024-03-28T01:04:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.489+0000 7fb212ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf454e310 0x557bf455f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.489+0000 7fb212ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf454e310 0x557bf455f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36096/0 (socket says 172.21.15.169:36096) 2024-03-28T01:04:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.489+0000 7fb212ffd640 1 -- 172.21.15.169:0/2440822257 learned_addr learned my addr 172.21.15.169:0/2440822257 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.489+0000 7fb213fff640 1 --2- 172.21.15.169:0/2440822257 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf4553480 0x557bf4557b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.489+0000 7fb2137fe640 1 --2- 172.21.15.169:0/2440822257 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454d920 0x557bf4555870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.489+0000 7fb212ffd640 1 -- 172.21.15.169:0/2440822257 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf4553480 msgr2=0x557bf4557b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.489+0000 7fb212ffd640 1 --2- 172.21.15.169:0/2440822257 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf4553480 0x557bf4557b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.489+0000 7fb212ffd640 1 -- 172.21.15.169:0/2440822257 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454d920 msgr2=0x557bf4555870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.489+0000 7fb212ffd640 1 --2- 172.21.15.169:0/2440822257 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454d920 0x557bf4555870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.489+0000 7fb212ffd640 1 -- 172.21.15.169:0/2440822257 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557bf4563970 con 0x557bf454e310 2024-03-28T01:04:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.489+0000 7fb213fff640 1 --2- 172.21.15.169:0/2440822257 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf4553480 0x557bf4557b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:04:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.490+0000 7fb212ffd640 1 --2- 172.21.15.169:0/2440822257 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf454e310 0x557bf455f0c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb2080a3580 tx=0x7fb208009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37652fa0ca9fe3a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.490+0000 7fb1f97fa640 1 -- 172.21.15.169:0/2440822257 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb208098df0 con 0x557bf454e310 2024-03-28T01:04:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.490+0000 7fb1f97fa640 1 -- 172.21.15.169:0/2440822257 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb208097340 con 0x557bf454e310 2024-03-28T01:04:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.490+0000 7fb1f97fa640 1 -- 172.21.15.169:0/2440822257 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2080b9170 con 0x557bf454e310 2024-03-28T01:04:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.490+0000 7fb21cb8f740 1 -- 172.21.15.169:0/2440822257 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf454e310 msgr2=0x557bf455f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.490+0000 7fb21cb8f740 1 --2- 172.21.15.169:0/2440822257 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf454e310 0x557bf455f0c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb2080a3580 tx=0x7fb208009700 comp rx=0 tx=0).stop 2024-03-28T01:04:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.491+0000 7fb21cb8f740 1 -- 172.21.15.169:0/2440822257 shutdown_connections 2024-03-28T01:04:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.491+0000 7fb21cb8f740 1 --2- 172.21.15.169:0/2440822257 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf4553480 0x557bf4557b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.491+0000 7fb21cb8f740 1 --2- 172.21.15.169:0/2440822257 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf454e310 0x557bf455f0c0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.491+0000 7fb21cb8f740 1 --2- 172.21.15.169:0/2440822257 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454d920 0x557bf4555870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.491+0000 7fb21cb8f740 1 -- 172.21.15.169:0/2440822257 >> 172.21.15.169:0/2440822257 conn(0x557bf4548e90 msgr2=0x557bf4549270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.491+0000 7fb21cb8f740 1 -- 172.21.15.169:0/2440822257 shutdown_connections 2024-03-28T01:04:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.491+0000 7fb21cb8f740 1 -- 172.21.15.169:0/2440822257 wait complete. 2024-03-28T01:04:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.491+0000 7fb21cb8f740 1 -- start start 2024-03-28T01:04:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.491+0000 7fb21cb8f740 1 Processor -- start 2024-03-28T01:04:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.492+0000 7fb21cb8f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf454d920 0x557bf455c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.492+0000 7fb21cb8f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454e310 0x557bf455c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.492+0000 7fb21cb8f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf4553480 0x557bf455d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.492+0000 7fb21cb8f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2141a66a0 con 0x557bf454d920 2024-03-28T01:04:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.492+0000 7fb21cb8f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2141a79a0 con 0x557bf454e310 2024-03-28T01:04:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.492+0000 7fb21cb8f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2141a6ce0 con 0x557bf4553480 2024-03-28T01:04:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.492+0000 7fb2137fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf454d920 0x557bf455c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.492+0000 7fb2137fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf454d920 0x557bf455c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36102/0 (socket says 172.21.15.169:36102) 2024-03-28T01:04:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.492+0000 7fb21cb8f740 10 client.0 _ll_register_callbacks cb 0x7fb20c003e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:04:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.492+0000 7fb2137fe640 1 -- 172.21.15.169:0/3504813445 learned_addr learned my addr 172.21.15.169:0/3504813445 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.492+0000 7fb212ffd640 1 --2- 172.21.15.169:0/3504813445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454e310 0x557bf455c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.492+0000 7fb213fff640 1 --2- 172.21.15.169:0/3504813445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf4553480 0x557bf455d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.492+0000 7fb2137fe640 1 -- 172.21.15.169:0/3504813445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf4553480 msgr2=0x557bf455d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.492+0000 7fb2137fe640 1 --2- 172.21.15.169:0/3504813445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf4553480 0x557bf455d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.492+0000 7fb2137fe640 1 -- 172.21.15.169:0/3504813445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454e310 msgr2=0x557bf455c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.492+0000 7fb2137fe640 1 --2- 172.21.15.169:0/3504813445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454e310 0x557bf455c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.492+0000 7fb2137fe640 1 -- 172.21.15.169:0/3504813445 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557bf455dfa0 con 0x557bf454d920 2024-03-28T01:04:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.492+0000 7fb2137fe640 1 --2- 172.21.15.169:0/3504813445 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf454d920 0x557bf455c460 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb20402c7b0 tx=0x7fb20402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59d1008d69b3f421 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.492+0000 7fb1f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:04:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.493+0000 7fb1f2ffd640 1 -- 172.21.15.169:0/3504813445 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb204005250 con 0x557bf454d920 2024-03-28T01:04:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.493+0000 7fb1f2ffd640 1 -- 172.21.15.169:0/3504813445 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb2040053f0 con 0x557bf454d920 2024-03-28T01:04:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.493+0000 7fb21cb8f740 1 -- 172.21.15.169:0/3504813445 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb208097340 con 0x557bf454d920 2024-03-28T01:04:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.493+0000 7fb1f2ffd640 1 -- 172.21.15.169:0/3504813445 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb204061050 con 0x557bf454d920 2024-03-28T01:04:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.493+0000 7fb21cb8f740 10 client.16953 fetch_fsmap learned FSMap version 25 2024-03-28T01:04:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.493+0000 7fb21cb8f740 1 -- 172.21.15.169:0/3504813445 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557bf455f320 con 0x557bf454d920 2024-03-28T01:04:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.494+0000 7fb1f2ffd640 1 -- 172.21.15.169:0/3504813445 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb208097340 con 0x557bf454d920 2024-03-28T01:04:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.494+0000 7fb1f2ffd640 1 -- 172.21.15.169:0/3504813445 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb2040037f0 con 0x557bf454d920 2024-03-28T01:04:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.494+0000 7fb21cb8f740 10 client.16953 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:04:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.494+0000 7fb21cb8f740 10 client.16953 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:04:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.494+0000 7fb21cb8f740 20 client.16953 populate_metadata read hostname 'smithi169' 2024-03-28T01:04:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.494+0000 7fb21cb8f740 1 --2- 172.21.15.169:0/3504813445 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557bf45bfb50 0x557bf45c1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.494+0000 7fb21cb8f740 4 client.16953 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:04:22.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.494+0000 7fb21cb8f740 1 -- 172.21.15.169:0/3504813445 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557bf44ba3a0 con 0x557bf45bfb50 2024-03-28T01:04:22.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.495+0000 7fb212ffd640 1 --2- 172.21.15.169:0/3504813445 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557bf45bfb50 0x557bf45c1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:22.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.495+0000 7fb1f2ffd640 10 client.16953 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:22.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.495+0000 7fb212ffd640 1 --2- 172.21.15.169:0/3504813445 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557bf45bfb50 0x557bf45c1f70 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2971011ece69f95 server_cookie=dc2adf199492c13 in_seq=0 out_seq=0 2024-03-28T01:04:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.518+0000 7fb1f2ffd640 1 -- 172.21.15.169:0/3504813445 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557bf44ba3a0 con 0x557bf45bfb50 2024-03-28T01:04:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.518+0000 7fb1f2ffd640 10 client.16953 handle_command_reply: tid=0 2024-03-28T01:04:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.518+0000 7fb21cb8f740 1 client.16953 shutdown 2024-03-28T01:04:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.518+0000 7fb21cb8f740 1 -- 172.21.15.169:0/3504813445 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf454d920 msgr2=0x557bf455c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.518+0000 7fb21cb8f740 1 --2- 172.21.15.169:0/3504813445 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf454d920 0x557bf455c460 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb20402c7b0 tx=0x7fb20402bc30 comp rx=0 tx=0).stop 2024-03-28T01:04:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.518+0000 7fb21cb8f740 1 -- 172.21.15.169:0/3504813445 shutdown_connections 2024-03-28T01:04:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.518+0000 7fb21cb8f740 1 --2- 172.21.15.169:0/3504813445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557bf4553480 0x557bf455d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.518+0000 7fb21cb8f740 1 --2- 172.21.15.169:0/3504813445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bf454e310 0x557bf455c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.518+0000 7fb21cb8f740 1 --2- 172.21.15.169:0/3504813445 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x557bf45bfb50 0x557bf45c1f70 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.518+0000 7fb21cb8f740 1 --2- 172.21.15.169:0/3504813445 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557bf454d920 0x557bf455c460 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.518+0000 7fb21cb8f740 1 -- 172.21.15.169:0/3504813445 >> 172.21.15.169:0/3504813445 conn(0x557bf4548e90 msgr2=0x557bf4557ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.518+0000 7fb1f2ffd640 0 client.16953 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.518+0000 7fb21cb8f740 1 -- 172.21.15.169:0/3504813445 shutdown_connections 2024-03-28T01:04:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.518+0000 7fb21cb8f740 1 -- 172.21.15.169:0/3504813445 wait complete. 2024-03-28T01:04:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.518+0000 7fb21cb8f740 20 client.16953 trim_cache size 0 max 16384 2024-03-28T01:04:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:04:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:04:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:04:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:04:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.519+0000 7fb219fa9640 1 -- 172.21.15.169:0/1942732422 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1fc0a28a0 msgr2=0x7fb1fc0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.519+0000 7fb219fa9640 1 --2- 172.21.15.169:0/1942732422 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1fc0a28a0 0x7fb1fc0a4d60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb204000c00 tx=0x7fb204005fd0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.519+0000 7fb219fa9640 1 -- 172.21.15.169:0/1942732422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2141011e0 msgr2=0x7fb214196da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.519+0000 7fb219fa9640 1 --2- 172.21.15.169:0/1942732422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2141011e0 0x7fb214196da0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb20800e9f0 tx=0x7fb20800eeb0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.519+0000 7fb219fa9640 1 -- 172.21.15.169:0/1942732422 shutdown_connections 2024-03-28T01:04:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.519+0000 7fb219fa9640 1 --2- 172.21.15.169:0/1942732422 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb1fc0a28a0 0x7fb1fc0a4d60 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.519+0000 7fb219fa9640 1 --2- 172.21.15.169:0/1942732422 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb214101bd0 0x7fb21419de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.519+0000 7fb219fa9640 1 --2- 172.21.15.169:0/1942732422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2141011e0 0x7fb214196da0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.519+0000 7fb219fa9640 1 --2- 172.21.15.169:0/1942732422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb214100820 0x7fb214196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.519+0000 7fb219fa9640 1 -- 172.21.15.169:0/1942732422 >> 172.21.15.169:0/1942732422 conn(0x7fb2140fb2f0 msgr2=0x7fb214104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.519+0000 7fb219fa9640 1 -- 172.21.15.169:0/1942732422 shutdown_connections 2024-03-28T01:04:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:22.519+0000 7fb219fa9640 1 -- 172.21.15.169:0/1942732422 wait complete. 2024-03-28T01:04:22.936 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:04:22.936 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:04:23.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:23 smithi174 ceph-mon[36060]: pgmap v587: 129 pgs: 129 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 171 MiB/s rd, 363 MiB/s wr, 4.70k op/s 2024-03-28T01:04:23.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:23 smithi177 ceph-mon[36206]: pgmap v587: 129 pgs: 129 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 171 MiB/s rd, 363 MiB/s wr, 4.70k op/s 2024-03-28T01:04:24.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:23 smithi169 ceph-mon[32274]: pgmap v587: 129 pgs: 129 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 171 MiB/s rd, 363 MiB/s wr, 4.70k op/s 2024-03-28T01:04:24.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.752+0000 7f1e212c0640 1 -- 172.21.15.169:0/3592446598 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e1c10b010 msgr2=0x7f1e1c10d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:24.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.752+0000 7f1e212c0640 1 --2- 172.21.15.169:0/3592446598 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e1c10b010 0x7f1e1c10d480 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c009860 tx=0x7f1e0c02bb70 comp rx=0 tx=0).stop 2024-03-28T01:04:24.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.752+0000 7f1e212c0640 1 -- 172.21.15.169:0/3592446598 shutdown_connections 2024-03-28T01:04:24.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.752+0000 7f1e212c0640 1 --2- 172.21.15.169:0/3592446598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e1c10d9c0 0x7f1e1c10de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.752+0000 7f1e212c0640 1 --2- 172.21.15.169:0/3592446598 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e1c10b010 0x7f1e1c10d480 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.752+0000 7f1e212c0640 1 --2- 172.21.15.169:0/3592446598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e1c1086a0 0x7f1e1c10aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.752+0000 7f1e212c0640 1 -- 172.21.15.169:0/3592446598 >> 172.21.15.169:0/3592446598 conn(0x7f1e1c06e500 msgr2=0x7f1e1c070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:24.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.754+0000 7f1e212c0640 1 -- 172.21.15.169:0/3592446598 shutdown_connections 2024-03-28T01:04:24.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.755+0000 7f1e212c0640 1 -- 172.21.15.169:0/3592446598 wait complete. 2024-03-28T01:04:24.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.757+0000 7f1e212c0640 1 Processor -- start 2024-03-28T01:04:24.763 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.760+0000 7f1e212c0640 1 -- start start 2024-03-28T01:04:24.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.760+0000 7f1e212c0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e1c1086a0 0x7f1e1c19e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:24.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.760+0000 7f1e212c0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e1c10b010 0x7f1e1c19e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:24.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.760+0000 7f1e212c0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e1c10d9c0 0x7f1e1c1a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:24.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.760+0000 7f1e212c0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c1119a0 con 0x7f1e1c10d9c0 2024-03-28T01:04:24.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.760+0000 7f1e212c0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c111820 con 0x7f1e1c10b010 2024-03-28T01:04:24.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.760+0000 7f1e212c0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c111b20 con 0x7f1e1c1086a0 2024-03-28T01:04:24.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.761+0000 7f1e1affd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e1c1086a0 0x7f1e1c19e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:24.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.761+0000 7f1e1affd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e1c1086a0 0x7f1e1c19e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:49710/0 (socket says 172.21.15.169:49710) 2024-03-28T01:04:24.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.761+0000 7f1e1affd640 1 -- 172.21.15.169:0/310599169 learned_addr learned my addr 172.21.15.169:0/310599169 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:24.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.763+0000 7f1e1a7fc640 1 --2- 172.21.15.169:0/310599169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e1c10b010 0x7f1e1c19e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:24.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.764+0000 7f1e1affd640 1 -- 172.21.15.169:0/310599169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e1c10b010 msgr2=0x7f1e1c19e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:24.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.764+0000 7f1e1affd640 1 --2- 172.21.15.169:0/310599169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e1c10b010 0x7f1e1c19e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.764+0000 7f1e1affd640 1 -- 172.21.15.169:0/310599169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e1c10d9c0 msgr2=0x7f1e1c1a5660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:04:24.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.764+0000 7f1e1affd640 1 --2- 172.21.15.169:0/310599169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e1c10d9c0 0x7f1e1c1a5660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.764+0000 7f1e1affd640 1 -- 172.21.15.169:0/310599169 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1c1a5cd0 con 0x7f1e1c1086a0 2024-03-28T01:04:24.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.764+0000 7f1e1affd640 1 --2- 172.21.15.169:0/310599169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e1c1086a0 0x7f1e1c19e0b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1e1001ea20 tx=0x7f1e1001eee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:24.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.766+0000 7f1e03fff640 1 -- 172.21.15.169:0/310599169 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e10028070 con 0x7f1e1c1086a0 2024-03-28T01:04:24.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.766+0000 7f1e03fff640 1 -- 172.21.15.169:0/310599169 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e1001abc0 con 0x7f1e1c1086a0 2024-03-28T01:04:24.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.766+0000 7f1e03fff640 1 -- 172.21.15.169:0/310599169 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e10020630 con 0x7f1e1c1086a0 2024-03-28T01:04:24.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.769+0000 7f1e212c0640 1 -- 172.21.15.169:0/310599169 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e1c1a5fc0 con 0x7f1e1c1086a0 2024-03-28T01:04:24.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.769+0000 7f1e212c0640 1 -- 172.21.15.169:0/310599169 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e1c1a63f0 con 0x7f1e1c1086a0 2024-03-28T01:04:24.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.777+0000 7f1e23f87740 1 Processor -- start 2024-03-28T01:04:24.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.782+0000 7f1e23f87740 1 -- start start 2024-03-28T01:04:24.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.782+0000 7f1e23f87740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e78a66da0 0x558e78a671c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:24.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.782+0000 7f1e23f87740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a67790 0x558e78a6c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:24.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.782+0000 7f1e23f87740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e78a6c900 0x558e78a6ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:24.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.782+0000 7f1e23f87740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e789cc3d0 con 0x558e78a66da0 2024-03-28T01:04:24.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.782+0000 7f1e23f87740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e78974ad0 con 0x558e78a67790 2024-03-28T01:04:24.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.782+0000 7f1e23f87740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e789c8470 con 0x558e78a6c900 2024-03-28T01:04:24.789 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.785+0000 7f1e03fff640 1 -- 172.21.15.169:0/310599169 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1e10021470 con 0x7f1e1c1086a0 2024-03-28T01:04:24.789 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.785+0000 7f1e1b7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e78a66da0 0x558e78a671c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:24.789 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.785+0000 7f1e1b7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e78a66da0 0x558e78a671c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36110/0 (socket says 172.21.15.169:36110) 2024-03-28T01:04:24.789 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.785+0000 7f1e1b7fe640 1 -- 172.21.15.169:0/2678934543 learned_addr learned my addr 172.21.15.169:0/2678934543 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:24.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.786+0000 7f1e03fff640 1 --2- 172.21.15.169:0/310599169 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1e040a2950 0x7f1e040a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:24.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.786+0000 7f1e1affd640 1 --2- 172.21.15.169:0/2678934543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a67790 0x558e78a6c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:24.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.786+0000 7f1e1a7fc640 1 --2- 172.21.15.169:0/2678934543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e78a6c900 0x558e78a6ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:24.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.786+0000 7f1e03fff640 1 -- 172.21.15.169:0/310599169 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f1e100df450 con 0x7f1e1c1086a0 2024-03-28T01:04:24.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.786+0000 7f1e1b7fe640 1 --2- 172.21.15.169:0/310599169 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1e040a2950 0x7f1e040a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:24.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.787+0000 7f1e1b7fe640 1 -- 172.21.15.169:0/2678934543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e78a6c900 msgr2=0x558e78a6ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:24.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.787+0000 7f1e1b7fe640 1 --2- 172.21.15.169:0/2678934543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e78a6c900 0x558e78a6ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.787+0000 7f1e1b7fe640 1 -- 172.21.15.169:0/2678934543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a67790 msgr2=0x558e78a6c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:24.792 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.787+0000 7f1e1b7fe640 1 --2- 172.21.15.169:0/2678934543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a67790 0x558e78a6c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.792 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.787+0000 7f1e1b7fe640 1 -- 172.21.15.169:0/2678934543 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e787aa7d0 con 0x558e78a66da0 2024-03-28T01:04:24.792 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.787+0000 7f1e1b7fe640 1 --2- 172.21.15.169:0/310599169 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1e040a2950 0x7f1e040a4e10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1e14012870 tx=0x7f1e14009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:24.792 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.787+0000 7f1e1b7fe640 1 --2- 172.21.15.169:0/2678934543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e78a66da0 0x558e78a671c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f1e14009710 tx=0x7f1e14017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e71598b45965ff33 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:24.792 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.788+0000 7f1e017fa640 1 -- 172.21.15.169:0/2678934543 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1401d070 con 0x558e78a66da0 2024-03-28T01:04:24.792 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.788+0000 7f1e017fa640 1 -- 172.21.15.169:0/2678934543 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e14013040 con 0x558e78a66da0 2024-03-28T01:04:24.792 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.788+0000 7f1e017fa640 1 -- 172.21.15.169:0/2678934543 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e14018cc0 con 0x558e78a66da0 2024-03-28T01:04:24.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.794+0000 7f1e23f87740 1 -- 172.21.15.169:0/2678934543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e78a66da0 msgr2=0x558e78a671c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:24.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.794+0000 7f1e23f87740 1 --2- 172.21.15.169:0/2678934543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e78a66da0 0x558e78a671c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f1e14009710 tx=0x7f1e14017000 comp rx=0 tx=0).stop 2024-03-28T01:04:24.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.798+0000 7f1e23f87740 1 -- 172.21.15.169:0/2678934543 shutdown_connections 2024-03-28T01:04:24.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.798+0000 7f1e23f87740 1 --2- 172.21.15.169:0/2678934543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e78a6c900 0x558e78a6ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.798+0000 7f1e23f87740 1 --2- 172.21.15.169:0/2678934543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a67790 0x558e78a6c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.798+0000 7f1e23f87740 1 --2- 172.21.15.169:0/2678934543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e78a66da0 0x558e78a671c0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.798+0000 7f1e23f87740 1 -- 172.21.15.169:0/2678934543 >> 172.21.15.169:0/2678934543 conn(0x558e78a62310 msgr2=0x558e78a62710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:24.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.801+0000 7f1e23f87740 1 -- 172.21.15.169:0/2678934543 shutdown_connections 2024-03-28T01:04:24.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.801+0000 7f1e23f87740 1 -- 172.21.15.169:0/2678934543 wait complete. 2024-03-28T01:04:24.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.801+0000 7f1e23f87740 1 -- start start 2024-03-28T01:04:24.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.801+0000 7f1e23f87740 1 Processor -- start 2024-03-28T01:04:24.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.801+0000 7f1e23f87740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a66da0 0x558e78a711c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:24.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.802+0000 7f1e23f87740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e78a67790 0x558e78a71700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:24.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.802+0000 7f1e23f87740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e78a6c900 0x558e78a78740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:24.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.802+0000 7f1e23f87740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e78a6f7f0 con 0x558e78a6c900 2024-03-28T01:04:24.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.802+0000 7f1e23f87740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e78988060 con 0x558e78a66da0 2024-03-28T01:04:24.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.802+0000 7f1e23f87740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e78a6f970 con 0x558e78a67790 2024-03-28T01:04:24.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.802+0000 7f1e1affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a66da0 0x558e78a711c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:24.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.802+0000 7f1e1affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a66da0 0x558e78a711c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57512/0 (socket says 172.21.15.169:57512) 2024-03-28T01:04:24.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.802+0000 7f1e1affd640 1 -- 172.21.15.169:0/2558212560 learned_addr learned my addr 172.21.15.169:0/2558212560 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:24.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.802+0000 7f1e23f87740 10 client.0 _ll_register_callbacks cb 0x558e7896a6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:04:24.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.802+0000 7f1e1b7fe640 1 --2- 172.21.15.169:0/2558212560 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e78a6c900 0x558e78a78740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:24.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.802+0000 7f1e1affd640 1 -- 172.21.15.169:0/2558212560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e78a67790 msgr2=0x558e78a71700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:04:24.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.802+0000 7f1e1affd640 1 --2- 172.21.15.169:0/2558212560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e78a67790 0x558e78a71700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.802+0000 7f1e1affd640 1 -- 172.21.15.169:0/2558212560 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e78a6c900 msgr2=0x558e78a78740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:24.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.802+0000 7f1e1affd640 1 --2- 172.21.15.169:0/2558212560 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e78a6c900 0x558e78a78740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.802+0000 7f1e1affd640 1 -- 172.21.15.169:0/2558212560 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558e786e03f0 con 0x558e78a66da0 2024-03-28T01:04:24.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.803+0000 7f1e1affd640 1 --2- 172.21.15.169:0/2558212560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a66da0 0x558e78a711c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1e100b3c60 tx=0x7f1e10076c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f07e451521dd7fe4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:24.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.803+0000 7f1dfeff5640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:04:24.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.805+0000 7f1dfeff5640 1 -- 172.21.15.169:0/2558212560 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e100ac0f0 con 0x558e78a66da0 2024-03-28T01:04:24.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.805+0000 7f1dfeff5640 1 -- 172.21.15.169:0/2558212560 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f1e100df110 con 0x558e78a66da0 2024-03-28T01:04:24.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.806+0000 7f1e23f87740 1 -- 172.21.15.169:0/2558212560 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1e14013040 con 0x558e78a66da0 2024-03-28T01:04:24.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.807+0000 7f1dfeff5640 1 -- 172.21.15.169:0/2558212560 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e100ab9f0 con 0x558e78a66da0 2024-03-28T01:04:24.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.808+0000 7f1e23f87740 10 client.26822 fetch_fsmap learned FSMap version 25 2024-03-28T01:04:24.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.808+0000 7f1e23f87740 1 -- 172.21.15.169:0/2558212560 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558e786a00f0 con 0x558e78a66da0 2024-03-28T01:04:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.811+0000 7f1dfeff5640 1 -- 172.21.15.169:0/2558212560 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e14013040 con 0x558e78a66da0 2024-03-28T01:04:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.811+0000 7f1dfeff5640 1 -- 172.21.15.169:0/2558212560 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f1e1001f040 con 0x558e78a66da0 2024-03-28T01:04:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.811+0000 7f1e23f87740 10 client.26822 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:04:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.811+0000 7f1e23f87740 10 client.26822 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:04:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.811+0000 7f1e23f87740 20 client.26822 populate_metadata read hostname 'smithi169' 2024-03-28T01:04:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.811+0000 7f1e23f87740 1 --2- 172.21.15.169:0/2558212560 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558e78a7d070 0x558e78a7f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.811+0000 7f1e23f87740 4 client.26822 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:04:24.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.811+0000 7f1e23f87740 1 -- 172.21.15.169:0/2558212560 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558e789d9910 con 0x558e78a7d070 2024-03-28T01:04:24.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.814+0000 7f1e1a7fc640 1 --2- 172.21.15.169:0/2558212560 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558e78a7d070 0x558e78a7f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:24.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.815+0000 7f1e1a7fc640 1 --2- 172.21.15.169:0/2558212560 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558e78a7d070 0x558e78a7f4e0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6b7c5457b14f2ea server_cookie=9947cb7c8be3f042 in_seq=0 out_seq=0 2024-03-28T01:04:24.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.815+0000 7f1dfeff5640 10 client.26822 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:24.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.817+0000 7f1dfeff5640 1 -- 172.21.15.169:0/2558212560 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558e789d9910 con 0x558e78a7d070 2024-03-28T01:04:24.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.817+0000 7f1dfeff5640 10 client.26822 handle_command_reply: tid=0 2024-03-28T01:04:24.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.817+0000 7f1e23f87740 1 client.26822 shutdown 2024-03-28T01:04:24.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.821+0000 7f1e23f87740 1 -- 172.21.15.169:0/2558212560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a66da0 msgr2=0x558e78a711c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:24.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.821+0000 7f1e23f87740 1 --2- 172.21.15.169:0/2558212560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a66da0 0x558e78a711c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1e100b3c60 tx=0x7f1e10076c20 comp rx=0 tx=0).stop 2024-03-28T01:04:24.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.828+0000 7f1e23f87740 1 -- 172.21.15.169:0/2558212560 shutdown_connections 2024-03-28T01:04:24.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.828+0000 7f1e23f87740 1 --2- 172.21.15.169:0/2558212560 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558e78a7d070 0x558e78a7f4e0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.828+0000 7f1e23f87740 1 --2- 172.21.15.169:0/2558212560 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e78a6c900 0x558e78a78740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.828+0000 7f1e23f87740 1 --2- 172.21.15.169:0/2558212560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e78a67790 0x558e78a71700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.828+0000 7f1e23f87740 1 --2- 172.21.15.169:0/2558212560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a66da0 0x558e78a711c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.828+0000 7f1e23f87740 1 -- 172.21.15.169:0/2558212560 >> 172.21.15.169:0/2558212560 conn(0x558e78a62310 msgr2=0x558e78a61530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:24.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.829+0000 7f1dfeff5640 0 client.26822 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:24.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.830+0000 7f1e23f87740 1 -- 172.21.15.169:0/2558212560 shutdown_connections 2024-03-28T01:04:24.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.830+0000 7f1e23f87740 1 -- 172.21.15.169:0/2558212560 wait complete. 2024-03-28T01:04:24.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.830+0000 7f1e23f87740 20 client.26822 trim_cache size 0 max 16384 2024-03-28T01:04:24.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.862+0000 7f1e23f87740 1 Processor -- start 2024-03-28T01:04:24.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.866+0000 7f1e23f87740 1 -- start start 2024-03-28T01:04:24.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.866+0000 7f1e23f87740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a66da0 0x558e78a70dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:24.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.866+0000 7f1e23f87740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e78a67790 0x558e78a78700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:24.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.866+0000 7f1e23f87740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e78a6c900 0x558e78a71310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:24.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.866+0000 7f1e23f87740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c1b2fd0 con 0x558e78a67790 2024-03-28T01:04:24.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.866+0000 7f1e23f87740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c1b0ad0 con 0x558e78a66da0 2024-03-28T01:04:24.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.866+0000 7f1e23f87740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c1aeb70 con 0x558e78a6c900 2024-03-28T01:04:24.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.866+0000 7f1e1b7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e78a6c900 0x558e78a71310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:24.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.866+0000 7f1e1b7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e78a6c900 0x558e78a71310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:49726/0 (socket says 172.21.15.169:49726) 2024-03-28T01:04:24.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.866+0000 7f1e1b7fe640 1 -- 172.21.15.169:0/757940432 learned_addr learned my addr 172.21.15.169:0/757940432 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:24.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.867+0000 7f1e1affd640 1 --2- 172.21.15.169:0/757940432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a66da0 0x558e78a70dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:24.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.868+0000 7f1e1b7fe640 1 -- 172.21.15.169:0/757940432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a66da0 msgr2=0x558e78a70dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:24.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.868+0000 7f1e1b7fe640 1 --2- 172.21.15.169:0/757940432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a66da0 0x558e78a70dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.868+0000 7f1e1b7fe640 1 -- 172.21.15.169:0/757940432 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e78a67790 msgr2=0x558e78a78700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:04:24.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.868+0000 7f1e1b7fe640 1 --2- 172.21.15.169:0/757940432 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e78a67790 0x558e78a78700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.868+0000 7f1e1b7fe640 1 -- 172.21.15.169:0/757940432 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e78a70270 con 0x558e78a6c900 2024-03-28T01:04:24.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.873+0000 7f1e1b7fe640 1 --2- 172.21.15.169:0/757940432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e78a6c900 0x558e78a71310 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1e14004a80 tx=0x7f1e140178e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ddf6150315f1589 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:24.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.877+0000 7f1e00ff9640 1 -- 172.21.15.169:0/757940432 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1401d040 con 0x558e78a6c900 2024-03-28T01:04:24.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.878+0000 7f1e00ff9640 1 -- 172.21.15.169:0/757940432 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e14007000 con 0x558e78a6c900 2024-03-28T01:04:24.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.878+0000 7f1e00ff9640 1 -- 172.21.15.169:0/757940432 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1402bb30 con 0x558e78a6c900 2024-03-28T01:04:24.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.881+0000 7f1e23f87740 1 -- 172.21.15.169:0/757940432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e78a6c900 msgr2=0x558e78a71310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:24.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.881+0000 7f1e23f87740 1 --2- 172.21.15.169:0/757940432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e78a6c900 0x558e78a71310 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1e14004a80 tx=0x7f1e140178e0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.884+0000 7f1e23f87740 1 -- 172.21.15.169:0/757940432 shutdown_connections 2024-03-28T01:04:24.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.884+0000 7f1e23f87740 1 --2- 172.21.15.169:0/757940432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e78a6c900 0x558e78a71310 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.884+0000 7f1e23f87740 1 --2- 172.21.15.169:0/757940432 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e78a67790 0x558e78a78700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.884+0000 7f1e23f87740 1 --2- 172.21.15.169:0/757940432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a66da0 0x558e78a70dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.884+0000 7f1e23f87740 1 -- 172.21.15.169:0/757940432 >> 172.21.15.169:0/757940432 conn(0x558e78a62310 msgr2=0x558e78a68190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:24.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.884+0000 7f1e23f87740 1 -- 172.21.15.169:0/757940432 shutdown_connections 2024-03-28T01:04:24.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.885+0000 7f1e23f87740 1 -- 172.21.15.169:0/757940432 wait complete. 2024-03-28T01:04:24.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.886+0000 7f1e23f87740 1 -- start start 2024-03-28T01:04:24.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.886+0000 7f1e23f87740 1 Processor -- start 2024-03-28T01:04:24.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.890+0000 7f1e23f87740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a66da0 0x558e78a75e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:24.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.890+0000 7f1e23f87740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e78a67790 0x558e78a76360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:24.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.890+0000 7f1e23f87740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e78a6c900 0x558e78a77320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:24.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.890+0000 7f1e23f87740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c1af170 con 0x558e78a67790 2024-03-28T01:04:24.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.890+0000 7f1e23f87740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c1b0470 con 0x558e78a66da0 2024-03-28T01:04:24.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.890+0000 7f1e23f87740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c1af7b0 con 0x558e78a6c900 2024-03-28T01:04:24.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.890+0000 7f1e23f87740 10 client.0 _ll_register_callbacks cb 0x7f1e100a7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:04:24.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.890+0000 7f1e1affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a66da0 0x558e78a75e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:24.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.890+0000 7f1e1affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a66da0 0x558e78a75e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57534/0 (socket says 172.21.15.169:57534) 2024-03-28T01:04:24.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.890+0000 7f1e1affd640 1 -- 172.21.15.169:0/127102488 learned_addr learned my addr 172.21.15.169:0/127102488 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:24.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.891+0000 7f1e1b7fe640 1 --2- 172.21.15.169:0/127102488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e78a6c900 0x558e78a77320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:24.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.894+0000 7f1e1a7fc640 1 --2- 172.21.15.169:0/127102488 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e78a67790 0x558e78a76360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:24.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.896+0000 7f1e1affd640 1 -- 172.21.15.169:0/127102488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e78a6c900 msgr2=0x558e78a77320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:24.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.896+0000 7f1e1affd640 1 --2- 172.21.15.169:0/127102488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e78a6c900 0x558e78a77320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.896+0000 7f1e1affd640 1 -- 172.21.15.169:0/127102488 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e78a67790 msgr2=0x558e78a76360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:24.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.896+0000 7f1e1affd640 1 --2- 172.21.15.169:0/127102488 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e78a67790 0x558e78a76360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.896+0000 7f1e1affd640 1 -- 172.21.15.169:0/127102488 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558e78a77910 con 0x558e78a66da0 2024-03-28T01:04:24.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.897+0000 7f1e1affd640 1 --2- 172.21.15.169:0/127102488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a66da0 0x558e78a75e20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1e100a6490 tx=0x7f1e100b3e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2c60dc5664f84b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:24.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.897+0000 7f1dfeff5640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:04:24.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.897+0000 7f1dfeff5640 1 -- 172.21.15.169:0/127102488 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1007d550 con 0x558e78a66da0 2024-03-28T01:04:24.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.897+0000 7f1dfeff5640 1 -- 172.21.15.169:0/127102488 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f1e100aae40 con 0x558e78a66da0 2024-03-28T01:04:24.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.897+0000 7f1e23f87740 1 -- 172.21.15.169:0/127102488 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1e14007000 con 0x558e78a66da0 2024-03-28T01:04:24.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.899+0000 7f1dfeff5640 1 -- 172.21.15.169:0/127102488 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e100a61c0 con 0x558e78a66da0 2024-03-28T01:04:24.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.899+0000 7f1e23f87740 10 client.26825 fetch_fsmap learned FSMap version 25 2024-03-28T01:04:24.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.899+0000 7f1e23f87740 1 -- 172.21.15.169:0/127102488 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558e78ad89b0 con 0x558e78a66da0 2024-03-28T01:04:24.906 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.903+0000 7f1dfeff5640 1 -- 172.21.15.169:0/127102488 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e14007000 con 0x558e78a66da0 2024-03-28T01:04:24.906 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.903+0000 7f1dfeff5640 1 -- 172.21.15.169:0/127102488 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f1e100c4ab0 con 0x558e78a66da0 2024-03-28T01:04:24.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.906+0000 7f1e23f87740 10 client.26825 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:04:24.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.906+0000 7f1e23f87740 10 client.26825 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:04:24.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.906+0000 7f1e23f87740 20 client.26825 populate_metadata read hostname 'smithi169' 2024-03-28T01:04:24.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.906+0000 7f1e23f87740 1 --2- 172.21.15.169:0/127102488 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558e78ad8fe0 0x558e78adb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:24.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.906+0000 7f1e23f87740 4 client.26825 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:04:24.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.906+0000 7f1e23f87740 1 -- 172.21.15.169:0/127102488 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558e789d3c60 con 0x558e78ad8fe0 2024-03-28T01:04:24.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.907+0000 7f1e1a7fc640 1 --2- 172.21.15.169:0/127102488 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558e78ad8fe0 0x558e78adb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:24.914 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.911+0000 7f1e1a7fc640 1 --2- 172.21.15.169:0/127102488 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558e78ad8fe0 0x558e78adb3d0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6352fcf42d6fa8a8 server_cookie=16e74bc17aa04394 in_seq=0 out_seq=0 2024-03-28T01:04:24.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.912+0000 7f1dfeff5640 10 client.26825 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:24.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.916+0000 7f1dfeff5640 1 -- 172.21.15.169:0/127102488 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x558e789d3c60 con 0x558e78ad8fe0 2024-03-28T01:04:24.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.916+0000 7f1dfeff5640 10 client.26825 handle_command_reply: tid=0 2024-03-28T01:04:24.921 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.917+0000 7f1e23f87740 1 client.26825 shutdown 2024-03-28T01:04:24.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.919+0000 7f1e23f87740 1 -- 172.21.15.169:0/127102488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a66da0 msgr2=0x558e78a75e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:24.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.919+0000 7f1e23f87740 1 --2- 172.21.15.169:0/127102488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a66da0 0x558e78a75e20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1e100a6490 tx=0x7f1e100b3e00 comp rx=0 tx=0).stop 2024-03-28T01:04:24.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.926+0000 7f1e23f87740 1 -- 172.21.15.169:0/127102488 shutdown_connections 2024-03-28T01:04:24.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.926+0000 7f1e23f87740 1 --2- 172.21.15.169:0/127102488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558e78a6c900 0x558e78a77320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.926+0000 7f1e23f87740 1 --2- 172.21.15.169:0/127102488 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x558e78ad8fe0 0x558e78adb3d0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.926+0000 7f1e23f87740 1 --2- 172.21.15.169:0/127102488 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558e78a67790 0x558e78a76360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.926+0000 7f1e23f87740 1 --2- 172.21.15.169:0/127102488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e78a66da0 0x558e78a75e20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.926+0000 7f1e23f87740 1 -- 172.21.15.169:0/127102488 >> 172.21.15.169:0/127102488 conn(0x558e78a62310 msgr2=0x558e78a70dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:24.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.926+0000 7f1dfeff5640 0 client.26825 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:24.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.926+0000 7f1e23f87740 1 -- 172.21.15.169:0/127102488 shutdown_connections 2024-03-28T01:04:24.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.927+0000 7f1e23f87740 1 -- 172.21.15.169:0/127102488 wait complete. 2024-03-28T01:04:24.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.928+0000 7f1e23f87740 20 client.26825 trim_cache size 0 max 16384 2024-03-28T01:04:24.934 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:04:24.934 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "no active scrubs running", 2024-03-28T01:04:24.934 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": {} 2024-03-28T01:04:24.934 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:04:24.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.936+0000 7f1e212c0640 1 -- 172.21.15.169:0/310599169 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1e040a2950 msgr2=0x7f1e040a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:24.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.936+0000 7f1e212c0640 1 --2- 172.21.15.169:0/310599169 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1e040a2950 0x7f1e040a4e10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1e14012870 tx=0x7f1e14009920 comp rx=0 tx=0).stop 2024-03-28T01:04:24.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.936+0000 7f1e212c0640 1 -- 172.21.15.169:0/310599169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e1c1086a0 msgr2=0x7f1e1c19e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:24.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.936+0000 7f1e212c0640 1 --2- 172.21.15.169:0/310599169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e1c1086a0 0x7f1e1c19e0b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1e1001ea20 tx=0x7f1e1001eee0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.936+0000 7f1e212c0640 1 -- 172.21.15.169:0/310599169 shutdown_connections 2024-03-28T01:04:24.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.936+0000 7f1e212c0640 1 --2- 172.21.15.169:0/310599169 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1e040a2950 0x7f1e040a4e10 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.936+0000 7f1e212c0640 1 --2- 172.21.15.169:0/310599169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e1c10d9c0 0x7f1e1c1a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.936+0000 7f1e212c0640 1 --2- 172.21.15.169:0/310599169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e1c10b010 0x7f1e1c19e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.936+0000 7f1e212c0640 1 --2- 172.21.15.169:0/310599169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e1c1086a0 0x7f1e1c19e0b0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:24.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.936+0000 7f1e212c0640 1 -- 172.21.15.169:0/310599169 >> 172.21.15.169:0/310599169 conn(0x7f1e1c06e500 msgr2=0x7f1e1c10b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:24.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.936+0000 7f1e212c0640 1 -- 172.21.15.169:0/310599169 shutdown_connections 2024-03-28T01:04:24.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:24.937+0000 7f1e212c0640 1 -- 172.21.15.169:0/310599169 wait complete. 2024-03-28T01:04:24.973 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T01:04:24.974 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T01:04:25.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.156+0000 7f94061cd640 1 -- 172.21.15.169:0/2023693207 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94001017b0 msgr2=0x7f9400101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:25.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.156+0000 7f94061cd640 1 --2- 172.21.15.169:0/2023693207 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94001017b0 0x7f9400101c30 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f93f4009900 tx=0x7f93f402bc30 comp rx=0 tx=0).stop 2024-03-28T01:04:25.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.161+0000 7f94061cd640 1 -- 172.21.15.169:0/2023693207 shutdown_connections 2024-03-28T01:04:25.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.161+0000 7f94061cd640 1 --2- 172.21.15.169:0/2023693207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9400102170 0x7f9400108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:25.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.161+0000 7f94061cd640 1 --2- 172.21.15.169:0/2023693207 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94001017b0 0x7f9400101c30 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:25.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.161+0000 7f94061cd640 1 --2- 172.21.15.169:0/2023693207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94001005c0 0x7f94001009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:25.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.161+0000 7f94061cd640 1 -- 172.21.15.169:0/2023693207 >> 172.21.15.169:0/2023693207 conn(0x7f94000fbb30 msgr2=0x7f94000fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:25.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.169+0000 7f94061cd640 1 -- 172.21.15.169:0/2023693207 shutdown_connections 2024-03-28T01:04:25.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.169+0000 7f94061cd640 1 -- 172.21.15.169:0/2023693207 wait complete. 2024-03-28T01:04:25.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.170+0000 7f94061cd640 1 Processor -- start 2024-03-28T01:04:25.174 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.171+0000 7f94061cd640 1 -- start start 2024-03-28T01:04:25.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.171+0000 7f94061cd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94001005c0 0x7f9400195790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:25.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.171+0000 7f94061cd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94001017b0 0x7f9400195cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:25.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.171+0000 7f94061cd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9400102170 0x7f940019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:25.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.172+0000 7f94061cd640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940010b140 con 0x7f94001017b0 2024-03-28T01:04:25.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.172+0000 7f94061cd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940010afc0 con 0x7f9400102170 2024-03-28T01:04:25.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.172+0000 7f93fffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9400102170 0x7f940019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:25.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.172+0000 7f93fffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9400102170 0x7f940019cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57550/0 (socket says 172.21.15.169:57550) 2024-03-28T01:04:25.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.172+0000 7f93fffff640 1 -- 172.21.15.169:0/307265944 learned_addr learned my addr 172.21.15.169:0/307265944 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:25.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.172+0000 7f94061cd640 1 -- 172.21.15.169:0/307265944 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940010b2c0 con 0x7f94001005c0 2024-03-28T01:04:25.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.173+0000 7f93ff7fe640 1 --2- 172.21.15.169:0/307265944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94001005c0 0x7f9400195790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:25.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.175+0000 7f93ff7fe640 1 -- 172.21.15.169:0/307265944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9400102170 msgr2=0x7f940019cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:25.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.175+0000 7f93ff7fe640 1 --2- 172.21.15.169:0/307265944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9400102170 0x7f940019cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:25.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.175+0000 7f93ff7fe640 1 -- 172.21.15.169:0/307265944 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94001017b0 msgr2=0x7f9400195cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:25.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.175+0000 7f93ff7fe640 1 --2- 172.21.15.169:0/307265944 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94001017b0 0x7f9400195cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:25.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.175+0000 7f93ff7fe640 1 -- 172.21.15.169:0/307265944 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940019d3b0 con 0x7f94001005c0 2024-03-28T01:04:25.182 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.179+0000 7f93ff7fe640 1 --2- 172.21.15.169:0/307265944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94001005c0 0x7f9400195790 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f93f000e9c0 tx=0x7f93f000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:25.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.186+0000 7f93fcff9640 1 -- 172.21.15.169:0/307265944 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f000ccc0 con 0x7f94001005c0 2024-03-28T01:04:25.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.186+0000 7f93fcff9640 1 -- 172.21.15.169:0/307265944 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93f00035f0 con 0x7f94001005c0 2024-03-28T01:04:25.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.186+0000 7f93fcff9640 1 -- 172.21.15.169:0/307265944 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f000ce60 con 0x7f94001005c0 2024-03-28T01:04:25.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.187+0000 7f94061cd640 1 -- 172.21.15.169:0/307265944 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f940019d6a0 con 0x7f94001005c0 2024-03-28T01:04:25.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.187+0000 7f94061cd640 1 -- 172.21.15.169:0/307265944 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f940019dba0 con 0x7f94001005c0 2024-03-28T01:04:25.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.191+0000 7f94061cd640 1 -- 172.21.15.169:0/307265944 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93ec0052c0 con 0x7f94001005c0 2024-03-28T01:04:25.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.199+0000 7f93fcff9640 1 -- 172.21.15.169:0/307265944 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f93f0003f30 con 0x7f94001005c0 2024-03-28T01:04:25.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.200+0000 7f93fcff9640 1 --2- 172.21.15.169:0/307265944 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f93e80a6de0 0x7f93e80a92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:25.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.200+0000 7f93fcff9640 1 -- 172.21.15.169:0/307265944 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f93f00cf010 con 0x7f94001005c0 2024-03-28T01:04:25.207 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.205+0000 7f93feffd640 1 --2- 172.21.15.169:0/307265944 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f93e80a6de0 0x7f93e80a92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:25.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.216+0000 7f93fcff9640 1 -- 172.21.15.169:0/307265944 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f93f009bd30 con 0x7f94001005c0 2024-03-28T01:04:25.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.216+0000 7f93feffd640 1 --2- 172.21.15.169:0/307265944 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f93e80a6de0 0x7f93e80a92a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f93f4005ab0 tx=0x7f93f4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:25.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.705+0000 7f94061cd640 1 -- 172.21.15.169:0/307265944 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93ec0055b0 con 0x7f94001005c0 2024-03-28T01:04:25.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.709+0000 7f93fcff9640 1 -- 172.21.15.169:0/307265944 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7f93f009b630 con 0x7f94001005c0 2024-03-28T01:04:25.713 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T01:04:25.713 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T01:04:25.713 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T01:04:25.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.713+0000 7f94061cd640 1 -- 172.21.15.169:0/307265944 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f93e80a6de0 msgr2=0x7f93e80a92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:25.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.713+0000 7f94061cd640 1 --2- 172.21.15.169:0/307265944 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f93e80a6de0 0x7f93e80a92a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f93f4005ab0 tx=0x7f93f4000bc0 comp rx=0 tx=0).stop 2024-03-28T01:04:25.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.713+0000 7f94061cd640 1 -- 172.21.15.169:0/307265944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94001005c0 msgr2=0x7f9400195790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:25.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.713+0000 7f94061cd640 1 --2- 172.21.15.169:0/307265944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94001005c0 0x7f9400195790 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f93f000e9c0 tx=0x7f93f000ee80 comp rx=0 tx=0).stop 2024-03-28T01:04:25.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.713+0000 7f94061cd640 1 -- 172.21.15.169:0/307265944 shutdown_connections 2024-03-28T01:04:25.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.713+0000 7f94061cd640 1 --2- 172.21.15.169:0/307265944 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f93e80a6de0 0x7f93e80a92a0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:25.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.713+0000 7f94061cd640 1 --2- 172.21.15.169:0/307265944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9400102170 0x7f940019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:25.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.713+0000 7f94061cd640 1 --2- 172.21.15.169:0/307265944 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94001017b0 0x7f9400195cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:25.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.713+0000 7f94061cd640 1 --2- 172.21.15.169:0/307265944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94001005c0 0x7f9400195790 secure :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f93f000e9c0 tx=0x7f93f000ee80 comp rx=0 tx=0).stop 2024-03-28T01:04:25.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.713+0000 7f94061cd640 1 -- 172.21.15.169:0/307265944 >> 172.21.15.169:0/307265944 conn(0x7f94000fbb30 msgr2=0x7f9400104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:25.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.715+0000 7f94061cd640 1 -- 172.21.15.169:0/307265944 shutdown_connections 2024-03-28T01:04:25.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.715+0000 7f94061cd640 1 -- 172.21.15.169:0/307265944 wait complete. 2024-03-28T01:04:25.746 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:04:25.746 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T01:04:25.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:25 smithi174 ceph-mon[36060]: pgmap v588: 129 pgs: 129 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 158 MiB/s rd, 333 MiB/s wr, 4.55k op/s 2024-03-28T01:04:25.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:25 smithi177 ceph-mon[36206]: pgmap v588: 129 pgs: 129 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 158 MiB/s rd, 333 MiB/s wr, 4.55k op/s 2024-03-28T01:04:25.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.956+0000 7f59ab081640 1 -- 172.21.15.169:0/3682510388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59a41017d0 msgr2=0x7f59a4101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:25.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.956+0000 7f59ab081640 1 --2- 172.21.15.169:0/3682510388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59a41017d0 0x7f59a4101c50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f599400b9f0 tx=0x7f599402d930 comp rx=0 tx=0).stop 2024-03-28T01:04:25.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.960+0000 7f59ab081640 1 -- 172.21.15.169:0/3682510388 shutdown_connections 2024-03-28T01:04:25.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.960+0000 7f59ab081640 1 --2- 172.21.15.169:0/3682510388 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59a4102190 0x7f59a4108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:25.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.960+0000 7f59ab081640 1 --2- 172.21.15.169:0/3682510388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59a41017d0 0x7f59a4101c50 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:25.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.960+0000 7f59ab081640 1 --2- 172.21.15.169:0/3682510388 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f59a41005e0 0x7f59a4100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:25.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.960+0000 7f59ab081640 1 -- 172.21.15.169:0/3682510388 >> 172.21.15.169:0/3682510388 conn(0x7f59a40fbb50 msgr2=0x7f59a40fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:25.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.964+0000 7f59ab081640 1 -- 172.21.15.169:0/3682510388 shutdown_connections 2024-03-28T01:04:25.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.964+0000 7f59ab081640 1 -- 172.21.15.169:0/3682510388 wait complete. 2024-03-28T01:04:25.968 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.965+0000 7f59ab081640 1 Processor -- start 2024-03-28T01:04:25.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.971+0000 7f59ab081640 1 -- start start 2024-03-28T01:04:25.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.971+0000 7f59ab081640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59a41005e0 0x7f59a41a6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:25.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.971+0000 7f59ab081640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59a41017d0 0x7f59a41a6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:25.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.971+0000 7f59ab081640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f59a4102190 0x7f59a41ade00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:25.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.971+0000 7f59ab081640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a410b1b0 con 0x7f59a4102190 2024-03-28T01:04:25.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.971+0000 7f59ab081640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a410b030 con 0x7f59a41017d0 2024-03-28T01:04:25.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.971+0000 7f59ab081640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a410b330 con 0x7f59a41005e0 2024-03-28T01:04:25.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.974+0000 7f59a961e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f59a4102190 0x7f59a41ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:25.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.974+0000 7f59a961e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f59a4102190 0x7f59a41ade00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36182/0 (socket says 172.21.15.169:36182) 2024-03-28T01:04:25.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.974+0000 7f59a961e640 1 -- 172.21.15.169:0/2028257193 learned_addr learned my addr 172.21.15.169:0/2028257193 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:25.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.974+0000 7f59a3fff640 1 --2- 172.21.15.169:0/2028257193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59a41017d0 0x7f59a41a6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:25.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.975+0000 7f59a8e1d640 1 --2- 172.21.15.169:0/2028257193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59a41005e0 0x7f59a41a6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:25.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.976+0000 7f59a3fff640 1 -- 172.21.15.169:0/2028257193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59a41005e0 msgr2=0x7f59a41a6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:25.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.976+0000 7f59a3fff640 1 --2- 172.21.15.169:0/2028257193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59a41005e0 0x7f59a41a6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:25.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.976+0000 7f59a3fff640 1 -- 172.21.15.169:0/2028257193 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f59a4102190 msgr2=0x7f59a41ade00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:25.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.976+0000 7f59a3fff640 1 --2- 172.21.15.169:0/2028257193 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f59a4102190 0x7f59a41ade00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:25.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.976+0000 7f59a3fff640 1 -- 172.21.15.169:0/2028257193 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a41ae470 con 0x7f59a41017d0 2024-03-28T01:04:25.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.977+0000 7f59a3fff640 1 --2- 172.21.15.169:0/2028257193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59a41017d0 0x7f59a41a6d90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5994000c00 tx=0x7f5994033690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:25.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.977+0000 7f59a1ffb640 1 -- 172.21.15.169:0/2028257193 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5994037070 con 0x7f59a41017d0 2024-03-28T01:04:25.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.977+0000 7f59ab081640 1 -- 172.21.15.169:0/2028257193 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a41ae700 con 0x7f59a41017d0 2024-03-28T01:04:25.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.978+0000 7f59ab081640 1 -- 172.21.15.169:0/2028257193 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a41aebf0 con 0x7f59a41017d0 2024-03-28T01:04:25.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.978+0000 7f59a1ffb640 1 -- 172.21.15.169:0/2028257193 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5994033ac0 con 0x7f59a41017d0 2024-03-28T01:04:25.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.978+0000 7f59a1ffb640 1 -- 172.21.15.169:0/2028257193 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599403c820 con 0x7f59a41017d0 2024-03-28T01:04:25.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.985+0000 7f59adc6d740 1 Processor -- start 2024-03-28T01:04:25.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.985+0000 7f59adc6d740 1 -- start start 2024-03-28T01:04:25.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.985+0000 7f59adc6d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4a375f690 0x55b4a375fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:25.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.985+0000 7f59adc6d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a3760080 0x55b4a3764b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:25.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.985+0000 7f59adc6d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37651f0 0x55b4a3767680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:25.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.985+0000 7f59adc6d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4a3722610 con 0x55b4a37651f0 2024-03-28T01:04:25.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.985+0000 7f59adc6d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4a36c53c0 con 0x55b4a375f690 2024-03-28T01:04:25.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.985+0000 7f59adc6d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4a366dd60 con 0x55b4a3760080 2024-03-28T01:04:25.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.986+0000 7f59a1ffb640 1 -- 172.21.15.169:0/2028257193 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f599403a020 con 0x7f59a41017d0 2024-03-28T01:04:25.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.986+0000 7f59a961e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4a375f690 0x55b4a375fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:25.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.986+0000 7f59a961e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4a375f690 0x55b4a375fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57596/0 (socket says 172.21.15.169:57596) 2024-03-28T01:04:25.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.986+0000 7f59a961e640 1 -- 172.21.15.169:0/4185727480 learned_addr learned my addr 172.21.15.169:0/4185727480 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:25.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.986+0000 7f59a8e1d640 1 --2- 172.21.15.169:0/4185727480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a3760080 0x55b4a3764b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:25.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.986+0000 7f59a961e640 1 -- 172.21.15.169:0/4185727480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a3760080 msgr2=0x55b4a3764b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:25.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.986+0000 7f59a961e640 1 --2- 172.21.15.169:0/4185727480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a3760080 0x55b4a3764b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:25.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.986+0000 7f59a961e640 1 -- 172.21.15.169:0/4185727480 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37651f0 msgr2=0x55b4a3767680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:25.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.986+0000 7f59a961e640 1 --2- 172.21.15.169:0/4185727480 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37651f0 0x55b4a3767680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:25.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.986+0000 7f59a961e640 1 -- 172.21.15.169:0/4185727480 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b4a33d93f0 con 0x55b4a375f690 2024-03-28T01:04:25.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.987+0000 7f59a961e640 1 --2- 172.21.15.169:0/4185727480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4a375f690 0x55b4a375fab0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f599c00f5c0 tx=0x7f599c00fa80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d54c4033ace89650 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:25.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.987+0000 7f59a8e1d640 1 --2- 172.21.15.169:0/4185727480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a3760080 0x55b4a3764b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:04:25.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.987+0000 7f598affd640 1 -- 172.21.15.169:0/4185727480 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c01d070 con 0x55b4a375f690 2024-03-28T01:04:25.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.987+0000 7f598affd640 1 -- 172.21.15.169:0/4185727480 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f599c00cc80 con 0x55b4a375f690 2024-03-28T01:04:25.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.988+0000 7f598affd640 1 -- 172.21.15.169:0/4185727480 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c018e30 con 0x55b4a375f690 2024-03-28T01:04:25.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.989+0000 7f59adc6d740 1 -- 172.21.15.169:0/4185727480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4a375f690 msgr2=0x55b4a375fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:25.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.989+0000 7f59adc6d740 1 --2- 172.21.15.169:0/4185727480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4a375f690 0x55b4a375fab0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f599c00f5c0 tx=0x7f599c00fa80 comp rx=0 tx=0).stop 2024-03-28T01:04:25.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.989+0000 7f59a1ffb640 1 --2- 172.21.15.169:0/2028257193 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f598c0a2a00 0x7f598c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:25.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.990+0000 7f59a1ffb640 1 -- 172.21.15.169:0/2028257193 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f59940f0f30 con 0x7f59a41017d0 2024-03-28T01:04:25.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.990+0000 7f59adc6d740 1 -- 172.21.15.169:0/4185727480 shutdown_connections 2024-03-28T01:04:25.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.990+0000 7f59adc6d740 1 --2- 172.21.15.169:0/4185727480 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37651f0 0x55b4a3767680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:25.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.990+0000 7f59adc6d740 1 --2- 172.21.15.169:0/4185727480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a3760080 0x55b4a3764b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:25.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.991+0000 7f59adc6d740 1 --2- 172.21.15.169:0/4185727480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4a375f690 0x55b4a375fab0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:25.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.991+0000 7f59adc6d740 1 -- 172.21.15.169:0/4185727480 >> 172.21.15.169:0/4185727480 conn(0x55b4a375ac00 msgr2=0x55b4a375b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:25.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.991+0000 7f59a961e640 1 --2- 172.21.15.169:0/2028257193 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f598c0a2a00 0x7f598c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:25.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:25.995+0000 7f59a961e640 1 --2- 172.21.15.169:0/2028257193 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f598c0a2a00 0x7f598c0a4ec0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f599c009f70 tx=0x7f599c018970 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:26.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.000+0000 7f59adc6d740 1 -- 172.21.15.169:0/4185727480 shutdown_connections 2024-03-28T01:04:26.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.006+0000 7f59adc6d740 1 -- 172.21.15.169:0/4185727480 wait complete. 2024-03-28T01:04:26.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.006+0000 7f59adc6d740 1 -- start start 2024-03-28T01:04:26.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.007+0000 7f59adc6d740 1 Processor -- start 2024-03-28T01:04:26.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.008+0000 7f59adc6d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a375f690 0x55b4a377a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.008+0000 7f59adc6d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4a3760080 0x55b4a377ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.008+0000 7f59adc6d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37651f0 0x55b4a3781e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.008+0000 7f59adc6d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4a3681240 con 0x55b4a37651f0 2024-03-28T01:04:26.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.008+0000 7f59adc6d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4a36c1460 con 0x55b4a3760080 2024-03-28T01:04:26.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.008+0000 7f59adc6d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4a3779200 con 0x55b4a375f690 2024-03-28T01:04:26.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.008+0000 7f59adc6d740 10 client.0 _ll_register_callbacks cb 0x55b4a3680c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:04:26.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.008+0000 7f59a961e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37651f0 0x55b4a3781e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.008+0000 7f59a961e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37651f0 0x55b4a3781e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36206/0 (socket says 172.21.15.169:36206) 2024-03-28T01:04:26.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.008+0000 7f59a961e640 1 -- 172.21.15.169:0/1888043009 learned_addr learned my addr 172.21.15.169:0/1888043009 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:26.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.009+0000 7f59a3fff640 1 --2- 172.21.15.169:0/1888043009 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4a3760080 0x55b4a377ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.009+0000 7f59a8e1d640 1 --2- 172.21.15.169:0/1888043009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a375f690 0x55b4a377a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.009+0000 7f59a961e640 1 -- 172.21.15.169:0/1888043009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a375f690 msgr2=0x55b4a377a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:26.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.009+0000 7f59a961e640 1 --2- 172.21.15.169:0/1888043009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a375f690 0x55b4a377a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.009+0000 7f59a961e640 1 -- 172.21.15.169:0/1888043009 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4a3760080 msgr2=0x55b4a377ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:26.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.009+0000 7f59a961e640 1 --2- 172.21.15.169:0/1888043009 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4a3760080 0x55b4a377ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.009+0000 7f59a961e640 1 -- 172.21.15.169:0/1888043009 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b4a34a37d0 con 0x55b4a37651f0 2024-03-28T01:04:26.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.010+0000 7f59a961e640 1 --2- 172.21.15.169:0/1888043009 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37651f0 0x55b4a3781e30 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f599c01a000 tx=0x7f599c00fa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe4cd0b35104055f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:26.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.010+0000 7f5983fff640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:04:26.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.010+0000 7f5983fff640 1 -- 172.21.15.169:0/1888043009 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c057060 con 0x55b4a37651f0 2024-03-28T01:04:26.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.010+0000 7f59adc6d740 1 -- 172.21.15.169:0/1888043009 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f599c00cc80 con 0x55b4a37651f0 2024-03-28T01:04:26.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.010+0000 7f5983fff640 1 -- 172.21.15.169:0/1888043009 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f599c004a40 con 0x55b4a37651f0 2024-03-28T01:04:26.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.011+0000 7f5983fff640 1 -- 172.21.15.169:0/1888043009 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f599c05e050 con 0x55b4a37651f0 2024-03-28T01:04:26.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.012+0000 7f59adc6d740 10 client.16977 fetch_fsmap learned FSMap version 25 2024-03-28T01:04:26.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.012+0000 7f59adc6d740 1 -- 172.21.15.169:0/1888043009 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55b4a3767e10 con 0x55b4a37651f0 2024-03-28T01:04:26.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.013+0000 7f5983fff640 1 -- 172.21.15.169:0/1888043009 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f599c00cc80 con 0x55b4a37651f0 2024-03-28T01:04:26.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.013+0000 7f5983fff640 1 -- 172.21.15.169:0/1888043009 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f599c00b040 con 0x55b4a37651f0 2024-03-28T01:04:26.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.013+0000 7f59adc6d740 10 client.16977 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:04:26.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.013+0000 7f59adc6d740 10 client.16977 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:04:26.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.013+0000 7f59adc6d740 20 client.16977 populate_metadata read hostname 'smithi169' 2024-03-28T01:04:26.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.013+0000 7f59adc6d740 1 --2- 172.21.15.169:0/1888043009 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b4a3769890 0x55b4a376bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.013+0000 7f59adc6d740 4 client.16977 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:04:26.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.013+0000 7f59adc6d740 1 -- 172.21.15.169:0/1888043009 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b4a36d2900 con 0x55b4a3769890 2024-03-28T01:04:26.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.017+0000 7f59a8e1d640 1 --2- 172.21.15.169:0/1888043009 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b4a3769890 0x55b4a376bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.028+0000 7f59a8e1d640 1 --2- 172.21.15.169:0/1888043009 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b4a3769890 0x55b4a376bcf0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1e86b085bf43f9b3 server_cookie=d2c713e2ccbc5150 in_seq=0 out_seq=0 2024-03-28T01:04:26.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.029+0000 7f5983fff640 10 client.16977 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:04:26.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.032+0000 7f5983fff640 1 -- 172.21.15.169:0/1888043009 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b4a36d2900 con 0x55b4a3769890 2024-03-28T01:04:26.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.032+0000 7f5983fff640 10 client.16977 handle_command_reply: tid=0 2024-03-28T01:04:26.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.033+0000 7f59adc6d740 1 client.16977 shutdown 2024-03-28T01:04:26.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.040+0000 7f59adc6d740 1 -- 172.21.15.169:0/1888043009 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37651f0 msgr2=0x55b4a3781e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.040+0000 7f59adc6d740 1 --2- 172.21.15.169:0/1888043009 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37651f0 0x55b4a3781e30 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f599c01a000 tx=0x7f599c00fa80 comp rx=0 tx=0).stop 2024-03-28T01:04:26.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.043+0000 7f59adc6d740 1 -- 172.21.15.169:0/1888043009 shutdown_connections 2024-03-28T01:04:26.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.043+0000 7f59adc6d740 1 --2- 172.21.15.169:0/1888043009 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b4a3769890 0x55b4a376bcf0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.043+0000 7f59adc6d740 1 --2- 172.21.15.169:0/1888043009 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37651f0 0x55b4a3781e30 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.043+0000 7f59adc6d740 1 --2- 172.21.15.169:0/1888043009 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4a3760080 0x55b4a377ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.043+0000 7f59adc6d740 1 --2- 172.21.15.169:0/1888043009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a375f690 0x55b4a377a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.043+0000 7f59adc6d740 1 -- 172.21.15.169:0/1888043009 >> 172.21.15.169:0/1888043009 conn(0x55b4a375ac00 msgr2=0x55b4a3759e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:26.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.044+0000 7f5983fff640 0 client.16977 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:04:26.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.045+0000 7f59adc6d740 1 -- 172.21.15.169:0/1888043009 shutdown_connections 2024-03-28T01:04:26.049 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.046+0000 7f59adc6d740 1 -- 172.21.15.169:0/1888043009 wait complete. 2024-03-28T01:04:26.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.048+0000 7f59adc6d740 20 client.16977 trim_cache size 0 max 16384 2024-03-28T01:04:26.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:25 smithi169 ceph-mon[32274]: pgmap v588: 129 pgs: 129 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 158 MiB/s rd, 333 MiB/s wr, 4.55k op/s 2024-03-28T01:04:26.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.077+0000 7f59adc6d740 1 Processor -- start 2024-03-28T01:04:26.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.078+0000 7f59adc6d740 1 -- start start 2024-03-28T01:04:26.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.078+0000 7f59adc6d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a375f690 0x55b4a377a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.078+0000 7f59adc6d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4a3760080 0x55b4a377ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.078+0000 7f59adc6d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37651f0 0x55b4a3781d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.078+0000 7f59adc6d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59900120d0 con 0x55b4a37651f0 2024-03-28T01:04:26.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.078+0000 7f59adc6d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599000fbd0 con 0x55b4a3760080 2024-03-28T01:04:26.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.078+0000 7f59adc6d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599000dc70 con 0x55b4a375f690 2024-03-28T01:04:26.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.079+0000 7f59a8e1d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a375f690 0x55b4a377a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.079+0000 7f59a8e1d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a375f690 0x55b4a377a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:49786/0 (socket says 172.21.15.169:49786) 2024-03-28T01:04:26.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.079+0000 7f59a8e1d640 1 -- 172.21.15.169:0/3353398851 learned_addr learned my addr 172.21.15.169:0/3353398851 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:26.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.080+0000 7f59a3fff640 1 --2- 172.21.15.169:0/3353398851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4a3760080 0x55b4a377ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.081+0000 7f59a8e1d640 1 -- 172.21.15.169:0/3353398851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4a3760080 msgr2=0x55b4a377ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:26.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.081+0000 7f59a8e1d640 1 --2- 172.21.15.169:0/3353398851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4a3760080 0x55b4a377ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.081+0000 7f59a8e1d640 1 -- 172.21.15.169:0/3353398851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37651f0 msgr2=0x55b4a3781d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:04:26.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.081+0000 7f59a8e1d640 1 --2- 172.21.15.169:0/3353398851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37651f0 0x55b4a3781d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.081+0000 7f59a8e1d640 1 -- 172.21.15.169:0/3353398851 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b4a3756670 con 0x55b4a375f690 2024-03-28T01:04:26.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.088+0000 7f59a8e1d640 1 --2- 172.21.15.169:0/3353398851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a375f690 0x55b4a377a920 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x55b4a3760af0 tx=0x7f599800b350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71e17de3ce9bccf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:26.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.092+0000 7f598a7fc640 1 -- 172.21.15.169:0/3353398851 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998012650 con 0x55b4a375f690 2024-03-28T01:04:26.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.092+0000 7f598a7fc640 1 -- 172.21.15.169:0/3353398851 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5998012cd0 con 0x55b4a375f690 2024-03-28T01:04:26.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.092+0000 7f598a7fc640 1 -- 172.21.15.169:0/3353398851 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998011d50 con 0x55b4a375f690 2024-03-28T01:04:26.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.093+0000 7f59adc6d740 1 -- 172.21.15.169:0/3353398851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a375f690 msgr2=0x55b4a377a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.093+0000 7f59adc6d740 1 --2- 172.21.15.169:0/3353398851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a375f690 0x55b4a377a920 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x55b4a3760af0 tx=0x7f599800b350 comp rx=0 tx=0).stop 2024-03-28T01:04:26.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.094+0000 7f59adc6d740 1 -- 172.21.15.169:0/3353398851 shutdown_connections 2024-03-28T01:04:26.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.094+0000 7f59adc6d740 1 --2- 172.21.15.169:0/3353398851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37651f0 0x55b4a3781d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.094+0000 7f59adc6d740 1 --2- 172.21.15.169:0/3353398851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4a3760080 0x55b4a377ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.094+0000 7f59adc6d740 1 --2- 172.21.15.169:0/3353398851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a375f690 0x55b4a377a920 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.094+0000 7f59adc6d740 1 -- 172.21.15.169:0/3353398851 >> 172.21.15.169:0/3353398851 conn(0x55b4a375ac00 msgr2=0x55b4a3755cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:26.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.095+0000 7f59adc6d740 1 -- 172.21.15.169:0/3353398851 shutdown_connections 2024-03-28T01:04:26.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.096+0000 7f59adc6d740 1 -- 172.21.15.169:0/3353398851 wait complete. 2024-03-28T01:04:26.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.097+0000 7f59adc6d740 1 -- start start 2024-03-28T01:04:26.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.097+0000 7f59adc6d740 1 Processor -- start 2024-03-28T01:04:26.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.101+0000 7f59adc6d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a3760080 0x55b4a377d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.101+0000 7f59adc6d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4a37651f0 0x55b4a377d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.101+0000 7f59adc6d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37804b0 0x55b4a3780930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.101+0000 7f59adc6d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599000e270 con 0x55b4a37804b0 2024-03-28T01:04:26.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.101+0000 7f59adc6d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599000f570 con 0x55b4a37651f0 2024-03-28T01:04:26.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.101+0000 7f59adc6d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599000e8b0 con 0x55b4a3760080 2024-03-28T01:04:26.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.101+0000 7f59adc6d740 10 client.0 _ll_register_callbacks cb 0x7f599c00f490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:04:26.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.103+0000 7f59a961e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37804b0 0x55b4a3780930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.103+0000 7f59a961e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37804b0 0x55b4a3780930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36230/0 (socket says 172.21.15.169:36230) 2024-03-28T01:04:26.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.103+0000 7f59a961e640 1 -- 172.21.15.169:0/506565485 learned_addr learned my addr 172.21.15.169:0/506565485 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:26.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.105+0000 7f59a8e1d640 1 --2- 172.21.15.169:0/506565485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a3760080 0x55b4a377d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.106+0000 7f59a961e640 1 -- 172.21.15.169:0/506565485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a3760080 msgr2=0x55b4a377d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:26.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.106+0000 7f59a961e640 1 --2- 172.21.15.169:0/506565485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a3760080 0x55b4a377d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.106+0000 7f59a961e640 1 -- 172.21.15.169:0/506565485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4a37651f0 msgr2=0x55b4a377d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:26.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.106+0000 7f59a961e640 1 --2- 172.21.15.169:0/506565485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4a37651f0 0x55b4a377d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.106+0000 7f59a961e640 1 -- 172.21.15.169:0/506565485 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b4a3780e70 con 0x55b4a37804b0 2024-03-28T01:04:26.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.106+0000 7f59a961e640 1 --2- 172.21.15.169:0/506565485 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37804b0 0x55b4a3780930 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f599c0183e0 tx=0x7f599c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bad3e0e86d7b682 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:26.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.106+0000 7f5983fff640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:04:26.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.107+0000 7f5983fff640 1 -- 172.21.15.169:0/506565485 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c057040 con 0x55b4a37804b0 2024-03-28T01:04:26.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.107+0000 7f59adc6d740 1 -- 172.21.15.169:0/506565485 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5998012cd0 con 0x55b4a37804b0 2024-03-28T01:04:26.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.107+0000 7f5983fff640 1 -- 172.21.15.169:0/506565485 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f599c00cc30 con 0x55b4a37804b0 2024-03-28T01:04:26.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.107+0000 7f5983fff640 1 -- 172.21.15.169:0/506565485 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f599c01d030 con 0x55b4a37804b0 2024-03-28T01:04:26.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.107+0000 7f59adc6d740 10 client.16980 fetch_fsmap learned FSMap version 25 2024-03-28T01:04:26.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.107+0000 7f59adc6d740 1 -- 172.21.15.169:0/506565485 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55b4a3782170 con 0x55b4a37804b0 2024-03-28T01:04:26.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.108+0000 7f5983fff640 1 -- 172.21.15.169:0/506565485 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5998012cd0 con 0x55b4a37804b0 2024-03-28T01:04:26.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.108+0000 7f5983fff640 1 -- 172.21.15.169:0/506565485 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f599c00b050 con 0x55b4a37804b0 2024-03-28T01:04:26.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.108+0000 7f59adc6d740 10 client.16980 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:04:26.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.108+0000 7f59adc6d740 10 client.16980 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:04:26.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.108+0000 7f59adc6d740 20 client.16980 populate_metadata read hostname 'smithi169' 2024-03-28T01:04:26.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.108+0000 7f59adc6d740 1 --2- 172.21.15.169:0/506565485 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b4a37d18d0 0x55b4a37d3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.108+0000 7f59adc6d740 4 client.16980 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:04:26.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.108+0000 7f59adc6d740 1 -- 172.21.15.169:0/506565485 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b4a36ccc50 con 0x55b4a37d18d0 2024-03-28T01:04:26.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.114+0000 7f59a8e1d640 1 --2- 172.21.15.169:0/506565485 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b4a37d18d0 0x55b4a37d3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.125+0000 7f59a8e1d640 1 --2- 172.21.15.169:0/506565485 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b4a37d18d0 0x55b4a37d3cf0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e8c9ef9a9fff132 server_cookie=63b83d641aae5e7 in_seq=0 out_seq=0 2024-03-28T01:04:26.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.125+0000 7f5983fff640 10 client.16980 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:04:26.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.131+0000 7f5983fff640 1 -- 172.21.15.169:0/506565485 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b4a36ccc50 con 0x55b4a37d18d0 2024-03-28T01:04:26.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.131+0000 7f5983fff640 10 client.16980 handle_command_reply: tid=0 2024-03-28T01:04:26.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.131+0000 7f59adc6d740 1 client.16980 shutdown 2024-03-28T01:04:26.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.133+0000 7f59adc6d740 1 -- 172.21.15.169:0/506565485 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37804b0 msgr2=0x55b4a3780930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.133+0000 7f59adc6d740 1 --2- 172.21.15.169:0/506565485 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37804b0 0x55b4a3780930 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f599c0183e0 tx=0x7f599c0095c0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.136+0000 7f59adc6d740 1 -- 172.21.15.169:0/506565485 shutdown_connections 2024-03-28T01:04:26.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.136+0000 7f59adc6d740 1 --2- 172.21.15.169:0/506565485 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55b4a37d18d0 0x55b4a37d3cf0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.136+0000 7f59adc6d740 1 --2- 172.21.15.169:0/506565485 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55b4a37804b0 0x55b4a3780930 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.136+0000 7f59adc6d740 1 --2- 172.21.15.169:0/506565485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4a37651f0 0x55b4a377d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.136+0000 7f59adc6d740 1 --2- 172.21.15.169:0/506565485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b4a3760080 0x55b4a377d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.136+0000 7f59adc6d740 1 -- 172.21.15.169:0/506565485 >> 172.21.15.169:0/506565485 conn(0x55b4a375ac00 msgr2=0x55b4a377a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:26.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.136+0000 7f5983fff640 0 client.16980 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:04:26.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.141+0000 7f59adc6d740 1 -- 172.21.15.169:0/506565485 shutdown_connections 2024-03-28T01:04:26.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.142+0000 7f59adc6d740 1 -- 172.21.15.169:0/506565485 wait complete. 2024-03-28T01:04:26.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.144+0000 7f59adc6d740 20 client.16980 trim_cache size 0 max 16384 2024-03-28T01:04:26.149 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:04:26.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.149+0000 7f59ab081640 1 -- 172.21.15.169:0/2028257193 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f598c0a2a00 msgr2=0x7f598c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.149+0000 7f59ab081640 1 --2- 172.21.15.169:0/2028257193 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f598c0a2a00 0x7f598c0a4ec0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f599c009f70 tx=0x7f599c018970 comp rx=0 tx=0).stop 2024-03-28T01:04:26.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.149+0000 7f59ab081640 1 -- 172.21.15.169:0/2028257193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59a41017d0 msgr2=0x7f59a41a6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.149+0000 7f59ab081640 1 --2- 172.21.15.169:0/2028257193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59a41017d0 0x7f59a41a6d90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5994000c00 tx=0x7f5994033690 comp rx=0 tx=0).stop 2024-03-28T01:04:26.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.150+0000 7f59ab081640 1 -- 172.21.15.169:0/2028257193 shutdown_connections 2024-03-28T01:04:26.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.150+0000 7f59ab081640 1 --2- 172.21.15.169:0/2028257193 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f598c0a2a00 0x7f598c0a4ec0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.150+0000 7f59ab081640 1 --2- 172.21.15.169:0/2028257193 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f59a4102190 0x7f59a41ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.150+0000 7f59ab081640 1 --2- 172.21.15.169:0/2028257193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59a41017d0 0x7f59a41a6d90 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.150+0000 7f59ab081640 1 --2- 172.21.15.169:0/2028257193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59a41005e0 0x7f59a41a6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.150+0000 7f59ab081640 1 -- 172.21.15.169:0/2028257193 >> 172.21.15.169:0/2028257193 conn(0x7f59a40fbb50 msgr2=0x7f59a4104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:26.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.150+0000 7f59ab081640 1 -- 172.21.15.169:0/2028257193 shutdown_connections 2024-03-28T01:04:26.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.153+0000 7f59ab081640 1 -- 172.21.15.169:0/2028257193 wait complete. 2024-03-28T01:04:26.186 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:04:26.186 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T01:04:26.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.374+0000 7f3770d49640 1 -- 172.21.15.169:0/3869768335 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f376c1019b0 msgr2=0x7f376c101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.374+0000 7f3770d49640 1 --2- 172.21.15.169:0/3869768335 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f376c1019b0 0x7f376c101dd0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f375c009900 tx=0x7f375c02bc30 comp rx=0 tx=0).stop 2024-03-28T01:04:26.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.375+0000 7f3770d49640 1 -- 172.21.15.169:0/3869768335 shutdown_connections 2024-03-28T01:04:26.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.375+0000 7f3770d49640 1 --2- 172.21.15.169:0/3869768335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f376c100f70 0x7f376c108810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.375+0000 7f3770d49640 1 --2- 172.21.15.169:0/3869768335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f376c1005b0 0x7f376c100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.375+0000 7f3770d49640 1 --2- 172.21.15.169:0/3869768335 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f376c1019b0 0x7f376c101dd0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.375+0000 7f3770d49640 1 -- 172.21.15.169:0/3869768335 >> 172.21.15.169:0/3869768335 conn(0x7f376c0fb0c0 msgr2=0x7f376c0fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:26.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.375+0000 7f3770d49640 1 -- 172.21.15.169:0/3869768335 shutdown_connections 2024-03-28T01:04:26.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.375+0000 7f3770d49640 1 -- 172.21.15.169:0/3869768335 wait complete. 2024-03-28T01:04:26.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.376+0000 7f3770d49640 1 Processor -- start 2024-03-28T01:04:26.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.376+0000 7f3770d49640 1 -- start start 2024-03-28T01:04:26.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.377+0000 7f3770d49640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f376c1005b0 0x7f376c19de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.377+0000 7f3770d49640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f376c100f70 0x7f376c19e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.377+0000 7f3770d49640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f376c1019b0 0x7f376c1a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.377+0000 7f3770d49640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c10af40 con 0x7f376c1005b0 2024-03-28T01:04:26.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.377+0000 7f3770d49640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c10adc0 con 0x7f376c1019b0 2024-03-28T01:04:26.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.377+0000 7f3770d49640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c10b0c0 con 0x7f376c100f70 2024-03-28T01:04:26.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.377+0000 7f376b7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f376c1005b0 0x7f376c19de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.377+0000 7f376bfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f376c1019b0 0x7f376c1a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.377+0000 7f376b7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f376c1005b0 0x7f376c19de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36254/0 (socket says 172.21.15.169:36254) 2024-03-28T01:04:26.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.377+0000 7f376b7fe640 1 -- 172.21.15.169:0/68202052 learned_addr learned my addr 172.21.15.169:0/68202052 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:26.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.377+0000 7f376affd640 1 --2- 172.21.15.169:0/68202052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f376c100f70 0x7f376c19e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.378+0000 7f376b7fe640 1 -- 172.21.15.169:0/68202052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f376c100f70 msgr2=0x7f376c19e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.378+0000 7f376b7fe640 1 --2- 172.21.15.169:0/68202052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f376c100f70 0x7f376c19e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.378+0000 7f376b7fe640 1 -- 172.21.15.169:0/68202052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f376c1019b0 msgr2=0x7f376c1a53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.378+0000 7f376b7fe640 1 --2- 172.21.15.169:0/68202052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f376c1019b0 0x7f376c1a53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.378+0000 7f376b7fe640 1 -- 172.21.15.169:0/68202052 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376c1a5a20 con 0x7f376c1005b0 2024-03-28T01:04:26.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.378+0000 7f376b7fe640 1 --2- 172.21.15.169:0/68202052 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f376c1005b0 0x7f376c19de00 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f375c000c00 tx=0x7f375c02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:26.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.379+0000 7f3768ff9640 1 -- 172.21.15.169:0/68202052 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c041070 con 0x7f376c1005b0 2024-03-28T01:04:26.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.380+0000 7f3768ff9640 1 -- 172.21.15.169:0/68202052 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f375c004450 con 0x7f376c1005b0 2024-03-28T01:04:26.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.380+0000 7f3768ff9640 1 -- 172.21.15.169:0/68202052 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c034440 con 0x7f376c1005b0 2024-03-28T01:04:26.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.380+0000 7f3770d49640 1 -- 172.21.15.169:0/68202052 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f376c1a5cb0 con 0x7f376c1005b0 2024-03-28T01:04:26.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.380+0000 7f3770d49640 1 -- 172.21.15.169:0/68202052 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f376c1a6150 con 0x7f376c1005b0 2024-03-28T01:04:26.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.384+0000 7f3774b8b740 1 Processor -- start 2024-03-28T01:04:26.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.384+0000 7f3774b8b740 1 -- start start 2024-03-28T01:04:26.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.384+0000 7f3774b8b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c748a0 0x55eb37c74cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.384+0000 7f3774b8b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eb37c75290 0x55eb37c79d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.384+0000 7f3774b8b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c7a400 0x55eb37c7c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.384+0000 7f3774b8b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb37c37610 con 0x55eb37c75290 2024-03-28T01:04:26.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.384+0000 7f3774b8b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb37bda3c0 con 0x55eb37c7a400 2024-03-28T01:04:26.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.384+0000 7f3774b8b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb37b82d60 con 0x55eb37c748a0 2024-03-28T01:04:26.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.384+0000 7f3768ff9640 1 -- 172.21.15.169:0/68202052 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f375c044070 con 0x7f376c1005b0 2024-03-28T01:04:26.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.384+0000 7f376bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c748a0 0x55eb37c74cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.385+0000 7f376bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c748a0 0x55eb37c74cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:49818/0 (socket says 172.21.15.169:49818) 2024-03-28T01:04:26.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.385+0000 7f376bfff640 1 -- 172.21.15.169:0/3450033039 learned_addr learned my addr 172.21.15.169:0/3450033039 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:26.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.385+0000 7f376affd640 1 --2- 172.21.15.169:0/3450033039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c7a400 0x55eb37c7c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.385+0000 7f376b7fe640 1 --2- 172.21.15.169:0/3450033039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eb37c75290 0x55eb37c79d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.385+0000 7f376affd640 1 -- 172.21.15.169:0/3450033039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c748a0 msgr2=0x55eb37c74cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:26.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.385+0000 7f376affd640 1 --2- 172.21.15.169:0/3450033039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c748a0 0x55eb37c74cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.385+0000 7f376affd640 1 -- 172.21.15.169:0/3450033039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eb37c75290 msgr2=0x55eb37c79d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:26.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.385+0000 7f376affd640 1 --2- 172.21.15.169:0/3450033039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eb37c75290 0x55eb37c79d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.385+0000 7f376affd640 1 -- 172.21.15.169:0/3450033039 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb378ee3f0 con 0x55eb37c7a400 2024-03-28T01:04:26.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.385+0000 7f376bfff640 1 --2- 172.21.15.169:0/3450033039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c748a0 0x55eb37c74cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:04:26.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.385+0000 7f376b7fe640 1 --2- 172.21.15.169:0/3450033039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eb37c75290 0x55eb37c79d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:04:26.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.386+0000 7f376affd640 1 --2- 172.21.15.169:0/3450033039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c7a400 0x55eb37c7c890 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3760009bf0 tx=0x7f3760011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1ac9c04efbf1002 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:26.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.386+0000 7f3768ff9640 1 --2- 172.21.15.169:0/68202052 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f37540a2950 0x7f37540a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.386+0000 7f3751ffb640 1 -- 172.21.15.169:0/3450033039 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376001e070 con 0x55eb37c7a400 2024-03-28T01:04:26.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.386+0000 7f3751ffb640 1 -- 172.21.15.169:0/3450033039 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3760009da0 con 0x55eb37c7a400 2024-03-28T01:04:26.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.386+0000 7f3751ffb640 1 -- 172.21.15.169:0/3450033039 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3760018870 con 0x55eb37c7a400 2024-03-28T01:04:26.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.386+0000 7f376bfff640 1 --2- 172.21.15.169:0/68202052 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f37540a2950 0x7f37540a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.387+0000 7f3774b8b740 1 -- 172.21.15.169:0/3450033039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c7a400 msgr2=0x55eb37c7c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.387+0000 7f3774b8b740 1 --2- 172.21.15.169:0/3450033039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c7a400 0x55eb37c7c890 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3760009bf0 tx=0x7f3760011810 comp rx=0 tx=0).stop 2024-03-28T01:04:26.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.387+0000 7f376bfff640 1 --2- 172.21.15.169:0/68202052 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f37540a2950 0x7f37540a4e10 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x55eb37c75af0 tx=0x7f3764009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:26.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.387+0000 7f3768ff9640 1 -- 172.21.15.169:0/68202052 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f375c0f0820 con 0x7f376c1005b0 2024-03-28T01:04:26.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.388+0000 7f3774b8b740 1 -- 172.21.15.169:0/3450033039 shutdown_connections 2024-03-28T01:04:26.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.388+0000 7f3774b8b740 1 --2- 172.21.15.169:0/3450033039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c7a400 0x55eb37c7c890 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.388+0000 7f3774b8b740 1 --2- 172.21.15.169:0/3450033039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eb37c75290 0x55eb37c79d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.388+0000 7f3774b8b740 1 --2- 172.21.15.169:0/3450033039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c748a0 0x55eb37c74cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.388+0000 7f3774b8b740 1 -- 172.21.15.169:0/3450033039 >> 172.21.15.169:0/3450033039 conn(0x55eb37c6fe10 msgr2=0x55eb37c70210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:26.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.389+0000 7f3774b8b740 1 -- 172.21.15.169:0/3450033039 shutdown_connections 2024-03-28T01:04:26.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.389+0000 7f3774b8b740 1 -- 172.21.15.169:0/3450033039 wait complete. 2024-03-28T01:04:26.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.390+0000 7f3774b8b740 1 -- start start 2024-03-28T01:04:26.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.390+0000 7f3774b8b740 1 Processor -- start 2024-03-28T01:04:26.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.391+0000 7f3774b8b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c748a0 0x55eb37c7ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.391+0000 7f3774b8b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eb37c75290 0x55eb37c7f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.391+0000 7f3774b8b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c7a400 0x55eb37c861f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.391+0000 7f3774b8b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb37b96240 con 0x55eb37c75290 2024-03-28T01:04:26.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.391+0000 7f376b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c748a0 0x55eb37c7ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.391+0000 7f3774b8b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb37bd6460 con 0x55eb37c748a0 2024-03-28T01:04:26.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.391+0000 7f3774b8b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb37c7d3b0 con 0x55eb37c7a400 2024-03-28T01:04:26.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.391+0000 7f376b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c748a0 0x55eb37c7ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:57642/0 (socket says 172.21.15.169:57642) 2024-03-28T01:04:26.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.391+0000 7f376b7fe640 1 -- 172.21.15.169:0/1514933948 learned_addr learned my addr 172.21.15.169:0/1514933948 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:26.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.391+0000 7f3774b8b740 10 client.0 _ll_register_callbacks cb 0x55eb37b95c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:04:26.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.391+0000 7f376bfff640 1 --2- 172.21.15.169:0/1514933948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c7a400 0x55eb37c861f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.391+0000 7f376affd640 1 --2- 172.21.15.169:0/1514933948 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eb37c75290 0x55eb37c7f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.392+0000 7f376bfff640 1 -- 172.21.15.169:0/1514933948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c748a0 msgr2=0x55eb37c7ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:26.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.392+0000 7f376bfff640 1 --2- 172.21.15.169:0/1514933948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c748a0 0x55eb37c7ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.392+0000 7f376bfff640 1 -- 172.21.15.169:0/1514933948 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eb37c75290 msgr2=0x55eb37c7f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:26.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.392+0000 7f376bfff640 1 --2- 172.21.15.169:0/1514933948 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eb37c75290 0x55eb37c7f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.392+0000 7f376bfff640 1 -- 172.21.15.169:0/1514933948 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eb379b87d0 con 0x55eb37c7a400 2024-03-28T01:04:26.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.393+0000 7f376bfff640 1 --2- 172.21.15.169:0/1514933948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c7a400 0x55eb37c861f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3764009fd0 tx=0x7f3764004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c9f1440cd598d36 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:26.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.393+0000 7f374affd640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:04:26.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.393+0000 7f374affd640 1 -- 172.21.15.169:0/1514933948 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3764004310 con 0x55eb37c7a400 2024-03-28T01:04:26.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.393+0000 7f374affd640 1 -- 172.21.15.169:0/1514933948 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f37640044b0 con 0x55eb37c7a400 2024-03-28T01:04:26.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.393+0000 7f3774b8b740 1 -- 172.21.15.169:0/1514933948 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3760009da0 con 0x55eb37c7a400 2024-03-28T01:04:26.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.394+0000 7f374affd640 1 -- 172.21.15.169:0/1514933948 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f37640046b0 con 0x55eb37c7a400 2024-03-28T01:04:26.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.394+0000 7f3774b8b740 10 client.26794 fetch_fsmap learned FSMap version 25 2024-03-28T01:04:26.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.394+0000 7f3774b8b740 1 -- 172.21.15.169:0/1514933948 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55eb37c89630 con 0x55eb37c7a400 2024-03-28T01:04:26.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.394+0000 7f374affd640 1 -- 172.21.15.169:0/1514933948 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3760009da0 con 0x55eb37c7a400 2024-03-28T01:04:26.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.394+0000 7f374affd640 1 -- 172.21.15.169:0/1514933948 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f3764026070 con 0x55eb37c7a400 2024-03-28T01:04:26.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.395+0000 7f3774b8b740 10 client.26794 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:04:26.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.395+0000 7f3774b8b740 10 client.26794 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:04:26.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.395+0000 7f3774b8b740 20 client.26794 populate_metadata read hostname 'smithi169' 2024-03-28T01:04:26.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.395+0000 7f3774b8b740 1 --2- 172.21.15.169:0/1514933948 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55eb37c8ad60 0x55eb37c8d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.395+0000 7f3774b8b740 4 client.26794 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:04:26.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.395+0000 7f3774b8b740 1 -- 172.21.15.169:0/1514933948 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eb37be7900 con 0x55eb37c8ad60 2024-03-28T01:04:26.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.395+0000 7f376b7fe640 1 --2- 172.21.15.169:0/1514933948 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55eb37c8ad60 0x55eb37c8d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.396+0000 7f376b7fe640 1 --2- 172.21.15.169:0/1514933948 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55eb37c8ad60 0x55eb37c8d1e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2ab3082374eb33dc server_cookie=e69ba9e950f51708 in_seq=0 out_seq=0 2024-03-28T01:04:26.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.396+0000 7f374affd640 10 client.26794 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:04:26.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.397+0000 7f374affd640 1 -- 172.21.15.169:0/1514933948 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55eb37be7900 con 0x55eb37c8ad60 2024-03-28T01:04:26.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.397+0000 7f374affd640 10 client.26794 handle_command_reply: tid=0 2024-03-28T01:04:26.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.397+0000 7f3774b8b740 1 client.26794 shutdown 2024-03-28T01:04:26.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.397+0000 7f3774b8b740 1 -- 172.21.15.169:0/1514933948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c7a400 msgr2=0x55eb37c861f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.397+0000 7f3774b8b740 1 --2- 172.21.15.169:0/1514933948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c7a400 0x55eb37c861f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3764009fd0 tx=0x7f3764004ab0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.398+0000 7f3774b8b740 1 -- 172.21.15.169:0/1514933948 shutdown_connections 2024-03-28T01:04:26.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.398+0000 7f3774b8b740 1 --2- 172.21.15.169:0/1514933948 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55eb37c8ad60 0x55eb37c8d1e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.398+0000 7f3774b8b740 1 --2- 172.21.15.169:0/1514933948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c7a400 0x55eb37c861f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.398+0000 7f3774b8b740 1 --2- 172.21.15.169:0/1514933948 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eb37c75290 0x55eb37c7f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.398+0000 7f374affd640 0 client.26794 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:04:26.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.398+0000 7f3774b8b740 1 --2- 172.21.15.169:0/1514933948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c748a0 0x55eb37c7ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.398+0000 7f3774b8b740 1 -- 172.21.15.169:0/1514933948 >> 172.21.15.169:0/1514933948 conn(0x55eb37c6fe10 msgr2=0x55eb37c6f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:26.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.398+0000 7f3774b8b740 1 -- 172.21.15.169:0/1514933948 shutdown_connections 2024-03-28T01:04:26.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.398+0000 7f3774b8b740 1 -- 172.21.15.169:0/1514933948 wait complete. 2024-03-28T01:04:26.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.398+0000 7f3774b8b740 20 client.26794 trim_cache size 0 max 16384 2024-03-28T01:04:26.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.438+0000 7f3774b8b740 1 Processor -- start 2024-03-28T01:04:26.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.438+0000 7f3774b8b740 1 -- start start 2024-03-28T01:04:26.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.443+0000 7f3774b8b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c748a0 0x55eb37c7d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.443+0000 7f3774b8b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eb37c75290 0x55eb37c7dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.443+0000 7f3774b8b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c7a400 0x55eb37c861b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.443+0000 7f3774b8b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c1b2b60 con 0x55eb37c75290 2024-03-28T01:04:26.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.443+0000 7f3774b8b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c1b0660 con 0x55eb37c748a0 2024-03-28T01:04:26.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.443+0000 7f3774b8b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c1ae700 con 0x55eb37c7a400 2024-03-28T01:04:26.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.444+0000 7f376bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c7a400 0x55eb37c861b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.444+0000 7f376bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c7a400 0x55eb37c861b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56480/0 (socket says 172.21.15.169:56480) 2024-03-28T01:04:26.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.444+0000 7f376bfff640 1 -- 172.21.15.169:0/3470156951 learned_addr learned my addr 172.21.15.169:0/3470156951 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:26.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.446+0000 7f376b7fe640 1 --2- 172.21.15.169:0/3470156951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c748a0 0x55eb37c7d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.446+0000 7f376bfff640 1 -- 172.21.15.169:0/3470156951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c748a0 msgr2=0x55eb37c7d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:26.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.446+0000 7f376bfff640 1 --2- 172.21.15.169:0/3470156951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c748a0 0x55eb37c7d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.446+0000 7f376bfff640 1 -- 172.21.15.169:0/3470156951 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eb37c75290 msgr2=0x55eb37c7dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:26.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.446+0000 7f376bfff640 1 --2- 172.21.15.169:0/3470156951 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eb37c75290 0x55eb37c7dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.446+0000 7f376bfff640 1 -- 172.21.15.169:0/3470156951 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb37c7fdf0 con 0x55eb37c7a400 2024-03-28T01:04:26.453 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.449+0000 7f376bfff640 1 --2- 172.21.15.169:0/3470156951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c7a400 0x55eb37c861b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3764005cc0 tx=0x7f376400f870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b6ea02c89f276bf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:26.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.457+0000 7f37517fa640 1 -- 172.21.15.169:0/3470156951 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3764028680 con 0x55eb37c7a400 2024-03-28T01:04:26.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.457+0000 7f37517fa640 1 -- 172.21.15.169:0/3470156951 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3764037020 con 0x55eb37c7a400 2024-03-28T01:04:26.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.457+0000 7f37517fa640 1 -- 172.21.15.169:0/3470156951 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3764028e20 con 0x55eb37c7a400 2024-03-28T01:04:26.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.460+0000 7f3774b8b740 1 -- 172.21.15.169:0/3470156951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c7a400 msgr2=0x55eb37c861b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.460+0000 7f3774b8b740 1 --2- 172.21.15.169:0/3470156951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c7a400 0x55eb37c861b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3764005cc0 tx=0x7f376400f870 comp rx=0 tx=0).stop 2024-03-28T01:04:26.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.467+0000 7f3774b8b740 1 -- 172.21.15.169:0/3470156951 shutdown_connections 2024-03-28T01:04:26.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.467+0000 7f3774b8b740 1 --2- 172.21.15.169:0/3470156951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c7a400 0x55eb37c861b0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.467+0000 7f3774b8b740 1 --2- 172.21.15.169:0/3470156951 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eb37c75290 0x55eb37c7dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.467+0000 7f3774b8b740 1 --2- 172.21.15.169:0/3470156951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c748a0 0x55eb37c7d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.467+0000 7f3774b8b740 1 -- 172.21.15.169:0/3470156951 >> 172.21.15.169:0/3470156951 conn(0x55eb37c6fe10 msgr2=0x55eb37c7ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:26.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.473+0000 7f3774b8b740 1 -- 172.21.15.169:0/3470156951 shutdown_connections 2024-03-28T01:04:26.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.477+0000 7f3774b8b740 1 -- 172.21.15.169:0/3470156951 wait complete. 2024-03-28T01:04:26.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.477+0000 7f3774b8b740 1 -- start start 2024-03-28T01:04:26.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.478+0000 7f3774b8b740 1 Processor -- start 2024-03-28T01:04:26.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.478+0000 7f3774b8b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eb37c748a0 0x55eb37c83930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.479+0000 7f3774b8b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c75290 0x55eb37c83e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.479+0000 7f3774b8b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c7a400 0x55eb37c84e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.479+0000 7f3774b8b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c1aed00 con 0x55eb37c748a0 2024-03-28T01:04:26.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.479+0000 7f3774b8b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c1b0000 con 0x55eb37c7a400 2024-03-28T01:04:26.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.479+0000 7f3774b8b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c1af340 con 0x55eb37c75290 2024-03-28T01:04:26.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.479+0000 7f376affd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c75290 0x55eb37c83e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.479+0000 7f376affd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c75290 0x55eb37c83e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56496/0 (socket says 172.21.15.169:56496) 2024-03-28T01:04:26.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.479+0000 7f376affd640 1 -- 172.21.15.169:0/3536112024 learned_addr learned my addr 172.21.15.169:0/3536112024 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:26.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.479+0000 7f376bfff640 1 --2- 172.21.15.169:0/3536112024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c7a400 0x55eb37c84e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.480+0000 7f3774b8b740 10 client.0 _ll_register_callbacks cb 0x7f3764003a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:04:26.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.484+0000 7f376bfff640 1 -- 172.21.15.169:0/3536112024 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c75290 msgr2=0x55eb37c83e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:26.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.484+0000 7f376bfff640 1 --2- 172.21.15.169:0/3536112024 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c75290 0x55eb37c83e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.484+0000 7f376bfff640 1 -- 172.21.15.169:0/3536112024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eb37c748a0 msgr2=0x55eb37c83930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:26.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.484+0000 7f376bfff640 1 --2- 172.21.15.169:0/3536112024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eb37c748a0 0x55eb37c83930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.484+0000 7f376bfff640 1 -- 172.21.15.169:0/3536112024 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eb37c85470 con 0x55eb37c7a400 2024-03-28T01:04:26.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.487+0000 7f376bfff640 1 --2- 172.21.15.169:0/3536112024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c7a400 0x55eb37c84e80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3764018040 tx=0x7f37640288b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb86e9b15e98c311 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:26.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.489+0000 7f374affd640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:04:26.497 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.493+0000 7f374affd640 1 -- 172.21.15.169:0/3536112024 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376403f020 con 0x55eb37c7a400 2024-03-28T01:04:26.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.493+0000 7f374affd640 1 -- 172.21.15.169:0/3536112024 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f3764004a60 con 0x55eb37c7a400 2024-03-28T01:04:26.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.495+0000 7f3774b8b740 1 -- 172.21.15.169:0/3536112024 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3764037020 con 0x55eb37c7a400 2024-03-28T01:04:26.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.501+0000 7f374affd640 1 -- 172.21.15.169:0/3536112024 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3764022040 con 0x55eb37c7a400 2024-03-28T01:04:26.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.501+0000 7f3774b8b740 10 client.26861 fetch_fsmap learned FSMap version 25 2024-03-28T01:04:26.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.501+0000 7f3774b8b740 1 -- 172.21.15.169:0/3536112024 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55eb37ce6670 con 0x55eb37c7a400 2024-03-28T01:04:26.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.505+0000 7f374affd640 1 -- 172.21.15.169:0/3536112024 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3764004c80 con 0x55eb37c7a400 2024-03-28T01:04:26.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.506+0000 7f374affd640 1 -- 172.21.15.169:0/3536112024 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f3764026070 con 0x55eb37c7a400 2024-03-28T01:04:26.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.506+0000 7f3774b8b740 10 client.26861 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:04:26.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.506+0000 7f3774b8b740 10 client.26861 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:04:26.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.506+0000 7f3774b8b740 20 client.26861 populate_metadata read hostname 'smithi169' 2024-03-28T01:04:26.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.506+0000 7f3774b8b740 1 --2- 172.21.15.169:0/3536112024 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55eb37c8ad60 0x55eb37ce8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.506+0000 7f3774b8b740 4 client.26861 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:04:26.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.506+0000 7f3774b8b740 1 -- 172.21.15.169:0/3536112024 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f3764037020 con 0x55eb37c8ad60 2024-03-28T01:04:26.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.507+0000 7f376b7fe640 1 --2- 172.21.15.169:0/3536112024 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55eb37c8ad60 0x55eb37ce8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.515+0000 7f374affd640 10 client.26861 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:04:26.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.515+0000 7f376b7fe640 1 --2- 172.21.15.169:0/3536112024 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55eb37c8ad60 0x55eb37ce8e40 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9d9578a589f43f2c server_cookie=e7dc728cd45ac214 in_seq=0 out_seq=0 2024-03-28T01:04:26.519 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.516+0000 7f374affd640 1 -- 172.21.15.169:0/3536112024 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f3764037020 con 0x55eb37c8ad60 2024-03-28T01:04:26.519 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.517+0000 7f374affd640 10 client.26861 handle_command_reply: tid=0 2024-03-28T01:04:26.520 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.517+0000 7f3774b8b740 1 client.26861 shutdown 2024-03-28T01:04:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.522+0000 7f3774b8b740 1 -- 172.21.15.169:0/3536112024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c7a400 msgr2=0x55eb37c84e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.522+0000 7f3774b8b740 1 --2- 172.21.15.169:0/3536112024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c7a400 0x55eb37c84e80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3764018040 tx=0x7f37640288b0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.527+0000 7f3774b8b740 1 -- 172.21.15.169:0/3536112024 shutdown_connections 2024-03-28T01:04:26.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.527+0000 7f3774b8b740 1 --2- 172.21.15.169:0/3536112024 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55eb37c8ad60 0x55eb37ce8e40 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.527+0000 7f3774b8b740 1 --2- 172.21.15.169:0/3536112024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55eb37c7a400 0x55eb37c84e80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.527+0000 7f3774b8b740 1 --2- 172.21.15.169:0/3536112024 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb37c75290 0x55eb37c83e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.527+0000 7f3774b8b740 1 --2- 172.21.15.169:0/3536112024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55eb37c748a0 0x55eb37c83930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.527+0000 7f3774b8b740 1 -- 172.21.15.169:0/3536112024 >> 172.21.15.169:0/3536112024 conn(0x55eb37c6fe10 msgr2=0x55eb37c7ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:26.531 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.528+0000 7f374affd640 0 client.26861 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:04:26.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.529+0000 7f3774b8b740 1 -- 172.21.15.169:0/3536112024 shutdown_connections 2024-03-28T01:04:26.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.531+0000 7f3774b8b740 1 -- 172.21.15.169:0/3536112024 wait complete. 2024-03-28T01:04:26.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.533+0000 7f3774b8b740 20 client.26861 trim_cache size 0 max 16384 2024-03-28T01:04:26.539 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:04:26.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.539+0000 7f3770d49640 1 -- 172.21.15.169:0/68202052 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f37540a2950 msgr2=0x7f37540a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.539+0000 7f3770d49640 1 --2- 172.21.15.169:0/68202052 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f37540a2950 0x7f37540a4e10 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x55eb37c75af0 tx=0x7f3764009500 comp rx=0 tx=0).stop 2024-03-28T01:04:26.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.539+0000 7f3770d49640 1 -- 172.21.15.169:0/68202052 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f376c1005b0 msgr2=0x7f376c19de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.539+0000 7f3770d49640 1 --2- 172.21.15.169:0/68202052 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f376c1005b0 0x7f376c19de00 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f375c000c00 tx=0x7f375c02c640 comp rx=0 tx=0).stop 2024-03-28T01:04:26.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.541+0000 7f3770d49640 1 -- 172.21.15.169:0/68202052 shutdown_connections 2024-03-28T01:04:26.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.541+0000 7f3770d49640 1 --2- 172.21.15.169:0/68202052 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f37540a2950 0x7f37540a4e10 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.541+0000 7f3770d49640 1 --2- 172.21.15.169:0/68202052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f376c1019b0 0x7f376c1a53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.541+0000 7f3770d49640 1 --2- 172.21.15.169:0/68202052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f376c100f70 0x7f376c19e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.541+0000 7f3770d49640 1 --2- 172.21.15.169:0/68202052 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f376c1005b0 0x7f376c19de00 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.541+0000 7f3770d49640 1 -- 172.21.15.169:0/68202052 >> 172.21.15.169:0/68202052 conn(0x7f376c0fb0c0 msgr2=0x7f376c10d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:26.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.545+0000 7f3770d49640 1 -- 172.21.15.169:0/68202052 shutdown_connections 2024-03-28T01:04:26.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.546+0000 7f3770d49640 1 -- 172.21.15.169:0/68202052 wait complete. 2024-03-28T01:04:26.579 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:04:26.579 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T01:04:26.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.780+0000 7f9e382ac640 1 -- 172.21.15.169:0/3290440943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e30107380 msgr2=0x7f9e301097d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.780+0000 7f9e382ac640 1 --2- 172.21.15.169:0/3290440943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e30107380 0x7f9e301097d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9e28009be0 tx=0x7f9e2802bef0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.780+0000 7f9e382ac640 1 -- 172.21.15.169:0/3290440943 shutdown_connections 2024-03-28T01:04:26.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.780+0000 7f9e382ac640 1 --2- 172.21.15.169:0/3290440943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e30107380 0x7f9e301097d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.780+0000 7f9e382ac640 1 --2- 172.21.15.169:0/3290440943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e30104a50 0x7f9e30106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.780+0000 7f9e382ac640 1 --2- 172.21.15.169:0/3290440943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e30102120 0x7f9e30104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.780+0000 7f9e382ac640 1 -- 172.21.15.169:0/3290440943 >> 172.21.15.169:0/3290440943 conn(0x7f9e300fbb30 msgr2=0x7f9e300fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:26.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.780+0000 7f9e382ac640 1 -- 172.21.15.169:0/3290440943 shutdown_connections 2024-03-28T01:04:26.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.780+0000 7f9e382ac640 1 -- 172.21.15.169:0/3290440943 wait complete. 2024-03-28T01:04:26.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.781+0000 7f9e382ac640 1 Processor -- start 2024-03-28T01:04:26.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.781+0000 7f9e382ac640 1 -- start start 2024-03-28T01:04:26.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.781+0000 7f9e382ac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e30102120 0x7f9e301957a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.781+0000 7f9e382ac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e30104a50 0x7f9e30195ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.781+0000 7f9e382ac640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e30107380 0x7f9e301a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.781+0000 7f9e382ac640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3010d320 con 0x7f9e30102120 2024-03-28T01:04:26.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.781+0000 7f9e382ac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3010d1a0 con 0x7f9e30104a50 2024-03-28T01:04:26.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.781+0000 7f9e382ac640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3010d4a0 con 0x7f9e30107380 2024-03-28T01:04:26.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.781+0000 7f9e35847640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e30104a50 0x7f9e30195ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.781+0000 7f9e35847640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e30104a50 0x7f9e30195ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:47104/0 (socket says 172.21.15.169:47104) 2024-03-28T01:04:26.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.781+0000 7f9e35847640 1 -- 172.21.15.169:0/4030446007 learned_addr learned my addr 172.21.15.169:0/4030446007 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:26.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.781+0000 7f9e36048640 1 --2- 172.21.15.169:0/4030446007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e30102120 0x7f9e301957a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.782+0000 7f9e36849640 1 --2- 172.21.15.169:0/4030446007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e30107380 0x7f9e301a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.782+0000 7f9e35847640 1 -- 172.21.15.169:0/4030446007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e30107380 msgr2=0x7f9e301a5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.782+0000 7f9e35847640 1 --2- 172.21.15.169:0/4030446007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e30107380 0x7f9e301a5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.782+0000 7f9e35847640 1 -- 172.21.15.169:0/4030446007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e30102120 msgr2=0x7f9e301957a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.782+0000 7f9e35847640 1 --2- 172.21.15.169:0/4030446007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e30102120 0x7f9e301957a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.782+0000 7f9e35847640 1 -- 172.21.15.169:0/4030446007 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e3006b110 con 0x7f9e30104a50 2024-03-28T01:04:26.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.782+0000 7f9e35847640 1 --2- 172.21.15.169:0/4030446007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e30104a50 0x7f9e30195ce0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c00e9c0 tx=0x7f9e2c00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:26.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.782+0000 7f9e277fe640 1 -- 172.21.15.169:0/4030446007 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c00ccc0 con 0x7f9e30104a50 2024-03-28T01:04:26.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.782+0000 7f9e277fe640 1 -- 172.21.15.169:0/4030446007 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e2c005500 con 0x7f9e30104a50 2024-03-28T01:04:26.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.782+0000 7f9e277fe640 1 -- 172.21.15.169:0/4030446007 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c010430 con 0x7f9e30104a50 2024-03-28T01:04:26.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.782+0000 7f9e382ac640 1 -- 172.21.15.169:0/4030446007 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e3006b400 con 0x7f9e30104a50 2024-03-28T01:04:26.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.782+0000 7f9e382ac640 1 -- 172.21.15.169:0/4030446007 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e3006b8a0 con 0x7f9e30104a50 2024-03-28T01:04:26.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.786+0000 7f9e277fe640 1 -- 172.21.15.169:0/4030446007 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9e2c005000 con 0x7f9e30104a50 2024-03-28T01:04:26.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.786+0000 7f9e3af73740 1 Processor -- start 2024-03-28T01:04:26.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.786+0000 7f9e3af73740 1 -- start start 2024-03-28T01:04:26.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.786+0000 7f9e3af73740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d0e690 0x5653b2d0eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.786+0000 7f9e3af73740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d0f080 0x5653b2d13b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.787+0000 7f9e3af73740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d141f0 0x5653b2d16680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.787+0000 7f9e3af73740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653b2cd1610 con 0x5653b2d0f080 2024-03-28T01:04:26.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.787+0000 7f9e3af73740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653b2c743c0 con 0x5653b2d141f0 2024-03-28T01:04:26.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.787+0000 7f9e3af73740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653b2c1cd60 con 0x5653b2d0e690 2024-03-28T01:04:26.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.787+0000 7f9e277fe640 1 --2- 172.21.15.169:0/4030446007 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9e1c0a2880 0x7f9e1c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.787+0000 7f9e35847640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d141f0 0x5653b2d16680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.787+0000 7f9e36048640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d0f080 0x5653b2d13b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.787+0000 7f9e36849640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d0e690 0x5653b2d0eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.787+0000 7f9e36849640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d0e690 0x5653b2d0eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56522/0 (socket says 172.21.15.169:56522) 2024-03-28T01:04:26.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.787+0000 7f9e35847640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d141f0 0x5653b2d16680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:47110/0 (socket says 172.21.15.169:47110) 2024-03-28T01:04:26.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.787+0000 7f9e36048640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d0f080 0x5653b2d13b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39208/0 (socket says 172.21.15.169:39208) 2024-03-28T01:04:26.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.787+0000 7f9e36849640 1 -- 172.21.15.169:0/1601377279 learned_addr learned my addr 172.21.15.169:0/1601377279 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:26.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.787+0000 7f9e36849640 1 --2- 172.21.15.169:0/4030446007 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9e1c0a2880 0x7f9e1c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.792 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.787+0000 7f9e277fe640 1 -- 172.21.15.169:0/4030446007 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f9e2c0cf170 con 0x7f9e30104a50 2024-03-28T01:04:26.792 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.788+0000 7f9e36849640 1 -- 172.21.15.169:0/1601377279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d141f0 msgr2=0x5653b2d16680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:26.792 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.788+0000 7f9e36849640 1 --2- 172.21.15.169:0/1601377279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d141f0 0x5653b2d16680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.792 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.788+0000 7f9e36849640 1 -- 172.21.15.169:0/1601377279 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d0f080 msgr2=0x5653b2d13b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:26.792 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.788+0000 7f9e36849640 1 --2- 172.21.15.169:0/1601377279 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d0f080 0x5653b2d13b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.792 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.788+0000 7f9e36849640 1 -- 172.21.15.169:0/1601377279 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653b29883f0 con 0x5653b2d0e690 2024-03-28T01:04:26.792 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.788+0000 7f9e36849640 1 --2- 172.21.15.169:0/1601377279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d0e690 0x5653b2d0eab0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9e28009370 tx=0x7f9e28033850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3577874f9f06c27 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:26.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.788+0000 7f9e36849640 1 --2- 172.21.15.169:0/4030446007 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9e1c0a2880 0x7f9e1c0a4d40 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9e28004ab0 tx=0x7f9e28043000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:26.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.788+0000 7f9e35847640 1 --2- 172.21.15.169:0/1601377279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d141f0 0x5653b2d16680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:04:26.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.789+0000 7f9e24bfe640 1 -- 172.21.15.169:0/1601377279 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2802cc60 con 0x5653b2d0e690 2024-03-28T01:04:26.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.789+0000 7f9e24bfe640 1 -- 172.21.15.169:0/1601377279 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e28051070 con 0x5653b2d0e690 2024-03-28T01:04:26.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.789+0000 7f9e24bfe640 1 -- 172.21.15.169:0/1601377279 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e28004820 con 0x5653b2d0e690 2024-03-28T01:04:26.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.791+0000 7f9e3af73740 1 -- 172.21.15.169:0/1601377279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d0e690 msgr2=0x5653b2d0eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.794 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.791+0000 7f9e3af73740 1 --2- 172.21.15.169:0/1601377279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d0e690 0x5653b2d0eab0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9e28009370 tx=0x7f9e28033850 comp rx=0 tx=0).stop 2024-03-28T01:04:26.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.792+0000 7f9e3af73740 1 -- 172.21.15.169:0/1601377279 shutdown_connections 2024-03-28T01:04:26.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.792+0000 7f9e3af73740 1 --2- 172.21.15.169:0/1601377279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d141f0 0x5653b2d16680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.792+0000 7f9e3af73740 1 --2- 172.21.15.169:0/1601377279 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d0f080 0x5653b2d13b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.792+0000 7f9e3af73740 1 --2- 172.21.15.169:0/1601377279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d0e690 0x5653b2d0eab0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.792+0000 7f9e3af73740 1 -- 172.21.15.169:0/1601377279 >> 172.21.15.169:0/1601377279 conn(0x5653b2d09c00 msgr2=0x5653b2d0a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:26.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.792+0000 7f9e3af73740 1 -- 172.21.15.169:0/1601377279 shutdown_connections 2024-03-28T01:04:26.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.792+0000 7f9e3af73740 1 -- 172.21.15.169:0/1601377279 wait complete. 2024-03-28T01:04:26.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.793+0000 7f9e3af73740 1 -- start start 2024-03-28T01:04:26.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.793+0000 7f9e3af73740 1 Processor -- start 2024-03-28T01:04:26.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.793+0000 7f9e3af73740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d0e690 0x5653b2d18ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.793+0000 7f9e3af73740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d0f080 0x5653b2d19020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.793+0000 7f9e3af73740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d141f0 0x5653b2d20060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.793+0000 7f9e3af73740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653b2c30240 con 0x5653b2d141f0 2024-03-28T01:04:26.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.793+0000 7f9e3af73740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653b2c70460 con 0x5653b2d0f080 2024-03-28T01:04:26.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.793+0000 7f9e3af73740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653b2d17140 con 0x5653b2d0e690 2024-03-28T01:04:26.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.793+0000 7f9e3af73740 10 client.0 _ll_register_callbacks cb 0x5653b2c2fc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:04:26.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.793+0000 7f9e35847640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d0f080 0x5653b2d19020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.793+0000 7f9e35847640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d0f080 0x5653b2d19020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:47126/0 (socket says 172.21.15.169:47126) 2024-03-28T01:04:26.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.793+0000 7f9e35847640 1 -- 172.21.15.169:0/641125649 learned_addr learned my addr 172.21.15.169:0/641125649 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:26.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.793+0000 7f9e36048640 1 --2- 172.21.15.169:0/641125649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d0e690 0x5653b2d18ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.793+0000 7f9e36849640 1 --2- 172.21.15.169:0/641125649 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d141f0 0x5653b2d20060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.794+0000 7f9e35847640 1 -- 172.21.15.169:0/641125649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d0e690 msgr2=0x5653b2d18ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:26.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.794+0000 7f9e35847640 1 --2- 172.21.15.169:0/641125649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d0e690 0x5653b2d18ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.794+0000 7f9e35847640 1 -- 172.21.15.169:0/641125649 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d141f0 msgr2=0x5653b2d20060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:26.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.794+0000 7f9e35847640 1 --2- 172.21.15.169:0/641125649 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d141f0 0x5653b2d20060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.794+0000 7f9e35847640 1 -- 172.21.15.169:0/641125649 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5653b2a527d0 con 0x5653b2d0f080 2024-03-28T01:04:26.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.794+0000 7f9e35847640 1 --2- 172.21.15.169:0/641125649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d0f080 0x5653b2d19020 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c00cfd0 tx=0x7f9e2c004a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91ffc78b170e307b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:26.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.794+0000 7f9e19ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:04:26.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.794+0000 7f9e19ffb640 1 -- 172.21.15.169:0/641125649 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c09b770 con 0x5653b2d0f080 2024-03-28T01:04:26.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.794+0000 7f9e19ffb640 1 -- 172.21.15.169:0/641125649 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f9e2c086070 con 0x5653b2d0f080 2024-03-28T01:04:26.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.794+0000 7f9e3af73740 1 -- 172.21.15.169:0/641125649 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9e28051070 con 0x5653b2d0f080 2024-03-28T01:04:26.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.795+0000 7f9e19ffb640 1 -- 172.21.15.169:0/641125649 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e2c09f8c0 con 0x5653b2d0f080 2024-03-28T01:04:26.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.795+0000 7f9e3af73740 10 client.26873 fetch_fsmap learned FSMap version 25 2024-03-28T01:04:26.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.795+0000 7f9e3af73740 1 -- 172.21.15.169:0/641125649 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5653b2d23490 con 0x5653b2d0f080 2024-03-28T01:04:26.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.795+0000 7f9e19ffb640 1 -- 172.21.15.169:0/641125649 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e28051070 con 0x5653b2d0f080 2024-03-28T01:04:26.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.795+0000 7f9e19ffb640 1 -- 172.21.15.169:0/641125649 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9e2c0a3a80 con 0x5653b2d0f080 2024-03-28T01:04:26.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.796+0000 7f9e3af73740 10 client.26873 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:04:26.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.796+0000 7f9e3af73740 10 client.26873 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:04:26.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.796+0000 7f9e3af73740 20 client.26873 populate_metadata read hostname 'smithi169' 2024-03-28T01:04:26.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.796+0000 7f9e3af73740 1 --2- 172.21.15.169:0/641125649 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5653b2d24b20 0x5653b2d26f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.796+0000 7f9e3af73740 4 client.26873 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:04:26.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.796+0000 7f9e3af73740 1 -- 172.21.15.169:0/641125649 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5653b2c81900 con 0x5653b2d24b20 2024-03-28T01:04:26.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.796+0000 7f9e36048640 1 --2- 172.21.15.169:0/641125649 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5653b2d24b20 0x5653b2d26f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.796+0000 7f9e19ffb640 10 client.26873 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:26.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.797+0000 7f9e36048640 1 --2- 172.21.15.169:0/641125649 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5653b2d24b20 0x5653b2d26f10 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78a5d3153af4c8b7 server_cookie=c9f1e5d9288ddb5c in_seq=0 out_seq=0 2024-03-28T01:04:26.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.798+0000 7f9e19ffb640 1 -- 172.21.15.169:0/641125649 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5653b2c81900 con 0x5653b2d24b20 2024-03-28T01:04:26.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.798+0000 7f9e19ffb640 10 client.26873 handle_command_reply: tid=0 2024-03-28T01:04:26.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.798+0000 7f9e3af73740 1 client.26873 shutdown 2024-03-28T01:04:26.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.798+0000 7f9e3af73740 1 -- 172.21.15.169:0/641125649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d0f080 msgr2=0x5653b2d19020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.798+0000 7f9e3af73740 1 --2- 172.21.15.169:0/641125649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d0f080 0x5653b2d19020 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c00cfd0 tx=0x7f9e2c004a70 comp rx=0 tx=0).stop 2024-03-28T01:04:26.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.799+0000 7f9e3af73740 1 -- 172.21.15.169:0/641125649 shutdown_connections 2024-03-28T01:04:26.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.799+0000 7f9e3af73740 1 --2- 172.21.15.169:0/641125649 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5653b2d24b20 0x5653b2d26f10 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.799+0000 7f9e3af73740 1 --2- 172.21.15.169:0/641125649 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d141f0 0x5653b2d20060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.799+0000 7f9e3af73740 1 --2- 172.21.15.169:0/641125649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d0f080 0x5653b2d19020 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.799+0000 7f9e3af73740 1 --2- 172.21.15.169:0/641125649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d0e690 0x5653b2d18ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.799+0000 7f9e3af73740 1 -- 172.21.15.169:0/641125649 >> 172.21.15.169:0/641125649 conn(0x5653b2d09c00 msgr2=0x5653b2d08e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:26.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.799+0000 7f9e19ffb640 0 client.26873 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:26.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.799+0000 7f9e3af73740 1 -- 172.21.15.169:0/641125649 shutdown_connections 2024-03-28T01:04:26.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.799+0000 7f9e3af73740 1 -- 172.21.15.169:0/641125649 wait complete. 2024-03-28T01:04:26.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.800+0000 7f9e3af73740 20 client.26873 trim_cache size 0 max 16384 2024-03-28T01:04:26.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.826+0000 7f9e3af73740 1 Processor -- start 2024-03-28T01:04:26.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.826+0000 7f9e3af73740 1 -- start start 2024-03-28T01:04:26.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.826+0000 7f9e3af73740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d0e690 0x5653b2d04cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.826+0000 7f9e3af73740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d0f080 0x5653b2d20020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.826+0000 7f9e3af73740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d141f0 0x5653b2d08590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.826+0000 7f9e35847640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d0f080 0x5653b2d20020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.827+0000 7f9e35847640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d0f080 0x5653b2d20020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:47138/0 (socket says 172.21.15.169:47138) 2024-03-28T01:04:26.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.827+0000 7f9e35847640 1 -- 172.21.15.169:0/1770685013 learned_addr learned my addr 172.21.15.169:0/1770685013 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:26.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.827+0000 7f9e3af73740 1 -- 172.21.15.169:0/1770685013 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e301bb570 con 0x5653b2d0e690 2024-03-28T01:04:26.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.827+0000 7f9e3af73740 1 -- 172.21.15.169:0/1770685013 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e301b9070 con 0x5653b2d0f080 2024-03-28T01:04:26.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.827+0000 7f9e3af73740 1 -- 172.21.15.169:0/1770685013 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e30073520 con 0x5653b2d141f0 2024-03-28T01:04:26.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.827+0000 7f9e36849640 1 --2- 172.21.15.169:0/1770685013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d141f0 0x5653b2d08590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.827+0000 7f9e36048640 1 --2- 172.21.15.169:0/1770685013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d0e690 0x5653b2d04cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.827+0000 7f9e36849640 1 -- 172.21.15.169:0/1770685013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d0f080 msgr2=0x5653b2d20020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:26.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.827+0000 7f9e36849640 1 --2- 172.21.15.169:0/1770685013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d0f080 0x5653b2d20020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.827+0000 7f9e36849640 1 -- 172.21.15.169:0/1770685013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d0e690 msgr2=0x5653b2d04cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:26.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.827+0000 7f9e36849640 1 --2- 172.21.15.169:0/1770685013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d0e690 0x5653b2d04cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.827+0000 7f9e36849640 1 -- 172.21.15.169:0/1770685013 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653b2d05480 con 0x5653b2d141f0 2024-03-28T01:04:26.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.827+0000 7f9e36849640 1 --2- 172.21.15.169:0/1770685013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d141f0 0x5653b2d08590 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9e280099a0 tx=0x7f9e28004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecb9367bb46eb727 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:26.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.828+0000 7f9e1bfff640 1 -- 172.21.15.169:0/1770685013 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e28033c10 con 0x5653b2d141f0 2024-03-28T01:04:26.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.828+0000 7f9e1bfff640 1 -- 172.21.15.169:0/1770685013 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e28051040 con 0x5653b2d141f0 2024-03-28T01:04:26.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.828+0000 7f9e1bfff640 1 -- 172.21.15.169:0/1770685013 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e280344d0 con 0x5653b2d141f0 2024-03-28T01:04:26.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.829+0000 7f9e3af73740 1 -- 172.21.15.169:0/1770685013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d141f0 msgr2=0x5653b2d08590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.829+0000 7f9e3af73740 1 --2- 172.21.15.169:0/1770685013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d141f0 0x5653b2d08590 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9e280099a0 tx=0x7f9e28004a40 comp rx=0 tx=0).stop 2024-03-28T01:04:26.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.829+0000 7f9e3af73740 1 -- 172.21.15.169:0/1770685013 shutdown_connections 2024-03-28T01:04:26.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.829+0000 7f9e3af73740 1 --2- 172.21.15.169:0/1770685013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d141f0 0x5653b2d08590 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.829+0000 7f9e3af73740 1 --2- 172.21.15.169:0/1770685013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d0f080 0x5653b2d20020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.829+0000 7f9e3af73740 1 --2- 172.21.15.169:0/1770685013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d0e690 0x5653b2d04cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.829+0000 7f9e3af73740 1 -- 172.21.15.169:0/1770685013 >> 172.21.15.169:0/1770685013 conn(0x5653b2d09c00 msgr2=0x5653b2d17530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:26.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.830+0000 7f9e3af73740 1 -- 172.21.15.169:0/1770685013 shutdown_connections 2024-03-28T01:04:26.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.831+0000 7f9e3af73740 1 -- 172.21.15.169:0/1770685013 wait complete. 2024-03-28T01:04:26.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.831+0000 7f9e3af73740 1 -- start start 2024-03-28T01:04:26.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.831+0000 7f9e3af73740 1 Processor -- start 2024-03-28T01:04:26.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.832+0000 7f9e3af73740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d0e690 0x5653b2d1d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.832+0000 7f9e3af73740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d0f080 0x5653b2d1de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.832+0000 7f9e36048640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d0e690 0x5653b2d1d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.832+0000 7f9e3af73740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d141f0 0x5653b2d1edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.832+0000 7f9e36048640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d0e690 0x5653b2d1d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39226/0 (socket says 172.21.15.169:39226) 2024-03-28T01:04:26.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.832+0000 7f9e3af73740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e301b7710 con 0x5653b2d0e690 2024-03-28T01:04:26.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.833+0000 7f9e3af73740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e301b8a10 con 0x5653b2d0f080 2024-03-28T01:04:26.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.833+0000 7f9e3af73740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e301b7d50 con 0x5653b2d141f0 2024-03-28T01:04:26.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.833+0000 7f9e35847640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d0f080 0x5653b2d1de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.833+0000 7f9e35847640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d0f080 0x5653b2d1de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:47146/0 (socket says 172.21.15.169:47146) 2024-03-28T01:04:26.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.833+0000 7f9e35847640 1 -- 172.21.15.169:0/2620251461 learned_addr learned my addr 172.21.15.169:0/2620251461 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:26.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.833+0000 7f9e36849640 1 --2- 172.21.15.169:0/2620251461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d141f0 0x5653b2d1edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.833+0000 7f9e3af73740 10 client.0 _ll_register_callbacks cb 0x7f9e2c00ced0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:04:26.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.833+0000 7f9e35847640 1 -- 172.21.15.169:0/2620251461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d141f0 msgr2=0x5653b2d1edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:26.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.833+0000 7f9e35847640 1 --2- 172.21.15.169:0/2620251461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d141f0 0x5653b2d1edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.833+0000 7f9e35847640 1 -- 172.21.15.169:0/2620251461 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d0e690 msgr2=0x5653b2d1d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:26.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.833+0000 7f9e35847640 1 --2- 172.21.15.169:0/2620251461 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d0e690 0x5653b2d1d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.833+0000 7f9e35847640 1 -- 172.21.15.169:0/2620251461 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5653b2d1f3c0 con 0x5653b2d0f080 2024-03-28T01:04:26.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.833+0000 7f9e3af73740 1 -- 172.21.15.169:0/2620251461 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9e28051040 con 0x5653b2d0f080 2024-03-28T01:04:26.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.833+0000 7f9e35847640 1 --2- 172.21.15.169:0/2620251461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d0f080 0x5653b2d1de10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c00cfd0 tx=0x7f9e2c09bf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a051f582487c29d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:26.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.839+0000 7f9e19ffb640 10 client.26879 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:04:26.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.840+0000 7f9e19ffb640 1 -- 172.21.15.169:0/2620251461 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c07e020 con 0x5653b2d0f080 2024-03-28T01:04:26.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.840+0000 7f9e19ffb640 1 -- 172.21.15.169:0/2620251461 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f9e2c0860b0 con 0x5653b2d0f080 2024-03-28T01:04:26.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.840+0000 7f9e19ffb640 1 -- 172.21.15.169:0/2620251461 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e2c09b0d0 con 0x5653b2d0f080 2024-03-28T01:04:26.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.841+0000 7f9e3af73740 10 client.26879 fetch_fsmap learned FSMap version 25 2024-03-28T01:04:26.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.841+0000 7f9e3af73740 1 -- 172.21.15.169:0/2620251461 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x5653b2d803f0 con 0x5653b2d0f080 2024-03-28T01:04:26.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.844+0000 7f9e19ffb640 1 -- 172.21.15.169:0/2620251461 <== mon.2 v2:172.21.15.174:3300/0 4 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9e2c09abc0 con 0x5653b2d0f080 2024-03-28T01:04:26.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.845+0000 7f9e3af73740 10 client.26879 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:04:26.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.845+0000 7f9e3af73740 10 client.26879 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:04:26.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.845+0000 7f9e3af73740 20 client.26879 populate_metadata read hostname 'smithi169' 2024-03-28T01:04:26.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.845+0000 7f9e3af73740 1 --2- 172.21.15.169:0/2620251461 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5653b2d809c0 0x5653b2d82de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:26.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.845+0000 7f9e3af73740 4 client.26879 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:04:26.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.845+0000 7f9e3af73740 1 -- 172.21.15.169:0/2620251461 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5653b2c7bc50 con 0x5653b2d809c0 2024-03-28T01:04:26.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.846+0000 7f9e36048640 1 --2- 172.21.15.169:0/2620251461 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5653b2d809c0 0x5653b2d82de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:26.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.846+0000 7f9e36048640 1 --2- 172.21.15.169:0/2620251461 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5653b2d809c0 0x5653b2d82de0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=644a16e7672482e3 server_cookie=5ec6361cd2fde332 in_seq=0 out_seq=0 2024-03-28T01:04:26.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.846+0000 7f9e19ffb640 10 client.26879 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:26.850 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.847+0000 7f9e19ffb640 1 -- 172.21.15.169:0/2620251461 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5653b2c7bc50 con 0x5653b2d809c0 2024-03-28T01:04:26.850 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.847+0000 7f9e19ffb640 10 client.26879 handle_command_reply: tid=0 2024-03-28T01:04:26.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.848+0000 7f9e3af73740 1 client.26879 shutdown 2024-03-28T01:04:26.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.848+0000 7f9e3af73740 1 -- 172.21.15.169:0/2620251461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d0f080 msgr2=0x5653b2d1de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.848+0000 7f9e3af73740 1 --2- 172.21.15.169:0/2620251461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d0f080 0x5653b2d1de10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c00cfd0 tx=0x7f9e2c09bf70 comp rx=0 tx=0).stop 2024-03-28T01:04:26.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.849+0000 7f9e3af73740 1 -- 172.21.15.169:0/2620251461 shutdown_connections 2024-03-28T01:04:26.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.849+0000 7f9e3af73740 1 --2- 172.21.15.169:0/2620251461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5653b2d141f0 0x5653b2d1edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.849+0000 7f9e3af73740 1 --2- 172.21.15.169:0/2620251461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653b2d0f080 0x5653b2d1de10 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.849+0000 7f9e3af73740 1 --2- 172.21.15.169:0/2620251461 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5653b2d809c0 0x5653b2d82de0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.849+0000 7f9e19ffb640 0 client.26879 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:26.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.849+0000 7f9e3af73740 1 --2- 172.21.15.169:0/2620251461 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5653b2d0e690 0x5653b2d1d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.849+0000 7f9e3af73740 1 -- 172.21.15.169:0/2620251461 >> 172.21.15.169:0/2620251461 conn(0x5653b2d09c00 msgr2=0x5653b2d0ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:26.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.850+0000 7f9e3af73740 1 -- 172.21.15.169:0/2620251461 shutdown_connections 2024-03-28T01:04:26.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.850+0000 7f9e3af73740 1 -- 172.21.15.169:0/2620251461 wait complete. 2024-03-28T01:04:26.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.850+0000 7f9e3af73740 20 client.26879 trim_cache size 0 max 16384 2024-03-28T01:04:26.854 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:04:26.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:26 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/307265944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:04:26.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.855+0000 7f9e382ac640 1 -- 172.21.15.169:0/4030446007 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9e1c0a2880 msgr2=0x7f9e1c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.855+0000 7f9e382ac640 1 --2- 172.21.15.169:0/4030446007 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9e1c0a2880 0x7f9e1c0a4d40 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9e28004ab0 tx=0x7f9e28043000 comp rx=0 tx=0).stop 2024-03-28T01:04:26.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.855+0000 7f9e382ac640 1 -- 172.21.15.169:0/4030446007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e30104a50 msgr2=0x7f9e30195ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:26.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.855+0000 7f9e382ac640 1 --2- 172.21.15.169:0/4030446007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e30104a50 0x7f9e30195ce0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c00e9c0 tx=0x7f9e2c00edc0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.855+0000 7f9e382ac640 1 -- 172.21.15.169:0/4030446007 shutdown_connections 2024-03-28T01:04:26.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.855+0000 7f9e382ac640 1 --2- 172.21.15.169:0/4030446007 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9e1c0a2880 0x7f9e1c0a4d40 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.855+0000 7f9e382ac640 1 --2- 172.21.15.169:0/4030446007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e30107380 0x7f9e301a5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.855+0000 7f9e382ac640 1 --2- 172.21.15.169:0/4030446007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e30104a50 0x7f9e30195ce0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.855+0000 7f9e382ac640 1 --2- 172.21.15.169:0/4030446007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e30102120 0x7f9e301957a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:26.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.855+0000 7f9e382ac640 1 -- 172.21.15.169:0/4030446007 >> 172.21.15.169:0/4030446007 conn(0x7f9e300fbb30 msgr2=0x7f9e30101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:26.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.855+0000 7f9e382ac640 1 -- 172.21.15.169:0/4030446007 shutdown_connections 2024-03-28T01:04:26.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:26.856+0000 7f9e382ac640 1 -- 172.21.15.169:0/4030446007 wait complete. 2024-03-28T01:04:26.880 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:26 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/307265944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:04:26.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:26 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/307265944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:04:27.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:27 smithi174 ceph-mon[36060]: pgmap v589: 129 pgs: 129 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 162 MiB/s rd, 334 MiB/s wr, 4.58k op/s 2024-03-28T01:04:27.882 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T01:04:27.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T01:04:27.883 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T01:04:27.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:27 smithi177 ceph-mon[36206]: pgmap v589: 129 pgs: 129 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 162 MiB/s rd, 334 MiB/s wr, 4.58k op/s 2024-03-28T01:04:27.934 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:27 smithi169 ceph-mon[32274]: pgmap v589: 129 pgs: 129 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 162 MiB/s rd, 334 MiB/s wr, 4.58k op/s 2024-03-28T01:04:28.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.066+0000 7eff0d201640 1 -- 172.21.15.169:0/4102521941 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff000a64f0 msgr2=0x7eff000a88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:28.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.066+0000 7eff0d201640 1 --2- 172.21.15.169:0/4102521941 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff000a64f0 0x7eff000a88e0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7efef8009900 tx=0x7efef802bc30 comp rx=0 tx=0).stop 2024-03-28T01:04:28.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.066+0000 7eff0d201640 1 -- 172.21.15.169:0/4102521941 shutdown_connections 2024-03-28T01:04:28.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.066+0000 7eff0d201640 1 --2- 172.21.15.169:0/4102521941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff000ab860 0x7eff000add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.066+0000 7eff0d201640 1 --2- 172.21.15.169:0/4102521941 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff000a8e20 0x7eff000ab210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.066+0000 7eff0d201640 1 --2- 172.21.15.169:0/4102521941 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff000a64f0 0x7eff000a88e0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.066+0000 7eff0d201640 1 -- 172.21.15.169:0/4102521941 >> 172.21.15.169:0/4102521941 conn(0x7eff000a00c0 msgr2=0x7eff000a2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:28.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.066+0000 7eff0d201640 1 -- 172.21.15.169:0/4102521941 shutdown_connections 2024-03-28T01:04:28.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.066+0000 7eff0d201640 1 -- 172.21.15.169:0/4102521941 wait complete. 2024-03-28T01:04:28.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.066+0000 7eff0d201640 1 Processor -- start 2024-03-28T01:04:28.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.067+0000 7eff0d201640 1 -- start start 2024-03-28T01:04:28.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.067+0000 7eff0d201640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff000a64f0 0x7eff000a5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.067+0000 7eff0d201640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff000a8e20 0x7eff00141e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.067+0000 7eff0d201640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff000ab860 0x7eff00144480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.067+0000 7eff0d201640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff000b1960 con 0x7eff000a8e20 2024-03-28T01:04:28.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.067+0000 7eff0d201640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff000b17e0 con 0x7eff000a64f0 2024-03-28T01:04:28.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.067+0000 7eff0d201640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff000b1ae0 con 0x7eff000ab860 2024-03-28T01:04:28.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.067+0000 7eff0659c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff000a8e20 0x7eff00141e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.067+0000 7eff0659c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff000a8e20 0x7eff00141e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39244/0 (socket says 172.21.15.169:39244) 2024-03-28T01:04:28.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.067+0000 7eff0759e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff000ab860 0x7eff00144480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.067+0000 7eff06d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff000a64f0 0x7eff000a5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.067+0000 7eff0659c640 1 -- 172.21.15.169:0/4287785089 learned_addr learned my addr 172.21.15.169:0/4287785089 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:28.073 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.067+0000 7eff06d9d640 1 -- 172.21.15.169:0/4287785089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff000ab860 msgr2=0x7eff00144480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:28.073 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.067+0000 7eff06d9d640 1 --2- 172.21.15.169:0/4287785089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff000ab860 0x7eff00144480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.073 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.067+0000 7eff06d9d640 1 -- 172.21.15.169:0/4287785089 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff000a8e20 msgr2=0x7eff00141e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:28.073 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.067+0000 7eff06d9d640 1 --2- 172.21.15.169:0/4287785089 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff000a8e20 0x7eff00141e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.073 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.067+0000 7eff06d9d640 1 -- 172.21.15.169:0/4287785089 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff00144d90 con 0x7eff000a64f0 2024-03-28T01:04:28.073 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.067+0000 7eff0659c640 1 --2- 172.21.15.169:0/4287785089 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff000a8e20 0x7eff00141e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:04:28.073 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.067+0000 7eff06d9d640 1 --2- 172.21.15.169:0/4287785089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff000a64f0 0x7eff000a5d70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7efef8009230 tx=0x7efef802c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:28.073 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.068+0000 7efeeffff640 1 -- 172.21.15.169:0/4287785089 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef8043020 con 0x7eff000a64f0 2024-03-28T01:04:28.073 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.068+0000 7efeeffff640 1 -- 172.21.15.169:0/4287785089 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efef80054e0 con 0x7eff000a64f0 2024-03-28T01:04:28.073 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.068+0000 7efeeffff640 1 -- 172.21.15.169:0/4287785089 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef8003af0 con 0x7eff000a64f0 2024-03-28T01:04:28.073 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.068+0000 7eff0d201640 1 -- 172.21.15.169:0/4287785089 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff00144fc0 con 0x7eff000a64f0 2024-03-28T01:04:28.073 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.068+0000 7eff0d201640 1 -- 172.21.15.169:0/4287785089 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff001453d0 con 0x7eff000a64f0 2024-03-28T01:04:28.075 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.070+0000 7efeeffff640 1 -- 172.21.15.169:0/4287785089 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7efef8046020 con 0x7eff000a64f0 2024-03-28T01:04:28.075 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.070+0000 7eff0fddf740 1 Processor -- start 2024-03-28T01:04:28.075 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.070+0000 7eff0fddf740 1 -- start start 2024-03-28T01:04:28.075 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.071+0000 7efeeffff640 1 --2- 172.21.15.169:0/4287785089 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efef00a2800 0x7efef00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.075 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.071+0000 7eff0fddf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea78b20 0x55bdfea7af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.071+0000 7eff0fddf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea7b450 0x55bdfea7d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.071+0000 7eff0fddf740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7deb0 0x55bdfea80320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.071+0000 7eff0fddf740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdfe9afc90 con 0x55bdfea7deb0 2024-03-28T01:04:28.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.071+0000 7eff0fddf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdfe9d5070 con 0x55bdfea78b20 2024-03-28T01:04:28.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.071+0000 7eff0fddf740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdfe97dc80 con 0x55bdfea7b450 2024-03-28T01:04:28.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.071+0000 7eff0759e640 1 --2- 172.21.15.169:0/4287785089 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efef00a2800 0x7efef00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.071+0000 7eff06d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea78b20 0x55bdfea7af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.071+0000 7eff06d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea78b20 0x55bdfea7af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:47166/0 (socket says 172.21.15.169:47166) 2024-03-28T01:04:28.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.071+0000 7eff06d9d640 1 -- 172.21.15.169:0/4271628142 learned_addr learned my addr 172.21.15.169:0/4271628142 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:28.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.071+0000 7eff0759e640 1 --2- 172.21.15.169:0/4271628142 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7deb0 0x55bdfea80320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.071+0000 7eff0659c640 1 --2- 172.21.15.169:0/4271628142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea7b450 0x55bdfea7d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.071+0000 7eff06d9d640 1 -- 172.21.15.169:0/4271628142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea7b450 msgr2=0x55bdfea7d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:28.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.071+0000 7eff06d9d640 1 --2- 172.21.15.169:0/4271628142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea7b450 0x55bdfea7d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.071+0000 7eff06d9d640 1 -- 172.21.15.169:0/4271628142 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7deb0 msgr2=0x55bdfea80320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:28.077 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.071+0000 7eff06d9d640 1 --2- 172.21.15.169:0/4271628142 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7deb0 0x55bdfea80320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.077 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.071+0000 7eff06d9d640 1 -- 172.21.15.169:0/4271628142 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bdfe975d10 con 0x55bdfea78b20 2024-03-28T01:04:28.077 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.071+0000 7eff0659c640 1 --2- 172.21.15.169:0/4271628142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea7b450 0x55bdfea7d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:04:28.077 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.071+0000 7eff0759e640 1 --2- 172.21.15.169:0/4271628142 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7deb0 0x55bdfea80320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:04:28.077 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.071+0000 7efeeffff640 1 -- 172.21.15.169:0/4287785089 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7efef80f0df0 con 0x7eff000a64f0 2024-03-28T01:04:28.077 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.072+0000 7eff06d9d640 1 --2- 172.21.15.169:0/4271628142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea78b20 0x55bdfea7af10 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7efef80bd320 tx=0x7efef80035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d5cf0e7ec25b104 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:28.077 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.072+0000 7efeed7fa640 1 -- 172.21.15.169:0/4271628142 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef809aa00 con 0x55bdfea78b20 2024-03-28T01:04:28.077 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.072+0000 7efeed7fa640 1 -- 172.21.15.169:0/4271628142 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efef80bda90 con 0x55bdfea78b20 2024-03-28T01:04:28.077 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.072+0000 7efeed7fa640 1 -- 172.21.15.169:0/4271628142 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef809a820 con 0x55bdfea78b20 2024-03-28T01:04:28.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.073+0000 7eff0fddf740 1 -- 172.21.15.169:0/4271628142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea78b20 msgr2=0x55bdfea7af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:28.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.073+0000 7eff0759e640 1 --2- 172.21.15.169:0/4287785089 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efef00a2800 0x7efef00a4cc0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7eff08066e40 tx=0x7eff08066f20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:28.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.073+0000 7eff0fddf740 1 --2- 172.21.15.169:0/4271628142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea78b20 0x55bdfea7af10 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7efef80bd320 tx=0x7efef80035f0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.074+0000 7eff0fddf740 1 -- 172.21.15.169:0/4271628142 shutdown_connections 2024-03-28T01:04:28.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.074+0000 7eff0fddf740 1 --2- 172.21.15.169:0/4271628142 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7deb0 0x55bdfea80320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.074+0000 7eff0fddf740 1 --2- 172.21.15.169:0/4271628142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea7b450 0x55bdfea7d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.074+0000 7eff0fddf740 1 --2- 172.21.15.169:0/4271628142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea78b20 0x55bdfea7af10 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.074+0000 7eff0fddf740 1 -- 172.21.15.169:0/4271628142 >> 172.21.15.169:0/4271628142 conn(0x55bdfea381a0 msgr2=0x55bdfea6b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:28.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.074+0000 7eff0fddf740 1 -- 172.21.15.169:0/4271628142 shutdown_connections 2024-03-28T01:04:28.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.074+0000 7eff0fddf740 1 -- 172.21.15.169:0/4271628142 wait complete. 2024-03-28T01:04:28.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.074+0000 7eff0fddf740 1 -- start start 2024-03-28T01:04:28.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.074+0000 7eff0fddf740 1 Processor -- start 2024-03-28T01:04:28.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.074+0000 7eff0fddf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea78b20 0x55bdfea82330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.074+0000 7eff0fddf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea7b450 0x55bdfea82870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.074+0000 7eff0fddf740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7deb0 0x55bdfea898b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.074+0000 7eff0fddf740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdfe9910d0 con 0x55bdfea7deb0 2024-03-28T01:04:28.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.074+0000 7eff0fddf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdfe9d1110 con 0x55bdfea78b20 2024-03-28T01:04:28.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.074+0000 7eff0fddf740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdfe9d5070 con 0x55bdfea7b450 2024-03-28T01:04:28.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.074+0000 7eff0fddf740 10 client.0 _ll_register_callbacks cb 0x55bdfe9cce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:04:28.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.075+0000 7eff06d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea78b20 0x55bdfea82330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.075+0000 7eff0759e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7deb0 0x55bdfea898b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.075+0000 7eff06d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea78b20 0x55bdfea82330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:47172/0 (socket says 172.21.15.169:47172) 2024-03-28T01:04:28.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.075+0000 7eff06d9d640 1 -- 172.21.15.169:0/3100098789 learned_addr learned my addr 172.21.15.169:0/3100098789 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:28.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.075+0000 7eff0659c640 1 --2- 172.21.15.169:0/3100098789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea7b450 0x55bdfea82870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.075+0000 7eff06d9d640 1 -- 172.21.15.169:0/3100098789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea7b450 msgr2=0x55bdfea82870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:28.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.075+0000 7eff06d9d640 1 --2- 172.21.15.169:0/3100098789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea7b450 0x55bdfea82870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.075+0000 7eff06d9d640 1 -- 172.21.15.169:0/3100098789 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7deb0 msgr2=0x55bdfea898b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:28.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.075+0000 7eff06d9d640 1 --2- 172.21.15.169:0/3100098789 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7deb0 0x55bdfea898b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.075+0000 7eff06d9d640 1 -- 172.21.15.169:0/3100098789 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bdfe7b17e0 con 0x55bdfea78b20 2024-03-28T01:04:28.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.075+0000 7eff0659c640 1 --2- 172.21.15.169:0/3100098789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea7b450 0x55bdfea82870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:04:28.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.075+0000 7eff06d9d640 1 --2- 172.21.15.169:0/3100098789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea78b20 0x55bdfea82330 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7efef80b9d60 tx=0x7efef8005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e42ff4127fc5c10c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:28.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.075+0000 7efeeaff5640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:04:28.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.075+0000 7efeeaff5640 1 -- 172.21.15.169:0/3100098789 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef80c1150 con 0x55bdfea78b20 2024-03-28T01:04:28.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.075+0000 7efeeaff5640 1 -- 172.21.15.169:0/3100098789 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7efef809c200 con 0x55bdfea78b20 2024-03-28T01:04:28.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.075+0000 7eff0fddf740 1 -- 172.21.15.169:0/3100098789 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efef80bda90 con 0x55bdfea78b20 2024-03-28T01:04:28.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.076+0000 7efeeaff5640 1 -- 172.21.15.169:0/3100098789 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7efef80d6120 con 0x55bdfea78b20 2024-03-28T01:04:28.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.076+0000 7eff0fddf740 10 client.26891 fetch_fsmap learned FSMap version 25 2024-03-28T01:04:28.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.076+0000 7eff0fddf740 1 -- 172.21.15.169:0/3100098789 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55bdfe811f70 con 0x55bdfea78b20 2024-03-28T01:04:28.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.076+0000 7efeeaff5640 1 -- 172.21.15.169:0/3100098789 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efef80bda90 con 0x55bdfea78b20 2024-03-28T01:04:28.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.076+0000 7efeeaff5640 1 -- 172.21.15.169:0/3100098789 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7efef809a7e0 con 0x55bdfea78b20 2024-03-28T01:04:28.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.076+0000 7eff0fddf740 10 client.26891 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:04:28.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.076+0000 7eff0fddf740 10 client.26891 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:04:28.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.076+0000 7eff0fddf740 20 client.26891 populate_metadata read hostname 'smithi169' 2024-03-28T01:04:28.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.076+0000 7eff0fddf740 1 --2- 172.21.15.169:0/3100098789 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bdfea8de90 0x55bdfea902c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.076+0000 7eff0fddf740 4 client.26891 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:04:28.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.077+0000 7eff0fddf740 1 -- 172.21.15.169:0/3100098789 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bdfe9e25b0 con 0x55bdfea8de90 2024-03-28T01:04:28.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.077+0000 7eff0659c640 1 --2- 172.21.15.169:0/3100098789 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bdfea8de90 0x55bdfea902c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.077+0000 7eff0659c640 1 --2- 172.21.15.169:0/3100098789 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bdfea8de90 0x55bdfea902c0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0dd810df9143aa8 server_cookie=9a265c363048f4eb in_seq=0 out_seq=0 2024-03-28T01:04:28.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.077+0000 7efeeaff5640 10 client.26891 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:28.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.078+0000 7efeeaff5640 1 -- 172.21.15.169:0/3100098789 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bdfe9e25b0 con 0x55bdfea8de90 2024-03-28T01:04:28.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.078+0000 7efeeaff5640 10 client.26891 handle_command_reply: tid=0 2024-03-28T01:04:28.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.078+0000 7eff0fddf740 1 client.26891 shutdown 2024-03-28T01:04:28.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.078+0000 7eff0fddf740 1 -- 172.21.15.169:0/3100098789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea78b20 msgr2=0x55bdfea82330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:28.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.078+0000 7eff0fddf740 1 --2- 172.21.15.169:0/3100098789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea78b20 0x55bdfea82330 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7efef80b9d60 tx=0x7efef8005dd0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.078+0000 7eff0fddf740 1 -- 172.21.15.169:0/3100098789 shutdown_connections 2024-03-28T01:04:28.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.078+0000 7eff0fddf740 1 --2- 172.21.15.169:0/3100098789 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bdfea8de90 0x55bdfea902c0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.078+0000 7efeeaff5640 0 client.26891 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:28.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.078+0000 7eff0fddf740 1 --2- 172.21.15.169:0/3100098789 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7deb0 0x55bdfea898b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.078+0000 7eff0fddf740 1 --2- 172.21.15.169:0/3100098789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea7b450 0x55bdfea82870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.078+0000 7eff0fddf740 1 --2- 172.21.15.169:0/3100098789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea78b20 0x55bdfea82330 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.078+0000 7eff0fddf740 1 -- 172.21.15.169:0/3100098789 >> 172.21.15.169:0/3100098789 conn(0x55bdfea381a0 msgr2=0x55bdfea6a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:28.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.079+0000 7eff0fddf740 1 -- 172.21.15.169:0/3100098789 shutdown_connections 2024-03-28T01:04:28.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.079+0000 7eff0fddf740 1 -- 172.21.15.169:0/3100098789 wait complete. 2024-03-28T01:04:28.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.079+0000 7eff0fddf740 20 client.26891 trim_cache size 0 max 16384 2024-03-28T01:04:28.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.105+0000 7eff0fddf740 1 Processor -- start 2024-03-28T01:04:28.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.105+0000 7eff0fddf740 1 -- start start 2024-03-28T01:04:28.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.105+0000 7eff0fddf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea78b20 0x55bdfea82650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.105+0000 7eff0fddf740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7b450 0x55bdfea89840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.105+0000 7eff0fddf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea7deb0 0x55bdfea69f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.105+0000 7eff0fddf740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0014fb50 con 0x55bdfea7b450 2024-03-28T01:04:28.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.105+0000 7eff0fddf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0014d650 con 0x55bdfea7deb0 2024-03-28T01:04:28.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.105+0000 7eff0fddf740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0014b6f0 con 0x55bdfea78b20 2024-03-28T01:04:28.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.105+0000 7eff0659c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7b450 0x55bdfea89840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.105+0000 7eff0659c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7b450 0x55bdfea89840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39276/0 (socket says 172.21.15.169:39276) 2024-03-28T01:04:28.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.105+0000 7eff0659c640 1 -- 172.21.15.169:0/3790620885 learned_addr learned my addr 172.21.15.169:0/3790620885 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:28.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.106+0000 7eff0759e640 1 --2- 172.21.15.169:0/3790620885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea7deb0 0x55bdfea69f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.106+0000 7eff06d9d640 1 --2- 172.21.15.169:0/3790620885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea78b20 0x55bdfea82650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.106+0000 7eff0759e640 1 -- 172.21.15.169:0/3790620885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea78b20 msgr2=0x55bdfea82650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:28.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.106+0000 7eff0759e640 1 --2- 172.21.15.169:0/3790620885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea78b20 0x55bdfea82650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.106+0000 7eff0759e640 1 -- 172.21.15.169:0/3790620885 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7b450 msgr2=0x55bdfea89840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:28.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.106+0000 7eff0759e640 1 --2- 172.21.15.169:0/3790620885 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7b450 0x55bdfea89840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.106+0000 7eff0759e640 1 -- 172.21.15.169:0/3790620885 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bdfe6a9170 con 0x55bdfea7deb0 2024-03-28T01:04:28.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.106+0000 7eff0759e640 1 --2- 172.21.15.169:0/3790620885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea7deb0 0x55bdfea69f50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7eff0804f6d0 tx=0x7eff08050420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=214c7f1fc52a830f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:28.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.106+0000 7efeecff9640 1 -- 172.21.15.169:0/3790620885 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff08075cc0 con 0x55bdfea7deb0 2024-03-28T01:04:28.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.106+0000 7efeecff9640 1 -- 172.21.15.169:0/3790620885 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff08075e60 con 0x55bdfea7deb0 2024-03-28T01:04:28.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.107+0000 7efeecff9640 1 -- 172.21.15.169:0/3790620885 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff08086ba0 con 0x55bdfea7deb0 2024-03-28T01:04:28.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.107+0000 7eff0fddf740 1 -- 172.21.15.169:0/3790620885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea7deb0 msgr2=0x55bdfea69f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:28.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.107+0000 7eff0fddf740 1 --2- 172.21.15.169:0/3790620885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea7deb0 0x55bdfea69f50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7eff0804f6d0 tx=0x7eff08050420 comp rx=0 tx=0).stop 2024-03-28T01:04:28.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.107+0000 7eff0fddf740 1 -- 172.21.15.169:0/3790620885 shutdown_connections 2024-03-28T01:04:28.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.107+0000 7eff0fddf740 1 --2- 172.21.15.169:0/3790620885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea7deb0 0x55bdfea69f50 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.107+0000 7eff0fddf740 1 --2- 172.21.15.169:0/3790620885 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7b450 0x55bdfea89840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.107+0000 7eff0fddf740 1 --2- 172.21.15.169:0/3790620885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea78b20 0x55bdfea82650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.107+0000 7eff0fddf740 1 -- 172.21.15.169:0/3790620885 >> 172.21.15.169:0/3790620885 conn(0x55bdfea381a0 msgr2=0x55bdfea81f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:28.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.108+0000 7eff0fddf740 1 -- 172.21.15.169:0/3790620885 shutdown_connections 2024-03-28T01:04:28.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.108+0000 7eff0fddf740 1 -- 172.21.15.169:0/3790620885 wait complete. 2024-03-28T01:04:28.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.108+0000 7eff0fddf740 1 -- start start 2024-03-28T01:04:28.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.108+0000 7eff0fddf740 1 Processor -- start 2024-03-28T01:04:28.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.108+0000 7eff0fddf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea78b20 0x55bdfea898a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.108+0000 7eff0fddf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea7b450 0x55bdfea87f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.108+0000 7eff0fddf740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7deb0 0x55bdfea88530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.108+0000 7eff0fddf740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0014bcf0 con 0x55bdfea7deb0 2024-03-28T01:04:28.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.109+0000 7eff0fddf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0014cff0 con 0x55bdfea7b450 2024-03-28T01:04:28.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.109+0000 7eff0fddf740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0014c330 con 0x55bdfea78b20 2024-03-28T01:04:28.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.109+0000 7eff0fddf740 10 client.0 _ll_register_callbacks cb 0x7efef80bd1f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:04:28.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.109+0000 7eff0659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea7b450 0x55bdfea87f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.109+0000 7eff06d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea78b20 0x55bdfea898a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.109+0000 7eff0659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea7b450 0x55bdfea87f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:47192/0 (socket says 172.21.15.169:47192) 2024-03-28T01:04:28.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.109+0000 7eff0659c640 1 -- 172.21.15.169:0/3404008873 learned_addr learned my addr 172.21.15.169:0/3404008873 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:28.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.109+0000 7eff0759e640 1 --2- 172.21.15.169:0/3404008873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7deb0 0x55bdfea88530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.109+0000 7eff0659c640 1 -- 172.21.15.169:0/3404008873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea78b20 msgr2=0x55bdfea898a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:28.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.109+0000 7eff0659c640 1 --2- 172.21.15.169:0/3404008873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea78b20 0x55bdfea898a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.109+0000 7eff0659c640 1 -- 172.21.15.169:0/3404008873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7deb0 msgr2=0x55bdfea88530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:28.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.109+0000 7eff0659c640 1 --2- 172.21.15.169:0/3404008873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7deb0 0x55bdfea88530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.109+0000 7eff0659c640 1 -- 172.21.15.169:0/3404008873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bdfea88d60 con 0x55bdfea7b450 2024-03-28T01:04:28.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.110+0000 7eff0659c640 1 --2- 172.21.15.169:0/3404008873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea7b450 0x55bdfea87f20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7efefc013480 tx=0x7efefc013c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=958756fffaa68aec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:28.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.110+0000 7efeeaff5640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:04:28.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.110+0000 7efeeaff5640 1 -- 172.21.15.169:0/3404008873 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc005020 con 0x55bdfea7b450 2024-03-28T01:04:28.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.110+0000 7eff0fddf740 1 -- 172.21.15.169:0/3404008873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7eff08075e60 con 0x55bdfea7b450 2024-03-28T01:04:28.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.110+0000 7efeeaff5640 1 -- 172.21.15.169:0/3404008873 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7efefc0051c0 con 0x55bdfea7b450 2024-03-28T01:04:28.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.111+0000 7efeeaff5640 1 -- 172.21.15.169:0/3404008873 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7efefc02e050 con 0x55bdfea7b450 2024-03-28T01:04:28.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.111+0000 7eff0fddf740 10 client.26897 fetch_fsmap learned FSMap version 25 2024-03-28T01:04:28.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.111+0000 7eff0fddf740 1 -- 172.21.15.169:0/3404008873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55bdfea70590 con 0x55bdfea7b450 2024-03-28T01:04:28.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.111+0000 7efeeaff5640 1 -- 172.21.15.169:0/3404008873 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff08075e60 con 0x55bdfea7b450 2024-03-28T01:04:28.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.111+0000 7efeeaff5640 1 -- 172.21.15.169:0/3404008873 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7efefc012520 con 0x55bdfea7b450 2024-03-28T01:04:28.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.111+0000 7eff0fddf740 10 client.26897 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:04:28.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.111+0000 7eff0fddf740 10 client.26897 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:04:28.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.111+0000 7eff0fddf740 20 client.26897 populate_metadata read hostname 'smithi169' 2024-03-28T01:04:28.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.111+0000 7eff0fddf740 1 --2- 172.21.15.169:0/3404008873 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bdfea8de90 0x55bdfea72f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.111+0000 7eff0fddf740 4 client.26897 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T01:04:28.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.111+0000 7eff0fddf740 1 -- 172.21.15.169:0/3404008873 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55bdfe9dc900 con 0x55bdfea8de90 2024-03-28T01:04:28.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.111+0000 7eff06d9d640 1 --2- 172.21.15.169:0/3404008873 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bdfea8de90 0x55bdfea72f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.112+0000 7efeeaff5640 10 client.26897 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:28.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.113+0000 7eff06d9d640 1 --2- 172.21.15.169:0/3404008873 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bdfea8de90 0x55bdfea72f90 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83b8cf28a56feddd server_cookie=926b1d52a2fa1314 in_seq=0 out_seq=0 2024-03-28T01:04:28.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.114+0000 7efeeaff5640 1 -- 172.21.15.169:0/3404008873 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55bdfe9dc900 con 0x55bdfea8de90 2024-03-28T01:04:28.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.114+0000 7efeeaff5640 10 client.26897 handle_command_reply: tid=0 2024-03-28T01:04:28.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.114+0000 7eff0fddf740 1 client.26897 shutdown 2024-03-28T01:04:28.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.115+0000 7eff0fddf740 1 -- 172.21.15.169:0/3404008873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea7b450 msgr2=0x55bdfea87f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:28.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.115+0000 7eff0fddf740 1 --2- 172.21.15.169:0/3404008873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea7b450 0x55bdfea87f20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7efefc013480 tx=0x7efefc013c30 comp rx=0 tx=0).stop 2024-03-28T01:04:28.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.115+0000 7eff0fddf740 1 -- 172.21.15.169:0/3404008873 shutdown_connections 2024-03-28T01:04:28.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.115+0000 7eff0fddf740 1 --2- 172.21.15.169:0/3404008873 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55bdfea8de90 0x55bdfea72f90 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.115+0000 7eff0fddf740 1 --2- 172.21.15.169:0/3404008873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55bdfea7deb0 0x55bdfea88530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.115+0000 7eff0fddf740 1 --2- 172.21.15.169:0/3404008873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bdfea7b450 0x55bdfea87f20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.115+0000 7eff0fddf740 1 --2- 172.21.15.169:0/3404008873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdfea78b20 0x55bdfea898a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.115+0000 7eff0fddf740 1 -- 172.21.15.169:0/3404008873 >> 172.21.15.169:0/3404008873 conn(0x55bdfea381a0 msgr2=0x55bdfea783a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:28.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.115+0000 7efeeaff5640 0 client.26897 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:28.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.116+0000 7eff0fddf740 1 -- 172.21.15.169:0/3404008873 shutdown_connections 2024-03-28T01:04:28.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.116+0000 7eff0fddf740 1 -- 172.21.15.169:0/3404008873 wait complete. 2024-03-28T01:04:28.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.116+0000 7eff0fddf740 20 client.26897 trim_cache size 0 max 16384 2024-03-28T01:04:28.120 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:04:28.120 INFO:teuthology.orchestra.run.smithi169.stdout: "return_code": 0, 2024-03-28T01:04:28.120 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_tag": "924ab91a-d205-4422-8e64-30329ff89fea", 2024-03-28T01:04:28.120 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": "asynchronous" 2024-03-28T01:04:28.120 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:04:28.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.117+0000 7eff0d201640 1 -- 172.21.15.169:0/4287785089 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efef00a2800 msgr2=0x7efef00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:28.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.117+0000 7eff0d201640 1 --2- 172.21.15.169:0/4287785089 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efef00a2800 0x7efef00a4cc0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7eff08066e40 tx=0x7eff08066f20 comp rx=0 tx=0).stop 2024-03-28T01:04:28.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.117+0000 7eff0d201640 1 -- 172.21.15.169:0/4287785089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff000a64f0 msgr2=0x7eff000a5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:28.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.118+0000 7eff0d201640 1 --2- 172.21.15.169:0/4287785089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff000a64f0 0x7eff000a5d70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7efef8009230 tx=0x7efef802c5d0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.118+0000 7eff0d201640 1 -- 172.21.15.169:0/4287785089 shutdown_connections 2024-03-28T01:04:28.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.118+0000 7eff0d201640 1 --2- 172.21.15.169:0/4287785089 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efef00a2800 0x7efef00a4cc0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.118+0000 7eff0d201640 1 --2- 172.21.15.169:0/4287785089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff000ab860 0x7eff00144480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.118+0000 7eff0d201640 1 --2- 172.21.15.169:0/4287785089 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff000a8e20 0x7eff00141e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.118+0000 7eff0d201640 1 --2- 172.21.15.169:0/4287785089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff000a64f0 0x7eff000a5d70 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.118+0000 7eff0d201640 1 -- 172.21.15.169:0/4287785089 >> 172.21.15.169:0/4287785089 conn(0x7eff000a00c0 msgr2=0x7eff000a7260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:28.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.118+0000 7eff0d201640 1 -- 172.21.15.169:0/4287785089 shutdown_connections 2024-03-28T01:04:28.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.118+0000 7eff0d201640 1 -- 172.21.15.169:0/4287785089 wait complete. 2024-03-28T01:04:28.152 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:04:28.152 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:04:28.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.380+0000 7f921537e640 1 -- 172.21.15.169:0/3394596148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f921010b010 msgr2=0x7f921010d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:28.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.380+0000 7f921537e640 1 --2- 172.21.15.169:0/3394596148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f921010b010 0x7f921010d480 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f91f8009900 tx=0x7f91f802bc80 comp rx=0 tx=0).stop 2024-03-28T01:04:28.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.382+0000 7f921537e640 1 -- 172.21.15.169:0/3394596148 shutdown_connections 2024-03-28T01:04:28.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.382+0000 7f921537e640 1 --2- 172.21.15.169:0/3394596148 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f921010d9c0 0x7f921010de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.382+0000 7f921537e640 1 --2- 172.21.15.169:0/3394596148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f921010b010 0x7f921010d480 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.382+0000 7f921537e640 1 --2- 172.21.15.169:0/3394596148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92101086a0 0x7f921010aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.382+0000 7f921537e640 1 -- 172.21.15.169:0/3394596148 >> 172.21.15.169:0/3394596148 conn(0x7f921006e500 msgr2=0x7f9210070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:28.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.387+0000 7f921537e640 1 -- 172.21.15.169:0/3394596148 shutdown_connections 2024-03-28T01:04:28.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.390+0000 7f921537e640 1 -- 172.21.15.169:0/3394596148 wait complete. 2024-03-28T01:04:28.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.391+0000 7f921537e640 1 Processor -- start 2024-03-28T01:04:28.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.391+0000 7f921537e640 1 -- start start 2024-03-28T01:04:28.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.391+0000 7f921537e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92101086a0 0x7f9210114dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.392+0000 7f921537e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f921010b010 0x7f9210115310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.392+0000 7f921537e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f921010d9c0 0x7f92101b68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.392+0000 7f921537e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921011c1e0 con 0x7f921010b010 2024-03-28T01:04:28.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.392+0000 7f921537e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921011c060 con 0x7f92101086a0 2024-03-28T01:04:28.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.392+0000 7f921537e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921011c360 con 0x7f921010d9c0 2024-03-28T01:04:28.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.392+0000 7f920effd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92101086a0 0x7f9210114dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.392+0000 7f920effd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92101086a0 0x7f9210114dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:47206/0 (socket says 172.21.15.169:47206) 2024-03-28T01:04:28.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.392+0000 7f920effd640 1 -- 172.21.15.169:0/84850341 learned_addr learned my addr 172.21.15.169:0/84850341 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:28.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.394+0000 7f920effd640 1 -- 172.21.15.169:0/84850341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f921010d9c0 msgr2=0x7f92101b68f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:04:28.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.394+0000 7f920effd640 1 --2- 172.21.15.169:0/84850341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f921010d9c0 0x7f92101b68f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.394+0000 7f920effd640 1 -- 172.21.15.169:0/84850341 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f921010b010 msgr2=0x7f9210115310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:28.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.394+0000 7f920effd640 1 --2- 172.21.15.169:0/84850341 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f921010b010 0x7f9210115310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.394+0000 7f920effd640 1 -- 172.21.15.169:0/84850341 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92101b6e30 con 0x7f92101086a0 2024-03-28T01:04:28.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.395+0000 7f920effd640 1 --2- 172.21.15.169:0/84850341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92101086a0 0x7f9210114dd0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f920000ebc0 tx=0x7f920000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:28.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.397+0000 7f9207fff640 1 -- 172.21.15.169:0/84850341 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9200010630 con 0x7f92101086a0 2024-03-28T01:04:28.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.397+0000 7f9207fff640 1 -- 172.21.15.169:0/84850341 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9200010cb0 con 0x7f92101086a0 2024-03-28T01:04:28.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.397+0000 7f9207fff640 1 -- 172.21.15.169:0/84850341 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9200018c40 con 0x7f92101086a0 2024-03-28T01:04:28.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.399+0000 7f921537e640 1 -- 172.21.15.169:0/84850341 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92101b7000 con 0x7f92101086a0 2024-03-28T01:04:28.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.399+0000 7f921537e640 1 -- 172.21.15.169:0/84850341 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92101b7410 con 0x7f92101086a0 2024-03-28T01:04:28.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.403+0000 7f9217f6f740 1 Processor -- start 2024-03-28T01:04:28.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.405+0000 7f9207fff640 1 -- 172.21.15.169:0/84850341 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f92000107d0 con 0x7f92101086a0 2024-03-28T01:04:28.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.406+0000 7f9207fff640 1 --2- 172.21.15.169:0/84850341 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f91fc0a2a00 0x7f91fc0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.406+0000 7f9207fff640 1 -- 172.21.15.169:0/84850341 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f92000cf610 con 0x7f92101086a0 2024-03-28T01:04:28.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.407+0000 7f9217f6f740 1 -- start start 2024-03-28T01:04:28.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.408+0000 7f9217f6f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56074131eda0 0x56074131f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.408+0000 7f9217f6f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56074131f790 0x560741324290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.408+0000 7f9217f6f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560741324900 0x560741326d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.408+0000 7f9217f6f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607412843d0 con 0x56074131f790 2024-03-28T01:04:28.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.408+0000 7f9217f6f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x56074122cad0 con 0x560741324900 2024-03-28T01:04:28.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.408+0000 7f9217f6f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560741280470 con 0x56074131eda0 2024-03-28T01:04:28.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.409+0000 7f920f7fe640 1 --2- 172.21.15.169:0/84850341 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f91fc0a2a00 0x7f91fc0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.410+0000 7f920f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560741324900 0x560741326d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.414+0000 7f920f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560741324900 0x560741326d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:47210/0 (socket says 172.21.15.169:47210) 2024-03-28T01:04:28.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.414+0000 7f920f7fe640 1 -- 172.21.15.169:0/2979372385 learned_addr learned my addr 172.21.15.169:0/2979372385 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:28.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.414+0000 7f920e7fc640 1 --2- 172.21.15.169:0/2979372385 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56074131f790 0x560741324290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.416+0000 7f920e7fc640 1 -- 172.21.15.169:0/2979372385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56074131eda0 msgr2=0x56074131f1c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:04:28.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.416+0000 7f920e7fc640 1 --2- 172.21.15.169:0/2979372385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56074131eda0 0x56074131f1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.416+0000 7f920e7fc640 1 -- 172.21.15.169:0/2979372385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560741324900 msgr2=0x560741326d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:28.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.416+0000 7f920e7fc640 1 --2- 172.21.15.169:0/2979372385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560741324900 0x560741326d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.416+0000 7f920e7fc640 1 -- 172.21.15.169:0/2979372385 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607410627d0 con 0x56074131f790 2024-03-28T01:04:28.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.418+0000 7f920f7fe640 1 --2- 172.21.15.169:0/84850341 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f91fc0a2a00 0x7f91fc0a4ec0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9208009d50 tx=0x7f9208009690 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:28.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.421+0000 7f920e7fc640 1 --2- 172.21.15.169:0/2979372385 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56074131f790 0x560741324290 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f91f8009230 tx=0x7f91f8009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=413db2b9ee4bdfa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:28.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.426+0000 7f92057fa640 1 -- 172.21.15.169:0/2979372385 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f8039040 con 0x56074131f790 2024-03-28T01:04:28.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.426+0000 7f92057fa640 1 -- 172.21.15.169:0/2979372385 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f91f80046c0 con 0x56074131f790 2024-03-28T01:04:28.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.426+0000 7f92057fa640 1 -- 172.21.15.169:0/2979372385 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f802cce0 con 0x56074131f790 2024-03-28T01:04:28.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.428+0000 7f9217f6f740 1 -- 172.21.15.169:0/2979372385 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56074131f790 msgr2=0x560741324290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:28.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.428+0000 7f9217f6f740 1 --2- 172.21.15.169:0/2979372385 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56074131f790 0x560741324290 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f91f8009230 tx=0x7f91f8009560 comp rx=0 tx=0).stop 2024-03-28T01:04:28.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.433+0000 7f9217f6f740 1 -- 172.21.15.169:0/2979372385 shutdown_connections 2024-03-28T01:04:28.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.433+0000 7f9217f6f740 1 --2- 172.21.15.169:0/2979372385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560741324900 0x560741326d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.433+0000 7f9217f6f740 1 --2- 172.21.15.169:0/2979372385 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56074131f790 0x560741324290 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.433+0000 7f9217f6f740 1 --2- 172.21.15.169:0/2979372385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56074131eda0 0x56074131f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.433+0000 7f9217f6f740 1 -- 172.21.15.169:0/2979372385 >> 172.21.15.169:0/2979372385 conn(0x56074131a310 msgr2=0x56074131a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:28.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.436+0000 7f9217f6f740 1 -- 172.21.15.169:0/2979372385 shutdown_connections 2024-03-28T01:04:28.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.436+0000 7f9217f6f740 1 -- 172.21.15.169:0/2979372385 wait complete. 2024-03-28T01:04:28.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.437+0000 7f9217f6f740 1 -- start start 2024-03-28T01:04:28.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.437+0000 7f9217f6f740 1 Processor -- start 2024-03-28T01:04:28.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.437+0000 7f9217f6f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56074131eda0 0x560741329310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.437+0000 7f9217f6f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56074131f790 0x560741329850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.437+0000 7f9217f6f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560741324900 0x560741330890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.437+0000 7f9217f6f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x560741327850 con 0x56074131eda0 2024-03-28T01:04:28.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.437+0000 7f9217f6f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x560741240060 con 0x56074131f790 2024-03-28T01:04:28.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.437+0000 7f9217f6f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607413279d0 con 0x560741324900 2024-03-28T01:04:28.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.437+0000 7f9217f6f740 10 client.0 _ll_register_callbacks cb 0x5607412226a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:04:28.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.439+0000 7f920effd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56074131eda0 0x560741329310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.439+0000 7f920effd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56074131eda0 0x560741329310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39312/0 (socket says 172.21.15.169:39312) 2024-03-28T01:04:28.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.439+0000 7f920effd640 1 -- 172.21.15.169:0/387092795 learned_addr learned my addr 172.21.15.169:0/387092795 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:28.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.440+0000 7f920f7fe640 1 --2- 172.21.15.169:0/387092795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560741324900 0x560741330890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.440+0000 7f920effd640 1 -- 172.21.15.169:0/387092795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560741324900 msgr2=0x560741330890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:28.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.440+0000 7f920effd640 1 --2- 172.21.15.169:0/387092795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560741324900 0x560741330890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.440+0000 7f920effd640 1 -- 172.21.15.169:0/387092795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56074131f790 msgr2=0x560741329850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:28.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.440+0000 7f920effd640 1 --2- 172.21.15.169:0/387092795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56074131f790 0x560741329850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.440+0000 7f920effd640 1 -- 172.21.15.169:0/387092795 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560740f983f0 con 0x56074131eda0 2024-03-28T01:04:28.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.441+0000 7f920effd640 1 --2- 172.21.15.169:0/387092795 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56074131eda0 0x560741329310 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f92000a3e20 tx=0x7f920009b5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=423dbdd0bdfbda50 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:28.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.443+0000 7f91f67fc640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:04:28.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.443+0000 7f91f67fc640 1 -- 172.21.15.169:0/387092795 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92000b4060 con 0x56074131eda0 2024-03-28T01:04:28.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.443+0000 7f91f67fc640 1 -- 172.21.15.169:0/387092795 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f9200018420 con 0x56074131eda0 2024-03-28T01:04:28.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.444+0000 7f9217f6f740 1 -- 172.21.15.169:0/387092795 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f91f80046c0 con 0x56074131eda0 2024-03-28T01:04:28.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.444+0000 7f91f67fc640 1 -- 172.21.15.169:0/387092795 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92000b9050 con 0x56074131eda0 2024-03-28T01:04:28.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.444+0000 7f9217f6f740 10 client.17034 fetch_fsmap learned FSMap version 25 2024-03-28T01:04:28.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.444+0000 7f9217f6f740 1 -- 172.21.15.169:0/387092795 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x560740f580f0 con 0x56074131eda0 2024-03-28T01:04:28.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.445+0000 7f91f67fc640 1 -- 172.21.15.169:0/387092795 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f91f80046c0 con 0x56074131eda0 2024-03-28T01:04:28.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.445+0000 7f91f67fc640 1 -- 172.21.15.169:0/387092795 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f92000ba240 con 0x56074131eda0 2024-03-28T01:04:28.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.446+0000 7f9217f6f740 10 client.17034 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:04:28.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.446+0000 7f9217f6f740 10 client.17034 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:04:28.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.446+0000 7f9217f6f740 20 client.17034 populate_metadata read hostname 'smithi169' 2024-03-28T01:04:28.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.446+0000 7f9217f6f740 1 --2- 172.21.15.169:0/387092795 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5607413572e0 0x560741359760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.446+0000 7f9217f6f740 4 client.17034 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:04:28.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.446+0000 7f9217f6f740 1 -- 172.21.15.169:0/387092795 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560741291910 con 0x5607413572e0 2024-03-28T01:04:28.454 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.452+0000 7f920e7fc640 1 --2- 172.21.15.169:0/387092795 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5607413572e0 0x560741359760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.463+0000 7f920e7fc640 1 --2- 172.21.15.169:0/387092795 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5607413572e0 0x560741359760 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46b6df490520353c server_cookie=7b9e16bfa1589a8c in_seq=0 out_seq=0 2024-03-28T01:04:28.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.463+0000 7f91f67fc640 10 client.17034 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:28.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.467+0000 7f91f67fc640 1 -- 172.21.15.169:0/387092795 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560741291910 con 0x5607413572e0 2024-03-28T01:04:28.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.467+0000 7f91f67fc640 10 client.17034 handle_command_reply: tid=0 2024-03-28T01:04:28.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.467+0000 7f9217f6f740 1 client.17034 shutdown 2024-03-28T01:04:28.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.470+0000 7f9217f6f740 1 -- 172.21.15.169:0/387092795 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56074131eda0 msgr2=0x560741329310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:28.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.470+0000 7f9217f6f740 1 --2- 172.21.15.169:0/387092795 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56074131eda0 0x560741329310 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f92000a3e20 tx=0x7f920009b5d0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.473+0000 7f9217f6f740 1 -- 172.21.15.169:0/387092795 shutdown_connections 2024-03-28T01:04:28.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.473+0000 7f9217f6f740 1 --2- 172.21.15.169:0/387092795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560741324900 0x560741330890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.473+0000 7f9217f6f740 1 --2- 172.21.15.169:0/387092795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56074131f790 0x560741329850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.473+0000 7f9217f6f740 1 --2- 172.21.15.169:0/387092795 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5607413572e0 0x560741359760 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.473+0000 7f9217f6f740 1 --2- 172.21.15.169:0/387092795 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56074131eda0 0x560741329310 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.473+0000 7f9217f6f740 1 -- 172.21.15.169:0/387092795 >> 172.21.15.169:0/387092795 conn(0x56074131a310 msgr2=0x560741319530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:28.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.473+0000 7f91f67fc640 0 client.17034 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:28.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.478+0000 7f9217f6f740 1 -- 172.21.15.169:0/387092795 shutdown_connections 2024-03-28T01:04:28.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.478+0000 7f9217f6f740 1 -- 172.21.15.169:0/387092795 wait complete. 2024-03-28T01:04:28.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.480+0000 7f9217f6f740 20 client.17034 trim_cache size 0 max 16384 2024-03-28T01:04:28.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.514+0000 7f9217f6f740 1 Processor -- start 2024-03-28T01:04:28.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.514+0000 7f9217f6f740 1 -- start start 2024-03-28T01:04:28.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.514+0000 7f9217f6f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56074131eda0 0x560741329630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.514+0000 7f9217f6f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56074131f790 0x560741330850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.514+0000 7f9217f6f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560741324900 0x560741318ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.514+0000 7f9217f6f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92100800b0 con 0x560741324900 2024-03-28T01:04:28.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.514+0000 7f9217f6f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921007dbb0 con 0x56074131f790 2024-03-28T01:04:28.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.514+0000 7f9217f6f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921007bc50 con 0x56074131eda0 2024-03-28T01:04:28.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.515+0000 7f920f7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560741324900 0x560741318ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.515+0000 7f920f7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560741324900 0x560741318ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39314/0 (socket says 172.21.15.169:39314) 2024-03-28T01:04:28.519 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.515+0000 7f920f7fe640 1 -- 172.21.15.169:0/1184807566 learned_addr learned my addr 172.21.15.169:0/1184807566 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:28.519 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.516+0000 7f920f7fe640 1 -- 172.21.15.169:0/1184807566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56074131eda0 msgr2=0x560741329630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:28.519 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.516+0000 7f920f7fe640 1 --2- 172.21.15.169:0/1184807566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56074131eda0 0x560741329630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.519 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.516+0000 7f920f7fe640 1 -- 172.21.15.169:0/1184807566 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56074131f790 msgr2=0x560741330850 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:04:28.519 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.516+0000 7f920f7fe640 1 --2- 172.21.15.169:0/1184807566 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56074131f790 0x560741330850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.519 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.516+0000 7f920f7fe640 1 -- 172.21.15.169:0/1184807566 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560741324010 con 0x560741324900 2024-03-28T01:04:28.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.518+0000 7f920f7fe640 1 --2- 172.21.15.169:0/1184807566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560741324900 0x560741318ca0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9208012f90 tx=0x7f9208019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=553a665a854343ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:28.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.523+0000 7f9204ff9640 1 -- 172.21.15.169:0/1184807566 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920800f040 con 0x560741324900 2024-03-28T01:04:28.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.523+0000 7f9204ff9640 1 -- 172.21.15.169:0/1184807566 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9208025070 con 0x560741324900 2024-03-28T01:04:28.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.524+0000 7f9204ff9640 1 -- 172.21.15.169:0/1184807566 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9208020db0 con 0x560741324900 2024-03-28T01:04:28.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.524+0000 7f9217f6f740 1 -- 172.21.15.169:0/1184807566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560741324900 msgr2=0x560741318ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:28.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.524+0000 7f9217f6f740 1 --2- 172.21.15.169:0/1184807566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560741324900 0x560741318ca0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9208012f90 tx=0x7f9208019000 comp rx=0 tx=0).stop 2024-03-28T01:04:28.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.524+0000 7f9217f6f740 1 -- 172.21.15.169:0/1184807566 shutdown_connections 2024-03-28T01:04:28.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.524+0000 7f9217f6f740 1 --2- 172.21.15.169:0/1184807566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560741324900 0x560741318ca0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.524+0000 7f9217f6f740 1 --2- 172.21.15.169:0/1184807566 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56074131f790 0x560741330850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.524+0000 7f9217f6f740 1 --2- 172.21.15.169:0/1184807566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56074131eda0 0x560741329630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.524+0000 7f9217f6f740 1 -- 172.21.15.169:0/1184807566 >> 172.21.15.169:0/1184807566 conn(0x56074131a310 msgr2=0x560741328f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:28.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.526+0000 7f9217f6f740 1 -- 172.21.15.169:0/1184807566 shutdown_connections 2024-03-28T01:04:28.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.526+0000 7f9217f6f740 1 -- 172.21.15.169:0/1184807566 wait complete. 2024-03-28T01:04:28.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.526+0000 7f9217f6f740 1 -- start start 2024-03-28T01:04:28.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.526+0000 7f9217f6f740 1 Processor -- start 2024-03-28T01:04:28.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.530+0000 7f9217f6f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56074131eda0 0x56074132daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.530+0000 7f9217f6f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56074131f790 0x56074132dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.530+0000 7f9217f6f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560741324900 0x56074132efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.530+0000 7f9217f6f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921007c250 con 0x56074131f790 2024-03-28T01:04:28.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.530+0000 7f9217f6f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921007d550 con 0x560741324900 2024-03-28T01:04:28.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.530+0000 7f9217f6f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921007c890 con 0x56074131eda0 2024-03-28T01:04:28.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.530+0000 7f9217f6f740 10 client.0 _ll_register_callbacks cb 0x7f920009b6f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:04:28.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.532+0000 7f920e7fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56074131f790 0x56074132dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.532+0000 7f920e7fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56074131f790 0x56074132dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39328/0 (socket says 172.21.15.169:39328) 2024-03-28T01:04:28.535 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.532+0000 7f920e7fc640 1 -- 172.21.15.169:0/242474884 learned_addr learned my addr 172.21.15.169:0/242474884 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:28.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.535+0000 7f920effd640 1 --2- 172.21.15.169:0/242474884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56074131eda0 0x56074132daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.535+0000 7f920f7fe640 1 --2- 172.21.15.169:0/242474884 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560741324900 0x56074132efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.535+0000 7f920e7fc640 1 -- 172.21.15.169:0/242474884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56074131eda0 msgr2=0x56074132daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:28.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.535+0000 7f920e7fc640 1 --2- 172.21.15.169:0/242474884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56074131eda0 0x56074132daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.535+0000 7f920e7fc640 1 -- 172.21.15.169:0/242474884 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560741324900 msgr2=0x56074132efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:28.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.535+0000 7f920e7fc640 1 --2- 172.21.15.169:0/242474884 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560741324900 0x56074132efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.535+0000 7f920e7fc640 1 -- 172.21.15.169:0/242474884 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56074132f590 con 0x56074131f790 2024-03-28T01:04:28.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.539+0000 7f920e7fc640 1 --2- 172.21.15.169:0/242474884 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56074131f790 0x56074132dfe0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f91f8005c40 tx=0x7f91f8005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8421ad996dce028 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:28.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.539+0000 7f91f67fc640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:04:28.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.543+0000 7f91f67fc640 1 -- 172.21.15.169:0/242474884 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f8050020 con 0x56074131f790 2024-03-28T01:04:28.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.543+0000 7f91f67fc640 1 -- 172.21.15.169:0/242474884 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f91f8004930 con 0x56074131f790 2024-03-28T01:04:28.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.545+0000 7f9217f6f740 1 -- 172.21.15.169:0/242474884 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9208025070 con 0x56074131f790 2024-03-28T01:04:28.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.547+0000 7f91f67fc640 1 -- 172.21.15.169:0/242474884 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91f80046c0 con 0x56074131f790 2024-03-28T01:04:28.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.548+0000 7f9217f6f740 10 client.17040 fetch_fsmap learned FSMap version 25 2024-03-28T01:04:28.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.548+0000 7f9217f6f740 1 -- 172.21.15.169:0/242474884 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x560741330940 con 0x56074131f790 2024-03-28T01:04:28.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.549+0000 7f91f67fc640 1 -- 172.21.15.169:0/242474884 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9208025070 con 0x56074131f790 2024-03-28T01:04:28.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.549+0000 7f91f67fc640 1 -- 172.21.15.169:0/242474884 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f91f803c070 con 0x56074131f790 2024-03-28T01:04:28.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.549+0000 7f9217f6f740 10 client.17040 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:04:28.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.549+0000 7f9217f6f740 10 client.17040 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:04:28.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.549+0000 7f9217f6f740 20 client.17040 populate_metadata read hostname 'smithi169' 2024-03-28T01:04:28.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.549+0000 7f9217f6f740 1 --2- 172.21.15.169:0/242474884 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5607413b2d40 0x5607413b5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:28.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.549+0000 7f9217f6f740 4 client.17040 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:04:28.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.549+0000 7f9217f6f740 1 -- 172.21.15.169:0/242474884 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56074128bc60 con 0x5607413b2d40 2024-03-28T01:04:28.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.551+0000 7f920effd640 1 --2- 172.21.15.169:0/242474884 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5607413b2d40 0x5607413b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:28.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.561+0000 7f920effd640 1 --2- 172.21.15.169:0/242474884 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5607413b2d40 0x5607413b5130 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28a0e5d6141f9ed5 server_cookie=d2bd6cdcd979f78a in_seq=0 out_seq=0 2024-03-28T01:04:28.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.562+0000 7f91f67fc640 10 client.17040 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:28.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.564+0000 7f91f67fc640 1 -- 172.21.15.169:0/242474884 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56074128bc60 con 0x5607413b2d40 2024-03-28T01:04:28.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.564+0000 7f91f67fc640 10 client.17040 handle_command_reply: tid=0 2024-03-28T01:04:28.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.565+0000 7f9217f6f740 1 client.17040 shutdown 2024-03-28T01:04:28.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.571+0000 7f9217f6f740 1 -- 172.21.15.169:0/242474884 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56074131f790 msgr2=0x56074132dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:28.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.571+0000 7f9217f6f740 1 --2- 172.21.15.169:0/242474884 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56074131f790 0x56074132dfe0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f91f8005c40 tx=0x7f91f8005e80 comp rx=0 tx=0).stop 2024-03-28T01:04:28.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.571+0000 7f9217f6f740 1 -- 172.21.15.169:0/242474884 shutdown_connections 2024-03-28T01:04:28.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.571+0000 7f9217f6f740 1 --2- 172.21.15.169:0/242474884 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560741324900 0x56074132efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.571+0000 7f9217f6f740 1 --2- 172.21.15.169:0/242474884 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5607413b2d40 0x5607413b5130 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.571+0000 7f9217f6f740 1 --2- 172.21.15.169:0/242474884 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56074131f790 0x56074132dfe0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.571+0000 7f9217f6f740 1 --2- 172.21.15.169:0/242474884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56074131eda0 0x56074132daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.571+0000 7f9217f6f740 1 -- 172.21.15.169:0/242474884 >> 172.21.15.169:0/242474884 conn(0x56074131a310 msgr2=0x56074131d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:28.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.571+0000 7f91f67fc640 0 client.17040 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:28.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.571+0000 7f9217f6f740 1 -- 172.21.15.169:0/242474884 shutdown_connections 2024-03-28T01:04:28.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.571+0000 7f9217f6f740 1 -- 172.21.15.169:0/242474884 wait complete. 2024-03-28T01:04:28.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.572+0000 7f9217f6f740 20 client.17040 trim_cache size 0 max 16384 2024-03-28T01:04:28.576 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:04:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T01:04:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": { 2024-03-28T01:04:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: "924ab91a-d205-4422-8e64-30329ff89fea": { 2024-03-28T01:04:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: "path": "/", 2024-03-28T01:04:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: "tag": "924ab91a-d205-4422-8e64-30329ff89fea", 2024-03-28T01:04:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: "options": "recursive,force" 2024-03-28T01:04:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:04:28.576 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:04:28.576 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:04:28.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.572+0000 7f921537e640 1 -- 172.21.15.169:0/84850341 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f91fc0a2a00 msgr2=0x7f91fc0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:28.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.572+0000 7f921537e640 1 --2- 172.21.15.169:0/84850341 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f91fc0a2a00 0x7f91fc0a4ec0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9208009d50 tx=0x7f9208009690 comp rx=0 tx=0).stop 2024-03-28T01:04:28.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.572+0000 7f921537e640 1 -- 172.21.15.169:0/84850341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92101086a0 msgr2=0x7f9210114dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:28.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.572+0000 7f921537e640 1 --2- 172.21.15.169:0/84850341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92101086a0 0x7f9210114dd0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f920000ebc0 tx=0x7f920000c6a0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.572+0000 7f921537e640 1 -- 172.21.15.169:0/84850341 shutdown_connections 2024-03-28T01:04:28.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.572+0000 7f921537e640 1 --2- 172.21.15.169:0/84850341 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f91fc0a2a00 0x7f91fc0a4ec0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.572+0000 7f921537e640 1 --2- 172.21.15.169:0/84850341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f921010d9c0 0x7f92101b68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.572+0000 7f921537e640 1 --2- 172.21.15.169:0/84850341 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f921010b010 0x7f9210115310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.572+0000 7f921537e640 1 --2- 172.21.15.169:0/84850341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92101086a0 0x7f9210114dd0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:28.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.572+0000 7f921537e640 1 -- 172.21.15.169:0/84850341 >> 172.21.15.169:0/84850341 conn(0x7f921006e500 msgr2=0x7f9210109cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:28.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.573+0000 7f921537e640 1 -- 172.21.15.169:0/84850341 shutdown_connections 2024-03-28T01:04:28.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:04:28.573+0000 7f921537e640 1 -- 172.21.15.169:0/84850341 wait complete. 2024-03-28T01:04:28.600 INFO:tasks.cephfs.filesystem:scrub status for tag:924ab91a-d205-4422-8e64-30329ff89fea - {'path': '/', 'tag': '924ab91a-d205-4422-8e64-30329ff89fea', 'options': 'recursive,force'} 2024-03-28T01:04:29.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:29 smithi174 ceph-mon[36060]: scrub queued for path: / 2024-03-28T01:04:29.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:29 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:04:29.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:29 smithi174 ceph-mon[36060]: scrub summary: active paths [/] 2024-03-28T01:04:29.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:29 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:04:29.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:29 smithi174 ceph-mon[36060]: pgmap v590: 129 pgs: 129 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 144 MiB/s rd, 317 MiB/s wr, 4.18k op/s 2024-03-28T01:04:29.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:29 smithi174 ceph-mon[36060]: scrub summary: idle 2024-03-28T01:04:29.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:29 smithi177 ceph-mon[36206]: scrub queued for path: / 2024-03-28T01:04:29.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:29 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:04:29.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:29 smithi177 ceph-mon[36206]: scrub summary: active paths [/] 2024-03-28T01:04:29.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:29 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:04:29.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:29 smithi177 ceph-mon[36206]: pgmap v590: 129 pgs: 129 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 144 MiB/s rd, 317 MiB/s wr, 4.18k op/s 2024-03-28T01:04:29.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:29 smithi177 ceph-mon[36206]: scrub summary: idle 2024-03-28T01:04:30.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:29 smithi169 ceph-mon[32274]: scrub queued for path: / 2024-03-28T01:04:30.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:29 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:04:30.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:29 smithi169 ceph-mon[32274]: scrub summary: active paths [/] 2024-03-28T01:04:30.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:29 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:04:30.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:29 smithi169 ceph-mon[32274]: pgmap v590: 129 pgs: 129 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 144 MiB/s rd, 317 MiB/s wr, 4.18k op/s 2024-03-28T01:04:30.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:29 smithi169 ceph-mon[32274]: scrub summary: idle 2024-03-28T01:04:31.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:31 smithi177 ceph-mon[36206]: pgmap v591: 129 pgs: 129 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 146 MiB/s rd, 367 MiB/s wr, 4.53k op/s 2024-03-28T01:04:31.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:31 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:04:32.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:31 smithi169 ceph-mon[32274]: pgmap v591: 129 pgs: 129 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 146 MiB/s rd, 367 MiB/s wr, 4.53k op/s 2024-03-28T01:04:32.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:31 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:04:32.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:31 smithi174 ceph-mon[36060]: pgmap v591: 129 pgs: 129 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 146 MiB/s rd, 367 MiB/s wr, 4.53k op/s 2024-03-28T01:04:32.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:31 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:04:33.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:33 smithi177 ceph-mon[36206]: pgmap v592: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 132 MiB/s rd, 347 MiB/s wr, 4.22k op/s 2024-03-28T01:04:33.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:04:33.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:04:33.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:04:33.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:04:33.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:04:33.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:04:33.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:04:34.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:33 smithi169 ceph-mon[32274]: pgmap v592: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 132 MiB/s rd, 347 MiB/s wr, 4.22k op/s 2024-03-28T01:04:34.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:04:34.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:04:34.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:04:34.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:04:34.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:04:34.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:04:34.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:04:34.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:33 smithi174 ceph-mon[36060]: pgmap v592: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 132 MiB/s rd, 347 MiB/s wr, 4.22k op/s 2024-03-28T01:04:34.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:04:34.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:04:34.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:04:34.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T01:04:34.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:04:34.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:04:34.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:04:34.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:34 smithi177 ceph-mon[36206]: Adjusting osd_memory_target on smithi177 to 174.1M 2024-03-28T01:04:34.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:34 smithi177 ceph-mon[36206]: Unable to set osd_memory_target on smithi177 to 182575411: error parsing value: Value '182575411' is below minimum 939524096 2024-03-28T01:04:35.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:34 smithi169 ceph-mon[32274]: Adjusting osd_memory_target on smithi177 to 174.1M 2024-03-28T01:04:35.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:34 smithi169 ceph-mon[32274]: Unable to set osd_memory_target on smithi177 to 182575411: error parsing value: Value '182575411' is below minimum 939524096 2024-03-28T01:04:35.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:34 smithi174 ceph-mon[36060]: Adjusting osd_memory_target on smithi177 to 174.1M 2024-03-28T01:04:35.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:34 smithi174 ceph-mon[36060]: Unable to set osd_memory_target on smithi177 to 182575411: error parsing value: Value '182575411' is below minimum 939524096 2024-03-28T01:04:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.665+0000 7fcbc089d640 1 -- 172.21.15.169:0/1420489763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbbc076270 msgr2=0x7fcbbc0786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.665+0000 7fcbc089d640 1 --2- 172.21.15.169:0/1420489763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbbc076270 0x7fcbbc0786e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0009900 tx=0x7fcbb002bc40 comp rx=0 tx=0).stop 2024-03-28T01:04:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.665+0000 7fcbc089d640 1 -- 172.21.15.169:0/1420489763 shutdown_connections 2024-03-28T01:04:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.665+0000 7fcbc089d640 1 --2- 172.21.15.169:0/1420489763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbbc078c20 0x7fcbbc07b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.665+0000 7fcbc089d640 1 --2- 172.21.15.169:0/1420489763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbbc076270 0x7fcbbc0786e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.665+0000 7fcbc089d640 1 --2- 172.21.15.169:0/1420489763 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbbc073390 0x7fcbbc0737b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.665+0000 7fcbc089d640 1 -- 172.21.15.169:0/1420489763 >> 172.21.15.169:0/1420489763 conn(0x7fcbbc06e510 msgr2=0x7fcbbc070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.666+0000 7fcbc089d640 1 -- 172.21.15.169:0/1420489763 shutdown_connections 2024-03-28T01:04:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.666+0000 7fcbc089d640 1 -- 172.21.15.169:0/1420489763 wait complete. 2024-03-28T01:04:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.667+0000 7fcbc089d640 1 Processor -- start 2024-03-28T01:04:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.667+0000 7fcbc089d640 1 -- start start 2024-03-28T01:04:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.668+0000 7fcbc089d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbbc073390 0x7fcbbc07fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.668+0000 7fcbc089d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbbc076270 0x7fcbbc080410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:35.673 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.668+0000 7fcbc089d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbbc078c20 0x7fcbbc0834a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:35.673 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.668+0000 7fcbc089d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc07eab0 con 0x7fcbbc073390 2024-03-28T01:04:35.673 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.668+0000 7fcbc089d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc07e930 con 0x7fcbbc076270 2024-03-28T01:04:35.673 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.668+0000 7fcbc089d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc07ec30 con 0x7fcbbc078c20 2024-03-28T01:04:35.673 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.668+0000 7fcbbaffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbbc076270 0x7fcbbc080410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:35.673 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.668+0000 7fcbbb7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbbc073390 0x7fcbbc07fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:35.673 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.668+0000 7fcbbbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbbc078c20 0x7fcbbc0834a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:35.673 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.668+0000 7fcbbb7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbbc073390 0x7fcbbc07fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39350/0 (socket says 172.21.15.169:39350) 2024-03-28T01:04:35.673 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.668+0000 7fcbbb7fe640 1 -- 172.21.15.169:0/1101078124 learned_addr learned my addr 172.21.15.169:0/1101078124 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:35.673 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.669+0000 7fcbbb7fe640 1 -- 172.21.15.169:0/1101078124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbbc078c20 msgr2=0x7fcbbc0834a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:35.673 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.669+0000 7fcbbb7fe640 1 --2- 172.21.15.169:0/1101078124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbbc078c20 0x7fcbbc0834a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.674 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.669+0000 7fcbbb7fe640 1 -- 172.21.15.169:0/1101078124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbbc076270 msgr2=0x7fcbbc080410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:35.674 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.669+0000 7fcbbb7fe640 1 --2- 172.21.15.169:0/1101078124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbbc076270 0x7fcbbc080410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.674 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.669+0000 7fcbbb7fe640 1 -- 172.21.15.169:0/1101078124 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbbc083b40 con 0x7fcbbc073390 2024-03-28T01:04:35.675 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.670+0000 7fcbbb7fe640 1 --2- 172.21.15.169:0/1101078124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbbc073390 0x7fcbbc07fed0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fcbb400ebc0 tx=0x7fcbb400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:35.675 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.670+0000 7fcbb8ff9640 1 -- 172.21.15.169:0/1101078124 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4010630 con 0x7fcbbc073390 2024-03-28T01:04:35.675 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.670+0000 7fcbc089d640 1 -- 172.21.15.169:0/1101078124 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbbc083e30 con 0x7fcbbc073390 2024-03-28T01:04:35.675 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.670+0000 7fcbb8ff9640 1 -- 172.21.15.169:0/1101078124 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbb4010cb0 con 0x7fcbbc073390 2024-03-28T01:04:35.675 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.670+0000 7fcbb8ff9640 1 -- 172.21.15.169:0/1101078124 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4018c00 con 0x7fcbbc073390 2024-03-28T01:04:35.676 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.671+0000 7fcbc089d640 1 -- 172.21.15.169:0/1101078124 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbbc0843a0 con 0x7fcbbc073390 2024-03-28T01:04:35.678 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.675+0000 7fcbb8ff9640 1 -- 172.21.15.169:0/1101078124 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fcbb40107d0 con 0x7fcbbc073390 2024-03-28T01:04:35.679 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.676+0000 7fcbb8ff9640 1 --2- 172.21.15.169:0/1101078124 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcba40a2940 0x7fcba40a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.677+0000 7fcbb8ff9640 1 -- 172.21.15.169:0/1101078124 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fcbb40cf300 con 0x7fcbbc073390 2024-03-28T01:04:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.677+0000 7fcbc47ca740 1 Processor -- start 2024-03-28T01:04:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.677+0000 7fcbc47ca740 1 -- start start 2024-03-28T01:04:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.677+0000 7fcbbaffd640 1 --2- 172.21.15.169:0/1101078124 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcba40a2940 0x7fcba40a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.677+0000 7fcbc47ca740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565188a7b20 0x5565188a7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.677+0000 7fcbc47ca740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565188a8510 0x5565188ad010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.677+0000 7fcbc47ca740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188ad680 0x5565188afb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.677+0000 7fcbc47ca740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55651880cb10 con 0x5565188a8510 2024-03-28T01:04:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.677+0000 7fcbc47ca740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565187b5900 con 0x5565188ad680 2024-03-28T01:04:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.677+0000 7fcbc47ca740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x556518808bb0 con 0x5565188a7b20 2024-03-28T01:04:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.678+0000 7fcbbbfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188ad680 0x5565188afb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.678+0000 7fcbbbfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188ad680 0x5565188afb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:47260/0 (socket says 172.21.15.169:47260) 2024-03-28T01:04:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.678+0000 7fcbbbfff640 1 -- 172.21.15.169:0/95022583 learned_addr learned my addr 172.21.15.169:0/95022583 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.678+0000 7fcbbb7fe640 1 --2- 172.21.15.169:0/95022583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565188a7b20 0x5565188a7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.678+0000 7fcbbaffd640 1 --2- 172.21.15.169:0/95022583 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565188a8510 0x5565188ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.678+0000 7fcbbaffd640 1 --2- 172.21.15.169:0/1101078124 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcba40a2940 0x7fcba40a4e00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fcbb002c250 tx=0x7fcbb00095b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.678+0000 7fcbbbfff640 1 -- 172.21.15.169:0/95022583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565188a7b20 msgr2=0x5565188a7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.678+0000 7fcbbbfff640 1 --2- 172.21.15.169:0/95022583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565188a7b20 0x5565188a7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.678+0000 7fcbbbfff640 1 -- 172.21.15.169:0/95022583 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565188a8510 msgr2=0x5565188ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.678+0000 7fcbbbfff640 1 --2- 172.21.15.169:0/95022583 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565188a8510 0x5565188ad010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.678+0000 7fcbbbfff640 1 -- 172.21.15.169:0/95022583 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556518521210 con 0x5565188ad680 2024-03-28T01:04:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.679+0000 7fcbbbfff640 1 --2- 172.21.15.169:0/95022583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188ad680 0x5565188afb10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcbac00b700 tx=0x7fcbac00bbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=380075c7ec68208e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.679+0000 7fcba1ffb640 1 -- 172.21.15.169:0/95022583 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbac003dd0 con 0x5565188ad680 2024-03-28T01:04:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.679+0000 7fcba1ffb640 1 -- 172.21.15.169:0/95022583 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbac015070 con 0x5565188ad680 2024-03-28T01:04:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.679+0000 7fcba1ffb640 1 -- 172.21.15.169:0/95022583 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbac0106a0 con 0x5565188ad680 2024-03-28T01:04:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.680+0000 7fcbc47ca740 1 -- 172.21.15.169:0/95022583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188ad680 msgr2=0x5565188afb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.680+0000 7fcbc47ca740 1 --2- 172.21.15.169:0/95022583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188ad680 0x5565188afb10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcbac00b700 tx=0x7fcbac00bbc0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.680+0000 7fcbc47ca740 1 -- 172.21.15.169:0/95022583 shutdown_connections 2024-03-28T01:04:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.680+0000 7fcbc47ca740 1 --2- 172.21.15.169:0/95022583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188ad680 0x5565188afb10 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.680+0000 7fcbc47ca740 1 --2- 172.21.15.169:0/95022583 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565188a8510 0x5565188ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.680+0000 7fcbc47ca740 1 --2- 172.21.15.169:0/95022583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565188a7b20 0x5565188a7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.680+0000 7fcbc47ca740 1 -- 172.21.15.169:0/95022583 >> 172.21.15.169:0/95022583 conn(0x5565188a3090 msgr2=0x5565188a3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.680+0000 7fcbc47ca740 1 -- 172.21.15.169:0/95022583 shutdown_connections 2024-03-28T01:04:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.680+0000 7fcbc47ca740 1 -- 172.21.15.169:0/95022583 wait complete. 2024-03-28T01:04:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.680+0000 7fcbc47ca740 1 -- start start 2024-03-28T01:04:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.681+0000 7fcbc47ca740 1 Processor -- start 2024-03-28T01:04:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.681+0000 7fcbc47ca740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565188a7b20 0x5565188b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:35.692 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.681+0000 7fcbc47ca740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188a8510 0x5565188b2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:35.692 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.681+0000 7fcbc47ca740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565188ad680 0x5565188b93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:35.692 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.681+0000 7fcbc47ca740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565188b0630 con 0x5565188a7b20 2024-03-28T01:04:35.692 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.681+0000 7fcbc47ca740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565187c8d50 con 0x5565188a8510 2024-03-28T01:04:35.692 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.681+0000 7fcbc47ca740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565188b07b0 con 0x5565188ad680 2024-03-28T01:04:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.681+0000 7fcbc47ca740 10 client.0 _ll_register_callbacks cb 0x5565187c8720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:04:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.681+0000 7fcbbaffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188a8510 0x5565188b2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.681+0000 7fcbbaffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188a8510 0x5565188b2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:47264/0 (socket says 172.21.15.169:47264) 2024-03-28T01:04:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.681+0000 7fcbbaffd640 1 -- 172.21.15.169:0/1804047615 learned_addr learned my addr 172.21.15.169:0/1804047615 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.682+0000 7fcbbb7fe640 1 --2- 172.21.15.169:0/1804047615 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565188a7b20 0x5565188b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.682+0000 7fcbbaffd640 1 -- 172.21.15.169:0/1804047615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565188ad680 msgr2=0x5565188b93a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:04:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.682+0000 7fcbbaffd640 1 --2- 172.21.15.169:0/1804047615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565188ad680 0x5565188b93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.682+0000 7fcbbaffd640 1 -- 172.21.15.169:0/1804047615 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565188a7b20 msgr2=0x5565188b1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.682+0000 7fcbbaffd640 1 --2- 172.21.15.169:0/1804047615 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565188a7b20 0x5565188b1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.682+0000 7fcbbaffd640 1 -- 172.21.15.169:0/1804047615 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5565185ed620 con 0x5565188a8510 2024-03-28T01:04:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.682+0000 7fcbbaffd640 1 --2- 172.21.15.169:0/1804047615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188a8510 0x5565188b2360 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0009900 tx=0x7fcbb002bc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cbcf1197dfe1d00 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.682+0000 7fcb9affd640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:04:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.683+0000 7fcb9affd640 1 -- 172.21.15.169:0/1804047615 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb00053c0 con 0x5565188a8510 2024-03-28T01:04:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.683+0000 7fcb9affd640 1 -- 172.21.15.169:0/1804047615 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fcbb0005560 con 0x5565188a8510 2024-03-28T01:04:35.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.683+0000 7fcbc47ca740 1 -- 172.21.15.169:0/1804047615 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcbac015070 con 0x5565188a8510 2024-03-28T01:04:35.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.683+0000 7fcb9affd640 1 -- 172.21.15.169:0/1804047615 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbb00046c0 con 0x5565188a8510 2024-03-28T01:04:35.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.683+0000 7fcbc47ca740 10 client.26918 fetch_fsmap learned FSMap version 25 2024-03-28T01:04:35.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.683+0000 7fcbc47ca740 1 -- 172.21.15.169:0/1804047615 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5565188bc780 con 0x5565188a8510 2024-03-28T01:04:35.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.684+0000 7fcb9affd640 1 -- 172.21.15.169:0/1804047615 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbb0004d40 con 0x5565188a8510 2024-03-28T01:04:35.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.684+0000 7fcb9affd640 1 -- 172.21.15.169:0/1804047615 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fcbb0034440 con 0x5565188a8510 2024-03-28T01:04:35.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.684+0000 7fcbc47ca740 10 client.26918 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:04:35.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.684+0000 7fcbc47ca740 10 client.26918 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:04:35.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.684+0000 7fcbc47ca740 20 client.26918 populate_metadata read hostname 'smithi169' 2024-03-28T01:04:35.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.684+0000 7fcbc47ca740 1 --2- 172.21.15.169:0/1804047615 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5565188bde10 0x5565188c0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:35.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.684+0000 7fcbc47ca740 4 client.26918 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:04:35.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.684+0000 7fcbc47ca740 1 -- 172.21.15.169:0/1804047615 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbac015070 con 0x5565188bde10 2024-03-28T01:04:35.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.684+0000 7fcbbb7fe640 1 --2- 172.21.15.169:0/1804047615 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5565188bde10 0x5565188c0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:35.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.685+0000 7fcb9affd640 10 client.26918 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:35.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.685+0000 7fcbbb7fe640 1 --2- 172.21.15.169:0/1804047615 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5565188bde10 0x5565188c0240 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fece6c5a4921792a server_cookie=5c839c9f17324a1a in_seq=0 out_seq=0 2024-03-28T01:04:35.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.686+0000 7fcb9affd640 1 -- 172.21.15.169:0/1804047615 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7fcbac015070 con 0x5565188bde10 2024-03-28T01:04:35.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.686+0000 7fcb9affd640 10 client.26918 handle_command_reply: tid=0 2024-03-28T01:04:35.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.687+0000 7fcbc47ca740 1 client.26918 shutdown 2024-03-28T01:04:35.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.687+0000 7fcbc47ca740 1 -- 172.21.15.169:0/1804047615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188a8510 msgr2=0x5565188b2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.687+0000 7fcbc47ca740 1 --2- 172.21.15.169:0/1804047615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188a8510 0x5565188b2360 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0009900 tx=0x7fcbb002bc40 comp rx=0 tx=0).stop 2024-03-28T01:04:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.688+0000 7fcbc47ca740 1 -- 172.21.15.169:0/1804047615 shutdown_connections 2024-03-28T01:04:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.688+0000 7fcbc47ca740 1 --2- 172.21.15.169:0/1804047615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565188ad680 0x5565188b93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.688+0000 7fcbc47ca740 1 --2- 172.21.15.169:0/1804047615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188a8510 0x5565188b2360 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.688+0000 7fcbc47ca740 1 --2- 172.21.15.169:0/1804047615 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5565188bde10 0x5565188c0240 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.688+0000 7fcb9affd640 0 client.26918 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.688+0000 7fcbc47ca740 1 --2- 172.21.15.169:0/1804047615 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565188a7b20 0x5565188b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.688+0000 7fcbc47ca740 1 -- 172.21.15.169:0/1804047615 >> 172.21.15.169:0/1804047615 conn(0x5565188a3090 msgr2=0x5565188a2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.688+0000 7fcbc47ca740 1 -- 172.21.15.169:0/1804047615 shutdown_connections 2024-03-28T01:04:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.688+0000 7fcbc47ca740 1 -- 172.21.15.169:0/1804047615 wait complete. 2024-03-28T01:04:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.688+0000 7fcbc47ca740 20 client.26918 trim_cache size 0 max 16384 2024-03-28T01:04:35.711 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.708+0000 7fcbc47ca740 1 Processor -- start 2024-03-28T01:04:35.711 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.708+0000 7fcbc47ca740 1 -- start start 2024-03-28T01:04:35.711 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.709+0000 7fcbc47ca740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188a7b20 0x5565188a3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.709+0000 7fcbc47ca740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565188a8510 0x5565188b9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.709+0000 7fcbc47ca740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565188ad680 0x5565188a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.709+0000 7fcbc47ca740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc1c3cc0 con 0x5565188a8510 2024-03-28T01:04:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.709+0000 7fcbc47ca740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc1c17c0 con 0x5565188a7b20 2024-03-28T01:04:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.709+0000 7fcbc47ca740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc1bf860 con 0x5565188ad680 2024-03-28T01:04:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.709+0000 7fcbbbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565188ad680 0x5565188a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.709+0000 7fcbbbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565188ad680 0x5565188a62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56672/0 (socket says 172.21.15.169:56672) 2024-03-28T01:04:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.709+0000 7fcbbbfff640 1 -- 172.21.15.169:0/1451463326 learned_addr learned my addr 172.21.15.169:0/1451463326 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.709+0000 7fcbbb7fe640 1 --2- 172.21.15.169:0/1451463326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188a7b20 0x5565188a3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.709+0000 7fcbbaffd640 1 --2- 172.21.15.169:0/1451463326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565188a8510 0x5565188b9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.710+0000 7fcbbaffd640 1 -- 172.21.15.169:0/1451463326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565188ad680 msgr2=0x5565188a62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.710+0000 7fcbbaffd640 1 --2- 172.21.15.169:0/1451463326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565188ad680 0x5565188a62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.710+0000 7fcbbaffd640 1 -- 172.21.15.169:0/1451463326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188a7b20 msgr2=0x5565188a3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.710+0000 7fcbbaffd640 1 --2- 172.21.15.169:0/1451463326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188a7b20 0x5565188a3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.710+0000 7fcbbaffd640 1 -- 172.21.15.169:0/1451463326 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565188acad0 con 0x5565188a8510 2024-03-28T01:04:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.710+0000 7fcbbbfff640 1 --2- 172.21.15.169:0/1451463326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565188ad680 0x5565188a62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:04:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.710+0000 7fcbbaffd640 1 --2- 172.21.15.169:0/1451463326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565188a8510 0x5565188b9360 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0036040 tx=0x7fcbb002ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd35f06c8ba6d45 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.711+0000 7fcba17fa640 1 -- 172.21.15.169:0/1451463326 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0034640 con 0x5565188a8510 2024-03-28T01:04:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.711+0000 7fcba17fa640 1 -- 172.21.15.169:0/1451463326 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbb00051c0 con 0x5565188a8510 2024-03-28T01:04:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.711+0000 7fcba17fa640 1 -- 172.21.15.169:0/1451463326 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0004c20 con 0x5565188a8510 2024-03-28T01:04:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.712+0000 7fcbc47ca740 1 -- 172.21.15.169:0/1451463326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565188a8510 msgr2=0x5565188b9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.712+0000 7fcbc47ca740 1 --2- 172.21.15.169:0/1451463326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565188a8510 0x5565188b9360 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0036040 tx=0x7fcbb002ce80 comp rx=0 tx=0).stop 2024-03-28T01:04:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.713+0000 7fcbc47ca740 1 -- 172.21.15.169:0/1451463326 shutdown_connections 2024-03-28T01:04:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.713+0000 7fcbc47ca740 1 --2- 172.21.15.169:0/1451463326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565188ad680 0x5565188a62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.713+0000 7fcbc47ca740 1 --2- 172.21.15.169:0/1451463326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565188a8510 0x5565188b9360 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.713+0000 7fcbc47ca740 1 --2- 172.21.15.169:0/1451463326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188a7b20 0x5565188a3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.713+0000 7fcbc47ca740 1 -- 172.21.15.169:0/1451463326 >> 172.21.15.169:0/1451463326 conn(0x5565188a3090 msgr2=0x5565188ace50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.713+0000 7fcbc47ca740 1 -- 172.21.15.169:0/1451463326 shutdown_connections 2024-03-28T01:04:35.716 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.713+0000 7fcbc47ca740 1 -- 172.21.15.169:0/1451463326 wait complete. 2024-03-28T01:04:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.714+0000 7fcbc47ca740 1 -- start start 2024-03-28T01:04:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.714+0000 7fcbc47ca740 1 Processor -- start 2024-03-28T01:04:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.715+0000 7fcbc47ca740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188a7b20 0x5565188b90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:35.718 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.715+0000 7fcbc47ca740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565188ad680 0x5565188b76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.715+0000 7fcbc47ca740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565188bde10 0x5565188b7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.716+0000 7fcbc47ca740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc1bfe60 con 0x5565188ad680 2024-03-28T01:04:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.716+0000 7fcbbb7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188a7b20 0x5565188b90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.716+0000 7fcbbb7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188a7b20 0x5565188b90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:47294/0 (socket says 172.21.15.169:47294) 2024-03-28T01:04:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.716+0000 7fcbbb7fe640 1 -- 172.21.15.169:0/3404392745 learned_addr learned my addr 172.21.15.169:0/3404392745 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.716+0000 7fcbc47ca740 1 -- 172.21.15.169:0/3404392745 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc1c1160 con 0x5565188a7b20 2024-03-28T01:04:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.716+0000 7fcbc47ca740 1 -- 172.21.15.169:0/3404392745 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc1c04a0 con 0x5565188bde10 2024-03-28T01:04:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.716+0000 7fcbc47ca740 10 client.0 _ll_register_callbacks cb 0x7fcbb00040f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:04:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.716+0000 7fcbbbfff640 1 --2- 172.21.15.169:0/3404392745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565188bde10 0x5565188b7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.717+0000 7fcbbaffd640 1 --2- 172.21.15.169:0/3404392745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565188ad680 0x5565188b76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.717+0000 7fcbbb7fe640 1 -- 172.21.15.169:0/3404392745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565188bde10 msgr2=0x5565188b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.717+0000 7fcbbb7fe640 1 --2- 172.21.15.169:0/3404392745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565188bde10 0x5565188b7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.717+0000 7fcbbb7fe640 1 -- 172.21.15.169:0/3404392745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565188ad680 msgr2=0x5565188b76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.717+0000 7fcbbb7fe640 1 --2- 172.21.15.169:0/3404392745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565188ad680 0x5565188b76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.717+0000 7fcbbb7fe640 1 -- 172.21.15.169:0/3404392745 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5565188b8240 con 0x5565188a7b20 2024-03-28T01:04:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.717+0000 7fcbbb7fe640 1 --2- 172.21.15.169:0/3404392745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188a7b20 0x5565188b90b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4097db0 tx=0x7fcbb401a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ce159b8ca0f10c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.717+0000 7fcb9affd640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:04:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.718+0000 7fcb9affd640 1 -- 172.21.15.169:0/3404392745 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb40b8070 con 0x5565188a7b20 2024-03-28T01:04:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.718+0000 7fcbc47ca740 1 -- 172.21.15.169:0/3404392745 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcbb00051c0 con 0x5565188a7b20 2024-03-28T01:04:35.722 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.718+0000 7fcb9affd640 1 -- 172.21.15.169:0/3404392745 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fcbb4097fc0 con 0x5565188a7b20 2024-03-28T01:04:35.722 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.719+0000 7fcb9affd640 1 -- 172.21.15.169:0/3404392745 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbb4010e60 con 0x5565188a7b20 2024-03-28T01:04:35.722 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.719+0000 7fcbc47ca740 10 client.26924 fetch_fsmap learned FSMap version 25 2024-03-28T01:04:35.723 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.719+0000 7fcbc47ca740 1 -- 172.21.15.169:0/3404392745 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55651891a640 con 0x5565188a7b20 2024-03-28T01:04:35.724 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.721+0000 7fcb9affd640 1 -- 172.21.15.169:0/3404392745 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbb00051c0 con 0x5565188a7b20 2024-03-28T01:04:35.724 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.721+0000 7fcb9affd640 1 -- 172.21.15.169:0/3404392745 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fcbb40a3bb0 con 0x5565188a7b20 2024-03-28T01:04:35.724 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.721+0000 7fcbc47ca740 10 client.26924 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:04:35.724 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.721+0000 7fcbc47ca740 10 client.26924 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:04:35.724 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.721+0000 7fcbc47ca740 20 client.26924 populate_metadata read hostname 'smithi169' 2024-03-28T01:04:35.724 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.721+0000 7fcbc47ca740 1 --2- 172.21.15.169:0/3404392745 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55651891b040 0x55651891d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:35.725 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.721+0000 7fcbc47ca740 4 client.26924 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:04:35.725 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.721+0000 7fcbc47ca740 1 -- 172.21.15.169:0/3404392745 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55651881a050 con 0x55651891b040 2024-03-28T01:04:35.725 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.722+0000 7fcbbaffd640 1 --2- 172.21.15.169:0/3404392745 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55651891b040 0x55651891d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:35.727 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.724+0000 7fcbbaffd640 1 --2- 172.21.15.169:0/3404392745 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55651891b040 0x55651891d460 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0ea6b14f89123ac server_cookie=620056ba007bec89 in_seq=0 out_seq=0 2024-03-28T01:04:35.727 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.725+0000 7fcb9affd640 10 client.26924 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:35.755 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.752+0000 7fcb9affd640 1 -- 172.21.15.169:0/3404392745 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55651881a050 con 0x55651891b040 2024-03-28T01:04:35.755 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.752+0000 7fcb9affd640 10 client.26924 handle_command_reply: tid=0 2024-03-28T01:04:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.753+0000 7fcbc47ca740 1 client.26924 shutdown 2024-03-28T01:04:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.753+0000 7fcbc47ca740 1 -- 172.21.15.169:0/3404392745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188a7b20 msgr2=0x5565188b90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.753+0000 7fcbc47ca740 1 --2- 172.21.15.169:0/3404392745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188a7b20 0x5565188b90b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4097db0 tx=0x7fcbb401a040 comp rx=0 tx=0).stop 2024-03-28T01:04:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.753+0000 7fcbc47ca740 1 -- 172.21.15.169:0/3404392745 shutdown_connections 2024-03-28T01:04:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.753+0000 7fcbc47ca740 1 --2- 172.21.15.169:0/3404392745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565188bde10 0x5565188b7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.753+0000 7fcbc47ca740 1 --2- 172.21.15.169:0/3404392745 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55651891b040 0x55651891d460 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.753+0000 7fcbc47ca740 1 --2- 172.21.15.169:0/3404392745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565188ad680 0x5565188b76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.753+0000 7fcbc47ca740 1 --2- 172.21.15.169:0/3404392745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565188a7b20 0x5565188b90b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.753+0000 7fcbc47ca740 1 -- 172.21.15.169:0/3404392745 >> 172.21.15.169:0/3404392745 conn(0x5565188a3090 msgr2=0x5565188a3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.753+0000 7fcb9affd640 0 client.26924 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.754+0000 7fcbc47ca740 1 -- 172.21.15.169:0/3404392745 shutdown_connections 2024-03-28T01:04:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.754+0000 7fcbc47ca740 1 -- 172.21.15.169:0/3404392745 wait complete. 2024-03-28T01:04:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.754+0000 7fcbc47ca740 20 client.26924 trim_cache size 0 max 16384 2024-03-28T01:04:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:04:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:04:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:04:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:04:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.755+0000 7fcbc089d640 1 -- 172.21.15.169:0/1101078124 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcba40a2940 msgr2=0x7fcba40a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.755+0000 7fcbc089d640 1 --2- 172.21.15.169:0/1101078124 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcba40a2940 0x7fcba40a4e00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fcbb002c250 tx=0x7fcbb00095b0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.755+0000 7fcbc089d640 1 -- 172.21.15.169:0/1101078124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbbc073390 msgr2=0x7fcbbc07fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.755+0000 7fcbc089d640 1 --2- 172.21.15.169:0/1101078124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbbc073390 0x7fcbbc07fed0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fcbb400ebc0 tx=0x7fcbb400c6a0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.755+0000 7fcbc089d640 1 -- 172.21.15.169:0/1101078124 shutdown_connections 2024-03-28T01:04:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.755+0000 7fcbc089d640 1 --2- 172.21.15.169:0/1101078124 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fcba40a2940 0x7fcba40a4e00 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.755+0000 7fcbc089d640 1 --2- 172.21.15.169:0/1101078124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbbc078c20 0x7fcbbc0834a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.755+0000 7fcbc089d640 1 --2- 172.21.15.169:0/1101078124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbbc076270 0x7fcbbc080410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.755+0000 7fcbc089d640 1 --2- 172.21.15.169:0/1101078124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcbbc073390 0x7fcbbc07fed0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.755+0000 7fcbc089d640 1 -- 172.21.15.169:0/1101078124 >> 172.21.15.169:0/1101078124 conn(0x7fcbbc06e510 msgr2=0x7fcbbc079bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.755+0000 7fcbc089d640 1 -- 172.21.15.169:0/1101078124 shutdown_connections 2024-03-28T01:04:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:35.755+0000 7fcbc089d640 1 -- 172.21.15.169:0/1101078124 wait complete. 2024-03-28T01:04:35.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:35 smithi177 ceph-mon[36206]: pgmap v593: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 126 MiB/s rd, 329 MiB/s wr, 3.99k op/s 2024-03-28T01:04:36.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:35 smithi169 ceph-mon[32274]: pgmap v593: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 126 MiB/s rd, 329 MiB/s wr, 3.99k op/s 2024-03-28T01:04:36.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:35 smithi174 ceph-mon[36060]: pgmap v593: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 126 MiB/s rd, 329 MiB/s wr, 3.99k op/s 2024-03-28T01:04:37.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:37 smithi177 ceph-mon[36206]: pgmap v594: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 124 MiB/s rd, 332 MiB/s wr, 3.99k op/s 2024-03-28T01:04:38.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:37 smithi169 ceph-mon[32274]: pgmap v594: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 124 MiB/s rd, 332 MiB/s wr, 3.99k op/s 2024-03-28T01:04:38.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:37 smithi174 ceph-mon[36060]: pgmap v594: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 124 MiB/s rd, 332 MiB/s wr, 3.99k op/s 2024-03-28T01:04:39.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:38 smithi169 ceph-mon[32274]: pgmap v595: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s rd, 346 MiB/s wr, 4.16k op/s 2024-03-28T01:04:39.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:38 smithi174 ceph-mon[36060]: pgmap v595: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s rd, 346 MiB/s wr, 4.16k op/s 2024-03-28T01:04:39.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:38 smithi177 ceph-mon[36206]: pgmap v595: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s rd, 346 MiB/s wr, 4.16k op/s 2024-03-28T01:04:41.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:41 smithi174 ceph-mon[36060]: pgmap v596: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 151 MiB/s rd, 417 MiB/s wr, 5.10k op/s 2024-03-28T01:04:41.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:41 smithi177 ceph-mon[36206]: pgmap v596: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 151 MiB/s rd, 417 MiB/s wr, 5.10k op/s 2024-03-28T01:04:42.053 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:41 smithi169 ceph-mon[32274]: pgmap v596: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 151 MiB/s rd, 417 MiB/s wr, 5.10k op/s 2024-03-28T01:04:43.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:43 smithi174 ceph-mon[36060]: pgmap v597: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 141 MiB/s rd, 377 MiB/s wr, 4.86k op/s 2024-03-28T01:04:43.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:43 smithi177 ceph-mon[36206]: pgmap v597: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 141 MiB/s rd, 377 MiB/s wr, 4.86k op/s 2024-03-28T01:04:44.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:43 smithi169 ceph-mon[32274]: pgmap v597: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 141 MiB/s rd, 377 MiB/s wr, 4.86k op/s 2024-03-28T01:04:45.854 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:45 smithi169 ceph-mon[32274]: pgmap v598: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s rd, 342 MiB/s wr, 4.43k op/s 2024-03-28T01:04:45.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:45 smithi174 ceph-mon[36060]: pgmap v598: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s rd, 342 MiB/s wr, 4.43k op/s 2024-03-28T01:04:45.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:45 smithi177 ceph-mon[36206]: pgmap v598: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s rd, 342 MiB/s wr, 4.43k op/s 2024-03-28T01:04:47.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:47 smithi174 ceph-mon[36060]: pgmap v599: 129 pgs: 129 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 129 MiB/s rd, 354 MiB/s wr, 4.59k op/s 2024-03-28T01:04:47.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:47 smithi177 ceph-mon[36206]: pgmap v599: 129 pgs: 129 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 129 MiB/s rd, 354 MiB/s wr, 4.59k op/s 2024-03-28T01:04:48.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:47 smithi169 ceph-mon[32274]: pgmap v599: 129 pgs: 129 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 129 MiB/s rd, 354 MiB/s wr, 4.59k op/s 2024-03-28T01:04:49.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:49 smithi174 ceph-mon[36060]: pgmap v600: 129 pgs: 129 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 132 MiB/s rd, 369 MiB/s wr, 4.82k op/s 2024-03-28T01:04:49.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:49 smithi177 ceph-mon[36206]: pgmap v600: 129 pgs: 129 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 132 MiB/s rd, 369 MiB/s wr, 4.82k op/s 2024-03-28T01:04:50.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:49 smithi169 ceph-mon[32274]: pgmap v600: 129 pgs: 129 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 132 MiB/s rd, 369 MiB/s wr, 4.82k op/s 2024-03-28T01:04:50.369 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.366+0000 7fd84f525640 1 -- 172.21.15.169:0/3722378847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8400a2340 msgr2=0x7fd8400a4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.366+0000 7fd84f525640 1 --2- 172.21.15.169:0/3722378847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8400a2340 0x7fd8400a4730 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd838009890 tx=0x7fd83802bbb0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.371 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.367+0000 7fd84f525640 1 -- 172.21.15.169:0/3722378847 shutdown_connections 2024-03-28T01:04:50.371 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.367+0000 7fd84f525640 1 --2- 172.21.15.169:0/3722378847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8400a2340 0x7fd8400a4730 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.371 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.368+0000 7fd84f525640 1 --2- 172.21.15.169:0/3722378847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd840007ec0 0x7fd840008340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.371 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.368+0000 7fd84f525640 1 --2- 172.21.15.169:0/3722378847 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8400094a0 0x7fd8400078f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.371 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.368+0000 7fd84f525640 1 -- 172.21.15.169:0/3722378847 >> 172.21.15.169:0/3722378847 conn(0x7fd840091320 msgr2=0x7fd840093740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.372+0000 7fd84f525640 1 -- 172.21.15.169:0/3722378847 shutdown_connections 2024-03-28T01:04:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.375+0000 7fd84f525640 1 -- 172.21.15.169:0/3722378847 wait complete. 2024-03-28T01:04:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.377+0000 7fd84f525640 1 Processor -- start 2024-03-28T01:04:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.378+0000 7fd84f525640 1 -- start start 2024-03-28T01:04:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.378+0000 7fd84f525640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd840007ec0 0x7fd840132730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.378+0000 7fd84f525640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8400094a0 0x7fd840132c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.378+0000 7fd84f525640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8400a2340 0x7fd840139ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.379+0000 7fd84f525640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8400a8060 con 0x7fd8400094a0 2024-03-28T01:04:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.379+0000 7fd84f525640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8400a7ee0 con 0x7fd8400a2340 2024-03-28T01:04:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.379+0000 7fd84f525640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8400a81e0 con 0x7fd840007ec0 2024-03-28T01:04:50.384 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.382+0000 7fd84ed24640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8400a2340 0x7fd840139ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:50.384 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.382+0000 7fd84ed24640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8400a2340 0x7fd840139ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:46612/0 (socket says 172.21.15.169:46612) 2024-03-28T01:04:50.384 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.382+0000 7fd84ed24640 1 -- 172.21.15.169:0/2141019742 learned_addr learned my addr 172.21.15.169:0/2141019742 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:50.385 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.382+0000 7fd84ed24640 1 -- 172.21.15.169:0/2141019742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd840007ec0 msgr2=0x7fd840132730 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:04:50.385 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.382+0000 7fd84ed24640 1 --2- 172.21.15.169:0/2141019742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd840007ec0 0x7fd840132730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.385 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.382+0000 7fd84ed24640 1 -- 172.21.15.169:0/2141019742 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8400094a0 msgr2=0x7fd840132c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:50.385 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.382+0000 7fd84ed24640 1 --2- 172.21.15.169:0/2141019742 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8400094a0 0x7fd840132c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.385 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.382+0000 7fd84ed24640 1 -- 172.21.15.169:0/2141019742 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84013a350 con 0x7fd8400a2340 2024-03-28T01:04:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.382+0000 7fd84ed24640 1 --2- 172.21.15.169:0/2141019742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8400a2340 0x7fd840139ce0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd83802c1c0 tx=0x7fd838004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.387+0000 7fd83f7fe640 1 -- 172.21.15.169:0/2141019742 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd838041020 con 0x7fd8400a2340 2024-03-28T01:04:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.387+0000 7fd83f7fe640 1 -- 172.21.15.169:0/2141019742 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd83802ca30 con 0x7fd8400a2340 2024-03-28T01:04:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.389+0000 7fd83f7fe640 1 -- 172.21.15.169:0/2141019742 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd838039040 con 0x7fd8400a2340 2024-03-28T01:04:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.389+0000 7fd84f525640 1 -- 172.21.15.169:0/2141019742 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd84013a5e0 con 0x7fd8400a2340 2024-03-28T01:04:50.395 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.389+0000 7fd84f525640 1 -- 172.21.15.169:0/2141019742 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd84013aad0 con 0x7fd8400a2340 2024-03-28T01:04:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.392+0000 7fd853375740 1 Processor -- start 2024-03-28T01:04:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.393+0000 7fd853375740 1 -- start start 2024-03-28T01:04:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.393+0000 7fd853375740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa2099cb60 0x55aa2099cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.393+0000 7fd853375740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa2099d550 0x55aa209a2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.393+0000 7fd853375740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209a26c0 0x55aa209a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.393+0000 7fd853375740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa208f0b10 con 0x55aa2099d550 2024-03-28T01:04:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.393+0000 7fd853375740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa20899900 con 0x55aa2099cb60 2024-03-28T01:04:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.393+0000 7fd853375740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa208ecbb0 con 0x55aa209a26c0 2024-03-28T01:04:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.394+0000 7fd84dd22640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209a26c0 0x55aa209a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.394+0000 7fd84dd22640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209a26c0 0x55aa209a4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:44624/0 (socket says 172.21.15.169:44624) 2024-03-28T01:04:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.394+0000 7fd84dd22640 1 -- 172.21.15.169:0/2359017150 learned_addr learned my addr 172.21.15.169:0/2359017150 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.394+0000 7fd84dd22640 1 --2- 172.21.15.169:0/2359017150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa2099cb60 0x55aa2099cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.394+0000 7fd84e523640 1 --2- 172.21.15.169:0/2359017150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa2099d550 0x55aa209a2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:50.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.395+0000 7fd84e523640 1 -- 172.21.15.169:0/2359017150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209a26c0 msgr2=0x55aa209a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:50.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.395+0000 7fd84e523640 1 --2- 172.21.15.169:0/2359017150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209a26c0 0x55aa209a4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.395+0000 7fd84e523640 1 -- 172.21.15.169:0/2359017150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa2099cb60 msgr2=0x55aa2099cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:50.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.398+0000 7fd84e523640 1 --2- 172.21.15.169:0/2359017150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa2099cb60 0x55aa2099cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.398+0000 7fd84e523640 1 -- 172.21.15.169:0/2359017150 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aa20605210 con 0x55aa2099d550 2024-03-28T01:04:50.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.398+0000 7fd84dd22640 1 --2- 172.21.15.169:0/2359017150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa2099cb60 0x55aa2099cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:04:50.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.398+0000 7fd83f7fe640 1 -- 172.21.15.169:0/2141019742 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd838044020 con 0x7fd8400a2340 2024-03-28T01:04:50.402 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.399+0000 7fd83f7fe640 1 --2- 172.21.15.169:0/2141019742 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd8340a27f0 0x7fd8340a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:50.402 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.399+0000 7fd83f7fe640 1 -- 172.21.15.169:0/2141019742 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fd8380efc00 con 0x7fd8400a2340 2024-03-28T01:04:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.401+0000 7fd84e523640 1 --2- 172.21.15.169:0/2359017150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa2099d550 0x55aa209a2050 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd848066de0 tx=0x7fd84804f9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=883d6b4acfaa7a53 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.402+0000 7fd84dd22640 1 --2- 172.21.15.169:0/2141019742 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd8340a27f0 0x7fd8340a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.403+0000 7fd84f525640 1 -- 172.21.15.169:0/2359017150 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd848082070 con 0x55aa2099d550 2024-03-28T01:04:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.403+0000 7fd84f525640 1 -- 172.21.15.169:0/2359017150 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd84807e430 con 0x55aa2099d550 2024-03-28T01:04:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.404+0000 7fd84f525640 1 -- 172.21.15.169:0/2359017150 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84807d620 con 0x55aa2099d550 2024-03-28T01:04:50.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.405+0000 7fd853375740 1 -- 172.21.15.169:0/2359017150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa2099d550 msgr2=0x55aa209a2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:50.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.405+0000 7fd853375740 1 --2- 172.21.15.169:0/2359017150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa2099d550 0x55aa209a2050 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd848066de0 tx=0x7fd84804f9f0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.405+0000 7fd853375740 1 -- 172.21.15.169:0/2359017150 shutdown_connections 2024-03-28T01:04:50.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.405+0000 7fd853375740 1 --2- 172.21.15.169:0/2359017150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209a26c0 0x55aa209a4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.405+0000 7fd853375740 1 --2- 172.21.15.169:0/2359017150 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa2099d550 0x55aa209a2050 secure :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd848066de0 tx=0x7fd84804f9f0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.405+0000 7fd853375740 1 --2- 172.21.15.169:0/2359017150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa2099cb60 0x55aa2099cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.408 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.405+0000 7fd853375740 1 -- 172.21.15.169:0/2359017150 >> 172.21.15.169:0/2359017150 conn(0x55aa209873b0 msgr2=0x55aa209877b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:50.409 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.405+0000 7fd853375740 1 -- 172.21.15.169:0/2359017150 shutdown_connections 2024-03-28T01:04:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.408+0000 7fd84dd22640 1 --2- 172.21.15.169:0/2141019742 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd8340a27f0 0x7fd8340a4cb0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd844009fd0 tx=0x7fd844001120 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.408+0000 7fd853375740 1 -- 172.21.15.169:0/2359017150 wait complete. 2024-03-28T01:04:50.413 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.408+0000 7fd853375740 1 -- start start 2024-03-28T01:04:50.413 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.408+0000 7fd853375740 1 Processor -- start 2024-03-28T01:04:50.414 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.411+0000 7fd853375740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa2099cb60 0x55aa2098d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:50.414 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.411+0000 7fd853375740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209a26c0 0x55aa2098d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:50.414 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.411+0000 7fd853375740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa209928f0 0x55aa20994d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:50.414 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.411+0000 7fd853375740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa2098ba50 con 0x55aa209928f0 2024-03-28T01:04:50.414 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.411+0000 7fd853375740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa208acd50 con 0x55aa2099cb60 2024-03-28T01:04:50.414 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.411+0000 7fd853375740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa2098bbd0 con 0x55aa209a26c0 2024-03-28T01:04:50.414 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.411+0000 7fd853375740 10 client.0 _ll_register_callbacks cb 0x55aa208ac720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:04:50.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.412+0000 7fd84e523640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa2099cb60 0x55aa2098d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:50.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.412+0000 7fd84ed24640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa209928f0 0x55aa20994d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:50.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.412+0000 7fd84ed24640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa209928f0 0x55aa20994d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:52710/0 (socket says 172.21.15.169:52710) 2024-03-28T01:04:50.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.412+0000 7fd84ed24640 1 -- 172.21.15.169:0/1335939201 learned_addr learned my addr 172.21.15.169:0/1335939201 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:50.415 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.412+0000 7fd84dd22640 1 --2- 172.21.15.169:0/1335939201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209a26c0 0x55aa2098d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:50.420 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.415+0000 7fd84e523640 1 -- 172.21.15.169:0/1335939201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209a26c0 msgr2=0x55aa2098d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:50.420 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.415+0000 7fd84e523640 1 --2- 172.21.15.169:0/1335939201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209a26c0 0x55aa2098d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.420 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.415+0000 7fd84e523640 1 -- 172.21.15.169:0/1335939201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa209928f0 msgr2=0x55aa20994d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:50.420 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.415+0000 7fd84e523640 1 --2- 172.21.15.169:0/1335939201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa209928f0 0x55aa20994d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.420 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.415+0000 7fd84e523640 1 -- 172.21.15.169:0/1335939201 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aa206d1620 con 0x55aa2099cb60 2024-03-28T01:04:50.420 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.417+0000 7fd84ed24640 1 --2- 172.21.15.169:0/1335939201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa209928f0 0x55aa20994d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:04:50.427 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.418+0000 7fd84e523640 1 --2- 172.21.15.169:0/1335939201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa2099cb60 0x55aa2098d3d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x55aa2099dfc0 tx=0x7fd84807e810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a704fc22d47941fb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:50.427 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.420+0000 7fd831ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:04:50.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.429+0000 7fd831ffb640 1 -- 172.21.15.169:0/1335939201 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd848082040 con 0x55aa2099cb60 2024-03-28T01:04:50.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.429+0000 7fd853375740 1 -- 172.21.15.169:0/1335939201 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd84807e430 con 0x55aa2099cb60 2024-03-28T01:04:50.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.429+0000 7fd831ffb640 1 -- 172.21.15.169:0/1335939201 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fd848090420 con 0x55aa2099cb60 2024-03-28T01:04:50.439 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.433+0000 7fd831ffb640 1 -- 172.21.15.169:0/1335939201 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd848090660 con 0x55aa2099cb60 2024-03-28T01:04:50.439 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.434+0000 7fd853375740 10 client.26933 fetch_fsmap learned FSMap version 25 2024-03-28T01:04:50.439 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.434+0000 7fd853375740 1 -- 172.21.15.169:0/1335939201 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55aa20997fb0 con 0x55aa2099cb60 2024-03-28T01:04:50.440 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.436+0000 7fd831ffb640 1 -- 172.21.15.169:0/1335939201 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd84807e430 con 0x55aa2099cb60 2024-03-28T01:04:50.440 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.436+0000 7fd831ffb640 1 -- 172.21.15.169:0/1335939201 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd848092760 con 0x55aa2099cb60 2024-03-28T01:04:50.440 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.436+0000 7fd853375740 10 client.26933 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:04:50.440 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.436+0000 7fd853375740 10 client.26933 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:04:50.440 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.436+0000 7fd853375740 20 client.26933 populate_metadata read hostname 'smithi169' 2024-03-28T01:04:50.440 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.436+0000 7fd853375740 1 --2- 172.21.15.169:0/1335939201 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aa209999e0 0x55aa2099be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:50.440 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.436+0000 7fd853375740 4 client.26933 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:04:50.440 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.436+0000 7fd853375740 1 -- 172.21.15.169:0/1335939201 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aa208fe050 con 0x55aa209999e0 2024-03-28T01:04:50.444 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.437+0000 7fd84dd22640 1 --2- 172.21.15.169:0/1335939201 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aa209999e0 0x55aa2099be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:50.444 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.438+0000 7fd84dd22640 1 --2- 172.21.15.169:0/1335939201 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aa209999e0 0x55aa2099be40 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b28d788e01b1ec4 server_cookie=c75c8da27d3e7575 in_seq=0 out_seq=0 2024-03-28T01:04:50.444 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.438+0000 7fd831ffb640 10 client.26933 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:50.445 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.439+0000 7fd831ffb640 1 -- 172.21.15.169:0/1335939201 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55aa208fe050 con 0x55aa209999e0 2024-03-28T01:04:50.445 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.439+0000 7fd831ffb640 10 client.26933 handle_command_reply: tid=0 2024-03-28T01:04:50.445 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.440+0000 7fd853375740 1 client.26933 shutdown 2024-03-28T01:04:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.447+0000 7fd853375740 1 -- 172.21.15.169:0/1335939201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa2099cb60 msgr2=0x55aa2098d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.447+0000 7fd853375740 1 --2- 172.21.15.169:0/1335939201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa2099cb60 0x55aa2098d3d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x55aa2099dfc0 tx=0x7fd84807e810 comp rx=0 tx=0).stop 2024-03-28T01:04:50.452 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.447+0000 7fd853375740 1 -- 172.21.15.169:0/1335939201 shutdown_connections 2024-03-28T01:04:50.452 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.447+0000 7fd853375740 1 --2- 172.21.15.169:0/1335939201 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aa209999e0 0x55aa2099be40 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.452 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.448+0000 7fd853375740 1 --2- 172.21.15.169:0/1335939201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa209928f0 0x55aa20994d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.452 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.448+0000 7fd853375740 1 --2- 172.21.15.169:0/1335939201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209a26c0 0x55aa2098d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.452 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.448+0000 7fd853375740 1 --2- 172.21.15.169:0/1335939201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa2099cb60 0x55aa2098d3d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.452 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.448+0000 7fd853375740 1 -- 172.21.15.169:0/1335939201 >> 172.21.15.169:0/1335939201 conn(0x55aa209873b0 msgr2=0x55aa20986690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:50.452 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.449+0000 7fd831ffb640 0 client.26933 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:50.452 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.449+0000 7fd853375740 1 -- 172.21.15.169:0/1335939201 shutdown_connections 2024-03-28T01:04:50.454 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.452+0000 7fd853375740 1 -- 172.21.15.169:0/1335939201 wait complete. 2024-03-28T01:04:50.456 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.452+0000 7fd853375740 20 client.26933 trim_cache size 0 max 16384 2024-03-28T01:04:50.485 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.482+0000 7fd853375740 1 Processor -- start 2024-03-28T01:04:50.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.482+0000 7fd853375740 1 -- start start 2024-03-28T01:04:50.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.482+0000 7fd853375740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209928f0 0x55aa20987790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:50.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.482+0000 7fd853375740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa2099cb60 0x55aa2098a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:50.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.482+0000 7fd853375740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa209a26c0 0x55aa2098ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:50.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.482+0000 7fd84dd22640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa2099cb60 0x55aa2098a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:50.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.483+0000 7fd84dd22640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa2099cb60 0x55aa2098a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:52724/0 (socket says 172.21.15.169:52724) 2024-03-28T01:04:50.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.483+0000 7fd84dd22640 1 -- 172.21.15.169:0/151398515 learned_addr learned my addr 172.21.15.169:0/151398515 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:50.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.483+0000 7fd853375740 1 -- 172.21.15.169:0/151398515 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84013ec00 con 0x55aa2099cb60 2024-03-28T01:04:50.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.483+0000 7fd853375740 1 -- 172.21.15.169:0/151398515 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84013c700 con 0x55aa209a26c0 2024-03-28T01:04:50.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.483+0000 7fd853375740 1 -- 172.21.15.169:0/151398515 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8400a1ce0 con 0x55aa209928f0 2024-03-28T01:04:50.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.483+0000 7fd84dd22640 1 -- 172.21.15.169:0/151398515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209928f0 msgr2=0x55aa20987790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:04:50.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.483+0000 7fd84dd22640 1 --2- 172.21.15.169:0/151398515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209928f0 0x55aa20987790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.483+0000 7fd84dd22640 1 -- 172.21.15.169:0/151398515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa209a26c0 msgr2=0x55aa2098ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:50.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.483+0000 7fd84dd22640 1 --2- 172.21.15.169:0/151398515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa209a26c0 0x55aa2098ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.483+0000 7fd84dd22640 1 -- 172.21.15.169:0/151398515 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aa209a1d00 con 0x55aa2099cb60 2024-03-28T01:04:50.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.483+0000 7fd84dd22640 1 --2- 172.21.15.169:0/151398515 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa2099cb60 0x55aa2098a610 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fd844005fb0 tx=0x7fd844034000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a630dc308acf2f2e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:50.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.484+0000 7fd833fff640 1 -- 172.21.15.169:0/151398515 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84404adf0 con 0x55aa2099cb60 2024-03-28T01:04:50.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.484+0000 7fd833fff640 1 -- 172.21.15.169:0/151398515 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd844004500 con 0x55aa2099cb60 2024-03-28T01:04:50.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.484+0000 7fd833fff640 1 -- 172.21.15.169:0/151398515 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd844003a00 con 0x55aa2099cb60 2024-03-28T01:04:50.488 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.485+0000 7fd853375740 1 -- 172.21.15.169:0/151398515 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa2099cb60 msgr2=0x55aa2098a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:50.488 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.485+0000 7fd853375740 1 --2- 172.21.15.169:0/151398515 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa2099cb60 0x55aa2098a610 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fd844005fb0 tx=0x7fd844034000 comp rx=0 tx=0).stop 2024-03-28T01:04:50.490 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.488+0000 7fd853375740 1 -- 172.21.15.169:0/151398515 shutdown_connections 2024-03-28T01:04:50.490 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.488+0000 7fd853375740 1 --2- 172.21.15.169:0/151398515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa209a26c0 0x55aa2098ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.488+0000 7fd853375740 1 --2- 172.21.15.169:0/151398515 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa2099cb60 0x55aa2098a610 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.488+0000 7fd853375740 1 --2- 172.21.15.169:0/151398515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209928f0 0x55aa20987790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.488+0000 7fd853375740 1 -- 172.21.15.169:0/151398515 >> 172.21.15.169:0/151398515 conn(0x55aa209873b0 msgr2=0x55aa2098bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:50.492 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.489+0000 7fd853375740 1 -- 172.21.15.169:0/151398515 shutdown_connections 2024-03-28T01:04:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.492+0000 7fd853375740 1 -- 172.21.15.169:0/151398515 wait complete. 2024-03-28T01:04:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.493+0000 7fd853375740 1 -- start start 2024-03-28T01:04:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.493+0000 7fd853375740 1 Processor -- start 2024-03-28T01:04:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.493+0000 7fd853375740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa209928f0 0x55aa209b2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.493+0000 7fd853375740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa2099cb60 0x55aa209b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.493+0000 7fd853375740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209a26c0 0x55aa209b3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.493+0000 7fd853375740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84013adf0 con 0x55aa2099cb60 2024-03-28T01:04:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.493+0000 7fd853375740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84013c0a0 con 0x55aa209928f0 2024-03-28T01:04:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.493+0000 7fd853375740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84013b3e0 con 0x55aa209a26c0 2024-03-28T01:04:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.494+0000 7fd853375740 10 client.0 _ll_register_callbacks cb 0x7fd84807ce20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:04:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.494+0000 7fd84ed24640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209a26c0 0x55aa209b3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.494+0000 7fd84ed24640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209a26c0 0x55aa209b3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:44642/0 (socket says 172.21.15.169:44642) 2024-03-28T01:04:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.494+0000 7fd84ed24640 1 -- 172.21.15.169:0/2470403663 learned_addr learned my addr 172.21.15.169:0/2470403663 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:04:50.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.494+0000 7fd84dd22640 1 --2- 172.21.15.169:0/2470403663 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa2099cb60 0x55aa209b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:50.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.494+0000 7fd84ed24640 1 -- 172.21.15.169:0/2470403663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa209928f0 msgr2=0x55aa209b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:50.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.494+0000 7fd84ed24640 1 --2- 172.21.15.169:0/2470403663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa209928f0 0x55aa209b2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.494+0000 7fd84ed24640 1 -- 172.21.15.169:0/2470403663 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa2099cb60 msgr2=0x55aa209b2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:04:50.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.494+0000 7fd84ed24640 1 --2- 172.21.15.169:0/2470403663 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa2099cb60 0x55aa209b2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:50.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.494+0000 7fd84ed24640 1 -- 172.21.15.169:0/2470403663 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aa209b3f50 con 0x55aa209a26c0 2024-03-28T01:04:50.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.496+0000 7fd84ed24640 1 --2- 172.21.15.169:0/2470403663 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209a26c0 0x55aa209b3960 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd838032b50 tx=0x7fd8380bc140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d95f246a8526b77 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:04:50.500 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.497+0000 7fd831ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:04:50.503 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.500+0000 7fd831ffb640 1 -- 172.21.15.169:0/2470403663 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8380d4020 con 0x55aa209a26c0 2024-03-28T01:04:50.503 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.500+0000 7fd853375740 1 -- 172.21.15.169:0/2470403663 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd844004500 con 0x55aa209a26c0 2024-03-28T01:04:50.503 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.500+0000 7fd831ffb640 1 -- 172.21.15.169:0/2470403663 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fd8380ed410 con 0x55aa209a26c0 2024-03-28T01:04:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.504+0000 7fd831ffb640 1 -- 172.21.15.169:0/2470403663 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8380bc2b0 con 0x55aa209a26c0 2024-03-28T01:04:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.505+0000 7fd853375740 10 client.26863 fetch_fsmap learned FSMap version 25 2024-03-28T01:04:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.505+0000 7fd853375740 1 -- 172.21.15.169:0/2470403663 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55aa209fd5a0 con 0x55aa209a26c0 2024-03-28T01:04:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.512+0000 7fd831ffb640 1 -- 172.21.15.169:0/2470403663 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd844004500 con 0x55aa209a26c0 2024-03-28T01:04:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.512+0000 7fd831ffb640 1 -- 172.21.15.169:0/2470403663 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd83803c030 con 0x55aa209a26c0 2024-03-28T01:04:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.512+0000 7fd853375740 10 client.26863 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:04:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.512+0000 7fd853375740 10 client.26863 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:04:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.512+0000 7fd853375740 20 client.26863 populate_metadata read hostname 'smithi169' 2024-03-28T01:04:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.512+0000 7fd853375740 1 --2- 172.21.15.169:0/2470403663 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aa209fe020 0x55aa20a00440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:04:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.512+0000 7fd853375740 4 client.26863 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:04:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.512+0000 7fd853375740 1 -- 172.21.15.169:0/2470403663 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55aa208f83a0 con 0x55aa209fe020 2024-03-28T01:04:50.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.518+0000 7fd84e523640 1 --2- 172.21.15.169:0/2470403663 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aa209fe020 0x55aa20a00440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:04:50.529 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.525+0000 7fd84e523640 1 --2- 172.21.15.169:0/2470403663 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aa209fe020 0x55aa20a00440 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a5c8265f57a4d5d server_cookie=96009abe2b5e9bfa in_seq=0 out_seq=0 2024-03-28T01:04:50.529 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:50.525+0000 7fd831ffb640 10 client.26863 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:51.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:51 smithi174 ceph-mon[36060]: pgmap v601: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 133 MiB/s rd, 395 MiB/s wr, 5.19k op/s 2024-03-28T01:04:51.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:51 smithi177 ceph-mon[36206]: pgmap v601: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 133 MiB/s rd, 395 MiB/s wr, 5.19k op/s 2024-03-28T01:04:52.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:51 smithi169 ceph-mon[32274]: pgmap v601: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 133 MiB/s rd, 395 MiB/s wr, 5.19k op/s 2024-03-28T01:04:53.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:53 smithi174 ceph-mon[36060]: pgmap v602: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s rd, 316 MiB/s wr, 4.19k op/s 2024-03-28T01:04:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.879+0000 7fd831ffb640 1 -- 172.21.15.169:0/2470403663 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55aa208f83a0 con 0x55aa209fe020 2024-03-28T01:04:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.879+0000 7fd831ffb640 10 client.26863 handle_command_reply: tid=0 2024-03-28T01:04:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.879+0000 7fd853375740 1 client.26863 shutdown 2024-03-28T01:04:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.879+0000 7fd853375740 1 -- 172.21.15.169:0/2470403663 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209a26c0 msgr2=0x55aa209b3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.879+0000 7fd853375740 1 --2- 172.21.15.169:0/2470403663 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209a26c0 0x55aa209b3960 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd838032b50 tx=0x7fd8380bc140 comp rx=0 tx=0).stop 2024-03-28T01:04:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.879+0000 7fd853375740 1 -- 172.21.15.169:0/2470403663 shutdown_connections 2024-03-28T01:04:53.883 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.879+0000 7fd853375740 1 --2- 172.21.15.169:0/2470403663 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa209a26c0 0x55aa209b3960 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.879+0000 7fd853375740 1 --2- 172.21.15.169:0/2470403663 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aa209fe020 0x55aa20a00440 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.880+0000 7fd853375740 1 --2- 172.21.15.169:0/2470403663 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa2099cb60 0x55aa209b2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.880+0000 7fd853375740 1 --2- 172.21.15.169:0/2470403663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa209928f0 0x55aa209b2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.880+0000 7fd853375740 1 -- 172.21.15.169:0/2470403663 >> 172.21.15.169:0/2470403663 conn(0x55aa209873b0 msgr2=0x55aa20987790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.880+0000 7fd831ffb640 0 client.26863 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:04:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.880+0000 7fd853375740 1 -- 172.21.15.169:0/2470403663 shutdown_connections 2024-03-28T01:04:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.880+0000 7fd853375740 1 -- 172.21.15.169:0/2470403663 wait complete. 2024-03-28T01:04:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.880+0000 7fd853375740 20 client.26863 trim_cache size 0 max 16384 2024-03-28T01:04:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:04:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:04:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:04:53.884 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:04:53.885 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.881+0000 7fd83d7fa640 1 -- 172.21.15.169:0/2141019742 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd8340a27f0 msgr2=0x7fd8340a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:53.886 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.881+0000 7fd83d7fa640 1 --2- 172.21.15.169:0/2141019742 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd8340a27f0 0x7fd8340a4cb0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd844009fd0 tx=0x7fd844001120 comp rx=0 tx=0).stop 2024-03-28T01:04:53.886 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.881+0000 7fd83d7fa640 1 -- 172.21.15.169:0/2141019742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8400a2340 msgr2=0x7fd840139ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:04:53.886 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.881+0000 7fd83d7fa640 1 --2- 172.21.15.169:0/2141019742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8400a2340 0x7fd840139ce0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd83802c1c0 tx=0x7fd838004270 comp rx=0 tx=0).stop 2024-03-28T01:04:53.886 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.881+0000 7fd83d7fa640 1 -- 172.21.15.169:0/2141019742 shutdown_connections 2024-03-28T01:04:53.886 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.881+0000 7fd83d7fa640 1 --2- 172.21.15.169:0/2141019742 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd8340a27f0 0x7fd8340a4cb0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:53.886 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.881+0000 7fd83d7fa640 1 --2- 172.21.15.169:0/2141019742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8400a2340 0x7fd840139ce0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:53.886 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.881+0000 7fd83d7fa640 1 --2- 172.21.15.169:0/2141019742 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd8400094a0 0x7fd840132c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:53.886 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.881+0000 7fd83d7fa640 1 --2- 172.21.15.169:0/2141019742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd840007ec0 0x7fd840132730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:04:53.886 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.881+0000 7fd83d7fa640 1 -- 172.21.15.169:0/2141019742 >> 172.21.15.169:0/2141019742 conn(0x7fd840091320 msgr2=0x7fd8400956e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:04:53.886 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.881+0000 7fd83d7fa640 1 -- 172.21.15.169:0/2141019742 shutdown_connections 2024-03-28T01:04:53.886 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:04:53.881+0000 7fd83d7fa640 1 -- 172.21.15.169:0/2141019742 wait complete. 2024-03-28T01:04:53.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:53 smithi177 ceph-mon[36206]: pgmap v602: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s rd, 316 MiB/s wr, 4.19k op/s 2024-03-28T01:04:54.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:53 smithi169 ceph-mon[32274]: pgmap v602: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s rd, 316 MiB/s wr, 4.19k op/s 2024-03-28T01:04:55.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:55 smithi174 ceph-mon[36060]: pgmap v603: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s rd, 317 MiB/s wr, 4.10k op/s 2024-03-28T01:04:55.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:55 smithi177 ceph-mon[36206]: pgmap v603: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s rd, 317 MiB/s wr, 4.10k op/s 2024-03-28T01:04:56.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:55 smithi169 ceph-mon[32274]: pgmap v603: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s rd, 317 MiB/s wr, 4.10k op/s 2024-03-28T01:04:57.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:57 smithi177 ceph-mon[36206]: pgmap v604: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s rd, 356 MiB/s wr, 4.47k op/s 2024-03-28T01:04:58.053 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:57 smithi169 ceph-mon[32274]: pgmap v604: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s rd, 356 MiB/s wr, 4.47k op/s 2024-03-28T01:04:58.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:57 smithi174 ceph-mon[36060]: pgmap v604: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s rd, 356 MiB/s wr, 4.47k op/s 2024-03-28T01:04:58.601 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:04:58.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:04:59.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:04:59 smithi174 ceph-mon[36060]: pgmap v605: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s rd, 365 MiB/s wr, 4.46k op/s 2024-03-28T01:04:59.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:04:59 smithi177 ceph-mon[36206]: pgmap v605: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s rd, 365 MiB/s wr, 4.46k op/s 2024-03-28T01:05:00.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:04:59 smithi169 ceph-mon[32274]: pgmap v605: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s rd, 365 MiB/s wr, 4.46k op/s 2024-03-28T01:05:00.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.886+0000 7fd716ffd640 1 -- 172.21.15.169:0/2286681546 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7100353f0 con 0x7fd71810dd70 2024-03-28T01:05:00.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.887+0000 7fd71e28a640 1 -- 172.21.15.169:0/2286681546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd71810dd70 msgr2=0x7fd718110230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:00.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.887+0000 7fd71e28a640 1 --2- 172.21.15.169:0/2286681546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd71810dd70 0x7fd718110230 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fd710005fd0 tx=0x7fd71002cc70 comp rx=0 tx=0).stop 2024-03-28T01:05:00.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.888+0000 7fd71e28a640 1 -- 172.21.15.169:0/2286681546 shutdown_connections 2024-03-28T01:05:00.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.888+0000 7fd71e28a640 1 --2- 172.21.15.169:0/2286681546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd71810dd70 0x7fd718110230 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:00.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.888+0000 7fd71e28a640 1 --2- 172.21.15.169:0/2286681546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd71810b2d0 0x7fd71810d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:00.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.888+0000 7fd71e28a640 1 --2- 172.21.15.169:0/2286681546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd718108910 0x7fd71810ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:00.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.888+0000 7fd71e28a640 1 -- 172.21.15.169:0/2286681546 >> 172.21.15.169:0/2286681546 conn(0x7fd71806e500 msgr2=0x7fd718070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:00.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.891+0000 7fd71e28a640 1 -- 172.21.15.169:0/2286681546 shutdown_connections 2024-03-28T01:05:00.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.891+0000 7fd71e28a640 1 -- 172.21.15.169:0/2286681546 wait complete. 2024-03-28T01:05:00.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.892+0000 7fd71e28a640 1 Processor -- start 2024-03-28T01:05:00.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.892+0000 7fd71e28a640 1 -- start start 2024-03-28T01:05:00.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.892+0000 7fd71e28a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd718108910 0x7fd718119400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:00.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.892+0000 7fd71e28a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd71810b2d0 0x7fd718119940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:00.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.892+0000 7fd71e28a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd71810dd70 0x7fd7181b6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:00.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.892+0000 7fd71e28a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7181135f0 con 0x7fd71810b2d0 2024-03-28T01:05:00.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.892+0000 7fd71e28a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd718113470 con 0x7fd71810dd70 2024-03-28T01:05:00.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.892+0000 7fd71e28a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd718113770 con 0x7fd718108910 2024-03-28T01:05:00.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.892+0000 7fd71c827640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd71810dd70 0x7fd7181b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:00.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.893+0000 7fd71c827640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd71810dd70 0x7fd7181b6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:56614/0 (socket says 172.21.15.169:56614) 2024-03-28T01:05:00.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.893+0000 7fd71c827640 1 -- 172.21.15.169:0/2171338218 learned_addr learned my addr 172.21.15.169:0/2171338218 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:00.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.893+0000 7fd717fff640 1 --2- 172.21.15.169:0/2171338218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd718108910 0x7fd718119400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:00.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.893+0000 7fd7177fe640 1 --2- 172.21.15.169:0/2171338218 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd71810b2d0 0x7fd718119940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:00.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.893+0000 7fd717fff640 1 -- 172.21.15.169:0/2171338218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd71810dd70 msgr2=0x7fd7181b6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:00.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.893+0000 7fd717fff640 1 --2- 172.21.15.169:0/2171338218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd71810dd70 0x7fd7181b6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:00.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.893+0000 7fd717fff640 1 -- 172.21.15.169:0/2171338218 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd71810b2d0 msgr2=0x7fd718119940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:00.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.893+0000 7fd717fff640 1 --2- 172.21.15.169:0/2171338218 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd71810b2d0 0x7fd718119940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:00.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.893+0000 7fd717fff640 1 -- 172.21.15.169:0/2171338218 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7181b6e40 con 0x7fd718108910 2024-03-28T01:05:00.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.893+0000 7fd71c827640 1 --2- 172.21.15.169:0/2171338218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd71810dd70 0x7fd7181b6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:05:00.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.895+0000 7fd717fff640 1 --2- 172.21.15.169:0/2171338218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd718108910 0x7fd718119400 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd70c009fd0 tx=0x7fd70c00b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:00.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.899+0000 7fd7157fa640 1 -- 172.21.15.169:0/2171338218 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c015020 con 0x7fd718108910 2024-03-28T01:05:00.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.899+0000 7fd71e28a640 1 -- 172.21.15.169:0/2171338218 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7181b7070 con 0x7fd718108910 2024-03-28T01:05:00.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.899+0000 7fd71e28a640 1 -- 172.21.15.169:0/2171338218 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7181b7610 con 0x7fd718108910 2024-03-28T01:05:00.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.899+0000 7fd7157fa640 1 -- 172.21.15.169:0/2171338218 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd70c011d10 con 0x7fd718108910 2024-03-28T01:05:00.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.899+0000 7fd7157fa640 1 -- 172.21.15.169:0/2171338218 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c00f2e0 con 0x7fd718108910 2024-03-28T01:05:00.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.903+0000 7fd720e70740 1 Processor -- start 2024-03-28T01:05:00.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.904+0000 7fd7157fa640 1 -- 172.21.15.169:0/2171338218 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd70c018020 con 0x7fd718108910 2024-03-28T01:05:00.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.904+0000 7fd720e70740 1 -- start start 2024-03-28T01:05:00.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.905+0000 7fd720e70740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc95da0 0x55923cc961c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:00.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.905+0000 7fd720e70740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55923cc96790 0x55923cc9b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:00.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.905+0000 7fd720e70740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc9b900 0x55923cc9dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:00.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.905+0000 7fd720e70740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55923cbfb3d0 con 0x55923cc96790 2024-03-28T01:05:00.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.905+0000 7fd720e70740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55923cba3ad0 con 0x55923cc9b900 2024-03-28T01:05:00.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.905+0000 7fd720e70740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55923cbf7470 con 0x55923cc95da0 2024-03-28T01:05:00.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.905+0000 7fd7177fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc9b900 0x55923cc9dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:00.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.905+0000 7fd71c827640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc95da0 0x55923cc961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:00.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.905+0000 7fd71c827640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc95da0 0x55923cc961c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48772/0 (socket says 172.21.15.169:48772) 2024-03-28T01:05:00.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.905+0000 7fd71c827640 1 -- 172.21.15.169:0/2757042864 learned_addr learned my addr 172.21.15.169:0/2757042864 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:00.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.905+0000 7fd717fff640 1 --2- 172.21.15.169:0/2757042864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55923cc96790 0x55923cc9b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:00.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.905+0000 7fd7157fa640 1 --2- 172.21.15.169:0/2171338218 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd7000a29d0 0x7fd7000a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:00.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.905+0000 7fd71c827640 1 -- 172.21.15.169:0/2757042864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc9b900 msgr2=0x55923cc9dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:00.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.905+0000 7fd71c827640 1 --2- 172.21.15.169:0/2757042864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc9b900 0x55923cc9dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:00.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.905+0000 7fd71c827640 1 -- 172.21.15.169:0/2757042864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55923cc96790 msgr2=0x55923cc9b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:00.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.905+0000 7fd71c827640 1 --2- 172.21.15.169:0/2757042864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55923cc96790 0x55923cc9b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:00.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.905+0000 7fd71c827640 1 -- 172.21.15.169:0/2757042864 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55923c9d97d0 con 0x55923cc95da0 2024-03-28T01:05:00.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.906+0000 7fd7157fa640 1 -- 172.21.15.169:0/2171338218 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fd70c0ce4f0 con 0x7fd718108910 2024-03-28T01:05:00.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.906+0000 7fd71c827640 1 --2- 172.21.15.169:0/2757042864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc95da0 0x55923cc961c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd71000a6d0 tx=0x7fd71000a7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ea54ec21317c3d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:00.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.906+0000 7fd71c827640 1 --2- 172.21.15.169:0/2171338218 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd7000a29d0 0x7fd7000a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:00.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.906+0000 7fd6fe7fc640 1 -- 172.21.15.169:0/2757042864 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd710043020 con 0x55923cc95da0 2024-03-28T01:05:00.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.906+0000 7fd6fe7fc640 1 -- 172.21.15.169:0/2757042864 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd710003970 con 0x55923cc95da0 2024-03-28T01:05:00.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.906+0000 7fd6fe7fc640 1 -- 172.21.15.169:0/2757042864 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7100356f0 con 0x55923cc95da0 2024-03-28T01:05:00.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.907+0000 7fd71c827640 1 --2- 172.21.15.169:0/2171338218 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd7000a29d0 0x7fd7000a4e90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd710044000 tx=0x7fd7100359d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:00.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.907+0000 7fd720e70740 1 -- 172.21.15.169:0/2757042864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc95da0 msgr2=0x55923cc961c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:00.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.907+0000 7fd720e70740 1 --2- 172.21.15.169:0/2757042864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc95da0 0x55923cc961c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd71000a6d0 tx=0x7fd71000a7a0 comp rx=0 tx=0).stop 2024-03-28T01:05:00.913 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.910+0000 7fd720e70740 1 -- 172.21.15.169:0/2757042864 shutdown_connections 2024-03-28T01:05:00.913 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.910+0000 7fd720e70740 1 --2- 172.21.15.169:0/2757042864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc9b900 0x55923cc9dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:00.913 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.910+0000 7fd720e70740 1 --2- 172.21.15.169:0/2757042864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55923cc96790 0x55923cc9b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:00.913 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.910+0000 7fd720e70740 1 --2- 172.21.15.169:0/2757042864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc95da0 0x55923cc961c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:00.914 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.910+0000 7fd720e70740 1 -- 172.21.15.169:0/2757042864 >> 172.21.15.169:0/2757042864 conn(0x55923cc91310 msgr2=0x55923cc91710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:00.914 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.911+0000 7fd720e70740 1 -- 172.21.15.169:0/2757042864 shutdown_connections 2024-03-28T01:05:00.914 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.911+0000 7fd720e70740 1 -- 172.21.15.169:0/2757042864 wait complete. 2024-03-28T01:05:00.914 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.911+0000 7fd720e70740 1 -- start start 2024-03-28T01:05:00.914 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.912+0000 7fd720e70740 1 Processor -- start 2024-03-28T01:05:00.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.912+0000 7fd720e70740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc95da0 0x55923cc9ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:00.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.912+0000 7fd720e70740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55923cc96790 0x55923cca0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:00.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.913+0000 7fd720e70740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc9b900 0x55923cca7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:00.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.913+0000 7fd720e70740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55923cc9e850 con 0x55923cc96790 2024-03-28T01:05:00.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.913+0000 7fd720e70740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55923cbb7060 con 0x55923cc9b900 2024-03-28T01:05:00.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.913+0000 7fd720e70740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55923cc9e9d0 con 0x55923cc95da0 2024-03-28T01:05:00.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.913+0000 7fd720e70740 10 client.0 _ll_register_callbacks cb 0x55923cb996a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:00.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.913+0000 7fd717fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc95da0 0x55923cc9ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:00.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.913+0000 7fd717fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc95da0 0x55923cc9ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48778/0 (socket says 172.21.15.169:48778) 2024-03-28T01:05:00.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.913+0000 7fd717fff640 1 -- 172.21.15.169:0/2677007228 learned_addr learned my addr 172.21.15.169:0/2677007228 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:00.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.913+0000 7fd71c827640 1 --2- 172.21.15.169:0/2677007228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc9b900 0x55923cca7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:00.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.914+0000 7fd7177fe640 1 --2- 172.21.15.169:0/2677007228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55923cc96790 0x55923cca0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:00.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.914+0000 7fd71c827640 1 -- 172.21.15.169:0/2677007228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc95da0 msgr2=0x55923cc9ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:00.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.914+0000 7fd71c827640 1 --2- 172.21.15.169:0/2677007228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc95da0 0x55923cc9ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:00.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.914+0000 7fd71c827640 1 -- 172.21.15.169:0/2677007228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55923cc96790 msgr2=0x55923cca0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:00.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.914+0000 7fd71c827640 1 --2- 172.21.15.169:0/2677007228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55923cc96790 0x55923cca0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:00.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.914+0000 7fd71c827640 1 -- 172.21.15.169:0/2677007228 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55923c90f3f0 con 0x55923cc9b900 2024-03-28T01:05:00.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.914+0000 7fd71c827640 1 --2- 172.21.15.169:0/2677007228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc9b900 0x55923cca7560 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd71002d280 tx=0x7fd710045000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a50df4985a35326e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:00.921 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.915+0000 7fd6f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:05:00.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.921+0000 7fd6f77fe640 1 -- 172.21.15.169:0/2677007228 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd710043050 con 0x55923cc9b900 2024-03-28T01:05:00.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.921+0000 7fd6f77fe640 1 -- 172.21.15.169:0/2677007228 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fd710007a50 con 0x55923cc9b900 2024-03-28T01:05:00.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.922+0000 7fd720e70740 1 -- 172.21.15.169:0/2677007228 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd710003970 con 0x55923cc9b900 2024-03-28T01:05:00.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.922+0000 7fd6f77fe640 1 -- 172.21.15.169:0/2677007228 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd710063050 con 0x55923cc9b900 2024-03-28T01:05:00.926 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.923+0000 7fd720e70740 10 client.26942 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:00.926 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.923+0000 7fd720e70740 1 -- 172.21.15.169:0/2677007228 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55923c8cf0f0 con 0x55923cc9b900 2024-03-28T01:05:00.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.926+0000 7fd6f77fe640 1 -- 172.21.15.169:0/2677007228 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd71000a9e0 con 0x55923cc9b900 2024-03-28T01:05:00.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.926+0000 7fd6f77fe640 1 -- 172.21.15.169:0/2677007228 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd7100045f0 con 0x55923cc9b900 2024-03-28T01:05:00.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.926+0000 7fd720e70740 10 client.26942 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:00.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.926+0000 7fd720e70740 10 client.26942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:00.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.926+0000 7fd720e70740 20 client.26942 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:00.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.926+0000 7fd720e70740 1 --2- 172.21.15.169:0/2677007228 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55923ccab710 0x55923ccadb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:00.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.926+0000 7fd720e70740 4 client.26942 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:05:00.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.926+0000 7fd720e70740 1 -- 172.21.15.169:0/2677007228 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd710003970 con 0x55923ccab710 2024-03-28T01:05:00.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.926+0000 7fd717fff640 1 --2- 172.21.15.169:0/2677007228 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55923ccab710 0x55923ccadb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:00.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.930+0000 7fd6f77fe640 10 client.26942 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:00.933 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.930+0000 7fd717fff640 1 --2- 172.21.15.169:0/2677007228 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55923ccab710 0x55923ccadb00 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9247fae91e5eea1 server_cookie=e08339e98c94f26d in_seq=0 out_seq=0 2024-03-28T01:05:00.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.931+0000 7fd6f77fe640 1 -- 172.21.15.169:0/2677007228 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7fd710003970 con 0x55923ccab710 2024-03-28T01:05:00.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.931+0000 7fd6f77fe640 10 client.26942 handle_command_reply: tid=0 2024-03-28T01:05:00.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.932+0000 7fd720e70740 1 client.26942 shutdown 2024-03-28T01:05:00.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.940+0000 7fd720e70740 1 -- 172.21.15.169:0/2677007228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc9b900 msgr2=0x55923cca7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:00.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.940+0000 7fd720e70740 1 --2- 172.21.15.169:0/2677007228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc9b900 0x55923cca7560 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd71002d280 tx=0x7fd710045000 comp rx=0 tx=0).stop 2024-03-28T01:05:00.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.941+0000 7fd720e70740 1 -- 172.21.15.169:0/2677007228 shutdown_connections 2024-03-28T01:05:00.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.941+0000 7fd720e70740 1 --2- 172.21.15.169:0/2677007228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc9b900 0x55923cca7560 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:00.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.941+0000 7fd720e70740 1 --2- 172.21.15.169:0/2677007228 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55923ccab710 0x55923ccadb00 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:00.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.941+0000 7fd720e70740 1 --2- 172.21.15.169:0/2677007228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55923cc96790 0x55923cca0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:00.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.941+0000 7fd720e70740 1 --2- 172.21.15.169:0/2677007228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc95da0 0x55923cc9ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:00.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.941+0000 7fd720e70740 1 -- 172.21.15.169:0/2677007228 >> 172.21.15.169:0/2677007228 conn(0x55923cc91310 msgr2=0x55923cc90530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:00.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.941+0000 7fd6f77fe640 0 client.26942 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:00.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.941+0000 7fd720e70740 1 -- 172.21.15.169:0/2677007228 shutdown_connections 2024-03-28T01:05:00.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.941+0000 7fd720e70740 1 -- 172.21.15.169:0/2677007228 wait complete. 2024-03-28T01:05:00.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.942+0000 7fd720e70740 20 client.26942 trim_cache size 0 max 16384 2024-03-28T01:05:00.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.980+0000 7fd720e70740 1 Processor -- start 2024-03-28T01:05:00.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.980+0000 7fd720e70740 1 -- start start 2024-03-28T01:05:00.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.980+0000 7fd720e70740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55923cc95da0 0x55923cc9f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:00.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.980+0000 7fd720e70740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc96790 0x55923cca7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:00.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.980+0000 7fd720e70740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc9b900 0x55923cc9f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:00.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.980+0000 7fd720e70740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd704012090 con 0x55923cc95da0 2024-03-28T01:05:00.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.980+0000 7fd720e70740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70400fb90 con 0x55923cc9b900 2024-03-28T01:05:00.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.980+0000 7fd720e70740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70400dc30 con 0x55923cc96790 2024-03-28T01:05:00.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.980+0000 7fd717fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55923cc95da0 0x55923cc9f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:00.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.980+0000 7fd717fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55923cc95da0 0x55923cc9f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34102/0 (socket says 172.21.15.169:34102) 2024-03-28T01:05:00.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.980+0000 7fd717fff640 1 -- 172.21.15.169:0/1708682543 learned_addr learned my addr 172.21.15.169:0/1708682543 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:00.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.981+0000 7fd71c827640 1 --2- 172.21.15.169:0/1708682543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc9b900 0x55923cc9f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:00.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.982+0000 7fd7177fe640 1 --2- 172.21.15.169:0/1708682543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc96790 0x55923cca7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:00.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.983+0000 7fd717fff640 1 -- 172.21.15.169:0/1708682543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc96790 msgr2=0x55923cca7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:00.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.983+0000 7fd717fff640 1 --2- 172.21.15.169:0/1708682543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc96790 0x55923cca7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:00.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.983+0000 7fd717fff640 1 -- 172.21.15.169:0/1708682543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc9b900 msgr2=0x55923cc9f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:00.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.983+0000 7fd717fff640 1 --2- 172.21.15.169:0/1708682543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc9b900 0x55923cc9f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:00.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.983+0000 7fd717fff640 1 -- 172.21.15.169:0/1708682543 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55923cc92080 con 0x55923cc95da0 2024-03-28T01:05:01.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.998+0000 7fd717fff640 1 --2- 172.21.15.169:0/1708682543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55923cc95da0 0x55923cc9f2b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fd70c098b10 tx=0x7fd70c098aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57ebddd99cb5ce27 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:01.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:00.999+0000 7fd6fdffb640 1 -- 172.21.15.169:0/1708682543 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c0b3070 con 0x55923cc95da0 2024-03-28T01:05:01.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.000+0000 7fd6fdffb640 1 -- 172.21.15.169:0/1708682543 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd70c09b190 con 0x55923cc95da0 2024-03-28T01:05:01.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.000+0000 7fd6fdffb640 1 -- 172.21.15.169:0/1708682543 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c098d90 con 0x55923cc95da0 2024-03-28T01:05:01.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.001+0000 7fd720e70740 1 -- 172.21.15.169:0/1708682543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55923cc95da0 msgr2=0x55923cc9f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:01.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.001+0000 7fd720e70740 1 --2- 172.21.15.169:0/1708682543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55923cc95da0 0x55923cc9f2b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fd70c098b10 tx=0x7fd70c098aa0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.002+0000 7fd720e70740 1 -- 172.21.15.169:0/1708682543 shutdown_connections 2024-03-28T01:05:01.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.002+0000 7fd720e70740 1 --2- 172.21.15.169:0/1708682543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc9b900 0x55923cc9f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.002+0000 7fd720e70740 1 --2- 172.21.15.169:0/1708682543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc96790 0x55923cca7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.002+0000 7fd720e70740 1 --2- 172.21.15.169:0/1708682543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55923cc95da0 0x55923cc9f2b0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.002+0000 7fd720e70740 1 -- 172.21.15.169:0/1708682543 >> 172.21.15.169:0/1708682543 conn(0x55923cc91310 msgr2=0x55923cc94570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:01.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.004+0000 7fd720e70740 1 -- 172.21.15.169:0/1708682543 shutdown_connections 2024-03-28T01:05:01.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.004+0000 7fd720e70740 1 -- 172.21.15.169:0/1708682543 wait complete. 2024-03-28T01:05:01.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.004+0000 7fd720e70740 1 -- start start 2024-03-28T01:05:01.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.005+0000 7fd720e70740 1 Processor -- start 2024-03-28T01:05:01.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.005+0000 7fd720e70740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc95da0 0x55923cca50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:01.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.005+0000 7fd720e70740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc96790 0x55923cca5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:01.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.005+0000 7fd720e70740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55923cc9b900 0x55923cca6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:01.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.005+0000 7fd720e70740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70400e230 con 0x55923cc9b900 2024-03-28T01:05:01.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.005+0000 7fd720e70740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70400f530 con 0x55923cc96790 2024-03-28T01:05:01.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.005+0000 7fd720e70740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70400e870 con 0x55923cc95da0 2024-03-28T01:05:01.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.005+0000 7fd720e70740 10 client.0 _ll_register_callbacks cb 0x7fd7080093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:01.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.006+0000 7fd7177fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc96790 0x55923cca5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:01.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.006+0000 7fd7177fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc96790 0x55923cca5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:56640/0 (socket says 172.21.15.169:56640) 2024-03-28T01:05:01.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.006+0000 7fd7177fe640 1 -- 172.21.15.169:0/3981583303 learned_addr learned my addr 172.21.15.169:0/3981583303 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:01.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.008+0000 7fd717fff640 1 --2- 172.21.15.169:0/3981583303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc95da0 0x55923cca50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:01.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.009+0000 7fd7177fe640 1 -- 172.21.15.169:0/3981583303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc95da0 msgr2=0x55923cca50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:01.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.009+0000 7fd7177fe640 1 --2- 172.21.15.169:0/3981583303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc95da0 0x55923cca50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.009+0000 7fd7177fe640 1 -- 172.21.15.169:0/3981583303 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55923cc9b900 msgr2=0x55923cca6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:01.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.009+0000 7fd7177fe640 1 --2- 172.21.15.169:0/3981583303 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55923cc9b900 0x55923cca6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.009+0000 7fd7177fe640 1 -- 172.21.15.169:0/3981583303 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55923cca6b40 con 0x55923cc96790 2024-03-28T01:05:01.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.010+0000 7fd7177fe640 1 --2- 172.21.15.169:0/3981583303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc96790 0x55923cca5600 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd7180738a0 tx=0x7fd708017b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3227a75eca593a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:01.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.011+0000 7fd6f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:05:01.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.012+0000 7fd6f77fe640 1 -- 172.21.15.169:0/3981583303 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd708015aa0 con 0x55923cc96790 2024-03-28T01:05:01.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.012+0000 7fd6f77fe640 1 -- 172.21.15.169:0/3981583303 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fd708015c40 con 0x55923cc96790 2024-03-28T01:05:01.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.012+0000 7fd720e70740 1 -- 172.21.15.169:0/3981583303 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd70c09b190 con 0x55923cc96790 2024-03-28T01:05:01.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.012+0000 7fd6f77fe640 1 -- 172.21.15.169:0/3981583303 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd708015e40 con 0x55923cc96790 2024-03-28T01:05:01.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.014+0000 7fd720e70740 10 client.26948 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:01.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.014+0000 7fd720e70740 1 -- 172.21.15.169:0/3981583303 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55923cd073e0 con 0x55923cc96790 2024-03-28T01:05:01.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.014+0000 7fd6f77fe640 1 -- 172.21.15.169:0/3981583303 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd70c09b190 con 0x55923cc96790 2024-03-28T01:05:01.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.018+0000 7fd6f77fe640 1 -- 172.21.15.169:0/3981583303 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd70801e070 con 0x55923cc96790 2024-03-28T01:05:01.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.018+0000 7fd720e70740 10 client.26948 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:01.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.018+0000 7fd720e70740 10 client.26948 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:01.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.018+0000 7fd720e70740 20 client.26948 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:01.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.018+0000 7fd720e70740 1 --2- 172.21.15.169:0/3981583303 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55923cd07dc0 0x55923cd0a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:01.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.018+0000 7fd720e70740 4 client.26948 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:05:01.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.018+0000 7fd720e70740 1 -- 172.21.15.169:0/3981583303 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55923cc08910 con 0x55923cd07dc0 2024-03-28T01:05:01.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.019+0000 7fd717fff640 1 --2- 172.21.15.169:0/3981583303 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55923cd07dc0 0x55923cd0a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:01.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.026+0000 7fd717fff640 1 --2- 172.21.15.169:0/3981583303 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55923cd07dc0 0x55923cd0a1e0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34dc232f6da433c0 server_cookie=6da645d247cc5d6b in_seq=0 out_seq=0 2024-03-28T01:05:01.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.026+0000 7fd6f77fe640 10 client.26948 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:01.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.026+0000 7fd6f77fe640 1 -- 172.21.15.169:0/3981583303 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55923cc08910 con 0x55923cd07dc0 2024-03-28T01:05:01.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.026+0000 7fd6f77fe640 10 client.26948 handle_command_reply: tid=0 2024-03-28T01:05:01.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.026+0000 7fd720e70740 1 client.26948 shutdown 2024-03-28T01:05:01.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.035+0000 7fd720e70740 1 -- 172.21.15.169:0/3981583303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc96790 msgr2=0x55923cca5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:01.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.035+0000 7fd720e70740 1 --2- 172.21.15.169:0/3981583303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc96790 0x55923cca5600 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd7180738a0 tx=0x7fd708017b50 comp rx=0 tx=0).stop 2024-03-28T01:05:01.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.036+0000 7fd720e70740 1 -- 172.21.15.169:0/3981583303 shutdown_connections 2024-03-28T01:05:01.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.036+0000 7fd720e70740 1 --2- 172.21.15.169:0/3981583303 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55923cd07dc0 0x55923cd0a1e0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.036+0000 7fd720e70740 1 --2- 172.21.15.169:0/3981583303 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55923cc9b900 0x55923cca6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.036+0000 7fd720e70740 1 --2- 172.21.15.169:0/3981583303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55923cc96790 0x55923cca5600 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.036+0000 7fd720e70740 1 --2- 172.21.15.169:0/3981583303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55923cc95da0 0x55923cca50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.036+0000 7fd720e70740 1 -- 172.21.15.169:0/3981583303 >> 172.21.15.169:0/3981583303 conn(0x55923cc91310 msgr2=0x55923cc9f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:01.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.036+0000 7fd6f77fe640 0 client.26948 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:01.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.038+0000 7fd720e70740 1 -- 172.21.15.169:0/3981583303 shutdown_connections 2024-03-28T01:05:01.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.038+0000 7fd720e70740 1 -- 172.21.15.169:0/3981583303 wait complete. 2024-03-28T01:05:01.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.038+0000 7fd720e70740 20 client.26948 trim_cache size 0 max 16384 2024-03-28T01:05:01.043 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:05:01.043 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "no active scrubs running", 2024-03-28T01:05:01.043 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": {} 2024-03-28T01:05:01.043 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:05:01.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.041+0000 7fd71e28a640 1 -- 172.21.15.169:0/2171338218 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd7000a29d0 msgr2=0x7fd7000a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:01.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.041+0000 7fd71e28a640 1 --2- 172.21.15.169:0/2171338218 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd7000a29d0 0x7fd7000a4e90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd710044000 tx=0x7fd7100359d0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.042+0000 7fd71e28a640 1 -- 172.21.15.169:0/2171338218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd718108910 msgr2=0x7fd718119400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:01.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.042+0000 7fd71e28a640 1 --2- 172.21.15.169:0/2171338218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd718108910 0x7fd718119400 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd70c009fd0 tx=0x7fd70c00b870 comp rx=0 tx=0).stop 2024-03-28T01:05:01.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.042+0000 7fd71e28a640 1 -- 172.21.15.169:0/2171338218 shutdown_connections 2024-03-28T01:05:01.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.043+0000 7fd71e28a640 1 --2- 172.21.15.169:0/2171338218 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd7000a29d0 0x7fd7000a4e90 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.043+0000 7fd71e28a640 1 --2- 172.21.15.169:0/2171338218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd71810dd70 0x7fd7181b6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.043+0000 7fd71e28a640 1 --2- 172.21.15.169:0/2171338218 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd71810b2d0 0x7fd718119940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.043+0000 7fd71e28a640 1 --2- 172.21.15.169:0/2171338218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd718108910 0x7fd718119400 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.043+0000 7fd71e28a640 1 -- 172.21.15.169:0/2171338218 >> 172.21.15.169:0/2171338218 conn(0x7fd71806e500 msgr2=0x7fd718109ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:01.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.043+0000 7fd71e28a640 1 -- 172.21.15.169:0/2171338218 shutdown_connections 2024-03-28T01:05:01.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.043+0000 7fd71e28a640 1 -- 172.21.15.169:0/2171338218 wait complete. 2024-03-28T01:05:01.075 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T01:05:01.075 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T01:05:01.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.286+0000 7f310b59e640 1 -- 172.21.15.169:0/1717187260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3104096830 msgr2=0x7f310409d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:01.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.286+0000 7f310b59e640 1 --2- 172.21.15.169:0/1717187260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3104096830 0x7f310409d0c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f30fc0108e0 tx=0x7f30fc02fb30 comp rx=0 tx=0).stop 2024-03-28T01:05:01.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.287+0000 7f310b59e640 1 -- 172.21.15.169:0/1717187260 shutdown_connections 2024-03-28T01:05:01.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.287+0000 7f310b59e640 1 --2- 172.21.15.169:0/1717187260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3104096830 0x7f310409d0c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.287+0000 7f310b59e640 1 --2- 172.21.15.169:0/1717187260 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3104095e70 0x7f31040962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.287+0000 7f310b59e640 1 --2- 172.21.15.169:0/1717187260 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3104094c80 0x7f31040950a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.287+0000 7f310b59e640 1 -- 172.21.15.169:0/1717187260 >> 172.21.15.169:0/1717187260 conn(0x7f3104090210 msgr2=0x7f3104092630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:01.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.293+0000 7f310b59e640 1 -- 172.21.15.169:0/1717187260 shutdown_connections 2024-03-28T01:05:01.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.294+0000 7f310b59e640 1 -- 172.21.15.169:0/1717187260 wait complete. 2024-03-28T01:05:01.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.295+0000 7f310b59e640 1 Processor -- start 2024-03-28T01:05:01.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.297+0000 7f310b59e640 1 -- start start 2024-03-28T01:05:01.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.297+0000 7f310b59e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3104094c80 0x7f31040a3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:01.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.297+0000 7f310b59e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3104095e70 0x7f31040a39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:01.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.297+0000 7f310b59e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3104096830 0x7f3104142730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:01.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.297+0000 7f310b59e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31040a8020 con 0x7f3104094c80 2024-03-28T01:05:01.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.297+0000 7f310b59e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31040a7ea0 con 0x7f3104095e70 2024-03-28T01:05:01.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.297+0000 7f310b59e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31040a81a0 con 0x7f3104096830 2024-03-28T01:05:01.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.299+0000 7f310ad9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3104096830 0x7f3104142730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:01.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.299+0000 7f310ad9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3104096830 0x7f3104142730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48806/0 (socket says 172.21.15.169:48806) 2024-03-28T01:05:01.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.299+0000 7f310ad9d640 1 -- 172.21.15.169:0/2278605551 learned_addr learned my addr 172.21.15.169:0/2278605551 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:01.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.299+0000 7f3109d9b640 1 --2- 172.21.15.169:0/2278605551 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3104095e70 0x7f31040a39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:01.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.299+0000 7f310ad9d640 1 -- 172.21.15.169:0/2278605551 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3104095e70 msgr2=0x7f31040a39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:01.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.299+0000 7f310ad9d640 1 --2- 172.21.15.169:0/2278605551 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3104095e70 0x7f31040a39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.299+0000 7f310ad9d640 1 -- 172.21.15.169:0/2278605551 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3104094c80 msgr2=0x7f31040a3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:01.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.299+0000 7f310ad9d640 1 --2- 172.21.15.169:0/2278605551 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3104094c80 0x7f31040a3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.299+0000 7f310ad9d640 1 -- 172.21.15.169:0/2278605551 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3104142c70 con 0x7f3104096830 2024-03-28T01:05:01.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.300+0000 7f310ad9d640 1 --2- 172.21.15.169:0/2278605551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3104096830 0x7f3104142730 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f30fc030140 tx=0x7f30fc031dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:01.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.303+0000 7f30fb7fe640 1 -- 172.21.15.169:0/2278605551 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30fc048070 con 0x7f3104096830 2024-03-28T01:05:01.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.303+0000 7f30fb7fe640 1 -- 172.21.15.169:0/2278605551 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30fc039ab0 con 0x7f3104096830 2024-03-28T01:05:01.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.303+0000 7f30fb7fe640 1 -- 172.21.15.169:0/2278605551 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30fc043410 con 0x7f3104096830 2024-03-28T01:05:01.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.305+0000 7f310b59e640 1 -- 172.21.15.169:0/2278605551 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3104142e40 con 0x7f3104096830 2024-03-28T01:05:01.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.305+0000 7f310b59e640 1 -- 172.21.15.169:0/2278605551 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31041432d0 con 0x7f3104096830 2024-03-28T01:05:01.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.307+0000 7f30f97fa640 1 -- 172.21.15.169:0/2278605551 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f310409b8d0 con 0x7f3104096830 2024-03-28T01:05:01.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.311+0000 7f30fb7fe640 1 -- 172.21.15.169:0/2278605551 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f30fc0035f0 con 0x7f3104096830 2024-03-28T01:05:01.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.312+0000 7f30fb7fe640 1 --2- 172.21.15.169:0/2278605551 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f30f40a2a00 0x7f30f40a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:01.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.312+0000 7f30fb7fe640 1 -- 172.21.15.169:0/2278605551 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f30fc0f0370 con 0x7f3104096830 2024-03-28T01:05:01.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.316+0000 7f310a59c640 1 --2- 172.21.15.169:0/2278605551 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f30f40a2a00 0x7f30f40a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:01.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.319+0000 7f30fb7fe640 1 -- 172.21.15.169:0/2278605551 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f30fc0bd090 con 0x7f3104096830 2024-03-28T01:05:01.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.323+0000 7f310a59c640 1 --2- 172.21.15.169:0/2278605551 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f30f40a2a00 0x7f30f40a4ec0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3104095cd0 tx=0x7f3100006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:01.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.763+0000 7f30f97fa640 1 -- 172.21.15.169:0/2278605551 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f310409baa0 con 0x7f3104096830 2024-03-28T01:05:01.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.764+0000 7f30fb7fe640 1 -- 172.21.15.169:0/2278605551 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7f30fc039c60 con 0x7f3104096830 2024-03-28T01:05:01.768 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T01:05:01.768 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T01:05:01.768 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T01:05:01.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.768+0000 7f30f97fa640 1 -- 172.21.15.169:0/2278605551 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f30f40a2a00 msgr2=0x7f30f40a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:01.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.768+0000 7f30f97fa640 1 --2- 172.21.15.169:0/2278605551 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f30f40a2a00 0x7f30f40a4ec0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3104095cd0 tx=0x7f3100006c90 comp rx=0 tx=0).stop 2024-03-28T01:05:01.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.768+0000 7f30f97fa640 1 -- 172.21.15.169:0/2278605551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3104096830 msgr2=0x7f3104142730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:01.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.769+0000 7f30f97fa640 1 --2- 172.21.15.169:0/2278605551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3104096830 0x7f3104142730 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f30fc030140 tx=0x7f30fc031dd0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.770+0000 7f30f97fa640 1 -- 172.21.15.169:0/2278605551 shutdown_connections 2024-03-28T01:05:01.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.770+0000 7f30f97fa640 1 --2- 172.21.15.169:0/2278605551 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f30f40a2a00 0x7f30f40a4ec0 secure :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3104095cd0 tx=0x7f3100006c90 comp rx=0 tx=0).stop 2024-03-28T01:05:01.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.770+0000 7f30f97fa640 1 --2- 172.21.15.169:0/2278605551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3104096830 0x7f3104142730 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.770+0000 7f30f97fa640 1 --2- 172.21.15.169:0/2278605551 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3104095e70 0x7f31040a39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.770+0000 7f30f97fa640 1 --2- 172.21.15.169:0/2278605551 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3104094c80 0x7f31040a3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.770+0000 7f30f97fa640 1 -- 172.21.15.169:0/2278605551 >> 172.21.15.169:0/2278605551 conn(0x7f3104090210 msgr2=0x7f3104099230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:01.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.771+0000 7f30f97fa640 1 -- 172.21.15.169:0/2278605551 shutdown_connections 2024-03-28T01:05:01.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.773+0000 7f30f97fa640 1 -- 172.21.15.169:0/2278605551 wait complete. 2024-03-28T01:05:01.799 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:05:01.799 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T01:05:01.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:01 smithi177 ceph-mon[36206]: pgmap v606: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s rd, 376 MiB/s wr, 4.14k op/s 2024-03-28T01:05:01.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.982+0000 7fb536909640 1 -- 172.21.15.169:0/1415743693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb530101b50 msgr2=0x7fb530101f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:01.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.982+0000 7fb536909640 1 --2- 172.21.15.169:0/1415743693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb530101b50 0x7fb530101f70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb520009900 tx=0x7fb52002bc30 comp rx=0 tx=0).stop 2024-03-28T01:05:01.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.982+0000 7fb536909640 1 -- 172.21.15.169:0/1415743693 shutdown_connections 2024-03-28T01:05:01.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.982+0000 7fb536909640 1 --2- 172.21.15.169:0/1415743693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb530101240 0x7fb530108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.982+0000 7fb536909640 1 --2- 172.21.15.169:0/1415743693 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb530100880 0x7fb530100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.982+0000 7fb536909640 1 --2- 172.21.15.169:0/1415743693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb530101b50 0x7fb530101f70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.982+0000 7fb536909640 1 -- 172.21.15.169:0/1415743693 >> 172.21.15.169:0/1415743693 conn(0x7fb5300fb2d0 msgr2=0x7fb5300fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:01.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.982+0000 7fb536909640 1 -- 172.21.15.169:0/1415743693 shutdown_connections 2024-03-28T01:05:01.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.982+0000 7fb536909640 1 -- 172.21.15.169:0/1415743693 wait complete. 2024-03-28T01:05:01.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.983+0000 7fb536909640 1 Processor -- start 2024-03-28T01:05:01.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.983+0000 7fb536909640 1 -- start start 2024-03-28T01:05:01.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.983+0000 7fb536909640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb530100880 0x7fb530197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:01.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.983+0000 7fb536909640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb530101240 0x7fb530197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:01.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.983+0000 7fb536909640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb530101b50 0x7fb53019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:01.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.983+0000 7fb536909640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53010c260 con 0x7fb530101b50 2024-03-28T01:05:01.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.983+0000 7fb536909640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53010c0e0 con 0x7fb530100880 2024-03-28T01:05:01.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.983+0000 7fb536909640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53010c3e0 con 0x7fb530101240 2024-03-28T01:05:01.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.983+0000 7fb534ea6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb530101b50 0x7fb53019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:01.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.983+0000 7fb52f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb530101240 0x7fb530197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:01.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.983+0000 7fb52ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb530100880 0x7fb530197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:01.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.983+0000 7fb534ea6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb530101b50 0x7fb53019efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34154/0 (socket says 172.21.15.169:34154) 2024-03-28T01:05:01.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.983+0000 7fb52f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb530101240 0x7fb530197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48830/0 (socket says 172.21.15.169:48830) 2024-03-28T01:05:01.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.983+0000 7fb534ea6640 1 -- 172.21.15.169:0/3830338948 learned_addr learned my addr 172.21.15.169:0/3830338948 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:01.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.984+0000 7fb52ffff640 1 -- 172.21.15.169:0/3830338948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb530101240 msgr2=0x7fb530197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:01.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.984+0000 7fb52ffff640 1 --2- 172.21.15.169:0/3830338948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb530101240 0x7fb530197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.984+0000 7fb52ffff640 1 -- 172.21.15.169:0/3830338948 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb530101b50 msgr2=0x7fb53019efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:01.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.984+0000 7fb52ffff640 1 --2- 172.21.15.169:0/3830338948 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb530101b50 0x7fb53019efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.984+0000 7fb52ffff640 1 -- 172.21.15.169:0/3830338948 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53019f610 con 0x7fb530100880 2024-03-28T01:05:01.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.984+0000 7fb52f7fe640 1 --2- 172.21.15.169:0/3830338948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb530101240 0x7fb530197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:05:01.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.984+0000 7fb52ffff640 1 --2- 172.21.15.169:0/3830338948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb530100880 0x7fb530197a40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb52002c170 tx=0x7fb52002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:01.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.984+0000 7fb52d7fa640 1 -- 172.21.15.169:0/3830338948 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb520041070 con 0x7fb530100880 2024-03-28T01:05:01.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.984+0000 7fb536909640 1 -- 172.21.15.169:0/3830338948 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb53019f8a0 con 0x7fb530100880 2024-03-28T01:05:01.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.984+0000 7fb52d7fa640 1 -- 172.21.15.169:0/3830338948 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5200054e0 con 0x7fb530100880 2024-03-28T01:05:01.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.984+0000 7fb52d7fa640 1 -- 172.21.15.169:0/3830338948 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb520034440 con 0x7fb530100880 2024-03-28T01:05:01.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.984+0000 7fb536909640 1 -- 172.21.15.169:0/3830338948 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb53019fdb0 con 0x7fb530100880 2024-03-28T01:05:01.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.986+0000 7fb52d7fa640 1 -- 172.21.15.169:0/3830338948 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb520044070 con 0x7fb530100880 2024-03-28T01:05:01.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.987+0000 7fb52d7fa640 1 --2- 172.21.15.169:0/3830338948 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb5180a2800 0x7fb5180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:01.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.987+0000 7fb5395d2740 1 Processor -- start 2024-03-28T01:05:01.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.987+0000 7fb52d7fa640 1 -- 172.21.15.169:0/3830338948 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb5200f07c0 con 0x7fb530100880 2024-03-28T01:05:01.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.987+0000 7fb52f7fe640 1 --2- 172.21.15.169:0/3830338948 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb5180a2800 0x7fb5180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:01.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.987+0000 7fb5395d2740 1 -- start start 2024-03-28T01:05:01.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.987+0000 7fb5395d2740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f0690 0x5556134f0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:01.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.987+0000 7fb5395d2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f1080 0x5556134f5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:01.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.987+0000 7fb5395d2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f61f0 0x5556134f8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:01.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.987+0000 7fb5395d2740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556134b3610 con 0x5556134f0690 2024-03-28T01:05:01.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.987+0000 7fb5395d2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556134563c0 con 0x5556134f61f0 2024-03-28T01:05:01.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.987+0000 7fb5395d2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556133fed60 con 0x5556134f1080 2024-03-28T01:05:01.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.988+0000 7fb52ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f0690 0x5556134f0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:01.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.988+0000 7fb52ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f0690 0x5556134f0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34156/0 (socket says 172.21.15.169:34156) 2024-03-28T01:05:01.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.988+0000 7fb52ffff640 1 -- 172.21.15.169:0/19085753 learned_addr learned my addr 172.21.15.169:0/19085753 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:01.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.988+0000 7fb52f7fe640 1 --2- 172.21.15.169:0/19085753 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f1080 0x5556134f5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:01.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.988+0000 7fb534ea6640 1 --2- 172.21.15.169:0/19085753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f61f0 0x5556134f8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:01.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.988+0000 7fb52f7fe640 1 --2- 172.21.15.169:0/3830338948 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb5180a2800 0x7fb5180a4cc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb530198f00 tx=0x7fb524005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:01.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.988+0000 7fb52ffff640 1 -- 172.21.15.169:0/19085753 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f1080 msgr2=0x5556134f5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:01.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.988+0000 7fb52ffff640 1 --2- 172.21.15.169:0/19085753 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f1080 0x5556134f5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.988+0000 7fb52ffff640 1 -- 172.21.15.169:0/19085753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f61f0 msgr2=0x5556134f8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.988+0000 7fb52ffff640 1 --2- 172.21.15.169:0/19085753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f61f0 0x5556134f8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.988+0000 7fb52ffff640 1 -- 172.21.15.169:0/19085753 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55561316a3f0 con 0x5556134f0690 2024-03-28T01:05:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.988+0000 7fb52ffff640 1 --2- 172.21.15.169:0/19085753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f0690 0x5556134f0ab0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb5200c5030 tx=0x7fb520003990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeb9861df145e5ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.988+0000 7fb5167fc640 1 -- 172.21.15.169:0/19085753 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52004b040 con 0x5556134f0690 2024-03-28T01:05:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.988+0000 7fb5167fc640 1 -- 172.21.15.169:0/19085753 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb520004c80 con 0x5556134f0690 2024-03-28T01:05:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.988+0000 7fb5167fc640 1 -- 172.21.15.169:0/19085753 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5200c5560 con 0x5556134f0690 2024-03-28T01:05:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.989+0000 7fb5395d2740 1 -- 172.21.15.169:0/19085753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f0690 msgr2=0x5556134f0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.989+0000 7fb5395d2740 1 --2- 172.21.15.169:0/19085753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f0690 0x5556134f0ab0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb5200c5030 tx=0x7fb520003990 comp rx=0 tx=0).stop 2024-03-28T01:05:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.989+0000 7fb5395d2740 1 -- 172.21.15.169:0/19085753 shutdown_connections 2024-03-28T01:05:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.989+0000 7fb5395d2740 1 --2- 172.21.15.169:0/19085753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f61f0 0x5556134f8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.989+0000 7fb5395d2740 1 --2- 172.21.15.169:0/19085753 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f1080 0x5556134f5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.989+0000 7fb5395d2740 1 --2- 172.21.15.169:0/19085753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f0690 0x5556134f0ab0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.989+0000 7fb5395d2740 1 -- 172.21.15.169:0/19085753 >> 172.21.15.169:0/19085753 conn(0x5556134ebc00 msgr2=0x5556134ec000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.989+0000 7fb5395d2740 1 -- 172.21.15.169:0/19085753 shutdown_connections 2024-03-28T01:05:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.989+0000 7fb5395d2740 1 -- 172.21.15.169:0/19085753 wait complete. 2024-03-28T01:05:02.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.989+0000 7fb5395d2740 1 -- start start 2024-03-28T01:05:02.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.990+0000 7fb5395d2740 1 Processor -- start 2024-03-28T01:05:02.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.990+0000 7fb5395d2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f0690 0x5556134fa640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.990+0000 7fb5395d2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f1080 0x5556134fab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.990+0000 7fb5395d2740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f61f0 0x555613501bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.990+0000 7fb5395d2740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x555613412240 con 0x5556134f61f0 2024-03-28T01:05:02.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.990+0000 7fb5395d2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x555613452460 con 0x5556134f0690 2024-03-28T01:05:02.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.990+0000 7fb5395d2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556134f8f60 con 0x5556134f1080 2024-03-28T01:05:02.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.990+0000 7fb5395d2740 10 client.0 _ll_register_callbacks cb 0x555613411c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:02.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.990+0000 7fb534ea6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f61f0 0x555613501bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.990+0000 7fb52f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f1080 0x5556134fab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.990+0000 7fb534ea6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f61f0 0x555613501bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34166/0 (socket says 172.21.15.169:34166) 2024-03-28T01:05:02.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.990+0000 7fb534ea6640 1 -- 172.21.15.169:0/1550548987 learned_addr learned my addr 172.21.15.169:0/1550548987 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:02.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.990+0000 7fb52ffff640 1 --2- 172.21.15.169:0/1550548987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f0690 0x5556134fa640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.990+0000 7fb52f7fe640 1 -- 172.21.15.169:0/1550548987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f0690 msgr2=0x5556134fa640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:02.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.990+0000 7fb52f7fe640 1 --2- 172.21.15.169:0/1550548987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f0690 0x5556134fa640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.990+0000 7fb52f7fe640 1 -- 172.21.15.169:0/1550548987 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f61f0 msgr2=0x555613501bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:02.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.990+0000 7fb52f7fe640 1 --2- 172.21.15.169:0/1550548987 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f61f0 0x555613501bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.990+0000 7fb52f7fe640 1 -- 172.21.15.169:0/1550548987 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5556132347d0 con 0x5556134f1080 2024-03-28T01:05:02.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.991+0000 7fb52f7fe640 1 --2- 172.21.15.169:0/1550548987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f1080 0x5556134fab80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x5556134f1af0 tx=0x7fb524004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ea67040078edbd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:02.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.991+0000 7fb50f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:05:02.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.991+0000 7fb50f7fe640 1 -- 172.21.15.169:0/1550548987 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52400f810 con 0x5556134f1080 2024-03-28T01:05:02.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.991+0000 7fb50f7fe640 1 -- 172.21.15.169:0/1550548987 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb52400f9b0 con 0x5556134f1080 2024-03-28T01:05:02.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.991+0000 7fb5395d2740 1 -- 172.21.15.169:0/1550548987 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb520004c80 con 0x5556134f1080 2024-03-28T01:05:02.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.991+0000 7fb50f7fe640 1 -- 172.21.15.169:0/1550548987 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb52400fbb0 con 0x5556134f1080 2024-03-28T01:05:02.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.991+0000 7fb5395d2740 10 client.26896 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:02.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.991+0000 7fb5395d2740 1 -- 172.21.15.169:0/1550548987 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x555613504fa0 con 0x5556134f1080 2024-03-28T01:05:02.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.991+0000 7fb50f7fe640 1 -- 172.21.15.169:0/1550548987 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb520004c80 con 0x5556134f1080 2024-03-28T01:05:02.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.991+0000 7fb50f7fe640 1 -- 172.21.15.169:0/1550548987 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb52401b430 con 0x5556134f1080 2024-03-28T01:05:02.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.991+0000 7fb5395d2740 10 client.26896 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:02.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.991+0000 7fb5395d2740 10 client.26896 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:05:02.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.991+0000 7fb5395d2740 20 client.26896 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:02.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.991+0000 7fb5395d2740 1 --2- 172.21.15.169:0/1550548987 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x555613506650 0x555613508af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.991+0000 7fb5395d2740 4 client.26896 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:05:02.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.991+0000 7fb5395d2740 1 -- 172.21.15.169:0/1550548987 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555613463900 con 0x555613506650 2024-03-28T01:05:02.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.992+0000 7fb52ffff640 1 --2- 172.21.15.169:0/1550548987 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x555613506650 0x555613508af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.992+0000 7fb50f7fe640 10 client.26896 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:05:02.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.992+0000 7fb52ffff640 1 --2- 172.21.15.169:0/1550548987 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x555613506650 0x555613508af0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6e9ba7639fa806f6 server_cookie=aed7d2297af0bd08 in_seq=0 out_seq=0 2024-03-28T01:05:02.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.993+0000 7fb50f7fe640 1 -- 172.21.15.169:0/1550548987 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555613463900 con 0x555613506650 2024-03-28T01:05:02.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.993+0000 7fb50f7fe640 10 client.26896 handle_command_reply: tid=0 2024-03-28T01:05:02.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.993+0000 7fb5395d2740 1 client.26896 shutdown 2024-03-28T01:05:02.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.993+0000 7fb5395d2740 1 -- 172.21.15.169:0/1550548987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f1080 msgr2=0x5556134fab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:02.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.993+0000 7fb5395d2740 1 --2- 172.21.15.169:0/1550548987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f1080 0x5556134fab80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x5556134f1af0 tx=0x7fb524004730 comp rx=0 tx=0).stop 2024-03-28T01:05:02.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.993+0000 7fb5395d2740 1 -- 172.21.15.169:0/1550548987 shutdown_connections 2024-03-28T01:05:02.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.993+0000 7fb5395d2740 1 --2- 172.21.15.169:0/1550548987 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x555613506650 0x555613508af0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.993+0000 7fb5395d2740 1 --2- 172.21.15.169:0/1550548987 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f61f0 0x555613501bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.993+0000 7fb5395d2740 1 --2- 172.21.15.169:0/1550548987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f1080 0x5556134fab80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.993+0000 7fb50f7fe640 0 client.26896 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:05:02.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.993+0000 7fb5395d2740 1 --2- 172.21.15.169:0/1550548987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f0690 0x5556134fa640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.993+0000 7fb5395d2740 1 -- 172.21.15.169:0/1550548987 >> 172.21.15.169:0/1550548987 conn(0x5556134ebc00 msgr2=0x5556134eae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:02.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.993+0000 7fb5395d2740 1 -- 172.21.15.169:0/1550548987 shutdown_connections 2024-03-28T01:05:02.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.993+0000 7fb5395d2740 1 -- 172.21.15.169:0/1550548987 wait complete. 2024-03-28T01:05:02.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:01.993+0000 7fb5395d2740 20 client.26896 trim_cache size 0 max 16384 2024-03-28T01:05:02.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.006+0000 7fb5395d2740 1 Processor -- start 2024-03-28T01:05:02.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.006+0000 7fb5395d2740 1 -- start start 2024-03-28T01:05:02.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.006+0000 7fb5395d2740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f0690 0x5556134fa640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.006+0000 7fb5395d2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f1080 0x5556134fab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.006+0000 7fb5395d2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f61f0 0x555613501b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.006+0000 7fb5395d2740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5301ac690 con 0x5556134f0690 2024-03-28T01:05:02.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.006+0000 7fb5395d2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5301aa190 con 0x5556134f61f0 2024-03-28T01:05:02.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.006+0000 7fb5395d2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5301a8230 con 0x5556134f1080 2024-03-28T01:05:02.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.006+0000 7fb52ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f0690 0x5556134fa640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.006+0000 7fb534ea6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f61f0 0x555613501b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.006+0000 7fb52ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f0690 0x5556134fa640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34180/0 (socket says 172.21.15.169:34180) 2024-03-28T01:05:02.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.006+0000 7fb52f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f1080 0x5556134fab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.006+0000 7fb52ffff640 1 -- 172.21.15.169:0/1033631882 learned_addr learned my addr 172.21.15.169:0/1033631882 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:02.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.007+0000 7fb534ea6640 1 -- 172.21.15.169:0/1033631882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f1080 msgr2=0x5556134fab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:02.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.007+0000 7fb534ea6640 1 --2- 172.21.15.169:0/1033631882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f1080 0x5556134fab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.007+0000 7fb534ea6640 1 -- 172.21.15.169:0/1033631882 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f0690 msgr2=0x5556134fa640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:02.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.007+0000 7fb534ea6640 1 --2- 172.21.15.169:0/1033631882 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f0690 0x5556134fa640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.007+0000 7fb534ea6640 1 -- 172.21.15.169:0/1033631882 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556134f1910 con 0x5556134f61f0 2024-03-28T01:05:02.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.007+0000 7fb52f7fe640 1 --2- 172.21.15.169:0/1033631882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f1080 0x5556134fab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:05:02.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.007+0000 7fb534ea6640 1 --2- 172.21.15.169:0/1033631882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f61f0 0x555613501b80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb528012770 tx=0x7fb528009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0568b2be23ed2da server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:02.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.007+0000 7fb515ffb640 1 -- 172.21.15.169:0/1033631882 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52800f920 con 0x5556134f61f0 2024-03-28T01:05:02.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.007+0000 7fb515ffb640 1 -- 172.21.15.169:0/1033631882 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb528021070 con 0x5556134f61f0 2024-03-28T01:05:02.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.007+0000 7fb515ffb640 1 -- 172.21.15.169:0/1033631882 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb528010be0 con 0x5556134f61f0 2024-03-28T01:05:02.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.008+0000 7fb5395d2740 1 -- 172.21.15.169:0/1033631882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f61f0 msgr2=0x555613501b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:02.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.008+0000 7fb5395d2740 1 --2- 172.21.15.169:0/1033631882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f61f0 0x555613501b80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb528012770 tx=0x7fb528009500 comp rx=0 tx=0).stop 2024-03-28T01:05:02.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.008+0000 7fb5395d2740 1 -- 172.21.15.169:0/1033631882 shutdown_connections 2024-03-28T01:05:02.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.008+0000 7fb5395d2740 1 --2- 172.21.15.169:0/1033631882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f61f0 0x555613501b80 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.008+0000 7fb5395d2740 1 --2- 172.21.15.169:0/1033631882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f1080 0x5556134fab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.008+0000 7fb5395d2740 1 --2- 172.21.15.169:0/1033631882 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f0690 0x5556134fa640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.008+0000 7fb5395d2740 1 -- 172.21.15.169:0/1033631882 >> 172.21.15.169:0/1033631882 conn(0x5556134ebc00 msgr2=0x5556134ebfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:02.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.008+0000 7fb5395d2740 1 -- 172.21.15.169:0/1033631882 shutdown_connections 2024-03-28T01:05:02.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.008+0000 7fb5395d2740 1 -- 172.21.15.169:0/1033631882 wait complete. 2024-03-28T01:05:02.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.008+0000 7fb5395d2740 1 -- start start 2024-03-28T01:05:02.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb5395d2740 1 Processor -- start 2024-03-28T01:05:02.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb5395d2740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f0690 0x5556134ff930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb5395d2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f1080 0x5556134ffe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb5395d2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f61f0 0x555613500e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb5395d2740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5301a8830 con 0x5556134f0690 2024-03-28T01:05:02.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb5395d2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5301a9b30 con 0x5556134f1080 2024-03-28T01:05:02.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb5395d2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5301a8e70 con 0x5556134f61f0 2024-03-28T01:05:02.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb5395d2740 10 client.0 _ll_register_callbacks cb 0x7fb524004590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:02.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb52ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f0690 0x5556134ff930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb534ea6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f61f0 0x555613500e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb52ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f0690 0x5556134ff930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34196/0 (socket says 172.21.15.169:34196) 2024-03-28T01:05:02.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb52ffff640 1 -- 172.21.15.169:0/2019560863 learned_addr learned my addr 172.21.15.169:0/2019560863 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:02.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb52f7fe640 1 --2- 172.21.15.169:0/2019560863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f1080 0x5556134ffe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb52ffff640 1 -- 172.21.15.169:0/2019560863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f61f0 msgr2=0x555613500e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:02.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb52ffff640 1 --2- 172.21.15.169:0/2019560863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f61f0 0x555613500e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb52ffff640 1 -- 172.21.15.169:0/2019560863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f1080 msgr2=0x5556134ffe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:02.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb52ffff640 1 --2- 172.21.15.169:0/2019560863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f1080 0x5556134ffe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb52ffff640 1 -- 172.21.15.169:0/2019560863 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555613501420 con 0x5556134f0690 2024-03-28T01:05:02.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb534ea6640 1 --2- 172.21.15.169:0/2019560863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f61f0 0x555613500e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:05:02.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb52f7fe640 1 --2- 172.21.15.169:0/2019560863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f1080 0x5556134ffe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:05:02.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb52ffff640 1 --2- 172.21.15.169:0/2019560863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f0690 0x5556134ff930 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb5200b9840 tx=0x7fb520005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b6d4280169dbae7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:02.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.009+0000 7fb50f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:05:02.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.010+0000 7fb50f7fe640 1 -- 172.21.15.169:0/2019560863 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52009f070 con 0x5556134f0690 2024-03-28T01:05:02.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.010+0000 7fb50f7fe640 1 -- 172.21.15.169:0/2019560863 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb5200b94e0 con 0x5556134f0690 2024-03-28T01:05:02.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.010+0000 7fb5395d2740 1 -- 172.21.15.169:0/2019560863 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb528021070 con 0x5556134f0690 2024-03-28T01:05:02.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.010+0000 7fb50f7fe640 1 -- 172.21.15.169:0/2019560863 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb52004b040 con 0x5556134f0690 2024-03-28T01:05:02.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.010+0000 7fb5395d2740 10 client.17109 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:02.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.010+0000 7fb5395d2740 1 -- 172.21.15.169:0/2019560863 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5556135625b0 con 0x5556134f0690 2024-03-28T01:05:02.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.010+0000 7fb50f7fe640 1 -- 172.21.15.169:0/2019560863 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb528021070 con 0x5556134f0690 2024-03-28T01:05:02.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.010+0000 7fb50f7fe640 1 -- 172.21.15.169:0/2019560863 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb52009bce0 con 0x5556134f0690 2024-03-28T01:05:02.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.010+0000 7fb5395d2740 10 client.17109 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:02.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.010+0000 7fb5395d2740 10 client.17109 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:05:02.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.010+0000 7fb5395d2740 20 client.17109 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:02.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.010+0000 7fb5395d2740 1 --2- 172.21.15.169:0/2019560863 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x555613562b80 0x555613564fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.010+0000 7fb5395d2740 4 client.17109 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:05:02.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.010+0000 7fb5395d2740 1 -- 172.21.15.169:0/2019560863 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55561345dc50 con 0x555613562b80 2024-03-28T01:05:02.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.010+0000 7fb52f7fe640 1 --2- 172.21.15.169:0/2019560863 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x555613562b80 0x555613564fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.011+0000 7fb50f7fe640 10 client.17109 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:05:02.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.011+0000 7fb52f7fe640 1 --2- 172.21.15.169:0/2019560863 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x555613562b80 0x555613564fa0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3a96bd3f126ab09b server_cookie=88cafcd8ea127996 in_seq=0 out_seq=0 2024-03-28T01:05:02.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.011+0000 7fb50f7fe640 1 -- 172.21.15.169:0/2019560863 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55561345dc50 con 0x555613562b80 2024-03-28T01:05:02.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.011+0000 7fb50f7fe640 10 client.17109 handle_command_reply: tid=0 2024-03-28T01:05:02.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.011+0000 7fb5395d2740 1 client.17109 shutdown 2024-03-28T01:05:02.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.011+0000 7fb5395d2740 1 -- 172.21.15.169:0/2019560863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f0690 msgr2=0x5556134ff930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:02.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.011+0000 7fb5395d2740 1 --2- 172.21.15.169:0/2019560863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f0690 0x5556134ff930 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb5200b9840 tx=0x7fb520005b50 comp rx=0 tx=0).stop 2024-03-28T01:05:02.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.011+0000 7fb5395d2740 1 -- 172.21.15.169:0/2019560863 shutdown_connections 2024-03-28T01:05:02.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.011+0000 7fb5395d2740 1 --2- 172.21.15.169:0/2019560863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556134f61f0 0x555613500e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.011+0000 7fb5395d2740 1 --2- 172.21.15.169:0/2019560863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556134f1080 0x5556134ffe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.011+0000 7fb5395d2740 1 --2- 172.21.15.169:0/2019560863 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x555613562b80 0x555613564fa0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.011+0000 7fb5395d2740 1 --2- 172.21.15.169:0/2019560863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5556134f0690 0x5556134ff930 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.011+0000 7fb50f7fe640 0 client.17109 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:05:02.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.011+0000 7fb5395d2740 1 -- 172.21.15.169:0/2019560863 >> 172.21.15.169:0/2019560863 conn(0x5556134ebc00 msgr2=0x5556134fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:02.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.011+0000 7fb5395d2740 1 -- 172.21.15.169:0/2019560863 shutdown_connections 2024-03-28T01:05:02.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.012+0000 7fb5395d2740 1 -- 172.21.15.169:0/2019560863 wait complete. 2024-03-28T01:05:02.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.012+0000 7fb5395d2740 20 client.17109 trim_cache size 0 max 16384 2024-03-28T01:05:02.023 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:05:02.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.012+0000 7fb536909640 1 -- 172.21.15.169:0/3830338948 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb5180a2800 msgr2=0x7fb5180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:02.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.012+0000 7fb536909640 1 --2- 172.21.15.169:0/3830338948 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb5180a2800 0x7fb5180a4cc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb530198f00 tx=0x7fb524005f90 comp rx=0 tx=0).stop 2024-03-28T01:05:02.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.012+0000 7fb536909640 1 -- 172.21.15.169:0/3830338948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb530100880 msgr2=0x7fb530197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:02.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.012+0000 7fb536909640 1 --2- 172.21.15.169:0/3830338948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb530100880 0x7fb530197a40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb52002c170 tx=0x7fb52002c650 comp rx=0 tx=0).stop 2024-03-28T01:05:02.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.012+0000 7fb536909640 1 -- 172.21.15.169:0/3830338948 shutdown_connections 2024-03-28T01:05:02.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.012+0000 7fb536909640 1 --2- 172.21.15.169:0/3830338948 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb5180a2800 0x7fb5180a4cc0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.012+0000 7fb536909640 1 --2- 172.21.15.169:0/3830338948 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb530101b50 0x7fb53019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.012+0000 7fb536909640 1 --2- 172.21.15.169:0/3830338948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb530101240 0x7fb530197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.012+0000 7fb536909640 1 --2- 172.21.15.169:0/3830338948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb530100880 0x7fb530197a40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.012+0000 7fb536909640 1 -- 172.21.15.169:0/3830338948 >> 172.21.15.169:0/3830338948 conn(0x7fb5300fb2d0 msgr2=0x7fb5300fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:02.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.012+0000 7fb536909640 1 -- 172.21.15.169:0/3830338948 shutdown_connections 2024-03-28T01:05:02.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.012+0000 7fb536909640 1 -- 172.21.15.169:0/3830338948 wait complete. 2024-03-28T01:05:02.033 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:01 smithi169 ceph-mon[32274]: pgmap v606: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s rd, 376 MiB/s wr, 4.14k op/s 2024-03-28T01:05:02.034 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:05:02.034 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T01:05:02.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:01 smithi174 ceph-mon[36060]: pgmap v606: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s rd, 376 MiB/s wr, 4.14k op/s 2024-03-28T01:05:02.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.278+0000 7fdf48626640 1 -- 172.21.15.169:0/3380508238 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf4010b010 msgr2=0x7fdf4010d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:02.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.278+0000 7fdf48626640 1 --2- 172.21.15.169:0/3380508238 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf4010b010 0x7fdf4010d480 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fdf30009860 tx=0x7fdf3002bbb0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.283+0000 7fdf48626640 1 -- 172.21.15.169:0/3380508238 shutdown_connections 2024-03-28T01:05:02.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.283+0000 7fdf48626640 1 --2- 172.21.15.169:0/3380508238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf4010d9c0 0x7fdf4010de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.283+0000 7fdf48626640 1 --2- 172.21.15.169:0/3380508238 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf4010b010 0x7fdf4010d480 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.283+0000 7fdf48626640 1 --2- 172.21.15.169:0/3380508238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf401086a0 0x7fdf4010aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.283+0000 7fdf48626640 1 -- 172.21.15.169:0/3380508238 >> 172.21.15.169:0/3380508238 conn(0x7fdf4006e500 msgr2=0x7fdf40070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:02.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.290+0000 7fdf48626640 1 -- 172.21.15.169:0/3380508238 shutdown_connections 2024-03-28T01:05:02.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.290+0000 7fdf48626640 1 -- 172.21.15.169:0/3380508238 wait complete. 2024-03-28T01:05:02.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.291+0000 7fdf48626640 1 Processor -- start 2024-03-28T01:05:02.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.291+0000 7fdf48626640 1 -- start start 2024-03-28T01:05:02.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.292+0000 7fdf48626640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf401086a0 0x7fdf40119a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.292+0000 7fdf48626640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf4010b010 0x7fdf40119f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.292+0000 7fdf48626640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf4010d9c0 0x7fdf40118090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.292+0000 7fdf48626640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf40110820 con 0x7fdf4010d9c0 2024-03-28T01:05:02.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.292+0000 7fdf48626640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf401106a0 con 0x7fdf401086a0 2024-03-28T01:05:02.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.292+0000 7fdf48626640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf401109a0 con 0x7fdf4010b010 2024-03-28T01:05:02.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.293+0000 7fdf45bc1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf4010b010 0x7fdf40119f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.293+0000 7fdf45bc1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf4010b010 0x7fdf40119f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48892/0 (socket says 172.21.15.169:48892) 2024-03-28T01:05:02.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.293+0000 7fdf45bc1640 1 -- 172.21.15.169:0/3963703828 learned_addr learned my addr 172.21.15.169:0/3963703828 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:02.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.293+0000 7fdf45bc1640 1 -- 172.21.15.169:0/3963703828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf401086a0 msgr2=0x7fdf40119a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:05:02.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.293+0000 7fdf45bc1640 1 --2- 172.21.15.169:0/3963703828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf401086a0 0x7fdf40119a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.293+0000 7fdf45bc1640 1 -- 172.21.15.169:0/3963703828 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf4010d9c0 msgr2=0x7fdf40118090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:05:02.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.293+0000 7fdf45bc1640 1 --2- 172.21.15.169:0/3963703828 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf4010d9c0 0x7fdf40118090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.293+0000 7fdf45bc1640 1 -- 172.21.15.169:0/3963703828 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf401188b0 con 0x7fdf4010b010 2024-03-28T01:05:02.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.294+0000 7fdf45bc1640 1 --2- 172.21.15.169:0/3963703828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf4010b010 0x7fdf40119f40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fdf300349e0 tx=0x7fdf300043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:02.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.296+0000 7fdf377fe640 1 -- 172.21.15.169:0/3963703828 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf30039040 con 0x7fdf4010b010 2024-03-28T01:05:02.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.296+0000 7fdf48626640 1 -- 172.21.15.169:0/3963703828 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf40118b40 con 0x7fdf4010b010 2024-03-28T01:05:02.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.296+0000 7fdf48626640 1 -- 172.21.15.169:0/3963703828 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf401b7560 con 0x7fdf4010b010 2024-03-28T01:05:02.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.296+0000 7fdf377fe640 1 -- 172.21.15.169:0/3963703828 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdf30042070 con 0x7fdf4010b010 2024-03-28T01:05:02.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.296+0000 7fdf377fe640 1 -- 172.21.15.169:0/3963703828 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3003d690 con 0x7fdf4010b010 2024-03-28T01:05:02.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.301+0000 7fdf4b20c740 1 Processor -- start 2024-03-28T01:05:02.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.303+0000 7fdf377fe640 1 -- 172.21.15.169:0/3963703828 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fdf3002c730 con 0x7fdf4010b010 2024-03-28T01:05:02.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.305+0000 7fdf377fe640 1 --2- 172.21.15.169:0/3963703828 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdf2c0a2a00 0x7fdf2c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.305+0000 7fdf377fe640 1 -- 172.21.15.169:0/3963703828 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fdf300f07c0 con 0x7fdf4010b010 2024-03-28T01:05:02.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.307+0000 7fdf46bc3640 1 --2- 172.21.15.169:0/3963703828 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdf2c0a2a00 0x7fdf2c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.308+0000 7fdf4b20c740 1 -- start start 2024-03-28T01:05:02.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.308+0000 7fdf4b20c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a1754690 0x5564a1754ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.308+0000 7fdf4b20c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564a1755080 0x5564a1759b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.308+0000 7fdf4b20c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a175a1f0 0x5564a175c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.308+0000 7fdf4b20c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564a1717610 con 0x5564a1754690 2024-03-28T01:05:02.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.308+0000 7fdf4b20c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564a16ba3c0 con 0x5564a1755080 2024-03-28T01:05:02.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.308+0000 7fdf4b20c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564a1662d60 con 0x5564a175a1f0 2024-03-28T01:05:02.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.309+0000 7fdf45bc1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564a1755080 0x5564a1759b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.309+0000 7fdf45bc1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564a1755080 0x5564a1759b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:56706/0 (socket says 172.21.15.169:56706) 2024-03-28T01:05:02.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.309+0000 7fdf45bc1640 1 -- 172.21.15.169:0/3979110863 learned_addr learned my addr 172.21.15.169:0/3979110863 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:02.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.310+0000 7fdf46bc3640 1 --2- 172.21.15.169:0/3979110863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a175a1f0 0x5564a175c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.310+0000 7fdf463c2640 1 --2- 172.21.15.169:0/3979110863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a1754690 0x5564a1754ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.312+0000 7fdf46bc3640 1 --2- 172.21.15.169:0/3963703828 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdf2c0a2a00 0x7fdf2c0a4ec0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fdf4006b390 tx=0x7fdf3c009c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:02.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.312+0000 7fdf46bc3640 1 -- 172.21.15.169:0/3979110863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564a1755080 msgr2=0x5564a1759b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:02.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.312+0000 7fdf46bc3640 1 --2- 172.21.15.169:0/3979110863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564a1755080 0x5564a1759b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.312+0000 7fdf46bc3640 1 -- 172.21.15.169:0/3979110863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a1754690 msgr2=0x5564a1754ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:02.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.312+0000 7fdf46bc3640 1 --2- 172.21.15.169:0/3979110863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a1754690 0x5564a1754ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.312+0000 7fdf46bc3640 1 -- 172.21.15.169:0/3979110863 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564a13ce3f0 con 0x5564a175a1f0 2024-03-28T01:05:02.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.312+0000 7fdf46bc3640 1 --2- 172.21.15.169:0/3979110863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a175a1f0 0x5564a175c680 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00fad0 tx=0x7fdf3c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb612318df0dc437 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:02.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.314+0000 7fdf34bfe640 1 -- 172.21.15.169:0/3979110863 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c016e00 con 0x5564a175a1f0 2024-03-28T01:05:02.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.315+0000 7fdf34bfe640 1 -- 172.21.15.169:0/3979110863 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdf3c023070 con 0x5564a175a1f0 2024-03-28T01:05:02.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.316+0000 7fdf4b20c740 1 -- 172.21.15.169:0/3979110863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a175a1f0 msgr2=0x5564a175c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:02.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.316+0000 7fdf4b20c740 1 --2- 172.21.15.169:0/3979110863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a175a1f0 0x5564a175c680 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00fad0 tx=0x7fdf3c017000 comp rx=0 tx=0).stop 2024-03-28T01:05:02.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.321+0000 7fdf4b20c740 1 -- 172.21.15.169:0/3979110863 shutdown_connections 2024-03-28T01:05:02.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.321+0000 7fdf4b20c740 1 --2- 172.21.15.169:0/3979110863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a175a1f0 0x5564a175c680 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.321+0000 7fdf4b20c740 1 --2- 172.21.15.169:0/3979110863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564a1755080 0x5564a1759b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.321+0000 7fdf4b20c740 1 --2- 172.21.15.169:0/3979110863 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a1754690 0x5564a1754ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.321+0000 7fdf4b20c740 1 -- 172.21.15.169:0/3979110863 >> 172.21.15.169:0/3979110863 conn(0x5564a174fc00 msgr2=0x5564a1750000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:02.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.324+0000 7fdf4b20c740 1 -- 172.21.15.169:0/3979110863 shutdown_connections 2024-03-28T01:05:02.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.328+0000 7fdf4b20c740 1 -- 172.21.15.169:0/3979110863 wait complete. 2024-03-28T01:05:02.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.328+0000 7fdf4b20c740 1 -- start start 2024-03-28T01:05:02.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.328+0000 7fdf4b20c740 1 Processor -- start 2024-03-28T01:05:02.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.329+0000 7fdf4b20c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a1754690 0x5564a175e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.329+0000 7fdf4b20c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564a1755080 0x5564a175eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.329+0000 7fdf4b20c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a175a1f0 0x5564a1765f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.330+0000 7fdf4b20c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564a1676240 con 0x5564a175a1f0 2024-03-28T01:05:02.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.330+0000 7fdf4b20c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564a16b6460 con 0x5564a1755080 2024-03-28T01:05:02.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.330+0000 7fdf4b20c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564a175d1e0 con 0x5564a1754690 2024-03-28T01:05:02.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.330+0000 7fdf4b20c740 10 client.0 _ll_register_callbacks cb 0x5564a1675c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:02.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.330+0000 7fdf463c2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a1754690 0x5564a175e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.330+0000 7fdf463c2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a1754690 0x5564a175e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48912/0 (socket says 172.21.15.169:48912) 2024-03-28T01:05:02.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.330+0000 7fdf463c2640 1 -- 172.21.15.169:0/3001397169 learned_addr learned my addr 172.21.15.169:0/3001397169 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:02.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.330+0000 7fdf46bc3640 1 --2- 172.21.15.169:0/3001397169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a175a1f0 0x5564a1765f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.330+0000 7fdf463c2640 1 -- 172.21.15.169:0/3001397169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564a1755080 msgr2=0x5564a175eed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:05:02.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.330+0000 7fdf463c2640 1 --2- 172.21.15.169:0/3001397169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564a1755080 0x5564a175eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.330+0000 7fdf463c2640 1 -- 172.21.15.169:0/3001397169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a175a1f0 msgr2=0x5564a1765f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:02.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.330+0000 7fdf463c2640 1 --2- 172.21.15.169:0/3001397169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a175a1f0 0x5564a1765f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.330+0000 7fdf463c2640 1 -- 172.21.15.169:0/3001397169 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5564a14987d0 con 0x5564a1754690 2024-03-28T01:05:02.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.332+0000 7fdf463c2640 1 --2- 172.21.15.169:0/3001397169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a1754690 0x5564a175e990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdf38010a80 tx=0x7fdf38010f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25deb69f69d512c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:02.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.333+0000 7fdf29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:05:02.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.334+0000 7fdf29ffb640 1 -- 172.21.15.169:0/3001397169 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3800ede0 con 0x5564a1754690 2024-03-28T01:05:02.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.334+0000 7fdf4b20c740 1 -- 172.21.15.169:0/3001397169 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdf3c023070 con 0x5564a1754690 2024-03-28T01:05:02.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.334+0000 7fdf29ffb640 1 -- 172.21.15.169:0/3001397169 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fdf38004b20 con 0x5564a1754690 2024-03-28T01:05:02.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.338+0000 7fdf29ffb640 1 -- 172.21.15.169:0/3001397169 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdf38004d20 con 0x5564a1754690 2024-03-28T01:05:02.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.339+0000 7fdf4b20c740 10 client.26914 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:02.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.339+0000 7fdf4b20c740 1 -- 172.21.15.169:0/3001397169 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5564a1769350 con 0x5564a1754690 2024-03-28T01:05:02.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.345+0000 7fdf29ffb640 1 -- 172.21.15.169:0/3001397169 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdf3c023070 con 0x5564a1754690 2024-03-28T01:05:02.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.345+0000 7fdf29ffb640 1 -- 172.21.15.169:0/3001397169 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fdf3800b040 con 0x5564a1754690 2024-03-28T01:05:02.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.345+0000 7fdf4b20c740 10 client.26914 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:02.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.345+0000 7fdf4b20c740 10 client.26914 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:05:02.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.345+0000 7fdf4b20c740 20 client.26914 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:02.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.345+0000 7fdf4b20c740 1 --2- 172.21.15.169:0/3001397169 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5564a176ae60 0x5564a176d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.345+0000 7fdf4b20c740 4 client.26914 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:05:02.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.345+0000 7fdf4b20c740 1 -- 172.21.15.169:0/3001397169 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5564a16c7900 con 0x5564a176ae60 2024-03-28T01:05:02.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.348+0000 7fdf45bc1640 1 --2- 172.21.15.169:0/3001397169 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5564a176ae60 0x5564a176d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.356+0000 7fdf45bc1640 1 --2- 172.21.15.169:0/3001397169 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5564a176ae60 0x5564a176d2c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5690853179f03433 server_cookie=8551b7cadaae1454 in_seq=0 out_seq=0 2024-03-28T01:05:02.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.357+0000 7fdf29ffb640 10 client.26914 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:05:02.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.357+0000 7fdf29ffb640 1 -- 172.21.15.169:0/3001397169 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5564a16c7900 con 0x5564a176ae60 2024-03-28T01:05:02.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.357+0000 7fdf29ffb640 10 client.26914 handle_command_reply: tid=0 2024-03-28T01:05:02.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.357+0000 7fdf4b20c740 1 client.26914 shutdown 2024-03-28T01:05:02.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.360+0000 7fdf4b20c740 1 -- 172.21.15.169:0/3001397169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a1754690 msgr2=0x5564a175e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:02.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.360+0000 7fdf4b20c740 1 --2- 172.21.15.169:0/3001397169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a1754690 0x5564a175e990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdf38010a80 tx=0x7fdf38010f40 comp rx=0 tx=0).stop 2024-03-28T01:05:02.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.361+0000 7fdf4b20c740 1 -- 172.21.15.169:0/3001397169 shutdown_connections 2024-03-28T01:05:02.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.361+0000 7fdf4b20c740 1 --2- 172.21.15.169:0/3001397169 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5564a176ae60 0x5564a176d2c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.361+0000 7fdf4b20c740 1 --2- 172.21.15.169:0/3001397169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a175a1f0 0x5564a1765f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.361+0000 7fdf4b20c740 1 --2- 172.21.15.169:0/3001397169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564a1755080 0x5564a175eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.361+0000 7fdf4b20c740 1 --2- 172.21.15.169:0/3001397169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a1754690 0x5564a175e990 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.361+0000 7fdf4b20c740 1 -- 172.21.15.169:0/3001397169 >> 172.21.15.169:0/3001397169 conn(0x5564a174fc00 msgr2=0x5564a174ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:02.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.362+0000 7fdf29ffb640 0 client.26914 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:05:02.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.362+0000 7fdf4b20c740 1 -- 172.21.15.169:0/3001397169 shutdown_connections 2024-03-28T01:05:02.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.362+0000 7fdf4b20c740 1 -- 172.21.15.169:0/3001397169 wait complete. 2024-03-28T01:05:02.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.362+0000 7fdf4b20c740 20 client.26914 trim_cache size 0 max 16384 2024-03-28T01:05:02.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.395+0000 7fdf4b20c740 1 Processor -- start 2024-03-28T01:05:02.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.396+0000 7fdf4b20c740 1 -- start start 2024-03-28T01:05:02.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.396+0000 7fdf4b20c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564a1754690 0x5564a1756440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.396+0000 7fdf4b20c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a1755080 0x5564a1756980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.396+0000 7fdf4b20c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a175a1f0 0x5564a1765ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.396+0000 7fdf4b20c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf24012120 con 0x5564a1755080 2024-03-28T01:05:02.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.396+0000 7fdf4b20c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2400fc20 con 0x5564a1754690 2024-03-28T01:05:02.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.396+0000 7fdf4b20c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2400dcc0 con 0x5564a175a1f0 2024-03-28T01:05:02.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.398+0000 7fdf463c2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564a1754690 0x5564a1756440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.398+0000 7fdf463c2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564a1754690 0x5564a1756440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:56710/0 (socket says 172.21.15.169:56710) 2024-03-28T01:05:02.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.398+0000 7fdf463c2640 1 -- 172.21.15.169:0/728063709 learned_addr learned my addr 172.21.15.169:0/728063709 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:02.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.398+0000 7fdf45bc1640 1 --2- 172.21.15.169:0/728063709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a1755080 0x5564a1756980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.398+0000 7fdf46bc3640 1 --2- 172.21.15.169:0/728063709 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a175a1f0 0x5564a1765ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.399+0000 7fdf463c2640 1 -- 172.21.15.169:0/728063709 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a175a1f0 msgr2=0x5564a1765ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:02.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.399+0000 7fdf463c2640 1 --2- 172.21.15.169:0/728063709 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a175a1f0 0x5564a1765ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.399+0000 7fdf463c2640 1 -- 172.21.15.169:0/728063709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a1755080 msgr2=0x5564a1756980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:02.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.399+0000 7fdf463c2640 1 --2- 172.21.15.169:0/728063709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a1755080 0x5564a1756980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.399+0000 7fdf463c2640 1 -- 172.21.15.169:0/728063709 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564a174b670 con 0x5564a1754690 2024-03-28T01:05:02.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.401+0000 7fdf463c2640 1 --2- 172.21.15.169:0/728063709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564a1754690 0x5564a1756440 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fdf3800e760 tx=0x7fdf38009890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b91197d37ff01e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:02.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.402+0000 7fdf2bfff640 1 -- 172.21.15.169:0/728063709 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3801a9b0 con 0x5564a1754690 2024-03-28T01:05:02.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.402+0000 7fdf2bfff640 1 -- 172.21.15.169:0/728063709 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdf38003e50 con 0x5564a1754690 2024-03-28T01:05:02.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.403+0000 7fdf4b20c740 1 -- 172.21.15.169:0/728063709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564a1754690 msgr2=0x5564a1756440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:02.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.403+0000 7fdf4b20c740 1 --2- 172.21.15.169:0/728063709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564a1754690 0x5564a1756440 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fdf3800e760 tx=0x7fdf38009890 comp rx=0 tx=0).stop 2024-03-28T01:05:02.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.407+0000 7fdf4b20c740 1 -- 172.21.15.169:0/728063709 shutdown_connections 2024-03-28T01:05:02.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.407+0000 7fdf4b20c740 1 --2- 172.21.15.169:0/728063709 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a175a1f0 0x5564a1765ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.407+0000 7fdf4b20c740 1 --2- 172.21.15.169:0/728063709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a1755080 0x5564a1756980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.407+0000 7fdf4b20c740 1 --2- 172.21.15.169:0/728063709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564a1754690 0x5564a1756440 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.407+0000 7fdf4b20c740 1 -- 172.21.15.169:0/728063709 >> 172.21.15.169:0/728063709 conn(0x5564a174fc00 msgr2=0x5564a176ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:02.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.409+0000 7fdf4b20c740 1 -- 172.21.15.169:0/728063709 shutdown_connections 2024-03-28T01:05:02.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.411+0000 7fdf4b20c740 1 -- 172.21.15.169:0/728063709 wait complete. 2024-03-28T01:05:02.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.412+0000 7fdf4b20c740 1 -- start start 2024-03-28T01:05:02.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.412+0000 7fdf4b20c740 1 Processor -- start 2024-03-28T01:05:02.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.412+0000 7fdf4b20c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a1754690 0x5564a17633f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.412+0000 7fdf4b20c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564a1755080 0x5564a1763930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.412+0000 7fdf4b20c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a175a1f0 0x5564a1764940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.412+0000 7fdf4b20c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2400e2c0 con 0x5564a175a1f0 2024-03-28T01:05:02.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.412+0000 7fdf4b20c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2400f5c0 con 0x5564a1755080 2024-03-28T01:05:02.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.412+0000 7fdf4b20c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2400e900 con 0x5564a1754690 2024-03-28T01:05:02.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.412+0000 7fdf4b20c740 10 client.0 _ll_register_callbacks cb 0x7fdf38010640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:02.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.413+0000 7fdf46bc3640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a175a1f0 0x5564a1764940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.414+0000 7fdf46bc3640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a175a1f0 0x5564a1764940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34234/0 (socket says 172.21.15.169:34234) 2024-03-28T01:05:02.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.414+0000 7fdf46bc3640 1 -- 172.21.15.169:0/1946146909 learned_addr learned my addr 172.21.15.169:0/1946146909 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:02.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.414+0000 7fdf463c2640 1 --2- 172.21.15.169:0/1946146909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a1754690 0x5564a17633f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.414+0000 7fdf46bc3640 1 -- 172.21.15.169:0/1946146909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a1754690 msgr2=0x5564a17633f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:02.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.414+0000 7fdf46bc3640 1 --2- 172.21.15.169:0/1946146909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a1754690 0x5564a17633f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.414+0000 7fdf46bc3640 1 -- 172.21.15.169:0/1946146909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564a1755080 msgr2=0x5564a1763930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:05:02.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.414+0000 7fdf46bc3640 1 --2- 172.21.15.169:0/1946146909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564a1755080 0x5564a1763930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.414+0000 7fdf46bc3640 1 -- 172.21.15.169:0/1946146909 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5564a1764f30 con 0x5564a175a1f0 2024-03-28T01:05:02.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.414+0000 7fdf463c2640 1 --2- 172.21.15.169:0/1946146909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a1754690 0x5564a17633f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:05:02.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.415+0000 7fdf46bc3640 1 --2- 172.21.15.169:0/1946146909 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a175a1f0 0x5564a1764940 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c022000 tx=0x7fdf3c0092f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df95254873345649 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:02.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.415+0000 7fdf29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:05:02.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.415+0000 7fdf29ffb640 1 -- 172.21.15.169:0/1946146909 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c004030 con 0x5564a175a1f0 2024-03-28T01:05:02.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.415+0000 7fdf4b20c740 1 -- 172.21.15.169:0/1946146909 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdf38003e50 con 0x5564a175a1f0 2024-03-28T01:05:02.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.415+0000 7fdf29ffb640 1 -- 172.21.15.169:0/1946146909 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fdf3c0041d0 con 0x5564a175a1f0 2024-03-28T01:05:02.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.421+0000 7fdf29ffb640 1 -- 172.21.15.169:0/1946146909 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdf3c0043d0 con 0x5564a175a1f0 2024-03-28T01:05:02.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.423+0000 7fdf4b20c740 10 client.17124 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:02.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.423+0000 7fdf4b20c740 1 -- 172.21.15.169:0/1946146909 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5564a17662a0 con 0x5564a175a1f0 2024-03-28T01:05:02.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.426+0000 7fdf29ffb640 1 -- 172.21.15.169:0/1946146909 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdf38003e50 con 0x5564a175a1f0 2024-03-28T01:05:02.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.426+0000 7fdf29ffb640 1 -- 172.21.15.169:0/1946146909 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fdf3c01e560 con 0x5564a175a1f0 2024-03-28T01:05:02.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.427+0000 7fdf4b20c740 10 client.17124 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:02.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.427+0000 7fdf4b20c740 10 client.17124 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:05:02.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.427+0000 7fdf4b20c740 20 client.17124 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:02.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.427+0000 7fdf4b20c740 1 --2- 172.21.15.169:0/1946146909 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5564a17c6c60 0x5564a17c9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.427+0000 7fdf4b20c740 4 client.17124 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:05:02.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.427+0000 7fdf4b20c740 1 -- 172.21.15.169:0/1946146909 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5564a16c1c50 con 0x5564a17c6c60 2024-03-28T01:05:02.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.428+0000 7fdf463c2640 1 --2- 172.21.15.169:0/1946146909 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5564a17c6c60 0x5564a17c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.437+0000 7fdf463c2640 1 --2- 172.21.15.169:0/1946146909 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5564a17c6c60 0x5564a17c9080 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2be458fe898ca563 server_cookie=b2c1ce50589b9952 in_seq=0 out_seq=0 2024-03-28T01:05:02.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.438+0000 7fdf29ffb640 10 client.17124 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:05:02.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.442+0000 7fdf29ffb640 1 -- 172.21.15.169:0/1946146909 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5564a16c1c50 con 0x5564a17c6c60 2024-03-28T01:05:02.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.442+0000 7fdf29ffb640 10 client.17124 handle_command_reply: tid=0 2024-03-28T01:05:02.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.442+0000 7fdf4b20c740 1 client.17124 shutdown 2024-03-28T01:05:02.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.445+0000 7fdf4b20c740 1 -- 172.21.15.169:0/1946146909 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a175a1f0 msgr2=0x5564a1764940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:02.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.445+0000 7fdf4b20c740 1 --2- 172.21.15.169:0/1946146909 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a175a1f0 0x5564a1764940 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c022000 tx=0x7fdf3c0092f0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.454 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.451+0000 7fdf4b20c740 1 -- 172.21.15.169:0/1946146909 shutdown_connections 2024-03-28T01:05:02.454 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.451+0000 7fdf4b20c740 1 --2- 172.21.15.169:0/1946146909 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5564a17c6c60 0x5564a17c9080 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.454 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.451+0000 7fdf4b20c740 1 --2- 172.21.15.169:0/1946146909 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5564a175a1f0 0x5564a1764940 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.454 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.451+0000 7fdf4b20c740 1 --2- 172.21.15.169:0/1946146909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564a1755080 0x5564a1763930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.454 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.451+0000 7fdf4b20c740 1 --2- 172.21.15.169:0/1946146909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5564a1754690 0x5564a17633f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.454 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.451+0000 7fdf4b20c740 1 -- 172.21.15.169:0/1946146909 >> 172.21.15.169:0/1946146909 conn(0x5564a174fc00 msgr2=0x5564a1756440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:02.454 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.452+0000 7fdf29ffb640 0 client.17124 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:05:02.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.455+0000 7fdf4b20c740 1 -- 172.21.15.169:0/1946146909 shutdown_connections 2024-03-28T01:05:02.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.458+0000 7fdf4b20c740 1 -- 172.21.15.169:0/1946146909 wait complete. 2024-03-28T01:05:02.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.459+0000 7fdf4b20c740 20 client.17124 trim_cache size 0 max 16384 2024-03-28T01:05:02.469 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:05:02.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.467+0000 7fdf48626640 1 -- 172.21.15.169:0/3963703828 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdf2c0a2a00 msgr2=0x7fdf2c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:02.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.467+0000 7fdf48626640 1 --2- 172.21.15.169:0/3963703828 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdf2c0a2a00 0x7fdf2c0a4ec0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fdf4006b390 tx=0x7fdf3c009c20 comp rx=0 tx=0).stop 2024-03-28T01:05:02.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.467+0000 7fdf48626640 1 -- 172.21.15.169:0/3963703828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf4010b010 msgr2=0x7fdf40119f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:02.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.467+0000 7fdf48626640 1 --2- 172.21.15.169:0/3963703828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf4010b010 0x7fdf40119f40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fdf300349e0 tx=0x7fdf300043b0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.468+0000 7fdf48626640 1 -- 172.21.15.169:0/3963703828 shutdown_connections 2024-03-28T01:05:02.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.468+0000 7fdf48626640 1 --2- 172.21.15.169:0/3963703828 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdf2c0a2a00 0x7fdf2c0a4ec0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.468+0000 7fdf48626640 1 --2- 172.21.15.169:0/3963703828 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf4010d9c0 0x7fdf40118090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.468+0000 7fdf48626640 1 --2- 172.21.15.169:0/3963703828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf4010b010 0x7fdf40119f40 secure :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fdf300349e0 tx=0x7fdf300043b0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.468+0000 7fdf48626640 1 --2- 172.21.15.169:0/3963703828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf401086a0 0x7fdf40119a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.468+0000 7fdf48626640 1 -- 172.21.15.169:0/3963703828 >> 172.21.15.169:0/3963703828 conn(0x7fdf4006e500 msgr2=0x7fdf4010b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:02.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.470+0000 7fdf48626640 1 -- 172.21.15.169:0/3963703828 shutdown_connections 2024-03-28T01:05:02.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.470+0000 7fdf48626640 1 -- 172.21.15.169:0/3963703828 wait complete. 2024-03-28T01:05:02.510 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:05:02.510 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T01:05:02.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.729+0000 7f23c84b2640 1 -- 172.21.15.169:0/4077192832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c010d9c0 msgr2=0x7f23c010de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:02.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.729+0000 7f23c84b2640 1 --2- 172.21.15.169:0/4077192832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c010d9c0 0x7f23c010de40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f23b0009900 tx=0x7f23b002bc80 comp rx=0 tx=0).stop 2024-03-28T01:05:02.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.729+0000 7f23c84b2640 1 -- 172.21.15.169:0/4077192832 shutdown_connections 2024-03-28T01:05:02.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.729+0000 7f23c84b2640 1 --2- 172.21.15.169:0/4077192832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c010d9c0 0x7f23c010de40 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.729+0000 7f23c84b2640 1 --2- 172.21.15.169:0/4077192832 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23c010b010 0x7f23c010d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.729+0000 7f23c84b2640 1 --2- 172.21.15.169:0/4077192832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23c01086a0 0x7f23c010aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.729+0000 7f23c84b2640 1 -- 172.21.15.169:0/4077192832 >> 172.21.15.169:0/4077192832 conn(0x7f23c006e500 msgr2=0x7f23c0070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:02.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.730+0000 7f23c84b2640 1 -- 172.21.15.169:0/4077192832 shutdown_connections 2024-03-28T01:05:02.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.730+0000 7f23c84b2640 1 -- 172.21.15.169:0/4077192832 wait complete. 2024-03-28T01:05:02.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.731+0000 7f23c84b2640 1 Processor -- start 2024-03-28T01:05:02.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.731+0000 7f23c84b2640 1 -- start start 2024-03-28T01:05:02.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.731+0000 7f23c84b2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23c01086a0 0x7f23c0078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.731+0000 7f23c84b2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23c010b010 0x7f23c0078db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.731+0000 7f23c84b2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c010d9c0 0x7f23c007fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.731+0000 7f23c84b2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c011c260 con 0x7f23c01086a0 2024-03-28T01:05:02.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.731+0000 7f23c84b2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c011c0e0 con 0x7f23c010b010 2024-03-28T01:05:02.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.731+0000 7f23c84b2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c011c3e0 con 0x7f23c010d9c0 2024-03-28T01:05:02.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.731+0000 7f23c624e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23c01086a0 0x7f23c0078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.731+0000 7f23c624e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23c01086a0 0x7f23c0078870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34254/0 (socket says 172.21.15.169:34254) 2024-03-28T01:05:02.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.732+0000 7f23c624e640 1 -- 172.21.15.169:0/3118615899 learned_addr learned my addr 172.21.15.169:0/3118615899 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:02.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.732+0000 7f23c5a4d640 1 --2- 172.21.15.169:0/3118615899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23c010b010 0x7f23c0078db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.732+0000 7f23c624e640 1 -- 172.21.15.169:0/3118615899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c010d9c0 msgr2=0x7f23c007fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:02.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.732+0000 7f23c624e640 1 --2- 172.21.15.169:0/3118615899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c010d9c0 0x7f23c007fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.732+0000 7f23c624e640 1 -- 172.21.15.169:0/3118615899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23c010b010 msgr2=0x7f23c0078db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:02.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.732+0000 7f23c624e640 1 --2- 172.21.15.169:0/3118615899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23c010b010 0x7f23c0078db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.732+0000 7f23c624e640 1 -- 172.21.15.169:0/3118615899 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c0080440 con 0x7f23c01086a0 2024-03-28T01:05:02.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.732+0000 7f23c624e640 1 --2- 172.21.15.169:0/3118615899 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23c01086a0 0x7f23c0078870 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f23b800ba50 tx=0x7f23b800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:02.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.735+0000 7f23b77fe640 1 -- 172.21.15.169:0/3118615899 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b8019070 con 0x7f23c01086a0 2024-03-28T01:05:02.738 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.735+0000 7f23b77fe640 1 -- 172.21.15.169:0/3118615899 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23b800cc10 con 0x7f23c01086a0 2024-03-28T01:05:02.738 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.736+0000 7f23c84b2640 1 -- 172.21.15.169:0/3118615899 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c0080730 con 0x7f23c01086a0 2024-03-28T01:05:02.738 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.736+0000 7f23b77fe640 1 -- 172.21.15.169:0/3118615899 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b80076d0 con 0x7f23c01086a0 2024-03-28T01:05:02.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.736+0000 7f23c84b2640 1 -- 172.21.15.169:0/3118615899 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c0080c80 con 0x7f23c01086a0 2024-03-28T01:05:02.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.740+0000 7f23cb180740 1 Processor -- start 2024-03-28T01:05:02.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.741+0000 7f23b77fe640 1 -- 172.21.15.169:0/3118615899 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f23b801c070 con 0x7f23c01086a0 2024-03-28T01:05:02.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.742+0000 7f23b77fe640 1 --2- 172.21.15.169:0/3118615899 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f23ac0a2a10 0x7f23ac0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.742+0000 7f23cb180740 1 -- start start 2024-03-28T01:05:02.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.742+0000 7f23b77fe640 1 -- 172.21.15.169:0/3118615899 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f23b80ce500 con 0x7f23c01086a0 2024-03-28T01:05:02.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.744+0000 7f23cb180740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fac690 0x55aee5facab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.744+0000 7f23c6a4f640 1 --2- 172.21.15.169:0/3118615899 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f23ac0a2a10 0x7f23ac0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.744+0000 7f23cb180740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fad080 0x55aee5fb1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.744+0000 7f23cb180740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aee5fb21f0 0x55aee5fb4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.744+0000 7f23cb180740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aee5f6f610 con 0x55aee5fb21f0 2024-03-28T01:05:02.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.744+0000 7f23cb180740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aee5f123c0 con 0x55aee5fac690 2024-03-28T01:05:02.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.744+0000 7f23cb180740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aee5ebad60 con 0x55aee5fad080 2024-03-28T01:05:02.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.744+0000 7f23c624e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fac690 0x55aee5facab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.744+0000 7f23c624e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fac690 0x55aee5facab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:56738/0 (socket says 172.21.15.169:56738) 2024-03-28T01:05:02.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.744+0000 7f23c624e640 1 -- 172.21.15.169:0/2898810707 learned_addr learned my addr 172.21.15.169:0/2898810707 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:02.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.744+0000 7f23c5a4d640 1 --2- 172.21.15.169:0/2898810707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fad080 0x55aee5fb1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.744+0000 7f23c624e640 1 -- 172.21.15.169:0/2898810707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fad080 msgr2=0x55aee5fb1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:02.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.744+0000 7f23c624e640 1 --2- 172.21.15.169:0/2898810707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fad080 0x55aee5fb1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.744+0000 7f23c624e640 1 -- 172.21.15.169:0/2898810707 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aee5fb21f0 msgr2=0x55aee5fb4680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:05:02.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.744+0000 7f23c624e640 1 --2- 172.21.15.169:0/2898810707 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aee5fb21f0 0x55aee5fb4680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.745+0000 7f23c624e640 1 -- 172.21.15.169:0/2898810707 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aee5c263f0 con 0x55aee5fac690 2024-03-28T01:05:02.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.745+0000 7f23c5a4d640 1 --2- 172.21.15.169:0/2898810707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fad080 0x55aee5fb1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:05:02.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.745+0000 7f23c624e640 1 --2- 172.21.15.169:0/2898810707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fac690 0x55aee5facab0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f23b8097470 tx=0x7f23b8096e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=982f673fbf8c0026 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:02.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.747+0000 7f23b4bfe640 1 -- 172.21.15.169:0/2898810707 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b80a3240 con 0x55aee5fac690 2024-03-28T01:05:02.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.747+0000 7f23b4bfe640 1 -- 172.21.15.169:0/2898810707 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23b8098da0 con 0x55aee5fac690 2024-03-28T01:05:02.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.747+0000 7f23b4bfe640 1 -- 172.21.15.169:0/2898810707 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b809ad00 con 0x55aee5fac690 2024-03-28T01:05:02.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.749+0000 7f23c6a4f640 1 --2- 172.21.15.169:0/3118615899 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f23ac0a2a10 0x7f23ac0a4ed0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f23b0005ec0 tx=0x7f23b002c890 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:02.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.749+0000 7f23cb180740 1 -- 172.21.15.169:0/2898810707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fac690 msgr2=0x55aee5facab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:02.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.749+0000 7f23cb180740 1 --2- 172.21.15.169:0/2898810707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fac690 0x55aee5facab0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f23b8097470 tx=0x7f23b8096e70 comp rx=0 tx=0).stop 2024-03-28T01:05:02.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.749+0000 7f23cb180740 1 -- 172.21.15.169:0/2898810707 shutdown_connections 2024-03-28T01:05:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.749+0000 7f23cb180740 1 --2- 172.21.15.169:0/2898810707 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aee5fb21f0 0x55aee5fb4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.749+0000 7f23cb180740 1 --2- 172.21.15.169:0/2898810707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fad080 0x55aee5fb1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.749+0000 7f23cb180740 1 --2- 172.21.15.169:0/2898810707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fac690 0x55aee5facab0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.749+0000 7f23cb180740 1 -- 172.21.15.169:0/2898810707 >> 172.21.15.169:0/2898810707 conn(0x55aee5fa7c00 msgr2=0x55aee5fa8000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.749+0000 7f23cb180740 1 -- 172.21.15.169:0/2898810707 shutdown_connections 2024-03-28T01:05:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.749+0000 7f23cb180740 1 -- 172.21.15.169:0/2898810707 wait complete. 2024-03-28T01:05:02.755 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.750+0000 7f23cb180740 1 -- start start 2024-03-28T01:05:02.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.750+0000 7f23cb180740 1 Processor -- start 2024-03-28T01:05:02.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.750+0000 7f23cb180740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fac690 0x55aee5fb6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.750+0000 7f23cb180740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fad080 0x55aee5fb6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.750+0000 7f23cb180740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aee5fb21f0 0x55aee5fbde90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.751+0000 7f23cb180740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aee5ece240 con 0x55aee5fb21f0 2024-03-28T01:05:02.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.751+0000 7f23cb180740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aee5f0e460 con 0x55aee5fad080 2024-03-28T01:05:02.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.751+0000 7f23cb180740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aee5fb51a0 con 0x55aee5fac690 2024-03-28T01:05:02.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.751+0000 7f23c6a4f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aee5fb21f0 0x55aee5fbde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.751+0000 7f23c6a4f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aee5fb21f0 0x55aee5fbde90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34260/0 (socket says 172.21.15.169:34260) 2024-03-28T01:05:02.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.751+0000 7f23c6a4f640 1 -- 172.21.15.169:0/1431668410 learned_addr learned my addr 172.21.15.169:0/1431668410 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:02.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.751+0000 7f23cb180740 10 client.0 _ll_register_callbacks cb 0x55aee5ecdc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:02.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.751+0000 7f23c6a4f640 1 -- 172.21.15.169:0/1431668410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fac690 msgr2=0x55aee5fb6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:02.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.751+0000 7f23c6a4f640 1 --2- 172.21.15.169:0/1431668410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fac690 0x55aee5fb6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.751+0000 7f23c6a4f640 1 -- 172.21.15.169:0/1431668410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fad080 msgr2=0x55aee5fb6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:02.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.751+0000 7f23c6a4f640 1 --2- 172.21.15.169:0/1431668410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fad080 0x55aee5fb6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.751+0000 7f23c6a4f640 1 -- 172.21.15.169:0/1431668410 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aee5cf07d0 con 0x55aee5fb21f0 2024-03-28T01:05:02.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.751+0000 7f23c6a4f640 1 --2- 172.21.15.169:0/1431668410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aee5fb21f0 0x55aee5fbde90 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f23b0035000 tx=0x7f23b0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60b9df8c93212f93 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:02.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.752+0000 7f23a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:05:02.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.752+0000 7f23a9ffb640 1 -- 172.21.15.169:0/1431668410 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b0046040 con 0x55aee5fb21f0 2024-03-28T01:05:02.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.752+0000 7f23cb180740 1 -- 172.21.15.169:0/1431668410 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f23b8098da0 con 0x55aee5fb21f0 2024-03-28T01:05:02.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.752+0000 7f23a9ffb640 1 -- 172.21.15.169:0/1431668410 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f23b00046c0 con 0x55aee5fb21f0 2024-03-28T01:05:02.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.752+0000 7f23a9ffb640 1 -- 172.21.15.169:0/1431668410 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23b0058050 con 0x55aee5fb21f0 2024-03-28T01:05:02.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.752+0000 7f23cb180740 10 client.17130 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:02.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.752+0000 7f23cb180740 1 -- 172.21.15.169:0/1431668410 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55aee5fc1230 con 0x55aee5fb21f0 2024-03-28T01:05:02.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.753+0000 7f23a9ffb640 1 -- 172.21.15.169:0/1431668410 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23b8098da0 con 0x55aee5fb21f0 2024-03-28T01:05:02.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.753+0000 7f23a9ffb640 1 -- 172.21.15.169:0/1431668410 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f23b0003cd0 con 0x55aee5fb21f0 2024-03-28T01:05:02.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.754+0000 7f23cb180740 10 client.17130 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:02.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.754+0000 7f23cb180740 10 client.17130 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:02.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.754+0000 7f23cb180740 20 client.17130 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:02.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.754+0000 7f23cb180740 1 --2- 172.21.15.169:0/1431668410 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aee5fc2d90 0x55aee5fc5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.754+0000 7f23cb180740 4 client.17130 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:05:02.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.754+0000 7f23cb180740 1 -- 172.21.15.169:0/1431668410 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aee5f1f900 con 0x55aee5fc2d90 2024-03-28T01:05:02.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.755+0000 7f23c624e640 1 --2- 172.21.15.169:0/1431668410 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aee5fc2d90 0x55aee5fc5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.757+0000 7f23c624e640 1 --2- 172.21.15.169:0/1431668410 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aee5fc2d90 0x55aee5fc5230 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adce50ecd10f6afb server_cookie=969d834523b6a6fb in_seq=0 out_seq=0 2024-03-28T01:05:02.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.757+0000 7f23a9ffb640 10 client.17130 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:02.761 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.758+0000 7f23a9ffb640 1 -- 172.21.15.169:0/1431668410 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55aee5f1f900 con 0x55aee5fc2d90 2024-03-28T01:05:02.761 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.758+0000 7f23a9ffb640 10 client.17130 handle_command_reply: tid=0 2024-03-28T01:05:02.761 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.758+0000 7f23cb180740 1 client.17130 shutdown 2024-03-28T01:05:02.761 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.758+0000 7f23cb180740 1 -- 172.21.15.169:0/1431668410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aee5fb21f0 msgr2=0x55aee5fbde90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:02.762 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.758+0000 7f23cb180740 1 --2- 172.21.15.169:0/1431668410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aee5fb21f0 0x55aee5fbde90 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f23b0035000 tx=0x7f23b0009260 comp rx=0 tx=0).stop 2024-03-28T01:05:02.762 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.758+0000 7f23cb180740 1 -- 172.21.15.169:0/1431668410 shutdown_connections 2024-03-28T01:05:02.762 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.758+0000 7f23cb180740 1 --2- 172.21.15.169:0/1431668410 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aee5fc2d90 0x55aee5fc5230 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.762 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.758+0000 7f23cb180740 1 --2- 172.21.15.169:0/1431668410 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aee5fb21f0 0x55aee5fbde90 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.762 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.758+0000 7f23cb180740 1 --2- 172.21.15.169:0/1431668410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fad080 0x55aee5fb6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.762 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.758+0000 7f23cb180740 1 --2- 172.21.15.169:0/1431668410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fac690 0x55aee5fb6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.762 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.758+0000 7f23cb180740 1 -- 172.21.15.169:0/1431668410 >> 172.21.15.169:0/1431668410 conn(0x55aee5fa7c00 msgr2=0x55aee5fa6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:02.762 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.758+0000 7f23a9ffb640 0 client.17130 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:02.762 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.759+0000 7f23cb180740 1 -- 172.21.15.169:0/1431668410 shutdown_connections 2024-03-28T01:05:02.763 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.760+0000 7f23cb180740 1 -- 172.21.15.169:0/1431668410 wait complete. 2024-03-28T01:05:02.763 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.760+0000 7f23cb180740 20 client.17130 trim_cache size 0 max 16384 2024-03-28T01:05:02.795 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.790+0000 7f23cb180740 1 Processor -- start 2024-03-28T01:05:02.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.793+0000 7f23cb180740 1 -- start start 2024-03-28T01:05:02.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.793+0000 7f23cb180740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fac690 0x55aee5fb6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.793+0000 7f23cb180740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aee5fad080 0x55aee5fb6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.793+0000 7f23cb180740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fb21f0 0x55aee5fbde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.793+0000 7f23cb180740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c01d4ea0 con 0x55aee5fad080 2024-03-28T01:05:02.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.793+0000 7f23cb180740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c0116610 con 0x55aee5fb21f0 2024-03-28T01:05:02.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.793+0000 7f23cb180740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c01146b0 con 0x55aee5fac690 2024-03-28T01:05:02.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.793+0000 7f23c6a4f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fb21f0 0x55aee5fbde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.793+0000 7f23c6a4f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fb21f0 0x55aee5fbde50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:56748/0 (socket says 172.21.15.169:56748) 2024-03-28T01:05:02.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.793+0000 7f23c6a4f640 1 -- 172.21.15.169:0/1414005701 learned_addr learned my addr 172.21.15.169:0/1414005701 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:02.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.793+0000 7f23c5a4d640 1 --2- 172.21.15.169:0/1414005701 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aee5fad080 0x55aee5fb6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.793+0000 7f23c624e640 1 --2- 172.21.15.169:0/1414005701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fac690 0x55aee5fb6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.794+0000 7f23c5a4d640 1 -- 172.21.15.169:0/1414005701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fac690 msgr2=0x55aee5fb6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:02.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.794+0000 7f23c5a4d640 1 --2- 172.21.15.169:0/1414005701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fac690 0x55aee5fb6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.794+0000 7f23c5a4d640 1 -- 172.21.15.169:0/1414005701 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fb21f0 msgr2=0x55aee5fbde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:02.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.794+0000 7f23c5a4d640 1 --2- 172.21.15.169:0/1414005701 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fb21f0 0x55aee5fbde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.794+0000 7f23c5a4d640 1 -- 172.21.15.169:0/1414005701 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aee5fad910 con 0x55aee5fad080 2024-03-28T01:05:02.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.794+0000 7f23c6a4f640 1 --2- 172.21.15.169:0/1414005701 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fb21f0 0x55aee5fbde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:05:02.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.794+0000 7f23c624e640 1 --2- 172.21.15.169:0/1414005701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fac690 0x55aee5fb6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:05:02.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.795+0000 7f23c5a4d640 1 --2- 172.21.15.169:0/1414005701 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aee5fad080 0x55aee5fb6e50 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x55aee5fb7df0 tx=0x7f23bc009c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ec7bf865cbff413 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:02.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.796+0000 7f23abfff640 1 -- 172.21.15.169:0/1414005701 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc017020 con 0x55aee5fad080 2024-03-28T01:05:02.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.796+0000 7f23abfff640 1 -- 172.21.15.169:0/1414005701 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23bc00ee00 con 0x55aee5fad080 2024-03-28T01:05:02.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.796+0000 7f23abfff640 1 -- 172.21.15.169:0/1414005701 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc017020 con 0x55aee5fad080 2024-03-28T01:05:02.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.797+0000 7f23cb180740 1 -- 172.21.15.169:0/1414005701 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aee5fad080 msgr2=0x55aee5fb6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:02.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.798+0000 7f23cb180740 1 --2- 172.21.15.169:0/1414005701 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aee5fad080 0x55aee5fb6e50 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x55aee5fb7df0 tx=0x7f23bc009c10 comp rx=0 tx=0).stop 2024-03-28T01:05:02.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:02 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2278605551' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:05:02.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.799+0000 7f23cb180740 1 -- 172.21.15.169:0/1414005701 shutdown_connections 2024-03-28T01:05:02.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.799+0000 7f23cb180740 1 --2- 172.21.15.169:0/1414005701 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fb21f0 0x55aee5fbde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.799+0000 7f23cb180740 1 --2- 172.21.15.169:0/1414005701 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aee5fad080 0x55aee5fb6e50 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.799+0000 7f23cb180740 1 --2- 172.21.15.169:0/1414005701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fac690 0x55aee5fb6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.799+0000 7f23cb180740 1 -- 172.21.15.169:0/1414005701 >> 172.21.15.169:0/1414005701 conn(0x55aee5fa7c00 msgr2=0x55aee5fa7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:02.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.801+0000 7f23cb180740 1 -- 172.21.15.169:0/1414005701 shutdown_connections 2024-03-28T01:05:02.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.805+0000 7f23cb180740 1 -- 172.21.15.169:0/1414005701 wait complete. 2024-03-28T01:05:02.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.806+0000 7f23cb180740 1 -- start start 2024-03-28T01:05:02.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.806+0000 7f23cb180740 1 Processor -- start 2024-03-28T01:05:02.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.809+0000 7f23cb180740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fac690 0x55aee5fbb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.809+0000 7f23cb180740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aee5fad080 0x55aee5fbb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.809+0000 7f23c624e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fac690 0x55aee5fbb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.809+0000 7f23c624e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fac690 0x55aee5fbb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48976/0 (socket says 172.21.15.169:48976) 2024-03-28T01:05:02.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.809+0000 7f23c624e640 1 -- 172.21.15.169:0/2052219106 learned_addr learned my addr 172.21.15.169:0/2052219106 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:02.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.809+0000 7f23cb180740 1 --2- 172.21.15.169:0/2052219106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fb21f0 0x55aee5fbc770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.809+0000 7f23cb180740 1 -- 172.21.15.169:0/2052219106 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c0114cb0 con 0x55aee5fad080 2024-03-28T01:05:02.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.810+0000 7f23cb180740 1 -- 172.21.15.169:0/2052219106 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c0115fb0 con 0x55aee5fb21f0 2024-03-28T01:05:02.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.810+0000 7f23c6a4f640 1 --2- 172.21.15.169:0/2052219106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fb21f0 0x55aee5fbc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.810+0000 7f23cb180740 1 -- 172.21.15.169:0/2052219106 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c01152f0 con 0x55aee5fac690 2024-03-28T01:05:02.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.810+0000 7f23cb180740 10 client.0 _ll_register_callbacks cb 0x7f23b0004400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:02.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.810+0000 7f23c6a4f640 1 -- 172.21.15.169:0/2052219106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fac690 msgr2=0x55aee5fbb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:02.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.810+0000 7f23c6a4f640 1 --2- 172.21.15.169:0/2052219106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fac690 0x55aee5fbb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.810+0000 7f23c6a4f640 1 -- 172.21.15.169:0/2052219106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aee5fad080 msgr2=0x55aee5fbb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:02.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.810+0000 7f23c6a4f640 1 --2- 172.21.15.169:0/2052219106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aee5fad080 0x55aee5fbb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.810+0000 7f23c6a4f640 1 -- 172.21.15.169:0/2052219106 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aee5fbcd60 con 0x55aee5fb21f0 2024-03-28T01:05:02.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.811+0000 7f23c6a4f640 1 --2- 172.21.15.169:0/2052219106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fb21f0 0x55aee5fbc770 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f23b0040500 tx=0x7f23b0040d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57bdfbe794d6fd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:02.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.811+0000 7f23cb180740 1 -- 172.21.15.169:0/2052219106 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f23bc00ee00 con 0x55aee5fb21f0 2024-03-28T01:05:02.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.812+0000 7f23a9ffb640 10 client.26987 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:05:02.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.812+0000 7f23a9ffb640 1 -- 172.21.15.169:0/2052219106 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b0046040 con 0x55aee5fb21f0 2024-03-28T01:05:02.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.812+0000 7f23a9ffb640 1 -- 172.21.15.169:0/2052219106 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f23b00408b0 con 0x55aee5fb21f0 2024-03-28T01:05:02.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.812+0000 7f23a9ffb640 1 -- 172.21.15.169:0/2052219106 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23b002cca0 con 0x55aee5fb21f0 2024-03-28T01:05:02.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.812+0000 7f23cb180740 10 client.26987 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:02.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.812+0000 7f23cb180740 1 -- 172.21.15.169:0/2052219106 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x55aee5fbdfb0 con 0x55aee5fb21f0 2024-03-28T01:05:02.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.813+0000 7f23a9ffb640 1 -- 172.21.15.169:0/2052219106 <== mon.2 v2:172.21.15.174:3300/0 4 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f23b00046c0 con 0x55aee5fb21f0 2024-03-28T01:05:02.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.813+0000 7f23cb180740 10 client.26987 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:02.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.813+0000 7f23cb180740 10 client.26987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:02.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.813+0000 7f23cb180740 20 client.26987 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:02.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.813+0000 7f23cb180740 1 --2- 172.21.15.169:0/2052219106 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aee601eb60 0x55aee6020fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:02.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.813+0000 7f23cb180740 4 client.26987 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:05:02.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.813+0000 7f23cb180740 1 -- 172.21.15.169:0/2052219106 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f23bc00ee00 con 0x55aee601eb60 2024-03-28T01:05:02.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.814+0000 7f23c624e640 1 --2- 172.21.15.169:0/2052219106 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aee601eb60 0x55aee6020fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:02.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.819+0000 7f23c624e640 1 --2- 172.21.15.169:0/2052219106 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aee601eb60 0x55aee6020fc0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1eaf424f0d983a9c server_cookie=7a5ac17ccf40aae2 in_seq=0 out_seq=0 2024-03-28T01:05:02.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.820+0000 7f23a9ffb640 10 client.26987 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:02.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.820+0000 7f23a9ffb640 1 -- 172.21.15.169:0/2052219106 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f23bc00ee00 con 0x55aee601eb60 2024-03-28T01:05:02.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.820+0000 7f23a9ffb640 10 client.26987 handle_command_reply: tid=0 2024-03-28T01:05:02.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.821+0000 7f23cb180740 1 client.26987 shutdown 2024-03-28T01:05:02.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.825+0000 7f23cb180740 1 -- 172.21.15.169:0/2052219106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fb21f0 msgr2=0x55aee5fbc770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:02.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.825+0000 7f23cb180740 1 --2- 172.21.15.169:0/2052219106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fb21f0 0x55aee5fbc770 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f23b0040500 tx=0x7f23b0040d00 comp rx=0 tx=0).stop 2024-03-28T01:05:02.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.831+0000 7f23cb180740 1 -- 172.21.15.169:0/2052219106 shutdown_connections 2024-03-28T01:05:02.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.831+0000 7f23cb180740 1 --2- 172.21.15.169:0/2052219106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aee5fb21f0 0x55aee5fbc770 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.831+0000 7f23cb180740 1 --2- 172.21.15.169:0/2052219106 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aee601eb60 0x55aee6020fc0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.831+0000 7f23cb180740 1 --2- 172.21.15.169:0/2052219106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aee5fad080 0x55aee5fbb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.831+0000 7f23cb180740 1 --2- 172.21.15.169:0/2052219106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aee5fac690 0x55aee5fbb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.831+0000 7f23cb180740 1 -- 172.21.15.169:0/2052219106 >> 172.21.15.169:0/2052219106 conn(0x55aee5fa7c00 msgr2=0x55aee5fb68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:02.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.831+0000 7f23a9ffb640 0 client.26987 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:02.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.832+0000 7f23cb180740 1 -- 172.21.15.169:0/2052219106 shutdown_connections 2024-03-28T01:05:02.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.832+0000 7f23cb180740 1 -- 172.21.15.169:0/2052219106 wait complete. 2024-03-28T01:05:02.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.834+0000 7f23cb180740 20 client.26987 trim_cache size 0 max 16384 2024-03-28T01:05:02.838 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:05:02.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.836+0000 7f23c84b2640 1 -- 172.21.15.169:0/3118615899 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f23ac0a2a10 msgr2=0x7f23ac0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:02.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.836+0000 7f23c84b2640 1 --2- 172.21.15.169:0/3118615899 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f23ac0a2a10 0x7f23ac0a4ed0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f23b0005ec0 tx=0x7f23b002c890 comp rx=0 tx=0).stop 2024-03-28T01:05:02.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.836+0000 7f23c84b2640 1 -- 172.21.15.169:0/3118615899 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23c01086a0 msgr2=0x7f23c0078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:02.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.836+0000 7f23c84b2640 1 --2- 172.21.15.169:0/3118615899 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23c01086a0 0x7f23c0078870 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f23b800ba50 tx=0x7f23b800bf10 comp rx=0 tx=0).stop 2024-03-28T01:05:02.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.836+0000 7f23c84b2640 1 -- 172.21.15.169:0/3118615899 shutdown_connections 2024-03-28T01:05:02.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.836+0000 7f23c84b2640 1 --2- 172.21.15.169:0/3118615899 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f23ac0a2a10 0x7f23ac0a4ed0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.836+0000 7f23c84b2640 1 --2- 172.21.15.169:0/3118615899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c010d9c0 0x7f23c007fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.836+0000 7f23c84b2640 1 --2- 172.21.15.169:0/3118615899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23c010b010 0x7f23c0078db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.836+0000 7f23c84b2640 1 --2- 172.21.15.169:0/3118615899 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f23c01086a0 0x7f23c0078870 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:02.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.836+0000 7f23c84b2640 1 -- 172.21.15.169:0/3118615899 >> 172.21.15.169:0/3118615899 conn(0x7f23c006e500 msgr2=0x7f23c0108ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:02.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.836+0000 7f23c84b2640 1 -- 172.21.15.169:0/3118615899 shutdown_connections 2024-03-28T01:05:02.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:02.837+0000 7f23c84b2640 1 -- 172.21.15.169:0/3118615899 wait complete. 2024-03-28T01:05:02.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:02 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2278605551' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:05:03.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:02 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2278605551' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:05:03.865 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T01:05:03.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T01:05:03.865 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T01:05:03.917 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:03 smithi169 ceph-mon[32274]: pgmap v607: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s rd, 385 MiB/s wr, 3.54k op/s 2024-03-28T01:05:03.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:03 smithi177 ceph-mon[36206]: pgmap v607: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s rd, 385 MiB/s wr, 3.54k op/s 2024-03-28T01:05:04.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:03 smithi174 ceph-mon[36060]: pgmap v607: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s rd, 385 MiB/s wr, 3.54k op/s 2024-03-28T01:05:04.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.106+0000 7fc1d1efe640 1 -- 172.21.15.169:0/1727924914 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1cc108d70 msgr2=0x7fc1cc1091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:04.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.106+0000 7fc1d1efe640 1 --2- 172.21.15.169:0/1727924914 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1cc108d70 0x7fc1cc1091f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0009900 tx=0x7fc1c002bc30 comp rx=0 tx=0).stop 2024-03-28T01:05:04.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.108+0000 7fc1d1efe640 1 -- 172.21.15.169:0/1727924914 shutdown_connections 2024-03-28T01:05:04.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.108+0000 7fc1d1efe640 1 --2- 172.21.15.169:0/1727924914 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1cc109730 0x7fc1cc10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.108+0000 7fc1d1efe640 1 --2- 172.21.15.169:0/1727924914 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1cc108d70 0x7fc1cc1091f0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.108+0000 7fc1d1efe640 1 --2- 172.21.15.169:0/1727924914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1cc072ab0 0x7fc1cc072ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.108+0000 7fc1d1efe640 1 -- 172.21.15.169:0/1727924914 >> 172.21.15.169:0/1727924914 conn(0x7fc1cc06e580 msgr2=0x7fc1cc0709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:04.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.114+0000 7fc1d1efe640 1 -- 172.21.15.169:0/1727924914 shutdown_connections 2024-03-28T01:05:04.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.114+0000 7fc1d1efe640 1 -- 172.21.15.169:0/1727924914 wait complete. 2024-03-28T01:05:04.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.115+0000 7fc1d1efe640 1 Processor -- start 2024-03-28T01:05:04.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.116+0000 7fc1d1efe640 1 -- start start 2024-03-28T01:05:04.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.116+0000 7fc1d1efe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1cc072ab0 0x7fc1cc1a6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.116+0000 7fc1d1efe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1cc108d70 0x7fc1cc1a6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.116+0000 7fc1d1efe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1cc109730 0x7fc1cc1add30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.116+0000 7fc1d1efe640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc11c080 con 0x7fc1cc108d70 2024-03-28T01:05:04.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.116+0000 7fc1d1efe640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc11bf00 con 0x7fc1cc109730 2024-03-28T01:05:04.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.116+0000 7fc1d1efe640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc11c200 con 0x7fc1cc072ab0 2024-03-28T01:05:04.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.116+0000 7fc1d16fd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1cc109730 0x7fc1cc1add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.116+0000 7fc1d16fd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1cc109730 0x7fc1cc1add30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:56768/0 (socket says 172.21.15.169:56768) 2024-03-28T01:05:04.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.116+0000 7fc1d16fd640 1 -- 172.21.15.169:0/3995574653 learned_addr learned my addr 172.21.15.169:0/3995574653 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:04.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.116+0000 7fc1cbfff640 1 --2- 172.21.15.169:0/3995574653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1cc108d70 0x7fc1cc1a6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.116+0000 7fc1d0efc640 1 --2- 172.21.15.169:0/3995574653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1cc072ab0 0x7fc1cc1a6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.117+0000 7fc1cbfff640 1 -- 172.21.15.169:0/3995574653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1cc072ab0 msgr2=0x7fc1cc1a6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:04.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.117+0000 7fc1cbfff640 1 --2- 172.21.15.169:0/3995574653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1cc072ab0 0x7fc1cc1a6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.117+0000 7fc1cbfff640 1 -- 172.21.15.169:0/3995574653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1cc109730 msgr2=0x7fc1cc1add30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:04.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.117+0000 7fc1cbfff640 1 --2- 172.21.15.169:0/3995574653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1cc109730 0x7fc1cc1add30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.117+0000 7fc1cbfff640 1 -- 172.21.15.169:0/3995574653 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1cc1ae430 con 0x7fc1cc108d70 2024-03-28T01:05:04.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.118+0000 7fc1cbfff640 1 --2- 172.21.15.169:0/3995574653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1cc108d70 0x7fc1cc1a6cc0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0036040 tx=0x7fc1c0003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:04.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.118+0000 7fc1c9ffb640 1 -- 172.21.15.169:0/3995574653 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0043020 con 0x7fc1cc108d70 2024-03-28T01:05:04.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.118+0000 7fc1d1efe640 1 -- 172.21.15.169:0/3995574653 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1cc1ae6c0 con 0x7fc1cc108d70 2024-03-28T01:05:04.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.118+0000 7fc1d1efe640 1 -- 172.21.15.169:0/3995574653 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1cc1aec60 con 0x7fc1cc108d70 2024-03-28T01:05:04.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.119+0000 7fc1c9ffb640 1 -- 172.21.15.169:0/3995574653 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1c0003c20 con 0x7fc1cc108d70 2024-03-28T01:05:04.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.119+0000 7fc1c9ffb640 1 -- 172.21.15.169:0/3995574653 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c00346a0 con 0x7fc1cc108d70 2024-03-28T01:05:04.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.122+0000 7fc1d5e20740 1 Processor -- start 2024-03-28T01:05:04.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.123+0000 7fc1d5e20740 1 -- start start 2024-03-28T01:05:04.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.123+0000 7fc1d5e20740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca76b30 0x55c83ca76f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.123+0000 7fc1d5e20740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c83ca77520 0x55c83ca7c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.123+0000 7fc1d5e20740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c83ca7c690 0x55c83ca7eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.123+0000 7fc1d5e20740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c83c9b6c90 con 0x55c83ca7c690 2024-03-28T01:05:04.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.123+0000 7fc1d5e20740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c83c9dc070 con 0x55c83ca77520 2024-03-28T01:05:04.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.123+0000 7fc1d5e20740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c83c984c80 con 0x55c83ca76b30 2024-03-28T01:05:04.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.124+0000 7fc1d16fd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca76b30 0x55c83ca76f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.124+0000 7fc1d16fd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca76b30 0x55c83ca76f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:49004/0 (socket says 172.21.15.169:49004) 2024-03-28T01:05:04.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.124+0000 7fc1d16fd640 1 -- 172.21.15.169:0/727688766 learned_addr learned my addr 172.21.15.169:0/727688766 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:04.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.124+0000 7fc1c9ffb640 1 -- 172.21.15.169:0/3995574653 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc1c0046020 con 0x7fc1cc108d70 2024-03-28T01:05:04.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.124+0000 7fc1cbfff640 1 --2- 172.21.15.169:0/727688766 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c83ca7c690 0x55c83ca7eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.125+0000 7fc1cbfff640 1 -- 172.21.15.169:0/727688766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca76b30 msgr2=0x55c83ca76f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:04.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.125+0000 7fc1cbfff640 1 --2- 172.21.15.169:0/727688766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca76b30 0x55c83ca76f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.125+0000 7fc1cbfff640 1 -- 172.21.15.169:0/727688766 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c83ca77520 msgr2=0x55c83ca7c020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:05:04.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.125+0000 7fc1cbfff640 1 --2- 172.21.15.169:0/727688766 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c83ca77520 0x55c83ca7c020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.125+0000 7fc1cbfff640 1 -- 172.21.15.169:0/727688766 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c83c97cd10 con 0x55c83ca7c690 2024-03-28T01:05:04.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.125+0000 7fc1c9ffb640 1 --2- 172.21.15.169:0/3995574653 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc1b40a2940 0x7fc1b40a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.126+0000 7fc1c9ffb640 1 -- 172.21.15.169:0/3995574653 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc1c00f1920 con 0x7fc1cc108d70 2024-03-28T01:05:04.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.127+0000 7fc1d16fd640 1 --2- 172.21.15.169:0/3995574653 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc1b40a2940 0x7fc1b40a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.127+0000 7fc1cbfff640 1 --2- 172.21.15.169:0/727688766 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c83ca7c690 0x55c83ca7eb20 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc1c00eefc0 tx=0x7fc1c00f7a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=817b05b0617b053b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:04.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.128+0000 7fc1b2ffd640 1 -- 172.21.15.169:0/727688766 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c00ba0d0 con 0x55c83ca7c690 2024-03-28T01:05:04.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.128+0000 7fc1b2ffd640 1 -- 172.21.15.169:0/727688766 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1c00be5c0 con 0x55c83ca7c690 2024-03-28T01:05:04.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.128+0000 7fc1b2ffd640 1 -- 172.21.15.169:0/727688766 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c00bc000 con 0x55c83ca7c690 2024-03-28T01:05:04.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.130+0000 7fc1d16fd640 1 --2- 172.21.15.169:0/3995574653 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc1b40a2940 0x7fc1b40a4e00 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4009fd0 tx=0x7fc1c4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:04.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.133+0000 7fc1d5e20740 1 -- 172.21.15.169:0/727688766 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c83ca7c690 msgr2=0x55c83ca7eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:04.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.133+0000 7fc1d5e20740 1 --2- 172.21.15.169:0/727688766 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c83ca7c690 0x55c83ca7eb20 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc1c00eefc0 tx=0x7fc1c00f7a00 comp rx=0 tx=0).stop 2024-03-28T01:05:04.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.134+0000 7fc1d5e20740 1 -- 172.21.15.169:0/727688766 shutdown_connections 2024-03-28T01:05:04.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.134+0000 7fc1d5e20740 1 --2- 172.21.15.169:0/727688766 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c83ca7c690 0x55c83ca7eb20 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.134+0000 7fc1d5e20740 1 --2- 172.21.15.169:0/727688766 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c83ca77520 0x55c83ca7c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.134+0000 7fc1d5e20740 1 --2- 172.21.15.169:0/727688766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca76b30 0x55c83ca76f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.134+0000 7fc1d5e20740 1 -- 172.21.15.169:0/727688766 >> 172.21.15.169:0/727688766 conn(0x55c83ca3f1a0 msgr2=0x55c83ca724a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:04.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.134+0000 7fc1d5e20740 1 -- 172.21.15.169:0/727688766 shutdown_connections 2024-03-28T01:05:04.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.134+0000 7fc1d5e20740 1 -- 172.21.15.169:0/727688766 wait complete. 2024-03-28T01:05:04.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.135+0000 7fc1d5e20740 1 -- start start 2024-03-28T01:05:04.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.135+0000 7fc1d5e20740 1 Processor -- start 2024-03-28T01:05:04.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.135+0000 7fc1d5e20740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c83ca76b30 0x55c83ca80910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.136+0000 7fc1d5e20740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca77520 0x55c83ca80e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.136+0000 7fc1d5e20740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c83ca7c690 0x55c83ca87e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.136+0000 7fc1d5e20740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c83c9980d0 con 0x55c83ca76b30 2024-03-28T01:05:04.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.136+0000 7fc1cbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca77520 0x55c83ca80e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.136+0000 7fc1cbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca77520 0x55c83ca80e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:49008/0 (socket says 172.21.15.169:49008) 2024-03-28T01:05:04.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.136+0000 7fc1cbfff640 1 -- 172.21.15.169:0/4067021075 learned_addr learned my addr 172.21.15.169:0/4067021075 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:04.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.136+0000 7fc1d5e20740 1 -- 172.21.15.169:0/4067021075 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c83c9d8110 con 0x55c83ca7c690 2024-03-28T01:05:04.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.136+0000 7fc1d5e20740 1 -- 172.21.15.169:0/4067021075 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c83c9b6c90 con 0x55c83ca77520 2024-03-28T01:05:04.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.136+0000 7fc1d5e20740 10 client.0 _ll_register_callbacks cb 0x55c83c9d3e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:04.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.136+0000 7fc1d16fd640 1 --2- 172.21.15.169:0/4067021075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c83ca7c690 0x55c83ca87e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.137+0000 7fc1d16fd640 1 -- 172.21.15.169:0/4067021075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca77520 msgr2=0x55c83ca80e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:04.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.137+0000 7fc1d16fd640 1 --2- 172.21.15.169:0/4067021075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca77520 0x55c83ca80e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.137+0000 7fc1d16fd640 1 -- 172.21.15.169:0/4067021075 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c83ca76b30 msgr2=0x55c83ca80910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:04.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.137+0000 7fc1d16fd640 1 --2- 172.21.15.169:0/4067021075 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c83ca76b30 0x55c83ca80910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.137+0000 7fc1d16fd640 1 -- 172.21.15.169:0/4067021075 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c83c7b87e0 con 0x55c83ca7c690 2024-03-28T01:05:04.142 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.139+0000 7fc1d16fd640 1 --2- 172.21.15.169:0/4067021075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c83ca7c690 0x55c83ca87e90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400f690 tx=0x7fc1c4004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b54bb26c244728e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:04.142 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.139+0000 7fc1abfff640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:05:04.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.141+0000 7fc1abfff640 1 -- 172.21.15.169:0/4067021075 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c40118d0 con 0x55c83ca7c690 2024-03-28T01:05:04.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.141+0000 7fc1d5e20740 1 -- 172.21.15.169:0/4067021075 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc1c00be5c0 con 0x55c83ca7c690 2024-03-28T01:05:04.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.143+0000 7fc1abfff640 1 -- 172.21.15.169:0/4067021075 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc1c4011a70 con 0x55c83ca7c690 2024-03-28T01:05:04.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.144+0000 7fc1abfff640 1 -- 172.21.15.169:0/4067021075 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1c4011c70 con 0x55c83ca7c690 2024-03-28T01:05:04.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.145+0000 7fc1d5e20740 10 client.26993 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:04.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.145+0000 7fc1d5e20740 1 -- 172.21.15.169:0/4067021075 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c83c818f70 con 0x55c83ca7c690 2024-03-28T01:05:04.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.153+0000 7fc1abfff640 1 -- 172.21.15.169:0/4067021075 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1c00be5c0 con 0x55c83ca7c690 2024-03-28T01:05:04.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.153+0000 7fc1abfff640 1 -- 172.21.15.169:0/4067021075 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fc1c4027070 con 0x55c83ca7c690 2024-03-28T01:05:04.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.153+0000 7fc1d5e20740 10 client.26993 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:04.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.153+0000 7fc1d5e20740 10 client.26993 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:04.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.153+0000 7fc1d5e20740 20 client.26993 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:04.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.153+0000 7fc1d5e20740 1 --2- 172.21.15.169:0/4067021075 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55c83ca8cc50 0x55c83ca8f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.153+0000 7fc1d5e20740 4 client.26993 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:05:04.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.153+0000 7fc1d5e20740 1 -- 172.21.15.169:0/4067021075 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c83c9e95b0 con 0x55c83ca8cc50 2024-03-28T01:05:04.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.157+0000 7fc1d0efc640 1 --2- 172.21.15.169:0/4067021075 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55c83ca8cc50 0x55c83ca8f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.174+0000 7fc1d0efc640 1 --2- 172.21.15.169:0/4067021075 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55c83ca8cc50 0x55c83ca8f0b0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a24da123a3e8726f server_cookie=2f87d32cbe53846a in_seq=0 out_seq=0 2024-03-28T01:05:04.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.174+0000 7fc1abfff640 10 client.26993 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:04.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.182+0000 7fc1abfff640 1 -- 172.21.15.169:0/4067021075 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c83c9e95b0 con 0x55c83ca8cc50 2024-03-28T01:05:04.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.182+0000 7fc1abfff640 10 client.26993 handle_command_reply: tid=0 2024-03-28T01:05:04.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.182+0000 7fc1d5e20740 1 client.26993 shutdown 2024-03-28T01:05:04.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.182+0000 7fc1d5e20740 1 -- 172.21.15.169:0/4067021075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c83ca7c690 msgr2=0x55c83ca87e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:04.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.182+0000 7fc1d5e20740 1 --2- 172.21.15.169:0/4067021075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c83ca7c690 0x55c83ca87e90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400f690 tx=0x7fc1c4004ab0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.183+0000 7fc1d5e20740 1 -- 172.21.15.169:0/4067021075 shutdown_connections 2024-03-28T01:05:04.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.183+0000 7fc1d5e20740 1 --2- 172.21.15.169:0/4067021075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c83ca7c690 0x55c83ca87e90 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.183+0000 7fc1d5e20740 1 --2- 172.21.15.169:0/4067021075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca77520 0x55c83ca80e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.183+0000 7fc1d5e20740 1 --2- 172.21.15.169:0/4067021075 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55c83ca8cc50 0x55c83ca8f0b0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.183+0000 7fc1d5e20740 1 --2- 172.21.15.169:0/4067021075 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c83ca76b30 0x55c83ca80910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.183+0000 7fc1d5e20740 1 -- 172.21.15.169:0/4067021075 >> 172.21.15.169:0/4067021075 conn(0x55c83ca3f1a0 msgr2=0x55c83ca717e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:04.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.185+0000 7fc1abfff640 0 client.26993 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:04.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.185+0000 7fc1d5e20740 1 -- 172.21.15.169:0/4067021075 shutdown_connections 2024-03-28T01:05:04.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.186+0000 7fc1d5e20740 1 -- 172.21.15.169:0/4067021075 wait complete. 2024-03-28T01:05:04.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.188+0000 7fc1d5e20740 20 client.26993 trim_cache size 0 max 16384 2024-03-28T01:05:04.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.237+0000 7fc1d5e20740 1 Processor -- start 2024-03-28T01:05:04.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.237+0000 7fc1d5e20740 1 -- start start 2024-03-28T01:05:04.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.237+0000 7fc1d5e20740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c83ca76b30 0x55c83ca802f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.237+0000 7fc1d5e20740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca77520 0x55c83ca87e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.237+0000 7fc1d5e20740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c83ca7c690 0x55c83ca756e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.237+0000 7fc1d5e20740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc1bb640 con 0x55c83ca76b30 2024-03-28T01:05:04.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.237+0000 7fc1d5e20740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc1b9140 con 0x55c83ca7c690 2024-03-28T01:05:04.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.237+0000 7fc1d5e20740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc1b71e0 con 0x55c83ca77520 2024-03-28T01:05:04.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.238+0000 7fc1cbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca77520 0x55c83ca87e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.238+0000 7fc1cbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca77520 0x55c83ca87e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:49022/0 (socket says 172.21.15.169:49022) 2024-03-28T01:05:04.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.238+0000 7fc1cbfff640 1 -- 172.21.15.169:0/1512951245 learned_addr learned my addr 172.21.15.169:0/1512951245 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:04.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.238+0000 7fc1cbfff640 1 -- 172.21.15.169:0/1512951245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c83ca7c690 msgr2=0x55c83ca756e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:04.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.238+0000 7fc1cbfff640 1 --2- 172.21.15.169:0/1512951245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c83ca7c690 0x55c83ca756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.238+0000 7fc1cbfff640 1 -- 172.21.15.169:0/1512951245 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c83ca76b30 msgr2=0x55c83ca802f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:05:04.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.238+0000 7fc1cbfff640 1 --2- 172.21.15.169:0/1512951245 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c83ca76b30 0x55c83ca802f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.238+0000 7fc1cbfff640 1 -- 172.21.15.169:0/1512951245 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c83c6b0170 con 0x55c83ca77520 2024-03-28T01:05:04.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.239+0000 7fc1cbfff640 1 --2- 172.21.15.169:0/1512951245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca77520 0x55c83ca87e50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0009900 tx=0x7fc1c0000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49bacf9f683ff4d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:04.249 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.245+0000 7fc1b27fc640 1 -- 172.21.15.169:0/1512951245 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c00bd820 con 0x55c83ca77520 2024-03-28T01:05:04.249 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.245+0000 7fc1b27fc640 1 -- 172.21.15.169:0/1512951245 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1c00be5c0 con 0x55c83ca77520 2024-03-28T01:05:04.249 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.246+0000 7fc1d5e20740 1 -- 172.21.15.169:0/1512951245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca77520 msgr2=0x55c83ca87e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:04.249 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.246+0000 7fc1d5e20740 1 --2- 172.21.15.169:0/1512951245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca77520 0x55c83ca87e50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0009900 tx=0x7fc1c0000bc0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.248+0000 7fc1d5e20740 1 -- 172.21.15.169:0/1512951245 shutdown_connections 2024-03-28T01:05:04.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.248+0000 7fc1d5e20740 1 --2- 172.21.15.169:0/1512951245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c83ca7c690 0x55c83ca756e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.248+0000 7fc1d5e20740 1 --2- 172.21.15.169:0/1512951245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca77520 0x55c83ca87e50 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.248+0000 7fc1d5e20740 1 --2- 172.21.15.169:0/1512951245 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c83ca76b30 0x55c83ca802f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.248+0000 7fc1d5e20740 1 -- 172.21.15.169:0/1512951245 >> 172.21.15.169:0/1512951245 conn(0x55c83ca3f1a0 msgr2=0x55c83ca7fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:04.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.256+0000 7fc1d5e20740 1 -- 172.21.15.169:0/1512951245 shutdown_connections 2024-03-28T01:05:04.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.259+0000 7fc1d5e20740 1 -- 172.21.15.169:0/1512951245 wait complete. 2024-03-28T01:05:04.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.259+0000 7fc1d5e20740 1 -- start start 2024-03-28T01:05:04.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.259+0000 7fc1d5e20740 1 Processor -- start 2024-03-28T01:05:04.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.260+0000 7fc1d5e20740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c83ca76b30 0x55c83ca85c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.260+0000 7fc1d5e20740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca77520 0x55c83ca86180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.260+0000 7fc1d5e20740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c83ca7c690 0x55c83ca87190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.260+0000 7fc1d5e20740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc1b77e0 con 0x55c83ca76b30 2024-03-28T01:05:04.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.260+0000 7fc1d5e20740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc1b8ae0 con 0x55c83ca7c690 2024-03-28T01:05:04.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.260+0000 7fc1d5e20740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc1b7e20 con 0x55c83ca77520 2024-03-28T01:05:04.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.260+0000 7fc1d5e20740 10 client.0 _ll_register_callbacks cb 0x7fc1c40124a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:04.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.262+0000 7fc1cbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca77520 0x55c83ca86180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.262+0000 7fc1cbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca77520 0x55c83ca86180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:49034/0 (socket says 172.21.15.169:49034) 2024-03-28T01:05:04.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.262+0000 7fc1cbfff640 1 -- 172.21.15.169:0/3456421798 learned_addr learned my addr 172.21.15.169:0/3456421798 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:04.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.263+0000 7fc1cbfff640 1 -- 172.21.15.169:0/3456421798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c83ca7c690 msgr2=0x55c83ca87190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:04.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.263+0000 7fc1cbfff640 1 --2- 172.21.15.169:0/3456421798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c83ca7c690 0x55c83ca87190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.263+0000 7fc1cbfff640 1 -- 172.21.15.169:0/3456421798 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c83ca76b30 msgr2=0x55c83ca85c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:04.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.263+0000 7fc1cbfff640 1 --2- 172.21.15.169:0/3456421798 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c83ca76b30 0x55c83ca85c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.263+0000 7fc1cbfff640 1 -- 172.21.15.169:0/3456421798 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c83ca87780 con 0x55c83ca77520 2024-03-28T01:05:04.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.266+0000 7fc1cbfff640 1 --2- 172.21.15.169:0/3456421798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca77520 0x55c83ca86180 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0009850 tx=0x7fc1c002cc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6af8d5e44163fae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:04.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.266+0000 7fc1abfff640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:05:04.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.268+0000 7fc1abfff640 1 -- 172.21.15.169:0/3456421798 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c00bdc90 con 0x55c83ca77520 2024-03-28T01:05:04.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.269+0000 7fc1d5e20740 1 -- 172.21.15.169:0/3456421798 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc1c00be5c0 con 0x55c83ca77520 2024-03-28T01:05:04.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.269+0000 7fc1abfff640 1 -- 172.21.15.169:0/3456421798 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc1c00bc1c0 con 0x55c83ca77520 2024-03-28T01:05:04.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.270+0000 7fc1abfff640 1 -- 172.21.15.169:0/3456421798 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1c00bd820 con 0x55c83ca77520 2024-03-28T01:05:04.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.271+0000 7fc1d5e20740 10 client.26941 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:04.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.271+0000 7fc1d5e20740 1 -- 172.21.15.169:0/3456421798 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c83cae8c10 con 0x55c83ca77520 2024-03-28T01:05:04.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.271+0000 7fc1abfff640 1 -- 172.21.15.169:0/3456421798 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1c00be5c0 con 0x55c83ca77520 2024-03-28T01:05:04.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.271+0000 7fc1abfff640 1 -- 172.21.15.169:0/3456421798 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fc1c00d4070 con 0x55c83ca77520 2024-03-28T01:05:04.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.272+0000 7fc1d5e20740 10 client.26941 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:04.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.272+0000 7fc1d5e20740 10 client.26941 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:04.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.272+0000 7fc1d5e20740 20 client.26941 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:04.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.272+0000 7fc1d5e20740 1 --2- 172.21.15.169:0/3456421798 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55c83cae9420 0x55c83caeb840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.272+0000 7fc1d5e20740 4 client.26941 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T01:05:04.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.272+0000 7fc1d5e20740 1 -- 172.21.15.169:0/3456421798 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55c83c9e3900 con 0x55c83cae9420 2024-03-28T01:05:04.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.274+0000 7fc1d0efc640 1 --2- 172.21.15.169:0/3456421798 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55c83cae9420 0x55c83caeb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.277+0000 7fc1d0efc640 1 --2- 172.21.15.169:0/3456421798 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55c83cae9420 0x55c83caeb840 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9f3c37c7d82036b server_cookie=f575264b6679823d in_seq=0 out_seq=0 2024-03-28T01:05:04.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.277+0000 7fc1abfff640 10 client.26941 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:04.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.282+0000 7fc1abfff640 1 -- 172.21.15.169:0/3456421798 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55c83c9e3900 con 0x55c83cae9420 2024-03-28T01:05:04.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.282+0000 7fc1abfff640 10 client.26941 handle_command_reply: tid=0 2024-03-28T01:05:04.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.282+0000 7fc1d5e20740 1 client.26941 shutdown 2024-03-28T01:05:04.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.285+0000 7fc1d5e20740 1 -- 172.21.15.169:0/3456421798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca77520 msgr2=0x55c83ca86180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:04.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.285+0000 7fc1d5e20740 1 --2- 172.21.15.169:0/3456421798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca77520 0x55c83ca86180 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0009850 tx=0x7fc1c002cc50 comp rx=0 tx=0).stop 2024-03-28T01:05:04.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.285+0000 7fc1d5e20740 1 -- 172.21.15.169:0/3456421798 shutdown_connections 2024-03-28T01:05:04.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.285+0000 7fc1d5e20740 1 --2- 172.21.15.169:0/3456421798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c83ca7c690 0x55c83ca87190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.285+0000 7fc1d5e20740 1 --2- 172.21.15.169:0/3456421798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c83ca77520 0x55c83ca86180 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.285+0000 7fc1d5e20740 1 --2- 172.21.15.169:0/3456421798 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55c83cae9420 0x55c83caeb840 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.285+0000 7fc1d5e20740 1 --2- 172.21.15.169:0/3456421798 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c83ca76b30 0x55c83ca85c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.285+0000 7fc1d5e20740 1 -- 172.21.15.169:0/3456421798 >> 172.21.15.169:0/3456421798 conn(0x55c83ca3f1a0 msgr2=0x55c83ca756e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:04.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.285+0000 7fc1abfff640 0 client.26941 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:04.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.288+0000 7fc1d5e20740 1 -- 172.21.15.169:0/3456421798 shutdown_connections 2024-03-28T01:05:04.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.288+0000 7fc1d5e20740 1 -- 172.21.15.169:0/3456421798 wait complete. 2024-03-28T01:05:04.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.288+0000 7fc1d5e20740 20 client.26941 trim_cache size 0 max 16384 2024-03-28T01:05:04.299 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:05:04.300 INFO:teuthology.orchestra.run.smithi169.stdout: "return_code": 0, 2024-03-28T01:05:04.300 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_tag": "467d0b78-47ea-435a-910d-2fe6c4a39e08", 2024-03-28T01:05:04.300 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": "asynchronous" 2024-03-28T01:05:04.300 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:05:04.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.299+0000 7fc1d1efe640 1 -- 172.21.15.169:0/3995574653 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc1b40a2940 msgr2=0x7fc1b40a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:04.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.299+0000 7fc1d1efe640 1 --2- 172.21.15.169:0/3995574653 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc1b40a2940 0x7fc1b40a4e00 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4009fd0 tx=0x7fc1c4009500 comp rx=0 tx=0).stop 2024-03-28T01:05:04.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.299+0000 7fc1d1efe640 1 -- 172.21.15.169:0/3995574653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1cc108d70 msgr2=0x7fc1cc1a6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:04.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.299+0000 7fc1d1efe640 1 --2- 172.21.15.169:0/3995574653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1cc108d70 0x7fc1cc1a6cc0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0036040 tx=0x7fc1c0003820 comp rx=0 tx=0).stop 2024-03-28T01:05:04.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.299+0000 7fc1d1efe640 1 -- 172.21.15.169:0/3995574653 shutdown_connections 2024-03-28T01:05:04.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.299+0000 7fc1d1efe640 1 --2- 172.21.15.169:0/3995574653 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc1b40a2940 0x7fc1b40a4e00 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.299+0000 7fc1d1efe640 1 --2- 172.21.15.169:0/3995574653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1cc109730 0x7fc1cc1add30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.299+0000 7fc1d1efe640 1 --2- 172.21.15.169:0/3995574653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1cc108d70 0x7fc1cc1a6cc0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.299+0000 7fc1d1efe640 1 --2- 172.21.15.169:0/3995574653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1cc072ab0 0x7fc1cc1a6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.299+0000 7fc1d1efe640 1 -- 172.21.15.169:0/3995574653 >> 172.21.15.169:0/3995574653 conn(0x7fc1cc06e580 msgr2=0x7fc1cc06f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:04.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.301+0000 7fc1d1efe640 1 -- 172.21.15.169:0/3995574653 shutdown_connections 2024-03-28T01:05:04.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.301+0000 7fc1d1efe640 1 -- 172.21.15.169:0/3995574653 wait complete. 2024-03-28T01:05:04.343 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:05:04.343 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:05:04.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.635+0000 7f305e8ea640 1 -- 172.21.15.169:0/376168383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3058101080 msgr2=0x7f3058105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:04.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.635+0000 7f305e8ea640 1 --2- 172.21.15.169:0/376168383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3058101080 0x7f3058105920 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3050009900 tx=0x7f305002bc40 comp rx=0 tx=0).stop 2024-03-28T01:05:04.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.637+0000 7f305e8ea640 1 -- 172.21.15.169:0/376168383 shutdown_connections 2024-03-28T01:05:04.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.637+0000 7f305e8ea640 1 --2- 172.21.15.169:0/376168383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3058101080 0x7f3058105920 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.637+0000 7f305e8ea640 1 --2- 172.21.15.169:0/376168383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30581006c0 0x7f3058100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.637+0000 7f305e8ea640 1 --2- 172.21.15.169:0/376168383 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f30580ff960 0x7f30580ffd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.637+0000 7f305e8ea640 1 -- 172.21.15.169:0/376168383 >> 172.21.15.169:0/376168383 conn(0x7f30580fb290 msgr2=0x7f30580fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:04.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.638+0000 7f305e8ea640 1 -- 172.21.15.169:0/376168383 shutdown_connections 2024-03-28T01:05:04.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.640+0000 7f305e8ea640 1 -- 172.21.15.169:0/376168383 wait complete. 2024-03-28T01:05:04.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.640+0000 7f305e8ea640 1 Processor -- start 2024-03-28T01:05:04.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.644+0000 7f305e8ea640 1 -- start start 2024-03-28T01:05:04.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.645+0000 7f305e8ea640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30580ff960 0x7f305819dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.645+0000 7f305e8ea640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30581006c0 0x7f305819e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.645+0000 7f305e8ea640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3058101080 0x7f30581a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.645+0000 7f305e8ea640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3058113990 con 0x7f3058101080 2024-03-28T01:05:04.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.645+0000 7f305e8ea640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3058113810 con 0x7f30580ff960 2024-03-28T01:05:04.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.645+0000 7f305e8ea640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3058113b10 con 0x7f30581006c0 2024-03-28T01:05:04.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.646+0000 7f305ce87640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3058101080 0x7f30581a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.646+0000 7f3057fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30580ff960 0x7f305819dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.646+0000 7f305ce87640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3058101080 0x7f30581a5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34364/0 (socket says 172.21.15.169:34364) 2024-03-28T01:05:04.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.646+0000 7f305ce87640 1 -- 172.21.15.169:0/4084985974 learned_addr learned my addr 172.21.15.169:0/4084985974 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:04.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.647+0000 7f305ce87640 1 -- 172.21.15.169:0/4084985974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30581006c0 msgr2=0x7f305819e510 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:05:04.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.647+0000 7f305ce87640 1 --2- 172.21.15.169:0/4084985974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30581006c0 0x7f305819e510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.647+0000 7f305ce87640 1 -- 172.21.15.169:0/4084985974 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30580ff960 msgr2=0x7f305819dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:04.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.647+0000 7f305ce87640 1 --2- 172.21.15.169:0/4084985974 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30580ff960 0x7f305819dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.647+0000 7f305ce87640 1 -- 172.21.15.169:0/4084985974 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30581a5ba0 con 0x7f3058101080 2024-03-28T01:05:04.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.648+0000 7f305ce87640 1 --2- 172.21.15.169:0/4084985974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3058101080 0x7f30581a5530 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f305002efd0 tx=0x7f3050004f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:04.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.650+0000 7f30557fa640 1 -- 172.21.15.169:0/4084985974 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3050041070 con 0x7f3058101080 2024-03-28T01:05:04.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.650+0000 7f30557fa640 1 -- 172.21.15.169:0/4084985974 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3050004450 con 0x7f3058101080 2024-03-28T01:05:04.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.650+0000 7f30557fa640 1 -- 172.21.15.169:0/4084985974 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3050034440 con 0x7f3058101080 2024-03-28T01:05:04.653 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.651+0000 7f305e8ea640 1 -- 172.21.15.169:0/4084985974 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30581a5e30 con 0x7f3058101080 2024-03-28T01:05:04.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.651+0000 7f305e8ea640 1 -- 172.21.15.169:0/4084985974 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30581a62a0 con 0x7f3058101080 2024-03-28T01:05:04.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.655+0000 7f30615b4740 1 Processor -- start 2024-03-28T01:05:04.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.658+0000 7f30615b4740 1 -- start start 2024-03-28T01:05:04.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.658+0000 7f30557fa640 1 -- 172.21.15.169:0/4084985974 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3050044070 con 0x7f3058101080 2024-03-28T01:05:04.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.659+0000 7f30557fa640 1 --2- 172.21.15.169:0/4084985974 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f30440a2a00 0x7f30440a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.660+0000 7f30557fa640 1 -- 172.21.15.169:0/4084985974 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f30500f0680 con 0x7f3058101080 2024-03-28T01:05:04.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.660+0000 7f30615b4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560866fb6da0 0x560866fb71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.660+0000 7f30615b4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fb7790 0x560866fbc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.660+0000 7f30615b4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560866fbc900 0x560866fbed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.660+0000 7f30615b4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x560866f1c3d0 con 0x560866fbc900 2024-03-28T01:05:04.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.660+0000 7f30615b4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x560866ec4ad0 con 0x560866fb6da0 2024-03-28T01:05:04.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.660+0000 7f30615b4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560866f18470 con 0x560866fb7790 2024-03-28T01:05:04.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.660+0000 7f30577fe640 1 --2- 172.21.15.169:0/4084985974 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f30440a2a00 0x7f30440a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.662+0000 7f30577fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fb7790 0x560866fbc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.662+0000 7f30577fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fb7790 0x560866fbc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:49052/0 (socket says 172.21.15.169:49052) 2024-03-28T01:05:04.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.662+0000 7f30577fe640 1 -- 172.21.15.169:0/228456494 learned_addr learned my addr 172.21.15.169:0/228456494 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:04.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.666+0000 7f3057fff640 1 --2- 172.21.15.169:0/228456494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560866fb6da0 0x560866fb71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.667+0000 7f30577fe640 1 -- 172.21.15.169:0/228456494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560866fb6da0 msgr2=0x560866fb71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:04.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.667+0000 7f30577fe640 1 --2- 172.21.15.169:0/228456494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560866fb6da0 0x560866fb71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.667+0000 7f30577fe640 1 -- 172.21.15.169:0/228456494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560866fbc900 msgr2=0x560866fbed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:04.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.667+0000 7f30577fe640 1 --2- 172.21.15.169:0/228456494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560866fbc900 0x560866fbed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.667+0000 7f30577fe640 1 -- 172.21.15.169:0/228456494 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560866cfa7d0 con 0x560866fb7790 2024-03-28T01:05:04.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.670+0000 7f30577fe640 1 --2- 172.21.15.169:0/228456494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fb7790 0x560866fbc290 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3040013a10 tx=0x7f3040013ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0ab9ae701e251e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:04.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.674+0000 7f30577fe640 1 --2- 172.21.15.169:0/4084985974 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f30440a2a00 0x7f30440a4ec0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3040010740 tx=0x7f304000e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:04.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.674+0000 7f304e7fc640 1 -- 172.21.15.169:0/228456494 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040013250 con 0x560866fb7790 2024-03-28T01:05:04.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.674+0000 7f304e7fc640 1 -- 172.21.15.169:0/228456494 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30400133f0 con 0x560866fb7790 2024-03-28T01:05:04.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.674+0000 7f304e7fc640 1 -- 172.21.15.169:0/228456494 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040012960 con 0x560866fb7790 2024-03-28T01:05:04.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.676+0000 7f30615b4740 1 -- 172.21.15.169:0/228456494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fb7790 msgr2=0x560866fbc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:04.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.676+0000 7f30615b4740 1 --2- 172.21.15.169:0/228456494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fb7790 0x560866fbc290 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3040013a10 tx=0x7f3040013ed0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.681+0000 7f30615b4740 1 -- 172.21.15.169:0/228456494 shutdown_connections 2024-03-28T01:05:04.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.681+0000 7f30615b4740 1 --2- 172.21.15.169:0/228456494 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560866fbc900 0x560866fbed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.681+0000 7f30615b4740 1 --2- 172.21.15.169:0/228456494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fb7790 0x560866fbc290 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.681+0000 7f30615b4740 1 --2- 172.21.15.169:0/228456494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560866fb6da0 0x560866fb71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.681+0000 7f30615b4740 1 -- 172.21.15.169:0/228456494 >> 172.21.15.169:0/228456494 conn(0x560866fb2310 msgr2=0x560866fb2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:04.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.683+0000 7f30615b4740 1 -- 172.21.15.169:0/228456494 shutdown_connections 2024-03-28T01:05:04.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.689+0000 7f30615b4740 1 -- 172.21.15.169:0/228456494 wait complete. 2024-03-28T01:05:04.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.689+0000 7f30615b4740 1 -- start start 2024-03-28T01:05:04.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.690+0000 7f30615b4740 1 Processor -- start 2024-03-28T01:05:04.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.693+0000 7f30615b4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fb6da0 0x560866fc1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.693+0000 7f30615b4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560866fb7790 0x560866fc1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.693+0000 7f30615b4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560866fbc900 0x560866fc88a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.693+0000 7f30615b4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x560866fbf850 con 0x560866fbc900 2024-03-28T01:05:04.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.693+0000 7f30615b4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x560866ed8060 con 0x560866fb7790 2024-03-28T01:05:04.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.693+0000 7f30615b4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560866fbf9d0 con 0x560866fb6da0 2024-03-28T01:05:04.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.693+0000 7f30615b4740 10 client.0 _ll_register_callbacks cb 0x560866eba6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:04.698 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.694+0000 7f305ce87640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560866fbc900 0x560866fc88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.698 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.694+0000 7f305ce87640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560866fbc900 0x560866fc88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34384/0 (socket says 172.21.15.169:34384) 2024-03-28T01:05:04.698 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.694+0000 7f305ce87640 1 -- 172.21.15.169:0/4152263772 learned_addr learned my addr 172.21.15.169:0/4152263772 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:04.698 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.695+0000 7f3057fff640 1 --2- 172.21.15.169:0/4152263772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fb6da0 0x560866fc1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.695+0000 7f305ce87640 1 -- 172.21.15.169:0/4152263772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fb6da0 msgr2=0x560866fc1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:04.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.695+0000 7f305ce87640 1 --2- 172.21.15.169:0/4152263772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fb6da0 0x560866fc1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.695+0000 7f305ce87640 1 -- 172.21.15.169:0/4152263772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560866fb7790 msgr2=0x560866fc1860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:05:04.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.695+0000 7f305ce87640 1 --2- 172.21.15.169:0/4152263772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560866fb7790 0x560866fc1860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.695+0000 7f305ce87640 1 -- 172.21.15.169:0/4152263772 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560866c303f0 con 0x560866fbc900 2024-03-28T01:05:04.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.695+0000 7f305ce87640 1 --2- 172.21.15.169:0/4152263772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560866fbc900 0x560866fc88a0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f30500bcbb0 tx=0x7f3050003920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c30036877c4a0e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:04.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.698+0000 7f30377fe640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:05:04.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.698+0000 7f30377fe640 1 -- 172.21.15.169:0/4152263772 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30500bad60 con 0x560866fbc900 2024-03-28T01:05:04.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.698+0000 7f30377fe640 1 -- 172.21.15.169:0/4152263772 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f30500bce80 con 0x560866fbc900 2024-03-28T01:05:04.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.699+0000 7f30615b4740 1 -- 172.21.15.169:0/4152263772 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f30400133f0 con 0x560866fbc900 2024-03-28T01:05:04.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.713+0000 7f30377fe640 1 -- 172.21.15.169:0/4152263772 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3050004c80 con 0x560866fbc900 2024-03-28T01:05:04.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.715+0000 7f30615b4740 10 client.17151 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:04.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.715+0000 7f30615b4740 1 -- 172.21.15.169:0/4152263772 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x560866bf00f0 con 0x560866fbc900 2024-03-28T01:05:04.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.716+0000 7f30377fe640 1 -- 172.21.15.169:0/4152263772 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30400133f0 con 0x560866fbc900 2024-03-28T01:05:04.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.717+0000 7f30377fe640 1 -- 172.21.15.169:0/4152263772 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f30500d5ce0 con 0x560866fbc900 2024-03-28T01:05:04.720 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.717+0000 7f30615b4740 10 client.17151 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:04.720 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.717+0000 7f30615b4740 10 client.17151 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:04.720 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.717+0000 7f30615b4740 20 client.17151 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:04.720 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.717+0000 7f30615b4740 1 --2- 172.21.15.169:0/4152263772 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560866fde070 0x560866fe04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.720 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.717+0000 7f30615b4740 4 client.17151 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:05:04.720 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.717+0000 7f30615b4740 1 -- 172.21.15.169:0/4152263772 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560866f29910 con 0x560866fde070 2024-03-28T01:05:04.720 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.717+0000 7f3057fff640 1 --2- 172.21.15.169:0/4152263772 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560866fde070 0x560866fe04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.718+0000 7f30377fe640 10 client.17151 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:04.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.718+0000 7f3057fff640 1 --2- 172.21.15.169:0/4152263772 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560866fde070 0x560866fe04f0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cf79bedd45e9ea2 server_cookie=436afc573c0ec1e6 in_seq=0 out_seq=0 2024-03-28T01:05:04.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.719+0000 7f30377fe640 1 -- 172.21.15.169:0/4152263772 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560866f29910 con 0x560866fde070 2024-03-28T01:05:04.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.719+0000 7f30377fe640 10 client.17151 handle_command_reply: tid=0 2024-03-28T01:05:04.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.719+0000 7f30615b4740 1 client.17151 shutdown 2024-03-28T01:05:04.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.724+0000 7f30615b4740 1 -- 172.21.15.169:0/4152263772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560866fbc900 msgr2=0x560866fc88a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:04.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.724+0000 7f30615b4740 1 --2- 172.21.15.169:0/4152263772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560866fbc900 0x560866fc88a0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f30500bcbb0 tx=0x7f3050003920 comp rx=0 tx=0).stop 2024-03-28T01:05:04.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.728+0000 7f30615b4740 1 -- 172.21.15.169:0/4152263772 shutdown_connections 2024-03-28T01:05:04.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.728+0000 7f30615b4740 1 --2- 172.21.15.169:0/4152263772 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560866fde070 0x560866fe04f0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.728+0000 7f30615b4740 1 --2- 172.21.15.169:0/4152263772 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560866fbc900 0x560866fc88a0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.728+0000 7f30615b4740 1 --2- 172.21.15.169:0/4152263772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560866fb7790 0x560866fc1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.728+0000 7f30615b4740 1 --2- 172.21.15.169:0/4152263772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fb6da0 0x560866fc1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.728+0000 7f30615b4740 1 -- 172.21.15.169:0/4152263772 >> 172.21.15.169:0/4152263772 conn(0x560866fb2310 msgr2=0x560866fb1530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:04.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.728+0000 7f30377fe640 0 client.17151 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:04.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.730+0000 7f30615b4740 1 -- 172.21.15.169:0/4152263772 shutdown_connections 2024-03-28T01:05:04.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.734+0000 7f30615b4740 1 -- 172.21.15.169:0/4152263772 wait complete. 2024-03-28T01:05:04.740 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.737+0000 7f30615b4740 20 client.17151 trim_cache size 0 max 16384 2024-03-28T01:05:04.777 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.774+0000 7f30615b4740 1 Processor -- start 2024-03-28T01:05:04.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.775+0000 7f30615b4740 1 -- start start 2024-03-28T01:05:04.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.775+0000 7f30615b4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560866fb6da0 0x560866fc1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.775+0000 7f30615b4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560866fb7790 0x560866fc8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.775+0000 7f30615b4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fbc900 0x560866fc1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.775+0000 7f30615b4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30581aa3f0 con 0x560866fb6da0 2024-03-28T01:05:04.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.775+0000 7f30615b4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30581a7ef0 con 0x560866fb7790 2024-03-28T01:05:04.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.775+0000 7f30615b4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30580732a0 con 0x560866fbc900 2024-03-28T01:05:04.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.775+0000 7f30577fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560866fb7790 0x560866fc8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.775+0000 7f30577fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560866fb7790 0x560866fc8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:56844/0 (socket says 172.21.15.169:56844) 2024-03-28T01:05:04.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.775+0000 7f30577fe640 1 -- 172.21.15.169:0/2557951179 learned_addr learned my addr 172.21.15.169:0/2557951179 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:04.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.776+0000 7f30577fe640 1 -- 172.21.15.169:0/2557951179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fbc900 msgr2=0x560866fc1860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:05:04.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.776+0000 7f30577fe640 1 --2- 172.21.15.169:0/2557951179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fbc900 0x560866fc1860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.776+0000 7f30577fe640 1 -- 172.21.15.169:0/2557951179 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560866fb6da0 msgr2=0x560866fc1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:04.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.776+0000 7f30577fe640 1 --2- 172.21.15.169:0/2557951179 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560866fb6da0 0x560866fc1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.776+0000 7f30577fe640 1 -- 172.21.15.169:0/2557951179 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560866fb3080 con 0x560866fb7790 2024-03-28T01:05:04.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.776+0000 7f30577fe640 1 --2- 172.21.15.169:0/2557951179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560866fb7790 0x560866fc8860 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3040005f90 tx=0x7f3040005fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b5dba021f4a1b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:04.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.777+0000 7f304dffb640 1 -- 172.21.15.169:0/2557951179 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040013250 con 0x560866fb7790 2024-03-28T01:05:04.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.777+0000 7f304dffb640 1 -- 172.21.15.169:0/2557951179 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3040005520 con 0x560866fb7790 2024-03-28T01:05:04.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.778+0000 7f30615b4740 1 -- 172.21.15.169:0/2557951179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560866fb7790 msgr2=0x560866fc8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:04.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.778+0000 7f30615b4740 1 --2- 172.21.15.169:0/2557951179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560866fb7790 0x560866fc8860 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3040005f90 tx=0x7f3040005fc0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.782+0000 7f30615b4740 1 -- 172.21.15.169:0/2557951179 shutdown_connections 2024-03-28T01:05:04.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.782+0000 7f30615b4740 1 --2- 172.21.15.169:0/2557951179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fbc900 0x560866fc1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.782+0000 7f30615b4740 1 --2- 172.21.15.169:0/2557951179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560866fb7790 0x560866fc8860 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.782+0000 7f30615b4740 1 --2- 172.21.15.169:0/2557951179 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560866fb6da0 0x560866fc1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.782+0000 7f30615b4740 1 -- 172.21.15.169:0/2557951179 >> 172.21.15.169:0/2557951179 conn(0x560866fb2310 msgr2=0x560866fb26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:04.789 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.787+0000 7f30615b4740 1 -- 172.21.15.169:0/2557951179 shutdown_connections 2024-03-28T01:05:04.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.789+0000 7f30615b4740 1 -- 172.21.15.169:0/2557951179 wait complete. 2024-03-28T01:05:04.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.790+0000 7f30615b4740 1 -- start start 2024-03-28T01:05:04.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.790+0000 7f30615b4740 1 Processor -- start 2024-03-28T01:05:04.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.793+0000 7f30615b4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fb6da0 0x560866fc3b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.793+0000 7f30615b4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560866fb7790 0x560866fc4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.793+0000 7f30615b4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560866fbc900 0x560866fc7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.793+0000 7f30615b4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30581a65c0 con 0x560866fb7790 2024-03-28T01:05:04.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.793+0000 7f30615b4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30581a7890 con 0x560866fbc900 2024-03-28T01:05:04.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.793+0000 7f30615b4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30581a6bd0 con 0x560866fb6da0 2024-03-28T01:05:04.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.793+0000 7f30615b4740 10 client.0 _ll_register_callbacks cb 0x7f3050009260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:04.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.794+0000 7f3057fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fb6da0 0x560866fc3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.794+0000 7f3057fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fb6da0 0x560866fc3b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:49062/0 (socket says 172.21.15.169:49062) 2024-03-28T01:05:04.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.794+0000 7f3057fff640 1 -- 172.21.15.169:0/3078160723 learned_addr learned my addr 172.21.15.169:0/3078160723 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:04.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.794+0000 7f305ce87640 1 --2- 172.21.15.169:0/3078160723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560866fbc900 0x560866fc7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.794+0000 7f3057fff640 1 -- 172.21.15.169:0/3078160723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560866fbc900 msgr2=0x560866fc7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:04.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.794+0000 7f3057fff640 1 --2- 172.21.15.169:0/3078160723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560866fbc900 0x560866fc7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.794+0000 7f3057fff640 1 -- 172.21.15.169:0/3078160723 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560866fb7790 msgr2=0x560866fc4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:04.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.794+0000 7f3057fff640 1 --2- 172.21.15.169:0/3078160723 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560866fb7790 0x560866fc4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.795+0000 7f3057fff640 1 -- 172.21.15.169:0/3078160723 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560866fc7610 con 0x560866fb6da0 2024-03-28T01:05:04.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.795+0000 7f3057fff640 1 --2- 172.21.15.169:0/3078160723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fb6da0 0x560866fc3b10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3048003990 tx=0x7f304801db10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d52ec69e0f2baed5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:04.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.797+0000 7f30377fe640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:05:04.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.797+0000 7f30377fe640 1 -- 172.21.15.169:0/3078160723 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304801d020 con 0x560866fb6da0 2024-03-28T01:05:04.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.797+0000 7f30615b4740 1 -- 172.21.15.169:0/3078160723 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3040005520 con 0x560866fb6da0 2024-03-28T01:05:04.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.798+0000 7f30377fe640 1 -- 172.21.15.169:0/3078160723 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f304801d1c0 con 0x560866fb6da0 2024-03-28T01:05:04.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.798+0000 7f30377fe640 1 -- 172.21.15.169:0/3078160723 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3048029050 con 0x560866fb6da0 2024-03-28T01:05:04.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.798+0000 7f30615b4740 10 client.26950 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:04.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.799+0000 7f30615b4740 1 -- 172.21.15.169:0/3078160723 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x560866fc8a20 con 0x560866fb6da0 2024-03-28T01:05:04.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.799+0000 7f30377fe640 1 -- 172.21.15.169:0/3078160723 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3040005520 con 0x560866fb6da0 2024-03-28T01:05:04.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.799+0000 7f30377fe640 1 -- 172.21.15.169:0/3078160723 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f3048011910 con 0x560866fb6da0 2024-03-28T01:05:04.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.801+0000 7f30615b4740 10 client.26950 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:04.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.801+0000 7f30615b4740 10 client.26950 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:04.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.801+0000 7f30615b4740 20 client.26950 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:04.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.801+0000 7f30615b4740 1 --2- 172.21.15.169:0/3078160723 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56086704af90 0x56086704d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:04.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.801+0000 7f30615b4740 4 client.26950 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:05:04.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.801+0000 7f30615b4740 1 -- 172.21.15.169:0/3078160723 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560866f23c60 con 0x56086704af90 2024-03-28T01:05:04.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.803+0000 7f30577fe640 1 --2- 172.21.15.169:0/3078160723 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56086704af90 0x56086704d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:04.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.808+0000 7f30377fe640 10 client.26950 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:04.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.808+0000 7f30577fe640 1 --2- 172.21.15.169:0/3078160723 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56086704af90 0x56086704d380 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=474038c4f59e87a1 server_cookie=3dcbaeb11746167f in_seq=0 out_seq=0 2024-03-28T01:05:04.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.809+0000 7f30377fe640 1 -- 172.21.15.169:0/3078160723 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x560866f23c60 con 0x56086704af90 2024-03-28T01:05:04.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.809+0000 7f30377fe640 10 client.26950 handle_command_reply: tid=0 2024-03-28T01:05:04.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.809+0000 7f30615b4740 1 client.26950 shutdown 2024-03-28T01:05:04.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.822+0000 7f30615b4740 1 -- 172.21.15.169:0/3078160723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fb6da0 msgr2=0x560866fc3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:04.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.822+0000 7f30615b4740 1 --2- 172.21.15.169:0/3078160723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fb6da0 0x560866fc3b10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3048003990 tx=0x7f304801db10 comp rx=0 tx=0).stop 2024-03-28T01:05:04.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.822+0000 7f30615b4740 1 -- 172.21.15.169:0/3078160723 shutdown_connections 2024-03-28T01:05:04.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.822+0000 7f30615b4740 1 --2- 172.21.15.169:0/3078160723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560866fbc900 0x560866fc7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.822+0000 7f30615b4740 1 --2- 172.21.15.169:0/3078160723 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56086704af90 0x56086704d380 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.822+0000 7f30615b4740 1 --2- 172.21.15.169:0/3078160723 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560866fb7790 0x560866fc4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.822+0000 7f30615b4740 1 --2- 172.21.15.169:0/3078160723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560866fb6da0 0x560866fc3b10 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.822+0000 7f30615b4740 1 -- 172.21.15.169:0/3078160723 >> 172.21.15.169:0/3078160723 conn(0x560866fb2310 msgr2=0x560866fb8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:04.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.822+0000 7f30377fe640 0 client.26950 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:04.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.824+0000 7f30615b4740 1 -- 172.21.15.169:0/3078160723 shutdown_connections 2024-03-28T01:05:04.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.824+0000 7f30615b4740 1 -- 172.21.15.169:0/3078160723 wait complete. 2024-03-28T01:05:04.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.824+0000 7f30615b4740 20 client.26950 trim_cache size 0 max 16384 2024-03-28T01:05:04.828 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:05:04.828 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "scrub active (0 inodes in the stack)", 2024-03-28T01:05:04.828 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": { 2024-03-28T01:05:04.828 INFO:teuthology.orchestra.run.smithi169.stdout: "467d0b78-47ea-435a-910d-2fe6c4a39e08": { 2024-03-28T01:05:04.828 INFO:teuthology.orchestra.run.smithi169.stdout: "path": "/", 2024-03-28T01:05:04.829 INFO:teuthology.orchestra.run.smithi169.stdout: "tag": "467d0b78-47ea-435a-910d-2fe6c4a39e08", 2024-03-28T01:05:04.829 INFO:teuthology.orchestra.run.smithi169.stdout: "options": "recursive,force" 2024-03-28T01:05:04.829 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:05:04.829 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:05:04.829 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:05:04.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.827+0000 7f305e8ea640 1 -- 172.21.15.169:0/4084985974 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f30440a2a00 msgr2=0x7f30440a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:04.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.827+0000 7f305e8ea640 1 --2- 172.21.15.169:0/4084985974 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f30440a2a00 0x7f30440a4ec0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3040010740 tx=0x7f304000e000 comp rx=0 tx=0).stop 2024-03-28T01:05:04.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.827+0000 7f305e8ea640 1 -- 172.21.15.169:0/4084985974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3058101080 msgr2=0x7f30581a5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:04.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.827+0000 7f305e8ea640 1 --2- 172.21.15.169:0/4084985974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3058101080 0x7f30581a5530 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f305002efd0 tx=0x7f3050004f00 comp rx=0 tx=0).stop 2024-03-28T01:05:04.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.828+0000 7f305e8ea640 1 -- 172.21.15.169:0/4084985974 shutdown_connections 2024-03-28T01:05:04.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.828+0000 7f305e8ea640 1 --2- 172.21.15.169:0/4084985974 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f30440a2a00 0x7f30440a4ec0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.828+0000 7f305e8ea640 1 --2- 172.21.15.169:0/4084985974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3058101080 0x7f30581a5530 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.828+0000 7f305e8ea640 1 --2- 172.21.15.169:0/4084985974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30581006c0 0x7f305819e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.828+0000 7f305e8ea640 1 --2- 172.21.15.169:0/4084985974 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30580ff960 0x7f305819dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:04.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.828+0000 7f305e8ea640 1 -- 172.21.15.169:0/4084985974 >> 172.21.15.169:0/4084985974 conn(0x7f30580fb290 msgr2=0x7f30581018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:04.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.831+0000 7f305e8ea640 1 -- 172.21.15.169:0/4084985974 shutdown_connections 2024-03-28T01:05:04.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:04.832+0000 7f305e8ea640 1 -- 172.21.15.169:0/4084985974 wait complete. 2024-03-28T01:05:04.857 INFO:tasks.cephfs.filesystem:scrub status for tag:467d0b78-47ea-435a-910d-2fe6c4a39e08 - {'path': '/', 'tag': '467d0b78-47ea-435a-910d-2fe6c4a39e08', 'options': 'recursive,force'} 2024-03-28T01:05:05.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:05 smithi177 ceph-mon[36206]: scrub queued for path: / 2024-03-28T01:05:05.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:05 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:05:05.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:05 smithi177 ceph-mon[36206]: scrub summary: active paths [/] 2024-03-28T01:05:05.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:05 smithi177 ceph-mon[36206]: pgmap v608: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s rd, 440 MiB/s wr, 3.79k op/s 2024-03-28T01:05:05.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:05 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:05:06.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:05 smithi169 ceph-mon[32274]: scrub queued for path: / 2024-03-28T01:05:06.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:05 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:05:06.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:05 smithi169 ceph-mon[32274]: scrub summary: active paths [/] 2024-03-28T01:05:06.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:05 smithi169 ceph-mon[32274]: pgmap v608: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s rd, 440 MiB/s wr, 3.79k op/s 2024-03-28T01:05:06.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:05 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:05:06.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:05 smithi174 ceph-mon[36060]: scrub queued for path: / 2024-03-28T01:05:06.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:05 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:05:06.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:05 smithi174 ceph-mon[36060]: scrub summary: active paths [/] 2024-03-28T01:05:06.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:05 smithi174 ceph-mon[36060]: pgmap v608: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s rd, 440 MiB/s wr, 3.79k op/s 2024-03-28T01:05:06.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:05 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:05:07.050 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.044+0000 7f30c84ac640 1 -- 172.21.15.169:0/1374858039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f30c0100600 msgr2=0x7f30c0100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:07.050 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.044+0000 7f30c84ac640 1 --2- 172.21.15.169:0/1374858039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f30c0100600 0x7f30c0100a20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f30bc009900 tx=0x7f30bc02bc30 comp rx=0 tx=0).stop 2024-03-28T01:05:07.050 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.044+0000 7f30c84ac640 1 -- 172.21.15.169:0/1374858039 shutdown_connections 2024-03-28T01:05:07.050 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.044+0000 7f30c84ac640 1 --2- 172.21.15.169:0/1374858039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30c01021b0 0x7f30c0108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.050 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.044+0000 7f30c84ac640 1 --2- 172.21.15.169:0/1374858039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30c01017f0 0x7f30c0101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.050 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.044+0000 7f30c84ac640 1 --2- 172.21.15.169:0/1374858039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f30c0100600 0x7f30c0100a20 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.050 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.044+0000 7f30c84ac640 1 -- 172.21.15.169:0/1374858039 >> 172.21.15.169:0/1374858039 conn(0x7f30c00fbb70 msgr2=0x7f30c00fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:07.050 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.044+0000 7f30c84ac640 1 -- 172.21.15.169:0/1374858039 shutdown_connections 2024-03-28T01:05:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.044+0000 7f30c84ac640 1 -- 172.21.15.169:0/1374858039 wait complete. 2024-03-28T01:05:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.045+0000 7f30c84ac640 1 Processor -- start 2024-03-28T01:05:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.045+0000 7f30c84ac640 1 -- start start 2024-03-28T01:05:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.045+0000 7f30c84ac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30c0100600 0x7f30c01979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.045+0000 7f30c84ac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f30c01017f0 0x7f30c0197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.045+0000 7f30c84ac640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30c01021b0 0x7f30c019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.045+0000 7f30c84ac640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c010c2d0 con 0x7f30c01017f0 2024-03-28T01:05:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.045+0000 7f30c84ac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c010c150 con 0x7f30c0100600 2024-03-28T01:05:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.045+0000 7f30c84ac640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c010c450 con 0x7f30c01021b0 2024-03-28T01:05:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.045+0000 7f30c5a47640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f30c01017f0 0x7f30c0197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.045+0000 7f30c6248640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30c0100600 0x7f30c01979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.045+0000 7f30c5a47640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f30c01017f0 0x7f30c0197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39078/0 (socket says 172.21.15.169:39078) 2024-03-28T01:05:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.045+0000 7f30c5a47640 1 -- 172.21.15.169:0/42554357 learned_addr learned my addr 172.21.15.169:0/42554357 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.045+0000 7f30c6a49640 1 --2- 172.21.15.169:0/42554357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30c01021b0 0x7f30c019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.045+0000 7f30c5a47640 1 -- 172.21.15.169:0/42554357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30c01021b0 msgr2=0x7f30c019ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.045+0000 7f30c5a47640 1 --2- 172.21.15.169:0/42554357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30c01021b0 0x7f30c019ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.045+0000 7f30c5a47640 1 -- 172.21.15.169:0/42554357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30c0100600 msgr2=0x7f30c01979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.045+0000 7f30c5a47640 1 --2- 172.21.15.169:0/42554357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30c0100600 0x7f30c01979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.045+0000 7f30c5a47640 1 -- 172.21.15.169:0/42554357 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c019f5f0 con 0x7f30c01017f0 2024-03-28T01:05:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.045+0000 7f30c6248640 1 --2- 172.21.15.169:0/42554357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30c0100600 0x7f30c01979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:05:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.045+0000 7f30c6a49640 1 --2- 172.21.15.169:0/42554357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30c01021b0 0x7f30c019ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:05:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.045+0000 7f30c5a47640 1 --2- 172.21.15.169:0/42554357 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f30c01017f0 0x7f30c0197f10 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f30b000e9f0 tx=0x7f30b000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.046+0000 7f30b77fe640 1 -- 172.21.15.169:0/42554357 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b000cd80 con 0x7f30c01017f0 2024-03-28T01:05:07.053 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.046+0000 7f30b77fe640 1 -- 172.21.15.169:0/42554357 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30b0005500 con 0x7f30c01017f0 2024-03-28T01:05:07.053 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.046+0000 7f30b77fe640 1 -- 172.21.15.169:0/42554357 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b0010690 con 0x7f30c01017f0 2024-03-28T01:05:07.053 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.046+0000 7f30c84ac640 1 -- 172.21.15.169:0/42554357 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c019f8e0 con 0x7f30c01017f0 2024-03-28T01:05:07.053 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.046+0000 7f30c84ac640 1 -- 172.21.15.169:0/42554357 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c019fde0 con 0x7f30c01017f0 2024-03-28T01:05:07.059 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.048+0000 7f30cb172740 1 Processor -- start 2024-03-28T01:05:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.049+0000 7f30cb172740 1 -- start start 2024-03-28T01:05:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.049+0000 7f30cb172740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a527e920 0x5652a527ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.049+0000 7f30cb172740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527f310 0x5652a5283e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.049+0000 7f30cb172740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a5284480 0x5652a5286910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.049+0000 7f30cb172740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652a51e3b10 con 0x5652a527f310 2024-03-28T01:05:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.049+0000 7f30cb172740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652a518c900 con 0x5652a5284480 2024-03-28T01:05:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.049+0000 7f30cb172740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652a51dfbb0 con 0x5652a527e920 2024-03-28T01:05:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.049+0000 7f30b77fe640 1 -- 172.21.15.169:0/42554357 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f30b00046c0 con 0x7f30c01017f0 2024-03-28T01:05:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.049+0000 7f30c6a49640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a527e920 0x5652a527ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.049+0000 7f30c6a49640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a527e920 0x5652a527ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:46440/0 (socket says 172.21.15.169:46440) 2024-03-28T01:05:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.049+0000 7f30c6248640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527f310 0x5652a5283e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.049+0000 7f30c6a49640 1 -- 172.21.15.169:0/2361078844 learned_addr learned my addr 172.21.15.169:0/2361078844 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.049+0000 7f30c5a47640 1 --2- 172.21.15.169:0/2361078844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a5284480 0x5652a5286910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.049+0000 7f30c6a49640 1 -- 172.21.15.169:0/2361078844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a5284480 msgr2=0x5652a5286910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.049+0000 7f30c6a49640 1 --2- 172.21.15.169:0/2361078844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a5284480 0x5652a5286910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.049+0000 7f30c6a49640 1 -- 172.21.15.169:0/2361078844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527f310 msgr2=0x5652a5283e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.049+0000 7f30c6a49640 1 --2- 172.21.15.169:0/2361078844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527f310 0x5652a5283e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.049+0000 7f30c6a49640 1 -- 172.21.15.169:0/2361078844 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652a4ef8210 con 0x5652a527e920 2024-03-28T01:05:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.050+0000 7f30b77fe640 1 --2- 172.21.15.169:0/42554357 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f30ac0a2920 0x7f30ac0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.050+0000 7f30c6a49640 1 --2- 172.21.15.169:0/2361078844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a527e920 0x5652a527ed40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f30b80117e0 tx=0x7f30b8011ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37303a4ba1c6383b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.050+0000 7f30c6248640 1 --2- 172.21.15.169:0/42554357 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f30ac0a2920 0x7f30ac0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:07.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.050+0000 7f30b4bfe640 1 -- 172.21.15.169:0/2361078844 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8011310 con 0x5652a527e920 2024-03-28T01:05:07.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.050+0000 7f30b77fe640 1 -- 172.21.15.169:0/42554357 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f30b00cf1e0 con 0x7f30c01017f0 2024-03-28T01:05:07.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.050+0000 7f30b4bfe640 1 -- 172.21.15.169:0/2361078844 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30b8003e50 con 0x5652a527e920 2024-03-28T01:05:07.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.050+0000 7f30b4bfe640 1 -- 172.21.15.169:0/2361078844 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8004590 con 0x5652a527e920 2024-03-28T01:05:07.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.051+0000 7f30c6248640 1 --2- 172.21.15.169:0/42554357 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f30ac0a2920 0x7f30ac0a4de0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f30bc02c170 tx=0x7f30bc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:07.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.051+0000 7f30cb172740 1 -- 172.21.15.169:0/2361078844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a527e920 msgr2=0x5652a527ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:07.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.051+0000 7f30cb172740 1 --2- 172.21.15.169:0/2361078844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a527e920 0x5652a527ed40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f30b80117e0 tx=0x7f30b8011ca0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.051+0000 7f30cb172740 1 -- 172.21.15.169:0/2361078844 shutdown_connections 2024-03-28T01:05:07.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.051+0000 7f30cb172740 1 --2- 172.21.15.169:0/2361078844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a5284480 0x5652a5286910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.051+0000 7f30cb172740 1 --2- 172.21.15.169:0/2361078844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527f310 0x5652a5283e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.062 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.051+0000 7f30cb172740 1 --2- 172.21.15.169:0/2361078844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a527e920 0x5652a527ed40 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.051+0000 7f30cb172740 1 -- 172.21.15.169:0/2361078844 >> 172.21.15.169:0/2361078844 conn(0x5652a5279e90 msgr2=0x5652a527a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.051+0000 7f30cb172740 1 -- 172.21.15.169:0/2361078844 shutdown_connections 2024-03-28T01:05:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.051+0000 7f30cb172740 1 -- 172.21.15.169:0/2361078844 wait complete. 2024-03-28T01:05:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.051+0000 7f30cb172740 1 -- start start 2024-03-28T01:05:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.051+0000 7f30cb172740 1 Processor -- start 2024-03-28T01:05:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.051+0000 7f30cb172740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527e920 0x5652a528d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.051+0000 7f30cb172740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a527f310 0x5652a528d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30cb172740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a5284480 0x5652a5294640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30cb172740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652a528b7b0 con 0x5652a527e920 2024-03-28T01:05:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30cb172740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652a519fd50 con 0x5652a527f310 2024-03-28T01:05:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30cb172740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652a528b930 con 0x5652a5284480 2024-03-28T01:05:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30cb172740 10 client.0 _ll_register_callbacks cb 0x5652a519f720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:07.064 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30c5a47640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a527f310 0x5652a528d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:07.064 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30c6a49640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a5284480 0x5652a5294640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:07.064 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30c5a47640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a527f310 0x5652a528d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:50088/0 (socket says 172.21.15.169:50088) 2024-03-28T01:05:07.064 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30c6248640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527e920 0x5652a528d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:07.064 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30c5a47640 1 -- 172.21.15.169:0/771503567 learned_addr learned my addr 172.21.15.169:0/771503567 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:07.064 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30c5a47640 1 -- 172.21.15.169:0/771503567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a5284480 msgr2=0x5652a5294640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:07.064 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30c5a47640 1 --2- 172.21.15.169:0/771503567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a5284480 0x5652a5294640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.064 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30c5a47640 1 -- 172.21.15.169:0/771503567 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527e920 msgr2=0x5652a528d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:07.064 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30c5a47640 1 --2- 172.21.15.169:0/771503567 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527e920 0x5652a528d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.064 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30c5a47640 1 -- 172.21.15.169:0/771503567 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652a4fc4620 con 0x5652a527f310 2024-03-28T01:05:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30c5a47640 1 --2- 172.21.15.169:0/771503567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a527f310 0x5652a528d600 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f30b00a3a50 tx=0x7f30b0004ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2aae487885774b69 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:05:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30a9ffb640 1 -- 172.21.15.169:0/771503567 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b009b320 con 0x5652a527f310 2024-03-28T01:05:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30a9ffb640 1 -- 172.21.15.169:0/771503567 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f30b00860b0 con 0x5652a527f310 2024-03-28T01:05:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30cb172740 1 -- 172.21.15.169:0/771503567 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f30b8003e50 con 0x5652a527f310 2024-03-28T01:05:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30a9ffb640 1 -- 172.21.15.169:0/771503567 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30b00056b0 con 0x5652a527f310 2024-03-28T01:05:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30cb172740 10 client.27017 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.052+0000 7f30cb172740 1 -- 172.21.15.169:0/771503567 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5652a5297a70 con 0x5652a527f310 2024-03-28T01:05:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.053+0000 7f30a9ffb640 1 -- 172.21.15.169:0/771503567 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30b8003e50 con 0x5652a527f310 2024-03-28T01:05:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.053+0000 7f30a9ffb640 1 -- 172.21.15.169:0/771503567 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f30b0097e50 con 0x5652a527f310 2024-03-28T01:05:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.053+0000 7f30cb172740 10 client.27017 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:07.065 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.053+0000 7f30cb172740 10 client.27017 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:07.066 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.053+0000 7f30cb172740 20 client.27017 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:07.066 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.053+0000 7f30cb172740 1 --2- 172.21.15.169:0/771503567 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5652a5299000 0x5652a529b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:07.066 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.053+0000 7f30cb172740 4 client.27017 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:05:07.066 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.053+0000 7f30cb172740 1 -- 172.21.15.169:0/771503567 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5652a51f1050 con 0x5652a5299000 2024-03-28T01:05:07.066 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.053+0000 7f30c6248640 1 --2- 172.21.15.169:0/771503567 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5652a5299000 0x5652a529b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:07.066 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.053+0000 7f30a9ffb640 10 client.27017 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:07.066 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.053+0000 7f30c6248640 1 --2- 172.21.15.169:0/771503567 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5652a5299000 0x5652a529b3f0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb4c0c2d6e3a9125 server_cookie=3b51f2f91506a8c5 in_seq=0 out_seq=0 2024-03-28T01:05:07.066 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.054+0000 7f30a9ffb640 1 -- 172.21.15.169:0/771503567 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5652a51f1050 con 0x5652a5299000 2024-03-28T01:05:07.066 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.054+0000 7f30a9ffb640 10 client.27017 handle_command_reply: tid=0 2024-03-28T01:05:07.066 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.054+0000 7f30cb172740 1 client.27017 shutdown 2024-03-28T01:05:07.066 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.055+0000 7f30cb172740 1 -- 172.21.15.169:0/771503567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a527f310 msgr2=0x5652a528d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:07.067 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.055+0000 7f30cb172740 1 --2- 172.21.15.169:0/771503567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a527f310 0x5652a528d600 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f30b00a3a50 tx=0x7f30b0004ba0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.067 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.055+0000 7f30cb172740 1 -- 172.21.15.169:0/771503567 shutdown_connections 2024-03-28T01:05:07.067 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.055+0000 7f30cb172740 1 --2- 172.21.15.169:0/771503567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a5284480 0x5652a5294640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.067 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.055+0000 7f30cb172740 1 --2- 172.21.15.169:0/771503567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a527f310 0x5652a528d600 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.067 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.055+0000 7f30cb172740 1 --2- 172.21.15.169:0/771503567 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5652a5299000 0x5652a529b3f0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.067 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.055+0000 7f30cb172740 1 --2- 172.21.15.169:0/771503567 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527e920 0x5652a528d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.067 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.055+0000 7f30a9ffb640 0 client.27017 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:07.067 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.055+0000 7f30cb172740 1 -- 172.21.15.169:0/771503567 >> 172.21.15.169:0/771503567 conn(0x5652a5279e90 msgr2=0x5652a5279020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:07.067 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.055+0000 7f30cb172740 1 -- 172.21.15.169:0/771503567 shutdown_connections 2024-03-28T01:05:07.067 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.055+0000 7f30cb172740 1 -- 172.21.15.169:0/771503567 wait complete. 2024-03-28T01:05:07.067 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.055+0000 7f30cb172740 20 client.27017 trim_cache size 0 max 16384 2024-03-28T01:05:07.077 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.069+0000 7f30cb172740 1 Processor -- start 2024-03-28T01:05:07.077 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.069+0000 7f30cb172740 1 -- start start 2024-03-28T01:05:07.078 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.069+0000 7f30cb172740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527e920 0x5652a5286870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:07.078 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.069+0000 7f30cb172740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a527f310 0x5652a5294600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:07.078 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.069+0000 7f30cb172740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a5284480 0x5652a5280710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:07.078 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.069+0000 7f30cb172740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c01aa560 con 0x5652a527e920 2024-03-28T01:05:07.078 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.069+0000 7f30cb172740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c01a8060 con 0x5652a5284480 2024-03-28T01:05:07.078 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.069+0000 7f30cb172740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c01a6100 con 0x5652a527f310 2024-03-28T01:05:07.078 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.069+0000 7f30c5a47640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a527f310 0x5652a5294600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:07.078 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.069+0000 7f30c6248640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527e920 0x5652a5286870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:07.078 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.069+0000 7f30c6a49640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a5284480 0x5652a5280710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:07.078 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.069+0000 7f30c6a49640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a5284480 0x5652a5280710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:50100/0 (socket says 172.21.15.169:50100) 2024-03-28T01:05:07.078 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.069+0000 7f30c6a49640 1 -- 172.21.15.169:0/2160355371 learned_addr learned my addr 172.21.15.169:0/2160355371 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:07.079 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.069+0000 7f30c5a47640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a527f310 0x5652a5294600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:46456/0 (socket says 172.21.15.169:46456) 2024-03-28T01:05:07.079 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.069+0000 7f30c5a47640 1 -- 172.21.15.169:0/2160355371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a5284480 msgr2=0x5652a5280710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:07.079 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.069+0000 7f30c5a47640 1 --2- 172.21.15.169:0/2160355371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a5284480 0x5652a5280710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.079 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.069+0000 7f30c5a47640 1 -- 172.21.15.169:0/2160355371 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527e920 msgr2=0x5652a5286870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:07.079 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.069+0000 7f30c5a47640 1 --2- 172.21.15.169:0/2160355371 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527e920 0x5652a5286870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.079 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.069+0000 7f30c5a47640 1 -- 172.21.15.169:0/2160355371 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652a52839b0 con 0x5652a527f310 2024-03-28T01:05:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.069+0000 7f30c6a49640 1 --2- 172.21.15.169:0/2160355371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a5284480 0x5652a5280710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:05:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.070+0000 7f30c5a47640 1 --2- 172.21.15.169:0/2160355371 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a527f310 0x5652a5294600 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f30b000c6a0 tx=0x7f30b0099cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fc86aa3be832ed9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.070+0000 7f30abfff640 1 -- 172.21.15.169:0/2160355371 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b009b140 con 0x5652a527f310 2024-03-28T01:05:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.070+0000 7f30abfff640 1 -- 172.21.15.169:0/2160355371 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30b009ac30 con 0x5652a527f310 2024-03-28T01:05:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.070+0000 7f30abfff640 1 -- 172.21.15.169:0/2160355371 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b009f930 con 0x5652a527f310 2024-03-28T01:05:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30cb172740 1 -- 172.21.15.169:0/2160355371 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a527f310 msgr2=0x5652a5294600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30cb172740 1 --2- 172.21.15.169:0/2160355371 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a527f310 0x5652a5294600 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f30b000c6a0 tx=0x7f30b0099cf0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30cb172740 1 -- 172.21.15.169:0/2160355371 shutdown_connections 2024-03-28T01:05:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30cb172740 1 --2- 172.21.15.169:0/2160355371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a5284480 0x5652a5280710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30cb172740 1 --2- 172.21.15.169:0/2160355371 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a527f310 0x5652a5294600 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30cb172740 1 --2- 172.21.15.169:0/2160355371 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527e920 0x5652a5286870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30cb172740 1 -- 172.21.15.169:0/2160355371 >> 172.21.15.169:0/2160355371 conn(0x5652a5279e90 msgr2=0x5652a527a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30cb172740 1 -- 172.21.15.169:0/2160355371 shutdown_connections 2024-03-28T01:05:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30cb172740 1 -- 172.21.15.169:0/2160355371 wait complete. 2024-03-28T01:05:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30cb172740 1 -- start start 2024-03-28T01:05:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30cb172740 1 Processor -- start 2024-03-28T01:05:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30cb172740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527e920 0x5652a5291ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30cb172740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a527f310 0x5652a5292420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30cb172740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a5284480 0x5652a52933c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30cb172740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c01a6700 con 0x5652a527e920 2024-03-28T01:05:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30cb172740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c01a7a00 con 0x5652a527f310 2024-03-28T01:05:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30cb172740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c01a6d40 con 0x5652a5284480 2024-03-28T01:05:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30cb172740 10 client.0 _ll_register_callbacks cb 0x7f30b000cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30c6248640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527e920 0x5652a5291ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30c5a47640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a527f310 0x5652a5292420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30c6248640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527e920 0x5652a5291ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39126/0 (socket says 172.21.15.169:39126) 2024-03-28T01:05:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30c5a47640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a527f310 0x5652a5292420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:50112/0 (socket says 172.21.15.169:50112) 2024-03-28T01:05:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30c6248640 1 -- 172.21.15.169:0/1870433104 learned_addr learned my addr 172.21.15.169:0/1870433104 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.071+0000 7f30c6a49640 1 --2- 172.21.15.169:0/1870433104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a5284480 0x5652a52933c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.072+0000 7f30c6248640 1 -- 172.21.15.169:0/1870433104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a5284480 msgr2=0x5652a52933c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.072+0000 7f30c6248640 1 --2- 172.21.15.169:0/1870433104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a5284480 0x5652a52933c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.072+0000 7f30c6248640 1 -- 172.21.15.169:0/1870433104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a527f310 msgr2=0x5652a5292420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.072+0000 7f30c6248640 1 --2- 172.21.15.169:0/1870433104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a527f310 0x5652a5292420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.072+0000 7f30c6248640 1 -- 172.21.15.169:0/1870433104 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652a52939b0 con 0x5652a527e920 2024-03-28T01:05:07.083 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.072+0000 7f30c5a47640 1 --2- 172.21.15.169:0/1870433104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a527f310 0x5652a5292420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:05:07.083 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.072+0000 7f30c6a49640 1 --2- 172.21.15.169:0/1870433104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a5284480 0x5652a52933c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:05:07.083 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.072+0000 7f30c6248640 1 --2- 172.21.15.169:0/1870433104 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527e920 0x5652a5291ee0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f30bc0096c0 tx=0x7f30bc02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46042c85dffe7d81 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:07.083 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.072+0000 7f30a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:05:07.083 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.072+0000 7f30a9ffb640 1 -- 172.21.15.169:0/1870433104 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc046070 con 0x5652a527e920 2024-03-28T01:05:07.083 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.072+0000 7f30a9ffb640 1 -- 172.21.15.169:0/1870433104 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f30bc004780 con 0x5652a527e920 2024-03-28T01:05:07.083 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.072+0000 7f30cb172740 1 -- 172.21.15.169:0/1870433104 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5652a51eb3a0 con 0x5652a527e920 2024-03-28T01:05:07.083 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.072+0000 7f30a9ffb640 1 -- 172.21.15.169:0/1870433104 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30bc06a050 con 0x5652a527e920 2024-03-28T01:05:07.083 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.072+0000 7f30cb172740 10 client.17169 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:07.083 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.072+0000 7f30cb172740 1 -- 172.21.15.169:0/1870433104 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5652a52f4a70 con 0x5652a527e920 2024-03-28T01:05:07.083 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.072+0000 7f30a9ffb640 1 -- 172.21.15.169:0/1870433104 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5652a51eb3a0 con 0x5652a527e920 2024-03-28T01:05:07.083 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.073+0000 7f30a9ffb640 1 -- 172.21.15.169:0/1870433104 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f30bc004f00 con 0x5652a527e920 2024-03-28T01:05:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.073+0000 7f30cb172740 10 client.17169 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.073+0000 7f30cb172740 10 client.17169 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.073+0000 7f30cb172740 20 client.17169 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.073+0000 7f30cb172740 1 --2- 172.21.15.169:0/1870433104 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5652a52f5400 0x5652a52f7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.073+0000 7f30cb172740 4 client.17169 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:05:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.073+0000 7f30cb172740 1 -- 172.21.15.169:0/1870433104 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5652a5169710 con 0x5652a52f5400 2024-03-28T01:05:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.073+0000 7f30c5a47640 1 --2- 172.21.15.169:0/1870433104 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5652a52f5400 0x5652a52f7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.073+0000 7f30a9ffb640 10 client.17169 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:07.073+0000 7f30c5a47640 1 --2- 172.21.15.169:0/1870433104 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5652a52f5400 0x5652a52f7820 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b44f5bab3adc0838 server_cookie=229ae12541fbaa7d in_seq=0 out_seq=0 2024-03-28T01:05:07.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:07 smithi177 ceph-mon[36206]: pgmap v609: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 460 MiB/s wr, 3.82k op/s 2024-03-28T01:05:08.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:07 smithi169 ceph-mon[32274]: pgmap v609: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 460 MiB/s wr, 3.82k op/s 2024-03-28T01:05:08.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:07 smithi174 ceph-mon[36060]: pgmap v609: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 460 MiB/s wr, 3.82k op/s 2024-03-28T01:05:09.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:09 smithi177 ceph-mon[36206]: pgmap v610: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 470 MiB/s wr, 3.62k op/s 2024-03-28T01:05:09.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:09 smithi177 ceph-mon[36206]: scrub summary: idle 2024-03-28T01:05:10.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:09 smithi169 ceph-mon[32274]: pgmap v610: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 470 MiB/s wr, 3.62k op/s 2024-03-28T01:05:10.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:09 smithi169 ceph-mon[32274]: scrub summary: idle 2024-03-28T01:05:10.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:09 smithi174 ceph-mon[36060]: pgmap v610: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 470 MiB/s wr, 3.62k op/s 2024-03-28T01:05:10.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:09 smithi174 ceph-mon[36060]: scrub summary: idle 2024-03-28T01:05:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.201+0000 7f30a9ffb640 1 -- 172.21.15.169:0/1870433104 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5652a5169710 con 0x5652a52f5400 2024-03-28T01:05:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.201+0000 7f30a9ffb640 10 client.17169 handle_command_reply: tid=0 2024-03-28T01:05:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.201+0000 7f30cb172740 1 client.17169 shutdown 2024-03-28T01:05:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.201+0000 7f30cb172740 1 -- 172.21.15.169:0/1870433104 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527e920 msgr2=0x5652a5291ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.201+0000 7f30cb172740 1 --2- 172.21.15.169:0/1870433104 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527e920 0x5652a5291ee0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f30bc0096c0 tx=0x7f30bc02bc30 comp rx=0 tx=0).stop 2024-03-28T01:05:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.201+0000 7f30cb172740 1 -- 172.21.15.169:0/1870433104 shutdown_connections 2024-03-28T01:05:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.201+0000 7f30cb172740 1 --2- 172.21.15.169:0/1870433104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5652a5284480 0x5652a52933c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.201+0000 7f30cb172740 1 --2- 172.21.15.169:0/1870433104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652a527f310 0x5652a5292420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.201+0000 7f30cb172740 1 --2- 172.21.15.169:0/1870433104 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5652a52f5400 0x5652a52f7820 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.201+0000 7f30cb172740 1 --2- 172.21.15.169:0/1870433104 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5652a527e920 0x5652a5291ee0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.201+0000 7f30cb172740 1 -- 172.21.15.169:0/1870433104 >> 172.21.15.169:0/1870433104 conn(0x5652a5279e90 msgr2=0x5652a5280710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.201+0000 7f30a9ffb640 0 client.17169 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.202+0000 7f30cb172740 1 -- 172.21.15.169:0/1870433104 shutdown_connections 2024-03-28T01:05:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.202+0000 7f30cb172740 1 -- 172.21.15.169:0/1870433104 wait complete. 2024-03-28T01:05:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.202+0000 7f30cb172740 20 client.17169 trim_cache size 0 max 16384 2024-03-28T01:05:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:05:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:05:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:05:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:05:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.203+0000 7f30c84ac640 1 -- 172.21.15.169:0/42554357 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f30ac0a2920 msgr2=0x7f30ac0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.203+0000 7f30c84ac640 1 --2- 172.21.15.169:0/42554357 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f30ac0a2920 0x7f30ac0a4de0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f30bc02c170 tx=0x7f30bc02df90 comp rx=0 tx=0).stop 2024-03-28T01:05:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.203+0000 7f30c84ac640 1 -- 172.21.15.169:0/42554357 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f30c01017f0 msgr2=0x7f30c0197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.203+0000 7f30c84ac640 1 --2- 172.21.15.169:0/42554357 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f30c01017f0 0x7f30c0197f10 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f30b000e9f0 tx=0x7f30b000edf0 comp rx=0 tx=0).stop 2024-03-28T01:05:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.203+0000 7f30c84ac640 1 -- 172.21.15.169:0/42554357 shutdown_connections 2024-03-28T01:05:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.203+0000 7f30c84ac640 1 --2- 172.21.15.169:0/42554357 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f30ac0a2920 0x7f30ac0a4de0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.203+0000 7f30c84ac640 1 --2- 172.21.15.169:0/42554357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30c01021b0 0x7f30c019ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.203+0000 7f30c84ac640 1 --2- 172.21.15.169:0/42554357 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f30c01017f0 0x7f30c0197f10 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.203+0000 7f30c84ac640 1 --2- 172.21.15.169:0/42554357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30c0100600 0x7f30c01979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.203+0000 7f30c84ac640 1 -- 172.21.15.169:0/42554357 >> 172.21.15.169:0/42554357 conn(0x7f30c00fbb70 msgr2=0x7f30c00fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.204+0000 7f30c84ac640 1 -- 172.21.15.169:0/42554357 shutdown_connections 2024-03-28T01:05:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:10.204+0000 7f30c84ac640 1 -- 172.21.15.169:0/42554357 wait complete. 2024-03-28T01:05:11.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:11 smithi177 ceph-mon[36206]: pgmap v611: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 486 MiB/s wr, 3.25k op/s 2024-03-28T01:05:12.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:11 smithi169 ceph-mon[32274]: pgmap v611: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 486 MiB/s wr, 3.25k op/s 2024-03-28T01:05:12.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:11 smithi174 ceph-mon[36060]: pgmap v611: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 486 MiB/s wr, 3.25k op/s 2024-03-28T01:05:13.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:12 smithi169 ceph-mon[32274]: pgmap v612: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 530 MiB/s wr, 3.12k op/s 2024-03-28T01:05:13.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:12 smithi174 ceph-mon[36060]: pgmap v612: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 530 MiB/s wr, 3.12k op/s 2024-03-28T01:05:13.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:12 smithi177 ceph-mon[36206]: pgmap v612: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 530 MiB/s wr, 3.12k op/s 2024-03-28T01:05:15.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:15 smithi174 ceph-mon[36060]: pgmap v613: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 536 MiB/s wr, 2.85k op/s 2024-03-28T01:05:15.868 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:15 smithi177 ceph-mon[36206]: pgmap v613: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 536 MiB/s wr, 2.85k op/s 2024-03-28T01:05:16.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:15 smithi169 ceph-mon[32274]: pgmap v613: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 536 MiB/s wr, 2.85k op/s 2024-03-28T01:05:17.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:17 smithi174 ceph-mon[36060]: pgmap v614: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 564 MiB/s wr, 2.63k op/s 2024-03-28T01:05:17.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:17 smithi177 ceph-mon[36206]: pgmap v614: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 564 MiB/s wr, 2.63k op/s 2024-03-28T01:05:18.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:17 smithi169 ceph-mon[32274]: pgmap v614: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 564 MiB/s wr, 2.63k op/s 2024-03-28T01:05:19.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:19 smithi177 ceph-mon[36206]: pgmap v615: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 586 MiB/s wr, 2.19k op/s 2024-03-28T01:05:20.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:19 smithi169 ceph-mon[32274]: pgmap v615: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 586 MiB/s wr, 2.19k op/s 2024-03-28T01:05:20.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:19 smithi174 ceph-mon[36060]: pgmap v615: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 586 MiB/s wr, 2.19k op/s 2024-03-28T01:05:21.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:21 smithi177 ceph-mon[36206]: pgmap v616: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 593 MiB/s wr, 1.82k op/s 2024-03-28T01:05:22.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:21 smithi169 ceph-mon[32274]: pgmap v616: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 593 MiB/s wr, 1.82k op/s 2024-03-28T01:05:22.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:21 smithi174 ceph-mon[36060]: pgmap v616: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 593 MiB/s wr, 1.82k op/s 2024-03-28T01:05:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.481+0000 7f2bda088640 1 -- 172.21.15.169:0/136328557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bd40ffd60 msgr2=0x7f2bd4102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.481+0000 7f2bda088640 1 --2- 172.21.15.169:0/136328557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bd40ffd60 0x7f2bd4102190 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2bc4009860 tx=0x7f2bc402bb90 comp rx=0 tx=0).stop 2024-03-28T01:05:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.486+0000 7f2bda088640 1 -- 172.21.15.169:0/136328557 shutdown_connections 2024-03-28T01:05:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.486+0000 7f2bda088640 1 --2- 172.21.15.169:0/136328557 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2bd4105080 0x7f2bd4105500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.486+0000 7f2bda088640 1 --2- 172.21.15.169:0/136328557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bd41026d0 0x7f2bd4104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.486+0000 7f2bda088640 1 --2- 172.21.15.169:0/136328557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bd40ffd60 0x7f2bd4102190 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.486+0000 7f2bda088640 1 -- 172.21.15.169:0/136328557 >> 172.21.15.169:0/136328557 conn(0x7f2bd40fb980 msgr2=0x7f2bd40fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:23.493 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.490+0000 7f2bda088640 1 -- 172.21.15.169:0/136328557 shutdown_connections 2024-03-28T01:05:23.494 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.490+0000 7f2bda088640 1 -- 172.21.15.169:0/136328557 wait complete. 2024-03-28T01:05:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.491+0000 7f2bda088640 1 Processor -- start 2024-03-28T01:05:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.491+0000 7f2bda088640 1 -- start start 2024-03-28T01:05:23.496 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.492+0000 7f2bda088640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bd40ffd60 0x7f2bd419dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.492+0000 7f2bda088640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2bd41026d0 0x7f2bd419e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.492+0000 7f2bda088640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bd4105080 0x7f2bd41a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.492+0000 7f2bd8885640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2bd41026d0 0x7f2bd419e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.492+0000 7f2bd8885640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2bd41026d0 0x7f2bd419e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:59192/0 (socket says 172.21.15.169:59192) 2024-03-28T01:05:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.492+0000 7f2bd8885640 1 -- 172.21.15.169:0/4076723490 learned_addr learned my addr 172.21.15.169:0/4076723490 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.492+0000 7f2bd9086640 1 --2- 172.21.15.169:0/4076723490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bd40ffd60 0x7f2bd419dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.493+0000 7f2bda088640 1 -- 172.21.15.169:0/4076723490 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd41137d0 con 0x7f2bd41026d0 2024-03-28T01:05:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.493+0000 7f2bda088640 1 -- 172.21.15.169:0/4076723490 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd4113650 con 0x7f2bd40ffd60 2024-03-28T01:05:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.493+0000 7f2bda088640 1 -- 172.21.15.169:0/4076723490 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd4113950 con 0x7f2bd4105080 2024-03-28T01:05:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.493+0000 7f2bd9086640 1 -- 172.21.15.169:0/4076723490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bd4105080 msgr2=0x7f2bd41a5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.493+0000 7f2bd9086640 1 --2- 172.21.15.169:0/4076723490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bd4105080 0x7f2bd41a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.493+0000 7f2bd9086640 1 -- 172.21.15.169:0/4076723490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2bd41026d0 msgr2=0x7f2bd419e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.493+0000 7f2bd9086640 1 --2- 172.21.15.169:0/4076723490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2bd41026d0 0x7f2bd419e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.493+0000 7f2bd9086640 1 -- 172.21.15.169:0/4076723490 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd41a5a00 con 0x7f2bd40ffd60 2024-03-28T01:05:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.494+0000 7f2bd9086640 1 --2- 172.21.15.169:0/4076723490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bd40ffd60 0x7f2bd419dde0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2bc402c0d0 tx=0x7f2bc40042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.494+0000 7f2bd8885640 1 --2- 172.21.15.169:0/4076723490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2bd41026d0 0x7f2bd419e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:05:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.499+0000 7f2bca7fc640 1 -- 172.21.15.169:0/4076723490 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4041020 con 0x7f2bd40ffd60 2024-03-28T01:05:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.499+0000 7f2bca7fc640 1 -- 172.21.15.169:0/4076723490 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2bc4032640 con 0x7f2bd40ffd60 2024-03-28T01:05:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.499+0000 7f2bda088640 1 -- 172.21.15.169:0/4076723490 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd41a5c90 con 0x7f2bd40ffd60 2024-03-28T01:05:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.499+0000 7f2bca7fc640 1 -- 172.21.15.169:0/4076723490 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4039040 con 0x7f2bd40ffd60 2024-03-28T01:05:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.499+0000 7f2bda088640 1 -- 172.21.15.169:0/4076723490 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd41a6180 con 0x7f2bd40ffd60 2024-03-28T01:05:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.505+0000 7f2bca7fc640 1 -- 172.21.15.169:0/4076723490 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2bc4044020 con 0x7f2bd40ffd60 2024-03-28T01:05:23.510 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.506+0000 7f2bddfb2740 1 Processor -- start 2024-03-28T01:05:23.510 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.507+0000 7f2bddfb2740 1 -- start start 2024-03-28T01:05:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.507+0000 7f2bddfb2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d87463fb20 0x55d87463ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.507+0000 7f2bddfb2740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d874640510 0x55d874645010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.508+0000 7f2bd9887640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d87463fb20 0x55d87463ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.508+0000 7f2bd9887640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d87463fb20 0x55d87463ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:43062/0 (socket says 172.21.15.169:43062) 2024-03-28T01:05:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.508+0000 7f2bd9887640 1 -- 172.21.15.169:0/584988564 learned_addr learned my addr 172.21.15.169:0/584988564 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.508+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/584988564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d874645680 0x55d874647b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.508+0000 7f2bddfb2740 1 -- 172.21.15.169:0/584988564 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8745a4b10 con 0x55d874640510 2024-03-28T01:05:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.508+0000 7f2bddfb2740 1 -- 172.21.15.169:0/584988564 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d87454d900 con 0x55d87463fb20 2024-03-28T01:05:23.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.508+0000 7f2bddfb2740 1 -- 172.21.15.169:0/584988564 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8745a0bb0 con 0x55d874645680 2024-03-28T01:05:23.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.508+0000 7f2bd8885640 1 --2- 172.21.15.169:0/584988564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d874645680 0x55d874647b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:23.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.508+0000 7f2bd9086640 1 --2- 172.21.15.169:0/584988564 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d874640510 0x55d874645010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:23.513 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.509+0000 7f2bd8885640 1 -- 172.21.15.169:0/584988564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d87463fb20 msgr2=0x55d87463ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.509+0000 7f2bd8885640 1 --2- 172.21.15.169:0/584988564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d87463fb20 0x55d87463ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.509+0000 7f2bd8885640 1 -- 172.21.15.169:0/584988564 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d874640510 msgr2=0x55d874645010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.509+0000 7f2bd8885640 1 --2- 172.21.15.169:0/584988564 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d874640510 0x55d874645010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.509+0000 7f2bd8885640 1 -- 172.21.15.169:0/584988564 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8742b9210 con 0x55d874645680 2024-03-28T01:05:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.509+0000 7f2bd8885640 1 --2- 172.21.15.169:0/584988564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d874645680 0x55d874647b10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc007f80 tx=0x7f2bcc00c750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66fe9d4e901a5615 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.509+0000 7f2bbf7fe640 1 -- 172.21.15.169:0/584988564 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc00d800 con 0x55d874645680 2024-03-28T01:05:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.510+0000 7f2bbf7fe640 1 -- 172.21.15.169:0/584988564 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2bcc0153f0 con 0x55d874645680 2024-03-28T01:05:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.510+0000 7f2bbf7fe640 1 -- 172.21.15.169:0/584988564 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc00a600 con 0x55d874645680 2024-03-28T01:05:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.511+0000 7f2bddfb2740 1 -- 172.21.15.169:0/584988564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d874645680 msgr2=0x55d874647b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.511+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/584988564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d874645680 0x55d874647b10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc007f80 tx=0x7f2bcc00c750 comp rx=0 tx=0).stop 2024-03-28T01:05:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.511+0000 7f2bca7fc640 1 --2- 172.21.15.169:0/4076723490 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2bc00a29c0 0x7f2bc00a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.511+0000 7f2bddfb2740 1 -- 172.21.15.169:0/584988564 shutdown_connections 2024-03-28T01:05:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.511+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/584988564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d874645680 0x55d874647b10 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.511+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/584988564 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d874640510 0x55d874645010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.512+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/584988564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d87463fb20 0x55d87463ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.512+0000 7f2bddfb2740 1 -- 172.21.15.169:0/584988564 >> 172.21.15.169:0/584988564 conn(0x55d87463b090 msgr2=0x55d87463b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.512+0000 7f2bca7fc640 1 -- 172.21.15.169:0/4076723490 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2bc40f1970 con 0x7f2bd40ffd60 2024-03-28T01:05:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.512+0000 7f2bd9887640 1 --2- 172.21.15.169:0/4076723490 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2bc00a29c0 0x7f2bc00a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.517+0000 7f2bddfb2740 1 -- 172.21.15.169:0/584988564 shutdown_connections 2024-03-28T01:05:23.522 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.518+0000 7f2bddfb2740 1 -- 172.21.15.169:0/584988564 wait complete. 2024-03-28T01:05:23.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.519+0000 7f2bddfb2740 1 -- start start 2024-03-28T01:05:23.523 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.519+0000 7f2bddfb2740 1 Processor -- start 2024-03-28T01:05:23.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.520+0000 7f2bddfb2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d87463fb20 0x55d874649df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:23.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.520+0000 7f2bddfb2740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d874640510 0x55d87464a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:23.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.520+0000 7f2bd9086640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d87463fb20 0x55d874649df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:23.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.520+0000 7f2bd9086640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d87463fb20 0x55d874649df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:43074/0 (socket says 172.21.15.169:43074) 2024-03-28T01:05:23.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.521+0000 7f2bd9086640 1 -- 172.21.15.169:0/1296228013 learned_addr learned my addr 172.21.15.169:0/1296228013 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:23.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.521+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/1296228013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d874645680 0x55d874651370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:23.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.521+0000 7f2bddfb2740 1 -- 172.21.15.169:0/1296228013 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d874648680 con 0x55d874640510 2024-03-28T01:05:23.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.521+0000 7f2bddfb2740 1 -- 172.21.15.169:0/1296228013 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d874560d50 con 0x55d87463fb20 2024-03-28T01:05:23.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.521+0000 7f2bd8885640 1 --2- 172.21.15.169:0/1296228013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d874640510 0x55d87464a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:23.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.521+0000 7f2bddfb2740 1 -- 172.21.15.169:0/1296228013 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d874648800 con 0x55d874645680 2024-03-28T01:05:23.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.521+0000 7f2bd9887640 1 --2- 172.21.15.169:0/4076723490 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2bc00a29c0 0x7f2bc00a4e80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x55d874640d70 tx=0x7f2bd00046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:23.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.521+0000 7f2bd9086640 1 -- 172.21.15.169:0/1296228013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d874645680 msgr2=0x55d874651370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:23.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.521+0000 7f2bd9086640 1 --2- 172.21.15.169:0/1296228013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d874645680 0x55d874651370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.521+0000 7f2bd9086640 1 -- 172.21.15.169:0/1296228013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d874640510 msgr2=0x55d87464a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:23.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.521+0000 7f2bd9086640 1 --2- 172.21.15.169:0/1296228013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d874640510 0x55d87464a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.521+0000 7f2bd9086640 1 -- 172.21.15.169:0/1296228013 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d874385620 con 0x55d87463fb20 2024-03-28T01:05:23.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.522+0000 7f2bddfb2740 10 client.0 _ll_register_callbacks cb 0x55d874560720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:23.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.522+0000 7f2bd9086640 1 --2- 172.21.15.169:0/1296228013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d87463fb20 0x55d874649df0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2bc40bdea0 tx=0x7f2bc403d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85f029089c6f982 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:23.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.522+0000 7f2bbcff9640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:05:23.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.522+0000 7f2bbcff9640 1 -- 172.21.15.169:0/1296228013 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc40044e0 con 0x55d87463fb20 2024-03-28T01:05:23.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.522+0000 7f2bbcff9640 1 -- 172.21.15.169:0/1296228013 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2bc40be370 con 0x55d87463fb20 2024-03-28T01:05:23.527 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.523+0000 7f2bddfb2740 1 -- 172.21.15.169:0/1296228013 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2bcc0153f0 con 0x55d87463fb20 2024-03-28T01:05:23.528 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.524+0000 7f2bbcff9640 1 -- 172.21.15.169:0/1296228013 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2bc40be610 con 0x55d87463fb20 2024-03-28T01:05:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.527+0000 7f2bddfb2740 10 client.27035 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.528+0000 7f2bddfb2740 1 -- 172.21.15.169:0/1296228013 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x55d874654290 con 0x55d87463fb20 2024-03-28T01:05:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.528+0000 7f2bbcff9640 1 -- 172.21.15.169:0/1296228013 <== mon.2 v2:172.21.15.174:3300/0 4 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2bc40d3020 con 0x55d87463fb20 2024-03-28T01:05:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.528+0000 7f2bddfb2740 10 client.27035 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.529+0000 7f2bddfb2740 10 client.27035 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.529+0000 7f2bddfb2740 20 client.27035 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.529+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/1296228013 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d874655980 0x55d874657e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.529+0000 7f2bddfb2740 4 client.27035 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:05:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.529+0000 7f2bddfb2740 1 -- 172.21.15.169:0/1296228013 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2bcc0153f0 con 0x55d874655980 2024-03-28T01:05:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.529+0000 7f2bd8885640 1 --2- 172.21.15.169:0/1296228013 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d874655980 0x55d874657e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.534+0000 7f2bd8885640 1 --2- 172.21.15.169:0/1296228013 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d874655980 0x55d874657e40 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23f86bfa85135d0 server_cookie=30de3239d3eee831 in_seq=0 out_seq=0 2024-03-28T01:05:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.534+0000 7f2bbcff9640 10 client.27035 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.538+0000 7f2bbcff9640 1 -- 172.21.15.169:0/1296228013 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7f2bcc0153f0 con 0x55d874655980 2024-03-28T01:05:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.538+0000 7f2bbcff9640 10 client.27035 handle_command_reply: tid=0 2024-03-28T01:05:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.538+0000 7f2bddfb2740 1 client.27035 shutdown 2024-03-28T01:05:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.541+0000 7f2bddfb2740 1 -- 172.21.15.169:0/1296228013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d87463fb20 msgr2=0x55d874649df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.541+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/1296228013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d87463fb20 0x55d874649df0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2bc40bdea0 tx=0x7f2bc403d000 comp rx=0 tx=0).stop 2024-03-28T01:05:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.544+0000 7f2bddfb2740 1 -- 172.21.15.169:0/1296228013 shutdown_connections 2024-03-28T01:05:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.544+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/1296228013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d874645680 0x55d874651370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.544+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/1296228013 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d874655980 0x55d874657e40 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.545+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/1296228013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d874640510 0x55d87464a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.545+0000 7f2bbcff9640 0 client.27035 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.545+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/1296228013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d87463fb20 0x55d874649df0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.545+0000 7f2bddfb2740 1 -- 172.21.15.169:0/1296228013 >> 172.21.15.169:0/1296228013 conn(0x55d87463b090 msgr2=0x55d87463a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.546+0000 7f2bddfb2740 1 -- 172.21.15.169:0/1296228013 shutdown_connections 2024-03-28T01:05:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.550+0000 7f2bddfb2740 1 -- 172.21.15.169:0/1296228013 wait complete. 2024-03-28T01:05:23.558 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.553+0000 7f2bddfb2740 20 client.27035 trim_cache size 0 max 16384 2024-03-28T01:05:23.617 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.613+0000 7f2bddfb2740 1 Processor -- start 2024-03-28T01:05:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.619+0000 7f2bddfb2740 1 -- start start 2024-03-28T01:05:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.619+0000 7f2bddfb2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d87463fb20 0x55d87463b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.619+0000 7f2bddfb2740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d874640510 0x55d874651330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.619+0000 7f2bddfb2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d874645680 0x55d874649a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.619+0000 7f2bddfb2740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd41b2aa0 con 0x55d874640510 2024-03-28T01:05:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.619+0000 7f2bddfb2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd41b05a0 con 0x55d874645680 2024-03-28T01:05:23.624 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.619+0000 7f2bddfb2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd41ae660 con 0x55d87463fb20 2024-03-28T01:05:23.625 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.620+0000 7f2bd9887640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d874645680 0x55d874649a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:23.625 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.620+0000 7f2bd9887640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d874645680 0x55d874649a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:43090/0 (socket says 172.21.15.169:43090) 2024-03-28T01:05:23.625 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.620+0000 7f2bd9887640 1 -- 172.21.15.169:0/1911007217 learned_addr learned my addr 172.21.15.169:0/1911007217 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:23.625 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.620+0000 7f2bd9086640 1 --2- 172.21.15.169:0/1911007217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d87463fb20 0x55d87463b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.621+0000 7f2bd9887640 1 -- 172.21.15.169:0/1911007217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d87463fb20 msgr2=0x55d87463b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.621+0000 7f2bd9887640 1 --2- 172.21.15.169:0/1911007217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d87463fb20 0x55d87463b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.621+0000 7f2bd9887640 1 -- 172.21.15.169:0/1911007217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d874640510 msgr2=0x55d874651330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.621+0000 7f2bd9887640 1 --2- 172.21.15.169:0/1911007217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d874640510 0x55d874651330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.621+0000 7f2bd9887640 1 -- 172.21.15.169:0/1911007217 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d87463be00 con 0x55d874645680 2024-03-28T01:05:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.621+0000 7f2bd9887640 1 --2- 172.21.15.169:0/1911007217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d874645680 0x55d874649a00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2bd000ff70 tx=0x7f2bd0005a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75eb1ac5e4aad06f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.623+0000 7f2bbeffd640 1 -- 172.21.15.169:0/1911007217 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd0016070 con 0x55d874645680 2024-03-28T01:05:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.623+0000 7f2bbeffd640 1 -- 172.21.15.169:0/1911007217 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2bd001c900 con 0x55d874645680 2024-03-28T01:05:23.630 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.626+0000 7f2bddfb2740 1 -- 172.21.15.169:0/1911007217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d874645680 msgr2=0x55d874649a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:23.630 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.626+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/1911007217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d874645680 0x55d874649a00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2bd000ff70 tx=0x7f2bd0005a00 comp rx=0 tx=0).stop 2024-03-28T01:05:23.635 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.631+0000 7f2bddfb2740 1 -- 172.21.15.169:0/1911007217 shutdown_connections 2024-03-28T01:05:23.635 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.631+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/1911007217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d874645680 0x55d874649a00 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.635 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.631+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/1911007217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d874640510 0x55d874651330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.635 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.631+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/1911007217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d87463fb20 0x55d87463b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.635 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.631+0000 7f2bddfb2740 1 -- 172.21.15.169:0/1911007217 >> 172.21.15.169:0/1911007217 conn(0x55d87463b090 msgr2=0x55d874644dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.652+0000 7f2bddfb2740 1 -- 172.21.15.169:0/1911007217 shutdown_connections 2024-03-28T01:05:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.652+0000 7f2bddfb2740 1 -- 172.21.15.169:0/1911007217 wait complete. 2024-03-28T01:05:23.657 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.653+0000 7f2bddfb2740 1 -- start start 2024-03-28T01:05:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.653+0000 7f2bddfb2740 1 Processor -- start 2024-03-28T01:05:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.654+0000 7f2bddfb2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d87463fb20 0x55d87464f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.654+0000 7f2bddfb2740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d874640510 0x55d87464f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.655+0000 7f2bddfb2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d874645680 0x55d874650680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.655+0000 7f2bddfb2740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd41aec60 con 0x55d874640510 2024-03-28T01:05:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.655+0000 7f2bddfb2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd41aff40 con 0x55d874645680 2024-03-28T01:05:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.655+0000 7f2bddfb2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd41af280 con 0x55d87463fb20 2024-03-28T01:05:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.656+0000 7f2bddfb2740 10 client.0 _ll_register_callbacks cb 0x7f2bc4009490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.656+0000 7f2bd9086640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d87463fb20 0x55d87464f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.656+0000 7f2bd9086640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d87463fb20 0x55d87464f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:36304/0 (socket says 172.21.15.169:36304) 2024-03-28T01:05:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.656+0000 7f2bd9086640 1 -- 172.21.15.169:0/3402127233 learned_addr learned my addr 172.21.15.169:0/3402127233 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.656+0000 7f2bd9887640 1 --2- 172.21.15.169:0/3402127233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d874645680 0x55d874650680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.658+0000 7f2bd9887640 1 -- 172.21.15.169:0/3402127233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d87463fb20 msgr2=0x55d87464f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.658+0000 7f2bd9887640 1 --2- 172.21.15.169:0/3402127233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d87463fb20 0x55d87464f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.658+0000 7f2bd9887640 1 -- 172.21.15.169:0/3402127233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d874640510 msgr2=0x55d87464f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.658+0000 7f2bd9887640 1 --2- 172.21.15.169:0/3402127233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d874640510 0x55d87464f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.658+0000 7f2bd9887640 1 -- 172.21.15.169:0/3402127233 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d874650c70 con 0x55d874645680 2024-03-28T01:05:23.666 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.660+0000 7f2bd9887640 1 --2- 172.21.15.169:0/3402127233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d874645680 0x55d874650680 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0010c90 tx=0x7f2bd001c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d00680e575f4828 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:23.666 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.662+0000 7f2bbcff9640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:05:23.668 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.665+0000 7f2bbcff9640 1 -- 172.21.15.169:0/3402127233 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd0034060 con 0x55d874645680 2024-03-28T01:05:23.669 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.665+0000 7f2bddfb2740 1 -- 172.21.15.169:0/3402127233 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2bd001c900 con 0x55d874645680 2024-03-28T01:05:23.669 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.665+0000 7f2bbcff9640 1 -- 172.21.15.169:0/3402127233 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2bd001ccc0 con 0x55d874645680 2024-03-28T01:05:23.683 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.679+0000 7f2bbcff9640 1 -- 172.21.15.169:0/3402127233 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2bd003b050 con 0x55d874645680 2024-03-28T01:05:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.682+0000 7f2bddfb2740 10 client.27041 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.682+0000 7f2bddfb2740 1 -- 172.21.15.169:0/3402127233 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d8746b1a30 con 0x55d874645680 2024-03-28T01:05:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.686+0000 7f2bbcff9640 1 -- 172.21.15.169:0/3402127233 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2bd001c900 con 0x55d874645680 2024-03-28T01:05:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.686+0000 7f2bbcff9640 1 -- 172.21.15.169:0/3402127233 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2bd0025a00 con 0x55d874645680 2024-03-28T01:05:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.686+0000 7f2bddfb2740 10 client.27041 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.686+0000 7f2bddfb2740 10 client.27041 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.686+0000 7f2bddfb2740 20 client.27041 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.686+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/3402127233 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d874655980 0x55d8746b4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.686+0000 7f2bddfb2740 4 client.27041 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:05:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.686+0000 7f2bddfb2740 1 -- 172.21.15.169:0/3402127233 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d8745b2050 con 0x55d874655980 2024-03-28T01:05:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.690+0000 7f2bd9086640 1 --2- 172.21.15.169:0/3402127233 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d874655980 0x55d8746b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.692+0000 7f2bd9086640 1 --2- 172.21.15.169:0/3402127233 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d874655980 0x55d8746b4260 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfa4b693ef0df8ac server_cookie=2331fd57c117a931 in_seq=0 out_seq=0 2024-03-28T01:05:23.699 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:23.695+0000 7f2bbcff9640 10 client.27041 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:24.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:23 smithi169 ceph-mon[32274]: pgmap v617: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 645 MiB/s wr, 1.66k op/s 2024-03-28T01:05:24.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:23 smithi174 ceph-mon[36060]: pgmap v617: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 645 MiB/s wr, 1.66k op/s 2024-03-28T01:05:24.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:23 smithi177 ceph-mon[36206]: pgmap v617: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 645 MiB/s wr, 1.66k op/s 2024-03-28T01:05:25.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:24 smithi169 ceph-mon[32274]: pgmap v618: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 595 MiB/s wr, 1.44k op/s 2024-03-28T01:05:25.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:24 smithi174 ceph-mon[36060]: pgmap v618: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 595 MiB/s wr, 1.44k op/s 2024-03-28T01:05:25.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:24 smithi177 ceph-mon[36206]: pgmap v618: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 595 MiB/s wr, 1.44k op/s 2024-03-28T01:05:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.786+0000 7f2bbcff9640 1 -- 172.21.15.169:0/3402127233 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d8745b2050 con 0x55d874655980 2024-03-28T01:05:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.786+0000 7f2bbcff9640 10 client.27041 handle_command_reply: tid=0 2024-03-28T01:05:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.786+0000 7f2bddfb2740 1 client.27041 shutdown 2024-03-28T01:05:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.786+0000 7f2bddfb2740 1 -- 172.21.15.169:0/3402127233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d874645680 msgr2=0x55d874650680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.786+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/3402127233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d874645680 0x55d874650680 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0010c90 tx=0x7f2bd001c650 comp rx=0 tx=0).stop 2024-03-28T01:05:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.786+0000 7f2bddfb2740 1 -- 172.21.15.169:0/3402127233 shutdown_connections 2024-03-28T01:05:25.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.786+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/3402127233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d874645680 0x55d874650680 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:25.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.786+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/3402127233 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d874655980 0x55d8746b4260 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:25.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.786+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/3402127233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d874640510 0x55d87464f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:25.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.786+0000 7f2bddfb2740 1 --2- 172.21.15.169:0/3402127233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d87463fb20 0x55d87464f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:25.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.786+0000 7f2bddfb2740 1 -- 172.21.15.169:0/3402127233 >> 172.21.15.169:0/3402127233 conn(0x55d87463b090 msgr2=0x55d87463e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:25.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.786+0000 7f2bbcff9640 0 client.27041 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:25.791 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.787+0000 7f2bddfb2740 1 -- 172.21.15.169:0/3402127233 shutdown_connections 2024-03-28T01:05:25.792 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.787+0000 7f2bddfb2740 1 -- 172.21.15.169:0/3402127233 wait complete. 2024-03-28T01:05:25.792 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.787+0000 7f2bddfb2740 20 client.27041 trim_cache size 0 max 16384 2024-03-28T01:05:25.792 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:05:25.792 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:05:25.792 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:05:25.792 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:05:25.794 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.790+0000 7f2bda088640 1 -- 172.21.15.169:0/4076723490 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2bc00a29c0 msgr2=0x7f2bc00a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:25.794 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.790+0000 7f2bda088640 1 --2- 172.21.15.169:0/4076723490 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2bc00a29c0 0x7f2bc00a4e80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x55d874640d70 tx=0x7f2bd00046c0 comp rx=0 tx=0).stop 2024-03-28T01:05:25.794 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.790+0000 7f2bda088640 1 -- 172.21.15.169:0/4076723490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bd40ffd60 msgr2=0x7f2bd419dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:25.795 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.790+0000 7f2bda088640 1 --2- 172.21.15.169:0/4076723490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bd40ffd60 0x7f2bd419dde0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2bc402c0d0 tx=0x7f2bc40042f0 comp rx=0 tx=0).stop 2024-03-28T01:05:25.799 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.796+0000 7f2bda088640 1 -- 172.21.15.169:0/4076723490 shutdown_connections 2024-03-28T01:05:25.800 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.796+0000 7f2bda088640 1 --2- 172.21.15.169:0/4076723490 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2bc00a29c0 0x7f2bc00a4e80 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:25.800 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.796+0000 7f2bda088640 1 --2- 172.21.15.169:0/4076723490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bd4105080 0x7f2bd41a5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:25.800 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.796+0000 7f2bda088640 1 --2- 172.21.15.169:0/4076723490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2bd41026d0 0x7f2bd419e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:25.800 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.796+0000 7f2bda088640 1 --2- 172.21.15.169:0/4076723490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bd40ffd60 0x7f2bd419dde0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:25.800 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.796+0000 7f2bda088640 1 -- 172.21.15.169:0/4076723490 >> 172.21.15.169:0/4076723490 conn(0x7f2bd40fb980 msgr2=0x7f2bd4103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:25.802 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.798+0000 7f2bda088640 1 -- 172.21.15.169:0/4076723490 shutdown_connections 2024-03-28T01:05:25.804 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:25.799+0000 7f2bda088640 1 -- 172.21.15.169:0/4076723490 wait complete. 2024-03-28T01:05:27.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:27 smithi174 ceph-mon[36060]: pgmap v619: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 604 MiB/s wr, 1.36k op/s 2024-03-28T01:05:27.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:27 smithi177 ceph-mon[36206]: pgmap v619: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 604 MiB/s wr, 1.36k op/s 2024-03-28T01:05:28.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:27 smithi169 ceph-mon[32274]: pgmap v619: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 604 MiB/s wr, 1.36k op/s 2024-03-28T01:05:29.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:29 smithi174 ceph-mon[36060]: pgmap v620: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 621 MiB/s wr, 1.35k op/s 2024-03-28T01:05:29.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:29 smithi177 ceph-mon[36206]: pgmap v620: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 621 MiB/s wr, 1.35k op/s 2024-03-28T01:05:30.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:29 smithi169 ceph-mon[32274]: pgmap v620: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 621 MiB/s wr, 1.35k op/s 2024-03-28T01:05:31.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:31 smithi174 ceph-mon[36060]: pgmap v621: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 559 MiB/s wr, 1.15k op/s 2024-03-28T01:05:31.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:31 smithi177 ceph-mon[36206]: pgmap v621: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 559 MiB/s wr, 1.15k op/s 2024-03-28T01:05:32.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:31 smithi169 ceph-mon[32274]: pgmap v621: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 559 MiB/s wr, 1.15k op/s 2024-03-28T01:05:33.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:33 smithi177 ceph-mon[36206]: pgmap v622: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 648 MiB/s wr, 1.19k op/s 2024-03-28T01:05:33.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:33 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:05:34.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:33 smithi169 ceph-mon[32274]: pgmap v622: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 648 MiB/s wr, 1.19k op/s 2024-03-28T01:05:34.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:33 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:05:34.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:33 smithi174 ceph-mon[36060]: pgmap v622: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 648 MiB/s wr, 1.19k op/s 2024-03-28T01:05:34.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:33 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:05:34.858 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:05:34.859 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:05:35.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.226+0000 7fca76989640 1 -- 172.21.15.169:0/2114099782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca70072ab0 msgr2=0x7fca70072ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:35.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.226+0000 7fca76989640 1 --2- 172.21.15.169:0/2114099782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca70072ab0 0x7fca70072ed0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fca6c009860 tx=0x7fca6c02bbb0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.228+0000 7fca76989640 1 -- 172.21.15.169:0/2114099782 shutdown_connections 2024-03-28T01:05:35.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.228+0000 7fca76989640 1 --2- 172.21.15.169:0/2114099782 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fca70109730 0x7fca7010dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.228+0000 7fca76989640 1 --2- 172.21.15.169:0/2114099782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca70108d70 0x7fca701091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.228+0000 7fca76989640 1 --2- 172.21.15.169:0/2114099782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca70072ab0 0x7fca70072ed0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.228+0000 7fca76989640 1 -- 172.21.15.169:0/2114099782 >> 172.21.15.169:0/2114099782 conn(0x7fca7006e580 msgr2=0x7fca700709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:35.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.232+0000 7fca76989640 1 -- 172.21.15.169:0/2114099782 shutdown_connections 2024-03-28T01:05:35.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.236+0000 7fca76989640 1 -- 172.21.15.169:0/2114099782 wait complete. 2024-03-28T01:05:35.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.237+0000 7fca76989640 1 Processor -- start 2024-03-28T01:05:35.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.240+0000 7fca76989640 1 -- start start 2024-03-28T01:05:35.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.240+0000 7fca76989640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca70072ab0 0x7fca701a66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.240+0000 7fca76989640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fca70108d70 0x7fca701a6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.240+0000 7fca76989640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca70109730 0x7fca701adc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.240+0000 7fca76989640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7011c040 con 0x7fca70108d70 2024-03-28T01:05:35.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.240+0000 7fca76989640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7011bec0 con 0x7fca70072ab0 2024-03-28T01:05:35.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.240+0000 7fca76989640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7011c1c0 con 0x7fca70109730 2024-03-28T01:05:35.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.242+0000 7fca75186640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fca70108d70 0x7fca701a6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:35.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.242+0000 7fca75186640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fca70108d70 0x7fca701a6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53824/0 (socket says 172.21.15.169:53824) 2024-03-28T01:05:35.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.242+0000 7fca75186640 1 -- 172.21.15.169:0/1425378405 learned_addr learned my addr 172.21.15.169:0/1425378405 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:35.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.244+0000 7fca75186640 1 -- 172.21.15.169:0/1425378405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca70109730 msgr2=0x7fca701adc90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:05:35.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.244+0000 7fca75186640 1 --2- 172.21.15.169:0/1425378405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca70109730 0x7fca701adc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.244+0000 7fca75186640 1 -- 172.21.15.169:0/1425378405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca70072ab0 msgr2=0x7fca701a66e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:05:35.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.244+0000 7fca75186640 1 --2- 172.21.15.169:0/1425378405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca70072ab0 0x7fca701a66e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.244+0000 7fca75186640 1 -- 172.21.15.169:0/1425378405 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca701ae300 con 0x7fca70108d70 2024-03-28T01:05:35.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.244+0000 7fca75186640 1 --2- 172.21.15.169:0/1425378405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fca70108d70 0x7fca701a6c20 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fca6800efb0 tx=0x7fca6800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:35.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.248+0000 7fca66ffd640 1 -- 172.21.15.169:0/1425378405 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6801b070 con 0x7fca70108d70 2024-03-28T01:05:35.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.248+0000 7fca66ffd640 1 -- 172.21.15.169:0/1425378405 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca6800ba50 con 0x7fca70108d70 2024-03-28T01:05:35.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.248+0000 7fca66ffd640 1 -- 172.21.15.169:0/1425378405 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca68016cb0 con 0x7fca70108d70 2024-03-28T01:05:35.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.250+0000 7fca76989640 1 -- 172.21.15.169:0/1425378405 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca701ae5f0 con 0x7fca70108d70 2024-03-28T01:05:35.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.250+0000 7fca76989640 1 -- 172.21.15.169:0/1425378405 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca701aeaf0 con 0x7fca70108d70 2024-03-28T01:05:35.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.259+0000 7fca7a7cd740 1 Processor -- start 2024-03-28T01:05:35.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.260+0000 7fca7a7cd740 1 -- start start 2024-03-28T01:05:35.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.260+0000 7fca7a7cd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905765f80 0x5649057663a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.260+0000 7fca7a7cd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564905766970 0x56490576b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.260+0000 7fca7a7cd740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56490576bae0 0x56490576df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.260+0000 7fca7a7cd740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649056cb3d0 con 0x56490576bae0 2024-03-28T01:05:35.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.260+0000 7fca7a7cd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x564905673ad0 con 0x564905766970 2024-03-28T01:05:35.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.260+0000 7fca7a7cd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649056c7470 con 0x564905765f80 2024-03-28T01:05:35.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.261+0000 7fca76188640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905765f80 0x5649057663a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:35.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.261+0000 7fca76188640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905765f80 0x5649057663a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:57514/0 (socket says 172.21.15.169:57514) 2024-03-28T01:05:35.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.261+0000 7fca76188640 1 -- 172.21.15.169:0/30529500 learned_addr learned my addr 172.21.15.169:0/30529500 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:35.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.266+0000 7fca76188640 1 -- 172.21.15.169:0/30529500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564905766970 msgr2=0x56490576b470 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:05:35.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.266+0000 7fca76188640 1 --2- 172.21.15.169:0/30529500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564905766970 0x56490576b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.266+0000 7fca76188640 1 -- 172.21.15.169:0/30529500 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56490576bae0 msgr2=0x56490576df70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:05:35.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.266+0000 7fca76188640 1 --2- 172.21.15.169:0/30529500 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56490576bae0 0x56490576df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.266+0000 7fca76188640 1 -- 172.21.15.169:0/30529500 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649054a97d0 con 0x564905765f80 2024-03-28T01:05:35.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.269+0000 7fca66ffd640 1 -- 172.21.15.169:0/1425378405 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fca6801e070 con 0x7fca70108d70 2024-03-28T01:05:35.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.270+0000 7fca76188640 1 --2- 172.21.15.169:0/30529500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905765f80 0x5649057663a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fca60009bd0 tx=0x7fca6000e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e26740ec48668e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:35.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.270+0000 7fca66ffd640 1 --2- 172.21.15.169:0/1425378405 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fca5c0a2a00 0x7fca5c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.272+0000 7fca66ffd640 1 -- 172.21.15.169:0/1425378405 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fca680cf2a0 con 0x7fca70108d70 2024-03-28T01:05:35.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.274+0000 7fca5bfff640 1 -- 172.21.15.169:0/30529500 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca60005020 con 0x564905765f80 2024-03-28T01:05:35.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.274+0000 7fca5bfff640 1 -- 172.21.15.169:0/30529500 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca600047a0 con 0x564905765f80 2024-03-28T01:05:35.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.274+0000 7fca5bfff640 1 -- 172.21.15.169:0/30529500 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca60010690 con 0x564905765f80 2024-03-28T01:05:35.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.275+0000 7fca7a7cd740 1 -- 172.21.15.169:0/30529500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905765f80 msgr2=0x5649057663a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:35.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.275+0000 7fca7a7cd740 1 --2- 172.21.15.169:0/30529500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905765f80 0x5649057663a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fca60009bd0 tx=0x7fca6000e720 comp rx=0 tx=0).stop 2024-03-28T01:05:35.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.276+0000 7fca7a7cd740 1 -- 172.21.15.169:0/30529500 shutdown_connections 2024-03-28T01:05:35.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.276+0000 7fca7a7cd740 1 --2- 172.21.15.169:0/30529500 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56490576bae0 0x56490576df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.276+0000 7fca75987640 1 --2- 172.21.15.169:0/1425378405 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fca5c0a2a00 0x7fca5c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:35.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.276+0000 7fca7a7cd740 1 --2- 172.21.15.169:0/30529500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564905766970 0x56490576b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.276+0000 7fca7a7cd740 1 --2- 172.21.15.169:0/30529500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905765f80 0x5649057663a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.276+0000 7fca7a7cd740 1 -- 172.21.15.169:0/30529500 >> 172.21.15.169:0/30529500 conn(0x5649057614f0 msgr2=0x5649057618f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:35.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.280+0000 7fca7a7cd740 1 -- 172.21.15.169:0/30529500 shutdown_connections 2024-03-28T01:05:35.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.284+0000 7fca7a7cd740 1 -- 172.21.15.169:0/30529500 wait complete. 2024-03-28T01:05:35.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.284+0000 7fca7a7cd740 1 -- start start 2024-03-28T01:05:35.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.284+0000 7fca7a7cd740 1 Processor -- start 2024-03-28T01:05:35.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.285+0000 7fca7a7cd740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564905765f80 0x564905781520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.285+0000 7fca7a7cd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905766970 0x564905781a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.285+0000 7fca7a7cd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56490576bae0 0x564905788a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.285+0000 7fca7a7cd740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56490577faa0 con 0x564905765f80 2024-03-28T01:05:35.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.285+0000 7fca7a7cd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x564905687060 con 0x56490576bae0 2024-03-28T01:05:35.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.285+0000 7fca7a7cd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56490577fc20 con 0x564905766970 2024-03-28T01:05:35.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.285+0000 7fca7a7cd740 10 client.0 _ll_register_callbacks cb 0x5649056696a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:35.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.285+0000 7fca76188640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56490576bae0 0x564905788a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:35.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.285+0000 7fca76188640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56490576bae0 0x564905788a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:55184/0 (socket says 172.21.15.169:55184) 2024-03-28T01:05:35.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.285+0000 7fca76188640 1 -- 172.21.15.169:0/3996647636 learned_addr learned my addr 172.21.15.169:0/3996647636 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:35.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.286+0000 7fca75186640 1 --2- 172.21.15.169:0/3996647636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905766970 0x564905781a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:35.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.288+0000 7fca76188640 1 -- 172.21.15.169:0/3996647636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905766970 msgr2=0x564905781a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:35.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.288+0000 7fca76188640 1 --2- 172.21.15.169:0/3996647636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905766970 0x564905781a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.288+0000 7fca76188640 1 -- 172.21.15.169:0/3996647636 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564905765f80 msgr2=0x564905781520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:35.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.288+0000 7fca76188640 1 --2- 172.21.15.169:0/3996647636 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564905765f80 0x564905781520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.288+0000 7fca76188640 1 -- 172.21.15.169:0/3996647636 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5649053df3f0 con 0x56490576bae0 2024-03-28T01:05:35.292 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.289+0000 7fca76188640 1 --2- 172.21.15.169:0/3996647636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56490576bae0 0x564905788a30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fca60005250 tx=0x7fca60005280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1449728df2da1d82 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:35.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.290+0000 7fca597fa640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:05:35.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.290+0000 7fca597fa640 1 -- 172.21.15.169:0/3996647636 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6001d040 con 0x56490576bae0 2024-03-28T01:05:35.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.290+0000 7fca7a7cd740 1 -- 172.21.15.169:0/3996647636 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fca600047a0 con 0x56490576bae0 2024-03-28T01:05:35.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.290+0000 7fca597fa640 1 -- 172.21.15.169:0/3996647636 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fca60005420 con 0x56490576bae0 2024-03-28T01:05:35.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.291+0000 7fca597fa640 1 -- 172.21.15.169:0/3996647636 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca6002f050 con 0x56490576bae0 2024-03-28T01:05:35.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.291+0000 7fca7a7cd740 10 client.27047 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:35.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.291+0000 7fca7a7cd740 1 -- 172.21.15.169:0/3996647636 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56490539f0f0 con 0x56490576bae0 2024-03-28T01:05:35.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.294+0000 7fca75987640 1 --2- 172.21.15.169:0/1425378405 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fca5c0a2a00 0x7fca5c0a4ec0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fca6c037040 tx=0x7fca6c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:35.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.294+0000 7fca597fa640 1 -- 172.21.15.169:0/3996647636 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca600047a0 con 0x56490576bae0 2024-03-28T01:05:35.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.294+0000 7fca597fa640 1 -- 172.21.15.169:0/3996647636 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fca6002a030 con 0x56490576bae0 2024-03-28T01:05:35.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.295+0000 7fca7a7cd740 10 client.27047 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:35.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.295+0000 7fca7a7cd740 10 client.27047 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:35.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.295+0000 7fca7a7cd740 20 client.27047 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:35.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.295+0000 7fca7a7cd740 1 --2- 172.21.15.169:0/3996647636 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56490578cb90 0x56490578ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.295+0000 7fca7a7cd740 4 client.27047 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:05:35.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.295+0000 7fca7a7cd740 1 -- 172.21.15.169:0/3996647636 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5649056d8910 con 0x56490578cb90 2024-03-28T01:05:35.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.299+0000 7fca75987640 1 --2- 172.21.15.169:0/3996647636 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56490578cb90 0x56490578ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:35.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.308+0000 7fca75987640 1 --2- 172.21.15.169:0/3996647636 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56490578cb90 0x56490578ef80 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5f048448c72b91d server_cookie=1072b50573ca769d in_seq=0 out_seq=0 2024-03-28T01:05:35.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.309+0000 7fca597fa640 10 client.27047 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:35.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.315+0000 7fca597fa640 1 -- 172.21.15.169:0/3996647636 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5649056d8910 con 0x56490578cb90 2024-03-28T01:05:35.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.315+0000 7fca597fa640 10 client.27047 handle_command_reply: tid=0 2024-03-28T01:05:35.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.315+0000 7fca7a7cd740 1 client.27047 shutdown 2024-03-28T01:05:35.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.317+0000 7fca7a7cd740 1 -- 172.21.15.169:0/3996647636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56490576bae0 msgr2=0x564905788a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:35.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.317+0000 7fca7a7cd740 1 --2- 172.21.15.169:0/3996647636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56490576bae0 0x564905788a30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fca60005250 tx=0x7fca60005280 comp rx=0 tx=0).stop 2024-03-28T01:05:35.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.318+0000 7fca7a7cd740 1 -- 172.21.15.169:0/3996647636 shutdown_connections 2024-03-28T01:05:35.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.319+0000 7fca7a7cd740 1 --2- 172.21.15.169:0/3996647636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56490576bae0 0x564905788a30 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.319+0000 7fca7a7cd740 1 --2- 172.21.15.169:0/3996647636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905766970 0x564905781a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.319+0000 7fca7a7cd740 1 --2- 172.21.15.169:0/3996647636 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56490578cb90 0x56490578ef80 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.319+0000 7fca7a7cd740 1 --2- 172.21.15.169:0/3996647636 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564905765f80 0x564905781520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.319+0000 7fca7a7cd740 1 -- 172.21.15.169:0/3996647636 >> 172.21.15.169:0/3996647636 conn(0x5649057614f0 msgr2=0x5649057607b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:35.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.321+0000 7fca597fa640 0 client.27047 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:35.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.331+0000 7fca7a7cd740 1 -- 172.21.15.169:0/3996647636 shutdown_connections 2024-03-28T01:05:35.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.333+0000 7fca7a7cd740 1 -- 172.21.15.169:0/3996647636 wait complete. 2024-03-28T01:05:35.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.333+0000 7fca7a7cd740 20 client.27047 trim_cache size 0 max 16384 2024-03-28T01:05:35.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.390+0000 7fca7a7cd740 1 Processor -- start 2024-03-28T01:05:35.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.396+0000 7fca7a7cd740 1 -- start start 2024-03-28T01:05:35.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.397+0000 7fca7a7cd740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564905765f80 0x564905780b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.397+0000 7fca7a7cd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905766970 0x5649057889f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.397+0000 7fca7a7cd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56490576bae0 0x56490575ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.397+0000 7fca7a7cd740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca540120e0 con 0x564905765f80 2024-03-28T01:05:35.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.398+0000 7fca7a7cd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5400fbe0 con 0x56490576bae0 2024-03-28T01:05:35.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.398+0000 7fca7a7cd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5400dc80 con 0x564905766970 2024-03-28T01:05:35.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.398+0000 7fca76188640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56490576bae0 0x56490575ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:35.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.398+0000 7fca76188640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56490576bae0 0x56490575ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:55200/0 (socket says 172.21.15.169:55200) 2024-03-28T01:05:35.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.398+0000 7fca76188640 1 -- 172.21.15.169:0/1177658615 learned_addr learned my addr 172.21.15.169:0/1177658615 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:35.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.399+0000 7fca75186640 1 --2- 172.21.15.169:0/1177658615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905766970 0x5649057889f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:35.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.399+0000 7fca75186640 1 -- 172.21.15.169:0/1177658615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56490576bae0 msgr2=0x56490575ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:35.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.399+0000 7fca75186640 1 --2- 172.21.15.169:0/1177658615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56490576bae0 0x56490575ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.399+0000 7fca75186640 1 -- 172.21.15.169:0/1177658615 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564905765f80 msgr2=0x564905780b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:35.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.399+0000 7fca75186640 1 --2- 172.21.15.169:0/1177658615 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564905765f80 0x564905780b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.399+0000 7fca75186640 1 -- 172.21.15.169:0/1177658615 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56490576e490 con 0x564905766970 2024-03-28T01:05:35.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.399+0000 7fca75186640 1 --2- 172.21.15.169:0/1177658615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905766970 0x5649057889f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fca6800ef80 tx=0x7fca68097c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=243ae1e01a9b9336 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:35.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.399+0000 7fca5b7fe640 1 -- 172.21.15.169:0/1177658615 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca680b4070 con 0x564905766970 2024-03-28T01:05:35.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.400+0000 7fca5b7fe640 1 -- 172.21.15.169:0/1177658615 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca680a3fe0 con 0x564905766970 2024-03-28T01:05:35.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.400+0000 7fca5b7fe640 1 -- 172.21.15.169:0/1177658615 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca680cce60 con 0x564905766970 2024-03-28T01:05:35.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.401+0000 7fca76188640 1 --2- 172.21.15.169:0/1177658615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56490576bae0 0x56490575ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:05:35.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.401+0000 7fca7a7cd740 1 -- 172.21.15.169:0/1177658615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905766970 msgr2=0x5649057889f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:35.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.401+0000 7fca7a7cd740 1 --2- 172.21.15.169:0/1177658615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905766970 0x5649057889f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fca6800ef80 tx=0x7fca68097c10 comp rx=0 tx=0).stop 2024-03-28T01:05:35.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.406+0000 7fca7a7cd740 1 -- 172.21.15.169:0/1177658615 shutdown_connections 2024-03-28T01:05:35.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.406+0000 7fca7a7cd740 1 --2- 172.21.15.169:0/1177658615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56490576bae0 0x56490575ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.406+0000 7fca7a7cd740 1 --2- 172.21.15.169:0/1177658615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905766970 0x5649057889f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.406+0000 7fca7a7cd740 1 --2- 172.21.15.169:0/1177658615 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564905765f80 0x564905780b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.406+0000 7fca7a7cd740 1 -- 172.21.15.169:0/1177658615 >> 172.21.15.169:0/1177658615 conn(0x5649057614f0 msgr2=0x564905767eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:35.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.407+0000 7fca7a7cd740 1 -- 172.21.15.169:0/1177658615 shutdown_connections 2024-03-28T01:05:35.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.410+0000 7fca7a7cd740 1 -- 172.21.15.169:0/1177658615 wait complete. 2024-03-28T01:05:35.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.410+0000 7fca7a7cd740 1 -- start start 2024-03-28T01:05:35.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.411+0000 7fca7a7cd740 1 Processor -- start 2024-03-28T01:05:35.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.411+0000 7fca7a7cd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564905765f80 0x564905788230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.411+0000 7fca7a7cd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905766970 0x564905788770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.411+0000 7fca7a7cd740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56490576bae0 0x564905786890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.411+0000 7fca7a7cd740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5400e280 con 0x56490576bae0 2024-03-28T01:05:35.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.411+0000 7fca7a7cd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5400f580 con 0x564905765f80 2024-03-28T01:05:35.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.411+0000 7fca7a7cd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5400e8c0 con 0x564905766970 2024-03-28T01:05:35.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.411+0000 7fca7a7cd740 10 client.0 _ll_register_callbacks cb 0x7fca60009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:35.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.412+0000 7fca75186640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905766970 0x564905788770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:35.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.412+0000 7fca75186640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905766970 0x564905788770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:57560/0 (socket says 172.21.15.169:57560) 2024-03-28T01:05:35.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.412+0000 7fca75186640 1 -- 172.21.15.169:0/1480044996 learned_addr learned my addr 172.21.15.169:0/1480044996 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:35.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.412+0000 7fca76188640 1 --2- 172.21.15.169:0/1480044996 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56490576bae0 0x564905786890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:35.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.412+0000 7fca75987640 1 --2- 172.21.15.169:0/1480044996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564905765f80 0x564905788230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:35.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.412+0000 7fca75186640 1 -- 172.21.15.169:0/1480044996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564905765f80 msgr2=0x564905788230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:35.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.412+0000 7fca75186640 1 --2- 172.21.15.169:0/1480044996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564905765f80 0x564905788230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.412+0000 7fca75186640 1 -- 172.21.15.169:0/1480044996 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56490576bae0 msgr2=0x564905786890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:35.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.412+0000 7fca75186640 1 --2- 172.21.15.169:0/1480044996 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56490576bae0 0x564905786890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.412+0000 7fca75186640 1 -- 172.21.15.169:0/1480044996 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564905786e80 con 0x564905766970 2024-03-28T01:05:35.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.412+0000 7fca75186640 1 --2- 172.21.15.169:0/1480044996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905766970 0x564905788770 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fca6809b7d0 tx=0x7fca680097f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=618ee3ddd1e9e2d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:35.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.414+0000 7fca597fa640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:05:35.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.414+0000 7fca597fa640 1 -- 172.21.15.169:0/1480044996 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca680b4040 con 0x564905766970 2024-03-28T01:05:35.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.414+0000 7fca7a7cd740 1 -- 172.21.15.169:0/1480044996 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fca680a3fe0 con 0x564905766970 2024-03-28T01:05:35.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.414+0000 7fca597fa640 1 -- 172.21.15.169:0/1480044996 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fca6809cd70 con 0x564905766970 2024-03-28T01:05:35.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.414+0000 7fca597fa640 1 -- 172.21.15.169:0/1480044996 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca680af050 con 0x564905766970 2024-03-28T01:05:35.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.415+0000 7fca7a7cd740 10 client.26992 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:35.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.415+0000 7fca7a7cd740 1 -- 172.21.15.169:0/1480044996 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5649057e8c60 con 0x564905766970 2024-03-28T01:05:35.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.417+0000 7fca597fa640 1 -- 172.21.15.169:0/1480044996 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca680a3fe0 con 0x564905766970 2024-03-28T01:05:35.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.417+0000 7fca597fa640 1 -- 172.21.15.169:0/1480044996 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fca6800b040 con 0x564905766970 2024-03-28T01:05:35.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.417+0000 7fca7a7cd740 10 client.26992 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:35.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.417+0000 7fca7a7cd740 10 client.26992 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:35.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.417+0000 7fca7a7cd740 20 client.26992 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:35.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.417+0000 7fca7a7cd740 1 --2- 172.21.15.169:0/1480044996 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5649057e9630 0x5649057eba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.417+0000 7fca7a7cd740 4 client.26992 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:05:35.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.417+0000 7fca7a7cd740 1 -- 172.21.15.169:0/1480044996 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564905781310 con 0x5649057e9630 2024-03-28T01:05:35.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.419+0000 7fca75987640 1 --2- 172.21.15.169:0/1480044996 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5649057e9630 0x5649057eba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:35.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.420+0000 7fca597fa640 10 client.26992 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:35.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.420+0000 7fca75987640 1 --2- 172.21.15.169:0/1480044996 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5649057e9630 0x5649057eba50 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8ac4e236dcdfe5 server_cookie=f6931c080de2015f in_seq=0 out_seq=0 2024-03-28T01:05:35.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.422+0000 7fca597fa640 1 -- 172.21.15.169:0/1480044996 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x564905781310 con 0x5649057e9630 2024-03-28T01:05:35.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.422+0000 7fca597fa640 10 client.26992 handle_command_reply: tid=0 2024-03-28T01:05:35.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.422+0000 7fca7a7cd740 1 client.26992 shutdown 2024-03-28T01:05:35.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.422+0000 7fca7a7cd740 1 -- 172.21.15.169:0/1480044996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905766970 msgr2=0x564905788770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:35.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.422+0000 7fca7a7cd740 1 --2- 172.21.15.169:0/1480044996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905766970 0x564905788770 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fca6809b7d0 tx=0x7fca680097f0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.423+0000 7fca7a7cd740 1 -- 172.21.15.169:0/1480044996 shutdown_connections 2024-03-28T01:05:35.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.423+0000 7fca7a7cd740 1 --2- 172.21.15.169:0/1480044996 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5649057e9630 0x5649057eba50 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.423+0000 7fca7a7cd740 1 --2- 172.21.15.169:0/1480044996 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56490576bae0 0x564905786890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.423+0000 7fca7a7cd740 1 --2- 172.21.15.169:0/1480044996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564905766970 0x564905788770 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.423+0000 7fca7a7cd740 1 --2- 172.21.15.169:0/1480044996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564905765f80 0x564905788230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.423+0000 7fca7a7cd740 1 -- 172.21.15.169:0/1480044996 >> 172.21.15.169:0/1480044996 conn(0x5649057614f0 msgr2=0x564905764750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:35.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.424+0000 7fca597fa640 0 client.26992 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:35.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.424+0000 7fca7a7cd740 1 -- 172.21.15.169:0/1480044996 shutdown_connections 2024-03-28T01:05:35.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.424+0000 7fca7a7cd740 1 -- 172.21.15.169:0/1480044996 wait complete. 2024-03-28T01:05:35.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.424+0000 7fca7a7cd740 20 client.26992 trim_cache size 0 max 16384 2024-03-28T01:05:35.428 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:05:35.429 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "no active scrubs running", 2024-03-28T01:05:35.429 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": {} 2024-03-28T01:05:35.429 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:05:35.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.429+0000 7fca76989640 1 -- 172.21.15.169:0/1425378405 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fca5c0a2a00 msgr2=0x7fca5c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:35.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.429+0000 7fca76989640 1 --2- 172.21.15.169:0/1425378405 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fca5c0a2a00 0x7fca5c0a4ec0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fca6c037040 tx=0x7fca6c000bc0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.429+0000 7fca76989640 1 -- 172.21.15.169:0/1425378405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fca70108d70 msgr2=0x7fca701a6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:35.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.429+0000 7fca76989640 1 --2- 172.21.15.169:0/1425378405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fca70108d70 0x7fca701a6c20 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fca6800efb0 tx=0x7fca6800c800 comp rx=0 tx=0).stop 2024-03-28T01:05:35.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.430+0000 7fca76989640 1 -- 172.21.15.169:0/1425378405 shutdown_connections 2024-03-28T01:05:35.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.430+0000 7fca76989640 1 --2- 172.21.15.169:0/1425378405 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fca5c0a2a00 0x7fca5c0a4ec0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.430+0000 7fca76989640 1 --2- 172.21.15.169:0/1425378405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca70109730 0x7fca701adc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.430+0000 7fca76989640 1 --2- 172.21.15.169:0/1425378405 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fca70108d70 0x7fca701a6c20 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.430+0000 7fca76989640 1 --2- 172.21.15.169:0/1425378405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca70072ab0 0x7fca701a66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.430+0000 7fca76989640 1 -- 172.21.15.169:0/1425378405 >> 172.21.15.169:0/1425378405 conn(0x7fca7006e580 msgr2=0x7fca70070220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:35.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.431+0000 7fca76989640 1 -- 172.21.15.169:0/1425378405 shutdown_connections 2024-03-28T01:05:35.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.431+0000 7fca76989640 1 -- 172.21.15.169:0/1425378405 wait complete. 2024-03-28T01:05:35.473 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T01:05:35.474 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T01:05:35.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.782+0000 7fc30bfff640 1 -- 172.21.15.169:0/1423125438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc30c10d760 msgr2=0x7fc30c1185e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:35.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.782+0000 7fc30bfff640 1 --2- 172.21.15.169:0/1423125438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc30c10d760 0x7fc30c1185e0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fc300009900 tx=0x7fc30002bc80 comp rx=0 tx=0).stop 2024-03-28T01:05:35.789 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.783+0000 7fc30bfff640 1 -- 172.21.15.169:0/1423125438 shutdown_connections 2024-03-28T01:05:35.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.783+0000 7fc30bfff640 1 --2- 172.21.15.169:0/1423125438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc30c10d760 0x7fc30c1185e0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.783+0000 7fc30bfff640 1 --2- 172.21.15.169:0/1423125438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc30c10ac80 0x7fc30c10d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.783+0000 7fc30bfff640 1 --2- 172.21.15.169:0/1423125438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc30c0734a0 0x7fc30c10a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.783+0000 7fc30bfff640 1 -- 172.21.15.169:0/1423125438 >> 172.21.15.169:0/1423125438 conn(0x7fc30c06e3d0 msgr2=0x7fc30c070810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:35.793 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.789+0000 7fc30bfff640 1 -- 172.21.15.169:0/1423125438 shutdown_connections 2024-03-28T01:05:35.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.793+0000 7fc30bfff640 1 -- 172.21.15.169:0/1423125438 wait complete. 2024-03-28T01:05:35.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.794+0000 7fc30bfff640 1 Processor -- start 2024-03-28T01:05:35.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.797+0000 7fc30bfff640 1 -- start start 2024-03-28T01:05:35.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.797+0000 7fc30bfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc30c0734a0 0x7fc30c1a6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.797+0000 7fc30bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc30c10ac80 0x7fc30c1a6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.797+0000 7fc30bfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc30c10d760 0x7fc30c1adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.797+0000 7fc30bfff640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc30c11c010 con 0x7fc30c10d760 2024-03-28T01:05:35.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.797+0000 7fc30bfff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc30c11be90 con 0x7fc30c0734a0 2024-03-28T01:05:35.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.797+0000 7fc30bfff640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc30c11c190 con 0x7fc30c10ac80 2024-03-28T01:05:35.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:35 smithi169 ceph-mon[32274]: pgmap v623: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 576 MiB/s wr, 1.04k op/s 2024-03-28T01:05:35.804 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:35 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:05:35.804 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:35 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:05:35.804 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:35 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:05:35.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.799+0000 7fc30a7fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc30c10ac80 0x7fc30c1a6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:35.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.802+0000 7fc30b7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc30c10d760 0x7fc30c1adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:35.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.802+0000 7fc30b7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc30c10d760 0x7fc30c1adb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53876/0 (socket says 172.21.15.169:53876) 2024-03-28T01:05:35.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.802+0000 7fc30b7fe640 1 -- 172.21.15.169:0/3000373631 learned_addr learned my addr 172.21.15.169:0/3000373631 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:35.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.804+0000 7fc30affd640 1 --2- 172.21.15.169:0/3000373631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc30c0734a0 0x7fc30c1a6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:35.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.805+0000 7fc30a7fc640 1 -- 172.21.15.169:0/3000373631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc30c0734a0 msgr2=0x7fc30c1a6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:35.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.805+0000 7fc30a7fc640 1 --2- 172.21.15.169:0/3000373631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc30c0734a0 0x7fc30c1a6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.805+0000 7fc30a7fc640 1 -- 172.21.15.169:0/3000373631 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc30c10d760 msgr2=0x7fc30c1adb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:35.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.805+0000 7fc30a7fc640 1 --2- 172.21.15.169:0/3000373631 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc30c10d760 0x7fc30c1adb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:35.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.805+0000 7fc30a7fc640 1 -- 172.21.15.169:0/3000373631 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc30c1ae190 con 0x7fc30c10ac80 2024-03-28T01:05:35.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.806+0000 7fc30a7fc640 1 --2- 172.21.15.169:0/3000373631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc30c10ac80 0x7fc30c1a6ab0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc00c970 tx=0x7fc2fc00ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:35.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.807+0000 7fc2f3fff640 1 -- 172.21.15.169:0/3000373631 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc007bb0 con 0x7fc30c10ac80 2024-03-28T01:05:35.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.807+0000 7fc30bfff640 1 -- 172.21.15.169:0/3000373631 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc30c1ae480 con 0x7fc30c10ac80 2024-03-28T01:05:35.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.807+0000 7fc30bfff640 1 -- 172.21.15.169:0/3000373631 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc30c1ae9d0 con 0x7fc30c10ac80 2024-03-28T01:05:35.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.808+0000 7fc2f3fff640 1 -- 172.21.15.169:0/3000373631 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2fc007d50 con 0x7fc30c10ac80 2024-03-28T01:05:35.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.808+0000 7fc2f3fff640 1 -- 172.21.15.169:0/3000373631 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc00f450 con 0x7fc30c10ac80 2024-03-28T01:05:35.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.814+0000 7fc2f3fff640 1 -- 172.21.15.169:0/3000373631 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc2fc0046c0 con 0x7fc30c10ac80 2024-03-28T01:05:35.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.815+0000 7fc2f3fff640 1 --2- 172.21.15.169:0/3000373631 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc2f40a2a10 0x7fc2f40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:35.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.815+0000 7fc2f3fff640 1 -- 172.21.15.169:0/3000373631 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fc2fc0ce960 con 0x7fc30c10ac80 2024-03-28T01:05:35.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.818+0000 7fc30affd640 1 --2- 172.21.15.169:0/3000373631 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc2f40a2a10 0x7fc2f40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:35.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.818+0000 7fc30affd640 1 --2- 172.21.15.169:0/3000373631 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc2f40a2a10 0x7fc2f40a4ed0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc30c0738a0 tx=0x7fc3040094f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:35.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.819+0000 7fc30bfff640 1 -- 172.21.15.169:0/3000373631 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f80052c0 con 0x7fc30c10ac80 2024-03-28T01:05:35.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:35.828+0000 7fc2f3fff640 1 -- 172.21.15.169:0/3000373631 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc2fc014030 con 0x7fc30c10ac80 2024-03-28T01:05:35.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:35 smithi177 ceph-mon[36206]: pgmap v623: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 576 MiB/s wr, 1.04k op/s 2024-03-28T01:05:35.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:35 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:05:35.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:35 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:05:35.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:35 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:05:36.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:35 smithi174 ceph-mon[36060]: pgmap v623: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 576 MiB/s wr, 1.04k op/s 2024-03-28T01:05:36.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:35 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:05:36.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:35 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:05:36.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:35 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:05:36.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.352+0000 7fc30bfff640 1 -- 172.21.15.169:0/3000373631 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2f80055b0 con 0x7fc30c10ac80 2024-03-28T01:05:36.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.356+0000 7fc2f3fff640 1 -- 172.21.15.169:0/3000373631 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7fc2fc09b600 con 0x7fc30c10ac80 2024-03-28T01:05:36.360 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T01:05:36.360 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T01:05:36.361 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T01:05:36.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.365+0000 7fc2f1ffb640 1 -- 172.21.15.169:0/3000373631 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc2f40a2a10 msgr2=0x7fc2f40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:36.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.365+0000 7fc2f1ffb640 1 --2- 172.21.15.169:0/3000373631 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc2f40a2a10 0x7fc2f40a4ed0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc30c0738a0 tx=0x7fc3040094f0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.365+0000 7fc2f1ffb640 1 -- 172.21.15.169:0/3000373631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc30c10ac80 msgr2=0x7fc30c1a6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:36.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.365+0000 7fc2f1ffb640 1 --2- 172.21.15.169:0/3000373631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc30c10ac80 0x7fc30c1a6ab0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc00c970 tx=0x7fc2fc00ce30 comp rx=0 tx=0).stop 2024-03-28T01:05:36.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.368+0000 7fc2f1ffb640 1 -- 172.21.15.169:0/3000373631 shutdown_connections 2024-03-28T01:05:36.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.368+0000 7fc2f1ffb640 1 --2- 172.21.15.169:0/3000373631 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fc2f40a2a10 0x7fc2f40a4ed0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.368+0000 7fc2f1ffb640 1 --2- 172.21.15.169:0/3000373631 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc30c10d760 0x7fc30c1adb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.368+0000 7fc2f1ffb640 1 --2- 172.21.15.169:0/3000373631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc30c10ac80 0x7fc30c1a6ab0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.368+0000 7fc2f1ffb640 1 --2- 172.21.15.169:0/3000373631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc30c0734a0 0x7fc30c1a6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.368+0000 7fc2f1ffb640 1 -- 172.21.15.169:0/3000373631 >> 172.21.15.169:0/3000373631 conn(0x7fc30c06e3d0 msgr2=0x7fc30c10a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:36.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.379+0000 7fc2f1ffb640 1 -- 172.21.15.169:0/3000373631 shutdown_connections 2024-03-28T01:05:36.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.380+0000 7fc2f1ffb640 1 -- 172.21.15.169:0/3000373631 wait complete. 2024-03-28T01:05:36.435 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:05:36.435 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T01:05:36.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.708+0000 7f1ea5673640 1 -- 172.21.15.169:0/3745895138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ea010b010 msgr2=0x7f1ea010d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:36.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.708+0000 7f1ea5673640 1 --2- 172.21.15.169:0/3745895138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ea010b010 0x7f1ea010d480 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1e9401a910 tx=0x7f1e9403cca0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.717+0000 7f1ea5673640 1 -- 172.21.15.169:0/3745895138 shutdown_connections 2024-03-28T01:05:36.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.717+0000 7f1ea5673640 1 --2- 172.21.15.169:0/3745895138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ea010d9c0 0x7f1ea010de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.717+0000 7f1ea5673640 1 --2- 172.21.15.169:0/3745895138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ea010b010 0x7f1ea010d480 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.717+0000 7f1ea5673640 1 --2- 172.21.15.169:0/3745895138 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ea01086a0 0x7f1ea010aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.717+0000 7f1ea5673640 1 -- 172.21.15.169:0/3745895138 >> 172.21.15.169:0/3745895138 conn(0x7f1ea006e500 msgr2=0x7f1ea0070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:36.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.724+0000 7f1ea5673640 1 -- 172.21.15.169:0/3745895138 shutdown_connections 2024-03-28T01:05:36.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.724+0000 7f1ea5673640 1 -- 172.21.15.169:0/3745895138 wait complete. 2024-03-28T01:05:36.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.725+0000 7f1ea5673640 1 Processor -- start 2024-03-28T01:05:36.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.725+0000 7f1ea5673640 1 -- start start 2024-03-28T01:05:36.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.725+0000 7f1ea5673640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ea01086a0 0x7f1ea019dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:36.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.725+0000 7f1ea5673640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ea010b010 0x7f1ea019e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:36.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.725+0000 7f1ea5673640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ea010d9c0 0x7f1ea01a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:36.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.725+0000 7f1ea5673640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea0111960 con 0x7f1ea010d9c0 2024-03-28T01:05:36.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.725+0000 7f1ea5673640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea01117e0 con 0x7f1ea01086a0 2024-03-28T01:05:36.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.725+0000 7f1ea5673640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea0111ae0 con 0x7f1ea010b010 2024-03-28T01:05:36.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.726+0000 7f1e9e7fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ea010b010 0x7f1ea019e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:36.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.726+0000 7f1e9f7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ea010d9c0 0x7f1ea01a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:36.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.726+0000 7f1e9e7fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ea010b010 0x7f1ea019e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:42808/0 (socket says 172.21.15.169:42808) 2024-03-28T01:05:36.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.726+0000 7f1e9e7fc640 1 -- 172.21.15.169:0/3366423930 learned_addr learned my addr 172.21.15.169:0/3366423930 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:36.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.726+0000 7f1e9effd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ea01086a0 0x7f1ea019dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:36.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.727+0000 7f1e9effd640 1 -- 172.21.15.169:0/3366423930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ea010b010 msgr2=0x7f1ea019e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:36.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.727+0000 7f1e9effd640 1 --2- 172.21.15.169:0/3366423930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ea010b010 0x7f1ea019e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.727+0000 7f1e9effd640 1 -- 172.21.15.169:0/3366423930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ea010d9c0 msgr2=0x7f1ea01a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:36.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.727+0000 7f1e9effd640 1 --2- 172.21.15.169:0/3366423930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ea010d9c0 0x7f1ea01a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.727+0000 7f1e9effd640 1 -- 172.21.15.169:0/3366423930 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea01a5c00 con 0x7f1ea01086a0 2024-03-28T01:05:36.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.727+0000 7f1e9effd640 1 --2- 172.21.15.169:0/3366423930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ea01086a0 0x7f1ea019dfe0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1e9000e9f0 tx=0x7f1e9000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:36.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.730+0000 7f1e87fff640 1 -- 172.21.15.169:0/3366423930 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9000cd80 con 0x7f1ea01086a0 2024-03-28T01:05:36.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.730+0000 7f1e87fff640 1 -- 172.21.15.169:0/3366423930 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e90011950 con 0x7f1ea01086a0 2024-03-28T01:05:36.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.731+0000 7f1e87fff640 1 -- 172.21.15.169:0/3366423930 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9001aa10 con 0x7f1ea01086a0 2024-03-28T01:05:36.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.732+0000 7f1ea5673640 1 -- 172.21.15.169:0/3366423930 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea01a5ef0 con 0x7f1ea01086a0 2024-03-28T01:05:36.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.732+0000 7f1ea5673640 1 -- 172.21.15.169:0/3366423930 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea01a6300 con 0x7f1ea01086a0 2024-03-28T01:05:36.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.736+0000 7f1ea825d740 1 Processor -- start 2024-03-28T01:05:36.740 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.738+0000 7f1ea825d740 1 -- start start 2024-03-28T01:05:36.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.738+0000 7f1e87fff640 1 -- 172.21.15.169:0/3366423930 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1e9001abb0 con 0x7f1ea01086a0 2024-03-28T01:05:36.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.738+0000 7f1e87fff640 1 --2- 172.21.15.169:0/3366423930 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1e880a2a00 0x7f1e880a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:36.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.739+0000 7f1ea825d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560096866690 0x560096866ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:36.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.739+0000 7f1ea825d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 0x56009686bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:36.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.739+0000 7f1ea825d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56009686c1f0 0x56009686e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:36.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.739+0000 7f1ea825d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x560096829610 con 0x56009686c1f0 2024-03-28T01:05:36.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.739+0000 7f1ea825d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600967cc3c0 con 0x560096867080 2024-03-28T01:05:36.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.739+0000 7f1ea825d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560096774d60 con 0x560096866690 2024-03-28T01:05:36.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.739+0000 7f1e9effd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560096866690 0x560096866ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:36.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.739+0000 7f1e9effd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560096866690 0x560096866ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:42814/0 (socket says 172.21.15.169:42814) 2024-03-28T01:05:36.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.739+0000 7f1e9effd640 1 -- 172.21.15.169:0/1430676032 learned_addr learned my addr 172.21.15.169:0/1430676032 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:36.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.739+0000 7f1e9e7fc640 1 --2- 172.21.15.169:0/1430676032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 0x56009686bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:36.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.739+0000 7f1e87fff640 1 -- 172.21.15.169:0/3366423930 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f1e900d01d0 con 0x7f1ea01086a0 2024-03-28T01:05:36.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.739+0000 7f1e9effd640 1 -- 172.21.15.169:0/1430676032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 msgr2=0x56009686bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:36.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.739+0000 7f1e9effd640 1 --2- 172.21.15.169:0/1430676032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 0x56009686bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.740+0000 7f1e9effd640 1 -- 172.21.15.169:0/1430676032 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56009686c1f0 msgr2=0x56009686e680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:05:36.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.740+0000 7f1e9effd640 1 --2- 172.21.15.169:0/1430676032 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56009686c1f0 0x56009686e680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.740+0000 7f1e9effd640 1 -- 172.21.15.169:0/1430676032 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600964e03f0 con 0x560096866690 2024-03-28T01:05:36.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.740+0000 7f1e9effd640 1 --2- 172.21.15.169:0/1430676032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560096866690 0x560096866ab0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1e90098bd0 tx=0x7f1e90007f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f3544d867a163ce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:36.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.743+0000 7f1e857fa640 1 -- 172.21.15.169:0/1430676032 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9009c2b0 con 0x560096866690 2024-03-28T01:05:36.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.743+0000 7f1e857fa640 1 -- 172.21.15.169:0/1430676032 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e9009ce70 con 0x560096866690 2024-03-28T01:05:36.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.743+0000 7f1e857fa640 1 -- 172.21.15.169:0/1430676032 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e90003970 con 0x560096866690 2024-03-28T01:05:36.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.743+0000 7f1e9f7fe640 1 --2- 172.21.15.169:0/3366423930 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1e880a2a00 0x7f1e880a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:36.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.745+0000 7f1ea825d740 1 -- 172.21.15.169:0/1430676032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560096866690 msgr2=0x560096866ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:36.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.745+0000 7f1ea825d740 1 --2- 172.21.15.169:0/1430676032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560096866690 0x560096866ab0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1e90098bd0 tx=0x7f1e90007f70 comp rx=0 tx=0).stop 2024-03-28T01:05:36.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.749+0000 7f1e9f7fe640 1 --2- 172.21.15.169:0/3366423930 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1e880a2a00 0x7f1e880a4ec0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1e98005fd0 tx=0x7f1e98005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:36.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.749+0000 7f1ea825d740 1 -- 172.21.15.169:0/1430676032 shutdown_connections 2024-03-28T01:05:36.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.749+0000 7f1ea825d740 1 --2- 172.21.15.169:0/1430676032 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56009686c1f0 0x56009686e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.749+0000 7f1ea825d740 1 --2- 172.21.15.169:0/1430676032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 0x56009686bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.749+0000 7f1ea825d740 1 --2- 172.21.15.169:0/1430676032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560096866690 0x560096866ab0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.749+0000 7f1ea825d740 1 -- 172.21.15.169:0/1430676032 >> 172.21.15.169:0/1430676032 conn(0x560096861c00 msgr2=0x560096862000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:36.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.751+0000 7f1ea825d740 1 -- 172.21.15.169:0/1430676032 shutdown_connections 2024-03-28T01:05:36.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.753+0000 7f1ea825d740 1 -- 172.21.15.169:0/1430676032 wait complete. 2024-03-28T01:05:36.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.754+0000 7f1ea825d740 1 -- start start 2024-03-28T01:05:36.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.754+0000 7f1ea825d740 1 Processor -- start 2024-03-28T01:05:36.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.754+0000 7f1ea825d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560096866690 0x560096870a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:36.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.754+0000 7f1ea825d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 0x560096870f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:36.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.754+0000 7f1ea825d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56009686c1f0 0x560096877f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:36.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.754+0000 7f1ea825d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x560096788240 con 0x560096866690 2024-03-28T01:05:36.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.754+0000 7f1ea825d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600967c8460 con 0x560096867080 2024-03-28T01:05:36.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.754+0000 7f1ea825d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56009686f1a0 con 0x56009686c1f0 2024-03-28T01:05:36.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.754+0000 7f1ea825d740 10 client.0 _ll_register_callbacks cb 0x560096787c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:36.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.755+0000 7f1e9f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56009686c1f0 0x560096877f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:36.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.755+0000 7f1e9f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56009686c1f0 0x560096877f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:42828/0 (socket says 172.21.15.169:42828) 2024-03-28T01:05:36.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.755+0000 7f1e9f7fe640 1 -- 172.21.15.169:0/3923097033 learned_addr learned my addr 172.21.15.169:0/3923097033 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:36.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.755+0000 7f1e9f7fe640 1 -- 172.21.15.169:0/3923097033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 msgr2=0x560096870f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:36.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.755+0000 7f1e9e7fc640 1 --2- 172.21.15.169:0/3923097033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 0x560096870f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:36.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.755+0000 7f1e9f7fe640 1 --2- 172.21.15.169:0/3923097033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 0x560096870f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.755+0000 7f1e9f7fe640 1 -- 172.21.15.169:0/3923097033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560096866690 msgr2=0x560096870a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:36.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.755+0000 7f1e9f7fe640 1 --2- 172.21.15.169:0/3923097033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560096866690 0x560096870a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.755+0000 7f1e9f7fe640 1 -- 172.21.15.169:0/3923097033 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5600965aa7d0 con 0x56009686c1f0 2024-03-28T01:05:36.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.755+0000 7f1e9e7fc640 1 --2- 172.21.15.169:0/3923097033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 0x560096870f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:05:36.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.755+0000 7f1e9f7fe640 1 --2- 172.21.15.169:0/3923097033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56009686c1f0 0x560096877f80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1e9800ffa0 tx=0x7f1e980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ae7ae27d5193a00 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:36.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.757+0000 7f1e82ff5640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:05:36.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.757+0000 7f1e82ff5640 1 -- 172.21.15.169:0/3923097033 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e98021070 con 0x56009686c1f0 2024-03-28T01:05:36.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.757+0000 7f1ea825d740 1 -- 172.21.15.169:0/3923097033 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1e9009ce70 con 0x56009686c1f0 2024-03-28T01:05:36.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.757+0000 7f1e82ff5640 1 -- 172.21.15.169:0/3923097033 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f1e9801d830 con 0x56009686c1f0 2024-03-28T01:05:36.763 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.761+0000 7f1e82ff5640 1 -- 172.21.15.169:0/3923097033 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e9801da30 con 0x56009686c1f0 2024-03-28T01:05:36.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.761+0000 7f1ea825d740 10 client.27007 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:36.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.761+0000 7f1ea825d740 1 -- 172.21.15.169:0/3923097033 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56009687b3c0 con 0x56009686c1f0 2024-03-28T01:05:36.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.765+0000 7f1e82ff5640 1 -- 172.21.15.169:0/3923097033 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e9009ce70 con 0x56009686c1f0 2024-03-28T01:05:36.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.765+0000 7f1e82ff5640 1 -- 172.21.15.169:0/3923097033 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f1e98030420 con 0x56009686c1f0 2024-03-28T01:05:36.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.765+0000 7f1ea825d740 10 client.27007 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:36.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.765+0000 7f1ea825d740 10 client.27007 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:05:36.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.765+0000 7f1ea825d740 20 client.27007 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:36.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.765+0000 7f1ea825d740 1 --2- 172.21.15.169:0/3923097033 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x56009687ced0 0x56009687f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:36.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.765+0000 7f1ea825d740 4 client.27007 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:05:36.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.765+0000 7f1ea825d740 1 -- 172.21.15.169:0/3923097033 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5600967d9900 con 0x56009687ced0 2024-03-28T01:05:36.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.766+0000 7f1e9effd640 1 --2- 172.21.15.169:0/3923097033 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x56009687ced0 0x56009687f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:36.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.768+0000 7f1e9effd640 1 --2- 172.21.15.169:0/3923097033 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x56009687ced0 0x56009687f370 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f98fafe12a8d0dcf server_cookie=d2eb7f99fa62195b in_seq=0 out_seq=0 2024-03-28T01:05:36.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.770+0000 7f1e82ff5640 10 client.27007 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:05:36.777 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.772+0000 7f1e82ff5640 1 -- 172.21.15.169:0/3923097033 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5600967d9900 con 0x56009687ced0 2024-03-28T01:05:36.777 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.772+0000 7f1e82ff5640 10 client.27007 handle_command_reply: tid=0 2024-03-28T01:05:36.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.772+0000 7f1ea825d740 1 client.27007 shutdown 2024-03-28T01:05:36.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.774+0000 7f1ea825d740 1 -- 172.21.15.169:0/3923097033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56009686c1f0 msgr2=0x560096877f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:36.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.774+0000 7f1ea825d740 1 --2- 172.21.15.169:0/3923097033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56009686c1f0 0x560096877f80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1e9800ffa0 tx=0x7f1e980046c0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.774+0000 7f1ea825d740 1 -- 172.21.15.169:0/3923097033 shutdown_connections 2024-03-28T01:05:36.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.774+0000 7f1ea825d740 1 --2- 172.21.15.169:0/3923097033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56009686c1f0 0x560096877f80 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.774+0000 7f1ea825d740 1 --2- 172.21.15.169:0/3923097033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 0x560096870f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.774+0000 7f1ea825d740 1 --2- 172.21.15.169:0/3923097033 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x56009687ced0 0x56009687f370 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.774+0000 7f1ea825d740 1 --2- 172.21.15.169:0/3923097033 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560096866690 0x560096870a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.774+0000 7f1ea825d740 1 -- 172.21.15.169:0/3923097033 >> 172.21.15.169:0/3923097033 conn(0x560096861c00 msgr2=0x560096860e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:36.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.774+0000 7f1e82ff5640 0 client.27007 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:05:36.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.782+0000 7f1ea825d740 1 -- 172.21.15.169:0/3923097033 shutdown_connections 2024-03-28T01:05:36.788 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.784+0000 7f1ea825d740 1 -- 172.21.15.169:0/3923097033 wait complete. 2024-03-28T01:05:36.789 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.786+0000 7f1ea825d740 20 client.27007 trim_cache size 0 max 16384 2024-03-28T01:05:36.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:36 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3000373631' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:05:36.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.830+0000 7f1ea825d740 1 Processor -- start 2024-03-28T01:05:36.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.833+0000 7f1ea825d740 1 -- start start 2024-03-28T01:05:36.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.833+0000 7f1ea825d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560096866690 0x560096870a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:36.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.833+0000 7f1ea825d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 0x560096870f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:36.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.834+0000 7f1ea825d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56009686c1f0 0x560096877ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:36.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.834+0000 7f1e9effd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560096866690 0x560096870a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:36.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.834+0000 7f1ea825d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c0120e0 con 0x560096866690 2024-03-28T01:05:36.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.834+0000 7f1e9effd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560096866690 0x560096870a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:57790/0 (socket says 172.21.15.169:57790) 2024-03-28T01:05:36.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.834+0000 7f1e9effd640 1 -- 172.21.15.169:0/3525989923 learned_addr learned my addr 172.21.15.169:0/3525989923 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:36.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.834+0000 7f1e9e7fc640 1 --2- 172.21.15.169:0/3525989923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 0x560096870f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:36.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.834+0000 7f1e9f7fe640 1 --2- 172.21.15.169:0/3525989923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56009686c1f0 0x560096877ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:36.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.834+0000 7f1ea825d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c00fbe0 con 0x560096867080 2024-03-28T01:05:36.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.834+0000 7f1ea825d740 1 -- 172.21.15.169:0/3525989923 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c00dc80 con 0x56009686c1f0 2024-03-28T01:05:36.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.834+0000 7f1e9e7fc640 1 -- 172.21.15.169:0/3525989923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56009686c1f0 msgr2=0x560096877ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:36.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.834+0000 7f1e9e7fc640 1 --2- 172.21.15.169:0/3525989923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56009686c1f0 0x560096877ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.834+0000 7f1e9e7fc640 1 -- 172.21.15.169:0/3525989923 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560096866690 msgr2=0x560096870a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:36.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.834+0000 7f1e9e7fc640 1 --2- 172.21.15.169:0/3525989923 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560096866690 0x560096870a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.834+0000 7f1e9e7fc640 1 -- 172.21.15.169:0/3525989923 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560096861a30 con 0x560096867080 2024-03-28T01:05:36.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.835+0000 7f1e9e7fc640 1 --2- 172.21.15.169:0/3525989923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 0x560096870f40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1e94006010 tx=0x7f1e94004920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7d87abd8b68a2f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:36.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.837+0000 7f1e84ff9640 1 -- 172.21.15.169:0/3525989923 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e940042b0 con 0x560096867080 2024-03-28T01:05:36.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.838+0000 7f1e84ff9640 1 -- 172.21.15.169:0/3525989923 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e94004450 con 0x560096867080 2024-03-28T01:05:36.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.839+0000 7f1ea825d740 1 -- 172.21.15.169:0/3525989923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 msgr2=0x560096870f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:36.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.839+0000 7f1ea825d740 1 --2- 172.21.15.169:0/3525989923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 0x560096870f40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1e94006010 tx=0x7f1e94004920 comp rx=0 tx=0).stop 2024-03-28T01:05:36.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.840+0000 7f1ea825d740 1 -- 172.21.15.169:0/3525989923 shutdown_connections 2024-03-28T01:05:36.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.840+0000 7f1ea825d740 1 --2- 172.21.15.169:0/3525989923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56009686c1f0 0x560096877ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.840+0000 7f1ea825d740 1 --2- 172.21.15.169:0/3525989923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 0x560096870f40 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.840+0000 7f1ea825d740 1 --2- 172.21.15.169:0/3525989923 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560096866690 0x560096870a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.840+0000 7f1ea825d740 1 -- 172.21.15.169:0/3525989923 >> 172.21.15.169:0/3525989923 conn(0x560096861c00 msgr2=0x560096861fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:36.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.842+0000 7f1ea825d740 1 -- 172.21.15.169:0/3525989923 shutdown_connections 2024-03-28T01:05:36.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.842+0000 7f1ea825d740 1 -- 172.21.15.169:0/3525989923 wait complete. 2024-03-28T01:05:36.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.843+0000 7f1ea825d740 1 -- start start 2024-03-28T01:05:36.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.843+0000 7f1ea825d740 1 Processor -- start 2024-03-28T01:05:36.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.843+0000 7f1ea825d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560096866690 0x5600968755c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:36.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.843+0000 7f1ea825d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 0x560096875b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:36.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.843+0000 7f1ea825d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56009686c1f0 0x560096876b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:36.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.843+0000 7f1ea825d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c00e280 con 0x560096866690 2024-03-28T01:05:36.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.843+0000 7f1ea825d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c00f580 con 0x560096867080 2024-03-28T01:05:36.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.843+0000 7f1ea825d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c00e8c0 con 0x56009686c1f0 2024-03-28T01:05:36.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.843+0000 7f1ea825d740 10 client.0 _ll_register_callbacks cb 0x7f1e9800fc50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:36.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.844+0000 7f1e9f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56009686c1f0 0x560096876b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:36.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.844+0000 7f1e9f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56009686c1f0 0x560096876b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:42846/0 (socket says 172.21.15.169:42846) 2024-03-28T01:05:36.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.844+0000 7f1e9f7fe640 1 -- 172.21.15.169:0/2767651847 learned_addr learned my addr 172.21.15.169:0/2767651847 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:36.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.844+0000 7f1e9e7fc640 1 --2- 172.21.15.169:0/2767651847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 0x560096875b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:36.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.844+0000 7f1e9f7fe640 1 -- 172.21.15.169:0/2767651847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 msgr2=0x560096875b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:36.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.844+0000 7f1e9f7fe640 1 --2- 172.21.15.169:0/2767651847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 0x560096875b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.844+0000 7f1e9f7fe640 1 -- 172.21.15.169:0/2767651847 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560096866690 msgr2=0x5600968755c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:36.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.844+0000 7f1e9f7fe640 1 --2- 172.21.15.169:0/2767651847 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560096866690 0x5600968755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.844+0000 7f1e9f7fe640 1 -- 172.21.15.169:0/2767651847 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560096877100 con 0x56009686c1f0 2024-03-28T01:05:36.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.845+0000 7f1e9f7fe640 1 --2- 172.21.15.169:0/2767651847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56009686c1f0 0x560096876b10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1e980058c0 tx=0x7f1e9800fa80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f9e46fb82148f77 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:36.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.846+0000 7f1e82ff5640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:05:36.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.846+0000 7f1e82ff5640 1 -- 172.21.15.169:0/2767651847 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9803f020 con 0x56009686c1f0 2024-03-28T01:05:36.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.846+0000 7f1ea825d740 1 -- 172.21.15.169:0/2767651847 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1e94004450 con 0x56009686c1f0 2024-03-28T01:05:36.850 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.846+0000 7f1e82ff5640 1 -- 172.21.15.169:0/2767651847 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f1e9801ddd0 con 0x56009686c1f0 2024-03-28T01:05:36.850 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.846+0000 7f1e82ff5640 1 -- 172.21.15.169:0/2767651847 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e98047050 con 0x56009686c1f0 2024-03-28T01:05:36.850 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.848+0000 7f1ea825d740 10 client.27013 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:36.850 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.848+0000 7f1ea825d740 1 -- 172.21.15.169:0/2767651847 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5600968d8800 con 0x56009686c1f0 2024-03-28T01:05:36.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.851+0000 7f1e82ff5640 1 -- 172.21.15.169:0/2767651847 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e94004450 con 0x56009686c1f0 2024-03-28T01:05:36.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.851+0000 7f1e82ff5640 1 -- 172.21.15.169:0/2767651847 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f1e98004ce0 con 0x56009686c1f0 2024-03-28T01:05:36.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.853+0000 7f1ea825d740 10 client.27013 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:36.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.853+0000 7f1ea825d740 10 client.27013 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:05:36.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.853+0000 7f1ea825d740 20 client.27013 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:36.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.853+0000 7f1ea825d740 1 --2- 172.21.15.169:0/2767651847 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x5600968d9070 0x5600968db4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:36.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.853+0000 7f1ea825d740 4 client.27013 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:05:36.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.853+0000 7f1ea825d740 1 -- 172.21.15.169:0/2767651847 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5600967d3c50 con 0x5600968d9070 2024-03-28T01:05:36.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.853+0000 7f1e9effd640 1 --2- 172.21.15.169:0/2767651847 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x5600968d9070 0x5600968db4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:36.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.855+0000 7f1e9effd640 1 --2- 172.21.15.169:0/2767651847 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x5600968d9070 0x5600968db4b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c6cdbee8402d4379 server_cookie=bd664afc17f3b1f5 in_seq=0 out_seq=0 2024-03-28T01:05:36.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.855+0000 7f1e82ff5640 10 client.27013 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:05:36.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.861+0000 7f1e82ff5640 1 -- 172.21.15.169:0/2767651847 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5600967d3c50 con 0x5600968d9070 2024-03-28T01:05:36.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.861+0000 7f1e82ff5640 10 client.27013 handle_command_reply: tid=0 2024-03-28T01:05:36.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.862+0000 7f1ea825d740 1 client.27013 shutdown 2024-03-28T01:05:36.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.871+0000 7f1ea825d740 1 -- 172.21.15.169:0/2767651847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56009686c1f0 msgr2=0x560096876b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:36.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.871+0000 7f1ea825d740 1 --2- 172.21.15.169:0/2767651847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56009686c1f0 0x560096876b10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1e980058c0 tx=0x7f1e9800fa80 comp rx=0 tx=0).stop 2024-03-28T01:05:36.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.872+0000 7f1ea825d740 1 -- 172.21.15.169:0/2767651847 shutdown_connections 2024-03-28T01:05:36.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.872+0000 7f1ea825d740 1 --2- 172.21.15.169:0/2767651847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56009686c1f0 0x560096876b10 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.872+0000 7f1ea825d740 1 --2- 172.21.15.169:0/2767651847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560096867080 0x560096875b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.872+0000 7f1ea825d740 1 --2- 172.21.15.169:0/2767651847 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x5600968d9070 0x5600968db4b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.872+0000 7f1ea825d740 1 --2- 172.21.15.169:0/2767651847 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560096866690 0x5600968755c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.872+0000 7f1ea825d740 1 -- 172.21.15.169:0/2767651847 >> 172.21.15.169:0/2767651847 conn(0x560096861c00 msgr2=0x560096868580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:36.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.872+0000 7f1e82ff5640 0 client.27013 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:05:36.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.872+0000 7f1ea825d740 1 -- 172.21.15.169:0/2767651847 shutdown_connections 2024-03-28T01:05:36.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.872+0000 7f1ea825d740 1 -- 172.21.15.169:0/2767651847 wait complete. 2024-03-28T01:05:36.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.877+0000 7f1ea825d740 20 client.27013 trim_cache size 0 max 16384 2024-03-28T01:05:36.891 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:05:36.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.900+0000 7f1ea5673640 1 -- 172.21.15.169:0/3366423930 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1e880a2a00 msgr2=0x7f1e880a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:36.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.900+0000 7f1ea5673640 1 --2- 172.21.15.169:0/3366423930 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1e880a2a00 0x7f1e880a4ec0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1e98005fd0 tx=0x7f1e98005850 comp rx=0 tx=0).stop 2024-03-28T01:05:36.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.900+0000 7f1ea5673640 1 -- 172.21.15.169:0/3366423930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ea01086a0 msgr2=0x7f1ea019dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:36.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.900+0000 7f1ea5673640 1 --2- 172.21.15.169:0/3366423930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ea01086a0 0x7f1ea019dfe0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1e9000e9f0 tx=0x7f1e9000eeb0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.900+0000 7f1ea5673640 1 -- 172.21.15.169:0/3366423930 shutdown_connections 2024-03-28T01:05:36.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.900+0000 7f1ea5673640 1 --2- 172.21.15.169:0/3366423930 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f1e880a2a00 0x7f1e880a4ec0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.900+0000 7f1ea5673640 1 --2- 172.21.15.169:0/3366423930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1ea010d9c0 0x7f1ea01a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.900+0000 7f1ea5673640 1 --2- 172.21.15.169:0/3366423930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ea010b010 0x7f1ea019e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.900+0000 7f1ea5673640 1 --2- 172.21.15.169:0/3366423930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ea01086a0 0x7f1ea019dfe0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:36.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.900+0000 7f1ea5673640 1 -- 172.21.15.169:0/3366423930 >> 172.21.15.169:0/3366423930 conn(0x7f1ea006e500 msgr2=0x7f1ea010b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:36.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.901+0000 7f1ea5673640 1 -- 172.21.15.169:0/3366423930 shutdown_connections 2024-03-28T01:05:36.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:36.901+0000 7f1ea5673640 1 -- 172.21.15.169:0/3366423930 wait complete. 2024-03-28T01:05:36.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:36 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3000373631' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:05:36.961 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:05:36.961 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T01:05:37.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:36 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3000373631' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:05:37.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.249+0000 7f952959a640 1 -- 172.21.15.169:0/1153717981 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9520032340 con 0x7f952c10b010 2024-03-28T01:05:37.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.249+0000 7f9530fb3640 1 -- 172.21.15.169:0/1153717981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f952c10b010 msgr2=0x7f952c10d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:37.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.249+0000 7f9530fb3640 1 --2- 172.21.15.169:0/1153717981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f952c10b010 0x7f952c10d480 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9520009860 tx=0x7f952002bbb0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.250+0000 7f9530fb3640 1 -- 172.21.15.169:0/1153717981 shutdown_connections 2024-03-28T01:05:37.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.250+0000 7f9530fb3640 1 --2- 172.21.15.169:0/1153717981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f952c10d9c0 0x7f952c10de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.250+0000 7f9530fb3640 1 --2- 172.21.15.169:0/1153717981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f952c10b010 0x7f952c10d480 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.250+0000 7f9530fb3640 1 --2- 172.21.15.169:0/1153717981 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f952c1086a0 0x7f952c10aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.250+0000 7f9530fb3640 1 -- 172.21.15.169:0/1153717981 >> 172.21.15.169:0/1153717981 conn(0x7f952c06e500 msgr2=0x7f952c070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:37.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.251+0000 7f9530fb3640 1 -- 172.21.15.169:0/1153717981 shutdown_connections 2024-03-28T01:05:37.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.255+0000 7f9530fb3640 1 -- 172.21.15.169:0/1153717981 wait complete. 2024-03-28T01:05:37.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.256+0000 7f9530fb3640 1 Processor -- start 2024-03-28T01:05:37.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.257+0000 7f9530fb3640 1 -- start start 2024-03-28T01:05:37.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.257+0000 7f9530fb3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f952c1086a0 0x7f952c1a67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.257+0000 7f9530fb3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f952c10b010 0x7f952c1a6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.257+0000 7f9530fb3640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f952c10d9c0 0x7f952c1adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.257+0000 7f9530fb3640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c11c280 con 0x7f952c10d9c0 2024-03-28T01:05:37.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.257+0000 7f9530fb3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c11c100 con 0x7f952c10b010 2024-03-28T01:05:37.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.257+0000 7f9530fb3640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c11c400 con 0x7f952c1086a0 2024-03-28T01:05:37.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.257+0000 7f952a59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f952c1086a0 0x7f952c1a67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.260+0000 7f952a59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f952c1086a0 0x7f952c1a67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:42870/0 (socket says 172.21.15.169:42870) 2024-03-28T01:05:37.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.260+0000 7f952a59c640 1 -- 172.21.15.169:0/2051405938 learned_addr learned my addr 172.21.15.169:0/2051405938 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:37.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.260+0000 7f952ad9d640 1 --2- 172.21.15.169:0/2051405938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f952c10d9c0 0x7f952c1adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.261+0000 7f952a59c640 1 -- 172.21.15.169:0/2051405938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f952c10b010 msgr2=0x7f952c1a6d30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:05:37.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.261+0000 7f952a59c640 1 --2- 172.21.15.169:0/2051405938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f952c10b010 0x7f952c1a6d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.261+0000 7f952a59c640 1 -- 172.21.15.169:0/2051405938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f952c10d9c0 msgr2=0x7f952c1adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:37.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.261+0000 7f952a59c640 1 --2- 172.21.15.169:0/2051405938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f952c10d9c0 0x7f952c1adda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.261+0000 7f952a59c640 1 -- 172.21.15.169:0/2051405938 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952c10ff00 con 0x7f952c1086a0 2024-03-28T01:05:37.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.264+0000 7f952a59c640 1 --2- 172.21.15.169:0/2051405938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f952c1086a0 0x7f952c1a67f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f952400efb0 tx=0x7f952400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:37.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.268+0000 7f951b7fe640 1 -- 172.21.15.169:0/2051405938 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952401b070 con 0x7f952c1086a0 2024-03-28T01:05:37.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.268+0000 7f9530fb3640 1 -- 172.21.15.169:0/2051405938 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f952c1101f0 con 0x7f952c1086a0 2024-03-28T01:05:37.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.268+0000 7f9530fb3640 1 -- 172.21.15.169:0/2051405938 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f952c110740 con 0x7f952c1086a0 2024-03-28T01:05:37.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.269+0000 7f951b7fe640 1 -- 172.21.15.169:0/2051405938 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f952400ba50 con 0x7f952c1086a0 2024-03-28T01:05:37.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.269+0000 7f951b7fe640 1 -- 172.21.15.169:0/2051405938 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9524016c60 con 0x7f952c1086a0 2024-03-28T01:05:37.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.272+0000 7f951b7fe640 1 -- 172.21.15.169:0/2051405938 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f952401e070 con 0x7f952c1086a0 2024-03-28T01:05:37.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.274+0000 7f951b7fe640 1 --2- 172.21.15.169:0/2051405938 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f95140a29d0 0x7f95140a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.274+0000 7f951b7fe640 1 -- 172.21.15.169:0/2051405938 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f95240ced20 con 0x7f952c1086a0 2024-03-28T01:05:37.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.278+0000 7f9529d9b640 1 --2- 172.21.15.169:0/2051405938 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f95140a29d0 0x7f95140a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.278+0000 7f9533b9a740 1 Processor -- start 2024-03-28T01:05:37.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.278+0000 7f9533b9a740 1 -- start start 2024-03-28T01:05:37.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.279+0000 7f9533b9a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c505791690 0x55c505791ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.279+0000 7f9533b9a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c505792080 0x55c505796b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.279+0000 7f9533b9a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c5057971f0 0x55c505799680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.279+0000 7f9533b9a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c505754610 con 0x55c5057971f0 2024-03-28T01:05:37.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.279+0000 7f9533b9a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5056f73c0 con 0x55c505792080 2024-03-28T01:05:37.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.279+0000 7f9533b9a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c50569fd60 con 0x55c505791690 2024-03-28T01:05:37.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.279+0000 7f952ad9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c5057971f0 0x55c505799680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.279+0000 7f952ad9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c5057971f0 0x55c505799680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:57818/0 (socket says 172.21.15.169:57818) 2024-03-28T01:05:37.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.279+0000 7f952ad9d640 1 -- 172.21.15.169:0/3151103474 learned_addr learned my addr 172.21.15.169:0/3151103474 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:37.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.281+0000 7f952ad9d640 1 -- 172.21.15.169:0/3151103474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c505791690 msgr2=0x55c505791ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:05:37.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.281+0000 7f952ad9d640 1 --2- 172.21.15.169:0/3151103474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c505791690 0x55c505791ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.281+0000 7f952ad9d640 1 -- 172.21.15.169:0/3151103474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c505792080 msgr2=0x55c505796b80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:05:37.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.281+0000 7f952ad9d640 1 --2- 172.21.15.169:0/3151103474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c505792080 0x55c505796b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.281+0000 7f952ad9d640 1 -- 172.21.15.169:0/3151103474 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c50540b3f0 con 0x55c5057971f0 2024-03-28T01:05:37.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.284+0000 7f952ad9d640 1 --2- 172.21.15.169:0/3151103474 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c5057971f0 0x55c505799680 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f951c00cb10 tx=0x7f951c00e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa20c88d1427f2c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:37.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.285+0000 7f9518ff9640 1 -- 172.21.15.169:0/3151103474 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c004020 con 0x55c5057971f0 2024-03-28T01:05:37.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.285+0000 7f9518ff9640 1 -- 172.21.15.169:0/3151103474 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f951c0041c0 con 0x55c5057971f0 2024-03-28T01:05:37.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.286+0000 7f9533b9a740 1 -- 172.21.15.169:0/3151103474 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c5057971f0 msgr2=0x55c505799680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:37.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.286+0000 7f9533b9a740 1 --2- 172.21.15.169:0/3151103474 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c5057971f0 0x55c505799680 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f951c00cb10 tx=0x7f951c00e350 comp rx=0 tx=0).stop 2024-03-28T01:05:37.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.287+0000 7f9533b9a740 1 -- 172.21.15.169:0/3151103474 shutdown_connections 2024-03-28T01:05:37.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.287+0000 7f9533b9a740 1 --2- 172.21.15.169:0/3151103474 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c5057971f0 0x55c505799680 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.287+0000 7f9533b9a740 1 --2- 172.21.15.169:0/3151103474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c505792080 0x55c505796b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.287+0000 7f9533b9a740 1 --2- 172.21.15.169:0/3151103474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c505791690 0x55c505791ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.287+0000 7f9533b9a740 1 -- 172.21.15.169:0/3151103474 >> 172.21.15.169:0/3151103474 conn(0x55c50578cc00 msgr2=0x55c50578d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:37.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.289+0000 7f9529d9b640 1 --2- 172.21.15.169:0/2051405938 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f95140a29d0 0x7f95140a4e90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9520037040 tx=0x7f9520000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:37.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.290+0000 7f9533b9a740 1 -- 172.21.15.169:0/3151103474 shutdown_connections 2024-03-28T01:05:37.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.290+0000 7f9533b9a740 1 -- 172.21.15.169:0/3151103474 wait complete. 2024-03-28T01:05:37.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.291+0000 7f9533b9a740 1 -- start start 2024-03-28T01:05:37.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.291+0000 7f9533b9a740 1 Processor -- start 2024-03-28T01:05:37.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.295+0000 7f9533b9a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c505791690 0x55c50579ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.295+0000 7f9533b9a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c505792080 0x55c50579bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.295+0000 7f9533b9a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5057971f0 0x55c5057a2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.295+0000 7f9533b9a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5056b3240 con 0x55c505791690 2024-03-28T01:05:37.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.295+0000 7f9533b9a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5056f3460 con 0x55c5057971f0 2024-03-28T01:05:37.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.295+0000 7f9533b9a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c50579a1f0 con 0x55c505792080 2024-03-28T01:05:37.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.295+0000 7f9533b9a740 10 client.0 _ll_register_callbacks cb 0x55c5056b2c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:37.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.296+0000 7f952ad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5057971f0 0x55c5057a2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.296+0000 7f952a59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c505791690 0x55c50579ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.296+0000 7f952a59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c505791690 0x55c50579ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:57830/0 (socket says 172.21.15.169:57830) 2024-03-28T01:05:37.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.296+0000 7f952a59c640 1 -- 172.21.15.169:0/958348471 learned_addr learned my addr 172.21.15.169:0/958348471 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:37.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.296+0000 7f9529d9b640 1 --2- 172.21.15.169:0/958348471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c505792080 0x55c50579bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.296+0000 7f952ad9d640 1 -- 172.21.15.169:0/958348471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c505792080 msgr2=0x55c50579bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:37.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.296+0000 7f952ad9d640 1 --2- 172.21.15.169:0/958348471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c505792080 0x55c50579bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.296+0000 7f952ad9d640 1 -- 172.21.15.169:0/958348471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c505791690 msgr2=0x55c50579ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:37.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.296+0000 7f952ad9d640 1 --2- 172.21.15.169:0/958348471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c505791690 0x55c50579ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.296+0000 7f952ad9d640 1 -- 172.21.15.169:0/958348471 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c5054d57d0 con 0x55c5057971f0 2024-03-28T01:05:37.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.296+0000 7f9529d9b640 1 --2- 172.21.15.169:0/958348471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c505792080 0x55c50579bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:05:37.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.297+0000 7f952ad9d640 1 --2- 172.21.15.169:0/958348471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5057971f0 0x55c5057a2fb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f951c00cae0 tx=0x7f951c003c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66722ae708f64d1a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:37.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.298+0000 7f9511ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:05:37.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.298+0000 7f9511ffb640 1 -- 172.21.15.169:0/958348471 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c00c6e0 con 0x55c5057971f0 2024-03-28T01:05:37.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.298+0000 7f9533b9a740 1 -- 172.21.15.169:0/958348471 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f951c0041c0 con 0x55c5057971f0 2024-03-28T01:05:37.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.298+0000 7f9511ffb640 1 -- 172.21.15.169:0/958348471 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f951c00ed60 con 0x55c5057971f0 2024-03-28T01:05:37.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.300+0000 7f9511ffb640 1 -- 172.21.15.169:0/958348471 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f951c0109c0 con 0x55c5057971f0 2024-03-28T01:05:37.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.301+0000 7f9533b9a740 10 client.27074 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:37.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.301+0000 7f9533b9a740 1 -- 172.21.15.169:0/958348471 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c5057a6320 con 0x55c5057971f0 2024-03-28T01:05:37.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.302+0000 7f9511ffb640 1 -- 172.21.15.169:0/958348471 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f951c0041c0 con 0x55c5057971f0 2024-03-28T01:05:37.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.302+0000 7f9511ffb640 1 -- 172.21.15.169:0/958348471 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f951c005030 con 0x55c5057971f0 2024-03-28T01:05:37.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.303+0000 7f9533b9a740 10 client.27074 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:37.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.303+0000 7f9533b9a740 10 client.27074 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:05:37.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.303+0000 7f9533b9a740 20 client.27074 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:37.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.303+0000 7f9533b9a740 1 --2- 172.21.15.169:0/958348471 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55c5057a7d20 0x55c5057aa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.303+0000 7f9533b9a740 4 client.27074 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:05:37.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.303+0000 7f9533b9a740 1 -- 172.21.15.169:0/958348471 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c505704900 con 0x55c5057a7d20 2024-03-28T01:05:37.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.304+0000 7f952a59c640 1 --2- 172.21.15.169:0/958348471 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55c5057a7d20 0x55c5057aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.304+0000 7f952a59c640 1 --2- 172.21.15.169:0/958348471 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55c5057a7d20 0x55c5057aa140 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4e6fb70cd6f7e7d0 server_cookie=326208d7baddcd6b in_seq=0 out_seq=0 2024-03-28T01:05:37.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.304+0000 7f9511ffb640 10 client.27074 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:05:37.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.306+0000 7f9511ffb640 1 -- 172.21.15.169:0/958348471 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c505704900 con 0x55c5057a7d20 2024-03-28T01:05:37.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.306+0000 7f9511ffb640 10 client.27074 handle_command_reply: tid=0 2024-03-28T01:05:37.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.306+0000 7f9533b9a740 1 client.27074 shutdown 2024-03-28T01:05:37.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.310+0000 7f9533b9a740 1 -- 172.21.15.169:0/958348471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5057971f0 msgr2=0x55c5057a2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:37.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.310+0000 7f9533b9a740 1 --2- 172.21.15.169:0/958348471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5057971f0 0x55c5057a2fb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f951c00cae0 tx=0x7f951c003c50 comp rx=0 tx=0).stop 2024-03-28T01:05:37.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.310+0000 7f9533b9a740 1 -- 172.21.15.169:0/958348471 shutdown_connections 2024-03-28T01:05:37.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.310+0000 7f9533b9a740 1 --2- 172.21.15.169:0/958348471 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55c5057a7d20 0x55c5057aa140 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.310+0000 7f9533b9a740 1 --2- 172.21.15.169:0/958348471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5057971f0 0x55c5057a2fb0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.310+0000 7f9533b9a740 1 --2- 172.21.15.169:0/958348471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c505792080 0x55c50579bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.310+0000 7f9533b9a740 1 --2- 172.21.15.169:0/958348471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c505791690 0x55c50579ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.310+0000 7f9533b9a740 1 -- 172.21.15.169:0/958348471 >> 172.21.15.169:0/958348471 conn(0x55c50578cc00 msgr2=0x55c50578be20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:37.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.310+0000 7f9511ffb640 0 client.27074 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:05:37.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.311+0000 7f9533b9a740 1 -- 172.21.15.169:0/958348471 shutdown_connections 2024-03-28T01:05:37.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.311+0000 7f9533b9a740 1 -- 172.21.15.169:0/958348471 wait complete. 2024-03-28T01:05:37.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.311+0000 7f9533b9a740 20 client.27074 trim_cache size 0 max 16384 2024-03-28T01:05:37.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.339+0000 7f9533b9a740 1 Processor -- start 2024-03-28T01:05:37.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.339+0000 7f9533b9a740 1 -- start start 2024-03-28T01:05:37.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.339+0000 7f9533b9a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c505791690 0x55c505787cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.339+0000 7f9533b9a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c505792080 0x55c5057a2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.339+0000 7f9533b9a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5057971f0 0x55c505793580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.340+0000 7f9533b9a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c0120d0 con 0x55c505791690 2024-03-28T01:05:37.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.340+0000 7f9533b9a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c00fbd0 con 0x55c505792080 2024-03-28T01:05:37.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.340+0000 7f9533b9a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c00dc70 con 0x55c5057971f0 2024-03-28T01:05:37.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.340+0000 7f952ad9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5057971f0 0x55c505793580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.340+0000 7f952ad9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5057971f0 0x55c505793580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:42894/0 (socket says 172.21.15.169:42894) 2024-03-28T01:05:37.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.340+0000 7f952ad9d640 1 -- 172.21.15.169:0/3493092493 learned_addr learned my addr 172.21.15.169:0/3493092493 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:37.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.340+0000 7f9529d9b640 1 --2- 172.21.15.169:0/3493092493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c505792080 0x55c5057a2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.340+0000 7f952a59c640 1 --2- 172.21.15.169:0/3493092493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c505791690 0x55c505787cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.340+0000 7f952ad9d640 1 -- 172.21.15.169:0/3493092493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c505792080 msgr2=0x55c5057a2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:37.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.340+0000 7f952ad9d640 1 --2- 172.21.15.169:0/3493092493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c505792080 0x55c5057a2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.340+0000 7f952ad9d640 1 -- 172.21.15.169:0/3493092493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c505791690 msgr2=0x55c505787cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:37.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.340+0000 7f952ad9d640 1 --2- 172.21.15.169:0/3493092493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c505791690 0x55c505787cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.340+0000 7f952ad9d640 1 -- 172.21.15.169:0/3493092493 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c505792460 con 0x55c5057971f0 2024-03-28T01:05:37.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.340+0000 7f9529d9b640 1 --2- 172.21.15.169:0/3493092493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c505792080 0x55c5057a2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:05:37.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.341+0000 7f952ad9d640 1 --2- 172.21.15.169:0/3493092493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5057971f0 0x55c505793580 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f951c026040 tx=0x7f951c001e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63a3d340a276c098 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:37.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.341+0000 7f9513fff640 1 -- 172.21.15.169:0/3493092493 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c00c400 con 0x55c5057971f0 2024-03-28T01:05:37.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.341+0000 7f9513fff640 1 -- 172.21.15.169:0/3493092493 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f951c004190 con 0x55c5057971f0 2024-03-28T01:05:37.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.342+0000 7f9513fff640 1 -- 172.21.15.169:0/3493092493 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c010b60 con 0x55c5057971f0 2024-03-28T01:05:37.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.342+0000 7f9533b9a740 1 -- 172.21.15.169:0/3493092493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5057971f0 msgr2=0x55c505793580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:37.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.342+0000 7f9533b9a740 1 --2- 172.21.15.169:0/3493092493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5057971f0 0x55c505793580 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f951c026040 tx=0x7f951c001e50 comp rx=0 tx=0).stop 2024-03-28T01:05:37.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.342+0000 7f9533b9a740 1 -- 172.21.15.169:0/3493092493 shutdown_connections 2024-03-28T01:05:37.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.342+0000 7f9533b9a740 1 --2- 172.21.15.169:0/3493092493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5057971f0 0x55c505793580 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.342+0000 7f9533b9a740 1 --2- 172.21.15.169:0/3493092493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c505792080 0x55c5057a2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.342+0000 7f9533b9a740 1 --2- 172.21.15.169:0/3493092493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c505791690 0x55c505787cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.342+0000 7f9533b9a740 1 -- 172.21.15.169:0/3493092493 >> 172.21.15.169:0/3493092493 conn(0x55c50578cc00 msgr2=0x55c50578cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:37.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.342+0000 7f9533b9a740 1 -- 172.21.15.169:0/3493092493 shutdown_connections 2024-03-28T01:05:37.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.342+0000 7f9533b9a740 1 -- 172.21.15.169:0/3493092493 wait complete. 2024-03-28T01:05:37.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.342+0000 7f9533b9a740 1 -- start start 2024-03-28T01:05:37.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.343+0000 7f9533b9a740 1 Processor -- start 2024-03-28T01:05:37.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.343+0000 7f9533b9a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c505791690 0x55c50579fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.343+0000 7f9533b9a740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c505792080 0x55c5057a0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.343+0000 7f9533b9a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5057971f0 0x55c5058015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.343+0000 7f9533b9a740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c00e270 con 0x55c505792080 2024-03-28T01:05:37.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.343+0000 7f9533b9a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c00f570 con 0x55c505791690 2024-03-28T01:05:37.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.343+0000 7f9533b9a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c00e8b0 con 0x55c5057971f0 2024-03-28T01:05:37.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.343+0000 7f9533b9a740 10 client.0 _ll_register_callbacks cb 0x7f951c003990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:37.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.343+0000 7f9529d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c505792080 0x55c5057a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.343+0000 7f9529d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c505792080 0x55c5057a0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:57848/0 (socket says 172.21.15.169:57848) 2024-03-28T01:05:37.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.343+0000 7f9529d9b640 1 -- 172.21.15.169:0/2928095847 learned_addr learned my addr 172.21.15.169:0/2928095847 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:37.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.344+0000 7f952a59c640 1 --2- 172.21.15.169:0/2928095847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c505791690 0x55c50579fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.344+0000 7f952ad9d640 1 --2- 172.21.15.169:0/2928095847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5057971f0 0x55c5058015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.344+0000 7f952a59c640 1 -- 172.21.15.169:0/2928095847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5057971f0 msgr2=0x55c5058015e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:37.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.344+0000 7f952a59c640 1 --2- 172.21.15.169:0/2928095847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5057971f0 0x55c5058015e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.344+0000 7f952a59c640 1 -- 172.21.15.169:0/2928095847 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c505792080 msgr2=0x55c5057a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:37.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.344+0000 7f952a59c640 1 --2- 172.21.15.169:0/2928095847 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c505792080 0x55c5057a0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.344+0000 7f952a59c640 1 -- 172.21.15.169:0/2928095847 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c5057a31d0 con 0x55c505791690 2024-03-28T01:05:37.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.345+0000 7f952a59c640 1 --2- 172.21.15.169:0/2928095847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c505791690 0x55c50579fc10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f95240987c0 tx=0x7f9524018040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f6435ac5a713867 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:37.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.345+0000 7f9511ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:05:37.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.345+0000 7f9511ffb640 1 -- 172.21.15.169:0/2928095847 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952409a660 con 0x55c505791690 2024-03-28T01:05:37.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.345+0000 7f9511ffb640 1 -- 172.21.15.169:0/2928095847 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f95240cc530 con 0x55c505791690 2024-03-28T01:05:37.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.345+0000 7f9533b9a740 1 -- 172.21.15.169:0/2928095847 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f951c004190 con 0x55c505791690 2024-03-28T01:05:37.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.346+0000 7f9511ffb640 1 -- 172.21.15.169:0/2928095847 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9524016410 con 0x55c505791690 2024-03-28T01:05:37.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.348+0000 7f9533b9a740 10 client.27080 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:37.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.348+0000 7f9533b9a740 1 -- 172.21.15.169:0/2928095847 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c505802bc0 con 0x55c505791690 2024-03-28T01:05:37.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.350+0000 7f9511ffb640 1 -- 172.21.15.169:0/2928095847 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f951c004190 con 0x55c505791690 2024-03-28T01:05:37.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.350+0000 7f9511ffb640 1 -- 172.21.15.169:0/2928095847 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f952400b040 con 0x55c505791690 2024-03-28T01:05:37.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.352+0000 7f9533b9a740 10 client.27080 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:37.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.352+0000 7f9533b9a740 10 client.27080 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:05:37.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.352+0000 7f9533b9a740 20 client.27080 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:37.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.353+0000 7f9533b9a740 1 --2- 172.21.15.169:0/2928095847 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55c505803480 0x55c5058058a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.353+0000 7f9533b9a740 4 client.27080 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:05:37.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.353+0000 7f9533b9a740 1 -- 172.21.15.169:0/2928095847 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c5056fec50 con 0x55c505803480 2024-03-28T01:05:37.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.353+0000 7f9529d9b640 1 --2- 172.21.15.169:0/2928095847 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55c505803480 0x55c5058058a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.354+0000 7f9529d9b640 1 --2- 172.21.15.169:0/2928095847 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55c505803480 0x55c5058058a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fd8f4fd9214d3799 server_cookie=ac3c1ec08d202301 in_seq=0 out_seq=0 2024-03-28T01:05:37.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.354+0000 7f9511ffb640 10 client.27080 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:05:37.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.355+0000 7f9511ffb640 1 -- 172.21.15.169:0/2928095847 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c5056fec50 con 0x55c505803480 2024-03-28T01:05:37.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.355+0000 7f9511ffb640 10 client.27080 handle_command_reply: tid=0 2024-03-28T01:05:37.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.355+0000 7f9533b9a740 1 client.27080 shutdown 2024-03-28T01:05:37.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.358+0000 7f9533b9a740 1 -- 172.21.15.169:0/2928095847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c505791690 msgr2=0x55c50579fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:37.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.358+0000 7f9533b9a740 1 --2- 172.21.15.169:0/2928095847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c505791690 0x55c50579fc10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f95240987c0 tx=0x7f9524018040 comp rx=0 tx=0).stop 2024-03-28T01:05:37.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.359+0000 7f9533b9a740 1 -- 172.21.15.169:0/2928095847 shutdown_connections 2024-03-28T01:05:37.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.359+0000 7f9533b9a740 1 --2- 172.21.15.169:0/2928095847 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55c505803480 0x55c5058058a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.359+0000 7f9533b9a740 1 --2- 172.21.15.169:0/2928095847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5057971f0 0x55c5058015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.359+0000 7f9533b9a740 1 --2- 172.21.15.169:0/2928095847 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55c505792080 0x55c5057a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.359+0000 7f9533b9a740 1 --2- 172.21.15.169:0/2928095847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c505791690 0x55c50579fc10 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.359+0000 7f9533b9a740 1 -- 172.21.15.169:0/2928095847 >> 172.21.15.169:0/2928095847 conn(0x55c50578cc00 msgr2=0x55c505787cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:37.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.359+0000 7f9511ffb640 0 client.27080 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:05:37.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.359+0000 7f9533b9a740 1 -- 172.21.15.169:0/2928095847 shutdown_connections 2024-03-28T01:05:37.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.359+0000 7f9533b9a740 1 -- 172.21.15.169:0/2928095847 wait complete. 2024-03-28T01:05:37.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.359+0000 7f9533b9a740 20 client.27080 trim_cache size 0 max 16384 2024-03-28T01:05:37.366 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:05:37.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.369+0000 7f9530fb3640 1 -- 172.21.15.169:0/2051405938 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f95140a29d0 msgr2=0x7f95140a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:37.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.369+0000 7f9530fb3640 1 --2- 172.21.15.169:0/2051405938 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f95140a29d0 0x7f95140a4e90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9520037040 tx=0x7f9520000bc0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.369+0000 7f9530fb3640 1 -- 172.21.15.169:0/2051405938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f952c1086a0 msgr2=0x7f952c1a67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:37.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.369+0000 7f9530fb3640 1 --2- 172.21.15.169:0/2051405938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f952c1086a0 0x7f952c1a67f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f952400efb0 tx=0x7f952400c800 comp rx=0 tx=0).stop 2024-03-28T01:05:37.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.369+0000 7f9530fb3640 1 -- 172.21.15.169:0/2051405938 shutdown_connections 2024-03-28T01:05:37.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.369+0000 7f9530fb3640 1 --2- 172.21.15.169:0/2051405938 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f95140a29d0 0x7f95140a4e90 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.370+0000 7f9530fb3640 1 --2- 172.21.15.169:0/2051405938 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f952c10d9c0 0x7f952c1adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.370+0000 7f9530fb3640 1 --2- 172.21.15.169:0/2051405938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f952c10b010 0x7f952c1a6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.370+0000 7f9530fb3640 1 --2- 172.21.15.169:0/2051405938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f952c1086a0 0x7f952c1a67f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.370+0000 7f9530fb3640 1 -- 172.21.15.169:0/2051405938 >> 172.21.15.169:0/2051405938 conn(0x7f952c06e500 msgr2=0x7f952c10a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:37.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.370+0000 7f9530fb3640 1 -- 172.21.15.169:0/2051405938 shutdown_connections 2024-03-28T01:05:37.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.370+0000 7f9530fb3640 1 -- 172.21.15.169:0/2051405938 wait complete. 2024-03-28T01:05:37.399 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:05:37.399 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T01:05:37.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.750+0000 7f5e855b3640 1 -- 172.21.15.169:0/3994067427 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e8010d9c0 msgr2=0x7f5e8010de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:37.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.750+0000 7f5e855b3640 1 --2- 172.21.15.169:0/3994067427 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e8010d9c0 0x7f5e8010de40 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f5e74009900 tx=0x7f5e7402bc80 comp rx=0 tx=0).stop 2024-03-28T01:05:37.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.764+0000 7f5e855b3640 1 -- 172.21.15.169:0/3994067427 shutdown_connections 2024-03-28T01:05:37.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.764+0000 7f5e855b3640 1 --2- 172.21.15.169:0/3994067427 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e8010d9c0 0x7f5e8010de40 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.764+0000 7f5e855b3640 1 --2- 172.21.15.169:0/3994067427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e8010b010 0x7f5e8010d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.764+0000 7f5e855b3640 1 --2- 172.21.15.169:0/3994067427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e801086a0 0x7f5e8010aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.764+0000 7f5e855b3640 1 -- 172.21.15.169:0/3994067427 >> 172.21.15.169:0/3994067427 conn(0x7f5e8006e500 msgr2=0x7f5e80070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:37.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.769+0000 7f5e855b3640 1 -- 172.21.15.169:0/3994067427 shutdown_connections 2024-03-28T01:05:37.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.769+0000 7f5e855b3640 1 -- 172.21.15.169:0/3994067427 wait complete. 2024-03-28T01:05:37.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.773+0000 7f5e855b3640 1 Processor -- start 2024-03-28T01:05:37.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.773+0000 7f5e855b3640 1 -- start start 2024-03-28T01:05:37.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.773+0000 7f5e855b3640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e801086a0 0x7f5e80117ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.773+0000 7f5e855b3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e8010b010 0x7f5e80118400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.773+0000 7f5e855b3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e8010d9c0 0x7f5e80119440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.773+0000 7f5e855b3640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e801103e0 con 0x7f5e801086a0 2024-03-28T01:05:37.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.773+0000 7f5e855b3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e80110260 con 0x7f5e8010d9c0 2024-03-28T01:05:37.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.773+0000 7f5e855b3640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e80110560 con 0x7f5e8010b010 2024-03-28T01:05:37.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.774+0000 7f5e7effd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e801086a0 0x7f5e80117ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.774+0000 7f5e7f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e8010d9c0 0x7f5e80119440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.774+0000 7f5e7effd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e801086a0 0x7f5e80117ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:57854/0 (socket says 172.21.15.169:57854) 2024-03-28T01:05:37.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.774+0000 7f5e7f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e8010d9c0 0x7f5e80119440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:51814/0 (socket says 172.21.15.169:51814) 2024-03-28T01:05:37.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.774+0000 7f5e7effd640 1 -- 172.21.15.169:0/3249981196 learned_addr learned my addr 172.21.15.169:0/3249981196 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:37.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.774+0000 7f5e7e7fc640 1 --2- 172.21.15.169:0/3249981196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e8010b010 0x7f5e80118400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.774+0000 7f5e7f7fe640 1 -- 172.21.15.169:0/3249981196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e8010b010 msgr2=0x7f5e80118400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:37.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.774+0000 7f5e7f7fe640 1 --2- 172.21.15.169:0/3249981196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e8010b010 0x7f5e80118400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.774+0000 7f5e7f7fe640 1 -- 172.21.15.169:0/3249981196 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e801086a0 msgr2=0x7f5e80117ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:37.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.774+0000 7f5e7f7fe640 1 --2- 172.21.15.169:0/3249981196 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e801086a0 0x7f5e80117ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.774+0000 7f5e7f7fe640 1 -- 172.21.15.169:0/3249981196 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e80119ab0 con 0x7f5e8010d9c0 2024-03-28T01:05:37.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.774+0000 7f5e7f7fe640 1 --2- 172.21.15.169:0/3249981196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e8010d9c0 0x7f5e80119440 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5e7402c800 tx=0x7f5e740045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:37.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.776+0000 7f5e6ffff640 1 -- 172.21.15.169:0/3249981196 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e74042020 con 0x7f5e8010d9c0 2024-03-28T01:05:37.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.777+0000 7f5e855b3640 1 -- 172.21.15.169:0/3249981196 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e80119d40 con 0x7f5e8010d9c0 2024-03-28T01:05:37.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.777+0000 7f5e855b3640 1 -- 172.21.15.169:0/3249981196 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e8011a200 con 0x7f5e8010d9c0 2024-03-28T01:05:37.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.777+0000 7f5e6ffff640 1 -- 172.21.15.169:0/3249981196 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e74005410 con 0x7f5e8010d9c0 2024-03-28T01:05:37.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.777+0000 7f5e6ffff640 1 -- 172.21.15.169:0/3249981196 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7403e870 con 0x7f5e8010d9c0 2024-03-28T01:05:37.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.781+0000 7f5e6ffff640 1 -- 172.21.15.169:0/3249981196 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5e74045020 con 0x7f5e8010d9c0 2024-03-28T01:05:37.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.782+0000 7f5e6ffff640 1 --2- 172.21.15.169:0/3249981196 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5e680a2a10 0x7f5e680a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.783+0000 7f5e7effd640 1 --2- 172.21.15.169:0/3249981196 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5e680a2a10 0x7f5e680a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.783+0000 7f5e6ffff640 1 -- 172.21.15.169:0/3249981196 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f5e740f0f40 con 0x7f5e8010d9c0 2024-03-28T01:05:37.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.790+0000 7f5e7effd640 1 --2- 172.21.15.169:0/3249981196 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5e680a2a10 0x7f5e680a4ed0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5e78009fd0 tx=0x7f5e7800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:37.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.790+0000 7f5e88199740 1 Processor -- start 2024-03-28T01:05:37.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.790+0000 7f5e88199740 1 -- start start 2024-03-28T01:05:37.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.791+0000 7f5e88199740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562711299690 0x562711299ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.791+0000 7f5e88199740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56271129a080 0x56271129eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.791+0000 7f5e88199740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271129f1f0 0x5627112a1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.791+0000 7f5e88199740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56271125c610 con 0x562711299690 2024-03-28T01:05:37.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.791+0000 7f5e88199740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627111ff3c0 con 0x56271129f1f0 2024-03-28T01:05:37.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.791+0000 7f5e88199740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627111a7d60 con 0x56271129a080 2024-03-28T01:05:37.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.791+0000 7f5e7effd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562711299690 0x562711299ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.791+0000 7f5e7f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271129f1f0 0x5627112a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.791+0000 7f5e7f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271129f1f0 0x5627112a1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:51822/0 (socket says 172.21.15.169:51822) 2024-03-28T01:05:37.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.791+0000 7f5e7f7fe640 1 -- 172.21.15.169:0/642023208 learned_addr learned my addr 172.21.15.169:0/642023208 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:37.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.791+0000 7f5e7effd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562711299690 0x562711299ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:57868/0 (socket says 172.21.15.169:57868) 2024-03-28T01:05:37.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.792+0000 7f5e7f7fe640 1 -- 172.21.15.169:0/642023208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56271129a080 msgr2=0x56271129eb80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:05:37.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.792+0000 7f5e7f7fe640 1 --2- 172.21.15.169:0/642023208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56271129a080 0x56271129eb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.792+0000 7f5e7f7fe640 1 -- 172.21.15.169:0/642023208 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562711299690 msgr2=0x562711299ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:37.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.792+0000 7f5e7f7fe640 1 --2- 172.21.15.169:0/642023208 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562711299690 0x562711299ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.792+0000 7f5e7f7fe640 1 -- 172.21.15.169:0/642023208 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562710f133f0 con 0x56271129f1f0 2024-03-28T01:05:37.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.792+0000 7f5e7f7fe640 1 --2- 172.21.15.169:0/642023208 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271129f1f0 0x5627112a1680 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5e740be270 tx=0x7f5e74005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6aeed3731f66f10 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:37.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:37 smithi169 ceph-mon[32274]: pgmap v624: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 624 MiB/s wr, 1.02k op/s 2024-03-28T01:05:37.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.797+0000 7f5e6d7fa640 1 -- 172.21.15.169:0/642023208 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e740bec70 con 0x56271129f1f0 2024-03-28T01:05:37.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.797+0000 7f5e6d7fa640 1 -- 172.21.15.169:0/642023208 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e740be570 con 0x56271129f1f0 2024-03-28T01:05:37.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.797+0000 7f5e6d7fa640 1 -- 172.21.15.169:0/642023208 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7402cc90 con 0x56271129f1f0 2024-03-28T01:05:37.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.798+0000 7f5e88199740 1 -- 172.21.15.169:0/642023208 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271129f1f0 msgr2=0x5627112a1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:37.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.798+0000 7f5e88199740 1 --2- 172.21.15.169:0/642023208 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271129f1f0 0x5627112a1680 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5e740be270 tx=0x7f5e74005fd0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.800+0000 7f5e88199740 1 -- 172.21.15.169:0/642023208 shutdown_connections 2024-03-28T01:05:37.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.800+0000 7f5e88199740 1 --2- 172.21.15.169:0/642023208 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271129f1f0 0x5627112a1680 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.800+0000 7f5e88199740 1 --2- 172.21.15.169:0/642023208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56271129a080 0x56271129eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.800+0000 7f5e88199740 1 --2- 172.21.15.169:0/642023208 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562711299690 0x562711299ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.800+0000 7f5e88199740 1 -- 172.21.15.169:0/642023208 >> 172.21.15.169:0/642023208 conn(0x562711294c00 msgr2=0x562711295000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:37.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.801+0000 7f5e88199740 1 -- 172.21.15.169:0/642023208 shutdown_connections 2024-03-28T01:05:37.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.803+0000 7f5e88199740 1 -- 172.21.15.169:0/642023208 wait complete. 2024-03-28T01:05:37.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.804+0000 7f5e88199740 1 -- start start 2024-03-28T01:05:37.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.804+0000 7f5e88199740 1 Processor -- start 2024-03-28T01:05:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.807+0000 7f5e88199740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562711299690 0x5627112a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.807+0000 7f5e88199740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271129a080 0x5627112a3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.807+0000 7f5e88199740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56271129f1f0 0x5627112aacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.807+0000 7f5e88199740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627111bb240 con 0x562711299690 2024-03-28T01:05:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.807+0000 7f5e88199740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627111fb460 con 0x56271129a080 2024-03-28T01:05:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.807+0000 7f5e88199740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627112a1f60 con 0x56271129f1f0 2024-03-28T01:05:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.807+0000 7f5e88199740 10 client.0 _ll_register_callbacks cb 0x5627111bac10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.808+0000 7f5e7f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56271129f1f0 0x5627112aacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.808+0000 7f5e7f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56271129f1f0 0x5627112aacb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:42930/0 (socket says 172.21.15.169:42930) 2024-03-28T01:05:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.808+0000 7f5e7f7fe640 1 -- 172.21.15.169:0/819212646 learned_addr learned my addr 172.21.15.169:0/819212646 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.808+0000 7f5e7e7fc640 1 --2- 172.21.15.169:0/819212646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271129a080 0x5627112a3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.808+0000 7f5e7effd640 1 --2- 172.21.15.169:0/819212646 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562711299690 0x5627112a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.808+0000 7f5e7f7fe640 1 -- 172.21.15.169:0/819212646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271129a080 msgr2=0x5627112a3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.808+0000 7f5e7f7fe640 1 --2- 172.21.15.169:0/819212646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271129a080 0x5627112a3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.808+0000 7f5e7f7fe640 1 -- 172.21.15.169:0/819212646 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562711299690 msgr2=0x5627112a37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.808+0000 7f5e7f7fe640 1 --2- 172.21.15.169:0/819212646 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562711299690 0x5627112a37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.808+0000 7f5e7f7fe640 1 -- 172.21.15.169:0/819212646 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562710fdd7d0 con 0x56271129f1f0 2024-03-28T01:05:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.809+0000 7f5e7f7fe640 1 --2- 172.21.15.169:0/819212646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56271129f1f0 0x5627112aacb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f5e740ee960 tx=0x7f5e740bae10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=968ec512141f932e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.809+0000 7f5e667fc640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:05:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.809+0000 7f5e667fc640 1 -- 172.21.15.169:0/819212646 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e740cb060 con 0x56271129f1f0 2024-03-28T01:05:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.809+0000 7f5e667fc640 1 -- 172.21.15.169:0/819212646 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f5e740c6a90 con 0x56271129f1f0 2024-03-28T01:05:37.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.810+0000 7f5e88199740 1 -- 172.21.15.169:0/819212646 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e740be570 con 0x56271129f1f0 2024-03-28T01:05:37.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.813+0000 7f5e667fc640 1 -- 172.21.15.169:0/819212646 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e740bec70 con 0x56271129f1f0 2024-03-28T01:05:37.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.813+0000 7f5e88199740 10 client.27034 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:37.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.813+0000 7f5e88199740 1 -- 172.21.15.169:0/819212646 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5627112ae020 con 0x56271129f1f0 2024-03-28T01:05:37.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.814+0000 7f5e667fc640 1 -- 172.21.15.169:0/819212646 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e740be570 con 0x56271129f1f0 2024-03-28T01:05:37.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.815+0000 7f5e667fc640 1 -- 172.21.15.169:0/819212646 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f5e740c2120 con 0x56271129f1f0 2024-03-28T01:05:37.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.815+0000 7f5e88199740 10 client.27034 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:37.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.815+0000 7f5e88199740 10 client.27034 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:37.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.815+0000 7f5e88199740 20 client.27034 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:37.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.815+0000 7f5e88199740 1 --2- 172.21.15.169:0/819212646 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5627112af480 0x5627112b1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.815+0000 7f5e88199740 4 client.27034 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:05:37.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.815+0000 7f5e88199740 1 -- 172.21.15.169:0/819212646 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56271120c900 con 0x5627112af480 2024-03-28T01:05:37.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.816+0000 7f5e7effd640 1 --2- 172.21.15.169:0/819212646 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5627112af480 0x5627112b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.818+0000 7f5e7effd640 1 --2- 172.21.15.169:0/819212646 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5627112af480 0x5627112b1870 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1770de890880c84c server_cookie=b18281c1982249b4 in_seq=0 out_seq=0 2024-03-28T01:05:37.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.819+0000 7f5e667fc640 10 client.27034 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:37.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.820+0000 7f5e667fc640 1 -- 172.21.15.169:0/819212646 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56271120c900 con 0x5627112af480 2024-03-28T01:05:37.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.821+0000 7f5e667fc640 10 client.27034 handle_command_reply: tid=0 2024-03-28T01:05:37.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.821+0000 7f5e88199740 1 client.27034 shutdown 2024-03-28T01:05:37.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.826+0000 7f5e88199740 1 -- 172.21.15.169:0/819212646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56271129f1f0 msgr2=0x5627112aacb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:37.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.826+0000 7f5e88199740 1 --2- 172.21.15.169:0/819212646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56271129f1f0 0x5627112aacb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f5e740ee960 tx=0x7f5e740bae10 comp rx=0 tx=0).stop 2024-03-28T01:05:37.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.827+0000 7f5e88199740 1 -- 172.21.15.169:0/819212646 shutdown_connections 2024-03-28T01:05:37.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.827+0000 7f5e88199740 1 --2- 172.21.15.169:0/819212646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56271129f1f0 0x5627112aacb0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.827+0000 7f5e88199740 1 --2- 172.21.15.169:0/819212646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271129a080 0x5627112a3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.827+0000 7f5e88199740 1 --2- 172.21.15.169:0/819212646 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5627112af480 0x5627112b1870 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.827+0000 7f5e88199740 1 --2- 172.21.15.169:0/819212646 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562711299690 0x5627112a37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.827+0000 7f5e88199740 1 -- 172.21.15.169:0/819212646 >> 172.21.15.169:0/819212646 conn(0x562711294c00 msgr2=0x562711293e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:37.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.827+0000 7f5e667fc640 0 client.27034 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:37.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.828+0000 7f5e88199740 1 -- 172.21.15.169:0/819212646 shutdown_connections 2024-03-28T01:05:37.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.828+0000 7f5e88199740 1 -- 172.21.15.169:0/819212646 wait complete. 2024-03-28T01:05:37.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.828+0000 7f5e88199740 20 client.27034 trim_cache size 0 max 16384 2024-03-28T01:05:37.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.857+0000 7f5e88199740 1 Processor -- start 2024-03-28T01:05:37.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.857+0000 7f5e88199740 1 -- start start 2024-03-28T01:05:37.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.857+0000 7f5e88199740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562711299690 0x5627112a2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.858+0000 7f5e88199740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271129a080 0x5627112aac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.858+0000 7f5e88199740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56271129f1f0 0x5627112a2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.858+0000 7f5e88199740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e60011ed0 con 0x56271129f1f0 2024-03-28T01:05:37.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.858+0000 7f5e88199740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6000f9d0 con 0x56271129a080 2024-03-28T01:05:37.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.858+0000 7f5e88199740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6000db20 con 0x562711299690 2024-03-28T01:05:37.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.858+0000 7f5e7f7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56271129f1f0 0x5627112a2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.858+0000 7f5e7f7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56271129f1f0 0x5627112a2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:57892/0 (socket says 172.21.15.169:57892) 2024-03-28T01:05:37.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.858+0000 7f5e7f7fe640 1 -- 172.21.15.169:0/81257451 learned_addr learned my addr 172.21.15.169:0/81257451 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:37.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.858+0000 7f5e7effd640 1 --2- 172.21.15.169:0/81257451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562711299690 0x5627112a2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.859+0000 7f5e7f7fe640 1 -- 172.21.15.169:0/81257451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562711299690 msgr2=0x5627112a2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:37.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.859+0000 7f5e7e7fc640 1 --2- 172.21.15.169:0/81257451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271129a080 0x5627112aac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.859+0000 7f5e7f7fe640 1 --2- 172.21.15.169:0/81257451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562711299690 0x5627112a2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.859+0000 7f5e7f7fe640 1 -- 172.21.15.169:0/81257451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271129a080 msgr2=0x5627112aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:37.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.859+0000 7f5e7f7fe640 1 --2- 172.21.15.169:0/81257451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271129a080 0x5627112aac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.859+0000 7f5e7f7fe640 1 -- 172.21.15.169:0/81257451 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562711295670 con 0x56271129f1f0 2024-03-28T01:05:37.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.859+0000 7f5e7effd640 1 --2- 172.21.15.169:0/81257451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562711299690 0x5627112a2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:05:37.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.859+0000 7f5e7e7fc640 1 --2- 172.21.15.169:0/81257451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271129a080 0x5627112aac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:05:37.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.859+0000 7f5e7f7fe640 1 --2- 172.21.15.169:0/81257451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56271129f1f0 0x5627112a2790 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5e74004650 tx=0x7f5e740c25f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd692f003379832 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:37.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.863+0000 7f5e6cff9640 1 -- 172.21.15.169:0/81257451 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e740cb070 con 0x56271129f1f0 2024-03-28T01:05:37.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.863+0000 7f5e6cff9640 1 -- 172.21.15.169:0/81257451 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e740bac50 con 0x56271129f1f0 2024-03-28T01:05:37.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.863+0000 7f5e6cff9640 1 -- 172.21.15.169:0/81257451 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e740dd410 con 0x56271129f1f0 2024-03-28T01:05:37.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.864+0000 7f5e88199740 1 -- 172.21.15.169:0/81257451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56271129f1f0 msgr2=0x5627112a2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:37.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.864+0000 7f5e88199740 1 --2- 172.21.15.169:0/81257451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56271129f1f0 0x5627112a2790 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5e74004650 tx=0x7f5e740c25f0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.864+0000 7f5e88199740 1 -- 172.21.15.169:0/81257451 shutdown_connections 2024-03-28T01:05:37.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.864+0000 7f5e88199740 1 --2- 172.21.15.169:0/81257451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56271129f1f0 0x5627112a2790 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.864+0000 7f5e88199740 1 --2- 172.21.15.169:0/81257451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271129a080 0x5627112aac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.864+0000 7f5e88199740 1 --2- 172.21.15.169:0/81257451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562711299690 0x5627112a2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.864+0000 7f5e88199740 1 -- 172.21.15.169:0/81257451 >> 172.21.15.169:0/81257451 conn(0x562711294c00 msgr2=0x56271129b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:37.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.865+0000 7f5e88199740 1 -- 172.21.15.169:0/81257451 shutdown_connections 2024-03-28T01:05:37.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.865+0000 7f5e88199740 1 -- 172.21.15.169:0/81257451 wait complete. 2024-03-28T01:05:37.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.865+0000 7f5e88199740 1 -- start start 2024-03-28T01:05:37.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.866+0000 7f5e88199740 1 Processor -- start 2024-03-28T01:05:37.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.866+0000 7f5e88199740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562711299690 0x5627112a8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.866+0000 7f5e88199740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56271129a080 0x5627112a85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.866+0000 7f5e7effd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562711299690 0x5627112a8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.867+0000 7f5e7effd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562711299690 0x5627112a8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:51852/0 (socket says 172.21.15.169:51852) 2024-03-28T01:05:37.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.867+0000 7f5e7effd640 1 -- 172.21.15.169:0/4005331055 learned_addr learned my addr 172.21.15.169:0/4005331055 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:37.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.867+0000 7f5e7e7fc640 1 --2- 172.21.15.169:0/4005331055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56271129a080 0x5627112a85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.867+0000 7f5e88199740 1 --2- 172.21.15.169:0/4005331055 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56271129f1f0 0x5627112a95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.867+0000 7f5e88199740 1 -- 172.21.15.169:0/4005331055 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6000e090 con 0x56271129f1f0 2024-03-28T01:05:37.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.867+0000 7f5e88199740 1 -- 172.21.15.169:0/4005331055 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6000f370 con 0x562711299690 2024-03-28T01:05:37.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.867+0000 7f5e88199740 1 -- 172.21.15.169:0/4005331055 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6000e6b0 con 0x56271129a080 2024-03-28T01:05:37.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.868+0000 7f5e7effd640 1 -- 172.21.15.169:0/4005331055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56271129a080 msgr2=0x5627112a85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:37.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.868+0000 7f5e7effd640 1 --2- 172.21.15.169:0/4005331055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56271129a080 0x5627112a85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.868+0000 7f5e7effd640 1 -- 172.21.15.169:0/4005331055 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56271129f1f0 msgr2=0x5627112a95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:37.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.868+0000 7f5e7effd640 1 --2- 172.21.15.169:0/4005331055 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56271129f1f0 0x5627112a95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.868+0000 7f5e7effd640 1 -- 172.21.15.169:0/4005331055 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5627112a9bc0 con 0x562711299690 2024-03-28T01:05:37.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.868+0000 7f5e7effd640 1 --2- 172.21.15.169:0/4005331055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562711299690 0x5627112a8080 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x5627112b20a0 tx=0x7f5e78004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3409e4741ac41e35 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:37.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.868+0000 7f5e88199740 10 client.0 _ll_register_callbacks cb 0x7f5e7402cd80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:37.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.868+0000 7f5e88199740 1 -- 172.21.15.169:0/4005331055 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56271129ef60 con 0x562711299690 2024-03-28T01:05:37.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.875+0000 7f5e667fc640 10 client.27092 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:05:37.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.875+0000 7f5e667fc640 1 -- 172.21.15.169:0/4005331055 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7801b040 con 0x562711299690 2024-03-28T01:05:37.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.875+0000 7f5e667fc640 1 -- 172.21.15.169:0/4005331055 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f5e780227b0 con 0x562711299690 2024-03-28T01:05:37.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.876+0000 7f5e667fc640 1 -- 172.21.15.169:0/4005331055 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e7803a050 con 0x562711299690 2024-03-28T01:05:37.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.876+0000 7f5e88199740 10 client.27092 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:37.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.876+0000 7f5e88199740 1 -- 172.21.15.169:0/4005331055 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x5627112aad70 con 0x562711299690 2024-03-28T01:05:37.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.876+0000 7f5e667fc640 1 -- 172.21.15.169:0/4005331055 <== mon.2 v2:172.21.15.174:3300/0 4 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f5e780229b0 con 0x562711299690 2024-03-28T01:05:37.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.877+0000 7f5e88199740 10 client.27092 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:37.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.877+0000 7f5e88199740 10 client.27092 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:37.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.877+0000 7f5e88199740 20 client.27092 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:37.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.877+0000 7f5e88199740 1 --2- 172.21.15.169:0/4005331055 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56271130b1d0 0x56271130d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:37.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.877+0000 7f5e88199740 4 client.27092 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:05:37.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.877+0000 7f5e88199740 1 -- 172.21.15.169:0/4005331055 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56271129ef60 con 0x56271130b1d0 2024-03-28T01:05:37.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.877+0000 7f5e7e7fc640 1 --2- 172.21.15.169:0/4005331055 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56271130b1d0 0x56271130d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:37.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.878+0000 7f5e7e7fc640 1 --2- 172.21.15.169:0/4005331055 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56271130b1d0 0x56271130d5e0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=748ed02ec5d54fe server_cookie=411fcc0618995522 in_seq=0 out_seq=0 2024-03-28T01:05:37.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.878+0000 7f5e667fc640 10 client.27092 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:37.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.878+0000 7f5e667fc640 1 -- 172.21.15.169:0/4005331055 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56271129ef60 con 0x56271130b1d0 2024-03-28T01:05:37.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.878+0000 7f5e667fc640 10 client.27092 handle_command_reply: tid=0 2024-03-28T01:05:37.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.879+0000 7f5e88199740 1 client.27092 shutdown 2024-03-28T01:05:37.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.883+0000 7f5e88199740 1 -- 172.21.15.169:0/4005331055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562711299690 msgr2=0x5627112a8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:37.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.883+0000 7f5e88199740 1 --2- 172.21.15.169:0/4005331055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562711299690 0x5627112a8080 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x5627112b20a0 tx=0x7f5e78004ad0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.883+0000 7f5e88199740 1 -- 172.21.15.169:0/4005331055 shutdown_connections 2024-03-28T01:05:37.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.883+0000 7f5e88199740 1 --2- 172.21.15.169:0/4005331055 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56271130b1d0 0x56271130d5e0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.883+0000 7f5e88199740 1 --2- 172.21.15.169:0/4005331055 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56271129f1f0 0x5627112a95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.883+0000 7f5e88199740 1 --2- 172.21.15.169:0/4005331055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56271129a080 0x5627112a85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.883+0000 7f5e88199740 1 --2- 172.21.15.169:0/4005331055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562711299690 0x5627112a8080 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.884+0000 7f5e88199740 1 -- 172.21.15.169:0/4005331055 >> 172.21.15.169:0/4005331055 conn(0x562711294c00 msgr2=0x56271129b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:37.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.883+0000 7f5e667fc640 0 client.27092 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:37.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.887+0000 7f5e88199740 1 -- 172.21.15.169:0/4005331055 shutdown_connections 2024-03-28T01:05:37.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.887+0000 7f5e88199740 1 -- 172.21.15.169:0/4005331055 wait complete. 2024-03-28T01:05:37.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.888+0000 7f5e88199740 20 client.27092 trim_cache size 0 max 16384 2024-03-28T01:05:37.902 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:05:37.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.903+0000 7f5e855b3640 1 -- 172.21.15.169:0/3249981196 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5e680a2a10 msgr2=0x7f5e680a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:37.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.903+0000 7f5e855b3640 1 --2- 172.21.15.169:0/3249981196 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5e680a2a10 0x7f5e680a4ed0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5e78009fd0 tx=0x7f5e7800b040 comp rx=0 tx=0).stop 2024-03-28T01:05:37.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.903+0000 7f5e855b3640 1 -- 172.21.15.169:0/3249981196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e8010d9c0 msgr2=0x7f5e80119440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:37.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.903+0000 7f5e855b3640 1 --2- 172.21.15.169:0/3249981196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e8010d9c0 0x7f5e80119440 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5e7402c800 tx=0x7f5e740045e0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.907+0000 7f5e855b3640 1 -- 172.21.15.169:0/3249981196 shutdown_connections 2024-03-28T01:05:37.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.907+0000 7f5e855b3640 1 --2- 172.21.15.169:0/3249981196 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5e680a2a10 0x7f5e680a4ed0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.907+0000 7f5e855b3640 1 --2- 172.21.15.169:0/3249981196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e8010d9c0 0x7f5e80119440 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.907+0000 7f5e855b3640 1 --2- 172.21.15.169:0/3249981196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e8010b010 0x7f5e80118400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.907+0000 7f5e855b3640 1 --2- 172.21.15.169:0/3249981196 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e801086a0 0x7f5e80117ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:37.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.907+0000 7f5e855b3640 1 -- 172.21.15.169:0/3249981196 >> 172.21.15.169:0/3249981196 conn(0x7f5e8006e500 msgr2=0x7f5e8010b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:37.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:37 smithi177 ceph-mon[36206]: pgmap v624: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 624 MiB/s wr, 1.02k op/s 2024-03-28T01:05:37.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.910+0000 7f5e855b3640 1 -- 172.21.15.169:0/3249981196 shutdown_connections 2024-03-28T01:05:37.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:37.910+0000 7f5e855b3640 1 -- 172.21.15.169:0/3249981196 wait complete. 2024-03-28T01:05:38.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:37 smithi174 ceph-mon[36060]: pgmap v624: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 624 MiB/s wr, 1.02k op/s 2024-03-28T01:05:38.981 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T01:05:38.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T01:05:38.982 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T01:05:39.106 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.096+0000 7f8954d7e640 1 -- 172.21.15.169:0/1679357082 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f89501086a0 msgr2=0x7f895010aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:39.106 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.096+0000 7f8954d7e640 1 --2- 172.21.15.169:0/1679357082 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f89501086a0 0x7f895010aad0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8944009900 tx=0x7f894402bc50 comp rx=0 tx=0).stop 2024-03-28T01:05:39.113 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.110+0000 7f8954d7e640 1 -- 172.21.15.169:0/1679357082 shutdown_connections 2024-03-28T01:05:39.113 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.110+0000 7f8954d7e640 1 --2- 172.21.15.169:0/1679357082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f895010d9c0 0x7f895010de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.113 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.110+0000 7f8954d7e640 1 --2- 172.21.15.169:0/1679357082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f895010b010 0x7f895010d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.110+0000 7f8954d7e640 1 --2- 172.21.15.169:0/1679357082 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f89501086a0 0x7f895010aad0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.110+0000 7f8954d7e640 1 -- 172.21.15.169:0/1679357082 >> 172.21.15.169:0/1679357082 conn(0x7f895006e6e0 msgr2=0x7f8950070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:39.116 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.113+0000 7f8954d7e640 1 -- 172.21.15.169:0/1679357082 shutdown_connections 2024-03-28T01:05:39.120 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.115+0000 7f8954d7e640 1 -- 172.21.15.169:0/1679357082 wait complete. 2024-03-28T01:05:39.121 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.117+0000 7f8954d7e640 1 Processor -- start 2024-03-28T01:05:39.121 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.117+0000 7f8954d7e640 1 -- start start 2024-03-28T01:05:39.121 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.117+0000 7f8954d7e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f89501086a0 0x7f8950119810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.121 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.117+0000 7f8954d7e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f895010b010 0x7f8950119d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.121 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.117+0000 7f8954d7e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f895010d9c0 0x7f8950117e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.123 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.117+0000 7f8954d7e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8950110820 con 0x7f89501086a0 2024-03-28T01:05:39.123 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.117+0000 7f8954d7e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89501106a0 con 0x7f895010d9c0 2024-03-28T01:05:39.124 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.117+0000 7f8954d7e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89501109a0 con 0x7f895010b010 2024-03-28T01:05:39.124 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.117+0000 7f894dd9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f895010b010 0x7f8950119d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.124 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.117+0000 7f894dd9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f895010b010 0x7f8950119d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:42960/0 (socket says 172.21.15.169:42960) 2024-03-28T01:05:39.124 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.117+0000 7f894dd9b640 1 -- 172.21.15.169:0/1483999266 learned_addr learned my addr 172.21.15.169:0/1483999266 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:39.124 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.118+0000 7f894ed9d640 1 --2- 172.21.15.169:0/1483999266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f895010d9c0 0x7f8950117e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.124 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.118+0000 7f894e59c640 1 --2- 172.21.15.169:0/1483999266 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f89501086a0 0x7f8950119810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.124 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.118+0000 7f894dd9b640 1 -- 172.21.15.169:0/1483999266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f895010d9c0 msgr2=0x7f8950117e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:39.124 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.118+0000 7f894dd9b640 1 --2- 172.21.15.169:0/1483999266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f895010d9c0 0x7f8950117e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.124 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.118+0000 7f894dd9b640 1 -- 172.21.15.169:0/1483999266 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f89501086a0 msgr2=0x7f8950119810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:39.124 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.118+0000 7f894dd9b640 1 --2- 172.21.15.169:0/1483999266 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f89501086a0 0x7f8950119810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.124 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.118+0000 7f894dd9b640 1 -- 172.21.15.169:0/1483999266 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89501185c0 con 0x7f895010b010 2024-03-28T01:05:39.124 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.119+0000 7f894dd9b640 1 --2- 172.21.15.169:0/1483999266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f895010b010 0x7f8950119d50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f894000d770 tx=0x7f894000dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:39.124 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.119+0000 7f893f7fe640 1 -- 172.21.15.169:0/1483999266 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8940017070 con 0x7f895010b010 2024-03-28T01:05:39.124 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.119+0000 7f893f7fe640 1 -- 172.21.15.169:0/1483999266 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8940005500 con 0x7f895010b010 2024-03-28T01:05:39.125 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.119+0000 7f8954d7e640 1 -- 172.21.15.169:0/1483999266 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8950118850 con 0x7f895010b010 2024-03-28T01:05:39.125 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.119+0000 7f8954d7e640 1 -- 172.21.15.169:0/1483999266 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8950079900 con 0x7f895010b010 2024-03-28T01:05:39.126 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.123+0000 7f893f7fe640 1 -- 172.21.15.169:0/1483999266 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8940004e00 con 0x7f895010b010 2024-03-28T01:05:39.128 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.125+0000 7f893f7fe640 1 -- 172.21.15.169:0/1483999266 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8940010460 con 0x7f895010b010 2024-03-28T01:05:39.129 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.126+0000 7f893f7fe640 1 --2- 172.21.15.169:0/1483999266 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f89380a2a60 0x7f89380a4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.130 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.127+0000 7f893f7fe640 1 -- 172.21.15.169:0/1483999266 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f89400cf8e0 con 0x7f895010b010 2024-03-28T01:05:39.132 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.129+0000 7f8957969740 1 Processor -- start 2024-03-28T01:05:39.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.130+0000 7f8957969740 1 -- start start 2024-03-28T01:05:39.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.131+0000 7f8957969740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a36a93e920 0x55a36a93ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.131+0000 7f8957969740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a93f310 0x55a36a943e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.131+0000 7f8957969740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a944480 0x55a36a946910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.131+0000 7f8957969740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a36a8a3b10 con 0x55a36a944480 2024-03-28T01:05:39.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.131+0000 7f8957969740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a36a84c900 con 0x55a36a93e920 2024-03-28T01:05:39.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.131+0000 7f8957969740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a36a89fbb0 con 0x55a36a93f310 2024-03-28T01:05:39.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.131+0000 7f894dd9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a93f310 0x55a36a943e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.131+0000 7f894ed9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a944480 0x55a36a946910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.131+0000 7f894ed9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a944480 0x55a36a946910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:57928/0 (socket says 172.21.15.169:57928) 2024-03-28T01:05:39.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.131+0000 7f894ed9d640 1 -- 172.21.15.169:0/2154909951 learned_addr learned my addr 172.21.15.169:0/2154909951 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:39.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.132+0000 7f894ed9d640 1 -- 172.21.15.169:0/2154909951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a93f310 msgr2=0x55a36a943e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:39.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.132+0000 7f894ed9d640 1 --2- 172.21.15.169:0/2154909951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a93f310 0x55a36a943e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.132+0000 7f894ed9d640 1 -- 172.21.15.169:0/2154909951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a36a93e920 msgr2=0x55a36a93ed40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:05:39.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.132+0000 7f894ed9d640 1 --2- 172.21.15.169:0/2154909951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a36a93e920 0x55a36a93ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.132+0000 7f894ed9d640 1 -- 172.21.15.169:0/2154909951 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a36a5b8210 con 0x55a36a944480 2024-03-28T01:05:39.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.132+0000 7f894ed9d640 1 --2- 172.21.15.169:0/2154909951 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a944480 0x55a36a946910 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f8948007d90 tx=0x7f894800c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c56d8a7c1fd222 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:39.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.133+0000 7f894e59c640 1 --2- 172.21.15.169:0/1483999266 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f89380a2a60 0x7f89380a4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.133+0000 7f893cff9640 1 -- 172.21.15.169:0/2154909951 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8948011430 con 0x55a36a944480 2024-03-28T01:05:39.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.133+0000 7f893cff9640 1 -- 172.21.15.169:0/2154909951 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8948037020 con 0x55a36a944480 2024-03-28T01:05:39.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.133+0000 7f893cff9640 1 -- 172.21.15.169:0/2154909951 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8948010970 con 0x55a36a944480 2024-03-28T01:05:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.136+0000 7f8957969740 1 -- 172.21.15.169:0/2154909951 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a944480 msgr2=0x55a36a946910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.136+0000 7f8957969740 1 --2- 172.21.15.169:0/2154909951 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a944480 0x55a36a946910 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f8948007d90 tx=0x7f894800c560 comp rx=0 tx=0).stop 2024-03-28T01:05:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.136+0000 7f8957969740 1 -- 172.21.15.169:0/2154909951 shutdown_connections 2024-03-28T01:05:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.136+0000 7f8957969740 1 --2- 172.21.15.169:0/2154909951 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a944480 0x55a36a946910 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.136+0000 7f8957969740 1 --2- 172.21.15.169:0/2154909951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a93f310 0x55a36a943e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.136+0000 7f8957969740 1 --2- 172.21.15.169:0/2154909951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a36a93e920 0x55a36a93ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.136+0000 7f8957969740 1 -- 172.21.15.169:0/2154909951 >> 172.21.15.169:0/2154909951 conn(0x55a36a939e90 msgr2=0x55a36a93a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:39.141 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.138+0000 7f894e59c640 1 --2- 172.21.15.169:0/1483999266 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f89380a2a60 0x7f89380a4f20 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8944036040 tx=0x7f8944005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:39.146 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.143+0000 7f8957969740 1 -- 172.21.15.169:0/2154909951 shutdown_connections 2024-03-28T01:05:39.146 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.143+0000 7f8957969740 1 -- 172.21.15.169:0/2154909951 wait complete. 2024-03-28T01:05:39.146 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.144+0000 7f8957969740 1 -- start start 2024-03-28T01:05:39.147 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.144+0000 7f8957969740 1 Processor -- start 2024-03-28T01:05:39.147 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.144+0000 7f8957969740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a36a93e920 0x55a36a948c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.147 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.144+0000 7f8957969740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a93f310 0x55a36a9491c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.151 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.145+0000 7f8957969740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a944480 0x55a36a950200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.152 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.145+0000 7f8957969740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a36a947430 con 0x55a36a93f310 2024-03-28T01:05:39.152 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.145+0000 7f8957969740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a36a85fd50 con 0x55a36a93e920 2024-03-28T01:05:39.152 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.145+0000 7f8957969740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a36a9475b0 con 0x55a36a944480 2024-03-28T01:05:39.152 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.145+0000 7f8957969740 10 client.0 _ll_register_callbacks cb 0x55a36a85f720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:39.152 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.145+0000 7f894dd9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a93f310 0x55a36a9491c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.152 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.145+0000 7f894dd9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a93f310 0x55a36a9491c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:57936/0 (socket says 172.21.15.169:57936) 2024-03-28T01:05:39.152 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.145+0000 7f894dd9b640 1 -- 172.21.15.169:0/2824859308 learned_addr learned my addr 172.21.15.169:0/2824859308 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:39.152 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.145+0000 7f894ed9d640 1 --2- 172.21.15.169:0/2824859308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a944480 0x55a36a950200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.152 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.145+0000 7f894dd9b640 1 -- 172.21.15.169:0/2824859308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a944480 msgr2=0x55a36a950200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:39.152 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.145+0000 7f894dd9b640 1 --2- 172.21.15.169:0/2824859308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a944480 0x55a36a950200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.152 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.145+0000 7f894dd9b640 1 -- 172.21.15.169:0/2824859308 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a36a93e920 msgr2=0x55a36a948c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:39.152 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.145+0000 7f894dd9b640 1 --2- 172.21.15.169:0/2824859308 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a36a93e920 0x55a36a948c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.152 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.145+0000 7f894dd9b640 1 -- 172.21.15.169:0/2824859308 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a36a684620 con 0x55a36a93f310 2024-03-28T01:05:39.152 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.146+0000 7f894dd9b640 1 --2- 172.21.15.169:0/2824859308 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a93f310 0x55a36a9491c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f8940009610 tx=0x7f89400a0090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb9a395125ec13af server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:39.166 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.163+0000 7f8935ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:05:39.166 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.163+0000 7f8935ffb640 1 -- 172.21.15.169:0/2824859308 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8940020040 con 0x55a36a93f310 2024-03-28T01:05:39.166 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.163+0000 7f8935ffb640 1 -- 172.21.15.169:0/2824859308 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f89400984f0 con 0x55a36a93f310 2024-03-28T01:05:39.166 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.163+0000 7f8957969740 1 -- 172.21.15.169:0/2824859308 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8948037020 con 0x55a36a93f310 2024-03-28T01:05:39.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.169+0000 7f8935ffb640 1 -- 172.21.15.169:0/2824859308 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89400a4190 con 0x55a36a93f310 2024-03-28T01:05:39.173 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.171+0000 7f8957969740 10 client.17241 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:39.174 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.171+0000 7f8957969740 1 -- 172.21.15.169:0/2824859308 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a36a953640 con 0x55a36a93f310 2024-03-28T01:05:39.174 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.171+0000 7f8935ffb640 1 -- 172.21.15.169:0/2824859308 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8948037020 con 0x55a36a93f310 2024-03-28T01:05:39.174 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.171+0000 7f8935ffb640 1 -- 172.21.15.169:0/2824859308 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f894009ad40 con 0x55a36a93f310 2024-03-28T01:05:39.174 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.172+0000 7f8957969740 10 client.17241 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:39.175 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.172+0000 7f8957969740 10 client.17241 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:39.175 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.172+0000 7f8957969740 20 client.17241 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:39.175 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.172+0000 7f8957969740 1 --2- 172.21.15.169:0/2824859308 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a36a954ce0 0x55a36a957130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.175 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.172+0000 7f8957969740 4 client.17241 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:05:39.175 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.172+0000 7f8957969740 1 -- 172.21.15.169:0/2824859308 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a36a8b1050 con 0x55a36a954ce0 2024-03-28T01:05:39.177 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.174+0000 7f894e59c640 1 --2- 172.21.15.169:0/2824859308 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a36a954ce0 0x55a36a957130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.187 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.180+0000 7f8935ffb640 10 client.17241 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:39.187 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.180+0000 7f894e59c640 1 --2- 172.21.15.169:0/2824859308 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a36a954ce0 0x55a36a957130 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61372a295486d6d0 server_cookie=9bf69f3048c2410b in_seq=0 out_seq=0 2024-03-28T01:05:39.187 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.183+0000 7f8935ffb640 1 -- 172.21.15.169:0/2824859308 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a36a8b1050 con 0x55a36a954ce0 2024-03-28T01:05:39.187 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.183+0000 7f8935ffb640 10 client.17241 handle_command_reply: tid=0 2024-03-28T01:05:39.187 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.184+0000 7f8957969740 1 client.17241 shutdown 2024-03-28T01:05:39.190 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.187+0000 7f8957969740 1 -- 172.21.15.169:0/2824859308 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a93f310 msgr2=0x55a36a9491c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:39.190 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.187+0000 7f8957969740 1 --2- 172.21.15.169:0/2824859308 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a93f310 0x55a36a9491c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f8940009610 tx=0x7f89400a0090 comp rx=0 tx=0).stop 2024-03-28T01:05:39.190 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.187+0000 7f8957969740 1 -- 172.21.15.169:0/2824859308 shutdown_connections 2024-03-28T01:05:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.187+0000 7f8957969740 1 --2- 172.21.15.169:0/2824859308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a944480 0x55a36a950200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.187+0000 7f8957969740 1 --2- 172.21.15.169:0/2824859308 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a36a954ce0 0x55a36a957130 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.187+0000 7f8957969740 1 --2- 172.21.15.169:0/2824859308 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a93f310 0x55a36a9491c0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.187+0000 7f8957969740 1 --2- 172.21.15.169:0/2824859308 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a36a93e920 0x55a36a948c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.187+0000 7f8957969740 1 -- 172.21.15.169:0/2824859308 >> 172.21.15.169:0/2824859308 conn(0x55a36a939e90 msgr2=0x55a36a939020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.187+0000 7f8935ffb640 0 client.17241 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.187+0000 7f8957969740 1 -- 172.21.15.169:0/2824859308 shutdown_connections 2024-03-28T01:05:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.188+0000 7f8957969740 1 -- 172.21.15.169:0/2824859308 wait complete. 2024-03-28T01:05:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.188+0000 7f8957969740 20 client.17241 trim_cache size 0 max 16384 2024-03-28T01:05:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.230+0000 7f8957969740 1 Processor -- start 2024-03-28T01:05:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.230+0000 7f8957969740 1 -- start start 2024-03-28T01:05:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.231+0000 7f8957969740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a93e920 0x55a36a946870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.231+0000 7f8957969740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a36a93f310 0x55a36a9501c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.231+0000 7f8957969740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a944480 0x55a36a93d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.231+0000 7f8957969740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89300120e0 con 0x55a36a93e920 2024-03-28T01:05:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.231+0000 7f8957969740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893000fbe0 con 0x55a36a93f310 2024-03-28T01:05:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.231+0000 7f8957969740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893000dc80 con 0x55a36a944480 2024-03-28T01:05:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.231+0000 7f894e59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a93e920 0x55a36a946870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.235 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.231+0000 7f894e59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a93e920 0x55a36a946870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:57946/0 (socket says 172.21.15.169:57946) 2024-03-28T01:05:39.235 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.231+0000 7f894e59c640 1 -- 172.21.15.169:0/93086274 learned_addr learned my addr 172.21.15.169:0/93086274 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:39.235 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.231+0000 7f894ed9d640 1 --2- 172.21.15.169:0/93086274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a944480 0x55a36a93d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.232+0000 7f894ed9d640 1 -- 172.21.15.169:0/93086274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a36a93f310 msgr2=0x55a36a9501c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.232+0000 7f894ed9d640 1 --2- 172.21.15.169:0/93086274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a36a93f310 0x55a36a9501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.232+0000 7f894ed9d640 1 -- 172.21.15.169:0/93086274 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a93e920 msgr2=0x55a36a946870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.232+0000 7f894ed9d640 1 --2- 172.21.15.169:0/93086274 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a93e920 0x55a36a946870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.232+0000 7f894ed9d640 1 -- 172.21.15.169:0/93086274 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a36a93f6f0 con 0x55a36a944480 2024-03-28T01:05:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.233+0000 7f894ed9d640 1 --2- 172.21.15.169:0/93086274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a944480 0x55a36a93d0f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8948000f80 tx=0x7f8948007080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5ce587244eb4257 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.234+0000 7f8937fff640 1 -- 172.21.15.169:0/93086274 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f894804b070 con 0x55a36a944480 2024-03-28T01:05:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.234+0000 7f8937fff640 1 -- 172.21.15.169:0/93086274 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8948037040 con 0x55a36a944480 2024-03-28T01:05:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.234+0000 7f8937fff640 1 -- 172.21.15.169:0/93086274 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f894803c430 con 0x55a36a944480 2024-03-28T01:05:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.236+0000 7f8957969740 1 -- 172.21.15.169:0/93086274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a944480 msgr2=0x55a36a93d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.236+0000 7f8957969740 1 --2- 172.21.15.169:0/93086274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a944480 0x55a36a93d0f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8948000f80 tx=0x7f8948007080 comp rx=0 tx=0).stop 2024-03-28T01:05:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.236+0000 7f8957969740 1 -- 172.21.15.169:0/93086274 shutdown_connections 2024-03-28T01:05:39.240 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.236+0000 7f8957969740 1 --2- 172.21.15.169:0/93086274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a944480 0x55a36a93d0f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.240 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.236+0000 7f8957969740 1 --2- 172.21.15.169:0/93086274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a36a93f310 0x55a36a9501c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.240 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.236+0000 7f8957969740 1 --2- 172.21.15.169:0/93086274 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a93e920 0x55a36a946870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.240 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.236+0000 7f8957969740 1 -- 172.21.15.169:0/93086274 >> 172.21.15.169:0/93086274 conn(0x55a36a939e90 msgr2=0x55a36a93a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.239+0000 7f8957969740 1 -- 172.21.15.169:0/93086274 shutdown_connections 2024-03-28T01:05:39.243 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.239+0000 7f8957969740 1 -- 172.21.15.169:0/93086274 wait complete. 2024-03-28T01:05:39.243 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.240+0000 7f8957969740 1 -- start start 2024-03-28T01:05:39.243 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.240+0000 7f8957969740 1 Processor -- start 2024-03-28T01:05:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.240+0000 7f8957969740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a93e920 0x55a36a94d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.240+0000 7f8957969740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a36a93f310 0x55a36a94de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.240+0000 7f8957969740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a944480 0x55a36a94edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.240+0000 7f8957969740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893000e280 con 0x55a36a93e920 2024-03-28T01:05:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.240+0000 7f8957969740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893000f580 con 0x55a36a93f310 2024-03-28T01:05:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.240+0000 7f8957969740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893000e8c0 con 0x55a36a944480 2024-03-28T01:05:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.240+0000 7f8957969740 10 client.0 _ll_register_callbacks cb 0x7f894009be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:39.248 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.241+0000 7f894ed9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a944480 0x55a36a94edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.248 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.241+0000 7f894ed9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a944480 0x55a36a94edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:42998/0 (socket says 172.21.15.169:42998) 2024-03-28T01:05:39.248 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.241+0000 7f894ed9d640 1 -- 172.21.15.169:0/552455735 learned_addr learned my addr 172.21.15.169:0/552455735 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:39.248 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.245+0000 7f894dd9b640 1 --2- 172.21.15.169:0/552455735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a36a93f310 0x55a36a94de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.248 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.245+0000 7f894e59c640 1 --2- 172.21.15.169:0/552455735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a93e920 0x55a36a94d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.245+0000 7f894ed9d640 1 -- 172.21.15.169:0/552455735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a36a93f310 msgr2=0x55a36a94de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.245+0000 7f894ed9d640 1 --2- 172.21.15.169:0/552455735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a36a93f310 0x55a36a94de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.245+0000 7f894ed9d640 1 -- 172.21.15.169:0/552455735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a93e920 msgr2=0x55a36a94d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.245+0000 7f894ed9d640 1 --2- 172.21.15.169:0/552455735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a93e920 0x55a36a94d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.245+0000 7f894ed9d640 1 -- 172.21.15.169:0/552455735 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a36a94f3e0 con 0x55a36a944480 2024-03-28T01:05:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.245+0000 7f894dd9b640 1 --2- 172.21.15.169:0/552455735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a36a93f310 0x55a36a94de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:05:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.253+0000 7f894ed9d640 1 --2- 172.21.15.169:0/552455735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a944480 0x55a36a94edf0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8948039040 tx=0x7f8948007900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae91c50c8dfb1b5e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.253+0000 7f8935ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:05:39.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.259+0000 7f8935ffb640 1 -- 172.21.15.169:0/552455735 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f894804b070 con 0x55a36a944480 2024-03-28T01:05:39.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.259+0000 7f8935ffb640 1 -- 172.21.15.169:0/552455735 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f8948004410 con 0x55a36a944480 2024-03-28T01:05:39.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.259+0000 7f8957969740 1 -- 172.21.15.169:0/552455735 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8948037040 con 0x55a36a944480 2024-03-28T01:05:39.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.261+0000 7f8935ffb640 1 -- 172.21.15.169:0/552455735 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89480070f0 con 0x55a36a944480 2024-03-28T01:05:39.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.261+0000 7f8957969740 10 client.27055 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:39.264 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.261+0000 7f8957969740 1 -- 172.21.15.169:0/552455735 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a36a9b0600 con 0x55a36a944480 2024-03-28T01:05:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.263+0000 7f8935ffb640 1 -- 172.21.15.169:0/552455735 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8948037040 con 0x55a36a944480 2024-03-28T01:05:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.263+0000 7f8935ffb640 1 -- 172.21.15.169:0/552455735 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f894803c6a0 con 0x55a36a944480 2024-03-28T01:05:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.264+0000 7f8957969740 10 client.27055 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.264+0000 7f8957969740 10 client.27055 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.264+0000 7f8957969740 20 client.27055 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.264+0000 7f8957969740 1 --2- 172.21.15.169:0/552455735 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a36a9b0c40 0x55a36a9b3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.264+0000 7f8957969740 4 client.27055 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:05:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.264+0000 7f8957969740 1 -- 172.21.15.169:0/552455735 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a36a8ab3a0 con 0x55a36a9b0c40 2024-03-28T01:05:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.270+0000 7f894e59c640 1 --2- 172.21.15.169:0/552455735 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a36a9b0c40 0x55a36a9b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.280+0000 7f894e59c640 1 --2- 172.21.15.169:0/552455735 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a36a9b0c40 0x55a36a9b3060 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=815042f17376fcd9 server_cookie=ab08a7a9de4f31c5 in_seq=0 out_seq=0 2024-03-28T01:05:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:39.281+0000 7f8935ffb640 10 client.27055 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:39.440 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.437+0000 7fee4bfff640 1 -- 172.21.15.169:0/3883118988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee4c072ab0 msgr2=0x7fee4c072ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:39.440 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.437+0000 7fee4bfff640 1 --2- 172.21.15.169:0/3883118988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee4c072ab0 0x7fee4c072ed0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009900 tx=0x7fee3c02bc80 comp rx=0 tx=0).stop 2024-03-28T01:05:39.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.439+0000 7fee4bfff640 1 -- 172.21.15.169:0/3883118988 shutdown_connections 2024-03-28T01:05:39.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.439+0000 7fee4bfff640 1 --2- 172.21.15.169:0/3883118988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fee4c109730 0x7fee4c10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.439+0000 7fee4bfff640 1 --2- 172.21.15.169:0/3883118988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee4c108d70 0x7fee4c1091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.439+0000 7fee4bfff640 1 --2- 172.21.15.169:0/3883118988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee4c072ab0 0x7fee4c072ed0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.439+0000 7fee4bfff640 1 -- 172.21.15.169:0/3883118988 >> 172.21.15.169:0/3883118988 conn(0x7fee4c06e580 msgr2=0x7fee4c0709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:39.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.440+0000 7fee4bfff640 1 -- 172.21.15.169:0/3883118988 shutdown_connections 2024-03-28T01:05:39.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.441+0000 7fee4bfff640 1 -- 172.21.15.169:0/3883118988 wait complete. 2024-03-28T01:05:39.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.442+0000 7fee4bfff640 1 Processor -- start 2024-03-28T01:05:39.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.443+0000 7fee4bfff640 1 -- start start 2024-03-28T01:05:39.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.444+0000 7fee4bfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fee4c072ab0 0x7fee4c117d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.444+0000 7fee4bfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee4c108d70 0x7fee4c118260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.444+0000 7fee4bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee4c109730 0x7fee4c1192a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.444+0000 7fee4bfff640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c1103e0 con 0x7fee4c072ab0 2024-03-28T01:05:39.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.444+0000 7fee4bfff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c110260 con 0x7fee4c108d70 2024-03-28T01:05:39.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.444+0000 7fee4bfff640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c110560 con 0x7fee4c109730 2024-03-28T01:05:39.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.444+0000 7fee43fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee4c108d70 0x7fee4c118260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.444+0000 7fee43fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee4c108d70 0x7fee4c118260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:51906/0 (socket says 172.21.15.169:51906) 2024-03-28T01:05:39.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.444+0000 7fee43fff640 1 -- 172.21.15.169:0/2425125008 learned_addr learned my addr 172.21.15.169:0/2425125008 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:39.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.446+0000 7fee43fff640 1 -- 172.21.15.169:0/2425125008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee4c109730 msgr2=0x7fee4c1192a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:05:39.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.446+0000 7fee43fff640 1 --2- 172.21.15.169:0/2425125008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee4c109730 0x7fee4c1192a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.446+0000 7fee43fff640 1 -- 172.21.15.169:0/2425125008 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fee4c072ab0 msgr2=0x7fee4c117d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:39.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.446+0000 7fee43fff640 1 --2- 172.21.15.169:0/2425125008 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fee4c072ab0 0x7fee4c117d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.446+0000 7fee43fff640 1 -- 172.21.15.169:0/2425125008 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c119910 con 0x7fee4c108d70 2024-03-28T01:05:39.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.446+0000 7fee43fff640 1 --2- 172.21.15.169:0/2425125008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee4c108d70 0x7fee4c118260 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fee3400b450 tx=0x7fee3400b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:39.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.448+0000 7fee48ff9640 1 -- 172.21.15.169:0/2425125008 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee34015020 con 0x7fee4c108d70 2024-03-28T01:05:39.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.448+0000 7fee48ff9640 1 -- 172.21.15.169:0/2425125008 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee34011d70 con 0x7fee4c108d70 2024-03-28T01:05:39.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.448+0000 7fee48ff9640 1 -- 172.21.15.169:0/2425125008 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3400f340 con 0x7fee4c108d70 2024-03-28T01:05:39.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.448+0000 7fee4bfff640 1 -- 172.21.15.169:0/2425125008 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee4c119c00 con 0x7fee4c108d70 2024-03-28T01:05:39.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.448+0000 7fee4bfff640 1 -- 172.21.15.169:0/2425125008 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4c11a100 con 0x7fee4c108d70 2024-03-28T01:05:39.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.456+0000 7fee53ff5740 1 Processor -- start 2024-03-28T01:05:39.461 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.458+0000 7fee53ff5740 1 -- start start 2024-03-28T01:05:39.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.458+0000 7fee48ff9640 1 -- 172.21.15.169:0/2425125008 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fee34018020 con 0x7fee4c108d70 2024-03-28T01:05:39.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.459+0000 7fee53ff5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d5712b30 0x55d4d5712f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.459+0000 7fee53ff5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d4d5713520 0x55d4d5718020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.459+0000 7fee53ff5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5718690 0x55d4d571ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.459+0000 7fee53ff5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d5652c90 con 0x55d4d5713520 2024-03-28T01:05:39.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.459+0000 7fee53ff5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d5678070 con 0x55d4d5718690 2024-03-28T01:05:39.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.459+0000 7fee53ff5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d5620c80 con 0x55d4d5712b30 2024-03-28T01:05:39.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.460+0000 7fee48ff9640 1 --2- 172.21.15.169:0/2425125008 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fee380a2a10 0x7fee380a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.460+0000 7fee48ff9640 1 -- 172.21.15.169:0/2425125008 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fee340cf980 con 0x7fee4c108d70 2024-03-28T01:05:39.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.461+0000 7fee4b7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d5712b30 0x55d4d5712f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.461+0000 7fee4b7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d5712b30 0x55d4d5712f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:43022/0 (socket says 172.21.15.169:43022) 2024-03-28T01:05:39.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.461+0000 7fee4b7fe640 1 -- 172.21.15.169:0/3563566256 learned_addr learned my addr 172.21.15.169:0/3563566256 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:39.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.461+0000 7fee4b7fe640 1 --2- 172.21.15.169:0/2425125008 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fee380a2a10 0x7fee380a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.461+0000 7fee43fff640 1 --2- 172.21.15.169:0/3563566256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5718690 0x55d4d571ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.461+0000 7fee4affd640 1 --2- 172.21.15.169:0/3563566256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d4d5713520 0x55d4d5718020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.461+0000 7fee4b7fe640 1 -- 172.21.15.169:0/3563566256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5718690 msgr2=0x55d4d571ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:39.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.461+0000 7fee4b7fe640 1 --2- 172.21.15.169:0/3563566256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5718690 0x55d4d571ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.461+0000 7fee4b7fe640 1 -- 172.21.15.169:0/3563566256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d4d5713520 msgr2=0x55d4d5718020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:39.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.461+0000 7fee4b7fe640 1 --2- 172.21.15.169:0/3563566256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d4d5713520 0x55d4d5718020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.462+0000 7fee4b7fe640 1 -- 172.21.15.169:0/3563566256 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4d5618d10 con 0x55d4d5712b30 2024-03-28T01:05:39.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.462+0000 7fee43fff640 1 --2- 172.21.15.169:0/3563566256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5718690 0x55d4d571ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:05:39.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.462+0000 7fee4b7fe640 1 --2- 172.21.15.169:0/3563566256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d5712b30 0x55d4d5712f50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fee4400d990 tx=0x7fee4400de50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e026cc9625eb9d6b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:39.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.463+0000 7fee4b7fe640 1 --2- 172.21.15.169:0/2425125008 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fee380a2a10 0x7fee380a4ed0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fee44014530 tx=0x7fee44005000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:39.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.463+0000 7fee417fa640 1 -- 172.21.15.169:0/3563566256 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4401f070 con 0x55d4d5712b30 2024-03-28T01:05:39.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.463+0000 7fee417fa640 1 -- 172.21.15.169:0/3563566256 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee4400d510 con 0x55d4d5712b30 2024-03-28T01:05:39.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.464+0000 7fee53ff5740 1 -- 172.21.15.169:0/3563566256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d5712b30 msgr2=0x55d4d5712f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:39.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.464+0000 7fee53ff5740 1 --2- 172.21.15.169:0/3563566256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d5712b30 0x55d4d5712f50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fee4400d990 tx=0x7fee4400de50 comp rx=0 tx=0).stop 2024-03-28T01:05:39.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.466+0000 7fee53ff5740 1 -- 172.21.15.169:0/3563566256 shutdown_connections 2024-03-28T01:05:39.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.466+0000 7fee53ff5740 1 --2- 172.21.15.169:0/3563566256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5718690 0x55d4d571ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.466+0000 7fee53ff5740 1 --2- 172.21.15.169:0/3563566256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d4d5713520 0x55d4d5718020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.466+0000 7fee53ff5740 1 --2- 172.21.15.169:0/3563566256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d5712b30 0x55d4d5712f50 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.466+0000 7fee53ff5740 1 -- 172.21.15.169:0/3563566256 >> 172.21.15.169:0/3563566256 conn(0x55d4d56db1a0 msgr2=0x55d4d570e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:39.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.468+0000 7fee53ff5740 1 -- 172.21.15.169:0/3563566256 shutdown_connections 2024-03-28T01:05:39.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.468+0000 7fee53ff5740 1 -- 172.21.15.169:0/3563566256 wait complete. 2024-03-28T01:05:39.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.468+0000 7fee53ff5740 1 -- start start 2024-03-28T01:05:39.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.468+0000 7fee53ff5740 1 Processor -- start 2024-03-28T01:05:39.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.469+0000 7fee53ff5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d5712b30 0x55d4d571ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.469+0000 7fee53ff5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d4d5713520 0x55d4d571d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.469+0000 7fee53ff5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5718690 0x55d4d57243f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.469+0000 7fee53ff5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d56340d0 con 0x55d4d5713520 2024-03-28T01:05:39.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.469+0000 7fee53ff5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d5674110 con 0x55d4d5718690 2024-03-28T01:05:39.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.469+0000 7fee53ff5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4d571b630 con 0x55d4d5712b30 2024-03-28T01:05:39.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.469+0000 7fee53ff5740 10 client.0 _ll_register_callbacks cb 0x55d4d566fe60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:39.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.470+0000 7fee4b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5718690 0x55d4d57243f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.470+0000 7fee4b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5718690 0x55d4d57243f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:51914/0 (socket says 172.21.15.169:51914) 2024-03-28T01:05:39.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.470+0000 7fee4b7fe640 1 -- 172.21.15.169:0/12470376 learned_addr learned my addr 172.21.15.169:0/12470376 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:39.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.472+0000 7fee4b7fe640 1 -- 172.21.15.169:0/12470376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d5712b30 msgr2=0x55d4d571ce70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:05:39.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.472+0000 7fee4b7fe640 1 --2- 172.21.15.169:0/12470376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d5712b30 0x55d4d571ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.472+0000 7fee4b7fe640 1 -- 172.21.15.169:0/12470376 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d4d5713520 msgr2=0x55d4d571d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:39.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.472+0000 7fee4b7fe640 1 --2- 172.21.15.169:0/12470376 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d4d5713520 0x55d4d571d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:39.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.472+0000 7fee4b7fe640 1 -- 172.21.15.169:0/12470376 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d4d54547e0 con 0x55d4d5718690 2024-03-28T01:05:39.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.474+0000 7fee4b7fe640 1 --2- 172.21.15.169:0/12470376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5718690 0x55d4d57243f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fee44006c40 tx=0x7fee44006c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ff522292d588586 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:39.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.474+0000 7fee327fc640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:05:39.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.474+0000 7fee327fc640 1 -- 172.21.15.169:0/12470376 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4402a070 con 0x55d4d5718690 2024-03-28T01:05:39.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.475+0000 7fee53ff5740 1 -- 172.21.15.169:0/12470376 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d4d56855b0 con 0x55d4d5718690 2024-03-28T01:05:39.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.475+0000 7fee327fc640 1 -- 172.21.15.169:0/12470376 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fee440062e0 con 0x55d4d5718690 2024-03-28T01:05:39.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.475+0000 7fee327fc640 1 -- 172.21.15.169:0/12470376 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fee4401f050 con 0x55d4d5718690 2024-03-28T01:05:39.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.475+0000 7fee53ff5740 10 client.27110 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:39.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.475+0000 7fee53ff5740 1 -- 172.21.15.169:0/12470376 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d4d54b4f70 con 0x55d4d5718690 2024-03-28T01:05:39.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.481+0000 7fee327fc640 1 -- 172.21.15.169:0/12470376 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d4d56855b0 con 0x55d4d5718690 2024-03-28T01:05:39.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.481+0000 7fee327fc640 1 -- 172.21.15.169:0/12470376 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fee4401a070 con 0x55d4d5718690 2024-03-28T01:05:39.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.481+0000 7fee53ff5740 10 client.27110 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:39.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.481+0000 7fee53ff5740 10 client.27110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:39.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.481+0000 7fee53ff5740 20 client.27110 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:39.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.481+0000 7fee53ff5740 1 --2- 172.21.15.169:0/12470376 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d4d5729090 0x55d4d572b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:39.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.481+0000 7fee53ff5740 4 client.27110 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:05:39.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.481+0000 7fee53ff5740 1 -- 172.21.15.169:0/12470376 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d4d567f900 con 0x55d4d5729090 2024-03-28T01:05:39.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.497+0000 7fee4affd640 1 --2- 172.21.15.169:0/12470376 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d4d5729090 0x55d4d572b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:39.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.515+0000 7fee4affd640 1 --2- 172.21.15.169:0/12470376 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d4d5729090 0x55d4d572b4f0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85cb00568ddcb149 server_cookie=8c0d5ca55247d38b in_seq=0 out_seq=0 2024-03-28T01:05:39.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:39.515+0000 7fee327fc640 10 client.27110 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:39.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:39 smithi177 ceph-mon[36206]: pgmap v625: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 654 MiB/s wr, 968 op/s 2024-03-28T01:05:40.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:39 smithi169 ceph-mon[32274]: pgmap v625: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 654 MiB/s wr, 968 op/s 2024-03-28T01:05:40.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:39 smithi174 ceph-mon[36060]: pgmap v625: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 654 MiB/s wr, 968 op/s 2024-03-28T01:05:41.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:41 smithi177 ceph-mon[36206]: pgmap v626: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 543 MiB/s wr, 659 op/s 2024-03-28T01:05:42.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:41 smithi169 ceph-mon[32274]: pgmap v626: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 543 MiB/s wr, 659 op/s 2024-03-28T01:05:42.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:41 smithi174 ceph-mon[36060]: pgmap v626: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 543 MiB/s wr, 659 op/s 2024-03-28T01:05:43.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:43 smithi177 ceph-mon[36206]: pgmap v627: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 658 MiB/s wr, 914 op/s 2024-03-28T01:05:44.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:43 smithi169 ceph-mon[32274]: pgmap v627: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 658 MiB/s wr, 914 op/s 2024-03-28T01:05:44.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:43 smithi174 ceph-mon[36060]: pgmap v627: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 658 MiB/s wr, 914 op/s 2024-03-28T01:05:45.112 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.108+0000 7f8935ffb640 1 -- 172.21.15.169:0/552455735 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a36a8ab3a0 con 0x55a36a9b0c40 2024-03-28T01:05:45.112 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.108+0000 7f8935ffb640 10 client.27055 handle_command_reply: tid=0 2024-03-28T01:05:45.112 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.108+0000 7f8957969740 1 client.27055 shutdown 2024-03-28T01:05:45.112 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.109+0000 7f8957969740 1 -- 172.21.15.169:0/552455735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a944480 msgr2=0x55a36a94edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.109+0000 7f8957969740 1 --2- 172.21.15.169:0/552455735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a944480 0x55a36a94edf0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8948039040 tx=0x7f8948007900 comp rx=0 tx=0).stop 2024-03-28T01:05:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.109+0000 7f8957969740 1 -- 172.21.15.169:0/552455735 shutdown_connections 2024-03-28T01:05:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.109+0000 7f8957969740 1 --2- 172.21.15.169:0/552455735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a36a944480 0x55a36a94edf0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.109+0000 7f8957969740 1 --2- 172.21.15.169:0/552455735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a36a93f310 0x55a36a94de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.109+0000 7f8957969740 1 --2- 172.21.15.169:0/552455735 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a36a9b0c40 0x55a36a9b3060 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.109+0000 7f8957969740 1 --2- 172.21.15.169:0/552455735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a36a93e920 0x55a36a94d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.109+0000 7f8957969740 1 -- 172.21.15.169:0/552455735 >> 172.21.15.169:0/552455735 conn(0x55a36a939e90 msgr2=0x55a36a93d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.109+0000 7f8935ffb640 0 client.27055 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:45.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.109+0000 7fee327fc640 1 -- 172.21.15.169:0/12470376 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d4d567f900 con 0x55d4d5729090 2024-03-28T01:05:45.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.109+0000 7fee327fc640 10 client.27110 handle_command_reply: tid=0 2024-03-28T01:05:45.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.109+0000 7fee53ff5740 1 client.27110 shutdown 2024-03-28T01:05:45.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.109+0000 7fee53ff5740 1 -- 172.21.15.169:0/12470376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5718690 msgr2=0x55d4d57243f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:45.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.109+0000 7fee53ff5740 1 --2- 172.21.15.169:0/12470376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5718690 0x55d4d57243f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fee44006c40 tx=0x7fee44006c70 comp rx=0 tx=0).stop 2024-03-28T01:05:45.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.110+0000 7fee53ff5740 1 -- 172.21.15.169:0/12470376 shutdown_connections 2024-03-28T01:05:45.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.110+0000 7fee53ff5740 1 --2- 172.21.15.169:0/12470376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5718690 0x55d4d57243f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:45.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.110+0000 7fee53ff5740 1 --2- 172.21.15.169:0/12470376 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d4d5729090 0x55d4d572b4f0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:45.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.110+0000 7fee53ff5740 1 --2- 172.21.15.169:0/12470376 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d4d5713520 0x55d4d571d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:45.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.110+0000 7fee53ff5740 1 --2- 172.21.15.169:0/12470376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d5712b30 0x55d4d571ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:45.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.110+0000 7fee53ff5740 1 -- 172.21.15.169:0/12470376 >> 172.21.15.169:0/12470376 conn(0x55d4d56db1a0 msgr2=0x55d4d570d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:45.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.110+0000 7fee327fc640 0 client.27110 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:45.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.110+0000 7f8957969740 1 -- 172.21.15.169:0/552455735 shutdown_connections 2024-03-28T01:05:45.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.110+0000 7fee53ff5740 1 -- 172.21.15.169:0/12470376 shutdown_connections 2024-03-28T01:05:45.115 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.110+0000 7f8957969740 1 -- 172.21.15.169:0/552455735 wait complete. 2024-03-28T01:05:45.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.110+0000 7fee53ff5740 1 -- 172.21.15.169:0/12470376 wait complete. 2024-03-28T01:05:45.116 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.110+0000 7f8957969740 20 client.27055 trim_cache size 0 max 16384 2024-03-28T01:05:45.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.111+0000 7fee53ff5740 20 client.27110 trim_cache size 0 max 16384 2024-03-28T01:05:45.116 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:05:45.116 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:05:45.116 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:05:45.116 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:05:45.119 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.115+0000 7f8954d7e640 1 -- 172.21.15.169:0/1483999266 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f89380a2a60 msgr2=0x7f89380a4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:45.119 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.115+0000 7f8954d7e640 1 --2- 172.21.15.169:0/1483999266 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f89380a2a60 0x7f89380a4f20 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8944036040 tx=0x7f8944005fd0 comp rx=0 tx=0).stop 2024-03-28T01:05:45.119 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.115+0000 7f8954d7e640 1 -- 172.21.15.169:0/1483999266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f895010b010 msgr2=0x7f8950119d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:45.119 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.115+0000 7f8954d7e640 1 --2- 172.21.15.169:0/1483999266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f895010b010 0x7f8950119d50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f894000d770 tx=0x7f894000dc30 comp rx=0 tx=0).stop 2024-03-28T01:05:45.126 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.123+0000 7f8954d7e640 1 -- 172.21.15.169:0/1483999266 shutdown_connections 2024-03-28T01:05:45.126 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.123+0000 7f8954d7e640 1 --2- 172.21.15.169:0/1483999266 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f89380a2a60 0x7f89380a4f20 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:45.126 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.123+0000 7f8954d7e640 1 --2- 172.21.15.169:0/1483999266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f895010d9c0 0x7f8950117e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:45.126 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.123+0000 7f8954d7e640 1 --2- 172.21.15.169:0/1483999266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f895010b010 0x7f8950119d50 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:45.126 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.123+0000 7f8954d7e640 1 --2- 172.21.15.169:0/1483999266 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f89501086a0 0x7f8950119810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:45.126 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.123+0000 7f8954d7e640 1 -- 172.21.15.169:0/1483999266 >> 172.21.15.169:0/1483999266 conn(0x7f895006e6e0 msgr2=0x7f8950108d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:45.126 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.123+0000 7f8954d7e640 1 -- 172.21.15.169:0/1483999266 shutdown_connections 2024-03-28T01:05:45.126 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:45.123+0000 7f8954d7e640 1 -- 172.21.15.169:0/1483999266 wait complete. 2024-03-28T01:05:45.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.162+0000 7fee53ff5740 1 Processor -- start 2024-03-28T01:05:45.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.163+0000 7fee53ff5740 1 -- start start 2024-03-28T01:05:45.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.163+0000 7fee53ff5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d4d5712b30 0x55d4d5711df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:45.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.164+0000 7fee53ff5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5713520 0x55d4d57243b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:45.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.164+0000 7fee53ff5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d5718690 0x55d4d570cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:45.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.164+0000 7fee43fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5713520 0x55d4d57243b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:45.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.164+0000 7fee43fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5713520 0x55d4d57243b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:51930/0 (socket says 172.21.15.169:51930) 2024-03-28T01:05:45.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.164+0000 7fee43fff640 1 -- 172.21.15.169:0/2070523345 learned_addr learned my addr 172.21.15.169:0/2070523345 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:45.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.164+0000 7fee4b7fe640 1 --2- 172.21.15.169:0/2070523345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d5718690 0x55d4d570cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:45.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.164+0000 7fee53ff5740 1 -- 172.21.15.169:0/2070523345 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c012120 con 0x55d4d5712b30 2024-03-28T01:05:45.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.164+0000 7fee53ff5740 1 -- 172.21.15.169:0/2070523345 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c00fc20 con 0x55d4d5713520 2024-03-28T01:05:45.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.164+0000 7fee53ff5740 1 -- 172.21.15.169:0/2070523345 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c00dcc0 con 0x55d4d5718690 2024-03-28T01:05:45.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.165+0000 7fee4affd640 1 --2- 172.21.15.169:0/2070523345 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d4d5712b30 0x55d4d5711df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:45.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.165+0000 7fee43fff640 1 -- 172.21.15.169:0/2070523345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d5718690 msgr2=0x55d4d570cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:45.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.165+0000 7fee43fff640 1 --2- 172.21.15.169:0/2070523345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d5718690 0x55d4d570cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:45.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.165+0000 7fee43fff640 1 -- 172.21.15.169:0/2070523345 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d4d5712b30 msgr2=0x55d4d5711df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:45.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.165+0000 7fee43fff640 1 --2- 172.21.15.169:0/2070523345 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d4d5712b30 0x55d4d5711df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:45.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.165+0000 7fee43fff640 1 -- 172.21.15.169:0/2070523345 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4d534c170 con 0x55d4d5713520 2024-03-28T01:05:45.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.166+0000 7fee43fff640 1 --2- 172.21.15.169:0/2070523345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5713520 0x55d4d57243b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fee3400be10 tx=0x7fee34013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b2423c1b6bfae1b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:45.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.168+0000 7fee40ff9640 1 -- 172.21.15.169:0/2070523345 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3409bf20 con 0x55d4d5713520 2024-03-28T01:05:45.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.169+0000 7fee40ff9640 1 -- 172.21.15.169:0/2070523345 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee340a0070 con 0x55d4d5713520 2024-03-28T01:05:45.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.170+0000 7fee53ff5740 1 -- 172.21.15.169:0/2070523345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5713520 msgr2=0x55d4d57243b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:45.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.170+0000 7fee53ff5740 1 --2- 172.21.15.169:0/2070523345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5713520 0x55d4d57243b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fee3400be10 tx=0x7fee34013040 comp rx=0 tx=0).stop 2024-03-28T01:05:45.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.170+0000 7fee53ff5740 1 -- 172.21.15.169:0/2070523345 shutdown_connections 2024-03-28T01:05:45.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.170+0000 7fee53ff5740 1 --2- 172.21.15.169:0/2070523345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d5718690 0x55d4d570cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:45.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.170+0000 7fee53ff5740 1 --2- 172.21.15.169:0/2070523345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5713520 0x55d4d57243b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:45.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.170+0000 7fee53ff5740 1 --2- 172.21.15.169:0/2070523345 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d4d5712b30 0x55d4d5711df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:45.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.170+0000 7fee53ff5740 1 -- 172.21.15.169:0/2070523345 >> 172.21.15.169:0/2070523345 conn(0x55d4d56db1a0 msgr2=0x55d4d57116e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:45.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.175+0000 7fee53ff5740 1 -- 172.21.15.169:0/2070523345 shutdown_connections 2024-03-28T01:05:45.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.176+0000 7fee53ff5740 1 -- 172.21.15.169:0/2070523345 wait complete. 2024-03-28T01:05:45.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.177+0000 7fee53ff5740 1 -- start start 2024-03-28T01:05:45.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.177+0000 7fee53ff5740 1 Processor -- start 2024-03-28T01:05:45.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.177+0000 7fee53ff5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d5712b30 0x55d4d5721db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:45.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.177+0000 7fee53ff5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d4d5713520 0x55d4d57222f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:45.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.177+0000 7fee53ff5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5718690 0x55d4d5723300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:45.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.177+0000 7fee53ff5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c00e2c0 con 0x55d4d5713520 2024-03-28T01:05:45.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.177+0000 7fee53ff5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c00f5c0 con 0x55d4d5718690 2024-03-28T01:05:45.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.177+0000 7fee53ff5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c00e900 con 0x55d4d5712b30 2024-03-28T01:05:45.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.178+0000 7fee53ff5740 10 client.0 _ll_register_callbacks cb 0x7fee4400d760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:45.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.178+0000 7fee4b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5718690 0x55d4d5723300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:45.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.178+0000 7fee4b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5718690 0x55d4d5723300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:51940/0 (socket says 172.21.15.169:51940) 2024-03-28T01:05:45.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.178+0000 7fee4b7fe640 1 -- 172.21.15.169:0/4111898722 learned_addr learned my addr 172.21.15.169:0/4111898722 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:45.182 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.179+0000 7fee4b7fe640 1 -- 172.21.15.169:0/4111898722 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d5712b30 msgr2=0x55d4d5721db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:05:45.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.179+0000 7fee4b7fe640 1 --2- 172.21.15.169:0/4111898722 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d5712b30 0x55d4d5721db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:45.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.179+0000 7fee4b7fe640 1 -- 172.21.15.169:0/4111898722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d4d5713520 msgr2=0x55d4d57222f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:45.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.179+0000 7fee4b7fe640 1 --2- 172.21.15.169:0/4111898722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d4d5713520 0x55d4d57222f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:45.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.179+0000 7fee4b7fe640 1 -- 172.21.15.169:0/4111898722 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d4d57238f0 con 0x55d4d5718690 2024-03-28T01:05:45.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.179+0000 7fee4b7fe640 1 --2- 172.21.15.169:0/4111898722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5718690 0x55d4d5723300 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fee44004560 tx=0x7fee44004980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eddfe56855208fd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:45.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.179+0000 7fee327fc640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:05:45.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.179+0000 7fee327fc640 1 -- 172.21.15.169:0/4111898722 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4402a040 con 0x55d4d5718690 2024-03-28T01:05:45.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.179+0000 7fee53ff5740 1 -- 172.21.15.169:0/4111898722 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fee340a0070 con 0x55d4d5718690 2024-03-28T01:05:45.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.179+0000 7fee327fc640 1 -- 172.21.15.169:0/4111898722 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fee44004260 con 0x55d4d5718690 2024-03-28T01:05:45.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.180+0000 7fee327fc640 1 -- 172.21.15.169:0/4111898722 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fee4401b830 con 0x55d4d5718690 2024-03-28T01:05:45.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.180+0000 7fee53ff5740 10 client.27116 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:45.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.180+0000 7fee53ff5740 1 -- 172.21.15.169:0/4111898722 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d4d5784c60 con 0x55d4d5718690 2024-03-28T01:05:45.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.180+0000 7fee327fc640 1 -- 172.21.15.169:0/4111898722 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee340a0070 con 0x55d4d5718690 2024-03-28T01:05:45.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.180+0000 7fee327fc640 1 -- 172.21.15.169:0/4111898722 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fee44038070 con 0x55d4d5718690 2024-03-28T01:05:45.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.181+0000 7fee53ff5740 10 client.27116 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:45.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.181+0000 7fee53ff5740 10 client.27116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:45.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.181+0000 7fee53ff5740 20 client.27116 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:45.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.181+0000 7fee53ff5740 1 --2- 172.21.15.169:0/4111898722 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d4d5785320 0x55d4d5787740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:45.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.181+0000 7fee53ff5740 4 client.27116 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T01:05:45.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.181+0000 7fee53ff5740 1 -- 172.21.15.169:0/4111898722 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55d4d55fdb80 con 0x55d4d5785320 2024-03-28T01:05:45.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.182+0000 7fee4affd640 1 --2- 172.21.15.169:0/4111898722 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d4d5785320 0x55d4d5787740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:45.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.182+0000 7fee4affd640 1 --2- 172.21.15.169:0/4111898722 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d4d5785320 0x55d4d5787740 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2be016ce5aafba3 server_cookie=a99cf591d90b2bba in_seq=0 out_seq=0 2024-03-28T01:05:45.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:45.182+0000 7fee327fc640 10 client.27116 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:45.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:45 smithi177 ceph-mon[36206]: pgmap v628: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 562 MiB/s wr, 956 op/s 2024-03-28T01:05:46.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:45 smithi169 ceph-mon[32274]: pgmap v628: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 562 MiB/s wr, 956 op/s 2024-03-28T01:05:46.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:45 smithi174 ceph-mon[36060]: pgmap v628: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 562 MiB/s wr, 956 op/s 2024-03-28T01:05:46.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.145+0000 7fee327fc640 1 -- 172.21.15.169:0/4111898722 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55d4d55fdb80 con 0x55d4d5785320 2024-03-28T01:05:46.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.145+0000 7fee327fc640 10 client.27116 handle_command_reply: tid=0 2024-03-28T01:05:46.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.146+0000 7fee53ff5740 1 client.27116 shutdown 2024-03-28T01:05:46.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.146+0000 7fee53ff5740 1 -- 172.21.15.169:0/4111898722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5718690 msgr2=0x55d4d5723300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:46.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.146+0000 7fee53ff5740 1 --2- 172.21.15.169:0/4111898722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5718690 0x55d4d5723300 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fee44004560 tx=0x7fee44004980 comp rx=0 tx=0).stop 2024-03-28T01:05:46.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.146+0000 7fee53ff5740 1 -- 172.21.15.169:0/4111898722 shutdown_connections 2024-03-28T01:05:46.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.146+0000 7fee53ff5740 1 --2- 172.21.15.169:0/4111898722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d4d5718690 0x55d4d5723300 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.146+0000 7fee53ff5740 1 --2- 172.21.15.169:0/4111898722 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55d4d5785320 0x55d4d5787740 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.146+0000 7fee53ff5740 1 --2- 172.21.15.169:0/4111898722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d4d5713520 0x55d4d57222f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.146+0000 7fee53ff5740 1 --2- 172.21.15.169:0/4111898722 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4d5712b30 0x55d4d5721db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.146+0000 7fee53ff5740 1 -- 172.21.15.169:0/4111898722 >> 172.21.15.169:0/4111898722 conn(0x55d4d56db1a0 msgr2=0x55d4d570cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:46.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.147+0000 7fee327fc640 0 client.27116 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:46.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.147+0000 7fee53ff5740 1 -- 172.21.15.169:0/4111898722 shutdown_connections 2024-03-28T01:05:46.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.147+0000 7fee53ff5740 1 -- 172.21.15.169:0/4111898722 wait complete. 2024-03-28T01:05:46.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.148+0000 7fee53ff5740 20 client.27116 trim_cache size 0 max 16384 2024-03-28T01:05:46.151 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:05:46.151 INFO:teuthology.orchestra.run.smithi169.stdout: "return_code": 0, 2024-03-28T01:05:46.151 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_tag": "cf297dd7-e3ae-4246-8507-9dbfaaed3b5a", 2024-03-28T01:05:46.151 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": "asynchronous" 2024-03-28T01:05:46.152 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:05:46.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.154+0000 7fee4bfff640 1 -- 172.21.15.169:0/2425125008 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fee380a2a10 msgr2=0x7fee380a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:46.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.154+0000 7fee4bfff640 1 --2- 172.21.15.169:0/2425125008 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fee380a2a10 0x7fee380a4ed0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fee44014530 tx=0x7fee44005000 comp rx=0 tx=0).stop 2024-03-28T01:05:46.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.154+0000 7fee4bfff640 1 -- 172.21.15.169:0/2425125008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee4c108d70 msgr2=0x7fee4c118260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:46.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.154+0000 7fee4bfff640 1 --2- 172.21.15.169:0/2425125008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee4c108d70 0x7fee4c118260 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fee3400b450 tx=0x7fee3400b910 comp rx=0 tx=0).stop 2024-03-28T01:05:46.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.155+0000 7fee4bfff640 1 -- 172.21.15.169:0/2425125008 shutdown_connections 2024-03-28T01:05:46.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.155+0000 7fee4bfff640 1 --2- 172.21.15.169:0/2425125008 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fee380a2a10 0x7fee380a4ed0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.155+0000 7fee4bfff640 1 --2- 172.21.15.169:0/2425125008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee4c109730 0x7fee4c1192a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.155+0000 7fee4bfff640 1 --2- 172.21.15.169:0/2425125008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee4c108d70 0x7fee4c118260 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.155+0000 7fee4bfff640 1 --2- 172.21.15.169:0/2425125008 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fee4c072ab0 0x7fee4c117d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.155+0000 7fee4bfff640 1 -- 172.21.15.169:0/2425125008 >> 172.21.15.169:0/2425125008 conn(0x7fee4c06e580 msgr2=0x7fee4c10c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:46.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.158+0000 7fee4bfff640 1 -- 172.21.15.169:0/2425125008 shutdown_connections 2024-03-28T01:05:46.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.161+0000 7fee4bfff640 1 -- 172.21.15.169:0/2425125008 wait complete. 2024-03-28T01:05:46.207 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:05:46.207 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:05:46.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.501+0000 7f5a0404b640 1 -- 172.21.15.169:0/1071672794 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f59fc1086a0 msgr2=0x7f59fc10aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:46.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.501+0000 7f5a0404b640 1 --2- 172.21.15.169:0/1071672794 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f59fc1086a0 0x7f59fc10aad0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f59ec009900 tx=0x7f59ec02bc80 comp rx=0 tx=0).stop 2024-03-28T01:05:46.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.501+0000 7f5a0404b640 1 -- 172.21.15.169:0/1071672794 shutdown_connections 2024-03-28T01:05:46.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.501+0000 7f5a0404b640 1 --2- 172.21.15.169:0/1071672794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59fc10d9c0 0x7f59fc10de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.501+0000 7f5a0404b640 1 --2- 172.21.15.169:0/1071672794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc10b010 0x7f59fc10d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.501+0000 7f5a0404b640 1 --2- 172.21.15.169:0/1071672794 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f59fc1086a0 0x7f59fc10aad0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.501+0000 7f5a0404b640 1 -- 172.21.15.169:0/1071672794 >> 172.21.15.169:0/1071672794 conn(0x7f59fc06e500 msgr2=0x7f59fc070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:46.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.503+0000 7f5a0404b640 1 -- 172.21.15.169:0/1071672794 shutdown_connections 2024-03-28T01:05:46.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.506+0000 7f5a0404b640 1 -- 172.21.15.169:0/1071672794 wait complete. 2024-03-28T01:05:46.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.506+0000 7f5a0404b640 1 Processor -- start 2024-03-28T01:05:46.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.507+0000 7f5a0404b640 1 -- start start 2024-03-28T01:05:46.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.507+0000 7f5a0404b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59fc10b010 0x7f59fc1a6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:46.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.507+0000 7f5a0404b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f59fc10d9c0 0x7f59fc1a6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:46.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.507+0000 7f5a0404b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc0778f0 0x7f59fc079db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:46.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.507+0000 7f5a0404b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc11c1e0 con 0x7f59fc10d9c0 2024-03-28T01:05:46.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.507+0000 7f5a0404b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc11c060 con 0x7f59fc0778f0 2024-03-28T01:05:46.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.507+0000 7f5a0404b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc11c360 con 0x7f59fc10b010 2024-03-28T01:05:46.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.507+0000 7f5a025e8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc0778f0 0x7f59fc079db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:46.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.508+0000 7f5a025e8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc0778f0 0x7f59fc079db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:53424/0 (socket says 172.21.15.169:53424) 2024-03-28T01:05:46.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.508+0000 7f5a025e8640 1 -- 172.21.15.169:0/519622873 learned_addr learned my addr 172.21.15.169:0/519622873 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:46.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.508+0000 7f5a025e8640 1 -- 172.21.15.169:0/519622873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59fc10b010 msgr2=0x7f59fc1a6700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:05:46.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.508+0000 7f5a025e8640 1 --2- 172.21.15.169:0/519622873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59fc10b010 0x7f59fc1a6700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.508+0000 7f5a025e8640 1 -- 172.21.15.169:0/519622873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f59fc10d9c0 msgr2=0x7f59fc1a6c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:05:46.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.508+0000 7f5a025e8640 1 --2- 172.21.15.169:0/519622873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f59fc10d9c0 0x7f59fc1a6c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.508+0000 7f5a025e8640 1 -- 172.21.15.169:0/519622873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc07a3f0 con 0x7f59fc0778f0 2024-03-28T01:05:46.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.509+0000 7f5a025e8640 1 --2- 172.21.15.169:0/519622873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc0778f0 0x7f59fc079db0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f59f800b3c0 tx=0x7f59f800b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:46.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.515+0000 7f59f2ffd640 1 -- 172.21.15.169:0/519622873 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f8011830 con 0x7f59fc0778f0 2024-03-28T01:05:46.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.515+0000 7f5a0404b640 1 -- 172.21.15.169:0/519622873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59fc07a6e0 con 0x7f59fc0778f0 2024-03-28T01:05:46.519 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.515+0000 7f5a0404b640 1 -- 172.21.15.169:0/519622873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59fc07ac30 con 0x7f59fc0778f0 2024-03-28T01:05:46.519 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.515+0000 7f59f2ffd640 1 -- 172.21.15.169:0/519622873 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59f8016070 con 0x7f59fc0778f0 2024-03-28T01:05:46.519 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.516+0000 7f59f2ffd640 1 -- 172.21.15.169:0/519622873 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f800f2e0 con 0x7f59fc0778f0 2024-03-28T01:05:46.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.523+0000 7f59f2ffd640 1 -- 172.21.15.169:0/519622873 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f59f80119d0 con 0x7f59fc0778f0 2024-03-28T01:05:46.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.525+0000 7f59f2ffd640 1 --2- 172.21.15.169:0/519622873 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f59e80a2990 0x7f59e80a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:46.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.525+0000 7f59f2ffd640 1 -- 172.21.15.169:0/519622873 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f59f80ce370 con 0x7f59fc0778f0 2024-03-28T01:05:46.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.526+0000 7f5a06c38740 1 Processor -- start 2024-03-28T01:05:46.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.527+0000 7f5a01de7640 1 --2- 172.21.15.169:0/519622873 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f59e80a2990 0x7f59e80a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:46.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.535+0000 7f5a06c38740 1 -- start start 2024-03-28T01:05:46.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.535+0000 7f5a06c38740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905ea4da0 0x55f905ea51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:46.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.536+0000 7f5a01de7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905ea4da0 0x55f905ea51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:46.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.536+0000 7f5a01de7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905ea4da0 0x55f905ea51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:35988/0 (socket says 172.21.15.169:35988) 2024-03-28T01:05:46.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.536+0000 7f5a01de7640 1 -- 172.21.15.169:0/2523930061 learned_addr learned my addr 172.21.15.169:0/2523930061 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:46.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.536+0000 7f5a06c38740 1 --2- 172.21.15.169:0/2523930061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f905ea5790 0x55f905eaa290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:46.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.536+0000 7f5a06c38740 1 --2- 172.21.15.169:0/2523930061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f905eaa900 0x55f905eacd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:46.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.536+0000 7f5a06c38740 1 -- 172.21.15.169:0/2523930061 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f905e0a3d0 con 0x55f905ea4da0 2024-03-28T01:05:46.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.536+0000 7f5a06c38740 1 -- 172.21.15.169:0/2523930061 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f905db2ad0 con 0x55f905ea5790 2024-03-28T01:05:46.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.536+0000 7f5a06c38740 1 -- 172.21.15.169:0/2523930061 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f905e06470 con 0x55f905eaa900 2024-03-28T01:05:46.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.537+0000 7f5a015e6640 1 --2- 172.21.15.169:0/2523930061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f905ea5790 0x55f905eaa290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:46.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.538+0000 7f5a015e6640 1 -- 172.21.15.169:0/2523930061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f905eaa900 msgr2=0x55f905eacd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:46.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.538+0000 7f5a015e6640 1 --2- 172.21.15.169:0/2523930061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f905eaa900 0x55f905eacd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.538+0000 7f5a015e6640 1 -- 172.21.15.169:0/2523930061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905ea4da0 msgr2=0x55f905ea51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:46.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.538+0000 7f5a015e6640 1 --2- 172.21.15.169:0/2523930061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905ea4da0 0x55f905ea51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.538+0000 7f5a015e6640 1 -- 172.21.15.169:0/2523930061 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f905be87d0 con 0x55f905ea5790 2024-03-28T01:05:46.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.538+0000 7f5a015e6640 1 --2- 172.21.15.169:0/2523930061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f905ea5790 0x55f905eaa290 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f59f4005fd0 tx=0x7f59f400e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dae80bae0535457 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:46.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.539+0000 7f5a01de7640 1 --2- 172.21.15.169:0/2523930061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905ea4da0 0x55f905ea51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:05:46.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.540+0000 7f5a01de7640 1 --2- 172.21.15.169:0/519622873 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f59e80a2990 0x7f59e80a4e50 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f59ec005c20 tx=0x7f59ec000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:46.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.541+0000 7f59e7fff640 1 -- 172.21.15.169:0/2523930061 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4016070 con 0x55f905ea5790 2024-03-28T01:05:46.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.541+0000 7f59e7fff640 1 -- 172.21.15.169:0/2523930061 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59f4003970 con 0x55f905ea5790 2024-03-28T01:05:46.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.541+0000 7f59e7fff640 1 -- 172.21.15.169:0/2523930061 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4004620 con 0x55f905ea5790 2024-03-28T01:05:46.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.544+0000 7f5a06c38740 1 -- 172.21.15.169:0/2523930061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f905ea5790 msgr2=0x55f905eaa290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:46.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.544+0000 7f5a06c38740 1 --2- 172.21.15.169:0/2523930061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f905ea5790 0x55f905eaa290 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f59f4005fd0 tx=0x7f59f400e6c0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.547+0000 7f5a06c38740 1 -- 172.21.15.169:0/2523930061 shutdown_connections 2024-03-28T01:05:46.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.548+0000 7f5a06c38740 1 --2- 172.21.15.169:0/2523930061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f905eaa900 0x55f905eacd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.548+0000 7f5a06c38740 1 --2- 172.21.15.169:0/2523930061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f905ea5790 0x55f905eaa290 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.548+0000 7f5a06c38740 1 --2- 172.21.15.169:0/2523930061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905ea4da0 0x55f905ea51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.548+0000 7f5a06c38740 1 -- 172.21.15.169:0/2523930061 >> 172.21.15.169:0/2523930061 conn(0x55f905ea0310 msgr2=0x55f905ea0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:46.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.551+0000 7f5a06c38740 1 -- 172.21.15.169:0/2523930061 shutdown_connections 2024-03-28T01:05:46.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.552+0000 7f5a06c38740 1 -- 172.21.15.169:0/2523930061 wait complete. 2024-03-28T01:05:46.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.552+0000 7f5a06c38740 1 -- start start 2024-03-28T01:05:46.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.552+0000 7f5a06c38740 1 Processor -- start 2024-03-28T01:05:46.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.567+0000 7f5a06c38740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f905ea4da0 0x55f905eaf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:46.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.567+0000 7f5a06c38740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905ea5790 0x55f905eaf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:46.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.567+0000 7f5a06c38740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f905eaa900 0x55f905eb6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:46.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.567+0000 7f5a06c38740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f905ead8b0 con 0x55f905ea5790 2024-03-28T01:05:46.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.567+0000 7f5a06c38740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f905dc6060 con 0x55f905eaa900 2024-03-28T01:05:46.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.567+0000 7f5a06c38740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f905eada30 con 0x55f905ea4da0 2024-03-28T01:05:46.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.567+0000 7f5a06c38740 10 client.0 _ll_register_callbacks cb 0x55f905da86a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:46.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.567+0000 7f5a015e6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905ea5790 0x55f905eaf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:46.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.567+0000 7f5a015e6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905ea5790 0x55f905eaf850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:35994/0 (socket says 172.21.15.169:35994) 2024-03-28T01:05:46.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.567+0000 7f5a015e6640 1 -- 172.21.15.169:0/343722225 learned_addr learned my addr 172.21.15.169:0/343722225 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:46.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.567+0000 7f5a015e6640 1 -- 172.21.15.169:0/343722225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f905ea4da0 msgr2=0x55f905eaf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:46.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.567+0000 7f5a015e6640 1 --2- 172.21.15.169:0/343722225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f905ea4da0 0x55f905eaf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.567+0000 7f5a015e6640 1 -- 172.21.15.169:0/343722225 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f905eaa900 msgr2=0x55f905eb6890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:05:46.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.567+0000 7f5a015e6640 1 --2- 172.21.15.169:0/343722225 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f905eaa900 0x55f905eb6890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.568+0000 7f5a015e6640 1 -- 172.21.15.169:0/343722225 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f905b1e3f0 con 0x55f905ea5790 2024-03-28T01:05:46.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.571+0000 7f5a015e6640 1 --2- 172.21.15.169:0/343722225 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905ea5790 0x55f905eaf850 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f59f4017000 tx=0x7f59f4005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86f46bab15c1f667 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:46.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.584+0000 7f59e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:05:46.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.584+0000 7f59e57fa640 1 -- 172.21.15.169:0/343722225 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4016040 con 0x55f905ea5790 2024-03-28T01:05:46.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.584+0000 7f59e57fa640 1 -- 172.21.15.169:0/343722225 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f59f4005800 con 0x55f905ea5790 2024-03-28T01:05:46.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.586+0000 7f5a06c38740 1 -- 172.21.15.169:0/343722225 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f59f4003970 con 0x55f905ea5790 2024-03-28T01:05:46.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.600+0000 7f59e57fa640 1 -- 172.21.15.169:0/343722225 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59f4011470 con 0x55f905ea5790 2024-03-28T01:05:46.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.601+0000 7f5a06c38740 10 client.17259 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:46.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.601+0000 7f5a06c38740 1 -- 172.21.15.169:0/343722225 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f905ade0f0 con 0x55f905ea5790 2024-03-28T01:05:46.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.608+0000 7f59e57fa640 1 -- 172.21.15.169:0/343722225 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59f4003970 con 0x55f905ea5790 2024-03-28T01:05:46.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.608+0000 7f59e57fa640 1 -- 172.21.15.169:0/343722225 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f59f402f020 con 0x55f905ea5790 2024-03-28T01:05:46.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.610+0000 7f5a06c38740 10 client.17259 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:46.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.610+0000 7f5a06c38740 10 client.17259 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:46.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.610+0000 7f5a06c38740 20 client.17259 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:46.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.610+0000 7f5a06c38740 1 --2- 172.21.15.169:0/343722225 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f905eba9e0 0x55f905ebce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:46.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.610+0000 7f5a06c38740 4 client.17259 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:05:46.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.610+0000 7f5a06c38740 1 -- 172.21.15.169:0/343722225 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f905e17910 con 0x55f905eba9e0 2024-03-28T01:05:46.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.614+0000 7f5a01de7640 1 --2- 172.21.15.169:0/343722225 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f905eba9e0 0x55f905ebce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:46.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.631+0000 7f5a01de7640 1 --2- 172.21.15.169:0/343722225 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f905eba9e0 0x55f905ebce30 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49e28b13c8a88085 server_cookie=c18e5c0acc435190 in_seq=0 out_seq=0 2024-03-28T01:05:46.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.632+0000 7f59e57fa640 10 client.17259 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:46.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.640+0000 7f59e57fa640 1 -- 172.21.15.169:0/343722225 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f905e17910 con 0x55f905eba9e0 2024-03-28T01:05:46.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.640+0000 7f59e57fa640 10 client.17259 handle_command_reply: tid=0 2024-03-28T01:05:46.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.642+0000 7f5a06c38740 1 client.17259 shutdown 2024-03-28T01:05:46.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.643+0000 7f5a06c38740 1 -- 172.21.15.169:0/343722225 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905ea5790 msgr2=0x55f905eaf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:46.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.643+0000 7f5a06c38740 1 --2- 172.21.15.169:0/343722225 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905ea5790 0x55f905eaf850 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f59f4017000 tx=0x7f59f4005250 comp rx=0 tx=0).stop 2024-03-28T01:05:46.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.651+0000 7f5a06c38740 1 -- 172.21.15.169:0/343722225 shutdown_connections 2024-03-28T01:05:46.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.652+0000 7f5a06c38740 1 --2- 172.21.15.169:0/343722225 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f905eaa900 0x55f905eb6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.652+0000 7f5a06c38740 1 --2- 172.21.15.169:0/343722225 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f905eba9e0 0x55f905ebce30 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.652+0000 7f5a06c38740 1 --2- 172.21.15.169:0/343722225 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905ea5790 0x55f905eaf850 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.652+0000 7f5a06c38740 1 --2- 172.21.15.169:0/343722225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f905ea4da0 0x55f905eaf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.652+0000 7f5a06c38740 1 -- 172.21.15.169:0/343722225 >> 172.21.15.169:0/343722225 conn(0x55f905ea0310 msgr2=0x55f905e9f530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:46.655 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.652+0000 7f59e57fa640 0 client.17259 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:46.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.655+0000 7f5a06c38740 1 -- 172.21.15.169:0/343722225 shutdown_connections 2024-03-28T01:05:46.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.659+0000 7f5a06c38740 1 -- 172.21.15.169:0/343722225 wait complete. 2024-03-28T01:05:46.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.662+0000 7f5a06c38740 20 client.17259 trim_cache size 0 max 16384 2024-03-28T01:05:46.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.710+0000 7f5a06c38740 1 Processor -- start 2024-03-28T01:05:46.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.710+0000 7f5a06c38740 1 -- start start 2024-03-28T01:05:46.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.711+0000 7f5a06c38740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f905ea4da0 0x55f905e9f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:46.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.711+0000 7f5a06c38740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f905ea5790 0x55f905eb6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:46.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.711+0000 7f5a06c38740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905eaa900 0x55f905ec1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:46.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.712+0000 7f5a06c38740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e00120e0 con 0x55f905eaa900 2024-03-28T01:05:46.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.712+0000 7f5a06c38740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e000fbe0 con 0x55f905ea5790 2024-03-28T01:05:46.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.712+0000 7f5a06c38740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e000dc80 con 0x55f905ea4da0 2024-03-28T01:05:46.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.712+0000 7f5a025e8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905eaa900 0x55f905ec1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:46.715 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.712+0000 7f5a025e8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905eaa900 0x55f905ec1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36010/0 (socket says 172.21.15.169:36010) 2024-03-28T01:05:46.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.712+0000 7f5a025e8640 1 -- 172.21.15.169:0/3841187032 learned_addr learned my addr 172.21.15.169:0/3841187032 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:46.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.716+0000 7f5a025e8640 1 -- 172.21.15.169:0/3841187032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f905ea4da0 msgr2=0x55f905e9f3b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:05:46.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.716+0000 7f5a025e8640 1 --2- 172.21.15.169:0/3841187032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f905ea4da0 0x55f905e9f3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.716+0000 7f5a025e8640 1 -- 172.21.15.169:0/3841187032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f905ea5790 msgr2=0x55f905eb6850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:05:46.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.716+0000 7f5a025e8640 1 --2- 172.21.15.169:0/3841187032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f905ea5790 0x55f905eb6850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.716+0000 7f5a025e8640 1 -- 172.21.15.169:0/3841187032 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f905e9fc60 con 0x55f905eaa900 2024-03-28T01:05:46.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.717+0000 7f5a025e8640 1 --2- 172.21.15.169:0/3841187032 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905eaa900 0x55f905ec1730 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f59f8006e50 tx=0x7f59f809a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c869f6b227f7d15 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:46.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.723+0000 7f59e77fe640 1 -- 172.21.15.169:0/3841187032 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f80b3070 con 0x55f905eaa900 2024-03-28T01:05:46.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.723+0000 7f59e77fe640 1 -- 172.21.15.169:0/3841187032 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59f809a970 con 0x55f905eaa900 2024-03-28T01:05:46.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.723+0000 7f59e77fe640 1 -- 172.21.15.169:0/3841187032 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f809a020 con 0x55f905eaa900 2024-03-28T01:05:46.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.724+0000 7f5a06c38740 1 -- 172.21.15.169:0/3841187032 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905eaa900 msgr2=0x55f905ec1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:46.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.724+0000 7f5a06c38740 1 --2- 172.21.15.169:0/3841187032 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905eaa900 0x55f905ec1730 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f59f8006e50 tx=0x7f59f809a570 comp rx=0 tx=0).stop 2024-03-28T01:05:46.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.728+0000 7f5a06c38740 1 -- 172.21.15.169:0/3841187032 shutdown_connections 2024-03-28T01:05:46.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.728+0000 7f5a06c38740 1 --2- 172.21.15.169:0/3841187032 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905eaa900 0x55f905ec1730 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.728+0000 7f5a06c38740 1 --2- 172.21.15.169:0/3841187032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f905ea5790 0x55f905eb6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.728+0000 7f5a06c38740 1 --2- 172.21.15.169:0/3841187032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f905ea4da0 0x55f905e9f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.728+0000 7f5a06c38740 1 -- 172.21.15.169:0/3841187032 >> 172.21.15.169:0/3841187032 conn(0x55f905ea0310 msgr2=0x55f905e9eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:46.738 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.735+0000 7f5a06c38740 1 -- 172.21.15.169:0/3841187032 shutdown_connections 2024-03-28T01:05:46.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.740+0000 7f5a06c38740 1 -- 172.21.15.169:0/3841187032 wait complete. 2024-03-28T01:05:46.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.740+0000 7f5a06c38740 1 -- start start 2024-03-28T01:05:46.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.741+0000 7f5a06c38740 1 Processor -- start 2024-03-28T01:05:46.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.742+0000 7f5a06c38740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f905ea4da0 0x55f905eb6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:46.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.742+0000 7f5a06c38740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905ea5790 0x55f905eb4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:46.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.742+0000 7f5a06c38740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f905eaa900 0x55f905eb4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:46.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.742+0000 7f5a06c38740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e000e280 con 0x55f905ea5790 2024-03-28T01:05:46.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.742+0000 7f5a06c38740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e000f580 con 0x55f905ea4da0 2024-03-28T01:05:46.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.743+0000 7f5a06c38740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e000e8c0 con 0x55f905eaa900 2024-03-28T01:05:46.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.743+0000 7f5a025e8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f905eaa900 0x55f905eb4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:46.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.743+0000 7f5a025e8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f905eaa900 0x55f905eb4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:51674/0 (socket says 172.21.15.169:51674) 2024-03-28T01:05:46.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.743+0000 7f5a025e8640 1 -- 172.21.15.169:0/3466220851 learned_addr learned my addr 172.21.15.169:0/3466220851 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:46.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.743+0000 7f5a015e6640 1 --2- 172.21.15.169:0/3466220851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905ea5790 0x55f905eb4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:46.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.743+0000 7f5a06c38740 10 client.0 _ll_register_callbacks cb 0x7f59f4009b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:46.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.743+0000 7f5a025e8640 1 -- 172.21.15.169:0/3466220851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f905ea4da0 msgr2=0x55f905eb6370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:05:46.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.743+0000 7f5a025e8640 1 --2- 172.21.15.169:0/3466220851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f905ea4da0 0x55f905eb6370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.743+0000 7f5a025e8640 1 -- 172.21.15.169:0/3466220851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905ea5790 msgr2=0x55f905eb4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:46.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.743+0000 7f5a025e8640 1 --2- 172.21.15.169:0/3466220851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905ea5790 0x55f905eb4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.743+0000 7f5a025e8640 1 -- 172.21.15.169:0/3466220851 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f905ea1080 con 0x55f905eaa900 2024-03-28T01:05:46.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.748+0000 7f5a025e8640 1 --2- 172.21.15.169:0/3466220851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f905eaa900 0x55f905eb4ec0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f59f809a670 tx=0x7f59f80a3200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37c333e001f95b3f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:46.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.748+0000 7f59e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:05:46.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.748+0000 7f59e57fa640 1 -- 172.21.15.169:0/3466220851 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f80b3040 con 0x55f905eaa900 2024-03-28T01:05:46.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.749+0000 7f59e57fa640 1 -- 172.21.15.169:0/3466220851 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f59f8099dc0 con 0x55f905eaa900 2024-03-28T01:05:46.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.751+0000 7f5a06c38740 1 -- 172.21.15.169:0/3466220851 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f59f809a970 con 0x55f905eaa900 2024-03-28T01:05:46.761 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.755+0000 7f59e57fa640 1 -- 172.21.15.169:0/3466220851 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59f8099fc0 con 0x55f905eaa900 2024-03-28T01:05:46.761 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.758+0000 7f5a06c38740 10 client.27064 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:46.761 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.758+0000 7f5a06c38740 1 -- 172.21.15.169:0/3466220851 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f905f16d30 con 0x55f905eaa900 2024-03-28T01:05:46.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.762+0000 7f59e57fa640 1 -- 172.21.15.169:0/3466220851 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59f809a970 con 0x55f905eaa900 2024-03-28T01:05:46.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.762+0000 7f59e57fa640 1 -- 172.21.15.169:0/3466220851 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f59f8099830 con 0x55f905eaa900 2024-03-28T01:05:46.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.762+0000 7f5a06c38740 10 client.27064 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:46.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.762+0000 7f5a06c38740 10 client.27064 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:46.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.762+0000 7f5a06c38740 20 client.27064 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:46.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.762+0000 7f5a06c38740 1 --2- 172.21.15.169:0/3466220851 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f905f17440 0x55f905f19860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:46.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.762+0000 7f5a06c38740 4 client.27064 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:05:46.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.762+0000 7f5a06c38740 1 -- 172.21.15.169:0/3466220851 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f905e11c60 con 0x55f905f17440 2024-03-28T01:05:46.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.766+0000 7f5a01de7640 1 --2- 172.21.15.169:0/3466220851 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f905f17440 0x55f905f19860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:46.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.795+0000 7f5a01de7640 1 --2- 172.21.15.169:0/3466220851 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f905f17440 0x55f905f19860 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aacc531fed9e30e server_cookie=b230234023f6b477 in_seq=0 out_seq=0 2024-03-28T01:05:46.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.795+0000 7f59e57fa640 10 client.27064 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:46.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.799+0000 7f59e57fa640 1 -- 172.21.15.169:0/3466220851 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x55f905e11c60 con 0x55f905f17440 2024-03-28T01:05:46.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.799+0000 7f59e57fa640 10 client.27064 handle_command_reply: tid=0 2024-03-28T01:05:46.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.803+0000 7f5a06c38740 1 client.27064 shutdown 2024-03-28T01:05:46.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.804+0000 7f5a06c38740 1 -- 172.21.15.169:0/3466220851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f905eaa900 msgr2=0x55f905eb4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:46.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.804+0000 7f5a06c38740 1 --2- 172.21.15.169:0/3466220851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f905eaa900 0x55f905eb4ec0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f59f809a670 tx=0x7f59f80a3200 comp rx=0 tx=0).stop 2024-03-28T01:05:46.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.808+0000 7f5a06c38740 1 -- 172.21.15.169:0/3466220851 shutdown_connections 2024-03-28T01:05:46.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.808+0000 7f5a06c38740 1 --2- 172.21.15.169:0/3466220851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f905eaa900 0x55f905eb4ec0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.811 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.808+0000 7f5a06c38740 1 --2- 172.21.15.169:0/3466220851 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f905f17440 0x55f905f19860 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.811 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.808+0000 7f5a06c38740 1 --2- 172.21.15.169:0/3466220851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f905ea5790 0x55f905eb4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.811 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.808+0000 7f5a06c38740 1 --2- 172.21.15.169:0/3466220851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f905ea4da0 0x55f905eb6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.811 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.808+0000 7f5a06c38740 1 -- 172.21.15.169:0/3466220851 >> 172.21.15.169:0/3466220851 conn(0x55f905ea0310 msgr2=0x55f905e9f530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:46.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.808+0000 7f59e57fa640 0 client.27064 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:46.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.814+0000 7f5a06c38740 1 -- 172.21.15.169:0/3466220851 shutdown_connections 2024-03-28T01:05:46.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.815+0000 7f5a06c38740 1 -- 172.21.15.169:0/3466220851 wait complete. 2024-03-28T01:05:46.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.815+0000 7f5a06c38740 20 client.27064 trim_cache size 0 max 16384 2024-03-28T01:05:46.822 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:05:46.822 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "scrub active (0 inodes in the stack)", 2024-03-28T01:05:46.822 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": { 2024-03-28T01:05:46.823 INFO:teuthology.orchestra.run.smithi169.stdout: "cf297dd7-e3ae-4246-8507-9dbfaaed3b5a": { 2024-03-28T01:05:46.823 INFO:teuthology.orchestra.run.smithi169.stdout: "path": "/", 2024-03-28T01:05:46.823 INFO:teuthology.orchestra.run.smithi169.stdout: "tag": "cf297dd7-e3ae-4246-8507-9dbfaaed3b5a", 2024-03-28T01:05:46.823 INFO:teuthology.orchestra.run.smithi169.stdout: "options": "recursive,force" 2024-03-28T01:05:46.823 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:05:46.823 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:05:46.823 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:05:46.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.831+0000 7f5a0404b640 1 -- 172.21.15.169:0/519622873 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f59e80a2990 msgr2=0x7f59e80a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:46.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.832+0000 7f5a0404b640 1 --2- 172.21.15.169:0/519622873 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f59e80a2990 0x7f59e80a4e50 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f59ec005c20 tx=0x7f59ec000bc0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.832+0000 7f5a0404b640 1 -- 172.21.15.169:0/519622873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc0778f0 msgr2=0x7f59fc079db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:46.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.832+0000 7f5a0404b640 1 --2- 172.21.15.169:0/519622873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc0778f0 0x7f59fc079db0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f59f800b3c0 tx=0x7f59f800b880 comp rx=0 tx=0).stop 2024-03-28T01:05:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.833+0000 7f5a0404b640 1 -- 172.21.15.169:0/519622873 shutdown_connections 2024-03-28T01:05:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.833+0000 7f5a0404b640 1 --2- 172.21.15.169:0/519622873 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f59e80a2990 0x7f59e80a4e50 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.833+0000 7f5a0404b640 1 --2- 172.21.15.169:0/519622873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc0778f0 0x7f59fc079db0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.833+0000 7f5a0404b640 1 --2- 172.21.15.169:0/519622873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f59fc10d9c0 0x7f59fc1a6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.833+0000 7f5a0404b640 1 --2- 172.21.15.169:0/519622873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59fc10b010 0x7f59fc1a6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:46.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.833+0000 7f5a0404b640 1 -- 172.21.15.169:0/519622873 >> 172.21.15.169:0/519622873 conn(0x7f59fc06e500 msgr2=0x7f59fc10a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:46.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.835+0000 7f5a0404b640 1 -- 172.21.15.169:0/519622873 shutdown_connections 2024-03-28T01:05:46.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:05:46.838+0000 7f5a0404b640 1 -- 172.21.15.169:0/519622873 wait complete. 2024-03-28T01:05:46.898 INFO:tasks.cephfs.filesystem:scrub status for tag:cf297dd7-e3ae-4246-8507-9dbfaaed3b5a - {'path': '/', 'tag': 'cf297dd7-e3ae-4246-8507-9dbfaaed3b5a', 'options': 'recursive,force'} 2024-03-28T01:05:47.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:46 smithi169 ceph-mon[32274]: pgmap v629: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 633 MiB/s wr, 1.24k op/s 2024-03-28T01:05:47.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:46 smithi174 ceph-mon[36060]: pgmap v629: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 633 MiB/s wr, 1.24k op/s 2024-03-28T01:05:47.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:46 smithi177 ceph-mon[36206]: pgmap v629: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 633 MiB/s wr, 1.24k op/s 2024-03-28T01:05:48.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:47 smithi169 ceph-mon[32274]: scrub queued for path: / 2024-03-28T01:05:48.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:47 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:05:48.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:47 smithi169 ceph-mon[32274]: scrub summary: active paths [/] 2024-03-28T01:05:48.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:47 smithi174 ceph-mon[36060]: scrub queued for path: / 2024-03-28T01:05:48.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:47 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:05:48.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:47 smithi174 ceph-mon[36060]: scrub summary: active paths [/] 2024-03-28T01:05:48.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:47 smithi177 ceph-mon[36206]: scrub queued for path: / 2024-03-28T01:05:48.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:47 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:05:48.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:47 smithi177 ceph-mon[36206]: scrub summary: active paths [/] 2024-03-28T01:05:49.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:48 smithi169 ceph-mon[32274]: pgmap v630: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 621 MiB/s wr, 1.30k op/s 2024-03-28T01:05:49.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:48 smithi174 ceph-mon[36060]: pgmap v630: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 621 MiB/s wr, 1.30k op/s 2024-03-28T01:05:49.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:48 smithi177 ceph-mon[36206]: pgmap v630: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 621 MiB/s wr, 1.30k op/s 2024-03-28T01:05:51.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:51 smithi177 ceph-mon[36206]: pgmap v631: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 498 MiB/s wr, 1.18k op/s 2024-03-28T01:05:51.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:51 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:05:52.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:51 smithi169 ceph-mon[32274]: pgmap v631: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 498 MiB/s wr, 1.18k op/s 2024-03-28T01:05:52.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:51 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:05:52.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:51 smithi174 ceph-mon[36060]: pgmap v631: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 498 MiB/s wr, 1.18k op/s 2024-03-28T01:05:52.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:51 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:05:53.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:53 smithi177 ceph-mon[36206]: pgmap v632: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 708 MiB/s wr, 1.29k op/s 2024-03-28T01:05:54.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:53 smithi169 ceph-mon[32274]: pgmap v632: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 708 MiB/s wr, 1.29k op/s 2024-03-28T01:05:54.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:53 smithi174 ceph-mon[36060]: pgmap v632: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 708 MiB/s wr, 1.29k op/s 2024-03-28T01:05:54.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:54 smithi177 ceph-mon[36206]: scrub summary: idle 2024-03-28T01:05:55.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:54 smithi169 ceph-mon[32274]: scrub summary: idle 2024-03-28T01:05:55.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:54 smithi174 ceph-mon[36060]: scrub summary: idle 2024-03-28T01:05:55.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:55 smithi177 ceph-mon[36206]: pgmap v633: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 604 MiB/s wr, 983 op/s 2024-03-28T01:05:56.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:55 smithi169 ceph-mon[32274]: pgmap v633: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 604 MiB/s wr, 983 op/s 2024-03-28T01:05:56.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:55 smithi174 ceph-mon[36060]: pgmap v633: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 604 MiB/s wr, 983 op/s 2024-03-28T01:05:57.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:57 smithi177 ceph-mon[36206]: pgmap v634: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 642 MiB/s wr, 814 op/s 2024-03-28T01:05:58.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:57 smithi169 ceph-mon[32274]: pgmap v634: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 642 MiB/s wr, 814 op/s 2024-03-28T01:05:58.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:57 smithi174 ceph-mon[36060]: pgmap v634: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 642 MiB/s wr, 814 op/s 2024-03-28T01:05:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.410+0000 7f6582f62640 1 -- 172.21.15.169:0/710319167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f657c109700 msgr2=0x7f657c10ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.410+0000 7f6582f62640 1 --2- 172.21.15.169:0/710319167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f657c109700 0x7f657c10ff90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6570009900 tx=0x7f657002bc80 comp rx=0 tx=0).stop 2024-03-28T01:05:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.411+0000 7f6582f62640 1 -- 172.21.15.169:0/710319167 shutdown_connections 2024-03-28T01:05:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.411+0000 7f6582f62640 1 --2- 172.21.15.169:0/710319167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f657c109700 0x7f657c10ff90 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.411+0000 7f6582f62640 1 --2- 172.21.15.169:0/710319167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f657c108d40 0x7f657c1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.411+0000 7f6582f62640 1 --2- 172.21.15.169:0/710319167 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f657c107b50 0x7f657c107f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.411+0000 7f6582f62640 1 -- 172.21.15.169:0/710319167 >> 172.21.15.169:0/710319167 conn(0x7f657c076ce0 msgr2=0x7f657c079120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:58.418 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.415+0000 7f6582f62640 1 -- 172.21.15.169:0/710319167 shutdown_connections 2024-03-28T01:05:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.417+0000 7f6582f62640 1 -- 172.21.15.169:0/710319167 wait complete. 2024-03-28T01:05:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.418+0000 7f6582f62640 1 Processor -- start 2024-03-28T01:05:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.418+0000 7f6582f62640 1 -- start start 2024-03-28T01:05:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.418+0000 7f6582f62640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f657c107b50 0x7f657c0821c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.418+0000 7f6582f62640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f657c108d40 0x7f657c07ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.418+0000 7f6582f62640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f657c109700 0x7f657c07f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.418+0000 7f6582f62640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c113850 con 0x7f657c109700 2024-03-28T01:05:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.419+0000 7f6582f62640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c1136d0 con 0x7f657c107b50 2024-03-28T01:05:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.419+0000 7f6582f62640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c1139d0 con 0x7f657c108d40 2024-03-28T01:05:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.419+0000 7f657bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f657c108d40 0x7f657c07ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.419+0000 7f657bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f657c108d40 0x7f657c07ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:49704/0 (socket says 172.21.15.169:49704) 2024-03-28T01:05:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.419+0000 7f657bfff640 1 -- 172.21.15.169:0/471315593 learned_addr learned my addr 172.21.15.169:0/471315593 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.419+0000 7f65814ff640 1 --2- 172.21.15.169:0/471315593 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f657c109700 0x7f657c07f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.419+0000 7f657bfff640 1 -- 172.21.15.169:0/471315593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f657c107b50 msgr2=0x7f657c0821c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:05:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.419+0000 7f657bfff640 1 --2- 172.21.15.169:0/471315593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f657c107b50 0x7f657c0821c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.419+0000 7f657bfff640 1 -- 172.21.15.169:0/471315593 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f657c109700 msgr2=0x7f657c07f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.419+0000 7f657bfff640 1 --2- 172.21.15.169:0/471315593 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f657c109700 0x7f657c07f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.419+0000 7f657bfff640 1 -- 172.21.15.169:0/471315593 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657c1183a0 con 0x7f657c108d40 2024-03-28T01:05:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.420+0000 7f657bfff640 1 --2- 172.21.15.169:0/471315593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f657c108d40 0x7f657c07ebf0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f656c00b450 tx=0x7f656c00b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.422+0000 7f6579ffb640 1 -- 172.21.15.169:0/471315593 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c015020 con 0x7f657c108d40 2024-03-28T01:05:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.422+0000 7f6579ffb640 1 -- 172.21.15.169:0/471315593 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f656c011d70 con 0x7f657c108d40 2024-03-28T01:05:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.422+0000 7f6582f62640 1 -- 172.21.15.169:0/471315593 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f657c118690 con 0x7f657c108d40 2024-03-28T01:05:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.422+0000 7f6579ffb640 1 -- 172.21.15.169:0/471315593 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c00f340 con 0x7f657c108d40 2024-03-28T01:05:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.422+0000 7f6582f62640 1 -- 172.21.15.169:0/471315593 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f657c118be0 con 0x7f657c108d40 2024-03-28T01:05:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.427+0000 7f6579ffb640 1 -- 172.21.15.169:0/471315593 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f656c018020 con 0x7f657c108d40 2024-03-28T01:05:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.428+0000 7f6579ffb640 1 --2- 172.21.15.169:0/471315593 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f65640a2980 0x7f65640a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.429+0000 7f6585c2f740 1 Processor -- start 2024-03-28T01:05:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.429+0000 7f6579ffb640 1 -- 172.21.15.169:0/471315593 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f656c0ce830 con 0x7f657c108d40 2024-03-28T01:05:58.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.431+0000 7f6585c2f740 1 -- start start 2024-03-28T01:05:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.432+0000 7f6580cfe640 1 --2- 172.21.15.169:0/471315593 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f65640a2980 0x7f65640a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.434+0000 7f6585c2f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f264c36b20 0x55f264c36f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.434+0000 7f6585c2f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c37510 0x55f264c3c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.434+0000 7f6585c2f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f264c3c680 0x55f264c3eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.434+0000 7f6585c2f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f264b9bb10 con 0x55f264c36b20 2024-03-28T01:05:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.434+0000 7f6585c2f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f264b44900 con 0x55f264c37510 2024-03-28T01:05:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.434+0000 7f6585c2f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f264b97bb0 con 0x55f264c3c680 2024-03-28T01:05:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.434+0000 7f657bfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c37510 0x55f264c3c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.434+0000 7f657bfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c37510 0x55f264c3c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:55814/0 (socket says 172.21.15.169:55814) 2024-03-28T01:05:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.435+0000 7f657bfff640 1 -- 172.21.15.169:0/2737764275 learned_addr learned my addr 172.21.15.169:0/2737764275 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.435+0000 7f65814ff640 1 --2- 172.21.15.169:0/2737764275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f264c3c680 0x55f264c3eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.435+0000 7f657bfff640 1 -- 172.21.15.169:0/2737764275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f264c3c680 msgr2=0x55f264c3eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.435+0000 7f657bfff640 1 --2- 172.21.15.169:0/2737764275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f264c3c680 0x55f264c3eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.435+0000 7f657bfff640 1 -- 172.21.15.169:0/2737764275 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f264c36b20 msgr2=0x55f264c36f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.435+0000 7f657bfff640 1 --2- 172.21.15.169:0/2737764275 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f264c36b20 0x55f264c36f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.435+0000 7f657bfff640 1 -- 172.21.15.169:0/2737764275 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2648b0210 con 0x55f264c37510 2024-03-28T01:05:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.435+0000 7f657bfff640 1 --2- 172.21.15.169:0/2737764275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c37510 0x55f264c3c010 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f656c0a30a0 tx=0x7f656c09a790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0b6101e1ee3bb4e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.437+0000 7f6562ffd640 1 -- 172.21.15.169:0/2737764275 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c09ef80 con 0x55f264c37510 2024-03-28T01:05:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.437+0000 7f6562ffd640 1 -- 172.21.15.169:0/2737764275 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f656c0a35d0 con 0x55f264c37510 2024-03-28T01:05:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.438+0000 7f6585c2f740 1 -- 172.21.15.169:0/2737764275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c37510 msgr2=0x55f264c3c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.438+0000 7f6585c2f740 1 --2- 172.21.15.169:0/2737764275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c37510 0x55f264c3c010 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f656c0a30a0 tx=0x7f656c09a790 comp rx=0 tx=0).stop 2024-03-28T01:05:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.441+0000 7f6585c2f740 1 -- 172.21.15.169:0/2737764275 shutdown_connections 2024-03-28T01:05:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.441+0000 7f6585c2f740 1 --2- 172.21.15.169:0/2737764275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f264c3c680 0x55f264c3eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.441+0000 7f6585c2f740 1 --2- 172.21.15.169:0/2737764275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c37510 0x55f264c3c010 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.441+0000 7f6585c2f740 1 --2- 172.21.15.169:0/2737764275 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f264c36b20 0x55f264c36f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.441+0000 7f6585c2f740 1 -- 172.21.15.169:0/2737764275 >> 172.21.15.169:0/2737764275 conn(0x55f264c32090 msgr2=0x55f264c32490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.443+0000 7f6585c2f740 1 -- 172.21.15.169:0/2737764275 shutdown_connections 2024-03-28T01:05:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.443+0000 7f6585c2f740 1 -- 172.21.15.169:0/2737764275 wait complete. 2024-03-28T01:05:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.443+0000 7f6585c2f740 1 -- start start 2024-03-28T01:05:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.443+0000 7f6585c2f740 1 Processor -- start 2024-03-28T01:05:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.443+0000 7f6580cfe640 1 --2- 172.21.15.169:0/471315593 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f65640a2980 0x7f65640a4e40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x55f264c37d70 tx=0x7f657400e6c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.444+0000 7f6585c2f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c36b20 0x55f264c40dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.444+0000 7f6585c2f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f264c37510 0x55f264c41310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.444+0000 7f6585c2f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f264c3c680 0x55f264c48350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.444+0000 7f6585c2f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f264c3f620 con 0x55f264c3c680 2024-03-28T01:05:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.444+0000 7f6585c2f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f264b57d50 con 0x55f264c36b20 2024-03-28T01:05:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.444+0000 7f6585c2f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f264c3f7a0 con 0x55f264c37510 2024-03-28T01:05:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.444+0000 7f6585c2f740 10 client.0 _ll_register_callbacks cb 0x55f264b57720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.445+0000 7f65814ff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f264c3c680 0x55f264c48350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:58.449 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.445+0000 7f65814ff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f264c3c680 0x55f264c48350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:37966/0 (socket says 172.21.15.169:37966) 2024-03-28T01:05:58.449 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.445+0000 7f65814ff640 1 -- 172.21.15.169:0/603178602 learned_addr learned my addr 172.21.15.169:0/603178602 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:58.449 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.446+0000 7f65814ff640 1 -- 172.21.15.169:0/603178602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f264c37510 msgr2=0x55f264c41310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:58.449 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.446+0000 7f65814ff640 1 --2- 172.21.15.169:0/603178602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f264c37510 0x55f264c41310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.449 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.446+0000 7f65814ff640 1 -- 172.21.15.169:0/603178602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c36b20 msgr2=0x55f264c40dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:05:58.450 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.446+0000 7f65814ff640 1 --2- 172.21.15.169:0/603178602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c36b20 0x55f264c40dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.450 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.446+0000 7f65814ff640 1 -- 172.21.15.169:0/603178602 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f26497c620 con 0x55f264c3c680 2024-03-28T01:05:58.450 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.446+0000 7f65814ff640 1 --2- 172.21.15.169:0/603178602 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f264c3c680 0x55f264c48350 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f6570009850 tx=0x7f65700042c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb0a660e833e4c73 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:58.450 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.446+0000 7f65607f8640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:05:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.447+0000 7f65607f8640 1 -- 172.21.15.169:0/603178602 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570005200 con 0x55f264c3c680 2024-03-28T01:05:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.447+0000 7f6585c2f740 1 -- 172.21.15.169:0/603178602 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f656c0a35d0 con 0x55f264c3c680 2024-03-28T01:05:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.447+0000 7f65607f8640 1 -- 172.21.15.169:0/603178602 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f65700053a0 con 0x55f264c3c680 2024-03-28T01:05:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.448+0000 7f65607f8640 1 -- 172.21.15.169:0/603178602 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65700055a0 con 0x55f264c3c680 2024-03-28T01:05:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.448+0000 7f6585c2f740 10 client.17274 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.448+0000 7f6585c2f740 1 -- 172.21.15.169:0/603178602 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f264c4b730 con 0x55f264c3c680 2024-03-28T01:05:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.449+0000 7f65607f8640 1 -- 172.21.15.169:0/603178602 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f656c0a35d0 con 0x55f264c3c680 2024-03-28T01:05:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.449+0000 7f65607f8640 1 -- 172.21.15.169:0/603178602 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f6570003bc0 con 0x55f264c3c680 2024-03-28T01:05:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.449+0000 7f6585c2f740 10 client.17274 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.449+0000 7f6585c2f740 10 client.17274 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.449+0000 7f6585c2f740 20 client.17274 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.449+0000 7f6585c2f740 1 --2- 172.21.15.169:0/603178602 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f264c4d290 0x55f264c4f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.449+0000 7f6585c2f740 4 client.17274 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:05:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.449+0000 7f6585c2f740 1 -- 172.21.15.169:0/603178602 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f264ba9050 con 0x55f264c4d290 2024-03-28T01:05:58.453 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.449+0000 7f6580cfe640 1 --2- 172.21.15.169:0/603178602 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f264c4d290 0x55f264c4f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:58.453 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.450+0000 7f6580cfe640 1 --2- 172.21.15.169:0/603178602 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f264c4d290 0x55f264c4f6f0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e8c00eb18f23e91 server_cookie=949a4d236a15dabc in_seq=0 out_seq=0 2024-03-28T01:05:58.454 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.450+0000 7f65607f8640 10 client.17274 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.453+0000 7f65607f8640 1 -- 172.21.15.169:0/603178602 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f264ba9050 con 0x55f264c4d290 2024-03-28T01:05:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.453+0000 7f65607f8640 10 client.17274 handle_command_reply: tid=0 2024-03-28T01:05:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.453+0000 7f6585c2f740 1 client.17274 shutdown 2024-03-28T01:05:58.457 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.454+0000 7f6585c2f740 1 -- 172.21.15.169:0/603178602 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f264c3c680 msgr2=0x55f264c48350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:58.457 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.454+0000 7f6585c2f740 1 --2- 172.21.15.169:0/603178602 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f264c3c680 0x55f264c48350 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f6570009850 tx=0x7f65700042c0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.457+0000 7f6585c2f740 1 -- 172.21.15.169:0/603178602 shutdown_connections 2024-03-28T01:05:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.457+0000 7f6585c2f740 1 --2- 172.21.15.169:0/603178602 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f264c4d290 0x55f264c4f6f0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.457+0000 7f6585c2f740 1 --2- 172.21.15.169:0/603178602 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f264c3c680 0x55f264c48350 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.457+0000 7f6585c2f740 1 --2- 172.21.15.169:0/603178602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f264c37510 0x55f264c41310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.457+0000 7f6585c2f740 1 --2- 172.21.15.169:0/603178602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c36b20 0x55f264c40dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.457+0000 7f6585c2f740 1 -- 172.21.15.169:0/603178602 >> 172.21.15.169:0/603178602 conn(0x55f264c32090 msgr2=0x55f264c31290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.457+0000 7f65607f8640 0 client.17274 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.457+0000 7f6585c2f740 1 -- 172.21.15.169:0/603178602 shutdown_connections 2024-03-28T01:05:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.457+0000 7f6585c2f740 1 -- 172.21.15.169:0/603178602 wait complete. 2024-03-28T01:05:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.458+0000 7f6585c2f740 20 client.17274 trim_cache size 0 max 16384 2024-03-28T01:05:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.486+0000 7f6585c2f740 1 Processor -- start 2024-03-28T01:05:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.486+0000 7f6585c2f740 1 -- start start 2024-03-28T01:05:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.487+0000 7f6585c2f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c36b20 0x55f264c32470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.487+0000 7f6585c2f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f264c37510 0x55f264c48310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.487+0000 7f6580cfe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c36b20 0x55f264c32470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.487+0000 7f6580cfe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c36b20 0x55f264c32470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:55834/0 (socket says 172.21.15.169:55834) 2024-03-28T01:05:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.487+0000 7f6580cfe640 1 -- 172.21.15.169:0/2849464277 learned_addr learned my addr 172.21.15.169:0/2849464277 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.487+0000 7f657bfff640 1 --2- 172.21.15.169:0/2849464277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f264c37510 0x55f264c48310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.487+0000 7f6585c2f740 1 --2- 172.21.15.169:0/2849464277 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f264c3c680 0x55f264c38790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.488+0000 7f6585c2f740 1 -- 172.21.15.169:0/2849464277 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c1cc290 con 0x55f264c3c680 2024-03-28T01:05:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.488+0000 7f6585c2f740 1 -- 172.21.15.169:0/2849464277 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c123070 con 0x55f264c36b20 2024-03-28T01:05:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.488+0000 7f6585c2f740 1 -- 172.21.15.169:0/2849464277 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c121110 con 0x55f264c37510 2024-03-28T01:05:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.488+0000 7f6580cfe640 1 -- 172.21.15.169:0/2849464277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f264c37510 msgr2=0x55f264c48310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:58.494 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.488+0000 7f6580cfe640 1 --2- 172.21.15.169:0/2849464277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f264c37510 0x55f264c48310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.488+0000 7f6580cfe640 1 -- 172.21.15.169:0/2849464277 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f264c3c680 msgr2=0x55f264c38790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:58.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.488+0000 7f6580cfe640 1 --2- 172.21.15.169:0/2849464277 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f264c3c680 0x55f264c38790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.495 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.488+0000 7f6580cfe640 1 -- 172.21.15.169:0/2849464277 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f264c41f50 con 0x55f264c36b20 2024-03-28T01:05:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.498+0000 7f6580cfe640 1 --2- 172.21.15.169:0/2849464277 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c36b20 0x55f264c32470 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f657c080140 tx=0x7f6574011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=370a4dfa2ee4c49 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.502+0000 7f65627fc640 1 -- 172.21.15.169:0/2849464277 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65740118c0 con 0x55f264c36b20 2024-03-28T01:05:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.502+0000 7f65627fc640 1 -- 172.21.15.169:0/2849464277 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6574017070 con 0x55f264c36b20 2024-03-28T01:05:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.502+0000 7f65627fc640 1 -- 172.21.15.169:0/2849464277 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657401fcd0 con 0x55f264c36b20 2024-03-28T01:05:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.503+0000 7f6585c2f740 1 -- 172.21.15.169:0/2849464277 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c36b20 msgr2=0x55f264c32470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:05:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.503+0000 7f6585c2f740 1 --2- 172.21.15.169:0/2849464277 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c36b20 0x55f264c32470 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f657c080140 tx=0x7f6574011000 comp rx=0 tx=0).stop 2024-03-28T01:05:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.503+0000 7f6585c2f740 1 -- 172.21.15.169:0/2849464277 shutdown_connections 2024-03-28T01:05:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.503+0000 7f6585c2f740 1 --2- 172.21.15.169:0/2849464277 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f264c3c680 0x55f264c38790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.503+0000 7f6585c2f740 1 --2- 172.21.15.169:0/2849464277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f264c37510 0x55f264c48310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.503+0000 7f6585c2f740 1 --2- 172.21.15.169:0/2849464277 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c36b20 0x55f264c32470 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.503+0000 7f6585c2f740 1 -- 172.21.15.169:0/2849464277 >> 172.21.15.169:0/2849464277 conn(0x55f264c32090 msgr2=0x55f264c3bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:05:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.504+0000 7f6585c2f740 1 -- 172.21.15.169:0/2849464277 shutdown_connections 2024-03-28T01:05:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.504+0000 7f6585c2f740 1 -- 172.21.15.169:0/2849464277 wait complete. 2024-03-28T01:05:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.504+0000 7f6585c2f740 1 -- start start 2024-03-28T01:05:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.504+0000 7f6585c2f740 1 Processor -- start 2024-03-28T01:05:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.505+0000 7f6585c2f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c36b20 0x55f264c451c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.505+0000 7f6585c2f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f264c37510 0x55f264c45700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.505+0000 7f6585c2f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f264c3c680 0x55f264ca6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.505+0000 7f6585c2f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c121710 con 0x55f264c3c680 2024-03-28T01:05:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.505+0000 7f6585c2f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c122a10 con 0x55f264c36b20 2024-03-28T01:05:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.505+0000 7f6585c2f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c121d50 con 0x55f264c37510 2024-03-28T01:05:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.505+0000 7f6585c2f740 10 client.0 _ll_register_callbacks cb 0x7f657002cdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:05:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.505+0000 7f657bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f264c37510 0x55f264c45700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.506+0000 7f657bfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f264c37510 0x55f264c45700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:49738/0 (socket says 172.21.15.169:49738) 2024-03-28T01:05:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.506+0000 7f657bfff640 1 -- 172.21.15.169:0/1801752614 learned_addr learned my addr 172.21.15.169:0/1801752614 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:05:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.506+0000 7f65814ff640 1 --2- 172.21.15.169:0/1801752614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f264c3c680 0x55f264ca6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.506+0000 7f6580cfe640 1 --2- 172.21.15.169:0/1801752614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c36b20 0x55f264c451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.506+0000 7f657bfff640 1 -- 172.21.15.169:0/1801752614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c36b20 msgr2=0x55f264c451c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.506+0000 7f657bfff640 1 --2- 172.21.15.169:0/1801752614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c36b20 0x55f264c451c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.507+0000 7f657bfff640 1 -- 172.21.15.169:0/1801752614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f264c3c680 msgr2=0x55f264ca6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:05:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.507+0000 7f657bfff640 1 --2- 172.21.15.169:0/1801752614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f264c3c680 0x55f264ca6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:05:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.507+0000 7f657bfff640 1 -- 172.21.15.169:0/1801752614 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f264c45da0 con 0x55f264c37510 2024-03-28T01:05:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.509+0000 7f657bfff640 1 --2- 172.21.15.169:0/1801752614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f264c37510 0x55f264c45700 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f656c09adc0 tx=0x7f656c0046e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a69066dc0a73713 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:05:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.513+0000 7f65607f8640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:05:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.513+0000 7f65607f8640 1 -- 172.21.15.169:0/1801752614 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c0a35d0 con 0x55f264c37510 2024-03-28T01:05:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.513+0000 7f6585c2f740 1 -- 172.21.15.169:0/1801752614 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6574017070 con 0x55f264c37510 2024-03-28T01:05:58.516 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.513+0000 7f65607f8640 1 -- 172.21.15.169:0/1801752614 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f656c099130 con 0x55f264c37510 2024-03-28T01:05:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.518+0000 7f65607f8640 1 -- 172.21.15.169:0/1801752614 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f656c09ef80 con 0x55f264c37510 2024-03-28T01:05:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.518+0000 7f6585c2f740 10 client.27076 fetch_fsmap learned FSMap version 25 2024-03-28T01:05:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.518+0000 7f6585c2f740 1 -- 172.21.15.169:0/1801752614 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f264cb92b0 con 0x55f264c37510 2024-03-28T01:05:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.521+0000 7f65607f8640 1 -- 172.21.15.169:0/1801752614 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6574017070 con 0x55f264c37510 2024-03-28T01:05:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.522+0000 7f65607f8640 1 -- 172.21.15.169:0/1801752614 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f656c099cf0 con 0x55f264c37510 2024-03-28T01:05:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.522+0000 7f6585c2f740 10 client.27076 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:05:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.522+0000 7f6585c2f740 10 client.27076 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:05:58.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.522+0000 7f6585c2f740 20 client.27076 populate_metadata read hostname 'smithi169' 2024-03-28T01:05:58.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.522+0000 7f6585c2f740 1 --2- 172.21.15.169:0/1801752614 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f264cb9be0 0x55f264cbc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:05:58.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.522+0000 7f6585c2f740 4 client.27076 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:05:58.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.522+0000 7f6585c2f740 1 -- 172.21.15.169:0/1801752614 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f264ba33a0 con 0x55f264cb9be0 2024-03-28T01:05:58.526 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.523+0000 7f6580cfe640 1 --2- 172.21.15.169:0/1801752614 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f264cb9be0 0x55f264cbc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:05:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.525+0000 7f6580cfe640 1 --2- 172.21.15.169:0/1801752614 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f264cb9be0 0x55f264cbc000 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77fbad3578643548 server_cookie=99cb912da024ea99 in_seq=0 out_seq=0 2024-03-28T01:05:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:05:58.525+0000 7f65607f8640 10 client.27076 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:05:59.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:05:59 smithi177 ceph-mon[36206]: pgmap v635: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 620 MiB/s wr, 543 op/s 2024-03-28T01:06:00.054 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:05:59 smithi169 ceph-mon[32274]: pgmap v635: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 620 MiB/s wr, 543 op/s 2024-03-28T01:06:00.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:05:59 smithi174 ceph-mon[36060]: pgmap v635: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 620 MiB/s wr, 543 op/s 2024-03-28T01:06:02.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:01 smithi169 ceph-mon[32274]: pgmap v636: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 570 MiB/s wr, 426 op/s 2024-03-28T01:06:02.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:01 smithi174 ceph-mon[36060]: pgmap v636: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 570 MiB/s wr, 426 op/s 2024-03-28T01:06:02.133 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.130+0000 7f65607f8640 1 -- 172.21.15.169:0/1801752614 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f264ba33a0 con 0x55f264cb9be0 2024-03-28T01:06:02.134 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.130+0000 7f65607f8640 10 client.27076 handle_command_reply: tid=0 2024-03-28T01:06:02.134 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.130+0000 7f6585c2f740 1 client.27076 shutdown 2024-03-28T01:06:02.134 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.131+0000 7f6585c2f740 1 -- 172.21.15.169:0/1801752614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f264c37510 msgr2=0x55f264c45700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:02.134 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.131+0000 7f6585c2f740 1 --2- 172.21.15.169:0/1801752614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f264c37510 0x55f264c45700 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f656c09adc0 tx=0x7f656c0046e0 comp rx=0 tx=0).stop 2024-03-28T01:06:02.135 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.132+0000 7f6585c2f740 1 -- 172.21.15.169:0/1801752614 shutdown_connections 2024-03-28T01:06:02.135 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.132+0000 7f6585c2f740 1 --2- 172.21.15.169:0/1801752614 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55f264cb9be0 0x55f264cbc000 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:02.135 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.132+0000 7f6585c2f740 1 --2- 172.21.15.169:0/1801752614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55f264c3c680 0x55f264ca6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:02.135 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.132+0000 7f6585c2f740 1 --2- 172.21.15.169:0/1801752614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f264c37510 0x55f264c45700 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:02.135 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.132+0000 7f6585c2f740 1 --2- 172.21.15.169:0/1801752614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f264c36b20 0x55f264c451c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:02.135 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.132+0000 7f6585c2f740 1 -- 172.21.15.169:0/1801752614 >> 172.21.15.169:0/1801752614 conn(0x55f264c32090 msgr2=0x55f264c32470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.132+0000 7f65607f8640 0 client.27076 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.132+0000 7f6585c2f740 1 -- 172.21.15.169:0/1801752614 shutdown_connections 2024-03-28T01:06:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.132+0000 7f6585c2f740 1 -- 172.21.15.169:0/1801752614 wait complete. 2024-03-28T01:06:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.132+0000 7f6585c2f740 20 client.27076 trim_cache size 0 max 16384 2024-03-28T01:06:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:06:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:06:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:06:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:06:02.146 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.143+0000 7f6582f62640 1 -- 172.21.15.169:0/471315593 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f65640a2980 msgr2=0x7f65640a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:02.146 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.143+0000 7f6582f62640 1 --2- 172.21.15.169:0/471315593 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f65640a2980 0x7f65640a4e40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x55f264c37d70 tx=0x7f657400e6c0 comp rx=0 tx=0).stop 2024-03-28T01:06:02.146 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.143+0000 7f6582f62640 1 -- 172.21.15.169:0/471315593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f657c108d40 msgr2=0x7f657c07ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:02.146 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.143+0000 7f6582f62640 1 --2- 172.21.15.169:0/471315593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f657c108d40 0x7f657c07ebf0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f656c00b450 tx=0x7f656c00b910 comp rx=0 tx=0).stop 2024-03-28T01:06:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.151+0000 7f6582f62640 1 -- 172.21.15.169:0/471315593 shutdown_connections 2024-03-28T01:06:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.151+0000 7f6582f62640 1 --2- 172.21.15.169:0/471315593 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f65640a2980 0x7f65640a4e40 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.151+0000 7f6582f62640 1 --2- 172.21.15.169:0/471315593 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f657c109700 0x7f657c07f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.151+0000 7f6582f62640 1 --2- 172.21.15.169:0/471315593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f657c108d40 0x7f657c07ebf0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.151+0000 7f6582f62640 1 --2- 172.21.15.169:0/471315593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f657c107b50 0x7f657c0821c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.151+0000 7f6582f62640 1 -- 172.21.15.169:0/471315593 >> 172.21.15.169:0/471315593 conn(0x7f657c076ce0 msgr2=0x7f657c10bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:02.156 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.153+0000 7f6582f62640 1 -- 172.21.15.169:0/471315593 shutdown_connections 2024-03-28T01:06:02.159 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:02.154+0000 7f6582f62640 1 -- 172.21.15.169:0/471315593 wait complete. 2024-03-28T01:06:02.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:01 smithi177 ceph-mon[36206]: pgmap v636: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 570 MiB/s wr, 426 op/s 2024-03-28T01:06:04.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:03 smithi169 ceph-mon[32274]: pgmap v637: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 744 MiB/s wr, 702 op/s 2024-03-28T01:06:04.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:03 smithi174 ceph-mon[36060]: pgmap v637: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 744 MiB/s wr, 702 op/s 2024-03-28T01:06:04.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:03 smithi177 ceph-mon[36206]: pgmap v637: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 744 MiB/s wr, 702 op/s 2024-03-28T01:06:05.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:04 smithi169 ceph-mon[32274]: pgmap v638: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 594 MiB/s wr, 778 op/s 2024-03-28T01:06:05.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:04 smithi174 ceph-mon[36060]: pgmap v638: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 594 MiB/s wr, 778 op/s 2024-03-28T01:06:05.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:04 smithi177 ceph-mon[36206]: pgmap v638: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 594 MiB/s wr, 778 op/s 2024-03-28T01:06:07.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:07 smithi177 ceph-mon[36206]: pgmap v639: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 658 MiB/s wr, 1.06k op/s 2024-03-28T01:06:08.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:07 smithi169 ceph-mon[32274]: pgmap v639: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 658 MiB/s wr, 1.06k op/s 2024-03-28T01:06:08.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:07 smithi174 ceph-mon[36060]: pgmap v639: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 658 MiB/s wr, 1.06k op/s 2024-03-28T01:06:09.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:09 smithi177 ceph-mon[36206]: pgmap v640: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 649 MiB/s wr, 1.07k op/s 2024-03-28T01:06:10.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:09 smithi169 ceph-mon[32274]: pgmap v640: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 649 MiB/s wr, 1.07k op/s 2024-03-28T01:06:10.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:09 smithi174 ceph-mon[36060]: pgmap v640: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 649 MiB/s wr, 1.07k op/s 2024-03-28T01:06:11.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:11 smithi177 ceph-mon[36206]: pgmap v641: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 579 MiB/s wr, 1.03k op/s 2024-03-28T01:06:12.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:11 smithi169 ceph-mon[32274]: pgmap v641: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 579 MiB/s wr, 1.03k op/s 2024-03-28T01:06:12.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:11 smithi174 ceph-mon[36060]: pgmap v641: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 579 MiB/s wr, 1.03k op/s 2024-03-28T01:06:14.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:13 smithi169 ceph-mon[32274]: pgmap v642: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 724 MiB/s wr, 1.11k op/s 2024-03-28T01:06:14.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:13 smithi174 ceph-mon[36060]: pgmap v642: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 724 MiB/s wr, 1.11k op/s 2024-03-28T01:06:14.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:13 smithi177 ceph-mon[36206]: pgmap v642: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 724 MiB/s wr, 1.11k op/s 2024-03-28T01:06:15.397 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.393+0000 7f75b570c640 1 -- 172.21.15.169:0/3500540352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75b0076440 msgr2=0x7f75b00788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:15.398 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.393+0000 7f75b570c640 1 --2- 172.21.15.169:0/3500540352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75b0076440 0x7f75b00788b0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f75a4009900 tx=0x7f75a402bc80 comp rx=0 tx=0).stop 2024-03-28T01:06:15.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.397+0000 7f75b570c640 1 -- 172.21.15.169:0/3500540352 shutdown_connections 2024-03-28T01:06:15.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.397+0000 7f75b570c640 1 --2- 172.21.15.169:0/3500540352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75b0078df0 0x7f75b007b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.398+0000 7f75b570c640 1 --2- 172.21.15.169:0/3500540352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75b0076440 0x7f75b00788b0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.398+0000 7f75b570c640 1 --2- 172.21.15.169:0/3500540352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75b0073560 0x7f75b0073980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.398+0000 7f75b570c640 1 -- 172.21.15.169:0/3500540352 >> 172.21.15.169:0/3500540352 conn(0x7f75b006e6e0 msgr2=0x7f75b0070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:15.405 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.402+0000 7f75b570c640 1 -- 172.21.15.169:0/3500540352 shutdown_connections 2024-03-28T01:06:15.407 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.404+0000 7f75b570c640 1 -- 172.21.15.169:0/3500540352 wait complete. 2024-03-28T01:06:15.407 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.404+0000 7f75b570c640 1 Processor -- start 2024-03-28T01:06:15.412 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.406+0000 7f75b570c640 1 -- start start 2024-03-28T01:06:15.412 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.406+0000 7f75b570c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75b0073560 0x7f75b0082af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:15.412 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.406+0000 7f75b570c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75b0076440 0x7f75b0083030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:15.412 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.406+0000 7f75b570c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75b0078df0 0x7f75b0083570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:15.412 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.406+0000 7f75b570c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b012b4b0 con 0x7f75b0073560 2024-03-28T01:06:15.412 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.406+0000 7f75b570c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b012b330 con 0x7f75b0076440 2024-03-28T01:06:15.412 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.406+0000 7f75b570c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b012b630 con 0x7f75b0078df0 2024-03-28T01:06:15.412 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.407+0000 7f75af7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75b0078df0 0x7f75b0083570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.407+0000 7f75af7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75b0078df0 0x7f75b0083570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:46952/0 (socket says 172.21.15.169:46952) 2024-03-28T01:06:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.407+0000 7f75af7fe640 1 -- 172.21.15.169:0/3092354706 learned_addr learned my addr 172.21.15.169:0/3092354706 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.407+0000 7f75ae7fc640 1 --2- 172.21.15.169:0/3092354706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75b0076440 0x7f75b0083030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.407+0000 7f75af7fe640 1 -- 172.21.15.169:0/3092354706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75b0076440 msgr2=0x7f75b0083030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.407+0000 7f75af7fe640 1 --2- 172.21.15.169:0/3092354706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75b0076440 0x7f75b0083030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.407+0000 7f75af7fe640 1 -- 172.21.15.169:0/3092354706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75b0073560 msgr2=0x7f75b0082af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:06:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.407+0000 7f75af7fe640 1 --2- 172.21.15.169:0/3092354706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75b0073560 0x7f75b0082af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.407+0000 7f75af7fe640 1 -- 172.21.15.169:0/3092354706 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b0131eb0 con 0x7f75b0078df0 2024-03-28T01:06:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.408+0000 7f75af7fe640 1 --2- 172.21.15.169:0/3092354706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75b0078df0 0x7f75b0083570 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f75a000c970 tx=0x7f75a000ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:15.421 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.410+0000 7f7597fff640 1 -- 172.21.15.169:0/3092354706 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a0007bb0 con 0x7f75b0078df0 2024-03-28T01:06:15.421 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.410+0000 7f75b570c640 1 -- 172.21.15.169:0/3092354706 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b01321a0 con 0x7f75b0078df0 2024-03-28T01:06:15.421 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.410+0000 7f75b570c640 1 -- 172.21.15.169:0/3092354706 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b01326c0 con 0x7f75b0078df0 2024-03-28T01:06:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.410+0000 7f7597fff640 1 -- 172.21.15.169:0/3092354706 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75a0007d50 con 0x7f75b0078df0 2024-03-28T01:06:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.410+0000 7f7597fff640 1 -- 172.21.15.169:0/3092354706 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a000f450 con 0x7f75b0078df0 2024-03-28T01:06:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.416+0000 7f75b83cc740 1 Processor -- start 2024-03-28T01:06:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.417+0000 7f75b83cc740 1 -- start start 2024-03-28T01:06:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.417+0000 7f75b83cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c60920 0x55ce53c60d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.417+0000 7f75b83cc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c61310 0x55ce53c65e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.417+0000 7f75b83cc740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c66480 0x55ce53c68910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.417+0000 7f75b83cc740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce53bc5b10 con 0x55ce53c66480 2024-03-28T01:06:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.417+0000 7f75b83cc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce53b6e900 con 0x55ce53c61310 2024-03-28T01:06:15.422 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.417+0000 7f75b83cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce53bc1bb0 con 0x55ce53c60920 2024-03-28T01:06:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.417+0000 7f75ae7fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c60920 0x55ce53c60d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.417+0000 7f75ae7fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c60920 0x55ce53c60d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:46964/0 (socket says 172.21.15.169:46964) 2024-03-28T01:06:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.417+0000 7f75ae7fc640 1 -- 172.21.15.169:0/2777777313 learned_addr learned my addr 172.21.15.169:0/2777777313 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.418+0000 7f75ae7fc640 1 -- 172.21.15.169:0/2777777313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c61310 msgr2=0x55ce53c65e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:06:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.418+0000 7f75ae7fc640 1 --2- 172.21.15.169:0/2777777313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c61310 0x55ce53c65e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.418+0000 7f75ae7fc640 1 -- 172.21.15.169:0/2777777313 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c66480 msgr2=0x55ce53c68910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.418+0000 7f75ae7fc640 1 --2- 172.21.15.169:0/2777777313 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c66480 0x55ce53c68910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.418+0000 7f75ae7fc640 1 -- 172.21.15.169:0/2777777313 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce538da210 con 0x55ce53c60920 2024-03-28T01:06:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.421+0000 7f75ae7fc640 1 --2- 172.21.15.169:0/2777777313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c60920 0x55ce53c60d40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f75a402c800 tx=0x7f75a402cad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc8d82c00ad36dcf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.421+0000 7f75957fa640 1 -- 172.21.15.169:0/2777777313 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a4004030 con 0x55ce53c60920 2024-03-28T01:06:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.421+0000 7f75957fa640 1 -- 172.21.15.169:0/2777777313 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75a4004ba0 con 0x55ce53c60920 2024-03-28T01:06:15.427 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.422+0000 7f7597fff640 1 -- 172.21.15.169:0/3092354706 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f75a00046c0 con 0x7f75b0078df0 2024-03-28T01:06:15.427 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.423+0000 7f75b83cc740 1 -- 172.21.15.169:0/2777777313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c60920 msgr2=0x55ce53c60d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:15.427 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.423+0000 7f75b83cc740 1 --2- 172.21.15.169:0/2777777313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c60920 0x55ce53c60d40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f75a402c800 tx=0x7f75a402cad0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.427 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.423+0000 7f7597fff640 1 --2- 172.21.15.169:0/3092354706 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f75980a2ae0 0x7f75980a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:15.427 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.423+0000 7f75b83cc740 1 -- 172.21.15.169:0/2777777313 shutdown_connections 2024-03-28T01:06:15.427 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.423+0000 7f75b83cc740 1 --2- 172.21.15.169:0/2777777313 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c66480 0x55ce53c68910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.427 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.423+0000 7f75b83cc740 1 --2- 172.21.15.169:0/2777777313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c61310 0x55ce53c65e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.427 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.423+0000 7f75b83cc740 1 --2- 172.21.15.169:0/2777777313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c60920 0x55ce53c60d40 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.427 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.423+0000 7f75b83cc740 1 -- 172.21.15.169:0/2777777313 >> 172.21.15.169:0/2777777313 conn(0x55ce53c5be90 msgr2=0x55ce53c5c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:15.428 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.424+0000 7f75b83cc740 1 -- 172.21.15.169:0/2777777313 shutdown_connections 2024-03-28T01:06:15.429 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.424+0000 7f7597fff640 1 -- 172.21.15.169:0/3092354706 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f75a00ced00 con 0x7f75b0078df0 2024-03-28T01:06:15.429 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.424+0000 7f75b83cc740 1 -- 172.21.15.169:0/2777777313 wait complete. 2024-03-28T01:06:15.430 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.424+0000 7f75aeffd640 1 --2- 172.21.15.169:0/3092354706 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f75980a2ae0 0x7f75980a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:15.430 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.424+0000 7f75b83cc740 1 -- start start 2024-03-28T01:06:15.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.424+0000 7f75b83cc740 1 Processor -- start 2024-03-28T01:06:15.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.425+0000 7f75b83cc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c61310 0x55ce53c6abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:15.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.425+0000 7f75b83cc740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c66480 0x55ce53c6b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:15.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.425+0000 7f75b83cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c70140 0x55ce53c72530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.425+0000 7f75b83cc740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce53c69420 con 0x55ce53c66480 2024-03-28T01:06:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.425+0000 7f75b83cc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce53b81d50 con 0x55ce53c61310 2024-03-28T01:06:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.425+0000 7f75b83cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce53c695a0 con 0x55ce53c70140 2024-03-28T01:06:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.425+0000 7f75b83cc740 10 client.0 _ll_register_callbacks cb 0x55ce53b81720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:06:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.426+0000 7f75ae7fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c66480 0x55ce53c6b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.426+0000 7f75ae7fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c66480 0x55ce53c6b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:59588/0 (socket says 172.21.15.169:59588) 2024-03-28T01:06:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.426+0000 7f75ae7fc640 1 -- 172.21.15.169:0/614569888 learned_addr learned my addr 172.21.15.169:0/614569888 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.426+0000 7f75af7fe640 1 --2- 172.21.15.169:0/614569888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c70140 0x55ce53c72530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.427+0000 7f75aeffd640 1 --2- 172.21.15.169:0/614569888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c61310 0x55ce53c6abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.427+0000 7f75af7fe640 1 -- 172.21.15.169:0/614569888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c61310 msgr2=0x55ce53c6abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.427+0000 7f75af7fe640 1 --2- 172.21.15.169:0/614569888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c61310 0x55ce53c6abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.427+0000 7f75af7fe640 1 -- 172.21.15.169:0/614569888 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c66480 msgr2=0x55ce53c6b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.427+0000 7f75af7fe640 1 --2- 172.21.15.169:0/614569888 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c66480 0x55ce53c6b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.427+0000 7f75af7fe640 1 -- 172.21.15.169:0/614569888 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ce539a6620 con 0x55ce53c70140 2024-03-28T01:06:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.427+0000 7f75af7fe640 1 --2- 172.21.15.169:0/614569888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c70140 0x55ce53c72530 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f75a0007590 tx=0x7f75a009bae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5eaa678c90407d76 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.428+0000 7f75aeffd640 1 --2- 172.21.15.169:0/614569888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c61310 0x55ce53c6abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:06:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.428+0000 7f7592ff5640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:06:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.428+0000 7f7592ff5640 1 -- 172.21.15.169:0/614569888 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a00b3020 con 0x55ce53c70140 2024-03-28T01:06:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.429+0000 7f75b83cc740 1 -- 172.21.15.169:0/614569888 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ce53bd3050 con 0x55ce53c70140 2024-03-28T01:06:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.429+0000 7f7592ff5640 1 -- 172.21.15.169:0/614569888 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f75a009a770 con 0x55ce53c70140 2024-03-28T01:06:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.429+0000 7f7592ff5640 1 -- 172.21.15.169:0/614569888 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75a00a3ac0 con 0x55ce53c70140 2024-03-28T01:06:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.429+0000 7f75b83cc740 10 client.27085 fetch_fsmap learned FSMap version 25 2024-03-28T01:06:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.429+0000 7f75b83cc740 1 -- 172.21.15.169:0/614569888 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ce53c97a10 con 0x55ce53c70140 2024-03-28T01:06:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.430+0000 7f7592ff5640 1 -- 172.21.15.169:0/614569888 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ce53bd3050 con 0x55ce53c70140 2024-03-28T01:06:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.430+0000 7f7592ff5640 1 -- 172.21.15.169:0/614569888 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f75a00a0390 con 0x55ce53c70140 2024-03-28T01:06:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.430+0000 7f75b83cc740 10 client.27085 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:06:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.430+0000 7f75b83cc740 10 client.27085 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:06:15.436 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.430+0000 7f75b83cc740 20 client.27085 populate_metadata read hostname 'smithi169' 2024-03-28T01:06:15.436 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.430+0000 7f75b83cc740 1 --2- 172.21.15.169:0/614569888 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ce53c99550 0x55ce53c9b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:15.436 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.430+0000 7f75b83cc740 4 client.27085 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:06:15.436 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.430+0000 7f75b83cc740 1 -- 172.21.15.169:0/614569888 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ce53bcd3a0 con 0x55ce53c99550 2024-03-28T01:06:15.439 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.436+0000 7f75aeffd640 1 --2- 172.21.15.169:0/3092354706 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f75980a2ae0 0x7f75980a4fa0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x55ce53c5ce00 tx=0x7f75a800a480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:15.439 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.436+0000 7f75aeffd640 1 --2- 172.21.15.169:0/614569888 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ce53c99550 0x55ce53c9b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:15.464 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.460+0000 7f7592ff5640 10 client.27085 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:15.464 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.460+0000 7f75aeffd640 1 --2- 172.21.15.169:0/614569888 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ce53c99550 0x55ce53c9b970 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83e85643054f8b37 server_cookie=f1b3328c10ae0a87 in_seq=0 out_seq=0 2024-03-28T01:06:15.464 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.461+0000 7f7592ff5640 1 -- 172.21.15.169:0/614569888 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ce53bcd3a0 con 0x55ce53c99550 2024-03-28T01:06:15.465 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.461+0000 7f7592ff5640 10 client.27085 handle_command_reply: tid=0 2024-03-28T01:06:15.465 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.462+0000 7f75b83cc740 1 client.27085 shutdown 2024-03-28T01:06:15.471 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.468+0000 7f75b83cc740 1 -- 172.21.15.169:0/614569888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c70140 msgr2=0x55ce53c72530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:15.471 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.468+0000 7f75b83cc740 1 --2- 172.21.15.169:0/614569888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c70140 0x55ce53c72530 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f75a0007590 tx=0x7f75a009bae0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.471 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.468+0000 7f75b83cc740 1 -- 172.21.15.169:0/614569888 shutdown_connections 2024-03-28T01:06:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.468+0000 7f75b83cc740 1 --2- 172.21.15.169:0/614569888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c70140 0x55ce53c72530 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.468+0000 7f75b83cc740 1 --2- 172.21.15.169:0/614569888 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ce53c99550 0x55ce53c9b970 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.468+0000 7f75b83cc740 1 --2- 172.21.15.169:0/614569888 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c66480 0x55ce53c6b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.468+0000 7f75b83cc740 1 --2- 172.21.15.169:0/614569888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c61310 0x55ce53c6abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.468+0000 7f75b83cc740 1 -- 172.21.15.169:0/614569888 >> 172.21.15.169:0/614569888 conn(0x55ce53c5be90 msgr2=0x55ce53c5a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.468+0000 7f7592ff5640 0 client.27085 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.468+0000 7f75b83cc740 1 -- 172.21.15.169:0/614569888 shutdown_connections 2024-03-28T01:06:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.468+0000 7f75b83cc740 1 -- 172.21.15.169:0/614569888 wait complete. 2024-03-28T01:06:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.468+0000 7f75b83cc740 20 client.27085 trim_cache size 0 max 16384 2024-03-28T01:06:15.520 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.515+0000 7f75b83cc740 1 Processor -- start 2024-03-28T01:06:15.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.521+0000 7f75b83cc740 1 -- start start 2024-03-28T01:06:15.524 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.521+0000 7f75b83cc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c61310 0x55ce53c5c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:15.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.521+0000 7f75b83cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c66480 0x55ce53c5f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:15.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.521+0000 7f75b83cc740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c70140 0x55ce53c5f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:15.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.521+0000 7f75b83cc740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c011fe0 con 0x55ce53c70140 2024-03-28T01:06:15.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.521+0000 7f75b83cc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c00fae0 con 0x55ce53c61310 2024-03-28T01:06:15.525 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.522+0000 7f75b83cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c00db80 con 0x55ce53c66480 2024-03-28T01:06:15.528 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.524+0000 7f75aeffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c61310 0x55ce53c5c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:15.529 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.524+0000 7f75ae7fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c66480 0x55ce53c5f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:15.529 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.524+0000 7f75ae7fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c66480 0x55ce53c5f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:46982/0 (socket says 172.21.15.169:46982) 2024-03-28T01:06:15.529 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.524+0000 7f75ae7fc640 1 -- 172.21.15.169:0/3698814769 learned_addr learned my addr 172.21.15.169:0/3698814769 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:15.529 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.524+0000 7f75af7fe640 1 --2- 172.21.15.169:0/3698814769 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c70140 0x55ce53c5f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.525+0000 7f75ae7fc640 1 -- 172.21.15.169:0/3698814769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c61310 msgr2=0x55ce53c5c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.528+0000 7f75ae7fc640 1 --2- 172.21.15.169:0/3698814769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c61310 0x55ce53c5c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.528+0000 7f75ae7fc640 1 -- 172.21.15.169:0/3698814769 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c70140 msgr2=0x55ce53c5f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.528+0000 7f75ae7fc640 1 --2- 172.21.15.169:0/3698814769 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c70140 0x55ce53c5f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.528+0000 7f75ae7fc640 1 -- 172.21.15.169:0/3698814769 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce53c9bfd0 con 0x55ce53c66480 2024-03-28T01:06:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.529+0000 7f75af7fe640 1 --2- 172.21.15.169:0/3698814769 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c70140 0x55ce53c5f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:06:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.529+0000 7f75aeffd640 1 --2- 172.21.15.169:0/3698814769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c61310 0x55ce53c5c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:06:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.538+0000 7f75ae7fc640 1 --2- 172.21.15.169:0/3698814769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c66480 0x55ce53c5f0f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f75a402c290 tx=0x7f75a402cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5951d4c33188ed3b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.542+0000 7f7594ff9640 1 -- 172.21.15.169:0/3698814769 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a4004040 con 0x55ce53c66480 2024-03-28T01:06:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.542+0000 7f7594ff9640 1 -- 172.21.15.169:0/3698814769 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75a404c070 con 0x55ce53c66480 2024-03-28T01:06:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.542+0000 7f7594ff9640 1 -- 172.21.15.169:0/3698814769 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a40035f0 con 0x55ce53c66480 2024-03-28T01:06:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.544+0000 7f75b83cc740 1 -- 172.21.15.169:0/3698814769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c66480 msgr2=0x55ce53c5f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.544+0000 7f75b83cc740 1 --2- 172.21.15.169:0/3698814769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c66480 0x55ce53c5f0f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f75a402c290 tx=0x7f75a402cd50 comp rx=0 tx=0).stop 2024-03-28T01:06:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.554+0000 7f75b83cc740 1 -- 172.21.15.169:0/3698814769 shutdown_connections 2024-03-28T01:06:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.554+0000 7f75b83cc740 1 --2- 172.21.15.169:0/3698814769 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c70140 0x55ce53c5f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.554+0000 7f75b83cc740 1 --2- 172.21.15.169:0/3698814769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c66480 0x55ce53c5f0f0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.554+0000 7f75b83cc740 1 --2- 172.21.15.169:0/3698814769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c61310 0x55ce53c5c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.554+0000 7f75b83cc740 1 -- 172.21.15.169:0/3698814769 >> 172.21.15.169:0/3698814769 conn(0x55ce53c5be90 msgr2=0x55ce53c65b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.554+0000 7f75b83cc740 1 -- 172.21.15.169:0/3698814769 shutdown_connections 2024-03-28T01:06:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.555+0000 7f75b83cc740 1 -- 172.21.15.169:0/3698814769 wait complete. 2024-03-28T01:06:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.555+0000 7f75b83cc740 1 -- start start 2024-03-28T01:06:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.556+0000 7f75b83cc740 1 Processor -- start 2024-03-28T01:06:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.556+0000 7f75b83cc740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c61310 0x55ce53cee2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.556+0000 7f75b83cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c66480 0x55ce53cf07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.556+0000 7f75b83cc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c70140 0x55ce53cf2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.556+0000 7f75b83cc740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c00e180 con 0x55ce53c61310 2024-03-28T01:06:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.556+0000 7f75b83cc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c00f480 con 0x55ce53c70140 2024-03-28T01:06:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.556+0000 7f75b83cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c00e7c0 con 0x55ce53c66480 2024-03-28T01:06:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.556+0000 7f75b83cc740 10 client.0 _ll_register_callbacks cb 0x7f75a009aed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:06:15.561 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.558+0000 7f75ae7fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c66480 0x55ce53cf07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:15.561 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.558+0000 7f75aeffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c61310 0x55ce53cee2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:15.562 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.558+0000 7f75aeffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c61310 0x55ce53cee2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:59604/0 (socket says 172.21.15.169:59604) 2024-03-28T01:06:15.562 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.558+0000 7f75aeffd640 1 -- 172.21.15.169:0/2534690516 learned_addr learned my addr 172.21.15.169:0/2534690516 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:15.562 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.558+0000 7f75af7fe640 1 --2- 172.21.15.169:0/2534690516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c70140 0x55ce53cf2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.560+0000 7f75af7fe640 1 -- 172.21.15.169:0/2534690516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c66480 msgr2=0x55ce53cf07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.560+0000 7f75af7fe640 1 --2- 172.21.15.169:0/2534690516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c66480 0x55ce53cf07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.560+0000 7f75af7fe640 1 -- 172.21.15.169:0/2534690516 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c61310 msgr2=0x55ce53cee2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.560+0000 7f75af7fe640 1 --2- 172.21.15.169:0/2534690516 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c61310 0x55ce53cee2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.560+0000 7f75af7fe640 1 -- 172.21.15.169:0/2534690516 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ce53c61fe0 con 0x55ce53c70140 2024-03-28T01:06:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.564+0000 7f75af7fe640 1 --2- 172.21.15.169:0/2534690516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c70140 0x55ce53cf2d40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f75a009b020 tx=0x7f75a009f3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66a7b372fd4f69c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.564+0000 7f7592ff5640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:06:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.566+0000 7f7592ff5640 1 -- 172.21.15.169:0/2534690516 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a00b3040 con 0x55ce53c70140 2024-03-28T01:06:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.566+0000 7f7592ff5640 1 -- 172.21.15.169:0/2534690516 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f75a0097070 con 0x55ce53c70140 2024-03-28T01:06:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.567+0000 7f75b83cc740 1 -- 172.21.15.169:0/2534690516 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f75a404c070 con 0x55ce53c70140 2024-03-28T01:06:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.568+0000 7f7592ff5640 1 -- 172.21.15.169:0/2534690516 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75a00972b0 con 0x55ce53c70140 2024-03-28T01:06:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.571+0000 7f75b83cc740 10 client.27146 fetch_fsmap learned FSMap version 25 2024-03-28T01:06:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.571+0000 7f75b83cc740 1 -- 172.21.15.169:0/2534690516 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ce53cf4430 con 0x55ce53c70140 2024-03-28T01:06:15.578 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.572+0000 7f7592ff5640 1 -- 172.21.15.169:0/2534690516 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75a404c070 con 0x55ce53c70140 2024-03-28T01:06:15.578 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.573+0000 7f7592ff5640 1 -- 172.21.15.169:0/2534690516 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f75a0004f10 con 0x55ce53c70140 2024-03-28T01:06:15.578 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.575+0000 7f75b83cc740 10 client.27146 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:06:15.578 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.575+0000 7f75b83cc740 10 client.27146 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:06:15.578 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.575+0000 7f75b83cc740 20 client.27146 populate_metadata read hostname 'smithi169' 2024-03-28T01:06:15.579 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.575+0000 7f75b83cc740 1 --2- 172.21.15.169:0/2534690516 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ce53cf4b20 0x55ce53cf6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:15.579 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.575+0000 7f75b83cc740 4 client.27146 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:06:15.579 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.575+0000 7f75b83cc740 1 -- 172.21.15.169:0/2534690516 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ce53b4b710 con 0x55ce53cf4b20 2024-03-28T01:06:15.580 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.577+0000 7f75aeffd640 1 --2- 172.21.15.169:0/2534690516 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ce53cf4b20 0x55ce53cf6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:15.591 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.588+0000 7f75aeffd640 1 --2- 172.21.15.169:0/2534690516 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ce53cf4b20 0x55ce53cf6f40 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f84534992e0bc70 server_cookie=b38359d39af1f6f6 in_seq=0 out_seq=0 2024-03-28T01:06:15.594 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:15.589+0000 7f7592ff5640 10 client.27146 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:15.968 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:15 smithi177 ceph-mon[36206]: pgmap v643: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 623 MiB/s wr, 874 op/s 2024-03-28T01:06:16.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:15 smithi169 ceph-mon[32274]: pgmap v643: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 623 MiB/s wr, 874 op/s 2024-03-28T01:06:16.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:15 smithi174 ceph-mon[36060]: pgmap v643: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 623 MiB/s wr, 874 op/s 2024-03-28T01:06:16.899 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:06:16.899 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:06:17.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.212+0000 7fd71eb64640 1 -- 172.21.15.169:0/3210713075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7180762f0 msgr2=0x7fd718078760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:17.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.212+0000 7fd71eb64640 1 --2- 172.21.15.169:0/3210713075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7180762f0 0x7fd718078760 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd708009860 tx=0x7fd70802bb90 comp rx=0 tx=0).stop 2024-03-28T01:06:17.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.213+0000 7fd71eb64640 1 -- 172.21.15.169:0/3210713075 shutdown_connections 2024-03-28T01:06:17.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.213+0000 7fd71eb64640 1 --2- 172.21.15.169:0/3210713075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd718078ca0 0x7fd71807b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:17.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.213+0000 7fd71eb64640 1 --2- 172.21.15.169:0/3210713075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7180762f0 0x7fd718078760 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:17.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.213+0000 7fd71eb64640 1 --2- 172.21.15.169:0/3210713075 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd718073410 0x7fd718073830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:17.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.213+0000 7fd71eb64640 1 -- 172.21.15.169:0/3210713075 >> 172.21.15.169:0/3210713075 conn(0x7fd71806e580 msgr2=0x7fd7180709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:17.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.215+0000 7fd71eb64640 1 -- 172.21.15.169:0/3210713075 shutdown_connections 2024-03-28T01:06:17.223 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.220+0000 7fd71eb64640 1 -- 172.21.15.169:0/3210713075 wait complete. 2024-03-28T01:06:17.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.221+0000 7fd71eb64640 1 Processor -- start 2024-03-28T01:06:17.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.221+0000 7fd71eb64640 1 -- start start 2024-03-28T01:06:17.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.221+0000 7fd71eb64640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd718073410 0x7fd71807fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:17.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.222+0000 7fd71eb64640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7180762f0 0x7fd7181b5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:17.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.222+0000 7fd71eb64640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd718078ca0 0x7fd7181b83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:17.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.222+0000 7fd71eb64640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71807e5b0 con 0x7fd7180762f0 2024-03-28T01:06:17.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.222+0000 7fd71eb64640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71807e430 con 0x7fd718078ca0 2024-03-28T01:06:17.225 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.222+0000 7fd71eb64640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71807e730 con 0x7fd718073410 2024-03-28T01:06:17.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.222+0000 7fd71e363640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd718078ca0 0x7fd7181b83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:17.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.222+0000 7fd71e363640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd718078ca0 0x7fd7181b83a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:38168/0 (socket says 172.21.15.169:38168) 2024-03-28T01:06:17.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.222+0000 7fd71e363640 1 -- 172.21.15.169:0/776337289 learned_addr learned my addr 172.21.15.169:0/776337289 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:17.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.222+0000 7fd71e363640 1 -- 172.21.15.169:0/776337289 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd718073410 msgr2=0x7fd71807fae0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:06:17.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.222+0000 7fd71e363640 1 --2- 172.21.15.169:0/776337289 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd718073410 0x7fd71807fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:17.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.222+0000 7fd71e363640 1 -- 172.21.15.169:0/776337289 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7180762f0 msgr2=0x7fd7181b5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:17.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.222+0000 7fd71e363640 1 --2- 172.21.15.169:0/776337289 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7180762f0 0x7fd7181b5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:17.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.222+0000 7fd71e363640 1 -- 172.21.15.169:0/776337289 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7181afc00 con 0x7fd718078ca0 2024-03-28T01:06:17.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.223+0000 7fd71e363640 1 --2- 172.21.15.169:0/776337289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd718078ca0 0x7fd7181b83a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd71000cad0 tx=0x7fd71000cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:17.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.223+0000 7fd70effd640 1 -- 172.21.15.169:0/776337289 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71000ae20 con 0x7fd718078ca0 2024-03-28T01:06:17.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.223+0000 7fd71eb64640 1 -- 172.21.15.169:0/776337289 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7181afef0 con 0x7fd718078ca0 2024-03-28T01:06:17.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.224+0000 7fd71eb64640 1 -- 172.21.15.169:0/776337289 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7181b0440 con 0x7fd718078ca0 2024-03-28T01:06:17.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.226+0000 7fd70effd640 1 -- 172.21.15.169:0/776337289 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd710005020 con 0x7fd718078ca0 2024-03-28T01:06:17.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.226+0000 7fd70effd640 1 -- 172.21.15.169:0/776337289 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71000f740 con 0x7fd718078ca0 2024-03-28T01:06:17.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.230+0000 7fd7229a8740 1 Processor -- start 2024-03-28T01:06:17.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.230+0000 7fd70effd640 1 -- 172.21.15.169:0/776337289 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd71000f9c0 con 0x7fd718078ca0 2024-03-28T01:06:17.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.231+0000 7fd7229a8740 1 -- start start 2024-03-28T01:06:17.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.231+0000 7fd7229a8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56254819d790 0x56254819dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:17.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.231+0000 7fd7229a8740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56254819e180 0x5625481a2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:17.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.231+0000 7fd7229a8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625481a32f0 0x5625481a5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:17.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.231+0000 7fd7229a8740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625480fa3d0 con 0x56254819e180 2024-03-28T01:06:17.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.231+0000 7fd7229a8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625480a2ad0 con 0x5625481a32f0 2024-03-28T01:06:17.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.231+0000 7fd7229a8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625480f6470 con 0x56254819d790 2024-03-28T01:06:17.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.231+0000 7fd71d361640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56254819d790 0x56254819dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:17.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.232+0000 7fd71d361640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56254819d790 0x56254819dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:54388/0 (socket says 172.21.15.169:54388) 2024-03-28T01:06:17.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.232+0000 7fd71d361640 1 -- 172.21.15.169:0/377727013 learned_addr learned my addr 172.21.15.169:0/377727013 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:17.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.232+0000 7fd71d361640 1 --2- 172.21.15.169:0/377727013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625481a32f0 0x5625481a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:17.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.232+0000 7fd71d361640 1 -- 172.21.15.169:0/377727013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625481a32f0 msgr2=0x5625481a5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:17.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.232+0000 7fd71d361640 1 --2- 172.21.15.169:0/377727013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625481a32f0 0x5625481a5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:17.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.232+0000 7fd71d361640 1 -- 172.21.15.169:0/377727013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56254819e180 msgr2=0x5625481a2c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:06:17.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.232+0000 7fd71d361640 1 --2- 172.21.15.169:0/377727013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56254819e180 0x5625481a2c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:17.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.232+0000 7fd71d361640 1 -- 172.21.15.169:0/377727013 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562547ed87d0 con 0x56254819d790 2024-03-28T01:06:17.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.232+0000 7fd71d361640 1 --2- 172.21.15.169:0/377727013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56254819d790 0x56254819dbb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd708036000 tx=0x7fd708000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e8e6a28757f3344 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:17.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.233+0000 7fd70effd640 1 --2- 172.21.15.169:0/776337289 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd7040a29b0 0x7fd7040a4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:17.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.233+0000 7fd70effd640 1 -- 172.21.15.169:0/776337289 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fd7100cee30 con 0x7fd718078ca0 2024-03-28T01:06:17.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.239+0000 7fd703fff640 1 -- 172.21.15.169:0/377727013 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd708039040 con 0x56254819d790 2024-03-28T01:06:17.242 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.239+0000 7fd703fff640 1 -- 172.21.15.169:0/377727013 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd708034640 con 0x56254819d790 2024-03-28T01:06:17.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.240+0000 7fd7229a8740 1 -- 172.21.15.169:0/377727013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56254819d790 msgr2=0x56254819dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:17.243 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.240+0000 7fd7229a8740 1 --2- 172.21.15.169:0/377727013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56254819d790 0x56254819dbb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd708036000 tx=0x7fd708000bc0 comp rx=0 tx=0).stop 2024-03-28T01:06:17.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.243+0000 7fd7229a8740 1 -- 172.21.15.169:0/377727013 shutdown_connections 2024-03-28T01:06:17.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.243+0000 7fd7229a8740 1 --2- 172.21.15.169:0/377727013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625481a32f0 0x5625481a5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:17.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.243+0000 7fd7229a8740 1 --2- 172.21.15.169:0/377727013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56254819e180 0x5625481a2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:17.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.243+0000 7fd7229a8740 1 --2- 172.21.15.169:0/377727013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56254819d790 0x56254819dbb0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:17.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.243+0000 7fd7229a8740 1 -- 172.21.15.169:0/377727013 >> 172.21.15.169:0/377727013 conn(0x5625481904f0 msgr2=0x5625481908f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:17.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.243+0000 7fd71db62640 1 --2- 172.21.15.169:0/776337289 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd7040a29b0 0x7fd7040a4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:17.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.244+0000 7fd7229a8740 1 -- 172.21.15.169:0/377727013 shutdown_connections 2024-03-28T01:06:17.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.245+0000 7fd7229a8740 1 -- 172.21.15.169:0/377727013 wait complete. 2024-03-28T01:06:17.249 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.245+0000 7fd7229a8740 1 -- start start 2024-03-28T01:06:17.249 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.245+0000 7fd7229a8740 1 Processor -- start 2024-03-28T01:06:17.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.246+0000 7fd7229a8740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56254819d790 0x562548196a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:17.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.246+0000 7fd7229a8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56254819e180 0x562548196f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:17.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.246+0000 7fd7229a8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625481a32f0 0x56254819bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:17.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.246+0000 7fd7229a8740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x562548195010 con 0x56254819d790 2024-03-28T01:06:17.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.246+0000 7fd7229a8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625480b6060 con 0x56254819e180 2024-03-28T01:06:17.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.246+0000 7fd7229a8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x562548195190 con 0x5625481a32f0 2024-03-28T01:06:17.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.246+0000 7fd7229a8740 10 client.0 _ll_register_callbacks cb 0x5625480986a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:06:17.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.250+0000 7fd71e363640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625481a32f0 0x56254819bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:17.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.250+0000 7fd71d361640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56254819e180 0x562548196f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:17.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.251+0000 7fd71e363640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625481a32f0 0x56254819bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:54404/0 (socket says 172.21.15.169:54404) 2024-03-28T01:06:17.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.251+0000 7fd71e363640 1 -- 172.21.15.169:0/1821972084 learned_addr learned my addr 172.21.15.169:0/1821972084 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:17.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.251+0000 7fd71db62640 1 --2- 172.21.15.169:0/1821972084 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56254819d790 0x562548196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:17.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.252+0000 7fd71db62640 1 --2- 172.21.15.169:0/776337289 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd7040a29b0 0x7fd7040a4e70 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x562548191460 tx=0x7fd714005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:17.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.253+0000 7fd71db62640 1 -- 172.21.15.169:0/1821972084 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625481a32f0 msgr2=0x56254819bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:17.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.253+0000 7fd71db62640 1 --2- 172.21.15.169:0/1821972084 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625481a32f0 0x56254819bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:17.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.253+0000 7fd71db62640 1 -- 172.21.15.169:0/1821972084 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56254819e180 msgr2=0x562548196f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:17.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.253+0000 7fd71db62640 1 --2- 172.21.15.169:0/1821972084 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56254819e180 0x562548196f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:17.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.253+0000 7fd71db62640 1 -- 172.21.15.169:0/1821972084 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562547e0e3f0 con 0x56254819d790 2024-03-28T01:06:17.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.254+0000 7fd71db62640 1 --2- 172.21.15.169:0/1821972084 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56254819d790 0x562548196a50 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd71400b2d0 tx=0x7fd714004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c6d47b7034ab4e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:17.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.254+0000 7fd7017fa640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:06:17.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.255+0000 7fd7017fa640 1 -- 172.21.15.169:0/1821972084 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd714020070 con 0x56254819d790 2024-03-28T01:06:17.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.255+0000 7fd7017fa640 1 -- 172.21.15.169:0/1821972084 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fd714005220 con 0x56254819d790 2024-03-28T01:06:17.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.255+0000 7fd7229a8740 1 -- 172.21.15.169:0/1821972084 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd708034640 con 0x56254819d790 2024-03-28T01:06:17.260 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.257+0000 7fd7017fa640 1 -- 172.21.15.169:0/1821972084 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd714029050 con 0x56254819d790 2024-03-28T01:06:17.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.260+0000 7fd7229a8740 10 client.17295 fetch_fsmap learned FSMap version 25 2024-03-28T01:06:17.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.260+0000 7fd7229a8740 1 -- 172.21.15.169:0/1821972084 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x562547dce0f0 con 0x56254819d790 2024-03-28T01:06:17.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.264+0000 7fd7017fa640 1 -- 172.21.15.169:0/1821972084 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd708034640 con 0x56254819d790 2024-03-28T01:06:17.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.265+0000 7fd7017fa640 1 -- 172.21.15.169:0/1821972084 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd71401c830 con 0x56254819d790 2024-03-28T01:06:17.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.266+0000 7fd7229a8740 10 client.17295 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:06:17.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.266+0000 7fd7229a8740 10 client.17295 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:06:17.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.266+0000 7fd7229a8740 20 client.17295 populate_metadata read hostname 'smithi169' 2024-03-28T01:06:17.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.266+0000 7fd7229a8740 1 --2- 172.21.15.169:0/1821972084 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5625481bbee0 0x5625481be340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:17.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.266+0000 7fd7229a8740 4 client.17295 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:06:17.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.266+0000 7fd7229a8740 1 -- 172.21.15.169:0/1821972084 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562548107910 con 0x5625481bbee0 2024-03-28T01:06:17.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.275+0000 7fd71d361640 1 --2- 172.21.15.169:0/1821972084 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5625481bbee0 0x5625481be340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:17.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.291+0000 7fd71d361640 1 --2- 172.21.15.169:0/1821972084 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5625481bbee0 0x5625481be340 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da6a4c8f07eabedd server_cookie=d6fc5bec480004e0 in_seq=0 out_seq=0 2024-03-28T01:06:17.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:17.292+0000 7fd7017fa640 10 client.17295 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:18.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:17 smithi169 ceph-mon[32274]: pgmap v644: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 682 MiB/s wr, 755 op/s 2024-03-28T01:06:18.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:17 smithi174 ceph-mon[36060]: pgmap v644: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 682 MiB/s wr, 755 op/s 2024-03-28T01:06:18.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:17 smithi177 ceph-mon[36206]: pgmap v644: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 682 MiB/s wr, 755 op/s 2024-03-28T01:06:20.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:19 smithi169 ceph-mon[32274]: pgmap v645: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 649 MiB/s wr, 482 op/s 2024-03-28T01:06:20.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:19 smithi174 ceph-mon[36060]: pgmap v645: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 649 MiB/s wr, 482 op/s 2024-03-28T01:06:20.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:19 smithi177 ceph-mon[36206]: pgmap v645: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 649 MiB/s wr, 482 op/s 2024-03-28T01:06:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.779+0000 7f7592ff5640 1 -- 172.21.15.169:0/2534690516 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ce53b4b710 con 0x55ce53cf4b20 2024-03-28T01:06:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.779+0000 7f7592ff5640 10 client.27146 handle_command_reply: tid=0 2024-03-28T01:06:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.779+0000 7f75b83cc740 1 client.27146 shutdown 2024-03-28T01:06:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.779+0000 7f75b83cc740 1 -- 172.21.15.169:0/2534690516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c70140 msgr2=0x55ce53cf2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.779+0000 7f75b83cc740 1 --2- 172.21.15.169:0/2534690516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c70140 0x55ce53cf2d40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f75a009b020 tx=0x7f75a009f3a0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.779+0000 7f75b83cc740 1 -- 172.21.15.169:0/2534690516 shutdown_connections 2024-03-28T01:06:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.779+0000 7f75b83cc740 1 --2- 172.21.15.169:0/2534690516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce53c70140 0x55ce53cf2d40 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.779+0000 7f75b83cc740 1 --2- 172.21.15.169:0/2534690516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce53c66480 0x55ce53cf07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.779+0000 7f75b83cc740 1 --2- 172.21.15.169:0/2534690516 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ce53cf4b20 0x55ce53cf6f40 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.779+0000 7f75b83cc740 1 --2- 172.21.15.169:0/2534690516 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ce53c61310 0x55ce53cee2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.779+0000 7f75b83cc740 1 -- 172.21.15.169:0/2534690516 >> 172.21.15.169:0/2534690516 conn(0x55ce53c5be90 msgr2=0x55ce53c5f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.779+0000 7f7592ff5640 0 client.27146 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.780+0000 7f75b83cc740 1 -- 172.21.15.169:0/2534690516 shutdown_connections 2024-03-28T01:06:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.780+0000 7f75b83cc740 1 -- 172.21.15.169:0/2534690516 wait complete. 2024-03-28T01:06:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.780+0000 7f75b83cc740 20 client.27146 trim_cache size 0 max 16384 2024-03-28T01:06:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:06:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:06:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:06:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:06:20.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.780+0000 7fd7017fa640 1 -- 172.21.15.169:0/1821972084 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562548107910 con 0x5625481bbee0 2024-03-28T01:06:20.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.780+0000 7fd7017fa640 10 client.17295 handle_command_reply: tid=0 2024-03-28T01:06:20.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.781+0000 7fd7229a8740 1 client.17295 shutdown 2024-03-28T01:06:20.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.781+0000 7fd7229a8740 1 -- 172.21.15.169:0/1821972084 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56254819d790 msgr2=0x562548196a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:20.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.781+0000 7fd7229a8740 1 --2- 172.21.15.169:0/1821972084 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56254819d790 0x562548196a50 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd71400b2d0 tx=0x7fd714004730 comp rx=0 tx=0).stop 2024-03-28T01:06:20.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.782+0000 7f75b570c640 1 -- 172.21.15.169:0/3092354706 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f75980a2ae0 msgr2=0x7f75980a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:20.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.782+0000 7f75b570c640 1 --2- 172.21.15.169:0/3092354706 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f75980a2ae0 0x7f75980a4fa0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x55ce53c5ce00 tx=0x7f75a800a480 comp rx=0 tx=0).stop 2024-03-28T01:06:20.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.782+0000 7f75b570c640 1 -- 172.21.15.169:0/3092354706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75b0078df0 msgr2=0x7f75b0083570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:20.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.782+0000 7f75b570c640 1 --2- 172.21.15.169:0/3092354706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75b0078df0 0x7f75b0083570 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f75a000c970 tx=0x7f75a000ce30 comp rx=0 tx=0).stop 2024-03-28T01:06:20.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.782+0000 7f75b570c640 1 -- 172.21.15.169:0/3092354706 shutdown_connections 2024-03-28T01:06:20.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.782+0000 7f75b570c640 1 --2- 172.21.15.169:0/3092354706 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f75980a2ae0 0x7f75980a4fa0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.782+0000 7f75b570c640 1 --2- 172.21.15.169:0/3092354706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75b0078df0 0x7f75b0083570 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.782+0000 7f75b570c640 1 --2- 172.21.15.169:0/3092354706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75b0076440 0x7f75b0083030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.782+0000 7f75b570c640 1 --2- 172.21.15.169:0/3092354706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75b0073560 0x7f75b0082af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.790 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.782+0000 7f75b570c640 1 -- 172.21.15.169:0/3092354706 >> 172.21.15.169:0/3092354706 conn(0x7f75b006e6e0 msgr2=0x7f75b007afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:20.790 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.781+0000 7fd7229a8740 1 -- 172.21.15.169:0/1821972084 shutdown_connections 2024-03-28T01:06:20.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.781+0000 7fd7229a8740 1 --2- 172.21.15.169:0/1821972084 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625481a32f0 0x56254819bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.781+0000 7fd7229a8740 1 --2- 172.21.15.169:0/1821972084 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56254819e180 0x562548196f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.781+0000 7fd7229a8740 1 --2- 172.21.15.169:0/1821972084 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5625481bbee0 0x5625481be340 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.781+0000 7fd7229a8740 1 --2- 172.21.15.169:0/1821972084 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56254819d790 0x562548196a50 secure :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd71400b2d0 tx=0x7fd714004730 comp rx=0 tx=0).stop 2024-03-28T01:06:20.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.781+0000 7fd7229a8740 1 -- 172.21.15.169:0/1821972084 >> 172.21.15.169:0/1821972084 conn(0x5625481904f0 msgr2=0x56254818f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:20.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.781+0000 7fd7017fa640 0 client.17295 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:20.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.781+0000 7fd7229a8740 1 -- 172.21.15.169:0/1821972084 shutdown_connections 2024-03-28T01:06:20.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.781+0000 7fd7229a8740 1 -- 172.21.15.169:0/1821972084 wait complete. 2024-03-28T01:06:20.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.781+0000 7fd7229a8740 20 client.17295 trim_cache size 0 max 16384 2024-03-28T01:06:20.802 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.799+0000 7f75b570c640 1 -- 172.21.15.169:0/3092354706 shutdown_connections 2024-03-28T01:06:20.802 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:20.800+0000 7f75b570c640 1 -- 172.21.15.169:0/3092354706 wait complete. 2024-03-28T01:06:20.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.829+0000 7fd7229a8740 1 Processor -- start 2024-03-28T01:06:20.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.835+0000 7fd7229a8740 1 -- start start 2024-03-28T01:06:20.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.835+0000 7fd7229a8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56254819e180 0x5625481bdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:20.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.835+0000 7fd7229a8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625481a32f0 0x562548196d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:20.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.835+0000 7fd7229a8740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56254818ef20 0x56254818f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:20.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.835+0000 7fd7229a8740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc0120d0 con 0x56254818ef20 2024-03-28T01:06:20.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.835+0000 7fd7229a8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc00fbd0 con 0x56254819e180 2024-03-28T01:06:20.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.835+0000 7fd7229a8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc00dc70 con 0x5625481a32f0 2024-03-28T01:06:20.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.839+0000 7fd71d361640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625481a32f0 0x562548196d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:20.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.839+0000 7fd71d361640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625481a32f0 0x562548196d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:54414/0 (socket says 172.21.15.169:54414) 2024-03-28T01:06:20.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.839+0000 7fd71d361640 1 -- 172.21.15.169:0/4257819344 learned_addr learned my addr 172.21.15.169:0/4257819344 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:20.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.839+0000 7fd71db62640 1 --2- 172.21.15.169:0/4257819344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56254819e180 0x5625481bdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:20.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.839+0000 7fd71e363640 1 --2- 172.21.15.169:0/4257819344 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56254818ef20 0x56254818f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:20.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.841+0000 7fd71d361640 1 -- 172.21.15.169:0/4257819344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56254819e180 msgr2=0x5625481bdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:20.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.841+0000 7fd71d361640 1 --2- 172.21.15.169:0/4257819344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56254819e180 0x5625481bdef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.841+0000 7fd71d361640 1 -- 172.21.15.169:0/4257819344 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56254818ef20 msgr2=0x56254818f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:20.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.841+0000 7fd71d361640 1 --2- 172.21.15.169:0/4257819344 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56254818ef20 0x56254818f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.841+0000 7fd71d361640 1 -- 172.21.15.169:0/4257819344 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56254818bf60 con 0x5625481a32f0 2024-03-28T01:06:20.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.841+0000 7fd71d361640 1 --2- 172.21.15.169:0/4257819344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625481a32f0 0x562548196d70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd708034ab0 tx=0x7fd708005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=538cceabb4771e05 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:20.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.842+0000 7fd7037fe640 1 -- 172.21.15.169:0/4257819344 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd708039040 con 0x5625481a32f0 2024-03-28T01:06:20.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.842+0000 7fd7037fe640 1 -- 172.21.15.169:0/4257819344 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd708005530 con 0x5625481a32f0 2024-03-28T01:06:20.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.842+0000 7fd7037fe640 1 -- 172.21.15.169:0/4257819344 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd708003e10 con 0x5625481a32f0 2024-03-28T01:06:20.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.848+0000 7fd7229a8740 1 -- 172.21.15.169:0/4257819344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625481a32f0 msgr2=0x562548196d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:20.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.848+0000 7fd7229a8740 1 --2- 172.21.15.169:0/4257819344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625481a32f0 0x562548196d70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd708034ab0 tx=0x7fd708005bd0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.848+0000 7fd7229a8740 1 -- 172.21.15.169:0/4257819344 shutdown_connections 2024-03-28T01:06:20.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.848+0000 7fd7229a8740 1 --2- 172.21.15.169:0/4257819344 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56254818ef20 0x56254818f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.848+0000 7fd7229a8740 1 --2- 172.21.15.169:0/4257819344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625481a32f0 0x562548196d70 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.848+0000 7fd7229a8740 1 --2- 172.21.15.169:0/4257819344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56254819e180 0x5625481bdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.848+0000 7fd7229a8740 1 -- 172.21.15.169:0/4257819344 >> 172.21.15.169:0/4257819344 conn(0x5625481904f0 msgr2=0x562548196660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:20.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.853+0000 7fd7229a8740 1 -- 172.21.15.169:0/4257819344 shutdown_connections 2024-03-28T01:06:20.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.853+0000 7fd7229a8740 1 -- 172.21.15.169:0/4257819344 wait complete. 2024-03-28T01:06:20.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.853+0000 7fd7229a8740 1 -- start start 2024-03-28T01:06:20.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.853+0000 7fd7229a8740 1 Processor -- start 2024-03-28T01:06:20.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.854+0000 7fd7229a8740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56254818ef20 0x5625481cd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:20.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.855+0000 7fd7229a8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56254819e180 0x5625481cd860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:20.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.855+0000 7fd7229a8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625481a32f0 0x562548216c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:20.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.855+0000 7fd7229a8740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc00e270 con 0x56254818ef20 2024-03-28T01:06:20.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.855+0000 7fd71d361640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56254819e180 0x5625481cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:20.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.855+0000 7fd7229a8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc00f570 con 0x5625481a32f0 2024-03-28T01:06:20.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.855+0000 7fd7229a8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc00e8b0 con 0x56254819e180 2024-03-28T01:06:20.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.855+0000 7fd71d361640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56254819e180 0x5625481cd860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:54428/0 (socket says 172.21.15.169:54428) 2024-03-28T01:06:20.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.855+0000 7fd71d361640 1 -- 172.21.15.169:0/4208437608 learned_addr learned my addr 172.21.15.169:0/4208437608 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:20.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.855+0000 7fd7229a8740 10 client.0 _ll_register_callbacks cb 0x7fd714015040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:06:20.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.855+0000 7fd71e363640 1 --2- 172.21.15.169:0/4208437608 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625481a32f0 0x562548216c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:20.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.855+0000 7fd71d361640 1 -- 172.21.15.169:0/4208437608 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625481a32f0 msgr2=0x562548216c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:20.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.855+0000 7fd71d361640 1 --2- 172.21.15.169:0/4208437608 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625481a32f0 0x562548216c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.855+0000 7fd71d361640 1 -- 172.21.15.169:0/4208437608 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56254818ef20 msgr2=0x5625481cd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:20.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.855+0000 7fd71d361640 1 --2- 172.21.15.169:0/4208437608 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56254818ef20 0x5625481cd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.855+0000 7fd71d361640 1 -- 172.21.15.169:0/4208437608 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5625482171c0 con 0x56254819e180 2024-03-28T01:06:20.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.859+0000 7fd71d361640 1 --2- 172.21.15.169:0/4208437608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56254819e180 0x5625481cd860 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd7080039e0 tx=0x7fd7080097f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c24d9b09e0df5b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:20.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.862+0000 7fd7017fa640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:06:20.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.862+0000 7fd7017fa640 1 -- 172.21.15.169:0/4208437608 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70805e060 con 0x56254819e180 2024-03-28T01:06:20.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.862+0000 7fd7017fa640 1 -- 172.21.15.169:0/4208437608 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fd708050170 con 0x56254819e180 2024-03-28T01:06:20.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.863+0000 7fd7229a8740 1 -- 172.21.15.169:0/4208437608 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd708005530 con 0x56254819e180 2024-03-28T01:06:20.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.865+0000 7fd7017fa640 1 -- 172.21.15.169:0/4208437608 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd708039070 con 0x56254819e180 2024-03-28T01:06:20.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.865+0000 7fd7229a8740 10 client.27106 fetch_fsmap learned FSMap version 25 2024-03-28T01:06:20.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.865+0000 7fd7229a8740 1 -- 172.21.15.169:0/4208437608 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5625482183b0 con 0x56254819e180 2024-03-28T01:06:20.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.868+0000 7fd7017fa640 1 -- 172.21.15.169:0/4208437608 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd708005530 con 0x56254819e180 2024-03-28T01:06:20.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.868+0000 7fd7017fa640 1 -- 172.21.15.169:0/4208437608 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fd70803f050 con 0x56254819e180 2024-03-28T01:06:20.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.869+0000 7fd7229a8740 10 client.27106 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:06:20.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.869+0000 7fd7229a8740 10 client.27106 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:06:20.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.869+0000 7fd7229a8740 20 client.27106 populate_metadata read hostname 'smithi169' 2024-03-28T01:06:20.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.869+0000 7fd7229a8740 1 --2- 172.21.15.169:0/4208437608 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562548218770 0x56254821ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:20.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.869+0000 7fd7229a8740 4 client.27106 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:06:20.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.869+0000 7fd7229a8740 1 -- 172.21.15.169:0/4208437608 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562548101c60 con 0x562548218770 2024-03-28T01:06:20.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.873+0000 7fd71db62640 1 --2- 172.21.15.169:0/4208437608 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562548218770 0x56254821ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:20.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.884+0000 7fd71db62640 1 --2- 172.21.15.169:0/4208437608 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562548218770 0x56254821ab60 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13d0b90c89073917 server_cookie=fb648d399f5954c5 in_seq=0 out_seq=0 2024-03-28T01:06:20.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.884+0000 7fd7017fa640 10 client.27106 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:20.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.887+0000 7fd7017fa640 1 -- 172.21.15.169:0/4208437608 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x562548101c60 con 0x562548218770 2024-03-28T01:06:20.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.887+0000 7fd7017fa640 10 client.27106 handle_command_reply: tid=0 2024-03-28T01:06:20.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.887+0000 7fd7229a8740 1 client.27106 shutdown 2024-03-28T01:06:20.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.888+0000 7fd7229a8740 1 -- 172.21.15.169:0/4208437608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56254819e180 msgr2=0x5625481cd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:20.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.888+0000 7fd7229a8740 1 --2- 172.21.15.169:0/4208437608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56254819e180 0x5625481cd860 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd7080039e0 tx=0x7fd7080097f0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.889+0000 7fd7229a8740 1 -- 172.21.15.169:0/4208437608 shutdown_connections 2024-03-28T01:06:20.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.889+0000 7fd7229a8740 1 --2- 172.21.15.169:0/4208437608 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625481a32f0 0x562548216c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.889+0000 7fd7229a8740 1 --2- 172.21.15.169:0/4208437608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56254819e180 0x5625481cd860 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.889+0000 7fd7229a8740 1 --2- 172.21.15.169:0/4208437608 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562548218770 0x56254821ab60 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.889+0000 7fd7229a8740 1 --2- 172.21.15.169:0/4208437608 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56254818ef20 0x5625481cd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.889+0000 7fd7229a8740 1 -- 172.21.15.169:0/4208437608 >> 172.21.15.169:0/4208437608 conn(0x5625481904f0 msgr2=0x562548193750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:20.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.891+0000 7fd7017fa640 0 client.27106 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:20.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.891+0000 7fd7229a8740 1 -- 172.21.15.169:0/4208437608 shutdown_connections 2024-03-28T01:06:20.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.891+0000 7fd7229a8740 1 -- 172.21.15.169:0/4208437608 wait complete. 2024-03-28T01:06:20.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.906+0000 7fd7229a8740 20 client.27106 trim_cache size 0 max 16384 2024-03-28T01:06:20.915 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:06:20.915 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "no active scrubs running", 2024-03-28T01:06:20.915 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": {} 2024-03-28T01:06:20.915 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:06:20.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.912+0000 7fd71eb64640 1 -- 172.21.15.169:0/776337289 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd7040a29b0 msgr2=0x7fd7040a4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:20.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.912+0000 7fd71eb64640 1 --2- 172.21.15.169:0/776337289 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd7040a29b0 0x7fd7040a4e70 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x562548191460 tx=0x7fd714005850 comp rx=0 tx=0).stop 2024-03-28T01:06:20.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.912+0000 7fd71eb64640 1 -- 172.21.15.169:0/776337289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd718078ca0 msgr2=0x7fd7181b83a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:20.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.912+0000 7fd71eb64640 1 --2- 172.21.15.169:0/776337289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd718078ca0 0x7fd7181b83a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd71000cad0 tx=0x7fd71000cf90 comp rx=0 tx=0).stop 2024-03-28T01:06:20.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.915+0000 7fd71eb64640 1 -- 172.21.15.169:0/776337289 shutdown_connections 2024-03-28T01:06:20.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.915+0000 7fd71eb64640 1 --2- 172.21.15.169:0/776337289 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd7040a29b0 0x7fd7040a4e70 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.915+0000 7fd71eb64640 1 --2- 172.21.15.169:0/776337289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd718078ca0 0x7fd7181b83a0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.915+0000 7fd71eb64640 1 --2- 172.21.15.169:0/776337289 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7180762f0 0x7fd7181b5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.915+0000 7fd71eb64640 1 --2- 172.21.15.169:0/776337289 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd718073410 0x7fd71807fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:20.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.915+0000 7fd71eb64640 1 -- 172.21.15.169:0/776337289 >> 172.21.15.169:0/776337289 conn(0x7fd71806e580 msgr2=0x7fd718079120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:20.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.920+0000 7fd71eb64640 1 -- 172.21.15.169:0/776337289 shutdown_connections 2024-03-28T01:06:20.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:20.920+0000 7fd71eb64640 1 -- 172.21.15.169:0/776337289 wait complete. 2024-03-28T01:06:20.956 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T01:06:20.956 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T01:06:21.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.296+0000 7fa039681640 1 -- 172.21.15.169:0/2486748769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa03410b010 msgr2=0x7fa03410d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:21.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.296+0000 7fa039681640 1 --2- 172.21.15.169:0/2486748769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa03410b010 0x7fa03410d4a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa028009900 tx=0x7fa02802bc80 comp rx=0 tx=0).stop 2024-03-28T01:06:21.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.303+0000 7fa039681640 1 -- 172.21.15.169:0/2486748769 shutdown_connections 2024-03-28T01:06:21.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.303+0000 7fa039681640 1 --2- 172.21.15.169:0/2486748769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa03410d9e0 0x7fa03410de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:21.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.303+0000 7fa039681640 1 --2- 172.21.15.169:0/2486748769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa03410b010 0x7fa03410d4a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:21.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.303+0000 7fa039681640 1 --2- 172.21.15.169:0/2486748769 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0341086a0 0x7fa03410aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:21.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.303+0000 7fa039681640 1 -- 172.21.15.169:0/2486748769 >> 172.21.15.169:0/2486748769 conn(0x7fa03406e510 msgr2=0x7fa034070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:21.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.313+0000 7fa039681640 1 -- 172.21.15.169:0/2486748769 shutdown_connections 2024-03-28T01:06:21.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.317+0000 7fa039681640 1 -- 172.21.15.169:0/2486748769 wait complete. 2024-03-28T01:06:21.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.317+0000 7fa039681640 1 Processor -- start 2024-03-28T01:06:21.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.319+0000 7fa039681640 1 -- start start 2024-03-28T01:06:21.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.320+0000 7fa039681640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0341086a0 0x7fa0341a6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:21.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.320+0000 7fa039681640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa03410b010 0x7fa0341a6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:21.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.320+0000 7fa039681640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa03410d9e0 0x7fa0341adcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:21.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.320+0000 7fa039681640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03411c2c0 con 0x7fa03410d9e0 2024-03-28T01:06:21.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.320+0000 7fa032ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0341086a0 0x7fa0341a6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:21.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.320+0000 7fa032ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0341086a0 0x7fa0341a6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:54436/0 (socket says 172.21.15.169:54436) 2024-03-28T01:06:21.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.320+0000 7fa032ffd640 1 -- 172.21.15.169:0/1184404439 learned_addr learned my addr 172.21.15.169:0/1184404439 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:21.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.320+0000 7fa039681640 1 -- 172.21.15.169:0/1184404439 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03411c140 con 0x7fa03410b010 2024-03-28T01:06:21.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.320+0000 7fa039681640 1 -- 172.21.15.169:0/1184404439 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03411c440 con 0x7fa0341086a0 2024-03-28T01:06:21.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.321+0000 7fa0337fe640 1 --2- 172.21.15.169:0/1184404439 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa03410d9e0 0x7fa0341adcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:21.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.322+0000 7fa0337fe640 1 -- 172.21.15.169:0/1184404439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0341086a0 msgr2=0x7fa0341a6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:21.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.322+0000 7fa0337fe640 1 --2- 172.21.15.169:0/1184404439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0341086a0 0x7fa0341a6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:21.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.322+0000 7fa0337fe640 1 -- 172.21.15.169:0/1184404439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa03410b010 msgr2=0x7fa0341a6cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:06:21.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.322+0000 7fa0337fe640 1 --2- 172.21.15.169:0/1184404439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa03410b010 0x7fa0341a6cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:21.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.322+0000 7fa0337fe640 1 -- 172.21.15.169:0/1184404439 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0341ae360 con 0x7fa03410d9e0 2024-03-28T01:06:21.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.324+0000 7fa0337fe640 1 --2- 172.21.15.169:0/1184404439 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa03410d9e0 0x7fa0341adcf0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fa02c00bdd0 tx=0x7fa02c009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:21.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.325+0000 7fa01bfff640 1 -- 172.21.15.169:0/1184404439 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c019070 con 0x7fa03410d9e0 2024-03-28T01:06:21.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.325+0000 7fa01bfff640 1 -- 172.21.15.169:0/1184404439 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa02c00cdf0 con 0x7fa03410d9e0 2024-03-28T01:06:21.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.326+0000 7fa01bfff640 1 -- 172.21.15.169:0/1184404439 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c012990 con 0x7fa03410d9e0 2024-03-28T01:06:21.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.326+0000 7fa039681640 1 -- 172.21.15.169:0/1184404439 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0341ae650 con 0x7fa03410d9e0 2024-03-28T01:06:21.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.326+0000 7fa039681640 1 -- 172.21.15.169:0/1184404439 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0341aeb50 con 0x7fa03410d9e0 2024-03-28T01:06:21.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.330+0000 7fa039681640 1 -- 172.21.15.169:0/1184404439 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0200052c0 con 0x7fa03410d9e0 2024-03-28T01:06:21.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.332+0000 7fa01bfff640 1 -- 172.21.15.169:0/1184404439 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa02c01c070 con 0x7fa03410d9e0 2024-03-28T01:06:21.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.335+0000 7fa01bfff640 1 --2- 172.21.15.169:0/1184404439 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa01c0a2950 0x7fa01c0a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:21.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.335+0000 7fa032ffd640 1 --2- 172.21.15.169:0/1184404439 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa01c0a2950 0x7fa01c0a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:21.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.336+0000 7fa01bfff640 1 -- 172.21.15.169:0/1184404439 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa02c098bf0 con 0x7fa03410d9e0 2024-03-28T01:06:21.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.336+0000 7fa032ffd640 1 --2- 172.21.15.169:0/1184404439 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa01c0a2950 0x7fa01c0a4e10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa024009910 tx=0x7fa024008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:21.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.354+0000 7fa01bfff640 1 -- 172.21.15.169:0/1184404439 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa02c0d0070 con 0x7fa03410d9e0 2024-03-28T01:06:21.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.863+0000 7fa039681640 1 -- 172.21.15.169:0/1184404439 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0200055b0 con 0x7fa03410d9e0 2024-03-28T01:06:21.870 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T01:06:21.870 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T01:06:21.871 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T01:06:21.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.866+0000 7fa01bfff640 1 -- 172.21.15.169:0/1184404439 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7fa02c09b250 con 0x7fa03410d9e0 2024-03-28T01:06:21.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.870+0000 7fa039681640 1 -- 172.21.15.169:0/1184404439 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa01c0a2950 msgr2=0x7fa01c0a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:21.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.870+0000 7fa039681640 1 --2- 172.21.15.169:0/1184404439 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa01c0a2950 0x7fa01c0a4e10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa024009910 tx=0x7fa024008040 comp rx=0 tx=0).stop 2024-03-28T01:06:21.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.870+0000 7fa039681640 1 -- 172.21.15.169:0/1184404439 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa03410d9e0 msgr2=0x7fa0341adcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:21.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.870+0000 7fa039681640 1 --2- 172.21.15.169:0/1184404439 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa03410d9e0 0x7fa0341adcf0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fa02c00bdd0 tx=0x7fa02c009ed0 comp rx=0 tx=0).stop 2024-03-28T01:06:21.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.871+0000 7fa039681640 1 -- 172.21.15.169:0/1184404439 shutdown_connections 2024-03-28T01:06:21.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.871+0000 7fa039681640 1 --2- 172.21.15.169:0/1184404439 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa01c0a2950 0x7fa01c0a4e10 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:21.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.871+0000 7fa039681640 1 --2- 172.21.15.169:0/1184404439 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa03410d9e0 0x7fa0341adcf0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:21.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.871+0000 7fa039681640 1 --2- 172.21.15.169:0/1184404439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa03410b010 0x7fa0341a6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:21.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.871+0000 7fa039681640 1 --2- 172.21.15.169:0/1184404439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0341086a0 0x7fa0341a6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:21.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.871+0000 7fa039681640 1 -- 172.21.15.169:0/1184404439 >> 172.21.15.169:0/1184404439 conn(0x7fa03406e510 msgr2=0x7fa0341093d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:21.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.871+0000 7fa039681640 1 -- 172.21.15.169:0/1184404439 shutdown_connections 2024-03-28T01:06:21.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:21.871+0000 7fa039681640 1 -- 172.21.15.169:0/1184404439 wait complete. 2024-03-28T01:06:21.901 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:06:21.901 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T01:06:22.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:21 smithi169 ceph-mon[32274]: pgmap v646: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 632 MiB/s wr, 484 op/s 2024-03-28T01:06:22.111 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:21 smithi174 ceph-mon[36060]: pgmap v646: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 632 MiB/s wr, 484 op/s 2024-03-28T01:06:22.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:21 smithi177 ceph-mon[36206]: pgmap v646: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 632 MiB/s wr, 484 op/s 2024-03-28T01:06:22.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.282+0000 7f3169861640 1 -- 172.21.15.169:0/1153827317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3164103260 msgr2=0x7f3164109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:22.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.282+0000 7f3169861640 1 --2- 172.21.15.169:0/1153827317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3164103260 0x7f3164109af0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f315c009ca0 tx=0x7f315c02c100 comp rx=0 tx=0).stop 2024-03-28T01:06:22.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.283+0000 7f3169861640 1 -- 172.21.15.169:0/1153827317 shutdown_connections 2024-03-28T01:06:22.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.283+0000 7f3169861640 1 --2- 172.21.15.169:0/1153827317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3164103260 0x7f3164109af0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.283+0000 7f3169861640 1 --2- 172.21.15.169:0/1153827317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31641028a0 0x7f3164102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.283+0000 7f3169861640 1 --2- 172.21.15.169:0/1153827317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31641016b0 0x7f3164101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.283+0000 7f3169861640 1 -- 172.21.15.169:0/1153827317 >> 172.21.15.169:0/1153827317 conn(0x7f31640fcc00 msgr2=0x7f31640ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:22.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.286+0000 7f3169861640 1 -- 172.21.15.169:0/1153827317 shutdown_connections 2024-03-28T01:06:22.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.287+0000 7f3169861640 1 -- 172.21.15.169:0/1153827317 wait complete. 2024-03-28T01:06:22.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.287+0000 7f3169861640 1 Processor -- start 2024-03-28T01:06:22.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.291+0000 7f3169861640 1 -- start start 2024-03-28T01:06:22.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.291+0000 7f3169861640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31641016b0 0x7f3164195740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.291+0000 7f3169861640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31641028a0 0x7f3164195c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.291+0000 7f3169861640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3164103260 0x7f316419ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.291+0000 7f3169861640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31640758e0 con 0x7f31641016b0 2024-03-28T01:06:22.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.291+0000 7f3169861640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3164075760 con 0x7f31641028a0 2024-03-28T01:06:22.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.291+0000 7f3169861640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316410f700 con 0x7f3164103260 2024-03-28T01:06:22.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.292+0000 7f3162ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31641016b0 0x7f3164195740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.292+0000 7f3162ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31641016b0 0x7f3164195740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:55762/0 (socket says 172.21.15.169:55762) 2024-03-28T01:06:22.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.292+0000 7f3162ffd640 1 -- 172.21.15.169:0/231939184 learned_addr learned my addr 172.21.15.169:0/231939184 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:22.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.292+0000 7f315a7fc640 1 --2- 172.21.15.169:0/231939184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31641028a0 0x7f3164195c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.293+0000 7f3162ffd640 1 -- 172.21.15.169:0/231939184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3164103260 msgr2=0x7f316419ccf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:06:22.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.293+0000 7f3162ffd640 1 --2- 172.21.15.169:0/231939184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3164103260 0x7f316419ccf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.293+0000 7f3162ffd640 1 -- 172.21.15.169:0/231939184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31641028a0 msgr2=0x7f3164195c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:22.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.293+0000 7f3162ffd640 1 --2- 172.21.15.169:0/231939184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31641028a0 0x7f3164195c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.293+0000 7f3162ffd640 1 -- 172.21.15.169:0/231939184 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31641ae400 con 0x7f31641016b0 2024-03-28T01:06:22.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.293+0000 7f3162ffd640 1 --2- 172.21.15.169:0/231939184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31641016b0 0x7f3164195740 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f315400e9c0 tx=0x7f315400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:22.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.297+0000 7f316885f640 1 -- 172.21.15.169:0/231939184 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315400cd80 con 0x7f31641016b0 2024-03-28T01:06:22.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.297+0000 7f316885f640 1 -- 172.21.15.169:0/231939184 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31540035f0 con 0x7f31641016b0 2024-03-28T01:06:22.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.297+0000 7f3169861640 1 -- 172.21.15.169:0/231939184 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31641ae690 con 0x7f31641016b0 2024-03-28T01:06:22.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.297+0000 7f316885f640 1 -- 172.21.15.169:0/231939184 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3154010690 con 0x7f31641016b0 2024-03-28T01:06:22.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.297+0000 7f3169861640 1 -- 172.21.15.169:0/231939184 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31641aec00 con 0x7f31641016b0 2024-03-28T01:06:22.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.300+0000 7f316c451740 1 Processor -- start 2024-03-28T01:06:22.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.301+0000 7f316c451740 1 -- start start 2024-03-28T01:06:22.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.301+0000 7f316c451740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a182c22690 0x55a182c22ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.301+0000 7f316c451740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a182c23080 0x55a182c27b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.301+0000 7f316c451740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c281f0 0x55a182c2a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.301+0000 7f316c451740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a182be5610 con 0x55a182c22690 2024-03-28T01:06:22.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.301+0000 7f316c451740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a182b883c0 con 0x55a182c281f0 2024-03-28T01:06:22.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.301+0000 7f316c451740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a182b30d60 con 0x55a182c23080 2024-03-28T01:06:22.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.302+0000 7f315a7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c281f0 0x55a182c2a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.302+0000 7f315a7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c281f0 0x55a182c2a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:38226/0 (socket says 172.21.15.169:38226) 2024-03-28T01:06:22.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.302+0000 7f315a7fc640 1 -- 172.21.15.169:0/2917441848 learned_addr learned my addr 172.21.15.169:0/2917441848 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:22.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.302+0000 7f31637fe640 1 --2- 172.21.15.169:0/2917441848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a182c22690 0x55a182c22ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.302+0000 7f31637fe640 1 -- 172.21.15.169:0/2917441848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a182c23080 msgr2=0x55a182c27b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:06:22.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.302+0000 7f31637fe640 1 --2- 172.21.15.169:0/2917441848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a182c23080 0x55a182c27b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.302+0000 7f31637fe640 1 -- 172.21.15.169:0/2917441848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c281f0 msgr2=0x55a182c2a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:22.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.302+0000 7f31637fe640 1 --2- 172.21.15.169:0/2917441848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c281f0 0x55a182c2a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.302+0000 7f31637fe640 1 -- 172.21.15.169:0/2917441848 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a18289c3f0 con 0x55a182c22690 2024-03-28T01:06:22.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.303+0000 7f316885f640 1 -- 172.21.15.169:0/231939184 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3154010830 con 0x7f31641016b0 2024-03-28T01:06:22.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.304+0000 7f31637fe640 1 --2- 172.21.15.169:0/2917441848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a182c22690 0x55a182c22ab0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f315c037040 tx=0x7f315c009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90fad93245aa005b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:22.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.304+0000 7f316885f640 1 --2- 172.21.15.169:0/231939184 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f31500a2880 0x7f31500a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.305+0000 7f3159ffb640 1 -- 172.21.15.169:0/2917441848 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c041020 con 0x55a182c22690 2024-03-28T01:06:22.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.305+0000 7f316885f640 1 -- 172.21.15.169:0/231939184 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f31540cf2f0 con 0x7f31641016b0 2024-03-28T01:06:22.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.305+0000 7f3159ffb640 1 -- 172.21.15.169:0/2917441848 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f315c003e50 con 0x55a182c22690 2024-03-28T01:06:22.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.306+0000 7f316c451740 1 -- 172.21.15.169:0/2917441848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a182c22690 msgr2=0x55a182c22ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:22.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.306+0000 7f316c451740 1 --2- 172.21.15.169:0/2917441848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a182c22690 0x55a182c22ab0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f315c037040 tx=0x7f315c009360 comp rx=0 tx=0).stop 2024-03-28T01:06:22.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.306+0000 7f3162ffd640 1 --2- 172.21.15.169:0/231939184 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f31500a2880 0x7f31500a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.308+0000 7f316c451740 1 -- 172.21.15.169:0/2917441848 shutdown_connections 2024-03-28T01:06:22.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.308+0000 7f316c451740 1 --2- 172.21.15.169:0/2917441848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c281f0 0x55a182c2a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.308+0000 7f316c451740 1 --2- 172.21.15.169:0/2917441848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a182c23080 0x55a182c27b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.308+0000 7f316c451740 1 --2- 172.21.15.169:0/2917441848 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a182c22690 0x55a182c22ab0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.308+0000 7f316c451740 1 -- 172.21.15.169:0/2917441848 >> 172.21.15.169:0/2917441848 conn(0x55a182c1dc00 msgr2=0x55a182c1e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:22.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.308+0000 7f3162ffd640 1 --2- 172.21.15.169:0/231939184 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f31500a2880 0x7f31500a4d40 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f315409b5f0 tx=0x7f3154020000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:22.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.308+0000 7f316c451740 1 -- 172.21.15.169:0/2917441848 shutdown_connections 2024-03-28T01:06:22.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.308+0000 7f316c451740 1 -- 172.21.15.169:0/2917441848 wait complete. 2024-03-28T01:06:22.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.309+0000 7f316c451740 1 -- start start 2024-03-28T01:06:22.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.309+0000 7f316c451740 1 Processor -- start 2024-03-28T01:06:22.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.309+0000 7f316c451740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c22690 0x55a182c3da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.309+0000 7f316c451740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a182c23080 0x55a182c3df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.309+0000 7f316c451740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a182c281f0 0x55a182c44fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.309+0000 7f316c451740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a182b44240 con 0x55a182c281f0 2024-03-28T01:06:22.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.309+0000 7f316c451740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a182b84460 con 0x55a182c22690 2024-03-28T01:06:22.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.309+0000 7f316c451740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a182c3c1a0 con 0x55a182c23080 2024-03-28T01:06:22.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.309+0000 7f316c451740 10 client.0 _ll_register_callbacks cb 0x55a182b43c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:06:22.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.313+0000 7f31637fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c22690 0x55a182c3da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.313+0000 7f31637fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c22690 0x55a182c3da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:38230/0 (socket says 172.21.15.169:38230) 2024-03-28T01:06:22.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.313+0000 7f31637fe640 1 -- 172.21.15.169:0/3217941064 learned_addr learned my addr 172.21.15.169:0/3217941064 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:22.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.313+0000 7f31637fe640 1 --2- 172.21.15.169:0/3217941064 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a182c281f0 0x55a182c44fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.313+0000 7f31637fe640 1 -- 172.21.15.169:0/3217941064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a182c23080 msgr2=0x55a182c3df60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:06:22.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.313+0000 7f31637fe640 1 --2- 172.21.15.169:0/3217941064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a182c23080 0x55a182c3df60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.313+0000 7f31637fe640 1 -- 172.21.15.169:0/3217941064 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a182c281f0 msgr2=0x55a182c44fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:22.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.313+0000 7f31637fe640 1 --2- 172.21.15.169:0/3217941064 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a182c281f0 0x55a182c44fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.313+0000 7f31637fe640 1 -- 172.21.15.169:0/3217941064 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a1829667d0 con 0x55a182c22690 2024-03-28T01:06:22.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.313+0000 7f31637fe640 1 --2- 172.21.15.169:0/3217941064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c22690 0x55a182c3da20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f315c004af0 tx=0x7f315c009540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fff4513c023549ac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:22.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.314+0000 7f3142ffd640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:06:22.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.314+0000 7f3142ffd640 1 -- 172.21.15.169:0/3217941064 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c041030 con 0x55a182c22690 2024-03-28T01:06:22.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.314+0000 7f3142ffd640 1 -- 172.21.15.169:0/3217941064 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f315c004520 con 0x55a182c22690 2024-03-28T01:06:22.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.314+0000 7f316c451740 1 -- 172.21.15.169:0/3217941064 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f315c003e50 con 0x55a182c22690 2024-03-28T01:06:22.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.316+0000 7f3142ffd640 1 -- 172.21.15.169:0/3217941064 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f315c055050 con 0x55a182c22690 2024-03-28T01:06:22.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.316+0000 7f316c451740 10 client.27173 fetch_fsmap learned FSMap version 25 2024-03-28T01:06:22.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.316+0000 7f316c451740 1 -- 172.21.15.169:0/3217941064 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a182c48310 con 0x55a182c22690 2024-03-28T01:06:22.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.316+0000 7f3142ffd640 1 -- 172.21.15.169:0/3217941064 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f315c003e50 con 0x55a182c22690 2024-03-28T01:06:22.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.316+0000 7f3142ffd640 1 -- 172.21.15.169:0/3217941064 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f315c03c030 con 0x55a182c22690 2024-03-28T01:06:22.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.317+0000 7f316c451740 10 client.27173 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:06:22.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.317+0000 7f316c451740 10 client.27173 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:06:22.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.318+0000 7f316c451740 20 client.27173 populate_metadata read hostname 'smithi169' 2024-03-28T01:06:22.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.318+0000 7f316c451740 1 --2- 172.21.15.169:0/3217941064 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55a182c49660 0x55a182c4ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.318+0000 7f316c451740 4 client.27173 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:06:22.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.318+0000 7f316c451740 1 -- 172.21.15.169:0/3217941064 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a182b95900 con 0x55a182c49660 2024-03-28T01:06:22.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.325+0000 7f315a7fc640 1 --2- 172.21.15.169:0/3217941064 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55a182c49660 0x55a182c4ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.329+0000 7f315a7fc640 1 --2- 172.21.15.169:0/3217941064 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55a182c49660 0x55a182c4ba50 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2ea98ea3c10198ec server_cookie=6291e9aca8c27eb7 in_seq=0 out_seq=0 2024-03-28T01:06:22.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.329+0000 7f3142ffd640 10 client.27173 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:06:22.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.330+0000 7f3142ffd640 1 -- 172.21.15.169:0/3217941064 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a182b95900 con 0x55a182c49660 2024-03-28T01:06:22.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.330+0000 7f3142ffd640 10 client.27173 handle_command_reply: tid=0 2024-03-28T01:06:22.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.330+0000 7f316c451740 1 client.27173 shutdown 2024-03-28T01:06:22.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.331+0000 7f316c451740 1 -- 172.21.15.169:0/3217941064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c22690 msgr2=0x55a182c3da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:22.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.331+0000 7f316c451740 1 --2- 172.21.15.169:0/3217941064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c22690 0x55a182c3da20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f315c004af0 tx=0x7f315c009540 comp rx=0 tx=0).stop 2024-03-28T01:06:22.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.331+0000 7f316c451740 1 -- 172.21.15.169:0/3217941064 shutdown_connections 2024-03-28T01:06:22.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.331+0000 7f316c451740 1 --2- 172.21.15.169:0/3217941064 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55a182c49660 0x55a182c4ba50 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.331+0000 7f316c451740 1 --2- 172.21.15.169:0/3217941064 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a182c281f0 0x55a182c44fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.331+0000 7f316c451740 1 --2- 172.21.15.169:0/3217941064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a182c23080 0x55a182c3df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.331+0000 7f316c451740 1 --2- 172.21.15.169:0/3217941064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c22690 0x55a182c3da20 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.331+0000 7f316c451740 1 -- 172.21.15.169:0/3217941064 >> 172.21.15.169:0/3217941064 conn(0x55a182c1dc00 msgr2=0x55a182c1ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:22.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.332+0000 7f3142ffd640 0 client.27173 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:06:22.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.332+0000 7f316c451740 1 -- 172.21.15.169:0/3217941064 shutdown_connections 2024-03-28T01:06:22.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.332+0000 7f316c451740 1 -- 172.21.15.169:0/3217941064 wait complete. 2024-03-28T01:06:22.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.332+0000 7f316c451740 20 client.27173 trim_cache size 0 max 16384 2024-03-28T01:06:22.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.357+0000 7f316c451740 1 Processor -- start 2024-03-28T01:06:22.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.357+0000 7f316c451740 1 -- start start 2024-03-28T01:06:22.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.358+0000 7f316c451740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c22690 0x55a182c3dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.358+0000 7f316c451740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a182c23080 0x55a182c44f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.358+0000 7f31637fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c22690 0x55a182c3dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.358+0000 7f31637fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c22690 0x55a182c3dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:38242/0 (socket says 172.21.15.169:38242) 2024-03-28T01:06:22.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.358+0000 7f31637fe640 1 -- 172.21.15.169:0/957620417 learned_addr learned my addr 172.21.15.169:0/957620417 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:22.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.358+0000 7f316c451740 1 --2- 172.21.15.169:0/957620417 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a182c281f0 0x55a182c1c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.358+0000 7f316c451740 1 -- 172.21.15.169:0/957620417 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31641b2da0 con 0x55a182c281f0 2024-03-28T01:06:22.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.358+0000 7f316c451740 1 -- 172.21.15.169:0/957620417 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31641b08a0 con 0x55a182c22690 2024-03-28T01:06:22.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.358+0000 7f316c451740 1 -- 172.21.15.169:0/957620417 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3164073370 con 0x55a182c23080 2024-03-28T01:06:22.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.359+0000 7f31637fe640 1 -- 172.21.15.169:0/957620417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a182c23080 msgr2=0x55a182c44f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:06:22.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.359+0000 7f31637fe640 1 --2- 172.21.15.169:0/957620417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a182c23080 0x55a182c44f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.359+0000 7f31637fe640 1 -- 172.21.15.169:0/957620417 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a182c281f0 msgr2=0x55a182c1c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:22.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.359+0000 7f31637fe640 1 --2- 172.21.15.169:0/957620417 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a182c281f0 0x55a182c1c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.359+0000 7f31637fe640 1 -- 172.21.15.169:0/957620417 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a182c19670 con 0x55a182c22690 2024-03-28T01:06:22.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.359+0000 7f31637fe640 1 --2- 172.21.15.169:0/957620417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c22690 0x55a182c3dd40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f315c0324c0 tx=0x7f315c032b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5101723528145cd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:22.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.360+0000 7f31597fa640 1 -- 172.21.15.169:0/957620417 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c041070 con 0x55a182c22690 2024-03-28T01:06:22.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.360+0000 7f31597fa640 1 -- 172.21.15.169:0/957620417 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f315c009870 con 0x55a182c22690 2024-03-28T01:06:22.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.360+0000 7f31597fa640 1 -- 172.21.15.169:0/957620417 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c032990 con 0x55a182c22690 2024-03-28T01:06:22.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.361+0000 7f316c451740 1 -- 172.21.15.169:0/957620417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c22690 msgr2=0x55a182c3dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:22.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.361+0000 7f316c451740 1 --2- 172.21.15.169:0/957620417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c22690 0x55a182c3dd40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f315c0324c0 tx=0x7f315c032b50 comp rx=0 tx=0).stop 2024-03-28T01:06:22.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.361+0000 7f316c451740 1 -- 172.21.15.169:0/957620417 shutdown_connections 2024-03-28T01:06:22.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.361+0000 7f316c451740 1 --2- 172.21.15.169:0/957620417 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a182c281f0 0x55a182c1c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.362+0000 7f316c451740 1 --2- 172.21.15.169:0/957620417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a182c23080 0x55a182c44f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.362+0000 7f316c451740 1 --2- 172.21.15.169:0/957620417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c22690 0x55a182c3dd40 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.362+0000 7f316c451740 1 -- 172.21.15.169:0/957620417 >> 172.21.15.169:0/957620417 conn(0x55a182c1dc00 msgr2=0x55a182c3d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:22.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.362+0000 7f316c451740 1 -- 172.21.15.169:0/957620417 shutdown_connections 2024-03-28T01:06:22.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.362+0000 7f316c451740 1 -- 172.21.15.169:0/957620417 wait complete. 2024-03-28T01:06:22.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.362+0000 7f316c451740 1 -- start start 2024-03-28T01:06:22.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.363+0000 7f316c451740 1 Processor -- start 2024-03-28T01:06:22.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.363+0000 7f316c451740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a182c22690 0x55a182c42cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.363+0000 7f316c451740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a182c23080 0x55a182c43230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.363+0000 7f316c451740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c281f0 0x55a182c44240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.363+0000 7f316c451740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31641aef70 con 0x55a182c23080 2024-03-28T01:06:22.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.363+0000 7f316c451740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31641b0240 con 0x55a182c281f0 2024-03-28T01:06:22.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.363+0000 7f316c451740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31641af580 con 0x55a182c22690 2024-03-28T01:06:22.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.363+0000 7f316c451740 10 client.0 _ll_register_callbacks cb 0x7f314c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:06:22.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.364+0000 7f31637fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a182c22690 0x55a182c42cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.364+0000 7f31637fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a182c22690 0x55a182c42cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:54466/0 (socket says 172.21.15.169:54466) 2024-03-28T01:06:22.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.364+0000 7f31637fe640 1 -- 172.21.15.169:0/320732190 learned_addr learned my addr 172.21.15.169:0/320732190 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:22.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.364+0000 7f31637fe640 1 --2- 172.21.15.169:0/320732190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c281f0 0x55a182c44240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.364+0000 7f315a7fc640 1 --2- 172.21.15.169:0/320732190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a182c23080 0x55a182c43230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.364+0000 7f31637fe640 1 -- 172.21.15.169:0/320732190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c281f0 msgr2=0x55a182c44240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:22.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.364+0000 7f31637fe640 1 --2- 172.21.15.169:0/320732190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c281f0 0x55a182c44240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.364+0000 7f31637fe640 1 -- 172.21.15.169:0/320732190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a182c23080 msgr2=0x55a182c43230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:22.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.364+0000 7f31637fe640 1 --2- 172.21.15.169:0/320732190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a182c23080 0x55a182c43230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.364+0000 7f31637fe640 1 -- 172.21.15.169:0/320732190 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a182c44830 con 0x55a182c22690 2024-03-28T01:06:22.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.365+0000 7f31637fe640 1 --2- 172.21.15.169:0/320732190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a182c22690 0x55a182c42cf0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f315c034d20 tx=0x7f315c02c780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1efe91ad3a224619 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:22.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.366+0000 7f3142ffd640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:06:22.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.366+0000 7f3142ffd640 1 -- 172.21.15.169:0/320732190 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c041030 con 0x55a182c22690 2024-03-28T01:06:22.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.366+0000 7f3142ffd640 1 -- 172.21.15.169:0/320732190 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f315c004d50 con 0x55a182c22690 2024-03-28T01:06:22.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.366+0000 7f316c451740 1 -- 172.21.15.169:0/320732190 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f315c009870 con 0x55a182c22690 2024-03-28T01:06:22.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.368+0000 7f3142ffd640 1 -- 172.21.15.169:0/320732190 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f315c032690 con 0x55a182c22690 2024-03-28T01:06:22.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.368+0000 7f316c451740 10 client.27118 fetch_fsmap learned FSMap version 25 2024-03-28T01:06:22.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.368+0000 7f316c451740 1 -- 172.21.15.169:0/320732190 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a182ca5420 con 0x55a182c22690 2024-03-28T01:06:22.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.371+0000 7f3142ffd640 1 -- 172.21.15.169:0/320732190 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f315c009870 con 0x55a182c22690 2024-03-28T01:06:22.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.371+0000 7f3142ffd640 1 -- 172.21.15.169:0/320732190 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f315c03c050 con 0x55a182c22690 2024-03-28T01:06:22.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.372+0000 7f316c451740 10 client.27118 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:06:22.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.372+0000 7f316c451740 10 client.27118 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:06:22.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.372+0000 7f316c451740 20 client.27118 populate_metadata read hostname 'smithi169' 2024-03-28T01:06:22.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.372+0000 7f316c451740 1 --2- 172.21.15.169:0/320732190 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55a182ca5c10 0x55a182ca8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.372+0000 7f316c451740 4 client.27118 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:06:22.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.372+0000 7f316c451740 1 -- 172.21.15.169:0/320732190 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a182b8fc50 con 0x55a182ca5c10 2024-03-28T01:06:22.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.374+0000 7f315a7fc640 1 --2- 172.21.15.169:0/320732190 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55a182ca5c10 0x55a182ca8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.398+0000 7f315a7fc640 1 --2- 172.21.15.169:0/320732190 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55a182ca5c10 0x55a182ca8030 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=846a0fd48cc4a44b server_cookie=cd448a4b6897ff5e in_seq=0 out_seq=0 2024-03-28T01:06:22.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.398+0000 7f3142ffd640 10 client.27118 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:06:22.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.400+0000 7f3142ffd640 1 -- 172.21.15.169:0/320732190 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a182b8fc50 con 0x55a182ca5c10 2024-03-28T01:06:22.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.400+0000 7f3142ffd640 10 client.27118 handle_command_reply: tid=0 2024-03-28T01:06:22.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.403+0000 7f316c451740 1 client.27118 shutdown 2024-03-28T01:06:22.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.409+0000 7f316c451740 1 -- 172.21.15.169:0/320732190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a182c22690 msgr2=0x55a182c42cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:22.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.409+0000 7f316c451740 1 --2- 172.21.15.169:0/320732190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a182c22690 0x55a182c42cf0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f315c034d20 tx=0x7f315c02c780 comp rx=0 tx=0).stop 2024-03-28T01:06:22.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.409+0000 7f316c451740 1 -- 172.21.15.169:0/320732190 shutdown_connections 2024-03-28T01:06:22.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.409+0000 7f316c451740 1 --2- 172.21.15.169:0/320732190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a182c281f0 0x55a182c44240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.409+0000 7f316c451740 1 --2- 172.21.15.169:0/320732190 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55a182ca5c10 0x55a182ca8030 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.409+0000 7f316c451740 1 --2- 172.21.15.169:0/320732190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a182c23080 0x55a182c43230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.409+0000 7f316c451740 1 --2- 172.21.15.169:0/320732190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a182c22690 0x55a182c42cf0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.409+0000 7f316c451740 1 -- 172.21.15.169:0/320732190 >> 172.21.15.169:0/320732190 conn(0x55a182c1dc00 msgr2=0x55a182c3ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:22.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.411+0000 7f3142ffd640 0 client.27118 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:06:22.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.417+0000 7f316c451740 1 -- 172.21.15.169:0/320732190 shutdown_connections 2024-03-28T01:06:22.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.419+0000 7f316c451740 1 -- 172.21.15.169:0/320732190 wait complete. 2024-03-28T01:06:22.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.423+0000 7f316c451740 20 client.27118 trim_cache size 0 max 16384 2024-03-28T01:06:22.426 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:06:22.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.424+0000 7f3169861640 1 -- 172.21.15.169:0/231939184 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f31500a2880 msgr2=0x7f31500a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:22.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.424+0000 7f3169861640 1 --2- 172.21.15.169:0/231939184 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f31500a2880 0x7f31500a4d40 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f315409b5f0 tx=0x7f3154020000 comp rx=0 tx=0).stop 2024-03-28T01:06:22.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.424+0000 7f3169861640 1 -- 172.21.15.169:0/231939184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31641016b0 msgr2=0x7f3164195740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:22.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.424+0000 7f3169861640 1 --2- 172.21.15.169:0/231939184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31641016b0 0x7f3164195740 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f315400e9c0 tx=0x7f315400ee80 comp rx=0 tx=0).stop 2024-03-28T01:06:22.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.435+0000 7f3169861640 1 -- 172.21.15.169:0/231939184 shutdown_connections 2024-03-28T01:06:22.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.435+0000 7f3169861640 1 --2- 172.21.15.169:0/231939184 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f31500a2880 0x7f31500a4d40 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.435+0000 7f3169861640 1 --2- 172.21.15.169:0/231939184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3164103260 0x7f316419ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.435+0000 7f3169861640 1 --2- 172.21.15.169:0/231939184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31641028a0 0x7f3164195c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.435+0000 7f3169861640 1 --2- 172.21.15.169:0/231939184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31641016b0 0x7f3164195740 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.435+0000 7f3169861640 1 -- 172.21.15.169:0/231939184 >> 172.21.15.169:0/231939184 conn(0x7f31640fcc00 msgr2=0x7f316410d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:22.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.443+0000 7f3169861640 1 -- 172.21.15.169:0/231939184 shutdown_connections 2024-03-28T01:06:22.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.445+0000 7f3169861640 1 -- 172.21.15.169:0/231939184 wait complete. 2024-03-28T01:06:22.511 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:06:22.511 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T01:06:22.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:22 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1184404439' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:06:22.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.852+0000 7f32f6f3b640 1 -- 172.21.15.169:0/3513680542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32f010b010 msgr2=0x7f32f010d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:22.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.852+0000 7f32f6f3b640 1 --2- 172.21.15.169:0/3513680542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32f010b010 0x7f32f010d480 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f32e401a910 tx=0x7f32e403cca0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.855+0000 7f32f6f3b640 1 -- 172.21.15.169:0/3513680542 shutdown_connections 2024-03-28T01:06:22.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.855+0000 7f32f6f3b640 1 --2- 172.21.15.169:0/3513680542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32f010d9c0 0x7f32f010de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.855+0000 7f32f6f3b640 1 --2- 172.21.15.169:0/3513680542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32f010b010 0x7f32f010d480 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.855+0000 7f32f6f3b640 1 --2- 172.21.15.169:0/3513680542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32f01086a0 0x7f32f010aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.855+0000 7f32f6f3b640 1 -- 172.21.15.169:0/3513680542 >> 172.21.15.169:0/3513680542 conn(0x7f32f006e500 msgr2=0x7f32f0070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:22.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.860+0000 7f32f6f3b640 1 -- 172.21.15.169:0/3513680542 shutdown_connections 2024-03-28T01:06:22.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.860+0000 7f32f6f3b640 1 -- 172.21.15.169:0/3513680542 wait complete. 2024-03-28T01:06:22.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.861+0000 7f32f6f3b640 1 Processor -- start 2024-03-28T01:06:22.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.861+0000 7f32f6f3b640 1 -- start start 2024-03-28T01:06:22.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.861+0000 7f32f6f3b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32f01086a0 0x7f32f019e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.861+0000 7f32f6f3b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32f010b010 0x7f32f019e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.861+0000 7f32f6f3b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32f010d9c0 0x7f32f01c7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.861+0000 7f32f6f3b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f0111a60 con 0x7f32f01086a0 2024-03-28T01:06:22.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.861+0000 7f32f6f3b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f01118e0 con 0x7f32f010b010 2024-03-28T01:06:22.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.861+0000 7f32f6f3b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f0111be0 con 0x7f32f010d9c0 2024-03-28T01:06:22.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.862+0000 7f32effff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32f010b010 0x7f32f019e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.862+0000 7f32effff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32f010b010 0x7f32f019e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:38246/0 (socket says 172.21.15.169:38246) 2024-03-28T01:06:22.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.862+0000 7f32effff640 1 -- 172.21.15.169:0/2112020971 learned_addr learned my addr 172.21.15.169:0/2112020971 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:22.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.862+0000 7f32f4cd7640 1 --2- 172.21.15.169:0/2112020971 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32f01086a0 0x7f32f019e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.862+0000 7f32effff640 1 -- 172.21.15.169:0/2112020971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32f010d9c0 msgr2=0x7f32f01c7770 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:06:22.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.862+0000 7f32effff640 1 --2- 172.21.15.169:0/2112020971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32f010d9c0 0x7f32f01c7770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.862+0000 7f32effff640 1 -- 172.21.15.169:0/2112020971 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32f01086a0 msgr2=0x7f32f019e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:22.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.862+0000 7f32effff640 1 --2- 172.21.15.169:0/2112020971 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32f01086a0 0x7f32f019e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.862+0000 7f32effff640 1 -- 172.21.15.169:0/2112020971 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f01c7de0 con 0x7f32f010b010 2024-03-28T01:06:22.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.862+0000 7f32effff640 1 --2- 172.21.15.169:0/2112020971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32f010b010 0x7f32f019e6f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f32e4005ab0 tx=0x7f32e4004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:22.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.863+0000 7f32edffb640 1 -- 172.21.15.169:0/2112020971 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e4011020 con 0x7f32f010b010 2024-03-28T01:06:22.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.863+0000 7f32edffb640 1 -- 172.21.15.169:0/2112020971 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32e40035f0 con 0x7f32f010b010 2024-03-28T01:06:22.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.863+0000 7f32edffb640 1 -- 172.21.15.169:0/2112020971 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e400d6c0 con 0x7f32f010b010 2024-03-28T01:06:22.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.863+0000 7f32f6f3b640 1 -- 172.21.15.169:0/2112020971 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f01c8070 con 0x7f32f010b010 2024-03-28T01:06:22.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.863+0000 7f32f6f3b640 1 -- 172.21.15.169:0/2112020971 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f01c84d0 con 0x7f32f010b010 2024-03-28T01:06:22.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.869+0000 7f32f9c09740 1 Processor -- start 2024-03-28T01:06:22.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.869+0000 7f32edffb640 1 -- 172.21.15.169:0/2112020971 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f32e4014020 con 0x7f32f010b010 2024-03-28T01:06:22.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.870+0000 7f32edffb640 1 --2- 172.21.15.169:0/2112020971 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f32d80a2950 0x7f32d80a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.871+0000 7f32edffb640 1 -- 172.21.15.169:0/2112020971 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f32e40f1790 con 0x7f32f010b010 2024-03-28T01:06:22.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.871+0000 7f32f9c09740 1 -- start start 2024-03-28T01:06:22.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.872+0000 7f32f9c09740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 0x560608e0aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.872+0000 7f32f9c09740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e0b080 0x560608e0fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.872+0000 7f32f9c09740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e101f0 0x560608e12680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.872+0000 7f32f9c09740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x560608dcd610 con 0x560608e0a690 2024-03-28T01:06:22.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.872+0000 7f32f9c09740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x560608d703c0 con 0x560608e101f0 2024-03-28T01:06:22.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.872+0000 7f32f9c09740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560608d18d60 con 0x560608e0b080 2024-03-28T01:06:22.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.876+0000 7f32f4cd7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 0x560608e0aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.876+0000 7f32f4cd7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 0x560608e0aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:55826/0 (socket says 172.21.15.169:55826) 2024-03-28T01:06:22.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.876+0000 7f32f4cd7640 1 -- 172.21.15.169:0/1841614916 learned_addr learned my addr 172.21.15.169:0/1841614916 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:22.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.876+0000 7f32effff640 1 --2- 172.21.15.169:0/1841614916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e0b080 0x560608e0fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.877+0000 7f32f54d8640 1 --2- 172.21.15.169:0/1841614916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e101f0 0x560608e12680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.877+0000 7f32f54d8640 1 --2- 172.21.15.169:0/2112020971 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f32d80a2950 0x7f32d80a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.877+0000 7f32f54d8640 1 -- 172.21.15.169:0/1841614916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e0b080 msgr2=0x560608e0fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:22.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.877+0000 7f32f54d8640 1 --2- 172.21.15.169:0/1841614916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e0b080 0x560608e0fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.877+0000 7f32f54d8640 1 -- 172.21.15.169:0/1841614916 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 msgr2=0x560608e0aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:22.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.877+0000 7f32f54d8640 1 --2- 172.21.15.169:0/1841614916 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 0x560608e0aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.877+0000 7f32f54d8640 1 -- 172.21.15.169:0/1841614916 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560608a843f0 con 0x560608e101f0 2024-03-28T01:06:22.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.878+0000 7f32f54d8640 1 --2- 172.21.15.169:0/1841614916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e101f0 0x560608e12680 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f32e8012a20 tx=0x7f32e8012ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=307e530ba90398dd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:22.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.878+0000 7f32f54d8640 1 --2- 172.21.15.169:0/2112020971 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f32d80a2950 0x7f32d80a4e10 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f32e8014870 tx=0x7f32e8010000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:22.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.879+0000 7f32d6ffd640 1 -- 172.21.15.169:0/1841614916 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e800ade0 con 0x560608e101f0 2024-03-28T01:06:22.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.879+0000 7f32d6ffd640 1 -- 172.21.15.169:0/1841614916 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32e8015d00 con 0x560608e101f0 2024-03-28T01:06:22.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.880+0000 7f32f9c09740 1 -- 172.21.15.169:0/1841614916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e101f0 msgr2=0x560608e12680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:22.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.880+0000 7f32f9c09740 1 --2- 172.21.15.169:0/1841614916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e101f0 0x560608e12680 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f32e8012a20 tx=0x7f32e8012ee0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.880+0000 7f32f9c09740 1 -- 172.21.15.169:0/1841614916 shutdown_connections 2024-03-28T01:06:22.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.880+0000 7f32f9c09740 1 --2- 172.21.15.169:0/1841614916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e101f0 0x560608e12680 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.880+0000 7f32f9c09740 1 --2- 172.21.15.169:0/1841614916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e0b080 0x560608e0fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.880+0000 7f32f9c09740 1 --2- 172.21.15.169:0/1841614916 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 0x560608e0aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.880+0000 7f32f9c09740 1 -- 172.21.15.169:0/1841614916 >> 172.21.15.169:0/1841614916 conn(0x560608e05c00 msgr2=0x560608e06000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:22.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.885+0000 7f32f9c09740 1 -- 172.21.15.169:0/1841614916 shutdown_connections 2024-03-28T01:06:22.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.888+0000 7f32f9c09740 1 -- 172.21.15.169:0/1841614916 wait complete. 2024-03-28T01:06:22.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.889+0000 7f32f9c09740 1 -- start start 2024-03-28T01:06:22.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.889+0000 7f32f9c09740 1 Processor -- start 2024-03-28T01:06:22.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.889+0000 7f32f9c09740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 0x560608e14c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.889+0000 7f32f9c09740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e0b080 0x560608e15180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.889+0000 7f32f9c09740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e101f0 0x560608e1c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.889+0000 7f32f9c09740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x560608d2c240 con 0x560608e0a690 2024-03-28T01:06:22.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.889+0000 7f32f9c09740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x560608d6c460 con 0x560608e0b080 2024-03-28T01:06:22.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.889+0000 7f32f9c09740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560608e13190 con 0x560608e101f0 2024-03-28T01:06:22.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.889+0000 7f32f9c09740 10 client.0 _ll_register_callbacks cb 0x560608d2bc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:06:22.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.893+0000 7f32f4cd7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 0x560608e14c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.893+0000 7f32f4cd7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 0x560608e14c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:55840/0 (socket says 172.21.15.169:55840) 2024-03-28T01:06:22.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.893+0000 7f32f4cd7640 1 -- 172.21.15.169:0/3595149189 learned_addr learned my addr 172.21.15.169:0/3595149189 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:22.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.893+0000 7f32f54d8640 1 --2- 172.21.15.169:0/3595149189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e101f0 0x560608e1c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.894+0000 7f32effff640 1 --2- 172.21.15.169:0/3595149189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e0b080 0x560608e15180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.898+0000 7f32f4cd7640 1 -- 172.21.15.169:0/3595149189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e101f0 msgr2=0x560608e1c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:22.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.898+0000 7f32f4cd7640 1 --2- 172.21.15.169:0/3595149189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e101f0 0x560608e1c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.898+0000 7f32f4cd7640 1 -- 172.21.15.169:0/3595149189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e0b080 msgr2=0x560608e15180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:22.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.898+0000 7f32f4cd7640 1 --2- 172.21.15.169:0/3595149189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e0b080 0x560608e15180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.898+0000 7f32f4cd7640 1 -- 172.21.15.169:0/3595149189 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560608b4e7d0 con 0x560608e0a690 2024-03-28T01:06:22.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.898+0000 7f32effff640 1 --2- 172.21.15.169:0/3595149189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e0b080 0x560608e15180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:06:22.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.899+0000 7f32f4cd7640 1 --2- 172.21.15.169:0/3595149189 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 0x560608e14c40 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f32e000de90 tx=0x7f32e000f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809e2e089f49f76e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:22.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.899+0000 7f32cffff640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:06:22.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.899+0000 7f32cffff640 1 -- 172.21.15.169:0/3595149189 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e0019070 con 0x560608e0a690 2024-03-28T01:06:22.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.900+0000 7f32f9c09740 1 -- 172.21.15.169:0/3595149189 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560608d7d900 con 0x560608e0a690 2024-03-28T01:06:22.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.900+0000 7f32cffff640 1 -- 172.21.15.169:0/3595149189 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f32e0014660 con 0x560608e0a690 2024-03-28T01:06:22.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.901+0000 7f32cffff640 1 -- 172.21.15.169:0/3595149189 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32e0024050 con 0x560608e0a690 2024-03-28T01:06:22.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.901+0000 7f32f9c09740 10 client.17328 fetch_fsmap learned FSMap version 25 2024-03-28T01:06:22.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.901+0000 7f32f9c09740 1 -- 172.21.15.169:0/3595149189 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x560608e1f690 con 0x560608e0a690 2024-03-28T01:06:22.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.902+0000 7f32cffff640 1 -- 172.21.15.169:0/3595149189 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560608d7d900 con 0x560608e0a690 2024-03-28T01:06:22.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.904+0000 7f32cffff640 1 -- 172.21.15.169:0/3595149189 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f32e0015aa0 con 0x560608e0a690 2024-03-28T01:06:22.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.904+0000 7f32f9c09740 10 client.17328 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:06:22.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.904+0000 7f32f9c09740 10 client.17328 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:06:22.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.904+0000 7f32f9c09740 20 client.17328 populate_metadata read hostname 'smithi169' 2024-03-28T01:06:22.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.904+0000 7f32f9c09740 1 --2- 172.21.15.169:0/3595149189 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x560608e21160 0x560608e235c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.904+0000 7f32f9c09740 4 client.17328 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:06:22.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.904+0000 7f32f9c09740 1 -- 172.21.15.169:0/3595149189 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560608d77c50 con 0x560608e21160 2024-03-28T01:06:22.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.908+0000 7f32effff640 1 --2- 172.21.15.169:0/3595149189 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x560608e21160 0x560608e235c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.912+0000 7f32effff640 1 --2- 172.21.15.169:0/3595149189 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x560608e21160 0x560608e235c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d6bd5a0577d042d9 server_cookie=943dd828ff7fda74 in_seq=0 out_seq=0 2024-03-28T01:06:22.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.913+0000 7f32cffff640 10 client.17328 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:06:22.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.914+0000 7f32cffff640 1 -- 172.21.15.169:0/3595149189 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560608d77c50 con 0x560608e21160 2024-03-28T01:06:22.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.914+0000 7f32cffff640 10 client.17328 handle_command_reply: tid=0 2024-03-28T01:06:22.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.915+0000 7f32f9c09740 1 client.17328 shutdown 2024-03-28T01:06:22.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.917+0000 7f32f9c09740 1 -- 172.21.15.169:0/3595149189 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 msgr2=0x560608e14c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:22.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.917+0000 7f32f9c09740 1 --2- 172.21.15.169:0/3595149189 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 0x560608e14c40 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f32e000de90 tx=0x7f32e000f5b0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.919+0000 7f32f9c09740 1 -- 172.21.15.169:0/3595149189 shutdown_connections 2024-03-28T01:06:22.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.919+0000 7f32f9c09740 1 --2- 172.21.15.169:0/3595149189 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x560608e21160 0x560608e235c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.919+0000 7f32f9c09740 1 --2- 172.21.15.169:0/3595149189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e101f0 0x560608e1c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.919+0000 7f32f9c09740 1 --2- 172.21.15.169:0/3595149189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e0b080 0x560608e15180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.919+0000 7f32f9c09740 1 --2- 172.21.15.169:0/3595149189 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 0x560608e14c40 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.919+0000 7f32f9c09740 1 -- 172.21.15.169:0/3595149189 >> 172.21.15.169:0/3595149189 conn(0x560608e05c00 msgr2=0x560608e04e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:22.924 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.922+0000 7f32cffff640 0 client.17328 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:06:22.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.926+0000 7f32f9c09740 1 -- 172.21.15.169:0/3595149189 shutdown_connections 2024-03-28T01:06:22.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.926+0000 7f32f9c09740 1 -- 172.21.15.169:0/3595149189 wait complete. 2024-03-28T01:06:22.929 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.926+0000 7f32f9c09740 20 client.17328 trim_cache size 0 max 16384 2024-03-28T01:06:22.977 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.973+0000 7f32f9c09740 1 Processor -- start 2024-03-28T01:06:22.978 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.975+0000 7f32f9c09740 1 -- start start 2024-03-28T01:06:22.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.976+0000 7f32f9c09740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 0x560608e0bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.976+0000 7f32f9c09740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e0b080 0x560608e1c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.976+0000 7f32f9c09740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e101f0 0x560608e04590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.976+0000 7f32f9c09740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f01d4e40 con 0x560608e0a690 2024-03-28T01:06:22.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.977+0000 7f32f9c09740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f01d2940 con 0x560608e0b080 2024-03-28T01:06:22.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.977+0000 7f32f9c09740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f01d0a90 con 0x560608e101f0 2024-03-28T01:06:22.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.977+0000 7f32f54d8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e101f0 0x560608e04590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.977+0000 7f32f54d8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e101f0 0x560608e04590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:54502/0 (socket says 172.21.15.169:54502) 2024-03-28T01:06:22.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.977+0000 7f32f54d8640 1 -- 172.21.15.169:0/2156761104 learned_addr learned my addr 172.21.15.169:0/2156761104 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:22.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.978+0000 7f32f54d8640 1 -- 172.21.15.169:0/2156761104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e0b080 msgr2=0x560608e1c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:22.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.978+0000 7f32f54d8640 1 --2- 172.21.15.169:0/2156761104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e0b080 0x560608e1c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.978+0000 7f32f54d8640 1 -- 172.21.15.169:0/2156761104 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 msgr2=0x560608e0bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:22.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.978+0000 7f32f54d8640 1 --2- 172.21.15.169:0/2156761104 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 0x560608e0bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.978+0000 7f32f54d8640 1 -- 172.21.15.169:0/2156761104 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560608e0b460 con 0x560608e101f0 2024-03-28T01:06:22.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.978+0000 7f32f54d8640 1 --2- 172.21.15.169:0/2156761104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e101f0 0x560608e04590 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f32e8013010 tx=0x7f32e800a6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec5bfc75e4cb20b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:22.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.979+0000 7f32d67fc640 1 -- 172.21.15.169:0/2156761104 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e8004e10 con 0x560608e101f0 2024-03-28T01:06:22.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.981+0000 7f32d67fc640 1 -- 172.21.15.169:0/2156761104 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32e802e070 con 0x560608e101f0 2024-03-28T01:06:22.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.981+0000 7f32d67fc640 1 -- 172.21.15.169:0/2156761104 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e8015a80 con 0x560608e101f0 2024-03-28T01:06:22.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.983+0000 7f32f9c09740 1 -- 172.21.15.169:0/2156761104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e101f0 msgr2=0x560608e04590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:22.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.983+0000 7f32f9c09740 1 --2- 172.21.15.169:0/2156761104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e101f0 0x560608e04590 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f32e8013010 tx=0x7f32e800a6c0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.987+0000 7f32f9c09740 1 -- 172.21.15.169:0/2156761104 shutdown_connections 2024-03-28T01:06:22.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.987+0000 7f32f9c09740 1 --2- 172.21.15.169:0/2156761104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e101f0 0x560608e04590 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.987+0000 7f32f9c09740 1 --2- 172.21.15.169:0/2156761104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e0b080 0x560608e1c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.987+0000 7f32f9c09740 1 --2- 172.21.15.169:0/2156761104 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 0x560608e0bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:22.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.987+0000 7f32f9c09740 1 -- 172.21.15.169:0/2156761104 >> 172.21.15.169:0/2156761104 conn(0x560608e05c00 msgr2=0x560608e0c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:22.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.987+0000 7f32f9c09740 1 -- 172.21.15.169:0/2156761104 shutdown_connections 2024-03-28T01:06:22.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.988+0000 7f32f9c09740 1 -- 172.21.15.169:0/2156761104 wait complete. 2024-03-28T01:06:22.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.988+0000 7f32f9c09740 1 -- start start 2024-03-28T01:06:22.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.989+0000 7f32f9c09740 1 Processor -- start 2024-03-28T01:06:22.993 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.990+0000 7f32f9c09740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 0x560608e19030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.992+0000 7f32f4cd7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 0x560608e19030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:22.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.992+0000 7f32f4cd7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 0x560608e19030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:55860/0 (socket says 172.21.15.169:55860) 2024-03-28T01:06:22.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.993+0000 7f32f4cd7640 1 -- 172.21.15.169:0/1328904977 learned_addr learned my addr 172.21.15.169:0/1328904977 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:22.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.993+0000 7f32f9c09740 1 --2- 172.21.15.169:0/1328904977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e0b080 0x560608e19570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.993+0000 7f32f9c09740 1 --2- 172.21.15.169:0/1328904977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e101f0 0x560608e7aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:22.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.993+0000 7f32f9c09740 1 -- 172.21.15.169:0/1328904977 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f01d1000 con 0x560608e0a690 2024-03-28T01:06:22.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.993+0000 7f32f9c09740 1 -- 172.21.15.169:0/1328904977 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f01d22e0 con 0x560608e101f0 2024-03-28T01:06:22.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.993+0000 7f32f9c09740 1 -- 172.21.15.169:0/1328904977 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f01d1620 con 0x560608e0b080 2024-03-28T01:06:22.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.994+0000 7f32f9c09740 10 client.0 _ll_register_callbacks cb 0x7f32e000da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:06:22.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.994+0000 7f32f54d8640 1 --2- 172.21.15.169:0/1328904977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e101f0 0x560608e7aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:23.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.997+0000 7f32effff640 1 --2- 172.21.15.169:0/1328904977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e0b080 0x560608e19570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:23.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.997+0000 7f32f54d8640 1 -- 172.21.15.169:0/1328904977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e0b080 msgr2=0x560608e19570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:23.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.997+0000 7f32f54d8640 1 --2- 172.21.15.169:0/1328904977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e0b080 0x560608e19570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.997+0000 7f32f54d8640 1 -- 172.21.15.169:0/1328904977 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 msgr2=0x560608e19030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:23.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.997+0000 7f32f54d8640 1 --2- 172.21.15.169:0/1328904977 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 0x560608e19030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.997+0000 7f32f54d8640 1 -- 172.21.15.169:0/1328904977 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560608e7afc0 con 0x560608e101f0 2024-03-28T01:06:23.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.999+0000 7f32f54d8640 1 --2- 172.21.15.169:0/1328904977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e101f0 0x560608e7aa80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f32e80128f0 tx=0x7f32e800a3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd8432fe2e526cc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:23.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.999+0000 7f32cffff640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:06:23.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:22.999+0000 7f32cffff640 1 -- 172.21.15.169:0/1328904977 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e8039060 con 0x560608e101f0 2024-03-28T01:06:23.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.000+0000 7f32f9c09740 1 -- 172.21.15.169:0/1328904977 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f32e802e070 con 0x560608e101f0 2024-03-28T01:06:23.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.000+0000 7f32cffff640 1 -- 172.21.15.169:0/1328904977 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f32e8003c30 con 0x560608e101f0 2024-03-28T01:06:23.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.002+0000 7f32cffff640 1 -- 172.21.15.169:0/1328904977 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32e8003e30 con 0x560608e101f0 2024-03-28T01:06:23.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.002+0000 7f32f9c09740 10 client.27191 fetch_fsmap learned FSMap version 25 2024-03-28T01:06:23.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.002+0000 7f32f9c09740 1 -- 172.21.15.169:0/1328904977 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x560608e7c1c0 con 0x560608e101f0 2024-03-28T01:06:23.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.006+0000 7f32cffff640 1 -- 172.21.15.169:0/1328904977 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32e8015d00 con 0x560608e101f0 2024-03-28T01:06:23.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.006+0000 7f32cffff640 1 -- 172.21.15.169:0/1328904977 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f32e8020480 con 0x560608e101f0 2024-03-28T01:06:23.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.007+0000 7f32f9c09740 10 client.27191 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:06:23.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.007+0000 7f32f9c09740 10 client.27191 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:06:23.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.007+0000 7f32f9c09740 20 client.27191 populate_metadata read hostname 'smithi169' 2024-03-28T01:06:23.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.007+0000 7f32f9c09740 1 --2- 172.21.15.169:0/1328904977 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x560608e7c830 0x560608e7ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:23.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.007+0000 7f32f9c09740 4 client.27191 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:06:23.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.007+0000 7f32f9c09740 1 -- 172.21.15.169:0/1328904977 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f32e802e070 con 0x560608e7c830 2024-03-28T01:06:23.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.016+0000 7f32f4cd7640 1 --2- 172.21.15.169:0/1328904977 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x560608e7c830 0x560608e7ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:23.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.048+0000 7f32f4cd7640 1 --2- 172.21.15.169:0/1328904977 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x560608e7c830 0x560608e7ec50 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=280ff3803e378f64 server_cookie=9d7cbe6f5b8487d1 in_seq=0 out_seq=0 2024-03-28T01:06:23.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.048+0000 7f32cffff640 10 client.27191 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:06:23.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.052+0000 7f32cffff640 1 -- 172.21.15.169:0/1328904977 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f32e802e070 con 0x560608e7c830 2024-03-28T01:06:23.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.052+0000 7f32cffff640 10 client.27191 handle_command_reply: tid=0 2024-03-28T01:06:23.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.052+0000 7f32f9c09740 1 client.27191 shutdown 2024-03-28T01:06:23.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.052+0000 7f32f9c09740 1 -- 172.21.15.169:0/1328904977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e101f0 msgr2=0x560608e7aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:23.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.052+0000 7f32f9c09740 1 --2- 172.21.15.169:0/1328904977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e101f0 0x560608e7aa80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f32e80128f0 tx=0x7f32e800a3b0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.052+0000 7f32f9c09740 1 -- 172.21.15.169:0/1328904977 shutdown_connections 2024-03-28T01:06:23.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.052+0000 7f32f9c09740 1 --2- 172.21.15.169:0/1328904977 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x560608e7c830 0x560608e7ec50 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.052+0000 7f32f9c09740 1 --2- 172.21.15.169:0/1328904977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560608e101f0 0x560608e7aa80 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.052+0000 7f32f9c09740 1 --2- 172.21.15.169:0/1328904977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560608e0b080 0x560608e19570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.052+0000 7f32f9c09740 1 --2- 172.21.15.169:0/1328904977 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560608e0a690 0x560608e19030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.052+0000 7f32f9c09740 1 -- 172.21.15.169:0/1328904977 >> 172.21.15.169:0/1328904977 conn(0x560608e05c00 msgr2=0x560608e08e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:23.057 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.054+0000 7f32cffff640 0 client.27191 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:06:23.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.058+0000 7f32f9c09740 1 -- 172.21.15.169:0/1328904977 shutdown_connections 2024-03-28T01:06:23.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.061+0000 7f32f9c09740 1 -- 172.21.15.169:0/1328904977 wait complete. 2024-03-28T01:06:23.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.061+0000 7f32f9c09740 20 client.27191 trim_cache size 0 max 16384 2024-03-28T01:06:23.067 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:06:23.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.066+0000 7f32f6f3b640 1 -- 172.21.15.169:0/2112020971 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f32d80a2950 msgr2=0x7f32d80a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:23.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.066+0000 7f32f6f3b640 1 --2- 172.21.15.169:0/2112020971 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f32d80a2950 0x7f32d80a4e10 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f32e8014870 tx=0x7f32e8010000 comp rx=0 tx=0).stop 2024-03-28T01:06:23.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.066+0000 7f32f6f3b640 1 -- 172.21.15.169:0/2112020971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32f010b010 msgr2=0x7f32f019e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:23.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.066+0000 7f32f6f3b640 1 --2- 172.21.15.169:0/2112020971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32f010b010 0x7f32f019e6f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f32e4005ab0 tx=0x7f32e4004270 comp rx=0 tx=0).stop 2024-03-28T01:06:23.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.066+0000 7f32f6f3b640 1 -- 172.21.15.169:0/2112020971 shutdown_connections 2024-03-28T01:06:23.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.067+0000 7f32f6f3b640 1 --2- 172.21.15.169:0/2112020971 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f32d80a2950 0x7f32d80a4e10 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.067+0000 7f32f6f3b640 1 --2- 172.21.15.169:0/2112020971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32f010d9c0 0x7f32f01c7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.067+0000 7f32f6f3b640 1 --2- 172.21.15.169:0/2112020971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32f010b010 0x7f32f019e6f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.067+0000 7f32f6f3b640 1 --2- 172.21.15.169:0/2112020971 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32f01086a0 0x7f32f019e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.067+0000 7f32f6f3b640 1 -- 172.21.15.169:0/2112020971 >> 172.21.15.169:0/2112020971 conn(0x7f32f006e500 msgr2=0x7f32f010b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:23.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.067+0000 7f32f6f3b640 1 -- 172.21.15.169:0/2112020971 shutdown_connections 2024-03-28T01:06:23.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.067+0000 7f32f6f3b640 1 -- 172.21.15.169:0/2112020971 wait complete. 2024-03-28T01:06:23.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:22 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1184404439' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:06:23.121 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:06:23.121 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T01:06:23.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:22 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1184404439' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:06:23.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.519+0000 7f3528bc5640 1 -- 172.21.15.169:0/3861676939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3524108880 msgr2=0x7f352410ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:23.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.519+0000 7f3528bc5640 1 --2- 172.21.15.169:0/3861676939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3524108880 0x7f352410ac70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3514009860 tx=0x7f351402bb90 comp rx=0 tx=0).stop 2024-03-28T01:06:23.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.523+0000 7f3528bc5640 1 -- 172.21.15.169:0/3861676939 shutdown_connections 2024-03-28T01:06:23.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.523+0000 7f3528bc5640 1 --2- 172.21.15.169:0/3861676939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f352410dc30 0x7f35241100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.523+0000 7f3528bc5640 1 --2- 172.21.15.169:0/3861676939 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f352410b1b0 0x7f352410d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.523+0000 7f3528bc5640 1 --2- 172.21.15.169:0/3861676939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3524108880 0x7f352410ac70 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.523+0000 7f3528bc5640 1 -- 172.21.15.169:0/3861676939 >> 172.21.15.169:0/3861676939 conn(0x7f352406e580 msgr2=0x7f35240709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:23.534 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.531+0000 7f3528bc5640 1 -- 172.21.15.169:0/3861676939 shutdown_connections 2024-03-28T01:06:23.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.534+0000 7f3528bc5640 1 -- 172.21.15.169:0/3861676939 wait complete. 2024-03-28T01:06:23.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.535+0000 7f3528bc5640 1 Processor -- start 2024-03-28T01:06:23.538 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.535+0000 7f3528bc5640 1 -- start start 2024-03-28T01:06:23.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.535+0000 7f3528bc5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3524108880 0x7f35241a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:23.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.535+0000 7f3528bc5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f352410b1b0 0x7f35241a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:23.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.535+0000 7f3528bc5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f352410dc30 0x7f35241a7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:23.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.535+0000 7f3528bc5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3524113860 con 0x7f3524108880 2024-03-28T01:06:23.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.535+0000 7f3528bc5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35241136e0 con 0x7f352410b1b0 2024-03-28T01:06:23.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.535+0000 7f3528bc5640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35241139e0 con 0x7f352410dc30 2024-03-28T01:06:23.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.536+0000 7f3522ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f352410b1b0 0x7f35241a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:23.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.536+0000 7f3522ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f352410b1b0 0x7f35241a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:38294/0 (socket says 172.21.15.169:38294) 2024-03-28T01:06:23.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.536+0000 7f3522ffd640 1 -- 172.21.15.169:0/799174961 learned_addr learned my addr 172.21.15.169:0/799174961 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:23.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.536+0000 7f35237fe640 1 --2- 172.21.15.169:0/799174961 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3524108880 0x7f35241a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:23.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.537+0000 7f3522ffd640 1 -- 172.21.15.169:0/799174961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f352410dc30 msgr2=0x7f35241a7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:23.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.537+0000 7f3522ffd640 1 --2- 172.21.15.169:0/799174961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f352410dc30 0x7f35241a7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.537+0000 7f3522ffd640 1 -- 172.21.15.169:0/799174961 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3524108880 msgr2=0x7f35241a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:23.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.537+0000 7f3522ffd640 1 --2- 172.21.15.169:0/799174961 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3524108880 0x7f35241a0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.537+0000 7f3522ffd640 1 -- 172.21.15.169:0/799174961 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35241a7d70 con 0x7f352410b1b0 2024-03-28T01:06:23.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.537+0000 7f3522ffd640 1 --2- 172.21.15.169:0/799174961 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f352410b1b0 0x7f35241a0690 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3518009780 tx=0x7f3518009c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:23.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.538+0000 7f3520ff9640 1 -- 172.21.15.169:0/799174961 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3518004220 con 0x7f352410b1b0 2024-03-28T01:06:23.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.538+0000 7f3520ff9640 1 -- 172.21.15.169:0/799174961 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35180043c0 con 0x7f352410b1b0 2024-03-28T01:06:23.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.539+0000 7f3520ff9640 1 -- 172.21.15.169:0/799174961 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3518003a90 con 0x7f352410b1b0 2024-03-28T01:06:23.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.543+0000 7f3528bc5640 1 -- 172.21.15.169:0/799174961 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f352419db60 con 0x7f352410b1b0 2024-03-28T01:06:23.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.543+0000 7f3528bc5640 1 -- 172.21.15.169:0/799174961 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f352419e060 con 0x7f352410b1b0 2024-03-28T01:06:23.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.551+0000 7f352ca12740 1 Processor -- start 2024-03-28T01:06:23.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.551+0000 7f3520ff9640 1 -- 172.21.15.169:0/799174961 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f351800ec20 con 0x7f352410b1b0 2024-03-28T01:06:23.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.552+0000 7f352ca12740 1 -- start start 2024-03-28T01:06:23.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.553+0000 7f352ca12740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf8bc0 0x5557f0cf8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:23.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.553+0000 7f352ca12740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5557f0cf95b0 0x5557f0cfe0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:23.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.553+0000 7f352ca12740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cfe720 0x5557f0d16d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:23.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.553+0000 7f352ca12740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557f0cc0610 con 0x5557f0cf95b0 2024-03-28T01:06:23.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.553+0000 7f352ca12740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557f0c633c0 con 0x5557f0cf8bc0 2024-03-28T01:06:23.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.553+0000 7f352ca12740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557f0c0bd60 con 0x5557f0cfe720 2024-03-28T01:06:23.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.554+0000 7f35237fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5557f0cf95b0 0x5557f0cfe0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:23.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.554+0000 7f35237fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5557f0cf95b0 0x5557f0cfe0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:55882/0 (socket says 172.21.15.169:55882) 2024-03-28T01:06:23.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.554+0000 7f35237fe640 1 -- 172.21.15.169:0/3622276228 learned_addr learned my addr 172.21.15.169:0/3622276228 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:23.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.554+0000 7f3523fff640 1 --2- 172.21.15.169:0/3622276228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf8bc0 0x5557f0cf8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:23.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.554+0000 7f3522ffd640 1 --2- 172.21.15.169:0/3622276228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cfe720 0x5557f0d16d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:23.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.555+0000 7f3523fff640 1 -- 172.21.15.169:0/3622276228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cfe720 msgr2=0x5557f0d16d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:23.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.555+0000 7f3523fff640 1 --2- 172.21.15.169:0/3622276228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cfe720 0x5557f0d16d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.555+0000 7f3523fff640 1 -- 172.21.15.169:0/3622276228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5557f0cf95b0 msgr2=0x5557f0cfe0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:23.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.555+0000 7f3523fff640 1 --2- 172.21.15.169:0/3622276228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5557f0cf95b0 0x5557f0cfe0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.555+0000 7f3523fff640 1 -- 172.21.15.169:0/3622276228 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557f09773f0 con 0x5557f0cf8bc0 2024-03-28T01:06:23.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.555+0000 7f3520ff9640 1 --2- 172.21.15.169:0/799174961 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f350c0a28d0 0x7f350c0a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:23.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.555+0000 7f3520ff9640 1 -- 172.21.15.169:0/799174961 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f35180d25f0 con 0x7f352410b1b0 2024-03-28T01:06:23.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.556+0000 7f3523fff640 1 --2- 172.21.15.169:0/3622276228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf8bc0 0x5557f0cf8fe0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f351c009d20 tx=0x7f351c00d6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97ff236bfee2e8ac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:23.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.559+0000 7f3509ffb640 1 -- 172.21.15.169:0/3622276228 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351c003970 con 0x5557f0cf8bc0 2024-03-28T01:06:23.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.559+0000 7f3509ffb640 1 -- 172.21.15.169:0/3622276228 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f351c004030 con 0x5557f0cf8bc0 2024-03-28T01:06:23.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.561+0000 7f352ca12740 1 -- 172.21.15.169:0/3622276228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf8bc0 msgr2=0x5557f0cf8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:23.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.561+0000 7f352ca12740 1 --2- 172.21.15.169:0/3622276228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf8bc0 0x5557f0cf8fe0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f351c009d20 tx=0x7f351c00d6b0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.563+0000 7f352ca12740 1 -- 172.21.15.169:0/3622276228 shutdown_connections 2024-03-28T01:06:23.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.563+0000 7f352ca12740 1 --2- 172.21.15.169:0/3622276228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cfe720 0x5557f0d16d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.563+0000 7f352ca12740 1 --2- 172.21.15.169:0/3622276228 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5557f0cf95b0 0x5557f0cfe0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.563+0000 7f352ca12740 1 --2- 172.21.15.169:0/3622276228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf8bc0 0x5557f0cf8fe0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.563+0000 7f352ca12740 1 -- 172.21.15.169:0/3622276228 >> 172.21.15.169:0/3622276228 conn(0x5557f0d015e0 msgr2=0x5557f0d019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:23.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.568+0000 7f35237fe640 1 --2- 172.21.15.169:0/799174961 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f350c0a28d0 0x7f350c0a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:23.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.569+0000 7f352ca12740 1 -- 172.21.15.169:0/3622276228 shutdown_connections 2024-03-28T01:06:23.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.569+0000 7f352ca12740 1 -- 172.21.15.169:0/3622276228 wait complete. 2024-03-28T01:06:23.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.569+0000 7f352ca12740 1 -- start start 2024-03-28T01:06:23.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.569+0000 7f352ca12740 1 Processor -- start 2024-03-28T01:06:23.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.570+0000 7f352ca12740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cf8bc0 0x5557f0d07c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:23.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.570+0000 7f352ca12740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf95b0 0x5557f0d08140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:23.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.570+0000 7f352ca12740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5557f0cfe720 0x5557f0d0f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:23.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.570+0000 7f352ca12740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557f0c1f240 con 0x5557f0cfe720 2024-03-28T01:06:23.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.570+0000 7f352ca12740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557f0c5f460 con 0x5557f0cf95b0 2024-03-28T01:06:23.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.570+0000 7f352ca12740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557f0d05ef0 con 0x5557f0cf8bc0 2024-03-28T01:06:23.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.570+0000 7f352ca12740 10 client.0 _ll_register_callbacks cb 0x5557f0c1ec10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:06:23.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.570+0000 7f3522ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf95b0 0x5557f0d08140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:23.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.570+0000 7f3522ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf95b0 0x5557f0d08140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:38298/0 (socket says 172.21.15.169:38298) 2024-03-28T01:06:23.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.570+0000 7f3522ffd640 1 -- 172.21.15.169:0/2727488432 learned_addr learned my addr 172.21.15.169:0/2727488432 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:23.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.571+0000 7f35237fe640 1 --2- 172.21.15.169:0/2727488432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cf8bc0 0x5557f0d07c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:23.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.571+0000 7f3522ffd640 1 -- 172.21.15.169:0/2727488432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cf8bc0 msgr2=0x5557f0d07c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:23.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.571+0000 7f3522ffd640 1 --2- 172.21.15.169:0/2727488432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cf8bc0 0x5557f0d07c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.571+0000 7f3522ffd640 1 -- 172.21.15.169:0/2727488432 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5557f0cfe720 msgr2=0x5557f0d0f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:23.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.571+0000 7f3522ffd640 1 --2- 172.21.15.169:0/2727488432 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5557f0cfe720 0x5557f0d0f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:23.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.571+0000 7f3522ffd640 1 -- 172.21.15.169:0/2727488432 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5557f0a417d0 con 0x5557f0cf95b0 2024-03-28T01:06:23.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.572+0000 7f3522ffd640 1 --2- 172.21.15.169:0/2727488432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf95b0 0x5557f0d08140 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f351809aff0 tx=0x7f351809ea20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3b59079db6dfd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:23.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.579+0000 7f35077f6640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:06:23.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.580+0000 7f35077f6640 1 -- 172.21.15.169:0/2727488432 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351809eb90 con 0x5557f0cf95b0 2024-03-28T01:06:23.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.580+0000 7f352ca12740 1 -- 172.21.15.169:0/2727488432 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f351c004030 con 0x5557f0cf95b0 2024-03-28T01:06:23.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.580+0000 7f35077f6640 1 -- 172.21.15.169:0/2727488432 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f351809ce90 con 0x5557f0cf95b0 2024-03-28T01:06:23.584 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.580+0000 7f35077f6640 1 -- 172.21.15.169:0/2727488432 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35180a2ce0 con 0x5557f0cf95b0 2024-03-28T01:06:23.584 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.581+0000 7f352ca12740 10 client.27203 fetch_fsmap learned FSMap version 25 2024-03-28T01:06:23.584 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.581+0000 7f352ca12740 1 -- 172.21.15.169:0/2727488432 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5557f0d125b0 con 0x5557f0cf95b0 2024-03-28T01:06:23.584 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.581+0000 7f35077f6640 1 -- 172.21.15.169:0/2727488432 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f351c004030 con 0x5557f0cf95b0 2024-03-28T01:06:23.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.583+0000 7f35077f6640 1 -- 172.21.15.169:0/2727488432 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f35180a6ea0 con 0x5557f0cf95b0 2024-03-28T01:06:23.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.583+0000 7f352ca12740 10 client.27203 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:06:23.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.583+0000 7f352ca12740 10 client.27203 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:06:23.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.583+0000 7f352ca12740 20 client.27203 populate_metadata read hostname 'smithi169' 2024-03-28T01:06:23.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.583+0000 7f352ca12740 1 --2- 172.21.15.169:0/2727488432 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5557f0d140d0 0x5557f0d16550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:23.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.583+0000 7f352ca12740 4 client.27203 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:06:23.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.583+0000 7f352ca12740 1 -- 172.21.15.169:0/2727488432 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5557f0c70900 con 0x5557f0d140d0 2024-03-28T01:06:23.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:23.590+0000 7f35237fe640 1 --2- 172.21.15.169:0/799174961 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f350c0a28d0 0x7f350c0a4d90 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x5557f0cf99b0 tx=0x7f3514009430 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:24.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:23 smithi169 ceph-mon[32274]: pgmap v647: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 771 MiB/s wr, 710 op/s 2024-03-28T01:06:24.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:23 smithi174 ceph-mon[36060]: pgmap v647: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 771 MiB/s wr, 710 op/s 2024-03-28T01:06:24.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:23 smithi177 ceph-mon[36206]: pgmap v647: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 771 MiB/s wr, 710 op/s 2024-03-28T01:06:24.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.624+0000 7f35237fe640 1 --2- 172.21.15.169:0/2727488432 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5557f0d140d0 0x5557f0d16550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:24.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.627+0000 7f35237fe640 1 --2- 172.21.15.169:0/2727488432 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5557f0d140d0 0x5557f0d16550 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddc2998a2ed5b57f server_cookie=ad7b0bfa9cb5650c in_seq=0 out_seq=0 2024-03-28T01:06:24.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.627+0000 7f35077f6640 10 client.27203 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:24.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.634+0000 7f35077f6640 1 -- 172.21.15.169:0/2727488432 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5557f0c70900 con 0x5557f0d140d0 2024-03-28T01:06:24.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.634+0000 7f35077f6640 10 client.27203 handle_command_reply: tid=0 2024-03-28T01:06:24.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.634+0000 7f352ca12740 1 client.27203 shutdown 2024-03-28T01:06:24.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.634+0000 7f352ca12740 1 -- 172.21.15.169:0/2727488432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf95b0 msgr2=0x5557f0d08140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:24.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.634+0000 7f352ca12740 1 --2- 172.21.15.169:0/2727488432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf95b0 0x5557f0d08140 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f351809aff0 tx=0x7f351809ea20 comp rx=0 tx=0).stop 2024-03-28T01:06:24.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.634+0000 7f352ca12740 1 -- 172.21.15.169:0/2727488432 shutdown_connections 2024-03-28T01:06:24.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.634+0000 7f352ca12740 1 --2- 172.21.15.169:0/2727488432 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5557f0d140d0 0x5557f0d16550 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:24.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.634+0000 7f352ca12740 1 --2- 172.21.15.169:0/2727488432 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5557f0cfe720 0x5557f0d0f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:24.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.634+0000 7f352ca12740 1 --2- 172.21.15.169:0/2727488432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf95b0 0x5557f0d08140 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:24.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.634+0000 7f352ca12740 1 --2- 172.21.15.169:0/2727488432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cf8bc0 0x5557f0d07c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:24.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.634+0000 7f352ca12740 1 -- 172.21.15.169:0/2727488432 >> 172.21.15.169:0/2727488432 conn(0x5557f0d015e0 msgr2=0x5557f0cf8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:24.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.635+0000 7f35077f6640 0 client.27203 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:24.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.635+0000 7f352ca12740 1 -- 172.21.15.169:0/2727488432 shutdown_connections 2024-03-28T01:06:24.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.635+0000 7f352ca12740 1 -- 172.21.15.169:0/2727488432 wait complete. 2024-03-28T01:06:24.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.635+0000 7f352ca12740 20 client.27203 trim_cache size 0 max 16384 2024-03-28T01:06:24.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.667+0000 7f352ca12740 1 Processor -- start 2024-03-28T01:06:24.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.667+0000 7f352ca12740 1 -- start start 2024-03-28T01:06:24.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.668+0000 7f352ca12740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cf8bc0 0x5557f0d06b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:24.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.668+0000 7f352ca12740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf95b0 0x5557f0d0f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:24.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.668+0000 7f35237fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cf8bc0 0x5557f0d06b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:24.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.668+0000 7f35237fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cf8bc0 0x5557f0d06b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:54564/0 (socket says 172.21.15.169:54564) 2024-03-28T01:06:24.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.668+0000 7f3522ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf95b0 0x5557f0d0f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:24.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.668+0000 7f3522ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf95b0 0x5557f0d0f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:38314/0 (socket says 172.21.15.169:38314) 2024-03-28T01:06:24.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.668+0000 7f35237fe640 1 -- 172.21.15.169:0/1956258919 learned_addr learned my addr 172.21.15.169:0/1956258919 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:24.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.669+0000 7f352ca12740 1 --2- 172.21.15.169:0/1956258919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5557f0cfe720 0x5557f0d04f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:24.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.669+0000 7f352ca12740 1 -- 172.21.15.169:0/1956258919 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35241b2bc0 con 0x5557f0cfe720 2024-03-28T01:06:24.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.669+0000 7f352ca12740 1 -- 172.21.15.169:0/1956258919 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35241b06c0 con 0x5557f0cf95b0 2024-03-28T01:06:24.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.669+0000 7f352ca12740 1 -- 172.21.15.169:0/1956258919 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35241ae810 con 0x5557f0cf8bc0 2024-03-28T01:06:24.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.669+0000 7f3523fff640 1 --2- 172.21.15.169:0/1956258919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5557f0cfe720 0x5557f0d04f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:24.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.670+0000 7f3522ffd640 1 -- 172.21.15.169:0/1956258919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cf8bc0 msgr2=0x5557f0d06b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:24.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.670+0000 7f3522ffd640 1 --2- 172.21.15.169:0/1956258919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cf8bc0 0x5557f0d06b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:24.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.670+0000 7f3522ffd640 1 -- 172.21.15.169:0/1956258919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5557f0cfe720 msgr2=0x5557f0d04f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:24.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.670+0000 7f3522ffd640 1 --2- 172.21.15.169:0/1956258919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5557f0cfe720 0x5557f0d04f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:24.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.670+0000 7f3522ffd640 1 -- 172.21.15.169:0/1956258919 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557f0cf87f0 con 0x5557f0cf95b0 2024-03-28T01:06:24.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.671+0000 7f3522ffd640 1 --2- 172.21.15.169:0/1956258919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf95b0 0x5557f0d0f140 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f35180a6e00 tx=0x7f3518009f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56cb96cc45059c3e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:24.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.675+0000 7f35097fa640 1 -- 172.21.15.169:0/1956258919 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351809eb90 con 0x5557f0cf95b0 2024-03-28T01:06:24.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.675+0000 7f35097fa640 1 -- 172.21.15.169:0/1956258919 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35180a7330 con 0x5557f0cf95b0 2024-03-28T01:06:24.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.676+0000 7f35097fa640 1 -- 172.21.15.169:0/1956258919 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351809f030 con 0x5557f0cf95b0 2024-03-28T01:06:24.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.677+0000 7f352ca12740 1 -- 172.21.15.169:0/1956258919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf95b0 msgr2=0x5557f0d0f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:24.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.677+0000 7f352ca12740 1 --2- 172.21.15.169:0/1956258919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf95b0 0x5557f0d0f140 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f35180a6e00 tx=0x7f3518009f50 comp rx=0 tx=0).stop 2024-03-28T01:06:24.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.678+0000 7f352ca12740 1 -- 172.21.15.169:0/1956258919 shutdown_connections 2024-03-28T01:06:24.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.678+0000 7f352ca12740 1 --2- 172.21.15.169:0/1956258919 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5557f0cfe720 0x5557f0d04f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:24.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.678+0000 7f352ca12740 1 --2- 172.21.15.169:0/1956258919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf95b0 0x5557f0d0f140 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:24.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.678+0000 7f352ca12740 1 --2- 172.21.15.169:0/1956258919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cf8bc0 0x5557f0d06b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:24.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.678+0000 7f352ca12740 1 -- 172.21.15.169:0/1956258919 >> 172.21.15.169:0/1956258919 conn(0x5557f0d015e0 msgr2=0x5557f0d04840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:24.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.679+0000 7f352ca12740 1 -- 172.21.15.169:0/1956258919 shutdown_connections 2024-03-28T01:06:24.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.680+0000 7f352ca12740 1 -- 172.21.15.169:0/1956258919 wait complete. 2024-03-28T01:06:24.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.680+0000 7f352ca12740 1 -- start start 2024-03-28T01:06:24.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.680+0000 7f352ca12740 1 Processor -- start 2024-03-28T01:06:24.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.680+0000 7f352ca12740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cf8bc0 0x5557f0d0ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:24.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.680+0000 7f352ca12740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf95b0 0x5557f0d0d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:24.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.680+0000 7f352ca12740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5557f0cfe720 0x5557f0d0e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:24.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.681+0000 7f352ca12740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35241aed80 con 0x5557f0cfe720 2024-03-28T01:06:24.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.681+0000 7f352ca12740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35241b0060 con 0x5557f0cf95b0 2024-03-28T01:06:24.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.681+0000 7f352ca12740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35241af3a0 con 0x5557f0cf8bc0 2024-03-28T01:06:24.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.681+0000 7f35237fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cf8bc0 0x5557f0d0ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:24.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.681+0000 7f35237fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cf8bc0 0x5557f0d0ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:54574/0 (socket says 172.21.15.169:54574) 2024-03-28T01:06:24.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.681+0000 7f35237fe640 1 -- 172.21.15.169:0/3170209559 learned_addr learned my addr 172.21.15.169:0/3170209559 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:24.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.681+0000 7f352ca12740 10 client.0 _ll_register_callbacks cb 0x7f351809a710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:06:24.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.681+0000 7f35237fe640 1 -- 172.21.15.169:0/3170209559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf95b0 msgr2=0x5557f0d0d350 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:06:24.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.681+0000 7f3523fff640 1 --2- 172.21.15.169:0/3170209559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5557f0cfe720 0x5557f0d0e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:24.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.681+0000 7f35237fe640 1 --2- 172.21.15.169:0/3170209559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf95b0 0x5557f0d0d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:24.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.681+0000 7f35237fe640 1 -- 172.21.15.169:0/3170209559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5557f0cfe720 msgr2=0x5557f0d0e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:24.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.681+0000 7f35237fe640 1 --2- 172.21.15.169:0/3170209559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5557f0cfe720 0x5557f0d0e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:24.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.681+0000 7f35237fe640 1 -- 172.21.15.169:0/3170209559 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5557f0d0e950 con 0x5557f0cf8bc0 2024-03-28T01:06:24.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.681+0000 7f3523fff640 1 --2- 172.21.15.169:0/3170209559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5557f0cfe720 0x5557f0d0e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:06:24.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.682+0000 7f35237fe640 1 --2- 172.21.15.169:0/3170209559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cf8bc0 0x5557f0d0ce10 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f35140323d0 tx=0x7f3514034a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12fa198c775e9d3b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:24.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.682+0000 7f35077f6640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:06:24.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.682+0000 7f35077f6640 1 -- 172.21.15.169:0/3170209559 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3514004240 con 0x5557f0cf8bc0 2024-03-28T01:06:24.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.682+0000 7f35077f6640 1 -- 172.21.15.169:0/3170209559 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f35140043e0 con 0x5557f0cf8bc0 2024-03-28T01:06:24.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.682+0000 7f352ca12740 1 -- 172.21.15.169:0/3170209559 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f35180a7330 con 0x5557f0cf8bc0 2024-03-28T01:06:24.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.682+0000 7f35077f6640 1 -- 172.21.15.169:0/3170209559 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35140732f0 con 0x5557f0cf8bc0 2024-03-28T01:06:24.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.682+0000 7f352ca12740 10 client.27142 fetch_fsmap learned FSMap version 25 2024-03-28T01:06:24.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.682+0000 7f352ca12740 1 -- 172.21.15.169:0/3170209559 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5557f0d70060 con 0x5557f0cf8bc0 2024-03-28T01:06:24.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.682+0000 7f35077f6640 1 -- 172.21.15.169:0/3170209559 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35180a7330 con 0x5557f0cf8bc0 2024-03-28T01:06:24.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.683+0000 7f35077f6640 1 -- 172.21.15.169:0/3170209559 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f351402c530 con 0x5557f0cf8bc0 2024-03-28T01:06:24.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.683+0000 7f352ca12740 10 client.27142 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:06:24.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.683+0000 7f352ca12740 10 client.27142 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:06:24.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.683+0000 7f352ca12740 20 client.27142 populate_metadata read hostname 'smithi169' 2024-03-28T01:06:24.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.683+0000 7f352ca12740 1 --2- 172.21.15.169:0/3170209559 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5557f0d709f0 0x5557f0d72e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:24.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.683+0000 7f352ca12740 4 client.27142 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:06:24.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.683+0000 7f352ca12740 1 -- 172.21.15.169:0/3170209559 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5557f0c6ac50 con 0x5557f0d709f0 2024-03-28T01:06:24.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.684+0000 7f3522ffd640 1 --2- 172.21.15.169:0/3170209559 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5557f0d709f0 0x5557f0d72e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:24.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.684+0000 7f3522ffd640 1 --2- 172.21.15.169:0/3170209559 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5557f0d709f0 0x5557f0d72e10 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b185067462e9f0d server_cookie=d03244e6e7cd85d2 in_seq=0 out_seq=0 2024-03-28T01:06:24.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.684+0000 7f35077f6640 10 client.27142 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:24.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.685+0000 7f35077f6640 1 -- 172.21.15.169:0/3170209559 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5557f0c6ac50 con 0x5557f0d709f0 2024-03-28T01:06:24.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.686+0000 7f35077f6640 10 client.27142 handle_command_reply: tid=0 2024-03-28T01:06:24.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.686+0000 7f352ca12740 1 client.27142 shutdown 2024-03-28T01:06:24.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.687+0000 7f352ca12740 1 -- 172.21.15.169:0/3170209559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cf8bc0 msgr2=0x5557f0d0ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:24.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.688+0000 7f352ca12740 1 --2- 172.21.15.169:0/3170209559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cf8bc0 0x5557f0d0ce10 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f35140323d0 tx=0x7f3514034a50 comp rx=0 tx=0).stop 2024-03-28T01:06:24.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.688+0000 7f352ca12740 1 -- 172.21.15.169:0/3170209559 shutdown_connections 2024-03-28T01:06:24.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.688+0000 7f352ca12740 1 --2- 172.21.15.169:0/3170209559 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5557f0d709f0 0x5557f0d72e10 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:24.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.688+0000 7f352ca12740 1 --2- 172.21.15.169:0/3170209559 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5557f0cfe720 0x5557f0d0e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:24.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.688+0000 7f35077f6640 0 client.27142 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:24.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.688+0000 7f352ca12740 1 --2- 172.21.15.169:0/3170209559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5557f0cf95b0 0x5557f0d0d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:24.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.688+0000 7f352ca12740 1 --2- 172.21.15.169:0/3170209559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557f0cf8bc0 0x5557f0d0ce10 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:24.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.688+0000 7f352ca12740 1 -- 172.21.15.169:0/3170209559 >> 172.21.15.169:0/3170209559 conn(0x5557f0d015e0 msgr2=0x5557f0d04840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:24.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.688+0000 7f352ca12740 1 -- 172.21.15.169:0/3170209559 shutdown_connections 2024-03-28T01:06:24.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.689+0000 7f352ca12740 1 -- 172.21.15.169:0/3170209559 wait complete. 2024-03-28T01:06:24.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.689+0000 7f352ca12740 20 client.27142 trim_cache size 0 max 16384 2024-03-28T01:06:24.695 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:06:24.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.693+0000 7f3528bc5640 1 -- 172.21.15.169:0/799174961 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f350c0a28d0 msgr2=0x7f350c0a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:24.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.693+0000 7f3528bc5640 1 --2- 172.21.15.169:0/799174961 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f350c0a28d0 0x7f350c0a4d90 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x5557f0cf99b0 tx=0x7f3514009430 comp rx=0 tx=0).stop 2024-03-28T01:06:24.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.693+0000 7f3528bc5640 1 -- 172.21.15.169:0/799174961 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f352410b1b0 msgr2=0x7f35241a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:24.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.693+0000 7f3528bc5640 1 --2- 172.21.15.169:0/799174961 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f352410b1b0 0x7f35241a0690 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3518009780 tx=0x7f3518009c40 comp rx=0 tx=0).stop 2024-03-28T01:06:24.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.693+0000 7f3528bc5640 1 -- 172.21.15.169:0/799174961 shutdown_connections 2024-03-28T01:06:24.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.693+0000 7f3528bc5640 1 --2- 172.21.15.169:0/799174961 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f350c0a28d0 0x7f350c0a4d90 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:24.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.693+0000 7f3528bc5640 1 --2- 172.21.15.169:0/799174961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f352410dc30 0x7f35241a7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:24.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.693+0000 7f3528bc5640 1 --2- 172.21.15.169:0/799174961 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f352410b1b0 0x7f35241a0690 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:24.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.693+0000 7f3528bc5640 1 --2- 172.21.15.169:0/799174961 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3524108880 0x7f35241a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:24.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.693+0000 7f3528bc5640 1 -- 172.21.15.169:0/799174961 >> 172.21.15.169:0/799174961 conn(0x7f352406e580 msgr2=0x7f352410aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:24.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.694+0000 7f3528bc5640 1 -- 172.21.15.169:0/799174961 shutdown_connections 2024-03-28T01:06:24.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:24.696+0000 7f3528bc5640 1 -- 172.21.15.169:0/799174961 wait complete. 2024-03-28T01:06:25.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:24 smithi169 ceph-mon[32274]: pgmap v648: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 657 MiB/s wr, 706 op/s 2024-03-28T01:06:25.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:24 smithi174 ceph-mon[36060]: pgmap v648: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 657 MiB/s wr, 706 op/s 2024-03-28T01:06:25.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:24 smithi177 ceph-mon[36206]: pgmap v648: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 657 MiB/s wr, 706 op/s 2024-03-28T01:06:25.725 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T01:06:25.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T01:06:25.726 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T01:06:26.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.083+0000 7f488aef9640 1 -- 172.21.15.169:0/1489302797 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4884108d70 msgr2=0x7f48841091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:26.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.083+0000 7f488aef9640 1 --2- 172.21.15.169:0/1489302797 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4884108d70 0x7f48841091f0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f487401a910 tx=0x7f487403cca0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.093+0000 7f488aef9640 1 -- 172.21.15.169:0/1489302797 shutdown_connections 2024-03-28T01:06:26.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.093+0000 7f488aef9640 1 --2- 172.21.15.169:0/1489302797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4884109730 0x7f488410dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.093+0000 7f488aef9640 1 --2- 172.21.15.169:0/1489302797 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4884108d70 0x7f48841091f0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.093+0000 7f488aef9640 1 --2- 172.21.15.169:0/1489302797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4884072ab0 0x7f4884072ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.093+0000 7f488aef9640 1 -- 172.21.15.169:0/1489302797 >> 172.21.15.169:0/1489302797 conn(0x7f488406e580 msgr2=0x7f48840709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:26.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.096+0000 7f488aef9640 1 -- 172.21.15.169:0/1489302797 shutdown_connections 2024-03-28T01:06:26.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.101+0000 7f488aef9640 1 -- 172.21.15.169:0/1489302797 wait complete. 2024-03-28T01:06:26.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.101+0000 7f488aef9640 1 Processor -- start 2024-03-28T01:06:26.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.102+0000 7f488aef9640 1 -- start start 2024-03-28T01:06:26.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.102+0000 7f488aef9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4884072ab0 0x7f488419e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:26.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.102+0000 7f488aef9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4884108d70 0x7f488419e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:26.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.102+0000 7f488aef9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4884109730 0x7f48841a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:26.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.102+0000 7f488aef9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4884111ab0 con 0x7f4884072ab0 2024-03-28T01:06:26.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.102+0000 7f488aef9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4884111930 con 0x7f4884109730 2024-03-28T01:06:26.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.102+0000 7f488aef9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4884111c30 con 0x7f4884108d70 2024-03-28T01:06:26.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.103+0000 7f48896f6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4884108d70 0x7f488419e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:26.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.103+0000 7f48896f6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4884108d70 0x7f488419e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:54590/0 (socket says 172.21.15.169:54590) 2024-03-28T01:06:26.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.103+0000 7f48896f6640 1 -- 172.21.15.169:0/2565544293 learned_addr learned my addr 172.21.15.169:0/2565544293 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:26.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.104+0000 7f4889ef7640 1 --2- 172.21.15.169:0/2565544293 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4884072ab0 0x7f488419e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:26.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.104+0000 7f4889ef7640 1 -- 172.21.15.169:0/2565544293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4884108d70 msgr2=0x7f488419e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:26.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.104+0000 7f4889ef7640 1 --2- 172.21.15.169:0/2565544293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4884108d70 0x7f488419e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.104+0000 7f4889ef7640 1 -- 172.21.15.169:0/2565544293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4884109730 msgr2=0x7f48841a55c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:06:26.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.104+0000 7f4889ef7640 1 --2- 172.21.15.169:0/2565544293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4884109730 0x7f48841a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.104+0000 7f4889ef7640 1 -- 172.21.15.169:0/2565544293 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48841a5c30 con 0x7f4884072ab0 2024-03-28T01:06:26.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.105+0000 7f4889ef7640 1 --2- 172.21.15.169:0/2565544293 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4884072ab0 0x7f488419e010 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f4880009fd0 tx=0x7f488000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:26.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.110+0000 7f487affd640 1 -- 172.21.15.169:0/2565544293 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4880015020 con 0x7f4884072ab0 2024-03-28T01:06:26.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.110+0000 7f488aef9640 1 -- 172.21.15.169:0/2565544293 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48841a5f20 con 0x7f4884072ab0 2024-03-28T01:06:26.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.110+0000 7f488aef9640 1 -- 172.21.15.169:0/2565544293 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48841a6400 con 0x7f4884072ab0 2024-03-28T01:06:26.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.110+0000 7f487affd640 1 -- 172.21.15.169:0/2565544293 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4880011d10 con 0x7f4884072ab0 2024-03-28T01:06:26.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.110+0000 7f487affd640 1 -- 172.21.15.169:0/2565544293 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488000f2e0 con 0x7f4884072ab0 2024-03-28T01:06:26.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.118+0000 7f488ee22740 1 Processor -- start 2024-03-28T01:06:26.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.118+0000 7f487affd640 1 -- 172.21.15.169:0/2565544293 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4880018020 con 0x7f4884072ab0 2024-03-28T01:06:26.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.118+0000 7f488ee22740 1 -- start start 2024-03-28T01:06:26.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.119+0000 7f488ee22740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561ef109bb30 0x561ef109bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:26.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.119+0000 7f488ee22740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109c520 0x561ef10a1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:26.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.119+0000 7f488ee22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ef10a1690 0x561ef10a3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:26.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.119+0000 7f488ee22740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef0fdbc90 con 0x561ef109bb30 2024-03-28T01:06:26.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.119+0000 7f488ee22740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef1001070 con 0x561ef109c520 2024-03-28T01:06:26.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.119+0000 7f488ee22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef0fa9c80 con 0x561ef10a1690 2024-03-28T01:06:26.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.119+0000 7f487affd640 1 --2- 172.21.15.169:0/2565544293 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f48700a2a00 0x7f48700a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:26.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.120+0000 7f487affd640 1 -- 172.21.15.169:0/2565544293 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f48800ce5a0 con 0x7f4884072ab0 2024-03-28T01:06:26.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.120+0000 7f488a6f8640 1 --2- 172.21.15.169:0/2565544293 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f48700a2a00 0x7f48700a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:26.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.120+0000 7f488a6f8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561ef109bb30 0x561ef109bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:26.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.120+0000 7f488a6f8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561ef109bb30 0x561ef109bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:55940/0 (socket says 172.21.15.169:55940) 2024-03-28T01:06:26.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.120+0000 7f488a6f8640 1 -- 172.21.15.169:0/320178562 learned_addr learned my addr 172.21.15.169:0/320178562 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:26.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.120+0000 7f4889ef7640 1 --2- 172.21.15.169:0/320178562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109c520 0x561ef10a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:26.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.121+0000 7f488a6f8640 1 -- 172.21.15.169:0/320178562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ef10a1690 msgr2=0x561ef10a3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:26.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.121+0000 7f488a6f8640 1 --2- 172.21.15.169:0/320178562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ef10a1690 0x561ef10a3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.121+0000 7f488a6f8640 1 -- 172.21.15.169:0/320178562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109c520 msgr2=0x561ef10a1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:26.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.121+0000 7f488a6f8640 1 --2- 172.21.15.169:0/320178562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109c520 0x561ef10a1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.121+0000 7f488a6f8640 1 -- 172.21.15.169:0/320178562 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ef0fa1d10 con 0x561ef109bb30 2024-03-28T01:06:26.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.121+0000 7f4889ef7640 1 --2- 172.21.15.169:0/320178562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109c520 0x561ef10a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:06:26.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.122+0000 7f488a6f8640 1 --2- 172.21.15.169:0/2565544293 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f48700a2a00 0x7f48700a4ec0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f487c00cef0 tx=0x7f487c013820 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:26.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.126+0000 7f488a6f8640 1 --2- 172.21.15.169:0/320178562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561ef109bb30 0x561ef109bf50 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f487c013d20 tx=0x7f487c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14f4608e9c0d2bd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:26.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.132+0000 7f486ffff640 1 -- 172.21.15.169:0/320178562 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c020070 con 0x561ef109bb30 2024-03-28T01:06:26.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.132+0000 7f486ffff640 1 -- 172.21.15.169:0/320178562 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f487c006660 con 0x561ef109bb30 2024-03-28T01:06:26.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.132+0000 7f486ffff640 1 -- 172.21.15.169:0/320178562 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c0117d0 con 0x561ef109bb30 2024-03-28T01:06:26.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.153+0000 7f488ee22740 1 -- 172.21.15.169:0/320178562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561ef109bb30 msgr2=0x561ef109bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:26.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.153+0000 7f488ee22740 1 --2- 172.21.15.169:0/320178562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561ef109bb30 0x561ef109bf50 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f487c013d20 tx=0x7f487c01a000 comp rx=0 tx=0).stop 2024-03-28T01:06:26.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.157+0000 7f488ee22740 1 -- 172.21.15.169:0/320178562 shutdown_connections 2024-03-28T01:06:26.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.157+0000 7f488ee22740 1 --2- 172.21.15.169:0/320178562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ef10a1690 0x561ef10a3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.157+0000 7f488ee22740 1 --2- 172.21.15.169:0/320178562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109c520 0x561ef10a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.157+0000 7f488ee22740 1 --2- 172.21.15.169:0/320178562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561ef109bb30 0x561ef109bf50 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.157+0000 7f488ee22740 1 -- 172.21.15.169:0/320178562 >> 172.21.15.169:0/320178562 conn(0x561ef10641a0 msgr2=0x561ef10974a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:26.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.168+0000 7f488ee22740 1 -- 172.21.15.169:0/320178562 shutdown_connections 2024-03-28T01:06:26.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.169+0000 7f488ee22740 1 -- 172.21.15.169:0/320178562 wait complete. 2024-03-28T01:06:26.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.170+0000 7f488ee22740 1 -- start start 2024-03-28T01:06:26.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.170+0000 7f488ee22740 1 Processor -- start 2024-03-28T01:06:26.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.175+0000 7f488ee22740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109bb30 0x561ef10a5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:26.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.175+0000 7f488ee22740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561ef109c520 0x561ef10a6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:26.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.175+0000 7f488ee22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ef10a1690 0x561ef10ad480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:26.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.175+0000 7f488ee22740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef0fbd0d0 con 0x561ef109c520 2024-03-28T01:06:26.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.175+0000 7f488ee22740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef0ffd110 con 0x561ef109bb30 2024-03-28T01:06:26.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.175+0000 7f488ee22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef10a45e0 con 0x561ef10a1690 2024-03-28T01:06:26.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.175+0000 7f488ee22740 10 client.0 _ll_register_callbacks cb 0x561ef0ff8e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:06:26.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.176+0000 7f4889ef7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109bb30 0x561ef10a5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:26.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.176+0000 7f4889ef7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109bb30 0x561ef10a5f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:38348/0 (socket says 172.21.15.169:38348) 2024-03-28T01:06:26.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.176+0000 7f4889ef7640 1 -- 172.21.15.169:0/4126342408 learned_addr learned my addr 172.21.15.169:0/4126342408 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:26.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.177+0000 7f4889ef7640 1 -- 172.21.15.169:0/4126342408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ef10a1690 msgr2=0x561ef10ad480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:06:26.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.177+0000 7f4889ef7640 1 --2- 172.21.15.169:0/4126342408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ef10a1690 0x561ef10ad480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.177+0000 7f4889ef7640 1 -- 172.21.15.169:0/4126342408 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561ef109c520 msgr2=0x561ef10a6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:26.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.177+0000 7f4889ef7640 1 --2- 172.21.15.169:0/4126342408 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561ef109c520 0x561ef10a6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.177+0000 7f4889ef7640 1 -- 172.21.15.169:0/4126342408 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ef0ddd7e0 con 0x561ef109bb30 2024-03-28T01:06:26.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.178+0000 7f4889ef7640 1 --2- 172.21.15.169:0/4126342408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109bb30 0x561ef10a5f00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4880096fa0 tx=0x7f488009b370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbb4df39d086027f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:26.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.178+0000 7f486d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:06:26.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.178+0000 7f486d7fa640 1 -- 172.21.15.169:0/4126342408 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48800b2070 con 0x561ef109bb30 2024-03-28T01:06:26.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.179+0000 7f486d7fa640 1 -- 172.21.15.169:0/4126342408 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f48800ce240 con 0x561ef109bb30 2024-03-28T01:06:26.182 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.179+0000 7f488ee22740 1 -- 172.21.15.169:0/4126342408 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f487c006660 con 0x561ef109bb30 2024-03-28T01:06:26.182 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.179+0000 7f486d7fa640 1 -- 172.21.15.169:0/4126342408 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48800a32e0 con 0x561ef109bb30 2024-03-28T01:06:26.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.182+0000 7f488ee22740 10 client.27212 fetch_fsmap learned FSMap version 25 2024-03-28T01:06:26.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.182+0000 7f488ee22740 1 -- 172.21.15.169:0/4126342408 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561ef0e3df70 con 0x561ef109bb30 2024-03-28T01:06:26.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.183+0000 7f486d7fa640 1 -- 172.21.15.169:0/4126342408 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f487c006660 con 0x561ef109bb30 2024-03-28T01:06:26.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.184+0000 7f486d7fa640 1 -- 172.21.15.169:0/4126342408 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f48800a2e50 con 0x561ef109bb30 2024-03-28T01:06:26.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.184+0000 7f488ee22740 10 client.27212 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:06:26.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.184+0000 7f488ee22740 10 client.27212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:06:26.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.184+0000 7f488ee22740 20 client.27212 populate_metadata read hostname 'smithi169' 2024-03-28T01:06:26.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.184+0000 7f488ee22740 1 --2- 172.21.15.169:0/4126342408 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561ef10b2150 0x561ef10b45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:26.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.184+0000 7f488ee22740 4 client.27212 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:06:26.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.184+0000 7f488ee22740 1 -- 172.21.15.169:0/4126342408 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561ef100e5b0 con 0x561ef10b2150 2024-03-28T01:06:26.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.185+0000 7f48896f6640 1 --2- 172.21.15.169:0/4126342408 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561ef10b2150 0x561ef10b45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:26.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.186+0000 7f48896f6640 1 --2- 172.21.15.169:0/4126342408 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561ef10b2150 0x561ef10b45f0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b0a6133a0be126c server_cookie=a2ef70bc1fe97eaa in_seq=0 out_seq=0 2024-03-28T01:06:26.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.186+0000 7f486d7fa640 10 client.27212 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:26.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.190+0000 7f486d7fa640 1 -- 172.21.15.169:0/4126342408 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561ef100e5b0 con 0x561ef10b2150 2024-03-28T01:06:26.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.190+0000 7f486d7fa640 10 client.27212 handle_command_reply: tid=0 2024-03-28T01:06:26.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.190+0000 7f488ee22740 1 client.27212 shutdown 2024-03-28T01:06:26.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.195+0000 7f488ee22740 1 -- 172.21.15.169:0/4126342408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109bb30 msgr2=0x561ef10a5f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:26.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.195+0000 7f488ee22740 1 --2- 172.21.15.169:0/4126342408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109bb30 0x561ef10a5f00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4880096fa0 tx=0x7f488009b370 comp rx=0 tx=0).stop 2024-03-28T01:06:26.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.203+0000 7f488ee22740 1 -- 172.21.15.169:0/4126342408 shutdown_connections 2024-03-28T01:06:26.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.203+0000 7f488ee22740 1 --2- 172.21.15.169:0/4126342408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ef10a1690 0x561ef10ad480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.203+0000 7f488ee22740 1 --2- 172.21.15.169:0/4126342408 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561ef10b2150 0x561ef10b45f0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.203+0000 7f488ee22740 1 --2- 172.21.15.169:0/4126342408 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561ef109c520 0x561ef10a6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.203+0000 7f488ee22740 1 --2- 172.21.15.169:0/4126342408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109bb30 0x561ef10a5f00 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.203+0000 7f488ee22740 1 -- 172.21.15.169:0/4126342408 >> 172.21.15.169:0/4126342408 conn(0x561ef10641a0 msgr2=0x561ef10967e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:26.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.203+0000 7f486d7fa640 0 client.27212 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:26.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.209+0000 7f488ee22740 1 -- 172.21.15.169:0/4126342408 shutdown_connections 2024-03-28T01:06:26.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.210+0000 7f488ee22740 1 -- 172.21.15.169:0/4126342408 wait complete. 2024-03-28T01:06:26.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.214+0000 7f488ee22740 20 client.27212 trim_cache size 0 max 16384 2024-03-28T01:06:26.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.263+0000 7f488ee22740 1 Processor -- start 2024-03-28T01:06:26.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.264+0000 7f488ee22740 1 -- start start 2024-03-28T01:06:26.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.265+0000 7f488ee22740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109bb30 0x561ef109a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:26.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.265+0000 7f488ee22740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561ef109c520 0x561ef10ad440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:26.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.265+0000 7f488ee22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ef10a1690 0x561ef109ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:26.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.265+0000 7f488ee22740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4868011e70 con 0x561ef109c520 2024-03-28T01:06:26.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.265+0000 7f488ee22740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486800f970 con 0x561ef109bb30 2024-03-28T01:06:26.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.265+0000 7f488ee22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486800dac0 con 0x561ef10a1690 2024-03-28T01:06:26.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.266+0000 7f4889ef7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109bb30 0x561ef109a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:26.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.266+0000 7f4889ef7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109bb30 0x561ef109a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:38354/0 (socket says 172.21.15.169:38354) 2024-03-28T01:06:26.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.266+0000 7f4889ef7640 1 -- 172.21.15.169:0/751167060 learned_addr learned my addr 172.21.15.169:0/751167060 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:26.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.267+0000 7f488a6f8640 1 --2- 172.21.15.169:0/751167060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ef10a1690 0x561ef109ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:26.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.268+0000 7f48896f6640 1 --2- 172.21.15.169:0/751167060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561ef109c520 0x561ef10ad440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:26.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.270+0000 7f4889ef7640 1 -- 172.21.15.169:0/751167060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ef10a1690 msgr2=0x561ef109ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:26.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.270+0000 7f4889ef7640 1 --2- 172.21.15.169:0/751167060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ef10a1690 0x561ef109ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.270+0000 7f4889ef7640 1 -- 172.21.15.169:0/751167060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561ef109c520 msgr2=0x561ef10ad440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:26.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.270+0000 7f4889ef7640 1 --2- 172.21.15.169:0/751167060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561ef109c520 0x561ef10ad440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.270+0000 7f4889ef7640 1 -- 172.21.15.169:0/751167060 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ef0cd5170 con 0x561ef109bb30 2024-03-28T01:06:26.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.271+0000 7f4889ef7640 1 --2- 172.21.15.169:0/751167060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109bb30 0x561ef109a6e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f48800a5040 tx=0x7f4880009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f3213be249598d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:26.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.272+0000 7f486f7fe640 1 -- 172.21.15.169:0/751167060 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48800b2070 con 0x561ef109bb30 2024-03-28T01:06:26.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.272+0000 7f486f7fe640 1 -- 172.21.15.169:0/751167060 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48800aa020 con 0x561ef109bb30 2024-03-28T01:06:26.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.273+0000 7f488ee22740 1 -- 172.21.15.169:0/751167060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109bb30 msgr2=0x561ef109a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:26.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.273+0000 7f488ee22740 1 --2- 172.21.15.169:0/751167060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109bb30 0x561ef109a6e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f48800a5040 tx=0x7f4880009260 comp rx=0 tx=0).stop 2024-03-28T01:06:26.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.274+0000 7f488ee22740 1 -- 172.21.15.169:0/751167060 shutdown_connections 2024-03-28T01:06:26.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.274+0000 7f488ee22740 1 --2- 172.21.15.169:0/751167060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ef10a1690 0x561ef109ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.274+0000 7f488ee22740 1 --2- 172.21.15.169:0/751167060 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561ef109c520 0x561ef10ad440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.274+0000 7f488ee22740 1 --2- 172.21.15.169:0/751167060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109bb30 0x561ef109a6e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.274+0000 7f488ee22740 1 -- 172.21.15.169:0/751167060 >> 172.21.15.169:0/751167060 conn(0x561ef10641a0 msgr2=0x561ef109dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:26.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.276+0000 7f488ee22740 1 -- 172.21.15.169:0/751167060 shutdown_connections 2024-03-28T01:06:26.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.280+0000 7f488ee22740 1 -- 172.21.15.169:0/751167060 wait complete. 2024-03-28T01:06:26.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.280+0000 7f488ee22740 1 -- start start 2024-03-28T01:06:26.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.280+0000 7f488ee22740 1 Processor -- start 2024-03-28T01:06:26.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.283+0000 7f488ee22740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109bb30 0x561ef10aae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:26.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.283+0000 7f488ee22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ef109c520 0x561ef10ab3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:26.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.283+0000 7f488ee22740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561ef10a1690 0x561ef10ac3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:26.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.283+0000 7f488ee22740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486800e030 con 0x561ef10a1690 2024-03-28T01:06:26.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.283+0000 7f488ee22740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486800f310 con 0x561ef109bb30 2024-03-28T01:06:26.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.283+0000 7f488ee22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486800e650 con 0x561ef109c520 2024-03-28T01:06:26.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.283+0000 7f488ee22740 10 client.0 _ll_register_callbacks cb 0x7f488009a680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:06:26.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.285+0000 7f48896f6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ef109c520 0x561ef10ab3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:26.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.288+0000 7f48896f6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ef109c520 0x561ef10ab3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:54634/0 (socket says 172.21.15.169:54634) 2024-03-28T01:06:26.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.288+0000 7f48896f6640 1 -- 172.21.15.169:0/1894907815 learned_addr learned my addr 172.21.15.169:0/1894907815 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:26.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.289+0000 7f4889ef7640 1 --2- 172.21.15.169:0/1894907815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109bb30 0x561ef10aae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:26.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.289+0000 7f488a6f8640 1 --2- 172.21.15.169:0/1894907815 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561ef10a1690 0x561ef10ac3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:26.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.294+0000 7f48896f6640 1 -- 172.21.15.169:0/1894907815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109bb30 msgr2=0x561ef10aae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:26.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.294+0000 7f48896f6640 1 --2- 172.21.15.169:0/1894907815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109bb30 0x561ef10aae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.294+0000 7f48896f6640 1 -- 172.21.15.169:0/1894907815 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561ef10a1690 msgr2=0x561ef10ac3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:26.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.294+0000 7f48896f6640 1 --2- 172.21.15.169:0/1894907815 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561ef10a1690 0x561ef10ac3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:26.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.294+0000 7f48896f6640 1 -- 172.21.15.169:0/1894907815 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ef10ac9b0 con 0x561ef109c520 2024-03-28T01:06:26.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.297+0000 7f48896f6640 1 --2- 172.21.15.169:0/1894907815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ef109c520 0x561ef10ab3b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4874005ab0 tx=0x7f4874004e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2095dfbe1cd20f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:26.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.297+0000 7f486d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:06:26.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.298+0000 7f486d7fa640 1 -- 172.21.15.169:0/1894907815 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4874004310 con 0x561ef109c520 2024-03-28T01:06:26.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.299+0000 7f486d7fa640 1 -- 172.21.15.169:0/1894907815 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f487403d850 con 0x561ef109c520 2024-03-28T01:06:26.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.304+0000 7f488ee22740 1 -- 172.21.15.169:0/1894907815 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561ef1008900 con 0x561ef109c520 2024-03-28T01:06:26.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.305+0000 7f486d7fa640 1 -- 172.21.15.169:0/1894907815 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48740044b0 con 0x561ef109c520 2024-03-28T01:06:26.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.306+0000 7f488ee22740 10 client.27151 fetch_fsmap learned FSMap version 25 2024-03-28T01:06:26.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.306+0000 7f488ee22740 1 -- 172.21.15.169:0/1894907815 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561ef110def0 con 0x561ef109c520 2024-03-28T01:06:26.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.307+0000 7f486d7fa640 1 -- 172.21.15.169:0/1894907815 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x561ef1008900 con 0x561ef109c520 2024-03-28T01:06:26.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.308+0000 7f486d7fa640 1 -- 172.21.15.169:0/1894907815 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f4874061030 con 0x561ef109c520 2024-03-28T01:06:26.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.308+0000 7f488ee22740 10 client.27151 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:06:26.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.308+0000 7f488ee22740 10 client.27151 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:06:26.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.308+0000 7f488ee22740 20 client.27151 populate_metadata read hostname 'smithi169' 2024-03-28T01:06:26.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.308+0000 7f488ee22740 1 --2- 172.21.15.169:0/1894907815 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561ef110e8d0 0x561ef1110cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:26.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.308+0000 7f488ee22740 4 client.27151 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T01:06:26.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.308+0000 7f488ee22740 1 -- 172.21.15.169:0/1894907815 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x561ef0f86b80 con 0x561ef110e8d0 2024-03-28T01:06:26.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.311+0000 7f4889ef7640 1 --2- 172.21.15.169:0/1894907815 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561ef110e8d0 0x561ef1110cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:26.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.344+0000 7f4889ef7640 1 --2- 172.21.15.169:0/1894907815 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561ef110e8d0 0x561ef1110cf0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d4c68c411b33602 server_cookie=8f4c30b668f5d6c9 in_seq=0 out_seq=0 2024-03-28T01:06:26.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:26.345+0000 7f486d7fa640 10 client.27151 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:27.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.130+0000 7f486d7fa640 1 -- 172.21.15.169:0/1894907815 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x561ef0f86b80 con 0x561ef110e8d0 2024-03-28T01:06:27.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.130+0000 7f486d7fa640 10 client.27151 handle_command_reply: tid=0 2024-03-28T01:06:27.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.130+0000 7f488ee22740 1 client.27151 shutdown 2024-03-28T01:06:27.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.130+0000 7f488ee22740 1 -- 172.21.15.169:0/1894907815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ef109c520 msgr2=0x561ef10ab3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:27.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.130+0000 7f488ee22740 1 --2- 172.21.15.169:0/1894907815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ef109c520 0x561ef10ab3b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4874005ab0 tx=0x7f4874004e40 comp rx=0 tx=0).stop 2024-03-28T01:06:27.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.130+0000 7f488ee22740 1 -- 172.21.15.169:0/1894907815 shutdown_connections 2024-03-28T01:06:27.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.130+0000 7f488ee22740 1 --2- 172.21.15.169:0/1894907815 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561ef110e8d0 0x561ef1110cf0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.130+0000 7f488ee22740 1 --2- 172.21.15.169:0/1894907815 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561ef10a1690 0x561ef10ac3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.130+0000 7f488ee22740 1 --2- 172.21.15.169:0/1894907815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ef109c520 0x561ef10ab3b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.130+0000 7f488ee22740 1 --2- 172.21.15.169:0/1894907815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ef109bb30 0x561ef10aae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.131+0000 7f488ee22740 1 -- 172.21.15.169:0/1894907815 >> 172.21.15.169:0/1894907815 conn(0x561ef10641a0 msgr2=0x561ef109a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:27.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.131+0000 7f486d7fa640 0 client.27151 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:27.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.131+0000 7f488ee22740 1 -- 172.21.15.169:0/1894907815 shutdown_connections 2024-03-28T01:06:27.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.131+0000 7f488ee22740 1 -- 172.21.15.169:0/1894907815 wait complete. 2024-03-28T01:06:27.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.131+0000 7f488ee22740 20 client.27151 trim_cache size 0 max 16384 2024-03-28T01:06:27.136 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:06:27.136 INFO:teuthology.orchestra.run.smithi169.stdout: "return_code": 0, 2024-03-28T01:06:27.136 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_tag": "c3e8db4a-a3e6-4a6d-a6cd-d79ef89e7230", 2024-03-28T01:06:27.136 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": "asynchronous" 2024-03-28T01:06:27.136 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:06:27.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.134+0000 7f488aef9640 1 -- 172.21.15.169:0/2565544293 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f48700a2a00 msgr2=0x7f48700a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:27.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.134+0000 7f488aef9640 1 --2- 172.21.15.169:0/2565544293 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f48700a2a00 0x7f48700a4ec0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f487c00cef0 tx=0x7f487c013820 comp rx=0 tx=0).stop 2024-03-28T01:06:27.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.134+0000 7f488aef9640 1 -- 172.21.15.169:0/2565544293 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4884072ab0 msgr2=0x7f488419e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:27.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.134+0000 7f488aef9640 1 --2- 172.21.15.169:0/2565544293 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4884072ab0 0x7f488419e010 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f4880009fd0 tx=0x7f488000b870 comp rx=0 tx=0).stop 2024-03-28T01:06:27.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.135+0000 7f488aef9640 1 -- 172.21.15.169:0/2565544293 shutdown_connections 2024-03-28T01:06:27.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.135+0000 7f488aef9640 1 --2- 172.21.15.169:0/2565544293 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f48700a2a00 0x7f48700a4ec0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.135+0000 7f488aef9640 1 --2- 172.21.15.169:0/2565544293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4884109730 0x7f48841a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.135+0000 7f488aef9640 1 --2- 172.21.15.169:0/2565544293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4884108d70 0x7f488419e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.135+0000 7f488aef9640 1 --2- 172.21.15.169:0/2565544293 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4884072ab0 0x7f488419e010 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.135+0000 7f488aef9640 1 -- 172.21.15.169:0/2565544293 >> 172.21.15.169:0/2565544293 conn(0x7f488406e580 msgr2=0x7f488410bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:27.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.136+0000 7f488aef9640 1 -- 172.21.15.169:0/2565544293 shutdown_connections 2024-03-28T01:06:27.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.136+0000 7f488aef9640 1 -- 172.21.15.169:0/2565544293 wait complete. 2024-03-28T01:06:27.182 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:06:27.182 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:06:27.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.541+0000 7f82e1250640 1 -- 172.21.15.169:0/2237967528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82dc109750 msgr2=0x7f82dc10ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:27.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.542+0000 7f82e1250640 1 --2- 172.21.15.169:0/2237967528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82dc109750 0x7f82dc10ffe0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009ca0 tx=0x7f82d402c120 comp rx=0 tx=0).stop 2024-03-28T01:06:27.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.552+0000 7f82e1250640 1 -- 172.21.15.169:0/2237967528 shutdown_connections 2024-03-28T01:06:27.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.552+0000 7f82e1250640 1 --2- 172.21.15.169:0/2237967528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82dc109750 0x7f82dc10ffe0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.552+0000 7f82e1250640 1 --2- 172.21.15.169:0/2237967528 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82dc108d90 0x7f82dc109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.552+0000 7f82e1250640 1 --2- 172.21.15.169:0/2237967528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82dc107ba0 0x7f82dc107fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.552+0000 7f82e1250640 1 -- 172.21.15.169:0/2237967528 >> 172.21.15.169:0/2237967528 conn(0x7f82dc076b40 msgr2=0x7f82dc078f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:27.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.554+0000 7f82e1250640 1 -- 172.21.15.169:0/2237967528 shutdown_connections 2024-03-28T01:06:27.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.556+0000 7f82e1250640 1 -- 172.21.15.169:0/2237967528 wait complete. 2024-03-28T01:06:27.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.558+0000 7f82e1250640 1 Processor -- start 2024-03-28T01:06:27.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.560+0000 7f82e1250640 1 -- start start 2024-03-28T01:06:27.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.561+0000 7f82e1250640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82dc107ba0 0x7f82dc19e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:27.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.561+0000 7f82e1250640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82dc108d90 0x7f82dc19e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:27.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.561+0000 7f82e1250640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82dc109750 0x7f82dc1a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:27.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.561+0000 7f82e1250640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc113910 con 0x7f82dc107ba0 2024-03-28T01:06:27.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.561+0000 7f82e1250640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc113790 con 0x7f82dc108d90 2024-03-28T01:06:27.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.561+0000 7f82e1250640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc113a90 con 0x7f82dc109750 2024-03-28T01:06:27.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.561+0000 7f82dbfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82dc107ba0 0x7f82dc19e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:27.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.561+0000 7f82dbfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82dc107ba0 0x7f82dc19e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34388/0 (socket says 172.21.15.169:34388) 2024-03-28T01:06:27.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.561+0000 7f82dbfff640 1 -- 172.21.15.169:0/3795086344 learned_addr learned my addr 172.21.15.169:0/3795086344 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:27.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.561+0000 7f82e0a4f640 1 --2- 172.21.15.169:0/3795086344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82dc109750 0x7f82dc1a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:27.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.562+0000 7f82e0a4f640 1 -- 172.21.15.169:0/3795086344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82dc108d90 msgr2=0x7f82dc19e5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:06:27.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.562+0000 7f82e0a4f640 1 --2- 172.21.15.169:0/3795086344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82dc108d90 0x7f82dc19e5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.562+0000 7f82e0a4f640 1 -- 172.21.15.169:0/3795086344 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82dc107ba0 msgr2=0x7f82dc19e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:27.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.562+0000 7f82e0a4f640 1 --2- 172.21.15.169:0/3795086344 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82dc107ba0 0x7f82dc19e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.562+0000 7f82e0a4f640 1 -- 172.21.15.169:0/3795086344 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82dc1a5c30 con 0x7f82dc109750 2024-03-28T01:06:27.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.562+0000 7f82e0a4f640 1 --2- 172.21.15.169:0/3795086344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82dc109750 0x7f82dc1a55c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009bf0 tx=0x7f82d4034e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:27.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.563+0000 7f82d97fa640 1 -- 172.21.15.169:0/3795086344 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d4041030 con 0x7f82dc109750 2024-03-28T01:06:27.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.563+0000 7f82d97fa640 1 -- 172.21.15.169:0/3795086344 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82d4003970 con 0x7f82dc109750 2024-03-28T01:06:27.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.563+0000 7f82d97fa640 1 -- 172.21.15.169:0/3795086344 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d4039040 con 0x7f82dc109750 2024-03-28T01:06:27.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.563+0000 7f82e1250640 1 -- 172.21.15.169:0/3795086344 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82dc1a5ec0 con 0x7f82dc109750 2024-03-28T01:06:27.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.563+0000 7f82e1250640 1 -- 172.21.15.169:0/3795086344 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82dc1a63b0 con 0x7f82dc109750 2024-03-28T01:06:27.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.569+0000 7f82e517b740 1 Processor -- start 2024-03-28T01:06:27.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.569+0000 7f82d97fa640 1 -- 172.21.15.169:0/3795086344 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f82d4044070 con 0x7f82dc109750 2024-03-28T01:06:27.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.570+0000 7f82d97fa640 1 --2- 172.21.15.169:0/3795086344 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f82c40a2980 0x7f82c40a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:27.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.572+0000 7f82d97fa640 1 -- 172.21.15.169:0/3795086344 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f82d4033e60 con 0x7f82dc109750 2024-03-28T01:06:27.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.573+0000 7f82e517b740 1 -- start start 2024-03-28T01:06:27.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.573+0000 7f82e517b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901acfdf80 0x55901acfe3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:27.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.574+0000 7f82e517b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55901acfe970 0x55901ad03470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:27.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.574+0000 7f82e517b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901ad03ae0 0x55901ad05f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:27.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.574+0000 7f82e517b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55901ac633d0 con 0x55901acfe970 2024-03-28T01:06:27.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.574+0000 7f82e517b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55901ac0bad0 con 0x55901ad03ae0 2024-03-28T01:06:27.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.574+0000 7f82e517b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55901ac5f470 con 0x55901acfdf80 2024-03-28T01:06:27.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.574+0000 7f82dbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901acfdf80 0x55901acfe3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:27.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.574+0000 7f82dbfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901acfdf80 0x55901acfe3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:42364/0 (socket says 172.21.15.169:42364) 2024-03-28T01:06:27.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.574+0000 7f82dbfff640 1 -- 172.21.15.169:0/2616114273 learned_addr learned my addr 172.21.15.169:0/2616114273 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:27.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.574+0000 7f82e0a4f640 1 --2- 172.21.15.169:0/2616114273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901ad03ae0 0x55901ad05f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:27.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.574+0000 7f82dbfff640 1 -- 172.21.15.169:0/2616114273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901ad03ae0 msgr2=0x55901ad05f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:27.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.574+0000 7f82dbfff640 1 --2- 172.21.15.169:0/2616114273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901ad03ae0 0x55901ad05f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.574+0000 7f82dbfff640 1 -- 172.21.15.169:0/2616114273 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55901acfe970 msgr2=0x55901ad03470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:27.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.574+0000 7f82dbfff640 1 --2- 172.21.15.169:0/2616114273 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55901acfe970 0x55901ad03470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.574+0000 7f82dbfff640 1 -- 172.21.15.169:0/2616114273 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55901aa417d0 con 0x55901acfdf80 2024-03-28T01:06:27.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.574+0000 7f82e0a4f640 1 --2- 172.21.15.169:0/2616114273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901ad03ae0 0x55901ad05f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:06:27.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.574+0000 7f82dbfff640 1 --2- 172.21.15.169:0/2616114273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901acfdf80 0x55901acfe3a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f82cc007810 tx=0x7f82cc007cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2074aac91548ec1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:27.584 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.579+0000 7f82db7fe640 1 --2- 172.21.15.169:0/3795086344 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f82c40a2980 0x7f82c40a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:27.584 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.580+0000 7f82db7fe640 1 --2- 172.21.15.169:0/3795086344 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f82c40a2980 0x7f82c40a4e40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x55901acfed70 tx=0x7f82d0007ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:27.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.583+0000 7f82c27fc640 1 -- 172.21.15.169:0/2616114273 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc00d950 con 0x55901acfdf80 2024-03-28T01:06:27.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.583+0000 7f82c27fc640 1 -- 172.21.15.169:0/2616114273 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82cc018070 con 0x55901acfdf80 2024-03-28T01:06:27.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.583+0000 7f82c27fc640 1 -- 172.21.15.169:0/2616114273 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc0123e0 con 0x55901acfdf80 2024-03-28T01:06:27.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.585+0000 7f82e517b740 1 -- 172.21.15.169:0/2616114273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901acfdf80 msgr2=0x55901acfe3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:27.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.585+0000 7f82e517b740 1 --2- 172.21.15.169:0/2616114273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901acfdf80 0x55901acfe3a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f82cc007810 tx=0x7f82cc007cd0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.592+0000 7f82e517b740 1 -- 172.21.15.169:0/2616114273 shutdown_connections 2024-03-28T01:06:27.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.592+0000 7f82e517b740 1 --2- 172.21.15.169:0/2616114273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901ad03ae0 0x55901ad05f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.592+0000 7f82e517b740 1 --2- 172.21.15.169:0/2616114273 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55901acfe970 0x55901ad03470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.592+0000 7f82e517b740 1 --2- 172.21.15.169:0/2616114273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901acfdf80 0x55901acfe3a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.592+0000 7f82e517b740 1 -- 172.21.15.169:0/2616114273 >> 172.21.15.169:0/2616114273 conn(0x55901acf94f0 msgr2=0x55901acf98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:27.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.592+0000 7f82e517b740 1 -- 172.21.15.169:0/2616114273 shutdown_connections 2024-03-28T01:06:27.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.596+0000 7f82e517b740 1 -- 172.21.15.169:0/2616114273 wait complete. 2024-03-28T01:06:27.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.596+0000 7f82e517b740 1 -- start start 2024-03-28T01:06:27.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.596+0000 7f82e517b740 1 Processor -- start 2024-03-28T01:06:27.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.597+0000 7f82e517b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55901acfdf80 0x55901ad08460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:27.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.597+0000 7f82e517b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901acfe970 0x55901ad089a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:27.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.597+0000 7f82e517b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901ad03ae0 0x55901ad0f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:27.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.597+0000 7f82e517b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55901ad06a90 con 0x55901acfdf80 2024-03-28T01:06:27.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.597+0000 7f82e517b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55901ac1f060 con 0x55901ad03ae0 2024-03-28T01:06:27.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.597+0000 7f82e517b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55901ad06c10 con 0x55901acfe970 2024-03-28T01:06:27.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.597+0000 7f82e517b740 10 client.0 _ll_register_callbacks cb 0x55901ac016a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:06:27.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.598+0000 7f82e0a4f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901ad03ae0 0x55901ad0f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:27.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.598+0000 7f82dbfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55901acfdf80 0x55901ad08460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:27.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.598+0000 7f82e0a4f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901ad03ae0 0x55901ad0f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:39348/0 (socket says 172.21.15.169:39348) 2024-03-28T01:06:27.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.598+0000 7f82e0a4f640 1 -- 172.21.15.169:0/1798585356 learned_addr learned my addr 172.21.15.169:0/1798585356 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:27.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.598+0000 7f82e0a4f640 1 -- 172.21.15.169:0/1798585356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901acfe970 msgr2=0x55901ad089a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:06:27.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.598+0000 7f82e0a4f640 1 --2- 172.21.15.169:0/1798585356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901acfe970 0x55901ad089a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.598+0000 7f82e0a4f640 1 -- 172.21.15.169:0/1798585356 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55901acfdf80 msgr2=0x55901ad08460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:27.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.598+0000 7f82e0a4f640 1 --2- 172.21.15.169:0/1798585356 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55901acfdf80 0x55901ad08460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.598+0000 7f82e0a4f640 1 -- 172.21.15.169:0/1798585356 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55901a9773f0 con 0x55901ad03ae0 2024-03-28T01:06:27.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.599+0000 7f82e0a4f640 1 --2- 172.21.15.169:0/1798585356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901ad03ae0 0x55901ad0f9e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f82d40ba250 tx=0x7f82d40b7ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8805f85adfd270ba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:27.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.612+0000 7f82bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:06:27.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.612+0000 7f82bb7fe640 1 -- 172.21.15.169:0/1798585356 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d40b9a00 con 0x55901ad03ae0 2024-03-28T01:06:27.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.612+0000 7f82e517b740 1 -- 172.21.15.169:0/1798585356 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f82cc018070 con 0x55901ad03ae0 2024-03-28T01:06:27.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.612+0000 7f82bb7fe640 1 -- 172.21.15.169:0/1798585356 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f82d4003b20 con 0x55901ad03ae0 2024-03-28T01:06:27.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.612+0000 7f82bb7fe640 1 -- 172.21.15.169:0/1798585356 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82d40b9ba0 con 0x55901ad03ae0 2024-03-28T01:06:27.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.624+0000 7f82e517b740 10 client.27224 fetch_fsmap learned FSMap version 25 2024-03-28T01:06:27.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.624+0000 7f82e517b740 1 -- 172.21.15.169:0/1798585356 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55901a9370f0 con 0x55901ad03ae0 2024-03-28T01:06:27.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.624+0000 7f82bb7fe640 1 -- 172.21.15.169:0/1798585356 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82cc018070 con 0x55901ad03ae0 2024-03-28T01:06:27.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.624+0000 7f82bb7fe640 1 -- 172.21.15.169:0/1798585356 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f82d403c030 con 0x55901ad03ae0 2024-03-28T01:06:27.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.624+0000 7f82e517b740 10 client.27224 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:06:27.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.624+0000 7f82e517b740 10 client.27224 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:06:27.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.624+0000 7f82e517b740 20 client.27224 populate_metadata read hostname 'smithi169' 2024-03-28T01:06:27.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.625+0000 7f82e517b740 1 --2- 172.21.15.169:0/1798585356 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55901ad24d20 0x55901ad27160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:27.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.625+0000 7f82e517b740 4 client.27224 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:06:27.628 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.625+0000 7f82e517b740 1 -- 172.21.15.169:0/1798585356 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55901ac70910 con 0x55901ad24d20 2024-03-28T01:06:27.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.625+0000 7f82dbfff640 1 --2- 172.21.15.169:0/1798585356 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55901ad24d20 0x55901ad27160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:27.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.626+0000 7f82dbfff640 1 --2- 172.21.15.169:0/1798585356 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55901ad24d20 0x55901ad27160 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23367e4f1063975 server_cookie=698f0aa235cbad59 in_seq=0 out_seq=0 2024-03-28T01:06:27.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.626+0000 7f82bb7fe640 10 client.27224 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:27.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.628+0000 7f82bb7fe640 1 -- 172.21.15.169:0/1798585356 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55901ac70910 con 0x55901ad24d20 2024-03-28T01:06:27.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.628+0000 7f82bb7fe640 10 client.27224 handle_command_reply: tid=0 2024-03-28T01:06:27.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.628+0000 7f82e517b740 1 client.27224 shutdown 2024-03-28T01:06:27.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.629+0000 7f82e517b740 1 -- 172.21.15.169:0/1798585356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901ad03ae0 msgr2=0x55901ad0f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:27.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.629+0000 7f82e517b740 1 --2- 172.21.15.169:0/1798585356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901ad03ae0 0x55901ad0f9e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f82d40ba250 tx=0x7f82d40b7ad0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.629+0000 7f82e517b740 1 -- 172.21.15.169:0/1798585356 shutdown_connections 2024-03-28T01:06:27.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.629+0000 7f82e517b740 1 --2- 172.21.15.169:0/1798585356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901ad03ae0 0x55901ad0f9e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.629+0000 7f82e517b740 1 --2- 172.21.15.169:0/1798585356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901acfe970 0x55901ad089a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.629+0000 7f82e517b740 1 --2- 172.21.15.169:0/1798585356 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55901ad24d20 0x55901ad27160 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.629+0000 7f82e517b740 1 --2- 172.21.15.169:0/1798585356 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55901acfdf80 0x55901ad08460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.629+0000 7f82e517b740 1 -- 172.21.15.169:0/1798585356 >> 172.21.15.169:0/1798585356 conn(0x55901acf94f0 msgr2=0x55901acf87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:27.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.629+0000 7f82bb7fe640 0 client.27224 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:27.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.632+0000 7f82e517b740 1 -- 172.21.15.169:0/1798585356 shutdown_connections 2024-03-28T01:06:27.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.639+0000 7f82e517b740 1 -- 172.21.15.169:0/1798585356 wait complete. 2024-03-28T01:06:27.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.640+0000 7f82e517b740 20 client.27224 trim_cache size 0 max 16384 2024-03-28T01:06:27.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.728+0000 7f82e517b740 1 Processor -- start 2024-03-28T01:06:27.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.728+0000 7f82e517b740 1 -- start start 2024-03-28T01:06:27.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.728+0000 7f82e517b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901acfdf80 0x55901acfce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:27.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.728+0000 7f82e517b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55901acfe970 0x55901ad0f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:27.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.728+0000 7f82e517b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901ad03ae0 0x55901ad2ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:27.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.729+0000 7f82e517b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc1aa460 con 0x55901acfe970 2024-03-28T01:06:27.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.729+0000 7f82e517b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc1a7f60 con 0x55901acfdf80 2024-03-28T01:06:27.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.729+0000 7f82e517b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc0732e0 con 0x55901ad03ae0 2024-03-28T01:06:27.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.729+0000 7f82dbfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901acfdf80 0x55901acfce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:27.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.729+0000 7f82dbfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901acfdf80 0x55901acfce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:39364/0 (socket says 172.21.15.169:39364) 2024-03-28T01:06:27.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.729+0000 7f82dbfff640 1 -- 172.21.15.169:0/319615520 learned_addr learned my addr 172.21.15.169:0/319615520 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:27.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.729+0000 7f82e0a4f640 1 --2- 172.21.15.169:0/319615520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901ad03ae0 0x55901ad2ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:27.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.730+0000 7f82dbfff640 1 -- 172.21.15.169:0/319615520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901ad03ae0 msgr2=0x55901ad2ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:27.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.730+0000 7f82dbfff640 1 --2- 172.21.15.169:0/319615520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901ad03ae0 0x55901ad2ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.730+0000 7f82dbfff640 1 -- 172.21.15.169:0/319615520 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55901acfe970 msgr2=0x55901ad0f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:27.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.730+0000 7f82dbfff640 1 --2- 172.21.15.169:0/319615520 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55901acfe970 0x55901ad0f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.730+0000 7f82dbfff640 1 -- 172.21.15.169:0/319615520 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55901acf9f60 con 0x55901acfdf80 2024-03-28T01:06:27.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.730+0000 7f82dbfff640 1 --2- 172.21.15.169:0/319615520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901acfdf80 0x55901acfce60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f82cc007760 tx=0x7f82cc008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a7b89ca1f3a39eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:27.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.738+0000 7f82c1ffb640 1 -- 172.21.15.169:0/319615520 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc0252d0 con 0x55901acfdf80 2024-03-28T01:06:27.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.738+0000 7f82c1ffb640 1 -- 172.21.15.169:0/319615520 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82cc02a070 con 0x55901acfdf80 2024-03-28T01:06:27.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.741+0000 7f82c1ffb640 1 -- 172.21.15.169:0/319615520 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc0257f0 con 0x55901acfdf80 2024-03-28T01:06:27.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.742+0000 7f82e517b740 1 -- 172.21.15.169:0/319615520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901acfdf80 msgr2=0x55901acfce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:27.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.743+0000 7f82e517b740 1 --2- 172.21.15.169:0/319615520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901acfdf80 0x55901acfce60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f82cc007760 tx=0x7f82cc008040 comp rx=0 tx=0).stop 2024-03-28T01:06:27.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.743+0000 7f82e517b740 1 -- 172.21.15.169:0/319615520 shutdown_connections 2024-03-28T01:06:27.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.743+0000 7f82e517b740 1 --2- 172.21.15.169:0/319615520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901ad03ae0 0x55901ad2ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.743+0000 7f82e517b740 1 --2- 172.21.15.169:0/319615520 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55901acfe970 0x55901ad0f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.743+0000 7f82e517b740 1 --2- 172.21.15.169:0/319615520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901acfdf80 0x55901acfce60 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.743+0000 7f82e517b740 1 -- 172.21.15.169:0/319615520 >> 172.21.15.169:0/319615520 conn(0x55901acf94f0 msgr2=0x55901acfc750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:27.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.743+0000 7f82e517b740 1 -- 172.21.15.169:0/319615520 shutdown_connections 2024-03-28T01:06:27.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.743+0000 7f82e517b740 1 -- 172.21.15.169:0/319615520 wait complete. 2024-03-28T01:06:27.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.744+0000 7f82e517b740 1 -- start start 2024-03-28T01:06:27.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.744+0000 7f82e517b740 1 Processor -- start 2024-03-28T01:06:27.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.745+0000 7f82e517b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55901acfdf80 0x55901ad0d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:27.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.745+0000 7f82e517b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901acfe970 0x55901ad0dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:27.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.745+0000 7f82e517b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901ad03ae0 0x55901ad0eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:27.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.745+0000 7f82e517b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc1a6630 con 0x55901acfdf80 2024-03-28T01:06:27.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.745+0000 7f82db7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901acfe970 0x55901ad0dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:27.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.745+0000 7f82db7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901acfe970 0x55901ad0dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:42390/0 (socket says 172.21.15.169:42390) 2024-03-28T01:06:27.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.745+0000 7f82db7fe640 1 -- 172.21.15.169:0/2560499991 learned_addr learned my addr 172.21.15.169:0/2560499991 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:27.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.745+0000 7f82e0a4f640 1 --2- 172.21.15.169:0/2560499991 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901ad03ae0 0x55901ad0eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:27.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.745+0000 7f82e517b740 1 -- 172.21.15.169:0/2560499991 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc1a7900 con 0x55901ad03ae0 2024-03-28T01:06:27.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.745+0000 7f82e517b740 1 -- 172.21.15.169:0/2560499991 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc1a6c40 con 0x55901acfe970 2024-03-28T01:06:27.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.746+0000 7f82e517b740 10 client.0 _ll_register_callbacks cb 0x7f82d4033db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:06:27.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.746+0000 7f82e0a4f640 1 -- 172.21.15.169:0/2560499991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901acfe970 msgr2=0x55901ad0dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:27.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.746+0000 7f82e0a4f640 1 --2- 172.21.15.169:0/2560499991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901acfe970 0x55901ad0dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.746+0000 7f82e0a4f640 1 -- 172.21.15.169:0/2560499991 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55901acfdf80 msgr2=0x55901ad0d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:27.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.746+0000 7f82e0a4f640 1 --2- 172.21.15.169:0/2560499991 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55901acfdf80 0x55901ad0d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.746+0000 7f82e0a4f640 1 -- 172.21.15.169:0/2560499991 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55901ad0f290 con 0x55901ad03ae0 2024-03-28T01:06:27.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.747+0000 7f82e0a4f640 1 --2- 172.21.15.169:0/2560499991 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901ad03ae0 0x55901ad0eca0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f82d40b9ee0 tx=0x7f82d40c5770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5e8491dd291f265 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:27.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.753+0000 7f82bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:06:27.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.753+0000 7f82bb7fe640 1 -- 172.21.15.169:0/2560499991 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d40b9d40 con 0x55901ad03ae0 2024-03-28T01:06:27.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.753+0000 7f82bb7fe640 1 -- 172.21.15.169:0/2560499991 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f82d4004650 con 0x55901ad03ae0 2024-03-28T01:06:27.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.753+0000 7f82e517b740 1 -- 172.21.15.169:0/2560499991 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f82cc02a070 con 0x55901ad03ae0 2024-03-28T01:06:27.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.757+0000 7f82bb7fe640 1 -- 172.21.15.169:0/2560499991 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82d40d6050 con 0x55901ad03ae0 2024-03-28T01:06:27.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.764+0000 7f82e517b740 10 client.27230 fetch_fsmap learned FSMap version 25 2024-03-28T01:06:27.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.764+0000 7f82e517b740 1 -- 172.21.15.169:0/2560499991 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55901ad0f290 con 0x55901ad03ae0 2024-03-28T01:06:27.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.765+0000 7f82bb7fe640 1 -- 172.21.15.169:0/2560499991 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82cc02a070 con 0x55901ad03ae0 2024-03-28T01:06:27.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.767+0000 7f82bb7fe640 1 -- 172.21.15.169:0/2560499991 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f82d40bc930 con 0x55901ad03ae0 2024-03-28T01:06:27.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.767+0000 7f82e517b740 10 client.27230 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:06:27.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.767+0000 7f82e517b740 10 client.27230 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:06:27.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.767+0000 7f82e517b740 20 client.27230 populate_metadata read hostname 'smithi169' 2024-03-28T01:06:27.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.767+0000 7f82e517b740 1 --2- 172.21.15.169:0/2560499991 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55901ad81280 0x55901ad836a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:27.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.767+0000 7f82e517b740 4 client.27230 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:06:27.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.767+0000 7f82e517b740 1 -- 172.21.15.169:0/2560499991 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55901ad08250 con 0x55901ad81280 2024-03-28T01:06:27.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.768+0000 7f82dbfff640 1 --2- 172.21.15.169:0/2560499991 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55901ad81280 0x55901ad836a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:27.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.769+0000 7f82bb7fe640 10 client.27230 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:27.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.769+0000 7f82dbfff640 1 --2- 172.21.15.169:0/2560499991 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55901ad81280 0x55901ad836a0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9164f41fb01d1049 server_cookie=30c0cc77405cf65a in_seq=0 out_seq=0 2024-03-28T01:06:27.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.770+0000 7f82bb7fe640 1 -- 172.21.15.169:0/2560499991 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x55901ad08250 con 0x55901ad81280 2024-03-28T01:06:27.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.770+0000 7f82bb7fe640 10 client.27230 handle_command_reply: tid=0 2024-03-28T01:06:27.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.770+0000 7f82e517b740 1 client.27230 shutdown 2024-03-28T01:06:27.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.771+0000 7f82e517b740 1 -- 172.21.15.169:0/2560499991 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901ad03ae0 msgr2=0x55901ad0eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:27.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.771+0000 7f82e517b740 1 --2- 172.21.15.169:0/2560499991 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901ad03ae0 0x55901ad0eca0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f82d40b9ee0 tx=0x7f82d40c5770 comp rx=0 tx=0).stop 2024-03-28T01:06:27.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.771+0000 7f82e517b740 1 -- 172.21.15.169:0/2560499991 shutdown_connections 2024-03-28T01:06:27.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.771+0000 7f82e517b740 1 --2- 172.21.15.169:0/2560499991 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55901ad03ae0 0x55901ad0eca0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.771+0000 7f82e517b740 1 --2- 172.21.15.169:0/2560499991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55901acfe970 0x55901ad0dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.771+0000 7f82e517b740 1 --2- 172.21.15.169:0/2560499991 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55901ad81280 0x55901ad836a0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.771+0000 7f82e517b740 1 --2- 172.21.15.169:0/2560499991 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55901acfdf80 0x55901ad0d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.771+0000 7f82bb7fe640 0 client.27230 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:27.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.771+0000 7f82e517b740 1 -- 172.21.15.169:0/2560499991 >> 172.21.15.169:0/2560499991 conn(0x55901acf94f0 msgr2=0x55901acfcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:27.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.771+0000 7f82e517b740 1 -- 172.21.15.169:0/2560499991 shutdown_connections 2024-03-28T01:06:27.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.771+0000 7f82e517b740 1 -- 172.21.15.169:0/2560499991 wait complete. 2024-03-28T01:06:27.776 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.772+0000 7f82e517b740 20 client.27230 trim_cache size 0 max 16384 2024-03-28T01:06:27.783 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:06:27.783 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "scrub active (0 inodes in the stack)", 2024-03-28T01:06:27.783 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": { 2024-03-28T01:06:27.783 INFO:teuthology.orchestra.run.smithi169.stdout: "c3e8db4a-a3e6-4a6d-a6cd-d79ef89e7230": { 2024-03-28T01:06:27.783 INFO:teuthology.orchestra.run.smithi169.stdout: "path": "/", 2024-03-28T01:06:27.783 INFO:teuthology.orchestra.run.smithi169.stdout: "tag": "c3e8db4a-a3e6-4a6d-a6cd-d79ef89e7230", 2024-03-28T01:06:27.783 INFO:teuthology.orchestra.run.smithi169.stdout: "options": "recursive,force" 2024-03-28T01:06:27.783 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:06:27.783 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:06:27.783 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:06:27.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.784+0000 7f82e1250640 1 -- 172.21.15.169:0/3795086344 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f82c40a2980 msgr2=0x7f82c40a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:27.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.784+0000 7f82e1250640 1 --2- 172.21.15.169:0/3795086344 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f82c40a2980 0x7f82c40a4e40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x55901acfed70 tx=0x7f82d0007ab0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.784+0000 7f82e1250640 1 -- 172.21.15.169:0/3795086344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82dc109750 msgr2=0x7f82dc1a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:27.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.784+0000 7f82e1250640 1 --2- 172.21.15.169:0/3795086344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82dc109750 0x7f82dc1a55c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009bf0 tx=0x7f82d4034e40 comp rx=0 tx=0).stop 2024-03-28T01:06:27.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.788+0000 7f82e1250640 1 -- 172.21.15.169:0/3795086344 shutdown_connections 2024-03-28T01:06:27.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.788+0000 7f82e1250640 1 --2- 172.21.15.169:0/3795086344 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f82c40a2980 0x7f82c40a4e40 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.788+0000 7f82e1250640 1 --2- 172.21.15.169:0/3795086344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82dc109750 0x7f82dc1a55c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.788+0000 7f82e1250640 1 --2- 172.21.15.169:0/3795086344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82dc108d90 0x7f82dc19e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.788+0000 7f82e1250640 1 --2- 172.21.15.169:0/3795086344 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f82dc107ba0 0x7f82dc19e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:27.791 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.788+0000 7f82e1250640 1 -- 172.21.15.169:0/3795086344 >> 172.21.15.169:0/3795086344 conn(0x7f82dc076b40 msgr2=0x7f82dc10c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:27.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.793+0000 7f82e1250640 1 -- 172.21.15.169:0/3795086344 shutdown_connections 2024-03-28T01:06:27.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:27.795+0000 7f82e1250640 1 -- 172.21.15.169:0/3795086344 wait complete. 2024-03-28T01:06:27.842 INFO:tasks.cephfs.filesystem:scrub status for tag:c3e8db4a-a3e6-4a6d-a6cd-d79ef89e7230 - {'path': '/', 'tag': 'c3e8db4a-a3e6-4a6d-a6cd-d79ef89e7230', 'options': 'recursive,force'} 2024-03-28T01:06:28.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:27 smithi169 ceph-mon[32274]: pgmap v649: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 695 MiB/s wr, 790 op/s 2024-03-28T01:06:28.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:27 smithi169 ceph-mon[32274]: scrub queued for path: / 2024-03-28T01:06:28.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:27 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:06:28.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:27 smithi169 ceph-mon[32274]: scrub summary: active paths [/] 2024-03-28T01:06:28.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:27 smithi174 ceph-mon[36060]: pgmap v649: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 695 MiB/s wr, 790 op/s 2024-03-28T01:06:28.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:27 smithi174 ceph-mon[36060]: scrub queued for path: / 2024-03-28T01:06:28.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:27 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:06:28.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:27 smithi174 ceph-mon[36060]: scrub summary: active paths [/] 2024-03-28T01:06:28.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:27 smithi177 ceph-mon[36206]: pgmap v649: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 695 MiB/s wr, 790 op/s 2024-03-28T01:06:28.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:27 smithi177 ceph-mon[36206]: scrub queued for path: / 2024-03-28T01:06:28.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:27 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:06:28.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:27 smithi177 ceph-mon[36206]: scrub summary: active paths [/] 2024-03-28T01:06:30.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:29 smithi169 ceph-mon[32274]: pgmap v650: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 633 MiB/s wr, 764 op/s 2024-03-28T01:06:30.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:29 smithi174 ceph-mon[36060]: pgmap v650: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 633 MiB/s wr, 764 op/s 2024-03-28T01:06:30.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:29 smithi177 ceph-mon[36206]: pgmap v650: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 633 MiB/s wr, 764 op/s 2024-03-28T01:06:32.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:31 smithi169 ceph-mon[32274]: pgmap v651: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 652 MiB/s wr, 783 op/s 2024-03-28T01:06:32.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:31 smithi174 ceph-mon[36060]: pgmap v651: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 652 MiB/s wr, 783 op/s 2024-03-28T01:06:32.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:31 smithi177 ceph-mon[36206]: pgmap v651: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 652 MiB/s wr, 783 op/s 2024-03-28T01:06:33.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:32 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:06:33.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:32 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:06:33.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:32 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:06:34.053 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:33 smithi169 ceph-mon[32274]: pgmap v652: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 746 MiB/s wr, 848 op/s 2024-03-28T01:06:34.098 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.093+0000 7f72e20d1640 1 -- 172.21.15.169:0/2250027884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72dc1086a0 msgr2=0x7f72dc10aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:34.098 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.093+0000 7f72e20d1640 1 --2- 172.21.15.169:0/2250027884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72dc1086a0 0x7f72dc10aad0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009900 tx=0x7f72cc02bc60 comp rx=0 tx=0).stop 2024-03-28T01:06:34.098 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.093+0000 7f72e20d1640 1 -- 172.21.15.169:0/2250027884 shutdown_connections 2024-03-28T01:06:34.098 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.093+0000 7f72e20d1640 1 --2- 172.21.15.169:0/2250027884 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72dc10d9c0 0x7f72dc10de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.098 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.093+0000 7f72e20d1640 1 --2- 172.21.15.169:0/2250027884 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f72dc10b010 0x7f72dc10d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.098 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.093+0000 7f72e20d1640 1 --2- 172.21.15.169:0/2250027884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72dc1086a0 0x7f72dc10aad0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.098 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.093+0000 7f72e20d1640 1 -- 172.21.15.169:0/2250027884 >> 172.21.15.169:0/2250027884 conn(0x7f72dc06e6e0 msgr2=0x7f72dc070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.096+0000 7f72e20d1640 1 -- 172.21.15.169:0/2250027884 shutdown_connections 2024-03-28T01:06:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.096+0000 7f72e20d1640 1 -- 172.21.15.169:0/2250027884 wait complete. 2024-03-28T01:06:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.097+0000 7f72e20d1640 1 Processor -- start 2024-03-28T01:06:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.098+0000 7f72e20d1640 1 -- start start 2024-03-28T01:06:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.098+0000 7f72e20d1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f72dc1086a0 0x7f72dc118040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:34.102 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.098+0000 7f72e20d1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72dc10b010 0x7f72dc114a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:34.102 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.098+0000 7f72e20d1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72dc10d9c0 0x7f72dc1150b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.098+0000 7f72e20d1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc11c190 con 0x7f72dc1086a0 2024-03-28T01:06:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.098+0000 7f72e20d1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc11c010 con 0x7f72dc10d9c0 2024-03-28T01:06:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.098+0000 7f72e20d1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc11c310 con 0x7f72dc10b010 2024-03-28T01:06:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.099+0000 7f72db7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f72dc1086a0 0x7f72dc118040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.099+0000 7f72db7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f72dc1086a0 0x7f72dc118040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34428/0 (socket says 172.21.15.169:34428) 2024-03-28T01:06:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.099+0000 7f72db7fe640 1 -- 172.21.15.169:0/2609638375 learned_addr learned my addr 172.21.15.169:0/2609638375 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.100+0000 7f72daffd640 1 --2- 172.21.15.169:0/2609638375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72dc10b010 0x7f72dc114a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:34.107 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.102+0000 7f72db7fe640 1 -- 172.21.15.169:0/2609638375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72dc10b010 msgr2=0x7f72dc114a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:34.107 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.102+0000 7f72db7fe640 1 --2- 172.21.15.169:0/2609638375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72dc10b010 0x7f72dc114a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.107 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.102+0000 7f72db7fe640 1 -- 172.21.15.169:0/2609638375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72dc10d9c0 msgr2=0x7f72dc1150b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:06:34.107 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.102+0000 7f72db7fe640 1 --2- 172.21.15.169:0/2609638375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72dc10d9c0 0x7f72dc1150b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.108 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.102+0000 7f72db7fe640 1 -- 172.21.15.169:0/2609638375 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72dc1a5920 con 0x7f72dc1086a0 2024-03-28T01:06:34.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:33 smithi174 ceph-mon[36060]: pgmap v652: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 746 MiB/s wr, 848 op/s 2024-03-28T01:06:34.109 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.106+0000 7f72db7fe640 1 --2- 172.21.15.169:0/2609638375 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f72dc1086a0 0x7f72dc118040 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009850 tx=0x7f72cc02ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.107+0000 7f72d8ff9640 1 -- 172.21.15.169:0/2609638375 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc041020 con 0x7f72dc1086a0 2024-03-28T01:06:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.107+0000 7f72e20d1640 1 -- 172.21.15.169:0/2609638375 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72dc1a5bb0 con 0x7f72dc1086a0 2024-03-28T01:06:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.107+0000 7f72e20d1640 1 -- 172.21.15.169:0/2609638375 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72dc1a60a0 con 0x7f72dc1086a0 2024-03-28T01:06:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.108+0000 7f72d8ff9640 1 -- 172.21.15.169:0/2609638375 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72cc02e2e0 con 0x7f72dc1086a0 2024-03-28T01:06:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.108+0000 7f72d8ff9640 1 -- 172.21.15.169:0/2609638375 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc005130 con 0x7f72dc1086a0 2024-03-28T01:06:34.117 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.114+0000 7f72d8ff9640 1 -- 172.21.15.169:0/2609638375 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f72cc044020 con 0x7f72dc1086a0 2024-03-28T01:06:34.121 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.118+0000 7f72d8ff9640 1 --2- 172.21.15.169:0/2609638375 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f72c40a2ae0 0x7f72c40a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:34.122 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.119+0000 7f72d8ff9640 1 -- 172.21.15.169:0/2609638375 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f72cc0f1970 con 0x7f72dc1086a0 2024-03-28T01:06:34.122 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.119+0000 7f72e4d96740 1 Processor -- start 2024-03-28T01:06:34.126 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.124+0000 7f72daffd640 1 --2- 172.21.15.169:0/2609638375 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f72c40a2ae0 0x7f72c40a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:34.129 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.126+0000 7f72e4d96740 1 -- start start 2024-03-28T01:06:34.130 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.126+0000 7f72e4d96740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa89d97920 0x55aa89d97d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:34.130 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.126+0000 7f72e4d96740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d98310 0x55aa89d9ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:34.130 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.126+0000 7f72e4d96740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa89d9d480 0x55aa89d9f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:34.130 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.126+0000 7f72e4d96740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa89cfcb10 con 0x55aa89d9d480 2024-03-28T01:06:34.130 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.126+0000 7f72e4d96740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa89ca5900 con 0x55aa89d97920 2024-03-28T01:06:34.130 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.126+0000 7f72e4d96740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa89cf8bb0 con 0x55aa89d98310 2024-03-28T01:06:34.134 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.131+0000 7f72dbfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa89d9d480 0x55aa89d9f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:34.134 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.131+0000 7f72dbfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa89d9d480 0x55aa89d9f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34434/0 (socket says 172.21.15.169:34434) 2024-03-28T01:06:34.134 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.131+0000 7f72dbfff640 1 -- 172.21.15.169:0/3079165368 learned_addr learned my addr 172.21.15.169:0/3079165368 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:34.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.131+0000 7f72daffd640 1 --2- 172.21.15.169:0/3079165368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d98310 0x55aa89d9ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:34.136 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.133+0000 7f72db7fe640 1 --2- 172.21.15.169:0/3079165368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa89d97920 0x55aa89d97d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:34.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.134+0000 7f72dbfff640 1 -- 172.21.15.169:0/3079165368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d98310 msgr2=0x55aa89d9ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:34.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.134+0000 7f72dbfff640 1 --2- 172.21.15.169:0/3079165368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d98310 0x55aa89d9ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.134+0000 7f72dbfff640 1 -- 172.21.15.169:0/3079165368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa89d97920 msgr2=0x55aa89d97d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:34.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.134+0000 7f72dbfff640 1 --2- 172.21.15.169:0/3079165368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa89d97920 0x55aa89d97d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.134+0000 7f72dbfff640 1 -- 172.21.15.169:0/3079165368 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aa89a11210 con 0x55aa89d9d480 2024-03-28T01:06:34.140 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.136+0000 7f72daffd640 1 --2- 172.21.15.169:0/2609638375 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f72c40a2ae0 0x7f72c40a4fa0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x55aa89d98710 tx=0x7f72d0009b10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:34.141 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.138+0000 7f72dbfff640 1 --2- 172.21.15.169:0/3079165368 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa89d9d480 0x55aa89d9f910 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f72d4009d00 tx=0x7f72d400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18249296b195ff11 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:34.142 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.140+0000 7f72c1ffb640 1 -- 172.21.15.169:0/3079165368 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d4016070 con 0x55aa89d9d480 2024-03-28T01:06:34.143 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.140+0000 7f72c1ffb640 1 -- 172.21.15.169:0/3079165368 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72d4005020 con 0x55aa89d9d480 2024-03-28T01:06:34.144 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.141+0000 7f72e4d96740 1 -- 172.21.15.169:0/3079165368 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa89d9d480 msgr2=0x55aa89d9f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:34.144 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.141+0000 7f72e4d96740 1 --2- 172.21.15.169:0/3079165368 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa89d9d480 0x55aa89d9f910 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f72d4009d00 tx=0x7f72d400e6c0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.146 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.143+0000 7f72e4d96740 1 -- 172.21.15.169:0/3079165368 shutdown_connections 2024-03-28T01:06:34.146 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.143+0000 7f72e4d96740 1 --2- 172.21.15.169:0/3079165368 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa89d9d480 0x55aa89d9f910 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.146 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.143+0000 7f72e4d96740 1 --2- 172.21.15.169:0/3079165368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d98310 0x55aa89d9ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.147 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.143+0000 7f72e4d96740 1 --2- 172.21.15.169:0/3079165368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa89d97920 0x55aa89d97d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.147 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.143+0000 7f72e4d96740 1 -- 172.21.15.169:0/3079165368 >> 172.21.15.169:0/3079165368 conn(0x55aa89d92e90 msgr2=0x55aa89d93290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.150+0000 7f72e4d96740 1 -- 172.21.15.169:0/3079165368 shutdown_connections 2024-03-28T01:06:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.151+0000 7f72e4d96740 1 -- 172.21.15.169:0/3079165368 wait complete. 2024-03-28T01:06:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.151+0000 7f72e4d96740 1 -- start start 2024-03-28T01:06:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.152+0000 7f72e4d96740 1 Processor -- start 2024-03-28T01:06:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.152+0000 7f72e4d96740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d97920 0x55aa89da1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.152+0000 7f72e4d96740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa89d98310 0x55aa89da2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.152+0000 7f72e4d96740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa89d9d480 0x55aa89da91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.152+0000 7f72e4d96740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa89da0480 con 0x55aa89d9d480 2024-03-28T01:06:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.152+0000 7f72e4d96740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa89cb8d50 con 0x55aa89d98310 2024-03-28T01:06:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.152+0000 7f72e4d96740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa89da0600 con 0x55aa89d97920 2024-03-28T01:06:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.152+0000 7f72e4d96740 10 client.0 _ll_register_callbacks cb 0x55aa89cb8720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:06:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.153+0000 7f72db7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d97920 0x55aa89da1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.153+0000 7f72db7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d97920 0x55aa89da1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:42430/0 (socket says 172.21.15.169:42430) 2024-03-28T01:06:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.153+0000 7f72db7fe640 1 -- 172.21.15.169:0/3056674188 learned_addr learned my addr 172.21.15.169:0/3056674188 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:34.160 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.154+0000 7f72db7fe640 1 -- 172.21.15.169:0/3056674188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa89d98310 msgr2=0x55aa89da2180 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:06:34.160 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.154+0000 7f72db7fe640 1 --2- 172.21.15.169:0/3056674188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa89d98310 0x55aa89da2180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.160 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.154+0000 7f72db7fe640 1 -- 172.21.15.169:0/3056674188 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa89d9d480 msgr2=0x55aa89da91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:34.160 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.154+0000 7f72db7fe640 1 --2- 172.21.15.169:0/3056674188 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa89d9d480 0x55aa89da91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.160 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.154+0000 7f72db7fe640 1 -- 172.21.15.169:0/3056674188 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aa89add620 con 0x55aa89d97920 2024-03-28T01:06:34.160 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.155+0000 7f72db7fe640 1 --2- 172.21.15.169:0/3056674188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d97920 0x55aa89da1c40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f72cc0bdea0 tx=0x7f72cc0bd870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e45fea171888f7a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:34.161 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.157+0000 7f72baffd640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:06:34.161 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.157+0000 7f72baffd640 1 -- 172.21.15.169:0/3056674188 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc0bd560 con 0x55aa89d97920 2024-03-28T01:06:34.161 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.157+0000 7f72baffd640 1 -- 172.21.15.169:0/3056674188 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f72cc0b9c60 con 0x55aa89d97920 2024-03-28T01:06:34.161 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.157+0000 7f72e4d96740 1 -- 172.21.15.169:0/3056674188 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f72d4005020 con 0x55aa89d97920 2024-03-28T01:06:34.163 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.160+0000 7f72baffd640 1 -- 172.21.15.169:0/3056674188 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72cc0c2060 con 0x55aa89d97920 2024-03-28T01:06:34.163 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.160+0000 7f72e4d96740 10 client.27172 fetch_fsmap learned FSMap version 25 2024-03-28T01:06:34.163 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.160+0000 7f72e4d96740 1 -- 172.21.15.169:0/3056674188 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55aa89da04f0 con 0x55aa89d97920 2024-03-28T01:06:34.164 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.161+0000 7f72baffd640 1 -- 172.21.15.169:0/3056674188 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72d4005020 con 0x55aa89d97920 2024-03-28T01:06:34.164 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.161+0000 7f72baffd640 1 -- 172.21.15.169:0/3056674188 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f72cc0d3070 con 0x55aa89d97920 2024-03-28T01:06:34.165 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.162+0000 7f72e4d96740 10 client.27172 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:06:34.165 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.162+0000 7f72e4d96740 10 client.27172 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:06:34.165 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.162+0000 7f72e4d96740 20 client.27172 populate_metadata read hostname 'smithi169' 2024-03-28T01:06:34.165 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.162+0000 7f72e4d96740 1 --2- 172.21.15.169:0/3056674188 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aa89dadc40 0x55aa89db0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:34.165 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.162+0000 7f72e4d96740 4 client.27172 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:06:34.165 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.162+0000 7f72e4d96740 1 -- 172.21.15.169:0/3056674188 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aa89d0a050 con 0x55aa89dadc40 2024-03-28T01:06:34.166 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.163+0000 7f72daffd640 1 --2- 172.21.15.169:0/3056674188 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aa89dadc40 0x55aa89db0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:34.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:33 smithi177 ceph-mon[36206]: pgmap v652: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 746 MiB/s wr, 848 op/s 2024-03-28T01:06:34.171 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.169+0000 7f72baffd640 10 client.27172 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:34.171 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.169+0000 7f72daffd640 1 --2- 172.21.15.169:0/3056674188 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aa89dadc40 0x55aa89db0100 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8847a53a0e82956 server_cookie=5bc856f68c6aab7f in_seq=0 out_seq=0 2024-03-28T01:06:34.181 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.178+0000 7f72baffd640 1 -- 172.21.15.169:0/3056674188 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55aa89d0a050 con 0x55aa89dadc40 2024-03-28T01:06:34.181 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.178+0000 7f72baffd640 10 client.27172 handle_command_reply: tid=0 2024-03-28T01:06:34.182 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.179+0000 7f72e4d96740 1 client.27172 shutdown 2024-03-28T01:06:34.192 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.189+0000 7f72e4d96740 1 -- 172.21.15.169:0/3056674188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d97920 msgr2=0x55aa89da1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:34.192 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.189+0000 7f72e4d96740 1 --2- 172.21.15.169:0/3056674188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d97920 0x55aa89da1c40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f72cc0bdea0 tx=0x7f72cc0bd870 comp rx=0 tx=0).stop 2024-03-28T01:06:34.194 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.191+0000 7f72e4d96740 1 -- 172.21.15.169:0/3056674188 shutdown_connections 2024-03-28T01:06:34.194 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.191+0000 7f72e4d96740 1 --2- 172.21.15.169:0/3056674188 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aa89dadc40 0x55aa89db0100 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.194 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.191+0000 7f72e4d96740 1 --2- 172.21.15.169:0/3056674188 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa89d9d480 0x55aa89da91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.194 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.191+0000 7f72e4d96740 1 --2- 172.21.15.169:0/3056674188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa89d98310 0x55aa89da2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.194 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.191+0000 7f72e4d96740 1 --2- 172.21.15.169:0/3056674188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d97920 0x55aa89da1c40 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.194 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.191+0000 7f72baffd640 0 client.27172 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:34.194 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.191+0000 7f72e4d96740 1 -- 172.21.15.169:0/3056674188 >> 172.21.15.169:0/3056674188 conn(0x55aa89d92e90 msgr2=0x55aa89d92020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:34.201 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.198+0000 7f72e4d96740 1 -- 172.21.15.169:0/3056674188 shutdown_connections 2024-03-28T01:06:34.202 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.199+0000 7f72e4d96740 1 -- 172.21.15.169:0/3056674188 wait complete. 2024-03-28T01:06:34.204 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.200+0000 7f72e4d96740 20 client.27172 trim_cache size 0 max 16384 2024-03-28T01:06:34.245 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.242+0000 7f72e4d96740 1 Processor -- start 2024-03-28T01:06:34.246 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.243+0000 7f72e4d96740 1 -- start start 2024-03-28T01:06:34.247 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.243+0000 7f72e4d96740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d97920 0x55aa89d93270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:34.247 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.244+0000 7f72e4d96740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa89d98310 0x55aa89da9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:34.247 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.244+0000 7f72e4d96740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa89d9d480 0x55aa89d960f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:34.247 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.244+0000 7f72e4d96740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c8012080 con 0x55aa89d9d480 2024-03-28T01:06:34.247 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.244+0000 7f72e4d96740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c800fb80 con 0x55aa89d98310 2024-03-28T01:06:34.247 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.244+0000 7f72e4d96740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c800dc20 con 0x55aa89d97920 2024-03-28T01:06:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.245+0000 7f72daffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa89d98310 0x55aa89da9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.245+0000 7f72db7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d97920 0x55aa89d93270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.245+0000 7f72db7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d97920 0x55aa89d93270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:42440/0 (socket says 172.21.15.169:42440) 2024-03-28T01:06:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.245+0000 7f72db7fe640 1 -- 172.21.15.169:0/2914828999 learned_addr learned my addr 172.21.15.169:0/2914828999 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.246+0000 7f72db7fe640 1 -- 172.21.15.169:0/2914828999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa89d98310 msgr2=0x55aa89da9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.246+0000 7f72db7fe640 1 --2- 172.21.15.169:0/2914828999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa89d98310 0x55aa89da9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.246+0000 7f72db7fe640 1 -- 172.21.15.169:0/2914828999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa89d9d480 msgr2=0x55aa89d960f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:06:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.246+0000 7f72dbfff640 1 --2- 172.21.15.169:0/2914828999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa89d9d480 0x55aa89d960f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.246+0000 7f72db7fe640 1 --2- 172.21.15.169:0/2914828999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa89d9d480 0x55aa89d960f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.246+0000 7f72db7fe640 1 -- 172.21.15.169:0/2914828999 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aa89da0310 con 0x55aa89d97920 2024-03-28T01:06:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.246+0000 7f72db7fe640 1 --2- 172.21.15.169:0/2914828999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d97920 0x55aa89d93270 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f72cc0c6180 tx=0x7f72cc0bd9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7470f5ed76815ed6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:34.259 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.256+0000 7f72c17fa640 1 -- 172.21.15.169:0/2914828999 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc0b8d50 con 0x55aa89d97920 2024-03-28T01:06:34.259 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.257+0000 7f72c17fa640 1 -- 172.21.15.169:0/2914828999 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72cc0c66b0 con 0x55aa89d97920 2024-03-28T01:06:34.261 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.258+0000 7f72e4d96740 1 -- 172.21.15.169:0/2914828999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d97920 msgr2=0x55aa89d93270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:34.261 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.258+0000 7f72e4d96740 1 --2- 172.21.15.169:0/2914828999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d97920 0x55aa89d93270 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f72cc0c6180 tx=0x7f72cc0bd9a0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.263 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.260+0000 7f72e4d96740 1 -- 172.21.15.169:0/2914828999 shutdown_connections 2024-03-28T01:06:34.263 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.260+0000 7f72e4d96740 1 --2- 172.21.15.169:0/2914828999 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa89d9d480 0x55aa89d960f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.263 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.260+0000 7f72e4d96740 1 --2- 172.21.15.169:0/2914828999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa89d98310 0x55aa89da9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.263 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.260+0000 7f72e4d96740 1 --2- 172.21.15.169:0/2914828999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d97920 0x55aa89d93270 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.263 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.260+0000 7f72e4d96740 1 -- 172.21.15.169:0/2914828999 >> 172.21.15.169:0/2914828999 conn(0x55aa89d92e90 msgr2=0x55aa89d9cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:34.269 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.266+0000 7f72e4d96740 1 -- 172.21.15.169:0/2914828999 shutdown_connections 2024-03-28T01:06:34.271 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.268+0000 7f72e4d96740 1 -- 172.21.15.169:0/2914828999 wait complete. 2024-03-28T01:06:34.271 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.268+0000 7f72e4d96740 1 -- start start 2024-03-28T01:06:34.271 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.268+0000 7f72e4d96740 1 Processor -- start 2024-03-28T01:06:34.274 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.271+0000 7f72e4d96740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa89d97920 0x55aa89da6790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:34.274 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.271+0000 7f72e4d96740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d98310 0x55aa89da6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:34.274 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.271+0000 7f72e4d96740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa89d9d480 0x55aa89da7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:34.274 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.271+0000 7f72e4d96740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c800e220 con 0x55aa89d9d480 2024-03-28T01:06:34.274 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.271+0000 7f72e4d96740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c800f520 con 0x55aa89d97920 2024-03-28T01:06:34.274 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.271+0000 7f72e4d96740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c800e860 con 0x55aa89d98310 2024-03-28T01:06:34.274 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.271+0000 7f72e4d96740 10 client.0 _ll_register_callbacks cb 0x7f72cc0ba1b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:06:34.275 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.272+0000 7f72db7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa89d97920 0x55aa89da6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:34.275 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.272+0000 7f72db7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa89d97920 0x55aa89da6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:39442/0 (socket says 172.21.15.169:39442) 2024-03-28T01:06:34.275 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.272+0000 7f72db7fe640 1 -- 172.21.15.169:0/3577718965 learned_addr learned my addr 172.21.15.169:0/3577718965 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:34.275 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.272+0000 7f72dbfff640 1 --2- 172.21.15.169:0/3577718965 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa89d9d480 0x55aa89da7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.272+0000 7f72daffd640 1 --2- 172.21.15.169:0/3577718965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d98310 0x55aa89da6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.272+0000 7f72daffd640 1 -- 172.21.15.169:0/3577718965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa89d97920 msgr2=0x55aa89da6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.272+0000 7f72daffd640 1 --2- 172.21.15.169:0/3577718965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa89d97920 0x55aa89da6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.272+0000 7f72daffd640 1 -- 172.21.15.169:0/3577718965 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa89d9d480 msgr2=0x55aa89da7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.272+0000 7f72daffd640 1 --2- 172.21.15.169:0/3577718965 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa89d9d480 0x55aa89da7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.272+0000 7f72daffd640 1 -- 172.21.15.169:0/3577718965 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aa89da8230 con 0x55aa89d98310 2024-03-28T01:06:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.272+0000 7f72daffd640 1 --2- 172.21.15.169:0/3577718965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d98310 0x55aa89da6cd0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f72cc0bc1c0 tx=0x7f72d00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f812062aee8f82 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:34.298 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.294+0000 7f72baffd640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:06:34.298 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.294+0000 7f72baffd640 1 -- 172.21.15.169:0/3577718965 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d002e070 con 0x55aa89d98310 2024-03-28T01:06:34.298 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.294+0000 7f72baffd640 1 -- 172.21.15.169:0/3577718965 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f72d0005220 con 0x55aa89d98310 2024-03-28T01:06:34.298 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.294+0000 7f72e4d96740 1 -- 172.21.15.169:0/3577718965 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f72cc0c66b0 con 0x55aa89d98310 2024-03-28T01:06:34.298 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.295+0000 7f72baffd640 1 -- 172.21.15.169:0/3577718965 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72d0037050 con 0x55aa89d98310 2024-03-28T01:06:34.299 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.295+0000 7f72e4d96740 10 client.27178 fetch_fsmap learned FSMap version 25 2024-03-28T01:06:34.299 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.296+0000 7f72e4d96740 1 -- 172.21.15.169:0/3577718965 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55aa89e09590 con 0x55aa89d98310 2024-03-28T01:06:34.304 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.299+0000 7f72baffd640 1 -- 172.21.15.169:0/3577718965 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72cc0c66b0 con 0x55aa89d98310 2024-03-28T01:06:34.304 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.300+0000 7f72baffd640 1 -- 172.21.15.169:0/3577718965 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f72d0027070 con 0x55aa89d98310 2024-03-28T01:06:34.304 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.300+0000 7f72e4d96740 10 client.27178 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:06:34.304 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.300+0000 7f72e4d96740 10 client.27178 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:06:34.304 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.301+0000 7f72e4d96740 20 client.27178 populate_metadata read hostname 'smithi169' 2024-03-28T01:06:34.304 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.301+0000 7f72e4d96740 1 --2- 172.21.15.169:0/3577718965 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aa89e09e80 0x55aa89e0c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:34.304 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.301+0000 7f72e4d96740 4 client.27178 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:06:34.304 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.301+0000 7f72e4d96740 1 -- 172.21.15.169:0/3577718965 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55aa89d043a0 con 0x55aa89e09e80 2024-03-28T01:06:34.318 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.314+0000 7f72db7fe640 1 --2- 172.21.15.169:0/3577718965 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aa89e09e80 0x55aa89e0c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:34.334 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.329+0000 7f72baffd640 10 client.27178 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:34.335 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:34.329+0000 7f72db7fe640 1 --2- 172.21.15.169:0/3577718965 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aa89e09e80 0x55aa89e0c2a0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6cd75981a942259 server_cookie=483e243a2e817777 in_seq=0 out_seq=0 2024-03-28T01:06:35.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:34 smithi169 ceph-mon[32274]: scrub summary: idle 2024-03-28T01:06:35.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:34 smithi174 ceph-mon[36060]: scrub summary: idle 2024-03-28T01:06:35.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:34 smithi177 ceph-mon[36206]: scrub summary: idle 2024-03-28T01:06:36.069 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:35 smithi169 ceph-mon[32274]: pgmap v653: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 590 MiB/s wr, 605 op/s 2024-03-28T01:06:36.070 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:35 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:06:36.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:35 smithi174 ceph-mon[36060]: pgmap v653: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 590 MiB/s wr, 605 op/s 2024-03-28T01:06:36.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:35 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:06:36.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:35 smithi177 ceph-mon[36206]: pgmap v653: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 590 MiB/s wr, 605 op/s 2024-03-28T01:06:36.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:35 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:06:37.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:36 smithi169 ceph-mon[32274]: pgmap v654: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 698 MiB/s wr, 660 op/s 2024-03-28T01:06:37.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:36 smithi174 ceph-mon[36060]: pgmap v654: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 698 MiB/s wr, 660 op/s 2024-03-28T01:06:37.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:36 smithi177 ceph-mon[36206]: pgmap v654: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 698 MiB/s wr, 660 op/s 2024-03-28T01:06:38.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:06:38.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:06:38.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:06:38.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:37 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:06:38.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:37 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:06:38.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:37 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:06:38.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:37 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:06:38.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:37 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:06:38.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:37 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:06:39.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:38 smithi169 ceph-mon[32274]: pgmap v655: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 640 MiB/s wr, 567 op/s 2024-03-28T01:06:39.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:38 smithi174 ceph-mon[36060]: pgmap v655: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 640 MiB/s wr, 567 op/s 2024-03-28T01:06:39.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:38 smithi177 ceph-mon[36206]: pgmap v655: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 640 MiB/s wr, 567 op/s 2024-03-28T01:06:40.322 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.319+0000 7f72baffd640 1 -- 172.21.15.169:0/3577718965 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55aa89d043a0 con 0x55aa89e09e80 2024-03-28T01:06:40.322 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.319+0000 7f72baffd640 10 client.27178 handle_command_reply: tid=0 2024-03-28T01:06:40.322 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.319+0000 7f72e4d96740 1 client.27178 shutdown 2024-03-28T01:06:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.321+0000 7f72e4d96740 1 -- 172.21.15.169:0/3577718965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d98310 msgr2=0x55aa89da6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.321+0000 7f72e4d96740 1 --2- 172.21.15.169:0/3577718965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d98310 0x55aa89da6cd0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f72cc0bc1c0 tx=0x7f72d00046c0 comp rx=0 tx=0).stop 2024-03-28T01:06:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.323+0000 7f72e4d96740 1 -- 172.21.15.169:0/3577718965 shutdown_connections 2024-03-28T01:06:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.323+0000 7f72e4d96740 1 --2- 172.21.15.169:0/3577718965 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55aa89e09e80 0x55aa89e0c2a0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.323+0000 7f72e4d96740 1 --2- 172.21.15.169:0/3577718965 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55aa89d9d480 0x55aa89da7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.323+0000 7f72e4d96740 1 --2- 172.21.15.169:0/3577718965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aa89d98310 0x55aa89da6cd0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.323+0000 7f72e4d96740 1 --2- 172.21.15.169:0/3577718965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa89d97920 0x55aa89da6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.323+0000 7f72e4d96740 1 -- 172.21.15.169:0/3577718965 >> 172.21.15.169:0/3577718965 conn(0x55aa89d92e90 msgr2=0x55aa89d93270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.323+0000 7f72baffd640 0 client.27178 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.323+0000 7f72e4d96740 1 -- 172.21.15.169:0/3577718965 shutdown_connections 2024-03-28T01:06:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.323+0000 7f72e4d96740 1 -- 172.21.15.169:0/3577718965 wait complete. 2024-03-28T01:06:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.324+0000 7f72e4d96740 20 client.27178 trim_cache size 0 max 16384 2024-03-28T01:06:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:06:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:06:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:06:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:06:40.331 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.327+0000 7f72e20d1640 1 -- 172.21.15.169:0/2609638375 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f72c40a2ae0 msgr2=0x7f72c40a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:40.331 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.328+0000 7f72e20d1640 1 --2- 172.21.15.169:0/2609638375 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f72c40a2ae0 0x7f72c40a4fa0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x55aa89d98710 tx=0x7f72d0009b10 comp rx=0 tx=0).stop 2024-03-28T01:06:40.331 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.328+0000 7f72e20d1640 1 -- 172.21.15.169:0/2609638375 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f72dc1086a0 msgr2=0x7f72dc118040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:40.331 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.328+0000 7f72e20d1640 1 --2- 172.21.15.169:0/2609638375 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f72dc1086a0 0x7f72dc118040 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009850 tx=0x7f72cc02ed00 comp rx=0 tx=0).stop 2024-03-28T01:06:40.336 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.333+0000 7f72e20d1640 1 -- 172.21.15.169:0/2609638375 shutdown_connections 2024-03-28T01:06:40.336 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.333+0000 7f72e20d1640 1 --2- 172.21.15.169:0/2609638375 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f72c40a2ae0 0x7f72c40a4fa0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:40.336 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.333+0000 7f72e20d1640 1 --2- 172.21.15.169:0/2609638375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72dc10d9c0 0x7f72dc1150b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:40.336 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.333+0000 7f72e20d1640 1 --2- 172.21.15.169:0/2609638375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72dc10b010 0x7f72dc114a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:40.336 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.333+0000 7f72e20d1640 1 --2- 172.21.15.169:0/2609638375 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f72dc1086a0 0x7f72dc118040 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:40.336 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.333+0000 7f72e20d1640 1 -- 172.21.15.169:0/2609638375 >> 172.21.15.169:0/2609638375 conn(0x7f72dc06e6e0 msgr2=0x7f72dc10ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:40.339 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.336+0000 7f72e20d1640 1 -- 172.21.15.169:0/2609638375 shutdown_connections 2024-03-28T01:06:40.339 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:40.336+0000 7f72e20d1640 1 -- 172.21.15.169:0/2609638375 wait complete. 2024-03-28T01:06:42.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:41 smithi169 ceph-mon[32274]: pgmap v656: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 604 MiB/s wr, 520 op/s 2024-03-28T01:06:42.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:41 smithi174 ceph-mon[36060]: pgmap v656: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 604 MiB/s wr, 520 op/s 2024-03-28T01:06:42.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:41 smithi177 ceph-mon[36206]: pgmap v656: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 604 MiB/s wr, 520 op/s 2024-03-28T01:06:44.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:43 smithi169 ceph-mon[32274]: pgmap v657: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 748 MiB/s wr, 616 op/s 2024-03-28T01:06:44.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:43 smithi174 ceph-mon[36060]: pgmap v657: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 748 MiB/s wr, 616 op/s 2024-03-28T01:06:44.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:43 smithi177 ceph-mon[36206]: pgmap v657: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 748 MiB/s wr, 616 op/s 2024-03-28T01:06:45.968 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:45 smithi177 ceph-mon[36206]: pgmap v658: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 609 MiB/s wr, 473 op/s 2024-03-28T01:06:46.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:45 smithi169 ceph-mon[32274]: pgmap v658: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 609 MiB/s wr, 473 op/s 2024-03-28T01:06:46.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:45 smithi174 ceph-mon[36060]: pgmap v658: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 609 MiB/s wr, 473 op/s 2024-03-28T01:06:48.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:47 smithi169 ceph-mon[32274]: pgmap v659: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 746 MiB/s wr, 661 op/s 2024-03-28T01:06:48.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:47 smithi174 ceph-mon[36060]: pgmap v659: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 746 MiB/s wr, 661 op/s 2024-03-28T01:06:48.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:47 smithi177 ceph-mon[36206]: pgmap v659: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 746 MiB/s wr, 661 op/s 2024-03-28T01:06:50.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:49 smithi169 ceph-mon[32274]: pgmap v660: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 622 MiB/s wr, 569 op/s 2024-03-28T01:06:50.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:49 smithi174 ceph-mon[36060]: pgmap v660: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 622 MiB/s wr, 569 op/s 2024-03-28T01:06:50.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:49 smithi177 ceph-mon[36206]: pgmap v660: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 622 MiB/s wr, 569 op/s 2024-03-28T01:06:52.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:51 smithi169 ceph-mon[32274]: pgmap v661: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 589 MiB/s wr, 541 op/s 2024-03-28T01:06:52.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:51 smithi174 ceph-mon[36060]: pgmap v661: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 589 MiB/s wr, 541 op/s 2024-03-28T01:06:52.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:51 smithi177 ceph-mon[36206]: pgmap v661: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 589 MiB/s wr, 541 op/s 2024-03-28T01:06:53.617 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.613+0000 7fecc4ce8640 1 -- 172.21.15.169:0/408450897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecc0072970 msgr2=0x7fecc0072d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:53.617 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.613+0000 7fecc4ce8640 1 --2- 172.21.15.169:0/408450897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecc0072970 0x7fecc0072d90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fecb4009900 tx=0x7fecb402bc50 comp rx=0 tx=0).stop 2024-03-28T01:06:53.622 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.619+0000 7fecc4ce8640 1 -- 172.21.15.169:0/408450897 shutdown_connections 2024-03-28T01:06:53.622 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.619+0000 7fecc4ce8640 1 --2- 172.21.15.169:0/408450897 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fecc01093d0 0x7fecc010e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.623 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.619+0000 7fecc4ce8640 1 --2- 172.21.15.169:0/408450897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecc00732d0 0x7fecc0108d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.623 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.619+0000 7fecc4ce8640 1 --2- 172.21.15.169:0/408450897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecc0072970 0x7fecc0072d90 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.623 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.619+0000 7fecc4ce8640 1 -- 172.21.15.169:0/408450897 >> 172.21.15.169:0/408450897 conn(0x7fecc006e510 msgr2=0x7fecc0070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:53.629 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.626+0000 7fecc4ce8640 1 -- 172.21.15.169:0/408450897 shutdown_connections 2024-03-28T01:06:53.629 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.626+0000 7fecc4ce8640 1 -- 172.21.15.169:0/408450897 wait complete. 2024-03-28T01:06:53.631 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.628+0000 7fecc4ce8640 1 Processor -- start 2024-03-28T01:06:53.632 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.629+0000 7fecc4ce8640 1 -- start start 2024-03-28T01:06:53.633 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.630+0000 7fecc4ce8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecc0072970 0x7fecc01a6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:53.633 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.630+0000 7fecc4ce8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecc00732d0 0x7fecc01a6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:53.633 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.630+0000 7fecbe59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecc0072970 0x7fecc01a6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:53.633 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.630+0000 7fecbe59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecc0072970 0x7fecc01a6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:41968/0 (socket says 172.21.15.169:41968) 2024-03-28T01:06:53.634 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.630+0000 7fecbe59c640 1 -- 172.21.15.169:0/155654157 learned_addr learned my addr 172.21.15.169:0/155654157 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:53.634 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.631+0000 7fecc4ce8640 1 --2- 172.21.15.169:0/155654157 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fecc01093d0 0x7fecc01adc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:53.634 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.631+0000 7fecbdd9b640 1 --2- 172.21.15.169:0/155654157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecc00732d0 0x7fecc01a6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:53.634 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.631+0000 7fecc4ce8640 1 -- 172.21.15.169:0/155654157 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc011c010 con 0x7fecc01093d0 2024-03-28T01:06:53.635 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.632+0000 7fecc4ce8640 1 -- 172.21.15.169:0/155654157 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc011be90 con 0x7fecc0072970 2024-03-28T01:06:53.635 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.632+0000 7fecc4ce8640 1 -- 172.21.15.169:0/155654157 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc011c190 con 0x7fecc00732d0 2024-03-28T01:06:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.633+0000 7fecbed9d640 1 --2- 172.21.15.169:0/155654157 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fecc01093d0 0x7fecc01adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.633+0000 7fecbe59c640 1 -- 172.21.15.169:0/155654157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecc00732d0 msgr2=0x7fecc01a6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.633+0000 7fecbe59c640 1 --2- 172.21.15.169:0/155654157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecc00732d0 0x7fecc01a6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.633+0000 7fecbe59c640 1 -- 172.21.15.169:0/155654157 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fecc01093d0 msgr2=0x7fecc01adc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.633+0000 7fecbe59c640 1 --2- 172.21.15.169:0/155654157 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fecc01093d0 0x7fecc01adc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.633+0000 7fecbe59c640 1 -- 172.21.15.169:0/155654157 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc01ae270 con 0x7fecc0072970 2024-03-28T01:06:53.637 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.634+0000 7fecbe59c640 1 --2- 172.21.15.169:0/155654157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecc0072970 0x7fecc01a6650 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fecb4000c00 tx=0x7fecb40052f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:53.641 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.636+0000 7fecaf7fe640 1 -- 172.21.15.169:0/155654157 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb4041020 con 0x7fecc0072970 2024-03-28T01:06:53.641 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.636+0000 7fecaf7fe640 1 -- 172.21.15.169:0/155654157 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fecb402cad0 con 0x7fecc0072970 2024-03-28T01:06:53.641 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.636+0000 7fecaf7fe640 1 -- 172.21.15.169:0/155654157 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb4034c20 con 0x7fecc0072970 2024-03-28T01:06:53.642 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.638+0000 7fecc4ce8640 1 -- 172.21.15.169:0/155654157 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc01ae500 con 0x7fecc0072970 2024-03-28T01:06:53.642 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.638+0000 7fecc4ce8640 1 -- 172.21.15.169:0/155654157 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc01ae8b0 con 0x7fecc0072970 2024-03-28T01:06:53.654 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.649+0000 7fecaf7fe640 1 -- 172.21.15.169:0/155654157 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fecb4044020 con 0x7fecc0072970 2024-03-28T01:06:53.654 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.650+0000 7fecaf7fe640 1 --2- 172.21.15.169:0/155654157 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7feca80a2a00 0x7feca80a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:53.654 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.650+0000 7fecaf7fe640 1 -- 172.21.15.169:0/155654157 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fecb40f0060 con 0x7fecc0072970 2024-03-28T01:06:53.658 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.652+0000 7fecbdd9b640 1 --2- 172.21.15.169:0/155654157 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7feca80a2a00 0x7feca80a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:53.658 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.653+0000 7fecc79ab740 1 Processor -- start 2024-03-28T01:06:53.659 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.654+0000 7fecc79ab740 1 -- start start 2024-03-28T01:06:53.659 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.654+0000 7fecc79ab740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594787cbb20 0x5594787cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:53.659 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.654+0000 7fecc79ab740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787cc510 0x5594787d1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:53.659 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.654+0000 7fecc79ab740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5594787d1680 0x5594787d3b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:53.659 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.655+0000 7fecc79ab740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x559478730b10 con 0x5594787d1680 2024-03-28T01:06:53.659 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.655+0000 7fecc79ab740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594786d9900 con 0x5594787cc510 2024-03-28T01:06:53.659 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.655+0000 7fecc79ab740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55947872cbb0 con 0x5594787cbb20 2024-03-28T01:06:53.659 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.655+0000 7fecbe59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594787cbb20 0x5594787cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:53.659 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.655+0000 7fecbe59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594787cbb20 0x5594787cbf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:60814/0 (socket says 172.21.15.169:60814) 2024-03-28T01:06:53.659 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.655+0000 7fecbe59c640 1 -- 172.21.15.169:0/3888104627 learned_addr learned my addr 172.21.15.169:0/3888104627 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:53.660 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.657+0000 7fecbed9d640 1 --2- 172.21.15.169:0/3888104627 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5594787d1680 0x5594787d3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.657+0000 7fecbdd9b640 1 --2- 172.21.15.169:0/3888104627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787cc510 0x5594787d1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.658+0000 7fecbe59c640 1 -- 172.21.15.169:0/3888104627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787cc510 msgr2=0x5594787d1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.658+0000 7fecbe59c640 1 --2- 172.21.15.169:0/3888104627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787cc510 0x5594787d1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.658+0000 7fecbe59c640 1 -- 172.21.15.169:0/3888104627 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5594787d1680 msgr2=0x5594787d3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.658+0000 7fecbe59c640 1 --2- 172.21.15.169:0/3888104627 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5594787d1680 0x5594787d3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.658+0000 7fecbe59c640 1 -- 172.21.15.169:0/3888104627 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559478445210 con 0x5594787cbb20 2024-03-28T01:06:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.658+0000 7fecbdd9b640 1 --2- 172.21.15.169:0/3888104627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787cc510 0x5594787d1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:06:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.658+0000 7fecbe59c640 1 --2- 172.21.15.169:0/3888104627 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594787cbb20 0x5594787cbf40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fecb40b89d0 tx=0x7fecb4009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54c20c904c86f0cf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:53.663 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.660+0000 7fecbdd9b640 1 --2- 172.21.15.169:0/155654157 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7feca80a2a00 0x7feca80a4ec0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fecc01a7b60 tx=0x7fecb0013830 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:53.670 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.667+0000 7fecacff9640 1 -- 172.21.15.169:0/3888104627 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb40d5070 con 0x5594787cbb20 2024-03-28T01:06:53.670 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.667+0000 7fecacff9640 1 -- 172.21.15.169:0/3888104627 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fecb40bcd00 con 0x5594787cbb20 2024-03-28T01:06:53.670 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.667+0000 7fecacff9640 1 -- 172.21.15.169:0/3888104627 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb402cc80 con 0x5594787cbb20 2024-03-28T01:06:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.670+0000 7fecc79ab740 1 -- 172.21.15.169:0/3888104627 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594787cbb20 msgr2=0x5594787cbf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.670+0000 7fecc79ab740 1 --2- 172.21.15.169:0/3888104627 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594787cbb20 0x5594787cbf40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fecb40b89d0 tx=0x7fecb4009260 comp rx=0 tx=0).stop 2024-03-28T01:06:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.670+0000 7fecc79ab740 1 -- 172.21.15.169:0/3888104627 shutdown_connections 2024-03-28T01:06:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.670+0000 7fecc79ab740 1 --2- 172.21.15.169:0/3888104627 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5594787d1680 0x5594787d3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.670+0000 7fecc79ab740 1 --2- 172.21.15.169:0/3888104627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787cc510 0x5594787d1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.670+0000 7fecc79ab740 1 --2- 172.21.15.169:0/3888104627 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594787cbb20 0x5594787cbf40 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.671+0000 7fecc79ab740 1 -- 172.21.15.169:0/3888104627 >> 172.21.15.169:0/3888104627 conn(0x5594787c7090 msgr2=0x5594787c7490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.675+0000 7fecc79ab740 1 -- 172.21.15.169:0/3888104627 shutdown_connections 2024-03-28T01:06:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.675+0000 7fecc79ab740 1 -- 172.21.15.169:0/3888104627 wait complete. 2024-03-28T01:06:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.676+0000 7fecc79ab740 1 -- start start 2024-03-28T01:06:53.679 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.676+0000 7fecc79ab740 1 Processor -- start 2024-03-28T01:06:53.680 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.676+0000 7fecc79ab740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594787cbb20 0x5594787d5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:53.680 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.677+0000 7fecc79ab740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5594787cc510 0x5594787d6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:53.680 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.677+0000 7fecc79ab740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787d1680 0x5594787dd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:53.680 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.677+0000 7fecc79ab740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594787d43f0 con 0x5594787cc510 2024-03-28T01:06:53.680 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.677+0000 7fecc79ab740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594786ecd50 con 0x5594787d1680 2024-03-28T01:06:53.680 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.677+0000 7fecc79ab740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594787d4570 con 0x5594787cbb20 2024-03-28T01:06:53.680 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.677+0000 7fecc79ab740 10 client.0 _ll_register_callbacks cb 0x5594786ec720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:06:53.680 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.677+0000 7fecbe59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594787cbb20 0x5594787d5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:53.681 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.677+0000 7fecbed9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787d1680 0x5594787dd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:53.681 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.677+0000 7fecbed9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787d1680 0x5594787dd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:41994/0 (socket says 172.21.15.169:41994) 2024-03-28T01:06:53.681 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.677+0000 7fecbed9d640 1 -- 172.21.15.169:0/1294129063 learned_addr learned my addr 172.21.15.169:0/1294129063 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:53.682 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.679+0000 7fecbed9d640 1 -- 172.21.15.169:0/1294129063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594787cbb20 msgr2=0x5594787d5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:53.682 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.679+0000 7fecbed9d640 1 --2- 172.21.15.169:0/1294129063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594787cbb20 0x5594787d5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.682 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.679+0000 7fecbed9d640 1 -- 172.21.15.169:0/1294129063 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5594787cc510 msgr2=0x5594787d6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:53.682 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.679+0000 7fecbed9d640 1 --2- 172.21.15.169:0/1294129063 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5594787cc510 0x5594787d6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.682 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.679+0000 7fecbed9d640 1 -- 172.21.15.169:0/1294129063 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559478511620 con 0x5594787d1680 2024-03-28T01:06:53.683 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.680+0000 7fecbed9d640 1 --2- 172.21.15.169:0/1294129063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787d1680 0x5594787dd180 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fecb8003f90 tx=0x7fecb80044d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=692608c0db7afee4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:53.683 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.680+0000 7feca5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:06:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.684+0000 7feca5ffb640 1 -- 172.21.15.169:0/1294129063 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb8004d20 con 0x5594787d1680 2024-03-28T01:06:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.684+0000 7feca5ffb640 1 -- 172.21.15.169:0/1294129063 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fecb80128e0 con 0x5594787d1680 2024-03-28T01:06:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.686+0000 7fecc79ab740 1 -- 172.21.15.169:0/1294129063 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fecb40bcd00 con 0x5594787d1680 2024-03-28T01:06:53.693 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.691+0000 7feca5ffb640 1 -- 172.21.15.169:0/1294129063 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fecb8012ae0 con 0x5594787d1680 2024-03-28T01:06:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.691+0000 7fecc79ab740 10 client.27245 fetch_fsmap learned FSMap version 25 2024-03-28T01:06:53.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.692+0000 7fecc79ab740 1 -- 172.21.15.169:0/1294129063 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5594787e0520 con 0x5594787d1680 2024-03-28T01:06:53.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.692+0000 7feca5ffb640 1 -- 172.21.15.169:0/1294129063 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fecb40bcd00 con 0x5594787d1680 2024-03-28T01:06:53.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.692+0000 7feca5ffb640 1 -- 172.21.15.169:0/1294129063 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fecb8019070 con 0x5594787d1680 2024-03-28T01:06:53.695 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.692+0000 7fecc79ab740 10 client.27245 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:06:53.696 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.692+0000 7fecc79ab740 10 client.27245 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:06:53.696 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.692+0000 7fecc79ab740 20 client.27245 populate_metadata read hostname 'smithi169' 2024-03-28T01:06:53.696 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.692+0000 7fecc79ab740 1 --2- 172.21.15.169:0/1294129063 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5594787e2030 0x5594787e4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:53.696 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.692+0000 7fecc79ab740 4 client.27245 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:06:53.696 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.692+0000 7fecc79ab740 1 -- 172.21.15.169:0/1294129063 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55947873e050 con 0x5594787e2030 2024-03-28T01:06:53.696 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.693+0000 7fecbe59c640 1 --2- 172.21.15.169:0/1294129063 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5594787e2030 0x5594787e4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:53.697 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.694+0000 7fecbe59c640 1 --2- 172.21.15.169:0/1294129063 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5594787e2030 0x5594787e4470 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd231532a0010f89 server_cookie=da7f4c684ce90052 in_seq=0 out_seq=0 2024-03-28T01:06:53.697 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.694+0000 7feca5ffb640 10 client.27245 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:53.700 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.696+0000 7feca5ffb640 1 -- 172.21.15.169:0/1294129063 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55947873e050 con 0x5594787e2030 2024-03-28T01:06:53.700 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.696+0000 7feca5ffb640 10 client.27245 handle_command_reply: tid=0 2024-03-28T01:06:53.701 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.696+0000 7fecc79ab740 1 client.27245 shutdown 2024-03-28T01:06:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.699+0000 7fecc79ab740 1 -- 172.21.15.169:0/1294129063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787d1680 msgr2=0x5594787dd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.699+0000 7fecc79ab740 1 --2- 172.21.15.169:0/1294129063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787d1680 0x5594787dd180 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fecb8003f90 tx=0x7fecb80044d0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.703 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.699+0000 7fecc79ab740 1 -- 172.21.15.169:0/1294129063 shutdown_connections 2024-03-28T01:06:53.703 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.699+0000 7fecc79ab740 1 --2- 172.21.15.169:0/1294129063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787d1680 0x5594787dd180 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.703 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.699+0000 7fecc79ab740 1 --2- 172.21.15.169:0/1294129063 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5594787e2030 0x5594787e4470 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.703 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.699+0000 7fecc79ab740 1 --2- 172.21.15.169:0/1294129063 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5594787cc510 0x5594787d6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.703 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.699+0000 7fecc79ab740 1 --2- 172.21.15.169:0/1294129063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594787cbb20 0x5594787d5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.703 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.699+0000 7fecc79ab740 1 -- 172.21.15.169:0/1294129063 >> 172.21.15.169:0/1294129063 conn(0x5594787c7090 msgr2=0x5594787c6290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:53.705 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.701+0000 7feca5ffb640 0 client.27245 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.704+0000 7fecc79ab740 1 -- 172.21.15.169:0/1294129063 shutdown_connections 2024-03-28T01:06:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.704+0000 7fecc79ab740 1 -- 172.21.15.169:0/1294129063 wait complete. 2024-03-28T01:06:53.708 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.705+0000 7fecc79ab740 20 client.27245 trim_cache size 0 max 16384 2024-03-28T01:06:53.740 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.737+0000 7fecc79ab740 1 Processor -- start 2024-03-28T01:06:53.744 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.739+0000 7fecc79ab740 1 -- start start 2024-03-28T01:06:53.744 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.739+0000 7fecc79ab740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5594787cbb20 0x5594787d5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:53.744 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.739+0000 7fecc79ab740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594787cc510 0x5594787dd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:53.744 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.739+0000 7fecc79ab740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787d1680 0x5594787d6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:53.744 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.739+0000 7fecc79ab740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc01bb360 con 0x5594787cbb20 2024-03-28T01:06:53.745 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.739+0000 7fecc79ab740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc01b8e60 con 0x5594787d1680 2024-03-28T01:06:53.745 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.739+0000 7fecc79ab740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc01b6f00 con 0x5594787cc510 2024-03-28T01:06:53.745 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.740+0000 7fecbe59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5594787cbb20 0x5594787d5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:53.745 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.740+0000 7fecbe59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5594787cbb20 0x5594787d5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:32786/0 (socket says 172.21.15.169:32786) 2024-03-28T01:06:53.745 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.740+0000 7fecbe59c640 1 -- 172.21.15.169:0/2263486931 learned_addr learned my addr 172.21.15.169:0/2263486931 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:53.745 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.741+0000 7fecbe59c640 1 -- 172.21.15.169:0/2263486931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594787cc510 msgr2=0x5594787dd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:53.745 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.741+0000 7fecbe59c640 1 --2- 172.21.15.169:0/2263486931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594787cc510 0x5594787dd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.745 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.741+0000 7fecbed9d640 1 --2- 172.21.15.169:0/2263486931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787d1680 0x5594787d6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:53.745 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.741+0000 7fecbe59c640 1 -- 172.21.15.169:0/2263486931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787d1680 msgr2=0x5594787d6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:53.745 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.741+0000 7fecbe59c640 1 --2- 172.21.15.169:0/2263486931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787d1680 0x5594787d6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.745 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.741+0000 7fecbe59c640 1 -- 172.21.15.169:0/2263486931 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594787c7e00 con 0x5594787cbb20 2024-03-28T01:06:53.745 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.741+0000 7fecbed9d640 1 --2- 172.21.15.169:0/2263486931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787d1680 0x5594787d6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:06:53.747 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.742+0000 7fecbe59c640 1 --2- 172.21.15.169:0/2263486931 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5594787cbb20 0x5594787d5c00 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fecb4006010 tx=0x7fecb402bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46f11156dd8f95de server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:53.749 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.746+0000 7feca7fff640 1 -- 172.21.15.169:0/2263486931 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb40d5040 con 0x5594787cbb20 2024-03-28T01:06:53.749 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.746+0000 7feca7fff640 1 -- 172.21.15.169:0/2263486931 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fecb40bcd00 con 0x5594787cbb20 2024-03-28T01:06:53.749 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.746+0000 7feca7fff640 1 -- 172.21.15.169:0/2263486931 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb4034440 con 0x5594787cbb20 2024-03-28T01:06:53.750 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.747+0000 7fecc79ab740 1 -- 172.21.15.169:0/2263486931 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5594787cbb20 msgr2=0x5594787d5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:53.750 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.747+0000 7fecc79ab740 1 --2- 172.21.15.169:0/2263486931 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5594787cbb20 0x5594787d5c00 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fecb4006010 tx=0x7fecb402bc50 comp rx=0 tx=0).stop 2024-03-28T01:06:53.751 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.748+0000 7fecc79ab740 1 -- 172.21.15.169:0/2263486931 shutdown_connections 2024-03-28T01:06:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.748+0000 7fecc79ab740 1 --2- 172.21.15.169:0/2263486931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787d1680 0x5594787d6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.748+0000 7fecc79ab740 1 --2- 172.21.15.169:0/2263486931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594787cc510 0x5594787dd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.748+0000 7fecc79ab740 1 --2- 172.21.15.169:0/2263486931 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5594787cbb20 0x5594787d5c00 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.748+0000 7fecc79ab740 1 -- 172.21.15.169:0/2263486931 >> 172.21.15.169:0/2263486931 conn(0x5594787c7090 msgr2=0x5594787c7470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.748+0000 7fecc79ab740 1 -- 172.21.15.169:0/2263486931 shutdown_connections 2024-03-28T01:06:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.748+0000 7fecc79ab740 1 -- 172.21.15.169:0/2263486931 wait complete. 2024-03-28T01:06:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.749+0000 7fecc79ab740 1 -- start start 2024-03-28T01:06:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.749+0000 7fecc79ab740 1 Processor -- start 2024-03-28T01:06:53.753 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.749+0000 7fecc79ab740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594787cbb20 0x5594787d8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:53.753 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.749+0000 7fecc79ab740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787cc510 0x5594787d8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:53.753 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.749+0000 7fecc79ab740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5594787d1680 0x5594787db870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:53.753 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.749+0000 7fecc79ab740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc01b7500 con 0x5594787d1680 2024-03-28T01:06:53.753 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.749+0000 7fecc79ab740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc01b8800 con 0x5594787cc510 2024-03-28T01:06:53.753 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.749+0000 7fecc79ab740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc01b7b40 con 0x5594787cbb20 2024-03-28T01:06:53.753 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.749+0000 7fecc79ab740 10 client.0 _ll_register_callbacks cb 0x7fecb8003e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:06:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.749+0000 7fecbdd9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787cc510 0x5594787d8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.749+0000 7fecbdd9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787cc510 0x5594787d8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:42002/0 (socket says 172.21.15.169:42002) 2024-03-28T01:06:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.750+0000 7fecbdd9b640 1 -- 172.21.15.169:0/3110462810 learned_addr learned my addr 172.21.15.169:0/3110462810 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.750+0000 7fecbdd9b640 1 -- 172.21.15.169:0/3110462810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594787cbb20 msgr2=0x5594787d8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.750+0000 7fecbdd9b640 1 --2- 172.21.15.169:0/3110462810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594787cbb20 0x5594787d8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.750+0000 7fecbdd9b640 1 -- 172.21.15.169:0/3110462810 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5594787d1680 msgr2=0x5594787db870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.750+0000 7fecbdd9b640 1 --2- 172.21.15.169:0/3110462810 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5594787d1680 0x5594787db870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.750+0000 7fecbdd9b640 1 -- 172.21.15.169:0/3110462810 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5594787dbe60 con 0x5594787cc510 2024-03-28T01:06:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.750+0000 7fecbdd9b640 1 --2- 172.21.15.169:0/3110462810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787cc510 0x5594787d8850 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fecc00738a0 tx=0x7fecb00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac0e5836134bdf49 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:53.755 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.751+0000 7feca5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:06:53.755 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.752+0000 7feca5ffb640 1 -- 172.21.15.169:0/3110462810 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb0005220 con 0x5594787cc510 2024-03-28T01:06:53.755 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.752+0000 7feca5ffb640 1 -- 172.21.15.169:0/3110462810 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fecb00053c0 con 0x5594787cc510 2024-03-28T01:06:53.755 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.752+0000 7fecc79ab740 1 -- 172.21.15.169:0/3110462810 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fecb40bcd00 con 0x5594787cc510 2024-03-28T01:06:53.756 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.753+0000 7feca5ffb640 1 -- 172.21.15.169:0/3110462810 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fecb00055c0 con 0x5594787cc510 2024-03-28T01:06:53.756 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.753+0000 7fecc79ab740 10 client.27248 fetch_fsmap learned FSMap version 25 2024-03-28T01:06:53.756 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.753+0000 7fecc79ab740 1 -- 172.21.15.169:0/3110462810 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5594787dd180 con 0x5594787cc510 2024-03-28T01:06:53.756 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.753+0000 7feca5ffb640 1 -- 172.21.15.169:0/3110462810 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fecb40bcd00 con 0x5594787cc510 2024-03-28T01:06:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.755+0000 7feca5ffb640 1 -- 172.21.15.169:0/3110462810 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fecb001c040 con 0x5594787cc510 2024-03-28T01:06:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.755+0000 7fecc79ab740 10 client.27248 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:06:53.759 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.755+0000 7fecc79ab740 10 client.27248 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:06:53.759 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.755+0000 7fecc79ab740 20 client.27248 populate_metadata read hostname 'smithi169' 2024-03-28T01:06:53.759 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.755+0000 7fecc79ab740 1 --2- 172.21.15.169:0/3110462810 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55947884ede0 0x559478851200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:53.759 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.755+0000 7fecc79ab740 4 client.27248 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:06:53.759 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.755+0000 7fecc79ab740 1 -- 172.21.15.169:0/3110462810 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5594787383a0 con 0x55947884ede0 2024-03-28T01:06:53.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.757+0000 7fecbe59c640 1 --2- 172.21.15.169:0/3110462810 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55947884ede0 0x559478851200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:53.764 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.761+0000 7fecbe59c640 1 --2- 172.21.15.169:0/3110462810 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55947884ede0 0x559478851200 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=938d31a0e7381e6f server_cookie=94a0dd2cc7b05a02 in_seq=0 out_seq=0 2024-03-28T01:06:53.764 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:06:53.761+0000 7feca5ffb640 10 client.27248 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:06:54.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:53 smithi169 ceph-mon[32274]: pgmap v662: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 803 MiB/s wr, 718 op/s 2024-03-28T01:06:54.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:53 smithi174 ceph-mon[36060]: pgmap v662: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 803 MiB/s wr, 718 op/s 2024-03-28T01:06:54.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:53 smithi177 ceph-mon[36206]: pgmap v662: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 803 MiB/s wr, 718 op/s 2024-03-28T01:06:56.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:55 smithi169 ceph-mon[32274]: pgmap v663: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 600 MiB/s wr, 668 op/s 2024-03-28T01:06:56.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:55 smithi174 ceph-mon[36060]: pgmap v663: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 600 MiB/s wr, 668 op/s 2024-03-28T01:06:56.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:55 smithi177 ceph-mon[36206]: pgmap v663: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 600 MiB/s wr, 668 op/s 2024-03-28T01:06:57.843 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:06:57.843 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:06:58.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:57 smithi169 ceph-mon[32274]: pgmap v664: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 673 MiB/s wr, 972 op/s 2024-03-28T01:06:58.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:57 smithi174 ceph-mon[36060]: pgmap v664: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 673 MiB/s wr, 972 op/s 2024-03-28T01:06:58.142 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.138+0000 7ffa2621c640 1 -- 172.21.15.169:0/1637630867 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa20109730 msgr2=0x7ffa2010dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:58.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.138+0000 7ffa2621c640 1 --2- 172.21.15.169:0/1637630867 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa20109730 0x7ffa2010dfd0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7ffa18009ca0 tx=0x7ffa1802c140 comp rx=0 tx=0).stop 2024-03-28T01:06:58.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.139+0000 7ffa2621c640 1 -- 172.21.15.169:0/1637630867 shutdown_connections 2024-03-28T01:06:58.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.139+0000 7ffa2621c640 1 --2- 172.21.15.169:0/1637630867 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa20109730 0x7ffa2010dfd0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:58.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.139+0000 7ffa2621c640 1 --2- 172.21.15.169:0/1637630867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa20108d70 0x7ffa201091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:58.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.139+0000 7ffa2621c640 1 --2- 172.21.15.169:0/1637630867 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa20072ab0 0x7ffa20072ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:58.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.139+0000 7ffa2621c640 1 -- 172.21.15.169:0/1637630867 >> 172.21.15.169:0/1637630867 conn(0x7ffa2006e580 msgr2=0x7ffa200709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:58.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.141+0000 7ffa2621c640 1 -- 172.21.15.169:0/1637630867 shutdown_connections 2024-03-28T01:06:58.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.141+0000 7ffa2621c640 1 -- 172.21.15.169:0/1637630867 wait complete. 2024-03-28T01:06:58.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.143+0000 7ffa2621c640 1 Processor -- start 2024-03-28T01:06:58.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.145+0000 7ffa2621c640 1 -- start start 2024-03-28T01:06:58.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.145+0000 7ffa2621c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa20072ab0 0x7ffa201a6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:58.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.145+0000 7ffa2621c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa20108d70 0x7ffa201a6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:58.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.145+0000 7ffa2621c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa20109730 0x7ffa201addb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:58.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.145+0000 7ffa2621c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2011c110 con 0x7ffa20109730 2024-03-28T01:06:58.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.145+0000 7ffa2621c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2011bf90 con 0x7ffa20108d70 2024-03-28T01:06:58.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.145+0000 7ffa2621c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2011c290 con 0x7ffa20072ab0 2024-03-28T01:06:58.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.146+0000 7ffa24a19640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa20108d70 0x7ffa201a6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:58.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.146+0000 7ffa24a19640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa20108d70 0x7ffa201a6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33542/0 (socket says 172.21.15.169:33542) 2024-03-28T01:06:58.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.146+0000 7ffa24a19640 1 -- 172.21.15.169:0/2925806197 learned_addr learned my addr 172.21.15.169:0/2925806197 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:58.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.146+0000 7ffa24a19640 1 -- 172.21.15.169:0/2925806197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa20072ab0 msgr2=0x7ffa201a6800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:06:58.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.146+0000 7ffa24a19640 1 --2- 172.21.15.169:0/2925806197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa20072ab0 0x7ffa201a6800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:58.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.146+0000 7ffa24a19640 1 -- 172.21.15.169:0/2925806197 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa20109730 msgr2=0x7ffa201addb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:06:58.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.146+0000 7ffa24a19640 1 --2- 172.21.15.169:0/2925806197 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa20109730 0x7ffa201addb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:58.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.146+0000 7ffa24a19640 1 -- 172.21.15.169:0/2925806197 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa201ae420 con 0x7ffa20108d70 2024-03-28T01:06:58.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.146+0000 7ffa24a19640 1 --2- 172.21.15.169:0/2925806197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa20108d70 0x7ffa201a6d40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ffa1000b450 tx=0x7ffa1000b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:58.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.148+0000 7ffa167fc640 1 -- 172.21.15.169:0/2925806197 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa10015020 con 0x7ffa20108d70 2024-03-28T01:06:58.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.148+0000 7ffa167fc640 1 -- 172.21.15.169:0/2925806197 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa10011d70 con 0x7ffa20108d70 2024-03-28T01:06:58.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.148+0000 7ffa2621c640 1 -- 172.21.15.169:0/2925806197 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa201ae710 con 0x7ffa20108d70 2024-03-28T01:06:58.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.148+0000 7ffa2621c640 1 -- 172.21.15.169:0/2925806197 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa201aece0 con 0x7ffa20108d70 2024-03-28T01:06:58.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.150+0000 7ffa167fc640 1 -- 172.21.15.169:0/2925806197 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1000f340 con 0x7ffa20108d70 2024-03-28T01:06:58.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.152+0000 7ffa2a068740 1 Processor -- start 2024-03-28T01:06:58.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.152+0000 7ffa2a068740 1 -- start start 2024-03-28T01:06:58.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.152+0000 7ffa2a068740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324ecf80 0x5606324ed3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:58.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.152+0000 7ffa2a068740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606324ed970 0x5606324f2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:58.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.152+0000 7ffa2a068740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606324f2ae0 0x5606324f4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:58.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.152+0000 7ffa2a068740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606324523d0 con 0x5606324ed970 2024-03-28T01:06:58.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.152+0000 7ffa2a068740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606323faad0 con 0x5606324ecf80 2024-03-28T01:06:58.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.152+0000 7ffa2a068740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56063244e470 con 0x5606324f2ae0 2024-03-28T01:06:58.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.153+0000 7ffa25a1b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324ecf80 0x5606324ed3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:58.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.153+0000 7ffa25a1b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324ecf80 0x5606324ed3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33544/0 (socket says 172.21.15.169:33544) 2024-03-28T01:06:58.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.153+0000 7ffa25a1b640 1 -- 172.21.15.169:0/80887644 learned_addr learned my addr 172.21.15.169:0/80887644 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:58.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:57 smithi177 ceph-mon[36206]: pgmap v664: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 673 MiB/s wr, 972 op/s 2024-03-28T01:06:58.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.154+0000 7ffa25a1b640 1 -- 172.21.15.169:0/80887644 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606324f2ae0 msgr2=0x5606324f4f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:06:58.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.154+0000 7ffa25a1b640 1 --2- 172.21.15.169:0/80887644 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606324f2ae0 0x5606324f4f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:58.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.154+0000 7ffa25a1b640 1 -- 172.21.15.169:0/80887644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606324ed970 msgr2=0x5606324f2470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:06:58.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.154+0000 7ffa25a1b640 1 --2- 172.21.15.169:0/80887644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606324ed970 0x5606324f2470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:58.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.154+0000 7ffa25a1b640 1 -- 172.21.15.169:0/80887644 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606322307d0 con 0x5606324ecf80 2024-03-28T01:06:58.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.155+0000 7ffa167fc640 1 -- 172.21.15.169:0/2925806197 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ffa10018020 con 0x7ffa20108d70 2024-03-28T01:06:58.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.155+0000 7ffa25a1b640 1 --2- 172.21.15.169:0/80887644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324ecf80 0x5606324ed3a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ffa18037040 tx=0x7ffa180323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61e59244c637855 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:58.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.163+0000 7ffa0b7fe640 1 -- 172.21.15.169:0/80887644 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa18052020 con 0x5606324ecf80 2024-03-28T01:06:58.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.167+0000 7ffa0b7fe640 1 -- 172.21.15.169:0/80887644 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa180098e0 con 0x5606324ecf80 2024-03-28T01:06:58.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.168+0000 7ffa2a068740 1 -- 172.21.15.169:0/80887644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324ecf80 msgr2=0x5606324ed3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:06:58.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.168+0000 7ffa2a068740 1 --2- 172.21.15.169:0/80887644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324ecf80 0x5606324ed3a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ffa18037040 tx=0x7ffa180323d0 comp rx=0 tx=0).stop 2024-03-28T01:06:58.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.170+0000 7ffa167fc640 1 --2- 172.21.15.169:0/2925806197 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffa0c0a2a00 0x7ffa0c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:58.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.170+0000 7ffa167fc640 1 -- 172.21.15.169:0/2925806197 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ffa100ce830 con 0x7ffa20108d70 2024-03-28T01:06:58.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.170+0000 7ffa2a068740 1 -- 172.21.15.169:0/80887644 shutdown_connections 2024-03-28T01:06:58.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.170+0000 7ffa2a068740 1 --2- 172.21.15.169:0/80887644 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606324f2ae0 0x5606324f4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:58.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.170+0000 7ffa2a068740 1 --2- 172.21.15.169:0/80887644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606324ed970 0x5606324f2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:58.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.170+0000 7ffa2a068740 1 --2- 172.21.15.169:0/80887644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324ecf80 0x5606324ed3a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:58.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.170+0000 7ffa2a068740 1 -- 172.21.15.169:0/80887644 >> 172.21.15.169:0/80887644 conn(0x5606324e84f0 msgr2=0x5606324e88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:06:58.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.170+0000 7ffa2a068740 1 -- 172.21.15.169:0/80887644 shutdown_connections 2024-03-28T01:06:58.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.170+0000 7ffa2a068740 1 -- 172.21.15.169:0/80887644 wait complete. 2024-03-28T01:06:58.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.171+0000 7ffa25a1b640 1 --2- 172.21.15.169:0/2925806197 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffa0c0a2a00 0x7ffa0c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:58.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.171+0000 7ffa2a068740 1 -- start start 2024-03-28T01:06:58.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.171+0000 7ffa2a068740 1 Processor -- start 2024-03-28T01:06:58.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.172+0000 7ffa2a068740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606324ecf80 0x5606324f73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:58.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.172+0000 7ffa2a068740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324ed970 0x5606324f7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:58.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.172+0000 7ffa2a068740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606324f2ae0 0x5606324fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:58.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.172+0000 7ffa2a068740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606324f5a80 con 0x5606324ecf80 2024-03-28T01:06:58.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.172+0000 7ffa2a068740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x56063240e060 con 0x5606324ed970 2024-03-28T01:06:58.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.172+0000 7ffa2a068740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606324f5c00 con 0x5606324f2ae0 2024-03-28T01:06:58.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.172+0000 7ffa2a068740 10 client.0 _ll_register_callbacks cb 0x5606323f06a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:06:58.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.172+0000 7ffa25a1b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606324f2ae0 0x5606324fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:58.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.172+0000 7ffa25a1b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606324f2ae0 0x5606324fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56114/0 (socket says 172.21.15.169:56114) 2024-03-28T01:06:58.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.172+0000 7ffa25a1b640 1 -- 172.21.15.169:0/1610964775 learned_addr learned my addr 172.21.15.169:0/1610964775 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:06:58.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.173+0000 7ffa24a19640 1 --2- 172.21.15.169:0/1610964775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324ed970 0x5606324f7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:58.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.173+0000 7ffa25a1b640 1 --2- 172.21.15.169:0/2925806197 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffa0c0a2a00 0x7ffa0c0a4ec0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ffa18003ea0 tx=0x7ffa1802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:58.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.173+0000 7ffa25a1b640 1 -- 172.21.15.169:0/1610964775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324ed970 msgr2=0x5606324f7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:06:58.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.173+0000 7ffa25a1b640 1 --2- 172.21.15.169:0/1610964775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324ed970 0x5606324f7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:58.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.173+0000 7ffa25a1b640 1 -- 172.21.15.169:0/1610964775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606324ecf80 msgr2=0x5606324f73c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:06:58.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.173+0000 7ffa25a1b640 1 --2- 172.21.15.169:0/1610964775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606324ecf80 0x5606324f73c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:06:58.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.173+0000 7ffa25a1b640 1 -- 172.21.15.169:0/1610964775 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5606321663f0 con 0x5606324f2ae0 2024-03-28T01:06:58.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.173+0000 7ffa25a1b640 1 --2- 172.21.15.169:0/1610964775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606324f2ae0 0x5606324fe940 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ffa18053000 tx=0x7ffa180323d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75d0b0e1d8d8886c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:06:58.181 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.173+0000 7ffa08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:06:58.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.173+0000 7ffa08ff9640 1 -- 172.21.15.169:0/1610964775 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa18052040 con 0x5606324f2ae0 2024-03-28T01:06:58.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.174+0000 7ffa2a068740 1 -- 172.21.15.169:0/1610964775 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffa180098e0 con 0x5606324f2ae0 2024-03-28T01:06:58.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.174+0000 7ffa08ff9640 1 -- 172.21.15.169:0/1610964775 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ffa18064440 con 0x5606324f2ae0 2024-03-28T01:06:58.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.174+0000 7ffa08ff9640 1 -- 172.21.15.169:0/1610964775 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa18072050 con 0x5606324f2ae0 2024-03-28T01:06:58.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.174+0000 7ffa2a068740 10 client.27193 fetch_fsmap learned FSMap version 25 2024-03-28T01:06:58.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.174+0000 7ffa2a068740 1 -- 172.21.15.169:0/1610964775 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5606321260f0 con 0x5606324f2ae0 2024-03-28T01:06:58.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.174+0000 7ffa08ff9640 1 -- 172.21.15.169:0/1610964775 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa180098e0 con 0x5606324f2ae0 2024-03-28T01:06:58.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.174+0000 7ffa08ff9640 1 -- 172.21.15.169:0/1610964775 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ffa18060070 con 0x5606324f2ae0 2024-03-28T01:06:58.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.174+0000 7ffa2a068740 10 client.27193 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:06:58.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.174+0000 7ffa2a068740 10 client.27193 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:06:58.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.174+0000 7ffa2a068740 20 client.27193 populate_metadata read hostname 'smithi169' 2024-03-28T01:06:58.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.174+0000 7ffa2a068740 1 --2- 172.21.15.169:0/1610964775 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560632503080 0x560632505540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:06:58.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.174+0000 7ffa2a068740 4 client.27193 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:06:58.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.174+0000 7ffa2a068740 1 -- 172.21.15.169:0/1610964775 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56063245f910 con 0x560632503080 2024-03-28T01:06:58.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.176+0000 7ffa2521a640 1 --2- 172.21.15.169:0/1610964775 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560632503080 0x560632505540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:06:58.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.177+0000 7ffa2521a640 1 --2- 172.21.15.169:0/1610964775 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560632503080 0x560632505540 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63e846cb3d600667 server_cookie=b5c475c1930ed612 in_seq=0 out_seq=0 2024-03-28T01:06:58.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:06:58.177+0000 7ffa08ff9640 10 client.27193 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:00.053 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:06:59 smithi169 ceph-mon[32274]: pgmap v665: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 608 MiB/s wr, 1.03k op/s 2024-03-28T01:07:00.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:06:59 smithi174 ceph-mon[36060]: pgmap v665: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 608 MiB/s wr, 1.03k op/s 2024-03-28T01:07:00.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:06:59 smithi177 ceph-mon[36206]: pgmap v665: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 608 MiB/s wr, 1.03k op/s 2024-03-28T01:07:00.304 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.301+0000 7feca5ffb640 1 -- 172.21.15.169:0/3110462810 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5594787383a0 con 0x55947884ede0 2024-03-28T01:07:00.304 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.301+0000 7feca5ffb640 10 client.27248 handle_command_reply: tid=0 2024-03-28T01:07:00.304 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.301+0000 7fecc79ab740 1 client.27248 shutdown 2024-03-28T01:07:00.304 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.301+0000 7fecc79ab740 1 -- 172.21.15.169:0/3110462810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787cc510 msgr2=0x5594787d8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:00.304 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.301+0000 7fecc79ab740 1 --2- 172.21.15.169:0/3110462810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787cc510 0x5594787d8850 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fecc00738a0 tx=0x7fecb00046c0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.301+0000 7fecc79ab740 1 -- 172.21.15.169:0/3110462810 shutdown_connections 2024-03-28T01:07:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.301+0000 7fecc79ab740 1 --2- 172.21.15.169:0/3110462810 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55947884ede0 0x559478851200 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.301+0000 7fecc79ab740 1 --2- 172.21.15.169:0/3110462810 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5594787d1680 0x5594787db870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.301+0000 7fecc79ab740 1 --2- 172.21.15.169:0/3110462810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5594787cc510 0x5594787d8850 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.301+0000 7fecc79ab740 1 --2- 172.21.15.169:0/3110462810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594787cbb20 0x5594787d8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.301+0000 7fecc79ab740 1 -- 172.21.15.169:0/3110462810 >> 172.21.15.169:0/3110462810 conn(0x5594787c7090 msgr2=0x5594787d5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.301+0000 7feca5ffb640 0 client.27248 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.301+0000 7fecc79ab740 1 -- 172.21.15.169:0/3110462810 shutdown_connections 2024-03-28T01:07:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.301+0000 7fecc79ab740 1 -- 172.21.15.169:0/3110462810 wait complete. 2024-03-28T01:07:00.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.301+0000 7ffa08ff9640 1 -- 172.21.15.169:0/1610964775 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56063245f910 con 0x560632503080 2024-03-28T01:07:00.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.301+0000 7ffa08ff9640 10 client.27193 handle_command_reply: tid=0 2024-03-28T01:07:00.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.302+0000 7ffa2a068740 1 client.27193 shutdown 2024-03-28T01:07:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.302+0000 7fecc79ab740 20 client.27248 trim_cache size 0 max 16384 2024-03-28T01:07:00.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.302+0000 7ffa2a068740 1 -- 172.21.15.169:0/1610964775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606324f2ae0 msgr2=0x5606324fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:00.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.302+0000 7ffa2a068740 1 --2- 172.21.15.169:0/1610964775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606324f2ae0 0x5606324fe940 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ffa18053000 tx=0x7ffa180323d0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.302+0000 7ffa2a068740 1 -- 172.21.15.169:0/1610964775 shutdown_connections 2024-03-28T01:07:00.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.302+0000 7ffa2a068740 1 --2- 172.21.15.169:0/1610964775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606324f2ae0 0x5606324fe940 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.302+0000 7ffa2a068740 1 --2- 172.21.15.169:0/1610964775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324ed970 0x5606324f7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.302+0000 7ffa2a068740 1 --2- 172.21.15.169:0/1610964775 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x560632503080 0x560632505540 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.302+0000 7ffa2a068740 1 --2- 172.21.15.169:0/1610964775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606324ecf80 0x5606324f73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.302+0000 7ffa2a068740 1 -- 172.21.15.169:0/1610964775 >> 172.21.15.169:0/1610964775 conn(0x5606324e84f0 msgr2=0x5606324e77b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:00.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.302+0000 7ffa08ff9640 0 client.27193 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:00.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.302+0000 7ffa2a068740 1 -- 172.21.15.169:0/1610964775 shutdown_connections 2024-03-28T01:07:00.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.302+0000 7ffa2a068740 1 -- 172.21.15.169:0/1610964775 wait complete. 2024-03-28T01:07:00.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.302+0000 7ffa2a068740 20 client.27193 trim_cache size 0 max 16384 2024-03-28T01:07:00.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:07:00.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:07:00.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:07:00.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:07:00.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.324+0000 7fecc4ce8640 1 -- 172.21.15.169:0/155654157 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7feca80a2a00 msgr2=0x7feca80a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:00.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.324+0000 7fecc4ce8640 1 --2- 172.21.15.169:0/155654157 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7feca80a2a00 0x7feca80a4ec0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fecc01a7b60 tx=0x7fecb0013830 comp rx=0 tx=0).stop 2024-03-28T01:07:00.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.324+0000 7fecc4ce8640 1 -- 172.21.15.169:0/155654157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecc0072970 msgr2=0x7fecc01a6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:00.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.324+0000 7fecc4ce8640 1 --2- 172.21.15.169:0/155654157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecc0072970 0x7fecc01a6650 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fecb4000c00 tx=0x7fecb40052f0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.329 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.325+0000 7fecc4ce8640 1 -- 172.21.15.169:0/155654157 shutdown_connections 2024-03-28T01:07:00.329 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.325+0000 7fecc4ce8640 1 --2- 172.21.15.169:0/155654157 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7feca80a2a00 0x7feca80a4ec0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.329 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.326+0000 7fecc4ce8640 1 --2- 172.21.15.169:0/155654157 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fecc01093d0 0x7fecc01adc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.329 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.326+0000 7fecc4ce8640 1 --2- 172.21.15.169:0/155654157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecc00732d0 0x7fecc01a6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.329 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.326+0000 7fecc4ce8640 1 --2- 172.21.15.169:0/155654157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecc0072970 0x7fecc01a6650 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.329 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.326+0000 7fecc4ce8640 1 -- 172.21.15.169:0/155654157 >> 172.21.15.169:0/155654157 conn(0x7fecc006e510 msgr2=0x7fecc0070120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:00.330 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.326+0000 7fecc4ce8640 1 -- 172.21.15.169:0/155654157 shutdown_connections 2024-03-28T01:07:00.330 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:00.326+0000 7fecc4ce8640 1 -- 172.21.15.169:0/155654157 wait complete. 2024-03-28T01:07:00.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.346+0000 7ffa2a068740 1 Processor -- start 2024-03-28T01:07:00.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.349+0000 7ffa2a068740 1 -- start start 2024-03-28T01:07:00.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.350+0000 7ffa2a068740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606324ecf80 0x5606324ebe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:00.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.350+0000 7ffa2a068740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606324ed970 0x5606324fe8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:00.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.350+0000 7ffa2a068740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324f2ae0 0x560632509e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:00.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.350+0000 7ffa2a068740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa040120d0 con 0x5606324ed970 2024-03-28T01:07:00.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.350+0000 7ffa2a068740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0400fbd0 con 0x5606324f2ae0 2024-03-28T01:07:00.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.350+0000 7ffa2a068740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0400dc70 con 0x5606324ecf80 2024-03-28T01:07:00.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.350+0000 7ffa25a1b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324f2ae0 0x560632509e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:00.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.350+0000 7ffa25a1b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324f2ae0 0x560632509e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33566/0 (socket says 172.21.15.169:33566) 2024-03-28T01:07:00.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.350+0000 7ffa25a1b640 1 -- 172.21.15.169:0/3274148871 learned_addr learned my addr 172.21.15.169:0/3274148871 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:00.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.350+0000 7ffa2521a640 1 --2- 172.21.15.169:0/3274148871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606324ecf80 0x5606324ebe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:00.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.351+0000 7ffa24a19640 1 --2- 172.21.15.169:0/3274148871 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606324ed970 0x5606324fe8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:00.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.351+0000 7ffa25a1b640 1 -- 172.21.15.169:0/3274148871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606324ecf80 msgr2=0x5606324ebe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:00.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.351+0000 7ffa25a1b640 1 --2- 172.21.15.169:0/3274148871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606324ecf80 0x5606324ebe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.351+0000 7ffa25a1b640 1 -- 172.21.15.169:0/3274148871 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606324ed970 msgr2=0x5606324fe8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:00.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.351+0000 7ffa25a1b640 1 --2- 172.21.15.169:0/3274148871 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606324ed970 0x5606324fe8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.351+0000 7ffa25a1b640 1 -- 172.21.15.169:0/3274148871 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560632505bc0 con 0x5606324f2ae0 2024-03-28T01:07:00.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.352+0000 7ffa25a1b640 1 --2- 172.21.15.169:0/3274148871 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324f2ae0 0x560632509e40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ffa18058800 tx=0x7ffa1802c140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ed3bd5a831bd5ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:00.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.353+0000 7ffa0affd640 1 -- 172.21.15.169:0/3274148871 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1807b070 con 0x5606324f2ae0 2024-03-28T01:07:00.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.354+0000 7ffa0affd640 1 -- 172.21.15.169:0/3274148871 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa18069070 con 0x5606324f2ae0 2024-03-28T01:07:00.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.355+0000 7ffa2a068740 1 -- 172.21.15.169:0/3274148871 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324f2ae0 msgr2=0x560632509e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:00.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.355+0000 7ffa2a068740 1 --2- 172.21.15.169:0/3274148871 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324f2ae0 0x560632509e40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ffa18058800 tx=0x7ffa1802c140 comp rx=0 tx=0).stop 2024-03-28T01:07:00.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.359+0000 7ffa2a068740 1 -- 172.21.15.169:0/3274148871 shutdown_connections 2024-03-28T01:07:00.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.359+0000 7ffa2a068740 1 --2- 172.21.15.169:0/3274148871 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324f2ae0 0x560632509e40 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.359+0000 7ffa2a068740 1 --2- 172.21.15.169:0/3274148871 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606324ed970 0x5606324fe8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.359+0000 7ffa2a068740 1 --2- 172.21.15.169:0/3274148871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606324ecf80 0x5606324ebe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.359+0000 7ffa2a068740 1 -- 172.21.15.169:0/3274148871 >> 172.21.15.169:0/3274148871 conn(0x5606324e84f0 msgr2=0x5606324eb750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:00.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.373+0000 7ffa2a068740 1 -- 172.21.15.169:0/3274148871 shutdown_connections 2024-03-28T01:07:00.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.379+0000 7ffa2a068740 1 -- 172.21.15.169:0/3274148871 wait complete. 2024-03-28T01:07:00.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.379+0000 7ffa2a068740 1 -- start start 2024-03-28T01:07:00.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.380+0000 7ffa2a068740 1 Processor -- start 2024-03-28T01:07:00.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.380+0000 7ffa2a068740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606324ecf80 0x5606324fbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:00.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.380+0000 7ffa2a068740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324ed970 0x5606324fc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:00.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.380+0000 7ffa2a068740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606324f2ae0 0x5606324fd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:00.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.380+0000 7ffa2a068740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0400e270 con 0x5606324f2ae0 2024-03-28T01:07:00.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.380+0000 7ffa2a068740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0400f570 con 0x5606324ed970 2024-03-28T01:07:00.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.380+0000 7ffa2a068740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0400e8b0 con 0x5606324ecf80 2024-03-28T01:07:00.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.380+0000 7ffa2a068740 10 client.0 _ll_register_callbacks cb 0x7ffa1805b030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:00.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.381+0000 7ffa2521a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606324ecf80 0x5606324fbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:00.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.383+0000 7ffa25a1b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606324f2ae0 0x5606324fd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:00.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.383+0000 7ffa24a19640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324ed970 0x5606324fc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:00.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.383+0000 7ffa25a1b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606324f2ae0 0x5606324fd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:35848/0 (socket says 172.21.15.169:35848) 2024-03-28T01:07:00.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.383+0000 7ffa25a1b640 1 -- 172.21.15.169:0/2260929530 learned_addr learned my addr 172.21.15.169:0/2260929530 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:00.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.383+0000 7ffa25a1b640 1 -- 172.21.15.169:0/2260929530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606324ecf80 msgr2=0x5606324fbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:00.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.383+0000 7ffa25a1b640 1 --2- 172.21.15.169:0/2260929530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606324ecf80 0x5606324fbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.383+0000 7ffa25a1b640 1 -- 172.21.15.169:0/2260929530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324ed970 msgr2=0x5606324fc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:00.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.383+0000 7ffa25a1b640 1 --2- 172.21.15.169:0/2260929530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324ed970 0x5606324fc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.383+0000 7ffa25a1b640 1 -- 172.21.15.169:0/2260929530 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5606324fd9b0 con 0x5606324f2ae0 2024-03-28T01:07:00.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.384+0000 7ffa25a1b640 1 --2- 172.21.15.169:0/2260929530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606324f2ae0 0x5606324fd3c0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7ffa18079040 tx=0x7ffa18005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80085b34491847e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:00.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.389+0000 7ffa08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:07:00.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.389+0000 7ffa08ff9640 1 -- 172.21.15.169:0/2260929530 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1807b070 con 0x5606324f2ae0 2024-03-28T01:07:00.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.389+0000 7ffa08ff9640 1 -- 172.21.15.169:0/2260929530 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ffa18080460 con 0x5606324f2ae0 2024-03-28T01:07:00.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.389+0000 7ffa2a068740 1 -- 172.21.15.169:0/2260929530 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffa18069070 con 0x5606324f2ae0 2024-03-28T01:07:00.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.403+0000 7ffa08ff9640 1 -- 172.21.15.169:0/2260929530 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa18080660 con 0x5606324f2ae0 2024-03-28T01:07:00.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.404+0000 7ffa2a068740 10 client.17394 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:00.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.404+0000 7ffa2a068740 1 -- 172.21.15.169:0/2260929530 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56063255ea10 con 0x5606324f2ae0 2024-03-28T01:07:00.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.411+0000 7ffa08ff9640 1 -- 172.21.15.169:0/2260929530 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa18069070 con 0x5606324f2ae0 2024-03-28T01:07:00.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.411+0000 7ffa08ff9640 1 -- 172.21.15.169:0/2260929530 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ffa18060070 con 0x5606324f2ae0 2024-03-28T01:07:00.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.411+0000 7ffa2a068740 10 client.17394 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:00.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.411+0000 7ffa2a068740 10 client.17394 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:07:00.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.411+0000 7ffa2a068740 20 client.17394 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:00.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.411+0000 7ffa2a068740 1 --2- 172.21.15.169:0/2260929530 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56063255ef90 0x560632561380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:00.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.411+0000 7ffa2a068740 4 client.17394 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:07:00.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.411+0000 7ffa2a068740 1 -- 172.21.15.169:0/2260929530 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5606324f28e0 con 0x56063255ef90 2024-03-28T01:07:00.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.414+0000 7ffa2521a640 1 --2- 172.21.15.169:0/2260929530 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56063255ef90 0x560632561380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:00.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.423+0000 7ffa2521a640 1 --2- 172.21.15.169:0/2260929530 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56063255ef90 0x560632561380 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=927378ab6c2305d7 server_cookie=8ef180ad66aa58d2 in_seq=0 out_seq=0 2024-03-28T01:07:00.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.424+0000 7ffa08ff9640 10 client.17394 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:00.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.427+0000 7ffa08ff9640 1 -- 172.21.15.169:0/2260929530 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5606324f28e0 con 0x56063255ef90 2024-03-28T01:07:00.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.428+0000 7ffa08ff9640 10 client.17394 handle_command_reply: tid=0 2024-03-28T01:07:00.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.435+0000 7ffa2a068740 1 client.17394 shutdown 2024-03-28T01:07:00.440 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.436+0000 7ffa2a068740 1 -- 172.21.15.169:0/2260929530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606324f2ae0 msgr2=0x5606324fd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:00.440 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.436+0000 7ffa2a068740 1 --2- 172.21.15.169:0/2260929530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606324f2ae0 0x5606324fd3c0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7ffa18079040 tx=0x7ffa18005ef0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.440 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.436+0000 7ffa2a068740 1 -- 172.21.15.169:0/2260929530 shutdown_connections 2024-03-28T01:07:00.440 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.436+0000 7ffa2a068740 1 --2- 172.21.15.169:0/2260929530 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56063255ef90 0x560632561380 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.440 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.436+0000 7ffa2a068740 1 --2- 172.21.15.169:0/2260929530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5606324f2ae0 0x5606324fd3c0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.440 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.436+0000 7ffa2a068740 1 --2- 172.21.15.169:0/2260929530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5606324ed970 0x5606324fc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.440 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.436+0000 7ffa2a068740 1 --2- 172.21.15.169:0/2260929530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606324ecf80 0x5606324fbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.440 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.436+0000 7ffa2a068740 1 -- 172.21.15.169:0/2260929530 >> 172.21.15.169:0/2260929530 conn(0x5606324e84f0 msgr2=0x5606324ebfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:00.440 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.437+0000 7ffa08ff9640 0 client.17394 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:00.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.438+0000 7ffa2a068740 1 -- 172.21.15.169:0/2260929530 shutdown_connections 2024-03-28T01:07:00.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.438+0000 7ffa2a068740 1 -- 172.21.15.169:0/2260929530 wait complete. 2024-03-28T01:07:00.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.438+0000 7ffa2a068740 20 client.17394 trim_cache size 0 max 16384 2024-03-28T01:07:00.443 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:07:00.443 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "no active scrubs running", 2024-03-28T01:07:00.443 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": {} 2024-03-28T01:07:00.443 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:07:00.454 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.450+0000 7ffa2621c640 1 -- 172.21.15.169:0/2925806197 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffa0c0a2a00 msgr2=0x7ffa0c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:00.454 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.450+0000 7ffa2621c640 1 --2- 172.21.15.169:0/2925806197 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffa0c0a2a00 0x7ffa0c0a4ec0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ffa18003ea0 tx=0x7ffa1802e040 comp rx=0 tx=0).stop 2024-03-28T01:07:00.454 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.450+0000 7ffa2621c640 1 -- 172.21.15.169:0/2925806197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa20108d70 msgr2=0x7ffa201a6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:00.454 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.450+0000 7ffa2621c640 1 --2- 172.21.15.169:0/2925806197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa20108d70 0x7ffa201a6d40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ffa1000b450 tx=0x7ffa1000b910 comp rx=0 tx=0).stop 2024-03-28T01:07:00.455 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.452+0000 7ffa2621c640 1 -- 172.21.15.169:0/2925806197 shutdown_connections 2024-03-28T01:07:00.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.452+0000 7ffa2621c640 1 --2- 172.21.15.169:0/2925806197 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffa0c0a2a00 0x7ffa0c0a4ec0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.452+0000 7ffa2621c640 1 --2- 172.21.15.169:0/2925806197 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa20109730 0x7ffa201addb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.452+0000 7ffa2621c640 1 --2- 172.21.15.169:0/2925806197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa20108d70 0x7ffa201a6d40 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.452+0000 7ffa2621c640 1 --2- 172.21.15.169:0/2925806197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa20072ab0 0x7ffa201a6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.456 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.452+0000 7ffa2621c640 1 -- 172.21.15.169:0/2925806197 >> 172.21.15.169:0/2925806197 conn(0x7ffa2006e580 msgr2=0x7ffa20070250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:00.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.456+0000 7ffa2621c640 1 -- 172.21.15.169:0/2925806197 shutdown_connections 2024-03-28T01:07:00.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.456+0000 7ffa2621c640 1 -- 172.21.15.169:0/2925806197 wait complete. 2024-03-28T01:07:00.511 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T01:07:00.512 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T01:07:00.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.768+0000 7ffab359e640 1 -- 172.21.15.169:0/2307655244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffab410ac80 msgr2=0x7ffab410d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:00.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.768+0000 7ffab359e640 1 --2- 172.21.15.169:0/2307655244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffab410ac80 0x7ffab410d0f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4009900 tx=0x7ffaa402bc50 comp rx=0 tx=0).stop 2024-03-28T01:07:00.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.769+0000 7ffab359e640 1 -- 172.21.15.169:0/2307655244 shutdown_connections 2024-03-28T01:07:00.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.769+0000 7ffab359e640 1 --2- 172.21.15.169:0/2307655244 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffab410d760 0x7ffab41185e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.769+0000 7ffab359e640 1 --2- 172.21.15.169:0/2307655244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffab410ac80 0x7ffab410d0f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.769+0000 7ffab359e640 1 --2- 172.21.15.169:0/2307655244 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffab40734a0 0x7ffab410a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.769+0000 7ffab359e640 1 -- 172.21.15.169:0/2307655244 >> 172.21.15.169:0/2307655244 conn(0x7ffab406e3d0 msgr2=0x7ffab4070810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:00.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.773+0000 7ffab359e640 1 -- 172.21.15.169:0/2307655244 shutdown_connections 2024-03-28T01:07:00.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.773+0000 7ffab359e640 1 -- 172.21.15.169:0/2307655244 wait complete. 2024-03-28T01:07:00.776 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.774+0000 7ffab359e640 1 Processor -- start 2024-03-28T01:07:00.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.774+0000 7ffab359e640 1 -- start start 2024-03-28T01:07:00.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.774+0000 7ffab359e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffab40734a0 0x7ffab4115270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:00.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.774+0000 7ffab359e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffab410ac80 0x7ffab41157b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:00.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.774+0000 7ffab359e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffab410d760 0x7ffab41b66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:00.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.774+0000 7ffab359e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab411bfb0 con 0x7ffab410ac80 2024-03-28T01:07:00.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.774+0000 7ffab359e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab411be30 con 0x7ffab40734a0 2024-03-28T01:07:00.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.774+0000 7ffab359e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab411c130 con 0x7ffab410d760 2024-03-28T01:07:00.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.774+0000 7ffab259c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffab40734a0 0x7ffab4115270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:00.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.774+0000 7ffab259c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffab40734a0 0x7ffab4115270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33588/0 (socket says 172.21.15.169:33588) 2024-03-28T01:07:00.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.774+0000 7ffab259c640 1 -- 172.21.15.169:0/3487612420 learned_addr learned my addr 172.21.15.169:0/3487612420 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:00.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.774+0000 7ffab2d9d640 1 --2- 172.21.15.169:0/3487612420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffab410d760 0x7ffab41b66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:00.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.774+0000 7ffab1d9b640 1 --2- 172.21.15.169:0/3487612420 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffab410ac80 0x7ffab41157b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:00.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.775+0000 7ffab259c640 1 -- 172.21.15.169:0/3487612420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffab410d760 msgr2=0x7ffab41b66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:00.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.775+0000 7ffab259c640 1 --2- 172.21.15.169:0/3487612420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffab410d760 0x7ffab41b66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.775+0000 7ffab259c640 1 -- 172.21.15.169:0/3487612420 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffab410ac80 msgr2=0x7ffab41157b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:00.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.775+0000 7ffab259c640 1 --2- 172.21.15.169:0/3487612420 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffab410ac80 0x7ffab41157b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:00.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.775+0000 7ffab259c640 1 -- 172.21.15.169:0/3487612420 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab41b6c00 con 0x7ffab40734a0 2024-03-28T01:07:00.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.775+0000 7ffab259c640 1 --2- 172.21.15.169:0/3487612420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffab40734a0 0x7ffab4115270 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800d770 tx=0x7ffaa800dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:00.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.775+0000 7ffaa37fe640 1 -- 172.21.15.169:0/3487612420 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8017070 con 0x7ffab40734a0 2024-03-28T01:07:00.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.775+0000 7ffaa37fe640 1 -- 172.21.15.169:0/3487612420 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffaa8009c30 con 0x7ffab40734a0 2024-03-28T01:07:00.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.775+0000 7ffaa37fe640 1 -- 172.21.15.169:0/3487612420 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa800f620 con 0x7ffab40734a0 2024-03-28T01:07:00.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.776+0000 7ffab359e640 1 -- 172.21.15.169:0/3487612420 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab41b6dd0 con 0x7ffab40734a0 2024-03-28T01:07:00.780 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.776+0000 7ffab359e640 1 -- 172.21.15.169:0/3487612420 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab41b7180 con 0x7ffab40734a0 2024-03-28T01:07:00.782 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.777+0000 7ffaa17fa640 1 -- 172.21.15.169:0/3487612420 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa98005290 con 0x7ffab40734a0 2024-03-28T01:07:00.785 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:00 smithi169 ceph-mon[32274]: pgmap v666: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 594 MiB/s wr, 1.02k op/s 2024-03-28T01:07:00.786 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.782+0000 7ffaa37fe640 1 -- 172.21.15.169:0/3487612420 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ffaa801a070 con 0x7ffab40734a0 2024-03-28T01:07:00.787 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.783+0000 7ffaa37fe640 1 --2- 172.21.15.169:0/3487612420 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffa9c0a2a00 0x7ffa9c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:00.788 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.784+0000 7ffaa37fe640 1 -- 172.21.15.169:0/3487612420 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ffaa80ceb20 con 0x7ffab40734a0 2024-03-28T01:07:00.788 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.785+0000 7ffab1d9b640 1 --2- 172.21.15.169:0/3487612420 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffa9c0a2a00 0x7ffa9c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:00.788 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.786+0000 7ffab1d9b640 1 --2- 172.21.15.169:0/3487612420 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffa9c0a2a00 0x7ffa9c0a4ec0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ffaa402c260 tx=0x7ffaa4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:00.792 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:00.789+0000 7ffaa37fe640 1 -- 172.21.15.169:0/3487612420 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffaa801b3b0 con 0x7ffab40734a0 2024-03-28T01:07:01.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:00 smithi174 ceph-mon[36060]: pgmap v666: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 594 MiB/s wr, 1.02k op/s 2024-03-28T01:07:01.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:00 smithi177 ceph-mon[36206]: pgmap v666: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 594 MiB/s wr, 1.02k op/s 2024-03-28T01:07:01.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.381+0000 7ffaa17fa640 1 -- 172.21.15.169:0/3487612420 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa98005580 con 0x7ffab40734a0 2024-03-28T01:07:01.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.386+0000 7ffaa37fe640 1 -- 172.21.15.169:0/3487612420 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7ffaa809b8c0 con 0x7ffab40734a0 2024-03-28T01:07:01.393 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T01:07:01.393 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T01:07:01.394 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T01:07:01.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.397+0000 7ffab359e640 1 -- 172.21.15.169:0/3487612420 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffa9c0a2a00 msgr2=0x7ffa9c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:01.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.397+0000 7ffab359e640 1 --2- 172.21.15.169:0/3487612420 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffa9c0a2a00 0x7ffa9c0a4ec0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ffaa402c260 tx=0x7ffaa4005fd0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.397+0000 7ffab359e640 1 -- 172.21.15.169:0/3487612420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffab40734a0 msgr2=0x7ffab4115270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:01.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.397+0000 7ffab359e640 1 --2- 172.21.15.169:0/3487612420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffab40734a0 0x7ffab4115270 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800d770 tx=0x7ffaa800dc30 comp rx=0 tx=0).stop 2024-03-28T01:07:01.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.400+0000 7ffab359e640 1 -- 172.21.15.169:0/3487612420 shutdown_connections 2024-03-28T01:07:01.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.400+0000 7ffab359e640 1 --2- 172.21.15.169:0/3487612420 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ffa9c0a2a00 0x7ffa9c0a4ec0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.400+0000 7ffab359e640 1 --2- 172.21.15.169:0/3487612420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffab410d760 0x7ffab41b66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.400+0000 7ffab359e640 1 --2- 172.21.15.169:0/3487612420 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffab410ac80 0x7ffab41157b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.400+0000 7ffab359e640 1 --2- 172.21.15.169:0/3487612420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffab40734a0 0x7ffab4115270 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.400+0000 7ffab359e640 1 -- 172.21.15.169:0/3487612420 >> 172.21.15.169:0/3487612420 conn(0x7ffab406e3d0 msgr2=0x7ffab406e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:01.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.409+0000 7ffab359e640 1 -- 172.21.15.169:0/3487612420 shutdown_connections 2024-03-28T01:07:01.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.409+0000 7ffab359e640 1 -- 172.21.15.169:0/3487612420 wait complete. 2024-03-28T01:07:01.462 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:07:01.462 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T01:07:01.797 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:01 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3487612420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:07:01.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.832+0000 7f053cc96640 1 -- 172.21.15.169:0/2983856484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0538108ca0 msgr2=0x7f0538109100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:01.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.832+0000 7f053cc96640 1 --2- 172.21.15.169:0/2983856484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0538108ca0 0x7f0538109100 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0530009be0 tx=0x7f053002bf10 comp rx=0 tx=0).stop 2024-03-28T01:07:01.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.832+0000 7f053cc96640 1 -- 172.21.15.169:0/2983856484 shutdown_connections 2024-03-28T01:07:01.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.832+0000 7f053cc96640 1 --2- 172.21.15.169:0/2983856484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0538109640 0x7f053810df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.832+0000 7f053cc96640 1 --2- 172.21.15.169:0/2983856484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0538108ca0 0x7f0538109100 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.832+0000 7f053cc96640 1 --2- 172.21.15.169:0/2983856484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f05380729f0 0x7f0538072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.832+0000 7f053cc96640 1 -- 172.21.15.169:0/2983856484 >> 172.21.15.169:0/2983856484 conn(0x7f053806e4c0 msgr2=0x7f0538070900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:01.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.833+0000 7f053cc96640 1 -- 172.21.15.169:0/2983856484 shutdown_connections 2024-03-28T01:07:01.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.833+0000 7f053cc96640 1 -- 172.21.15.169:0/2983856484 wait complete. 2024-03-28T01:07:01.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.834+0000 7f053cc96640 1 Processor -- start 2024-03-28T01:07:01.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.836+0000 7f053cc96640 1 -- start start 2024-03-28T01:07:01.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.836+0000 7f053cc96640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05380729f0 0x7f05381a65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:01.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.836+0000 7f053cc96640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0538108ca0 0x7f05381a6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:01.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.836+0000 7f053cc96640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0538109640 0x7f05381adb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:01.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.836+0000 7f053cc96640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053811bfa0 con 0x7f0538109640 2024-03-28T01:07:01.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.836+0000 7f053cc96640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053811be20 con 0x7f0538108ca0 2024-03-28T01:07:01.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.836+0000 7f053cc96640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053811c120 con 0x7f05380729f0 2024-03-28T01:07:01.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.837+0000 7f0536ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0538108ca0 0x7f05381a6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:01.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.837+0000 7f0536ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0538108ca0 0x7f05381a6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33614/0 (socket says 172.21.15.169:33614) 2024-03-28T01:07:01.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.837+0000 7f0536ffd640 1 -- 172.21.15.169:0/999328013 learned_addr learned my addr 172.21.15.169:0/999328013 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:01.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.838+0000 7f05377fe640 1 --2- 172.21.15.169:0/999328013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05380729f0 0x7f05381a65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:01.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.839+0000 7f0536ffd640 1 -- 172.21.15.169:0/999328013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05380729f0 msgr2=0x7f05381a65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:01.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.839+0000 7f0536ffd640 1 --2- 172.21.15.169:0/999328013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05380729f0 0x7f05381a65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.839+0000 7f0536ffd640 1 -- 172.21.15.169:0/999328013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0538109640 msgr2=0x7f05381adb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:01.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.839+0000 7f0536ffd640 1 --2- 172.21.15.169:0/999328013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0538109640 0x7f05381adb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.839+0000 7f0536ffd640 1 -- 172.21.15.169:0/999328013 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053810ff00 con 0x7f0538108ca0 2024-03-28T01:07:01.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.841+0000 7f0536ffd640 1 --2- 172.21.15.169:0/999328013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0538108ca0 0x7f05381a6ae0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f053002c520 tx=0x7f05300337f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:01.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.851+0000 7f0534ff9640 1 -- 172.21.15.169:0/999328013 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0530043020 con 0x7f0538108ca0 2024-03-28T01:07:01.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.851+0000 7f053cc96640 1 -- 172.21.15.169:0/999328013 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0538110190 con 0x7f0538108ca0 2024-03-28T01:07:01.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.851+0000 7f053cc96640 1 -- 172.21.15.169:0/999328013 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05381106e0 con 0x7f0538108ca0 2024-03-28T01:07:01.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.851+0000 7f0534ff9640 1 -- 172.21.15.169:0/999328013 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0530003b80 con 0x7f0538108ca0 2024-03-28T01:07:01.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.851+0000 7f0534ff9640 1 -- 172.21.15.169:0/999328013 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0530034720 con 0x7f0538108ca0 2024-03-28T01:07:01.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.856+0000 7f0534ff9640 1 -- 172.21.15.169:0/999328013 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0530046020 con 0x7f0538108ca0 2024-03-28T01:07:01.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.857+0000 7f0534ff9640 1 --2- 172.21.15.169:0/999328013 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f05200a2980 0x7f05200a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:01.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.857+0000 7f0540bc1740 1 Processor -- start 2024-03-28T01:07:01.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.857+0000 7f0540bc1740 1 -- start start 2024-03-28T01:07:01.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.857+0000 7f0540bc1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895aba68a0 0x55895aba6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:01.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.857+0000 7f0540bc1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55895aba7290 0x55895ababd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:01.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.857+0000 7f0540bc1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895abac400 0x55895abae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:01.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.857+0000 7f0540bc1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895ab69610 con 0x55895aba68a0 2024-03-28T01:07:01.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.857+0000 7f0540bc1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895ab0c3c0 con 0x55895abac400 2024-03-28T01:07:01.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.857+0000 7f0540bc1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895aab4d60 con 0x55895aba7290 2024-03-28T01:07:01.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.858+0000 7f0534ff9640 1 -- 172.21.15.169:0/999328013 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f05300f2700 con 0x7f0538108ca0 2024-03-28T01:07:01.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.858+0000 7f0537fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895abac400 0x55895abae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:01.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.858+0000 7f0537fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895abac400 0x55895abae890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33626/0 (socket says 172.21.15.169:33626) 2024-03-28T01:07:01.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.858+0000 7f0537fff640 1 -- 172.21.15.169:0/2988309490 learned_addr learned my addr 172.21.15.169:0/2988309490 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:01.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.859+0000 7f05377fe640 1 --2- 172.21.15.169:0/999328013 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f05200a2980 0x7f05200a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:01.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.859+0000 7f05377fe640 1 --2- 172.21.15.169:0/2988309490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895aba68a0 0x55895aba6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:01.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.859+0000 7f0537fff640 1 -- 172.21.15.169:0/2988309490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55895aba7290 msgr2=0x55895ababd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:01.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.859+0000 7f0537fff640 1 --2- 172.21.15.169:0/2988309490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55895aba7290 0x55895ababd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.859+0000 7f0537fff640 1 -- 172.21.15.169:0/2988309490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895aba68a0 msgr2=0x55895aba6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:01.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.859+0000 7f0537fff640 1 --2- 172.21.15.169:0/2988309490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895aba68a0 0x55895aba6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.859+0000 7f0537fff640 1 -- 172.21.15.169:0/2988309490 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55895a8203f0 con 0x55895abac400 2024-03-28T01:07:01.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.867+0000 7f05377fe640 1 --2- 172.21.15.169:0/2988309490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895aba68a0 0x55895aba6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:07:01.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.870+0000 7f0537fff640 1 --2- 172.21.15.169:0/2988309490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895abac400 0x55895abae890 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0528009c40 tx=0x7f052800e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=797ec2457c0f7d75 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:01.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.872+0000 7f051dffb640 1 -- 172.21.15.169:0/2988309490 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0528005020 con 0x55895abac400 2024-03-28T01:07:01.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.872+0000 7f051dffb640 1 -- 172.21.15.169:0/2988309490 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05280047a0 con 0x55895abac400 2024-03-28T01:07:01.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.872+0000 7f051dffb640 1 -- 172.21.15.169:0/2988309490 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0528010690 con 0x55895abac400 2024-03-28T01:07:01.876 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.874+0000 7f0540bc1740 1 -- 172.21.15.169:0/2988309490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895abac400 msgr2=0x55895abae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:01.877 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.874+0000 7f0540bc1740 1 --2- 172.21.15.169:0/2988309490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895abac400 0x55895abae890 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0528009c40 tx=0x7f052800e720 comp rx=0 tx=0).stop 2024-03-28T01:07:01.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.876+0000 7f0540bc1740 1 -- 172.21.15.169:0/2988309490 shutdown_connections 2024-03-28T01:07:01.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.876+0000 7f05377fe640 1 --2- 172.21.15.169:0/999328013 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f05200a2980 0x7f05200a4e40 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x55895aba7af0 tx=0x7f052c0094d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:01.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.876+0000 7f0540bc1740 1 --2- 172.21.15.169:0/2988309490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895abac400 0x55895abae890 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.876+0000 7f0540bc1740 1 --2- 172.21.15.169:0/2988309490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55895aba7290 0x55895ababd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.876+0000 7f0540bc1740 1 --2- 172.21.15.169:0/2988309490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895aba68a0 0x55895aba6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.876+0000 7f0540bc1740 1 -- 172.21.15.169:0/2988309490 >> 172.21.15.169:0/2988309490 conn(0x55895aba1e10 msgr2=0x55895aba2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:01.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.885+0000 7f0540bc1740 1 -- 172.21.15.169:0/2988309490 shutdown_connections 2024-03-28T01:07:01.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.885+0000 7f0540bc1740 1 -- 172.21.15.169:0/2988309490 wait complete. 2024-03-28T01:07:01.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.885+0000 7f0540bc1740 1 -- start start 2024-03-28T01:07:01.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.886+0000 7f0540bc1740 1 Processor -- start 2024-03-28T01:07:01.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.889+0000 7f0540bc1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55895aba68a0 0x55895abb0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:01.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.889+0000 7f0540bc1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895aba7290 0x55895abb1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:01.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.889+0000 7f0540bc1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895abac400 0x55895abb8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:01.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.889+0000 7f0540bc1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895aac8240 con 0x55895aba7290 2024-03-28T01:07:01.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.889+0000 7f0540bc1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895ab08460 con 0x55895abac400 2024-03-28T01:07:01.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.889+0000 7f0540bc1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895abaf3b0 con 0x55895aba68a0 2024-03-28T01:07:01.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.889+0000 7f0540bc1740 10 client.0 _ll_register_callbacks cb 0x55895aac7c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:01.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.889+0000 7f0536ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895aba7290 0x55895abb1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:01.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.890+0000 7f0536ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895aba7290 0x55895abb1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:35908/0 (socket says 172.21.15.169:35908) 2024-03-28T01:07:01.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.890+0000 7f0536ffd640 1 -- 172.21.15.169:0/2120648983 learned_addr learned my addr 172.21.15.169:0/2120648983 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:01.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.890+0000 7f0536ffd640 1 -- 172.21.15.169:0/2120648983 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55895aba68a0 msgr2=0x55895abb0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:01.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.890+0000 7f0536ffd640 1 --2- 172.21.15.169:0/2120648983 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55895aba68a0 0x55895abb0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.890+0000 7f0536ffd640 1 -- 172.21.15.169:0/2120648983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895abac400 msgr2=0x55895abb8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:01.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.890+0000 7f0536ffd640 1 --2- 172.21.15.169:0/2120648983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895abac400 0x55895abb8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.890+0000 7f0536ffd640 1 -- 172.21.15.169:0/2120648983 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55895a8ea7d0 con 0x55895aba7290 2024-03-28T01:07:01.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.890+0000 7f0536ffd640 1 --2- 172.21.15.169:0/2120648983 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895aba7290 0x55895abb1240 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f05300bec90 tx=0x7f05300bab50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e853eda01b02729 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:01.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.898+0000 7f0516ffd640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:07:01.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.898+0000 7f0516ffd640 1 -- 172.21.15.169:0/2120648983 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05300bf400 con 0x55895aba7290 2024-03-28T01:07:01.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.898+0000 7f0540bc1740 1 -- 172.21.15.169:0/2120648983 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f05280047a0 con 0x55895aba7290 2024-03-28T01:07:01.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.898+0000 7f0516ffd640 1 -- 172.21.15.169:0/2120648983 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f05300bd000 con 0x55895aba7290 2024-03-28T01:07:01.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.899+0000 7f0516ffd640 1 -- 172.21.15.169:0/2120648983 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05300bed00 con 0x55895aba7290 2024-03-28T01:07:01.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.899+0000 7f0540bc1740 10 client.17406 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:01.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.899+0000 7f0540bc1740 1 -- 172.21.15.169:0/2120648983 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55895abbb560 con 0x55895aba7290 2024-03-28T01:07:01.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.904+0000 7f0516ffd640 1 -- 172.21.15.169:0/2120648983 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05280047a0 con 0x55895aba7290 2024-03-28T01:07:01.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.905+0000 7f0516ffd640 1 -- 172.21.15.169:0/2120648983 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f05300dd400 con 0x55895aba7290 2024-03-28T01:07:01.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.905+0000 7f0540bc1740 10 client.17406 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:01.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.905+0000 7f0540bc1740 10 client.17406 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:07:01.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.905+0000 7f0540bc1740 20 client.17406 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:01.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.905+0000 7f0540bc1740 1 --2- 172.21.15.169:0/2120648983 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55895abbceb0 0x55895abbf350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:01.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.905+0000 7f0540bc1740 4 client.17406 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:07:01.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.905+0000 7f0540bc1740 1 -- 172.21.15.169:0/2120648983 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55895ab19900 con 0x55895abbceb0 2024-03-28T01:07:01.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.907+0000 7f05377fe640 1 --2- 172.21.15.169:0/2120648983 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55895abbceb0 0x55895abbf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:01.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.920+0000 7f0516ffd640 10 client.17406 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:07:01.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.920+0000 7f05377fe640 1 --2- 172.21.15.169:0/2120648983 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55895abbceb0 0x55895abbf350 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ffdaa16355cc69d7 server_cookie=36acc92638e19403 in_seq=0 out_seq=0 2024-03-28T01:07:01.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.925+0000 7f0516ffd640 1 -- 172.21.15.169:0/2120648983 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55895ab19900 con 0x55895abbceb0 2024-03-28T01:07:01.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.925+0000 7f0516ffd640 10 client.17406 handle_command_reply: tid=0 2024-03-28T01:07:01.930 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.926+0000 7f0540bc1740 1 client.17406 shutdown 2024-03-28T01:07:01.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.928+0000 7f0540bc1740 1 -- 172.21.15.169:0/2120648983 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895aba7290 msgr2=0x55895abb1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:01.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.928+0000 7f0540bc1740 1 --2- 172.21.15.169:0/2120648983 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895aba7290 0x55895abb1240 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f05300bec90 tx=0x7f05300bab50 comp rx=0 tx=0).stop 2024-03-28T01:07:01.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.929+0000 7f0540bc1740 1 -- 172.21.15.169:0/2120648983 shutdown_connections 2024-03-28T01:07:01.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.929+0000 7f0540bc1740 1 --2- 172.21.15.169:0/2120648983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895abac400 0x55895abb8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.929+0000 7f0540bc1740 1 --2- 172.21.15.169:0/2120648983 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55895abbceb0 0x55895abbf350 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.929+0000 7f0540bc1740 1 --2- 172.21.15.169:0/2120648983 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895aba7290 0x55895abb1240 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.929+0000 7f0540bc1740 1 --2- 172.21.15.169:0/2120648983 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55895aba68a0 0x55895abb0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.929+0000 7f0516ffd640 0 client.17406 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:07:01.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.929+0000 7f0540bc1740 1 -- 172.21.15.169:0/2120648983 >> 172.21.15.169:0/2120648983 conn(0x55895aba1e10 msgr2=0x55895aba1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:01.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.932+0000 7f0540bc1740 1 -- 172.21.15.169:0/2120648983 shutdown_connections 2024-03-28T01:07:01.937 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.934+0000 7f0540bc1740 1 -- 172.21.15.169:0/2120648983 wait complete. 2024-03-28T01:07:01.937 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.934+0000 7f0540bc1740 20 client.17406 trim_cache size 0 max 16384 2024-03-28T01:07:01.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.987+0000 7f0540bc1740 1 Processor -- start 2024-03-28T01:07:01.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.992+0000 7f0540bc1740 1 -- start start 2024-03-28T01:07:01.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.992+0000 7f0540bc1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895aba68a0 0x55895abafff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:01.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.992+0000 7f0540bc1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55895aba7290 0x55895abb8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:01.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.992+0000 7f0540bc1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895abac400 0x55895abb0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:01.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.993+0000 7f0537fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895abac400 0x55895abb0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:01.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.993+0000 7f0537fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895abac400 0x55895abb0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:35922/0 (socket says 172.21.15.169:35922) 2024-03-28T01:07:01.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.993+0000 7f0537fff640 1 -- 172.21.15.169:0/3330905844 learned_addr learned my addr 172.21.15.169:0/3330905844 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.993+0000 7f0536ffd640 1 --2- 172.21.15.169:0/3330905844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55895aba7290 0x55895abb8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.993+0000 7f0540bc1740 1 -- 172.21.15.169:0/3330905844 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0524011f00 con 0x55895abac400 2024-03-28T01:07:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.993+0000 7f0540bc1740 1 -- 172.21.15.169:0/3330905844 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052400fa00 con 0x55895aba68a0 2024-03-28T01:07:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.993+0000 7f0540bc1740 1 -- 172.21.15.169:0/3330905844 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052400db50 con 0x55895aba7290 2024-03-28T01:07:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.994+0000 7f0537fff640 1 -- 172.21.15.169:0/3330905844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55895aba7290 msgr2=0x55895abb8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.994+0000 7f0537fff640 1 --2- 172.21.15.169:0/3330905844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55895aba7290 0x55895abb8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:01.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.994+0000 7f0537fff640 1 -- 172.21.15.169:0/3330905844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895aba68a0 msgr2=0x55895abafff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:07:02.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.994+0000 7f0537fff640 1 --2- 172.21.15.169:0/3330905844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895aba68a0 0x55895abafff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.994+0000 7f0537fff640 1 -- 172.21.15.169:0/3330905844 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55895ababaa0 con 0x55895abac400 2024-03-28T01:07:02.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.996+0000 7f0537fff640 1 --2- 172.21.15.169:0/3330905844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895abac400 0x55895abb0530 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f05280096d0 tx=0x7f052800eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4c3545594a0d769 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:02.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.997+0000 7f051d7fa640 1 -- 172.21.15.169:0/3330905844 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05280097d0 con 0x55895abac400 2024-03-28T01:07:02.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.997+0000 7f051d7fa640 1 -- 172.21.15.169:0/3330905844 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0528004520 con 0x55895abac400 2024-03-28T01:07:02.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.998+0000 7f0540bc1740 1 -- 172.21.15.169:0/3330905844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895abac400 msgr2=0x55895abb0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:02.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:01.998+0000 7f0540bc1740 1 --2- 172.21.15.169:0/3330905844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895abac400 0x55895abb0530 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f05280096d0 tx=0x7f052800eed0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.003+0000 7f0540bc1740 1 -- 172.21.15.169:0/3330905844 shutdown_connections 2024-03-28T01:07:02.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.003+0000 7f0540bc1740 1 --2- 172.21.15.169:0/3330905844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895abac400 0x55895abb0530 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.003+0000 7f0540bc1740 1 --2- 172.21.15.169:0/3330905844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55895aba7290 0x55895abb8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.003+0000 7f0540bc1740 1 --2- 172.21.15.169:0/3330905844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895aba68a0 0x55895abafff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.003+0000 7f0540bc1740 1 -- 172.21.15.169:0/3330905844 >> 172.21.15.169:0/3330905844 conn(0x55895aba1e10 msgr2=0x55895abb0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:02.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.007+0000 7f0540bc1740 1 -- 172.21.15.169:0/3330905844 shutdown_connections 2024-03-28T01:07:02.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.007+0000 7f0540bc1740 1 -- 172.21.15.169:0/3330905844 wait complete. 2024-03-28T01:07:02.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.007+0000 7f0540bc1740 1 -- start start 2024-03-28T01:07:02.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.008+0000 7f0540bc1740 1 Processor -- start 2024-03-28T01:07:02.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.008+0000 7f0540bc1740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895aba68a0 0x55895abb5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.008+0000 7f0540bc1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895aba7290 0x55895abb5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.008+0000 7f0540bc1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55895abac400 0x55895abb6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.008+0000 7f0540bc1740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052400e0c0 con 0x55895aba68a0 2024-03-28T01:07:02.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.008+0000 7f0540bc1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052400f3a0 con 0x55895aba7290 2024-03-28T01:07:02.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.008+0000 7f0540bc1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052400e6e0 con 0x55895abac400 2024-03-28T01:07:02.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.008+0000 7f0540bc1740 10 client.0 _ll_register_callbacks cb 0x7f05300ba8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:02.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.009+0000 7f0537fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55895abac400 0x55895abb6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:02.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.009+0000 7f0537fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55895abac400 0x55895abb6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56186/0 (socket says 172.21.15.169:56186) 2024-03-28T01:07:02.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.009+0000 7f0537fff640 1 -- 172.21.15.169:0/788671821 learned_addr learned my addr 172.21.15.169:0/788671821 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:02.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.010+0000 7f0536ffd640 1 --2- 172.21.15.169:0/788671821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895aba7290 0x55895abb5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:02.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.010+0000 7f0537fff640 1 -- 172.21.15.169:0/788671821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895aba7290 msgr2=0x55895abb5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:02.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.010+0000 7f0537fff640 1 --2- 172.21.15.169:0/788671821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895aba7290 0x55895abb5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.010+0000 7f0537fff640 1 -- 172.21.15.169:0/788671821 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895aba68a0 msgr2=0x55895abb5910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:07:02.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.010+0000 7f0537fff640 1 --2- 172.21.15.169:0/788671821 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895aba68a0 0x55895abb5910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.010+0000 7f0537fff640 1 -- 172.21.15.169:0/788671821 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55895abb7400 con 0x55895abac400 2024-03-28T01:07:02.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.011+0000 7f0537fff640 1 --2- 172.21.15.169:0/788671821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55895abac400 0x55895abb6e10 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0528024040 tx=0x7f0528009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=769aaa226dc69760 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:02.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.018+0000 7f0516ffd640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:07:02.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.018+0000 7f0516ffd640 1 -- 172.21.15.169:0/788671821 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052801d040 con 0x55895abac400 2024-03-28T01:07:02.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.018+0000 7f0540bc1740 1 -- 172.21.15.169:0/788671821 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0528004520 con 0x55895abac400 2024-03-28T01:07:02.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.018+0000 7f0516ffd640 1 -- 172.21.15.169:0/788671821 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f0528004bd0 con 0x55895abac400 2024-03-28T01:07:02.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.019+0000 7f0516ffd640 1 -- 172.21.15.169:0/788671821 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0528039050 con 0x55895abac400 2024-03-28T01:07:02.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.020+0000 7f0540bc1740 10 client.27217 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:02.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.020+0000 7f0540bc1740 1 -- 172.21.15.169:0/788671821 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55895ac187e0 con 0x55895abac400 2024-03-28T01:07:02.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.021+0000 7f0516ffd640 1 -- 172.21.15.169:0/788671821 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0528004520 con 0x55895abac400 2024-03-28T01:07:02.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.022+0000 7f0516ffd640 1 -- 172.21.15.169:0/788671821 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f0528018430 con 0x55895abac400 2024-03-28T01:07:02.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.023+0000 7f0540bc1740 10 client.27217 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:02.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.023+0000 7f0540bc1740 10 client.27217 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:07:02.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.023+0000 7f0540bc1740 20 client.27217 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:02.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.023+0000 7f0540bc1740 1 --2- 172.21.15.169:0/788671821 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55895ac19020 0x55895ac1b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.023+0000 7f0540bc1740 4 client.27217 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:07:02.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.023+0000 7f0540bc1740 1 -- 172.21.15.169:0/788671821 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55895ab13c50 con 0x55895ac19020 2024-03-28T01:07:02.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.027+0000 7f05377fe640 1 --2- 172.21.15.169:0/788671821 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55895ac19020 0x55895ac1b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:02.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.036+0000 7f0516ffd640 10 client.27217 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:07:02.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.036+0000 7f05377fe640 1 --2- 172.21.15.169:0/788671821 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55895ac19020 0x55895ac1b440 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8277b230f6ccb22 server_cookie=342998345ce9957b in_seq=0 out_seq=0 2024-03-28T01:07:02.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.039+0000 7f0516ffd640 1 -- 172.21.15.169:0/788671821 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55895ab13c50 con 0x55895ac19020 2024-03-28T01:07:02.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.039+0000 7f0516ffd640 10 client.27217 handle_command_reply: tid=0 2024-03-28T01:07:02.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.039+0000 7f0540bc1740 1 client.27217 shutdown 2024-03-28T01:07:02.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.040+0000 7f0540bc1740 1 -- 172.21.15.169:0/788671821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55895abac400 msgr2=0x55895abb6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:02.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.040+0000 7f0540bc1740 1 --2- 172.21.15.169:0/788671821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55895abac400 0x55895abb6e10 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0528024040 tx=0x7f0528009c00 comp rx=0 tx=0).stop 2024-03-28T01:07:02.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.041+0000 7f0540bc1740 1 -- 172.21.15.169:0/788671821 shutdown_connections 2024-03-28T01:07:02.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.041+0000 7f0540bc1740 1 --2- 172.21.15.169:0/788671821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55895abac400 0x55895abb6e10 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.041+0000 7f0540bc1740 1 --2- 172.21.15.169:0/788671821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895aba7290 0x55895abb5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.041+0000 7f0540bc1740 1 --2- 172.21.15.169:0/788671821 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55895ac19020 0x55895ac1b440 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.041+0000 7f0540bc1740 1 --2- 172.21.15.169:0/788671821 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55895aba68a0 0x55895abb5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.041+0000 7f0516ffd640 0 client.27217 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:07:02.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.041+0000 7f0540bc1740 1 -- 172.21.15.169:0/788671821 >> 172.21.15.169:0/788671821 conn(0x55895aba1e10 msgr2=0x55895aba5900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:02.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.041+0000 7f0540bc1740 1 -- 172.21.15.169:0/788671821 shutdown_connections 2024-03-28T01:07:02.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.041+0000 7f0540bc1740 1 -- 172.21.15.169:0/788671821 wait complete. 2024-03-28T01:07:02.046 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.043+0000 7f0540bc1740 20 client.27217 trim_cache size 0 max 16384 2024-03-28T01:07:02.046 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:07:02.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.052+0000 7f053cc96640 1 -- 172.21.15.169:0/999328013 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f05200a2980 msgr2=0x7f05200a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:02.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.052+0000 7f053cc96640 1 --2- 172.21.15.169:0/999328013 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f05200a2980 0x7f05200a4e40 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x55895aba7af0 tx=0x7f052c0094d0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.052+0000 7f053cc96640 1 -- 172.21.15.169:0/999328013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0538108ca0 msgr2=0x7f05381a6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:02.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.052+0000 7f053cc96640 1 --2- 172.21.15.169:0/999328013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0538108ca0 0x7f05381a6ae0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f053002c520 tx=0x7f05300337f0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.052+0000 7f053cc96640 1 -- 172.21.15.169:0/999328013 shutdown_connections 2024-03-28T01:07:02.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.052+0000 7f053cc96640 1 --2- 172.21.15.169:0/999328013 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f05200a2980 0x7f05200a4e40 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.052+0000 7f053cc96640 1 --2- 172.21.15.169:0/999328013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0538109640 0x7f05381adb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.052+0000 7f053cc96640 1 --2- 172.21.15.169:0/999328013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0538108ca0 0x7f05381a6ae0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.052+0000 7f053cc96640 1 --2- 172.21.15.169:0/999328013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05380729f0 0x7f05381a65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.052+0000 7f053cc96640 1 -- 172.21.15.169:0/999328013 >> 172.21.15.169:0/999328013 conn(0x7f053806e4c0 msgr2=0x7f0538070120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:02.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.057+0000 7f053cc96640 1 -- 172.21.15.169:0/999328013 shutdown_connections 2024-03-28T01:07:02.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.058+0000 7f053cc96640 1 -- 172.21.15.169:0/999328013 wait complete. 2024-03-28T01:07:02.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:01 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3487612420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:07:02.109 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:07:02.109 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T01:07:02.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:01 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3487612420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:07:02.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.458+0000 7f15897e2640 1 -- 172.21.15.169:0/128329102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15840729f0 msgr2=0x7f1584072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:02.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.458+0000 7f15897e2640 1 --2- 172.21.15.169:0/128329102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15840729f0 0x7f1584072e10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f157c009be0 tx=0x7f157c02bf10 comp rx=0 tx=0).stop 2024-03-28T01:07:02.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.460+0000 7f15897e2640 1 -- 172.21.15.169:0/128329102 shutdown_connections 2024-03-28T01:07:02.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.460+0000 7f15897e2640 1 --2- 172.21.15.169:0/128329102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1584109640 0x7f158410df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.460+0000 7f15897e2640 1 --2- 172.21.15.169:0/128329102 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1584108ca0 0x7f1584109100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.460+0000 7f15897e2640 1 --2- 172.21.15.169:0/128329102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15840729f0 0x7f1584072e10 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.460+0000 7f15897e2640 1 -- 172.21.15.169:0/128329102 >> 172.21.15.169:0/128329102 conn(0x7f158406e4c0 msgr2=0x7f1584070900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:02.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.470+0000 7f15897e2640 1 -- 172.21.15.169:0/128329102 shutdown_connections 2024-03-28T01:07:02.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.470+0000 7f15897e2640 1 -- 172.21.15.169:0/128329102 wait complete. 2024-03-28T01:07:02.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.471+0000 7f15897e2640 1 Processor -- start 2024-03-28T01:07:02.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.471+0000 7f15897e2640 1 -- start start 2024-03-28T01:07:02.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.477+0000 7f15897e2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15840729f0 0x7f1584114d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.477+0000 7f15897e2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1584108ca0 0x7f1584115260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.477+0000 7f15897e2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1584109640 0x7f15841b6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.477+0000 7f15897e2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158411bfa0 con 0x7f1584109640 2024-03-28T01:07:02.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.477+0000 7f15897e2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158411be20 con 0x7f1584108ca0 2024-03-28T01:07:02.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.477+0000 7f15897e2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158411c120 con 0x7f15840729f0 2024-03-28T01:07:02.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.478+0000 7f1588fe1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1584109640 0x7f15841b6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:02.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.478+0000 7f1588fe1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1584109640 0x7f15841b6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:35938/0 (socket says 172.21.15.169:35938) 2024-03-28T01:07:02.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.478+0000 7f1588fe1640 1 -- 172.21.15.169:0/3619975976 learned_addr learned my addr 172.21.15.169:0/3619975976 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:02.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.478+0000 7f15837fe640 1 --2- 172.21.15.169:0/3619975976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1584108ca0 0x7f1584115260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:02.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.478+0000 7f1583fff640 1 --2- 172.21.15.169:0/3619975976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15840729f0 0x7f1584114d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:02.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.480+0000 7f1588fe1640 1 -- 172.21.15.169:0/3619975976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15840729f0 msgr2=0x7f1584114d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:02.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.480+0000 7f1588fe1640 1 --2- 172.21.15.169:0/3619975976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15840729f0 0x7f1584114d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.480+0000 7f1588fe1640 1 -- 172.21.15.169:0/3619975976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1584108ca0 msgr2=0x7f1584115260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:02.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.480+0000 7f1588fe1640 1 --2- 172.21.15.169:0/3619975976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1584108ca0 0x7f1584115260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.485 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.481+0000 7f1588fe1640 1 -- 172.21.15.169:0/3619975976 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15841b6bb0 con 0x7f1584109640 2024-03-28T01:07:02.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.488+0000 7f1588fe1640 1 --2- 172.21.15.169:0/3619975976 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1584109640 0x7f15841b6670 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f157800c970 tx=0x7f157800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:02.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.492+0000 7f15817fa640 1 -- 172.21.15.169:0/3619975976 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157800ae60 con 0x7f1584109640 2024-03-28T01:07:02.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.492+0000 7f15817fa640 1 -- 172.21.15.169:0/3619975976 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1578005500 con 0x7f1584109640 2024-03-28T01:07:02.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.492+0000 7f15817fa640 1 -- 172.21.15.169:0/3619975976 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157800f450 con 0x7f1584109640 2024-03-28T01:07:02.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.492+0000 7f15897e2640 1 -- 172.21.15.169:0/3619975976 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15841b6d80 con 0x7f1584109640 2024-03-28T01:07:02.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.492+0000 7f15897e2640 1 -- 172.21.15.169:0/3619975976 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15841b7190 con 0x7f1584109640 2024-03-28T01:07:02.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.499+0000 7f15817fa640 1 -- 172.21.15.169:0/3619975976 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f157800f5f0 con 0x7f1584109640 2024-03-28T01:07:02.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.500+0000 7f158d62c740 1 Processor -- start 2024-03-28T01:07:02.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.500+0000 7f158d62c740 1 -- start start 2024-03-28T01:07:02.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.501+0000 7f158d62c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565e96cb8a0 0x5565e96cbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.501+0000 7f158d62c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565e96cc290 0x5565e96d0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.501+0000 7f158d62c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96d1400 0x5565e96d3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.501+0000 7f158d62c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565e968e610 con 0x5565e96cc290 2024-03-28T01:07:02.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.501+0000 7f158d62c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565e96313c0 con 0x5565e96d1400 2024-03-28T01:07:02.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.501+0000 7f158d62c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565e95d9d60 con 0x5565e96cb8a0 2024-03-28T01:07:02.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.503+0000 7f15837fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96d1400 0x5565e96d3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:02.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.503+0000 7f15817fa640 1 --2- 172.21.15.169:0/3619975976 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f156c0a2a10 0x7f156c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.503+0000 7f1588fe1640 1 --2- 172.21.15.169:0/3619975976 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f156c0a2a10 0x7f156c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:02.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.504+0000 7f1588fe1640 1 --2- 172.21.15.169:0/3619975976 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f156c0a2a10 0x7f156c0a4ed0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1578096b40 tx=0x7f15780969d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:02.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.504+0000 7f15817fa640 1 -- 172.21.15.169:0/3619975976 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f15780ce8c0 con 0x7f1584109640 2024-03-28T01:07:02.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.504+0000 7f15837fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96d1400 0x5565e96d3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33664/0 (socket says 172.21.15.169:33664) 2024-03-28T01:07:02.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.504+0000 7f15837fe640 1 -- 172.21.15.169:0/298515752 learned_addr learned my addr 172.21.15.169:0/298515752 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:02.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.504+0000 7f15837fe640 1 --2- 172.21.15.169:0/298515752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565e96cb8a0 0x5565e96cbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:02.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.508+0000 7f15837fe640 1 -- 172.21.15.169:0/298515752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565e96cb8a0 msgr2=0x5565e96cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:02.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.508+0000 7f15837fe640 1 --2- 172.21.15.169:0/298515752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565e96cb8a0 0x5565e96cbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.508+0000 7f15837fe640 1 -- 172.21.15.169:0/298515752 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565e96cc290 msgr2=0x5565e96d0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:02.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.508+0000 7f15837fe640 1 --2- 172.21.15.169:0/298515752 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565e96cc290 0x5565e96d0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.508+0000 7f15837fe640 1 -- 172.21.15.169:0/298515752 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565e93453f0 con 0x5565e96d1400 2024-03-28T01:07:02.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.516+0000 7f15837fe640 1 --2- 172.21.15.169:0/298515752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96d1400 0x5565e96d3890 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1574009fd0 tx=0x7f1574005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36c14770bb6e3521 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:02.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.521+0000 7f156a7fc640 1 -- 172.21.15.169:0/298515752 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1574004050 con 0x5565e96d1400 2024-03-28T01:07:02.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.521+0000 7f156a7fc640 1 -- 172.21.15.169:0/298515752 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1574005500 con 0x5565e96d1400 2024-03-28T01:07:02.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.522+0000 7f158d62c740 1 -- 172.21.15.169:0/298515752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96d1400 msgr2=0x5565e96d3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:02.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.522+0000 7f158d62c740 1 --2- 172.21.15.169:0/298515752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96d1400 0x5565e96d3890 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1574009fd0 tx=0x7f1574005850 comp rx=0 tx=0).stop 2024-03-28T01:07:02.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.523+0000 7f158d62c740 1 -- 172.21.15.169:0/298515752 shutdown_connections 2024-03-28T01:07:02.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.523+0000 7f158d62c740 1 --2- 172.21.15.169:0/298515752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96d1400 0x5565e96d3890 secure :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1574009fd0 tx=0x7f1574005850 comp rx=0 tx=0).stop 2024-03-28T01:07:02.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.523+0000 7f158d62c740 1 --2- 172.21.15.169:0/298515752 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565e96cc290 0x5565e96d0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.523+0000 7f158d62c740 1 --2- 172.21.15.169:0/298515752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565e96cb8a0 0x5565e96cbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.523+0000 7f158d62c740 1 -- 172.21.15.169:0/298515752 >> 172.21.15.169:0/298515752 conn(0x5565e96c6e10 msgr2=0x5565e96c7210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:02.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.523+0000 7f158d62c740 1 -- 172.21.15.169:0/298515752 shutdown_connections 2024-03-28T01:07:02.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.526+0000 7f158d62c740 1 -- 172.21.15.169:0/298515752 wait complete. 2024-03-28T01:07:02.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.526+0000 7f158d62c740 1 -- start start 2024-03-28T01:07:02.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.527+0000 7f158d62c740 1 Processor -- start 2024-03-28T01:07:02.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.529+0000 7f158d62c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565e96cb8a0 0x5565e96d5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.529+0000 7f158d62c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96cc290 0x5565e96d62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.529+0000 7f158d62c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565e96db2e0 0x5565e96dd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.529+0000 7f158d62c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565e95ed240 con 0x5565e96db2e0 2024-03-28T01:07:02.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.529+0000 7f158d62c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565e962d460 con 0x5565e96cc290 2024-03-28T01:07:02.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.529+0000 7f158d62c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565e96d4400 con 0x5565e96cb8a0 2024-03-28T01:07:02.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.529+0000 7f158d62c740 10 client.0 _ll_register_callbacks cb 0x5565e95ecc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:02.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.534+0000 7f1583fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96cc290 0x5565e96d62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:02.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.534+0000 7f1583fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96cc290 0x5565e96d62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33676/0 (socket says 172.21.15.169:33676) 2024-03-28T01:07:02.537 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.534+0000 7f1583fff640 1 -- 172.21.15.169:0/2560285891 learned_addr learned my addr 172.21.15.169:0/2560285891 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:02.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.536+0000 7f15837fe640 1 --2- 172.21.15.169:0/2560285891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565e96cb8a0 0x5565e96d5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:02.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.536+0000 7f15837fe640 1 --2- 172.21.15.169:0/2560285891 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565e96db2e0 0x5565e96dd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:02.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.540+0000 7f15837fe640 1 -- 172.21.15.169:0/2560285891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565e96cb8a0 msgr2=0x5565e96d5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:02.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.540+0000 7f15837fe640 1 --2- 172.21.15.169:0/2560285891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565e96cb8a0 0x5565e96d5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.541+0000 7f15837fe640 1 -- 172.21.15.169:0/2560285891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96cc290 msgr2=0x5565e96d62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:02.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.541+0000 7f15837fe640 1 --2- 172.21.15.169:0/2560285891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96cc290 0x5565e96d62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.541+0000 7f15837fe640 1 -- 172.21.15.169:0/2560285891 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5565e940f7d0 con 0x5565e96db2e0 2024-03-28T01:07:02.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.541+0000 7f15837fe640 1 --2- 172.21.15.169:0/2560285891 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565e96db2e0 0x5565e96dd6d0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f157400e000 tx=0x7f1574000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea2f5dcbeec908e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:02.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.541+0000 7f15637fe640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:07:02.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.542+0000 7f15637fe640 1 -- 172.21.15.169:0/2560285891 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1574029040 con 0x5565e96db2e0 2024-03-28T01:07:02.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.543+0000 7f158d62c740 1 -- 172.21.15.169:0/2560285891 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1574005500 con 0x5565e96db2e0 2024-03-28T01:07:02.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.543+0000 7f15637fe640 1 -- 172.21.15.169:0/2560285891 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f157400f990 con 0x5565e96db2e0 2024-03-28T01:07:02.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.546+0000 7f15637fe640 1 -- 172.21.15.169:0/2560285891 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1574035050 con 0x5565e96db2e0 2024-03-28T01:07:02.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.546+0000 7f158d62c740 10 client.17418 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:02.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.546+0000 7f158d62c740 1 -- 172.21.15.169:0/2560285891 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5565e96e0a40 con 0x5565e96db2e0 2024-03-28T01:07:02.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.558+0000 7f15637fe640 1 -- 172.21.15.169:0/2560285891 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1574005500 con 0x5565e96db2e0 2024-03-28T01:07:02.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.558+0000 7f15637fe640 1 -- 172.21.15.169:0/2560285891 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f1574003ad0 con 0x5565e96db2e0 2024-03-28T01:07:02.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.558+0000 7f158d62c740 10 client.17418 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:02.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.558+0000 7f158d62c740 10 client.17418 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:07:02.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.558+0000 7f158d62c740 20 client.17418 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:02.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.558+0000 7f158d62c740 1 --2- 172.21.15.169:0/2560285891 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5565e96e2340 0x5565e96e4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.558+0000 7f158d62c740 4 client.17418 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:07:02.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.558+0000 7f158d62c740 1 -- 172.21.15.169:0/2560285891 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5565e963e900 con 0x5565e96e2340 2024-03-28T01:07:02.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.560+0000 7f1583fff640 1 --2- 172.21.15.169:0/2560285891 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5565e96e2340 0x5565e96e4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:02.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.566+0000 7f1583fff640 1 --2- 172.21.15.169:0/2560285891 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5565e96e2340 0x5565e96e4760 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ca2cbf0856b3b02e server_cookie=f9675c157c0370ad in_seq=0 out_seq=0 2024-03-28T01:07:02.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.566+0000 7f15637fe640 10 client.17418 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:07:02.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.568+0000 7f15637fe640 1 -- 172.21.15.169:0/2560285891 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5565e963e900 con 0x5565e96e2340 2024-03-28T01:07:02.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.568+0000 7f15637fe640 10 client.17418 handle_command_reply: tid=0 2024-03-28T01:07:02.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.568+0000 7f158d62c740 1 client.17418 shutdown 2024-03-28T01:07:02.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.576+0000 7f158d62c740 1 -- 172.21.15.169:0/2560285891 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565e96db2e0 msgr2=0x5565e96dd6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:02.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.576+0000 7f158d62c740 1 --2- 172.21.15.169:0/2560285891 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565e96db2e0 0x5565e96dd6d0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f157400e000 tx=0x7f1574000bc0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.576+0000 7f158d62c740 1 -- 172.21.15.169:0/2560285891 shutdown_connections 2024-03-28T01:07:02.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.576+0000 7f158d62c740 1 --2- 172.21.15.169:0/2560285891 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5565e96e2340 0x5565e96e4760 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.576+0000 7f158d62c740 1 --2- 172.21.15.169:0/2560285891 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565e96db2e0 0x5565e96dd6d0 secure :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f157400e000 tx=0x7f1574000bc0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.576+0000 7f158d62c740 1 --2- 172.21.15.169:0/2560285891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96cc290 0x5565e96d62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.576+0000 7f158d62c740 1 --2- 172.21.15.169:0/2560285891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565e96cb8a0 0x5565e96d5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.576+0000 7f158d62c740 1 -- 172.21.15.169:0/2560285891 >> 172.21.15.169:0/2560285891 conn(0x5565e96c6e10 msgr2=0x5565e96c6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:02.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.576+0000 7f15637fe640 0 client.17418 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:07:02.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.587+0000 7f158d62c740 1 -- 172.21.15.169:0/2560285891 shutdown_connections 2024-03-28T01:07:02.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.588+0000 7f158d62c740 1 -- 172.21.15.169:0/2560285891 wait complete. 2024-03-28T01:07:02.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.588+0000 7f158d62c740 20 client.17418 trim_cache size 0 max 16384 2024-03-28T01:07:02.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.643+0000 7f158d62c740 1 Processor -- start 2024-03-28T01:07:02.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.645+0000 7f158d62c740 1 -- start start 2024-03-28T01:07:02.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.645+0000 7f158d62c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96cb8a0 0x5565e96c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.645+0000 7f158d62c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565e96cc290 0x5565e96d5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.645+0000 7f158d62c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565e96d62b0 0x5565e96cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.645+0000 7f158d62c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1570012120 con 0x5565e96cc290 2024-03-28T01:07:02.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.645+0000 7f158d62c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157000fc20 con 0x5565e96cb8a0 2024-03-28T01:07:02.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.645+0000 7f158d62c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157000dcc0 con 0x5565e96d62b0 2024-03-28T01:07:02.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.646+0000 7f15837fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96cb8a0 0x5565e96c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:02.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.646+0000 7f15837fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96cb8a0 0x5565e96c71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33684/0 (socket says 172.21.15.169:33684) 2024-03-28T01:07:02.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.646+0000 7f15837fe640 1 -- 172.21.15.169:0/148696242 learned_addr learned my addr 172.21.15.169:0/148696242 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:02.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.646+0000 7f15837fe640 1 --2- 172.21.15.169:0/148696242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565e96d62b0 0x5565e96cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:02.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.646+0000 7f15837fe640 1 -- 172.21.15.169:0/148696242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565e96d62b0 msgr2=0x5565e96cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:02.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.646+0000 7f15837fe640 1 --2- 172.21.15.169:0/148696242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565e96d62b0 0x5565e96cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.646+0000 7f15837fe640 1 -- 172.21.15.169:0/148696242 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565e96cc290 msgr2=0x5565e96d5d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:07:02.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.646+0000 7f15837fe640 1 --2- 172.21.15.169:0/148696242 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565e96cc290 0x5565e96d5d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.646+0000 7f15837fe640 1 -- 172.21.15.169:0/148696242 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565e96c1f00 con 0x5565e96cb8a0 2024-03-28T01:07:02.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.647+0000 7f15837fe640 1 --2- 172.21.15.169:0/148696242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96cb8a0 0x5565e96c71f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1574009fd0 tx=0x7f1574022040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27337b5aad01a32b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:02.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.654+0000 7f1569ffb640 1 -- 172.21.15.169:0/148696242 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1574009c70 con 0x5565e96cb8a0 2024-03-28T01:07:02.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.654+0000 7f1569ffb640 1 -- 172.21.15.169:0/148696242 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f157401f590 con 0x5565e96cb8a0 2024-03-28T01:07:02.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.654+0000 7f1569ffb640 1 -- 172.21.15.169:0/148696242 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1574005360 con 0x5565e96cb8a0 2024-03-28T01:07:02.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.655+0000 7f158d62c740 1 -- 172.21.15.169:0/148696242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96cb8a0 msgr2=0x5565e96c71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:02.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.655+0000 7f158d62c740 1 --2- 172.21.15.169:0/148696242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96cb8a0 0x5565e96c71f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1574009fd0 tx=0x7f1574022040 comp rx=0 tx=0).stop 2024-03-28T01:07:02.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.656+0000 7f158d62c740 1 -- 172.21.15.169:0/148696242 shutdown_connections 2024-03-28T01:07:02.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.656+0000 7f158d62c740 1 --2- 172.21.15.169:0/148696242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565e96d62b0 0x5565e96cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.656+0000 7f158d62c740 1 --2- 172.21.15.169:0/148696242 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565e96cc290 0x5565e96d5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.656+0000 7f158d62c740 1 --2- 172.21.15.169:0/148696242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96cb8a0 0x5565e96c71f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.656+0000 7f158d62c740 1 -- 172.21.15.169:0/148696242 >> 172.21.15.169:0/148696242 conn(0x5565e96c6e10 msgr2=0x5565e96cccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:02.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.657+0000 7f158d62c740 1 -- 172.21.15.169:0/148696242 shutdown_connections 2024-03-28T01:07:02.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.658+0000 7f158d62c740 1 -- 172.21.15.169:0/148696242 wait complete. 2024-03-28T01:07:02.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.658+0000 7f158d62c740 1 -- start start 2024-03-28T01:07:02.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.658+0000 7f158d62c740 1 Processor -- start 2024-03-28T01:07:02.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.659+0000 7f158d62c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565e96cc290 0x5565e96f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.659+0000 7f158d62c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565e96d62b0 0x5565e96f2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.659+0000 7f158d62c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96f3920 0x5565e96f3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.659+0000 7f158d62c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157000e2c0 con 0x5565e96cc290 2024-03-28T01:07:02.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.659+0000 7f158d62c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157000f5c0 con 0x5565e96f3920 2024-03-28T01:07:02.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.659+0000 7f158d62c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157000e900 con 0x5565e96d62b0 2024-03-28T01:07:02.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.659+0000 7f158d62c740 10 client.0 _ll_register_callbacks cb 0x7f1574004580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.662+0000 7f1583fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565e96d62b0 0x5565e96f2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.662+0000 7f1583fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565e96d62b0 0x5565e96f2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56246/0 (socket says 172.21.15.169:56246) 2024-03-28T01:07:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.662+0000 7f1583fff640 1 -- 172.21.15.169:0/2443701711 learned_addr learned my addr 172.21.15.169:0/2443701711 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.662+0000 7f15837fe640 1 --2- 172.21.15.169:0/2443701711 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96f3920 0x5565e96f3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.662+0000 7f15837fe640 1 --2- 172.21.15.169:0/2443701711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565e96cc290 0x5565e96f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:02.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.662+0000 7f15837fe640 1 -- 172.21.15.169:0/2443701711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565e96d62b0 msgr2=0x5565e96f2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:02.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.662+0000 7f15837fe640 1 --2- 172.21.15.169:0/2443701711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565e96d62b0 0x5565e96f2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.662+0000 7f15837fe640 1 -- 172.21.15.169:0/2443701711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565e96cc290 msgr2=0x5565e96f23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:02.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.662+0000 7f15837fe640 1 --2- 172.21.15.169:0/2443701711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565e96cc290 0x5565e96f23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.662+0000 7f15837fe640 1 -- 172.21.15.169:0/2443701711 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5565e96f42e0 con 0x5565e96f3920 2024-03-28T01:07:02.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.664+0000 7f15837fe640 1 --2- 172.21.15.169:0/2443701711 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96f3920 0x5565e96f3da0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x5565e96d0de0 tx=0x7f1574004db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4353321708b3227 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:02.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.664+0000 7f15637fe640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:07:02.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.674+0000 7f15637fe640 1 -- 172.21.15.169:0/2443701711 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1574009c70 con 0x5565e96f3920 2024-03-28T01:07:02.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.674+0000 7f158d62c740 1 -- 172.21.15.169:0/2443701711 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f157401f590 con 0x5565e96f3920 2024-03-28T01:07:02.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.674+0000 7f15637fe640 1 -- 172.21.15.169:0/2443701711 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f157402ebe0 con 0x5565e96f3920 2024-03-28T01:07:02.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.683+0000 7f15637fe640 1 -- 172.21.15.169:0/2443701711 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f157402ede0 con 0x5565e96f3920 2024-03-28T01:07:02.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.685+0000 7f158d62c740 10 client.27296 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:02.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.685+0000 7f158d62c740 1 -- 172.21.15.169:0/2443701711 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5565e973d820 con 0x5565e96f3920 2024-03-28T01:07:02.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.693+0000 7f15637fe640 1 -- 172.21.15.169:0/2443701711 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f157401f590 con 0x5565e96f3920 2024-03-28T01:07:02.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.693+0000 7f15637fe640 1 -- 172.21.15.169:0/2443701711 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f157400f790 con 0x5565e96f3920 2024-03-28T01:07:02.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.693+0000 7f158d62c740 10 client.27296 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:02.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.693+0000 7f158d62c740 10 client.27296 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:07:02.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.693+0000 7f158d62c740 20 client.27296 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:02.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.693+0000 7f158d62c740 1 --2- 172.21.15.169:0/2443701711 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5565e973e0c0 0x5565e97404b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:02.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.693+0000 7f158d62c740 4 client.27296 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:07:02.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.693+0000 7f158d62c740 1 -- 172.21.15.169:0/2443701711 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5565e9638c50 con 0x5565e973e0c0 2024-03-28T01:07:02.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.696+0000 7f1583fff640 1 --2- 172.21.15.169:0/2443701711 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5565e973e0c0 0x5565e97404b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:02.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.705+0000 7f1583fff640 1 --2- 172.21.15.169:0/2443701711 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5565e973e0c0 0x5565e97404b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6038f1997418cc2b server_cookie=e3f9e87ac8379ef in_seq=0 out_seq=0 2024-03-28T01:07:02.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.706+0000 7f15637fe640 10 client.27296 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:07:02.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.706+0000 7f15637fe640 1 -- 172.21.15.169:0/2443701711 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5565e9638c50 con 0x5565e973e0c0 2024-03-28T01:07:02.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.706+0000 7f15637fe640 10 client.27296 handle_command_reply: tid=0 2024-03-28T01:07:02.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.706+0000 7f158d62c740 1 client.27296 shutdown 2024-03-28T01:07:02.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.709+0000 7f158d62c740 1 -- 172.21.15.169:0/2443701711 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96f3920 msgr2=0x5565e96f3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:02.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.709+0000 7f158d62c740 1 --2- 172.21.15.169:0/2443701711 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96f3920 0x5565e96f3da0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x5565e96d0de0 tx=0x7f1574004db0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.710+0000 7f158d62c740 1 -- 172.21.15.169:0/2443701711 shutdown_connections 2024-03-28T01:07:02.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.710+0000 7f158d62c740 1 --2- 172.21.15.169:0/2443701711 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x5565e973e0c0 0x5565e97404b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.710+0000 7f158d62c740 1 --2- 172.21.15.169:0/2443701711 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565e96f3920 0x5565e96f3da0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.710+0000 7f158d62c740 1 --2- 172.21.15.169:0/2443701711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5565e96d62b0 0x5565e96f2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.710+0000 7f158d62c740 1 --2- 172.21.15.169:0/2443701711 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5565e96cc290 0x5565e96f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.710+0000 7f158d62c740 1 -- 172.21.15.169:0/2443701711 >> 172.21.15.169:0/2443701711 conn(0x5565e96c6e10 msgr2=0x5565e96ccad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:02.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.710+0000 7f15637fe640 0 client.27296 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:07:02.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.710+0000 7f158d62c740 1 -- 172.21.15.169:0/2443701711 shutdown_connections 2024-03-28T01:07:02.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.710+0000 7f158d62c740 1 -- 172.21.15.169:0/2443701711 wait complete. 2024-03-28T01:07:02.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.710+0000 7f158d62c740 20 client.27296 trim_cache size 0 max 16384 2024-03-28T01:07:02.715 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:07:02.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.724+0000 7f15897e2640 1 -- 172.21.15.169:0/3619975976 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f156c0a2a10 msgr2=0x7f156c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:02.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.724+0000 7f15897e2640 1 --2- 172.21.15.169:0/3619975976 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f156c0a2a10 0x7f156c0a4ed0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1578096b40 tx=0x7f15780969d0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.724+0000 7f15897e2640 1 -- 172.21.15.169:0/3619975976 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1584109640 msgr2=0x7f15841b6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:02.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.724+0000 7f15897e2640 1 --2- 172.21.15.169:0/3619975976 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1584109640 0x7f15841b6670 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f157800c970 tx=0x7f157800cd70 comp rx=0 tx=0).stop 2024-03-28T01:07:02.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.729+0000 7f15897e2640 1 -- 172.21.15.169:0/3619975976 shutdown_connections 2024-03-28T01:07:02.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.729+0000 7f15897e2640 1 --2- 172.21.15.169:0/3619975976 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f156c0a2a10 0x7f156c0a4ed0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.729+0000 7f15897e2640 1 --2- 172.21.15.169:0/3619975976 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1584109640 0x7f15841b6670 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.729+0000 7f15897e2640 1 --2- 172.21.15.169:0/3619975976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1584108ca0 0x7f1584115260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.729+0000 7f15897e2640 1 --2- 172.21.15.169:0/3619975976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15840729f0 0x7f1584114d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:02.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.729+0000 7f15897e2640 1 -- 172.21.15.169:0/3619975976 >> 172.21.15.169:0/3619975976 conn(0x7f158406e4c0 msgr2=0x7f15840700e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:02.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.732+0000 7f15897e2640 1 -- 172.21.15.169:0/3619975976 shutdown_connections 2024-03-28T01:07:02.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:02.732+0000 7f15897e2640 1 -- 172.21.15.169:0/3619975976 wait complete. 2024-03-28T01:07:02.770 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:07:02.770 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T01:07:03.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:02 smithi169 ceph-mon[32274]: pgmap v667: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 767 MiB/s wr, 1.22k op/s 2024-03-28T01:07:03.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:02 smithi174 ceph-mon[36060]: pgmap v667: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 767 MiB/s wr, 1.22k op/s 2024-03-28T01:07:03.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.122+0000 7f2eae07e640 1 -- 172.21.15.169:0/3525160701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ea810d9c0 msgr2=0x7f2ea810de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:03.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.122+0000 7f2eae07e640 1 --2- 172.21.15.169:0/3525160701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ea810d9c0 0x7f2ea810de40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c01a910 tx=0x7f2e9c03cca0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.123+0000 7f2eae07e640 1 -- 172.21.15.169:0/3525160701 shutdown_connections 2024-03-28T01:07:03.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.123+0000 7f2eae07e640 1 --2- 172.21.15.169:0/3525160701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ea810d9c0 0x7f2ea810de40 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.123+0000 7f2eae07e640 1 --2- 172.21.15.169:0/3525160701 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ea810b010 0x7f2ea810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.123+0000 7f2eae07e640 1 --2- 172.21.15.169:0/3525160701 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ea81086a0 0x7f2ea810aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.123+0000 7f2eae07e640 1 -- 172.21.15.169:0/3525160701 >> 172.21.15.169:0/3525160701 conn(0x7f2ea806e500 msgr2=0x7f2ea8070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:03.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.125+0000 7f2eae07e640 1 -- 172.21.15.169:0/3525160701 shutdown_connections 2024-03-28T01:07:03.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.127+0000 7f2eae07e640 1 -- 172.21.15.169:0/3525160701 wait complete. 2024-03-28T01:07:03.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.128+0000 7f2eae07e640 1 Processor -- start 2024-03-28T01:07:03.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.129+0000 7f2eae07e640 1 -- start start 2024-03-28T01:07:03.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.129+0000 7f2eae07e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ea81086a0 0x7f2ea819e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:03.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.129+0000 7f2eae07e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ea810b010 0x7f2ea819e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:03.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.129+0000 7f2eae07e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ea810d9c0 0x7f2ea81a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:03.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.129+0000 7f2eae07e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea81119c0 con 0x7f2ea810b010 2024-03-28T01:07:03.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.129+0000 7f2eae07e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea8111840 con 0x7f2ea81086a0 2024-03-28T01:07:03.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.129+0000 7f2eae07e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea8111b40 con 0x7f2ea810d9c0 2024-03-28T01:07:03.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.130+0000 7f2ea77fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ea81086a0 0x7f2ea819e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:03.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.130+0000 7f2ea77fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ea81086a0 0x7f2ea819e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33718/0 (socket says 172.21.15.169:33718) 2024-03-28T01:07:03.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.130+0000 7f2ea77fe640 1 -- 172.21.15.169:0/480030496 learned_addr learned my addr 172.21.15.169:0/480030496 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:03.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.130+0000 7f2ea77fe640 1 -- 172.21.15.169:0/480030496 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ea810d9c0 msgr2=0x7f2ea81a55b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:07:03.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.130+0000 7f2ea77fe640 1 --2- 172.21.15.169:0/480030496 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ea810d9c0 0x7f2ea81a55b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.130+0000 7f2ea77fe640 1 -- 172.21.15.169:0/480030496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ea810b010 msgr2=0x7f2ea819e540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:07:03.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.130+0000 7f2ea77fe640 1 --2- 172.21.15.169:0/480030496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ea810b010 0x7f2ea819e540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.130+0000 7f2ea77fe640 1 -- 172.21.15.169:0/480030496 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea81a5c20 con 0x7f2ea81086a0 2024-03-28T01:07:03.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.131+0000 7f2ea77fe640 1 --2- 172.21.15.169:0/480030496 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ea81086a0 0x7f2ea819e000 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2e9800b450 tx=0x7f2e9800b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:03.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.141+0000 7f2ea4ff9640 1 -- 172.21.15.169:0/480030496 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e98015020 con 0x7f2ea81086a0 2024-03-28T01:07:03.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.141+0000 7f2ea4ff9640 1 -- 172.21.15.169:0/480030496 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e98011d70 con 0x7f2ea81086a0 2024-03-28T01:07:03.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.141+0000 7f2ea4ff9640 1 -- 172.21.15.169:0/480030496 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9800f340 con 0x7f2ea81086a0 2024-03-28T01:07:03.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.141+0000 7f2eae07e640 1 -- 172.21.15.169:0/480030496 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea81a5f10 con 0x7f2ea81086a0 2024-03-28T01:07:03.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.141+0000 7f2eae07e640 1 -- 172.21.15.169:0/480030496 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea81a6400 con 0x7f2ea81086a0 2024-03-28T01:07:03.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.144+0000 7f2ea4ff9640 1 -- 172.21.15.169:0/480030496 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2e98018020 con 0x7f2ea81086a0 2024-03-28T01:07:03.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.145+0000 7f2ea4ff9640 1 --2- 172.21.15.169:0/480030496 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2e900a29d0 0x7f2e900a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:03.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.146+0000 7f2ea4ff9640 1 -- 172.21.15.169:0/480030496 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2e980cf960 con 0x7f2ea81086a0 2024-03-28T01:07:03.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.148+0000 7f2eb0c65740 1 Processor -- start 2024-03-28T01:07:03.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.150+0000 7f2eb0c65740 1 -- start start 2024-03-28T01:07:03.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.150+0000 7f2eb0c65740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ebfff690 0x55a1ebfffab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:03.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.151+0000 7f2eb0c65740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a1ec000080 0x55a1ec004b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:03.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.151+0000 7f2ea6ffd640 1 --2- 172.21.15.169:0/480030496 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2e900a29d0 0x7f2e900a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:03.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.151+0000 7f2eb0c65740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec0051f0 0x55a1ec007680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:03.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.151+0000 7f2eb0c65740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1ebfc2610 con 0x55a1ec000080 2024-03-28T01:07:03.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.151+0000 7f2eb0c65740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1ebf653c0 con 0x55a1ebfff690 2024-03-28T01:07:03.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.151+0000 7f2eb0c65740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1ebf0dd60 con 0x55a1ec0051f0 2024-03-28T01:07:03.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.151+0000 7f2ea77fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ebfff690 0x55a1ebfffab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:03.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.151+0000 7f2ea77fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ebfff690 0x55a1ebfffab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33730/0 (socket says 172.21.15.169:33730) 2024-03-28T01:07:03.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.151+0000 7f2ea77fe640 1 -- 172.21.15.169:0/3135277684 learned_addr learned my addr 172.21.15.169:0/3135277684 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:03.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.152+0000 7f2ea77fe640 1 -- 172.21.15.169:0/3135277684 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec0051f0 msgr2=0x55a1ec007680 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:07:03.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.152+0000 7f2ea77fe640 1 --2- 172.21.15.169:0/3135277684 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec0051f0 0x55a1ec007680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.152+0000 7f2ea77fe640 1 -- 172.21.15.169:0/3135277684 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a1ec000080 msgr2=0x55a1ec004b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:03.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.152+0000 7f2ea77fe640 1 --2- 172.21.15.169:0/3135277684 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a1ec000080 0x55a1ec004b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.152+0000 7f2ea77fe640 1 -- 172.21.15.169:0/3135277684 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a1ebc793f0 con 0x55a1ebfff690 2024-03-28T01:07:03.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.152+0000 7f2ea77fe640 1 --2- 172.21.15.169:0/3135277684 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ebfff690 0x55a1ebfffab0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2e980a4170 tx=0x7f2e9809bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb971ef5ee8a98e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:03.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.155+0000 7f2e8dffb640 1 -- 172.21.15.169:0/3135277684 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9809a200 con 0x55a1ebfff690 2024-03-28T01:07:03.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.155+0000 7f2e8dffb640 1 -- 172.21.15.169:0/3135277684 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e980cd170 con 0x55a1ebfff690 2024-03-28T01:07:03.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.156+0000 7f2eb0c65740 1 -- 172.21.15.169:0/3135277684 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ebfff690 msgr2=0x55a1ebfffab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:03.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.156+0000 7f2eb0c65740 1 --2- 172.21.15.169:0/3135277684 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ebfff690 0x55a1ebfffab0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2e980a4170 tx=0x7f2e9809bb70 comp rx=0 tx=0).stop 2024-03-28T01:07:03.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.160+0000 7f2ea6ffd640 1 --2- 172.21.15.169:0/480030496 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2e900a29d0 0x7f2e900a4e90 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x55a1ec000480 tx=0x7f2ea0009990 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:03.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.160+0000 7f2eb0c65740 1 -- 172.21.15.169:0/3135277684 shutdown_connections 2024-03-28T01:07:03.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.160+0000 7f2eb0c65740 1 --2- 172.21.15.169:0/3135277684 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec0051f0 0x55a1ec007680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.160+0000 7f2eb0c65740 1 --2- 172.21.15.169:0/3135277684 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a1ec000080 0x55a1ec004b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.160+0000 7f2eb0c65740 1 --2- 172.21.15.169:0/3135277684 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ebfff690 0x55a1ebfffab0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.163 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.160+0000 7f2eb0c65740 1 -- 172.21.15.169:0/3135277684 >> 172.21.15.169:0/3135277684 conn(0x55a1ebffac00 msgr2=0x55a1ebffb000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:03.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:02 smithi177 ceph-mon[36206]: pgmap v667: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 767 MiB/s wr, 1.22k op/s 2024-03-28T01:07:03.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.169+0000 7f2eb0c65740 1 -- 172.21.15.169:0/3135277684 shutdown_connections 2024-03-28T01:07:03.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.169+0000 7f2eb0c65740 1 -- 172.21.15.169:0/3135277684 wait complete. 2024-03-28T01:07:03.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.169+0000 7f2eb0c65740 1 -- start start 2024-03-28T01:07:03.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.170+0000 7f2eb0c65740 1 Processor -- start 2024-03-28T01:07:03.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.170+0000 7f2eb0c65740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a1ebfff690 0x55a1ec009a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:03.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.170+0000 7f2eb0c65740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ec000080 0x55a1ec009fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:03.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.170+0000 7f2eb0c65740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec0051f0 0x55a1ec010ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:03.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.170+0000 7f2eb0c65740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1ebf21240 con 0x55a1ebfff690 2024-03-28T01:07:03.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.170+0000 7f2eb0c65740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1ebf61460 con 0x55a1ec000080 2024-03-28T01:07:03.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.170+0000 7f2eb0c65740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1ec0081f0 con 0x55a1ec0051f0 2024-03-28T01:07:03.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.170+0000 7f2eb0c65740 10 client.0 _ll_register_callbacks cb 0x55a1ebf20c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:03.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.172+0000 7f2ea77fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a1ebfff690 0x55a1ec009a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:03.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.172+0000 7f2ea6ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ec000080 0x55a1ec009fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:03.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.173+0000 7f2ea7fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec0051f0 0x55a1ec010ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:03.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.173+0000 7f2ea7fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec0051f0 0x55a1ec010ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56262/0 (socket says 172.21.15.169:56262) 2024-03-28T01:07:03.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.173+0000 7f2ea7fff640 1 -- 172.21.15.169:0/693425921 learned_addr learned my addr 172.21.15.169:0/693425921 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:03.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.177+0000 7f2ea6ffd640 1 -- 172.21.15.169:0/693425921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec0051f0 msgr2=0x55a1ec010ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:03.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.177+0000 7f2ea6ffd640 1 --2- 172.21.15.169:0/693425921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec0051f0 0x55a1ec010ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.177+0000 7f2ea6ffd640 1 -- 172.21.15.169:0/693425921 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a1ebfff690 msgr2=0x55a1ec009a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:03.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.177+0000 7f2ea6ffd640 1 --2- 172.21.15.169:0/693425921 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a1ebfff690 0x55a1ec009a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.177+0000 7f2ea6ffd640 1 -- 172.21.15.169:0/693425921 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a1ebd437d0 con 0x55a1ec000080 2024-03-28T01:07:03.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.177+0000 7f2ea7fff640 1 --2- 172.21.15.169:0/693425921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec0051f0 0x55a1ec010ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:07:03.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.180+0000 7f2ea6ffd640 1 --2- 172.21.15.169:0/693425921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ec000080 0x55a1ec009fb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2ea819f510 tx=0x7f2ea0004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d45c0b267c5cd3f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:03.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.181+0000 7f2e8b7f6640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:07:03.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.181+0000 7f2e8b7f6640 1 -- 172.21.15.169:0/693425921 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0023070 con 0x55a1ec000080 2024-03-28T01:07:03.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.183+0000 7f2eb0c65740 1 -- 172.21.15.169:0/693425921 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2e980cd170 con 0x55a1ec000080 2024-03-28T01:07:03.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.184+0000 7f2e8b7f6640 1 -- 172.21.15.169:0/693425921 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2ea001f3f0 con 0x55a1ec000080 2024-03-28T01:07:03.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.186+0000 7f2e8b7f6640 1 -- 172.21.15.169:0/693425921 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ea0012e40 con 0x55a1ec000080 2024-03-28T01:07:03.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.196+0000 7f2eb0c65740 10 client.27308 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:03.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.196+0000 7f2eb0c65740 1 -- 172.21.15.169:0/693425921 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a1ec014450 con 0x55a1ec000080 2024-03-28T01:07:03.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.200+0000 7f2e8b7f6640 1 -- 172.21.15.169:0/693425921 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e980cd170 con 0x55a1ec000080 2024-03-28T01:07:03.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.200+0000 7f2e8b7f6640 1 -- 172.21.15.169:0/693425921 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2ea001e070 con 0x55a1ec000080 2024-03-28T01:07:03.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.200+0000 7f2eb0c65740 10 client.27308 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:03.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.200+0000 7f2eb0c65740 10 client.27308 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:07:03.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.200+0000 7f2eb0c65740 20 client.27308 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:03.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.200+0000 7f2eb0c65740 1 --2- 172.21.15.169:0/693425921 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a1ec015a60 0x55a1ec017ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:03.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.201+0000 7f2eb0c65740 4 client.27308 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:07:03.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.201+0000 7f2eb0c65740 1 -- 172.21.15.169:0/693425921 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a1ebf72900 con 0x55a1ec015a60 2024-03-28T01:07:03.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.207+0000 7f2ea77fe640 1 --2- 172.21.15.169:0/693425921 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a1ec015a60 0x55a1ec017ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:03.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.229+0000 7f2ea77fe640 1 --2- 172.21.15.169:0/693425921 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a1ec015a60 0x55a1ec017ea0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23b6249d07639450 server_cookie=abaab29cf8c0b5a9 in_seq=0 out_seq=0 2024-03-28T01:07:03.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.230+0000 7f2e8b7f6640 10 client.27308 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:03.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.255+0000 7f2e8b7f6640 1 -- 172.21.15.169:0/693425921 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a1ebf72900 con 0x55a1ec015a60 2024-03-28T01:07:03.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.255+0000 7f2e8b7f6640 10 client.27308 handle_command_reply: tid=0 2024-03-28T01:07:03.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.255+0000 7f2eb0c65740 1 client.27308 shutdown 2024-03-28T01:07:03.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.256+0000 7f2eb0c65740 1 -- 172.21.15.169:0/693425921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ec000080 msgr2=0x55a1ec009fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:03.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.256+0000 7f2eb0c65740 1 --2- 172.21.15.169:0/693425921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ec000080 0x55a1ec009fb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2ea819f510 tx=0x7f2ea0004ab0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.258+0000 7f2eb0c65740 1 -- 172.21.15.169:0/693425921 shutdown_connections 2024-03-28T01:07:03.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.258+0000 7f2eb0c65740 1 --2- 172.21.15.169:0/693425921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec0051f0 0x55a1ec010ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.258+0000 7f2eb0c65740 1 --2- 172.21.15.169:0/693425921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ec000080 0x55a1ec009fb0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.258+0000 7f2eb0c65740 1 --2- 172.21.15.169:0/693425921 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a1ec015a60 0x55a1ec017ea0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.258+0000 7f2eb0c65740 1 --2- 172.21.15.169:0/693425921 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a1ebfff690 0x55a1ec009a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.258+0000 7f2eb0c65740 1 -- 172.21.15.169:0/693425921 >> 172.21.15.169:0/693425921 conn(0x55a1ebffac00 msgr2=0x55a1ebff9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:03.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.258+0000 7f2e8b7f6640 0 client.27308 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:03.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.262+0000 7f2eb0c65740 1 -- 172.21.15.169:0/693425921 shutdown_connections 2024-03-28T01:07:03.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.267+0000 7f2eb0c65740 1 -- 172.21.15.169:0/693425921 wait complete. 2024-03-28T01:07:03.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.270+0000 7f2eb0c65740 20 client.27308 trim_cache size 0 max 16384 2024-03-28T01:07:03.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.299+0000 7f2eb0c65740 1 Processor -- start 2024-03-28T01:07:03.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.300+0000 7f2eb0c65740 1 -- start start 2024-03-28T01:07:03.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.300+0000 7f2eb0c65740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a1ebfff690 0x55a1ebff9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:03.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.300+0000 7f2eb0c65740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec000080 0x55a1ec010fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:03.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.301+0000 7f2eb0c65740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ec0051f0 0x55a1ebff9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:03.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.301+0000 7f2eb0c65740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e94012000 con 0x55a1ebfff690 2024-03-28T01:07:03.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.301+0000 7f2ea7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ec0051f0 0x55a1ebff9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:03.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.301+0000 7f2ea7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ec0051f0 0x55a1ebff9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33750/0 (socket says 172.21.15.169:33750) 2024-03-28T01:07:03.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.301+0000 7f2ea7fff640 1 -- 172.21.15.169:0/4175528583 learned_addr learned my addr 172.21.15.169:0/4175528583 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:03.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.301+0000 7f2ea6ffd640 1 --2- 172.21.15.169:0/4175528583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec000080 0x55a1ec010fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:03.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.301+0000 7f2eb0c65740 1 -- 172.21.15.169:0/4175528583 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9400fb00 con 0x55a1ec0051f0 2024-03-28T01:07:03.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.301+0000 7f2eb0c65740 1 -- 172.21.15.169:0/4175528583 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9400dba0 con 0x55a1ec000080 2024-03-28T01:07:03.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.301+0000 7f2ea77fe640 1 --2- 172.21.15.169:0/4175528583 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a1ebfff690 0x55a1ebff9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:03.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.307+0000 7f2ea6ffd640 1 -- 172.21.15.169:0/4175528583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ec0051f0 msgr2=0x55a1ebff9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:03.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.307+0000 7f2ea6ffd640 1 --2- 172.21.15.169:0/4175528583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ec0051f0 0x55a1ebff9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.307+0000 7f2ea6ffd640 1 -- 172.21.15.169:0/4175528583 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a1ebfff690 msgr2=0x55a1ebff9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:03.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.307+0000 7f2ea6ffd640 1 --2- 172.21.15.169:0/4175528583 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a1ebfff690 0x55a1ebff9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.307+0000 7f2ea6ffd640 1 -- 172.21.15.169:0/4175528583 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a1ebff6480 con 0x55a1ec000080 2024-03-28T01:07:03.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.309+0000 7f2ea6ffd640 1 --2- 172.21.15.169:0/4175528583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec000080 0x55a1ec010fb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2ea001e720 tx=0x7f2ea0011f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f836f15b5d6767b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:03.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.314+0000 7f2e8d7fa640 1 -- 172.21.15.169:0/4175528583 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0023040 con 0x55a1ec000080 2024-03-28T01:07:03.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.315+0000 7f2e8d7fa640 1 -- 172.21.15.169:0/4175528583 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ea0019070 con 0x55a1ec000080 2024-03-28T01:07:03.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.315+0000 7f2e8d7fa640 1 -- 172.21.15.169:0/4175528583 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea001e990 con 0x55a1ec000080 2024-03-28T01:07:03.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.317+0000 7f2eb0c65740 1 -- 172.21.15.169:0/4175528583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec000080 msgr2=0x55a1ec010fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:03.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.317+0000 7f2eb0c65740 1 --2- 172.21.15.169:0/4175528583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec000080 0x55a1ec010fb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2ea001e720 tx=0x7f2ea0011f90 comp rx=0 tx=0).stop 2024-03-28T01:07:03.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.321+0000 7f2eb0c65740 1 -- 172.21.15.169:0/4175528583 shutdown_connections 2024-03-28T01:07:03.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.322+0000 7f2eb0c65740 1 --2- 172.21.15.169:0/4175528583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ec0051f0 0x55a1ebff9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.322+0000 7f2eb0c65740 1 --2- 172.21.15.169:0/4175528583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec000080 0x55a1ec010fb0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.322+0000 7f2eb0c65740 1 --2- 172.21.15.169:0/4175528583 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a1ebfff690 0x55a1ebff9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.322+0000 7f2eb0c65740 1 -- 172.21.15.169:0/4175528583 >> 172.21.15.169:0/4175528583 conn(0x55a1ebffac00 msgr2=0x55a1ec0014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:03.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.328+0000 7f2eb0c65740 1 -- 172.21.15.169:0/4175528583 shutdown_connections 2024-03-28T01:07:03.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.329+0000 7f2eb0c65740 1 -- 172.21.15.169:0/4175528583 wait complete. 2024-03-28T01:07:03.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.329+0000 7f2eb0c65740 1 -- start start 2024-03-28T01:07:03.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.329+0000 7f2eb0c65740 1 Processor -- start 2024-03-28T01:07:03.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.333+0000 7f2eb0c65740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a1ebfff690 0x55a1ec00eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:03.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.333+0000 7f2eb0c65740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec000080 0x55a1ec00f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:03.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.333+0000 7f2eb0c65740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ec0051f0 0x55a1ec010010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:03.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.334+0000 7f2eb0c65740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9400e1a0 con 0x55a1ebfff690 2024-03-28T01:07:03.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.334+0000 7f2ea6ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec000080 0x55a1ec00f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:03.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.334+0000 7f2ea6ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec000080 0x55a1ec00f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56282/0 (socket says 172.21.15.169:56282) 2024-03-28T01:07:03.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.334+0000 7f2ea6ffd640 1 -- 172.21.15.169:0/6546157 learned_addr learned my addr 172.21.15.169:0/6546157 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:03.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.334+0000 7f2ea7fff640 1 --2- 172.21.15.169:0/6546157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ec0051f0 0x55a1ec010010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:03.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.335+0000 7f2ea77fe640 1 --2- 172.21.15.169:0/6546157 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a1ebfff690 0x55a1ec00eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:03.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.334+0000 7f2eb0c65740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9400f4a0 con 0x55a1ec0051f0 2024-03-28T01:07:03.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.335+0000 7f2eb0c65740 1 -- 172.21.15.169:0/6546157 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9400e7e0 con 0x55a1ec000080 2024-03-28T01:07:03.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.336+0000 7f2ea7fff640 1 -- 172.21.15.169:0/6546157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec000080 msgr2=0x55a1ec00f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:03.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.336+0000 7f2ea7fff640 1 --2- 172.21.15.169:0/6546157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec000080 0x55a1ec00f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.336+0000 7f2ea7fff640 1 -- 172.21.15.169:0/6546157 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a1ebfff690 msgr2=0x55a1ec00eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:03.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.336+0000 7f2ea7fff640 1 --2- 172.21.15.169:0/6546157 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a1ebfff690 0x55a1ec00eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.336+0000 7f2ea7fff640 1 -- 172.21.15.169:0/6546157 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a1ec010600 con 0x55a1ec0051f0 2024-03-28T01:07:03.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.336+0000 7f2ea7fff640 1 --2- 172.21.15.169:0/6546157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ec0051f0 0x55a1ec010010 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x55a1ebffffc0 tx=0x7f2e9c0048f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8591c01f3173ba83 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:03.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.339+0000 7f2eb0c65740 10 client.0 _ll_register_callbacks cb 0x7f2ea0012820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:03.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.339+0000 7f2eb0c65740 1 -- 172.21.15.169:0/6546157 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2ea0019070 con 0x55a1ec0051f0 2024-03-28T01:07:03.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.342+0000 7f2e8b7f6640 10 client.27314 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:07:03.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.343+0000 7f2e8b7f6640 1 -- 172.21.15.169:0/6546157 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c005d10 con 0x55a1ec0051f0 2024-03-28T01:07:03.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.343+0000 7f2e8b7f6640 1 -- 172.21.15.169:0/6546157 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2e9c04a040 con 0x55a1ec0051f0 2024-03-28T01:07:03.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.343+0000 7f2e8b7f6640 1 -- 172.21.15.169:0/6546157 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e9c061050 con 0x55a1ec0051f0 2024-03-28T01:07:03.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.348+0000 7f2eb0c65740 10 client.27314 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:03.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.348+0000 7f2eb0c65740 1 -- 172.21.15.169:0/6546157 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x55a1ec0716d0 con 0x55a1ec0051f0 2024-03-28T01:07:03.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.348+0000 7f2e8b7f6640 1 -- 172.21.15.169:0/6546157 <== mon.2 v2:172.21.15.174:3300/0 4 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2e9c056020 con 0x55a1ec0051f0 2024-03-28T01:07:03.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.354+0000 7f2eb0c65740 10 client.27314 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:03.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.354+0000 7f2eb0c65740 10 client.27314 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:07:03.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.354+0000 7f2eb0c65740 20 client.27314 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:03.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.354+0000 7f2eb0c65740 1 --2- 172.21.15.169:0/6546157 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a1ec071df0 0x55a1ec074210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:03.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.354+0000 7f2eb0c65740 4 client.27314 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:07:03.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.354+0000 7f2eb0c65740 1 -- 172.21.15.169:0/6546157 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a1ebf6cc50 con 0x55a1ec071df0 2024-03-28T01:07:03.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.354+0000 7f2ea77fe640 1 --2- 172.21.15.169:0/6546157 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a1ec071df0 0x55a1ec074210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:03.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.355+0000 7f2ea77fe640 1 --2- 172.21.15.169:0/6546157 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a1ec071df0 0x55a1ec074210 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad00e900f9819bf0 server_cookie=682f24e63d83ce0 in_seq=0 out_seq=0 2024-03-28T01:07:03.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.355+0000 7f2e8b7f6640 10 client.27314 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:03.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.359+0000 7f2e8b7f6640 1 -- 172.21.15.169:0/6546157 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a1ebf6cc50 con 0x55a1ec071df0 2024-03-28T01:07:03.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.359+0000 7f2e8b7f6640 10 client.27314 handle_command_reply: tid=0 2024-03-28T01:07:03.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.359+0000 7f2eb0c65740 1 client.27314 shutdown 2024-03-28T01:07:03.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.365+0000 7f2eb0c65740 1 -- 172.21.15.169:0/6546157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ec0051f0 msgr2=0x55a1ec010010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:03.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.365+0000 7f2eb0c65740 1 --2- 172.21.15.169:0/6546157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ec0051f0 0x55a1ec010010 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x55a1ebffffc0 tx=0x7f2e9c0048f0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.368+0000 7f2eb0c65740 1 -- 172.21.15.169:0/6546157 shutdown_connections 2024-03-28T01:07:03.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.368+0000 7f2eb0c65740 1 --2- 172.21.15.169:0/6546157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1ec0051f0 0x55a1ec010010 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.368+0000 7f2eb0c65740 1 --2- 172.21.15.169:0/6546157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a1ec000080 0x55a1ec00f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.368+0000 7f2eb0c65740 1 --2- 172.21.15.169:0/6546157 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55a1ec071df0 0x55a1ec074210 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.368+0000 7f2eb0c65740 1 --2- 172.21.15.169:0/6546157 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55a1ebfff690 0x55a1ec00eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.368+0000 7f2eb0c65740 1 -- 172.21.15.169:0/6546157 >> 172.21.15.169:0/6546157 conn(0x55a1ebffac00 msgr2=0x55a1ebffde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:03.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.368+0000 7f2e8b7f6640 0 client.27314 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:03.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.371+0000 7f2eb0c65740 1 -- 172.21.15.169:0/6546157 shutdown_connections 2024-03-28T01:07:03.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.374+0000 7f2eb0c65740 1 -- 172.21.15.169:0/6546157 wait complete. 2024-03-28T01:07:03.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.374+0000 7f2eb0c65740 20 client.27314 trim_cache size 0 max 16384 2024-03-28T01:07:03.382 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:07:03.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.387+0000 7f2eae07e640 1 -- 172.21.15.169:0/480030496 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2e900a29d0 msgr2=0x7f2e900a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:03.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.387+0000 7f2eae07e640 1 --2- 172.21.15.169:0/480030496 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2e900a29d0 0x7f2e900a4e90 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x55a1ec000480 tx=0x7f2ea0009990 comp rx=0 tx=0).stop 2024-03-28T01:07:03.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.387+0000 7f2eae07e640 1 -- 172.21.15.169:0/480030496 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ea81086a0 msgr2=0x7f2ea819e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:03.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.387+0000 7f2eae07e640 1 --2- 172.21.15.169:0/480030496 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ea81086a0 0x7f2ea819e000 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2e9800b450 tx=0x7f2e9800b910 comp rx=0 tx=0).stop 2024-03-28T01:07:03.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.392+0000 7f2eae07e640 1 -- 172.21.15.169:0/480030496 shutdown_connections 2024-03-28T01:07:03.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.392+0000 7f2eae07e640 1 --2- 172.21.15.169:0/480030496 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2e900a29d0 0x7f2e900a4e90 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.392+0000 7f2eae07e640 1 --2- 172.21.15.169:0/480030496 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ea810d9c0 0x7f2ea81a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.392+0000 7f2eae07e640 1 --2- 172.21.15.169:0/480030496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ea810b010 0x7f2ea819e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.392+0000 7f2eae07e640 1 --2- 172.21.15.169:0/480030496 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ea81086a0 0x7f2ea819e000 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:03.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.392+0000 7f2eae07e640 1 -- 172.21.15.169:0/480030496 >> 172.21.15.169:0/480030496 conn(0x7f2ea806e500 msgr2=0x7f2ea810b790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:03.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.397+0000 7f2eae07e640 1 -- 172.21.15.169:0/480030496 shutdown_connections 2024-03-28T01:07:03.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:03.402+0000 7f2eae07e640 1 -- 172.21.15.169:0/480030496 wait complete. 2024-03-28T01:07:04.437 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T01:07:04.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T01:07:04.438 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T01:07:04.782 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.779+0000 7fa5d400b640 1 -- 172.21.15.169:0/3763880854 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5cc0ff330 msgr2=0x7fa5cc0ff750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:04.783 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.779+0000 7fa5d400b640 1 --2- 172.21.15.169:0/3763880854 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5cc0ff330 0x7fa5cc0ff750 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009ca0 tx=0x7fa5c402c140 comp rx=0 tx=0).stop 2024-03-28T01:07:04.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.793+0000 7fa5d400b640 1 -- 172.21.15.169:0/3763880854 shutdown_connections 2024-03-28T01:07:04.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.793+0000 7fa5d400b640 1 --2- 172.21.15.169:0/3763880854 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5cc107220 0x7fa5cc1096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.793+0000 7fa5d400b640 1 --2- 172.21.15.169:0/3763880854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5cc0ffc90 0x7fa5cc106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.793+0000 7fa5d400b640 1 --2- 172.21.15.169:0/3763880854 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5cc0ff330 0x7fa5cc0ff750 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.796 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.793+0000 7fa5d400b640 1 -- 172.21.15.169:0/3763880854 >> 172.21.15.169:0/3763880854 conn(0x7fa5cc0fad60 msgr2=0x7fa5cc0fd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:04.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.799+0000 7fa5d400b640 1 -- 172.21.15.169:0/3763880854 shutdown_connections 2024-03-28T01:07:04.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.803+0000 7fa5d400b640 1 -- 172.21.15.169:0/3763880854 wait complete. 2024-03-28T01:07:04.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.804+0000 7fa5d400b640 1 Processor -- start 2024-03-28T01:07:04.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.804+0000 7fa5d400b640 1 -- start start 2024-03-28T01:07:04.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.804+0000 7fa5d400b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5cc0ff330 0x7fa5cc06f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:04.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.804+0000 7fa5d400b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5cc0ffc90 0x7fa5cc06f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:04.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.804+0000 7fa5d400b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5cc107220 0x7fa5cc0728d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:04.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.804+0000 7fa5d400b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc06bb10 con 0x7fa5cc0ffc90 2024-03-28T01:07:04.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.804+0000 7fa5d400b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc06b990 con 0x7fa5cc107220 2024-03-28T01:07:04.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.804+0000 7fa5d400b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc06bc90 con 0x7fa5cc0ff330 2024-03-28T01:07:04.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.805+0000 7fa5d1da7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5cc0ff330 0x7fa5cc06f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:04.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.805+0000 7fa5d1da7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5cc0ff330 0x7fa5cc06f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56296/0 (socket says 172.21.15.169:56296) 2024-03-28T01:07:04.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.805+0000 7fa5d1da7640 1 -- 172.21.15.169:0/790199969 learned_addr learned my addr 172.21.15.169:0/790199969 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:04.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.806+0000 7fa5d25a8640 1 --2- 172.21.15.169:0/790199969 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5cc107220 0x7fa5cc0728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:04.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.806+0000 7fa5d1da7640 1 -- 172.21.15.169:0/790199969 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5cc107220 msgr2=0x7fa5cc0728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:04.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.806+0000 7fa5d1da7640 1 --2- 172.21.15.169:0/790199969 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5cc107220 0x7fa5cc0728d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.806+0000 7fa5d1da7640 1 -- 172.21.15.169:0/790199969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5cc0ffc90 msgr2=0x7fa5cc06f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:04.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.806+0000 7fa5d1da7640 1 --2- 172.21.15.169:0/790199969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5cc0ffc90 0x7fa5cc06f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.806+0000 7fa5d1da7640 1 -- 172.21.15.169:0/790199969 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5cc072f40 con 0x7fa5cc0ff330 2024-03-28T01:07:04.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.809+0000 7fa5d1da7640 1 --2- 172.21.15.169:0/790199969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5cc0ff330 0x7fa5cc06f340 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4037040 tx=0x7fa5c4035e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:04.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.814+0000 7fa5caffd640 1 -- 172.21.15.169:0/790199969 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4039040 con 0x7fa5cc0ff330 2024-03-28T01:07:04.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.814+0000 7fa5caffd640 1 -- 172.21.15.169:0/790199969 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5c4044070 con 0x7fa5cc0ff330 2024-03-28T01:07:04.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.814+0000 7fa5d400b640 1 -- 172.21.15.169:0/790199969 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5cc0731d0 con 0x7fa5cc0ff330 2024-03-28T01:07:04.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.816+0000 7fa5d400b640 1 -- 172.21.15.169:0/790199969 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5cc1aee90 con 0x7fa5cc0ff330 2024-03-28T01:07:04.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.821+0000 7fa5d6bf3740 1 Processor -- start 2024-03-28T01:07:04.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.821+0000 7fa5caffd640 1 -- 172.21.15.169:0/790199969 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4004800 con 0x7fa5cc0ff330 2024-03-28T01:07:04.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.822+0000 7fa5d6bf3740 1 -- start start 2024-03-28T01:07:04.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.822+0000 7fa5d6bf3740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819f7b30 0x55dd819f7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:04.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.823+0000 7fa5d6bf3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd819f8520 0x55dd819fd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:04.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.823+0000 7fa5d6bf3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd819fd690 0x55dd819ffb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:04.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.823+0000 7fa5d6bf3740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd81937c90 con 0x55dd819f7b30 2024-03-28T01:07:04.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.823+0000 7fa5d6bf3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8195d070 con 0x55dd819f8520 2024-03-28T01:07:04.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.823+0000 7fa5d6bf3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd81905c80 con 0x55dd819fd690 2024-03-28T01:07:04.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.823+0000 7fa5d15a6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd819fd690 0x55dd819ffb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:04.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.824+0000 7fa5d15a6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd819fd690 0x55dd819ffb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56304/0 (socket says 172.21.15.169:56304) 2024-03-28T01:07:04.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.824+0000 7fa5d15a6640 1 -- 172.21.15.169:0/1385689876 learned_addr learned my addr 172.21.15.169:0/1385689876 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:04.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.826+0000 7fa5d15a6640 1 -- 172.21.15.169:0/1385689876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd819f8520 msgr2=0x55dd819fd020 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:07:04.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.826+0000 7fa5d15a6640 1 --2- 172.21.15.169:0/1385689876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd819f8520 0x55dd819fd020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.826+0000 7fa5d15a6640 1 -- 172.21.15.169:0/1385689876 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819f7b30 msgr2=0x55dd819f7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:04.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.826+0000 7fa5d15a6640 1 --2- 172.21.15.169:0/1385689876 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819f7b30 0x55dd819f7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.826+0000 7fa5d15a6640 1 -- 172.21.15.169:0/1385689876 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd818fdd10 con 0x55dd819fd690 2024-03-28T01:07:04.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.828+0000 7fa5d15a6640 1 --2- 172.21.15.169:0/1385689876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd819fd690 0x55dd819ffb20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc009790 tx=0x7fa5bc009c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d60ad2a568ef0062 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:04.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.830+0000 7fa5b3fff640 1 -- 172.21.15.169:0/1385689876 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc015020 con 0x55dd819fd690 2024-03-28T01:07:04.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.830+0000 7fa5b3fff640 1 -- 172.21.15.169:0/1385689876 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5bc011420 con 0x55dd819fd690 2024-03-28T01:07:04.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.830+0000 7fa5caffd640 1 -- 172.21.15.169:0/790199969 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa5c4003970 con 0x7fa5cc0ff330 2024-03-28T01:07:04.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.831+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/1385689876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd819fd690 msgr2=0x55dd819ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:04.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.831+0000 7fa5d6bf3740 1 --2- 172.21.15.169:0/1385689876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd819fd690 0x55dd819ffb20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc009790 tx=0x7fa5bc009c50 comp rx=0 tx=0).stop 2024-03-28T01:07:04.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.832+0000 7fa5caffd640 1 --2- 172.21.15.169:0/790199969 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa5b40a2820 0x7fa5b40a4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:04.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.832+0000 7fa5caffd640 1 -- 172.21.15.169:0/790199969 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa5c40f1050 con 0x7fa5cc0ff330 2024-03-28T01:07:04.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.837+0000 7fa5d25a8640 1 --2- 172.21.15.169:0/790199969 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa5b40a2820 0x7fa5b40a4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:04.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.838+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/1385689876 shutdown_connections 2024-03-28T01:07:04.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.838+0000 7fa5d6bf3740 1 --2- 172.21.15.169:0/1385689876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd819fd690 0x55dd819ffb20 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.838+0000 7fa5d6bf3740 1 --2- 172.21.15.169:0/1385689876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd819f8520 0x55dd819fd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.838+0000 7fa5d6bf3740 1 --2- 172.21.15.169:0/1385689876 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819f7b30 0x55dd819f7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.838+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/1385689876 >> 172.21.15.169:0/1385689876 conn(0x55dd819c01a0 msgr2=0x55dd819f34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:04.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.846+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/1385689876 shutdown_connections 2024-03-28T01:07:04.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.847+0000 7fa5d25a8640 1 --2- 172.21.15.169:0/790199969 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa5b40a2820 0x7fa5b40a4ce0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x55dd819f8d80 tx=0x7fa5c0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:04.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.849+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/1385689876 wait complete. 2024-03-28T01:07:04.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.849+0000 7fa5d6bf3740 1 -- start start 2024-03-28T01:07:04.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.850+0000 7fa5d6bf3740 1 Processor -- start 2024-03-28T01:07:04.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.860+0000 7fa5d6bf3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd819f7b30 0x55dd81a12c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:04.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.860+0000 7fa5d6bf3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd819f8520 0x55dd81a13150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:04.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.860+0000 7fa5d6bf3740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819fd690 0x55dd81a1a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:04.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.860+0000 7fa5d6bf3740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd819190d0 con 0x55dd819fd690 2024-03-28T01:07:04.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.860+0000 7fa5d6bf3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd81959110 con 0x55dd819f8520 2024-03-28T01:07:04.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.860+0000 7fa5d6bf3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd81a116a0 con 0x55dd819f7b30 2024-03-28T01:07:04.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.860+0000 7fa5d6bf3740 10 client.0 _ll_register_callbacks cb 0x55dd81954e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:04.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.861+0000 7fa5d25a8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819fd690 0x55dd81a1a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:04.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.861+0000 7fa5d25a8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819fd690 0x55dd81a1a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36030/0 (socket says 172.21.15.169:36030) 2024-03-28T01:07:04.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.861+0000 7fa5d25a8640 1 -- 172.21.15.169:0/943494869 learned_addr learned my addr 172.21.15.169:0/943494869 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:04.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.862+0000 7fa5d15a6640 1 --2- 172.21.15.169:0/943494869 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd819f8520 0x55dd81a13150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:04.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.864+0000 7fa5d1da7640 1 --2- 172.21.15.169:0/943494869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd819f7b30 0x55dd81a12c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:04.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.864+0000 7fa5d25a8640 1 -- 172.21.15.169:0/943494869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd819f7b30 msgr2=0x55dd81a12c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:04.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.864+0000 7fa5d25a8640 1 --2- 172.21.15.169:0/943494869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd819f7b30 0x55dd81a12c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.864+0000 7fa5d25a8640 1 -- 172.21.15.169:0/943494869 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd819f8520 msgr2=0x55dd81a13150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:04.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.864+0000 7fa5d25a8640 1 --2- 172.21.15.169:0/943494869 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd819f8520 0x55dd81a13150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.864+0000 7fa5d25a8640 1 -- 172.21.15.169:0/943494869 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dd817397e0 con 0x55dd819fd690 2024-03-28T01:07:04.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.869+0000 7fa5d25a8640 1 --2- 172.21.15.169:0/943494869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819fd690 0x55dd81a1a190 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fa5c000cc60 tx=0x7fa5c0006a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8871d21caf6da93d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:04.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.870+0000 7fa5b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:07:04.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.870+0000 7fa5b17fa640 1 -- 172.21.15.169:0/943494869 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0021020 con 0x55dd819fd690 2024-03-28T01:07:04.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.870+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/943494869 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5bc011420 con 0x55dd819fd690 2024-03-28T01:07:04.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.870+0000 7fa5b17fa640 1 -- 172.21.15.169:0/943494869 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa5c001d3f0 con 0x55dd819fd690 2024-03-28T01:07:04.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.875+0000 7fa5b17fa640 1 -- 172.21.15.169:0/943494869 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5c002c050 con 0x55dd819fd690 2024-03-28T01:07:04.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.875+0000 7fa5d6bf3740 10 client.17433 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:04.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.875+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/943494869 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55dd81799f70 con 0x55dd819fd690 2024-03-28T01:07:04.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.880+0000 7fa5b17fa640 1 -- 172.21.15.169:0/943494869 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5bc011420 con 0x55dd819fd690 2024-03-28T01:07:04.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.880+0000 7fa5b17fa640 1 -- 172.21.15.169:0/943494869 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa5c001c030 con 0x55dd819fd690 2024-03-28T01:07:04.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.880+0000 7fa5d6bf3740 10 client.17433 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:04.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.880+0000 7fa5d6bf3740 10 client.17433 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:07:04.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.880+0000 7fa5d6bf3740 20 client.17433 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:04.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.880+0000 7fa5d6bf3740 1 --2- 172.21.15.169:0/943494869 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55dd81a1ef30 0x55dd81a21390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:04.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.880+0000 7fa5d6bf3740 4 client.17433 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:07:04.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.880+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/943494869 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dd8196a5b0 con 0x55dd81a1ef30 2024-03-28T01:07:04.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.883+0000 7fa5d1da7640 1 --2- 172.21.15.169:0/943494869 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55dd81a1ef30 0x55dd81a21390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:04.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.895+0000 7fa5d1da7640 1 --2- 172.21.15.169:0/943494869 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55dd81a1ef30 0x55dd81a21390 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9b145eca40d012 server_cookie=2ad53045795433d3 in_seq=0 out_seq=0 2024-03-28T01:07:04.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.895+0000 7fa5b17fa640 10 client.17433 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:04.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.898+0000 7fa5b17fa640 1 -- 172.21.15.169:0/943494869 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55dd8196a5b0 con 0x55dd81a1ef30 2024-03-28T01:07:04.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.898+0000 7fa5b17fa640 10 client.17433 handle_command_reply: tid=0 2024-03-28T01:07:04.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.898+0000 7fa5d6bf3740 1 client.17433 shutdown 2024-03-28T01:07:04.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.907+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/943494869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819fd690 msgr2=0x55dd81a1a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:04.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.907+0000 7fa5d6bf3740 1 --2- 172.21.15.169:0/943494869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819fd690 0x55dd81a1a190 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fa5c000cc60 tx=0x7fa5c0006a20 comp rx=0 tx=0).stop 2024-03-28T01:07:04.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.908+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/943494869 shutdown_connections 2024-03-28T01:07:04.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.908+0000 7fa5d6bf3740 1 --2- 172.21.15.169:0/943494869 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55dd81a1ef30 0x55dd81a21390 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.908+0000 7fa5d6bf3740 1 --2- 172.21.15.169:0/943494869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819fd690 0x55dd81a1a190 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.908+0000 7fa5d6bf3740 1 --2- 172.21.15.169:0/943494869 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd819f8520 0x55dd81a13150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.908+0000 7fa5d6bf3740 1 --2- 172.21.15.169:0/943494869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd819f7b30 0x55dd81a12c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.908+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/943494869 >> 172.21.15.169:0/943494869 conn(0x55dd819c01a0 msgr2=0x55dd819f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:04.912 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.908+0000 7fa5b17fa640 0 client.17433 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:04.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.913+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/943494869 shutdown_connections 2024-03-28T01:07:04.924 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.919+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/943494869 wait complete. 2024-03-28T01:07:04.924 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.919+0000 7fa5d6bf3740 20 client.17433 trim_cache size 0 max 16384 2024-03-28T01:07:04.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.967+0000 7fa5d6bf3740 1 Processor -- start 2024-03-28T01:07:04.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.967+0000 7fa5d6bf3740 1 -- start start 2024-03-28T01:07:04.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.967+0000 7fa5d6bf3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd819f7b30 0x55dd819f2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:04.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.967+0000 7fa5d6bf3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd819f8520 0x55dd81a1a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:04.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.967+0000 7fa5d6bf3740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819fd690 0x55dd819f3500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:04.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.968+0000 7fa5d6bf3740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc1b2c80 con 0x55dd819fd690 2024-03-28T01:07:04.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.968+0000 7fa5d6bf3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc1b0780 con 0x55dd819f7b30 2024-03-28T01:07:04.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.968+0000 7fa5d6bf3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc1aa630 con 0x55dd819f8520 2024-03-28T01:07:04.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.968+0000 7fa5d25a8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819fd690 0x55dd819f3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:04.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.968+0000 7fa5d25a8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819fd690 0x55dd819f3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36044/0 (socket says 172.21.15.169:36044) 2024-03-28T01:07:04.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.968+0000 7fa5d25a8640 1 -- 172.21.15.169:0/3558888241 learned_addr learned my addr 172.21.15.169:0/3558888241 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:04.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.968+0000 7fa5d25a8640 1 -- 172.21.15.169:0/3558888241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd819f8520 msgr2=0x55dd81a1a150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:07:04.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.968+0000 7fa5d25a8640 1 --2- 172.21.15.169:0/3558888241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd819f8520 0x55dd81a1a150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.968+0000 7fa5d25a8640 1 -- 172.21.15.169:0/3558888241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd819f7b30 msgr2=0x55dd819f2fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:07:04.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.968+0000 7fa5d25a8640 1 --2- 172.21.15.169:0/3558888241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd819f7b30 0x55dd819f2fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.968+0000 7fa5d25a8640 1 -- 172.21.15.169:0/3558888241 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd81631170 con 0x55dd819fd690 2024-03-28T01:07:04.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.969+0000 7fa5d25a8640 1 --2- 172.21.15.169:0/3558888241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819fd690 0x55dd819f3500 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0015010 tx=0x7fa5c000c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4923b3186923738 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:04.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.971+0000 7fa5b37fe640 1 -- 172.21.15.169:0/3558888241 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0021040 con 0x55dd819fd690 2024-03-28T01:07:04.974 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.971+0000 7fa5b37fe640 1 -- 172.21.15.169:0/3558888241 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5c0011e70 con 0x55dd819fd690 2024-03-28T01:07:04.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.972+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/3558888241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819fd690 msgr2=0x55dd819f3500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:04.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.972+0000 7fa5d6bf3740 1 --2- 172.21.15.169:0/3558888241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819fd690 0x55dd819f3500 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0015010 tx=0x7fa5c000c320 comp rx=0 tx=0).stop 2024-03-28T01:07:04.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.978+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/3558888241 shutdown_connections 2024-03-28T01:07:04.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.978+0000 7fa5d6bf3740 1 --2- 172.21.15.169:0/3558888241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819fd690 0x55dd819f3500 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.978+0000 7fa5d6bf3740 1 --2- 172.21.15.169:0/3558888241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd819f8520 0x55dd81a1a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.978+0000 7fa5d6bf3740 1 --2- 172.21.15.169:0/3558888241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd819f7b30 0x55dd819f2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.978+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/3558888241 >> 172.21.15.169:0/3558888241 conn(0x55dd819c01a0 msgr2=0x55dd819f96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:04.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.986+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/3558888241 shutdown_connections 2024-03-28T01:07:04.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.989+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/3558888241 wait complete. 2024-03-28T01:07:04.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.989+0000 7fa5d6bf3740 1 -- start start 2024-03-28T01:07:04.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.989+0000 7fa5d6bf3740 1 Processor -- start 2024-03-28T01:07:04.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.993+0000 7fa5d6bf3740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819f7b30 0x55dd81a178b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:04.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.993+0000 7fa5d6bf3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd819f8520 0x55dd81a17df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:04.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.993+0000 7fa5d6bf3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd819fd690 0x55dd81a18e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:04.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.993+0000 7fa5d6bf3740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc1aac30 con 0x55dd819f7b30 2024-03-28T01:07:04.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.993+0000 7fa5d6bf3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc1b0120 con 0x55dd819fd690 2024-03-28T01:07:04.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.993+0000 7fa5d6bf3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc1af460 con 0x55dd819f8520 2024-03-28T01:07:04.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.993+0000 7fa5d6bf3740 10 client.0 _ll_register_callbacks cb 0x7fa5c000c910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:04.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.994+0000 7fa5d25a8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd819fd690 0x55dd81a18e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:04.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.994+0000 7fa5d25a8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd819fd690 0x55dd81a18e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33788/0 (socket says 172.21.15.169:33788) 2024-03-28T01:07:04.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.994+0000 7fa5d25a8640 1 -- 172.21.15.169:0/4240203836 learned_addr learned my addr 172.21.15.169:0/4240203836 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:04.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.994+0000 7fa5d25a8640 1 -- 172.21.15.169:0/4240203836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd819f8520 msgr2=0x55dd81a17df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:04.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.994+0000 7fa5d25a8640 1 --2- 172.21.15.169:0/4240203836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd819f8520 0x55dd81a17df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.994+0000 7fa5d25a8640 1 -- 172.21.15.169:0/4240203836 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819f7b30 msgr2=0x55dd81a178b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:04.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.994+0000 7fa5d25a8640 1 --2- 172.21.15.169:0/4240203836 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819f7b30 0x55dd81a178b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:04.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.994+0000 7fa5d25a8640 1 -- 172.21.15.169:0/4240203836 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dd81a193f0 con 0x55dd819fd690 2024-03-28T01:07:05.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:04.998+0000 7fa5d25a8640 1 --2- 172.21.15.169:0/4240203836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd819fd690 0x55dd81a18e00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa5c000e010 tx=0x7fa5c0011600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd7479385db5c35b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:05.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.006+0000 7fa5b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:07:05.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.007+0000 7fa5b17fa640 1 -- 172.21.15.169:0/4240203836 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0021050 con 0x55dd819fd690 2024-03-28T01:07:05.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.007+0000 7fa5b17fa640 1 -- 172.21.15.169:0/4240203836 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa5c0011ba0 con 0x55dd819fd690 2024-03-28T01:07:05.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.007+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/4240203836 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5c0011e70 con 0x55dd819fd690 2024-03-28T01:07:05.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.017+0000 7fa5b17fa640 1 -- 172.21.15.169:0/4240203836 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5c00039b0 con 0x55dd819fd690 2024-03-28T01:07:05.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.019+0000 7fa5d6bf3740 10 client.27320 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:05.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.019+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/4240203836 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55dd81a7a8c0 con 0x55dd819fd690 2024-03-28T01:07:05.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.027+0000 7fa5b17fa640 1 -- 172.21.15.169:0/4240203836 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5c0011e70 con 0x55dd819fd690 2024-03-28T01:07:05.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.027+0000 7fa5b17fa640 1 -- 172.21.15.169:0/4240203836 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa5c001c040 con 0x55dd819fd690 2024-03-28T01:07:05.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.028+0000 7fa5d6bf3740 10 client.27320 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:05.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.028+0000 7fa5d6bf3740 10 client.27320 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:07:05.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.028+0000 7fa5d6bf3740 20 client.27320 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:05.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.028+0000 7fa5d6bf3740 1 --2- 172.21.15.169:0/4240203836 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55dd81a7b100 0x55dd81a7d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:05.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.028+0000 7fa5d6bf3740 4 client.27320 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T01:07:05.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.028+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/4240203836 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55dd81964900 con 0x55dd81a7b100 2024-03-28T01:07:05.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.031+0000 7fa5d1da7640 1 --2- 172.21.15.169:0/4240203836 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55dd81a7b100 0x55dd81a7d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:05.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.036+0000 7fa5d1da7640 1 --2- 172.21.15.169:0/4240203836 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55dd81a7b100 0x55dd81a7d520 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fd7c8ae8012a670 server_cookie=dde82492efd53f9a in_seq=0 out_seq=0 2024-03-28T01:07:05.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.038+0000 7fa5b17fa640 10 client.27320 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:05.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.039+0000 7fa5b17fa640 1 -- 172.21.15.169:0/4240203836 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55dd81964900 con 0x55dd81a7b100 2024-03-28T01:07:05.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.039+0000 7fa5b17fa640 10 client.27320 handle_command_reply: tid=0 2024-03-28T01:07:05.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.040+0000 7fa5d6bf3740 1 client.27320 shutdown 2024-03-28T01:07:05.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.044+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/4240203836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd819fd690 msgr2=0x55dd81a18e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:05.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.044+0000 7fa5d6bf3740 1 --2- 172.21.15.169:0/4240203836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd819fd690 0x55dd81a18e00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa5c000e010 tx=0x7fa5c0011600 comp rx=0 tx=0).stop 2024-03-28T01:07:05.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.047+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/4240203836 shutdown_connections 2024-03-28T01:07:05.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.047+0000 7fa5d6bf3740 1 --2- 172.21.15.169:0/4240203836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd819fd690 0x55dd81a18e00 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.047+0000 7fa5d6bf3740 1 --2- 172.21.15.169:0/4240203836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd819f8520 0x55dd81a17df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.047+0000 7fa5d6bf3740 1 --2- 172.21.15.169:0/4240203836 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55dd81a7b100 0x55dd81a7d520 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.047+0000 7fa5d6bf3740 1 --2- 172.21.15.169:0/4240203836 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55dd819f7b30 0x55dd81a178b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.047+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/4240203836 >> 172.21.15.169:0/4240203836 conn(0x55dd819c01a0 msgr2=0x55dd819f2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:05.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.047+0000 7fa5b17fa640 0 client.27320 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:05.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.052+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/4240203836 shutdown_connections 2024-03-28T01:07:05.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.052+0000 7fa5d6bf3740 1 -- 172.21.15.169:0/4240203836 wait complete. 2024-03-28T01:07:05.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.053+0000 7fa5d6bf3740 20 client.27320 trim_cache size 0 max 16384 2024-03-28T01:07:05.056 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:07:05.056 INFO:teuthology.orchestra.run.smithi169.stdout: "return_code": 0, 2024-03-28T01:07:05.056 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_tag": "8168f164-0632-4993-822e-da1651bae568", 2024-03-28T01:07:05.056 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": "asynchronous" 2024-03-28T01:07:05.057 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:07:05.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.055+0000 7fa5d400b640 1 -- 172.21.15.169:0/790199969 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa5b40a2820 msgr2=0x7fa5b40a4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:05.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.055+0000 7fa5d400b640 1 --2- 172.21.15.169:0/790199969 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa5b40a2820 0x7fa5b40a4ce0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x55dd819f8d80 tx=0x7fa5c0008040 comp rx=0 tx=0).stop 2024-03-28T01:07:05.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.055+0000 7fa5d400b640 1 -- 172.21.15.169:0/790199969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5cc0ff330 msgr2=0x7fa5cc06f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:05.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.055+0000 7fa5d400b640 1 --2- 172.21.15.169:0/790199969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5cc0ff330 0x7fa5cc06f340 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4037040 tx=0x7fa5c4035e50 comp rx=0 tx=0).stop 2024-03-28T01:07:05.059 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.056+0000 7fa5d400b640 1 -- 172.21.15.169:0/790199969 shutdown_connections 2024-03-28T01:07:05.059 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.056+0000 7fa5d400b640 1 --2- 172.21.15.169:0/790199969 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa5b40a2820 0x7fa5b40a4ce0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.059 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.056+0000 7fa5d400b640 1 --2- 172.21.15.169:0/790199969 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5cc107220 0x7fa5cc0728d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.059 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.056+0000 7fa5d400b640 1 --2- 172.21.15.169:0/790199969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5cc0ffc90 0x7fa5cc06f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.056+0000 7fa5d400b640 1 --2- 172.21.15.169:0/790199969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5cc0ff330 0x7fa5cc06f340 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.060 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.056+0000 7fa5d400b640 1 -- 172.21.15.169:0/790199969 >> 172.21.15.169:0/790199969 conn(0x7fa5cc0fad60 msgr2=0x7fa5cc100a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:05.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.059+0000 7fa5d400b640 1 -- 172.21.15.169:0/790199969 shutdown_connections 2024-03-28T01:07:05.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.059+0000 7fa5d400b640 1 -- 172.21.15.169:0/790199969 wait complete. 2024-03-28T01:07:05.098 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:07:05.098 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:07:05.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.393+0000 7fb9dc79e640 1 -- 172.21.15.169:0/3181243247 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9d4076430 msgr2=0x7fb9d40788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:05.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.393+0000 7fb9dc79e640 1 --2- 172.21.15.169:0/3181243247 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9d4076430 0x7fb9d40788a0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4009900 tx=0x7fb9c402bc80 comp rx=0 tx=0).stop 2024-03-28T01:07:05.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.401+0000 7fb9dc79e640 1 -- 172.21.15.169:0/3181243247 shutdown_connections 2024-03-28T01:07:05.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.401+0000 7fb9dc79e640 1 --2- 172.21.15.169:0/3181243247 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9d4078de0 0x7fb9d407b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.401+0000 7fb9dc79e640 1 --2- 172.21.15.169:0/3181243247 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9d4076430 0x7fb9d40788a0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.401+0000 7fb9dc79e640 1 --2- 172.21.15.169:0/3181243247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9d4073550 0x7fb9d4073970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.401+0000 7fb9dc79e640 1 -- 172.21.15.169:0/3181243247 >> 172.21.15.169:0/3181243247 conn(0x7fb9d406e500 msgr2=0x7fb9d4070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:05.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.408+0000 7fb9dc79e640 1 -- 172.21.15.169:0/3181243247 shutdown_connections 2024-03-28T01:07:05.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.413+0000 7fb9dc79e640 1 -- 172.21.15.169:0/3181243247 wait complete. 2024-03-28T01:07:05.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.414+0000 7fb9dc79e640 1 Processor -- start 2024-03-28T01:07:05.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.415+0000 7fb9dc79e640 1 -- start start 2024-03-28T01:07:05.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.416+0000 7fb9dc79e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9d4073550 0x7fb9d41b39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:05.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.416+0000 7fb9dc79e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9d4076430 0x7fb9d41b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:05.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.416+0000 7fb9dc79e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9d4078de0 0x7fb9d41ba9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:05.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.416+0000 7fb9dc79e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d407e450 con 0x7fb9d4078de0 2024-03-28T01:07:05.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.416+0000 7fb9dc79e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d407e2d0 con 0x7fb9d4076430 2024-03-28T01:07:05.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.416+0000 7fb9dc79e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d407e5d0 con 0x7fb9d4073550 2024-03-28T01:07:05.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.417+0000 7fb9dad3b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9d4078de0 0x7fb9d41ba9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:05.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.417+0000 7fb9dad3b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9d4078de0 0x7fb9d41ba9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36060/0 (socket says 172.21.15.169:36060) 2024-03-28T01:07:05.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.417+0000 7fb9dad3b640 1 -- 172.21.15.169:0/1378700355 learned_addr learned my addr 172.21.15.169:0/1378700355 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:05.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.417+0000 7fb9d9d39640 1 --2- 172.21.15.169:0/1378700355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9d4076430 0x7fb9d41b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:05.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.418+0000 7fb9d9d39640 1 -- 172.21.15.169:0/1378700355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9d4073550 msgr2=0x7fb9d41b39f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:07:05.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.418+0000 7fb9d9d39640 1 --2- 172.21.15.169:0/1378700355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9d4073550 0x7fb9d41b39f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.418+0000 7fb9d9d39640 1 -- 172.21.15.169:0/1378700355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9d4078de0 msgr2=0x7fb9d41ba9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:05.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.418+0000 7fb9d9d39640 1 --2- 172.21.15.169:0/1378700355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9d4078de0 0x7fb9d41ba9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.418+0000 7fb9d9d39640 1 -- 172.21.15.169:0/1378700355 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d41bb080 con 0x7fb9d4076430 2024-03-28T01:07:05.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.419+0000 7fb9d9d39640 1 --2- 172.21.15.169:0/1378700355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9d4076430 0x7fb9d41b3f30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4009850 tx=0x7fb9c40045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:05.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.419+0000 7fb9cb7fe640 1 -- 172.21.15.169:0/1378700355 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4042020 con 0x7fb9d4076430 2024-03-28T01:07:05.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.419+0000 7fb9dc79e640 1 -- 172.21.15.169:0/1378700355 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9d41bb310 con 0x7fb9d4076430 2024-03-28T01:07:05.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.419+0000 7fb9dc79e640 1 -- 172.21.15.169:0/1378700355 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9d41bb820 con 0x7fb9d4076430 2024-03-28T01:07:05.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.420+0000 7fb9cb7fe640 1 -- 172.21.15.169:0/1378700355 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb9c40035f0 con 0x7fb9d4076430 2024-03-28T01:07:05.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.420+0000 7fb9cb7fe640 1 -- 172.21.15.169:0/1378700355 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c403e6c0 con 0x7fb9d4076430 2024-03-28T01:07:05.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.426+0000 7fb9cb7fe640 1 -- 172.21.15.169:0/1378700355 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb9c4045020 con 0x7fb9d4076430 2024-03-28T01:07:05.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.426+0000 7fb9cb7fe640 1 --2- 172.21.15.169:0/1378700355 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb9c00a2970 0x7fb9c00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:05.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.427+0000 7fb9cb7fe640 1 -- 172.21.15.169:0/1378700355 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb9c40f1fa0 con 0x7fb9d4076430 2024-03-28T01:07:05.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.427+0000 7fb9df389740 1 Processor -- start 2024-03-28T01:07:05.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.430+0000 7fb9df389740 1 -- start start 2024-03-28T01:07:05.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.430+0000 7fb9df389740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0122b65b0 0x55e0122b69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:05.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.430+0000 7fb9df389740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b6fa0 0x55e0122bbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:05.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.430+0000 7fb9df389740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122bc110 0x55e0122be5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:05.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.430+0000 7fb9df389740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0122133d0 con 0x55e0122b65b0 2024-03-28T01:07:05.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.430+0000 7fb9df389740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0121bbad0 con 0x55e0122bc110 2024-03-28T01:07:05.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.430+0000 7fb9df389740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e01220f470 con 0x55e0122b6fa0 2024-03-28T01:07:05.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.432+0000 7fb9d9d39640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b6fa0 0x55e0122bbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:05.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.433+0000 7fb9da53a640 1 --2- 172.21.15.169:0/1378700355 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb9c00a2970 0x7fb9c00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:05.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.433+0000 7fb9dad3b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122bc110 0x55e0122be5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:05.437 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.433+0000 7fb9d9d39640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b6fa0 0x55e0122bbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56346/0 (socket says 172.21.15.169:56346) 2024-03-28T01:07:05.438 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.433+0000 7fb9d9d39640 1 -- 172.21.15.169:0/2640396124 learned_addr learned my addr 172.21.15.169:0/2640396124 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:05.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.436+0000 7fb9da53a640 1 --2- 172.21.15.169:0/2640396124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0122b65b0 0x55e0122b69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:05.440 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.438+0000 7fb9d9d39640 1 -- 172.21.15.169:0/2640396124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122bc110 msgr2=0x55e0122be5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:05.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.438+0000 7fb9d9d39640 1 --2- 172.21.15.169:0/2640396124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122bc110 0x55e0122be5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.438+0000 7fb9d9d39640 1 -- 172.21.15.169:0/2640396124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0122b65b0 msgr2=0x55e0122b69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:05.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.438+0000 7fb9d9d39640 1 --2- 172.21.15.169:0/2640396124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0122b65b0 0x55e0122b69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.441 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.438+0000 7fb9d9d39640 1 -- 172.21.15.169:0/2640396124 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e011ff17d0 con 0x55e0122b6fa0 2024-03-28T01:07:05.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.439+0000 7fb9da53a640 1 --2- 172.21.15.169:0/1378700355 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb9c00a2970 0x7fb9c00a4e30 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc005fd0 tx=0x7fb9cc007b60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:05.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.443+0000 7fb9d9d39640 1 --2- 172.21.15.169:0/2640396124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b6fa0 0x55e0122bbaa0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb9c402c290 tx=0x7fb9c402c720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59d07a2e24ca7fab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:05.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.446+0000 7fb9c8ff9640 1 -- 172.21.15.169:0/2640396124 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c402ccd0 con 0x55e0122b6fa0 2024-03-28T01:07:05.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.446+0000 7fb9c8ff9640 1 -- 172.21.15.169:0/2640396124 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb9c40bec40 con 0x55e0122b6fa0 2024-03-28T01:07:05.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.448+0000 7fb9df389740 1 -- 172.21.15.169:0/2640396124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b6fa0 msgr2=0x55e0122bbaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:05.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.448+0000 7fb9df389740 1 --2- 172.21.15.169:0/2640396124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b6fa0 0x55e0122bbaa0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb9c402c290 tx=0x7fb9c402c720 comp rx=0 tx=0).stop 2024-03-28T01:07:05.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.449+0000 7fb9df389740 1 -- 172.21.15.169:0/2640396124 shutdown_connections 2024-03-28T01:07:05.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.449+0000 7fb9df389740 1 --2- 172.21.15.169:0/2640396124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122bc110 0x55e0122be5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.449+0000 7fb9df389740 1 --2- 172.21.15.169:0/2640396124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b6fa0 0x55e0122bbaa0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.449+0000 7fb9df389740 1 --2- 172.21.15.169:0/2640396124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0122b65b0 0x55e0122b69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.449+0000 7fb9df389740 1 -- 172.21.15.169:0/2640396124 >> 172.21.15.169:0/2640396124 conn(0x55e0122a9310 msgr2=0x55e0122a9710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:05.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.456+0000 7fb9df389740 1 -- 172.21.15.169:0/2640396124 shutdown_connections 2024-03-28T01:07:05.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.459+0000 7fb9df389740 1 -- 172.21.15.169:0/2640396124 wait complete. 2024-03-28T01:07:05.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.459+0000 7fb9df389740 1 -- start start 2024-03-28T01:07:05.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.459+0000 7fb9df389740 1 Processor -- start 2024-03-28T01:07:05.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.461+0000 7fb9df389740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b65b0 0x55e0122af560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:05.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.461+0000 7fb9df389740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122b6fa0 0x55e0122afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:05.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.461+0000 7fb9df389740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0122bc110 0x55e0122b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:05.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.461+0000 7fb9df389740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0122adc60 con 0x55e0122bc110 2024-03-28T01:07:05.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.461+0000 7fb9df389740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0121cf060 con 0x55e0122b6fa0 2024-03-28T01:07:05.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.461+0000 7fb9df389740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0122adde0 con 0x55e0122b65b0 2024-03-28T01:07:05.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.461+0000 7fb9df389740 10 client.0 _ll_register_callbacks cb 0x55e0121b16a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:05.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.462+0000 7fb9dad3b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0122bc110 0x55e0122b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:05.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.462+0000 7fb9dad3b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0122bc110 0x55e0122b4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36076/0 (socket says 172.21.15.169:36076) 2024-03-28T01:07:05.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.462+0000 7fb9dad3b640 1 -- 172.21.15.169:0/3047413298 learned_addr learned my addr 172.21.15.169:0/3047413298 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:05.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.463+0000 7fb9dad3b640 1 -- 172.21.15.169:0/3047413298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b65b0 msgr2=0x55e0122af560 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:07:05.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.463+0000 7fb9dad3b640 1 --2- 172.21.15.169:0/3047413298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b65b0 0x55e0122af560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.463+0000 7fb9dad3b640 1 -- 172.21.15.169:0/3047413298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122b6fa0 msgr2=0x55e0122afaa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:07:05.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.463+0000 7fb9dad3b640 1 --2- 172.21.15.169:0/3047413298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122b6fa0 0x55e0122afaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.463+0000 7fb9dad3b640 1 -- 172.21.15.169:0/3047413298 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e011f273f0 con 0x55e0122bc110 2024-03-28T01:07:05.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.464+0000 7fb9dad3b640 1 --2- 172.21.15.169:0/3047413298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0122bc110 0x55e0122b4ad0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb9d00136e0 tx=0x7fb9d0013ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b423f10a76fd24 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:05.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.464+0000 7fb9bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:07:05.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.464+0000 7fb9bdffb640 1 -- 172.21.15.169:0/3047413298 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0011a40 con 0x55e0122bc110 2024-03-28T01:07:05.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.465+0000 7fb9df389740 1 -- 172.21.15.169:0/3047413298 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb9c40bec40 con 0x55e0122bc110 2024-03-28T01:07:05.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.465+0000 7fb9bdffb640 1 -- 172.21.15.169:0/3047413298 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb9d0011be0 con 0x55e0122bc110 2024-03-28T01:07:05.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.465+0000 7fb9bdffb640 1 -- 172.21.15.169:0/3047413298 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb9d0028050 con 0x55e0122bc110 2024-03-28T01:07:05.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.466+0000 7fb9df389740 10 client.17448 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:05.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.466+0000 7fb9df389740 1 -- 172.21.15.169:0/3047413298 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55e011ee70f0 con 0x55e0122bc110 2024-03-28T01:07:05.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.467+0000 7fb9bdffb640 1 -- 172.21.15.169:0/3047413298 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb9c40bec40 con 0x55e0122bc110 2024-03-28T01:07:05.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.469+0000 7fb9bdffb640 1 -- 172.21.15.169:0/3047413298 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb9d001d450 con 0x55e0122bc110 2024-03-28T01:07:05.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.472+0000 7fb9df389740 10 client.17448 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:05.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.472+0000 7fb9df389740 10 client.17448 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:07:05.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.472+0000 7fb9df389740 20 client.17448 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:05.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.472+0000 7fb9df389740 1 --2- 172.21.15.169:0/3047413298 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e0122d4b10 0x55e0122d6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:05.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.473+0000 7fb9df389740 4 client.17448 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:07:05.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.473+0000 7fb9df389740 1 -- 172.21.15.169:0/3047413298 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e012220910 con 0x55e0122d4b10 2024-03-28T01:07:05.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.474+0000 7fb9da53a640 1 --2- 172.21.15.169:0/3047413298 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e0122d4b10 0x55e0122d6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:05.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.477+0000 7fb9da53a640 1 --2- 172.21.15.169:0/3047413298 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e0122d4b10 0x55e0122d6fa0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=365358fadaa0e494 server_cookie=29ba04b3402da2d7 in_seq=0 out_seq=0 2024-03-28T01:07:05.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.478+0000 7fb9bdffb640 10 client.17448 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:05.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.481+0000 7fb9bdffb640 1 -- 172.21.15.169:0/3047413298 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e012220910 con 0x55e0122d4b10 2024-03-28T01:07:05.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.481+0000 7fb9bdffb640 10 client.17448 handle_command_reply: tid=0 2024-03-28T01:07:05.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.481+0000 7fb9df389740 1 client.17448 shutdown 2024-03-28T01:07:05.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.482+0000 7fb9df389740 1 -- 172.21.15.169:0/3047413298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0122bc110 msgr2=0x55e0122b4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:05.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.482+0000 7fb9df389740 1 --2- 172.21.15.169:0/3047413298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0122bc110 0x55e0122b4ad0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb9d00136e0 tx=0x7fb9d0013ae0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.485+0000 7fb9df389740 1 -- 172.21.15.169:0/3047413298 shutdown_connections 2024-03-28T01:07:05.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.485+0000 7fb9df389740 1 --2- 172.21.15.169:0/3047413298 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e0122d4b10 0x55e0122d6fa0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.485+0000 7fb9df389740 1 --2- 172.21.15.169:0/3047413298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0122bc110 0x55e0122b4ad0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.485+0000 7fb9df389740 1 --2- 172.21.15.169:0/3047413298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122b6fa0 0x55e0122afaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.485+0000 7fb9df389740 1 --2- 172.21.15.169:0/3047413298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b65b0 0x55e0122af560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.485+0000 7fb9df389740 1 -- 172.21.15.169:0/3047413298 >> 172.21.15.169:0/3047413298 conn(0x55e0122a9310 msgr2=0x55e0122a8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:05.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.485+0000 7fb9bdffb640 0 client.17448 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:05.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.487+0000 7fb9df389740 1 -- 172.21.15.169:0/3047413298 shutdown_connections 2024-03-28T01:07:05.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.487+0000 7fb9df389740 1 -- 172.21.15.169:0/3047413298 wait complete. 2024-03-28T01:07:05.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.488+0000 7fb9df389740 20 client.17448 trim_cache size 0 max 16384 2024-03-28T01:07:05.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.541+0000 7fb9df389740 1 Processor -- start 2024-03-28T01:07:05.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.541+0000 7fb9df389740 1 -- start start 2024-03-28T01:07:05.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.542+0000 7fb9df389740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122b65b0 0x55e0122ae850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:05.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.542+0000 7fb9df389740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b6fa0 0x55e0122aed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:05.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.542+0000 7fb9df389740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0122bc110 0x55e0122a7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:05.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.543+0000 7fb9df389740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b8011f60 con 0x55e0122bc110 2024-03-28T01:07:05.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.543+0000 7fb9da53a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122b65b0 0x55e0122ae850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:05.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.543+0000 7fb9da53a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122b65b0 0x55e0122ae850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33840/0 (socket says 172.21.15.169:33840) 2024-03-28T01:07:05.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.543+0000 7fb9da53a640 1 -- 172.21.15.169:0/3928724582 learned_addr learned my addr 172.21.15.169:0/3928724582 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:05.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.543+0000 7fb9d9d39640 1 --2- 172.21.15.169:0/3928724582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b6fa0 0x55e0122aed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:05.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.543+0000 7fb9df389740 1 -- 172.21.15.169:0/3928724582 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b800fa60 con 0x55e0122b65b0 2024-03-28T01:07:05.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.543+0000 7fb9df389740 1 -- 172.21.15.169:0/3928724582 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b800dbb0 con 0x55e0122b6fa0 2024-03-28T01:07:05.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.544+0000 7fb9dad3b640 1 --2- 172.21.15.169:0/3928724582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0122bc110 0x55e0122a7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:05.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.544+0000 7fb9da53a640 1 -- 172.21.15.169:0/3928724582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b6fa0 msgr2=0x55e0122aed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:05.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.544+0000 7fb9da53a640 1 --2- 172.21.15.169:0/3928724582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b6fa0 0x55e0122aed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.544+0000 7fb9da53a640 1 -- 172.21.15.169:0/3928724582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0122bc110 msgr2=0x55e0122a7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:05.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.544+0000 7fb9da53a640 1 --2- 172.21.15.169:0/3928724582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0122bc110 0x55e0122a7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.544+0000 7fb9da53a640 1 -- 172.21.15.169:0/3928724582 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e0122b7db0 con 0x55e0122b65b0 2024-03-28T01:07:05.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.545+0000 7fb9da53a640 1 --2- 172.21.15.169:0/3928724582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122b65b0 0x55e0122ae850 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc005c40 tx=0x7fb9cc016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cb141197f8d3cf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:05.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.546+0000 7fb9bffff640 1 -- 172.21.15.169:0/3928724582 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc003b70 con 0x55e0122b65b0 2024-03-28T01:07:05.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.546+0000 7fb9bffff640 1 -- 172.21.15.169:0/3928724582 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb9cc003d10 con 0x55e0122b65b0 2024-03-28T01:07:05.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.546+0000 7fb9bffff640 1 -- 172.21.15.169:0/3928724582 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc01f8b0 con 0x55e0122b65b0 2024-03-28T01:07:05.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.547+0000 7fb9df389740 1 -- 172.21.15.169:0/3928724582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122b65b0 msgr2=0x55e0122ae850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:05.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.547+0000 7fb9df389740 1 --2- 172.21.15.169:0/3928724582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122b65b0 0x55e0122ae850 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc005c40 tx=0x7fb9cc016000 comp rx=0 tx=0).stop 2024-03-28T01:07:05.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.553+0000 7fb9df389740 1 -- 172.21.15.169:0/3928724582 shutdown_connections 2024-03-28T01:07:05.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.553+0000 7fb9df389740 1 --2- 172.21.15.169:0/3928724582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0122bc110 0x55e0122a7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.553+0000 7fb9df389740 1 --2- 172.21.15.169:0/3928724582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b6fa0 0x55e0122aed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.553+0000 7fb9df389740 1 --2- 172.21.15.169:0/3928724582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122b65b0 0x55e0122ae850 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.553+0000 7fb9df389740 1 -- 172.21.15.169:0/3928724582 >> 172.21.15.169:0/3928724582 conn(0x55e0122a9310 msgr2=0x55e0122a96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:05.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.565+0000 7fb9df389740 1 -- 172.21.15.169:0/3928724582 shutdown_connections 2024-03-28T01:07:05.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.565+0000 7fb9df389740 1 -- 172.21.15.169:0/3928724582 wait complete. 2024-03-28T01:07:05.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.565+0000 7fb9df389740 1 -- start start 2024-03-28T01:07:05.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.565+0000 7fb9df389740 1 Processor -- start 2024-03-28T01:07:05.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.566+0000 7fb9df389740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b65b0 0x55e0122e59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:05.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.566+0000 7fb9df389740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122b6fa0 0x55e0122e5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:05.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.566+0000 7fb9da53a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b65b0 0x55e0122e59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:05.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.566+0000 7fb9da53a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b65b0 0x55e0122e59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:56386/0 (socket says 172.21.15.169:56386) 2024-03-28T01:07:05.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.566+0000 7fb9da53a640 1 -- 172.21.15.169:0/1090861747 learned_addr learned my addr 172.21.15.169:0/1090861747 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:05.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.567+0000 7fb9d9d39640 1 --2- 172.21.15.169:0/1090861747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122b6fa0 0x55e0122e5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:05.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.567+0000 7fb9df389740 1 --2- 172.21.15.169:0/1090861747 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0122bc110 0x55e0122e6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:05.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.567+0000 7fb9df389740 1 -- 172.21.15.169:0/1090861747 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b800e120 con 0x55e0122bc110 2024-03-28T01:07:05.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.567+0000 7fb9df389740 1 -- 172.21.15.169:0/1090861747 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b800f400 con 0x55e0122b6fa0 2024-03-28T01:07:05.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.567+0000 7fb9df389740 1 -- 172.21.15.169:0/1090861747 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b800e740 con 0x55e0122b65b0 2024-03-28T01:07:05.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.567+0000 7fb9df389740 10 client.0 _ll_register_callbacks cb 0x7fb9d00096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:05.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.567+0000 7fb9da53a640 1 -- 172.21.15.169:0/1090861747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122b6fa0 msgr2=0x55e0122e5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:05.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.567+0000 7fb9da53a640 1 --2- 172.21.15.169:0/1090861747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122b6fa0 0x55e0122e5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.568+0000 7fb9da53a640 1 -- 172.21.15.169:0/1090861747 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0122bc110 msgr2=0x55e0122e6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:05.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.568+0000 7fb9da53a640 1 --2- 172.21.15.169:0/1090861747 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0122bc110 0x55e0122e6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.568+0000 7fb9da53a640 1 -- 172.21.15.169:0/1090861747 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e01232f990 con 0x55e0122b65b0 2024-03-28T01:07:05.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.568+0000 7fb9d9d39640 1 --2- 172.21.15.169:0/1090861747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122b6fa0 0x55e0122e5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:07:05.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.568+0000 7fb9df389740 1 -- 172.21.15.169:0/1090861747 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb9cc003d10 con 0x55e0122b65b0 2024-03-28T01:07:05.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.569+0000 7fb9da53a640 1 --2- 172.21.15.169:0/1090861747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b65b0 0x55e0122e59b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc005f40 tx=0x7fb9cc003b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbce8b641b0bd873 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:05.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.569+0000 7fb9bdffb640 10 client.27262 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:07:05.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.569+0000 7fb9bdffb640 1 -- 172.21.15.169:0/1090861747 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc01f840 con 0x55e0122b65b0 2024-03-28T01:07:05.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.569+0000 7fb9bdffb640 1 -- 172.21.15.169:0/1090861747 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb9cc01f9e0 con 0x55e0122b65b0 2024-03-28T01:07:05.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.570+0000 7fb9bdffb640 1 -- 172.21.15.169:0/1090861747 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb9cc01fc20 con 0x55e0122b65b0 2024-03-28T01:07:05.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.574+0000 7fb9df389740 10 client.27262 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:05.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.574+0000 7fb9df389740 1 -- 172.21.15.169:0/1090861747 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x55e012330bf0 con 0x55e0122b65b0 2024-03-28T01:07:05.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.577+0000 7fb9bdffb640 1 -- 172.21.15.169:0/1090861747 <== mon.1 v2:172.21.15.177:3300/0 4 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb9cc01d040 con 0x55e0122b65b0 2024-03-28T01:07:05.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.577+0000 7fb9df389740 10 client.27262 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:05.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.577+0000 7fb9df389740 10 client.27262 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:07:05.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.577+0000 7fb9df389740 20 client.27262 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:05.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.577+0000 7fb9df389740 1 --2- 172.21.15.169:0/1090861747 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e0123314d0 0x55e0123338f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:05.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.577+0000 7fb9df389740 4 client.27262 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:07:05.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.577+0000 7fb9df389740 1 -- 172.21.15.169:0/1090861747 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e01221ac60 con 0x55e0123314d0 2024-03-28T01:07:05.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.624+0000 7fb9d9d39640 1 --2- 172.21.15.169:0/1090861747 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e0123314d0 0x55e0123338f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:05.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.626+0000 7fb9d9d39640 1 --2- 172.21.15.169:0/1090861747 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e0123314d0 0x55e0123338f0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0b173ba81877b4 server_cookie=960a5062261fc7c6 in_seq=0 out_seq=0 2024-03-28T01:07:05.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.626+0000 7fb9bdffb640 10 client.27262 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:05.640 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.637+0000 7fb9bdffb640 1 -- 172.21.15.169:0/1090861747 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x55e01221ac60 con 0x55e0123314d0 2024-03-28T01:07:05.640 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.637+0000 7fb9bdffb640 10 client.27262 handle_command_reply: tid=0 2024-03-28T01:07:05.640 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.637+0000 7fb9df389740 1 client.27262 shutdown 2024-03-28T01:07:05.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.637+0000 7fb9df389740 1 -- 172.21.15.169:0/1090861747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b65b0 msgr2=0x55e0122e59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:05.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.637+0000 7fb9df389740 1 --2- 172.21.15.169:0/1090861747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b65b0 0x55e0122e59b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc005f40 tx=0x7fb9cc003b80 comp rx=0 tx=0).stop 2024-03-28T01:07:05.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.637+0000 7fb9df389740 1 -- 172.21.15.169:0/1090861747 shutdown_connections 2024-03-28T01:07:05.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.637+0000 7fb9df389740 1 --2- 172.21.15.169:0/1090861747 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55e0123314d0 0x55e0123338f0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.637+0000 7fb9df389740 1 --2- 172.21.15.169:0/1090861747 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55e0122bc110 0x55e0122e6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.637+0000 7fb9df389740 1 --2- 172.21.15.169:0/1090861747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0122b6fa0 0x55e0122e5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.637+0000 7fb9df389740 1 --2- 172.21.15.169:0/1090861747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0122b65b0 0x55e0122e59b0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.637+0000 7fb9bdffb640 0 client.27262 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:05.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.637+0000 7fb9df389740 1 -- 172.21.15.169:0/1090861747 >> 172.21.15.169:0/1090861747 conn(0x55e0122a9310 msgr2=0x55e0122ae850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:05.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.638+0000 7fb9df389740 1 -- 172.21.15.169:0/1090861747 shutdown_connections 2024-03-28T01:07:05.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.638+0000 7fb9df389740 1 -- 172.21.15.169:0/1090861747 wait complete. 2024-03-28T01:07:05.642 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.639+0000 7fb9df389740 20 client.27262 trim_cache size 0 max 16384 2024-03-28T01:07:05.642 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:07:05.642 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "scrub active (0 inodes in the stack)", 2024-03-28T01:07:05.643 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": { 2024-03-28T01:07:05.643 INFO:teuthology.orchestra.run.smithi169.stdout: "8168f164-0632-4993-822e-da1651bae568": { 2024-03-28T01:07:05.643 INFO:teuthology.orchestra.run.smithi169.stdout: "path": "/", 2024-03-28T01:07:05.643 INFO:teuthology.orchestra.run.smithi169.stdout: "tag": "8168f164-0632-4993-822e-da1651bae568", 2024-03-28T01:07:05.643 INFO:teuthology.orchestra.run.smithi169.stdout: "options": "recursive,force" 2024-03-28T01:07:05.643 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:07:05.643 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:07:05.643 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:07:05.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.642+0000 7fb9dc79e640 1 -- 172.21.15.169:0/1378700355 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb9c00a2970 msgr2=0x7fb9c00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:05.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.642+0000 7fb9dc79e640 1 --2- 172.21.15.169:0/1378700355 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb9c00a2970 0x7fb9c00a4e30 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc005fd0 tx=0x7fb9cc007b60 comp rx=0 tx=0).stop 2024-03-28T01:07:05.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.642+0000 7fb9dc79e640 1 -- 172.21.15.169:0/1378700355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9d4076430 msgr2=0x7fb9d41b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:05.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.642+0000 7fb9dc79e640 1 --2- 172.21.15.169:0/1378700355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9d4076430 0x7fb9d41b3f30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4009850 tx=0x7fb9c40045e0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.642+0000 7fb9dc79e640 1 -- 172.21.15.169:0/1378700355 shutdown_connections 2024-03-28T01:07:05.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.642+0000 7fb9dc79e640 1 --2- 172.21.15.169:0/1378700355 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb9c00a2970 0x7fb9c00a4e30 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.642+0000 7fb9dc79e640 1 --2- 172.21.15.169:0/1378700355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9d4078de0 0x7fb9d41ba9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.642+0000 7fb9dc79e640 1 --2- 172.21.15.169:0/1378700355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9d4076430 0x7fb9d41b3f30 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.642+0000 7fb9dc79e640 1 --2- 172.21.15.169:0/1378700355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9d4073550 0x7fb9d41b39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:05.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.642+0000 7fb9dc79e640 1 -- 172.21.15.169:0/1378700355 >> 172.21.15.169:0/1378700355 conn(0x7fb9d406e500 msgr2=0x7fb9d4079c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:05.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.643+0000 7fb9dc79e640 1 -- 172.21.15.169:0/1378700355 shutdown_connections 2024-03-28T01:07:05.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:05.643+0000 7fb9dc79e640 1 -- 172.21.15.169:0/1378700355 wait complete. 2024-03-28T01:07:05.674 INFO:tasks.cephfs.filesystem:scrub status for tag:8168f164-0632-4993-822e-da1651bae568 - {'path': '/', 'tag': '8168f164-0632-4993-822e-da1651bae568', 'options': 'recursive,force'} 2024-03-28T01:07:06.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:05 smithi169 ceph-mon[32274]: pgmap v668: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 559 MiB/s wr, 1.14k op/s 2024-03-28T01:07:06.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:05 smithi169 ceph-mon[32274]: scrub queued for path: / 2024-03-28T01:07:06.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:05 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:07:06.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:05 smithi169 ceph-mon[32274]: scrub summary: active paths [/] 2024-03-28T01:07:06.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:05 smithi174 ceph-mon[36060]: pgmap v668: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 559 MiB/s wr, 1.14k op/s 2024-03-28T01:07:06.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:05 smithi174 ceph-mon[36060]: scrub queued for path: / 2024-03-28T01:07:06.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:05 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:07:06.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:05 smithi174 ceph-mon[36060]: scrub summary: active paths [/] 2024-03-28T01:07:06.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:05 smithi177 ceph-mon[36206]: pgmap v668: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 559 MiB/s wr, 1.14k op/s 2024-03-28T01:07:06.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:05 smithi177 ceph-mon[36206]: scrub queued for path: / 2024-03-28T01:07:06.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:05 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:07:06.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:05 smithi177 ceph-mon[36206]: scrub summary: active paths [/] 2024-03-28T01:07:08.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:07 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:07:08.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:07 smithi169 ceph-mon[32274]: pgmap v669: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 633 MiB/s wr, 1.40k op/s 2024-03-28T01:07:08.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:07 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:07:08.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:07 smithi174 ceph-mon[36060]: pgmap v669: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 633 MiB/s wr, 1.40k op/s 2024-03-28T01:07:08.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:07 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:07:08.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:07 smithi177 ceph-mon[36206]: pgmap v669: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 633 MiB/s wr, 1.40k op/s 2024-03-28T01:07:10.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:09 smithi169 ceph-mon[32274]: pgmap v670: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 609 MiB/s wr, 1.32k op/s 2024-03-28T01:07:10.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:09 smithi169 ceph-mon[32274]: scrub summary: idle 2024-03-28T01:07:10.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:09 smithi174 ceph-mon[36060]: pgmap v670: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 609 MiB/s wr, 1.32k op/s 2024-03-28T01:07:10.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:09 smithi174 ceph-mon[36060]: scrub summary: idle 2024-03-28T01:07:10.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:09 smithi177 ceph-mon[36206]: pgmap v670: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 609 MiB/s wr, 1.32k op/s 2024-03-28T01:07:10.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:09 smithi177 ceph-mon[36206]: scrub summary: idle 2024-03-28T01:07:12.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:11 smithi169 ceph-mon[32274]: pgmap v671: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 556 MiB/s wr, 1.19k op/s 2024-03-28T01:07:12.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:11 smithi174 ceph-mon[36060]: pgmap v671: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 556 MiB/s wr, 1.19k op/s 2024-03-28T01:07:12.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:11 smithi177 ceph-mon[36206]: pgmap v671: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 556 MiB/s wr, 1.19k op/s 2024-03-28T01:07:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.559+0000 7f33a3fff640 1 -- 172.21.15.169:0/4174726269 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33a410dbb0 msgr2=0x7f33a4110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.559+0000 7f33a3fff640 1 --2- 172.21.15.169:0/4174726269 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33a410dbb0 0x7f33a4110070 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f339c005fd0 tx=0x7f339c02cc70 comp rx=0 tx=0).stop 2024-03-28T01:07:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.560+0000 7f33a3fff640 1 -- 172.21.15.169:0/4174726269 shutdown_connections 2024-03-28T01:07:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.560+0000 7f33a3fff640 1 --2- 172.21.15.169:0/4174726269 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33a410dbb0 0x7f33a4110070 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.560+0000 7f33a3fff640 1 --2- 172.21.15.169:0/4174726269 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33a410b0f0 0x7f33a410d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.560+0000 7f33a3fff640 1 --2- 172.21.15.169:0/4174726269 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33a4108730 0x7f33a410ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.560+0000 7f33a3fff640 1 -- 172.21.15.169:0/4174726269 >> 172.21.15.169:0/4174726269 conn(0x7f33a406e510 msgr2=0x7f33a4070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.561+0000 7f33a3fff640 1 -- 172.21.15.169:0/4174726269 shutdown_connections 2024-03-28T01:07:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.561+0000 7f33a3fff640 1 -- 172.21.15.169:0/4174726269 wait complete. 2024-03-28T01:07:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.562+0000 7f33a3fff640 1 Processor -- start 2024-03-28T01:07:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.563+0000 7f33a3fff640 1 -- start start 2024-03-28T01:07:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.563+0000 7f33a3fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33a4108730 0x7f33a41a6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.563+0000 7f33a3fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33a410b0f0 0x7f33a41a6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.563+0000 7f33a3fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33a410dbb0 0x7f33a41adbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.563+0000 7f33a3fff640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a41132c0 con 0x7f33a4108730 2024-03-28T01:07:13.568 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.563+0000 7f33a3fff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a4113140 con 0x7f33a410dbb0 2024-03-28T01:07:13.568 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.563+0000 7f33a3fff640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a4113440 con 0x7f33a410b0f0 2024-03-28T01:07:13.568 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.563+0000 7f33a27fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33a410b0f0 0x7f33a41a6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:13.568 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.563+0000 7f33a27fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33a410b0f0 0x7f33a41a6b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:53908/0 (socket says 172.21.15.169:53908) 2024-03-28T01:07:13.568 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.563+0000 7f33a27fc640 1 -- 172.21.15.169:0/2640833923 learned_addr learned my addr 172.21.15.169:0/2640833923 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:13.568 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.564+0000 7f33a27fc640 1 -- 172.21.15.169:0/2640833923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33a410dbb0 msgr2=0x7f33a41adbf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:07:13.568 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.564+0000 7f33a27fc640 1 --2- 172.21.15.169:0/2640833923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33a410dbb0 0x7f33a41adbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.568 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.564+0000 7f33a27fc640 1 -- 172.21.15.169:0/2640833923 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33a4108730 msgr2=0x7f33a41a6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:13.568 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.564+0000 7f33a2ffd640 1 --2- 172.21.15.169:0/2640833923 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33a4108730 0x7f33a41a6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:13.568 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.564+0000 7f33a27fc640 1 --2- 172.21.15.169:0/2640833923 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33a4108730 0x7f33a41a6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.568 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.564+0000 7f33a27fc640 1 -- 172.21.15.169:0/2640833923 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a41186d0 con 0x7f33a410b0f0 2024-03-28T01:07:13.568 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.564+0000 7f33a2ffd640 1 --2- 172.21.15.169:0/2640833923 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33a4108730 0x7f33a41a6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:07:13.568 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.564+0000 7f33a27fc640 1 --2- 172.21.15.169:0/2640833923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33a410b0f0 0x7f33a41a6b80 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f339400e9f0 tx=0x7f339400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.570+0000 7f33a8849640 1 -- 172.21.15.169:0/2640833923 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339400ccc0 con 0x7f33a410b0f0 2024-03-28T01:07:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.570+0000 7f33a3fff640 1 -- 172.21.15.169:0/2640833923 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a41189c0 con 0x7f33a410b0f0 2024-03-28T01:07:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.570+0000 7f33a3fff640 1 -- 172.21.15.169:0/2640833923 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a4118f10 con 0x7f33a410b0f0 2024-03-28T01:07:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.571+0000 7f33a8849640 1 -- 172.21.15.169:0/2640833923 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3394010910 con 0x7f33a410b0f0 2024-03-28T01:07:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.571+0000 7f33a8849640 1 -- 172.21.15.169:0/2640833923 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3394018860 con 0x7f33a410b0f0 2024-03-28T01:07:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.574+0000 7f33ac432740 1 Processor -- start 2024-03-28T01:07:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.575+0000 7f33ac432740 1 -- start start 2024-03-28T01:07:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.575+0000 7f33ac432740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56143a883b20 0x56143a883f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.575+0000 7f33ac432740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 0x56143a889010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.575+0000 7f33ac432740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 0x56143a88bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.575+0000 7f33ac432740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56143a7e8b10 con 0x56143a883b20 2024-03-28T01:07:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.575+0000 7f33ac432740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x56143a791900 con 0x56143a889680 2024-03-28T01:07:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.575+0000 7f33ac432740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56143a7e4bb0 con 0x56143a884510 2024-03-28T01:07:13.579 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.576+0000 7f33a8849640 1 -- 172.21.15.169:0/2640833923 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3394018a00 con 0x7f33a410b0f0 2024-03-28T01:07:13.579 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.576+0000 7f33a2ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 0x56143a889010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:13.579 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.576+0000 7f33a27fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 0x56143a88bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:13.579 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.576+0000 7f33a27fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 0x56143a88bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:42456/0 (socket says 172.21.15.169:42456) 2024-03-28T01:07:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.576+0000 7f33a27fc640 1 -- 172.21.15.169:0/1258900599 learned_addr learned my addr 172.21.15.169:0/1258900599 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.577+0000 7f33a27fc640 1 -- 172.21.15.169:0/1258900599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 msgr2=0x56143a889010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.577+0000 7f33a27fc640 1 --2- 172.21.15.169:0/1258900599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 0x56143a889010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.577+0000 7f33a27fc640 1 -- 172.21.15.169:0/1258900599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56143a883b20 msgr2=0x56143a883f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.577+0000 7f33a27fc640 1 --2- 172.21.15.169:0/1258900599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56143a883b20 0x56143a883f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.577+0000 7f33a27fc640 1 -- 172.21.15.169:0/1258900599 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56143a4fd210 con 0x56143a889680 2024-03-28T01:07:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.577+0000 7f33a27fc640 1 --2- 172.21.15.169:0/1258900599 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 0x56143a88bb10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f33940d8320 tx=0x7f33940d3f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9e5d64e591fbc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.578+0000 7f33a8849640 1 --2- 172.21.15.169:0/2640833923 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f338c0a2a00 0x7f338c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.579+0000 7f33a8849640 1 -- 172.21.15.169:0/2640833923 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f33940d01a0 con 0x7f33a410b0f0 2024-03-28T01:07:13.583 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.580+0000 7f33a37fe640 1 --2- 172.21.15.169:0/2640833923 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f338c0a2a00 0x7f338c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:13.591 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.588+0000 7f3389ffb640 1 -- 172.21.15.169:0/1258900599 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33940e3070 con 0x56143a889680 2024-03-28T01:07:13.591 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.588+0000 7f3389ffb640 1 -- 172.21.15.169:0/1258900599 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33940d0ab0 con 0x56143a889680 2024-03-28T01:07:13.591 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.588+0000 7f3389ffb640 1 -- 172.21.15.169:0/1258900599 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33940deb50 con 0x56143a889680 2024-03-28T01:07:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.590+0000 7f33a37fe640 1 --2- 172.21.15.169:0/2640833923 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f338c0a2a00 0x7f338c0a4ec0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f339c037040 tx=0x7f339c005eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:13.593 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.591+0000 7f33ac432740 1 -- 172.21.15.169:0/1258900599 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 msgr2=0x56143a88bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:13.594 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.591+0000 7f33ac432740 1 --2- 172.21.15.169:0/1258900599 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 0x56143a88bb10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f33940d8320 tx=0x7f33940d3f90 comp rx=0 tx=0).stop 2024-03-28T01:07:13.594 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.591+0000 7f33ac432740 1 -- 172.21.15.169:0/1258900599 shutdown_connections 2024-03-28T01:07:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.591+0000 7f33ac432740 1 --2- 172.21.15.169:0/1258900599 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 0x56143a88bb10 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.591+0000 7f33ac432740 1 --2- 172.21.15.169:0/1258900599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 0x56143a889010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.591+0000 7f33ac432740 1 --2- 172.21.15.169:0/1258900599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56143a883b20 0x56143a883f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.591+0000 7f33ac432740 1 -- 172.21.15.169:0/1258900599 >> 172.21.15.169:0/1258900599 conn(0x56143a87f090 msgr2=0x56143a87f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.597+0000 7f33ac432740 1 -- 172.21.15.169:0/1258900599 shutdown_connections 2024-03-28T01:07:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.597+0000 7f33ac432740 1 -- 172.21.15.169:0/1258900599 wait complete. 2024-03-28T01:07:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.597+0000 7f33ac432740 1 -- start start 2024-03-28T01:07:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.598+0000 7f33ac432740 1 Processor -- start 2024-03-28T01:07:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.599+0000 7f33ac432740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56143a883b20 0x56143a88dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.599+0000 7f33ac432740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 0x56143a88e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.599+0000 7f33ac432740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 0x56143a895440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.600+0000 7f33a27fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 0x56143a88e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.600+0000 7f33a27fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 0x56143a88e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:53928/0 (socket says 172.21.15.169:53928) 2024-03-28T01:07:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.600+0000 7f33ac432740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56143a88c630 con 0x56143a883b20 2024-03-28T01:07:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.600+0000 7f33ac432740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x56143a7a4d50 con 0x56143a889680 2024-03-28T01:07:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.600+0000 7f33ac432740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56143a88c7b0 con 0x56143a884510 2024-03-28T01:07:13.604 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.600+0000 7f33ac432740 10 client.0 _ll_register_callbacks cb 0x56143a7a4720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:13.604 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.600+0000 7f33a27fc640 1 -- 172.21.15.169:0/2100416544 learned_addr learned my addr 172.21.15.169:0/2100416544 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:13.604 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.600+0000 7f33a27fc640 1 -- 172.21.15.169:0/2100416544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 msgr2=0x56143a895440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:07:13.604 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.600+0000 7f33a27fc640 1 --2- 172.21.15.169:0/2100416544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 0x56143a895440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.604 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.600+0000 7f33a27fc640 1 -- 172.21.15.169:0/2100416544 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56143a883b20 msgr2=0x56143a88dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:13.604 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.600+0000 7f33a27fc640 1 --2- 172.21.15.169:0/2100416544 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56143a883b20 0x56143a88dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.604 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.600+0000 7f33a27fc640 1 -- 172.21.15.169:0/2100416544 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56143a5c9620 con 0x56143a884510 2024-03-28T01:07:13.605 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.601+0000 7f33a27fc640 1 --2- 172.21.15.169:0/2100416544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 0x56143a88e400 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f33940d8c70 tx=0x7f33940d3c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8517ed2c374a66c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:13.609 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.606+0000 7f3382ffd640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:07:13.609 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.606+0000 7f3382ffd640 1 -- 172.21.15.169:0/2100416544 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33940989e0 con 0x56143a884510 2024-03-28T01:07:13.609 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.606+0000 7f3382ffd640 1 -- 172.21.15.169:0/2100416544 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f33940d22e0 con 0x56143a884510 2024-03-28T01:07:13.609 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.606+0000 7f33ac432740 1 -- 172.21.15.169:0/2100416544 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f33940d0ab0 con 0x56143a884510 2024-03-28T01:07:13.610 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.607+0000 7f3382ffd640 1 -- 172.21.15.169:0/2100416544 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f339409c0a0 con 0x56143a884510 2024-03-28T01:07:13.610 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.607+0000 7f33ac432740 10 client.27271 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:13.610 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.607+0000 7f33ac432740 1 -- 172.21.15.169:0/2100416544 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56143a88c630 con 0x56143a884510 2024-03-28T01:07:13.611 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.608+0000 7f3382ffd640 1 -- 172.21.15.169:0/2100416544 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33940d0ab0 con 0x56143a884510 2024-03-28T01:07:13.612 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.609+0000 7f3382ffd640 1 -- 172.21.15.169:0/2100416544 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f339409c9a0 con 0x56143a884510 2024-03-28T01:07:13.612 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.609+0000 7f33ac432740 10 client.27271 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:13.612 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.609+0000 7f33ac432740 10 client.27271 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:07:13.612 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.609+0000 7f33ac432740 20 client.27271 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:13.612 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.609+0000 7f33ac432740 1 --2- 172.21.15.169:0/2100416544 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56143a899930 0x56143a89bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.609+0000 7f33ac432740 4 client.27271 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:07:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.609+0000 7f33ac432740 1 -- 172.21.15.169:0/2100416544 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56143a7f6050 con 0x56143a899930 2024-03-28T01:07:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.609+0000 7f33a2ffd640 1 --2- 172.21.15.169:0/2100416544 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56143a899930 0x56143a89bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.610+0000 7f33a2ffd640 1 --2- 172.21.15.169:0/2100416544 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56143a899930 0x56143a89bd20 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a413fa8726fc3584 server_cookie=8b56a23c1c11f3d0 in_seq=0 out_seq=0 2024-03-28T01:07:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.610+0000 7f3382ffd640 10 client.27271 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:13.618 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.614+0000 7f3382ffd640 1 -- 172.21.15.169:0/2100416544 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56143a7f6050 con 0x56143a899930 2024-03-28T01:07:13.618 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.614+0000 7f3382ffd640 10 client.27271 handle_command_reply: tid=0 2024-03-28T01:07:13.618 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.615+0000 7f33ac432740 1 client.27271 shutdown 2024-03-28T01:07:13.620 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.617+0000 7f33ac432740 1 -- 172.21.15.169:0/2100416544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 msgr2=0x56143a88e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:13.620 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.617+0000 7f33ac432740 1 --2- 172.21.15.169:0/2100416544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 0x56143a88e400 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f33940d8c70 tx=0x7f33940d3c60 comp rx=0 tx=0).stop 2024-03-28T01:07:13.621 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.617+0000 7f33ac432740 1 -- 172.21.15.169:0/2100416544 shutdown_connections 2024-03-28T01:07:13.621 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.617+0000 7f33ac432740 1 --2- 172.21.15.169:0/2100416544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 0x56143a895440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.621 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.617+0000 7f33ac432740 1 --2- 172.21.15.169:0/2100416544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 0x56143a88e400 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.621 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.617+0000 7f33ac432740 1 --2- 172.21.15.169:0/2100416544 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56143a899930 0x56143a89bd20 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.621 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.617+0000 7f33ac432740 1 --2- 172.21.15.169:0/2100416544 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56143a883b20 0x56143a88dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.621 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.618+0000 7f33ac432740 1 -- 172.21.15.169:0/2100416544 >> 172.21.15.169:0/2100416544 conn(0x56143a87f090 msgr2=0x56143a87e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:13.621 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.618+0000 7f3382ffd640 0 client.27271 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:13.621 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.618+0000 7f33ac432740 1 -- 172.21.15.169:0/2100416544 shutdown_connections 2024-03-28T01:07:13.621 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.618+0000 7f33ac432740 1 -- 172.21.15.169:0/2100416544 wait complete. 2024-03-28T01:07:13.627 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.624+0000 7f33ac432740 20 client.27271 trim_cache size 0 max 16384 2024-03-28T01:07:13.670 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.664+0000 7f33ac432740 1 Processor -- start 2024-03-28T01:07:13.671 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.667+0000 7f33ac432740 1 -- start start 2024-03-28T01:07:13.671 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.667+0000 7f33ac432740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56143a883b20 0x56143a8857f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:13.671 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.667+0000 7f33ac432740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 0x56143a895400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:13.671 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.667+0000 7f33ac432740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 0x56143a885d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:13.671 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.667+0000 7f33ac432740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3390012090 con 0x56143a883b20 2024-03-28T01:07:13.672 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.667+0000 7f33ac432740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339000fb90 con 0x56143a889680 2024-03-28T01:07:13.672 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.667+0000 7f33ac432740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339000dc30 con 0x56143a884510 2024-03-28T01:07:13.673 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.669+0000 7f33a27fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 0x56143a895400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:13.673 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.670+0000 7f33a37fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 0x56143a885d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:13.673 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.670+0000 7f33a2ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56143a883b20 0x56143a8857f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:13.673 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.670+0000 7f33a2ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56143a883b20 0x56143a8857f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36298/0 (socket says 172.21.15.169:36298) 2024-03-28T01:07:13.673 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.670+0000 7f33a2ffd640 1 -- 172.21.15.169:0/2650045524 learned_addr learned my addr 172.21.15.169:0/2650045524 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.676+0000 7f33a27fc640 1 -- 172.21.15.169:0/2650045524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 msgr2=0x56143a885d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.676+0000 7f33a27fc640 1 --2- 172.21.15.169:0/2650045524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 0x56143a885d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.676+0000 7f33a27fc640 1 -- 172.21.15.169:0/2650045524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56143a883b20 msgr2=0x56143a8857f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.676+0000 7f33a27fc640 1 --2- 172.21.15.169:0/2650045524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56143a883b20 0x56143a8857f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.676+0000 7f33a27fc640 1 -- 172.21.15.169:0/2650045524 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56143a87fe00 con 0x56143a884510 2024-03-28T01:07:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.700+0000 7f33a27fc640 1 --2- 172.21.15.169:0/2650045524 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 0x56143a895400 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f339409cda0 tx=0x7f3394009720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5febbb2cdaed8aff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.704+0000 7f33897fa640 1 -- 172.21.15.169:0/2650045524 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339409c280 con 0x56143a884510 2024-03-28T01:07:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.704+0000 7f33897fa640 1 -- 172.21.15.169:0/2650045524 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f339409ce40 con 0x56143a884510 2024-03-28T01:07:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.706+0000 7f33ac432740 1 -- 172.21.15.169:0/2650045524 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 msgr2=0x56143a895400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.706+0000 7f33ac432740 1 --2- 172.21.15.169:0/2650045524 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 0x56143a895400 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f339409cda0 tx=0x7f3394009720 comp rx=0 tx=0).stop 2024-03-28T01:07:13.713 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.710+0000 7f33ac432740 1 -- 172.21.15.169:0/2650045524 shutdown_connections 2024-03-28T01:07:13.713 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.710+0000 7f33ac432740 1 --2- 172.21.15.169:0/2650045524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 0x56143a885d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.713 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.710+0000 7f33ac432740 1 --2- 172.21.15.169:0/2650045524 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 0x56143a895400 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.713 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.710+0000 7f33ac432740 1 --2- 172.21.15.169:0/2650045524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56143a883b20 0x56143a8857f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.713 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.710+0000 7f33ac432740 1 -- 172.21.15.169:0/2650045524 >> 172.21.15.169:0/2650045524 conn(0x56143a87f090 msgr2=0x56143a87f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.720+0000 7f33ac432740 1 -- 172.21.15.169:0/2650045524 shutdown_connections 2024-03-28T01:07:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.724+0000 7f33ac432740 1 -- 172.21.15.169:0/2650045524 wait complete. 2024-03-28T01:07:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.724+0000 7f33ac432740 1 -- start start 2024-03-28T01:07:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.725+0000 7f33ac432740 1 Processor -- start 2024-03-28T01:07:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.725+0000 7f33ac432740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56143a883b20 0x56143a892a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.725+0000 7f33ac432740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 0x56143a892f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.725+0000 7f33ac432740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 0x56143a893fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.725+0000 7f33ac432740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339000e230 con 0x56143a883b20 2024-03-28T01:07:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.725+0000 7f33ac432740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339000f530 con 0x56143a889680 2024-03-28T01:07:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.725+0000 7f33ac432740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339000e870 con 0x56143a884510 2024-03-28T01:07:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.725+0000 7f33ac432740 10 client.0 _ll_register_callbacks cb 0x7f33940d8270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:13.731 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.728+0000 7f33a37fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 0x56143a893fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:13.731 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.728+0000 7f33a37fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 0x56143a893fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:42478/0 (socket says 172.21.15.169:42478) 2024-03-28T01:07:13.731 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.728+0000 7f33a37fe640 1 -- 172.21.15.169:0/3563632789 learned_addr learned my addr 172.21.15.169:0/3563632789 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:13.731 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.728+0000 7f33a2ffd640 1 --2- 172.21.15.169:0/3563632789 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56143a883b20 0x56143a892a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.730+0000 7f33a27fc640 1 --2- 172.21.15.169:0/3563632789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 0x56143a892f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.731+0000 7f33a2ffd640 1 -- 172.21.15.169:0/3563632789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 msgr2=0x56143a892f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.731+0000 7f33a2ffd640 1 --2- 172.21.15.169:0/3563632789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 0x56143a892f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.731+0000 7f33a2ffd640 1 -- 172.21.15.169:0/3563632789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 msgr2=0x56143a893fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.731+0000 7f33a2ffd640 1 --2- 172.21.15.169:0/3563632789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 0x56143a893fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.731+0000 7f33a2ffd640 1 -- 172.21.15.169:0/3563632789 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56143a894590 con 0x56143a883b20 2024-03-28T01:07:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.735+0000 7f33a2ffd640 1 --2- 172.21.15.169:0/3563632789 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56143a883b20 0x56143a892a50 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f339800b860 tx=0x7f339800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=615559d6cfa66796 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.735+0000 7f3382ffd640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:07:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.739+0000 7f3382ffd640 1 -- 172.21.15.169:0/3563632789 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3398013810 con 0x56143a883b20 2024-03-28T01:07:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.739+0000 7f33ac432740 1 -- 172.21.15.169:0/3563632789 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f339409ce40 con 0x56143a883b20 2024-03-28T01:07:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.739+0000 7f3382ffd640 1 -- 172.21.15.169:0/3563632789 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f33980139b0 con 0x56143a883b20 2024-03-28T01:07:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.753+0000 7f3382ffd640 1 -- 172.21.15.169:0/3563632789 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3398029050 con 0x56143a883b20 2024-03-28T01:07:13.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.756+0000 7f33ac432740 10 client.17457 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:13.760 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.756+0000 7f33ac432740 1 -- 172.21.15.169:0/3563632789 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56143a8f5180 con 0x56143a883b20 2024-03-28T01:07:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.761+0000 7f3382ffd640 1 -- 172.21.15.169:0/3563632789 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f339409ce40 con 0x56143a883b20 2024-03-28T01:07:13.766 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.763+0000 7f3382ffd640 1 -- 172.21.15.169:0/3563632789 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f339801e020 con 0x56143a883b20 2024-03-28T01:07:13.766 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.763+0000 7f33ac432740 10 client.17457 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:13.766 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.763+0000 7f33ac432740 10 client.17457 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:07:13.766 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.763+0000 7f33ac432740 20 client.17457 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:13.766 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.763+0000 7f33ac432740 1 --2- 172.21.15.169:0/3563632789 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56143a8f5a40 0x56143a8f7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:13.766 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.763+0000 7f33ac432740 4 client.17457 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:07:13.766 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.763+0000 7f33ac432740 1 -- 172.21.15.169:0/3563632789 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56143a7f03a0 con 0x56143a8f5a40 2024-03-28T01:07:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.767+0000 7f33a27fc640 1 --2- 172.21.15.169:0/3563632789 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56143a8f5a40 0x56143a8f7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:13.789 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.785+0000 7f33a27fc640 1 --2- 172.21.15.169:0/3563632789 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56143a8f5a40 0x56143a8f7e60 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7a8afc0c8b60ee7 server_cookie=265d6ddde63ac52d in_seq=0 out_seq=0 2024-03-28T01:07:13.789 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:13.786+0000 7f3382ffd640 10 client.17457 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:14.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:13 smithi169 ceph-mon[32274]: pgmap v672: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 663 MiB/s wr, 1.54k op/s 2024-03-28T01:07:14.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:13 smithi174 ceph-mon[36060]: pgmap v672: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 663 MiB/s wr, 1.54k op/s 2024-03-28T01:07:14.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:13 smithi177 ceph-mon[36206]: pgmap v672: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 663 MiB/s wr, 1.54k op/s 2024-03-28T01:07:16.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:15 smithi169 ceph-mon[32274]: pgmap v673: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 524 MiB/s wr, 1.52k op/s 2024-03-28T01:07:16.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:15 smithi174 ceph-mon[36060]: pgmap v673: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 524 MiB/s wr, 1.52k op/s 2024-03-28T01:07:16.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:15 smithi177 ceph-mon[36206]: pgmap v673: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 524 MiB/s wr, 1.52k op/s 2024-03-28T01:07:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.070+0000 7f3382ffd640 1 -- 172.21.15.169:0/3563632789 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56143a7f03a0 con 0x56143a8f5a40 2024-03-28T01:07:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.070+0000 7f3382ffd640 10 client.17457 handle_command_reply: tid=0 2024-03-28T01:07:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.070+0000 7f33ac432740 1 client.17457 shutdown 2024-03-28T01:07:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.070+0000 7f33ac432740 1 -- 172.21.15.169:0/3563632789 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56143a883b20 msgr2=0x56143a892a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.070+0000 7f33ac432740 1 --2- 172.21.15.169:0/3563632789 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56143a883b20 0x56143a892a50 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f339800b860 tx=0x7f339800be80 comp rx=0 tx=0).stop 2024-03-28T01:07:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.070+0000 7f33ac432740 1 -- 172.21.15.169:0/3563632789 shutdown_connections 2024-03-28T01:07:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.070+0000 7f33ac432740 1 --2- 172.21.15.169:0/3563632789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56143a889680 0x56143a893fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:17.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.071+0000 7f33ac432740 1 --2- 172.21.15.169:0/3563632789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56143a884510 0x56143a892f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:17.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.071+0000 7f33ac432740 1 --2- 172.21.15.169:0/3563632789 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56143a8f5a40 0x56143a8f7e60 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:17.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.071+0000 7f33ac432740 1 --2- 172.21.15.169:0/3563632789 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56143a883b20 0x56143a892a50 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:17.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.071+0000 7f33ac432740 1 -- 172.21.15.169:0/3563632789 >> 172.21.15.169:0/3563632789 conn(0x56143a87f090 msgr2=0x56143a87f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:17.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.071+0000 7f3382ffd640 0 client.17457 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:17.077 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.074+0000 7f33ac432740 1 -- 172.21.15.169:0/3563632789 shutdown_connections 2024-03-28T01:07:17.077 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.074+0000 7f33ac432740 1 -- 172.21.15.169:0/3563632789 wait complete. 2024-03-28T01:07:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.075+0000 7f33ac432740 20 client.17457 trim_cache size 0 max 16384 2024-03-28T01:07:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:07:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:07:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:07:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:07:17.088 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.085+0000 7f33a3fff640 1 -- 172.21.15.169:0/2640833923 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f338c0a2a00 msgr2=0x7f338c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.085+0000 7f33a3fff640 1 --2- 172.21.15.169:0/2640833923 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f338c0a2a00 0x7f338c0a4ec0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f339c037040 tx=0x7f339c005eb0 comp rx=0 tx=0).stop 2024-03-28T01:07:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.085+0000 7f33a3fff640 1 -- 172.21.15.169:0/2640833923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33a410b0f0 msgr2=0x7f33a41a6b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.085+0000 7f33a3fff640 1 --2- 172.21.15.169:0/2640833923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33a410b0f0 0x7f33a41a6b80 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f339400e9f0 tx=0x7f339400eeb0 comp rx=0 tx=0).stop 2024-03-28T01:07:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.085+0000 7f33a3fff640 1 -- 172.21.15.169:0/2640833923 shutdown_connections 2024-03-28T01:07:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.085+0000 7f33a3fff640 1 --2- 172.21.15.169:0/2640833923 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f338c0a2a00 0x7f338c0a4ec0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.085+0000 7f33a3fff640 1 --2- 172.21.15.169:0/2640833923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33a410dbb0 0x7f33a41adbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.085+0000 7f33a3fff640 1 --2- 172.21.15.169:0/2640833923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33a410b0f0 0x7f33a41a6b80 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.085+0000 7f33a3fff640 1 --2- 172.21.15.169:0/2640833923 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33a4108730 0x7f33a41a6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.085+0000 7f33a3fff640 1 -- 172.21.15.169:0/2640833923 >> 172.21.15.169:0/2640833923 conn(0x7f33a406e510 msgr2=0x7f33a4108b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.085+0000 7f33a3fff640 1 -- 172.21.15.169:0/2640833923 shutdown_connections 2024-03-28T01:07:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:17.085+0000 7f33a3fff640 1 -- 172.21.15.169:0/2640833923 wait complete. 2024-03-28T01:07:18.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:17 smithi169 ceph-mon[32274]: pgmap v674: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 615 MiB/s wr, 1.60k op/s 2024-03-28T01:07:18.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:17 smithi174 ceph-mon[36060]: pgmap v674: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 615 MiB/s wr, 1.60k op/s 2024-03-28T01:07:18.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:17 smithi177 ceph-mon[36206]: pgmap v674: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 615 MiB/s wr, 1.60k op/s 2024-03-28T01:07:20.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:19 smithi169 ceph-mon[32274]: pgmap v675: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 580 MiB/s wr, 1.35k op/s 2024-03-28T01:07:20.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:19 smithi174 ceph-mon[36060]: pgmap v675: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 580 MiB/s wr, 1.35k op/s 2024-03-28T01:07:20.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:19 smithi177 ceph-mon[36206]: pgmap v675: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 580 MiB/s wr, 1.35k op/s 2024-03-28T01:07:22.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:21 smithi169 ceph-mon[32274]: pgmap v676: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 565 MiB/s wr, 1.20k op/s 2024-03-28T01:07:22.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:21 smithi174 ceph-mon[36060]: pgmap v676: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 565 MiB/s wr, 1.20k op/s 2024-03-28T01:07:22.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:21 smithi177 ceph-mon[36206]: pgmap v676: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 565 MiB/s wr, 1.20k op/s 2024-03-28T01:07:24.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:23 smithi169 ceph-mon[32274]: pgmap v677: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 683 MiB/s wr, 1.56k op/s 2024-03-28T01:07:24.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:23 smithi174 ceph-mon[36060]: pgmap v677: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 683 MiB/s wr, 1.56k op/s 2024-03-28T01:07:24.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:23 smithi177 ceph-mon[36206]: pgmap v677: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 683 MiB/s wr, 1.56k op/s 2024-03-28T01:07:26.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:25 smithi169 ceph-mon[32274]: pgmap v678: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 560 MiB/s wr, 1.26k op/s 2024-03-28T01:07:26.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:25 smithi174 ceph-mon[36060]: pgmap v678: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 560 MiB/s wr, 1.26k op/s 2024-03-28T01:07:26.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:25 smithi177 ceph-mon[36206]: pgmap v678: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 560 MiB/s wr, 1.26k op/s 2024-03-28T01:07:28.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:27 smithi169 ceph-mon[32274]: pgmap v679: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 690 MiB/s wr, 1.24k op/s 2024-03-28T01:07:28.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:27 smithi174 ceph-mon[36060]: pgmap v679: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 690 MiB/s wr, 1.24k op/s 2024-03-28T01:07:28.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:27 smithi177 ceph-mon[36206]: pgmap v679: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 690 MiB/s wr, 1.24k op/s 2024-03-28T01:07:30.056 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:29 smithi169 ceph-mon[32274]: pgmap v680: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 615 MiB/s wr, 1.11k op/s 2024-03-28T01:07:30.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:29 smithi174 ceph-mon[36060]: pgmap v680: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 615 MiB/s wr, 1.11k op/s 2024-03-28T01:07:30.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:29 smithi177 ceph-mon[36206]: pgmap v680: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 615 MiB/s wr, 1.11k op/s 2024-03-28T01:07:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.387+0000 7f9a290c6640 1 -- 172.21.15.169:0/733798902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a2410d9c0 msgr2=0x7f9a2410de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.387+0000 7f9a290c6640 1 --2- 172.21.15.169:0/733798902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a2410d9c0 0x7f9a2410de40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c009be0 tx=0x7f9a1c02bf10 comp rx=0 tx=0).stop 2024-03-28T01:07:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.387+0000 7f9a290c6640 1 -- 172.21.15.169:0/733798902 shutdown_connections 2024-03-28T01:07:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.387+0000 7f9a290c6640 1 --2- 172.21.15.169:0/733798902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a2410d9c0 0x7f9a2410de40 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.388+0000 7f9a290c6640 1 --2- 172.21.15.169:0/733798902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a2410b010 0x7f9a2410d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.388+0000 7f9a290c6640 1 --2- 172.21.15.169:0/733798902 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a241086a0 0x7f9a2410aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.388+0000 7f9a290c6640 1 -- 172.21.15.169:0/733798902 >> 172.21.15.169:0/733798902 conn(0x7f9a2406e6e0 msgr2=0x7f9a24070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:30.396 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.393+0000 7f9a290c6640 1 -- 172.21.15.169:0/733798902 shutdown_connections 2024-03-28T01:07:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.396+0000 7f9a290c6640 1 -- 172.21.15.169:0/733798902 wait complete. 2024-03-28T01:07:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.398+0000 7f9a290c6640 1 Processor -- start 2024-03-28T01:07:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.398+0000 7f9a290c6640 1 -- start start 2024-03-28T01:07:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.398+0000 7f9a290c6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a241086a0 0x7f9a24114cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.398+0000 7f9a290c6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a2410b010 0x7f9a24115210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.398+0000 7f9a290c6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a2410d9c0 0x7f9a241b6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.399+0000 7f9a290c6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2411c180 con 0x7f9a241086a0 2024-03-28T01:07:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.399+0000 7f9a290c6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2411c000 con 0x7f9a2410b010 2024-03-28T01:07:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.399+0000 7f9a290c6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2411c300 con 0x7f9a2410d9c0 2024-03-28T01:07:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.399+0000 7f9a22d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a241086a0 0x7f9a24114cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.399+0000 7f9a22d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a241086a0 0x7f9a24114cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:55232/0 (socket says 172.21.15.169:55232) 2024-03-28T01:07:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.399+0000 7f9a22d9d640 1 -- 172.21.15.169:0/2038396685 learned_addr learned my addr 172.21.15.169:0/2038396685 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.399+0000 7f9a2359e640 1 --2- 172.21.15.169:0/2038396685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a2410d9c0 0x7f9a241b6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.401+0000 7f9a22d9d640 1 -- 172.21.15.169:0/2038396685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a2410d9c0 msgr2=0x7f9a241b6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.401+0000 7f9a22d9d640 1 --2- 172.21.15.169:0/2038396685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a2410d9c0 0x7f9a241b6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.401+0000 7f9a22d9d640 1 -- 172.21.15.169:0/2038396685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a2410b010 msgr2=0x7f9a24115210 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:07:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.401+0000 7f9a22d9d640 1 --2- 172.21.15.169:0/2038396685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a2410b010 0x7f9a24115210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.401+0000 7f9a22d9d640 1 -- 172.21.15.169:0/2038396685 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a24118340 con 0x7f9a241086a0 2024-03-28T01:07:30.407 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.401+0000 7f9a2359e640 1 --2- 172.21.15.169:0/2038396685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a2410d9c0 0x7f9a241b6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:07:30.407 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.402+0000 7f9a22d9d640 1 --2- 172.21.15.169:0/2038396685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a241086a0 0x7f9a24114cd0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f9a1800b4a0 tx=0x7f9a1800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:30.413 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.410+0000 7f9a0bfff640 1 -- 172.21.15.169:0/2038396685 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a18014070 con 0x7f9a241086a0 2024-03-28T01:07:30.413 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.410+0000 7f9a290c6640 1 -- 172.21.15.169:0/2038396685 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a241b6e30 con 0x7f9a241086a0 2024-03-28T01:07:30.413 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.410+0000 7f9a290c6640 1 -- 172.21.15.169:0/2038396685 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a241b7290 con 0x7f9a241086a0 2024-03-28T01:07:30.414 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.410+0000 7f9a0bfff640 1 -- 172.21.15.169:0/2038396685 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a18010af0 con 0x7f9a241086a0 2024-03-28T01:07:30.414 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.410+0000 7f9a0bfff640 1 -- 172.21.15.169:0/2038396685 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a18017c50 con 0x7f9a241086a0 2024-03-28T01:07:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.414+0000 7f9a2bd8f740 1 Processor -- start 2024-03-28T01:07:30.427 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.416+0000 7f9a0bfff640 1 -- 172.21.15.169:0/2038396685 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9a18017070 con 0x7f9a241086a0 2024-03-28T01:07:30.428 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.417+0000 7f9a0bfff640 1 --2- 172.21.15.169:0/2038396685 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9a0c0a2ae0 0x7f9a0c0a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:30.428 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.417+0000 7f9a0bfff640 1 -- 172.21.15.169:0/2038396685 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f9a180ce140 con 0x7f9a241086a0 2024-03-28T01:07:30.428 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.417+0000 7f9a2359e640 1 --2- 172.21.15.169:0/2038396685 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9a0c0a2ae0 0x7f9a0c0a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:30.428 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.420+0000 7f9a2359e640 1 --2- 172.21.15.169:0/2038396685 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9a0c0a2ae0 0x7f9a0c0a4fa0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c033450 tx=0x7f9a1c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:30.428 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.425+0000 7f9a2bd8f740 1 -- start start 2024-03-28T01:07:30.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.425+0000 7f9a2bd8f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6283920 0x5603d6283d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:30.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.425+0000 7f9a2bd8f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6284310 0x5603d6288e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:30.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.425+0000 7f9a2bd8f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603d6289480 0x5603d628b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:30.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.425+0000 7f9a2bd8f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603d61e8b10 con 0x5603d6284310 2024-03-28T01:07:30.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.425+0000 7f9a2bd8f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603d6191900 con 0x5603d6289480 2024-03-28T01:07:30.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.425+0000 7f9a2bd8f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603d61e4bb0 con 0x5603d6283920 2024-03-28T01:07:30.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.425+0000 7f9a22d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6283920 0x5603d6283d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:30.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.425+0000 7f9a22d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6283920 0x5603d6283d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:44680/0 (socket says 172.21.15.169:44680) 2024-03-28T01:07:30.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.425+0000 7f9a22d9d640 1 -- 172.21.15.169:0/3707189505 learned_addr learned my addr 172.21.15.169:0/3707189505 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:30.433 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.426+0000 7f9a2359e640 1 --2- 172.21.15.169:0/3707189505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603d6289480 0x5603d628b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:30.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.427+0000 7f9a22d9d640 1 -- 172.21.15.169:0/3707189505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603d6289480 msgr2=0x5603d628b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:30.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.427+0000 7f9a22d9d640 1 --2- 172.21.15.169:0/3707189505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603d6289480 0x5603d628b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.427+0000 7f9a22d9d640 1 -- 172.21.15.169:0/3707189505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6284310 msgr2=0x5603d6288e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:07:30.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.427+0000 7f9a22d9d640 1 --2- 172.21.15.169:0/3707189505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6284310 0x5603d6288e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.427+0000 7f9a22d9d640 1 -- 172.21.15.169:0/3707189505 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5603d5efd210 con 0x5603d6283920 2024-03-28T01:07:30.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.428+0000 7f9a22d9d640 1 --2- 172.21.15.169:0/3707189505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6283920 0x5603d6283d40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9a180970b0 tx=0x7f9a180047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e633a898fa5f7e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:30.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.428+0000 7f9a097fa640 1 -- 172.21.15.169:0/3707189505 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1801e040 con 0x5603d6283920 2024-03-28T01:07:30.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.428+0000 7f9a097fa640 1 -- 172.21.15.169:0/3707189505 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a1809e830 con 0x5603d6283920 2024-03-28T01:07:30.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.429+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/3707189505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6283920 msgr2=0x5603d6283d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:30.434 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.429+0000 7f9a2bd8f740 1 --2- 172.21.15.169:0/3707189505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6283920 0x5603d6283d40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9a180970b0 tx=0x7f9a180047d0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.436 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.433+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/3707189505 shutdown_connections 2024-03-28T01:07:30.436 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.433+0000 7f9a2bd8f740 1 --2- 172.21.15.169:0/3707189505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603d6289480 0x5603d628b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.436 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.433+0000 7f9a2bd8f740 1 --2- 172.21.15.169:0/3707189505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6284310 0x5603d6288e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.436 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.433+0000 7f9a2bd8f740 1 --2- 172.21.15.169:0/3707189505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6283920 0x5603d6283d40 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.436 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.433+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/3707189505 >> 172.21.15.169:0/3707189505 conn(0x5603d627ee90 msgr2=0x5603d627f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:30.440 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.437+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/3707189505 shutdown_connections 2024-03-28T01:07:30.442 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.439+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/3707189505 wait complete. 2024-03-28T01:07:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.439+0000 7f9a2bd8f740 1 -- start start 2024-03-28T01:07:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.439+0000 7f9a2bd8f740 1 Processor -- start 2024-03-28T01:07:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.439+0000 7f9a2bd8f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6283920 0x5603d628dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.440+0000 7f9a2bd8f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603d6284310 0x5603d628e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.440+0000 7f9a2bd8f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6289480 0x5603d6295250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.440+0000 7f9a2bd8f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603d628c480 con 0x5603d6289480 2024-03-28T01:07:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.440+0000 7f9a2bd8f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603d61a4d50 con 0x5603d6284310 2024-03-28T01:07:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.440+0000 7f9a2bd8f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603d628c600 con 0x5603d6283920 2024-03-28T01:07:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.440+0000 7f9a2bd8f740 10 client.0 _ll_register_callbacks cb 0x5603d61a4720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:30.444 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.441+0000 7f9a22d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6283920 0x5603d628dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:30.444 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.441+0000 7f9a2359e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6289480 0x5603d6295250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:30.444 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.441+0000 7f9a2359e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6289480 0x5603d6295250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:55248/0 (socket says 172.21.15.169:55248) 2024-03-28T01:07:30.444 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.441+0000 7f9a2359e640 1 -- 172.21.15.169:0/2880096532 learned_addr learned my addr 172.21.15.169:0/2880096532 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:30.446 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.443+0000 7f9a2259c640 1 --2- 172.21.15.169:0/2880096532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603d6284310 0x5603d628e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:30.450 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.447+0000 7f9a22d9d640 1 -- 172.21.15.169:0/2880096532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603d6284310 msgr2=0x5603d628e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:30.450 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.447+0000 7f9a22d9d640 1 --2- 172.21.15.169:0/2880096532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603d6284310 0x5603d628e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.450 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.447+0000 7f9a22d9d640 1 -- 172.21.15.169:0/2880096532 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6289480 msgr2=0x5603d6295250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:30.450 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.447+0000 7f9a22d9d640 1 --2- 172.21.15.169:0/2880096532 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6289480 0x5603d6295250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.450 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.447+0000 7f9a22d9d640 1 -- 172.21.15.169:0/2880096532 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5603d5fc9620 con 0x5603d6283920 2024-03-28T01:07:30.453 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.451+0000 7f9a22d9d640 1 --2- 172.21.15.169:0/2880096532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6283920 0x5603d628dd40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9a1800b470 tx=0x7f9a18009230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f89ca71986ce2b05 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:30.454 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.452+0000 7f9a06ff5640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:07:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.452+0000 7f9a06ff5640 1 -- 172.21.15.169:0/2880096532 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a180b2030 con 0x5603d6283920 2024-03-28T01:07:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.452+0000 7f9a06ff5640 1 -- 172.21.15.169:0/2880096532 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f9a180b3070 con 0x5603d6283920 2024-03-28T01:07:30.456 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.454+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/2880096532 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9a1809e830 con 0x5603d6283920 2024-03-28T01:07:30.460 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.457+0000 7f9a06ff5640 1 -- 172.21.15.169:0/2880096532 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a1801e040 con 0x5603d6283920 2024-03-28T01:07:30.462 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.460+0000 7f9a2bd8f740 10 client.27289 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:30.462 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.460+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/2880096532 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5603d628c480 con 0x5603d6283920 2024-03-28T01:07:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.461+0000 7f9a06ff5640 1 -- 172.21.15.169:0/2880096532 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a1809e830 con 0x5603d6283920 2024-03-28T01:07:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.462+0000 7f9a06ff5640 1 -- 172.21.15.169:0/2880096532 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9a18099b30 con 0x5603d6283920 2024-03-28T01:07:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.462+0000 7f9a2bd8f740 10 client.27289 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.462+0000 7f9a2bd8f740 10 client.27289 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:07:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.462+0000 7f9a2bd8f740 20 client.27289 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.462+0000 7f9a2bd8f740 1 --2- 172.21.15.169:0/2880096532 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5603d6299cc0 0x5603d629c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.462+0000 7f9a2bd8f740 4 client.27289 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:07:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.462+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/2880096532 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5603d61f6050 con 0x5603d6299cc0 2024-03-28T01:07:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.462+0000 7f9a2259c640 1 --2- 172.21.15.169:0/2880096532 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5603d6299cc0 0x5603d629c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.468+0000 7f9a2259c640 1 --2- 172.21.15.169:0/2880096532 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5603d6299cc0 0x5603d629c0e0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a7383dd41dfd9e8 server_cookie=620e3f60228e1e93 in_seq=0 out_seq=0 2024-03-28T01:07:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.468+0000 7f9a06ff5640 10 client.27289 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.470+0000 7f9a06ff5640 1 -- 172.21.15.169:0/2880096532 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5603d61f6050 con 0x5603d6299cc0 2024-03-28T01:07:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.470+0000 7f9a06ff5640 10 client.27289 handle_command_reply: tid=0 2024-03-28T01:07:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.470+0000 7f9a2bd8f740 1 client.27289 shutdown 2024-03-28T01:07:30.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.479+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/2880096532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6283920 msgr2=0x5603d628dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:30.483 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.479+0000 7f9a2bd8f740 1 --2- 172.21.15.169:0/2880096532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6283920 0x5603d628dd40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9a1800b470 tx=0x7f9a18009230 comp rx=0 tx=0).stop 2024-03-28T01:07:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.483+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/2880096532 shutdown_connections 2024-03-28T01:07:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.483+0000 7f9a2bd8f740 1 --2- 172.21.15.169:0/2880096532 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5603d6299cc0 0x5603d629c0e0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.483+0000 7f9a2bd8f740 1 --2- 172.21.15.169:0/2880096532 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6289480 0x5603d6295250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.483+0000 7f9a2bd8f740 1 --2- 172.21.15.169:0/2880096532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603d6284310 0x5603d628e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.483+0000 7f9a2bd8f740 1 --2- 172.21.15.169:0/2880096532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6283920 0x5603d628dd40 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.483+0000 7f9a06ff5640 0 client.27289 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.483+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/2880096532 >> 172.21.15.169:0/2880096532 conn(0x5603d627ee90 msgr2=0x5603d627e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:30.489 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.486+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/2880096532 shutdown_connections 2024-03-28T01:07:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.488+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/2880096532 wait complete. 2024-03-28T01:07:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.489+0000 7f9a2bd8f740 20 client.27289 trim_cache size 0 max 16384 2024-03-28T01:07:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.541+0000 7f9a2bd8f740 1 Processor -- start 2024-03-28T01:07:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.541+0000 7f9a2bd8f740 1 -- start start 2024-03-28T01:07:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.541+0000 7f9a2bd8f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6283920 0x5603d628d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.541+0000 7f9a2bd8f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6284310 0x5603d6295210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.541+0000 7f9a2bd8f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603d6289480 0x5603d628de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.541+0000 7f9a2bd8f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a10011fa0 con 0x5603d6283920 2024-03-28T01:07:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.541+0000 7f9a2bd8f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1000faa0 con 0x5603d6289480 2024-03-28T01:07:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.541+0000 7f9a2bd8f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1000db20 con 0x5603d6284310 2024-03-28T01:07:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.543+0000 7f9a2259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6284310 0x5603d6295210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.543+0000 7f9a2359e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603d6289480 0x5603d628de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.543+0000 7f9a22d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6283920 0x5603d628d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.543+0000 7f9a22d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6283920 0x5603d628d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:55252/0 (socket says 172.21.15.169:55252) 2024-03-28T01:07:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.543+0000 7f9a22d9d640 1 -- 172.21.15.169:0/2090724835 learned_addr learned my addr 172.21.15.169:0/2090724835 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:30.549 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.546+0000 7f9a22d9d640 1 -- 172.21.15.169:0/2090724835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6284310 msgr2=0x5603d6295210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:30.549 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.546+0000 7f9a22d9d640 1 --2- 172.21.15.169:0/2090724835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6284310 0x5603d6295210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.549 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.546+0000 7f9a22d9d640 1 -- 172.21.15.169:0/2090724835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603d6289480 msgr2=0x5603d628de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:30.550 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.546+0000 7f9a22d9d640 1 --2- 172.21.15.169:0/2090724835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603d6289480 0x5603d628de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.550 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.546+0000 7f9a22d9d640 1 -- 172.21.15.169:0/2090724835 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5603d6285120 con 0x5603d6283920 2024-03-28T01:07:30.553 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.550+0000 7f9a22d9d640 1 --2- 172.21.15.169:0/2090724835 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6283920 0x5603d628d950 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f9a1809a3e0 tx=0x7f9a18006e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fecf03af96bdf862 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:30.559 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.555+0000 7f9a08ff9640 1 -- 172.21.15.169:0/2090724835 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a180b2070 con 0x5603d6283920 2024-03-28T01:07:30.559 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.556+0000 7f9a08ff9640 1 -- 172.21.15.169:0/2090724835 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a1807e070 con 0x5603d6283920 2024-03-28T01:07:30.559 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.556+0000 7f9a08ff9640 1 -- 172.21.15.169:0/2090724835 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a180b3480 con 0x5603d6283920 2024-03-28T01:07:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.566+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/2090724835 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6283920 msgr2=0x5603d628d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.566+0000 7f9a2bd8f740 1 --2- 172.21.15.169:0/2090724835 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6283920 0x5603d628d950 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f9a1809a3e0 tx=0x7f9a18006e40 comp rx=0 tx=0).stop 2024-03-28T01:07:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.574+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/2090724835 shutdown_connections 2024-03-28T01:07:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.574+0000 7f9a2bd8f740 1 --2- 172.21.15.169:0/2090724835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603d6289480 0x5603d628de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.574+0000 7f9a2bd8f740 1 --2- 172.21.15.169:0/2090724835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6284310 0x5603d6295210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.574+0000 7f9a2bd8f740 1 --2- 172.21.15.169:0/2090724835 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6283920 0x5603d628d950 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.574+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/2090724835 >> 172.21.15.169:0/2090724835 conn(0x5603d627ee90 msgr2=0x5603d6288c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.574+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/2090724835 shutdown_connections 2024-03-28T01:07:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.575+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/2090724835 wait complete. 2024-03-28T01:07:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.575+0000 7f9a2bd8f740 1 -- start start 2024-03-28T01:07:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.575+0000 7f9a2bd8f740 1 Processor -- start 2024-03-28T01:07:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.575+0000 7f9a2bd8f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6283920 0x5603d62928b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.575+0000 7f9a2bd8f740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6284310 0x5603d6292df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.576+0000 7f9a2bd8f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603d6289480 0x5603d6293d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.576+0000 7f9a2bd8f740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1000e120 con 0x5603d6284310 2024-03-28T01:07:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.576+0000 7f9a2bd8f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1000f440 con 0x5603d6289480 2024-03-28T01:07:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.576+0000 7f9a2259c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6284310 0x5603d6292df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.576+0000 7f9a2bd8f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1000e780 con 0x5603d6283920 2024-03-28T01:07:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.576+0000 7f9a2259c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6284310 0x5603d6292df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:55260/0 (socket says 172.21.15.169:55260) 2024-03-28T01:07:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.576+0000 7f9a2259c640 1 -- 172.21.15.169:0/605795325 learned_addr learned my addr 172.21.15.169:0/605795325 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.576+0000 7f9a2bd8f740 10 client.0 _ll_register_callbacks cb 0x7f9a18010ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.576+0000 7f9a2359e640 1 --2- 172.21.15.169:0/605795325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603d6289480 0x5603d6293d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.576+0000 7f9a22d9d640 1 --2- 172.21.15.169:0/605795325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6283920 0x5603d62928b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.577+0000 7f9a22d9d640 1 -- 172.21.15.169:0/605795325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603d6289480 msgr2=0x5603d6293d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.577+0000 7f9a22d9d640 1 --2- 172.21.15.169:0/605795325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603d6289480 0x5603d6293d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.577+0000 7f9a22d9d640 1 -- 172.21.15.169:0/605795325 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6284310 msgr2=0x5603d6292df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.577+0000 7f9a22d9d640 1 --2- 172.21.15.169:0/605795325 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6284310 0x5603d6292df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.577+0000 7f9a22d9d640 1 -- 172.21.15.169:0/605795325 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5603d6294380 con 0x5603d6283920 2024-03-28T01:07:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.578+0000 7f9a22d9d640 1 --2- 172.21.15.169:0/605795325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6283920 0x5603d62928b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9a18098990 tx=0x7f9a180a2fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42ef62d4d8c69a2e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.581+0000 7f9a06ff5640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:07:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.581+0000 7f9a06ff5640 1 -- 172.21.15.169:0/605795325 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a180b2070 con 0x5603d6283920 2024-03-28T01:07:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.581+0000 7f9a06ff5640 1 -- 172.21.15.169:0/605795325 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f9a180b3480 con 0x5603d6283920 2024-03-28T01:07:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.581+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/605795325 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9a1807e070 con 0x5603d6283920 2024-03-28T01:07:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.587+0000 7f9a06ff5640 1 -- 172.21.15.169:0/605795325 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a1808a050 con 0x5603d6283920 2024-03-28T01:07:30.595 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.588+0000 7f9a2bd8f740 10 client.27295 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:30.595 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.588+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/605795325 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5603d6294380 con 0x5603d6283920 2024-03-28T01:07:30.595 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.592+0000 7f9a06ff5640 1 -- 172.21.15.169:0/605795325 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a1807e070 con 0x5603d6283920 2024-03-28T01:07:30.595 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.592+0000 7f9a06ff5640 1 -- 172.21.15.169:0/605795325 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f9a180aecd0 con 0x5603d6283920 2024-03-28T01:07:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.592+0000 7f9a2bd8f740 10 client.27295 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.592+0000 7f9a2bd8f740 10 client.27295 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:07:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.592+0000 7f9a2bd8f740 20 client.27295 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.592+0000 7f9a2bd8f740 1 --2- 172.21.15.169:0/605795325 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5603d62f5990 0x5603d62f7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.592+0000 7f9a2bd8f740 4 client.27295 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:07:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.592+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/605795325 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5603d61f03a0 con 0x5603d62f5990 2024-03-28T01:07:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.594+0000 7f9a2259c640 1 --2- 172.21.15.169:0/605795325 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5603d62f5990 0x5603d62f7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.599+0000 7f9a2259c640 1 --2- 172.21.15.169:0/605795325 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5603d62f5990 0x5603d62f7db0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61a1e0a38ec04627 server_cookie=7aa3c1363f36834c in_seq=0 out_seq=0 2024-03-28T01:07:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:30.603+0000 7f9a06ff5640 10 client.27295 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:31.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:30 smithi174 ceph-mon[36060]: pgmap v681: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 602 MiB/s wr, 1.11k op/s 2024-03-28T01:07:31.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:30 smithi177 ceph-mon[36206]: pgmap v681: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 602 MiB/s wr, 1.11k op/s 2024-03-28T01:07:31.306 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:30 smithi169 ceph-mon[32274]: pgmap v681: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 602 MiB/s wr, 1.11k op/s 2024-03-28T01:07:34.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:33 smithi169 ceph-mon[32274]: pgmap v682: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 686 MiB/s wr, 1.28k op/s 2024-03-28T01:07:34.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:33 smithi174 ceph-mon[36060]: pgmap v682: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 686 MiB/s wr, 1.28k op/s 2024-03-28T01:07:34.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:33 smithi177 ceph-mon[36206]: pgmap v682: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 686 MiB/s wr, 1.28k op/s 2024-03-28T01:07:34.807 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.804+0000 7f9a06ff5640 1 -- 172.21.15.169:0/605795325 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5603d61f03a0 con 0x5603d62f5990 2024-03-28T01:07:34.808 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.804+0000 7f9a06ff5640 10 client.27295 handle_command_reply: tid=0 2024-03-28T01:07:34.808 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.804+0000 7f9a2bd8f740 1 client.27295 shutdown 2024-03-28T01:07:34.808 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.804+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/605795325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6283920 msgr2=0x5603d62928b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:34.808 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.804+0000 7f9a2bd8f740 1 --2- 172.21.15.169:0/605795325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6283920 0x5603d62928b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9a18098990 tx=0x7f9a180a2fc0 comp rx=0 tx=0).stop 2024-03-28T01:07:34.808 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.804+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/605795325 shutdown_connections 2024-03-28T01:07:34.808 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.804+0000 7f9a2bd8f740 1 --2- 172.21.15.169:0/605795325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603d6289480 0x5603d6293d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:34.808 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.804+0000 7f9a2bd8f740 1 --2- 172.21.15.169:0/605795325 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5603d62f5990 0x5603d62f7db0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:34.808 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.804+0000 7f9a2bd8f740 1 --2- 172.21.15.169:0/605795325 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5603d6284310 0x5603d6292df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:34.808 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.804+0000 7f9a2bd8f740 1 --2- 172.21.15.169:0/605795325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d6283920 0x5603d62928b0 secure :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9a18098990 tx=0x7f9a180a2fc0 comp rx=0 tx=0).stop 2024-03-28T01:07:34.808 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.804+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/605795325 >> 172.21.15.169:0/605795325 conn(0x5603d627ee90 msgr2=0x5603d628d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:34.808 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.805+0000 7f9a06ff5640 0 client.27295 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:34.812 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.809+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/605795325 shutdown_connections 2024-03-28T01:07:34.812 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.809+0000 7f9a2bd8f740 1 -- 172.21.15.169:0/605795325 wait complete. 2024-03-28T01:07:34.812 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.809+0000 7f9a2bd8f740 20 client.27295 trim_cache size 0 max 16384 2024-03-28T01:07:34.812 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:07:34.812 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:07:34.813 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:07:34.813 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:07:34.813 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.810+0000 7f9a290c6640 1 -- 172.21.15.169:0/2038396685 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9a0c0a2ae0 msgr2=0x7f9a0c0a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:34.813 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.810+0000 7f9a290c6640 1 --2- 172.21.15.169:0/2038396685 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9a0c0a2ae0 0x7f9a0c0a4fa0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c033450 tx=0x7f9a1c005fd0 comp rx=0 tx=0).stop 2024-03-28T01:07:34.813 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.810+0000 7f9a290c6640 1 -- 172.21.15.169:0/2038396685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a241086a0 msgr2=0x7f9a24114cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:34.814 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.810+0000 7f9a290c6640 1 --2- 172.21.15.169:0/2038396685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a241086a0 0x7f9a24114cd0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f9a1800b4a0 tx=0x7f9a1800b960 comp rx=0 tx=0).stop 2024-03-28T01:07:34.814 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.810+0000 7f9a290c6640 1 -- 172.21.15.169:0/2038396685 shutdown_connections 2024-03-28T01:07:34.814 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.810+0000 7f9a290c6640 1 --2- 172.21.15.169:0/2038396685 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f9a0c0a2ae0 0x7f9a0c0a4fa0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:34.814 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.810+0000 7f9a290c6640 1 --2- 172.21.15.169:0/2038396685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a2410d9c0 0x7f9a241b6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:34.814 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.810+0000 7f9a290c6640 1 --2- 172.21.15.169:0/2038396685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a2410b010 0x7f9a24115210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:34.814 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.810+0000 7f9a290c6640 1 --2- 172.21.15.169:0/2038396685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9a241086a0 0x7f9a24114cd0 secure :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f9a1800b4a0 tx=0x7f9a1800b960 comp rx=0 tx=0).stop 2024-03-28T01:07:34.814 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.810+0000 7f9a290c6640 1 -- 172.21.15.169:0/2038396685 >> 172.21.15.169:0/2038396685 conn(0x7f9a2406e6e0 msgr2=0x7f9a2410be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:34.815 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.812+0000 7f9a290c6640 1 -- 172.21.15.169:0/2038396685 shutdown_connections 2024-03-28T01:07:34.815 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:34.812+0000 7f9a290c6640 1 -- 172.21.15.169:0/2038396685 wait complete. 2024-03-28T01:07:35.675 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:07:35.675 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:07:36.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.045+0000 7f3bd359e640 1 -- 172.21.15.169:0/1588865286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bd4101f70 msgr2=0x7f3bd4108800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:36.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.045+0000 7f3bd359e640 1 --2- 172.21.15.169:0/1588865286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bd4101f70 0x7f3bd4108800 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800d910 tx=0x7f3bc802dbf0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.058 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:35 smithi169 ceph-mon[32274]: pgmap v683: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 563 MiB/s wr, 800 op/s 2024-03-28T01:07:36.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.051+0000 7f3bd359e640 1 -- 172.21.15.169:0/1588865286 shutdown_connections 2024-03-28T01:07:36.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.051+0000 7f3bd359e640 1 --2- 172.21.15.169:0/1588865286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bd4101f70 0x7f3bd4108800 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.051+0000 7f3bd359e640 1 --2- 172.21.15.169:0/1588865286 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bd41015b0 0x7f3bd4101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.051+0000 7f3bd359e640 1 --2- 172.21.15.169:0/1588865286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3bd41003c0 0x7f3bd41007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.051+0000 7f3bd359e640 1 -- 172.21.15.169:0/1588865286 >> 172.21.15.169:0/1588865286 conn(0x7f3bd40fb970 msgr2=0x7f3bd40fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:36.059 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.053+0000 7f3bd359e640 1 -- 172.21.15.169:0/1588865286 shutdown_connections 2024-03-28T01:07:36.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.053+0000 7f3bd359e640 1 -- 172.21.15.169:0/1588865286 wait complete. 2024-03-28T01:07:36.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.054+0000 7f3bd359e640 1 Processor -- start 2024-03-28T01:07:36.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.055+0000 7f3bd359e640 1 -- start start 2024-03-28T01:07:36.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.055+0000 7f3bd359e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bd41003c0 0x7f3bd4195700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.055+0000 7f3bd359e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bd41015b0 0x7f3bd4195c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.055+0000 7f3bd359e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3bd4101f70 0x7f3bd419ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.055+0000 7f3bd359e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd410b040 con 0x7f3bd4101f70 2024-03-28T01:07:36.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.055+0000 7f3bd359e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd410aec0 con 0x7f3bd41015b0 2024-03-28T01:07:36.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.055+0000 7f3bd359e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd410b1c0 con 0x7f3bd41003c0 2024-03-28T01:07:36.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.055+0000 7f3bd2d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3bd4101f70 0x7f3bd419ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:36.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.055+0000 7f3bd2d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3bd4101f70 0x7f3bd419ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:55274/0 (socket says 172.21.15.169:55274) 2024-03-28T01:07:36.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.055+0000 7f3bd2d9d640 1 -- 172.21.15.169:0/249842690 learned_addr learned my addr 172.21.15.169:0/249842690 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:36.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.056+0000 7f3bd2d9d640 1 -- 172.21.15.169:0/249842690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bd41003c0 msgr2=0x7f3bd4195700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:36.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.056+0000 7f3bd2d9d640 1 --2- 172.21.15.169:0/249842690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bd41003c0 0x7f3bd4195700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.056+0000 7f3bd2d9d640 1 -- 172.21.15.169:0/249842690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bd41015b0 msgr2=0x7f3bd4195c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:07:36.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.056+0000 7f3bd2d9d640 1 --2- 172.21.15.169:0/249842690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bd41015b0 0x7f3bd4195c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.056+0000 7f3bd2d9d640 1 -- 172.21.15.169:0/249842690 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd419d390 con 0x7f3bd4101f70 2024-03-28T01:07:36.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.056+0000 7f3bd2d9d640 1 --2- 172.21.15.169:0/249842690 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3bd4101f70 0x7f3bd419ccf0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8005ab0 tx=0x7f3bc80053c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:36.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.071+0000 7f3bc37fe640 1 -- 172.21.15.169:0/249842690 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc8004020 con 0x7f3bd4101f70 2024-03-28T01:07:36.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.071+0000 7f3bd359e640 1 -- 172.21.15.169:0/249842690 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd419d620 con 0x7f3bd4101f70 2024-03-28T01:07:36.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.072+0000 7f3bd359e640 1 -- 172.21.15.169:0/249842690 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd419db30 con 0x7f3bd4101f70 2024-03-28T01:07:36.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.072+0000 7f3bc37fe640 1 -- 172.21.15.169:0/249842690 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bc80041c0 con 0x7f3bd4101f70 2024-03-28T01:07:36.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.072+0000 7f3bc37fe640 1 -- 172.21.15.169:0/249842690 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc802e5f0 con 0x7f3bd4101f70 2024-03-28T01:07:36.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.080+0000 7f3bdb981740 1 Processor -- start 2024-03-28T01:07:36.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.081+0000 7f3bc37fe640 1 -- 172.21.15.169:0/249842690 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3bc8045030 con 0x7f3bd4101f70 2024-03-28T01:07:36.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.082+0000 7f3bdb981740 1 -- start start 2024-03-28T01:07:36.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.082+0000 7f3bdb981740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ed3a341fa0 0x55ed3a3423c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.082+0000 7f3bdb981740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a342990 0x55ed3a347490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.082+0000 7f3bdb981740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a347b00 0x55ed3a349f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.082+0000 7f3bdb981740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed3a2a73d0 con 0x55ed3a341fa0 2024-03-28T01:07:36.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.082+0000 7f3bdb981740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed3a24fad0 con 0x55ed3a342990 2024-03-28T01:07:36.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.082+0000 7f3bdb981740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed3a2a3470 con 0x55ed3a347b00 2024-03-28T01:07:36.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.083+0000 7f3bc37fe640 1 --2- 172.21.15.169:0/249842690 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3bbc0a2980 0x7f3bbc0a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.083+0000 7f3bc37fe640 1 -- 172.21.15.169:0/249842690 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f3bc80f0040 con 0x7f3bd4101f70 2024-03-28T01:07:36.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.084+0000 7f3bd259c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a342990 0x55ed3a347490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:36.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.084+0000 7f3bd259c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a342990 0x55ed3a347490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:50946/0 (socket says 172.21.15.169:50946) 2024-03-28T01:07:36.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.084+0000 7f3bd259c640 1 -- 172.21.15.169:0/1140405119 learned_addr learned my addr 172.21.15.169:0/1140405119 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:36.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.084+0000 7f3bd259c640 1 -- 172.21.15.169:0/1140405119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a347b00 msgr2=0x55ed3a349f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:07:36.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.084+0000 7f3bd259c640 1 --2- 172.21.15.169:0/1140405119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a347b00 0x55ed3a349f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.084+0000 7f3bd259c640 1 -- 172.21.15.169:0/1140405119 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ed3a341fa0 msgr2=0x55ed3a3423c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:07:36.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.084+0000 7f3bd259c640 1 --2- 172.21.15.169:0/1140405119 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ed3a341fa0 0x55ed3a3423c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.084+0000 7f3bd259c640 1 -- 172.21.15.169:0/1140405119 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed3a0857d0 con 0x55ed3a342990 2024-03-28T01:07:36.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.084+0000 7f3bd259c640 1 --2- 172.21.15.169:0/1140405119 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a342990 0x55ed3a347490 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3bc40097b0 tx=0x7f3bc4009c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c44a2286ad35082 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:36.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.085+0000 7f3bc0ff9640 1 -- 172.21.15.169:0/1140405119 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4015020 con 0x55ed3a342990 2024-03-28T01:07:36.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.086+0000 7f3bc0ff9640 1 -- 172.21.15.169:0/1140405119 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bc4011420 con 0x55ed3a342990 2024-03-28T01:07:36.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.088+0000 7f3bd2d9d640 1 --2- 172.21.15.169:0/249842690 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3bbc0a2980 0x7f3bbc0a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:36.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.088+0000 7f3bc0ff9640 1 -- 172.21.15.169:0/1140405119 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc40105e0 con 0x55ed3a342990 2024-03-28T01:07:36.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.089+0000 7f3bdb981740 1 -- 172.21.15.169:0/1140405119 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a342990 msgr2=0x55ed3a347490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:36.096 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.089+0000 7f3bdb981740 1 --2- 172.21.15.169:0/1140405119 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a342990 0x55ed3a347490 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3bc40097b0 tx=0x7f3bc4009c70 comp rx=0 tx=0).stop 2024-03-28T01:07:36.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.095+0000 7f3bdb981740 1 -- 172.21.15.169:0/1140405119 shutdown_connections 2024-03-28T01:07:36.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.095+0000 7f3bdb981740 1 --2- 172.21.15.169:0/1140405119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a347b00 0x55ed3a349f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.095+0000 7f3bdb981740 1 --2- 172.21.15.169:0/1140405119 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a342990 0x55ed3a347490 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.095+0000 7f3bdb981740 1 --2- 172.21.15.169:0/1140405119 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ed3a341fa0 0x55ed3a3423c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.095+0000 7f3bdb981740 1 -- 172.21.15.169:0/1140405119 >> 172.21.15.169:0/1140405119 conn(0x55ed3a33d510 msgr2=0x55ed3a33d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:36.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.100+0000 7f3bd2d9d640 1 --2- 172.21.15.169:0/249842690 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3bbc0a2980 0x7f3bbc0a4e40 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8004bb0 tx=0x7f3bc8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:36.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.100+0000 7f3bdb981740 1 -- 172.21.15.169:0/1140405119 shutdown_connections 2024-03-28T01:07:36.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.100+0000 7f3bdb981740 1 -- 172.21.15.169:0/1140405119 wait complete. 2024-03-28T01:07:36.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.100+0000 7f3bdb981740 1 -- start start 2024-03-28T01:07:36.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.101+0000 7f3bdb981740 1 Processor -- start 2024-03-28T01:07:36.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.101+0000 7f3bdb981740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ed3a341fa0 0x55ed3a34c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.101+0000 7f3bdb981740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a342990 0x55ed3a34ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.101+0000 7f3bdb981740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a347b00 0x55ed3a353a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.101+0000 7f3bdb981740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed3a34aab0 con 0x55ed3a341fa0 2024-03-28T01:07:36.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.101+0000 7f3bdb981740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed3a263060 con 0x55ed3a347b00 2024-03-28T01:07:36.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.101+0000 7f3bdb981740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed3a34ac30 con 0x55ed3a342990 2024-03-28T01:07:36.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.101+0000 7f3bdb981740 10 client.0 _ll_register_callbacks cb 0x55ed3a2456a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:36.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.103+0000 7f3bd259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a342990 0x55ed3a34ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:36.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.103+0000 7f3bd259c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a342990 0x55ed3a34ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:44734/0 (socket says 172.21.15.169:44734) 2024-03-28T01:07:36.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.103+0000 7f3bd259c640 1 -- 172.21.15.169:0/3429174046 learned_addr learned my addr 172.21.15.169:0/3429174046 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:36.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.104+0000 7f3bd1d9b640 1 --2- 172.21.15.169:0/3429174046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a347b00 0x55ed3a353a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:36.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:35 smithi174 ceph-mon[36060]: pgmap v683: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 563 MiB/s wr, 800 op/s 2024-03-28T01:07:36.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.105+0000 7f3bd1d9b640 1 -- 172.21.15.169:0/3429174046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a342990 msgr2=0x55ed3a34ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:36.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.105+0000 7f3bd1d9b640 1 --2- 172.21.15.169:0/3429174046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a342990 0x55ed3a34ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.105+0000 7f3bd1d9b640 1 -- 172.21.15.169:0/3429174046 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ed3a341fa0 msgr2=0x55ed3a34c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:36.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.105+0000 7f3bd1d9b640 1 --2- 172.21.15.169:0/3429174046 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ed3a341fa0 0x55ed3a34c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.105+0000 7f3bd1d9b640 1 -- 172.21.15.169:0/3429174046 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed39fbb3f0 con 0x55ed3a347b00 2024-03-28T01:07:36.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.105+0000 7f3bd1d9b640 1 --2- 172.21.15.169:0/3429174046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a347b00 0x55ed3a353a40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4196820 tx=0x7f3bcc00da40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62527b13a0d43b3c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:36.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.110+0000 7f3bb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:07:36.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.110+0000 7f3bb9ffb640 1 -- 172.21.15.169:0/3429174046 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc004d60 con 0x55ed3a347b00 2024-03-28T01:07:36.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.111+0000 7f3bdb981740 1 -- 172.21.15.169:0/3429174046 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3bc4011420 con 0x55ed3a347b00 2024-03-28T01:07:36.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.111+0000 7f3bb9ffb640 1 -- 172.21.15.169:0/3429174046 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f3bcc00f6e0 con 0x55ed3a347b00 2024-03-28T01:07:36.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.111+0000 7f3bb9ffb640 1 -- 172.21.15.169:0/3429174046 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3bcc020050 con 0x55ed3a347b00 2024-03-28T01:07:36.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.112+0000 7f3bdb981740 10 client.27368 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:36.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.112+0000 7f3bdb981740 1 -- 172.21.15.169:0/3429174046 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ed39f7b0f0 con 0x55ed3a347b00 2024-03-28T01:07:36.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.112+0000 7f3bb9ffb640 1 -- 172.21.15.169:0/3429174046 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bc4011420 con 0x55ed3a347b00 2024-03-28T01:07:36.115 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.112+0000 7f3bb9ffb640 1 -- 172.21.15.169:0/3429174046 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f3bcc006800 con 0x55ed3a347b00 2024-03-28T01:07:36.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.112+0000 7f3bdb981740 10 client.27368 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:36.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.112+0000 7f3bdb981740 10 client.27368 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:07:36.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.112+0000 7f3bdb981740 20 client.27368 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:36.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.112+0000 7f3bdb981740 1 --2- 172.21.15.169:0/3429174046 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ed3a3583f0 0x55ed3a35a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.112+0000 7f3bdb981740 4 client.27368 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:07:36.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.112+0000 7f3bdb981740 1 -- 172.21.15.169:0/3429174046 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ed3a2b4910 con 0x55ed3a3583f0 2024-03-28T01:07:36.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:35 smithi177 ceph-mon[36206]: pgmap v683: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 563 MiB/s wr, 800 op/s 2024-03-28T01:07:36.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.232+0000 7f3bd259c640 1 --2- 172.21.15.169:0/3429174046 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ed3a3583f0 0x55ed3a35a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:36.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.243+0000 7f3bd259c640 1 --2- 172.21.15.169:0/3429174046 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ed3a3583f0 0x55ed3a35a850 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1c341d82d25438e server_cookie=37877f19e335c7c6 in_seq=0 out_seq=0 2024-03-28T01:07:36.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.243+0000 7f3bb9ffb640 10 client.27368 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:36.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.250+0000 7f3bb9ffb640 1 -- 172.21.15.169:0/3429174046 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ed3a2b4910 con 0x55ed3a3583f0 2024-03-28T01:07:36.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.250+0000 7f3bb9ffb640 10 client.27368 handle_command_reply: tid=0 2024-03-28T01:07:36.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.250+0000 7f3bdb981740 1 client.27368 shutdown 2024-03-28T01:07:36.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.252+0000 7f3bdb981740 1 -- 172.21.15.169:0/3429174046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a347b00 msgr2=0x55ed3a353a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:36.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.252+0000 7f3bdb981740 1 --2- 172.21.15.169:0/3429174046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a347b00 0x55ed3a353a40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4196820 tx=0x7f3bcc00da40 comp rx=0 tx=0).stop 2024-03-28T01:07:36.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.254+0000 7f3bdb981740 1 -- 172.21.15.169:0/3429174046 shutdown_connections 2024-03-28T01:07:36.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.254+0000 7f3bdb981740 1 --2- 172.21.15.169:0/3429174046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a347b00 0x55ed3a353a40 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.254+0000 7f3bdb981740 1 --2- 172.21.15.169:0/3429174046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a342990 0x55ed3a34ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.254+0000 7f3bdb981740 1 --2- 172.21.15.169:0/3429174046 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ed3a3583f0 0x55ed3a35a850 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.254+0000 7f3bdb981740 1 --2- 172.21.15.169:0/3429174046 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ed3a341fa0 0x55ed3a34c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.254+0000 7f3bdb981740 1 -- 172.21.15.169:0/3429174046 >> 172.21.15.169:0/3429174046 conn(0x55ed3a33d510 msgr2=0x55ed3a33c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:36.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.254+0000 7f3bb9ffb640 0 client.27368 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:36.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.254+0000 7f3bdb981740 1 -- 172.21.15.169:0/3429174046 shutdown_connections 2024-03-28T01:07:36.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.254+0000 7f3bdb981740 1 -- 172.21.15.169:0/3429174046 wait complete. 2024-03-28T01:07:36.259 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.256+0000 7f3bdb981740 20 client.27368 trim_cache size 0 max 16384 2024-03-28T01:07:36.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.307+0000 7f3bdb981740 1 Processor -- start 2024-03-28T01:07:36.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.308+0000 7f3bdb981740 1 -- start start 2024-03-28T01:07:36.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.309+0000 7f3bdb981740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a341fa0 0x55ed3a33bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.309+0000 7f3bdb981740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ed3a342990 0x55ed3a353a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.309+0000 7f3bdb981740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a347b00 0x55ed3a33c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.309+0000 7f3bdb981740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb4012060 con 0x55ed3a342990 2024-03-28T01:07:36.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.309+0000 7f3bdb981740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb400fb60 con 0x55ed3a341fa0 2024-03-28T01:07:36.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.309+0000 7f3bdb981740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb400dc00 con 0x55ed3a347b00 2024-03-28T01:07:36.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.309+0000 7f3bd1d9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a347b00 0x55ed3a33c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:36.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.309+0000 7f3bd1d9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a347b00 0x55ed3a33c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:44746/0 (socket says 172.21.15.169:44746) 2024-03-28T01:07:36.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.309+0000 7f3bd1d9b640 1 -- 172.21.15.169:0/1816412785 learned_addr learned my addr 172.21.15.169:0/1816412785 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:36.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.310+0000 7f3bd1d9b640 1 --2- 172.21.15.169:0/1816412785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a341fa0 0x55ed3a33bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:36.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.310+0000 7f3bd1d9b640 1 -- 172.21.15.169:0/1816412785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a341fa0 msgr2=0x55ed3a33bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:36.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.310+0000 7f3bd1d9b640 1 --2- 172.21.15.169:0/1816412785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a341fa0 0x55ed3a33bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.310+0000 7f3bd1d9b640 1 -- 172.21.15.169:0/1816412785 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ed3a342990 msgr2=0x55ed3a353a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:36.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.310+0000 7f3bd259c640 1 --2- 172.21.15.169:0/1816412785 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ed3a342990 0x55ed3a353a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:36.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.310+0000 7f3bd1d9b640 1 --2- 172.21.15.169:0/1816412785 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ed3a342990 0x55ed3a353a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.310+0000 7f3bd1d9b640 1 -- 172.21.15.169:0/1816412785 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed3a33e480 con 0x55ed3a347b00 2024-03-28T01:07:36.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.310+0000 7f3bd1d9b640 1 --2- 172.21.15.169:0/1816412785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a347b00 0x55ed3a33c460 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc00b300 tx=0x7f3bcc00bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5973fbee359f1ab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:36.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.320+0000 7f3bbbfff640 1 -- 172.21.15.169:0/1816412785 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc029070 con 0x55ed3a347b00 2024-03-28T01:07:36.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.320+0000 7f3bbbfff640 1 -- 172.21.15.169:0/1816412785 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bcc004d50 con 0x55ed3a347b00 2024-03-28T01:07:36.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.320+0000 7f3bbbfff640 1 -- 172.21.15.169:0/1816412785 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc01d040 con 0x55ed3a347b00 2024-03-28T01:07:36.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.321+0000 7f3bdb981740 1 -- 172.21.15.169:0/1816412785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a347b00 msgr2=0x55ed3a33c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:36.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.321+0000 7f3bdb981740 1 --2- 172.21.15.169:0/1816412785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a347b00 0x55ed3a33c460 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc00b300 tx=0x7f3bcc00bee0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.323+0000 7f3bdb981740 1 -- 172.21.15.169:0/1816412785 shutdown_connections 2024-03-28T01:07:36.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.323+0000 7f3bdb981740 1 --2- 172.21.15.169:0/1816412785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a347b00 0x55ed3a33c460 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.323+0000 7f3bdb981740 1 --2- 172.21.15.169:0/1816412785 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ed3a342990 0x55ed3a353a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.323+0000 7f3bdb981740 1 --2- 172.21.15.169:0/1816412785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a341fa0 0x55ed3a33bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.323+0000 7f3bdb981740 1 -- 172.21.15.169:0/1816412785 >> 172.21.15.169:0/1816412785 conn(0x55ed3a33d510 msgr2=0x55ed3a343df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:36.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.327+0000 7f3bdb981740 1 -- 172.21.15.169:0/1816412785 shutdown_connections 2024-03-28T01:07:36.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.327+0000 7f3bdb981740 1 -- 172.21.15.169:0/1816412785 wait complete. 2024-03-28T01:07:36.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.328+0000 7f3bdb981740 1 -- start start 2024-03-28T01:07:36.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.328+0000 7f3bdb981740 1 Processor -- start 2024-03-28T01:07:36.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.328+0000 7f3bdb981740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a341fa0 0x55ed3a34fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.328+0000 7f3bdb981740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ed3a342990 0x55ed3a3500d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.328+0000 7f3bdb981740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a347b00 0x55ed3a3530f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.328+0000 7f3bdb981740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb400e200 con 0x55ed3a342990 2024-03-28T01:07:36.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.328+0000 7f3bdb981740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb400f500 con 0x55ed3a341fa0 2024-03-28T01:07:36.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.328+0000 7f3bdb981740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb400e840 con 0x55ed3a347b00 2024-03-28T01:07:36.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.328+0000 7f3bdb981740 10 client.0 _ll_register_callbacks cb 0x7f3bcc00b390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:36.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.329+0000 7f3bd1d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a341fa0 0x55ed3a34fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:36.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.329+0000 7f3bd1d9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a347b00 0x55ed3a3530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:36.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.329+0000 7f3bd1d9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a347b00 0x55ed3a3530f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:44752/0 (socket says 172.21.15.169:44752) 2024-03-28T01:07:36.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.329+0000 7f3bd1d9b640 1 -- 172.21.15.169:0/296120768 learned_addr learned my addr 172.21.15.169:0/296120768 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:36.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.329+0000 7f3bd1d9b640 1 -- 172.21.15.169:0/296120768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a341fa0 msgr2=0x55ed3a34fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:36.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.329+0000 7f3bd1d9b640 1 --2- 172.21.15.169:0/296120768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a341fa0 0x55ed3a34fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.329+0000 7f3bd1d9b640 1 -- 172.21.15.169:0/296120768 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ed3a342990 msgr2=0x55ed3a3500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:36.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.329+0000 7f3bd1d9b640 1 --2- 172.21.15.169:0/296120768 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ed3a342990 0x55ed3a3500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.329+0000 7f3bd1d9b640 1 -- 172.21.15.169:0/296120768 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed3a3536e0 con 0x55ed3a347b00 2024-03-28T01:07:36.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.331+0000 7f3bd1d9b640 1 --2- 172.21.15.169:0/296120768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a347b00 0x55ed3a3530f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc0176f0 tx=0x7f3bcc00b5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2feedfa5977f0ace server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:36.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.361+0000 7f3bb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:07:36.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.361+0000 7f3bb9ffb640 1 -- 172.21.15.169:0/296120768 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc029040 con 0x55ed3a347b00 2024-03-28T01:07:36.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.361+0000 7f3bb9ffb640 1 -- 172.21.15.169:0/296120768 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f3bcc00f9c0 con 0x55ed3a347b00 2024-03-28T01:07:36.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.361+0000 7f3bdb981740 1 -- 172.21.15.169:0/296120768 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3bcc004d50 con 0x55ed3a347b00 2024-03-28T01:07:36.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.367+0000 7f3bb9ffb640 1 -- 172.21.15.169:0/296120768 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3bcc03f050 con 0x55ed3a347b00 2024-03-28T01:07:36.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.367+0000 7f3bdb981740 10 client.27304 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:36.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.367+0000 7f3bdb981740 1 -- 172.21.15.169:0/296120768 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ed3a3c57e0 con 0x55ed3a347b00 2024-03-28T01:07:36.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.372+0000 7f3bb9ffb640 1 -- 172.21.15.169:0/296120768 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bcc004d50 con 0x55ed3a347b00 2024-03-28T01:07:36.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.372+0000 7f3bb9ffb640 1 -- 172.21.15.169:0/296120768 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f3bcc0067f0 con 0x55ed3a347b00 2024-03-28T01:07:36.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.372+0000 7f3bdb981740 10 client.27304 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:36.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.372+0000 7f3bdb981740 10 client.27304 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:07:36.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.372+0000 7f3bdb981740 20 client.27304 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:36.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.372+0000 7f3bdb981740 1 --2- 172.21.15.169:0/296120768 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ed3a3c6020 0x55ed3a3c8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.372+0000 7f3bdb981740 4 client.27304 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:07:36.376 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.372+0000 7f3bdb981740 1 -- 172.21.15.169:0/296120768 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ed3a2aec60 con 0x55ed3a3c6020 2024-03-28T01:07:36.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.376+0000 7f3bd259c640 1 --2- 172.21.15.169:0/296120768 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ed3a3c6020 0x55ed3a3c8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:36.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.390+0000 7f3bd259c640 1 --2- 172.21.15.169:0/296120768 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ed3a3c6020 0x55ed3a3c8440 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd413473d2b3f8f9 server_cookie=c1832fe511715e2e in_seq=0 out_seq=0 2024-03-28T01:07:36.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.391+0000 7f3bb9ffb640 10 client.27304 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:36.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.395+0000 7f3bb9ffb640 1 -- 172.21.15.169:0/296120768 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55ed3a2aec60 con 0x55ed3a3c6020 2024-03-28T01:07:36.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.395+0000 7f3bb9ffb640 10 client.27304 handle_command_reply: tid=0 2024-03-28T01:07:36.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.395+0000 7f3bdb981740 1 client.27304 shutdown 2024-03-28T01:07:36.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.402+0000 7f3bdb981740 1 -- 172.21.15.169:0/296120768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a347b00 msgr2=0x55ed3a3530f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:36.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.402+0000 7f3bdb981740 1 --2- 172.21.15.169:0/296120768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a347b00 0x55ed3a3530f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc0176f0 tx=0x7f3bcc00b5a0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.402+0000 7f3bdb981740 1 -- 172.21.15.169:0/296120768 shutdown_connections 2024-03-28T01:07:36.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.402+0000 7f3bdb981740 1 --2- 172.21.15.169:0/296120768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed3a347b00 0x55ed3a3530f0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.402+0000 7f3bdb981740 1 --2- 172.21.15.169:0/296120768 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55ed3a3c6020 0x55ed3a3c8440 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.402+0000 7f3bdb981740 1 --2- 172.21.15.169:0/296120768 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55ed3a342990 0x55ed3a3500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.402+0000 7f3bdb981740 1 --2- 172.21.15.169:0/296120768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed3a341fa0 0x55ed3a34fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.402+0000 7f3bdb981740 1 -- 172.21.15.169:0/296120768 >> 172.21.15.169:0/296120768 conn(0x55ed3a33d510 msgr2=0x55ed3a340770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:36.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.402+0000 7f3bb9ffb640 0 client.27304 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:36.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.402+0000 7f3bdb981740 1 -- 172.21.15.169:0/296120768 shutdown_connections 2024-03-28T01:07:36.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.402+0000 7f3bdb981740 1 -- 172.21.15.169:0/296120768 wait complete. 2024-03-28T01:07:36.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.405+0000 7f3bdb981740 20 client.27304 trim_cache size 0 max 16384 2024-03-28T01:07:36.412 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:07:36.412 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "no active scrubs running", 2024-03-28T01:07:36.412 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": {} 2024-03-28T01:07:36.412 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:07:36.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.418+0000 7f3bd359e640 1 -- 172.21.15.169:0/249842690 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3bbc0a2980 msgr2=0x7f3bbc0a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:36.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.418+0000 7f3bd359e640 1 --2- 172.21.15.169:0/249842690 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3bbc0a2980 0x7f3bbc0a4e40 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8004bb0 tx=0x7f3bc8009260 comp rx=0 tx=0).stop 2024-03-28T01:07:36.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.418+0000 7f3bd359e640 1 -- 172.21.15.169:0/249842690 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3bd4101f70 msgr2=0x7f3bd419ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:36.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.418+0000 7f3bd359e640 1 --2- 172.21.15.169:0/249842690 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3bd4101f70 0x7f3bd419ccf0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8005ab0 tx=0x7f3bc80053c0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.423+0000 7f3bd359e640 1 -- 172.21.15.169:0/249842690 shutdown_connections 2024-03-28T01:07:36.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.423+0000 7f3bd359e640 1 --2- 172.21.15.169:0/249842690 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f3bbc0a2980 0x7f3bbc0a4e40 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.423+0000 7f3bd359e640 1 --2- 172.21.15.169:0/249842690 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3bd4101f70 0x7f3bd419ccf0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.423+0000 7f3bd359e640 1 --2- 172.21.15.169:0/249842690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bd41015b0 0x7f3bd4195c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.423+0000 7f3bd359e640 1 --2- 172.21.15.169:0/249842690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3bd41003c0 0x7f3bd4195700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.423+0000 7f3bd359e640 1 -- 172.21.15.169:0/249842690 >> 172.21.15.169:0/249842690 conn(0x7f3bd40fb970 msgr2=0x7f3bd41048b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:36.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.426+0000 7f3bd359e640 1 -- 172.21.15.169:0/249842690 shutdown_connections 2024-03-28T01:07:36.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.432+0000 7f3bd359e640 1 -- 172.21.15.169:0/249842690 wait complete. 2024-03-28T01:07:36.487 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T01:07:36.488 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T01:07:36.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.819+0000 7fcecd3c4640 1 -- 172.21.15.169:0/3613712777 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcec8108a40 msgr2=0x7fcec8108e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:36.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.819+0000 7fcecd3c4640 1 --2- 172.21.15.169:0/3613712777 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcec8108a40 0x7fcec8108e60 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fceb8009900 tx=0x7fceb802df80 comp rx=0 tx=0).stop 2024-03-28T01:07:36.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.824+0000 7fcecd3c4640 1 -- 172.21.15.169:0/3613712777 shutdown_connections 2024-03-28T01:07:36.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.824+0000 7fcecd3c4640 1 --2- 172.21.15.169:0/3613712777 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcec810a0f0 0x7fcec81109a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.824+0000 7fcecd3c4640 1 --2- 172.21.15.169:0/3613712777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcec8109750 0x7fcec8109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.824+0000 7fcecd3c4640 1 --2- 172.21.15.169:0/3613712777 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcec8108a40 0x7fcec8108e60 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.824+0000 7fcecd3c4640 1 -- 172.21.15.169:0/3613712777 >> 172.21.15.169:0/3613712777 conn(0x7fcec806e510 msgr2=0x7fcec8070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:36.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.829+0000 7fcecd3c4640 1 -- 172.21.15.169:0/3613712777 shutdown_connections 2024-03-28T01:07:36.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.833+0000 7fcecd3c4640 1 -- 172.21.15.169:0/3613712777 wait complete. 2024-03-28T01:07:36.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.834+0000 7fcecd3c4640 1 Processor -- start 2024-03-28T01:07:36.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.834+0000 7fcecd3c4640 1 -- start start 2024-03-28T01:07:36.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.835+0000 7fcecd3c4640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcec8108a40 0x7fcec819e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.835+0000 7fcecd3c4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcec8109750 0x7fcec819e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.835+0000 7fcecd3c4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcec810a0f0 0x7fcec81a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.835+0000 7fcecd3c4640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec81139f0 con 0x7fcec8108a40 2024-03-28T01:07:36.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.835+0000 7fcecd3c4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec8113870 con 0x7fcec8109750 2024-03-28T01:07:36.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.835+0000 7fcec77fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcec810a0f0 0x7fcec81a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:36.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.836+0000 7fcec77fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcec810a0f0 0x7fcec81a56e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:47066/0 (socket says 172.21.15.169:47066) 2024-03-28T01:07:36.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.836+0000 7fcec77fe640 1 -- 172.21.15.169:0/3975665323 learned_addr learned my addr 172.21.15.169:0/3975665323 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:36.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.836+0000 7fcecd3c4640 1 -- 172.21.15.169:0/3975665323 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec8113b70 con 0x7fcec810a0f0 2024-03-28T01:07:36.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.836+0000 7fcec6ffd640 1 --2- 172.21.15.169:0/3975665323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcec8108a40 0x7fcec819e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:36.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.837+0000 7fcec6ffd640 1 -- 172.21.15.169:0/3975665323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcec810a0f0 msgr2=0x7fcec81a56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:36.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.837+0000 7fcec6ffd640 1 --2- 172.21.15.169:0/3975665323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcec810a0f0 0x7fcec81a56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.837+0000 7fcec6ffd640 1 -- 172.21.15.169:0/3975665323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcec8109750 msgr2=0x7fcec819e670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:07:36.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.837+0000 7fcec6ffd640 1 --2- 172.21.15.169:0/3975665323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcec8109750 0x7fcec819e670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:36.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.837+0000 7fcec6ffd640 1 -- 172.21.15.169:0/3975665323 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec81a5d50 con 0x7fcec8108a40 2024-03-28T01:07:36.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.837+0000 7fcec77fe640 1 --2- 172.21.15.169:0/3975665323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcec810a0f0 0x7fcec81a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:07:36.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.837+0000 7fcec6ffd640 1 --2- 172.21.15.169:0/3975665323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcec8108a40 0x7fcec819e130 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fceb8009850 tx=0x7fceb8003960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:36.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.840+0000 7fceaffff640 1 -- 172.21.15.169:0/3975665323 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb8042020 con 0x7fcec8108a40 2024-03-28T01:07:36.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.840+0000 7fcecd3c4640 1 -- 172.21.15.169:0/3975665323 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcec81a5fe0 con 0x7fcec8108a40 2024-03-28T01:07:36.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.840+0000 7fcecd3c4640 1 -- 172.21.15.169:0/3975665323 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcec81a6460 con 0x7fcec8108a40 2024-03-28T01:07:36.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.840+0000 7fceaffff640 1 -- 172.21.15.169:0/3975665323 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fceb8003d20 con 0x7fcec8108a40 2024-03-28T01:07:36.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.841+0000 7fceaffff640 1 -- 172.21.15.169:0/3975665323 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb803e770 con 0x7fcec8108a40 2024-03-28T01:07:36.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.846+0000 7fceaffff640 1 -- 172.21.15.169:0/3975665323 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fceb8045020 con 0x7fcec8108a40 2024-03-28T01:07:36.850 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.847+0000 7fcecd3c4640 1 -- 172.21.15.169:0/3975665323 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb40052c0 con 0x7fcec8108a40 2024-03-28T01:07:36.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.848+0000 7fceaffff640 1 --2- 172.21.15.169:0/3975665323 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fceb00a2a00 0x7fceb00a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:36.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.848+0000 7fceaffff640 1 -- 172.21.15.169:0/3975665323 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fceb80f2b90 con 0x7fcec8108a40 2024-03-28T01:07:36.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.856+0000 7fcec67fc640 1 --2- 172.21.15.169:0/3975665323 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fceb00a2a00 0x7fceb00a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:36.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.859+0000 7fceaffff640 1 -- 172.21.15.169:0/3975665323 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fceb804c020 con 0x7fcec8108a40 2024-03-28T01:07:36.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:36.860+0000 7fcec67fc640 1 --2- 172.21.15.169:0/3975665323 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fceb00a2a00 0x7fceb00a4ec0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fcec819f640 tx=0x7fcebc006c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:37.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.379+0000 7fcecd3c4640 1 -- 172.21.15.169:0/3975665323 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fceb40055b0 con 0x7fcec8108a40 2024-03-28T01:07:37.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.388+0000 7fceaffff640 1 -- 172.21.15.169:0/3975665323 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7fceb80bf830 con 0x7fcec8108a40 2024-03-28T01:07:37.392 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T01:07:37.393 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T01:07:37.393 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T01:07:37.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.394+0000 7fcecd3c4640 1 -- 172.21.15.169:0/3975665323 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fceb00a2a00 msgr2=0x7fceb00a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:37.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.394+0000 7fcecd3c4640 1 --2- 172.21.15.169:0/3975665323 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fceb00a2a00 0x7fceb00a4ec0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fcec819f640 tx=0x7fcebc006c20 comp rx=0 tx=0).stop 2024-03-28T01:07:37.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.394+0000 7fcecd3c4640 1 -- 172.21.15.169:0/3975665323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcec8108a40 msgr2=0x7fcec819e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:37.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.394+0000 7fcecd3c4640 1 --2- 172.21.15.169:0/3975665323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcec8108a40 0x7fcec819e130 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fceb8009850 tx=0x7fceb8003960 comp rx=0 tx=0).stop 2024-03-28T01:07:37.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.397+0000 7fcecd3c4640 1 -- 172.21.15.169:0/3975665323 shutdown_connections 2024-03-28T01:07:37.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.397+0000 7fcecd3c4640 1 --2- 172.21.15.169:0/3975665323 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fceb00a2a00 0x7fceb00a4ec0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.397+0000 7fcecd3c4640 1 --2- 172.21.15.169:0/3975665323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcec810a0f0 0x7fcec81a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.397+0000 7fcecd3c4640 1 --2- 172.21.15.169:0/3975665323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcec8109750 0x7fcec819e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.397+0000 7fcecd3c4640 1 --2- 172.21.15.169:0/3975665323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcec8108a40 0x7fcec819e130 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.397+0000 7fcecd3c4640 1 -- 172.21.15.169:0/3975665323 >> 172.21.15.169:0/3975665323 conn(0x7fcec806e510 msgr2=0x7fcec810cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:37.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.400+0000 7fcecd3c4640 1 -- 172.21.15.169:0/3975665323 shutdown_connections 2024-03-28T01:07:37.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.402+0000 7fcecd3c4640 1 -- 172.21.15.169:0/3975665323 wait complete. 2024-03-28T01:07:37.470 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:07:37.470 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T01:07:37.769 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:37 smithi169 ceph-mon[32274]: pgmap v684: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 748 MiB/s wr, 834 op/s 2024-03-28T01:07:37.769 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:37 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3975665323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:07:37.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.809+0000 7f264d59a640 1 -- 172.21.15.169:0/639935403 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263802ccd0 con 0x7f265010b010 2024-03-28T01:07:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.811+0000 7f2654fcd640 1 -- 172.21.15.169:0/639935403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f265010b010 msgr2=0x7f265010d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:37.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.811+0000 7f2654fcd640 1 --2- 172.21.15.169:0/639935403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f265010b010 0x7f265010d480 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2638009900 tx=0x7f263802bc80 comp rx=0 tx=0).stop 2024-03-28T01:07:37.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.812+0000 7f2654fcd640 1 -- 172.21.15.169:0/639935403 shutdown_connections 2024-03-28T01:07:37.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.812+0000 7f2654fcd640 1 --2- 172.21.15.169:0/639935403 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f265010d9c0 0x7f265010de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.812+0000 7f2654fcd640 1 --2- 172.21.15.169:0/639935403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f265010b010 0x7f265010d480 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.812+0000 7f2654fcd640 1 --2- 172.21.15.169:0/639935403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26501086a0 0x7f265010aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.812+0000 7f2654fcd640 1 -- 172.21.15.169:0/639935403 >> 172.21.15.169:0/639935403 conn(0x7f265006e500 msgr2=0x7f2650070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:37.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.816+0000 7f2654fcd640 1 -- 172.21.15.169:0/639935403 shutdown_connections 2024-03-28T01:07:37.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.818+0000 7f2654fcd640 1 -- 172.21.15.169:0/639935403 wait complete. 2024-03-28T01:07:37.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.819+0000 7f2654fcd640 1 Processor -- start 2024-03-28T01:07:37.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.819+0000 7f2654fcd640 1 -- start start 2024-03-28T01:07:37.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.819+0000 7f2654fcd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26501086a0 0x7f26501a68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:37.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.819+0000 7f2654fcd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f265010b010 0x7f26501a6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:37.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.819+0000 7f2654fcd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f265010d9c0 0x7f26501ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:37.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.819+0000 7f2654fcd640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265011c280 con 0x7f26501086a0 2024-03-28T01:07:37.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.819+0000 7f2654fcd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265011c100 con 0x7f265010b010 2024-03-28T01:07:37.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.819+0000 7f2654fcd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265011c400 con 0x7f265010d9c0 2024-03-28T01:07:37.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.820+0000 7f264dd9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f265010b010 0x7f26501a6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:37.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.821+0000 7f264dd9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f265010b010 0x7f26501a6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:58596/0 (socket says 172.21.15.169:58596) 2024-03-28T01:07:37.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.821+0000 7f264dd9b640 1 -- 172.21.15.169:0/3842597553 learned_addr learned my addr 172.21.15.169:0/3842597553 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:37.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.822+0000 7f264ed9d640 1 --2- 172.21.15.169:0/3842597553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f265010d9c0 0x7f26501ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:37.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.823+0000 7f264e59c640 1 --2- 172.21.15.169:0/3842597553 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26501086a0 0x7f26501a68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:37.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.824+0000 7f264dd9b640 1 -- 172.21.15.169:0/3842597553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f265010d9c0 msgr2=0x7f26501ade20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:37.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.824+0000 7f264dd9b640 1 --2- 172.21.15.169:0/3842597553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f265010d9c0 0x7f26501ade20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.824+0000 7f264dd9b640 1 -- 172.21.15.169:0/3842597553 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26501086a0 msgr2=0x7f26501a68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:37.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.824+0000 7f264dd9b640 1 --2- 172.21.15.169:0/3842597553 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26501086a0 0x7f26501a68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.824+0000 7f264dd9b640 1 -- 172.21.15.169:0/3842597553 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265010ff00 con 0x7f265010b010 2024-03-28T01:07:37.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.824+0000 7f264dd9b640 1 --2- 172.21.15.169:0/3842597553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f265010b010 0x7f26501a6e00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2638000c00 tx=0x7f2638004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:37.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.828+0000 7f26477fe640 1 -- 172.21.15.169:0/3842597553 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2638041070 con 0x7f265010b010 2024-03-28T01:07:37.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.828+0000 7f26477fe640 1 -- 172.21.15.169:0/3842597553 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26380035f0 con 0x7f265010b010 2024-03-28T01:07:37.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.828+0000 7f26477fe640 1 -- 172.21.15.169:0/3842597553 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26380346c0 con 0x7f265010b010 2024-03-28T01:07:37.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.832+0000 7f2654fcd640 1 -- 172.21.15.169:0/3842597553 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2650110190 con 0x7f265010b010 2024-03-28T01:07:37.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.832+0000 7f2654fcd640 1 -- 172.21.15.169:0/3842597553 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2650110600 con 0x7f265010b010 2024-03-28T01:07:37.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.842+0000 7f26477fe640 1 -- 172.21.15.169:0/3842597553 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2638044070 con 0x7f265010b010 2024-03-28T01:07:37.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.843+0000 7f26477fe640 1 --2- 172.21.15.169:0/3842597553 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f263c0a2a00 0x7f263c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:37.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.843+0000 7f26477fe640 1 -- 172.21.15.169:0/3842597553 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f26380f0ad0 con 0x7f265010b010 2024-03-28T01:07:37.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.846+0000 7f264e59c640 1 --2- 172.21.15.169:0/3842597553 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f263c0a2a00 0x7f263c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:37.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.848+0000 7f264e59c640 1 --2- 172.21.15.169:0/3842597553 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f263c0a2a00 0x7f263c0a4ec0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f26480058c0 tx=0x7f264800a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:37.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.849+0000 7f2657bb0740 1 Processor -- start 2024-03-28T01:07:37.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.852+0000 7f2657bb0740 1 -- start start 2024-03-28T01:07:37.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.852+0000 7f2657bb0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55675c93a690 0x55675c93aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:37.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.852+0000 7f2657bb0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c93b080 0x55675c93fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:37.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.852+0000 7f2657bb0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c9401f0 0x55675c942680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:37.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.852+0000 7f2657bb0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55675c8fd610 con 0x55675c93a690 2024-03-28T01:07:37.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.852+0000 7f2657bb0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55675c8a03c0 con 0x55675c9401f0 2024-03-28T01:07:37.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.852+0000 7f2657bb0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55675c848d60 con 0x55675c93b080 2024-03-28T01:07:37.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.853+0000 7f264dd9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c93b080 0x55675c93fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:37.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.853+0000 7f264dd9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c93b080 0x55675c93fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:47092/0 (socket says 172.21.15.169:47092) 2024-03-28T01:07:37.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.853+0000 7f264dd9b640 1 -- 172.21.15.169:0/4224257157 learned_addr learned my addr 172.21.15.169:0/4224257157 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:37.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.853+0000 7f264dd9b640 1 -- 172.21.15.169:0/4224257157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c9401f0 msgr2=0x55675c942680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:07:37.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.853+0000 7f264dd9b640 1 --2- 172.21.15.169:0/4224257157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c9401f0 0x55675c942680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.853+0000 7f264dd9b640 1 -- 172.21.15.169:0/4224257157 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55675c93a690 msgr2=0x55675c93aab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:07:37.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.853+0000 7f264dd9b640 1 --2- 172.21.15.169:0/4224257157 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55675c93a690 0x55675c93aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.853+0000 7f264dd9b640 1 -- 172.21.15.169:0/4224257157 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55675c5b43f0 con 0x55675c93b080 2024-03-28T01:07:37.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.854+0000 7f264dd9b640 1 --2- 172.21.15.169:0/4224257157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c93b080 0x55675c93fb80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f26380c52e0 tx=0x7f2638003eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef1802d96669e6c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:37.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.856+0000 7f2644ff9640 1 -- 172.21.15.169:0/4224257157 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26380bd770 con 0x55675c93b080 2024-03-28T01:07:37.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.856+0000 7f2644ff9640 1 -- 172.21.15.169:0/4224257157 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26380bd070 con 0x55675c93b080 2024-03-28T01:07:37.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.856+0000 7f2644ff9640 1 -- 172.21.15.169:0/4224257157 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2638003b10 con 0x55675c93b080 2024-03-28T01:07:37.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.857+0000 7f2657bb0740 1 -- 172.21.15.169:0/4224257157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c93b080 msgr2=0x55675c93fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:37.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.857+0000 7f2657bb0740 1 --2- 172.21.15.169:0/4224257157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c93b080 0x55675c93fb80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f26380c52e0 tx=0x7f2638003eb0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.858+0000 7f2657bb0740 1 -- 172.21.15.169:0/4224257157 shutdown_connections 2024-03-28T01:07:37.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.858+0000 7f2657bb0740 1 --2- 172.21.15.169:0/4224257157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c9401f0 0x55675c942680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.858+0000 7f2657bb0740 1 --2- 172.21.15.169:0/4224257157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c93b080 0x55675c93fb80 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.858+0000 7f2657bb0740 1 --2- 172.21.15.169:0/4224257157 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55675c93a690 0x55675c93aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.858+0000 7f2657bb0740 1 -- 172.21.15.169:0/4224257157 >> 172.21.15.169:0/4224257157 conn(0x55675c935c00 msgr2=0x55675c936000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:37.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.864+0000 7f2657bb0740 1 -- 172.21.15.169:0/4224257157 shutdown_connections 2024-03-28T01:07:37.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.866+0000 7f2657bb0740 1 -- 172.21.15.169:0/4224257157 wait complete. 2024-03-28T01:07:37.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.866+0000 7f2657bb0740 1 -- start start 2024-03-28T01:07:37.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.866+0000 7f2657bb0740 1 Processor -- start 2024-03-28T01:07:37.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.867+0000 7f2657bb0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55675c93a690 0x55675c966460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:37.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.867+0000 7f2657bb0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c9401f0 0x55675c9669a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:37.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.867+0000 7f2657bb0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c96b9d0 0x55675c96ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:37.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.867+0000 7f2657bb0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55675c85c240 con 0x55675c93a690 2024-03-28T01:07:37.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.867+0000 7f2657bb0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55675c89c460 con 0x55675c96b9d0 2024-03-28T01:07:37.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.867+0000 7f2657bb0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55675c8fd610 con 0x55675c9401f0 2024-03-28T01:07:37.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.867+0000 7f2657bb0740 10 client.0 _ll_register_callbacks cb 0x55675c85bc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:37.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.869+0000 7f264dd9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c9401f0 0x55675c9669a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:37.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.869+0000 7f264dd9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c9401f0 0x55675c9669a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:47096/0 (socket says 172.21.15.169:47096) 2024-03-28T01:07:37.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.869+0000 7f264dd9b640 1 -- 172.21.15.169:0/273243194 learned_addr learned my addr 172.21.15.169:0/273243194 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:37.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.870+0000 7f264dd9b640 1 -- 172.21.15.169:0/273243194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c96b9d0 msgr2=0x55675c96ddc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:07:37.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.870+0000 7f264dd9b640 1 --2- 172.21.15.169:0/273243194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c96b9d0 0x55675c96ddc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.870+0000 7f264dd9b640 1 -- 172.21.15.169:0/273243194 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55675c93a690 msgr2=0x55675c966460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:37.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.870+0000 7f264dd9b640 1 --2- 172.21.15.169:0/273243194 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55675c93a690 0x55675c966460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.870+0000 7f264dd9b640 1 -- 172.21.15.169:0/273243194 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55675c67e7d0 con 0x55675c9401f0 2024-03-28T01:07:37.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.870+0000 7f264dd9b640 1 --2- 172.21.15.169:0/273243194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c9401f0 0x55675c9669a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2638005820 tx=0x7f26380b98e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afb7a5cac4ef8f1d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:37.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.871+0000 7f2635ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:07:37.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.872+0000 7f2635ffb640 1 -- 172.21.15.169:0/273243194 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26380c5810 con 0x55675c9401f0 2024-03-28T01:07:37.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.872+0000 7f2635ffb640 1 -- 172.21.15.169:0/273243194 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f26380bb370 con 0x55675c9401f0 2024-03-28T01:07:37.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.877+0000 7f2657bb0740 1 -- 172.21.15.169:0/273243194 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f26380bd070 con 0x55675c9401f0 2024-03-28T01:07:37.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.881+0000 7f2635ffb640 1 -- 172.21.15.169:0/273243194 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26380bd770 con 0x55675c9401f0 2024-03-28T01:07:37.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.881+0000 7f2657bb0740 10 client.27316 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:37.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.881+0000 7f2657bb0740 1 -- 172.21.15.169:0/273243194 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55675c971130 con 0x55675c9401f0 2024-03-28T01:07:37.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.890+0000 7f2635ffb640 1 -- 172.21.15.169:0/273243194 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26380ca070 con 0x55675c9401f0 2024-03-28T01:07:37.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.890+0000 7f2635ffb640 1 -- 172.21.15.169:0/273243194 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f26380dbb70 con 0x55675c9401f0 2024-03-28T01:07:37.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.891+0000 7f2657bb0740 10 client.27316 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:37.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.891+0000 7f2657bb0740 10 client.27316 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:07:37.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.891+0000 7f2657bb0740 20 client.27316 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:37.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.891+0000 7f2657bb0740 1 --2- 172.21.15.169:0/273243194 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55675c972750 0x55675c974b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:37.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.891+0000 7f2657bb0740 4 client.27316 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:07:37.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.891+0000 7f2657bb0740 1 -- 172.21.15.169:0/273243194 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55675c8ad900 con 0x55675c972750 2024-03-28T01:07:37.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.893+0000 7f264e59c640 1 --2- 172.21.15.169:0/273243194 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55675c972750 0x55675c974b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:37.921 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.917+0000 7f264e59c640 1 --2- 172.21.15.169:0/273243194 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55675c972750 0x55675c974b60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f8dd58fc3488bfe6 server_cookie=af2e836470b1296f in_seq=0 out_seq=0 2024-03-28T01:07:37.921 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.917+0000 7f2635ffb640 10 client.27316 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:07:37.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.919+0000 7f2635ffb640 1 -- 172.21.15.169:0/273243194 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55675c8ad900 con 0x55675c972750 2024-03-28T01:07:37.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.919+0000 7f2635ffb640 10 client.27316 handle_command_reply: tid=0 2024-03-28T01:07:37.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.919+0000 7f2657bb0740 1 client.27316 shutdown 2024-03-28T01:07:37.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.931+0000 7f2657bb0740 1 -- 172.21.15.169:0/273243194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c9401f0 msgr2=0x55675c9669a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:37.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.931+0000 7f2657bb0740 1 --2- 172.21.15.169:0/273243194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c9401f0 0x55675c9669a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2638005820 tx=0x7f26380b98e0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.931+0000 7f2657bb0740 1 -- 172.21.15.169:0/273243194 shutdown_connections 2024-03-28T01:07:37.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.931+0000 7f2657bb0740 1 --2- 172.21.15.169:0/273243194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c96b9d0 0x55675c96ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.931+0000 7f2657bb0740 1 --2- 172.21.15.169:0/273243194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c9401f0 0x55675c9669a0 secure :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2638005820 tx=0x7f26380b98e0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.931+0000 7f2657bb0740 1 --2- 172.21.15.169:0/273243194 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55675c972750 0x55675c974b60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.931+0000 7f2657bb0740 1 --2- 172.21.15.169:0/273243194 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55675c93a690 0x55675c966460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:37.934 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.931+0000 7f2657bb0740 1 -- 172.21.15.169:0/273243194 >> 172.21.15.169:0/273243194 conn(0x55675c935c00 msgr2=0x55675c934e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:37.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.931+0000 7f2635ffb640 0 client.27316 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:07:37.938 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.934+0000 7f2657bb0740 1 -- 172.21.15.169:0/273243194 shutdown_connections 2024-03-28T01:07:37.938 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.935+0000 7f2657bb0740 1 -- 172.21.15.169:0/273243194 wait complete. 2024-03-28T01:07:37.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.940+0000 7f2657bb0740 20 client.27316 trim_cache size 0 max 16384 2024-03-28T01:07:37.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.991+0000 7f2657bb0740 1 Processor -- start 2024-03-28T01:07:37.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.996+0000 7f2657bb0740 1 -- start start 2024-03-28T01:07:37.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.996+0000 7f2657bb0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55675c93a690 0x55675c966460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:37.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.996+0000 7f2657bb0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c96b9d0 0x55675c9669a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:37.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.996+0000 7f2657bb0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c938e60 0x55675c9392c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:37.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.996+0000 7f2657bb0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2650075e70 con 0x55675c93a690 2024-03-28T01:07:38.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.996+0000 7f2657bb0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2650118300 con 0x55675c938e60 2024-03-28T01:07:38.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.996+0000 7f2657bb0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2650075810 con 0x55675c96b9d0 2024-03-28T01:07:38.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.997+0000 7f264ed9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c938e60 0x55675c9392c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:38.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.997+0000 7f264ed9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c938e60 0x55675c9392c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:58616/0 (socket says 172.21.15.169:58616) 2024-03-28T01:07:38.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.997+0000 7f264ed9d640 1 -- 172.21.15.169:0/2279544146 learned_addr learned my addr 172.21.15.169:0/2279544146 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:38.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.997+0000 7f264dd9b640 1 --2- 172.21.15.169:0/2279544146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c96b9d0 0x55675c9669a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:38.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:37.998+0000 7f264e59c640 1 --2- 172.21.15.169:0/2279544146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55675c93a690 0x55675c966460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:38.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.001+0000 7f264ed9d640 1 -- 172.21.15.169:0/2279544146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c96b9d0 msgr2=0x55675c9669a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:38.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.001+0000 7f264ed9d640 1 --2- 172.21.15.169:0/2279544146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c96b9d0 0x55675c9669a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.001+0000 7f264ed9d640 1 -- 172.21.15.169:0/2279544146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55675c93a690 msgr2=0x55675c966460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:38.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.001+0000 7f264ed9d640 1 --2- 172.21.15.169:0/2279544146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55675c93a690 0x55675c966460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.001+0000 7f264ed9d640 1 -- 172.21.15.169:0/2279544146 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55675c93a2d0 con 0x55675c938e60 2024-03-28T01:07:38.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.002+0000 7f264ed9d640 1 --2- 172.21.15.169:0/2279544146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c938e60 0x55675c9392c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2640009e60 tx=0x7f2640009630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4b8662e999b2219 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:38.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.008+0000 7f2637fff640 1 -- 172.21.15.169:0/2279544146 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264000a7f0 con 0x55675c938e60 2024-03-28T01:07:38.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.008+0000 7f2637fff640 1 -- 172.21.15.169:0/2279544146 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2640019070 con 0x55675c938e60 2024-03-28T01:07:38.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.009+0000 7f2657bb0740 1 -- 172.21.15.169:0/2279544146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c938e60 msgr2=0x55675c9392c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:38.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.009+0000 7f2657bb0740 1 --2- 172.21.15.169:0/2279544146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c938e60 0x55675c9392c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2640009e60 tx=0x7f2640009630 comp rx=0 tx=0).stop 2024-03-28T01:07:38.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.022+0000 7f2657bb0740 1 -- 172.21.15.169:0/2279544146 shutdown_connections 2024-03-28T01:07:38.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.022+0000 7f2657bb0740 1 --2- 172.21.15.169:0/2279544146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c938e60 0x55675c9392c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.022+0000 7f2657bb0740 1 --2- 172.21.15.169:0/2279544146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c96b9d0 0x55675c9669a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.022+0000 7f2657bb0740 1 --2- 172.21.15.169:0/2279544146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55675c93a690 0x55675c966460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.022+0000 7f2657bb0740 1 -- 172.21.15.169:0/2279544146 >> 172.21.15.169:0/2279544146 conn(0x55675c935c00 msgr2=0x55675c965a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:38.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.031+0000 7f2657bb0740 1 -- 172.21.15.169:0/2279544146 shutdown_connections 2024-03-28T01:07:38.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.031+0000 7f2657bb0740 1 -- 172.21.15.169:0/2279544146 wait complete. 2024-03-28T01:07:38.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.031+0000 7f2657bb0740 1 -- start start 2024-03-28T01:07:38.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.031+0000 7f2657bb0740 1 Processor -- start 2024-03-28T01:07:38.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.032+0000 7f2657bb0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c938e60 0x55675c983600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.032+0000 7f2657bb0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c93a690 0x55675c983b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.032+0000 7f2657bb0740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55675c96b9d0 0x55675c984080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.032+0000 7f2657bb0740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2650074b50 con 0x55675c96b9d0 2024-03-28T01:07:38.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.032+0000 7f2657bb0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2650074510 con 0x55675c938e60 2024-03-28T01:07:38.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.032+0000 7f2657bb0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26500751b0 con 0x55675c93a690 2024-03-28T01:07:38.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.032+0000 7f2657bb0740 10 client.0 _ll_register_callbacks cb 0x7f26380ee140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:38.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.033+0000 7f264e59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c938e60 0x55675c983600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:38.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.033+0000 7f264e59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c938e60 0x55675c983600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:58620/0 (socket says 172.21.15.169:58620) 2024-03-28T01:07:38.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.033+0000 7f264e59c640 1 -- 172.21.15.169:0/2925946334 learned_addr learned my addr 172.21.15.169:0/2925946334 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:38.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.033+0000 7f264dd9b640 1 --2- 172.21.15.169:0/2925946334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c93a690 0x55675c983b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:38.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.033+0000 7f264ed9d640 1 --2- 172.21.15.169:0/2925946334 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55675c96b9d0 0x55675c984080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:38.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.034+0000 7f264e59c640 1 -- 172.21.15.169:0/2925946334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c93a690 msgr2=0x55675c983b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:38.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.034+0000 7f264e59c640 1 --2- 172.21.15.169:0/2925946334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c93a690 0x55675c983b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.034+0000 7f264e59c640 1 -- 172.21.15.169:0/2925946334 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55675c96b9d0 msgr2=0x55675c984080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:38.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.034+0000 7f264e59c640 1 --2- 172.21.15.169:0/2925946334 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55675c96b9d0 0x55675c984080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.034+0000 7f264e59c640 1 -- 172.21.15.169:0/2925946334 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55675c9cd5d0 con 0x55675c938e60 2024-03-28T01:07:38.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.034+0000 7f264e59c640 1 --2- 172.21.15.169:0/2925946334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c938e60 0x55675c983600 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2648000c00 tx=0x7f26480046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef42fcbf70b6d56a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:38.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.044+0000 7f2635ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:07:38.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.044+0000 7f2635ffb640 1 -- 172.21.15.169:0/2925946334 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2648004d70 con 0x55675c938e60 2024-03-28T01:07:38.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.044+0000 7f2657bb0740 1 -- 172.21.15.169:0/2925946334 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2640019070 con 0x55675c938e60 2024-03-28T01:07:38.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.044+0000 7f2635ffb640 1 -- 172.21.15.169:0/2925946334 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2648024520 con 0x55675c938e60 2024-03-28T01:07:38.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.045+0000 7f2635ffb640 1 -- 172.21.15.169:0/2925946334 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2648034050 con 0x55675c938e60 2024-03-28T01:07:38.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.046+0000 7f2657bb0740 10 client.27386 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:38.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.046+0000 7f2657bb0740 1 -- 172.21.15.169:0/2925946334 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55675c9ce9a0 con 0x55675c938e60 2024-03-28T01:07:38.049 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.047+0000 7f2635ffb640 1 -- 172.21.15.169:0/2925946334 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2640019070 con 0x55675c938e60 2024-03-28T01:07:38.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.047+0000 7f2635ffb640 1 -- 172.21.15.169:0/2925946334 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f2648010730 con 0x55675c938e60 2024-03-28T01:07:38.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.047+0000 7f2657bb0740 10 client.27386 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:38.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.047+0000 7f2657bb0740 10 client.27386 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:07:38.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.047+0000 7f2657bb0740 20 client.27386 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:38.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.047+0000 7f2657bb0740 1 --2- 172.21.15.169:0/2925946334 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55675c9cf330 0x55675c9d1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.047+0000 7f2657bb0740 4 client.27386 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:07:38.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.047+0000 7f2657bb0740 1 -- 172.21.15.169:0/2925946334 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55675c8a7c50 con 0x55675c9cf330 2024-03-28T01:07:38.051 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.048+0000 7f264dd9b640 1 --2- 172.21.15.169:0/2925946334 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55675c9cf330 0x55675c9d1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:38.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.050+0000 7f2635ffb640 10 client.27386 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:07:38.053 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.050+0000 7f264dd9b640 1 --2- 172.21.15.169:0/2925946334 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55675c9cf330 0x55675c9d1750 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=57ff64eda07a57a9 server_cookie=e2c95d2c18e62d7b in_seq=0 out_seq=0 2024-03-28T01:07:38.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.051+0000 7f2635ffb640 1 -- 172.21.15.169:0/2925946334 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55675c8a7c50 con 0x55675c9cf330 2024-03-28T01:07:38.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.051+0000 7f2635ffb640 10 client.27386 handle_command_reply: tid=0 2024-03-28T01:07:38.054 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.051+0000 7f2657bb0740 1 client.27386 shutdown 2024-03-28T01:07:38.056 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:37 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:07:38.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.056+0000 7f2657bb0740 1 -- 172.21.15.169:0/2925946334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c938e60 msgr2=0x55675c983600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:38.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.056+0000 7f2657bb0740 1 --2- 172.21.15.169:0/2925946334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c938e60 0x55675c983600 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2648000c00 tx=0x7f26480046c0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.056+0000 7f2657bb0740 1 -- 172.21.15.169:0/2925946334 shutdown_connections 2024-03-28T01:07:38.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.056+0000 7f2657bb0740 1 --2- 172.21.15.169:0/2925946334 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55675c9cf330 0x55675c9d1750 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.056+0000 7f2657bb0740 1 --2- 172.21.15.169:0/2925946334 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55675c96b9d0 0x55675c984080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.056+0000 7f2657bb0740 1 --2- 172.21.15.169:0/2925946334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55675c93a690 0x55675c983b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.056+0000 7f2657bb0740 1 --2- 172.21.15.169:0/2925946334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55675c938e60 0x55675c983600 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.056+0000 7f2635ffb640 0 client.27386 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:07:38.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.056+0000 7f2657bb0740 1 -- 172.21.15.169:0/2925946334 >> 172.21.15.169:0/2925946334 conn(0x55675c935c00 msgr2=0x55675c965a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:38.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.058+0000 7f2657bb0740 1 -- 172.21.15.169:0/2925946334 shutdown_connections 2024-03-28T01:07:38.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.058+0000 7f2657bb0740 1 -- 172.21.15.169:0/2925946334 wait complete. 2024-03-28T01:07:38.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.063+0000 7f2657bb0740 20 client.27386 trim_cache size 0 max 16384 2024-03-28T01:07:38.074 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:07:38.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.076+0000 7f2654fcd640 1 -- 172.21.15.169:0/3842597553 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f263c0a2a00 msgr2=0x7f263c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:38.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.076+0000 7f2654fcd640 1 --2- 172.21.15.169:0/3842597553 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f263c0a2a00 0x7f263c0a4ec0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f26480058c0 tx=0x7f264800a4c0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.076+0000 7f2654fcd640 1 -- 172.21.15.169:0/3842597553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f265010b010 msgr2=0x7f26501a6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:38.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.076+0000 7f2654fcd640 1 --2- 172.21.15.169:0/3842597553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f265010b010 0x7f26501a6e00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2638000c00 tx=0x7f2638004270 comp rx=0 tx=0).stop 2024-03-28T01:07:38.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.077+0000 7f2654fcd640 1 -- 172.21.15.169:0/3842597553 shutdown_connections 2024-03-28T01:07:38.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.077+0000 7f2654fcd640 1 --2- 172.21.15.169:0/3842597553 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f263c0a2a00 0x7f263c0a4ec0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.077+0000 7f2654fcd640 1 --2- 172.21.15.169:0/3842597553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f265010d9c0 0x7f26501ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.077+0000 7f2654fcd640 1 --2- 172.21.15.169:0/3842597553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f265010b010 0x7f26501a6e00 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.077+0000 7f2654fcd640 1 --2- 172.21.15.169:0/3842597553 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26501086a0 0x7f26501a68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.077+0000 7f2654fcd640 1 -- 172.21.15.169:0/3842597553 >> 172.21.15.169:0/3842597553 conn(0x7f265006e500 msgr2=0x7f2650109c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:38.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.078+0000 7f2654fcd640 1 -- 172.21.15.169:0/3842597553 shutdown_connections 2024-03-28T01:07:38.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.078+0000 7f2654fcd640 1 -- 172.21.15.169:0/3842597553 wait complete. 2024-03-28T01:07:38.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:37 smithi174 ceph-mon[36060]: pgmap v684: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 748 MiB/s wr, 834 op/s 2024-03-28T01:07:38.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:37 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3975665323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:07:38.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:37 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:07:38.117 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:07:38.117 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T01:07:38.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:37 smithi177 ceph-mon[36206]: pgmap v684: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 748 MiB/s wr, 834 op/s 2024-03-28T01:07:38.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:37 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3975665323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:07:38.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:37 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:07:38.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.461+0000 7f4aac8ab640 1 -- 172.21.15.169:0/2636744274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aa8109730 msgr2=0x7f4aa810dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:38.465 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.461+0000 7f4aac8ab640 1 --2- 172.21.15.169:0/2636744274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aa8109730 0x7f4aa810dfd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0009ca0 tx=0x7f4aa002c100 comp rx=0 tx=0).stop 2024-03-28T01:07:38.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.464+0000 7f4aac8ab640 1 -- 172.21.15.169:0/2636744274 shutdown_connections 2024-03-28T01:07:38.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.464+0000 7f4aac8ab640 1 --2- 172.21.15.169:0/2636744274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aa8109730 0x7f4aa810dfd0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.464+0000 7f4aac8ab640 1 --2- 172.21.15.169:0/2636744274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4aa8108d70 0x7f4aa81091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.464+0000 7f4aac8ab640 1 --2- 172.21.15.169:0/2636744274 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4aa8072ab0 0x7f4aa8072ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.464+0000 7f4aac8ab640 1 -- 172.21.15.169:0/2636744274 >> 172.21.15.169:0/2636744274 conn(0x7f4aa806e580 msgr2=0x7f4aa80709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:38.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.469+0000 7f4aac8ab640 1 -- 172.21.15.169:0/2636744274 shutdown_connections 2024-03-28T01:07:38.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.469+0000 7f4aac8ab640 1 -- 172.21.15.169:0/2636744274 wait complete. 2024-03-28T01:07:38.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.470+0000 7f4aac8ab640 1 Processor -- start 2024-03-28T01:07:38.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.471+0000 7f4aac8ab640 1 -- start start 2024-03-28T01:07:38.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.471+0000 7f4aac8ab640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4aa8072ab0 0x7f4aa8114c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.471+0000 7f4aac8ab640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4aa8108d70 0x7f4aa8115170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.471+0000 7f4aac8ab640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aa8109730 0x7f4aa81b6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.471+0000 7f4aac8ab640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa811c140 con 0x7f4aa8072ab0 2024-03-28T01:07:38.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.471+0000 7f4aac8ab640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa811bfc0 con 0x7f4aa8108d70 2024-03-28T01:07:38.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.471+0000 7f4aac8ab640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa811c2c0 con 0x7f4aa8109730 2024-03-28T01:07:38.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.472+0000 7f4aa759e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aa8109730 0x7f4aa81b6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:38.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.472+0000 7f4aa759e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aa8109730 0x7f4aa81b6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:47130/0 (socket says 172.21.15.169:47130) 2024-03-28T01:07:38.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.472+0000 7f4aa759e640 1 -- 172.21.15.169:0/885711847 learned_addr learned my addr 172.21.15.169:0/885711847 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:38.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.474+0000 7f4aa759e640 1 -- 172.21.15.169:0/885711847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4aa8108d70 msgr2=0x7f4aa8115170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:38.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.474+0000 7f4aa759e640 1 --2- 172.21.15.169:0/885711847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4aa8108d70 0x7f4aa8115170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.474+0000 7f4aa759e640 1 -- 172.21.15.169:0/885711847 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4aa8072ab0 msgr2=0x7f4aa8114c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:38.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.474+0000 7f4aa759e640 1 --2- 172.21.15.169:0/885711847 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4aa8072ab0 0x7f4aa8114c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.474+0000 7f4aa759e640 1 -- 172.21.15.169:0/885711847 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa81182f0 con 0x7f4aa8109730 2024-03-28T01:07:38.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.475+0000 7f4aa759e640 1 --2- 172.21.15.169:0/885711847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aa8109730 0x7f4aa81b6850 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0037040 tx=0x7f4aa0035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:38.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.479+0000 7f4a8ffff640 1 -- 172.21.15.169:0/885711847 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0039040 con 0x7f4aa8109730 2024-03-28T01:07:38.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.479+0000 7f4aac8ab640 1 -- 172.21.15.169:0/885711847 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa81b6d90 con 0x7f4aa8109730 2024-03-28T01:07:38.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.479+0000 7f4aac8ab640 1 -- 172.21.15.169:0/885711847 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa81b72d0 con 0x7f4aa8109730 2024-03-28T01:07:38.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.479+0000 7f4a8ffff640 1 -- 172.21.15.169:0/885711847 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4aa0044070 con 0x7f4aa8109730 2024-03-28T01:07:38.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.484+0000 7f4a8ffff640 1 -- 172.21.15.169:0/885711847 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa003f6a0 con 0x7f4aa8109730 2024-03-28T01:07:38.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.487+0000 7f4a8ffff640 1 -- 172.21.15.169:0/885711847 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4aa0004040 con 0x7f4aa8109730 2024-03-28T01:07:38.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.488+0000 7f4a8ffff640 1 --2- 172.21.15.169:0/885711847 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4a900a2a10 0x7f4a900a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.489+0000 7f4a8ffff640 1 -- 172.21.15.169:0/885711847 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f4aa00f1200 con 0x7f4aa8109730 2024-03-28T01:07:38.510 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.490+0000 7f4aa6d9d640 1 --2- 172.21.15.169:0/885711847 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4a900a2a10 0x7f4a900a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:38.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.510+0000 7f4aafd73740 1 Processor -- start 2024-03-28T01:07:38.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.513+0000 7f4aafd73740 1 -- start start 2024-03-28T01:07:38.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.513+0000 7f4aafd73740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b88860 0x560dc5b88c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.513+0000 7f4aafd73740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b89250 0x560dc5b8dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.513+0000 7f4aafd73740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560dc5b8e3c0 0x560dc5b90850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.513+0000 7f4aafd73740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dc5b4b610 con 0x560dc5b88860 2024-03-28T01:07:38.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.513+0000 7f4aafd73740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dc5aee3c0 con 0x560dc5b89250 2024-03-28T01:07:38.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.514+0000 7f4aa659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b89250 0x560dc5b8dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:38.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.514+0000 7f4aa759e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560dc5b8e3c0 0x560dc5b90850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:38.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.513+0000 7f4aafd73740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dc5a96d60 con 0x560dc5b8e3c0 2024-03-28T01:07:38.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.517+0000 7f4aa759e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560dc5b8e3c0 0x560dc5b90850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:47134/0 (socket says 172.21.15.169:47134) 2024-03-28T01:07:38.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.517+0000 7f4aa759e640 1 -- 172.21.15.169:0/4015903035 learned_addr learned my addr 172.21.15.169:0/4015903035 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:38.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.519+0000 7f4aa6d9d640 1 --2- 172.21.15.169:0/885711847 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4a900a2a10 0x7f4a900a4ed0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8073800 tx=0x7f4a98006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:38.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.519+0000 7f4aa6d9d640 1 --2- 172.21.15.169:0/4015903035 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b88860 0x560dc5b88c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:38.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.521+0000 7f4aa659c640 1 -- 172.21.15.169:0/4015903035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560dc5b8e3c0 msgr2=0x560dc5b90850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:38.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.521+0000 7f4aa659c640 1 --2- 172.21.15.169:0/4015903035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560dc5b8e3c0 0x560dc5b90850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.521+0000 7f4aa659c640 1 -- 172.21.15.169:0/4015903035 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b88860 msgr2=0x560dc5b88c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:38.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.521+0000 7f4aa659c640 1 --2- 172.21.15.169:0/4015903035 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b88860 0x560dc5b88c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.521+0000 7f4aa659c640 1 -- 172.21.15.169:0/4015903035 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560dc58023f0 con 0x560dc5b89250 2024-03-28T01:07:38.531 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.526+0000 7f4aa659c640 1 --2- 172.21.15.169:0/4015903035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b89250 0x560dc5b8dd50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009720 tx=0x7f4a9c009be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5306f9a091ec726 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:38.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.529+0000 7f4a8d7fa640 1 -- 172.21.15.169:0/4015903035 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c015020 con 0x560dc5b89250 2024-03-28T01:07:38.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.529+0000 7f4a8d7fa640 1 -- 172.21.15.169:0/4015903035 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a9c009e30 con 0x560dc5b89250 2024-03-28T01:07:38.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.529+0000 7f4a8d7fa640 1 -- 172.21.15.169:0/4015903035 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c0103a0 con 0x560dc5b89250 2024-03-28T01:07:38.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.532+0000 7f4aafd73740 1 -- 172.21.15.169:0/4015903035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b89250 msgr2=0x560dc5b8dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:38.536 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.532+0000 7f4aafd73740 1 --2- 172.21.15.169:0/4015903035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b89250 0x560dc5b8dd50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009720 tx=0x7f4a9c009be0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.536+0000 7f4aafd73740 1 -- 172.21.15.169:0/4015903035 shutdown_connections 2024-03-28T01:07:38.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.536+0000 7f4aafd73740 1 --2- 172.21.15.169:0/4015903035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560dc5b8e3c0 0x560dc5b90850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.536+0000 7f4aafd73740 1 --2- 172.21.15.169:0/4015903035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b89250 0x560dc5b8dd50 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.536+0000 7f4aafd73740 1 --2- 172.21.15.169:0/4015903035 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b88860 0x560dc5b88c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.536+0000 7f4aafd73740 1 -- 172.21.15.169:0/4015903035 >> 172.21.15.169:0/4015903035 conn(0x560dc5b83dd0 msgr2=0x560dc5b841d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:38.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.542+0000 7f4aafd73740 1 -- 172.21.15.169:0/4015903035 shutdown_connections 2024-03-28T01:07:38.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.546+0000 7f4aafd73740 1 -- 172.21.15.169:0/4015903035 wait complete. 2024-03-28T01:07:38.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.546+0000 7f4aafd73740 1 -- start start 2024-03-28T01:07:38.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.546+0000 7f4aafd73740 1 Processor -- start 2024-03-28T01:07:38.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.547+0000 7f4aafd73740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b88860 0x560dc5b92a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.547+0000 7f4aafd73740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b89250 0x560dc5b92f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.547+0000 7f4aafd73740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560dc5b8e3c0 0x560dc5b99fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.547+0000 7f4aafd73740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dc5aaa240 con 0x560dc5b88860 2024-03-28T01:07:38.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.547+0000 7f4aafd73740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dc5aea460 con 0x560dc5b89250 2024-03-28T01:07:38.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.547+0000 7f4aafd73740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dc5b91370 con 0x560dc5b8e3c0 2024-03-28T01:07:38.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.547+0000 7f4aafd73740 10 client.0 _ll_register_callbacks cb 0x560dc5aa9c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:38.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.548+0000 7f4aa6d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b88860 0x560dc5b92a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:38.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.548+0000 7f4aa6d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b88860 0x560dc5b92a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38284/0 (socket says 172.21.15.169:38284) 2024-03-28T01:07:38.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.548+0000 7f4aa6d9d640 1 -- 172.21.15.169:0/807345503 learned_addr learned my addr 172.21.15.169:0/807345503 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:38.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.549+0000 7f4aa759e640 1 --2- 172.21.15.169:0/807345503 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560dc5b8e3c0 0x560dc5b99fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:38.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.549+0000 7f4aa659c640 1 --2- 172.21.15.169:0/807345503 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b89250 0x560dc5b92f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:38.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.549+0000 7f4aa6d9d640 1 -- 172.21.15.169:0/807345503 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560dc5b8e3c0 msgr2=0x560dc5b99fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:38.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.549+0000 7f4aa6d9d640 1 --2- 172.21.15.169:0/807345503 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560dc5b8e3c0 0x560dc5b99fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.549+0000 7f4aa6d9d640 1 -- 172.21.15.169:0/807345503 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b89250 msgr2=0x560dc5b92f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:38.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.549+0000 7f4aa6d9d640 1 --2- 172.21.15.169:0/807345503 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b89250 0x560dc5b92f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.549+0000 7f4aa6d9d640 1 -- 172.21.15.169:0/807345503 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560dc58cc7d0 con 0x560dc5b88860 2024-03-28T01:07:38.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.571+0000 7f4aa6d9d640 1 --2- 172.21.15.169:0/807345503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b88860 0x560dc5b92a20 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f4a9800b4a0 tx=0x7f4a980047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a082621d1de7e537 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:38.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.571+0000 7f4a8aff5640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:07:38.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.585+0000 7f4a8aff5640 1 -- 172.21.15.169:0/807345503 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a98024060 con 0x560dc5b88860 2024-03-28T01:07:38.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.585+0000 7f4a8aff5640 1 -- 172.21.15.169:0/807345503 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f4a980127c0 con 0x560dc5b88860 2024-03-28T01:07:38.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.585+0000 7f4aafd73740 1 -- 172.21.15.169:0/807345503 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4a9c009e30 con 0x560dc5b88860 2024-03-28T01:07:38.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.594+0000 7f4a8aff5640 1 -- 172.21.15.169:0/807345503 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a9802b050 con 0x560dc5b88860 2024-03-28T01:07:38.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.594+0000 7f4aafd73740 10 client.17493 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:38.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.594+0000 7f4aafd73740 1 -- 172.21.15.169:0/807345503 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x560dc5b9d380 con 0x560dc5b88860 2024-03-28T01:07:38.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.601+0000 7f4a8aff5640 1 -- 172.21.15.169:0/807345503 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a9c009e30 con 0x560dc5b88860 2024-03-28T01:07:38.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.601+0000 7f4a8aff5640 1 -- 172.21.15.169:0/807345503 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f4a9801e070 con 0x560dc5b88860 2024-03-28T01:07:38.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.601+0000 7f4aafd73740 10 client.17493 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:38.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.601+0000 7f4aafd73740 10 client.17493 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:07:38.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.601+0000 7f4aafd73740 20 client.17493 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:38.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.601+0000 7f4aafd73740 1 --2- 172.21.15.169:0/807345503 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x560dc5b9ea80 0x560dc5ba0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.601+0000 7f4aafd73740 4 client.17493 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:07:38.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.601+0000 7f4aafd73740 1 -- 172.21.15.169:0/807345503 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560dc5afb900 con 0x560dc5b9ea80 2024-03-28T01:07:38.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.610+0000 7f4aa659c640 1 --2- 172.21.15.169:0/807345503 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x560dc5b9ea80 0x560dc5ba0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:38.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.627+0000 7f4a8aff5640 10 client.17493 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:07:38.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.627+0000 7f4aa659c640 1 --2- 172.21.15.169:0/807345503 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x560dc5b9ea80 0x560dc5ba0e70 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=afa284e0eefc4d63 server_cookie=b06cea5b3b6b1176 in_seq=0 out_seq=0 2024-03-28T01:07:38.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.634+0000 7f4a8aff5640 1 -- 172.21.15.169:0/807345503 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560dc5afb900 con 0x560dc5b9ea80 2024-03-28T01:07:38.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.634+0000 7f4a8aff5640 10 client.17493 handle_command_reply: tid=0 2024-03-28T01:07:38.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.634+0000 7f4aafd73740 1 client.17493 shutdown 2024-03-28T01:07:38.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.635+0000 7f4aafd73740 1 -- 172.21.15.169:0/807345503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b88860 msgr2=0x560dc5b92a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:38.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.635+0000 7f4aafd73740 1 --2- 172.21.15.169:0/807345503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b88860 0x560dc5b92a20 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f4a9800b4a0 tx=0x7f4a980047d0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.635+0000 7f4aafd73740 1 -- 172.21.15.169:0/807345503 shutdown_connections 2024-03-28T01:07:38.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.635+0000 7f4aafd73740 1 --2- 172.21.15.169:0/807345503 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x560dc5b9ea80 0x560dc5ba0e70 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.635+0000 7f4aafd73740 1 --2- 172.21.15.169:0/807345503 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560dc5b8e3c0 0x560dc5b99fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.635+0000 7f4aafd73740 1 --2- 172.21.15.169:0/807345503 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b89250 0x560dc5b92f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.635+0000 7f4aafd73740 1 --2- 172.21.15.169:0/807345503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b88860 0x560dc5b92a20 secure :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f4a9800b4a0 tx=0x7f4a980047d0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.635+0000 7f4aafd73740 1 -- 172.21.15.169:0/807345503 >> 172.21.15.169:0/807345503 conn(0x560dc5b83dd0 msgr2=0x560dc5b83070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:38.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.635+0000 7f4a8aff5640 0 client.17493 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:07:38.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.635+0000 7f4aafd73740 1 -- 172.21.15.169:0/807345503 shutdown_connections 2024-03-28T01:07:38.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.636+0000 7f4aafd73740 1 -- 172.21.15.169:0/807345503 wait complete. 2024-03-28T01:07:38.640 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.636+0000 7f4aafd73740 20 client.17493 trim_cache size 0 max 16384 2024-03-28T01:07:38.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.693+0000 7f4aafd73740 1 Processor -- start 2024-03-28T01:07:38.698 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.695+0000 7f4aafd73740 1 -- start start 2024-03-28T01:07:38.698 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.695+0000 7f4aafd73740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b89250 0x560dc5b92ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.698 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.695+0000 7f4aafd73740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b8e3c0 0x560dc5b99f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.695+0000 7f4aafd73740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560dc5b917a0 0x560dc5b827e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.695+0000 7f4aafd73740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a94012080 con 0x560dc5b8e3c0 2024-03-28T01:07:38.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.695+0000 7f4aafd73740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9400fb80 con 0x560dc5b89250 2024-03-28T01:07:38.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.695+0000 7f4aafd73740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9400dc20 con 0x560dc5b917a0 2024-03-28T01:07:38.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.698+0000 7f4aa6d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b89250 0x560dc5b92ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:38.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.699+0000 7f4aa659c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b8e3c0 0x560dc5b99f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:38.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.699+0000 7f4aa659c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b8e3c0 0x560dc5b99f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38294/0 (socket says 172.21.15.169:38294) 2024-03-28T01:07:38.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.699+0000 7f4aa659c640 1 -- 172.21.15.169:0/823332846 learned_addr learned my addr 172.21.15.169:0/823332846 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:38.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.699+0000 7f4aa759e640 1 --2- 172.21.15.169:0/823332846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560dc5b917a0 0x560dc5b827e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:38.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.700+0000 7f4aa759e640 1 -- 172.21.15.169:0/823332846 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b89250 msgr2=0x560dc5b92ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:38.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.700+0000 7f4aa759e640 1 --2- 172.21.15.169:0/823332846 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b89250 0x560dc5b92ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.700+0000 7f4aa759e640 1 -- 172.21.15.169:0/823332846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b8e3c0 msgr2=0x560dc5b99f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:38.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.700+0000 7f4aa759e640 1 --2- 172.21.15.169:0/823332846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b8e3c0 0x560dc5b99f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.700+0000 7f4aa759e640 1 -- 172.21.15.169:0/823332846 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560dc5b88c40 con 0x560dc5b917a0 2024-03-28T01:07:38.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.700+0000 7f4aa759e640 1 --2- 172.21.15.169:0/823332846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560dc5b917a0 0x560dc5b827e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00bd4a0 tx=0x7f4aa002e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e40c2460222acfa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:38.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.713+0000 7f4a8cff9640 1 -- 172.21.15.169:0/823332846 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa00d6070 con 0x560dc5b917a0 2024-03-28T01:07:38.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.713+0000 7f4a8cff9640 1 -- 172.21.15.169:0/823332846 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4aa00bd7a0 con 0x560dc5b917a0 2024-03-28T01:07:38.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.715+0000 7f4aafd73740 1 -- 172.21.15.169:0/823332846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560dc5b917a0 msgr2=0x560dc5b827e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:38.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.715+0000 7f4aafd73740 1 --2- 172.21.15.169:0/823332846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560dc5b917a0 0x560dc5b827e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00bd4a0 tx=0x7f4aa002e040 comp rx=0 tx=0).stop 2024-03-28T01:07:38.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.715+0000 7f4aafd73740 1 -- 172.21.15.169:0/823332846 shutdown_connections 2024-03-28T01:07:38.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.715+0000 7f4aafd73740 1 --2- 172.21.15.169:0/823332846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560dc5b917a0 0x560dc5b827e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.715+0000 7f4aafd73740 1 --2- 172.21.15.169:0/823332846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b8e3c0 0x560dc5b99f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.715+0000 7f4aafd73740 1 --2- 172.21.15.169:0/823332846 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b89250 0x560dc5b92ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.715+0000 7f4aafd73740 1 -- 172.21.15.169:0/823332846 >> 172.21.15.169:0/823332846 conn(0x560dc5b83dd0 msgr2=0x560dc5b841b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:38.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.724+0000 7f4aafd73740 1 -- 172.21.15.169:0/823332846 shutdown_connections 2024-03-28T01:07:38.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.725+0000 7f4aafd73740 1 -- 172.21.15.169:0/823332846 wait complete. 2024-03-28T01:07:38.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.725+0000 7f4aafd73740 1 -- start start 2024-03-28T01:07:38.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.725+0000 7f4aafd73740 1 Processor -- start 2024-03-28T01:07:38.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.726+0000 7f4aafd73740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560dc5b89250 0x560dc5b97e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.726+0000 7f4aafd73740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b8e3c0 0x560dc5b98350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.726+0000 7f4aafd73740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b917a0 0x560dc5b99310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.726+0000 7f4aafd73740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9400e220 con 0x560dc5b917a0 2024-03-28T01:07:38.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.726+0000 7f4aafd73740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9400f520 con 0x560dc5b8e3c0 2024-03-28T01:07:38.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.726+0000 7f4aafd73740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9400e860 con 0x560dc5b89250 2024-03-28T01:07:38.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.726+0000 7f4aafd73740 10 client.0 _ll_register_callbacks cb 0x7f4a98013d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:38.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.727+0000 7f4aa659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b8e3c0 0x560dc5b98350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:38.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.727+0000 7f4aa659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b8e3c0 0x560dc5b98350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:58688/0 (socket says 172.21.15.169:58688) 2024-03-28T01:07:38.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.727+0000 7f4aa659c640 1 -- 172.21.15.169:0/233576879 learned_addr learned my addr 172.21.15.169:0/233576879 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:38.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.728+0000 7f4aa659c640 1 -- 172.21.15.169:0/233576879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560dc5b89250 msgr2=0x560dc5b97e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:38.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.728+0000 7f4aa659c640 1 --2- 172.21.15.169:0/233576879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560dc5b89250 0x560dc5b97e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.728+0000 7f4aa659c640 1 -- 172.21.15.169:0/233576879 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b917a0 msgr2=0x560dc5b99310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:38.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.728+0000 7f4aa659c640 1 --2- 172.21.15.169:0/233576879 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b917a0 0x560dc5b99310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.728+0000 7f4aa659c640 1 -- 172.21.15.169:0/233576879 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560dc5b99900 con 0x560dc5b8e3c0 2024-03-28T01:07:38.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.729+0000 7f4aa659c640 1 --2- 172.21.15.169:0/233576879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b8e3c0 0x560dc5b98350 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009350 tx=0x7f4a9c011a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40005ff63137e75b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:38.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.729+0000 7f4a8aff5640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:07:38.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.729+0000 7f4a8aff5640 1 -- 172.21.15.169:0/233576879 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c015040 con 0x560dc5b8e3c0 2024-03-28T01:07:38.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.729+0000 7f4aafd73740 1 -- 172.21.15.169:0/233576879 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4aa00bd7a0 con 0x560dc5b8e3c0 2024-03-28T01:07:38.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.729+0000 7f4a8aff5640 1 -- 172.21.15.169:0/233576879 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f4a9c0114c0 con 0x560dc5b8e3c0 2024-03-28T01:07:38.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.730+0000 7f4a8aff5640 1 -- 172.21.15.169:0/233576879 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a9c033050 con 0x560dc5b8e3c0 2024-03-28T01:07:38.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.734+0000 7f4aafd73740 10 client.27395 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:38.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.734+0000 7f4aafd73740 1 -- 172.21.15.169:0/233576879 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x560dc5bfab00 con 0x560dc5b8e3c0 2024-03-28T01:07:38.738 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.735+0000 7f4a8aff5640 1 -- 172.21.15.169:0/233576879 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4aa00bd7a0 con 0x560dc5b8e3c0 2024-03-28T01:07:38.738 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.735+0000 7f4a8aff5640 1 -- 172.21.15.169:0/233576879 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f4a9c025950 con 0x560dc5b8e3c0 2024-03-28T01:07:38.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.739+0000 7f4aafd73740 10 client.27395 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:38.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.739+0000 7f4aafd73740 10 client.27395 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:07:38.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.739+0000 7f4aafd73740 20 client.27395 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:38.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.739+0000 7f4aafd73740 1 --2- 172.21.15.169:0/233576879 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x560dc5bfb310 0x560dc5bfd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:38.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.739+0000 7f4aafd73740 4 client.27395 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:07:38.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.739+0000 7f4aafd73740 1 -- 172.21.15.169:0/233576879 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560dc5af5c50 con 0x560dc5bfb310 2024-03-28T01:07:38.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.745+0000 7f4aa6d9d640 1 --2- 172.21.15.169:0/233576879 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x560dc5bfb310 0x560dc5bfd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:38.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.747+0000 7f4aa6d9d640 1 --2- 172.21.15.169:0/233576879 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x560dc5bfb310 0x560dc5bfd730 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=675276b71097a820 server_cookie=6906505e96f54aca in_seq=0 out_seq=0 2024-03-28T01:07:38.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.749+0000 7f4a8aff5640 10 client.27395 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:07:38.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.749+0000 7f4a8aff5640 1 -- 172.21.15.169:0/233576879 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560dc5af5c50 con 0x560dc5bfb310 2024-03-28T01:07:38.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.749+0000 7f4a8aff5640 10 client.27395 handle_command_reply: tid=0 2024-03-28T01:07:38.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.749+0000 7f4aafd73740 1 client.27395 shutdown 2024-03-28T01:07:38.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.755+0000 7f4aafd73740 1 -- 172.21.15.169:0/233576879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b8e3c0 msgr2=0x560dc5b98350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:38.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.755+0000 7f4aafd73740 1 --2- 172.21.15.169:0/233576879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b8e3c0 0x560dc5b98350 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009350 tx=0x7f4a9c011a20 comp rx=0 tx=0).stop 2024-03-28T01:07:38.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.755+0000 7f4aafd73740 1 -- 172.21.15.169:0/233576879 shutdown_connections 2024-03-28T01:07:38.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.755+0000 7f4aafd73740 1 --2- 172.21.15.169:0/233576879 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x560dc5bfb310 0x560dc5bfd730 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.755+0000 7f4aafd73740 1 --2- 172.21.15.169:0/233576879 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x560dc5b917a0 0x560dc5b99310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.755+0000 7f4aafd73740 1 --2- 172.21.15.169:0/233576879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dc5b8e3c0 0x560dc5b98350 secure :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009350 tx=0x7f4a9c011a20 comp rx=0 tx=0).stop 2024-03-28T01:07:38.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.755+0000 7f4aafd73740 1 --2- 172.21.15.169:0/233576879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560dc5b89250 0x560dc5b97e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.755+0000 7f4aafd73740 1 -- 172.21.15.169:0/233576879 >> 172.21.15.169:0/233576879 conn(0x560dc5b83dd0 msgr2=0x560dc5b87030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:38.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.756+0000 7f4a8aff5640 0 client.27395 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:07:38.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.761+0000 7f4aafd73740 1 -- 172.21.15.169:0/233576879 shutdown_connections 2024-03-28T01:07:38.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.768+0000 7f4aafd73740 1 -- 172.21.15.169:0/233576879 wait complete. 2024-03-28T01:07:38.776 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.772+0000 7f4aafd73740 20 client.27395 trim_cache size 0 max 16384 2024-03-28T01:07:38.784 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:07:38.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.795+0000 7f4aac8ab640 1 -- 172.21.15.169:0/885711847 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4a900a2a10 msgr2=0x7f4a900a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:38.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.795+0000 7f4aac8ab640 1 --2- 172.21.15.169:0/885711847 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4a900a2a10 0x7f4a900a4ed0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8073800 tx=0x7f4a98006c90 comp rx=0 tx=0).stop 2024-03-28T01:07:38.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.795+0000 7f4aac8ab640 1 -- 172.21.15.169:0/885711847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aa8109730 msgr2=0x7f4aa81b6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:38.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.795+0000 7f4aac8ab640 1 --2- 172.21.15.169:0/885711847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aa8109730 0x7f4aa81b6850 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0037040 tx=0x7f4aa0035d10 comp rx=0 tx=0).stop 2024-03-28T01:07:38.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.798+0000 7f4aac8ab640 1 -- 172.21.15.169:0/885711847 shutdown_connections 2024-03-28T01:07:38.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.798+0000 7f4aac8ab640 1 --2- 172.21.15.169:0/885711847 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4a900a2a10 0x7f4a900a4ed0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.798+0000 7f4aac8ab640 1 --2- 172.21.15.169:0/885711847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aa8109730 0x7f4aa81b6850 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.798+0000 7f4aac8ab640 1 --2- 172.21.15.169:0/885711847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4aa8108d70 0x7f4aa8115170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.798+0000 7f4aac8ab640 1 --2- 172.21.15.169:0/885711847 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4aa8072ab0 0x7f4aa8114c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:38.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.798+0000 7f4aac8ab640 1 -- 172.21.15.169:0/885711847 >> 172.21.15.169:0/885711847 conn(0x7f4aa806e580 msgr2=0x7f4aa8109fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:38.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.802+0000 7f4aac8ab640 1 -- 172.21.15.169:0/885711847 shutdown_connections 2024-03-28T01:07:38.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:38.805+0000 7f4aac8ab640 1 -- 172.21.15.169:0/885711847 wait complete. 2024-03-28T01:07:38.872 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:07:38.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T01:07:39.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.313+0000 7fbd66ca7640 1 -- 172.21.15.169:0/4000431076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd600729f0 msgr2=0x7fbd60072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:39.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.313+0000 7fbd66ca7640 1 --2- 172.21.15.169:0/4000431076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd600729f0 0x7fbd60072e10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fbd5801acb0 tx=0x7fbd5803d170 comp rx=0 tx=0).stop 2024-03-28T01:07:39.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.331+0000 7fbd66ca7640 1 -- 172.21.15.169:0/4000431076 shutdown_connections 2024-03-28T01:07:39.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.331+0000 7fbd66ca7640 1 --2- 172.21.15.169:0/4000431076 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd60109640 0x7fbd6010df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.331+0000 7fbd66ca7640 1 --2- 172.21.15.169:0/4000431076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd60108ca0 0x7fbd60109100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.331+0000 7fbd66ca7640 1 --2- 172.21.15.169:0/4000431076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd600729f0 0x7fbd60072e10 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.331+0000 7fbd66ca7640 1 -- 172.21.15.169:0/4000431076 >> 172.21.15.169:0/4000431076 conn(0x7fbd6006e4c0 msgr2=0x7fbd60070900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:39.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.335+0000 7fbd66ca7640 1 -- 172.21.15.169:0/4000431076 shutdown_connections 2024-03-28T01:07:39.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.338+0000 7fbd66ca7640 1 -- 172.21.15.169:0/4000431076 wait complete. 2024-03-28T01:07:39.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.342+0000 7fbd66ca7640 1 Processor -- start 2024-03-28T01:07:39.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.343+0000 7fbd66ca7640 1 -- start start 2024-03-28T01:07:39.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.343+0000 7fbd66ca7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd600729f0 0x7fbd6019df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:39.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.343+0000 7fbd66ca7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd60108ca0 0x7fbd6019e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:39.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.344+0000 7fbd66ca7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd60109640 0x7fbd601a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:39.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.344+0000 7fbd66ca7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd601119a0 con 0x7fbd60108ca0 2024-03-28T01:07:39.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.344+0000 7fbd65ca5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd600729f0 0x7fbd6019df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:39.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.344+0000 7fbd65ca5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd600729f0 0x7fbd6019df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:47170/0 (socket says 172.21.15.169:47170) 2024-03-28T01:07:39.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.344+0000 7fbd65ca5640 1 -- 172.21.15.169:0/3840441458 learned_addr learned my addr 172.21.15.169:0/3840441458 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:39.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.344+0000 7fbd66ca7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd60111820 con 0x7fbd60109640 2024-03-28T01:07:39.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.344+0000 7fbd66ca7640 1 -- 172.21.15.169:0/3840441458 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd60111b20 con 0x7fbd600729f0 2024-03-28T01:07:39.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.346+0000 7fbd65ca5640 1 -- 172.21.15.169:0/3840441458 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd60109640 msgr2=0x7fbd601a5500 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:07:39.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.346+0000 7fbd65ca5640 1 --2- 172.21.15.169:0/3840441458 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd60109640 0x7fbd601a5500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.346+0000 7fbd65ca5640 1 -- 172.21.15.169:0/3840441458 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd60108ca0 msgr2=0x7fbd6019e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:39.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.346+0000 7fbd65ca5640 1 --2- 172.21.15.169:0/3840441458 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd60108ca0 0x7fbd6019e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.346+0000 7fbd65ca5640 1 -- 172.21.15.169:0/3840441458 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd601a5b70 con 0x7fbd600729f0 2024-03-28T01:07:39.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.347+0000 7fbd65ca5640 1 --2- 172.21.15.169:0/3840441458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd600729f0 0x7fbd6019df50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fbd5801add0 tx=0x7fbd58003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:39.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.349+0000 7fbd56ffd640 1 -- 172.21.15.169:0/3840441458 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd58052020 con 0x7fbd600729f0 2024-03-28T01:07:39.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.349+0000 7fbd56ffd640 1 -- 172.21.15.169:0/3840441458 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd58003e50 con 0x7fbd600729f0 2024-03-28T01:07:39.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.349+0000 7fbd56ffd640 1 -- 172.21.15.169:0/3840441458 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5804a040 con 0x7fbd600729f0 2024-03-28T01:07:39.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.351+0000 7fbd66ca7640 1 -- 172.21.15.169:0/3840441458 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd601a5e00 con 0x7fbd600729f0 2024-03-28T01:07:39.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.351+0000 7fbd66ca7640 1 -- 172.21.15.169:0/3840441458 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd601a6290 con 0x7fbd600729f0 2024-03-28T01:07:39.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.355+0000 7fbd56ffd640 1 -- 172.21.15.169:0/3840441458 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbd58055020 con 0x7fbd600729f0 2024-03-28T01:07:39.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.356+0000 7fbd56ffd640 1 --2- 172.21.15.169:0/3840441458 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbd4c0a2a00 0x7fbd4c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:39.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.356+0000 7fbd56ffd640 1 -- 172.21.15.169:0/3840441458 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fbd58102fb0 con 0x7fbd600729f0 2024-03-28T01:07:39.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.357+0000 7fbd654a4640 1 --2- 172.21.15.169:0/3840441458 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbd4c0a2a00 0x7fbd4c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:39.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.361+0000 7fbd6abd2740 1 Processor -- start 2024-03-28T01:07:39.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.361+0000 7fbd654a4640 1 --2- 172.21.15.169:0/3840441458 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbd4c0a2a00 0x7fbd4c0a4ec0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fbd6019f460 tx=0x7fbd5c006c40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:39.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.362+0000 7fbd6abd2740 1 -- start start 2024-03-28T01:07:39.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.362+0000 7fbd6abd2740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556efcc6a8a0 0x556efcc6acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:39.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.362+0000 7fbd6abd2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6b290 0x556efcc6fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:39.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.362+0000 7fbd6abd2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc70400 0x556efcc72890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:39.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.362+0000 7fbd6abd2740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x556efcc2d610 con 0x556efcc6a8a0 2024-03-28T01:07:39.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.362+0000 7fbd6abd2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x556efcbd03c0 con 0x556efcc6b290 2024-03-28T01:07:39.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.362+0000 7fbd6abd2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x556efcb78d60 con 0x556efcc70400 2024-03-28T01:07:39.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.363+0000 7fbd65ca5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556efcc6a8a0 0x556efcc6acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:39.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.363+0000 7fbd65ca5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556efcc6a8a0 0x556efcc6acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38316/0 (socket says 172.21.15.169:38316) 2024-03-28T01:07:39.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.363+0000 7fbd65ca5640 1 -- 172.21.15.169:0/355578786 learned_addr learned my addr 172.21.15.169:0/355578786 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:39.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.364+0000 7fbd664a6640 1 --2- 172.21.15.169:0/355578786 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc70400 0x556efcc72890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:39.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.365+0000 7fbd654a4640 1 --2- 172.21.15.169:0/355578786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6b290 0x556efcc6fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:39.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.367+0000 7fbd664a6640 1 -- 172.21.15.169:0/355578786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6b290 msgr2=0x556efcc6fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:39.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.367+0000 7fbd664a6640 1 --2- 172.21.15.169:0/355578786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6b290 0x556efcc6fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.367+0000 7fbd664a6640 1 -- 172.21.15.169:0/355578786 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556efcc6a8a0 msgr2=0x556efcc6acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:39.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.367+0000 7fbd664a6640 1 --2- 172.21.15.169:0/355578786 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556efcc6a8a0 0x556efcc6acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.367+0000 7fbd664a6640 1 -- 172.21.15.169:0/355578786 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556efc8e43f0 con 0x556efcc70400 2024-03-28T01:07:39.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.374+0000 7fbd664a6640 1 --2- 172.21.15.169:0/355578786 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc70400 0x556efcc72890 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009630 tx=0x7fbd50009af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82db44e087a31c2a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:39.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.375+0000 7fbd4bfff640 1 -- 172.21.15.169:0/355578786 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50015020 con 0x556efcc70400 2024-03-28T01:07:39.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.375+0000 7fbd4bfff640 1 -- 172.21.15.169:0/355578786 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd50011d10 con 0x556efcc70400 2024-03-28T01:07:39.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.376+0000 7fbd6abd2740 1 -- 172.21.15.169:0/355578786 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc70400 msgr2=0x556efcc72890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:39.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.376+0000 7fbd6abd2740 1 --2- 172.21.15.169:0/355578786 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc70400 0x556efcc72890 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009630 tx=0x7fbd50009af0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.377+0000 7fbd6abd2740 1 -- 172.21.15.169:0/355578786 shutdown_connections 2024-03-28T01:07:39.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.377+0000 7fbd6abd2740 1 --2- 172.21.15.169:0/355578786 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc70400 0x556efcc72890 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.377+0000 7fbd6abd2740 1 --2- 172.21.15.169:0/355578786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6b290 0x556efcc6fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.377+0000 7fbd6abd2740 1 --2- 172.21.15.169:0/355578786 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556efcc6a8a0 0x556efcc6acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.377+0000 7fbd6abd2740 1 -- 172.21.15.169:0/355578786 >> 172.21.15.169:0/355578786 conn(0x556efcc65e10 msgr2=0x556efcc66210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:39.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.391+0000 7fbd6abd2740 1 -- 172.21.15.169:0/355578786 shutdown_connections 2024-03-28T01:07:39.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.391+0000 7fbd6abd2740 1 -- 172.21.15.169:0/355578786 wait complete. 2024-03-28T01:07:39.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.391+0000 7fbd6abd2740 1 -- start start 2024-03-28T01:07:39.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.391+0000 7fbd6abd2740 1 Processor -- start 2024-03-28T01:07:39.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.393+0000 7fbd6abd2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6a8a0 0x556efcc74bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:39.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.393+0000 7fbd6abd2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc6b290 0x556efcc75100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:39.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.393+0000 7fbd6abd2740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556efcc70400 0x556efcc7c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:39.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.393+0000 7fbd6abd2740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x556efcb8c240 con 0x556efcc70400 2024-03-28T01:07:39.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.393+0000 7fbd6abd2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x556efcbcc460 con 0x556efcc6a8a0 2024-03-28T01:07:39.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.393+0000 7fbd6abd2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x556efcc73400 con 0x556efcc6b290 2024-03-28T01:07:39.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.393+0000 7fbd6abd2740 10 client.0 _ll_register_callbacks cb 0x556efcb8bc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:39.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.393+0000 7fbd65ca5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6a8a0 0x556efcc74bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:39.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.393+0000 7fbd65ca5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6a8a0 0x556efcc74bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:58714/0 (socket says 172.21.15.169:58714) 2024-03-28T01:07:39.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.393+0000 7fbd65ca5640 1 -- 172.21.15.169:0/1395946256 learned_addr learned my addr 172.21.15.169:0/1395946256 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:39.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.393+0000 7fbd654a4640 1 --2- 172.21.15.169:0/1395946256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc6b290 0x556efcc75100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:39.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.394+0000 7fbd654a4640 1 -- 172.21.15.169:0/1395946256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6a8a0 msgr2=0x556efcc74bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:39.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.394+0000 7fbd654a4640 1 --2- 172.21.15.169:0/1395946256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6a8a0 0x556efcc74bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.394+0000 7fbd654a4640 1 -- 172.21.15.169:0/1395946256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556efcc70400 msgr2=0x556efcc7c140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:07:39.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.394+0000 7fbd654a4640 1 --2- 172.21.15.169:0/1395946256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556efcc70400 0x556efcc7c140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.394+0000 7fbd654a4640 1 -- 172.21.15.169:0/1395946256 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556efc9ae7d0 con 0x556efcc6b290 2024-03-28T01:07:39.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.394+0000 7fbd654a4640 1 --2- 172.21.15.169:0/1395946256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc6b290 0x556efcc75100 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x556efcc6bd00 tx=0x7fbd5c0047e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ca1db89ba218353 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:39.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.397+0000 7fbd497fa640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:07:39.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.397+0000 7fbd497fa640 1 -- 172.21.15.169:0/1395946256 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c005a00 con 0x556efcc6b290 2024-03-28T01:07:39.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.397+0000 7fbd497fa640 1 -- 172.21.15.169:0/1395946256 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fbd5c005ba0 con 0x556efcc6b290 2024-03-28T01:07:39.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.401+0000 7fbd6abd2740 1 -- 172.21.15.169:0/1395946256 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbd50011d10 con 0x556efcc6b290 2024-03-28T01:07:39.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.401+0000 7fbd497fa640 1 -- 172.21.15.169:0/1395946256 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd5c005da0 con 0x556efcc6b290 2024-03-28T01:07:39.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.402+0000 7fbd6abd2740 10 client.27349 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:39.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.402+0000 7fbd6abd2740 1 -- 172.21.15.169:0/1395946256 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556efcc7f550 con 0x556efcc6b290 2024-03-28T01:07:39.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.402+0000 7fbd497fa640 1 -- 172.21.15.169:0/1395946256 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd50011d10 con 0x556efcc6b290 2024-03-28T01:07:39.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.402+0000 7fbd497fa640 1 -- 172.21.15.169:0/1395946256 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fbd5c012070 con 0x556efcc6b290 2024-03-28T01:07:39.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.402+0000 7fbd6abd2740 10 client.27349 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:39.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.402+0000 7fbd6abd2740 10 client.27349 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:07:39.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.402+0000 7fbd6abd2740 20 client.27349 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:39.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.402+0000 7fbd6abd2740 1 --2- 172.21.15.169:0/1395946256 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x556efcc80d20 0x556efcc83160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:39.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.402+0000 7fbd6abd2740 4 client.27349 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:07:39.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.402+0000 7fbd6abd2740 1 -- 172.21.15.169:0/1395946256 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556efcbdd900 con 0x556efcc80d20 2024-03-28T01:07:39.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.403+0000 7fbd65ca5640 1 --2- 172.21.15.169:0/1395946256 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x556efcc80d20 0x556efcc83160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:39.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.404+0000 7fbd65ca5640 1 --2- 172.21.15.169:0/1395946256 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x556efcc80d20 0x556efcc83160 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b4779fbb615f6f3 server_cookie=fee2e821ca64576a in_seq=0 out_seq=0 2024-03-28T01:07:39.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.406+0000 7fbd497fa640 10 client.27349 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:39.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.409+0000 7fbd497fa640 1 -- 172.21.15.169:0/1395946256 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556efcbdd900 con 0x556efcc80d20 2024-03-28T01:07:39.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.409+0000 7fbd497fa640 10 client.27349 handle_command_reply: tid=0 2024-03-28T01:07:39.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.410+0000 7fbd6abd2740 1 client.27349 shutdown 2024-03-28T01:07:39.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.410+0000 7fbd6abd2740 1 -- 172.21.15.169:0/1395946256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc6b290 msgr2=0x556efcc75100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:39.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.410+0000 7fbd6abd2740 1 --2- 172.21.15.169:0/1395946256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc6b290 0x556efcc75100 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x556efcc6bd00 tx=0x7fbd5c0047e0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.411+0000 7fbd6abd2740 1 -- 172.21.15.169:0/1395946256 shutdown_connections 2024-03-28T01:07:39.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.411+0000 7fbd6abd2740 1 --2- 172.21.15.169:0/1395946256 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x556efcc80d20 0x556efcc83160 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.411+0000 7fbd6abd2740 1 --2- 172.21.15.169:0/1395946256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556efcc70400 0x556efcc7c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.411+0000 7fbd6abd2740 1 --2- 172.21.15.169:0/1395946256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc6b290 0x556efcc75100 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.411+0000 7fbd6abd2740 1 --2- 172.21.15.169:0/1395946256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6a8a0 0x556efcc74bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.411+0000 7fbd497fa640 0 client.27349 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:39.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.411+0000 7fbd6abd2740 1 -- 172.21.15.169:0/1395946256 >> 172.21.15.169:0/1395946256 conn(0x556efcc65e10 msgr2=0x556efcc65070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:39.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.411+0000 7fbd6abd2740 1 -- 172.21.15.169:0/1395946256 shutdown_connections 2024-03-28T01:07:39.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.411+0000 7fbd6abd2740 1 -- 172.21.15.169:0/1395946256 wait complete. 2024-03-28T01:07:39.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.411+0000 7fbd6abd2740 20 client.27349 trim_cache size 0 max 16384 2024-03-28T01:07:39.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.440+0000 7fbd6abd2740 1 Processor -- start 2024-03-28T01:07:39.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.444+0000 7fbd6abd2740 1 -- start start 2024-03-28T01:07:39.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.444+0000 7fbd6abd2740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556efcc6a8a0 0x556efcc745c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:39.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.444+0000 7fbd6abd2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6b290 0x556efcc7c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:39.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.445+0000 7fbd6abd2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc70400 0x556efcc647e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:39.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.445+0000 7fbd65ca5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556efcc6a8a0 0x556efcc745c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:39.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.445+0000 7fbd6abd2740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd440120a0 con 0x556efcc6a8a0 2024-03-28T01:07:39.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.445+0000 7fbd65ca5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556efcc6a8a0 0x556efcc745c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38324/0 (socket says 172.21.15.169:38324) 2024-03-28T01:07:39.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.445+0000 7fbd65ca5640 1 -- 172.21.15.169:0/472815326 learned_addr learned my addr 172.21.15.169:0/472815326 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:39.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.449+0000 7fbd664a6640 1 --2- 172.21.15.169:0/472815326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc70400 0x556efcc647e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:39.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.449+0000 7fbd6abd2740 1 -- 172.21.15.169:0/472815326 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4400fba0 con 0x556efcc6b290 2024-03-28T01:07:39.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.449+0000 7fbd6abd2740 1 -- 172.21.15.169:0/472815326 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4400dc40 con 0x556efcc70400 2024-03-28T01:07:39.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.449+0000 7fbd654a4640 1 --2- 172.21.15.169:0/472815326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6b290 0x556efcc7c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:39.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.453+0000 7fbd664a6640 1 -- 172.21.15.169:0/472815326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6b290 msgr2=0x556efcc7c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:39.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.453+0000 7fbd664a6640 1 --2- 172.21.15.169:0/472815326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6b290 0x556efcc7c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.453+0000 7fbd664a6640 1 -- 172.21.15.169:0/472815326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556efcc6a8a0 msgr2=0x556efcc745c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:39.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.453+0000 7fbd664a6640 1 --2- 172.21.15.169:0/472815326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556efcc6a8a0 0x556efcc745c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.457 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.453+0000 7fbd664a6640 1 -- 172.21.15.169:0/472815326 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556efcc6b670 con 0x556efcc70400 2024-03-28T01:07:39.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.454+0000 7fbd654a4640 1 --2- 172.21.15.169:0/472815326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6b290 0x556efcc7c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:07:39.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.455+0000 7fbd664a6640 1 --2- 172.21.15.169:0/472815326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc70400 0x556efcc647e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009550 tx=0x7fbd500092d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6166cff3c423630a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:39.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.455+0000 7fbd4b7fe640 1 -- 172.21.15.169:0/472815326 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5000b410 con 0x556efcc70400 2024-03-28T01:07:39.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.460+0000 7fbd4b7fe640 1 -- 172.21.15.169:0/472815326 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd5001e070 con 0x556efcc70400 2024-03-28T01:07:39.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.461+0000 7fbd6abd2740 1 -- 172.21.15.169:0/472815326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc70400 msgr2=0x556efcc647e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:39.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.461+0000 7fbd6abd2740 1 --2- 172.21.15.169:0/472815326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc70400 0x556efcc647e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009550 tx=0x7fbd500092d0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.461+0000 7fbd6abd2740 1 -- 172.21.15.169:0/472815326 shutdown_connections 2024-03-28T01:07:39.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.461+0000 7fbd6abd2740 1 --2- 172.21.15.169:0/472815326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc70400 0x556efcc647e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.461+0000 7fbd6abd2740 1 --2- 172.21.15.169:0/472815326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6b290 0x556efcc7c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.461+0000 7fbd6abd2740 1 --2- 172.21.15.169:0/472815326 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556efcc6a8a0 0x556efcc745c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.461+0000 7fbd6abd2740 1 -- 172.21.15.169:0/472815326 >> 172.21.15.169:0/472815326 conn(0x556efcc65e10 msgr2=0x556efcc73eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:39.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.464+0000 7fbd6abd2740 1 -- 172.21.15.169:0/472815326 shutdown_connections 2024-03-28T01:07:39.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.464+0000 7fbd6abd2740 1 -- 172.21.15.169:0/472815326 wait complete. 2024-03-28T01:07:39.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.464+0000 7fbd6abd2740 1 -- start start 2024-03-28T01:07:39.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.464+0000 7fbd6abd2740 1 Processor -- start 2024-03-28T01:07:39.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.465+0000 7fbd6abd2740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556efcc6a8a0 0x556efcc7bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:39.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.465+0000 7fbd6abd2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6b290 0x556efcc7c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:39.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.465+0000 7fbd6abd2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc70400 0x556efcc7a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:39.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.465+0000 7fbd6abd2740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4400e240 con 0x556efcc6a8a0 2024-03-28T01:07:39.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.465+0000 7fbd6abd2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4400f540 con 0x556efcc6b290 2024-03-28T01:07:39.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.465+0000 7fbd6abd2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4400e880 con 0x556efcc70400 2024-03-28T01:07:39.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.465+0000 7fbd6abd2740 10 client.0 _ll_register_callbacks cb 0x7fbd5c015da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:39.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.466+0000 7fbd654a4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6b290 0x556efcc7c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:39.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.466+0000 7fbd654a4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6b290 0x556efcc7c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:58728/0 (socket says 172.21.15.169:58728) 2024-03-28T01:07:39.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.466+0000 7fbd654a4640 1 -- 172.21.15.169:0/2067925960 learned_addr learned my addr 172.21.15.169:0/2067925960 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:39.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.466+0000 7fbd664a6640 1 --2- 172.21.15.169:0/2067925960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc70400 0x556efcc7a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:39.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.466+0000 7fbd65ca5640 1 --2- 172.21.15.169:0/2067925960 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556efcc6a8a0 0x556efcc7bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:39.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.466+0000 7fbd664a6640 1 -- 172.21.15.169:0/2067925960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6b290 msgr2=0x556efcc7c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:39.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.466+0000 7fbd664a6640 1 --2- 172.21.15.169:0/2067925960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6b290 0x556efcc7c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.466+0000 7fbd664a6640 1 -- 172.21.15.169:0/2067925960 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556efcc6a8a0 msgr2=0x556efcc7bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:39.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.466+0000 7fbd664a6640 1 --2- 172.21.15.169:0/2067925960 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556efcc6a8a0 0x556efcc7bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.466+0000 7fbd664a6640 1 -- 172.21.15.169:0/2067925960 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556efcc7aa60 con 0x556efcc70400 2024-03-28T01:07:39.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.467+0000 7fbd664a6640 1 --2- 172.21.15.169:0/2067925960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc70400 0x556efcc7a230 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009600 tx=0x7fbd50004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4281a8f892f34c26 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:39.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.468+0000 7fbd497fa640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:07:39.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.468+0000 7fbd497fa640 1 -- 172.21.15.169:0/2067925960 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50015040 con 0x556efcc70400 2024-03-28T01:07:39.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.468+0000 7fbd497fa640 1 -- 172.21.15.169:0/2067925960 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fbd5000b8e0 con 0x556efcc70400 2024-03-28T01:07:39.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.469+0000 7fbd6abd2740 1 -- 172.21.15.169:0/2067925960 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbd5001e070 con 0x556efcc70400 2024-03-28T01:07:39.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.474+0000 7fbd497fa640 1 -- 172.21.15.169:0/2067925960 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd5000bae0 con 0x556efcc70400 2024-03-28T01:07:39.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.475+0000 7fbd6abd2740 10 client.27355 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:39.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.475+0000 7fbd6abd2740 1 -- 172.21.15.169:0/2067925960 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556efccdd0c0 con 0x556efcc70400 2024-03-28T01:07:39.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.477+0000 7fbd497fa640 1 -- 172.21.15.169:0/2067925960 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd5001e070 con 0x556efcc70400 2024-03-28T01:07:39.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.477+0000 7fbd497fa640 1 -- 172.21.15.169:0/2067925960 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fbd50010980 con 0x556efcc70400 2024-03-28T01:07:39.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.477+0000 7fbd6abd2740 10 client.27355 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:39.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.477+0000 7fbd6abd2740 10 client.27355 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:07:39.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.477+0000 7fbd6abd2740 20 client.27355 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:39.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.477+0000 7fbd6abd2740 1 --2- 172.21.15.169:0/2067925960 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x556efccdd920 0x556efccdfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:39.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.477+0000 7fbd6abd2740 4 client.27355 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:07:39.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.477+0000 7fbd6abd2740 1 -- 172.21.15.169:0/2067925960 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556efcbd7c50 con 0x556efccdd920 2024-03-28T01:07:39.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.478+0000 7fbd65ca5640 1 --2- 172.21.15.169:0/2067925960 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x556efccdd920 0x556efccdfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:39.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.480+0000 7fbd65ca5640 1 --2- 172.21.15.169:0/2067925960 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x556efccdd920 0x556efccdfd40 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82a18eff6ee2243f server_cookie=725c233634a84b9 in_seq=0 out_seq=0 2024-03-28T01:07:39.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.481+0000 7fbd497fa640 10 client.27355 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:39.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.484+0000 7fbd497fa640 1 -- 172.21.15.169:0/2067925960 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556efcbd7c50 con 0x556efccdd920 2024-03-28T01:07:39.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.484+0000 7fbd497fa640 10 client.27355 handle_command_reply: tid=0 2024-03-28T01:07:39.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.484+0000 7fbd6abd2740 1 client.27355 shutdown 2024-03-28T01:07:39.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.485+0000 7fbd6abd2740 1 -- 172.21.15.169:0/2067925960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc70400 msgr2=0x556efcc7a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:39.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.485+0000 7fbd6abd2740 1 --2- 172.21.15.169:0/2067925960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc70400 0x556efcc7a230 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009600 tx=0x7fbd50004740 comp rx=0 tx=0).stop 2024-03-28T01:07:39.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.486+0000 7fbd6abd2740 1 -- 172.21.15.169:0/2067925960 shutdown_connections 2024-03-28T01:07:39.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.486+0000 7fbd6abd2740 1 --2- 172.21.15.169:0/2067925960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556efcc70400 0x556efcc7a230 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.486+0000 7fbd6abd2740 1 --2- 172.21.15.169:0/2067925960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556efcc6b290 0x556efcc7c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.486+0000 7fbd6abd2740 1 --2- 172.21.15.169:0/2067925960 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x556efccdd920 0x556efccdfd40 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.486+0000 7fbd6abd2740 1 --2- 172.21.15.169:0/2067925960 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556efcc6a8a0 0x556efcc7bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.486+0000 7fbd6abd2740 1 -- 172.21.15.169:0/2067925960 >> 172.21.15.169:0/2067925960 conn(0x556efcc65e10 msgr2=0x556efcc69070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:39.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.486+0000 7fbd497fa640 0 client.27355 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:39.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.486+0000 7fbd6abd2740 1 -- 172.21.15.169:0/2067925960 shutdown_connections 2024-03-28T01:07:39.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.486+0000 7fbd6abd2740 1 -- 172.21.15.169:0/2067925960 wait complete. 2024-03-28T01:07:39.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.486+0000 7fbd6abd2740 20 client.27355 trim_cache size 0 max 16384 2024-03-28T01:07:39.493 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:07:39.497 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.492+0000 7fbd66ca7640 1 -- 172.21.15.169:0/3840441458 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbd4c0a2a00 msgr2=0x7fbd4c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:39.497 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.492+0000 7fbd66ca7640 1 --2- 172.21.15.169:0/3840441458 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbd4c0a2a00 0x7fbd4c0a4ec0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fbd6019f460 tx=0x7fbd5c006c40 comp rx=0 tx=0).stop 2024-03-28T01:07:39.497 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.492+0000 7fbd66ca7640 1 -- 172.21.15.169:0/3840441458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd600729f0 msgr2=0x7fbd6019df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:39.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.492+0000 7fbd66ca7640 1 --2- 172.21.15.169:0/3840441458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd600729f0 0x7fbd6019df50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fbd5801add0 tx=0x7fbd58003ba0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.492+0000 7fbd66ca7640 1 -- 172.21.15.169:0/3840441458 shutdown_connections 2024-03-28T01:07:39.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.492+0000 7fbd66ca7640 1 --2- 172.21.15.169:0/3840441458 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbd4c0a2a00 0x7fbd4c0a4ec0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.492+0000 7fbd66ca7640 1 --2- 172.21.15.169:0/3840441458 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd60109640 0x7fbd601a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.492+0000 7fbd66ca7640 1 --2- 172.21.15.169:0/3840441458 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd60108ca0 0x7fbd6019e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.492+0000 7fbd66ca7640 1 --2- 172.21.15.169:0/3840441458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd600729f0 0x7fbd6019df50 secure :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fbd5801add0 tx=0x7fbd58003ba0 comp rx=0 tx=0).stop 2024-03-28T01:07:39.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.492+0000 7fbd66ca7640 1 -- 172.21.15.169:0/3840441458 >> 172.21.15.169:0/3840441458 conn(0x7fbd6006e4c0 msgr2=0x7fbd6010bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:39.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.493+0000 7fbd66ca7640 1 -- 172.21.15.169:0/3840441458 shutdown_connections 2024-03-28T01:07:39.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:39.493+0000 7fbd66ca7640 1 -- 172.21.15.169:0/3840441458 wait complete. 2024-03-28T01:07:39.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:39 smithi169 ceph-mon[32274]: pgmap v685: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 581 MiB/s wr, 685 op/s 2024-03-28T01:07:40.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:39 smithi174 ceph-mon[36060]: pgmap v685: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 581 MiB/s wr, 685 op/s 2024-03-28T01:07:40.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:39 smithi177 ceph-mon[36206]: pgmap v685: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 581 MiB/s wr, 685 op/s 2024-03-28T01:07:40.533 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T01:07:40.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T01:07:40.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T01:07:40.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.809+0000 7fa95281f640 1 -- 172.21.15.169:0/1308154008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa94c072ab0 msgr2=0x7fa94c072ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:40.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.809+0000 7fa95281f640 1 --2- 172.21.15.169:0/1308154008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa94c072ab0 0x7fa94c072ed0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa93c01a910 tx=0x7fa93c03cca0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.815+0000 7fa95281f640 1 -- 172.21.15.169:0/1308154008 shutdown_connections 2024-03-28T01:07:40.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.815+0000 7fa95281f640 1 --2- 172.21.15.169:0/1308154008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa94c109730 0x7fa94c10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.815+0000 7fa95281f640 1 --2- 172.21.15.169:0/1308154008 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa94c108d70 0x7fa94c1091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.815+0000 7fa95281f640 1 --2- 172.21.15.169:0/1308154008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa94c072ab0 0x7fa94c072ed0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.815+0000 7fa95281f640 1 -- 172.21.15.169:0/1308154008 >> 172.21.15.169:0/1308154008 conn(0x7fa94c06e580 msgr2=0x7fa94c0709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:40.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.817+0000 7fa95281f640 1 -- 172.21.15.169:0/1308154008 shutdown_connections 2024-03-28T01:07:40.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.819+0000 7fa95281f640 1 -- 172.21.15.169:0/1308154008 wait complete. 2024-03-28T01:07:40.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.820+0000 7fa95281f640 1 Processor -- start 2024-03-28T01:07:40.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.822+0000 7fa95281f640 1 -- start start 2024-03-28T01:07:40.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.822+0000 7fa95281f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa94c072ab0 0x7fa94c19e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:40.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.822+0000 7fa95281f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa94c108d70 0x7fa94c19e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:40.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.822+0000 7fa95281f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa94c109730 0x7fa94c1a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:40.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.822+0000 7fa95281f640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c111a90 con 0x7fa94c072ab0 2024-03-28T01:07:40.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.822+0000 7fa95281f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c111910 con 0x7fa94c108d70 2024-03-28T01:07:40.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.822+0000 7fa95281f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c111c10 con 0x7fa94c109730 2024-03-28T01:07:40.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.823+0000 7fa95201e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa94c109730 0x7fa94c1a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:40.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.823+0000 7fa95201e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa94c109730 0x7fa94c1a55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:47214/0 (socket says 172.21.15.169:47214) 2024-03-28T01:07:40.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.823+0000 7fa95201e640 1 -- 172.21.15.169:0/395895980 learned_addr learned my addr 172.21.15.169:0/395895980 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:40.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.823+0000 7fa95201e640 1 -- 172.21.15.169:0/395895980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa94c108d70 msgr2=0x7fa94c19e570 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:07:40.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.823+0000 7fa95201e640 1 --2- 172.21.15.169:0/395895980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa94c108d70 0x7fa94c19e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.823+0000 7fa95201e640 1 -- 172.21.15.169:0/395895980 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa94c072ab0 msgr2=0x7fa94c19e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:40.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.823+0000 7fa95201e640 1 --2- 172.21.15.169:0/395895980 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa94c072ab0 0x7fa94c19e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.823+0000 7fa95201e640 1 -- 172.21.15.169:0/395895980 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94c1a5c50 con 0x7fa94c109730 2024-03-28T01:07:40.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.823+0000 7fa95201e640 1 --2- 172.21.15.169:0/395895980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa94c109730 0x7fa94c1a55e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa94400b970 tx=0x7fa94400be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:40.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.830+0000 7fa942ffd640 1 -- 172.21.15.169:0/395895980 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa944019070 con 0x7fa94c109730 2024-03-28T01:07:40.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.830+0000 7fa942ffd640 1 -- 172.21.15.169:0/395895980 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa94400cb40 con 0x7fa94c109730 2024-03-28T01:07:40.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.830+0000 7fa942ffd640 1 -- 172.21.15.169:0/395895980 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9440076e0 con 0x7fa94c109730 2024-03-28T01:07:40.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.831+0000 7fa95281f640 1 -- 172.21.15.169:0/395895980 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa94c1a5f40 con 0x7fa94c109730 2024-03-28T01:07:40.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.831+0000 7fa95281f640 1 -- 172.21.15.169:0/395895980 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa94c1a63d0 con 0x7fa94c109730 2024-03-28T01:07:40.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.836+0000 7fa95666b740 1 Processor -- start 2024-03-28T01:07:40.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.838+0000 7fa95666b740 1 -- start start 2024-03-28T01:07:40.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.839+0000 7fa95666b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564fc1e58b30 0x564fc1e58f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:40.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.839+0000 7fa95666b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e59520 0x564fc1e5e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:40.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.839+0000 7fa95666b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e5e690 0x564fc1e60b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:40.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.839+0000 7fa95666b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fc1d98c90 con 0x564fc1e58b30 2024-03-28T01:07:40.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.839+0000 7fa95666b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fc1dbe070 con 0x564fc1e5e690 2024-03-28T01:07:40.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.839+0000 7fa95666b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fc1d66c80 con 0x564fc1e59520 2024-03-28T01:07:40.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.839+0000 7fa942ffd640 1 -- 172.21.15.169:0/395895980 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa94401c070 con 0x7fa94c109730 2024-03-28T01:07:40.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.840+0000 7fa95181d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e59520 0x564fc1e5e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:40.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.840+0000 7fa95181d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e59520 0x564fc1e5e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:47230/0 (socket says 172.21.15.169:47230) 2024-03-28T01:07:40.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.840+0000 7fa95181d640 1 -- 172.21.15.169:0/113496935 learned_addr learned my addr 172.21.15.169:0/113496935 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:40.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.840+0000 7fa95101c640 1 --2- 172.21.15.169:0/113496935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e5e690 0x564fc1e60b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:40.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.840+0000 7fa95181d640 1 -- 172.21.15.169:0/113496935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e5e690 msgr2=0x564fc1e60b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:40.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.840+0000 7fa95181d640 1 --2- 172.21.15.169:0/113496935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e5e690 0x564fc1e60b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.840+0000 7fa95181d640 1 -- 172.21.15.169:0/113496935 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564fc1e58b30 msgr2=0x564fc1e58f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:40.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.840+0000 7fa95181d640 1 --2- 172.21.15.169:0/113496935 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564fc1e58b30 0x564fc1e58f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.840+0000 7fa95181d640 1 -- 172.21.15.169:0/113496935 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564fc1d5ed10 con 0x564fc1e59520 2024-03-28T01:07:40.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.840+0000 7fa942ffd640 1 --2- 172.21.15.169:0/395895980 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa9380a29d0 0x7fa9380a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:40.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.840+0000 7fa95101c640 1 --2- 172.21.15.169:0/113496935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e5e690 0x564fc1e60b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:07:40.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.840+0000 7fa942ffd640 1 -- 172.21.15.169:0/395895980 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa9440ce3b0 con 0x7fa94c109730 2024-03-28T01:07:40.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.841+0000 7fa95181d640 1 --2- 172.21.15.169:0/113496935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e59520 0x564fc1e5e020 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa93c01a610 tx=0x7fa93c01a640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6eda7cdd590f2a7b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:40.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.842+0000 7fa95101c640 1 --2- 172.21.15.169:0/395895980 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa9380a29d0 0x7fa9380a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:40.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.844+0000 7fa95101c640 1 --2- 172.21.15.169:0/395895980 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa9380a29d0 0x7fa9380a4e90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa948008fd0 tx=0x7fa948008ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:40.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.851+0000 7fa937fff640 1 -- 172.21.15.169:0/113496935 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93c052020 con 0x564fc1e59520 2024-03-28T01:07:40.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.852+0000 7fa937fff640 1 -- 172.21.15.169:0/113496935 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa93c004b30 con 0x564fc1e59520 2024-03-28T01:07:40.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.856+0000 7fa937fff640 1 -- 172.21.15.169:0/113496935 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93c005210 con 0x564fc1e59520 2024-03-28T01:07:40.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.858+0000 7fa95666b740 1 -- 172.21.15.169:0/113496935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e59520 msgr2=0x564fc1e5e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:40.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.858+0000 7fa95666b740 1 --2- 172.21.15.169:0/113496935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e59520 0x564fc1e5e020 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa93c01a610 tx=0x7fa93c01a640 comp rx=0 tx=0).stop 2024-03-28T01:07:40.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.862+0000 7fa95666b740 1 -- 172.21.15.169:0/113496935 shutdown_connections 2024-03-28T01:07:40.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.862+0000 7fa95666b740 1 --2- 172.21.15.169:0/113496935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e5e690 0x564fc1e60b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.862+0000 7fa95666b740 1 --2- 172.21.15.169:0/113496935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e59520 0x564fc1e5e020 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.862+0000 7fa95666b740 1 --2- 172.21.15.169:0/113496935 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564fc1e58b30 0x564fc1e58f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.862+0000 7fa95666b740 1 -- 172.21.15.169:0/113496935 >> 172.21.15.169:0/113496935 conn(0x564fc1e211a0 msgr2=0x564fc1e544a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:40.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.863+0000 7fa95666b740 1 -- 172.21.15.169:0/113496935 shutdown_connections 2024-03-28T01:07:40.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.863+0000 7fa95666b740 1 -- 172.21.15.169:0/113496935 wait complete. 2024-03-28T01:07:40.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.864+0000 7fa95666b740 1 -- start start 2024-03-28T01:07:40.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.864+0000 7fa95666b740 1 Processor -- start 2024-03-28T01:07:40.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.864+0000 7fa95666b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564fc1e58b30 0x564fc1e62f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:40.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.864+0000 7fa95666b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e59520 0x564fc1e63460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:40.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.864+0000 7fa95666b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e5e690 0x564fc1e6a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:40.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.864+0000 7fa95666b740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fc1d7a0d0 con 0x564fc1e58b30 2024-03-28T01:07:40.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.865+0000 7fa95666b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fc1dba110 con 0x564fc1e59520 2024-03-28T01:07:40.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.865+0000 7fa95666b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fc1e615e0 con 0x564fc1e5e690 2024-03-28T01:07:40.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.865+0000 7fa95666b740 10 client.0 _ll_register_callbacks cb 0x564fc1db5e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:40.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.865+0000 7fa95201e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e5e690 0x564fc1e6a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:40.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.865+0000 7fa95201e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e5e690 0x564fc1e6a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:47244/0 (socket says 172.21.15.169:47244) 2024-03-28T01:07:40.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.865+0000 7fa95201e640 1 -- 172.21.15.169:0/401635639 learned_addr learned my addr 172.21.15.169:0/401635639 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:40.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.865+0000 7fa95101c640 1 --2- 172.21.15.169:0/401635639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e59520 0x564fc1e63460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:40.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.866+0000 7fa95101c640 1 -- 172.21.15.169:0/401635639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e5e690 msgr2=0x564fc1e6a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:40.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.866+0000 7fa95101c640 1 --2- 172.21.15.169:0/401635639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e5e690 0x564fc1e6a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.866+0000 7fa95101c640 1 -- 172.21.15.169:0/401635639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564fc1e58b30 msgr2=0x564fc1e62f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:40.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.866+0000 7fa95101c640 1 --2- 172.21.15.169:0/401635639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564fc1e58b30 0x564fc1e62f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.866+0000 7fa95101c640 1 -- 172.21.15.169:0/401635639 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564fc1b9a7e0 con 0x564fc1e59520 2024-03-28T01:07:40.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.869+0000 7fa95101c640 1 --2- 172.21.15.169:0/401635639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e59520 0x564fc1e63460 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x564fc1e59d80 tx=0x7fa948006880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=622f56031ab412d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:40.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.869+0000 7fa9357fa640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:07:40.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.869+0000 7fa9357fa640 1 -- 172.21.15.169:0/401635639 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94800ce40 con 0x564fc1e59520 2024-03-28T01:07:40.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.869+0000 7fa95666b740 1 -- 172.21.15.169:0/401635639 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa93c004b30 con 0x564fc1e59520 2024-03-28T01:07:40.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.869+0000 7fa9357fa640 1 -- 172.21.15.169:0/401635639 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa948016c40 con 0x564fc1e59520 2024-03-28T01:07:40.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.870+0000 7fa9357fa640 1 -- 172.21.15.169:0/401635639 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa94802a050 con 0x564fc1e59520 2024-03-28T01:07:40.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.870+0000 7fa95666b740 10 client.27410 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:40.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.870+0000 7fa95666b740 1 -- 172.21.15.169:0/401635639 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x564fc1bfaf70 con 0x564fc1e59520 2024-03-28T01:07:40.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.871+0000 7fa9357fa640 1 -- 172.21.15.169:0/401635639 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa93c004b30 con 0x564fc1e59520 2024-03-28T01:07:40.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.872+0000 7fa9357fa640 1 -- 172.21.15.169:0/401635639 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa948014410 con 0x564fc1e59520 2024-03-28T01:07:40.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.872+0000 7fa95666b740 10 client.27410 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:40.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.872+0000 7fa95666b740 10 client.27410 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:07:40.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.872+0000 7fa95666b740 20 client.27410 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:40.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.872+0000 7fa95666b740 1 --2- 172.21.15.169:0/401635639 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564fc1e6f170 0x564fc1e715f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:40.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.872+0000 7fa95666b740 4 client.27410 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:07:40.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.872+0000 7fa95666b740 1 -- 172.21.15.169:0/401635639 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564fc1dcb5b0 con 0x564fc1e6f170 2024-03-28T01:07:40.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.875+0000 7fa95181d640 1 --2- 172.21.15.169:0/401635639 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564fc1e6f170 0x564fc1e715f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:40.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.878+0000 7fa95181d640 1 --2- 172.21.15.169:0/401635639 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564fc1e6f170 0x564fc1e715f0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5afd4995e9e06b4a server_cookie=55b34aee052cd14d in_seq=0 out_seq=0 2024-03-28T01:07:40.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.879+0000 7fa9357fa640 10 client.27410 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:40.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.881+0000 7fa9357fa640 1 -- 172.21.15.169:0/401635639 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564fc1dcb5b0 con 0x564fc1e6f170 2024-03-28T01:07:40.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.881+0000 7fa9357fa640 10 client.27410 handle_command_reply: tid=0 2024-03-28T01:07:40.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.881+0000 7fa95666b740 1 client.27410 shutdown 2024-03-28T01:07:40.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.883+0000 7fa95666b740 1 -- 172.21.15.169:0/401635639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e59520 msgr2=0x564fc1e63460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:40.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.883+0000 7fa95666b740 1 --2- 172.21.15.169:0/401635639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e59520 0x564fc1e63460 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x564fc1e59d80 tx=0x7fa948006880 comp rx=0 tx=0).stop 2024-03-28T01:07:40.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.887+0000 7fa95666b740 1 -- 172.21.15.169:0/401635639 shutdown_connections 2024-03-28T01:07:40.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.887+0000 7fa95666b740 1 --2- 172.21.15.169:0/401635639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e5e690 0x564fc1e6a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.887+0000 7fa95666b740 1 --2- 172.21.15.169:0/401635639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e59520 0x564fc1e63460 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.887+0000 7fa95666b740 1 --2- 172.21.15.169:0/401635639 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564fc1e6f170 0x564fc1e715f0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.887+0000 7fa95666b740 1 --2- 172.21.15.169:0/401635639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564fc1e58b30 0x564fc1e62f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.887+0000 7fa95666b740 1 -- 172.21.15.169:0/401635639 >> 172.21.15.169:0/401635639 conn(0x564fc1e211a0 msgr2=0x564fc1e537e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:40.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.888+0000 7fa9357fa640 0 client.27410 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:40.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.892+0000 7fa95666b740 1 -- 172.21.15.169:0/401635639 shutdown_connections 2024-03-28T01:07:40.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.896+0000 7fa95666b740 1 -- 172.21.15.169:0/401635639 wait complete. 2024-03-28T01:07:40.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.901+0000 7fa95666b740 20 client.27410 trim_cache size 0 max 16384 2024-03-28T01:07:40.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.941+0000 7fa95666b740 1 Processor -- start 2024-03-28T01:07:40.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.942+0000 7fa95666b740 1 -- start start 2024-03-28T01:07:40.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.942+0000 7fa95666b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e58b30 0x564fc1e53660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:40.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.942+0000 7fa95666b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e59520 0x564fc1e6a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:40.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.942+0000 7fa95181d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e58b30 0x564fc1e53660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:40.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.942+0000 7fa95181d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e58b30 0x564fc1e53660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:47258/0 (socket says 172.21.15.169:47258) 2024-03-28T01:07:40.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.943+0000 7fa95101c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e59520 0x564fc1e6a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:40.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.943+0000 7fa95101c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e59520 0x564fc1e6a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:58768/0 (socket says 172.21.15.169:58768) 2024-03-28T01:07:40.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.943+0000 7fa95181d640 1 -- 172.21.15.169:0/3984762934 learned_addr learned my addr 172.21.15.169:0/3984762934 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:40.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.943+0000 7fa95666b740 1 --2- 172.21.15.169:0/3984762934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564fc1e5e690 0x564fc1e75ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:40.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.943+0000 7fa95666b740 1 -- 172.21.15.169:0/3984762934 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c1b2de0 con 0x564fc1e5e690 2024-03-28T01:07:40.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.943+0000 7fa95666b740 1 -- 172.21.15.169:0/3984762934 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c1b08e0 con 0x564fc1e59520 2024-03-28T01:07:40.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.943+0000 7fa95201e640 1 --2- 172.21.15.169:0/3984762934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564fc1e5e690 0x564fc1e75ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:40.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.943+0000 7fa95666b740 1 -- 172.21.15.169:0/3984762934 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c1ae980 con 0x564fc1e58b30 2024-03-28T01:07:40.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.945+0000 7fa95201e640 1 -- 172.21.15.169:0/3984762934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e58b30 msgr2=0x564fc1e53660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:40.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.945+0000 7fa95201e640 1 --2- 172.21.15.169:0/3984762934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e58b30 0x564fc1e53660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.945+0000 7fa95201e640 1 -- 172.21.15.169:0/3984762934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e59520 msgr2=0x564fc1e6a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:40.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.945+0000 7fa95201e640 1 --2- 172.21.15.169:0/3984762934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e59520 0x564fc1e6a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.945+0000 7fa95201e640 1 -- 172.21.15.169:0/3984762934 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564fc1a92170 con 0x564fc1e5e690 2024-03-28T01:07:40.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.945+0000 7fa95201e640 1 --2- 172.21.15.169:0/3984762934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564fc1e5e690 0x564fc1e75ef0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fa944097320 tx=0x7fa94409a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be12a7abfd344be0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:40.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.946+0000 7fa9377fe640 1 -- 172.21.15.169:0/3984762934 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9440a30f0 con 0x564fc1e5e690 2024-03-28T01:07:40.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.946+0000 7fa9377fe640 1 -- 172.21.15.169:0/3984762934 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9440b3070 con 0x564fc1e5e690 2024-03-28T01:07:40.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.946+0000 7fa9377fe640 1 -- 172.21.15.169:0/3984762934 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa944096fc0 con 0x564fc1e5e690 2024-03-28T01:07:40.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.949+0000 7fa95666b740 1 -- 172.21.15.169:0/3984762934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564fc1e5e690 msgr2=0x564fc1e75ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:40.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.949+0000 7fa95666b740 1 --2- 172.21.15.169:0/3984762934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564fc1e5e690 0x564fc1e75ef0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fa944097320 tx=0x7fa94409a7d0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.950+0000 7fa95666b740 1 -- 172.21.15.169:0/3984762934 shutdown_connections 2024-03-28T01:07:40.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.950+0000 7fa95666b740 1 --2- 172.21.15.169:0/3984762934 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564fc1e5e690 0x564fc1e75ef0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.950+0000 7fa95666b740 1 --2- 172.21.15.169:0/3984762934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e59520 0x564fc1e6a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.950+0000 7fa95666b740 1 --2- 172.21.15.169:0/3984762934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e58b30 0x564fc1e53660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.950+0000 7fa95666b740 1 -- 172.21.15.169:0/3984762934 >> 172.21.15.169:0/3984762934 conn(0x564fc1e211a0 msgr2=0x564fc1e52f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:40.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.950+0000 7fa95666b740 1 -- 172.21.15.169:0/3984762934 shutdown_connections 2024-03-28T01:07:40.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.950+0000 7fa95666b740 1 -- 172.21.15.169:0/3984762934 wait complete. 2024-03-28T01:07:40.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.951+0000 7fa95666b740 1 -- start start 2024-03-28T01:07:40.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.951+0000 7fa95666b740 1 Processor -- start 2024-03-28T01:07:40.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.952+0000 7fa95666b740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564fc1e58b30 0x564fc1e678d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:40.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.952+0000 7fa95666b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e59520 0x564fc1e67e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:40.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.952+0000 7fa95666b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e5e690 0x564fc1e68db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:40.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.953+0000 7fa95101c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e59520 0x564fc1e67e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:40.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.953+0000 7fa95101c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e59520 0x564fc1e67e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:47270/0 (socket says 172.21.15.169:47270) 2024-03-28T01:07:40.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.953+0000 7fa95101c640 1 -- 172.21.15.169:0/1130029020 learned_addr learned my addr 172.21.15.169:0/1130029020 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:40.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.953+0000 7fa95666b740 1 -- 172.21.15.169:0/1130029020 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c1aef80 con 0x564fc1e58b30 2024-03-28T01:07:40.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.953+0000 7fa95201e640 1 --2- 172.21.15.169:0/1130029020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e5e690 0x564fc1e68db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:40.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.953+0000 7fa95666b740 1 -- 172.21.15.169:0/1130029020 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c1b0280 con 0x564fc1e5e690 2024-03-28T01:07:40.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.953+0000 7fa95666b740 1 -- 172.21.15.169:0/1130029020 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c1af5c0 con 0x564fc1e59520 2024-03-28T01:07:40.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.953+0000 7fa95666b740 10 client.0 _ll_register_callbacks cb 0x7fa948004d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:40.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.953+0000 7fa95101c640 1 -- 172.21.15.169:0/1130029020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e5e690 msgr2=0x564fc1e68db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:40.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.953+0000 7fa95101c640 1 --2- 172.21.15.169:0/1130029020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e5e690 0x564fc1e68db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.954+0000 7fa95181d640 1 --2- 172.21.15.169:0/1130029020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564fc1e58b30 0x564fc1e678d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:40.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.954+0000 7fa95101c640 1 -- 172.21.15.169:0/1130029020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564fc1e58b30 msgr2=0x564fc1e678d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:40.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.954+0000 7fa95101c640 1 --2- 172.21.15.169:0/1130029020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564fc1e58b30 0x564fc1e678d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:40.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.954+0000 7fa95101c640 1 -- 172.21.15.169:0/1130029020 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564fc1e693a0 con 0x564fc1e59520 2024-03-28T01:07:40.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.954+0000 7fa95181d640 1 --2- 172.21.15.169:0/1130029020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564fc1e58b30 0x564fc1e678d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:07:40.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.954+0000 7fa95666b740 1 -- 172.21.15.169:0/1130029020 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa9440b3070 con 0x564fc1e59520 2024-03-28T01:07:40.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.954+0000 7fa95101c640 1 --2- 172.21.15.169:0/1130029020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e59520 0x564fc1e67e10 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa948004f80 tx=0x7fa94800c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ececb6079d9ef1e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:40.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.958+0000 7fa9357fa640 10 client.27370 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:07:40.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.958+0000 7fa9357fa640 1 -- 172.21.15.169:0/1130029020 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa948016dd0 con 0x564fc1e59520 2024-03-28T01:07:40.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.958+0000 7fa9357fa640 1 -- 172.21.15.169:0/1130029020 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fa9480149f0 con 0x564fc1e59520 2024-03-28T01:07:40.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.959+0000 7fa9357fa640 1 -- 172.21.15.169:0/1130029020 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa948014c10 con 0x564fc1e59520 2024-03-28T01:07:40.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.959+0000 7fa95666b740 10 client.27370 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:40.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.959+0000 7fa95666b740 1 -- 172.21.15.169:0/1130029020 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x564fc1e6a600 con 0x564fc1e59520 2024-03-28T01:07:40.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.962+0000 7fa9357fa640 1 -- 172.21.15.169:0/1130029020 <== mon.1 v2:172.21.15.177:3300/0 4 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fa948004040 con 0x564fc1e59520 2024-03-28T01:07:40.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.963+0000 7fa95666b740 10 client.27370 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:40.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.963+0000 7fa95666b740 10 client.27370 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:07:40.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.963+0000 7fa95666b740 20 client.27370 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:40.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.963+0000 7fa95666b740 1 --2- 172.21.15.169:0/1130029020 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564fc1ecaef0 0x564fc1ecd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:40.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.963+0000 7fa95666b740 4 client.27370 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T01:07:40.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.963+0000 7fa95666b740 1 -- 172.21.15.169:0/1130029020 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x564fc1dc5900 con 0x564fc1ecaef0 2024-03-28T01:07:40.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.963+0000 7fa95181d640 1 --2- 172.21.15.169:0/1130029020 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564fc1ecaef0 0x564fc1ecd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:40.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.969+0000 7fa95181d640 1 --2- 172.21.15.169:0/1130029020 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564fc1ecaef0 0x564fc1ecd2e0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4093636c16b9d5b server_cookie=59b8a5fbec515179 in_seq=0 out_seq=0 2024-03-28T01:07:40.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:40.969+0000 7fa9357fa640 10 client.27370 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:41.321 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.318+0000 7fa9357fa640 1 -- 172.21.15.169:0/1130029020 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x564fc1dc5900 con 0x564fc1ecaef0 2024-03-28T01:07:41.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.318+0000 7fa9357fa640 10 client.27370 handle_command_reply: tid=0 2024-03-28T01:07:41.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.318+0000 7fa95666b740 1 client.27370 shutdown 2024-03-28T01:07:41.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.318+0000 7fa95666b740 1 -- 172.21.15.169:0/1130029020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e59520 msgr2=0x564fc1e67e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:41.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.319+0000 7fa95666b740 1 --2- 172.21.15.169:0/1130029020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e59520 0x564fc1e67e10 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa948004f80 tx=0x7fa94800c560 comp rx=0 tx=0).stop 2024-03-28T01:07:41.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.319+0000 7fa95666b740 1 -- 172.21.15.169:0/1130029020 shutdown_connections 2024-03-28T01:07:41.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.319+0000 7fa95666b740 1 --2- 172.21.15.169:0/1130029020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fc1e5e690 0x564fc1e68db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.319+0000 7fa95666b740 1 --2- 172.21.15.169:0/1130029020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fc1e59520 0x564fc1e67e10 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.319+0000 7fa95666b740 1 --2- 172.21.15.169:0/1130029020 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x564fc1ecaef0 0x564fc1ecd2e0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.319+0000 7fa95666b740 1 --2- 172.21.15.169:0/1130029020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x564fc1e58b30 0x564fc1e678d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.319+0000 7fa9357fa640 0 client.27370 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:41.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.319+0000 7fa95666b740 1 -- 172.21.15.169:0/1130029020 >> 172.21.15.169:0/1130029020 conn(0x564fc1e211a0 msgr2=0x564fc1e537e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:41.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.319+0000 7fa95666b740 1 -- 172.21.15.169:0/1130029020 shutdown_connections 2024-03-28T01:07:41.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.319+0000 7fa95666b740 1 -- 172.21.15.169:0/1130029020 wait complete. 2024-03-28T01:07:41.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.319+0000 7fa95666b740 20 client.27370 trim_cache size 0 max 16384 2024-03-28T01:07:41.324 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:07:41.324 INFO:teuthology.orchestra.run.smithi169.stdout: "return_code": 0, 2024-03-28T01:07:41.324 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_tag": "a734cdcd-bb7b-42d1-9000-2598b06563d8", 2024-03-28T01:07:41.324 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": "asynchronous" 2024-03-28T01:07:41.324 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:07:41.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.320+0000 7fa95281f640 1 -- 172.21.15.169:0/395895980 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa9380a29d0 msgr2=0x7fa9380a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:41.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.320+0000 7fa95281f640 1 --2- 172.21.15.169:0/395895980 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa9380a29d0 0x7fa9380a4e90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa948008fd0 tx=0x7fa948008ea0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.320+0000 7fa95281f640 1 -- 172.21.15.169:0/395895980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa94c109730 msgr2=0x7fa94c1a55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:41.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.320+0000 7fa95281f640 1 --2- 172.21.15.169:0/395895980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa94c109730 0x7fa94c1a55e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa94400b970 tx=0x7fa94400be30 comp rx=0 tx=0).stop 2024-03-28T01:07:41.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.320+0000 7fa95281f640 1 -- 172.21.15.169:0/395895980 shutdown_connections 2024-03-28T01:07:41.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.320+0000 7fa95281f640 1 --2- 172.21.15.169:0/395895980 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fa9380a29d0 0x7fa9380a4e90 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.320+0000 7fa95281f640 1 --2- 172.21.15.169:0/395895980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa94c109730 0x7fa94c1a55e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.320+0000 7fa95281f640 1 --2- 172.21.15.169:0/395895980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa94c108d70 0x7fa94c19e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.320+0000 7fa95281f640 1 --2- 172.21.15.169:0/395895980 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa94c072ab0 0x7fa94c19e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.320+0000 7fa95281f640 1 -- 172.21.15.169:0/395895980 >> 172.21.15.169:0/395895980 conn(0x7fa94c06e580 msgr2=0x7fa94c10c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:41.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.320+0000 7fa95281f640 1 -- 172.21.15.169:0/395895980 shutdown_connections 2024-03-28T01:07:41.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.320+0000 7fa95281f640 1 -- 172.21.15.169:0/395895980 wait complete. 2024-03-28T01:07:41.341 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:07:41.341 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:07:41.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.615+0000 7fbff1a75640 1 -- 172.21.15.169:0/1314607162 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfec073550 msgr2=0x7fbfec073970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:41.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.615+0000 7fbff1a75640 1 --2- 172.21.15.169:0/1314607162 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfec073550 0x7fbfec073970 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc009900 tx=0x7fbfdc02bc80 comp rx=0 tx=0).stop 2024-03-28T01:07:41.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.618+0000 7fbff1a75640 1 -- 172.21.15.169:0/1314607162 shutdown_connections 2024-03-28T01:07:41.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.618+0000 7fbff1a75640 1 --2- 172.21.15.169:0/1314607162 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfec078de0 0x7fbfec07b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.618+0000 7fbff1a75640 1 --2- 172.21.15.169:0/1314607162 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfec076430 0x7fbfec0788a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.618+0000 7fbff1a75640 1 --2- 172.21.15.169:0/1314607162 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfec073550 0x7fbfec073970 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.618+0000 7fbff1a75640 1 -- 172.21.15.169:0/1314607162 >> 172.21.15.169:0/1314607162 conn(0x7fbfec06e500 msgr2=0x7fbfec070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:41.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.619+0000 7fbff1a75640 1 -- 172.21.15.169:0/1314607162 shutdown_connections 2024-03-28T01:07:41.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.623+0000 7fbff1a75640 1 -- 172.21.15.169:0/1314607162 wait complete. 2024-03-28T01:07:41.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.623+0000 7fbff1a75640 1 Processor -- start 2024-03-28T01:07:41.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.627+0000 7fbff1a75640 1 -- start start 2024-03-28T01:07:41.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.627+0000 7fbff1a75640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfec073550 0x7fbfec082ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:41.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.627+0000 7fbff1a75640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfec076430 0x7fbfec0831e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:41.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.627+0000 7fbff1a75640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfec078de0 0x7fbfec083720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:41.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.627+0000 7fbff1a75640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec12b4b0 con 0x7fbfec073550 2024-03-28T01:07:41.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.627+0000 7fbff1a75640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec12b330 con 0x7fbfec076430 2024-03-28T01:07:41.633 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.627+0000 7fbff1a75640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec12b630 con 0x7fbfec078de0 2024-03-28T01:07:41.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.628+0000 7fbfebfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfec078de0 0x7fbfec083720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:41.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.628+0000 7fbfebfff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfec078de0 0x7fbfec083720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:47296/0 (socket says 172.21.15.169:47296) 2024-03-28T01:07:41.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.628+0000 7fbfebfff640 1 -- 172.21.15.169:0/816127661 learned_addr learned my addr 172.21.15.169:0/816127661 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:41.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.628+0000 7fbfebfff640 1 -- 172.21.15.169:0/816127661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfec076430 msgr2=0x7fbfec0831e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T01:07:41.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.628+0000 7fbfebfff640 1 --2- 172.21.15.169:0/816127661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfec076430 0x7fbfec0831e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.628+0000 7fbfebfff640 1 -- 172.21.15.169:0/816127661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfec073550 msgr2=0x7fbfec082ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:41.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.628+0000 7fbfebfff640 1 --2- 172.21.15.169:0/816127661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfec073550 0x7fbfec082ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.628+0000 7fbfebfff640 1 -- 172.21.15.169:0/816127661 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec1320a0 con 0x7fbfec078de0 2024-03-28T01:07:41.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.628+0000 7fbfebfff640 1 --2- 172.21.15.169:0/816127661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfec078de0 0x7fbfec083720 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400b9e0 tx=0x7fbfe400bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:41.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.629+0000 7fbfe97fa640 1 -- 172.21.15.169:0/816127661 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4019070 con 0x7fbfec078de0 2024-03-28T01:07:41.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.629+0000 7fbfe97fa640 1 -- 172.21.15.169:0/816127661 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbfe4014440 con 0x7fbfec078de0 2024-03-28T01:07:41.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.629+0000 7fbfe97fa640 1 -- 172.21.15.169:0/816127661 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4012620 con 0x7fbfec078de0 2024-03-28T01:07:41.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.631+0000 7fbff1a75640 1 -- 172.21.15.169:0/816127661 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfec132390 con 0x7fbfec078de0 2024-03-28T01:07:41.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.631+0000 7fbff1a75640 1 -- 172.21.15.169:0/816127661 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfec132860 con 0x7fbfec078de0 2024-03-28T01:07:41.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.634+0000 7fbfe97fa640 1 -- 172.21.15.169:0/816127661 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbfe4014980 con 0x7fbfec078de0 2024-03-28T01:07:41.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.635+0000 7fbff4659740 1 Processor -- start 2024-03-28T01:07:41.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.635+0000 7fbff4659740 1 -- start start 2024-03-28T01:07:41.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.635+0000 7fbff4659740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aa9da0 0x556db6aaa1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:41.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.635+0000 7fbff4659740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db6aaa790 0x556db6aaf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:41.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.635+0000 7fbff4659740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556db6aaf900 0x556db6ab1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:41.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.635+0000 7fbff4659740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db6a0f3d0 con 0x556db6aaf900 2024-03-28T01:07:41.640 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.635+0000 7fbff4659740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db69b7ad0 con 0x556db6aa9da0 2024-03-28T01:07:41.640 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.635+0000 7fbff4659740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db6a0b470 con 0x556db6aaa790 2024-03-28T01:07:41.640 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.635+0000 7fbfe97fa640 1 --2- 172.21.15.169:0/816127661 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbfd80a2950 0x7fbfd80a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:41.640 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.636+0000 7fbfe97fa640 1 -- 172.21.15.169:0/816127661 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fbfe40ce3d0 con 0x7fbfec078de0 2024-03-28T01:07:41.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.637+0000 7fbfe2ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aa9da0 0x556db6aaa1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:41.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.637+0000 7fbfe2ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aa9da0 0x556db6aaa1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:58796/0 (socket says 172.21.15.169:58796) 2024-03-28T01:07:41.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.637+0000 7fbfe2ffd640 1 -- 172.21.15.169:0/3854866823 learned_addr learned my addr 172.21.15.169:0/3854866823 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:41.641 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.637+0000 7fbfe2ffd640 1 --2- 172.21.15.169:0/3854866823 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556db6aaf900 0x556db6ab1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:41.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.638+0000 7fbfeb7fe640 1 --2- 172.21.15.169:0/3854866823 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db6aaa790 0x556db6aaf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:41.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.638+0000 7fbfe2ffd640 1 -- 172.21.15.169:0/3854866823 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db6aaa790 msgr2=0x556db6aaf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:41.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.638+0000 7fbfe2ffd640 1 --2- 172.21.15.169:0/3854866823 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db6aaa790 0x556db6aaf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.638+0000 7fbfe2ffd640 1 -- 172.21.15.169:0/3854866823 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556db6aaf900 msgr2=0x556db6ab1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:41.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.639+0000 7fbfe2ffd640 1 --2- 172.21.15.169:0/3854866823 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556db6aaf900 0x556db6ab1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.639+0000 7fbfe2ffd640 1 -- 172.21.15.169:0/3854866823 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556db67ed7d0 con 0x556db6aa9da0 2024-03-28T01:07:41.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.639+0000 7fbfebfff640 1 --2- 172.21.15.169:0/816127661 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbfd80a2950 0x7fbfd80a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:41.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.645+0000 7fbfe2ffd640 1 --2- 172.21.15.169:0/3854866823 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aa9da0 0x556db6aaa1c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fbfd40143a0 tx=0x7fbfd400b0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5329e02a77faadd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:41.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.646+0000 7fbfe17fa640 1 -- 172.21.15.169:0/3854866823 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4011040 con 0x556db6aa9da0 2024-03-28T01:07:41.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.647+0000 7fbfe17fa640 1 -- 172.21.15.169:0/3854866823 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbfd40065b0 con 0x556db6aa9da0 2024-03-28T01:07:41.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.647+0000 7fbfe17fa640 1 -- 172.21.15.169:0/3854866823 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4018690 con 0x556db6aa9da0 2024-03-28T01:07:41.656 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.651+0000 7fbff4659740 1 -- 172.21.15.169:0/3854866823 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aa9da0 msgr2=0x556db6aaa1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:41.656 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.651+0000 7fbff4659740 1 --2- 172.21.15.169:0/3854866823 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aa9da0 0x556db6aaa1c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fbfd40143a0 tx=0x7fbfd400b0a0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.655+0000 7fbfebfff640 1 --2- 172.21.15.169:0/816127661 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbfd80a2950 0x7fbfd80a4e10 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4096dd0 tx=0x7fbfe40ce620 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:41.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.655+0000 7fbff4659740 1 -- 172.21.15.169:0/3854866823 shutdown_connections 2024-03-28T01:07:41.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.655+0000 7fbff4659740 1 --2- 172.21.15.169:0/3854866823 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556db6aaf900 0x556db6ab1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.655+0000 7fbff4659740 1 --2- 172.21.15.169:0/3854866823 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db6aaa790 0x556db6aaf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.655+0000 7fbff4659740 1 --2- 172.21.15.169:0/3854866823 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aa9da0 0x556db6aaa1c0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.655+0000 7fbff4659740 1 -- 172.21.15.169:0/3854866823 >> 172.21.15.169:0/3854866823 conn(0x556db6aa5600 msgr2=0x556db6aa5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:41.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.657+0000 7fbff4659740 1 -- 172.21.15.169:0/3854866823 shutdown_connections 2024-03-28T01:07:41.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.659+0000 7fbff4659740 1 -- 172.21.15.169:0/3854866823 wait complete. 2024-03-28T01:07:41.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.660+0000 7fbff4659740 1 -- start start 2024-03-28T01:07:41.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.660+0000 7fbff4659740 1 Processor -- start 2024-03-28T01:07:41.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.660+0000 7fbff4659740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556db6aa9da0 0x556db6ab4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:41.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.660+0000 7fbff4659740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aaa790 0x556db6ab4790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:41.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.660+0000 7fbff4659740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db6aaf900 0x556db6abb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:41.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.660+0000 7fbff4659740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db6ab27f0 con 0x556db6aa9da0 2024-03-28T01:07:41.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.660+0000 7fbff4659740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db69cb060 con 0x556db6aaa790 2024-03-28T01:07:41.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.660+0000 7fbff4659740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db6ab2970 con 0x556db6aaf900 2024-03-28T01:07:41.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.660+0000 7fbff4659740 10 client.0 _ll_register_callbacks cb 0x556db69ad6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:41.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.661+0000 7fbfeb7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aaa790 0x556db6ab4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:41.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.661+0000 7fbfeb7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aaa790 0x556db6ab4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:58800/0 (socket says 172.21.15.169:58800) 2024-03-28T01:07:41.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.661+0000 7fbfeb7fe640 1 -- 172.21.15.169:0/2914693030 learned_addr learned my addr 172.21.15.169:0/2914693030 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:41.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.661+0000 7fbfeb7fe640 1 -- 172.21.15.169:0/2914693030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db6aaf900 msgr2=0x556db6abb760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:07:41.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.661+0000 7fbfeb7fe640 1 --2- 172.21.15.169:0/2914693030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db6aaf900 0x556db6abb760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.661+0000 7fbfeb7fe640 1 -- 172.21.15.169:0/2914693030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556db6aa9da0 msgr2=0x556db6ab4250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:07:41.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.661+0000 7fbfeb7fe640 1 --2- 172.21.15.169:0/2914693030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556db6aa9da0 0x556db6ab4250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.661+0000 7fbfeb7fe640 1 -- 172.21.15.169:0/2914693030 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556db67233f0 con 0x556db6aaa790 2024-03-28T01:07:41.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.662+0000 7fbfeb7fe640 1 --2- 172.21.15.169:0/2914693030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aaa790 0x556db6ab4790 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc02cfd0 tx=0x7fbfdc004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bda712ced0ee2c90 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:41.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.662+0000 7fbfcaffd640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:07:41.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.664+0000 7fbfcaffd640 1 -- 172.21.15.169:0/2914693030 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc042030 con 0x556db6aaa790 2024-03-28T01:07:41.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.664+0000 7fbff4659740 1 -- 172.21.15.169:0/2914693030 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbfd40065b0 con 0x556db6aaa790 2024-03-28T01:07:41.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.664+0000 7fbfcaffd640 1 -- 172.21.15.169:0/2914693030 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fbfdc005000 con 0x556db6aaa790 2024-03-28T01:07:41.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.664+0000 7fbfcaffd640 1 -- 172.21.15.169:0/2914693030 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbfdc04f050 con 0x556db6aaa790 2024-03-28T01:07:41.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.666+0000 7fbff4659740 10 client.27422 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:41.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.666+0000 7fbff4659740 1 -- 172.21.15.169:0/2914693030 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556db66e30f0 con 0x556db6aaa790 2024-03-28T01:07:41.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.666+0000 7fbfcaffd640 1 -- 172.21.15.169:0/2914693030 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbfd40065b0 con 0x556db6aaa790 2024-03-28T01:07:41.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.666+0000 7fbfcaffd640 1 -- 172.21.15.169:0/2914693030 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fbfdc0035f0 con 0x556db6aaa790 2024-03-28T01:07:41.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.666+0000 7fbff4659740 10 client.27422 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:41.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.666+0000 7fbff4659740 10 client.27422 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:07:41.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.666+0000 7fbff4659740 20 client.27422 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:41.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.666+0000 7fbff4659740 1 --2- 172.21.15.169:0/2914693030 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x556db6abfc80 0x556db6ac2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:41.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.666+0000 7fbff4659740 4 client.27422 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:07:41.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.666+0000 7fbff4659740 1 -- 172.21.15.169:0/2914693030 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556db6a1c910 con 0x556db6abfc80 2024-03-28T01:07:41.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.667+0000 7fbfe2ffd640 1 --2- 172.21.15.169:0/2914693030 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x556db6abfc80 0x556db6ac2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:41.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.668+0000 7fbfe2ffd640 1 --2- 172.21.15.169:0/2914693030 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x556db6abfc80 0x556db6ac2100 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d8cd5773bf35203 server_cookie=fbd7ce743b0b1fa4 in_seq=0 out_seq=0 2024-03-28T01:07:41.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.668+0000 7fbfcaffd640 10 client.27422 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:41.675 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.672+0000 7fbfcaffd640 1 -- 172.21.15.169:0/2914693030 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556db6a1c910 con 0x556db6abfc80 2024-03-28T01:07:41.675 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.672+0000 7fbfcaffd640 10 client.27422 handle_command_reply: tid=0 2024-03-28T01:07:41.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.678+0000 7fbff4659740 1 client.27422 shutdown 2024-03-28T01:07:41.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.680+0000 7fbff4659740 1 -- 172.21.15.169:0/2914693030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aaa790 msgr2=0x556db6ab4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:41.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.680+0000 7fbff4659740 1 --2- 172.21.15.169:0/2914693030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aaa790 0x556db6ab4790 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc02cfd0 tx=0x7fbfdc004270 comp rx=0 tx=0).stop 2024-03-28T01:07:41.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.680+0000 7fbff4659740 1 -- 172.21.15.169:0/2914693030 shutdown_connections 2024-03-28T01:07:41.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.680+0000 7fbff4659740 1 --2- 172.21.15.169:0/2914693030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db6aaf900 0x556db6abb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.680+0000 7fbff4659740 1 --2- 172.21.15.169:0/2914693030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aaa790 0x556db6ab4790 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.680+0000 7fbff4659740 1 --2- 172.21.15.169:0/2914693030 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x556db6abfc80 0x556db6ac2100 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.680+0000 7fbff4659740 1 --2- 172.21.15.169:0/2914693030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556db6aa9da0 0x556db6ab4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.680+0000 7fbff4659740 1 -- 172.21.15.169:0/2914693030 >> 172.21.15.169:0/2914693030 conn(0x556db6aa5600 msgr2=0x556db6aa4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:41.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.680+0000 7fbfcaffd640 0 client.27422 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:41.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.680+0000 7fbff4659740 1 -- 172.21.15.169:0/2914693030 shutdown_connections 2024-03-28T01:07:41.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.681+0000 7fbff4659740 1 -- 172.21.15.169:0/2914693030 wait complete. 2024-03-28T01:07:41.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.681+0000 7fbff4659740 20 client.27422 trim_cache size 0 max 16384 2024-03-28T01:07:41.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.722+0000 7fbff4659740 1 Processor -- start 2024-03-28T01:07:41.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.722+0000 7fbff4659740 1 -- start start 2024-03-28T01:07:41.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.723+0000 7fbff4659740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aa9da0 0x556db6aa59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:41.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.723+0000 7fbff4659740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db6aaa790 0x556db6abb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:41.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.723+0000 7fbff4659740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556db6aaf900 0x556db6aabcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:41.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.723+0000 7fbff4659740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec1d4e40 con 0x556db6aaf900 2024-03-28T01:07:41.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.723+0000 7fbff4659740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec1d2940 con 0x556db6aa9da0 2024-03-28T01:07:41.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.723+0000 7fbff4659740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec1d09e0 con 0x556db6aaa790 2024-03-28T01:07:41.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.725+0000 7fbfe2ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aa9da0 0x556db6aa59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:41.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.725+0000 7fbfe2ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aa9da0 0x556db6aa59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:58804/0 (socket says 172.21.15.169:58804) 2024-03-28T01:07:41.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.725+0000 7fbfe2ffd640 1 -- 172.21.15.169:0/3219369438 learned_addr learned my addr 172.21.15.169:0/3219369438 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:41.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.725+0000 7fbfeb7fe640 1 --2- 172.21.15.169:0/3219369438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db6aaa790 0x556db6abb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:41.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.728+0000 7fbfe2ffd640 1 -- 172.21.15.169:0/3219369438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db6aaa790 msgr2=0x556db6abb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:41.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.728+0000 7fbfe2ffd640 1 --2- 172.21.15.169:0/3219369438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db6aaa790 0x556db6abb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.728+0000 7fbfe2ffd640 1 -- 172.21.15.169:0/3219369438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556db6aaf900 msgr2=0x556db6aabcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:41.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.728+0000 7fbfe2ffd640 1 --2- 172.21.15.169:0/3219369438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556db6aaf900 0x556db6aabcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.728+0000 7fbfe2ffd640 1 -- 172.21.15.169:0/3219369438 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556db6aaefc0 con 0x556db6aa9da0 2024-03-28T01:07:41.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.728+0000 7fbfe2ffd640 1 --2- 172.21.15.169:0/3219369438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aa9da0 0x556db6aa59e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fbfd40143a0 tx=0x7fbfd4008ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6a5ea7092f114be server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:41.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.731+0000 7fbff1a75640 1 -- 172.21.15.169:0/3219369438 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4014cf0 con 0x556db6aa9da0 2024-03-28T01:07:41.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.731+0000 7fbff1a75640 1 -- 172.21.15.169:0/3219369438 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbfd402a020 con 0x556db6aa9da0 2024-03-28T01:07:41.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.731+0000 7fbff1a75640 1 -- 172.21.15.169:0/3219369438 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd40269d0 con 0x556db6aa9da0 2024-03-28T01:07:41.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.732+0000 7fbff4659740 1 -- 172.21.15.169:0/3219369438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aa9da0 msgr2=0x556db6aa59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:41.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.732+0000 7fbff4659740 1 --2- 172.21.15.169:0/3219369438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aa9da0 0x556db6aa59e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fbfd40143a0 tx=0x7fbfd4008ba0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.732+0000 7fbff4659740 1 -- 172.21.15.169:0/3219369438 shutdown_connections 2024-03-28T01:07:41.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.732+0000 7fbff4659740 1 --2- 172.21.15.169:0/3219369438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556db6aaf900 0x556db6aabcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.732+0000 7fbff4659740 1 --2- 172.21.15.169:0/3219369438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db6aaa790 0x556db6abb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.732+0000 7fbff4659740 1 --2- 172.21.15.169:0/3219369438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aa9da0 0x556db6aa59e0 secure :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fbfd40143a0 tx=0x7fbfd4008ba0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.732+0000 7fbff4659740 1 -- 172.21.15.169:0/3219369438 >> 172.21.15.169:0/3219369438 conn(0x556db6aa5600 msgr2=0x556db6aab190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:41.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.739+0000 7fbff4659740 1 -- 172.21.15.169:0/3219369438 shutdown_connections 2024-03-28T01:07:41.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.740+0000 7fbff4659740 1 -- 172.21.15.169:0/3219369438 wait complete. 2024-03-28T01:07:41.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.740+0000 7fbff4659740 1 -- start start 2024-03-28T01:07:41.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.740+0000 7fbff4659740 1 Processor -- start 2024-03-28T01:07:41.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.741+0000 7fbff4659740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aaa790 0x556db6ab91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:41.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.741+0000 7fbff4659740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556db6aaf900 0x556db6ab9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:41.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.741+0000 7fbff4659740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db6aba650 0x556db6abaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:41.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.741+0000 7fbff4659740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec1d0fe0 con 0x556db6aaf900 2024-03-28T01:07:41.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.741+0000 7fbff4659740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec1d22e0 con 0x556db6aaa790 2024-03-28T01:07:41.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.741+0000 7fbff4659740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec1d1620 con 0x556db6aba650 2024-03-28T01:07:41.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.741+0000 7fbff4659740 10 client.0 _ll_register_callbacks cb 0x7fbfdc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:41.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.742+0000 7fbfeb7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556db6aaf900 0x556db6ab9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:41.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.742+0000 7fbfeb7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556db6aaf900 0x556db6ab9700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:38436/0 (socket says 172.21.15.169:38436) 2024-03-28T01:07:41.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.742+0000 7fbfeb7fe640 1 -- 172.21.15.169:0/818380669 learned_addr learned my addr 172.21.15.169:0/818380669 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:41.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.744+0000 7fbfe2ffd640 1 --2- 172.21.15.169:0/818380669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aaa790 0x556db6ab91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:41.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.744+0000 7fbfe2ffd640 1 --2- 172.21.15.169:0/818380669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db6aba650 0x556db6abaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:41.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.744+0000 7fbfe2ffd640 1 -- 172.21.15.169:0/818380669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db6aba650 msgr2=0x556db6abaad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:41.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.744+0000 7fbfe2ffd640 1 --2- 172.21.15.169:0/818380669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db6aba650 0x556db6abaad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.744+0000 7fbfe2ffd640 1 -- 172.21.15.169:0/818380669 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556db6aaf900 msgr2=0x556db6ab9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:41.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.744+0000 7fbfe2ffd640 1 --2- 172.21.15.169:0/818380669 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556db6aaf900 0x556db6ab9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.744+0000 7fbfe2ffd640 1 -- 172.21.15.169:0/818380669 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556db6abb010 con 0x556db6aaa790 2024-03-28T01:07:41.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.746+0000 7fbfe2ffd640 1 --2- 172.21.15.169:0/818380669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aaa790 0x556db6ab91c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x556db6ab5120 tx=0x7fbfd4028040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a518148b0b9d1e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:41.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.746+0000 7fbfcaffd640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:07:41.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.746+0000 7fbfcaffd640 1 -- 172.21.15.169:0/818380669 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4026990 con 0x556db6aaa790 2024-03-28T01:07:41.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.747+0000 7fbff4659740 1 -- 172.21.15.169:0/818380669 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbfd402a020 con 0x556db6aaa790 2024-03-28T01:07:41.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.747+0000 7fbfcaffd640 1 -- 172.21.15.169:0/818380669 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fbfd4006070 con 0x556db6aaa790 2024-03-28T01:07:41.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.750+0000 7fbfcaffd640 1 -- 172.21.15.169:0/818380669 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbfd4006270 con 0x556db6aaa790 2024-03-28T01:07:41.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.750+0000 7fbff4659740 10 client.27428 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:41.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.750+0000 7fbff4659740 1 -- 172.21.15.169:0/818380669 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556db6b1bb60 con 0x556db6aaa790 2024-03-28T01:07:41.755 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.752+0000 7fbfcaffd640 1 -- 172.21.15.169:0/818380669 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbfd402a020 con 0x556db6aaa790 2024-03-28T01:07:41.755 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.752+0000 7fbfcaffd640 1 -- 172.21.15.169:0/818380669 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fbfd4031450 con 0x556db6aaa790 2024-03-28T01:07:41.755 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.752+0000 7fbff4659740 10 client.27428 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:41.755 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.752+0000 7fbff4659740 10 client.27428 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:07:41.755 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.752+0000 7fbff4659740 20 client.27428 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:41.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.752+0000 7fbff4659740 1 --2- 172.21.15.169:0/818380669 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x556db6b1c2f0 0x556db6b1e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:41.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.752+0000 7fbff4659740 4 client.27428 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:07:41.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.752+0000 7fbff4659740 1 -- 172.21.15.169:0/818380669 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556db6a16c60 con 0x556db6b1c2f0 2024-03-28T01:07:41.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.754+0000 7fbfeb7fe640 1 --2- 172.21.15.169:0/818380669 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x556db6b1c2f0 0x556db6b1e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:41.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.755+0000 7fbfeb7fe640 1 --2- 172.21.15.169:0/818380669 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x556db6b1c2f0 0x556db6b1e6e0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=104c3c13b8c7aa81 server_cookie=50293ae2450e45cc in_seq=0 out_seq=0 2024-03-28T01:07:41.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.757+0000 7fbfcaffd640 10 client.27428 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:41.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.757+0000 7fbfcaffd640 1 -- 172.21.15.169:0/818380669 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x556db6a16c60 con 0x556db6b1c2f0 2024-03-28T01:07:41.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.757+0000 7fbfcaffd640 10 client.27428 handle_command_reply: tid=0 2024-03-28T01:07:41.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.757+0000 7fbff4659740 1 client.27428 shutdown 2024-03-28T01:07:41.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.757+0000 7fbff4659740 1 -- 172.21.15.169:0/818380669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aaa790 msgr2=0x556db6ab91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:41.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.757+0000 7fbff4659740 1 --2- 172.21.15.169:0/818380669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aaa790 0x556db6ab91c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x556db6ab5120 tx=0x7fbfd4028040 comp rx=0 tx=0).stop 2024-03-28T01:07:41.762 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.760+0000 7fbff4659740 1 -- 172.21.15.169:0/818380669 shutdown_connections 2024-03-28T01:07:41.763 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.760+0000 7fbff4659740 1 --2- 172.21.15.169:0/818380669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556db6aba650 0x556db6abaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.763 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.760+0000 7fbff4659740 1 --2- 172.21.15.169:0/818380669 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x556db6b1c2f0 0x556db6b1e6e0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.763 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.760+0000 7fbff4659740 1 --2- 172.21.15.169:0/818380669 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x556db6aaf900 0x556db6ab9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.763 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.760+0000 7fbff4659740 1 --2- 172.21.15.169:0/818380669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556db6aaa790 0x556db6ab91c0 secure :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0x556db6ab5120 tx=0x7fbfd4028040 comp rx=0 tx=0).stop 2024-03-28T01:07:41.763 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.760+0000 7fbff4659740 1 -- 172.21.15.169:0/818380669 >> 172.21.15.169:0/818380669 conn(0x556db6aa5600 msgr2=0x556db6aa59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:41.763 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.760+0000 7fbfcaffd640 0 client.27428 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:41.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.761+0000 7fbff4659740 1 -- 172.21.15.169:0/818380669 shutdown_connections 2024-03-28T01:07:41.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.761+0000 7fbff4659740 1 -- 172.21.15.169:0/818380669 wait complete. 2024-03-28T01:07:41.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.761+0000 7fbff4659740 20 client.27428 trim_cache size 0 max 16384 2024-03-28T01:07:41.765 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:07:41.765 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "scrub active (0 inodes in the stack)", 2024-03-28T01:07:41.765 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": { 2024-03-28T01:07:41.765 INFO:teuthology.orchestra.run.smithi169.stdout: "a734cdcd-bb7b-42d1-9000-2598b06563d8": { 2024-03-28T01:07:41.765 INFO:teuthology.orchestra.run.smithi169.stdout: "path": "/", 2024-03-28T01:07:41.765 INFO:teuthology.orchestra.run.smithi169.stdout: "tag": "a734cdcd-bb7b-42d1-9000-2598b06563d8", 2024-03-28T01:07:41.765 INFO:teuthology.orchestra.run.smithi169.stdout: "options": "recursive,force" 2024-03-28T01:07:41.765 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:07:41.765 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:07:41.765 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:07:41.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.762+0000 7fbfe27fc640 1 -- 172.21.15.169:0/816127661 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbfd80a2950 msgr2=0x7fbfd80a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:41.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.762+0000 7fbfe27fc640 1 --2- 172.21.15.169:0/816127661 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbfd80a2950 0x7fbfd80a4e10 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4096dd0 tx=0x7fbfe40ce620 comp rx=0 tx=0).stop 2024-03-28T01:07:41.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.762+0000 7fbfe27fc640 1 -- 172.21.15.169:0/816127661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfec078de0 msgr2=0x7fbfec083720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:41.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.762+0000 7fbfe27fc640 1 --2- 172.21.15.169:0/816127661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfec078de0 0x7fbfec083720 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400b9e0 tx=0x7fbfe400bea0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.762+0000 7fbfe27fc640 1 -- 172.21.15.169:0/816127661 shutdown_connections 2024-03-28T01:07:41.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.762+0000 7fbfe27fc640 1 --2- 172.21.15.169:0/816127661 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbfd80a2950 0x7fbfd80a4e10 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.762+0000 7fbfe27fc640 1 --2- 172.21.15.169:0/816127661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfec078de0 0x7fbfec083720 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.762+0000 7fbfe27fc640 1 --2- 172.21.15.169:0/816127661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfec076430 0x7fbfec0831e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.762+0000 7fbfe27fc640 1 --2- 172.21.15.169:0/816127661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbfec073550 0x7fbfec082ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:41.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.762+0000 7fbfe27fc640 1 -- 172.21.15.169:0/816127661 >> 172.21.15.169:0/816127661 conn(0x7fbfec06e500 msgr2=0x7fbfec077a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:41.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.762+0000 7fbfe27fc640 1 -- 172.21.15.169:0/816127661 shutdown_connections 2024-03-28T01:07:41.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:07:41.762+0000 7fbfe27fc640 1 -- 172.21.15.169:0/816127661 wait complete. 2024-03-28T01:07:41.780 INFO:tasks.cephfs.filesystem:scrub status for tag:a734cdcd-bb7b-42d1-9000-2598b06563d8 - {'path': '/', 'tag': 'a734cdcd-bb7b-42d1-9000-2598b06563d8', 'options': 'recursive,force'} 2024-03-28T01:07:42.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:41 smithi169 ceph-mon[32274]: pgmap v686: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 600 MiB/s wr, 702 op/s 2024-03-28T01:07:42.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:41 smithi174 ceph-mon[36060]: pgmap v686: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 600 MiB/s wr, 702 op/s 2024-03-28T01:07:42.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:41 smithi177 ceph-mon[36206]: pgmap v686: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 600 MiB/s wr, 702 op/s 2024-03-28T01:07:43.053 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:42 smithi169 ceph-mon[32274]: scrub queued for path: / 2024-03-28T01:07:43.053 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:42 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:07:43.053 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:42 smithi169 ceph-mon[32274]: scrub summary: active paths [/] 2024-03-28T01:07:43.053 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:42 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:07:43.053 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:42 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:07:43.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:42 smithi174 ceph-mon[36060]: scrub queued for path: / 2024-03-28T01:07:43.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:42 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:07:43.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:42 smithi174 ceph-mon[36060]: scrub summary: active paths [/] 2024-03-28T01:07:43.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:42 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:07:43.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:42 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:07:43.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:42 smithi177 ceph-mon[36206]: scrub queued for path: / 2024-03-28T01:07:43.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:42 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:07:43.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:42 smithi177 ceph-mon[36206]: scrub summary: active paths [/] 2024-03-28T01:07:43.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:42 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:07:43.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:42 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:07:44.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:43 smithi169 ceph-mon[32274]: pgmap v687: 129 pgs: 129 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 714 MiB/s wr, 1.16k op/s 2024-03-28T01:07:44.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:43 smithi174 ceph-mon[36060]: pgmap v687: 129 pgs: 129 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 714 MiB/s wr, 1.16k op/s 2024-03-28T01:07:44.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:43 smithi177 ceph-mon[36206]: pgmap v687: 129 pgs: 129 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 714 MiB/s wr, 1.16k op/s 2024-03-28T01:07:45.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:44 smithi177 ceph-mon[36206]: pgmap v688: 129 pgs: 129 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 570 MiB/s wr, 1.03k op/s 2024-03-28T01:07:45.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:44 smithi169 ceph-mon[32274]: pgmap v688: 129 pgs: 129 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 570 MiB/s wr, 1.03k op/s 2024-03-28T01:07:45.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:44 smithi174 ceph-mon[36060]: pgmap v688: 129 pgs: 129 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 570 MiB/s wr, 1.03k op/s 2024-03-28T01:07:46.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:45 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:07:46.203 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:45 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:07:46.209 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:45 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:07:46.907 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:46 smithi177 ceph-mon[36206]: pgmap v689: 129 pgs: 129 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 638 MiB/s wr, 1.55k op/s 2024-03-28T01:07:47.304 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:46 smithi169 ceph-mon[32274]: pgmap v689: 129 pgs: 129 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 638 MiB/s wr, 1.55k op/s 2024-03-28T01:07:47.362 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:46 smithi174 ceph-mon[36060]: pgmap v689: 129 pgs: 129 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 638 MiB/s wr, 1.55k op/s 2024-03-28T01:07:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.041+0000 7f006464b640 1 -- 172.21.15.169:0/1483701987 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f005c1092e0 msgr2=0x7f005c0726c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.041+0000 7f006464b640 1 --2- 172.21.15.169:0/1483701987 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f005c1092e0 0x7f005c0726c0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f0054009ca0 tx=0x7f005402c1c0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.041+0000 7f006464b640 1 -- 172.21.15.169:0/1483701987 shutdown_connections 2024-03-28T01:07:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.042+0000 7f006464b640 1 --2- 172.21.15.169:0/1483701987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f005c10f9e0 0x7f005c111dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.042+0000 7f006464b640 1 --2- 172.21.15.169:0/1483701987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f005c072c00 0x7f005c073060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.042+0000 7f006464b640 1 --2- 172.21.15.169:0/1483701987 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f005c1092e0 0x7f005c0726c0 secure :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f0054009ca0 tx=0x7f005402c1c0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.042+0000 7f006464b640 1 -- 172.21.15.169:0/1483701987 >> 172.21.15.169:0/1483701987 conn(0x7f005c06e510 msgr2=0x7f005c070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:48.051 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.048+0000 7f006464b640 1 -- 172.21.15.169:0/1483701987 shutdown_connections 2024-03-28T01:07:48.053 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.049+0000 7f006464b640 1 -- 172.21.15.169:0/1483701987 wait complete. 2024-03-28T01:07:48.053 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.049+0000 7f006464b640 1 Processor -- start 2024-03-28T01:07:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.050+0000 7f006464b640 1 -- start start 2024-03-28T01:07:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.050+0000 7f006464b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f005c072c00 0x7f005c119360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.050+0000 7f006464b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f005c10f9e0 0x7f005c1198a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.050+0000 7f006464b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f005c11e900 0x7f005c11edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.050+0000 7f006464b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c10b520 con 0x7f005c10f9e0 2024-03-28T01:07:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.050+0000 7f006464b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c10b3a0 con 0x7f005c072c00 2024-03-28T01:07:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.050+0000 7f006464b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c10b6a0 con 0x7f005c11e900 2024-03-28T01:07:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.050+0000 7f00623e7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f005c072c00 0x7f005c119360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.050+0000 7f00623e7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f005c072c00 0x7f005c119360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:54990/0 (socket says 172.21.15.169:54990) 2024-03-28T01:07:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.050+0000 7f00623e7640 1 -- 172.21.15.169:0/848071511 learned_addr learned my addr 172.21.15.169:0/848071511 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:48.056 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.051+0000 7f0062be8640 1 --2- 172.21.15.169:0/848071511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f005c11e900 0x7f005c11edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:48.056 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.051+0000 7f0062be8640 1 -- 172.21.15.169:0/848071511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f005c072c00 msgr2=0x7f005c119360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:48.056 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.051+0000 7f0062be8640 1 --2- 172.21.15.169:0/848071511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f005c072c00 0x7f005c119360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.056 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.051+0000 7f0062be8640 1 -- 172.21.15.169:0/848071511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f005c10f9e0 msgr2=0x7f005c1198a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:48.057 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.051+0000 7f0062be8640 1 --2- 172.21.15.169:0/848071511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f005c10f9e0 0x7f005c1198a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.057 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.051+0000 7f0062be8640 1 -- 172.21.15.169:0/848071511 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005c11f370 con 0x7f005c11e900 2024-03-28T01:07:48.057 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.051+0000 7f0062be8640 1 --2- 172.21.15.169:0/848071511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f005c11e900 0x7f005c11edb0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f005c0736a0 tx=0x7f004c00cba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:48.057 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.052+0000 7f00537fe640 1 -- 172.21.15.169:0/848071511 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c004220 con 0x7f005c11e900 2024-03-28T01:07:48.057 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.052+0000 7f00537fe640 1 -- 172.21.15.169:0/848071511 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f004c0043c0 con 0x7f005c11e900 2024-03-28T01:07:48.057 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.052+0000 7f00537fe640 1 -- 172.21.15.169:0/848071511 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c00f450 con 0x7f005c11e900 2024-03-28T01:07:48.059 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.054+0000 7f006464b640 1 -- 172.21.15.169:0/848071511 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f005c11f660 con 0x7f005c11e900 2024-03-28T01:07:48.059 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.054+0000 7f006464b640 1 -- 172.21.15.169:0/848071511 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f005c11fb60 con 0x7f005c11e900 2024-03-28T01:07:48.063 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.059+0000 7f00537fe640 1 -- 172.21.15.169:0/848071511 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f004c00f5f0 con 0x7f005c11e900 2024-03-28T01:07:48.064 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.060+0000 7f00537fe640 1 --2- 172.21.15.169:0/848071511 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f00480a2a00 0x7f00480a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:48.064 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.060+0000 7f00537fe640 1 -- 172.21.15.169:0/848071511 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f004c0cfa60 con 0x7f005c11e900 2024-03-28T01:07:48.065 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.061+0000 7f00623e7640 1 --2- 172.21.15.169:0/848071511 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f00480a2a00 0x7f00480a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:48.066 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.062+0000 7f006723c740 1 Processor -- start 2024-03-28T01:07:48.067 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.063+0000 7f006723c740 1 -- start start 2024-03-28T01:07:48.067 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.063+0000 7f006723c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56231bd6bb20 0x56231bd6bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:48.067 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.063+0000 7f006723c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56231bd6c510 0x56231bd71010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:48.067 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.063+0000 7f006723c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd71680 0x56231bd73b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:48.067 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.063+0000 7f006723c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56231bcd0b10 con 0x56231bd6c510 2024-03-28T01:07:48.067 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.063+0000 7f006723c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x56231bc79900 con 0x56231bd71680 2024-03-28T01:07:48.067 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.063+0000 7f006723c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56231bcccbb0 con 0x56231bd6bb20 2024-03-28T01:07:48.068 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.064+0000 7f0061be6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56231bd6c510 0x56231bd71010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:48.068 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.064+0000 7f0061be6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56231bd6c510 0x56231bd71010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:42678/0 (socket says 172.21.15.169:42678) 2024-03-28T01:07:48.068 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.064+0000 7f0061be6640 1 -- 172.21.15.169:0/1304629131 learned_addr learned my addr 172.21.15.169:0/1304629131 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:48.068 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.064+0000 7f0062be8640 1 --2- 172.21.15.169:0/1304629131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd71680 0x56231bd73b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:48.069 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.064+0000 7f0062be8640 1 -- 172.21.15.169:0/1304629131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56231bd6bb20 msgr2=0x56231bd6bf40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:07:48.069 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.064+0000 7f0062be8640 1 --2- 172.21.15.169:0/1304629131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56231bd6bb20 0x56231bd6bf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.069 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.064+0000 7f0062be8640 1 -- 172.21.15.169:0/1304629131 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56231bd6c510 msgr2=0x56231bd71010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:48.069 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.064+0000 7f0062be8640 1 --2- 172.21.15.169:0/1304629131 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56231bd6c510 0x56231bd71010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.069 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.064+0000 7f0062be8640 1 -- 172.21.15.169:0/1304629131 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56231b9e5210 con 0x56231bd71680 2024-03-28T01:07:48.069 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.065+0000 7f0062be8640 1 --2- 172.21.15.169:0/1304629131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd71680 0x56231bd73b10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f004c0989d0 tx=0x7f004c004570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9674f42356f6037 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:48.071 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.067+0000 7f0050bfe640 1 -- 172.21.15.169:0/1304629131 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c01d070 con 0x56231bd71680 2024-03-28T01:07:48.071 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.067+0000 7f0050bfe640 1 -- 172.21.15.169:0/1304629131 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f004c09c000 con 0x56231bd71680 2024-03-28T01:07:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.068+0000 7f006723c740 1 -- 172.21.15.169:0/1304629131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd71680 msgr2=0x56231bd73b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.068+0000 7f006723c740 1 --2- 172.21.15.169:0/1304629131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd71680 0x56231bd73b10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f004c0989d0 tx=0x7f004c004570 comp rx=0 tx=0).stop 2024-03-28T01:07:48.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.071+0000 7f006723c740 1 -- 172.21.15.169:0/1304629131 shutdown_connections 2024-03-28T01:07:48.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.071+0000 7f006723c740 1 --2- 172.21.15.169:0/1304629131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd71680 0x56231bd73b10 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.071+0000 7f006723c740 1 --2- 172.21.15.169:0/1304629131 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56231bd6c510 0x56231bd71010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.075 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.071+0000 7f006723c740 1 --2- 172.21.15.169:0/1304629131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56231bd6bb20 0x56231bd6bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.076 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.071+0000 7f006723c740 1 -- 172.21.15.169:0/1304629131 >> 172.21.15.169:0/1304629131 conn(0x56231bd67090 msgr2=0x56231bd67490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:48.076 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.072+0000 7f00623e7640 1 --2- 172.21.15.169:0/848071511 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f00480a2a00 0x7f00480a4ec0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0054006010 tx=0x7f005402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:48.076 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.072+0000 7f006723c740 1 -- 172.21.15.169:0/1304629131 shutdown_connections 2024-03-28T01:07:48.077 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.073+0000 7f006723c740 1 -- 172.21.15.169:0/1304629131 wait complete. 2024-03-28T01:07:48.077 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.073+0000 7f006723c740 1 -- start start 2024-03-28T01:07:48.077 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.073+0000 7f006723c740 1 Processor -- start 2024-03-28T01:07:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.073+0000 7f006723c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd6bb20 0x56231bd75e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.073+0000 7f006723c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56231bd6c510 0x56231bd763d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.073+0000 7f006723c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56231bd71680 0x56231bd7d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.074+0000 7f006723c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56231bd74680 con 0x56231bd6c510 2024-03-28T01:07:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.074+0000 7f006723c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x56231bc8cd50 con 0x56231bd6bb20 2024-03-28T01:07:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.074+0000 7f006723c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56231bd74800 con 0x56231bd71680 2024-03-28T01:07:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.074+0000 7f006723c740 10 client.0 _ll_register_callbacks cb 0x56231bc8c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:48.078 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.074+0000 7f0062be8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56231bd71680 0x56231bd7d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:48.079 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.074+0000 7f0062be8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56231bd71680 0x56231bd7d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:41174/0 (socket says 172.21.15.169:41174) 2024-03-28T01:07:48.079 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.074+0000 7f0062be8640 1 -- 172.21.15.169:0/2782723710 learned_addr learned my addr 172.21.15.169:0/2782723710 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:48.081 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.076+0000 7f00623e7640 1 --2- 172.21.15.169:0/2782723710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd6bb20 0x56231bd75e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:48.081 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.077+0000 7f0062be8640 1 -- 172.21.15.169:0/2782723710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd6bb20 msgr2=0x56231bd75e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:48.081 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.077+0000 7f0062be8640 1 --2- 172.21.15.169:0/2782723710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd6bb20 0x56231bd75e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.082 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.077+0000 7f0062be8640 1 -- 172.21.15.169:0/2782723710 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56231bd6c510 msgr2=0x56231bd763d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:48.082 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.077+0000 7f0062be8640 1 --2- 172.21.15.169:0/2782723710 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56231bd6c510 0x56231bd763d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.082 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.077+0000 7f0062be8640 1 -- 172.21.15.169:0/2782723710 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56231bab1620 con 0x56231bd71680 2024-03-28T01:07:48.087 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.083+0000 7f0062be8640 1 --2- 172.21.15.169:0/2782723710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56231bd71680 0x56231bd7d410 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f004c003900 tx=0x7f004c007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af80109dc38868e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:48.087 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.084+0000 7f0045ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:07:48.094 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.088+0000 7f0045ffb640 1 -- 172.21.15.169:0/2782723710 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c0a9060 con 0x56231bd71680 2024-03-28T01:07:48.094 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.088+0000 7f006723c740 1 -- 172.21.15.169:0/2782723710 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f004c09c000 con 0x56231bd71680 2024-03-28T01:07:48.094 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.088+0000 7f0045ffb640 1 -- 172.21.15.169:0/2782723710 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f004c097d50 con 0x56231bd71680 2024-03-28T01:07:48.094 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.088+0000 7f0045ffb640 1 -- 172.21.15.169:0/2782723710 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f004c0b0050 con 0x56231bd71680 2024-03-28T01:07:48.094 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.088+0000 7f006723c740 10 client.27388 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.088+0000 7f006723c740 1 -- 172.21.15.169:0/2782723710 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56231bd746d0 con 0x56231bd71680 2024-03-28T01:07:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.090+0000 7f0045ffb640 1 -- 172.21.15.169:0/2782723710 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f004c09c000 con 0x56231bd71680 2024-03-28T01:07:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.090+0000 7f0045ffb640 1 -- 172.21.15.169:0/2782723710 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f004c0a1210 con 0x56231bd71680 2024-03-28T01:07:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.090+0000 7f006723c740 10 client.27388 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.090+0000 7f006723c740 10 client.27388 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:07:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.090+0000 7f006723c740 20 client.27388 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.090+0000 7f006723c740 1 --2- 172.21.15.169:0/2782723710 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56231bd81ef0 0x56231bd84310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.090+0000 7f006723c740 4 client.27388 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:07:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.090+0000 7f006723c740 1 -- 172.21.15.169:0/2782723710 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56231bcde050 con 0x56231bd81ef0 2024-03-28T01:07:48.098 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.093+0000 7f00623e7640 1 --2- 172.21.15.169:0/2782723710 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56231bd81ef0 0x56231bd84310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:48.109 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.104+0000 7f0045ffb640 10 client.27388 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:48.110 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.104+0000 7f00623e7640 1 --2- 172.21.15.169:0/2782723710 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56231bd81ef0 0x56231bd84310 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a1bf84c9c25ec2a server_cookie=ac6de50893fa4c04 in_seq=0 out_seq=0 2024-03-28T01:07:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.108+0000 7f0045ffb640 1 -- 172.21.15.169:0/2782723710 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56231bcde050 con 0x56231bd81ef0 2024-03-28T01:07:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.108+0000 7f0045ffb640 10 client.27388 handle_command_reply: tid=0 2024-03-28T01:07:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.108+0000 7f006723c740 1 client.27388 shutdown 2024-03-28T01:07:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.109+0000 7f006723c740 1 -- 172.21.15.169:0/2782723710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56231bd71680 msgr2=0x56231bd7d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.109+0000 7f006723c740 1 --2- 172.21.15.169:0/2782723710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56231bd71680 0x56231bd7d410 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f004c003900 tx=0x7f004c007f90 comp rx=0 tx=0).stop 2024-03-28T01:07:48.113 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.109+0000 7f006723c740 1 -- 172.21.15.169:0/2782723710 shutdown_connections 2024-03-28T01:07:48.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.109+0000 7f006723c740 1 --2- 172.21.15.169:0/2782723710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56231bd71680 0x56231bd7d410 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.109+0000 7f006723c740 1 --2- 172.21.15.169:0/2782723710 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56231bd81ef0 0x56231bd84310 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.109+0000 7f006723c740 1 --2- 172.21.15.169:0/2782723710 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56231bd6c510 0x56231bd763d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.109+0000 7f006723c740 1 --2- 172.21.15.169:0/2782723710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd6bb20 0x56231bd75e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.109+0000 7f006723c740 1 -- 172.21.15.169:0/2782723710 >> 172.21.15.169:0/2782723710 conn(0x56231bd67090 msgr2=0x56231bd66290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:48.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.109+0000 7f0045ffb640 0 client.27388 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:48.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.109+0000 7f006723c740 1 -- 172.21.15.169:0/2782723710 shutdown_connections 2024-03-28T01:07:48.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.110+0000 7f006723c740 1 -- 172.21.15.169:0/2782723710 wait complete. 2024-03-28T01:07:48.114 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.110+0000 7f006723c740 20 client.27388 trim_cache size 0 max 16384 2024-03-28T01:07:48.188 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.185+0000 7f006723c740 1 Processor -- start 2024-03-28T01:07:48.188 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.185+0000 7f006723c740 1 -- start start 2024-03-28T01:07:48.189 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.186+0000 7f006723c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56231bd6bb20 0x56231bd6d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.186+0000 7f006723c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd6c510 0x56231bd6dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.187+0000 7f006723c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56231bd71680 0x56231bd7d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.187+0000 7f006723c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c1c42d0 con 0x56231bd6bb20 2024-03-28T01:07:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.187+0000 7f006723c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c1c1dd0 con 0x56231bd6c510 2024-03-28T01:07:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.187+0000 7f006723c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c1bfe70 con 0x56231bd71680 2024-03-28T01:07:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.190+0000 7f00623e7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56231bd6bb20 0x56231bd6d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.190+0000 7f00623e7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56231bd6bb20 0x56231bd6d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:42684/0 (socket says 172.21.15.169:42684) 2024-03-28T01:07:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.190+0000 7f00623e7640 1 -- 172.21.15.169:0/1971937572 learned_addr learned my addr 172.21.15.169:0/1971937572 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.190+0000 7f0062be8640 1 --2- 172.21.15.169:0/1971937572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56231bd71680 0x56231bd7d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.190+0000 7f0061be6640 1 --2- 172.21.15.169:0/1971937572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd6c510 0x56231bd6dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.198+0000 7f0062be8640 1 -- 172.21.15.169:0/1971937572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd6c510 msgr2=0x56231bd6dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.198+0000 7f0062be8640 1 --2- 172.21.15.169:0/1971937572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd6c510 0x56231bd6dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.198+0000 7f0062be8640 1 -- 172.21.15.169:0/1971937572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56231bd6bb20 msgr2=0x56231bd6d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.198+0000 7f0062be8640 1 --2- 172.21.15.169:0/1971937572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56231bd6bb20 0x56231bd6d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.198+0000 7f0062be8640 1 -- 172.21.15.169:0/1971937572 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56231bd67e00 con 0x56231bd71680 2024-03-28T01:07:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.203+0000 7f0062be8640 1 --2- 172.21.15.169:0/1971937572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56231bd71680 0x56231bd7d3d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f004c0989d0 tx=0x7f004c09a510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25b446b9a64e0818 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.204+0000 7f0047fff640 1 -- 172.21.15.169:0/1971937572 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c0a9040 con 0x56231bd71680 2024-03-28T01:07:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.204+0000 7f0047fff640 1 -- 172.21.15.169:0/1971937572 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f004c09c000 con 0x56231bd71680 2024-03-28T01:07:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.204+0000 7f0047fff640 1 -- 172.21.15.169:0/1971937572 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c01d030 con 0x56231bd71680 2024-03-28T01:07:48.210 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.205+0000 7f006723c740 1 -- 172.21.15.169:0/1971937572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56231bd71680 msgr2=0x56231bd7d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.205+0000 7f006723c740 1 --2- 172.21.15.169:0/1971937572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56231bd71680 0x56231bd7d3d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f004c0989d0 tx=0x7f004c09a510 comp rx=0 tx=0).stop 2024-03-28T01:07:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.207+0000 7f006723c740 1 -- 172.21.15.169:0/1971937572 shutdown_connections 2024-03-28T01:07:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.207+0000 7f006723c740 1 --2- 172.21.15.169:0/1971937572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56231bd71680 0x56231bd7d3d0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.207+0000 7f006723c740 1 --2- 172.21.15.169:0/1971937572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd6c510 0x56231bd6dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.207+0000 7f006723c740 1 --2- 172.21.15.169:0/1971937572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56231bd6bb20 0x56231bd6d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.207+0000 7f006723c740 1 -- 172.21.15.169:0/1971937572 >> 172.21.15.169:0/1971937572 conn(0x56231bd67090 msgr2=0x56231bd75e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.214+0000 7f006723c740 1 -- 172.21.15.169:0/1971937572 shutdown_connections 2024-03-28T01:07:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.214+0000 7f006723c740 1 -- 172.21.15.169:0/1971937572 wait complete. 2024-03-28T01:07:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.214+0000 7f006723c740 1 -- start start 2024-03-28T01:07:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.215+0000 7f006723c740 1 Processor -- start 2024-03-28T01:07:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.215+0000 7f006723c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56231bd6bb20 0x56231bd7a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.215+0000 7f006723c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd6c510 0x56231bd7a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.215+0000 7f006723c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56231bd71680 0x56231bddb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.216+0000 7f006723c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c1c0470 con 0x56231bd71680 2024-03-28T01:07:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.216+0000 7f006723c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c1c1770 con 0x56231bd6c510 2024-03-28T01:07:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.216+0000 7f006723c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c1c0ab0 con 0x56231bd6bb20 2024-03-28T01:07:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.216+0000 7f0061be6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd6c510 0x56231bd7a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.216+0000 7f0061be6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd6c510 0x56231bd7a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:55022/0 (socket says 172.21.15.169:55022) 2024-03-28T01:07:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.216+0000 7f0061be6640 1 -- 172.21.15.169:0/274648110 learned_addr learned my addr 172.21.15.169:0/274648110 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:07:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.216+0000 7f006723c740 10 client.0 _ll_register_callbacks cb 0x7f004c098390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:07:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.216+0000 7f0062be8640 1 --2- 172.21.15.169:0/274648110 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56231bd71680 0x56231bddb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.218+0000 7f0061be6640 1 -- 172.21.15.169:0/274648110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56231bd6bb20 msgr2=0x56231bd7a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.218+0000 7f0061be6640 1 --2- 172.21.15.169:0/274648110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56231bd6bb20 0x56231bd7a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.218+0000 7f0061be6640 1 -- 172.21.15.169:0/274648110 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56231bd71680 msgr2=0x56231bddb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:07:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.218+0000 7f0061be6640 1 --2- 172.21.15.169:0/274648110 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56231bd71680 0x56231bddb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.218+0000 7f0061be6640 1 -- 172.21.15.169:0/274648110 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56231bd7aba0 con 0x56231bd6c510 2024-03-28T01:07:48.229 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.226+0000 7f0061be6640 1 --2- 172.21.15.169:0/274648110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd6c510 0x56231bd7a5b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x56231bd6c910 tx=0x7f0058010b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d236eb46de6b9fe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:07:48.229 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.227+0000 7f0045ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:07:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.230+0000 7f0045ffb640 1 -- 172.21.15.169:0/274648110 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005800fba0 con 0x56231bd6c510 2024-03-28T01:07:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.231+0000 7f0045ffb640 1 -- 172.21.15.169:0/274648110 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f005800fd40 con 0x56231bd6c510 2024-03-28T01:07:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.232+0000 7f006723c740 1 -- 172.21.15.169:0/274648110 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f004c09c000 con 0x56231bd6c510 2024-03-28T01:07:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.234+0000 7f0045ffb640 1 -- 172.21.15.169:0/274648110 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0058004020 con 0x56231bd6c510 2024-03-28T01:07:48.243 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.240+0000 7f006723c740 10 client.27443 fetch_fsmap learned FSMap version 25 2024-03-28T01:07:48.243 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.240+0000 7f006723c740 1 -- 172.21.15.169:0/274648110 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56231bddcdf0 con 0x56231bd6c510 2024-03-28T01:07:48.245 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.242+0000 7f0045ffb640 1 -- 172.21.15.169:0/274648110 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f004c09c000 con 0x56231bd6c510 2024-03-28T01:07:48.247 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.243+0000 7f0045ffb640 1 -- 172.21.15.169:0/274648110 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f0058019410 con 0x56231bd6c510 2024-03-28T01:07:48.249 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.245+0000 7f006723c740 10 client.27443 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:07:48.249 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.245+0000 7f006723c740 10 client.27443 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:07:48.249 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.245+0000 7f006723c740 20 client.27443 populate_metadata read hostname 'smithi169' 2024-03-28T01:07:48.249 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.245+0000 7f006723c740 1 --2- 172.21.15.169:0/274648110 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56231bddd740 0x56231bddfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:07:48.249 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.245+0000 7f006723c740 4 client.27443 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:07:48.249 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.245+0000 7f006723c740 1 -- 172.21.15.169:0/274648110 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56231bcd83a0 con 0x56231bddd740 2024-03-28T01:07:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.248+0000 7f00623e7640 1 --2- 172.21.15.169:0/274648110 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56231bddd740 0x56231bddfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:07:48.267 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.259+0000 7f00623e7640 1 --2- 172.21.15.169:0/274648110 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56231bddd740 0x56231bddfb60 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38343ecc7b3d8990 server_cookie=6d44e754d84d15e6 in_seq=0 out_seq=0 2024-03-28T01:07:48.267 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:48.259+0000 7f0045ffb640 10 client.27443 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:48.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:48 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:07:48.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:48 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:07:48.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:48 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:07:48.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:48 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:07:49.100 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:48 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:07:49.101 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:48 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:07:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.792+0000 7f0045ffb640 1 -- 172.21.15.169:0/274648110 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56231bcd83a0 con 0x56231bddd740 2024-03-28T01:07:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.792+0000 7f0045ffb640 10 client.27443 handle_command_reply: tid=0 2024-03-28T01:07:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.794+0000 7f006723c740 1 client.27443 shutdown 2024-03-28T01:07:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.794+0000 7f006723c740 1 -- 172.21.15.169:0/274648110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd6c510 msgr2=0x56231bd7a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.794+0000 7f006723c740 1 --2- 172.21.15.169:0/274648110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd6c510 0x56231bd7a5b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x56231bd6c910 tx=0x7f0058010b40 comp rx=0 tx=0).stop 2024-03-28T01:07:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.794+0000 7f006723c740 1 -- 172.21.15.169:0/274648110 shutdown_connections 2024-03-28T01:07:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.794+0000 7f006723c740 1 --2- 172.21.15.169:0/274648110 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56231bddd740 0x56231bddfb60 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.794+0000 7f006723c740 1 --2- 172.21.15.169:0/274648110 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56231bd71680 0x56231bddb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.794+0000 7f006723c740 1 --2- 172.21.15.169:0/274648110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56231bd6c510 0x56231bd7a5b0 secure :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0x56231bd6c910 tx=0x7f0058010b40 comp rx=0 tx=0).stop 2024-03-28T01:07:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.794+0000 7f006723c740 1 --2- 172.21.15.169:0/274648110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56231bd6bb20 0x56231bd7a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.795+0000 7f006723c740 1 -- 172.21.15.169:0/274648110 >> 172.21.15.169:0/274648110 conn(0x56231bd67090 msgr2=0x56231bd6d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.797+0000 7f0045ffb640 0 client.27443 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:07:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.797+0000 7f006723c740 1 -- 172.21.15.169:0/274648110 shutdown_connections 2024-03-28T01:07:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.797+0000 7f006723c740 1 -- 172.21.15.169:0/274648110 wait complete. 2024-03-28T01:07:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.797+0000 7f006723c740 20 client.27443 trim_cache size 0 max 16384 2024-03-28T01:07:49.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:49 smithi169 ceph-mon[32274]: pgmap v690: 129 pgs: 129 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 451 MiB/s wr, 1.55k op/s 2024-03-28T01:07:49.803 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:49 smithi169 ceph-mon[32274]: scrub summary: idle 2024-03-28T01:07:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:07:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:07:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:07:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:07:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.806+0000 7f006464b640 1 -- 172.21.15.169:0/848071511 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f00480a2a00 msgr2=0x7f00480a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.806+0000 7f006464b640 1 --2- 172.21.15.169:0/848071511 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f00480a2a00 0x7f00480a4ec0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0054006010 tx=0x7f005402e040 comp rx=0 tx=0).stop 2024-03-28T01:07:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.806+0000 7f006464b640 1 -- 172.21.15.169:0/848071511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f005c11e900 msgr2=0x7f005c11edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:07:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.806+0000 7f006464b640 1 --2- 172.21.15.169:0/848071511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f005c11e900 0x7f005c11edb0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f005c0736a0 tx=0x7f004c00cba0 comp rx=0 tx=0).stop 2024-03-28T01:07:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.807+0000 7f006464b640 1 -- 172.21.15.169:0/848071511 shutdown_connections 2024-03-28T01:07:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.807+0000 7f006464b640 1 --2- 172.21.15.169:0/848071511 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f00480a2a00 0x7f00480a4ec0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.807+0000 7f006464b640 1 --2- 172.21.15.169:0/848071511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f005c11e900 0x7f005c11edb0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.807+0000 7f006464b640 1 --2- 172.21.15.169:0/848071511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f005c10f9e0 0x7f005c1198a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.807+0000 7f006464b640 1 --2- 172.21.15.169:0/848071511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f005c072c00 0x7f005c119360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:07:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.808+0000 7f006464b640 1 -- 172.21.15.169:0/848071511 >> 172.21.15.169:0/848071511 conn(0x7f005c06e510 msgr2=0x7f005c10b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:07:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.811+0000 7f006464b640 1 -- 172.21.15.169:0/848071511 shutdown_connections 2024-03-28T01:07:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:07:49.811+0000 7f006464b640 1 -- 172.21.15.169:0/848071511 wait complete. 2024-03-28T01:07:49.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:49 smithi174 ceph-mon[36060]: pgmap v690: 129 pgs: 129 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 451 MiB/s wr, 1.55k op/s 2024-03-28T01:07:49.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:49 smithi174 ceph-mon[36060]: scrub summary: idle 2024-03-28T01:07:49.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:49 smithi177 ceph-mon[36206]: pgmap v690: 129 pgs: 129 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 451 MiB/s wr, 1.55k op/s 2024-03-28T01:07:49.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:49 smithi177 ceph-mon[36206]: scrub summary: idle 2024-03-28T01:07:52.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:51 smithi169 ceph-mon[32274]: pgmap v691: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 491 MiB/s wr, 1.58k op/s 2024-03-28T01:07:52.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:51 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:07:52.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:51 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:07:52.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:51 smithi174 ceph-mon[36060]: pgmap v691: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 491 MiB/s wr, 1.58k op/s 2024-03-28T01:07:52.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:51 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:07:52.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:51 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:07:52.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:51 smithi177 ceph-mon[36206]: pgmap v691: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 491 MiB/s wr, 1.58k op/s 2024-03-28T01:07:52.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:51 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:07:52.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:51 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:07:54.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:53 smithi169 ceph-mon[32274]: pgmap v692: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 619 MiB/s wr, 1.58k op/s 2024-03-28T01:07:54.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:53 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:07:54.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:53 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:07:54.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:53 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:07:54.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:53 smithi174 ceph-mon[36060]: pgmap v692: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 619 MiB/s wr, 1.58k op/s 2024-03-28T01:07:54.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:53 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:07:54.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:53 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:07:54.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:53 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:07:54.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:53 smithi177 ceph-mon[36206]: pgmap v692: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 619 MiB/s wr, 1.58k op/s 2024-03-28T01:07:54.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:53 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:07:54.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:53 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:07:54.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:53 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:07:56.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:55 smithi169 ceph-mon[32274]: pgmap v693: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 447 MiB/s wr, 1.01k op/s 2024-03-28T01:07:56.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:55 smithi174 ceph-mon[36060]: pgmap v693: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 447 MiB/s wr, 1.01k op/s 2024-03-28T01:07:56.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:55 smithi177 ceph-mon[36206]: pgmap v693: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 447 MiB/s wr, 1.01k op/s 2024-03-28T01:07:58.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:57 smithi169 ceph-mon[32274]: pgmap v694: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 651 MiB/s wr, 1.05k op/s 2024-03-28T01:07:58.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:57 smithi174 ceph-mon[36060]: pgmap v694: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 651 MiB/s wr, 1.05k op/s 2024-03-28T01:07:58.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:57 smithi177 ceph-mon[36206]: pgmap v694: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 651 MiB/s wr, 1.05k op/s 2024-03-28T01:08:00.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:07:59 smithi169 ceph-mon[32274]: pgmap v695: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 606 MiB/s wr, 582 op/s 2024-03-28T01:08:00.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:07:59 smithi174 ceph-mon[36060]: pgmap v695: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 606 MiB/s wr, 582 op/s 2024-03-28T01:08:00.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:07:59 smithi177 ceph-mon[36206]: pgmap v695: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 606 MiB/s wr, 582 op/s 2024-03-28T01:08:02.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:01 smithi174 ceph-mon[36060]: pgmap v696: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 639 MiB/s wr, 656 op/s 2024-03-28T01:08:02.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:01 smithi177 ceph-mon[36206]: pgmap v696: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 639 MiB/s wr, 656 op/s 2024-03-28T01:08:02.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:01 smithi169 ceph-mon[32274]: pgmap v696: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 639 MiB/s wr, 656 op/s 2024-03-28T01:08:03.121 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.117+0000 7f36d679d640 1 -- 172.21.15.169:0/1508911438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36d00732b0 msgr2=0x7f36d0108cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.117+0000 7f36d679d640 1 --2- 172.21.15.169:0/1508911438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36d00732b0 0x7f36d0108cd0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f36c8009ca0 tx=0x7f36c802c120 comp rx=0 tx=0).stop 2024-03-28T01:08:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.118+0000 7f36d679d640 1 -- 172.21.15.169:0/1508911438 shutdown_connections 2024-03-28T01:08:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.118+0000 7f36d679d640 1 --2- 172.21.15.169:0/1508911438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36d0109390 0x7f36d010e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.118+0000 7f36d679d640 1 --2- 172.21.15.169:0/1508911438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36d00732b0 0x7f36d0108cd0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.118+0000 7f36d679d640 1 --2- 172.21.15.169:0/1508911438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36d0072950 0x7f36d0072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.118+0000 7f36d679d640 1 -- 172.21.15.169:0/1508911438 >> 172.21.15.169:0/1508911438 conn(0x7f36d006e510 msgr2=0x7f36d0070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:03.131 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.129+0000 7f36d679d640 1 -- 172.21.15.169:0/1508911438 shutdown_connections 2024-03-28T01:08:03.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.133+0000 7f36d679d640 1 -- 172.21.15.169:0/1508911438 wait complete. 2024-03-28T01:08:03.137 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.133+0000 7f36d679d640 1 Processor -- start 2024-03-28T01:08:03.138 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.134+0000 7f36d679d640 1 -- start start 2024-03-28T01:08:03.138 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.134+0000 7f36d679d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36d0072950 0x7f36d0114d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:03.138 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.134+0000 7f36d679d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36d00732b0 0x7f36d01152a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:03.138 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.134+0000 7f36d679d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36d0109390 0x7f36d01b66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:03.138 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.134+0000 7f36d679d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d011bf90 con 0x7f36d0072950 2024-03-28T01:08:03.138 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.134+0000 7f36d679d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d011be10 con 0x7f36d0109390 2024-03-28T01:08:03.138 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.134+0000 7f36d679d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d011c110 con 0x7f36d00732b0 2024-03-28T01:08:03.138 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.135+0000 7f36d4f9a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36d00732b0 0x7f36d01152a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:03.140 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.137+0000 7f36d5f9c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36d0109390 0x7f36d01b66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:03.140 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.137+0000 7f36d4f9a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36d00732b0 0x7f36d01152a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:57298/0 (socket says 172.21.15.169:57298) 2024-03-28T01:08:03.140 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.137+0000 7f36d4f9a640 1 -- 172.21.15.169:0/3640599821 learned_addr learned my addr 172.21.15.169:0/3640599821 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:03.141 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.139+0000 7f36d579b640 1 --2- 172.21.15.169:0/3640599821 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36d0072950 0x7f36d0114d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:03.143 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.140+0000 7f36d4f9a640 1 -- 172.21.15.169:0/3640599821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36d0109390 msgr2=0x7f36d01b66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:03.144 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.140+0000 7f36d4f9a640 1 --2- 172.21.15.169:0/3640599821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36d0109390 0x7f36d01b66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.144 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.140+0000 7f36d4f9a640 1 -- 172.21.15.169:0/3640599821 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36d0072950 msgr2=0x7f36d0114d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:03.144 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.140+0000 7f36d4f9a640 1 --2- 172.21.15.169:0/3640599821 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36d0072950 0x7f36d0114d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.144 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.140+0000 7f36d4f9a640 1 -- 172.21.15.169:0/3640599821 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d01b6be0 con 0x7f36d00732b0 2024-03-28T01:08:03.148 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.145+0000 7f36d4f9a640 1 --2- 172.21.15.169:0/3640599821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36d00732b0 0x7f36d01152a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f36c8006010 tx=0x7f36c8035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:03.154 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.150+0000 7f36c67fc640 1 -- 172.21.15.169:0/3640599821 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c8039040 con 0x7f36d00732b0 2024-03-28T01:08:03.154 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.151+0000 7f36d679d640 1 -- 172.21.15.169:0/3640599821 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d01b6db0 con 0x7f36d00732b0 2024-03-28T01:08:03.154 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.151+0000 7f36d679d640 1 -- 172.21.15.169:0/3640599821 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d01b71b0 con 0x7f36d00732b0 2024-03-28T01:08:03.154 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.151+0000 7f36c67fc640 1 -- 172.21.15.169:0/3640599821 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36c8044070 con 0x7f36d00732b0 2024-03-28T01:08:03.154 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.151+0000 7f36c67fc640 1 -- 172.21.15.169:0/3640599821 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c803f6a0 con 0x7f36d00732b0 2024-03-28T01:08:03.168 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.161+0000 7f36da5ec740 1 Processor -- start 2024-03-28T01:08:03.171 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.168+0000 7f36c67fc640 1 -- 172.21.15.169:0/3640599821 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f36c8003b70 con 0x7f36d00732b0 2024-03-28T01:08:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.169+0000 7f36c67fc640 1 --2- 172.21.15.169:0/3640599821 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36bc0a2a00 0x7f36bc0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.170+0000 7f36c67fc640 1 -- 172.21.15.169:0/3640599821 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f36c80f11e0 con 0x7f36d00732b0 2024-03-28T01:08:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.174+0000 7f36da5ec740 1 -- start start 2024-03-28T01:08:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.174+0000 7f36da5ec740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3e6b20 0x55af5b3e6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.174+0000 7f36da5ec740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af5b3e7510 0x55af5b3ec010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.174+0000 7f36da5ec740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5b3ec680 0x55af5b3eeb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.174+0000 7f36da5ec740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af5b34bb10 con 0x55af5b3e6b20 2024-03-28T01:08:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.174+0000 7f36da5ec740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af5b2f4900 con 0x55af5b3ec680 2024-03-28T01:08:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.175+0000 7f36da5ec740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af5b347bb0 con 0x55af5b3e7510 2024-03-28T01:08:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.175+0000 7f36d5f9c640 1 --2- 172.21.15.169:0/3640599821 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36bc0a2a00 0x7f36bc0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.176+0000 7f36d579b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3e6b20 0x55af5b3e6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.176+0000 7f36d579b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3e6b20 0x55af5b3e6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34912/0 (socket says 172.21.15.169:34912) 2024-03-28T01:08:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.176+0000 7f36d579b640 1 -- 172.21.15.169:0/3654934235 learned_addr learned my addr 172.21.15.169:0/3654934235 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:03.183 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.177+0000 7f36d579b640 1 -- 172.21.15.169:0/3654934235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af5b3e7510 msgr2=0x55af5b3ec010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:08:03.183 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.177+0000 7f36d579b640 1 --2- 172.21.15.169:0/3654934235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af5b3e7510 0x55af5b3ec010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.183 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.177+0000 7f36d579b640 1 -- 172.21.15.169:0/3654934235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5b3ec680 msgr2=0x55af5b3eeb10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:08:03.183 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.177+0000 7f36d579b640 1 --2- 172.21.15.169:0/3654934235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5b3ec680 0x55af5b3eeb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.183 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.177+0000 7f36d579b640 1 -- 172.21.15.169:0/3654934235 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af5b060210 con 0x55af5b3e6b20 2024-03-28T01:08:03.183 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.177+0000 7f36d579b640 1 --2- 172.21.15.169:0/3654934235 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3e6b20 0x55af5b3e6f40 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f36cc00cbf0 tx=0x7f36cc00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d411ea7ff830b305 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:03.183 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.180+0000 7f36bb7fe640 1 -- 172.21.15.169:0/3654934235 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc010630 con 0x55af5b3e6b20 2024-03-28T01:08:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.180+0000 7f36bb7fe640 1 -- 172.21.15.169:0/3654934235 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36cc010cb0 con 0x55af5b3e6b20 2024-03-28T01:08:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.181+0000 7f36da5ec740 1 -- 172.21.15.169:0/3654934235 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3e6b20 msgr2=0x55af5b3e6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.181+0000 7f36da5ec740 1 --2- 172.21.15.169:0/3654934235 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3e6b20 0x55af5b3e6f40 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f36cc00cbf0 tx=0x7f36cc00e720 comp rx=0 tx=0).stop 2024-03-28T01:08:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.185+0000 7f36d5f9c640 1 --2- 172.21.15.169:0/3640599821 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36bc0a2a00 0x7f36bc0a4ec0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f36c0005fd0 tx=0x7f36c0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.185+0000 7f36da5ec740 1 -- 172.21.15.169:0/3654934235 shutdown_connections 2024-03-28T01:08:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.185+0000 7f36da5ec740 1 --2- 172.21.15.169:0/3654934235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5b3ec680 0x55af5b3eeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.185+0000 7f36da5ec740 1 --2- 172.21.15.169:0/3654934235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af5b3e7510 0x55af5b3ec010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.185+0000 7f36da5ec740 1 --2- 172.21.15.169:0/3654934235 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3e6b20 0x55af5b3e6f40 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.185+0000 7f36da5ec740 1 -- 172.21.15.169:0/3654934235 >> 172.21.15.169:0/3654934235 conn(0x55af5b3e2090 msgr2=0x55af5b3e2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:03.198 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.196+0000 7f36da5ec740 1 -- 172.21.15.169:0/3654934235 shutdown_connections 2024-03-28T01:08:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.196+0000 7f36da5ec740 1 -- 172.21.15.169:0/3654934235 wait complete. 2024-03-28T01:08:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.196+0000 7f36da5ec740 1 -- start start 2024-03-28T01:08:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.196+0000 7f36da5ec740 1 Processor -- start 2024-03-28T01:08:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.197+0000 7f36da5ec740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5b3e6b20 0x55af5b3f0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.198+0000 7f36da5ec740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af5b3e7510 0x55af5b3f13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:03.201 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.198+0000 7f36da5ec740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3ec680 0x55af5b3f83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:03.201 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.198+0000 7f36da5ec740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af5b3ef680 con 0x55af5b3ec680 2024-03-28T01:08:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.198+0000 7f36da5ec740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af5b307d50 con 0x55af5b3e6b20 2024-03-28T01:08:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.198+0000 7f36da5ec740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af5b3ef800 con 0x55af5b3e7510 2024-03-28T01:08:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.198+0000 7f36da5ec740 10 client.0 _ll_register_callbacks cb 0x55af5b307720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.199+0000 7f36d5f9c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3ec680 0x55af5b3f83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.199+0000 7f36d5f9c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3ec680 0x55af5b3f83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34928/0 (socket says 172.21.15.169:34928) 2024-03-28T01:08:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.199+0000 7f36d5f9c640 1 -- 172.21.15.169:0/2467774693 learned_addr learned my addr 172.21.15.169:0/2467774693 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.201+0000 7f36d5f9c640 1 -- 172.21.15.169:0/2467774693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af5b3e7510 msgr2=0x55af5b3f13b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:08:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.201+0000 7f36d5f9c640 1 --2- 172.21.15.169:0/2467774693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af5b3e7510 0x55af5b3f13b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.201+0000 7f36d5f9c640 1 -- 172.21.15.169:0/2467774693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5b3e6b20 msgr2=0x55af5b3f0e70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:08:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.201+0000 7f36d5f9c640 1 --2- 172.21.15.169:0/2467774693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5b3e6b20 0x55af5b3f0e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.201+0000 7f36d5f9c640 1 -- 172.21.15.169:0/2467774693 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55af5b12c620 con 0x55af5b3ec680 2024-03-28T01:08:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.201+0000 7f36d5f9c640 1 --2- 172.21.15.169:0/2467774693 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3ec680 0x55af5b3f83f0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f36c000b7d0 tx=0x7f36c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fb2b4bffc28de71 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.201+0000 7f36b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:08:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.202+0000 7f36b8ff9640 1 -- 172.21.15.169:0/2467774693 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0022070 con 0x55af5b3ec680 2024-03-28T01:08:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.202+0000 7f36da5ec740 1 -- 172.21.15.169:0/2467774693 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36cc010cb0 con 0x55af5b3ec680 2024-03-28T01:08:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.202+0000 7f36b8ff9640 1 -- 172.21.15.169:0/2467774693 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f36c001e860 con 0x55af5b3ec680 2024-03-28T01:08:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.203+0000 7f36b8ff9640 1 -- 172.21.15.169:0/2467774693 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36c002e050 con 0x55af5b3ec680 2024-03-28T01:08:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.203+0000 7f36da5ec740 10 client.17541 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.203+0000 7f36da5ec740 1 -- 172.21.15.169:0/2467774693 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55af5b3fb860 con 0x55af5b3ec680 2024-03-28T01:08:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.204+0000 7f36b8ff9640 1 -- 172.21.15.169:0/2467774693 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36cc010cb0 con 0x55af5b3ec680 2024-03-28T01:08:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.204+0000 7f36b8ff9640 1 -- 172.21.15.169:0/2467774693 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f36c0010c70 con 0x55af5b3ec680 2024-03-28T01:08:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.204+0000 7f36da5ec740 10 client.17541 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.204+0000 7f36da5ec740 10 client.17541 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:08:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.204+0000 7f36da5ec740 20 client.17541 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.204+0000 7f36da5ec740 1 --2- 172.21.15.169:0/2467774693 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55af5b3fd420 0x55af5b3ff880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.204+0000 7f36da5ec740 4 client.17541 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:08:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.205+0000 7f36da5ec740 1 -- 172.21.15.169:0/2467774693 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55af5b359050 con 0x55af5b3fd420 2024-03-28T01:08:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.211+0000 7f36d579b640 1 --2- 172.21.15.169:0/2467774693 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55af5b3fd420 0x55af5b3ff880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:03.238 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.230+0000 7f36d579b640 1 --2- 172.21.15.169:0/2467774693 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55af5b3fd420 0x55af5b3ff880 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f211e697a22ff8bb server_cookie=acf3dcb948733712 in_seq=0 out_seq=0 2024-03-28T01:08:03.239 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.230+0000 7f36b8ff9640 10 client.17541 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.236+0000 7f36b8ff9640 1 -- 172.21.15.169:0/2467774693 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55af5b359050 con 0x55af5b3fd420 2024-03-28T01:08:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.236+0000 7f36b8ff9640 10 client.17541 handle_command_reply: tid=0 2024-03-28T01:08:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.236+0000 7f36da5ec740 1 client.17541 shutdown 2024-03-28T01:08:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.242+0000 7f36da5ec740 1 -- 172.21.15.169:0/2467774693 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3ec680 msgr2=0x55af5b3f83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.242+0000 7f36da5ec740 1 --2- 172.21.15.169:0/2467774693 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3ec680 0x55af5b3f83f0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f36c000b7d0 tx=0x7f36c00046c0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.248 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.244+0000 7f36da5ec740 1 -- 172.21.15.169:0/2467774693 shutdown_connections 2024-03-28T01:08:03.248 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.244+0000 7f36da5ec740 1 --2- 172.21.15.169:0/2467774693 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55af5b3fd420 0x55af5b3ff880 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.248 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.245+0000 7f36da5ec740 1 --2- 172.21.15.169:0/2467774693 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3ec680 0x55af5b3f83f0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.248 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.245+0000 7f36da5ec740 1 --2- 172.21.15.169:0/2467774693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af5b3e7510 0x55af5b3f13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.248 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.245+0000 7f36da5ec740 1 --2- 172.21.15.169:0/2467774693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5b3e6b20 0x55af5b3f0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.249 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.245+0000 7f36b8ff9640 0 client.17541 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:03.249 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.245+0000 7f36da5ec740 1 -- 172.21.15.169:0/2467774693 >> 172.21.15.169:0/2467774693 conn(0x55af5b3e2090 msgr2=0x55af5b3e1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:03.251 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.247+0000 7f36da5ec740 1 -- 172.21.15.169:0/2467774693 shutdown_connections 2024-03-28T01:08:03.251 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.247+0000 7f36da5ec740 1 -- 172.21.15.169:0/2467774693 wait complete. 2024-03-28T01:08:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.249+0000 7f36da5ec740 20 client.17541 trim_cache size 0 max 16384 2024-03-28T01:08:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.292+0000 7f36da5ec740 1 Processor -- start 2024-03-28T01:08:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.293+0000 7f36da5ec740 1 -- start start 2024-03-28T01:08:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.293+0000 7f36da5ec740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5b3e6b20 0x55af5b3ef510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.293+0000 7f36da5ec740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3e7510 0x55af5b3f83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.293+0000 7f36da5ec740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af5b3ec680 0x55af5b3e52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.293+0000 7f36d579b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5b3e6b20 0x55af5b3ef510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.293+0000 7f36d579b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5b3e6b20 0x55af5b3ef510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:33620/0 (socket says 172.21.15.169:33620) 2024-03-28T01:08:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.293+0000 7f36d4f9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3e7510 0x55af5b3f83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.293+0000 7f36d4f9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3e7510 0x55af5b3f83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34934/0 (socket says 172.21.15.169:34934) 2024-03-28T01:08:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.294+0000 7f36da5ec740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b4012120 con 0x55af5b3e7510 2024-03-28T01:08:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.294+0000 7f36da5ec740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b400fc20 con 0x55af5b3e6b20 2024-03-28T01:08:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.294+0000 7f36da5ec740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b400dcc0 con 0x55af5b3ec680 2024-03-28T01:08:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.294+0000 7f36d579b640 1 -- 172.21.15.169:0/1449410535 learned_addr learned my addr 172.21.15.169:0/1449410535 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.294+0000 7f36d5f9c640 1 --2- 172.21.15.169:0/1449410535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af5b3ec680 0x55af5b3e52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.294+0000 7f36d579b640 1 -- 172.21.15.169:0/1449410535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af5b3ec680 msgr2=0x55af5b3e52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.294+0000 7f36d579b640 1 --2- 172.21.15.169:0/1449410535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af5b3ec680 0x55af5b3e52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.294+0000 7f36d579b640 1 -- 172.21.15.169:0/1449410535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3e7510 msgr2=0x55af5b3f83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.294+0000 7f36d579b640 1 --2- 172.21.15.169:0/1449410535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3e7510 0x55af5b3f83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.294+0000 7f36d579b640 1 -- 172.21.15.169:0/1449410535 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af5b3f1f00 con 0x55af5b3e6b20 2024-03-28T01:08:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.295+0000 7f36d579b640 1 --2- 172.21.15.169:0/1449410535 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5b3e6b20 0x55af5b3ef510 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f36cc00cbf0 tx=0x7f36cc005ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=650cbdaf3ac1117b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.303+0000 7f36baffd640 1 -- 172.21.15.169:0/1449410535 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc04c070 con 0x55af5b3e6b20 2024-03-28T01:08:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.303+0000 7f36baffd640 1 -- 172.21.15.169:0/1449410535 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36cc005520 con 0x55af5b3e6b20 2024-03-28T01:08:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.303+0000 7f36baffd640 1 -- 172.21.15.169:0/1449410535 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc0048c0 con 0x55af5b3e6b20 2024-03-28T01:08:03.309 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.306+0000 7f36da5ec740 1 -- 172.21.15.169:0/1449410535 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5b3e6b20 msgr2=0x55af5b3ef510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:03.309 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.306+0000 7f36da5ec740 1 --2- 172.21.15.169:0/1449410535 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5b3e6b20 0x55af5b3ef510 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f36cc00cbf0 tx=0x7f36cc005ce0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.311+0000 7f36da5ec740 1 -- 172.21.15.169:0/1449410535 shutdown_connections 2024-03-28T01:08:03.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.311+0000 7f36da5ec740 1 --2- 172.21.15.169:0/1449410535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af5b3ec680 0x55af5b3e52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.311+0000 7f36da5ec740 1 --2- 172.21.15.169:0/1449410535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3e7510 0x55af5b3f83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.314 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.311+0000 7f36da5ec740 1 --2- 172.21.15.169:0/1449410535 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5b3e6b20 0x55af5b3ef510 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.315 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.311+0000 7f36da5ec740 1 -- 172.21.15.169:0/1449410535 >> 172.21.15.169:0/1449410535 conn(0x55af5b3e2090 msgr2=0x55af5b3e2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:03.321 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.319+0000 7f36da5ec740 1 -- 172.21.15.169:0/1449410535 shutdown_connections 2024-03-28T01:08:03.323 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.321+0000 7f36da5ec740 1 -- 172.21.15.169:0/1449410535 wait complete. 2024-03-28T01:08:03.324 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.321+0000 7f36da5ec740 1 -- start start 2024-03-28T01:08:03.324 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.322+0000 7f36da5ec740 1 Processor -- start 2024-03-28T01:08:03.326 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.323+0000 7f36da5ec740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3e6b20 0x55af5b3f5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:03.326 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.323+0000 7f36da5ec740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af5b3e7510 0x55af5b3f5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:03.326 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.323+0000 7f36da5ec740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5b3ec680 0x55af5b3f6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:03.326 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.323+0000 7f36da5ec740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b400e2c0 con 0x55af5b3e6b20 2024-03-28T01:08:03.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.323+0000 7f36da5ec740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b400f5c0 con 0x55af5b3ec680 2024-03-28T01:08:03.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.323+0000 7f36da5ec740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b400e900 con 0x55af5b3e7510 2024-03-28T01:08:03.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.323+0000 7f36da5ec740 10 client.0 _ll_register_callbacks cb 0x7f36c000fc90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:03.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.324+0000 7f36d4f9a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af5b3e7510 0x55af5b3f5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:03.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.324+0000 7f36d4f9a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af5b3e7510 0x55af5b3f5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:57330/0 (socket says 172.21.15.169:57330) 2024-03-28T01:08:03.327 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.324+0000 7f36d4f9a640 1 -- 172.21.15.169:0/1150933477 learned_addr learned my addr 172.21.15.169:0/1150933477 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:03.328 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.325+0000 7f36d4f9a640 1 -- 172.21.15.169:0/1150933477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5b3ec680 msgr2=0x55af5b3f6d50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:08:03.328 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.325+0000 7f36d4f9a640 1 --2- 172.21.15.169:0/1150933477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5b3ec680 0x55af5b3f6d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.328 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.325+0000 7f36d4f9a640 1 -- 172.21.15.169:0/1150933477 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3e6b20 msgr2=0x55af5b3f5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:03.328 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.325+0000 7f36d4f9a640 1 --2- 172.21.15.169:0/1150933477 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3e6b20 0x55af5b3f5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:03.328 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.325+0000 7f36d4f9a640 1 -- 172.21.15.169:0/1150933477 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55af5b3f7340 con 0x55af5b3e7510 2024-03-28T01:08:03.328 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.325+0000 7f36d4f9a640 1 --2- 172.21.15.169:0/1150933477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af5b3e7510 0x55af5b3f5d40 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f36c8009bf0 tx=0x7f36c8009a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a19345da9153b396 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:03.329 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.325+0000 7f36b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:08:03.329 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.326+0000 7f36b8ff9640 1 -- 172.21.15.169:0/1150933477 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c80bd0e0 con 0x55af5b3e7510 2024-03-28T01:08:03.330 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.327+0000 7f36da5ec740 1 -- 172.21.15.169:0/1150933477 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36cc005520 con 0x55af5b3e7510 2024-03-28T01:08:03.330 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.327+0000 7f36b8ff9640 1 -- 172.21.15.169:0/1150933477 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f36c80ee9f0 con 0x55af5b3e7510 2024-03-28T01:08:03.337 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.334+0000 7f36b8ff9640 1 -- 172.21.15.169:0/1150933477 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36c8003e60 con 0x55af5b3e7510 2024-03-28T01:08:03.337 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.335+0000 7f36da5ec740 10 client.27403 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:03.337 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.335+0000 7f36da5ec740 1 -- 172.21.15.169:0/1150933477 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55af5b3f8720 con 0x55af5b3e7510 2024-03-28T01:08:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.342+0000 7f36b8ff9640 1 -- 172.21.15.169:0/1150933477 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36cc005520 con 0x55af5b3e7510 2024-03-28T01:08:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.342+0000 7f36b8ff9640 1 -- 172.21.15.169:0/1150933477 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f36c80b9040 con 0x55af5b3e7510 2024-03-28T01:08:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.342+0000 7f36da5ec740 10 client.27403 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.342+0000 7f36da5ec740 10 client.27403 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T01:08:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.342+0000 7f36da5ec740 20 client.27403 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.342+0000 7f36da5ec740 1 --2- 172.21.15.169:0/1150933477 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55af5b459490 0x55af5b45b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.342+0000 7f36da5ec740 4 client.27403 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T01:08:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.342+0000 7f36da5ec740 1 -- 172.21.15.169:0/1150933477 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55af5b3533a0 con 0x55af5b459490 2024-03-28T01:08:03.349 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.346+0000 7f36d579b640 1 --2- 172.21.15.169:0/1150933477 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55af5b459490 0x55af5b45b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:03.350 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.347+0000 7f36d579b640 1 --2- 172.21.15.169:0/1150933477 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55af5b459490 0x55af5b45b8b0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6e1bb22a0062c60 server_cookie=36f2ed8c31c84bc7 in_seq=0 out_seq=0 2024-03-28T01:08:03.350 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:03.347+0000 7f36b8ff9640 10 client.27403 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:04.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:03 smithi174 ceph-mon[36060]: pgmap v697: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 715 MiB/s wr, 1.13k op/s 2024-03-28T01:08:04.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:03 smithi177 ceph-mon[36206]: pgmap v697: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 715 MiB/s wr, 1.13k op/s 2024-03-28T01:08:04.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:03 smithi169 ceph-mon[32274]: pgmap v697: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 715 MiB/s wr, 1.13k op/s 2024-03-28T01:08:06.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:05 smithi174 ceph-mon[36060]: pgmap v698: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 545 MiB/s wr, 1.12k op/s 2024-03-28T01:08:06.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:05 smithi177 ceph-mon[36206]: pgmap v698: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 545 MiB/s wr, 1.12k op/s 2024-03-28T01:08:06.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:05 smithi169 ceph-mon[32274]: pgmap v698: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 545 MiB/s wr, 1.12k op/s 2024-03-28T01:08:07.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:06 smithi177 ceph-mon[36206]: pgmap v699: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 782 MiB/s wr, 1.34k op/s 2024-03-28T01:08:07.303 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:06 smithi169 ceph-mon[32274]: pgmap v699: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 782 MiB/s wr, 1.34k op/s 2024-03-28T01:08:07.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:06 smithi174 ceph-mon[36060]: pgmap v699: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 782 MiB/s wr, 1.34k op/s 2024-03-28T01:08:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.910+0000 7f36b8ff9640 1 -- 172.21.15.169:0/1150933477 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55af5b3533a0 con 0x55af5b459490 2024-03-28T01:08:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.910+0000 7f36b8ff9640 10 client.27403 handle_command_reply: tid=0 2024-03-28T01:08:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.910+0000 7f36da5ec740 1 client.27403 shutdown 2024-03-28T01:08:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.910+0000 7f36da5ec740 1 -- 172.21.15.169:0/1150933477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af5b3e7510 msgr2=0x55af5b3f5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.910+0000 7f36da5ec740 1 --2- 172.21.15.169:0/1150933477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af5b3e7510 0x55af5b3f5d40 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f36c8009bf0 tx=0x7f36c8009a10 comp rx=0 tx=0).stop 2024-03-28T01:08:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.910+0000 7f36da5ec740 1 -- 172.21.15.169:0/1150933477 shutdown_connections 2024-03-28T01:08:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.910+0000 7f36da5ec740 1 --2- 172.21.15.169:0/1150933477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5b3ec680 0x55af5b3f6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.910+0000 7f36da5ec740 1 --2- 172.21.15.169:0/1150933477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af5b3e7510 0x55af5b3f5d40 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.910+0000 7f36da5ec740 1 --2- 172.21.15.169:0/1150933477 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x55af5b459490 0x55af5b45b8b0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.910+0000 7f36da5ec740 1 --2- 172.21.15.169:0/1150933477 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55af5b3e6b20 0x55af5b3f5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.910+0000 7f36da5ec740 1 -- 172.21.15.169:0/1150933477 >> 172.21.15.169:0/1150933477 conn(0x55af5b3e2090 msgr2=0x55af5b3e52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.910+0000 7f36b8ff9640 0 client.27403 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:08.915 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.911+0000 7f36da5ec740 1 -- 172.21.15.169:0/1150933477 shutdown_connections 2024-03-28T01:08:08.915 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.911+0000 7f36da5ec740 1 -- 172.21.15.169:0/1150933477 wait complete. 2024-03-28T01:08:08.917 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.915+0000 7f36da5ec740 20 client.27403 trim_cache size 0 max 16384 2024-03-28T01:08:08.921 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:{ 2024-03-28T01:08:08.921 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "message": "", 2024-03-28T01:08:08.921 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout: "return_code": 0 2024-03-28T01:08:08.921 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stdout:} 2024-03-28T01:08:08.925 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.922+0000 7f36d679d640 1 -- 172.21.15.169:0/3640599821 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36bc0a2a00 msgr2=0x7f36bc0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.922+0000 7f36d679d640 1 --2- 172.21.15.169:0/3640599821 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36bc0a2a00 0x7f36bc0a4ec0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f36c0005fd0 tx=0x7f36c0005850 comp rx=0 tx=0).stop 2024-03-28T01:08:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.922+0000 7f36d679d640 1 -- 172.21.15.169:0/3640599821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36d00732b0 msgr2=0x7f36d01152a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.922+0000 7f36d679d640 1 --2- 172.21.15.169:0/3640599821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36d00732b0 0x7f36d01152a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f36c8006010 tx=0x7f36c8035d10 comp rx=0 tx=0).stop 2024-03-28T01:08:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.922+0000 7f36d679d640 1 -- 172.21.15.169:0/3640599821 shutdown_connections 2024-03-28T01:08:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.922+0000 7f36d679d640 1 --2- 172.21.15.169:0/3640599821 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36bc0a2a00 0x7f36bc0a4ec0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.922+0000 7f36d679d640 1 --2- 172.21.15.169:0/3640599821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36d0109390 0x7f36d01b66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.922+0000 7f36d679d640 1 --2- 172.21.15.169:0/3640599821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36d00732b0 0x7f36d01152a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.922+0000 7f36d679d640 1 --2- 172.21.15.169:0/3640599821 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36d0072950 0x7f36d0114d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.922+0000 7f36d679d640 1 -- 172.21.15.169:0/3640599821 >> 172.21.15.169:0/3640599821 conn(0x7f36d006e510 msgr2=0x7f36d00701a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.922+0000 7f36d679d640 1 -- 172.21.15.169:0/3640599821 shutdown_connections 2024-03-28T01:08:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi169.front.sepia.ceph.com.smithi169.stderr:2024-03-28T01:08:08.922+0000 7f36d679d640 1 -- 172.21.15.169:0/3640599821 wait complete. 2024-03-28T01:08:10.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:09 smithi169 ceph-mon[32274]: pgmap v700: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 601 MiB/s wr, 1.17k op/s 2024-03-28T01:08:10.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:09 smithi174 ceph-mon[36060]: pgmap v700: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 601 MiB/s wr, 1.17k op/s 2024-03-28T01:08:10.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:09 smithi177 ceph-mon[36206]: pgmap v700: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 601 MiB/s wr, 1.17k op/s 2024-03-28T01:08:11.781 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:08:11.781 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:08:12.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:11 smithi169 ceph-mon[32274]: pgmap v701: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 602 MiB/s wr, 1.23k op/s 2024-03-28T01:08:12.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:11 smithi174 ceph-mon[36060]: pgmap v701: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 602 MiB/s wr, 1.23k op/s 2024-03-28T01:08:12.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.114+0000 7f90c4db7640 1 -- 172.21.15.169:0/3368888156 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90c0109730 msgr2=0x7f90c010dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:12.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.114+0000 7f90c4db7640 1 --2- 172.21.15.169:0/3368888156 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90c0109730 0x7f90c010dfd0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f90a8009900 tx=0x7f90a802bc80 comp rx=0 tx=0).stop 2024-03-28T01:08:12.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.123+0000 7f90c4db7640 1 -- 172.21.15.169:0/3368888156 shutdown_connections 2024-03-28T01:08:12.126 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.123+0000 7f90c4db7640 1 --2- 172.21.15.169:0/3368888156 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90c0109730 0x7f90c010dfd0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.123+0000 7f90c4db7640 1 --2- 172.21.15.169:0/3368888156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90c0108d70 0x7f90c01091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.123+0000 7f90c4db7640 1 --2- 172.21.15.169:0/3368888156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90c0072ab0 0x7f90c0072ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.123+0000 7f90c4db7640 1 -- 172.21.15.169:0/3368888156 >> 172.21.15.169:0/3368888156 conn(0x7f90c006e580 msgr2=0x7f90c00709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:12.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.127+0000 7f90c4db7640 1 -- 172.21.15.169:0/3368888156 shutdown_connections 2024-03-28T01:08:12.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.129+0000 7f90c4db7640 1 -- 172.21.15.169:0/3368888156 wait complete. 2024-03-28T01:08:12.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.130+0000 7f90c4db7640 1 Processor -- start 2024-03-28T01:08:12.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.131+0000 7f90c4db7640 1 -- start start 2024-03-28T01:08:12.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.131+0000 7f90c4db7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90c0072ab0 0x7f90c01a6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.132+0000 7f90c4db7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90c0108d70 0x7f90c01a6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.132+0000 7f90c4db7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90c0109730 0x7f90c01adc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.132+0000 7f90c4db7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c011c070 con 0x7f90c0109730 2024-03-28T01:08:12.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.132+0000 7f90c4db7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c011bef0 con 0x7f90c0108d70 2024-03-28T01:08:12.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.132+0000 7f90beffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90c0108d70 0x7f90c01a6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:12.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.132+0000 7f90c4db7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c011c1f0 con 0x7f90c0072ab0 2024-03-28T01:08:12.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.132+0000 7f90beffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90c0108d70 0x7f90c01a6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:47658/0 (socket says 172.21.15.169:47658) 2024-03-28T01:08:12.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.132+0000 7f90beffd640 1 -- 172.21.15.169:0/2666643883 learned_addr learned my addr 172.21.15.169:0/2666643883 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:12.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.133+0000 7f90bffff640 1 --2- 172.21.15.169:0/2666643883 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90c0109730 0x7f90c01adc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:12.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.133+0000 7f90bffff640 1 -- 172.21.15.169:0/2666643883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90c0072ab0 msgr2=0x7f90c01a6680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:08:12.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.133+0000 7f90bffff640 1 --2- 172.21.15.169:0/2666643883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90c0072ab0 0x7f90c01a6680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.133+0000 7f90bffff640 1 -- 172.21.15.169:0/2666643883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90c0108d70 msgr2=0x7f90c01a6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:12.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.133+0000 7f90bffff640 1 --2- 172.21.15.169:0/2666643883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90c0108d70 0x7f90c01a6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.133+0000 7f90bffff640 1 -- 172.21.15.169:0/2666643883 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c01ae2a0 con 0x7f90c0109730 2024-03-28T01:08:12.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.134+0000 7f90bffff640 1 --2- 172.21.15.169:0/2666643883 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90c0109730 0x7f90c01adc30 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f90a8009850 tx=0x7f90a8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:12.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.134+0000 7f90bcff9640 1 -- 172.21.15.169:0/2666643883 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a8043020 con 0x7f90c0109730 2024-03-28T01:08:12.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.134+0000 7f90bcff9640 1 -- 172.21.15.169:0/2666643883 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90a8004480 con 0x7f90c0109730 2024-03-28T01:08:12.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.134+0000 7f90bcff9640 1 -- 172.21.15.169:0/2666643883 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a8003d80 con 0x7f90c0109730 2024-03-28T01:08:12.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.137+0000 7f90c4db7640 1 -- 172.21.15.169:0/2666643883 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c01ae530 con 0x7f90c0109730 2024-03-28T01:08:12.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.138+0000 7f90c4db7640 1 -- 172.21.15.169:0/2666643883 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c01ae940 con 0x7f90c0109730 2024-03-28T01:08:12.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.144+0000 7f90bcff9640 1 -- 172.21.15.169:0/2666643883 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f90a8046020 con 0x7f90c0109730 2024-03-28T01:08:12.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.145+0000 7f90bcff9640 1 --2- 172.21.15.169:0/2666643883 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f90ac0a2a00 0x7f90ac0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.146+0000 7f90bcff9640 1 -- 172.21.15.169:0/2666643883 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f90a80f0cc0 con 0x7f90c0109730 2024-03-28T01:08:12.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.156+0000 7f90c8c07740 1 Processor -- start 2024-03-28T01:08:12.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.156+0000 7f90bf7fe640 1 --2- 172.21.15.169:0/2666643883 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f90ac0a2a00 0x7f90ac0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:12.164 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.161+0000 7f90c8c07740 1 -- start start 2024-03-28T01:08:12.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.162+0000 7f90c8c07740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3ebf80 0x563e6c3ec3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.162+0000 7f90bf7fe640 1 --2- 172.21.15.169:0/2666643883 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f90ac0a2a00 0x7f90ac0a4ec0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f90c0073800 tx=0x7f90b8006bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:12.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.162+0000 7f90c8c07740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e6c3ec970 0x563e6c3f1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.162+0000 7f90c8c07740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3f1ae0 0x563e6c3f3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.162+0000 7f90bf7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3ebf80 0x563e6c3ec3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:12.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.162+0000 7f90bf7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3ebf80 0x563e6c3ec3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:58254/0 (socket says 172.21.15.169:58254) 2024-03-28T01:08:12.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.163+0000 7f90bf7fe640 1 -- 172.21.15.169:0/4019663502 learned_addr learned my addr 172.21.15.169:0/4019663502 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:12.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.163+0000 7f90c8c07740 1 -- 172.21.15.169:0/4019663502 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e6c3513d0 con 0x563e6c3f1ae0 2024-03-28T01:08:12.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.163+0000 7f90c8c07740 1 -- 172.21.15.169:0/4019663502 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e6c2f9ad0 con 0x563e6c3ec970 2024-03-28T01:08:12.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.163+0000 7f90c8c07740 1 -- 172.21.15.169:0/4019663502 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e6c34d470 con 0x563e6c3ebf80 2024-03-28T01:08:12.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:11 smithi177 ceph-mon[36206]: pgmap v701: 129 pgs: 129 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 602 MiB/s wr, 1.23k op/s 2024-03-28T01:08:12.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.164+0000 7f90beffd640 1 --2- 172.21.15.169:0/4019663502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e6c3ec970 0x563e6c3f1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:12.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.167+0000 7f90bffff640 1 --2- 172.21.15.169:0/4019663502 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3f1ae0 0x563e6c3f3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:12.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.172+0000 7f90bf7fe640 1 -- 172.21.15.169:0/4019663502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e6c3ec970 msgr2=0x563e6c3f1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:12.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.172+0000 7f90bf7fe640 1 --2- 172.21.15.169:0/4019663502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e6c3ec970 0x563e6c3f1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.172+0000 7f90bf7fe640 1 -- 172.21.15.169:0/4019663502 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3f1ae0 msgr2=0x563e6c3f3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:12.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.172+0000 7f90bf7fe640 1 --2- 172.21.15.169:0/4019663502 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3f1ae0 0x563e6c3f3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.172+0000 7f90bf7fe640 1 -- 172.21.15.169:0/4019663502 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e6c12f7d0 con 0x563e6c3ebf80 2024-03-28T01:08:12.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.177+0000 7f90bf7fe640 1 --2- 172.21.15.169:0/4019663502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3ebf80 0x563e6c3ec3a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f90b80138b0 tx=0x7f90b800e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d490d7599030ede server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:12.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.181+0000 7f90b5ffb640 1 -- 172.21.15.169:0/4019663502 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b8012ca0 con 0x563e6c3ebf80 2024-03-28T01:08:12.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.181+0000 7f90b5ffb640 1 -- 172.21.15.169:0/4019663502 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90b8022070 con 0x563e6c3ebf80 2024-03-28T01:08:12.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.183+0000 7f90c8c07740 1 -- 172.21.15.169:0/4019663502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3ebf80 msgr2=0x563e6c3ec3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:12.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.183+0000 7f90c8c07740 1 --2- 172.21.15.169:0/4019663502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3ebf80 0x563e6c3ec3a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f90b80138b0 tx=0x7f90b800e000 comp rx=0 tx=0).stop 2024-03-28T01:08:12.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.184+0000 7f90c8c07740 1 -- 172.21.15.169:0/4019663502 shutdown_connections 2024-03-28T01:08:12.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.184+0000 7f90c8c07740 1 --2- 172.21.15.169:0/4019663502 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3f1ae0 0x563e6c3f3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.184+0000 7f90c8c07740 1 --2- 172.21.15.169:0/4019663502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e6c3ec970 0x563e6c3f1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.184+0000 7f90c8c07740 1 --2- 172.21.15.169:0/4019663502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3ebf80 0x563e6c3ec3a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.184+0000 7f90c8c07740 1 -- 172.21.15.169:0/4019663502 >> 172.21.15.169:0/4019663502 conn(0x563e6c3e74f0 msgr2=0x563e6c3e78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:12.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.188+0000 7f90c8c07740 1 -- 172.21.15.169:0/4019663502 shutdown_connections 2024-03-28T01:08:12.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.188+0000 7f90c8c07740 1 -- 172.21.15.169:0/4019663502 wait complete. 2024-03-28T01:08:12.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.189+0000 7f90c8c07740 1 -- start start 2024-03-28T01:08:12.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.189+0000 7f90c8c07740 1 Processor -- start 2024-03-28T01:08:12.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.189+0000 7f90c8c07740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3ebf80 0x563e6c3f6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.189+0000 7f90c8c07740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e6c3ec970 0x563e6c3f6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.190+0000 7f90c8c07740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3f1ae0 0x563e6c3fd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.190+0000 7f90c8c07740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e6c3f4ae0 con 0x563e6c3ebf80 2024-03-28T01:08:12.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.190+0000 7f90c8c07740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e6c30d060 con 0x563e6c3ec970 2024-03-28T01:08:12.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.190+0000 7f90c8c07740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e6c3f4c60 con 0x563e6c3f1ae0 2024-03-28T01:08:12.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.190+0000 7f90c8c07740 10 client.0 _ll_register_callbacks cb 0x563e6c2ef6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:12.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.190+0000 7f90bffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3f1ae0 0x563e6c3fd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:12.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.190+0000 7f90bffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3f1ae0 0x563e6c3fd9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:58256/0 (socket says 172.21.15.169:58256) 2024-03-28T01:08:12.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.190+0000 7f90bffff640 1 -- 172.21.15.169:0/3123816546 learned_addr learned my addr 172.21.15.169:0/3123816546 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:12.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.190+0000 7f90beffd640 1 --2- 172.21.15.169:0/3123816546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e6c3ec970 0x563e6c3f6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:12.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.190+0000 7f90bffff640 1 -- 172.21.15.169:0/3123816546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e6c3ec970 msgr2=0x563e6c3f6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:12.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.190+0000 7f90bffff640 1 --2- 172.21.15.169:0/3123816546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e6c3ec970 0x563e6c3f6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.190+0000 7f90bffff640 1 -- 172.21.15.169:0/3123816546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3ebf80 msgr2=0x563e6c3f6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:12.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.190+0000 7f90bffff640 1 --2- 172.21.15.169:0/3123816546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3ebf80 0x563e6c3f6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.190+0000 7f90bffff640 1 -- 172.21.15.169:0/3123816546 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563e6c0653f0 con 0x563e6c3f1ae0 2024-03-28T01:08:12.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.191+0000 7f90bffff640 1 --2- 172.21.15.169:0/3123816546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3f1ae0 0x563e6c3fd9a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f90a802c290 tx=0x7f90a8004c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93266874308ff003 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:12.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.191+0000 7f909effd640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:08:12.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.191+0000 7f909effd640 1 -- 172.21.15.169:0/3123816546 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a80bd260 con 0x563e6c3f1ae0 2024-03-28T01:08:12.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.192+0000 7f909effd640 1 -- 172.21.15.169:0/3123816546 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f90a80ee4d0 con 0x563e6c3f1ae0 2024-03-28T01:08:12.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.194+0000 7f90c8c07740 1 -- 172.21.15.169:0/3123816546 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f90b8022070 con 0x563e6c3f1ae0 2024-03-28T01:08:12.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.194+0000 7f909effd640 1 -- 172.21.15.169:0/3123816546 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90a80c13b0 con 0x563e6c3f1ae0 2024-03-28T01:08:12.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.197+0000 7f90c8c07740 10 client.27409 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:12.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.197+0000 7f90c8c07740 1 -- 172.21.15.169:0/3123816546 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563e6c0250f0 con 0x563e6c3f1ae0 2024-03-28T01:08:12.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.201+0000 7f909effd640 1 -- 172.21.15.169:0/3123816546 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90b8022070 con 0x563e6c3f1ae0 2024-03-28T01:08:12.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.201+0000 7f909effd640 1 -- 172.21.15.169:0/3123816546 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f90a80c5570 con 0x563e6c3f1ae0 2024-03-28T01:08:12.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.202+0000 7f90c8c07740 10 client.27409 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:12.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.202+0000 7f90c8c07740 10 client.27409 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:08:12.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.203+0000 7f90c8c07740 20 client.27409 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:12.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.203+0000 7f90c8c07740 1 --2- 172.21.15.169:0/3123816546 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563e6c401f30 0x563e6c404320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.203+0000 7f90c8c07740 4 client.27409 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:08:12.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.203+0000 7f90c8c07740 1 -- 172.21.15.169:0/3123816546 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563e6c35e910 con 0x563e6c401f30 2024-03-28T01:08:12.207 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.204+0000 7f90bf7fe640 1 --2- 172.21.15.169:0/3123816546 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563e6c401f30 0x563e6c404320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:12.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.224+0000 7f90bf7fe640 1 --2- 172.21.15.169:0/3123816546 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563e6c401f30 0x563e6c404320 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7c772d9af6f8ab8 server_cookie=3a88ae52dcb79c55 in_seq=0 out_seq=0 2024-03-28T01:08:12.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.225+0000 7f909effd640 10 client.27409 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:12.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.228+0000 7f909effd640 1 -- 172.21.15.169:0/3123816546 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563e6c35e910 con 0x563e6c401f30 2024-03-28T01:08:12.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.228+0000 7f909effd640 10 client.27409 handle_command_reply: tid=0 2024-03-28T01:08:12.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.228+0000 7f90c8c07740 1 client.27409 shutdown 2024-03-28T01:08:12.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.236+0000 7f90c8c07740 1 -- 172.21.15.169:0/3123816546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3f1ae0 msgr2=0x563e6c3fd9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:12.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.236+0000 7f90c8c07740 1 --2- 172.21.15.169:0/3123816546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3f1ae0 0x563e6c3fd9a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f90a802c290 tx=0x7f90a8004c60 comp rx=0 tx=0).stop 2024-03-28T01:08:12.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.243+0000 7f90c8c07740 1 -- 172.21.15.169:0/3123816546 shutdown_connections 2024-03-28T01:08:12.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.243+0000 7f90c8c07740 1 --2- 172.21.15.169:0/3123816546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3f1ae0 0x563e6c3fd9a0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.243+0000 7f90c8c07740 1 --2- 172.21.15.169:0/3123816546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e6c3ec970 0x563e6c3f6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.243+0000 7f90c8c07740 1 --2- 172.21.15.169:0/3123816546 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563e6c401f30 0x563e6c404320 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.243+0000 7f90c8c07740 1 --2- 172.21.15.169:0/3123816546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3ebf80 0x563e6c3f6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.243+0000 7f90c8c07740 1 -- 172.21.15.169:0/3123816546 >> 172.21.15.169:0/3123816546 conn(0x563e6c3e74f0 msgr2=0x563e6c3e67b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:12.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.244+0000 7f909effd640 0 client.27409 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:12.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.248+0000 7f90c8c07740 1 -- 172.21.15.169:0/3123816546 shutdown_connections 2024-03-28T01:08:12.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.249+0000 7f90c8c07740 1 -- 172.21.15.169:0/3123816546 wait complete. 2024-03-28T01:08:12.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.249+0000 7f90c8c07740 20 client.27409 trim_cache size 0 max 16384 2024-03-28T01:08:12.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.292+0000 7f90c8c07740 1 Processor -- start 2024-03-28T01:08:12.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.292+0000 7f90c8c07740 1 -- start start 2024-03-28T01:08:12.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.292+0000 7f90c8c07740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3ebf80 0x563e6c403f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.294+0000 7f90c8c07740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e6c3ec970 0x563e6c3fd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.294+0000 7f90bf7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3ebf80 0x563e6c403f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:12.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.294+0000 7f90bf7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3ebf80 0x563e6c403f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51810/0 (socket says 172.21.15.169:51810) 2024-03-28T01:08:12.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.294+0000 7f90bf7fe640 1 -- 172.21.15.169:0/2592646684 learned_addr learned my addr 172.21.15.169:0/2592646684 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:12.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.294+0000 7f90c8c07740 1 --2- 172.21.15.169:0/2592646684 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3f1ae0 0x563e6c3e5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.294+0000 7f90c8c07740 1 -- 172.21.15.169:0/2592646684 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c01bb340 con 0x563e6c3ebf80 2024-03-28T01:08:12.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.294+0000 7f90c8c07740 1 -- 172.21.15.169:0/2592646684 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c01b8e40 con 0x563e6c3ec970 2024-03-28T01:08:12.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.294+0000 7f90bffff640 1 --2- 172.21.15.169:0/2592646684 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3f1ae0 0x563e6c3e5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:12.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.295+0000 7f90c8c07740 1 -- 172.21.15.169:0/2592646684 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c01b6ee0 con 0x563e6c3f1ae0 2024-03-28T01:08:12.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.299+0000 7f90bffff640 1 -- 172.21.15.169:0/2592646684 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e6c3ec970 msgr2=0x563e6c3fd960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T01:08:12.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.299+0000 7f90bffff640 1 --2- 172.21.15.169:0/2592646684 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e6c3ec970 0x563e6c3fd960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.299+0000 7f90bffff640 1 -- 172.21.15.169:0/2592646684 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3ebf80 msgr2=0x563e6c403f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:12.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.299+0000 7f90bffff640 1 --2- 172.21.15.169:0/2592646684 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3ebf80 0x563e6c403f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.299+0000 7f90bffff640 1 -- 172.21.15.169:0/2592646684 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e6c3e8260 con 0x563e6c3f1ae0 2024-03-28T01:08:12.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.299+0000 7f90bffff640 1 --2- 172.21.15.169:0/2592646684 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3f1ae0 0x563e6c3e5f20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f90a80bd1f0 tx=0x7f90a80b9630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55927672aef4749e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:12.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.300+0000 7f90b57fa640 1 -- 172.21.15.169:0/2592646684 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a80bcda0 con 0x563e6c3f1ae0 2024-03-28T01:08:12.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.300+0000 7f90b57fa640 1 -- 172.21.15.169:0/2592646684 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90a80bd260 con 0x563e6c3f1ae0 2024-03-28T01:08:12.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.301+0000 7f90c8c07740 1 -- 172.21.15.169:0/2592646684 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3f1ae0 msgr2=0x563e6c3e5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:12.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.301+0000 7f90c8c07740 1 --2- 172.21.15.169:0/2592646684 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3f1ae0 0x563e6c3e5f20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f90a80bd1f0 tx=0x7f90a80b9630 comp rx=0 tx=0).stop 2024-03-28T01:08:12.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.308+0000 7f90c8c07740 1 -- 172.21.15.169:0/2592646684 shutdown_connections 2024-03-28T01:08:12.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.308+0000 7f90c8c07740 1 --2- 172.21.15.169:0/2592646684 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3f1ae0 0x563e6c3e5f20 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.308+0000 7f90c8c07740 1 --2- 172.21.15.169:0/2592646684 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e6c3ec970 0x563e6c3fd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.308+0000 7f90c8c07740 1 --2- 172.21.15.169:0/2592646684 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3ebf80 0x563e6c403f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.308+0000 7f90c8c07740 1 -- 172.21.15.169:0/2592646684 >> 172.21.15.169:0/2592646684 conn(0x563e6c3e74f0 msgr2=0x563e6c3f6420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:12.318 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.316+0000 7f90c8c07740 1 -- 172.21.15.169:0/2592646684 shutdown_connections 2024-03-28T01:08:12.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.318+0000 7f90c8c07740 1 -- 172.21.15.169:0/2592646684 wait complete. 2024-03-28T01:08:12.322 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.319+0000 7f90c8c07740 1 -- start start 2024-03-28T01:08:12.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.319+0000 7f90c8c07740 1 Processor -- start 2024-03-28T01:08:12.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.321+0000 7f90c8c07740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e6c3ebf80 0x563e6c3fb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.321+0000 7f90c8c07740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3ec970 0x563e6c3fbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.322+0000 7f90c8c07740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3f1ae0 0x563e6c3fce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.322+0000 7f90c8c07740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c01b74e0 con 0x563e6c3ec970 2024-03-28T01:08:12.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.322+0000 7f90c8c07740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c01b87e0 con 0x563e6c3ebf80 2024-03-28T01:08:12.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.322+0000 7f90c8c07740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c01b7b20 con 0x563e6c3f1ae0 2024-03-28T01:08:12.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.322+0000 7f90c8c07740 10 client.0 _ll_register_callbacks cb 0x7f90a80bd030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:12.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.322+0000 7f90beffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3ec970 0x563e6c3fbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:12.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.322+0000 7f90beffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3ec970 0x563e6c3fbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51820/0 (socket says 172.21.15.169:51820) 2024-03-28T01:08:12.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.322+0000 7f90beffd640 1 -- 172.21.15.169:0/1530011754 learned_addr learned my addr 172.21.15.169:0/1530011754 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:12.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.323+0000 7f90beffd640 1 -- 172.21.15.169:0/1530011754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3f1ae0 msgr2=0x563e6c3fce00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:08:12.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.323+0000 7f90beffd640 1 --2- 172.21.15.169:0/1530011754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3f1ae0 0x563e6c3fce00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.323+0000 7f90beffd640 1 -- 172.21.15.169:0/1530011754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e6c3ebf80 msgr2=0x563e6c3fb9a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:08:12.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.323+0000 7f90beffd640 1 --2- 172.21.15.169:0/1530011754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e6c3ebf80 0x563e6c3fb9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.323+0000 7f90beffd640 1 -- 172.21.15.169:0/1530011754 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563e6c3fd3f0 con 0x563e6c3ec970 2024-03-28T01:08:12.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.325+0000 7f90beffd640 1 --2- 172.21.15.169:0/1530011754 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3ec970 0x563e6c3fbee0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f90c01a7b90 tx=0x7f90b0011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c39dbc6a4327642 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:12.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.326+0000 7f909effd640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:08:12.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.328+0000 7f909effd640 1 -- 172.21.15.169:0/1530011754 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b000fba0 con 0x563e6c3ec970 2024-03-28T01:08:12.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.328+0000 7f90c8c07740 1 -- 172.21.15.169:0/1530011754 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563e6c358c60 con 0x563e6c3ec970 2024-03-28T01:08:12.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.328+0000 7f909effd640 1 -- 172.21.15.169:0/1530011754 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f90b000fd40 con 0x563e6c3ec970 2024-03-28T01:08:12.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.329+0000 7f909effd640 1 -- 172.21.15.169:0/1530011754 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90b0025050 con 0x563e6c3ec970 2024-03-28T01:08:12.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.330+0000 7f90c8c07740 10 client.17559 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:12.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.330+0000 7f90c8c07740 1 -- 172.21.15.169:0/1530011754 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563e6c45e0c0 con 0x563e6c3ec970 2024-03-28T01:08:12.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.331+0000 7f909effd640 1 -- 172.21.15.169:0/1530011754 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563e6c358c60 con 0x563e6c3ec970 2024-03-28T01:08:12.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.331+0000 7f909effd640 1 -- 172.21.15.169:0/1530011754 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f90b000e710 con 0x563e6c3ec970 2024-03-28T01:08:12.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.331+0000 7f90c8c07740 10 client.17559 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:12.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.331+0000 7f90c8c07740 10 client.17559 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:08:12.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.331+0000 7f90c8c07740 20 client.17559 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:12.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.331+0000 7f90c8c07740 1 --2- 172.21.15.169:0/1530011754 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563e6c45ead0 0x563e6c460ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.331+0000 7f90c8c07740 4 client.17559 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:08:12.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.331+0000 7f90c8c07740 1 -- 172.21.15.169:0/1530011754 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563e6c2d64e0 con 0x563e6c45ead0 2024-03-28T01:08:12.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.334+0000 7f90bf7fe640 1 --2- 172.21.15.169:0/1530011754 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563e6c45ead0 0x563e6c460ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:12.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.341+0000 7f90bf7fe640 1 --2- 172.21.15.169:0/1530011754 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563e6c45ead0 0x563e6c460ef0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec46a096f09b7525 server_cookie=7d42b663d89fada3 in_seq=0 out_seq=0 2024-03-28T01:08:12.344 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.342+0000 7f909effd640 10 client.17559 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:12.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.349+0000 7f909effd640 1 -- 172.21.15.169:0/1530011754 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x563e6c2d64e0 con 0x563e6c45ead0 2024-03-28T01:08:12.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.349+0000 7f909effd640 10 client.17559 handle_command_reply: tid=0 2024-03-28T01:08:12.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.349+0000 7f90c8c07740 1 client.17559 shutdown 2024-03-28T01:08:12.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.355+0000 7f90c8c07740 1 -- 172.21.15.169:0/1530011754 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3ec970 msgr2=0x563e6c3fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:12.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.355+0000 7f90c8c07740 1 --2- 172.21.15.169:0/1530011754 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3ec970 0x563e6c3fbee0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f90c01a7b90 tx=0x7f90b0011ea0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.356+0000 7f90c8c07740 1 -- 172.21.15.169:0/1530011754 shutdown_connections 2024-03-28T01:08:12.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.356+0000 7f90c8c07740 1 --2- 172.21.15.169:0/1530011754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563e6c3f1ae0 0x563e6c3fce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.356+0000 7f90c8c07740 1 --2- 172.21.15.169:0/1530011754 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x563e6c45ead0 0x563e6c460ef0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.356+0000 7f90c8c07740 1 --2- 172.21.15.169:0/1530011754 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x563e6c3ec970 0x563e6c3fbee0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.356+0000 7f90c8c07740 1 --2- 172.21.15.169:0/1530011754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e6c3ebf80 0x563e6c3fb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.356+0000 7f90c8c07740 1 -- 172.21.15.169:0/1530011754 >> 172.21.15.169:0/1530011754 conn(0x563e6c3e74f0 msgr2=0x563e6c3ea750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:12.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.356+0000 7f909effd640 0 client.17559 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:12.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.359+0000 7f90c8c07740 1 -- 172.21.15.169:0/1530011754 shutdown_connections 2024-03-28T01:08:12.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.362+0000 7f90c8c07740 1 -- 172.21.15.169:0/1530011754 wait complete. 2024-03-28T01:08:12.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.363+0000 7f90c8c07740 20 client.17559 trim_cache size 0 max 16384 2024-03-28T01:08:12.367 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:08:12.368 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "no active scrubs running", 2024-03-28T01:08:12.368 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": {} 2024-03-28T01:08:12.368 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:08:12.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.366+0000 7f90c4db7640 1 -- 172.21.15.169:0/2666643883 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f90ac0a2a00 msgr2=0x7f90ac0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:12.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.366+0000 7f90c4db7640 1 --2- 172.21.15.169:0/2666643883 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f90ac0a2a00 0x7f90ac0a4ec0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f90c0073800 tx=0x7f90b8006bd0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.366+0000 7f90c4db7640 1 -- 172.21.15.169:0/2666643883 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90c0109730 msgr2=0x7f90c01adc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:12.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.366+0000 7f90c4db7640 1 --2- 172.21.15.169:0/2666643883 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90c0109730 0x7f90c01adc30 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f90a8009850 tx=0x7f90a8004040 comp rx=0 tx=0).stop 2024-03-28T01:08:12.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.370+0000 7f90c4db7640 1 -- 172.21.15.169:0/2666643883 shutdown_connections 2024-03-28T01:08:12.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.370+0000 7f90c4db7640 1 --2- 172.21.15.169:0/2666643883 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f90ac0a2a00 0x7f90ac0a4ec0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.370+0000 7f90c4db7640 1 --2- 172.21.15.169:0/2666643883 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f90c0109730 0x7f90c01adc30 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.371+0000 7f90c4db7640 1 --2- 172.21.15.169:0/2666643883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90c0108d70 0x7f90c01a6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.371+0000 7f90c4db7640 1 --2- 172.21.15.169:0/2666643883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90c0072ab0 0x7f90c01a6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.371+0000 7f90c4db7640 1 -- 172.21.15.169:0/2666643883 >> 172.21.15.169:0/2666643883 conn(0x7f90c006e580 msgr2=0x7f90c0070150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:12.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.376+0000 7f90c4db7640 1 -- 172.21.15.169:0/2666643883 shutdown_connections 2024-03-28T01:08:12.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.376+0000 7f90c4db7640 1 -- 172.21.15.169:0/2666643883 wait complete. 2024-03-28T01:08:12.408 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T01:08:12.409 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T01:08:12.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.837+0000 7efcac4ea640 1 -- 172.21.15.169:0/2419061096 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efca41086a0 msgr2=0x7efca410aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:12.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.837+0000 7efcac4ea640 1 --2- 172.21.15.169:0/2419061096 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efca41086a0 0x7efca410aad0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7efc94009900 tx=0x7efc9402bc80 comp rx=0 tx=0).stop 2024-03-28T01:08:12.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.841+0000 7efcac4ea640 1 -- 172.21.15.169:0/2419061096 shutdown_connections 2024-03-28T01:08:12.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.841+0000 7efcac4ea640 1 --2- 172.21.15.169:0/2419061096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efca410d9e0 0x7efca410de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.841+0000 7efcac4ea640 1 --2- 172.21.15.169:0/2419061096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efca410b010 0x7efca410d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.841+0000 7efcac4ea640 1 --2- 172.21.15.169:0/2419061096 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efca41086a0 0x7efca410aad0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.841+0000 7efcac4ea640 1 -- 172.21.15.169:0/2419061096 >> 172.21.15.169:0/2419061096 conn(0x7efca406e510 msgr2=0x7efca4070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:12.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.841+0000 7efcac4ea640 1 -- 172.21.15.169:0/2419061096 shutdown_connections 2024-03-28T01:08:12.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.842+0000 7efcac4ea640 1 -- 172.21.15.169:0/2419061096 wait complete. 2024-03-28T01:08:12.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.844+0000 7efcac4ea640 1 Processor -- start 2024-03-28T01:08:12.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.844+0000 7efcac4ea640 1 -- start start 2024-03-28T01:08:12.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.844+0000 7efcac4ea640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efca41086a0 0x7efca41a6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.844+0000 7efcac4ea640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efca410b010 0x7efca41a6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.844+0000 7efcac4ea640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efca410d9e0 0x7efca41adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.844+0000 7efcac4ea640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca411c220 con 0x7efca410b010 2024-03-28T01:08:12.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.844+0000 7efcac4ea640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca411c0a0 con 0x7efca41086a0 2024-03-28T01:08:12.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.844+0000 7efcac4ea640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca411c3a0 con 0x7efca410d9e0 2024-03-28T01:08:12.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.845+0000 7efca9a85640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efca410b010 0x7efca41a6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:12.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.845+0000 7efca9a85640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efca410b010 0x7efca41a6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51838/0 (socket says 172.21.15.169:51838) 2024-03-28T01:08:12.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.845+0000 7efca9a85640 1 -- 172.21.15.169:0/2784206075 learned_addr learned my addr 172.21.15.169:0/2784206075 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:12.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.845+0000 7efcaa286640 1 --2- 172.21.15.169:0/2784206075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efca41086a0 0x7efca41a6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:12.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.846+0000 7efcaaa87640 1 --2- 172.21.15.169:0/2784206075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efca410d9e0 0x7efca41adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:12.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.846+0000 7efca9a85640 1 -- 172.21.15.169:0/2784206075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efca410d9e0 msgr2=0x7efca41adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:12.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.846+0000 7efca9a85640 1 --2- 172.21.15.169:0/2784206075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efca410d9e0 0x7efca41adda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.846+0000 7efca9a85640 1 -- 172.21.15.169:0/2784206075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efca41086a0 msgr2=0x7efca41a6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:12.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.846+0000 7efca9a85640 1 --2- 172.21.15.169:0/2784206075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efca41086a0 0x7efca41a6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:12.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.846+0000 7efca9a85640 1 -- 172.21.15.169:0/2784206075 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca41ae410 con 0x7efca410b010 2024-03-28T01:08:12.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.847+0000 7efca9a85640 1 --2- 172.21.15.169:0/2784206075 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efca410b010 0x7efca41a6dd0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7efca000e9f0 tx=0x7efca000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:12.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.847+0000 7efc9b7fe640 1 -- 172.21.15.169:0/2784206075 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca000cd80 con 0x7efca410b010 2024-03-28T01:08:12.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.850+0000 7efcac4ea640 1 -- 172.21.15.169:0/2784206075 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca41ae700 con 0x7efca410b010 2024-03-28T01:08:12.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.850+0000 7efcac4ea640 1 -- 172.21.15.169:0/2784206075 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca41aeca0 con 0x7efca410b010 2024-03-28T01:08:12.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.851+0000 7efc9b7fe640 1 -- 172.21.15.169:0/2784206075 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efca0010910 con 0x7efca410b010 2024-03-28T01:08:12.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.851+0000 7efc9b7fe640 1 -- 172.21.15.169:0/2784206075 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca00188f0 con 0x7efca410b010 2024-03-28T01:08:12.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.856+0000 7efcac4ea640 1 -- 172.21.15.169:0/2784206075 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc880052c0 con 0x7efca410b010 2024-03-28T01:08:12.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.864+0000 7efc9b7fe640 1 -- 172.21.15.169:0/2784206075 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7efca0010430 con 0x7efca410b010 2024-03-28T01:08:12.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.865+0000 7efc9b7fe640 1 --2- 172.21.15.169:0/2784206075 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efc900a2ab0 0x7efc900a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:12.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.865+0000 7efc9b7fe640 1 -- 172.21.15.169:0/2784206075 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7efca0018690 con 0x7efca410b010 2024-03-28T01:08:12.873 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.870+0000 7efcaa286640 1 --2- 172.21.15.169:0/2784206075 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efc900a2ab0 0x7efc900a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:12.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.871+0000 7efcaa286640 1 --2- 172.21.15.169:0/2784206075 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efc900a2ab0 0x7efc900a4f70 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7efc9402c290 tx=0x7efc94000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:12.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:12.872+0000 7efc9b7fe640 1 -- 172.21.15.169:0/2784206075 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efca009bcd0 con 0x7efca410b010 2024-03-28T01:08:13.040 INFO:tasks.workunit.client.0.smithi169.stdout:stopping iogen 2024-03-28T01:08:13.040 INFO:tasks.workunit.client.0.smithi169.stderr:+ echo 'stopping iogen' 2024-03-28T01:08:13.040 INFO:tasks.workunit.client.0.smithi169.stderr:+ ./iogen -k 2024-03-28T01:08:13.130 INFO:tasks.workunit.client.0.smithi169.stdout:OK 2024-03-28T01:08:13.131 INFO:tasks.workunit.client.0.smithi169.stderr:+ echo OK 2024-03-28T01:08:13.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.444+0000 7efcac4ea640 1 -- 172.21.15.169:0/2784206075 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc880055b0 con 0x7efca410b010 2024-03-28T01:08:13.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.446+0000 7efc9b7fe640 1 -- 172.21.15.169:0/2784206075 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7efca009b5d0 con 0x7efca410b010 2024-03-28T01:08:13.451 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T01:08:13.452 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T01:08:13.452 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T01:08:13.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.458+0000 7efcac4ea640 1 -- 172.21.15.169:0/2784206075 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efc900a2ab0 msgr2=0x7efc900a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:13.461 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.458+0000 7efcac4ea640 1 --2- 172.21.15.169:0/2784206075 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efc900a2ab0 0x7efc900a4f70 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7efc9402c290 tx=0x7efc94000bc0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.461 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.458+0000 7efcac4ea640 1 -- 172.21.15.169:0/2784206075 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efca410b010 msgr2=0x7efca41a6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:13.461 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.458+0000 7efcac4ea640 1 --2- 172.21.15.169:0/2784206075 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efca410b010 0x7efca41a6dd0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7efca000e9f0 tx=0x7efca000eeb0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.463+0000 7efcac4ea640 1 -- 172.21.15.169:0/2784206075 shutdown_connections 2024-03-28T01:08:13.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.463+0000 7efcac4ea640 1 --2- 172.21.15.169:0/2784206075 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7efc900a2ab0 0x7efc900a4f70 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.463+0000 7efcac4ea640 1 --2- 172.21.15.169:0/2784206075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efca410d9e0 0x7efca41adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.463+0000 7efcac4ea640 1 --2- 172.21.15.169:0/2784206075 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efca410b010 0x7efca41a6dd0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.463+0000 7efcac4ea640 1 --2- 172.21.15.169:0/2784206075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efca41086a0 0x7efca41a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.463+0000 7efcac4ea640 1 -- 172.21.15.169:0/2784206075 >> 172.21.15.169:0/2784206075 conn(0x7efca406e510 msgr2=0x7efca410b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:13.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.465+0000 7efcac4ea640 1 -- 172.21.15.169:0/2784206075 shutdown_connections 2024-03-28T01:08:13.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.466+0000 7efcac4ea640 1 -- 172.21.15.169:0/2784206075 wait complete. 2024-03-28T01:08:13.507 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:08:13.507 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T01:08:13.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.851+0000 7f36b5d10640 1 -- 172.21.15.169:0/33004055 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36b010b2d0 msgr2=0x7f36b010d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:13.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.851+0000 7f36b5d10640 1 --2- 172.21.15.169:0/33004055 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36b010b2d0 0x7f36b010d6e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f36a400af00 tx=0x7f36a402fe60 comp rx=0 tx=0).stop 2024-03-28T01:08:13.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.852+0000 7f36b5d10640 1 -- 172.21.15.169:0/33004055 shutdown_connections 2024-03-28T01:08:13.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.852+0000 7f36b5d10640 1 --2- 172.21.15.169:0/33004055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36b010dd70 0x7f36b0110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.852+0000 7f36b5d10640 1 --2- 172.21.15.169:0/33004055 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36b010b2d0 0x7f36b010d6e0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.852+0000 7f36b5d10640 1 --2- 172.21.15.169:0/33004055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36b0108910 0x7f36b010ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.852+0000 7f36b5d10640 1 -- 172.21.15.169:0/33004055 >> 172.21.15.169:0/33004055 conn(0x7f36b006e500 msgr2=0x7f36b0070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:13.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.856+0000 7f36b5d10640 1 -- 172.21.15.169:0/33004055 shutdown_connections 2024-03-28T01:08:13.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.857+0000 7f36b5d10640 1 -- 172.21.15.169:0/33004055 wait complete. 2024-03-28T01:08:13.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.858+0000 7f36b5d10640 1 Processor -- start 2024-03-28T01:08:13.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.859+0000 7f36b5d10640 1 -- start start 2024-03-28T01:08:13.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.860+0000 7f36b5d10640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36b0108910 0x7f36b019ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:13.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.860+0000 7f36b5d10640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36b010b2d0 0x7f36b019e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:13.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.860+0000 7f36b5d10640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36b010dd70 0x7f36b01a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:13.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.860+0000 7f36b5d10640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b0113a00 con 0x7f36b010dd70 2024-03-28T01:08:13.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.860+0000 7f36b5d10640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b0113880 con 0x7f36b0108910 2024-03-28T01:08:13.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.860+0000 7f36b5d10640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b0113b80 con 0x7f36b010b2d0 2024-03-28T01:08:13.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.861+0000 7f36affff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36b010dd70 0x7f36b01a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:13.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.861+0000 7f36affff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36b010dd70 0x7f36b01a5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51854/0 (socket says 172.21.15.169:51854) 2024-03-28T01:08:13.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.861+0000 7f36affff640 1 -- 172.21.15.169:0/600812685 learned_addr learned my addr 172.21.15.169:0/600812685 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:13.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.862+0000 7f36affff640 1 -- 172.21.15.169:0/600812685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36b010b2d0 msgr2=0x7f36b019e410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:08:13.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.862+0000 7f36affff640 1 --2- 172.21.15.169:0/600812685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36b010b2d0 0x7f36b019e410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.862+0000 7f36affff640 1 -- 172.21.15.169:0/600812685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36b0108910 msgr2=0x7f36b019ded0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:08:13.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.862+0000 7f36affff640 1 --2- 172.21.15.169:0/600812685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36b0108910 0x7f36b019ded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.862+0000 7f36affff640 1 -- 172.21.15.169:0/600812685 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b01a5af0 con 0x7f36b010dd70 2024-03-28T01:08:13.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.863+0000 7f36affff640 1 --2- 172.21.15.169:0/600812685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36b010dd70 0x7f36b01a5480 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f36a800b9e0 tx=0x7f36a800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:13.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.867+0000 7f36acff9640 1 -- 172.21.15.169:0/600812685 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a8019070 con 0x7f36b010dd70 2024-03-28T01:08:13.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.867+0000 7f36acff9640 1 -- 172.21.15.169:0/600812685 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36a8014440 con 0x7f36b010dd70 2024-03-28T01:08:13.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.867+0000 7f36acff9640 1 -- 172.21.15.169:0/600812685 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a8012620 con 0x7f36b010dd70 2024-03-28T01:08:13.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.867+0000 7f36b5d10640 1 -- 172.21.15.169:0/600812685 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b01a5de0 con 0x7f36b010dd70 2024-03-28T01:08:13.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.867+0000 7f36b5d10640 1 -- 172.21.15.169:0/600812685 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b01aea80 con 0x7f36b010dd70 2024-03-28T01:08:13.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.874+0000 7f36acff9640 1 -- 172.21.15.169:0/600812685 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f36a801c070 con 0x7f36b010dd70 2024-03-28T01:08:13.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.875+0000 7f36acff9640 1 --2- 172.21.15.169:0/600812685 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36980a29d0 0x7f36980a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:13.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.875+0000 7f36acff9640 1 -- 172.21.15.169:0/600812685 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f36a80cf430 con 0x7f36b010dd70 2024-03-28T01:08:13.879 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.875+0000 7f36af7fe640 1 --2- 172.21.15.169:0/600812685 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36980a29d0 0x7f36980a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:13.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.885+0000 7f36b89dd740 1 Processor -- start 2024-03-28T01:08:13.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.885+0000 7f36b89dd740 1 -- start start 2024-03-28T01:08:13.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.886+0000 7f36b89dd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557d64ce0690 0x557d64ce0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:13.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.886+0000 7f36b89dd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce1080 0x557d64ce5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:13.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.886+0000 7f36b89dd740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d64ce61f0 0x557d64ce8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:13.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.886+0000 7f36b89dd740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d64ca3610 con 0x557d64ce61f0 2024-03-28T01:08:13.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.886+0000 7f36b89dd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d64c463c0 con 0x557d64ce0690 2024-03-28T01:08:13.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.886+0000 7f36b89dd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d64beed60 con 0x557d64ce1080 2024-03-28T01:08:13.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.889+0000 7f36af7fe640 1 --2- 172.21.15.169:0/600812685 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36980a29d0 0x7f36980a4e90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f36a000a930 tx=0x7f36a0005c00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:13.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.889+0000 7f36affff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d64ce61f0 0x557d64ce8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:13.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.889+0000 7f36affff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d64ce61f0 0x557d64ce8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51864/0 (socket says 172.21.15.169:51864) 2024-03-28T01:08:13.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.889+0000 7f36affff640 1 -- 172.21.15.169:0/1857302159 learned_addr learned my addr 172.21.15.169:0/1857302159 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:13.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.889+0000 7f36aeffd640 1 --2- 172.21.15.169:0/1857302159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce1080 0x557d64ce5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:13.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.890+0000 7f36affff640 1 -- 172.21.15.169:0/1857302159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce1080 msgr2=0x557d64ce5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:13.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.890+0000 7f36affff640 1 --2- 172.21.15.169:0/1857302159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce1080 0x557d64ce5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.890+0000 7f36affff640 1 -- 172.21.15.169:0/1857302159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557d64ce0690 msgr2=0x557d64ce0ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:08:13.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.890+0000 7f36affff640 1 --2- 172.21.15.169:0/1857302159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557d64ce0690 0x557d64ce0ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.890+0000 7f36affff640 1 -- 172.21.15.169:0/1857302159 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d6495a3f0 con 0x557d64ce61f0 2024-03-28T01:08:13.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.897+0000 7f36affff640 1 --2- 172.21.15.169:0/1857302159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d64ce61f0 0x557d64ce8680 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f36a8014fd0 tx=0x7f36a8004a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65e2f09014b4ea61 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:13.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.901+0000 7f3695ffb640 1 -- 172.21.15.169:0/1857302159 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a809fb20 con 0x557d64ce61f0 2024-03-28T01:08:13.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.901+0000 7f3695ffb640 1 -- 172.21.15.169:0/1857302159 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36a80a4170 con 0x557d64ce61f0 2024-03-28T01:08:13.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.901+0000 7f3695ffb640 1 -- 172.21.15.169:0/1857302159 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a809be90 con 0x557d64ce61f0 2024-03-28T01:08:13.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.902+0000 7f36b89dd740 1 -- 172.21.15.169:0/1857302159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d64ce61f0 msgr2=0x557d64ce8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:13.906 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.902+0000 7f36b89dd740 1 --2- 172.21.15.169:0/1857302159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d64ce61f0 0x557d64ce8680 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f36a8014fd0 tx=0x7f36a8004a60 comp rx=0 tx=0).stop 2024-03-28T01:08:13.906 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.902+0000 7f36b89dd740 1 -- 172.21.15.169:0/1857302159 shutdown_connections 2024-03-28T01:08:13.906 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.902+0000 7f36b89dd740 1 --2- 172.21.15.169:0/1857302159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d64ce61f0 0x557d64ce8680 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.906 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.902+0000 7f36b89dd740 1 --2- 172.21.15.169:0/1857302159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce1080 0x557d64ce5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.906 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.902+0000 7f36b89dd740 1 --2- 172.21.15.169:0/1857302159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557d64ce0690 0x557d64ce0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.906 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.903+0000 7f36b89dd740 1 -- 172.21.15.169:0/1857302159 >> 172.21.15.169:0/1857302159 conn(0x557d64cdbc00 msgr2=0x557d64cdc000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:13.912 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.909+0000 7f36b89dd740 1 -- 172.21.15.169:0/1857302159 shutdown_connections 2024-03-28T01:08:13.912 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.909+0000 7f36b89dd740 1 -- 172.21.15.169:0/1857302159 wait complete. 2024-03-28T01:08:13.912 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.909+0000 7f36b89dd740 1 -- start start 2024-03-28T01:08:13.913 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.910+0000 7f36b89dd740 1 Processor -- start 2024-03-28T01:08:13.914 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.910+0000 7f36b89dd740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d64ce0690 0x557d64cfbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:13.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.911+0000 7f36b89dd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce1080 0x557d64cfc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:13.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.911+0000 7f36b89dd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557d64ce61f0 0x557d64d03080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:13.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.911+0000 7f36b89dd740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d64c02240 con 0x557d64ce0690 2024-03-28T01:08:13.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.911+0000 7f36b89dd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d64c42460 con 0x557d64ce61f0 2024-03-28T01:08:13.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.911+0000 7f36b89dd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d64cfa200 con 0x557d64ce1080 2024-03-28T01:08:13.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.911+0000 7f36b89dd740 10 client.0 _ll_register_callbacks cb 0x557d64c01c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:13.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.912+0000 7f36aeffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce1080 0x557d64cfc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:13.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.912+0000 7f36aeffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce1080 0x557d64cfc040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:58322/0 (socket says 172.21.15.169:58322) 2024-03-28T01:08:13.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.912+0000 7f36aeffd640 1 -- 172.21.15.169:0/1063395964 learned_addr learned my addr 172.21.15.169:0/1063395964 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:13.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.912+0000 7f36aeffd640 1 -- 172.21.15.169:0/1063395964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557d64ce61f0 msgr2=0x557d64d03080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:13.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.912+0000 7f36aeffd640 1 --2- 172.21.15.169:0/1063395964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557d64ce61f0 0x557d64d03080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.912+0000 7f36aeffd640 1 -- 172.21.15.169:0/1063395964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d64ce0690 msgr2=0x557d64cfbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:13.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.912+0000 7f36aeffd640 1 --2- 172.21.15.169:0/1063395964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d64ce0690 0x557d64cfbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.912+0000 7f36aeffd640 1 -- 172.21.15.169:0/1063395964 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d64a247d0 con 0x557d64ce1080 2024-03-28T01:08:13.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.914+0000 7f36aeffd640 1 --2- 172.21.15.169:0/1063395964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce1080 0x557d64cfc040 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f36b00737f0 tx=0x7f36a4012650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ace0d6cc86eb7b22 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:13.918 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.914+0000 7f368effd640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:08:13.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.916+0000 7f368effd640 1 -- 172.21.15.169:0/1063395964 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a4045060 con 0x557d64ce1080 2024-03-28T01:08:13.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.916+0000 7f368effd640 1 -- 172.21.15.169:0/1063395964 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f36a4004300 con 0x557d64ce1080 2024-03-28T01:08:13.921 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.918+0000 7f36b89dd740 1 -- 172.21.15.169:0/1063395964 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36a80a4170 con 0x557d64ce1080 2024-03-28T01:08:13.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.920+0000 7f368effd640 1 -- 172.21.15.169:0/1063395964 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36a400a210 con 0x557d64ce1080 2024-03-28T01:08:13.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.921+0000 7f36b89dd740 10 client.27421 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:13.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.921+0000 7f36b89dd740 1 -- 172.21.15.169:0/1063395964 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557d64d06490 con 0x557d64ce1080 2024-03-28T01:08:13.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.926+0000 7f368effd640 1 -- 172.21.15.169:0/1063395964 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36a80a4170 con 0x557d64ce1080 2024-03-28T01:08:13.931 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.926+0000 7f368effd640 1 -- 172.21.15.169:0/1063395964 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f36a4003b00 con 0x557d64ce1080 2024-03-28T01:08:13.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.929+0000 7f36b89dd740 10 client.27421 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:13.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.929+0000 7f36b89dd740 10 client.27421 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:08:13.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.929+0000 7f36b89dd740 20 client.27421 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:13.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.929+0000 7f36b89dd740 1 --2- 172.21.15.169:0/1063395964 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x557d64d07b00 0x557d64d09f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:13.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.929+0000 7f36b89dd740 4 client.27421 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:08:13.932 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.929+0000 7f36b89dd740 1 -- 172.21.15.169:0/1063395964 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557d64c53900 con 0x557d64d07b00 2024-03-28T01:08:13.935 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.931+0000 7f36af7fe640 1 --2- 172.21.15.169:0/1063395964 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x557d64d07b00 0x557d64d09f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:13.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.941+0000 7f36af7fe640 1 --2- 172.21.15.169:0/1063395964 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x557d64d07b00 0x557d64d09f40 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=90d199701e1d0a77 server_cookie=d12c45579e4b38d7 in_seq=0 out_seq=0 2024-03-28T01:08:13.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.941+0000 7f368effd640 10 client.27421 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:08:13.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.943+0000 7f368effd640 1 -- 172.21.15.169:0/1063395964 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557d64c53900 con 0x557d64d07b00 2024-03-28T01:08:13.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.944+0000 7f368effd640 10 client.27421 handle_command_reply: tid=0 2024-03-28T01:08:13.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.944+0000 7f36b89dd740 1 client.27421 shutdown 2024-03-28T01:08:13.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.949+0000 7f36b89dd740 1 -- 172.21.15.169:0/1063395964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce1080 msgr2=0x557d64cfc040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:13.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.949+0000 7f36b89dd740 1 --2- 172.21.15.169:0/1063395964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce1080 0x557d64cfc040 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f36b00737f0 tx=0x7f36a4012650 comp rx=0 tx=0).stop 2024-03-28T01:08:13.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.955+0000 7f36b89dd740 1 -- 172.21.15.169:0/1063395964 shutdown_connections 2024-03-28T01:08:13.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.955+0000 7f36b89dd740 1 --2- 172.21.15.169:0/1063395964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557d64ce61f0 0x557d64d03080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.955+0000 7f36b89dd740 1 --2- 172.21.15.169:0/1063395964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce1080 0x557d64cfc040 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.955+0000 7f36b89dd740 1 --2- 172.21.15.169:0/1063395964 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x557d64d07b00 0x557d64d09f40 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.955+0000 7f36b89dd740 1 --2- 172.21.15.169:0/1063395964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d64ce0690 0x557d64cfbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:13.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.955+0000 7f36b89dd740 1 -- 172.21.15.169:0/1063395964 >> 172.21.15.169:0/1063395964 conn(0x557d64cdbc00 msgr2=0x557d64cdae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:13.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.955+0000 7f368effd640 0 client.27421 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:08:13.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.956+0000 7f36b89dd740 1 -- 172.21.15.169:0/1063395964 shutdown_connections 2024-03-28T01:08:13.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.960+0000 7f36b89dd740 1 -- 172.21.15.169:0/1063395964 wait complete. 2024-03-28T01:08:13.967 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.962+0000 7f36b89dd740 20 client.27421 trim_cache size 0 max 16384 2024-03-28T01:08:14.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.997+0000 7f36b89dd740 1 Processor -- start 2024-03-28T01:08:14.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.997+0000 7f36b89dd740 1 -- start start 2024-03-28T01:08:14.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.997+0000 7f36b89dd740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d64ce0690 0x557d64ce2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.997+0000 7f36b89dd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce1080 0x557d64d03040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.997+0000 7f36b89dd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557d64ce61f0 0x557d64ce2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.000 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.997+0000 7f36b89dd740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c012120 con 0x557d64ce0690 2024-03-28T01:08:14.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.997+0000 7f36b89dd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c00fc20 con 0x557d64ce61f0 2024-03-28T01:08:14.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.997+0000 7f36b89dd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c00dcc0 con 0x557d64ce1080 2024-03-28T01:08:14.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.998+0000 7f36affff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557d64ce61f0 0x557d64ce2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.998+0000 7f36aeffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce1080 0x557d64d03040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.998+0000 7f36aeffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce1080 0x557d64d03040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:58330/0 (socket says 172.21.15.169:58330) 2024-03-28T01:08:14.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:13.998+0000 7f36aeffd640 1 -- 172.21.15.169:0/2574124352 learned_addr learned my addr 172.21.15.169:0/2574124352 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:14.003 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.000+0000 7f36af7fe640 1 --2- 172.21.15.169:0/2574124352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d64ce0690 0x557d64ce2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.001+0000 7f36aeffd640 1 -- 172.21.15.169:0/2574124352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557d64ce61f0 msgr2=0x557d64ce2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:14.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.001+0000 7f36aeffd640 1 --2- 172.21.15.169:0/2574124352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557d64ce61f0 0x557d64ce2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.001+0000 7f36aeffd640 1 -- 172.21.15.169:0/2574124352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d64ce0690 msgr2=0x557d64ce2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:14.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.001+0000 7f36aeffd640 1 --2- 172.21.15.169:0/2574124352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d64ce0690 0x557d64ce2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.001+0000 7f36aeffd640 1 -- 172.21.15.169:0/2574124352 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d64cd7670 con 0x557d64ce1080 2024-03-28T01:08:14.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.003+0000 7f36aeffd640 1 --2- 172.21.15.169:0/2574124352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce1080 0x557d64d03040 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f36a4003ed0 tx=0x7f36a402fe60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8352d0e4d6d7a1cc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:14.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.008+0000 7f36957fa640 1 -- 172.21.15.169:0/2574124352 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a4045070 con 0x557d64ce1080 2024-03-28T01:08:14.011 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.008+0000 7f36957fa640 1 -- 172.21.15.169:0/2574124352 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36a4005850 con 0x557d64ce1080 2024-03-28T01:08:14.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.009+0000 7f36b89dd740 1 -- 172.21.15.169:0/2574124352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce1080 msgr2=0x557d64d03040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:14.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.009+0000 7f36b89dd740 1 --2- 172.21.15.169:0/2574124352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce1080 0x557d64d03040 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f36a4003ed0 tx=0x7f36a402fe60 comp rx=0 tx=0).stop 2024-03-28T01:08:14.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.010+0000 7f36b89dd740 1 -- 172.21.15.169:0/2574124352 shutdown_connections 2024-03-28T01:08:14.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.010+0000 7f36b89dd740 1 --2- 172.21.15.169:0/2574124352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557d64ce61f0 0x557d64ce2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.010+0000 7f36b89dd740 1 --2- 172.21.15.169:0/2574124352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce1080 0x557d64d03040 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.010+0000 7f36b89dd740 1 --2- 172.21.15.169:0/2574124352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d64ce0690 0x557d64ce2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.010+0000 7f36b89dd740 1 -- 172.21.15.169:0/2574124352 >> 172.21.15.169:0/2574124352 conn(0x557d64cdbc00 msgr2=0x557d64cfa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:14.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.013+0000 7f36b89dd740 1 -- 172.21.15.169:0/2574124352 shutdown_connections 2024-03-28T01:08:14.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.013+0000 7f36b89dd740 1 -- 172.21.15.169:0/2574124352 wait complete. 2024-03-28T01:08:14.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.013+0000 7f36b89dd740 1 -- start start 2024-03-28T01:08:14.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.014+0000 7f36b89dd740 1 Processor -- start 2024-03-28T01:08:14.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.014+0000 7f36b89dd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce0690 0x557d64d00790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.014+0000 7f36b89dd740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d64ce1080 0x557d64d00cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.014+0000 7f36b89dd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557d64ce61f0 0x557d64d01ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.014+0000 7f36b89dd740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c00e2c0 con 0x557d64ce1080 2024-03-28T01:08:14.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.014+0000 7f36b89dd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c00f5c0 con 0x557d64ce61f0 2024-03-28T01:08:14.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.014+0000 7f36b89dd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c00e900 con 0x557d64ce0690 2024-03-28T01:08:14.018 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.014+0000 7f36b89dd740 10 client.0 _ll_register_callbacks cb 0x7f36a4030ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:14.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.016+0000 7f36affff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557d64ce61f0 0x557d64d01ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.016+0000 7f36af7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce0690 0x557d64d00790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.016+0000 7f36affff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557d64ce61f0 0x557d64d01ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:47756/0 (socket says 172.21.15.169:47756) 2024-03-28T01:08:14.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.016+0000 7f36affff640 1 -- 172.21.15.169:0/2257826906 learned_addr learned my addr 172.21.15.169:0/2257826906 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:14.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.017+0000 7f36aeffd640 1 --2- 172.21.15.169:0/2257826906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d64ce1080 0x557d64d00cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.021+0000 7f36affff640 1 -- 172.21.15.169:0/2257826906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce0690 msgr2=0x557d64d00790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:14.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.021+0000 7f36affff640 1 --2- 172.21.15.169:0/2257826906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce0690 0x557d64d00790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.021+0000 7f36affff640 1 -- 172.21.15.169:0/2257826906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d64ce1080 msgr2=0x557d64d00cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:14.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.021+0000 7f36affff640 1 --2- 172.21.15.169:0/2257826906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d64ce1080 0x557d64d00cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.021+0000 7f36affff640 1 -- 172.21.15.169:0/2257826906 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d64ce02d0 con 0x557d64ce61f0 2024-03-28T01:08:14.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.021+0000 7f36affff640 1 --2- 172.21.15.169:0/2257826906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557d64ce61f0 0x557d64d01ce0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f36a8099c80 tx=0x7f36a8097d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bd567ec5fdec020 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:14.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.021+0000 7f368effd640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:08:14.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.022+0000 7f368effd640 1 -- 172.21.15.169:0/2257826906 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a80a4170 con 0x557d64ce61f0 2024-03-28T01:08:14.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.022+0000 7f36b89dd740 1 -- 172.21.15.169:0/2257826906 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36a4005850 con 0x557d64ce61f0 2024-03-28T01:08:14.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.022+0000 7f368effd640 1 -- 172.21.15.169:0/2257826906 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f36a80a3ce0 con 0x557d64ce61f0 2024-03-28T01:08:14.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.023+0000 7f368effd640 1 -- 172.21.15.169:0/2257826906 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36a80a3ee0 con 0x557d64ce61f0 2024-03-28T01:08:14.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.028+0000 7f36b89dd740 10 client.27470 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:14.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.028+0000 7f36b89dd740 1 -- 172.21.15.169:0/2257826906 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557d64d63410 con 0x557d64ce61f0 2024-03-28T01:08:14.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.034+0000 7f368effd640 1 -- 172.21.15.169:0/2257826906 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36a4005850 con 0x557d64ce61f0 2024-03-28T01:08:14.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.034+0000 7f368effd640 1 -- 172.21.15.169:0/2257826906 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f36a809ae20 con 0x557d64ce61f0 2024-03-28T01:08:14.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.034+0000 7f36b89dd740 10 client.27470 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:14.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.034+0000 7f36b89dd740 10 client.27470 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:08:14.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.034+0000 7f36b89dd740 20 client.27470 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:14.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.034+0000 7f36b89dd740 1 --2- 172.21.15.169:0/2257826906 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x557d64d63c00 0x557d64d66020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.034+0000 7f36b89dd740 4 client.27470 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:08:14.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.034+0000 7f36b89dd740 1 -- 172.21.15.169:0/2257826906 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557d64cfb8f0 con 0x557d64d63c00 2024-03-28T01:08:14.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.040+0000 7f36af7fe640 1 --2- 172.21.15.169:0/2257826906 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x557d64d63c00 0x557d64d66020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.043+0000 7f36af7fe640 1 --2- 172.21.15.169:0/2257826906 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x557d64d63c00 0x557d64d66020 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c1fcc577ec209283 server_cookie=704dcd2cc67d1173 in_seq=0 out_seq=0 2024-03-28T01:08:14.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.043+0000 7f368effd640 10 client.27470 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:08:14.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.043+0000 7f368effd640 1 -- 172.21.15.169:0/2257826906 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557d64cfb8f0 con 0x557d64d63c00 2024-03-28T01:08:14.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.043+0000 7f368effd640 10 client.27470 handle_command_reply: tid=0 2024-03-28T01:08:14.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.043+0000 7f36b89dd740 1 client.27470 shutdown 2024-03-28T01:08:14.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.047+0000 7f36b89dd740 1 -- 172.21.15.169:0/2257826906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557d64ce61f0 msgr2=0x557d64d01ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:14.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.047+0000 7f36b89dd740 1 --2- 172.21.15.169:0/2257826906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557d64ce61f0 0x557d64d01ce0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f36a8099c80 tx=0x7f36a8097d30 comp rx=0 tx=0).stop 2024-03-28T01:08:14.054 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:13 smithi169 ceph-mon[32274]: pgmap v702: 129 pgs: 129 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 773 MiB/s wr, 1.64k op/s 2024-03-28T01:08:14.055 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:13 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2784206075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:08:14.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.049+0000 7f36b89dd740 1 -- 172.21.15.169:0/2257826906 shutdown_connections 2024-03-28T01:08:14.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.049+0000 7f36b89dd740 1 --2- 172.21.15.169:0/2257826906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557d64ce61f0 0x557d64d01ce0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.049+0000 7f36b89dd740 1 --2- 172.21.15.169:0/2257826906 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x557d64d63c00 0x557d64d66020 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.049+0000 7f36b89dd740 1 --2- 172.21.15.169:0/2257826906 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x557d64ce1080 0x557d64d00cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.049+0000 7f36b89dd740 1 --2- 172.21.15.169:0/2257826906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d64ce0690 0x557d64d00790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.049+0000 7f36b89dd740 1 -- 172.21.15.169:0/2257826906 >> 172.21.15.169:0/2257826906 conn(0x557d64cdbc00 msgr2=0x557d64cfa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:14.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.050+0000 7f368effd640 0 client.27470 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:08:14.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.052+0000 7f36b89dd740 1 -- 172.21.15.169:0/2257826906 shutdown_connections 2024-03-28T01:08:14.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.052+0000 7f36b89dd740 1 -- 172.21.15.169:0/2257826906 wait complete. 2024-03-28T01:08:14.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.054+0000 7f36b89dd740 20 client.27470 trim_cache size 0 max 16384 2024-03-28T01:08:14.059 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:08:14.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.066+0000 7f36b5d10640 1 -- 172.21.15.169:0/600812685 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36980a29d0 msgr2=0x7f36980a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:14.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.067+0000 7f36b5d10640 1 --2- 172.21.15.169:0/600812685 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36980a29d0 0x7f36980a4e90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f36a000a930 tx=0x7f36a0005c00 comp rx=0 tx=0).stop 2024-03-28T01:08:14.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.067+0000 7f36b5d10640 1 -- 172.21.15.169:0/600812685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36b010dd70 msgr2=0x7f36b01a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:14.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.067+0000 7f36b5d10640 1 --2- 172.21.15.169:0/600812685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36b010dd70 0x7f36b01a5480 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f36a800b9e0 tx=0x7f36a800bea0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.069+0000 7f36b5d10640 1 -- 172.21.15.169:0/600812685 shutdown_connections 2024-03-28T01:08:14.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.069+0000 7f36b5d10640 1 --2- 172.21.15.169:0/600812685 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f36980a29d0 0x7f36980a4e90 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.069+0000 7f36b5d10640 1 --2- 172.21.15.169:0/600812685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f36b010dd70 0x7f36b01a5480 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.069+0000 7f36b5d10640 1 --2- 172.21.15.169:0/600812685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36b010b2d0 0x7f36b019e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.069+0000 7f36b5d10640 1 --2- 172.21.15.169:0/600812685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36b0108910 0x7f36b019ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.069+0000 7f36b5d10640 1 -- 172.21.15.169:0/600812685 >> 172.21.15.169:0/600812685 conn(0x7f36b006e500 msgr2=0x7f36b010c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:14.075 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.072+0000 7f36b5d10640 1 -- 172.21.15.169:0/600812685 shutdown_connections 2024-03-28T01:08:14.075 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.073+0000 7f36b5d10640 1 -- 172.21.15.169:0/600812685 wait complete. 2024-03-28T01:08:14.106 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:08:14.106 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T01:08:14.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:13 smithi174 ceph-mon[36060]: pgmap v702: 129 pgs: 129 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 773 MiB/s wr, 1.64k op/s 2024-03-28T01:08:14.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:13 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2784206075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:08:14.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:13 smithi177 ceph-mon[36206]: pgmap v702: 129 pgs: 129 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 773 MiB/s wr, 1.64k op/s 2024-03-28T01:08:14.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:13 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2784206075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:08:14.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.462+0000 7f4e3ffff640 1 -- 172.21.15.169:0/3257830659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e40108d70 msgr2=0x7f4e401091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:14.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.462+0000 7f4e3ffff640 1 --2- 172.21.15.169:0/3257830659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e40108d70 0x7f4e401091f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4e3401a910 tx=0x7f4e3403cc90 comp rx=0 tx=0).stop 2024-03-28T01:08:14.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.463+0000 7f4e3ffff640 1 -- 172.21.15.169:0/3257830659 shutdown_connections 2024-03-28T01:08:14.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.463+0000 7f4e3ffff640 1 --2- 172.21.15.169:0/3257830659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e40109730 0x7f4e4010dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.463+0000 7f4e3ffff640 1 --2- 172.21.15.169:0/3257830659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e40108d70 0x7f4e401091f0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.463+0000 7f4e3ffff640 1 --2- 172.21.15.169:0/3257830659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e40072ab0 0x7f4e40072ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.463+0000 7f4e3ffff640 1 -- 172.21.15.169:0/3257830659 >> 172.21.15.169:0/3257830659 conn(0x7f4e4006e580 msgr2=0x7f4e400709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:14.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.466+0000 7f4e3ffff640 1 -- 172.21.15.169:0/3257830659 shutdown_connections 2024-03-28T01:08:14.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.466+0000 7f4e3ffff640 1 -- 172.21.15.169:0/3257830659 wait complete. 2024-03-28T01:08:14.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.467+0000 7f4e3ffff640 1 Processor -- start 2024-03-28T01:08:14.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.467+0000 7f4e3ffff640 1 -- start start 2024-03-28T01:08:14.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.467+0000 7f4e3ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e40072ab0 0x7f4e40119180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.467+0000 7f4e3ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e40108d70 0x7f4e401196c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.467+0000 7f4e3ffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e40109730 0x7f4e401c7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.467+0000 7f4e3ffff640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e40111880 con 0x7f4e40108d70 2024-03-28T01:08:14.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.467+0000 7f4e3ffff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e40111700 con 0x7f4e40072ab0 2024-03-28T01:08:14.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.467+0000 7f4e3ffff640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e40111a00 con 0x7f4e40109730 2024-03-28T01:08:14.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.468+0000 7f4e3f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e40109730 0x7f4e401c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.468+0000 7f4e3f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e40109730 0x7f4e401c7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:58352/0 (socket says 172.21.15.169:58352) 2024-03-28T01:08:14.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.468+0000 7f4e3f7fe640 1 -- 172.21.15.169:0/3566685317 learned_addr learned my addr 172.21.15.169:0/3566685317 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:14.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.468+0000 7f4e3effd640 1 --2- 172.21.15.169:0/3566685317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e40072ab0 0x7f4e40119180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.469+0000 7f4e3f7fe640 1 -- 172.21.15.169:0/3566685317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e40072ab0 msgr2=0x7f4e40119180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:14.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.469+0000 7f4e3f7fe640 1 --2- 172.21.15.169:0/3566685317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e40072ab0 0x7f4e40119180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.469+0000 7f4e3f7fe640 1 -- 172.21.15.169:0/3566685317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e40108d70 msgr2=0x7f4e401196c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:14.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.469+0000 7f4e3f7fe640 1 --2- 172.21.15.169:0/3566685317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e40108d70 0x7f4e401196c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.469+0000 7f4e3f7fe640 1 -- 172.21.15.169:0/3566685317 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e401c7ae0 con 0x7f4e40109730 2024-03-28T01:08:14.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.472+0000 7f4e3f7fe640 1 --2- 172.21.15.169:0/3566685317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e40109730 0x7f4e401c7470 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4e3800b970 tx=0x7f4e3800be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:14.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.477+0000 7f4e27fff640 1 -- 172.21.15.169:0/3566685317 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e38019070 con 0x7f4e40109730 2024-03-28T01:08:14.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.477+0000 7f4e3ffff640 1 -- 172.21.15.169:0/3566685317 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e401c7dd0 con 0x7f4e40109730 2024-03-28T01:08:14.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.477+0000 7f4e3ffff640 1 -- 172.21.15.169:0/3566685317 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e401c8370 con 0x7f4e40109730 2024-03-28T01:08:14.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.477+0000 7f4e27fff640 1 -- 172.21.15.169:0/3566685317 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e3800cb40 con 0x7f4e40109730 2024-03-28T01:08:14.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.477+0000 7f4e27fff640 1 -- 172.21.15.169:0/3566685317 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e38007650 con 0x7f4e40109730 2024-03-28T01:08:14.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.484+0000 7f4e47d22740 1 Processor -- start 2024-03-28T01:08:14.514 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.494+0000 7f4e47d22740 1 -- start start 2024-03-28T01:08:14.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.494+0000 7f4e47d22740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f7ec86860 0x558f7ec86c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.494+0000 7f4e47d22740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec87250 0x558f7ec8bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.494+0000 7f4e47d22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec8c3c0 0x558f7ec8e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.494+0000 7f4e47d22740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f7ec49610 con 0x558f7ec87250 2024-03-28T01:08:14.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.494+0000 7f4e47d22740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f7ebec3c0 con 0x558f7ec86860 2024-03-28T01:08:14.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.494+0000 7f4e47d22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f7eb94d60 con 0x558f7ec8c3c0 2024-03-28T01:08:14.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.494+0000 7f4e27fff640 1 -- 172.21.15.169:0/3566685317 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4e3801c070 con 0x7f4e40109730 2024-03-28T01:08:14.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.494+0000 7f4e3e7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f7ec86860 0x558f7ec86c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.495+0000 7f4e3effd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec87250 0x558f7ec8bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.495+0000 7f4e3effd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec87250 0x558f7ec8bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51894/0 (socket says 172.21.15.169:51894) 2024-03-28T01:08:14.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.495+0000 7f4e3effd640 1 -- 172.21.15.169:0/2452022376 learned_addr learned my addr 172.21.15.169:0/2452022376 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:14.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.495+0000 7f4e3e7fc640 1 --2- 172.21.15.169:0/2452022376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec8c3c0 0x558f7ec8e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.495+0000 7f4e27fff640 1 --2- 172.21.15.169:0/3566685317 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4e280a2a00 0x7f4e280a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.496+0000 7f4e27fff640 1 -- 172.21.15.169:0/3566685317 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f4e380ce210 con 0x7f4e40109730 2024-03-28T01:08:14.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.496+0000 7f4e3effd640 1 -- 172.21.15.169:0/2452022376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec8c3c0 msgr2=0x558f7ec8e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:14.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.496+0000 7f4e3effd640 1 --2- 172.21.15.169:0/2452022376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec8c3c0 0x558f7ec8e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.496+0000 7f4e3effd640 1 -- 172.21.15.169:0/2452022376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f7ec86860 msgr2=0x558f7ec86c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:14.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.496+0000 7f4e3effd640 1 --2- 172.21.15.169:0/2452022376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f7ec86860 0x558f7ec86c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.496+0000 7f4e3effd640 1 -- 172.21.15.169:0/2452022376 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f7e9003f0 con 0x558f7ec87250 2024-03-28T01:08:14.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.497+0000 7f4e3effd640 1 --2- 172.21.15.169:0/2452022376 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec87250 0x558f7ec8bd50 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f4e3000bc70 tx=0x7f4e300107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3ef3abece392be9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:14.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.497+0000 7f4e3f7fe640 1 --2- 172.21.15.169:0/3566685317 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4e280a2a00 0x7f4e280a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.498+0000 7f4e257fa640 1 -- 172.21.15.169:0/2452022376 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e30004020 con 0x558f7ec87250 2024-03-28T01:08:14.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.498+0000 7f4e257fa640 1 -- 172.21.15.169:0/2452022376 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e300041c0 con 0x558f7ec87250 2024-03-28T01:08:14.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.499+0000 7f4e47d22740 1 -- 172.21.15.169:0/2452022376 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec87250 msgr2=0x558f7ec8bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:14.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.499+0000 7f4e47d22740 1 --2- 172.21.15.169:0/2452022376 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec87250 0x558f7ec8bd50 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f4e3000bc70 tx=0x7f4e300107a0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.501+0000 7f4e3f7fe640 1 --2- 172.21.15.169:0/3566685317 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4e280a2a00 0x7f4e280a4ec0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4e38097230 tx=0x7f4e3800e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:14.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.501+0000 7f4e47d22740 1 -- 172.21.15.169:0/2452022376 shutdown_connections 2024-03-28T01:08:14.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.502+0000 7f4e47d22740 1 --2- 172.21.15.169:0/2452022376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec8c3c0 0x558f7ec8e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.502+0000 7f4e47d22740 1 --2- 172.21.15.169:0/2452022376 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec87250 0x558f7ec8bd50 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.502+0000 7f4e47d22740 1 --2- 172.21.15.169:0/2452022376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f7ec86860 0x558f7ec86c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.502+0000 7f4e47d22740 1 -- 172.21.15.169:0/2452022376 >> 172.21.15.169:0/2452022376 conn(0x558f7ec81dd0 msgr2=0x558f7ec821d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:14.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.504+0000 7f4e47d22740 1 -- 172.21.15.169:0/2452022376 shutdown_connections 2024-03-28T01:08:14.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.504+0000 7f4e47d22740 1 -- 172.21.15.169:0/2452022376 wait complete. 2024-03-28T01:08:14.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.505+0000 7f4e47d22740 1 -- start start 2024-03-28T01:08:14.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.505+0000 7f4e47d22740 1 Processor -- start 2024-03-28T01:08:14.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.505+0000 7f4e47d22740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f7ec86860 0x558f7ecd4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.505+0000 7f4e47d22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec87250 0x558f7ecd5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.506+0000 7f4e47d22740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec8c3c0 0x558f7ecdc490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.506+0000 7f4e47d22740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f7eba8240 con 0x558f7ec8c3c0 2024-03-28T01:08:14.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.506+0000 7f4e47d22740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f7ebe8460 con 0x558f7ec86860 2024-03-28T01:08:14.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.506+0000 7f4e47d22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f7ec8f3c0 con 0x558f7ec87250 2024-03-28T01:08:14.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.506+0000 7f4e47d22740 10 client.0 _ll_register_callbacks cb 0x558f7eba7c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:14.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.506+0000 7f4e3e7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f7ec86860 0x558f7ecd4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.506+0000 7f4e3e7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f7ec86860 0x558f7ecd4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:47792/0 (socket says 172.21.15.169:47792) 2024-03-28T01:08:14.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.506+0000 7f4e3e7fc640 1 -- 172.21.15.169:0/3728120849 learned_addr learned my addr 172.21.15.169:0/3728120849 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:14.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.506+0000 7f4e3e7fc640 1 --2- 172.21.15.169:0/3728120849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec8c3c0 0x558f7ecdc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.506+0000 7f4e3effd640 1 --2- 172.21.15.169:0/3728120849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec87250 0x558f7ecd5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.507+0000 7f4e3effd640 1 -- 172.21.15.169:0/3728120849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f7ec86860 msgr2=0x558f7ecd4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:14.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.507+0000 7f4e3effd640 1 --2- 172.21.15.169:0/3728120849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f7ec86860 0x558f7ecd4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.507+0000 7f4e3effd640 1 -- 172.21.15.169:0/3728120849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec8c3c0 msgr2=0x558f7ecdc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:14.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.507+0000 7f4e3effd640 1 --2- 172.21.15.169:0/3728120849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec8c3c0 0x558f7ecdc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.507+0000 7f4e3effd640 1 -- 172.21.15.169:0/3728120849 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f7e9ca7d0 con 0x558f7ec87250 2024-03-28T01:08:14.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.507+0000 7f4e3effd640 1 --2- 172.21.15.169:0/3728120849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec87250 0x558f7ecd5450 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x558f7ec87650 tx=0x7f4e30005940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1563a5382c5a036d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:14.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.517+0000 7f4e22ff5640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:08:14.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.517+0000 7f4e22ff5640 1 -- 172.21.15.169:0/3728120849 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e300119f0 con 0x558f7ec87250 2024-03-28T01:08:14.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.517+0000 7f4e47d22740 1 -- 172.21.15.169:0/3728120849 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558f7ebf9900 con 0x558f7ec87250 2024-03-28T01:08:14.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.517+0000 7f4e22ff5640 1 -- 172.21.15.169:0/3728120849 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f4e30011b90 con 0x558f7ec87250 2024-03-28T01:08:14.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.519+0000 7f4e22ff5640 1 -- 172.21.15.169:0/3728120849 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e30011d90 con 0x558f7ec87250 2024-03-28T01:08:14.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.520+0000 7f4e47d22740 10 client.27439 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:14.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.520+0000 7f4e47d22740 1 -- 172.21.15.169:0/3728120849 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558f7ecdf890 con 0x558f7ec87250 2024-03-28T01:08:14.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.522+0000 7f4e22ff5640 1 -- 172.21.15.169:0/3728120849 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558f7ebf9900 con 0x558f7ec87250 2024-03-28T01:08:14.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.522+0000 7f4e22ff5640 1 -- 172.21.15.169:0/3728120849 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f4e3002ec60 con 0x558f7ec87250 2024-03-28T01:08:14.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.526+0000 7f4e47d22740 10 client.27439 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:14.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.526+0000 7f4e47d22740 10 client.27439 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:08:14.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.526+0000 7f4e47d22740 20 client.27439 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:14.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.526+0000 7f4e47d22740 1 --2- 172.21.15.169:0/3728120849 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x558f7ece0cb0 0x558f7ece30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.526+0000 7f4e47d22740 4 client.27439 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:08:14.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.526+0000 7f4e47d22740 1 -- 172.21.15.169:0/3728120849 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558f7ebf3c50 con 0x558f7ece0cb0 2024-03-28T01:08:14.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.534+0000 7f4e3e7fc640 1 --2- 172.21.15.169:0/3728120849 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x558f7ece0cb0 0x558f7ece30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.557+0000 7f4e22ff5640 10 client.27439 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:08:14.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.557+0000 7f4e3e7fc640 1 --2- 172.21.15.169:0/3728120849 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x558f7ece0cb0 0x558f7ece30a0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ed49a5854700acfa server_cookie=a90dbf5666e54d1a in_seq=0 out_seq=0 2024-03-28T01:08:14.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.560+0000 7f4e22ff5640 1 -- 172.21.15.169:0/3728120849 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558f7ebf3c50 con 0x558f7ece0cb0 2024-03-28T01:08:14.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.560+0000 7f4e22ff5640 10 client.27439 handle_command_reply: tid=0 2024-03-28T01:08:14.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.560+0000 7f4e47d22740 1 client.27439 shutdown 2024-03-28T01:08:14.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.562+0000 7f4e47d22740 1 -- 172.21.15.169:0/3728120849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec87250 msgr2=0x558f7ecd5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:14.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.562+0000 7f4e47d22740 1 --2- 172.21.15.169:0/3728120849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec87250 0x558f7ecd5450 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x558f7ec87650 tx=0x7f4e30005940 comp rx=0 tx=0).stop 2024-03-28T01:08:14.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.563+0000 7f4e47d22740 1 -- 172.21.15.169:0/3728120849 shutdown_connections 2024-03-28T01:08:14.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.563+0000 7f4e47d22740 1 --2- 172.21.15.169:0/3728120849 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x558f7ece0cb0 0x558f7ece30a0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.563+0000 7f4e47d22740 1 --2- 172.21.15.169:0/3728120849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec8c3c0 0x558f7ecdc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.563+0000 7f4e47d22740 1 --2- 172.21.15.169:0/3728120849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec87250 0x558f7ecd5450 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.563+0000 7f4e47d22740 1 --2- 172.21.15.169:0/3728120849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f7ec86860 0x558f7ecd4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.563+0000 7f4e22ff5640 0 client.27439 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:08:14.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.563+0000 7f4e47d22740 1 -- 172.21.15.169:0/3728120849 >> 172.21.15.169:0/3728120849 conn(0x558f7ec81dd0 msgr2=0x558f7ec81070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:14.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.565+0000 7f4e47d22740 1 -- 172.21.15.169:0/3728120849 shutdown_connections 2024-03-28T01:08:14.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.566+0000 7f4e47d22740 1 -- 172.21.15.169:0/3728120849 wait complete. 2024-03-28T01:08:14.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.566+0000 7f4e47d22740 20 client.27439 trim_cache size 0 max 16384 2024-03-28T01:08:14.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.596+0000 7f4e47d22740 1 Processor -- start 2024-03-28T01:08:14.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.596+0000 7f4e47d22740 1 -- start start 2024-03-28T01:08:14.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.596+0000 7f4e47d22740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f7ec86860 0x558f7ec87d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.596+0000 7f4e47d22740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec87250 0x558f7ecdc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.596+0000 7f4e47d22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec8c3c0 0x558f7ec88930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.596+0000 7f4e47d22740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c012120 con 0x558f7ec87250 2024-03-28T01:08:14.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.596+0000 7f4e47d22740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c00fc20 con 0x558f7ec86860 2024-03-28T01:08:14.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.596+0000 7f4e47d22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c00dcc0 con 0x558f7ec8c3c0 2024-03-28T01:08:14.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.597+0000 7f4e3effd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec87250 0x558f7ecdc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.597+0000 7f4e3effd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec87250 0x558f7ecdc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51910/0 (socket says 172.21.15.169:51910) 2024-03-28T01:08:14.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.597+0000 7f4e3effd640 1 -- 172.21.15.169:0/2150551473 learned_addr learned my addr 172.21.15.169:0/2150551473 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:14.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.603+0000 7f4e3e7fc640 1 --2- 172.21.15.169:0/2150551473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f7ec86860 0x558f7ec87d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.603+0000 7f4e3e7fc640 1 --2- 172.21.15.169:0/2150551473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec8c3c0 0x558f7ec88930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.604+0000 7f4e3e7fc640 1 -- 172.21.15.169:0/2150551473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f7ec86860 msgr2=0x558f7ec87d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:14.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.604+0000 7f4e3e7fc640 1 --2- 172.21.15.169:0/2150551473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f7ec86860 0x558f7ec87d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.604+0000 7f4e3e7fc640 1 -- 172.21.15.169:0/2150551473 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec87250 msgr2=0x558f7ecdc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:14.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.604+0000 7f4e3e7fc640 1 --2- 172.21.15.169:0/2150551473 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec87250 0x558f7ecdc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.604+0000 7f4e3e7fc640 1 -- 172.21.15.169:0/2150551473 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f7ecd5db0 con 0x558f7ec8c3c0 2024-03-28T01:08:14.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.606+0000 7f4e3e7fc640 1 --2- 172.21.15.169:0/2150551473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec8c3c0 0x558f7ec88930 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x558f7ece38a0 tx=0x7f4e34005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3684bb2bae6468e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:14.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.612+0000 7f4e24ff9640 1 -- 172.21.15.169:0/2150551473 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e34004030 con 0x558f7ec8c3c0 2024-03-28T01:08:14.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.612+0000 7f4e24ff9640 1 -- 172.21.15.169:0/2150551473 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e34052020 con 0x558f7ec8c3c0 2024-03-28T01:08:14.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.612+0000 7f4e24ff9640 1 -- 172.21.15.169:0/2150551473 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e340035f0 con 0x558f7ec8c3c0 2024-03-28T01:08:14.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.614+0000 7f4e47d22740 1 -- 172.21.15.169:0/2150551473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec8c3c0 msgr2=0x558f7ec88930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:14.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.614+0000 7f4e47d22740 1 --2- 172.21.15.169:0/2150551473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec8c3c0 0x558f7ec88930 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x558f7ece38a0 tx=0x7f4e34005fd0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.614+0000 7f4e47d22740 1 -- 172.21.15.169:0/2150551473 shutdown_connections 2024-03-28T01:08:14.617 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.614+0000 7f4e47d22740 1 --2- 172.21.15.169:0/2150551473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec8c3c0 0x558f7ec88930 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.614+0000 7f4e47d22740 1 --2- 172.21.15.169:0/2150551473 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec87250 0x558f7ecdc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.614+0000 7f4e47d22740 1 --2- 172.21.15.169:0/2150551473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f7ec86860 0x558f7ec87d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.614+0000 7f4e47d22740 1 -- 172.21.15.169:0/2150551473 >> 172.21.15.169:0/2150551473 conn(0x558f7ec81dd0 msgr2=0x558f7ecd4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:14.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.616+0000 7f4e47d22740 1 -- 172.21.15.169:0/2150551473 shutdown_connections 2024-03-28T01:08:14.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.616+0000 7f4e47d22740 1 -- 172.21.15.169:0/2150551473 wait complete. 2024-03-28T01:08:14.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.616+0000 7f4e47d22740 1 -- start start 2024-03-28T01:08:14.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.616+0000 7f4e47d22740 1 Processor -- start 2024-03-28T01:08:14.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.617+0000 7f4e47d22740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec86860 0x558f7ecd98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.617+0000 7f4e47d22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec87250 0x558f7ecd9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.617+0000 7f4e47d22740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f7ec8c3c0 0x558f7ecdae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.617+0000 7f4e47d22740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c00e2c0 con 0x558f7ec86860 2024-03-28T01:08:14.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.617+0000 7f4e47d22740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c00f5c0 con 0x558f7ec8c3c0 2024-03-28T01:08:14.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.617+0000 7f4e47d22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c00e900 con 0x558f7ec87250 2024-03-28T01:08:14.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.617+0000 7f4e47d22740 10 client.0 _ll_register_callbacks cb 0x7f4e30005850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:14.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.618+0000 7f4e3effd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec87250 0x558f7ecd9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.618+0000 7f4e3effd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec87250 0x558f7ecd9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:58378/0 (socket says 172.21.15.169:58378) 2024-03-28T01:08:14.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.618+0000 7f4e3effd640 1 -- 172.21.15.169:0/2356036695 learned_addr learned my addr 172.21.15.169:0/2356036695 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:14.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.618+0000 7f4e3effd640 1 -- 172.21.15.169:0/2356036695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f7ec8c3c0 msgr2=0x558f7ecdae20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:08:14.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.618+0000 7f4e3effd640 1 --2- 172.21.15.169:0/2356036695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f7ec8c3c0 0x558f7ecdae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.618+0000 7f4e3effd640 1 -- 172.21.15.169:0/2356036695 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec86860 msgr2=0x558f7ecd98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:14.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.618+0000 7f4e3effd640 1 --2- 172.21.15.169:0/2356036695 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec86860 0x558f7ecd98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.619+0000 7f4e3effd640 1 -- 172.21.15.169:0/2356036695 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f7ecdb410 con 0x558f7ec87250 2024-03-28T01:08:14.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.619+0000 7f4e3effd640 1 --2- 172.21.15.169:0/2356036695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec87250 0x558f7ecd9e10 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4e3000c010 tx=0x7f4e30005c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c15dddb1ed9740a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:14.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.620+0000 7f4e22ff5640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:08:14.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.620+0000 7f4e22ff5640 1 -- 172.21.15.169:0/2356036695 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e30033030 con 0x558f7ec87250 2024-03-28T01:08:14.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.620+0000 7f4e22ff5640 1 -- 172.21.15.169:0/2356036695 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f4e30021070 con 0x558f7ec87250 2024-03-28T01:08:14.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.620+0000 7f4e47d22740 1 -- 172.21.15.169:0/2356036695 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4e34052020 con 0x558f7ec87250 2024-03-28T01:08:14.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.621+0000 7f4e22ff5640 1 -- 172.21.15.169:0/2356036695 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e3001b810 con 0x558f7ec87250 2024-03-28T01:08:14.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.621+0000 7f4e47d22740 10 client.27445 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:14.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.621+0000 7f4e47d22740 1 -- 172.21.15.169:0/2356036695 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558f7ecdc710 con 0x558f7ec87250 2024-03-28T01:08:14.629 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.626+0000 7f4e22ff5640 1 -- 172.21.15.169:0/2356036695 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e34052020 con 0x558f7ec87250 2024-03-28T01:08:14.630 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.627+0000 7f4e22ff5640 1 -- 172.21.15.169:0/2356036695 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f4e300037f0 con 0x558f7ec87250 2024-03-28T01:08:14.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.627+0000 7f4e47d22740 10 client.27445 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:14.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.627+0000 7f4e47d22740 10 client.27445 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:08:14.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.627+0000 7f4e47d22740 20 client.27445 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:14.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.628+0000 7f4e47d22740 1 --2- 172.21.15.169:0/2356036695 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x558f7ed3cbb0 0x558f7ed3efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:14.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.628+0000 7f4e47d22740 4 client.27445 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:08:14.631 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.628+0000 7f4e47d22740 1 -- 172.21.15.169:0/2356036695 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558f7eb71640 con 0x558f7ed3cbb0 2024-03-28T01:08:14.632 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.628+0000 7f4e3e7fc640 1 --2- 172.21.15.169:0/2356036695 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x558f7ed3cbb0 0x558f7ed3efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:14.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.635+0000 7f4e22ff5640 10 client.27445 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:08:14.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.635+0000 7f4e3e7fc640 1 --2- 172.21.15.169:0/2356036695 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x558f7ed3cbb0 0x558f7ed3efd0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6b15ca4c1ad3140e server_cookie=677623835c53ff3c in_seq=0 out_seq=0 2024-03-28T01:08:14.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.639+0000 7f4e22ff5640 1 -- 172.21.15.169:0/2356036695 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558f7eb71640 con 0x558f7ed3cbb0 2024-03-28T01:08:14.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.639+0000 7f4e22ff5640 10 client.27445 handle_command_reply: tid=0 2024-03-28T01:08:14.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.639+0000 7f4e47d22740 1 client.27445 shutdown 2024-03-28T01:08:14.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.643+0000 7f4e47d22740 1 -- 172.21.15.169:0/2356036695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec87250 msgr2=0x558f7ecd9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:14.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.643+0000 7f4e47d22740 1 --2- 172.21.15.169:0/2356036695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec87250 0x558f7ecd9e10 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4e3000c010 tx=0x7f4e30005c80 comp rx=0 tx=0).stop 2024-03-28T01:08:14.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.644+0000 7f4e47d22740 1 -- 172.21.15.169:0/2356036695 shutdown_connections 2024-03-28T01:08:14.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.644+0000 7f4e47d22740 1 --2- 172.21.15.169:0/2356036695 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x558f7ed3cbb0 0x558f7ed3efd0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.644+0000 7f4e47d22740 1 --2- 172.21.15.169:0/2356036695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f7ec8c3c0 0x558f7ecdae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.644+0000 7f4e47d22740 1 --2- 172.21.15.169:0/2356036695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f7ec87250 0x558f7ecd9e10 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.644+0000 7f4e47d22740 1 --2- 172.21.15.169:0/2356036695 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x558f7ec86860 0x558f7ecd98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.644+0000 7f4e47d22740 1 -- 172.21.15.169:0/2356036695 >> 172.21.15.169:0/2356036695 conn(0x558f7ec81dd0 msgr2=0x558f7ecd4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:14.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.644+0000 7f4e22ff5640 0 client.27445 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:08:14.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.644+0000 7f4e47d22740 1 -- 172.21.15.169:0/2356036695 shutdown_connections 2024-03-28T01:08:14.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.644+0000 7f4e47d22740 1 -- 172.21.15.169:0/2356036695 wait complete. 2024-03-28T01:08:14.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.644+0000 7f4e47d22740 20 client.27445 trim_cache size 0 max 16384 2024-03-28T01:08:14.654 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:08:14.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.654+0000 7f4e3ffff640 1 -- 172.21.15.169:0/3566685317 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4e280a2a00 msgr2=0x7f4e280a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:14.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.654+0000 7f4e3ffff640 1 --2- 172.21.15.169:0/3566685317 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4e280a2a00 0x7f4e280a4ec0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4e38097230 tx=0x7f4e3800e040 comp rx=0 tx=0).stop 2024-03-28T01:08:14.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.654+0000 7f4e3ffff640 1 -- 172.21.15.169:0/3566685317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e40109730 msgr2=0x7f4e401c7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:14.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.654+0000 7f4e3ffff640 1 --2- 172.21.15.169:0/3566685317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e40109730 0x7f4e401c7470 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4e3800b970 tx=0x7f4e3800be30 comp rx=0 tx=0).stop 2024-03-28T01:08:14.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.658+0000 7f4e3ffff640 1 -- 172.21.15.169:0/3566685317 shutdown_connections 2024-03-28T01:08:14.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.658+0000 7f4e3ffff640 1 --2- 172.21.15.169:0/3566685317 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f4e280a2a00 0x7f4e280a4ec0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.658+0000 7f4e3ffff640 1 --2- 172.21.15.169:0/3566685317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e40109730 0x7f4e401c7470 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.658+0000 7f4e3ffff640 1 --2- 172.21.15.169:0/3566685317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4e40108d70 0x7f4e401196c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.658+0000 7f4e3ffff640 1 --2- 172.21.15.169:0/3566685317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e40072ab0 0x7f4e40119180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:14.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.658+0000 7f4e3ffff640 1 -- 172.21.15.169:0/3566685317 >> 172.21.15.169:0/3566685317 conn(0x7f4e4006e580 msgr2=0x7f4e4010c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:14.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.664+0000 7f4e3ffff640 1 -- 172.21.15.169:0/3566685317 shutdown_connections 2024-03-28T01:08:14.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:14.665+0000 7f4e3ffff640 1 -- 172.21.15.169:0/3566685317 wait complete. 2024-03-28T01:08:14.727 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:08:14.727 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T01:08:15.183 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.179+0000 7ff8894ea640 1 -- 172.21.15.169:0/859794114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff884108ce0 msgr2=0x7ff884109160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:15.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.179+0000 7ff8894ea640 1 --2- 172.21.15.169:0/859794114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff884108ce0 0x7ff884109160 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff87c009be0 tx=0x7ff87c02bef0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.180+0000 7ff8894ea640 1 -- 172.21.15.169:0/859794114 shutdown_connections 2024-03-28T01:08:15.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.180+0000 7ff8894ea640 1 --2- 172.21.15.169:0/859794114 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8841096a0 0x7ff88410df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.180+0000 7ff8894ea640 1 --2- 172.21.15.169:0/859794114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff884108ce0 0x7ff884109160 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.180+0000 7ff8894ea640 1 --2- 172.21.15.169:0/859794114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff884072a10 0x7ff884072e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.180+0000 7ff8894ea640 1 -- 172.21.15.169:0/859794114 >> 172.21.15.169:0/859794114 conn(0x7ff88406e4c0 msgr2=0x7ff884070900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:15.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.180+0000 7ff8894ea640 1 -- 172.21.15.169:0/859794114 shutdown_connections 2024-03-28T01:08:15.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.180+0000 7ff8894ea640 1 -- 172.21.15.169:0/859794114 wait complete. 2024-03-28T01:08:15.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.182+0000 7ff8894ea640 1 Processor -- start 2024-03-28T01:08:15.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.182+0000 7ff8894ea640 1 -- start start 2024-03-28T01:08:15.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.182+0000 7ff8894ea640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff884072a10 0x7ff8841a6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:15.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.183+0000 7ff8894ea640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff884108ce0 0x7ff8841a6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:15.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.183+0000 7ff8894ea640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8841096a0 0x7ff8841adbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:15.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.183+0000 7ff8894ea640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88411bff0 con 0x7ff8841096a0 2024-03-28T01:08:15.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.183+0000 7ff8894ea640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88411be70 con 0x7ff884072a10 2024-03-28T01:08:15.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.183+0000 7ff8894ea640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88411c170 con 0x7ff884108ce0 2024-03-28T01:08:15.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.183+0000 7ff8837fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8841096a0 0x7ff8841adbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:15.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.183+0000 7ff8837fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8841096a0 0x7ff8841adbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51934/0 (socket says 172.21.15.169:51934) 2024-03-28T01:08:15.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.183+0000 7ff8837fe640 1 -- 172.21.15.169:0/3666815317 learned_addr learned my addr 172.21.15.169:0/3666815317 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:15.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.183+0000 7ff882ffd640 1 --2- 172.21.15.169:0/3666815317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff884072a10 0x7ff8841a6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:15.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.183+0000 7ff8837fe640 1 -- 172.21.15.169:0/3666815317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff884108ce0 msgr2=0x7ff8841a6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:15.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.183+0000 7ff8837fe640 1 --2- 172.21.15.169:0/3666815317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff884108ce0 0x7ff8841a6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.183+0000 7ff8837fe640 1 -- 172.21.15.169:0/3666815317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff884072a10 msgr2=0x7ff8841a6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:15.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.183+0000 7ff8837fe640 1 --2- 172.21.15.169:0/3666815317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff884072a10 0x7ff8841a6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.183+0000 7ff8837fe640 1 -- 172.21.15.169:0/3666815317 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8841ae220 con 0x7ff8841096a0 2024-03-28T01:08:15.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.184+0000 7ff8837fe640 1 --2- 172.21.15.169:0/3666815317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8841096a0 0x7ff8841adbb0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff87400e970 tx=0x7ff87400ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:15.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.184+0000 7ff86bfff640 1 -- 172.21.15.169:0/3666815317 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87400ccc0 con 0x7ff8841096a0 2024-03-28T01:08:15.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.184+0000 7ff86bfff640 1 -- 172.21.15.169:0/3666815317 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff874005500 con 0x7ff8841096a0 2024-03-28T01:08:15.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.184+0000 7ff86bfff640 1 -- 172.21.15.169:0/3666815317 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff874010430 con 0x7ff8841096a0 2024-03-28T01:08:15.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.186+0000 7ff8894ea640 1 -- 172.21.15.169:0/3666815317 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8841ae510 con 0x7ff8841096a0 2024-03-28T01:08:15.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.186+0000 7ff8894ea640 1 -- 172.21.15.169:0/3666815317 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8841aea10 con 0x7ff8841096a0 2024-03-28T01:08:15.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.191+0000 7ff86bfff640 1 -- 172.21.15.169:0/3666815317 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff874005000 con 0x7ff8841096a0 2024-03-28T01:08:15.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.192+0000 7ff86bfff640 1 --2- 172.21.15.169:0/3666815317 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff86c0a28d0 0x7ff86c0a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:15.196 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T01:08:15.196 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T01:08:15.196 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.192+0000 7ff86bfff640 1 -- 172.21.15.169:0/3666815317 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ff8740d2170 con 0x7ff8841096a0 2024-03-28T01:08:15.196 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.193+0000 7ff88c1b5740 1 Processor -- start 2024-03-28T01:08:15.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.194+0000 7ff88c1b5740 1 -- start start 2024-03-28T01:08:15.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.194+0000 7ff88c1b5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56548fc6d860 0x56548fc6dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:15.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.194+0000 7ff88c1b5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc6e250 0x56548fc72d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:15.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.194+0000 7ff88c1b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc733c0 0x56548fc75850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:15.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.194+0000 7ff88c1b5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56548fc30610 con 0x56548fc6e250 2024-03-28T01:08:15.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.194+0000 7ff88c1b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x56548fbd33c0 con 0x56548fc733c0 2024-03-28T01:08:15.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.194+0000 7ff88c1b5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56548fb7bd60 con 0x56548fc6d860 2024-03-28T01:08:15.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.195+0000 7ff882ffd640 1 --2- 172.21.15.169:0/3666815317 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff86c0a28d0 0x7ff86c0a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:15.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.195+0000 7ff8827fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc6e250 0x56548fc72d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:15.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.195+0000 7ff8827fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc6e250 0x56548fc72d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51940/0 (socket says 172.21.15.169:51940) 2024-03-28T01:08:15.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.195+0000 7ff8827fc640 1 -- 172.21.15.169:0/3476783753 learned_addr learned my addr 172.21.15.169:0/3476783753 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:15.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.195+0000 7ff8837fe640 1 --2- 172.21.15.169:0/3476783753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc733c0 0x56548fc75850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:15.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.195+0000 7ff8827fc640 1 -- 172.21.15.169:0/3476783753 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56548fc6d860 msgr2=0x56548fc6dc80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:08:15.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.195+0000 7ff8827fc640 1 --2- 172.21.15.169:0/3476783753 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56548fc6d860 0x56548fc6dc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.195+0000 7ff8827fc640 1 -- 172.21.15.169:0/3476783753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc733c0 msgr2=0x56548fc75850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:15.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.195+0000 7ff8827fc640 1 --2- 172.21.15.169:0/3476783753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc733c0 0x56548fc75850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.195+0000 7ff8827fc640 1 -- 172.21.15.169:0/3476783753 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56548f8e73f0 con 0x56548fc6e250 2024-03-28T01:08:15.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.196+0000 7ff8827fc640 1 --2- 172.21.15.169:0/3476783753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc6e250 0x56548fc72d50 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff87c009b30 tx=0x7ff87c034f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62cb2bd00627c7b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:15.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.196+0000 7ff882ffd640 1 --2- 172.21.15.169:0/3666815317 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff86c0a28d0 0x7ff86c0a4d90 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff878009c20 tx=0x7ff878005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:15.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.200+0000 7ff8697fa640 1 -- 172.21.15.169:0/3476783753 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c042020 con 0x56548fc6e250 2024-03-28T01:08:15.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.200+0000 7ff8697fa640 1 -- 172.21.15.169:0/3476783753 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff87c003e50 con 0x56548fc6e250 2024-03-28T01:08:15.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.200+0000 7ff8697fa640 1 -- 172.21.15.169:0/3476783753 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c0043b0 con 0x56548fc6e250 2024-03-28T01:08:15.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.202+0000 7ff88c1b5740 1 -- 172.21.15.169:0/3476783753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc6e250 msgr2=0x56548fc72d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:15.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.202+0000 7ff88c1b5740 1 --2- 172.21.15.169:0/3476783753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc6e250 0x56548fc72d50 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff87c009b30 tx=0x7ff87c034f90 comp rx=0 tx=0).stop 2024-03-28T01:08:15.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.202+0000 7ff88c1b5740 1 -- 172.21.15.169:0/3476783753 shutdown_connections 2024-03-28T01:08:15.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.203+0000 7ff88c1b5740 1 --2- 172.21.15.169:0/3476783753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc733c0 0x56548fc75850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.203+0000 7ff88c1b5740 1 --2- 172.21.15.169:0/3476783753 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc6e250 0x56548fc72d50 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.203+0000 7ff88c1b5740 1 --2- 172.21.15.169:0/3476783753 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56548fc6d860 0x56548fc6dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.203+0000 7ff88c1b5740 1 -- 172.21.15.169:0/3476783753 >> 172.21.15.169:0/3476783753 conn(0x56548fc68dd0 msgr2=0x56548fc691d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:15.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.205+0000 7ff88c1b5740 1 -- 172.21.15.169:0/3476783753 shutdown_connections 2024-03-28T01:08:15.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.205+0000 7ff88c1b5740 1 -- 172.21.15.169:0/3476783753 wait complete. 2024-03-28T01:08:15.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.206+0000 7ff88c1b5740 1 -- start start 2024-03-28T01:08:15.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.206+0000 7ff88c1b5740 1 Processor -- start 2024-03-28T01:08:15.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.206+0000 7ff88c1b5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc6d860 0x56548fc77a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:15.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.206+0000 7ff88c1b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc6e250 0x56548fc77fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:15.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.206+0000 7ff88c1b5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56548fc733c0 0x56548fc7eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:15.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.206+0000 7ff88c1b5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x56548fb8f240 con 0x56548fc6d860 2024-03-28T01:08:15.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.206+0000 7ff88c1b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x56548fbcf460 con 0x56548fc6e250 2024-03-28T01:08:15.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.206+0000 7ff88c1b5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56548fc76310 con 0x56548fc733c0 2024-03-28T01:08:15.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.206+0000 7ff88c1b5740 10 client.0 _ll_register_callbacks cb 0x56548fb8ec10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:15.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.207+0000 7ff8837fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56548fc733c0 0x56548fc7eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:15.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.207+0000 7ff8837fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56548fc733c0 0x56548fc7eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:58422/0 (socket says 172.21.15.169:58422) 2024-03-28T01:08:15.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.207+0000 7ff8837fe640 1 -- 172.21.15.169:0/1334029424 learned_addr learned my addr 172.21.15.169:0/1334029424 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:15.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.208+0000 7ff8827fc640 1 --2- 172.21.15.169:0/1334029424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc6e250 0x56548fc77fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:15.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.208+0000 7ff882ffd640 1 --2- 172.21.15.169:0/1334029424 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc6d860 0x56548fc77a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:15.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.208+0000 7ff8837fe640 1 -- 172.21.15.169:0/1334029424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc6e250 msgr2=0x56548fc77fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:15.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.208+0000 7ff8837fe640 1 --2- 172.21.15.169:0/1334029424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc6e250 0x56548fc77fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.208+0000 7ff8837fe640 1 -- 172.21.15.169:0/1334029424 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc6d860 msgr2=0x56548fc77a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:15.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.208+0000 7ff8837fe640 1 --2- 172.21.15.169:0/1334029424 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc6d860 0x56548fc77a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.208+0000 7ff8837fe640 1 -- 172.21.15.169:0/1334029424 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56548f9b17d0 con 0x56548fc733c0 2024-03-28T01:08:15.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.209+0000 7ff8837fe640 1 --2- 172.21.15.169:0/1334029424 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56548fc733c0 0x56548fc7eff0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff87409afc0 tx=0x7ff87409ac30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad5f349ad57344d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:15.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.209+0000 7ff866ff5640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:08:15.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.210+0000 7ff866ff5640 1 -- 172.21.15.169:0/1334029424 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8740d3060 con 0x56548fc733c0 2024-03-28T01:08:15.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.210+0000 7ff866ff5640 1 -- 172.21.15.169:0/1334029424 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ff874095ba0 con 0x56548fc733c0 2024-03-28T01:08:15.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.210+0000 7ff88c1b5740 1 -- 172.21.15.169:0/1334029424 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff87c003e50 con 0x56548fc733c0 2024-03-28T01:08:15.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.212+0000 7ff866ff5640 1 -- 172.21.15.169:0/1334029424 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8740046c0 con 0x56548fc733c0 2024-03-28T01:08:15.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.216+0000 7ff88c1b5740 10 client.27451 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:15.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.216+0000 7ff88c1b5740 1 -- 172.21.15.169:0/1334029424 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56548fc823d0 con 0x56548fc733c0 2024-03-28T01:08:15.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.217+0000 7ff866ff5640 1 -- 172.21.15.169:0/1334029424 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff87c003e50 con 0x56548fc733c0 2024-03-28T01:08:15.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.217+0000 7ff866ff5640 1 -- 172.21.15.169:0/1334029424 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ff8740b9b30 con 0x56548fc733c0 2024-03-28T01:08:15.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.218+0000 7ff88c1b5740 10 client.27451 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:15.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.218+0000 7ff88c1b5740 10 client.27451 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:08:15.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.218+0000 7ff88c1b5740 20 client.27451 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:15.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.218+0000 7ff88c1b5740 1 --2- 172.21.15.169:0/1334029424 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56548fc83ad0 0x56548fc85ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:15.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.218+0000 7ff88c1b5740 4 client.27451 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:08:15.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.218+0000 7ff88c1b5740 1 -- 172.21.15.169:0/1334029424 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56548fbe0900 con 0x56548fc83ad0 2024-03-28T01:08:15.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.218+0000 7ff882ffd640 1 --2- 172.21.15.169:0/1334029424 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56548fc83ad0 0x56548fc85ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:15.223 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.220+0000 7ff882ffd640 1 --2- 172.21.15.169:0/1334029424 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56548fc83ad0 0x56548fc85ee0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a25172589924467 server_cookie=729e7d8ebc123627 in_seq=0 out_seq=0 2024-03-28T01:08:15.223 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.220+0000 7ff866ff5640 10 client.27451 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:15.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.224+0000 7ff866ff5640 1 -- 172.21.15.169:0/1334029424 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56548fbe0900 con 0x56548fc83ad0 2024-03-28T01:08:15.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.224+0000 7ff866ff5640 10 client.27451 handle_command_reply: tid=0 2024-03-28T01:08:15.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.227+0000 7ff88c1b5740 1 client.27451 shutdown 2024-03-28T01:08:15.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.228+0000 7ff88c1b5740 1 -- 172.21.15.169:0/1334029424 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56548fc733c0 msgr2=0x56548fc7eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:15.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.228+0000 7ff88c1b5740 1 --2- 172.21.15.169:0/1334029424 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56548fc733c0 0x56548fc7eff0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff87409afc0 tx=0x7ff87409ac30 comp rx=0 tx=0).stop 2024-03-28T01:08:15.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.228+0000 7ff88c1b5740 1 -- 172.21.15.169:0/1334029424 shutdown_connections 2024-03-28T01:08:15.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.228+0000 7ff88c1b5740 1 --2- 172.21.15.169:0/1334029424 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56548fc733c0 0x56548fc7eff0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.228+0000 7ff88c1b5740 1 --2- 172.21.15.169:0/1334029424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc6e250 0x56548fc77fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.228+0000 7ff88c1b5740 1 --2- 172.21.15.169:0/1334029424 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56548fc83ad0 0x56548fc85ee0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.228+0000 7ff88c1b5740 1 --2- 172.21.15.169:0/1334029424 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc6d860 0x56548fc77a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.228+0000 7ff88c1b5740 1 -- 172.21.15.169:0/1334029424 >> 172.21.15.169:0/1334029424 conn(0x56548fc68dd0 msgr2=0x56548fc68070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:15.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.228+0000 7ff866ff5640 0 client.27451 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:15.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.229+0000 7ff88c1b5740 1 -- 172.21.15.169:0/1334029424 shutdown_connections 2024-03-28T01:08:15.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.229+0000 7ff88c1b5740 1 -- 172.21.15.169:0/1334029424 wait complete. 2024-03-28T01:08:15.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.231+0000 7ff88c1b5740 20 client.27451 trim_cache size 0 max 16384 2024-03-28T01:08:15.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.260+0000 7ff88c1b5740 1 Processor -- start 2024-03-28T01:08:15.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.260+0000 7ff88c1b5740 1 -- start start 2024-03-28T01:08:15.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.261+0000 7ff88c1b5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56548fc6d860 0x56548fc77090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:15.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.261+0000 7ff88c1b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc6e250 0x56548fc7efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:15.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.261+0000 7ff88c1b5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc733c0 0x56548fc677e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:15.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.261+0000 7ff88c1b5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8841bb530 con 0x56548fc733c0 2024-03-28T01:08:15.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.261+0000 7ff88c1b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8841b9030 con 0x56548fc6e250 2024-03-28T01:08:15.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.261+0000 7ff88c1b5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8841b70d0 con 0x56548fc6d860 2024-03-28T01:08:15.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.261+0000 7ff8837fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc733c0 0x56548fc677e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:15.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.261+0000 7ff8827fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc6e250 0x56548fc7efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:15.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.261+0000 7ff8837fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc733c0 0x56548fc677e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51964/0 (socket says 172.21.15.169:51964) 2024-03-28T01:08:15.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.261+0000 7ff8837fe640 1 -- 172.21.15.169:0/4163383879 learned_addr learned my addr 172.21.15.169:0/4163383879 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:15.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.262+0000 7ff8837fe640 1 -- 172.21.15.169:0/4163383879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56548fc6d860 msgr2=0x56548fc77090 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:08:15.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.262+0000 7ff8837fe640 1 --2- 172.21.15.169:0/4163383879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56548fc6d860 0x56548fc77090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.262+0000 7ff8837fe640 1 -- 172.21.15.169:0/4163383879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc6e250 msgr2=0x56548fc7efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:15.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.262+0000 7ff8837fe640 1 --2- 172.21.15.169:0/4163383879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc6e250 0x56548fc7efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.262+0000 7ff8837fe640 1 -- 172.21.15.169:0/4163383879 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56548fc6eae0 con 0x56548fc733c0 2024-03-28T01:08:15.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.262+0000 7ff8827fc640 1 --2- 172.21.15.169:0/4163383879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc6e250 0x56548fc7efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:08:15.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.262+0000 7ff8837fe640 1 --2- 172.21.15.169:0/4163383879 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc733c0 0x56548fc677e0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ff87400e940 tx=0x7ff87409aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae09e86160cb18b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:15.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.262+0000 7ff868ff9640 1 -- 172.21.15.169:0/4163383879 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87407d030 con 0x56548fc733c0 2024-03-28T01:08:15.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.262+0000 7ff868ff9640 1 -- 172.21.15.169:0/4163383879 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff87409b730 con 0x56548fc733c0 2024-03-28T01:08:15.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.263+0000 7ff868ff9640 1 -- 172.21.15.169:0/4163383879 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8740b9b30 con 0x56548fc733c0 2024-03-28T01:08:15.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.264+0000 7ff88c1b5740 1 -- 172.21.15.169:0/4163383879 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc733c0 msgr2=0x56548fc677e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:15.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.264+0000 7ff88c1b5740 1 --2- 172.21.15.169:0/4163383879 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc733c0 0x56548fc677e0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ff87400e940 tx=0x7ff87409aca0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.264+0000 7ff88c1b5740 1 -- 172.21.15.169:0/4163383879 shutdown_connections 2024-03-28T01:08:15.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.265+0000 7ff88c1b5740 1 --2- 172.21.15.169:0/4163383879 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc733c0 0x56548fc677e0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.265+0000 7ff88c1b5740 1 --2- 172.21.15.169:0/4163383879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc6e250 0x56548fc7efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.265+0000 7ff88c1b5740 1 --2- 172.21.15.169:0/4163383879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56548fc6d860 0x56548fc77090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.265+0000 7ff88c1b5740 1 -- 172.21.15.169:0/4163383879 >> 172.21.15.169:0/4163383879 conn(0x56548fc68dd0 msgr2=0x56548fc6f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:15.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.270+0000 7ff88c1b5740 1 -- 172.21.15.169:0/4163383879 shutdown_connections 2024-03-28T01:08:15.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.270+0000 7ff88c1b5740 1 -- 172.21.15.169:0/4163383879 wait complete. 2024-03-28T01:08:15.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.271+0000 7ff88c1b5740 1 -- start start 2024-03-28T01:08:15.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.271+0000 7ff88c1b5740 1 Processor -- start 2024-03-28T01:08:15.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.271+0000 7ff88c1b5740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc6d860 0x56548fc7d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:15.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.271+0000 7ff88c1b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc6e250 0x56548fc7d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:15.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.271+0000 7ff88c1b5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56548fc733c0 0x56548fc7e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:15.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.271+0000 7ff88c1b5740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8841b76d0 con 0x56548fc6d860 2024-03-28T01:08:15.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.271+0000 7ff88c1b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8841b89d0 con 0x56548fc6e250 2024-03-28T01:08:15.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.271+0000 7ff88c1b5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8841b7d10 con 0x56548fc733c0 2024-03-28T01:08:15.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.271+0000 7ff88c1b5740 10 client.0 _ll_register_callbacks cb 0x7ff87409ae00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:15.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.272+0000 7ff8827fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc6e250 0x56548fc7d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:15.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.272+0000 7ff882ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc6d860 0x56548fc7d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:15.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.272+0000 7ff8827fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc6e250 0x56548fc7d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:47864/0 (socket says 172.21.15.169:47864) 2024-03-28T01:08:15.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.272+0000 7ff8827fc640 1 -- 172.21.15.169:0/528578124 learned_addr learned my addr 172.21.15.169:0/528578124 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:15.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.272+0000 7ff8837fe640 1 --2- 172.21.15.169:0/528578124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56548fc733c0 0x56548fc7e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:15.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.272+0000 7ff882ffd640 1 -- 172.21.15.169:0/528578124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56548fc733c0 msgr2=0x56548fc7e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:15.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.272+0000 7ff882ffd640 1 --2- 172.21.15.169:0/528578124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56548fc733c0 0x56548fc7e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.272+0000 7ff882ffd640 1 -- 172.21.15.169:0/528578124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc6e250 msgr2=0x56548fc7d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:15.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.272+0000 7ff882ffd640 1 --2- 172.21.15.169:0/528578124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc6e250 0x56548fc7d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.272+0000 7ff882ffd640 1 -- 172.21.15.169:0/528578124 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56548fc7eb60 con 0x56548fc6d860 2024-03-28T01:08:15.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.272+0000 7ff8827fc640 1 --2- 172.21.15.169:0/528578124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc6e250 0x56548fc7d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:08:15.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.272+0000 7ff882ffd640 1 --2- 172.21.15.169:0/528578124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc6d860 0x56548fc7d1a0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x56548fc77060 tx=0x7ff878013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f35baab640de8804 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:15.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.272+0000 7ff866ff5640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:08:15.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.272+0000 7ff866ff5640 1 -- 172.21.15.169:0/528578124 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff878004030 con 0x56548fc6d860 2024-03-28T01:08:15.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.272+0000 7ff88c1b5740 1 -- 172.21.15.169:0/528578124 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff87409b730 con 0x56548fc6d860 2024-03-28T01:08:15.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.274+0000 7ff866ff5640 1 -- 172.21.15.169:0/528578124 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7ff8780041d0 con 0x56548fc6d860 2024-03-28T01:08:15.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.274+0000 7ff866ff5640 1 -- 172.21.15.169:0/528578124 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff878004410 con 0x56548fc6d860 2024-03-28T01:08:15.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.274+0000 7ff88c1b5740 10 client.17607 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:15.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.274+0000 7ff88c1b5740 1 -- 172.21.15.169:0/528578124 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56548fcdfe00 con 0x56548fc6d860 2024-03-28T01:08:15.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.275+0000 7ff866ff5640 1 -- 172.21.15.169:0/528578124 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff878005500 con 0x56548fc6d860 2024-03-28T01:08:15.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.275+0000 7ff866ff5640 1 -- 172.21.15.169:0/528578124 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7ff8780035f0 con 0x56548fc6d860 2024-03-28T01:08:15.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.275+0000 7ff88c1b5740 10 client.17607 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:15.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.275+0000 7ff88c1b5740 10 client.17607 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:08:15.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.275+0000 7ff88c1b5740 20 client.17607 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:15.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.276+0000 7ff88c1b5740 1 --2- 172.21.15.169:0/528578124 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56548fce07b0 0x56548fce2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:15.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.276+0000 7ff88c1b5740 4 client.17607 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:08:15.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.276+0000 7ff88c1b5740 1 -- 172.21.15.169:0/528578124 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7ff87409b730 con 0x56548fce07b0 2024-03-28T01:08:15.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.276+0000 7ff8827fc640 1 --2- 172.21.15.169:0/528578124 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56548fce07b0 0x56548fce2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.276+0000 7ff866ff5640 10 client.17607 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.277+0000 7ff8827fc640 1 --2- 172.21.15.169:0/528578124 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56548fce07b0 0x56548fce2bd0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70289ddd3487742c server_cookie=ce5409dba5599014 in_seq=0 out_seq=0 2024-03-28T01:08:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.277+0000 7ff866ff5640 1 -- 172.21.15.169:0/528578124 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7ff87409b730 con 0x56548fce07b0 2024-03-28T01:08:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.277+0000 7ff866ff5640 10 client.17607 handle_command_reply: tid=0 2024-03-28T01:08:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.277+0000 7ff88c1b5740 1 client.17607 shutdown 2024-03-28T01:08:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.277+0000 7ff88c1b5740 1 -- 172.21.15.169:0/528578124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc6d860 msgr2=0x56548fc7d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.277+0000 7ff88c1b5740 1 --2- 172.21.15.169:0/528578124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc6d860 0x56548fc7d1a0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x56548fc77060 tx=0x7ff878013040 comp rx=0 tx=0).stop 2024-03-28T01:08:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.277+0000 7ff88c1b5740 1 -- 172.21.15.169:0/528578124 shutdown_connections 2024-03-28T01:08:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.277+0000 7ff88c1b5740 1 --2- 172.21.15.169:0/528578124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56548fc733c0 0x56548fc7e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.277+0000 7ff88c1b5740 1 --2- 172.21.15.169:0/528578124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56548fc6e250 0x56548fc7d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.277+0000 7ff88c1b5740 1 --2- 172.21.15.169:0/528578124 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x56548fce07b0 0x56548fce2bd0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.277+0000 7ff88c1b5740 1 --2- 172.21.15.169:0/528578124 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x56548fc6d860 0x56548fc7d1a0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.277+0000 7ff88c1b5740 1 -- 172.21.15.169:0/528578124 >> 172.21.15.169:0/528578124 conn(0x56548fc68dd0 msgr2=0x56548fc6c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:15.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.277+0000 7ff866ff5640 0 client.17607 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:15.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.277+0000 7ff88c1b5740 1 -- 172.21.15.169:0/528578124 shutdown_connections 2024-03-28T01:08:15.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.277+0000 7ff88c1b5740 1 -- 172.21.15.169:0/528578124 wait complete. 2024-03-28T01:08:15.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.277+0000 7ff88c1b5740 20 client.17607 trim_cache size 0 max 16384 2024-03-28T01:08:15.282 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:08:15.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.278+0000 7ff8894ea640 1 -- 172.21.15.169:0/3666815317 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff86c0a28d0 msgr2=0x7ff86c0a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:15.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.278+0000 7ff8894ea640 1 --2- 172.21.15.169:0/3666815317 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff86c0a28d0 0x7ff86c0a4d90 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff878009c20 tx=0x7ff878005f90 comp rx=0 tx=0).stop 2024-03-28T01:08:15.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.278+0000 7ff8894ea640 1 -- 172.21.15.169:0/3666815317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8841096a0 msgr2=0x7ff8841adbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:15.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.278+0000 7ff8894ea640 1 --2- 172.21.15.169:0/3666815317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8841096a0 0x7ff8841adbb0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff87400e970 tx=0x7ff87400ee30 comp rx=0 tx=0).stop 2024-03-28T01:08:15.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.278+0000 7ff8894ea640 1 -- 172.21.15.169:0/3666815317 shutdown_connections 2024-03-28T01:08:15.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.278+0000 7ff8894ea640 1 --2- 172.21.15.169:0/3666815317 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff86c0a28d0 0x7ff86c0a4d90 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.278+0000 7ff8894ea640 1 --2- 172.21.15.169:0/3666815317 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8841096a0 0x7ff8841adbb0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.278+0000 7ff8894ea640 1 --2- 172.21.15.169:0/3666815317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff884108ce0 0x7ff8841a6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.278+0000 7ff8894ea640 1 --2- 172.21.15.169:0/3666815317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff884072a10 0x7ff8841a6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:15.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.278+0000 7ff8894ea640 1 -- 172.21.15.169:0/3666815317 >> 172.21.15.169:0/3666815317 conn(0x7ff88406e4c0 msgr2=0x7ff884070100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:15.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.279+0000 7ff8894ea640 1 -- 172.21.15.169:0/3666815317 shutdown_connections 2024-03-28T01:08:15.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:15.279+0000 7ff8894ea640 1 -- 172.21.15.169:0/3666815317 wait complete. 2024-03-28T01:08:15.604 INFO:tasks.workunit:Stopping ['suites/iogen.sh'] on client.0... 2024-03-28T01:08:15.604 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-28T01:08:16.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:15 smithi169 ceph-mon[32274]: pgmap v703: 129 pgs: 129 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 634 MiB/s wr, 1.14k op/s 2024-03-28T01:08:16.089 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:15 smithi177 ceph-mon[36206]: pgmap v703: 129 pgs: 129 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 634 MiB/s wr, 1.14k op/s 2024-03-28T01:08:16.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:15 smithi174 ceph-mon[36060]: pgmap v703: 129 pgs: 129 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 634 MiB/s wr, 1.14k op/s 2024-03-28T01:08:16.302 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T01:08:16.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T01:08:16.302 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T01:08:16.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.409+0000 7f811075b640 1 -- 172.21.15.169:0/133493826 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81081028c0 msgr2=0x7f8108102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:16.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.409+0000 7f811075b640 1 --2- 172.21.15.169:0/133493826 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81081028c0 0x7f8108102d40 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f8104009900 tx=0x7f810402bc30 comp rx=0 tx=0).stop 2024-03-28T01:08:16.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.410+0000 7f811075b640 1 -- 172.21.15.169:0/133493826 shutdown_connections 2024-03-28T01:08:16.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.410+0000 7f811075b640 1 --2- 172.21.15.169:0/133493826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8108103280 0x7f8108109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.410+0000 7f811075b640 1 --2- 172.21.15.169:0/133493826 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81081028c0 0x7f8108102d40 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.410+0000 7f811075b640 1 --2- 172.21.15.169:0/133493826 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81081016d0 0x7f8108101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.410+0000 7f811075b640 1 -- 172.21.15.169:0/133493826 >> 172.21.15.169:0/133493826 conn(0x7f81080fcc40 msgr2=0x7f81080ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:16.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.410+0000 7f811075b640 1 -- 172.21.15.169:0/133493826 shutdown_connections 2024-03-28T01:08:16.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.410+0000 7f811075b640 1 -- 172.21.15.169:0/133493826 wait complete. 2024-03-28T01:08:16.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.410+0000 7f811075b640 1 Processor -- start 2024-03-28T01:08:16.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.410+0000 7f811075b640 1 -- start start 2024-03-28T01:08:16.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.410+0000 7f811075b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81081016d0 0x7f8108197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.410+0000 7f811075b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81081028c0 0x7f8108197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.410+0000 7f811075b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8108103280 0x7f810819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.410+0000 7f811075b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810810d330 con 0x7f81081028c0 2024-03-28T01:08:16.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.410+0000 7f811075b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810810d1b0 con 0x7f81081016d0 2024-03-28T01:08:16.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.410+0000 7f811075b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810810d4b0 con 0x7f8108103280 2024-03-28T01:08:16.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.411+0000 7f810dcf6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81081028c0 0x7f8108197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.411+0000 7f810e4f7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81081016d0 0x7f8108197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.411+0000 7f810ecf8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8108103280 0x7f810819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.411+0000 7f810dcf6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81081028c0 0x7f8108197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:51990/0 (socket says 172.21.15.169:51990) 2024-03-28T01:08:16.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.411+0000 7f810dcf6640 1 -- 172.21.15.169:0/1824769436 learned_addr learned my addr 172.21.15.169:0/1824769436 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:16.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.411+0000 7f810dcf6640 1 -- 172.21.15.169:0/1824769436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8108103280 msgr2=0x7f810819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:16.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.411+0000 7f810dcf6640 1 --2- 172.21.15.169:0/1824769436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8108103280 0x7f810819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.411+0000 7f810dcf6640 1 -- 172.21.15.169:0/1824769436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81081016d0 msgr2=0x7f8108197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:16.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.411+0000 7f810dcf6640 1 --2- 172.21.15.169:0/1824769436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81081016d0 0x7f8108197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.411+0000 7f810dcf6640 1 -- 172.21.15.169:0/1824769436 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810819f600 con 0x7f81081028c0 2024-03-28T01:08:16.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.411+0000 7f810dcf6640 1 --2- 172.21.15.169:0/1824769436 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81081028c0 0x7f8108197f70 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f8104009230 tx=0x7f810402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:16.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.411+0000 7f80ff7fe640 1 -- 172.21.15.169:0/1824769436 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8104041070 con 0x7f81081028c0 2024-03-28T01:08:16.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.411+0000 7f811075b640 1 -- 172.21.15.169:0/1824769436 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f810819f890 con 0x7f81081028c0 2024-03-28T01:08:16.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.411+0000 7f80ff7fe640 1 -- 172.21.15.169:0/1824769436 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8104004450 con 0x7f81081028c0 2024-03-28T01:08:16.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.411+0000 7f80ff7fe640 1 -- 172.21.15.169:0/1824769436 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8104003e20 con 0x7f81081028c0 2024-03-28T01:08:16.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.411+0000 7f811075b640 1 -- 172.21.15.169:0/1824769436 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f810819fd80 con 0x7f81081028c0 2024-03-28T01:08:16.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.413+0000 7f80ff7fe640 1 -- 172.21.15.169:0/1824769436 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8104044070 con 0x7f81081028c0 2024-03-28T01:08:16.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.413+0000 7f8113423740 1 Processor -- start 2024-03-28T01:08:16.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.414+0000 7f8113423740 1 -- start start 2024-03-28T01:08:16.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.414+0000 7f8113423740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596f532f960 0x5596f532fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.414+0000 7f8113423740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f5330350 0x5596f5334e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.414+0000 7f8113423740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f53354c0 0x5596f5337950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.414+0000 7f8113423740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596f526fc90 con 0x5596f53354c0 2024-03-28T01:08:16.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.414+0000 7f8113423740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596f5295070 con 0x5596f532f960 2024-03-28T01:08:16.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.414+0000 7f8113423740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596f523dc80 con 0x5596f5330350 2024-03-28T01:08:16.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.414+0000 7f810dcf6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f53354c0 0x5596f5337950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.414+0000 7f810e4f7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f5330350 0x5596f5334e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.414+0000 7f810e4f7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f5330350 0x5596f5334e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:58462/0 (socket says 172.21.15.169:58462) 2024-03-28T01:08:16.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.414+0000 7f810e4f7640 1 -- 172.21.15.169:0/3238610014 learned_addr learned my addr 172.21.15.169:0/3238610014 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:16.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.414+0000 7f810dcf6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f53354c0 0x5596f5337950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:52006/0 (socket says 172.21.15.169:52006) 2024-03-28T01:08:16.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.414+0000 7f810ecf8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596f532f960 0x5596f532fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.414+0000 7f80ff7fe640 1 --2- 172.21.15.169:0/1824769436 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f80f40a2800 0x7f80f40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.414+0000 7f810ecf8640 1 --2- 172.21.15.169:0/1824769436 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f80f40a2800 0x7f80f40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.414+0000 7f810e4f7640 1 -- 172.21.15.169:0/3238610014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596f532f960 msgr2=0x5596f532fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.414+0000 7f810e4f7640 1 --2- 172.21.15.169:0/3238610014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596f532f960 0x5596f532fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.414+0000 7f810e4f7640 1 -- 172.21.15.169:0/3238610014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f53354c0 msgr2=0x5596f5337950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.414+0000 7f810e4f7640 1 --2- 172.21.15.169:0/3238610014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f53354c0 0x5596f5337950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.414+0000 7f810e4f7640 1 -- 172.21.15.169:0/3238610014 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596f5235d10 con 0x5596f5330350 2024-03-28T01:08:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.414+0000 7f80ff7fe640 1 -- 172.21.15.169:0/1824769436 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f81040f0990 con 0x7f81081028c0 2024-03-28T01:08:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.415+0000 7f810e4f7640 1 --2- 172.21.15.169:0/3238610014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f5330350 0x5596f5334e50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f80f8009ca0 tx=0x7f80f8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=199766065ef92d48 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.415+0000 7f810ecf8640 1 --2- 172.21.15.169:0/1824769436 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f80f40a2800 0x7f80f40a4cc0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x5596f5330bb0 tx=0x7f8100004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.415+0000 7f80fcff9640 1 -- 172.21.15.169:0/3238610014 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8004220 con 0x5596f5330350 2024-03-28T01:08:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.415+0000 7f80fcff9640 1 -- 172.21.15.169:0/3238610014 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80f80043c0 con 0x5596f5330350 2024-03-28T01:08:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.415+0000 7f80fcff9640 1 -- 172.21.15.169:0/3238610014 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8019930 con 0x5596f5330350 2024-03-28T01:08:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.415+0000 7f8113423740 1 -- 172.21.15.169:0/3238610014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f5330350 msgr2=0x5596f5334e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.415+0000 7f8113423740 1 --2- 172.21.15.169:0/3238610014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f5330350 0x5596f5334e50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f80f8009ca0 tx=0x7f80f8011810 comp rx=0 tx=0).stop 2024-03-28T01:08:16.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.416+0000 7f8113423740 1 -- 172.21.15.169:0/3238610014 shutdown_connections 2024-03-28T01:08:16.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.416+0000 7f8113423740 1 --2- 172.21.15.169:0/3238610014 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f53354c0 0x5596f5337950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.416+0000 7f8113423740 1 --2- 172.21.15.169:0/3238610014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f5330350 0x5596f5334e50 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.416+0000 7f8113423740 1 --2- 172.21.15.169:0/3238610014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596f532f960 0x5596f532fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.416+0000 7f8113423740 1 -- 172.21.15.169:0/3238610014 >> 172.21.15.169:0/3238610014 conn(0x5596f52f81a0 msgr2=0x5596f532b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:16.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.416+0000 7f8113423740 1 -- 172.21.15.169:0/3238610014 shutdown_connections 2024-03-28T01:08:16.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.416+0000 7f8113423740 1 -- 172.21.15.169:0/3238610014 wait complete. 2024-03-28T01:08:16.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.416+0000 7f8113423740 1 -- start start 2024-03-28T01:08:16.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.416+0000 7f8113423740 1 Processor -- start 2024-03-28T01:08:16.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.416+0000 7f8113423740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596f532f960 0x5596f5339cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.416+0000 7f8113423740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f5330350 0x5596f533a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.416+0000 7f8113423740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f53354c0 0x5596f5341250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.416+0000 7f8113423740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596f52510d0 con 0x5596f5330350 2024-03-28T01:08:16.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.416+0000 7f8113423740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596f5291110 con 0x5596f532f960 2024-03-28T01:08:16.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.416+0000 7f8113423740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596f5338470 con 0x5596f53354c0 2024-03-28T01:08:16.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.416+0000 7f8113423740 10 client.0 _ll_register_callbacks cb 0x5596f528ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:16.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.416+0000 7f810dcf6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f5330350 0x5596f533a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.416+0000 7f810dcf6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f5330350 0x5596f533a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:52008/0 (socket says 172.21.15.169:52008) 2024-03-28T01:08:16.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.416+0000 7f810dcf6640 1 -- 172.21.15.169:0/1628761631 learned_addr learned my addr 172.21.15.169:0/1628761631 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:16.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.416+0000 7f810ecf8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f53354c0 0x5596f5341250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.416+0000 7f810e4f7640 1 --2- 172.21.15.169:0/1628761631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596f532f960 0x5596f5339cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.417+0000 7f810dcf6640 1 -- 172.21.15.169:0/1628761631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f53354c0 msgr2=0x5596f5341250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:16.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.417+0000 7f810dcf6640 1 --2- 172.21.15.169:0/1628761631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f53354c0 0x5596f5341250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.417+0000 7f810dcf6640 1 -- 172.21.15.169:0/1628761631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596f532f960 msgr2=0x5596f5339cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:16.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.417+0000 7f810dcf6640 1 --2- 172.21.15.169:0/1628761631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596f532f960 0x5596f5339cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.417+0000 7f810dcf6640 1 -- 172.21.15.169:0/1628761631 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5596f50717e0 con 0x5596f5330350 2024-03-28T01:08:16.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.417+0000 7f810ecf8640 1 --2- 172.21.15.169:0/1628761631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f53354c0 0x5596f5341250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:08:16.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.417+0000 7f810dcf6640 1 --2- 172.21.15.169:0/1628761631 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f5330350 0x5596f533a210 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f81040bcfd0 tx=0x7f8104005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6feedb4b9544c162 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:16.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.417+0000 7f80f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:08:16.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.417+0000 7f80f1ffb640 1 -- 172.21.15.169:0/1628761631 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810404b030 con 0x5596f5330350 2024-03-28T01:08:16.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.417+0000 7f80f1ffb640 1 -- 172.21.15.169:0/1628761631 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f81040b96b0 con 0x5596f5330350 2024-03-28T01:08:16.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.417+0000 7f8113423740 1 -- 172.21.15.169:0/1628761631 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f80f80043c0 con 0x5596f5330350 2024-03-28T01:08:16.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.417+0000 7f80f1ffb640 1 -- 172.21.15.169:0/1628761631 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f81040bc9a0 con 0x5596f5330350 2024-03-28T01:08:16.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.417+0000 7f8113423740 10 client.17619 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:16.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.417+0000 7f8113423740 1 -- 172.21.15.169:0/1628761631 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5596f50d1f70 con 0x5596f5330350 2024-03-28T01:08:16.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.417+0000 7f80f1ffb640 1 -- 172.21.15.169:0/1628761631 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80f80043c0 con 0x5596f5330350 2024-03-28T01:08:16.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.417+0000 7f80f1ffb640 1 -- 172.21.15.169:0/1628761631 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f81040c52a0 con 0x5596f5330350 2024-03-28T01:08:16.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.417+0000 7f8113423740 10 client.17619 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:16.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.417+0000 7f8113423740 10 client.17619 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:08:16.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.417+0000 7f8113423740 20 client.17619 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:16.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.418+0000 7f8113423740 1 --2- 172.21.15.169:0/1628761631 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5596f5345ad0 0x5596f5347ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.418+0000 7f8113423740 4 client.17619 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:08:16.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.418+0000 7f8113423740 1 -- 172.21.15.169:0/1628761631 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5596f52a25b0 con 0x5596f5345ad0 2024-03-28T01:08:16.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.418+0000 7f810e4f7640 1 --2- 172.21.15.169:0/1628761631 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5596f5345ad0 0x5596f5347ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.418+0000 7f810e4f7640 1 --2- 172.21.15.169:0/1628761631 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5596f5345ad0 0x5596f5347ee0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8ee289993a1b7e0 server_cookie=f741cf732631e1a6 in_seq=0 out_seq=0 2024-03-28T01:08:16.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.418+0000 7f80f1ffb640 10 client.17619 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:16.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.419+0000 7f80f1ffb640 1 -- 172.21.15.169:0/1628761631 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5596f52a25b0 con 0x5596f5345ad0 2024-03-28T01:08:16.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.419+0000 7f80f1ffb640 10 client.17619 handle_command_reply: tid=0 2024-03-28T01:08:16.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.419+0000 7f8113423740 1 client.17619 shutdown 2024-03-28T01:08:16.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.419+0000 7f8113423740 1 -- 172.21.15.169:0/1628761631 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f5330350 msgr2=0x5596f533a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:16.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.419+0000 7f8113423740 1 --2- 172.21.15.169:0/1628761631 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f5330350 0x5596f533a210 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f81040bcfd0 tx=0x7f8104005c90 comp rx=0 tx=0).stop 2024-03-28T01:08:16.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.419+0000 7f8113423740 1 -- 172.21.15.169:0/1628761631 shutdown_connections 2024-03-28T01:08:16.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.419+0000 7f8113423740 1 --2- 172.21.15.169:0/1628761631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f53354c0 0x5596f5341250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.419+0000 7f8113423740 1 --2- 172.21.15.169:0/1628761631 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5596f5345ad0 0x5596f5347ee0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.419+0000 7f8113423740 1 --2- 172.21.15.169:0/1628761631 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f5330350 0x5596f533a210 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.419+0000 7f8113423740 1 --2- 172.21.15.169:0/1628761631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596f532f960 0x5596f5339cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.419+0000 7f80f1ffb640 0 client.17619 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:16.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.419+0000 7f8113423740 1 -- 172.21.15.169:0/1628761631 >> 172.21.15.169:0/1628761631 conn(0x5596f52f81a0 msgr2=0x5596f532a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:16.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.419+0000 7f8113423740 1 -- 172.21.15.169:0/1628761631 shutdown_connections 2024-03-28T01:08:16.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.419+0000 7f8113423740 1 -- 172.21.15.169:0/1628761631 wait complete. 2024-03-28T01:08:16.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.419+0000 7f8113423740 20 client.17619 trim_cache size 0 max 16384 2024-03-28T01:08:16.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.435+0000 7f8113423740 1 Processor -- start 2024-03-28T01:08:16.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.435+0000 7f8113423740 1 -- start start 2024-03-28T01:08:16.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.435+0000 7f8113423740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f532f960 0x5596f534c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.435+0000 7f8113423740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f5330350 0x5596f534ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.435+0000 7f8113423740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596f53354c0 0x5596f534d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.435+0000 7f8113423740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81081ac6c0 con 0x5596f5330350 2024-03-28T01:08:16.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.435+0000 7f8113423740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81081aa1c0 con 0x5596f53354c0 2024-03-28T01:08:16.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.435+0000 7f8113423740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81081a8260 con 0x5596f532f960 2024-03-28T01:08:16.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.435+0000 7f810dcf6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f5330350 0x5596f534ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.435+0000 7f810dcf6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f5330350 0x5596f534ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34676/0 (socket says 172.21.15.169:34676) 2024-03-28T01:08:16.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.435+0000 7f810dcf6640 1 -- 172.21.15.169:0/461832460 learned_addr learned my addr 172.21.15.169:0/461832460 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:16.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.435+0000 7f810e4f7640 1 --2- 172.21.15.169:0/461832460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f532f960 0x5596f534c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.435+0000 7f810ecf8640 1 --2- 172.21.15.169:0/461832460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596f53354c0 0x5596f534d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.436+0000 7f810dcf6640 1 -- 172.21.15.169:0/461832460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f532f960 msgr2=0x5596f534c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:16.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.436+0000 7f810dcf6640 1 --2- 172.21.15.169:0/461832460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f532f960 0x5596f534c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.436+0000 7f810dcf6640 1 -- 172.21.15.169:0/461832460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596f53354c0 msgr2=0x5596f534d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:16.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.436+0000 7f810dcf6640 1 --2- 172.21.15.169:0/461832460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596f53354c0 0x5596f534d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.436+0000 7f810dcf6640 1 -- 172.21.15.169:0/461832460 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596f4f69170 con 0x5596f5330350 2024-03-28T01:08:16.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.436+0000 7f810dcf6640 1 --2- 172.21.15.169:0/461832460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f5330350 0x5596f534ccb0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f8104000c00 tx=0x7f81040b9410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4edcac22b7e0c0f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:16.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.436+0000 7f80f3fff640 1 -- 172.21.15.169:0/461832460 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81040f6020 con 0x5596f5330350 2024-03-28T01:08:16.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.436+0000 7f80f3fff640 1 -- 172.21.15.169:0/461832460 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8104004c60 con 0x5596f5330350 2024-03-28T01:08:16.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.436+0000 7f80f3fff640 1 -- 172.21.15.169:0/461832460 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810404b070 con 0x5596f5330350 2024-03-28T01:08:16.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f8113423740 1 -- 172.21.15.169:0/461832460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f5330350 msgr2=0x5596f534ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:16.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f8113423740 1 --2- 172.21.15.169:0/461832460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f5330350 0x5596f534ccb0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f8104000c00 tx=0x7f81040b9410 comp rx=0 tx=0).stop 2024-03-28T01:08:16.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f8113423740 1 -- 172.21.15.169:0/461832460 shutdown_connections 2024-03-28T01:08:16.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f8113423740 1 --2- 172.21.15.169:0/461832460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596f53354c0 0x5596f534d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f8113423740 1 --2- 172.21.15.169:0/461832460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f5330350 0x5596f534ccb0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f8113423740 1 --2- 172.21.15.169:0/461832460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f532f960 0x5596f534c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f8113423740 1 -- 172.21.15.169:0/461832460 >> 172.21.15.169:0/461832460 conn(0x5596f52f81a0 msgr2=0x5596f532a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:16.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f8113423740 1 -- 172.21.15.169:0/461832460 shutdown_connections 2024-03-28T01:08:16.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f8113423740 1 -- 172.21.15.169:0/461832460 wait complete. 2024-03-28T01:08:16.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f8113423740 1 -- start start 2024-03-28T01:08:16.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f8113423740 1 Processor -- start 2024-03-28T01:08:16.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f8113423740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f532f960 0x5596f539ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f8113423740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596f5330350 0x5596f539e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f8113423740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f53354c0 0x5596f53a5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f8113423740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81081a8860 con 0x5596f532f960 2024-03-28T01:08:16.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f8113423740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81081a9b60 con 0x5596f5330350 2024-03-28T01:08:16.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f8113423740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81081a8ea0 con 0x5596f53354c0 2024-03-28T01:08:16.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f8113423740 10 client.0 _ll_register_callbacks cb 0x7f8104005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:16.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f810e4f7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f532f960 0x5596f539ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f810e4f7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f532f960 0x5596f539ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34690/0 (socket says 172.21.15.169:34690) 2024-03-28T01:08:16.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f810e4f7640 1 -- 172.21.15.169:0/487938682 learned_addr learned my addr 172.21.15.169:0/487938682 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:16.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f810dcf6640 1 --2- 172.21.15.169:0/487938682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596f5330350 0x5596f539e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.437+0000 7f810ecf8640 1 --2- 172.21.15.169:0/487938682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f53354c0 0x5596f53a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f810e4f7640 1 -- 172.21.15.169:0/487938682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f53354c0 msgr2=0x5596f53a5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:16.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f810e4f7640 1 --2- 172.21.15.169:0/487938682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f53354c0 0x5596f53a5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f810e4f7640 1 -- 172.21.15.169:0/487938682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596f5330350 msgr2=0x5596f539e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:16.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f810e4f7640 1 --2- 172.21.15.169:0/487938682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596f5330350 0x5596f539e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f810e4f7640 1 -- 172.21.15.169:0/487938682 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5596f53a5940 con 0x5596f532f960 2024-03-28T01:08:16.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f810dcf6640 1 --2- 172.21.15.169:0/487938682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596f5330350 0x5596f539e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:08:16.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f810e4f7640 1 --2- 172.21.15.169:0/487938682 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f532f960 0x5596f539ddd0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f80f8009ca0 tx=0x7f80f8003dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f0d1c766f77d71 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:16.447 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f80f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:08:16.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f80f1ffb640 1 -- 172.21.15.169:0/487938682 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f802f070 con 0x5596f532f960 2024-03-28T01:08:16.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f80f1ffb640 1 -- 172.21.15.169:0/487938682 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f80f802b2b0 con 0x5596f532f960 2024-03-28T01:08:16.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f8113423740 1 -- 172.21.15.169:0/487938682 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8104004c60 con 0x5596f532f960 2024-03-28T01:08:16.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f80f1ffb640 1 -- 172.21.15.169:0/487938682 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80f8005080 con 0x5596f532f960 2024-03-28T01:08:16.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f8113423740 10 client.17625 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:16.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f8113423740 1 -- 172.21.15.169:0/487938682 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5596f53a6cf0 con 0x5596f532f960 2024-03-28T01:08:16.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f80f1ffb640 1 -- 172.21.15.169:0/487938682 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8104004c60 con 0x5596f532f960 2024-03-28T01:08:16.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f80f1ffb640 1 -- 172.21.15.169:0/487938682 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f80f802b520 con 0x5596f532f960 2024-03-28T01:08:16.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f8113423740 10 client.17625 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:16.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f8113423740 10 client.17625 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:08:16.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f8113423740 20 client.17625 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:16.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f8113423740 1 --2- 172.21.15.169:0/487938682 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5596f53a7560 0x5596f53a9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f8113423740 4 client.17625 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T01:08:16.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.438+0000 7f8113423740 1 -- 172.21.15.169:0/487938682 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5596f529c900 con 0x5596f53a7560 2024-03-28T01:08:16.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.439+0000 7f810dcf6640 1 --2- 172.21.15.169:0/487938682 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5596f53a7560 0x5596f53a9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.439+0000 7f80f1ffb640 10 client.17625 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:16.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.439+0000 7f810dcf6640 1 --2- 172.21.15.169:0/487938682 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5596f53a7560 0x5596f53a9980 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f62ff54c3466dc47 server_cookie=7881532402ef3842 in_seq=0 out_seq=0 2024-03-28T01:08:16.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.440+0000 7f80f1ffb640 1 -- 172.21.15.169:0/487938682 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5596f529c900 con 0x5596f53a7560 2024-03-28T01:08:16.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.440+0000 7f80f1ffb640 10 client.17625 handle_command_reply: tid=0 2024-03-28T01:08:16.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.440+0000 7f8113423740 1 client.17625 shutdown 2024-03-28T01:08:16.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.440+0000 7f8113423740 1 -- 172.21.15.169:0/487938682 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f532f960 msgr2=0x5596f539ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:16.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.440+0000 7f8113423740 1 --2- 172.21.15.169:0/487938682 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f532f960 0x5596f539ddd0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f80f8009ca0 tx=0x7f80f8003dd0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.440+0000 7f8113423740 1 -- 172.21.15.169:0/487938682 shutdown_connections 2024-03-28T01:08:16.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.440+0000 7f8113423740 1 --2- 172.21.15.169:0/487938682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596f53354c0 0x5596f53a5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.440+0000 7f8113423740 1 --2- 172.21.15.169:0/487938682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596f5330350 0x5596f539e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.440+0000 7f8113423740 1 --2- 172.21.15.169:0/487938682 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5596f53a7560 0x5596f53a9980 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.440+0000 7f8113423740 1 --2- 172.21.15.169:0/487938682 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5596f532f960 0x5596f539ddd0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.440+0000 7f80f1ffb640 0 client.17625 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:16.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.440+0000 7f8113423740 1 -- 172.21.15.169:0/487938682 >> 172.21.15.169:0/487938682 conn(0x5596f52f81a0 msgr2=0x5596f5329ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:16.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.440+0000 7f8113423740 1 -- 172.21.15.169:0/487938682 shutdown_connections 2024-03-28T01:08:16.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.440+0000 7f8113423740 1 -- 172.21.15.169:0/487938682 wait complete. 2024-03-28T01:08:16.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.440+0000 7f8113423740 20 client.17625 trim_cache size 0 max 16384 2024-03-28T01:08:16.450 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:08:16.450 INFO:teuthology.orchestra.run.smithi169.stdout: "return_code": 0, 2024-03-28T01:08:16.450 INFO:teuthology.orchestra.run.smithi169.stdout: "scrub_tag": "bdf1aae6-278d-4d20-9ff2-1169a5fd3b02", 2024-03-28T01:08:16.450 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": "asynchronous" 2024-03-28T01:08:16.450 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:08:16.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.441+0000 7f811075b640 1 -- 172.21.15.169:0/1824769436 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f80f40a2800 msgr2=0x7f80f40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:16.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.441+0000 7f811075b640 1 --2- 172.21.15.169:0/1824769436 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f80f40a2800 0x7f80f40a4cc0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x5596f5330bb0 tx=0x7f8100004ab0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.441+0000 7f811075b640 1 -- 172.21.15.169:0/1824769436 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81081028c0 msgr2=0x7f8108197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:16.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.441+0000 7f811075b640 1 --2- 172.21.15.169:0/1824769436 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81081028c0 0x7f8108197f70 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f8104009230 tx=0x7f810402c650 comp rx=0 tx=0).stop 2024-03-28T01:08:16.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.441+0000 7f811075b640 1 -- 172.21.15.169:0/1824769436 shutdown_connections 2024-03-28T01:08:16.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.441+0000 7f811075b640 1 --2- 172.21.15.169:0/1824769436 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f80f40a2800 0x7f80f40a4cc0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.441+0000 7f811075b640 1 --2- 172.21.15.169:0/1824769436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8108103280 0x7f810819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.441+0000 7f811075b640 1 --2- 172.21.15.169:0/1824769436 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f81081028c0 0x7f8108197f70 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.441+0000 7f811075b640 1 --2- 172.21.15.169:0/1824769436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81081016d0 0x7f8108197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.441+0000 7f811075b640 1 -- 172.21.15.169:0/1824769436 >> 172.21.15.169:0/1824769436 conn(0x7f81080fcc40 msgr2=0x7f8108105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:16.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.441+0000 7f811075b640 1 -- 172.21.15.169:0/1824769436 shutdown_connections 2024-03-28T01:08:16.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.441+0000 7f811075b640 1 -- 172.21.15.169:0/1824769436 wait complete. 2024-03-28T01:08:16.455 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:08:16.455 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:08:16.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.557+0000 7fddb92b2640 1 -- 172.21.15.169:0/3339999353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddb4102150 msgr2=0x7fddb41089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:16.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.557+0000 7fddb92b2640 1 --2- 172.21.15.169:0/3339999353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddb4102150 0x7fddb41089e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fddac009ca0 tx=0x7fddac02c0f0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.557+0000 7fddb92b2640 1 -- 172.21.15.169:0/3339999353 shutdown_connections 2024-03-28T01:08:16.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.557+0000 7fddb92b2640 1 --2- 172.21.15.169:0/3339999353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddb4102150 0x7fddb41089e0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.557+0000 7fddb92b2640 1 --2- 172.21.15.169:0/3339999353 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fddb4101790 0x7fddb4101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.557+0000 7fddb92b2640 1 --2- 172.21.15.169:0/3339999353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddb41005a0 0x7fddb41009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.557+0000 7fddb92b2640 1 -- 172.21.15.169:0/3339999353 >> 172.21.15.169:0/3339999353 conn(0x7fddb40fbb30 msgr2=0x7fddb40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:16.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.557+0000 7fddb92b2640 1 -- 172.21.15.169:0/3339999353 shutdown_connections 2024-03-28T01:08:16.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.557+0000 7fddb92b2640 1 -- 172.21.15.169:0/3339999353 wait complete. 2024-03-28T01:08:16.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.558+0000 7fddb92b2640 1 Processor -- start 2024-03-28T01:08:16.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.558+0000 7fddb92b2640 1 -- start start 2024-03-28T01:08:16.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.558+0000 7fddb92b2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddb41005a0 0x7fddb4069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.558+0000 7fddb92b2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddb4101790 0x7fddb419c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.558+0000 7fddb92b2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fddb4102150 0x7fddb419ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.558+0000 7fddb92b2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb410c230 con 0x7fddb4102150 2024-03-28T01:08:16.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.558+0000 7fddb92b2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb410c0b0 con 0x7fddb4101790 2024-03-28T01:08:16.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.558+0000 7fddb92b2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb410c3b0 con 0x7fddb41005a0 2024-03-28T01:08:16.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.558+0000 7fddb37fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fddb4102150 0x7fddb419ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.558+0000 7fddb37fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fddb4102150 0x7fddb419ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34708/0 (socket says 172.21.15.169:34708) 2024-03-28T01:08:16.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.558+0000 7fddb37fe640 1 -- 172.21.15.169:0/2121609816 learned_addr learned my addr 172.21.15.169:0/2121609816 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:16.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.558+0000 7fddb2ffd640 1 --2- 172.21.15.169:0/2121609816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddb41005a0 0x7fddb4069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.558+0000 7fddb27fc640 1 --2- 172.21.15.169:0/2121609816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddb4101790 0x7fddb419c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.559+0000 7fddb37fe640 1 -- 172.21.15.169:0/2121609816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddb41005a0 msgr2=0x7fddb4069390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:16.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.559+0000 7fddb37fe640 1 --2- 172.21.15.169:0/2121609816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddb41005a0 0x7fddb4069390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.559+0000 7fddb37fe640 1 -- 172.21.15.169:0/2121609816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddb4101790 msgr2=0x7fddb419c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:16.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.559+0000 7fddb37fe640 1 --2- 172.21.15.169:0/2121609816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddb4101790 0x7fddb419c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.559+0000 7fddb37fe640 1 -- 172.21.15.169:0/2121609816 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb419f580 con 0x7fddb4102150 2024-03-28T01:08:16.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.559+0000 7fddb37fe640 1 --2- 172.21.15.169:0/2121609816 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fddb4102150 0x7fddb419ecc0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fddac037040 tx=0x7fddac032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:16.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.559+0000 7fdd9bfff640 1 -- 172.21.15.169:0/2121609816 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac041020 con 0x7fddb4102150 2024-03-28T01:08:16.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.559+0000 7fddb92b2640 1 -- 172.21.15.169:0/2121609816 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb419f810 con 0x7fddb4102150 2024-03-28T01:08:16.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.559+0000 7fdd9bfff640 1 -- 172.21.15.169:0/2121609816 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fddac034cd0 con 0x7fddb4102150 2024-03-28T01:08:16.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.559+0000 7fdd9bfff640 1 -- 172.21.15.169:0/2121609816 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac039040 con 0x7fddb4102150 2024-03-28T01:08:16.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.559+0000 7fddb92b2640 1 -- 172.21.15.169:0/2121609816 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb419fd50 con 0x7fddb4102150 2024-03-28T01:08:16.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.561+0000 7fdd9bfff640 1 -- 172.21.15.169:0/2121609816 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fddac044020 con 0x7fddb4102150 2024-03-28T01:08:16.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.561+0000 7fddbbf7c740 1 Processor -- start 2024-03-28T01:08:16.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.561+0000 7fddbbf7c740 1 -- start start 2024-03-28T01:08:16.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.562+0000 7fddbbf7c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562acd906da0 0x562acd9071c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.562+0000 7fddbbf7c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd907790 0x562acd90c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.562+0000 7fddbbf7c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd90c900 0x562acd90ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.562+0000 7fddbbf7c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x562acd86c3d0 con 0x562acd90c900 2024-03-28T01:08:16.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.562+0000 7fddbbf7c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x562acd814ad0 con 0x562acd907790 2024-03-28T01:08:16.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.562+0000 7fddbbf7c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x562acd868470 con 0x562acd906da0 2024-03-28T01:08:16.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.562+0000 7fddb2ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd907790 0x562acd90c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.562+0000 7fddb27fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562acd906da0 0x562acd9071c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.562+0000 7fddb2ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd907790 0x562acd90c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:34632/0 (socket says 172.21.15.169:34632) 2024-03-28T01:08:16.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.562+0000 7fddb27fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd90c900 0x562acd90ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.562+0000 7fddb2ffd640 1 -- 172.21.15.169:0/1244418814 learned_addr learned my addr 172.21.15.169:0/1244418814 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:16.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.562+0000 7fdd9bfff640 1 --2- 172.21.15.169:0/2121609816 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdd9c0a2800 0x7fdd9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.562+0000 7fddb2ffd640 1 -- 172.21.15.169:0/1244418814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562acd906da0 msgr2=0x562acd9071c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:16.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.562+0000 7fddb2ffd640 1 --2- 172.21.15.169:0/1244418814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562acd906da0 0x562acd9071c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.562+0000 7fddb2ffd640 1 -- 172.21.15.169:0/1244418814 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd90c900 msgr2=0x562acd90ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:16.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.562+0000 7fddb37fe640 1 --2- 172.21.15.169:0/2121609816 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdd9c0a2800 0x7fdd9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.562+0000 7fddb2ffd640 1 --2- 172.21.15.169:0/1244418814 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd90c900 0x562acd90ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.562+0000 7fddb2ffd640 1 -- 172.21.15.169:0/1244418814 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562acd64a7d0 con 0x562acd907790 2024-03-28T01:08:16.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.562+0000 7fddb27fc640 1 --2- 172.21.15.169:0/1244418814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562acd906da0 0x562acd9071c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:08:16.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.562+0000 7fdd9bfff640 1 -- 172.21.15.169:0/2121609816 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fddac0f0c90 con 0x7fddb4102150 2024-03-28T01:08:16.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.562+0000 7fddb2ffd640 1 --2- 172.21.15.169:0/1244418814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd907790 0x562acd90c290 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fdda8009ca0 tx=0x7fdda8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12edfb5840694627 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:16.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.563+0000 7fdd997fa640 1 -- 172.21.15.169:0/1244418814 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda8004220 con 0x562acd907790 2024-03-28T01:08:16.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.563+0000 7fdd997fa640 1 -- 172.21.15.169:0/1244418814 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdda80043c0 con 0x562acd907790 2024-03-28T01:08:16.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.563+0000 7fddb37fe640 1 --2- 172.21.15.169:0/2121609816 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdd9c0a2800 0x7fdd9c0a4cc0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fddac0c5500 tx=0x7fddac039bb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:16.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.563+0000 7fdd997fa640 1 -- 172.21.15.169:0/1244418814 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda8019830 con 0x562acd907790 2024-03-28T01:08:16.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.563+0000 7fddbbf7c740 1 -- 172.21.15.169:0/1244418814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd907790 msgr2=0x562acd90c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:16.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.563+0000 7fddbbf7c740 1 --2- 172.21.15.169:0/1244418814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd907790 0x562acd90c290 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fdda8009ca0 tx=0x7fdda8011810 comp rx=0 tx=0).stop 2024-03-28T01:08:16.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.563+0000 7fddbbf7c740 1 -- 172.21.15.169:0/1244418814 shutdown_connections 2024-03-28T01:08:16.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.563+0000 7fddbbf7c740 1 --2- 172.21.15.169:0/1244418814 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd90c900 0x562acd90ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.563+0000 7fddbbf7c740 1 --2- 172.21.15.169:0/1244418814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd907790 0x562acd90c290 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.563+0000 7fddbbf7c740 1 --2- 172.21.15.169:0/1244418814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562acd906da0 0x562acd9071c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.563+0000 7fddbbf7c740 1 -- 172.21.15.169:0/1244418814 >> 172.21.15.169:0/1244418814 conn(0x562acd902310 msgr2=0x562acd902710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:16.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.563+0000 7fddbbf7c740 1 -- 172.21.15.169:0/1244418814 shutdown_connections 2024-03-28T01:08:16.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.563+0000 7fddbbf7c740 1 -- 172.21.15.169:0/1244418814 wait complete. 2024-03-28T01:08:16.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fddbbf7c740 1 -- start start 2024-03-28T01:08:16.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fddbbf7c740 1 Processor -- start 2024-03-28T01:08:16.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fddbbf7c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd906da0 0x562acd911220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fddbbf7c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd907790 0x562acd911760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fddbbf7c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562acd90c900 0x562acd9187a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fddbbf7c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x562acd90f8b0 con 0x562acd906da0 2024-03-28T01:08:16.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fddbbf7c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x562acd828060 con 0x562acd907790 2024-03-28T01:08:16.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fddbbf7c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x562acd90fa30 con 0x562acd90c900 2024-03-28T01:08:16.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fddbbf7c740 10 client.0 _ll_register_callbacks cb 0x562acd80a6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:16.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fddb27fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd906da0 0x562acd911220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fddb27fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd906da0 0x562acd911220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34728/0 (socket says 172.21.15.169:34728) 2024-03-28T01:08:16.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fddb27fc640 1 -- 172.21.15.169:0/686242613 learned_addr learned my addr 172.21.15.169:0/686242613 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:16.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fddb2ffd640 1 --2- 172.21.15.169:0/686242613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd907790 0x562acd911760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fddb27fc640 1 --2- 172.21.15.169:0/686242613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562acd90c900 0x562acd9187a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fddb2ffd640 1 -- 172.21.15.169:0/686242613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562acd90c900 msgr2=0x562acd9187a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:16.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fddb2ffd640 1 --2- 172.21.15.169:0/686242613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562acd90c900 0x562acd9187a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fddb2ffd640 1 -- 172.21.15.169:0/686242613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd906da0 msgr2=0x562acd911220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:16.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fddb2ffd640 1 --2- 172.21.15.169:0/686242613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd906da0 0x562acd911220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fddb2ffd640 1 -- 172.21.15.169:0/686242613 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562acd5803f0 con 0x562acd907790 2024-03-28T01:08:16.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fddb27fc640 1 --2- 172.21.15.169:0/686242613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd906da0 0x562acd911220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:08:16.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fddb2ffd640 1 --2- 172.21.15.169:0/686242613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd907790 0x562acd911760 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x562acd903280 tx=0x7fdda80038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16313a059aebee81 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:16.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.564+0000 7fdd96ff5640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:08:16.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.565+0000 7fdd96ff5640 1 -- 172.21.15.169:0/686242613 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda8028070 con 0x562acd907790 2024-03-28T01:08:16.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.565+0000 7fdd96ff5640 1 -- 172.21.15.169:0/686242613 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fdda8005040 con 0x562acd907790 2024-03-28T01:08:16.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.565+0000 7fddbbf7c740 1 -- 172.21.15.169:0/686242613 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdda80043c0 con 0x562acd907790 2024-03-28T01:08:16.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.565+0000 7fdd96ff5640 1 -- 172.21.15.169:0/686242613 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdda8034050 con 0x562acd907790 2024-03-28T01:08:16.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.565+0000 7fddbbf7c740 10 client.27530 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:16.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.565+0000 7fddbbf7c740 1 -- 172.21.15.169:0/686242613 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x562acd5400f0 con 0x562acd907790 2024-03-28T01:08:16.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.565+0000 7fdd96ff5640 1 -- 172.21.15.169:0/686242613 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdda80043c0 con 0x562acd907790 2024-03-28T01:08:16.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.565+0000 7fdd96ff5640 1 -- 172.21.15.169:0/686242613 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fdda8004a80 con 0x562acd907790 2024-03-28T01:08:16.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.565+0000 7fddbbf7c740 10 client.27530 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:16.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.565+0000 7fddbbf7c740 10 client.27530 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:08:16.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.565+0000 7fddbbf7c740 20 client.27530 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:16.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.565+0000 7fddbbf7c740 1 --2- 172.21.15.169:0/686242613 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562acd91c9b0 0x562acd91edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.565+0000 7fddbbf7c740 4 client.27530 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:08:16.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.565+0000 7fddbbf7c740 1 -- 172.21.15.169:0/686242613 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562acd879910 con 0x562acd91c9b0 2024-03-28T01:08:16.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.565+0000 7fddb27fc640 1 --2- 172.21.15.169:0/686242613 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562acd91c9b0 0x562acd91edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.566+0000 7fddb27fc640 1 --2- 172.21.15.169:0/686242613 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562acd91c9b0 0x562acd91edc0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb12dbc5d9ed3a07 server_cookie=48aec17c6f87f4a4 in_seq=0 out_seq=0 2024-03-28T01:08:16.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.566+0000 7fdd96ff5640 10 client.27530 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:16.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.566+0000 7fdd96ff5640 1 -- 172.21.15.169:0/686242613 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562acd879910 con 0x562acd91c9b0 2024-03-28T01:08:16.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.566+0000 7fdd96ff5640 10 client.27530 handle_command_reply: tid=0 2024-03-28T01:08:16.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.566+0000 7fddbbf7c740 1 client.27530 shutdown 2024-03-28T01:08:16.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.566+0000 7fddbbf7c740 1 -- 172.21.15.169:0/686242613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd907790 msgr2=0x562acd911760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:16.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.566+0000 7fddbbf7c740 1 --2- 172.21.15.169:0/686242613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd907790 0x562acd911760 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x562acd903280 tx=0x7fdda80038a0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.566+0000 7fddbbf7c740 1 -- 172.21.15.169:0/686242613 shutdown_connections 2024-03-28T01:08:16.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.566+0000 7fddbbf7c740 1 --2- 172.21.15.169:0/686242613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562acd90c900 0x562acd9187a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.566+0000 7fddbbf7c740 1 --2- 172.21.15.169:0/686242613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd907790 0x562acd911760 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.566+0000 7fddbbf7c740 1 --2- 172.21.15.169:0/686242613 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562acd91c9b0 0x562acd91edc0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.566+0000 7fddbbf7c740 1 --2- 172.21.15.169:0/686242613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd906da0 0x562acd911220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.566+0000 7fdd96ff5640 0 client.27530 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:16.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.566+0000 7fddbbf7c740 1 -- 172.21.15.169:0/686242613 >> 172.21.15.169:0/686242613 conn(0x562acd902310 msgr2=0x562acd901530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:16.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.566+0000 7fddbbf7c740 1 -- 172.21.15.169:0/686242613 shutdown_connections 2024-03-28T01:08:16.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.566+0000 7fddbbf7c740 1 -- 172.21.15.169:0/686242613 wait complete. 2024-03-28T01:08:16.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.567+0000 7fddbbf7c740 20 client.27530 trim_cache size 0 max 16384 2024-03-28T01:08:16.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.583+0000 7fddbbf7c740 1 Processor -- start 2024-03-28T01:08:16.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.583+0000 7fddbbf7c740 1 -- start start 2024-03-28T01:08:16.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.583+0000 7fddbbf7c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd906da0 0x562acd908bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.583+0000 7fddbbf7c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd907790 0x562acd918760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.583+0000 7fddbbf7c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562acd90c900 0x562acd909130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.583+0000 7fddbbf7c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb41ac6d0 con 0x562acd907790 2024-03-28T01:08:16.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.583+0000 7fddbbf7c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb41aa1d0 con 0x562acd906da0 2024-03-28T01:08:16.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.583+0000 7fddbbf7c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb41a8270 con 0x562acd90c900 2024-03-28T01:08:16.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.583+0000 7fddb2ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd907790 0x562acd918760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.583+0000 7fddb2ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd907790 0x562acd918760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34730/0 (socket says 172.21.15.169:34730) 2024-03-28T01:08:16.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.583+0000 7fddb2ffd640 1 -- 172.21.15.169:0/4078884493 learned_addr learned my addr 172.21.15.169:0/4078884493 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:16.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.583+0000 7fddb27fc640 1 --2- 172.21.15.169:0/4078884493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd906da0 0x562acd908bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.583+0000 7fddb27fc640 1 --2- 172.21.15.169:0/4078884493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562acd90c900 0x562acd909130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.584+0000 7fddb2ffd640 1 -- 172.21.15.169:0/4078884493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562acd90c900 msgr2=0x562acd909130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:16.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.584+0000 7fddb2ffd640 1 --2- 172.21.15.169:0/4078884493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562acd90c900 0x562acd909130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.584+0000 7fddb2ffd640 1 -- 172.21.15.169:0/4078884493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd906da0 msgr2=0x562acd908bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:16.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.584+0000 7fddb2ffd640 1 --2- 172.21.15.169:0/4078884493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd906da0 0x562acd908bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.584+0000 7fddb2ffd640 1 -- 172.21.15.169:0/4078884493 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562acd903080 con 0x562acd907790 2024-03-28T01:08:16.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.584+0000 7fddb2ffd640 1 --2- 172.21.15.169:0/4078884493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd907790 0x562acd918760 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fdda8004e50 tx=0x7fdda8011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a5090ba155ac979 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:16.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.584+0000 7fdd98ff9640 1 -- 172.21.15.169:0/4078884493 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda8028040 con 0x562acd907790 2024-03-28T01:08:16.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.584+0000 7fdd98ff9640 1 -- 172.21.15.169:0/4078884493 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdda8019e20 con 0x562acd907790 2024-03-28T01:08:16.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.584+0000 7fdd98ff9640 1 -- 172.21.15.169:0/4078884493 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda8019690 con 0x562acd907790 2024-03-28T01:08:16.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.584+0000 7fddbbf7c740 1 -- 172.21.15.169:0/4078884493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd907790 msgr2=0x562acd918760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:16.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.584+0000 7fddbbf7c740 1 --2- 172.21.15.169:0/4078884493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd907790 0x562acd918760 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fdda8004e50 tx=0x7fdda8011dc0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.584+0000 7fddbbf7c740 1 -- 172.21.15.169:0/4078884493 shutdown_connections 2024-03-28T01:08:16.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.584+0000 7fddbbf7c740 1 --2- 172.21.15.169:0/4078884493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562acd90c900 0x562acd909130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.584+0000 7fddbbf7c740 1 --2- 172.21.15.169:0/4078884493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd907790 0x562acd918760 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddbbf7c740 1 --2- 172.21.15.169:0/4078884493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd906da0 0x562acd908bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddbbf7c740 1 -- 172.21.15.169:0/4078884493 >> 172.21.15.169:0/4078884493 conn(0x562acd902310 msgr2=0x562acd911220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:16.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddbbf7c740 1 -- 172.21.15.169:0/4078884493 shutdown_connections 2024-03-28T01:08:16.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddbbf7c740 1 -- 172.21.15.169:0/4078884493 wait complete. 2024-03-28T01:08:16.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddbbf7c740 1 -- start start 2024-03-28T01:08:16.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddbbf7c740 1 Processor -- start 2024-03-28T01:08:16.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddbbf7c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562acd906da0 0x562acd915c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddbbf7c740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd907790 0x562acd916160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddbbf7c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd90c900 0x562acd917170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddbbf7c740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb41a8870 con 0x562acd907790 2024-03-28T01:08:16.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddbbf7c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb41a9b70 con 0x562acd90c900 2024-03-28T01:08:16.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddbbf7c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb41a8eb0 con 0x562acd906da0 2024-03-28T01:08:16.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddbbf7c740 10 client.0 _ll_register_callbacks cb 0x7fdda4009dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:16.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddb2ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd907790 0x562acd916160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddb2ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd907790 0x562acd916160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:34746/0 (socket says 172.21.15.169:34746) 2024-03-28T01:08:16.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddb2ffd640 1 -- 172.21.15.169:0/419854363 learned_addr learned my addr 172.21.15.169:0/419854363 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:16.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddb27fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562acd906da0 0x562acd915c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddb27fc640 1 --2- 172.21.15.169:0/419854363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd90c900 0x562acd917170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddb2ffd640 1 -- 172.21.15.169:0/419854363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562acd906da0 msgr2=0x562acd915c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:16.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddb2ffd640 1 --2- 172.21.15.169:0/419854363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562acd906da0 0x562acd915c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddb2ffd640 1 -- 172.21.15.169:0/419854363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd90c900 msgr2=0x562acd917170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:16.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddb2ffd640 1 --2- 172.21.15.169:0/419854363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd90c900 0x562acd917170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.585+0000 7fddb2ffd640 1 -- 172.21.15.169:0/419854363 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562acd917760 con 0x562acd907790 2024-03-28T01:08:16.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.586+0000 7fddb2ffd640 1 --2- 172.21.15.169:0/419854363 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd907790 0x562acd916160 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fdda8013040 tx=0x7fdda8009b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfadba6eeea2169a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:16.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.586+0000 7fdd96ff5640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:08:16.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.586+0000 7fdd96ff5640 1 -- 172.21.15.169:0/419854363 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda8028070 con 0x562acd907790 2024-03-28T01:08:16.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.586+0000 7fdd96ff5640 1 -- 172.21.15.169:0/419854363 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fdda80051b0 con 0x562acd907790 2024-03-28T01:08:16.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.586+0000 7fddbbf7c740 1 -- 172.21.15.169:0/419854363 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdda8019e20 con 0x562acd907790 2024-03-28T01:08:16.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.586+0000 7fdd96ff5640 1 -- 172.21.15.169:0/419854363 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdda804e050 con 0x562acd907790 2024-03-28T01:08:16.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.586+0000 7fddbbf7c740 10 client.17643 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:16.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.586+0000 7fddbbf7c740 1 -- 172.21.15.169:0/419854363 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x562acd918a70 con 0x562acd907790 2024-03-28T01:08:16.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.586+0000 7fdd96ff5640 1 -- 172.21.15.169:0/419854363 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdda8019e20 con 0x562acd907790 2024-03-28T01:08:16.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.586+0000 7fdd96ff5640 1 -- 172.21.15.169:0/419854363 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fdda8019a80 con 0x562acd907790 2024-03-28T01:08:16.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.586+0000 7fddbbf7c740 10 client.17643 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:16.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.586+0000 7fddbbf7c740 10 client.17643 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:08:16.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.586+0000 7fddbbf7c740 20 client.17643 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:16.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.586+0000 7fddbbf7c740 1 --2- 172.21.15.169:0/419854363 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562acd978940 0x562acd97ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:16.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.586+0000 7fddbbf7c740 4 client.17643 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:08:16.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.586+0000 7fddbbf7c740 1 -- 172.21.15.169:0/419854363 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562acd873c60 con 0x562acd978940 2024-03-28T01:08:16.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.586+0000 7fddb27fc640 1 --2- 172.21.15.169:0/419854363 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562acd978940 0x562acd97ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:16.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.587+0000 7fddb27fc640 1 --2- 172.21.15.169:0/419854363 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562acd978940 0x562acd97ad60 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d309ec9414f187be server_cookie=4ab46bd7eb5b1b8 in_seq=0 out_seq=0 2024-03-28T01:08:16.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.587+0000 7fdd96ff5640 10 client.17643 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:16.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.587+0000 7fdd96ff5640 1 -- 172.21.15.169:0/419854363 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x562acd873c60 con 0x562acd978940 2024-03-28T01:08:16.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.587+0000 7fdd96ff5640 10 client.17643 handle_command_reply: tid=0 2024-03-28T01:08:16.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.587+0000 7fddbbf7c740 1 client.17643 shutdown 2024-03-28T01:08:16.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.587+0000 7fddbbf7c740 1 -- 172.21.15.169:0/419854363 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd907790 msgr2=0x562acd916160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:16.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.587+0000 7fddbbf7c740 1 --2- 172.21.15.169:0/419854363 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd907790 0x562acd916160 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fdda8013040 tx=0x7fdda8009b70 comp rx=0 tx=0).stop 2024-03-28T01:08:16.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.587+0000 7fddbbf7c740 1 -- 172.21.15.169:0/419854363 shutdown_connections 2024-03-28T01:08:16.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.587+0000 7fddbbf7c740 1 --2- 172.21.15.169:0/419854363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562acd90c900 0x562acd917170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.587+0000 7fddbbf7c740 1 --2- 172.21.15.169:0/419854363 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x562acd978940 0x562acd97ad60 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.587+0000 7fddbbf7c740 1 --2- 172.21.15.169:0/419854363 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x562acd907790 0x562acd916160 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.587+0000 7fddbbf7c740 1 --2- 172.21.15.169:0/419854363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562acd906da0 0x562acd915c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.587+0000 7fdd96ff5640 0 client.17643 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:16.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.587+0000 7fddbbf7c740 1 -- 172.21.15.169:0/419854363 >> 172.21.15.169:0/419854363 conn(0x562acd902310 msgr2=0x562acd908bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:16.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.587+0000 7fddbbf7c740 1 -- 172.21.15.169:0/419854363 shutdown_connections 2024-03-28T01:08:16.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.587+0000 7fddbbf7c740 1 -- 172.21.15.169:0/419854363 wait complete. 2024-03-28T01:08:16.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.587+0000 7fddbbf7c740 20 client.17643 trim_cache size 0 max 16384 2024-03-28T01:08:16.598 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:08:16.598 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T01:08:16.598 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": { 2024-03-28T01:08:16.598 INFO:teuthology.orchestra.run.smithi169.stdout: "bdf1aae6-278d-4d20-9ff2-1169a5fd3b02": { 2024-03-28T01:08:16.598 INFO:teuthology.orchestra.run.smithi169.stdout: "path": "/", 2024-03-28T01:08:16.598 INFO:teuthology.orchestra.run.smithi169.stdout: "tag": "bdf1aae6-278d-4d20-9ff2-1169a5fd3b02", 2024-03-28T01:08:16.598 INFO:teuthology.orchestra.run.smithi169.stdout: "options": "recursive,force" 2024-03-28T01:08:16.599 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:16.599 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:16.599 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:08:16.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.588+0000 7fddb92b2640 1 -- 172.21.15.169:0/2121609816 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdd9c0a2800 msgr2=0x7fdd9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:16.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.588+0000 7fddb92b2640 1 --2- 172.21.15.169:0/2121609816 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdd9c0a2800 0x7fdd9c0a4cc0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fddac0c5500 tx=0x7fddac039bb0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.588+0000 7fddb92b2640 1 -- 172.21.15.169:0/2121609816 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fddb4102150 msgr2=0x7fddb419ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:16.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.588+0000 7fddb92b2640 1 --2- 172.21.15.169:0/2121609816 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fddb4102150 0x7fddb419ecc0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fddac037040 tx=0x7fddac032f40 comp rx=0 tx=0).stop 2024-03-28T01:08:16.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.588+0000 7fddb92b2640 1 -- 172.21.15.169:0/2121609816 shutdown_connections 2024-03-28T01:08:16.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.588+0000 7fddb92b2640 1 --2- 172.21.15.169:0/2121609816 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fdd9c0a2800 0x7fdd9c0a4cc0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.588+0000 7fddb92b2640 1 --2- 172.21.15.169:0/2121609816 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fddb4102150 0x7fddb419ecc0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.588+0000 7fddb92b2640 1 --2- 172.21.15.169:0/2121609816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddb4101790 0x7fddb419c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.588+0000 7fddb92b2640 1 --2- 172.21.15.169:0/2121609816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddb41005a0 0x7fddb4069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:16.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.588+0000 7fddb92b2640 1 -- 172.21.15.169:0/2121609816 >> 172.21.15.169:0/2121609816 conn(0x7fddb40fbb30 msgr2=0x7fddb4104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:16.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.588+0000 7fddb92b2640 1 -- 172.21.15.169:0/2121609816 shutdown_connections 2024-03-28T01:08:16.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:16.588+0000 7fddb92b2640 1 -- 172.21.15.169:0/2121609816 wait complete. 2024-03-28T01:08:16.602 INFO:tasks.cephfs.filesystem:scrub status for tag:bdf1aae6-278d-4d20-9ff2-1169a5fd3b02 - {'path': '/', 'tag': 'bdf1aae6-278d-4d20-9ff2-1169a5fd3b02', 'options': 'recursive,force'} 2024-03-28T01:08:17.844 DEBUG:teuthology.parallel:result is None 2024-03-28T01:08:17.844 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-28T01:08:17.880 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-28T01:08:17.881 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-28T01:08:17.892 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi169.front.sepia.ceph.com 2024-03-28T01:08:17.927 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:17 smithi169 ceph-mon[32274]: scrub queued for path: / 2024-03-28T01:08:17.928 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:17 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:08:17.928 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:17 smithi169 ceph-mon[32274]: scrub summary: active paths [/] 2024-03-28T01:08:17.928 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:17 smithi169 ceph-mon[32274]: scrub summary: idle+waiting paths [/] 2024-03-28T01:08:17.928 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:17 smithi169 ceph-mon[32274]: pgmap v704: 129 pgs: 129 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 804 MiB/s wr, 1.45k op/s 2024-03-28T01:08:17.929 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-28T01:08:17.938 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-28T01:08:18.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:17 smithi174 ceph-mon[36060]: scrub queued for path: / 2024-03-28T01:08:18.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:17 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:08:18.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:17 smithi174 ceph-mon[36060]: scrub summary: active paths [/] 2024-03-28T01:08:18.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:17 smithi174 ceph-mon[36060]: scrub summary: idle+waiting paths [/] 2024-03-28T01:08:18.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:17 smithi174 ceph-mon[36060]: pgmap v704: 129 pgs: 129 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 804 MiB/s wr, 1.45k op/s 2024-03-28T01:08:18.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:17 smithi177 ceph-mon[36206]: scrub queued for path: / 2024-03-28T01:08:18.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:17 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:08:18.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:17 smithi177 ceph-mon[36206]: scrub summary: active paths [/] 2024-03-28T01:08:18.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:17 smithi177 ceph-mon[36206]: scrub summary: idle+waiting paths [/] 2024-03-28T01:08:18.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:17 smithi177 ceph-mon[36206]: pgmap v704: 129 pgs: 129 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 804 MiB/s wr, 1.45k op/s 2024-03-28T01:08:20.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:19 smithi169 ceph-mon[32274]: pgmap v705: 129 pgs: 129 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 576 MiB/s wr, 1.26k op/s 2024-03-28T01:08:20.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:19 smithi169 ceph-mon[32274]: scrub summary: idle 2024-03-28T01:08:20.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:19 smithi174 ceph-mon[36060]: pgmap v705: 129 pgs: 129 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 576 MiB/s wr, 1.26k op/s 2024-03-28T01:08:20.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:19 smithi174 ceph-mon[36060]: scrub summary: idle 2024-03-28T01:08:20.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:19 smithi177 ceph-mon[36206]: pgmap v705: 129 pgs: 129 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 576 MiB/s wr, 1.26k op/s 2024-03-28T01:08:20.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:19 smithi177 ceph-mon[36206]: scrub summary: idle 2024-03-28T01:08:22.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:21 smithi169 ceph-mon[32274]: pgmap v706: 129 pgs: 129 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 539 MiB/s wr, 1.23k op/s 2024-03-28T01:08:22.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:21 smithi174 ceph-mon[36060]: pgmap v706: 129 pgs: 129 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 539 MiB/s wr, 1.23k op/s 2024-03-28T01:08:22.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:21 smithi177 ceph-mon[36206]: pgmap v706: 129 pgs: 129 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 539 MiB/s wr, 1.23k op/s 2024-03-28T01:08:24.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:23 smithi169 ceph-mon[32274]: pgmap v707: 129 pgs: 129 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 499 MiB/s wr, 1.15k op/s 2024-03-28T01:08:24.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:23 smithi174 ceph-mon[36060]: pgmap v707: 129 pgs: 129 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 499 MiB/s wr, 1.15k op/s 2024-03-28T01:08:24.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:23 smithi177 ceph-mon[36206]: pgmap v707: 129 pgs: 129 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 499 MiB/s wr, 1.15k op/s 2024-03-28T01:08:26.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:25 smithi174 ceph-mon[36060]: pgmap v708: 129 pgs: 129 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 263 MiB/s wr, 513 op/s 2024-03-28T01:08:26.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:25 smithi177 ceph-mon[36206]: pgmap v708: 129 pgs: 129 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 263 MiB/s wr, 513 op/s 2024-03-28T01:08:26.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:25 smithi169 ceph-mon[32274]: pgmap v708: 129 pgs: 129 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 263 MiB/s wr, 513 op/s 2024-03-28T01:08:28.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:27 smithi174 ceph-mon[36060]: pgmap v709: 129 pgs: 129 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 281 MiB/s wr, 549 op/s 2024-03-28T01:08:28.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:27 smithi177 ceph-mon[36206]: pgmap v709: 129 pgs: 129 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 281 MiB/s wr, 549 op/s 2024-03-28T01:08:28.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:27 smithi169 ceph-mon[32274]: pgmap v709: 129 pgs: 129 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 281 MiB/s wr, 549 op/s 2024-03-28T01:08:30.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:29 smithi174 ceph-mon[36060]: pgmap v710: 129 pgs: 129 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 77 MiB/s wr, 160 op/s 2024-03-28T01:08:30.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:29 smithi177 ceph-mon[36206]: pgmap v710: 129 pgs: 129 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 77 MiB/s wr, 160 op/s 2024-03-28T01:08:30.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:29 smithi169 ceph-mon[32274]: pgmap v710: 129 pgs: 129 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 77 MiB/s wr, 160 op/s 2024-03-28T01:08:32.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:31 smithi174 ceph-mon[36060]: pgmap v711: 129 pgs: 129 active+clean; 9.8 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 70 MiB/s wr, 152 op/s 2024-03-28T01:08:32.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:31 smithi177 ceph-mon[36206]: pgmap v711: 129 pgs: 129 active+clean; 9.8 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 70 MiB/s wr, 152 op/s 2024-03-28T01:08:32.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:31 smithi169 ceph-mon[32274]: pgmap v711: 129 pgs: 129 active+clean; 9.8 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 70 MiB/s wr, 152 op/s 2024-03-28T01:08:34.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:33 smithi174 ceph-mon[36060]: pgmap v712: 129 pgs: 129 active+clean; 8.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 86 MiB/s wr, 221 op/s 2024-03-28T01:08:34.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:33 smithi177 ceph-mon[36206]: pgmap v712: 129 pgs: 129 active+clean; 8.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 86 MiB/s wr, 221 op/s 2024-03-28T01:08:34.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:33 smithi169 ceph-mon[32274]: pgmap v712: 129 pgs: 129 active+clean; 8.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 86 MiB/s wr, 221 op/s 2024-03-28T01:08:36.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:35 smithi174 ceph-mon[36060]: pgmap v713: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 64 MiB/s wr, 175 op/s 2024-03-28T01:08:36.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:35 smithi177 ceph-mon[36206]: pgmap v713: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 64 MiB/s wr, 175 op/s 2024-03-28T01:08:36.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:35 smithi169 ceph-mon[32274]: pgmap v713: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 64 MiB/s wr, 175 op/s 2024-03-28T01:08:38.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:37 smithi174 ceph-mon[36060]: pgmap v714: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 88 MiB/s wr, 222 op/s 2024-03-28T01:08:38.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:37 smithi177 ceph-mon[36206]: pgmap v714: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 88 MiB/s wr, 222 op/s 2024-03-28T01:08:38.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:37 smithi169 ceph-mon[32274]: pgmap v714: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 88 MiB/s wr, 222 op/s 2024-03-28T01:08:40.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:39 smithi174 ceph-mon[36060]: pgmap v715: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 63 MiB/s wr, 170 op/s 2024-03-28T01:08:40.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:39 smithi177 ceph-mon[36206]: pgmap v715: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 63 MiB/s wr, 170 op/s 2024-03-28T01:08:40.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:39 smithi169 ceph-mon[32274]: pgmap v715: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 63 MiB/s wr, 170 op/s 2024-03-28T01:08:42.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:41 smithi174 ceph-mon[36060]: pgmap v716: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 64 MiB/s wr, 171 op/s 2024-03-28T01:08:42.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:41 smithi177 ceph-mon[36206]: pgmap v716: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 64 MiB/s wr, 171 op/s 2024-03-28T01:08:42.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:41 smithi169 ceph-mon[32274]: pgmap v716: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 64 MiB/s wr, 171 op/s 2024-03-28T01:08:44.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:43 smithi174 ceph-mon[36060]: pgmap v717: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 74 MiB/s wr, 253 op/s 2024-03-28T01:08:44.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:43 smithi177 ceph-mon[36206]: pgmap v717: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 74 MiB/s wr, 253 op/s 2024-03-28T01:08:44.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:43 smithi169 ceph-mon[32274]: pgmap v717: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 74 MiB/s wr, 253 op/s 2024-03-28T01:08:46.103 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:45 smithi169 ceph-mon[32274]: pgmap v718: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 54 MiB/s wr, 185 op/s 2024-03-28T01:08:46.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:45 smithi174 ceph-mon[36060]: pgmap v718: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 54 MiB/s wr, 185 op/s 2024-03-28T01:08:46.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:45 smithi177 ceph-mon[36206]: pgmap v718: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 54 MiB/s wr, 185 op/s 2024-03-28T01:08:46.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T01:08:46.604 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T01:08:46.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.710+0000 7fb3d49c6640 1 -- 172.21.15.169:0/3835593985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3d0103260 msgr2=0x7fb3d0109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:46.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.710+0000 7fb3d49c6640 1 --2- 172.21.15.169:0/3835593985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3d0103260 0x7fb3d0109af0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8009ca0 tx=0x7fb3c802c0f0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.710+0000 7fb3d49c6640 1 -- 172.21.15.169:0/3835593985 shutdown_connections 2024-03-28T01:08:46.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.710+0000 7fb3d49c6640 1 --2- 172.21.15.169:0/3835593985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3d0103260 0x7fb3d0109af0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.710+0000 7fb3d49c6640 1 --2- 172.21.15.169:0/3835593985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3d01028a0 0x7fb3d0102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.710+0000 7fb3d49c6640 1 --2- 172.21.15.169:0/3835593985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3d01016b0 0x7fb3d0101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.710+0000 7fb3d49c6640 1 -- 172.21.15.169:0/3835593985 >> 172.21.15.169:0/3835593985 conn(0x7fb3d00fcc40 msgr2=0x7fb3d00ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:46.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.710+0000 7fb3d49c6640 1 -- 172.21.15.169:0/3835593985 shutdown_connections 2024-03-28T01:08:46.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.710+0000 7fb3d49c6640 1 -- 172.21.15.169:0/3835593985 wait complete. 2024-03-28T01:08:46.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.711+0000 7fb3d49c6640 1 Processor -- start 2024-03-28T01:08:46.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.711+0000 7fb3d49c6640 1 -- start start 2024-03-28T01:08:46.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.711+0000 7fb3d49c6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3d01016b0 0x7fb3d0075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.711+0000 7fb3d49c6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3d01028a0 0x7fb3d0073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.711+0000 7fb3d49c6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3d0103260 0x7fb3d0073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.711+0000 7fb3d49c6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d010d330 con 0x7fb3d0103260 2024-03-28T01:08:46.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.711+0000 7fb3d49c6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d010d1b0 con 0x7fb3d01016b0 2024-03-28T01:08:46.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.711+0000 7fb3d49c6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d010d4b0 con 0x7fb3d01028a0 2024-03-28T01:08:46.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.711+0000 7fb3ced9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3d0103260 0x7fb3d0073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.711+0000 7fb3ced9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3d0103260 0x7fb3d0073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53248/0 (socket says 172.21.15.169:53248) 2024-03-28T01:08:46.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.711+0000 7fb3ced9d640 1 -- 172.21.15.169:0/3648375935 learned_addr learned my addr 172.21.15.169:0/3648375935 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:46.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.712+0000 7fb3cdd9b640 1 --2- 172.21.15.169:0/3648375935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3d01028a0 0x7fb3d0073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.712+0000 7fb3ce59c640 1 --2- 172.21.15.169:0/3648375935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3d01016b0 0x7fb3d0075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.712+0000 7fb3ced9d640 1 -- 172.21.15.169:0/3648375935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3d01028a0 msgr2=0x7fb3d0073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:46.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.712+0000 7fb3ced9d640 1 --2- 172.21.15.169:0/3648375935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3d01028a0 0x7fb3d0073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.712+0000 7fb3ced9d640 1 -- 172.21.15.169:0/3648375935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3d01016b0 msgr2=0x7fb3d0075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:46.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.712+0000 7fb3ced9d640 1 --2- 172.21.15.169:0/3648375935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3d01016b0 0x7fb3d0075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.712+0000 7fb3ced9d640 1 -- 172.21.15.169:0/3648375935 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d0074860 con 0x7fb3d0103260 2024-03-28T01:08:46.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.712+0000 7fb3ced9d640 1 --2- 172.21.15.169:0/3648375935 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3d0103260 0x7fb3d0073fd0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8037040 tx=0x7fb3c8032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:46.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.712+0000 7fb3bf7fe640 1 -- 172.21.15.169:0/3648375935 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8041020 con 0x7fb3d0103260 2024-03-28T01:08:46.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.712+0000 7fb3bf7fe640 1 -- 172.21.15.169:0/3648375935 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3c8034cd0 con 0x7fb3d0103260 2024-03-28T01:08:46.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.712+0000 7fb3d49c6640 1 -- 172.21.15.169:0/3648375935 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d01a1a40 con 0x7fb3d0103260 2024-03-28T01:08:46.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.712+0000 7fb3bf7fe640 1 -- 172.21.15.169:0/3648375935 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8039040 con 0x7fb3d0103260 2024-03-28T01:08:46.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.712+0000 7fb3d49c6640 1 -- 172.21.15.169:0/3648375935 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d01a1f90 con 0x7fb3d0103260 2024-03-28T01:08:46.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.714+0000 7fb3bf7fe640 1 -- 172.21.15.169:0/3648375935 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb3c8044020 con 0x7fb3d0103260 2024-03-28T01:08:46.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.715+0000 7fb3d75b4740 1 Processor -- start 2024-03-28T01:08:46.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.715+0000 7fb3d75b4740 1 -- start start 2024-03-28T01:08:46.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.715+0000 7fb3d75b4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e273da0 0x561e2e2741c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.715+0000 7fb3d75b4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e274790 0x561e2e279290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.715+0000 7fb3d75b4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e279900 0x561e2e27bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.715+0000 7fb3d75b4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e2e1d93d0 con 0x561e2e279900 2024-03-28T01:08:46.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.715+0000 7fb3d75b4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e2e181ad0 con 0x561e2e274790 2024-03-28T01:08:46.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.715+0000 7fb3d75b4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e2e1d5470 con 0x561e2e273da0 2024-03-28T01:08:46.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.715+0000 7fb3bf7fe640 1 --2- 172.21.15.169:0/3648375935 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb3b80a2800 0x7fb3b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.715+0000 7fb3ce59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e274790 0x561e2e279290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.715+0000 7fb3ce59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e274790 0x561e2e279290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:55242/0 (socket says 172.21.15.169:55242) 2024-03-28T01:08:46.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.715+0000 7fb3ce59c640 1 -- 172.21.15.169:0/2654036564 learned_addr learned my addr 172.21.15.169:0/2654036564 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:46.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.715+0000 7fb3cdd9b640 1 --2- 172.21.15.169:0/2654036564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e273da0 0x561e2e2741c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.715+0000 7fb3ced9d640 1 --2- 172.21.15.169:0/3648375935 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb3b80a2800 0x7fb3b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.715+0000 7fb3cdd9b640 1 --2- 172.21.15.169:0/2654036564 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e279900 0x561e2e27bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.715+0000 7fb3bf7fe640 1 -- 172.21.15.169:0/3648375935 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb3c80f0d00 con 0x7fb3d0103260 2024-03-28T01:08:46.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.716+0000 7fb3ce59c640 1 -- 172.21.15.169:0/2654036564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e273da0 msgr2=0x561e2e2741c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:46.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.716+0000 7fb3ce59c640 1 --2- 172.21.15.169:0/2654036564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e273da0 0x561e2e2741c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.716+0000 7fb3ce59c640 1 -- 172.21.15.169:0/2654036564 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e279900 msgr2=0x561e2e27bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:46.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.716+0000 7fb3ce59c640 1 --2- 172.21.15.169:0/2654036564 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e279900 0x561e2e27bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.716+0000 7fb3ce59c640 1 -- 172.21.15.169:0/2654036564 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e2dfb77d0 con 0x561e2e274790 2024-03-28T01:08:46.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.716+0000 7fb3cdd9b640 1 --2- 172.21.15.169:0/2654036564 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e279900 0x561e2e27bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T01:08:46.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.716+0000 7fb3ce59c640 1 --2- 172.21.15.169:0/2654036564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e274790 0x561e2e279290 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0009ca0 tx=0x7fb3c0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4646cf930cb1882d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:46.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.716+0000 7fb3ced9d640 1 --2- 172.21.15.169:0/3648375935 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb3b80a2800 0x7fb3b80a4cc0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8032420 tx=0x7fb3c80b2810 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:46.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.716+0000 7fb3bcff9640 1 -- 172.21.15.169:0/2654036564 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0004220 con 0x561e2e274790 2024-03-28T01:08:46.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.716+0000 7fb3bcff9640 1 -- 172.21.15.169:0/2654036564 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3c00043c0 con 0x561e2e274790 2024-03-28T01:08:46.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.716+0000 7fb3bcff9640 1 -- 172.21.15.169:0/2654036564 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0019930 con 0x561e2e274790 2024-03-28T01:08:46.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.717+0000 7fb3d75b4740 1 -- 172.21.15.169:0/2654036564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e274790 msgr2=0x561e2e279290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:46.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.717+0000 7fb3d75b4740 1 --2- 172.21.15.169:0/2654036564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e274790 0x561e2e279290 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0009ca0 tx=0x7fb3c0011810 comp rx=0 tx=0).stop 2024-03-28T01:08:46.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.717+0000 7fb3d75b4740 1 -- 172.21.15.169:0/2654036564 shutdown_connections 2024-03-28T01:08:46.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.717+0000 7fb3d75b4740 1 --2- 172.21.15.169:0/2654036564 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e279900 0x561e2e27bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.717+0000 7fb3d75b4740 1 --2- 172.21.15.169:0/2654036564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e274790 0x561e2e279290 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.717+0000 7fb3d75b4740 1 --2- 172.21.15.169:0/2654036564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e273da0 0x561e2e2741c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.717+0000 7fb3d75b4740 1 -- 172.21.15.169:0/2654036564 >> 172.21.15.169:0/2654036564 conn(0x561e2e26f310 msgr2=0x561e2e26f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:46.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.717+0000 7fb3d75b4740 1 -- 172.21.15.169:0/2654036564 shutdown_connections 2024-03-28T01:08:46.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.717+0000 7fb3d75b4740 1 -- 172.21.15.169:0/2654036564 wait complete. 2024-03-28T01:08:46.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.717+0000 7fb3d75b4740 1 -- start start 2024-03-28T01:08:46.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.717+0000 7fb3d75b4740 1 Processor -- start 2024-03-28T01:08:46.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.717+0000 7fb3d75b4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e273da0 0x561e2e27e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.717+0000 7fb3d75b4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e274790 0x561e2e27e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.717+0000 7fb3d75b4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e279900 0x561e2e2857d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.717+0000 7fb3d75b4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e2e27c8b0 con 0x561e2e279900 2024-03-28T01:08:46.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.717+0000 7fb3d75b4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e2e195060 con 0x561e2e273da0 2024-03-28T01:08:46.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.717+0000 7fb3d75b4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e2e27ca30 con 0x561e2e274790 2024-03-28T01:08:46.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.717+0000 7fb3d75b4740 10 client.0 _ll_register_callbacks cb 0x561e2e1776a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:46.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.718+0000 7fb3cdd9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e279900 0x561e2e2857d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.718+0000 7fb3ce59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e274790 0x561e2e27e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.718+0000 7fb3cdd9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e279900 0x561e2e2857d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53270/0 (socket says 172.21.15.169:53270) 2024-03-28T01:08:46.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.718+0000 7fb3ce59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e274790 0x561e2e27e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48398/0 (socket says 172.21.15.169:48398) 2024-03-28T01:08:46.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.718+0000 7fb3cdd9b640 1 -- 172.21.15.169:0/1031102283 learned_addr learned my addr 172.21.15.169:0/1031102283 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:46.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.718+0000 7fb3cdd9b640 1 --2- 172.21.15.169:0/1031102283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e273da0 0x561e2e27e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.718+0000 7fb3cdd9b640 1 -- 172.21.15.169:0/1031102283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e274790 msgr2=0x561e2e27e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:46.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.718+0000 7fb3cdd9b640 1 --2- 172.21.15.169:0/1031102283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e274790 0x561e2e27e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.718+0000 7fb3cdd9b640 1 -- 172.21.15.169:0/1031102283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e273da0 msgr2=0x561e2e27e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:46.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.718+0000 7fb3cdd9b640 1 --2- 172.21.15.169:0/1031102283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e273da0 0x561e2e27e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.718+0000 7fb3cdd9b640 1 -- 172.21.15.169:0/1031102283 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e2deed3f0 con 0x561e2e279900 2024-03-28T01:08:46.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.718+0000 7fb3ce59c640 1 --2- 172.21.15.169:0/1031102283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e274790 0x561e2e27e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:08:46.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.718+0000 7fb3cdd9b640 1 --2- 172.21.15.169:0/1031102283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e279900 0x561e2e2857d0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4011e00 tx=0x7fb3c4009aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3c3bf9621a2ef89 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:46.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.718+0000 7fb3b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:08:46.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.718+0000 7fb3b5ffb640 1 -- 172.21.15.169:0/1031102283 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4004260 con 0x561e2e279900 2024-03-28T01:08:46.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.718+0000 7fb3b5ffb640 1 -- 172.21.15.169:0/1031102283 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb3c4009040 con 0x561e2e279900 2024-03-28T01:08:46.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.718+0000 7fb3d75b4740 1 -- 172.21.15.169:0/1031102283 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb3c00043c0 con 0x561e2e279900 2024-03-28T01:08:46.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.718+0000 7fb3b5ffb640 1 -- 172.21.15.169:0/1031102283 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3c4004400 con 0x561e2e279900 2024-03-28T01:08:46.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.718+0000 7fb3d75b4740 10 client.17655 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:46.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.718+0000 7fb3d75b4740 1 -- 172.21.15.169:0/1031102283 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561e2dead0f0 con 0x561e2e279900 2024-03-28T01:08:46.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.719+0000 7fb3b5ffb640 1 -- 172.21.15.169:0/1031102283 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3c00043c0 con 0x561e2e279900 2024-03-28T01:08:46.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.719+0000 7fb3b5ffb640 1 -- 172.21.15.169:0/1031102283 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb3c4020040 con 0x561e2e279900 2024-03-28T01:08:46.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.719+0000 7fb3d75b4740 10 client.17655 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:46.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.719+0000 7fb3d75b4740 10 client.17655 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:08:46.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.719+0000 7fb3d75b4740 20 client.17655 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:46.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.719+0000 7fb3d75b4740 1 --2- 172.21.15.169:0/1031102283 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561e2e289ce0 0x561e2e28c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.719+0000 7fb3d75b4740 4 client.17655 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:08:46.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.719+0000 7fb3d75b4740 1 -- 172.21.15.169:0/1031102283 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561e2e1e6910 con 0x561e2e289ce0 2024-03-28T01:08:46.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.719+0000 7fb3ce59c640 1 --2- 172.21.15.169:0/1031102283 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561e2e289ce0 0x561e2e28c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.719+0000 7fb3b5ffb640 10 client.17655 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:46.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.719+0000 7fb3ce59c640 1 --2- 172.21.15.169:0/1031102283 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561e2e289ce0 0x561e2e28c140 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cf84f3f90551848 server_cookie=93668c4f15735e93 in_seq=0 out_seq=0 2024-03-28T01:08:46.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.720+0000 7fb3b5ffb640 1 -- 172.21.15.169:0/1031102283 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561e2e1e6910 con 0x561e2e289ce0 2024-03-28T01:08:46.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.720+0000 7fb3b5ffb640 10 client.17655 handle_command_reply: tid=0 2024-03-28T01:08:46.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.720+0000 7fb3d75b4740 1 client.17655 shutdown 2024-03-28T01:08:46.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.720+0000 7fb3d75b4740 1 -- 172.21.15.169:0/1031102283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e279900 msgr2=0x561e2e2857d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:46.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.720+0000 7fb3d75b4740 1 --2- 172.21.15.169:0/1031102283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e279900 0x561e2e2857d0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4011e00 tx=0x7fb3c4009aa0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.720+0000 7fb3d75b4740 1 -- 172.21.15.169:0/1031102283 shutdown_connections 2024-03-28T01:08:46.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.720+0000 7fb3d75b4740 1 --2- 172.21.15.169:0/1031102283 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561e2e289ce0 0x561e2e28c140 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.720+0000 7fb3d75b4740 1 --2- 172.21.15.169:0/1031102283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e279900 0x561e2e2857d0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.720+0000 7fb3d75b4740 1 --2- 172.21.15.169:0/1031102283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e274790 0x561e2e27e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.720+0000 7fb3b5ffb640 0 client.17655 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:46.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.720+0000 7fb3d75b4740 1 --2- 172.21.15.169:0/1031102283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e273da0 0x561e2e27e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.720+0000 7fb3d75b4740 1 -- 172.21.15.169:0/1031102283 >> 172.21.15.169:0/1031102283 conn(0x561e2e26f310 msgr2=0x561e2e26e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:46.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.720+0000 7fb3d75b4740 1 -- 172.21.15.169:0/1031102283 shutdown_connections 2024-03-28T01:08:46.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.720+0000 7fb3d75b4740 1 -- 172.21.15.169:0/1031102283 wait complete. 2024-03-28T01:08:46.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.721+0000 7fb3d75b4740 20 client.17655 trim_cache size 0 max 16384 2024-03-28T01:08:46.745 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.737+0000 7fb3d75b4740 1 Processor -- start 2024-03-28T01:08:46.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.737+0000 7fb3d75b4740 1 -- start start 2024-03-28T01:08:46.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.737+0000 7fb3d75b4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e273da0 0x561e2e26dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.737+0000 7fb3d75b4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e274790 0x561e2e285790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.737+0000 7fb3d75b4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e279900 0x561e2e26e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.737+0000 7fb3d75b4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d01ae8b0 con 0x561e2e274790 2024-03-28T01:08:46.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.737+0000 7fb3d75b4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d01ac3b0 con 0x561e2e273da0 2024-03-28T01:08:46.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.737+0000 7fb3d75b4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d01aa450 con 0x561e2e279900 2024-03-28T01:08:46.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.737+0000 7fb3ce59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e274790 0x561e2e285790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.737+0000 7fb3ce59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e274790 0x561e2e285790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53276/0 (socket says 172.21.15.169:53276) 2024-03-28T01:08:46.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.737+0000 7fb3ce59c640 1 -- 172.21.15.169:0/3163269357 learned_addr learned my addr 172.21.15.169:0/3163269357 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:46.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.737+0000 7fb3cdd9b640 1 --2- 172.21.15.169:0/3163269357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e273da0 0x561e2e26dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.737+0000 7fb3cdd9b640 1 --2- 172.21.15.169:0/3163269357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e279900 0x561e2e26e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.737+0000 7fb3ce59c640 1 -- 172.21.15.169:0/3163269357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e279900 msgr2=0x561e2e26e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:46.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.737+0000 7fb3ce59c640 1 --2- 172.21.15.169:0/3163269357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e279900 0x561e2e26e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.737+0000 7fb3ce59c640 1 -- 172.21.15.169:0/3163269357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e273da0 msgr2=0x561e2e26dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:46.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.737+0000 7fb3ce59c640 1 --2- 172.21.15.169:0/3163269357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e273da0 0x561e2e26dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.737+0000 7fb3ce59c640 1 -- 172.21.15.169:0/3163269357 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e2e26f140 con 0x561e2e274790 2024-03-28T01:08:46.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.737+0000 7fb3ce59c640 1 --2- 172.21.15.169:0/3163269357 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e274790 0x561e2e285790 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0013010 tx=0x7fb3c00096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c751b86c716aef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:46.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.738+0000 7fb3b7fff640 1 -- 172.21.15.169:0/3163269357 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0030070 con 0x561e2e274790 2024-03-28T01:08:46.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.738+0000 7fb3b7fff640 1 -- 172.21.15.169:0/3163269357 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3c002c2d0 con 0x561e2e274790 2024-03-28T01:08:46.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.738+0000 7fb3b7fff640 1 -- 172.21.15.169:0/3163269357 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0003a30 con 0x561e2e274790 2024-03-28T01:08:46.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.738+0000 7fb3d75b4740 1 -- 172.21.15.169:0/3163269357 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e274790 msgr2=0x561e2e285790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:46.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.738+0000 7fb3d75b4740 1 --2- 172.21.15.169:0/3163269357 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e274790 0x561e2e285790 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0013010 tx=0x7fb3c00096a0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.738+0000 7fb3d75b4740 1 -- 172.21.15.169:0/3163269357 shutdown_connections 2024-03-28T01:08:46.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.738+0000 7fb3d75b4740 1 --2- 172.21.15.169:0/3163269357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e279900 0x561e2e26e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.738+0000 7fb3d75b4740 1 --2- 172.21.15.169:0/3163269357 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e274790 0x561e2e285790 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.738+0000 7fb3d75b4740 1 --2- 172.21.15.169:0/3163269357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e273da0 0x561e2e26dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.738+0000 7fb3d75b4740 1 -- 172.21.15.169:0/3163269357 >> 172.21.15.169:0/3163269357 conn(0x561e2e26f310 msgr2=0x561e2e275c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:46.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.738+0000 7fb3d75b4740 1 -- 172.21.15.169:0/3163269357 shutdown_connections 2024-03-28T01:08:46.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.738+0000 7fb3d75b4740 1 -- 172.21.15.169:0/3163269357 wait complete. 2024-03-28T01:08:46.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.739+0000 7fb3d75b4740 1 -- start start 2024-03-28T01:08:46.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.739+0000 7fb3d75b4740 1 Processor -- start 2024-03-28T01:08:46.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.739+0000 7fb3d75b4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e273da0 0x561e2e283440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.739+0000 7fb3d75b4740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e274790 0x561e2e283980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.739+0000 7fb3d75b4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e279900 0x561e2e284990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.739+0000 7fb3d75b4740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d01aaa50 con 0x561e2e274790 2024-03-28T01:08:46.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.739+0000 7fb3d75b4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d01abd50 con 0x561e2e273da0 2024-03-28T01:08:46.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.739+0000 7fb3d75b4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d01ab090 con 0x561e2e279900 2024-03-28T01:08:46.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.739+0000 7fb3d75b4740 10 client.0 _ll_register_callbacks cb 0x7fb3c4009dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:46.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.739+0000 7fb3ce59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e274790 0x561e2e283980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.739+0000 7fb3cdd9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e273da0 0x561e2e283440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.739+0000 7fb3ce59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e274790 0x561e2e283980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53290/0 (socket says 172.21.15.169:53290) 2024-03-28T01:08:46.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.739+0000 7fb3ce59c640 1 -- 172.21.15.169:0/803163278 learned_addr learned my addr 172.21.15.169:0/803163278 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:46.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.739+0000 7fb3cdd9b640 1 --2- 172.21.15.169:0/803163278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e279900 0x561e2e284990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.739+0000 7fb3ce59c640 1 -- 172.21.15.169:0/803163278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e279900 msgr2=0x561e2e284990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:46.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.739+0000 7fb3ce59c640 1 --2- 172.21.15.169:0/803163278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e279900 0x561e2e284990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.739+0000 7fb3ce59c640 1 -- 172.21.15.169:0/803163278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e273da0 msgr2=0x561e2e283440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:46.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.739+0000 7fb3ce59c640 1 --2- 172.21.15.169:0/803163278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e273da0 0x561e2e283440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.739+0000 7fb3ce59c640 1 -- 172.21.15.169:0/803163278 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e2e284f80 con 0x561e2e274790 2024-03-28T01:08:46.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.739+0000 7fb3ce59c640 1 --2- 172.21.15.169:0/803163278 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e274790 0x561e2e283980 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0009a30 tx=0x7fb3c0011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc73437a14dc5178 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:46.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.739+0000 7fb3b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.169:3300/0 2024-03-28T01:08:46.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.740+0000 7fb3b5ffb640 1 -- 172.21.15.169:0/803163278 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c003b070 con 0x561e2e274790 2024-03-28T01:08:46.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.740+0000 7fb3b5ffb640 1 -- 172.21.15.169:0/803163278 <== mon.0 v2:172.21.15.169:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb3c0003b40 con 0x561e2e274790 2024-03-28T01:08:46.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.740+0000 7fb3d75b4740 1 -- 172.21.15.169:0/803163278 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb3c002c2d0 con 0x561e2e274790 2024-03-28T01:08:46.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.740+0000 7fb3b5ffb640 1 -- 172.21.15.169:0/803163278 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3c0030040 con 0x561e2e274790 2024-03-28T01:08:46.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.740+0000 7fb3d75b4740 10 client.17661 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:46.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.740+0000 7fb3d75b4740 1 -- 172.21.15.169:0/803163278 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561e2e2e5aa0 con 0x561e2e274790 2024-03-28T01:08:46.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.740+0000 7fb3b5ffb640 1 -- 172.21.15.169:0/803163278 <== mon.0 v2:172.21.15.169:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3c002c2d0 con 0x561e2e274790 2024-03-28T01:08:46.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.740+0000 7fb3b5ffb640 1 -- 172.21.15.169:0/803163278 <== mon.0 v2:172.21.15.169:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb3c002c890 con 0x561e2e274790 2024-03-28T01:08:46.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.740+0000 7fb3d75b4740 10 client.17661 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:46.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.740+0000 7fb3d75b4740 10 client.17661 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:08:46.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.740+0000 7fb3d75b4740 20 client.17661 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:46.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.740+0000 7fb3d75b4740 1 --2- 172.21.15.169:0/803163278 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561e2e2e61f0 0x561e2e2e8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.740+0000 7fb3d75b4740 4 client.17661 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T01:08:46.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.740+0000 7fb3d75b4740 1 -- 172.21.15.169:0/803163278 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561e2e1e0c60 con 0x561e2e2e61f0 2024-03-28T01:08:46.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.740+0000 7fb3cdd9b640 1 --2- 172.21.15.169:0/803163278 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561e2e2e61f0 0x561e2e2e8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.741+0000 7fb3b5ffb640 10 client.17661 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:46.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.741+0000 7fb3cdd9b640 1 --2- 172.21.15.169:0/803163278 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561e2e2e61f0 0x561e2e2e8600 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26461339ec18e753 server_cookie=68e4cd536737cf2 in_seq=0 out_seq=0 2024-03-28T01:08:46.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.741+0000 7fb3b5ffb640 1 -- 172.21.15.169:0/803163278 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561e2e1e0c60 con 0x561e2e2e61f0 2024-03-28T01:08:46.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.741+0000 7fb3b5ffb640 10 client.17661 handle_command_reply: tid=0 2024-03-28T01:08:46.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.741+0000 7fb3d75b4740 1 client.17661 shutdown 2024-03-28T01:08:46.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.741+0000 7fb3d75b4740 1 -- 172.21.15.169:0/803163278 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e274790 msgr2=0x561e2e283980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:46.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.741+0000 7fb3d75b4740 1 --2- 172.21.15.169:0/803163278 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e274790 0x561e2e283980 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0009a30 tx=0x7fb3c0011dc0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.741+0000 7fb3d75b4740 1 -- 172.21.15.169:0/803163278 shutdown_connections 2024-03-28T01:08:46.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.741+0000 7fb3d75b4740 1 --2- 172.21.15.169:0/803163278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561e2e279900 0x561e2e284990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.741+0000 7fb3d75b4740 1 --2- 172.21.15.169:0/803163278 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x561e2e2e61f0 0x561e2e2e8600 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.741+0000 7fb3d75b4740 1 --2- 172.21.15.169:0/803163278 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x561e2e274790 0x561e2e283980 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.741+0000 7fb3d75b4740 1 --2- 172.21.15.169:0/803163278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2e273da0 0x561e2e283440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.741+0000 7fb3b5ffb640 0 client.17661 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:46.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.741+0000 7fb3d75b4740 1 -- 172.21.15.169:0/803163278 >> 172.21.15.169:0/803163278 conn(0x561e2e26f310 msgr2=0x561e2e272570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:46.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.741+0000 7fb3d75b4740 1 -- 172.21.15.169:0/803163278 shutdown_connections 2024-03-28T01:08:46.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.741+0000 7fb3d75b4740 1 -- 172.21.15.169:0/803163278 wait complete. 2024-03-28T01:08:46.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.741+0000 7fb3d75b4740 20 client.17661 trim_cache size 0 max 16384 2024-03-28T01:08:46.751 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:08:46.752 INFO:teuthology.orchestra.run.smithi169.stdout: "status": "no active scrubs running", 2024-03-28T01:08:46.752 INFO:teuthology.orchestra.run.smithi169.stdout: "scrubs": {} 2024-03-28T01:08:46.752 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:08:46.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.741+0000 7fb3d49c6640 1 -- 172.21.15.169:0/3648375935 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb3b80a2800 msgr2=0x7fb3b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:46.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.741+0000 7fb3d49c6640 1 --2- 172.21.15.169:0/3648375935 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb3b80a2800 0x7fb3b80a4cc0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8032420 tx=0x7fb3c80b2810 comp rx=0 tx=0).stop 2024-03-28T01:08:46.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.741+0000 7fb3d49c6640 1 -- 172.21.15.169:0/3648375935 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3d0103260 msgr2=0x7fb3d0073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:46.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.741+0000 7fb3d49c6640 1 --2- 172.21.15.169:0/3648375935 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3d0103260 0x7fb3d0073fd0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8037040 tx=0x7fb3c8032f40 comp rx=0 tx=0).stop 2024-03-28T01:08:46.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.742+0000 7fb3d49c6640 1 -- 172.21.15.169:0/3648375935 shutdown_connections 2024-03-28T01:08:46.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.742+0000 7fb3d49c6640 1 --2- 172.21.15.169:0/3648375935 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb3b80a2800 0x7fb3b80a4cc0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.742+0000 7fb3d49c6640 1 --2- 172.21.15.169:0/3648375935 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb3d0103260 0x7fb3d0073fd0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.742+0000 7fb3d49c6640 1 --2- 172.21.15.169:0/3648375935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3d01028a0 0x7fb3d0073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.742+0000 7fb3d49c6640 1 --2- 172.21.15.169:0/3648375935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3d01016b0 0x7fb3d0075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.742+0000 7fb3d49c6640 1 -- 172.21.15.169:0/3648375935 >> 172.21.15.169:0/3648375935 conn(0x7fb3d00fcc40 msgr2=0x7fb3d0105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:46.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.742+0000 7fb3d49c6640 1 -- 172.21.15.169:0/3648375935 shutdown_connections 2024-03-28T01:08:46.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.742+0000 7fb3d49c6640 1 -- 172.21.15.169:0/3648375935 wait complete. 2024-03-28T01:08:46.756 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T01:08:46.756 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T01:08:46.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.855+0000 7f15c8787640 1 -- 172.21.15.169:0/2880562024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15c01005c0 msgr2=0x7f15c01009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:46.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.855+0000 7f15c8787640 1 --2- 172.21.15.169:0/2880562024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15c01005c0 0x7f15c01009e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f15bc009900 tx=0x7f15bc02bc30 comp rx=0 tx=0).stop 2024-03-28T01:08:46.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.855+0000 7f15c8787640 1 -- 172.21.15.169:0/2880562024 shutdown_connections 2024-03-28T01:08:46.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.855+0000 7f15c8787640 1 --2- 172.21.15.169:0/2880562024 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15c0102170 0x7f15c0108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.855+0000 7f15c8787640 1 --2- 172.21.15.169:0/2880562024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15c01017b0 0x7f15c0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.855+0000 7f15c8787640 1 --2- 172.21.15.169:0/2880562024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15c01005c0 0x7f15c01009e0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.855+0000 7f15c8787640 1 -- 172.21.15.169:0/2880562024 >> 172.21.15.169:0/2880562024 conn(0x7f15c00fbb30 msgr2=0x7f15c00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:46.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.855+0000 7f15c8787640 1 -- 172.21.15.169:0/2880562024 shutdown_connections 2024-03-28T01:08:46.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.855+0000 7f15c8787640 1 -- 172.21.15.169:0/2880562024 wait complete. 2024-03-28T01:08:46.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.856+0000 7f15c8787640 1 Processor -- start 2024-03-28T01:08:46.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.856+0000 7f15c8787640 1 -- start start 2024-03-28T01:08:46.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.856+0000 7f15c8787640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15c01005c0 0x7f15c01979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.856+0000 7f15c8787640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15c01017b0 0x7f15c0197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.856+0000 7f15c8787640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15c0102170 0x7f15c019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.856+0000 7f15c8787640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c010d360 con 0x7f15c0102170 2024-03-28T01:08:46.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.856+0000 7f15c8787640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c010d1e0 con 0x7f15c01005c0 2024-03-28T01:08:46.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.856+0000 7f15c8787640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c010d4e0 con 0x7f15c01017b0 2024-03-28T01:08:46.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.856+0000 7f15c6d24640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15c0102170 0x7f15c019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.856+0000 7f15c6d24640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15c0102170 0x7f15c019ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53304/0 (socket says 172.21.15.169:53304) 2024-03-28T01:08:46.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.856+0000 7f15c6523640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15c01005c0 0x7f15c01979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.856+0000 7f15c6d24640 1 -- 172.21.15.169:0/3631918448 learned_addr learned my addr 172.21.15.169:0/3631918448 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:46.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.856+0000 7f15c5d22640 1 --2- 172.21.15.169:0/3631918448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15c01017b0 0x7f15c0197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.856+0000 7f15c6d24640 1 -- 172.21.15.169:0/3631918448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15c01017b0 msgr2=0x7f15c0197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:46.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.856+0000 7f15c6d24640 1 --2- 172.21.15.169:0/3631918448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15c01017b0 0x7f15c0197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.856+0000 7f15c6d24640 1 -- 172.21.15.169:0/3631918448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15c01005c0 msgr2=0x7f15c01979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:46.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.856+0000 7f15c6d24640 1 --2- 172.21.15.169:0/3631918448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15c01005c0 0x7f15c01979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:46.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.856+0000 7f15c6d24640 1 -- 172.21.15.169:0/3631918448 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c019f600 con 0x7f15c0102170 2024-03-28T01:08:46.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.856+0000 7f15c6523640 1 --2- 172.21.15.169:0/3631918448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15c01005c0 0x7f15c01979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:08:46.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.857+0000 7f15c6d24640 1 --2- 172.21.15.169:0/3631918448 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15c0102170 0x7f15c019ef90 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f15b800eb10 tx=0x7f15b800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:46.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.857+0000 7f15b77fe640 1 -- 172.21.15.169:0/3631918448 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b8004b20 con 0x7f15c0102170 2024-03-28T01:08:46.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.857+0000 7f15b77fe640 1 -- 172.21.15.169:0/3631918448 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15b8004cc0 con 0x7f15c0102170 2024-03-28T01:08:46.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.857+0000 7f15c8787640 1 -- 172.21.15.169:0/3631918448 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c019f8f0 con 0x7f15c0102170 2024-03-28T01:08:46.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.857+0000 7f15b77fe640 1 -- 172.21.15.169:0/3631918448 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b8010760 con 0x7f15c0102170 2024-03-28T01:08:46.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.857+0000 7f15c8787640 1 -- 172.21.15.169:0/3631918448 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c019fe90 con 0x7f15c0102170 2024-03-28T01:08:46.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.858+0000 7f15c8787640 1 -- 172.21.15.169:0/3631918448 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c0107960 con 0x7f15c0102170 2024-03-28T01:08:46.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.859+0000 7f15b77fe640 1 -- 172.21.15.169:0/3631918448 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f15b8005000 con 0x7f15c0102170 2024-03-28T01:08:46.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.859+0000 7f15b77fe640 1 --2- 172.21.15.169:0/3631918448 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f15ac0a2880 0x7f15ac0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:46.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.860+0000 7f15c6523640 1 --2- 172.21.15.169:0/3631918448 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f15ac0a2880 0x7f15ac0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:46.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.860+0000 7f15b77fe640 1 -- 172.21.15.169:0/3631918448 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f15b80cf7d0 con 0x7f15c0102170 2024-03-28T01:08:46.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.860+0000 7f15c6523640 1 --2- 172.21.15.169:0/3631918448 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f15ac0a2880 0x7f15ac0a4d40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f15bc000c00 tx=0x7f15bc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:46.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:46.863+0000 7f15b77fe640 1 -- 172.21.15.169:0/3631918448 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f15b809c4d0 con 0x7f15c0102170 2024-03-28T01:08:47.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.083+0000 7f15c8787640 1 -- 172.21.15.169:0/3631918448 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15c002d070 con 0x7f15c0102170 2024-03-28T01:08:47.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.084+0000 7f15b77fe640 1 -- 172.21.15.169:0/3631918448 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7f15b80b3cb0 con 0x7f15c0102170 2024-03-28T01:08:47.087 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T01:08:47.087 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T01:08:47.087 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T01:08:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.086+0000 7f15c8787640 1 -- 172.21.15.169:0/3631918448 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f15ac0a2880 msgr2=0x7f15ac0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.086+0000 7f15c8787640 1 --2- 172.21.15.169:0/3631918448 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f15ac0a2880 0x7f15ac0a4d40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f15bc000c00 tx=0x7f15bc005fd0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.086+0000 7f15c8787640 1 -- 172.21.15.169:0/3631918448 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15c0102170 msgr2=0x7f15c019ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.086+0000 7f15c8787640 1 --2- 172.21.15.169:0/3631918448 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15c0102170 0x7f15c019ef90 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f15b800eb10 tx=0x7f15b800c2d0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.086+0000 7f15c8787640 1 -- 172.21.15.169:0/3631918448 shutdown_connections 2024-03-28T01:08:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.086+0000 7f15c8787640 1 --2- 172.21.15.169:0/3631918448 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f15ac0a2880 0x7f15ac0a4d40 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.086+0000 7f15c8787640 1 --2- 172.21.15.169:0/3631918448 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15c0102170 0x7f15c019ef90 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.086+0000 7f15c8787640 1 --2- 172.21.15.169:0/3631918448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15c01017b0 0x7f15c0197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.086+0000 7f15c8787640 1 --2- 172.21.15.169:0/3631918448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15c01005c0 0x7f15c01979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.086+0000 7f15c8787640 1 -- 172.21.15.169:0/3631918448 >> 172.21.15.169:0/3631918448 conn(0x7f15c00fbb30 msgr2=0x7f15c0104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.086+0000 7f15c8787640 1 -- 172.21.15.169:0/3631918448 shutdown_connections 2024-03-28T01:08:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.086+0000 7f15c8787640 1 -- 172.21.15.169:0/3631918448 wait complete. 2024-03-28T01:08:47.103 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:08:47.103 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T01:08:47.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.215+0000 7fab8c77e640 1 -- 172.21.15.169:0/760851703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab841005a0 msgr2=0x7fab841009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.215+0000 7fab8c77e640 1 --2- 172.21.15.169:0/760851703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab841005a0 0x7fab841009c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fab74009900 tx=0x7fab7402bc30 comp rx=0 tx=0).stop 2024-03-28T01:08:47.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.216+0000 7fab8c77e640 1 -- 172.21.15.169:0/760851703 shutdown_connections 2024-03-28T01:08:47.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.216+0000 7fab8c77e640 1 --2- 172.21.15.169:0/760851703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab84102150 0x7fab841089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.216+0000 7fab8c77e640 1 --2- 172.21.15.169:0/760851703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab84101790 0x7fab84101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.216+0000 7fab8c77e640 1 --2- 172.21.15.169:0/760851703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab841005a0 0x7fab841009c0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.216+0000 7fab8c77e640 1 -- 172.21.15.169:0/760851703 >> 172.21.15.169:0/760851703 conn(0x7fab840fbb30 msgr2=0x7fab840fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:47.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.216+0000 7fab8c77e640 1 -- 172.21.15.169:0/760851703 shutdown_connections 2024-03-28T01:08:47.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.216+0000 7fab8c77e640 1 -- 172.21.15.169:0/760851703 wait complete. 2024-03-28T01:08:47.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.216+0000 7fab8c77e640 1 Processor -- start 2024-03-28T01:08:47.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.216+0000 7fab8c77e640 1 -- start start 2024-03-28T01:08:47.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.216+0000 7fab8c77e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab841005a0 0x7fab84197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.216+0000 7fab8c77e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab84101790 0x7fab84197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.216+0000 7fab8c77e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab84102150 0x7fab8419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.216+0000 7fab8c77e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8410c230 con 0x7fab84102150 2024-03-28T01:08:47.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.216+0000 7fab8c77e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8410c0b0 con 0x7fab841005a0 2024-03-28T01:08:47.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.216+0000 7fab8c77e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8410c3b0 con 0x7fab84101790 2024-03-28T01:08:47.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.217+0000 7fab8ad1b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab84102150 0x7fab8419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.217+0000 7fab8ad1b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab84102150 0x7fab8419f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53314/0 (socket says 172.21.15.169:53314) 2024-03-28T01:08:47.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.217+0000 7fab8ad1b640 1 -- 172.21.15.169:0/339911449 learned_addr learned my addr 172.21.15.169:0/339911449 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:47.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.217+0000 7fab8a51a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab841005a0 0x7fab84197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.217+0000 7fab89d19640 1 --2- 172.21.15.169:0/339911449 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab84101790 0x7fab84197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.217+0000 7fab8a51a640 1 -- 172.21.15.169:0/339911449 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab84101790 msgr2=0x7fab84197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.217+0000 7fab8a51a640 1 --2- 172.21.15.169:0/339911449 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab84101790 0x7fab84197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.217+0000 7fab8a51a640 1 -- 172.21.15.169:0/339911449 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab84102150 msgr2=0x7fab8419f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.217+0000 7fab8a51a640 1 --2- 172.21.15.169:0/339911449 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab84102150 0x7fab8419f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.217+0000 7fab8a51a640 1 -- 172.21.15.169:0/339911449 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab8419f690 con 0x7fab841005a0 2024-03-28T01:08:47.223 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.217+0000 7fab8ad1b640 1 --2- 172.21.15.169:0/339911449 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab84102150 0x7fab8419f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:08:47.223 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.217+0000 7fab8a51a640 1 --2- 172.21.15.169:0/339911449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab841005a0 0x7fab84197a70 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fab7402c170 tx=0x7fab7402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:47.223 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.217+0000 7fab7b7fe640 1 -- 172.21.15.169:0/339911449 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab74041070 con 0x7fab841005a0 2024-03-28T01:08:47.223 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.217+0000 7fab7b7fe640 1 -- 172.21.15.169:0/339911449 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab74004450 con 0x7fab841005a0 2024-03-28T01:08:47.223 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.217+0000 7fab8c77e640 1 -- 172.21.15.169:0/339911449 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab8419f920 con 0x7fab841005a0 2024-03-28T01:08:47.223 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.217+0000 7fab7b7fe640 1 -- 172.21.15.169:0/339911449 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab74003e50 con 0x7fab841005a0 2024-03-28T01:08:47.223 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.217+0000 7fab8c77e640 1 -- 172.21.15.169:0/339911449 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab8419fde0 con 0x7fab841005a0 2024-03-28T01:08:47.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.219+0000 7fab7b7fe640 1 -- 172.21.15.169:0/339911449 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fab74044070 con 0x7fab841005a0 2024-03-28T01:08:47.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.220+0000 7fab7b7fe640 1 --2- 172.21.15.169:0/339911449 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fab700a29a0 0x7fab700a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.220+0000 7fab7b7fe640 1 -- 172.21.15.169:0/339911449 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fab740f0900 con 0x7fab841005a0 2024-03-28T01:08:47.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.220+0000 7fab89d19640 1 --2- 172.21.15.169:0/339911449 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fab700a29a0 0x7fab700a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.220+0000 7fab8f369740 1 Processor -- start 2024-03-28T01:08:47.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.221+0000 7fab8f369740 1 -- start start 2024-03-28T01:08:47.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.221+0000 7fab8f369740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806e9690 0x55da806e9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.221+0000 7fab89d19640 1 --2- 172.21.15.169:0/339911449 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fab700a29a0 0x7fab700a4e60 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fab800095f0 tx=0x7fab80005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:47.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.221+0000 7fab8f369740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806ea080 0x55da806eeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.221+0000 7fab8f369740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806ef1f0 0x55da806f1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.221+0000 7fab8f369740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da806ac610 con 0x55da806ea080 2024-03-28T01:08:47.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.221+0000 7fab8f369740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da8064f3c0 con 0x55da806e9690 2024-03-28T01:08:47.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.221+0000 7fab8f369740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da805f7d60 con 0x55da806ef1f0 2024-03-28T01:08:47.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.221+0000 7fab89d19640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806ea080 0x55da806eeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.221+0000 7fab8ad1b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806ef1f0 0x55da806f1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.221+0000 7fab8a51a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806e9690 0x55da806e9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.221+0000 7fab89d19640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806ea080 0x55da806eeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53316/0 (socket says 172.21.15.169:53316) 2024-03-28T01:08:47.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.221+0000 7fab89d19640 1 -- 172.21.15.169:0/2371561398 learned_addr learned my addr 172.21.15.169:0/2371561398 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:47.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.221+0000 7fab8ad1b640 1 -- 172.21.15.169:0/2371561398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806e9690 msgr2=0x55da806e9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.221+0000 7fab8ad1b640 1 --2- 172.21.15.169:0/2371561398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806e9690 0x55da806e9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.221+0000 7fab8ad1b640 1 -- 172.21.15.169:0/2371561398 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806ea080 msgr2=0x55da806eeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.221+0000 7fab8ad1b640 1 --2- 172.21.15.169:0/2371561398 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806ea080 0x55da806eeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.221+0000 7fab8ad1b640 1 -- 172.21.15.169:0/2371561398 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da803633f0 con 0x55da806ef1f0 2024-03-28T01:08:47.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.221+0000 7fab8a51a640 1 --2- 172.21.15.169:0/2371561398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806e9690 0x55da806e9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:08:47.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.221+0000 7fab8ad1b640 1 --2- 172.21.15.169:0/2371561398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806ef1f0 0x55da806f1680 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fab7c00e800 tx=0x7fab7c00ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=557f54675f00bab7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:47.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.222+0000 7fab78ff9640 1 -- 172.21.15.169:0/2371561398 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c011c20 con 0x55da806ef1f0 2024-03-28T01:08:47.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.222+0000 7fab78ff9640 1 -- 172.21.15.169:0/2371561398 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab7c011dc0 con 0x55da806ef1f0 2024-03-28T01:08:47.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.222+0000 7fab78ff9640 1 -- 172.21.15.169:0/2371561398 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c00f740 con 0x55da806ef1f0 2024-03-28T01:08:47.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.222+0000 7fab8f369740 1 -- 172.21.15.169:0/2371561398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806ef1f0 msgr2=0x55da806f1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.222+0000 7fab8f369740 1 --2- 172.21.15.169:0/2371561398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806ef1f0 0x55da806f1680 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fab7c00e800 tx=0x7fab7c00ecc0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.222+0000 7fab8f369740 1 -- 172.21.15.169:0/2371561398 shutdown_connections 2024-03-28T01:08:47.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.222+0000 7fab8f369740 1 --2- 172.21.15.169:0/2371561398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806ef1f0 0x55da806f1680 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.222+0000 7fab8f369740 1 --2- 172.21.15.169:0/2371561398 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806ea080 0x55da806eeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.222+0000 7fab8f369740 1 --2- 172.21.15.169:0/2371561398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806e9690 0x55da806e9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.222+0000 7fab8f369740 1 -- 172.21.15.169:0/2371561398 >> 172.21.15.169:0/2371561398 conn(0x55da806e4c00 msgr2=0x55da806e5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:47.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.223+0000 7fab8f369740 1 -- 172.21.15.169:0/2371561398 shutdown_connections 2024-03-28T01:08:47.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.223+0000 7fab8f369740 1 -- 172.21.15.169:0/2371561398 wait complete. 2024-03-28T01:08:47.232 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.223+0000 7fab8f369740 1 -- start start 2024-03-28T01:08:47.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.223+0000 7fab8f369740 1 Processor -- start 2024-03-28T01:08:47.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.223+0000 7fab8f369740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806e9690 0x55da806f3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.223+0000 7fab8f369740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806ea080 0x55da806f3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.223+0000 7fab8f369740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806ef1f0 0x55da806faf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.223+0000 7fab8f369740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da8060b240 con 0x55da806e9690 2024-03-28T01:08:47.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.223+0000 7fab8f369740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da8064b460 con 0x55da806ea080 2024-03-28T01:08:47.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.223+0000 7fab8f369740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da806f21a0 con 0x55da806ef1f0 2024-03-28T01:08:47.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.223+0000 7fab8f369740 10 client.0 _ll_register_callbacks cb 0x55da8060ac10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:47.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.223+0000 7fab8ad1b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806ef1f0 0x55da806faf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.223+0000 7fab8a51a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806e9690 0x55da806f3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.223+0000 7fab89d19640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806ea080 0x55da806f3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.223+0000 7fab8ad1b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806ef1f0 0x55da806faf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48462/0 (socket says 172.21.15.169:48462) 2024-03-28T01:08:47.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.223+0000 7fab8ad1b640 1 -- 172.21.15.169:0/3714895002 learned_addr learned my addr 172.21.15.169:0/3714895002 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:47.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.224+0000 7fab8ad1b640 1 -- 172.21.15.169:0/3714895002 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806ea080 msgr2=0x55da806f3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.233 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.224+0000 7fab8ad1b640 1 --2- 172.21.15.169:0/3714895002 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806ea080 0x55da806f3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.224+0000 7fab8ad1b640 1 -- 172.21.15.169:0/3714895002 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806e9690 msgr2=0x55da806f3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.224+0000 7fab8ad1b640 1 --2- 172.21.15.169:0/3714895002 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806e9690 0x55da806f3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.224+0000 7fab8ad1b640 1 -- 172.21.15.169:0/3714895002 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55da8042d7d0 con 0x55da806ef1f0 2024-03-28T01:08:47.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.224+0000 7fab8ad1b640 1 --2- 172.21.15.169:0/3714895002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806ef1f0 0x55da806faf80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fab7c000f80 tx=0x7fab7c004c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=108d37fd0c78f8c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:47.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.224+0000 7fab6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:08:47.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.224+0000 7fab6dffb640 1 -- 172.21.15.169:0/3714895002 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c01c660 con 0x55da806ef1f0 2024-03-28T01:08:47.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.224+0000 7fab8f369740 1 -- 172.21.15.169:0/3714895002 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fab7c011dc0 con 0x55da806ef1f0 2024-03-28T01:08:47.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.224+0000 7fab6dffb640 1 -- 172.21.15.169:0/3714895002 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fab7c01c800 con 0x55da806ef1f0 2024-03-28T01:08:47.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.224+0000 7fab6dffb640 1 -- 172.21.15.169:0/3714895002 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab7c032050 con 0x55da806ef1f0 2024-03-28T01:08:47.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.224+0000 7fab8f369740 10 client.27526 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:47.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.224+0000 7fab8f369740 1 -- 172.21.15.169:0/3714895002 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55da806fe2f0 con 0x55da806ef1f0 2024-03-28T01:08:47.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.224+0000 7fab6dffb640 1 -- 172.21.15.169:0/3714895002 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab7c011dc0 con 0x55da806ef1f0 2024-03-28T01:08:47.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.225+0000 7fab6dffb640 1 -- 172.21.15.169:0/3714895002 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fab7c02d030 con 0x55da806ef1f0 2024-03-28T01:08:47.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.225+0000 7fab8f369740 10 client.27526 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:47.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.225+0000 7fab8f369740 10 client.27526 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:08:47.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.225+0000 7fab8f369740 20 client.27526 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:47.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.225+0000 7fab8f369740 1 --2- 172.21.15.169:0/3714895002 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55da806ff7f0 0x55da80701c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.225+0000 7fab8f369740 4 client.27526 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:08:47.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.225+0000 7fab8f369740 1 -- 172.21.15.169:0/3714895002 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55da8065c900 con 0x55da806ff7f0 2024-03-28T01:08:47.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.225+0000 7fab8a51a640 1 --2- 172.21.15.169:0/3714895002 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55da806ff7f0 0x55da80701c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.225+0000 7fab8a51a640 1 --2- 172.21.15.169:0/3714895002 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55da806ff7f0 0x55da80701c40 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=119dbc30cee9226 server_cookie=899ed80590779acc in_seq=0 out_seq=0 2024-03-28T01:08:47.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.225+0000 7fab6dffb640 10 client.27526 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:08:47.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.226+0000 7fab6dffb640 1 -- 172.21.15.169:0/3714895002 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55da8065c900 con 0x55da806ff7f0 2024-03-28T01:08:47.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.226+0000 7fab6dffb640 10 client.27526 handle_command_reply: tid=0 2024-03-28T01:08:47.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.226+0000 7fab8f369740 1 client.27526 shutdown 2024-03-28T01:08:47.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.226+0000 7fab8f369740 1 -- 172.21.15.169:0/3714895002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806ef1f0 msgr2=0x55da806faf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.226+0000 7fab8f369740 1 --2- 172.21.15.169:0/3714895002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806ef1f0 0x55da806faf80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fab7c000f80 tx=0x7fab7c004c70 comp rx=0 tx=0).stop 2024-03-28T01:08:47.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.226+0000 7fab8f369740 1 -- 172.21.15.169:0/3714895002 shutdown_connections 2024-03-28T01:08:47.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.226+0000 7fab8f369740 1 --2- 172.21.15.169:0/3714895002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806ef1f0 0x55da806faf80 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.226+0000 7fab8f369740 1 --2- 172.21.15.169:0/3714895002 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806ea080 0x55da806f3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.226+0000 7fab8f369740 1 --2- 172.21.15.169:0/3714895002 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55da806ff7f0 0x55da80701c40 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.226+0000 7fab8f369740 1 --2- 172.21.15.169:0/3714895002 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806e9690 0x55da806f3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.226+0000 7fab6dffb640 0 client.27526 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:08:47.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.226+0000 7fab8f369740 1 -- 172.21.15.169:0/3714895002 >> 172.21.15.169:0/3714895002 conn(0x55da806e4c00 msgr2=0x55da806e3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:47.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.226+0000 7fab8f369740 1 -- 172.21.15.169:0/3714895002 shutdown_connections 2024-03-28T01:08:47.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.226+0000 7fab8f369740 1 -- 172.21.15.169:0/3714895002 wait complete. 2024-03-28T01:08:47.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.226+0000 7fab8f369740 20 client.27526 trim_cache size 0 max 16384 2024-03-28T01:08:47.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.251+0000 7fab8f369740 1 Processor -- start 2024-03-28T01:08:47.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.251+0000 7fab8f369740 1 -- start start 2024-03-28T01:08:47.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.251+0000 7fab8f369740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806e9690 0x55da806e8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.251+0000 7fab8f369740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806ea080 0x55da806faf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.251+0000 7fab8f369740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806ef1f0 0x55da80706500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.251+0000 7fab8f369740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab841ac6c0 con 0x55da806ea080 2024-03-28T01:08:47.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.251+0000 7fab8f369740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab841aa1c0 con 0x55da806ef1f0 2024-03-28T01:08:47.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.251+0000 7fab8f369740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab841a8260 con 0x55da806e9690 2024-03-28T01:08:47.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.251+0000 7fab8ad1b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806ef1f0 0x55da80706500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.251+0000 7fab89d19640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806ea080 0x55da806faf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.251+0000 7fab8a51a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806e9690 0x55da806e8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.251+0000 7fab8ad1b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806ef1f0 0x55da80706500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:55340/0 (socket says 172.21.15.169:55340) 2024-03-28T01:08:47.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.251+0000 7fab8ad1b640 1 -- 172.21.15.169:0/3503351662 learned_addr learned my addr 172.21.15.169:0/3503351662 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:47.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.251+0000 7fab8ad1b640 1 -- 172.21.15.169:0/3503351662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806e9690 msgr2=0x55da806e8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.251+0000 7fab8ad1b640 1 --2- 172.21.15.169:0/3503351662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806e9690 0x55da806e8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.251+0000 7fab8ad1b640 1 -- 172.21.15.169:0/3503351662 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806ea080 msgr2=0x55da806faf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.251+0000 7fab8ad1b640 1 --2- 172.21.15.169:0/3503351662 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806ea080 0x55da806faf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.251+0000 7fab8ad1b640 1 -- 172.21.15.169:0/3503351662 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da806df230 con 0x55da806ef1f0 2024-03-28T01:08:47.262 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.252+0000 7fab8ad1b640 1 --2- 172.21.15.169:0/3503351662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806ef1f0 0x55da80706500 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fab7c00ff40 tx=0x7fab7c005d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d15769103e2310d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:47.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.252+0000 7fab6ffff640 1 -- 172.21.15.169:0/3503351662 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c011dc0 con 0x55da806ef1f0 2024-03-28T01:08:47.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.252+0000 7fab6ffff640 1 -- 172.21.15.169:0/3503351662 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab7c025660 con 0x55da806ef1f0 2024-03-28T01:08:47.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.252+0000 7fab6ffff640 1 -- 172.21.15.169:0/3503351662 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c00f3c0 con 0x55da806ef1f0 2024-03-28T01:08:47.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.252+0000 7fab8f369740 1 -- 172.21.15.169:0/3503351662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806ef1f0 msgr2=0x55da80706500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.252+0000 7fab8f369740 1 --2- 172.21.15.169:0/3503351662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806ef1f0 0x55da80706500 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fab7c00ff40 tx=0x7fab7c005d30 comp rx=0 tx=0).stop 2024-03-28T01:08:47.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.252+0000 7fab8f369740 1 -- 172.21.15.169:0/3503351662 shutdown_connections 2024-03-28T01:08:47.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.252+0000 7fab8f369740 1 --2- 172.21.15.169:0/3503351662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806ef1f0 0x55da80706500 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.252+0000 7fab8f369740 1 --2- 172.21.15.169:0/3503351662 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806ea080 0x55da806faf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.252+0000 7fab8f369740 1 --2- 172.21.15.169:0/3503351662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806e9690 0x55da806e8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.253+0000 7fab8f369740 1 -- 172.21.15.169:0/3503351662 >> 172.21.15.169:0/3503351662 conn(0x55da806e4c00 msgr2=0x55da806e7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:47.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.253+0000 7fab8f369740 1 -- 172.21.15.169:0/3503351662 shutdown_connections 2024-03-28T01:08:47.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.253+0000 7fab8f369740 1 -- 172.21.15.169:0/3503351662 wait complete. 2024-03-28T01:08:47.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.253+0000 7fab8f369740 1 -- start start 2024-03-28T01:08:47.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.253+0000 7fab8f369740 1 Processor -- start 2024-03-28T01:08:47.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.253+0000 7fab8f369740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806e9690 0x55da806f8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.253+0000 7fab8f369740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806ea080 0x55da806f9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.253+0000 7fab8f369740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806ef1f0 0x55da806fa150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.253+0000 7fab8f369740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab841a8860 con 0x55da806e9690 2024-03-28T01:08:47.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.253+0000 7fab8f369740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab841a9b60 con 0x55da806ef1f0 2024-03-28T01:08:47.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.253+0000 7fab8f369740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab841a8ea0 con 0x55da806ea080 2024-03-28T01:08:47.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.253+0000 7fab8f369740 10 client.0 _ll_register_callbacks cb 0x7fab7c00e700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:47.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.253+0000 7fab8a51a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806e9690 0x55da806f8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.253+0000 7fab89d19640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806ea080 0x55da806f9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.253+0000 7fab8a51a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806e9690 0x55da806f8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53340/0 (socket says 172.21.15.169:53340) 2024-03-28T01:08:47.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.253+0000 7fab8a51a640 1 -- 172.21.15.169:0/559959107 learned_addr learned my addr 172.21.15.169:0/559959107 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:47.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.253+0000 7fab8ad1b640 1 --2- 172.21.15.169:0/559959107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806ef1f0 0x55da806fa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.254+0000 7fab89d19640 1 -- 172.21.15.169:0/559959107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806ef1f0 msgr2=0x55da806fa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.254+0000 7fab89d19640 1 --2- 172.21.15.169:0/559959107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806ef1f0 0x55da806fa150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.254+0000 7fab89d19640 1 -- 172.21.15.169:0/559959107 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806e9690 msgr2=0x55da806f8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.254+0000 7fab89d19640 1 --2- 172.21.15.169:0/559959107 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806e9690 0x55da806f8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.254+0000 7fab89d19640 1 -- 172.21.15.169:0/559959107 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55da806fa740 con 0x55da806ea080 2024-03-28T01:08:47.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.254+0000 7fab8a51a640 1 --2- 172.21.15.169:0/559959107 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806e9690 0x55da806f8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:08:47.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.254+0000 7fab8ad1b640 1 --2- 172.21.15.169:0/559959107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806ef1f0 0x55da806fa150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:08:47.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.254+0000 7fab89d19640 1 --2- 172.21.15.169:0/559959107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806ea080 0x55da806f9140 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fab8000e4a0 tx=0x7fab80004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a032914d2efb18c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:47.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.254+0000 7fab6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:08:47.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.254+0000 7fab6dffb640 1 -- 172.21.15.169:0/559959107 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8002e060 con 0x55da806ea080 2024-03-28T01:08:47.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.254+0000 7fab8f369740 1 -- 172.21.15.169:0/559959107 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55da80656c50 con 0x55da806ea080 2024-03-28T01:08:47.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.254+0000 7fab6dffb640 1 -- 172.21.15.169:0/559959107 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fab8000eae0 con 0x55da806ea080 2024-03-28T01:08:47.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.254+0000 7fab6dffb640 1 -- 172.21.15.169:0/559959107 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab80035050 con 0x55da806ea080 2024-03-28T01:08:47.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.254+0000 7fab8f369740 10 client.27532 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:47.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.254+0000 7fab8f369740 1 -- 172.21.15.169:0/559959107 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55da8075b700 con 0x55da806ea080 2024-03-28T01:08:47.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.254+0000 7fab6dffb640 1 -- 172.21.15.169:0/559959107 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55da80656c50 con 0x55da806ea080 2024-03-28T01:08:47.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.255+0000 7fab6dffb640 1 -- 172.21.15.169:0/559959107 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fab8001d410 con 0x55da806ea080 2024-03-28T01:08:47.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.255+0000 7fab8f369740 10 client.27532 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:47.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.255+0000 7fab8f369740 10 client.27532 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T01:08:47.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.255+0000 7fab8f369740 20 client.27532 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:47.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.255+0000 7fab8f369740 1 --2- 172.21.15.169:0/559959107 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55da8075c090 0x55da8075e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.255+0000 7fab8f369740 4 client.27532 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:08:47.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.255+0000 7fab8f369740 1 -- 172.21.15.169:0/559959107 --> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55da805d4640 con 0x55da8075c090 2024-03-28T01:08:47.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.255+0000 7fab8a51a640 1 --2- 172.21.15.169:0/559959107 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55da8075c090 0x55da8075e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.255+0000 7fab6dffb640 10 client.27532 ms_handle_connect on v2:172.21.15.177:6836/3931247435 2024-03-28T01:08:47.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.255+0000 7fab8a51a640 1 --2- 172.21.15.169:0/559959107 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55da8075c090 0x55da8075e4b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=116f5fd08839b74e server_cookie=1736c8973e7bdbc2 in_seq=0 out_seq=0 2024-03-28T01:08:47.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.255+0000 7fab6dffb640 1 -- 172.21.15.169:0/559959107 <== mds.2 v2:172.21.15.177:6836/3931247435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55da805d4640 con 0x55da8075c090 2024-03-28T01:08:47.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.255+0000 7fab6dffb640 10 client.27532 handle_command_reply: tid=0 2024-03-28T01:08:47.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.255+0000 7fab8f369740 1 client.27532 shutdown 2024-03-28T01:08:47.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8f369740 1 -- 172.21.15.169:0/559959107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806ea080 msgr2=0x55da806f9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8f369740 1 --2- 172.21.15.169:0/559959107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806ea080 0x55da806f9140 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fab8000e4a0 tx=0x7fab80004730 comp rx=0 tx=0).stop 2024-03-28T01:08:47.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8f369740 1 -- 172.21.15.169:0/559959107 shutdown_connections 2024-03-28T01:08:47.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8f369740 1 --2- 172.21.15.169:0/559959107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55da806ef1f0 0x55da806fa150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8f369740 1 --2- 172.21.15.169:0/559959107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da806ea080 0x55da806f9140 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8f369740 1 --2- 172.21.15.169:0/559959107 >> [v2:172.21.15.177:6836/3931247435,v1:172.21.15.177:6837/3931247435] conn(0x55da8075c090 0x55da8075e4b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8f369740 1 --2- 172.21.15.169:0/559959107 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55da806e9690 0x55da806f8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab6dffb640 0 client.27532 ms_handle_reset on v2:172.21.15.177:6836/3931247435 2024-03-28T01:08:47.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8f369740 1 -- 172.21.15.169:0/559959107 >> 172.21.15.169:0/559959107 conn(0x55da806e4c00 msgr2=0x55da806e86f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:47.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8f369740 1 -- 172.21.15.169:0/559959107 shutdown_connections 2024-03-28T01:08:47.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8f369740 1 -- 172.21.15.169:0/559959107 wait complete. 2024-03-28T01:08:47.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8f369740 20 client.27532 trim_cache size 0 max 16384 2024-03-28T01:08:47.268 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:08:47.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8c77e640 1 -- 172.21.15.169:0/339911449 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fab700a29a0 msgr2=0x7fab700a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8c77e640 1 --2- 172.21.15.169:0/339911449 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fab700a29a0 0x7fab700a4e60 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fab800095f0 tx=0x7fab80005f90 comp rx=0 tx=0).stop 2024-03-28T01:08:47.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8c77e640 1 -- 172.21.15.169:0/339911449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab841005a0 msgr2=0x7fab84197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8c77e640 1 --2- 172.21.15.169:0/339911449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab841005a0 0x7fab84197a70 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fab7402c170 tx=0x7fab7402c650 comp rx=0 tx=0).stop 2024-03-28T01:08:47.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8c77e640 1 -- 172.21.15.169:0/339911449 shutdown_connections 2024-03-28T01:08:47.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8c77e640 1 --2- 172.21.15.169:0/339911449 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fab700a29a0 0x7fab700a4e60 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8c77e640 1 --2- 172.21.15.169:0/339911449 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab84102150 0x7fab8419f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8c77e640 1 --2- 172.21.15.169:0/339911449 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab84101790 0x7fab84197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8c77e640 1 --2- 172.21.15.169:0/339911449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab841005a0 0x7fab84197a70 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8c77e640 1 -- 172.21.15.169:0/339911449 >> 172.21.15.169:0/339911449 conn(0x7fab840fbb30 msgr2=0x7fab84068850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:47.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.256+0000 7fab8c77e640 1 -- 172.21.15.169:0/339911449 shutdown_connections 2024-03-28T01:08:47.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.257+0000 7fab8c77e640 1 -- 172.21.15.169:0/339911449 wait complete. 2024-03-28T01:08:47.273 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:08:47.273 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T01:08:47.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.386+0000 7f94be577640 1 -- 172.21.15.169:0/3541457527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94b81021e0 msgr2=0x7f94b81045d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.386+0000 7f94be577640 1 --2- 172.21.15.169:0/3541457527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94b81021e0 0x7f94b81045d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f94a0009900 tx=0x7f94a002bc30 comp rx=0 tx=0).stop 2024-03-28T01:08:47.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.386+0000 7f94be577640 1 -- 172.21.15.169:0/3541457527 shutdown_connections 2024-03-28T01:08:47.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.386+0000 7f94be577640 1 --2- 172.21.15.169:0/3541457527 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94b8107440 0x7f94b8109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.386+0000 7f94be577640 1 --2- 172.21.15.169:0/3541457527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94b8104b10 0x7f94b8106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.386+0000 7f94be577640 1 --2- 172.21.15.169:0/3541457527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94b81021e0 0x7f94b81045d0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.386+0000 7f94be577640 1 -- 172.21.15.169:0/3541457527 >> 172.21.15.169:0/3541457527 conn(0x7f94b80fbb50 msgr2=0x7f94b80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:47.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.386+0000 7f94be577640 1 -- 172.21.15.169:0/3541457527 shutdown_connections 2024-03-28T01:08:47.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.386+0000 7f94be577640 1 -- 172.21.15.169:0/3541457527 wait complete. 2024-03-28T01:08:47.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.386+0000 7f94be577640 1 Processor -- start 2024-03-28T01:08:47.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.386+0000 7f94be577640 1 -- start start 2024-03-28T01:08:47.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94be577640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94b81021e0 0x7f94b8197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94be577640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94b8104b10 0x7f94b8197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94be577640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94b8107440 0x7f94b819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94be577640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b810d3b0 con 0x7f94b8104b10 2024-03-28T01:08:47.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94be577640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b810d230 con 0x7f94b8107440 2024-03-28T01:08:47.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94be577640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b810d530 con 0x7f94b81021e0 2024-03-28T01:08:47.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94b7fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94b81021e0 0x7f94b8197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94b77fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94b8104b10 0x7f94b8197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94b7fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94b81021e0 0x7f94b8197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48500/0 (socket says 172.21.15.169:48500) 2024-03-28T01:08:47.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94b7fff640 1 -- 172.21.15.169:0/1176921737 learned_addr learned my addr 172.21.15.169:0/1176921737 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:47.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94bcb14640 1 --2- 172.21.15.169:0/1176921737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94b8107440 0x7f94b819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94b7fff640 1 -- 172.21.15.169:0/1176921737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94b8107440 msgr2=0x7f94b819efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94b7fff640 1 --2- 172.21.15.169:0/1176921737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94b8107440 0x7f94b819efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94b7fff640 1 -- 172.21.15.169:0/1176921737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94b8104b10 msgr2=0x7f94b8197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94b7fff640 1 --2- 172.21.15.169:0/1176921737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94b8104b10 0x7f94b8197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94b7fff640 1 -- 172.21.15.169:0/1176921737 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b819f640 con 0x7f94b81021e0 2024-03-28T01:08:47.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94b7fff640 1 --2- 172.21.15.169:0/1176921737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94b81021e0 0x7f94b8197a20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f94a0000c00 tx=0x7f94a002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:47.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94b57fa640 1 -- 172.21.15.169:0/1176921737 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a0043020 con 0x7f94b81021e0 2024-03-28T01:08:47.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94b57fa640 1 -- 172.21.15.169:0/1176921737 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94a00054e0 con 0x7f94b81021e0 2024-03-28T01:08:47.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94be577640 1 -- 172.21.15.169:0/1176921737 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94b819f8d0 con 0x7f94b81021e0 2024-03-28T01:08:47.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94b57fa640 1 -- 172.21.15.169:0/1176921737 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a0003c40 con 0x7f94b81021e0 2024-03-28T01:08:47.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.387+0000 7f94be577640 1 -- 172.21.15.169:0/1176921737 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94b819fe70 con 0x7f94b81021e0 2024-03-28T01:08:47.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.390+0000 7f94c0dfd740 1 Processor -- start 2024-03-28T01:08:47.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.390+0000 7f94c0dfd740 1 -- start start 2024-03-28T01:08:47.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.390+0000 7f94b57fa640 1 -- 172.21.15.169:0/1176921737 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f94a0046020 con 0x7f94b81021e0 2024-03-28T01:08:47.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.390+0000 7f94c0dfd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d6299690 0x55d0d6299ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.390+0000 7f94c0dfd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 0x55d0d629eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.390+0000 7f94c0dfd740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d629f1f0 0x55d0d62a1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.390+0000 7f94c0dfd740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0d625c610 con 0x55d0d629f1f0 2024-03-28T01:08:47.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.390+0000 7f94c0dfd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0d61ff3c0 con 0x55d0d629a080 2024-03-28T01:08:47.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.390+0000 7f94c0dfd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0d61a7d60 con 0x55d0d6299690 2024-03-28T01:08:47.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.391+0000 7f94b77fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d629f1f0 0x55d0d62a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.391+0000 7f94b7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 0x55d0d629eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.391+0000 7f94bcb14640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d6299690 0x55d0d6299ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.391+0000 7f94bcb14640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d6299690 0x55d0d6299ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48516/0 (socket says 172.21.15.169:48516) 2024-03-28T01:08:47.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.391+0000 7f94b77fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d629f1f0 0x55d0d62a1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53374/0 (socket says 172.21.15.169:53374) 2024-03-28T01:08:47.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.391+0000 7f94bcb14640 1 -- 172.21.15.169:0/2438222165 learned_addr learned my addr 172.21.15.169:0/2438222165 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:47.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.391+0000 7f94b7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 0x55d0d629eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:55370/0 (socket says 172.21.15.169:55370) 2024-03-28T01:08:47.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.391+0000 7f94bcb14640 1 -- 172.21.15.169:0/2438222165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 msgr2=0x55d0d629eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.391+0000 7f94bcb14640 1 --2- 172.21.15.169:0/2438222165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 0x55d0d629eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.391+0000 7f94bcb14640 1 -- 172.21.15.169:0/2438222165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d629f1f0 msgr2=0x55d0d62a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.391+0000 7f94bcb14640 1 --2- 172.21.15.169:0/2438222165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d629f1f0 0x55d0d62a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.391+0000 7f94bcb14640 1 -- 172.21.15.169:0/2438222165 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0d5f133f0 con 0x55d0d6299690 2024-03-28T01:08:47.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.391+0000 7f94b7fff640 1 --2- 172.21.15.169:0/2438222165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 0x55d0d629eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:08:47.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.391+0000 7f94b57fa640 1 --2- 172.21.15.169:0/1176921737 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f94a40a2880 0x7f94a40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.391+0000 7f94bcb14640 1 --2- 172.21.15.169:0/2438222165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d6299690 0x55d0d6299ab0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f94b0009fd0 tx=0x7f94b00118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8632d711cec274c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:47.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.391+0000 7f94ae7fc640 1 -- 172.21.15.169:0/2438222165 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b0004d20 con 0x55d0d6299690 2024-03-28T01:08:47.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.391+0000 7f94b57fa640 1 -- 172.21.15.169:0/1176921737 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f94a00f0de0 con 0x7f94b81021e0 2024-03-28T01:08:47.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.391+0000 7f94ae7fc640 1 -- 172.21.15.169:0/2438222165 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94b0004050 con 0x55d0d6299690 2024-03-28T01:08:47.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.391+0000 7f94ae7fc640 1 -- 172.21.15.169:0/2438222165 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b0005260 con 0x55d0d6299690 2024-03-28T01:08:47.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.392+0000 7f94b7fff640 1 --2- 172.21.15.169:0/1176921737 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f94a40a2880 0x7f94a40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.392+0000 7f94c0dfd740 1 -- 172.21.15.169:0/2438222165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d6299690 msgr2=0x55d0d6299ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.392+0000 7f94c0dfd740 1 --2- 172.21.15.169:0/2438222165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d6299690 0x55d0d6299ab0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f94b0009fd0 tx=0x7f94b00118d0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.392+0000 7f94b7fff640 1 --2- 172.21.15.169:0/1176921737 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f94a40a2880 0x7f94a40a4d40 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f94a00ee8b0 tx=0x7f94a00fbb80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:47.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.392+0000 7f94c0dfd740 1 -- 172.21.15.169:0/2438222165 shutdown_connections 2024-03-28T01:08:47.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.392+0000 7f94c0dfd740 1 --2- 172.21.15.169:0/2438222165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d629f1f0 0x55d0d62a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.392+0000 7f94c0dfd740 1 --2- 172.21.15.169:0/2438222165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 0x55d0d629eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.392+0000 7f94c0dfd740 1 --2- 172.21.15.169:0/2438222165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d6299690 0x55d0d6299ab0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.392+0000 7f94c0dfd740 1 -- 172.21.15.169:0/2438222165 >> 172.21.15.169:0/2438222165 conn(0x55d0d6294c00 msgr2=0x55d0d6295000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:47.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.392+0000 7f94c0dfd740 1 -- 172.21.15.169:0/2438222165 shutdown_connections 2024-03-28T01:08:47.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.392+0000 7f94c0dfd740 1 -- 172.21.15.169:0/2438222165 wait complete. 2024-03-28T01:08:47.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.393+0000 7f94c0dfd740 1 -- start start 2024-03-28T01:08:47.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.393+0000 7f94c0dfd740 1 Processor -- start 2024-03-28T01:08:47.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.393+0000 7f94c0dfd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d6299690 0x55d0d62a3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.393+0000 7f94c0dfd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 0x55d0d62a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.393+0000 7f94c0dfd740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d629f1f0 0x55d0d62aaf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.393+0000 7f94c0dfd740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0d61bb240 con 0x55d0d629f1f0 2024-03-28T01:08:47.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.393+0000 7f94c0dfd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0d61fb460 con 0x55d0d629a080 2024-03-28T01:08:47.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.393+0000 7f94c0dfd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0d62a21a0 con 0x55d0d6299690 2024-03-28T01:08:47.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.393+0000 7f94c0dfd740 10 client.0 _ll_register_callbacks cb 0x55d0d61bac10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:47.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.393+0000 7f94bcb14640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d629f1f0 0x55d0d62aaf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.393+0000 7f94b77fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 0x55d0d62a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.393+0000 7f94bcb14640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d6299690 0x55d0d62a3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.393+0000 7f94b77fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 0x55d0d62a3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:55382/0 (socket says 172.21.15.169:55382) 2024-03-28T01:08:47.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.393+0000 7f94bcb14640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d6299690 0x55d0d62a3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48526/0 (socket says 172.21.15.169:48526) 2024-03-28T01:08:47.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.393+0000 7f94b77fe640 1 -- 172.21.15.169:0/3686146797 learned_addr learned my addr 172.21.15.169:0/3686146797 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:47.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.394+0000 7f94b77fe640 1 -- 172.21.15.169:0/3686146797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d6299690 msgr2=0x55d0d62a3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.394+0000 7f94b77fe640 1 --2- 172.21.15.169:0/3686146797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d6299690 0x55d0d62a3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.394+0000 7f94b77fe640 1 -- 172.21.15.169:0/3686146797 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d629f1f0 msgr2=0x55d0d62aaf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.394+0000 7f94b77fe640 1 --2- 172.21.15.169:0/3686146797 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d629f1f0 0x55d0d62aaf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.394+0000 7f94b77fe640 1 -- 172.21.15.169:0/3686146797 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d0d5fdd7d0 con 0x55d0d629a080 2024-03-28T01:08:47.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.394+0000 7f94bcb14640 1 --2- 172.21.15.169:0/3686146797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d6299690 0x55d0d62a3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:08:47.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.394+0000 7f94b77fe640 1 --2- 172.21.15.169:0/3686146797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 0x55d0d62a3f50 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f94a8012700 tx=0x7f94a8012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d7e689b1612e3e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:47.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.394+0000 7f94977fe640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:08:47.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.394+0000 7f94977fe640 1 -- 172.21.15.169:0/3686146797 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a8004020 con 0x55d0d629a080 2024-03-28T01:08:47.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.394+0000 7f94977fe640 1 -- 172.21.15.169:0/3686146797 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f94a80041c0 con 0x55d0d629a080 2024-03-28T01:08:47.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.394+0000 7f94c0dfd740 1 -- 172.21.15.169:0/3686146797 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f94b0004050 con 0x55d0d629a080 2024-03-28T01:08:47.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.394+0000 7f94977fe640 1 -- 172.21.15.169:0/3686146797 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94a80043c0 con 0x55d0d629a080 2024-03-28T01:08:47.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.394+0000 7f94c0dfd740 10 client.27590 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:47.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.394+0000 7f94c0dfd740 1 -- 172.21.15.169:0/3686146797 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d0d62ae370 con 0x55d0d629a080 2024-03-28T01:08:47.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.394+0000 7f94977fe640 1 -- 172.21.15.169:0/3686146797 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94b0004050 con 0x55d0d629a080 2024-03-28T01:08:47.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.394+0000 7f94977fe640 1 -- 172.21.15.169:0/3686146797 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f94a8003ba0 con 0x55d0d629a080 2024-03-28T01:08:47.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.394+0000 7f94c0dfd740 10 client.27590 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:47.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.394+0000 7f94c0dfd740 10 client.27590 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:08:47.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.395+0000 7f94c0dfd740 20 client.27590 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:47.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.395+0000 7f94c0dfd740 1 --2- 172.21.15.169:0/3686146797 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55d0d62afa10 0x55d0d62b1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.395+0000 7f94c0dfd740 4 client.27590 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:08:47.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.395+0000 7f94c0dfd740 1 -- 172.21.15.169:0/3686146797 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d0d620c900 con 0x55d0d62afa10 2024-03-28T01:08:47.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.395+0000 7f94bcb14640 1 --2- 172.21.15.169:0/3686146797 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55d0d62afa10 0x55d0d62b1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.395+0000 7f94977fe640 10 client.27590 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:08:47.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.395+0000 7f94bcb14640 1 --2- 172.21.15.169:0/3686146797 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55d0d62afa10 0x55d0d62b1e30 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d229538065b04cb5 server_cookie=159d5f6a47516c1c in_seq=0 out_seq=0 2024-03-28T01:08:47.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.396+0000 7f94977fe640 1 -- 172.21.15.169:0/3686146797 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d0d620c900 con 0x55d0d62afa10 2024-03-28T01:08:47.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.396+0000 7f94977fe640 10 client.27590 handle_command_reply: tid=0 2024-03-28T01:08:47.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.396+0000 7f94c0dfd740 1 client.27590 shutdown 2024-03-28T01:08:47.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.396+0000 7f94c0dfd740 1 -- 172.21.15.169:0/3686146797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 msgr2=0x55d0d62a3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.396+0000 7f94c0dfd740 1 --2- 172.21.15.169:0/3686146797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 0x55d0d62a3f50 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f94a8012700 tx=0x7f94a8012bc0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.396+0000 7f94c0dfd740 1 -- 172.21.15.169:0/3686146797 shutdown_connections 2024-03-28T01:08:47.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.396+0000 7f94c0dfd740 1 --2- 172.21.15.169:0/3686146797 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55d0d62afa10 0x55d0d62b1e30 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.396+0000 7f94c0dfd740 1 --2- 172.21.15.169:0/3686146797 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d629f1f0 0x55d0d62aaf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.396+0000 7f94c0dfd740 1 --2- 172.21.15.169:0/3686146797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 0x55d0d62a3f50 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.396+0000 7f94977fe640 0 client.27590 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:08:47.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.396+0000 7f94c0dfd740 1 --2- 172.21.15.169:0/3686146797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d6299690 0x55d0d62a3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.396+0000 7f94c0dfd740 1 -- 172.21.15.169:0/3686146797 >> 172.21.15.169:0/3686146797 conn(0x55d0d6294c00 msgr2=0x55d0d6293e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:47.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.396+0000 7f94c0dfd740 1 -- 172.21.15.169:0/3686146797 shutdown_connections 2024-03-28T01:08:47.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.396+0000 7f94c0dfd740 1 -- 172.21.15.169:0/3686146797 wait complete. 2024-03-28T01:08:47.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.396+0000 7f94c0dfd740 20 client.27590 trim_cache size 0 max 16384 2024-03-28T01:08:47.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.409+0000 7f94c0dfd740 1 Processor -- start 2024-03-28T01:08:47.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.409+0000 7f94c0dfd740 1 -- start start 2024-03-28T01:08:47.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.409+0000 7f94c0dfd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d6299690 0x55d0d6293590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.409+0000 7f94c0dfd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 0x55d0d62aaf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.409+0000 7f94c0dfd740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d629f1f0 0x55d0d6293ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.409+0000 7f94c0dfd740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b81aa570 con 0x55d0d629f1f0 2024-03-28T01:08:47.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.409+0000 7f94c0dfd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b81a8070 con 0x55d0d629a080 2024-03-28T01:08:47.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.409+0000 7f94c0dfd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b81a6110 con 0x55d0d6299690 2024-03-28T01:08:47.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.409+0000 7f94bcb14640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d6299690 0x55d0d6293590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.409+0000 7f94bcb14640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d6299690 0x55d0d6293590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48534/0 (socket says 172.21.15.169:48534) 2024-03-28T01:08:47.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.409+0000 7f94bcb14640 1 -- 172.21.15.169:0/4114873929 learned_addr learned my addr 172.21.15.169:0/4114873929 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:47.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.409+0000 7f94b77fe640 1 --2- 172.21.15.169:0/4114873929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 0x55d0d62aaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.409+0000 7f94bcb14640 1 --2- 172.21.15.169:0/4114873929 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d629f1f0 0x55d0d6293ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.409+0000 7f94bcb14640 1 -- 172.21.15.169:0/4114873929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 msgr2=0x55d0d62aaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.409+0000 7f94bcb14640 1 --2- 172.21.15.169:0/4114873929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 0x55d0d62aaf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.409+0000 7f94bcb14640 1 -- 172.21.15.169:0/4114873929 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d629f1f0 msgr2=0x55d0d6293ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.409+0000 7f94bcb14640 1 --2- 172.21.15.169:0/4114873929 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d629f1f0 0x55d0d6293ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.409+0000 7f94bcb14640 1 -- 172.21.15.169:0/4114873929 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0d62a4a00 con 0x55d0d6299690 2024-03-28T01:08:47.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.410+0000 7f94bcb14640 1 --2- 172.21.15.169:0/4114873929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d6299690 0x55d0d6293590 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f94b001b040 tx=0x7f94b0000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddb2c6db9173837e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:47.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.410+0000 7f94adffb640 1 -- 172.21.15.169:0/4114873929 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b0009be0 con 0x55d0d6299690 2024-03-28T01:08:47.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.410+0000 7f94adffb640 1 -- 172.21.15.169:0/4114873929 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94b0004840 con 0x55d0d6299690 2024-03-28T01:08:47.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.410+0000 7f94adffb640 1 -- 172.21.15.169:0/4114873929 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b002f9b0 con 0x55d0d6299690 2024-03-28T01:08:47.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.410+0000 7f94c0dfd740 1 -- 172.21.15.169:0/4114873929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d6299690 msgr2=0x55d0d6293590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.410+0000 7f94c0dfd740 1 --2- 172.21.15.169:0/4114873929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d6299690 0x55d0d6293590 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f94b001b040 tx=0x7f94b0000f40 comp rx=0 tx=0).stop 2024-03-28T01:08:47.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.410+0000 7f94c0dfd740 1 -- 172.21.15.169:0/4114873929 shutdown_connections 2024-03-28T01:08:47.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.411+0000 7f94c0dfd740 1 --2- 172.21.15.169:0/4114873929 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d629f1f0 0x55d0d6293ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.411+0000 7f94c0dfd740 1 --2- 172.21.15.169:0/4114873929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 0x55d0d62aaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.411+0000 7f94c0dfd740 1 --2- 172.21.15.169:0/4114873929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d6299690 0x55d0d6293590 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.411+0000 7f94c0dfd740 1 -- 172.21.15.169:0/4114873929 >> 172.21.15.169:0/4114873929 conn(0x55d0d6294c00 msgr2=0x55d0d62a3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:47.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.411+0000 7f94c0dfd740 1 -- 172.21.15.169:0/4114873929 shutdown_connections 2024-03-28T01:08:47.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.411+0000 7f94c0dfd740 1 -- 172.21.15.169:0/4114873929 wait complete. 2024-03-28T01:08:47.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.411+0000 7f94c0dfd740 1 -- start start 2024-03-28T01:08:47.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.411+0000 7f94c0dfd740 1 Processor -- start 2024-03-28T01:08:47.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.411+0000 7f94c0dfd740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d6299690 0x55d0d62a8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.411+0000 7f94c0dfd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 0x55d0d62a93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.411+0000 7f94c0dfd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d629f1f0 0x55d0d62aa3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.411+0000 7f94c0dfd740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b81a6710 con 0x55d0d6299690 2024-03-28T01:08:47.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.411+0000 7f94c0dfd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b81a7a10 con 0x55d0d629a080 2024-03-28T01:08:47.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.411+0000 7f94c0dfd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b81a6d50 con 0x55d0d629f1f0 2024-03-28T01:08:47.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.411+0000 7f94c0dfd740 10 client.0 _ll_register_callbacks cb 0x7f94a8009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:47.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.411+0000 7f94bcb14640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d6299690 0x55d0d62a8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.412+0000 7f94bcb14640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d6299690 0x55d0d62a8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53394/0 (socket says 172.21.15.169:53394) 2024-03-28T01:08:47.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.412+0000 7f94b77fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 0x55d0d62a93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.412+0000 7f94bcb14640 1 -- 172.21.15.169:0/3917658995 learned_addr learned my addr 172.21.15.169:0/3917658995 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:47.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.412+0000 7f94bcb14640 1 --2- 172.21.15.169:0/3917658995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d629f1f0 0x55d0d62aa3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.412+0000 7f94b77fe640 1 -- 172.21.15.169:0/3917658995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d629f1f0 msgr2=0x55d0d62aa3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.412+0000 7f94b77fe640 1 --2- 172.21.15.169:0/3917658995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d629f1f0 0x55d0d62aa3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.412+0000 7f94b77fe640 1 -- 172.21.15.169:0/3917658995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d6299690 msgr2=0x55d0d62a8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.412+0000 7f94b77fe640 1 --2- 172.21.15.169:0/3917658995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d6299690 0x55d0d62a8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.412+0000 7f94b77fe640 1 -- 172.21.15.169:0/3917658995 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d0d62aa9d0 con 0x55d0d629a080 2024-03-28T01:08:47.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.412+0000 7f94b77fe640 1 --2- 172.21.15.169:0/3917658995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 0x55d0d62a93d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f94a800a010 tx=0x7f94a8009420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcf55cf2ad800642 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:47.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.412+0000 7f94977fe640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:08:47.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.412+0000 7f94977fe640 1 -- 172.21.15.169:0/3917658995 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a800fe60 con 0x55d0d629a080 2024-03-28T01:08:47.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.412+0000 7f94977fe640 1 -- 172.21.15.169:0/3917658995 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f94a8005030 con 0x55d0d629a080 2024-03-28T01:08:47.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.412+0000 7f94c0dfd740 1 -- 172.21.15.169:0/3917658995 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f94b0004840 con 0x55d0d629a080 2024-03-28T01:08:47.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.412+0000 7f94977fe640 1 -- 172.21.15.169:0/3917658995 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94a8036050 con 0x55d0d629a080 2024-03-28T01:08:47.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.412+0000 7f94c0dfd740 10 client.27596 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:47.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.412+0000 7f94c0dfd740 1 -- 172.21.15.169:0/3917658995 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d0d630bad0 con 0x55d0d629a080 2024-03-28T01:08:47.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.412+0000 7f94977fe640 1 -- 172.21.15.169:0/3917658995 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94b0004840 con 0x55d0d629a080 2024-03-28T01:08:47.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.413+0000 7f94977fe640 1 -- 172.21.15.169:0/3917658995 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7f94a8031030 con 0x55d0d629a080 2024-03-28T01:08:47.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.413+0000 7f94c0dfd740 10 client.27596 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:47.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.413+0000 7f94c0dfd740 10 client.27596 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T01:08:47.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.413+0000 7f94c0dfd740 20 client.27596 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:47.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.413+0000 7f94c0dfd740 1 --2- 172.21.15.169:0/3917658995 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55d0d630c3c0 0x55d0d630e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.413+0000 7f94c0dfd740 4 client.27596 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:08:47.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.413+0000 7f94c0dfd740 1 -- 172.21.15.169:0/3917658995 --> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d0d6206c50 con 0x55d0d630c3c0 2024-03-28T01:08:47.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.413+0000 7f94bcb14640 1 --2- 172.21.15.169:0/3917658995 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55d0d630c3c0 0x55d0d630e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.413+0000 7f94977fe640 10 client.27596 ms_handle_connect on v2:172.21.15.177:6838/152605729 2024-03-28T01:08:47.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.413+0000 7f94bcb14640 1 --2- 172.21.15.169:0/3917658995 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55d0d630c3c0 0x55d0d630e7e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4c79acc1de948b1f server_cookie=f41ef13ae666b899 in_seq=0 out_seq=0 2024-03-28T01:08:47.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.413+0000 7f94977fe640 1 -- 172.21.15.169:0/3917658995 <== mds.1 v2:172.21.15.177:6838/152605729 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d0d6206c50 con 0x55d0d630c3c0 2024-03-28T01:08:47.422 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.413+0000 7f94977fe640 10 client.27596 handle_command_reply: tid=0 2024-03-28T01:08:47.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.413+0000 7f94c0dfd740 1 client.27596 shutdown 2024-03-28T01:08:47.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.414+0000 7f94c0dfd740 1 -- 172.21.15.169:0/3917658995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 msgr2=0x55d0d62a93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.414+0000 7f94c0dfd740 1 --2- 172.21.15.169:0/3917658995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 0x55d0d62a93d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f94a800a010 tx=0x7f94a8009420 comp rx=0 tx=0).stop 2024-03-28T01:08:47.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.414+0000 7f94c0dfd740 1 -- 172.21.15.169:0/3917658995 shutdown_connections 2024-03-28T01:08:47.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.414+0000 7f94c0dfd740 1 --2- 172.21.15.169:0/3917658995 >> [v2:172.21.15.177:6838/152605729,v1:172.21.15.177:6839/152605729] conn(0x55d0d630c3c0 0x55d0d630e7e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.414+0000 7f94c0dfd740 1 --2- 172.21.15.169:0/3917658995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0d629f1f0 0x55d0d62aa3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.414+0000 7f94c0dfd740 1 --2- 172.21.15.169:0/3917658995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0d629a080 0x55d0d62a93d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.414+0000 7f94977fe640 0 client.27596 ms_handle_reset on v2:172.21.15.177:6838/152605729 2024-03-28T01:08:47.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.414+0000 7f94c0dfd740 1 --2- 172.21.15.169:0/3917658995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x55d0d6299690 0x55d0d62a8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.414+0000 7f94c0dfd740 1 -- 172.21.15.169:0/3917658995 >> 172.21.15.169:0/3917658995 conn(0x55d0d6294c00 msgr2=0x55d0d6297e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:47.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.414+0000 7f94c0dfd740 1 -- 172.21.15.169:0/3917658995 shutdown_connections 2024-03-28T01:08:47.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.414+0000 7f94c0dfd740 1 -- 172.21.15.169:0/3917658995 wait complete. 2024-03-28T01:08:47.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.414+0000 7f94c0dfd740 20 client.27596 trim_cache size 0 max 16384 2024-03-28T01:08:47.424 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:08:47.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.414+0000 7f94be577640 1 -- 172.21.15.169:0/1176921737 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f94a40a2880 msgr2=0x7f94a40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.414+0000 7f94be577640 1 --2- 172.21.15.169:0/1176921737 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f94a40a2880 0x7f94a40a4d40 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f94a00ee8b0 tx=0x7f94a00fbb80 comp rx=0 tx=0).stop 2024-03-28T01:08:47.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.414+0000 7f94be577640 1 -- 172.21.15.169:0/1176921737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94b81021e0 msgr2=0x7f94b8197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.414+0000 7f94be577640 1 --2- 172.21.15.169:0/1176921737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94b81021e0 0x7f94b8197a20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f94a0000c00 tx=0x7f94a002c650 comp rx=0 tx=0).stop 2024-03-28T01:08:47.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.414+0000 7f94be577640 1 -- 172.21.15.169:0/1176921737 shutdown_connections 2024-03-28T01:08:47.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.415+0000 7f94be577640 1 --2- 172.21.15.169:0/1176921737 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f94a40a2880 0x7f94a40a4d40 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.415+0000 7f94be577640 1 --2- 172.21.15.169:0/1176921737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94b8107440 0x7f94b819efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.415+0000 7f94be577640 1 --2- 172.21.15.169:0/1176921737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f94b8104b10 0x7f94b8197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.415+0000 7f94be577640 1 --2- 172.21.15.169:0/1176921737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94b81021e0 0x7f94b8197a20 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.415+0000 7f94be577640 1 -- 172.21.15.169:0/1176921737 >> 172.21.15.169:0/1176921737 conn(0x7f94b80fbb50 msgr2=0x7f94b8102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:47.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.415+0000 7f94be577640 1 -- 172.21.15.169:0/1176921737 shutdown_connections 2024-03-28T01:08:47.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.415+0000 7f94be577640 1 -- 172.21.15.169:0/1176921737 wait complete. 2024-03-28T01:08:47.431 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T01:08:47.431 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T01:08:47.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.544+0000 7fb075681640 1 -- 172.21.15.169:0/2333798155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0701000b0 msgr2=0x7fb070109c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.544+0000 7fb075681640 1 --2- 172.21.15.169:0/2333798155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0701000b0 0x7fb070109c60 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb068009ca0 tx=0x7fb06802c0f0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.544+0000 7fb075681640 1 -- 172.21.15.169:0/2333798155 shutdown_connections 2024-03-28T01:08:47.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.544+0000 7fb075681640 1 --2- 172.21.15.169:0/2333798155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0701000b0 0x7fb070109c60 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.544+0000 7fb075681640 1 --2- 172.21.15.169:0/2333798155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0700ff6f0 0x7fb0700ffb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.544+0000 7fb075681640 1 --2- 172.21.15.169:0/2333798155 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb070102c80 0x7fb0701030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.544+0000 7fb075681640 1 -- 172.21.15.169:0/2333798155 >> 172.21.15.169:0/2333798155 conn(0x7fb0700fb290 msgr2=0x7fb0700fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:47.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.544+0000 7fb075681640 1 -- 172.21.15.169:0/2333798155 shutdown_connections 2024-03-28T01:08:47.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.544+0000 7fb075681640 1 -- 172.21.15.169:0/2333798155 wait complete. 2024-03-28T01:08:47.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.544+0000 7fb075681640 1 Processor -- start 2024-03-28T01:08:47.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.545+0000 7fb075681640 1 -- start start 2024-03-28T01:08:47.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.545+0000 7fb075681640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0700ff6f0 0x7fb070197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.545+0000 7fb075681640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0701000b0 0x7fb070197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.545+0000 7fb075681640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb070102c80 0x7fb07019ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.545+0000 7fb075681640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07010d300 con 0x7fb0701000b0 2024-03-28T01:08:47.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.545+0000 7fb075681640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07010d180 con 0x7fb070102c80 2024-03-28T01:08:47.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.545+0000 7fb075681640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07010d480 con 0x7fb0700ff6f0 2024-03-28T01:08:47.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.545+0000 7fb06e7fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0701000b0 0x7fb070197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.545+0000 7fb06effd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0700ff6f0 0x7fb070197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.545+0000 7fb06effd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0700ff6f0 0x7fb070197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48562/0 (socket says 172.21.15.169:48562) 2024-03-28T01:08:47.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.545+0000 7fb06f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb070102c80 0x7fb07019ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.545+0000 7fb06effd640 1 -- 172.21.15.169:0/672843497 learned_addr learned my addr 172.21.15.169:0/672843497 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.545+0000 7fb06e7fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0701000b0 0x7fb070197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53412/0 (socket says 172.21.15.169:53412) 2024-03-28T01:08:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.545+0000 7fb06effd640 1 -- 172.21.15.169:0/672843497 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb070102c80 msgr2=0x7fb07019ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.545+0000 7fb06effd640 1 --2- 172.21.15.169:0/672843497 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb070102c80 0x7fb07019ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.545+0000 7fb06effd640 1 -- 172.21.15.169:0/672843497 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0701000b0 msgr2=0x7fb070197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.545+0000 7fb06effd640 1 --2- 172.21.15.169:0/672843497 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0701000b0 0x7fb070197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.545+0000 7fb06effd640 1 -- 172.21.15.169:0/672843497 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07019f590 con 0x7fb0700ff6f0 2024-03-28T01:08:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.545+0000 7fb06f7fe640 1 --2- 172.21.15.169:0/672843497 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb070102c80 0x7fb07019ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:08:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.545+0000 7fb06effd640 1 --2- 172.21.15.169:0/672843497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0700ff6f0 0x7fb070197970 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb06400e9c0 tx=0x7fb06400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.546+0000 7fb057fff640 1 -- 172.21.15.169:0/672843497 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06400cde0 con 0x7fb0700ff6f0 2024-03-28T01:08:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.546+0000 7fb057fff640 1 -- 172.21.15.169:0/672843497 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb064010910 con 0x7fb0700ff6f0 2024-03-28T01:08:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.546+0000 7fb075681640 1 -- 172.21.15.169:0/672843497 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb07019f880 con 0x7fb0700ff6f0 2024-03-28T01:08:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.546+0000 7fb057fff640 1 -- 172.21.15.169:0/672843497 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb064018a00 con 0x7fb0700ff6f0 2024-03-28T01:08:47.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.546+0000 7fb075681640 1 -- 172.21.15.169:0/672843497 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb07019fe20 con 0x7fb0700ff6f0 2024-03-28T01:08:47.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.548+0000 7fb057fff640 1 -- 172.21.15.169:0/672843497 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb06401f020 con 0x7fb0700ff6f0 2024-03-28T01:08:47.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.548+0000 7fb07826d740 1 Processor -- start 2024-03-28T01:08:47.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.549+0000 7fb07826d740 1 -- start start 2024-03-28T01:08:47.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.549+0000 7fb057fff640 1 --2- 172.21.15.169:0/672843497 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb0580a2880 0x7fb0580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.549+0000 7fb07826d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9a690 0x5581b6b9aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.549+0000 7fb07826d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6b9b080 0x5581b6b9fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.549+0000 7fb07826d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6ba01f0 0x5581b6ba2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.549+0000 7fb07826d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581b6b5d610 con 0x5581b6b9a690 2024-03-28T01:08:47.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.549+0000 7fb07826d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581b6b003c0 con 0x5581b6ba01f0 2024-03-28T01:08:47.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.549+0000 7fb07826d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581b6aa8d60 con 0x5581b6b9b080 2024-03-28T01:08:47.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.549+0000 7fb06f7fe640 1 --2- 172.21.15.169:0/672843497 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb0580a2880 0x7fb0580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.549+0000 7fb057fff640 1 -- 172.21.15.169:0/672843497 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb0640cefc0 con 0x7fb0700ff6f0 2024-03-28T01:08:47.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.549+0000 7fb06effd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9a690 0x5581b6b9aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.549+0000 7fb06e7fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6b9b080 0x5581b6b9fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.549+0000 7fb06e7fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6b9b080 0x5581b6b9fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48566/0 (socket says 172.21.15.169:48566) 2024-03-28T01:08:47.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.549+0000 7fb06e7fc640 1 -- 172.21.15.169:0/2508609064 learned_addr learned my addr 172.21.15.169:0/2508609064 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:47.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.549+0000 7fb06e7fc640 1 -- 172.21.15.169:0/2508609064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6ba01f0 msgr2=0x5581b6ba2680 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T01:08:47.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.549+0000 7fb06e7fc640 1 --2- 172.21.15.169:0/2508609064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6ba01f0 0x5581b6ba2680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.549+0000 7fb06e7fc640 1 -- 172.21.15.169:0/2508609064 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9a690 msgr2=0x5581b6b9aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.549+0000 7fb06e7fc640 1 --2- 172.21.15.169:0/2508609064 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9a690 0x5581b6b9aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.549+0000 7fb06e7fc640 1 -- 172.21.15.169:0/2508609064 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581b68143f0 con 0x5581b6b9b080 2024-03-28T01:08:47.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.549+0000 7fb06f7fe640 1 --2- 172.21.15.169:0/672843497 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb0580a2880 0x7fb0580a4d40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb06802c630 tx=0x7fb068032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:47.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.549+0000 7fb06e7fc640 1 --2- 172.21.15.169:0/2508609064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6b9b080 0x5581b6b9fb80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fb060009c90 tx=0x7fb060011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d9a297ed1ce5881 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:47.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.550+0000 7fb0557fa640 1 -- 172.21.15.169:0/2508609064 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06001e070 con 0x5581b6b9b080 2024-03-28T01:08:47.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.550+0000 7fb0557fa640 1 -- 172.21.15.169:0/2508609064 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb060019b10 con 0x5581b6b9b080 2024-03-28T01:08:47.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.550+0000 7fb0557fa640 1 -- 172.21.15.169:0/2508609064 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06000eac0 con 0x5581b6b9b080 2024-03-28T01:08:47.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.550+0000 7fb07826d740 1 -- 172.21.15.169:0/2508609064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6b9b080 msgr2=0x5581b6b9fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.550+0000 7fb07826d740 1 --2- 172.21.15.169:0/2508609064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6b9b080 0x5581b6b9fb80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fb060009c90 tx=0x7fb060011810 comp rx=0 tx=0).stop 2024-03-28T01:08:47.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.550+0000 7fb07826d740 1 -- 172.21.15.169:0/2508609064 shutdown_connections 2024-03-28T01:08:47.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.550+0000 7fb07826d740 1 --2- 172.21.15.169:0/2508609064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6ba01f0 0x5581b6ba2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.550+0000 7fb07826d740 1 --2- 172.21.15.169:0/2508609064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6b9b080 0x5581b6b9fb80 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.550+0000 7fb07826d740 1 --2- 172.21.15.169:0/2508609064 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9a690 0x5581b6b9aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.550+0000 7fb07826d740 1 -- 172.21.15.169:0/2508609064 >> 172.21.15.169:0/2508609064 conn(0x5581b6b95c00 msgr2=0x5581b6b96000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:47.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.550+0000 7fb07826d740 1 -- 172.21.15.169:0/2508609064 shutdown_connections 2024-03-28T01:08:47.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.550+0000 7fb07826d740 1 -- 172.21.15.169:0/2508609064 wait complete. 2024-03-28T01:08:47.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.551+0000 7fb07826d740 1 -- start start 2024-03-28T01:08:47.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.551+0000 7fb07826d740 1 Processor -- start 2024-03-28T01:08:47.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.551+0000 7fb07826d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6b9a690 0x5581b6ba4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.551+0000 7fb07826d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9b080 0x5581b6ba5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.551+0000 7fb07826d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6ba01f0 0x5581b6babfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.551+0000 7fb07826d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581b6abc240 con 0x5581b6b9b080 2024-03-28T01:08:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.551+0000 7fb07826d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581b6afc460 con 0x5581b6b9a690 2024-03-28T01:08:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.551+0000 7fb07826d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581b6ba31a0 con 0x5581b6ba01f0 2024-03-28T01:08:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.551+0000 7fb07826d740 10 client.0 _ll_register_callbacks cb 0x5581b6abbc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.551+0000 7fb06e7fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9b080 0x5581b6ba5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.551+0000 7fb06effd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6b9a690 0x5581b6ba4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.551+0000 7fb06e7fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9b080 0x5581b6ba5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53434/0 (socket says 172.21.15.169:53434) 2024-03-28T01:08:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.551+0000 7fb06f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6ba01f0 0x5581b6babfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.551+0000 7fb06e7fc640 1 -- 172.21.15.169:0/4110214995 learned_addr learned my addr 172.21.15.169:0/4110214995 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.551+0000 7fb06effd640 1 -- 172.21.15.169:0/4110214995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6ba01f0 msgr2=0x5581b6babfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.551+0000 7fb06effd640 1 --2- 172.21.15.169:0/4110214995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6ba01f0 0x5581b6babfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.551+0000 7fb06effd640 1 -- 172.21.15.169:0/4110214995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9b080 msgr2=0x5581b6ba5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.552+0000 7fb06effd640 1 --2- 172.21.15.169:0/4110214995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9b080 0x5581b6ba5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.552+0000 7fb06effd640 1 -- 172.21.15.169:0/4110214995 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5581b68de7d0 con 0x5581b6b9a690 2024-03-28T01:08:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.552+0000 7fb06effd640 1 --2- 172.21.15.169:0/4110214995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6b9a690 0x5581b6ba4ad0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb06400c6a0 tx=0x7fb06409af80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e10d2884c9a1c0b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.552+0000 7fb052ff5640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-03-28T01:08:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.552+0000 7fb052ff5640 1 -- 172.21.15.169:0/4110214995 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06409bc60 con 0x5581b6b9a690 2024-03-28T01:08:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.552+0000 7fb052ff5640 1 -- 172.21.15.169:0/4110214995 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb064010ac0 con 0x5581b6b9a690 2024-03-28T01:08:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.552+0000 7fb07826d740 1 -- 172.21.15.169:0/4110214995 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb060019b10 con 0x5581b6b9a690 2024-03-28T01:08:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.552+0000 7fb052ff5640 1 -- 172.21.15.169:0/4110214995 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb06401e4a0 con 0x5581b6b9a690 2024-03-28T01:08:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.552+0000 7fb07826d740 10 client.27605 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.552+0000 7fb07826d740 1 -- 172.21.15.169:0/4110214995 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5581b6baf3c0 con 0x5581b6b9a690 2024-03-28T01:08:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.552+0000 7fb052ff5640 1 -- 172.21.15.169:0/4110214995 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb060019b10 con 0x5581b6b9a690 2024-03-28T01:08:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.552+0000 7fb052ff5640 1 -- 172.21.15.169:0/4110214995 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb064013ad0 con 0x5581b6b9a690 2024-03-28T01:08:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.552+0000 7fb07826d740 10 client.27605 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.552+0000 7fb07826d740 10 client.27605 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:08:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.552+0000 7fb07826d740 20 client.27605 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.552+0000 7fb07826d740 1 --2- 172.21.15.169:0/4110214995 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5581b6bb09d0 0x5581b6bb2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.552+0000 7fb07826d740 4 client.27605 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T01:08:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.552+0000 7fb07826d740 1 -- 172.21.15.169:0/4110214995 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5581b6b0d900 con 0x5581b6bb09d0 2024-03-28T01:08:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.553+0000 7fb06e7fc640 1 --2- 172.21.15.169:0/4110214995 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5581b6bb09d0 0x5581b6bb2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.553+0000 7fb052ff5640 10 client.27605 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.553+0000 7fb06e7fc640 1 --2- 172.21.15.169:0/4110214995 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5581b6bb09d0 0x5581b6bb2e20 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d9dd4a2d371d0b server_cookie=dc79046ca8428126 in_seq=0 out_seq=0 2024-03-28T01:08:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.553+0000 7fb052ff5640 1 -- 172.21.15.169:0/4110214995 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5581b6b0d900 con 0x5581b6bb09d0 2024-03-28T01:08:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.553+0000 7fb052ff5640 10 client.27605 handle_command_reply: tid=0 2024-03-28T01:08:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.554+0000 7fb07826d740 1 client.27605 shutdown 2024-03-28T01:08:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.554+0000 7fb07826d740 1 -- 172.21.15.169:0/4110214995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6b9a690 msgr2=0x5581b6ba4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.554+0000 7fb07826d740 1 --2- 172.21.15.169:0/4110214995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6b9a690 0x5581b6ba4ad0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb06400c6a0 tx=0x7fb06409af80 comp rx=0 tx=0).stop 2024-03-28T01:08:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.554+0000 7fb07826d740 1 -- 172.21.15.169:0/4110214995 shutdown_connections 2024-03-28T01:08:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.554+0000 7fb07826d740 1 --2- 172.21.15.169:0/4110214995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6ba01f0 0x5581b6babfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.554+0000 7fb07826d740 1 --2- 172.21.15.169:0/4110214995 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5581b6bb09d0 0x5581b6bb2e20 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.554+0000 7fb07826d740 1 --2- 172.21.15.169:0/4110214995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9b080 0x5581b6ba5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.554+0000 7fb07826d740 1 --2- 172.21.15.169:0/4110214995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6b9a690 0x5581b6ba4ad0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.554+0000 7fb052ff5640 0 client.27605 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:47.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.554+0000 7fb07826d740 1 -- 172.21.15.169:0/4110214995 >> 172.21.15.169:0/4110214995 conn(0x5581b6b95c00 msgr2=0x5581b6b94e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:47.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.554+0000 7fb07826d740 1 -- 172.21.15.169:0/4110214995 shutdown_connections 2024-03-28T01:08:47.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.554+0000 7fb07826d740 1 -- 172.21.15.169:0/4110214995 wait complete. 2024-03-28T01:08:47.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.554+0000 7fb07826d740 20 client.27605 trim_cache size 0 max 16384 2024-03-28T01:08:47.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.566+0000 7fb07826d740 1 Processor -- start 2024-03-28T01:08:47.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.566+0000 7fb07826d740 1 -- start start 2024-03-28T01:08:47.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.567+0000 7fb07826d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6b9a690 0x5581b6ba46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.567+0000 7fb07826d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9b080 0x5581b6babfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.567+0000 7fb07826d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6ba01f0 0x5581b6ba4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.567+0000 7fb07826d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0701ac660 con 0x5581b6b9b080 2024-03-28T01:08:47.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.567+0000 7fb07826d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0701aa160 con 0x5581b6b9a690 2024-03-28T01:08:47.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.567+0000 7fb07826d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0701a8200 con 0x5581b6ba01f0 2024-03-28T01:08:47.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.567+0000 7fb06f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6ba01f0 0x5581b6ba4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.567+0000 7fb06effd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6b9a690 0x5581b6ba46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.567+0000 7fb06e7fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9b080 0x5581b6babfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.567+0000 7fb06effd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6b9a690 0x5581b6ba46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:55448/0 (socket says 172.21.15.169:55448) 2024-03-28T01:08:47.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.567+0000 7fb06f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6ba01f0 0x5581b6ba4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48586/0 (socket says 172.21.15.169:48586) 2024-03-28T01:08:47.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.567+0000 7fb06effd640 1 -- 172.21.15.169:0/1069761962 learned_addr learned my addr 172.21.15.169:0/1069761962 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:47.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.567+0000 7fb06e7fc640 1 -- 172.21.15.169:0/1069761962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6ba01f0 msgr2=0x5581b6ba4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.567+0000 7fb06e7fc640 1 --2- 172.21.15.169:0/1069761962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6ba01f0 0x5581b6ba4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.567+0000 7fb06e7fc640 1 -- 172.21.15.169:0/1069761962 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6b9a690 msgr2=0x5581b6ba46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.567+0000 7fb06e7fc640 1 --2- 172.21.15.169:0/1069761962 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6b9a690 0x5581b6ba46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.567+0000 7fb06e7fc640 1 -- 172.21.15.169:0/1069761962 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581b6b9b460 con 0x5581b6b9b080 2024-03-28T01:08:47.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.567+0000 7fb06f7fe640 1 --2- 172.21.15.169:0/1069761962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6ba01f0 0x5581b6ba4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:08:47.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.567+0000 7fb06effd640 1 --2- 172.21.15.169:0/1069761962 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6b9a690 0x5581b6ba46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:08:47.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.567+0000 7fb06e7fc640 1 --2- 172.21.15.169:0/1069761962 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9b080 0x5581b6babfa0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x5581b6bb3660 tx=0x7fb0600097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afad4bcd75e4d34b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:47.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.568+0000 7fb054ff9640 1 -- 172.21.15.169:0/1069761962 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb060033070 con 0x5581b6b9b080 2024-03-28T01:08:47.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.568+0000 7fb054ff9640 1 -- 172.21.15.169:0/1069761962 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb06001f040 con 0x5581b6b9b080 2024-03-28T01:08:47.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.568+0000 7fb054ff9640 1 -- 172.21.15.169:0/1069761962 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06001e040 con 0x5581b6b9b080 2024-03-28T01:08:47.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.568+0000 7fb07826d740 1 -- 172.21.15.169:0/1069761962 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9b080 msgr2=0x5581b6babfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.568+0000 7fb07826d740 1 --2- 172.21.15.169:0/1069761962 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9b080 0x5581b6babfa0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x5581b6bb3660 tx=0x7fb0600097a0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.568+0000 7fb07826d740 1 -- 172.21.15.169:0/1069761962 shutdown_connections 2024-03-28T01:08:47.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.568+0000 7fb07826d740 1 --2- 172.21.15.169:0/1069761962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6ba01f0 0x5581b6ba4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.568+0000 7fb07826d740 1 --2- 172.21.15.169:0/1069761962 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9b080 0x5581b6babfa0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.568+0000 7fb07826d740 1 --2- 172.21.15.169:0/1069761962 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6b9a690 0x5581b6ba46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.568+0000 7fb07826d740 1 -- 172.21.15.169:0/1069761962 >> 172.21.15.169:0/1069761962 conn(0x5581b6b95c00 msgr2=0x5581b6b95fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:47.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.569+0000 7fb07826d740 1 -- 172.21.15.169:0/1069761962 shutdown_connections 2024-03-28T01:08:47.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.569+0000 7fb07826d740 1 -- 172.21.15.169:0/1069761962 wait complete. 2024-03-28T01:08:47.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.569+0000 7fb07826d740 1 -- start start 2024-03-28T01:08:47.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.569+0000 7fb07826d740 1 Processor -- start 2024-03-28T01:08:47.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.569+0000 7fb07826d740 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9a690 0x5581b6ba96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.569+0000 7fb07826d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6b9b080 0x5581b6ba9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.569+0000 7fb07826d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6ba01f0 0x5581b6baabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.569+0000 7fb07826d740 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0701a8800 con 0x5581b6b9a690 2024-03-28T01:08:47.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.569+0000 7fb07826d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0701a9b00 con 0x5581b6ba01f0 2024-03-28T01:08:47.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.569+0000 7fb07826d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0701a8e40 con 0x5581b6b9b080 2024-03-28T01:08:47.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.569+0000 7fb07826d740 10 client.0 _ll_register_callbacks cb 0x7fb06409b330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T01:08:47.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.569+0000 7fb06e7fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6b9b080 0x5581b6ba9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.569+0000 7fb06effd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9a690 0x5581b6ba96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.569+0000 7fb06e7fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6b9b080 0x5581b6ba9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48592/0 (socket says 172.21.15.169:48592) 2024-03-28T01:08:47.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.569+0000 7fb06e7fc640 1 -- 172.21.15.169:0/934947732 learned_addr learned my addr 172.21.15.169:0/934947732 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:47.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.569+0000 7fb06f7fe640 1 --2- 172.21.15.169:0/934947732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6ba01f0 0x5581b6baabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.570+0000 7fb06e7fc640 1 -- 172.21.15.169:0/934947732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6ba01f0 msgr2=0x5581b6baabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.570+0000 7fb06e7fc640 1 --2- 172.21.15.169:0/934947732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6ba01f0 0x5581b6baabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.570+0000 7fb06e7fc640 1 -- 172.21.15.169:0/934947732 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9a690 msgr2=0x5581b6ba96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T01:08:47.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.570+0000 7fb06e7fc640 1 --2- 172.21.15.169:0/934947732 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9a690 0x5581b6ba96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.570+0000 7fb06e7fc640 1 -- 172.21.15.169:0/934947732 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5581b6bab1b0 con 0x5581b6b9b080 2024-03-28T01:08:47.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.570+0000 7fb06f7fe640 1 --2- 172.21.15.169:0/934947732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6ba01f0 0x5581b6baabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:08:47.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.570+0000 7fb06e7fc640 1 --2- 172.21.15.169:0/934947732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6b9b080 0x5581b6ba9c00 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb060009460 tx=0x7fb060009bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1045e00e75da8209 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:47.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.570+0000 7fb052ff5640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-03-28T01:08:47.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.570+0000 7fb052ff5640 1 -- 172.21.15.169:0/934947732 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb060033070 con 0x5581b6b9b080 2024-03-28T01:08:47.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.570+0000 7fb052ff5640 1 -- 172.21.15.169:0/934947732 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb060005030 con 0x5581b6b9b080 2024-03-28T01:08:47.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.570+0000 7fb07826d740 1 -- 172.21.15.169:0/934947732 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb06001f040 con 0x5581b6b9b080 2024-03-28T01:08:47.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.570+0000 7fb052ff5640 1 -- 172.21.15.169:0/934947732 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb06003f050 con 0x5581b6b9b080 2024-03-28T01:08:47.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.570+0000 7fb07826d740 10 client.27568 fetch_fsmap learned FSMap version 25 2024-03-28T01:08:47.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.570+0000 7fb07826d740 1 -- 172.21.15.169:0/934947732 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5581b6c0c280 con 0x5581b6b9b080 2024-03-28T01:08:47.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.570+0000 7fb052ff5640 1 -- 172.21.15.169:0/934947732 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb06001f040 con 0x5581b6b9b080 2024-03-28T01:08:47.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.570+0000 7fb052ff5640 1 -- 172.21.15.169:0/934947732 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) v1 ==== 7926+0+0 (secure 0 0 0) 0x7fb06002e040 con 0x5581b6b9b080 2024-03-28T01:08:47.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.571+0000 7fb07826d740 10 client.27568 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T01:08:47.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.571+0000 7fb07826d740 10 client.27568 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T01:08:47.581 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.571+0000 7fb07826d740 20 client.27568 populate_metadata read hostname 'smithi169' 2024-03-28T01:08:47.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.571+0000 7fb07826d740 1 --2- 172.21.15.169:0/934947732 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5581b6c0c990 0x5581b6c0edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:47.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.571+0000 7fb07826d740 4 client.27568 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T01:08:47.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.571+0000 7fb07826d740 1 -- 172.21.15.169:0/934947732 --> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5581b6b07c50 con 0x5581b6c0c990 2024-03-28T01:08:47.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.571+0000 7fb06effd640 1 --2- 172.21.15.169:0/934947732 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5581b6c0c990 0x5581b6c0edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:47.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.571+0000 7fb052ff5640 10 client.27568 ms_handle_connect on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:47.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.571+0000 7fb06effd640 1 --2- 172.21.15.169:0/934947732 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5581b6c0c990 0x5581b6c0edb0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=156c0bdf7b4f24bb server_cookie=c5a67eaf54f786d0 in_seq=0 out_seq=0 2024-03-28T01:08:47.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.571+0000 7fb052ff5640 1 -- 172.21.15.169:0/934947732 <== mds.0 v2:172.21.15.174:6838/1152414321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5581b6b07c50 con 0x5581b6c0c990 2024-03-28T01:08:47.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.571+0000 7fb052ff5640 10 client.27568 handle_command_reply: tid=0 2024-03-28T01:08:47.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.571+0000 7fb07826d740 1 client.27568 shutdown 2024-03-28T01:08:47.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.571+0000 7fb07826d740 1 -- 172.21.15.169:0/934947732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6b9b080 msgr2=0x5581b6ba9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.571+0000 7fb07826d740 1 --2- 172.21.15.169:0/934947732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6b9b080 0x5581b6ba9c00 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb060009460 tx=0x7fb060009bc0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.571+0000 7fb07826d740 1 -- 172.21.15.169:0/934947732 shutdown_connections 2024-03-28T01:08:47.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.571+0000 7fb07826d740 1 --2- 172.21.15.169:0/934947732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b6ba01f0 0x5581b6baabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.571+0000 7fb07826d740 1 --2- 172.21.15.169:0/934947732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5581b6b9b080 0x5581b6ba9c00 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.584 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.571+0000 7fb07826d740 1 --2- 172.21.15.169:0/934947732 >> [v2:172.21.15.174:6838/1152414321,v1:172.21.15.174:6839/1152414321] conn(0x5581b6c0c990 0x5581b6c0edb0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.584 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.571+0000 7fb07826d740 1 --2- 172.21.15.169:0/934947732 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x5581b6b9a690 0x5581b6ba96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.584 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.571+0000 7fb052ff5640 0 client.27568 ms_handle_reset on v2:172.21.15.174:6838/1152414321 2024-03-28T01:08:47.584 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.571+0000 7fb07826d740 1 -- 172.21.15.169:0/934947732 >> 172.21.15.169:0/934947732 conn(0x5581b6b95c00 msgr2=0x5581b6ba46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:47.584 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.572+0000 7fb07826d740 1 -- 172.21.15.169:0/934947732 shutdown_connections 2024-03-28T01:08:47.584 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.572+0000 7fb07826d740 1 -- 172.21.15.169:0/934947732 wait complete. 2024-03-28T01:08:47.584 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.572+0000 7fb07826d740 20 client.27568 trim_cache size 0 max 16384 2024-03-28T01:08:47.584 INFO:teuthology.orchestra.run.smithi169.stdout:[] 2024-03-28T01:08:47.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.572+0000 7fb075681640 1 -- 172.21.15.169:0/672843497 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb0580a2880 msgr2=0x7fb0580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.572+0000 7fb075681640 1 --2- 172.21.15.169:0/672843497 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb0580a2880 0x7fb0580a4d40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb06802c630 tx=0x7fb068032590 comp rx=0 tx=0).stop 2024-03-28T01:08:47.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.572+0000 7fb075681640 1 -- 172.21.15.169:0/672843497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0700ff6f0 msgr2=0x7fb070197970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:47.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.572+0000 7fb075681640 1 --2- 172.21.15.169:0/672843497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0700ff6f0 0x7fb070197970 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb06400e9c0 tx=0x7fb06400ee80 comp rx=0 tx=0).stop 2024-03-28T01:08:47.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.572+0000 7fb075681640 1 -- 172.21.15.169:0/672843497 shutdown_connections 2024-03-28T01:08:47.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.572+0000 7fb075681640 1 --2- 172.21.15.169:0/672843497 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb0580a2880 0x7fb0580a4d40 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.572+0000 7fb075681640 1 --2- 172.21.15.169:0/672843497 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb070102c80 0x7fb07019ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.572+0000 7fb075681640 1 --2- 172.21.15.169:0/672843497 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0701000b0 0x7fb070197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.572+0000 7fb075681640 1 --2- 172.21.15.169:0/672843497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0700ff6f0 0x7fb070197970 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:47.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.572+0000 7fb075681640 1 -- 172.21.15.169:0/672843497 >> 172.21.15.169:0/672843497 conn(0x7fb0700fb290 msgr2=0x7fb070105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:47.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.572+0000 7fb075681640 1 -- 172.21.15.169:0/672843497 shutdown_connections 2024-03-28T01:08:47.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:47.572+0000 7fb075681640 1 -- 172.21.15.169:0/672843497 wait complete. 2024-03-28T01:08:48.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:47 smithi174 ceph-mon[36060]: pgmap v719: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 67 MiB/s wr, 326 op/s 2024-03-28T01:08:48.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:47 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3631918448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:08:48.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:47 smithi177 ceph-mon[36206]: pgmap v719: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 67 MiB/s wr, 326 op/s 2024-03-28T01:08:48.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:47 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3631918448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:08:48.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:47 smithi169 ceph-mon[32274]: pgmap v719: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 67 MiB/s wr, 326 op/s 2024-03-28T01:08:48.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:47 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3631918448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:08:48.591 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-28T01:08:48.591 INFO:tasks.fwd_scrub:done joining 2024-03-28T01:08:48.591 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-28T01:08:48.602 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T01:08:48.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.714+0000 7fbd98d97640 1 -- 172.21.15.169:0/1607021587 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd941017b0 msgr2=0x7fbd94101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:48.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.714+0000 7fbd98d97640 1 --2- 172.21.15.169:0/1607021587 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd941017b0 0x7fbd94101c30 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c009900 tx=0x7fbd7c02bc30 comp rx=0 tx=0).stop 2024-03-28T01:08:48.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.714+0000 7fbd98d97640 1 -- 172.21.15.169:0/1607021587 shutdown_connections 2024-03-28T01:08:48.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.714+0000 7fbd98d97640 1 --2- 172.21.15.169:0/1607021587 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd94102170 0x7fbd94108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:48.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.714+0000 7fbd98d97640 1 --2- 172.21.15.169:0/1607021587 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd941017b0 0x7fbd94101c30 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:48.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.714+0000 7fbd98d97640 1 --2- 172.21.15.169:0/1607021587 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd941005c0 0x7fbd941009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:48.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.714+0000 7fbd98d97640 1 -- 172.21.15.169:0/1607021587 >> 172.21.15.169:0/1607021587 conn(0x7fbd940fbb30 msgr2=0x7fbd940fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:48.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.714+0000 7fbd98d97640 1 -- 172.21.15.169:0/1607021587 shutdown_connections 2024-03-28T01:08:48.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.714+0000 7fbd98d97640 1 -- 172.21.15.169:0/1607021587 wait complete. 2024-03-28T01:08:48.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.715+0000 7fbd98d97640 1 Processor -- start 2024-03-28T01:08:48.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.715+0000 7fbd98d97640 1 -- start start 2024-03-28T01:08:48.720 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.715+0000 7fbd98d97640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd941005c0 0x7fbd94069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:48.720 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.715+0000 7fbd98d97640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd941017b0 0x7fbd9419c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:48.720 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.715+0000 7fbd98d97640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd94102170 0x7fbd9419ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:48.720 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.715+0000 7fbd98d97640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9410c240 con 0x7fbd94102170 2024-03-28T01:08:48.720 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.715+0000 7fbd98d97640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9410c0c0 con 0x7fbd941005c0 2024-03-28T01:08:48.720 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.715+0000 7fbd98d97640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9410c3c0 con 0x7fbd941017b0 2024-03-28T01:08:48.720 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.715+0000 7fbd9259c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd941005c0 0x7fbd94069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:48.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.715+0000 7fbd92d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd94102170 0x7fbd9419ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:48.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.715+0000 7fbd91d9b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd941017b0 0x7fbd9419c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:48.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.715+0000 7fbd9259c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd941005c0 0x7fbd94069370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:55462/0 (socket says 172.21.15.169:55462) 2024-03-28T01:08:48.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.715+0000 7fbd92d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd94102170 0x7fbd9419ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53476/0 (socket says 172.21.15.169:53476) 2024-03-28T01:08:48.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.715+0000 7fbd9259c640 1 -- 172.21.15.169:0/3715729473 learned_addr learned my addr 172.21.15.169:0/3715729473 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:48.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.715+0000 7fbd92d9d640 1 -- 172.21.15.169:0/3715729473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd941017b0 msgr2=0x7fbd9419c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:48.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.715+0000 7fbd92d9d640 1 --2- 172.21.15.169:0/3715729473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd941017b0 0x7fbd9419c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:48.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.715+0000 7fbd92d9d640 1 -- 172.21.15.169:0/3715729473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd941005c0 msgr2=0x7fbd94069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:48.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.715+0000 7fbd92d9d640 1 --2- 172.21.15.169:0/3715729473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd941005c0 0x7fbd94069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:48.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.715+0000 7fbd92d9d640 1 -- 172.21.15.169:0/3715729473 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9419f660 con 0x7fbd94102170 2024-03-28T01:08:48.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.715+0000 7fbd9259c640 1 --2- 172.21.15.169:0/3715729473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd941005c0 0x7fbd94069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:08:48.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.715+0000 7fbd92d9d640 1 --2- 172.21.15.169:0/3715729473 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd94102170 0x7fbd9419ed50 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fbd8400e970 tx=0x7fbd8400ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:48.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.716+0000 7fbd8b7fe640 1 -- 172.21.15.169:0/3715729473 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8400ccc0 con 0x7fbd94102170 2024-03-28T01:08:48.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.716+0000 7fbd98d97640 1 -- 172.21.15.169:0/3715729473 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd9419f8f0 con 0x7fbd94102170 2024-03-28T01:08:48.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.716+0000 7fbd8b7fe640 1 -- 172.21.15.169:0/3715729473 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd84005020 con 0x7fbd94102170 2024-03-28T01:08:48.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.716+0000 7fbd8b7fe640 1 -- 172.21.15.169:0/3715729473 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd84004a00 con 0x7fbd94102170 2024-03-28T01:08:48.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.716+0000 7fbd98d97640 1 -- 172.21.15.169:0/3715729473 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd9419fe60 con 0x7fbd94102170 2024-03-28T01:08:48.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.717+0000 7fbd98d97640 1 -- 172.21.15.169:0/3715729473 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd941078e0 con 0x7fbd94102170 2024-03-28T01:08:48.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.718+0000 7fbd8b7fe640 1 -- 172.21.15.169:0/3715729473 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbd840051d0 con 0x7fbd94102170 2024-03-28T01:08:48.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.718+0000 7fbd8b7fe640 1 --2- 172.21.15.169:0/3715729473 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbd800a2800 0x7fbd800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:48.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.718+0000 7fbd8b7fe640 1 -- 172.21.15.169:0/3715729473 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fbd840ced40 con 0x7fbd94102170 2024-03-28T01:08:48.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.718+0000 7fbd9259c640 1 --2- 172.21.15.169:0/3715729473 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbd800a2800 0x7fbd800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:48.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.719+0000 7fbd9259c640 1 --2- 172.21.15.169:0/3715729473 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbd800a2800 0x7fbd800a4cc0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c005c40 tx=0x7fbd8c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:48.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.723+0000 7fbd8b7fe640 1 -- 172.21.15.169:0/3715729473 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbd8409ba40 con 0x7fbd94102170 2024-03-28T01:08:48.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.901+0000 7fbd98d97640 1 -- 172.21.15.169:0/3715729473 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd940622a0 con 0x7fbd94102170 2024-03-28T01:08:48.906 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.903+0000 7fbd8b7fe640 1 -- 172.21.15.169:0/3715729473 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+24823 (secure 0 0 0) 0x7fbd84081110 con 0x7fbd94102170 2024-03-28T01:08:48.906 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T01:08:48.907 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":94,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:56:38.281826+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9},{"from":5,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T01:08:48.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.905+0000 7fbd98d97640 1 -- 172.21.15.169:0/3715729473 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbd800a2800 msgr2=0x7fbd800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:48.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.905+0000 7fbd98d97640 1 --2- 172.21.15.169:0/3715729473 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbd800a2800 0x7fbd800a4cc0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c005c40 tx=0x7fbd8c009500 comp rx=0 tx=0).stop 2024-03-28T01:08:48.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.905+0000 7fbd98d97640 1 -- 172.21.15.169:0/3715729473 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd94102170 msgr2=0x7fbd9419ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:48.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.905+0000 7fbd98d97640 1 --2- 172.21.15.169:0/3715729473 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd94102170 0x7fbd9419ed50 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fbd8400e970 tx=0x7fbd8400ee30 comp rx=0 tx=0).stop 2024-03-28T01:08:48.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.905+0000 7fbd98d97640 1 -- 172.21.15.169:0/3715729473 shutdown_connections 2024-03-28T01:08:48.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.905+0000 7fbd98d97640 1 --2- 172.21.15.169:0/3715729473 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fbd800a2800 0x7fbd800a4cc0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:48.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.905+0000 7fbd98d97640 1 --2- 172.21.15.169:0/3715729473 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd94102170 0x7fbd9419ed50 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:48.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.905+0000 7fbd98d97640 1 --2- 172.21.15.169:0/3715729473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd941017b0 0x7fbd9419c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:48.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.905+0000 7fbd98d97640 1 --2- 172.21.15.169:0/3715729473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd941005c0 0x7fbd94069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:48.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.905+0000 7fbd98d97640 1 -- 172.21.15.169:0/3715729473 >> 172.21.15.169:0/3715729473 conn(0x7fbd940fbb30 msgr2=0x7fbd94104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:48.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.905+0000 7fbd98d97640 1 -- 172.21.15.169:0/3715729473 shutdown_connections 2024-03-28T01:08:48.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:48.905+0000 7fbd98d97640 1 -- 172.21.15.169:0/3715729473 wait complete. 2024-03-28T01:08:48.920 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T00:49:56.339525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T00:54:31.182562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T00:54:31.780429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T00:54:35.035724+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T01:08:48.921 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T01:08:49.029 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.024+0000 7fd98abc6640 1 -- 172.21.15.169:0/2577206859 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd984103260 msgr2=0x7fd984109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.024+0000 7fd98abc6640 1 --2- 172.21.15.169:0/2577206859 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd984103260 0x7fd984109af0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fd97c009900 tx=0x7fd97c02bc30 comp rx=0 tx=0).stop 2024-03-28T01:08:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.024+0000 7fd98abc6640 1 -- 172.21.15.169:0/2577206859 shutdown_connections 2024-03-28T01:08:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.024+0000 7fd98abc6640 1 --2- 172.21.15.169:0/2577206859 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd984103260 0x7fd984109af0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.024+0000 7fd98abc6640 1 --2- 172.21.15.169:0/2577206859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9841028a0 0x7fd984102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.024+0000 7fd98abc6640 1 --2- 172.21.15.169:0/2577206859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9841016b0 0x7fd984101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.024+0000 7fd98abc6640 1 -- 172.21.15.169:0/2577206859 >> 172.21.15.169:0/2577206859 conn(0x7fd9840fcc40 msgr2=0x7fd9840ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.024+0000 7fd98abc6640 1 -- 172.21.15.169:0/2577206859 shutdown_connections 2024-03-28T01:08:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.024+0000 7fd98abc6640 1 -- 172.21.15.169:0/2577206859 wait complete. 2024-03-28T01:08:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.024+0000 7fd98abc6640 1 Processor -- start 2024-03-28T01:08:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.024+0000 7fd98abc6640 1 -- start start 2024-03-28T01:08:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd98abc6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9841016b0 0x7fd984075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd98abc6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9841028a0 0x7fd984073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:49.030 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd98abc6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd984103260 0x7fd984073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:49.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd98abc6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98410d330 con 0x7fd9841016b0 2024-03-28T01:08:49.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd98abc6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98410d1b0 con 0x7fd984103260 2024-03-28T01:08:49.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd98abc6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98410d4b0 con 0x7fd9841028a0 2024-03-28T01:08:49.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd988962640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9841016b0 0x7fd984075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:49.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd988962640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9841016b0 0x7fd984075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53488/0 (socket says 172.21.15.169:53488) 2024-03-28T01:08:49.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd988962640 1 -- 172.21.15.169:0/3273436473 learned_addr learned my addr 172.21.15.169:0/3273436473 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:49.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd983fff640 1 --2- 172.21.15.169:0/3273436473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9841028a0 0x7fd984073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:49.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd989163640 1 --2- 172.21.15.169:0/3273436473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd984103260 0x7fd984073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:49.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd988962640 1 -- 172.21.15.169:0/3273436473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9841028a0 msgr2=0x7fd984073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:49.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd988962640 1 --2- 172.21.15.169:0/3273436473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9841028a0 0x7fd984073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd988962640 1 -- 172.21.15.169:0/3273436473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd984103260 msgr2=0x7fd984073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:49.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd988962640 1 --2- 172.21.15.169:0/3273436473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd984103260 0x7fd984073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd988962640 1 -- 172.21.15.169:0/3273436473 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd984074860 con 0x7fd9841016b0 2024-03-28T01:08:49.031 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd988962640 1 --2- 172.21.15.169:0/3273436473 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9841016b0 0x7fd984075450 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fd97400ebc0 tx=0x7fd97400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:49.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd981ffb640 1 -- 172.21.15.169:0/3273436473 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd974003b70 con 0x7fd9841016b0 2024-03-28T01:08:49.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd981ffb640 1 -- 172.21.15.169:0/3273436473 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd974003d10 con 0x7fd9841016b0 2024-03-28T01:08:49.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd98abc6640 1 -- 172.21.15.169:0/3273436473 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9841a1a90 con 0x7fd9841016b0 2024-03-28T01:08:49.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd981ffb640 1 -- 172.21.15.169:0/3273436473 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd974019980 con 0x7fd9841016b0 2024-03-28T01:08:49.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.025+0000 7fd98abc6640 1 -- 172.21.15.169:0/3273436473 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9841a2030 con 0x7fd9841016b0 2024-03-28T01:08:49.032 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.026+0000 7fd98abc6640 1 -- 172.21.15.169:0/3273436473 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9841089d0 con 0x7fd9841016b0 2024-03-28T01:08:49.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.027+0000 7fd981ffb640 1 -- 172.21.15.169:0/3273436473 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd974019b20 con 0x7fd9841016b0 2024-03-28T01:08:49.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.028+0000 7fd981ffb640 1 --2- 172.21.15.169:0/3273436473 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd9700a2880 0x7fd9700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:49.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.028+0000 7fd983fff640 1 --2- 172.21.15.169:0/3273436473 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd9700a2880 0x7fd9700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:49.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.028+0000 7fd981ffb640 1 -- 172.21.15.169:0/3273436473 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fd9740cf5b0 con 0x7fd9841016b0 2024-03-28T01:08:49.033 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.028+0000 7fd983fff640 1 --2- 172.21.15.169:0/3273436473 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd9700a2880 0x7fd9700a4d40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd9840751c0 tx=0x7fd96c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:49.035 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.032+0000 7fd981ffb640 1 -- 172.21.15.169:0/3273436473 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd97409c2b0 con 0x7fd9841016b0 2024-03-28T01:08:49.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.207+0000 7fd98abc6640 1 -- 172.21.15.169:0/3273436473 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd9840622a0 con 0x7fd9841016b0 2024-03-28T01:08:49.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.207+0000 7fd981ffb640 1 -- 172.21.15.169:0/3273436473 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd97409bbb0 con 0x7fd9841016b0 2024-03-28T01:08:49.211 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 1 2024-03-28T01:08:49.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.209+0000 7fd98abc6640 1 -- 172.21.15.169:0/3273436473 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd9700a2880 msgr2=0x7fd9700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:49.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.209+0000 7fd98abc6640 1 --2- 172.21.15.169:0/3273436473 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd9700a2880 0x7fd9700a4d40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd9840751c0 tx=0x7fd96c005f90 comp rx=0 tx=0).stop 2024-03-28T01:08:49.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.209+0000 7fd98abc6640 1 -- 172.21.15.169:0/3273436473 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9841016b0 msgr2=0x7fd984075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:49.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.209+0000 7fd98abc6640 1 --2- 172.21.15.169:0/3273436473 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9841016b0 0x7fd984075450 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fd97400ebc0 tx=0x7fd97400c6a0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.209+0000 7fd98abc6640 1 -- 172.21.15.169:0/3273436473 shutdown_connections 2024-03-28T01:08:49.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.209+0000 7fd98abc6640 1 --2- 172.21.15.169:0/3273436473 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd9700a2880 0x7fd9700a4d40 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.209+0000 7fd98abc6640 1 --2- 172.21.15.169:0/3273436473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd984103260 0x7fd984073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.209+0000 7fd98abc6640 1 --2- 172.21.15.169:0/3273436473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9841028a0 0x7fd984073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.209+0000 7fd98abc6640 1 --2- 172.21.15.169:0/3273436473 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9841016b0 0x7fd984075450 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.209+0000 7fd98abc6640 1 -- 172.21.15.169:0/3273436473 >> 172.21.15.169:0/3273436473 conn(0x7fd9840fcc40 msgr2=0x7fd984105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:49.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.209+0000 7fd98abc6640 1 -- 172.21.15.169:0/3273436473 shutdown_connections 2024-03-28T01:08:49.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.209+0000 7fd98abc6640 1 -- 172.21.15.169:0/3273436473 wait complete. 2024-03-28T01:08:49.224 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T01:08:49.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.327+0000 7f2f1dcf1640 1 -- 172.21.15.169:0/2356496454 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f18102cc0 msgr2=0x7f2f181030e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:49.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.327+0000 7f2f1dcf1640 1 --2- 172.21.15.169:0/2356496454 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f18102cc0 0x7f2f181030e0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009900 tx=0x7f2f0802bc30 comp rx=0 tx=0).stop 2024-03-28T01:08:49.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.327+0000 7f2f1dcf1640 1 -- 172.21.15.169:0/2356496454 shutdown_connections 2024-03-28T01:08:49.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.327+0000 7f2f1dcf1640 1 --2- 172.21.15.169:0/2356496454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f181000f0 0x7f2f18109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.327+0000 7f2f1dcf1640 1 --2- 172.21.15.169:0/2356496454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f180ff730 0x7f2f180ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.327+0000 7f2f1dcf1640 1 --2- 172.21.15.169:0/2356496454 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f18102cc0 0x7f2f181030e0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.327+0000 7f2f1dcf1640 1 -- 172.21.15.169:0/2356496454 >> 172.21.15.169:0/2356496454 conn(0x7f2f180fb290 msgr2=0x7f2f180fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:49.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.327+0000 7f2f1dcf1640 1 -- 172.21.15.169:0/2356496454 shutdown_connections 2024-03-28T01:08:49.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.327+0000 7f2f1dcf1640 1 -- 172.21.15.169:0/2356496454 wait complete. 2024-03-28T01:08:49.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.327+0000 7f2f1dcf1640 1 Processor -- start 2024-03-28T01:08:49.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.327+0000 7f2f1dcf1640 1 -- start start 2024-03-28T01:08:49.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f1dcf1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f180ff730 0x7f2f181023c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:49.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f1dcf1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f181000f0 0x7f2f18100a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:49.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f1dcf1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f18102cc0 0x7f2f18100f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:49.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f1dcf1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1810d320 con 0x7f2f181000f0 2024-03-28T01:08:49.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f1dcf1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1810d1a0 con 0x7f2f18102cc0 2024-03-28T01:08:49.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f1dcf1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1810d4a0 con 0x7f2f180ff730 2024-03-28T01:08:49.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f0ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f181000f0 0x7f2f18100a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:49.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f0ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f181000f0 0x7f2f18100a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53508/0 (socket says 172.21.15.169:53508) 2024-03-28T01:08:49.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f0ffff640 1 -- 172.21.15.169:0/3908570736 learned_addr learned my addr 172.21.15.169:0/3908570736 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:49.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f177fe640 1 --2- 172.21.15.169:0/3908570736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f180ff730 0x7f2f181023c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:49.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f17fff640 1 --2- 172.21.15.169:0/3908570736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f18102cc0 0x7f2f18100f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:49.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f0ffff640 1 -- 172.21.15.169:0/3908570736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f180ff730 msgr2=0x7f2f181023c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:49.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f0ffff640 1 --2- 172.21.15.169:0/3908570736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f180ff730 0x7f2f181023c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f0ffff640 1 -- 172.21.15.169:0/3908570736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f18102cc0 msgr2=0x7f2f18100f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:49.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f0ffff640 1 --2- 172.21.15.169:0/3908570736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f18102cc0 0x7f2f18100f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f0ffff640 1 -- 172.21.15.169:0/3908570736 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f181017d0 con 0x7f2f181000f0 2024-03-28T01:08:49.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f0ffff640 1 --2- 172.21.15.169:0/3908570736 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f181000f0 0x7f2f18100a00 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2f0000e9f0 tx=0x7f2f0000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:49.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f157fa640 1 -- 172.21.15.169:0/3908570736 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0000cd80 con 0x7f2f181000f0 2024-03-28T01:08:49.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f157fa640 1 -- 172.21.15.169:0/3908570736 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f00010910 con 0x7f2f181000f0 2024-03-28T01:08:49.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f157fa640 1 -- 172.21.15.169:0/3908570736 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f00018990 con 0x7f2f181000f0 2024-03-28T01:08:49.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f1dcf1640 1 -- 172.21.15.169:0/3908570736 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f181a1af0 con 0x7f2f181000f0 2024-03-28T01:08:49.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.328+0000 7f2f1dcf1640 1 -- 172.21.15.169:0/3908570736 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f181a2040 con 0x7f2f181000f0 2024-03-28T01:08:49.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.329+0000 7f2f1dcf1640 1 -- 172.21.15.169:0/3908570736 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f18108ba0 con 0x7f2f181000f0 2024-03-28T01:08:49.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.331+0000 7f2f157fa640 1 -- 172.21.15.169:0/3908570736 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2f00010ac0 con 0x7f2f181000f0 2024-03-28T01:08:49.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.331+0000 7f2f157fa640 1 --2- 172.21.15.169:0/3908570736 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2f040a2800 0x7f2f040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:49.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.331+0000 7f2f177fe640 1 --2- 172.21.15.169:0/3908570736 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2f040a2800 0x7f2f040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:49.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.331+0000 7f2f157fa640 1 -- 172.21.15.169:0/3908570736 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f2f000cf500 con 0x7f2f181000f0 2024-03-28T01:08:49.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.332+0000 7f2f177fe640 1 --2- 172.21.15.169:0/3908570736 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2f040a2800 0x7f2f040a4cc0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2f08000c00 tx=0x7f2f08005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:49.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.335+0000 7f2f157fa640 1 -- 172.21.15.169:0/3908570736 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2f0009bbb0 con 0x7f2f181000f0 2024-03-28T01:08:49.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.596+0000 7f2f1dcf1640 1 -- 172.21.15.169:0/3908570736 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2f1802d070 con 0x7f2f181000f0 2024-03-28T01:08:49.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.600+0000 7f2f157fa640 1 -- 172.21.15.169:0/3908570736 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2f0009fc50 con 0x7f2f181000f0 2024-03-28T01:08:49.603 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T01:08:49.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.604+0000 7f2f1dcf1640 1 -- 172.21.15.169:0/3908570736 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2f040a2800 msgr2=0x7f2f040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:49.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.604+0000 7f2f1dcf1640 1 --2- 172.21.15.169:0/3908570736 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2f040a2800 0x7f2f040a4cc0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2f08000c00 tx=0x7f2f08005fd0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.604+0000 7f2f1dcf1640 1 -- 172.21.15.169:0/3908570736 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f181000f0 msgr2=0x7f2f18100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:49.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.604+0000 7f2f1dcf1640 1 --2- 172.21.15.169:0/3908570736 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f181000f0 0x7f2f18100a00 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2f0000e9f0 tx=0x7f2f0000eeb0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.604+0000 7f2f1dcf1640 1 -- 172.21.15.169:0/3908570736 shutdown_connections 2024-03-28T01:08:49.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.604+0000 7f2f1dcf1640 1 --2- 172.21.15.169:0/3908570736 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2f040a2800 0x7f2f040a4cc0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.604+0000 7f2f1dcf1640 1 --2- 172.21.15.169:0/3908570736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f18102cc0 0x7f2f18100f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.604+0000 7f2f1dcf1640 1 --2- 172.21.15.169:0/3908570736 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2f181000f0 0x7f2f18100a00 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.604+0000 7f2f1dcf1640 1 --2- 172.21.15.169:0/3908570736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f180ff730 0x7f2f181023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.604+0000 7f2f1dcf1640 1 -- 172.21.15.169:0/3908570736 >> 172.21.15.169:0/3908570736 conn(0x7f2f180fb290 msgr2=0x7f2f18105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:49.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.605+0000 7f2f1dcf1640 1 -- 172.21.15.169:0/3908570736 shutdown_connections 2024-03-28T01:08:49.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.605+0000 7f2f1dcf1640 1 -- 172.21.15.169:0/3908570736 wait complete. 2024-03-28T01:08:49.627 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T01:08:49.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.799+0000 7fef6007e640 1 -- 172.21.15.169:0/688167556 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef58102900 msgr2=0x7fef58104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:49.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.799+0000 7fef6007e640 1 --2- 172.21.15.169:0/688167556 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef58102900 0x7fef58104d70 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fef5401a910 tx=0x7fef5403cca0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.802+0000 7fef6007e640 1 -- 172.21.15.169:0/688167556 shutdown_connections 2024-03-28T01:08:49.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.802+0000 7fef6007e640 1 --2- 172.21.15.169:0/688167556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef581052b0 0x7fef58105730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.802+0000 7fef6007e640 1 --2- 172.21.15.169:0/688167556 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef58102900 0x7fef58104d70 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.802+0000 7fef6007e640 1 --2- 172.21.15.169:0/688167556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef580fff90 0x7fef581023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.802+0000 7fef6007e640 1 -- 172.21.15.169:0/688167556 >> 172.21.15.169:0/688167556 conn(0x7fef580fbbd0 msgr2=0x7fef580fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:49.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.804+0000 7fef6007e640 1 -- 172.21.15.169:0/688167556 shutdown_connections 2024-03-28T01:08:49.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.805+0000 7fef6007e640 1 -- 172.21.15.169:0/688167556 wait complete. 2024-03-28T01:08:49.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.806+0000 7fef6007e640 1 Processor -- start 2024-03-28T01:08:49.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.806+0000 7fef6007e640 1 -- start start 2024-03-28T01:08:49.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.806+0000 7fef6007e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef580fff90 0x7fef5806b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:49.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.806+0000 7fef6007e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef58102900 0x7fef5806b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:49.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.806+0000 7fef6007e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef581052b0 0x7fef580724c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:49.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.806+0000 7fef6007e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef58109240 con 0x7fef581052b0 2024-03-28T01:08:49.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.806+0000 7fef6007e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef581090c0 con 0x7fef58102900 2024-03-28T01:08:49.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.806+0000 7fef6007e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef581093c0 con 0x7fef580fff90 2024-03-28T01:08:49.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.807+0000 7fef5de1a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef580fff90 0x7fef5806b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:49.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.807+0000 7fef5de1a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef580fff90 0x7fef5806b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48660/0 (socket says 172.21.15.169:48660) 2024-03-28T01:08:49.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.807+0000 7fef5de1a640 1 -- 172.21.15.169:0/2833594692 learned_addr learned my addr 172.21.15.169:0/2833594692 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:49.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.807+0000 7fef5d619640 1 --2- 172.21.15.169:0/2833594692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef58102900 0x7fef5806b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:49.811 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.807+0000 7fef5e61b640 1 --2- 172.21.15.169:0/2833594692 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef581052b0 0x7fef580724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:49.812 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.807+0000 7fef5de1a640 1 -- 172.21.15.169:0/2833594692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef58102900 msgr2=0x7fef5806b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:49.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.807+0000 7fef5de1a640 1 --2- 172.21.15.169:0/2833594692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef58102900 0x7fef5806b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.807+0000 7fef5de1a640 1 -- 172.21.15.169:0/2833594692 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef581052b0 msgr2=0x7fef580724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:49.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.807+0000 7fef5de1a640 1 --2- 172.21.15.169:0/2833594692 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef581052b0 0x7fef580724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:49.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.807+0000 7fef5de1a640 1 -- 172.21.15.169:0/2833594692 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef58072b30 con 0x7fef580fff90 2024-03-28T01:08:49.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.807+0000 7fef5de1a640 1 --2- 172.21.15.169:0/2833594692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef580fff90 0x7fef5806b270 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fef4800b450 tx=0x7fef4800b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:49.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.808+0000 7fef4effd640 1 -- 172.21.15.169:0/2833594692 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef48015020 con 0x7fef580fff90 2024-03-28T01:08:49.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.808+0000 7fef4effd640 1 -- 172.21.15.169:0/2833594692 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef48011d70 con 0x7fef580fff90 2024-03-28T01:08:49.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.808+0000 7fef6007e640 1 -- 172.21.15.169:0/2833594692 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef58072e20 con 0x7fef580fff90 2024-03-28T01:08:49.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.808+0000 7fef6007e640 1 -- 172.21.15.169:0/2833594692 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef580733c0 con 0x7fef580fff90 2024-03-28T01:08:49.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.811+0000 7fef4effd640 1 -- 172.21.15.169:0/2833594692 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4800f340 con 0x7fef580fff90 2024-03-28T01:08:49.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.811+0000 7fef6007e640 1 -- 172.21.15.169:0/2833594692 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef580688e0 con 0x7fef580fff90 2024-03-28T01:08:49.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.816+0000 7fef4effd640 1 -- 172.21.15.169:0/2833594692 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fef48018020 con 0x7fef580fff90 2024-03-28T01:08:49.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.816+0000 7fef4effd640 1 --2- 172.21.15.169:0/2833594692 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fef440a2a00 0x7fef440a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:49.820 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.817+0000 7fef4effd640 1 -- 172.21.15.169:0/2833594692 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fef480ceec0 con 0x7fef580fff90 2024-03-28T01:08:49.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.820+0000 7fef5d619640 1 --2- 172.21.15.169:0/2833594692 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fef440a2a00 0x7fef440a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:49.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.824+0000 7fef4effd640 1 -- 172.21.15.169:0/2833594692 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fef4801f020 con 0x7fef580fff90 2024-03-28T01:08:49.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:49.825+0000 7fef5d619640 1 --2- 172.21.15.169:0/2833594692 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fef440a2a00 0x7fef440a4ec0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fef54006010 tx=0x7fef5403f040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:50.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:49 smithi169 ceph-mon[32274]: pgmap v720: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 46 MiB/s wr, 324 op/s 2024-03-28T01:08:50.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:49 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3715729473' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T01:08:50.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:49 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3273436473' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T01:08:50.052 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:49 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3908570736' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T01:08:50.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:49 smithi174 ceph-mon[36060]: pgmap v720: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 46 MiB/s wr, 324 op/s 2024-03-28T01:08:50.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:49 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3715729473' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T01:08:50.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:49 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3273436473' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T01:08:50.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:49 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3908570736' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T01:08:50.169 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:49 smithi177 ceph-mon[36206]: pgmap v720: 129 pgs: 129 active+clean; 8.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 46 MiB/s wr, 324 op/s 2024-03-28T01:08:50.170 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:49 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3715729473' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T01:08:50.170 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:49 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3273436473' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T01:08:50.170 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:49 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3908570736' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T01:08:50.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.169+0000 7fef6007e640 1 -- 172.21.15.169:0/2833594692 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fef5810ab80 con 0x7fef580fff90 2024-03-28T01:08:50.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.169+0000 7fef4effd640 1 -- 172.21.15.169:0/2833594692 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) v1 ==== 100+0+11 (secure 0 0 0) 0x7fef4800d040 con 0x7fef580fff90 2024-03-28T01:08:50.172 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 64 2024-03-28T01:08:50.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.173+0000 7fef6007e640 1 -- 172.21.15.169:0/2833594692 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fef440a2a00 msgr2=0x7fef440a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:50.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.173+0000 7fef6007e640 1 --2- 172.21.15.169:0/2833594692 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fef440a2a00 0x7fef440a4ec0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fef54006010 tx=0x7fef5403f040 comp rx=0 tx=0).stop 2024-03-28T01:08:50.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.173+0000 7fef6007e640 1 -- 172.21.15.169:0/2833594692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef580fff90 msgr2=0x7fef5806b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:50.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.173+0000 7fef6007e640 1 --2- 172.21.15.169:0/2833594692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef580fff90 0x7fef5806b270 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fef4800b450 tx=0x7fef4800b910 comp rx=0 tx=0).stop 2024-03-28T01:08:50.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.174+0000 7fef6007e640 1 -- 172.21.15.169:0/2833594692 shutdown_connections 2024-03-28T01:08:50.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.174+0000 7fef6007e640 1 --2- 172.21.15.169:0/2833594692 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fef440a2a00 0x7fef440a4ec0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:50.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.174+0000 7fef6007e640 1 --2- 172.21.15.169:0/2833594692 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef581052b0 0x7fef580724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:50.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.174+0000 7fef6007e640 1 --2- 172.21.15.169:0/2833594692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef58102900 0x7fef5806b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:50.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.174+0000 7fef6007e640 1 --2- 172.21.15.169:0/2833594692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef580fff90 0x7fef5806b270 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:50.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.174+0000 7fef6007e640 1 -- 172.21.15.169:0/2833594692 >> 172.21.15.169:0/2833594692 conn(0x7fef580fbbd0 msgr2=0x7fef580fbfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:50.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.174+0000 7fef6007e640 1 -- 172.21.15.169:0/2833594692 shutdown_connections 2024-03-28T01:08:50.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.174+0000 7fef6007e640 1 -- 172.21.15.169:0/2833594692 wait complete. 2024-03-28T01:08:50.196 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T01:08:50.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.368+0000 7fefec6f8640 1 -- 172.21.15.169:0/1182864214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fefe41038c0 msgr2=0x7fefe4103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:50.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.368+0000 7fefec6f8640 1 --2- 172.21.15.169:0/1182864214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fefe41038c0 0x7fefe4103ce0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fefe0009900 tx=0x7fefe002bc30 comp rx=0 tx=0).stop 2024-03-28T01:08:50.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.368+0000 7fefec6f8640 1 -- 172.21.15.169:0/1182864214 shutdown_connections 2024-03-28T01:08:50.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.368+0000 7fefec6f8640 1 --2- 172.21.15.169:0/1182864214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefe4105470 0x7fefe410bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:50.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.368+0000 7fefec6f8640 1 --2- 172.21.15.169:0/1182864214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefe4104ab0 0x7fefe4104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:50.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.368+0000 7fefec6f8640 1 --2- 172.21.15.169:0/1182864214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fefe41038c0 0x7fefe4103ce0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:50.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.368+0000 7fefec6f8640 1 -- 172.21.15.169:0/1182864214 >> 172.21.15.169:0/1182864214 conn(0x7fefe40fee50 msgr2=0x7fefe4101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:50.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.368+0000 7fefec6f8640 1 -- 172.21.15.169:0/1182864214 shutdown_connections 2024-03-28T01:08:50.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.369+0000 7fefec6f8640 1 -- 172.21.15.169:0/1182864214 wait complete. 2024-03-28T01:08:50.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.369+0000 7fefec6f8640 1 Processor -- start 2024-03-28T01:08:50.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.369+0000 7fefec6f8640 1 -- start start 2024-03-28T01:08:50.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.369+0000 7fefec6f8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefe41038c0 0x7fefe4199cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:50.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.369+0000 7fefec6f8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fefe4104ab0 0x7fefe419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:50.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.369+0000 7fefec6f8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefe4105470 0x7fefe41a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:50.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.369+0000 7fefec6f8640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe410f510 con 0x7fefe4104ab0 2024-03-28T01:08:50.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.369+0000 7fefec6f8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe410f390 con 0x7fefe4105470 2024-03-28T01:08:50.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.369+0000 7fefec6f8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe410f690 con 0x7fefe41038c0 2024-03-28T01:08:50.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.369+0000 7fefe9c93640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fefe4104ab0 0x7fefe419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:50.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.369+0000 7fefea494640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefe41038c0 0x7fefe4199cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:50.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.369+0000 7fefe9c93640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fefe4104ab0 0x7fefe419a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53522/0 (socket says 172.21.15.169:53522) 2024-03-28T01:08:50.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.369+0000 7fefe9c93640 1 -- 172.21.15.169:0/570433433 learned_addr learned my addr 172.21.15.169:0/570433433 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:50.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.369+0000 7fefeac95640 1 --2- 172.21.15.169:0/570433433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefe4105470 0x7fefe41a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:50.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.370+0000 7fefe9c93640 1 -- 172.21.15.169:0/570433433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefe41038c0 msgr2=0x7fefe4199cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:50.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.370+0000 7fefe9c93640 1 --2- 172.21.15.169:0/570433433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefe41038c0 0x7fefe4199cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:50.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.370+0000 7fefe9c93640 1 -- 172.21.15.169:0/570433433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefe4105470 msgr2=0x7fefe41a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:50.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.370+0000 7fefe9c93640 1 --2- 172.21.15.169:0/570433433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefe4105470 0x7fefe41a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:50.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.370+0000 7fefe9c93640 1 -- 172.21.15.169:0/570433433 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe41a1840 con 0x7fefe4104ab0 2024-03-28T01:08:50.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.370+0000 7fefe9c93640 1 --2- 172.21.15.169:0/570433433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fefe4104ab0 0x7fefe419a210 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fefd400ea60 tx=0x7fefd400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:50.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.370+0000 7fefdb7fe640 1 -- 172.21.15.169:0/570433433 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd400cde0 con 0x7fefe4104ab0 2024-03-28T01:08:50.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.370+0000 7fefdb7fe640 1 -- 172.21.15.169:0/570433433 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fefd40035f0 con 0x7fefe4104ab0 2024-03-28T01:08:50.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.370+0000 7fefec6f8640 1 -- 172.21.15.169:0/570433433 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe41a1b30 con 0x7fefe4104ab0 2024-03-28T01:08:50.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.370+0000 7fefdb7fe640 1 -- 172.21.15.169:0/570433433 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd4010670 con 0x7fefe4104ab0 2024-03-28T01:08:50.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.370+0000 7fefec6f8640 1 -- 172.21.15.169:0/570433433 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe41a2080 con 0x7fefe4104ab0 2024-03-28T01:08:50.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.371+0000 7fefec6f8640 1 -- 172.21.15.169:0/570433433 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe410ac00 con 0x7fefe4104ab0 2024-03-28T01:08:50.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.372+0000 7fefdb7fe640 1 -- 172.21.15.169:0/570433433 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fefd40037a0 con 0x7fefe4104ab0 2024-03-28T01:08:50.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.372+0000 7fefdb7fe640 1 --2- 172.21.15.169:0/570433433 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fefd00a2800 0x7fefd00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:50.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.373+0000 7fefdb7fe640 1 -- 172.21.15.169:0/570433433 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fefd40cf180 con 0x7fefe4104ab0 2024-03-28T01:08:50.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.373+0000 7fefea494640 1 --2- 172.21.15.169:0/570433433 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fefd00a2800 0x7fefd00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:50.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.373+0000 7fefea494640 1 --2- 172.21.15.169:0/570433433 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fefd00a2800 0x7fefd00a4cc0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fefe419b100 tx=0x7fefe002c880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:50.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.377+0000 7fefdb7fe640 1 -- 172.21.15.169:0/570433433 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fefd409be80 con 0x7fefe4104ab0 2024-03-28T01:08:50.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.553+0000 7fefec6f8640 1 -- 172.21.15.169:0/570433433 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fefe4062340 con 0x7fefe4104ab0 2024-03-28T01:08:50.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.554+0000 7fefdb7fe640 1 -- 172.21.15.169:0/570433433 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) v1 ==== 103+0+11 (secure 0 0 0) 0x7fefd409b780 con 0x7fefe4104ab0 2024-03-28T01:08:50.557 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T01:08:50.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.556+0000 7fefec6f8640 1 -- 172.21.15.169:0/570433433 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fefd00a2800 msgr2=0x7fefd00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:50.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.556+0000 7fefec6f8640 1 --2- 172.21.15.169:0/570433433 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fefd00a2800 0x7fefd00a4cc0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fefe419b100 tx=0x7fefe002c880 comp rx=0 tx=0).stop 2024-03-28T01:08:50.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.556+0000 7fefec6f8640 1 -- 172.21.15.169:0/570433433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fefe4104ab0 msgr2=0x7fefe419a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:50.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.556+0000 7fefec6f8640 1 --2- 172.21.15.169:0/570433433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fefe4104ab0 0x7fefe419a210 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fefd400ea60 tx=0x7fefd400ef20 comp rx=0 tx=0).stop 2024-03-28T01:08:50.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.556+0000 7fefec6f8640 1 -- 172.21.15.169:0/570433433 shutdown_connections 2024-03-28T01:08:50.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.556+0000 7fefec6f8640 1 --2- 172.21.15.169:0/570433433 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fefd00a2800 0x7fefd00a4cc0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:50.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.556+0000 7fefec6f8640 1 --2- 172.21.15.169:0/570433433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefe4105470 0x7fefe41a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:50.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.556+0000 7fefec6f8640 1 --2- 172.21.15.169:0/570433433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fefe4104ab0 0x7fefe419a210 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:50.559 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.556+0000 7fefec6f8640 1 --2- 172.21.15.169:0/570433433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefe41038c0 0x7fefe4199cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:50.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.556+0000 7fefec6f8640 1 -- 172.21.15.169:0/570433433 >> 172.21.15.169:0/570433433 conn(0x7fefe40fee50 msgr2=0x7fefe4107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:50.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.556+0000 7fefec6f8640 1 -- 172.21.15.169:0/570433433 shutdown_connections 2024-03-28T01:08:50.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.556+0000 7fefec6f8640 1 -- 172.21.15.169:0/570433433 wait complete. 2024-03-28T01:08:50.570 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-28T01:08:50.675 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.669+0000 7f7e0e1e6640 1 -- 172.21.15.169:0/2973481930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e081017b0 msgr2=0x7f7e08101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:50.675 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.669+0000 7f7e0e1e6640 1 --2- 172.21.15.169:0/2973481930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e081017b0 0x7f7e08101c30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7df0009900 tx=0x7f7df002bc30 comp rx=0 tx=0).stop 2024-03-28T01:08:50.675 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.669+0000 7f7e0e1e6640 1 -- 172.21.15.169:0/2973481930 shutdown_connections 2024-03-28T01:08:50.675 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.669+0000 7f7e0e1e6640 1 --2- 172.21.15.169:0/2973481930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e08102170 0x7f7e08108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:50.675 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.669+0000 7f7e0e1e6640 1 --2- 172.21.15.169:0/2973481930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e081017b0 0x7f7e08101c30 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:50.675 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.669+0000 7f7e0e1e6640 1 --2- 172.21.15.169:0/2973481930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7e081005c0 0x7f7e081009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:50.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.669+0000 7f7e0e1e6640 1 -- 172.21.15.169:0/2973481930 >> 172.21.15.169:0/2973481930 conn(0x7f7e080fbb30 msgr2=0x7f7e080fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:50.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.669+0000 7f7e0e1e6640 1 -- 172.21.15.169:0/2973481930 shutdown_connections 2024-03-28T01:08:50.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.669+0000 7f7e0e1e6640 1 -- 172.21.15.169:0/2973481930 wait complete. 2024-03-28T01:08:50.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.670+0000 7f7e0e1e6640 1 Processor -- start 2024-03-28T01:08:50.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.670+0000 7f7e0e1e6640 1 -- start start 2024-03-28T01:08:50.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.670+0000 7f7e0e1e6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e081005c0 0x7f7e081968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:50.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.670+0000 7f7e0e1e6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7e081017b0 0x7f7e08196df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:50.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.670+0000 7f7e0e1e6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e08102170 0x7f7e0819de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:50.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.670+0000 7f7e0e1e6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0810c2a0 con 0x7f7e081017b0 2024-03-28T01:08:50.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.670+0000 7f7e0e1e6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0810c120 con 0x7f7e08102170 2024-03-28T01:08:50.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.670+0000 7f7e0e1e6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0810c420 con 0x7f7e081005c0 2024-03-28T01:08:50.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.670+0000 7f7e06ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7e081017b0 0x7f7e08196df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:50.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.670+0000 7f7e06ffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7e081017b0 0x7f7e08196df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:53552/0 (socket says 172.21.15.169:53552) 2024-03-28T01:08:50.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.670+0000 7f7e06ffd640 1 -- 172.21.15.169:0/435307857 learned_addr learned my addr 172.21.15.169:0/435307857 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:08:50.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.670+0000 7f7e07fff640 1 --2- 172.21.15.169:0/435307857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e08102170 0x7f7e0819de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:50.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.670+0000 7f7e077fe640 1 --2- 172.21.15.169:0/435307857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e081005c0 0x7f7e081968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:50.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.670+0000 7f7e06ffd640 1 -- 172.21.15.169:0/435307857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e081005c0 msgr2=0x7f7e081968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:50.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.670+0000 7f7e06ffd640 1 --2- 172.21.15.169:0/435307857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e081005c0 0x7f7e081968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:50.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.670+0000 7f7e06ffd640 1 -- 172.21.15.169:0/435307857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e08102170 msgr2=0x7f7e0819de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:50.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.670+0000 7f7e06ffd640 1 --2- 172.21.15.169:0/435307857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e08102170 0x7f7e0819de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:50.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.670+0000 7f7e06ffd640 1 -- 172.21.15.169:0/435307857 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0819e4d0 con 0x7f7e081017b0 2024-03-28T01:08:50.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.670+0000 7f7e077fe640 1 --2- 172.21.15.169:0/435307857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e081005c0 0x7f7e081968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:08:50.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.670+0000 7f7e07fff640 1 --2- 172.21.15.169:0/435307857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e08102170 0x7f7e0819de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:08:50.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.671+0000 7f7e06ffd640 1 --2- 172.21.15.169:0/435307857 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7e081017b0 0x7f7e08196df0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f7df0000c00 tx=0x7f7df002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:50.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.671+0000 7f7e04ff9640 1 -- 172.21.15.169:0/435307857 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df0043020 con 0x7f7e081017b0 2024-03-28T01:08:50.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.671+0000 7f7e0e1e6640 1 -- 172.21.15.169:0/435307857 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e0819e760 con 0x7f7e081017b0 2024-03-28T01:08:50.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.671+0000 7f7e04ff9640 1 -- 172.21.15.169:0/435307857 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7df00054e0 con 0x7f7e081017b0 2024-03-28T01:08:50.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.671+0000 7f7e04ff9640 1 -- 172.21.15.169:0/435307857 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df0003c40 con 0x7f7e081017b0 2024-03-28T01:08:50.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.671+0000 7f7e0e1e6640 1 -- 172.21.15.169:0/435307857 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e0819ecb0 con 0x7f7e081017b0 2024-03-28T01:08:50.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.672+0000 7f7e0e1e6640 1 -- 172.21.15.169:0/435307857 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e081078e0 con 0x7f7e081017b0 2024-03-28T01:08:50.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.673+0000 7f7e04ff9640 1 -- 172.21.15.169:0/435307857 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7df0046020 con 0x7f7e081017b0 2024-03-28T01:08:50.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.673+0000 7f7e04ff9640 1 --2- 172.21.15.169:0/435307857 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7df40a2800 0x7f7df40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:08:50.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.673+0000 7f7e077fe640 1 --2- 172.21.15.169:0/435307857 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7df40a2800 0x7f7df40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:08:50.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.674+0000 7f7e04ff9640 1 -- 172.21.15.169:0/435307857 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f7df00f1200 con 0x7f7e081017b0 2024-03-28T01:08:50.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.674+0000 7f7e077fe640 1 --2- 172.21.15.169:0/435307857 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7df40a2800 0x7f7df40a4cc0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f7e08101610 tx=0x7f7df8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:08:50.680 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.677+0000 7f7e04ff9640 1 -- 172.21.15.169:0/435307857 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7df004d020 con 0x7f7e081017b0 2024-03-28T01:08:50.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.901+0000 7f7e0e1e6640 1 -- 172.21.15.169:0/435307857 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f7e0802d240 con 0x7f7e081017b0 2024-03-28T01:08:50.912 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.909+0000 7f7e04ff9640 1 -- 172.21.15.169:0/435307857 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v20) v1 ==== 81+0+739838 (secure 0 0 0) 0x7f7df00bdf00 con 0x7f7e081017b0 2024-03-28T01:08:50.929 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T01:08:50.929 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:08:50.929 INFO:teuthology.orchestra.run.smithi169.stdout: "epoch": 20, 2024-03-28T01:08:50.929 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.929 INFO:teuthology.orchestra.run.smithi169.stdout: "active_gid": 14152, 2024-03-28T01:08:50.930 INFO:teuthology.orchestra.run.smithi169.stdout: "active_name": "x", 2024-03-28T01:08:50.930 INFO:teuthology.orchestra.run.smithi169.stdout: "active_addrs": { 2024-03-28T01:08:50.930 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-28T01:08:50.930 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:50.930 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-28T01:08:50.930 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6800", 2024-03-28T01:08:50.930 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 1069722251 2024-03-28T01:08:50.930 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.930 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:50.930 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v1", 2024-03-28T01:08:50.930 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:6801", 2024-03-28T01:08:50.931 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 1069722251 2024-03-28T01:08:50.931 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:50.931 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-28T01:08:50.931 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.931 INFO:teuthology.orchestra.run.smithi169.stdout: "active_addr": "172.21.15.169:6801/1069722251", 2024-03-28T01:08:50.931 INFO:teuthology.orchestra.run.smithi169.stdout: "active_change": "2024-03-28T00:47:35.997884+0000", 2024-03-28T01:08:50.931 INFO:teuthology.orchestra.run.smithi169.stdout: "active_mgr_features": 4540701547738038271, 2024-03-28T01:08:50.931 INFO:teuthology.orchestra.run.smithi169.stdout: "available": true, 2024-03-28T01:08:50.931 INFO:teuthology.orchestra.run.smithi169.stdout: "standbys": [ 2024-03-28T01:08:50.931 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:50.931 INFO:teuthology.orchestra.run.smithi169.stdout: "gid": 24109, 2024-03-28T01:08:50.931 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "z", 2024-03-28T01:08:50.931 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_features": 4540701547738038271, 2024-03-28T01:08:50.932 INFO:teuthology.orchestra.run.smithi169.stdout: "available_modules": [ 2024-03-28T01:08:50.932 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:50.932 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "alerts", 2024-03-28T01:08:50.932 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:50.932 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:50.932 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:50.932 INFO:teuthology.orchestra.run.smithi169.stdout: "interval": { 2024-03-28T01:08:50.932 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "interval", 2024-03-28T01:08:50.932 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:50.932 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.932 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.932 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "60", 2024-03-28T01:08:50.932 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.933 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.933 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.933 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T01:08:50.933 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.933 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.933 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.933 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.933 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:50.933 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:50.933 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.933 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.933 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.933 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:50.933 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.934 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.934 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:50.934 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:50.934 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:50.934 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:50.934 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:50.934 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:50.934 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:50.934 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:50.934 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:50.934 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.934 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.934 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.935 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.935 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:50.935 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:50.935 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:50.935 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.935 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.935 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:50.935 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.935 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.935 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.935 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:50.935 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.935 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.936 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.936 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.936 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:50.936 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:50.936 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.936 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.936 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.936 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:50.936 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.937 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.937 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:50.937 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:50.937 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:50.937 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:50.937 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:50.937 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:50.937 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:50.937 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:50.937 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:50.937 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.937 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.937 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.937 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.937 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:50.938 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:50.938 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:50.938 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.938 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.938 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:50.938 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.938 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.938 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.938 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:50.938 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.938 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.938 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.939 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_destination": { 2024-03-28T01:08:50.939 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_destination", 2024-03-28T01:08:50.939 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.939 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.939 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.939 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:50.939 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.939 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.940 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.940 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Email address to send alerts to", 2024-03-28T01:08:50.940 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.940 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.940 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.940 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.940 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_from_name": { 2024-03-28T01:08:50.940 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_from_name", 2024-03-28T01:08:50.940 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.940 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.940 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.940 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "Ceph", 2024-03-28T01:08:50.940 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.941 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.941 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.941 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Email From: name", 2024-03-28T01:08:50.941 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.941 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.941 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.942 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.942 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_host": { 2024-03-28T01:08:50.942 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_host", 2024-03-28T01:08:50.942 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.942 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.942 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.942 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:50.942 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.942 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.942 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.942 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "SMTP server", 2024-03-28T01:08:50.942 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.942 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.942 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.942 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.943 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_password": { 2024-03-28T01:08:50.943 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_password", 2024-03-28T01:08:50.943 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.943 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.943 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.943 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:50.943 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.943 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.943 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.943 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Password to authenticate with", 2024-03-28T01:08:50.943 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.943 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.943 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.943 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.944 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_port": { 2024-03-28T01:08:50.944 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_port", 2024-03-28T01:08:50.944 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:50.944 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.944 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.944 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "465", 2024-03-28T01:08:50.944 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.944 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.944 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.944 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "SMTP port", 2024-03-28T01:08:50.944 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.944 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.944 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.944 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.945 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_sender": { 2024-03-28T01:08:50.945 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_sender", 2024-03-28T01:08:50.945 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.945 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.945 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.945 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:50.945 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.945 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.945 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.945 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "SMTP envelope sender", 2024-03-28T01:08:50.945 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.945 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.945 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.946 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.946 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_ssl": { 2024-03-28T01:08:50.946 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_ssl", 2024-03-28T01:08:50.946 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:50.946 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.946 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.946 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:50.946 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.946 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.946 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.946 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T01:08:50.946 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.946 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.947 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.947 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.947 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_user": { 2024-03-28T01:08:50.947 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_user", 2024-03-28T01:08:50.947 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.947 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.947 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.947 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:50.947 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.947 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.947 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.947 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "User to authenticate as", 2024-03-28T01:08:50.947 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.947 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.948 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.948 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:50.948 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:50.948 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.948 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:50.948 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "balancer", 2024-03-28T01:08:50.948 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:50.948 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:50.948 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:50.948 INFO:teuthology.orchestra.run.smithi169.stdout: "active": { 2024-03-28T01:08:50.948 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "active", 2024-03-28T01:08:50.948 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:50.948 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.948 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.949 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:50.949 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.949 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.949 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.949 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T01:08:50.949 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.949 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.949 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.949 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.949 INFO:teuthology.orchestra.run.smithi169.stdout: "begin_time": { 2024-03-28T01:08:50.949 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "begin_time", 2024-03-28T01:08:50.949 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.949 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.949 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.950 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0000", 2024-03-28T01:08:50.950 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.950 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.950 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.950 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T01:08:50.950 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T01:08:50.950 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.950 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.950 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.950 INFO:teuthology.orchestra.run.smithi169.stdout: "begin_weekday": { 2024-03-28T01:08:50.950 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "begin_weekday", 2024-03-28T01:08:50.950 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "uint", 2024-03-28T01:08:50.950 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.950 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.951 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0", 2024-03-28T01:08:50.951 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "0", 2024-03-28T01:08:50.951 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "6", 2024-03-28T01:08:50.951 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.951 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T01:08:50.951 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T01:08:50.951 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.951 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.951 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.951 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_compat_max_iterations": { 2024-03-28T01:08:50.951 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "crush_compat_max_iterations", 2024-03-28T01:08:50.951 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "uint", 2024-03-28T01:08:50.951 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.951 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.951 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "25", 2024-03-28T01:08:50.952 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "1", 2024-03-28T01:08:50.952 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "250", 2024-03-28T01:08:50.952 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.952 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T01:08:50.952 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.952 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.952 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.952 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.952 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_compat_metrics": { 2024-03-28T01:08:50.952 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "crush_compat_metrics", 2024-03-28T01:08:50.952 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.952 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.952 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.953 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T01:08:50.953 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.953 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.953 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.953 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T01:08:50.953 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T01:08:50.953 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.953 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.953 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.953 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_compat_step": { 2024-03-28T01:08:50.953 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "crush_compat_step", 2024-03-28T01:08:50.953 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "float", 2024-03-28T01:08:50.953 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.953 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.954 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0.5", 2024-03-28T01:08:50.954 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "0.001", 2024-03-28T01:08:50.954 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "0.999", 2024-03-28T01:08:50.954 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.954 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "aggressiveness of optimization", 2024-03-28T01:08:50.954 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T01:08:50.954 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.954 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.954 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.954 INFO:teuthology.orchestra.run.smithi169.stdout: "end_time": { 2024-03-28T01:08:50.954 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "end_time", 2024-03-28T01:08:50.954 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.954 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.954 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.955 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "2359", 2024-03-28T01:08:50.955 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.955 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.955 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.955 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T01:08:50.955 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T01:08:50.955 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.955 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.955 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.955 INFO:teuthology.orchestra.run.smithi169.stdout: "end_weekday": { 2024-03-28T01:08:50.955 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "end_weekday", 2024-03-28T01:08:50.955 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "uint", 2024-03-28T01:08:50.955 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.955 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.955 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0", 2024-03-28T01:08:50.956 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "0", 2024-03-28T01:08:50.956 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "6", 2024-03-28T01:08:50.956 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.956 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T01:08:50.956 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T01:08:50.956 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.956 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.956 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.956 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:50.956 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:50.956 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.956 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.956 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.956 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:50.957 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.957 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.957 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:50.957 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:50.957 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:50.957 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:50.957 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:50.957 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:50.957 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:50.957 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:50.957 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:50.957 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.957 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.957 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.957 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.958 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:50.958 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:50.958 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:50.958 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.958 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.958 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:50.958 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.958 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.958 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.958 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:50.958 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.958 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.959 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.959 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.959 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:50.959 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:50.959 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.959 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.959 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.959 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:50.959 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.959 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.959 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:50.959 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:50.959 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:50.959 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:50.960 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:50.960 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:50.960 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:50.960 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:50.960 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:50.960 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.960 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.960 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.960 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.960 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:50.960 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:50.960 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:50.960 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.960 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.960 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:50.961 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.961 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.961 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.961 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:50.961 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.961 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.961 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.961 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.961 INFO:teuthology.orchestra.run.smithi169.stdout: "min_score": { 2024-03-28T01:08:50.961 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "min_score", 2024-03-28T01:08:50.961 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "float", 2024-03-28T01:08:50.961 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.961 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.961 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0", 2024-03-28T01:08:50.961 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.962 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.962 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.962 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T01:08:50.962 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.962 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.962 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.962 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.962 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": { 2024-03-28T01:08:50.962 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "mode", 2024-03-28T01:08:50.962 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.962 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.962 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.963 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "upmap", 2024-03-28T01:08:50.963 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.963 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.964 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:50.964 INFO:teuthology.orchestra.run.smithi169.stdout: "crush-compat", 2024-03-28T01:08:50.964 INFO:teuthology.orchestra.run.smithi169.stdout: "none", 2024-03-28T01:08:50.964 INFO:teuthology.orchestra.run.smithi169.stdout: "read", 2024-03-28T01:08:50.964 INFO:teuthology.orchestra.run.smithi169.stdout: "upmap", 2024-03-28T01:08:50.964 INFO:teuthology.orchestra.run.smithi169.stdout: "upmap-read" 2024-03-28T01:08:50.964 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:50.964 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Balancer mode", 2024-03-28T01:08:50.964 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.965 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.965 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.965 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.965 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_ids": { 2024-03-28T01:08:50.965 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "pool_ids", 2024-03-28T01:08:50.965 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.965 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.965 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.965 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:50.965 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.965 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.965 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.966 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T01:08:50.966 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.966 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.966 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.966 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.966 INFO:teuthology.orchestra.run.smithi169.stdout: "sleep_interval": { 2024-03-28T01:08:50.966 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "sleep_interval", 2024-03-28T01:08:50.966 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:50.966 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.966 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.966 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "60", 2024-03-28T01:08:50.966 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.966 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.966 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.967 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T01:08:50.967 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.967 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.967 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.967 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.967 INFO:teuthology.orchestra.run.smithi169.stdout: "upmap_max_deviation": { 2024-03-28T01:08:50.967 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "upmap_max_deviation", 2024-03-28T01:08:50.967 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:50.967 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.967 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.967 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "5", 2024-03-28T01:08:50.967 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "1", 2024-03-28T01:08:50.967 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.967 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.967 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T01:08:50.968 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T01:08:50.968 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.968 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.968 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.968 INFO:teuthology.orchestra.run.smithi169.stdout: "upmap_max_optimizations": { 2024-03-28T01:08:50.968 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "upmap_max_optimizations", 2024-03-28T01:08:50.968 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "uint", 2024-03-28T01:08:50.968 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.968 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.969 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:50.969 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.969 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.969 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.969 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T01:08:50.969 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.969 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.969 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.969 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:50.969 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:50.969 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.969 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:50.969 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "cephadm", 2024-03-28T01:08:50.969 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:50.970 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:50.970 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:50.970 INFO:teuthology.orchestra.run.smithi169.stdout: "agent_down_multiplier": { 2024-03-28T01:08:50.970 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "agent_down_multiplier", 2024-03-28T01:08:50.970 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "float", 2024-03-28T01:08:50.970 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.970 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.970 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "3.0", 2024-03-28T01:08:50.970 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.970 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.970 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.970 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T01:08:50.970 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.970 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.970 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.971 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.971 INFO:teuthology.orchestra.run.smithi169.stdout: "agent_refresh_rate": { 2024-03-28T01:08:50.971 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "agent_refresh_rate", 2024-03-28T01:08:50.971 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:50.971 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.971 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.971 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "20", 2024-03-28T01:08:50.971 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.971 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.971 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.971 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T01:08:50.971 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.971 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.971 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.971 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.972 INFO:teuthology.orchestra.run.smithi169.stdout: "agent_starting_port": { 2024-03-28T01:08:50.972 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "agent_starting_port", 2024-03-28T01:08:50.972 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:50.972 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.972 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.972 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "4721", 2024-03-28T01:08:50.972 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.972 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.972 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.972 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T01:08:50.972 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.972 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.972 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.972 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.973 INFO:teuthology.orchestra.run.smithi169.stdout: "allow_ptrace": { 2024-03-28T01:08:50.973 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "allow_ptrace", 2024-03-28T01:08:50.973 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:50.973 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.973 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.973 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:50.973 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.973 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.973 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.973 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T01:08:50.973 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T01:08:50.973 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.973 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.973 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.973 INFO:teuthology.orchestra.run.smithi169.stdout: "autotune_interval": { 2024-03-28T01:08:50.974 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "autotune_interval", 2024-03-28T01:08:50.974 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:50.974 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.974 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.974 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "600", 2024-03-28T01:08:50.974 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.974 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.974 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.974 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T01:08:50.974 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.974 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.974 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.974 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.974 INFO:teuthology.orchestra.run.smithi169.stdout: "autotune_memory_target_ratio": { 2024-03-28T01:08:50.974 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T01:08:50.975 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "float", 2024-03-28T01:08:50.975 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.975 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.975 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0.7", 2024-03-28T01:08:50.975 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.975 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.975 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.975 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T01:08:50.975 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.975 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.975 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.975 INFO:teuthology.orchestra.run.smithi169.stdout: "cephadm_log_destination": { 2024-03-28T01:08:50.975 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "cephadm_log_destination", 2024-03-28T01:08:50.975 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.976 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.976 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.976 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:50.976 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.976 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.976 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:50.976 INFO:teuthology.orchestra.run.smithi169.stdout: "file", 2024-03-28T01:08:50.976 INFO:teuthology.orchestra.run.smithi169.stdout: "file,syslog", 2024-03-28T01:08:50.976 INFO:teuthology.orchestra.run.smithi169.stdout: "syslog" 2024-03-28T01:08:50.976 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:50.976 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T01:08:50.976 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.976 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.976 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.976 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.977 INFO:teuthology.orchestra.run.smithi169.stdout: "cgroups_split": { 2024-03-28T01:08:50.977 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "cgroups_split", 2024-03-28T01:08:50.977 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:50.977 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.977 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.977 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:50.977 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.977 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.977 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.977 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T01:08:50.977 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.977 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.977 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.977 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.977 INFO:teuthology.orchestra.run.smithi169.stdout: "config_checks_enabled": { 2024-03-28T01:08:50.978 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "config_checks_enabled", 2024-03-28T01:08:50.978 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:50.978 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.978 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.978 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:50.978 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.978 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.978 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.978 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T01:08:50.978 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.978 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.978 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.978 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.978 INFO:teuthology.orchestra.run.smithi169.stdout: "config_dashboard": { 2024-03-28T01:08:50.979 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "config_dashboard", 2024-03-28T01:08:50.979 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:50.979 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.979 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.979 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:50.979 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.979 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.979 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.979 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T01:08:50.979 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.979 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.979 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.979 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.979 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_alertmanager": { 2024-03-28T01:08:50.979 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_alertmanager", 2024-03-28T01:08:50.979 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.980 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.980 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.980 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T01:08:50.980 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.980 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.980 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.980 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Prometheus container image", 2024-03-28T01:08:50.980 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.980 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.980 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.980 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.980 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_base": { 2024-03-28T01:08:50.980 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_base", 2024-03-28T01:08:50.981 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.981 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.981 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:50.981 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T01:08:50.981 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.981 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.981 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.981 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Container image name, without the tag", 2024-03-28T01:08:50.981 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.981 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.981 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.981 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.982 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_elasticsearch": { 2024-03-28T01:08:50.982 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_elasticsearch", 2024-03-28T01:08:50.982 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.982 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.982 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.982 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T01:08:50.982 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.982 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.982 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.982 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "elasticsearch container image", 2024-03-28T01:08:50.982 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.983 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.983 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.983 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.983 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_grafana": { 2024-03-28T01:08:50.983 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_grafana", 2024-03-28T01:08:50.983 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.983 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.983 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.983 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T01:08:50.983 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.983 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.983 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.984 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Prometheus container image", 2024-03-28T01:08:50.984 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.984 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.984 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.984 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.984 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_haproxy": { 2024-03-28T01:08:50.984 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_haproxy", 2024-03-28T01:08:50.984 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.984 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.984 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.984 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T01:08:50.984 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.984 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.984 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.985 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "HAproxy container image", 2024-03-28T01:08:50.985 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.985 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.985 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.985 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.985 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_jaeger_agent": { 2024-03-28T01:08:50.985 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_jaeger_agent", 2024-03-28T01:08:50.985 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.985 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.985 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.985 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T01:08:50.985 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.985 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.985 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.985 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Jaeger agent container image", 2024-03-28T01:08:50.986 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.986 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.986 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.986 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.986 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_jaeger_collector": { 2024-03-28T01:08:50.986 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_jaeger_collector", 2024-03-28T01:08:50.986 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.986 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.986 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.986 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T01:08:50.986 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.986 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.986 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.986 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Jaeger collector container image", 2024-03-28T01:08:50.986 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.987 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.987 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.987 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.987 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_jaeger_query": { 2024-03-28T01:08:50.987 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_jaeger_query", 2024-03-28T01:08:50.987 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.987 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.987 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.987 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T01:08:50.987 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.987 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.987 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.987 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Jaeger query container image", 2024-03-28T01:08:50.987 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.988 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.988 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.988 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.988 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_keepalived": { 2024-03-28T01:08:50.988 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_keepalived", 2024-03-28T01:08:50.988 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.988 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.988 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.988 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T01:08:50.988 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.988 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.988 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.988 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Keepalived container image", 2024-03-28T01:08:50.988 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.988 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.989 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.989 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.989 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_loki": { 2024-03-28T01:08:50.989 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_loki", 2024-03-28T01:08:50.989 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.989 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.989 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.989 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T01:08:50.989 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.989 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.989 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.989 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Loki container image", 2024-03-28T01:08:50.989 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.989 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.989 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.989 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.990 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_node_exporter": { 2024-03-28T01:08:50.990 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_node_exporter", 2024-03-28T01:08:50.990 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.990 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.990 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.990 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T01:08:50.990 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.990 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.990 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.990 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Prometheus container image", 2024-03-28T01:08:50.990 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.990 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.990 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.990 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.991 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_nvmeof": { 2024-03-28T01:08:50.991 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_nvmeof", 2024-03-28T01:08:50.991 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.991 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.991 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.991 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-28T01:08:50.991 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.991 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.991 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.991 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Nvme-of container image", 2024-03-28T01:08:50.991 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.991 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.991 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.991 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.991 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_prometheus": { 2024-03-28T01:08:50.992 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_prometheus", 2024-03-28T01:08:50.992 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.992 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.992 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.992 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T01:08:50.992 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.992 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.992 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.992 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Prometheus container image", 2024-03-28T01:08:50.992 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.992 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.992 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.992 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.992 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_promtail": { 2024-03-28T01:08:50.992 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_promtail", 2024-03-28T01:08:50.993 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.993 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.993 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.993 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T01:08:50.993 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.993 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.993 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.993 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Promtail container image", 2024-03-28T01:08:50.993 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.993 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.993 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.993 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.993 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_snmp_gateway": { 2024-03-28T01:08:50.993 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_snmp_gateway", 2024-03-28T01:08:50.993 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.994 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.994 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.994 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T01:08:50.994 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.994 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.994 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.994 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "SNMP Gateway container image", 2024-03-28T01:08:50.994 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.994 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.994 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.994 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.994 INFO:teuthology.orchestra.run.smithi169.stdout: "container_init": { 2024-03-28T01:08:50.994 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_init", 2024-03-28T01:08:50.994 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:50.994 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.995 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.995 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:50.995 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.995 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.995 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.995 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T01:08:50.995 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.995 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.995 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.995 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.995 INFO:teuthology.orchestra.run.smithi169.stdout: "daemon_cache_timeout": { 2024-03-28T01:08:50.995 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "daemon_cache_timeout", 2024-03-28T01:08:50.995 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:50.995 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.995 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.995 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "600", 2024-03-28T01:08:50.996 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.996 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.996 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.996 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T01:08:50.996 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.996 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.996 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.996 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.996 INFO:teuthology.orchestra.run.smithi169.stdout: "default_cephadm_command_timeout": { 2024-03-28T01:08:50.996 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T01:08:50.996 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:50.996 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.996 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.996 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "900", 2024-03-28T01:08:50.997 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.997 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.997 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.997 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T01:08:50.997 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.997 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.997 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.997 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.997 INFO:teuthology.orchestra.run.smithi169.stdout: "default_registry": { 2024-03-28T01:08:50.997 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "default_registry", 2024-03-28T01:08:50.997 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:50.997 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.997 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.997 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "docker.io", 2024-03-28T01:08:50.997 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.998 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.998 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.998 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T01:08:50.998 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.998 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.998 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.998 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.998 INFO:teuthology.orchestra.run.smithi169.stdout: "device_cache_timeout": { 2024-03-28T01:08:50.998 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "device_cache_timeout", 2024-03-28T01:08:50.998 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:50.998 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.998 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.998 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "1800", 2024-03-28T01:08:50.998 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.999 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:50.999 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:50.999 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "seconds to cache device inventory", 2024-03-28T01:08:50.999 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:50.999 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:50.999 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:50.999 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:50.999 INFO:teuthology.orchestra.run.smithi169.stdout: "device_enhanced_scan": { 2024-03-28T01:08:50.999 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "device_enhanced_scan", 2024-03-28T01:08:50.999 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:50.999 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:50.999 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:50.999 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:50.999 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:50.999 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.000 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.000 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T01:08:51.000 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.000 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.000 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.000 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.000 INFO:teuthology.orchestra.run.smithi169.stdout: "facts_cache_timeout": { 2024-03-28T01:08:51.000 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "facts_cache_timeout", 2024-03-28T01:08:51.000 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.000 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.000 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.000 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "60", 2024-03-28T01:08:51.000 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.000 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.000 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.001 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "seconds to cache host facts data", 2024-03-28T01:08:51.001 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.001 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.001 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.001 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.001 INFO:teuthology.orchestra.run.smithi169.stdout: "host_check_interval": { 2024-03-28T01:08:51.001 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "host_check_interval", 2024-03-28T01:08:51.001 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.001 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.001 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.001 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "600", 2024-03-28T01:08:51.001 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.001 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.001 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.001 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how frequently to perform a host check", 2024-03-28T01:08:51.001 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.002 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.002 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.002 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.002 INFO:teuthology.orchestra.run.smithi169.stdout: "hw_monitoring": { 2024-03-28T01:08:51.002 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "hw_monitoring", 2024-03-28T01:08:51.002 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.002 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.002 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.002 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.002 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.002 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.002 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.002 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T01:08:51.003 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.003 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.003 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.003 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.003 INFO:teuthology.orchestra.run.smithi169.stdout: "inventory_list_all": { 2024-03-28T01:08:51.003 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "inventory_list_all", 2024-03-28T01:08:51.003 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.003 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.003 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.003 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.003 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.003 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.003 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.003 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T01:08:51.003 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.003 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.004 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.004 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.004 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.004 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.004 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.004 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.004 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.004 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.004 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.004 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.004 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.004 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.004 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.004 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.004 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.004 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.005 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.005 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.005 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.005 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.005 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.005 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.005 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.005 INFO:teuthology.orchestra.run.smithi169.stdout: "log_refresh_metadata": { 2024-03-28T01:08:51.005 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_refresh_metadata", 2024-03-28T01:08:51.005 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.005 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.005 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.005 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.005 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.005 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.006 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.006 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T01:08:51.006 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.006 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.006 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.006 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.006 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.006 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.006 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.006 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.006 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.006 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.006 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.006 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.007 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.007 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T01:08:51.007 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.007 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.007 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.007 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.007 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.007 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.007 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.007 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.007 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.007 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.007 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.007 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.008 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.008 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.008 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.008 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.008 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.008 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.008 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.008 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.008 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.008 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.008 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.008 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.008 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.008 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.008 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.009 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.009 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.009 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.009 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.009 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.009 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.009 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.009 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.009 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.009 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.009 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.009 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.009 INFO:teuthology.orchestra.run.smithi169.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T01:08:51.009 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T01:08:51.009 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.010 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.010 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.010 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.010 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.010 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.010 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.010 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T01:08:51.010 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.010 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.010 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.010 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.010 INFO:teuthology.orchestra.run.smithi169.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T01:08:51.010 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T01:08:51.011 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.011 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.011 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.011 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "*", 2024-03-28T01:08:51.011 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.011 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.011 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.011 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T01:08:51.011 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.011 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.011 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.011 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.011 INFO:teuthology.orchestra.run.smithi169.stdout: "max_count_per_host": { 2024-03-28T01:08:51.011 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "max_count_per_host", 2024-03-28T01:08:51.011 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.012 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.012 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.012 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.012 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.012 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.012 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.012 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "max number of daemons per service per host", 2024-03-28T01:08:51.012 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.012 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.012 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.012 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.012 INFO:teuthology.orchestra.run.smithi169.stdout: "max_osd_draining_count": { 2024-03-28T01:08:51.012 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "max_osd_draining_count", 2024-03-28T01:08:51.012 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.013 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.013 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.013 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.013 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.013 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.013 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.013 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T01:08:51.013 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.013 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.013 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.013 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.013 INFO:teuthology.orchestra.run.smithi169.stdout: "migration_current": { 2024-03-28T01:08:51.013 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "migration_current", 2024-03-28T01:08:51.013 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.013 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.013 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.014 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.014 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.014 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.014 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.014 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "internal - do not modify", 2024-03-28T01:08:51.014 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.014 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.014 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.014 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.014 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": { 2024-03-28T01:08:51.014 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "mode", 2024-03-28T01:08:51.014 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.014 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.014 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.015 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "root", 2024-03-28T01:08:51.015 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.015 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.015 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.015 INFO:teuthology.orchestra.run.smithi169.stdout: "cephadm-package", 2024-03-28T01:08:51.015 INFO:teuthology.orchestra.run.smithi169.stdout: "root" 2024-03-28T01:08:51.015 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.015 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T01:08:51.015 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.015 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.015 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.015 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.015 INFO:teuthology.orchestra.run.smithi169.stdout: "oob_default_addr": { 2024-03-28T01:08:51.015 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "oob_default_addr", 2024-03-28T01:08:51.015 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.016 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.016 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.016 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "169.254.1.1", 2024-03-28T01:08:51.016 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.016 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.016 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.016 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T01:08:51.016 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.016 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.016 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.016 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.016 INFO:teuthology.orchestra.run.smithi169.stdout: "prometheus_alerts_path": { 2024-03-28T01:08:51.016 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "prometheus_alerts_path", 2024-03-28T01:08:51.016 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.016 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.016 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.017 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T01:08:51.017 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.017 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.017 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.017 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T01:08:51.017 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.017 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.017 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.017 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.017 INFO:teuthology.orchestra.run.smithi169.stdout: "registry_insecure": { 2024-03-28T01:08:51.017 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "registry_insecure", 2024-03-28T01:08:51.017 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.017 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.017 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.017 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.018 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.018 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.018 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.018 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T01:08:51.018 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.018 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.018 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.018 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.018 INFO:teuthology.orchestra.run.smithi169.stdout: "registry_password": { 2024-03-28T01:08:51.018 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "registry_password", 2024-03-28T01:08:51.018 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.018 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.018 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.018 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.019 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.019 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.019 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.019 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T01:08:51.019 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.019 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.019 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.019 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.019 INFO:teuthology.orchestra.run.smithi169.stdout: "registry_url": { 2024-03-28T01:08:51.019 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "registry_url", 2024-03-28T01:08:51.019 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.019 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.019 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.019 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.020 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.020 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.020 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.020 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T01:08:51.020 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.020 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.020 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.020 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.020 INFO:teuthology.orchestra.run.smithi169.stdout: "registry_username": { 2024-03-28T01:08:51.020 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "registry_username", 2024-03-28T01:08:51.020 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.020 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.020 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.020 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.020 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.021 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.021 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.021 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T01:08:51.021 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.021 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.021 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.021 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.021 INFO:teuthology.orchestra.run.smithi169.stdout: "secure_monitoring_stack": { 2024-03-28T01:08:51.021 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "secure_monitoring_stack", 2024-03-28T01:08:51.021 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.021 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.021 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.021 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.021 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.021 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.022 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.022 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T01:08:51.022 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.022 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.022 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.022 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.022 INFO:teuthology.orchestra.run.smithi169.stdout: "service_discovery_port": { 2024-03-28T01:08:51.022 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "service_discovery_port", 2024-03-28T01:08:51.022 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.022 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.022 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.022 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "8765", 2024-03-28T01:08:51.022 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.022 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.023 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.023 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "cephadm service discovery port", 2024-03-28T01:08:51.023 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.023 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.023 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.023 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.023 INFO:teuthology.orchestra.run.smithi169.stdout: "ssh_config_file": { 2024-03-28T01:08:51.023 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ssh_config_file", 2024-03-28T01:08:51.023 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.023 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.023 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.023 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.023 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.023 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.023 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.024 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T01:08:51.024 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.024 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.024 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.024 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.024 INFO:teuthology.orchestra.run.smithi169.stdout: "use_agent": { 2024-03-28T01:08:51.024 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "use_agent", 2024-03-28T01:08:51.024 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.024 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.024 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.024 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.024 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.024 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.024 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.024 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T01:08:51.024 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.025 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.025 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.025 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.025 INFO:teuthology.orchestra.run.smithi169.stdout: "use_repo_digest": { 2024-03-28T01:08:51.025 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "use_repo_digest", 2024-03-28T01:08:51.025 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.025 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.025 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.025 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.025 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.025 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.025 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.025 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T01:08:51.025 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.025 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.025 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.026 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.026 INFO:teuthology.orchestra.run.smithi169.stdout: "warn_on_failed_host_check": { 2024-03-28T01:08:51.026 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "warn_on_failed_host_check", 2024-03-28T01:08:51.026 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.026 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.026 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.026 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.026 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.026 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.026 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.026 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T01:08:51.026 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.026 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.026 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.027 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.027 INFO:teuthology.orchestra.run.smithi169.stdout: "warn_on_stray_daemons": { 2024-03-28T01:08:51.027 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "warn_on_stray_daemons", 2024-03-28T01:08:51.027 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.027 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.027 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.027 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.027 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.027 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.027 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.027 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T01:08:51.027 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.027 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.027 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.028 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.028 INFO:teuthology.orchestra.run.smithi169.stdout: "warn_on_stray_hosts": { 2024-03-28T01:08:51.028 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "warn_on_stray_hosts", 2024-03-28T01:08:51.028 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.028 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.028 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.028 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.028 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.028 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.028 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.028 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T01:08:51.028 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.028 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.028 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.028 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.028 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.029 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.029 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.029 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "crash", 2024-03-28T01:08:51.029 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.029 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.029 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.029 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.029 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.029 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.029 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.029 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.029 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.029 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.029 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.029 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.029 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.030 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.030 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.030 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.030 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.030 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.030 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.030 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.030 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.030 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.030 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.030 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.030 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.030 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.030 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.030 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.031 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.031 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.031 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.031 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.031 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.031 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.031 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.031 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.031 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.031 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.031 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.031 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.031 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.031 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.032 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.032 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.032 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.032 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.032 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.032 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.032 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.032 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.032 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.032 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.032 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.032 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.032 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.032 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.032 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.033 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.033 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.033 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.033 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.033 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.033 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.033 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.033 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.033 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.033 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.033 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.033 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.033 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.033 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.033 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.034 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.034 INFO:teuthology.orchestra.run.smithi169.stdout: "retain_interval": { 2024-03-28T01:08:51.034 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "retain_interval", 2024-03-28T01:08:51.034 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.034 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.034 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.034 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "31536000", 2024-03-28T01:08:51.034 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.034 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.034 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.034 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T01:08:51.034 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.034 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.034 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.034 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.034 INFO:teuthology.orchestra.run.smithi169.stdout: "warn_recent_interval": { 2024-03-28T01:08:51.035 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "warn_recent_interval", 2024-03-28T01:08:51.035 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.035 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.035 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.035 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "1209600", 2024-03-28T01:08:51.035 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.035 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.035 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.035 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T01:08:51.035 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.035 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.035 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.035 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.035 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.035 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.036 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.036 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "dashboard", 2024-03-28T01:08:51.036 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.036 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.036 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.036 INFO:teuthology.orchestra.run.smithi169.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T01:08:51.036 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T01:08:51.036 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.036 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.036 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.036 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.036 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.036 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.036 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.036 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.037 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.037 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.037 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.037 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.037 INFO:teuthology.orchestra.run.smithi169.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T01:08:51.037 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T01:08:51.037 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.037 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.037 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.037 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.037 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.037 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.037 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.037 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.037 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.038 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.038 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.038 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.038 INFO:teuthology.orchestra.run.smithi169.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T01:08:51.038 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T01:08:51.038 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.038 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.038 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.038 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.038 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.038 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.038 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.038 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.038 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.038 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.039 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.039 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.039 INFO:teuthology.orchestra.run.smithi169.stdout: "AUDIT_API_ENABLED": { 2024-03-28T01:08:51.039 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T01:08:51.039 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.039 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.039 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.039 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.039 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.039 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.039 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.039 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.039 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.039 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.039 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.040 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.040 INFO:teuthology.orchestra.run.smithi169.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T01:08:51.040 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T01:08:51.040 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.040 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.040 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.040 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.040 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.040 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.040 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.040 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.040 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.040 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.040 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.040 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.041 INFO:teuthology.orchestra.run.smithi169.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T01:08:51.041 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T01:08:51.041 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.041 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.041 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.041 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.041 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.041 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.041 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.041 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.041 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.041 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.041 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.041 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.041 INFO:teuthology.orchestra.run.smithi169.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T01:08:51.042 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T01:08:51.042 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.042 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.042 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.042 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.042 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.042 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.042 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.042 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.042 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.042 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.042 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.042 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.042 INFO:teuthology.orchestra.run.smithi169.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T01:08:51.043 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T01:08:51.043 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.043 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.043 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.043 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.043 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.043 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.043 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.043 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.043 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.043 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.043 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.043 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.043 INFO:teuthology.orchestra.run.smithi169.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T01:08:51.043 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T01:08:51.044 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.044 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.044 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.044 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.044 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.044 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.044 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.044 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.044 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.044 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.044 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.044 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.044 INFO:teuthology.orchestra.run.smithi169.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T01:08:51.044 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T01:08:51.044 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.045 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.045 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.045 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.045 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.045 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.045 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.045 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.045 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.045 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.045 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.045 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.045 INFO:teuthology.orchestra.run.smithi169.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T01:08:51.045 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T01:08:51.045 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.046 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.046 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.046 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.046 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.046 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.046 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.046 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.046 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.046 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.046 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.046 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.046 INFO:teuthology.orchestra.run.smithi169.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T01:08:51.046 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T01:08:51.046 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.047 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.047 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.047 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.047 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.047 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.047 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.047 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.047 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.047 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.047 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.047 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.047 INFO:teuthology.orchestra.run.smithi169.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T01:08:51.047 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T01:08:51.047 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.047 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.047 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.048 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.048 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.048 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.048 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.048 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.048 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.048 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.048 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.048 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.048 INFO:teuthology.orchestra.run.smithi169.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T01:08:51.048 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T01:08:51.048 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.048 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.048 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.049 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.049 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.049 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.049 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.049 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.049 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.049 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.049 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.049 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.049 INFO:teuthology.orchestra.run.smithi169.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T01:08:51.049 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T01:08:51.049 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.049 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.049 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.049 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "admin", 2024-03-28T01:08:51.050 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.050 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.050 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.050 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.050 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.050 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.050 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.050 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.050 INFO:teuthology.orchestra.run.smithi169.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T01:08:51.050 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T01:08:51.050 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.050 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.050 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.050 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.050 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.051 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.051 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.051 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.051 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.051 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.051 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.051 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.051 INFO:teuthology.orchestra.run.smithi169.stdout: "GRAFANA_API_URL": { 2024-03-28T01:08:51.051 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "GRAFANA_API_URL", 2024-03-28T01:08:51.051 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.051 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.051 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.051 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.051 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.051 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.052 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.052 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.052 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.052 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.052 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.052 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.052 INFO:teuthology.orchestra.run.smithi169.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T01:08:51.052 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T01:08:51.052 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.052 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.052 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.052 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "admin", 2024-03-28T01:08:51.052 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.052 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.052 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.052 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.053 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.053 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.053 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.053 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.053 INFO:teuthology.orchestra.run.smithi169.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T01:08:51.053 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T01:08:51.053 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.053 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.053 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.053 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.053 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.053 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.053 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.053 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.053 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.054 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.054 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.054 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.054 INFO:teuthology.orchestra.run.smithi169.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T01:08:51.054 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T01:08:51.054 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.054 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.054 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.054 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.054 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.054 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.054 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.054 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.054 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.055 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.055 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.055 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.055 INFO:teuthology.orchestra.run.smithi169.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T01:08:51.055 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T01:08:51.055 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.055 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.055 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.055 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.055 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.055 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.055 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.055 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.055 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.055 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.055 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.056 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.056 INFO:teuthology.orchestra.run.smithi169.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T01:08:51.056 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T01:08:51.056 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.056 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.056 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.056 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.056 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.056 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.056 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.056 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.056 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.056 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.056 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.057 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.057 INFO:teuthology.orchestra.run.smithi169.stdout: "MULTICLUSTER_CONFIG": { 2024-03-28T01:08:51.057 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-28T01:08:51.057 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.057 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.057 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.057 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "{}", 2024-03-28T01:08:51.057 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.057 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.057 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.057 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.057 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.057 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.057 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.058 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.058 INFO:teuthology.orchestra.run.smithi169.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T01:08:51.058 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T01:08:51.058 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.058 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.058 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.058 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.058 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.058 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.058 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.058 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.058 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.058 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.058 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.059 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.059 INFO:teuthology.orchestra.run.smithi169.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T01:08:51.059 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T01:08:51.059 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.059 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.059 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.059 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.059 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.059 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.059 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.059 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.059 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.059 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.059 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.059 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.060 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T01:08:51.060 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T01:08:51.060 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.060 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.060 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.060 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.060 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.060 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.060 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.060 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.060 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.060 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.060 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.060 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.060 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T01:08:51.060 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T01:08:51.061 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.061 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.061 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.061 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.061 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.061 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.061 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.061 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.061 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.061 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.061 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.061 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.061 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T01:08:51.061 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T01:08:51.061 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.062 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.062 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.062 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.062 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.062 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.062 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.062 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.062 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.062 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.062 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.062 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.062 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T01:08:51.062 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T01:08:51.062 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.063 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.063 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.063 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.063 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.063 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.063 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.063 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.063 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.063 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.063 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.063 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.063 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T01:08:51.063 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T01:08:51.063 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.063 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.064 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.064 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.064 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.064 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.064 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.064 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.064 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.064 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.064 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.064 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.064 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T01:08:51.064 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T01:08:51.064 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.064 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.064 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.064 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.065 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.065 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.065 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.065 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.065 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.065 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.065 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.065 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.065 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T01:08:51.065 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T01:08:51.065 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.065 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.065 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.065 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.065 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.066 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.066 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.066 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.066 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.066 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.066 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.066 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.066 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T01:08:51.066 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T01:08:51.066 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.066 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.066 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.066 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.066 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.066 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.067 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.067 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.067 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.067 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.067 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.067 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.067 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T01:08:51.067 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T01:08:51.067 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.067 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.067 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.067 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T01:08:51.067 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.067 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.067 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.068 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.068 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.068 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.068 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.068 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T01:08:51.068 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T01:08:51.068 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.068 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.068 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.068 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.068 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.068 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.068 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.068 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.069 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.069 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.069 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.069 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.069 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T01:08:51.069 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T01:08:51.069 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.069 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.069 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.069 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "8", 2024-03-28T01:08:51.069 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.069 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.069 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.069 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.069 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.070 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.070 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.070 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.070 INFO:teuthology.orchestra.run.smithi169.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T01:08:51.070 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T01:08:51.070 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.070 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.070 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.070 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "45", 2024-03-28T01:08:51.070 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.070 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.070 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.070 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.070 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.070 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.071 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.071 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.071 INFO:teuthology.orchestra.run.smithi169.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T01:08:51.071 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T01:08:51.071 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.071 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.071 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.071 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.071 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.071 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.071 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.071 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.071 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.071 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.071 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.072 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.072 INFO:teuthology.orchestra.run.smithi169.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T01:08:51.072 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T01:08:51.072 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.072 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.072 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.072 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "admin", 2024-03-28T01:08:51.072 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.072 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.072 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.072 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.072 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.072 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.072 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.072 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.072 INFO:teuthology.orchestra.run.smithi169.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T01:08:51.073 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T01:08:51.073 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.073 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.073 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.073 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.073 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.073 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.073 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.073 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.073 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.073 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.073 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.073 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.073 INFO:teuthology.orchestra.run.smithi169.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T01:08:51.073 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T01:08:51.074 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.074 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.074 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.074 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.074 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.074 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.074 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.074 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.074 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.074 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.074 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.074 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.074 INFO:teuthology.orchestra.run.smithi169.stdout: "UNSAFE_TLS_v1_2": { 2024-03-28T01:08:51.074 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-28T01:08:51.075 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.075 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.075 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.075 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.075 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.075 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.075 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.075 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.075 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.075 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.075 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.075 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.075 INFO:teuthology.orchestra.run.smithi169.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T01:08:51.075 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T01:08:51.075 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.076 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.076 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.076 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0", 2024-03-28T01:08:51.076 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.076 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.076 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.076 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.076 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.076 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.076 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.076 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.076 INFO:teuthology.orchestra.run.smithi169.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T01:08:51.076 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T01:08:51.076 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.076 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.076 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.077 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.077 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.077 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.077 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.077 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.077 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.077 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.077 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.077 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.077 INFO:teuthology.orchestra.run.smithi169.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T01:08:51.077 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T01:08:51.077 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.077 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.077 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.077 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "5", 2024-03-28T01:08:51.078 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.078 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.078 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.078 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.078 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.078 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.078 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.078 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.078 INFO:teuthology.orchestra.run.smithi169.stdout: "cross_origin_url": { 2024-03-28T01:08:51.078 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "cross_origin_url", 2024-03-28T01:08:51.078 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.078 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.078 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.078 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.078 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.079 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.079 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.079 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.079 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.079 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.079 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.079 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.079 INFO:teuthology.orchestra.run.smithi169.stdout: "crt_file": { 2024-03-28T01:08:51.079 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "crt_file", 2024-03-28T01:08:51.079 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.079 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.079 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.079 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.079 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.079 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.079 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.080 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.080 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.080 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.080 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.080 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.080 INFO:teuthology.orchestra.run.smithi169.stdout: "debug": { 2024-03-28T01:08:51.080 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "debug", 2024-03-28T01:08:51.080 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.080 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.080 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.080 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.080 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.080 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.080 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.081 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Enable/disable debug options", 2024-03-28T01:08:51.081 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.081 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.081 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.081 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.081 INFO:teuthology.orchestra.run.smithi169.stdout: "jwt_token_ttl": { 2024-03-28T01:08:51.081 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "jwt_token_ttl", 2024-03-28T01:08:51.081 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.081 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.081 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.081 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "28800", 2024-03-28T01:08:51.081 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.081 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.081 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.081 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.082 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.082 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.082 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.082 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.082 INFO:teuthology.orchestra.run.smithi169.stdout: "key_file": { 2024-03-28T01:08:51.082 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "key_file", 2024-03-28T01:08:51.082 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.082 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.082 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.082 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.082 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.082 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.082 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.082 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.082 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.083 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.083 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.083 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.083 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.083 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.083 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.083 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.083 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.083 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.083 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.083 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.083 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.083 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.083 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.084 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.084 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.084 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.084 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.084 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.084 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.084 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.084 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.084 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.084 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.084 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.084 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.084 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.084 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.084 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.084 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.085 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.085 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.085 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.085 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.085 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.085 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.085 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.085 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.085 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.085 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.085 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.085 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.085 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.085 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.085 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.086 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.086 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.086 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.086 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.086 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.086 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.086 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.086 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.086 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.086 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.086 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.086 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.086 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.086 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.086 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.087 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.087 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.087 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.087 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.087 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.087 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.087 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.087 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.087 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.087 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.087 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.087 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.087 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.087 INFO:teuthology.orchestra.run.smithi169.stdout: "motd": { 2024-03-28T01:08:51.087 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "motd", 2024-03-28T01:08:51.087 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.088 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.088 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.088 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.088 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.088 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.088 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.088 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "The message of the day", 2024-03-28T01:08:51.088 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.088 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.088 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.088 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.088 INFO:teuthology.orchestra.run.smithi169.stdout: "redirect_resolve_ip_addr": { 2024-03-28T01:08:51.088 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T01:08:51.088 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.088 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.089 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.089 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.089 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.089 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.089 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.089 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.089 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.089 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.089 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.089 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.089 INFO:teuthology.orchestra.run.smithi169.stdout: "server_addr": { 2024-03-28T01:08:51.089 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "server_addr", 2024-03-28T01:08:51.089 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.089 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.089 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.089 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "::", 2024-03-28T01:08:51.090 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.090 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.090 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.090 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.090 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.090 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.090 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.090 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.090 INFO:teuthology.orchestra.run.smithi169.stdout: "server_port": { 2024-03-28T01:08:51.090 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "server_port", 2024-03-28T01:08:51.090 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.090 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.090 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.090 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "8080", 2024-03-28T01:08:51.090 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.091 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.091 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.091 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.091 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.091 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.091 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.091 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.091 INFO:teuthology.orchestra.run.smithi169.stdout: "ssl": { 2024-03-28T01:08:51.091 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ssl", 2024-03-28T01:08:51.091 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.091 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.091 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.091 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.091 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.091 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.092 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.092 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.092 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.092 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.092 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.092 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.092 INFO:teuthology.orchestra.run.smithi169.stdout: "ssl_server_port": { 2024-03-28T01:08:51.092 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ssl_server_port", 2024-03-28T01:08:51.092 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.092 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.092 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.092 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "8443", 2024-03-28T01:08:51.092 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.092 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.092 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.093 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.093 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.093 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.093 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.093 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.093 INFO:teuthology.orchestra.run.smithi169.stdout: "standby_behaviour": { 2024-03-28T01:08:51.093 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "standby_behaviour", 2024-03-28T01:08:51.093 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.093 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.093 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.093 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "redirect", 2024-03-28T01:08:51.093 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.093 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.093 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.093 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.094 INFO:teuthology.orchestra.run.smithi169.stdout: "redirect" 2024-03-28T01:08:51.094 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.094 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.094 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.094 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.094 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.094 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.094 INFO:teuthology.orchestra.run.smithi169.stdout: "standby_error_status_code": { 2024-03-28T01:08:51.094 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "standby_error_status_code", 2024-03-28T01:08:51.094 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.094 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.094 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.094 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "500", 2024-03-28T01:08:51.094 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "400", 2024-03-28T01:08:51.095 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "599", 2024-03-28T01:08:51.095 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.095 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.095 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.095 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.095 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.095 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.095 INFO:teuthology.orchestra.run.smithi169.stdout: "url_prefix": { 2024-03-28T01:08:51.095 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "url_prefix", 2024-03-28T01:08:51.095 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.095 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.095 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.095 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.095 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.096 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.096 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.096 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.096 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.096 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.096 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.096 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.096 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.096 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.096 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.096 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "devicehealth", 2024-03-28T01:08:51.096 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.096 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.096 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.096 INFO:teuthology.orchestra.run.smithi169.stdout: "enable_monitoring": { 2024-03-28T01:08:51.097 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "enable_monitoring", 2024-03-28T01:08:51.097 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.097 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.097 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.097 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.097 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.097 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.097 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.097 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "monitor device health metrics", 2024-03-28T01:08:51.097 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.097 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.097 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.097 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.097 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.098 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.098 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.098 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.098 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.098 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.098 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.098 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.098 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.098 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.098 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.098 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.098 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.098 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.098 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.099 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.099 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.099 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.099 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.099 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.099 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.099 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.099 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.099 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.099 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.099 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.099 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.099 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.099 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.100 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.100 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.100 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.100 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.100 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.100 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.100 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.100 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.100 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.100 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.100 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.100 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.100 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.100 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.100 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.101 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.101 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.101 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.101 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.101 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.101 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.101 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.101 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.101 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.101 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.101 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.101 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.101 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.101 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.101 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.102 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.102 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.102 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.102 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.102 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.102 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.102 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.102 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.102 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.102 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.102 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.102 INFO:teuthology.orchestra.run.smithi169.stdout: "mark_out_threshold": { 2024-03-28T01:08:51.102 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "mark_out_threshold", 2024-03-28T01:08:51.102 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.102 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.103 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.103 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "2419200", 2024-03-28T01:08:51.103 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.103 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.103 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.103 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T01:08:51.103 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.103 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.103 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.103 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.103 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": { 2024-03-28T01:08:51.103 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "pool_name", 2024-03-28T01:08:51.103 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.103 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.103 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.104 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "device_health_metrics", 2024-03-28T01:08:51.104 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.104 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.104 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.104 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T01:08:51.104 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.104 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.104 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.104 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.104 INFO:teuthology.orchestra.run.smithi169.stdout: "retention_period": { 2024-03-28T01:08:51.104 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "retention_period", 2024-03-28T01:08:51.104 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.104 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.104 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.104 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "15552000", 2024-03-28T01:08:51.105 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.105 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.105 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.105 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how long to retain device health metrics", 2024-03-28T01:08:51.105 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.105 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.105 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.105 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.105 INFO:teuthology.orchestra.run.smithi169.stdout: "scrape_frequency": { 2024-03-28T01:08:51.105 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "scrape_frequency", 2024-03-28T01:08:51.105 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.105 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.105 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.105 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "86400", 2024-03-28T01:08:51.105 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.105 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.106 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.106 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T01:08:51.106 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.106 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.106 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.106 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.106 INFO:teuthology.orchestra.run.smithi169.stdout: "self_heal": { 2024-03-28T01:08:51.106 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "self_heal", 2024-03-28T01:08:51.106 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.106 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.106 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.106 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.106 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.107 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.107 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.107 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T01:08:51.107 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.107 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.107 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.107 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.107 INFO:teuthology.orchestra.run.smithi169.stdout: "sleep_interval": { 2024-03-28T01:08:51.107 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "sleep_interval", 2024-03-28T01:08:51.107 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.107 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.107 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.107 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "600", 2024-03-28T01:08:51.107 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.107 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.108 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.108 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T01:08:51.108 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.108 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.108 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.108 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.108 INFO:teuthology.orchestra.run.smithi169.stdout: "warn_threshold": { 2024-03-28T01:08:51.108 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "warn_threshold", 2024-03-28T01:08:51.108 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.108 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.108 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.108 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "7257600", 2024-03-28T01:08:51.108 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.108 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.108 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.108 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T01:08:51.109 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.109 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.109 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.109 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.109 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.109 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.109 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.109 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "diskprediction_local", 2024-03-28T01:08:51.109 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.109 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.109 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.109 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.109 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.109 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.109 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.110 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.110 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.110 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.110 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.110 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.110 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.110 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.110 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.110 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.110 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.110 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.110 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.110 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.110 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.111 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.111 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.111 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.111 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.111 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.111 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.111 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.111 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.111 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.111 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.111 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.111 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.111 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.111 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.111 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.112 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.112 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.112 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.112 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.112 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.112 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.112 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.112 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.112 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.112 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.112 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.112 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.112 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.112 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.112 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.113 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.113 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.113 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.113 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.113 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.113 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.113 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.113 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.113 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.113 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.113 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.113 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.113 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.113 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.114 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.114 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.114 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.114 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.114 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.114 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.114 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.114 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.114 INFO:teuthology.orchestra.run.smithi169.stdout: "predict_interval": { 2024-03-28T01:08:51.114 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "predict_interval", 2024-03-28T01:08:51.114 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.114 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.114 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.114 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "86400", 2024-03-28T01:08:51.115 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.115 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.115 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.115 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.115 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.115 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.115 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.115 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.115 INFO:teuthology.orchestra.run.smithi169.stdout: "predictor_model": { 2024-03-28T01:08:51.115 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "predictor_model", 2024-03-28T01:08:51.115 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.115 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.115 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.115 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "prophetstor", 2024-03-28T01:08:51.115 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.115 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.116 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.116 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.116 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.116 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.116 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.116 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.116 INFO:teuthology.orchestra.run.smithi169.stdout: "sleep_interval": { 2024-03-28T01:08:51.116 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "sleep_interval", 2024-03-28T01:08:51.116 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.116 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.116 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.116 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "600", 2024-03-28T01:08:51.116 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.116 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.116 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.116 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.117 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.117 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.117 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.117 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.117 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.117 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.117 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.117 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "influx", 2024-03-28T01:08:51.117 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": false, 2024-03-28T01:08:51.117 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "influxdb python module not found", 2024-03-28T01:08:51.117 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.117 INFO:teuthology.orchestra.run.smithi169.stdout: "batch_size": { 2024-03-28T01:08:51.117 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "batch_size", 2024-03-28T01:08:51.117 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.118 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.118 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.118 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "5000", 2024-03-28T01:08:51.118 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.118 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.118 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.118 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T01:08:51.118 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.118 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.118 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.118 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.118 INFO:teuthology.orchestra.run.smithi169.stdout: "database": { 2024-03-28T01:08:51.118 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "database", 2024-03-28T01:08:51.118 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.119 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.119 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.119 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "ceph", 2024-03-28T01:08:51.119 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.119 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.119 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.119 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T01:08:51.119 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.119 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.119 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.119 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.119 INFO:teuthology.orchestra.run.smithi169.stdout: "hostname": { 2024-03-28T01:08:51.119 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "hostname", 2024-03-28T01:08:51.119 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.120 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.120 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.120 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.120 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.120 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.120 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.120 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "InfluxDB server hostname", 2024-03-28T01:08:51.120 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.120 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.120 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.120 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.120 INFO:teuthology.orchestra.run.smithi169.stdout: "interval": { 2024-03-28T01:08:51.120 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "interval", 2024-03-28T01:08:51.120 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.120 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.121 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.121 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "30", 2024-03-28T01:08:51.121 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "5", 2024-03-28T01:08:51.121 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.121 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.121 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T01:08:51.121 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.121 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.121 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.121 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.121 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.121 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.121 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.121 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.121 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.122 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.122 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.122 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.122 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.122 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.122 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.122 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.122 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.122 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.122 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.122 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.122 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.122 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.122 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.122 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.123 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.123 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.123 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.123 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.123 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.123 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.123 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.123 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.123 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.123 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.123 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.123 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.123 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.123 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.123 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.124 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.124 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.124 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.124 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.124 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.124 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.124 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.124 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.124 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.124 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.124 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.124 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.124 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.124 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.125 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.125 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.125 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.125 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.125 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.125 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.125 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.125 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.125 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.125 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.125 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.125 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.125 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.125 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.125 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.125 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.126 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.126 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.126 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.126 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.126 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.126 INFO:teuthology.orchestra.run.smithi169.stdout: "password": { 2024-03-28T01:08:51.126 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "password", 2024-03-28T01:08:51.126 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.126 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.126 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.126 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.126 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.126 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.126 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.126 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "password of InfluxDB server user", 2024-03-28T01:08:51.127 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.127 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.127 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.127 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.127 INFO:teuthology.orchestra.run.smithi169.stdout: "port": { 2024-03-28T01:08:51.127 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "port", 2024-03-28T01:08:51.127 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.127 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.127 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.127 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "8086", 2024-03-28T01:08:51.127 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.127 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.127 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.127 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "InfluxDB server port", 2024-03-28T01:08:51.127 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.128 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.128 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.128 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.128 INFO:teuthology.orchestra.run.smithi169.stdout: "ssl": { 2024-03-28T01:08:51.128 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ssl", 2024-03-28T01:08:51.128 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.128 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.128 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.128 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "false", 2024-03-28T01:08:51.128 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.128 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.128 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.128 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T01:08:51.128 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.128 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.129 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.129 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.129 INFO:teuthology.orchestra.run.smithi169.stdout: "threads": { 2024-03-28T01:08:51.129 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "threads", 2024-03-28T01:08:51.129 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.129 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.129 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.129 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "5", 2024-03-28T01:08:51.129 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "1", 2024-03-28T01:08:51.129 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "32", 2024-03-28T01:08:51.129 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.129 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T01:08:51.129 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.129 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.129 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.130 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.130 INFO:teuthology.orchestra.run.smithi169.stdout: "username": { 2024-03-28T01:08:51.130 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "username", 2024-03-28T01:08:51.130 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.130 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.130 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.130 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.130 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.130 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.130 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.130 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "username of InfluxDB server user", 2024-03-28T01:08:51.130 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.130 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.130 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.130 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.131 INFO:teuthology.orchestra.run.smithi169.stdout: "verify_ssl": { 2024-03-28T01:08:51.131 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "verify_ssl", 2024-03-28T01:08:51.131 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.131 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.131 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.131 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "true", 2024-03-28T01:08:51.131 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.131 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.131 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.131 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T01:08:51.131 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.131 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.131 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.131 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.132 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.132 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.132 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.132 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "insights", 2024-03-28T01:08:51.132 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.132 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.132 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.132 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.132 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.132 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.132 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.132 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.132 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.132 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.132 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.133 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.133 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.133 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.133 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.133 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.133 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.133 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.133 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.133 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.133 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.133 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.133 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.133 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.133 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.133 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.134 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.134 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.134 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.134 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.134 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.134 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.134 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.134 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.134 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.134 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.134 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.134 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.134 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.134 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.134 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.135 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.135 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.135 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.135 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.135 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.135 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.135 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.135 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.135 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.135 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.135 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.135 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.135 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.135 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.135 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.136 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.136 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.136 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.136 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.136 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.136 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.136 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.136 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.136 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.136 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.136 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.136 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.136 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.136 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.136 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.137 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.137 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.137 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.137 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.137 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.137 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "iostat", 2024-03-28T01:08:51.137 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.137 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.137 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.137 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.137 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.137 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.137 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.137 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.138 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.138 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.138 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.138 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.138 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.138 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.138 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.138 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.138 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.138 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.138 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.138 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.138 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.138 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.138 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.139 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.139 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.139 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.139 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.139 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.139 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.139 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.139 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.139 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.139 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.139 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.139 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.139 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.139 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.140 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.140 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.140 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.140 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.140 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.140 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.140 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.140 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.140 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.140 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.140 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.140 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.140 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.140 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.140 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.141 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.141 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.141 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.141 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.141 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.141 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.141 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.141 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.141 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.141 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.141 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.141 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.141 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.141 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.141 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.141 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.142 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.142 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.142 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.142 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.142 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.142 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.142 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.142 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.142 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "k8sevents", 2024-03-28T01:08:51.142 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.142 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.142 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.142 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_event_retention_days": { 2024-03-28T01:08:51.142 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ceph_event_retention_days", 2024-03-28T01:08:51.142 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.143 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.143 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.143 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "7", 2024-03-28T01:08:51.143 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.143 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.143 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.143 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T01:08:51.143 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.143 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.143 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.143 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.143 INFO:teuthology.orchestra.run.smithi169.stdout: "config_check_secs": { 2024-03-28T01:08:51.143 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "config_check_secs", 2024-03-28T01:08:51.143 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.143 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.143 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.144 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.144 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "10", 2024-03-28T01:08:51.144 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.144 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.144 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T01:08:51.144 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.144 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.144 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.144 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.144 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.144 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.144 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.144 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.145 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.145 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.145 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.145 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.145 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.145 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.145 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.145 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.145 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.145 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.145 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.145 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.145 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.145 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.146 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.146 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.146 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.146 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.146 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.146 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.146 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.146 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.146 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.146 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.146 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.146 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.146 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.146 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.147 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.147 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.147 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.147 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.147 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.147 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.147 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.147 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.147 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.147 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.147 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.147 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.147 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.147 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.147 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.148 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.148 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.148 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.148 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.148 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.148 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.148 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.148 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.148 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.148 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.148 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.148 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.148 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.148 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.148 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.149 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.149 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.149 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.149 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.149 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.149 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.149 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.149 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.149 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.149 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.149 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.149 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "localpool", 2024-03-28T01:08:51.149 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.149 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.149 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.149 INFO:teuthology.orchestra.run.smithi169.stdout: "failure_domain": { 2024-03-28T01:08:51.150 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "failure_domain", 2024-03-28T01:08:51.150 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.150 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.150 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.150 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "host", 2024-03-28T01:08:51.150 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.150 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.150 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.150 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "failure domain for any created local pool", 2024-03-28T01:08:51.150 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T01:08:51.150 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.150 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.150 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.151 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.151 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.151 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.151 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.151 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.151 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.151 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.151 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.151 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.151 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.151 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.151 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.151 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.151 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.151 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.151 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.152 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.152 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.152 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.152 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.152 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.152 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.152 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.152 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.152 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.152 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.152 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.152 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.152 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.152 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.152 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.153 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.153 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.153 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.153 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.153 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.153 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.153 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.153 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.153 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.153 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.153 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.153 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.153 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.153 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.154 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.154 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.154 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.154 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.154 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.154 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.154 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.154 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.154 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.154 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.154 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.154 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.154 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.155 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.155 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.155 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.155 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.155 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.155 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.155 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.155 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.155 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.155 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.155 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.155 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.155 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": { 2024-03-28T01:08:51.155 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "min_size", 2024-03-28T01:08:51.155 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.156 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.156 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.156 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.156 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.156 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.156 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.156 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "default min_size for any created local pool", 2024-03-28T01:08:51.156 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T01:08:51.156 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.156 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.156 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.156 INFO:teuthology.orchestra.run.smithi169.stdout: "num_rep": { 2024-03-28T01:08:51.156 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "num_rep", 2024-03-28T01:08:51.156 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.157 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.157 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.157 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "3", 2024-03-28T01:08:51.157 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.157 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.157 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.157 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "default replica count for any created local pool", 2024-03-28T01:08:51.157 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.157 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.157 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.157 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.157 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": { 2024-03-28T01:08:51.157 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "pg_num", 2024-03-28T01:08:51.157 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.158 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.158 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.158 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "128", 2024-03-28T01:08:51.158 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.158 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.158 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.158 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T01:08:51.158 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.158 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.158 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.158 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.158 INFO:teuthology.orchestra.run.smithi169.stdout: "prefix": { 2024-03-28T01:08:51.158 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "prefix", 2024-03-28T01:08:51.158 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.158 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.159 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.159 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.159 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.159 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.159 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.159 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "name prefix for any created local pool", 2024-03-28T01:08:51.159 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.159 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.159 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.159 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.159 INFO:teuthology.orchestra.run.smithi169.stdout: "subtree": { 2024-03-28T01:08:51.159 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "subtree", 2024-03-28T01:08:51.159 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.159 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.159 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.160 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "rack", 2024-03-28T01:08:51.160 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.160 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.160 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.160 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T01:08:51.160 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T01:08:51.160 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.160 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.160 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.160 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.160 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.160 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.160 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "mds_autoscaler", 2024-03-28T01:08:51.160 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.160 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.160 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.161 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.161 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.161 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.161 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.161 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.161 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.161 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.161 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.161 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.161 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.161 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.161 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.161 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.161 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.161 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.162 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.162 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.162 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.162 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.162 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.162 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.162 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.162 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.162 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.162 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.162 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.162 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.162 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.163 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.164 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.165 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "mirroring", 2024-03-28T01:08:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.166 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.167 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.168 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.170 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "nfs", 2024-03-28T01:08:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.171 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.172 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.173 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.174 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.175 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "orchestrator", 2024-03-28T01:08:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: "fail_fs": { 2024-03-28T01:08:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "fail_fs", 2024-03-28T01:08:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.176 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T01:08:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.177 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.178 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.179 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.180 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.181 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: "orchestrator": { 2024-03-28T01:08:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "orchestrator", 2024-03-28T01:08:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: "cephadm", 2024-03-28T01:08:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: "rook", 2024-03-28T01:08:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: "test_orchestrator" 2024-03-28T01:08:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.182 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Orchestrator backend", 2024-03-28T01:08:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "osd_perf_query", 2024-03-28T01:08:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.183 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.184 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.185 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.186 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.187 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.188 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.188 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.188 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "osd_support", 2024-03-28T01:08:51.190 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.191 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.192 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.193 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.194 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "pg_autoscaler", 2024-03-28T01:08:51.195 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.197 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.198 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.199 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.199 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.199 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.199 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.199 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.199 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.199 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.199 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.199 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.199 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.199 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.199 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.199 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.199 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.199 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "sleep_interval": { 2024-03-28T01:08:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "sleep_interval", 2024-03-28T01:08:51.200 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "60", 2024-03-28T01:08:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "threshold": { 2024-03-28T01:08:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "threshold", 2024-03-28T01:08:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "float", 2024-03-28T01:08:51.201 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "3.0", 2024-03-28T01:08:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "1.0", 2024-03-28T01:08:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "scaling threshold", 2024-03-28T01:08:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T01:08:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "progress", 2024-03-28T01:08:51.202 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "allow_pg_recovery_event": { 2024-03-28T01:08:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "allow_pg_recovery_event", 2024-03-28T01:08:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T01:08:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.203 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "enabled": { 2024-03-28T01:08:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "enabled", 2024-03-28T01:08:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.204 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.205 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.206 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.207 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.208 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "max_completed_events": { 2024-03-28T01:08:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "max_completed_events", 2024-03-28T01:08:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "50", 2024-03-28T01:08:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.209 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "number of past completed events to remember", 2024-03-28T01:08:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "sleep_interval": { 2024-03-28T01:08:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "sleep_interval", 2024-03-28T01:08:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "5", 2024-03-28T01:08:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.210 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how long the module is going to sleep", 2024-03-28T01:08:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "prometheus", 2024-03-28T01:08:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.211 INFO:teuthology.orchestra.run.smithi169.stdout: "cache": { 2024-03-28T01:08:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "cache", 2024-03-28T01:08:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "exclude_perf_counters": { 2024-03-28T01:08:51.212 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "exclude_perf_counters", 2024-03-28T01:08:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T01:08:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T01:08:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.213 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.214 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.215 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.216 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.217 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_stats_pools": { 2024-03-28T01:08:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rbd_stats_pools", 2024-03-28T01:08:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.218 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.219 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.219 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.219 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.219 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.219 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.219 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.219 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T01:08:51.219 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T01:08:51.219 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.219 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.219 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.219 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "300", 2024-03-28T01:08:51.219 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.219 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.219 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.220 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.220 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.220 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.220 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.220 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.220 INFO:teuthology.orchestra.run.smithi169.stdout: "scrape_interval": { 2024-03-28T01:08:51.220 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "scrape_interval", 2024-03-28T01:08:51.220 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "float", 2024-03-28T01:08:51.220 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.220 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.220 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "15.0", 2024-03-28T01:08:51.220 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.220 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.220 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.220 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.220 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.221 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.221 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.221 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.221 INFO:teuthology.orchestra.run.smithi169.stdout: "server_addr": { 2024-03-28T01:08:51.221 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "server_addr", 2024-03-28T01:08:51.221 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.221 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.221 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.221 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "::", 2024-03-28T01:08:51.221 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.221 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.221 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.221 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T01:08:51.221 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.221 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.222 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.222 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.222 INFO:teuthology.orchestra.run.smithi169.stdout: "server_port": { 2024-03-28T01:08:51.222 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "server_port", 2024-03-28T01:08:51.222 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.222 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.222 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.222 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "9283", 2024-03-28T01:08:51.222 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.222 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.222 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.222 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T01:08:51.222 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.222 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.223 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.223 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.223 INFO:teuthology.orchestra.run.smithi169.stdout: "stale_cache_strategy": { 2024-03-28T01:08:51.223 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "stale_cache_strategy", 2024-03-28T01:08:51.223 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.223 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.223 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.223 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "log", 2024-03-28T01:08:51.223 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.223 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.223 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.223 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.223 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.223 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.224 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.224 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.224 INFO:teuthology.orchestra.run.smithi169.stdout: "standby_behaviour": { 2024-03-28T01:08:51.224 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "standby_behaviour", 2024-03-28T01:08:51.224 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.224 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.224 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.224 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "default", 2024-03-28T01:08:51.224 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.224 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.224 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.224 INFO:teuthology.orchestra.run.smithi169.stdout: "default", 2024-03-28T01:08:51.224 INFO:teuthology.orchestra.run.smithi169.stdout: "error" 2024-03-28T01:08:51.224 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.224 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.225 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.225 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.225 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.225 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.225 INFO:teuthology.orchestra.run.smithi169.stdout: "standby_error_status_code": { 2024-03-28T01:08:51.225 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "standby_error_status_code", 2024-03-28T01:08:51.225 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.225 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.225 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.225 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "500", 2024-03-28T01:08:51.225 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "400", 2024-03-28T01:08:51.225 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "599", 2024-03-28T01:08:51.225 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.225 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.225 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.226 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.226 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.226 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.226 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.226 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.226 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.226 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rbd_support", 2024-03-28T01:08:51.226 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.226 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.226 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.226 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.226 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.226 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.226 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.226 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.227 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.227 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.227 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.227 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.227 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.227 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.227 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.227 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.227 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.227 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.227 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.227 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.227 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.227 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.227 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.228 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.228 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.228 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.228 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.228 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.228 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.228 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.228 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.228 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.228 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.228 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.228 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.228 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.228 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.228 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.229 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.229 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.229 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.229 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.229 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.229 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.229 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.229 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.229 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.229 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.229 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.229 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.229 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.229 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.229 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.230 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.230 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.230 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.230 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.230 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.230 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.230 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.230 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.230 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.230 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.230 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.230 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.230 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.230 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.231 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.231 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.231 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.231 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.231 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.231 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.231 INFO:teuthology.orchestra.run.smithi169.stdout: "max_concurrent_snap_create": { 2024-03-28T01:08:51.231 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "max_concurrent_snap_create", 2024-03-28T01:08:51.231 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.231 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.231 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.231 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.231 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.231 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.231 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.231 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.232 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.232 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.232 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.232 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.232 INFO:teuthology.orchestra.run.smithi169.stdout: "mirror_snapshot_schedule": { 2024-03-28T01:08:51.232 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T01:08:51.232 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.232 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.232 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.232 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.232 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.232 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.232 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.232 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.232 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.232 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.233 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.233 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.233 INFO:teuthology.orchestra.run.smithi169.stdout: "trash_purge_schedule": { 2024-03-28T01:08:51.233 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "trash_purge_schedule", 2024-03-28T01:08:51.233 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.233 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.233 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.233 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.233 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.233 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.233 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.233 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.233 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.233 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.233 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.234 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.234 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.234 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.234 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.234 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "restful", 2024-03-28T01:08:51.234 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.234 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.234 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.234 INFO:teuthology.orchestra.run.smithi169.stdout: "enable_auth": { 2024-03-28T01:08:51.234 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "enable_auth", 2024-03-28T01:08:51.234 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.234 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.234 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.234 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.234 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.235 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.235 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.235 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.235 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.235 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.235 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.235 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.235 INFO:teuthology.orchestra.run.smithi169.stdout: "key_file": { 2024-03-28T01:08:51.235 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "key_file", 2024-03-28T01:08:51.235 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.235 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.235 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.235 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.235 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.235 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.236 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.236 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.236 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.236 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.236 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.236 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.236 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.236 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.236 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.236 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.236 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.237 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.237 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.237 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.237 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.237 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.237 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.237 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.237 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.237 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.237 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.237 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.237 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.237 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.237 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.238 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.238 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.238 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.238 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.238 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.238 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.238 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.238 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.238 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.238 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.238 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.238 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.238 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.238 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.238 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.239 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.239 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.239 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.239 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.239 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.239 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.239 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.239 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.239 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.239 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.239 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.239 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.239 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.239 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.240 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.240 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.240 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.240 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.240 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.240 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.240 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.240 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.240 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.240 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.240 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.240 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.240 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.240 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.240 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.241 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.241 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.241 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.241 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.241 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.241 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.241 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.241 INFO:teuthology.orchestra.run.smithi169.stdout: "server_addr": { 2024-03-28T01:08:51.241 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "server_addr", 2024-03-28T01:08:51.241 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.241 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.241 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.241 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.241 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.241 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.242 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.242 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.242 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.242 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.242 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.242 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.242 INFO:teuthology.orchestra.run.smithi169.stdout: "server_port": { 2024-03-28T01:08:51.242 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "server_port", 2024-03-28T01:08:51.242 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.242 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.242 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.242 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.242 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.242 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.242 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.243 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.243 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.243 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.243 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.243 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.243 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.243 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.243 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.243 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rgw", 2024-03-28T01:08:51.243 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.243 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.243 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.243 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.243 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.243 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.244 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.244 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.244 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.244 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.244 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.244 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.244 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.244 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.244 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.244 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.244 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.244 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.244 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.244 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.244 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.244 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.245 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.245 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.245 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.245 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.245 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.245 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.245 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.245 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.245 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.245 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.245 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.245 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.245 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.245 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.245 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.245 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.246 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.246 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.246 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.246 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.246 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.246 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.246 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.246 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.246 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.246 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.246 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.246 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.246 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.246 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.246 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.247 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.247 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.247 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.247 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.247 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.247 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.247 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.247 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.247 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.247 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.247 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.247 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.247 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.247 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.247 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.247 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.248 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.248 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.248 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.248 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.248 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.248 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.248 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.248 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rook", 2024-03-28T01:08:51.248 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.248 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.248 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.248 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.248 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.248 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.248 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.249 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.249 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.249 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.249 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.249 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.249 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.249 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.249 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.249 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.249 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.249 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.249 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.249 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.249 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.249 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.250 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.250 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.250 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.250 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.250 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.250 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.250 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.250 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.250 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.250 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.250 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.250 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.250 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.250 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.251 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.251 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.251 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.251 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.251 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.251 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.251 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.251 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.251 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.251 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.251 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.251 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.251 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.251 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.251 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.252 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.252 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.252 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.252 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.252 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.252 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.252 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.252 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.252 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.252 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.252 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.252 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.252 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.252 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.252 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.252 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.253 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.253 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.253 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.253 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.253 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.253 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.253 INFO:teuthology.orchestra.run.smithi169.stdout: "storage_class": { 2024-03-28T01:08:51.253 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "storage_class", 2024-03-28T01:08:51.253 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.253 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.253 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.253 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "local", 2024-03-28T01:08:51.253 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.253 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.253 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.254 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T01:08:51.254 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.254 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.254 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.254 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.254 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.254 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.254 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.254 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "selftest", 2024-03-28T01:08:51.254 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.254 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.254 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.254 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.254 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.254 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.255 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.255 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.255 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.255 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.255 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.255 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.255 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.255 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.255 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.255 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.255 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.255 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.255 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.255 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.255 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.256 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.256 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.256 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.256 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.256 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.256 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.256 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.256 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.256 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.256 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.256 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.256 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.256 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.256 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.256 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.256 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.257 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.257 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.257 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.257 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.257 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.257 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.257 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.257 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.257 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.257 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.257 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.257 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.257 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.257 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.257 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.257 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.258 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.258 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.258 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.258 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.258 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.258 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.258 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.258 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.258 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.258 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.258 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.258 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.258 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.258 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.259 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.259 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.259 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.259 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.259 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.259 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.259 INFO:teuthology.orchestra.run.smithi169.stdout: "roption1": { 2024-03-28T01:08:51.259 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "roption1", 2024-03-28T01:08:51.259 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.259 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.259 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.259 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.259 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.259 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.259 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.259 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.260 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.260 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.260 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.260 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.260 INFO:teuthology.orchestra.run.smithi169.stdout: "roption2": { 2024-03-28T01:08:51.260 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "roption2", 2024-03-28T01:08:51.260 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.260 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.260 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.260 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "xyz", 2024-03-28T01:08:51.260 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.260 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.260 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.260 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.260 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.260 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.261 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.261 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.261 INFO:teuthology.orchestra.run.smithi169.stdout: "rwoption1": { 2024-03-28T01:08:51.261 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rwoption1", 2024-03-28T01:08:51.261 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.261 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.261 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.261 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.261 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.261 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.261 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.261 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.261 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.261 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.261 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.261 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.262 INFO:teuthology.orchestra.run.smithi169.stdout: "rwoption2": { 2024-03-28T01:08:51.262 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rwoption2", 2024-03-28T01:08:51.262 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.262 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.262 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.262 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.262 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.262 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.262 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.262 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.262 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.262 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.262 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.263 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.263 INFO:teuthology.orchestra.run.smithi169.stdout: "rwoption3": { 2024-03-28T01:08:51.263 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rwoption3", 2024-03-28T01:08:51.263 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "float", 2024-03-28T01:08:51.263 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.263 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.263 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.263 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.263 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.263 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.263 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.263 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.263 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.263 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.263 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.264 INFO:teuthology.orchestra.run.smithi169.stdout: "rwoption4": { 2024-03-28T01:08:51.264 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rwoption4", 2024-03-28T01:08:51.264 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.264 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.264 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.264 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.264 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.264 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.264 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.264 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.264 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.264 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.264 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.264 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.264 INFO:teuthology.orchestra.run.smithi169.stdout: "rwoption5": { 2024-03-28T01:08:51.264 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rwoption5", 2024-03-28T01:08:51.265 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.265 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.265 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.265 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.265 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.265 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.265 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.265 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.265 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.265 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.265 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.265 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.265 INFO:teuthology.orchestra.run.smithi169.stdout: "rwoption6": { 2024-03-28T01:08:51.265 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rwoption6", 2024-03-28T01:08:51.265 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.266 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.266 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.266 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.266 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.266 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.266 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.266 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.266 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.266 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.266 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.266 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.266 INFO:teuthology.orchestra.run.smithi169.stdout: "rwoption7": { 2024-03-28T01:08:51.266 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rwoption7", 2024-03-28T01:08:51.266 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.266 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.267 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.267 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.267 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "1", 2024-03-28T01:08:51.267 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "42", 2024-03-28T01:08:51.267 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.267 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.267 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.267 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.267 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.267 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.267 INFO:teuthology.orchestra.run.smithi169.stdout: "testkey": { 2024-03-28T01:08:51.267 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "testkey", 2024-03-28T01:08:51.267 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.267 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.267 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.268 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.268 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.268 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.268 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.268 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.268 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.268 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.268 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.268 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.268 INFO:teuthology.orchestra.run.smithi169.stdout: "testlkey": { 2024-03-28T01:08:51.268 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "testlkey", 2024-03-28T01:08:51.268 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.268 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.268 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.268 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.268 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.269 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.269 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.269 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.269 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.269 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.269 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.269 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.269 INFO:teuthology.orchestra.run.smithi169.stdout: "testnewline": { 2024-03-28T01:08:51.269 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "testnewline", 2024-03-28T01:08:51.269 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.269 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.269 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.269 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.269 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.269 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.269 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.270 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.270 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.270 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.270 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.270 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.270 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.270 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.270 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.270 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "snap_schedule", 2024-03-28T01:08:51.270 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.270 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.270 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.270 INFO:teuthology.orchestra.run.smithi169.stdout: "allow_m_granularity": { 2024-03-28T01:08:51.270 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "allow_m_granularity", 2024-03-28T01:08:51.270 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.271 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.271 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.271 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.271 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.271 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.271 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.271 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T01:08:51.271 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.271 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.271 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.271 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.271 INFO:teuthology.orchestra.run.smithi169.stdout: "dump_on_update": { 2024-03-28T01:08:51.271 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "dump_on_update", 2024-03-28T01:08:51.271 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.271 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.272 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.272 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.272 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.272 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.272 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.272 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "dump database to debug log on update", 2024-03-28T01:08:51.272 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.272 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.272 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.272 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.272 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.272 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.272 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.272 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.272 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.272 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.273 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.273 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.273 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.273 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.273 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.273 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.273 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.273 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.273 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.273 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.273 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.273 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.273 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.273 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.273 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.274 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.274 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.274 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.274 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.274 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.274 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.274 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.274 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.274 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.274 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.274 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.274 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.274 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.274 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.275 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.275 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.275 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.275 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.275 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.275 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.275 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.275 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.275 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.275 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.275 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.275 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.275 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.275 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.276 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.276 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.276 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.276 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.276 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.276 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.276 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.276 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.276 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.276 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.276 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.276 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.276 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.276 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.276 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.277 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.277 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.277 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.277 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.277 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.277 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.277 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.277 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.277 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.277 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "stats", 2024-03-28T01:08:51.277 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.277 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.277 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.277 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.277 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.278 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.278 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.278 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.278 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.278 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.278 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.278 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.278 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.278 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.278 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.278 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.278 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.278 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.278 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.279 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.279 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.279 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.279 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.279 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.279 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.279 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.279 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.279 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.279 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.279 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.279 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.279 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.279 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.279 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.279 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.280 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.280 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.280 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.280 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.280 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.280 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.280 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.280 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.280 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.280 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.280 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.280 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.280 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.280 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.280 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.281 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.281 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.281 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.281 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.281 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.281 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.281 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.281 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.281 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.281 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.281 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.281 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.281 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.281 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.281 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.282 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.282 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.282 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.282 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.282 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.282 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.282 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.282 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.282 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.282 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.282 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.282 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "status", 2024-03-28T01:08:51.282 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.282 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.282 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.283 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.283 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.283 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.283 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.283 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.283 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.283 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.283 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.283 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.283 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.283 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.283 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.283 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.283 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.283 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.284 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.284 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.284 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.284 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.284 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.284 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.284 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.284 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.284 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.284 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.284 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.284 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.284 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.284 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.284 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.284 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.285 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.285 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.285 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.285 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.285 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.285 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.285 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.285 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.285 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.285 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.285 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.285 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.285 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.285 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.285 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.285 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.286 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.286 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.286 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.286 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.286 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.286 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.286 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.286 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.286 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.286 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.286 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.286 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.286 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.286 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.287 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.287 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.287 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.287 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.287 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.287 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.287 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.287 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.287 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.287 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.287 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.287 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.287 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "telegraf", 2024-03-28T01:08:51.287 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.287 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.288 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.288 INFO:teuthology.orchestra.run.smithi169.stdout: "address": { 2024-03-28T01:08:51.288 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "address", 2024-03-28T01:08:51.288 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.288 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.288 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.288 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T01:08:51.288 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.288 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.288 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.288 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.288 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.288 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.288 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.288 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.289 INFO:teuthology.orchestra.run.smithi169.stdout: "interval": { 2024-03-28T01:08:51.289 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "interval", 2024-03-28T01:08:51.289 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.289 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.289 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.289 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "15", 2024-03-28T01:08:51.289 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.289 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.289 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.289 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.289 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.289 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.289 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.289 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.289 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.290 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.290 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.290 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.290 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.290 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.290 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.290 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.290 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.290 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.290 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.290 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.290 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.290 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.290 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.291 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.291 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.291 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.291 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.291 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.291 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.291 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.291 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.291 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.291 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.291 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.291 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.291 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.291 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.291 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.292 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.292 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.292 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.292 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.292 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.292 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.292 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.292 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.292 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.292 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.292 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.292 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.292 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.292 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.292 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.293 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.293 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.293 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.293 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.293 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.293 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.293 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.293 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.293 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.293 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.293 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.293 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.293 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.293 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.293 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.293 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.294 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.294 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.294 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.294 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.294 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.294 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.294 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.294 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.294 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.294 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.294 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.294 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.294 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "telemetry", 2024-03-28T01:08:51.294 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.295 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.295 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.295 INFO:teuthology.orchestra.run.smithi169.stdout: "channel_basic": { 2024-03-28T01:08:51.295 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "channel_basic", 2024-03-28T01:08:51.295 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.295 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.295 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.295 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.295 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.295 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.295 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.295 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T01:08:51.295 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.295 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.295 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.296 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.296 INFO:teuthology.orchestra.run.smithi169.stdout: "channel_crash": { 2024-03-28T01:08:51.296 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "channel_crash", 2024-03-28T01:08:51.296 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.296 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.296 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.296 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.296 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.296 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.296 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.296 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T01:08:51.296 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.296 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.296 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.296 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.297 INFO:teuthology.orchestra.run.smithi169.stdout: "channel_device": { 2024-03-28T01:08:51.297 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "channel_device", 2024-03-28T01:08:51.297 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.297 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.297 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.297 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.297 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.297 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.297 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.297 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T01:08:51.297 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.297 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.297 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.297 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.297 INFO:teuthology.orchestra.run.smithi169.stdout: "channel_ident": { 2024-03-28T01:08:51.298 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "channel_ident", 2024-03-28T01:08:51.298 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.298 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.298 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.298 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.298 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.298 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.298 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.298 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T01:08:51.298 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.298 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.298 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.298 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.298 INFO:teuthology.orchestra.run.smithi169.stdout: "channel_perf": { 2024-03-28T01:08:51.298 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "channel_perf", 2024-03-28T01:08:51.299 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.299 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.299 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.299 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.299 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.299 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.299 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.299 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T01:08:51.299 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.299 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.299 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.299 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.299 INFO:teuthology.orchestra.run.smithi169.stdout: "contact": { 2024-03-28T01:08:51.299 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "contact", 2024-03-28T01:08:51.299 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.300 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.300 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.300 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.300 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.300 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.300 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.300 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.300 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.300 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.300 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.300 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.300 INFO:teuthology.orchestra.run.smithi169.stdout: "description": { 2024-03-28T01:08:51.300 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "description", 2024-03-28T01:08:51.300 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.300 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.300 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.301 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.301 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.301 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.301 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.301 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.301 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.301 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.301 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.301 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.301 INFO:teuthology.orchestra.run.smithi169.stdout: "device_url": { 2024-03-28T01:08:51.301 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "device_url", 2024-03-28T01:08:51.301 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.301 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.301 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.301 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T01:08:51.302 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.302 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.302 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.302 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.302 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.302 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.302 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.302 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.302 INFO:teuthology.orchestra.run.smithi169.stdout: "enabled": { 2024-03-28T01:08:51.302 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "enabled", 2024-03-28T01:08:51.302 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.302 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.302 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.302 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.302 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.303 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.303 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.303 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.303 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.303 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.303 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.303 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.303 INFO:teuthology.orchestra.run.smithi169.stdout: "interval": { 2024-03-28T01:08:51.303 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "interval", 2024-03-28T01:08:51.303 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.303 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.303 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.303 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "24", 2024-03-28T01:08:51.303 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "8", 2024-03-28T01:08:51.303 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.304 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.304 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.304 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.304 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.304 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.304 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.304 INFO:teuthology.orchestra.run.smithi169.stdout: "last_opt_revision": { 2024-03-28T01:08:51.304 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "last_opt_revision", 2024-03-28T01:08:51.304 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.304 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.304 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.304 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "1", 2024-03-28T01:08:51.304 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.304 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.304 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.305 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.305 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.305 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.305 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.305 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.305 INFO:teuthology.orchestra.run.smithi169.stdout: "leaderboard": { 2024-03-28T01:08:51.305 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "leaderboard", 2024-03-28T01:08:51.305 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.305 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.305 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.305 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.305 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.305 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.305 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.305 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.305 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.306 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.306 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.306 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.306 INFO:teuthology.orchestra.run.smithi169.stdout: "leaderboard_description": { 2024-03-28T01:08:51.306 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "leaderboard_description", 2024-03-28T01:08:51.306 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.306 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.306 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.306 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.306 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.306 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.306 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.306 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.306 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.307 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.307 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.307 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.307 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.307 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.307 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.307 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.307 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.307 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.307 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.307 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.307 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.307 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.307 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.307 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.308 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.308 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.308 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.308 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.308 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.308 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.308 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.308 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.308 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.308 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.308 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.308 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.308 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.308 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.309 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.309 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.309 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.309 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.309 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.309 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.309 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.309 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.309 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.309 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.309 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.309 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.309 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.309 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.309 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.309 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.310 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.310 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.310 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.310 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.310 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.310 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.310 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.310 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.310 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.310 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.310 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.310 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.310 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.310 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.310 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.311 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.311 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.311 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.311 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.311 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.311 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.311 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.311 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.311 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.311 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.311 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.311 INFO:teuthology.orchestra.run.smithi169.stdout: "organization": { 2024-03-28T01:08:51.311 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "organization", 2024-03-28T01:08:51.311 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.311 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.312 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.312 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.312 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.312 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.312 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.312 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.312 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.312 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.312 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.312 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.312 INFO:teuthology.orchestra.run.smithi169.stdout: "proxy": { 2024-03-28T01:08:51.312 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "proxy", 2024-03-28T01:08:51.312 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.312 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.312 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.313 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.313 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.313 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.313 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.313 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.313 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.313 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.313 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.313 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.313 INFO:teuthology.orchestra.run.smithi169.stdout: "url": { 2024-03-28T01:08:51.313 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "url", 2024-03-28T01:08:51.313 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.313 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.313 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.314 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T01:08:51.314 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.314 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.314 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.314 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.314 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.314 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.314 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.314 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.314 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.314 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.314 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.314 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "test_orchestrator", 2024-03-28T01:08:51.314 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.314 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.315 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.315 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.315 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.315 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.315 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.315 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.315 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.315 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.315 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.315 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.315 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.315 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.315 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.315 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.315 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.315 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.316 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.316 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.316 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.316 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.316 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.316 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.316 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.316 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.316 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.316 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.316 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.316 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.316 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.316 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.317 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.317 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.317 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.317 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.317 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.317 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.317 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.317 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.317 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.317 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.317 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.317 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.317 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.317 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.318 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.318 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.318 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.318 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.318 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.318 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.318 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.318 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.318 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.318 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.318 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.318 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.318 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.318 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.319 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.319 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.319 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.319 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.319 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.319 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.319 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.319 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.319 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.319 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.319 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.319 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.319 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.320 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "volumes", 2024-03-28T01:08:51.320 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.320 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.320 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.320 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.320 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.320 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.320 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.320 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.320 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.320 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.320 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.320 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.320 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.320 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.321 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.321 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.321 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.321 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.321 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.321 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.321 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.321 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.321 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.321 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.321 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.321 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.321 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.321 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.321 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.322 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.322 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.322 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.322 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.322 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.322 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.322 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.322 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.322 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.322 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.322 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.322 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.322 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.322 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.323 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.323 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.323 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.323 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.323 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.323 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.323 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.323 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.323 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.323 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.323 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.323 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.323 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.323 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.324 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.324 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.324 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.324 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.324 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.324 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.324 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.324 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.324 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.324 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.324 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.324 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.324 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.324 INFO:teuthology.orchestra.run.smithi169.stdout: "max_concurrent_clones": { 2024-03-28T01:08:51.325 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "max_concurrent_clones", 2024-03-28T01:08:51.325 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.325 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.325 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.325 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "4", 2024-03-28T01:08:51.325 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.325 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.325 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.325 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T01:08:51.325 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.325 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.325 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.325 INFO:teuthology.orchestra.run.smithi169.stdout: "periodic_async_work": { 2024-03-28T01:08:51.325 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "periodic_async_work", 2024-03-28T01:08:51.326 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.326 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.326 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.326 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.326 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.326 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.326 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.326 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Periodically check for async work", 2024-03-28T01:08:51.326 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.326 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.326 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.326 INFO:teuthology.orchestra.run.smithi169.stdout: "snapshot_clone_delay": { 2024-03-28T01:08:51.326 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "snapshot_clone_delay", 2024-03-28T01:08:51.327 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.327 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.327 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.327 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0", 2024-03-28T01:08:51.327 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.327 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.327 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.327 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T01:08:51.327 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.327 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.327 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.327 INFO:teuthology.orchestra.run.smithi169.stdout: "snapshot_clone_no_wait": { 2024-03-28T01:08:51.327 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T01:08:51.328 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.328 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.328 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.328 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.328 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.328 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.328 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.328 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T01:08:51.328 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.328 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.328 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.328 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.328 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.329 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "zabbix", 2024-03-28T01:08:51.329 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.329 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.329 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.329 INFO:teuthology.orchestra.run.smithi169.stdout: "discovery_interval": { 2024-03-28T01:08:51.329 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "discovery_interval", 2024-03-28T01:08:51.329 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "uint", 2024-03-28T01:08:51.329 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.329 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.329 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "100", 2024-03-28T01:08:51.329 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.329 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.329 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.329 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.329 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.330 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.330 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.330 INFO:teuthology.orchestra.run.smithi169.stdout: "identifier": { 2024-03-28T01:08:51.330 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "identifier", 2024-03-28T01:08:51.330 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.330 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.330 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.330 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.330 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.330 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.330 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.330 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.330 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.331 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.331 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.331 INFO:teuthology.orchestra.run.smithi169.stdout: "interval": { 2024-03-28T01:08:51.331 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "interval", 2024-03-28T01:08:51.331 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.331 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.331 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.331 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "60", 2024-03-28T01:08:51.331 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.331 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.331 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.331 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.331 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.331 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.332 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.332 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.332 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.332 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.332 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.332 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.332 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.332 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.332 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.332 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.332 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.332 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.332 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.332 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.332 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.333 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.333 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.333 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.333 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.333 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.333 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.333 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.333 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.333 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.333 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.333 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.333 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.333 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.333 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.334 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.334 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.334 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.334 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.334 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.334 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.334 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.334 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.334 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.334 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.334 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.334 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.334 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.334 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.335 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.335 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.335 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.335 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.335 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.335 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.335 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.335 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.335 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.335 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.335 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.335 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.335 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.335 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.335 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.335 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.336 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.336 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.336 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.336 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.336 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.336 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.336 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.336 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.336 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.336 INFO:teuthology.orchestra.run.smithi169.stdout: "zabbix_host": { 2024-03-28T01:08:51.336 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "zabbix_host", 2024-03-28T01:08:51.336 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.336 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.336 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.336 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.337 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.337 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.337 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.337 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.337 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.337 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.337 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.337 INFO:teuthology.orchestra.run.smithi169.stdout: "zabbix_port": { 2024-03-28T01:08:51.337 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "zabbix_port", 2024-03-28T01:08:51.337 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.337 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.337 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.337 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10051", 2024-03-28T01:08:51.337 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.338 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.338 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.338 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.338 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.338 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.338 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.338 INFO:teuthology.orchestra.run.smithi169.stdout: "zabbix_sender": { 2024-03-28T01:08:51.338 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "zabbix_sender", 2024-03-28T01:08:51.338 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.338 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.338 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.338 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T01:08:51.338 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.338 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.338 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.339 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.339 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.339 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.339 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.339 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.339 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.339 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.339 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-28T01:08:51.339 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.339 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.339 INFO:teuthology.orchestra.run.smithi169.stdout: "gid": 24116, 2024-03-28T01:08:51.339 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "y", 2024-03-28T01:08:51.339 INFO:teuthology.orchestra.run.smithi169.stdout: "mgr_features": 4540701547738038271, 2024-03-28T01:08:51.339 INFO:teuthology.orchestra.run.smithi169.stdout: "available_modules": [ 2024-03-28T01:08:51.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.340 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "alerts", 2024-03-28T01:08:51.340 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.340 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.340 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.340 INFO:teuthology.orchestra.run.smithi169.stdout: "interval": { 2024-03-28T01:08:51.340 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "interval", 2024-03-28T01:08:51.340 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.340 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.340 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.340 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "60", 2024-03-28T01:08:51.340 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.340 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.340 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.340 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T01:08:51.341 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.341 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.341 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.341 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.341 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.341 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.341 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.341 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.341 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.341 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.341 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.341 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.341 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.341 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.342 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.342 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.342 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.342 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.342 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.342 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.342 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.342 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.342 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.342 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.342 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.342 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.342 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.342 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.343 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.343 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.343 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.343 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.343 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.343 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.343 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.343 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.343 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.343 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.343 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.343 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.343 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.344 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.344 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.344 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.344 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.344 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.344 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.344 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.344 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.344 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.344 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.344 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.344 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.344 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.344 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.345 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.345 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.345 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.345 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.345 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.345 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.345 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.345 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.345 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.345 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.345 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.345 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.345 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.345 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.345 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.346 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_destination": { 2024-03-28T01:08:51.346 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_destination", 2024-03-28T01:08:51.346 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.346 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.346 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.346 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.346 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.346 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.346 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.346 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Email address to send alerts to", 2024-03-28T01:08:51.346 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.346 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.346 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.347 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_from_name": { 2024-03-28T01:08:51.347 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_from_name", 2024-03-28T01:08:51.347 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.347 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.347 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.347 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "Ceph", 2024-03-28T01:08:51.347 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.347 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.347 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.347 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Email From: name", 2024-03-28T01:08:51.347 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.347 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.347 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.348 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_host": { 2024-03-28T01:08:51.348 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_host", 2024-03-28T01:08:51.348 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.348 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.348 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.348 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.348 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.348 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.348 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.348 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "SMTP server", 2024-03-28T01:08:51.348 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.348 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.348 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.348 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.348 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_password": { 2024-03-28T01:08:51.348 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_password", 2024-03-28T01:08:51.349 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.349 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.349 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.349 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.349 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.349 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.349 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.349 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Password to authenticate with", 2024-03-28T01:08:51.349 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.349 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.349 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.349 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.349 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_port": { 2024-03-28T01:08:51.349 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_port", 2024-03-28T01:08:51.349 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.350 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.350 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.350 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "465", 2024-03-28T01:08:51.350 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.350 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.350 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.350 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "SMTP port", 2024-03-28T01:08:51.350 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.350 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.350 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.350 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_sender": { 2024-03-28T01:08:51.350 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_sender", 2024-03-28T01:08:51.350 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.351 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.351 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.351 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.351 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.351 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.351 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.351 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "SMTP envelope sender", 2024-03-28T01:08:51.351 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.351 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.351 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.351 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_ssl": { 2024-03-28T01:08:51.351 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_ssl", 2024-03-28T01:08:51.351 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.351 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.351 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.351 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.352 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.352 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.352 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.352 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T01:08:51.352 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.352 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.352 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.352 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_user": { 2024-03-28T01:08:51.352 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_user", 2024-03-28T01:08:51.352 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.352 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.352 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.352 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "User to authenticate as", 2024-03-28T01:08:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "balancer", 2024-03-28T01:08:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.354 INFO:teuthology.orchestra.run.smithi169.stdout: "active": { 2024-03-28T01:08:51.354 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "active", 2024-03-28T01:08:51.354 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.354 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.354 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.354 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.354 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.354 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.354 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.354 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T01:08:51.354 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.354 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.354 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.355 INFO:teuthology.orchestra.run.smithi169.stdout: "begin_time": { 2024-03-28T01:08:51.355 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "begin_time", 2024-03-28T01:08:51.355 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.355 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.355 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.355 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0000", 2024-03-28T01:08:51.355 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.355 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.355 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.355 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T01:08:51.355 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T01:08:51.355 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.355 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.355 INFO:teuthology.orchestra.run.smithi169.stdout: "begin_weekday": { 2024-03-28T01:08:51.356 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "begin_weekday", 2024-03-28T01:08:51.356 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "uint", 2024-03-28T01:08:51.356 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.356 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.356 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0", 2024-03-28T01:08:51.356 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "0", 2024-03-28T01:08:51.356 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "6", 2024-03-28T01:08:51.356 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.356 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T01:08:51.356 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T01:08:51.356 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.356 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.356 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_compat_max_iterations": { 2024-03-28T01:08:51.356 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "crush_compat_max_iterations", 2024-03-28T01:08:51.357 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "uint", 2024-03-28T01:08:51.357 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.357 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.357 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "25", 2024-03-28T01:08:51.357 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "1", 2024-03-28T01:08:51.357 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "250", 2024-03-28T01:08:51.357 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.357 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T01:08:51.357 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.357 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.357 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.357 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.357 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_compat_metrics": { 2024-03-28T01:08:51.357 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "crush_compat_metrics", 2024-03-28T01:08:51.357 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.358 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.358 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.358 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T01:08:51.358 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.358 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.358 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.358 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T01:08:51.358 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T01:08:51.358 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.358 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.358 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_compat_step": { 2024-03-28T01:08:51.358 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "crush_compat_step", 2024-03-28T01:08:51.358 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "float", 2024-03-28T01:08:51.359 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.359 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.359 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0.5", 2024-03-28T01:08:51.359 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "0.001", 2024-03-28T01:08:51.359 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "0.999", 2024-03-28T01:08:51.359 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.359 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "aggressiveness of optimization", 2024-03-28T01:08:51.359 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T01:08:51.359 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.359 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.359 INFO:teuthology.orchestra.run.smithi169.stdout: "end_time": { 2024-03-28T01:08:51.359 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "end_time", 2024-03-28T01:08:51.359 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.359 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.359 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.360 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "2359", 2024-03-28T01:08:51.360 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.360 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.360 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.360 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T01:08:51.360 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T01:08:51.360 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.360 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.360 INFO:teuthology.orchestra.run.smithi169.stdout: "end_weekday": { 2024-03-28T01:08:51.360 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "end_weekday", 2024-03-28T01:08:51.360 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "uint", 2024-03-28T01:08:51.360 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.360 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.360 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0", 2024-03-28T01:08:51.360 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "0", 2024-03-28T01:08:51.361 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "6", 2024-03-28T01:08:51.361 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.361 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T01:08:51.361 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T01:08:51.361 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.361 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.361 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.361 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.361 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.361 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.361 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.361 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.361 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.361 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.361 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.362 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.362 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.362 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.362 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.362 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.362 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.362 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.362 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.362 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.362 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.362 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.362 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.362 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.362 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.363 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.363 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.363 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.363 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.363 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.363 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.363 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.363 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.363 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.363 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.363 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.363 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.363 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.363 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.364 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.364 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.364 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.364 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.364 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.364 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.364 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.364 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.364 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.364 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.364 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.364 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.364 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.364 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.364 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.365 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.365 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.365 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.365 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.365 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.365 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.365 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.365 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.365 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.365 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.365 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.365 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.365 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.365 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.366 INFO:teuthology.orchestra.run.smithi169.stdout: "min_score": { 2024-03-28T01:08:51.366 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "min_score", 2024-03-28T01:08:51.366 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "float", 2024-03-28T01:08:51.366 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.366 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.366 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0", 2024-03-28T01:08:51.366 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.366 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.366 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.366 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T01:08:51.366 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.366 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.366 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.367 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": { 2024-03-28T01:08:51.367 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "mode", 2024-03-28T01:08:51.367 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.367 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.367 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.367 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "upmap", 2024-03-28T01:08:51.367 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.367 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.367 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.367 INFO:teuthology.orchestra.run.smithi169.stdout: "crush-compat", 2024-03-28T01:08:51.367 INFO:teuthology.orchestra.run.smithi169.stdout: "none", 2024-03-28T01:08:51.367 INFO:teuthology.orchestra.run.smithi169.stdout: "read", 2024-03-28T01:08:51.367 INFO:teuthology.orchestra.run.smithi169.stdout: "upmap", 2024-03-28T01:08:51.367 INFO:teuthology.orchestra.run.smithi169.stdout: "upmap-read" 2024-03-28T01:08:51.367 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.367 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Balancer mode", 2024-03-28T01:08:51.368 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.368 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.368 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.368 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_ids": { 2024-03-28T01:08:51.368 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "pool_ids", 2024-03-28T01:08:51.368 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.368 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.368 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.368 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.368 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.368 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.368 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.368 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T01:08:51.369 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.369 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.369 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.369 INFO:teuthology.orchestra.run.smithi169.stdout: "sleep_interval": { 2024-03-28T01:08:51.369 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "sleep_interval", 2024-03-28T01:08:51.369 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.369 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.369 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.369 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "60", 2024-03-28T01:08:51.369 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.369 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.369 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.369 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T01:08:51.369 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.369 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.370 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.370 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.370 INFO:teuthology.orchestra.run.smithi169.stdout: "upmap_max_deviation": { 2024-03-28T01:08:51.370 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "upmap_max_deviation", 2024-03-28T01:08:51.370 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.370 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.370 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.370 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "5", 2024-03-28T01:08:51.370 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "1", 2024-03-28T01:08:51.370 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.370 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.370 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T01:08:51.370 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T01:08:51.370 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.370 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.371 INFO:teuthology.orchestra.run.smithi169.stdout: "upmap_max_optimizations": { 2024-03-28T01:08:51.371 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "upmap_max_optimizations", 2024-03-28T01:08:51.371 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "uint", 2024-03-28T01:08:51.371 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.371 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.371 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.371 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.371 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.371 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.371 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T01:08:51.371 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.371 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.371 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.371 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.372 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.372 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "cephadm", 2024-03-28T01:08:51.372 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.372 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.372 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.372 INFO:teuthology.orchestra.run.smithi169.stdout: "agent_down_multiplier": { 2024-03-28T01:08:51.372 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "agent_down_multiplier", 2024-03-28T01:08:51.372 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "float", 2024-03-28T01:08:51.372 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.372 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.372 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "3.0", 2024-03-28T01:08:51.372 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.372 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.372 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.373 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T01:08:51.373 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.373 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.373 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.373 INFO:teuthology.orchestra.run.smithi169.stdout: "agent_refresh_rate": { 2024-03-28T01:08:51.373 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "agent_refresh_rate", 2024-03-28T01:08:51.373 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.373 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.373 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.373 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "20", 2024-03-28T01:08:51.373 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.373 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.373 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.373 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T01:08:51.374 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.374 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.374 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.374 INFO:teuthology.orchestra.run.smithi169.stdout: "agent_starting_port": { 2024-03-28T01:08:51.374 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "agent_starting_port", 2024-03-28T01:08:51.374 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.374 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.374 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.374 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "4721", 2024-03-28T01:08:51.374 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.374 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.374 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.374 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T01:08:51.374 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.375 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.375 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.375 INFO:teuthology.orchestra.run.smithi169.stdout: "allow_ptrace": { 2024-03-28T01:08:51.375 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "allow_ptrace", 2024-03-28T01:08:51.375 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.375 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.375 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.375 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.375 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.375 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.375 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.375 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T01:08:51.375 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T01:08:51.376 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.376 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.376 INFO:teuthology.orchestra.run.smithi169.stdout: "autotune_interval": { 2024-03-28T01:08:51.376 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "autotune_interval", 2024-03-28T01:08:51.376 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.376 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.376 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.376 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "600", 2024-03-28T01:08:51.376 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.376 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.376 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.376 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T01:08:51.376 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.377 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.377 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.377 INFO:teuthology.orchestra.run.smithi169.stdout: "autotune_memory_target_ratio": { 2024-03-28T01:08:51.377 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T01:08:51.377 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "float", 2024-03-28T01:08:51.377 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.377 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.377 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0.7", 2024-03-28T01:08:51.377 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.377 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.377 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.377 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T01:08:51.377 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.377 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.377 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.378 INFO:teuthology.orchestra.run.smithi169.stdout: "cephadm_log_destination": { 2024-03-28T01:08:51.378 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "cephadm_log_destination", 2024-03-28T01:08:51.378 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.378 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.378 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.378 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.378 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.378 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.378 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.378 INFO:teuthology.orchestra.run.smithi169.stdout: "file", 2024-03-28T01:08:51.378 INFO:teuthology.orchestra.run.smithi169.stdout: "file,syslog", 2024-03-28T01:08:51.378 INFO:teuthology.orchestra.run.smithi169.stdout: "syslog" 2024-03-28T01:08:51.378 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.379 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T01:08:51.379 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.379 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.379 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.379 INFO:teuthology.orchestra.run.smithi169.stdout: "cgroups_split": { 2024-03-28T01:08:51.379 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "cgroups_split", 2024-03-28T01:08:51.379 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.379 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.379 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.379 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.379 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.379 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.379 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.379 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T01:08:51.380 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.380 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.380 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.380 INFO:teuthology.orchestra.run.smithi169.stdout: "config_checks_enabled": { 2024-03-28T01:08:51.380 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "config_checks_enabled", 2024-03-28T01:08:51.380 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.380 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.380 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.380 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.380 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.380 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.380 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.380 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T01:08:51.381 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.381 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.381 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.381 INFO:teuthology.orchestra.run.smithi169.stdout: "config_dashboard": { 2024-03-28T01:08:51.381 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "config_dashboard", 2024-03-28T01:08:51.381 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.381 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.381 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.381 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.381 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.381 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.381 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.381 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T01:08:51.381 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.382 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.382 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.382 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_alertmanager": { 2024-03-28T01:08:51.382 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_alertmanager", 2024-03-28T01:08:51.382 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.382 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.382 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.382 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T01:08:51.382 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.382 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.382 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.382 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Prometheus container image", 2024-03-28T01:08:51.382 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.382 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.383 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.383 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_base": { 2024-03-28T01:08:51.383 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_base", 2024-03-28T01:08:51.383 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.383 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.383 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.383 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T01:08:51.383 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.383 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.383 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.383 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Container image name, without the tag", 2024-03-28T01:08:51.383 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.383 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.383 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.384 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_elasticsearch": { 2024-03-28T01:08:51.384 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_elasticsearch", 2024-03-28T01:08:51.384 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.384 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.384 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.384 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T01:08:51.384 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.384 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.384 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.384 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "elasticsearch container image", 2024-03-28T01:08:51.384 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.384 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.384 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.385 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_grafana": { 2024-03-28T01:08:51.385 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_grafana", 2024-03-28T01:08:51.385 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.385 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.385 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.385 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T01:08:51.385 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.385 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.385 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.385 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Prometheus container image", 2024-03-28T01:08:51.385 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.385 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.385 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.385 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_haproxy": { 2024-03-28T01:08:51.386 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_haproxy", 2024-03-28T01:08:51.386 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.386 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.386 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.386 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T01:08:51.386 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.386 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.386 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.386 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "HAproxy container image", 2024-03-28T01:08:51.386 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.386 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.386 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.386 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_jaeger_agent": { 2024-03-28T01:08:51.386 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_jaeger_agent", 2024-03-28T01:08:51.387 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.387 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.387 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.387 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T01:08:51.387 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.387 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.387 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.387 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Jaeger agent container image", 2024-03-28T01:08:51.387 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.387 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.387 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.387 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_jaeger_collector": { 2024-03-28T01:08:51.387 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_jaeger_collector", 2024-03-28T01:08:51.388 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.388 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.388 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.388 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T01:08:51.388 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.388 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.388 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.388 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Jaeger collector container image", 2024-03-28T01:08:51.388 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.388 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.388 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.388 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_jaeger_query": { 2024-03-28T01:08:51.388 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_jaeger_query", 2024-03-28T01:08:51.389 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.389 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.389 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.389 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T01:08:51.389 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.389 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.389 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.389 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Jaeger query container image", 2024-03-28T01:08:51.389 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.389 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.389 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.389 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_keepalived": { 2024-03-28T01:08:51.389 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_keepalived", 2024-03-28T01:08:51.389 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.390 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.390 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.390 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T01:08:51.390 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.390 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.390 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.390 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Keepalived container image", 2024-03-28T01:08:51.390 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.390 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.390 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.390 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_loki": { 2024-03-28T01:08:51.390 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_loki", 2024-03-28T01:08:51.390 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.391 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.393 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.393 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T01:08:51.393 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.393 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.393 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.393 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Loki container image", 2024-03-28T01:08:51.393 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.394 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.394 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.394 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_node_exporter": { 2024-03-28T01:08:51.394 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_node_exporter", 2024-03-28T01:08:51.394 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.394 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.394 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.394 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T01:08:51.394 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.394 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.394 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.394 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Prometheus container image", 2024-03-28T01:08:51.394 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.394 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.395 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.395 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_nvmeof": { 2024-03-28T01:08:51.395 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_nvmeof", 2024-03-28T01:08:51.395 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.395 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.395 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.395 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-28T01:08:51.395 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.395 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.395 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.395 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Nvme-of container image", 2024-03-28T01:08:51.395 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.395 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.395 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.396 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_prometheus": { 2024-03-28T01:08:51.396 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_prometheus", 2024-03-28T01:08:51.396 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.396 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.396 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.396 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T01:08:51.396 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.396 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.396 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.396 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Prometheus container image", 2024-03-28T01:08:51.396 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.396 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.396 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.397 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_promtail": { 2024-03-28T01:08:51.397 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_promtail", 2024-03-28T01:08:51.397 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.397 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.397 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.397 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T01:08:51.397 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.397 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.397 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.397 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Promtail container image", 2024-03-28T01:08:51.397 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.397 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.397 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.398 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_snmp_gateway": { 2024-03-28T01:08:51.398 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_snmp_gateway", 2024-03-28T01:08:51.398 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.398 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.398 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.398 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T01:08:51.398 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.398 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.398 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.398 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "SNMP Gateway container image", 2024-03-28T01:08:51.398 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.398 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.398 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.398 INFO:teuthology.orchestra.run.smithi169.stdout: "container_init": { 2024-03-28T01:08:51.399 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_init", 2024-03-28T01:08:51.399 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.399 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.399 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.399 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.399 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.399 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.399 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.399 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T01:08:51.399 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.399 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.399 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.399 INFO:teuthology.orchestra.run.smithi169.stdout: "daemon_cache_timeout": { 2024-03-28T01:08:51.399 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "daemon_cache_timeout", 2024-03-28T01:08:51.400 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.400 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.400 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.400 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "600", 2024-03-28T01:08:51.400 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.400 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.400 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.400 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T01:08:51.400 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.400 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.400 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.400 INFO:teuthology.orchestra.run.smithi169.stdout: "default_cephadm_command_timeout": { 2024-03-28T01:08:51.400 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T01:08:51.400 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.400 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.401 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.401 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "900", 2024-03-28T01:08:51.401 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.401 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.401 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.401 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T01:08:51.401 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.401 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.401 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.401 INFO:teuthology.orchestra.run.smithi169.stdout: "default_registry": { 2024-03-28T01:08:51.401 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "default_registry", 2024-03-28T01:08:51.401 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.402 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.402 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.402 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "docker.io", 2024-03-28T01:08:51.402 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.402 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.402 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.402 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T01:08:51.402 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.402 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.402 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.402 INFO:teuthology.orchestra.run.smithi169.stdout: "device_cache_timeout": { 2024-03-28T01:08:51.402 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "device_cache_timeout", 2024-03-28T01:08:51.402 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.402 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.403 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.403 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "1800", 2024-03-28T01:08:51.403 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.403 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.403 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.403 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "seconds to cache device inventory", 2024-03-28T01:08:51.403 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.403 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.403 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.403 INFO:teuthology.orchestra.run.smithi169.stdout: "device_enhanced_scan": { 2024-03-28T01:08:51.403 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "device_enhanced_scan", 2024-03-28T01:08:51.403 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.403 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.403 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.404 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.404 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.404 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.404 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.404 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T01:08:51.404 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.404 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.404 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.404 INFO:teuthology.orchestra.run.smithi169.stdout: "facts_cache_timeout": { 2024-03-28T01:08:51.404 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "facts_cache_timeout", 2024-03-28T01:08:51.404 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.404 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.404 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.404 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "60", 2024-03-28T01:08:51.405 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.405 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.405 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.405 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "seconds to cache host facts data", 2024-03-28T01:08:51.405 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.405 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.405 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.405 INFO:teuthology.orchestra.run.smithi169.stdout: "host_check_interval": { 2024-03-28T01:08:51.405 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "host_check_interval", 2024-03-28T01:08:51.405 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.405 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.405 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.405 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "600", 2024-03-28T01:08:51.405 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.406 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.406 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.406 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how frequently to perform a host check", 2024-03-28T01:08:51.406 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.406 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.406 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.406 INFO:teuthology.orchestra.run.smithi169.stdout: "hw_monitoring": { 2024-03-28T01:08:51.406 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "hw_monitoring", 2024-03-28T01:08:51.406 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.406 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.406 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.406 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.406 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.407 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.407 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.407 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T01:08:51.407 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.407 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.407 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.407 INFO:teuthology.orchestra.run.smithi169.stdout: "inventory_list_all": { 2024-03-28T01:08:51.407 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "inventory_list_all", 2024-03-28T01:08:51.407 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.407 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.407 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.407 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.407 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.407 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.408 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.408 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T01:08:51.408 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.408 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.408 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.408 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.408 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.408 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.408 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.408 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.408 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.408 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.408 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.408 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.409 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.409 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.409 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.409 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.409 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.409 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.409 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.409 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.409 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.409 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.409 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.409 INFO:teuthology.orchestra.run.smithi169.stdout: "log_refresh_metadata": { 2024-03-28T01:08:51.409 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_refresh_metadata", 2024-03-28T01:08:51.409 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.410 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.410 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.410 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.410 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.410 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.410 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.410 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T01:08:51.410 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.410 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.410 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.410 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.410 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.410 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.411 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.411 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.411 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.411 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.411 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.411 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.411 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T01:08:51.411 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.411 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.411 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.411 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.411 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.411 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.411 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.411 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.412 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.412 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.412 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.412 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.412 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.412 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.412 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.412 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.412 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.412 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.412 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.412 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.412 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.412 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.412 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.413 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.413 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.413 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.413 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.413 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.413 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.413 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.413 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.413 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.413 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.413 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.413 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.413 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.414 INFO:teuthology.orchestra.run.smithi169.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T01:08:51.414 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T01:08:51.414 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.414 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.414 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.414 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.414 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.414 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.414 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.414 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T01:08:51.414 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.414 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.414 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.415 INFO:teuthology.orchestra.run.smithi169.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T01:08:51.415 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T01:08:51.415 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.415 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.415 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.415 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "*", 2024-03-28T01:08:51.415 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.415 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.415 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.415 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T01:08:51.415 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.415 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.415 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.416 INFO:teuthology.orchestra.run.smithi169.stdout: "max_count_per_host": { 2024-03-28T01:08:51.416 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "max_count_per_host", 2024-03-28T01:08:51.416 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.416 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.416 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.416 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.416 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.416 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.416 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.416 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "max number of daemons per service per host", 2024-03-28T01:08:51.416 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.416 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.416 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.416 INFO:teuthology.orchestra.run.smithi169.stdout: "max_osd_draining_count": { 2024-03-28T01:08:51.417 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "max_osd_draining_count", 2024-03-28T01:08:51.417 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.417 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.417 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.417 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.417 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.417 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.417 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.417 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T01:08:51.417 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.417 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.417 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.417 INFO:teuthology.orchestra.run.smithi169.stdout: "migration_current": { 2024-03-28T01:08:51.417 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "migration_current", 2024-03-28T01:08:51.418 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.418 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.418 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.418 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.418 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.418 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.418 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.418 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "internal - do not modify", 2024-03-28T01:08:51.418 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.418 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.418 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.418 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": { 2024-03-28T01:08:51.418 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "mode", 2024-03-28T01:08:51.419 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.419 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.419 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.419 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "root", 2024-03-28T01:08:51.419 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.419 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.419 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.419 INFO:teuthology.orchestra.run.smithi169.stdout: "cephadm-package", 2024-03-28T01:08:51.419 INFO:teuthology.orchestra.run.smithi169.stdout: "root" 2024-03-28T01:08:51.419 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.419 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T01:08:51.419 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.419 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.419 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.420 INFO:teuthology.orchestra.run.smithi169.stdout: "oob_default_addr": { 2024-03-28T01:08:51.420 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "oob_default_addr", 2024-03-28T01:08:51.420 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.420 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.420 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.420 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "169.254.1.1", 2024-03-28T01:08:51.420 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.420 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.420 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.420 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T01:08:51.420 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.420 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.420 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.421 INFO:teuthology.orchestra.run.smithi169.stdout: "prometheus_alerts_path": { 2024-03-28T01:08:51.421 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "prometheus_alerts_path", 2024-03-28T01:08:51.421 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.421 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.421 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.421 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T01:08:51.421 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.421 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.421 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.421 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T01:08:51.421 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.421 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.421 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.421 INFO:teuthology.orchestra.run.smithi169.stdout: "registry_insecure": { 2024-03-28T01:08:51.421 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "registry_insecure", 2024-03-28T01:08:51.422 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.422 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.422 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.422 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.422 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.422 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.422 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.422 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T01:08:51.422 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.422 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.422 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.422 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.422 INFO:teuthology.orchestra.run.smithi169.stdout: "registry_password": { 2024-03-28T01:08:51.422 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "registry_password", 2024-03-28T01:08:51.423 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.423 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.423 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.423 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.423 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.423 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.423 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.423 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T01:08:51.423 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.423 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.423 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.423 INFO:teuthology.orchestra.run.smithi169.stdout: "registry_url": { 2024-03-28T01:08:51.423 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "registry_url", 2024-03-28T01:08:51.423 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.423 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.424 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.424 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.424 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.424 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.424 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.424 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T01:08:51.424 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.424 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.424 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.424 INFO:teuthology.orchestra.run.smithi169.stdout: "registry_username": { 2024-03-28T01:08:51.424 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "registry_username", 2024-03-28T01:08:51.424 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.424 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.425 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.425 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.425 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.425 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.425 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.425 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T01:08:51.425 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.425 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.425 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.425 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.425 INFO:teuthology.orchestra.run.smithi169.stdout: "secure_monitoring_stack": { 2024-03-28T01:08:51.425 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "secure_monitoring_stack", 2024-03-28T01:08:51.425 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.425 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.425 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.426 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.426 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.426 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.426 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.426 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T01:08:51.426 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.426 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.426 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.426 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.426 INFO:teuthology.orchestra.run.smithi169.stdout: "service_discovery_port": { 2024-03-28T01:08:51.426 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "service_discovery_port", 2024-03-28T01:08:51.426 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.426 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.426 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.426 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "8765", 2024-03-28T01:08:51.427 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.427 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.427 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.427 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "cephadm service discovery port", 2024-03-28T01:08:51.427 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.427 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.427 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.427 INFO:teuthology.orchestra.run.smithi169.stdout: "ssh_config_file": { 2024-03-28T01:08:51.427 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ssh_config_file", 2024-03-28T01:08:51.427 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.427 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.428 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.428 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.428 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.428 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.428 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.428 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T01:08:51.428 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.428 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.428 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.428 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.428 INFO:teuthology.orchestra.run.smithi169.stdout: "use_agent": { 2024-03-28T01:08:51.428 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "use_agent", 2024-03-28T01:08:51.428 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.428 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.429 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.429 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.429 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.429 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.429 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.429 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T01:08:51.429 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.429 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.429 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.429 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.429 INFO:teuthology.orchestra.run.smithi169.stdout: "use_repo_digest": { 2024-03-28T01:08:51.429 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "use_repo_digest", 2024-03-28T01:08:51.429 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.429 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.429 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.430 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.430 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.430 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.430 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.430 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T01:08:51.430 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.430 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.430 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.430 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.430 INFO:teuthology.orchestra.run.smithi169.stdout: "warn_on_failed_host_check": { 2024-03-28T01:08:51.430 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "warn_on_failed_host_check", 2024-03-28T01:08:51.430 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.430 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.430 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.430 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.431 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.431 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.431 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.431 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T01:08:51.431 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.431 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.431 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.431 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.431 INFO:teuthology.orchestra.run.smithi169.stdout: "warn_on_stray_daemons": { 2024-03-28T01:08:51.431 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "warn_on_stray_daemons", 2024-03-28T01:08:51.431 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.431 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.431 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.431 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.431 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.432 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.432 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.432 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T01:08:51.432 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.432 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.432 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.432 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.432 INFO:teuthology.orchestra.run.smithi169.stdout: "warn_on_stray_hosts": { 2024-03-28T01:08:51.432 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "warn_on_stray_hosts", 2024-03-28T01:08:51.432 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.432 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.432 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.432 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.432 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.432 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.432 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.433 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T01:08:51.433 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.433 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.433 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.433 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.433 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.433 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.433 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.433 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "crash", 2024-03-28T01:08:51.433 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.433 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.433 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.433 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.433 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.433 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.434 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.434 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.434 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.434 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.434 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.434 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.434 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.434 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.434 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.434 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.434 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.434 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.434 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.434 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.434 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.435 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.435 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.435 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.435 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.435 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.435 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.435 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.435 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.435 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.435 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.435 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.435 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.435 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.435 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.435 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.436 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.436 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.436 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.436 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.436 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.436 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.436 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.436 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.436 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.436 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.436 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.436 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.436 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.437 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.437 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.437 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.437 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.437 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.437 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.437 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.437 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.437 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.437 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.437 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.437 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.437 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.437 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.438 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.438 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.438 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.438 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.438 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.438 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.438 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.438 INFO:teuthology.orchestra.run.smithi169.stdout: "retain_interval": { 2024-03-28T01:08:51.438 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "retain_interval", 2024-03-28T01:08:51.438 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.438 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.438 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.438 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "31536000", 2024-03-28T01:08:51.438 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.438 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.439 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.439 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T01:08:51.439 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.439 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.439 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.439 INFO:teuthology.orchestra.run.smithi169.stdout: "warn_recent_interval": { 2024-03-28T01:08:51.439 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "warn_recent_interval", 2024-03-28T01:08:51.439 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.439 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.439 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.439 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "1209600", 2024-03-28T01:08:51.439 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.439 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.439 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.440 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T01:08:51.440 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.440 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.440 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.440 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.440 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.440 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "dashboard", 2024-03-28T01:08:51.440 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.440 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.440 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.440 INFO:teuthology.orchestra.run.smithi169.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T01:08:51.440 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T01:08:51.440 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.441 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.441 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.441 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.441 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.441 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.441 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.441 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.441 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.441 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.441 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.441 INFO:teuthology.orchestra.run.smithi169.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T01:08:51.441 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T01:08:51.441 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.442 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.442 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.442 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.442 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.442 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.442 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.442 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.442 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.442 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.442 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.442 INFO:teuthology.orchestra.run.smithi169.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T01:08:51.442 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T01:08:51.442 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.442 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.443 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.443 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.443 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.443 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.443 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.443 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.443 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.443 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.443 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.443 INFO:teuthology.orchestra.run.smithi169.stdout: "AUDIT_API_ENABLED": { 2024-03-28T01:08:51.443 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T01:08:51.443 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.443 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.443 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.444 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.444 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.444 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.444 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.444 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.444 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.444 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.444 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.444 INFO:teuthology.orchestra.run.smithi169.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T01:08:51.444 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T01:08:51.444 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.444 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.444 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.445 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.445 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.445 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.445 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.445 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.445 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.445 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.445 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.445 INFO:teuthology.orchestra.run.smithi169.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T01:08:51.445 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T01:08:51.445 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.445 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.445 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.445 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.446 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.446 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.446 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.446 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.446 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.446 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.446 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.446 INFO:teuthology.orchestra.run.smithi169.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T01:08:51.446 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T01:08:51.446 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.446 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.446 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.446 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.447 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.447 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.447 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.447 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.447 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.447 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.447 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.447 INFO:teuthology.orchestra.run.smithi169.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T01:08:51.447 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T01:08:51.447 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.447 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.447 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.447 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.448 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.448 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.448 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.448 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.448 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.448 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.448 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.448 INFO:teuthology.orchestra.run.smithi169.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T01:08:51.448 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T01:08:51.448 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.448 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.448 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.448 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.448 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.449 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.449 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.449 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.449 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.449 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.449 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.449 INFO:teuthology.orchestra.run.smithi169.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T01:08:51.449 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T01:08:51.449 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.449 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.449 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.449 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.449 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.450 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.450 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.450 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.450 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.450 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.450 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.450 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.450 INFO:teuthology.orchestra.run.smithi169.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T01:08:51.450 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T01:08:51.450 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.450 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.450 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.450 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.450 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.451 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.451 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.451 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.451 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.451 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.451 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.451 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.451 INFO:teuthology.orchestra.run.smithi169.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T01:08:51.451 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T01:08:51.451 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.451 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.451 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.451 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.451 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.451 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.452 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.452 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.452 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.452 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.452 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.452 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.452 INFO:teuthology.orchestra.run.smithi169.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T01:08:51.452 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T01:08:51.452 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.452 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.452 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.452 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.452 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.452 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.453 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.453 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.453 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.453 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.453 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.453 INFO:teuthology.orchestra.run.smithi169.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T01:08:51.453 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T01:08:51.453 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.453 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.453 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.453 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.453 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.453 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.453 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.454 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.454 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.454 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.454 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.454 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.454 INFO:teuthology.orchestra.run.smithi169.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T01:08:51.454 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T01:08:51.454 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.454 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.454 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.454 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "admin", 2024-03-28T01:08:51.454 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.454 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.454 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.455 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.455 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.455 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.455 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.455 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.455 INFO:teuthology.orchestra.run.smithi169.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T01:08:51.455 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T01:08:51.455 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.455 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.455 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.455 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.455 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.455 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.455 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.456 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.456 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.456 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.456 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.456 INFO:teuthology.orchestra.run.smithi169.stdout: "GRAFANA_API_URL": { 2024-03-28T01:08:51.456 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "GRAFANA_API_URL", 2024-03-28T01:08:51.456 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.456 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.456 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.456 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.456 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.456 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.456 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.456 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.456 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.457 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.457 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.457 INFO:teuthology.orchestra.run.smithi169.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T01:08:51.457 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T01:08:51.457 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.457 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.457 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.457 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "admin", 2024-03-28T01:08:51.457 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.457 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.457 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.457 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.457 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.457 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.458 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.458 INFO:teuthology.orchestra.run.smithi169.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T01:08:51.458 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T01:08:51.458 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.458 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.458 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.458 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.458 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.458 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.458 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.458 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.458 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.458 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.459 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.459 INFO:teuthology.orchestra.run.smithi169.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T01:08:51.459 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T01:08:51.459 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.459 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.459 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.459 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.459 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.459 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.459 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.459 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.459 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.459 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.459 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.460 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.460 INFO:teuthology.orchestra.run.smithi169.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T01:08:51.460 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T01:08:51.460 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.460 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.460 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.460 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.460 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.460 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.460 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.460 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.460 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.460 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.460 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.460 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.460 INFO:teuthology.orchestra.run.smithi169.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T01:08:51.461 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T01:08:51.461 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.461 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.461 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.461 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.461 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.461 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.461 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.461 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.461 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.461 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.461 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.461 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.461 INFO:teuthology.orchestra.run.smithi169.stdout: "MULTICLUSTER_CONFIG": { 2024-03-28T01:08:51.461 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-28T01:08:51.462 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.462 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.462 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.462 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "{}", 2024-03-28T01:08:51.462 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.462 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.462 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.462 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.462 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.462 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.462 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.462 INFO:teuthology.orchestra.run.smithi169.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T01:08:51.463 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T01:08:51.463 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.463 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.463 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.463 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.463 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.463 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.463 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.463 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.463 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.463 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.463 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.463 INFO:teuthology.orchestra.run.smithi169.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T01:08:51.463 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T01:08:51.463 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.464 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.464 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.464 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.464 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.464 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.464 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.464 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.464 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.464 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.464 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.464 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.464 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T01:08:51.464 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T01:08:51.464 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.464 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.465 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.465 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.465 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.465 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.465 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.465 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.465 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.465 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.465 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.465 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.465 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T01:08:51.465 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T01:08:51.465 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.465 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.465 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.465 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.465 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.466 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.466 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.466 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.466 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.466 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.466 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.466 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T01:08:51.466 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T01:08:51.466 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.466 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.466 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.466 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.466 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.466 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.467 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.467 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.467 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.467 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.467 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.467 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T01:08:51.467 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T01:08:51.467 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.467 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.467 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.467 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.467 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.467 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.467 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.468 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.468 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.468 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.468 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.468 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.468 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T01:08:51.468 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T01:08:51.468 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.468 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.468 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.468 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.468 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.468 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.468 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.468 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.469 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.469 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.469 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.469 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.469 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T01:08:51.469 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T01:08:51.469 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.469 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.469 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.469 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.469 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.469 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.469 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.469 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.469 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T01:08:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T01:08:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.471 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.471 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.471 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T01:08:51.471 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T01:08:51.471 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.471 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.471 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.471 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.471 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.471 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.471 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.471 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.471 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.471 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.471 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.471 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.472 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T01:08:51.472 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T01:08:51.472 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.472 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.472 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.472 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T01:08:51.472 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.472 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.472 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.472 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.472 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.472 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.472 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.472 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.472 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T01:08:51.472 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T01:08:51.473 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.473 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.473 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.473 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.473 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.473 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.473 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.473 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.473 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.473 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.473 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.473 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.473 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T01:08:51.473 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T01:08:51.473 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.474 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.474 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.474 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "8", 2024-03-28T01:08:51.474 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.474 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.474 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.474 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.474 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.474 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.474 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.474 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.474 INFO:teuthology.orchestra.run.smithi169.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T01:08:51.474 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T01:08:51.474 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.475 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.475 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.475 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "45", 2024-03-28T01:08:51.475 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.475 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.475 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.475 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.475 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.475 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.475 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.475 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.475 INFO:teuthology.orchestra.run.smithi169.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T01:08:51.475 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T01:08:51.475 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.475 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.476 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.476 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.476 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.476 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.476 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.476 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.476 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.476 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.476 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.476 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.476 INFO:teuthology.orchestra.run.smithi169.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T01:08:51.476 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T01:08:51.476 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.476 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.476 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.477 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "admin", 2024-03-28T01:08:51.477 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.477 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.477 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.477 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.477 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.477 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.477 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.477 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.477 INFO:teuthology.orchestra.run.smithi169.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T01:08:51.477 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T01:08:51.477 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.477 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.477 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.478 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.478 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.478 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.478 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.478 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.478 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.478 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.478 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.478 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.478 INFO:teuthology.orchestra.run.smithi169.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T01:08:51.478 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T01:08:51.478 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.478 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.478 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.478 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.479 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.479 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.479 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.479 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.479 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.479 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.479 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.479 INFO:teuthology.orchestra.run.smithi169.stdout: "UNSAFE_TLS_v1_2": { 2024-03-28T01:08:51.479 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-28T01:08:51.479 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.479 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.479 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.479 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.479 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.480 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.480 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.480 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.480 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.480 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.480 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.480 INFO:teuthology.orchestra.run.smithi169.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T01:08:51.480 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T01:08:51.480 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.480 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.480 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.480 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0", 2024-03-28T01:08:51.480 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.480 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.480 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.481 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.481 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.481 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.481 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.481 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.481 INFO:teuthology.orchestra.run.smithi169.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T01:08:51.481 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T01:08:51.481 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.481 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.481 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.481 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.481 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.481 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.481 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.481 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.481 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.482 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.482 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.482 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.482 INFO:teuthology.orchestra.run.smithi169.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T01:08:51.482 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T01:08:51.482 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.482 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.482 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.482 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "5", 2024-03-28T01:08:51.482 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.482 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.482 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.482 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.482 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.483 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.483 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.483 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.483 INFO:teuthology.orchestra.run.smithi169.stdout: "cross_origin_url": { 2024-03-28T01:08:51.483 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "cross_origin_url", 2024-03-28T01:08:51.483 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.483 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.483 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.483 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.483 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.483 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.483 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.483 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.483 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.483 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.483 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.484 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.484 INFO:teuthology.orchestra.run.smithi169.stdout: "crt_file": { 2024-03-28T01:08:51.484 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "crt_file", 2024-03-28T01:08:51.484 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.484 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.484 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.484 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.484 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.484 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.484 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.484 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.484 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.484 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.484 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.484 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.484 INFO:teuthology.orchestra.run.smithi169.stdout: "debug": { 2024-03-28T01:08:51.485 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "debug", 2024-03-28T01:08:51.485 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.485 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.485 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.485 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.485 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.485 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.485 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.485 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Enable/disable debug options", 2024-03-28T01:08:51.485 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.485 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.485 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.485 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.485 INFO:teuthology.orchestra.run.smithi169.stdout: "jwt_token_ttl": { 2024-03-28T01:08:51.485 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "jwt_token_ttl", 2024-03-28T01:08:51.485 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.486 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.486 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.486 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "28800", 2024-03-28T01:08:51.486 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.486 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.486 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.486 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.486 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.486 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.486 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.486 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.486 INFO:teuthology.orchestra.run.smithi169.stdout: "key_file": { 2024-03-28T01:08:51.486 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "key_file", 2024-03-28T01:08:51.486 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.487 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.487 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.487 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.487 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.487 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.487 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.487 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.487 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.487 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.487 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.487 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.487 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.487 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.487 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.487 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.488 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.488 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.488 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.488 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.488 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.488 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.488 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.488 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.488 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.488 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.488 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.488 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.488 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.488 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.488 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.489 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.489 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.489 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.489 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.489 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.489 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.489 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.489 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.489 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.489 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.489 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.489 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.489 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.489 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.489 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.489 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.490 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.490 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.490 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.490 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.490 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.490 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.490 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.490 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.490 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.490 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.490 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.490 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.490 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.490 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.491 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.491 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.491 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.491 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.491 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.491 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.491 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.491 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.491 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.491 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.491 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.491 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.491 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.491 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.491 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.491 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.492 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.492 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.492 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.492 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.492 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.492 INFO:teuthology.orchestra.run.smithi169.stdout: "motd": { 2024-03-28T01:08:51.492 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "motd", 2024-03-28T01:08:51.492 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.492 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.492 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.492 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.492 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.492 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.492 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.492 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "The message of the day", 2024-03-28T01:08:51.493 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.493 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.493 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.493 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.493 INFO:teuthology.orchestra.run.smithi169.stdout: "redirect_resolve_ip_addr": { 2024-03-28T01:08:51.493 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T01:08:51.493 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.493 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.493 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.493 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.493 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.493 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.493 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.493 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.494 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.494 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.494 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.494 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.494 INFO:teuthology.orchestra.run.smithi169.stdout: "server_addr": { 2024-03-28T01:08:51.494 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "server_addr", 2024-03-28T01:08:51.494 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.494 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.494 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.494 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "::", 2024-03-28T01:08:51.494 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.494 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.494 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.494 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.494 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.495 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.495 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.495 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.495 INFO:teuthology.orchestra.run.smithi169.stdout: "server_port": { 2024-03-28T01:08:51.495 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "server_port", 2024-03-28T01:08:51.495 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.495 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.495 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.495 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "8080", 2024-03-28T01:08:51.495 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.495 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.495 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.495 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.495 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.495 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.495 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.496 INFO:teuthology.orchestra.run.smithi169.stdout: "ssl": { 2024-03-28T01:08:51.496 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ssl", 2024-03-28T01:08:51.496 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.496 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.496 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.496 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.496 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.496 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.496 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.496 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.496 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.496 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.496 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.496 INFO:teuthology.orchestra.run.smithi169.stdout: "ssl_server_port": { 2024-03-28T01:08:51.497 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ssl_server_port", 2024-03-28T01:08:51.497 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.497 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.497 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.497 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "8443", 2024-03-28T01:08:51.497 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.497 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.497 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.497 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.497 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.497 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.497 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.497 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.497 INFO:teuthology.orchestra.run.smithi169.stdout: "standby_behaviour": { 2024-03-28T01:08:51.497 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "standby_behaviour", 2024-03-28T01:08:51.497 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.498 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.498 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.498 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "redirect", 2024-03-28T01:08:51.498 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.498 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.498 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.498 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.498 INFO:teuthology.orchestra.run.smithi169.stdout: "redirect" 2024-03-28T01:08:51.498 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.498 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.498 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.498 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.498 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.498 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.498 INFO:teuthology.orchestra.run.smithi169.stdout: "standby_error_status_code": { 2024-03-28T01:08:51.499 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "standby_error_status_code", 2024-03-28T01:08:51.499 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.499 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.499 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.499 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "500", 2024-03-28T01:08:51.499 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "400", 2024-03-28T01:08:51.499 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "599", 2024-03-28T01:08:51.499 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.499 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.499 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.499 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.499 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.499 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.499 INFO:teuthology.orchestra.run.smithi169.stdout: "url_prefix": { 2024-03-28T01:08:51.500 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "url_prefix", 2024-03-28T01:08:51.500 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.500 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.500 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.500 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.500 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.500 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.500 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.500 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.500 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.500 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.500 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.500 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.500 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.500 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.500 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.501 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "devicehealth", 2024-03-28T01:08:51.501 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.501 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.501 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.501 INFO:teuthology.orchestra.run.smithi169.stdout: "enable_monitoring": { 2024-03-28T01:08:51.501 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "enable_monitoring", 2024-03-28T01:08:51.501 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.501 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.501 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.501 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.501 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.501 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.501 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.501 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "monitor device health metrics", 2024-03-28T01:08:51.501 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.501 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.501 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.502 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.502 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.502 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.502 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.502 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.502 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.502 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.502 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.502 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.502 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.502 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.502 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.502 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.502 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.503 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.503 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.503 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.503 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.503 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.503 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.503 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.503 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.503 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.503 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.503 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.503 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.503 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.503 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.503 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.503 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.504 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.504 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.504 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.504 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.504 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.504 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.504 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.504 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.504 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.504 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.504 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.504 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.504 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.504 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.504 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.505 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.505 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.505 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.505 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.505 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.505 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.505 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.505 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.505 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.505 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.505 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.505 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.505 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.505 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.505 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.505 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.505 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.506 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.506 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.506 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.506 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.506 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.506 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.506 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.506 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.506 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.506 INFO:teuthology.orchestra.run.smithi169.stdout: "mark_out_threshold": { 2024-03-28T01:08:51.506 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "mark_out_threshold", 2024-03-28T01:08:51.506 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.506 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.506 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.507 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "2419200", 2024-03-28T01:08:51.507 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.507 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.507 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.507 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T01:08:51.507 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.507 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.507 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.507 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.507 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": { 2024-03-28T01:08:51.507 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "pool_name", 2024-03-28T01:08:51.507 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.507 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.507 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.507 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "device_health_metrics", 2024-03-28T01:08:51.507 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.507 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.508 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.508 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T01:08:51.508 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.508 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.508 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.508 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.508 INFO:teuthology.orchestra.run.smithi169.stdout: "retention_period": { 2024-03-28T01:08:51.508 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "retention_period", 2024-03-28T01:08:51.508 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.508 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.508 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.508 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "15552000", 2024-03-28T01:08:51.508 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.508 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.508 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.509 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how long to retain device health metrics", 2024-03-28T01:08:51.509 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.509 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.509 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.509 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.509 INFO:teuthology.orchestra.run.smithi169.stdout: "scrape_frequency": { 2024-03-28T01:08:51.509 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "scrape_frequency", 2024-03-28T01:08:51.509 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.509 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.509 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.509 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "86400", 2024-03-28T01:08:51.509 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.509 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.509 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.509 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T01:08:51.509 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.510 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.510 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.510 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.510 INFO:teuthology.orchestra.run.smithi169.stdout: "self_heal": { 2024-03-28T01:08:51.510 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "self_heal", 2024-03-28T01:08:51.510 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.510 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.510 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.510 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.510 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.510 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.510 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.510 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T01:08:51.510 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.510 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.511 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.511 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.511 INFO:teuthology.orchestra.run.smithi169.stdout: "sleep_interval": { 2024-03-28T01:08:51.511 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "sleep_interval", 2024-03-28T01:08:51.511 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.511 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.511 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.511 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "600", 2024-03-28T01:08:51.511 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.511 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.511 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.511 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T01:08:51.511 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.511 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.511 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.512 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.512 INFO:teuthology.orchestra.run.smithi169.stdout: "warn_threshold": { 2024-03-28T01:08:51.512 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "warn_threshold", 2024-03-28T01:08:51.512 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.512 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.512 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.512 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "7257600", 2024-03-28T01:08:51.512 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.512 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.512 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.512 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T01:08:51.512 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.512 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.512 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.512 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.512 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.513 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.513 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.513 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "diskprediction_local", 2024-03-28T01:08:51.513 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.513 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.513 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.513 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.513 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.513 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.513 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.513 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.513 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.513 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.513 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.513 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.513 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.514 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.514 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.514 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.514 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.514 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.514 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.514 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.514 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.514 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.514 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.514 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.514 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.514 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.514 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.515 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.515 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.515 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.515 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.515 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.515 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.515 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.515 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.515 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.515 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.515 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.515 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.515 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.515 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.515 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.515 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.516 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.516 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.516 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.516 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.516 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.516 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.516 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.516 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.516 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.516 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.516 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.516 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.516 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.516 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.516 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.516 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.517 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.517 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.517 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.517 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.517 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.517 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.517 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.517 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.517 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.517 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.517 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.517 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.517 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.517 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.517 INFO:teuthology.orchestra.run.smithi169.stdout: "predict_interval": { 2024-03-28T01:08:51.517 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "predict_interval", 2024-03-28T01:08:51.518 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.518 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.518 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.518 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "86400", 2024-03-28T01:08:51.518 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.518 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.518 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.518 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.518 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.518 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.518 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.518 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.518 INFO:teuthology.orchestra.run.smithi169.stdout: "predictor_model": { 2024-03-28T01:08:51.518 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "predictor_model", 2024-03-28T01:08:51.518 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.519 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.519 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.519 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "prophetstor", 2024-03-28T01:08:51.519 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.519 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.519 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.519 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.519 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.519 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.519 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.519 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.519 INFO:teuthology.orchestra.run.smithi169.stdout: "sleep_interval": { 2024-03-28T01:08:51.519 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "sleep_interval", 2024-03-28T01:08:51.519 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.519 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.519 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.520 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "600", 2024-03-28T01:08:51.520 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.520 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.520 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.520 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.520 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.520 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.520 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.520 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.520 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.520 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.520 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.520 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "influx", 2024-03-28T01:08:51.520 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": false, 2024-03-28T01:08:51.520 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "influxdb python module not found", 2024-03-28T01:08:51.520 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.521 INFO:teuthology.orchestra.run.smithi169.stdout: "batch_size": { 2024-03-28T01:08:51.521 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "batch_size", 2024-03-28T01:08:51.521 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.521 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.521 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.521 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "5000", 2024-03-28T01:08:51.521 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.521 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.521 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.521 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T01:08:51.521 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.521 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.521 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.521 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.521 INFO:teuthology.orchestra.run.smithi169.stdout: "database": { 2024-03-28T01:08:51.521 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "database", 2024-03-28T01:08:51.522 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.522 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.522 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.522 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "ceph", 2024-03-28T01:08:51.522 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.522 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.522 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.522 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T01:08:51.522 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.522 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.522 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.522 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.522 INFO:teuthology.orchestra.run.smithi169.stdout: "hostname": { 2024-03-28T01:08:51.522 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "hostname", 2024-03-28T01:08:51.522 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.523 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.523 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.523 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.523 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.523 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.523 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.523 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "InfluxDB server hostname", 2024-03-28T01:08:51.523 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.523 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.523 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.523 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.523 INFO:teuthology.orchestra.run.smithi169.stdout: "interval": { 2024-03-28T01:08:51.523 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "interval", 2024-03-28T01:08:51.523 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.523 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.524 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.524 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "30", 2024-03-28T01:08:51.524 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "5", 2024-03-28T01:08:51.524 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.524 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.524 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T01:08:51.524 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.524 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.524 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.524 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.524 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.524 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.524 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.524 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.524 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.524 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.525 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.525 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.525 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.525 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.525 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.525 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.525 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.525 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.525 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.525 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.525 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.525 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.525 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.525 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.525 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.525 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.526 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.526 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.526 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.526 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.526 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.526 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.526 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.526 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.526 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.526 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.526 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.526 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.526 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.526 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.526 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.527 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.527 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.527 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.527 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.527 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.527 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.527 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.527 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.527 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.527 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.527 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.527 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.527 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.527 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.527 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.527 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.528 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.528 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.528 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.528 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.528 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.528 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.528 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.528 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.528 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.528 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.528 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.528 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.528 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.528 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.528 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.529 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.529 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.529 INFO:teuthology.orchestra.run.smithi169.stdout: "password": { 2024-03-28T01:08:51.529 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "password", 2024-03-28T01:08:51.529 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.529 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.529 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.529 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.529 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.529 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.529 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.529 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "password of InfluxDB server user", 2024-03-28T01:08:51.529 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.529 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.529 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.530 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.530 INFO:teuthology.orchestra.run.smithi169.stdout: "port": { 2024-03-28T01:08:51.530 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "port", 2024-03-28T01:08:51.530 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.530 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.530 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.530 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "8086", 2024-03-28T01:08:51.530 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.530 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.530 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.530 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "InfluxDB server port", 2024-03-28T01:08:51.530 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.530 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.530 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.530 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.531 INFO:teuthology.orchestra.run.smithi169.stdout: "ssl": { 2024-03-28T01:08:51.531 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ssl", 2024-03-28T01:08:51.531 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.531 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.531 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.531 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "false", 2024-03-28T01:08:51.531 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.531 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.531 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.531 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T01:08:51.531 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.531 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.531 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.531 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.531 INFO:teuthology.orchestra.run.smithi169.stdout: "threads": { 2024-03-28T01:08:51.531 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "threads", 2024-03-28T01:08:51.532 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.532 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.532 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.532 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "5", 2024-03-28T01:08:51.532 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "1", 2024-03-28T01:08:51.532 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "32", 2024-03-28T01:08:51.532 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.532 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T01:08:51.532 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.532 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.532 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.532 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.532 INFO:teuthology.orchestra.run.smithi169.stdout: "username": { 2024-03-28T01:08:51.532 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "username", 2024-03-28T01:08:51.532 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.533 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.533 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.533 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.533 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.533 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.533 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.533 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "username of InfluxDB server user", 2024-03-28T01:08:51.533 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.533 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.533 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.533 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.533 INFO:teuthology.orchestra.run.smithi169.stdout: "verify_ssl": { 2024-03-28T01:08:51.533 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "verify_ssl", 2024-03-28T01:08:51.533 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.533 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.533 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.534 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "true", 2024-03-28T01:08:51.534 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.534 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.534 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.534 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T01:08:51.534 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.534 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.534 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.534 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.534 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.534 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.534 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.534 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "insights", 2024-03-28T01:08:51.534 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.534 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.535 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.535 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.535 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.535 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.535 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.535 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.535 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.535 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.535 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.535 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.535 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.535 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.535 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.535 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.535 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.536 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.536 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.536 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.536 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.536 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.536 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.536 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.536 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.536 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.536 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.536 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.536 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.536 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.536 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.536 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.537 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.537 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.537 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.537 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.537 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.537 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.537 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.537 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.537 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.537 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.537 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.537 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.537 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.537 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.537 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.538 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.538 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.538 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.538 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.538 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.538 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.538 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.538 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.538 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.538 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.538 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.538 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.538 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.538 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.538 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.539 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.539 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.539 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.539 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.539 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.539 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.539 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.539 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.539 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.539 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.539 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.539 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.539 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.539 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.539 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "iostat", 2024-03-28T01:08:51.539 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.540 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.540 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.540 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.540 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.540 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.540 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.540 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.540 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.540 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.540 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.540 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.540 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.540 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.540 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.540 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.541 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.541 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.541 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.541 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.541 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.541 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.541 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.541 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.541 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.541 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.541 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.541 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.541 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.541 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.541 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.542 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.542 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.542 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.542 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.542 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.542 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.542 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.542 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.542 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.542 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.542 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.542 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.542 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.542 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.542 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.543 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.543 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.543 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.543 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.543 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.543 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.543 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.543 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.543 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.543 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.543 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.543 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.543 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.543 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.543 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.544 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.544 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.544 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.544 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.544 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.544 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.544 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.544 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.544 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.544 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.544 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.544 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.544 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.544 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.544 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.545 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "k8sevents", 2024-03-28T01:08:51.545 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.545 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.545 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.545 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_event_retention_days": { 2024-03-28T01:08:51.545 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ceph_event_retention_days", 2024-03-28T01:08:51.545 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.545 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.545 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.545 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "7", 2024-03-28T01:08:51.545 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.545 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.545 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.545 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T01:08:51.545 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.546 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.546 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.546 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.546 INFO:teuthology.orchestra.run.smithi169.stdout: "config_check_secs": { 2024-03-28T01:08:51.546 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "config_check_secs", 2024-03-28T01:08:51.546 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.546 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.546 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.546 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.546 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "10", 2024-03-28T01:08:51.546 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.546 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.546 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T01:08:51.546 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.546 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.547 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.547 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.547 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.547 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.547 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.547 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.547 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.547 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.547 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.547 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.547 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.547 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.547 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.547 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.547 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.547 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.548 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.548 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.548 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.548 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.548 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.548 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.548 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.548 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.548 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.548 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.548 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.548 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.548 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.548 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.548 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.549 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.550 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.551 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "localpool", 2024-03-28T01:08:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "failure_domain": { 2024-03-28T01:08:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "failure_domain", 2024-03-28T01:08:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "host", 2024-03-28T01:08:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "failure domain for any created local pool", 2024-03-28T01:08:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T01:08:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.552 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.553 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.554 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.555 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.556 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": { 2024-03-28T01:08:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "min_size", 2024-03-28T01:08:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.557 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "default min_size for any created local pool", 2024-03-28T01:08:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T01:08:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "num_rep": { 2024-03-28T01:08:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "num_rep", 2024-03-28T01:08:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.558 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "3", 2024-03-28T01:08:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "default replica count for any created local pool", 2024-03-28T01:08:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": { 2024-03-28T01:08:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "pg_num", 2024-03-28T01:08:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "128", 2024-03-28T01:08:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.559 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T01:08:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "prefix": { 2024-03-28T01:08:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "prefix", 2024-03-28T01:08:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.560 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "name prefix for any created local pool", 2024-03-28T01:08:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "subtree": { 2024-03-28T01:08:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "subtree", 2024-03-28T01:08:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.561 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "rack", 2024-03-28T01:08:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T01:08:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T01:08:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "mds_autoscaler", 2024-03-28T01:08:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.562 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.563 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.564 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.565 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.566 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "mirroring", 2024-03-28T01:08:51.567 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.568 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.570 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.571 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.572 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "nfs", 2024-03-28T01:08:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.573 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.574 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.575 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.576 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.577 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "orchestrator", 2024-03-28T01:08:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "fail_fs": { 2024-03-28T01:08:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "fail_fs", 2024-03-28T01:08:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.578 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T01:08:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.579 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.580 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.582 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.583 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "orchestrator": { 2024-03-28T01:08:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "orchestrator", 2024-03-28T01:08:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.584 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "cephadm", 2024-03-28T01:08:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "rook", 2024-03-28T01:08:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "test_orchestrator" 2024-03-28T01:08:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Orchestrator backend", 2024-03-28T01:08:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "osd_perf_query", 2024-03-28T01:08:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.585 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.586 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.587 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.588 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.589 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.590 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "osd_support", 2024-03-28T01:08:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.591 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.592 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.593 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.594 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.595 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.596 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.596 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.596 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.596 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.596 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.596 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.596 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.596 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "pg_autoscaler", 2024-03-28T01:08:51.596 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.596 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.596 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.596 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.596 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.596 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.597 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.597 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.597 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.597 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.597 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.597 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.597 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.597 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.597 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.597 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.597 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.597 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.597 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.597 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.597 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.598 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.598 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.598 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.598 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.598 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.598 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.598 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.598 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.598 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.598 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.598 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.598 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.598 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.598 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.598 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.599 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.599 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.599 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.599 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.599 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.599 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.599 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.599 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.599 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.599 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.599 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.599 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.599 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.599 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.599 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.600 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.600 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.601 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.601 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.601 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.601 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.601 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.601 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.601 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.601 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.602 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.602 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.602 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.602 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.602 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.602 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.602 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.602 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.602 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.602 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.602 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.602 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.602 INFO:teuthology.orchestra.run.smithi169.stdout: "sleep_interval": { 2024-03-28T01:08:51.603 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "sleep_interval", 2024-03-28T01:08:51.603 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.603 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.603 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.603 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "60", 2024-03-28T01:08:51.603 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.603 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.603 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.603 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.603 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.603 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.603 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.603 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.603 INFO:teuthology.orchestra.run.smithi169.stdout: "threshold": { 2024-03-28T01:08:51.603 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "threshold", 2024-03-28T01:08:51.604 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "float", 2024-03-28T01:08:51.604 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.604 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.604 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "3.0", 2024-03-28T01:08:51.604 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "1.0", 2024-03-28T01:08:51.604 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.604 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.604 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "scaling threshold", 2024-03-28T01:08:51.604 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T01:08:51.604 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.604 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.604 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.604 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.604 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.604 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.605 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "progress", 2024-03-28T01:08:51.605 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.605 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.605 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.605 INFO:teuthology.orchestra.run.smithi169.stdout: "allow_pg_recovery_event": { 2024-03-28T01:08:51.605 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "allow_pg_recovery_event", 2024-03-28T01:08:51.605 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.605 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.605 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.605 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.605 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.605 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.605 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.605 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T01:08:51.605 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.606 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.606 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.606 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.606 INFO:teuthology.orchestra.run.smithi169.stdout: "enabled": { 2024-03-28T01:08:51.606 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "enabled", 2024-03-28T01:08:51.606 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.606 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.606 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.606 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.606 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.606 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.606 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.606 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.606 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.606 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.607 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.607 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.607 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.607 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.607 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.607 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.607 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.607 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.607 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.607 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.607 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.607 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.607 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.607 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.607 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.608 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.608 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.608 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.608 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.608 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.608 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.608 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.608 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.608 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.608 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.608 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.608 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.608 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.608 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.608 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.609 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.609 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.609 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.609 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.609 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.609 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.609 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.609 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.609 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.609 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.609 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.609 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.609 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.609 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.609 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.610 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.610 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.610 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.610 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.610 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.610 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.610 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.610 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.610 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.610 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.610 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.610 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.610 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.610 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.610 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.611 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.611 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.611 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.611 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.611 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.611 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.611 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.611 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.611 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.611 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.611 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.611 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.611 INFO:teuthology.orchestra.run.smithi169.stdout: "max_completed_events": { 2024-03-28T01:08:51.611 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "max_completed_events", 2024-03-28T01:08:51.611 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.612 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.612 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.612 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "50", 2024-03-28T01:08:51.612 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.612 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.612 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.612 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "number of past completed events to remember", 2024-03-28T01:08:51.612 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.612 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.612 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.612 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.612 INFO:teuthology.orchestra.run.smithi169.stdout: "sleep_interval": { 2024-03-28T01:08:51.612 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "sleep_interval", 2024-03-28T01:08:51.612 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.612 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.612 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.613 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "5", 2024-03-28T01:08:51.613 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.613 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.613 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.613 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how long the module is going to sleep", 2024-03-28T01:08:51.613 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.613 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.613 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.613 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.613 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.613 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.613 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.613 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "prometheus", 2024-03-28T01:08:51.613 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.613 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.614 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.614 INFO:teuthology.orchestra.run.smithi169.stdout: "cache": { 2024-03-28T01:08:51.614 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "cache", 2024-03-28T01:08:51.614 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.614 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.614 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.614 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.614 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.614 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.614 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.614 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.614 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.614 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.615 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.615 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.615 INFO:teuthology.orchestra.run.smithi169.stdout: "exclude_perf_counters": { 2024-03-28T01:08:51.615 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "exclude_perf_counters", 2024-03-28T01:08:51.615 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.615 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.615 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.615 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.615 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.615 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.615 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.615 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T01:08:51.615 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T01:08:51.615 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.616 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.616 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.616 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.616 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.616 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.616 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.616 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.616 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.616 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.616 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.616 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.616 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.616 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.616 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.617 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.617 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.617 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.617 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.617 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.617 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.617 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.617 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.617 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.617 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.617 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.617 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.617 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.617 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.618 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.618 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.618 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.618 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.618 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.618 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.618 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.618 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.618 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.618 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.618 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.618 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.618 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.618 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.618 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.619 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.619 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.619 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.619 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.619 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.619 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.619 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.619 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.619 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.619 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.619 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.619 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.619 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.619 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.620 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.620 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.620 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.620 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.620 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.620 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.620 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.620 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.620 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.620 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.620 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.620 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.620 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.620 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_stats_pools": { 2024-03-28T01:08:51.620 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rbd_stats_pools", 2024-03-28T01:08:51.620 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.621 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.621 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.621 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.621 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.621 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.621 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.621 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.621 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.621 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.621 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.621 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.621 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T01:08:51.621 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T01:08:51.621 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.621 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.621 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.622 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "300", 2024-03-28T01:08:51.622 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.622 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.622 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.622 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.622 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.622 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.622 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.622 INFO:teuthology.orchestra.run.smithi169.stdout: "scrape_interval": { 2024-03-28T01:08:51.622 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "scrape_interval", 2024-03-28T01:08:51.622 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "float", 2024-03-28T01:08:51.622 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.622 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.622 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "15.0", 2024-03-28T01:08:51.623 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.623 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.623 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.623 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.623 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.623 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.623 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.623 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.623 INFO:teuthology.orchestra.run.smithi169.stdout: "server_addr": { 2024-03-28T01:08:51.623 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "server_addr", 2024-03-28T01:08:51.623 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.623 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.623 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.623 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "::", 2024-03-28T01:08:51.623 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.624 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.624 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.624 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T01:08:51.624 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.624 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.624 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.624 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.624 INFO:teuthology.orchestra.run.smithi169.stdout: "server_port": { 2024-03-28T01:08:51.624 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "server_port", 2024-03-28T01:08:51.624 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.624 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.624 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.624 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "9283", 2024-03-28T01:08:51.624 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.624 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.625 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.625 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T01:08:51.625 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.625 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.625 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.625 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.625 INFO:teuthology.orchestra.run.smithi169.stdout: "stale_cache_strategy": { 2024-03-28T01:08:51.625 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "stale_cache_strategy", 2024-03-28T01:08:51.625 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.625 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.625 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.625 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "log", 2024-03-28T01:08:51.625 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.625 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.625 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.626 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.626 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.626 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.626 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.626 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.626 INFO:teuthology.orchestra.run.smithi169.stdout: "standby_behaviour": { 2024-03-28T01:08:51.626 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "standby_behaviour", 2024-03-28T01:08:51.626 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.626 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.626 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.626 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "default", 2024-03-28T01:08:51.626 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.626 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.626 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.626 INFO:teuthology.orchestra.run.smithi169.stdout: "default", 2024-03-28T01:08:51.627 INFO:teuthology.orchestra.run.smithi169.stdout: "error" 2024-03-28T01:08:51.627 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.627 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.627 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.627 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.627 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.627 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.627 INFO:teuthology.orchestra.run.smithi169.stdout: "standby_error_status_code": { 2024-03-28T01:08:51.627 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "standby_error_status_code", 2024-03-28T01:08:51.627 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.627 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.627 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.627 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "500", 2024-03-28T01:08:51.627 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "400", 2024-03-28T01:08:51.627 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "599", 2024-03-28T01:08:51.627 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.628 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.628 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.628 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.628 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.628 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.628 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.628 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.628 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.628 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rbd_support", 2024-03-28T01:08:51.628 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.628 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.628 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.628 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.628 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.629 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.629 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.629 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.629 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.629 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.629 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.629 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.629 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.629 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.629 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.629 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.629 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.629 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.629 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.629 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.629 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.630 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.630 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.630 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.630 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.630 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.630 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.630 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.630 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.630 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.630 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.630 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.630 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.630 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.630 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.630 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.631 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.631 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.631 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.631 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.631 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.631 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.631 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.631 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.631 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.631 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.631 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.631 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.631 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.631 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.631 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.632 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.632 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.632 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.632 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.632 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.632 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.632 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.632 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.632 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.632 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.632 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.632 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.632 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.632 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.632 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.633 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.633 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.633 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.633 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.633 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.633 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.633 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.633 INFO:teuthology.orchestra.run.smithi169.stdout: "max_concurrent_snap_create": { 2024-03-28T01:08:51.633 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "max_concurrent_snap_create", 2024-03-28T01:08:51.633 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.633 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.633 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.633 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.633 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.633 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.634 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.634 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.634 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.634 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.634 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.634 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.634 INFO:teuthology.orchestra.run.smithi169.stdout: "mirror_snapshot_schedule": { 2024-03-28T01:08:51.634 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T01:08:51.634 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.634 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.634 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.634 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.634 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.634 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.634 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.634 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.635 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.635 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.635 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.635 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.635 INFO:teuthology.orchestra.run.smithi169.stdout: "trash_purge_schedule": { 2024-03-28T01:08:51.635 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "trash_purge_schedule", 2024-03-28T01:08:51.635 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.635 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.635 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.635 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.635 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.635 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.635 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.635 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.636 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.636 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.636 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.636 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.636 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.636 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "restful", 2024-03-28T01:08:51.636 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.636 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.636 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.636 INFO:teuthology.orchestra.run.smithi169.stdout: "enable_auth": { 2024-03-28T01:08:51.636 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "enable_auth", 2024-03-28T01:08:51.636 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.636 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.636 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.637 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.637 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.637 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.637 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.637 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.637 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.637 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.637 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.637 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.637 INFO:teuthology.orchestra.run.smithi169.stdout: "key_file": { 2024-03-28T01:08:51.637 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "key_file", 2024-03-28T01:08:51.637 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.637 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.637 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.637 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.638 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.638 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.638 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.638 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.638 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.638 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.638 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.638 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.638 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.638 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.638 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.638 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.638 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.638 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.638 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.639 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.639 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.639 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.639 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.639 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.639 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.639 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.639 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.639 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.639 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.639 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.639 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.639 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.640 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.640 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.640 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.640 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.640 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.640 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.640 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.640 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.640 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.640 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.640 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.640 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.640 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.640 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.640 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.641 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.641 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.641 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.641 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.641 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.641 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.641 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.641 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.641 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.641 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.641 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.641 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.641 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.641 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.642 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.642 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.642 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.642 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.642 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.642 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.642 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.642 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.642 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.642 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.642 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.642 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.642 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.642 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.643 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.643 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.643 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.643 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.643 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.643 INFO:teuthology.orchestra.run.smithi169.stdout: "server_addr": { 2024-03-28T01:08:51.643 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "server_addr", 2024-03-28T01:08:51.643 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.643 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.643 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.643 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.643 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.643 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.643 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.643 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.644 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.644 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.644 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.644 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.644 INFO:teuthology.orchestra.run.smithi169.stdout: "server_port": { 2024-03-28T01:08:51.644 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "server_port", 2024-03-28T01:08:51.644 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.644 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.644 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.644 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.644 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.644 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.644 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.644 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.644 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.645 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.645 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.645 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.645 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.645 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.645 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rgw", 2024-03-28T01:08:51.645 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.645 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.645 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.645 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.645 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.645 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.645 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.646 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.646 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.646 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.646 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.646 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.646 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.646 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.646 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.646 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.646 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.646 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.646 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.646 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.646 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.646 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.647 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.647 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.647 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.647 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.647 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.647 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.647 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.647 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.647 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.647 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.647 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.647 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.647 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.647 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.647 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.648 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.648 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.648 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.648 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.648 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.648 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.648 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.648 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.648 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.648 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.648 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.648 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.648 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.648 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.648 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.649 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.649 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.649 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.649 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.649 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.649 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.649 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.649 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.649 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.649 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.649 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.649 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.649 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.649 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.649 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.649 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.650 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.650 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.650 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.650 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.650 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.650 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.650 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.650 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.650 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rook", 2024-03-28T01:08:51.650 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.650 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.650 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.650 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.650 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.650 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.651 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.651 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.651 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.651 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.651 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.651 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.651 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.651 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.651 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.651 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.651 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.651 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.651 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.651 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.651 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.652 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.652 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.652 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.652 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.652 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.652 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.652 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.652 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.652 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.652 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.652 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.652 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.652 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.652 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.652 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.653 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.653 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.653 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.653 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.653 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.653 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.653 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.653 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.653 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.653 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.653 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.653 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.653 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.653 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.654 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.654 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.654 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.654 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.654 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.654 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.654 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.654 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.654 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.654 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.654 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.654 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.654 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.654 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.654 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.655 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.655 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.655 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.655 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.655 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.655 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.655 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.655 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.655 INFO:teuthology.orchestra.run.smithi169.stdout: "storage_class": { 2024-03-28T01:08:51.655 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "storage_class", 2024-03-28T01:08:51.655 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.655 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.655 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.655 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "local", 2024-03-28T01:08:51.655 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.656 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.656 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.656 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T01:08:51.656 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.656 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.656 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.656 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.656 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.656 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.656 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "selftest", 2024-03-28T01:08:51.656 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.656 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.656 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.656 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.657 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.657 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.657 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.657 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.657 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.657 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.657 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.657 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.657 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.657 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.657 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.657 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.657 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.657 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.657 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.658 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.658 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.658 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.658 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.658 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.658 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.658 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.658 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.658 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.658 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.658 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.658 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.658 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.658 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.659 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.659 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.659 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.659 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.659 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.659 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.659 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.659 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.659 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.659 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.659 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.659 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.659 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.659 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.660 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.660 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.660 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.660 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.660 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.660 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.660 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.660 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.660 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.660 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.660 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.660 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.660 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.660 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.661 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.661 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.661 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.661 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.661 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.661 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.661 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.661 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.661 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.661 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.661 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.661 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.661 INFO:teuthology.orchestra.run.smithi169.stdout: "roption1": { 2024-03-28T01:08:51.661 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "roption1", 2024-03-28T01:08:51.661 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.661 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.662 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.662 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.662 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.662 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.662 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.662 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.662 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.662 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.662 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.662 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.662 INFO:teuthology.orchestra.run.smithi169.stdout: "roption2": { 2024-03-28T01:08:51.662 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "roption2", 2024-03-28T01:08:51.662 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.662 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.663 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.663 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "xyz", 2024-03-28T01:08:51.663 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.663 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.663 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.663 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.663 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.663 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.663 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.663 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.663 INFO:teuthology.orchestra.run.smithi169.stdout: "rwoption1": { 2024-03-28T01:08:51.663 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rwoption1", 2024-03-28T01:08:51.663 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.663 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.663 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.664 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.664 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.664 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.664 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.664 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.664 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.664 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.664 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.664 INFO:teuthology.orchestra.run.smithi169.stdout: "rwoption2": { 2024-03-28T01:08:51.664 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rwoption2", 2024-03-28T01:08:51.664 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.664 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.664 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.664 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.664 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.665 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.665 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.665 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.665 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.665 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.665 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.665 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.665 INFO:teuthology.orchestra.run.smithi169.stdout: "rwoption3": { 2024-03-28T01:08:51.665 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rwoption3", 2024-03-28T01:08:51.665 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "float", 2024-03-28T01:08:51.665 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.665 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.665 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.665 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.665 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.666 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.666 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.666 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.666 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.666 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.666 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.666 INFO:teuthology.orchestra.run.smithi169.stdout: "rwoption4": { 2024-03-28T01:08:51.666 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rwoption4", 2024-03-28T01:08:51.666 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.666 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.666 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.666 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.666 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.667 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.667 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.667 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.667 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.667 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.667 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.667 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.667 INFO:teuthology.orchestra.run.smithi169.stdout: "rwoption5": { 2024-03-28T01:08:51.667 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rwoption5", 2024-03-28T01:08:51.667 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.667 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.667 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.667 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.667 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.667 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.668 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.668 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.668 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.668 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.668 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.668 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.668 INFO:teuthology.orchestra.run.smithi169.stdout: "rwoption6": { 2024-03-28T01:08:51.668 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rwoption6", 2024-03-28T01:08:51.668 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.668 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.668 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.668 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.668 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.668 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.668 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.669 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.669 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.669 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.669 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.669 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.669 INFO:teuthology.orchestra.run.smithi169.stdout: "rwoption7": { 2024-03-28T01:08:51.669 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rwoption7", 2024-03-28T01:08:51.669 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.669 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.669 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.669 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.669 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "1", 2024-03-28T01:08:51.669 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "42", 2024-03-28T01:08:51.669 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.669 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.670 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.670 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.670 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.670 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.670 INFO:teuthology.orchestra.run.smithi169.stdout: "testkey": { 2024-03-28T01:08:51.670 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "testkey", 2024-03-28T01:08:51.670 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.670 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.670 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.670 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.670 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.670 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.670 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.670 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.671 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.671 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.671 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.671 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.671 INFO:teuthology.orchestra.run.smithi169.stdout: "testlkey": { 2024-03-28T01:08:51.671 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "testlkey", 2024-03-28T01:08:51.671 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.671 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.671 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.671 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.671 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.671 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.671 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.671 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.671 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.672 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.672 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.672 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.672 INFO:teuthology.orchestra.run.smithi169.stdout: "testnewline": { 2024-03-28T01:08:51.672 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "testnewline", 2024-03-28T01:08:51.672 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.672 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.672 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.672 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.672 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.672 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.672 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.672 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.672 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.672 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.673 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.673 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.673 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.673 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.673 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.673 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "snap_schedule", 2024-03-28T01:08:51.673 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.673 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.673 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.673 INFO:teuthology.orchestra.run.smithi169.stdout: "allow_m_granularity": { 2024-03-28T01:08:51.673 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "allow_m_granularity", 2024-03-28T01:08:51.673 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.673 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.673 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.673 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.673 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.674 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.674 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.674 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T01:08:51.674 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.674 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.674 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.674 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.674 INFO:teuthology.orchestra.run.smithi169.stdout: "dump_on_update": { 2024-03-28T01:08:51.674 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "dump_on_update", 2024-03-28T01:08:51.674 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.674 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.674 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.674 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.674 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.675 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.675 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.675 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "dump database to debug log on update", 2024-03-28T01:08:51.675 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.675 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.675 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.675 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.675 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.675 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.675 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.675 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.675 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.675 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.676 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.676 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.676 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.676 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.676 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.676 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.676 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.676 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.676 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.676 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.676 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.676 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.676 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.676 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.676 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.676 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.677 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.677 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.677 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.677 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.677 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.677 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.677 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.677 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.677 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.677 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.677 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.677 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.677 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.677 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.677 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.678 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.678 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.678 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.678 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.678 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.678 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.678 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.678 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.678 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.678 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.678 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.678 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.678 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.678 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.679 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.679 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.679 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.679 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.679 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.679 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.679 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.679 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.679 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.679 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.679 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.679 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.679 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.679 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.679 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.680 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.680 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.680 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.680 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.680 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.680 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.680 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.680 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "stats", 2024-03-28T01:08:51.680 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.680 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.680 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.680 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.680 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.680 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.680 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.681 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.681 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.681 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.681 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.681 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.681 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.681 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.681 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.681 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.681 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.681 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.681 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.681 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.681 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.681 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.682 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.682 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.682 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.682 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.682 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.682 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.682 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.682 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.682 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.682 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.682 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.682 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.682 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.682 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.682 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.683 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.683 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.683 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.683 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.683 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.683 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.683 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.683 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.683 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.683 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.683 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.683 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.683 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.683 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.683 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.684 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.684 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.684 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.684 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.684 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.684 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.684 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.684 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.684 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.684 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.684 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.684 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.684 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.684 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.684 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.685 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.685 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.685 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.685 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.685 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.685 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.685 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.685 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.685 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.685 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "status", 2024-03-28T01:08:51.685 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.685 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.685 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.685 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.685 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.686 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.686 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.686 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.686 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.686 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.686 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.686 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.686 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.686 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.686 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.686 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.686 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.686 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.686 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.686 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.687 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.687 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.687 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.687 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.687 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.687 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.687 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.687 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.687 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.687 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.687 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.687 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.687 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.687 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.687 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.688 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.688 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.688 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.688 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.688 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.688 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.688 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.688 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.688 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.688 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.688 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.688 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.688 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.688 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.688 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.689 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.689 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.689 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.689 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.689 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.689 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.689 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.689 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.689 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.689 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.689 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.689 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.689 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.689 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.689 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.689 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.690 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.690 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.690 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.690 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.690 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.690 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.690 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.690 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.690 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.690 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.690 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "telegraf", 2024-03-28T01:08:51.690 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.690 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.690 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.691 INFO:teuthology.orchestra.run.smithi169.stdout: "address": { 2024-03-28T01:08:51.691 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "address", 2024-03-28T01:08:51.691 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.691 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.691 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.691 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T01:08:51.691 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.691 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.691 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.691 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.691 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.691 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.691 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.691 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.691 INFO:teuthology.orchestra.run.smithi169.stdout: "interval": { 2024-03-28T01:08:51.691 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "interval", 2024-03-28T01:08:51.692 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.692 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.692 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.692 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "15", 2024-03-28T01:08:51.692 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.692 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.692 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.692 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.692 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.692 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.692 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.692 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.692 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.692 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.693 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.693 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.693 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.693 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.693 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.693 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.693 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.693 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.693 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.693 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.693 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.693 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.693 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.693 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.693 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.694 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.694 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.694 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.694 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.694 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.694 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.694 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.694 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.694 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.694 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.694 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.694 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.694 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.694 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.694 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.695 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.695 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.695 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.695 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.695 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.695 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.695 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.695 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.695 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.695 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.695 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.695 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.695 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.695 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.695 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.696 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.696 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.696 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.696 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.696 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.696 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.696 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.696 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.696 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.696 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.696 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.696 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.696 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.696 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.697 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.697 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.697 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.697 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.697 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.697 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.697 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.697 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.697 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.697 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.697 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.697 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.697 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "telemetry", 2024-03-28T01:08:51.697 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.697 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "channel_basic": { 2024-03-28T01:08:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "channel_basic", 2024-03-28T01:08:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T01:08:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.698 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "channel_crash": { 2024-03-28T01:08:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "channel_crash", 2024-03-28T01:08:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T01:08:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.699 INFO:teuthology.orchestra.run.smithi169.stdout: "channel_device": { 2024-03-28T01:08:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "channel_device", 2024-03-28T01:08:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T01:08:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "channel_ident": { 2024-03-28T01:08:51.700 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "channel_ident", 2024-03-28T01:08:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T01:08:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: "channel_perf": { 2024-03-28T01:08:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "channel_perf", 2024-03-28T01:08:51.701 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T01:08:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "contact": { 2024-03-28T01:08:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "contact", 2024-03-28T01:08:51.702 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "description": { 2024-03-28T01:08:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "description", 2024-03-28T01:08:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.703 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: "device_url": { 2024-03-28T01:08:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "device_url", 2024-03-28T01:08:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.704 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T01:08:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "enabled": { 2024-03-28T01:08:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "enabled", 2024-03-28T01:08:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.705 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "interval": { 2024-03-28T01:08:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "interval", 2024-03-28T01:08:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.706 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "24", 2024-03-28T01:08:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "8", 2024-03-28T01:08:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "last_opt_revision": { 2024-03-28T01:08:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "last_opt_revision", 2024-03-28T01:08:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "1", 2024-03-28T01:08:51.707 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "leaderboard": { 2024-03-28T01:08:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "leaderboard", 2024-03-28T01:08:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.708 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "leaderboard_description": { 2024-03-28T01:08:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "leaderboard_description", 2024-03-28T01:08:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.709 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.710 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.711 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.712 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.713 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.714 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "organization": { 2024-03-28T01:08:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "organization", 2024-03-28T01:08:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.715 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "proxy": { 2024-03-28T01:08:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "proxy", 2024-03-28T01:08:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.716 INFO:teuthology.orchestra.run.smithi169.stdout: "url": { 2024-03-28T01:08:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "url", 2024-03-28T01:08:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T01:08:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.717 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "test_orchestrator", 2024-03-28T01:08:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.718 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.719 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.720 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.721 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.722 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "volumes", 2024-03-28T01:08:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.723 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.724 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.725 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.727 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "max_concurrent_clones": { 2024-03-28T01:08:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "max_concurrent_clones", 2024-03-28T01:08:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "4", 2024-03-28T01:08:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T01:08:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "periodic_async_work": { 2024-03-28T01:08:51.728 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "periodic_async_work", 2024-03-28T01:08:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Periodically check for async work", 2024-03-28T01:08:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "snapshot_clone_delay": { 2024-03-28T01:08:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "snapshot_clone_delay", 2024-03-28T01:08:51.729 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0", 2024-03-28T01:08:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T01:08:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: "snapshot_clone_no_wait": { 2024-03-28T01:08:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T01:08:51.730 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T01:08:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "zabbix", 2024-03-28T01:08:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "discovery_interval": { 2024-03-28T01:08:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "discovery_interval", 2024-03-28T01:08:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "uint", 2024-03-28T01:08:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "100", 2024-03-28T01:08:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.732 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "identifier": { 2024-03-28T01:08:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "identifier", 2024-03-28T01:08:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.733 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "interval": { 2024-03-28T01:08:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "interval", 2024-03-28T01:08:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "60", 2024-03-28T01:08:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.734 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.735 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.736 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.737 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.738 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.738 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.738 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.738 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.738 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.738 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.738 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.738 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.738 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.738 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.738 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.738 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.738 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.738 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "zabbix_host": { 2024-03-28T01:08:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "zabbix_host", 2024-03-28T01:08:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.739 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "zabbix_port": { 2024-03-28T01:08:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "zabbix_port", 2024-03-28T01:08:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10051", 2024-03-28T01:08:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "zabbix_sender": { 2024-03-28T01:08:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "zabbix_sender", 2024-03-28T01:08:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T01:08:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.741 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-28T01:08:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: "modules": [ 2024-03-28T01:08:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: "cephadm", 2024-03-28T01:08:51.742 INFO:teuthology.orchestra.run.smithi169.stdout: "dashboard", 2024-03-28T01:08:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "iostat", 2024-03-28T01:08:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "nfs", 2024-03-28T01:08:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "restful" 2024-03-28T01:08:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "available_modules": [ 2024-03-28T01:08:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "alerts", 2024-03-28T01:08:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "interval": { 2024-03-28T01:08:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "interval", 2024-03-28T01:08:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.743 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "60", 2024-03-28T01:08:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T01:08:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.744 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.745 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.746 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.747 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.748 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_destination": { 2024-03-28T01:08:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_destination", 2024-03-28T01:08:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Email address to send alerts to", 2024-03-28T01:08:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.749 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_from_name": { 2024-03-28T01:08:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_from_name", 2024-03-28T01:08:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "Ceph", 2024-03-28T01:08:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Email From: name", 2024-03-28T01:08:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.750 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_host": { 2024-03-28T01:08:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_host", 2024-03-28T01:08:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "SMTP server", 2024-03-28T01:08:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.751 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_password": { 2024-03-28T01:08:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_password", 2024-03-28T01:08:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Password to authenticate with", 2024-03-28T01:08:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.752 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_port": { 2024-03-28T01:08:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_port", 2024-03-28T01:08:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "465", 2024-03-28T01:08:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "SMTP port", 2024-03-28T01:08:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.753 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_sender": { 2024-03-28T01:08:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_sender", 2024-03-28T01:08:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "SMTP envelope sender", 2024-03-28T01:08:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_ssl": { 2024-03-28T01:08:51.754 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_ssl", 2024-03-28T01:08:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T01:08:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "smtp_user": { 2024-03-28T01:08:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "smtp_user", 2024-03-28T01:08:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.755 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "User to authenticate as", 2024-03-28T01:08:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "balancer", 2024-03-28T01:08:51.756 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "active": { 2024-03-28T01:08:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "active", 2024-03-28T01:08:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T01:08:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.757 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: "begin_time": { 2024-03-28T01:08:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "begin_time", 2024-03-28T01:08:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0000", 2024-03-28T01:08:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T01:08:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T01:08:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.758 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "begin_weekday": { 2024-03-28T01:08:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "begin_weekday", 2024-03-28T01:08:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "uint", 2024-03-28T01:08:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0", 2024-03-28T01:08:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "0", 2024-03-28T01:08:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "6", 2024-03-28T01:08:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T01:08:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T01:08:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.759 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_compat_max_iterations": { 2024-03-28T01:08:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "crush_compat_max_iterations", 2024-03-28T01:08:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "uint", 2024-03-28T01:08:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "25", 2024-03-28T01:08:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "1", 2024-03-28T01:08:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "250", 2024-03-28T01:08:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T01:08:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.760 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_compat_metrics": { 2024-03-28T01:08:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "crush_compat_metrics", 2024-03-28T01:08:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T01:08:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T01:08:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T01:08:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.761 INFO:teuthology.orchestra.run.smithi169.stdout: "crush_compat_step": { 2024-03-28T01:08:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "crush_compat_step", 2024-03-28T01:08:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "float", 2024-03-28T01:08:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0.5", 2024-03-28T01:08:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "0.001", 2024-03-28T01:08:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "0.999", 2024-03-28T01:08:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "aggressiveness of optimization", 2024-03-28T01:08:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T01:08:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "end_time": { 2024-03-28T01:08:51.762 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "end_time", 2024-03-28T01:08:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "2359", 2024-03-28T01:08:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T01:08:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T01:08:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "end_weekday": { 2024-03-28T01:08:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "end_weekday", 2024-03-28T01:08:51.763 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "uint", 2024-03-28T01:08:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0", 2024-03-28T01:08:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "0", 2024-03-28T01:08:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "6", 2024-03-28T01:08:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T01:08:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T01:08:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.764 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.765 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.766 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.767 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.768 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "min_score": { 2024-03-28T01:08:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "min_score", 2024-03-28T01:08:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "float", 2024-03-28T01:08:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0", 2024-03-28T01:08:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.769 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T01:08:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": { 2024-03-28T01:08:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "mode", 2024-03-28T01:08:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "upmap", 2024-03-28T01:08:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.770 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: "crush-compat", 2024-03-28T01:08:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: "none", 2024-03-28T01:08:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: "read", 2024-03-28T01:08:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: "upmap", 2024-03-28T01:08:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: "upmap-read" 2024-03-28T01:08:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Balancer mode", 2024-03-28T01:08:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_ids": { 2024-03-28T01:08:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "pool_ids", 2024-03-28T01:08:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.771 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T01:08:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "sleep_interval": { 2024-03-28T01:08:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "sleep_interval", 2024-03-28T01:08:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.772 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "60", 2024-03-28T01:08:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T01:08:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "upmap_max_deviation": { 2024-03-28T01:08:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "upmap_max_deviation", 2024-03-28T01:08:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "5", 2024-03-28T01:08:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "1", 2024-03-28T01:08:51.773 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T01:08:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T01:08:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: "upmap_max_optimizations": { 2024-03-28T01:08:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "upmap_max_optimizations", 2024-03-28T01:08:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "uint", 2024-03-28T01:08:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.774 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T01:08:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "cephadm", 2024-03-28T01:08:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: "agent_down_multiplier": { 2024-03-28T01:08:51.775 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "agent_down_multiplier", 2024-03-28T01:08:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "float", 2024-03-28T01:08:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "3.0", 2024-03-28T01:08:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T01:08:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "agent_refresh_rate": { 2024-03-28T01:08:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "agent_refresh_rate", 2024-03-28T01:08:51.776 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "20", 2024-03-28T01:08:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T01:08:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "agent_starting_port": { 2024-03-28T01:08:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "agent_starting_port", 2024-03-28T01:08:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.777 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.778 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.778 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "4721", 2024-03-28T01:08:51.778 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.778 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.778 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.778 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T01:08:51.778 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.778 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.778 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.778 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.778 INFO:teuthology.orchestra.run.smithi169.stdout: "allow_ptrace": { 2024-03-28T01:08:51.778 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "allow_ptrace", 2024-03-28T01:08:51.778 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.779 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.779 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.779 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.779 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.779 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.779 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.779 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T01:08:51.779 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T01:08:51.779 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.779 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.779 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.779 INFO:teuthology.orchestra.run.smithi169.stdout: "autotune_interval": { 2024-03-28T01:08:51.779 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "autotune_interval", 2024-03-28T01:08:51.779 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "600", 2024-03-28T01:08:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T01:08:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "autotune_memory_target_ratio": { 2024-03-28T01:08:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T01:08:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "float", 2024-03-28T01:08:51.780 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0.7", 2024-03-28T01:08:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T01:08:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "cephadm_log_destination": { 2024-03-28T01:08:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "cephadm_log_destination", 2024-03-28T01:08:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.781 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: "file", 2024-03-28T01:08:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: "file,syslog", 2024-03-28T01:08:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: "syslog" 2024-03-28T01:08:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T01:08:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: "cgroups_split": { 2024-03-28T01:08:51.782 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "cgroups_split", 2024-03-28T01:08:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T01:08:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "config_checks_enabled": { 2024-03-28T01:08:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "config_checks_enabled", 2024-03-28T01:08:51.783 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T01:08:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "config_dashboard": { 2024-03-28T01:08:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "config_dashboard", 2024-03-28T01:08:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.784 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T01:08:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_alertmanager": { 2024-03-28T01:08:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_alertmanager", 2024-03-28T01:08:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T01:08:51.785 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Prometheus container image", 2024-03-28T01:08:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_base": { 2024-03-28T01:08:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_base", 2024-03-28T01:08:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T01:08:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.786 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Container image name, without the tag", 2024-03-28T01:08:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_elasticsearch": { 2024-03-28T01:08:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_elasticsearch", 2024-03-28T01:08:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T01:08:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "elasticsearch container image", 2024-03-28T01:08:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_grafana": { 2024-03-28T01:08:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_grafana", 2024-03-28T01:08:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T01:08:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Prometheus container image", 2024-03-28T01:08:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_haproxy": { 2024-03-28T01:08:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_haproxy", 2024-03-28T01:08:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T01:08:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "HAproxy container image", 2024-03-28T01:08:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_jaeger_agent": { 2024-03-28T01:08:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_jaeger_agent", 2024-03-28T01:08:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T01:08:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Jaeger agent container image", 2024-03-28T01:08:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_jaeger_collector": { 2024-03-28T01:08:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_jaeger_collector", 2024-03-28T01:08:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T01:08:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Jaeger collector container image", 2024-03-28T01:08:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_jaeger_query": { 2024-03-28T01:08:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_jaeger_query", 2024-03-28T01:08:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T01:08:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Jaeger query container image", 2024-03-28T01:08:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_keepalived": { 2024-03-28T01:08:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_keepalived", 2024-03-28T01:08:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T01:08:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Keepalived container image", 2024-03-28T01:08:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_loki": { 2024-03-28T01:08:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_loki", 2024-03-28T01:08:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T01:08:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Loki container image", 2024-03-28T01:08:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_node_exporter": { 2024-03-28T01:08:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_node_exporter", 2024-03-28T01:08:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T01:08:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Prometheus container image", 2024-03-28T01:08:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_nvmeof": { 2024-03-28T01:08:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_nvmeof", 2024-03-28T01:08:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-28T01:08:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Nvme-of container image", 2024-03-28T01:08:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_prometheus": { 2024-03-28T01:08:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_prometheus", 2024-03-28T01:08:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T01:08:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Prometheus container image", 2024-03-28T01:08:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_promtail": { 2024-03-28T01:08:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_promtail", 2024-03-28T01:08:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T01:08:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Promtail container image", 2024-03-28T01:08:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "container_image_snmp_gateway": { 2024-03-28T01:08:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_image_snmp_gateway", 2024-03-28T01:08:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T01:08:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "SNMP Gateway container image", 2024-03-28T01:08:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "container_init": { 2024-03-28T01:08:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "container_init", 2024-03-28T01:08:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T01:08:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "daemon_cache_timeout": { 2024-03-28T01:08:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "daemon_cache_timeout", 2024-03-28T01:08:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "600", 2024-03-28T01:08:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T01:08:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "default_cephadm_command_timeout": { 2024-03-28T01:08:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T01:08:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "900", 2024-03-28T01:08:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T01:08:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "default_registry": { 2024-03-28T01:08:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "default_registry", 2024-03-28T01:08:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "docker.io", 2024-03-28T01:08:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T01:08:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "device_cache_timeout": { 2024-03-28T01:08:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "device_cache_timeout", 2024-03-28T01:08:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "1800", 2024-03-28T01:08:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "seconds to cache device inventory", 2024-03-28T01:08:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "device_enhanced_scan": { 2024-03-28T01:08:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "device_enhanced_scan", 2024-03-28T01:08:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T01:08:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "facts_cache_timeout": { 2024-03-28T01:08:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "facts_cache_timeout", 2024-03-28T01:08:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "60", 2024-03-28T01:08:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "seconds to cache host facts data", 2024-03-28T01:08:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "host_check_interval": { 2024-03-28T01:08:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "host_check_interval", 2024-03-28T01:08:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "600", 2024-03-28T01:08:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how frequently to perform a host check", 2024-03-28T01:08:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "hw_monitoring": { 2024-03-28T01:08:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "hw_monitoring", 2024-03-28T01:08:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T01:08:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "inventory_list_all": { 2024-03-28T01:08:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "inventory_list_all", 2024-03-28T01:08:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T01:08:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "log_refresh_metadata": { 2024-03-28T01:08:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_refresh_metadata", 2024-03-28T01:08:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T01:08:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T01:08:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T01:08:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T01:08:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T01:08:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T01:08:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T01:08:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "*", 2024-03-28T01:08:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T01:08:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "max_count_per_host": { 2024-03-28T01:08:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "max_count_per_host", 2024-03-28T01:08:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "max number of daemons per service per host", 2024-03-28T01:08:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "max_osd_draining_count": { 2024-03-28T01:08:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "max_osd_draining_count", 2024-03-28T01:08:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T01:08:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "migration_current": { 2024-03-28T01:08:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "migration_current", 2024-03-28T01:08:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "internal - do not modify", 2024-03-28T01:08:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "mode": { 2024-03-28T01:08:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "mode", 2024-03-28T01:08:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "root", 2024-03-28T01:08:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "cephadm-package", 2024-03-28T01:08:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "root" 2024-03-28T01:08:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T01:08:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "oob_default_addr": { 2024-03-28T01:08:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "oob_default_addr", 2024-03-28T01:08:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "169.254.1.1", 2024-03-28T01:08:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T01:08:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "prometheus_alerts_path": { 2024-03-28T01:08:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "prometheus_alerts_path", 2024-03-28T01:08:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T01:08:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T01:08:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "registry_insecure": { 2024-03-28T01:08:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "registry_insecure", 2024-03-28T01:08:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T01:08:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "registry_password": { 2024-03-28T01:08:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "registry_password", 2024-03-28T01:08:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T01:08:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "registry_url": { 2024-03-28T01:08:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "registry_url", 2024-03-28T01:08:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T01:08:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "registry_username": { 2024-03-28T01:08:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "registry_username", 2024-03-28T01:08:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T01:08:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "secure_monitoring_stack": { 2024-03-28T01:08:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "secure_monitoring_stack", 2024-03-28T01:08:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T01:08:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "service_discovery_port": { 2024-03-28T01:08:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "service_discovery_port", 2024-03-28T01:08:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "8765", 2024-03-28T01:08:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "cephadm service discovery port", 2024-03-28T01:08:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: "ssh_config_file": { 2024-03-28T01:08:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ssh_config_file", 2024-03-28T01:08:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T01:08:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "use_agent": { 2024-03-28T01:08:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "use_agent", 2024-03-28T01:08:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T01:08:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: "use_repo_digest": { 2024-03-28T01:08:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "use_repo_digest", 2024-03-28T01:08:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T01:08:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "warn_on_failed_host_check": { 2024-03-28T01:08:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "warn_on_failed_host_check", 2024-03-28T01:08:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T01:08:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: "warn_on_stray_daemons": { 2024-03-28T01:08:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "warn_on_stray_daemons", 2024-03-28T01:08:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T01:08:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: "warn_on_stray_hosts": { 2024-03-28T01:08:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "warn_on_stray_hosts", 2024-03-28T01:08:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T01:08:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "crash", 2024-03-28T01:08:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: "retain_interval": { 2024-03-28T01:08:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "retain_interval", 2024-03-28T01:08:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "31536000", 2024-03-28T01:08:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T01:08:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "warn_recent_interval": { 2024-03-28T01:08:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "warn_recent_interval", 2024-03-28T01:08:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "1209600", 2024-03-28T01:08:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T01:08:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "dashboard", 2024-03-28T01:08:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T01:08:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T01:08:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T01:08:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T01:08:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T01:08:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T01:08:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "AUDIT_API_ENABLED": { 2024-03-28T01:08:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T01:08:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T01:08:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T01:08:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T01:08:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T01:08:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T01:08:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T01:08:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T01:08:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T01:08:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T01:08:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T01:08:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T01:08:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T01:08:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T01:08:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T01:08:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T01:08:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T01:08:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T01:08:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T01:08:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T01:08:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T01:08:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T01:08:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T01:08:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "admin", 2024-03-28T01:08:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T01:08:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T01:08:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "GRAFANA_API_URL": { 2024-03-28T01:08:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "GRAFANA_API_URL", 2024-03-28T01:08:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T01:08:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T01:08:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "admin", 2024-03-28T01:08:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T01:08:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T01:08:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T01:08:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T01:08:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T01:08:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T01:08:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T01:08:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T01:08:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: "MULTICLUSTER_CONFIG": { 2024-03-28T01:08:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-28T01:08:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "{}", 2024-03-28T01:08:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T01:08:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T01:08:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T01:08:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T01:08:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T01:08:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T01:08:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T01:08:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T01:08:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T01:08:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T01:08:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T01:08:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T01:08:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T01:08:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T01:08:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T01:08:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T01:08:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T01:08:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T01:08:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T01:08:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T01:08:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T01:08:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T01:08:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T01:08:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T01:08:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T01:08:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T01:08:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T01:08:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "8", 2024-03-28T01:08:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T01:08:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T01:08:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "45", 2024-03-28T01:08:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T01:08:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T01:08:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T01:08:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T01:08:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "admin", 2024-03-28T01:08:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T01:08:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T01:08:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T01:08:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T01:08:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "UNSAFE_TLS_v1_2": { 2024-03-28T01:08:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-28T01:08:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T01:08:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T01:08:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0", 2024-03-28T01:08:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T01:08:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T01:08:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T01:08:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T01:08:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "5", 2024-03-28T01:08:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: "cross_origin_url": { 2024-03-28T01:08:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "cross_origin_url", 2024-03-28T01:08:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.886 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.886 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.886 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.886 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.886 INFO:teuthology.orchestra.run.smithi169.stdout: "crt_file": { 2024-03-28T01:08:51.886 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "crt_file", 2024-03-28T01:08:51.886 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.886 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.886 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.886 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.886 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.886 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.886 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.886 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.887 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.887 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.887 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.887 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.887 INFO:teuthology.orchestra.run.smithi169.stdout: "debug": { 2024-03-28T01:08:51.887 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "debug", 2024-03-28T01:08:51.887 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.887 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.887 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.887 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.887 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.887 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.887 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.887 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Enable/disable debug options", 2024-03-28T01:08:51.887 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.887 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.888 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.888 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.888 INFO:teuthology.orchestra.run.smithi169.stdout: "jwt_token_ttl": { 2024-03-28T01:08:51.888 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "jwt_token_ttl", 2024-03-28T01:08:51.888 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.888 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.888 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.888 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "28800", 2024-03-28T01:08:51.888 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.888 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.888 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.888 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.888 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.888 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.888 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.889 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.889 INFO:teuthology.orchestra.run.smithi169.stdout: "key_file": { 2024-03-28T01:08:51.889 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "key_file", 2024-03-28T01:08:51.889 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.889 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.889 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.889 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.889 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.889 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.889 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.889 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.889 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.889 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.889 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.889 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.889 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.890 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.890 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.890 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.890 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.890 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.890 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.890 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.890 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.890 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.890 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.890 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.890 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.890 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.890 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.890 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.891 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.891 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.891 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.891 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.891 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.891 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.891 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.891 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.891 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.891 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.891 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.891 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.891 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.891 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.891 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.892 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.892 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.892 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.892 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.892 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.892 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.892 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.892 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.892 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.892 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.892 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.892 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.892 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.892 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.892 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.892 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.893 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.893 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.893 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.893 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.893 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.893 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.893 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.893 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.893 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.893 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.893 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.893 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.893 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.893 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.893 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.893 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.894 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.894 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.894 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.894 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.894 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.894 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.894 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.894 INFO:teuthology.orchestra.run.smithi169.stdout: "motd": { 2024-03-28T01:08:51.894 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "motd", 2024-03-28T01:08:51.894 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.894 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.894 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.894 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.894 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.895 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.895 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.895 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "The message of the day", 2024-03-28T01:08:51.895 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.895 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.895 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.895 INFO:teuthology.orchestra.run.smithi169.stdout: "redirect_resolve_ip_addr": { 2024-03-28T01:08:51.895 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T01:08:51.895 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.895 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.895 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.895 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.895 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.895 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.895 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.896 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.896 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.896 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.896 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.896 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.896 INFO:teuthology.orchestra.run.smithi169.stdout: "server_addr": { 2024-03-28T01:08:51.896 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "server_addr", 2024-03-28T01:08:51.896 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.896 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.896 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.896 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "::", 2024-03-28T01:08:51.896 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.896 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.896 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.896 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.897 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.897 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.897 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.897 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.897 INFO:teuthology.orchestra.run.smithi169.stdout: "server_port": { 2024-03-28T01:08:51.897 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "server_port", 2024-03-28T01:08:51.897 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.897 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.897 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.897 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "8080", 2024-03-28T01:08:51.897 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.897 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.897 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.897 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.897 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.897 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.898 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.898 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.898 INFO:teuthology.orchestra.run.smithi169.stdout: "ssl": { 2024-03-28T01:08:51.898 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ssl", 2024-03-28T01:08:51.898 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.898 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.898 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.898 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.898 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.898 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.898 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.898 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.898 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.898 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.898 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.899 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.899 INFO:teuthology.orchestra.run.smithi169.stdout: "ssl_server_port": { 2024-03-28T01:08:51.899 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ssl_server_port", 2024-03-28T01:08:51.899 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.899 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.899 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.899 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "8443", 2024-03-28T01:08:51.899 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.899 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.899 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.899 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.899 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.899 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.899 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.899 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.899 INFO:teuthology.orchestra.run.smithi169.stdout: "standby_behaviour": { 2024-03-28T01:08:51.900 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "standby_behaviour", 2024-03-28T01:08:51.900 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.900 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.900 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.900 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "redirect", 2024-03-28T01:08:51.900 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.900 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.900 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.900 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.900 INFO:teuthology.orchestra.run.smithi169.stdout: "redirect" 2024-03-28T01:08:51.900 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.900 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.900 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.900 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.900 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.900 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.901 INFO:teuthology.orchestra.run.smithi169.stdout: "standby_error_status_code": { 2024-03-28T01:08:51.901 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "standby_error_status_code", 2024-03-28T01:08:51.901 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.901 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.901 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.901 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "500", 2024-03-28T01:08:51.901 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "400", 2024-03-28T01:08:51.901 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "599", 2024-03-28T01:08:51.901 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.901 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.901 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.901 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.901 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.901 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.901 INFO:teuthology.orchestra.run.smithi169.stdout: "url_prefix": { 2024-03-28T01:08:51.902 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "url_prefix", 2024-03-28T01:08:51.902 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.902 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.902 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.902 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.902 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.902 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.902 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.902 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.902 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.902 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.902 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.902 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.902 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.902 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.903 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "devicehealth", 2024-03-28T01:08:51.903 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.903 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.903 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.903 INFO:teuthology.orchestra.run.smithi169.stdout: "enable_monitoring": { 2024-03-28T01:08:51.903 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "enable_monitoring", 2024-03-28T01:08:51.903 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.903 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.903 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.903 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.903 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.903 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.903 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.903 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "monitor device health metrics", 2024-03-28T01:08:51.903 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.904 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.904 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.904 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.904 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.904 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.904 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.904 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.904 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.904 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.904 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.904 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.904 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.904 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.904 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.904 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.905 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.905 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.905 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.905 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.905 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.905 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.905 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.905 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.905 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.905 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.905 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.905 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.905 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.905 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.905 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.905 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.906 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.906 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.906 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.906 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.906 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.906 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.906 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.906 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.906 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.906 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.906 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.906 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.906 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.906 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.906 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.907 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.907 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.907 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.907 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.907 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.907 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.907 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.907 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.907 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.907 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.907 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.907 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.907 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.907 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.907 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.908 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.908 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.908 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.908 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.908 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.908 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.908 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.908 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.908 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.908 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.908 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.908 INFO:teuthology.orchestra.run.smithi169.stdout: "mark_out_threshold": { 2024-03-28T01:08:51.908 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "mark_out_threshold", 2024-03-28T01:08:51.908 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.908 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.908 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.909 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "2419200", 2024-03-28T01:08:51.909 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.909 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.909 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.909 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T01:08:51.909 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.909 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.909 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.909 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.909 INFO:teuthology.orchestra.run.smithi169.stdout: "pool_name": { 2024-03-28T01:08:51.909 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "pool_name", 2024-03-28T01:08:51.909 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.909 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.909 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.909 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "device_health_metrics", 2024-03-28T01:08:51.910 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.910 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.910 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.910 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T01:08:51.910 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.910 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.910 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.910 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.910 INFO:teuthology.orchestra.run.smithi169.stdout: "retention_period": { 2024-03-28T01:08:51.910 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "retention_period", 2024-03-28T01:08:51.910 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.910 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.910 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.910 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "15552000", 2024-03-28T01:08:51.911 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.911 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.911 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.911 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how long to retain device health metrics", 2024-03-28T01:08:51.911 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.911 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.911 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.911 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.911 INFO:teuthology.orchestra.run.smithi169.stdout: "scrape_frequency": { 2024-03-28T01:08:51.911 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "scrape_frequency", 2024-03-28T01:08:51.911 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.911 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.911 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.911 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "86400", 2024-03-28T01:08:51.912 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.912 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.912 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.912 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T01:08:51.912 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.912 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.912 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.912 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.912 INFO:teuthology.orchestra.run.smithi169.stdout: "self_heal": { 2024-03-28T01:08:51.912 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "self_heal", 2024-03-28T01:08:51.912 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.912 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.912 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.912 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:51.912 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.913 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.913 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.913 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T01:08:51.913 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.913 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.913 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.913 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.913 INFO:teuthology.orchestra.run.smithi169.stdout: "sleep_interval": { 2024-03-28T01:08:51.913 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "sleep_interval", 2024-03-28T01:08:51.913 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.913 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.913 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.913 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "600", 2024-03-28T01:08:51.913 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.913 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.913 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.914 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T01:08:51.914 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.914 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.914 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.914 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.914 INFO:teuthology.orchestra.run.smithi169.stdout: "warn_threshold": { 2024-03-28T01:08:51.914 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "warn_threshold", 2024-03-28T01:08:51.914 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.914 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.914 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.914 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "7257600", 2024-03-28T01:08:51.914 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.914 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.914 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.914 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T01:08:51.915 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.915 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.915 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.915 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.915 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.915 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.915 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.915 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "diskprediction_local", 2024-03-28T01:08:51.915 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.915 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.915 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.915 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.915 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.915 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.915 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.916 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.916 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.916 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.916 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.916 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.916 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.916 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.916 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.916 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.916 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.916 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.916 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.916 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.916 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.916 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.916 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.917 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.917 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.917 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.917 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.917 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.917 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.917 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.917 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.917 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.917 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.917 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.917 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.917 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.917 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.918 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.918 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.918 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.918 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.918 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.918 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.918 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.918 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.918 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.918 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.918 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.918 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.918 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.918 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.919 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.919 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.919 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.919 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.919 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.919 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.919 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.919 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.919 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.919 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.919 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.919 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.919 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.919 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.919 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.919 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.920 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.920 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.920 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.920 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.920 INFO:teuthology.orchestra.run.smithi169.stdout: "predict_interval": { 2024-03-28T01:08:51.920 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "predict_interval", 2024-03-28T01:08:51.920 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.920 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.920 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.920 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "86400", 2024-03-28T01:08:51.920 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.920 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.920 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.920 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.920 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.920 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.921 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.921 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.921 INFO:teuthology.orchestra.run.smithi169.stdout: "predictor_model": { 2024-03-28T01:08:51.921 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "predictor_model", 2024-03-28T01:08:51.921 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.921 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.921 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.921 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "prophetstor", 2024-03-28T01:08:51.921 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.921 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.921 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.921 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.922 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.922 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.922 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.922 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.922 INFO:teuthology.orchestra.run.smithi169.stdout: "sleep_interval": { 2024-03-28T01:08:51.922 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "sleep_interval", 2024-03-28T01:08:51.922 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.922 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.922 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.922 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "600", 2024-03-28T01:08:51.922 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.922 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.922 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.922 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.923 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.923 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.923 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.923 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.923 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.923 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.923 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.923 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "influx", 2024-03-28T01:08:51.923 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": false, 2024-03-28T01:08:51.923 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "influxdb python module not found", 2024-03-28T01:08:51.923 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.923 INFO:teuthology.orchestra.run.smithi169.stdout: "batch_size": { 2024-03-28T01:08:51.923 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "batch_size", 2024-03-28T01:08:51.923 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.923 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.924 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.924 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "5000", 2024-03-28T01:08:51.924 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.924 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.924 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.924 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T01:08:51.924 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.924 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.924 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.924 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.924 INFO:teuthology.orchestra.run.smithi169.stdout: "database": { 2024-03-28T01:08:51.924 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "database", 2024-03-28T01:08:51.924 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.924 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.924 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.924 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "ceph", 2024-03-28T01:08:51.925 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.925 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.925 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.925 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T01:08:51.925 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.925 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.925 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.925 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.925 INFO:teuthology.orchestra.run.smithi169.stdout: "hostname": { 2024-03-28T01:08:51.925 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "hostname", 2024-03-28T01:08:51.925 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.925 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.925 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.925 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.925 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.925 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.926 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.926 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "InfluxDB server hostname", 2024-03-28T01:08:51.926 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.926 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.926 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.926 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.926 INFO:teuthology.orchestra.run.smithi169.stdout: "interval": { 2024-03-28T01:08:51.926 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "interval", 2024-03-28T01:08:51.926 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:51.926 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.926 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.926 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "30", 2024-03-28T01:08:51.926 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "5", 2024-03-28T01:08:51.926 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.927 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.927 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T01:08:51.927 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.927 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.927 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.927 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.927 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.927 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.927 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.927 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.927 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.927 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.927 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.927 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.927 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.928 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.928 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.928 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.928 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.928 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.928 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.928 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.928 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.928 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.928 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.928 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.928 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.928 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.928 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.928 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.928 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.929 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.929 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.929 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.929 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.929 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.929 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.929 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.929 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.929 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.929 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.929 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.929 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.929 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.929 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.929 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.929 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.930 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.930 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.930 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.930 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.930 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.930 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.930 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.930 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.930 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.930 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.930 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.930 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.930 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.930 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.930 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.931 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.931 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.931 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.931 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.931 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.931 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.931 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.931 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.931 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.931 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.931 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.931 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.931 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.931 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.931 INFO:teuthology.orchestra.run.smithi169.stdout: "password": { 2024-03-28T01:08:51.931 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "password", 2024-03-28T01:08:51.932 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.932 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.932 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.932 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.932 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.932 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.932 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.932 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "password of InfluxDB server user", 2024-03-28T01:08:51.932 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.932 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.932 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.932 INFO:teuthology.orchestra.run.smithi169.stdout: "port": { 2024-03-28T01:08:51.932 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "port", 2024-03-28T01:08:51.932 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.932 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.933 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.933 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "8086", 2024-03-28T01:08:51.933 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.933 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.933 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.933 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "InfluxDB server port", 2024-03-28T01:08:51.933 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.933 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.933 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.933 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.933 INFO:teuthology.orchestra.run.smithi169.stdout: "ssl": { 2024-03-28T01:08:51.933 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ssl", 2024-03-28T01:08:51.933 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.933 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.933 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.934 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "false", 2024-03-28T01:08:51.934 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.934 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.934 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.934 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T01:08:51.934 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.934 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.934 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.934 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.934 INFO:teuthology.orchestra.run.smithi169.stdout: "threads": { 2024-03-28T01:08:51.934 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "threads", 2024-03-28T01:08:51.934 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.934 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.934 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.934 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "5", 2024-03-28T01:08:51.934 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "1", 2024-03-28T01:08:51.935 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "32", 2024-03-28T01:08:51.935 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.935 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T01:08:51.935 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.935 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.935 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.935 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.935 INFO:teuthology.orchestra.run.smithi169.stdout: "username": { 2024-03-28T01:08:51.935 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "username", 2024-03-28T01:08:51.935 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.935 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.935 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.935 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.935 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.935 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.936 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.936 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "username of InfluxDB server user", 2024-03-28T01:08:51.936 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.936 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.936 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.936 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.936 INFO:teuthology.orchestra.run.smithi169.stdout: "verify_ssl": { 2024-03-28T01:08:51.936 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "verify_ssl", 2024-03-28T01:08:51.936 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.936 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.936 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.936 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "true", 2024-03-28T01:08:51.936 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.936 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.937 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.937 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T01:08:51.937 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.937 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.937 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.937 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.937 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.937 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.937 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.937 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "insights", 2024-03-28T01:08:51.937 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.937 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.937 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.937 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.937 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.938 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.938 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.938 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.938 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.938 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.938 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.938 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.938 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.938 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.938 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.938 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.938 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.938 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.938 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.938 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.939 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.939 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.939 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.939 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.939 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.939 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.939 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.939 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.939 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.939 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.939 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.939 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.939 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.939 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.939 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.939 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.940 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.940 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.940 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.940 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.940 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.940 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.940 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.940 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.940 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.940 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.940 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.940 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.940 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.940 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.940 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.941 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.941 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.941 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.941 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.941 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.941 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.941 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.941 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.941 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.941 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.941 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.941 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.941 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.941 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.941 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.942 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.942 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.942 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.942 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.942 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.942 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.942 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.942 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.942 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.942 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.942 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "iostat", 2024-03-28T01:08:51.942 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.942 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.942 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.942 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.943 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.943 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.943 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.943 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.943 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.943 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.943 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.943 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.943 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.943 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.943 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.943 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.943 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.943 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.943 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.944 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.944 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.944 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.944 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.944 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.944 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.944 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.944 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.944 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.944 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.944 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.944 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.944 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.944 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.944 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.944 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.945 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.945 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.945 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.945 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.945 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.945 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.945 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.945 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.945 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.945 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.945 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.945 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.945 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.945 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.945 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.946 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.946 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.946 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.946 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.946 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.946 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.946 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.946 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.946 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.946 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.946 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.946 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.946 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.946 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.946 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.947 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.947 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.947 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.947 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.947 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.947 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.947 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.947 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.947 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.947 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.947 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.947 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "k8sevents", 2024-03-28T01:08:51.947 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.947 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.947 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.947 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_event_retention_days": { 2024-03-28T01:08:51.948 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "ceph_event_retention_days", 2024-03-28T01:08:51.948 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.948 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.948 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.948 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "7", 2024-03-28T01:08:51.948 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.948 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.948 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.948 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T01:08:51.948 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.948 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.948 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.948 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.948 INFO:teuthology.orchestra.run.smithi169.stdout: "config_check_secs": { 2024-03-28T01:08:51.948 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "config_check_secs", 2024-03-28T01:08:51.948 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.949 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.949 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.949 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:51.949 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "10", 2024-03-28T01:08:51.949 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.949 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.949 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T01:08:51.949 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.949 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.949 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.949 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.949 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.949 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.949 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.949 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.950 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.950 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.950 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.950 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.950 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.950 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.950 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.950 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.950 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.950 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.950 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.950 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.950 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.950 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.950 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.951 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.951 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.951 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.951 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.951 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.951 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.951 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.951 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.951 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.951 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.951 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.951 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.951 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.951 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.951 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.952 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.952 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.952 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.952 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.952 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.952 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.952 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.952 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.952 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.952 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.952 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.952 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.952 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.952 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.952 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.953 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.953 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.953 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.953 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.953 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.953 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.953 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.953 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.953 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.953 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.953 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.953 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.953 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.953 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.953 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.953 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.954 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.954 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.954 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.954 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.954 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.954 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.954 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.954 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.954 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "localpool", 2024-03-28T01:08:51.954 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.954 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.954 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.954 INFO:teuthology.orchestra.run.smithi169.stdout: "failure_domain": { 2024-03-28T01:08:51.954 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "failure_domain", 2024-03-28T01:08:51.954 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.955 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.955 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.955 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "host", 2024-03-28T01:08:51.955 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.955 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.955 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.955 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "failure domain for any created local pool", 2024-03-28T01:08:51.955 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T01:08:51.955 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.955 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.955 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.955 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.955 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.955 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.955 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.956 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.956 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.956 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.956 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.956 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.956 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.956 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.956 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.956 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.956 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.956 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.956 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.956 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.956 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.956 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.957 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.957 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.957 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.957 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.957 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.957 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.957 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.957 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.957 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.957 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.957 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.957 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.957 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.957 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.957 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.957 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.958 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.958 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.958 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.958 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.958 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.958 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.958 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.958 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.958 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.958 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.958 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.958 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.958 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.958 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.958 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.959 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.959 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.959 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.959 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.959 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.959 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.959 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.959 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.959 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.959 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.959 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.959 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.959 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.959 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.959 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "min_size": { 2024-03-28T01:08:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "min_size", 2024-03-28T01:08:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "default min_size for any created local pool", 2024-03-28T01:08:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T01:08:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: "num_rep": { 2024-03-28T01:08:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "num_rep", 2024-03-28T01:08:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "3", 2024-03-28T01:08:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "default replica count for any created local pool", 2024-03-28T01:08:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_num": { 2024-03-28T01:08:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "pg_num", 2024-03-28T01:08:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "128", 2024-03-28T01:08:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T01:08:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: "prefix": { 2024-03-28T01:08:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "prefix", 2024-03-28T01:08:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "name prefix for any created local pool", 2024-03-28T01:08:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: "subtree": { 2024-03-28T01:08:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "subtree", 2024-03-28T01:08:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "rack", 2024-03-28T01:08:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T01:08:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T01:08:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "mds_autoscaler", 2024-03-28T01:08:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "mirroring", 2024-03-28T01:08:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "nfs", 2024-03-28T01:08:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "orchestrator", 2024-03-28T01:08:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: "fail_fs": { 2024-03-28T01:08:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "fail_fs", 2024-03-28T01:08:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T01:08:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "orchestrator": { 2024-03-28T01:08:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "orchestrator", 2024-03-28T01:08:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: "cephadm", 2024-03-28T01:08:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: "rook", 2024-03-28T01:08:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: "test_orchestrator" 2024-03-28T01:08:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Orchestrator backend", 2024-03-28T01:08:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "osd_perf_query", 2024-03-28T01:08:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "osd_support", 2024-03-28T01:08:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "pg_autoscaler", 2024-03-28T01:08:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "sleep_interval": { 2024-03-28T01:08:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "sleep_interval", 2024-03-28T01:08:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "60", 2024-03-28T01:08:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: "threshold": { 2024-03-28T01:08:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "threshold", 2024-03-28T01:08:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "float", 2024-03-28T01:08:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "3.0", 2024-03-28T01:08:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "1.0", 2024-03-28T01:08:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.006 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "scaling threshold", 2024-03-28T01:08:52.006 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T01:08:52.006 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.006 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.006 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.006 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.006 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.006 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.006 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "progress", 2024-03-28T01:08:52.006 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:52.006 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:52.006 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:52.006 INFO:teuthology.orchestra.run.smithi169.stdout: "allow_pg_recovery_event": { 2024-03-28T01:08:52.006 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "allow_pg_recovery_event", 2024-03-28T01:08:52.007 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.007 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.007 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.007 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.007 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.007 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.007 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.007 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T01:08:52.007 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.007 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.007 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.007 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.007 INFO:teuthology.orchestra.run.smithi169.stdout: "enabled": { 2024-03-28T01:08:52.008 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "enabled", 2024-03-28T01:08:52.008 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.008 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.008 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.008 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:52.008 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.008 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.008 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.008 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.008 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.008 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.008 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.008 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.008 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:52.008 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.012 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: "max_completed_events": { 2024-03-28T01:08:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "max_completed_events", 2024-03-28T01:08:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "50", 2024-03-28T01:08:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "number of past completed events to remember", 2024-03-28T01:08:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: "sleep_interval": { 2024-03-28T01:08:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "sleep_interval", 2024-03-28T01:08:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "5", 2024-03-28T01:08:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "how long the module is going to sleep", 2024-03-28T01:08:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "prometheus", 2024-03-28T01:08:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "cache": { 2024-03-28T01:08:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "cache", 2024-03-28T01:08:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: "exclude_perf_counters": { 2024-03-28T01:08:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "exclude_perf_counters", 2024-03-28T01:08:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T01:08:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T01:08:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_stats_pools": { 2024-03-28T01:08:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rbd_stats_pools", 2024-03-28T01:08:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T01:08:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T01:08:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "300", 2024-03-28T01:08:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "scrape_interval": { 2024-03-28T01:08:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "scrape_interval", 2024-03-28T01:08:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "float", 2024-03-28T01:08:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "15.0", 2024-03-28T01:08:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "server_addr": { 2024-03-28T01:08:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "server_addr", 2024-03-28T01:08:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "::", 2024-03-28T01:08:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T01:08:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: "server_port": { 2024-03-28T01:08:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "server_port", 2024-03-28T01:08:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "9283", 2024-03-28T01:08:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T01:08:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "stale_cache_strategy": { 2024-03-28T01:08:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "stale_cache_strategy", 2024-03-28T01:08:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "log", 2024-03-28T01:08:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: "standby_behaviour": { 2024-03-28T01:08:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "standby_behaviour", 2024-03-28T01:08:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "default", 2024-03-28T01:08:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "default", 2024-03-28T01:08:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "error" 2024-03-28T01:08:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "standby_error_status_code": { 2024-03-28T01:08:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "standby_error_status_code", 2024-03-28T01:08:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "500", 2024-03-28T01:08:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "400", 2024-03-28T01:08:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "599", 2024-03-28T01:08:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rbd_support", 2024-03-28T01:08:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "max_concurrent_snap_create": { 2024-03-28T01:08:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "max_concurrent_snap_create", 2024-03-28T01:08:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10", 2024-03-28T01:08:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "mirror_snapshot_schedule": { 2024-03-28T01:08:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T01:08:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: "trash_purge_schedule": { 2024-03-28T01:08:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "trash_purge_schedule", 2024-03-28T01:08:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "restful", 2024-03-28T01:08:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "enable_auth": { 2024-03-28T01:08:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "enable_auth", 2024-03-28T01:08:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: "key_file": { 2024-03-28T01:08:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "key_file", 2024-03-28T01:08:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "server_addr": { 2024-03-28T01:08:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "server_addr", 2024-03-28T01:08:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: "server_port": { 2024-03-28T01:08:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "server_port", 2024-03-28T01:08:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rgw", 2024-03-28T01:08:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rook", 2024-03-28T01:08:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: "storage_class": { 2024-03-28T01:08:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "storage_class", 2024-03-28T01:08:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "local", 2024-03-28T01:08:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T01:08:52.060 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.060 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.060 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.060 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.060 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.060 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.060 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.060 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "selftest", 2024-03-28T01:08:52.060 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:52.060 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:52.060 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:52.060 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:52.060 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:52.060 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.060 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.061 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.061 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.061 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.061 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.061 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.061 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.061 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.061 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.061 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.061 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.061 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.061 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.061 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.061 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.061 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.062 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.062 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.062 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:52.062 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:52.062 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.062 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.062 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.062 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.062 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.062 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.062 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.062 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.062 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.062 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.063 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.063 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.063 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:52.063 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:52.063 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.063 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.063 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.063 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:52.063 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.063 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.063 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.063 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.063 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.063 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.063 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.064 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.064 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.064 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.064 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.064 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.064 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.064 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.064 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.064 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:52.064 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:52.064 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.064 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.064 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.064 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.064 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.065 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.065 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.065 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.065 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.065 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.065 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.065 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.065 INFO:teuthology.orchestra.run.smithi169.stdout: "roption1": { 2024-03-28T01:08:52.065 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "roption1", 2024-03-28T01:08:52.065 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.065 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.065 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.065 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.065 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.065 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.066 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.066 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.066 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.066 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.066 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.066 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.066 INFO:teuthology.orchestra.run.smithi169.stdout: "roption2": { 2024-03-28T01:08:52.066 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "roption2", 2024-03-28T01:08:52.066 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.066 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.066 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.066 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "xyz", 2024-03-28T01:08:52.066 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.066 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.067 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.067 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.067 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.067 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.067 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.067 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.067 INFO:teuthology.orchestra.run.smithi169.stdout: "rwoption1": { 2024-03-28T01:08:52.067 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rwoption1", 2024-03-28T01:08:52.067 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.067 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.067 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.067 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.067 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.067 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.067 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.068 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.068 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.068 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.068 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.068 INFO:teuthology.orchestra.run.smithi169.stdout: "rwoption2": { 2024-03-28T01:08:52.068 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rwoption2", 2024-03-28T01:08:52.068 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:52.068 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.068 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.068 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.068 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.068 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.068 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.069 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.069 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.069 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.069 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.069 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.069 INFO:teuthology.orchestra.run.smithi169.stdout: "rwoption3": { 2024-03-28T01:08:52.069 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rwoption3", 2024-03-28T01:08:52.069 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "float", 2024-03-28T01:08:52.069 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.069 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.069 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.069 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.069 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.069 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.069 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.070 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.070 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.070 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.070 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.070 INFO:teuthology.orchestra.run.smithi169.stdout: "rwoption4": { 2024-03-28T01:08:52.070 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rwoption4", 2024-03-28T01:08:52.070 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.070 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.070 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.070 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.070 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.070 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.070 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.070 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.071 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.071 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.071 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.071 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.071 INFO:teuthology.orchestra.run.smithi169.stdout: "rwoption5": { 2024-03-28T01:08:52.071 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rwoption5", 2024-03-28T01:08:52.071 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.071 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.071 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.071 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.071 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.071 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.071 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.071 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.071 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.071 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.072 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.072 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.072 INFO:teuthology.orchestra.run.smithi169.stdout: "rwoption6": { 2024-03-28T01:08:52.072 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rwoption6", 2024-03-28T01:08:52.072 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.072 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.072 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.072 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:52.072 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.072 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.072 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.072 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.072 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.072 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.073 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.073 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.073 INFO:teuthology.orchestra.run.smithi169.stdout: "rwoption7": { 2024-03-28T01:08:52.073 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rwoption7", 2024-03-28T01:08:52.073 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:52.073 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.073 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.073 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.073 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "1", 2024-03-28T01:08:52.073 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "42", 2024-03-28T01:08:52.073 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.073 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.073 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.073 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.073 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.074 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.074 INFO:teuthology.orchestra.run.smithi169.stdout: "testkey": { 2024-03-28T01:08:52.074 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "testkey", 2024-03-28T01:08:52.074 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.074 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.074 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.074 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.074 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.074 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.074 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.074 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.074 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.074 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.074 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.074 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.075 INFO:teuthology.orchestra.run.smithi169.stdout: "testlkey": { 2024-03-28T01:08:52.075 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "testlkey", 2024-03-28T01:08:52.075 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.075 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.075 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.075 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.075 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.075 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.075 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.075 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.075 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.075 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.075 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.075 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.075 INFO:teuthology.orchestra.run.smithi169.stdout: "testnewline": { 2024-03-28T01:08:52.076 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "testnewline", 2024-03-28T01:08:52.076 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.076 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.076 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.076 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.076 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.076 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.076 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.076 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.076 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.076 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.076 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.076 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.076 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.076 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.077 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.077 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "snap_schedule", 2024-03-28T01:08:52.077 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:52.077 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:52.077 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:52.077 INFO:teuthology.orchestra.run.smithi169.stdout: "allow_m_granularity": { 2024-03-28T01:08:52.077 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "allow_m_granularity", 2024-03-28T01:08:52.077 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.077 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.077 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.077 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.077 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.077 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.077 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.077 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T01:08:52.078 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.078 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.078 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.078 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.078 INFO:teuthology.orchestra.run.smithi169.stdout: "dump_on_update": { 2024-03-28T01:08:52.078 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "dump_on_update", 2024-03-28T01:08:52.078 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.078 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.078 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.078 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.078 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.078 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.078 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.078 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "dump database to debug log on update", 2024-03-28T01:08:52.078 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.079 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.079 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.079 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.079 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:52.079 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:52.079 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.079 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.079 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.079 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.079 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.079 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.079 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.079 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.079 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.080 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.080 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.080 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.080 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.080 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.080 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.080 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.080 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.080 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.080 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.080 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:52.080 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:52.080 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.080 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.080 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.081 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.081 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.081 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.081 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.081 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.081 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.081 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.081 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.081 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.081 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:52.081 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:52.081 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.081 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.081 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.081 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:52.082 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.082 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.082 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.082 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.082 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.082 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.082 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.082 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.082 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.082 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.082 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.082 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.082 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.082 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.082 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.083 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:52.083 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:52.083 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.083 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.083 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.083 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.083 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.083 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.083 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.083 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.083 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.083 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.083 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.083 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.083 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.084 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.084 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.084 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "stats", 2024-03-28T01:08:52.084 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:52.084 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:52.084 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:52.084 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:52.084 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:52.084 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.084 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.084 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.084 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.084 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.084 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.085 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.085 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.085 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.085 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.085 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.085 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.085 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.085 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.085 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.085 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.085 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.085 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.085 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.085 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:52.085 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:52.085 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.086 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.086 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.086 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.086 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.086 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.086 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.086 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.086 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.086 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.086 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.086 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.086 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:52.086 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:52.086 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.087 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.087 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.087 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:52.087 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.087 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.087 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.087 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.087 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.087 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.087 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.087 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.087 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.087 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.087 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.087 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.088 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.088 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.088 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.088 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:52.088 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:52.088 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.088 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.088 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.088 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.088 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.088 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.088 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.088 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.088 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.088 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.088 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.089 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.089 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.089 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.089 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.089 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "status", 2024-03-28T01:08:52.089 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:52.089 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:52.089 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:52.089 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:52.089 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:52.089 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.089 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.089 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.089 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.090 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.090 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.090 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.090 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.090 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.090 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.090 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.090 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.090 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.090 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.090 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.090 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.090 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.090 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.091 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.091 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:52.091 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:52.091 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.091 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.091 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.091 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.091 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.091 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.091 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.091 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.091 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.091 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.091 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.091 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.092 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:52.092 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:52.092 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.092 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.092 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.092 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:52.092 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.092 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.092 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.092 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.092 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.092 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.092 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.092 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.092 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.093 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.093 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.093 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.093 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.093 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.093 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.093 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:52.093 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:52.093 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.093 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.093 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.093 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.093 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.093 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.093 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.093 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.094 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.094 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.094 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.094 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.094 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.094 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.094 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.094 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "telegraf", 2024-03-28T01:08:52.094 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:52.094 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:52.094 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:52.094 INFO:teuthology.orchestra.run.smithi169.stdout: "address": { 2024-03-28T01:08:52.094 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "address", 2024-03-28T01:08:52.094 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.095 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.095 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.095 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T01:08:52.095 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.095 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.095 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.095 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.095 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.095 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.095 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.095 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.095 INFO:teuthology.orchestra.run.smithi169.stdout: "interval": { 2024-03-28T01:08:52.095 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "interval", 2024-03-28T01:08:52.095 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:52.095 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.096 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.096 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "15", 2024-03-28T01:08:52.096 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.096 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.096 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.096 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.096 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.096 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.096 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.096 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.096 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:52.096 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:52.096 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.096 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.096 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.097 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.097 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.097 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.097 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.097 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.097 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.097 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.097 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.097 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.097 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.097 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.097 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.097 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.097 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.097 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.097 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.098 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:52.098 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:52.098 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.098 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.098 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.098 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.098 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.098 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.098 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.098 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.098 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.098 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.098 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.098 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.099 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:52.099 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:52.099 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.099 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.099 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.099 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:52.099 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.099 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.099 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.099 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.099 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.099 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.099 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.099 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.099 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.100 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.100 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.100 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.100 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.100 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.100 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.100 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:52.100 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:52.100 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.100 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.100 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.100 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.100 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.100 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.101 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.101 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.101 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.101 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.101 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.101 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.101 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.101 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.101 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.101 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "telemetry", 2024-03-28T01:08:52.101 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:52.101 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:52.101 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:52.101 INFO:teuthology.orchestra.run.smithi169.stdout: "channel_basic": { 2024-03-28T01:08:52.101 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "channel_basic", 2024-03-28T01:08:52.102 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.102 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.102 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.102 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:52.102 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.102 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.102 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.102 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T01:08:52.102 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.102 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.102 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.102 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.102 INFO:teuthology.orchestra.run.smithi169.stdout: "channel_crash": { 2024-03-28T01:08:52.102 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "channel_crash", 2024-03-28T01:08:52.103 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.103 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.103 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.103 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:52.103 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.103 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.103 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.103 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T01:08:52.103 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.103 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.103 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.103 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.103 INFO:teuthology.orchestra.run.smithi169.stdout: "channel_device": { 2024-03-28T01:08:52.103 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "channel_device", 2024-03-28T01:08:52.103 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.104 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.104 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.104 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:52.104 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.104 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.104 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.104 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T01:08:52.104 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.104 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.104 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.104 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.104 INFO:teuthology.orchestra.run.smithi169.stdout: "channel_ident": { 2024-03-28T01:08:52.104 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "channel_ident", 2024-03-28T01:08:52.104 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.104 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.105 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.105 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.105 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.105 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.105 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.105 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T01:08:52.105 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.105 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.105 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.105 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.105 INFO:teuthology.orchestra.run.smithi169.stdout: "channel_perf": { 2024-03-28T01:08:52.105 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "channel_perf", 2024-03-28T01:08:52.105 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.105 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.105 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.106 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.106 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.106 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.106 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.106 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T01:08:52.106 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.106 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.106 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.106 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.106 INFO:teuthology.orchestra.run.smithi169.stdout: "contact": { 2024-03-28T01:08:52.106 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "contact", 2024-03-28T01:08:52.106 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.106 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.106 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.106 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.107 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.107 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.107 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.107 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.107 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.107 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.107 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.107 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.107 INFO:teuthology.orchestra.run.smithi169.stdout: "description": { 2024-03-28T01:08:52.107 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "description", 2024-03-28T01:08:52.107 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.107 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.107 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.107 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.107 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.108 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.108 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.108 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.108 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.108 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.108 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.108 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.108 INFO:teuthology.orchestra.run.smithi169.stdout: "device_url": { 2024-03-28T01:08:52.108 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "device_url", 2024-03-28T01:08:52.108 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.108 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.108 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.108 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T01:08:52.108 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.109 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.109 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.109 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.109 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.109 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.109 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.109 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.109 INFO:teuthology.orchestra.run.smithi169.stdout: "enabled": { 2024-03-28T01:08:52.109 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "enabled", 2024-03-28T01:08:52.109 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.109 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.109 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.109 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.109 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.109 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.109 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.110 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.110 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.110 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.110 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.110 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.110 INFO:teuthology.orchestra.run.smithi169.stdout: "interval": { 2024-03-28T01:08:52.110 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "interval", 2024-03-28T01:08:52.110 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:52.110 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.110 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.110 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "24", 2024-03-28T01:08:52.110 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "8", 2024-03-28T01:08:52.110 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.110 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.111 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.111 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.111 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.111 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.111 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.111 INFO:teuthology.orchestra.run.smithi169.stdout: "last_opt_revision": { 2024-03-28T01:08:52.111 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "last_opt_revision", 2024-03-28T01:08:52.111 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:52.111 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.111 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.111 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "1", 2024-03-28T01:08:52.111 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.111 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.111 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.111 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.112 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.112 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.112 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.112 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.112 INFO:teuthology.orchestra.run.smithi169.stdout: "leaderboard": { 2024-03-28T01:08:52.112 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "leaderboard", 2024-03-28T01:08:52.112 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.112 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.112 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.112 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.112 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.112 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.112 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.112 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.112 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.113 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.113 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.113 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.113 INFO:teuthology.orchestra.run.smithi169.stdout: "leaderboard_description": { 2024-03-28T01:08:52.113 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "leaderboard_description", 2024-03-28T01:08:52.113 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.113 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.113 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.113 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.113 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.113 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.113 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.113 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.113 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.113 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.113 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.114 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.114 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:52.114 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:52.114 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.114 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.114 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.114 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.114 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.114 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.114 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.114 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.114 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.114 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.114 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.114 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.115 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.115 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.115 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.115 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.115 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.115 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.115 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.115 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:52.115 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:52.115 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.115 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.115 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.115 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.115 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.115 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.116 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.116 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.116 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.116 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.116 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.116 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.116 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:52.116 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:52.116 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.116 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.116 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.116 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:52.116 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.116 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.116 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.116 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.117 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.117 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.117 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.117 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.117 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.117 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.117 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.117 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.117 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.117 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.117 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.117 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:52.117 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:52.117 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.117 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.117 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.118 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.118 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.118 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.118 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.118 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.118 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.118 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.118 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.118 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.118 INFO:teuthology.orchestra.run.smithi169.stdout: "organization": { 2024-03-28T01:08:52.118 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "organization", 2024-03-28T01:08:52.118 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.118 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.118 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.118 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.119 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.119 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.119 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.119 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.119 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.119 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.119 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.119 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.119 INFO:teuthology.orchestra.run.smithi169.stdout: "proxy": { 2024-03-28T01:08:52.119 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "proxy", 2024-03-28T01:08:52.119 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.119 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.119 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.119 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.120 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.120 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.120 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.120 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.120 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.120 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.120 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.120 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.120 INFO:teuthology.orchestra.run.smithi169.stdout: "url": { 2024-03-28T01:08:52.120 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "url", 2024-03-28T01:08:52.120 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.120 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.120 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.120 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T01:08:52.120 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.121 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.121 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.121 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.121 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.121 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.121 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.121 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.121 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.121 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.121 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.121 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "test_orchestrator", 2024-03-28T01:08:52.121 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:52.121 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:52.121 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:52.121 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:52.122 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:52.122 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.122 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.122 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.122 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.122 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.122 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.122 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.122 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.122 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.122 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.122 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.122 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.122 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.122 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.123 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.123 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.123 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.123 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.123 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.123 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:52.123 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:52.123 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.123 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.123 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.123 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.123 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.123 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.123 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.123 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.124 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.124 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.124 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.124 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.124 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:52.124 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:52.124 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.124 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.124 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.124 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:52.124 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.124 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.124 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.124 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.124 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.125 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.125 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.125 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.125 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.125 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.125 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.125 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.125 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.125 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.125 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.125 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:52.125 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:52.125 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.125 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.125 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.126 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.126 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.126 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.126 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.126 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.126 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.126 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.126 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.126 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.126 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.126 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.126 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.126 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "volumes", 2024-03-28T01:08:52.126 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:52.127 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:52.127 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:52.127 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:52.127 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:52.127 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.127 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.127 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.127 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.127 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.127 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.127 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.127 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.127 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.127 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.127 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.127 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.128 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.128 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.128 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.128 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.128 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.128 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.128 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.128 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:52.128 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:52.128 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.128 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.128 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.128 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.128 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.129 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.129 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.129 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.129 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.129 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.129 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.129 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.129 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:52.129 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:52.129 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.129 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.129 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.129 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:52.129 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.129 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.130 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.130 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.130 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.130 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.130 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.130 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.130 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.130 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.130 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.130 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.130 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.130 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.130 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.130 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:52.130 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:52.131 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.131 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.131 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.131 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.131 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.131 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.131 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.131 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.131 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.131 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.131 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.131 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.131 INFO:teuthology.orchestra.run.smithi169.stdout: "max_concurrent_clones": { 2024-03-28T01:08:52.131 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "max_concurrent_clones", 2024-03-28T01:08:52.131 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:52.132 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.132 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.132 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "4", 2024-03-28T01:08:52.132 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.132 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.132 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.132 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T01:08:52.132 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.132 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.132 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.132 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.132 INFO:teuthology.orchestra.run.smithi169.stdout: "periodic_async_work": { 2024-03-28T01:08:52.132 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "periodic_async_work", 2024-03-28T01:08:52.132 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.132 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.133 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.133 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.133 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.133 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.133 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.133 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Periodically check for async work", 2024-03-28T01:08:52.133 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.133 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.133 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.133 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.133 INFO:teuthology.orchestra.run.smithi169.stdout: "snapshot_clone_delay": { 2024-03-28T01:08:52.133 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "snapshot_clone_delay", 2024-03-28T01:08:52.133 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:52.133 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.134 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.134 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "0", 2024-03-28T01:08:52.134 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.134 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.134 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.134 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T01:08:52.134 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.134 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.134 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.134 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.134 INFO:teuthology.orchestra.run.smithi169.stdout: "snapshot_clone_no_wait": { 2024-03-28T01:08:52.134 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T01:08:52.134 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.134 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.134 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.135 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "True", 2024-03-28T01:08:52.135 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.135 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.135 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.135 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T01:08:52.135 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.135 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.135 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.135 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.135 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.135 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.135 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.135 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "zabbix", 2024-03-28T01:08:52.135 INFO:teuthology.orchestra.run.smithi169.stdout: "can_run": true, 2024-03-28T01:08:52.135 INFO:teuthology.orchestra.run.smithi169.stdout: "error_string": "", 2024-03-28T01:08:52.136 INFO:teuthology.orchestra.run.smithi169.stdout: "module_options": { 2024-03-28T01:08:52.136 INFO:teuthology.orchestra.run.smithi169.stdout: "discovery_interval": { 2024-03-28T01:08:52.136 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "discovery_interval", 2024-03-28T01:08:52.136 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "uint", 2024-03-28T01:08:52.136 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.136 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.136 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "100", 2024-03-28T01:08:52.136 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.136 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.136 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.136 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.136 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.136 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.136 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.136 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.137 INFO:teuthology.orchestra.run.smithi169.stdout: "identifier": { 2024-03-28T01:08:52.137 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "identifier", 2024-03-28T01:08:52.137 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.137 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.137 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.137 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.137 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.137 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.137 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.137 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.137 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.137 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.137 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.137 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.137 INFO:teuthology.orchestra.run.smithi169.stdout: "interval": { 2024-03-28T01:08:52.138 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "interval", 2024-03-28T01:08:52.138 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "secs", 2024-03-28T01:08:52.138 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.138 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.138 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "60", 2024-03-28T01:08:52.138 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.138 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.138 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.138 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.138 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.138 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.138 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.138 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.138 INFO:teuthology.orchestra.run.smithi169.stdout: "log_level": { 2024-03-28T01:08:52.138 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_level", 2024-03-28T01:08:52.139 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.139 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.139 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.139 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.139 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.139 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.139 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.139 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.139 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.139 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.139 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.139 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.139 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.139 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.139 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster": { 2024-03-28T01:08:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster", 2024-03-28T01:08:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_cluster_level": { 2024-03-28T01:08:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_cluster_level", 2024-03-28T01:08:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "info", 2024-03-28T01:08:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [ 2024-03-28T01:08:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: "", 2024-03-28T01:08:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: "critical", 2024-03-28T01:08:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: "debug", 2024-03-28T01:08:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: "error", 2024-03-28T01:08:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: "info", 2024-03-28T01:08:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: "warning" 2024-03-28T01:08:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: "log_to_file": { 2024-03-28T01:08:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "log_to_file", 2024-03-28T01:08:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "bool", 2024-03-28T01:08:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 1, 2024-03-28T01:08:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "False", 2024-03-28T01:08:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.143 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.143 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.143 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.143 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.143 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.143 INFO:teuthology.orchestra.run.smithi169.stdout: "zabbix_host": { 2024-03-28T01:08:52.143 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "zabbix_host", 2024-03-28T01:08:52.143 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.143 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.143 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.143 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "", 2024-03-28T01:08:52.143 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.143 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.143 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.143 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.144 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.144 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.144 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.144 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.144 INFO:teuthology.orchestra.run.smithi169.stdout: "zabbix_port": { 2024-03-28T01:08:52.144 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "zabbix_port", 2024-03-28T01:08:52.144 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "int", 2024-03-28T01:08:52.144 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.144 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.144 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "10051", 2024-03-28T01:08:52.144 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.144 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.144 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.144 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.144 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.144 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.145 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.145 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.145 INFO:teuthology.orchestra.run.smithi169.stdout: "zabbix_sender": { 2024-03-28T01:08:52.145 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "zabbix_sender", 2024-03-28T01:08:52.145 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "str", 2024-03-28T01:08:52.145 INFO:teuthology.orchestra.run.smithi169.stdout: "level": "advanced", 2024-03-28T01:08:52.145 INFO:teuthology.orchestra.run.smithi169.stdout: "flags": 0, 2024-03-28T01:08:52.145 INFO:teuthology.orchestra.run.smithi169.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T01:08:52.145 INFO:teuthology.orchestra.run.smithi169.stdout: "min": "", 2024-03-28T01:08:52.145 INFO:teuthology.orchestra.run.smithi169.stdout: "max": "", 2024-03-28T01:08:52.145 INFO:teuthology.orchestra.run.smithi169.stdout: "enum_allowed": [], 2024-03-28T01:08:52.145 INFO:teuthology.orchestra.run.smithi169.stdout: "desc": "", 2024-03-28T01:08:52.145 INFO:teuthology.orchestra.run.smithi169.stdout: "long_desc": "", 2024-03-28T01:08:52.145 INFO:teuthology.orchestra.run.smithi169.stdout: "tags": [], 2024-03-28T01:08:52.145 INFO:teuthology.orchestra.run.smithi169.stdout: "see_also": [] 2024-03-28T01:08:52.146 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.146 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.146 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.146 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.146 INFO:teuthology.orchestra.run.smithi169.stdout: "services": { 2024-03-28T01:08:52.146 INFO:teuthology.orchestra.run.smithi169.stdout: "dashboard": "https://172.21.15.169:8443/" 2024-03-28T01:08:52.146 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.146 INFO:teuthology.orchestra.run.smithi169.stdout: "always_on_modules": { 2024-03-28T01:08:52.146 INFO:teuthology.orchestra.run.smithi169.stdout: "octopus": [ 2024-03-28T01:08:52.146 INFO:teuthology.orchestra.run.smithi169.stdout: "balancer", 2024-03-28T01:08:52.146 INFO:teuthology.orchestra.run.smithi169.stdout: "crash", 2024-03-28T01:08:52.146 INFO:teuthology.orchestra.run.smithi169.stdout: "devicehealth", 2024-03-28T01:08:52.146 INFO:teuthology.orchestra.run.smithi169.stdout: "orchestrator", 2024-03-28T01:08:52.146 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscaler", 2024-03-28T01:08:52.147 INFO:teuthology.orchestra.run.smithi169.stdout: "progress", 2024-03-28T01:08:52.147 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_support", 2024-03-28T01:08:52.147 INFO:teuthology.orchestra.run.smithi169.stdout: "status", 2024-03-28T01:08:52.147 INFO:teuthology.orchestra.run.smithi169.stdout: "telemetry", 2024-03-28T01:08:52.147 INFO:teuthology.orchestra.run.smithi169.stdout: "volumes" 2024-03-28T01:08:52.147 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.147 INFO:teuthology.orchestra.run.smithi169.stdout: "pacific": [ 2024-03-28T01:08:52.147 INFO:teuthology.orchestra.run.smithi169.stdout: "balancer", 2024-03-28T01:08:52.147 INFO:teuthology.orchestra.run.smithi169.stdout: "crash", 2024-03-28T01:08:52.147 INFO:teuthology.orchestra.run.smithi169.stdout: "devicehealth", 2024-03-28T01:08:52.147 INFO:teuthology.orchestra.run.smithi169.stdout: "orchestrator", 2024-03-28T01:08:52.147 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscaler", 2024-03-28T01:08:52.147 INFO:teuthology.orchestra.run.smithi169.stdout: "progress", 2024-03-28T01:08:52.147 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_support", 2024-03-28T01:08:52.147 INFO:teuthology.orchestra.run.smithi169.stdout: "status", 2024-03-28T01:08:52.148 INFO:teuthology.orchestra.run.smithi169.stdout: "telemetry", 2024-03-28T01:08:52.148 INFO:teuthology.orchestra.run.smithi169.stdout: "volumes" 2024-03-28T01:08:52.148 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.148 INFO:teuthology.orchestra.run.smithi169.stdout: "quincy": [ 2024-03-28T01:08:52.148 INFO:teuthology.orchestra.run.smithi169.stdout: "balancer", 2024-03-28T01:08:52.148 INFO:teuthology.orchestra.run.smithi169.stdout: "crash", 2024-03-28T01:08:52.148 INFO:teuthology.orchestra.run.smithi169.stdout: "devicehealth", 2024-03-28T01:08:52.148 INFO:teuthology.orchestra.run.smithi169.stdout: "orchestrator", 2024-03-28T01:08:52.148 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscaler", 2024-03-28T01:08:52.148 INFO:teuthology.orchestra.run.smithi169.stdout: "progress", 2024-03-28T01:08:52.148 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_support", 2024-03-28T01:08:52.148 INFO:teuthology.orchestra.run.smithi169.stdout: "status", 2024-03-28T01:08:52.148 INFO:teuthology.orchestra.run.smithi169.stdout: "telemetry", 2024-03-28T01:08:52.148 INFO:teuthology.orchestra.run.smithi169.stdout: "volumes" 2024-03-28T01:08:52.148 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.148 INFO:teuthology.orchestra.run.smithi169.stdout: "reef": [ 2024-03-28T01:08:52.149 INFO:teuthology.orchestra.run.smithi169.stdout: "balancer", 2024-03-28T01:08:52.149 INFO:teuthology.orchestra.run.smithi169.stdout: "crash", 2024-03-28T01:08:52.149 INFO:teuthology.orchestra.run.smithi169.stdout: "devicehealth", 2024-03-28T01:08:52.149 INFO:teuthology.orchestra.run.smithi169.stdout: "orchestrator", 2024-03-28T01:08:52.149 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscaler", 2024-03-28T01:08:52.149 INFO:teuthology.orchestra.run.smithi169.stdout: "progress", 2024-03-28T01:08:52.149 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_support", 2024-03-28T01:08:52.149 INFO:teuthology.orchestra.run.smithi169.stdout: "status", 2024-03-28T01:08:52.149 INFO:teuthology.orchestra.run.smithi169.stdout: "telemetry", 2024-03-28T01:08:52.149 INFO:teuthology.orchestra.run.smithi169.stdout: "volumes" 2024-03-28T01:08:52.149 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-03-28T01:08:52.149 INFO:teuthology.orchestra.run.smithi169.stdout: "squid": [ 2024-03-28T01:08:52.149 INFO:teuthology.orchestra.run.smithi169.stdout: "balancer", 2024-03-28T01:08:52.149 INFO:teuthology.orchestra.run.smithi169.stdout: "crash", 2024-03-28T01:08:52.149 INFO:teuthology.orchestra.run.smithi169.stdout: "devicehealth", 2024-03-28T01:08:52.150 INFO:teuthology.orchestra.run.smithi169.stdout: "orchestrator", 2024-03-28T01:08:52.150 INFO:teuthology.orchestra.run.smithi169.stdout: "pg_autoscaler", 2024-03-28T01:08:52.150 INFO:teuthology.orchestra.run.smithi169.stdout: "progress", 2024-03-28T01:08:52.150 INFO:teuthology.orchestra.run.smithi169.stdout: "rbd_support", 2024-03-28T01:08:52.150 INFO:teuthology.orchestra.run.smithi169.stdout: "status", 2024-03-28T01:08:52.150 INFO:teuthology.orchestra.run.smithi169.stdout: "telemetry", 2024-03-28T01:08:52.150 INFO:teuthology.orchestra.run.smithi169.stdout: "volumes" 2024-03-28T01:08:52.150 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-28T01:08:52.150 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.150 INFO:teuthology.orchestra.run.smithi169.stdout: "last_failure_osd_epoch": 3, 2024-03-28T01:08:52.150 INFO:teuthology.orchestra.run.smithi169.stdout: "active_clients": [ 2024-03-28T01:08:52.150 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.150 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "devicehealth", 2024-03-28T01:08:52.150 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-28T01:08:52.150 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.151 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-28T01:08:52.151 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:0", 2024-03-28T01:08:52.151 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 1250999478 2024-03-28T01:08:52.151 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.151 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-28T01:08:52.151 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.151 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.151 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "libcephsqlite", 2024-03-28T01:08:52.151 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-28T01:08:52.151 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.151 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-28T01:08:52.151 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:0", 2024-03-28T01:08:52.151 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 511304831 2024-03-28T01:08:52.151 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.151 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-28T01:08:52.151 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.152 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.152 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "rbd_support", 2024-03-28T01:08:52.152 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-28T01:08:52.152 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.152 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-28T01:08:52.152 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:0", 2024-03-28T01:08:52.152 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 1494547996 2024-03-28T01:08:52.152 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.152 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-28T01:08:52.152 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:52.152 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.152 INFO:teuthology.orchestra.run.smithi169.stdout: "name": "volumes", 2024-03-28T01:08:52.152 INFO:teuthology.orchestra.run.smithi169.stdout: "addrvec": [ 2024-03-28T01:08:52.152 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-03-28T01:08:52.152 INFO:teuthology.orchestra.run.smithi169.stdout: "type": "v2", 2024-03-28T01:08:52.152 INFO:teuthology.orchestra.run.smithi169.stdout: "addr": "172.21.15.169:0", 2024-03-28T01:08:52.153 INFO:teuthology.orchestra.run.smithi169.stdout: "nonce": 1388450074 2024-03-28T01:08:52.153 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.153 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-28T01:08:52.153 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:52.153 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-03-28T01:08:52.153 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:08:52.154 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:50 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2833594692' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T01:08:52.154 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:50 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/570433433' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T01:08:52.154 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:50 smithi177 ceph-mon[36206]: pgmap v721: 129 pgs: 129 active+clean; 7.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 46 MiB/s wr, 372 op/s 2024-03-28T01:08:52.155 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:50 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2833594692' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T01:08:52.155 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:50 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/570433433' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T01:08:52.155 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:50 smithi174 ceph-mon[36060]: pgmap v721: 129 pgs: 129 active+clean; 7.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 46 MiB/s wr, 372 op/s 2024-03-28T01:08:52.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.914+0000 7f7e0e1e6640 1 -- 172.21.15.169:0/435307857 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7df40a2800 msgr2=0x7f7df40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:52.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.914+0000 7f7e0e1e6640 1 --2- 172.21.15.169:0/435307857 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7df40a2800 0x7f7df40a4cc0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f7e08101610 tx=0x7f7df8005f90 comp rx=0 tx=0).stop 2024-03-28T01:08:52.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.914+0000 7f7e0e1e6640 1 -- 172.21.15.169:0/435307857 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7e081017b0 msgr2=0x7f7e08196df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:08:52.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.914+0000 7f7e0e1e6640 1 --2- 172.21.15.169:0/435307857 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7e081017b0 0x7f7e08196df0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f7df0000c00 tx=0x7f7df002c650 comp rx=0 tx=0).stop 2024-03-28T01:08:52.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.914+0000 7f7e0e1e6640 1 -- 172.21.15.169:0/435307857 shutdown_connections 2024-03-28T01:08:52.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.914+0000 7f7e0e1e6640 1 --2- 172.21.15.169:0/435307857 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f7df40a2800 0x7f7df40a4cc0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:52.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.914+0000 7f7e0e1e6640 1 --2- 172.21.15.169:0/435307857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e08102170 0x7f7e0819de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:52.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.914+0000 7f7e0e1e6640 1 --2- 172.21.15.169:0/435307857 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7e081017b0 0x7f7e08196df0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:52.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.914+0000 7f7e0e1e6640 1 --2- 172.21.15.169:0/435307857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e081005c0 0x7f7e081968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:08:52.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.914+0000 7f7e0e1e6640 1 -- 172.21.15.169:0/435307857 >> 172.21.15.169:0/435307857 conn(0x7f7e080fbb30 msgr2=0x7f7e08104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:08:52.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.914+0000 7f7e0e1e6640 1 -- 172.21.15.169:0/435307857 shutdown_connections 2024-03-28T01:08:52.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:08:50.914+0000 7f7e0e1e6640 1 -- 172.21.15.169:0/435307857 wait complete. 2024-03-28T01:08:52.161 DEBUG:tasks.check_counter:Getting stats from a 2024-03-28T01:08:52.162 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-28T01:08:52.162 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:50 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2833594692' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T01:08:52.162 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:50 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/570433433' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T01:08:52.162 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:50 smithi169 ceph-mon[32274]: pgmap v721: 129 pgs: 129 active+clean; 7.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 46 MiB/s wr, 372 op/s 2024-03-28T01:08:52.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:51 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/435307857' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T01:08:52.190 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:51 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/435307857' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T01:08:52.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:51 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/435307857' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T01:08:52.435 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T01:08:53.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:52 smithi177 ceph-mon[36206]: pgmap v722: 129 pgs: 129 active+clean; 6.3 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 63 MiB/s wr, 478 op/s 2024-03-28T01:08:53.200 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:52 smithi169 ceph-mon[32274]: pgmap v722: 129 pgs: 129 active+clean; 6.3 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 63 MiB/s wr, 478 op/s 2024-03-28T01:08:53.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:52 smithi174 ceph-mon[36060]: pgmap v722: 129 pgs: 129 active+clean; 6.3 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 63 MiB/s wr, 478 op/s 2024-03-28T01:08:54.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:53 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:08:54.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:53 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:08:54.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:53 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T01:08:55.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:55 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:08:55.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:55 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:08:55.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:55 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:08:55.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:55 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:08:55.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:55 smithi169 ceph-mon[32274]: pgmap v723: 129 pgs: 129 active+clean; 6.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 49 MiB/s wr, 390 op/s 2024-03-28T01:08:55.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:55 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:08:55.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:55 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:08:55.302 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:55 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:08:55.315 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:08:55.315 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T01:08:55.315 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 4, 2024-03-28T01:08:55.315 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 4, 2024-03-28T01:08:55.315 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 3072, 2024-03-28T01:08:55.315 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 448, 2024-03-28T01:08:55.315 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 4, 2024-03-28T01:08:55.315 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 0, 2024-03-28T01:08:55.315 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.001704192, 2024-03-28T01:08:55.315 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.000034058, 2024-03-28T01:08:55.316 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.000654082, 2024-03-28T01:08:55.316 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T01:08:55.316 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:08:55.316 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 3, 2024-03-28T01:08:55.316 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000778212, 2024-03-28T01:08:55.316 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000259404 2024-03-28T01:08:55.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:55.316 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:08:55.316 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-28T01:08:55.316 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-28T01:08:55.316 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-28T01:08:55.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:55.316 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-28T01:08:55.316 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T01:08:55.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:55.317 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T01:08:55.317 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 1, 2024-03-28T01:08:55.317 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 177, 2024-03-28T01:08:55.317 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 160, 2024-03-28T01:08:55.317 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 108720, 2024-03-28T01:08:55.317 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 3, 2024-03-28T01:08:55.317 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 2, 2024-03-28T01:08:55.317 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.017398525, 2024-03-28T01:08:55.317 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.013477068, 2024-03-28T01:08:55.317 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.001338740, 2024-03-28T01:08:55.317 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T01:08:55.317 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:08:55.317 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 175, 2024-03-28T01:08:55.317 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.011906838, 2024-03-28T01:08:55.317 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000068039 2024-03-28T01:08:55.317 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:55.317 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:08:55.318 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-28T01:08:55.318 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-28T01:08:55.318 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-28T01:08:55.318 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:55.318 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-28T01:08:55.318 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 108720 2024-03-28T01:08:55.318 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:55.318 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T01:08:55.318 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 238, 2024-03-28T01:08:55.318 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 233, 2024-03-28T01:08:55.318 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 756176, 2024-03-28T01:08:55.318 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 139920, 2024-03-28T01:08:55.318 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 2, 2024-03-28T01:08:55.318 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 1, 2024-03-28T01:08:55.318 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.043422933, 2024-03-28T01:08:55.319 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.017084351, 2024-03-28T01:08:55.319 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.019612759, 2024-03-28T01:08:55.319 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.007523109, 2024-03-28T01:08:55.319 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:08:55.319 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 232, 2024-03-28T01:08:55.319 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.029837614, 2024-03-28T01:08:55.319 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000128610 2024-03-28T01:08:55.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:55.319 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:08:55.319 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-28T01:08:55.319 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-28T01:08:55.319 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-28T01:08:55.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:55.319 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 756176, 2024-03-28T01:08:55.319 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 139920 2024-03-28T01:08:55.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:55.320 INFO:teuthology.orchestra.run.smithi169.stdout: "cct": { 2024-03-28T01:08:55.320 INFO:teuthology.orchestra.run.smithi169.stdout: "total_workers": 0, 2024-03-28T01:08:55.320 INFO:teuthology.orchestra.run.smithi169.stdout: "unhealthy_workers": 0 2024-03-28T01:08:55.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:55.320 INFO:teuthology.orchestra.run.smithi169.stdout: "mempool": { 2024-03-28T01:08:55.320 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_bytes": 0, 2024-03-28T01:08:55.320 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_items": 0, 2024-03-28T01:08:55.320 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T01:08:55.320 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_items": 0, 2024-03-28T01:08:55.320 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T01:08:55.320 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_items": 0, 2024-03-28T01:08:55.320 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T01:08:55.320 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T01:08:55.320 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T01:08:55.320 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T01:08:55.320 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T01:08:55.321 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_items": 0, 2024-03-28T01:08:55.321 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T01:08:55.321 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T01:08:55.321 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_bytes": 0, 2024-03-28T01:08:55.321 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_items": 0, 2024-03-28T01:08:55.321 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_bytes": 0, 2024-03-28T01:08:55.321 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_items": 0, 2024-03-28T01:08:55.321 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T01:08:55.321 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T01:08:55.321 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T01:08:55.321 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T01:08:55.321 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T01:08:55.321 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_items": 0, 2024-03-28T01:08:55.321 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_bytes": 0, 2024-03-28T01:08:55.321 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_items": 0, 2024-03-28T01:08:55.321 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T01:08:55.322 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T01:08:55.322 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_bytes": 0, 2024-03-28T01:08:55.322 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_items": 0, 2024-03-28T01:08:55.322 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_bytes": 0, 2024-03-28T01:08:55.322 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_items": 0, 2024-03-28T01:08:55.322 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T01:08:55.322 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_items": 0, 2024-03-28T01:08:55.322 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T01:08:55.322 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_items": 0, 2024-03-28T01:08:55.322 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_bytes": 148984, 2024-03-28T01:08:55.322 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_items": 30, 2024-03-28T01:08:55.322 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_bytes": 0, 2024-03-28T01:08:55.322 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_items": 0, 2024-03-28T01:08:55.322 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_bytes": 0, 2024-03-28T01:08:55.322 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_items": 0, 2024-03-28T01:08:55.322 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_bytes": 0, 2024-03-28T01:08:55.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_items": 0, 2024-03-28T01:08:55.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_bytes": 0, 2024-03-28T01:08:55.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_items": 0, 2024-03-28T01:08:55.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_bytes": 0, 2024-03-28T01:08:55.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_items": 0, 2024-03-28T01:08:55.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T01:08:55.323 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_items": 0, 2024-03-28T01:08:55.323 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_bytes": 0, 2024-03-28T01:08:55.323 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_items": 0, 2024-03-28T01:08:55.323 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_bytes": 928, 2024-03-28T01:08:55.323 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_items": 2, 2024-03-28T01:08:55.323 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_bytes": 0, 2024-03-28T01:08:55.323 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_items": 0, 2024-03-28T01:08:55.323 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_bytes": 0, 2024-03-28T01:08:55.323 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_items": 0 2024-03-28T01:08:55.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:55.324 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T01:08:55.324 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-03-28T01:08:55.324 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 104857600, 2024-03-28T01:08:55.324 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-03-28T01:08:55.324 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 239, 2024-03-28T01:08:55.324 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 722565, 2024-03-28T01:08:55.324 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-03-28T01:08:55.324 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 239, 2024-03-28T01:08:55.324 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-03-28T01:08:55.324 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-03-28T01:08:55.324 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 239, 2024-03-28T01:08:55.324 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 722565, 2024-03-28T01:08:55.324 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-03-28T01:08:55.324 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-28T01:08:55.324 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-28T01:08:55.325 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-28T01:08:55.325 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:55.325 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:55.325 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:08:55.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:55 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:08:55.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:55 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:08:55.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:55 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:08:55.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:55 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:08:55.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:55 smithi174 ceph-mon[36060]: pgmap v723: 129 pgs: 129 active+clean; 6.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 49 MiB/s wr, 390 op/s 2024-03-28T01:08:55.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:55 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:08:55.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:55 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:08:55.608 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:55 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:08:55.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:55 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:08:55.666 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:55 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:08:55.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:55 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:08:55.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:55 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:08:55.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:55 smithi177 ceph-mon[36206]: pgmap v723: 129 pgs: 129 active+clean; 6.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 49 MiB/s wr, 390 op/s 2024-03-28T01:08:55.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:55 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T01:08:55.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:55 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T01:08:55.667 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:55 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' 2024-03-28T01:08:56.857 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-03-28T01:08:56.857 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-03-28T01:08:56.857 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-03-28T01:08:56.858 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-03-28T01:08:56.858 DEBUG:tasks.check_counter:Getting stats from d 2024-03-28T01:08:56.858 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-28T01:08:57.152 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T01:08:58.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:57 smithi169 ceph-mon[32274]: pgmap v724: 129 pgs: 129 active+clean; 6.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 63 MiB/s wr, 411 op/s 2024-03-28T01:08:58.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:57 smithi174 ceph-mon[36060]: pgmap v724: 129 pgs: 129 active+clean; 6.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 63 MiB/s wr, 411 op/s 2024-03-28T01:08:58.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:57 smithi177 ceph-mon[36206]: pgmap v724: 129 pgs: 129 active+clean; 6.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 63 MiB/s wr, 411 op/s 2024-03-28T01:08:59.075 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:08:59.076 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T01:08:59.076 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 248, 2024-03-28T01:08:59.076 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 241, 2024-03-28T01:08:59.076 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 760880, 2024-03-28T01:08:59.076 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 144112, 2024-03-28T01:08:59.076 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 4, 2024-03-28T01:08:59.076 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 0, 2024-03-28T01:08:59.076 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.053029911, 2024-03-28T01:08:59.076 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.021848390, 2024-03-28T01:08:59.076 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.021447521, 2024-03-28T01:08:59.076 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.007511074, 2024-03-28T01:08:59.076 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:08:59.076 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 241, 2024-03-28T01:08:59.076 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.031100720, 2024-03-28T01:08:59.076 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000129048 2024-03-28T01:08:59.077 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:59.077 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:08:59.077 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-28T01:08:59.077 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-28T01:08:59.077 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-28T01:08:59.077 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:59.077 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 760880, 2024-03-28T01:08:59.077 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 144112 2024-03-28T01:08:59.077 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:59.077 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T01:08:59.077 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 1, 2024-03-28T01:08:59.077 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 182, 2024-03-28T01:08:59.077 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 160, 2024-03-28T01:08:59.077 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 109600, 2024-03-28T01:08:59.077 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 3, 2024-03-28T01:08:59.078 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 2, 2024-03-28T01:08:59.078 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.017821883, 2024-03-28T01:08:59.078 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.013274908, 2024-03-28T01:08:59.078 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.001446890, 2024-03-28T01:08:59.078 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T01:08:59.078 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:08:59.078 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 180, 2024-03-28T01:08:59.078 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.017233905, 2024-03-28T01:08:59.078 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000095743 2024-03-28T01:08:59.078 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:59.078 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:08:59.078 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-28T01:08:59.078 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-28T01:08:59.078 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-28T01:08:59.079 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:59.079 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-28T01:08:59.079 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 109600 2024-03-28T01:08:59.079 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:59.079 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T01:08:59.079 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 0, 2024-03-28T01:08:59.079 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 2, 2024-03-28T01:08:59.079 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 0, 2024-03-28T01:08:59.079 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 0, 2024-03-28T01:08:59.079 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 2, 2024-03-28T01:08:59.079 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 1, 2024-03-28T01:08:59.079 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.000372062, 2024-03-28T01:08:59.079 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.000000000, 2024-03-28T01:08:59.079 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.000271843, 2024-03-28T01:08:59.079 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T01:08:59.080 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:08:59.080 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-28T01:08:59.080 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-28T01:08:59.080 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-28T01:08:59.080 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:59.080 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:08:59.080 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-28T01:08:59.080 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-28T01:08:59.080 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-28T01:08:59.080 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:59.080 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T01:08:59.080 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T01:08:59.080 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:59.080 INFO:teuthology.orchestra.run.smithi169.stdout: "cct": { 2024-03-28T01:08:59.081 INFO:teuthology.orchestra.run.smithi169.stdout: "total_workers": 0, 2024-03-28T01:08:59.081 INFO:teuthology.orchestra.run.smithi169.stdout: "unhealthy_workers": 0 2024-03-28T01:08:59.081 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:59.081 INFO:teuthology.orchestra.run.smithi169.stdout: "mempool": { 2024-03-28T01:08:59.081 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_bytes": 0, 2024-03-28T01:08:59.081 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_items": 0, 2024-03-28T01:08:59.081 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T01:08:59.081 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_items": 0, 2024-03-28T01:08:59.081 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T01:08:59.081 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_items": 0, 2024-03-28T01:08:59.081 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T01:08:59.081 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T01:08:59.081 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T01:08:59.081 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T01:08:59.081 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T01:08:59.082 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_items": 0, 2024-03-28T01:08:59.082 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T01:08:59.082 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T01:08:59.082 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_bytes": 0, 2024-03-28T01:08:59.082 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_items": 0, 2024-03-28T01:08:59.082 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_bytes": 0, 2024-03-28T01:08:59.082 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_items": 0, 2024-03-28T01:08:59.082 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T01:08:59.082 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T01:08:59.082 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T01:08:59.082 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T01:08:59.082 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T01:08:59.082 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_items": 0, 2024-03-28T01:08:59.082 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_bytes": 0, 2024-03-28T01:08:59.083 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_items": 0, 2024-03-28T01:08:59.083 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T01:08:59.083 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T01:08:59.083 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_bytes": 0, 2024-03-28T01:08:59.083 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_items": 0, 2024-03-28T01:08:59.083 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_bytes": 0, 2024-03-28T01:08:59.083 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_items": 0, 2024-03-28T01:08:59.083 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T01:08:59.083 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_items": 0, 2024-03-28T01:08:59.083 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T01:08:59.083 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_items": 0, 2024-03-28T01:08:59.083 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_bytes": 149047, 2024-03-28T01:08:59.083 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_items": 31, 2024-03-28T01:08:59.084 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_bytes": 0, 2024-03-28T01:08:59.084 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_items": 0, 2024-03-28T01:08:59.084 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_bytes": 0, 2024-03-28T01:08:59.084 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_items": 0, 2024-03-28T01:08:59.084 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_bytes": 0, 2024-03-28T01:08:59.084 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_items": 0, 2024-03-28T01:08:59.084 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_bytes": 0, 2024-03-28T01:08:59.084 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_items": 0, 2024-03-28T01:08:59.084 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_bytes": 0, 2024-03-28T01:08:59.084 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_items": 0, 2024-03-28T01:08:59.084 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T01:08:59.084 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_items": 0, 2024-03-28T01:08:59.084 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_bytes": 0, 2024-03-28T01:08:59.085 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_items": 0, 2024-03-28T01:08:59.085 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_bytes": 928, 2024-03-28T01:08:59.085 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_items": 2, 2024-03-28T01:08:59.085 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_bytes": 0, 2024-03-28T01:08:59.085 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_items": 0, 2024-03-28T01:08:59.085 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_bytes": 0, 2024-03-28T01:08:59.085 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_items": 0 2024-03-28T01:08:59.085 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:08:59.085 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T01:08:59.085 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-03-28T01:08:59.085 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 104857600, 2024-03-28T01:08:59.085 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-03-28T01:08:59.085 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 245, 2024-03-28T01:08:59.085 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 723345, 2024-03-28T01:08:59.085 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-03-28T01:08:59.085 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 245, 2024-03-28T01:08:59.086 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-03-28T01:08:59.086 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-03-28T01:08:59.086 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 245, 2024-03-28T01:08:59.086 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 723345, 2024-03-28T01:08:59.086 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-03-28T01:08:59.086 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-28T01:08:59.086 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-28T01:08:59.086 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-28T01:08:59.086 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:59.086 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:08:59.086 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:08:59.771 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-28T01:08:59.771 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-28T01:08:59.771 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-03-28T01:08:59.771 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-03-28T01:08:59.771 DEBUG:tasks.check_counter:Getting stats from g 2024-03-28T01:08:59.772 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-28T01:09:00.019 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T01:09:00.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:08:59 smithi169 ceph-mon[32274]: pgmap v725: 129 pgs: 129 active+clean; 6.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 53 MiB/s wr, 273 op/s 2024-03-28T01:09:00.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:08:59 smithi174 ceph-mon[36060]: pgmap v725: 129 pgs: 129 active+clean; 6.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 53 MiB/s wr, 273 op/s 2024-03-28T01:09:00.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:08:59 smithi177 ceph-mon[36206]: pgmap v725: 129 pgs: 129 active+clean; 6.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 53 MiB/s wr, 273 op/s 2024-03-28T01:09:01.705 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:09:01.706 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T01:09:01.706 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 240, 2024-03-28T01:09:01.706 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 235, 2024-03-28T01:09:01.706 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 756720, 2024-03-28T01:09:01.706 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 141168, 2024-03-28T01:09:01.706 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 4, 2024-03-28T01:09:01.706 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 0, 2024-03-28T01:09:01.706 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.047154165, 2024-03-28T01:09:01.706 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.018004783, 2024-03-28T01:09:01.706 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.021211600, 2024-03-28T01:09:01.706 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.008355146, 2024-03-28T01:09:01.706 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:01.706 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 234, 2024-03-28T01:09:01.707 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.041487321, 2024-03-28T01:09:01.707 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000177296 2024-03-28T01:09:01.707 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:01.707 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:01.707 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-28T01:09:01.707 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-28T01:09:01.707 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-28T01:09:01.707 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:01.707 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 756720, 2024-03-28T01:09:01.707 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 141168 2024-03-28T01:09:01.707 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:01.707 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T01:09:01.707 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 5, 2024-03-28T01:09:01.707 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 181, 2024-03-28T01:09:01.708 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 3232, 2024-03-28T01:09:01.708 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 109520, 2024-03-28T01:09:01.708 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 3, 2024-03-28T01:09:01.708 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 2, 2024-03-28T01:09:01.708 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.017185392, 2024-03-28T01:09:01.708 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.013459234, 2024-03-28T01:09:01.708 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.001374435, 2024-03-28T01:09:01.708 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T01:09:01.708 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:01.708 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 180, 2024-03-28T01:09:01.708 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.015385247, 2024-03-28T01:09:01.708 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000085473 2024-03-28T01:09:01.708 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:01.708 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:01.709 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-28T01:09:01.709 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-28T01:09:01.709 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-28T01:09:01.709 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:01.709 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 3232, 2024-03-28T01:09:01.709 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 109520 2024-03-28T01:09:01.709 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:01.709 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T01:09:01.709 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 0, 2024-03-28T01:09:01.709 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 2, 2024-03-28T01:09:01.709 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 0, 2024-03-28T01:09:01.709 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 0, 2024-03-28T01:09:01.710 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 2, 2024-03-28T01:09:01.710 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 1, 2024-03-28T01:09:01.710 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.000314165, 2024-03-28T01:09:01.710 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.000000000, 2024-03-28T01:09:01.710 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.000245446, 2024-03-28T01:09:01.710 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T01:09:01.710 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:01.710 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-28T01:09:01.710 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-28T01:09:01.710 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-28T01:09:01.710 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:01.710 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:01.710 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-28T01:09:01.710 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-28T01:09:01.711 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-28T01:09:01.711 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:01.711 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T01:09:01.711 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T01:09:01.711 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:01.711 INFO:teuthology.orchestra.run.smithi169.stdout: "cct": { 2024-03-28T01:09:01.711 INFO:teuthology.orchestra.run.smithi169.stdout: "total_workers": 0, 2024-03-28T01:09:01.711 INFO:teuthology.orchestra.run.smithi169.stdout: "unhealthy_workers": 0 2024-03-28T01:09:01.711 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:01.711 INFO:teuthology.orchestra.run.smithi169.stdout: "mempool": { 2024-03-28T01:09:01.711 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_bytes": 0, 2024-03-28T01:09:01.712 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_items": 0, 2024-03-28T01:09:01.712 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T01:09:01.712 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_items": 0, 2024-03-28T01:09:01.712 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T01:09:01.712 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_items": 0, 2024-03-28T01:09:01.712 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T01:09:01.712 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T01:09:01.712 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T01:09:01.712 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T01:09:01.712 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T01:09:01.712 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_items": 0, 2024-03-28T01:09:01.712 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T01:09:01.712 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T01:09:01.712 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_bytes": 0, 2024-03-28T01:09:01.713 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_items": 0, 2024-03-28T01:09:01.713 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_bytes": 0, 2024-03-28T01:09:01.713 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_items": 0, 2024-03-28T01:09:01.713 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T01:09:01.713 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T01:09:01.713 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T01:09:01.713 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T01:09:01.713 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T01:09:01.713 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_items": 0, 2024-03-28T01:09:01.713 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_bytes": 0, 2024-03-28T01:09:01.713 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_items": 0, 2024-03-28T01:09:01.713 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T01:09:01.714 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T01:09:01.714 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_bytes": 0, 2024-03-28T01:09:01.714 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_items": 0, 2024-03-28T01:09:01.714 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_bytes": 0, 2024-03-28T01:09:01.714 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_items": 0, 2024-03-28T01:09:01.714 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T01:09:01.714 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_items": 0, 2024-03-28T01:09:01.714 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T01:09:01.714 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_items": 0, 2024-03-28T01:09:01.714 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_bytes": 148967, 2024-03-28T01:09:01.714 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_items": 29, 2024-03-28T01:09:01.715 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_bytes": 0, 2024-03-28T01:09:01.715 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_items": 0, 2024-03-28T01:09:01.715 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_bytes": 0, 2024-03-28T01:09:01.715 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_items": 0, 2024-03-28T01:09:01.715 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_bytes": 0, 2024-03-28T01:09:01.715 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_items": 0, 2024-03-28T01:09:01.715 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_bytes": 0, 2024-03-28T01:09:01.715 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_items": 0, 2024-03-28T01:09:01.715 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_bytes": 0, 2024-03-28T01:09:01.715 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_items": 0, 2024-03-28T01:09:01.715 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T01:09:01.715 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_items": 0, 2024-03-28T01:09:01.715 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_bytes": 0, 2024-03-28T01:09:01.715 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_items": 0, 2024-03-28T01:09:01.715 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_bytes": 928, 2024-03-28T01:09:01.716 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_items": 2, 2024-03-28T01:09:01.716 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_bytes": 0, 2024-03-28T01:09:01.716 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_items": 0, 2024-03-28T01:09:01.716 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_bytes": 0, 2024-03-28T01:09:01.716 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_items": 0 2024-03-28T01:09:01.716 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:01.716 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T01:09:01.716 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-03-28T01:09:01.716 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 104857600, 2024-03-28T01:09:01.716 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-03-28T01:09:01.716 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 241, 2024-03-28T01:09:01.716 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 722825, 2024-03-28T01:09:01.716 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:01.716 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 241, 2024-03-28T01:09:01.716 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-03-28T01:09:01.716 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-03-28T01:09:01.717 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 241, 2024-03-28T01:09:01.717 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 722825, 2024-03-28T01:09:01.717 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-03-28T01:09:01.717 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-28T01:09:01.717 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-28T01:09:01.717 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-28T01:09:01.717 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:09:01.717 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:09:01.717 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:09:02.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:01 smithi169 ceph-mon[32274]: pgmap v726: 129 pgs: 129 active+clean; 6.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 53 MiB/s wr, 235 op/s 2024-03-28T01:09:02.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:01 smithi174 ceph-mon[36060]: pgmap v726: 129 pgs: 129 active+clean; 6.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 53 MiB/s wr, 235 op/s 2024-03-28T01:09:02.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:01 smithi177 ceph-mon[36206]: pgmap v726: 129 pgs: 129 active+clean; 6.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 53 MiB/s wr, 235 op/s 2024-03-28T01:09:02.364 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-28T01:09:02.364 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-28T01:09:02.364 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-03-28T01:09:02.364 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-03-28T01:09:02.364 DEBUG:tasks.check_counter:Getting stats from j 2024-03-28T01:09:02.364 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-28T01:09:02.609 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T01:09:04.030 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:03 smithi169 ceph-mon[32274]: pgmap v727: 129 pgs: 129 active+clean; 6.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 57 MiB/s wr, 198 op/s 2024-03-28T01:09:04.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:03 smithi174 ceph-mon[36060]: pgmap v727: 129 pgs: 129 active+clean; 6.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 57 MiB/s wr, 198 op/s 2024-03-28T01:09:04.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:03 smithi177 ceph-mon[36206]: pgmap v727: 129 pgs: 129 active+clean; 6.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 57 MiB/s wr, 198 op/s 2024-03-28T01:09:04.443 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-03-28T01:09:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T01:09:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 0, 2024-03-28T01:09:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 2, 2024-03-28T01:09:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 0, 2024-03-28T01:09:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 0, 2024-03-28T01:09:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 4, 2024-03-28T01:09:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 0, 2024-03-28T01:09:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.001318285, 2024-03-28T01:09:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.000000000, 2024-03-28T01:09:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.000381580, 2024-03-28T01:09:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T01:09:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:04.444 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-28T01:09:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-28T01:09:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-28T01:09:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-28T01:09:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-28T01:09:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-28T01:09:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T01:09:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T01:09:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T01:09:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 251, 2024-03-28T01:09:04.445 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 244, 2024-03-28T01:09:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 762320, 2024-03-28T01:09:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 145696, 2024-03-28T01:09:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 2, 2024-03-28T01:09:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 1, 2024-03-28T01:09:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.046472887, 2024-03-28T01:09:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.018086300, 2024-03-28T01:09:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.022478743, 2024-03-28T01:09:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.007158699, 2024-03-28T01:09:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 244, 2024-03-28T01:09:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.042394426, 2024-03-28T01:09:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000173747 2024-03-28T01:09:04.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-28T01:09:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-28T01:09:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-28T01:09:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 762320, 2024-03-28T01:09:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 145696 2024-03-28T01:09:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T01:09:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_messages": 2, 2024-03-28T01:09:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages": 185, 2024-03-28T01:09:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_bytes": 320, 2024-03-28T01:09:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_bytes": 188048, 2024-03-28T01:09:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_created_connections": 4, 2024-03-28T01:09:04.447 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_active_connections": 2, 2024-03-28T01:09:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_total_time": 0.017461454, 2024-03-28T01:09:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_send_time": 0.012579538, 2024-03-28T01:09:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_recv_time": 0.001570050, 2024-03-28T01:09:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T01:09:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 183, 2024-03-28T01:09:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.061368497, 2024-03-28T01:09:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000335346 2024-03-28T01:09:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-28T01:09:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-28T01:09:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-28T01:09:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:04.448 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T01:09:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "msgr_send_encrypted_bytes": 188048 2024-03-28T01:09:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "cct": { 2024-03-28T01:09:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "total_workers": 0, 2024-03-28T01:09:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "unhealthy_workers": 0 2024-03-28T01:09:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "mempool": { 2024-03-28T01:09:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_bytes": 0, 2024-03-28T01:09:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "bloom_filter_items": 0, 2024-03-28T01:09:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T01:09:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_alloc_items": 0, 2024-03-28T01:09:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T01:09:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_data_items": 0, 2024-03-28T01:09:04.449 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T01:09:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T01:09:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T01:09:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T01:09:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T01:09:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_other_items": 0, 2024-03-28T01:09:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T01:09:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T01:09:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_bytes": 0, 2024-03-28T01:09:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_extent_items": 0, 2024-03-28T01:09:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_bytes": 0, 2024-03-28T01:09:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_blob_items": 0, 2024-03-28T01:09:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T01:09:04.450 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T01:09:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T01:09:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T01:09:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T01:09:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_fsck_items": 0, 2024-03-28T01:09:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_bytes": 0, 2024-03-28T01:09:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_txc_items": 0, 2024-03-28T01:09:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T01:09:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T01:09:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_bytes": 0, 2024-03-28T01:09:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "bluestore_writing_items": 0, 2024-03-28T01:09:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_bytes": 0, 2024-03-28T01:09:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_items": 0, 2024-03-28T01:09:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T01:09:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_reader_items": 0, 2024-03-28T01:09:04.451 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T01:09:04.452 INFO:teuthology.orchestra.run.smithi169.stdout: "bluefs_file_writer_items": 0, 2024-03-28T01:09:04.452 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_bytes": 148967, 2024-03-28T01:09:04.452 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_anon_items": 29, 2024-03-28T01:09:04.452 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_bytes": 0, 2024-03-28T01:09:04.452 INFO:teuthology.orchestra.run.smithi169.stdout: "buffer_meta_items": 0, 2024-03-28T01:09:04.452 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_bytes": 0, 2024-03-28T01:09:04.452 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_items": 0, 2024-03-28T01:09:04.452 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_bytes": 0, 2024-03-28T01:09:04.452 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_mapbl_items": 0, 2024-03-28T01:09:04.452 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_bytes": 0, 2024-03-28T01:09:04.452 INFO:teuthology.orchestra.run.smithi169.stdout: "osd_pglog_items": 0, 2024-03-28T01:09:04.452 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_bytes": 0, 2024-03-28T01:09:04.452 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_items": 0, 2024-03-28T01:09:04.452 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T01:09:04.452 INFO:teuthology.orchestra.run.smithi169.stdout: "osdmap_mapping_items": 0, 2024-03-28T01:09:04.453 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_bytes": 0, 2024-03-28T01:09:04.453 INFO:teuthology.orchestra.run.smithi169.stdout: "pgmap_items": 0, 2024-03-28T01:09:04.453 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_bytes": 928, 2024-03-28T01:09:04.453 INFO:teuthology.orchestra.run.smithi169.stdout: "mds_co_items": 2, 2024-03-28T01:09:04.453 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_bytes": 0, 2024-03-28T01:09:04.453 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_1_items": 0, 2024-03-28T01:09:04.453 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_bytes": 0, 2024-03-28T01:09:04.453 INFO:teuthology.orchestra.run.smithi169.stdout: "unittest_2_items": 0 2024-03-28T01:09:04.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-03-28T01:09:04.453 INFO:teuthology.orchestra.run.smithi169.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T01:09:04.453 INFO:teuthology.orchestra.run.smithi169.stdout: "val": 0, 2024-03-28T01:09:04.453 INFO:teuthology.orchestra.run.smithi169.stdout: "max": 104857600, 2024-03-28T01:09:04.453 INFO:teuthology.orchestra.run.smithi169.stdout: "get_started": 0, 2024-03-28T01:09:04.453 INFO:teuthology.orchestra.run.smithi169.stdout: "get": 249, 2024-03-28T01:09:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: "get_sum": 724399, 2024-03-28T01:09:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: "get_or_fail_success": 249, 2024-03-28T01:09:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: "take": 0, 2024-03-28T01:09:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: "take_sum": 0, 2024-03-28T01:09:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: "put": 249, 2024-03-28T01:09:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: "put_sum": 724399, 2024-03-28T01:09:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: "wait": { 2024-03-28T01:09:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: "avgcount": 0, 2024-03-28T01:09:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: "sum": 0.000000000, 2024-03-28T01:09:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: "avgtime": 0.000000000 2024-03-28T01:09:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:09:04.454 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-03-28T01:09:04.454 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-03-28T01:09:05.142 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-28T01:09:05.142 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-28T01:09:05.142 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-28T01:09:05.142 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-28T01:09:05.142 DEBUG:tasks.check_counter:Getting stats from b 2024-03-28T01:09:05.142 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-28T01:09:05.441 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T01:09:05.816 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:05 smithi174 ceph-mon[36060]: pgmap v728: 129 pgs: 129 active+clean; 6.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 36 MiB/s wr, 83 op/s 2024-03-28T01:09:06.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:05 smithi169 ceph-mon[32274]: pgmap v728: 129 pgs: 129 active+clean; 6.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 36 MiB/s wr, 83 op/s 2024-03-28T01:09:06.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:05 smithi177 ceph-mon[36206]: pgmap v728: 129 pgs: 129 active+clean; 6.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 36 MiB/s wr, 83 op/s 2024-03-28T01:09:07.375 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-28T01:09:07.375 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T01:09:07.375 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 5552, 2024-03-28T01:09:07.376 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 6205, 2024-03-28T01:09:07.376 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 1505055, 2024-03-28T01:09:07.376 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 398182434, 2024-03-28T01:09:07.376 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 204, 2024-03-28T01:09:07.376 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 3, 2024-03-28T01:09:07.376 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 2.952880646, 2024-03-28T01:09:07.376 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 1.712449269, 2024-03-28T01:09:07.376 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.259047472, 2024-03-28T01:09:07.376 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.786840334, 2024-03-28T01:09:07.376 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:07.376 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 6206, 2024-03-28T01:09:07.376 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 3.051991717, 2024-03-28T01:09:07.377 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000491780 2024-03-28T01:09:07.377 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.377 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:07.377 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2674, 2024-03-28T01:09:07.377 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000797627, 2024-03-28T01:09:07.377 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000298 2024-03-28T01:09:07.377 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.377 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 38384, 2024-03-28T01:09:07.377 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 1456 2024-03-28T01:09:07.377 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.377 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T01:09:07.377 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 4134, 2024-03-28T01:09:07.377 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 4851, 2024-03-28T01:09:07.377 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 1856338, 2024-03-28T01:09:07.377 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 212609527, 2024-03-28T01:09:07.378 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 45, 2024-03-28T01:09:07.378 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 5, 2024-03-28T01:09:07.378 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 1.553011518, 2024-03-28T01:09:07.378 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.962748532, 2024-03-28T01:09:07.378 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.135273200, 2024-03-28T01:09:07.378 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.442543306, 2024-03-28T01:09:07.378 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:07.378 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 4853, 2024-03-28T01:09:07.378 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 3.491654673, 2024-03-28T01:09:07.378 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000719483 2024-03-28T01:09:07.378 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.378 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:07.379 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2435, 2024-03-28T01:09:07.379 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000669545, 2024-03-28T01:09:07.379 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000274 2024-03-28T01:09:07.379 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.379 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 810384, 2024-03-28T01:09:07.379 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 172960 2024-03-28T01:09:07.379 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.379 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T01:09:07.379 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 413182, 2024-03-28T01:09:07.379 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 415655, 2024-03-28T01:09:07.379 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 134675875, 2024-03-28T01:09:07.379 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 277318008, 2024-03-28T01:09:07.379 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 18, 2024-03-28T01:09:07.379 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 8, 2024-03-28T01:09:07.379 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 20.628375069, 2024-03-28T01:09:07.379 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 8.275838296, 2024-03-28T01:09:07.380 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 28.532099483, 2024-03-28T01:09:07.380 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.854300763, 2024-03-28T01:09:07.380 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:07.380 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 415648, 2024-03-28T01:09:07.380 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 128.568167612, 2024-03-28T01:09:07.380 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000309319 2024-03-28T01:09:07.380 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.380 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:07.380 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 655329, 2024-03-28T01:09:07.380 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.196562118, 2024-03-28T01:09:07.380 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000299 2024-03-28T01:09:07.380 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.380 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 64416, 2024-03-28T01:09:07.381 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 440000 2024-03-28T01:09:07.381 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.381 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-03-28T01:09:07.381 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 1, 2024-03-28T01:09:07.381 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-03-28T01:09:07.381 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.381 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-MDSRank": { 2024-03-28T01:09:07.381 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-03-28T01:09:07.381 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-03-28T01:09:07.381 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3260, 2024-03-28T01:09:07.381 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 331.340962212, 2024-03-28T01:09:07.381 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.101638331 2024-03-28T01:09:07.381 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:07.381 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.382 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-PurgeQueue": { 2024-03-28T01:09:07.382 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-03-28T01:09:07.382 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-03-28T01:09:07.382 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1022, 2024-03-28T01:09:07.382 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.035896709, 2024-03-28T01:09:07.382 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000035123 2024-03-28T01:09:07.382 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:07.382 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.382 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": { 2024-03-28T01:09:07.382 INFO:teuthology.orchestra.run.smithi174.stdout: "request": 166, 2024-03-28T01:09:07.382 INFO:teuthology.orchestra.run.smithi174.stdout: "reply": 166, 2024-03-28T01:09:07.382 INFO:teuthology.orchestra.run.smithi174.stdout: "reply_latency": { 2024-03-28T01:09:07.382 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 166, 2024-03-28T01:09:07.383 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.393752725, 2024-03-28T01:09:07.383 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.002372004 2024-03-28T01:09:07.383 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.383 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_reply": 0, 2024-03-28T01:09:07.383 INFO:teuthology.orchestra.run.smithi174.stdout: "forward": 0, 2024-03-28T01:09:07.383 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_complete": 0, 2024-03-28T01:09:07.383 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_keys": 0, 2024-03-28T01:09:07.383 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_commit": 150, 2024-03-28T01:09:07.383 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_split": 0, 2024-03-28T01:09:07.383 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_merge": 0, 2024-03-28T01:09:07.383 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes": 29, 2024-03-28T01:09:07.383 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_top": 6, 2024-03-28T01:09:07.383 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_bottom": 23, 2024-03-28T01:09:07.383 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pin_tail": 0, 2024-03-28T01:09:07.383 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pinned": 20, 2024-03-28T01:09:07.383 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_expired": 0, 2024-03-28T01:09:07.384 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_with_caps": 4, 2024-03-28T01:09:07.384 INFO:teuthology.orchestra.run.smithi174.stdout: "caps": 4, 2024-03-28T01:09:07.384 INFO:teuthology.orchestra.run.smithi174.stdout: "subtrees": 2, 2024-03-28T01:09:07.384 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse": 336, 2024-03-28T01:09:07.384 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_hit": 230, 2024-03-28T01:09:07.384 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_forward": 0, 2024-03-28T01:09:07.384 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_discover": 0, 2024-03-28T01:09:07.384 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_dir_fetch": 0, 2024-03-28T01:09:07.384 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_remote_ino": 0, 2024-03-28T01:09:07.384 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_lock": 0, 2024-03-28T01:09:07.384 INFO:teuthology.orchestra.run.smithi174.stdout: "load_cent": 0, 2024-03-28T01:09:07.384 INFO:teuthology.orchestra.run.smithi174.stdout: "q": 0, 2024-03-28T01:09:07.384 INFO:teuthology.orchestra.run.smithi174.stdout: "exported": 0, 2024-03-28T01:09:07.384 INFO:teuthology.orchestra.run.smithi174.stdout: "exported_inodes": 0, 2024-03-28T01:09:07.384 INFO:teuthology.orchestra.run.smithi174.stdout: "imported": 0, 2024-03-28T01:09:07.385 INFO:teuthology.orchestra.run.smithi174.stdout: "imported_inodes": 0, 2024-03-28T01:09:07.385 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_dir_fetch": 0, 2024-03-28T01:09:07.385 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_backtrace_fetch": 544, 2024-03-28T01:09:07.385 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_peer_discover": 0, 2024-03-28T01:09:07.385 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rfiles": 4, 2024-03-28T01:09:07.385 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rbytes": 234, 2024-03-28T01:09:07.385 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rsnaps": 0, 2024-03-28T01:09:07.385 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_fetch": 544, 2024-03-28T01:09:07.385 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_set_tag": 0, 2024-03-28T01:09:07.385 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T01:09:07.385 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_inotable_repaired": 0, 2024-03-28T01:09:07.385 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_inodes": 198, 2024-03-28T01:09:07.385 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_base_inodes": 19, 2024-03-28T01:09:07.385 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dirfrag_rstats": 179, 2024-03-28T01:09:07.386 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_file_inodes": 346, 2024-03-28T01:09:07.386 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_inode_file_caps": 0, 2024-03-28T01:09:07.386 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_revoke": 35, 2024-03-28T01:09:07.386 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_grant": 3082, 2024-03-28T01:09:07.386 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T01:09:07.386 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T01:09:07.386 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T01:09:07.386 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps": 410175, 2024-03-28T01:09:07.386 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps_dirty": 410118, 2024-03-28T01:09:07.386 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_cap_release": 8, 2024-03-28T01:09:07.386 INFO:teuthology.orchestra.run.smithi174.stdout: "process_request_cap_release": 84 2024-03-28T01:09:07.386 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.386 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_cache": { 2024-03-28T01:09:07.386 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update": 8, 2024-03-28T01:09:07.387 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update_receipt": 0, 2024-03-28T01:09:07.387 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_try_discover": 0, 2024-03-28T01:09:07.387 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_send_discover": 0, 2024-03-28T01:09:07.387 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_handle_discover": 19, 2024-03-28T01:09:07.387 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays": 4, 2024-03-28T01:09:07.387 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_delayed": 0, 2024-03-28T01:09:07.387 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_enqueuing": 0, 2024-03-28T01:09:07.387 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_created": 22, 2024-03-28T01:09:07.387 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_enqueued": 18, 2024-03-28T01:09:07.387 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_reintegrated": 0, 2024-03-28T01:09:07.387 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_migrated": 0, 2024-03-28T01:09:07.387 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_processing": 0, 2024-03-28T01:09:07.387 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_enqueued": 0, 2024-03-28T01:09:07.387 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_prioritized": 0, 2024-03-28T01:09:07.387 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_started": 0, 2024-03-28T01:09:07.388 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_completed": 0, 2024-03-28T01:09:07.388 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_enqueue_scrub": 19, 2024-03-28T01:09:07.388 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_exportdir": 0, 2024-03-28T01:09:07.388 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_flush": 0, 2024-03-28T01:09:07.388 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragmentdir": 0, 2024-03-28T01:09:07.388 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragstats": 0, 2024-03-28T01:09:07.388 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_inodestats": 0 2024-03-28T01:09:07.388 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.388 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_log": { 2024-03-28T01:09:07.388 INFO:teuthology.orchestra.run.smithi174.stdout: "evlrg": 0, 2024-03-28T01:09:07.388 INFO:teuthology.orchestra.run.smithi174.stdout: "evadd": 411186, 2024-03-28T01:09:07.388 INFO:teuthology.orchestra.run.smithi174.stdout: "evex": 403735, 2024-03-28T01:09:07.388 INFO:teuthology.orchestra.run.smithi174.stdout: "evtrm": 402479, 2024-03-28T01:09:07.388 INFO:teuthology.orchestra.run.smithi174.stdout: "ev": 8707, 2024-03-28T01:09:07.388 INFO:teuthology.orchestra.run.smithi174.stdout: "evexg": 0, 2024-03-28T01:09:07.388 INFO:teuthology.orchestra.run.smithi174.stdout: "evexd": 12563, 2024-03-28T01:09:07.389 INFO:teuthology.orchestra.run.smithi174.stdout: "segadd": 488, 2024-03-28T01:09:07.389 INFO:teuthology.orchestra.run.smithi174.stdout: "segex": 480, 2024-03-28T01:09:07.389 INFO:teuthology.orchestra.run.smithi174.stdout: "segtrm": 478, 2024-03-28T01:09:07.389 INFO:teuthology.orchestra.run.smithi174.stdout: "seg": 10, 2024-03-28T01:09:07.389 INFO:teuthology.orchestra.run.smithi174.stdout: "segmjr": 1, 2024-03-28T01:09:07.389 INFO:teuthology.orchestra.run.smithi174.stdout: "segexg": 0, 2024-03-28T01:09:07.389 INFO:teuthology.orchestra.run.smithi174.stdout: "segexd": 14, 2024-03-28T01:09:07.389 INFO:teuthology.orchestra.run.smithi174.stdout: "expos": 817398663, 2024-03-28T01:09:07.389 INFO:teuthology.orchestra.run.smithi174.stdout: "wrpos": 835797005, 2024-03-28T01:09:07.389 INFO:teuthology.orchestra.run.smithi174.stdout: "rdpos": 0, 2024-03-28T01:09:07.389 INFO:teuthology.orchestra.run.smithi174.stdout: "jlat": { 2024-03-28T01:09:07.389 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 5624, 2024-03-28T01:09:07.389 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 2232.360432908, 2024-03-28T01:09:07.389 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.396934643 2024-03-28T01:09:07.389 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.389 INFO:teuthology.orchestra.run.smithi174.stdout: "replayed": 0 2024-03-28T01:09:07.390 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.390 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_mem": { 2024-03-28T01:09:07.390 INFO:teuthology.orchestra.run.smithi174.stdout: "ino": 27, 2024-03-28T01:09:07.390 INFO:teuthology.orchestra.run.smithi174.stdout: "ino+": 64, 2024-03-28T01:09:07.390 INFO:teuthology.orchestra.run.smithi174.stdout: "ino-": 37, 2024-03-28T01:09:07.390 INFO:teuthology.orchestra.run.smithi174.stdout: "dir": 19, 2024-03-28T01:09:07.390 INFO:teuthology.orchestra.run.smithi174.stdout: "dir+": 22, 2024-03-28T01:09:07.390 INFO:teuthology.orchestra.run.smithi174.stdout: "dir-": 3, 2024-03-28T01:09:07.390 INFO:teuthology.orchestra.run.smithi174.stdout: "dn": 29, 2024-03-28T01:09:07.390 INFO:teuthology.orchestra.run.smithi174.stdout: "dn+": 69, 2024-03-28T01:09:07.390 INFO:teuthology.orchestra.run.smithi174.stdout: "dn-": 40, 2024-03-28T01:09:07.390 INFO:teuthology.orchestra.run.smithi174.stdout: "cap": 3, 2024-03-28T01:09:07.390 INFO:teuthology.orchestra.run.smithi174.stdout: "cap+": 41, 2024-03-28T01:09:07.391 INFO:teuthology.orchestra.run.smithi174.stdout: "cap-": 38, 2024-03-28T01:09:07.391 INFO:teuthology.orchestra.run.smithi174.stdout: "rss": 81552, 2024-03-28T01:09:07.391 INFO:teuthology.orchestra.run.smithi174.stdout: "heap": 231804 2024-03-28T01:09:07.391 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.391 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_server": { 2024-03-28T01:09:07.391 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_client_request": 278, 2024-03-28T01:09:07.391 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_server_request": 0, 2024-03-28T01:09:07.391 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_request": 166, 2024-03-28T01:09:07.391 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_session": 24, 2024-03-28T01:09:07.391 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_peer_request": 0, 2024-03-28T01:09:07.391 INFO:teuthology.orchestra.run.smithi174.stdout: "req_create_latency": { 2024-03-28T01:09:07.391 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 22, 2024-03-28T01:09:07.391 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.064144633, 2024-03-28T01:09:07.391 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.002915665 2024-03-28T01:09:07.391 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.392 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getattr_latency": { 2024-03-28T01:09:07.392 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 28, 2024-03-28T01:09:07.392 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.007391197, 2024-03-28T01:09:07.392 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000263971 2024-03-28T01:09:07.392 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.392 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getfilelock_latency": { 2024-03-28T01:09:07.392 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.392 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.392 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.392 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.392 INFO:teuthology.orchestra.run.smithi174.stdout: "req_link_latency": { 2024-03-28T01:09:07.392 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.392 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.392 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.392 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.392 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookup_latency": { 2024-03-28T01:09:07.393 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 61, 2024-03-28T01:09:07.393 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.065452233, 2024-03-28T01:09:07.393 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.001072987 2024-03-28T01:09:07.393 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.393 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookuphash_latency": { 2024-03-28T01:09:07.393 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.393 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.393 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.393 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.393 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupino_latency": { 2024-03-28T01:09:07.393 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.393 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.393 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.393 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.393 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupname_latency": { 2024-03-28T01:09:07.394 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.394 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.394 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.394 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.394 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupparent_latency": { 2024-03-28T01:09:07.394 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.394 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.394 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.394 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.394 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupsnap_latency": { 2024-03-28T01:09:07.394 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.394 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.394 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.394 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.394 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lssnap_latency": { 2024-03-28T01:09:07.395 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.395 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.395 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.395 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.396 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mkdir_latency": { 2024-03-28T01:09:07.396 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 10, 2024-03-28T01:09:07.396 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.015330084, 2024-03-28T01:09:07.396 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.001533008 2024-03-28T01:09:07.396 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.396 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mknod_latency": { 2024-03-28T01:09:07.396 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.396 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.396 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.396 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.396 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mksnap_latency": { 2024-03-28T01:09:07.396 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.396 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.396 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.396 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.397 INFO:teuthology.orchestra.run.smithi174.stdout: "req_open_latency": { 2024-03-28T01:09:07.397 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2, 2024-03-28T01:09:07.397 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000503948, 2024-03-28T01:09:07.397 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000251974 2024-03-28T01:09:07.397 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.397 INFO:teuthology.orchestra.run.smithi174.stdout: "req_readdir_latency": { 2024-03-28T01:09:07.397 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 6, 2024-03-28T01:09:07.397 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.004454827, 2024-03-28T01:09:07.397 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000742471 2024-03-28T01:09:07.397 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.397 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rename_latency": { 2024-03-28T01:09:07.397 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2, 2024-03-28T01:09:07.397 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.003450862, 2024-03-28T01:09:07.397 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.001725431 2024-03-28T01:09:07.397 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.398 INFO:teuthology.orchestra.run.smithi174.stdout: "req_renamesnap_latency": { 2024-03-28T01:09:07.398 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.398 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.398 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.398 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.398 INFO:teuthology.orchestra.run.smithi174.stdout: "req_snapdiff_latency": { 2024-03-28T01:09:07.398 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.398 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.398 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.398 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.398 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmdir_latency": { 2024-03-28T01:09:07.398 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 4, 2024-03-28T01:09:07.398 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.026290941, 2024-03-28T01:09:07.398 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.006572735 2024-03-28T01:09:07.398 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.399 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmsnap_latency": { 2024-03-28T01:09:07.399 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.399 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.399 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.399 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.399 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmxattr_latency": { 2024-03-28T01:09:07.399 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.399 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.399 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.399 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.399 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setattr_latency": { 2024-03-28T01:09:07.399 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 4, 2024-03-28T01:09:07.399 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.005861046, 2024-03-28T01:09:07.399 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.001465261 2024-03-28T01:09:07.400 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.400 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setdirlayout_latency": { 2024-03-28T01:09:07.400 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.400 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.400 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.400 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.400 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setfilelock_latency": { 2024-03-28T01:09:07.400 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.400 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.400 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.400 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.400 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setlayout_latency": { 2024-03-28T01:09:07.400 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.400 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.400 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.401 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.401 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setxattr_latency": { 2024-03-28T01:09:07.401 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 9, 2024-03-28T01:09:07.401 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.021341652, 2024-03-28T01:09:07.401 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.002371294 2024-03-28T01:09:07.401 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.401 INFO:teuthology.orchestra.run.smithi174.stdout: "req_symlink_latency": { 2024-03-28T01:09:07.401 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.401 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.401 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.401 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.401 INFO:teuthology.orchestra.run.smithi174.stdout: "req_unlink_latency": { 2024-03-28T01:09:07.401 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 18, 2024-03-28T01:09:07.401 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.179531302, 2024-03-28T01:09:07.401 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.009973961 2024-03-28T01:09:07.402 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.402 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_revoke_eviction": 0, 2024-03-28T01:09:07.402 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_acquisition_throttle": 0, 2024-03-28T01:09:07.402 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getvxattr_latency": { 2024-03-28T01:09:07.402 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.402 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.402 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.402 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:07.402 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.402 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_sessions": { 2024-03-28T01:09:07.402 INFO:teuthology.orchestra.run.smithi174.stdout: "session_count": 1, 2024-03-28T01:09:07.402 INFO:teuthology.orchestra.run.smithi174.stdout: "session_add": 4, 2024-03-28T01:09:07.402 INFO:teuthology.orchestra.run.smithi174.stdout: "session_remove": 3, 2024-03-28T01:09:07.402 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_open": 1, 2024-03-28T01:09:07.403 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_stale": 0, 2024-03-28T01:09:07.403 INFO:teuthology.orchestra.run.smithi174.stdout: "total_load": 44, 2024-03-28T01:09:07.403 INFO:teuthology.orchestra.run.smithi174.stdout: "average_load": 44, 2024-03-28T01:09:07.403 INFO:teuthology.orchestra.run.smithi174.stdout: "avg_session_uptime": 739, 2024-03-28T01:09:07.403 INFO:teuthology.orchestra.run.smithi174.stdout: "mdthresh_evicted": 0 2024-03-28T01:09:07.403 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.403 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-03-28T01:09:07.403 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-03-28T01:09:07.403 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-03-28T01:09:07.403 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T01:09:07.403 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-03-28T01:09:07.403 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T01:09:07.403 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-03-28T01:09:07.403 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T01:09:07.403 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T01:09:07.403 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T01:09:07.404 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T01:09:07.404 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T01:09:07.404 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-03-28T01:09:07.404 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T01:09:07.404 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T01:09:07.404 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-03-28T01:09:07.404 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-03-28T01:09:07.404 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-03-28T01:09:07.404 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-03-28T01:09:07.404 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T01:09:07.404 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T01:09:07.404 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T01:09:07.404 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T01:09:07.404 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T01:09:07.404 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-03-28T01:09:07.405 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-03-28T01:09:07.405 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-03-28T01:09:07.405 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T01:09:07.405 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T01:09:07.405 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-03-28T01:09:07.405 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-03-28T01:09:07.405 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-03-28T01:09:07.405 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-03-28T01:09:07.405 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T01:09:07.405 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-03-28T01:09:07.405 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T01:09:07.405 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-03-28T01:09:07.405 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 287881, 2024-03-28T01:09:07.405 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 102, 2024-03-28T01:09:07.405 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 0, 2024-03-28T01:09:07.406 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 0, 2024-03-28T01:09:07.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-03-28T01:09:07.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-03-28T01:09:07.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-03-28T01:09:07.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-03-28T01:09:07.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-03-28T01:09:07.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-03-28T01:09:07.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 8896, 2024-03-28T01:09:07.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 171, 2024-03-28T01:09:07.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T01:09:07.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-03-28T01:09:07.406 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-03-28T01:09:07.406 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-03-28T01:09:07.406 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 110039, 2024-03-28T01:09:07.406 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 1207, 2024-03-28T01:09:07.407 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-03-28T01:09:07.407 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-03-28T01:09:07.407 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-03-28T01:09:07.407 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-03-28T01:09:07.407 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.407 INFO:teuthology.orchestra.run.smithi174.stdout: "objecter": { 2024-03-28T01:09:07.407 INFO:teuthology.orchestra.run.smithi174.stdout: "op_active": 0, 2024-03-28T01:09:07.407 INFO:teuthology.orchestra.run.smithi174.stdout: "op_laggy": 0, 2024-03-28T01:09:07.407 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send": 8962, 2024-03-28T01:09:07.407 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send_bytes": 831969872, 2024-03-28T01:09:07.407 INFO:teuthology.orchestra.run.smithi174.stdout: "op_resend": 37, 2024-03-28T01:09:07.407 INFO:teuthology.orchestra.run.smithi174.stdout: "op_reply": 8962, 2024-03-28T01:09:07.407 INFO:teuthology.orchestra.run.smithi174.stdout: "op_latency": { 2024-03-28T01:09:07.407 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 8962, 2024-03-28T01:09:07.407 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 971.911273165, 2024-03-28T01:09:07.408 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.108448033 2024-03-28T01:09:07.408 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.408 INFO:teuthology.orchestra.run.smithi174.stdout: "op_inflight": 0, 2024-03-28T01:09:07.408 INFO:teuthology.orchestra.run.smithi174.stdout: "oplen_avg": { 2024-03-28T01:09:07.408 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 8962, 2024-03-28T01:09:07.408 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 9366 2024-03-28T01:09:07.408 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.408 INFO:teuthology.orchestra.run.smithi174.stdout: "op": 8962, 2024-03-28T01:09:07.408 INFO:teuthology.orchestra.run.smithi174.stdout: "op_r": 587, 2024-03-28T01:09:07.408 INFO:teuthology.orchestra.run.smithi174.stdout: "op_w": 8375, 2024-03-28T01:09:07.408 INFO:teuthology.orchestra.run.smithi174.stdout: "op_rmw": 0, 2024-03-28T01:09:07.408 INFO:teuthology.orchestra.run.smithi174.stdout: "op_pg": 0, 2024-03-28T01:09:07.408 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_stat": 128, 2024-03-28T01:09:07.408 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_create": 66, 2024-03-28T01:09:07.408 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_read": 24, 2024-03-28T01:09:07.409 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_write": 5629, 2024-03-28T01:09:07.409 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writefull": 163, 2024-03-28T01:09:07.409 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writesame": 0, 2024-03-28T01:09:07.409 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_append": 0, 2024-03-28T01:09:07.409 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_zero": 0, 2024-03-28T01:09:07.409 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_truncate": 0, 2024-03-28T01:09:07.409 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_delete": 1960, 2024-03-28T01:09:07.409 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_mapext": 0, 2024-03-28T01:09:07.409 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_sparse_read": 0, 2024-03-28T01:09:07.409 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_clonerange": 0, 2024-03-28T01:09:07.409 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_getxattr": 563, 2024-03-28T01:09:07.409 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_setxattr": 110, 2024-03-28T01:09:07.409 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_cmpxattr": 0, 2024-03-28T01:09:07.409 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_rmxattr": 0, 2024-03-28T01:09:07.409 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_resetxattrs": 0, 2024-03-28T01:09:07.410 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_call": 0, 2024-03-28T01:09:07.410 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_watch": 0, 2024-03-28T01:09:07.410 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_notify": 0, 2024-03-28T01:09:07.410 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T01:09:07.410 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls": 0, 2024-03-28T01:09:07.410 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls_filter": 0, 2024-03-28T01:09:07.410 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_other": 0, 2024-03-28T01:09:07.410 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_active": 0, 2024-03-28T01:09:07.410 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_send": 0, 2024-03-28T01:09:07.410 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_resend": 0, 2024-03-28T01:09:07.410 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_ping": 0, 2024-03-28T01:09:07.410 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_active": 0, 2024-03-28T01:09:07.410 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_send": 0, 2024-03-28T01:09:07.410 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_resend": 0, 2024-03-28T01:09:07.411 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_active": 0, 2024-03-28T01:09:07.411 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_send": 0, 2024-03-28T01:09:07.411 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_resend": 0, 2024-03-28T01:09:07.411 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_active": 0, 2024-03-28T01:09:07.411 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_send": 0, 2024-03-28T01:09:07.411 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_resend": 0, 2024-03-28T01:09:07.411 INFO:teuthology.orchestra.run.smithi174.stdout: "command_active": 0, 2024-03-28T01:09:07.411 INFO:teuthology.orchestra.run.smithi174.stdout: "command_send": 0, 2024-03-28T01:09:07.411 INFO:teuthology.orchestra.run.smithi174.stdout: "command_resend": 0, 2024-03-28T01:09:07.411 INFO:teuthology.orchestra.run.smithi174.stdout: "map_epoch": 94, 2024-03-28T01:09:07.411 INFO:teuthology.orchestra.run.smithi174.stdout: "map_full": 0, 2024-03-28T01:09:07.411 INFO:teuthology.orchestra.run.smithi174.stdout: "map_inc": 10, 2024-03-28T01:09:07.411 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_sessions": 12, 2024-03-28T01:09:07.411 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_open": 12, 2024-03-28T01:09:07.411 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_close": 0, 2024-03-28T01:09:07.411 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_laggy": 0, 2024-03-28T01:09:07.412 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_wr": 706, 2024-03-28T01:09:07.412 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_rd": 0, 2024-03-28T01:09:07.412 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_del": 17 2024-03-28T01:09:07.412 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.412 INFO:teuthology.orchestra.run.smithi174.stdout: "oft": { 2024-03-28T01:09:07.412 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_objs": 1, 2024-03-28T01:09:07.412 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_kv_pairs": 0, 2024-03-28T01:09:07.412 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_updates": 0, 2024-03-28T01:09:07.412 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_removes": 0 2024-03-28T01:09:07.412 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.412 INFO:teuthology.orchestra.run.smithi174.stdout: "osd-slow-ops": { 2024-03-28T01:09:07.412 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_ops_count": 0 2024-03-28T01:09:07.412 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.412 INFO:teuthology.orchestra.run.smithi174.stdout: "purge_queue": { 2024-03-28T01:09:07.412 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops": 0, 2024-03-28T01:09:07.413 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops_high_water": 516, 2024-03-28T01:09:07.413 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing": 0, 2024-03-28T01:09:07.413 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_high_water": 9, 2024-03-28T01:09:07.413 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed_ops": 1057, 2024-03-28T01:09:07.413 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed": 18, 2024-03-28T01:09:07.413 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_item_in_journal": 0 2024-03-28T01:09:07.413 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.413 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T01:09:07.413 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-28T01:09:07.413 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-03-28T01:09:07.413 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-28T01:09:07.413 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 422864, 2024-03-28T01:09:07.413 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 99948358, 2024-03-28T01:09:07.413 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:07.414 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 422864, 2024-03-28T01:09:07.414 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-03-28T01:09:07.414 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-03-28T01:09:07.414 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 422864, 2024-03-28T01:09:07.414 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 99948358, 2024-03-28T01:09:07.414 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-28T01:09:07.414 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.414 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.414 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.414 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:07.414 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.414 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_bytes": { 2024-03-28T01:09:07.414 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-28T01:09:07.414 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-03-28T01:09:07.414 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-28T01:09:07.415 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-03-28T01:09:07.415 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-03-28T01:09:07.415 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:07.415 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-03-28T01:09:07.415 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 8962, 2024-03-28T01:09:07.415 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 831971810, 2024-03-28T01:09:07.415 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 8962, 2024-03-28T01:09:07.415 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 831971810, 2024-03-28T01:09:07.415 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-28T01:09:07.415 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.415 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.415 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.415 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:07.415 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.415 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_ops": { 2024-03-28T01:09:07.416 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-28T01:09:07.416 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 1024, 2024-03-28T01:09:07.416 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-28T01:09:07.416 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-03-28T01:09:07.416 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-03-28T01:09:07.416 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:07.416 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-03-28T01:09:07.416 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 8962, 2024-03-28T01:09:07.416 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 8962, 2024-03-28T01:09:07.416 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 8962, 2024-03-28T01:09:07.416 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 8962, 2024-03-28T01:09:07.416 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-28T01:09:07.416 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.416 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.416 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.417 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:07.417 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.417 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle": { 2024-03-28T01:09:07.417 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-28T01:09:07.417 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-03-28T01:09:07.417 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-28T01:09:07.417 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 18, 2024-03-28T01:09:07.417 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 1938, 2024-03-28T01:09:07.417 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:07.417 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 18, 2024-03-28T01:09:07.417 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-03-28T01:09:07.417 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-03-28T01:09:07.417 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 5, 2024-03-28T01:09:07.417 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 1938, 2024-03-28T01:09:07.417 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-28T01:09:07.418 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.418 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.418 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.418 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:07.418 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:07.418 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle-0x5606758ca500": { 2024-03-28T01:09:07.418 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 2654, 2024-03-28T01:09:07.418 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-03-28T01:09:07.418 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-28T01:09:07.418 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 411186, 2024-03-28T01:09:07.418 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 831602701, 2024-03-28T01:09:07.418 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:07.418 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 411186, 2024-03-28T01:09:07.418 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-03-28T01:09:07.418 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-03-28T01:09:07.419 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 5624, 2024-03-28T01:09:07.419 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 831600047, 2024-03-28T01:09:07.419 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-28T01:09:07.419 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:07.419 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:07.419 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:07.419 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:07.419 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:07.419 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-28T01:09:08.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:07 smithi169 ceph-mon[32274]: pgmap v729: 129 pgs: 129 active+clean; 6.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 34 MiB/s wr, 74 op/s 2024-03-28T01:09:08.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:07 smithi174 ceph-mon[36060]: pgmap v729: 129 pgs: 129 active+clean; 6.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 34 MiB/s wr, 74 op/s 2024-03-28T01:09:08.170 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:07 smithi177 ceph-mon[36206]: pgmap v729: 129 pgs: 129 active+clean; 6.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 34 MiB/s wr, 74 op/s 2024-03-28T01:09:08.267 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-03-28T01:09:08.267 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-03-28T01:09:08.267 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=8 2024-03-28T01:09:08.267 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-03-28T01:09:08.268 DEBUG:tasks.check_counter:Getting stats from e 2024-03-28T01:09:08.268 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-28T01:09:08.563 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T01:09:10.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:09 smithi169 ceph-mon[32274]: pgmap v730: 129 pgs: 129 active+clean; 6.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 20 MiB/s wr, 46 op/s 2024-03-28T01:09:10.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:09 smithi174 ceph-mon[36060]: pgmap v730: 129 pgs: 129 active+clean; 6.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 20 MiB/s wr, 46 op/s 2024-03-28T01:09:10.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:09 smithi177 ceph-mon[36206]: pgmap v730: 129 pgs: 129 active+clean; 6.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 20 MiB/s wr, 46 op/s 2024-03-28T01:09:11.216 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-28T01:09:11.217 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T01:09:11.217 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 258, 2024-03-28T01:09:11.217 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 252, 2024-03-28T01:09:11.217 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 779906, 2024-03-28T01:09:11.217 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 147108, 2024-03-28T01:09:11.217 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 5, 2024-03-28T01:09:11.217 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 1, 2024-03-28T01:09:11.217 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.037261440, 2024-03-28T01:09:11.217 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.012435704, 2024-03-28T01:09:11.217 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.020418651, 2024-03-28T01:09:11.217 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.006289658, 2024-03-28T01:09:11.217 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:11.218 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 251, 2024-03-28T01:09:11.218 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.013396243, 2024-03-28T01:09:11.218 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000053371 2024-03-28T01:09:11.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.218 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:11.218 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.218 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.218 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.218 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 779072, 2024-03-28T01:09:11.218 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 146784 2024-03-28T01:09:11.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.218 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T01:09:11.219 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 640, 2024-03-28T01:09:11.219 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 823, 2024-03-28T01:09:11.219 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 157290, 2024-03-28T01:09:11.219 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 684345, 2024-03-28T01:09:11.219 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 6, 2024-03-28T01:09:11.219 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 4, 2024-03-28T01:09:11.219 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.075790141, 2024-03-28T01:09:11.219 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.020192670, 2024-03-28T01:09:11.219 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.016575721, 2024-03-28T01:09:11.219 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.028957951, 2024-03-28T01:09:11.219 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:11.219 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 821, 2024-03-28T01:09:11.219 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.215112801, 2024-03-28T01:09:11.219 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000262013 2024-03-28T01:09:11.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.220 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:11.220 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.220 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.220 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.220 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T01:09:11.220 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 481456 2024-03-28T01:09:11.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.220 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T01:09:11.220 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 1682, 2024-03-28T01:09:11.220 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 1682, 2024-03-28T01:09:11.220 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 505577, 2024-03-28T01:09:11.220 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 534110, 2024-03-28T01:09:11.220 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 4, 2024-03-28T01:09:11.221 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 3, 2024-03-28T01:09:11.221 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.146829933, 2024-03-28T01:09:11.221 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.042639686, 2024-03-28T01:09:11.221 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.039683125, 2024-03-28T01:09:11.221 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.053251856, 2024-03-28T01:09:11.221 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:11.221 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1681, 2024-03-28T01:09:11.221 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.094227315, 2024-03-28T01:09:11.221 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000056054 2024-03-28T01:09:11.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.221 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:11.221 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.221 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.221 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.222 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-28T01:09:11.222 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T01:09:11.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.222 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-03-28T01:09:11.222 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 1, 2024-03-28T01:09:11.222 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-03-28T01:09:11.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.222 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-MDSRank": { 2024-03-28T01:09:11.222 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-03-28T01:09:11.222 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-03-28T01:09:11.222 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3981, 2024-03-28T01:09:11.222 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.118894861, 2024-03-28T01:09:11.223 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000029865 2024-03-28T01:09:11.223 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:11.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.223 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-PurgeQueue": { 2024-03-28T01:09:11.223 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-03-28T01:09:11.223 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-03-28T01:09:11.223 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.223 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.223 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.223 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:11.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.223 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": { 2024-03-28T01:09:11.223 INFO:teuthology.orchestra.run.smithi174.stdout: "request": 0, 2024-03-28T01:09:11.223 INFO:teuthology.orchestra.run.smithi174.stdout: "reply": 0, 2024-03-28T01:09:11.223 INFO:teuthology.orchestra.run.smithi174.stdout: "reply_latency": { 2024-03-28T01:09:11.224 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.224 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.224 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.224 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_reply": 0, 2024-03-28T01:09:11.224 INFO:teuthology.orchestra.run.smithi174.stdout: "forward": 0, 2024-03-28T01:09:11.224 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_complete": 0, 2024-03-28T01:09:11.224 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_keys": 0, 2024-03-28T01:09:11.224 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_commit": 0, 2024-03-28T01:09:11.224 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_split": 0, 2024-03-28T01:09:11.224 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_merge": 0, 2024-03-28T01:09:11.224 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes": 0, 2024-03-28T01:09:11.224 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_top": 0, 2024-03-28T01:09:11.224 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_bottom": 0, 2024-03-28T01:09:11.225 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pin_tail": 0, 2024-03-28T01:09:11.225 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pinned": 0, 2024-03-28T01:09:11.225 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_expired": 0, 2024-03-28T01:09:11.225 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_with_caps": 0, 2024-03-28T01:09:11.225 INFO:teuthology.orchestra.run.smithi174.stdout: "caps": 0, 2024-03-28T01:09:11.225 INFO:teuthology.orchestra.run.smithi174.stdout: "subtrees": 1, 2024-03-28T01:09:11.225 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse": 0, 2024-03-28T01:09:11.225 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_hit": 0, 2024-03-28T01:09:11.225 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_forward": 0, 2024-03-28T01:09:11.225 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_discover": 0, 2024-03-28T01:09:11.225 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_dir_fetch": 0, 2024-03-28T01:09:11.225 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_remote_ino": 0, 2024-03-28T01:09:11.225 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_lock": 0, 2024-03-28T01:09:11.225 INFO:teuthology.orchestra.run.smithi174.stdout: "load_cent": 0, 2024-03-28T01:09:11.225 INFO:teuthology.orchestra.run.smithi174.stdout: "q": 0, 2024-03-28T01:09:11.226 INFO:teuthology.orchestra.run.smithi174.stdout: "exported": 0, 2024-03-28T01:09:11.226 INFO:teuthology.orchestra.run.smithi174.stdout: "exported_inodes": 0, 2024-03-28T01:09:11.226 INFO:teuthology.orchestra.run.smithi174.stdout: "imported": 0, 2024-03-28T01:09:11.226 INFO:teuthology.orchestra.run.smithi174.stdout: "imported_inodes": 0, 2024-03-28T01:09:11.226 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_dir_fetch": 0, 2024-03-28T01:09:11.226 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_backtrace_fetch": 0, 2024-03-28T01:09:11.226 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_peer_discover": 0, 2024-03-28T01:09:11.226 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rfiles": 0, 2024-03-28T01:09:11.226 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rbytes": 0, 2024-03-28T01:09:11.226 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rsnaps": 0, 2024-03-28T01:09:11.226 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T01:09:11.226 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_set_tag": 0, 2024-03-28T01:09:11.226 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T01:09:11.226 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_inotable_repaired": 0, 2024-03-28T01:09:11.227 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_inodes": 0, 2024-03-28T01:09:11.227 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T01:09:11.227 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T01:09:11.227 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_file_inodes": 0, 2024-03-28T01:09:11.227 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_inode_file_caps": 0, 2024-03-28T01:09:11.227 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T01:09:11.227 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_grant": 0, 2024-03-28T01:09:11.227 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T01:09:11.227 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T01:09:11.227 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T01:09:11.227 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps": 0, 2024-03-28T01:09:11.227 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps_dirty": 0, 2024-03-28T01:09:11.227 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_cap_release": 0, 2024-03-28T01:09:11.227 INFO:teuthology.orchestra.run.smithi174.stdout: "process_request_cap_release": 0 2024-03-28T01:09:11.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.228 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_cache": { 2024-03-28T01:09:11.228 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update": 0, 2024-03-28T01:09:11.228 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update_receipt": 0, 2024-03-28T01:09:11.228 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_try_discover": 0, 2024-03-28T01:09:11.228 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_send_discover": 0, 2024-03-28T01:09:11.228 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_handle_discover": 0, 2024-03-28T01:09:11.228 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays": 0, 2024-03-28T01:09:11.228 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_delayed": 0, 2024-03-28T01:09:11.228 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_enqueuing": 0, 2024-03-28T01:09:11.228 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_created": 0, 2024-03-28T01:09:11.228 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_enqueued": 0, 2024-03-28T01:09:11.228 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_reintegrated": 0, 2024-03-28T01:09:11.228 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_migrated": 0, 2024-03-28T01:09:11.228 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_processing": 0, 2024-03-28T01:09:11.228 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_enqueued": 0, 2024-03-28T01:09:11.229 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_prioritized": 0, 2024-03-28T01:09:11.229 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_started": 0, 2024-03-28T01:09:11.229 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_completed": 0, 2024-03-28T01:09:11.229 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T01:09:11.229 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_exportdir": 0, 2024-03-28T01:09:11.229 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_flush": 0, 2024-03-28T01:09:11.229 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragmentdir": 0, 2024-03-28T01:09:11.229 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragstats": 0, 2024-03-28T01:09:11.229 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_inodestats": 0 2024-03-28T01:09:11.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.229 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_log": { 2024-03-28T01:09:11.229 INFO:teuthology.orchestra.run.smithi174.stdout: "evlrg": 0, 2024-03-28T01:09:11.229 INFO:teuthology.orchestra.run.smithi174.stdout: "evadd": 0, 2024-03-28T01:09:11.229 INFO:teuthology.orchestra.run.smithi174.stdout: "evex": 0, 2024-03-28T01:09:11.229 INFO:teuthology.orchestra.run.smithi174.stdout: "evtrm": 0, 2024-03-28T01:09:11.229 INFO:teuthology.orchestra.run.smithi174.stdout: "ev": 3, 2024-03-28T01:09:11.230 INFO:teuthology.orchestra.run.smithi174.stdout: "evexg": 0, 2024-03-28T01:09:11.230 INFO:teuthology.orchestra.run.smithi174.stdout: "evexd": 0, 2024-03-28T01:09:11.230 INFO:teuthology.orchestra.run.smithi174.stdout: "segadd": 0, 2024-03-28T01:09:11.230 INFO:teuthology.orchestra.run.smithi174.stdout: "segex": 0, 2024-03-28T01:09:11.230 INFO:teuthology.orchestra.run.smithi174.stdout: "segtrm": 0, 2024-03-28T01:09:11.230 INFO:teuthology.orchestra.run.smithi174.stdout: "seg": 2, 2024-03-28T01:09:11.230 INFO:teuthology.orchestra.run.smithi174.stdout: "segmjr": 2, 2024-03-28T01:09:11.230 INFO:teuthology.orchestra.run.smithi174.stdout: "segexg": 0, 2024-03-28T01:09:11.230 INFO:teuthology.orchestra.run.smithi174.stdout: "segexd": 0, 2024-03-28T01:09:11.230 INFO:teuthology.orchestra.run.smithi174.stdout: "expos": 4194304, 2024-03-28T01:09:11.230 INFO:teuthology.orchestra.run.smithi174.stdout: "wrpos": 4195200, 2024-03-28T01:09:11.230 INFO:teuthology.orchestra.run.smithi174.stdout: "rdpos": 4194892, 2024-03-28T01:09:11.230 INFO:teuthology.orchestra.run.smithi174.stdout: "jlat": { 2024-03-28T01:09:11.231 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.231 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.231 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.231 INFO:teuthology.orchestra.run.smithi174.stdout: "replayed": 3 2024-03-28T01:09:11.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.231 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_mem": { 2024-03-28T01:09:11.231 INFO:teuthology.orchestra.run.smithi174.stdout: "ino": 3, 2024-03-28T01:09:11.231 INFO:teuthology.orchestra.run.smithi174.stdout: "ino+": 3, 2024-03-28T01:09:11.231 INFO:teuthology.orchestra.run.smithi174.stdout: "ino-": 0, 2024-03-28T01:09:11.231 INFO:teuthology.orchestra.run.smithi174.stdout: "dir": 1, 2024-03-28T01:09:11.231 INFO:teuthology.orchestra.run.smithi174.stdout: "dir+": 1, 2024-03-28T01:09:11.231 INFO:teuthology.orchestra.run.smithi174.stdout: "dir-": 0, 2024-03-28T01:09:11.231 INFO:teuthology.orchestra.run.smithi174.stdout: "dn": 0, 2024-03-28T01:09:11.231 INFO:teuthology.orchestra.run.smithi174.stdout: "dn+": 0, 2024-03-28T01:09:11.231 INFO:teuthology.orchestra.run.smithi174.stdout: "dn-": 0, 2024-03-28T01:09:11.232 INFO:teuthology.orchestra.run.smithi174.stdout: "cap": 0, 2024-03-28T01:09:11.232 INFO:teuthology.orchestra.run.smithi174.stdout: "cap+": 0, 2024-03-28T01:09:11.232 INFO:teuthology.orchestra.run.smithi174.stdout: "cap-": 0, 2024-03-28T01:09:11.232 INFO:teuthology.orchestra.run.smithi174.stdout: "rss": 46252, 2024-03-28T01:09:11.232 INFO:teuthology.orchestra.run.smithi174.stdout: "heap": 215420 2024-03-28T01:09:11.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.232 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_server": { 2024-03-28T01:09:11.232 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_client_request": 0, 2024-03-28T01:09:11.232 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_server_request": 0, 2024-03-28T01:09:11.232 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_request": 0, 2024-03-28T01:09:11.232 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_session": 0, 2024-03-28T01:09:11.232 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_peer_request": 0, 2024-03-28T01:09:11.232 INFO:teuthology.orchestra.run.smithi174.stdout: "req_create_latency": { 2024-03-28T01:09:11.232 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.232 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.233 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.233 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getattr_latency": { 2024-03-28T01:09:11.233 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.233 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.233 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.233 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getfilelock_latency": { 2024-03-28T01:09:11.233 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.233 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.233 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.233 INFO:teuthology.orchestra.run.smithi174.stdout: "req_link_latency": { 2024-03-28T01:09:11.233 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.234 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.234 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.234 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookup_latency": { 2024-03-28T01:09:11.234 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.234 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.234 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.234 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookuphash_latency": { 2024-03-28T01:09:11.234 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.234 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.234 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.234 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupino_latency": { 2024-03-28T01:09:11.235 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.235 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.235 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.235 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupname_latency": { 2024-03-28T01:09:11.235 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.235 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.235 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.235 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupparent_latency": { 2024-03-28T01:09:11.235 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.235 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.235 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.235 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupsnap_latency": { 2024-03-28T01:09:11.235 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.236 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.236 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.236 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lssnap_latency": { 2024-03-28T01:09:11.236 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.236 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.236 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.236 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mkdir_latency": { 2024-03-28T01:09:11.236 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.236 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.236 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.237 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mknod_latency": { 2024-03-28T01:09:11.237 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.237 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.237 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.237 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mksnap_latency": { 2024-03-28T01:09:11.237 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.238 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.238 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.238 INFO:teuthology.orchestra.run.smithi174.stdout: "req_open_latency": { 2024-03-28T01:09:11.238 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.238 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.238 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.238 INFO:teuthology.orchestra.run.smithi174.stdout: "req_readdir_latency": { 2024-03-28T01:09:11.238 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.238 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.238 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.238 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rename_latency": { 2024-03-28T01:09:11.239 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.239 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.239 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.239 INFO:teuthology.orchestra.run.smithi174.stdout: "req_renamesnap_latency": { 2024-03-28T01:09:11.239 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.239 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.239 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.239 INFO:teuthology.orchestra.run.smithi174.stdout: "req_snapdiff_latency": { 2024-03-28T01:09:11.239 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.239 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.239 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.240 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmdir_latency": { 2024-03-28T01:09:11.240 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.240 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.240 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.240 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmsnap_latency": { 2024-03-28T01:09:11.240 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.240 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.240 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.240 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmxattr_latency": { 2024-03-28T01:09:11.240 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.240 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.240 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.241 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setattr_latency": { 2024-03-28T01:09:11.241 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.241 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.241 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.241 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setdirlayout_latency": { 2024-03-28T01:09:11.241 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.241 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.241 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.241 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setfilelock_latency": { 2024-03-28T01:09:11.241 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.241 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.242 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.242 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setlayout_latency": { 2024-03-28T01:09:11.242 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.242 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.242 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.242 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setxattr_latency": { 2024-03-28T01:09:11.242 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.242 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.242 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.242 INFO:teuthology.orchestra.run.smithi174.stdout: "req_symlink_latency": { 2024-03-28T01:09:11.242 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.242 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.243 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.243 INFO:teuthology.orchestra.run.smithi174.stdout: "req_unlink_latency": { 2024-03-28T01:09:11.243 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.243 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.243 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.243 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_revoke_eviction": 0, 2024-03-28T01:09:11.243 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_acquisition_throttle": 0, 2024-03-28T01:09:11.243 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getvxattr_latency": { 2024-03-28T01:09:11.243 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.243 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.243 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.243 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:11.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.244 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_sessions": { 2024-03-28T01:09:11.244 INFO:teuthology.orchestra.run.smithi174.stdout: "session_count": 1, 2024-03-28T01:09:11.244 INFO:teuthology.orchestra.run.smithi174.stdout: "session_add": 1, 2024-03-28T01:09:11.244 INFO:teuthology.orchestra.run.smithi174.stdout: "session_remove": 0, 2024-03-28T01:09:11.244 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_open": 1, 2024-03-28T01:09:11.244 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_stale": 0, 2024-03-28T01:09:11.244 INFO:teuthology.orchestra.run.smithi174.stdout: "total_load": 0, 2024-03-28T01:09:11.244 INFO:teuthology.orchestra.run.smithi174.stdout: "average_load": 0, 2024-03-28T01:09:11.244 INFO:teuthology.orchestra.run.smithi174.stdout: "avg_session_uptime": 2120, 2024-03-28T01:09:11.244 INFO:teuthology.orchestra.run.smithi174.stdout: "mdthresh_evicted": 0 2024-03-28T01:09:11.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.244 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-03-28T01:09:11.244 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-03-28T01:09:11.244 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-03-28T01:09:11.244 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T01:09:11.245 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-03-28T01:09:11.245 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T01:09:11.245 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-03-28T01:09:11.245 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T01:09:11.245 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T01:09:11.245 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T01:09:11.245 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T01:09:11.245 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T01:09:11.245 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-03-28T01:09:11.245 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T01:09:11.245 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T01:09:11.245 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-03-28T01:09:11.245 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-03-28T01:09:11.245 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-03-28T01:09:11.246 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-03-28T01:09:11.246 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T01:09:11.246 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T01:09:11.246 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T01:09:11.246 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T01:09:11.246 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T01:09:11.246 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-03-28T01:09:11.246 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-03-28T01:09:11.246 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-03-28T01:09:11.246 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T01:09:11.246 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T01:09:11.246 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-03-28T01:09:11.246 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-03-28T01:09:11.246 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-03-28T01:09:11.247 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-03-28T01:09:11.247 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T01:09:11.247 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-03-28T01:09:11.247 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T01:09:11.247 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-03-28T01:09:11.247 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 190167, 2024-03-28T01:09:11.247 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 52, 2024-03-28T01:09:11.247 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 0, 2024-03-28T01:09:11.247 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 0, 2024-03-28T01:09:11.247 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-03-28T01:09:11.247 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-03-28T01:09:11.247 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-03-28T01:09:11.247 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-03-28T01:09:11.247 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-03-28T01:09:11.247 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-03-28T01:09:11.248 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 8896, 2024-03-28T01:09:11.248 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 171, 2024-03-28T01:09:11.248 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T01:09:11.248 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-03-28T01:09:11.248 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-03-28T01:09:11.248 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-03-28T01:09:11.248 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 8320, 2024-03-28T01:09:11.248 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 13, 2024-03-28T01:09:11.248 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-03-28T01:09:11.248 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-03-28T01:09:11.248 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-03-28T01:09:11.248 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-03-28T01:09:11.248 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.248 INFO:teuthology.orchestra.run.smithi174.stdout: "objecter": { 2024-03-28T01:09:11.249 INFO:teuthology.orchestra.run.smithi174.stdout: "op_active": 0, 2024-03-28T01:09:11.249 INFO:teuthology.orchestra.run.smithi174.stdout: "op_laggy": 0, 2024-03-28T01:09:11.249 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send": 2317, 2024-03-28T01:09:11.249 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send_bytes": 21, 2024-03-28T01:09:11.249 INFO:teuthology.orchestra.run.smithi174.stdout: "op_resend": 3, 2024-03-28T01:09:11.249 INFO:teuthology.orchestra.run.smithi174.stdout: "op_reply": 2317, 2024-03-28T01:09:11.249 INFO:teuthology.orchestra.run.smithi174.stdout: "op_latency": { 2024-03-28T01:09:11.249 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2317, 2024-03-28T01:09:11.249 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 23.210696733, 2024-03-28T01:09:11.249 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.010017564 2024-03-28T01:09:11.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.249 INFO:teuthology.orchestra.run.smithi174.stdout: "op_inflight": 0, 2024-03-28T01:09:11.249 INFO:teuthology.orchestra.run.smithi174.stdout: "oplen_avg": { 2024-03-28T01:09:11.249 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2317, 2024-03-28T01:09:11.249 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 2318 2024-03-28T01:09:11.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.250 INFO:teuthology.orchestra.run.smithi174.stdout: "op": 2317, 2024-03-28T01:09:11.250 INFO:teuthology.orchestra.run.smithi174.stdout: "op_r": 2317, 2024-03-28T01:09:11.250 INFO:teuthology.orchestra.run.smithi174.stdout: "op_w": 0, 2024-03-28T01:09:11.250 INFO:teuthology.orchestra.run.smithi174.stdout: "op_rmw": 0, 2024-03-28T01:09:11.250 INFO:teuthology.orchestra.run.smithi174.stdout: "op_pg": 0, 2024-03-28T01:09:11.250 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_stat": 1473, 2024-03-28T01:09:11.250 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_create": 0, 2024-03-28T01:09:11.250 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_read": 842, 2024-03-28T01:09:11.250 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_write": 0, 2024-03-28T01:09:11.250 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writefull": 0, 2024-03-28T01:09:11.250 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writesame": 0, 2024-03-28T01:09:11.250 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_append": 0, 2024-03-28T01:09:11.250 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_zero": 0, 2024-03-28T01:09:11.250 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_truncate": 0, 2024-03-28T01:09:11.251 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_delete": 0, 2024-03-28T01:09:11.251 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_mapext": 0, 2024-03-28T01:09:11.251 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_sparse_read": 0, 2024-03-28T01:09:11.251 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_clonerange": 0, 2024-03-28T01:09:11.251 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_getxattr": 1, 2024-03-28T01:09:11.251 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_setxattr": 0, 2024-03-28T01:09:11.251 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_cmpxattr": 0, 2024-03-28T01:09:11.251 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_rmxattr": 0, 2024-03-28T01:09:11.251 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_resetxattrs": 0, 2024-03-28T01:09:11.251 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_call": 0, 2024-03-28T01:09:11.251 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_watch": 0, 2024-03-28T01:09:11.251 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_notify": 0, 2024-03-28T01:09:11.251 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T01:09:11.251 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls": 0, 2024-03-28T01:09:11.252 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls_filter": 0, 2024-03-28T01:09:11.252 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_other": 0, 2024-03-28T01:09:11.252 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_active": 0, 2024-03-28T01:09:11.252 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_send": 0, 2024-03-28T01:09:11.252 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_resend": 0, 2024-03-28T01:09:11.252 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_ping": 0, 2024-03-28T01:09:11.252 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_active": 0, 2024-03-28T01:09:11.252 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_send": 0, 2024-03-28T01:09:11.252 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_resend": 0, 2024-03-28T01:09:11.252 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_active": 0, 2024-03-28T01:09:11.252 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_send": 0, 2024-03-28T01:09:11.252 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_resend": 0, 2024-03-28T01:09:11.252 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_active": 0, 2024-03-28T01:09:11.252 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_send": 0, 2024-03-28T01:09:11.252 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_resend": 0, 2024-03-28T01:09:11.252 INFO:teuthology.orchestra.run.smithi174.stdout: "command_active": 0, 2024-03-28T01:09:11.253 INFO:teuthology.orchestra.run.smithi174.stdout: "command_send": 0, 2024-03-28T01:09:11.253 INFO:teuthology.orchestra.run.smithi174.stdout: "command_resend": 0, 2024-03-28T01:09:11.253 INFO:teuthology.orchestra.run.smithi174.stdout: "map_epoch": 94, 2024-03-28T01:09:11.253 INFO:teuthology.orchestra.run.smithi174.stdout: "map_full": 0, 2024-03-28T01:09:11.253 INFO:teuthology.orchestra.run.smithi174.stdout: "map_inc": 4, 2024-03-28T01:09:11.253 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_sessions": 5, 2024-03-28T01:09:11.253 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_open": 5, 2024-03-28T01:09:11.253 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_close": 0, 2024-03-28T01:09:11.253 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_laggy": 0, 2024-03-28T01:09:11.253 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_wr": 0, 2024-03-28T01:09:11.253 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_rd": 2, 2024-03-28T01:09:11.253 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_del": 0 2024-03-28T01:09:11.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.253 INFO:teuthology.orchestra.run.smithi174.stdout: "oft": { 2024-03-28T01:09:11.253 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_objs": 0, 2024-03-28T01:09:11.254 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_kv_pairs": 0, 2024-03-28T01:09:11.254 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_updates": 0, 2024-03-28T01:09:11.254 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_removes": 0 2024-03-28T01:09:11.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.254 INFO:teuthology.orchestra.run.smithi174.stdout: "osd-slow-ops": { 2024-03-28T01:09:11.254 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_ops_count": 0 2024-03-28T01:09:11.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.254 INFO:teuthology.orchestra.run.smithi174.stdout: "purge_queue": { 2024-03-28T01:09:11.254 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops": 0, 2024-03-28T01:09:11.254 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T01:09:11.254 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing": 0, 2024-03-28T01:09:11.254 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_high_water": 0, 2024-03-28T01:09:11.254 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed_ops": 0, 2024-03-28T01:09:11.254 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed": 0, 2024-03-28T01:09:11.255 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_item_in_journal": 0 2024-03-28T01:09:11.255 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.255 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T01:09:11.255 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-28T01:09:11.255 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-03-28T01:09:11.255 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-28T01:09:11.255 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 2576, 2024-03-28T01:09:11.255 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 1194638, 2024-03-28T01:09:11.255 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:11.255 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 2576, 2024-03-28T01:09:11.255 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-03-28T01:09:11.255 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-03-28T01:09:11.255 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2576, 2024-03-28T01:09:11.255 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 1194638, 2024-03-28T01:09:11.255 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-28T01:09:11.256 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.256 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.256 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.256 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:11.256 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.256 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_bytes": { 2024-03-28T01:09:11.256 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-28T01:09:11.256 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-03-28T01:09:11.256 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-28T01:09:11.256 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-03-28T01:09:11.256 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-03-28T01:09:11.256 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:11.256 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-03-28T01:09:11.256 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 2317, 2024-03-28T01:09:11.256 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 901, 2024-03-28T01:09:11.256 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2317, 2024-03-28T01:09:11.257 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 901, 2024-03-28T01:09:11.257 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-28T01:09:11.257 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.257 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.257 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.257 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:11.257 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.257 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_ops": { 2024-03-28T01:09:11.257 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-28T01:09:11.257 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 1024, 2024-03-28T01:09:11.257 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-28T01:09:11.257 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-03-28T01:09:11.257 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-03-28T01:09:11.257 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:11.258 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-03-28T01:09:11.258 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 2317, 2024-03-28T01:09:11.258 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 2317, 2024-03-28T01:09:11.258 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2317, 2024-03-28T01:09:11.258 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 2317, 2024-03-28T01:09:11.258 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-28T01:09:11.258 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.258 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.258 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.258 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:11.258 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.258 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle": { 2024-03-28T01:09:11.258 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-28T01:09:11.258 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-03-28T01:09:11.259 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-28T01:09:11.259 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-03-28T01:09:11.259 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-03-28T01:09:11.259 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:11.259 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-03-28T01:09:11.259 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-03-28T01:09:11.259 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-03-28T01:09:11.259 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-03-28T01:09:11.259 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-03-28T01:09:11.259 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-28T01:09:11.259 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.259 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.259 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.259 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:11.260 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:11.260 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle-0x56026d2306e0": { 2024-03-28T01:09:11.260 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-28T01:09:11.260 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-03-28T01:09:11.260 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-28T01:09:11.260 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-03-28T01:09:11.260 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-03-28T01:09:11.260 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:11.260 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-03-28T01:09:11.260 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-03-28T01:09:11.260 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-03-28T01:09:11.260 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-03-28T01:09:11.260 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-03-28T01:09:11.260 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-28T01:09:11.260 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:11.260 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:11.260 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:11.261 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:11.261 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:11.261 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-28T01:09:12.126 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:11 smithi174 ceph-mon[36060]: pgmap v731: 129 pgs: 129 active+clean; 5.8 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 19 MiB/s wr, 46 op/s 2024-03-28T01:09:12.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:11 smithi177 ceph-mon[36206]: pgmap v731: 129 pgs: 129 active+clean; 5.8 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 19 MiB/s wr, 46 op/s 2024-03-28T01:09:12.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:11 smithi169 ceph-mon[32274]: pgmap v731: 129 pgs: 129 active+clean; 5.8 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 19 MiB/s wr, 46 op/s 2024-03-28T01:09:12.919 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-28T01:09:12.919 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-28T01:09:12.919 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-03-28T01:09:12.919 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-03-28T01:09:12.919 DEBUG:tasks.check_counter:Getting stats from h 2024-03-28T01:09:12.920 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-28T01:09:13.325 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T01:09:14.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:13 smithi174 ceph-mon[36060]: pgmap v732: 129 pgs: 129 active+clean; 4.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 24 MiB/s wr, 102 op/s 2024-03-28T01:09:14.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:13 smithi177 ceph-mon[36206]: pgmap v732: 129 pgs: 129 active+clean; 4.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 24 MiB/s wr, 102 op/s 2024-03-28T01:09:14.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:13 smithi169 ceph-mon[32274]: pgmap v732: 129 pgs: 129 active+clean; 4.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 24 MiB/s wr, 102 op/s 2024-03-28T01:09:15.108 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-28T01:09:15.109 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T01:09:15.109 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 1907, 2024-03-28T01:09:15.109 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 1899, 2024-03-28T01:09:15.109 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 223609574, 2024-03-28T01:09:15.109 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 670182, 2024-03-28T01:09:15.109 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 7, 2024-03-28T01:09:15.109 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 3, 2024-03-28T01:09:15.109 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.289104066, 2024-03-28T01:09:15.109 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.062943714, 2024-03-28T01:09:15.109 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.162992183, 2024-03-28T01:09:15.109 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.054994475, 2024-03-28T01:09:15.109 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:15.110 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1899, 2024-03-28T01:09:15.110 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.081471193, 2024-03-28T01:09:15.110 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000042902 2024-03-28T01:09:15.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.110 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:15.110 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.110 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.110 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.110 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 781136, 2024-03-28T01:09:15.110 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 146432 2024-03-28T01:09:15.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.110 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T01:09:15.111 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 1004, 2024-03-28T01:09:15.111 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 1186, 2024-03-28T01:09:15.111 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 277147021, 2024-03-28T01:09:15.111 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 714700, 2024-03-28T01:09:15.111 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 7, 2024-03-28T01:09:15.111 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 6, 2024-03-28T01:09:15.111 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.287436796, 2024-03-28T01:09:15.111 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.028126650, 2024-03-28T01:09:15.111 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.213875207, 2024-03-28T01:09:15.111 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.036559000, 2024-03-28T01:09:15.111 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:15.111 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1184, 2024-03-28T01:09:15.111 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.071821605, 2024-03-28T01:09:15.111 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000060660 2024-03-28T01:09:15.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.112 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:15.112 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.112 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.112 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.112 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-28T01:09:15.112 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 395680 2024-03-28T01:09:15.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.112 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T01:09:15.112 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 1173, 2024-03-28T01:09:15.112 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 1175, 2024-03-28T01:09:15.112 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 332851229, 2024-03-28T01:09:15.112 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 373021, 2024-03-28T01:09:15.112 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 6, 2024-03-28T01:09:15.113 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 5, 2024-03-28T01:09:15.113 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.338717919, 2024-03-28T01:09:15.113 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.021461287, 2024-03-28T01:09:15.113 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.266564947, 2024-03-28T01:09:15.113 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.040406069, 2024-03-28T01:09:15.113 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:15.113 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1173, 2024-03-28T01:09:15.113 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.113838810, 2024-03-28T01:09:15.113 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000097049 2024-03-28T01:09:15.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.113 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:15.113 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.113 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.114 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.114 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.114 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T01:09:15.114 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T01:09:15.114 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.114 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-03-28T01:09:15.114 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 1, 2024-03-28T01:09:15.114 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-03-28T01:09:15.114 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.114 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-MDSRank": { 2024-03-28T01:09:15.114 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-03-28T01:09:15.114 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-03-28T01:09:15.114 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 5734, 2024-03-28T01:09:15.114 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.153112381, 2024-03-28T01:09:15.115 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000026702 2024-03-28T01:09:15.115 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:15.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.115 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-PurgeQueue": { 2024-03-28T01:09:15.115 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-03-28T01:09:15.115 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-03-28T01:09:15.115 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.115 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.115 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.115 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:15.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.115 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": { 2024-03-28T01:09:15.115 INFO:teuthology.orchestra.run.smithi174.stdout: "request": 0, 2024-03-28T01:09:15.115 INFO:teuthology.orchestra.run.smithi174.stdout: "reply": 0, 2024-03-28T01:09:15.115 INFO:teuthology.orchestra.run.smithi174.stdout: "reply_latency": { 2024-03-28T01:09:15.115 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.116 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.116 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.116 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.116 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_reply": 0, 2024-03-28T01:09:15.116 INFO:teuthology.orchestra.run.smithi174.stdout: "forward": 0, 2024-03-28T01:09:15.116 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_complete": 0, 2024-03-28T01:09:15.116 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_keys": 0, 2024-03-28T01:09:15.116 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_commit": 0, 2024-03-28T01:09:15.116 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_split": 0, 2024-03-28T01:09:15.116 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_merge": 0, 2024-03-28T01:09:15.116 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes": 52, 2024-03-28T01:09:15.116 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_top": 4, 2024-03-28T01:09:15.116 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_bottom": 48, 2024-03-28T01:09:15.116 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pin_tail": 0, 2024-03-28T01:09:15.116 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pinned": 45, 2024-03-28T01:09:15.117 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_expired": 0, 2024-03-28T01:09:15.117 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_with_caps": 0, 2024-03-28T01:09:15.117 INFO:teuthology.orchestra.run.smithi174.stdout: "caps": 0, 2024-03-28T01:09:15.117 INFO:teuthology.orchestra.run.smithi174.stdout: "subtrees": 2, 2024-03-28T01:09:15.117 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse": 0, 2024-03-28T01:09:15.117 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_hit": 0, 2024-03-28T01:09:15.117 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_forward": 0, 2024-03-28T01:09:15.117 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_discover": 0, 2024-03-28T01:09:15.117 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_dir_fetch": 0, 2024-03-28T01:09:15.117 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_remote_ino": 0, 2024-03-28T01:09:15.117 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_lock": 0, 2024-03-28T01:09:15.117 INFO:teuthology.orchestra.run.smithi174.stdout: "load_cent": 0, 2024-03-28T01:09:15.117 INFO:teuthology.orchestra.run.smithi174.stdout: "q": 0, 2024-03-28T01:09:15.118 INFO:teuthology.orchestra.run.smithi174.stdout: "exported": 0, 2024-03-28T01:09:15.118 INFO:teuthology.orchestra.run.smithi174.stdout: "exported_inodes": 0, 2024-03-28T01:09:15.118 INFO:teuthology.orchestra.run.smithi174.stdout: "imported": 0, 2024-03-28T01:09:15.118 INFO:teuthology.orchestra.run.smithi174.stdout: "imported_inodes": 0, 2024-03-28T01:09:15.118 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_dir_fetch": 0, 2024-03-28T01:09:15.118 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_backtrace_fetch": 0, 2024-03-28T01:09:15.118 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_peer_discover": 0, 2024-03-28T01:09:15.118 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rfiles": 4, 2024-03-28T01:09:15.118 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rbytes": 234, 2024-03-28T01:09:15.118 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rsnaps": 0, 2024-03-28T01:09:15.118 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T01:09:15.118 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_set_tag": 0, 2024-03-28T01:09:15.118 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T01:09:15.118 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_inotable_repaired": 0, 2024-03-28T01:09:15.119 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_inodes": 0, 2024-03-28T01:09:15.119 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T01:09:15.119 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T01:09:15.119 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_file_inodes": 0, 2024-03-28T01:09:15.119 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_inode_file_caps": 0, 2024-03-28T01:09:15.119 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T01:09:15.119 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_grant": 0, 2024-03-28T01:09:15.119 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T01:09:15.119 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T01:09:15.119 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T01:09:15.119 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps": 0, 2024-03-28T01:09:15.119 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps_dirty": 0, 2024-03-28T01:09:15.119 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_cap_release": 0, 2024-03-28T01:09:15.119 INFO:teuthology.orchestra.run.smithi174.stdout: "process_request_cap_release": 0 2024-03-28T01:09:15.119 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.120 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_cache": { 2024-03-28T01:09:15.120 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update": 0, 2024-03-28T01:09:15.120 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update_receipt": 0, 2024-03-28T01:09:15.120 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_try_discover": 0, 2024-03-28T01:09:15.120 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_send_discover": 0, 2024-03-28T01:09:15.120 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_handle_discover": 0, 2024-03-28T01:09:15.120 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays": 0, 2024-03-28T01:09:15.120 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_delayed": 0, 2024-03-28T01:09:15.120 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_enqueuing": 0, 2024-03-28T01:09:15.120 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_created": 0, 2024-03-28T01:09:15.120 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_enqueued": 0, 2024-03-28T01:09:15.120 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_reintegrated": 0, 2024-03-28T01:09:15.120 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_migrated": 0, 2024-03-28T01:09:15.120 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_processing": 0, 2024-03-28T01:09:15.120 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_enqueued": 0, 2024-03-28T01:09:15.121 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_prioritized": 0, 2024-03-28T01:09:15.121 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_started": 0, 2024-03-28T01:09:15.121 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_completed": 0, 2024-03-28T01:09:15.121 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T01:09:15.121 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_exportdir": 0, 2024-03-28T01:09:15.121 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_flush": 0, 2024-03-28T01:09:15.121 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragmentdir": 0, 2024-03-28T01:09:15.121 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragstats": 0, 2024-03-28T01:09:15.121 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_inodestats": 0 2024-03-28T01:09:15.121 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.121 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_log": { 2024-03-28T01:09:15.121 INFO:teuthology.orchestra.run.smithi174.stdout: "evlrg": 0, 2024-03-28T01:09:15.121 INFO:teuthology.orchestra.run.smithi174.stdout: "evadd": 0, 2024-03-28T01:09:15.121 INFO:teuthology.orchestra.run.smithi174.stdout: "evex": 0, 2024-03-28T01:09:15.122 INFO:teuthology.orchestra.run.smithi174.stdout: "evtrm": 0, 2024-03-28T01:09:15.122 INFO:teuthology.orchestra.run.smithi174.stdout: "ev": 411187, 2024-03-28T01:09:15.122 INFO:teuthology.orchestra.run.smithi174.stdout: "evexg": 0, 2024-03-28T01:09:15.122 INFO:teuthology.orchestra.run.smithi174.stdout: "evexd": 0, 2024-03-28T01:09:15.122 INFO:teuthology.orchestra.run.smithi174.stdout: "segadd": 0, 2024-03-28T01:09:15.122 INFO:teuthology.orchestra.run.smithi174.stdout: "segex": 0, 2024-03-28T01:09:15.122 INFO:teuthology.orchestra.run.smithi174.stdout: "segtrm": 0, 2024-03-28T01:09:15.122 INFO:teuthology.orchestra.run.smithi174.stdout: "seg": 10, 2024-03-28T01:09:15.122 INFO:teuthology.orchestra.run.smithi174.stdout: "segmjr": 63, 2024-03-28T01:09:15.122 INFO:teuthology.orchestra.run.smithi174.stdout: "segexg": 0, 2024-03-28T01:09:15.122 INFO:teuthology.orchestra.run.smithi174.stdout: "segexd": 0, 2024-03-28T01:09:15.122 INFO:teuthology.orchestra.run.smithi174.stdout: "expos": 817398663, 2024-03-28T01:09:15.123 INFO:teuthology.orchestra.run.smithi174.stdout: "wrpos": 835797604, 2024-03-28T01:09:15.123 INFO:teuthology.orchestra.run.smithi174.stdout: "rdpos": 835797005, 2024-03-28T01:09:15.123 INFO:teuthology.orchestra.run.smithi174.stdout: "jlat": { 2024-03-28T01:09:15.123 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.123 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.123 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.123 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.123 INFO:teuthology.orchestra.run.smithi174.stdout: "replayed": 411187 2024-03-28T01:09:15.123 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.123 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_mem": { 2024-03-28T01:09:15.123 INFO:teuthology.orchestra.run.smithi174.stdout: "ino": 17, 2024-03-28T01:09:15.123 INFO:teuthology.orchestra.run.smithi174.stdout: "ino+": 36, 2024-03-28T01:09:15.123 INFO:teuthology.orchestra.run.smithi174.stdout: "ino-": 19, 2024-03-28T01:09:15.123 INFO:teuthology.orchestra.run.smithi174.stdout: "dir": 10, 2024-03-28T01:09:15.124 INFO:teuthology.orchestra.run.smithi174.stdout: "dir+": 13, 2024-03-28T01:09:15.124 INFO:teuthology.orchestra.run.smithi174.stdout: "dir-": 3, 2024-03-28T01:09:15.124 INFO:teuthology.orchestra.run.smithi174.stdout: "dn": 52, 2024-03-28T01:09:15.124 INFO:teuthology.orchestra.run.smithi174.stdout: "dn+": 57, 2024-03-28T01:09:15.124 INFO:teuthology.orchestra.run.smithi174.stdout: "dn-": 5, 2024-03-28T01:09:15.124 INFO:teuthology.orchestra.run.smithi174.stdout: "cap": 0, 2024-03-28T01:09:15.124 INFO:teuthology.orchestra.run.smithi174.stdout: "cap+": 0, 2024-03-28T01:09:15.124 INFO:teuthology.orchestra.run.smithi174.stdout: "cap-": 0, 2024-03-28T01:09:15.124 INFO:teuthology.orchestra.run.smithi174.stdout: "rss": 51604, 2024-03-28T01:09:15.124 INFO:teuthology.orchestra.run.smithi174.stdout: "heap": 215420 2024-03-28T01:09:15.124 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.124 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_server": { 2024-03-28T01:09:15.124 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_client_request": 0, 2024-03-28T01:09:15.125 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_server_request": 0, 2024-03-28T01:09:15.125 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_request": 0, 2024-03-28T01:09:15.125 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_session": 0, 2024-03-28T01:09:15.125 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_peer_request": 0, 2024-03-28T01:09:15.125 INFO:teuthology.orchestra.run.smithi174.stdout: "req_create_latency": { 2024-03-28T01:09:15.125 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.125 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.125 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.125 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.125 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getattr_latency": { 2024-03-28T01:09:15.125 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.125 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.125 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.126 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.126 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getfilelock_latency": { 2024-03-28T01:09:15.126 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.126 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.126 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.126 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.126 INFO:teuthology.orchestra.run.smithi174.stdout: "req_link_latency": { 2024-03-28T01:09:15.126 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.126 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.126 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.126 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.126 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookup_latency": { 2024-03-28T01:09:15.126 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.126 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.126 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.127 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.127 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookuphash_latency": { 2024-03-28T01:09:15.127 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.127 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.127 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.127 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.127 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupino_latency": { 2024-03-28T01:09:15.127 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.127 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.127 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.127 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.127 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupname_latency": { 2024-03-28T01:09:15.127 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.127 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.127 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.128 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.128 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupparent_latency": { 2024-03-28T01:09:15.128 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.128 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.128 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.128 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.128 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupsnap_latency": { 2024-03-28T01:09:15.128 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.128 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.128 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.128 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.128 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lssnap_latency": { 2024-03-28T01:09:15.128 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.128 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.128 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.129 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.129 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mkdir_latency": { 2024-03-28T01:09:15.129 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.129 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.129 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.129 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.129 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mknod_latency": { 2024-03-28T01:09:15.129 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.129 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.129 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.129 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.129 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mksnap_latency": { 2024-03-28T01:09:15.129 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.130 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.130 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.130 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.130 INFO:teuthology.orchestra.run.smithi174.stdout: "req_open_latency": { 2024-03-28T01:09:15.130 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.130 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.130 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.130 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.130 INFO:teuthology.orchestra.run.smithi174.stdout: "req_readdir_latency": { 2024-03-28T01:09:15.130 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.130 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.130 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.130 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.130 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rename_latency": { 2024-03-28T01:09:15.130 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.131 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.131 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.131 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.131 INFO:teuthology.orchestra.run.smithi174.stdout: "req_renamesnap_latency": { 2024-03-28T01:09:15.131 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.131 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.131 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.131 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.131 INFO:teuthology.orchestra.run.smithi174.stdout: "req_snapdiff_latency": { 2024-03-28T01:09:15.131 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.131 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.131 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.131 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.131 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmdir_latency": { 2024-03-28T01:09:15.131 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.132 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.132 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.132 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.132 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmsnap_latency": { 2024-03-28T01:09:15.132 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.132 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.132 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.132 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.132 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmxattr_latency": { 2024-03-28T01:09:15.132 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.132 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.132 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.132 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.132 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setattr_latency": { 2024-03-28T01:09:15.133 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.133 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.133 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.133 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setdirlayout_latency": { 2024-03-28T01:09:15.133 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.133 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.133 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.133 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setfilelock_latency": { 2024-03-28T01:09:15.133 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.133 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.133 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.133 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setlayout_latency": { 2024-03-28T01:09:15.134 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.134 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.134 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.134 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setxattr_latency": { 2024-03-28T01:09:15.134 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.134 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.134 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.134 INFO:teuthology.orchestra.run.smithi174.stdout: "req_symlink_latency": { 2024-03-28T01:09:15.134 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.134 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.134 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "req_unlink_latency": { 2024-03-28T01:09:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_revoke_eviction": 0, 2024-03-28T01:09:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_acquisition_throttle": 0, 2024-03-28T01:09:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getvxattr_latency": { 2024-03-28T01:09:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:15.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_sessions": { 2024-03-28T01:09:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "session_count": 1, 2024-03-28T01:09:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "session_add": 4, 2024-03-28T01:09:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "session_remove": 3, 2024-03-28T01:09:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_open": 2, 2024-03-28T01:09:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_stale": 0, 2024-03-28T01:09:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "total_load": 0, 2024-03-28T01:09:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "average_load": 0, 2024-03-28T01:09:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "avg_session_uptime": 3501, 2024-03-28T01:09:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "mdthresh_evicted": 0 2024-03-28T01:09:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-03-28T01:09:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-03-28T01:09:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-03-28T01:09:15.136 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T01:09:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-03-28T01:09:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T01:09:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-03-28T01:09:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T01:09:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T01:09:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T01:09:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T01:09:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T01:09:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-03-28T01:09:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T01:09:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T01:09:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-03-28T01:09:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-03-28T01:09:15.137 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-03-28T01:09:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-03-28T01:09:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T01:09:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T01:09:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T01:09:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T01:09:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T01:09:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-03-28T01:09:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-03-28T01:09:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-03-28T01:09:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T01:09:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T01:09:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-03-28T01:09:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-03-28T01:09:15.138 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-03-28T01:09:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-03-28T01:09:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T01:09:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-03-28T01:09:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T01:09:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-03-28T01:09:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 238835, 2024-03-28T01:09:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 81, 2024-03-28T01:09:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 0, 2024-03-28T01:09:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 0, 2024-03-28T01:09:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-03-28T01:09:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-03-28T01:09:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-03-28T01:09:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-03-28T01:09:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-03-28T01:09:15.139 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-03-28T01:09:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 8896, 2024-03-28T01:09:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 171, 2024-03-28T01:09:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T01:09:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-03-28T01:09:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-03-28T01:09:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-03-28T01:09:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 89631, 2024-03-28T01:09:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 866, 2024-03-28T01:09:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-03-28T01:09:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-03-28T01:09:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-03-28T01:09:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-03-28T01:09:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: "objecter": { 2024-03-28T01:09:15.140 INFO:teuthology.orchestra.run.smithi174.stdout: "op_active": 0, 2024-03-28T01:09:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "op_laggy": 0, 2024-03-28T01:09:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send": 3823, 2024-03-28T01:09:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send_bytes": 26, 2024-03-28T01:09:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "op_resend": 4, 2024-03-28T01:09:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "op_reply": 3823, 2024-03-28T01:09:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "op_latency": { 2024-03-28T01:09:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3823, 2024-03-28T01:09:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 61.914198523, 2024-03-28T01:09:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.016195186 2024-03-28T01:09:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "op_inflight": 0, 2024-03-28T01:09:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "oplen_avg": { 2024-03-28T01:09:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3823, 2024-03-28T01:09:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 3824 2024-03-28T01:09:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.141 INFO:teuthology.orchestra.run.smithi174.stdout: "op": 3823, 2024-03-28T01:09:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "op_r": 3823, 2024-03-28T01:09:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "op_w": 0, 2024-03-28T01:09:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "op_rmw": 0, 2024-03-28T01:09:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "op_pg": 0, 2024-03-28T01:09:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_stat": 2440, 2024-03-28T01:09:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_create": 0, 2024-03-28T01:09:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_read": 1380, 2024-03-28T01:09:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_write": 0, 2024-03-28T01:09:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writefull": 0, 2024-03-28T01:09:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writesame": 0, 2024-03-28T01:09:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_append": 0, 2024-03-28T01:09:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_zero": 0, 2024-03-28T01:09:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_truncate": 0, 2024-03-28T01:09:15.142 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_delete": 0, 2024-03-28T01:09:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_mapext": 0, 2024-03-28T01:09:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_sparse_read": 0, 2024-03-28T01:09:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_clonerange": 0, 2024-03-28T01:09:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_getxattr": 2, 2024-03-28T01:09:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_setxattr": 0, 2024-03-28T01:09:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_cmpxattr": 0, 2024-03-28T01:09:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_rmxattr": 0, 2024-03-28T01:09:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_resetxattrs": 0, 2024-03-28T01:09:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_call": 0, 2024-03-28T01:09:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_watch": 0, 2024-03-28T01:09:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_notify": 0, 2024-03-28T01:09:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T01:09:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls": 0, 2024-03-28T01:09:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls_filter": 0, 2024-03-28T01:09:15.143 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_other": 0, 2024-03-28T01:09:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_active": 0, 2024-03-28T01:09:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_send": 0, 2024-03-28T01:09:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_resend": 0, 2024-03-28T01:09:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_ping": 0, 2024-03-28T01:09:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_active": 0, 2024-03-28T01:09:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_send": 0, 2024-03-28T01:09:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_resend": 0, 2024-03-28T01:09:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_active": 0, 2024-03-28T01:09:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_send": 0, 2024-03-28T01:09:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_resend": 0, 2024-03-28T01:09:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_active": 0, 2024-03-28T01:09:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_send": 0, 2024-03-28T01:09:15.144 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_resend": 0, 2024-03-28T01:09:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "command_active": 0, 2024-03-28T01:09:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "command_send": 0, 2024-03-28T01:09:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "command_resend": 0, 2024-03-28T01:09:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "map_epoch": 94, 2024-03-28T01:09:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "map_full": 0, 2024-03-28T01:09:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "map_inc": 4, 2024-03-28T01:09:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_sessions": 11, 2024-03-28T01:09:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_open": 11, 2024-03-28T01:09:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_close": 0, 2024-03-28T01:09:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_laggy": 0, 2024-03-28T01:09:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_wr": 0, 2024-03-28T01:09:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_rd": 2, 2024-03-28T01:09:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_del": 0 2024-03-28T01:09:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "oft": { 2024-03-28T01:09:15.145 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_objs": 0, 2024-03-28T01:09:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_kv_pairs": 0, 2024-03-28T01:09:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_updates": 0, 2024-03-28T01:09:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_removes": 0 2024-03-28T01:09:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: "osd-slow-ops": { 2024-03-28T01:09:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_ops_count": 0 2024-03-28T01:09:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: "purge_queue": { 2024-03-28T01:09:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops": 0, 2024-03-28T01:09:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T01:09:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing": 0, 2024-03-28T01:09:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_high_water": 0, 2024-03-28T01:09:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed_ops": 0, 2024-03-28T01:09:15.146 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed": 0, 2024-03-28T01:09:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_item_in_journal": 0 2024-03-28T01:09:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T01:09:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-28T01:09:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-03-28T01:09:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-28T01:09:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 4080, 2024-03-28T01:09:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 833224417, 2024-03-28T01:09:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 4080, 2024-03-28T01:09:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-03-28T01:09:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-03-28T01:09:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 4080, 2024-03-28T01:09:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 833224417, 2024-03-28T01:09:15.147 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-28T01:09:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_bytes": { 2024-03-28T01:09:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-28T01:09:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-03-28T01:09:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-28T01:09:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-03-28T01:09:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-03-28T01:09:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-03-28T01:09:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 3823, 2024-03-28T01:09:15.148 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 831786278, 2024-03-28T01:09:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 3823, 2024-03-28T01:09:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 831786278, 2024-03-28T01:09:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-28T01:09:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:15.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_ops": { 2024-03-28T01:09:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-28T01:09:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 1024, 2024-03-28T01:09:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-28T01:09:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-03-28T01:09:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-03-28T01:09:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-03-28T01:09:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 3823, 2024-03-28T01:09:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 3823, 2024-03-28T01:09:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 3823, 2024-03-28T01:09:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 3823, 2024-03-28T01:09:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-28T01:09:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle": { 2024-03-28T01:09:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-28T01:09:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-03-28T01:09:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-28T01:09:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-03-28T01:09:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-03-28T01:09:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-03-28T01:09:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-03-28T01:09:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-03-28T01:09:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-03-28T01:09:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-03-28T01:09:15.151 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-28T01:09:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle-0x563abafb86e0": { 2024-03-28T01:09:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-28T01:09:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-03-28T01:09:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-28T01:09:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-03-28T01:09:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-03-28T01:09:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-03-28T01:09:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-03-28T01:09:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-03-28T01:09:15.152 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-03-28T01:09:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-03-28T01:09:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-28T01:09:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:15.153 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:15.153 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-28T01:09:15.861 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-28T01:09:15.861 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-28T01:09:15.861 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-03-28T01:09:15.861 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=0 2024-03-28T01:09:15.861 DEBUG:tasks.check_counter:Getting stats from k 2024-03-28T01:09:15.862 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-28T01:09:16.104 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:15 smithi169 ceph-mon[32274]: pgmap v733: 129 pgs: 129 active+clean; 4.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 21 MiB/s wr, 111 op/s 2024-03-28T01:09:16.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:15 smithi174 ceph-mon[36060]: pgmap v733: 129 pgs: 129 active+clean; 4.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 21 MiB/s wr, 111 op/s 2024-03-28T01:09:16.110 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T01:09:16.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:15 smithi177 ceph-mon[36206]: pgmap v733: 129 pgs: 129 active+clean; 4.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 21 MiB/s wr, 111 op/s 2024-03-28T01:09:18.002 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:17 smithi174 ceph-mon[36060]: pgmap v734: 129 pgs: 129 active+clean; 4.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 40 MiB/s wr, 224 op/s 2024-03-28T01:09:18.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:17 smithi177 ceph-mon[36206]: pgmap v734: 129 pgs: 129 active+clean; 4.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 40 MiB/s wr, 224 op/s 2024-03-28T01:09:18.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:17 smithi169 ceph-mon[32274]: pgmap v734: 129 pgs: 129 active+clean; 4.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 40 MiB/s wr, 224 op/s 2024-03-28T01:09:18.506 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-28T01:09:18.506 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T01:09:18.506 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 4, 2024-03-28T01:09:18.506 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 4, 2024-03-28T01:09:18.507 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 3072, 2024-03-28T01:09:18.507 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 448, 2024-03-28T01:09:18.507 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 4, 2024-03-28T01:09:18.507 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 0, 2024-03-28T01:09:18.507 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.001605790, 2024-03-28T01:09:18.507 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.000035790, 2024-03-28T01:09:18.507 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.000675461, 2024-03-28T01:09:18.507 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T01:09:18.507 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:18.508 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3, 2024-03-28T01:09:18.508 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000822757, 2024-03-28T01:09:18.508 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000274252 2024-03-28T01:09:18.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:18.508 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:18.508 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:18.508 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:18.508 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:18.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:18.508 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-28T01:09:18.509 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T01:09:18.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:18.509 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T01:09:18.509 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 251, 2024-03-28T01:09:18.509 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 246, 2024-03-28T01:09:18.509 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 760336, 2024-03-28T01:09:18.509 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 147744, 2024-03-28T01:09:18.509 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 2, 2024-03-28T01:09:18.509 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 1, 2024-03-28T01:09:18.510 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.037494833, 2024-03-28T01:09:18.510 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.012168348, 2024-03-28T01:09:18.510 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.021677134, 2024-03-28T01:09:18.510 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.006258598, 2024-03-28T01:09:18.510 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:18.510 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 245, 2024-03-28T01:09:18.510 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.013370495, 2024-03-28T01:09:18.510 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000054573 2024-03-28T01:09:18.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:18.511 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:18.511 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:18.511 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:18.511 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:18.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:18.511 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 760336, 2024-03-28T01:09:18.511 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 147744 2024-03-28T01:09:18.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:18.511 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T01:09:18.511 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 2, 2024-03-28T01:09:18.511 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 188, 2024-03-28T01:09:18.512 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 320, 2024-03-28T01:09:18.512 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 188576, 2024-03-28T01:09:18.512 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 4, 2024-03-28T01:09:18.512 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 2, 2024-03-28T01:09:18.512 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.009550028, 2024-03-28T01:09:18.512 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.006508398, 2024-03-28T01:09:18.512 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.001261795, 2024-03-28T01:09:18.512 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T01:09:18.512 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:18.513 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 186, 2024-03-28T01:09:18.513 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.008647797, 2024-03-28T01:09:18.513 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000046493 2024-03-28T01:09:18.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:18.513 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:18.513 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:18.514 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:18.514 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:18.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:18.514 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T01:09:18.514 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 188576 2024-03-28T01:09:18.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:18.514 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-03-28T01:09:18.514 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 0, 2024-03-28T01:09:18.514 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-03-28T01:09:18.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:18.514 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-03-28T01:09:18.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-03-28T01:09:18.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-03-28T01:09:18.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T01:09:18.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-03-28T01:09:18.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T01:09:18.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-03-28T01:09:18.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T01:09:18.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T01:09:18.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T01:09:18.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T01:09:18.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T01:09:18.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-03-28T01:09:18.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T01:09:18.515 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T01:09:18.516 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-03-28T01:09:18.516 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-03-28T01:09:18.516 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-03-28T01:09:18.516 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-03-28T01:09:18.516 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T01:09:18.516 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T01:09:18.516 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T01:09:18.516 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T01:09:18.516 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T01:09:18.516 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-03-28T01:09:18.516 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-03-28T01:09:18.516 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-03-28T01:09:18.516 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T01:09:18.516 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T01:09:18.517 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-03-28T01:09:18.517 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-03-28T01:09:18.517 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-03-28T01:09:18.517 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-03-28T01:09:18.517 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T01:09:18.517 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-03-28T01:09:18.517 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T01:09:18.517 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-03-28T01:09:18.517 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 147960, 2024-03-28T01:09:18.517 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 28, 2024-03-28T01:09:18.517 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 0, 2024-03-28T01:09:18.517 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 0, 2024-03-28T01:09:18.517 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-03-28T01:09:18.517 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-03-28T01:09:18.517 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-03-28T01:09:18.518 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-03-28T01:09:18.518 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-03-28T01:09:18.518 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-03-28T01:09:18.518 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 0, 2024-03-28T01:09:18.518 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 0, 2024-03-28T01:09:18.518 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T01:09:18.518 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-03-28T01:09:18.518 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-03-28T01:09:18.518 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-03-28T01:09:18.518 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 928, 2024-03-28T01:09:18.518 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 2, 2024-03-28T01:09:18.518 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-03-28T01:09:18.518 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-03-28T01:09:18.518 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-03-28T01:09:18.519 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-03-28T01:09:18.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-03-28T01:09:18.519 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T01:09:18.519 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-03-28T01:09:18.519 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-03-28T01:09:18.519 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-03-28T01:09:18.519 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 253, 2024-03-28T01:09:18.519 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 724919, 2024-03-28T01:09:18.519 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:18.519 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 253, 2024-03-28T01:09:18.519 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-03-28T01:09:18.519 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-03-28T01:09:18.519 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 253, 2024-03-28T01:09:18.519 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 724919, 2024-03-28T01:09:18.519 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-03-28T01:09:18.520 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-03-28T01:09:18.520 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-03-28T01:09:18.520 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-03-28T01:09:18.520 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:18.520 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-03-28T01:09:18.520 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-28T01:09:19.219 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-03-28T01:09:19.219 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-03-28T01:09:19.219 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-03-28T01:09:19.219 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-03-28T01:09:19.219 DEBUG:tasks.check_counter:Getting stats from c 2024-03-28T01:09:19.219 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-28T01:09:19.599 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T01:09:20.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:19 smithi174 ceph-mon[36060]: pgmap v735: 129 pgs: 129 active+clean; 4.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 45 MiB/s wr, 259 op/s 2024-03-28T01:09:20.112 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:19 smithi177 ceph-mon[36206]: pgmap v735: 129 pgs: 129 active+clean; 4.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 45 MiB/s wr, 259 op/s 2024-03-28T01:09:20.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:19 smithi169 ceph-mon[32274]: pgmap v735: 129 pgs: 129 active+clean; 4.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 45 MiB/s wr, 259 op/s 2024-03-28T01:09:22.101 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:21 smithi177 ceph-mon[36206]: pgmap v736: 129 pgs: 129 active+clean; 4.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 46 MiB/s wr, 276 op/s 2024-03-28T01:09:22.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:21 smithi174 ceph-mon[36060]: pgmap v736: 129 pgs: 129 active+clean; 4.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 46 MiB/s wr, 276 op/s 2024-03-28T01:09:22.274 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-03-28T01:09:22.274 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T01:09:22.274 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 1773, 2024-03-28T01:09:22.274 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 1044, 2024-03-28T01:09:22.274 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 429928, 2024-03-28T01:09:22.274 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 313701, 2024-03-28T01:09:22.275 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 8, 2024-03-28T01:09:22.275 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 3, 2024-03-28T01:09:22.275 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.101477021, 2024-03-28T01:09:22.275 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.033899577, 2024-03-28T01:09:22.275 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.085299327, 2024-03-28T01:09:22.275 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.004758357, 2024-03-28T01:09:22.275 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:22.275 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 1042, 2024-03-28T01:09:22.275 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.102226116, 2024-03-28T01:09:22.275 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000098105 2024-03-28T01:09:22.275 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.275 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:22.275 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2579, 2024-03-28T01:09:22.275 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000610634, 2024-03-28T01:09:22.276 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000236 2024-03-28T01:09:22.276 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.276 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T01:09:22.276 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T01:09:22.276 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.276 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T01:09:22.276 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 54, 2024-03-28T01:09:22.276 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 236, 2024-03-28T01:09:22.276 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 12095, 2024-03-28T01:09:22.276 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 868987, 2024-03-28T01:09:22.276 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 41, 2024-03-28T01:09:22.276 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 4, 2024-03-28T01:09:22.276 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.031881519, 2024-03-28T01:09:22.277 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.009098757, 2024-03-28T01:09:22.277 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.018373615, 2024-03-28T01:09:22.277 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000289608, 2024-03-28T01:09:22.277 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:22.277 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 235, 2024-03-28T01:09:22.277 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.027108065, 2024-03-28T01:09:22.277 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000115353 2024-03-28T01:09:22.277 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.277 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:22.277 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 34, 2024-03-28T01:09:22.277 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000006443, 2024-03-28T01:09:22.277 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000189 2024-03-28T01:09:22.277 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.277 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-03-28T01:09:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 485280 2024-03-28T01:09:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T01:09:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 310, 2024-03-28T01:09:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 302, 2024-03-28T01:09:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 792796, 2024-03-28T01:09:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 244020, 2024-03-28T01:09:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 14, 2024-03-28T01:09:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 3, 2024-03-28T01:09:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.049693080, 2024-03-28T01:09:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.015943725, 2024-03-28T01:09:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.029691505, 2024-03-28T01:09:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.007228786, 2024-03-28T01:09:22.278 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 301, 2024-03-28T01:09:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.027090231, 2024-03-28T01:09:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000090000 2024-03-28T01:09:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 28, 2024-03-28T01:09:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000005292, 2024-03-28T01:09:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000189 2024-03-28T01:09:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 788752, 2024-03-28T01:09:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 152192 2024-03-28T01:09:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-03-28T01:09:22.279 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 1, 2024-03-28T01:09:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-03-28T01:09:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-MDSRank": { 2024-03-28T01:09:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-03-28T01:09:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-03-28T01:09:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 38, 2024-03-28T01:09:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.001969180, 2024-03-28T01:09:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000051820 2024-03-28T01:09:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-PurgeQueue": { 2024-03-28T01:09:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-03-28T01:09:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-03-28T01:09:22.280 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2, 2024-03-28T01:09:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000005993, 2024-03-28T01:09:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000002996 2024-03-28T01:09:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: "mds": { 2024-03-28T01:09:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: "request": 1, 2024-03-28T01:09:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: "reply": 0, 2024-03-28T01:09:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: "reply_latency": { 2024-03-28T01:09:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_reply": 0, 2024-03-28T01:09:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: "forward": 1, 2024-03-28T01:09:22.281 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_complete": 0, 2024-03-28T01:09:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_keys": 0, 2024-03-28T01:09:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_commit": 11, 2024-03-28T01:09:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_split": 0, 2024-03-28T01:09:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_merge": 0, 2024-03-28T01:09:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes": 16, 2024-03-28T01:09:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_top": 0, 2024-03-28T01:09:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_bottom": 16, 2024-03-28T01:09:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pin_tail": 0, 2024-03-28T01:09:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pinned": 13, 2024-03-28T01:09:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_expired": 8, 2024-03-28T01:09:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_with_caps": 0, 2024-03-28T01:09:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: "caps": 0, 2024-03-28T01:09:22.282 INFO:teuthology.orchestra.run.smithi177.stdout: "subtrees": 3, 2024-03-28T01:09:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse": 9, 2024-03-28T01:09:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_hit": 4, 2024-03-28T01:09:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_forward": 1, 2024-03-28T01:09:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_discover": 4, 2024-03-28T01:09:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_dir_fetch": 0, 2024-03-28T01:09:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_remote_ino": 0, 2024-03-28T01:09:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_lock": 0, 2024-03-28T01:09:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "load_cent": 0, 2024-03-28T01:09:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "q": 0, 2024-03-28T01:09:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "exported": 0, 2024-03-28T01:09:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "exported_inodes": 0, 2024-03-28T01:09:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "imported": 0, 2024-03-28T01:09:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "imported_inodes": 0, 2024-03-28T01:09:22.283 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_dir_fetch": 0, 2024-03-28T01:09:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_backtrace_fetch": 0, 2024-03-28T01:09:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_peer_discover": 0, 2024-03-28T01:09:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rfiles": 22, 2024-03-28T01:09:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rbytes": 10738150859, 2024-03-28T01:09:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rsnaps": 0, 2024-03-28T01:09:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T01:09:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_set_tag": 0, 2024-03-28T01:09:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T01:09:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_inotable_repaired": 0, 2024-03-28T01:09:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_inodes": 0, 2024-03-28T01:09:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T01:09:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T01:09:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_file_inodes": 0, 2024-03-28T01:09:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_inode_file_caps": 0, 2024-03-28T01:09:22.284 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T01:09:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_grant": 0, 2024-03-28T01:09:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T01:09:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T01:09:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T01:09:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps": 0, 2024-03-28T01:09:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps_dirty": 0, 2024-03-28T01:09:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_cap_release": 0, 2024-03-28T01:09:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: "process_request_cap_release": 0 2024-03-28T01:09:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_cache": { 2024-03-28T01:09:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update": 0, 2024-03-28T01:09:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update_receipt": 12, 2024-03-28T01:09:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_try_discover": 8, 2024-03-28T01:09:22.285 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_send_discover": 9, 2024-03-28T01:09:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_handle_discover": 0, 2024-03-28T01:09:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays": 0, 2024-03-28T01:09:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_delayed": 0, 2024-03-28T01:09:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_enqueuing": 0, 2024-03-28T01:09:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_created": 0, 2024-03-28T01:09:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_enqueued": 0, 2024-03-28T01:09:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_reintegrated": 0, 2024-03-28T01:09:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_migrated": 0, 2024-03-28T01:09:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_processing": 0, 2024-03-28T01:09:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_enqueued": 0, 2024-03-28T01:09:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_prioritized": 0, 2024-03-28T01:09:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_started": 0, 2024-03-28T01:09:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_completed": 0, 2024-03-28T01:09:22.286 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T01:09:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_exportdir": 0, 2024-03-28T01:09:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_flush": 0, 2024-03-28T01:09:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragmentdir": 0, 2024-03-28T01:09:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragstats": 0, 2024-03-28T01:09:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_inodestats": 0 2024-03-28T01:09:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_log": { 2024-03-28T01:09:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: "evlrg": 0, 2024-03-28T01:09:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: "evadd": 3, 2024-03-28T01:09:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: "evex": 0, 2024-03-28T01:09:22.287 INFO:teuthology.orchestra.run.smithi177.stdout: "evtrm": 0, 2024-03-28T01:09:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: "ev": 3, 2024-03-28T01:09:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: "evexg": 0, 2024-03-28T01:09:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: "evexd": 0, 2024-03-28T01:09:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: "segadd": 2, 2024-03-28T01:09:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: "segex": 0, 2024-03-28T01:09:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: "segtrm": 0, 2024-03-28T01:09:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: "seg": 2, 2024-03-28T01:09:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: "segmjr": 2, 2024-03-28T01:09:22.289 INFO:teuthology.orchestra.run.smithi177.stdout: "segexg": 0, 2024-03-28T01:09:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: "segexd": 0, 2024-03-28T01:09:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: "expos": 4194304, 2024-03-28T01:09:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: "wrpos": 4195200, 2024-03-28T01:09:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: "rdpos": 0, 2024-03-28T01:09:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: "jlat": { 2024-03-28T01:09:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2, 2024-03-28T01:09:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.002656857, 2024-03-28T01:09:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.001328428 2024-03-28T01:09:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: "replayed": 0 2024-03-28T01:09:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_mem": { 2024-03-28T01:09:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: "ino": 20, 2024-03-28T01:09:22.290 INFO:teuthology.orchestra.run.smithi177.stdout: "ino+": 24, 2024-03-28T01:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "ino-": 4, 2024-03-28T01:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "dir": 19, 2024-03-28T01:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "dir+": 23, 2024-03-28T01:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "dir-": 4, 2024-03-28T01:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "dn": 16, 2024-03-28T01:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "dn+": 24, 2024-03-28T01:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "dn-": 8, 2024-03-28T01:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "cap": 0, 2024-03-28T01:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "cap+": 0, 2024-03-28T01:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "cap-": 0, 2024-03-28T01:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "rss": 49656, 2024-03-28T01:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "heap": 223612 2024-03-28T01:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_server": { 2024-03-28T01:09:22.291 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_client_request": 1, 2024-03-28T01:09:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_server_request": 0, 2024-03-28T01:09:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_request": 1, 2024-03-28T01:09:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_session": 9, 2024-03-28T01:09:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_peer_request": 0, 2024-03-28T01:09:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: "req_create_latency": { 2024-03-28T01:09:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getattr_latency": { 2024-03-28T01:09:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.292 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getfilelock_latency": { 2024-03-28T01:09:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "req_link_latency": { 2024-03-28T01:09:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookup_latency": { 2024-03-28T01:09:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookuphash_latency": { 2024-03-28T01:09:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupino_latency": { 2024-03-28T01:09:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupname_latency": { 2024-03-28T01:09:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupparent_latency": { 2024-03-28T01:09:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupsnap_latency": { 2024-03-28T01:09:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lssnap_latency": { 2024-03-28T01:09:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.295 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.296 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mkdir_latency": { 2024-03-28T01:09:22.296 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.296 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.296 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mknod_latency": { 2024-03-28T01:09:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mksnap_latency": { 2024-03-28T01:09:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: "req_open_latency": { 2024-03-28T01:09:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.297 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "req_readdir_latency": { 2024-03-28T01:09:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rename_latency": { 2024-03-28T01:09:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "req_renamesnap_latency": { 2024-03-28T01:09:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.298 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: "req_snapdiff_latency": { 2024-03-28T01:09:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmdir_latency": { 2024-03-28T01:09:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmsnap_latency": { 2024-03-28T01:09:22.299 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmxattr_latency": { 2024-03-28T01:09:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setattr_latency": { 2024-03-28T01:09:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setdirlayout_latency": { 2024-03-28T01:09:22.300 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setfilelock_latency": { 2024-03-28T01:09:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setlayout_latency": { 2024-03-28T01:09:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setxattr_latency": { 2024-03-28T01:09:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.301 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: "req_symlink_latency": { 2024-03-28T01:09:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: "req_unlink_latency": { 2024-03-28T01:09:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_revoke_eviction": 0, 2024-03-28T01:09:22.302 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_acquisition_throttle": 0, 2024-03-28T01:09:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getvxattr_latency": { 2024-03-28T01:09:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_sessions": { 2024-03-28T01:09:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "session_count": 1, 2024-03-28T01:09:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "session_add": 1, 2024-03-28T01:09:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "session_remove": 0, 2024-03-28T01:09:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_open": 1, 2024-03-28T01:09:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_stale": 0, 2024-03-28T01:09:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "total_load": 0, 2024-03-28T01:09:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "average_load": 0, 2024-03-28T01:09:22.303 INFO:teuthology.orchestra.run.smithi177.stdout: "avg_session_uptime": 666, 2024-03-28T01:09:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "mdthresh_evicted": 0 2024-03-28T01:09:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-03-28T01:09:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-03-28T01:09:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-03-28T01:09:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T01:09:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-03-28T01:09:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T01:09:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-03-28T01:09:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T01:09:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T01:09:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T01:09:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T01:09:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T01:09:22.304 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-03-28T01:09:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T01:09:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T01:09:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-03-28T01:09:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-03-28T01:09:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-03-28T01:09:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-03-28T01:09:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T01:09:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T01:09:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T01:09:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T01:09:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T01:09:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-03-28T01:09:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-03-28T01:09:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-03-28T01:09:22.305 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T01:09:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T01:09:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-03-28T01:09:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-03-28T01:09:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-03-28T01:09:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-03-28T01:09:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T01:09:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-03-28T01:09:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T01:09:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-03-28T01:09:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 229974, 2024-03-28T01:09:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 70, 2024-03-28T01:09:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 0, 2024-03-28T01:09:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 0, 2024-03-28T01:09:22.306 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-03-28T01:09:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-03-28T01:09:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-03-28T01:09:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-03-28T01:09:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-03-28T01:09:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-03-28T01:09:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 8896, 2024-03-28T01:09:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 171, 2024-03-28T01:09:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T01:09:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-03-28T01:09:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-03-28T01:09:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-03-28T01:09:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 79629, 2024-03-28T01:09:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 320, 2024-03-28T01:09:22.307 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-03-28T01:09:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-03-28T01:09:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-03-28T01:09:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-03-28T01:09:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: "objecter": { 2024-03-28T01:09:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: "op_active": 0, 2024-03-28T01:09:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: "op_laggy": 0, 2024-03-28T01:09:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send": 37, 2024-03-28T01:09:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send_bytes": 11108, 2024-03-28T01:09:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: "op_resend": 29, 2024-03-28T01:09:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: "op_reply": 37, 2024-03-28T01:09:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: "op_latency": { 2024-03-28T01:09:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 37, 2024-03-28T01:09:22.308 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.209993974, 2024-03-28T01:09:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.005675512 2024-03-28T01:09:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "op_inflight": 0, 2024-03-28T01:09:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "oplen_avg": { 2024-03-28T01:09:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 37, 2024-03-28T01:09:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 59 2024-03-28T01:09:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "op": 37, 2024-03-28T01:09:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "op_r": 0, 2024-03-28T01:09:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "op_w": 37, 2024-03-28T01:09:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "op_rmw": 0, 2024-03-28T01:09:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "op_pg": 0, 2024-03-28T01:09:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_stat": 0, 2024-03-28T01:09:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_create": 10, 2024-03-28T01:09:22.309 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_read": 0, 2024-03-28T01:09:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_write": 2, 2024-03-28T01:09:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writefull": 6, 2024-03-28T01:09:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writesame": 0, 2024-03-28T01:09:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_append": 0, 2024-03-28T01:09:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_zero": 0, 2024-03-28T01:09:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_truncate": 0, 2024-03-28T01:09:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_delete": 6, 2024-03-28T01:09:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_mapext": 0, 2024-03-28T01:09:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_sparse_read": 0, 2024-03-28T01:09:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_clonerange": 0, 2024-03-28T01:09:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_getxattr": 0, 2024-03-28T01:09:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_setxattr": 20, 2024-03-28T01:09:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_cmpxattr": 0, 2024-03-28T01:09:22.310 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_rmxattr": 0, 2024-03-28T01:09:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_resetxattrs": 0, 2024-03-28T01:09:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_call": 0, 2024-03-28T01:09:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_watch": 0, 2024-03-28T01:09:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_notify": 0, 2024-03-28T01:09:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T01:09:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls": 0, 2024-03-28T01:09:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls_filter": 0, 2024-03-28T01:09:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_other": 0, 2024-03-28T01:09:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_active": 0, 2024-03-28T01:09:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_send": 0, 2024-03-28T01:09:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_resend": 0, 2024-03-28T01:09:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_ping": 0, 2024-03-28T01:09:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_active": 0, 2024-03-28T01:09:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_send": 0, 2024-03-28T01:09:22.311 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_resend": 0, 2024-03-28T01:09:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_active": 0, 2024-03-28T01:09:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_send": 0, 2024-03-28T01:09:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_resend": 0, 2024-03-28T01:09:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_active": 0, 2024-03-28T01:09:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_send": 0, 2024-03-28T01:09:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_resend": 0, 2024-03-28T01:09:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: "command_active": 0, 2024-03-28T01:09:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: "command_send": 0, 2024-03-28T01:09:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: "command_resend": 0, 2024-03-28T01:09:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: "map_epoch": 94, 2024-03-28T01:09:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: "map_full": 0, 2024-03-28T01:09:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: "map_inc": 4, 2024-03-28T01:09:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_sessions": 8, 2024-03-28T01:09:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_open": 8, 2024-03-28T01:09:22.312 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_close": 0, 2024-03-28T01:09:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_laggy": 0, 2024-03-28T01:09:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_wr": 14, 2024-03-28T01:09:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_rd": 0, 2024-03-28T01:09:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_del": 1 2024-03-28T01:09:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "oft": { 2024-03-28T01:09:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_objs": 1, 2024-03-28T01:09:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_kv_pairs": 0, 2024-03-28T01:09:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_updates": 0, 2024-03-28T01:09:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_removes": 0 2024-03-28T01:09:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "osd-slow-ops": { 2024-03-28T01:09:22.313 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_ops_count": 0 2024-03-28T01:09:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "purge_queue": { 2024-03-28T01:09:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops": 0, 2024-03-28T01:09:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T01:09:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing": 0, 2024-03-28T01:09:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_high_water": 0, 2024-03-28T01:09:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed_ops": 0, 2024-03-28T01:09:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed": 0, 2024-03-28T01:09:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_item_in_journal": 0 2024-03-28T01:09:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T01:09:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-28T01:09:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-03-28T01:09:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-28T01:09:22.314 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 2133, 2024-03-28T01:09:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 1025158, 2024-03-28T01:09:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 2133, 2024-03-28T01:09:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-03-28T01:09:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-03-28T01:09:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 2133, 2024-03-28T01:09:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 1025158, 2024-03-28T01:09:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-28T01:09:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_bytes": { 2024-03-28T01:09:22.315 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-28T01:09:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-03-28T01:09:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-28T01:09:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-03-28T01:09:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-03-28T01:09:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-03-28T01:09:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 37, 2024-03-28T01:09:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 11108, 2024-03-28T01:09:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 37, 2024-03-28T01:09:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 11108, 2024-03-28T01:09:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-28T01:09:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.316 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_ops": { 2024-03-28T01:09:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-28T01:09:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 1024, 2024-03-28T01:09:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-28T01:09:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-03-28T01:09:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-03-28T01:09:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-03-28T01:09:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 37, 2024-03-28T01:09:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 37, 2024-03-28T01:09:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 37, 2024-03-28T01:09:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 37, 2024-03-28T01:09:22.317 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-28T01:09:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle": { 2024-03-28T01:09:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-28T01:09:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-03-28T01:09:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-28T01:09:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-03-28T01:09:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-03-28T01:09:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-03-28T01:09:22.318 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-03-28T01:09:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-03-28T01:09:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-03-28T01:09:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-03-28T01:09:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-28T01:09:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle-0x55eebac0a500": { 2024-03-28T01:09:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-28T01:09:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-03-28T01:09:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-28T01:09:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 3, 2024-03-28T01:09:22.319 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 896, 2024-03-28T01:09:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 3, 2024-03-28T01:09:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-03-28T01:09:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-03-28T01:09:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 2, 2024-03-28T01:09:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 896, 2024-03-28T01:09:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-28T01:09:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:22.320 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:22.320 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-03-28T01:09:22.321 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:21 smithi169 ceph-mon[32274]: pgmap v736: 129 pgs: 129 active+clean; 4.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 46 MiB/s wr, 276 op/s 2024-03-28T01:09:24.118 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:23 smithi174 ceph-mon[36060]: pgmap v737: 129 pgs: 129 active+clean; 4.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 60 MiB/s wr, 368 op/s 2024-03-28T01:09:24.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:23 smithi177 ceph-mon[36206]: pgmap v737: 129 pgs: 129 active+clean; 4.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 60 MiB/s wr, 368 op/s 2024-03-28T01:09:24.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:23 smithi169 ceph-mon[32274]: pgmap v737: 129 pgs: 129 active+clean; 4.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 60 MiB/s wr, 368 op/s 2024-03-28T01:09:24.371 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-28T01:09:24.371 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-28T01:09:24.371 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-03-28T01:09:24.371 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=12 2024-03-28T01:09:24.371 DEBUG:tasks.check_counter:Getting stats from f 2024-03-28T01:09:24.371 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-28T01:09:24.690 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T01:09:25.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:25 smithi177 ceph-mon[36206]: pgmap v738: 129 pgs: 129 active+clean; 4.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 53 MiB/s wr, 318 op/s 2024-03-28T01:09:26.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:25 smithi174 ceph-mon[36060]: pgmap v738: 129 pgs: 129 active+clean; 4.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 53 MiB/s wr, 318 op/s 2024-03-28T01:09:26.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:25 smithi169 ceph-mon[32274]: pgmap v738: 129 pgs: 129 active+clean; 4.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 53 MiB/s wr, 318 op/s 2024-03-28T01:09:26.429 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-03-28T01:09:26.429 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T01:09:26.429 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 2526, 2024-03-28T01:09:26.429 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 2528, 2024-03-28T01:09:26.429 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 715563, 2024-03-28T01:09:26.429 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 803275, 2024-03-28T01:09:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 6, 2024-03-28T01:09:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 2, 2024-03-28T01:09:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.221941002, 2024-03-28T01:09:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.054934326, 2024-03-28T01:09:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.055726325, 2024-03-28T01:09:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.098629106, 2024-03-28T01:09:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2526, 2024-03-28T01:09:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.104790011, 2024-03-28T01:09:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000041484 2024-03-28T01:09:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T01:09:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T01:09:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T01:09:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 266, 2024-03-28T01:09:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 258, 2024-03-28T01:09:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 783516, 2024-03-28T01:09:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 149802, 2024-03-28T01:09:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 3, 2024-03-28T01:09:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 2, 2024-03-28T01:09:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.040510263, 2024-03-28T01:09:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.013474907, 2024-03-28T01:09:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.022175358, 2024-03-28T01:09:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.007467328, 2024-03-28T01:09:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 258, 2024-03-28T01:09:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.012743961, 2024-03-28T01:09:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000049395 2024-03-28T01:09:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 782960, 2024-03-28T01:09:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 149104 2024-03-28T01:09:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T01:09:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 3, 2024-03-28T01:09:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 188, 2024-03-28T01:09:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 588, 2024-03-28T01:09:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 485326, 2024-03-28T01:09:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 5, 2024-03-28T01:09:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 3, 2024-03-28T01:09:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.010361585, 2024-03-28T01:09:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.007634494, 2024-03-28T01:09:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.001329225, 2024-03-28T01:09:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000027906, 2024-03-28T01:09:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 186, 2024-03-28T01:09:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.009368257, 2024-03-28T01:09:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000050366 2024-03-28T01:09:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T01:09:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 485008 2024-03-28T01:09:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-03-28T01:09:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 1, 2024-03-28T01:09:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-03-28T01:09:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-MDSRank": { 2024-03-28T01:09:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-03-28T01:09:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-03-28T01:09:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 4105, 2024-03-28T01:09:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.114968679, 2024-03-28T01:09:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000028006 2024-03-28T01:09:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-PurgeQueue": { 2024-03-28T01:09:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-03-28T01:09:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-03-28T01:09:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: "mds": { 2024-03-28T01:09:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: "request": 0, 2024-03-28T01:09:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: "reply": 0, 2024-03-28T01:09:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "reply_latency": { 2024-03-28T01:09:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_reply": 0, 2024-03-28T01:09:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "forward": 0, 2024-03-28T01:09:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_complete": 0, 2024-03-28T01:09:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_keys": 0, 2024-03-28T01:09:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_commit": 0, 2024-03-28T01:09:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_split": 0, 2024-03-28T01:09:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_merge": 0, 2024-03-28T01:09:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes": 0, 2024-03-28T01:09:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_top": 0, 2024-03-28T01:09:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_bottom": 0, 2024-03-28T01:09:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pin_tail": 0, 2024-03-28T01:09:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pinned": 0, 2024-03-28T01:09:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_expired": 0, 2024-03-28T01:09:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_with_caps": 0, 2024-03-28T01:09:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "caps": 0, 2024-03-28T01:09:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "subtrees": 1, 2024-03-28T01:09:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse": 0, 2024-03-28T01:09:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_hit": 0, 2024-03-28T01:09:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_forward": 0, 2024-03-28T01:09:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_discover": 0, 2024-03-28T01:09:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_dir_fetch": 0, 2024-03-28T01:09:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_remote_ino": 0, 2024-03-28T01:09:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_lock": 0, 2024-03-28T01:09:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "load_cent": 0, 2024-03-28T01:09:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "q": 0, 2024-03-28T01:09:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "exported": 0, 2024-03-28T01:09:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "exported_inodes": 0, 2024-03-28T01:09:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "imported": 0, 2024-03-28T01:09:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "imported_inodes": 0, 2024-03-28T01:09:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_dir_fetch": 0, 2024-03-28T01:09:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_backtrace_fetch": 0, 2024-03-28T01:09:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_peer_discover": 0, 2024-03-28T01:09:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rfiles": 0, 2024-03-28T01:09:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rbytes": 0, 2024-03-28T01:09:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rsnaps": 0, 2024-03-28T01:09:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T01:09:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_set_tag": 0, 2024-03-28T01:09:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T01:09:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_inotable_repaired": 0, 2024-03-28T01:09:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_inodes": 0, 2024-03-28T01:09:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T01:09:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T01:09:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_file_inodes": 0, 2024-03-28T01:09:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_inode_file_caps": 0, 2024-03-28T01:09:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T01:09:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_grant": 0, 2024-03-28T01:09:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T01:09:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T01:09:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T01:09:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps": 0, 2024-03-28T01:09:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps_dirty": 0, 2024-03-28T01:09:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_cap_release": 0, 2024-03-28T01:09:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "process_request_cap_release": 0 2024-03-28T01:09:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_cache": { 2024-03-28T01:09:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update": 0, 2024-03-28T01:09:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update_receipt": 0, 2024-03-28T01:09:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_try_discover": 0, 2024-03-28T01:09:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_send_discover": 0, 2024-03-28T01:09:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_handle_discover": 0, 2024-03-28T01:09:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays": 0, 2024-03-28T01:09:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_delayed": 0, 2024-03-28T01:09:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_enqueuing": 0, 2024-03-28T01:09:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_created": 0, 2024-03-28T01:09:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_enqueued": 0, 2024-03-28T01:09:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_reintegrated": 0, 2024-03-28T01:09:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_migrated": 0, 2024-03-28T01:09:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_processing": 0, 2024-03-28T01:09:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_enqueued": 0, 2024-03-28T01:09:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_prioritized": 0, 2024-03-28T01:09:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_started": 0, 2024-03-28T01:09:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_completed": 0, 2024-03-28T01:09:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T01:09:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_exportdir": 0, 2024-03-28T01:09:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_flush": 0, 2024-03-28T01:09:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragmentdir": 0, 2024-03-28T01:09:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragstats": 0, 2024-03-28T01:09:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_inodestats": 0 2024-03-28T01:09:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_log": { 2024-03-28T01:09:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "evlrg": 0, 2024-03-28T01:09:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "evadd": 0, 2024-03-28T01:09:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "evex": 0, 2024-03-28T01:09:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "evtrm": 0, 2024-03-28T01:09:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "ev": 7, 2024-03-28T01:09:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "evexg": 0, 2024-03-28T01:09:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "evexd": 0, 2024-03-28T01:09:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "segadd": 0, 2024-03-28T01:09:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "segex": 0, 2024-03-28T01:09:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "segtrm": 0, 2024-03-28T01:09:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "seg": 2, 2024-03-28T01:09:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "segmjr": 2, 2024-03-28T01:09:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "segexg": 0, 2024-03-28T01:09:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "segexd": 0, 2024-03-28T01:09:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "expos": 4194304, 2024-03-28T01:09:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "wrpos": 4196264, 2024-03-28T01:09:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "rdpos": 4196123, 2024-03-28T01:09:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "jlat": { 2024-03-28T01:09:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "replayed": 7 2024-03-28T01:09:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_mem": { 2024-03-28T01:09:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "ino": 3, 2024-03-28T01:09:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "ino+": 3, 2024-03-28T01:09:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "ino-": 0, 2024-03-28T01:09:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "dir": 1, 2024-03-28T01:09:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "dir+": 1, 2024-03-28T01:09:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "dir-": 0, 2024-03-28T01:09:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "dn": 0, 2024-03-28T01:09:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "dn+": 0, 2024-03-28T01:09:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "dn-": 0, 2024-03-28T01:09:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "cap": 0, 2024-03-28T01:09:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "cap+": 0, 2024-03-28T01:09:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "cap-": 0, 2024-03-28T01:09:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "rss": 48400, 2024-03-28T01:09:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "heap": 215420 2024-03-28T01:09:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_server": { 2024-03-28T01:09:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_client_request": 0, 2024-03-28T01:09:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_server_request": 0, 2024-03-28T01:09:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_request": 0, 2024-03-28T01:09:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_session": 0, 2024-03-28T01:09:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_peer_request": 0, 2024-03-28T01:09:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "req_create_latency": { 2024-03-28T01:09:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getattr_latency": { 2024-03-28T01:09:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getfilelock_latency": { 2024-03-28T01:09:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: "req_link_latency": { 2024-03-28T01:09:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookup_latency": { 2024-03-28T01:09:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookuphash_latency": { 2024-03-28T01:09:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupino_latency": { 2024-03-28T01:09:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupname_latency": { 2024-03-28T01:09:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupparent_latency": { 2024-03-28T01:09:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupsnap_latency": { 2024-03-28T01:09:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lssnap_latency": { 2024-03-28T01:09:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mkdir_latency": { 2024-03-28T01:09:26.448 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.448 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.448 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.448 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mknod_latency": { 2024-03-28T01:09:26.448 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.448 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mksnap_latency": { 2024-03-28T01:09:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: "req_open_latency": { 2024-03-28T01:09:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: "req_readdir_latency": { 2024-03-28T01:09:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rename_latency": { 2024-03-28T01:09:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: "req_renamesnap_latency": { 2024-03-28T01:09:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: "req_snapdiff_latency": { 2024-03-28T01:09:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmdir_latency": { 2024-03-28T01:09:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmsnap_latency": { 2024-03-28T01:09:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmxattr_latency": { 2024-03-28T01:09:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setattr_latency": { 2024-03-28T01:09:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setdirlayout_latency": { 2024-03-28T01:09:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setfilelock_latency": { 2024-03-28T01:09:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setlayout_latency": { 2024-03-28T01:09:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setxattr_latency": { 2024-03-28T01:09:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "req_symlink_latency": { 2024-03-28T01:09:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: "req_unlink_latency": { 2024-03-28T01:09:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_revoke_eviction": 0, 2024-03-28T01:09:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_acquisition_throttle": 0, 2024-03-28T01:09:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getvxattr_latency": { 2024-03-28T01:09:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_sessions": { 2024-03-28T01:09:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "session_count": 1, 2024-03-28T01:09:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "session_add": 3, 2024-03-28T01:09:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "session_remove": 2, 2024-03-28T01:09:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_open": 2, 2024-03-28T01:09:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_stale": 0, 2024-03-28T01:09:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "total_load": 0, 2024-03-28T01:09:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "average_load": 0, 2024-03-28T01:09:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "avg_session_uptime": 3508, 2024-03-28T01:09:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "mdthresh_evicted": 0 2024-03-28T01:09:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-03-28T01:09:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-03-28T01:09:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-03-28T01:09:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T01:09:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-03-28T01:09:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T01:09:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-03-28T01:09:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T01:09:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T01:09:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T01:09:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T01:09:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T01:09:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-03-28T01:09:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T01:09:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T01:09:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-03-28T01:09:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-03-28T01:09:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-03-28T01:09:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-03-28T01:09:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T01:09:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T01:09:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T01:09:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T01:09:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T01:09:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-03-28T01:09:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-03-28T01:09:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-03-28T01:09:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T01:09:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T01:09:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-03-28T01:09:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-03-28T01:09:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-03-28T01:09:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-03-28T01:09:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T01:09:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-03-28T01:09:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T01:09:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-03-28T01:09:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 181327, 2024-03-28T01:09:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 46, 2024-03-28T01:09:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 0, 2024-03-28T01:09:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 0, 2024-03-28T01:09:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-03-28T01:09:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-03-28T01:09:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-03-28T01:09:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-03-28T01:09:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-03-28T01:09:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-03-28T01:09:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 8896, 2024-03-28T01:09:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 171, 2024-03-28T01:09:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T01:09:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-03-28T01:09:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-03-28T01:09:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-03-28T01:09:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 8320, 2024-03-28T01:09:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 13, 2024-03-28T01:09:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-03-28T01:09:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-03-28T01:09:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-03-28T01:09:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-03-28T01:09:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "objecter": { 2024-03-28T01:09:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "op_active": 0, 2024-03-28T01:09:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "op_laggy": 0, 2024-03-28T01:09:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send": 2529, 2024-03-28T01:09:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send_bytes": 21, 2024-03-28T01:09:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "op_resend": 3, 2024-03-28T01:09:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "op_reply": 2529, 2024-03-28T01:09:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "op_latency": { 2024-03-28T01:09:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2529, 2024-03-28T01:09:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 17.362528438, 2024-03-28T01:09:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.006865373 2024-03-28T01:09:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "op_inflight": 0, 2024-03-28T01:09:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "oplen_avg": { 2024-03-28T01:09:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2529, 2024-03-28T01:09:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 2530 2024-03-28T01:09:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "op": 2529, 2024-03-28T01:09:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "op_r": 2529, 2024-03-28T01:09:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "op_w": 0, 2024-03-28T01:09:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "op_rmw": 0, 2024-03-28T01:09:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "op_pg": 0, 2024-03-28T01:09:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_stat": 1654, 2024-03-28T01:09:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_create": 0, 2024-03-28T01:09:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_read": 873, 2024-03-28T01:09:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_write": 0, 2024-03-28T01:09:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writefull": 0, 2024-03-28T01:09:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writesame": 0, 2024-03-28T01:09:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_append": 0, 2024-03-28T01:09:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_zero": 0, 2024-03-28T01:09:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_truncate": 0, 2024-03-28T01:09:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_delete": 0, 2024-03-28T01:09:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_mapext": 0, 2024-03-28T01:09:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_sparse_read": 0, 2024-03-28T01:09:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_clonerange": 0, 2024-03-28T01:09:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_getxattr": 1, 2024-03-28T01:09:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_setxattr": 0, 2024-03-28T01:09:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_cmpxattr": 0, 2024-03-28T01:09:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_rmxattr": 0, 2024-03-28T01:09:26.462 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_resetxattrs": 0, 2024-03-28T01:09:26.462 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_call": 0, 2024-03-28T01:09:26.462 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_watch": 0, 2024-03-28T01:09:26.462 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_notify": 0, 2024-03-28T01:09:26.462 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T01:09:26.462 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls": 0, 2024-03-28T01:09:26.462 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls_filter": 0, 2024-03-28T01:09:26.462 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_other": 0, 2024-03-28T01:09:26.462 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_active": 0, 2024-03-28T01:09:26.462 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_send": 0, 2024-03-28T01:09:26.462 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_resend": 0, 2024-03-28T01:09:26.462 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_ping": 0, 2024-03-28T01:09:26.462 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_active": 0, 2024-03-28T01:09:26.462 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_send": 0, 2024-03-28T01:09:26.462 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_resend": 0, 2024-03-28T01:09:26.463 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_active": 0, 2024-03-28T01:09:26.463 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_send": 0, 2024-03-28T01:09:26.463 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_resend": 0, 2024-03-28T01:09:26.463 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_active": 0, 2024-03-28T01:09:26.463 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_send": 0, 2024-03-28T01:09:26.463 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_resend": 0, 2024-03-28T01:09:26.463 INFO:teuthology.orchestra.run.smithi177.stdout: "command_active": 0, 2024-03-28T01:09:26.463 INFO:teuthology.orchestra.run.smithi177.stdout: "command_send": 0, 2024-03-28T01:09:26.463 INFO:teuthology.orchestra.run.smithi177.stdout: "command_resend": 0, 2024-03-28T01:09:26.463 INFO:teuthology.orchestra.run.smithi177.stdout: "map_epoch": 94, 2024-03-28T01:09:26.463 INFO:teuthology.orchestra.run.smithi177.stdout: "map_full": 0, 2024-03-28T01:09:26.463 INFO:teuthology.orchestra.run.smithi177.stdout: "map_inc": 4, 2024-03-28T01:09:26.463 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_sessions": 4, 2024-03-28T01:09:26.463 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_open": 4, 2024-03-28T01:09:26.463 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_close": 0, 2024-03-28T01:09:26.463 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_laggy": 0, 2024-03-28T01:09:26.464 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_wr": 0, 2024-03-28T01:09:26.464 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_rd": 2, 2024-03-28T01:09:26.464 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_del": 0 2024-03-28T01:09:26.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.464 INFO:teuthology.orchestra.run.smithi177.stdout: "oft": { 2024-03-28T01:09:26.464 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_objs": 0, 2024-03-28T01:09:26.464 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_kv_pairs": 0, 2024-03-28T01:09:26.464 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_updates": 0, 2024-03-28T01:09:26.464 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_removes": 0 2024-03-28T01:09:26.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.464 INFO:teuthology.orchestra.run.smithi177.stdout: "osd-slow-ops": { 2024-03-28T01:09:26.464 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_ops_count": 0 2024-03-28T01:09:26.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.464 INFO:teuthology.orchestra.run.smithi177.stdout: "purge_queue": { 2024-03-28T01:09:26.464 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops": 0, 2024-03-28T01:09:26.464 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T01:09:26.465 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing": 0, 2024-03-28T01:09:26.465 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_high_water": 0, 2024-03-28T01:09:26.465 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed_ops": 0, 2024-03-28T01:09:26.465 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed": 0, 2024-03-28T01:09:26.465 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_item_in_journal": 0 2024-03-28T01:09:26.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.465 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T01:09:26.465 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-28T01:09:26.465 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-03-28T01:09:26.465 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-28T01:09:26.465 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 2791, 2024-03-28T01:09:26.465 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 1232026, 2024-03-28T01:09:26.465 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:26.465 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 2791, 2024-03-28T01:09:26.465 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-03-28T01:09:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-03-28T01:09:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 2791, 2024-03-28T01:09:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 1232026, 2024-03-28T01:09:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-28T01:09:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_bytes": { 2024-03-28T01:09:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-28T01:09:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-03-28T01:09:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-28T01:09:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-03-28T01:09:26.466 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-03-28T01:09:26.467 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:26.467 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-03-28T01:09:26.467 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 2529, 2024-03-28T01:09:26.467 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 1965, 2024-03-28T01:09:26.467 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 2529, 2024-03-28T01:09:26.467 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 1965, 2024-03-28T01:09:26.467 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-28T01:09:26.467 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.467 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.467 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.467 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:26.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.467 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_ops": { 2024-03-28T01:09:26.467 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-28T01:09:26.467 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 1024, 2024-03-28T01:09:26.468 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-28T01:09:26.468 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-03-28T01:09:26.468 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-03-28T01:09:26.468 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:26.468 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-03-28T01:09:26.468 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 2529, 2024-03-28T01:09:26.468 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 2529, 2024-03-28T01:09:26.468 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 2529, 2024-03-28T01:09:26.468 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 2529, 2024-03-28T01:09:26.468 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-28T01:09:26.468 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.468 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.468 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.468 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:26.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.469 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle": { 2024-03-28T01:09:26.469 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-28T01:09:26.469 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-03-28T01:09:26.469 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-28T01:09:26.469 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-03-28T01:09:26.469 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-03-28T01:09:26.469 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:26.469 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-03-28T01:09:26.469 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-03-28T01:09:26.469 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-03-28T01:09:26.469 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-03-28T01:09:26.469 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-03-28T01:09:26.469 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-28T01:09:26.469 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.470 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.470 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.470 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:26.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:26.470 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle-0x56490e5fe6e0": { 2024-03-28T01:09:26.470 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-28T01:09:26.470 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-03-28T01:09:26.470 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-28T01:09:26.470 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-03-28T01:09:26.470 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-03-28T01:09:26.470 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:26.470 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-03-28T01:09:26.470 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-03-28T01:09:26.470 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-03-28T01:09:26.470 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-03-28T01:09:26.471 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-03-28T01:09:26.471 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-28T01:09:26.471 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:26.471 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:26.471 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:26.471 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:26.471 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:26.471 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-03-28T01:09:27.791 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-28T01:09:27.792 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-28T01:09:27.792 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-03-28T01:09:27.792 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-03-28T01:09:27.792 DEBUG:tasks.check_counter:Getting stats from i 2024-03-28T01:09:27.792 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-28T01:09:28.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:27 smithi174 ceph-mon[36060]: pgmap v739: 129 pgs: 129 active+clean; 4.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 69 MiB/s wr, 406 op/s 2024-03-28T01:09:28.131 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T01:09:28.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:27 smithi177 ceph-mon[36206]: pgmap v739: 129 pgs: 129 active+clean; 4.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 69 MiB/s wr, 406 op/s 2024-03-28T01:09:28.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:27 smithi169 ceph-mon[32274]: pgmap v739: 129 pgs: 129 active+clean; 4.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 69 MiB/s wr, 406 op/s 2024-03-28T01:09:29.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:29 smithi177 ceph-mon[36206]: pgmap v740: 129 pgs: 129 active+clean; 4.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 56 MiB/s wr, 345 op/s 2024-03-28T01:09:30.010 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-03-28T01:09:30.010 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T01:09:30.010 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 338, 2024-03-28T01:09:30.010 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 328, 2024-03-28T01:09:30.011 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 802762, 2024-03-28T01:09:30.011 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 253897, 2024-03-28T01:09:30.011 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 17, 2024-03-28T01:09:30.011 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 2, 2024-03-28T01:09:30.011 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.054957760, 2024-03-28T01:09:30.011 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.015469771, 2024-03-28T01:09:30.011 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.030792568, 2024-03-28T01:09:30.011 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.007154855, 2024-03-28T01:09:30.011 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:30.011 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 327, 2024-03-28T01:09:30.011 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.026077097, 2024-03-28T01:09:30.011 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000079746 2024-03-28T01:09:30.012 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.012 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:30.012 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 46, 2024-03-28T01:09:30.012 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000007786, 2024-03-28T01:09:30.012 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000169 2024-03-28T01:09:30.012 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.012 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 795136, 2024-03-28T01:09:30.012 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 154416 2024-03-28T01:09:30.012 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.012 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T01:09:30.012 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 1782, 2024-03-28T01:09:30.012 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 1237, 2024-03-28T01:09:30.012 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 433158, 2024-03-28T01:09:30.013 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 841782, 2024-03-28T01:09:30.013 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 8, 2024-03-28T01:09:30.013 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 5, 2024-03-28T01:09:30.013 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.112848514, 2024-03-28T01:09:30.013 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.042910685, 2024-03-28T01:09:30.013 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.093148722, 2024-03-28T01:09:30.013 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.005335972, 2024-03-28T01:09:30.013 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:30.013 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 1236, 2024-03-28T01:09:30.013 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.116237129, 2024-03-28T01:09:30.013 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000094042 2024-03-28T01:09:30.013 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.013 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:30.013 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2603, 2024-03-28T01:09:30.014 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000622458, 2024-03-28T01:09:30.014 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000239 2024-03-28T01:09:30.014 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.014 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-03-28T01:09:30.014 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 488480 2024-03-28T01:09:30.014 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.014 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T01:09:30.014 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 62, 2024-03-28T01:09:30.014 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 64, 2024-03-28T01:09:30.014 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 12664, 2024-03-28T01:09:30.014 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 389442, 2024-03-28T01:09:30.014 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 41, 2024-03-28T01:09:30.014 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 6, 2024-03-28T01:09:30.014 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.021653066, 2024-03-28T01:09:30.014 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.001897498, 2024-03-28T01:09:30.015 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.018198961, 2024-03-28T01:09:30.015 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000538118, 2024-03-28T01:09:30.015 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:30.015 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 62, 2024-03-28T01:09:30.015 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.012899224, 2024-03-28T01:09:30.015 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000208052 2024-03-28T01:09:30.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.015 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:30.015 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 51, 2024-03-28T01:09:30.015 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000009045, 2024-03-28T01:09:30.015 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000177 2024-03-28T01:09:30.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.015 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 2512, 2024-03-28T01:09:30.016 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 3472 2024-03-28T01:09:30.016 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.016 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-03-28T01:09:30.016 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 1, 2024-03-28T01:09:30.016 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-03-28T01:09:30.016 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.016 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-MDSRank": { 2024-03-28T01:09:30.016 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-03-28T01:09:30.016 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-03-28T01:09:30.016 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 50, 2024-03-28T01:09:30.016 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.002493590, 2024-03-28T01:09:30.017 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000049871 2024-03-28T01:09:30.017 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:30.017 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.017 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-PurgeQueue": { 2024-03-28T01:09:30.017 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-03-28T01:09:30.017 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-03-28T01:09:30.017 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2, 2024-03-28T01:09:30.017 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000006405, 2024-03-28T01:09:30.017 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000003202 2024-03-28T01:09:30.017 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:30.017 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.017 INFO:teuthology.orchestra.run.smithi177.stdout: "mds": { 2024-03-28T01:09:30.017 INFO:teuthology.orchestra.run.smithi177.stdout: "request": 11, 2024-03-28T01:09:30.017 INFO:teuthology.orchestra.run.smithi177.stdout: "reply": 6, 2024-03-28T01:09:30.018 INFO:teuthology.orchestra.run.smithi177.stdout: "reply_latency": { 2024-03-28T01:09:30.018 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 6, 2024-03-28T01:09:30.018 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000982565, 2024-03-28T01:09:30.018 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000163760 2024-03-28T01:09:30.018 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.018 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_reply": 0, 2024-03-28T01:09:30.018 INFO:teuthology.orchestra.run.smithi177.stdout: "forward": 5, 2024-03-28T01:09:30.018 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_complete": 0, 2024-03-28T01:09:30.018 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_keys": 0, 2024-03-28T01:09:30.018 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_commit": 11, 2024-03-28T01:09:30.018 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_split": 0, 2024-03-28T01:09:30.018 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_merge": 0, 2024-03-28T01:09:30.019 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes": 16, 2024-03-28T01:09:30.019 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_top": 0, 2024-03-28T01:09:30.019 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_bottom": 16, 2024-03-28T01:09:30.019 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pin_tail": 0, 2024-03-28T01:09:30.019 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pinned": 13, 2024-03-28T01:09:30.019 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_expired": 8, 2024-03-28T01:09:30.019 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_with_caps": 2, 2024-03-28T01:09:30.019 INFO:teuthology.orchestra.run.smithi177.stdout: "caps": 2, 2024-03-28T01:09:30.019 INFO:teuthology.orchestra.run.smithi177.stdout: "subtrees": 3, 2024-03-28T01:09:30.019 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse": 20, 2024-03-28T01:09:30.019 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_hit": 12, 2024-03-28T01:09:30.019 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_forward": 1, 2024-03-28T01:09:30.019 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_discover": 5, 2024-03-28T01:09:30.019 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_dir_fetch": 0, 2024-03-28T01:09:30.019 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_remote_ino": 0, 2024-03-28T01:09:30.020 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_lock": 0, 2024-03-28T01:09:30.020 INFO:teuthology.orchestra.run.smithi177.stdout: "load_cent": 0, 2024-03-28T01:09:30.020 INFO:teuthology.orchestra.run.smithi177.stdout: "q": 0, 2024-03-28T01:09:30.020 INFO:teuthology.orchestra.run.smithi177.stdout: "exported": 0, 2024-03-28T01:09:30.020 INFO:teuthology.orchestra.run.smithi177.stdout: "exported_inodes": 0, 2024-03-28T01:09:30.020 INFO:teuthology.orchestra.run.smithi177.stdout: "imported": 0, 2024-03-28T01:09:30.020 INFO:teuthology.orchestra.run.smithi177.stdout: "imported_inodes": 0, 2024-03-28T01:09:30.020 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_dir_fetch": 0, 2024-03-28T01:09:30.020 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_backtrace_fetch": 0, 2024-03-28T01:09:30.020 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_peer_discover": 0, 2024-03-28T01:09:30.020 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rfiles": 22, 2024-03-28T01:09:30.020 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rbytes": 10738150859, 2024-03-28T01:09:30.020 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rsnaps": 0, 2024-03-28T01:09:30.020 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T01:09:30.021 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_set_tag": 0, 2024-03-28T01:09:30.021 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T01:09:30.021 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_inotable_repaired": 0, 2024-03-28T01:09:30.021 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_inodes": 0, 2024-03-28T01:09:30.021 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T01:09:30.021 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T01:09:30.021 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_file_inodes": 0, 2024-03-28T01:09:30.021 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_inode_file_caps": 0, 2024-03-28T01:09:30.021 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T01:09:30.021 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_grant": 0, 2024-03-28T01:09:30.021 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T01:09:30.021 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T01:09:30.021 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T01:09:30.021 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps": 0, 2024-03-28T01:09:30.022 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps_dirty": 0, 2024-03-28T01:09:30.022 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_cap_release": 0, 2024-03-28T01:09:30.022 INFO:teuthology.orchestra.run.smithi177.stdout: "process_request_cap_release": 2 2024-03-28T01:09:30.022 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.022 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_cache": { 2024-03-28T01:09:30.022 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update": 0, 2024-03-28T01:09:30.022 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update_receipt": 12, 2024-03-28T01:09:30.022 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_try_discover": 8, 2024-03-28T01:09:30.022 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_send_discover": 10, 2024-03-28T01:09:30.022 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_handle_discover": 0, 2024-03-28T01:09:30.022 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays": 0, 2024-03-28T01:09:30.022 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_delayed": 0, 2024-03-28T01:09:30.022 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_enqueuing": 0, 2024-03-28T01:09:30.022 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_created": 0, 2024-03-28T01:09:30.023 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_enqueued": 0, 2024-03-28T01:09:30.023 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_reintegrated": 0, 2024-03-28T01:09:30.023 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_migrated": 0, 2024-03-28T01:09:30.023 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_processing": 0, 2024-03-28T01:09:30.023 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_enqueued": 0, 2024-03-28T01:09:30.023 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_prioritized": 0, 2024-03-28T01:09:30.023 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_started": 0, 2024-03-28T01:09:30.023 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_completed": 0, 2024-03-28T01:09:30.023 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T01:09:30.023 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_exportdir": 0, 2024-03-28T01:09:30.023 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_flush": 0, 2024-03-28T01:09:30.023 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragmentdir": 0, 2024-03-28T01:09:30.023 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragstats": 0, 2024-03-28T01:09:30.023 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_inodestats": 0 2024-03-28T01:09:30.024 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.024 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_log": { 2024-03-28T01:09:30.024 INFO:teuthology.orchestra.run.smithi177.stdout: "evlrg": 0, 2024-03-28T01:09:30.024 INFO:teuthology.orchestra.run.smithi177.stdout: "evadd": 7, 2024-03-28T01:09:30.024 INFO:teuthology.orchestra.run.smithi177.stdout: "evex": 0, 2024-03-28T01:09:30.024 INFO:teuthology.orchestra.run.smithi177.stdout: "evtrm": 0, 2024-03-28T01:09:30.024 INFO:teuthology.orchestra.run.smithi177.stdout: "ev": 7, 2024-03-28T01:09:30.024 INFO:teuthology.orchestra.run.smithi177.stdout: "evexg": 0, 2024-03-28T01:09:30.024 INFO:teuthology.orchestra.run.smithi177.stdout: "evexd": 0, 2024-03-28T01:09:30.024 INFO:teuthology.orchestra.run.smithi177.stdout: "segadd": 2, 2024-03-28T01:09:30.024 INFO:teuthology.orchestra.run.smithi177.stdout: "segex": 0, 2024-03-28T01:09:30.024 INFO:teuthology.orchestra.run.smithi177.stdout: "segtrm": 0, 2024-03-28T01:09:30.024 INFO:teuthology.orchestra.run.smithi177.stdout: "seg": 2, 2024-03-28T01:09:30.024 INFO:teuthology.orchestra.run.smithi177.stdout: "segmjr": 2, 2024-03-28T01:09:30.025 INFO:teuthology.orchestra.run.smithi177.stdout: "segexg": 0, 2024-03-28T01:09:30.025 INFO:teuthology.orchestra.run.smithi177.stdout: "segexd": 0, 2024-03-28T01:09:30.025 INFO:teuthology.orchestra.run.smithi177.stdout: "expos": 4194304, 2024-03-28T01:09:30.025 INFO:teuthology.orchestra.run.smithi177.stdout: "wrpos": 4196264, 2024-03-28T01:09:30.025 INFO:teuthology.orchestra.run.smithi177.stdout: "rdpos": 0, 2024-03-28T01:09:30.025 INFO:teuthology.orchestra.run.smithi177.stdout: "jlat": { 2024-03-28T01:09:30.025 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 6, 2024-03-28T01:09:30.025 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.008750864, 2024-03-28T01:09:30.025 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.001458477 2024-03-28T01:09:30.025 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.025 INFO:teuthology.orchestra.run.smithi177.stdout: "replayed": 0 2024-03-28T01:09:30.025 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.025 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_mem": { 2024-03-28T01:09:30.025 INFO:teuthology.orchestra.run.smithi177.stdout: "ino": 20, 2024-03-28T01:09:30.026 INFO:teuthology.orchestra.run.smithi177.stdout: "ino+": 24, 2024-03-28T01:09:30.026 INFO:teuthology.orchestra.run.smithi177.stdout: "ino-": 4, 2024-03-28T01:09:30.026 INFO:teuthology.orchestra.run.smithi177.stdout: "dir": 19, 2024-03-28T01:09:30.026 INFO:teuthology.orchestra.run.smithi177.stdout: "dir+": 24, 2024-03-28T01:09:30.026 INFO:teuthology.orchestra.run.smithi177.stdout: "dir-": 5, 2024-03-28T01:09:30.026 INFO:teuthology.orchestra.run.smithi177.stdout: "dn": 16, 2024-03-28T01:09:30.026 INFO:teuthology.orchestra.run.smithi177.stdout: "dn+": 24, 2024-03-28T01:09:30.026 INFO:teuthology.orchestra.run.smithi177.stdout: "dn-": 8, 2024-03-28T01:09:30.026 INFO:teuthology.orchestra.run.smithi177.stdout: "cap": 2, 2024-03-28T01:09:30.026 INFO:teuthology.orchestra.run.smithi177.stdout: "cap+": 6, 2024-03-28T01:09:30.026 INFO:teuthology.orchestra.run.smithi177.stdout: "cap-": 4, 2024-03-28T01:09:30.026 INFO:teuthology.orchestra.run.smithi177.stdout: "rss": 51012, 2024-03-28T01:09:30.026 INFO:teuthology.orchestra.run.smithi177.stdout: "heap": 223612 2024-03-28T01:09:30.027 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.027 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_server": { 2024-03-28T01:09:30.027 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_client_request": 12, 2024-03-28T01:09:30.027 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_server_request": 0, 2024-03-28T01:09:30.027 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_request": 11, 2024-03-28T01:09:30.027 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_session": 18, 2024-03-28T01:09:30.027 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_peer_request": 0, 2024-03-28T01:09:30.027 INFO:teuthology.orchestra.run.smithi177.stdout: "req_create_latency": { 2024-03-28T01:09:30.027 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.027 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.027 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.027 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.027 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getattr_latency": { 2024-03-28T01:09:30.027 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2, 2024-03-28T01:09:30.028 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000391730, 2024-03-28T01:09:30.028 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000195865 2024-03-28T01:09:30.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.028 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getfilelock_latency": { 2024-03-28T01:09:30.028 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.028 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.028 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.028 INFO:teuthology.orchestra.run.smithi177.stdout: "req_link_latency": { 2024-03-28T01:09:30.028 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.028 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.028 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.028 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookup_latency": { 2024-03-28T01:09:30.029 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2, 2024-03-28T01:09:30.029 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000332133, 2024-03-28T01:09:30.029 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000166066 2024-03-28T01:09:30.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.029 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookuphash_latency": { 2024-03-28T01:09:30.029 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.029 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.029 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.029 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupino_latency": { 2024-03-28T01:09:30.029 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2, 2024-03-28T01:09:30.029 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000258702, 2024-03-28T01:09:30.029 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000129351 2024-03-28T01:09:30.029 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.030 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupname_latency": { 2024-03-28T01:09:30.030 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.030 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.030 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.030 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.030 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupparent_latency": { 2024-03-28T01:09:30.030 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.030 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.030 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.030 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.030 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupsnap_latency": { 2024-03-28T01:09:30.030 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.030 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.031 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.031 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.031 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lssnap_latency": { 2024-03-28T01:09:30.031 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.031 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.031 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.031 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.031 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mkdir_latency": { 2024-03-28T01:09:30.031 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.031 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.032 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.032 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.032 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mknod_latency": { 2024-03-28T01:09:30.032 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.032 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.032 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.032 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.033 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mksnap_latency": { 2024-03-28T01:09:30.033 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.033 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.033 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.033 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.033 INFO:teuthology.orchestra.run.smithi177.stdout: "req_open_latency": { 2024-03-28T01:09:30.033 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.033 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.033 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.033 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.033 INFO:teuthology.orchestra.run.smithi177.stdout: "req_readdir_latency": { 2024-03-28T01:09:30.033 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.034 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.034 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.034 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.034 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rename_latency": { 2024-03-28T01:09:30.034 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.034 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.034 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.034 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.034 INFO:teuthology.orchestra.run.smithi177.stdout: "req_renamesnap_latency": { 2024-03-28T01:09:30.034 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.034 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.034 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.034 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.034 INFO:teuthology.orchestra.run.smithi177.stdout: "req_snapdiff_latency": { 2024-03-28T01:09:30.035 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.035 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.035 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.035 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.035 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmdir_latency": { 2024-03-28T01:09:30.035 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.035 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.035 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.035 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.035 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmsnap_latency": { 2024-03-28T01:09:30.035 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.035 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.035 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.035 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.035 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmxattr_latency": { 2024-03-28T01:09:30.036 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.036 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.036 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.036 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.036 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setattr_latency": { 2024-03-28T01:09:30.036 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.036 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.036 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.036 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.036 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setdirlayout_latency": { 2024-03-28T01:09:30.036 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.036 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.036 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.037 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setfilelock_latency": { 2024-03-28T01:09:30.037 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.037 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.037 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.037 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setlayout_latency": { 2024-03-28T01:09:30.037 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.037 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.037 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.037 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setxattr_latency": { 2024-03-28T01:09:30.037 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.038 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.038 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.038 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.038 INFO:teuthology.orchestra.run.smithi177.stdout: "req_symlink_latency": { 2024-03-28T01:09:30.038 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.038 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.038 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.038 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.038 INFO:teuthology.orchestra.run.smithi177.stdout: "req_unlink_latency": { 2024-03-28T01:09:30.038 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.038 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.038 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.038 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.038 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_revoke_eviction": 0, 2024-03-28T01:09:30.039 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_acquisition_throttle": 0, 2024-03-28T01:09:30.039 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getvxattr_latency": { 2024-03-28T01:09:30.039 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.039 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.039 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.039 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:30.039 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.039 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_sessions": { 2024-03-28T01:09:30.039 INFO:teuthology.orchestra.run.smithi177.stdout: "session_count": 1, 2024-03-28T01:09:30.039 INFO:teuthology.orchestra.run.smithi177.stdout: "session_add": 3, 2024-03-28T01:09:30.039 INFO:teuthology.orchestra.run.smithi177.stdout: "session_remove": 2, 2024-03-28T01:09:30.039 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_open": 1, 2024-03-28T01:09:30.039 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_stale": 0, 2024-03-28T01:09:30.039 INFO:teuthology.orchestra.run.smithi177.stdout: "total_load": 5, 2024-03-28T01:09:30.040 INFO:teuthology.orchestra.run.smithi177.stdout: "average_load": 2, 2024-03-28T01:09:30.040 INFO:teuthology.orchestra.run.smithi177.stdout: "avg_session_uptime": 761, 2024-03-28T01:09:30.040 INFO:teuthology.orchestra.run.smithi177.stdout: "mdthresh_evicted": 0 2024-03-28T01:09:30.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.040 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-03-28T01:09:30.040 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-03-28T01:09:30.040 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-03-28T01:09:30.040 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T01:09:30.040 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-03-28T01:09:30.040 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T01:09:30.040 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-03-28T01:09:30.040 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T01:09:30.040 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T01:09:30.041 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T01:09:30.041 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T01:09:30.041 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T01:09:30.041 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-03-28T01:09:30.041 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T01:09:30.041 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T01:09:30.041 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-03-28T01:09:30.041 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-03-28T01:09:30.041 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-03-28T01:09:30.041 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-03-28T01:09:30.041 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T01:09:30.041 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T01:09:30.041 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T01:09:30.042 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T01:09:30.042 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T01:09:30.042 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-03-28T01:09:30.042 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-03-28T01:09:30.042 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-03-28T01:09:30.042 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T01:09:30.042 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T01:09:30.042 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-03-28T01:09:30.042 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-03-28T01:09:30.042 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-03-28T01:09:30.042 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-03-28T01:09:30.042 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T01:09:30.042 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-03-28T01:09:30.043 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T01:09:30.043 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-03-28T01:09:30.043 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 256683, 2024-03-28T01:09:30.043 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 98, 2024-03-28T01:09:30.043 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 176, 2024-03-28T01:09:30.043 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 2, 2024-03-28T01:09:30.043 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-03-28T01:09:30.043 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-03-28T01:09:30.043 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-03-28T01:09:30.043 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-03-28T01:09:30.043 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-03-28T01:09:30.043 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-03-28T01:09:30.043 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 8896, 2024-03-28T01:09:30.043 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 171, 2024-03-28T01:09:30.044 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T01:09:30.044 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-03-28T01:09:30.044 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-03-28T01:09:30.044 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-03-28T01:09:30.044 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 80477, 2024-03-28T01:09:30.044 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 338, 2024-03-28T01:09:30.044 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-03-28T01:09:30.044 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-03-28T01:09:30.044 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-03-28T01:09:30.044 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-03-28T01:09:30.044 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.044 INFO:teuthology.orchestra.run.smithi177.stdout: "objecter": { 2024-03-28T01:09:30.044 INFO:teuthology.orchestra.run.smithi177.stdout: "op_active": 0, 2024-03-28T01:09:30.044 INFO:teuthology.orchestra.run.smithi177.stdout: "op_laggy": 0, 2024-03-28T01:09:30.044 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send": 46, 2024-03-28T01:09:30.045 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send_bytes": 12768, 2024-03-28T01:09:30.045 INFO:teuthology.orchestra.run.smithi177.stdout: "op_resend": 33, 2024-03-28T01:09:30.045 INFO:teuthology.orchestra.run.smithi177.stdout: "op_reply": 46, 2024-03-28T01:09:30.045 INFO:teuthology.orchestra.run.smithi177.stdout: "op_latency": { 2024-03-28T01:09:30.045 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 46, 2024-03-28T01:09:30.045 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.219993700, 2024-03-28T01:09:30.045 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.004782471 2024-03-28T01:09:30.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.045 INFO:teuthology.orchestra.run.smithi177.stdout: "op_inflight": 0, 2024-03-28T01:09:30.045 INFO:teuthology.orchestra.run.smithi177.stdout: "oplen_avg": { 2024-03-28T01:09:30.045 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 46, 2024-03-28T01:09:30.045 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 71 2024-03-28T01:09:30.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.045 INFO:teuthology.orchestra.run.smithi177.stdout: "op": 46, 2024-03-28T01:09:30.046 INFO:teuthology.orchestra.run.smithi177.stdout: "op_r": 0, 2024-03-28T01:09:30.046 INFO:teuthology.orchestra.run.smithi177.stdout: "op_w": 46, 2024-03-28T01:09:30.046 INFO:teuthology.orchestra.run.smithi177.stdout: "op_rmw": 0, 2024-03-28T01:09:30.046 INFO:teuthology.orchestra.run.smithi177.stdout: "op_pg": 0, 2024-03-28T01:09:30.046 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_stat": 0, 2024-03-28T01:09:30.046 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_create": 10, 2024-03-28T01:09:30.046 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_read": 0, 2024-03-28T01:09:30.046 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_write": 6, 2024-03-28T01:09:30.046 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writefull": 8, 2024-03-28T01:09:30.046 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writesame": 0, 2024-03-28T01:09:30.046 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_append": 0, 2024-03-28T01:09:30.046 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_zero": 0, 2024-03-28T01:09:30.046 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_truncate": 0, 2024-03-28T01:09:30.046 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_delete": 6, 2024-03-28T01:09:30.046 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_mapext": 0, 2024-03-28T01:09:30.047 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_sparse_read": 0, 2024-03-28T01:09:30.047 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_clonerange": 0, 2024-03-28T01:09:30.047 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_getxattr": 0, 2024-03-28T01:09:30.047 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_setxattr": 20, 2024-03-28T01:09:30.047 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_cmpxattr": 0, 2024-03-28T01:09:30.047 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_rmxattr": 0, 2024-03-28T01:09:30.047 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_resetxattrs": 0, 2024-03-28T01:09:30.047 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_call": 0, 2024-03-28T01:09:30.047 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_watch": 0, 2024-03-28T01:09:30.047 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_notify": 0, 2024-03-28T01:09:30.047 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T01:09:30.047 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls": 0, 2024-03-28T01:09:30.047 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls_filter": 0, 2024-03-28T01:09:30.047 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_other": 0, 2024-03-28T01:09:30.048 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_active": 0, 2024-03-28T01:09:30.048 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_send": 0, 2024-03-28T01:09:30.048 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_resend": 0, 2024-03-28T01:09:30.048 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_ping": 0, 2024-03-28T01:09:30.048 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_active": 0, 2024-03-28T01:09:30.048 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_send": 0, 2024-03-28T01:09:30.048 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_resend": 0, 2024-03-28T01:09:30.048 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_active": 0, 2024-03-28T01:09:30.048 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_send": 0, 2024-03-28T01:09:30.048 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_resend": 0, 2024-03-28T01:09:30.048 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_active": 0, 2024-03-28T01:09:30.048 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_send": 0, 2024-03-28T01:09:30.049 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_resend": 0, 2024-03-28T01:09:30.049 INFO:teuthology.orchestra.run.smithi177.stdout: "command_active": 0, 2024-03-28T01:09:30.049 INFO:teuthology.orchestra.run.smithi177.stdout: "command_send": 0, 2024-03-28T01:09:30.049 INFO:teuthology.orchestra.run.smithi177.stdout: "command_resend": 0, 2024-03-28T01:09:30.049 INFO:teuthology.orchestra.run.smithi177.stdout: "map_epoch": 94, 2024-03-28T01:09:30.049 INFO:teuthology.orchestra.run.smithi177.stdout: "map_full": 0, 2024-03-28T01:09:30.049 INFO:teuthology.orchestra.run.smithi177.stdout: "map_inc": 4, 2024-03-28T01:09:30.049 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_sessions": 9, 2024-03-28T01:09:30.049 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_open": 9, 2024-03-28T01:09:30.049 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_close": 0, 2024-03-28T01:09:30.049 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_laggy": 0, 2024-03-28T01:09:30.049 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_wr": 20, 2024-03-28T01:09:30.049 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_rd": 0, 2024-03-28T01:09:30.049 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_del": 1 2024-03-28T01:09:30.049 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.050 INFO:teuthology.orchestra.run.smithi177.stdout: "oft": { 2024-03-28T01:09:30.050 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_objs": 1, 2024-03-28T01:09:30.050 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_kv_pairs": 0, 2024-03-28T01:09:30.050 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_updates": 0, 2024-03-28T01:09:30.050 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_removes": 0 2024-03-28T01:09:30.050 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.050 INFO:teuthology.orchestra.run.smithi177.stdout: "osd-slow-ops": { 2024-03-28T01:09:30.050 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_ops_count": 0 2024-03-28T01:09:30.050 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.050 INFO:teuthology.orchestra.run.smithi177.stdout: "purge_queue": { 2024-03-28T01:09:30.050 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops": 0, 2024-03-28T01:09:30.050 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T01:09:30.050 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing": 0, 2024-03-28T01:09:30.050 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_high_water": 0, 2024-03-28T01:09:30.051 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed_ops": 0, 2024-03-28T01:09:30.051 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed": 0, 2024-03-28T01:09:30.051 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_item_in_journal": 0 2024-03-28T01:09:30.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.051 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T01:09:30.051 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-28T01:09:30.051 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-03-28T01:09:30.051 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-28T01:09:30.051 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 2178, 2024-03-28T01:09:30.051 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 1033878, 2024-03-28T01:09:30.051 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:30.051 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 2178, 2024-03-28T01:09:30.051 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-03-28T01:09:30.051 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-03-28T01:09:30.051 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 2178, 2024-03-28T01:09:30.052 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 1033878, 2024-03-28T01:09:30.052 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-28T01:09:30.052 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.052 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.052 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.052 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:30.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.052 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_bytes": { 2024-03-28T01:09:30.052 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-28T01:09:30.052 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-03-28T01:09:30.052 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-28T01:09:30.052 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-03-28T01:09:30.052 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-03-28T01:09:30.052 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:30.052 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-03-28T01:09:30.053 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 46, 2024-03-28T01:09:30.053 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 12768, 2024-03-28T01:09:30.053 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 46, 2024-03-28T01:09:30.053 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 12768, 2024-03-28T01:09:30.053 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-28T01:09:30.053 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.053 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.053 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.053 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:30.053 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.053 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_ops": { 2024-03-28T01:09:30.053 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-28T01:09:30.054 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 1024, 2024-03-28T01:09:30.054 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-28T01:09:30.054 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-03-28T01:09:30.054 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-03-28T01:09:30.054 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:30.054 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-03-28T01:09:30.054 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 46, 2024-03-28T01:09:30.054 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 46, 2024-03-28T01:09:30.054 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 46, 2024-03-28T01:09:30.054 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 46, 2024-03-28T01:09:30.054 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-28T01:09:30.054 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.054 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.054 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.055 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:30.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.055 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle": { 2024-03-28T01:09:30.055 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-28T01:09:30.055 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-03-28T01:09:30.055 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-28T01:09:30.055 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-03-28T01:09:30.055 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-03-28T01:09:30.055 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:30.055 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-03-28T01:09:30.055 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-03-28T01:09:30.055 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-03-28T01:09:30.055 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-03-28T01:09:30.055 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-03-28T01:09:30.056 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-28T01:09:30.056 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.056 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.056 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.056 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:30.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:30.056 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle-0x5624acc60500": { 2024-03-28T01:09:30.056 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-28T01:09:30.056 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-03-28T01:09:30.056 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-28T01:09:30.056 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 7, 2024-03-28T01:09:30.056 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 1960, 2024-03-28T01:09:30.056 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:30.056 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 7, 2024-03-28T01:09:30.056 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-03-28T01:09:30.057 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-03-28T01:09:30.057 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 6, 2024-03-28T01:09:30.057 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 1960, 2024-03-28T01:09:30.057 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-28T01:09:30.057 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:30.057 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:30.057 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:30.057 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:30.057 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:30.057 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-03-28T01:09:30.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:29 smithi174 ceph-mon[36060]: pgmap v740: 129 pgs: 129 active+clean; 4.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 56 MiB/s wr, 345 op/s 2024-03-28T01:09:30.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:29 smithi169 ceph-mon[32274]: pgmap v740: 129 pgs: 129 active+clean; 4.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 56 MiB/s wr, 345 op/s 2024-03-28T01:09:30.907 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-28T01:09:30.907 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-28T01:09:30.907 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-03-28T01:09:30.907 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=12 2024-03-28T01:09:30.907 DEBUG:tasks.check_counter:Getting stats from l 2024-03-28T01:09:30.908 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 833368da-ec9c-11ee-95d0-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-28T01:09:31.159 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T01:09:32.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:31 smithi174 ceph-mon[36060]: pgmap v741: 129 pgs: 129 active+clean; 3.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 56 MiB/s wr, 372 op/s 2024-03-28T01:09:32.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:31 smithi177 ceph-mon[36206]: pgmap v741: 129 pgs: 129 active+clean; 3.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 56 MiB/s wr, 372 op/s 2024-03-28T01:09:32.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:31 smithi169 ceph-mon[32274]: pgmap v741: 129 pgs: 129 active+clean; 3.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 56 MiB/s wr, 372 op/s 2024-03-28T01:09:34.133 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:33 smithi174 ceph-mon[36060]: pgmap v742: 129 pgs: 129 active+clean; 2.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 73 MiB/s wr, 570 op/s 2024-03-28T01:09:34.155 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:33 smithi177 ceph-mon[36206]: pgmap v742: 129 pgs: 129 active+clean; 2.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 73 MiB/s wr, 570 op/s 2024-03-28T01:09:34.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:33 smithi169 ceph-mon[32274]: pgmap v742: 129 pgs: 129 active+clean; 2.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 73 MiB/s wr, 570 op/s 2024-03-28T01:09:34.636 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-03-28T01:09:34.637 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T01:09:34.637 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 4, 2024-03-28T01:09:34.637 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 4, 2024-03-28T01:09:34.637 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 3072, 2024-03-28T01:09:34.637 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 448, 2024-03-28T01:09:34.637 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 4, 2024-03-28T01:09:34.637 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 0, 2024-03-28T01:09:34.637 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.001588890, 2024-03-28T01:09:34.637 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.000031414, 2024-03-28T01:09:34.637 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.000616323, 2024-03-28T01:09:34.637 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T01:09:34.637 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:34.637 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 3, 2024-03-28T01:09:34.638 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000647239, 2024-03-28T01:09:34.638 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000215746 2024-03-28T01:09:34.638 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:34.638 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:34.638 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:34.638 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:34.638 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:34.638 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:34.638 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-28T01:09:34.638 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T01:09:34.638 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:34.638 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T01:09:34.638 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 1, 2024-03-28T01:09:34.638 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 191, 2024-03-28T01:09:34.638 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 160, 2024-03-28T01:09:34.639 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 111184, 2024-03-28T01:09:34.639 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 3, 2024-03-28T01:09:34.639 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 2, 2024-03-28T01:09:34.639 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.009526854, 2024-03-28T01:09:34.639 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.006942461, 2024-03-28T01:09:34.639 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.000777504, 2024-03-28T01:09:34.639 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T01:09:34.639 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:34.639 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 189, 2024-03-28T01:09:34.639 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.011559877, 2024-03-28T01:09:34.639 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000061163 2024-03-28T01:09:34.639 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:34.639 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:34.639 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:34.639 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:34.640 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:34.640 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:34.640 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-28T01:09:34.640 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 111184 2024-03-28T01:09:34.640 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:34.640 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T01:09:34.640 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 256, 2024-03-28T01:09:34.640 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 251, 2024-03-28T01:09:34.640 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 761696, 2024-03-28T01:09:34.640 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 150864, 2024-03-28T01:09:34.640 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 2, 2024-03-28T01:09:34.640 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 1, 2024-03-28T01:09:34.640 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.038908064, 2024-03-28T01:09:34.640 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.013139455, 2024-03-28T01:09:34.640 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.021406454, 2024-03-28T01:09:34.641 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.006647822, 2024-03-28T01:09:34.641 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T01:09:34.641 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 250, 2024-03-28T01:09:34.641 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.010014155, 2024-03-28T01:09:34.641 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000040056 2024-03-28T01:09:34.641 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:34.641 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-03-28T01:09:34.641 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:34.641 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:34.641 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:34.641 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:34.641 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 761696, 2024-03-28T01:09:34.641 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 150864 2024-03-28T01:09:34.641 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:34.641 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-03-28T01:09:34.641 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 0, 2024-03-28T01:09:34.642 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-03-28T01:09:34.642 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:34.642 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-03-28T01:09:34.642 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-03-28T01:09:34.642 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-03-28T01:09:34.642 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T01:09:34.642 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-03-28T01:09:34.642 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T01:09:34.642 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-03-28T01:09:34.642 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T01:09:34.642 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T01:09:34.643 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T01:09:34.643 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T01:09:34.643 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T01:09:34.644 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-03-28T01:09:34.644 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T01:09:34.644 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T01:09:34.644 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-03-28T01:09:34.644 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-03-28T01:09:34.644 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-03-28T01:09:34.644 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-03-28T01:09:34.644 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T01:09:34.644 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T01:09:34.644 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T01:09:34.644 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T01:09:34.644 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T01:09:34.645 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-03-28T01:09:34.645 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-03-28T01:09:34.645 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-03-28T01:09:34.645 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T01:09:34.645 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T01:09:34.645 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-03-28T01:09:34.645 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-03-28T01:09:34.645 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-03-28T01:09:34.645 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-03-28T01:09:34.645 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T01:09:34.645 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-03-28T01:09:34.645 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T01:09:34.645 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-03-28T01:09:34.645 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 149047, 2024-03-28T01:09:34.645 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 31, 2024-03-28T01:09:34.646 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 0, 2024-03-28T01:09:34.646 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 0, 2024-03-28T01:09:34.646 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-03-28T01:09:34.646 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-03-28T01:09:34.646 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-03-28T01:09:34.646 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-03-28T01:09:34.646 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-03-28T01:09:34.646 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-03-28T01:09:34.646 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 0, 2024-03-28T01:09:34.646 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 0, 2024-03-28T01:09:34.646 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T01:09:34.646 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-03-28T01:09:34.646 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-03-28T01:09:34.647 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-03-28T01:09:34.647 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 928, 2024-03-28T01:09:34.647 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 2, 2024-03-28T01:09:34.647 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-03-28T01:09:34.647 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-03-28T01:09:34.647 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-03-28T01:09:34.647 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-03-28T01:09:34.647 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-03-28T01:09:34.647 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T01:09:34.647 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-03-28T01:09:34.647 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-03-28T01:09:34.647 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-03-28T01:09:34.647 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 257, 2024-03-28T01:09:34.647 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 725542, 2024-03-28T01:09:34.647 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-03-28T01:09:34.648 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 257, 2024-03-28T01:09:34.648 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-03-28T01:09:34.648 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-03-28T01:09:34.648 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 257, 2024-03-28T01:09:34.648 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 725542, 2024-03-28T01:09:34.648 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-03-28T01:09:34.648 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-03-28T01:09:34.648 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-03-28T01:09:34.648 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-03-28T01:09:34.648 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:34.648 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-03-28T01:09:34.648 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-03-28T01:09:35.375 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-03-28T01:09:35.375 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-03-28T01:09:35.375 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-03-28T01:09:35.375 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-03-28T01:09:35.375 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-28T01:09:35.386 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-28T01:09:35.386 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-28T01:09:35.386 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T01:09:35.417 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-28T01:09:35.417 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T01:09:35.476 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T01:09:35.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.629+0000 7f04f1059640 1 -- 172.21.15.169:0/3419364371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04ec104a70 msgr2=0x7f04ec106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:35.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.629+0000 7f04f1059640 1 --2- 172.21.15.169:0/3419364371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04ec104a70 0x7f04ec106e60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f04e0009900 tx=0x7f04e002bc30 comp rx=0 tx=0).stop 2024-03-28T01:09:35.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.629+0000 7f04f1059640 1 -- 172.21.15.169:0/3419364371 shutdown_connections 2024-03-28T01:09:35.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.629+0000 7f04f1059640 1 --2- 172.21.15.169:0/3419364371 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04ec1073a0 0x7f04ec1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:35.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.629+0000 7f04f1059640 1 --2- 172.21.15.169:0/3419364371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04ec104a70 0x7f04ec106e60 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:35.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.629+0000 7f04f1059640 1 --2- 172.21.15.169:0/3419364371 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04ec102140 0x7f04ec104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:35.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.629+0000 7f04f1059640 1 -- 172.21.15.169:0/3419364371 >> 172.21.15.169:0/3419364371 conn(0x7f04ec0fbb30 msgr2=0x7f04ec0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:35.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.629+0000 7f04f1059640 1 -- 172.21.15.169:0/3419364371 shutdown_connections 2024-03-28T01:09:35.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.629+0000 7f04f1059640 1 -- 172.21.15.169:0/3419364371 wait complete. 2024-03-28T01:09:35.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.629+0000 7f04f1059640 1 Processor -- start 2024-03-28T01:09:35.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.630+0000 7f04f1059640 1 -- start start 2024-03-28T01:09:35.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.630+0000 7f04f1059640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04ec102140 0x7f04ec101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:35.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.630+0000 7f04f1059640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04ec104a70 0x7f04ec1000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:35.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.630+0000 7f04f1059640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04ec1073a0 0x7f04ec100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:35.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.630+0000 7f04f1059640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec10d340 con 0x7f04ec104a70 2024-03-28T01:09:35.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.630+0000 7f04f1059640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec10d1c0 con 0x7f04ec102140 2024-03-28T01:09:35.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.630+0000 7f04f1059640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec10d4c0 con 0x7f04ec1073a0 2024-03-28T01:09:35.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.630+0000 7f04ea59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04ec104a70 0x7f04ec1000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:35.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.630+0000 7f04ea59c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04ec104a70 0x7f04ec1000f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60102/0 (socket says 172.21.15.169:60102) 2024-03-28T01:09:35.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.630+0000 7f04ea59c640 1 -- 172.21.15.169:0/4262446165 learned_addr learned my addr 172.21.15.169:0/4262446165 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:09:35.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.630+0000 7f04ead9d640 1 --2- 172.21.15.169:0/4262446165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04ec102140 0x7f04ec101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:35.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.630+0000 7f04eb59e640 1 --2- 172.21.15.169:0/4262446165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04ec1073a0 0x7f04ec100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:35.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.630+0000 7f04ea59c640 1 -- 172.21.15.169:0/4262446165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04ec1073a0 msgr2=0x7f04ec100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:35.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.630+0000 7f04ea59c640 1 --2- 172.21.15.169:0/4262446165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04ec1073a0 0x7f04ec100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:35.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.630+0000 7f04ea59c640 1 -- 172.21.15.169:0/4262446165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04ec102140 msgr2=0x7f04ec101ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:35.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.630+0000 7f04ea59c640 1 --2- 172.21.15.169:0/4262446165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04ec102140 0x7f04ec101ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:35.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.630+0000 7f04ea59c640 1 -- 172.21.15.169:0/4262446165 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ec100ec0 con 0x7f04ec104a70 2024-03-28T01:09:35.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.630+0000 7f04ea59c640 1 --2- 172.21.15.169:0/4262446165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04ec104a70 0x7f04ec1000f0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f04e0000c00 tx=0x7f04e002c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:35.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.631+0000 7f04d3fff640 1 -- 172.21.15.169:0/4262446165 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e0041070 con 0x7f04ec104a70 2024-03-28T01:09:35.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.631+0000 7f04f1059640 1 -- 172.21.15.169:0/4262446165 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ec1a1ab0 con 0x7f04ec104a70 2024-03-28T01:09:35.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.631+0000 7f04d3fff640 1 -- 172.21.15.169:0/4262446165 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04e0004450 con 0x7f04ec104a70 2024-03-28T01:09:35.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.631+0000 7f04d3fff640 1 -- 172.21.15.169:0/4262446165 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e0034440 con 0x7f04ec104a70 2024-03-28T01:09:35.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.631+0000 7f04f1059640 1 -- 172.21.15.169:0/4262446165 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ec1a2050 con 0x7f04ec104a70 2024-03-28T01:09:35.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.632+0000 7f04f1059640 1 -- 172.21.15.169:0/4262446165 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04ec0fd620 con 0x7f04ec104a70 2024-03-28T01:09:35.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.633+0000 7f04d3fff640 1 -- 172.21.15.169:0/4262446165 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f04e0044070 con 0x7f04ec104a70 2024-03-28T01:09:35.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.634+0000 7f04d3fff640 1 --2- 172.21.15.169:0/4262446165 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04d40a2800 0x7f04d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:35.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.634+0000 7f04ead9d640 1 --2- 172.21.15.169:0/4262446165 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04d40a2800 0x7f04d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:35.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.634+0000 7f04d3fff640 1 -- 172.21.15.169:0/4262446165 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f04e00f0ad0 con 0x7f04ec104a70 2024-03-28T01:09:35.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.634+0000 7f04ead9d640 1 --2- 172.21.15.169:0/4262446165 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04d40a2800 0x7f04d40a4cc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0095f0 tx=0x7f04dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:35.640 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.637+0000 7f04d3fff640 1 -- 172.21.15.169:0/4262446165 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f04e00bd210 con 0x7f04ec104a70 2024-03-28T01:09:35.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.815+0000 7f04f1059640 1 -- 172.21.15.169:0/4262446165 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04ec02d1f0 con 0x7f04ec104a70 2024-03-28T01:09:35.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.816+0000 7f04d3fff640 1 -- 172.21.15.169:0/4262446165 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+24823 (secure 0 0 0) 0x7f04e00c12b0 con 0x7f04ec104a70 2024-03-28T01:09:35.820 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T01:09:35.820 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":94,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:56:38.281826+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9},{"from":5,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T01:09:35.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.818+0000 7f04f1059640 1 -- 172.21.15.169:0/4262446165 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04d40a2800 msgr2=0x7f04d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:35.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.818+0000 7f04f1059640 1 --2- 172.21.15.169:0/4262446165 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04d40a2800 0x7f04d40a4cc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0095f0 tx=0x7f04dc005f90 comp rx=0 tx=0).stop 2024-03-28T01:09:35.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.818+0000 7f04f1059640 1 -- 172.21.15.169:0/4262446165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04ec104a70 msgr2=0x7f04ec1000f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:35.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.818+0000 7f04f1059640 1 --2- 172.21.15.169:0/4262446165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04ec104a70 0x7f04ec1000f0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f04e0000c00 tx=0x7f04e002c640 comp rx=0 tx=0).stop 2024-03-28T01:09:35.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.819+0000 7f04f1059640 1 -- 172.21.15.169:0/4262446165 shutdown_connections 2024-03-28T01:09:35.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.819+0000 7f04f1059640 1 --2- 172.21.15.169:0/4262446165 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04d40a2800 0x7f04d40a4cc0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:35.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.819+0000 7f04f1059640 1 --2- 172.21.15.169:0/4262446165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04ec1073a0 0x7f04ec100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:35.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.819+0000 7f04f1059640 1 --2- 172.21.15.169:0/4262446165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04ec104a70 0x7f04ec1000f0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:35.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.819+0000 7f04f1059640 1 --2- 172.21.15.169:0/4262446165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04ec102140 0x7f04ec101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:35.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.819+0000 7f04f1059640 1 -- 172.21.15.169:0/4262446165 >> 172.21.15.169:0/4262446165 conn(0x7f04ec0fbb30 msgr2=0x7f04ec102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:35.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.819+0000 7f04f1059640 1 -- 172.21.15.169:0/4262446165 shutdown_connections 2024-03-28T01:09:35.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.819+0000 7f04f1059640 1 -- 172.21.15.169:0/4262446165 wait complete. 2024-03-28T01:09:35.833 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T00:49:56.339525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T00:54:31.182562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T00:54:31.780429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T00:54:35.035724+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T01:09:35.834 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T01:09:35.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.936+0000 7f06c811b640 1 -- 172.21.15.169:0/887356007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06c01018f0 msgr2=0x7f06c0108180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:35.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.936+0000 7f06c811b640 1 --2- 172.21.15.169:0/887356007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06c01018f0 0x7f06c0108180 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f06b8009ca0 tx=0x7f06b802c0f0 comp rx=0 tx=0).stop 2024-03-28T01:09:35.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.936+0000 7f06c811b640 1 -- 172.21.15.169:0/887356007 shutdown_connections 2024-03-28T01:09:35.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.936+0000 7f06c811b640 1 --2- 172.21.15.169:0/887356007 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06c01018f0 0x7f06c0108180 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:35.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.936+0000 7f06c811b640 1 --2- 172.21.15.169:0/887356007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06c0100f30 0x7f06c01013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:35.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.936+0000 7f06c811b640 1 --2- 172.21.15.169:0/887356007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06c00ffd40 0x7f06c0100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:35.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.936+0000 7f06c811b640 1 -- 172.21.15.169:0/887356007 >> 172.21.15.169:0/887356007 conn(0x7f06c00fb2d0 msgr2=0x7f06c00fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:35.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.937+0000 7f06c811b640 1 -- 172.21.15.169:0/887356007 shutdown_connections 2024-03-28T01:09:35.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.937+0000 7f06c811b640 1 -- 172.21.15.169:0/887356007 wait complete. 2024-03-28T01:09:35.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.937+0000 7f06c811b640 1 Processor -- start 2024-03-28T01:09:35.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.937+0000 7f06c811b640 1 -- start start 2024-03-28T01:09:35.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.937+0000 7f06c811b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06c00ffd40 0x7f06c0196890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:35.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.937+0000 7f06c811b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06c0100f30 0x7f06c0196dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:35.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.937+0000 7f06c811b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06c01018f0 0x7f06c019de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:35.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.937+0000 7f06c811b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c010c250 con 0x7f06c01018f0 2024-03-28T01:09:35.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.937+0000 7f06c811b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c010c0d0 con 0x7f06c00ffd40 2024-03-28T01:09:35.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.937+0000 7f06c811b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c010c3d0 con 0x7f06c0100f30 2024-03-28T01:09:35.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.937+0000 7f06c66b8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06c01018f0 0x7f06c019de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:35.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.937+0000 7f06c66b8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06c01018f0 0x7f06c019de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60112/0 (socket says 172.21.15.169:60112) 2024-03-28T01:09:35.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.937+0000 7f06c66b8640 1 -- 172.21.15.169:0/1654604510 learned_addr learned my addr 172.21.15.169:0/1654604510 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:09:35.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.937+0000 7f06c56b6640 1 --2- 172.21.15.169:0/1654604510 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06c0100f30 0x7f06c0196dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:35.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.937+0000 7f06c5eb7640 1 --2- 172.21.15.169:0/1654604510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06c00ffd40 0x7f06c0196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:35.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.938+0000 7f06c66b8640 1 -- 172.21.15.169:0/1654604510 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06c0100f30 msgr2=0x7f06c0196dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:35.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.938+0000 7f06c66b8640 1 --2- 172.21.15.169:0/1654604510 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06c0100f30 0x7f06c0196dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:35.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.938+0000 7f06c66b8640 1 -- 172.21.15.169:0/1654604510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06c00ffd40 msgr2=0x7f06c0196890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:35.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.938+0000 7f06c66b8640 1 --2- 172.21.15.169:0/1654604510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06c00ffd40 0x7f06c0196890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:35.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.938+0000 7f06c66b8640 1 -- 172.21.15.169:0/1654604510 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c019e4b0 con 0x7f06c01018f0 2024-03-28T01:09:35.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.938+0000 7f06c66b8640 1 --2- 172.21.15.169:0/1654604510 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06c01018f0 0x7f06c019de40 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f06b8037040 tx=0x7f06b8032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:35.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.938+0000 7f06b6ffd640 1 -- 172.21.15.169:0/1654604510 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b8041020 con 0x7f06c01018f0 2024-03-28T01:09:35.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.938+0000 7f06b6ffd640 1 -- 172.21.15.169:0/1654604510 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f06b8034cd0 con 0x7f06c01018f0 2024-03-28T01:09:35.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.938+0000 7f06c811b640 1 -- 172.21.15.169:0/1654604510 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c019e740 con 0x7f06c01018f0 2024-03-28T01:09:35.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.938+0000 7f06c811b640 1 -- 172.21.15.169:0/1654604510 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c019ec30 con 0x7f06c01018f0 2024-03-28T01:09:35.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.938+0000 7f06b6ffd640 1 -- 172.21.15.169:0/1654604510 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b8039040 con 0x7f06c01018f0 2024-03-28T01:09:35.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.939+0000 7f06c811b640 1 -- 172.21.15.169:0/1654604510 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a40052c0 con 0x7f06c01018f0 2024-03-28T01:09:35.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.940+0000 7f06b6ffd640 1 -- 172.21.15.169:0/1654604510 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f06b8044020 con 0x7f06c01018f0 2024-03-28T01:09:35.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.941+0000 7f06b6ffd640 1 --2- 172.21.15.169:0/1654604510 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f06ac0a2800 0x7f06ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:35.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.941+0000 7f06b6ffd640 1 -- 172.21.15.169:0/1654604510 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f06b80f0b80 con 0x7f06c01018f0 2024-03-28T01:09:35.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.941+0000 7f06c5eb7640 1 --2- 172.21.15.169:0/1654604510 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f06ac0a2800 0x7f06ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:35.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.941+0000 7f06c5eb7640 1 --2- 172.21.15.169:0/1654604510 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f06ac0a2800 0x7f06ac0a4cc0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f06bc0095f0 tx=0x7f06bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:35.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:35.945+0000 7f06b6ffd640 1 -- 172.21.15.169:0/1654604510 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f06b804b020 con 0x7f06c01018f0 2024-03-28T01:09:36.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:35 smithi174 ceph-mon[36060]: pgmap v743: 129 pgs: 129 active+clean; 2.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 58 MiB/s wr, 499 op/s 2024-03-28T01:09:36.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:35 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/4262446165' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T01:09:36.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.121+0000 7f06c811b640 1 -- 172.21.15.169:0/1654604510 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f06a4005dc0 con 0x7f06c01018f0 2024-03-28T01:09:36.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.122+0000 7f06b6ffd640 1 -- 172.21.15.169:0/1654604510 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7f06b80bd910 con 0x7f06c01018f0 2024-03-28T01:09:36.125 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 1 2024-03-28T01:09:36.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.124+0000 7f06c811b640 1 -- 172.21.15.169:0/1654604510 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f06ac0a2800 msgr2=0x7f06ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:36.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.124+0000 7f06c811b640 1 --2- 172.21.15.169:0/1654604510 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f06ac0a2800 0x7f06ac0a4cc0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f06bc0095f0 tx=0x7f06bc005f90 comp rx=0 tx=0).stop 2024-03-28T01:09:36.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.124+0000 7f06c811b640 1 -- 172.21.15.169:0/1654604510 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06c01018f0 msgr2=0x7f06c019de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:36.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.124+0000 7f06c811b640 1 --2- 172.21.15.169:0/1654604510 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06c01018f0 0x7f06c019de40 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f06b8037040 tx=0x7f06b8032f40 comp rx=0 tx=0).stop 2024-03-28T01:09:36.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.124+0000 7f06c811b640 1 -- 172.21.15.169:0/1654604510 shutdown_connections 2024-03-28T01:09:36.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.124+0000 7f06c811b640 1 --2- 172.21.15.169:0/1654604510 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f06ac0a2800 0x7f06ac0a4cc0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.124+0000 7f06c811b640 1 --2- 172.21.15.169:0/1654604510 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f06c01018f0 0x7f06c019de40 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.124+0000 7f06c811b640 1 --2- 172.21.15.169:0/1654604510 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06c0100f30 0x7f06c0196dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.124+0000 7f06c811b640 1 --2- 172.21.15.169:0/1654604510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06c00ffd40 0x7f06c0196890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.124+0000 7f06c811b640 1 -- 172.21.15.169:0/1654604510 >> 172.21.15.169:0/1654604510 conn(0x7f06c00fb2d0 msgr2=0x7f06c0104230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:36.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.124+0000 7f06c811b640 1 -- 172.21.15.169:0/1654604510 shutdown_connections 2024-03-28T01:09:36.128 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.124+0000 7f06c811b640 1 -- 172.21.15.169:0/1654604510 wait complete. 2024-03-28T01:09:36.137 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:35 smithi169 ceph-mon[32274]: pgmap v743: 129 pgs: 129 active+clean; 2.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 58 MiB/s wr, 499 op/s 2024-03-28T01:09:36.137 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:35 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4262446165' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T01:09:36.138 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T01:09:36.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:35 smithi177 ceph-mon[36206]: pgmap v743: 129 pgs: 129 active+clean; 2.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 58 MiB/s wr, 499 op/s 2024-03-28T01:09:36.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:35 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/4262446165' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T01:09:36.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.240+0000 7f661bd7e640 1 -- 172.21.15.169:0/878768297 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6614101ba0 msgr2=0x7f6614101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:36.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.240+0000 7f661bd7e640 1 --2- 172.21.15.169:0/878768297 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6614101ba0 0x7f6614101fc0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f6610009930 tx=0x7f661002bc30 comp rx=0 tx=0).stop 2024-03-28T01:09:36.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.240+0000 7f661bd7e640 1 -- 172.21.15.169:0/878768297 shutdown_connections 2024-03-28T01:09:36.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.240+0000 7f661bd7e640 1 --2- 172.21.15.169:0/878768297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6614101200 0x7f6614108aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.240+0000 7f661bd7e640 1 --2- 172.21.15.169:0/878768297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6614100840 0x7f6614100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.240+0000 7f661bd7e640 1 --2- 172.21.15.169:0/878768297 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6614101ba0 0x7f6614101fc0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.246 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.240+0000 7f661bd7e640 1 -- 172.21.15.169:0/878768297 >> 172.21.15.169:0/878768297 conn(0x7f66140fb2d0 msgr2=0x7f66140fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:36.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.241+0000 7f661bd7e640 1 -- 172.21.15.169:0/878768297 shutdown_connections 2024-03-28T01:09:36.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.241+0000 7f661bd7e640 1 -- 172.21.15.169:0/878768297 wait complete. 2024-03-28T01:09:36.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.241+0000 7f661bd7e640 1 Processor -- start 2024-03-28T01:09:36.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.241+0000 7f661bd7e640 1 -- start start 2024-03-28T01:09:36.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.241+0000 7f661bd7e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6614100840 0x7f6614197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:36.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.241+0000 7f661bd7e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6614101200 0x7f6614197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:36.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.241+0000 7f661bd7e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6614101ba0 0x7f661419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:36.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.241+0000 7f661bd7e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661410d340 con 0x7f6614100840 2024-03-28T01:09:36.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.241+0000 7f661bd7e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661410d1c0 con 0x7f6614101ba0 2024-03-28T01:09:36.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.241+0000 7f661bd7e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661410d4c0 con 0x7f6614101200 2024-03-28T01:09:36.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.241+0000 7f6619b1a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6614100840 0x7f6614197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:36.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.241+0000 7f6619b1a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6614100840 0x7f6614197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:60130/0 (socket says 172.21.15.169:60130) 2024-03-28T01:09:36.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.241+0000 7f6619b1a640 1 -- 172.21.15.169:0/1456541546 learned_addr learned my addr 172.21.15.169:0/1456541546 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:09:36.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.241+0000 7f6619319640 1 --2- 172.21.15.169:0/1456541546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6614101200 0x7f6614197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:36.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.242+0000 7f661a31b640 1 --2- 172.21.15.169:0/1456541546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6614101ba0 0x7f661419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:36.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.242+0000 7f6619b1a640 1 -- 172.21.15.169:0/1456541546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6614101200 msgr2=0x7f6614197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:36.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.242+0000 7f6619b1a640 1 --2- 172.21.15.169:0/1456541546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6614101200 0x7f6614197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.242+0000 7f6619b1a640 1 -- 172.21.15.169:0/1456541546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6614101ba0 msgr2=0x7f661419eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:36.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.242+0000 7f6619b1a640 1 --2- 172.21.15.169:0/1456541546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6614101ba0 0x7f661419eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.242+0000 7f6619b1a640 1 -- 172.21.15.169:0/1456541546 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661419f660 con 0x7f6614100840 2024-03-28T01:09:36.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.242+0000 7f6619b1a640 1 --2- 172.21.15.169:0/1456541546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6614100840 0x7f6614197a40 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f661002c7b0 tx=0x7f661002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:36.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.242+0000 7f660affd640 1 -- 172.21.15.169:0/1456541546 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6610041070 con 0x7f6614100840 2024-03-28T01:09:36.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.242+0000 7f660affd640 1 -- 172.21.15.169:0/1456541546 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66100054e0 con 0x7f6614100840 2024-03-28T01:09:36.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.242+0000 7f660affd640 1 -- 172.21.15.169:0/1456541546 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6610034440 con 0x7f6614100840 2024-03-28T01:09:36.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.242+0000 7f661bd7e640 1 -- 172.21.15.169:0/1456541546 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661419f8f0 con 0x7f6614100840 2024-03-28T01:09:36.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.242+0000 7f661bd7e640 1 -- 172.21.15.169:0/1456541546 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661419fd90 con 0x7f6614100840 2024-03-28T01:09:36.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.243+0000 7f661bd7e640 1 -- 172.21.15.169:0/1456541546 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66141079f0 con 0x7f6614100840 2024-03-28T01:09:36.249 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.244+0000 7f660affd640 1 -- 172.21.15.169:0/1456541546 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6610044070 con 0x7f6614100840 2024-03-28T01:09:36.249 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.245+0000 7f660affd640 1 --2- 172.21.15.169:0/1456541546 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f66000a2800 0x7f66000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:36.249 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.245+0000 7f6619319640 1 --2- 172.21.15.169:0/1456541546 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f66000a2800 0x7f66000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:36.249 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.245+0000 7f660affd640 1 -- 172.21.15.169:0/1456541546 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f66100f0bd0 con 0x7f6614100840 2024-03-28T01:09:36.249 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.245+0000 7f6619319640 1 --2- 172.21.15.169:0/1456541546 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f66000a2800 0x7f66000a4cc0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f66040095f0 tx=0x7f6604005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:36.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.248+0000 7f660affd640 1 -- 172.21.15.169:0/1456541546 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f66100bd280 con 0x7f6614100840 2024-03-28T01:09:36.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.425+0000 7f661bd7e640 1 -- 172.21.15.169:0/1456541546 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f661402d2e0 con 0x7f6614100840 2024-03-28T01:09:36.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.425+0000 7f660affd640 1 -- 172.21.15.169:0/1456541546 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) v1 ==== 104+0+11 (secure 0 0 0) 0x7f66100c1320 con 0x7f6614100840 2024-03-28T01:09:36.428 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T01:09:36.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.427+0000 7f661bd7e640 1 -- 172.21.15.169:0/1456541546 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f66000a2800 msgr2=0x7f66000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:36.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.427+0000 7f661bd7e640 1 --2- 172.21.15.169:0/1456541546 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f66000a2800 0x7f66000a4cc0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f66040095f0 tx=0x7f6604005f90 comp rx=0 tx=0).stop 2024-03-28T01:09:36.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.427+0000 7f661bd7e640 1 -- 172.21.15.169:0/1456541546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6614100840 msgr2=0x7f6614197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:36.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.427+0000 7f661bd7e640 1 --2- 172.21.15.169:0/1456541546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6614100840 0x7f6614197a40 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f661002c7b0 tx=0x7f661002c650 comp rx=0 tx=0).stop 2024-03-28T01:09:36.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.427+0000 7f661bd7e640 1 -- 172.21.15.169:0/1456541546 shutdown_connections 2024-03-28T01:09:36.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.427+0000 7f661bd7e640 1 --2- 172.21.15.169:0/1456541546 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f66000a2800 0x7f66000a4cc0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.427+0000 7f661bd7e640 1 --2- 172.21.15.169:0/1456541546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6614101ba0 0x7f661419eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.427+0000 7f661bd7e640 1 --2- 172.21.15.169:0/1456541546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6614101200 0x7f6614197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.427+0000 7f661bd7e640 1 --2- 172.21.15.169:0/1456541546 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6614100840 0x7f6614197a40 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.427+0000 7f661bd7e640 1 -- 172.21.15.169:0/1456541546 >> 172.21.15.169:0/1456541546 conn(0x7f66140fb2d0 msgr2=0x7f6614104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:36.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.427+0000 7f661bd7e640 1 -- 172.21.15.169:0/1456541546 shutdown_connections 2024-03-28T01:09:36.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.427+0000 7f661bd7e640 1 -- 172.21.15.169:0/1456541546 wait complete. 2024-03-28T01:09:36.442 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T01:09:36.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.544+0000 7fe688ce8640 1 -- 172.21.15.169:0/2022076479 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6841017a0 msgr2=0x7fe684101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:36.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.544+0000 7fe688ce8640 1 --2- 172.21.15.169:0/2022076479 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6841017a0 0x7fe684101c20 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fe678009900 tx=0x7fe67802bc30 comp rx=0 tx=0).stop 2024-03-28T01:09:36.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.544+0000 7fe688ce8640 1 -- 172.21.15.169:0/2022076479 shutdown_connections 2024-03-28T01:09:36.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.544+0000 7fe688ce8640 1 --2- 172.21.15.169:0/2022076479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe684102160 0x7fe6841089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.544+0000 7fe688ce8640 1 --2- 172.21.15.169:0/2022076479 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6841017a0 0x7fe684101c20 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.544+0000 7fe688ce8640 1 --2- 172.21.15.169:0/2022076479 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6841005b0 0x7fe6841009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.544+0000 7fe688ce8640 1 -- 172.21.15.169:0/2022076479 >> 172.21.15.169:0/2022076479 conn(0x7fe6840fbb40 msgr2=0x7fe6840fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:36.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.544+0000 7fe688ce8640 1 -- 172.21.15.169:0/2022076479 shutdown_connections 2024-03-28T01:09:36.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.544+0000 7fe688ce8640 1 -- 172.21.15.169:0/2022076479 wait complete. 2024-03-28T01:09:36.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.545+0000 7fe688ce8640 1 Processor -- start 2024-03-28T01:09:36.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.545+0000 7fe688ce8640 1 -- start start 2024-03-28T01:09:36.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.545+0000 7fe688ce8640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6841005b0 0x7fe6841979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:36.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.545+0000 7fe688ce8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6841017a0 0x7fe684197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:36.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.545+0000 7fe688ce8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe684102160 0x7fe68419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:36.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.545+0000 7fe688ce8640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68410c210 con 0x7fe6841005b0 2024-03-28T01:09:36.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.545+0000 7fe688ce8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68410c090 con 0x7fe6841017a0 2024-03-28T01:09:36.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.545+0000 7fe688ce8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68410c390 con 0x7fe684102160 2024-03-28T01:09:36.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.545+0000 7fe68259c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6841005b0 0x7fe6841979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:36.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.545+0000 7fe68259c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6841005b0 0x7fe6841979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49412/0 (socket says 172.21.15.169:49412) 2024-03-28T01:09:36.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.545+0000 7fe68259c640 1 -- 172.21.15.169:0/1016967896 learned_addr learned my addr 172.21.15.169:0/1016967896 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:09:36.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.545+0000 7fe682d9d640 1 --2- 172.21.15.169:0/1016967896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe684102160 0x7fe68419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:36.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.545+0000 7fe681d9b640 1 --2- 172.21.15.169:0/1016967896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6841017a0 0x7fe684197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:36.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.545+0000 7fe68259c640 1 -- 172.21.15.169:0/1016967896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe684102160 msgr2=0x7fe68419ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:36.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.545+0000 7fe68259c640 1 --2- 172.21.15.169:0/1016967896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe684102160 0x7fe68419ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.545+0000 7fe68259c640 1 -- 172.21.15.169:0/1016967896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6841017a0 msgr2=0x7fe684197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:36.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.545+0000 7fe68259c640 1 --2- 172.21.15.169:0/1016967896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6841017a0 0x7fe684197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.545+0000 7fe68259c640 1 -- 172.21.15.169:0/1016967896 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe68419f5f0 con 0x7fe6841005b0 2024-03-28T01:09:36.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.545+0000 7fe681d9b640 1 --2- 172.21.15.169:0/1016967896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6841017a0 0x7fe684197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:09:36.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.546+0000 7fe68259c640 1 --2- 172.21.15.169:0/1016967896 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6841005b0 0x7fe6841979d0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fe67400e9c0 tx=0x7fe67400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:36.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.546+0000 7fe6737fe640 1 -- 172.21.15.169:0/1016967896 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67400cde0 con 0x7fe6841005b0 2024-03-28T01:09:36.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.546+0000 7fe6737fe640 1 -- 172.21.15.169:0/1016967896 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe674010910 con 0x7fe6841005b0 2024-03-28T01:09:36.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.546+0000 7fe688ce8640 1 -- 172.21.15.169:0/1016967896 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe68419f8e0 con 0x7fe6841005b0 2024-03-28T01:09:36.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.546+0000 7fe6737fe640 1 -- 172.21.15.169:0/1016967896 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe674018a00 con 0x7fe6841005b0 2024-03-28T01:09:36.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.546+0000 7fe688ce8640 1 -- 172.21.15.169:0/1016967896 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe68419fe00 con 0x7fe6841005b0 2024-03-28T01:09:36.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.547+0000 7fe688ce8640 1 -- 172.21.15.169:0/1016967896 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6841a0030 con 0x7fe6841005b0 2024-03-28T01:09:36.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.548+0000 7fe6737fe640 1 -- 172.21.15.169:0/1016967896 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe67401f020 con 0x7fe6841005b0 2024-03-28T01:09:36.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.548+0000 7fe6737fe640 1 --2- 172.21.15.169:0/1016967896 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe66c0a2800 0x7fe66c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:36.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.548+0000 7fe681d9b640 1 --2- 172.21.15.169:0/1016967896 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe66c0a2800 0x7fe66c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:36.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.548+0000 7fe6737fe640 1 -- 172.21.15.169:0/1016967896 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fe6740cf5b0 con 0x7fe6841005b0 2024-03-28T01:09:36.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.549+0000 7fe681d9b640 1 --2- 172.21.15.169:0/1016967896 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe66c0a2800 0x7fe66c0a4cc0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fe67802c7b0 tx=0x7fe678005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:36.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.552+0000 7fe6737fe640 1 -- 172.21.15.169:0/1016967896 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe67409c2b0 con 0x7fe6841005b0 2024-03-28T01:09:36.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.727+0000 7fe688ce8640 1 -- 172.21.15.169:0/1016967896 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe6841a03c0 con 0x7fe6841005b0 2024-03-28T01:09:36.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.728+0000 7fe6737fe640 1 -- 172.21.15.169:0/1016967896 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe67409bbb0 con 0x7fe6841005b0 2024-03-28T01:09:36.731 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 64 2024-03-28T01:09:36.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.730+0000 7fe688ce8640 1 -- 172.21.15.169:0/1016967896 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe66c0a2800 msgr2=0x7fe66c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:36.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.730+0000 7fe688ce8640 1 --2- 172.21.15.169:0/1016967896 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe66c0a2800 0x7fe66c0a4cc0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fe67802c7b0 tx=0x7fe678005fd0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.730+0000 7fe688ce8640 1 -- 172.21.15.169:0/1016967896 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6841005b0 msgr2=0x7fe6841979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:36.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.730+0000 7fe688ce8640 1 --2- 172.21.15.169:0/1016967896 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6841005b0 0x7fe6841979d0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fe67400e9c0 tx=0x7fe67400ee80 comp rx=0 tx=0).stop 2024-03-28T01:09:36.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.730+0000 7fe688ce8640 1 -- 172.21.15.169:0/1016967896 shutdown_connections 2024-03-28T01:09:36.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.730+0000 7fe688ce8640 1 --2- 172.21.15.169:0/1016967896 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fe66c0a2800 0x7fe66c0a4cc0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.730+0000 7fe688ce8640 1 --2- 172.21.15.169:0/1016967896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe684102160 0x7fe68419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.730+0000 7fe688ce8640 1 --2- 172.21.15.169:0/1016967896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6841017a0 0x7fe684197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.730+0000 7fe688ce8640 1 --2- 172.21.15.169:0/1016967896 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6841005b0 0x7fe6841979d0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.730+0000 7fe688ce8640 1 -- 172.21.15.169:0/1016967896 >> 172.21.15.169:0/1016967896 conn(0x7fe6840fbb40 msgr2=0x7fe684069c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:36.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.730+0000 7fe688ce8640 1 -- 172.21.15.169:0/1016967896 shutdown_connections 2024-03-28T01:09:36.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.730+0000 7fe688ce8640 1 -- 172.21.15.169:0/1016967896 wait complete. 2024-03-28T01:09:36.744 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T01:09:36.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.848+0000 7f04611a9640 1 -- 172.21.15.169:0/3367566659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f045c1005a0 msgr2=0x7f045c1009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:36.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.848+0000 7f04611a9640 1 --2- 172.21.15.169:0/3367566659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f045c1005a0 0x7f045c1009c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f044c009ca0 tx=0x7f044c02c0f0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.848+0000 7f04611a9640 1 -- 172.21.15.169:0/3367566659 shutdown_connections 2024-03-28T01:09:36.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.848+0000 7f04611a9640 1 --2- 172.21.15.169:0/3367566659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f045c102150 0x7f045c1089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.848+0000 7f04611a9640 1 --2- 172.21.15.169:0/3367566659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f045c101790 0x7f045c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.848+0000 7f04611a9640 1 --2- 172.21.15.169:0/3367566659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f045c1005a0 0x7f045c1009c0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.848+0000 7f04611a9640 1 -- 172.21.15.169:0/3367566659 >> 172.21.15.169:0/3367566659 conn(0x7f045c0fbb30 msgr2=0x7f045c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:36.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.848+0000 7f04611a9640 1 -- 172.21.15.169:0/3367566659 shutdown_connections 2024-03-28T01:09:36.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.848+0000 7f04611a9640 1 -- 172.21.15.169:0/3367566659 wait complete. 2024-03-28T01:09:36.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.848+0000 7f04611a9640 1 Processor -- start 2024-03-28T01:09:36.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.849+0000 7f04611a9640 1 -- start start 2024-03-28T01:09:36.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.849+0000 7f04611a9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f045c1005a0 0x7f045c197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:36.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.849+0000 7f04611a9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f045c101790 0x7f045c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:36.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.849+0000 7f04611a9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f045c102150 0x7f045c19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:36.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.849+0000 7f04611a9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045c10c230 con 0x7f045c1005a0 2024-03-28T01:09:36.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.849+0000 7f04611a9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045c10c0b0 con 0x7f045c102150 2024-03-28T01:09:36.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.849+0000 7f04611a9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045c10c3b0 con 0x7f045c101790 2024-03-28T01:09:36.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.849+0000 7f045ad9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f045c1005a0 0x7f045c197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:36.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.849+0000 7f045ad9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f045c1005a0 0x7f045c197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49434/0 (socket says 172.21.15.169:49434) 2024-03-28T01:09:36.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.849+0000 7f045ad9d640 1 -- 172.21.15.169:0/2405462205 learned_addr learned my addr 172.21.15.169:0/2405462205 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:09:36.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.849+0000 7f045a59c640 1 --2- 172.21.15.169:0/2405462205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f045c101790 0x7f045c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:36.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.849+0000 7f045b59e640 1 --2- 172.21.15.169:0/2405462205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f045c102150 0x7f045c19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:36.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.849+0000 7f045ad9d640 1 -- 172.21.15.169:0/2405462205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f045c101790 msgr2=0x7f045c197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:36.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.849+0000 7f045ad9d640 1 --2- 172.21.15.169:0/2405462205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f045c101790 0x7f045c197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.849+0000 7f045ad9d640 1 -- 172.21.15.169:0/2405462205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f045c102150 msgr2=0x7f045c19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:36.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.849+0000 7f045ad9d640 1 --2- 172.21.15.169:0/2405462205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f045c102150 0x7f045c19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:36.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.849+0000 7f045ad9d640 1 -- 172.21.15.169:0/2405462205 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045c19f660 con 0x7f045c1005a0 2024-03-28T01:09:36.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.849+0000 7f045a59c640 1 --2- 172.21.15.169:0/2405462205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f045c101790 0x7f045c197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:09:36.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.849+0000 7f045ad9d640 1 --2- 172.21.15.169:0/2405462205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f045c1005a0 0x7f045c197a40 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f044c000f80 tx=0x7f044c032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:36.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.850+0000 7f0443fff640 1 -- 172.21.15.169:0/2405462205 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044c041020 con 0x7f045c1005a0 2024-03-28T01:09:36.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.850+0000 7f04611a9640 1 -- 172.21.15.169:0/2405462205 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f045c19f8f0 con 0x7f045c1005a0 2024-03-28T01:09:36.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.850+0000 7f0443fff640 1 -- 172.21.15.169:0/2405462205 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f044c034cd0 con 0x7f045c1005a0 2024-03-28T01:09:36.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.850+0000 7f0443fff640 1 -- 172.21.15.169:0/2405462205 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044c039040 con 0x7f045c1005a0 2024-03-28T01:09:36.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.850+0000 7f04611a9640 1 -- 172.21.15.169:0/2405462205 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f045c19fdb0 con 0x7f045c1005a0 2024-03-28T01:09:36.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.851+0000 7f04611a9640 1 -- 172.21.15.169:0/2405462205 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f045c19ffb0 con 0x7f045c1005a0 2024-03-28T01:09:36.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.852+0000 7f0443fff640 1 -- 172.21.15.169:0/2405462205 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f044c044020 con 0x7f045c1005a0 2024-03-28T01:09:36.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.852+0000 7f0443fff640 1 --2- 172.21.15.169:0/2405462205 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04440a2800 0x7f04440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:36.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.852+0000 7f045a59c640 1 --2- 172.21.15.169:0/2405462205 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04440a2800 0x7f04440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:36.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.852+0000 7f0443fff640 1 -- 172.21.15.169:0/2405462205 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f044c0f0ba0 con 0x7f045c1005a0 2024-03-28T01:09:36.857 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.853+0000 7f045a59c640 1 --2- 172.21.15.169:0/2405462205 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04440a2800 0x7f04440a4cc0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f045c198f50 tx=0x7f0450005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:36.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:36.857+0000 7f0443fff640 1 -- 172.21.15.169:0/2405462205 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f044c04b020 con 0x7f045c1005a0 2024-03-28T01:09:37.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.033+0000 7f04611a9640 1 -- 172.21.15.169:0/2405462205 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f045c1a0180 con 0x7f045c1005a0 2024-03-28T01:09:37.037 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.033+0000 7f0443fff640 1 -- 172.21.15.169:0/2405462205 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) v1 ==== 103+0+11 (secure 0 0 0) 0x7f044c0348f0 con 0x7f045c1005a0 2024-03-28T01:09:37.037 INFO:teuthology.orchestra.run.smithi169.stdout:pg_num: 32 2024-03-28T01:09:37.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.035+0000 7f04611a9640 1 -- 172.21.15.169:0/2405462205 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04440a2800 msgr2=0x7f04440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:37.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.035+0000 7f04611a9640 1 --2- 172.21.15.169:0/2405462205 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04440a2800 0x7f04440a4cc0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f045c198f50 tx=0x7f0450005f90 comp rx=0 tx=0).stop 2024-03-28T01:09:37.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.035+0000 7f04611a9640 1 -- 172.21.15.169:0/2405462205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f045c1005a0 msgr2=0x7f045c197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:37.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.035+0000 7f04611a9640 1 --2- 172.21.15.169:0/2405462205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f045c1005a0 0x7f045c197a40 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f044c000f80 tx=0x7f044c032ec0 comp rx=0 tx=0).stop 2024-03-28T01:09:37.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.035+0000 7f04611a9640 1 -- 172.21.15.169:0/2405462205 shutdown_connections 2024-03-28T01:09:37.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.035+0000 7f04611a9640 1 --2- 172.21.15.169:0/2405462205 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04440a2800 0x7f04440a4cc0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:37.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.035+0000 7f04611a9640 1 --2- 172.21.15.169:0/2405462205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f045c102150 0x7f045c19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:37.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.035+0000 7f04611a9640 1 --2- 172.21.15.169:0/2405462205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f045c101790 0x7f045c197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:37.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.035+0000 7f04611a9640 1 --2- 172.21.15.169:0/2405462205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f045c1005a0 0x7f045c197a40 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:37.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.035+0000 7f04611a9640 1 -- 172.21.15.169:0/2405462205 >> 172.21.15.169:0/2405462205 conn(0x7f045c0fbb30 msgr2=0x7f045c068800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:37.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.036+0000 7f04611a9640 1 -- 172.21.15.169:0/2405462205 shutdown_connections 2024-03-28T01:09:37.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.036+0000 7f04611a9640 1 -- 172.21.15.169:0/2405462205 wait complete. 2024-03-28T01:09:37.049 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:36 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1654604510' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T01:09:37.049 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:36 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1456541546' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T01:09:37.049 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:36 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1016967896' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T01:09:37.050 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T01:09:37.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:36 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1654604510' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T01:09:37.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:36 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1456541546' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T01:09:37.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:36 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1016967896' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T01:09:37.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:36 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1654604510' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T01:09:37.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:36 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1456541546' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T01:09:37.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:36 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1016967896' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T01:09:37.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.239+0000 7f0d25c9a640 1 -- 172.21.15.169:0/1518472721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d201028a0 msgr2=0x7f0d20104d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:37.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.239+0000 7f0d25c9a640 1 --2- 172.21.15.169:0/1518472721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d201028a0 0x7f0d20104d10 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f0d10009900 tx=0x7f0d1002bc60 comp rx=0 tx=0).stop 2024-03-28T01:09:37.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.245+0000 7f0d25c9a640 1 -- 172.21.15.169:0/1518472721 shutdown_connections 2024-03-28T01:09:37.249 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.245+0000 7f0d25c9a640 1 --2- 172.21.15.169:0/1518472721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d20105250 0x7f0d201056d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:37.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.245+0000 7f0d25c9a640 1 --2- 172.21.15.169:0/1518472721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d201028a0 0x7f0d20104d10 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:37.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.245+0000 7f0d25c9a640 1 --2- 172.21.15.169:0/1518472721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d200fff30 0x7f0d20102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:37.250 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.245+0000 7f0d25c9a640 1 -- 172.21.15.169:0/1518472721 >> 172.21.15.169:0/1518472721 conn(0x7f0d200fbb50 msgr2=0x7f0d200fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:37.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.251+0000 7f0d25c9a640 1 -- 172.21.15.169:0/1518472721 shutdown_connections 2024-03-28T01:09:37.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.252+0000 7f0d25c9a640 1 -- 172.21.15.169:0/1518472721 wait complete. 2024-03-28T01:09:37.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.253+0000 7f0d25c9a640 1 Processor -- start 2024-03-28T01:09:37.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.253+0000 7f0d25c9a640 1 -- start start 2024-03-28T01:09:37.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.253+0000 7f0d25c9a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d200fff30 0x7f0d20072ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:37.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.253+0000 7f0d25c9a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d201028a0 0x7f0d200730e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:37.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.253+0000 7f0d25c9a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d20105250 0x7f0d200712e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:37.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.253+0000 7f0d25c9a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2006bb10 con 0x7f0d20105250 2024-03-28T01:09:37.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.253+0000 7f0d25c9a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2006b990 con 0x7f0d201028a0 2024-03-28T01:09:37.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.253+0000 7f0d25c9a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2006bc90 con 0x7f0d200fff30 2024-03-28T01:09:37.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.255+0000 7f0d1f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d200fff30 0x7f0d20072ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:37.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.255+0000 7f0d1f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d200fff30 0x7f0d20072ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.169:48294/0 (socket says 172.21.15.169:48294) 2024-03-28T01:09:37.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.255+0000 7f0d1f7fe640 1 -- 172.21.15.169:0/133019154 learned_addr learned my addr 172.21.15.169:0/133019154 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:09:37.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.255+0000 7f0d17fff640 1 --2- 172.21.15.169:0/133019154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d201028a0 0x7f0d200730e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:37.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.257+0000 7f0d1ffff640 1 --2- 172.21.15.169:0/133019154 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d20105250 0x7f0d200712e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:37.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.260+0000 7f0d17fff640 1 -- 172.21.15.169:0/133019154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d200fff30 msgr2=0x7f0d20072ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:37.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.260+0000 7f0d17fff640 1 --2- 172.21.15.169:0/133019154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d200fff30 0x7f0d20072ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:37.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.260+0000 7f0d17fff640 1 -- 172.21.15.169:0/133019154 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d20105250 msgr2=0x7f0d200712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:37.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.260+0000 7f0d17fff640 1 --2- 172.21.15.169:0/133019154 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d20105250 0x7f0d200712e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:37.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.260+0000 7f0d17fff640 1 -- 172.21.15.169:0/133019154 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d20071a60 con 0x7f0d201028a0 2024-03-28T01:09:37.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.265+0000 7f0d17fff640 1 --2- 172.21.15.169:0/133019154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d201028a0 0x7f0d200730e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0d10036040 tx=0x7f0d1002c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:37.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.276+0000 7f0d1d7fa640 1 -- 172.21.15.169:0/133019154 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10041070 con 0x7f0d201028a0 2024-03-28T01:09:37.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.276+0000 7f0d1d7fa640 1 -- 172.21.15.169:0/133019154 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d100054e0 con 0x7f0d201028a0 2024-03-28T01:09:37.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.276+0000 7f0d1d7fa640 1 -- 172.21.15.169:0/133019154 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10034440 con 0x7f0d201028a0 2024-03-28T01:09:37.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.279+0000 7f0d25c9a640 1 -- 172.21.15.169:0/133019154 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d20071cf0 con 0x7f0d201028a0 2024-03-28T01:09:37.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.279+0000 7f0d25c9a640 1 -- 172.21.15.169:0/133019154 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d2010c580 con 0x7f0d201028a0 2024-03-28T01:09:37.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.282+0000 7f0d25c9a640 1 -- 172.21.15.169:0/133019154 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d20067c00 con 0x7f0d201028a0 2024-03-28T01:09:37.290 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.287+0000 7f0d1d7fa640 1 -- 172.21.15.169:0/133019154 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0d10044070 con 0x7f0d201028a0 2024-03-28T01:09:37.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.288+0000 7f0d1d7fa640 1 --2- 172.21.15.169:0/133019154 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0d0c0a2a00 0x7f0d0c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:37.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.289+0000 7f0d1d7fa640 1 -- 172.21.15.169:0/133019154 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f0d100f0bd0 con 0x7f0d201028a0 2024-03-28T01:09:37.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.290+0000 7f0d1f7fe640 1 --2- 172.21.15.169:0/133019154 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0d0c0a2a00 0x7f0d0c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:37.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.292+0000 7f0d1f7fe640 1 --2- 172.21.15.169:0/133019154 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0d0c0a2a00 0x7f0d0c0a4ec0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f0d20072950 tx=0x7f0d08009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:37.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.294+0000 7f0d1d7fa640 1 -- 172.21.15.169:0/133019154 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0d100f0e20 con 0x7f0d201028a0 2024-03-28T01:09:37.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.617+0000 7f0d25c9a640 1 -- 172.21.15.169:0/133019154 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0d200fd640 con 0x7f0d201028a0 2024-03-28T01:09:37.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.618+0000 7f0d1d7fa640 1 -- 172.21.15.169:0/133019154 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v94) v1 ==== 81+0+600 (secure 0 0 0) 0x7f0d100bd900 con 0x7f0d201028a0 2024-03-28T01:09:37.621 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:0/2299683991 2024-03-29T00:47:35.997770+0000 2024-03-28T01:09:37.621 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:6801/3995234407 2024-03-29T00:47:35.997770+0000 2024-03-28T01:09:37.621 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:0/3955974323 2024-03-29T00:47:10.143139+0000 2024-03-28T01:09:37.621 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:6800/3995234407 2024-03-29T00:47:35.997770+0000 2024-03-28T01:09:37.621 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:0/1567973250 2024-03-29T00:47:10.143139+0000 2024-03-28T01:09:37.621 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:0/1799473331 2024-03-29T00:47:10.143139+0000 2024-03-28T01:09:37.621 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:0/1949169969 2024-03-29T00:47:35.997770+0000 2024-03-28T01:09:37.621 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:6800/311965133 2024-03-29T00:47:10.143139+0000 2024-03-28T01:09:37.622 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:6801/311965133 2024-03-29T00:47:10.143139+0000 2024-03-28T01:09:37.622 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169:0/2928378386 2024-03-29T00:47:35.997770+0000 2024-03-28T01:09:37.622 INFO:teuthology.orchestra.run.smithi169.stderr:listed 10 entries 2024-03-28T01:09:37.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.620+0000 7f0d25c9a640 1 -- 172.21.15.169:0/133019154 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0d0c0a2a00 msgr2=0x7f0d0c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:37.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.620+0000 7f0d25c9a640 1 --2- 172.21.15.169:0/133019154 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0d0c0a2a00 0x7f0d0c0a4ec0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f0d20072950 tx=0x7f0d08009190 comp rx=0 tx=0).stop 2024-03-28T01:09:37.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.620+0000 7f0d25c9a640 1 -- 172.21.15.169:0/133019154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d201028a0 msgr2=0x7f0d200730e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:37.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.620+0000 7f0d25c9a640 1 --2- 172.21.15.169:0/133019154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d201028a0 0x7f0d200730e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0d10036040 tx=0x7f0d1002c670 comp rx=0 tx=0).stop 2024-03-28T01:09:37.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.620+0000 7f0d25c9a640 1 -- 172.21.15.169:0/133019154 shutdown_connections 2024-03-28T01:09:37.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.620+0000 7f0d25c9a640 1 --2- 172.21.15.169:0/133019154 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f0d0c0a2a00 0x7f0d0c0a4ec0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:37.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.620+0000 7f0d25c9a640 1 --2- 172.21.15.169:0/133019154 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0d20105250 0x7f0d200712e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:37.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.620+0000 7f0d25c9a640 1 --2- 172.21.15.169:0/133019154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d201028a0 0x7f0d200730e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:37.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.620+0000 7f0d25c9a640 1 --2- 172.21.15.169:0/133019154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d200fff30 0x7f0d20072ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:37.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.620+0000 7f0d25c9a640 1 -- 172.21.15.169:0/133019154 >> 172.21.15.169:0/133019154 conn(0x7f0d200fbb50 msgr2=0x7f0d20100870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:37.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.620+0000 7f0d25c9a640 1 -- 172.21.15.169:0/133019154 shutdown_connections 2024-03-28T01:09:37.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:37.620+0000 7f0d25c9a640 1 -- 172.21.15.169:0/133019154 wait complete. 2024-03-28T01:09:37.635 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-28T01:09:37.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T01:09:37.635 DEBUG:teuthology.orchestra.run.smithi169:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-28T01:09:37.751 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi169.front.sepia.ceph.com 2024-03-28T01:09:37.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T01:09:37.751 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-28T01:09:37.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T01:09:37.798 DEBUG:teuthology.orchestra.run.smithi169:> ip netns list 2024-03-28T01:09:38.029 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-28T01:09:38.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T01:09:38.029 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T01:09:38.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:37 smithi169 ceph-mon[32274]: pgmap v744: 129 pgs: 129 active+clean; 2.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 74 MiB/s wr, 668 op/s 2024-03-28T01:09:38.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:37 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2405462205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T01:09:38.051 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:37 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/133019154' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T01:09:38.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T01:09:38.057 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link delete ceph-brx 2024-03-28T01:09:38.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:37 smithi174 ceph-mon[36060]: pgmap v744: 129 pgs: 129 active+clean; 2.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 74 MiB/s wr, 668 op/s 2024-03-28T01:09:38.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:37 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2405462205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T01:09:38.109 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:37 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/133019154' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T01:09:38.159 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-28T01:09:38.169 INFO:tasks.cephadm:Teardown begin 2024-03-28T01:09:38.170 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T01:09:38.170 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T01:09:38.170 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:37 smithi177 ceph-mon[36206]: pgmap v744: 129 pgs: 129 active+clean; 2.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 74 MiB/s wr, 668 op/s 2024-03-28T01:09:38.170 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:37 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2405462205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T01:09:38.171 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:37 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/133019154' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T01:09:38.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.272+0000 7fd8736c0640 1 -- 172.21.15.169:0/1217461722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd86c100100 msgr2=0x7fd86c1005b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:38.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.272+0000 7fd8736c0640 1 --2- 172.21.15.169:0/1217461722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd86c100100 0x7fd86c1005b0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd864009ca0 tx=0x7fd86402c0f0 comp rx=0 tx=0).stop 2024-03-28T01:09:38.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.272+0000 7fd8736c0640 1 -- 172.21.15.169:0/1217461722 shutdown_connections 2024-03-28T01:09:38.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.272+0000 7fd8736c0640 1 --2- 172.21.15.169:0/1217461722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd86c100100 0x7fd86c1005b0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:38.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.272+0000 7fd8736c0640 1 --2- 172.21.15.169:0/1217461722 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd86c0ff740 0x7fd86c0ffbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:38.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.272+0000 7fd8736c0640 1 --2- 172.21.15.169:0/1217461722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd86c101c50 0x7fd86c102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:38.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.272+0000 7fd8736c0640 1 -- 172.21.15.169:0/1217461722 >> 172.21.15.169:0/1217461722 conn(0x7fd86c0fb2c0 msgr2=0x7fd86c0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:38.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.272+0000 7fd8736c0640 1 -- 172.21.15.169:0/1217461722 shutdown_connections 2024-03-28T01:09:38.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.272+0000 7fd8736c0640 1 -- 172.21.15.169:0/1217461722 wait complete. 2024-03-28T01:09:38.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.273+0000 7fd8736c0640 1 Processor -- start 2024-03-28T01:09:38.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.273+0000 7fd8736c0640 1 -- start start 2024-03-28T01:09:38.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.273+0000 7fd8736c0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd86c0ff740 0x7fd86c199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:38.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.273+0000 7fd8736c0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd86c100100 0x7fd86c19a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:38.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.273+0000 7fd8736c0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd86c101c50 0x7fd86c1a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:38.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.273+0000 7fd8736c0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c10f560 con 0x7fd86c100100 2024-03-28T01:09:38.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.273+0000 7fd8736c0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c10f3e0 con 0x7fd86c101c50 2024-03-28T01:09:38.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.273+0000 7fd8736c0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c10f6e0 con 0x7fd86c0ff740 2024-03-28T01:09:38.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.273+0000 7fd870c5b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd86c100100 0x7fd86c19a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:38.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.273+0000 7fd87145c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd86c0ff740 0x7fd86c199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:38.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.273+0000 7fd870c5b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd86c100100 0x7fd86c19a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49476/0 (socket says 172.21.15.169:49476) 2024-03-28T01:09:38.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.273+0000 7fd870c5b640 1 -- 172.21.15.169:0/4203334146 learned_addr learned my addr 172.21.15.169:0/4203334146 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:09:38.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.273+0000 7fd871c5d640 1 --2- 172.21.15.169:0/4203334146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd86c101c50 0x7fd86c1a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:38.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.273+0000 7fd87145c640 1 -- 172.21.15.169:0/4203334146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd86c101c50 msgr2=0x7fd86c1a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:38.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.273+0000 7fd87145c640 1 --2- 172.21.15.169:0/4203334146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd86c101c50 0x7fd86c1a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:38.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.273+0000 7fd87145c640 1 -- 172.21.15.169:0/4203334146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd86c100100 msgr2=0x7fd86c19a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:38.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.273+0000 7fd87145c640 1 --2- 172.21.15.169:0/4203334146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd86c100100 0x7fd86c19a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:38.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.273+0000 7fd87145c640 1 -- 172.21.15.169:0/4203334146 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c1a1790 con 0x7fd86c0ff740 2024-03-28T01:09:38.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.273+0000 7fd871c5d640 1 --2- 172.21.15.169:0/4203334146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd86c101c50 0x7fd86c1a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:09:38.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.273+0000 7fd870c5b640 1 --2- 172.21.15.169:0/4203334146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd86c100100 0x7fd86c19a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:09:38.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.274+0000 7fd87145c640 1 --2- 172.21.15.169:0/4203334146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd86c0ff740 0x7fd86c199b70 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd86800e9f0 tx=0x7fd86800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:38.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.274+0000 7fd8627fc640 1 -- 172.21.15.169:0/4203334146 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86800ccc0 con 0x7fd86c0ff740 2024-03-28T01:09:38.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.274+0000 7fd8627fc640 1 -- 172.21.15.169:0/4203334146 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd868010910 con 0x7fd86c0ff740 2024-03-28T01:09:38.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.274+0000 7fd8736c0640 1 -- 172.21.15.169:0/4203334146 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86c1a1a80 con 0x7fd86c0ff740 2024-03-28T01:09:38.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.274+0000 7fd8627fc640 1 -- 172.21.15.169:0/4203334146 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868018960 con 0x7fd86c0ff740 2024-03-28T01:09:38.280 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.274+0000 7fd8736c0640 1 -- 172.21.15.169:0/4203334146 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd86c1a1fd0 con 0x7fd86c0ff740 2024-03-28T01:09:38.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.275+0000 7fd8736c0640 1 -- 172.21.15.169:0/4203334146 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd86c10b6a0 con 0x7fd86c0ff740 2024-03-28T01:09:38.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.276+0000 7fd8627fc640 1 -- 172.21.15.169:0/4203334146 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd868010ac0 con 0x7fd86c0ff740 2024-03-28T01:09:38.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.277+0000 7fd8627fc640 1 --2- 172.21.15.169:0/4203334146 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd8580a2800 0x7fd8580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:38.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.277+0000 7fd870c5b640 1 --2- 172.21.15.169:0/4203334146 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd8580a2800 0x7fd8580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:38.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.277+0000 7fd8627fc640 1 -- 172.21.15.169:0/4203334146 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fd8680cf6b0 con 0x7fd86c0ff740 2024-03-28T01:09:38.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.277+0000 7fd870c5b640 1 --2- 172.21.15.169:0/4203334146 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd8580a2800 0x7fd8580a4cc0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd86c19b080 tx=0x7fd85c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:38.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.280+0000 7fd8627fc640 1 -- 172.21.15.169:0/4203334146 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd86809bd00 con 0x7fd86c0ff740 2024-03-28T01:09:38.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.553+0000 7fd8736c0640 1 -- 172.21.15.169:0/4203334146 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd86c02d070 con 0x7fd86c0ff740 2024-03-28T01:09:38.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.554+0000 7fd8627fc640 1 -- 172.21.15.169:0/4203334146 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7fd86809fda0 con 0x7fd86c0ff740 2024-03-28T01:09:38.557 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T01:09:38.557 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T01:09:38.557 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T01:09:38.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.557+0000 7fd8736c0640 1 -- 172.21.15.169:0/4203334146 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd8580a2800 msgr2=0x7fd8580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:38.560 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.557+0000 7fd8736c0640 1 --2- 172.21.15.169:0/4203334146 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd8580a2800 0x7fd8580a4cc0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd86c19b080 tx=0x7fd85c005f90 comp rx=0 tx=0).stop 2024-03-28T01:09:38.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.557+0000 7fd8736c0640 1 -- 172.21.15.169:0/4203334146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd86c0ff740 msgr2=0x7fd86c199b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:38.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.557+0000 7fd8736c0640 1 --2- 172.21.15.169:0/4203334146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd86c0ff740 0x7fd86c199b70 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd86800e9f0 tx=0x7fd86800eeb0 comp rx=0 tx=0).stop 2024-03-28T01:09:38.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.558+0000 7fd8736c0640 1 -- 172.21.15.169:0/4203334146 shutdown_connections 2024-03-28T01:09:38.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.558+0000 7fd8736c0640 1 --2- 172.21.15.169:0/4203334146 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fd8580a2800 0x7fd8580a4cc0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:38.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.558+0000 7fd8736c0640 1 --2- 172.21.15.169:0/4203334146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd86c101c50 0x7fd86c1a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:38.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.558+0000 7fd8736c0640 1 --2- 172.21.15.169:0/4203334146 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd86c100100 0x7fd86c19a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:38.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.558+0000 7fd8736c0640 1 --2- 172.21.15.169:0/4203334146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd86c0ff740 0x7fd86c199b70 secure :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd86800e9f0 tx=0x7fd86800eeb0 comp rx=0 tx=0).stop 2024-03-28T01:09:38.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.558+0000 7fd8736c0640 1 -- 172.21.15.169:0/4203334146 >> 172.21.15.169:0/4203334146 conn(0x7fd86c0fb2c0 msgr2=0x7fd86c0fc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:38.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.558+0000 7fd8736c0640 1 -- 172.21.15.169:0/4203334146 shutdown_connections 2024-03-28T01:09:38.562 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.559+0000 7fd8736c0640 1 -- 172.21.15.169:0/4203334146 wait complete. 2024-03-28T01:09:38.582 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T01:09:38.761 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.756+0000 7fb510acc640 1 -- 172.21.15.169:0/534109185 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb50c10d9e0 msgr2=0x7fb50c10de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:38.761 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.756+0000 7fb510acc640 1 --2- 172.21.15.169:0/534109185 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb50c10d9e0 0x7fb50c10de90 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc01a910 tx=0x7fb4fc03cc80 comp rx=0 tx=0).stop 2024-03-28T01:09:38.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.759+0000 7fb510acc640 1 -- 172.21.15.169:0/534109185 shutdown_connections 2024-03-28T01:09:38.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.759+0000 7fb510acc640 1 --2- 172.21.15.169:0/534109185 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb50c10d9e0 0x7fb50c10de90 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:38.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.759+0000 7fb510acc640 1 --2- 172.21.15.169:0/534109185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb50c10b010 0x7fb50c10d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:38.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.759+0000 7fb510acc640 1 --2- 172.21.15.169:0/534109185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb50c1086a0 0x7fb50c10aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:38.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.759+0000 7fb510acc640 1 -- 172.21.15.169:0/534109185 >> 172.21.15.169:0/534109185 conn(0x7fb50c06e510 msgr2=0x7fb50c070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:38.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.760+0000 7fb510acc640 1 -- 172.21.15.169:0/534109185 shutdown_connections 2024-03-28T01:09:38.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.762+0000 7fb510acc640 1 -- 172.21.15.169:0/534109185 wait complete. 2024-03-28T01:09:38.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.763+0000 7fb510acc640 1 Processor -- start 2024-03-28T01:09:38.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.765+0000 7fb510acc640 1 -- start start 2024-03-28T01:09:38.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.765+0000 7fb510acc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb50c1086a0 0x7fb50c19e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:38.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.765+0000 7fb510acc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb50c10b010 0x7fb50c19e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:38.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.765+0000 7fb510acc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb50c10d9e0 0x7fb50c1a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:38.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.765+0000 7fb510acc640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50c111a50 con 0x7fb50c10d9e0 2024-03-28T01:09:38.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.765+0000 7fb510acc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50c1118d0 con 0x7fb50c1086a0 2024-03-28T01:09:38.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.765+0000 7fb510acc640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50c111bd0 con 0x7fb50c10b010 2024-03-28T01:09:38.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.765+0000 7fb50a59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb50c1086a0 0x7fb50c19e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:38.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.765+0000 7fb50a59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb50c1086a0 0x7fb50c19e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:46450/0 (socket says 172.21.15.169:46450) 2024-03-28T01:09:38.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.765+0000 7fb50a59c640 1 -- 172.21.15.169:0/2187864309 learned_addr learned my addr 172.21.15.169:0/2187864309 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:09:38.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.765+0000 7fb509d9b640 1 --2- 172.21.15.169:0/2187864309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb50c10b010 0x7fb50c19e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:38.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.766+0000 7fb50a59c640 1 -- 172.21.15.169:0/2187864309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb50c10b010 msgr2=0x7fb50c19e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:38.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.766+0000 7fb50a59c640 1 --2- 172.21.15.169:0/2187864309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb50c10b010 0x7fb50c19e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:38.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.766+0000 7fb50a59c640 1 -- 172.21.15.169:0/2187864309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb50c10d9e0 msgr2=0x7fb50c1a5730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T01:09:38.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.766+0000 7fb50a59c640 1 --2- 172.21.15.169:0/2187864309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb50c10d9e0 0x7fb50c1a5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:38.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.766+0000 7fb50a59c640 1 -- 172.21.15.169:0/2187864309 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb50c1a5da0 con 0x7fb50c1086a0 2024-03-28T01:09:38.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.766+0000 7fb50a59c640 1 --2- 172.21.15.169:0/2187864309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb50c1086a0 0x7fb50c19e1d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb50000b450 tx=0x7fb50000b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:38.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.767+0000 7fb4fb7fe640 1 -- 172.21.15.169:0/2187864309 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb500015020 con 0x7fb50c1086a0 2024-03-28T01:09:38.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.767+0000 7fb4fb7fe640 1 -- 172.21.15.169:0/2187864309 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb500011d70 con 0x7fb50c1086a0 2024-03-28T01:09:38.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.767+0000 7fb4fb7fe640 1 -- 172.21.15.169:0/2187864309 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50000f340 con 0x7fb50c1086a0 2024-03-28T01:09:38.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.767+0000 7fb510acc640 1 -- 172.21.15.169:0/2187864309 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb50c1a6090 con 0x7fb50c1086a0 2024-03-28T01:09:38.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.767+0000 7fb510acc640 1 -- 172.21.15.169:0/2187864309 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb50c1a6610 con 0x7fb50c1086a0 2024-03-28T01:09:38.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.769+0000 7fb510acc640 1 -- 172.21.15.169:0/2187864309 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4ec005290 con 0x7fb50c1086a0 2024-03-28T01:09:38.777 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.773+0000 7fb4fb7fe640 1 -- 172.21.15.169:0/2187864309 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb500018020 con 0x7fb50c1086a0 2024-03-28T01:09:38.777 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.774+0000 7fb4fb7fe640 1 --2- 172.21.15.169:0/2187864309 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb4f40a2980 0x7fb4f40a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:38.777 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.774+0000 7fb4fb7fe640 1 -- 172.21.15.169:0/2187864309 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7fb5000cec50 con 0x7fb50c1086a0 2024-03-28T01:09:38.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.775+0000 7fb509d9b640 1 --2- 172.21.15.169:0/2187864309 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb4f40a2980 0x7fb4f40a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:38.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.775+0000 7fb509d9b640 1 --2- 172.21.15.169:0/2187864309 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb4f40a2980 0x7fb4f40a4e40 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fb50c19f690 tx=0x7fb50400d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:38.788 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:38.781+0000 7fb4fb7fe640 1 -- 172.21.15.169:0/2187864309 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb50000d040 con 0x7fb50c1086a0 2024-03-28T01:09:39.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:38 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/4203334146' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:09:39.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.196+0000 7fb510acc640 1 -- 172.21.15.169:0/2187864309 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4ec005580 con 0x7fb50c1086a0 2024-03-28T01:09:39.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.198+0000 7fb4fb7fe640 1 -- 172.21.15.169:0/2187864309 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11420 (secure 0 0 0) 0x7fb50009b9d0 con 0x7fb50c1086a0 2024-03-28T01:09:39.201 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T01:09:39.201 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6835/3964885254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6834","nonce":3964885254},{"type":"v1","addr":"172.21.15.169:6835","nonce":3964885254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6837/2130558218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6836","nonce":2130558218},{"type":"v1","addr":"172.21.15.169:6837","nonce":2130558218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6839/4072876730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6838","nonce":4072876730},{"type":"v1","addr":"172.21.15.169:6839","nonce":4072876730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.169:6841/2980464895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6840","nonce":2980464895},{"type":"v1","addr":"172.21.15.169:6841","nonce":2980464895}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1840169743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1840169743},{"type":"v1","addr":"172.21.15.174:6833","nonce":1840169743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3466292652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3466292652},{"type":"v1","addr":"172.21.15.177:6833","nonce":3466292652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T00:54:32.613335+0000","modified":"2024-03-28T00:56:45.097353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24478,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3213155599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3213155599},{"type":"v1","addr":"172.21.15.174:6835","nonce":3213155599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/1050217570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1050217570},{"type":"v1","addr":"172.21.15.177:6835","nonce":1050217570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24452":{"gid":24452,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/3629228577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3629228577},{"type":"v1","addr":"172.21.15.174:6837","nonce":3629228577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.177:6837/3931247435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":3931247435},{"type":"v1","addr":"172.21.15.177:6837","nonce":3931247435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6839/152605729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":152605729},{"type":"v1","addr":"172.21.15.177:6839","nonce":152605729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6839/1152414321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":1152414321},{"type":"v1","addr":"172.21.15.174:6839","nonce":1152414321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24479,24478,24457]},"id":1}]} 2024-03-28T01:09:39.201 INFO:teuthology.orchestra.run.smithi169.stderr:dumped fsmap epoch 25 2024-03-28T01:09:39.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.201+0000 7fb4f97fa640 1 -- 172.21.15.169:0/2187864309 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb4f40a2980 msgr2=0x7fb4f40a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:39.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.201+0000 7fb4f97fa640 1 --2- 172.21.15.169:0/2187864309 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb4f40a2980 0x7fb4f40a4e40 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fb50c19f690 tx=0x7fb50400d040 comp rx=0 tx=0).stop 2024-03-28T01:09:39.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.201+0000 7fb4f97fa640 1 -- 172.21.15.169:0/2187864309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb50c1086a0 msgr2=0x7fb50c19e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:39.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.201+0000 7fb4f97fa640 1 --2- 172.21.15.169:0/2187864309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb50c1086a0 0x7fb50c19e1d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb50000b450 tx=0x7fb50000b910 comp rx=0 tx=0).stop 2024-03-28T01:09:39.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.201+0000 7fb4f97fa640 1 -- 172.21.15.169:0/2187864309 shutdown_connections 2024-03-28T01:09:39.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.201+0000 7fb4f97fa640 1 --2- 172.21.15.169:0/2187864309 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7fb4f40a2980 0x7fb4f40a4e40 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.201+0000 7fb4f97fa640 1 --2- 172.21.15.169:0/2187864309 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb50c10d9e0 0x7fb50c1a5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.201+0000 7fb4f97fa640 1 --2- 172.21.15.169:0/2187864309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb50c10b010 0x7fb50c19e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.201+0000 7fb4f97fa640 1 --2- 172.21.15.169:0/2187864309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb50c1086a0 0x7fb50c19e1d0 secure :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb50000b450 tx=0x7fb50000b910 comp rx=0 tx=0).stop 2024-03-28T01:09:39.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.201+0000 7fb4f97fa640 1 -- 172.21.15.169:0/2187864309 >> 172.21.15.169:0/2187864309 conn(0x7fb50c06e510 msgr2=0x7fb50c10b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:39.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.201+0000 7fb4f97fa640 1 -- 172.21.15.169:0/2187864309 shutdown_connections 2024-03-28T01:09:39.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.201+0000 7fb4f97fa640 1 -- 172.21.15.169:0/2187864309 wait complete. 2024-03-28T01:09:39.221 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:38 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/4203334146' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:09:39.223 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T01:09:39.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:38 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/4203334146' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:09:39.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.405+0000 7f96dad1b640 1 -- 172.21.15.169:0/1278520582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96d410d9e0 msgr2=0x7f96d410de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:39.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.405+0000 7f96dad1b640 1 --2- 172.21.15.169:0/1278520582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96d410d9e0 0x7f96d410de90 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009900 tx=0x7f96c402bc60 comp rx=0 tx=0).stop 2024-03-28T01:09:39.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.406+0000 7f96dad1b640 1 -- 172.21.15.169:0/1278520582 shutdown_connections 2024-03-28T01:09:39.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.406+0000 7f96dad1b640 1 --2- 172.21.15.169:0/1278520582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96d410d9e0 0x7f96d410de90 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.406+0000 7f96dad1b640 1 --2- 172.21.15.169:0/1278520582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96d410b010 0x7f96d410d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.406+0000 7f96dad1b640 1 --2- 172.21.15.169:0/1278520582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96d41086a0 0x7f96d410aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.406+0000 7f96dad1b640 1 -- 172.21.15.169:0/1278520582 >> 172.21.15.169:0/1278520582 conn(0x7f96d406e510 msgr2=0x7f96d4070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:39.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.406+0000 7f96dad1b640 1 -- 172.21.15.169:0/1278520582 shutdown_connections 2024-03-28T01:09:39.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.406+0000 7f96dad1b640 1 -- 172.21.15.169:0/1278520582 wait complete. 2024-03-28T01:09:39.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.407+0000 7f96dad1b640 1 Processor -- start 2024-03-28T01:09:39.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.407+0000 7f96dad1b640 1 -- start start 2024-03-28T01:09:39.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.407+0000 7f96dad1b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96d41086a0 0x7f96d4114d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:39.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.407+0000 7f96dad1b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96d410b010 0x7f96d4115260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:39.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.408+0000 7f96dad1b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96d410d9e0 0x7f96d41b6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:39.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.408+0000 7f96dad1b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d411c190 con 0x7f96d410b010 2024-03-28T01:09:39.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.408+0000 7f96dad1b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d411c010 con 0x7f96d41086a0 2024-03-28T01:09:39.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.408+0000 7f96dad1b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d411c310 con 0x7f96d410d9e0 2024-03-28T01:09:39.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.408+0000 7f96d8ab7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96d41086a0 0x7f96d4114d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:39.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.408+0000 7f96d8ab7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96d41086a0 0x7f96d4114d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.169:46460/0 (socket says 172.21.15.169:46460) 2024-03-28T01:09:39.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.408+0000 7f96d8ab7640 1 -- 172.21.15.169:0/2121708360 learned_addr learned my addr 172.21.15.169:0/2121708360 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:09:39.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.408+0000 7f96d3fff640 1 --2- 172.21.15.169:0/2121708360 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96d410b010 0x7f96d4115260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:39.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.408+0000 7f96d92b8640 1 --2- 172.21.15.169:0/2121708360 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96d410d9e0 0x7f96d41b6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:39.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.408+0000 7f96d8ab7640 1 -- 172.21.15.169:0/2121708360 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96d410d9e0 msgr2=0x7f96d41b6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:39.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.408+0000 7f96d8ab7640 1 --2- 172.21.15.169:0/2121708360 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96d410d9e0 0x7f96d41b6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.408+0000 7f96d8ab7640 1 -- 172.21.15.169:0/2121708360 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96d410b010 msgr2=0x7f96d4115260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:39.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.408+0000 7f96d8ab7640 1 --2- 172.21.15.169:0/2121708360 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96d410b010 0x7f96d4115260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.408+0000 7f96d8ab7640 1 -- 172.21.15.169:0/2121708360 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d41b6d80 con 0x7f96d41086a0 2024-03-28T01:09:39.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.408+0000 7f96d8ab7640 1 --2- 172.21.15.169:0/2121708360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96d41086a0 0x7f96d4114d20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009fd0 tx=0x7f96cc00c270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:39.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.409+0000 7f96d1ffb640 1 -- 172.21.15.169:0/2121708360 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc011470 con 0x7f96d41086a0 2024-03-28T01:09:39.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.409+0000 7f96dad1b640 1 -- 172.21.15.169:0/2121708360 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96d41b6f50 con 0x7f96d41086a0 2024-03-28T01:09:39.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.409+0000 7f96dad1b640 1 -- 172.21.15.169:0/2121708360 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96d41b7350 con 0x7f96d41086a0 2024-03-28T01:09:39.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.409+0000 7f96d1ffb640 1 -- 172.21.15.169:0/2121708360 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96cc011af0 con 0x7f96d41086a0 2024-03-28T01:09:39.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.409+0000 7f96d1ffb640 1 -- 172.21.15.169:0/2121708360 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc01ab90 con 0x7f96d41086a0 2024-03-28T01:09:39.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.411+0000 7f96dad1b640 1 -- 172.21.15.169:0/2121708360 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c00052c0 con 0x7f96d41086a0 2024-03-28T01:09:39.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.412+0000 7f96d1ffb640 1 -- 172.21.15.169:0/2121708360 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f96cc011610 con 0x7f96d41086a0 2024-03-28T01:09:39.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.413+0000 7f96d1ffb640 1 --2- 172.21.15.169:0/2121708360 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f96bc0a28d0 0x7f96bc0a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:39.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.413+0000 7f96d1ffb640 1 -- 172.21.15.169:0/2121708360 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f96cc0cf2c0 con 0x7f96d41086a0 2024-03-28T01:09:39.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.414+0000 7f96d3fff640 1 --2- 172.21.15.169:0/2121708360 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f96bc0a28d0 0x7f96bc0a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:39.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.414+0000 7f96d3fff640 1 --2- 172.21.15.169:0/2121708360 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f96bc0a28d0 0x7f96bc0a4d90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f96d4116230 tx=0x7f96c800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:39.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.416+0000 7f96d1ffb640 1 -- 172.21.15.169:0/2121708360 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f96cc016030 con 0x7f96d41086a0 2024-03-28T01:09:39.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.596+0000 7f96dad1b640 1 -- 172.21.15.169:0/2121708360 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96c0005880 con 0x7f96d41086a0 2024-03-28T01:09:39.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.598+0000 7f96d1ffb640 1 -- 172.21.15.169:0/2121708360 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+24823 (secure 0 0 0) 0x7f96cc09c040 con 0x7f96d41086a0 2024-03-28T01:09:39.601 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-28T01:09:39.601 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":94,"fsid":"833368da-ec9c-11ee-95d0-87774f69a715","created":"2024-03-28T00:46:42.432153+0000","modified":"2024-03-28T00:56:38.281826+0000","last_up_change":"2024-03-28T00:53:42.689299+0000","last_in_change":"2024-03-28T00:53:22.226921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T00:49:56.339525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T00:54:31.182562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T00:54:31.780429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T00:54:35.035724+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37a188e7-9de4-4f22-a2d2-f524c64e1923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6803","nonce":1362598737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6805","nonce":1362598737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6809","nonce":1362598737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":1362598737},{"type":"v1","addr":"172.21.15.169:6807","nonce":1362598737}]},"public_addr":"172.21.15.169:6803/1362598737","cluster_addr":"172.21.15.169:6805/1362598737","heartbeat_back_addr":"172.21.15.169:6809/1362598737","heartbeat_front_addr":"172.21.15.169:6807/1362598737","state":["exists","up"]},{"osd":1,"uuid":"269d9b9a-b3cd-4dda-bc9e-7d980cad5d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6801","nonce":3333696361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6803","nonce":3333696361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6807","nonce":3333696361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3333696361},{"type":"v1","addr":"172.21.15.174:6805","nonce":3333696361}]},"public_addr":"172.21.15.174:6801/3333696361","cluster_addr":"172.21.15.174:6803/3333696361","heartbeat_back_addr":"172.21.15.174:6807/3333696361","heartbeat_front_addr":"172.21.15.174:6805/3333696361","state":["exists","up"]},{"osd":2,"uuid":"4a6108e3-e6eb-489b-8f90-1caba1b3da34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6801","nonce":1495865602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6803","nonce":1495865602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6807","nonce":1495865602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1495865602},{"type":"v1","addr":"172.21.15.177:6805","nonce":1495865602}]},"public_addr":"172.21.15.177:6801/1495865602","cluster_addr":"172.21.15.177:6803/1495865602","heartbeat_back_addr":"172.21.15.177:6807/1495865602","heartbeat_front_addr":"172.21.15.177:6805/1495865602","state":["exists","up"]},{"osd":3,"uuid":"a7eae99d-476b-43c0-968f-8b5bf6d07222","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6811","nonce":149376866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6813","nonce":149376866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6817","nonce":149376866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":149376866},{"type":"v1","addr":"172.21.15.169:6815","nonce":149376866}]},"public_addr":"172.21.15.169:6811/149376866","cluster_addr":"172.21.15.169:6813/149376866","heartbeat_back_addr":"172.21.15.169:6817/149376866","heartbeat_front_addr":"172.21.15.169:6815/149376866","state":["exists","up"]},{"osd":4,"uuid":"73da9d2f-f108-40a2-a13d-639b240706f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6809","nonce":701854773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6811","nonce":701854773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6815","nonce":701854773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":701854773},{"type":"v1","addr":"172.21.15.174:6813","nonce":701854773}]},"public_addr":"172.21.15.174:6809/701854773","cluster_addr":"172.21.15.174:6811/701854773","heartbeat_back_addr":"172.21.15.174:6815/701854773","heartbeat_front_addr":"172.21.15.174:6813/701854773","state":["exists","up"]},{"osd":5,"uuid":"dd80a3c2-8ec1-495c-a8cb-b662242df637","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6809","nonce":3583242703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6811","nonce":3583242703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6815","nonce":3583242703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3583242703},{"type":"v1","addr":"172.21.15.177:6813","nonce":3583242703}]},"public_addr":"172.21.15.177:6809/3583242703","cluster_addr":"172.21.15.177:6811/3583242703","heartbeat_back_addr":"172.21.15.177:6815/3583242703","heartbeat_front_addr":"172.21.15.177:6813/3583242703","state":["exists","up"]},{"osd":6,"uuid":"7f15ef55-6eb2-4b77-891b-bb7a5fcd70c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6819","nonce":783670252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6821","nonce":783670252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6825","nonce":783670252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":783670252},{"type":"v1","addr":"172.21.15.169:6823","nonce":783670252}]},"public_addr":"172.21.15.169:6819/783670252","cluster_addr":"172.21.15.169:6821/783670252","heartbeat_back_addr":"172.21.15.169:6825/783670252","heartbeat_front_addr":"172.21.15.169:6823/783670252","state":["exists","up"]},{"osd":7,"uuid":"5378575b-122b-4369-9fc8-1230aee2dbd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6817","nonce":2274626586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6819","nonce":2274626586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6823","nonce":2274626586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2274626586},{"type":"v1","addr":"172.21.15.174:6821","nonce":2274626586}]},"public_addr":"172.21.15.174:6817/2274626586","cluster_addr":"172.21.15.174:6819/2274626586","heartbeat_back_addr":"172.21.15.174:6823/2274626586","heartbeat_front_addr":"172.21.15.174:6821/2274626586","state":["exists","up"]},{"osd":8,"uuid":"874c0869-f1fe-4927-9549-61d6e46c498a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6817","nonce":4108616824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6819","nonce":4108616824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6823","nonce":4108616824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4108616824},{"type":"v1","addr":"172.21.15.177:6821","nonce":4108616824}]},"public_addr":"172.21.15.177:6817/4108616824","cluster_addr":"172.21.15.177:6819/4108616824","heartbeat_back_addr":"172.21.15.177:6823/4108616824","heartbeat_front_addr":"172.21.15.177:6821/4108616824","state":["exists","up"]},{"osd":9,"uuid":"e6a429af-798a-48e3-8368-ad9338daa9b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6827","nonce":3660250290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6829","nonce":3660250290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6832","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6833","nonce":3660250290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":3660250290},{"type":"v1","addr":"172.21.15.169:6831","nonce":3660250290}]},"public_addr":"172.21.15.169:6827/3660250290","cluster_addr":"172.21.15.169:6829/3660250290","heartbeat_back_addr":"172.21.15.169:6833/3660250290","heartbeat_front_addr":"172.21.15.169:6831/3660250290","state":["exists","up"]},{"osd":10,"uuid":"ba098e93-e8fd-4dca-b2ff-1722425e2524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6825","nonce":3371468956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6827","nonce":3371468956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6831","nonce":3371468956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":3371468956},{"type":"v1","addr":"172.21.15.174:6829","nonce":3371468956}]},"public_addr":"172.21.15.174:6825/3371468956","cluster_addr":"172.21.15.174:6827/3371468956","heartbeat_back_addr":"172.21.15.174:6831/3371468956","heartbeat_front_addr":"172.21.15.174:6829/3371468956","state":["exists","up"]},{"osd":11,"uuid":"b1e65122-4e70-4022-933a-c9bf2f0f2e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6825","nonce":2109004868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6827","nonce":2109004868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6831","nonce":2109004868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2109004868},{"type":"v1","addr":"172.21.15.177:6829","nonce":2109004868}]},"public_addr":"172.21.15.177:6825/2109004868","cluster_addr":"172.21.15.177:6827/2109004868","heartbeat_back_addr":"172.21.15.177:6831/2109004868","heartbeat_front_addr":"172.21.15.177:6829/2109004868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:07.482024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:30.871963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:49:54.179658+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:18.707886+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:50:43.579515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:07.921457+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:32.462648+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:51:57.707683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:23.393399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:52:48.777428+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:14.350705+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T00:53:40.530209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9},{"from":5,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.169:0/2299683991":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6801/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/3955974323":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6800/3995234407":"2024-03-29T00:47:35.997770+0000","172.21.15.169:0/1567973250":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1799473331":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/1949169969":"2024-03-29T00:47:35.997770+0000","172.21.15.169:6800/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:6801/311965133":"2024-03-29T00:47:10.143139+0000","172.21.15.169:0/2928378386":"2024-03-29T00:47:35.997770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T01:09:39.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.600+0000 7f96dad1b640 1 -- 172.21.15.169:0/2121708360 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f96bc0a28d0 msgr2=0x7f96bc0a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:39.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.600+0000 7f96dad1b640 1 --2- 172.21.15.169:0/2121708360 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f96bc0a28d0 0x7f96bc0a4d90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f96d4116230 tx=0x7f96c800b390 comp rx=0 tx=0).stop 2024-03-28T01:09:39.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.600+0000 7f96dad1b640 1 -- 172.21.15.169:0/2121708360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96d41086a0 msgr2=0x7f96d4114d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:39.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.600+0000 7f96dad1b640 1 --2- 172.21.15.169:0/2121708360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96d41086a0 0x7f96d4114d20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009fd0 tx=0x7f96cc00c270 comp rx=0 tx=0).stop 2024-03-28T01:09:39.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.600+0000 7f96dad1b640 1 -- 172.21.15.169:0/2121708360 shutdown_connections 2024-03-28T01:09:39.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.600+0000 7f96dad1b640 1 --2- 172.21.15.169:0/2121708360 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f96bc0a28d0 0x7f96bc0a4d90 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.600+0000 7f96dad1b640 1 --2- 172.21.15.169:0/2121708360 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96d410d9e0 0x7f96d41b6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.600+0000 7f96dad1b640 1 --2- 172.21.15.169:0/2121708360 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f96d410b010 0x7f96d4115260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.600+0000 7f96dad1b640 1 --2- 172.21.15.169:0/2121708360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96d41086a0 0x7f96d4114d20 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.600+0000 7f96dad1b640 1 -- 172.21.15.169:0/2121708360 >> 172.21.15.169:0/2121708360 conn(0x7f96d406e510 msgr2=0x7f96d410a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:39.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.600+0000 7f96dad1b640 1 -- 172.21.15.169:0/2121708360 shutdown_connections 2024-03-28T01:09:39.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.600+0000 7f96dad1b640 1 -- 172.21.15.169:0/2121708360 wait complete. 2024-03-28T01:09:39.615 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-28T01:09:39.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.716+0000 7f39b15b9640 1 -- 172.21.15.169:0/852536468 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39ac1017a0 msgr2=0x7f39ac101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:39.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.716+0000 7f39b15b9640 1 --2- 172.21.15.169:0/852536468 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39ac1017a0 0x7f39ac101c20 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f3994009900 tx=0x7f399402bc30 comp rx=0 tx=0).stop 2024-03-28T01:09:39.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.716+0000 7f39b15b9640 1 -- 172.21.15.169:0/852536468 shutdown_connections 2024-03-28T01:09:39.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.716+0000 7f39b15b9640 1 --2- 172.21.15.169:0/852536468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39ac102160 0x7f39ac1089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.716+0000 7f39b15b9640 1 --2- 172.21.15.169:0/852536468 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39ac1017a0 0x7f39ac101c20 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.716+0000 7f39b15b9640 1 --2- 172.21.15.169:0/852536468 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39ac1005b0 0x7f39ac1009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.716+0000 7f39b15b9640 1 -- 172.21.15.169:0/852536468 >> 172.21.15.169:0/852536468 conn(0x7f39ac0fbb40 msgr2=0x7f39ac0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:39.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.716+0000 7f39b15b9640 1 -- 172.21.15.169:0/852536468 shutdown_connections 2024-03-28T01:09:39.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.716+0000 7f39b15b9640 1 -- 172.21.15.169:0/852536468 wait complete. 2024-03-28T01:09:39.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.717+0000 7f39b15b9640 1 Processor -- start 2024-03-28T01:09:39.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.717+0000 7f39b15b9640 1 -- start start 2024-03-28T01:09:39.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.717+0000 7f39b15b9640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39ac1005b0 0x7f39ac06a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:39.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.717+0000 7f39b15b9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39ac1017a0 0x7f39ac19c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:39.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.717+0000 7f39b15b9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39ac102160 0x7f39ac19ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:39.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.717+0000 7f39b15b9640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac10c230 con 0x7f39ac1005b0 2024-03-28T01:09:39.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.717+0000 7f39b15b9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac10c0b0 con 0x7f39ac1017a0 2024-03-28T01:09:39.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.717+0000 7f39b15b9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac10c3b0 con 0x7f39ac102160 2024-03-28T01:09:39.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.717+0000 7f39aaffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39ac1005b0 0x7f39ac06a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:39.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.717+0000 7f39aaffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39ac1005b0 0x7f39ac06a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49500/0 (socket says 172.21.15.169:49500) 2024-03-28T01:09:39.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.717+0000 7f39aaffd640 1 -- 172.21.15.169:0/64644448 learned_addr learned my addr 172.21.15.169:0/64644448 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:09:39.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.717+0000 7f39a3fff640 1 --2- 172.21.15.169:0/64644448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39ac1017a0 0x7f39ac19c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:39.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.717+0000 7f39ab7fe640 1 --2- 172.21.15.169:0/64644448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39ac102160 0x7f39ac19ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:39.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.717+0000 7f39aaffd640 1 -- 172.21.15.169:0/64644448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39ac102160 msgr2=0x7f39ac19ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:39.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.717+0000 7f39aaffd640 1 --2- 172.21.15.169:0/64644448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39ac102160 0x7f39ac19ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.717+0000 7f39aaffd640 1 -- 172.21.15.169:0/64644448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39ac1017a0 msgr2=0x7f39ac19c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:39.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.717+0000 7f39aaffd640 1 --2- 172.21.15.169:0/64644448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39ac1017a0 0x7f39ac19c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.717+0000 7f39aaffd640 1 -- 172.21.15.169:0/64644448 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ac19f5d0 con 0x7f39ac1005b0 2024-03-28T01:09:39.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.717+0000 7f39a3fff640 1 --2- 172.21.15.169:0/64644448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39ac1017a0 0x7f39ac19c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T01:09:39.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.718+0000 7f39aaffd640 1 --2- 172.21.15.169:0/64644448 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39ac1005b0 0x7f39ac06a860 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f399c00e9c0 tx=0x7f399c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:39.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.718+0000 7f39a8ff9640 1 -- 172.21.15.169:0/64644448 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c00cd80 con 0x7f39ac1005b0 2024-03-28T01:09:39.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.718+0000 7f39b15b9640 1 -- 172.21.15.169:0/64644448 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39ac19f8c0 con 0x7f39ac1005b0 2024-03-28T01:09:39.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.718+0000 7f39a8ff9640 1 -- 172.21.15.169:0/64644448 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f399c0035f0 con 0x7f39ac1005b0 2024-03-28T01:09:39.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.718+0000 7f39a8ff9640 1 -- 172.21.15.169:0/64644448 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c010690 con 0x7f39ac1005b0 2024-03-28T01:09:39.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.718+0000 7f39b15b9640 1 -- 172.21.15.169:0/64644448 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39ac19fde0 con 0x7f39ac1005b0 2024-03-28T01:09:39.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.719+0000 7f39b15b9640 1 -- 172.21.15.169:0/64644448 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39ac1079a0 con 0x7f39ac1005b0 2024-03-28T01:09:39.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.720+0000 7f39a8ff9640 1 -- 172.21.15.169:0/64644448 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f399c0037a0 con 0x7f39ac1005b0 2024-03-28T01:09:39.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.720+0000 7f39a8ff9640 1 --2- 172.21.15.169:0/64644448 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f39980a2800 0x7f39980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:39.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.721+0000 7f39a3fff640 1 --2- 172.21.15.169:0/64644448 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f39980a2800 0x7f39980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:39.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.721+0000 7f39a8ff9640 1 -- 172.21.15.169:0/64644448 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9158+0+0 (secure 0 0 0) 0x7f399c0cf1b0 con 0x7f39ac1005b0 2024-03-28T01:09:39.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.721+0000 7f39a3fff640 1 --2- 172.21.15.169:0/64644448 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f39980a2800 0x7f39980a4cc0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f399402c7b0 tx=0x7f3994005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:39.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.725+0000 7f39a8ff9640 1 -- 172.21.15.169:0/64644448 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f399c09beb0 con 0x7f39ac1005b0 2024-03-28T01:09:39.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.938+0000 7f39b15b9640 1 -- 172.21.15.169:0/64644448 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f39ac02d070 con 0x7f39ac1005b0 2024-03-28T01:09:39.942 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:39 smithi169 ceph-mon[32274]: pgmap v745: 129 pgs: 129 active+clean; 2.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 59 MiB/s wr, 595 op/s 2024-03-28T01:09:39.942 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:39 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2187864309' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:09:39.943 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:39 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/2121708360' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T01:09:39.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.942+0000 7f39a8ff9640 1 -- 172.21.15.169:0/64644448 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) v1 ==== 161+0+0 (secure 0 0 0) 0x7f399c09b7b0 con 0x7f39ac1005b0 2024-03-28T01:09:39.945 INFO:teuthology.orchestra.run.smithi169.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T01:09:39.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.944+0000 7f39b15b9640 1 -- 172.21.15.169:0/64644448 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f39980a2800 msgr2=0x7f39980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:39.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.944+0000 7f39b15b9640 1 --2- 172.21.15.169:0/64644448 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f39980a2800 0x7f39980a4cc0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f399402c7b0 tx=0x7f3994005fd0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.944+0000 7f39b15b9640 1 -- 172.21.15.169:0/64644448 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39ac1005b0 msgr2=0x7f39ac06a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:39.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.944+0000 7f39b15b9640 1 --2- 172.21.15.169:0/64644448 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39ac1005b0 0x7f39ac06a860 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f399c00e9c0 tx=0x7f399c00ee80 comp rx=0 tx=0).stop 2024-03-28T01:09:39.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.944+0000 7f39b15b9640 1 -- 172.21.15.169:0/64644448 shutdown_connections 2024-03-28T01:09:39.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.944+0000 7f39b15b9640 1 --2- 172.21.15.169:0/64644448 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f39980a2800 0x7f39980a4cc0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.944+0000 7f39b15b9640 1 --2- 172.21.15.169:0/64644448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39ac102160 0x7f39ac19ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.944+0000 7f39b15b9640 1 --2- 172.21.15.169:0/64644448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39ac1017a0 0x7f39ac19c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.944+0000 7f39b15b9640 1 --2- 172.21.15.169:0/64644448 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39ac1005b0 0x7f39ac06a860 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:39.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.944+0000 7f39b15b9640 1 -- 172.21.15.169:0/64644448 >> 172.21.15.169:0/64644448 conn(0x7f39ac0fbb40 msgr2=0x7f39ac104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:39.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.944+0000 7f39b15b9640 1 -- 172.21.15.169:0/64644448 shutdown_connections 2024-03-28T01:09:39.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:39.944+0000 7f39b15b9640 1 -- 172.21.15.169:0/64644448 wait complete. 2024-03-28T01:09:39.960 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-28T01:09:40.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.063+0000 7ff3b5fc6640 1 -- 172.21.15.169:0/3166757463 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3b0102150 msgr2=0x7ff3b01089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:40.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.063+0000 7ff3b5fc6640 1 --2- 172.21.15.169:0/3166757463 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3b0102150 0x7ff3b01089e0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8009ca0 tx=0x7ff3a802c0f0 comp rx=0 tx=0).stop 2024-03-28T01:09:40.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.064+0000 7ff3b5fc6640 1 -- 172.21.15.169:0/3166757463 shutdown_connections 2024-03-28T01:09:40.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.064+0000 7ff3b5fc6640 1 --2- 172.21.15.169:0/3166757463 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3b0102150 0x7ff3b01089e0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:40.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.064+0000 7ff3b5fc6640 1 --2- 172.21.15.169:0/3166757463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b0101790 0x7ff3b0101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:40.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.064+0000 7ff3b5fc6640 1 --2- 172.21.15.169:0/3166757463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3b01005a0 0x7ff3b01009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:40.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.064+0000 7ff3b5fc6640 1 -- 172.21.15.169:0/3166757463 >> 172.21.15.169:0/3166757463 conn(0x7ff3b00fbb30 msgr2=0x7ff3b00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:40.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.064+0000 7ff3b5fc6640 1 -- 172.21.15.169:0/3166757463 shutdown_connections 2024-03-28T01:09:40.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.064+0000 7ff3b5fc6640 1 -- 172.21.15.169:0/3166757463 wait complete. 2024-03-28T01:09:40.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.064+0000 7ff3b5fc6640 1 Processor -- start 2024-03-28T01:09:40.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.064+0000 7ff3b5fc6640 1 -- start start 2024-03-28T01:09:40.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.064+0000 7ff3b5fc6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b01005a0 0x7ff3b0199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:40.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.064+0000 7ff3b5fc6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3b0101790 0x7ff3b019a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:40.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.064+0000 7ff3b5fc6640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3b0102150 0x7ff3b01a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:40.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.064+0000 7ff3b5fc6640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b010c230 con 0x7ff3b0102150 2024-03-28T01:09:40.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.064+0000 7ff3b5fc6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b010c0b0 con 0x7ff3b01005a0 2024-03-28T01:09:40.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.064+0000 7ff3b5fc6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b010c3b0 con 0x7ff3b0101790 2024-03-28T01:09:40.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.065+0000 7ff3affff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3b0102150 0x7ff3b01a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:40.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.065+0000 7ff3affff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3b0102150 0x7ff3b01a1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49520/0 (socket says 172.21.15.169:49520) 2024-03-28T01:09:40.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.065+0000 7ff3affff640 1 -- 172.21.15.169:0/1261835183 learned_addr learned my addr 172.21.15.169:0/1261835183 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:09:40.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.065+0000 7ff3aeffd640 1 --2- 172.21.15.169:0/1261835183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3b0101790 0x7ff3b019a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:40.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.065+0000 7ff3af7fe640 1 --2- 172.21.15.169:0/1261835183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b01005a0 0x7ff3b0199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:40.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.065+0000 7ff3affff640 1 -- 172.21.15.169:0/1261835183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3b0101790 msgr2=0x7ff3b019a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:40.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.065+0000 7ff3affff640 1 --2- 172.21.15.169:0/1261835183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3b0101790 0x7ff3b019a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:40.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.065+0000 7ff3affff640 1 -- 172.21.15.169:0/1261835183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b01005a0 msgr2=0x7ff3b0199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:40.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.065+0000 7ff3affff640 1 --2- 172.21.15.169:0/1261835183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b01005a0 0x7ff3b0199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:40.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.065+0000 7ff3affff640 1 -- 172.21.15.169:0/1261835183 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b01a17a0 con 0x7ff3b0102150 2024-03-28T01:09:40.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.065+0000 7ff3affff640 1 --2- 172.21.15.169:0/1261835183 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3b0102150 0x7ff3b01a1130 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8037040 tx=0x7ff3a8032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:40.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.065+0000 7ff3acff9640 1 -- 172.21.15.169:0/1261835183 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8041020 con 0x7ff3b0102150 2024-03-28T01:09:40.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.065+0000 7ff3acff9640 1 -- 172.21.15.169:0/1261835183 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3a8034cd0 con 0x7ff3b0102150 2024-03-28T01:09:40.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.065+0000 7ff3acff9640 1 -- 172.21.15.169:0/1261835183 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8039040 con 0x7ff3b0102150 2024-03-28T01:09:40.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.065+0000 7ff3b5fc6640 1 -- 172.21.15.169:0/1261835183 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b01a1a30 con 0x7ff3b0102150 2024-03-28T01:09:40.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.065+0000 7ff3b5fc6640 1 -- 172.21.15.169:0/1261835183 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b01a1f70 con 0x7ff3b0102150 2024-03-28T01:09:40.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.066+0000 7ff3b5fc6640 1 -- 172.21.15.169:0/1261835183 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b01a21a0 con 0x7ff3b0102150 2024-03-28T01:09:40.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.067+0000 7ff3acff9640 1 -- 172.21.15.169:0/1261835183 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff3a8044020 con 0x7ff3b0102150 2024-03-28T01:09:40.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.068+0000 7ff3acff9640 1 --2- 172.21.15.169:0/1261835183 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff3980a2880 0x7ff3980a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:40.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.068+0000 7ff3af7fe640 1 --2- 172.21.15.169:0/1261835183 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff3980a2880 0x7ff3980a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:40.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.068+0000 7ff3acff9640 1 -- 172.21.15.169:0/1261835183 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9416+0+0 (secure 0 0 0) 0x7ff3a80f10c0 con 0x7ff3b0102150 2024-03-28T01:09:40.072 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.068+0000 7ff3af7fe640 1 --2- 172.21.15.169:0/1261835183 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff3980a2880 0x7ff3980a4d40 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff3a00095f0 tx=0x7ff3a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:40.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.072+0000 7ff3acff9640 1 -- 172.21.15.169:0/1261835183 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff3a804b020 con 0x7ff3b0102150 2024-03-28T01:09:40.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:39 smithi177 ceph-mon[36206]: pgmap v745: 129 pgs: 129 active+clean; 2.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 59 MiB/s wr, 595 op/s 2024-03-28T01:09:40.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:39 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2187864309' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:09:40.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:39 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/2121708360' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T01:09:40.167 INFO:journalctl@ceph.mds.i.smithi177.stdout:Mar 28 01:09:40 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-i[67676]: ignoring --setuser ceph since I am not root 2024-03-28T01:09:40.167 INFO:journalctl@ceph.mds.i.smithi177.stdout:Mar 28 01:09:40 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-i[67676]: ignoring --setgroup ceph since I am not root 2024-03-28T01:09:40.167 INFO:journalctl@ceph.mds.i.smithi177.stdout:Mar 28 01:09:40 smithi177 ceph-mds[67680]: -- 172.21.15.177:0/2891739147 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561c98f95a40 con 0x561c98f68800 2024-03-28T01:09:40.167 INFO:journalctl@ceph.mds.i.smithi177.stdout:Mar 28 01:09:40 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-i[67676]: starting mds.i at 2024-03-28T01:09:40.168 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 01:09:40 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-c[67357]: ignoring --setuser ceph since I am not root 2024-03-28T01:09:40.168 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 01:09:40 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-c[67357]: ignoring --setgroup ceph since I am not root 2024-03-28T01:09:40.168 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 01:09:40 smithi177 ceph-mds[67361]: -- 172.21.15.177:0/2369948479 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562410bc7a40 con 0x562410b9b000 2024-03-28T01:09:40.168 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 01:09:40 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-c[67357]: starting mds.c at 2024-03-28T01:09:40.168 INFO:journalctl@ceph.mds.f.smithi177.stdout:Mar 28 01:09:40 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-f[67040]: ignoring --setuser ceph since I am not root 2024-03-28T01:09:40.168 INFO:journalctl@ceph.mds.f.smithi177.stdout:Mar 28 01:09:40 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-f[67040]: ignoring --setgroup ceph since I am not root 2024-03-28T01:09:40.168 INFO:journalctl@ceph.mds.f.smithi177.stdout:Mar 28 01:09:40 smithi177 ceph-mds[67044]: -- 172.21.15.177:0/1594021466 <== mon.2 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55be3480ba40 con 0x55be347de800 2024-03-28T01:09:40.168 INFO:journalctl@ceph.mds.f.smithi177.stdout:Mar 28 01:09:40 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-f[67040]: starting mds.f at 2024-03-28T01:09:40.288 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.285+0000 7ff3b5fc6640 1 -- 172.21.15.169:0/1261835183 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff3b01a2370 con 0x7ff3b0102150 2024-03-28T01:09:40.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:39 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-a[32270]: 2024-03-28T01:09:39.938+0000 7f2d5bed2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T01:09:40.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:39 smithi174 ceph-mon[36060]: pgmap v745: 129 pgs: 129 active+clean; 2.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 59 MiB/s wr, 595 op/s 2024-03-28T01:09:40.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:39 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2187864309' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T01:09:40.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:39 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/2121708360' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T01:09:40.358 INFO:journalctl@ceph.mds.h.smithi174.stdout:Mar 28 01:09:40 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-h[65800]: ignoring --setuser ceph since I am not root 2024-03-28T01:09:40.358 INFO:journalctl@ceph.mds.h.smithi174.stdout:Mar 28 01:09:40 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-h[65800]: ignoring --setgroup ceph since I am not root 2024-03-28T01:09:40.358 INFO:journalctl@ceph.mds.h.smithi174.stdout:Mar 28 01:09:40 smithi174 ceph-mds[65804]: -- 172.21.15.174:0/1599532323 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5583703fda40 con 0x5583703d1000 2024-03-28T01:09:40.358 INFO:journalctl@ceph.mds.h.smithi174.stdout:Mar 28 01:09:40 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-h[65800]: starting mds.h at 2024-03-28T01:09:40.359 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 01:09:40 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-b[66103]: ignoring --setuser ceph since I am not root 2024-03-28T01:09:40.359 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 01:09:40 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-b[66103]: ignoring --setgroup ceph since I am not root 2024-03-28T01:09:40.359 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 01:09:40 smithi174 ceph-mds[66107]: -- 172.21.15.174:0/192364245 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b95ec3da40 con 0x55b95ec11000 2024-03-28T01:09:40.359 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 01:09:40 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-b[66103]: starting mds.b at 2024-03-28T01:09:40.359 INFO:journalctl@ceph.mds.e.smithi174.stdout:Mar 28 01:09:40 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-e[65479]: ignoring --setuser ceph since I am not root 2024-03-28T01:09:40.359 INFO:journalctl@ceph.mds.e.smithi174.stdout:Mar 28 01:09:40 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-e[65479]: ignoring --setgroup ceph since I am not root 2024-03-28T01:09:40.359 INFO:journalctl@ceph.mds.e.smithi174.stdout:Mar 28 01:09:40 smithi174 ceph-mds[65483]: -- 172.21.15.174:0/162824197 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560b06d05a40 con 0x560b06cd8800 2024-03-28T01:09:40.359 INFO:journalctl@ceph.mds.e.smithi174.stdout:Mar 28 01:09:40 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-e[65479]: starting mds.e at 2024-03-28T01:09:40.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.949+0000 7ff3acff9640 1 -- 172.21.15.169:0/1261835183 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff3a80bdd50 con 0x7ff3b0102150 2024-03-28T01:09:40.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.952+0000 7ff3b5fc6640 1 -- 172.21.15.169:0/1261835183 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff3980a2880 msgr2=0x7ff3980a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:40.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.952+0000 7ff3b5fc6640 1 --2- 172.21.15.169:0/1261835183 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff3980a2880 0x7ff3980a4d40 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff3a00095f0 tx=0x7ff3a0005f90 comp rx=0 tx=0).stop 2024-03-28T01:09:40.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.952+0000 7ff3b5fc6640 1 -- 172.21.15.169:0/1261835183 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3b0102150 msgr2=0x7ff3b01a1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:40.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.952+0000 7ff3b5fc6640 1 --2- 172.21.15.169:0/1261835183 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3b0102150 0x7ff3b01a1130 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8037040 tx=0x7ff3a8032f40 comp rx=0 tx=0).stop 2024-03-28T01:09:40.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.952+0000 7ff3b5fc6640 1 -- 172.21.15.169:0/1261835183 shutdown_connections 2024-03-28T01:09:40.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.952+0000 7ff3b5fc6640 1 --2- 172.21.15.169:0/1261835183 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7ff3980a2880 0x7ff3980a4d40 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:40.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.952+0000 7ff3b5fc6640 1 --2- 172.21.15.169:0/1261835183 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3b0102150 0x7ff3b01a1130 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:40.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.952+0000 7ff3b5fc6640 1 --2- 172.21.15.169:0/1261835183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3b0101790 0x7ff3b019a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:40.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.952+0000 7ff3b5fc6640 1 --2- 172.21.15.169:0/1261835183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3b01005a0 0x7ff3b0199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:40.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.952+0000 7ff3b5fc6640 1 -- 172.21.15.169:0/1261835183 >> 172.21.15.169:0/1261835183 conn(0x7ff3b00fbb30 msgr2=0x7ff3b00fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:40.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.952+0000 7ff3b5fc6640 1 -- 172.21.15.169:0/1261835183 shutdown_connections 2024-03-28T01:09:40.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:40.952+0000 7ff3b5fc6640 1 -- 172.21.15.169:0/1261835183 wait complete. 2024-03-28T01:09:40.969 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-28T01:09:41.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.075+0000 7f0502d10640 1 -- 172.21.15.169:0/258386403 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04fc102180 msgr2=0x7f04fc104570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:41.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.075+0000 7f0502d10640 1 --2- 172.21.15.169:0/258386403 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04fc102180 0x7f04fc104570 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f04f0009900 tx=0x7f04f002bc30 comp rx=0 tx=0).stop 2024-03-28T01:09:41.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.075+0000 7f0502d10640 1 -- 172.21.15.169:0/258386403 shutdown_connections 2024-03-28T01:09:41.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.075+0000 7f0502d10640 1 --2- 172.21.15.169:0/258386403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04fc1073e0 0x7f04fc1097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:41.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.075+0000 7f0502d10640 1 --2- 172.21.15.169:0/258386403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04fc104ab0 0x7f04fc106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:41.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.075+0000 7f0502d10640 1 --2- 172.21.15.169:0/258386403 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04fc102180 0x7f04fc104570 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:41.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.075+0000 7f0502d10640 1 -- 172.21.15.169:0/258386403 >> 172.21.15.169:0/258386403 conn(0x7f04fc0fbb10 msgr2=0x7f04fc0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:41.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.075+0000 7f0502d10640 1 -- 172.21.15.169:0/258386403 shutdown_connections 2024-03-28T01:09:41.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.075+0000 7f0502d10640 1 -- 172.21.15.169:0/258386403 wait complete. 2024-03-28T01:09:41.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.075+0000 7f0502d10640 1 Processor -- start 2024-03-28T01:09:41.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f0502d10640 1 -- start start 2024-03-28T01:09:41.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f0502d10640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04fc102180 0x7f04fc197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:41.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f0502d10640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04fc104ab0 0x7f04fc197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:41.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f0502d10640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04fc1073e0 0x7f04fc19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:41.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f0502d10640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc10d340 con 0x7f04fc102180 2024-03-28T01:09:41.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f0502d10640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc10d1c0 con 0x7f04fc1073e0 2024-03-28T01:09:41.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f0502d10640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc10d4c0 con 0x7f04fc104ab0 2024-03-28T01:09:41.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f0500aac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04fc102180 0x7f04fc197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:41.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f0500aac640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04fc102180 0x7f04fc197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49548/0 (socket says 172.21.15.169:49548) 2024-03-28T01:09:41.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f0500aac640 1 -- 172.21.15.169:0/3819127998 learned_addr learned my addr 172.21.15.169:0/3819127998 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:09:41.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f04fbfff640 1 --2- 172.21.15.169:0/3819127998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04fc104ab0 0x7f04fc197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:41.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f05012ad640 1 --2- 172.21.15.169:0/3819127998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04fc1073e0 0x7f04fc19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:41.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f0500aac640 1 -- 172.21.15.169:0/3819127998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04fc104ab0 msgr2=0x7f04fc197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:41.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f0500aac640 1 --2- 172.21.15.169:0/3819127998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04fc104ab0 0x7f04fc197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:41.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f0500aac640 1 -- 172.21.15.169:0/3819127998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04fc1073e0 msgr2=0x7f04fc19efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:41.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f0500aac640 1 --2- 172.21.15.169:0/3819127998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04fc1073e0 0x7f04fc19efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:41.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f0500aac640 1 -- 172.21.15.169:0/3819127998 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc19f630 con 0x7f04fc102180 2024-03-28T01:09:41.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f05012ad640 1 --2- 172.21.15.169:0/3819127998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04fc1073e0 0x7f04fc19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T01:09:41.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f0500aac640 1 --2- 172.21.15.169:0/3819127998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04fc102180 0x7f04fc197a10 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f04f0009230 tx=0x7f04f002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:41.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f04f9ffb640 1 -- 172.21.15.169:0/3819127998 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f0041070 con 0x7f04fc102180 2024-03-28T01:09:41.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f04f9ffb640 1 -- 172.21.15.169:0/3819127998 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04f0004450 con 0x7f04fc102180 2024-03-28T01:09:41.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f04f9ffb640 1 -- 172.21.15.169:0/3819127998 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f0003e20 con 0x7f04fc102180 2024-03-28T01:09:41.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.076+0000 7f0502d10640 1 -- 172.21.15.169:0/3819127998 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04fc19f8c0 con 0x7f04fc102180 2024-03-28T01:09:41.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.077+0000 7f0502d10640 1 -- 172.21.15.169:0/3819127998 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04fc19fe00 con 0x7f04fc102180 2024-03-28T01:09:41.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.078+0000 7f0502d10640 1 -- 172.21.15.169:0/3819127998 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04fc0fd600 con 0x7f04fc102180 2024-03-28T01:09:41.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.079+0000 7f04f9ffb640 1 -- 172.21.15.169:0/3819127998 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f04f0044070 con 0x7f04fc102180 2024-03-28T01:09:41.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.079+0000 7f04f9ffb640 1 --2- 172.21.15.169:0/3819127998 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04ec0a2880 0x7f04ec0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:41.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.079+0000 7f04fbfff640 1 --2- 172.21.15.169:0/3819127998 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04ec0a2880 0x7f04ec0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:41.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.079+0000 7f04f9ffb640 1 -- 172.21.15.169:0/3819127998 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9416+0+0 (secure 0 0 0) 0x7f04f00f0ec0 con 0x7f04fc102180 2024-03-28T01:09:41.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.080+0000 7f04fbfff640 1 --2- 172.21.15.169:0/3819127998 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04ec0a2880 0x7f04ec0a4d40 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f04fc198f20 tx=0x7f04e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:41.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.083+0000 7f04f9ffb640 1 -- 172.21.15.169:0/3819127998 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f04f00bdac0 con 0x7f04fc102180 2024-03-28T01:09:41.166 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:40 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/64644448' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T01:09:41.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:40 smithi177 ceph-mon[36206]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T01:09:41.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:40 smithi177 ceph-mon[36206]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T01:09:41.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:40 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/64644448' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T01:09:41.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:40 smithi177 ceph-mon[36206]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T01:09:41.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:40 smithi177 ceph-mon[36206]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-28T01:09:41.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:40 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1261835183' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T01:09:41.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:40 smithi177 ceph-mon[36206]: pgmap v747: 129 pgs: 129 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 59 MiB/s wr, 620 op/s 2024-03-28T01:09:41.248 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:40 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/64644448' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T01:09:41.248 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:40 smithi169 ceph-mon[32274]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T01:09:41.248 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:40 smithi169 ceph-mon[32274]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T01:09:41.248 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:40 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/64644448' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T01:09:41.248 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:40 smithi169 ceph-mon[32274]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T01:09:41.248 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:40 smithi169 ceph-mon[32274]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-28T01:09:41.249 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:40 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1261835183' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T01:09:41.249 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:40 smithi169 ceph-mon[32274]: pgmap v747: 129 pgs: 129 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 59 MiB/s wr, 620 op/s 2024-03-28T01:09:41.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.268+0000 7f0502d10640 1 -- 172.21.15.169:0/3819127998 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f04fc02d070 con 0x7f04fc102180 2024-03-28T01:09:41.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:40 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/64644448' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T01:09:41.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:40 smithi174 ceph-mon[36060]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T01:09:41.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:40 smithi174 ceph-mon[36060]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T01:09:41.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:40 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/64644448' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T01:09:41.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:40 smithi174 ceph-mon[36060]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T01:09:41.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:40 smithi174 ceph-mon[36060]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-28T01:09:41.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:40 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1261835183' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T01:09:41.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:40 smithi174 ceph-mon[36060]: pgmap v747: 129 pgs: 129 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 59 MiB/s wr, 620 op/s 2024-03-28T01:09:41.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.951+0000 7f04f9ffb640 1 -- 172.21.15.169:0/3819127998 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v96) v1 ==== 181+0+0 (secure 0 0 0) 0x7f04f00bd3c0 con 0x7f04fc102180 2024-03-28T01:09:41.954 INFO:teuthology.orchestra.run.smithi169.stderr:pool 'cephfs_metadata' removed 2024-03-28T01:09:41.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.953+0000 7f0502d10640 1 -- 172.21.15.169:0/3819127998 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04ec0a2880 msgr2=0x7f04ec0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:41.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.953+0000 7f0502d10640 1 --2- 172.21.15.169:0/3819127998 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04ec0a2880 0x7f04ec0a4d40 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f04fc198f20 tx=0x7f04e8005f90 comp rx=0 tx=0).stop 2024-03-28T01:09:41.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.953+0000 7f0502d10640 1 -- 172.21.15.169:0/3819127998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04fc102180 msgr2=0x7f04fc197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:41.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.953+0000 7f0502d10640 1 --2- 172.21.15.169:0/3819127998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04fc102180 0x7f04fc197a10 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f04f0009230 tx=0x7f04f002c650 comp rx=0 tx=0).stop 2024-03-28T01:09:41.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.953+0000 7f0502d10640 1 -- 172.21.15.169:0/3819127998 shutdown_connections 2024-03-28T01:09:41.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.953+0000 7f0502d10640 1 --2- 172.21.15.169:0/3819127998 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f04ec0a2880 0x7f04ec0a4d40 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:41.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.953+0000 7f0502d10640 1 --2- 172.21.15.169:0/3819127998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04fc1073e0 0x7f04fc19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:41.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.953+0000 7f0502d10640 1 --2- 172.21.15.169:0/3819127998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04fc104ab0 0x7f04fc197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:41.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.953+0000 7f0502d10640 1 --2- 172.21.15.169:0/3819127998 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04fc102180 0x7f04fc197a10 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:41.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.953+0000 7f0502d10640 1 -- 172.21.15.169:0/3819127998 >> 172.21.15.169:0/3819127998 conn(0x7f04fc0fbb10 msgr2=0x7f04fc102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:41.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.953+0000 7f0502d10640 1 -- 172.21.15.169:0/3819127998 shutdown_connections 2024-03-28T01:09:41.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:41.953+0000 7f0502d10640 1 -- 172.21.15.169:0/3819127998 wait complete. 2024-03-28T01:09:41.970 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-28T01:09:42.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.075+0000 7f2d4900b640 1 -- 172.21.15.169:0/1446153539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d44101790 msgr2=0x7f2d44101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:42.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.075+0000 7f2d4900b640 1 --2- 172.21.15.169:0/1446153539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d44101790 0x7f2d44101c10 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f2d34009900 tx=0x7f2d3402bc30 comp rx=0 tx=0).stop 2024-03-28T01:09:42.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.075+0000 7f2d4900b640 1 -- 172.21.15.169:0/1446153539 shutdown_connections 2024-03-28T01:09:42.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.075+0000 7f2d4900b640 1 --2- 172.21.15.169:0/1446153539 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d44102150 0x7f2d441089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:42.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.075+0000 7f2d4900b640 1 --2- 172.21.15.169:0/1446153539 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d44101790 0x7f2d44101c10 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:42.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.075+0000 7f2d4900b640 1 --2- 172.21.15.169:0/1446153539 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d441005a0 0x7f2d441009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:42.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.075+0000 7f2d4900b640 1 -- 172.21.15.169:0/1446153539 >> 172.21.15.169:0/1446153539 conn(0x7f2d440fbb30 msgr2=0x7f2d440fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:42.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.076+0000 7f2d4900b640 1 -- 172.21.15.169:0/1446153539 shutdown_connections 2024-03-28T01:09:42.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.076+0000 7f2d4900b640 1 -- 172.21.15.169:0/1446153539 wait complete. 2024-03-28T01:09:42.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.076+0000 7f2d4900b640 1 Processor -- start 2024-03-28T01:09:42.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.076+0000 7f2d4900b640 1 -- start start 2024-03-28T01:09:42.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d4900b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d441005a0 0x7f2d44069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:42.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d4900b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d44101790 0x7f2d4419c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:42.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d4900b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d44102150 0x7f2d4419ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:42.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d4900b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4410c230 con 0x7f2d441005a0 2024-03-28T01:09:42.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d4900b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4410c0b0 con 0x7f2d44102150 2024-03-28T01:09:42.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d4900b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4410c3b0 con 0x7f2d44101790 2024-03-28T01:09:42.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d42d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d441005a0 0x7f2d44069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:42.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d42d9d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d441005a0 0x7f2d44069390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49568/0 (socket says 172.21.15.169:49568) 2024-03-28T01:09:42.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d42d9d640 1 -- 172.21.15.169:0/956709805 learned_addr learned my addr 172.21.15.169:0/956709805 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:09:42.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d4259c640 1 --2- 172.21.15.169:0/956709805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d44101790 0x7f2d4419c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:42.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d4359e640 1 --2- 172.21.15.169:0/956709805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d44102150 0x7f2d4419ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:42.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d42d9d640 1 -- 172.21.15.169:0/956709805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d44101790 msgr2=0x7f2d4419c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:42.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d42d9d640 1 --2- 172.21.15.169:0/956709805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d44101790 0x7f2d4419c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:42.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d42d9d640 1 -- 172.21.15.169:0/956709805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d44102150 msgr2=0x7f2d4419ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:42.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d42d9d640 1 --2- 172.21.15.169:0/956709805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d44102150 0x7f2d4419ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:42.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d42d9d640 1 -- 172.21.15.169:0/956709805 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d4419f600 con 0x7f2d441005a0 2024-03-28T01:09:42.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d42d9d640 1 --2- 172.21.15.169:0/956709805 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d441005a0 0x7f2d44069390 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f2d3800d770 tx=0x7f2d3800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:42.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d2bfff640 1 -- 172.21.15.169:0/956709805 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d38017070 con 0x7f2d441005a0 2024-03-28T01:09:42.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d2bfff640 1 -- 172.21.15.169:0/956709805 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d38009c30 con 0x7f2d441005a0 2024-03-28T01:09:42.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d4900b640 1 -- 172.21.15.169:0/956709805 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d4419f8f0 con 0x7f2d441005a0 2024-03-28T01:09:42.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d2bfff640 1 -- 172.21.15.169:0/956709805 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d38003dd0 con 0x7f2d441005a0 2024-03-28T01:09:42.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.077+0000 7f2d4900b640 1 -- 172.21.15.169:0/956709805 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d4419fe10 con 0x7f2d441005a0 2024-03-28T01:09:42.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.078+0000 7f2d4900b640 1 -- 172.21.15.169:0/956709805 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d44068490 con 0x7f2d441005a0 2024-03-28T01:09:42.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.080+0000 7f2d2bfff640 1 -- 172.21.15.169:0/956709805 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2d38003fb0 con 0x7f2d441005a0 2024-03-28T01:09:42.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.080+0000 7f2d2bfff640 1 --2- 172.21.15.169:0/956709805 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2d2c0a2800 0x7f2d2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:42.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.080+0000 7f2d2bfff640 1 -- 172.21.15.169:0/956709805 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 8920+0+0 (secure 0 0 0) 0x7f2d380ceb60 con 0x7f2d441005a0 2024-03-28T01:09:42.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.080+0000 7f2d4259c640 1 --2- 172.21.15.169:0/956709805 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2d2c0a2800 0x7f2d2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:42.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.081+0000 7f2d4259c640 1 --2- 172.21.15.169:0/956709805 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2d2c0a2800 0x7f2d2c0a4cc0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2d34006010 tx=0x7f2d34000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:42.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.090+0000 7f2d2bfff640 1 -- 172.21.15.169:0/956709805 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2d38009de0 con 0x7f2d441005a0 2024-03-28T01:09:42.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:41 smithi177 ceph-mon[36206]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T01:09:42.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:41 smithi177 ceph-mon[36206]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T01:09:42.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:41 smithi177 ceph-mon[36206]: Cluster is now healthy 2024-03-28T01:09:42.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:41 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1261835183' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T01:09:42.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:41 smithi177 ceph-mon[36206]: mds.? [v2:172.21.15.174:6834/3840554136,v1:172.21.15.174:6837/3840554136] up:boot 2024-03-28T01:09:42.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:41 smithi177 ceph-mon[36206]: mds.? [v2:172.21.15.174:6835/4047300119,v1:172.21.15.174:6838/4047300119] up:boot 2024-03-28T01:09:42.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:41 smithi177 ceph-mon[36206]: mds.? [v2:172.21.15.177:6837/1866372185,v1:172.21.15.177:6839/1866372185] up:boot 2024-03-28T01:09:42.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:41 smithi177 ceph-mon[36206]: mds.? [v2:172.21.15.177:6834/3755445102,v1:172.21.15.177:6835/3755445102] up:boot 2024-03-28T01:09:42.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:41 smithi177 ceph-mon[36206]: mds.? [v2:172.21.15.174:6836/3636915453,v1:172.21.15.174:6839/3636915453] up:boot 2024-03-28T01:09:42.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:41 smithi177 ceph-mon[36206]: mds.? [v2:172.21.15.177:6836/1778294653,v1:172.21.15.177:6838/1778294653] up:boot 2024-03-28T01:09:42.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:41 smithi177 ceph-mon[36206]: fsmap 12 up:standby 2024-03-28T01:09:42.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:41 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T01:09:42.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:41 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T01:09:42.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:41 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T01:09:42.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:41 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T01:09:42.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:41 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T01:09:42.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:41 smithi177 ceph-mon[36206]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T01:09:42.168 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:41 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3819127998' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T01:09:42.218 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:41 smithi169 ceph-mon[32274]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T01:09:42.218 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:41 smithi169 ceph-mon[32274]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T01:09:42.219 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:41 smithi169 ceph-mon[32274]: Cluster is now healthy 2024-03-28T01:09:42.219 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:41 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1261835183' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T01:09:42.219 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:41 smithi169 ceph-mon[32274]: mds.? [v2:172.21.15.174:6834/3840554136,v1:172.21.15.174:6837/3840554136] up:boot 2024-03-28T01:09:42.219 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:41 smithi169 ceph-mon[32274]: mds.? [v2:172.21.15.174:6835/4047300119,v1:172.21.15.174:6838/4047300119] up:boot 2024-03-28T01:09:42.219 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:41 smithi169 ceph-mon[32274]: mds.? [v2:172.21.15.177:6837/1866372185,v1:172.21.15.177:6839/1866372185] up:boot 2024-03-28T01:09:42.219 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:41 smithi169 ceph-mon[32274]: mds.? [v2:172.21.15.177:6834/3755445102,v1:172.21.15.177:6835/3755445102] up:boot 2024-03-28T01:09:42.219 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:41 smithi169 ceph-mon[32274]: mds.? [v2:172.21.15.174:6836/3636915453,v1:172.21.15.174:6839/3636915453] up:boot 2024-03-28T01:09:42.219 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:41 smithi169 ceph-mon[32274]: mds.? [v2:172.21.15.177:6836/1778294653,v1:172.21.15.177:6838/1778294653] up:boot 2024-03-28T01:09:42.219 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:41 smithi169 ceph-mon[32274]: fsmap 12 up:standby 2024-03-28T01:09:42.219 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:41 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T01:09:42.219 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:41 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T01:09:42.219 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:41 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T01:09:42.219 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:41 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T01:09:42.219 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:41 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T01:09:42.220 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:41 smithi169 ceph-mon[32274]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T01:09:42.220 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:41 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3819127998' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T01:09:42.277 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.274+0000 7f2d4900b640 1 -- 172.21.15.169:0/956709805 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2d44107930 con 0x7f2d441005a0 2024-03-28T01:09:42.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:41 smithi174 ceph-mon[36060]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T01:09:42.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:41 smithi174 ceph-mon[36060]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T01:09:42.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:41 smithi174 ceph-mon[36060]: Cluster is now healthy 2024-03-28T01:09:42.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:41 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1261835183' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T01:09:42.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:41 smithi174 ceph-mon[36060]: mds.? [v2:172.21.15.174:6834/3840554136,v1:172.21.15.174:6837/3840554136] up:boot 2024-03-28T01:09:42.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:41 smithi174 ceph-mon[36060]: mds.? [v2:172.21.15.174:6835/4047300119,v1:172.21.15.174:6838/4047300119] up:boot 2024-03-28T01:09:42.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:41 smithi174 ceph-mon[36060]: mds.? [v2:172.21.15.177:6837/1866372185,v1:172.21.15.177:6839/1866372185] up:boot 2024-03-28T01:09:42.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:41 smithi174 ceph-mon[36060]: mds.? [v2:172.21.15.177:6834/3755445102,v1:172.21.15.177:6835/3755445102] up:boot 2024-03-28T01:09:42.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:41 smithi174 ceph-mon[36060]: mds.? [v2:172.21.15.174:6836/3636915453,v1:172.21.15.174:6839/3636915453] up:boot 2024-03-28T01:09:42.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:41 smithi174 ceph-mon[36060]: mds.? [v2:172.21.15.177:6836/1778294653,v1:172.21.15.177:6838/1778294653] up:boot 2024-03-28T01:09:42.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:41 smithi174 ceph-mon[36060]: fsmap 12 up:standby 2024-03-28T01:09:42.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:41 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T01:09:42.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:41 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T01:09:42.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:41 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T01:09:42.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:41 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T01:09:42.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:41 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T01:09:42.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:41 smithi174 ceph-mon[36060]: from='mgr.14152 172.21.15.169:0/2479329701' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T01:09:42.359 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:41 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3819127998' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T01:09:42.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.956+0000 7f2d2bfff640 1 -- 172.21.15.169:0/956709805 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v97) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2d44107930 con 0x7f2d441005a0 2024-03-28T01:09:42.960 INFO:teuthology.orchestra.run.smithi169.stderr:pool 'cephfs_data' removed 2024-03-28T01:09:42.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.960+0000 7f2d4900b640 1 -- 172.21.15.169:0/956709805 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2d2c0a2800 msgr2=0x7f2d2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:42.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.960+0000 7f2d4900b640 1 --2- 172.21.15.169:0/956709805 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2d2c0a2800 0x7f2d2c0a4cc0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2d34006010 tx=0x7f2d34000bc0 comp rx=0 tx=0).stop 2024-03-28T01:09:42.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.960+0000 7f2d4900b640 1 -- 172.21.15.169:0/956709805 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d441005a0 msgr2=0x7f2d44069390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:42.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.960+0000 7f2d4900b640 1 --2- 172.21.15.169:0/956709805 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d441005a0 0x7f2d44069390 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f2d3800d770 tx=0x7f2d3800dc30 comp rx=0 tx=0).stop 2024-03-28T01:09:42.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.960+0000 7f2d4900b640 1 -- 172.21.15.169:0/956709805 shutdown_connections 2024-03-28T01:09:42.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.960+0000 7f2d4900b640 1 --2- 172.21.15.169:0/956709805 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f2d2c0a2800 0x7f2d2c0a4cc0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:42.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.961+0000 7f2d4900b640 1 --2- 172.21.15.169:0/956709805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d44102150 0x7f2d4419ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:42.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.961+0000 7f2d4900b640 1 --2- 172.21.15.169:0/956709805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d44101790 0x7f2d4419c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:42.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.961+0000 7f2d4900b640 1 --2- 172.21.15.169:0/956709805 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d441005a0 0x7f2d44069390 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:42.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.961+0000 7f2d4900b640 1 -- 172.21.15.169:0/956709805 >> 172.21.15.169:0/956709805 conn(0x7f2d440fbb30 msgr2=0x7f2d44104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:42.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.961+0000 7f2d4900b640 1 -- 172.21.15.169:0/956709805 shutdown_connections 2024-03-28T01:09:42.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:42.961+0000 7f2d4900b640 1 -- 172.21.15.169:0/956709805 wait complete. 2024-03-28T01:09:42.980 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-28T01:09:43.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.087+0000 7f5e252a1640 1 -- 172.21.15.169:0/3924877632 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e20107390 msgr2=0x7f5e201097e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:43.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.087+0000 7f5e252a1640 1 --2- 172.21.15.169:0/3924877632 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e20107390 0x7f5e201097e0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f5e18009ca0 tx=0x7f5e1802c0f0 comp rx=0 tx=0).stop 2024-03-28T01:09:43.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.087+0000 7f5e252a1640 1 -- 172.21.15.169:0/3924877632 shutdown_connections 2024-03-28T01:09:43.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.087+0000 7f5e252a1640 1 --2- 172.21.15.169:0/3924877632 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e20107390 0x7f5e201097e0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:43.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.087+0000 7f5e252a1640 1 --2- 172.21.15.169:0/3924877632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e20104a60 0x7f5e20106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:43.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.087+0000 7f5e252a1640 1 --2- 172.21.15.169:0/3924877632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e20102130 0x7f5e20104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:43.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.087+0000 7f5e252a1640 1 -- 172.21.15.169:0/3924877632 >> 172.21.15.169:0/3924877632 conn(0x7f5e200fbb20 msgr2=0x7f5e200fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:43.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.087+0000 7f5e252a1640 1 -- 172.21.15.169:0/3924877632 shutdown_connections 2024-03-28T01:09:43.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.087+0000 7f5e252a1640 1 -- 172.21.15.169:0/3924877632 wait complete. 2024-03-28T01:09:43.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.088+0000 7f5e252a1640 1 Processor -- start 2024-03-28T01:09:43.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.088+0000 7f5e252a1640 1 -- start start 2024-03-28T01:09:43.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.088+0000 7f5e252a1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e20102130 0x7f5e20197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:43.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.088+0000 7f5e252a1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e20104a60 0x7f5e20197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:43.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.088+0000 7f5e252a1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e20107390 0x7f5e2019ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:43.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.088+0000 7f5e252a1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2010d350 con 0x7f5e20104a60 2024-03-28T01:09:43.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.088+0000 7f5e252a1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2010d1d0 con 0x7f5e20107390 2024-03-28T01:09:43.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.088+0000 7f5e252a1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2010d4d0 con 0x7f5e20102130 2024-03-28T01:09:43.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.088+0000 7f5e167fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e20104a60 0x7f5e20197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:43.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.088+0000 7f5e167fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e20104a60 0x7f5e20197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49594/0 (socket says 172.21.15.169:49594) 2024-03-28T01:09:43.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.088+0000 7f5e167fc640 1 -- 172.21.15.169:0/1414325171 learned_addr learned my addr 172.21.15.169:0/1414325171 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-03-28T01:09:43.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.088+0000 7f5e1effd640 1 --2- 172.21.15.169:0/1414325171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e20102130 0x7f5e20197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:43.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.088+0000 7f5e1f7fe640 1 --2- 172.21.15.169:0/1414325171 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e20107390 0x7f5e2019ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:43.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.088+0000 7f5e167fc640 1 -- 172.21.15.169:0/1414325171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e20102130 msgr2=0x7f5e20197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:43.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.088+0000 7f5e167fc640 1 --2- 172.21.15.169:0/1414325171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e20102130 0x7f5e20197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:43.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.088+0000 7f5e167fc640 1 -- 172.21.15.169:0/1414325171 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e20107390 msgr2=0x7f5e2019ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:43.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.088+0000 7f5e167fc640 1 --2- 172.21.15.169:0/1414325171 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e20107390 0x7f5e2019ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:43.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.088+0000 7f5e167fc640 1 -- 172.21.15.169:0/1414325171 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2019f570 con 0x7f5e20104a60 2024-03-28T01:09:43.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.089+0000 7f5e167fc640 1 --2- 172.21.15.169:0/1414325171 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e20104a60 0x7f5e20197e90 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f5e0800d770 tx=0x7f5e0800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:43.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.089+0000 7f5e1cff9640 1 -- 172.21.15.169:0/1414325171 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e08017070 con 0x7f5e20104a60 2024-03-28T01:09:43.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.089+0000 7f5e1cff9640 1 -- 172.21.15.169:0/1414325171 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e08009c30 con 0x7f5e20104a60 2024-03-28T01:09:43.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.089+0000 7f5e252a1640 1 -- 172.21.15.169:0/1414325171 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e2019f860 con 0x7f5e20104a60 2024-03-28T01:09:43.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.089+0000 7f5e1cff9640 1 -- 172.21.15.169:0/1414325171 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0800f620 con 0x7f5e20104a60 2024-03-28T01:09:43.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.089+0000 7f5e252a1640 1 -- 172.21.15.169:0/1414325171 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e2019fdb0 con 0x7f5e20104a60 2024-03-28T01:09:43.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.090+0000 7f5e252a1640 1 -- 172.21.15.169:0/1414325171 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e200fd610 con 0x7f5e20104a60 2024-03-28T01:09:43.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.091+0000 7f5e1cff9640 1 -- 172.21.15.169:0/1414325171 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5e0801a070 con 0x7f5e20104a60 2024-03-28T01:09:43.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.091+0000 7f5e1cff9640 1 --2- 172.21.15.169:0/1414325171 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5e0c0a2800 0x7f5e0c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T01:09:43.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.092+0000 7f5e1cff9640 1 -- 172.21.15.169:0/1414325171 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8324+0+0 (secure 0 0 0) 0x7f5e080ce4e0 con 0x7f5e20104a60 2024-03-28T01:09:43.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.092+0000 7f5e1effd640 1 --2- 172.21.15.169:0/1414325171 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5e0c0a2800 0x7f5e0c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T01:09:43.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.092+0000 7f5e1effd640 1 --2- 172.21.15.169:0/1414325171 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5e0c0a2800 0x7f5e0c0a4cc0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5e10009bd0 tx=0x7f5e10005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T01:09:43.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.096+0000 7f5e1cff9640 1 -- 172.21.15.169:0/1414325171 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5e08009de0 con 0x7f5e20104a60 2024-03-28T01:09:43.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.286+0000 7f5e252a1640 1 -- 172.21.15.169:0/1414325171 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5e2002d070 con 0x7f5e20104a60 2024-03-28T01:09:43.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:42 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/3819127998' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T01:09:43.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:42 smithi169 ceph-mon[32274]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T01:09:43.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:42 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/956709805' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T01:09:43.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:42 smithi169 ceph-mon[32274]: pgmap v749: 97 pgs: 97 active+clean; 229 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T01:09:43.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:42 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/3819127998' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T01:09:43.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:42 smithi174 ceph-mon[36060]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T01:09:43.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:42 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/956709805' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T01:09:43.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:42 smithi174 ceph-mon[36060]: pgmap v749: 97 pgs: 97 active+clean; 229 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T01:09:43.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:42 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/3819127998' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T01:09:43.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:42 smithi177 ceph-mon[36206]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T01:09:43.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:42 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/956709805' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T01:09:43.417 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:42 smithi177 ceph-mon[36206]: pgmap v749: 97 pgs: 97 active+clean; 229 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T01:09:43.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.958+0000 7f5e1cff9640 1 -- 172.21.15.169:0/1414325171 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v98) v1 ==== 178+0+0 (secure 0 0 0) 0x7f5e0809aed0 con 0x7f5e20104a60 2024-03-28T01:09:43.961 INFO:teuthology.orchestra.run.smithi169.stderr:pool 'cephfs_data_ec' removed 2024-03-28T01:09:43.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.961+0000 7f5e252a1640 1 -- 172.21.15.169:0/1414325171 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5e0c0a2800 msgr2=0x7f5e0c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:43.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.961+0000 7f5e252a1640 1 --2- 172.21.15.169:0/1414325171 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5e0c0a2800 0x7f5e0c0a4cc0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5e10009bd0 tx=0x7f5e10005f90 comp rx=0 tx=0).stop 2024-03-28T01:09:43.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.961+0000 7f5e252a1640 1 -- 172.21.15.169:0/1414325171 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e20104a60 msgr2=0x7f5e20197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T01:09:43.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.961+0000 7f5e252a1640 1 --2- 172.21.15.169:0/1414325171 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e20104a60 0x7f5e20197e90 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f5e0800d770 tx=0x7f5e0800dc30 comp rx=0 tx=0).stop 2024-03-28T01:09:43.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.961+0000 7f5e252a1640 1 -- 172.21.15.169:0/1414325171 shutdown_connections 2024-03-28T01:09:43.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.961+0000 7f5e252a1640 1 --2- 172.21.15.169:0/1414325171 >> [v2:172.21.15.169:6800/1069722251,v1:172.21.15.169:6801/1069722251] conn(0x7f5e0c0a2800 0x7f5e0c0a4cc0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:43.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.961+0000 7f5e252a1640 1 --2- 172.21.15.169:0/1414325171 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e20107390 0x7f5e2019ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:43.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.961+0000 7f5e252a1640 1 --2- 172.21.15.169:0/1414325171 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e20104a60 0x7f5e20197e90 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:43.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.961+0000 7f5e252a1640 1 --2- 172.21.15.169:0/1414325171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e20102130 0x7f5e20197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T01:09:43.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.961+0000 7f5e252a1640 1 -- 172.21.15.169:0/1414325171 >> 172.21.15.169:0/1414325171 conn(0x7f5e200fbb20 msgr2=0x7f5e20102a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T01:09:43.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.961+0000 7f5e252a1640 1 -- 172.21.15.169:0/1414325171 shutdown_connections 2024-03-28T01:09:43.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-28T01:09:43.961+0000 7f5e252a1640 1 -- 172.21.15.169:0/1414325171 wait complete. 2024-03-28T01:09:43.984 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T01:09:44.027 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T01:09:44.291 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:43 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/956709805' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T01:09:44.291 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:43 smithi174 ceph-mon[36060]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T01:09:44.291 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:43 smithi174 ceph-mon[36060]: from='client.? 172.21.15.169:0/1414325171' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T01:09:44.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:43 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/956709805' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T01:09:44.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:43 smithi169 ceph-mon[32274]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T01:09:44.301 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:43 smithi169 ceph-mon[32274]: from='client.? 172.21.15.169:0/1414325171' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T01:09:44.308 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T01:09:44.346 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:43 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/956709805' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T01:09:44.346 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:43 smithi177 ceph-mon[36206]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T01:09:44.346 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:43 smithi177 ceph-mon[36206]: from='client.? 172.21.15.169:0/1414325171' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T01:09:44.350 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-28T01:09:44.350 DEBUG:teuthology.orchestra.run.smithi169:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T01:09:44.370 DEBUG:teuthology.orchestra.run.smithi174:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T01:09:44.385 DEBUG:teuthology.orchestra.run.smithi177:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T01:09:44.406 INFO:tasks.cephadm:Stopping all daemons... 2024-03-28T01:09:44.406 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-28T01:09:44.406 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@mon.a 2024-03-28T01:09:44.802 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:44 smithi169 systemd[1]: Stopping Ceph mon.a for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:09:45.164 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:44 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-a[32270]: 2024-03-28T01:09:44.876+0000 7f2d616dd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:09:45.164 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:44 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-a[32270]: 2024-03-28T01:09:44.876+0000 7f2d616dd700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-28T01:09:45.478 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:45 smithi169 podman[99634]: 2024-03-28 01:09:45.161923479 +0000 UTC m=+0.409879267 container died 559d191b9d44bf626ddc94c817add793be07571fbeda37425420013eca489a78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T01:09:45.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:45 smithi169 podman[99634]: 2024-03-28 01:09:45.577190419 +0000 UTC m=+0.825146195 container cleanup 559d191b9d44bf626ddc94c817add793be07571fbeda37425420013eca489a78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-28T01:09:45.801 INFO:journalctl@ceph.mon.a.smithi169.stdout:Mar 28 01:09:45 smithi169 bash[99634]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-a 2024-03-28T01:09:45.983 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mon.a.service' 2024-03-28T01:09:46.025 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:09:46.025 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-28T01:09:46.026 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-28T01:09:46.026 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@mon.b 2024-03-28T01:09:46.358 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:46 smithi174 systemd[1]: Stopping Ceph mon.b for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:09:46.797 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:46 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-b[36055]: 2024-03-28T01:09:46.399+0000 7fb826c50700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:09:46.797 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:46 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-b[36055]: 2024-03-28T01:09:46.399+0000 7fb826c50700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-28T01:09:46.797 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:46 smithi174 podman[70983]: 2024-03-28 01:09:46.460469444 +0000 UTC m=+0.223611771 container died 19bd7a2dd6fec9218147a3188bb19d3962f792cde05ec15c5571e92af254d1e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-b, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-28T01:09:47.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:46 smithi174 podman[70983]: 2024-03-28 01:09:46.935837468 +0000 UTC m=+0.698979789 container cleanup 19bd7a2dd6fec9218147a3188bb19d3962f792cde05ec15c5571e92af254d1e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-b, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T01:09:47.108 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 28 01:09:46 smithi174 bash[70983]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-b 2024-03-28T01:09:47.479 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mon.b.service' 2024-03-28T01:09:47.533 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:09:47.533 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-28T01:09:47.534 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-28T01:09:47.534 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@mon.c 2024-03-28T01:09:47.878 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:47 smithi177 systemd[1]: Stopping Ceph mon.c for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:09:48.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:47 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-c[36187]: 2024-03-28T01:09:47.874+0000 7f52500cb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:09:48.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:47 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-c[36187]: 2024-03-28T01:09:47.874+0000 7f52500cb700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-28T01:09:48.167 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:47 smithi177 podman[72245]: 2024-03-28 01:09:47.905568675 +0000 UTC m=+0.134656579 container died e37383dc5f0d4f108e093043ceb3f246ae85b2c7d9d8498a955bc2a4912e0b7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T01:09:48.612 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:48 smithi177 podman[72245]: 2024-03-28 01:09:48.342168132 +0000 UTC m=+0.571256029 container cleanup e37383dc5f0d4f108e093043ceb3f246ae85b2c7d9d8498a955bc2a4912e0b7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-c, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T01:09:48.612 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:48 smithi177 bash[72245]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-c 2024-03-28T01:09:48.847 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mon.c.service' 2024-03-28T01:09:48.872 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:48 smithi177 podman[72259]: 2024-03-28 01:09:48.610328138 +0000 UTC m=+0.692067901 container remove e37383dc5f0d4f108e093043ceb3f246ae85b2c7d9d8498a955bc2a4912e0b7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mon-c, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-28T01:09:48.872 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:48 smithi177 systemd[1]: ceph-833368da-ec9c-11ee-95d0-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-28T01:09:48.872 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:48 smithi177 systemd[1]: Stopped Ceph mon.c for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T01:09:48.872 INFO:journalctl@ceph.mon.c.smithi177.stdout:Mar 28 01:09:48 smithi177 systemd[1]: ceph-833368da-ec9c-11ee-95d0-87774f69a715@mon.c.service: Consumed 14.334s CPU time. 2024-03-28T01:09:48.907 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:09:48.907 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-28T01:09:48.907 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-28T01:09:48.907 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@mgr.x 2024-03-28T01:09:49.301 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 01:09:48 smithi169 systemd[1]: Stopping Ceph mgr.x for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:09:49.737 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 01:09:49 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x[32507]: 2024-03-28T01:09:49.307+0000 7f52c9091700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:09:49.738 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 01:09:49 smithi169 podman[99754]: 2024-03-28 01:09:49.349355763 +0000 UTC m=+0.243699701 container died baba397a44a4429c5d6604d52739ce4b6b38d09d16f4c9860afb72300488736c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-28T01:09:50.051 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 01:09:49 smithi169 podman[99754]: 2024-03-28 01:09:49.850952217 +0000 UTC m=+0.745296145 container cleanup baba397a44a4429c5d6604d52739ce4b6b38d09d16f4c9860afb72300488736c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=) 2024-03-28T01:09:50.052 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Mar 28 01:09:49 smithi169 bash[99754]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-x 2024-03-28T01:09:50.249 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-28T01:09:50.288 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:09:50.289 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-28T01:09:50.289 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-28T01:09:50.289 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@mgr.y 2024-03-28T01:09:50.608 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 01:09:50 smithi174 systemd[1]: Stopping Ceph mgr.y for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:09:50.918 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 01:09:50 smithi174 podman[71115]: 2024-03-28 01:09:50.628416163 +0000 UTC m=+0.148779581 container died fbd1ef7452fc0364dda36b7ed13789ff03f3c3a6d573ba088c749e48285dd336 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319) 2024-03-28T01:09:51.294 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 01:09:50 smithi174 podman[71115]: 2024-03-28 01:09:50.982499712 +0000 UTC m=+0.502863144 container cleanup fbd1ef7452fc0364dda36b7ed13789ff03f3c3a6d573ba088c749e48285dd336 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T01:09:51.294 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Mar 28 01:09:50 smithi174 bash[71115]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-y 2024-03-28T01:09:51.444 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-28T01:09:51.482 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:09:51.482 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-28T01:09:51.482 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-28T01:09:51.482 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@mgr.z 2024-03-28T01:09:51.837 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 01:09:51 smithi177 systemd[1]: Stopping Ceph mgr.z for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:09:52.134 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 01:09:51 smithi177 podman[72380]: 2024-03-28 01:09:51.834255364 +0000 UTC m=+0.153325392 container died a1b0b6a946b255625c6f21de2e7ecd4d87085aa8f6cdcbecb018d29bc99bc3f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-28T01:09:52.416 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 01:09:52 smithi177 podman[72380]: 2024-03-28 01:09:52.239998789 +0000 UTC m=+0.559068816 container cleanup a1b0b6a946b255625c6f21de2e7ecd4d87085aa8f6cdcbecb018d29bc99bc3f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-28T01:09:52.417 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Mar 28 01:09:52 smithi177 bash[72380]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-mgr-z 2024-03-28T01:09:52.617 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mgr.z.service' 2024-03-28T01:09:52.655 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:09:52.656 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-28T01:09:52.656 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-28T01:09:52.656 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.0 2024-03-28T01:09:52.941 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 01:09:52 smithi169 systemd[1]: Stopping Ceph osd.0 for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:09:53.301 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 01:09:52 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-0[44478]: 2024-03-28T01:09:52.938+0000 7f7e2dc30700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:09:53.302 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 01:09:52 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-0[44478]: 2024-03-28T01:09:52.938+0000 7f7e2dc30700 -1 osd.0 98 *** Got signal Terminated *** 2024-03-28T01:09:53.302 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 01:09:52 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-0[44478]: 2024-03-28T01:09:52.938+0000 7f7e2dc30700 -1 osd.0 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T01:09:58.751 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 01:09:58 smithi169 podman[99879]: 2024-03-28 01:09:58.46322762 +0000 UTC m=+5.620444406 container died d9b13d3fea969c5376f50d7aa8fd53d94068ec71e0302a0bf3ea9757b7ff147f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-28T01:09:59.051 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 01:09:58 smithi169 podman[99879]: 2024-03-28 01:09:58.839831039 +0000 UTC m=+5.997047825 container cleanup d9b13d3fea969c5376f50d7aa8fd53d94068ec71e0302a0bf3ea9757b7ff147f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-28T01:09:59.051 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 01:09:58 smithi169 bash[99879]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-0 2024-03-28T01:09:59.452 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 01:09:59 smithi169 podman[99897]: 2024-03-28 01:09:59.136715711 +0000 UTC m=+0.672651775 container remove d9b13d3fea969c5376f50d7aa8fd53d94068ec71e0302a0bf3ea9757b7ff147f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T01:09:59.781 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 01:09:59 smithi169 podman[99989]: 2024-03-28 01:09:59.352541074 +0000 UTC m=+0.017931270 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T01:09:59.781 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 01:09:59 smithi169 podman[99989]: 2024-03-28 01:09:59.484209866 +0000 UTC m=+0.149600104 container create 589d7a0e2571116e3c503a59fa1cc1ef38c268c40c93b713ee735296c3d630a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T01:09:59.781 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 01:09:59 smithi169 podman[99989]: 2024-03-28 01:09:59.779200811 +0000 UTC m=+0.444591003 container init 589d7a0e2571116e3c503a59fa1cc1ef38c268c40c93b713ee735296c3d630a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T01:10:00.051 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 01:09:59 smithi169 podman[99989]: 2024-03-28 01:09:59.782532869 +0000 UTC m=+0.447923065 container start 589d7a0e2571116e3c503a59fa1cc1ef38c268c40c93b713ee735296c3d630a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-0-deactivate, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-28T01:10:00.052 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 01:09:59 smithi169 podman[99989]: 2024-03-28 01:09:59.929297089 +0000 UTC m=+0.594687283 container attach 589d7a0e2571116e3c503a59fa1cc1ef38c268c40c93b713ee735296c3d630a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T01:10:01.702 INFO:journalctl@ceph.osd.0.smithi169.stdout:Mar 28 01:10:01 smithi169 podman[99989]: 2024-03-28 01:10:01.354480985 +0000 UTC m=+2.019871178 container died 589d7a0e2571116e3c503a59fa1cc1ef38c268c40c93b713ee735296c3d630a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-28T01:10:02.015 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.0.service' 2024-03-28T01:10:02.063 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:10:02.064 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-28T01:10:02.064 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-28T01:10:02.064 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.1 2024-03-28T01:10:02.358 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 01:10:02 smithi174 systemd[1]: Stopping Ceph osd.1 for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:10:02.858 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 01:10:02 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-1[40666]: 2024-03-28T01:10:02.362+0000 7fbd74f19700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:10:02.858 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 01:10:02 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-1[40666]: 2024-03-28T01:10:02.362+0000 7fbd74f19700 -1 osd.1 98 *** Got signal Terminated *** 2024-03-28T01:10:02.858 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 01:10:02 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-1[40666]: 2024-03-28T01:10:02.362+0000 7fbd74f19700 -1 osd.1 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T01:10:08.358 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 01:10:08 smithi174 podman[71243]: 2024-03-28 01:10:08.059892938 +0000 UTC m=+5.805724439 container died 01586815ac4a5623aaf7ce43942c763b8442b28bbbdb37d549bccab9c84e81d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-28T01:10:08.740 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 01:10:08 smithi174 podman[71243]: 2024-03-28 01:10:08.461973689 +0000 UTC m=+6.207805178 container cleanup 01586815ac4a5623aaf7ce43942c763b8442b28bbbdb37d549bccab9c84e81d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-1, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-28T01:10:08.740 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 01:10:08 smithi174 bash[71243]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-1 2024-03-28T01:10:09.033 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 01:10:08 smithi174 podman[71260]: 2024-03-28 01:10:08.738458614 +0000 UTC m=+0.673477960 container remove 01586815ac4a5623aaf7ce43942c763b8442b28bbbdb37d549bccab9c84e81d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-1, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T01:10:09.311 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 01:10:09 smithi174 podman[71352]: 2024-03-28 01:10:08.934703381 +0000 UTC m=+0.018119580 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T01:10:09.311 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 01:10:09 smithi174 podman[71352]: 2024-03-28 01:10:09.064091291 +0000 UTC m=+0.147507480 container create 8ac4f7d10fff731da6db3fcccbc78980e293b103d88945972ea9c9336c92875e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T01:10:09.311 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 01:10:09 smithi174 podman[71352]: 2024-03-28 01:10:09.3084425 +0000 UTC m=+0.391858694 container init 8ac4f7d10fff731da6db3fcccbc78980e293b103d88945972ea9c9336c92875e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-1-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-28T01:10:09.608 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 01:10:09 smithi174 podman[71352]: 2024-03-28 01:10:09.312141662 +0000 UTC m=+0.395557853 container start 8ac4f7d10fff731da6db3fcccbc78980e293b103d88945972ea9c9336c92875e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-28T01:10:09.608 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 01:10:09 smithi174 podman[71352]: 2024-03-28 01:10:09.450490106 +0000 UTC m=+0.533906307 container attach 8ac4f7d10fff731da6db3fcccbc78980e293b103d88945972ea9c9336c92875e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-28T01:10:10.158 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 28 01:10:09 smithi174 podman[71352]: 2024-03-28 01:10:09.86258329 +0000 UTC m=+0.945999480 container died 8ac4f7d10fff731da6db3fcccbc78980e293b103d88945972ea9c9336c92875e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-28T01:10:10.483 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.1.service' 2024-03-28T01:10:10.525 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:10:10.525 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-28T01:10:10.526 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-28T01:10:10.526 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.2 2024-03-28T01:10:10.847 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 01:10:10 smithi177 systemd[1]: Stopping Ceph osd.2 for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:10:11.167 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 01:10:10 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-2[41334]: 2024-03-28T01:10:10.843+0000 7ff88cb37700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:10:11.167 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 01:10:10 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-2[41334]: 2024-03-28T01:10:10.843+0000 7ff88cb37700 -1 osd.2 98 *** Got signal Terminated *** 2024-03-28T01:10:11.167 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 01:10:10 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-2[41334]: 2024-03-28T01:10:10.843+0000 7ff88cb37700 -1 osd.2 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T01:10:16.740 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 01:10:16 smithi177 podman[72504]: 2024-03-28 01:10:16.469600681 +0000 UTC m=+5.750878016 container died cc239ec94f15007c67b80015f2829b8329933a02aeaaf8c8dd756ce83b47a215 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-28T01:10:17.167 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 01:10:16 smithi177 podman[72504]: 2024-03-28 01:10:16.83629794 +0000 UTC m=+6.117575275 container cleanup cc239ec94f15007c67b80015f2829b8329933a02aeaaf8c8dd756ce83b47a215 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T01:10:17.167 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 01:10:16 smithi177 bash[72504]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-2 2024-03-28T01:10:17.463 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 01:10:17 smithi177 podman[72523]: 2024-03-28 01:10:17.171925392 +0000 UTC m=+0.700185653 container remove cc239ec94f15007c67b80015f2829b8329933a02aeaaf8c8dd756ce83b47a215 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-28T01:10:17.753 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 01:10:17 smithi177 podman[72615]: 2024-03-28 01:10:17.363695983 +0000 UTC m=+0.017493467 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T01:10:17.754 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 01:10:17 smithi177 podman[72615]: 2024-03-28 01:10:17.49289114 +0000 UTC m=+0.146688612 container create 5e15a6c54612bc8d0cf55f761c79f791a815e24bcf9ac7ea0ab7fa023e14adc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T01:10:18.167 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 01:10:17 smithi177 podman[72615]: 2024-03-28 01:10:17.751003143 +0000 UTC m=+0.404800619 container init 5e15a6c54612bc8d0cf55f761c79f791a815e24bcf9ac7ea0ab7fa023e14adc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T01:10:18.167 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 01:10:17 smithi177 podman[72615]: 2024-03-28 01:10:17.755357112 +0000 UTC m=+0.409154586 container start 5e15a6c54612bc8d0cf55f761c79f791a815e24bcf9ac7ea0ab7fa023e14adc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True) 2024-03-28T01:10:18.167 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 01:10:17 smithi177 podman[72615]: 2024-03-28 01:10:17.876277138 +0000 UTC m=+0.530074634 container attach 5e15a6c54612bc8d0cf55f761c79f791a815e24bcf9ac7ea0ab7fa023e14adc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T01:10:18.605 INFO:journalctl@ceph.osd.2.smithi177.stdout:Mar 28 01:10:18 smithi177 podman[72615]: 2024-03-28 01:10:18.334127711 +0000 UTC m=+0.987925193 container died 5e15a6c54612bc8d0cf55f761c79f791a815e24bcf9ac7ea0ab7fa023e14adc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-2-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-28T01:10:18.979 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.2.service' 2024-03-28T01:10:19.022 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:10:19.023 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-28T01:10:19.023 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-28T01:10:19.023 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.3 2024-03-28T01:10:19.330 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 28 01:10:19 smithi169 systemd[1]: Stopping Ceph osd.3 for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:10:19.605 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 28 01:10:19 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-3[51405]: 2024-03-28T01:10:19.327+0000 7fb55c9bd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:10:19.605 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 28 01:10:19 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-3[51405]: 2024-03-28T01:10:19.327+0000 7fb55c9bd700 -1 osd.3 98 *** Got signal Terminated *** 2024-03-28T01:10:19.605 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 28 01:10:19 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-3[51405]: 2024-03-28T01:10:19.327+0000 7fb55c9bd700 -1 osd.3 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T01:10:20.167 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:19 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:19.848+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:20.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:20 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:20.174+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:21.167 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:20 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:20.829+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:21.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:20 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:20.802+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:21.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:21 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:21.166+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:22.166 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:21 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:21.834+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:22.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:21 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:21.848+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:22.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:22 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:22.215+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:23.051 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 28 01:10:22 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-3[51405]: 2024-03-28T01:10:22.673+0000 7fb558fb1700 -1 osd.3 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:56.500773+0000 front 2024-03-28T01:09:56.500830+0000 (oldest deadline 2024-03-28T01:10:21.800899+0000) 2024-03-28T01:10:23.051 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:22 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:22.857+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:23.167 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:22 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:22.801+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:23.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:23 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:23.189+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:24.051 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 28 01:10:23 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-3[51405]: 2024-03-28T01:10:23.717+0000 7fb558fb1700 -1 osd.3 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:56.500773+0000 front 2024-03-28T01:09:56.500830+0000 (oldest deadline 2024-03-28T01:10:21.800899+0000) 2024-03-28T01:10:24.052 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:23 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:23.886+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:24.167 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:23 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:23.812+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:24.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:23 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:23.877+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:24.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:24 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:24.143+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:25.167 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:24 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:24.857+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:25.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:24 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:24.851+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:25.301 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 28 01:10:25 smithi169 podman[100102]: 2024-03-28 01:10:25.078275983 +0000 UTC m=+5.864414872 container died 14077a9a999a19f09964b7e48a5452537b9aed6f99454c3d00b2817cf5342d59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-3, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-28T01:10:25.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:24 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:24.916+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:25.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:25 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:25.188+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:25.774 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 28 01:10:25 smithi169 podman[100102]: 2024-03-28 01:10:25.486361581 +0000 UTC m=+6.272500466 container cleanup 14077a9a999a19f09964b7e48a5452537b9aed6f99454c3d00b2817cf5342d59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-3, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-28T01:10:25.774 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 28 01:10:25 smithi169 bash[100102]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-3 2024-03-28T01:10:26.051 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 28 01:10:25 smithi169 podman[100133]: 2024-03-28 01:10:25.77167371 +0000 UTC m=+0.682878467 container remove 14077a9a999a19f09964b7e48a5452537b9aed6f99454c3d00b2817cf5342d59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-3, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T01:10:26.051 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:25 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:25.946+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:26.167 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:25 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:25.849+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:26.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:25 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:25.840+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:26.372 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 28 01:10:26 smithi169 podman[100223]: 2024-03-28 01:10:25.999851274 +0000 UTC m=+0.017963834 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T01:10:26.372 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 28 01:10:26 smithi169 podman[100223]: 2024-03-28 01:10:26.124922869 +0000 UTC m=+0.143035407 container create d2579453868d1943abf9b0c72909f17a07d4c6e984969d8175ded503316c7601 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-28T01:10:26.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:26 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:26.213+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:26.664 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 28 01:10:26 smithi169 podman[100223]: 2024-03-28 01:10:26.369760927 +0000 UTC m=+0.387873466 container init d2579453868d1943abf9b0c72909f17a07d4c6e984969d8175ded503316c7601 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-28T01:10:26.664 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 28 01:10:26 smithi169 podman[100223]: 2024-03-28 01:10:26.373517649 +0000 UTC m=+0.391630192 container start d2579453868d1943abf9b0c72909f17a07d4c6e984969d8175ded503316c7601 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T01:10:26.664 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 28 01:10:26 smithi169 podman[100223]: 2024-03-28 01:10:26.461105883 +0000 UTC m=+0.479218421 container attach d2579453868d1943abf9b0c72909f17a07d4c6e984969d8175ded503316c7601 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-3-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-03-28T01:10:26.976 INFO:journalctl@ceph.osd.3.smithi169.stdout:Mar 28 01:10:26 smithi169 podman[100223]: 2024-03-28 01:10:26.661426599 +0000 UTC m=+0.679539149 container died d2579453868d1943abf9b0c72909f17a07d4c6e984969d8175ded503316c7601 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T01:10:27.167 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:26 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:26.804+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:27.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:26 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:26.861+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:27.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:26 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:26.973+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:27.371 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.3.service' 2024-03-28T01:10:27.417 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:10:27.417 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-28T01:10:27.417 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-28T01:10:27.417 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.4 2024-03-28T01:10:27.608 INFO:journalctl@ceph.osd.4.smithi174.stdout:Mar 28 01:10:27 smithi174 systemd[1]: Stopping Ceph osd.4 for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:10:27.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:27 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:27.223+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:28.108 INFO:journalctl@ceph.osd.4.smithi174.stdout:Mar 28 01:10:27 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-4[47179]: 2024-03-28T01:10:27.721+0000 7f12ab10a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:10:28.108 INFO:journalctl@ceph.osd.4.smithi174.stdout:Mar 28 01:10:27 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-4[47179]: 2024-03-28T01:10:27.721+0000 7f12ab10a700 -1 osd.4 98 *** Got signal Terminated *** 2024-03-28T01:10:28.108 INFO:journalctl@ceph.osd.4.smithi174.stdout:Mar 28 01:10:27 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-4[47179]: 2024-03-28T01:10:27.721+0000 7f12ab10a700 -1 osd.4 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T01:10:28.166 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:27 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:27.817+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:28.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:27 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:27.902+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:28.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:27 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:27.993+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:28.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:28 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:28.227+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:29.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:28 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:28.657+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:29.167 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:28 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:28.800+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:29.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:28 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:28.915+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:29.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:28 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:28.915+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:29.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:29 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:29.042+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:29.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:29 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:29.228+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:29.608 INFO:journalctl@ceph.osd.4.smithi174.stdout:Mar 28 01:10:29 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-4[47179]: 2024-03-28T01:10:29.176+0000 7f12a36f6700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.170974+0000 front 2024-03-28T01:10:06.171007+0000 (oldest deadline 2024-03-28T01:10:28.471060+0000) 2024-03-28T01:10:29.916 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:29 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:29.776+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:29.916 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:29 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:29.776+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.187350+0000 front 2024-03-28T01:10:04.187368+0000 (oldest deadline 2024-03-28T01:10:29.487422+0000) 2024-03-28T01:10:30.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:29 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:29.701+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:30.166 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:29 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:29.914+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:30.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:29 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:29.914+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:30.551 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:30 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:30.090+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:30.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:30 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:30.103+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.063693+0000 front 2024-03-28T01:10:04.063666+0000 (oldest deadline 2024-03-28T01:10:29.363749+0000) 2024-03-28T01:10:30.608 INFO:journalctl@ceph.osd.4.smithi174.stdout:Mar 28 01:10:30 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-4[47179]: 2024-03-28T01:10:30.185+0000 7f12a36f6700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.170974+0000 front 2024-03-28T01:10:06.171007+0000 (oldest deadline 2024-03-28T01:10:28.471060+0000) 2024-03-28T01:10:30.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:30 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:30.218+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:30.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:30 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:30.652+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:30.917 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:30 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:30.800+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:30.917 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:30 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:30.800+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.187350+0000 front 2024-03-28T01:10:04.187368+0000 (oldest deadline 2024-03-28T01:10:29.487422+0000) 2024-03-28T01:10:31.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:30 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:30.663+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:31.416 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:30 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:30.961+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:31.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:30 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:30.961+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:31.444 INFO:journalctl@ceph.osd.4.smithi174.stdout:Mar 28 01:10:31 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-4[47179]: 2024-03-28T01:10:31.176+0000 7f12a36f6700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.170974+0000 front 2024-03-28T01:10:06.171007+0000 (oldest deadline 2024-03-28T01:10:28.471060+0000) 2024-03-28T01:10:31.445 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:31 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:31.210+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:31.551 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:31 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:31.087+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:31.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:31 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:31.122+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.063693+0000 front 2024-03-28T01:10:04.063666+0000 (oldest deadline 2024-03-28T01:10:29.363749+0000) 2024-03-28T01:10:31.858 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:31 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:31.680+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:31.916 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:31 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:31.612+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:31.917 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:31 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:31.805+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:31.917 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:31 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:31.805+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.187350+0000 front 2024-03-28T01:10:04.187368+0000 (oldest deadline 2024-03-28T01:10:29.487422+0000) 2024-03-28T01:10:32.358 INFO:journalctl@ceph.osd.4.smithi174.stdout:Mar 28 01:10:32 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-4[47179]: 2024-03-28T01:10:32.132+0000 7f12a36f6700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.170974+0000 front 2024-03-28T01:10:06.171007+0000 (oldest deadline 2024-03-28T01:10:28.471060+0000) 2024-03-28T01:10:32.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:32 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:32.169+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:32.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:31 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:31.985+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:32.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:31 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:31.985+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:32.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:32 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:32.145+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.063693+0000 front 2024-03-28T01:10:04.063666+0000 (oldest deadline 2024-03-28T01:10:29.363749+0000) 2024-03-28T01:10:32.551 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:32 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:32.109+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:32.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:32 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:32.109+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:32.850 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:32 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:32.562+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:33.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:32 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:32.707+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:33.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:33 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:33.032+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:33.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:33 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:33.032+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:33.167 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:32 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:32.848+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:33.167 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:32 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:32.848+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.187350+0000 front 2024-03-28T01:10:04.187368+0000 (oldest deadline 2024-03-28T01:10:29.487422+0000) 2024-03-28T01:10:33.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:33 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:33.104+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.063693+0000 front 2024-03-28T01:10:04.063666+0000 (oldest deadline 2024-03-28T01:10:29.363749+0000) 2024-03-28T01:10:33.551 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:33 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:33.137+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:33.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:33 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:33.137+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:33.608 INFO:journalctl@ceph.osd.4.smithi174.stdout:Mar 28 01:10:33 smithi174 podman[71474]: 2024-03-28 01:10:33.360790236 +0000 UTC m=+5.751839799 container died 592754e094e7a1ef1f2b5e5b8cbd00e659bb74e22fe1f5be92fa2b5e45354d3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-28T01:10:33.609 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:33 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:33.133+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:33.609 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:33 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:33.133+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:33.821 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:33 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:33.551+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:34.001 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:33 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:33.721+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:34.001 INFO:journalctl@ceph.osd.4.smithi174.stdout:Mar 28 01:10:33 smithi174 podman[71474]: 2024-03-28 01:10:33.747806184 +0000 UTC m=+6.138855741 container cleanup 592754e094e7a1ef1f2b5e5b8cbd00e659bb74e22fe1f5be92fa2b5e45354d3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-4, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-28T01:10:34.001 INFO:journalctl@ceph.osd.4.smithi174.stdout:Mar 28 01:10:33 smithi174 bash[71474]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-4 2024-03-28T01:10:34.166 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:33 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:33.818+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:34.167 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:33 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:33.818+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.187350+0000 front 2024-03-28T01:10:04.187368+0000 (oldest deadline 2024-03-28T01:10:29.487422+0000) 2024-03-28T01:10:34.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:34 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:34.065+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:34.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:34 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:34.065+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:34.310 INFO:journalctl@ceph.osd.4.smithi174.stdout:Mar 28 01:10:33 smithi174 podman[71492]: 2024-03-28 01:10:33.999018003 +0000 UTC m=+0.633674767 container remove 592754e094e7a1ef1f2b5e5b8cbd00e659bb74e22fe1f5be92fa2b5e45354d3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-4, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-28T01:10:34.311 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:34 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:34.131+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:34.311 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:34 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:34.131+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:34.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:34 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:34.094+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.063693+0000 front 2024-03-28T01:10:04.063666+0000 (oldest deadline 2024-03-28T01:10:29.363749+0000) 2024-03-28T01:10:34.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:34 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:34.138+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:34.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:34 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:34.138+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:34.608 INFO:journalctl@ceph.osd.4.smithi174.stdout:Mar 28 01:10:34 smithi174 podman[71586]: 2024-03-28 01:10:34.211413657 +0000 UTC m=+0.017057499 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T01:10:34.608 INFO:journalctl@ceph.osd.4.smithi174.stdout:Mar 28 01:10:34 smithi174 podman[71586]: 2024-03-28 01:10:34.324750561 +0000 UTC m=+0.130394386 container create 551f7b93529d5e9cf3ea7e66332897826dffb0d5ce98f560a133b69541c1a7aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T01:10:34.608 INFO:journalctl@ceph.osd.4.smithi174.stdout:Mar 28 01:10:34 smithi174 podman[71586]: 2024-03-28 01:10:34.527385145 +0000 UTC m=+0.333028989 container init 551f7b93529d5e9cf3ea7e66332897826dffb0d5ce98f560a133b69541c1a7aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-28T01:10:34.608 INFO:journalctl@ceph.osd.4.smithi174.stdout:Mar 28 01:10:34 smithi174 podman[71586]: 2024-03-28 01:10:34.530376614 +0000 UTC m=+0.336020446 container start 551f7b93529d5e9cf3ea7e66332897826dffb0d5ce98f560a133b69541c1a7aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-4-deactivate, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T01:10:34.869 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:34 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:34.545+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:35.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:34 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:34.744+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:35.108 INFO:journalctl@ceph.osd.4.smithi174.stdout:Mar 28 01:10:34 smithi174 podman[71586]: 2024-03-28 01:10:34.627173667 +0000 UTC m=+0.432817515 container attach 551f7b93529d5e9cf3ea7e66332897826dffb0d5ce98f560a133b69541c1a7aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=) 2024-03-28T01:10:35.108 INFO:journalctl@ceph.osd.4.smithi174.stdout:Mar 28 01:10:34 smithi174 podman[71586]: 2024-03-28 01:10:34.821052546 +0000 UTC m=+0.626696387 container died 551f7b93529d5e9cf3ea7e66332897826dffb0d5ce98f560a133b69541c1a7aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-28T01:10:35.166 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:35 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:35.029+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:35.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:35 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:35.029+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:35.167 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:34 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:34.867+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:35.167 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:34 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:34.867+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.187350+0000 front 2024-03-28T01:10:04.187368+0000 (oldest deadline 2024-03-28T01:10:29.487422+0000) 2024-03-28T01:10:35.501 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:35 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:35.129+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:35.501 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:35 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:35.129+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:35.522 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.4.service' 2024-03-28T01:10:35.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:35 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:35.120+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.063693+0000 front 2024-03-28T01:10:04.063666+0000 (oldest deadline 2024-03-28T01:10:29.363749+0000) 2024-03-28T01:10:35.551 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:35 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:35.186+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:35.551 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:35 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:35.186+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:35.565 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:10:35.565 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-28T01:10:35.565 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-28T01:10:35.566 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.5 2024-03-28T01:10:35.849 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:35 smithi177 systemd[1]: Stopping Ceph osd.5 for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:10:35.849 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:35 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:35.571+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:35.857 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:35 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:35.742+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:36.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:36 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:36.022+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:36.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:36 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:36.022+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:36.167 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:35 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:35.846+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:36.167 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:35 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:35.846+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.187350+0000 front 2024-03-28T01:10:04.187368+0000 (oldest deadline 2024-03-28T01:10:29.487422+0000) 2024-03-28T01:10:36.167 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:35 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:35.883+0000 7faf8887e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:10:36.167 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:35 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:35.883+0000 7faf8887e700 -1 osd.5 98 *** Got signal Terminated *** 2024-03-28T01:10:36.167 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:35 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:35.883+0000 7faf8887e700 -1 osd.5 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T01:10:36.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:36 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:36.080+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:36.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:36 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:36.080+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:36.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:36 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:36.120+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.063693+0000 front 2024-03-28T01:10:04.063666+0000 (oldest deadline 2024-03-28T01:10:29.363749+0000) 2024-03-28T01:10:36.551 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:36 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:36.209+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:36.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:36 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:36.209+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:36.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:36 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:36.564+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:36.917 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:36 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:36.805+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:36.917 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:36 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:36.805+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.187350+0000 front 2024-03-28T01:10:04.187368+0000 (oldest deadline 2024-03-28T01:10:29.487422+0000) 2024-03-28T01:10:37.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:36 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:36.722+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:37.416 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:37 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:37.028+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:37.416 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:37 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:37.028+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:37.428 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:37 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:37.086+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.063693+0000 front 2024-03-28T01:10:04.063666+0000 (oldest deadline 2024-03-28T01:10:29.363749+0000) 2024-03-28T01:10:37.429 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:37 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:37.173+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:37.429 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:37 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:37.173+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:37.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:37 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:37.129+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:37.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:37 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:37.129+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:37.917 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:37 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:37.773+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:37.917 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:37 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:37.773+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.187350+0000 front 2024-03-28T01:10:04.187368+0000 (oldest deadline 2024-03-28T01:10:29.487422+0000) 2024-03-28T01:10:37.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:37 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:37.527+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:37.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:37 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:37.527+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:38.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:37 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:37.719+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:38.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:38 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:38.067+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:38.418 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:38 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:38.067+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:38.418 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:38 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:38.067+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:38.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:38 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:38.070+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.063693+0000 front 2024-03-28T01:10:04.063666+0000 (oldest deadline 2024-03-28T01:10:29.363749+0000) 2024-03-28T01:10:38.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:38 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:38.182+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:38.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:38 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:38.182+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:38.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:38 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:38.133+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:38.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:38 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:38.133+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:38.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:38 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:38.547+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:38.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:38 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:38.547+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:38.917 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:38 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:38.770+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:38.917 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:38 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:38.770+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.187350+0000 front 2024-03-28T01:10:04.187368+0000 (oldest deadline 2024-03-28T01:10:29.487422+0000) 2024-03-28T01:10:39.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:38 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:38.757+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:39.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:38 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:38.757+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.370246+0000 front 2024-03-28T01:10:14.370338+0000 (oldest deadline 2024-03-28T01:10:38.470180+0000) 2024-03-28T01:10:39.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:39 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:39.099+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:39.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:39 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:39.099+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:39.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:39 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:39.099+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:39.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:39 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:39.119+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.063693+0000 front 2024-03-28T01:10:04.063666+0000 (oldest deadline 2024-03-28T01:10:29.363749+0000) 2024-03-28T01:10:39.551 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:39 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:39.146+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:39.551 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:39 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:39.146+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:39.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:39 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:39.146+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:10:39.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:39 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:39.128+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:39.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:39 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:39.128+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:39.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:39 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:39.128+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:10:39.820 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:39 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:39.556+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:39.821 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:39 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:39.556+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:40.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:39 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:39.719+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:40.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:39 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:39.719+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.370246+0000 front 2024-03-28T01:10:14.370338+0000 (oldest deadline 2024-03-28T01:10:38.470180+0000) 2024-03-28T01:10:40.150 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:39 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:39.817+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:40.150 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:39 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:39.817+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.187350+0000 front 2024-03-28T01:10:04.187368+0000 (oldest deadline 2024-03-28T01:10:29.487422+0000) 2024-03-28T01:10:40.150 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:39 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:39.817+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:15.388401+0000 front 2024-03-28T01:10:15.388425+0000 (oldest deadline 2024-03-28T01:10:39.488524+0000) 2024-03-28T01:10:40.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:40 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:40.146+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:40.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:40 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:40.146+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:40.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:40 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:40.146+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:40.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:40 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:40.126+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.063693+0000 front 2024-03-28T01:10:04.063666+0000 (oldest deadline 2024-03-28T01:10:29.363749+0000) 2024-03-28T01:10:40.551 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:40 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:40.161+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:40.551 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:40 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:40.161+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:40.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:40 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:40.161+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:10:40.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:40 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:40.158+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:40.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:40 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:40.158+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:40.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:40 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:40.158+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:10:40.865 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:40 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:40.569+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:40.866 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:40 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:40.569+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:41.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:40 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:40.757+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:41.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:40 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:40.757+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.370246+0000 front 2024-03-28T01:10:14.370338+0000 (oldest deadline 2024-03-28T01:10:38.470180+0000) 2024-03-28T01:10:41.130 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:40 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:40.860+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:52.986239+0000 front 2024-03-28T01:09:52.986228+0000 (oldest deadline 2024-03-28T01:10:18.886200+0000) 2024-03-28T01:10:41.130 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:40 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:40.860+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.187350+0000 front 2024-03-28T01:10:04.187368+0000 (oldest deadline 2024-03-28T01:10:29.487422+0000) 2024-03-28T01:10:41.130 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:40 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5[47992]: 2024-03-28T01:10:40.860+0000 7faf84e72700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:15.388401+0000 front 2024-03-28T01:10:15.388425+0000 (oldest deadline 2024-03-28T01:10:39.488524+0000) 2024-03-28T01:10:41.416 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:41 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:41.127+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:41.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:41 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:41.127+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:41.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:41 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:41.127+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:41.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:41 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:41.134+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.063693+0000 front 2024-03-28T01:10:04.063666+0000 (oldest deadline 2024-03-28T01:10:29.363749+0000) 2024-03-28T01:10:41.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:41 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:41.134+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.865083+0000 front 2024-03-28T01:10:12.865012+0000 (oldest deadline 2024-03-28T01:10:40.465474+0000) 2024-03-28T01:10:41.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:41 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:41.128+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:41.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:41 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:41.128+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:41.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:41 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:41.128+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:10:41.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:41 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:41.137+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:41.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:41 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:41.137+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:41.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:41 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:41.137+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:10:41.916 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:41 smithi177 podman[72733]: 2024-03-28 01:10:41.733303015 +0000 UTC m=+5.977999642 container died 2a21aa6e83e330d6f35a89349f9742c954f17b3c198cd47b67edb9fee2f501df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-28T01:10:41.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:41 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:41.590+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:41.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:41 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:41.590+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:42.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:41 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:41.768+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:42.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:41 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:41.768+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.370246+0000 front 2024-03-28T01:10:14.370338+0000 (oldest deadline 2024-03-28T01:10:38.470180+0000) 2024-03-28T01:10:42.417 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:42 smithi177 podman[72733]: 2024-03-28 01:10:42.140770186 +0000 UTC m=+6.385466796 container cleanup 2a21aa6e83e330d6f35a89349f9742c954f17b3c198cd47b67edb9fee2f501df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2) 2024-03-28T01:10:42.417 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:42 smithi177 bash[72733]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5 2024-03-28T01:10:42.418 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:42 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:42.137+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:42.418 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:42 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:42.137+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:42.418 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:42 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:42.137+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:42.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:42 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:42.149+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.063693+0000 front 2024-03-28T01:10:04.063666+0000 (oldest deadline 2024-03-28T01:10:29.363749+0000) 2024-03-28T01:10:42.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:42 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:42.149+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.865083+0000 front 2024-03-28T01:10:12.865012+0000 (oldest deadline 2024-03-28T01:10:40.465474+0000) 2024-03-28T01:10:42.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:42 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:42.120+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:42.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:42 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:42.120+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:42.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:42 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:42.120+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:10:42.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:42 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:42.143+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:42.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:42 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:42.143+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:42.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:42 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:42.143+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:10:42.739 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:42 smithi177 podman[72750]: 2024-03-28 01:10:42.433825758 +0000 UTC m=+0.697570737 container remove 2a21aa6e83e330d6f35a89349f9742c954f17b3c198cd47b67edb9fee2f501df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-28T01:10:42.740 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:42 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:42.574+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:42.740 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:42 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:42.574+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:43.065 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:42 smithi177 podman[72839]: 2024-03-28 01:10:42.640827844 +0000 UTC m=+0.017858083 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T01:10:43.065 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:42 smithi177 podman[72839]: 2024-03-28 01:10:42.768189868 +0000 UTC m=+0.145220090 container create eac4265b54715ca8397170f30acaf83069a54df0946dd6deb6233232607aad28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS) 2024-03-28T01:10:43.065 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:42 smithi177 podman[72839]: 2024-03-28 01:10:42.970674818 +0000 UTC m=+0.347705039 container init eac4265b54715ca8397170f30acaf83069a54df0946dd6deb6233232607aad28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-28T01:10:43.065 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:42 smithi177 podman[72839]: 2024-03-28 01:10:42.973966502 +0000 UTC m=+0.350996728 container start eac4265b54715ca8397170f30acaf83069a54df0946dd6deb6233232607aad28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-28T01:10:43.065 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:43 smithi177 podman[72839]: 2024-03-28 01:10:43.062596959 +0000 UTC m=+0.439627194 container attach eac4265b54715ca8397170f30acaf83069a54df0946dd6deb6233232607aad28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T01:10:43.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:42 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:42.733+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:43.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:42 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:42.733+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.370246+0000 front 2024-03-28T01:10:14.370338+0000 (oldest deadline 2024-03-28T01:10:38.470180+0000) 2024-03-28T01:10:43.417 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 28 01:10:43 smithi177 podman[72839]: 2024-03-28 01:10:43.272234031 +0000 UTC m=+0.649264271 container died eac4265b54715ca8397170f30acaf83069a54df0946dd6deb6233232607aad28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T01:10:43.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:43 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:43.096+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:43.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:43 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:43.096+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:43.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:43 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:43.096+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:43.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:43 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:43.160+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.063693+0000 front 2024-03-28T01:10:04.063666+0000 (oldest deadline 2024-03-28T01:10:29.363749+0000) 2024-03-28T01:10:43.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:43 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:43.160+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.865083+0000 front 2024-03-28T01:10:12.865012+0000 (oldest deadline 2024-03-28T01:10:40.465474+0000) 2024-03-28T01:10:43.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:43 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:43.101+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:43.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:43 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:43.101+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:43.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:43 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:43.101+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:10:43.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:43 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:43.107+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:43.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:43 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:43.107+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:43.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:43 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:43.107+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:10:43.893 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:43 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:43.605+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:43.893 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:43 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:43.605+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:43.914 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.5.service' 2024-03-28T01:10:43.956 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:10:43.956 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-28T01:10:43.956 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-28T01:10:43.956 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.6 2024-03-28T01:10:44.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:43 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:43.756+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:44.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:43 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:43.756+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.370246+0000 front 2024-03-28T01:10:14.370338+0000 (oldest deadline 2024-03-28T01:10:38.470180+0000) 2024-03-28T01:10:44.166 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:44 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:44.123+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:44.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:44 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:44.123+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:44.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:44 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:44.123+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:44.295 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:44 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:44.093+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:44.296 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:44 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:44.093+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:44.296 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:44 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:44.093+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:10:44.296 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:44 smithi169 systemd[1]: Stopping Ceph osd.6 for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:10:44.296 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:44 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:44.154+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.063693+0000 front 2024-03-28T01:10:04.063666+0000 (oldest deadline 2024-03-28T01:10:29.363749+0000) 2024-03-28T01:10:44.296 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:44 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:44.154+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.865083+0000 front 2024-03-28T01:10:12.865012+0000 (oldest deadline 2024-03-28T01:10:40.465474+0000) 2024-03-28T01:10:44.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:44 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:44.292+0000 7fa2f5642700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:10:44.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:44 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:44.292+0000 7fa2f5642700 -1 osd.6 98 *** Got signal Terminated *** 2024-03-28T01:10:44.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:44 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:44.292+0000 7fa2f5642700 -1 osd.6 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T01:10:44.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:44 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:44.153+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:44.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:44 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:44.153+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:44.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:44 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:44.153+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:10:44.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:44 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:44.631+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:44.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:44 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:44.631+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:45.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:44 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:44.790+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:45.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:44 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:44.790+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.370246+0000 front 2024-03-28T01:10:14.370338+0000 (oldest deadline 2024-03-28T01:10:38.470180+0000) 2024-03-28T01:10:45.416 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:45 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:45.088+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:45.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:45 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:45.088+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:45.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:45 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:45.088+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:45.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:45 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:45.179+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.063693+0000 front 2024-03-28T01:10:04.063666+0000 (oldest deadline 2024-03-28T01:10:29.363749+0000) 2024-03-28T01:10:45.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:45 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:45.179+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.865083+0000 front 2024-03-28T01:10:12.865012+0000 (oldest deadline 2024-03-28T01:10:40.465474+0000) 2024-03-28T01:10:45.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:45 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:45.082+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:45.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:45 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:45.082+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:45.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:45 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:45.082+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:10:45.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:45 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:45.132+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:45.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:45 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:45.132+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:45.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:45 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:45.132+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:10:45.916 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:45 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:45.649+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:45.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:45 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:45.649+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:46.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:45 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:45.789+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:46.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:45 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:45.789+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.370246+0000 front 2024-03-28T01:10:14.370338+0000 (oldest deadline 2024-03-28T01:10:38.470180+0000) 2024-03-28T01:10:46.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:45 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:45.789+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:19.571333+0000 front 2024-03-28T01:10:19.571346+0000 (oldest deadline 2024-03-28T01:10:45.471267+0000) 2024-03-28T01:10:46.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:46 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:46.044+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:46.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:46 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:46.044+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:46.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:46 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:46.044+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:46.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:46 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:46.132+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.063693+0000 front 2024-03-28T01:10:04.063666+0000 (oldest deadline 2024-03-28T01:10:29.363749+0000) 2024-03-28T01:10:46.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:46 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:46.132+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.865083+0000 front 2024-03-28T01:10:12.865012+0000 (oldest deadline 2024-03-28T01:10:40.465474+0000) 2024-03-28T01:10:46.551 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:46 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:46.102+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:46.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:46 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:46.102+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:46.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:46 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:46.102+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:10:46.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:46 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:46.102+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:10:46.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:46 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:46.137+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:46.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:46 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:46.137+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:46.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:46 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:46.137+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:10:46.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:46 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:46.628+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:46.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:46 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:46.628+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:47.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:46 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:46.838+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:47.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:46 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:46.838+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.370246+0000 front 2024-03-28T01:10:14.370338+0000 (oldest deadline 2024-03-28T01:10:38.470180+0000) 2024-03-28T01:10:47.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:46 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:46.838+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:19.571333+0000 front 2024-03-28T01:10:19.571346+0000 (oldest deadline 2024-03-28T01:10:45.471267+0000) 2024-03-28T01:10:47.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:47 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:47.045+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:47.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:47 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:47.045+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:47.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:47 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:47.045+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:47.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:47 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:47.176+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.063693+0000 front 2024-03-28T01:10:04.063666+0000 (oldest deadline 2024-03-28T01:10:29.363749+0000) 2024-03-28T01:10:47.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:47 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:47.176+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.865083+0000 front 2024-03-28T01:10:12.865012+0000 (oldest deadline 2024-03-28T01:10:40.465474+0000) 2024-03-28T01:10:47.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:47 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:47.073+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:47.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:47 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:47.073+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:47.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:47 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:47.073+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:10:47.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:47 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:47.073+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:10:47.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:47 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:47.184+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:47.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:47 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:47.184+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:47.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:47 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:47.184+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:10:47.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:47 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:47.617+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:47.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:47 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:47.617+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:47.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:47 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:47.617+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:20.747734+0000 front 2024-03-28T01:10:20.747924+0000 (oldest deadline 2024-03-28T01:10:46.647752+0000) 2024-03-28T01:10:48.140 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:47 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:47.882+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:48.140 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:47 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:47.882+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.370246+0000 front 2024-03-28T01:10:14.370338+0000 (oldest deadline 2024-03-28T01:10:38.470180+0000) 2024-03-28T01:10:48.140 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:47 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:47.882+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:19.571333+0000 front 2024-03-28T01:10:19.571346+0000 (oldest deadline 2024-03-28T01:10:45.471267+0000) 2024-03-28T01:10:48.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:48 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:48.088+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:48.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:48 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:48.088+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:48.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:48 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:48.088+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:48.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:48 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:48.195+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.063693+0000 front 2024-03-28T01:10:04.063666+0000 (oldest deadline 2024-03-28T01:10:29.363749+0000) 2024-03-28T01:10:48.551 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:48 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:48.195+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.865083+0000 front 2024-03-28T01:10:12.865012+0000 (oldest deadline 2024-03-28T01:10:40.465474+0000) 2024-03-28T01:10:48.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:48 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:48.075+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:48.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:48 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:48.075+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:48.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:48 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:48.075+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:10:48.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:48 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:48.075+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:10:48.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:48 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:48.136+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:48.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:48 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:48.136+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:48.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:48 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:48.136+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:10:48.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:48 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:48.136+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:10:48.687 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:48 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:48.587+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:48.687 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:48 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:48.587+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:48.687 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:48 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:48.587+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:20.747734+0000 front 2024-03-28T01:10:20.747924+0000 (oldest deadline 2024-03-28T01:10:46.647752+0000) 2024-03-28T01:10:49.105 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:48 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:48.842+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:49.105 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:48 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:48.842+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.370246+0000 front 2024-03-28T01:10:14.370338+0000 (oldest deadline 2024-03-28T01:10:38.470180+0000) 2024-03-28T01:10:49.105 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:48 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:48.842+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:19.571333+0000 front 2024-03-28T01:10:19.571346+0000 (oldest deadline 2024-03-28T01:10:45.471267+0000) 2024-03-28T01:10:49.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:49 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:49.102+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:49.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:49 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:49.102+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:49.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:49 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:49.102+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:10:49.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:49 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:49.102+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:10:49.416 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:49 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:49.130+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:49.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:49 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:49.130+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:49.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:49 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:49.130+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:49.452 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:49 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:49.071+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:49.452 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:49 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:49.071+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:49.452 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:49 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:49.071+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:10:49.452 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:49 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:49.071+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:10:49.453 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:49 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:49.230+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:04.063693+0000 front 2024-03-28T01:10:04.063666+0000 (oldest deadline 2024-03-28T01:10:29.363749+0000) 2024-03-28T01:10:49.453 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:49 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:49.230+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.865083+0000 front 2024-03-28T01:10:12.865012+0000 (oldest deadline 2024-03-28T01:10:40.465474+0000) 2024-03-28T01:10:49.453 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:49 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6[58399]: 2024-03-28T01:10:49.230+0000 7fa2edc2e700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.366449+0000 front 2024-03-28T01:10:23.366428+0000 (oldest deadline 2024-03-28T01:10:48.666615+0000) 2024-03-28T01:10:49.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:49 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:49.637+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:49.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:49 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:49.637+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:49.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:49 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:49.637+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:20.747734+0000 front 2024-03-28T01:10:20.747924+0000 (oldest deadline 2024-03-28T01:10:46.647752+0000) 2024-03-28T01:10:50.111 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:49 smithi169 podman[100328]: 2024-03-28 01:10:49.85744282 +0000 UTC m=+5.708127961 container died 8df1010710b3bbffbea16faacd7165596ccbe99f15aea18cc7b6e71d4f8638ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-28T01:10:50.259 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:49 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:49.872+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:50.259 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:49 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:49.872+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.370246+0000 front 2024-03-28T01:10:14.370338+0000 (oldest deadline 2024-03-28T01:10:38.470180+0000) 2024-03-28T01:10:50.260 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:49 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:49.872+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:19.571333+0000 front 2024-03-28T01:10:19.571346+0000 (oldest deadline 2024-03-28T01:10:45.471267+0000) 2024-03-28T01:10:50.260 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:50 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:50.062+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:50.260 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:50 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:50.062+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:50.260 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:50 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:50.062+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:10:50.260 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:50 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:50.062+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:10:50.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:50 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:50.098+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:50.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:50 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:50.098+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:50.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:50 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:50.098+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:50.513 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:50 smithi169 podman[100328]: 2024-03-28 01:10:50.25189151 +0000 UTC m=+6.102576664 container cleanup 8df1010710b3bbffbea16faacd7165596ccbe99f15aea18cc7b6e71d4f8638ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True) 2024-03-28T01:10:50.513 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:50 smithi169 bash[100328]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6 2024-03-28T01:10:50.514 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:50 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:50.107+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:50.514 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:50 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:50.107+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:50.514 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:50 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:50.107+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:10:50.514 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:50 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:50.107+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:10:50.801 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:50 smithi169 podman[100345]: 2024-03-28 01:10:50.520057503 +0000 UTC m=+0.663306855 container remove 8df1010710b3bbffbea16faacd7165596ccbe99f15aea18cc7b6e71d4f8638ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T01:10:50.916 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:50 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:50.640+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:50.916 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:50 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:50.640+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:50.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:50 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:50.640+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:20.747734+0000 front 2024-03-28T01:10:20.747924+0000 (oldest deadline 2024-03-28T01:10:46.647752+0000) 2024-03-28T01:10:51.168 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:50 smithi169 podman[100446]: 2024-03-28 01:10:50.79777533 +0000 UTC m=+0.019786558 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T01:10:51.168 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:50 smithi169 podman[100446]: 2024-03-28 01:10:50.92113002 +0000 UTC m=+0.143141231 container create 8f5a44a4f2ef4e0f36e49bb4e264da967ebd129c4c7c6a090e8f69bc508bffce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-28T01:10:51.168 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:51 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:51.062+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:51.168 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:51 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:51.062+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:51.168 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:51 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:51.062+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:10:51.169 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:51 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:51.062+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:10:51.358 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:50 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:50.909+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:51.358 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:50 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:50.909+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.370246+0000 front 2024-03-28T01:10:14.370338+0000 (oldest deadline 2024-03-28T01:10:38.470180+0000) 2024-03-28T01:10:51.358 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:50 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:50.909+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:19.571333+0000 front 2024-03-28T01:10:19.571346+0000 (oldest deadline 2024-03-28T01:10:45.471267+0000) 2024-03-28T01:10:51.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:51 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:51.028+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:51.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:51 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:51.028+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:51.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:51 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:51.028+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:10:51.359 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:51 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:51.028+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:10:51.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:51 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:51.129+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:51.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:51 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:51.129+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:51.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:51 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:51.129+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:51.474 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:51 smithi169 podman[100446]: 2024-03-28 01:10:51.165650337 +0000 UTC m=+0.387661556 container init 8f5a44a4f2ef4e0f36e49bb4e264da967ebd129c4c7c6a090e8f69bc508bffce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-28T01:10:51.475 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:51 smithi169 podman[100446]: 2024-03-28 01:10:51.169108376 +0000 UTC m=+0.391119595 container start 8f5a44a4f2ef4e0f36e49bb4e264da967ebd129c4c7c6a090e8f69bc508bffce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-28T01:10:51.475 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:51 smithi169 podman[100446]: 2024-03-28 01:10:51.240848296 +0000 UTC m=+0.462859519 container attach 8f5a44a4f2ef4e0f36e49bb4e264da967ebd129c4c7c6a090e8f69bc508bffce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-28T01:10:51.796 INFO:journalctl@ceph.osd.6.smithi169.stdout:Mar 28 01:10:51 smithi169 podman[100446]: 2024-03-28 01:10:51.472613822 +0000 UTC m=+0.694625037 container died 8f5a44a4f2ef4e0f36e49bb4e264da967ebd129c4c7c6a090e8f69bc508bffce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-6-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-28T01:10:51.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:51 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:51.620+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:51.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:51 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:51.620+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:51.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:51 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:51.620+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:20.747734+0000 front 2024-03-28T01:10:20.747924+0000 (oldest deadline 2024-03-28T01:10:46.647752+0000) 2024-03-28T01:10:52.051 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:52 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:52.022+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:52.051 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:52 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:52.022+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:52.051 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:52 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:52.022+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:10:52.051 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:52 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:52.022+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:10:52.122 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.6.service' 2024-03-28T01:10:52.164 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:10:52.164 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-28T01:10:52.164 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-28T01:10:52.164 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.7 2024-03-28T01:10:52.194 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:51 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:51.892+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:52.194 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:51 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:51.892+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.370246+0000 front 2024-03-28T01:10:14.370338+0000 (oldest deadline 2024-03-28T01:10:38.470180+0000) 2024-03-28T01:10:52.194 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:51 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:51.892+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:19.571333+0000 front 2024-03-28T01:10:19.571346+0000 (oldest deadline 2024-03-28T01:10:45.471267+0000) 2024-03-28T01:10:52.194 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:52 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:52.008+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:52.195 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:52 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:52.008+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:52.195 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:52 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:52.008+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:10:52.195 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:52 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:52.008+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:10:52.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:52 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:52.087+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:52.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:52 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:52.087+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:52.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:52 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:52.087+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:52.494 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:52 smithi174 systemd[1]: Stopping Ceph osd.7 for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:10:52.858 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:52 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:52.490+0000 7fbaec83d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:10:52.858 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:52 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:52.490+0000 7fbaec83d700 -1 osd.7 98 *** Got signal Terminated *** 2024-03-28T01:10:52.858 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:52 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:52.490+0000 7fbaec83d700 -1 osd.7 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T01:10:52.916 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:52 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:52.650+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:52.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:52 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:52.650+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:52.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:52 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:52.650+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:20.747734+0000 front 2024-03-28T01:10:20.747924+0000 (oldest deadline 2024-03-28T01:10:46.647752+0000) 2024-03-28T01:10:53.358 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:52 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:52.858+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:53.358 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:52 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:52.858+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.370246+0000 front 2024-03-28T01:10:14.370338+0000 (oldest deadline 2024-03-28T01:10:38.470180+0000) 2024-03-28T01:10:53.358 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:52 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:52.858+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:19.571333+0000 front 2024-03-28T01:10:19.571346+0000 (oldest deadline 2024-03-28T01:10:45.471267+0000) 2024-03-28T01:10:53.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:53 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:53.049+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:53.359 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:53 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:53.049+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:53.359 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:53 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:53.049+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:10:53.359 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:53 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:53.049+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:10:53.416 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:53 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:53.123+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:53.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:53 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:53.123+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:53.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:53 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:53.123+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:53.551 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:53 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:53.065+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:53.551 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:53 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:53.065+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:53.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:53 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:53.065+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:10:53.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:53 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:53.065+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:10:54.099 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:53 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:53.668+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:54.100 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:53 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:53.668+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:54.100 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:53 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:53.668+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:20.747734+0000 front 2024-03-28T01:10:20.747924+0000 (oldest deadline 2024-03-28T01:10:46.647752+0000) 2024-03-28T01:10:54.100 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:53 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:53.668+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:32.548756+0000 front 2024-03-28T01:10:32.548850+0000 (oldest deadline 2024-03-28T01:10:53.648720+0000) 2024-03-28T01:10:54.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:53 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:53.823+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:54.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:53 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:53.823+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.370246+0000 front 2024-03-28T01:10:14.370338+0000 (oldest deadline 2024-03-28T01:10:38.470180+0000) 2024-03-28T01:10:54.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:53 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:53.823+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:19.571333+0000 front 2024-03-28T01:10:19.571346+0000 (oldest deadline 2024-03-28T01:10:45.471267+0000) 2024-03-28T01:10:54.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:53 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:53.823+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:32.472846+0000 front 2024-03-28T01:10:32.472889+0000 (oldest deadline 2024-03-28T01:10:53.572924+0000) 2024-03-28T01:10:54.108 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:54 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:54.058+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:54.108 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:54 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:54.058+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:54.108 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:54 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:54.058+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:10:54.109 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:54 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:54.058+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:10:54.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:54 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:54.033+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:54.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:54 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:54.033+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:54.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:54 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:54.033+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:10:54.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:54 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:54.033+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:10:54.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:54 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:54.096+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:54.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:54 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:54.096+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:54.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:54 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:54.096+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:54.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:54 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:54.096+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:10:54.916 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:54 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:54.640+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:54.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:54 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:54.640+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:54.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:54 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:54.640+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:20.747734+0000 front 2024-03-28T01:10:20.747924+0000 (oldest deadline 2024-03-28T01:10:46.647752+0000) 2024-03-28T01:10:54.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:54 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:54.640+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:32.548756+0000 front 2024-03-28T01:10:32.548850+0000 (oldest deadline 2024-03-28T01:10:53.648720+0000) 2024-03-28T01:10:55.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:54 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:54.834+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:55.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:54 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:54.834+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.370246+0000 front 2024-03-28T01:10:14.370338+0000 (oldest deadline 2024-03-28T01:10:38.470180+0000) 2024-03-28T01:10:55.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:54 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:54.834+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:19.571333+0000 front 2024-03-28T01:10:19.571346+0000 (oldest deadline 2024-03-28T01:10:45.471267+0000) 2024-03-28T01:10:55.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:54 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:54.834+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:32.472846+0000 front 2024-03-28T01:10:32.472889+0000 (oldest deadline 2024-03-28T01:10:53.572924+0000) 2024-03-28T01:10:55.109 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:55 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:55.059+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:55.109 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:55 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:55.059+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:55.109 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:55 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:55.059+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:10:55.109 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:55 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:55.059+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:10:55.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:54 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:54.992+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:55.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:54 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:54.992+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:55.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:54 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:54.992+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:10:55.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:54 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:54.992+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:10:55.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:54 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:54.992+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.795010+0000 front 2024-03-28T01:10:31.794967+0000 (oldest deadline 2024-03-28T01:10:54.094722+0000) 2024-03-28T01:10:55.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:55 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:55.065+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:55.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:55 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:55.065+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:55.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:55 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:55.065+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:55.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:55 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:55.065+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:10:56.032 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:55 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:55.689+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:56.032 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:55 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:55.689+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:56.032 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:55 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:55.689+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:20.747734+0000 front 2024-03-28T01:10:20.747924+0000 (oldest deadline 2024-03-28T01:10:46.647752+0000) 2024-03-28T01:10:56.032 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:55 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:55.689+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:32.548756+0000 front 2024-03-28T01:10:32.548850+0000 (oldest deadline 2024-03-28T01:10:53.648720+0000) 2024-03-28T01:10:56.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:56 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:56.036+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:56.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:56 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:56.036+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:56.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:56 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:56.036+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:10:56.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:56 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:56.036+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:10:56.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:56 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:56.036+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.795010+0000 front 2024-03-28T01:10:31.794967+0000 (oldest deadline 2024-03-28T01:10:54.094722+0000) 2024-03-28T01:10:56.358 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:55 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:55.872+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:56.358 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:55 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:55.872+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.370246+0000 front 2024-03-28T01:10:14.370338+0000 (oldest deadline 2024-03-28T01:10:38.470180+0000) 2024-03-28T01:10:56.358 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:55 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:55.872+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:19.571333+0000 front 2024-03-28T01:10:19.571346+0000 (oldest deadline 2024-03-28T01:10:45.471267+0000) 2024-03-28T01:10:56.358 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:55 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:55.872+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:32.472846+0000 front 2024-03-28T01:10:32.472889+0000 (oldest deadline 2024-03-28T01:10:53.572924+0000) 2024-03-28T01:10:56.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:56 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:56.093+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:56.359 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:56 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:56.093+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:56.359 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:56 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:56.093+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:10:56.359 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:56 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:56.093+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:10:56.359 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:56 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:56.093+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:10:56.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:56 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:56.029+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:56.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:56 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:56.029+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:56.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:56 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:56.029+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:56.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:56 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:56.029+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:10:57.025 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:56 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:56.727+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:57.025 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:56 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:56.727+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:57.025 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:56 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:56.727+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:20.747734+0000 front 2024-03-28T01:10:20.747924+0000 (oldest deadline 2024-03-28T01:10:46.647752+0000) 2024-03-28T01:10:57.025 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:56 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:56.727+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:32.548756+0000 front 2024-03-28T01:10:32.548850+0000 (oldest deadline 2024-03-28T01:10:53.648720+0000) 2024-03-28T01:10:57.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:57 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:57.040+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:57.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:57 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:57.040+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:57.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:57 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:57.040+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:10:57.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:57 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:57.040+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:10:57.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:57 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:57.040+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.795010+0000 front 2024-03-28T01:10:31.794967+0000 (oldest deadline 2024-03-28T01:10:54.094722+0000) 2024-03-28T01:10:57.358 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:56 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:56.887+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:03.169067+0000 front 2024-03-28T01:10:03.169048+0000 (oldest deadline 2024-03-28T01:10:28.469204+0000) 2024-03-28T01:10:57.358 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:56 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:56.887+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.370246+0000 front 2024-03-28T01:10:14.370338+0000 (oldest deadline 2024-03-28T01:10:38.470180+0000) 2024-03-28T01:10:57.358 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:56 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:56.887+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:19.571333+0000 front 2024-03-28T01:10:19.571346+0000 (oldest deadline 2024-03-28T01:10:45.471267+0000) 2024-03-28T01:10:57.358 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:56 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7[53766]: 2024-03-28T01:10:56.887+0000 7fbae8e31700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:32.472846+0000 front 2024-03-28T01:10:32.472889+0000 (oldest deadline 2024-03-28T01:10:53.572924+0000) 2024-03-28T01:10:57.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:57 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:57.112+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:57.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:57 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:57.112+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:57.359 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:57 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:57.112+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:10:57.359 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:57 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:57.112+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:10:57.359 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:57 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:57.112+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:10:57.416 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:57 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:57.022+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:57.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:57 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:57.022+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:57.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:57 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:57.022+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:57.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:57 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:57.022+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:10:58.026 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:57 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:57.725+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:58.026 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:57 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:57.725+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:58.026 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:57 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:57.725+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:20.747734+0000 front 2024-03-28T01:10:20.747924+0000 (oldest deadline 2024-03-28T01:10:46.647752+0000) 2024-03-28T01:10:58.026 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:57 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:57.725+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:32.548756+0000 front 2024-03-28T01:10:32.548850+0000 (oldest deadline 2024-03-28T01:10:53.648720+0000) 2024-03-28T01:10:58.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:58 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:58.043+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:58.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:58 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:58.043+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:58.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:58 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:58.043+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:10:58.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:58 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:58.043+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:10:58.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:58 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:58.043+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.795010+0000 front 2024-03-28T01:10:31.794967+0000 (oldest deadline 2024-03-28T01:10:54.094722+0000) 2024-03-28T01:10:58.358 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:58 smithi174 podman[71703]: 2024-03-28 01:10:58.102400117 +0000 UTC m=+5.745346676 container died 6df5fc0be6c872e13cf1642c402f9625ddb4281b2329597e1dab6f3b501b6f88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T01:10:58.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:58 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:58.136+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:58.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:58 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:58.136+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:58.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:58 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:58.136+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:10:58.359 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:58 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:58.136+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:10:58.359 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:58 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:58.136+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:10:58.416 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:58 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:58.023+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:58.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:58 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:58.023+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:58.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:58 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:58.023+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:58.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:58 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:58.023+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:10:58.858 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:58 smithi174 podman[71703]: 2024-03-28 01:10:58.432745971 +0000 UTC m=+6.075692512 container cleanup 6df5fc0be6c872e13cf1642c402f9625ddb4281b2329597e1dab6f3b501b6f88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-28T01:10:58.858 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:58 smithi174 bash[71703]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7 2024-03-28T01:10:58.858 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:58 smithi174 podman[71720]: 2024-03-28 01:10:58.584115003 +0000 UTC m=+0.485848146 container remove 6df5fc0be6c872e13cf1642c402f9625ddb4281b2329597e1dab6f3b501b6f88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-03-28T01:10:59.116 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:58 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:58.733+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:59.116 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:58 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:58.733+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:59.117 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:58 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:58.733+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:20.747734+0000 front 2024-03-28T01:10:20.747924+0000 (oldest deadline 2024-03-28T01:10:46.647752+0000) 2024-03-28T01:10:59.117 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:58 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:58.733+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:32.548756+0000 front 2024-03-28T01:10:32.548850+0000 (oldest deadline 2024-03-28T01:10:53.648720+0000) 2024-03-28T01:10:59.117 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:58 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:58.979+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:10:59.117 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:58 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:58.979+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:10:59.117 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:58 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:58.979+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:10:59.117 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:58 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:58.979+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:10:59.248 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:58 smithi174 podman[71816]: 2024-03-28 01:10:58.888861128 +0000 UTC m=+0.017595761 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T01:10:59.248 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:58 smithi174 podman[71816]: 2024-03-28 01:10:58.993037287 +0000 UTC m=+0.121771905 container create bb9342a834c9b5abde278220669d9a9325c5224144e07bd25f520816b89245d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7-deactivate, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-28T01:10:59.249 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:59 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:59.142+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:10:59.249 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:59 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:59.142+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:10:59.249 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:59 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:59.142+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:10:59.249 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:59 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:59.142+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:10:59.249 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:10:59 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:10:59.142+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:10:59.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:59 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:59.005+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:10:59.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:59 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:59.005+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:10:59.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:59 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:59.005+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:10:59.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:59 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:59.005+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:10:59.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:59 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:59.005+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.795010+0000 front 2024-03-28T01:10:31.794967+0000 (oldest deadline 2024-03-28T01:10:54.094722+0000) 2024-03-28T01:10:59.535 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:59 smithi174 podman[71816]: 2024-03-28 01:10:59.245838776 +0000 UTC m=+0.374573419 container init bb9342a834c9b5abde278220669d9a9325c5224144e07bd25f520816b89245d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-28T01:10:59.535 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:59 smithi174 podman[71816]: 2024-03-28 01:10:59.249847145 +0000 UTC m=+0.378581770 container start bb9342a834c9b5abde278220669d9a9325c5224144e07bd25f520816b89245d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T01:10:59.535 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:59 smithi174 podman[71816]: 2024-03-28 01:10:59.370372036 +0000 UTC m=+0.499106657 container attach bb9342a834c9b5abde278220669d9a9325c5224144e07bd25f520816b89245d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T01:10:59.955 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:59 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:59.725+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:10:59.956 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:59 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:59.725+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:10:59.956 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:59 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:59.725+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:20.747734+0000 front 2024-03-28T01:10:20.747924+0000 (oldest deadline 2024-03-28T01:10:46.647752+0000) 2024-03-28T01:10:59.956 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:10:59 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:10:59.725+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:32.548756+0000 front 2024-03-28T01:10:32.548850+0000 (oldest deadline 2024-03-28T01:10:53.648720+0000) 2024-03-28T01:11:00.108 INFO:journalctl@ceph.osd.7.smithi174.stdout:Mar 28 01:10:59 smithi174 podman[71816]: 2024-03-28 01:10:59.532607511 +0000 UTC m=+0.661342142 container died bb9342a834c9b5abde278220669d9a9325c5224144e07bd25f520816b89245d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T01:11:00.163 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.7.service' 2024-03-28T01:11:00.250 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:11:00.250 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-28T01:11:00.250 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-28T01:11:00.250 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.8 2024-03-28T01:11:00.281 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:59 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:59.952+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:00.281 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:59 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:59.952+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:00.281 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:59 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:59.952+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:00.281 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:10:59 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:10:59.952+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:00.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:59 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:59.967+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:11:00.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:59 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:59.967+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:11:00.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:59 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:59.967+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:11:00.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:59 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:59.967+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:11:00.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:10:59 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:10:59.967+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.795010+0000 front 2024-03-28T01:10:31.794967+0000 (oldest deadline 2024-03-28T01:10:54.094722+0000) 2024-03-28T01:11:00.554 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:00 smithi177 systemd[1]: Stopping Ceph osd.8 for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:11:00.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:00 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:00.106+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:00.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:00 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:00.106+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:00.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:00 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:00.106+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:00.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:00 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:00.106+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:00.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:00 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:00.106+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:00.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:00 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:00.550+0000 7fdc40b99700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:11:00.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:00 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:00.550+0000 7fdc40b99700 -1 osd.8 98 *** Got signal Terminated *** 2024-03-28T01:11:00.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:00 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:00.550+0000 7fdc40b99700 -1 osd.8 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T01:11:00.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:00 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:00.717+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:11:00.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:00 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:00.717+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:11:00.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:00 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:00.717+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:20.747734+0000 front 2024-03-28T01:10:20.747924+0000 (oldest deadline 2024-03-28T01:10:46.647752+0000) 2024-03-28T01:11:00.917 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:00 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:00.717+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:32.548756+0000 front 2024-03-28T01:10:32.548850+0000 (oldest deadline 2024-03-28T01:10:53.648720+0000) 2024-03-28T01:11:01.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:00 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:00.917+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:11:01.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:00 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:00.917+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:11:01.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:00 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:00.917+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:11:01.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:00 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:00.917+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:11:01.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:00 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:00.917+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.795010+0000 front 2024-03-28T01:10:31.794967+0000 (oldest deadline 2024-03-28T01:10:54.094722+0000) 2024-03-28T01:11:01.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:01 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:01.087+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:01.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:01 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:01.087+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:01.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:01 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:01.087+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:01.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:01 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:01.087+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:01.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:01 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:01.087+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:01.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:00 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:00.996+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:01.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:00 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:00.996+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:01.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:00 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:00.996+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:01.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:00 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:00.996+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:01.996 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:01 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:01.700+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:11:01.997 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:01 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:01.700+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:11:01.997 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:01 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:01.700+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:20.747734+0000 front 2024-03-28T01:10:20.747924+0000 (oldest deadline 2024-03-28T01:10:46.647752+0000) 2024-03-28T01:11:01.997 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:01 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:01.700+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:32.548756+0000 front 2024-03-28T01:10:32.548850+0000 (oldest deadline 2024-03-28T01:10:53.648720+0000) 2024-03-28T01:11:02.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:01 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:01.950+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:11:02.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:01 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:01.950+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:11:02.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:01 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:01.950+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:11:02.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:01 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:01.950+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:11:02.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:01 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:01.950+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.795010+0000 front 2024-03-28T01:10:31.794967+0000 (oldest deadline 2024-03-28T01:10:54.094722+0000) 2024-03-28T01:11:02.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:02 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:02.075+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:02.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:02 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:02.075+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:02.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:02 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:02.075+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:02.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:02 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:02.075+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:02.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:02 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:02.075+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:02.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:02 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:02.075+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.025466+0000 front 2024-03-28T01:10:36.025486+0000 (oldest deadline 2024-03-28T01:11:01.825685+0000) 2024-03-28T01:11:02.416 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:01 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:01.993+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:02.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:01 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:01.993+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:02.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:01 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:01.993+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:02.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:01 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:01.993+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:03.167 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:02 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:02.738+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:11:03.167 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:02 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:02.738+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:11:03.167 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:02 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:02.738+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:20.747734+0000 front 2024-03-28T01:10:20.747924+0000 (oldest deadline 2024-03-28T01:10:46.647752+0000) 2024-03-28T01:11:03.167 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:02 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:02.738+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:32.548756+0000 front 2024-03-28T01:10:32.548850+0000 (oldest deadline 2024-03-28T01:10:53.648720+0000) 2024-03-28T01:11:03.168 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:02 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:02.951+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:03.168 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:02 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:02.951+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:03.168 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:02 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:02.951+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:03.168 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:02 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:02.951+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:03.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:02 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:02.904+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:11:03.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:02 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:02.904+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:11:03.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:02 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:02.904+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:11:03.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:02 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:02.904+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:11:03.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:02 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:02.904+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.795010+0000 front 2024-03-28T01:10:31.794967+0000 (oldest deadline 2024-03-28T01:10:54.094722+0000) 2024-03-28T01:11:03.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:03 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:03.047+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:03.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:03 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:03.047+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:03.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:03 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:03.047+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:03.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:03 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:03.047+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:03.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:03 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:03.047+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:03.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:03 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:03.047+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.025466+0000 front 2024-03-28T01:10:36.025486+0000 (oldest deadline 2024-03-28T01:11:01.825685+0000) 2024-03-28T01:11:04.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:03 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:03.966+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:04.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:03 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:03.966+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:04.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:03 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:03.966+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:04.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:03 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:03.966+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:04.167 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:03 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:03.770+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:11:04.167 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:03 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:03.770+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:11:04.167 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:03 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:03.770+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:20.747734+0000 front 2024-03-28T01:10:20.747924+0000 (oldest deadline 2024-03-28T01:10:46.647752+0000) 2024-03-28T01:11:04.168 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:03 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:03.770+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:32.548756+0000 front 2024-03-28T01:10:32.548850+0000 (oldest deadline 2024-03-28T01:10:53.648720+0000) 2024-03-28T01:11:04.168 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:03 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:03.770+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:37.749619+0000 front 2024-03-28T01:10:37.749638+0000 (oldest deadline 2024-03-28T01:11:03.649674+0000) 2024-03-28T01:11:04.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:03 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:03.932+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:11:04.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:03 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:03.932+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:11:04.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:03 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:03.932+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:11:04.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:03 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:03.932+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:11:04.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:03 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:03.932+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.795010+0000 front 2024-03-28T01:10:31.794967+0000 (oldest deadline 2024-03-28T01:10:54.094722+0000) 2024-03-28T01:11:04.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:04 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:04.063+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:04.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:04 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:04.063+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:04.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:04 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:04.063+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:04.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:04 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:04.063+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:04.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:04 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:04.063+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:04.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:04 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:04.063+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.025466+0000 front 2024-03-28T01:10:36.025486+0000 (oldest deadline 2024-03-28T01:11:01.825685+0000) 2024-03-28T01:11:05.167 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:04 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:04.773+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:06.245483+0000 front 2024-03-28T01:10:06.245408+0000 (oldest deadline 2024-03-28T01:10:30.345390+0000) 2024-03-28T01:11:05.167 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:04 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:04.773+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:10.846184+0000 front 2024-03-28T01:10:10.846118+0000 (oldest deadline 2024-03-28T01:10:36.746254+0000) 2024-03-28T01:11:05.167 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:04 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:04.773+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:20.747734+0000 front 2024-03-28T01:10:20.747924+0000 (oldest deadline 2024-03-28T01:10:46.647752+0000) 2024-03-28T01:11:05.167 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:04 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:04.773+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:32.548756+0000 front 2024-03-28T01:10:32.548850+0000 (oldest deadline 2024-03-28T01:10:53.648720+0000) 2024-03-28T01:11:05.167 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:04 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8[54885]: 2024-03-28T01:11:04.773+0000 7fdc3d18d700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:37.749619+0000 front 2024-03-28T01:10:37.749638+0000 (oldest deadline 2024-03-28T01:11:03.649674+0000) 2024-03-28T01:11:05.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:04 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:04.972+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:05.168 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:04 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:04.972+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:05.168 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:04 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:04.972+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:05.168 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:04 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:04.972+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:05.168 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:04 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:04.972+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:05.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:04 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:04.919+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:11:05.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:04 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:04.919+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:11:05.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:04 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:04.919+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:11:05.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:04 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:04.919+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:11:05.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:04 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:04.919+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.795010+0000 front 2024-03-28T01:10:31.794967+0000 (oldest deadline 2024-03-28T01:10:54.094722+0000) 2024-03-28T01:11:05.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:04 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:04.919+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:39.896297+0000 front 2024-03-28T01:10:39.896345+0000 (oldest deadline 2024-03-28T01:11:03.996099+0000) 2024-03-28T01:11:05.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:05 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:05.028+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:05.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:05 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:05.028+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:05.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:05 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:05.028+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:05.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:05 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:05.028+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:05.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:05 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:05.028+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:05.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:05 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:05.028+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.025466+0000 front 2024-03-28T01:10:36.025486+0000 (oldest deadline 2024-03-28T01:11:01.825685+0000) 2024-03-28T01:11:06.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:05 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:05.951+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:11:06.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:05 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:05.951+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:11:06.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:05 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:05.951+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:11:06.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:05 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:05.951+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:11:06.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:05 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:05.951+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.795010+0000 front 2024-03-28T01:10:31.794967+0000 (oldest deadline 2024-03-28T01:10:54.094722+0000) 2024-03-28T01:11:06.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:05 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:05.951+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:39.896297+0000 front 2024-03-28T01:10:39.896345+0000 (oldest deadline 2024-03-28T01:11:03.996099+0000) 2024-03-28T01:11:06.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:06 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:06.060+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:06.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:06 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:06.060+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:06.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:06 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:06.060+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:06.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:06 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:06.060+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:06.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:06 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:06.060+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:06.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:06 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:06.060+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.025466+0000 front 2024-03-28T01:10:36.025486+0000 (oldest deadline 2024-03-28T01:11:01.825685+0000) 2024-03-28T01:11:06.405 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:06 smithi177 podman[72959]: 2024-03-28 01:11:06.187530691 +0000 UTC m=+5.746322596 container died b1ce4b588bacba1acefdd6c0fbca737086579b27deb5861ead646febdca373bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T01:11:06.405 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:05 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:05.976+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:06.405 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:05 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:05.976+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:06.405 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:05 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:05.976+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:06.405 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:05 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:05.976+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:06.405 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:05 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:05.976+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:06.667 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:06 smithi177 podman[72959]: 2024-03-28 01:11:06.492683484 +0000 UTC m=+6.051475368 container cleanup b1ce4b588bacba1acefdd6c0fbca737086579b27deb5861ead646febdca373bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T01:11:06.667 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:06 smithi177 bash[72959]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8 2024-03-28T01:11:07.167 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:06 smithi177 podman[72976]: 2024-03-28 01:11:06.860733216 +0000 UTC m=+0.672582945 container remove b1ce4b588bacba1acefdd6c0fbca737086579b27deb5861ead646febdca373bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-28T01:11:07.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:06 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:06.958+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:07.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:06 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:06.958+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:07.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:06 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:06.958+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:07.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:06 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:06.958+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:07.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:06 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:06.958+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:07.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:06 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:06.969+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:11:07.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:06 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:06.969+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:11:07.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:06 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:06.969+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:11:07.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:06 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:06.969+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:11:07.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:06 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:06.970+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.795010+0000 front 2024-03-28T01:10:31.794967+0000 (oldest deadline 2024-03-28T01:10:54.094722+0000) 2024-03-28T01:11:07.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:06 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:06.970+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:39.896297+0000 front 2024-03-28T01:10:39.896345+0000 (oldest deadline 2024-03-28T01:11:03.996099+0000) 2024-03-28T01:11:07.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:07 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:07.033+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:07.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:07 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:07.033+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:07.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:07 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:07.033+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:07.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:07 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:07.033+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:07.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:07 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:07.033+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:07.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:07 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:07.033+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.025466+0000 front 2024-03-28T01:10:36.025486+0000 (oldest deadline 2024-03-28T01:11:01.825685+0000) 2024-03-28T01:11:07.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:07 smithi177 podman[73063]: 2024-03-28 01:11:07.090524843 +0000 UTC m=+0.017898732 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T01:11:07.442 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:07 smithi177 podman[73063]: 2024-03-28 01:11:07.21184804 +0000 UTC m=+0.139221914 container create dc26fe95d78d333115ef1e946624be4c48b1e1546f959856f9943efb683a1b78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True) 2024-03-28T01:11:07.737 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:07 smithi177 podman[73063]: 2024-03-28 01:11:07.439776807 +0000 UTC m=+0.367150699 container init dc26fe95d78d333115ef1e946624be4c48b1e1546f959856f9943efb683a1b78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-28T01:11:07.737 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:07 smithi177 podman[73063]: 2024-03-28 01:11:07.443304908 +0000 UTC m=+0.370678783 container start dc26fe95d78d333115ef1e946624be4c48b1e1546f959856f9943efb683a1b78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True) 2024-03-28T01:11:07.737 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:07 smithi177 podman[73063]: 2024-03-28 01:11:07.547539068 +0000 UTC m=+0.474912957 container attach dc26fe95d78d333115ef1e946624be4c48b1e1546f959856f9943efb683a1b78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-28T01:11:08.007 INFO:journalctl@ceph.osd.8.smithi177.stdout:Mar 28 01:11:07 smithi177 podman[73063]: 2024-03-28 01:11:07.735005003 +0000 UTC m=+0.662378878 container died dc26fe95d78d333115ef1e946624be4c48b1e1546f959856f9943efb683a1b78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-8-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-28T01:11:08.287 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:08 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:08.004+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:08.287 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:08 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:08.004+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:08.287 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:08 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:08.004+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:08.287 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:08 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:08.004+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:08.287 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:08 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:08.004+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:08.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:07 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:07.989+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:11:08.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:07 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:07.989+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:11:08.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:07 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:07.989+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:11:08.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:07 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:07.989+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:11:08.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:07 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:07.989+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.795010+0000 front 2024-03-28T01:10:31.794967+0000 (oldest deadline 2024-03-28T01:10:54.094722+0000) 2024-03-28T01:11:08.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:07 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:07.989+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:39.896297+0000 front 2024-03-28T01:10:39.896345+0000 (oldest deadline 2024-03-28T01:11:03.996099+0000) 2024-03-28T01:11:08.313 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.8.service' 2024-03-28T01:11:08.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:07 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:07.992+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:08.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:07 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:07.992+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:08.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:07 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:07.992+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:08.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:07 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:07.992+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:08.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:07 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:07.992+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:08.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:07 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:07.992+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.025466+0000 front 2024-03-28T01:10:36.025486+0000 (oldest deadline 2024-03-28T01:11:01.825685+0000) 2024-03-28T01:11:08.394 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:11:08.395 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-28T01:11:08.395 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-28T01:11:08.395 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.9 2024-03-28T01:11:08.699 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:08 smithi169 systemd[1]: Stopping Ceph osd.9 for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:11:09.040 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:08 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:08.696+0000 7f5d306d4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:11:09.040 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:08 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:08.696+0000 7f5d306d4700 -1 osd.9 98 *** Got signal Terminated *** 2024-03-28T01:11:09.040 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:08 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:08.696+0000 7f5d306d4700 -1 osd.9 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T01:11:09.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:09 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:09.037+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:11:09.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:09 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:09.037+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:11:09.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:09 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:09.037+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:11:09.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:09 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:09.037+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:11:09.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:09 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:09.037+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.795010+0000 front 2024-03-28T01:10:31.794967+0000 (oldest deadline 2024-03-28T01:10:54.094722+0000) 2024-03-28T01:11:09.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:09 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:09.037+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:39.896297+0000 front 2024-03-28T01:10:39.896345+0000 (oldest deadline 2024-03-28T01:11:03.996099+0000) 2024-03-28T01:11:09.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:08 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:08.994+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:09.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:08 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:08.994+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:09.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:08 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:08.994+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:09.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:08 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:08.994+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:09.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:08 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:08.994+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:09.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:08 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:08.994+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.025466+0000 front 2024-03-28T01:10:36.025486+0000 (oldest deadline 2024-03-28T01:11:01.825685+0000) 2024-03-28T01:11:09.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:09 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:09.001+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:09.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:09 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:09.001+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:09.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:09 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:09.001+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:09.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:09 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:09.001+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:09.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:09 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:09.001+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:10.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:09 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:09.992+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:11:10.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:09 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:09.992+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:11:10.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:09 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:09.992+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:11:10.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:09 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:09.992+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:11:10.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:09 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:09.992+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.795010+0000 front 2024-03-28T01:10:31.794967+0000 (oldest deadline 2024-03-28T01:10:54.094722+0000) 2024-03-28T01:11:10.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:09 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:09.992+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:39.896297+0000 front 2024-03-28T01:10:39.896345+0000 (oldest deadline 2024-03-28T01:11:03.996099+0000) 2024-03-28T01:11:10.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:10 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:10.028+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:10.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:10 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:10.028+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:10.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:10 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:10.028+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:10.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:10 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:10.028+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:10.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:10 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:10.028+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:10.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:10 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:10.028+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.025466+0000 front 2024-03-28T01:10:36.025486+0000 (oldest deadline 2024-03-28T01:11:01.825685+0000) 2024-03-28T01:11:10.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:09 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:09.992+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:10.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:09 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:09.992+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:10.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:09 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:09.992+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:10.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:09 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:09.992+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:10.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:09 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:09.992+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:10.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:09 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:09.992+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:44.002455+0000 front 2024-03-28T01:10:44.002562+0000 (oldest deadline 2024-03-28T01:11:09.902384+0000) 2024-03-28T01:11:11.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:11 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:11.036+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:11:11.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:11 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:11.036+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:11:11.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:11 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:11.036+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:11:11.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:11 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:11.036+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:11:11.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:11 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:11.036+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.795010+0000 front 2024-03-28T01:10:31.794967+0000 (oldest deadline 2024-03-28T01:10:54.094722+0000) 2024-03-28T01:11:11.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:11 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:11.036+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:39.896297+0000 front 2024-03-28T01:10:39.896345+0000 (oldest deadline 2024-03-28T01:11:03.996099+0000) 2024-03-28T01:11:11.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:11 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:11.064+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:11.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:11 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:11.064+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:11.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:11 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:11.064+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:11.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:11 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:11.064+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:11.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:11 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:11.064+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:11.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:11 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:11.064+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.025466+0000 front 2024-03-28T01:10:36.025486+0000 (oldest deadline 2024-03-28T01:11:01.825685+0000) 2024-03-28T01:11:11.416 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:11 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:11.041+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:11.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:11 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:11.041+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:11.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:11 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:11.041+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:11.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:11 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:11.041+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:11.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:11 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:11.041+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:11.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:11 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:11.041+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:44.002455+0000 front 2024-03-28T01:10:44.002562+0000 (oldest deadline 2024-03-28T01:11:09.902384+0000) 2024-03-28T01:11:12.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:11 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:11.995+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:11:12.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:11 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:11.995+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:11:12.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:11 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:11.995+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:11:12.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:11 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:11.995+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:11:12.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:11 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:11.995+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.795010+0000 front 2024-03-28T01:10:31.794967+0000 (oldest deadline 2024-03-28T01:10:54.094722+0000) 2024-03-28T01:11:12.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:11 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:11.995+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:39.896297+0000 front 2024-03-28T01:10:39.896345+0000 (oldest deadline 2024-03-28T01:11:03.996099+0000) 2024-03-28T01:11:12.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:12 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:12.034+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:12.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:12 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:12.034+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:12.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:12 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:12.034+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:12.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:12 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:12.034+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:12.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:12 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:12.034+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:12.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:12 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:12.034+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.025466+0000 front 2024-03-28T01:10:36.025486+0000 (oldest deadline 2024-03-28T01:11:01.825685+0000) 2024-03-28T01:11:12.416 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:12 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:12.017+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:12.416 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:12 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:12.017+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:12.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:12 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:12.017+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:12.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:12 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:12.017+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:12.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:12 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:12.017+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:12.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:12 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:12.017+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:44.002455+0000 front 2024-03-28T01:10:44.002562+0000 (oldest deadline 2024-03-28T01:11:09.902384+0000) 2024-03-28T01:11:13.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:13 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:13.040+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.588556+0000 front 2024-03-28T01:09:54.588530+0000 (oldest deadline 2024-03-28T01:10:19.888708+0000) 2024-03-28T01:11:13.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:13 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:13.040+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:05.789809+0000 front 2024-03-28T01:10:05.789965+0000 (oldest deadline 2024-03-28T01:10:31.689682+0000) 2024-03-28T01:11:13.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:13 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:13.040+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:13.891439+0000 front 2024-03-28T01:10:13.891367+0000 (oldest deadline 2024-03-28T01:10:38.591278+0000) 2024-03-28T01:11:13.301 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:13 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:13.040+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:22.692294+0000 front 2024-03-28T01:10:22.692249+0000 (oldest deadline 2024-03-28T01:10:45.592309+0000) 2024-03-28T01:11:13.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:13 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:13.040+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.795010+0000 front 2024-03-28T01:10:31.794967+0000 (oldest deadline 2024-03-28T01:10:54.094722+0000) 2024-03-28T01:11:13.302 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:13 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9[65278]: 2024-03-28T01:11:13.040+0000 7f5d2ccc8700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:39.896297+0000 front 2024-03-28T01:10:39.896345+0000 (oldest deadline 2024-03-28T01:11:03.996099+0000) 2024-03-28T01:11:13.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:13 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:13.005+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:13.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:13 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:13.005+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:13.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:13 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:13.005+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:13.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:13 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:13.005+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:13.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:13 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:13.005+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:13.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:13 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:13.005+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.025466+0000 front 2024-03-28T01:10:36.025486+0000 (oldest deadline 2024-03-28T01:11:01.825685+0000) 2024-03-28T01:11:13.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:12 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:12.977+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:13.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:12 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:12.977+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:13.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:12 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:12.977+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:13.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:12 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:12.977+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:13.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:12 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:12.977+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:13.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:12 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:12.977+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:44.002455+0000 front 2024-03-28T01:10:44.002562+0000 (oldest deadline 2024-03-28T01:11:09.902384+0000) 2024-03-28T01:11:14.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:13 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:13.993+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:14.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:13 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:13.993+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:14.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:13 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:13.993+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:14.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:13 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:13.993+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:14.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:13 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:13.993+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:14.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:13 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:13.993+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.025466+0000 front 2024-03-28T01:10:36.025486+0000 (oldest deadline 2024-03-28T01:11:01.825685+0000) 2024-03-28T01:11:14.359 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:13 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:13.993+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:47.626919+0000 front 2024-03-28T01:10:47.626972+0000 (oldest deadline 2024-03-28T01:11:13.526848+0000) 2024-03-28T01:11:14.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:13 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:13.976+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:14.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:13 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:13.976+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:14.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:13 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:13.976+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:14.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:13 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:13.976+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:14.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:13 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:13.976+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:14.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:13 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:13.976+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:44.002455+0000 front 2024-03-28T01:10:44.002562+0000 (oldest deadline 2024-03-28T01:11:09.902384+0000) 2024-03-28T01:11:14.551 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:14 smithi169 podman[100553]: 2024-03-28 01:11:14.298089486 +0000 UTC m=+5.714391026 container died d68890c7d7f3fb6c8cba61b1b27b1011255add27e4d8c8c2b9a14b000dd0a372 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-28T01:11:14.859 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:14 smithi169 podman[100553]: 2024-03-28 01:11:14.638824385 +0000 UTC m=+6.055125924 container cleanup d68890c7d7f3fb6c8cba61b1b27b1011255add27e4d8c8c2b9a14b000dd0a372 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-28T01:11:14.860 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:14 smithi169 bash[100553]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9 2024-03-28T01:11:15.291 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:14 smithi169 podman[100571]: 2024-03-28 01:11:14.857052541 +0000 UTC m=+0.555061843 container remove d68890c7d7f3fb6c8cba61b1b27b1011255add27e4d8c8c2b9a14b000dd0a372 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-28T01:11:15.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:14 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:14.984+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:15.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:14 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:14.984+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:15.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:14 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:14.984+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:15.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:14 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:14.984+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:15.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:14 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:14.984+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:15.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:14 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:14.984+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.025466+0000 front 2024-03-28T01:10:36.025486+0000 (oldest deadline 2024-03-28T01:11:01.825685+0000) 2024-03-28T01:11:15.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:14 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:14.984+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:47.626919+0000 front 2024-03-28T01:10:47.626972+0000 (oldest deadline 2024-03-28T01:11:13.526848+0000) 2024-03-28T01:11:15.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:14 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:14.948+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:15.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:14 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:14.948+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:15.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:14 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:14.948+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:15.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:14 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:14.948+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:15.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:14 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:14.948+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:15.417 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:14 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:14.948+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:44.002455+0000 front 2024-03-28T01:10:44.002562+0000 (oldest deadline 2024-03-28T01:11:09.902384+0000) 2024-03-28T01:11:15.551 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:15 smithi169 podman[100664]: 2024-03-28 01:11:15.191889503 +0000 UTC m=+0.017804840 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T01:11:15.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:15 smithi169 podman[100664]: 2024-03-28 01:11:15.291043236 +0000 UTC m=+0.116958576 container create 8116dd4c1d440c5c27a1b0318d99ea0a6bdab127a4610e1d11ee4d71f9ce6f4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-28T01:11:15.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:15 smithi169 podman[100664]: 2024-03-28 01:11:15.501860364 +0000 UTC m=+0.327775701 container init 8116dd4c1d440c5c27a1b0318d99ea0a6bdab127a4610e1d11ee4d71f9ce6f4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-28T01:11:15.552 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:15 smithi169 podman[100664]: 2024-03-28 01:11:15.505171452 +0000 UTC m=+0.331086790 container start 8116dd4c1d440c5c27a1b0318d99ea0a6bdab127a4610e1d11ee4d71f9ce6f4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-28T01:11:15.809 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:15 smithi169 podman[100664]: 2024-03-28 01:11:15.585420039 +0000 UTC m=+0.411335394 container attach 8116dd4c1d440c5c27a1b0318d99ea0a6bdab127a4610e1d11ee4d71f9ce6f4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9-deactivate, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-28T01:11:16.083 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:15 smithi169 podman[100664]: 2024-03-28 01:11:15.807464613 +0000 UTC m=+0.633379960 container died 8116dd4c1d440c5c27a1b0318d99ea0a6bdab127a4610e1d11ee4d71f9ce6f4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-28T01:11:16.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:15 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:15.907+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:16.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:15 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:15.907+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:16.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:15 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:15.907+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:16.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:15 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:15.907+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:16.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:15 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:15.907+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:16.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:15 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:15.907+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:44.002455+0000 front 2024-03-28T01:10:44.002562+0000 (oldest deadline 2024-03-28T01:11:09.902384+0000) 2024-03-28T01:11:16.308 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.9.service' 2024-03-28T01:11:16.335 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:16 smithi169 podman[100690]: 2024-03-28 01:11:16.280998308 +0000 UTC m=+0.467953355 container remove 8116dd4c1d440c5c27a1b0318d99ea0a6bdab127a4610e1d11ee4d71f9ce6f4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-9-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-03-28T01:11:16.335 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:16 smithi169 systemd[1]: ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.9.service: Deactivated successfully. 2024-03-28T01:11:16.335 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:16 smithi169 systemd[1]: Stopped Ceph osd.9 for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T01:11:16.335 INFO:journalctl@ceph.osd.9.smithi169.stdout:Mar 28 01:11:16 smithi169 systemd[1]: ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.9.service: Consumed 14min 35.768s CPU time. 2024-03-28T01:11:16.350 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:11:16.351 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-28T01:11:16.351 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-28T01:11:16.351 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.10 2024-03-28T01:11:16.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:16 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:16.023+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:16.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:16 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:16.023+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:16.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:16 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:16.023+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:16.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:16 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:16.023+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:16.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:16 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:16.023+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:16.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:16 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:16.023+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.025466+0000 front 2024-03-28T01:10:36.025486+0000 (oldest deadline 2024-03-28T01:11:01.825685+0000) 2024-03-28T01:11:16.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:16 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:16.023+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:47.626919+0000 front 2024-03-28T01:10:47.626972+0000 (oldest deadline 2024-03-28T01:11:13.526848+0000) 2024-03-28T01:11:16.637 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:16 smithi174 systemd[1]: Stopping Ceph osd.10 for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:11:17.041 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:16 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:16.634+0000 7f128591a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:11:17.041 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:16 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:16.634+0000 7f128591a700 -1 osd.10 98 *** Got signal Terminated *** 2024-03-28T01:11:17.042 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:16 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:16.634+0000 7f128591a700 -1 osd.10 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T01:11:17.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:16 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:16.908+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:17.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:16 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:16.908+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:17.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:16 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:16.908+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:17.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:16 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:16.908+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:17.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:16 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:16.908+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:17.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:16 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:16.908+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:44.002455+0000 front 2024-03-28T01:10:44.002562+0000 (oldest deadline 2024-03-28T01:11:09.902384+0000) 2024-03-28T01:11:17.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:17 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:17.039+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:17.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:17 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:17.039+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:17.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:17 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:17.039+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:17.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:17 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:17.039+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:17.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:17 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:17.039+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:17.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:17 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:17.039+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.025466+0000 front 2024-03-28T01:10:36.025486+0000 (oldest deadline 2024-03-28T01:11:01.825685+0000) 2024-03-28T01:11:17.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:17 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:17.039+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:47.626919+0000 front 2024-03-28T01:10:47.626972+0000 (oldest deadline 2024-03-28T01:11:13.526848+0000) 2024-03-28T01:11:18.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:17 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:17.911+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:18.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:17 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:17.911+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:18.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:17 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:17.911+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:18.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:17 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:17.911+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:18.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:17 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:17.911+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:18.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:17 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:17.911+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:44.002455+0000 front 2024-03-28T01:10:44.002562+0000 (oldest deadline 2024-03-28T01:11:09.902384+0000) 2024-03-28T01:11:18.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:18 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:18.058+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:18.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:18 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:18.058+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:18.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:18 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:18.058+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:18.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:18 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:18.058+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:18.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:18 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:18.058+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:18.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:18 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:18.058+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.025466+0000 front 2024-03-28T01:10:36.025486+0000 (oldest deadline 2024-03-28T01:11:01.825685+0000) 2024-03-28T01:11:18.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:18 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:18.058+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:47.626919+0000 front 2024-03-28T01:10:47.626972+0000 (oldest deadline 2024-03-28T01:11:13.526848+0000) 2024-03-28T01:11:19.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:18 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:18.875+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:19.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:18 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:18.875+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:19.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:18 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:18.875+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:19.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:18 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:18.875+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:19.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:18 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:18.875+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:19.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:18 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:18.875+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:44.002455+0000 front 2024-03-28T01:10:44.002562+0000 (oldest deadline 2024-03-28T01:11:09.902384+0000) 2024-03-28T01:11:19.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:19 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:19.074+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:19.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:19 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:19.074+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:19.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:19 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:19.074+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:19.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:19 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:19.074+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:19.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:19 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:19.074+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:19.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:19 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:19.074+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.025466+0000 front 2024-03-28T01:10:36.025486+0000 (oldest deadline 2024-03-28T01:11:01.825685+0000) 2024-03-28T01:11:19.359 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:19 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:19.074+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:47.626919+0000 front 2024-03-28T01:10:47.626972+0000 (oldest deadline 2024-03-28T01:11:13.526848+0000) 2024-03-28T01:11:20.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:19 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:19.885+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:20.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:19 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:19.885+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:20.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:19 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:19.885+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:20.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:19 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:19.885+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:20.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:19 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:19.885+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:20.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:19 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:19.885+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:44.002455+0000 front 2024-03-28T01:10:44.002562+0000 (oldest deadline 2024-03-28T01:11:09.902384+0000) 2024-03-28T01:11:20.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:20 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:20.032+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:20.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:20 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:20.032+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:20.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:20 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:20.032+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:20.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:20 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:20.032+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:20.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:20 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:20.032+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:20.359 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:20 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:20.032+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.025466+0000 front 2024-03-28T01:10:36.025486+0000 (oldest deadline 2024-03-28T01:11:01.825685+0000) 2024-03-28T01:11:20.359 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:20 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:20.032+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:47.626919+0000 front 2024-03-28T01:10:47.626972+0000 (oldest deadline 2024-03-28T01:11:13.526848+0000) 2024-03-28T01:11:21.166 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:20 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:20.896+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:21.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:20 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:20.896+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:21.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:20 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:20.896+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:21.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:20 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:20.896+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:21.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:20 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:20.896+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:21.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:20 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:20.896+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:44.002455+0000 front 2024-03-28T01:10:44.002562+0000 (oldest deadline 2024-03-28T01:11:09.902384+0000) 2024-03-28T01:11:21.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:20 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:20.996+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:54.519314+0000 front 2024-03-28T01:09:54.519234+0000 (oldest deadline 2024-03-28T01:10:19.819130+0000) 2024-03-28T01:11:21.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:20 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:20.996+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.321247+0000 front 2024-03-28T01:10:07.321345+0000 (oldest deadline 2024-03-28T01:10:32.621396+0000) 2024-03-28T01:11:21.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:20 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:20.996+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:14.922430+0000 front 2024-03-28T01:10:14.922495+0000 (oldest deadline 2024-03-28T01:10:39.022508+0000) 2024-03-28T01:11:21.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:20 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:20.996+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6814 osd.3 since back 2024-03-28T01:10:23.123637+0000 front 2024-03-28T01:10:23.123564+0000 (oldest deadline 2024-03-28T01:10:47.823639+0000) 2024-03-28T01:11:21.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:20 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:20.996+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:31.924717+0000 front 2024-03-28T01:10:31.924758+0000 (oldest deadline 2024-03-28T01:10:56.024894+0000) 2024-03-28T01:11:21.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:20 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:20.996+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.025466+0000 front 2024-03-28T01:10:36.025486+0000 (oldest deadline 2024-03-28T01:11:01.825685+0000) 2024-03-28T01:11:21.359 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:20 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10[60841]: 2024-03-28T01:11:20.996+0000 7f127df06700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:47.626919+0000 front 2024-03-28T01:10:47.626972+0000 (oldest deadline 2024-03-28T01:11:13.526848+0000) 2024-03-28T01:11:22.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:21 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:21.859+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:22.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:21 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:21.859+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:22.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:21 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:21.859+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:22.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:21 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:21.859+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:22.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:21 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:21.859+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:22.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:21 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:21.859+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:44.002455+0000 front 2024-03-28T01:10:44.002562+0000 (oldest deadline 2024-03-28T01:11:09.902384+0000) 2024-03-28T01:11:22.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:21 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:21.859+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6820 osd.7 since back 2024-03-28T01:10:55.104312+0000 front 2024-03-28T01:10:55.104332+0000 (oldest deadline 2024-03-28T01:11:21.004186+0000) 2024-03-28T01:11:22.358 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:22 smithi174 podman[71921]: 2024-03-28 01:11:22.093786187 +0000 UTC m=+5.556737576 container died 84a25479b76fd8e16592c46e1ac743c49e49cac284b0270e21990f46f156cc0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-28T01:11:22.653 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:22 smithi174 podman[71921]: 2024-03-28 01:11:22.441866151 +0000 UTC m=+5.904817528 container cleanup 84a25479b76fd8e16592c46e1ac743c49e49cac284b0270e21990f46f156cc0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T01:11:22.653 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:22 smithi174 bash[71921]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10 2024-03-28T01:11:23.037 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:22 smithi174 podman[71938]: 2024-03-28 01:11:22.651408532 +0000 UTC m=+0.548885098 container remove 84a25479b76fd8e16592c46e1ac743c49e49cac284b0270e21990f46f156cc0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T01:11:23.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:22 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:22.819+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:23.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:22 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:22.819+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:23.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:22 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:22.819+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:23.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:22 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:22.819+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:23.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:22 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:22.819+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:23.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:22 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:22.819+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:44.002455+0000 front 2024-03-28T01:10:44.002562+0000 (oldest deadline 2024-03-28T01:11:09.902384+0000) 2024-03-28T01:11:23.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:22 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:22.819+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6820 osd.7 since back 2024-03-28T01:10:55.104312+0000 front 2024-03-28T01:10:55.104332+0000 (oldest deadline 2024-03-28T01:11:21.004186+0000) 2024-03-28T01:11:23.314 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:23 smithi174 podman[72027]: 2024-03-28 01:11:23.035372476 +0000 UTC m=+0.103559388 container create fb2f9e3c736a75740f70745a5dd6533ab0947f4176f0d27f8b34060128fea8bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T01:11:23.314 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:23 smithi174 podman[72027]: 2024-03-28 01:11:22.949353086 +0000 UTC m=+0.017540013 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T01:11:23.315 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:23 smithi174 podman[72027]: 2024-03-28 01:11:23.246464006 +0000 UTC m=+0.314650931 container init fb2f9e3c736a75740f70745a5dd6533ab0947f4176f0d27f8b34060128fea8bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T01:11:23.315 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:23 smithi174 podman[72027]: 2024-03-28 01:11:23.249685784 +0000 UTC m=+0.317872696 container start fb2f9e3c736a75740f70745a5dd6533ab0947f4176f0d27f8b34060128fea8bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T01:11:23.315 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:23 smithi174 podman[72027]: 2024-03-28 01:11:23.312468825 +0000 UTC m=+0.380655751 container attach fb2f9e3c736a75740f70745a5dd6533ab0947f4176f0d27f8b34060128fea8bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True) 2024-03-28T01:11:23.608 INFO:journalctl@ceph.osd.10.smithi174.stdout:Mar 28 01:11:23 smithi174 podman[72027]: 2024-03-28 01:11:23.539495529 +0000 UTC m=+0.607682475 container died fb2f9e3c736a75740f70745a5dd6533ab0947f4176f0d27f8b34060128fea8bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-10-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-28T01:11:24.096 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.10.service' 2024-03-28T01:11:24.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:23 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:23.793+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:24.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:23 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:23.793+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:24.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:23 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:23.793+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:24.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:23 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:23.793+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:24.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:23 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:23.793+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:24.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:23 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:23.793+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:44.002455+0000 front 2024-03-28T01:10:44.002562+0000 (oldest deadline 2024-03-28T01:11:09.902384+0000) 2024-03-28T01:11:24.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:23 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:23.793+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6820 osd.7 since back 2024-03-28T01:10:55.104312+0000 front 2024-03-28T01:10:55.104332+0000 (oldest deadline 2024-03-28T01:11:21.004186+0000) 2024-03-28T01:11:24.176 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:11:24.176 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-28T01:11:24.176 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-28T01:11:24.176 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.11 2024-03-28T01:11:24.497 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:24 smithi177 systemd[1]: Stopping Ceph osd.11 for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:11:24.776 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:24 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:24.494+0000 7feb9d79c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:11:24.776 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:24 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:24.494+0000 7feb9d79c700 -1 osd.11 98 *** Got signal Terminated *** 2024-03-28T01:11:24.776 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:24 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:24.494+0000 7feb9d79c700 -1 osd.11 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T01:11:24.776 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:24 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:24.772+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:24.776 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:24 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:24.772+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:25.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:24 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:24.772+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:25.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:24 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:24.772+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:25.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:24 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:24.772+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:25.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:24 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:24.772+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:44.002455+0000 front 2024-03-28T01:10:44.002562+0000 (oldest deadline 2024-03-28T01:11:09.902384+0000) 2024-03-28T01:11:25.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:24 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:24.772+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6820 osd.7 since back 2024-03-28T01:10:55.104312+0000 front 2024-03-28T01:10:55.104332+0000 (oldest deadline 2024-03-28T01:11:21.004186+0000) 2024-03-28T01:11:26.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:25 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:25.724+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:26.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:25 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:25.724+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:26.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:25 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:25.724+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:26.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:25 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:25.724+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:26.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:25 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:25.724+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:26.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:25 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:25.724+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:44.002455+0000 front 2024-03-28T01:10:44.002562+0000 (oldest deadline 2024-03-28T01:11:09.902384+0000) 2024-03-28T01:11:26.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:25 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:25.724+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6820 osd.7 since back 2024-03-28T01:10:55.104312+0000 front 2024-03-28T01:10:55.104332+0000 (oldest deadline 2024-03-28T01:11:21.004186+0000) 2024-03-28T01:11:27.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:26 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:26.721+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:27.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:26 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:26.721+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:27.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:26 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:26.721+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:27.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:26 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:26.721+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:27.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:26 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:26.721+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:27.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:26 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:26.721+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:44.002455+0000 front 2024-03-28T01:10:44.002562+0000 (oldest deadline 2024-03-28T01:11:09.902384+0000) 2024-03-28T01:11:27.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:26 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:26.721+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6820 osd.7 since back 2024-03-28T01:10:55.104312+0000 front 2024-03-28T01:10:55.104332+0000 (oldest deadline 2024-03-28T01:11:21.004186+0000) 2024-03-28T01:11:28.120 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:27 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:27.672+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:28.120 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:27 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:27.672+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:28.120 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:27 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:27.672+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:28.120 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:27 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:27.672+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:28.120 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:27 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:27.672+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:28.120 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:27 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:27.672+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:44.002455+0000 front 2024-03-28T01:10:44.002562+0000 (oldest deadline 2024-03-28T01:11:09.902384+0000) 2024-03-28T01:11:28.120 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:27 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:27.672+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6820 osd.7 since back 2024-03-28T01:10:55.104312+0000 front 2024-03-28T01:10:55.104332+0000 (oldest deadline 2024-03-28T01:11:21.004186+0000) 2024-03-28T01:11:28.120 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:27 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:27.672+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6820 osd.8 since back 2024-03-28T01:11:01.004529+0000 front 2024-03-28T01:11:01.004549+0000 (oldest deadline 2024-03-28T01:11:27.405059+0000) 2024-03-28T01:11:29.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:28 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:28.676+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6806 osd.0 since back 2024-03-28T01:09:55.193949+0000 front 2024-03-28T01:09:55.194084+0000 (oldest deadline 2024-03-28T01:10:23.394509+0000) 2024-03-28T01:11:29.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:28 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:28.676+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-03-28T01:10:07.296824+0000 front 2024-03-28T01:10:07.296702+0000 (oldest deadline 2024-03-28T01:10:28.396738+0000) 2024-03-28T01:11:29.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:28 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:28.676+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-03-28T01:10:12.497761+0000 front 2024-03-28T01:10:12.497924+0000 (oldest deadline 2024-03-28T01:10:37.197891+0000) 2024-03-28T01:11:29.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:28 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:28.676+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-03-28T01:10:30.700310+0000 front 2024-03-28T01:10:30.700234+0000 (oldest deadline 2024-03-28T01:10:53.500480+0000) 2024-03-28T01:11:29.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:28 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:28.676+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-03-28T01:10:36.401288+0000 front 2024-03-28T01:10:36.401325+0000 (oldest deadline 2024-03-28T01:11:04.001796+0000) 2024-03-28T01:11:29.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:28 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:28.676+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.169:6822 osd.6 since back 2024-03-28T01:10:44.002455+0000 front 2024-03-28T01:10:44.002562+0000 (oldest deadline 2024-03-28T01:11:09.902384+0000) 2024-03-28T01:11:29.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:28 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:28.676+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.174:6820 osd.7 since back 2024-03-28T01:10:55.104312+0000 front 2024-03-28T01:10:55.104332+0000 (oldest deadline 2024-03-28T01:11:21.004186+0000) 2024-03-28T01:11:29.167 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:28 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11[61972]: 2024-03-28T01:11:28.676+0000 7feb95d88700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.177:6820 osd.8 since back 2024-03-28T01:11:01.004529+0000 front 2024-03-28T01:11:01.004549+0000 (oldest deadline 2024-03-28T01:11:27.405059+0000) 2024-03-28T01:11:30.314 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:30 smithi177 podman[73179]: 2024-03-28 01:11:30.037388644 +0000 UTC m=+5.665551530 container died 03e05207a750a5d7c9e869329c8cdfa48eb5be15d33e01dbdc261b859f472373 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-28T01:11:30.664 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:30 smithi177 podman[73179]: 2024-03-28 01:11:30.385954568 +0000 UTC m=+6.014117452 container cleanup 03e05207a750a5d7c9e869329c8cdfa48eb5be15d33e01dbdc261b859f472373 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-28T01:11:30.664 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:30 smithi177 bash[73179]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11 2024-03-28T01:11:30.917 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:30 smithi177 podman[73194]: 2024-03-28 01:11:30.662396883 +0000 UTC m=+0.622148055 container remove 03e05207a750a5d7c9e869329c8cdfa48eb5be15d33e01dbdc261b859f472373 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-28T01:11:31.259 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:31 smithi177 podman[73285]: 2024-03-28 01:11:30.905813151 +0000 UTC m=+0.017778520 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T01:11:31.259 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:31 smithi177 podman[73285]: 2024-03-28 01:11:31.013431564 +0000 UTC m=+0.125396915 container create d877b124a2715d9a9e73257652bf74fce4c0aa783167143d079fad36dcb2a3a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-28T01:11:31.558 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:31 smithi177 podman[73285]: 2024-03-28 01:11:31.257171106 +0000 UTC m=+0.369136464 container init d877b124a2715d9a9e73257652bf74fce4c0aa783167143d079fad36dcb2a3a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-03-28T01:11:31.558 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:31 smithi177 podman[73285]: 2024-03-28 01:11:31.260219532 +0000 UTC m=+0.372184893 container start d877b124a2715d9a9e73257652bf74fce4c0aa783167143d079fad36dcb2a3a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-28T01:11:31.559 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:31 smithi177 podman[73285]: 2024-03-28 01:11:31.348865579 +0000 UTC m=+0.460830951 container attach d877b124a2715d9a9e73257652bf74fce4c0aa783167143d079fad36dcb2a3a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-28T01:11:31.866 INFO:journalctl@ceph.osd.11.smithi177.stdout:Mar 28 01:11:31 smithi177 podman[73285]: 2024-03-28 01:11:31.556669396 +0000 UTC m=+0.668634762 container died d877b124a2715d9a9e73257652bf74fce4c0aa783167143d079fad36dcb2a3a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-28T01:11:32.158 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@osd.11.service' 2024-03-28T01:11:32.199 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:11:32.200 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-28T01:11:32.200 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-28T01:11:32.200 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.a 2024-03-28T01:11:32.528 INFO:journalctl@ceph.mds.a.smithi169.stdout:Mar 28 01:11:32 smithi169 systemd[1]: Stopping Ceph mds.a for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:11:32.801 INFO:journalctl@ceph.mds.a.smithi169.stdout:Mar 28 01:11:32 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-a[72227]: 2024-03-28T01:11:32.524+0000 7f285302a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:11:32.801 INFO:journalctl@ceph.mds.a.smithi169.stdout:Mar 28 01:11:32 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-a[72227]: 2024-03-28T01:11:32.524+0000 7f285302a700 -1 mds.a *** got signal Terminated *** 2024-03-28T01:11:34.428 INFO:journalctl@ceph.mds.a.smithi169.stdout:Mar 28 01:11:34 smithi169 podman[100784]: 2024-03-28 01:11:34.154782762 +0000 UTC m=+1.758215913 container died d4b5685df0c66548a8fa3e1a36ddd7c036aabb86f460efb60bac56d364db0bf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-a, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T01:11:34.795 INFO:journalctl@ceph.mds.a.smithi169.stdout:Mar 28 01:11:34 smithi169 podman[100784]: 2024-03-28 01:11:34.516219923 +0000 UTC m=+2.119653073 container cleanup d4b5685df0c66548a8fa3e1a36ddd7c036aabb86f460efb60bac56d364db0bf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T01:11:34.795 INFO:journalctl@ceph.mds.a.smithi169.stdout:Mar 28 01:11:34 smithi169 bash[100784]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-a 2024-03-28T01:11:34.908 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.a.service' 2024-03-28T01:11:34.946 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:11:34.946 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-28T01:11:34.946 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-28T01:11:34.946 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.d 2024-03-28T01:11:35.051 INFO:journalctl@ceph.mds.d.smithi169.stdout:Mar 28 01:11:35 smithi169 systemd[1]: Stopping Ceph mds.d for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:11:35.551 INFO:journalctl@ceph.mds.d.smithi169.stdout:Mar 28 01:11:35 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-d[71277]: 2024-03-28T01:11:35.253+0000 7f571481c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:11:35.551 INFO:journalctl@ceph.mds.d.smithi169.stdout:Mar 28 01:11:35 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-d[71277]: 2024-03-28T01:11:35.253+0000 7f571481c700 -1 mds.d *** got signal Terminated *** 2024-03-28T01:11:38.757 INFO:journalctl@ceph.mds.d.smithi169.stdout:Mar 28 01:11:38 smithi169 podman[100907]: 2024-03-28 01:11:38.490336657 +0000 UTC m=+3.328750807 container died 95e1f3625dc388656717ebbadde95089f39f2bd12699dbe880b6788812247355 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-d, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T01:11:39.032 INFO:journalctl@ceph.mds.d.smithi169.stdout:Mar 28 01:11:38 smithi169 podman[100907]: 2024-03-28 01:11:38.828637211 +0000 UTC m=+3.667051361 container cleanup 95e1f3625dc388656717ebbadde95089f39f2bd12699dbe880b6788812247355 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-d, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T01:11:39.033 INFO:journalctl@ceph.mds.d.smithi169.stdout:Mar 28 01:11:38 smithi169 bash[100907]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-d 2024-03-28T01:11:39.251 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.d.service' 2024-03-28T01:11:39.287 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:11:39.287 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-28T01:11:39.287 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-28T01:11:39.288 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.g 2024-03-28T01:11:39.551 INFO:journalctl@ceph.mds.g.smithi169.stdout:Mar 28 01:11:39 smithi169 systemd[1]: Stopping Ceph mds.g for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:11:40.051 INFO:journalctl@ceph.mds.g.smithi169.stdout:Mar 28 01:11:39 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-g[71882]: 2024-03-28T01:11:39.607+0000 7f7657af8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:11:40.051 INFO:journalctl@ceph.mds.g.smithi169.stdout:Mar 28 01:11:39 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-g[71882]: 2024-03-28T01:11:39.607+0000 7f7657af8700 -1 mds.g *** got signal Terminated *** 2024-03-28T01:11:43.669 INFO:journalctl@ceph.mds.g.smithi169.stdout:Mar 28 01:11:43 smithi169 podman[101028]: 2024-03-28 01:11:43.410161482 +0000 UTC m=+3.907163856 container died 73be7983f9dae61f75fbb29a7c5b8ac728b372731674f786a4e7618d202dbe60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-g, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T01:11:43.928 INFO:journalctl@ceph.mds.g.smithi169.stdout:Mar 28 01:11:43 smithi169 podman[101028]: 2024-03-28 01:11:43.724581898 +0000 UTC m=+4.221584271 container cleanup 73be7983f9dae61f75fbb29a7c5b8ac728b372731674f786a4e7618d202dbe60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-g, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-28T01:11:43.928 INFO:journalctl@ceph.mds.g.smithi169.stdout:Mar 28 01:11:43 smithi169 bash[101028]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-g 2024-03-28T01:11:44.141 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.g.service' 2024-03-28T01:11:44.175 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:11:44.175 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-28T01:11:44.175 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-28T01:11:44.176 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.j 2024-03-28T01:11:44.505 INFO:journalctl@ceph.mds.j.smithi169.stdout:Mar 28 01:11:44 smithi169 systemd[1]: Stopping Ceph mds.j for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:11:44.801 INFO:journalctl@ceph.mds.j.smithi169.stdout:Mar 28 01:11:44 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-j[70891]: 2024-03-28T01:11:44.502+0000 7f2225f18700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:11:44.801 INFO:journalctl@ceph.mds.j.smithi169.stdout:Mar 28 01:11:44 smithi169 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-j[70891]: 2024-03-28T01:11:44.502+0000 7f2225f18700 -1 mds.j *** got signal Terminated *** 2024-03-28T01:11:47.860 INFO:journalctl@ceph.mds.j.smithi169.stdout:Mar 28 01:11:47 smithi169 podman[101150]: 2024-03-28 01:11:47.578995933 +0000 UTC m=+3.188342278 container died 0433ace5a545d5309d9c1312a4874749bec3eb9de09e3871d08babf0f7a3b702 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-28T01:11:48.198 INFO:journalctl@ceph.mds.j.smithi169.stdout:Mar 28 01:11:47 smithi169 podman[101150]: 2024-03-28 01:11:47.953241399 +0000 UTC m=+3.562587744 container cleanup 0433ace5a545d5309d9c1312a4874749bec3eb9de09e3871d08babf0f7a3b702 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-j, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-28T01:11:48.198 INFO:journalctl@ceph.mds.j.smithi169.stdout:Mar 28 01:11:47 smithi169 bash[101150]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-j 2024-03-28T01:11:48.328 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.j.service' 2024-03-28T01:11:48.361 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:11:48.362 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-28T01:11:48.362 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-28T01:11:48.362 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.b 2024-03-28T01:11:48.653 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 01:11:48 smithi174 systemd[1]: Stopping Ceph mds.b for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:11:49.108 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 01:11:48 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-b[66103]: 2024-03-28T01:11:48.649+0000 7fab308ee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:11:49.108 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 01:11:48 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-b[66103]: 2024-03-28T01:11:48.649+0000 7fab308ee700 -1 mds.b *** got signal Terminated *** 2024-03-28T01:11:52.312 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 01:11:52 smithi174 podman[72131]: 2024-03-28 01:11:52.053167452 +0000 UTC m=+3.501936227 container died 08f12830df86263c166a955fd1af29076d22527d64c31b952e2569d916a500f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T01:11:52.608 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 01:11:52 smithi174 podman[72131]: 2024-03-28 01:11:52.409206209 +0000 UTC m=+3.857974977 container cleanup 08f12830df86263c166a955fd1af29076d22527d64c31b952e2569d916a500f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-28T01:11:52.608 INFO:journalctl@ceph.mds.b.smithi174.stdout:Mar 28 01:11:52 smithi174 bash[72131]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-b 2024-03-28T01:11:52.888 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.b.service' 2024-03-28T01:11:52.925 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:11:52.925 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-28T01:11:52.925 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-28T01:11:52.925 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.e 2024-03-28T01:11:53.249 INFO:journalctl@ceph.mds.e.smithi174.stdout:Mar 28 01:11:53 smithi174 systemd[1]: Stopping Ceph mds.e for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:11:53.608 INFO:journalctl@ceph.mds.e.smithi174.stdout:Mar 28 01:11:53 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-e[65479]: 2024-03-28T01:11:53.245+0000 7f06e69a4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:11:53.608 INFO:journalctl@ceph.mds.e.smithi174.stdout:Mar 28 01:11:53 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-e[65479]: 2024-03-28T01:11:53.245+0000 7f06e69a4700 -1 mds.e *** got signal Terminated *** 2024-03-28T01:11:56.323 INFO:journalctl@ceph.mds.e.smithi174.stdout:Mar 28 01:11:56 smithi174 podman[72255]: 2024-03-28 01:11:56.048993741 +0000 UTC m=+2.908531190 container died 66562be0736e3bc3339f230efa51f144bb48a4a6b47c37a9626f12b1370c8306 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T01:11:56.608 INFO:journalctl@ceph.mds.e.smithi174.stdout:Mar 28 01:11:56 smithi174 podman[72255]: 2024-03-28 01:11:56.457379931 +0000 UTC m=+3.316917362 container cleanup 66562be0736e3bc3339f230efa51f144bb48a4a6b47c37a9626f12b1370c8306 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-28T01:11:56.608 INFO:journalctl@ceph.mds.e.smithi174.stdout:Mar 28 01:11:56 smithi174 bash[72255]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-e 2024-03-28T01:11:56.802 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.e.service' 2024-03-28T01:11:56.839 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:11:56.839 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-28T01:11:56.840 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-28T01:11:56.840 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.h 2024-03-28T01:11:57.108 INFO:journalctl@ceph.mds.h.smithi174.stdout:Mar 28 01:11:56 smithi174 systemd[1]: Stopping Ceph mds.h for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:11:57.608 INFO:journalctl@ceph.mds.h.smithi174.stdout:Mar 28 01:11:57 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-h[65800]: 2024-03-28T01:11:57.189+0000 7fe5cf1f9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:11:57.608 INFO:journalctl@ceph.mds.h.smithi174.stdout:Mar 28 01:11:57 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-h[65800]: 2024-03-28T01:11:57.189+0000 7fe5cf1f9700 -1 mds.h *** got signal Terminated *** 2024-03-28T01:12:00.319 INFO:journalctl@ceph.mds.h.smithi174.stdout:Mar 28 01:12:00 smithi174 podman[72387]: 2024-03-28 01:12:00.043222759 +0000 UTC m=+2.977842625 container died 7c47ea8c36a858aaaf82bcd5671484eed5a96cf29bafa8c1d53a0b830a36416c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-h, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-28T01:12:00.594 INFO:journalctl@ceph.mds.h.smithi174.stdout:Mar 28 01:12:00 smithi174 podman[72387]: 2024-03-28 01:12:00.390636064 +0000 UTC m=+3.325255945 container cleanup 7c47ea8c36a858aaaf82bcd5671484eed5a96cf29bafa8c1d53a0b830a36416c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-h, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-28T01:12:00.594 INFO:journalctl@ceph.mds.h.smithi174.stdout:Mar 28 01:12:00 smithi174 bash[72387]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-h 2024-03-28T01:12:00.813 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.h.service' 2024-03-28T01:12:00.849 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:12:00.849 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-28T01:12:00.849 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-28T01:12:00.849 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.k 2024-03-28T01:12:01.108 INFO:journalctl@ceph.mds.k.smithi174.stdout:Mar 28 01:12:00 smithi174 systemd[1]: Stopping Ceph mds.k for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:12:01.608 INFO:journalctl@ceph.mds.k.smithi174.stdout:Mar 28 01:12:01 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-k[65158]: 2024-03-28T01:12:01.185+0000 7f03eec99700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:12:01.608 INFO:journalctl@ceph.mds.k.smithi174.stdout:Mar 28 01:12:01 smithi174 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-k[65158]: 2024-03-28T01:12:01.185+0000 7f03eec99700 -1 mds.k *** got signal Terminated *** 2024-03-28T01:12:04.956 INFO:journalctl@ceph.mds.k.smithi174.stdout:Mar 28 01:12:04 smithi174 podman[72508]: 2024-03-28 01:12:04.678344515 +0000 UTC m=+3.611734964 container died d59d450f31ef1f29786e7a8df3e3c4805e7de6fa75124dc0c43b42be64a08ac3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-k, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319) 2024-03-28T01:12:05.298 INFO:journalctl@ceph.mds.k.smithi174.stdout:Mar 28 01:12:05 smithi174 podman[72508]: 2024-03-28 01:12:05.036165214 +0000 UTC m=+3.969555649 container cleanup d59d450f31ef1f29786e7a8df3e3c4805e7de6fa75124dc0c43b42be64a08ac3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-k, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-28T01:12:05.298 INFO:journalctl@ceph.mds.k.smithi174.stdout:Mar 28 01:12:05 smithi174 bash[72508]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-k 2024-03-28T01:12:05.419 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.k.service' 2024-03-28T01:12:05.454 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:12:05.454 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-28T01:12:05.455 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-28T01:12:05.455 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.c 2024-03-28T01:12:05.736 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 01:12:05 smithi177 systemd[1]: Stopping Ceph mds.c for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:12:05.991 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 01:12:05 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-c[67357]: 2024-03-28T01:12:05.732+0000 7f79cb383700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:12:05.991 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 01:12:05 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-c[67357]: 2024-03-28T01:12:05.732+0000 7f79cb383700 -1 mds.c *** got signal Terminated *** 2024-03-28T01:12:08.394 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 01:12:08 smithi177 podman[73390]: 2024-03-28 01:12:08.04575885 +0000 UTC m=+2.400419284 container died 729e7063b6ebb665fda672bf336029c586c6da36c5d5dc10aa6d72214811c61c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-c, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-28T01:12:08.667 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 01:12:08 smithi177 podman[73390]: 2024-03-28 01:12:08.474724235 +0000 UTC m=+2.829384669 container cleanup 729e7063b6ebb665fda672bf336029c586c6da36c5d5dc10aa6d72214811c61c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-c, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-28T01:12:08.667 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 01:12:08 smithi177 bash[73390]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-c 2024-03-28T01:12:08.957 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.c.service' 2024-03-28T01:12:08.982 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 01:12:08 smithi177 podman[73416]: 2024-03-28 01:12:08.709140735 +0000 UTC m=+0.660515746 container remove 729e7063b6ebb665fda672bf336029c586c6da36c5d5dc10aa6d72214811c61c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-c, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T01:12:08.982 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 01:12:08 smithi177 systemd[1]: ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.c.service: Deactivated successfully. 2024-03-28T01:12:08.982 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 01:12:08 smithi177 systemd[1]: Stopped Ceph mds.c for 833368da-ec9c-11ee-95d0-87774f69a715. 2024-03-28T01:12:08.982 INFO:journalctl@ceph.mds.c.smithi177.stdout:Mar 28 01:12:08 smithi177 systemd[1]: ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.c.service: Consumed 4min 39.344s CPU time. 2024-03-28T01:12:08.996 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:12:08.996 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-28T01:12:08.996 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-28T01:12:08.996 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.f 2024-03-28T01:12:09.323 INFO:journalctl@ceph.mds.f.smithi177.stdout:Mar 28 01:12:09 smithi177 systemd[1]: Stopping Ceph mds.f for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:12:09.666 INFO:journalctl@ceph.mds.f.smithi177.stdout:Mar 28 01:12:09 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-f[67040]: 2024-03-28T01:12:09.319+0000 7f0860baa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:12:09.667 INFO:journalctl@ceph.mds.f.smithi177.stdout:Mar 28 01:12:09 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-f[67040]: 2024-03-28T01:12:09.319+0000 7f0860baa700 -1 mds.f *** got signal Terminated *** 2024-03-28T01:12:12.331 INFO:journalctl@ceph.mds.f.smithi177.stdout:Mar 28 01:12:12 smithi177 podman[73524]: 2024-03-28 01:12:12.051225699 +0000 UTC m=+2.842970597 container died 8f7ef40b1835aeed41908232748f893a9e40bbeeb2fbc400ab19c1a4104a4fbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-f, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T01:12:12.667 INFO:journalctl@ceph.mds.f.smithi177.stdout:Mar 28 01:12:12 smithi177 podman[73524]: 2024-03-28 01:12:12.436407006 +0000 UTC m=+3.228151862 container cleanup 8f7ef40b1835aeed41908232748f893a9e40bbeeb2fbc400ab19c1a4104a4fbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-f, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-28T01:12:12.667 INFO:journalctl@ceph.mds.f.smithi177.stdout:Mar 28 01:12:12 smithi177 bash[73524]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-f 2024-03-28T01:12:12.861 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.f.service' 2024-03-28T01:12:12.897 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:12:12.897 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-28T01:12:12.898 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-28T01:12:12.898 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.i 2024-03-28T01:12:13.226 INFO:journalctl@ceph.mds.i.smithi177.stdout:Mar 28 01:12:12 smithi177 systemd[1]: Stopping Ceph mds.i for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:12:13.666 INFO:journalctl@ceph.mds.i.smithi177.stdout:Mar 28 01:12:13 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-i[67676]: 2024-03-28T01:12:13.222+0000 7ff2159ce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:12:13.667 INFO:journalctl@ceph.mds.i.smithi177.stdout:Mar 28 01:12:13 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-i[67676]: 2024-03-28T01:12:13.222+0000 7ff2159ce700 -1 mds.i *** got signal Terminated *** 2024-03-28T01:12:16.358 INFO:journalctl@ceph.mds.i.smithi177.stdout:Mar 28 01:12:16 smithi177 podman[73644]: 2024-03-28 01:12:16.050413302 +0000 UTC m=+2.939054733 container died 6863e73af0f029d5fb6b74fe8f227e93374d09a5a8cc8428558426824e5d4f49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-i, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T01:12:16.359 INFO:journalctl@ceph.mds.i.smithi177.stdout:Mar 28 01:12:16 smithi177 podman[73644]: 2024-03-28 01:12:16.356410698 +0000 UTC m=+3.245052103 container cleanup 6863e73af0f029d5fb6b74fe8f227e93374d09a5a8cc8428558426824e5d4f49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-i, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T01:12:16.666 INFO:journalctl@ceph.mds.i.smithi177.stdout:Mar 28 01:12:16 smithi177 bash[73644]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-i 2024-03-28T01:12:16.667 INFO:journalctl@ceph.mds.i.smithi177.stdout:Mar 28 01:12:16 smithi177 podman[73658]: 2024-03-28 01:12:16.582396546 +0000 UTC m=+0.529518250 container remove 6863e73af0f029d5fb6b74fe8f227e93374d09a5a8cc8428558426824e5d4f49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-i, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True) 2024-03-28T01:12:16.709 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.i.service' 2024-03-28T01:12:16.744 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:12:16.744 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-28T01:12:16.744 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-28T01:12:16.744 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.l 2024-03-28T01:12:16.993 INFO:journalctl@ceph.mds.l.smithi177.stdout:Mar 28 01:12:16 smithi177 systemd[1]: Stopping Ceph mds.l for 833368da-ec9c-11ee-95d0-87774f69a715... 2024-03-28T01:12:17.417 INFO:journalctl@ceph.mds.l.smithi177.stdout:Mar 28 01:12:17 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-l[66732]: 2024-03-28T01:12:17.092+0000 7fd9e8c91700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T01:12:17.417 INFO:journalctl@ceph.mds.l.smithi177.stdout:Mar 28 01:12:17 smithi177 ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-l[66732]: 2024-03-28T01:12:17.092+0000 7fd9e8c91700 -1 mds.l *** got signal Terminated *** 2024-03-28T01:12:21.417 INFO:journalctl@ceph.mds.l.smithi177.stdout:Mar 28 01:12:21 smithi177 podman[73764]: 2024-03-28 01:12:21.134253529 +0000 UTC m=+4.177150954 container died eda74141a6aaefbf1db0184b68eaffc66f71f05660683d9241d950aaa61b8de3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-28T01:12:21.787 INFO:journalctl@ceph.mds.l.smithi177.stdout:Mar 28 01:12:21 smithi177 podman[73764]: 2024-03-28 01:12:21.5519763 +0000 UTC m=+4.594873725 container cleanup eda74141a6aaefbf1db0184b68eaffc66f71f05660683d9241d950aaa61b8de3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-l, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T01:12:21.787 INFO:journalctl@ceph.mds.l.smithi177.stdout:Mar 28 01:12:21 smithi177 bash[73764]: ceph-833368da-ec9c-11ee-95d0-87774f69a715-mds-l 2024-03-28T01:12:21.893 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-833368da-ec9c-11ee-95d0-87774f69a715@mds.l.service' 2024-03-28T01:12:21.928 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T01:12:21.928 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-28T01:12:21.928 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 833368da-ec9c-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T01:12:22.120 INFO:teuthology.orchestra.run.smithi169.stdout:Deleting cluster with fsid: 833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T01:12:25.306 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 833368da-ec9c-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T01:12:25.497 INFO:teuthology.orchestra.run.smithi174.stdout:Deleting cluster with fsid: 833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T01:12:28.103 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 833368da-ec9c-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T01:12:28.294 INFO:teuthology.orchestra.run.smithi177.stdout:Deleting cluster with fsid: 833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T01:12:30.987 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T01:12:31.023 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T01:12:31.057 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T01:12:31.086 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-28T01:12:31.087 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625243/remote/smithi169/crash 2024-03-28T01:12:31.088 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/crash -- . 2024-03-28T01:12:31.134 INFO:teuthology.orchestra.run.smithi169.stderr:tar: /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T01:12:31.135 INFO:teuthology.orchestra.run.smithi169.stderr:tar: Error is not recoverable: exiting now 2024-03-28T01:12:31.137 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625243/remote/smithi174/crash 2024-03-28T01:12:31.138 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/crash -- . 2024-03-28T01:12:31.187 INFO:teuthology.orchestra.run.smithi174.stderr:tar: /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T01:12:31.188 INFO:teuthology.orchestra.run.smithi174.stderr:tar: Error is not recoverable: exiting now 2024-03-28T01:12:31.190 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625243/remote/smithi177/crash 2024-03-28T01:12:31.190 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/crash -- . 2024-03-28T01:12:31.233 INFO:teuthology.orchestra.run.smithi177.stderr:tar: /var/lib/ceph/833368da-ec9c-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T01:12:31.233 INFO:teuthology.orchestra.run.smithi177.stderr:tar: Error is not recoverable: exiting now 2024-03-28T01:12:31.234 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-28T01:12:31.235 DEBUG:teuthology.orchestra.run.smithi169:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-28T01:12:31.342 INFO:teuthology.orchestra.run.smithi169.stdout:2024-03-28T00:50:00.271375+0000 mon.a (mon.0) 414 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-03-28T01:12:31.342 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-28T01:12:31.343 DEBUG:teuthology.orchestra.run.smithi169:> sudo egrep '\[SEC\]' /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-28T01:12:31.377 DEBUG:teuthology.orchestra.run.smithi169:> sudo egrep '\[ERR\]' /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-28T01:12:31.451 DEBUG:teuthology.orchestra.run.smithi169:> sudo egrep '\[WRN\]' /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-28T01:12:31.525 INFO:teuthology.orchestra.run.smithi169.stdout:2024-03-28T00:50:00.271375+0000 mon.a (mon.0) 414 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-03-28T01:12:31.525 INFO:tasks.cephadm:Compressing logs... 2024-03-28T01:12:31.525 DEBUG:teuthology.orchestra.run.smithi169:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T01:12:31.568 DEBUG:teuthology.orchestra.run.smithi174:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T01:12:31.571 DEBUG:teuthology.orchestra.run.smithi177:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T01:12:31.622 INFO:teuthology.orchestra.run.smithi174.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-28T01:12:31.629 INFO:teuthology.orchestra.run.smithi169.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-28T01:12:31.629 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T01:12:31.630 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T01:12:31.631 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.10.log 2024-03-28T01:12:31.631 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T01:12:31.632 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.b.log 2024-03-28T01:12:31.632 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-28T01:12:31.632 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-28T01:12:31.639 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.k.log 2024-03-28T01:12:31.639 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.h.log 2024-03-28T01:12:31.639 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-28T01:12:31.639 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.e.log 2024-03-28T01:12:31.639 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-28T01:12:31.639 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T01:12:31.639 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.log 2024-03-28T01:12:31.639 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T01:12:31.640 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-28T01:12:31.640 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mon.b.log: 94.5% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.k.log.gz 2024-03-28T01:12:31.640 INFO:teuthology.orchestra.run.smithi174.stderr: 92.8% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-28T01:12:31.641 INFO:teuthology.orchestra.run.smithi177.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-28T01:12:31.641 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T01:12:31.641 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.l.log 2024-03-28T01:12:31.641 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T01:12:31.641 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.i.log 2024-03-28T01:12:31.641 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.l.log: /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.11.log 2024-03-28T01:12:31.641 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mgr.z.log 2024-03-28T01:12:31.641 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.8.log 2024-03-28T01:12:31.641 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-28T01:12:31.642 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.f.log 2024-03-28T01:12:31.642 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T01:12:31.642 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.log 2024-03-28T01:12:31.642 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-28T01:12:31.642 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.c.log 2024-03-28T01:12:31.642 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T01:12:31.642 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-28T01:12:31.642 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mon.c.log: 94.7% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.l.log.gz 2024-03-28T01:12:31.642 INFO:teuthology.orchestra.run.smithi177.stderr: 92.9% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mgr.z.log.gz 2024-03-28T01:12:31.647 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-28T01:12:31.647 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-28T01:12:31.647 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T01:12:31.647 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.g.log 2024-03-28T01:12:31.647 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.j.log 2024-03-28T01:12:31.647 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-28T01:12:31.647 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-28T01:12:31.648 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-28T01:12:31.648 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T01:12:31.648 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.log 2024-03-28T01:12:31.648 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T01:12:31.648 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.9.log 2024-03-28T01:12:31.648 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.d.log 2024-03-28T01:12:31.648 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.a.log 2024-03-28T01:12:31.648 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-28T01:12:31.648 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.log: 94.6% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.g.log.gz 2024-03-28T01:12:31.648 INFO:teuthology.orchestra.run.smithi169.stderr: 86.7% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T01:12:31.649 INFO:teuthology.orchestra.run.smithi174.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T01:12:31.649 INFO:teuthology.orchestra.run.smithi177.stderr: 92.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T01:12:31.737 INFO:teuthology.orchestra.run.smithi174.stderr: 82.6% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T01:12:31.761 INFO:teuthology.orchestra.run.smithi169.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-28T01:12:31.776 INFO:teuthology.orchestra.run.smithi174.stderr: 94.5% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.e.log.gz 2024-03-28T01:12:31.782 INFO:teuthology.orchestra.run.smithi177.stderr: 90.9% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T01:12:31.795 INFO:teuthology.orchestra.run.smithi177.stderr: 94.6% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.f.log.gz 2024-03-28T01:12:31.809 INFO:teuthology.orchestra.run.smithi169.stderr: 94.5% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.j.log.gz 2024-03-28T01:12:31.820 INFO:teuthology.orchestra.run.smithi177.stderr: 93.0% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.i.log.gz 2024-03-28T01:12:31.821 INFO:teuthology.orchestra.run.smithi177.stderr: 93.1% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.c.log.gz 2024-03-28T01:12:31.826 INFO:teuthology.orchestra.run.smithi177.stderr: 82.6% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T01:12:31.828 INFO:teuthology.orchestra.run.smithi177.stderr: 86.7% 90.4% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T01:12:31.828 INFO:teuthology.orchestra.run.smithi177.stderr: -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T01:12:31.830 INFO:teuthology.orchestra.run.smithi169.stderr: 93.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T01:12:31.851 INFO:teuthology.orchestra.run.smithi169.stderr: 83.0% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T01:12:31.886 INFO:teuthology.orchestra.run.smithi169.stderr: 90.2% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T01:12:31.890 INFO:teuthology.orchestra.run.smithi169.stderr: 94.6% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.a.log.gz 2024-03-28T01:12:31.904 INFO:teuthology.orchestra.run.smithi169.stderr: 91.0% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T01:12:31.954 INFO:teuthology.orchestra.run.smithi174.stderr: 91.0% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T01:12:32.065 INFO:teuthology.orchestra.run.smithi174.stderr: 86.7% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T01:12:32.094 INFO:teuthology.orchestra.run.smithi174.stderr: 90.3% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T01:12:32.146 INFO:teuthology.orchestra.run.smithi169.stderr: 94.5% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.d.log.gz 2024-03-28T01:12:33.328 INFO:teuthology.orchestra.run.smithi169.stderr: 90.0% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-28T01:12:33.340 INFO:teuthology.orchestra.run.smithi177.stderr: 91.7% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-28T01:12:35.468 INFO:teuthology.orchestra.run.smithi174.stderr: 91.7% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-28T01:12:55.191 INFO:teuthology.orchestra.run.smithi169.stderr: 90.3% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-28T01:19:14.895 INFO:teuthology.orchestra.run.smithi174.stderr: 97.4% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.h.log.gz 2024-03-28T01:24:07.960 INFO:teuthology.orchestra.run.smithi177.stderr: 92.7% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.8.log.gz 2024-03-28T01:24:13.100 INFO:teuthology.orchestra.run.smithi177.stderr: 92.6% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.11.log.gz 2024-03-28T01:27:40.629 INFO:teuthology.orchestra.run.smithi174.stderr: 92.8% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.10.log.gz 2024-03-28T01:28:05.072 INFO:teuthology.orchestra.run.smithi177.stderr: 92.6% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-28T01:29:17.611 INFO:teuthology.orchestra.run.smithi177.stderr: 92.4% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-28T01:29:17.612 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-28T01:29:17.612 INFO:teuthology.orchestra.run.smithi177.stderr:real 16m46.031s 2024-03-28T01:29:17.613 INFO:teuthology.orchestra.run.smithi177.stderr:user 7m26.621s 2024-03-28T01:29:17.613 INFO:teuthology.orchestra.run.smithi177.stderr:sys 0m20.676s 2024-03-28T01:34:13.559 INFO:teuthology.orchestra.run.smithi169.stderr: 92.6% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-28T01:34:57.534 INFO:teuthology.orchestra.run.smithi174.stderr: 92.6% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-28T01:36:02.575 INFO:teuthology.orchestra.run.smithi169.stderr: 92.6% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-28T01:37:19.410 INFO:teuthology.orchestra.run.smithi174.stderr: 92.6% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-28T01:38:27.200 INFO:teuthology.orchestra.run.smithi169.stderr: 92.5% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-28T01:39:04.120 INFO:teuthology.orchestra.run.smithi169.stderr: 92.4% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.9.log.gz 2024-03-28T01:39:04.146 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-28T01:39:04.146 INFO:teuthology.orchestra.run.smithi169.stderr:real 26m32.569s 2024-03-28T01:39:04.146 INFO:teuthology.orchestra.run.smithi169.stderr:user 8m42.647s 2024-03-28T01:39:04.146 INFO:teuthology.orchestra.run.smithi169.stderr:sys 0m25.902s 2024-03-28T01:40:04.172 INFO:teuthology.orchestra.run.smithi174.stderr: 92.6% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-28T01:40:41.462 INFO:teuthology.orchestra.run.smithi174.stderr: 94.2% -- replaced with /var/log/ceph/833368da-ec9c-11ee-95d0-87774f69a715/ceph-mds.b.log.gz 2024-03-28T01:40:41.471 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-28T01:40:41.471 INFO:teuthology.orchestra.run.smithi174.stderr:real 28m9.889s 2024-03-28T01:40:41.471 INFO:teuthology.orchestra.run.smithi174.stderr:user 9m38.660s 2024-03-28T01:40:41.471 INFO:teuthology.orchestra.run.smithi174.stderr:sys 0m28.317s 2024-03-28T01:40:41.471 INFO:tasks.cephadm:Archiving logs... 2024-03-28T01:40:41.472 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625243/remote/smithi169/log 2024-03-28T01:40:41.473 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T01:42:09.128 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625243/remote/smithi174/log 2024-03-28T01:42:09.129 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T01:43:42.775 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625243/remote/smithi177/log 2024-03-28T01:43:42.776 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T01:45:00.095 INFO:tasks.cephadm:Removing cluster... 2024-03-28T01:45:00.096 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 833368da-ec9c-11ee-95d0-87774f69a715 --force 2024-03-28T01:45:00.740 INFO:teuthology.orchestra.run.smithi169.stdout:Deleting cluster with fsid: 833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T01:45:01.602 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 833368da-ec9c-11ee-95d0-87774f69a715 --force 2024-03-28T01:45:01.995 INFO:teuthology.orchestra.run.smithi174.stdout:Deleting cluster with fsid: 833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T01:45:02.822 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 833368da-ec9c-11ee-95d0-87774f69a715 --force 2024-03-28T01:45:03.098 INFO:teuthology.orchestra.run.smithi177.stdout:Deleting cluster with fsid: 833368da-ec9c-11ee-95d0-87774f69a715 2024-03-28T01:45:03.835 INFO:tasks.cephadm:Removing cephadm ... 2024-03-28T01:45:03.835 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T01:45:03.883 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T01:45:03.901 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T01:45:03.916 INFO:tasks.cephadm:Teardown complete 2024-03-28T01:45:03.917 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-28T01:45:03.929 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-28T01:45:03.929 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T01:45:03.932 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T01:45:03.944 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T01:45:04.067 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-28T01:45:04.077 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-28T01:45:04.077 DEBUG:teuthology.orchestra.run.smithi169:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T01:45:04.080 DEBUG:teuthology.orchestra.run.smithi174:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T01:45:04.083 DEBUG:teuthology.orchestra.run.smithi177:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T01:45:04.096 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-03-28T01:45:04.097 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-03-28T01:45:04.114 INFO:teuthology.orchestra.run.smithi169.stderr:bash: line 1: ntpq: command not found 2024-03-28T01:45:04.161 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T01:45:04.161 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-03-28T01:45:04.161 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 62 +1623us[+1665us] +/- 73ms 2024-03-28T01:45:04.161 INFO:teuthology.orchestra.run.smithi169.stdout:^* hv02.front.sepia.ceph.com 3 6 377 62 -1411us[-1369us] +/- 50ms 2024-03-28T01:45:04.161 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 +617us[ +617us] +/- 80ms 2024-03-28T01:45:04.161 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T01:45:04.219 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T01:45:04.219 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-03-28T01:45:04.219 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 57 +1599us[+1599us] +/- 73ms 2024-03-28T01:45:04.219 INFO:teuthology.orchestra.run.smithi174.stdout:^* hv02.front.sepia.ceph.com 3 6 377 61 -1429us[-1285us] +/- 50ms 2024-03-28T01:45:04.219 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 59 +591us[ +591us] +/- 80ms 2024-03-28T01:45:04.220 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T01:45:04.232 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T01:45:04.232 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-03-28T01:45:04.232 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 61 +1584us[+1605us] +/- 73ms 2024-03-28T01:45:04.232 INFO:teuthology.orchestra.run.smithi177.stdout:^* hv02.front.sepia.ceph.com 3 6 377 54 -1443us[-1423us] +/- 50ms 2024-03-28T01:45:04.233 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 +561us[ +582us] +/- 80ms 2024-03-28T01:45:04.233 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T01:45:04.234 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-28T01:45:04.244 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-28T01:45:04.245 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-28T01:45:04.259 DEBUG:teuthology.orchestra.run.smithi169:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T01:45:04.427 DEBUG:teuthology.orchestra.run.smithi174:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T01:45:04.559 DEBUG:teuthology.orchestra.run.smithi177:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T01:45:04.680 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T01:45:04.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T01:45:04.715 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T01:45:04.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T01:45:04.748 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T01:45:04.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T01:45:04.778 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-28T01:45:04.789 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-28T01:45:04.804 INFO:teuthology.task.internal:Duration was 3995.322331 seconds 2024-03-28T01:45:04.805 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-28T01:45:04.820 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-28T01:45:04.820 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T01:45:04.823 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T01:45:04.826 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T01:45:04.881 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T01:45:04.886 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T01:45:04.912 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T01:45:05.216 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-28T01:45:05.216 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi169.front.sepia.ceph.com 2024-03-28T01:45:05.217 DEBUG:teuthology.orchestra.run.smithi169:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T01:45:05.254 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi174.front.sepia.ceph.com 2024-03-28T01:45:05.254 DEBUG:teuthology.orchestra.run.smithi174:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T01:45:05.288 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi177.front.sepia.ceph.com 2024-03-28T01:45:05.288 DEBUG:teuthology.orchestra.run.smithi177:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T01:45:05.320 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-28T01:45:05.320 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T01:45:05.323 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T01:45:05.331 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T01:45:05.399 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-28T01:45:05.399 DEBUG:teuthology.orchestra.run.smithi169:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T01:45:05.401 DEBUG:teuthology.orchestra.run.smithi174:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T01:45:05.404 DEBUG:teuthology.orchestra.run.smithi177:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T01:45:05.780 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-28T01:45:05.792 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-28T01:45:05.792 DEBUG:teuthology.orchestra.run.smithi169:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T01:45:05.823 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T01:45:05.863 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T01:45:05.902 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-28T01:45:05.915 DEBUG:teuthology.orchestra.run.smithi169:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T01:45:05.918 DEBUG:teuthology.orchestra.run.smithi174:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T01:45:05.920 DEBUG:teuthology.orchestra.run.smithi177:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T01:45:05.953 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = core 2024-03-28T01:45:05.958 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = core 2024-03-28T01:45:05.983 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = core 2024-03-28T01:45:06.014 DEBUG:teuthology.orchestra.run.smithi169:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T01:45:06.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T01:45:06.039 DEBUG:teuthology.orchestra.run.smithi174:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T01:45:06.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T01:45:06.052 DEBUG:teuthology.orchestra.run.smithi177:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T01:45:06.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T01:45:06.068 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-28T01:45:06.081 INFO:teuthology.task.internal:Transferring archived files... 2024-03-28T01:45:06.082 DEBUG:teuthology.misc:Transferring archived files from smithi169:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625243/remote/smithi169 2024-03-28T01:45:06.082 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T01:45:06.128 DEBUG:teuthology.misc:Transferring archived files from smithi174:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625243/remote/smithi174 2024-03-28T01:45:06.128 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T01:45:06.167 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625243/remote/smithi177 2024-03-28T01:45:06.167 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T01:45:06.206 INFO:teuthology.task.internal:Removing archive directory... 2024-03-28T01:45:06.206 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T01:45:06.208 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T01:45:06.211 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T01:45:06.254 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-28T01:45:06.267 INFO:teuthology.task.internal:Not uploading archives. 2024-03-28T01:45:06.267 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-28T01:45:06.282 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-28T01:45:06.282 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T01:45:06.285 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T01:45:06.286 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T01:45:06.298 INFO:teuthology.orchestra.run.smithi169.stdout: 265699 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 01:45 /home/ubuntu/cephtest 2024-03-28T01:45:06.299 INFO:teuthology.orchestra.run.smithi174.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 01:45 /home/ubuntu/cephtest 2024-03-28T01:45:06.312 INFO:teuthology.orchestra.run.smithi177.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 01:45 /home/ubuntu/cephtest 2024-03-28T01:45:06.313 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-28T01:45:06.327 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-28T01:45:06.356 INFO:teuthology.nuke:Checking targets against current locks 2024-03-28T01:45:06.383 DEBUG:teuthology.nuke:shortname: smithi169 2024-03-28T01:45:06.383 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T01:45:06.403 DEBUG:teuthology.nuke:shortname: smithi174 2024-03-28T01:45:06.404 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T01:45:06.408 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625243', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 00:29:37.922069', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDvqBRAdkinW6eathrE4TDlDVrjB27UcWoRVSmCzPzKFCw0tUqXvPEzss8vK65QBSeXuzWno0S74/iYffI/OgUM='} 2024-03-28T01:45:06.424 DEBUG:teuthology.nuke:shortname: smithi177 2024-03-28T01:45:06.424 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T01:45:06.431 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625243', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 00:29:37.923462', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE6h7J0lVtBzjACbHrnBa3xMal6Zg049NPtROAFInqMLyH5aSYl2cAGBPCnh7kVO8vNtaKX4kn4SDH7uhGbCog0='} 2024-03-28T01:45:06.448 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625243', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 00:29:37.924661', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAK9MjTg33ou9JzFIbnx85ybNfxorqE+LxEwM4+HOpi46iUBIbOrRDZmLvuz2SuqIBSPQbDQhdc056rFTG9dNfQ='} 2024-03-28T01:45:06.457 INFO:teuthology.orchestra.console.smithi169:Power off 2024-03-28T01:45:06.457 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T01:45:06.480 DEBUG:teuthology.orchestra.console.smithi169:power off output: Chassis Power Control: Down/Off 2024-03-28T01:45:06.481 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T01:45:06.496 INFO:teuthology.orchestra.console.smithi174:Power off 2024-03-28T01:45:06.496 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T01:45:06.516 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-28T01:45:06.521 DEBUG:teuthology.orchestra.console.smithi174:power off output: Chassis Power Control: Down/Off 2024-03-28T01:45:06.521 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T01:45:06.537 INFO:teuthology.orchestra.console.smithi177:Power off 2024-03-28T01:45:06.537 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T01:45:06.546 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2024-03-28T01:45:06.559 DEBUG:teuthology.orchestra.console.smithi177:power off output: Chassis Power Control: Down/Off 2024-03-28T01:45:06.559 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T01:45:06.582 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-03-28T01:45:10.516 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T01:45:10.547 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T01:45:10.583 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T01:45:10.626 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-28T01:45:10.657 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2024-03-28T01:45:10.693 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-03-28T01:45:14.630 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T01:45:14.658 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T01:45:14.694 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T01:45:14.742 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-28T01:45:14.768 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2024-03-28T01:45:14.804 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-03-28T01:45:18.747 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T01:45:18.770 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T01:45:18.805 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T01:45:18.861 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2024-03-28T01:45:18.878 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is off 2024-03-28T01:45:18.915 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off 2024-03-28T01:45:18.963 INFO:teuthology.orchestra.console.smithi169:Power off completed 2024-03-28T01:45:18.979 INFO:teuthology.orchestra.console.smithi174:Power off completed 2024-03-28T01:45:19.016 INFO:teuthology.orchestra.console.smithi177:Power off completed 2024-03-28T01:45:19.121 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} duration: 3995.3223309516907 failure_reason: '"2024-03-28T00:50:00.271375+0000 mon.a (mon.0) 414 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled" in cluster log' flavor: default owner: mchangir success: false 2024-03-28T01:45:19.122 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T01:45:19.220 INFO:teuthology.run:FAIL